[yast-devel] services and checks unification for yast modules
Hi, in last year we play a lot with different services and various automatic checks for yast modules. I think it make sense that we after SP2 and 42.2 branching start looking and discussing, what is useful, what not and what is promissing. Ideally if something is useful, then use it everywhere, if something is not useful, then remove POC and if something is promising, then try to invest time in its POC. Reason why I mention it is, that I face it in last days few times, e.g. that some modules do not use rubocop, so when I call rubocop --auto-correct it basically destroy my changes, some modules do not have spellchecker, so when we update in one module CONTRIBUTING.md, it pass, but in other modules it cause crash as osc vc do not pass spellchecker. So I think we can start with discussion what is useful, what not now, even before branching, so we can make conclusion and start working on it. I will write down all projects I am aware of together with its POC to see how it works. Feel free to react to any project how you see it or add project if I forget any. projects: travis build - https://travis-ci.org/yast/yast-yast2 - currently used only for ruby modules and in other projects in other languages jenkins build for pull requests - http://ci.opensuse.org/job/yast-registration-github-push/586//console - currently only in few yast modules coveralls - https://coveralls.io/r/yast/yast-yast2?branch=master - currently in few yast modules, do not support c++ codecov - https://codecov.io/gh/yast/yast-bootloader/pull/355?src=pr and https://codecov.io/gh/jreidinger/libstorage-ng/pull/1?src=pr - currently only POC for bootloader and libstorage-ng codeclimate - https://codeclimate.com/github/yast/yast-yast2 - currently for few yast modules and other ruby projects, do not support c++ codacy - https://www.codacy.com/app/YaST_Team/yast-bootloader/dashboard - similar to codeclimate, only POC for bootloader, can integrate test coverage, comment pull requests inch - http://inch-ci.org/github/yast/yast-registration - checker for documentation coverage, currently used only in yast2-registration pullreview - https://www.pullreview.com/github/yast/yast-registration/reviews/master - sends mails to lslezak, currently only used in yast2-registration rubylint - https://github.com/yast/yast-yast2/pull/480 - checker for errors in ruby, currently only POC rubocop - https://github.com/yast/yast-yast2/blob/master/.rubocop.yml - currently used in many yast modules, but not all spellchecking - https://github.com/yast/yast-yast2/blob/master/.travis.yml#L10 - currently in few yast modules -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Wed, 24 Aug 2016 16:27:03 +0200 Josef Reidinger <jreidinger@suse.cz> wrote:
Hi, in last year we play a lot with different services and various automatic checks for yast modules. I think it make sense that we after SP2 and 42.2 branching start looking and discussing, what is useful, what not and what is promissing. Ideally if something is useful, then use it everywhere, if something is not useful, then remove POC and if something is promising, then try to invest time in its POC. Reason why I mention it is, that I face it in last days few times, e.g. that some modules do not use rubocop, so when I call rubocop --auto-correct it basically destroy my changes, some modules do not have spellchecker, so when we update in one module CONTRIBUTING.md, it pass, but in other modules it cause crash as osc vc do not pass spellchecker. So I think we can start with discussion what is useful, what not now, even before branching, so we can make conclusion and start working on it. I will write down all projects I am aware of together with its POC to see how it works. Feel free to react to any project how you see it or add project if I forget any.
projects:
Hi, I do not see any response, so I will write my view and noone will object, I will start implementing it.
travis build - https://travis-ci.org/yast/yast-yast2 - currently used only for ruby modules and in other projects in other languages
keep travis until jenkins one is ready
jenkins build for pull requests - http://ci.opensuse.org/job/yast-registration-github-push/586//console - currently only in few yast modules
Until we can deploy it for all modules I suggest to keep it in POC state.
coveralls - https://coveralls.io/r/yast/yast-yast2?branch=master - currently in few yast modules, do not support c++
my impression is that similar code coverage provide also code quality services, so we can drop this service and report coverage to quality service.
codecov - https://codecov.io/gh/yast/yast-bootloader/pull/355?src=pr and https://codecov.io/gh/jreidinger/libstorage-ng/pull/1?src=pr - currently only POC for bootloader and libstorage-ng
C++ support is promissing for our C++ projects, so I think we should start checking coverage there.
codeclimate - https://codeclimate.com/github/yast/yast-yast2 - currently for few yast modules and other ruby projects, do not support c++
Currently codacy looks more promissing for me, as it integrate also test coverage ( so we have everything in place ).
codacy - https://www.codacy.com/app/YaST_Team/yast-bootloader/dashboard - similar to codeclimate, only POC for bootloader, can integrate test coverage, comment pull requests
I suggest to use this service to integrate all of it and due to its commenting ability.
inch - http://inch-ci.org/github/yast/yast-registration - checker for documentation coverage, currently used only in yast2-registration
as other services do not check this part I suggest to use it everywhere where service works.
pullreview - https://www.pullreview.com/github/yast/yast-registration/reviews/master - sends mails to lslezak, currently only used in yast2-registration
I think if we use codacy also for pull request commenting, then this one is duplicite.
rubylint - https://github.com/yast/yast-yast2/pull/480 - checker for errors in ruby, currently only POC
for me it is still not usable, so I propose to keep it as POC
rubocop - https://github.com/yast/yast-yast2/blob/master/.rubocop.yml - currently used in many yast modules, but not all
Suggest to use it everywhere where applicable.
spellchecking - https://github.com/yast/yast-yast2/blob/master/.travis.yml#L10 - currently in few yast modules
We should use it everywhere. Josef -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 1.9.2016 09:22, Josef Reidinger wrote:
Hi, I do not see any response, so I will write my view and noone will object, I will start implementing it.
Let's talk about it first. Thanks Lukas -- Lukas Ocilka, Systems Management (Yast) Team Leader SLE Department, SUSE Linux https://en.wikipedia.org/wiki/Scout_Promise#Czech_Republic http://www.scouting.org/Visitor/WhyScouting/ServingOthers.aspx -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 09/01/2016 09:22 AM, Josef Reidinger wrote:
On Wed, 24 Aug 2016 16:27:03 +0200 Josef Reidinger <jreidinger@suse.cz> wrote:
Hi, in last year we play a lot with different services and various automatic checks for yast modules. I think it make sense that we after SP2 and 42.2 branching start looking and discussing, what is useful, what not and what is promissing. Ideally if something is useful, then use it everywhere, if something is not useful, then remove POC and if something is promising, then try to invest time in its POC. Reason why I mention it is, that I face it in last days few times, e.g. that some modules do not use rubocop, so when I call rubocop --auto-correct it basically destroy my changes, some modules do not have spellchecker, so when we update in one module CONTRIBUTING.md, it pass, but in other modules it cause crash as osc vc do not pass spellchecker. So I think we can start with discussion what is useful, what not now, even before branching, so we can make conclusion and start working on it. I will write down all projects I am aware of together with its POC to see how it works. Feel free to react to any project how you see it or add project if I forget any.
projects:
Hi, I do not see any response, so I will write my view and noone will object, I will start implementing it.
Sorry. I forgot. First of all, I wonder about the relation of those hosted services with our mid-term goal of having code quality and code coverage history in a self-hosted fashion. https://trello.com/c/TVjjdrkn/506-yast-metrics IMHO, it would be nice to use the exact same metric for that and for checking incoming pull requests. So far, that means rubycritics for ruby quality (the only locally-runnable that will offer a global rate), simplecov for coverage and yardoc for documentation coverage.
travis build - https://travis-ci.org/yast/yast-yast2 - currently used only for ruby modules and in other projects in other languages
keep travis until jenkins one is ready
+1. Keeping Travis alive is quite some work (it was an endless source of pain for storage-ng), but we should not drop it before having support for pull requests in our Jenkins setup.
jenkins build for pull requests - http://ci.opensuse.org/job/yast-registration-github-push/586//console - currently only in few yast modules
Until we can deploy it for all modules I suggest to keep it in POC state.
This should probably be our main focus as a next step in this area. With jenkins we can potentially check everything (rubocop, test status, test coverage, documentation coverage with yardoc and test quality with rubycritics). Most of these tools are less cool than their hosted counterparts but, as already explained before, this would align Jenkins (CI) with https://github.com/mvidner/metrics (history view).
coveralls - https://coveralls.io/r/yast/yast-yast2?branch=master - currently in few yast modules, do not support c++
my impression is that similar code coverage provide also code quality services, so we can drop this service and report coverage to quality service.
Fine with me.
codecov - https://codecov.io/gh/yast/yast-bootloader/pull/355?src=pr and https://codecov.io/gh/jreidinger/libstorage-ng/pull/1?src=pr - currently only POC for bootloader and libstorage-ng
C++ support is promissing for our C++ projects, so I think we should start checking coverage there.
Doesn't this contradict the previous statement about using a quality service to also check the coverage?
codeclimate - https://codeclimate.com/github/yast/yast-yast2 - currently for few yast modules and other ruby projects, do not support c++
Currently codacy looks more promissing for me, as it integrate also test coverage ( so we have everything in place ).
+1
codacy - https://www.codacy.com/app/YaST_Team/yast-bootloader/dashboard - similar to codeclimate, only POC for bootloader, can integrate test coverage, comment pull requests
I suggest to use this service to integrate all of it and due to its commenting ability.
+1 Just one question, is there bi-directional sync between comments in Github and in that thing or will we have different sets of comments in each place? Well, two questions :-) Does it support C++? I'm trying to add my fork of libstorage-ng but it's taking ages.
inch - http://inch-ci.org/github/yast/yast-registration - checker for documentation coverage, currently used only in yast2-registration
as other services do not check this part I suggest to use it everywhere where service works.
Jenkins + yardoc could do it.
pullreview - https://www.pullreview.com/github/yast/yast-registration/reviews/master - sends mails to lslezak, currently only used in yast2-registration
I think if we use codacy also for pull request commenting, then this one is duplicite.
+1
rubylint - https://github.com/yast/yast-yast2/pull/480 - checker for errors in ruby, currently only POC
for me it is still not usable, so I propose to keep it as POC
+1
rubocop - https://github.com/yast/yast-yast2/blob/master/.rubocop.yml - currently used in many yast modules, but not all
Suggest to use it everywhere where applicable.
+1
spellchecking - https://github.com/yast/yast-yast2/blob/master/.travis.yml#L10 - currently in few yast modules
We should use it everywhere.
+1 (although sometimes it's a little bit annoying). Cheers. -- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On Thu, 1 Sep 2016 10:39:08 +0200 Ancor Gonzalez Sosa <ancor@suse.de> wrote:
On 09/01/2016 09:22 AM, Josef Reidinger wrote:
On Wed, 24 Aug 2016 16:27:03 +0200 Josef Reidinger <jreidinger@suse.cz> wrote:
Hi, in last year we play a lot with different services and various automatic checks for yast modules. I think it make sense that we after SP2 and 42.2 branching start looking and discussing, what is useful, what not and what is promissing. Ideally if something is useful, then use it everywhere, if something is not useful, then remove POC and if something is promising, then try to invest time in its POC. Reason why I mention it is, that I face it in last days few times, e.g. that some modules do not use rubocop, so when I call rubocop --auto-correct it basically destroy my changes, some modules do not have spellchecker, so when we update in one module CONTRIBUTING.md, it pass, but in other modules it cause crash as osc vc do not pass spellchecker. So I think we can start with discussion what is useful, what not now, even before branching, so we can make conclusion and start working on it. I will write down all projects I am aware of together with its POC to see how it works. Feel free to react to any project how you see it or add project if I forget any.
projects:
Hi, I do not see any response, so I will write my view and noone will object, I will start implementing it.
Sorry. I forgot.
First of all, I wonder about the relation of those hosted services with our mid-term goal of having code quality and code coverage history in a self-hosted fashion. https://trello.com/c/TVjjdrkn/506-yast-metrics
IMHO, it would be nice to use the exact same metric for that and for checking incoming pull requests. So far, that means rubycritics for ruby quality (the only locally-runnable that will offer a global rate), simplecov for coverage and yardoc for documentation coverage.
Well, we should at first discuss that goals, how to measure it and so on. And if we agreed on some tool and really invest time into its deployment, we should use it. Also that reports should be publicly visible so non-SUSE employees see metrics and we can have badges on github :) So this sounds like we probably should use some public cloud service for it?
travis build - https://travis-ci.org/yast/yast-yast2 - currently used only for ruby modules and in other projects in other languages
keep travis until jenkins one is ready
+1. Keeping Travis alive is quite some work (it was an endless source of pain for storage-ng), but we should not drop it before having support for pull requests in our Jenkins setup.
jenkins build for pull requests - http://ci.opensuse.org/job/yast-registration-github-push/586//console - currently only in few yast modules
Until we can deploy it for all modules I suggest to keep it in POC state.
This should probably be our main focus as a next step in this area. With jenkins we can potentially check everything (rubocop, test status, test coverage, documentation coverage with yardoc and test quality with rubycritics). Most of these tools are less cool than their hosted counterparts but, as already explained before, this would align Jenkins (CI) with https://github.com/mvidner/metrics (history view).
well, what I am missing probably most is comments to github, that says something like "code coverage decrease by 0.07%", "documentation: +1 documented method, -2 undocumented method" or comments to code what is not so nice or comment to given line that it break rubocop rule ( which I found in some services and sees it very useful ). Maybe it can be also added to jenkins as github have public API, but it is non-trivial work, so we should be sure we have time to do it. If not, we should look for easy to deploy existing solutions.
coveralls - https://coveralls.io/r/yast/yast-yast2?branch=master - currently in few yast modules, do not support c++
my impression is that similar code coverage provide also code quality services, so we can drop this service and report coverage to quality service.
Fine with me.
codecov - https://codecov.io/gh/yast/yast-bootloader/pull/355?src=pr and https://codecov.io/gh/jreidinger/libstorage-ng/pull/1?src=pr - currently only POC for bootloader and libstorage-ng
C++ support is promissing for our C++ projects, so I think we should start checking coverage there.
Doesn't this contradict the previous statement about using a quality service to also check the coverage?
yes, but I do not find any such C++ service, also there is not much tools for c++ for checking quality of code ( probably due to high complexity of c++ code ).
codeclimate - https://codeclimate.com/github/yast/yast-yast2 - currently for few yast modules and other ruby projects, do not support c++
Currently codacy looks more promissing for me, as it integrate also test coverage ( so we have everything in place ).
+1
codacy - https://www.codacy.com/app/YaST_Team/yast-bootloader/dashboard - similar to codeclimate, only POC for bootloader, can integrate test coverage, comment pull requests
I suggest to use this service to integrate all of it and due to its commenting ability.
+1
Just one question, is there bi-directional sync between comments in Github and in that thing or will we have different sets of comments in each place?
I do not get question. I expect we do not write comments in that service, only on github in pull requests.
Well, two questions :-) Does it support C++? I'm trying to add my fork of libstorage-ng but it's taking ages.
no, Codacy supports JavaScript, Scala, Java, PHP, Python, CoffeeScript, CSS and Ruby so no perl and no c++.
inch - http://inch-ci.org/github/yast/yast-registration - checker for documentation coverage, currently used only in yast2-registration
as other services do not check this part I suggest to use it everywhere where service works.
Jenkins + yardoc could do it.
well, we need somehow parse it and track progress.
pullreview - https://www.pullreview.com/github/yast/yast-registration/reviews/master - sends mails to lslezak, currently only used in yast2-registration
I think if we use codacy also for pull request commenting, then this one is duplicite.
+1
rubylint - https://github.com/yast/yast-yast2/pull/480 - checker for errors in ruby, currently only POC
for me it is still not usable, so I propose to keep it as POC
+1
rubocop - https://github.com/yast/yast-yast2/blob/master/.rubocop.yml - currently used in many yast modules, but not all
Suggest to use it everywhere where applicable.
+1
spellchecking - https://github.com/yast/yast-yast2/blob/master/.travis.yml#L10 - currently in few yast modules
We should use it everywhere.
+1 (although sometimes it's a little bit annoying).
Cheers.
Thanks for answers. Josef -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
On 09/01/2016 11:00 AM, Josef Reidinger wrote:
On Thu, 1 Sep 2016 10:39:08 +0200 Ancor Gonzalez Sosa <ancor@suse.de> wrote:
On 09/01/2016 09:22 AM, Josef Reidinger wrote:
On Wed, 24 Aug 2016 16:27:03 +0200 Josef Reidinger <jreidinger@suse.cz> wrote:
codacy - https://www.codacy.com/app/YaST_Team/yast-bootloader/dashboard - similar to codeclimate, only POC for bootloader, can integrate test coverage, comment pull requests
I suggest to use this service to integrate all of it and due to its commenting ability.
+1
Just one question, is there bi-directional sync between comments in Github and in that thing or will we have different sets of comments in each place?
I do not get question. I expect we do not write comments in that service, only on github in pull requests.
Well, two questions :-) Does it support C++? I'm trying to add my fork of libstorage-ng but it's taking ages.
no, Codacy supports JavaScript, Scala, Java, PHP, Python, CoffeeScript, CSS and Ruby
so no perl and no c++.
In fact, with some manual tweaking it does support C++ according to my experiments. Check this (use the "search file" filter to look for .cc files): https://www.codacy.com/app/ancorgs/libstorage-ng/files Cheers. -- Ancor González Sosa YaST Team at SUSE Linux GmbH -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 24.8.2016 v 16:27 Josef Reidinger napsal(a): [...]
Reason why I mention it is, that I face it in last days few times, e.g. that some modules do not use rubocop, so when I call rubocop --auto-correct it basically destroy my changes, some modules do not
There are several possibilities how to avoid this: 1) first run plain "rubocop" to see what issues it finds, if the result are reasonable run "rubocop -a", if the first scan reports hundreds issues something is very likely wrong... 2) commit your current changes first, run "rubocop -a", check the diff, if ok amend the commit, if not you can easily revert the changes 3) use Overcommit [1][2], this is basically the same as 1) but the "rubocop" part is called automatically by git hooks and additionally it runs rubocop only on the changed files so it's way faster. [...]
travis build - https://travis-ci.org/yast/yast-yast2 - currently used only for ruby modules and in other projects in other languages
Travis uses a completely different distro on the nodes (Ubuntu 12.04 or 14.04) which makes troubles when your project requires the latest compiler, Ruby, some libraries... And you cannot build the final RPM there, you can never be sure that you did not make a typo in *.spec file...
jenkins build for pull requests - http://ci.opensuse.org/job/yast-registration-github-push/586//console - currently only in few yast modules
This actually builds commits in all branches, not the pull requests, that feature is still missing. This is one step forward to replace Travis to avoid issues mentioned above, but we are still not there. In the long term we should replace Travis by Jenkins.
coveralls - https://coveralls.io/r/yast/yast-yast2?branch=master - currently in few yast modules, do not support c++
Actually there is a C++ support, it's provided by an external tool, see [3]. But I haven't tried it... You mention some other services related to code coverage below, I think we should evaluate pros&cons of each of them and stick to one solution if possible. Coveralls looks good to me, but maybe the other tools have some advantages. We will probably need to do some research here...
codecov - https://codecov.io/gh/yast/yast-bootloader/pull/355?src=pr and https://codecov.io/gh/jreidinger/libstorage-ng/pull/1?src=pr - currently only POC for bootloader and libstorage-ng
codeclimate - https://codeclimate.com/github/yast/yast-yast2 - currently for few yast modules and other ruby projects, do not support c++
Code climate is a nice tool, I like the ability to compare branches. I you do some bigger changes you can check whether the code quality improves or not before merging it. So far it's not configured to set status in Github, but I'll try it to see how it works. Additionaly they recently released a browser plugin [4] which adds the results directly into the pages at Github, but I haven't tried it yet.
codacy - https://www.codacy.com/app/YaST_Team/yast-bootloader/dashboard - similar to codeclimate, only POC for bootloader, can integrate test coverage, comment pull requests
And what is your experience with that tool? That dashboard looks a bit puzzling to me, it says "Project Certification A" and "Code Style 100%" but if you click the "Project quality" tab in the right bottom part you will see even some grade "F" parts. But I was not able to get more details like which files/methods are bad...
inch - http://inch-ci.org/github/yast/yast-registration - checker for documentation coverage, currently used only in yast2-registration
This is a completely server side service, the setup is trivial and it does not depend on Travis or Jenkins. Yardoc itself reports the documentation coverage, but the nice inch feature is that it rather suggests the places which should be improved. And it also prioritizes the issues, e.g. documenting a public method is more important than documenting a private method. See the registration example. It's not a killer tool, but if you have some time and want to improve the code documentation it can help you to find the most problematic places. But maybe some other tools can help with this as well. What do you think about it?
pullreview - https://www.pullreview.com/github/yast/yast-registration/reviews/master - sends mails to lslezak, currently only used in yast2-registration
This is similar to codeclimate, but I do not see a real value here, the codeclimate UI looks better IMHO and provides better results. After some time I stopped checking it and I ignore it now. On the other hand it runs some additional/different checks when compared to codeclimate. But we should prefer a single tool, check the link and see if it is worth trying it more...
rubylint - https://github.com/yast/yast-yast2/pull/480 - checker for errors in ruby, currently only POC
IIRC it reported too many false positives, so until it becomes usable is should stay PoC and just keep an eye on it.
rubocop - https://github.com/yast/yast-yast2/blob/master/.rubocop.yml - currently used in many yast modules, but not all
It should be used everywhere ;-)
spellchecking - https://github.com/yast/yast-yast2/blob/master/.travis.yml#L10 - currently in few yast modules
Again, IMHO this should be used more often. It's a bit annoying that you need to put many unknown words to the custom dictionary (the specific technical terms we usually use might not be covered) but on the other hand it helped me to find quite a lot of typos already. # Overcommit And here I'm adding one more tool to the list - Overcommit. I already mentioned it at the beginning, see [1] and [2] for more details. I have been using it for several months and it looks good. The only issue I found is that currently the .overcommit.yml config files are not committed to Git. When I run "rake osc:build" it complains about extra file in the checkout. So I remove it. But when I need to do some more fixes and build the package several times then I can easily forget to restore the config file back and then overcommit uses the default configuration which does not include running rubocop or running the tests. The main reason for using it is gone. Therefore I'd like to submit the .overcommit.yml file to Git. If you do not want to use overcommit then simply do not install it. By default git does not run the hooks, you have to manually enable them in the local checkout. (And you can still disable it for a single commit via "OVERCOMMIT_DISABLE=1 git commit".) What do you think about it? Ladislav [1] http://blog.ladslezak.cz/2016/06/06/overcommit/ [2] https://github.com/brigade/overcommit [3] https://github.com/eddyxu/cpp-coveralls [4] https://codeclimate.com/browser-extension -- Ladislav Slezák Appliance department / YaST Developer Lihovarská 1060/12 190 00 Prague 9 / Czech Republic tel: +420 284 028 960 lslezak@suse.com SUSE -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
participants (4)
-
Ancor Gonzalez Sosa -
Josef Reidinger -
Ladislav Slezak -
Lukas Ocilka