Re: [che-dev] che-bot PR comments - annoying or useful?

[Radim Hopp <rhopp@xxxxxxxxxx>]

Thanks for this discussion. I hope this email answers/explains most of the questions here.

Right now, as we are using jenkins, we use Jenkins github pull request builder plugin. This is the easiest way for us how to trigger PR checks and report back to the issue the result or anything else we can (like the image used for testing, URL's to the "try it now" workspaces on che.openshift.io, ...).

This plugin unfortunately (AFAIK) has some limitations in the reporting back to the issue:

1/ First way is putting a comment when the job succeeds/fails. There are basically no limitations here and we can put as much info there as required. It seems, this approach is not desirable as it could generate a lot of notifications, if the PR is still WIP/in active development.

2/ Second way the plugin is using to report back to the commit is Github statuses API. This just puts this line (screenshot below) to the PR. But the amount of info we can put there is very limited.  The message we can put there must be shorter than 140chars. Name of the status and the "details" link is automatically generated by the plugin. 

Yesterday I was looking and experimenting with GitHub checks API (result of my experiments). I think this would allow us to solve all the problems in nice manner, as it allows us to utilize the "Checks" tab Angel was talking about. But it would be relatively hard (or time consuming), as we would need to implement the reporting from jobs to PRs from scratch (and for every job we are using), instead of just reusing the Jenkins plugin capabilities (Or implement this functionality in the jenkins plugin itself of course 😀)

Another concern/idea was to keep editing one comment. Again - if we were to reimplement the reporting back to PR by ourselves, this would be possible, but the plugin we are using right now doesn't allow this.

I was also trying to find if the plugin is capable of not triggering the job, if the PR is in draft/WIP state, or if it has some label. Unfortunately I wasn't able to find anything in this regards, so I assume it doesn't have such capability.

As a conclusion I would like to propose plano to lower the comment&notification clutter:

On che repo:
* Disable comments from job running selenium tests on codenvy ci - Only `ci-integration-tests` check will remain

* For job running Happy Path I would suggest to turn off the "successful run" comments and leave the failed ones.

On che-theia repo:

* It seems to me, that contributors to this repo likes the comments in general, so I'm not sure if any change is required here.

I hope this helps.

Radim

On Thu, Dec 5, 2019 at 7:01 PM Angel Misevski <amisevsk@xxxxxxxxxx> wrote:

There is also a "Checks" tab on GitHub, e.g. [1], but I don't know if
it's accessible to our CI reporting:

[1] - https://github.com/eclipse/che/pull/15386/checks

On 12/5/19 11:21 AM, Dmytro Nochevnov wrote:
> I am afraid it's impossible to enclose all PR check results in ONE
> comment so as those comments have been sent from different CI and from
> different Jenkins builds asynchronously.
>
> At the same time we can make it possible to DISABLE PR check execution
> by providing special command to leave in PR title/description (e.g.
> *skip-pr-check*).
>
> On Thu, Dec 5, 2019 at 5:59 PM Michal Vala <mvala@xxxxxxxxxx
> <mailto:mvala@xxxxxxxxxx>> wrote:
>
>     Looks like info how to test changes, like name of built image or
>     link to che.openshift.io <http://che.openshift.io>, is important. I
>     can see that both of these names are based on PR id, thus it's the
>     same for all lifetime of the PR. So maybe just one comment after PR
>     is created with all test links would help with this. WDYT?
>
>     On Thu, Dec 5, 2019 at 2:59 PM Vladyslav Zhukovskyi
>     <vzhukovs@xxxxxxxxxx <mailto:vzhukovs@xxxxxxxxxx>> wrote:
>
>         I’d rather disable commenting PR on each commit and leave the
>         link to the test report in Details section:
>
>
>         Also, it would better to not perform testing when PR is in draft
>         mode. + Make ci-integration-tests required and allow PR merge
>         only when tests successfully passed. It would be incentive for
>         us to keep master stable.
>
>>         5 дек. 2019 г., в 15:44, Dmytro Nochevnov <dnochevn@xxxxxxxxxx
>>         <mailto:dnochevn@xxxxxxxxxx>> написал(а):
>>
>>         Hello everyone.
>>
>>         I would also make it clear why it was decided to leave che-bot
>>         comments: Che QE team don't have much resources to answer the
>>         questions like "/Where to find test/product logs?/", "/Which
>>         test(s) failed?/", "/Where to get product images which the
>>         tests were run against?/".
>>         Alternatively, we can prepare a short document about where to
>>         find that information in Jenkins build logs / artifacts, or it
>>         could be put into a short description of Jenkins build.
>>
>>         @Mykola Morhun <mailto:mmorhun@xxxxxxxxxx>:
>>
>>             Will it be possible to quickly (without digging in job
>>             details) see built by Ci image name?
>>
>>         Did you mean an addresses like "maxura/che-server:14540" in PR
>>         check status message on GitHub page?
>>
>>         On Thu, Dec 5, 2019 at 3:29 PM Mykola Morhun
>>         <mmorhun@xxxxxxxxxx <mailto:mmorhun@xxxxxxxxxx>> wrote:
>>
>>             Hello.
>>             Will it be possible to quickly (without digging in job
>>             details) see built by Ci image name?
>>
>>             On Thu, Dec 5, 2019 at 3:19 PM Radim Hopp
>>             <rhopp@xxxxxxxxxx <mailto:rhopp@xxxxxxxxxx>> wrote:
>>
>>                 Thanks for raising this topic.
>>                 Based on the comments I think it's safe to disable the
>>                 bot comments.
>>                 I'll wait one or two more days to see if any voices
>>                 requesting the bot-comments to stay/change will rise
>>                 up (I don't expect that). Otherwise I'll disable the
>>                 comments at all and leave just the github status check.
>>                 To the idea of editing one comment - we are using
>>                 jenkins job builder plugin and I don't know if such
>>                 functionality is there (I don't think so)
>>                 To Angel's nitpick - We are aware of the fact, that
>>                 having jobs behind VPN is not community friendly at
>>                 all. And it's on our to-do list to move them from
>>                 there right after we move jobs from codenvy CI infra.
>>
>>                 P.S.: I was quickly experimenting with putting the
>>                 info from comment into the text of the status check,
>>                 as it also supports markdown
>>                 (https://developer.github.com/v3/checks/runs/), but it
>>                 seems that the `message` argument is limited with
>>                 maximum size of 140 chars (this is not mentioned in
>>                 the github docs), so it's not fit for our usecase. But
>>                 anyway... The "details" link stays there and all
>>                 required info should be accessible from there.
>>
>>                 On Thu, Dec 5, 2019 at 12:40 PM Sun Tan
>>                 <sutan@xxxxxxxxxx <mailto:sutan@xxxxxxxxxx>> wrote:
>>
>>                     Updating a comment does not send any new
>>                     notifications ...
>>
>>                     Sun Tan
>>                     Senior Software Engineer
>>                     /Eclipse Che - CodeReady Workspaces /@ Red Hat
>>                     /Paris JUG leader/
>>
>>                     Red Hat Paris <https://www.redhat.com/>
>>
>>                     sutan@xxxxxxxxxx <mailto:sutan@xxxxxxxxxx>
>>                     M: +33621024173 <tel:+33621024173>
>>                     @sunsengdavidtan
>>                     <https://twitter.com/sunsengdavidtan> me
>>                     <https://www.linkedin.com/in/sun-seng-david-tan-b05a684/>
>>
>>                     <https://www.redhat.com/>
>>
>>
>>
>>                     On Thu, Dec 5, 2019 at 12:06 PM Michal Vala
>>                     <mvala@xxxxxxxxxx <mailto:mvala@xxxxxxxxxx>> wrote:
>>
>>                         If updating a comment fires the notification,
>>                         I'm against. Still option to enable/disable
>>                         looks better to me.
>>
>>                         On Thu, Dec 5, 2019 at 10:59 AM Sun Tan
>>                         <sutan@xxxxxxxxxx <mailto:sutan@xxxxxxxxxx>>
>>                         wrote:
>>
>>                             maybe the bot could just update the same
>>                             comment, adding or updating the status
>>
>>                             Sun Tan
>>                             Senior Software Engineer
>>                             /Eclipse Che - CodeReady Workspaces /@ Red Hat
>>                             /Paris JUG leader/
>>
>>                             Red Hat Paris <https://www.redhat.com/>
>>
>>                             sutan@xxxxxxxxxx <mailto:sutan@xxxxxxxxxx>
>>                             M: +33621024173 <tel:+33621024173>
>>                             @sunsengdavidtan
>>                             <https://twitter.com/sunsengdavidtan> me
>>                             <https://www.linkedin.com/in/sun-seng-david-tan-b05a684/>
>>
>>                             <https://www.redhat.com/>
>>
>>
>>
>>                             On Mon, Dec 2, 2019 at 11:39 PM Angel
>>                             Misevski <amisevsk@xxxxxxxxxx
>>                             <mailto:amisevsk@xxxxxxxxxx>> wrote:
>>
>>
>>
>>                                 On 11/27/19 8:04 AM, Sergii Kabashniuk
>>                                 wrote:
>>                                 >
>>                                 >
>>                                 > On Wed, Nov 27, 2019 at 1:44 PM
>>                                 Thomas Mäder <tmader@xxxxxxxxxx
>>                                 <mailto:tmader@xxxxxxxxxx>
>>                                 > <mailto:tmader@xxxxxxxxxx
>>                                 <mailto:tmader@xxxxxxxxxx>>> wrote:
>>                                 >
>>                                 >
>>                                 >     On 27/11/2019 12:58, Michal Vala
>>                                 wrote:
>>                                 >      > I'd like to propose to
>>                                 disable che-bot comments by default and
>>                                 >     make it
>>                                 >      > possible to enable it with a
>>                                 label. If disable by default is not
>>                                 >      > acceptable for whatever
>>                                 reason, at least make it possible to
>>                                 disable
>>                                 >      > che-bot comments with a label.
>>                                 >      >
>>                                 >      > What do you think?
>>                                 >
>>                                 >     +1
>>                                 >
>>                                 > +1
>>
>>                                 I'm also +1, but to add to the
>>                                 question: how useful are the comments
>>                                 left by che-bot in general?
>>
>>                                 The bot comment includes
>>                                 - A link to the build
>>                                 - A link to the test report (which can
>>                                 be retrieved from the build link)
>>                                 - the built-and-tagged image, if
>>                                 applicable
>>
>>                                 I've found having easy access to e.g.
>>                                 the built image for testing very
>>                                 useful in the past, but I don't know
>>                                 if having that is worth the extreme
>>                                 amount of comment spam we can have.
>>
>>                                 In addition, the happy path and OCP
>>                                 e2e tests seem to output different
>>                                 images, e.g. maxura/che-server:14540
>>                                 vs eclipseche/che-server:14540.
>>
>>                                 If we disable bot comments completely,
>>                                 the jobs should still be
>>                                 accessible via the details link in the
>>                                 checks section of the merge UI,
>>                                 so it would seem that having the
>>                                 comments at all is not incredibly useful.
>>
>>                                 Side nitpick: it's kind of strange
>>                                 that one of the CI jobs links to a CI
>>                                 that requires RH VPN.
>>
>>                                 >
>>                                 >
>>                                 >     /Thomas
>>                                 >
>>                                 >   
>>                                  _______________________________________________
>>                                 >     che-dev mailing list
>>                                 > che-dev@xxxxxxxxxxx
>>                                 <mailto:che-dev@xxxxxxxxxxx><mailto:che-dev@xxxxxxxxxxx
>>                                 <mailto:che-dev@xxxxxxxxxxx>>
>>                                 >     To change your delivery options,
>>                                 retrieve your password, or
>>                                 >     unsubscribe from this list, visit
>>                                 >
>>                                 https://www.eclipse.org/mailman/listinfo/che-dev
>>                                 >
>>                                 >
>>                                 >
>>                                 > --
>>                                 >
>>                                 > Sergii Kabashniuk
>>                                 >
>>                                 > Principal Software Engineer, DevTools
>>                                 >
>>                                 > Red Hat
>>                                 >
>>                                 >skabashniuk@xxxxxxxxxx
>>                                 <mailto:skabashniuk@xxxxxxxxxx><mailto:skabashniuk@xxxxxxxxxx
>>                                 <mailto:skabashniuk@xxxxxxxxxx>>
>>                                 >
>>                                 >
>>                                 >
>>                                 _______________________________________________
>>                                 > che-dev mailing list
>>                                 >che-dev@xxxxxxxxxxx
>>                                 <mailto:che-dev@xxxxxxxxxxx>
>>                                 > To change your delivery options,
>>                                 retrieve your password, or unsubscribe
>>                                 from this list, visit
>>                                 >https://www.eclipse.org/mailman/listinfo/che-dev
>>                                 >
>>
>>                                 _______________________________________________
>>                                 che-dev mailing list
>>                                 che-dev@xxxxxxxxxxx
>>                                 <mailto:che-dev@xxxxxxxxxxx>
>>                                 To change your delivery options,
>>                                 retrieve your password, or unsubscribe
>>                                 from this list, visit
>>                                 https://www.eclipse.org/mailman/listinfo/che-dev
>>
>>                             _______________________________________________
>>                             che-dev mailing list
>>                             che-dev@xxxxxxxxxxx
>>                             <mailto:che-dev@xxxxxxxxxxx>
>>                             To change your delivery options, retrieve
>>                             your password, or unsubscribe from this
>>                             list, visit
>>                             https://www.eclipse.org/mailman/listinfo/che-dev
>>
>>
>>
>>                         --
>>                         Michal Vala
>>                         Software Engineer, Eclipse Che
>>                         Red Hat Czech
>>                         _______________________________________________
>>                         che-dev mailing list
>>                         che-dev@xxxxxxxxxxx <mailto:che-dev@xxxxxxxxxxx>
>>                         To change your delivery options, retrieve your
>>                         password, or unsubscribe from this list, visit
>>                         https://www.eclipse.org/mailman/listinfo/che-dev
>>
>>                     _______________________________________________
>>                     che-dev mailing list
>>                     che-dev@xxxxxxxxxxx <mailto:che-dev@xxxxxxxxxxx>
>>                     To change your delivery options, retrieve your
>>                     password, or unsubscribe from this list, visit
>>                     https://www.eclipse.org/mailman/listinfo/che-dev
>>
>>                 _______________________________________________
>>                 che-dev mailing list
>>                 che-dev@xxxxxxxxxxx <mailto:che-dev@xxxxxxxxxxx>
>>                 To change your delivery options, retrieve your
>>                 password, or unsubscribe from this list, visit
>>                 https://www.eclipse.org/mailman/listinfo/che-dev
>>
>>
>>
>>             --
>>             MykolaMorhun
>>             Software Engineer
>>
>>             Red Hat<https://www.redhat.com/>
>>
>>             <https://www.redhat.com/>
>>
>>
>>
>>         --
>>         With best regards,
>>         Dmytro Nochevnov
>>         Senior Quality Engineer
>>         Che QE team
>>         _______________________________________________
>>         che-dev mailing list
>>         che-dev@xxxxxxxxxxx <mailto:che-dev@xxxxxxxxxxx>
>>         To change your delivery options, retrieve your password, or
>>         unsubscribe from this list, visit
>>         https://www.eclipse.org/mailman/listinfo/che-dev
>
>         _______________________________________________
>         che-dev mailing list
>         che-dev@xxxxxxxxxxx <mailto:che-dev@xxxxxxxxxxx>
>         To change your delivery options, retrieve your password, or
>         unsubscribe from this list, visit
>         https://www.eclipse.org/mailman/listinfo/che-dev
>
>
>
>     --
>     Michal Vala
>     Software Engineer, Eclipse Che
>     Red Hat Czech
>     _______________________________________________
>     che-dev mailing list
>     che-dev@xxxxxxxxxxx <mailto:che-dev@xxxxxxxxxxx>
>     To change your delivery options, retrieve your password, or
>     unsubscribe from this list, visit
>     https://www.eclipse.org/mailman/listinfo/che-dev
>
>
>
> --
> With best regards,
> Dmytro Nochevnov
> Senior Quality Engineer
> Che QE team
>
> _______________________________________________
> che-dev mailing list
> che-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/che-dev
>

_______________________________________________
che-dev mailing list
che-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/che-dev