So, we're finished investigation of this case, and we propose the following resolution and troubleshooting actions:
Issue details:
Due to error in DB schema introduced in version 6.10, there is possibility of situation when further rolling updates are failing.
This error affects workspaces which were in a running state during update to 6.10 version.
They are possibly lost theirs public/private signature key pairs and therefore their runtime states cannot be recovered during rolling update.
For more technical details, see
https://github.com/eclipse/che/issues/11087
Troubleshooting:
Che version 6.11 will contain fix of DB schema, but it is
strongly recommended to manually restart workspaces which may be affected by the error.
If those workspaces will not be restarted, after the automatic DB fix they may remain in undefined state that can be fixed by restart.
To determine lists of affected workspaces, the following query should be run on the Che DB before starting update:
SELECT kp.workspace_id FROM che_sign_key_pair kp
WHERE NOT EXISTS (
SELECT id
FROM che_sign_key k
WHERE k.id = kp.private_key
)
OR NOT EXISTS (
SELECT id
FROM che_sign_key k
WHERE k.id = kp.public_key
)
An empty result means that no workspaces are affected and no more manual actions needed.
After damaged workspaces restart finish, rolling update may be performed as usual.
If you require any further information, feel free to contact us.
On Thu, Sep 6, 2018 at 4:11 PM, Dave Neary <
dneary@xxxxxxxxxx> wrote:
>
> Thanks!
> Dave.
>
>
> On 09/06/2018 08:59 AM, Sergii Kabashniuk wrote:
> >
> >
> > On Thu, Sep 6, 2018 at 3:57 PM Dave Neary <
dneary@xxxxxxxxxx> > <mailto:
dneary@xxxxxxxxxx>> wrote:
> >
> > Hi,
> >
> > On 09/06/2018 08:50 AM, Sergii Kabashniuk wrote:
> > > @Dave you are asking about what to do in this particular use case
> > > or in general how to do an upgrade between versions?
> >
> > In this specific case. In your original email you said what not to do
> > (hot update) - and I'm just checking that the way I have upgraded Che in
> > the past (via a new deployment) is not going to hit the issue you
> > pointed to.
> >
> > yes, you right.
> >
> >
> > Thanks,
> > Dave.
> >
> > > On Thu, Sep 6, 2018 at 3:48 PM Dave Neary <
dneary@xxxxxxxxxx> > <mailto:
dneary@xxxxxxxxxx>
> > > <mailto:
dneary@xxxxxxxxxx <mailto:
dneary@xxxxxxxxxx>>> wrote:
> > >
> > > Thanks Sergii,
> > >
> > > Is the recommended way to do an update to do a new deployment on
> > > K8s/OpenShift? This is how I usually do it, and I had
> > understood that
> > > this would stop workspaces and the master, and redeploy and
> > restart
> > > them. Is this correct?
> > >
> > > Thanks,
> > > Dave.
> > >
> > > On 09/06/2018 05:47 AM, Sergii Kabashniuk wrote:
> > > > Hello
> > > > We noticed undesired behavior during the hot update of
> > Eclipse Che
> > > 6.9.0
> > > > -> 6.10.0 in case if you have running workspaces at that time.
> > > > As a result, you may have a state of a master then you can't
> > > restart it.
> > > > We are investigating this issue.
> > > > Our recommendation for now:
> > > > - Do not do hot update 6.9.0 -> 6.10.0
> > > > - If you need an update we recommend to stop all workspace
> > before you
> > > > are going to do that.
> > > > - Make DB backup before and after update
> > > >
> > > > --
> > > >
> > > > Sergii Kabashniuk
> > > >
> > > > Principal Software Engineer, DevTools
> > > >
> > > > Red Hat Ukraine
> > > >
> > > >
skabashniuk@xxxxxxxxxx <mailto:
skabashniuk@xxxxxxxxxx>
> > <mailto:
skabashniuk@xxxxxxxxxx <mailto:
skabashniuk@xxxxxxxxxx>>
> > > <mailto:
skabashniuk@xxxxxxxxxx <mailto:
skabashniuk@xxxxxxxxxx>
> > <mailto:
skabashniuk@xxxxxxxxxx <mailto:
skabashniuk@xxxxxxxxxx>>>
> > > >
> > >
> > > --
> > > Dave Neary - Open Source and Standards team
> > > Red Hat -
http://community.redhat.com> > > Ph: +1-978-399-2182 / Cell: +1-978-799-3338
> > >
> > >
> > >
> > > --
> > >
> > > Sergii Kabashniuk
> > >
> > > Principal Software Engineer, DevTools
> > >
> > > Red Hat Ukraine
> > >
> > >
skabashniuk@xxxxxxxxxx <mailto:
skabashniuk@xxxxxxxxxx>
> > <mailto:
skabashniuk@xxxxxxxxxx <mailto:
skabashniuk@xxxxxxxxxx>>
> > >
> >
> > --
> > Dave Neary - Open Source and Standards team
> > Red Hat -
http://community.redhat.com> > Ph: +1-978-399-2182 / Cell: +1-978-799-3338
> >
> >
> >
> > --
> >
> > Sergii Kabashniuk
> >
> > Principal Software Engineer, DevTools
> >
> > Red Hat Ukraine
> >
> >
skabashniuk@xxxxxxxxxx <mailto:
skabashniuk@xxxxxxxxxx>
> >
>
> --
> Dave Neary - Open Source and Standards team
> Red Hat -
http://community.redhat.com> Ph: +1-978-399-2182 / Cell: +1-978-799-3338
>
