Re: [che-dev] How to collect and persist all workspace logs?

[Michal Vala <mvala@xxxxxxxxxx>]

We've figured it out. Goals of effort Anatolii mentioned are mostly
targeted to admin, for collecting logs of current workspace and
che-server, and it is only for chectl. Our effort will not depend on
deployment, will primary target the user and user's workspaces and
will be there when something wrong happen with at least 5 last
workspace instances. At the end, chectl can consume new logs API of
che-server.

On Thu, Jan 23, 2020 at 11:02 AM Gennady Azarenkov <gazarenk@xxxxxxxxxx> wrote:
>
> @Anatolii Bazko is it a really duplicating? is not it Administrator Logs (yours) vs User Logs (Michal)?
> (though consolidated technical approach is always a good idea)
>
>
> On Thu, Jan 23, 2020 at 11:22 AM Thomas Mäder <tmader@xxxxxxxxxx> wrote:
>>
>> If it helps, there's info about logging in
>> https://github.com/eclipse/che/wiki/Language-support-troubleshooting-document
>>
>> On 23/01/2020 05:04, Michal Vala wrote:
>> > Hello team,
>> >
>> > we're currently working on improving diagnosis capabilities[1] of workspaces, to
>> > be more concrete, how to get all logs from the workspace[2]. We're in phase of
>> > investigating options and prototyping and we've came up with several variants
>> > how to achieve the goal. We would like to know your opinion and new ideas.
>> >
>> > Requirements:
>> >    - collect all logs of all containers from the workspace
>> >    - stdout/err as well as file logs inside the container
>> >    - keep history of last 5 runs of the workspace
>> >    - collect logs of crashed workspace
>> >    - make logs easily accessible to the user (rest API + dashboard view)
>> >
>> >
>> > I've splitted the effort into two sections:
>> >
>> >    ### How to collect:
>> >
>> >      # log everything to files to mounted PV
>> >        - just mount PV and log everything there
>> >        - pros
>> >          - not much extra overhead, only write stdout/err to the file
>> > and mount PV
>> >          - don't need extra hw resources (memory/cpu)
>> >        - cons
>> >          - we might need to override `command` of all containers. They will
>> >            have to run with extra parameters to write stdout/err to the file.
>> >            Something like `<command> 2>&1 | tee ws.log`
>> >
>> >      # workspace collector sidecar (kubernetes/client-go app?)
>> >        - pros
>> >          - per workspace
>> >          - dynamic and powerful
>> >        - cons
>> >          - very custom solution and might be hard to manage/maintain
>> >          - unknown performance and hw resources requirements
>> >          - hard when ws crash
>> >          - need more memory per workspace, even if user does not use it and
>> >            everything works as expected
>> >
>> >      # watch and collect from master
>> >        - pros
>> >          - easy to grab logs and events
>> >          - easy to access archived logs
>> >        - cons
>> >          - only container's stderr/out
>> >          - keep the connection to ws
>> >          - more network traffic
>> >          - increase memory footprint of mastaer
>> >
>> >      # kubernetes native
>> >        - change the logging backend of kubernetes [3]
>> >        - pros
>> >          - standard k8s way, "googleable"
>> >        - cons
>> >          - depends on kubernetes deployment
>> >          - needs extra cluster component/configuration
>> >          - only stdout/err of containers
>> >
>> >      # push logs directly from containers to logging backend
>> >        - cons
>> >          - customize all components to log to the backend
>> >          - performance and hw resources overhead
>> >
>> >      # collect on workspace exit
>> >        - mount PV and log there. When workspace exits, start collector pod that
>> >            grabs the logs and "archive" them.
>> >        - pros
>> >          - not much extra overhead
>> >        - cons
>> >          - don't have logs of running workspace
>> >          - custom collector pod
>> >
>> >
>> >    ### Where to store and how to access:
>> >
>> >      # Workspace PV
>> >        - pros
>> >          - easy to set quota per user
>> >        - cons
>> >          - harder to access (need to start some pod at workspace's namespace)
>> >          - lost when delete namespace
>> >
>> >      # Che PV
>> >        - pros
>> >          - easier to access
>> >        - cons
>> >          - harder to set quota per user
>> >          - harder to scale and manage
>> >          - possible performance bottleneck
>> >
>> >      # PostgreSQL
>> >        - pros
>> >          - the easiest to access
>> >        - cons
>> >          - harder to set quota per user
>> >          - harder to scale and manage
>> >          - possible performance bottleneck
>> >
>> >
>> > There is one remaining and very important question we have not investigated
>> > much. We need to somehow configure all plugins/editors and other components, to
>> > tell where they have all log files that should be collected. Otherwise, we
>> > would not be able to find the logs on containers. We would need to
>> > handle that in
>> > plugin's `meta.yaml` as well as in the devfile.
>> >
>> > What's next?
>> >    We would like to investigate and prototype following solution:
>> >      - collect all ws logs into files and store in PV in the workspace
>> >      - watch ws events from master and on exit, start the collector pod that will
>> >        collect all the logs and pass them to the backend. Logs backend
>> > is something
>> >        to be done. It might be only PV dedicated to archiving log, or some new
>> >        service, or Che master.
>> >      - prototype new Che master API to access the logs. If we store
>> > them in workspace's PV,
>> >        start the collector pod on demand to access the logs.
>> >
>> >
>> > We would very much welcome any opinions or ideas.
>> >
>> >
>> > [1] - https://github.com/eclipse/che/issues/15047
>> > [2] - https://github.com/eclipse/che/issues/15134
>> > [3] - https://kubernetes.io/docs/concepts/cluster-administration/logging/#sidecar-container-with-a-logging-agent
>> >
>> > _______________________________________________
>> > che-dev mailing list
>> > che-dev@xxxxxxxxxxx
>> > To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> > https://www.eclipse.org/mailman/listinfo/che-dev
>> >
>>
>> _______________________________________________
>> che-dev mailing list
>> che-dev@xxxxxxxxxxx
>> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
>> https://www.eclipse.org/mailman/listinfo/che-dev
>>
>>
> _______________________________________________
> che-dev mailing list
> che-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/che-dev



-- 
Michal Vala
Software Engineer, Eclipse Che
Red Hat Czech