Culling of Idle JupyterLab Instances
Why do we cull?
Well, the answer is simple enough. Resources are precious.
How do we cull?
JupyterHub uses a mix of the idle
culler service
and internal JupyterLab server configuration to periodically check for
idle servers to shut them down.
We have noticed that internal culler of the JupyterLab server does not reap all the child processes launched within a lab session. This can be a problem in interactive mode as left over processes can keep consuming resources potentially preventing users to launch lab servers. Check You had your slice of Pie mate!! section for more details.
jupyterhub-idle-culler culls servers via JupyterHub's REST API. It makes decisions based on information retrieved by JupyterHub REST API which reports information regularly updated by summarizing information gained by: asking the proxy about routes' activity, and by retaining activity information reported by the servers.
To help jupyterhub-idle-culler cull user servers, we configure the kernel manager to cull idle kernels that would otherwise make the user servers report themselves as active which is part of what jupyterhub-idle-culler considers.
The expected behavior is that the server will be shut down and removed from the JupyterHub namespace once all Terminals and Kernels are considered idle or terminated, as well as any remaining connection is closed.
Default settings
By default, JupyterHub will check for last activity of JupyterLab servers every 5 minutes. Every server found to be idle for more than 1 hour will be terminated.
On the JupyterLab side, we configure the kernels to be culled if they inactive for 1 hour as well. Similarly, JupyterLab server will cull itself if there is no activity for more than 1 hour. So, we have different configurations in place to cull idle servers both on Hub side and user's JupyterLab side.
If the user is executing a long running code in the notebook, normally the server will not cull itself even if there is no interaction from the user. If, by any chance, the server culled itself marking itself as inactive, please report to us.
Postface
Please close the browser tabs of JupyterLab servers when they are terminated (by user), killed (at the end of walltime) or culled.