There are a lot of low- and medium-hanging fruit in this space.
I take the view that there is a "warmth hierarchy" (an idea I spitballed with other folks in Knative before Matt Moore took it and filled it out a lot).
You have a request or message. You need to process it. The hierarchy runs CPU > Memory > Disk > Cold.
CPU-warm: immediately schedulable by the OS. Order of nanoseconds.
Memory-warm: in-memory but can't be immediately scheduled. Our reference case is processes that have been cgroup frozen. Order of tens of milliseconds.
Disk-warm: the necessary layers are all present on the node which is assigned to launch the process. Order of hundreds of milliseconds.
Cold start: nothing is present on the node which is assigned to launch the process. It will need to fetch some or all layers from a registry. Order of seconds to minutes.
Some of these problems can't be solved by Kubernetes (it doesn't know how to freeze a process), some of them can't be solved by Docker (it doesn't know about other nodes that might have warmer disk caches). Then there's the business of deciding where to route traffic, how to prewarm caches and based on what predictive approach, avoiding siloisation of processes due to feedback loops.
As these are knocked down they will pay dividends for everyone.
I take the view that there is a "warmth hierarchy" (an idea I spitballed with other folks in Knative before Matt Moore took it and filled it out a lot).
You have a request or message. You need to process it. The hierarchy runs CPU > Memory > Disk > Cold.
CPU-warm: immediately schedulable by the OS. Order of nanoseconds.
Memory-warm: in-memory but can't be immediately scheduled. Our reference case is processes that have been cgroup frozen. Order of tens of milliseconds.
Disk-warm: the necessary layers are all present on the node which is assigned to launch the process. Order of hundreds of milliseconds.
Cold start: nothing is present on the node which is assigned to launch the process. It will need to fetch some or all layers from a registry. Order of seconds to minutes.
Some of these problems can't be solved by Kubernetes (it doesn't know how to freeze a process), some of them can't be solved by Docker (it doesn't know about other nodes that might have warmer disk caches). Then there's the business of deciding where to route traffic, how to prewarm caches and based on what predictive approach, avoiding siloisation of processes due to feedback loops.
As these are knocked down they will pay dividends for everyone.