Yes, all the layer cache is saved on a persistent volume that's reused between jobs. In that respect, it's very similar to running `docker build` on your local laptop, where each build is incremental. But in Depot's case, that incremental experience is shared for all CI builds, and between all users running `depot build` on their local devices.
It sounds like the SSD provides the layer cache with `--cache-to=type=local`? Does Depot support connecting additional VMs to the volume if someone wants to scale up? I've been meaning to look into the new S3 cache to solve this issue.
Exactly, we effectively persist BuildKit's state directory, so it doesn't even need to tar and export any of the internal state or layer cache, it gets the same state volume for the next build.
Today we're using vertical scaling, so we run BuildKit on larger EC2 instances, tune `max-parallelism`, and let BuildKit handle processing multiple builds / deduplicating steps across builds / etc. What our website calls a "project" basically equates to a new EC2 instance (so two different projects are fully isolated from each other).
We'd like to expand into horizontal scaling, probably with some kind of tiered caching, so the builders would be able to use local cache on local SSDs if available, but fall back to copying cache from S3 if not available locally.
Today we're vertically scaling builders, so multiple builds run on the same large EC2 instance. The instances support processing multiple Docker builds at once, and thanks to BuildKit can even deduplicate work between multiple parallel builds, so like if two simultaneous builds share some steps, it can handle coordinating so those steps run just once.
We have plans to expand to more horizontal scaling with tiered caching, so we can keep the speedups we see today but further increase potential parallelism.
