there was a project trying to achieve some of those goals a few years ago using p2p: petals https://github.com/bigscience-workshop/petals

their bloom model was also a collaborative effort https://huggingface.co/docs/transformers/en/model_doc/bloom