Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

It seems that the only way to scale a site like this is to build decoupled services maybe using a messaging queue (AMQP, etc..). How can I learn more about this kind of systems? Also is there a open source application[s] built in a distributed way that I could learn from?


Exactly. If you don't know what Thrift is, check it out.

In addition to solving the problem space using simple, focused applications (in the classical Unix style), it solves other practical issues. Like PHP having shitty clients for most modern data stores (Cassandra, Redis, hell, even Memcached) compared to what you get in, say, Python, or a JVM language.


I hate to point out the obvious answer, but it's a good introduction with links to several implementations:

http://en.wikipedia.org/wiki/AMQP


I meant some open source applications that are built in a distributed way (hence using something like AMQP in a publish/subscribe fashion)


I don't really know of any. Nobody writes these sort of distributed applications from scratch. They're almost always evolved from monolithic database-backed applications. And the process of evolution is half development half ops. I doubt that tumblr as it exists today could be deployed from scratch, even if you had all of the source code of all of the systems.


However, if you want to go truly distributed (much more so than any website I know of) there are open source projects you can learn from:

http://freenetproject.org/

http://amule.org (or http://mldonkey.sourceforge.net/Main_Page if you're into functional programming)

http://www.torproject.org (especially the hidden services stuff)

And you probably want to read this: http://pmg.csail.mit.edu/papers/osdi99.pdf


mongrel2[1] (uses Ømq[2]) maybe?

[1]: http://mongrel2.org/

[2]: http://www.zeromq.org/


Gearman, briefly mentioned in the article, can probably help you as well. Let's you do cross language, cross platform (a)synchronous function calls . Looks cool, never used it..

http://gearman.org/ http://gearman.org/index.php#how_does_gearman_work




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: