It seems that the only way to scale a site like this is to build decoupled services maybe using a messaging queue (AMQP, etc..).
How can I learn more about this kind of systems? Also is there a open source application[s] built in a distributed way that I could learn from?
Exactly. If you don't know what Thrift is, check it out.
In addition to solving the problem space using simple, focused applications (in the classical Unix style), it solves other practical issues. Like PHP having shitty clients for most modern data stores (Cassandra, Redis, hell, even Memcached) compared to what you get in, say, Python, or a JVM language.
I don't really know of any. Nobody writes these sort of distributed applications from scratch. They're almost always evolved from monolithic database-backed applications. And the process of evolution is half development half ops. I doubt that tumblr as it exists today could be deployed from scratch, even if you had all of the source code of all of the systems.
Gearman, briefly mentioned in the article, can probably help you as well. Let's you do cross language, cross platform (a)synchronous function calls . Looks cool, never used it..