where i worked - a small shop with 10 engineers - we had 350 machines running in 5 different data centers in chicago, new jersey, and korea. each machine ran around 100 different processes, which were coordinated using a combination of several different pubsub systems:
- in house reliable tcp-based broker mediated (we called it pubsub3 )
- in house unrelaiable ip-multicast system (i named it blue
ray because i thought that sounded cool)
- 3rd party implementation of a PGM derivative that we licensed for god knows how much
the engines subscribed to various topics on the system, processed market ticks (sent by marketdata publishers over blue ray) and fired orders at whatever instrument they were handling
the risk engines modeled risk scenarios inresponse to trading shifts, powering through the risk models our quant trader wrote in c. we had a 'calcparser' system that constructed a DAG rerpresenting the AST of the c code and distributed the DAG across several machines.
The complexity is also in having to deal with limited resources, and factors outside of your control. Have you seen many bootstrapped HFT implementations? The budgets are just... different.
When you spend 10x or 100x hardware, get massively redundant backbone connectivity (connecting to a tier 1 core router directly, and not through an oversubscribed shared switch port that a typical bootstrapped startup would get), you end up with a much better controlled and somewhat predictable environment.
With a typical "web" (non-HFT) distributed system, you've got millions of "subscribers" aka users - and I don't think people outside of IT fully realize just how bad a typical user's [Windows] machine is messed up - and that was before they installed a couple of toolbars, adware, firewall, and AV products... And all of these subs are tying up your publishers in their own unique screwed up way, keeping sessions open, dropping packets, blocking ports...
I've designed global CDNs, and operated one for a decade - and the kind of randomness our distributed systems have to deal with are not always technical, even though we don't typically count microseconds.
where i worked - a small shop with 10 engineers - we had 350 machines running in 5 different data centers in chicago, new jersey, and korea. each machine ran around 100 different processes, which were coordinated using a combination of several different pubsub systems:
- in house reliable tcp-based broker mediated (we called it pubsub3 )
- in house unrelaiable ip-multicast system (i named it blue ray because i thought that sounded cool)
- 3rd party implementation of a PGM derivative that we licensed for god knows how much
the engines subscribed to various topics on the system, processed market ticks (sent by marketdata publishers over blue ray) and fired orders at whatever instrument they were handling
the risk engines modeled risk scenarios inresponse to trading shifts, powering through the risk models our quant trader wrote in c. we had a 'calcparser' system that constructed a DAG rerpresenting the AST of the c code and distributed the DAG across several machines.
shit was nuts. i miss it.