CentOS and Windows, Xeon 5400 series CPUs - cheap used Dells. OpenSIPS, FreeSWITCH, RabbitMQ, SQL Server, VoltDB, and a custom DB. We have a few hundred million routes loaded at any given time and run up to 50,000 calls/sec. SQL Server is going to get phased out in favour of Postgres, due to SQL Server's increasing licensing costs. ElasticSearch also looks like it might be a good candidate for crunching up a bunch of records.
All critical code is F# on Mono and CLR. Each voice-handling machine makes an HTTP request to a CLR process to get routing instructions, then the results of the call are handed off to a Mono process, which protobufs stuff into a local RabbitMQ instance, which gets shoveled into billing, analysis, etc.
Lots of "XML", as that's FreeSWITCH's preferred format (it's not real XML, but a psuedo-XML with inane encoding rules, for some reason). This is the cause of at least 20% CPU time when handling call records.
In general, scaling out by adding a few more machines to the mix is so easy we're not really pressed hard to make things go faster. But it's certainly fun to do. I'd guess we can improve many pieces up over 100% without doing anything really tricky.
As a comparison, I know of at least one successful competitor that has everything in PHP. Every call creates multiple processes and multiple (10+ sometimes) DB queries. Their entire scale-out process is to throw hardware at things. At one point they had over 100(!) servers just to hold call records (maybe 100M a month?). Inefficient? Hilariously so. Did the primary owners get rich from it? Absolutely. Engineering quality counts for surprisingly little, when it really comes down to it.
All critical code is F# on Mono and CLR. Each voice-handling machine makes an HTTP request to a CLR process to get routing instructions, then the results of the call are handed off to a Mono process, which protobufs stuff into a local RabbitMQ instance, which gets shoveled into billing, analysis, etc.
Lots of "XML", as that's FreeSWITCH's preferred format (it's not real XML, but a psuedo-XML with inane encoding rules, for some reason). This is the cause of at least 20% CPU time when handling call records.
In general, scaling out by adding a few more machines to the mix is so easy we're not really pressed hard to make things go faster. But it's certainly fun to do. I'd guess we can improve many pieces up over 100% without doing anything really tricky.
As a comparison, I know of at least one successful competitor that has everything in PHP. Every call creates multiple processes and multiple (10+ sometimes) DB queries. Their entire scale-out process is to throw hardware at things. At one point they had over 100(!) servers just to hold call records (maybe 100M a month?). Inefficient? Hilariously so. Did the primary owners get rich from it? Absolutely. Engineering quality counts for surprisingly little, when it really comes down to it.