Consultant for a big box retailer here. Yes, of course I can build a shopping cart site with a few good hackers, and I can find a few good hackers anywhere.

But that's not what I have. I have a pile of legacy crud that has to integrate with ten other piles of legacy crud, not to mention the external crud with which I must integrate.

And did I mention that during holiday the site gets a little more traffic than most sites? Or that the site makes a few hundred million dollars, and can't just be rewritten on a whim? To solve these sorts of problems, I don't need ten good hackers, I need more like 50. Or more. And that's the challenge the article is talking about. Because good hackers have options like Facebook - or their own startups - where they don't have to deal with that kind of noise.

It's a serious challenge for any big company which relies heavily on the web, which in 2012 is every big company.

"Or that the site makes a few hundred million dollars, and can't just be rewritten on a whim?"

Why not? If the site is unstable and can't stay up, you can either throw 50 programmers at it and still have it crash regularly or you can take far fewer and re-write it from scratch.

You're right about programmers not liking the cruft, but we also don't like most of the management processes that occur in large companies. I don't know if things have changed, but back when I worked for a large brake corporation, web programmers were devalued and distrusted. All of our decisions had to be signed off (in triplicate!) by a manager who didn't know how to program or the implications of the decisions being made.

To give you on example, one of the official company-wide policies was that if you wanted to transfer files between company offices over the internet, you had to use FTP for "security purposes." We were technically not allowed to use SSH or SFTP because, when I asked, my boss said nobody had heard of it. This was in 2006.

Rewriting a major ecommerce site in parallel with maintaining the current one, while ensuring that all functionality stays exactly the same, that all cross-pollination between systems stays exactly the same... you make it sound like it would be easy.

The stakes would be incredibly high, you would need extremely skilled, experienced and passionate engineers and a management environment that would give them enough breathing space to be successful. Everyone's careers would be on the line. Besides the fact that no one is going to want to stick their necks out for Target.com (whereas someone might consider it for a site that improves lives in a meaningful way), that's going to be a lot more difficult than you're making it sound.

Sounds like 150k to 200k to me.

Amazon actually dealt with this problem in a very elegant fashion, at some point they realized that crap cannot be excised from the system, the reality is due to integrations, 3rd party code and legacy that somewhere there is going to be crap. It was at this point that they started to move to a service based architecture in which they could isolate all business functions into black boxes or services. This way instead of having a monolithic crappy application you have disparate services with some good and some bad. Further they have service teams that are accountable to their business units for the quality of their code. So what they now had was a way to gauge very discrete units based on performance to identify bad solutions, it allowed them to surgically upgrade those areas while not affecting the rest of the system. They where able to replace those pieces of the system, by just ensuring that new solutions conformed to the old service contracts. I know that enterprise solutions don't get a lot of love here on HN because it is not the focus but SOA and by extension REST are very important technologies that help deal with the problem that you highlighted. Amazon, is a very good case study on the subject.

Hmm, that's interesting. Did you witness this first-hand? If so, what were some of the struggles they faced?

No I did not, I have contacts in the organization, but more importantly, they did a good deal of postmortems and white papers on the subject which where pretty frank about their problems and the solutions they tried using to address them, highlighting what worked and how it worked and what failed and why it failed. I read most that where available because it was a treasure trove of experience in aligning a large organizations development to improve it's quality. As such there are not many other examples that have had the success in doing it that Amazon did. AWS and S3 where two new lines of business that came out of that alignment. They streamlined the idea of business and technology as services to the point that it became a commodity in their organization.

The complication arises when interfacing the e-commerce with existing infrastructure management systems. If someone buys item X from the online store and says they want to pick it up at Store A, the commerce site needs to do several things:

1) Check if the item is in stock at Store A, if so, send a message to take it off the shelf to the store -- access to system wide inventory management. 2) If it's not in store A, find the nearest warehouse and route item X to store A. Give the user an estimate shipping time + Alert Store A the incoming item is reserved for the user. 3) Update global inventory lists and decrement one item X from stocks. 4) Update sales/accounting to let them know of the sale. etc.

The problem isn't in the ecommerce site so much as everything else it has to interface with and do. It has to interface with systems up and down the whole chain, and depending on how up to date and accessible those systems are, it could get real interesting trying to merge them together.

...except the ecommerce system won't do things like stock control or replenishment, it just sends messages to the systems that handle those things - so then how do you know when something is going to be ready at the store?

I'm a retail integration consultant (currently looking for my next gig, to any desperate Oracle Retail users out there :-).

You either 1) Emulate stock control and routing rules in your app (flaky), 2) Hook the ecommerce app deep into the DBs of the other systems (now almost impossible to change those systems due to the level of coupling and the politics it will involve) 3) Go fully realtime with all systems running on a common bus (seriously heavy infrastructure and support requirements)

Ideally you should do 3 but most of the time you end up with some combination of 1 and 2. Then the next "must have" idea comes along, and before you know it your fancy new e-commerce solution has become part of the legacy establishment.

Is there an option between 2 & 3?

My experience, albeit only the last 2 jobs, with the concept of "Enterprise Service Bus" is that it becomes a magnet for "Architecture Astronauts" with little functionality delivered :-(

But it sounds a hell of a lot more impressive than "we will wrap simple REST or XML-RPC services, for which clients can easily be implemented in various consumer systems, around needed queries and updates, using the technology most amenable to the data store in question, documenting the interfaces, using proper configuration management policies", though.

But maybe that's about what you meant, in a nutshell.

And it can go even further: now customer wants to return item X and get a refund! Where does he send it? Has it arrived? What about the money? Wow.

I'd always try with a single guy.