Absolutely agree that that is the first priority, but there are many factors you can keep in mind during development without slowing it down much, and this is one of them.

But your customers aren't going to be happy if your site is slow whilst your bleeding money.

Heroku/EC2 are fantastic for what they are but quite pricy.