I think Intel realized at the end of Netburst era that pushing 150+W of power through small piece of silicon is not trivial anymore. So to have better performance they were forced to improve perf/watt, and that is exactly what they have done since.

Definitely: for at least half a decade most of the HPC people I've worked with or for had power and cooling as the primary constraint, not space or budget. Things like switching to centralized DC (vs. in-server power supplies), getting disks out of cluster nodes, etc. but there was an obvious problem when a rack of servers was roughly comparable BTU-wise to a large gas grill running at maximum.

Yeah, buying your hardware is largely a one time capital cost, but power for it and for your cooling is a constant operating cost. Haven't I read somewhere that power is now the largest expense in the lifetime of a "supercomputer"?

Forget operating costs, one major biological research center I knew of hit (IIRC) 20% of their maximum server space before the power company said they'd need to run a new high-voltage line at $$$ and years of delays even if the neighbors didn't kill it. I believe they were able to jump on the lower-power hardware as that market developed but it certainly got a lot of attention in the community.

Wow. I suppose they didn't want to move the bulk of their servers because of their sunk costs in space, security, etc. ... but, yeah, I can see that getting a lot of attention.

And you implicitly bring up a point I've read about, that power and cooling are now the limiting factors in server spaces. Moore's Law plus the even faster doubling of disk platter capacity (as of late I've read it's doubling every year or so) has allowed us to pack rather a lot in rather small spaces.

I think for most any computer that is at constant high CPU utilization power will cost more than the hardware over its lifetime. Unless you have very cheap power.