Many years ago, back before the dawn of MIME, before HTTP roamed the earth, a population of digital machines arose. Some were small and permeated nearly every household, as if they had simply fallen from the trees. Others were giants born of the vast ocean Blue. Yet there was one class of machines that towered above all others. These systems were so awesomely powerful, so incomprehensibly fast, and so well adorned with nifty blinking lights, that they were not even known by the mere designation of "computer." Yes, this was the age when super-computers reigned supreme. And the scientist gazed upon the supercomputer, and said it was good.
At least until he got the bill.
The needs of the scientist gave rise to the first generation of supercomputers. These were purpose-built machines with custom hardware. Today, however, we live in the era commercial off the shelf (COTS) technology, which increasingly blurs the distinction between "supercomputer" and "a really big room filled with workstations" or even "a bunch of rooms across the country filled with workstations". Furthermore, the capabilities of COTS are in many ways the antithesis to the needs of scientific research. In this talk, I will describe what attributes characterize scientific codes, and how modern-day high-performance computing attempts to strike a cost-effective balance between customization and off-the-shelf. This will motivate discussion on the obstacles for application development on massively distributed compute resources, why scientists are being forced to rethink how they write programs, and what additional challenges the future will bring.