Deja vu: Storage goes the way of compute infrastructure

Deja vu: Storage goes the way of compute infrastructure

I grew up in a valley nestled in the lower Himalayas. When we needed to send a package to friends in different cities, my dad would send me to the local truck agency.  They would charge arbitrary, often exorbitant prices, and give no guarantees on timing or safe delivery of the goods. After all that effort, packages would often reach their destination damaged or even get lost. Times have changed; the advancement of technology and commoditization of the airline industry completely changed this landscape. I go back to my hometown now and use DHL (the variant of FedEx used in India) to have packages picked up from home, delivered on time, safe, and for lower comparative cost than I used to pay. In other words, shipping “infrastructure” has evolved to make available solutions simultaneously more efficient and more affordable. The computer storage industry is going through a very interesting, similar transformation due to the commoditization of the components used to build storage platforms and the advent of flash as a disruptive technology.

I read the announcement from Pure recently around a custom hardware platform to build a scale-out storage system that will ship sometime in the future. I am happy they finally acknowledged that scale-out architecture is the way to build storage systems. That said, a custom hardware platform? What are they thinking? The real advantage of scale-out is about flexibility, economics and managing risk. If you know exactly what your storage and application needs are and you can predict them for the lifecycle of the storage system, yes, you should buy custom hardware and provision it to match the needs of the applications. But in this age of accelerated data growth and decreasing hardware costs, purchasing proprietary hardware is equivalent to locking your business into inefficiency.

As somebody who has been building distributed systems for the past 2 decades, I can draw a parallel to when the compute side of infrastructure went through this transition in the 90’s. Who remembers the likes of Sequent and Pyramid Computers? These companies were big vendors built around the promise of scale up custom hardware & operating systems that were eventually taken out by the really simple commodity server running Linux. Why did the customers choose these less optimized commodity platforms over the scale up options? It was because customers cared about the flexibility provided by scale-out and the economic efficiency that commodity components help drive. Commodity systems give them the option to get the best economics. A scale-out architecture gives customers the flexibility to grow the infrastructure economically as their organization and application needs change or grow.

We have built an enterprise grade scale-out storage platform based on flash at Coho that has been in production for over 2 years. This platform leverages commodity hardware to provide a scale-out solution that allows the customers to ride the hardware cost curve and manage uncertainty around storage growth and needs in the most economical manner. What I have heard from many customers is that their application needs around performance, capacity and scale are unpredictable. A scale-out, commodity platform ensures they don’t spend a lot of money today to overprovision against their unpredictable growth. Instead, as their needs grow they can add commodity nodes that will typically have dropped in cost. This gives them the best option to invest in a system that is simple to use and can grow to their needs of both capacity and performance, while leveraging the economics that commoditization of storage will drive over time.

We live in interesting times in the technology world: new components are being developed that can deliver new levels of efficiency if, and only if, we build our solutions to leverage them effectively. I believe storage is going through the same transition as both package shipment and compute infrastructure has in past years, and that commodity hardware and scale-out will come out as the champions of this transformation. These are the key tenets of the products we build at Coho.

2,897 total views, 4 views today