Storage Performance is Critical for High Performance Computing

In many ways developments in IT storage have tended to be seen as unexciting to most people, except deeply technical storage architects…and storage vendors. My recent briefing from Jeff Whitaker, VP of Marketing and Product Management at Panasas showed me that he is determined to change that perception.

Panasas was formed in 1999 with a specific focus on the High-Performance Computing (HPC) market. Over the past 23 years they have built a solid reputation for providing linearly scalable, high performance, easy to manage solutions via its ActiveStor® parallel file system hardware appliances and PanFS® data management software. Within specific market segments, such as Life Sciences and Higher Educational research establishments, it has a core of loyal customers who value the reliability and ease of use features that a strong product management focus has delivered.

Conversely that strong product management focus and a certain lack of awareness about changes in market dynamics has led to one or two missteps that have dented growth prospects at times in the last 10 years. Previously, Panasas would have called themselves an HPC storage appliance and software company. But in 2018 Panasas switched focus away from building their own appliances and decided to use COTS (Commercial off the Shelf) hardware instead. This is not as straightforward as it sounds and certainly doesn’t mean that customers can buy any storage hardware and expect it meet the exacting performance characteristics demanded by many HPC use-cases. Panasas still sells hardware appliances with hardware supplied by their preferred partner, Supermicro, with which the PanFS parallel file system has been tightly integrated.

Making sure that this move didn’t impact existing customers and ensured on-going performance required a 4-year focus and effort and the briefing was ostensibly to talk about some new products and a more market-oriented approach.

There are two new products. ActiveStor® Flash, based entirely on NVMe storage is aimed at AI/ML training, trading strategy back-testing, as well as life sciences and electronic design automation (EDA) projects involving Metadata and smaller file sizes. The second product, ActiveStor® UltraXL expands its capacity and performance at the high-end into massive-scale data environments, often with very large file sizes, that should open up opportunities for Panasas in sectors such as seismic resource exploration; scientific, academic, and government research; manufacturing; and media and entertainment.

AI in the Cloud

Panasas have been, and remain, resolutely focused on providing on-premises based systems. Given the pervasiveness of Cloud today, and the growing interest in using public cloud for AI applications, I had to ask Jeff how that would impact Panasas strategy and positioning. The discussion is worth sharing as it raises some interesting questions customers need to consider when deciding whether to deploy AI solutions in the public cloud.

A lot of the focus has been on the performance benchmarks of the public cloud providers’ processor configurations. But, as Jeff Whitaker explained, given the importance of i/o performance in HPC systems in general, and AI in particular there isn’t an accepted storage performance benchmark. Panasas is co-chair of the ML Commons storage working group looking to define just such a benchmark that should help a more balanced understanding of how AI/ML systems perform in a public cloud or on-premises.

A further consideration is the cost of data ingress and egress in a public cloud. This can become very expensive, and it also threatens the capacity of public networks to handle huge amounts of data that should, in our opinion at Bloor, be handled close to the Edge. In this day and age, we are seeing a shift from data moving to the compute, to the compute moving to the data. This is true for AI as well as more obvious areas like IoT sensors.

Ultimately, Jeff thought that the Cloud would be used for AI and ML development and model training where smaller data files are more usual. Then large production systems would run at the on-premises. I’m sure Amazon, Microsoft and Google would like to dispute this, and I have no doubt they will be looking at how they deliver production performance of AI systems in the Cloud. But for the moment, customers should think carefully about the cost and performance implications of AI.

Panasas ActiveStor® and PanFS® are great products for the HPC market. They deserve to have their story told more engagingly. It looks like Panasas might be about to do that.