Optimizing Storage For Scientific Supercomputing Platforms

Researchers at Experts Laboratory continue to advance the development of their new data management software architecture focused on tiered storage for scientific workflows (MaDsTS). They have released a white paper explaining data-centric programming abstractions that allow users to better manage workflows with data across the various storage tiers for these applications.

Today, one of the most complex data management environments is tiered storage for supercomputing platforms, whose architecture must be tailored to the specific workflows to be executed on HPC platforms. Recently, a team of researchers at Lawrence Berkeley National Laboratory in California introduced a new data management software architecture for scientific workflows (MaDaTS) tailored to the hierarchical organization of HPC systems used in industry.

There is an increasing complexity of storage and memory systems, with technologies such as NVRAM, SSDs, platforms, and new generation data fabrics. Managing these data sets at different levels is a challenge that requires programming abstractions that make it easy to work with and extract as much performance as possible from storage and compute platforms. This has been one of the fields of study in which these experts have worked, and now they have presented a document explaining how they have developed these data-centric abstractions.

They explain that their designs allow users to manage a workflow around their data at the storage layer, with a specific focus on simplifying data management for scientific workflows in multi-tier storage systems, taking the maximum use of computational resources and storage capacity.

They have measured the overhead and effectiveness of the MaDaTS programming abstractions they have developed and concluded that they allow optimal use of available storage capacity at lower capacity tiers and simplify data management across the board system without affecting performance.

This will improve heterogeneous tiered storage platforms used in scientific supercomputing systems, a field where the industry is striving to improve technology and optimize systems. This field includes everything from meteorology and earth sciences to the development of new materials, health science, and other fields such as astrophysics and artificial intelligence.

Also Read: How To Build Big Data Applications With A Low-Code Platform