Using Redis supported by NVRAM in HPC applications

Abstract

Nowadays, the efficiency of a storage systems is a bottleneck in many moern HPC clusters. High performance in traditional approach - processing using files - is often difficult to obtain because of model complexity and its read/write patterns. Alternative approach is applying a key-value database, which usually has low latency and scales well. On the other hand, many key-value stores suffer from limitation of memory capacity and vulnerability to serious faiures, which is caused by processing in RAM. Moreover, some research suggests, that scientific data models are not applicable to storage structures of key-value databases. In this paper, the author proposes resolving mentioned issues by replacing RAM with NVRAM. Practical example is based on Redis NoSQL. The article contains also a three domain specific APIs, that show the idea bhind transformation from HPC data model to Redis structures, as well as two micro-benchmarks results.