Understanding Persistent Memory
What is persistent memory?
Any media with byte-addressability which provides the memory semantics (direct load/store accesses) along with the persistence properties of storage is termed as persistent memory. Persistent memory is connected to the system memory bus like a DRAM DIMM and often is referred to as NVDIMM (Non-volatile DIMM). It allows the applications to access the media as memory. So comes with all the benefits that memory provides (Main memory(DRAM) in systems today): cache line granularity access, cache coherency support, DMA support, etc. Supporting these memory semantics puts a constraint on the media that a load or store latency to the media should be fast enough that it is reasonable to stall a processor while the load or store instruction is accessing it.
Intel Optane DC Persistent Memory is the first commercially available persistent memory in the DIMM form and has latency in nanoseconds.
How an application access the persistent memory?
The persistent memory is exposed to the applications as memory-mapped files by the Operating System similar to the way the files on storage media like HDDs and SSDs are. But there is a huge difference in the way the normal accesses happen to a memory-mapped file on persistent memory vs a memory-mapped file on storage media.
How is persistent memory different from volatile DRAM memory?
Persistent memory is not anonymous like DRAM. Because the data of the applications are going to be present even after the power is off, applications need a specific way to connect to the content of the persistent memory and that is done by using the old concept of files on storage media which can be located via name and have permissions which can control which all applications can access a particular region on persistent memory. So allocation and management of persistent memory are done by the Operating Systems using the same semantics of file management on storage media providing the naming, access, and permission support along with memory-mapping the files.
How could the file-systems leverage the persistent memory semantics to provide better performance to the applications?
Storage-based file blocks are cached in the main memory by the OS using the concept of system page cache. So whenever the data needs to be persisted to the storage media, it is flushed from the page cache in blocks. So there is no direct access to the storage media at a low granularity, which is why the performance for a programming writing to a file in a storage media is abysmal. The normal file systems could still be used for persistent memory, very similar to how they deal with files on storage media (accessing at block granularity).
On the other hand, there is development of persistent memory aware file-systems that allows direct access to persistent memory without using the page cache. Linux and Windows OS both provide this feature and it is known as Direct Access(DAX).
Memory mapping a file on persistent memory and accessing via load/stores
Persistent memory files can be mapped into memory using standard system calls like mmap() on Linux. After memory mapping the file, the applications can directly access persistent memory using normal load/store instructions. Once the memory mappings are set-up after the initial page-fault (minor page-fault), a way is created for applications to access persistent memory via load/stores without any involvement of kernel.
Execution of load and stores does not guarantee the persistence of data to the media. The data could be present in some level of caches or even in-flight in memory controller buffers on natural eviction from caches. But there may be a requirement for the applications to ensure the persistence of data to guarantee consistency on a system failure. Similar to how the storage-media backed files are persisted using the range-based msync() call or file-based fsync() call, they could be used to guarantee the persistence of data to the persistent memory media (NVDIMM). Internally, these calls would ensure that if the data is cached at any level of the cache hierarchy, then that is flushed to the media via memory controller. Therefore, we can understand these calls like memory barriers which ensures that all the modifications to the data belonging to persistent memory before the call point are guaranteed to have made to the media.
Hardware instruction support for persistent memory
x86 provides multiple user-level instructions(CLFLUSH, CLFLUSHOPT, CLWB) to flush the data from caches to the persistent memory. CLFLUSHOPT and CLWB are not serialized instructions so they need to be followed by SFENCE instruction to ensure the flushes are complete before continuing.
Accessing persistent memory without any file system
Linux provides a feature called as Device-DAX which allows application to open a persistent memory device without a file system. Applications can memory map the whole device and then can use the normal load/stores followed by cache flush instructions and memory barriers(sfence) to ensure persistence.
Blogs to follow in the series:
- Crash consistency support for applications leveraging persistent memory
- Mechanism of Asynchronous DRAM Refresh (ADR)
- Memory controller and persistent memory interaction
- Getting started with programming for persistent memory
- Writing simple data structures residing on persistent memory
Reference: Andy Rudoff, Persistent Memory Programming, 2017
