Nearly 20 years ago Microsoft introduced the New Technology File System as part of Windows NT. Since then NTFS has accumulated numerous features that, while promising at the time, turned out to be bad ideas. Meanwhile the state of the art has continued to advance in ways that are incompatible with the NTFS architecture. After a couple of failed starts, Microsoft is now promising a new file system.
Known as ReFS or Resilient File System, this new file system is designed to improve reliability, especially in the event of power loss or media degradation (i.e. the disk simply wearing out). Some of this reliability comes from fundamentally changing where and how file metadata is stored and updated. Rather than using in-place updates paired with a journal, metadata is updated using allocate-on-write semantics. If the power does go out during a write the file system simply continues using the old metadata entry rather than the partially written new one.
ReFS is compatible with the disk-spanning technology Storage Spaces. By itself Storage Spaces has several ways to improve performance and reliability, especially when dealing with catastrophic disk failure. Unfortunately there are subtler forms of data corruption that can occur. Using a checksum system, ReFS can detect when these errors occur. When that happens it then polls all of the Storage Spaces drives to determine which has a good copy of the file.
In order to proactively ensure drive and file integrity, ReFS will occasionally spot check files using a background process. This process compares the metadata and content of each copy of a file and, like in the example above, uses the good copies to correct the others. This is particularly important for rarely accessed files where corruption in one copy may not otherwise be caught before other copies happen to be corrupted as well.
Another reliability mechanism is what they are calling “integrity streams”. Just like metadata, integrity streams will use allocate-on-write semantics to reduce the chance that a failure while writing to disk will result in corruption of the only good copy of the file’s content. Integrity streams are not appropriate for all types of files; applications that require control over the physical file structure (e.g. databases) should disable this feature.
In order to improve compatibility ReFS has been built to run under the NTFS APIs with only the lowest levels swapped out. That said, there are numerous NTFS features that will not be supported in ReFS including:
- Named streams
- Object IDs
- Short names
- Compression
- File level encryption (EFS)
- User data transactions
- Sparse Files
- Hard-links
- Extended attributes
- Quotas
Other features such as “BitLocker encryption, Access Control Lists for security, USN journal, change notifications, symbolic links, junction points, mount points, reparse points, volume snapshots, file IDs, and oplocks” will still be supported.
The initial release of ReFS will be targeting file servers. This means it will only be available on Windows 8 Server and will not support bootable drives. Over time ReFS will be offered to client versions of Windows and eventually may be used as a boot drive. Given how dangerous flaws are when it comes to file systems, Microsoft felt that this go-slow approach would be more prudent.
One cannot perform an in-place conversion from NTFS to ReFS, a new drive must be created and the data copied across.