BT
x Your opinion matters! Please fill in the InfoQ Survey about your reading habits!

New Features and Performance Improvements for System.IO

by Jonathan Allen on Oct 13, 2009 |

Microsoft is planning some simple but much welcomed performance improvements for the core System.IO functionality. These include convenience methods for reading and writing text-based files, significantly faster directory enumeration, and support for memory mapped files.

The first improvement is a replacement for the convenience method File.ReadAllLines. For small files this is a perfectly acceptable function, but as the file size increases so do the problems. The fundamental flaw is that ReadAllLines does just that, it pauses the program until the entire file can be read into an array of strings.

The replacement is File.ReadLines, which returns a string enumerator. This will lazily read the file, just as if you used the lower level stream objects. Also available are new overloads of File.WriteAllLines and File.AppendAllLines, both of which now take an enumerator instead of just an array.

DirectoryInfo.GetFiles has the same array problem, but they lies an even more serious issue underneath. When retrieving a list of files the Win32 API also returns basic information like size and last modified date. Unfortunately this information is discarded by .NET instead of being passed to the FileInfo objects. So when the program starts to loop through the files, perhaps to determine the directories overall size, it has to requery the file system one by one. What you end up with is a classic 1+N optimization issue. Both DirectoryInfo.GetFiles and the new DirectoryInfo.EnumerateFiles fix this problem.

Another major performance boost for .NET is support for memory-mapped files. Memory-mapped files are an operating system feature that links a block of memory to a file. Once linked, you can read and write to any part of the file as if it were nothing more than just an array of unmanaged memory. The operating system handles important details like paging different parts of the file into and out of memory as needed. Memory-mapped files allow applications to work with incredibly large files, even in excess of a gigabyte, in a highly efficient manner.

In addition to raw file I/O, memory-mapped files provide a powerful means of communication between processes. If two applications open the same memory-mapped file, changes made by one application will be immediately visible to the other application.

Despite the name, memory-mapped files are not necessarily real files. They can also be purely in-memory objects with no backing store. While potentially useful within an application, these are particularly applicable to cross-process communication.

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Network I/O by Alex Suvorov

I wish they also improved network I/O, current implementation consumes much CPU and memory. The only alternative is a commercial XF.Server component (www.kodart.com), but I wish that would be available in .NET for free.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

1 Discuss

Educational Content

General Feedback
Bugs
Advertising
Editorial
InfoQ.com and all content copyright © 2006-2014 C4Media Inc. InfoQ.com hosted at Contegix, the best ISP we've ever worked with.
Privacy policy
BT