BT

New Early adopter or innovator? InfoQ has been working on some new features for you. Learn more

New Features and Performance Improvements for System.IO

| by Jonathan Allen Follow 131 Followers on Oct 13, 2009. Estimated reading time: 2 minutes |

Microsoft is planning some simple but much welcomed performance improvements for the core System.IO functionality. These include convenience methods for reading and writing text-based files, significantly faster directory enumeration, and support for memory mapped files.

The first improvement is a replacement for the convenience method File.ReadAllLines. For small files this is a perfectly acceptable function, but as the file size increases so do the problems. The fundamental flaw is that ReadAllLines does just that, it pauses the program until the entire file can be read into an array of strings.

The replacement is File.ReadLines, which returns a string enumerator. This will lazily read the file, just as if you used the lower level stream objects. Also available are new overloads of File.WriteAllLines and File.AppendAllLines, both of which now take an enumerator instead of just an array.

DirectoryInfo.GetFiles has the same array problem, but they lies an even more serious issue underneath. When retrieving a list of files the Win32 API also returns basic information like size and last modified date. Unfortunately this information is discarded by .NET instead of being passed to the FileInfo objects. So when the program starts to loop through the files, perhaps to determine the directories overall size, it has to requery the file system one by one. What you end up with is a classic 1+N optimization issue. Both DirectoryInfo.GetFiles and the new DirectoryInfo.EnumerateFiles fix this problem.

Another major performance boost for .NET is support for memory-mapped files. Memory-mapped files are an operating system feature that links a block of memory to a file. Once linked, you can read and write to any part of the file as if it were nothing more than just an array of unmanaged memory. The operating system handles important details like paging different parts of the file into and out of memory as needed. Memory-mapped files allow applications to work with incredibly large files, even in excess of a gigabyte, in a highly efficient manner.

In addition to raw file I/O, memory-mapped files provide a powerful means of communication between processes. If two applications open the same memory-mapped file, changes made by one application will be immediately visible to the other application.

Despite the name, memory-mapped files are not necessarily real files. They can also be purely in-memory objects with no backing store. While potentially useful within an application, these are particularly applicable to cross-process communication.

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Network I/O by Alex Suvorov

I wish they also improved network I/O, current implementation consumes much CPU and memory. The only alternative is a commercial XF.Server component (www.kodart.com), but I wish that would be available in .NET for free.

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

1 Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT