Next eats up your Windows memory – and is proud of it!

Torben Riis

Your Windows server is running low on memory. 0 Mb free physical memory! Time to panic? Absolutely not – read on to learn why. But only if you truly enjoy the nitty-gritty details of memory management.

Memory mapped files in Next

Multi Support Next uses memory-mapped files for its indexes to increase IO performance. In short terms; a memory-mapped file is a file that has been mapped into virtual memory such that it looks like it has been loaded into memory.  The operating system will then transparently load parts of the file into physical memory as the application access them, and release them again if not needed anymore.  Thereby the application does not itself have to be concerned by which parts exists in memory and which does not, at any given time. 

This chapter describes how memory on Windows is affected by the memory-mapped files. As you will learn, a system which appears to allocate all memory is not a bad sign at all - the operating system is only optimizing IO. Expect Memory to be used - unused memory is wasted memory.

Working Set memory on Windows

The memory model in Windows is a quite complex thing and it includes a lot more than what the Task Manager and Resource Monitor shows for a given process. These tools do by default, only show the memory directly associated to a given process, also known as private memory, but memory consumptions for shared resources such as Memory Mapped IO isn’t shown by default. The memory consumed using Memory Mapped IO is also known as working set memory, and it can be displayed by adding the 'Working Set (Memory)' column to the task manager.

When is memory allocated and released?

Windows Servers and Windows desktop editions behave differently when it comes to releasing working set memory associated with a memory mapped file. Desktop editions will release this working set memory much quicker than the server editions, since Windows servers doesn’t expect a lot of applications to suddenly start, when the server is up and running.

But the key point is, the operating system will release it, if it needs it. Therefore it isn’t necessarily a problem if you see that a server has 0 Mb free physical memory and it uses memory mapped IO. It’s just an optimization from the operating system, it caches as much as possible of the memory mapped file in physical memory, in the case it should be accessed later. The important part is Windows will release parts of the physical memory allocated to Memory Mapped IO, if it is needed for other use, and if it is not, it will just keep it reserved.

As mentioned earlier pages in a memory-mapped file is only cached in physical memory if it has been accessed. Therefore the amount of used working set memory will typically increase slowly over time, with the exception of index creation. If indexes are created from scratch, every page in that memory mapped index file will be touched, and this means that the operating system will very quickly assign all its available memory as working set memory. It’s therefore expected that every available byte of physical memory will reserved during index creation, unless the index is smaller than the amount of physical memory.

Since no other applications access the memory-mapped files used by Multi Support Next, the reserved working set memory will be released when Next is shutdown. It’s natural and expected behavior.

Memory in a virtual server environment

It is very important that the virtualization server reserves as much physical memory to the guest OS running Multi Support Next as the guest OS is emulating. Otherwise ballooning will most likely occur due to the use of Memory Mapped IO. It is much better to assign less memory to a guest OS and guarantee that it always will be in physical memory than start paging memory that the guest OS believes is physical. This is also known as the ballooning effect.

Tools – Microsoft Sysinternals

Multiple tools from Microsoft have been released over the time to monitor and manage how physical memory is assigned. RAMMap and VMMap are such tools. With these tools you can monitor the usage of working set memory, but you can also release all the physical memory assigned to Memory Mapped IO, if you should have such interests.

RAMMap -

With VMMap you can monitor which files a process has opened as Memory Mapped files, and how much of each file is loaded into physical memory (Total WS).

VMMap -

If you got this far, chances are that you actually do enjoy the nitty-gritty details of Next. Me and my colleagues in Software Labs, will be back with more in depth stuff for you. If you have questions or suggestions don’t hesitate to leave a response to the blog entry, or send me a PM.



All blog posts

Get the 5 reasons why handling your
contracts in Excel can be a costly mistake.

STOP using Excel
for handling contracts