Python: Finding the Commit Charge Values in Windows

This week I was tasked with trying to find a way to find out what the Peak Commit value was on our virtual workstations. The reason being that we are trying to save money and were wondering if we were allocating too much memory or not. We didn't need the Total Commit Charge or the Limit Commit Charge values, but since I figured out how to get those during my research, I'll show how to get those as well.

When I first started searching on this topic, I tried such search terms as "python peak commit value" and variations thereof. That got me nowhere, so I replaced "oython" with "wmi" and found the Win32_PerfFormattedData_PerfOS_Memory Class on MSDN. I thought this was it, but it only gave me the Commit Charge Limit and the Total Commit Charge. Here's how I got those values using Tim Golden's WMI module:

import wmi

c = wmi.WMI()

for item in c.Win32_PerfFormattedData_PerfOS_Memory():
    commitChargeLimit = int(item.CommitLimit) / 1024
    
for item in c.Win32_PerfFormattedData_PerfOS_Memory():
    commitChargeTotal = int(item.CommittedBytes) / 1024
    
print "Commit Charge Limit: ", commitChargeLimit
print "Commit Charge Total: ", commitChargeTotal

This is neat stuff and shows how easy it is to take the documentation at MSDN and translate it into something usable in Python. Unfortunately, it just didn't give me the information I needed. My next stop was the PyWin32 mailing list where Mark Hammond informed me of the win32pdh and win32pdhutil modules. These expose performance counters, but I couldn't find a way to use it to get this information either. Fortunately, I found an old post on the sysinternals forum that gave me a clue. Here is what it said:

The only way I'm aware of that one can get this detail is from the uMmPeakCommitLimit member of the SYSTEM_PERFORMANCE_INFORMATION structure one passes to NtQuerySystemInformation when calling it with the SystemPerformanceInformation type.

I asked Mr. Hammond if that meant that I would need to use ctypes since the NtQuerySystemInformation class isn't exposed by PyWin32 and he said "probably". The ctypes module is pretty low-level and not something I've used except when I've copied a script from ActiveState. It's a pretty handy module though and was added to the standard library in version 2.5. From what I can tell, it was created by Thomas Heller.

Anyway, ctypes has its own mailing list, so I decided to try there. I received two replies, one of which was from the man himself (Heller). He gave me a script that didn't appear to work at first, but after a little back and forth with him, he got me straightened out. Here's the result:

from ctypes import *

SystemBasicInformation = 0
SystemPerformanceInformation = 2

class SYSTEM_BASIC_INFORMATION(Structure):
    _fields_ = [("Reserved1", c_long * 10),
                ("NumberOfProcessors", c_byte),
                ("bUnknown2", c_byte),
                ("bUnknown3", c_short)
                ]

class SYSTEM_PERFORMANCE_INFORMATION(Structure):
    _fields_ = [("IdleTime", c_int64),
                ("ReadTransferCount", c_int64),
                ("WriteTransferCount", c_int64),
                ("OtherTransferCount", c_int64),
                ("ReadOperationCount", c_ulong),
                ("WriteOperationCount", c_ulong),
                ("OtherOperationCount", c_ulong),
                ("AvailablePages", c_ulong),
                ("TotalCommittedPages", c_ulong),
                ("TotalCommitLimit", c_ulong),
                ("PeakCommitment", c_ulong),
                ("PageFaults", c_ulong),
                ("WriteCopyFaults", c_ulong),
                ("TransitionFaults", c_ulong),
                ("Reserved1", c_ulong),
                ("DemandZeroFaults", c_ulong),
                ("PagesRead", c_ulong),
                ("PageReadIos", c_ulong),
                ("Reserved2", c_ulong * 2),
                ("PagefilePagesWritten", c_ulong),
                ("PagefilePageWriteIos", c_ulong),
                ("MappedFilePagesWritten", c_ulong),
                ("MappedFilePageWriteIos", c_ulong),
                ("PagedPoolUsage", c_ulong),
                ("NonPagedPoolUsage", c_ulong),
                ("PagedPoolAllocs", c_ulong),
                ("PagedPoolFrees", c_ulong),
                ("NonPagedPoolAllocs", c_ulong),
                ("NonPagedPoolFrees", c_ulong),
                ("TotalFreeSystemPtes", c_ulong),
                ("SystemCodePage", c_ulong),
                ("TotalSystemDriverPages", c_ulong),
                ("TotalSystemCodePages", c_ulong),
                ("SmallNonPagedLookasideListAllocateHits", c_ulong),
                ("SmallPagedLookasideListAllocateHits", c_ulong),
                ("Reserved3", c_ulong),
                ("MmSystemCachePage", c_ulong),
                ("PagedPoolPage", c_ulong),
                ("SystemDriverPage", c_ulong),
                ("FastReadNoWait", c_ulong),
                ("FastReadWait", c_ulong),
                ("FastReadResourceMiss", c_ulong),
                ("FastReadNotPossible", c_ulong),
                ("FastMdlReadNoWait", c_ulong),
                ("FastMdlReadWait", c_ulong),
                ("FastMdlReadResourceMiss", c_ulong),
                ("FastMdlReadNotPossible", c_ulong),
                ("MapDataNoWait", c_ulong),
                ("MapDataWait", c_ulong),
                ("MapDataNoWaitMiss", c_ulong),
                ("MapDataWaitMiss", c_ulong),
                ("PinMappedDataCount", c_ulong),
                ("PinReadNoWait", c_ulong),
                ("PinReadWait", c_ulong),
                ("PinReadNoWaitMiss", c_ulong),
                ("PinReadWaitMiss", c_ulong),
                ("CopyReadNoWait", c_ulong),
                ("CopyReadWait", c_ulong),
                ("CopyReadNoWaitMiss", c_ulong),
                ("CopyReadWaitMiss", c_ulong),
                ("MdlReadNoWait", c_ulong),
                ("MdlReadWait", c_ulong),
                ("MdlReadNoWaitMiss", c_ulong),
                ("MdlReadWaitMiss", c_ulong),
                ("ReadAheadIos", c_ulong),
                ("LazyWriteIos", c_ulong),
                ("LazyWritePages", c_ulong),
                ("DataFlushes", c_ulong),
                ("DataPages", c_ulong),
                ("ContextSwitches", c_ulong),
                ("FirstLevelTbFills", c_ulong),
                ("SecondLevelTbFills", c_ulong),
                ("SystemCalls", c_ulong)]

sbi = SYSTEM_BASIC_INFORMATION()
retlen = c_ulong()

res = windll.ntdll.NtQuerySystemInformation(SystemBasicInformation,
                                            byref(sbi),
                                            sizeof(sbi),
                                            byref(retlen))
print res, retlen
print sbi.NumberOfProcessors


spi = SYSTEM_PERFORMANCE_INFORMATION()
retlen = c_ulong()

res = windll.ntdll.NtQuerySystemInformation(SystemPerformanceInformation,
                                            byref(spi),
                                            sizeof(spi),
                                            byref(retlen))
print res, retlen
print "Peak commit: ",
print spi.PeakCommitment * 4096 / 1024

I don't really understand everything that's going on here, but I'm glad it works. Well, I should say that it works on Windows XP Professional, 32-bit with Python 2.5. I tried it on Windows 7 64-bit as well and while the script ran, it returned "0L". I'm guessing that 64-bit operating systems need a slightly different script, but since all our workstations currently use 32-bit, it doesn't really matter at this point. Once again, the Python community came through for me and showed me just how amazing they are!