Why did DOS-based Windows require HIMEM.SYS to boot?

Question

My understanding is that all versions of Microsoft Windows that ran on top of DOS — that is, the lineage from Windows 1.0 up to Windows ME, even though the reliance on DOS diminished over time — required HIMEM.SYS, the extended memory manager for DOS, to be loaded in order to be able to start in protected mode (thus, from Windows 3.1 on, to be able to start at all).

Contemporary documentation used to make it very clear that the driver was mandatory, but did not elaborate on why it was so. HIMEM.SYS was supposed to provide XMS memory to real mode DOS programs, but since the protected mode Windows kernel was to take over memory management and access all installed memory directly, relying on a 16-bit DOS driver would seem rather nonsensical, especially considering that XMS itself could not be used to access memory arbitrarily but in pieces that had to be copied back and forth between conventional and extended memory.

So, what was the role of HIMEM.SYS with Windows? If it was used just as an intermediary for the Windows loader (which had to be a real mode DOS program) to be able to stuff the insides of Windows into memory before kernel start, how was the switching over to Windows managing the memory accomplished? Did this require HIMEM.SYS to "know" and cooperate with Windows in some way?

Answer 1

Windows in protected mode requires an XMS driver — HIMEM.SYS is the one provided by Microsoft, but others can be used¹ (and the Windows Me kernel includes its own XMS driver). The role of the XMS driver depends on the version of Windows and the mode it’s run in. The basic requirement is that Windows needs to be able to access extended memory, and it needs to do so without losing anything loaded before it (such as the SMARTDrive disk cache).

At the XMS level, there isn’t much cooperation involved: Windows allocates all the available XMS memory (after asking programs such as SMARTDrive to free some memory up if appropriate), and then handles XMS requests itself. XMS includes the necessary mechanism, a chain of hooks such that the calling address for XMS never changes, but the latest XMS handler ends up being called first. This allows Windows to handle XMS requests for programs which were loaded before it started. A good reference for this is Geoff Chappell’s DOS Internals.

Given that Windows, at least in enhanced mode, includes its own XMS handler, one may wonder why it needs to rely on HIMEM.SYS or another driver at all. I suspect that this is to avoid duplication of effort: HIMEM.SYS needs to deal with hardware specifics, A20 gate handling etc.; the Windows driver doesn’t need to worry about that.

Things get more interesting when more complex memory managers are loaded. Such memory managers can only run Windows in enhanced mode if they implement the Global Expanded Memory Manager Import interface, which allows Windows to retrieve all the necessary runtime information from a pre-existing memory manager.

In some cases more dubious shenanigans are required. For example, Windows 3.0 standard mode refuses to run if another enhanced memory manager is already loaded; to work around this, QEMM386 patches the Windows DOS extender (DOSX.EXE) as it loads!

---

¹ Including Microsoft’s own XMSMMGR.EXE, which is used during Windows setup on systems that don’t already have XMS.

Answer 2

It is entirely possible that it required himem simply because it had to support himem, to start with. I.e. if you have himem.sys loaded, then windows can't just look into e820 and map all the available memory. It needs to allocate if from himem. If you support also the no-himem case, then you'll need an entirely different code path, which needs to be maintained and tested. So probably they just wanted to reduce the maintenance burden and not introduce the second code path.

Another interesting scenario may be that windows used the big real mode during startup - that mode is set up by himem.sys if it itself was started in real mode. (of course this doesn't apply to the case when also emm386 is loaded, which needs an entirely different startup technique)

Yet another possibility is that windows just wanted to keep as much free conventional memory as possible, or otherwise many DOS programs would OOM in its DOS boxes. So it insisted on loading himem.sys, but this is less likely to be the reason because IIRC the DOS=HIGH wasn't a requirement.

One more thing I can very vaguely recall, is that there were non-standard PCs with the vendor-specific a20 gating implemented in a custom himem.sys from vendor. In that case loading himem.sys would be a mandatory.

Answer 3

Because until Windows 95 it was not always standard practice to load straight into Windows.

In the Windows 3.x days it was pretty common for only a few of your programs to be Windows apps. Everything else, and the vast majority of games that weren't Sim City 2000, ran in DOS. These programs still required more than low memory to run. So if you could manage to do it all in one autoexec.bat it was pretty common for the boot process to leave you at a DOS prompt with himem or some other memory manager running and let you invoke windows from the PATH if you needed to run Word or whatever.

It was also still pretty common to need a boot menu or boot disk with different startup config to get games running. Windows and games memory requirements were not particularly friendly to eachother.