PAE sucks (Why didn't MS take advantage of PAE?)

David W. Hess (dwhess@banishedsouls.org) on 5/13/07 wrote:
>
>I have tended to run into the physical memory space
>limitations instead of the single process memory
>limitations so I do not understand why Microsoft did not
>allow the use of PAE and more then 4GB of physical address
>space in their desktop operating systems.

PAE really really sucks.

The biggest single reason to go 64-bit is exactly
because of physical address space. Your virtual address
space needs to bea multiple of the physical one:
when you hit 1GB of RAM, 32-bit virtual memory is no longer
acceptable. You literally do need more virtual memory than
physical.

PAE turned that very simple fact on its head, and screwed
things up royally. Whoever came up with the idea was
totally incompetent, and had forgotten all the DOS HIGHMEM
pains. There's a damn good reason why we left the 286
behind, and started using 386's, instead of having HIGHMEM
crap with windows into a bigger physical space.

Repeat after me: virtual space needs to be bigger than
physical space. Not "as big". Not "smaller". It needs to
be bigger, by a factor of at least two, and that's
quite frankly pushing it, and you're much better off
having a factor of ten or more.

Anybody who doesn't get that is a moron. End of discussion.

Reasons for why you need a bigger virtual space:

- you need to map that physical memory somehow, and no,
tiny windows into the physical memory simply do not
cut it! If you cannot have normal pointers to the
physical space, it immediately means that you need to
jump through serious hoops to get there.

- you additionally need to be able to remap things
in alternate ways (ie user space) or make space for
non-memory issues (virtual page tables, IO, you name it)

Ergo, a factor-of-two is a requirement. PAE was a total
and utter disaster.

Yes, Linux supported it, and probably did so better than
anybody else. But "better than anybody else" still wasn't
very good. Because you couldn't use normal pointers to
point to arbitrary physical memory, all the memory that
couldn't be accessed directly (ie anythign that didn't fit
in the virtual address map, which also had the user
space memory in it) was basically limited to "special uses
only".

So you could allocate user pages in it, but you had huge
problems with things like internal kernel data structures,
which can be the bulk of your memory needs under some
(not that unusual) loads. Directory caches, inodes, etc
couldn't use it, and in general it meant that under Linux,
if you had more than 4GB of physical memory, you generally
ran into problems (since only 25% of memory was availble
for normal kernel stuff - the rest had to be addressed
through small holes in the tiny virtual address space).

I'm not at all surprised that Windows didn't push PAE
either. It was a total braindamage. I bet they supported
it in the server offerings just because they had to, and
I bet they did a much worse job of it than Linux did, and
the reason you can do that with servers is that the loads
are much easier, and you can expect maintainers to set
magic config entries to tweak it to make it appear
to work well for any particular load, when in reality it
is fragile as hell and works only with duct-tape and
prayers.

That kind of "duct-tape and prayers and lots of specialized
knowledge about the load" is simply not possible in
a desktop environment. Yeah, users have prayers, but they
lack the duct-tape and the knowledge to work around the
problems.

And dammit, in this age and date when almost everybody
has a gigabyte of RAM in any new machine, anybody who still
thinks that "not that many people need 64-bits" is simply
not aware of what he's speaking of.

Go back and play with HIGHMEM.SYS on a 286, and stop
blathering crap. When you've spent the last ten
years of your life working with HIGHMEM.SYS, then
you can come back and tell me that we still don't need
64 bits. Until that is the case, anybody who still doesn't
get why 64 bits is a requirement should just shut
up rather than make a total fool of himself.

So repeat after me: PAE didn't ever really fix anything.
It was a mistake. It was just a total failure, and the
result of hw engineers not understanding software.

So no, PAE does not mean that you can use more than
4GB of RAM. Even before PAE, the practical limit was around
1GB, and PAE didn't move that post a fraction of an inch!

Linus