142

In a book I read the following:

32-bit processors have 2^32 possible addresses, while current 64-bit processors have a 48-bit address space

My expectation was that if it's a 64-bit processor, the address space should also be 2^64.

So I was wondering what is the reason for this limitation?

Improve this question
11
  • 19
    The book must have been talking specifically about the current implementation of the AMD64 architecture (x86-64). Only the low-order 48 bits are used. This is not a hardware limitation, though--all 64 bits are available.
    – Cody Gray - on strike ♦
    Commented Jul 16, 2011 at 11:12
  • 13
    Always a good idea to identify the book.
    – Henk Holterman
    Commented Jul 16, 2011 at 11:14
  • 1
    I'm guessing that physical address lines aren't free (you need 16 extra cpu pins at least). And i'm not aware of any hardware that can fill a 48 bit space with physical RAM chips on the same processor yet. When this becomes feasible, i'm sure AMD will add the missing 16 pins :)
    – Torp
    Commented Jul 16, 2011 at 11:17
  • 8
    even, The 32-bit processors have 2^32 possible addresses is not necessarily true, there can exist 32bit cpu with only 24 "pins" for addressing memory. E.g. 68EC020 (cheaper 68020 version) is a 32bit cpu but with 24 bits for addressing memory.
    – ShinTakezou
    Commented Jul 16, 2011 at 11:53
  • 28
    There's a very real problem with 64-bit physical addressing, the virtual memory page size is too small. Which makes for enormous page directories and extremely expensive TLB cache flushes on every context switch. Moving from 4KB to 4MB pages is an option but very incompatible with current operating systems.
    – Hans Passant
    Commented Jul 16, 2011 at 13:32

11 Answers 11

Reset to default
186

Because that's all that's needed. 48 bits give you an address space of 256 terabyte. That's a lot. You're not going to see a system which needs more than that any time soon.

So CPU manufacturers took a shortcut. They use an instruction set which allows a full 64-bit address space, but current CPUs just only use the lower 48 bits. The alternative was wasting transistors on handling a bigger address space which wasn't going to be needed for many years.

So once we get near the 48-bit limit, it's just a matter of releasing CPUs that handle the full address space, but it won't require any changes to the instruction set, and it won't break compatibility.

Improve this answer
16
  • 157
    640kb is enough for anyone.
    – user684934
    Commented Jul 16, 2011 at 11:35
  • 9
    Are you still running an 8088 system, bdares?
    – Joe
    Commented Jul 16, 2011 at 11:39
  • 33
    @bdares: Bad analogy. The 8088/8086 arch's instruction set has a 640k limit built into it. Only making a new ISA (386) was it possible to break the barrier. x86_64 on the other hand supports all 64 bits in the ISA. It's just the current-generation hardware that can't make use of them all...
    – R.. GitHub STOP HELPING ICE
    Commented Jul 16, 2011 at 12:29
  • 18
    @R. Actually, the limitation in the CPU was one megabyte. The IBM PC designated a section of that for memory mapped peripherals, BIOS, etc. Some other 8088/8086 designs (Zenith Z100, if memory serves) designated less for peripherals and such, and correspondingly more for application programs.
    – Jerry Coffin
    Commented Jul 16, 2011 at 19:30
  • 30
    lwn.net/SubscriberLink/655437/9a48cd3e7a8cbe8a <-- three years after this reply, we are already hitting these limits :) The HP Machine will have 320TB of memory and they can't provide it as a flat address space because of the 48-bit addressing limitation.
    – agam
    Commented Aug 28, 2015 at 19:27
23

Any answer referring to the bus size and physical memory is slightly mistaken, since OP's question was about virtual address space not physical address space. For example the supposedly analogous limit on some 386's was a limit on the physical memory they could use, not the virtual address space, which was always a full 32 bits. In principle you could use a full 64 bits of virtual address space even with only a few MB of physical memory; of course you could do so by swapping, or for specialized tasks where you want to map the same page at most addresses (e.g. certain sparse-data operations).

I think the real answer is that AMD was just being cheap and hoped nobody would care for now, but I don't have references to cite.

Improve this answer
9
  • 21
    "Being cheap" I guess you mean not adding pins that will never be used, not taking up chip space for transistors that won't be used and using the freed space to make existing instructions faster? If that's being cheap, I'm in!
    – Olof Forshell
    Commented Jul 17, 2011 at 5:16
  • The 80386 allows 2 * 4096 selectors each containing up to 4GB of memory (32TB total). The 80286 allowed 2 * 4096 selectors each containing up to 64KB (1GB).
    – Olof Forshell
    Commented Jul 17, 2011 at 5:24
  • 1
    Non-linear segmented hacks do not count as address space in my book. There's no way for portable software to make any use of them.
    – R.. GitHub STOP HELPING ICE
    Commented Jul 17, 2011 at 6:00
  • @R.. - I thought the definition of portable software is that it can. :-) For example, C++ forbids comparing pointers into different arrays so that they can be in separate 4GB segments.
    – Bo Persson
    Commented Jul 17, 2011 at 7:48
  • 1
    If your compile actually generates huge pointers and loads a segment register for each memory dereference then yes. But in reality that's horribly slow, and instead everyone used small memory models and __far (or worse yet, FAR/far!) pointers...
    – R.. GitHub STOP HELPING ICE
    Commented Jul 17, 2011 at 13:04
16

There is a more severe reason than just saving transistors in the CPU address path: if you increase the size of the address space you need to increase the page size, increase the size of the page tables, or have a deeper page table structure (that is more levels of translation tables). All of these things increase the cost of a TLB miss, which hurts performance.

Improve this answer
2
  • 4
    Intel is proposing a 5-level paging scheme to extend from the current 48 bits to 57 bits. (Same 9 bits per level / 4k pages as current x86-64 page tables). Using 10 or 11 bits per level would have required changing the page-walk hardware, so this might not be the optimal design for huge memory, but it's a sensible extension for a dual-mode CPU that needs to also support maximum performance for 4-level tables in the current format.
    – Peter Cordes
    Commented Dec 4, 2017 at 2:22
  • 1
    Of course, with 2M or 1G hugepages, it's only 4 or 3 levels of page tables from top level to a huge-page table entry instead of a page directory pointer.
    – Peter Cordes
    Commented Dec 4, 2017 at 2:28
14

The internal native register/operation width does not need to be reflected in the external address bus width.

Say you have a 64 bit processor which only needs to access 1 megabyte of RAM. A 20 bit address bus is all that is required. Why bother with the cost and hardware complexity of all the extra pins that you won't use?

The Motorola 68000 was like this; 32 bit internally, but with a 23 bit address bus (and a 16 bit data bus). The CPU could access 16 megabytes of RAM, and to load the native data type (32 bits) took two memory accesses (each bearing 16 bits of data).

Improve this answer
6
  • 1
    but 68000 is considered as a "16/32 bit" cpu, not "full" 32 bit cpu so one could say it has still a foot in the 16bit past; I've picked the 68020 as an example, since its low-cost 68EC020 version has 24 bit only for addresses, though the 68020 is a "full" 32 bit cpu... +1 to have considered this wonderful processor family!
    – ShinTakezou
    Commented Jul 16, 2011 at 11:59
  • @ShinTakezou: honestly, was the 80386SX a 16-bit CPU (because it had an address space like the 80286) or was it 32-bit (because it had the internal architecture of an 80386DX)? One could say as you do but another (this one) says "internal is what counts" - and you can quote me on that.
    – Olof Forshell
    Commented Jul 17, 2011 at 17:24
  • @Olof I think that, in the context of the "memory" (which is the external world), external is what counts, so 68000 is a 16bit CPU (needing 2 "steps" to read 32 bit data) :D
    – ShinTakezou
    Commented Jul 17, 2011 at 19:24
  • @ShinTakezou: the memory context, even caches, is always external to the cpu itself even though they are extremely tightly coupled in modern processors. The 8088 was internally equal to the 8086 though it had eight data bus lines to the 8086's sixteen. I don't see what you apparently see as obvious, that the 8088 should be classified in the same group as the Z80, 8080, 8085 etc. The question of the width of the data bus seems trivial in that context
    – Olof Forshell
    Commented Jul 18, 2011 at 10:58
  • I am not an expert of such a matter at all,so I have nothing obvious to me.I wanted just to notice the need for a sharper cut with the past, where one could think 68000 is still an "old time" processor, so that it could seem "natural" that its address space is limited to less than 32 bit;while the 68020 can 32 bit, so that the existence of the 68EC020 with its limit makes clear that it's a choice not due to "limit of that (or this) time" but to other consideration (like to make it cheaper if there's no real advantage in having 64 pins), which is more or less the argument of this answer.
    – ShinTakezou
    Commented Jul 18, 2011 at 11:13
12

Many people have this misconception. But I am promising to you if you read this carefully, after reading this all your misconceptions will be clear.

To say a processor 32-bit or 64-bit doesn't signify it should have a 32-bit address bus or 64-bit address bus respectively!... I repeat it DOESN'T!!

32-bit processor means it has a 32-bit ALU (Arithmetic and Logic Unit)... which means it can operate on a 32-bit binary operand (or simply saying a binary number having 32 digits) and similarly 64-bit processor can operate on a 64-bit binary operand. So whether a processor 32-bit or 64-bit DOESN'T signify the maximum amount of memory can be installed. They just show how large the operand can be...(for analogy you can think of a 10-digit calculator that can calculate results up to 10 digits...it cannot give us 11 digits or any other bigger results... although it is in decimal but I am telling this analogy for simplicity)...but what you are saying is address space that is the maximum directly interfaceable size of memory (RAM). The RAM's maximum possible size is determined by the size of the address bus and it is not the size of the data bus or even ALU on which the processor's size is defined (32/64 bit). Yes if a processor has a 32-bit "Address bus" then it can address 2^32 byte=4GB of RAM (or for 64 bit it will be 2^64)...but saying a processor 32-bit or 64-bit has nothing relevant to this address space (address space = how far it can access to the memory or the maximum size of RAM), rather it is only depended on the size of its ALU. Of course, the data bus and the address bus may be of same sized and then it may seem that a 32-bit processor means it will access 2^32 byte or 4 GB memory...but it is a coincidence only and it won't be the same for all....for example, intel 8086 is a 16-bit processor (as it has 16 bit ALU) so as your saying it should have accessed to 2^16 byte=64 KB of memory but it is not true. It can access up to 1 MB of memory for having a 20-bit address bus.

Now coming to your question... a 64-bit processor doesn't mean that it must have a 64-bit address bus so there is nothing wrong with having a 48-bit address bus in a 64-bit processor...they kept the address space smaller to make the design and fabrication cheap....as nobody gonna use such a big memory (2^64 byte)...where 2^48 byte is more than enough nowadays.

Improve this answer
2
  • I think you made your point very clear, there is one thing I don't understand though in what you said about the 16 bits 8086 CPU : how can a 16 bits CPU handle a 20 bits address ? Does it handle it through a 2 steps operation ? Even if the address bus is 20 bits width, once it gets to the CPU, the register width can obviously only take 16 bits ... How do they do that ?
    – programmersn
    Commented Jan 1, 2019 at 13:06
  • 3
    Hmm...2 steps operation. Segment register contains only the upper 16 bits. Then it is multiplied by 10H to make it 20 bits and then the offset is added.
    – hafiz031
    Commented Jan 6, 2019 at 2:27
11

Read the limitations section of the wikipedia article:

A PC cannot contain 4 petabytes of memory (due to the size of current memory chips if nothing else) but AMD envisioned large servers, shared memory clusters, and other uses of physical address space that might approach this in the foreseeable future, and the 52 bit physical address provides ample room for expansion while not incurring the cost of implementing 64-bit physical addresses

That is, there's no point implementing full 64 bit addressing at this point, because we can't build a system that could utilize such an address space in full - so we pick something that's practical for today's (and tomorrow's) systems.

Improve this answer
2
  • Where does the 4 come from in the 4 petabytes? If we're talking 64 address lines we should end up with the square of the address space made possible by 32 address lines which is 4 gigabytes. Square that and we should have 16, not 4 petabytes. Am I missing something?
    – Olof Forshell
    Commented Jul 18, 2011 at 11:06
  • 1
    It comes from the current physical limit (52 bits) - the point being that we can't put enough RAM in a PC to support this restricted range, let alone what would be required for a full 64-bit address space.
    – Damien_The_Unbeliever
    Commented Jul 18, 2011 at 11:10
9

From my point of view, this is result from the page size.Each page at most contains 4096/8 =512 entries of page table. And 2^9 =512. So 9 * 4 + 12=48.

Improve this answer
5

To answer the original question: There was no need to add more than 48 Bits of PA.

Servers need the maximum amount of memory, so let's try to dig deeper.

1) The largest (commonly used) server configuration is an 8 Socket system. An 8S system is nothing but 8 Server CPU's connected by a high speed coherent interconnect (or simply, a high speed "bus") to form a single node. There are larger clusters out there but they are few and far between, we are talking commonly used configurations here. Note that in the real world usages, 2 Socket system is one of the most commonly used servers, and 8S is typically considered very high end.

2) The main types of memory used by servers are byte addressable regular DRAM memory (eg DDR3/DDR4 memory), Memory Mapped IO - MMIO (such as memory used by an add-in card), as well as Configuration Space used to configure the devices that are present in the system. The first type of memory is the one that are usually the biggest (and hence need the biggest number of address bits). Some high end servers use a large amount of MMIO as well depending on what the actual configuration of the system is.

3) Assume each server CPU can house 16 DDR4 DIMMs in each slot. With a maximum size DDR4 DIMM of 256GB. (Depending on the version of server, this number of possible DIMMs per socket is actually less than 16 DIMMs, but continue reading for the sake of the example).

So each socket can theoretically have 16*256GB=4096GB = 4 TB. For our example 8S system, the DRAM size can be a maximum of 4*8= 32 TB. This means that the max number of bits needed to address this DRAM space is 45 (=log2 32TB/log2 2).

We wont go into the details of the other types of memory (MMIO, MMCFG etc), but the point here is that the most "demanding" type of memory for an 8 Socket system with the largest types of DDR4 DIMMs available today (256 GB DIMMs) use only 45 bits.

For an OS that supports 48 bits (WS16 for example), there are (48-45=) 3 remaining bits. Which means that if we used the lower 45 bits solely for 32TB of DRAM, we still have 2^3 times of addressable memory which can be used for MMIO/MMCFG for a total of 256 TB of addressable space.

So, to summarize: 1) 48 bits of Physical address is plenty of bits to support the largest systems of today that are "fully loaded" with copious amounts of DDR4 and also plenty of other IO devices that demand MMIO space. 256TB to be exact.

Note that this 256TB address space (=48bits of physical address) does NOT include any disk drives like SATA drives because they are NOT part of the address map, they only include the memory that is byte-addressable, and is exposed to the OS.

2) CPU hardware may choose to implement 46, 48 or > 48 bits depending on the generation of the server. But another important factor is how many bits does the OS recognize. Today, WS16 supports 48 bit Physical addresses (=256 TB).

What this means to the user is, even though one has a large, ultra modern server CPU that can support >48 bits of addressing, if you run an OS that only supports 48 bits of PA, then you can only take advantage of 256 TB.

3) All in all, there are two main factors to take advantage of higher number of address bits (= more memory capacity).

a) How many bits does your CPU HW support? (This can be determined by CPUID instruction in Intel CPUs).

b) What OS version are you running and how many bits of PA does it recognize/support.

The min of (a,b) will ultimately determine the amount of addressable space your system can take advantage of.

I have written this response without looking into the other responses in detail. Also, I have not delved in detail into the nuances of MMIO, MMCFG and the entirety of the address map construction. But I do hope this helps.

Thanks, Anand K Enamandram, Server Platform Architect Intel Corporation

Improve this answer
2
  • This question is asking about 48-bit virtual address space size (requiring virtual addresses to be canonical). You do want more virtual bits than physical bits, so a high-half kernel can map all of physical memory into a single address space (it's own or user-space). As you say, HW only needs to implement as many PA bits as the DRAM controllers + MMIO can use, and can use any number up to the 52-bit limit in the x86-64 page-table format. (Why in 64bit the virtual address are 4 bits short (48bit long) compared with the physical address (52 bit long)?)
    – Peter Cordes
    Commented Oct 28, 2018 at 17:24
  • 2
    The 4-level page-table format also imposes the 48-bit VA limit, until HW + SW support PML5 page tables for 57-bit VAs. Anyway, this is a useful answer, but it seems to be posted under the wrong question. I'm not sure if there's a better place for it, so I guess we can leave it here, hopefully with an edit to add a header to say something about PA vs. VA.
    – Peter Cordes
    Commented Oct 28, 2018 at 17:29
3

It's not true that only the low-order 48 bits of a 64 bit VA are used, at least with Intel 64. The upper 16 bits are used, sort of, kind of.

Section 3.3.7.1 Canonical Addressing in the Intel® 64 and IA-32 Architectures Software Developer’s Manual says:

a canonical address must have bits 63 through 48 set to zeros or ones (depending on whether bit 47 is a zero or one)

So bits 47 thru 63 form a super-bit, either all 1 or all 0. If an address isn't in canonical form, the implementation should fault.

On AArch64, this is different. According to the ARMv8 Instruction Set Overview, it's a 49-bit VA.

The AArch64 memory translation system supports a 49-bit virtual address (48 bits per translation table). Virtual addresses are sign- extended from 49 bits, and stored within a 64-bit pointer. Optionally, under control of a system register, the most significant 8 bits of a 64-bit pointer may hold a “tag” which will be ignored when used as a load/store address or the target of an indirect branch

Improve this answer
6
  • 1
    Only the lower 48 are significant, but the hardware checks that it's correctly sign-extended to 64 bits. IDK why they didn't specify zero-extension; maybe they wanted to make it more convenient to check for a high vs. low half address (by just checking the sign bit). Or maybe to avoid making the 2^48 boundary special, and so addresses near the top can conveniently fit in 32-bit sign-extended constants. I think the latter is more likely.
    – Peter Cordes
    Commented Aug 9, 2017 at 1:31
  • Anyway, current HW checking for canonical prevents software from using ignored bits for tagged pointers that will break on future HW, so it's part of the mechanism that makes it possible to extend future hardware if/when it's needed. (Which could be sooner rather than they expected, thanks to non-volatile memory being hooked up directly into physical and virtual address space.)
    – Peter Cordes
    Commented Aug 9, 2017 at 1:34
  • procfs on Linux on my Core i5 says that it gets mapped to 7ffd5ea41000-7ffd5ea62000. This address range makes sense according to the above 'canonical' rule. Bit 48-63 are 0 making it a correct canonical address. What's a little strange are some addresses in the Linux source. In include/asm/pgtable_64_types it says #define __VMALLOC_BASE _AC(0xff92000000000000, UL). This is NOT a canonical address. Such an address would start with 0xffff8. Dunno why.
    – Olsonist
    Commented Oct 5, 2017 at 16:47
  • Yeah, IIRC Linux uses the low half of the canonical range for user-space, and (mostly) uses the high half for kernel-only mappings. But some kernel memory is exported to user-space, like the [vsyscall] page. (That may be exporting stuff like current PID so that getpid() is purely user-space. Also gettimeofday() can just use rdtsc in user-space + scale factors exported by the kernel. Although some of that is I think in [vdso], which is near the top of the bottom half.)
    – Peter Cordes
    Commented Oct 5, 2017 at 19:57
  • IDK what __VMALLOC_BASE does. Presumably it's not used directly.
    – Peter Cordes
    Commented Oct 5, 2017 at 19:58
1

I concur with @linzuojian's answer.

In 32-bit non-PAE mode where entry size == 4bytes you have 10-bits for indexing both PDE and PTE because 4k/4 == 1024 == 2^10 = 10 bits

In PAE the entry size is 8 bytes, I think intel is trying to limit the table of whatever level to 4K, you are left with 4k/8 == 512 == 2^9 == 9 bits

The 48 bits came from 9-bit PML4 + 9-bit PDPT + 9-bit PD + 9-bit PT + 12-bit page_offset.

interestingly, intel has extended the VA from 48 to 57 with PML5, which is exactly another 9 bits. I guess they can further extend it to 64 but then you are only able to index 2^7 == 128 entries in the supposed PML6

In conclusion, the only reasonable VA sizes would be 12 + 9x -> 39, 48, 57. you are free to pick any VA size >= the physical address size. Pick a small one to save some circuitry, pick a large one to allow for more non-memory backed region.

Why limiting tables to 4K? maybe it's for memory efficiency since each context can use as little as 8K for memory management data structure (PML4 + PDPT) or 12K (PML5 + PML4 + PDPT)

Improve this answer
4
  • 4K tables means OSes don't need to find 2 contiguous physical pages when allocating space for PTE and other levels of the table. They use physical addresses so I don't think there's any other major reason why they couldn't be 8K to still be 10 bits per level. And yes, in a hypothetical PML6, the topmost level would only be partial, same as with the top level of 32-bit PAE. But the PTE format only has room for 52 physical address bits, or a few more if they change some of the "ignored" bits to be part of the physical addresses; IDK how many bits most OS need for their own data.
    – Peter Cordes
    Commented Sep 9, 2023 at 11:59
  • 1
    good point about contiguous memory for table allocation! I guess we'll never see PML6 since MAXPHYADDR is 52
    – XYZ
    Commented Sep 9, 2023 at 12:16
  • 1
    Your point about not using as much space is also a good one; if you want to map an isolated 4K page somewhere, in the worst case it needs a PDPT, PD, and PT, each 8K instead of 4K. (And the top-level PML4 would be 8K instead of 4K, if it still used 4 levels for 52-bit virtual instead of 42-bit virtual.) Hmm, the design choice to stick with 4K tables was made for PAE; Intel wasn't considering a 64-bit extension to x86 at the time. Sticking to 10 bits per level would have been attractive for 32-bit, still being 2 levels. (Unlike with x86-64 where 9 bits per level works out nicely to 48-bit.)
    – Peter Cordes
    Commented Sep 9, 2023 at 12:23
  • right, I didn't pay attention that PML5 or PML4 can't be the final level
    – XYZ
    Commented Sep 9, 2023 at 21:14
0

A CPU is considered "N-bits" mainly upon its data-bus size, and upon big part of it's entities (internal architecture): Registers, Accumulators, Arithmetic-Logic-Unit (ALU), Instruction Set, etc. For example: The good old Motorola 6800 (or Intel 8050) CPU is a 8-bits CPU. It has a 8-bits data-bus, 8-bits internal architecture, & a 16-bits address-bus.

  • Although N-bits CPU may have some other than N-size entities. For example the impovments in the 6809 over the 6800 (both of them are 8-bits CPU with a 8-bits data-bus). Among the significant enhancements introduced in the 6809 were the use of two 8-bit accumulators (A and B, which could be combined into a single 16-bit register, D), two 16-bit index registers (X, Y) and two 16-bit stack pointers.
Improve this answer
4
  • There's already an answer making this point with Motorola 68000 / 68020 as an example. This question is really about x86-64 specifically, not old 8 / 16-bit CPUs. In the case of x86-64, one of the major factors is that wider virtual addresses would need a deeper page table, and that factor didn't exist for the old chips you're talking about.
    – Peter Cordes
    Commented Jul 22, 2018 at 7:53
  • data-bus width doesn't have to match register or ALU width. For example, P5 Pentium has a 64-bit data bus (aligned 64-bit loads/stores are guaranteed to be atomic), but registers/ALUs are only 32 bit (except for the integrated FPU, and in the later Pentium MMX the SIMD ALUs.)
    – Peter Cordes
    Commented Jul 22, 2018 at 7:55
  • OP write: "My expectation was that if it's a 64-bit processor, the address space should also be 2^64." ........ You write: "This question is really about x86-64 specifically, not old 8 / 16-bit CPUs". ........ I think you missed the essence of OP question. OP question is an outcome of the wrong assumption that a 64-bits CPU should have a 64-bits address-bus. About the ALU, I wrote big part of its entities; Not all of them.
    – Amit G.
    Commented Jul 22, 2018 at 8:55
  • Stop spamming me by reposting this comment. Yes of course the OP is wrong for the reason you describe, but I was pointing out that your answer looks like it makes a similar mistake. You say "and consequently big part of it's entities: Registers and Accumulators, Arithmetic-Logic-Unit (ALU) ...", which sounds like you're saying that those things match the data bus width. The phrase "a big part" implies that you're saying which parts, not that it's only sometimes true for those parts.
    – Peter Cordes
    Commented Jul 22, 2018 at 8:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.