1

On a HPE DL380 G10 Server with 2 Xeon(R) Gold 6246R (32 phys. Cores, 64 log. with HT), in the BIOS, using this setting:

System Configuration > BIOS/Platform Configuration (RBSU) > Performance Options > Advanced Performance Tuning Options > NUMA Group Size Optimization

We can choose of these 2 options:

  1. Clustered — Optimizes groups along NUMA boundaries, providing better performance.
  2. Flat - Enables applications that are not optimized to take advantage of processors spanning multiple groups to utilize morelogical processors.

The "Clustered" Option will force Windows to only reporting ONE NUMA Node BUT 2 Processor Groups, each one 32 Cores. The "Flat" Option on the other side shows 2 NUMA Nodes and only 1 Processor Group with 64 logical Cores.

Sure i know already a lot about NUMA and Processor groups, so i'm really not asking for any recommendations about the best Setting for my usecase or so. Instead the question is: Why would it make sense for windows to decide to go with 2 Processor Groups when there is only one NUMA node and vice-versa? Also, i never found any documentation that Windows (Server 2019) Splits 64 Cores into 2 Groups, is this expected behaviour at all?

Improve this question
6
  • It seems quite logical to me that "NUMA boundaries" for Clustered mean the same as "CPU boundaries". Since you have two CPUs, you end up with two processor groups. "Boundaries" then means to respect the physical architecture. And you're right: Impossible to find any documentation on the subject.
    – harrymc
    Commented Mar 19, 2021 at 14:25
  • I did find one supporting article which says: "The default setting is Clustered, which basically leads to the fact that Windows will create for each physical processor a processor group".
    – harrymc
    Commented Mar 19, 2021 at 16:01
  • Hey harry :-) hmmm the article seems to just say exactly what i said: the grouping is happening with the Clustered setting. This "boundaries" thing... i would kind of understand it if there were 2 numa nodes and 2 processor groups. Still i gues the main question is why the hell does windows group processors < 65 cores. All documents say you must have > 64 cores for grouping
    – Harry
    Commented Mar 20, 2021 at 11:44
  • My understanding is that 64 is the maximum in a group, not the minimum.
    – harrymc
    Commented Mar 20, 2021 at 11:53
  • Thats how it actually seems to be but reading docs.microsoft.com/en-us/windows/win32/procthread/… says "Support for systems that have more than 64 logical processors is based on the concept of a processor group," AND "On systems with 64 or fewer processors, existing applications will operate correctly without modification."
    – Harry
    Commented Mar 20, 2021 at 11:56

1 Answer 1

Reset to default
3
+25

The article you have found dates from the year 2008. At that time Windows was confronted with NUMA computers with more than 64 processors, while its implementation of processor groups was limited to 64. The solution then was to create automatically more than one such groups of no more than 64 processors each.

A more flexible solution was introduced end 2014. The earliest reference I have found is an HP Advisory note regarding HP Gen9 servers and dating from 2015-04-24:

In the Revision 1.30 (12/24/2014) and later versions of the System ROMs for Gen9 servers, a new ROM-Based Setup Utility, "NUMA Group Size Optimization," has been added that allows the user to change the behavior of reporting processors to the Operating System. This option will allow the OS to put all logical processors into a single group if there are 64 logical processors or fewer ("Flat".)

The Clustered option creates one processor group for the cores of each CPU. The Flat option is intended to let applications use all the computer cores, since in Windows by default an application is limited to a single group and therefore will see and use only its one group.

Regarding your questions:

Why would it make sense for windows to decide to go with 2 Processor Groups when there is only one NUMA node and vice-versa?

This is dictated by the Clustered option, which makes sense for an application and all its processes and threads to execute all of them on nearby memory for performance.

If an application needs in your case more than 32 cores, you should use the Flat option, to make all cores disponible.

I never found any documentation that Windows (Server 2019) Splits 64 Cores into 2 Groups, is this expected behaviour at all?

Yes, this is the expected behavior. It's not very well documented, but there are references for this behavior, for example Exchange performance:HP NUMA BIOS settings.

Improve this answer
15
  • Thanks a lot for all the investigations you did! It is clear that it obviously does what it does but unfortunately we still don't have any information about how it looks from windows perspective. All infos from HP basically just say like "it is as it is, we force windows to do some magical stuff" and the HP advisory points to the same MS Article from 2018 that i linked above (which again says that grouping needs at least 64 cores.. Did i overlook anything?
    – Harry
    Commented Mar 21, 2021 at 17:24
  • That's where is the main confusion: The article you found is marked as from 2018, but it's not. It applies at most to Windows Server 2008, which means that it was written in 2008. The 2018 date is probably the result of some automatic modification by the Microsoft website software (or an employee updating some format); it was not written on that date. It's now 13 years old, an eternity for computers.
    – harrymc
    Commented Mar 21, 2021 at 17:33
  • Interesting, Still the HP advisory liks to that article (and i didnt find any newer one), leaving me clueless and having no evidence that windows (e.g. 2019) is expected to group processors under 64 cores and also no clue how the decisions for grouping connects to NUMA nodes.
    – Harry
    Commented Mar 21, 2021 at 20:46
  • The rules you yourself listed in your post : flat/clustered. The advisory note links to that article for information about Processor Groups, which it describes with great detail.
    – harrymc
    Commented Mar 21, 2021 at 20:52
  • Sorry but i really don't get what you mean. Can yuo confirm "the link in the advisory note that describes it in great detail" is this link? msdn.microsoft.com/en-us/library/windows/desktop/… So the aricle from 2008?
    – Harry
    Commented Mar 21, 2021 at 21:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .