PGO tests: mimalloc: warning: unable to allocate OS memory

[furkanonder]

Bug report

Bug description:

Configuration:

./configure --enable-optimizations --with-valgrind --enable-profiling --enable-pystats --with-pydebug --with-trace-refs --with-assertions --with-lto=full --disable-gil

Output: configure-output.txt

Build:

# Next, run the profile task to generate the profile information.
./python -m test --pgo --timeout=
Using random seed: 2967610669
0:00:00 load avg: 9.52 Run 44 tests sequentially
0:00:00 load avg: 9.52 [ 1/44] test_array
0:00:13 load avg: 9.46 [ 2/44] test_base64
0:00:18 load avg: 9.11 [ 3/44] test_binascii -- test_base64 failed (env changed)
0:00:19 load avg: 9.11 [ 4/44] test_binop
0:00:19 load avg: 9.11 [ 5/44] test_bisect
0:00:20 load avg: 9.11 [ 6/44] test_bytes
0:01:34 load avg: 9.41 [ 7/44] test_bz2 -- test_bytes passed in 1 min 14 sec
0:01:40 load avg: 9.53 [ 8/44] test_cmath
0:01:43 load avg: 9.97 [ 9/44] test_codecs
0:02:02 load avg: 9.81 [10/44] test_collections
0:02:15 load avg: 9.99 [11/44] test_complex
0:02:18 load avg: 9.67 [12/44] test_dataclasses
0:02:22 load avg: 9.06 [13/44] test_datetime
sys:1: DeprecationWarning: Parsing dates involving a day of month without a year specified is ambiguious
and fails to parse leap day. The default behavior will change in Python 3.15
to either always raise an exception or to use a different default year (TBD).
To avoid trouble, add a specific year to the input & format.
See https://github.com/python/cpython/issues/70647.
sys:1: DeprecationWarning: Parsing dates involving a day of month without a year specified is ambiguious
and fails to parse leap day. The default behavior will change in Python 3.15
to either always raise an exception or to use a different default year (TBD).
To avoid trouble, add a specific year to the input & format.
See https://github.com/python/cpython/issues/70647.
sys:1: DeprecationWarning: Parsing dates involving a day of month without a year specified is ambiguious
and fails to parse leap day. The default behavior will change in Python 3.15
to either always raise an exception or to use a different default year (TBD).
To avoid trouble, add a specific year to the input & format.
See https://github.com/python/cpython/issues/70647.
sys:1: DeprecationWarning: Parsing dates involving a day of month without a year specified is ambiguious
and fails to parse leap day. The default behavior will change in Python 3.15
to either always raise an exception or to use a different default year (TBD).
To avoid trouble, add a specific year to the input & format.
See https://github.com/python/cpython/issues/70647.
sys:1: DeprecationWarning: Parsing dates involving a day of month without a year specified is ambiguious
and fails to parse leap day. The default behavior will change in Python 3.15
to either always raise an exception or to use a different default year (TBD).
To avoid trouble, add a specific year to the input & format.
See https://github.com/python/cpython/issues/70647.
sys:1: DeprecationWarning: Parsing dates involving a day of month without a year specified is ambiguious
and fails to parse leap day. The default behavior will change in Python 3.15
to either always raise an exception or to use a different default year (TBD).
To avoid trouble, add a specific year to the input & format.
See https://github.com/python/cpython/issues/70647.
0:03:28 load avg: 8.38 [14/44] test_decimal -- test_datetime passed in 1 min 5 sec
mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0xbafc24672800000 bytes, align: 0x2000000, commit: 1, allow large: 1)
mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0xbafc24672800000 bytes, align: 0x2000000, commit: 1, allow large: 1)
mimalloc: error: unable to allocate memory (842105263157894768 bytes)
mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0xbafc24672800000 bytes, align: 0x2000000, commit: 1, allow large: 1)
mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0xbafc24672800000 bytes, align: 0x2000000, commit: 1, allow large: 1)
mimalloc: error: unable to allocate memory (842105263157894768 bytes)
mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0x5d7e12339800000 bytes, align: 0x2000000, commit: 1, allow large: 1)
mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0x5d7e12339800000 bytes, align: 0x2000000, commit: 1, allow large: 1)
mimalloc: error: unable to allocate memory (421052631578947400 bytes)
mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0x5d7e12339800000 bytes, align: 0x2000000, commit: 1, allow large: 1)
mimalloc: warning: unable to allocate OS memory (error: 12 (0xc), size: 0x5d7e12339800000 bytes, align: 0x2000000, commit: 1, allow large: 1)
mimalloc: error: unable to allocate memory (421052631578947400 bytes)
0:04:51 load avg: 7.48 [15/44] test_difflib -- test_decimal passed in 1 min 23 sec
0:05:24 load avg: 7.89 [16/44] test_embed -- test_difflib passed in 33.0 sec
0:06:54 load avg: 8.15 [17/44] test_float -- test_embed passed in 1 min 30 sec
0:06:59 load avg: 8.54 [18/44] test_fstring
0:07:08 load avg: 8.07 [19/44] test_functools
0:07:10 load avg: 8.07 [20/44] test_generators
0:07:13 load avg: 7.50 [21/44] test_hashlib
0:07:15 load avg: 7.50 [22/44] test_heapq
0:07:27 load avg: 7.29 [23/44] test_int
0:07:31 load avg: 7.19 [24/44] test_itertools
0:08:22 load avg: 7.53 [25/44] test_json -- test_itertools passed in 50.9 sec
0:09:32 load avg: 7.85 [26/44] test_long -- test_json passed in 1 min 9 sec
0:10:36 load avg: 8.51 [27/44] test_lzma -- test_long passed in 1 min 4 sec
0:10:41 load avg: 8.39 [28/44] test_math
 0:11:39 load avg: 7.59 [29/44] test_memoryview -- test_math passed in 58.1 sec
0:11:43 load avg: 7.46 [30/44] test_operator
0:11:44 load avg: 7.46 [31/44] test_ordered_dict
0:11:55 load avg: 7.64 [32/44] test_patma
0:11:58 load avg: 7.99 [33/44] test_pickle
0:14:09 load avg: 6.94 [34/44] test_pprint -- test_pickle passed in 2 min 10 sec
0:14:13 load avg: 7.19 [35/44] test_re
 0:14:33 load avg: 7.41 [36/44] test_set
0:16:04 load avg: 6.40 [37/44] test_sqlite3 -- test_set passed in 1 min 31 sec
0:16:09 load avg: 6.21 [38/44] test_statistics
 0:19:00 load avg: 7.92 [39/44] test_str -- test_statistics passed in 2 min 51 sec
0:19:26 load avg: 6.99 [40/44] test_struct
0:19:34 load avg: 6.74 [41/44] test_tabnanny
0:19:38 load avg: 6.74 [42/44] test_time
0:19:41 load avg: 6.28 [43/44] test_xml_etree
0:19:48 load avg: 6.33 [44/44] test_xml_etree_c

Total duration: 19 min 58 sec
Total tests: run=9,203 skipped=191
Total test files: run=44/44 env_changed=1
Result: SUCCESS

Environment

➜  ~ uname -a
Linux akarsu 6.8.1-arch1-1 #1 SMP PREEMPT_DYNAMIC Sat, 16 Mar 2024 17:15:35 +0000 x86_64 GNU/Linux
➜  ~ cc --version        
cc (GCC) 13.2.1 20230801
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
➜  ~ 

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

[mdboom]

We regularly do PGO builds on the Faster CPython team and they currently work for us. Maybe there is some unexpected interaction with one of the many other flags in the ./configure line here. Have you tried turning each of them off individually to see which one it might be?

[furkanonder]

@mdboom Reduced configuration flags that create the same issue.

./configure --enable-optimizations --with-pydebug --disable-gil

[mdboom]

Because the experimental free threading is involved, I've add the topic-free-threading label and am summoning @colesbury.

[colesbury]

Thanks @mdboom

The mimalloc warning is expected, I think. See also #114331. There's a test in test_decimal that tries to allocate a huge amount of memory and mimalloc prints warnings in debug builds when allocations fail. We could disable the warning (or skip the test), but I'm not sure if we should.

I am confused about the test_httpservers failure. For some reason there is a gmon.out file in the temporary directories preventing their deletion.

[colesbury]

The test_httpservers is not related to --disable-gil. It also happens with:

./configure --enable-optimizations --with-valgrind --enable-profiling --enable-pystats --with-pydebug --with-trace-refs --with-assertions --with-lto

Beyond that, I'm not sure which combination of build flags leads to the problem.

[furkanonder]

The test_httpservers is not related to --disable-gil. It also happens with:

./configure --enable-optimizations --with-valgrind --enable-profiling --enable-pystats --with-pydebug --with-trace-refs --with-assertions --with-lto

Beyond that, I'm not sure which combination of build flags leads to the problem.

@colesbury Thank you very much for your time. I have created another issue with test_httpservers.