Setting negative tp_dictoffset on PyLong subclass leads to crash

[encukou]

Bug report

Bug description:

In Python 3.12, it is no longer possible to subclass int by extending basicsize and setting a negative tp_dictoffset.

The 3.11 docs suggest that this might work:

A negative offset is more expensive to use, and should only be used when the instance structure contains a variable-length part. This is used for example to add an instance variable dictionary to subtypes of str or tuple.

That is, while int is not given as an explicit example, but there's nothing to imply it would be different than str or tuple.

Indeed, it works on Python 3.11, but fails in 3.12 when accessing the __dict__, as Python calls _PyObject_ComputedDictPointer, which tries to compute the dict offset using abs(ob_size), which is invalid for int on 3.12.

I'm not the best person to judge whether this is a bad bug, acceptable backwards-incompatible change, or perhaps it's a misuse of 3.11 API (but I can't see exactly how it's misused).

The 3.12 What's new does mention tp_dictoffset-related breaking changes:

The use of tp_dictoffset and tp_weaklistoffset is still supported, but does not fully support multiple inheritance (gh-95589), and performance may be worse.

But multiple inheritance is not in play here.

@markshannon, any thoughts?

---

Reproducer (hope I didn't simplify too much):

extmodule.c:

#include "Python.h"
#include "structmember.h"   // for PyMemberDef API in 3.11 & below

static PyTypeObject *basetype = &PyLong_Type;

static int
mod_exec(PyObject *m)
{
    PyObject *type = PyType_FromModuleAndSpec(
        m,
        &(PyType_Spec) {
            .name = "ext.MySubclass",
            .basicsize = basetype->tp_basicsize + sizeof(PyDictObject*),
            .flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,
            .slots = (PyType_Slot[]) {
                {Py_tp_members, (PyMemberDef[]) {
                    {
                        "__dictoffset__", T_PYSSIZET,
                        -sizeof(PyDictObject*), READONLY},
                    {NULL},
                }},
                {0, 0}, 
            }},
        (PyObject*)basetype);
    if (!type) return -1;
    if (PyModule_AddType(m, (PyTypeObject*)type) == -1) return -1;
    Py_DECREF(type);

    return 0;
}

static struct PyModuleDef mod_def = {
    PyModuleDef_HEAD_INIT,
    .m_name = "ext",
    .m_slots = (PyModuleDef_Slot[]) {
        {Py_mod_exec, mod_exec},
        {0, NULL}
    },
};

PyMODINIT_FUNC
PyInit_ext(void)
{
    return PyModuleDef_Init(&mod_def);
}

repro.py:

import ext
print(ext)

obj = ext.MySubclass(1)
obj.attr = 'foo'
assert obj.attr == 'foo'

Compiling is tricky with distutils gone, but on Fedora a command to compile & run is:
gcc $(python3.12-config --cflags --ldflags) -fPIC --shared extmodule.c -o ext.so && python3.12d -Xdev repro.py

Crashes on 3.12, works on 3.11 or with basetype = &PyUnicode_Type.

CPython versions tested on:

3.11, 3.12

Operating systems tested on:

Linux