Align weak-reference collection with Python 3.15 finalization (#9168)

* Align weak-reference collection with Python 3.15 finalization

Trace and retain owned callback edges independently of weak referent links.
Clear callback-bearing weakrefs before finalizers, suppress callbacks from
unreachable weakrefs, and close remaining weakref ingress after resurrection
checks while the world is stopped. Publish and unlink intrusive nodes safely.

Refresh weakref and canonical weakref/GC/finalization tests through update_lib
from pinned CPython 3.15.0. Retire four obsolete guards only after fresh body
passes, retaining applicable RustPython limitations.

Assisted-by: OpenAI Codex:model version unavailable

* Refresh canonical IO tests for weakref finalization

Migrate the complete CPython 3.15.0 test_io package at d220409 with the
fail-closed legacy-map updater, preserving all ten existing guards. Retire
the seven consolidated standalone suites and update io/file dependency
paths with mapping regression tests.

The canonical GC tests ignore ResourceWarning without retaining its
source, so callback-free weakrefs correctly reflect surviving owners.
This closes the stale test dependency exposed by the lifecycle change.

Assisted-by: OpenAI Codex:model version unavailable

* Protect weakref callbacks with the original referent stripe

Follow CPython's free-threading weakref locking model instead of keeping
an independent callback RwLock. Retain the original referent stripe key
so callback access stays synchronized after GC clears the referent.
Unlink and steal callbacks under that stripe, then release ownership
outside the lock. Preserve synchronization for running-world GC paths
and gc.get_referents while keeping ordinary weakref upgrades unchanged.

Assisted-by: OpenAI Codex:model version unavailable

* Defer weakref GC tracking until referent publication

Keep freshly allocated weakrefs out of GC's object lists until they are
linked and their referent is published. Release the referent stripe before
tracking, since tracking can collect and invoke finalizers. Generic reuse
losers remain untracked.

Exercise the allocator boundary directly for reference, proxy, callable
proxy and subclass payloads, and ensure completed constructors track their
objects. This prevents observers from retaining a half-built dead weakref
that later becomes live.

Assisted-by: OpenAI Codex:model version unavailable

* Retain the original weakref stripe lock like CPython

Store a safe handle to the selected static weakref stripe instead of
retaining and rehashing the referent address. Keep the same lock after
referent clearing, matching CPython free-threaded weakrefs_lock. The
nonthreaded handle is zero-sized and introduces no locking.

Preserve lock selection, acquire/release ordering, fork reset, deferred
GC publication, and callback-owner release outside the stripe. No new
locks or unsafe code are needed.

Assisted-by: OpenAI Codex:model version unavailable

Lib/test/test_bufio.py

@@ -1,73 +0,0 @@
-import unittest
-from test.support import os_helper
-
-import io # C implementation.
-import _pyio as pyio # Python implementation.
-
-# Simple test to ensure that optimizations in the IO library deliver the
-# expected results.  For best testing, run this under a debug-build Python too
-# (to exercise asserts in the C code).
-
-lengths = list(range(1, 257)) + [512, 1000, 1024, 2048, 4096, 8192, 10000,
-                                 16384, 32768, 65536, 1000000]
-
-class BufferSizeTest:
-    def try_one(self, s):
-        # Write s + "\n" + s to file, then open it and ensure that successive
-        # .readline()s deliver what we wrote.
-
-        # Ensure we can open TESTFN for writing.
-        os_helper.unlink(os_helper.TESTFN)
-
-        # Since C doesn't guarantee we can write/read arbitrary bytes in text
-        # files, use binary mode.
-        f = self.open(os_helper.TESTFN, "wb")
-        try:
-            # write once with \n and once without
-            f.write(s)
-            f.write(b"\n")
-            f.write(s)
-            f.close()
-            f = self.open(os_helper.TESTFN, "rb")
-            line = f.readline()
-            self.assertEqual(line, s + b"\n")
-            line = f.readline()
-            self.assertEqual(line, s)
-            line = f.readline()
-            self.assertFalse(line) # Must be at EOF
-            f.close()
-        finally:
-            os_helper.unlink(os_helper.TESTFN)
-
-    def drive_one(self, pattern):
-        for length in lengths:
-            # Repeat string 'pattern' as often as needed to reach total length
-            # 'length'.  Then call try_one with that string, a string one larger
-            # than that, and a string one smaller than that.  Try this with all
-            # small sizes and various powers of 2, so we exercise all likely
-            # stdio buffer sizes, and "off by one" errors on both sides.
-            q, r = divmod(length, len(pattern))
-            teststring = pattern * q + pattern[:r]
-            self.assertEqual(len(teststring), length)
-            self.try_one(teststring)
-            self.try_one(teststring + b"x")
-            self.try_one(teststring[:-1])
-
-    def test_primepat(self):
-        # A pattern with prime length, to avoid simple relationships with
-        # stdio buffer sizes.
-        self.drive_one(b"1234567890\00\01\02\03\04\05\06")
-
-    def test_nullpat(self):
-        self.drive_one(b'\0' * 1000)
-
-
-class CBufferSizeTest(BufferSizeTest, unittest.TestCase):
-    open = io.open
-
-class PyBufferSizeTest(BufferSizeTest, unittest.TestCase):
-    open = staticmethod(pyio.open)
-
-
-if __name__ == "__main__":
-    unittest.main()

Lib/test/test_finalization.py

@@ -174,7 +174,7 @@ class SimpleFinalizationTest(TestBase, unittest.TestCase):
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_survivors([])
-            self.assertIs(wr(), None)
+            self.assertIsNone(wr())
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_survivors([])
@@ -188,12 +188,12 @@ class SimpleFinalizationTest(TestBase, unittest.TestCase):
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_survivors(ids)
-            self.assertIsNot(wr(), None)
+            self.assertIsNotNone(wr())
             self.clear_survivors()
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_survivors([])
-        self.assertIs(wr(), None)
+        self.assertIsNone(wr())
 
     @support.cpython_only
     def test_non_gc(self):
@@ -265,7 +265,7 @@ class SelfCycleFinalizationTest(TestBase, unittest.TestCase):
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_survivors([])
-            self.assertIs(wr(), None)
+            self.assertIsNone(wr())
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_survivors([])
@@ -276,19 +276,24 @@ class SelfCycleFinalizationTest(TestBase, unittest.TestCase):
             s = SelfCycleResurrector()
             ids = [id(s)]
             wr = weakref.ref(s)
+            wrc = weakref.ref(s, lambda x: None)
             del s
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_survivors(ids)
-            # XXX is this desirable?
-            self.assertIs(wr(), None)
+            # This used to be None because weakrefs were cleared before
+            # calling finalizers.  Now they are cleared after.
+            self.assertIsNotNone(wr())
+            # A weakref with a callback is still cleared before calling
+            # finalizers.
+            self.assertIsNone(wrc())
             # When trying to destroy the object a second time, __del__
             # isn't called anymore (and the object isn't resurrected).
             self.clear_survivors()
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_survivors([])
-            self.assertIs(wr(), None)
+            self.assertIsNone(wr())
 
     def test_simple_suicide(self):
         # Test the GC is able to deal with an object that kills its last
@@ -301,11 +306,11 @@ class SelfCycleFinalizationTest(TestBase, unittest.TestCase):
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_survivors([])
-            self.assertIs(wr(), None)
+            self.assertIsNone(wr())
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_survivors([])
-            self.assertIs(wr(), None)
+            self.assertIsNone(wr())
 
 
 class ChainedBase:
@@ -378,18 +383,27 @@ class CycleChainFinalizationTest(TestBase, unittest.TestCase):
 
     def check_resurrecting_chain(self, classes):
         N = len(classes)
+        def dummy_callback(ref):
+            pass
         with SimpleBase.test():
             nodes = self.build_chain(classes)
             N = len(nodes)
             ids = [id(s) for s in nodes]
             survivor_ids = [id(s) for s in nodes if isinstance(s, SimpleResurrector)]
             wrs = [weakref.ref(s) for s in nodes]
+            wrcs = [weakref.ref(s, dummy_callback) for s in nodes]
             del nodes
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_survivors(survivor_ids)
-            # XXX desirable?
-            self.assertEqual([wr() for wr in wrs], [None] * N)
+            for wr in wrs:
+                # These values used to be None because weakrefs were cleared
+                # before calling finalizers.  Now they are cleared after.
+                self.assertIsNotNone(wr())
+            for wr in wrcs:
+                # Weakrefs with callbacks are still cleared before calling
+                # finalizers.
+                self.assertIsNone(wr())
             self.clear_survivors()
             gc.collect()
             self.assert_del_calls(ids)
@@ -491,7 +505,7 @@ class LegacyFinalizationTest(TestBase, unittest.TestCase):
             self.assert_del_calls(ids)
             self.assert_tp_del_calls(ids)
             self.assert_survivors([])
-            self.assertIs(wr(), None)
+            self.assertIsNone(wr())
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_tp_del_calls(ids)
@@ -507,13 +521,13 @@ class LegacyFinalizationTest(TestBase, unittest.TestCase):
             self.assert_tp_del_calls(ids)
             self.assert_survivors(ids)
             # weakrefs are cleared before tp_del is called.
-            self.assertIs(wr(), None)
+            self.assertIsNone(wr())
             self.clear_survivors()
             gc.collect()
             self.assert_del_calls(ids)
             self.assert_tp_del_calls(ids * 2)
             self.assert_survivors(ids)
-        self.assertIs(wr(), None)
+        self.assertIsNone(wr())
 
     def test_legacy_self_cycle(self):
         # Self-cycles with legacy finalizers end up in gc.garbage.
@@ -527,11 +541,11 @@ class LegacyFinalizationTest(TestBase, unittest.TestCase):
             self.assert_tp_del_calls([])
             self.assert_survivors([])
             self.assert_garbage(ids)
-            self.assertIsNot(wr(), None)
+            self.assertIsNotNone(wr())
             # Break the cycle to allow collection
             gc.garbage[0].ref = None
         self.assert_garbage([])
-        self.assertIs(wr(), None)
+        self.assertIsNone(wr())
 
 
 if __name__ == "__main__":

Lib/test/test_gc.py

@@ -236,7 +236,6 @@ class GCTests(unittest.TestCase):
         # is 3 because it includes f's code object.
         self.assertIn(gc.collect(), (2, 3))
 
-    @unittest.expectedFailure  # TODO: RUSTPYTHON; - weakref clear ordering differs from 3.15+
     def test_function_tp_clear_leaves_consistent_state(self):
         # https://github.com/python/cpython/issues/91636
         code = """if 1:
@@ -310,6 +309,26 @@ class GCTests(unittest.TestCase):
         self.assertRegex(stdout, rb"""\A\s*func=None""")
         self.assertFalse(stderr)
 
+    def test_datetime_weakref_cycle(self):
+        # https://github.com/python/cpython/issues/132413
+        # If the weakref used by the datetime extension gets cleared by the GC (due to being
+        # in an unreachable cycle) then datetime functions would crash (get_module_state()
+        # was returning a NULL pointer).  This bug is fixed by clearing weakrefs without
+        # callbacks *after* running finalizers.
+        code = """if 1:
+        import _datetime
+        class C:
+            def __del__(self):
+                print('__del__ called')
+                _datetime.timedelta(days=1)  # crash?
+
+        l = [C()]
+        l.append(l)
+        """
+        rc, stdout, stderr = assert_python_ok("-c", code)
+        self.assertEqual(rc, 0)
+        self.assertEqual(stdout.strip(), b'__del__ called')
+
     @refcount_test
     def test_frame(self):
         def f():
@@ -667,9 +686,8 @@ class GCTests(unittest.TestCase):
         gc.collect()
         self.assertEqual(len(ouch), 2)  # else the callbacks didn't run
         for x in ouch:
-            # If the callback resurrected one of these guys, the instance
-            # would be damaged, with an empty __dict__.
-            self.assertEqual(x, None)
+            # The weakref should be cleared before executing the callback.
+            self.assertIsNone(x)
 
     def test_bug21435(self):
         # This is a poor test - its only virtue is that it happened to
@@ -831,17 +849,20 @@ class GCTests(unittest.TestCase):
         rc, out, err = assert_python_ok(TESTFN)
         self.assertEqual(out.strip(), b'__del__ called')
 
-    @unittest.expectedFailure  # TODO: RUSTPYTHON
     def test_get_stats(self):
         stats = gc.get_stats()
         self.assertEqual(len(stats), 3)
         for st in stats:
             self.assertIsInstance(st, dict)
-            self.assertEqual(set(st),
-                             {"collected", "collections", "uncollectable"})
+            self.assertEqual(
+                set(st),
+                {"collected", "collections", "uncollectable", "candidates", "duration"}
+            )
             self.assertGreaterEqual(st["collected"], 0)
             self.assertGreaterEqual(st["collections"], 0)
             self.assertGreaterEqual(st["uncollectable"], 0)
+            self.assertGreaterEqual(st["candidates"], 0)
+            self.assertGreaterEqual(st["duration"], 0)
         # Check that collection counts are incremented correctly
         if gc.isenabled():
             self.addCleanup(gc.enable)
@@ -852,11 +873,25 @@ class GCTests(unittest.TestCase):
         self.assertEqual(new[0]["collections"], old[0]["collections"] + 1)
         self.assertEqual(new[1]["collections"], old[1]["collections"])
         self.assertEqual(new[2]["collections"], old[2]["collections"])
+        self.assertGreater(new[0]["duration"], old[0]["duration"])
+        self.assertEqual(new[1]["duration"], old[1]["duration"])
+        self.assertEqual(new[2]["duration"], old[2]["duration"])
+        for stat in ["collected", "uncollectable", "candidates"]:
+            self.assertGreaterEqual(new[0][stat], old[0][stat])
+            self.assertEqual(new[1][stat], old[1][stat])
+            self.assertEqual(new[2][stat], old[2][stat])
         gc.collect(2)
-        new = gc.get_stats()
-        self.assertEqual(new[0]["collections"], old[0]["collections"] + 1)
+        old, new = new, gc.get_stats()
+        self.assertEqual(new[0]["collections"], old[0]["collections"])
         self.assertEqual(new[1]["collections"], old[1]["collections"])
         self.assertEqual(new[2]["collections"], old[2]["collections"] + 1)
+        self.assertEqual(new[0]["duration"], old[0]["duration"])
+        self.assertEqual(new[1]["duration"], old[1]["duration"])
+        self.assertGreater(new[2]["duration"], old[2]["duration"])
+        for stat in ["collected", "uncollectable", "candidates"]:
+            self.assertEqual(new[0][stat], old[0][stat])
+            self.assertEqual(new[1][stat], old[1][stat])
+            self.assertGreaterEqual(new[2][stat], old[2][stat])
 
     def test_freeze(self):
         gc.freeze()
@@ -1204,6 +1239,37 @@ class GCTests(unittest.TestCase):
         """)
         assert_python_ok("-c", source)
 
+    def test_do_not_cleanup_type_subclasses_before_finalization(self):
+        #  See https://github.com/python/cpython/issues/135552
+        # If we cleanup weakrefs for tp_subclasses before calling
+        # the finalizer (__del__) then the line `fail = BaseNode.next.next`
+        # should fail because we are trying to access a subclass
+        # attribute. But subclass type cache was not properly invalidated.
+        code = """
+            class BaseNode:
+                def __del__(self):
+                    BaseNode.next = BaseNode.next.next
+                    fail = BaseNode.next.next
+
+            class Node(BaseNode):
+                pass
+
+            BaseNode.next = Node()
+            BaseNode.next.next = Node()
+        """
+        # this test checks garbage collection while interp
+        # finalization
+        assert_python_ok("-c", textwrap.dedent(code))
+
+        code_inside_function = textwrap.dedent(F"""
+            def test():
+                {textwrap.indent(code, '    ')}
+
+            test()
+        """)
+        # this test checks regular garbage collection
+        assert_python_ok("-c", code_inside_function)
+
 
     @unittest.expectedFailure  # TODO: RUSTPYTHON; long_lived_total does not count untracked tuples
     @unittest.skipUnless(Py_GIL_DISABLED, "requires free-threaded GC")
@@ -1308,9 +1374,11 @@ class GCCallbackTests(unittest.TestCase):
         # Check that we got the right info dict for all callbacks
         for v in self.visit:
             info = v[2]
-            self.assertTrue("generation" in info)
-            self.assertTrue("collected" in info)
-            self.assertTrue("uncollectable" in info)
+            self.assertIn("generation", info)
+            self.assertIn("collected", info)
+            self.assertIn("uncollectable", info)
+            self.assertIn("candidates", info)
+            self.assertIn("duration", info)
 
     def test_collect_generation(self):
         self.preclean()
@@ -1498,6 +1566,7 @@ class GCTogglingTests(unittest.TestCase):
             self.assertEqual(x, None)
 
     @gc_threshold(1000, 0, 0)
+    @unittest.skipIf(Py_GIL_DISABLED, "requires GC generations or increments")
     def test_bug1055820d(self):
         # Corresponds to temp2d.py in the bug report.  This is very much like
         # test_bug1055820c, but uses a __del__ method instead of a weakref
@@ -1595,6 +1664,25 @@ class GCTogglingTests(unittest.TestCase):
         finally:
             gc.enable()
 
+    def test_get_count_nonnegative(self):
+        xs = [[] for _ in range(1500)]
+        gc.collect()
+        del xs[:1000]
+        self.assertGreaterEqual(gc.get_count()[0], 0)
+
+    @gc_threshold(1000, 0, 0)
+    def test_get_count_does_not_prevent_collection(self):
+        junk = []
+        gc.collect()
+        detector = GC_Detector()
+        for _ in range(10000):
+            junk.append([])
+            gc.get_count()
+            if detector.gc_happened:
+                break
+        else:
+            self.fail("gc didn't happen after 10000 iterations")
+
     # Ensure that setting *threshold0* to zero disables collection.
     @gc_threshold(0)
     def test_threshold_zero(self):

Lib/test/test_io/__init__.py

@@ -0,0 +1,28 @@
+"""Tests for the io module and its implementations (_io and _pyio)
+
+Tests are split across multiple files to increase
+parallelism and focus on specific implementation pieces.
+
+* test_io
+  * test_bufferedio - tests file buffering
+  * test_memoryio - tests BytesIO and StringIO
+  * test_fileio - tests FileIO
+  * test_file - tests the file interface
+  * test_general - tests everything else in the io module
+  * test_univnewlines - tests universal newline support
+  * test_largefile - tests operations on a file greater than 2**32 bytes
+    (only enabled with -ulargefile)
+* test_free_threading/test_io - tests thread safety of io objects
+
+.. attention::
+   When writing tests for io, it's important to test both the C and Python
+   implementations. This is usually done by writing a base test that refers to
+   the type it is testing as an attribute. Then it provides custom subclasses to
+   test both implementations. This directory contains lots of examples.
+"""
+
+import os
+from test.support import load_package_tests
+
+def load_tests(*args):
+    return load_package_tests(os.path.dirname(__file__), *args)

Lib/test/test_io/__main__.py

@@ -0,0 +1,4 @@
+from . import load_tests
+import unittest
+
+unittest.main()

Lib/test/test_io/test_largefile.py

@@ -56,9 +56,7 @@ class TestFileMethods(LargeFileTest):
     (i.e. > 2 GiB) files.
     """
 
-    # _pyio.FileIO.readall() uses a temporary bytearray then casted to bytes,
-    # so memuse=2 is needed
-    @bigmemtest(size=size, memuse=2, dry_run=False)
+    @bigmemtest(size=size, memuse=1, dry_run=False)
     def test_large_read(self, _size):
         # bpo-24658: Test that a read greater than 2GB does not fail.
         with self.open(TESTFN, "rb") as f:
@@ -154,7 +152,7 @@ class TestFileMethods(LargeFileTest):
                 f.seek(pos)
                 self.assertTrue(f.seekable())
 
-    @bigmemtest(size=size, memuse=2, dry_run=False)
+    @bigmemtest(size=size, memuse=1, dry_run=False)
     def test_seek_readall(self, _size):
         # Seek which doesn't change position should readall successfully.
         with self.open(TESTFN, 'rb') as f:

Lib/test/test_io/test_signals.py

@@ -0,0 +1,280 @@
+import errno
+import os
+import signal
+import threading
+import unittest
+from test import support
+from test.support import import_helper
+from .utils import PyTestCase, CTestCase
+
+
+requires_alarm = unittest.skipUnless(
+    hasattr(signal, "alarm"), "test requires signal.alarm()"
+)
+
+
+@unittest.skipIf(os.name == 'nt', 'POSIX signals required for this test.')
+class SignalsTest:
+
+    def setUp(self):
+        self.oldalrm = signal.signal(signal.SIGALRM, self.alarm_interrupt)
+
+    def tearDown(self):
+        signal.signal(signal.SIGALRM, self.oldalrm)
+
+    def alarm_interrupt(self, sig, frame):
+        1/0
+
+    def check_interrupted_write(self, item, bytes, **fdopen_kwargs):
+        """Check that a partial write, when it gets interrupted, properly
+        invokes the signal handler, and bubbles up the exception raised
+        in the latter."""
+
+        # XXX This test has three flaws that appear when objects are
+        # XXX not reference counted.
+
+        # - if wio.write() happens to trigger a garbage collection,
+        #   the signal exception may be raised when some __del__
+        #   method is running; it will not reach the assertRaises()
+        #   call.
+
+        # - more subtle, if the wio object is not destroyed at once
+        #   and survives this function, the next opened file is likely
+        #   to have the same fileno (since the file descriptor was
+        #   actively closed).  When wio.__del__ is finally called, it
+        #   will close the other's test file...  To trigger this with
+        #   CPython, try adding "global wio" in this function.
+
+        # - This happens only for streams created by the _pyio module,
+        #   because a wio.close() that fails still consider that the
+        #   file needs to be closed again.  You can try adding an
+        #   "assert wio.closed" at the end of the function.
+
+        # Fortunately, a little gc.collect() seems to be enough to
+        # work around all these issues.
+        support.gc_collect()  # For PyPy or other GCs.
+
+        read_results = []
+        def _read():
+            s = os.read(r, 1)
+            read_results.append(s)
+
+        t = threading.Thread(target=_read)
+        t.daemon = True
+        r, w = os.pipe()
+        fdopen_kwargs["closefd"] = False
+        large_data = item * (support.PIPE_MAX_SIZE // len(item) + 1)
+        try:
+            wio = self.io.open(w, **fdopen_kwargs)
+            if hasattr(signal, 'pthread_sigmask'):
+                # create the thread with SIGALRM signal blocked
+                signal.pthread_sigmask(signal.SIG_BLOCK, [signal.SIGALRM])
+                t.start()
+                signal.pthread_sigmask(signal.SIG_UNBLOCK, [signal.SIGALRM])
+            else:
+                t.start()
+
+            # Fill the pipe enough that the write will be blocking.
+            # It will be interrupted by the timer armed above.  Since the
+            # other thread has read one byte, the low-level write will
+            # return with a successful (partial) result rather than an EINTR.
+            # The buffered IO layer must check for pending signal
+            # handlers, which in this case will invoke alarm_interrupt().
+            signal.alarm(1)
+            try:
+                self.assertRaises(ZeroDivisionError, wio.write, large_data)
+            finally:
+                signal.alarm(0)
+                t.join()
+            # We got one byte, get another one and check that it isn't a
+            # repeat of the first one.
+            read_results.append(os.read(r, 1))
+            self.assertEqual(read_results, [bytes[0:1], bytes[1:2]])
+        finally:
+            os.close(w)
+            os.close(r)
+            # This is deliberate. If we didn't close the file descriptor
+            # before closing wio, wio would try to flush its internal
+            # buffer, and block again.
+            try:
+                wio.close()
+            except OSError as e:
+                if e.errno != errno.EBADF:
+                    raise
+
+    @requires_alarm
+    @unittest.skipUnless(hasattr(os, "pipe"), "requires os.pipe()")
+    def test_interrupted_write_unbuffered(self):
+        self.check_interrupted_write(b"xy", b"xy", mode="wb", buffering=0)
+
+    @requires_alarm
+    @unittest.skipUnless(hasattr(os, "pipe"), "requires os.pipe()")
+    def test_interrupted_write_buffered(self):
+        self.check_interrupted_write(b"xy", b"xy", mode="wb")
+
+    @requires_alarm
+    @unittest.skipUnless(hasattr(os, "pipe"), "requires os.pipe()")
+    def test_interrupted_write_text(self):
+        self.check_interrupted_write("xy", b"xy", mode="w", encoding="ascii")
+
+    @support.no_tracing
+    def check_reentrant_write(self, data, **fdopen_kwargs):
+        def on_alarm(*args):
+            # Will be called reentrantly from the same thread
+            wio.write(data)
+            1/0
+        signal.signal(signal.SIGALRM, on_alarm)
+        r, w = os.pipe()
+        wio = self.io.open(w, **fdopen_kwargs)
+        try:
+            signal.alarm(1)
+            # Either the reentrant call to wio.write() fails with RuntimeError,
+            # or the signal handler raises ZeroDivisionError.
+            with self.assertRaises((ZeroDivisionError, RuntimeError)) as cm:
+                while 1:
+                    for i in range(100):
+                        wio.write(data)
+                        wio.flush()
+                    # Make sure the buffer doesn't fill up and block further writes
+                    os.read(r, len(data) * 100)
+            exc = cm.exception
+            if isinstance(exc, RuntimeError):
+                self.assertStartsWith(str(exc), "reentrant call")
+        finally:
+            signal.alarm(0)
+            wio.close()
+            os.close(r)
+
+    @requires_alarm
+    def test_reentrant_write_buffered(self):
+        self.check_reentrant_write(b"xy", mode="wb")
+
+    @requires_alarm
+    def test_reentrant_write_text(self):
+        self.check_reentrant_write("xy", mode="w", encoding="ascii")
+
+    def check_interrupted_read_retry(self, decode, **fdopen_kwargs):
+        """Check that a buffered read, when it gets interrupted (either
+        returning a partial result or EINTR), properly invokes the signal
+        handler and retries if the latter returned successfully."""
+        r, w = os.pipe()
+        fdopen_kwargs["closefd"] = False
+        def alarm_handler(sig, frame):
+            os.write(w, b"bar")
+        signal.signal(signal.SIGALRM, alarm_handler)
+        try:
+            rio = self.io.open(r, **fdopen_kwargs)
+            os.write(w, b"foo")
+            signal.alarm(1)
+            # Expected behaviour:
+            # - first raw read() returns partial b"foo"
+            # - second raw read() returns EINTR
+            # - third raw read() returns b"bar"
+            self.assertEqual(decode(rio.read(6)), "foobar")
+        finally:
+            signal.alarm(0)
+            rio.close()
+            os.close(w)
+            os.close(r)
+
+    @requires_alarm
+    @support.requires_resource('walltime')
+    def test_interrupted_read_retry_buffered(self):
+        self.check_interrupted_read_retry(lambda x: x.decode('latin1'),
+                                          mode="rb")
+
+    @requires_alarm
+    @support.requires_resource('walltime')
+    def test_interrupted_read_retry_text(self):
+        self.check_interrupted_read_retry(lambda x: x,
+                                          mode="r", encoding="latin1")
+
+    def check_interrupted_write_retry(self, item, **fdopen_kwargs):
+        """Check that a buffered write, when it gets interrupted (either
+        returning a partial result or EINTR), properly invokes the signal
+        handler and retries if the latter returned successfully."""
+        select = import_helper.import_module("select")
+
+        # A quantity that exceeds the buffer size of an anonymous pipe's
+        # write end.
+        N = support.PIPE_MAX_SIZE
+        r, w = os.pipe()
+        fdopen_kwargs["closefd"] = False
+
+        # We need a separate thread to read from the pipe and allow the
+        # write() to finish.  This thread is started after the SIGALRM is
+        # received (forcing a first EINTR in write()).
+        read_results = []
+        write_finished = False
+        error = None
+        def _read():
+            try:
+                while not write_finished:
+                    while r in select.select([r], [], [], 1.0)[0]:
+                        s = os.read(r, 1024)
+                        read_results.append(s)
+            except BaseException as exc:
+                nonlocal error
+                error = exc
+        t = threading.Thread(target=_read)
+        t.daemon = True
+        def alarm1(sig, frame):
+            signal.signal(signal.SIGALRM, alarm2)
+            signal.alarm(1)
+        def alarm2(sig, frame):
+            t.start()
+
+        large_data = item * N
+        signal.signal(signal.SIGALRM, alarm1)
+        try:
+            wio = self.io.open(w, **fdopen_kwargs)
+            signal.alarm(1)
+            # Expected behaviour:
+            # - first raw write() is partial (because of the limited pipe buffer
+            #   and the first alarm)
+            # - second raw write() returns EINTR (because of the second alarm)
+            # - subsequent write()s are successful (either partial or complete)
+            written = wio.write(large_data)
+            self.assertEqual(N, written)
+
+            wio.flush()
+            write_finished = True
+            t.join()
+
+            self.assertIsNone(error)
+            self.assertEqual(N, sum(len(x) for x in read_results))
+        finally:
+            signal.alarm(0)
+            write_finished = True
+            os.close(w)
+            os.close(r)
+            # This is deliberate. If we didn't close the file descriptor
+            # before closing wio, wio would try to flush its internal
+            # buffer, and could block (in case of failure).
+            try:
+                wio.close()
+            except OSError as e:
+                if e.errno != errno.EBADF:
+                    raise
+
+    @requires_alarm
+    @support.requires_resource('walltime')
+    def test_interrupted_write_retry_buffered(self):
+        self.check_interrupted_write_retry(b"x", mode="wb")
+
+    @requires_alarm
+    @support.requires_resource('walltime')
+    def test_interrupted_write_retry_text(self):
+        self.check_interrupted_write_retry("x", mode="w", encoding="latin1")
+
+
+class CSignalsTest(SignalsTest, CTestCase):
+    pass
+
+class PySignalsTest(SignalsTest, PyTestCase):
+    pass
+
+    # Handling reentrancy issues would slow down _pyio even more, so the
+    # tests are disabled.
+    test_reentrant_write_buffered = None
+    test_reentrant_write_text = None

Lib/test/test_io/utils.py

@@ -0,0 +1,318 @@
+import array
+import threading
+import time
+import unittest
+
+import io  # C implementation of io
+import _pyio as pyio # Python implementation of io
+
+
+try:
+    import ctypes
+except ImportError:
+    def byteslike(*pos, **kw):
+        return array.array("b", bytes(*pos, **kw))
+else:
+    class EmptyStruct(ctypes.Structure):
+        pass
+
+    def byteslike(*pos, **kw):
+        """Create a bytes-like object having no string or sequence methods"""
+        data = bytes(*pos, **kw)
+        obj = EmptyStruct()
+        ctypes.resize(obj, len(data))
+        memoryview(obj).cast("B")[:] = data
+        return obj
+
+
+class MockRawIOWithoutRead:
+    """A RawIO implementation without read(), so as to exercise the default
+    RawIO.read() which calls readinto()."""
+
+    def __init__(self, read_stack=()):
+        self._read_stack = list(read_stack)
+        self._write_stack = []
+        self._reads = 0
+        self._extraneous_reads = 0
+
+    def write(self, b):
+        self._write_stack.append(bytes(b))
+        return len(b)
+
+    def writable(self):
+        return True
+
+    def fileno(self):
+        return 42
+
+    def readable(self):
+        return True
+
+    def seekable(self):
+        return True
+
+    def seek(self, pos, whence):
+        return 0   # wrong but we gotta return something
+
+    def tell(self):
+        return 0   # same comment as above
+
+    def readinto(self, buf):
+        self._reads += 1
+        max_len = len(buf)
+        try:
+            data = self._read_stack[0]
+        except IndexError:
+            self._extraneous_reads += 1
+            return 0
+        if data is None:
+            del self._read_stack[0]
+            return None
+        n = len(data)
+        if len(data) <= max_len:
+            del self._read_stack[0]
+            buf[:n] = data
+            return n
+        else:
+            buf[:] = data[:max_len]
+            self._read_stack[0] = data[max_len:]
+            return max_len
+
+    def truncate(self, pos=None):
+        return pos
+
+class CMockRawIOWithoutRead(MockRawIOWithoutRead, io.RawIOBase):
+    pass
+
+class PyMockRawIOWithoutRead(MockRawIOWithoutRead, pyio.RawIOBase):
+    pass
+
+
+class MockRawIO(MockRawIOWithoutRead):
+
+    def read(self, n=None):
+        self._reads += 1
+        try:
+            return self._read_stack.pop(0)
+        except:
+            self._extraneous_reads += 1
+            return b""
+
+class CMockRawIO(MockRawIO, io.RawIOBase):
+    pass
+
+class PyMockRawIO(MockRawIO, pyio.RawIOBase):
+    pass
+
+
+class MisbehavedRawIO(MockRawIO):
+    def write(self, b):
+        return super().write(b) * 2
+
+    def read(self, n=None):
+        return super().read(n) * 2
+
+    def seek(self, pos, whence):
+        return -123
+
+    def tell(self):
+        return -456
+
+    def readinto(self, buf):
+        super().readinto(buf)
+        return len(buf) * 5
+
+class CMisbehavedRawIO(MisbehavedRawIO, io.RawIOBase):
+    pass
+
+class PyMisbehavedRawIO(MisbehavedRawIO, pyio.RawIOBase):
+    pass
+
+
+class SlowFlushRawIO(MockRawIO):
+    def __init__(self):
+        super().__init__()
+        self.in_flush = threading.Event()
+
+    def flush(self):
+        self.in_flush.set()
+        time.sleep(0.25)
+
+class CSlowFlushRawIO(SlowFlushRawIO, io.RawIOBase):
+    pass
+
+class PySlowFlushRawIO(SlowFlushRawIO, pyio.RawIOBase):
+    pass
+
+
+class CloseFailureIO(MockRawIO):
+    closed = 0
+
+    def close(self):
+        if not self.closed:
+            self.closed = 1
+            raise OSError
+
+class CCloseFailureIO(CloseFailureIO, io.RawIOBase):
+    pass
+
+class PyCloseFailureIO(CloseFailureIO, pyio.RawIOBase):
+    pass
+
+
+class MockFileIO:
+
+    def __init__(self, data):
+        self.read_history = []
+        super().__init__(data)
+
+    def read(self, n=None):
+        res = super().read(n)
+        self.read_history.append(None if res is None else len(res))
+        return res
+
+    def readinto(self, b):
+        res = super().readinto(b)
+        self.read_history.append(res)
+        return res
+
+class CMockFileIO(MockFileIO, io.BytesIO):
+    pass
+
+class PyMockFileIO(MockFileIO, pyio.BytesIO):
+    pass
+
+
+class MockUnseekableIO:
+    def seekable(self):
+        return False
+
+    def seek(self, *args):
+        raise self.UnsupportedOperation("not seekable")
+
+    def tell(self, *args):
+        raise self.UnsupportedOperation("not seekable")
+
+    def truncate(self, *args):
+        raise self.UnsupportedOperation("not seekable")
+
+class CMockUnseekableIO(MockUnseekableIO, io.BytesIO):
+    UnsupportedOperation = io.UnsupportedOperation
+
+class PyMockUnseekableIO(MockUnseekableIO, pyio.BytesIO):
+    UnsupportedOperation = pyio.UnsupportedOperation
+
+
+class MockCharPseudoDevFileIO(MockFileIO):
+    # GH-95782
+    # ftruncate() does not work on these special files (and CPython then raises
+    # appropriate exceptions), so truncate() does not have to be accounted for
+    # here.
+    def __init__(self, data):
+        super().__init__(data)
+
+    def seek(self, *args):
+        return 0
+
+    def tell(self, *args):
+        return 0
+
+class CMockCharPseudoDevFileIO(MockCharPseudoDevFileIO, io.BytesIO):
+    pass
+
+class PyMockCharPseudoDevFileIO(MockCharPseudoDevFileIO, pyio.BytesIO):
+    pass
+
+
+class MockNonBlockWriterIO:
+
+    def __init__(self):
+        self._write_stack = []
+        self._blocker_char = None
+
+    def pop_written(self):
+        s = b"".join(self._write_stack)
+        self._write_stack[:] = []
+        return s
+
+    def block_on(self, char):
+        """Block when a given char is encountered."""
+        self._blocker_char = char
+
+    def readable(self):
+        return True
+
+    def seekable(self):
+        return True
+
+    def seek(self, pos, whence=0):
+        # naive implementation, enough for tests
+        return 0
+
+    def writable(self):
+        return True
+
+    def write(self, b):
+        b = bytes(b)
+        n = -1
+        if self._blocker_char:
+            try:
+                n = b.index(self._blocker_char)
+            except ValueError:
+                pass
+            else:
+                if n > 0:
+                    # write data up to the first blocker
+                    self._write_stack.append(b[:n])
+                    return n
+                else:
+                    # cancel blocker and indicate would block
+                    self._blocker_char = None
+                    return None
+        self._write_stack.append(b)
+        return len(b)
+
+class CMockNonBlockWriterIO(MockNonBlockWriterIO, io.RawIOBase):
+    BlockingIOError = io.BlockingIOError
+
+class PyMockNonBlockWriterIO(MockNonBlockWriterIO, pyio.RawIOBase):
+    BlockingIOError = pyio.BlockingIOError
+
+
+# Build classes which point to all the right mocks per io implementation
+class CTestCase(unittest.TestCase):
+    io = io
+    is_C = True
+
+    MockRawIO = CMockRawIO
+    MisbehavedRawIO = CMisbehavedRawIO
+    MockFileIO = CMockFileIO
+    CloseFailureIO = CCloseFailureIO
+    MockNonBlockWriterIO = CMockNonBlockWriterIO
+    MockUnseekableIO = CMockUnseekableIO
+    MockRawIOWithoutRead = CMockRawIOWithoutRead
+    SlowFlushRawIO = CSlowFlushRawIO
+    MockCharPseudoDevFileIO = CMockCharPseudoDevFileIO
+
+    # Use the class as a proxy to the io module members.
+    def __getattr__(self, name):
+        return getattr(io, name)
+
+
+class PyTestCase(unittest.TestCase):
+    io = pyio
+    is_C = False
+
+    MockRawIO = PyMockRawIO
+    MisbehavedRawIO = PyMisbehavedRawIO
+    MockFileIO = PyMockFileIO
+    CloseFailureIO = PyCloseFailureIO
+    MockNonBlockWriterIO = PyMockNonBlockWriterIO
+    MockUnseekableIO = PyMockUnseekableIO
+    MockRawIOWithoutRead = PyMockRawIOWithoutRead
+    SlowFlushRawIO = PySlowFlushRawIO
+    MockCharPseudoDevFileIO = PyMockCharPseudoDevFileIO
+
+    # Use the class as a proxy to the _pyio module members.
+    def __getattr__(self, name):
+        return getattr(pyio, name)

Lib/test/test_weakref.py

@@ -751,7 +751,6 @@ class ReferencesTestCase(TestBase):
         del c1, c2, C, D
         gc.collect()
 
-    @unittest.expectedFailure  # TODO: RUSTPYTHON
     def test_callback_in_cycle_resurrection(self):
         import gc
 
@@ -798,7 +797,6 @@ class ReferencesTestCase(TestBase):
         gc.collect()
         self.assertEqual(alist, [])
 
-    @unittest.expectedFailure  # TODO: RUSTPYTHON
     def test_callbacks_on_callback(self):
         import gc
 
@@ -1046,6 +1044,32 @@ class ReferencesTestCase(TestBase):
         stderr = res.err.decode("ascii", "backslashreplace")
         self.assertNotRegex(stderr, "_Py_Dealloc: Deallocator of type 'TestObj'")
 
+    def test_clearing_weakrefs_in_gc(self):
+        # This test checks that when finalizers are called:
+        # 1. weakrefs with callbacks have been cleared
+        # 2. weakrefs without callbacks have not been cleared
+        errors = []
+        def test():
+            class Class:
+                def __init__(self):
+                    self._self = self
+                    self.wr1 = weakref.ref(Class, lambda x: None)
+                    self.wr2 = weakref.ref(Class)
+
+                def __del__(self):
+                    # we can't use assert* here, because gc will swallow
+                    # exceptions
+                    if self.wr1() is not None:
+                        errors.append("weakref with callback as cleared")
+                    if self.wr2() is not Class:
+                        errors.append("weakref without callback was cleared")
+
+            Class()
+
+        test()
+        gc.collect()
+        self.assertEqual(errors, [])
+
 
 class SubclassableWeakrefTestCase(TestBase):
 
@@ -1791,6 +1815,11 @@ class MappingTestCase(TestBase):
     def test_weak_keyed_dict_update(self):
         self.check_update(weakref.WeakKeyDictionary,
                           {C(): 1, C(): 2, C(): 3})
+        d = weakref.WeakKeyDictionary()
+        msg = ("Keyword arguments are not supported: "
+               "cannot create weak reference to 'str' object")
+        with self.assertRaisesRegex(TypeError, msg):
+            d.update(k='v')
 
     def test_weak_keyed_delitem(self):
         d = weakref.WeakKeyDictionary()

Lib/weakref.py

@@ -408,14 +408,16 @@ class WeakKeyDictionary(_collections_abc.MutableMapping):
         return self.data.setdefault(ref(key, self._remove),default)
 
     def update(self, dict=None, /, **kwargs):
+        if kwargs:
+            msg = ("Keyword arguments are not supported: "
+                   "cannot create weak reference to 'str' object")
+            raise TypeError(msg)
         d = self.data
         if dict is not None:
             if not hasattr(dict, "items"):
                 dict = type({})(dict)
             for key, value in dict.items():
                 d[ref(key, self._remove)] = value
-        if len(kwargs):
-            self.update(kwargs)
 
     def __ior__(self, other):
         self.update(other)

crates/vm/src/builtins/weakref.rs

@@ -24,6 +24,9 @@ pub struct WeakNewArgs {
 }
 
 impl PyPayload for PyWeak {
+    // WeakRefList::add tracks only after linking and publishing the referent.
+    const NEW_REF_UNTRACKED: bool = true;
+
     #[inline]
     fn class(ctx: &Context) -> &'static Py<PyType> {
         ctx.types.weakref_type

crates/vm/src/gc_state.rs

@@ -921,11 +921,15 @@ impl GcState {
             })
             .collect();
 
-        // 6c: Clear existing weakrefs BEFORE calling __del__
+        // 6c: Clear callback-bearing weakrefs BEFORE calling __del__. Keep
+        // callback-free weakrefs usable throughout finalization.
+        let unreachable_set: GcSet<GcPtr> = unreachable.iter().copied().collect();
+        let is_unreachable =
+            |obj: &PyObject| unreachable_set.contains(&GcPtr(core::ptr::NonNull::from(obj)));
         let mut all_callbacks: Vec<(crate::PyRef<crate::object::PyWeak>, crate::PyObjectRef)> =
             Vec::new();
         for obj_ref in &unreachable_refs {
-            let callbacks = obj_ref.gc_clear_weakrefs_collect_callbacks();
+            let callbacks = obj_ref.gc_clear_weakrefs_collect_callbacks(&is_unreachable);
             all_callbacks.extend(callbacks);
         }
         for (wr, cb) in all_callbacks {
@@ -945,7 +949,6 @@ impl GcState {
 
         // Detect resurrection
         let mut resurrected_set: GcSet<GcPtr> = GcSet::default();
-        let unreachable_set: GcSet<GcPtr> = unreachable.iter().copied().collect();
 
         for obj in &unreachable_refs {
             let ptr = GcPtr(core::ptr::NonNull::from(obj.as_ref()));
@@ -1039,6 +1042,14 @@ impl GcState {
             // skips tp_clear for saved objects).
             let save_all = debug.contains(GcDebugFlags::SAVEALL);
 
+            // Callback-free weakrefs remain usable during finalizers, so a
+            // thread can still acquire an external reference at this point.
+            // Stop before taking generation/list locks, and keep the final
+            // liveness check and weakref closure in the same stopped phase.
+            // No callbacks, finalizers or tp_clear may run under this guard.
+            #[cfg(feature = "threading")]
+            let mut clear_stw = CollectStopTheWorld::new();
+
             // Untrack dead objects BEFORE clearing them, mirroring the
             // untrack-then-clear ordering of the refcount dealloc path.
             // A cleared object (e.g. a frame husk with iframe == None) must
@@ -1056,11 +1067,10 @@ impl GcState {
                     // One strong reference held by the `truly_dead` vec itself.
                     expected_counts.insert(GcPtr(NonNull::from(obj)), 1);
                 }
-                // With the objects out of the generation lists, no new external
-                // reference can appear. Count the references coming from within
-                // the dead set; any surplus in strong_count means another thread
-                // grabbed a reference before untracking (late resurrection) and
-                // the object must not be cleared.
+                // With the world stopped and these objects off the generation
+                // lists, count references from within the dead set. Any surplus
+                // means an external reference appeared before this stopped phase
+                // (late resurrection), and the object must not be cleared.
                 let mut referents: GcMap<GcPtr, Vec<NonNull<PyObject>>> = GcMap::default();
                 for obj_ref in &truly_dead {
                     let referent_ptrs = unsafe { obj_ref.gc_get_referent_ptrs() };
@@ -1106,6 +1116,20 @@ impl GcState {
                     unsafe { self.track_object(ptr.0, owner) };
                 }
             }
+            // Clear every weakref in or pointing to objects that will be
+            // cleared, including weakrefs created by finalizers. Do not run
+            // or drop callbacks here: all objects must remain intact until
+            // this pass finishes. Resurrected objects retain their weakrefs.
+            for obj_ref in &truly_dead {
+                if !late_resurrected.contains(&GcPtr(NonNull::from(obj_ref.as_ref()))) {
+                    obj_ref.gc_clear_weakrefs();
+                }
+            }
+            // Every remaining dead object is now unreachable through both
+            // generation lists and weakrefs. Resume before tp_clear can release
+            // edges or invoke arbitrary Python through deallocation.
+            #[cfg(feature = "threading")]
+            clear_stw.restart();
             rustpython_common::refcount::with_deferred_drops(|| {
                 if !save_all {
                     for obj_ref in &truly_dead {

crates/vm/src/object/core.rs

@@ -581,6 +581,9 @@ unsafe impl Traverse for PyObject {
 }
 
 // === Stripe lock for weakref list protection (WEAKREF_LIST_LOCK) ===
+// Like CPython's _Py_LOCK_DONT_DETACH weakref stripes, these critical sections
+// must not run Python, release Python references, or suspend an attached thread.
+// Traversal visitors must not recursively acquire a weakref stripe.
 
 #[cfg(feature = "threading")]
 mod weakref_lock {
@@ -590,27 +593,44 @@ mod weakref_lock {
 
     static LOCKS: [AtomicU8; NUM_WEAKREF_LOCKS] = [const { AtomicU8::new(0) }; NUM_WEAKREF_LOCKS];
 
+    #[derive(Debug)]
+    pub(super) struct WeakrefLock {
+        lock: &'static AtomicU8,
+    }
+
     pub(super) struct WeakrefLockGuard {
-        idx: usize,
+        lock: &'static AtomicU8,
     }
 
     impl Drop for WeakrefLockGuard {
         fn drop(&mut self) {
-            LOCKS[self.idx].store(0, Ordering::Release);
+            self.lock.store(0, Ordering::Release);
+        }
+    }
+
+    impl WeakrefLock {
+        pub(super) fn lock(&self) -> WeakrefLockGuard {
+            loop {
+                if self
+                    .lock
+                    .compare_exchange_weak(0, 1, Ordering::Acquire, Ordering::Relaxed)
+                    .is_ok()
+                {
+                    return WeakrefLockGuard { lock: self.lock };
+                }
+                core::hint::spin_loop();
+            }
+        }
+    }
+
+    pub(super) fn select(addr: usize) -> WeakrefLock {
+        WeakrefLock {
+            lock: &LOCKS[(addr >> 4) % NUM_WEAKREF_LOCKS],
         }
     }
 
     pub(super) fn lock(addr: usize) -> WeakrefLockGuard {
-        let idx = (addr >> 4) % NUM_WEAKREF_LOCKS;
-        loop {
-            if LOCKS[idx]
-                .compare_exchange_weak(0, 1, Ordering::Acquire, Ordering::Relaxed)
-                .is_ok()
-            {
-                return WeakrefLockGuard { idx };
-            }
-            core::hint::spin_loop();
-        }
+        select(addr).lock()
     }
 
     /// Reset all weakref stripe locks after fork in child process.
@@ -625,14 +645,27 @@ mod weakref_lock {
 
 #[cfg(not(feature = "threading"))]
 mod weakref_lock {
+    #[derive(Debug)]
+    pub(super) struct WeakrefLock;
+
     pub(super) struct WeakrefLockGuard;
 
     impl Drop for WeakrefLockGuard {
         fn drop(&mut self) {}
     }
 
-    pub(super) fn lock(_addr: usize) -> WeakrefLockGuard {
-        WeakrefLockGuard
+    impl WeakrefLock {
+        pub(super) fn lock(&self) -> WeakrefLockGuard {
+            WeakrefLockGuard
+        }
+    }
+
+    pub(super) fn select(_addr: usize) -> WeakrefLock {
+        WeakrefLock
+    }
+
+    pub(super) fn lock(addr: usize) -> WeakrefLockGuard {
+        select(addr).lock()
     }
 }
 
@@ -735,12 +768,13 @@ impl WeakRefList {
             }
         }
 
-        // Allocate OUTSIDE the stripe lock. PyRef::new_ref may trigger
-        // maybe_collect → GC → WeakRefList::clear on another object that
-        // hashes to the same stripe, which would deadlock on the spinlock.
+        // Allocate untracked: gc.get_objects() must not expose this node
+        // before its referent is published. Tracking below can trigger GC,
+        // so it must also happen outside the stripe lock.
         let weak_payload = PyWeak {
             pointers: Pointers::new(),
-            wr_object: Radium::new(obj as *const PyObject as *mut PyObject),
+            wr_object: Radium::new(ptr::null_mut()),
+            weakrefs_lock: weakref_lock::select(obj as *const PyObject as usize),
             callback: UnsafeCell::new(callback),
             hash: Radium::new(crate::common::hash::SENTINEL),
         };
@@ -760,7 +794,7 @@ impl WeakRefList {
             None
         };
         if let Some(existing) = existing {
-            // Nullify wr_object so drop_inner won't unlink an
+            // Nullify wr_object so clear_ref won't unlink an
             // un-inserted node (which would corrupt the list head).
             weak.wr_object.store(ptr::null_mut(), Ordering::Relaxed);
             return existing;
@@ -784,6 +818,16 @@ impl WeakRefList {
         if is_generic {
             self.generic.store(node_ptr.as_ptr(), Ordering::Relaxed);
         }
+        // Only linked nodes may advertise a referent: an allocation unwind
+        // must never try to unlink an unpublished node.
+        weak.wr_object
+            .store(obj as *const PyObject as *mut PyObject, Ordering::Release);
+
+        drop(_lock);
+        // SAFETY: this fresh allocation opted out of automatic tracking.
+        // Our strong reference keeps it alive, and it cannot enter a GC
+        // candidate list before tracking, so its gc_bits are still zero.
+        unsafe { crate::gc_state::track_new_object(NonNull::from(weak.as_object())) };
 
         weak
     }
@@ -828,7 +872,8 @@ impl WeakRefList {
         match NonNull::new(candidate_ptr) {
             Some(candidate) => {
                 let node = unsafe { candidate.as_ref() };
-                let has_callback = unsafe { (&*node.payload.callback.get()).is_some() };
+                // SAFETY: the referent stripe is held by the caller.
+                let has_callback = unsafe { &*node.payload.callback.get() }.is_some();
                 let node_cls = node.class();
                 // PyWeakref_CheckProxy: the basic-proxy slot is reserved for
                 // the canonical proxy type; subclasses and callback-less ref
@@ -849,99 +894,98 @@ impl WeakRefList {
     /// Called when the owner object is being dropped.
     // PyObject_ClearWeakRefs
     fn clear(&self, obj: &PyObject) {
-        let obj_addr = obj as *const PyObject as usize;
-        let _lock = weakref_lock::lock(obj_addr);
-
-        // Clear generic cache
+        // Owners precede the guard so unwinding also drops them after unlock.
+        let mut callbacks: Vec<(PyRef<PyWeak>, Option<PyObjectRef>)> = Vec::new();
+        let _lock = weakref_lock::lock(obj as *const PyObject as usize);
         self.generic.store(ptr::null_mut(), Ordering::Relaxed);
-
-        // Walk the list, collecting weakrefs with callbacks
-        let mut callbacks: Vec<(PyRef<PyWeak>, PyObjectRef)> = Vec::new();
         let mut current = NonNull::new(self.head.load(Ordering::Relaxed));
         while let Some(node) = current {
             let next = unsafe { WeakLink::pointers(node).as_ref().get_next() };
-
             let wr = unsafe { node.as_ref() };
-
-            // Mark weakref as dead
+            // Reserve before acquiring a Python owner; push cannot then unwind.
+            callbacks.reserve(1);
+            if wr.ref_count.safe_inc() {
+                let owner = unsafe { PyRef::<PyWeak>::from_raw(wr as *const Py<PyWeak>) };
+                callbacks.push((owner, None));
+                // Keep the owner in the outer batch before taking its callback.
+                let entry = callbacks.last_mut().unwrap();
+                // SAFETY: the referent stripe protects the callback as well.
+                entry.1 = unsafe { &mut *wr.payload.callback.get() }.take();
+            }
+            unsafe { unlink_weakref(self, node) };
+            // Publish death only after unlinking, while the original stripe
+            // still protects this node from a concurrent deallocator.
             wr.payload
                 .wr_object
-                .store(ptr::null_mut(), Ordering::Relaxed);
-
-            // Unlink from list
-            unsafe {
-                let mut ptrs = WeakLink::pointers(node);
-                ptrs.as_mut().set_prev(None);
-                ptrs.as_mut().set_next(None);
-            }
-
-            // Collect callback only if we can still acquire a strong ref.
-            if wr.ref_count.safe_inc() {
-                let wr_ref = unsafe { PyRef::<PyWeak>::from_raw(wr as *const Py<PyWeak>) };
-                let cb = unsafe { wr.payload.callback.get().replace(None) };
-                if let Some(cb) = cb {
-                    callbacks.push((wr_ref, cb));
-                }
-            }
-
+                .store(ptr::null_mut(), Ordering::Release);
             current = next;
         }
-        self.head.store(ptr::null_mut(), Ordering::Relaxed);
-
-        // Invoke callbacks outside the lock
         drop(_lock);
-        for (wr, cb) in callbacks {
-            crate::vm::thread::with_vm(&cb, |vm| {
-                let _ = cb.call((wr.clone(),), vm);
-            });
+        for (wr, callback) in callbacks {
+            if let Some(cb) = callback {
+                crate::vm::thread::with_vm(&cb, |vm| {
+                    let _ = cb.call((wr.clone(),), vm);
+                });
+            }
         }
     }
 
-    /// Clear all weakrefs but DON'T call callbacks. Instead, return them for later invocation.
-    /// Used by GC to ensure ALL weakrefs are cleared BEFORE any callbacks are invoked.
-    /// handle_weakrefs() clears all weakrefs first, then invokes callbacks.
-    fn clear_for_gc_collect_callbacks(&self, obj: &PyObject) -> Vec<(PyRef<PyWeak>, PyObjectRef)> {
-        let obj_addr = obj as *const PyObject as usize;
-        let _lock = weakref_lock::lock(obj_addr);
-
-        // Clear generic cache
-        self.generic.store(ptr::null_mut(), Ordering::Relaxed);
-
+    /// Clear callback-bearing weakrefs before finalizers. Retain callback
+    /// ownership on each weakref until its later tp_clear or deallocation.
+    fn clear_for_gc_collect_callbacks(
+        &self,
+        obj: &PyObject,
+        is_unreachable: &impl Fn(&PyObject) -> bool,
+    ) -> Vec<(PyRef<PyWeak>, PyObjectRef)> {
+        let mut owners: Vec<PyRef<PyWeak>> = Vec::new();
         let mut callbacks = Vec::new();
+        let _lock = weakref_lock::lock(obj as *const PyObject as usize);
         let mut current = NonNull::new(self.head.load(Ordering::Relaxed));
         while let Some(node) = current {
             let next = unsafe { WeakLink::pointers(node).as_ref().get_next() };
-
             let wr = unsafe { node.as_ref() };
-
-            // Mark weakref as dead
-            wr.payload
-                .wr_object
-                .store(ptr::null_mut(), Ordering::Relaxed);
-
-            // Unlink from list
-            unsafe {
-                let mut ptrs = WeakLink::pointers(node);
-                ptrs.as_mut().set_prev(None);
-                ptrs.as_mut().set_next(None);
-            }
-
-            // Collect callback without invoking only if we can keep weakref alive.
-            if wr.ref_count.safe_inc() {
-                let wr_ref = unsafe { PyRef::<PyWeak>::from_raw(wr as *const Py<PyWeak>) };
-                let cb = unsafe { wr.payload.callback.get().replace(None) };
-                if let Some(cb) = cb {
-                    callbacks.push((wr_ref, cb));
+            // SAFETY: the referent stripe protects this borrowed callback.
+            let callback = unsafe { &*wr.payload.callback.get() };
+            if callback.is_some() {
+                owners.reserve(1);
+                callbacks.reserve(1);
+                if !is_unreachable(wr.as_object()) && wr.ref_count.safe_inc() {
+                    let owner = unsafe { PyRef::<PyWeak>::from_raw(wr as *const Py<PyWeak>) };
+                    owners.push(owner);
+                    let callback = callback.clone();
+                    if let Some(callback) = callback {
+                        callbacks.push((owners.last().unwrap().clone(), callback));
+                    }
                 }
+                unsafe { unlink_weakref(self, node) };
+                // No node access after publishing null without a strong pin.
+                wr.payload
+                    .wr_object
+                    .store(ptr::null_mut(), Ordering::Release);
             }
-
             current = next;
         }
-        self.head.store(ptr::null_mut(), Ordering::Relaxed);
-
+        drop(_lock);
         callbacks
     }
 
+    /// Unlink remaining weakrefs without invoking or releasing callbacks.
+    fn clear_for_gc(&self, obj: &PyObject) {
+        let _lock = weakref_lock::lock(obj as *const PyObject as usize);
+        self.generic.store(ptr::null_mut(), Ordering::Relaxed);
+        let mut current = NonNull::new(self.head.load(Ordering::Relaxed));
+        while let Some(node) = current {
+            let next = unsafe { WeakLink::pointers(node).as_ref().get_next() };
+            let wr = unsafe { node.as_ref() };
+            unsafe { unlink_weakref(self, node) };
+            // Publishing null is the final access to this unpinned node.
+            wr.payload
+                .wr_object
+                .store(ptr::null_mut(), Ordering::Release);
+            current = next;
+        }
+    }
+
     fn count(&self, obj: &PyObject) -> usize {
         let _lock = weakref_lock::lock(obj as *const PyObject as usize);
         let mut count = 0usize;
@@ -956,11 +1000,12 @@ impl WeakRefList {
     }
 
     fn get_weak_references(&self, obj: &PyObject) -> Vec<PyRef<PyWeak>> {
-        let _lock = weakref_lock::lock(obj as *const PyObject as usize);
         let mut v = Vec::new();
+        let _lock = weakref_lock::lock(obj as *const PyObject as usize);
         let mut current = NonNull::new(self.head.load(Ordering::Relaxed));
         while let Some(node) = current {
             let wr = unsafe { node.as_ref() };
+            v.reserve(1);
             if wr.ref_count.safe_inc() {
                 v.push(unsafe { PyRef::<PyWeak>::from_raw(wr as *const Py<PyWeak>) });
             }
@@ -1000,20 +1045,27 @@ unsafe impl Link for WeakLink {
 }
 
 // PyWeakReference: each weakref holds a direct pointer to its referent.
-#[pyclass(name = "ReferenceType", module = "weakref")]
+#[pyclass(name = "ReferenceType", module = "weakref", traverse = "manual")]
 #[derive(Debug)]
 pub struct PyWeak {
     pointers: Pointers<Py<Self>>,
     /// Direct pointer to the referent object, null when dead.
     /// Equivalent to wr_object in PyWeakReference.
     wr_object: PyAtomic<*mut PyObject>,
-    /// Protected by stripe lock (keyed on wr_object address).
+    /// Handle to the original static referent stripe, like CPython's
+    /// weakrefs_lock. The same stripe protects the callback even after
+    /// wr_object becomes null and the referent is freed.
+    weakrefs_lock: weakref_lock::WeakrefLock,
+    /// Owned callback edge, protected by the original referent stripe.
+    /// GC can retain this edge after unlinking the weakref.
     callback: UnsafeCell<Option<PyObjectRef>>,
     pub(crate) hash: PyAtomic<crate::common::hash::PyHash>,
 }
 
 cfg_select! {
     feature = "threading" => {
+        // SAFETY: mutable callback and list access is serialized by the
+        // original referent stripe; referent and hash reads are atomic.
         unsafe impl Send for PyWeak {}
         unsafe impl Sync for PyWeak {}
     }
@@ -1048,42 +1100,40 @@ impl PyWeak {
         self.wr_object.load(Ordering::Acquire).is_null()
     }
 
-    /// Get the callback associated with this weak reference.
-    /// Returns `None` if there is no callback or if the referent has been
-    /// collected (at which point the callback was already consumed).
+    /// Get the owned callback, which GC can retain after clearing the referent.
     pub(crate) fn get_callback(&self) -> Option<PyObjectRef> {
-        let obj_ptr = self.wr_object.load(Ordering::Acquire);
-        if obj_ptr.is_null() {
-            // Dead weakref: callback was consumed during clear
-            return None;
-        }
-
-        let _lock = weakref_lock::lock(obj_ptr as usize);
-
-        // Double-check under lock (clear may have run between our check and lock)
-        let obj_ptr = self.wr_object.load(Ordering::Relaxed);
-        if obj_ptr.is_null() {
-            return None;
-        }
-
-        // Safety: we hold the stripe lock that protects the callback field
-        let callback = unsafe { &*self.callback.get() };
-        callback.clone()
+        let _lock = self.weakrefs_lock.lock();
+        // SAFETY: the original referent stripe protects the callback.
+        unsafe { &*self.callback.get() }.clone()
     }
 
-    /// weakref_dealloc: remove from list if still linked.
-    fn drop_inner(&self) {
-        let obj_ptr = self.wr_object.load(Ordering::Acquire);
-        if obj_ptr.is_null() {
-            return; // Already cleared by WeakRefList::clear()
+    /// Unlink and steal the callback under the original referent stripe.
+    /// Return ownership so callers release it only after unlocking.
+    fn clear(&self) -> Option<PyObjectRef> {
+        let _lock = self.weakrefs_lock.lock();
+        // SAFETY: the original referent stripe protects both fields, including
+        // callbacks retained by GC after the weakref was already unlinked.
+        unsafe {
+            self.clear_ref_lock_held();
+            (&mut *self.callback.get()).take()
         }
+    }
 
-        let _lock = weakref_lock::lock(obj_ptr as usize);
+    /// _PyWeakref_ClearRef: unlink without releasing the callback.
+    fn clear_ref(&self) {
+        let _lock = self.weakrefs_lock.lock();
+        // SAFETY: the original referent stripe is held.
+        unsafe { self.clear_ref_lock_held() };
+    }
 
-        // Double-check under lock
+    /// Unlink without releasing the callback.
+    ///
+    /// # Safety
+    /// The original referent stripe must be held.
+    unsafe fn clear_ref_lock_held(&self) {
         let obj_ptr = self.wr_object.load(Ordering::Relaxed);
         if obj_ptr.is_null() {
-            return; // Cleared between our check and lock acquisition
+            return;
         }
 
         let obj = unsafe { &*obj_ptr };
@@ -1106,12 +1156,31 @@ impl PyWeak {
     }
 }
 
+// SAFETY: the callback is the only owned Python edge; wr_object is borrowed.
+unsafe impl Traverse for PyWeak {
+    fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) {
+        // Stripe critical sections never suspend or run Python, so a stopped
+        // thread cannot own this lock. Also synchronize gc.get_referents(),
+        // which traverses while other threads are running.
+        let _lock = self.weakrefs_lock.lock();
+        // SAFETY: the original referent stripe protects the callback.
+        unsafe { &*self.callback.get() }.traverse(tracer_fn);
+    }
+
+    fn clear(&mut self, out: &mut Vec<PyObjectRef>) {
+        let callback = Self::clear(self);
+        if let Some(callback) = callback {
+            out.push(callback);
+        }
+    }
+}
+
 impl Drop for PyWeak {
     #[inline(always)]
     fn drop(&mut self) {
         // we do NOT have actual exclusive access!
         let me: &Self = self;
-        me.drop_inner();
+        drop(me.clear());
     }
 }
 
@@ -2133,7 +2202,7 @@ impl PyObject {
         }
 
         // Clear weak refs AFTER __del__.
-        // Note: This differs from GC behavior which clears weakrefs before finalizers,
+        // GC clears callback-bearing weakrefs before finalizers and the rest later,
         // but for direct deallocation (drop_slow_inner), we need to allow the finalizer
         // to run without triggering use-after-free from WeakRefList operations.
         if let Some(wrl) = self.weak_ref_list() {
@@ -2260,18 +2329,30 @@ impl PyObject {
         }
     }
 
-    /// Clear weakrefs but collect callbacks instead of calling them.
-    /// This is used by GC to ensure ALL weakrefs are cleared BEFORE any callbacks run.
-    /// Returns collected callbacks as (PyRef<PyWeak>, callback) pairs.
-    // = handle_weakrefs
-    pub fn gc_clear_weakrefs_collect_callbacks(&self) -> Vec<(PyRef<PyWeak>, PyObjectRef)> {
+    /// Clear callback-bearing weakrefs before finalizers and return callbacks
+    /// belonging to reachable weakrefs for invocation outside the list lock.
+    pub fn gc_clear_weakrefs_collect_callbacks(
+        &self,
+        is_unreachable: &impl Fn(&Self) -> bool,
+    ) -> Vec<(PyRef<PyWeak>, PyObjectRef)> {
         if let Some(wrl) = self.weak_ref_list() {
-            wrl.clear_for_gc_collect_callbacks(self)
+            wrl.clear_for_gc_collect_callbacks(self, is_unreachable)
         } else {
             vec![]
         }
     }
 
+    /// Clear weakrefs both in and pointing into the final unreachable set.
+    /// Callback edges remain owned until tp_clear or deallocation.
+    pub(crate) fn gc_clear_weakrefs(&self) {
+        if let Some(weakref) = self.downcast_ref::<PyWeak>() {
+            weakref.clear_ref();
+        }
+        if let Some(wrl) = self.weak_ref_list() {
+            wrl.clear_for_gc(self);
+        }
+    }
+
     /// Get raw pointers to referents without incrementing reference counts.
     /// This is used during GC to avoid reference count manipulation.
     /// tp_traverse visits objects without incref
@@ -3590,6 +3671,78 @@ mod tests {
         drop(obj);
     }
 
+    #[test]
+    fn weakref_allocation_defers_gc_tracking_until_publication() {
+        crate::Interpreter::without_stdlib(Default::default()).enter(|vm| {
+            let target = vm.ctx.new_class(
+                None,
+                "WeakrefTarget",
+                vm.ctx.types.object_type.to_owned(),
+                Default::default(),
+            );
+            let subclass = vm.ctx.new_class(
+                None,
+                "WeakrefSubclass",
+                vm.ctx.types.weakref_type.to_owned(),
+                Default::default(),
+            );
+            for class in [
+                vm.ctx.types.weakref_type.to_owned(),
+                vm.ctx.types.weakproxy_type.to_owned(),
+                vm.ctx.types.weakcallableproxy_type.to_owned(),
+                subclass,
+            ] {
+                let dict = class
+                    .slots
+                    .flags
+                    .has_feature(crate::types::PyTypeFlags::HAS_DICT)
+                    .then(|| vm.ctx.new_dict());
+                // Exercise the allocator boundary directly: an uninitialized
+                // weakref must not be published through GC generation lists.
+                let unpublished = PyRef::new_ref(
+                    PyWeak {
+                        pointers: Pointers::new(),
+                        wr_object: Radium::new(ptr::null_mut()),
+                        weakrefs_lock: weakref_lock::select(
+                            target.as_object() as *const PyObject as usize
+                        ),
+                        callback: UnsafeCell::new(None),
+                        hash: Radium::new(crate::common::hash::SENTINEL),
+                    },
+                    class.clone(),
+                    dict,
+                );
+                assert_eq!(unpublished.as_object().0.gc_bits.load(Ordering::Relaxed), 0);
+                assert!(!unpublished.as_object().is_gc_tracked());
+                assert!(
+                    !vm.state
+                        .gc
+                        .get_objects(None)
+                        .iter()
+                        .any(|object| object.is(unpublished.as_object()))
+                );
+                drop(unpublished);
+
+                let published = target
+                    .as_object()
+                    .downgrade_with_typ(None, class, vm)
+                    .unwrap();
+                assert!(published.as_object().is_gc_tracked());
+                assert!(
+                    vm.state
+                        .gc
+                        .get_objects(None)
+                        .iter()
+                        .any(|object| object.is(published.as_object()))
+                );
+                assert_eq!(
+                    published.wr_object.load(Ordering::Acquire),
+                    target.as_object() as *const PyObject as *mut PyObject,
+                );
+            }
+        });
+    }
+
     /// A weakref node stays linked into its target's list until its own
     /// `Drop` unlinks it, and `WeakRefList::add` reads the class off every
     /// node it walks looking for a proxy to reuse. A node that lost its class

scripts/update_lib/deps.py

@@ -570,10 +570,7 @@ DEPENDENCIES = {
     "io": {
         "hard_deps": ["_pyio.py"],
         "test": [
-            "test_io.py",
-            "test_bufio.py",
-            "test_fileio.py",
-            "test_memoryio.py",
+            "test_io",
         ],
     },
     "dbm": {
@@ -607,8 +604,7 @@ DEPENDENCIES = {
     "file": {
         "lib": [],
         "test": [
-            "test_file.py",
-            "test_largefile.py",
+            "test_io",
         ],
     },
     "fcntl": {

scripts/update_lib/tests/test_deps.py

@@ -110,6 +110,18 @@ class TestGetTestPaths(unittest.TestCase):
         self.assertEqual(len(paths), 1)
         self.assertEqual(paths[0], pathlib.Path("cpython/Lib/test/test_regrtest"))
 
+    def test_io_package(self):
+        self.assertEqual(
+            get_test_paths("io", "cpython"),
+            (pathlib.Path("cpython/Lib/test/test_io"),),
+        )
+
+    def test_file_alias_uses_io_package(self):
+        self.assertEqual(
+            get_test_paths("file", "cpython"),
+            (pathlib.Path("cpython/Lib/test/test_io"),),
+        )
+
     def test_default_directory(self):
         """Test default to test_name/ directory."""
         with tempfile.TemporaryDirectory() as tmpdir: