From afad92bf1b78e33c53b9018fef31233cefcb5225 Mon Sep 17 00:00:00 2001 From: "Jeong, YunWon" <69878+youknowone@users.noreply.github.com> Date: Thu, 24 Sep 2026 12:21:26 +0900 Subject: [PATCH] Add OrderedDict, marshal, inspect, and chflags bitflagset (#8780) * Implement _collections.OrderedDict Extracted from #6799: native OrderedDict as a dict subclass, iterator mutation tracking, and mapping-view set operations. * Add OrderedDict, warnings source lines, and socket sendmsg Implement _collections.OrderedDict as a dict subclass with a linked list of insertion-order nodes. Pack struct F/D as native float/double complex pairs, and report unexpected __replace__ field names as a list repr. Register encodings.latin_1 at startup. Reject NUL in input() prompts. Copy non-exact type dicts in mapping order. Fetch warning source lines through _bless_my_loader and unicode splitlines. Keep the last warnings.filters list after deletion. Snapshot sendmsg sequences before converting buffers, parse ancillary ints with __index__, and dispatch AF_CAN by protocol. Store sys.trace and sys.profile on each ThreadSlot so _settraceallthreads/_setprofileallthreads are visible to sys.gettrace/getprofile on other threads. Use as frozen importlib co_filename. Keep HEAPTYPE socket objects GC-tracked. Assisted-by: Grok:grok-4.6 * Load marshal via readinto and dump sets in dumps order marshal.load fills a scratch buffer with readinto on a contiguous writable memoryview, and rejects a returned length other than the requested size. set and frozenset dumps write elements in sorted(v, key=marshal.dumps) order. Assisted-by: Grok:grok-4.6 * Add classmethod_descriptor and ABC IS_ABSTRACT flags Give METH_CLASS methods a distinct classmethod_descriptor type. Store Py_TPFLAGS_IS_ABSTRACT from __abstractmethods__ and look up that attribute only in the type's own dict. Expose defaultdict.default_factory as a member descriptor. Inherit bool's unused number slots from int. Mark set and frozenset __contains__ as METH_COEXIST. Parse classmethod, staticmethod, object.__new__, and heap type text signatures. Register interactive source in linecache. Assisted-by: Grok:grok-4.6 * address review: odict reuse, descr call, new wrapper Reuse odict node slots and walk the live list in next_after. Require both | operands to be dicts. Drop the odict iterator on GC clear. Bind classmethod descriptors through descr_get and update IS_ABSTRACT under the type lock. Give odict keys and items sq_contains. Walk tp_new_wrapper past new_wrapper so HEAPTYPE socket subclasses can call socket.__new__. Split FreeBSD Bluetooth protocol constants from Linux values. Assisted-by: Grok:grok-4.6 * Represent chflags bits with bitflagset Assisted-by: Grok:grok-4.6 * Represent access and mode bits with bitflagset Assisted-by: Grok:grok-4.6 * Continue unfinished REPL suites as incomplete input Assisted-by: Grok:grok-4.6 * Add pickle and os.stat inspect signatures Give _pickle.Pickler the clinic constructor text signature and accept fix_imports and buffer_callback as positional-or-keyword. Allow #[pyfunction(text_signature)] and use it so os.stat reports (path, *, dir_fd=None, follow_symlinks=True). Uncomment import _pickle and drop the pickle inspect expectedFailures. Assisted-by: Grok:grok-4.6 * Add Unpickler, dump/load, and lstat signatures Give _pickle Unpickler, dump, dumps, load, and loads clinic text signatures, and os.lstat (path, *, dir_fd=None). Assisted-by: Grok:grok-4.6 * Complete OrderedDict with insertion-order links Move the native OrderedDict into ordered_dict.rs and track order with a linked list of nodes, keeping mapping-view set operations. Assisted-by: Grok:grok-4.6 * address CI: stat casts, HCI_DATA_DIR, passing xfails Compare stat permission bits as Mode so Linux clippy does not flag a u32-to-u32 cast. Export HCI_DATA_DIR with the Bluetooth socket option values. Drop expectedFailure from tests that now pass. Assisted-by: Grok:grok-4.6 * Fix wasm teardown deadlock and bluetooth constants Assisted-by: Grok:grok-4.6 * Clear process-wide eval breaker bits in STW test Assisted-by: Grok:grok-4.6 * Drop exclusive-held panic that tripped wasm clippy Assisted-by: Grok:grok-4.6 * Fix OrderedDict contains and export Windows BDADDR Assisted-by: Grok:grok-4.6 * Add cmp_to_key and hash-once LRU cache _functools.cmp_to_key now returns KeyWrapper. LRU lookups store the key hash on _lru_list_elem so __hash__ is not called again on hits or eviction. Assisted-by: Grok:grok-4.6 * Allow subclass keywords on list, tuple, and float list.__init__ rejects keywords only when tp_new is still list's. tuple/float.__new__ reject them only when tp_init is still the base's. Assisted-by: Grok:grok-4.6 * Fix CI lint, test re-export, and structseq members Drop the duplicate clear_eval_breaker_for_test re-export. Format structseq imports. Expose struct sequence fields as readonly members so assignment raises "readonly attribute". Assisted-by: Grok:grok-4.6 * Allow subclass keywords on itertools and filter Positional-only constructors ignore extra keywords when a subclass has replaced tp_init, matching list/tuple/float. Assisted-by: Grok:grok-4.6 * Remove text_signature derive parameter Drop the string override on #[pyfunction] and #[pyclass]. Signatures are generated from the Rust parameter list only. Restore inspect expectedFailures that depended on the override. Assisted-by: Grok:grok-4.6 * Rename set contains_method to __contains__ Keep a borrowed contains helper for slots and C-API. The coexist pymethod is named __contains__. Assisted-by: Grok:grok-4.6 * Use MonitoringEvent instead of EVENT_* aliases Drop the unused MonitoringEvents bitflags and the EVENT_PY_START style mask constants. Call sites use the enum and .mask(). Assisted-by: Grok:grok-4.6 * Store warnings filters as PyListRef The list already serializes its elements. Drop the extra mutex around the cached list pointer. Assisted-by: Grok:grok-4.6 * Re-export clear_eval_breaker_for_test after rebase Keep the STW test helper visible without bringing back stop_bit_set. Assisted-by: Grok:grok-4.6 * Fix groupby reentrancy and islice arguments Give _grouper its own tgtkey and take currvalue after the key comparison. Accept __index__ on islice bounds, reject step 0, and drain the source up to stop. Assisted-by: Grok:grok-4.6 * SETREF warnings filters, onceregistry, and defaultaction Keep the last list, dict, or str after the module attributes are deleted by swapping the interpreter cache pointer. Do not store a None fallback for _warnings_context. Assisted-by: Grok:grok-4.6 * Iterate OrderedDict inner dict via next_entry &PyDict is not an iterator after the borrowed-ref signature change. Assisted-by: Grok:grok-4.6 * Remove expectedFailure from testBluetoothConstants Linux now exports BTPROTO_HCI and the rest of the set. Assisted-by: Grok:grok-4.6 * Borrow get_filter arguments as PyObject Match the warn_explicit call site that passes borrowed handles after the internal-helper borrow change. Assisted-by: Grok:grok-4.6 * Fix bool slots and SemHandle safety docs Leave unused bool number slots unset so inspect reports __add__ on int. Document SemHandle::from_raw safety. Assisted-by: Grok:grok-4.6 * Keep sendmsg errors and drop duplicate linecache Replace sendmsg iterable TypeError only. Register REPL source in linecache once per compiled input. Assisted-by: Grok:grok-4.6 * Nest REPL parse or-patterns for clippy Assisted-by: Grok:grok-4.6 * Mark pydoc os.stat signature test expectedFailure Assisted-by: Grok:grok-4.6 --------- Co-authored-by: Lee Dogeon --- .cspell.dict/cpython.txt | 3 + Cargo.lock | 2 + Lib/test/test_abc.py | 1 - Lib/test/test_builtin.py | 9 - Lib/test/test_dataclasses/__init__.py | 1 - Lib/test/test_float.py | 1 - Lib/test/test_functools.py | 41 +- Lib/test/test_inspect/test_inspect.py | 19 +- Lib/test/test_itertools.py | 3 - Lib/test/test_list.py | 1 - Lib/test/test_marshal.py | 3 +- Lib/test/test_ordered_dict.py | 9 - Lib/test/test_pydoc/test_pydoc.py | 4 +- Lib/test/test_re.py | 1 - Lib/test/test_repl.py | 2 - Lib/test/test_socket.py | 11 - Lib/test/test_struct.py | 2 - Lib/test/test_structseq.py | 1 - Lib/test/test_sys.py | 3 - Lib/test/test_threading.py | 2 - Lib/test/test_trace.py | 2 - Lib/test/test_tuple.py | 1 - Lib/test/test_warnings/__init__.py | 35 - Lib/test/test_zipfile/test_core.py | 2 - Lib/test/test_zoneinfo/test_zoneinfo.py | 1 - crates/capi/src/setobject.rs | 6 +- crates/common/src/encodings/cjk/iso2022.rs | 27 +- crates/compiler-core/src/bytecode.rs | 43 +- crates/compiler-core/src/marshal.rs | 18 +- crates/host_env/Cargo.toml | 1 + crates/host_env/src/io.rs | 51 +- crates/host_env/src/io_unsupported.rs | 51 +- crates/host_env/src/multiprocessing.rs | 7 +- crates/host_env/src/os.rs | 147 +- crates/host_env/src/posix.rs | 14 +- crates/stdlib/src/pickle.rs | 4 +- crates/stdlib/src/pystruct.rs | 6 + crates/stdlib/src/socket.rs | 480 ++++++- crates/vm/Cargo.toml | 1 + .../vm/Lib/core_modules/encodings_latin_1.py | 50 + crates/vm/src/buffer.rs | 59 +- crates/vm/src/builtins/bool.rs | 2 +- crates/vm/src/builtins/code.rs | 36 +- crates/vm/src/builtins/descriptor.rs | 165 ++- crates/vm/src/builtins/dict.rs | 114 +- crates/vm/src/builtins/filter.rs | 1 + crates/vm/src/builtins/float.rs | 21 +- crates/vm/src/builtins/frame.rs | 41 +- crates/vm/src/builtins/list.rs | 24 +- crates/vm/src/builtins/set.rs | 18 +- crates/vm/src/builtins/str.rs | 6 +- crates/vm/src/builtins/tuple.rs | 18 +- crates/vm/src/builtins/type.rs | 162 ++- crates/vm/src/frame.rs | 85 +- crates/vm/src/function/method.rs | 11 +- crates/vm/src/import.rs | 6 +- crates/vm/src/intern.rs | 8 +- crates/vm/src/protocol/callable.rs | 24 +- crates/vm/src/signal.rs | 116 +- crates/vm/src/stdlib/_abc.rs | 5 + crates/vm/src/stdlib/_collections.rs | 46 +- .../src/stdlib/_collections/ordered_dict.rs | 1237 +++++++++++++++++ crates/vm/src/stdlib/_functools.rs | 141 +- crates/vm/src/stdlib/_interpchannels.rs | 113 +- crates/vm/src/stdlib/_interpqueues.rs | 35 +- crates/vm/src/stdlib/_io.rs | 12 +- crates/vm/src/stdlib/_stat.rs | 190 +-- crates/vm/src/stdlib/_testinternalcapi.rs | 2 +- crates/vm/src/stdlib/_types.rs | 3 +- crates/vm/src/stdlib/_warnings.rs | 64 +- crates/vm/src/stdlib/builtins.rs | 44 +- crates/vm/src/stdlib/itertools.rs | 184 ++- crates/vm/src/stdlib/marshal.rs | 141 +- crates/vm/src/stdlib/mod.rs | 1 + crates/vm/src/stdlib/sys.rs | 38 + crates/vm/src/stdlib/sys/monitoring.rs | 243 ++-- crates/vm/src/types/slot.rs | 28 + crates/vm/src/types/structseq.rs | 27 +- crates/vm/src/types/zoo.rs | 2 + crates/vm/src/vm/compile.rs | 55 +- crates/vm/src/vm/compile_mode.rs | 29 +- crates/vm/src/vm/context.rs | 28 +- crates/vm/src/vm/crossinterp.rs | 27 +- crates/vm/src/vm/interpreter.rs | 12 +- crates/vm/src/vm/mod.rs | 134 +- crates/vm/src/vm/thread.rs | 233 ++-- crates/vm/src/warn.rs | 225 ++- src/shell.rs | 59 +- 88 files changed, 4034 insertions(+), 1307 deletions(-) create mode 100644 crates/vm/Lib/core_modules/encodings_latin_1.py create mode 100644 crates/vm/src/stdlib/_collections/ordered_dict.rs diff --git a/.cspell.dict/cpython.txt b/.cspell.dict/cpython.txt index f0b5e608d..a00cd9e25 100644 --- a/.cspell.dict/cpython.txt +++ b/.cspell.dict/cpython.txt @@ -153,6 +153,7 @@ mimalloc miscompiles mult multibytecodec +mycmp mystricmp mystrnicmp nameobj @@ -187,6 +188,8 @@ numkwonly numposorkw nvars obmalloc +odict +ODICT opname opnames orelse diff --git a/Cargo.lock b/Cargo.lock index 5b972a79f..6f18b37a8 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -3867,6 +3867,7 @@ name = "rustpython-host_env" version = "0.6.0" dependencies = [ "bitflags 2.13.1", + "bitflagset", "cc", "der 0.8.2", "dns-lookup", @@ -4108,6 +4109,7 @@ version = "0.6.0" dependencies = [ "ascii", "bitflags 2.13.1", + "bitflagset", "bstr", "constant_time_eq 0.6.0", "crossbeam-utils", diff --git a/Lib/test/test_abc.py b/Lib/test/test_abc.py index e29fc5a23..80ee9e0ba 100644 --- a/Lib/test/test_abc.py +++ b/Lib/test/test_abc.py @@ -168,7 +168,6 @@ def test_factory(abc_ABCMeta, abc_get_cache_token): msg = r"class C without an implementation for abstract methods 'method_one', 'method_two'" self.assertRaisesRegex(TypeError, msg, C) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: False is not true def test_abstractmethod_integration(self): for abstractthing in [abc.abstractmethod, abc.abstractproperty, abc.abstractclassmethod, diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py index 937a0b6a1..8fdee358a 100644 --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -2711,43 +2711,35 @@ class PtyTests(unittest.TestCase): else: yield - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux def test_input_tty(self): # Test input() functionality when wired to a tty self.check_input_tty("prompt", b"quux") - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux def test_input_tty_non_ascii(self): # Check stdin/stdout encoding is used when invoking PyOS_Readline() self.check_input_tty("prompté", b"quux\xc3\xa9", "utf-8") - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux def test_input_tty_non_ascii_unicode_errors(self): # Check stdin/stdout error handler is used when invoking PyOS_Readline() self.check_input_tty("prompté", b"quux\xe9", "ascii") - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux def test_input_tty_null_in_prompt(self): self.check_input_tty("prompt\0", b"", expected='ValueError: input: prompt string cannot contain ' 'null characters') - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux def test_input_tty_nonencodable_prompt(self): self.check_input_tty("prompté", b"quux", "ascii", stdout_errors='strict', expected="UnicodeEncodeError: 'ascii' codec can't encode " "character '\\xe9' in position 6: ordinal not in " "range(128)") - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux def test_input_tty_nondecodable_input(self): self.check_input_tty("prompt", b"quux\xe9", "ascii", stdin_errors='strict', expected="UnicodeDecodeError: 'ascii' codec can't decode " "byte 0xe9 in position 4: ordinal not in " "range(128)") - @unittest.skip("TODO: RUSTPYTHON; FAILURE, WORKER BUG") - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux def test_input_no_stdout_fileno(self): # Issue #24402: If stdin is the original terminal but stdout.fileno() # fails, do not use the original stdout file descriptor @@ -3036,7 +3028,6 @@ class TestType(unittest.TestCase): with self.assertRaises(TypeError): type('A', (B,), {'__slots__': '__weakref__'}) - @unittest.skip("TODO: RUSTPYTHON; random failure") def test_namespace_order(self): # bpo-34320: namespace should preserve order od = collections.OrderedDict([('a', 1), ('b', 2)]) diff --git a/Lib/test/test_dataclasses/__init__.py b/Lib/test/test_dataclasses/__init__.py index 962d75abc..acbbf60ef 100644 --- a/Lib/test/test_dataclasses/__init__.py +++ b/Lib/test/test_dataclasses/__init__.py @@ -5000,7 +5000,6 @@ class TestAbstract(unittest.TestCase): self.assertFalse(inspect.isabstract(Date)) self.assertGreater(Date(2020,12,25), Date(2020,8,31)) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_maintain_abc(self): class A(abc.ABC): @abc.abstractmethod diff --git a/Lib/test/test_float.py b/Lib/test/test_float.py index 609a7164f..dd0ab00d3 100644 --- a/Lib/test/test_float.py +++ b/Lib/test/test_float.py @@ -255,7 +255,6 @@ class GeneralFloatCases(unittest.TestCase): with self.assertRaisesRegex(TypeError, 'keyword argument'): float(x='3.14') - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Unexpected keyword argument newarg def test_keywords_in_subclass(self): class subclass(float): pass diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py index 4ad6e52da..431e58648 100644 --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -1238,6 +1238,10 @@ class TestCmpToKeyC(TestCmpToKey, unittest.TestCase): if c_functools: cmp_to_key = c_functools.cmp_to_key + @unittest.expectedFailure # TODO: RUSTPYTHON; inspect.signature reports (args, /) until FromArgs fields generate __text_signature__ + def test_cmp_to_signature(self): + super().test_cmp_to_signature() + @support.cpython_only def test_disallow_instantiation(self): # Ensure that the type disallows instantiation (bpo-43916) @@ -1245,38 +1249,6 @@ class TestCmpToKeyC(TestCmpToKey, unittest.TestCase): self, type(c_functools.cmp_to_key(None)) ) - @unittest.expectedFailure # TODO: RUSTPYTHON; + (mycmp) - def test_cmp_to_signature(self): - return super().test_cmp_to_signature() - - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() got multiple values for argument 'mycmp' - def test_cmp_to_key_arguments(self): - return super().test_cmp_to_key_arguments() - - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() got multiple values for argument 'mycmp' - def test_obj_field(self): - return super().test_obj_field() - - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given - def test_bad_cmp(self): - return super().test_bad_cmp() - - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given - def test_cmp_to_key(self): - return super().test_cmp_to_key() - - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given - def test_hash(self): - return super().test_hash() - - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given - def test_sort_int(self): - return super().test_sort_int() - - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given - def test_sort_int_str(self): - return super().test_sort_int_str() - class TestCmpToKeyPy(TestCmpToKey, unittest.TestCase): cmp_to_key = staticmethod(py_functools.cmp_to_key) @@ -1763,7 +1735,6 @@ class TestLRU: f(0, **{}) self.assertEqual(f.cache_info().hits, 1) - @unittest.expectedFailure # TODO: RUSTPYTHON; Python lru_cache impl doesn't cache hash like C impl def test_lru_hash_only_once(self): # To protect against weird reentrancy bugs and to improve # efficiency when faced with slow __hash__ methods, the @@ -2266,6 +2237,10 @@ class TestLRUPy(TestLRU, unittest.TestCase): def cached_staticmeth(x, y): return 3 * x + y + @unittest.expectedFailure # TODO: RUSTPYTHON; Python lru_cache impl doesn't cache hash like C impl + def test_lru_hash_only_once(self): + return super().test_lru_hash_only_once() + @unittest.skipUnless(c_functools, 'requires the C _functools module') class TestLRUC(TestLRU, unittest.TestCase): diff --git a/Lib/test/test_inspect/test_inspect.py b/Lib/test/test_inspect/test_inspect.py index 9dc80d08f..8d9d3537e 100644 --- a/Lib/test/test_inspect/test_inspect.py +++ b/Lib/test/test_inspect/test_inspect.py @@ -14,7 +14,7 @@ import linecache import os import dis from os.path import normcase -# import _pickle # TODO: RUSTPYTHON +import _pickle import pickle import shutil import stat @@ -123,7 +123,6 @@ class IsTestBase(unittest.TestCase): else: self.assertFalse(other(obj), 'not %s(%s)' % (other.__name__, exp)) - @unittest.expectedFailure # TODO: RUSTPYTHON; First has 0, Second has 1: 'iskeyword' def test__all__(self): support.check__all__(self, inspect, not_exported=("modulesbyfile",), extra=("get_annotations",)) @@ -176,7 +175,7 @@ class custom_descriptor: class TestPredicates(IsTestBase): - @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: type object 'defaultdict' has no attribute 'default_factory' + @unittest.expectedFailure # TODO: RUSTPYTHON; datetime.timedelta.days is a property until _datetime members exist def test_excluding_predicates(self): global tb self.istest(inspect.isbuiltin, 'sys.exit') @@ -480,7 +479,6 @@ class TestPredicates(IsTestBase): self.assertIn('a', members) self.assertNotIn('b', members) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: False is not true def test_isabstract(self): from abc import ABCMeta, abstractmethod @@ -503,7 +501,6 @@ class TestPredicates(IsTestBase): self.assertFalse(inspect.isabstract(int)) self.assertFalse(inspect.isabstract(5)) - @unittest.expectedFailure # TODO: RUSTPYTHON; + [True, False] def test_isabstract_during_init_subclass(self): from abc import ABCMeta, abstractmethod isabstract_checks = [] @@ -1366,7 +1363,7 @@ class TestClassesAndFunctions(unittest.TestCase): spec = inspect.getfullargspec(test) self.assertEqual(test.__annotations__, spec.annotations) - @unittest.expectedFailure # TODO: RUSTPYTHON; NameError: name '_pickle' is not defined. Did you mean: 'pickle'? Or did you forget to import '_pickle'? + @unittest.expectedFailure # TODO: RUSTPYTHON; os.stat signature is generated as (path, dir_fd, follow_symlinks, /) @unittest.skipIf(MISSING_C_DOCSTRINGS, "Signature information for builtins requires docstrings") def test_getfullargspec_builtin_methods(self): @@ -1531,7 +1528,6 @@ class TestClassesAndFunctions(unittest.TestCase): self.assertIn(('md', 'method', A), attrs, 'missing method descriptor') self.assertIn(('dd', 'data', A), attrs, 'missing data descriptor') - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ('to_bytes', 'method', ) not found in [('__abs__', 'method', ), ('__add__', 'method', ), ('__and__', 'method', ), ('__bool__', 'method', ), ('__ceil__', 'class method', ), ('__class__', 'data', ), ('__delattr__', 'class method', ), ('__dir__', 'class method', ), ('__divmod__', 'method', ), ('__doc__', 'data', ), ('__eq__', 'class method', ), ('__float__', 'method', ), ('__floor__', 'class method', ), ('__floordiv__', 'method', ), ('__format__', 'class method', ), ('__ge__', 'class method', ), ('__getattribute__', 'class method', ), ('__getnewargs__', 'class method', ), ('__getstate__', 'class method', ), ('__gt__', 'class method', ), ('__hash__', 'method', ), ('__index__', 'method', ), ('__init__', 'method', ), ('__init_subclass__', 'class method', ), ('__int__', 'method', ), ('__invert__', 'method', ), ('__le__', 'class method', ), ('__lshift__', 'method', ), ('__lt__', 'class method', ), ('__mod__', 'method', ), ('__mul__', 'method', ), ('__ne__', 'class method', ), ('__neg__', 'method', ), ('__new__', 'static method', ), ('__or__', 'method', ), ('__pos__', 'method', ), ('__pow__', 'method', ), ('__radd__', 'method', ), ('__rand__', 'method', ), ('__rdivmod__', 'method', ), ('__reduce__', 'class method', ), ('__reduce_ex__', 'class method', ), ('__repr__', 'method', ), ('__rfloordiv__', 'method', ), ('__rlshift__', 'method', ), ('__rmod__', 'method', ), ('__rmul__', 'method', ), ('__ror__', 'method', ), ('__round__', 'class method', ), ('__rpow__', 'method', ), ('__rrshift__', 'method', ), ('__rshift__', 'method', ), ('__rsub__', 'method', ), ('__rtruediv__', 'method', ), ('__rxor__', 'method', ), ('__setattr__', 'class method', ), ('__sizeof__', 'class method', ), ('__str__', 'method', ), ('__sub__', 'method', ), ('__subclasshook__', 'class method', ), ('__truediv__', 'method', ), ('__trunc__', 'class method', ), ('__xor__', 'method', ), ('as_integer_ratio', 'class method', ), ('bit_count', 'class method', ), ('bit_length', 'class method', ), ('conjugate', 'class method', ), ('denominator', 'data', ), ('from_bytes', 'class method', ), ('imag', 'data', ), ('is_integer', 'class method', ), ('numerator', 'data', ), ('real', 'data', ), ('to_bytes', 'class method', )] : missing plain method def test_classify_builtin_types(self): # Simple sanity check that all built-in types can have their # attributes classified. @@ -4210,7 +4206,6 @@ class TestSignatureObject(unittest.TestCase): self.assertEqual(self.signature(C.__call__, follow_wrapped=False), varargs_signature) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_on_class_with_wrapped_init(self): class C: @identity_wrapper @@ -4460,7 +4455,6 @@ class TestSignatureObject(unittest.TestCase): ((('a', ..., ..., "positional_or_keyword"),), ...)) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_on_class_with_new(self): with self.subTest('FunctionType'): class C: @@ -4621,7 +4615,7 @@ class TestSignatureObject(unittest.TestCase): with self.assertRaisesRegex(ValueError, "callable.*is not supported"): self.assertEqual(inspect.signature(D), None) - @unittest.expectedFailure # TODO: RUSTPYTHON; NameError: name '_pickle' is not defined. Did you mean: 'pickle'? Or did you forget to import '_pickle'? + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: no signature found for builtin type @unittest.skipIf(MISSING_C_DOCSTRINGS, "Signature information for builtins requires docstrings") def test_signature_on_builtin_class(self): @@ -4647,7 +4641,6 @@ class TestSignatureObject(unittest.TestCase): pass self.assertEqual(str(inspect.signature(P4)), '(foo, bar)') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_signature_on_callable_objects(self): class Foo: def __call__(self, a): @@ -5164,7 +5157,7 @@ class TestSignatureObject(unittest.TestCase): foo_sig = MySignature.from_callable(foo) self.assertIsInstance(foo_sig, MySignature) - @unittest.expectedFailure # TODO: RUSTPYTHON; NameError: name '_pickle' is not defined. Did you mean: 'pickle'? Or did you forget to import '_pickle'? + @unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: no signature found for builtin type @unittest.skipIf(MISSING_C_DOCSTRINGS, "Signature information for builtins requires docstrings") def test_signature_from_callable_builtin_obj(self): @@ -6335,7 +6328,6 @@ class TestSignatureDefinitions(unittest.TestCase): with self.assertRaises(ValueError): inspect.signature(func) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: None != '(raw, buffer_size=DEFAULT_BUFFER_SIZE)' @support.requires_docstrings def test_base_class_have_text_signature(self): # see issue 43118 @@ -6567,7 +6559,6 @@ class TestRepl(unittest.TestCase): raise ValueError("Process didn't exit properly.") return output - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 'The source is: <<>>' not found in 'Traceback (most recent call last):\n File "", line 1, in \n File "crates/pylib/Lib/inspect.py", line 1161, in getsource\n lines, lnum = getsourcelines(object)\n ~~~~~~~~~~~~~~^^^^^^^^\n File "crates/pylib/Lib/inspect.py", line 1143, in getsourcelines\n lines, lnum = findsource(object)\n ~~~~~~~~~~^^^^^^^^\n File "crates/pylib/Lib/inspect.py", line 978, in findsource\n raise OSError(\'could not get source code\')\nOSError: could not get source code\n' @unittest.skipIf(not has_subprocess_support, "test requires subprocess") def test_getsource(self): output = self.run_on_interactive_mode(textwrap.dedent("""\ diff --git a/Lib/test/test_itertools.py b/Lib/test/test_itertools.py index c1695690b..cf579d4da 100644 --- a/Lib/test/test_itertools.py +++ b/Lib/test/test_itertools.py @@ -754,7 +754,6 @@ class TestBasicOps(unittest.TestCase): next(g) next(g) # must pass with address sanitizer - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 4 != 1 def test_grouper_reentrant_eq_does_not_crash(self): # regression test for gh-146613 grouper_iter = None @@ -1167,7 +1166,6 @@ class TestBasicOps(unittest.TestCase): self.assertRaises(ValueError, next, starmap(errfunc, [(4,5)])) self.assertRaises(TypeError, next, starmap(onearg, [(4,5)])) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_islice(self): for args in [ # islice(args) should agree with range(args) (10, 20, 3), @@ -2493,7 +2491,6 @@ class RegressionTests(unittest.TestCase): class SubclassWithKwargsTest(unittest.TestCase): - @unittest.expectedFailure # TODO: RUSTPYTHON def test_keywords_in_subclass(self): # count is not subclassable... testcases = [ diff --git a/Lib/test/test_list.py b/Lib/test/test_list.py index 82e6056e7..4d5cb11bd 100644 --- a/Lib/test/test_list.py +++ b/Lib/test/test_list.py @@ -49,7 +49,6 @@ class ListTest(list_tests.CommonTest): with self.assertRaisesRegex(TypeError, 'keyword argument'): list(sequence=[]) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_keywords_in_subclass(self): class subclass(list): pass diff --git a/Lib/test/test_marshal.py b/Lib/test/test_marshal.py index eb2e8bda9..9f5cef9be 100644 --- a/Lib/test/test_marshal.py +++ b/Lib/test/test_marshal.py @@ -493,7 +493,6 @@ class BugsTestCase(unittest.TestCase): unicode_string = 'T' self.assertRaises(TypeError, marshal.loads, unicode_string) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_bad_reader(self): class BadReader(io.BytesIO): def readinto(self, buf): @@ -510,7 +509,7 @@ class BugsTestCase(unittest.TestCase): for i in range(len(data)): self.assertRaises(EOFError, marshal.loads, data[0: i]) - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; set iteration is insertion order, so PYTHONHASHSEED does not scramble repr def test_deterministic_sets(self): # bpo-37596: To support reproducible builds, sets and frozensets need to # have their elements serialized in a consistent order (even when they diff --git a/Lib/test/test_ordered_dict.py b/Lib/test/test_ordered_dict.py index 01678de56..04942effa 100644 --- a/Lib/test/test_ordered_dict.py +++ b/Lib/test/test_ordered_dict.py @@ -805,7 +805,6 @@ class CPythonOrderedDictSideEffects: msg = re.escape("OrderedDict mutated during iteration") self.assertRaisesRegex(RuntimeError, msg, operator.eq, dict1, dict2) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeError not raised by eq def test_issue119004_change_size_by_clear(self): class Key(_TriggerSideEffectOnEqual): def side_effect(self): @@ -818,7 +817,6 @@ class CPythonOrderedDictSideEffects: self.assertDictEqual(dict1, {}) self.assertDictEqual(dict2, dict.fromkeys((0, Key(), 4.2))) - @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'NoneType' object has no attribute 'key' def test_issue119004_change_size_by_delete_key(self): class Key(_TriggerSideEffectOnEqual): def side_effect(self): @@ -832,7 +830,6 @@ class CPythonOrderedDictSideEffects: self.assertDictEqual(dict1, dict.fromkeys((0, 4.2))) self.assertDictEqual(dict2, dict.fromkeys((0, Key(), 4.2))) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeError not raised by eq def test_issue119004_change_linked_list_by_clear(self): class Key(_TriggerSideEffectOnEqual): def side_effect(self): @@ -846,7 +843,6 @@ class CPythonOrderedDictSideEffects: self.assertDictEqual(dict1, dict.fromkeys(('a', 'b'), 'c')) self.assertDictEqual(dict2, dict.fromkeys((0, Key(), 4.2))) - @unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'NoneType' object has no attribute 'key' def test_issue119004_change_linked_list_by_delete_key(self): class Key(_TriggerSideEffectOnEqual): def side_effect(self): @@ -861,7 +857,6 @@ class CPythonOrderedDictSideEffects: self.assertDictEqual(dict1, {0: None, 'a': 'c', 4.2: None}) self.assertDictEqual(dict2, dict.fromkeys((0, Key(), 4.2))) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 1 != 2 def test_issue119004_change_size_by_delete_key_in_dict_eq(self): class Key(_TriggerSideEffectOnEqual): trigger = 0 @@ -878,7 +873,6 @@ class CPythonOrderedDictSideEffects: self.assertDictEqual(dict1, dict.fromkeys((0, 4.2))) self.assertDictEqual(dict2, dict.fromkeys((0, Key(), 4.2))) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeError not raised by copy def test_issue148660_copy_clear_in_key_eq(self): # gh-148660: od.copy() must not crash when a key's __eq__ clears od # while copy() is inserting into the new dict. @@ -901,7 +895,6 @@ class CPythonOrderedDictSideEffects: msg = "OrderedDict mutated during iteration" self.assertRaisesRegex(RuntimeError, msg, od.copy) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeError not raised by copy def test_issue148660_copy_clear_in_subclass_getitem(self): # gh-148660: od.copy() must not crash when a subclass __getitem__ # clears od. @@ -956,7 +949,6 @@ class CPythonOrderedDictTests(OrderedDictTests, check(iter(od.items()), itersize) check(iter(od.values()), itersize) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeError not raised def test_key_change_during_iteration(self): OrderedDict = self.OrderedDict @@ -974,7 +966,6 @@ class CPythonOrderedDictTests(OrderedDictTests, del od['c'] self.assertEqual(list(od), list('bdeaf')) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_iterators_pickling(self): OrderedDict = self.OrderedDict pairs = [('c', 1), ('b', 2), ('a', 3), ('d', 4), ('e', 5), ('f', 6)] diff --git a/Lib/test/test_pydoc/test_pydoc.py b/Lib/test/test_pydoc/test_pydoc.py index dcb42a3a6..eeea10101 100644 --- a/Lib/test/test_pydoc/test_pydoc.py +++ b/Lib/test/test_pydoc/test_pydoc.py @@ -1573,7 +1573,7 @@ class TestDescriptions(unittest.TestCase): "dump(obj, /) method of _pickle.Pickler instance") # this should *never* include self! - @unittest.expectedFailure # TODO: RUSTPYTHON + @unittest.expectedFailure # TODO: RUSTPYTHON; os.stat signature is generated as (path, dir_fd, follow_symlinks, /) @requires_docstrings def test_module_level_callable(self): self.assertEqual(self._get_summary_line(os.stat), @@ -1994,7 +1994,6 @@ class PydocFodderTest(unittest.TestCase): self.assertIn(' | B_classmethod(x)', lines) self.assertIn(' | B_classmethod_alias = B_classmethod(x)', lines) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_html_doc_routines_in_class(self, cls=pydocfodder.B): doc = pydoc.HTMLDoc() result = doc.docclass(cls) @@ -2033,7 +2032,6 @@ class PydocFodderTest(unittest.TestCase): def test_text_doc_inherited_routines_in_class(self): self.test_text_doc_routines_in_class(pydocfodder.D) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_html_doc_inherited_routines_in_class(self): self.test_html_doc_routines_in_class(pydocfodder.D) diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index 21c846835..80c21f6b1 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -2939,7 +2939,6 @@ class ImplementationTest(unittest.TestCase): pat = re.compile("") check_disallow_instantiation(self, type(pat.scanner(""))) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_deprecated_modules(self): deprecated = { 'sre_compile': ['compile', 'error', diff --git a/Lib/test/test_repl.py b/Lib/test/test_repl.py index c80db8323..71dab8c38 100644 --- a/Lib/test/test_repl.py +++ b/Lib/test/test_repl.py @@ -207,7 +207,6 @@ class TestInteractiveInterpreter(unittest.TestCase): ] self.assertEqual(traceback_lines, expected_lines) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 101 != 0 def test_interactive_traceback_reporting_multiple_input(self): user_input1 = dedent(""" def foo(x): @@ -308,7 +307,6 @@ class TestInteractiveInterpreter(unittest.TestCase): ] self.assertEqual(output.splitlines()[4:-1], expected_lines) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 101 != 0 def test_interactive_source_is_in_linecache(self): user_input = dedent(""" def foo(x): diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py index 615cc1c65..0e20723b6 100644 --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -943,7 +943,6 @@ def requireSocket(*args): class GeneralModuleTests(unittest.TestCase): - @unittest.expectedFailure # TODO: RUSTPYTHON; gc.is_tracked not implemented @unittest.skipUnless(_socket is not None, 'need _socket module') def test_socket_type(self): self.assertTrue(gc.is_tracked(_socket.socket)) @@ -1007,7 +1006,6 @@ class GeneralModuleTests(unittest.TestCase): with self.assertRaises(OSError, msg=msg % 'socket.gaierror'): raise socket.gaierror - @unittest.expectedFailure # TODO: RUSTPYTHON; error message format differs def testSendtoErrors(self): # Testing that sendto doesn't mask failures. See #10169. s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) @@ -2230,7 +2228,6 @@ class GeneralModuleTests(unittest.TestCase): source=_socket) enum._test_simple_enum(CheckedAddressInfo, socket.AddressInfo) - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Expected type 'int' but 'Mut' found @unittest.skipUnless(hasattr(socket.socket, "sendmsg"),"sendmsg not supported") def test_sendmsg_reentrant_ancillary_mutation(self): @@ -2448,14 +2445,12 @@ class ISOTPTest(unittest.TestCase): with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_ISOTP) as s: pass - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: AF_CAN address must be a tuple (interface,) or (interface, addr) def testTooLongInterfaceName(self): # most systems limit IFNAMSIZ to 16, take 1024 to be sure with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_ISOTP) as s: with self.assertRaisesRegex(OSError, 'interface name too long'): s.bind(('x' * 1024, 1, 2)) - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: AF_CAN address must be a tuple (interface,) or (interface, addr) def testBind(self): try: with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_ISOTP) as s: @@ -2477,7 +2472,6 @@ class J1939Test(unittest.TestCase): super().__init__(*args, **kwargs) self.interface = "vcan0" - @unittest.expectedFailure # TODO: RUSTPYTHON; - J1939 constants not fully implemented @unittest.skipUnless(hasattr(socket, "CAN_J1939"), 'socket.CAN_J1939 required for this test.') def testJ1939Constants(self): @@ -2519,7 +2513,6 @@ class J1939Test(unittest.TestCase): with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_J1939) as s: pass - @unittest.expectedFailure # TODO: RUSTPYTHON; - AF_CAN J1939 address format not fully implemented def testBind(self): try: with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_J1939) as s: @@ -2698,9 +2691,6 @@ class BasicVSOCKTest(unittest.TestCase): 'Bluetooth sockets required for this test.') class BasicBluetoothTest(unittest.TestCase): - # Windows publishes the names below. Other platforms still miss - # BTPROTO_HCI and the rest of the non-Windows set. - @(unittest.expectedFailure if sys.platform != "win32" else (lambda func: func)) def testBluetoothConstants(self): socket.BDADDR_ANY socket.BDADDR_LOCAL @@ -7519,7 +7509,6 @@ class ReentrantMutationTests(unittest.TestCase): See: https://github.com/python/cpython/issues/143988 """ - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: b'Hello' != b'HelloWorldTest' @unittest.skipUnless(hasattr(socket.socket, "sendmsg"), "sendmsg not supported") def test_sendmsg_reentrant_data_mutation(self): diff --git a/Lib/test/test_struct.py b/Lib/test/test_struct.py index a570ec098..e18bad9c9 100644 --- a/Lib/test/test_struct.py +++ b/Lib/test/test_struct.py @@ -847,7 +847,6 @@ class StructTest(ComplexesAreIdenticalMixin, unittest.TestCase): s = struct.Struct('=i2H') self.assertEqual(repr(s), f'Struct({s.format!r})') - @unittest.expectedFailure # TODO: RUSTPYTHON def test_c_complex_round_trip(self): values = [complex(*_) for _ in combinations([1, -1, 0.0, -0.0, 2, -3, INF, -INF, NAN], 2)] @@ -874,7 +873,6 @@ class StructTest(ComplexesAreIdenticalMixin, unittest.TestCase): results = executor.map(exec, [code] * 5) self.assertListEqual(list(results), [None] * 5) - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: expected at least 1 arguments, got 0 def test_operations_on_half_initialized_Struct(self): S = struct.Struct.__new__(struct.Struct) diff --git a/Lib/test/test_structseq.py b/Lib/test/test_structseq.py index d4014a784..74506fc54 100644 --- a/Lib/test/test_structseq.py +++ b/Lib/test/test_structseq.py @@ -286,7 +286,6 @@ class StructSeqTest(unittest.TestCase): with self.assertRaisesRegex(TypeError, 'unexpected field name'): copy.replace(t, user=1, error=-1) - @unittest.expectedFailure # TODO: RUSTPYTHON; Wrong error message def test_copy_replace_with_invisible_fields(self): assert time.struct_time.n_unnamed_fields == 0 assert time.struct_time.n_sequence_fields < time.struct_time.n_fields diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 71c2f7512..f30b609af 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -209,7 +209,6 @@ class SysModuleTest(unittest.TestCase): def tearDown(self): test.support.reap_children() - @unittest.expectedFailure # TODO: RUSTPYTHON; latin-1 codec not registered def test_exit(self): # call with two arguments self.assertRaises(TypeError, sys.exit, 42, 42) @@ -1054,12 +1053,10 @@ class SysModuleTest(unittest.TestCase): 'stdout: surrogateescape\n' 'stderr: backslashreplace\n') - @unittest.expectedFailure # TODO: RUSTPYTHON; iso8859_1 codec not registered @support.requires_subprocess() def test_c_locale_surrogateescape(self): self.check_locale_surrogateescape('C') - @unittest.expectedFailure # TODO: RUSTPYTHON; iso8859_1 codec not registered @support.requires_subprocess() def test_posix_locale_surrogateescape(self): self.check_locale_surrogateescape('POSIX') diff --git a/Lib/test/test_threading.py b/Lib/test/test_threading.py index bd8145f22..605dffa52 100644 --- a/Lib/test/test_threading.py +++ b/Lib/test/test_threading.py @@ -1018,7 +1018,6 @@ class ThreadTests(BaseTestCase): finally: threading.settrace(old_trace) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_gettrace_all_threads(self): def fn(*args): pass old_trace = threading.gettrace() @@ -1057,7 +1056,6 @@ class ThreadTests(BaseTestCase): finally: threading.setprofile(old_profile) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_getprofile_all_threads(self): def fn(*args): pass old_profile = threading.getprofile() diff --git a/Lib/test/test_trace.py b/Lib/test/test_trace.py index f1b3a3827..d0e51c5c3 100644 --- a/Lib/test/test_trace.py +++ b/Lib/test/test_trace.py @@ -375,7 +375,6 @@ class TestCoverage(unittest.TestCase): r = tracer.results() r.write_results(show_missing=True, summary=True, coverdir=TESTFN) - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 'pprint.py' not found in '' @requires_resource('cpu') def test_coverage(self): tracer = trace.Trace(trace=0, count=1) @@ -400,7 +399,6 @@ class TestCoverage(unittest.TestCase): files = os.listdir(TESTFN) self.assertEqual(files, ['_importlib.cover']) # Ignore __import__ - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 'test.tracedmodules.testmod' not found in {} def test_issue9936(self): tracer = trace.Trace(trace=0, count=1) modname = 'test.tracedmodules.testmod' diff --git a/Lib/test/test_tuple.py b/Lib/test/test_tuple.py index a97ca8c9a..e533392b8 100644 --- a/Lib/test/test_tuple.py +++ b/Lib/test/test_tuple.py @@ -42,7 +42,6 @@ class TupleTest(seq_tests.CommonTest): with self.assertRaisesRegex(TypeError, 'keyword argument'): tuple(sequence=()) - @unittest.expectedFailure # TODO: RUSTPYTHON def test_keywords_in_subclass(self): class subclass(tuple): pass diff --git a/Lib/test/test_warnings/__init__.py b/Lib/test/test_warnings/__init__.py index bd7e3ec3e..bbb64faa4 100644 --- a/Lib/test/test_warnings/__init__.py +++ b/Lib/test/test_warnings/__init__.py @@ -607,7 +607,6 @@ class WarnTests(BaseTest): self.assertEqual(os.path.basename(w[-1].filename), "") - @unittest.expectedFailure # TODO: RUSTPYTHON; + crates/pylib/Lib/test/test_warnings/__init__.py def test_stacklevel_import(self): # Issue #24305: With stacklevel=2, module-level warnings should work. import_helper.unload('test.test_warnings.data.import_warning') @@ -853,38 +852,6 @@ class CWarnTests(WarnTests, unittest.TestCase): self.assertIsNot(original_warnings, self.module) self.assertNotHasAttr(self.module.warn, '__code__') - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 1 != 2 - def test_gh86298_loader_and_spec_loader_disagree(self): - return super().test_gh86298_loader_and_spec_loader_disagree() - - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 1 != 2 - def test_gh86298_no_spec(self): - return super().test_gh86298_no_spec() - - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 1 != 2 - def test_gh86298_no_spec_loader(self): - return super().test_gh86298_no_spec_loader() - - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 1 != 2 - def test_gh86298_spec_is_none(self): - return super().test_gh86298_spec_is_none() - - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: AttributeError not raised - def test_gh86298_no_loader_and_no_spec_loader(self): - return super().test_gh86298_no_loader_and_no_spec_loader() - - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ValueError not raised - def test_gh86298_loader_is_none_and_spec_is_none(self): - return super().test_gh86298_loader_is_none_and_spec_is_none() - - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ValueError not raised - def test_gh86298_loader_is_none_and_spec_loader_is_none(self): - return super().test_gh86298_loader_is_none_and_spec_loader_is_none() - - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ValueError not raised - def test_gh86298_no_loader_and_spec_is_none(self): - return super().test_gh86298_no_loader_and_spec_is_none() - class PyWarnTests(WarnTests, unittest.TestCase): module = py_warnings @@ -960,7 +927,6 @@ class _WarningsTests(BaseTest, unittest.TestCase): module = c_warnings - @unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: UserWarning not raised by warn def test_filter(self): # Everything should function even if 'filters' is not in warnings. with self.module.catch_warnings() as w: @@ -1130,7 +1096,6 @@ class _WarningsTests(BaseTest, unittest.TestCase): self.assertNotIn(b'Warning!', stderr) self.assertNotIn(b'Error', stderr) - @unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: 'int' object is not iterable def test_issue31285(self): # warn_explicit() should neither raise a SystemError nor cause an # assertion failure, in case the return value of get_source() has a diff --git a/Lib/test/test_zipfile/test_core.py b/Lib/test/test_zipfile/test_core.py index 23c6b0cb0..f9ad28ea0 100644 --- a/Lib/test/test_zipfile/test_core.py +++ b/Lib/test/test_zipfile/test_core.py @@ -3551,7 +3551,6 @@ class TestExecutablePrependedZip(unittest.TestCase): def test_read_zip64_with_exe_prepended(self): self._test_zip_works(self.exe_zip64) - @unittest.expectedFailure # TODO: RUSTPYTHON @unittest.skipUnless(sys.executable, 'sys.executable required.') @unittest.skipUnless(os.access('/bin/bash', os.X_OK), 'Test relies on #!/bin/bash working.') @@ -3560,7 +3559,6 @@ class TestExecutablePrependedZip(unittest.TestCase): output = subprocess.check_output([self.exe_zip, sys.executable]) self.assertIn(b'number in executable: 5', output) - @unittest.expectedFailure # TODO: RUSTPYTHON @unittest.skipUnless(sys.executable, 'sys.executable required.') @unittest.skipUnless(os.access('/bin/bash', os.X_OK), 'Test relies on #!/bin/bash working.') diff --git a/Lib/test/test_zoneinfo/test_zoneinfo.py b/Lib/test/test_zoneinfo/test_zoneinfo.py index 638b4c52a..98f76d4cd 100644 --- a/Lib/test/test_zoneinfo/test_zoneinfo.py +++ b/Lib/test/test_zoneinfo/test_zoneinfo.py @@ -1866,7 +1866,6 @@ class TzPathTest(TzPathUserMixin, ZoneInfoTestBase): with self.subTest("filtered", path_var=path_var): self.assertSequenceEqual(tzpath, expected_paths) - @unittest.expectedFailure # TODO: RUSTPYTHON; + /home/runner/work/RustPython/RustPython/crates/pylib/Lib/test/test_zoneinfo/test_zoneinfo.py def test_env_variable_relative_paths_warning_location(self): path_var = "path/to/somewhere" diff --git a/crates/capi/src/setobject.rs b/crates/capi/src/setobject.rs index 1036bb147..6ccbddbbd 100644 --- a/crates/capi/src/setobject.rs +++ b/crates/capi/src/setobject.rs @@ -66,9 +66,9 @@ pub unsafe extern "C" fn PySet_Contains(anyset: *mut PyObject, key: *mut PyObjec let key = unsafe { &*key }; if let Some(set) = anyset.downcast_ref::() { - set.__contains__(key, vm) + set.contains(key, vm) } else if let Some(frozenset) = anyset.downcast_ref::() { - frozenset.__contains__(key, vm) + frozenset.contains(key, vm) } else { Err(vm.new_type_error(format!( "expected set or frozenset, got '{}'", @@ -83,7 +83,7 @@ pub unsafe extern "C" fn PySet_Discard(set: *mut PyObject, key: *mut PyObject) - with_vm(|vm| { let set = unsafe { &*set }.try_downcast_ref::(vm)?; let key = unsafe { &*key }; - let had_item = set.__contains__(key, vm)?; + let had_item = set.contains(key, vm)?; if had_item { set.discard(key.to_owned(), vm)?; } diff --git a/crates/common/src/encodings/cjk/iso2022.rs b/crates/common/src/encodings/cjk/iso2022.rs index 93010513e..4ab7bc750 100644 --- a/crates/common/src/encodings/cjk/iso2022.rs +++ b/crates/common/src/encodings/cjk/iso2022.rs @@ -36,12 +36,29 @@ const CHARSET_JISX0213_2: u8 = b'P' | CHARSET_DBCS; const CHARSET_JISX0213_2004_1: u8 = b'Q' | CHARSET_DBCS; const CHARSET_JISX0208_O: u8 = b'@' | CHARSET_DBCS; -const F_SHIFTED: u8 = 0x01; -const F_ESCTHROUGHOUT: u8 = 0x02; +bitflags::bitflags! { + #[derive(Copy, Clone, Debug, PartialEq, Eq)] + struct Iso2022StateFlags: u8 { + const SHIFTED = 0x01; + const ESCTHROUGHOUT = 0x02; + } +} -const NO_SHIFT: u8 = 0x01; -const USE_G2: u8 = 0x02; -const USE_JISX0208_EXT: u8 = 0x04; +bitflags::bitflags! { + #[derive(Copy, Clone, Debug, PartialEq, Eq)] + struct Iso2022ConfigFlags: u8 { + const NO_SHIFT = 0x01; + const USE_G2 = 0x02; + const USE_JISX0208_EXT = 0x04; + } +} + +const F_SHIFTED: u8 = Iso2022StateFlags::SHIFTED.bits(); +const F_ESCTHROUGHOUT: u8 = Iso2022StateFlags::ESCTHROUGHOUT.bits(); + +const NO_SHIFT: u8 = Iso2022ConfigFlags::NO_SHIFT.bits(); +const USE_G2: u8 = Iso2022ConfigFlags::USE_G2.bits(); +const USE_JISX0208_EXT: u8 = Iso2022ConfigFlags::USE_JISX0208_EXT.bits(); #[derive(Clone, Copy)] struct Designation { diff --git a/crates/compiler-core/src/bytecode.rs b/crates/compiler-core/src/bytecode.rs index 2d02e6939..641601d92 100644 --- a/crates/compiler-core/src/bytecode.rs +++ b/crates/compiler-core/src/bytecode.rs @@ -425,15 +425,40 @@ impl IndexMut for [T] { } } -/// Per-slot kind flags for localsplus (co_localspluskinds). -pub const CO_FAST_ARG_POS: u8 = 0x02; -pub const CO_FAST_ARG_KW: u8 = 0x04; -pub const CO_FAST_ARG_VAR: u8 = 0x08; -pub const CO_FAST_ARG: u8 = CO_FAST_ARG_POS | CO_FAST_ARG_KW | CO_FAST_ARG_VAR; -pub const CO_FAST_HIDDEN: u8 = 0x10; -pub const CO_FAST_LOCAL: u8 = 0x20; -pub const CO_FAST_CELL: u8 = 0x40; -pub const CO_FAST_FREE: u8 = 0x80; +bitflagset::bitflag! { + /// Per-slot kind flags for localsplus (`co_localspluskinds`). + /// Values are bit *positions* (`Local` is bit 5 → mask `0x20`). + #[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)] + #[repr(u8)] + pub enum CoFastFlag { + ArgPos = 1, + ArgKw = 2, + ArgVar = 3, + Hidden = 4, + Local = 5, + Cell = 6, + Free = 7, + } +} + +bitflagset::bitflagset! { + #[derive(Copy, Clone, PartialEq, Eq)] + pub struct CoFastFlags(u8): CoFastFlag +} + +impl CoFastFlags { + pub const ARG: Self = + Self::from_slice(&[CoFastFlag::ArgPos, CoFastFlag::ArgKw, CoFastFlag::ArgVar]); +} + +pub const CO_FAST_ARG_POS: u8 = CoFastFlags::from_element(CoFastFlag::ArgPos).bits(); +pub const CO_FAST_ARG_KW: u8 = CoFastFlags::from_element(CoFastFlag::ArgKw).bits(); +pub const CO_FAST_ARG_VAR: u8 = CoFastFlags::from_element(CoFastFlag::ArgVar).bits(); +pub const CO_FAST_ARG: u8 = CoFastFlags::ARG.bits(); +pub const CO_FAST_HIDDEN: u8 = CoFastFlags::from_element(CoFastFlag::Hidden).bits(); +pub const CO_FAST_LOCAL: u8 = CoFastFlags::from_element(CoFastFlag::Local).bits(); +pub const CO_FAST_CELL: u8 = CoFastFlags::from_element(CoFastFlag::Cell).bits(); +pub const CO_FAST_FREE: u8 = CoFastFlags::from_element(CoFastFlag::Free).bits(); /// Primary container of a single code object. Each python function has /// a code object. Also a module has a code object. diff --git a/crates/compiler-core/src/marshal.rs b/crates/compiler-core/src/marshal.rs index acc45e75c..1615c1d6a 100644 --- a/crates/compiler-core/src/marshal.rs +++ b/crates/compiler-core/src/marshal.rs @@ -1616,10 +1616,6 @@ pub fn read_pstring(rdr: &mut R) -> Result<&[u8]> { rdr.read_slice(n as u32) } -const CO_FAST_LOCAL: u8 = 0x20; -const CO_FAST_CELL: u8 = 0x40; -const CO_FAST_FREE: u8 = 0x80; - pub struct LocalsPlusResult { pub varnames: Vec, pub cellvars: Vec, @@ -1645,10 +1641,11 @@ pub fn split_localplus( // First pass: collect varnames (LOCAL entries) and freevars for (name, &kind) in names.iter().zip(kinds.iter()) { - if kind & CO_FAST_LOCAL != 0 { + let kind = CoFastFlags::from_bits_retain(kind); + if kind.contains(&CoFastFlag::Local) { varnames.push(name.clone()); } - if kind & CO_FAST_FREE != 0 { + if kind.contains(&CoFastFlag::Free) { freevars.push(name.clone()); } } @@ -1659,8 +1656,9 @@ pub fn split_localplus( // This preserves the original ordering from localsplusnames. let mut arg_cell_positions = Vec::new(); // (cell_idx, localplus_idx) for (i, (name, &kind)) in names.iter().zip(kinds.iter()).enumerate() { - let is_local = kind & CO_FAST_LOCAL != 0; - let is_cell = kind & CO_FAST_CELL != 0; + let kind = CoFastFlags::from_bits_retain(kind); + let is_local = kind.contains(&CoFastFlag::Local); + let is_cell = kind.contains(&CoFastFlag::Cell); if is_cell { let cell_idx = cellvars.len(); cellvars.push(name.clone()); @@ -1701,7 +1699,7 @@ pub fn split_localplus( let mut deref_map = alloc::vec![u32::MAX; names.len()]; let mut cell_idx = 0u32; for (i, &kind) in kinds.iter().enumerate() { - if kind & CO_FAST_CELL != 0 { + if CoFastFlags::from_bits_retain(kind).contains(&CoFastFlag::Cell) { deref_map[i] = cell_idx; cell_idx += 1; } @@ -1709,7 +1707,7 @@ pub fn split_localplus( let ncells = cellvars.len(); let mut free_idx = 0u32; for (i, &kind) in kinds.iter().enumerate() { - if kind & CO_FAST_FREE != 0 { + if CoFastFlags::from_bits_retain(kind).contains(&CoFastFlag::Free) { deref_map[i] = ncells as u32 + free_idx; free_idx += 1; } diff --git a/crates/host_env/Cargo.toml b/crates/host_env/Cargo.toml index 8b3f0dd37..20578a710 100644 --- a/crates/host_env/Cargo.toml +++ b/crates/host_env/Cargo.toml @@ -29,6 +29,7 @@ rustls = [ rustpython-wtf8 = { workspace = true } bitflags = { workspace = true } +bitflagset = { workspace = true } der = { workspace = true, optional = true } getrandom = { workspace = true } libc = { workspace = true } diff --git a/crates/host_env/src/io.rs b/crates/host_env/src/io.rs index 1b205c252..134e99df1 100644 --- a/crates/host_env/src/io.rs +++ b/crates/host_env/src/io.rs @@ -9,16 +9,29 @@ use rustix::{fs::FileType, io::Errno}; use crate::fileutils; use crate::{crt_fd, os}; -bitflags::bitflags! { +bitflagset::bitflag! { #[derive(Copy, Clone, Debug, PartialEq, Eq)] - pub struct FileMode: u8 { - const CREATED = 0b0001; - const READABLE = 0b0010; - const WRITABLE = 0b0100; - const APPENDING = 0b1000; + #[repr(u8)] + pub enum FileModeFlag { + Created = 0, + Readable = 1, + Writable = 2, + Appending = 3, } } +bitflagset::bitflagset! { + #[derive(Copy, Clone, PartialEq, Eq)] + pub struct FileMode(u8): FileModeFlag +} + +impl FileMode { + pub const CREATED: Self = Self::from_element(FileModeFlag::Created); + pub const READABLE: Self = Self::from_element(FileModeFlag::Readable); + pub const WRITABLE: Self = Self::from_element(FileModeFlag::Writable); + pub const APPENDING: Self = Self::from_element(FileModeFlag::Appending); +} + #[derive(Clone, Copy, Debug, Eq, PartialEq)] pub enum FileModeError { Invalid, @@ -45,20 +58,20 @@ pub struct ParsedFileMode { impl FileMode { pub const fn raw_mode(self) -> &'static str { - if self.contains(Self::CREATED) { - if self.contains(Self::READABLE) { + if self.contains(&FileModeFlag::Created) { + if self.contains(&FileModeFlag::Readable) { "xb+" } else { "xb" } - } else if self.contains(Self::APPENDING) { - if self.contains(Self::READABLE) { + } else if self.contains(&FileModeFlag::Appending) { + if self.contains(&FileModeFlag::Readable) { "ab+" } else { "ab" } - } else if self.contains(Self::READABLE) { - if self.contains(Self::WRITABLE) { + } else if self.contains(&FileModeFlag::Readable) { + if self.contains(&FileModeFlag::Writable) { "rb+" } else { "rb" @@ -81,7 +94,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result { @@ -89,14 +102,14 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result { if rwa { return Err(FileModeError::BadRwa); } rwa = true; - mode.insert(FileMode::WRITABLE); + mode |= FileMode::WRITABLE; flags |= os::O_CREAT | os::O_TRUNC; } b'a' => { @@ -104,7 +117,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result { @@ -112,7 +125,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result {} _ => return Err(FileModeError::Invalid), @@ -123,9 +136,9 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result &'static str { - if self.contains(Self::CREATED) { - if self.contains(Self::READABLE) { + if self.contains(&FileModeFlag::Created) { + if self.contains(&FileModeFlag::Readable) { "xb+" } else { "xb" } - } else if self.contains(Self::APPENDING) { - if self.contains(Self::READABLE) { + } else if self.contains(&FileModeFlag::Appending) { + if self.contains(&FileModeFlag::Readable) { "ab+" } else { "ab" } - } else if self.contains(Self::READABLE) { - if self.contains(Self::WRITABLE) { + } else if self.contains(&FileModeFlag::Readable) { + if self.contains(&FileModeFlag::Writable) { "rb+" } else { "rb" @@ -78,7 +91,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result { @@ -86,14 +99,14 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result { if rwa { return Err(FileModeError::BadRwa); } rwa = true; - mode.insert(FileMode::WRITABLE); + mode |= FileMode::WRITABLE; flags |= O_CREAT | O_TRUNC; } b'a' => { @@ -101,7 +114,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result { @@ -109,7 +122,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result { if binary { @@ -125,9 +138,9 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result Self { Self { raw } diff --git a/crates/host_env/src/os.rs b/crates/host_env/src/os.rs index 6678572bf..6eda6030c 100644 --- a/crates/host_env/src/os.rs +++ b/crates/host_env/src/os.rs @@ -30,10 +30,25 @@ use { }, }; -pub const F_OK: u8 = 0; -pub const R_OK: u8 = 4; -pub const W_OK: u8 = 2; -pub const X_OK: u8 = 1; +bitflagset::bitflag! { + #[derive(Copy, Clone, Debug, PartialEq, Eq)] + #[repr(u8)] + pub enum AccessFlag { + X = 0, + W = 1, + R = 2, + } +} + +bitflagset::bitflagset! { + #[derive(Copy, Clone, PartialEq, Eq)] + pub struct AccessMode(u8): AccessFlag +} + +pub const F_OK: u8 = AccessMode::empty().bits(); +pub const X_OK: u8 = AccessMode::from_element(AccessFlag::X).bits(); +pub const W_OK: u8 = AccessMode::from_element(AccessFlag::W).bits(); +pub const R_OK: u8 = AccessMode::from_element(AccessFlag::R).bits(); #[cfg(any(unix, target_os = "wasi"))] pub use libc::AT_FDCWD; @@ -158,51 +173,91 @@ mod wasm_oflag { #[cfg(all(target_arch = "wasm32", target_os = "unknown"))] pub use wasm_oflag::*; -/// BSD `chflags` bits the `libc` crate does not bind. Published by `_stat` -/// on every platform. -pub const UF_SETTABLE: u32 = 0x0000ffff; -pub const UF_NOUNLINK: u32 = 0x00000010; -pub const UF_TRACKED: u32 = 0x00000040; -pub const UF_DATAVAULT: u32 = 0x00000080; -pub const SF_NOUNLINK: u32 = 0x00100000; -pub const SF_SNAPSHOT: u32 = 0x00200000; -pub const SF_FIRMLINK: u32 = 0x00800000; -pub const SF_DATALESS: u32 = 0x40000000; -pub const SF_SETTABLE: u32 = if cfg!(target_os = "macos") { - 0x3fff0000 -} else { - 0xffff0000 -}; -#[cfg(target_os = "macos")] -pub const SF_SUPPORTED: u32 = 0x009f0000; -#[cfg(target_os = "macos")] -pub const SF_SYNTHETIC: u32 = 0xc0000000; +bitflagset::bitflag! { + /// BSD `chflags` bits. Values are bit *positions*; masks are `1 << pos`. + #[derive(Copy, Clone, Debug, PartialEq, Eq)] + #[repr(u8)] + pub enum ChFlag { + UfNodump = 0, + UfImmutable = 1, + UfAppend = 2, + UfOpaque = 3, + UfNounlink = 4, + UfCompressed = 5, + UfTracked = 6, + UfDatavault = 7, + UfHidden = 15, + SfArchived = 16, + SfImmutable = 17, + SfAppend = 18, + SfNounlink = 20, + SfSnapshot = 21, + SfFirmlink = 23, + SfDataless = 30, + } +} -/// BSD `chflags` bits libc binds on macOS and `_stat.c` falls back to -/// elsewhere. Published by `_stat` on every platform. +bitflagset::bitflagset! { + #[derive(Copy, Clone, PartialEq, Eq)] + pub struct ChFlags(u32): ChFlag +} + +impl ChFlags { + /// Owner-settable bits, including reserved user-flag positions. + pub const UF_SETTABLE: Self = Self::from_bits_retain(0x0000ffff); + pub const SF_SETTABLE: Self = Self::from_bits_retain(if cfg!(target_os = "macos") { + 0x3fff0000 + } else { + 0xffff0000 + }); + #[cfg(target_os = "macos")] + pub const SF_SUPPORTED: Self = Self::from_bits_retain(0x009f0000); + #[cfg(target_os = "macos")] + pub const SF_SYNTHETIC: Self = Self::from_bits_retain(0xc0000000); +} + +pub const UF_NODUMP: u32 = ChFlags::from_element(ChFlag::UfNodump).bits(); +pub const UF_IMMUTABLE: u32 = ChFlags::from_element(ChFlag::UfImmutable).bits(); +pub const UF_APPEND: u32 = ChFlags::from_element(ChFlag::UfAppend).bits(); +pub const UF_OPAQUE: u32 = ChFlags::from_element(ChFlag::UfOpaque).bits(); +pub const UF_NOUNLINK: u32 = ChFlags::from_element(ChFlag::UfNounlink).bits(); +pub const UF_COMPRESSED: u32 = ChFlags::from_element(ChFlag::UfCompressed).bits(); +pub const UF_TRACKED: u32 = ChFlags::from_element(ChFlag::UfTracked).bits(); +pub const UF_DATAVAULT: u32 = ChFlags::from_element(ChFlag::UfDatavault).bits(); +pub const UF_HIDDEN: u32 = ChFlags::from_element(ChFlag::UfHidden).bits(); +pub const UF_SETTABLE: u32 = ChFlags::UF_SETTABLE.bits(); +pub const SF_ARCHIVED: u32 = ChFlags::from_element(ChFlag::SfArchived).bits(); +pub const SF_IMMUTABLE: u32 = ChFlags::from_element(ChFlag::SfImmutable).bits(); +pub const SF_APPEND: u32 = ChFlags::from_element(ChFlag::SfAppend).bits(); +pub const SF_NOUNLINK: u32 = ChFlags::from_element(ChFlag::SfNounlink).bits(); +pub const SF_SNAPSHOT: u32 = ChFlags::from_element(ChFlag::SfSnapshot).bits(); +pub const SF_FIRMLINK: u32 = ChFlags::from_element(ChFlag::SfFirmlink).bits(); +pub const SF_DATALESS: u32 = ChFlags::from_element(ChFlag::SfDataless).bits(); +pub const SF_SETTABLE: u32 = ChFlags::SF_SETTABLE.bits(); #[cfg(target_os = "macos")] -pub use libc::{ - SF_APPEND, SF_ARCHIVED, SF_IMMUTABLE, UF_APPEND, UF_COMPRESSED, UF_HIDDEN, UF_IMMUTABLE, - UF_NODUMP, UF_OPAQUE, +pub const SF_SUPPORTED: u32 = ChFlags::SF_SUPPORTED.bits(); +#[cfg(target_os = "macos")] +pub const SF_SYNTHETIC: u32 = ChFlags::SF_SYNTHETIC.bits(); + +const _: () = { + assert!(UF_NODUMP == 0x00000001); + assert!(UF_IMMUTABLE == 0x00000002); + assert!(UF_APPEND == 0x00000004); + assert!(UF_OPAQUE == 0x00000008); + assert!(UF_NOUNLINK == 0x00000010); + assert!(UF_COMPRESSED == 0x00000020); + assert!(UF_TRACKED == 0x00000040); + assert!(UF_DATAVAULT == 0x00000080); + assert!(UF_HIDDEN == 0x00008000); + assert!(UF_SETTABLE == 0x0000ffff); + assert!(SF_ARCHIVED == 0x00010000); + assert!(SF_IMMUTABLE == 0x00020000); + assert!(SF_APPEND == 0x00040000); + assert!(SF_NOUNLINK == 0x00100000); + assert!(SF_SNAPSHOT == 0x00200000); + assert!(SF_FIRMLINK == 0x00800000); + assert!(SF_DATALESS == 0x40000000); }; -#[cfg(not(target_os = "macos"))] -pub const UF_NODUMP: u32 = 0x00000001; -#[cfg(not(target_os = "macos"))] -pub const UF_IMMUTABLE: u32 = 0x00000002; -#[cfg(not(target_os = "macos"))] -pub const UF_APPEND: u32 = 0x00000004; -#[cfg(not(target_os = "macos"))] -pub const UF_OPAQUE: u32 = 0x00000008; -#[cfg(not(target_os = "macos"))] -pub const UF_COMPRESSED: u32 = 0x00000020; -#[cfg(not(target_os = "macos"))] -pub const UF_HIDDEN: u32 = 0x00008000; -#[cfg(not(target_os = "macos"))] -pub const SF_ARCHIVED: u32 = 0x00010000; -#[cfg(not(target_os = "macos"))] -pub const SF_IMMUTABLE: u32 = 0x00020000; -#[cfg(not(target_os = "macos"))] -pub const SF_APPEND: u32 = 0x00040000; /// Solaris door/port and BSD whiteout. `_stat` publishes 0 where the /// platform has no such file type. diff --git a/crates/host_env/src/posix.rs b/crates/host_env/src/posix.rs index b26b2c168..f46554cad 100644 --- a/crates/host_env/src/posix.rs +++ b/crates/host_env/src/posix.rs @@ -903,7 +903,7 @@ impl From for AccessError { } } -pub use crate::os::{F_OK, R_OK, W_OK, X_OK}; +pub use crate::os::{AccessMode, F_OK, R_OK, W_OK, X_OK}; fn get_permissions(mode: u32) -> Permissions { Permissions { @@ -971,24 +971,24 @@ pub fn getgroups() -> std::io::Result> { pub fn check_access(path: &Path, mode: u8) -> Result { use std::os::unix::fs::MetadataExt; - if mode & !(R_OK | W_OK | X_OK) != 0 { + let Some(mode) = AccessMode::from_bits(mode) else { return Err(AccessError::InvalidMode); - } + }; let metadata = match crate::fs::metadata(path) { Ok(m) => m, Err(_) => return Ok(false), }; - if mode == F_OK { + if mode.is_empty() { return Ok(true); } let perm = get_right_permission(metadata.mode(), metadata.uid(), metadata.gid())?; - let r_ok = (mode & R_OK == 0) || perm.is_readable; - let w_ok = (mode & W_OK == 0) || perm.is_writable; - let x_ok = (mode & X_OK == 0) || perm.is_executable; + let r_ok = !mode.contains(&crate::os::AccessFlag::R) || perm.is_readable; + let w_ok = !mode.contains(&crate::os::AccessFlag::W) || perm.is_writable; + let x_ok = !mode.contains(&crate::os::AccessFlag::X) || perm.is_executable; Ok(r_ok && w_ok && x_ok) } diff --git a/crates/stdlib/src/pickle.rs b/crates/stdlib/src/pickle.rs index 4eca3f9e2..40f8f79ea 100644 --- a/crates/stdlib/src/pickle.rs +++ b/crates/stdlib/src/pickle.rs @@ -1945,9 +1945,9 @@ mod _pickle { file: PyObjectRef, #[pyarg(any, optional)] protocol: OptionalArg, - #[pyarg(named, optional)] + #[pyarg(any, optional)] fix_imports: OptionalArg, - #[pyarg(named, optional)] + #[pyarg(any, optional)] buffer_callback: OptionalArg, } diff --git a/crates/stdlib/src/pystruct.rs b/crates/stdlib/src/pystruct.rs index ba1430094..5e0553b2f 100644 --- a/crates/stdlib/src/pystruct.rs +++ b/crates/stdlib/src/pystruct.rs @@ -365,6 +365,12 @@ pub(crate) mod _struct { let spec = self.ready(vm)?.spec.clone(); UnpackIterator::with_buffer(vm, spec, buffer) } + + #[pymethod] + fn __sizeof__(&self, vm: &VirtualMachine) -> PyResult { + let inner = self.ready(vm)?; + Ok(core::mem::size_of::() + inner.spec.codes_sizeof()) + } } impl Representable for PyStruct { diff --git a/crates/stdlib/src/socket.rs b/crates/stdlib/src/socket.rs index c8f6af6af..ff582ef4f 100644 --- a/crates/stdlib/src/socket.rs +++ b/crates/stdlib/src/socket.rs @@ -22,7 +22,7 @@ mod _socket { convert::{IntoPyException, ToPyObject, TryFromBorrowedObject, TryFromObject}, function::{ ArgBytesLike, ArgIntoFloat, ArgMemoryBuffer, ArgStrOrBytesLike, Either, FsPath, - OptionalArg, OptionalOption, + FuncArgs, OptionalArg, OptionalOption, }, types::{Constructor, DefaultConstructor, Destructor, Initializer, Representable}, utils::ToCString, @@ -135,6 +135,16 @@ mod _socket { SOL_CAN_RAW, }; + #[cfg(target_os = "linux")] + #[pyattr] + const J1939_EE_INFO_NONE: i32 = 0; + #[cfg(target_os = "linux")] + #[pyattr] + const J1939_EE_INFO_TX_ABORT: i32 = 1; + #[cfg(target_os = "linux")] + #[pyattr] + const J1939_FILTER_MAX: i32 = 512; + #[cfg(target_os = "linux")] #[pyattr] use host_socket::{ @@ -652,6 +662,260 @@ mod _socket { ))] #[pyattr] use host_socket::{BDADDR_ANY, BDADDR_LOCAL}; + + #[cfg(any( + target_os = "android", + target_os = "fuchsia", + target_os = "linux", + target_os = "openbsd" + ))] + #[pyattr] + const BTPROTO_L2CAP: i32 = 0; + #[cfg(target_os = "freebsd")] + #[pyattr] + const BTPROTO_L2CAP: i32 = 135; + #[cfg(any( + target_os = "android", + target_os = "fuchsia", + target_os = "linux", + target_os = "openbsd" + ))] + #[pyattr] + const BTPROTO_HCI: i32 = 1; + #[cfg(target_os = "freebsd")] + #[pyattr] + const BTPROTO_HCI: i32 = 134; + #[cfg(any( + target_os = "android", + target_os = "fuchsia", + target_os = "linux", + target_os = "openbsd" + ))] + #[pyattr] + const BTPROTO_SCO: i32 = 2; + #[cfg(target_os = "freebsd")] + #[pyattr] + const BTPROTO_SCO: i32 = 137; + #[cfg(any( + target_os = "android", + target_os = "fuchsia", + target_os = "linux", + target_os = "openbsd" + ))] + #[pyattr] + const BTPROTO_RFCOMM: i32 = 3; + #[cfg(target_os = "freebsd")] + #[pyattr] + const BTPROTO_RFCOMM: i32 = 136; + #[cfg(any( + target_os = "android", + target_os = "fuchsia", + target_os = "linux", + target_os = "openbsd" + ))] + #[pyattr] + const SOL_HCI: i32 = 0; + #[cfg(target_os = "freebsd")] + #[pyattr] + const SOL_HCI: i32 = 0x0802; + #[cfg(any( + target_os = "android", + target_os = "fuchsia", + target_os = "linux", + target_os = "openbsd" + ))] + #[pyattr] + const HCI_DATA_DIR: i32 = 1; + #[cfg(target_os = "freebsd")] + #[pyattr] + const HCI_DATA_DIR: i32 = 2; + #[cfg(any( + target_os = "android", + target_os = "fuchsia", + target_os = "linux", + target_os = "openbsd" + ))] + #[pyattr] + const SOL_L2CAP: i32 = 6; + #[cfg(target_os = "freebsd")] + #[pyattr] + const SOL_L2CAP: i32 = 0x1609; + #[cfg(any( + target_os = "android", + target_os = "fuchsia", + target_os = "linux", + target_os = "openbsd" + ))] + #[pyattr] + const SOL_SCO: i32 = 17; + #[cfg(target_os = "freebsd")] + #[pyattr] + const SOL_SCO: i32 = 0x0209; + #[cfg(any( + target_os = "android", + target_os = "fuchsia", + target_os = "linux", + target_os = "openbsd" + ))] + #[pyattr] + const SOL_RFCOMM: i32 = 18; + #[cfg(target_os = "freebsd")] + #[pyattr] + const SOL_RFCOMM: i32 = 0x0816; + + // Linux / / + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const SOL_BLUETOOTH: i32 = 274; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const HCI_DEV_NONE: i32 = 0xffff; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const HCI_CHANNEL_RAW: i32 = 0; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const HCI_CHANNEL_USER: i32 = 1; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const HCI_CHANNEL_MONITOR: i32 = 2; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const HCI_CHANNEL_CONTROL: i32 = 3; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const HCI_CHANNEL_LOGGING: i32 = 4; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const HCI_TIME_STAMP: i32 = 3; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_SECURITY: i32 = 4; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_SECURITY_SDP: i32 = 0; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_SECURITY_LOW: i32 = 1; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_SECURITY_MEDIUM: i32 = 2; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_SECURITY_HIGH: i32 = 3; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_SECURITY_FIPS: i32 = 4; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_FLUSHABLE: i32 = 8; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_FLUSHABLE_OFF: i32 = 0; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_FLUSHABLE_ON: i32 = 1; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_POWER: i32 = 9; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_POWER_FORCE_ACTIVE_OFF: i32 = 0; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_POWER_FORCE_ACTIVE_ON: i32 = 1; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_CHANNEL_POLICY: i32 = 10; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_CHANNEL_POLICY_BREDR_ONLY: i32 = 0; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_CHANNEL_POLICY_BREDR_PREFERRED: i32 = 1; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_CHANNEL_POLICY_AMP_PREFERRED: i32 = 2; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_VOICE: i32 = 11; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_VOICE_TRANSPARENT: i32 = 0x0003; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_VOICE_CVSD_16BIT: i32 = 0x0060; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_SNDMTU: i32 = 12; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_RCVMTU: i32 = 13; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_PHY: i32 = 14; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_PHY_BR_1M_1SLOT: i32 = 0x0001; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_MODE: i32 = 16; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const BT_MODE_BASIC: i32 = 0x00; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const L2CAP_LM: i32 = 0x03; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const L2CAP_LM_MASTER: i32 = 0x0001; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const L2CAP_LM_AUTH: i32 = 0x0002; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const L2CAP_LM_ENCRYPT: i32 = 0x0004; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const L2CAP_LM_TRUSTED: i32 = 0x0008; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const L2CAP_LM_RELIABLE: i32 = 0x0010; + #[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))] + #[pyattr] + const L2CAP_LM_SECURE: i32 = 0x0020; + #[cfg(any( + target_os = "android", + target_os = "freebsd", + target_os = "fuchsia", + target_os = "linux" + ))] + #[pyattr] + const BDADDR_BREDR: i32 = 0x00; + #[cfg(any( + target_os = "android", + target_os = "freebsd", + target_os = "fuchsia", + target_os = "linux" + ))] + #[pyattr] + const BDADDR_LE_PUBLIC: i32 = 0x01; + #[cfg(any( + target_os = "android", + target_os = "freebsd", + target_os = "fuchsia", + target_os = "linux" + ))] + #[pyattr] + const BDADDR_LE_RANDOM: i32 = 0x02; + #[cfg(any( + target_os = "android", + target_os = "freebsd", + target_os = "fuchsia", + target_os = "linux" + ))] + #[pyattr] + const HCI_FILTER: i32 = 2; // HERE IS WHERE THE BLUETOOTH CONSTANTS END #[cfg(windows)] @@ -1183,43 +1447,88 @@ mod _socket { obj.class().name() )) })?; - if tuple.is_empty() || tuple.len() > 2 { - return Err(vm - .new_type_error( - "AF_CAN address must be a tuple (interface,) or (interface, addr)", - ) - .into()); - } - let interface: PyStrRef = tuple[0].clone().downcast().map_err(|obj| { - vm.new_type_error(format!( - "{}(): AF_CAN interface must be str, not {}", - caller, - obj.class().name() - )) - })?; + let proto = self.proto.load(); + let interface: PyStrRef = tuple + .first() + .cloned() + .ok_or_else(|| { + vm.new_type_error("AF_CAN address must be a tuple (interface, )") + })? + .downcast() + .map_err(|obj| { + vm.new_type_error(format!( + "{}(): AF_CAN interface must be str, not {}", + caller, + obj.class().name() + )) + })?; let interface = interface.try_into_utf8(vm).map_err(IoOrPyException::from)?; let ifname = interface.as_str(); - // Get interface index let ifindex = if ifname.is_empty() { - 0 // Bind to all CAN interfaces + 0 + } else if ifname.len() >= 16 { + return Err(vm.new_os_error("AF_CAN interface name too long").into()); } else { - // Check interface name length (IFNAMSIZ is typically 16) - if ifname.len() >= 16 { - return Err(vm.new_os_error("interface name too long").into()); - } let cstr = alloc::ffi::CString::new(ifname) .map_err(|_| vm.new_os_error("invalid interface name"))?; host_socket::if_nametoindex_checked(cstr.as_c_str())? as i32 }; - // Create sockaddr_can let mut storage: c::sockaddr_storage = unsafe { core::mem::zeroed() }; let can_addr = &mut storage as *mut c::sockaddr_storage as *mut c::sockaddr_can; unsafe { (*can_addr).can_family = c::AF_CAN as c::sa_family_t; (*can_addr).can_ifindex = ifindex; } + + match proto { + c::CAN_RAW | c::CAN_BCM => { + if tuple.len() != 1 { + return Err(vm + .new_type_error("AF_CAN address must be a tuple (interface, )") + .into()); + } + } + c::CAN_ISOTP => { + if tuple.len() != 3 { + return Err(vm + .new_type_error( + "AF_CAN ISOTP address must be a tuple (interface, rx_id, tx_id)", + ) + .into()); + } + let rx_id = tuple[1].try_index(vm)?.try_to_primitive::(vm)?; + let tx_id = tuple[2].try_index(vm)?.try_to_primitive::(vm)?; + unsafe { + (*can_addr).can_addr.tp.rx_id = rx_id; + (*can_addr).can_addr.tp.tx_id = tx_id; + } + } + c::CAN_J1939 => { + if tuple.len() != 4 { + return Err(vm + .new_type_error( + "AF_CAN J1939 address must be a tuple (interface, name, pgn, addr)", + ) + .into()); + } + let name = tuple[1].try_index(vm)?.try_to_primitive::(vm)?; + let pgn = tuple[2].try_index(vm)?.try_to_primitive::(vm)?; + let jaddr = tuple[3].try_index(vm)?.try_to_primitive::(vm)?; + unsafe { + (*can_addr).can_addr.j1939.name = name; + (*can_addr).can_addr.j1939.pgn = pgn; + (*can_addr).can_addr.j1939.addr = jaddr; + } + } + _ => { + return Err(vm + .new_os_error(format!("{caller}(): unsupported CAN protocol")) + .into()); + } + } + let storage: host_socket::raw::SockAddrStorage = unsafe { core::mem::transmute(storage) }; Ok(unsafe { @@ -1398,7 +1707,7 @@ mod _socket { #[pyclass( with(Constructor, Initializer, Representable, Destructor), - flags(BASETYPE) + flags(BASETYPE, HEAPTYPE) )] impl PySocket { fn _init( @@ -1579,7 +1888,7 @@ mod _socket { self.sock_snapshot()?.accept_raw() })?; let fd = into_sock_fileno(sock); - Ok((fd, get_addr_tuple(&addr, vm))) + Ok((fd, get_addr_tuple(&addr, self.proto.load(), vm))) } #[pymethod] @@ -1652,7 +1961,7 @@ mod _socket { .recv_from_with_flags(buffer.spare_capacity_mut(), flags) })?; unsafe { buffer.set_len(n) }; - Ok((buffer, get_addr_tuple(&addr, vm))) + Ok((buffer, get_addr_tuple(&addr, self.proto.load(), vm))) } #[pymethod] @@ -1685,7 +1994,7 @@ mod _socket { })?; unsafe { scratch.set_len(n) }; buf.borrow_buf_mut()[..n].copy_from_slice(&scratch); - Ok((n, get_addr_tuple(&addr, vm))) + Ok((n, get_addr_tuple(&addr, self.proto.load(), vm))) } #[pymethod] @@ -1731,25 +2040,26 @@ mod _socket { } #[pymethod] - fn sendto( - &self, - bytes: ArgBytesLike, - arg2: PyObjectRef, - arg3: OptionalArg, - vm: &VirtualMachine, - ) -> Result { - // signature is bytes[, flags], address - let (flags, address) = match arg3 { - OptionalArg::Present(arg3) => { - // should just be i32::try_from_obj but tests check for error message - let int = arg2 - .try_index_opt(vm) - .unwrap_or_else(|| Err(vm.new_type_error("an integer is required")))?; - let flags = int.try_to_primitive::(vm)?; - (flags, arg3) + fn sendto(&self, args: FuncArgs, vm: &VirtualMachine) -> Result { + if !args.kwargs.is_empty() { + return Err(vm + .new_type_error("sendto() takes no keyword arguments") + .into()); + } + let n = args.args.len(); + let (bytes, flags, address) = match n { + 2 => (args.args[0].clone(), 0, args.args[1].clone()), + 3 => { + let flags = args.args[1].try_index(vm)?.try_to_primitive::(vm)?; + (args.args[0].clone(), flags, args.args[2].clone()) + } + _ => { + return Err(vm + .new_type_error(format!("sendto() takes 2 or 3 arguments ({n} given)")) + .into()); } - OptionalArg::Missing => (0, arg2), }; + let bytes = ArgBytesLike::try_from_object(vm, bytes)?; let addr = self.extract_address(address, "sendto", vm)?; let buf = bytes.borrow_buf_unlocked(vm)?; let buf = &*buf; @@ -1762,8 +2072,8 @@ mod _socket { #[pymethod] fn sendmsg( &self, - buffers: Vec, - ancdata: OptionalArg, + buffers: PyObjectRef, + ancdata: OptionalArg, flags: OptionalArg, addr: OptionalOption, vm: &VirtualMachine, @@ -1779,6 +2089,23 @@ mod _socket { msg = msg.with_addr(&sockaddr); } + let collect_iterable = + |obj: &PyObject, msg: &'static str| -> PyResult> { + match vm.extract_elements_with(obj, Ok) { + Ok(items) => Ok(items), + Err(e) if e.fast_isinstance(vm.ctx.exceptions.type_error) => { + Err(vm.new_type_error(msg)) + } + Err(e) => Err(e), + } + }; + + let buffer_items = + collect_iterable(&buffers, "sendmsg() argument 1 must be an iterable")?; + let buffers = buffer_items + .into_iter() + .map(|obj| ArgBytesLike::try_from_object(vm, obj)) + .collect::>>()?; let buffers = buffers .iter() .map(|buf| buf.borrow_buf_unlocked(vm)) @@ -1791,20 +2118,22 @@ mod _socket { let control_buf; if let OptionalArg::Present(ancdata) = ancdata { - let cmsgs = vm.extract_elements_with( - &ancdata, - |obj| -> PyResult<(i32, i32, ArgBytesLike)> { - let seq: Vec = obj.try_into_value(vm)?; + let cmsg_fast = + collect_iterable(&ancdata, "sendmsg() argument 2 must be an iterable")?; + let cmsgs = cmsg_fast + .into_iter() + .map(|item| -> PyResult<(i32, i32, ArgBytesLike)> { + let seq: Vec = item.try_into_value(vm)?; let [lvl, typ, data]: [PyObjectRef; 3] = seq .try_into() - .map_err(|_| vm.new_type_error("expected a sequence of length 3"))?; + .map_err(|_| vm.new_type_error("[sendmsg() ancillary data items]"))?; Ok(( - lvl.try_into_value(vm)?, - typ.try_into_value(vm)?, - data.try_into_value(vm)?, + lvl.try_index(vm)?.try_to_primitive::(vm)?, + typ.try_index(vm)?.try_to_primitive::(vm)?, + ArgBytesLike::try_from_object(vm, data)?, )) - }, - )?; + }) + .collect::>>()?; control_buf = Self::pack_cmsgs_to_send(&cmsgs, vm)?; if !control_buf.is_empty() { msg = msg.with_control(&control_buf); @@ -1895,7 +2224,7 @@ mod _socket { let storage: host_socket::raw::SockAddrStorage = unsafe { core::mem::transmute(address.storage) }; let addr = unsafe { host_socket::raw::SockAddr::new(storage, address.len as _) }; - get_addr_tuple(&addr, vm) + get_addr_tuple(&addr, self.proto.load(), vm) } else { vm.ctx.none() }; @@ -1979,14 +2308,14 @@ mod _socket { fn getsockname(&self, vm: &VirtualMachine) -> std::io::Result { let addr = self.sock()?.local_addr()?; - Ok(get_addr_tuple(&addr, vm)) + Ok(get_addr_tuple(&addr, self.proto.load(), vm)) } #[pymethod] fn getpeername(&self, vm: &VirtualMachine) -> std::io::Result { let addr = self.sock()?.peer_addr()?; - Ok(get_addr_tuple(&addr, vm)) + Ok(get_addr_tuple(&addr, self.proto.load(), vm)) } #[pymethod] @@ -2256,7 +2585,11 @@ mod _socket { } } - fn get_addr_tuple(addr: &host_socket::raw::SockAddr, vm: &VirtualMachine) -> PyObjectRef { + fn get_addr_tuple( + addr: &host_socket::raw::SockAddr, + #[cfg_attr(not(target_os = "linux"), allow(unused_variables))] proto: i32, + vm: &VirtualMachine, + ) -> PyObjectRef { if let Some(addr) = addr.as_socket() { return get_ip_addr_tuple(&addr, vm); } @@ -2277,7 +2610,6 @@ mod _socket { { let family = addr.family(); if family == c::AF_CAN as c::sa_family_t { - // AF_CAN address: (interface_name,) or (interface_name, can_id) let can_addr = unsafe { &*(addr.as_ptr() as *const c::sockaddr_can) }; let ifindex = can_addr.can_ifindex; let ifname = if ifindex == 0 { @@ -2285,7 +2617,33 @@ mod _socket { } else { host_socket::if_indextoname_checked(ifindex as u32).unwrap_or_default() }; - return vm.ctx.new_tuple(vec![vm.ctx.new_str(ifname).into()]).into(); + let ifname_obj: PyObjectRef = vm.ctx.new_str(ifname).into(); + return match proto { + c::CAN_ISOTP => vm + .ctx + .new_tuple(vec![ + ifname_obj, + vm.ctx.new_int(unsafe { can_addr.can_addr.tp.rx_id }).into(), + vm.ctx.new_int(unsafe { can_addr.can_addr.tp.tx_id }).into(), + ]) + .into(), + c::CAN_J1939 => vm + .ctx + .new_tuple(vec![ + ifname_obj, + vm.ctx + .new_int(unsafe { can_addr.can_addr.j1939.name }) + .into(), + vm.ctx + .new_int(unsafe { can_addr.can_addr.j1939.pgn }) + .into(), + vm.ctx + .new_int(unsafe { can_addr.can_addr.j1939.addr }) + .into(), + ]) + .into(), + _ => vm.ctx.new_tuple(vec![ifname_obj]).into(), + }; } if family == c::AF_ALG as c::sa_family_t { // AF_ALG address: (type, name) diff --git a/crates/vm/Cargo.toml b/crates/vm/Cargo.toml index b36b7bd83..2b6577e4a 100644 --- a/crates/vm/Cargo.toml +++ b/crates/vm/Cargo.toml @@ -47,6 +47,7 @@ rustpython-unicode = { workspace = true } ascii = { workspace = true } bitflags = { workspace = true } +bitflagset = { workspace = true } bstr = { workspace = true } crossbeam-utils = { workspace = true } constant_time_eq = { workspace = true } diff --git a/crates/vm/Lib/core_modules/encodings_latin_1.py b/crates/vm/Lib/core_modules/encodings_latin_1.py new file mode 100644 index 000000000..370160c0c --- /dev/null +++ b/crates/vm/Lib/core_modules/encodings_latin_1.py @@ -0,0 +1,50 @@ +""" Python 'latin-1' Codec + + +Written by Marc-Andre Lemburg (mal@lemburg.com). + +(c) Copyright CNRI, All Rights Reserved. NO WARRANTY. + +""" +import codecs + +### Codec APIs + +class Codec(codecs.Codec): + + # Note: Binding these as C functions will result in the class not + # converting them to methods. This is intended. + encode = codecs.latin_1_encode + decode = codecs.latin_1_decode + +class IncrementalEncoder(codecs.IncrementalEncoder): + def encode(self, input, final=False): + return codecs.latin_1_encode(input,self.errors)[0] + +class IncrementalDecoder(codecs.IncrementalDecoder): + def decode(self, input, final=False): + return codecs.latin_1_decode(input,self.errors)[0] + +class StreamWriter(Codec,codecs.StreamWriter): + pass + +class StreamReader(Codec,codecs.StreamReader): + pass + +class StreamConverter(StreamWriter,StreamReader): + + encode = codecs.latin_1_decode + decode = codecs.latin_1_encode + +### encodings module API + +def getregentry(): + return codecs.CodecInfo( + name='iso8859-1', + encode=Codec.encode, + decode=Codec.decode, + incrementalencoder=IncrementalEncoder, + incrementaldecoder=IncrementalDecoder, + streamreader=StreamReader, + streamwriter=StreamWriter, + ) diff --git a/crates/vm/src/buffer.rs b/crates/vm/src/buffer.rs index dd48d0065..4f23e16c5 100644 --- a/crates/vm/src/buffer.rs +++ b/crates/vm/src/buffer.rs @@ -4,7 +4,7 @@ use crate::{ common::{static_cell, str::wchar_t}, convert::ToPyObject, exceptions, - function::{ArgBytesLike, ArgIntoBool, ArgIntoFloat}, + function::{ArgBytesLike, ArgIntoBool, ArgIntoComplex, ArgIntoFloat}, }; use rustpython_common::wtf8::Wtf8Buf; @@ -13,6 +13,7 @@ use core::{fmt, iter::Peekable, mem}; use half::f16; use itertools::Itertools; use malachite_bigint::BigInt; +use num_complex::Complex64; use num_traits::{PrimInt, ToPrimitive}; use std::os::raw; @@ -157,6 +158,8 @@ pub(crate) enum FormatType { Float = b'f', Double = b'd', LongDouble = b'g', + FloatComplex = b'F', + DoubleComplex = b'D', VoidP = b'P', PyObject = b'O', } @@ -221,6 +224,8 @@ impl FormatType { Self::Float => nonnative_info!(f32, $end), Self::Double => nonnative_info!(f64, $end), Self::LongDouble => nonnative_info!(f64, $end), // long double same as double + Self::FloatComplex => nonnative_info!(PackFloatComplex, $end), + Self::DoubleComplex => nonnative_info!(PackDoubleComplex, $end), Self::PyObject => nonnative_info!(usize, $end), // pointer size _ => unreachable!(), // size_t or void* } @@ -260,6 +265,8 @@ impl FormatType { Self::Float => native_info!(raw::c_float), Self::Double => native_info!(raw::c_double), Self::LongDouble => native_info!(raw::c_double), // long double same as double for now + Self::FloatComplex => native_info!(PackFloatComplex), + Self::DoubleComplex => native_info!(PackDoubleComplex), Self::VoidP => native_info!(*mut raw::c_void), Self::PyObject => native_info!(*mut raw::c_void), // pointer to PyObject }, @@ -624,6 +631,11 @@ impl FormatSpec { pub const fn size(&self) -> usize { self.size } + + #[must_use] + pub fn codes_sizeof(&self) -> usize { + core::mem::size_of::() * (self.codes.len() + 1) + } } trait Packable { @@ -761,6 +773,51 @@ macro_rules! make_pack_float { make_pack_float!(f32, "f"); make_pack_float!(f64, "d"); +#[repr(C)] +struct PackFloatComplex(f32, f32); + +#[repr(C)] +struct PackDoubleComplex(f64, f64); + +macro_rules! make_pack_complex { + ($T:ty, $Elem:ty, $Bits:ty) => { + impl Packable for $T { + fn pack( + vm: &VirtualMachine, + _code: FormatType, + arg: PyObjectRef, + data: &mut [u8], + ) -> Result<(), PackError> { + let c = ArgIntoComplex::try_from_object(vm, arg) + .map_err(|_| { + PackError::new( + PackErrorKind::Type, + vm, + "required argument is not a complex", + ) + })? + .into_complex(); + let half = size_of::<$Elem>(); + (c.re as $Elem).to_bits().pack_int::(&mut data[..half]); + (c.im as $Elem) + .to_bits() + .pack_int::(&mut data[half..half * 2]); + Ok(()) + } + + fn unpack(vm: &VirtualMachine, rdr: &[u8]) -> PyObjectRef { + let half = size_of::<$Elem>(); + let re = <$Elem>::from_bits(<$Bits>::unpack_int::(&rdr[..half])) as f64; + let im = <$Elem>::from_bits(<$Bits>::unpack_int::(&rdr[half..half * 2])) as f64; + vm.ctx.new_complex(Complex64::new(re, im)).into() + } + } + }; +} + +make_pack_complex!(PackFloatComplex, f32, u32); +make_pack_complex!(PackDoubleComplex, f64, u64); + impl Packable for f16 { fn pack( vm: &VirtualMachine, diff --git a/crates/vm/src/builtins/bool.rs b/crates/vm/src/builtins/bool.rs index 28d598ebf..a613b3d9a 100644 --- a/crates/vm/src/builtins/bool.rs +++ b/crates/vm/src/builtins/bool.rs @@ -172,7 +172,7 @@ impl AsNumber for PyBool { and: Some(|a, b, vm| PyBool::__and__(a, b.to_owned(), vm).to_pyresult(vm)), xor: Some(|a, b, vm| PyBool::__xor__(a, b.to_owned(), vm).to_pyresult(vm)), or: Some(|a, b, vm| PyBool::__or__(a, b.to_owned(), vm).to_pyresult(vm)), - ..PyInt::AS_NUMBER + ..PyNumberMethods::NOT_IMPLEMENTED }; &AS_NUMBER } diff --git a/crates/vm/src/builtins/code.rs b/crates/vm/src/builtins/code.rs index d083f778a..4ed5ee60d 100644 --- a/crates/vm/src/builtins/code.rs +++ b/crates/vm/src/builtins/code.rs @@ -655,7 +655,9 @@ impl PyCode { vm: &VirtualMachine, code: frozen::FrozenCodeObject, ) -> PyRef { - Self::new_ref_with_bag(vm, code.decode(PyVmBag(vm))) + let py_code = Self::new_ref_with_bag(vm, code.decode(PyVmBag(vm))); + apply_frozen_co_filename(&py_code, vm); + py_code } #[cfg(feature = "host_env")] @@ -703,6 +705,38 @@ impl PyCode { } } +fn frozen_co_filename(path: &str) -> Option<&'static str> { + match path { + "_frozen_importlib" => Some(""), + "_frozen_importlib_external" => Some(""), + _ => None, + } +} + +fn apply_frozen_co_filename(code: &PyCode, vm: &VirtualMachine) { + let Some(new) = frozen_co_filename(code.source_path().as_str()) else { + return; + }; + set_source_path_tree(code, vm.ctx.intern_str(new)); +} + +fn set_source_path_tree(code: &PyCode, interned: &'static PyStrInterned) { + code.set_source_path(interned); + for constant in code.constants.iter() { + set_source_path_in_const(&constant.0, interned); + } +} + +fn set_source_path_in_const(obj: &PyObject, interned: &'static PyStrInterned) { + if let Some(inner) = obj.downcast_ref::() { + set_source_path_tree(inner, interned); + } else if let Some(tup) = obj.downcast_ref::() { + for item in tup { + set_source_path_in_const(item, interned); + } + } +} + impl fmt::Debug for PyCode { fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result { write!(f, "code: {:?}", self.code) diff --git a/crates/vm/src/builtins/descriptor.rs b/crates/vm/src/builtins/descriptor.rs index 50b35151d..97086cc74 100644 --- a/crates/vm/src/builtins/descriptor.rs +++ b/crates/vm/src/builtins/descriptor.rs @@ -1,4 +1,4 @@ -use super::{PyStr, PyStrInterned, PyType}; +use super::{PyStr, PyStrInterned, PyTuple, PyType}; use crate::{ AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, builtins::{PyTypeRef, builtin_func::PyNativeMethod, type_}, @@ -178,6 +178,157 @@ impl Representable for PyMethodDescriptor { } } +/// METH_CLASS descriptors. Same layout as method_descriptor; a distinct type. +#[pyclass(name = "classmethod_descriptor", module = false)] +pub struct PyClassMethodDescriptor { + pub common: PyDescriptor, + pub method: &'static PyMethodDef, + pub objclass: &'static Py, + pub(crate) _method_def_owner: Option, +} + +impl PyClassMethodDescriptor { + pub fn new(method: &'static PyMethodDef, typ: &'static Py, ctx: &Context) -> Self { + Self { + common: PyDescriptor { + typ, + name: ctx.intern_str(method.name), + qualname: PyRwLock::new(None), + }, + method, + objclass: typ, + _method_def_owner: None, + } + } + + pub fn bind(&self, obj: PyObjectRef, ctx: &Context) -> PyRef { + self.method.build_bound_method(ctx, obj, self.common.typ) + } +} + +impl PyPayload for PyClassMethodDescriptor { + fn class(ctx: &Context) -> &'static Py { + ctx.types.classmethod_descriptor_type + } +} + +impl core::fmt::Debug for PyClassMethodDescriptor { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + write!(f, "classmethod descriptor for '{}'", self.common.name) + } +} + +impl GetDescriptor for PyClassMethodDescriptor { + fn descr_get( + zelf: PyObjectRef, + obj: Option, + cls: Option, + vm: &VirtualMachine, + ) -> PyResult { + let descr = Self::_as_pyref(&zelf, vm).unwrap(); + let type_obj = match cls { + Some(typ) => typ, + None => match &obj { + Some(o) => o.class().to_owned().into(), + None => { + return Err(vm.new_type_error(format!( + "descriptor '{}' for type '{}' needs either an object or a type", + descr.common.name, + descr.common.typ.name() + ))); + } + }, + }; + if !type_obj.fast_isinstance(vm.ctx.types.type_type) { + return Err(vm.new_type_error(format!( + "descriptor '{}' for type '{}' needs a type, not '{}' as arg 2", + descr.common.name, + descr.common.typ.name(), + type_obj.class().name() + ))); + } + let typ = type_obj.downcast::().map_err(|obj| { + vm.new_type_error(format!( + "descriptor '{}' for type '{}' needs a type, not '{}' as arg 2", + descr.common.name, + descr.common.typ.name(), + obj.class().name() + )) + })?; + if !typ.fast_issubclass(descr.common.typ) { + return Err(vm.new_type_error(format!( + "descriptor '{}' requires a subtype of '{}' but received '{}'", + descr.common.name, + descr.common.typ.name(), + typ.name() + ))); + } + Ok(descr.bind(typ.into(), &vm.ctx).into()) + } +} + +impl Callable for PyClassMethodDescriptor { + type Args = FuncArgs; + fn call(zelf: &Py, mut args: FuncArgs, vm: &VirtualMachine) -> PyResult { + let Some(owner) = args.args.first().cloned() else { + return Err(vm.new_type_error(format!( + "descriptor '{}' of '{}' object needs an argument", + zelf.method.name, + zelf.common.typ.name() + ))); + }; + let bound = Self::descr_get(zelf.to_owned().into(), None, Some(owner), vm)?; + args.args.remove(0); + bound.call(args, vm) + } +} + +#[pyclass( + with(GetDescriptor, Callable, Representable), + flags(DISALLOW_INSTANTIATION) +)] +impl PyClassMethodDescriptor { + #[pygetset] + const fn __name__(&self) -> &'static PyStrInterned { + self.common.name + } + + #[pygetset] + fn __qualname__(&self) -> String { + format!("{}.{}", self.common.typ.name(), self.common.name) + } + + #[pygetset] + fn __doc__(&self) -> Option<&'static str> { + let doc = self.method.doc?; + type_::get_doc_from_internal_doc(self.method.name, doc) + } + + #[pygetset] + fn __text_signature__(&self) -> Option { + self.method.doc.and_then(|doc| { + type_::get_text_signature_from_internal_doc(self.method.name, doc) + .map(|signature| signature.to_string()) + }) + } + + #[pygetset] + fn __objclass__(&self) -> PyTypeRef { + self.objclass.to_owned() + } +} + +impl Representable for PyClassMethodDescriptor { + #[inline] + fn repr_str(zelf: &Py, _vm: &VirtualMachine) -> PyResult { + Ok(format!( + "", + zelf.method.name, + zelf.common.typ.name() + )) + } +} + #[derive(Debug)] pub enum MemberKind { Object = 6, @@ -191,6 +342,7 @@ pub(crate) type MemberSetterFunc = pub enum MemberGetter { Getter(fn(&VirtualMachine, PyObjectRef) -> PyResult), Offset(usize), + TupleItem(usize), } pub enum MemberSetter { @@ -211,6 +363,16 @@ impl PyMemberDef { match self.getter { MemberGetter::Getter(getter) => (getter)(vm, obj), MemberGetter::Offset(offset) => get_slot_from_object(&obj, offset, self, vm), + MemberGetter::TupleItem(index) => { + let tuple = obj.downcast_ref::().ok_or_else(|| { + vm.new_type_error("unexpected payload for struct sequence member") + })?; + tuple + .as_slice() + .get(index) + .cloned() + .ok_or_else(|| vm.new_index_error(format!("tuple index {index} out of range"))) + } } } @@ -497,6 +659,7 @@ pub(crate) fn init(ctx: &'static Context) { .slots .vectorcall .store(Some(vectorcall_method_descriptor)); + PyClassMethodDescriptor::extend_class(ctx, ctx.types.classmethod_descriptor_type); PyWrapper::extend_class(ctx, ctx.types.wrapper_descriptor_type); ctx.types .wrapper_descriptor_type diff --git a/crates/vm/src/builtins/dict.rs b/crates/vm/src/builtins/dict.rs index c69594e99..d4a5a6e10 100644 --- a/crates/vm/src/builtins/dict.rs +++ b/crates/vm/src/builtins/dict.rs @@ -4,6 +4,7 @@ use super::{ }; use crate::common::lock::LazyLock; use crate::object::{Traverse, TraverseFn}; +use crate::stdlib::PyOrderedDictItems; use crate::{ AsObject, Context, Py, PyExact, PyObject, PyObjectRef, PyPayload, PyRef, PyRefExact, PyResult, TryFromObject, atomic_func, @@ -213,7 +214,7 @@ impl PyDict { Ok((key.clone(), value.clone())) } - fn update_sequence_pair( + pub(crate) fn update_sequence_pair( element: PyObjectRef, index: usize, vm: &VirtualMachine, @@ -422,7 +423,7 @@ impl PyDict { } #[pymethod] - fn __sizeof__(&self) -> usize { + pub(crate) fn __sizeof__(&self) -> usize { core::mem::size_of::() + self.entries.sizeof() } @@ -492,8 +493,9 @@ impl PyDict { } fn __or__(&self, other: PyObjectRef, vm: &VirtualMachine) -> PyResult { - let other_dict: Result = other.downcast(); - if let Ok(other) = other_dict { + // Accept dict subclasses that inherit this implementation. Subclasses with + // their own reflected slot, such as OrderedDict, take precedence in dispatch. + if let Ok(other) = other.downcast::() { let self_cp = self.copy(); self_cp.merge_dict(&other, true, vm)?; return Ok(self_cp.into_pyobject(vm)); @@ -538,7 +540,7 @@ impl PyDict { #[pyclass] impl Py { - fn inner_cmp( + pub(crate) fn inner_cmp( &self, other: &Self, op: PyComparisonOp, @@ -820,6 +822,38 @@ impl Py { self.entries.get_known_hash(vm, key, hash) } + pub(crate) fn set_item_known_hash( + &self, + key: &PyObject, + hash: crate::common::hash::PyHash, + value: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult<()> { + self.entries.insert_known_hash(vm, key, hash, value) + } + + pub(crate) fn del_item_known_hash( + &self, + key: &PyObject, + hash: crate::common::hash::PyHash, + vm: &VirtualMachine, + ) -> PyResult<()> { + if self.entries.delete_if_exists_known_hash(vm, key, hash)? { + Ok(()) + } else { + Err(vm.new_key_error(key.to_owned())) + } + } + + pub(crate) fn contains_known_hash( + &self, + key: &PyObject, + hash: crate::common::hash::PyHash, + vm: &VirtualMachine, + ) -> PyResult { + self.entries.contains_known_hash(vm, key, hash) + } + pub fn get_item_opt( &self, key: &K, @@ -1663,7 +1697,7 @@ impl AsNumber for PyDictItems { static AS_NUMBER: PyNumberMethods = PyNumberMethods { subtract: Some(set_inner_number_subtract), and: Some(set_inner_number_and), - xor: Some(set_inner_number_xor), + xor: Some(dict_item_view_number_xor), or: Some(set_inner_number_or), ..PyNumberMethods::NOT_IMPLEMENTED }; @@ -1704,19 +1738,77 @@ where Ok(PySet { inner: f(a, b)? }.into_pyobject(vm)) } -fn set_inner_number_subtract(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult { +pub(crate) fn set_inner_number_subtract( + a: &PyObject, + b: &PyObject, + vm: &VirtualMachine, +) -> PyResult { set_inner_number_op(a, b, |a, b| a.difference(b, vm), vm) } -fn set_inner_number_and(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult { - set_inner_number_op(a, b, |a, b| a.intersection(b, vm), vm) +pub(crate) fn set_inner_number_and(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult { + let a_is_view = + a.downcast_ref::().is_some() || a.downcast_ref::().is_some(); + let (view, other) = if a_is_view { (a, b) } else { (b, a) }; + set_view_number_and(view, other, vm) } -fn set_inner_number_xor(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult { +pub(crate) fn set_view_number_and( + view: &PyObject, + other: &PyObject, + vm: &VirtualMachine, +) -> PyResult { + let other = ArgIterable::::try_from_object(vm, other.to_owned())?; + let result = PySet::default(); + for item in other.iter(vm)? { + let item = item?; + if view.sequence_unchecked().contains(&item, vm)? { + result.add(item, vm)?; + } + } + Ok(result.into_pyobject(vm)) +} + +pub(crate) fn set_inner_number_xor(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult { set_inner_number_op(a, b, |a, b| a.symmetric_difference(b, vm), vm) } -fn set_inner_number_or(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult { +pub(crate) fn set_item_view_number_xor( + a: &PyObject, + b: &PyObject, + both_are_item_views: bool, + vm: &VirtualMachine, +) -> PyResult { + if !both_are_item_views { + return set_inner_number_xor(a, b, vm); + } + let result = PySet::default(); + let a_iterable = ArgIterable::::try_from_object(vm, a.to_owned())?; + for item in a_iterable.iter(vm)? { + let item = item?; + if !b.sequence_unchecked().contains(&item, vm)? { + result.add(item, vm)?; + } + } + let b_iterable = ArgIterable::::try_from_object(vm, b.to_owned())?; + for item in b_iterable.iter(vm)? { + let item = item?; + if !a.sequence_unchecked().contains(&item, vm)? { + result.add(item, vm)?; + } + } + Ok(result.into_pyobject(vm)) +} + +fn dict_item_view_number_xor(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult { + let is_native_item_view = |obj: &PyObject| { + obj.downcast_ref::().is_some() + || obj.downcast_ref::().is_some() + }; + set_item_view_number_xor(a, b, is_native_item_view(a) && is_native_item_view(b), vm) +} + +pub(crate) fn set_inner_number_or(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult { set_inner_number_op(a, b, |a, b| a.union(b, vm), vm) } diff --git a/crates/vm/src/builtins/filter.rs b/crates/vm/src/builtins/filter.rs index 2bce5c782..d591629e0 100644 --- a/crates/vm/src/builtins/filter.rs +++ b/crates/vm/src/builtins/filter.rs @@ -23,6 +23,7 @@ impl PyPayload for PyFilter { impl Constructor for PyFilter { type Args = (PyObjectRef, PyIter); + const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true; fn py_new( _cls: &Py, diff --git a/crates/vm/src/builtins/float.rs b/crates/vm/src/builtins/float.rs index 8eadc0c04..154e19e63 100644 --- a/crates/vm/src/builtins/float.rs +++ b/crates/vm/src/builtins/float.rs @@ -176,9 +176,26 @@ impl Constructor for PyFloat { type Args = OptionalArg; fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { + let float_type = vm.ctx.types.float_type; + let uses_float_init = { + let cls_init = cls.slots.init.load().map(crate::types::fn_addr); + let float_init = float_type.slots.init.load().map(crate::types::fn_addr); + cls_init == float_init + }; // Bind before the fast path so FromArgs::arity decides how many arguments - // are acceptable, rather than a count repeated here. - let arg: Self::Args = args.bind_for(vm, Self::NAME)?; + // are acceptable, rather than a count repeated here. Extra keywords are + // accepted only when a subclass has replaced tp_init. + let arg: Self::Args = if cls.is(float_type) || uses_float_init { + args.bind_for(vm, Self::NAME)? + } else { + match args.args.as_slice() { + [] => OptionalArg::Missing, + [value] => OptionalArg::Present(value.clone()), + slice => { + return Err(vm.new_arity_type_error(Self::NAME, 0..=1, slice.len())); + } + } + }; // Optimization: return exact float as-is if cls.is(vm.ctx.types.float_type) diff --git a/crates/vm/src/builtins/frame.rs b/crates/vm/src/builtins/frame.rs index 721c1b1d5..045375f1a 100644 --- a/crates/vm/src/builtins/frame.rs +++ b/crates/vm/src/builtins/frame.rs @@ -523,42 +523,41 @@ impl FrameObject { } }; - let what_event = vm.what_event.get(); - if what_event < 0 { + let Some(what_event) = vm.what_event.get() else { return Err( vm.new_value_error("f_lineno can only be set in a trace function".to_owned()) ); - } + }; { - use crate::stdlib::sys::monitoring as mon; + use crate::stdlib::sys::monitoring::MonitoringEvent as Ev; match what_event { - mon::WHAT_PY_RESUME - | mon::WHAT_JUMP - | mon::WHAT_BRANCH - | mon::WHAT_BRANCH_LEFT - | mon::WHAT_BRANCH_RIGHT - | mon::WHAT_LINE - | mon::WHAT_PY_YIELD => {} - mon::WHAT_PY_START => { + Ev::PyResume + | Ev::Jump + | Ev::Branch + | Ev::BranchLeft + | Ev::BranchRight + | Ev::Line + | Ev::PyYield => {} + Ev::PyStart => { return Err(vm.new_value_error( "can't jump from the 'call' trace event of a new frame".to_owned(), )); } - mon::WHAT_CALL | mon::WHAT_C_RETURN => { + Ev::Call | Ev::CReturn => { return Err(vm.new_value_error("can't jump during a call".to_owned())); } - mon::WHAT_PY_RETURN - | mon::WHAT_PY_UNWIND - | mon::WHAT_PY_THROW - | mon::WHAT_RAISE - | mon::WHAT_C_RAISE - | mon::WHAT_INSTRUCTION - | mon::WHAT_EXCEPTION_HANDLED => { + Ev::PyReturn + | Ev::PyUnwind + | Ev::PyThrow + | Ev::Raise + | Ev::CRaise + | Ev::Instruction + | Ev::ExceptionHandled => { return Err( vm.new_value_error("can only jump from a 'line' trace event".to_owned()) ); } - _ => { + Ev::StopIteration | Ev::Reraise => { return Err(vm.new_system_error("unexpected event type".to_owned())); } } diff --git a/crates/vm/src/builtins/list.rs b/crates/vm/src/builtins/list.rs index 537f0ca67..8cff4500c 100644 --- a/crates/vm/src/builtins/list.rs +++ b/crates/vm/src/builtins/list.rs @@ -11,7 +11,7 @@ use crate::object::{Traverse, TraverseFn}; use crate::{ AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, builtins::{PyFloat, PyInt, PyStr, PyTuple}, - class::PyClassImpl, + class::{PyClassDef, PyClassImpl}, convert::ToPyObject, function::{ArgSize, Either, FuncArgs, OptionalArg, PyComparisonValue}, protocol::{PyIterReturn, PyMappingMethods, PySequenceMethods}, @@ -484,6 +484,28 @@ impl Constructor for PyList { impl Initializer for PyList { type Args = OptionalArg; + fn slot_init(zelf: PyObjectRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult<()> { + let list_type = vm.ctx.types.list_type; + let cls = zelf.class().to_owned(); + let uses_list_new = { + let cls_new = cls.slots.new.load().map(crate::types::fn_addr); + let list_new = list_type.slots.new.load().map(crate::types::fn_addr); + cls_new == list_new + }; + let iterable = if cls.is(list_type) || uses_list_new { + args.bind_for(vm, Self::NAME)? + } else { + match args.args.as_slice() { + [] => OptionalArg::Missing, + [iterable] => OptionalArg::Present(iterable.clone()), + slice => { + return Err(vm.new_arity_type_error(Self::NAME, 0..=1, slice.len())); + } + } + }; + Self::init(zelf.try_into_value(vm)?, iterable, vm) + } + fn init(zelf: PyRef, iterable: Self::Args, vm: &VirtualMachine) -> PyResult<()> { let mut elements = if let OptionalArg::Present(iterable) = iterable { vm.extract_elements_sized(&iterable, &|| 0, Ok)? diff --git a/crates/vm/src/builtins/set.rs b/crates/vm/src/builtins/set.rs index d9a82f337..1ae920b86 100644 --- a/crates/vm/src/builtins/set.rs +++ b/crates/vm/src/builtins/set.rs @@ -655,10 +655,15 @@ impl PySet { self.inner.len() } - pub fn __contains__(&self, needle: &PyObject, vm: &VirtualMachine) -> PyResult { + pub fn contains(&self, needle: &PyObject, vm: &VirtualMachine) -> PyResult { self.inner.contains(needle, vm) } + #[pymethod(coexist)] + fn __contains__(&self, needle: PyObjectRef, vm: &VirtualMachine) -> PyResult { + self.contains(&needle, vm) + } + #[pymethod] fn __sizeof__(&self) -> usize { core::mem::size_of::() + self.inner.sizeof() @@ -914,7 +919,7 @@ impl AsSequence for PySet { static AS_SEQUENCE: LazyLock = LazyLock::new(|| PySequenceMethods { length: atomic_func!(|seq, _vm| Ok(PySet::sequence_downcast(seq).__len__())), contains: atomic_func!( - |seq, needle, vm| PySet::sequence_downcast(seq).__contains__(needle, vm) + |seq, needle, vm| PySet::sequence_downcast(seq).contains(needle, vm) ), ..PySequenceMethods::NOT_IMPLEMENTED }); @@ -1148,10 +1153,15 @@ impl PyFrozenSet { self.inner.len() } - pub fn __contains__(&self, needle: &PyObject, vm: &VirtualMachine) -> PyResult { + pub fn contains(&self, needle: &PyObject, vm: &VirtualMachine) -> PyResult { self.inner.contains(needle, vm) } + #[pymethod(coexist)] + fn __contains__(&self, needle: PyObjectRef, vm: &VirtualMachine) -> PyResult { + self.contains(&needle, vm) + } + #[pymethod] fn __sizeof__(&self) -> usize { core::mem::size_of::() + self.inner.sizeof() @@ -1309,7 +1319,7 @@ impl AsSequence for PyFrozenSet { static AS_SEQUENCE: LazyLock = LazyLock::new(|| PySequenceMethods { length: atomic_func!(|seq, _vm| Ok(PyFrozenSet::sequence_downcast(seq).__len__())), contains: atomic_func!( - |seq, needle, vm| PyFrozenSet::sequence_downcast(seq).__contains__(needle, vm) + |seq, needle, vm| PyFrozenSet::sequence_downcast(seq).contains(needle, vm) ), ..PySequenceMethods::NOT_IMPLEMENTED }); diff --git a/crates/vm/src/builtins/str.rs b/crates/vm/src/builtins/str.rs index 10d891b6a..cae410673 100644 --- a/crates/vm/src/builtins/str.rs +++ b/crates/vm/src/builtins/str.rs @@ -1170,7 +1170,11 @@ impl PyStr { } #[pymethod] - fn splitlines(&self, args: anystr::SplitLinesArgs, vm: &VirtualMachine) -> Vec { + pub(crate) fn splitlines( + &self, + args: anystr::SplitLinesArgs, + vm: &VirtualMachine, + ) -> Vec { let into_wrapper = |s: &Wtf8| self.new_substr(s.to_owned()).to_pyobject(vm); let mut elements = Vec::new(); let mut last_i = 0; diff --git a/crates/vm/src/builtins/tuple.rs b/crates/vm/src/builtins/tuple.rs index 35121d083..b4edba7d9 100644 --- a/crates/vm/src/builtins/tuple.rs +++ b/crates/vm/src/builtins/tuple.rs @@ -221,7 +221,23 @@ impl Constructor for PyTuple { type Args = Vec; fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { - let iterable: OptionalArg = args.bind_for(vm, Self::NAME)?; + let tuple_type = vm.ctx.types.tuple_type; + let uses_tuple_init = { + let cls_init = cls.slots.init.load().map(crate::types::fn_addr); + let tuple_init = tuple_type.slots.init.load().map(crate::types::fn_addr); + cls_init == tuple_init + }; + let iterable: OptionalArg = if cls.is(tuple_type) || uses_tuple_init { + args.bind_for(vm, Self::NAME)? + } else { + match args.args.as_slice() { + [] => OptionalArg::Missing, + [iterable] => OptionalArg::Present(iterable.clone()), + slice => { + return Err(vm.new_arity_type_error(Self::NAME, 0..=1, slice.len())); + } + } + }; // Optimizations for exact tuple type if cls.is(vm.ctx.types.tuple_type) { diff --git a/crates/vm/src/builtins/type.rs b/crates/vm/src/builtins/type.rs index bb2b4aef5..b74e0d465 100644 --- a/crates/vm/src/builtins/type.rs +++ b/crates/vm/src/builtins/type.rs @@ -24,8 +24,9 @@ use crate::{ object::{Traverse, TraverseFn}, protocol::{PyIterReturn, PyNumberMethods}, types::{ - AsNumber, Callable, Constructor, GetAttr, Initializer, PyTypeFlags, PyTypeSlots, - Representable, SLOT_DEFS, SetAttr, TypeDataRef, TypeDataRefMut, TypeDataSlot, + AsNumber, Callable, Constructor, GetAttr, Initializer, NewFunc, PyTypeFlags, PyTypeSlots, + Representable, SLOT_DEFS, SetAttr, TypeDataRef, TypeDataRefMut, TypeDataSlot, fn_addr, + new_wrapper, }, }; use core::{ @@ -244,7 +245,10 @@ unsafe impl crate::object::Traverse for PyType { if let Some(base) = self.base.deref() { tracer_fn(base.as_object()); } - tracer_fn(self.bases.read_recursive().as_untyped().as_object()); + // Skip when a writer holds `bases` (same rule as `Traverse for PyRwLock`). + if let Some(bases) = self.bases.try_read_recursive() { + tracer_fn(bases.as_untyped().as_object()); + } self.mro.traverse(tracer_fn); self.subclasses.traverse(tracer_fn); self.attributes.traverse(tracer_fn); @@ -259,8 +263,11 @@ unsafe impl crate::object::Traverse for PyType { if let Some(base) = unsafe { self.base.swap(None) } { out.push(base.into()); } + // Clone the empty tuple before taking the write lock: `object`'s + // `bases` is this same lock, and `.read()` while exclusive panics + // on CellRwLock (and hangs on parking_lot). + let empty = object::PyBaseObject::static_type().bases.read().clone(); if let Some(mut bases) = self.bases.try_write() { - let empty = object::PyBaseObject::static_type().bases.read().clone(); let old_bases = core::mem::replace(&mut *bases, empty); out.push(old_bases.into_untyped().into()); } @@ -410,7 +417,11 @@ pub enum TypeNamespace { unsafe impl Traverse for TypeNamespace { fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { match self { - Self::Attributes(attrs) => attrs.read_recursive().traverse(tracer_fn), + Self::Attributes(attrs) => { + if let Some(attrs) = attrs.try_read_recursive() { + attrs.traverse(tracer_fn); + } + } Self::Dict(dict) => tracer_fn(dict.as_object()), } } @@ -474,18 +485,7 @@ impl TypeNamespace { /// Bind `name` to `value`, dropping whatever it displaced. pub fn set(&self, name: &'static PyStrInterned, value: PyObjectRef) { - match self { - Self::Attributes(attrs) => { - attrs.write().insert(name, value); - } - Self::Dict(dict) => { - if let Some(Err(_)) | None = - crate::vm::thread::try_with_current_vm(|vm| dict.set_item(name, value, vm)) - { - debug_assert!(false, "type namespace write without a running VM"); - } - } - } + drop(self.insert(name, value)); } /// Bind `name` to `value` and hand back what it displaced, so the caller @@ -495,7 +495,11 @@ impl TypeNamespace { Self::Attributes(attrs) => attrs.write().insert(name, value), Self::Dict(dict) => { let previous = dict_get(dict, name); - self.set(name, value); + if let Some(Err(_)) | None = + crate::vm::thread::try_with_current_vm(|vm| dict.set_item(name, value, vm)) + { + debug_assert!(false, "type namespace write without a running VM"); + } previous } } @@ -881,6 +885,16 @@ impl PyType { PyTypeFlags::from_bits_truncate(self.abc_tpflags.load(Ordering::Acquire)).contains(flag) } + pub fn set_is_abstract(&self, is_abstract: bool) { + const MASK: u64 = PyTypeFlags::IS_ABSTRACT.bits(); + let _ = self + .abc_tpflags + .try_update(Ordering::AcqRel, Ordering::Acquire, |old| { + Some(if is_abstract { old | MASK } else { old & !MASK }) + }); + self.modified(); + } + pub fn set_abc_collection_flags_recursive(&self, flags: PyTypeFlags) { const COLLECTION_FLAGS: PyTypeFlags = PyTypeFlags::from_bits_truncate( PyTypeFlags::SEQUENCE.bits() | PyTypeFlags::MAPPING.bits(), @@ -1115,11 +1129,14 @@ impl PyType { // Static types are not tracked by GC. // They are immortal and never participate in collectable cycles. - unsafe { - crate::gc_state::gc_state() - .untrack_object(core::ptr::NonNull::from(new_type.as_object())); + // Heap types from PyType_FromSpec stay tracked. + if !new_type.slots.flags.has_feature(PyTypeFlags::HEAPTYPE) { + unsafe { + crate::gc_state::gc_state() + .untrack_object(core::ptr::NonNull::from(new_type.as_object())); + } + new_type.as_object().clear_gc_tracked(); } - new_type.as_object().clear_gc_tracked(); new_type.mro.write().insert(0, new_type.clone()); @@ -1889,8 +1906,50 @@ impl PyType { } #[pygetset] - const fn __flags__(&self) -> u64 { + fn __flags__(&self) -> u64 { self.slots.flags.bits() + | (self.abc_tpflags.load(Ordering::Acquire) & PyTypeFlags::IS_ABSTRACT.bits()) + } + + #[pygetset] + fn __abstractmethods__(zelf: &Py, vm: &VirtualMachine) -> PyResult { + if zelf.is(vm.ctx.types.type_type) { + return Err(vm.new_attribute_error("__abstractmethods__")); + } + zelf.get_direct_attr(identifier!(vm, __abstractmethods__)) + .ok_or_else(|| vm.new_attribute_error("__abstractmethods__")) + } + + #[pygetset(setter)] + fn set___abstractmethods__(&self, value: PySetterValue, vm: &VirtualMachine) -> PyResult<()> { + let key = identifier!(vm, __abstractmethods__); + let is_abstract = match &value { + PySetterValue::Assign(val) => val.try_to_bool(vm)?, + PySetterValue::Delete => false, + }; + match value { + PySetterValue::Assign(val) => { + Self::with_type_lock(vm, || { + self.modified_inner(); + let _prev = self.attributes.insert(key, val); + self.set_is_abstract(is_abstract); + }); + } + PySetterValue::Delete => { + let removed = Self::with_type_lock(vm, || { + self.modified_inner(); + let removed = self.attributes.remove(key); + if removed.is_some() { + self.set_is_abstract(false); + } + removed + }); + if removed.is_none() { + return Err(vm.new_attribute_error("__abstractmethods__")); + } + } + } + Ok(()) } #[pygetset] @@ -2292,11 +2351,22 @@ impl PyType { } #[pygetset] - fn __text_signature__(&self) -> Option { - self.slots - .doc - .and_then(|doc| get_text_signature_from_internal_doc(&self.name(), doc)) - .map(|signature| signature.to_string()) + fn __text_signature__(&self, vm: &VirtualMachine) -> Option { + let name = self.name(); + if let Some(doc) = self.slots.doc + && let Some(signature) = get_text_signature_from_internal_doc(&name, doc) + { + return Some(signature.to_string()); + } + if self.slots.flags.has_feature(PyTypeFlags::HEAPTYPE) + && let Some(doc_attr) = self.get_direct_attr(identifier!(vm, __doc__)) + && let Some(doc) = doc_attr.downcast_ref::() + && let Some(doc) = doc.to_str() + && let Some(signature) = get_text_signature_from_internal_doc(&name, doc) + { + return Some(signature.to_string()); + } + None } #[pygetset] @@ -2356,6 +2426,17 @@ impl Constructor for PyType { let (name, bases, dict, kwargs): (PyStrRef, PyTupleRef, PyDictRef, KwArgs) = args.clone().bind_for(vm, Self::NAME)?; + // A mapping that is not an exact dict (e.g. OrderedDict) keeps its + // own iteration order; copy via the mapping protocol so that order + // lands in the type dict. + let dict = if args.args[2].class().is(vm.ctx.types.dict_type) { + dict + } else { + let copied = vm.ctx.new_dict(); + copied.merge_object(args.args[2].clone(), vm)?; + copied + }; + if name.as_bytes().contains(&0) { return Err(vm.new_value_error("type name must not contain null characters")); } @@ -3358,24 +3439,25 @@ pub(crate) fn call_slot_new( return Err(vm.new_type_error(format!("cannot create '{}' instances", subtype.slot_name()))); } - // "is not safe" check (tp_new_wrapper logic) - // Check that the user doesn't do something silly and unsafe like - // object.__new__(dict). To do this, we check that the most derived base - // that's not a heap type is this type. + // "is not safe" check (tp_new_wrapper). Walk past bases whose tp_new is + // new_wrapper so a Python subclass of a native heap type still reaches + // that type's tp_new; reject object.__new__(dict) and similar. let mut staticbase = subtype.clone(); - while staticbase.slots.flags.has_feature(PyTypeFlags::HEAPTYPE) { - if let Some(base) = staticbase.base.to_owned() { - staticbase = base; - } else { - break; + while staticbase + .slots + .new + .load() + .is_some_and(|f| fn_addr(f) == fn_addr(new_wrapper as NewFunc)) + { + match staticbase.base.to_owned() { + Some(base) => staticbase = base, + None => break, } } - // Check if staticbase's tp_new differs from typ's tp_new let typ_new = typ.slots.new.load(); let staticbase_new = staticbase.slots.new.load(); - if typ_new.map(|f| crate::types::fn_addr(f)) != staticbase_new.map(|f| crate::types::fn_addr(f)) - { + if typ_new.map(fn_addr) != staticbase_new.map(fn_addr) { return Err(vm.new_type_error(format!( "{}.__new__({}) is not safe, use {}.__new__()", typ.slot_name(), diff --git a/crates/vm/src/frame.rs b/crates/vm/src/frame.rs index 4428984a7..cf847d82b 100644 --- a/crates/vm/src/frame.rs +++ b/crates/vm/src/frame.rs @@ -29,7 +29,10 @@ use crate::{ protocol::{PyIter, PyIterReturn}, scope::Scope, sliceable::SliceableSequenceOp, - stdlib::{_typing, builtins, sys::monitoring}, + stdlib::{ + _typing, builtins, + sys::monitoring::{self, MonitoringEvent}, + }, types::{PyComparisonOp, PyTypeFlags}, vm::{Context, PyMethod}, }; @@ -2675,7 +2678,7 @@ pub(crate) fn trampoline_handle_exception( exec.fire_exception_trace(exception, vm)?; let exception = { let mon_events = vm.state.monitoring_events.load(); - if mon_events & monitoring::EVENT_RAISE != 0 { + if mon_events & MonitoringEvent::Raise.mask() != 0 { let offset = idx as u32 * 2; let exc_obj: PyObjectRef = exception.to_owned().into(); match monitoring::fire_raise(vm, exec.code, offset, &exc_obj) { @@ -3327,9 +3330,9 @@ impl ExecutingFrame<'_> { } // RESUME oparg 0 is PY_START; nonzero is PY_RESUME. let what = if resume_type == 0 { - monitoring::WHAT_PY_START + monitoring::MonitoringEvent::PyStart } else { - monitoring::WHAT_PY_RESUME + monitoring::MonitoringEvent::PyResume }; let trace_result = vm.trace_event_what(crate::protocol::TraceEvent::Call, what, None)?; if let Some(local_trace) = trace_result { @@ -3449,7 +3452,7 @@ impl ExecutingFrame<'_> { let value = self.top_value().to_owned(); vm.trace_event_what( crate::protocol::TraceEvent::Return, - monitoring::WHAT_PY_YIELD, + monitoring::MonitoringEvent::PyYield, Some(value), )?; if self.lasti() != lasti_before { @@ -3457,7 +3460,7 @@ impl ExecutingFrame<'_> { return Ok(true); } } - if self.monitoring_mask & monitoring::EVENT_PY_YIELD != 0 { + if self.monitoring_mask & MonitoringEvent::PyYield.mask() != 0 { let value = self.top_value().to_owned(); let offset = (self.lasti() - 1) * 2; monitoring::fire_py_yield(vm, self.code, offset, &value)?; @@ -3852,7 +3855,7 @@ impl ExecutingFrame<'_> { let exception = { let mon_events = vm.state.monitoring_events.load(); if is_reraise { - if mon_events & monitoring::EVENT_RERAISE != 0 { + if mon_events & MonitoringEvent::Reraise.mask() != 0 { let offset = idx as u32 * 2; let exc_obj: PyObjectRef = exception.clone().into(); match monitoring::fire_reraise(vm, self.code, offset, &exc_obj) { @@ -3862,7 +3865,7 @@ impl ExecutingFrame<'_> { } else { exception } - } else if mon_events & monitoring::EVENT_RAISE != 0 { + } else if mon_events & MonitoringEvent::Raise.mask() != 0 { let offset = idx as u32 * 2; let exc_obj: PyObjectRef = exception.clone().into(); match monitoring::fire_raise(vm, self.code, offset, &exc_obj) { @@ -3889,7 +3892,7 @@ impl ExecutingFrame<'_> { Err(exception) => { // Fire PY_UNWIND: exception escapes this frame let exception = if vm.state.monitoring_events.load() - & monitoring::EVENT_PY_UNWIND + & MonitoringEvent::PyUnwind.mask() != 0 { let offset = idx as u32 * 2; @@ -4113,7 +4116,7 @@ impl ExecutingFrame<'_> { // Fire PY_THROW and RAISE events before raising the exception. // If a monitoring callback fails, its exception replaces the original. if vm.use_tracing.get() { - let what = monitoring::WHAT_PY_THROW; + let what = monitoring::MonitoringEvent::PyThrow; if let Some(local_trace) = vm.trace_event_what(crate::protocol::TraceEvent::Call, what, None)? { @@ -4123,7 +4126,7 @@ impl ExecutingFrame<'_> { } let exception = { let mon_events = vm.state.monitoring_events.load(); - let exception = if mon_events & monitoring::EVENT_PY_THROW != 0 { + let exception = if mon_events & MonitoringEvent::PyThrow.mask() != 0 { let offset = idx as u32 * 2; let exc_obj: PyObjectRef = exception.clone().into(); match monitoring::fire_py_throw(vm, self.code, offset, &exc_obj) { @@ -4133,7 +4136,7 @@ impl ExecutingFrame<'_> { } else { exception }; - if mon_events & monitoring::EVENT_RAISE != 0 { + if mon_events & MonitoringEvent::Raise.mask() != 0 { let offset = idx as u32 * 2; let exc_obj: PyObjectRef = exception.clone().into(); match monitoring::fire_raise(vm, self.code, offset, &exc_obj) { @@ -4171,7 +4174,7 @@ impl ExecutingFrame<'_> { Err(exception) => { // Fire PY_UNWIND: exception escapes the generator frame. let exception = - if vm.state.monitoring_events.load() & monitoring::EVENT_PY_UNWIND != 0 { + if vm.state.monitoring_events.load() & MonitoringEvent::PyUnwind.mask() != 0 { let offset = idx as u32 * 2; let exc_obj: PyObjectRef = exception.clone().into(); match monitoring::fire_py_unwind(vm, self.code, offset, &exc_obj) { @@ -5256,7 +5259,7 @@ impl ExecutingFrame<'_> { ))); } }; - if seen_attrs.__contains__(attr_name.as_object(), vm)? { + if seen_attrs.contains(attr_name.as_object(), vm)? { let type_name = type_name(); let attr_repr = attr_name.as_object().repr(vm)?; return Err(vm.new_type_error(format!( @@ -5309,7 +5312,7 @@ impl ExecutingFrame<'_> { // Extract keyword attributes for name in kwd_attrs { let name_str = name.downcast_ref::().unwrap(); - if seen_attrs.__contains__(name_str.as_object(), vm)? { + if seen_attrs.contains(name_str.as_object(), vm)? { let type_name = type_name(); let attr_repr = name.as_object().repr(vm)?; return Err(vm.new_type_error(format!( @@ -5364,7 +5367,7 @@ impl ExecutingFrame<'_> { .new_base_object(vm.ctx.types.object_type.to_owned(), None); for key in keys { - if seen_keys.__contains__(key.as_object(), vm)? { + if seen_keys.contains(key.as_object(), vm)? { return Err(vm.new_value_error(format!( "mapping pattern checks duplicate key ({})", key.as_object().repr(vm)? @@ -5386,7 +5389,7 @@ impl ExecutingFrame<'_> { } else { // Fallback if .get() method is not available (shouldn't happen for mappings) for key in keys { - if seen_keys.__contains__(key.as_object(), vm)? { + if seen_keys.contains(key.as_object(), vm)? { return Err(vm.new_value_error(format!( "mapping pattern checks duplicate key ({})", key.as_object().repr(vm)? @@ -5556,7 +5559,7 @@ impl ExecutingFrame<'_> { if vm.use_tracing.get() { vm.trace_event_what( crate::protocol::TraceEvent::Return, - monitoring::WHAT_PY_RETURN, + monitoring::MonitoringEvent::PyReturn, Some(value.clone()), )?; } @@ -7876,7 +7879,8 @@ impl ExecutingFrame<'_> { Ok(PyIterReturn::StopIteration(value)) => { // FOR_ITER_GEN returns through END_FOR, which // fires STOP_ITERATION rather than RAISE. - if vm.state.monitoring_events.load() & monitoring::EVENT_STOP_ITERATION + if vm.state.monitoring_events.load() + & MonitoringEvent::StopIteration.mask() != 0 { let offset = (self.lasti() - 1) * 2; @@ -8015,10 +8019,10 @@ impl ExecutingFrame<'_> { let resume_type = u32::from(arg); let offset = (self.lasti() - 1) * 2; if resume_type == 0 { - if self.monitoring_mask & monitoring::EVENT_PY_START != 0 { + if self.monitoring_mask & MonitoringEvent::PyStart.mask() != 0 { monitoring::fire_py_start(vm, self.code, offset)?; } - } else if self.monitoring_mask & monitoring::EVENT_PY_RESUME != 0 { + } else if self.monitoring_mask & MonitoringEvent::PyResume.mask() != 0 { monitoring::fire_py_resume(vm, self.code, offset)?; } self.trace_call_from_resume(vm, resume_type)?; @@ -8029,11 +8033,11 @@ impl ExecutingFrame<'_> { if vm.use_tracing.get() { vm.trace_event_what( crate::protocol::TraceEvent::Return, - monitoring::WHAT_PY_RETURN, + monitoring::MonitoringEvent::PyReturn, Some(value.clone()), )?; } - if self.monitoring_mask & monitoring::EVENT_PY_RETURN != 0 { + if self.monitoring_mask & MonitoringEvent::PyReturn.mask() != 0 { let offset = (self.lasti() - 1) * 2; monitoring::fire_py_return(vm, self.code, offset, &value)?; } @@ -8070,7 +8074,7 @@ impl ExecutingFrame<'_> { let oparg = bytecode::LoadSuperAttr::from(u32::from(arg)); let offset = (self.lasti() - 1) * 2; // Fire CALL event before super() call - let call_args = if self.monitoring_mask & monitoring::EVENT_CALL != 0 { + let call_args = if self.monitoring_mask & MonitoringEvent::Call.mask() != 0 { let global_super: PyObjectRef = self.nth_value(2).to_owned(); let arg0 = if oparg.has_class() { self.nth_value(1).to_owned() @@ -8115,7 +8119,7 @@ impl ExecutingFrame<'_> { let target_idx = self.lasti() + u32::from(arg); let target = bytecode::Label::from_u32(target_idx); self.jump(target); - if self.monitoring_mask & monitoring::EVENT_JUMP != 0 { + if self.monitoring_mask & MonitoringEvent::Jump.mask() != 0 { monitoring::fire_jump(vm, self.code, src_offset, target.as_u32() * 2)?; } Ok(None) @@ -8126,7 +8130,7 @@ impl ExecutingFrame<'_> { let target_idx = self.lasti() + 1 - u32::from(arg); let target = bytecode::Label::from_u32(target_idx); self.jump(target); - if self.monitoring_mask & monitoring::EVENT_JUMP != 0 { + if self.monitoring_mask & MonitoringEvent::Jump.mask() != 0 { monitoring::fire_jump(vm, self.code, src_offset, target.as_u32() * 2)?; } self.trace_backward_same_line(from_idx, vm)?; @@ -8137,11 +8141,11 @@ impl ExecutingFrame<'_> { let target = bytecode::Label::from_u32(self.lasti() + 1 + u32::from(arg)); let continued = self.execute_for_iter(vm, target)?; if continued { - if self.monitoring_mask & monitoring::EVENT_BRANCH_LEFT != 0 { + if self.monitoring_mask & MonitoringEvent::BranchLeft.mask() != 0 { let dest_offset = (self.lasti() + 1) * 2; // after caches monitoring::fire_branch_left(vm, self.code, src_offset, dest_offset)?; } - } else if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 { + } else if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 { // INSTRUMENTED_POP_ITER: dest is the instruction after // POP_ITER (FOR_ITER jumps over END_FOR onto POP_ITER). let dest_offset = (self.lasti() + 1) * 2; @@ -8157,7 +8161,7 @@ impl ExecutingFrame<'_> { .downcast_ref::() .is_some(); let value = self.pop_value(); - if is_gen && self.monitoring_mask & monitoring::EVENT_STOP_ITERATION != 0 { + if is_gen && self.monitoring_mask & MonitoringEvent::StopIteration.mask() != 0 { let offset = (self.lasti() - 1) * 2; monitoring::fire_stop_iteration(vm, self.code, offset, &value)?; } @@ -8173,7 +8177,9 @@ impl ExecutingFrame<'_> { || receiver .downcast_ref::() .is_some(); - if is_gen_or_coro && self.monitoring_mask & monitoring::EVENT_STOP_ITERATION != 0 { + if is_gen_or_coro + && self.monitoring_mask & MonitoringEvent::StopIteration.mask() != 0 + { let offset = (self.lasti() - 1) * 2; monitoring::fire_stop_iteration(vm, self.code, offset, &value)?; } @@ -8187,7 +8193,7 @@ impl ExecutingFrame<'_> { let value = obj.try_to_bool(vm)?; if value { self.jump(bytecode::Label::from_u32(target_idx)); - if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 { + if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 { monitoring::fire_branch_right(vm, self.code, src_offset, target_idx * 2)?; } } @@ -8200,7 +8206,7 @@ impl ExecutingFrame<'_> { let value = obj.try_to_bool(vm)?; if !value { self.jump(bytecode::Label::from_u32(target_idx)); - if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 { + if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 { monitoring::fire_branch_right(vm, self.code, src_offset, target_idx * 2)?; } } @@ -8212,7 +8218,7 @@ impl ExecutingFrame<'_> { let value = self.pop_value(); if vm.is_none(&value) { self.jump(bytecode::Label::from_u32(target_idx)); - if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 { + if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 { monitoring::fire_branch_right(vm, self.code, src_offset, target_idx * 2)?; } } @@ -8224,14 +8230,14 @@ impl ExecutingFrame<'_> { let value = self.pop_value(); if !vm.is_none(&value) { self.jump(bytecode::Label::from_u32(target_idx)); - if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 { + if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 { monitoring::fire_branch_right(vm, self.code, src_offset, target_idx * 2)?; } } Ok(None) } Instruction::InstrumentedNotTaken => { - if self.monitoring_mask & monitoring::EVENT_BRANCH_LEFT != 0 { + if self.monitoring_mask & MonitoringEvent::BranchLeft.mask() != 0 { let not_taken_idx = self.lasti() as usize - 1; // Scan backwards past CACHE entries to find the branch instruction let mut branch_idx = not_taken_idx.saturating_sub(1); @@ -8255,7 +8261,7 @@ impl ExecutingFrame<'_> { Ok(None) } Instruction::InstrumentedEndAsyncFor => { - if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 { + if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 { let oparg_val = u32::from(arg); // src = next_instr - oparg (END_SEND position) let src_offset = (self.lasti() - oparg_val) * 2; @@ -8607,7 +8613,8 @@ impl ExecutingFrame<'_> { // Fire EXCEPTION_HANDLED before setting up handler. // If the callback raises, the handler is NOT set up and the // new exception propagates instead. - if vm.state.monitoring_events.load() & monitoring::EVENT_EXCEPTION_HANDLED != 0 + if vm.state.monitoring_events.load() & MonitoringEvent::ExceptionHandled.mask() + != 0 { let byte_offset = offset * 2; let exc_obj: PyObjectRef = exception.clone().into(); @@ -9011,7 +9018,7 @@ impl ExecutingFrame<'_> { let is_python_call = callable.downcast_ref_if_exact::(vm).is_some(); // Fire CALL event - let call_arg0 = if self.monitoring_mask & monitoring::EVENT_CALL != 0 { + let call_arg0 = if self.monitoring_mask & MonitoringEvent::Call.mask() != 0 { let arg0 = final_args .args .first() @@ -9324,7 +9331,7 @@ impl ExecutingFrame<'_> { { return Ok(()); } - let need_raise = vm.state.monitoring_events.load() & monitoring::EVENT_RAISE != 0; + let need_raise = vm.state.monitoring_events.load() & MonitoringEvent::Raise.mask() != 0; let need_trace = vm.use_tracing.get() && self.trace_is_set(vm); if !need_raise && !need_trace { return Ok(()); diff --git a/crates/vm/src/function/method.rs b/crates/vm/src/function/method.rs index c1a947c8d..d76fb31e7 100644 --- a/crates/vm/src/function/method.rs +++ b/crates/vm/src/function/method.rs @@ -3,7 +3,7 @@ use crate::{ builtins::{ PyType, builtin_func::{PyNativeFunction, PyNativeMethod}, - descriptor::PyMethodDescriptor, + descriptor::{PyClassMethodDescriptor, PyMethodDescriptor}, }, class::PyClassDef, function::{IntoPyNativeFn, PyNativeFn}, @@ -202,12 +202,9 @@ impl PyMethodDef { &'static self, ctx: &Context, class: &'static Py, - ) -> PyRef { - PyRef::new_ref( - self.to_method(class, ctx), - ctx.types.method_descriptor_type.to_owned(), - None, - ) + ) -> PyRef { + debug_assert!(self.flags.contains(PyMethodFlags::CLASS)); + PyClassMethodDescriptor::new(self, class, ctx).into_ref(ctx) } pub fn build_staticmethod( diff --git a/crates/vm/src/import.rs b/crates/vm/src/import.rs index dce9f9781..97c0e7a3f 100644 --- a/crates/vm/src/import.rs +++ b/crates/vm/src/import.rs @@ -233,7 +233,11 @@ fn remove_importlib_frames_inner( let (inner_tb, mut now_in_importlib) = remove_importlib_frames_inner(vm, traceback.next.lock().clone(), always_trim); - if file_name == "_frozen_importlib" || file_name == "_frozen_importlib_external" { + if file_name == "" + || file_name == "" + || file_name == "_frozen_importlib" + || file_name == "_frozen_importlib_external" + { if traceback.frame.iframe().code().obj_name.as_str() == "_call_with_frames_removed" { now_in_importlib = true; } diff --git a/crates/vm/src/intern.rs b/crates/vm/src/intern.rs index 82c2d347b..bc8209397 100644 --- a/crates/vm/src/intern.rs +++ b/crates/vm/src/intern.rs @@ -54,8 +54,9 @@ impl StringPool { #[cold] fn miss(zelf: &StringPool, s: PyRefExact) -> &'static PyStrInterned { let cache = CachedPyStrRef { inner: s }; - let inserted = zelf.inner.write().insert(cache.clone()); - if inserted { + let mut inner = zelf.inner.write(); + if inner.insert(cache.clone()) { + drop(inner); let interned = unsafe { cache.as_interned_str() }; // `mark_intern` also makes the object immortal: the pool // never gives an entry up and its refcount could already never @@ -67,8 +68,7 @@ impl StringPool { interned } else { unsafe { - zelf.inner - .read() + inner .get(cache.as_ref()) .expect("inserted is false") .as_interned_str() diff --git a/crates/vm/src/protocol/callable.rs b/crates/vm/src/protocol/callable.rs index 82c57633a..0bcd756d1 100644 --- a/crates/vm/src/protocol/callable.rs +++ b/crates/vm/src/protocol/callable.rs @@ -179,17 +179,17 @@ impl TraceEvent { /// Default `what_event` for this legacy event. #[must_use] - const fn default_what(self) -> i32 { - use crate::stdlib::sys::monitoring as mon; + const fn default_what(self) -> crate::stdlib::sys::monitoring::MonitoringEvent { + use crate::stdlib::sys::monitoring::MonitoringEvent as Ev; match self { - Self::Call => mon::WHAT_PY_START, - Self::Return => mon::WHAT_PY_RETURN, - Self::Exception => mon::WHAT_RAISE, - Self::Line => mon::WHAT_LINE, - Self::Opcode => mon::WHAT_INSTRUCTION, - Self::CCall => mon::WHAT_CALL, - Self::CReturn => mon::WHAT_C_RETURN, - Self::CException => mon::WHAT_C_RAISE, + Self::Call => Ev::PyStart, + Self::Return => Ev::PyReturn, + Self::Exception => Ev::Raise, + Self::Line => Ev::Line, + Self::Opcode => Ev::Instruction, + Self::CCall => Ev::Call, + Self::CReturn => Ev::CReturn, + Self::CException => Ev::CRaise, } } } @@ -235,11 +235,11 @@ impl VirtualMachine { pub(crate) fn trace_event_what( &self, event: TraceEvent, - what: i32, + what: crate::stdlib::sys::monitoring::MonitoringEvent, arg: Option, ) -> PyResult> { if self.use_tracing.get() && !self.tracing_is_suppressed() { - let old = self.what_event.replace(what); + let old = self.what_event.replace(Some(what)); let result = self._trace_event_inner(event, arg); self.what_event.set(old); result diff --git a/crates/vm/src/signal.rs b/crates/vm/src/signal.rs index 5395580b0..bf2aa7650 100644 --- a/crates/vm/src/signal.rs +++ b/crates/vm/src/signal.rs @@ -13,13 +13,40 @@ use crate::{PyObjectRef, PyResult, TryFromBorrowedObject, TryFromObject, Virtual pub(crate) const NSIG: usize = 64; -/// Eval-breaker word: bit flags checked once per bytecode instruction. +#[cfg(not(feature = "threading"))] +bitflagset::bitflag! { + /// Eval-breaker bits checked once per bytecode instruction. + #[derive(Copy, Clone, Debug, PartialEq, Eq)] + #[repr(u8)] + enum EvalBreakerFlag { + Signal = 0, + } +} + +#[cfg(feature = "threading")] +bitflagset::bitflag! { + /// Eval-breaker bits checked once per bytecode instruction. + #[derive(Copy, Clone, Debug, PartialEq, Eq)] + #[repr(u8)] + enum EvalBreakerFlag { + Signal = 0, + Qsbr = 1, + Gc = 2, + Stop = 3, + Finalizing = 4, + } +} + +bitflagset::bitflagset! { + #[derive(Copy, Clone, PartialEq, Eq)] + struct EvalBreakerBits(u8): EvalBreakerFlag +} + +bitflagset::atomic_bitflagset!(struct EvalBreaker(AtomicU8) on EvalBreakerBits); + /// Signal handlers and QSBR set bits with fetch_or (async-signal-safe, /// lock-free); consumers clear only their own bit with fetch_and. -static EVAL_BREAKER: AtomicU8 = AtomicU8::new(0); - -/// A signal handler recorded a pending signal. -const SIGNAL_BIT: u8 = 1 << 0; +static EVAL_BREAKER: EvalBreaker = EvalBreaker::new(); #[expect( clippy::declare_interior_mutable_const, @@ -55,12 +82,12 @@ pub fn check_signals(vm: &VirtualMachine) -> PyResult<()> { // Read-only check first: avoids cache-line invalidation on every // instruction when no signal is pending (the common case). - if EVAL_BREAKER.load(Ordering::Relaxed) & SIGNAL_BIT == 0 { + if !EVAL_BREAKER.contains(&EvalBreakerFlag::Signal) { return Ok(()); } // Atomic RMW only when a signal is actually pending. - if EVAL_BREAKER.fetch_and(!SIGNAL_BIT, Ordering::Acquire) & SIGNAL_BIT == 0 { + if !EVAL_BREAKER.remove(EvalBreakerFlag::Signal) { return Ok(()); } @@ -115,86 +142,70 @@ fn trigger_signals(vm: &VirtualMachine) -> PyResult<()> { pub(crate) fn set_triggered() { // fetch_or (not store) so a signal handler never clobbers the QSBR bit; // this compiles to a lock-free RMW, safe to call from a signal handler. - EVAL_BREAKER.fetch_or(SIGNAL_BIT, Ordering::Release); + EVAL_BREAKER.insert(EvalBreakerFlag::Signal); } /// Any eval-breaker bit pending? One relaxed load; checked per instruction. #[inline(always)] pub(crate) fn eval_breaker_pending() -> bool { - EVAL_BREAKER.load(Ordering::Relaxed) != 0 + !EVAL_BREAKER.is_empty() } /// Extra eval-breaker bits used only when more than one thread can run. #[cfg(feature = "threading")] mod mt { - use super::EVAL_BREAKER; - use core::sync::atomic::Ordering; + use super::{EVAL_BREAKER, EvalBreakerFlag}; /// QSBR has retired allocations pending reclamation. - const QSBR_BIT: u8 = 1 << 1; - /// An automatic collection was scheduled by `maybe_collect` and must run - /// at the next bytecode safepoint rather than synchronously inside the - /// allocation that tripped the threshold. - const GC_BIT: u8 = 1 << 2; - /// At least one thread's `stop_requested` flag may be set (stop-the-world - /// in progress). This bit is shared by every thread rather than being - /// per-thread: the fast (common) path checked once per bytecode - /// instruction becomes a single relaxed load of this word instead of a - /// thread-local lookup plus atomic load of that thread's own flag. Once - /// set, the bit is sticky until `start_the_world`/`reset_after_fork` - /// clears it (both run under the single stop-the-world exclusion, so - /// clearing cannot race a new requester) — every thread takes the slow - /// path for the (short) duration of a stop-the-world request, which only - /// affects programs with more than one live thread; single-threaded - /// programs never set this bit at all, since `stop_the_world` never sets - /// any per-thread `stop_requested` flag when there are no other threads - /// to stop. - const STOP_BIT: u8 = 1 << 3; - /// The interpreter has begun finalizing (see `Interpreter::finalize`). - /// Set once, near process/interpreter shutdown, and never cleared — from - /// that point on every thread takes the slow path once per instruction, - /// which is fine because finalization is a one-time, short-lived, - /// non-performance-sensitive window. - const FINALIZING_BIT: u8 = 1 << 4; - pub(crate) fn set_qsbr_bit() { - EVAL_BREAKER.fetch_or(QSBR_BIT, Ordering::Release); + EVAL_BREAKER.insert(EvalBreakerFlag::Qsbr); } pub(crate) fn clear_qsbr_bit() { - EVAL_BREAKER.fetch_and(!QSBR_BIT, Ordering::Release); + EVAL_BREAKER.remove(EvalBreakerFlag::Qsbr); } pub(crate) fn qsbr_bit_set() -> bool { - EVAL_BREAKER.load(Ordering::Relaxed) & QSBR_BIT != 0 + EVAL_BREAKER.contains(&EvalBreakerFlag::Qsbr) } - /// Record that some thread's `stop_requested` flag was (or may have been) - /// set. See `STOP_BIT`. + /// Record that some thread's `stop_requested` flag may be set. Shared by + /// every thread rather than being per-thread: the fast path checked once + /// per bytecode instruction becomes a single relaxed load of this word. + /// Sticky until `start_the_world`/`reset_after_fork` clears it. pub(crate) fn set_stop_bit() { - EVAL_BREAKER.fetch_or(STOP_BIT, Ordering::Release); + EVAL_BREAKER.insert(EvalBreakerFlag::Stop); } /// Clear the shared stop-the-world bit. Only safe to call from - /// `start_the_world`/`reset_after_fork`, which run under the single - /// stop-the-world exclusion (see `STOP_BIT`). + /// `start_the_world`/`reset_after_fork`. pub(crate) fn clear_stop_bit() { - EVAL_BREAKER.fetch_and(!STOP_BIT, Ordering::Release); + EVAL_BREAKER.remove(EvalBreakerFlag::Stop); } - /// Record that finalization has begun. See `FINALIZING_BIT`. + /// Record that finalization has begun (`Interpreter::finalize`). Set once + /// and never cleared. pub(crate) fn set_finalizing_bit() { - EVAL_BREAKER.fetch_or(FINALIZING_BIT, Ordering::Release); + EVAL_BREAKER.insert(EvalBreakerFlag::Finalizing); } /// Schedule an automatic collection to run at the next bytecode safepoint. pub(crate) fn schedule_gc() { - EVAL_BREAKER.fetch_or(GC_BIT, Ordering::Release); + EVAL_BREAKER.insert(EvalBreakerFlag::Gc); } /// Clear the scheduled-GC bit, returning whether it had been set. pub(crate) fn take_gc_scheduled() -> bool { - EVAL_BREAKER.fetch_and(!GC_BIT, Ordering::Acquire) & GC_BIT != 0 + EVAL_BREAKER.remove(EvalBreakerFlag::Gc) + } + + /// Drop every process-wide eval-breaker bit. Tests that assert a single + /// thread's `stop_requested` must not trip `eval_breaker_pending` have to + /// start from a clean word: cargo's Windows runner shares the process + /// across `#[test]` functions, so a sibling can leave SIGNAL/QSBR/GC/STOP. + #[cfg(test)] + pub(crate) fn clear_eval_breaker_for_test() { + EVAL_BREAKER.clear(); } } @@ -204,11 +215,14 @@ pub(crate) use mt::{ set_stop_bit, take_gc_scheduled, }; +#[cfg(all(test, feature = "threading"))] +pub(crate) use mt::clear_eval_breaker_for_test; + /// Reset all signal trigger state after fork in child process. /// Stale triggers from the parent must not fire in the child. #[cfg(all(unix, feature = "host_env"))] pub(crate) fn clear_after_fork() { - EVAL_BREAKER.fetch_and(!SIGNAL_BIT, Ordering::Release); + EVAL_BREAKER.remove(EvalBreakerFlag::Signal); for trigger in &TRIGGERS { trigger.store(false, Ordering::Relaxed); } diff --git a/crates/vm/src/stdlib/_abc.rs b/crates/vm/src/stdlib/_abc.rs index b818cb91a..ffe0735cd 100644 --- a/crates/vm/src/stdlib/_abc.rs +++ b/crates/vm/src/stdlib/_abc.rs @@ -206,6 +206,11 @@ mod _abc { let data = AbcData::new(); cls.set_attr("_abc_impl", data.to_pyobject(vm), vm)?; + if let Some(cls_type) = cls.downcast_ref::() { + let abstracts = cls.get_attr("__abstractmethods__", vm)?; + cls_type.set_is_abstract(abstracts.try_to_bool(vm)?); + } + Ok(()) } diff --git a/crates/vm/src/stdlib/_collections.rs b/crates/vm/src/stdlib/_collections.rs index 9ec676471..9761d1ddb 100644 --- a/crates/vm/src/stdlib/_collections.rs +++ b/crates/vm/src/stdlib/_collections.rs @@ -1,6 +1,10 @@ +// cspell:ignore odict + pub(crate) use _collections::module_def; -#[pymodule] +pub(crate) mod ordered_dict; + +#[pymodule(with(ordered_dict::ordered_dict))] mod _collections { use crate::{ AsObject, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, @@ -12,7 +16,7 @@ mod _collections { }, common::lock::{PyMutex, PyRwLock, PyRwLockReadGuard, PyRwLockWriteGuard}, convert::ToPyObject, - function::{FuncArgs, KwArgs, OptionalArg, PyComparisonValue}, + function::{FuncArgs, KwArgs, OptionalArg, PyComparisonValue, PySetterValue}, object::{Traverse, TraverseFn}, protocol::{PyIterReturn, PyMappingMethods, PyNumberMethods, PySequenceMethods}, recursion::ReprGuard, @@ -885,23 +889,33 @@ mod _collections { flags(BASETYPE, MAPPING, HAS_DICT) )] impl PyDefaultDict { - #[pygetset] - fn default_factory(&self) -> Option { - self.default_factory.read().clone() + #[pymember(type = "object")] + fn default_factory(vm: &VirtualMachine, zelf: PyObjectRef) -> PyResult { + let zelf: &Py = zelf.try_to_value(vm)?; + Ok(zelf + .default_factory + .read() + .clone() + .unwrap_or_else(|| vm.ctx.none())) } - #[pygetset(name = "default_factory", setter)] - fn default_factory_setter(&self, value: PyObjectRef, vm: &VirtualMachine) { - *self.default_factory.write() = if value.is(&vm.ctx.none()) { - None - } else { - Some(value) + #[pymember(type = "object", setter)] + fn set_default_factory( + vm: &VirtualMachine, + zelf: PyObjectRef, + value: PySetterValue, + ) -> PyResult<()> { + let zelf: &Py = zelf.try_to_value(vm)?; + *zelf.default_factory.write() = match value { + PySetterValue::Assign(v) if !v.is(&vm.ctx.none()) => Some(v), + _ => None, }; + Ok(()) } #[pymethod] fn __missing__(&self, key: PyObjectRef, vm: &VirtualMachine) -> PyResult { - let factory = self.default_factory(); + let factory = self.default_factory.read().clone(); if let Some(f) = factory { let value = f.call((), vm)?; @@ -914,7 +928,7 @@ mod _collections { #[pymethod] #[pymethod(name = "__copy__")] fn copy(&self) -> Self { - let default_factory = self.default_factory(); + let default_factory = self.default_factory.read().clone(); Self { dict: self.dict.copy(), @@ -926,7 +940,7 @@ mod _collections { fn __reduce__(zelf: PyRef, vm: &VirtualMachine) -> PyResult { let cls = zelf.class().to_owned(); - let default_factory = zelf.default_factory(); + let default_factory = zelf.default_factory.read().clone(); let factory_tuple_elements = default_factory.map_or_else(Vec::new, |factory| vec![factory]); let factory_tuple = vm.ctx.new_tuple(factory_tuple_elements); @@ -958,13 +972,13 @@ mod _collections { return not_implemented(); } - (zelf.default_factory(), zelf.dict.copy()) + (zelf.default_factory.read().clone(), zelf.dict.copy()) } else if let Some(zelf) = rhs.downcast_ref::() { let Some(dict) = lhs.downcast_ref::() else { return not_implemented(); }; - (zelf.default_factory(), dict.copy()) + (zelf.default_factory.read().clone(), dict.copy()) } else { return Err(vm.new_type_error(format!( "unsupported operand type(s) for |: '{}' and '{}'", diff --git a/crates/vm/src/stdlib/_collections/ordered_dict.rs b/crates/vm/src/stdlib/_collections/ordered_dict.rs new file mode 100644 index 000000000..56302c8b9 --- /dev/null +++ b/crates/vm/src/stdlib/_collections/ordered_dict.rs @@ -0,0 +1,1237 @@ +// OrderedDict implementation +// cspell:ignore odict + +#[pymodule(sub)] +pub(crate) mod ordered_dict { + use crate::{ + AsObject, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, + atomic_func, + builtins::{ + PyDict, PyGenericAlias, PyMappingProxy, PyTuple, PyTypeRef, + dict::{ + PyDictItems, set_inner_number_or, set_inner_number_subtract, set_inner_number_xor, + set_item_view_number_xor, set_view_number_and, + }, + iter::builtins_iter, + }, + class::StaticType, + common::{hash::PyHash, lock::PyMutex}, + convert::ToPyObject, + dict_inner::DictKey, + function::{ArgIterable, FuncArgs, OptionalArg, PyArithmeticValue, PyComparisonValue}, + object::{Traverse, TraverseFn}, + protocol::{PyIterReturn, PyMappingMethods, PyNumberMethods, PySequenceMethods}, + recursion::ReprGuard, + types::{ + AsMapping, AsNumber, AsSequence, Comparable, Constructor, DefaultConstructor, + Initializer, IterNext, Iterable, PyComparisonOp, Representable, SelfIter, + }, + }; + use std::collections::HashMap; + + const ODICT_ITER_REVERSED: u8 = 1; + const ODICT_ITER_KEYS: u8 = 2; + const ODICT_ITER_VALUES: u8 = 4; + const ODICT_ITER_ITEMS: u8 = ODICT_ITER_KEYS | ODICT_ITER_VALUES; + + #[derive(Debug)] + struct ODictNode { + key: PyObjectRef, + hash: PyHash, + prev: Option, + next: Option, + } + + #[derive(Debug, Default)] + struct ODictLinks { + first: Option, + last: Option, + nodes: Vec>, + free: Vec, + by_hash: HashMap>, + state: usize, + } + + impl ODictLinks { + fn alloc_node(&mut self, node: ODictNode) -> usize { + if let Some(idx) = self.free.pop() { + self.nodes[idx] = Some(node); + idx + } else { + let idx = self.nodes.len(); + self.nodes.push(Some(node)); + idx + } + } + + fn add_tail(&mut self, key: PyObjectRef, hash: PyHash) { + let prev = self.last; + let idx = self.alloc_node(ODictNode { + key, + hash, + prev, + next: None, + }); + if let Some(prev) = prev { + if let Some(node) = self.nodes[prev].as_mut() { + node.next = Some(idx); + } + } else { + self.first = Some(idx); + } + self.last = Some(idx); + self.by_hash.entry(hash).or_default().push(idx); + self.state += 1; + } + + fn unlink(&mut self, idx: usize) -> Option { + let node = self.nodes[idx].take()?; + self.free.push(idx); + if let Some(prev) = node.prev { + if let Some(prev_node) = self.nodes[prev].as_mut() { + prev_node.next = node.next; + } + } else { + self.first = node.next; + } + if let Some(next) = node.next { + if let Some(next_node) = self.nodes[next].as_mut() { + next_node.prev = node.prev; + } + } else { + self.last = node.prev; + } + if let Some(bucket) = self.by_hash.get_mut(&node.hash) { + bucket.retain(|&i| i != idx); + } + self.state += 1; + Some(node) + } + + fn key_at(&self, idx: usize) -> Option { + self.nodes[idx].as_ref().map(|n| n.key.clone()) + } + + fn next_after(&self, key: &PyObject, reversed: bool) -> Option { + let mut idx = if reversed { self.last } else { self.first }; + while let Some(i) = idx { + let Some(node) = self.nodes[i].as_ref() else { + break; + }; + if node.key.is(key) { + let nxt = if reversed { node.prev } else { node.next }; + return nxt.and_then(|j| self.key_at(j)); + } + idx = if reversed { node.prev } else { node.next }; + } + None + } + } + + #[pyattr] + #[pyclass( + module = "collections", + name = "OrderedDict", + base = PyDict, + unhashable = true, + traverse = "manual" + )] + #[derive(Debug)] + struct PyOrderedDict { + dict: PyDict, + links: PyMutex, + } + + impl Default for PyOrderedDict { + fn default() -> Self { + Self { + dict: PyDict::default(), + links: PyMutex::new(ODictLinks::default()), + } + } + } + + // SAFETY: Traverse visits each owned Python reference at most once. + unsafe impl Traverse for PyOrderedDict { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.dict.traverse(tracer_fn); + if let Some(links) = self.links.try_lock() { + for node in links.nodes.iter().flatten() { + node.key.traverse(tracer_fn); + } + } + } + + fn clear(&mut self, out: &mut Vec) { + Traverse::clear(&mut self.dict, out); + let links = self.links.get_mut(); + for node in links.nodes.drain(..).flatten() { + out.push(node.key); + } + links.first = None; + links.last = None; + links.free.clear(); + links.by_hash.clear(); + links.state += 1; + } + } + + impl PyOrderedDict { + fn is_exact(zelf: &Py) -> bool { + zelf.class().is(Self::static_type()) + } + + fn mutated_error(vm: &VirtualMachine) -> crate::builtins::PyBaseExceptionRef { + vm.new_runtime_error("OrderedDict mutated during iteration") + } + + fn find_node( + &self, + key: &PyObject, + hash: PyHash, + vm: &VirtualMachine, + ) -> PyResult> { + let candidates: Vec<(usize, PyObjectRef)>; + { + let links = self.links.lock(); + let Some(idxs) = links.by_hash.get(&hash) else { + return Ok(None); + }; + for &i in idxs { + if let Some(node) = links.nodes[i].as_ref() + && node.key.is(key) + { + return Ok(Some(i)); + } + } + candidates = idxs + .iter() + .filter_map(|&i| links.nodes[i].as_ref().map(|n| (i, n.key.clone()))) + .collect(); + } + for (i, k) in candidates { + if vm.bool_eq(&k, key)? { + return Ok(Some(i)); + } + } + Ok(None) + } + + fn setitem_impl( + &self, + key: PyObjectRef, + value: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult<()> { + let hash = key.key_hash(vm)?; + self.dict.inner_setitem(&*key, value, vm)?; + if self.find_node(&key, hash, vm)?.is_none() { + self.links.lock().add_tail(key, hash); + } + Ok(()) + } + + fn delitem_impl(&self, key: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + let hash = key.key_hash(vm)?; + let Some(idx) = self.find_node(&key, hash, vm)? else { + return Err(vm.new_key_error(key)); + }; + self.links.lock().unlink(idx); + self.dict.inner_delitem(&*key, vm) + } + + fn dicts_equal(a: &PyDict, b: &PyDict, vm: &VirtualMachine) -> PyResult { + if a.__len__() != b.__len__() { + return Ok(false); + } + let mut pos = 0; + while let Some((next, k, v1)) = a.next_entry(pos) { + pos = next; + match b.inner_getitem_opt(&*k, vm)? { + Some(v2) if v1.is(&v2) || vm.bool_eq(&v1, &v2)? => {} + _ => return Ok(false), + } + } + Ok(true) + } + + fn keys_equal(&self, other: &Self, vm: &VirtualMachine) -> PyResult { + if core::ptr::eq(self, other) { + return Ok(true); + } + let (state_a, state_b, mut key_a, mut key_b) = { + let la = self.links.lock(); + let lb = other.links.lock(); + ( + la.state, + lb.state, + la.first.and_then(|i| la.key_at(i)), + lb.first.and_then(|i| lb.key_at(i)), + ) + }; + loop { + match (key_a, key_b) { + (None, None) => return Ok(true), + (None, Some(_)) | (Some(_), None) => return Ok(false), + (Some(a), Some(b)) => { + let eq = vm.bool_eq(&a, &b)?; + let (next_a, next_b, mutated) = { + let la = self.links.lock(); + let lb = other.links.lock(); + if la.state != state_a || lb.state != state_b { + (None, None, true) + } else if !eq { + return Ok(false); + } else { + (la.next_after(&a, false), lb.next_after(&b, false), false) + } + }; + if mutated { + return Err(Self::mutated_error(vm)); + } + key_a = next_a; + key_b = next_b; + } + } + } + } + + fn first_or_last_key(&self, last: bool) -> Option { + let links = self.links.lock(); + let idx = if last { links.last } else { links.first }?; + links.key_at(idx) + } + + fn iter_kind(zelf: PyRef, kind: u8, vm: &VirtualMachine) -> PyObjectRef { + PyODictIter::new(zelf, kind).to_pyobject(vm) + } + } + + #[derive(FromArgs)] + struct ODictPopArgs { + #[pyarg(any)] + key: PyObjectRef, + #[pyarg(any, optional)] + default: OptionalArg, + } + + #[derive(FromArgs)] + struct ODictPopItemArgs { + #[pyarg(any, optional)] + last: OptionalArg, + } + + #[derive(FromArgs)] + struct ODictSetDefaultArgs { + #[pyarg(any)] + key: PyObjectRef, + #[pyarg(any, optional)] + default: OptionalArg, + } + + #[derive(FromArgs)] + struct ODictMoveToEndArgs { + #[pyarg(any)] + key: PyObjectRef, + #[pyarg(any, optional)] + last: OptionalArg, + } + + #[derive(FromArgs)] + struct ODictFromKeysArgs { + #[pyarg(positional)] + iterable: PyObjectRef, + #[pyarg(any, optional)] + value: OptionalArg, + } + + #[pyclass( + with( + AsMapping, + AsNumber, + AsSequence, + Comparable, + Constructor, + Initializer, + Iterable, + Representable + ), + flags(BASETYPE, MAPPING, HAS_DICT, HAS_WEAKREF) + )] + impl PyOrderedDict { + #[pymethod] + fn clear(&self) { + self.dict.clear(); + let mut links = self.links.lock(); + *links = ODictLinks { + state: links.state.wrapping_add(1), + ..ODictLinks::default() + }; + } + + #[pymethod] + fn copy(zelf: PyRef, vm: &VirtualMachine) -> PyResult { + let exact = Self::is_exact(&zelf); + let od_copy = if exact { + Self::default().into_pyobject(vm) + } else { + let cls = zelf.class().to_owned(); + cls.as_object().call((), vm)? + }; + let state = zelf.links.lock().state; + let mut current = zelf.first_or_last_key(false); + while let Some(key) = current { + let value = if exact { + zelf.dict + .inner_getitem_opt(&*key, vm)? + .ok_or_else(|| vm.new_key_error(key.clone()))? + } else { + zelf.as_object().get_item(&*key, vm)? + }; + od_copy.set_item(&*key, value, vm)?; + let (next, mutated) = { + let links = zelf.links.lock(); + if links.state != state { + (None, true) + } else { + (links.next_after(&key, false), false) + } + }; + if mutated { + return Err(Self::mutated_error(vm)); + } + current = next; + } + Ok(od_copy) + } + + #[pymethod] + fn update(zelf: PyRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult<()> { + if args.args.len() > 1 { + return Err(vm.new_type_error(format!( + "update expected at most 1 argument, got {}", + args.args.len() + ))); + } + if let Some(arg) = args.args.first() { + odict_update_arg(zelf.as_object(), arg.clone(), vm)?; + } + for (key, value) in args.kwargs { + zelf.as_object().set_item(&key, value, vm)?; + } + Ok(()) + } + + #[pymethod] + fn pop(zelf: PyRef, args: ODictPopArgs, vm: &VirtualMachine) -> PyResult { + let key = args.key; + let hash = key.key_hash(vm)?; + match zelf.find_node(&key, hash, vm)? { + Some(idx) => { + zelf.links.lock().unlink(idx); + let value = zelf + .dict + .inner_getitem_opt(&*key, vm)? + .ok_or_else(|| vm.new_key_error(key.clone()))?; + zelf.dict.inner_delitem(&*key, vm)?; + Ok(value) + } + None => args.default.ok_or_else(|| vm.new_key_error(key)), + } + } + + #[pymethod] + fn popitem( + zelf: PyRef, + args: ODictPopItemArgs, + vm: &VirtualMachine, + ) -> PyResult<(PyObjectRef, PyObjectRef)> { + let last = args.last.unwrap_or(true); + let Some(key) = zelf.first_or_last_key(last) else { + return Err(vm.new_key_error(vm.ctx.new_str("dictionary is empty").into())); + }; + let hash = key.key_hash(vm)?; + if let Some(idx) = zelf.find_node(&key, hash, vm)? { + zelf.links.lock().unlink(idx); + } + let value = zelf + .dict + .inner_getitem_opt(&*key, vm)? + .ok_or_else(|| vm.new_key_error(key.clone()))?; + zelf.dict.inner_delitem(&*key, vm)?; + Ok((key, value)) + } + + #[pymethod] + fn setdefault( + zelf: PyRef, + args: ODictSetDefaultArgs, + vm: &VirtualMachine, + ) -> PyResult { + let key = args.key; + let default = args.default.unwrap_or_none(vm); + if Self::is_exact(&zelf) { + if let Some(value) = zelf.dict.inner_getitem_opt(&*key, vm)? { + return Ok(value); + } + zelf.setitem_impl(key, default.clone(), vm)?; + Ok(default) + } else if zelf.as_object().sequence_unchecked().contains(&key, vm)? { + zelf.as_object().get_item(&*key, vm) + } else { + zelf.as_object().set_item(&*key, default.clone(), vm)?; + Ok(default) + } + } + + #[pymethod] + fn move_to_end(&self, args: ODictMoveToEndArgs, vm: &VirtualMachine) -> PyResult<()> { + let key = args.key; + let last = args.last.unwrap_or(true); + let hash = key.key_hash(vm)?; + let Some(idx) = self.find_node(&key, hash, vm)? else { + return Err(vm.new_key_error(key)); + }; + let mut links = self.links.lock(); + let already = if last { + links.last == Some(idx) + } else { + links.first == Some(idx) + }; + if already { + return Ok(()); + } + let Some(node) = links.unlink(idx) else { + return Err(vm.new_key_error(key)); + }; + if last { + links.add_tail(node.key, node.hash); + } else { + let next = links.first; + let new_idx = links.alloc_node(ODictNode { + key: node.key, + hash: node.hash, + prev: None, + next, + }); + if let Some(next) = next { + if let Some(next_node) = links.nodes[next].as_mut() { + next_node.prev = Some(new_idx); + } + } else { + links.last = Some(new_idx); + } + links.first = Some(new_idx); + links.by_hash.entry(node.hash).or_default().push(new_idx); + links.state += 1; + } + Ok(()) + } + + #[pymethod] + fn keys(zelf: PyRef, vm: &VirtualMachine) -> PyObjectRef { + PyODictKeys { od: zelf }.to_pyobject(vm) + } + + #[pymethod] + fn values(zelf: PyRef, vm: &VirtualMachine) -> PyObjectRef { + PyODictValues { od: zelf }.to_pyobject(vm) + } + + #[pymethod] + fn items(zelf: PyRef, vm: &VirtualMachine) -> PyObjectRef { + PyOrderedDictItems { od: zelf }.to_pyobject(vm) + } + + #[pymethod] + fn __reversed__(zelf: PyRef, vm: &VirtualMachine) -> PyObjectRef { + Self::iter_kind(zelf, ODICT_ITER_KEYS | ODICT_ITER_REVERSED, vm) + } + + #[pymethod] + fn __reduce__(zelf: PyRef, vm: &VirtualMachine) -> PyResult { + let items = zelf.as_object().get_attr("items", vm)?.call((), vm)?; + let items_iter = items.get_iter(vm)?; + let state = vm.call_method(zelf.as_object(), "__getstate__", ())?; + let state = match state.downcast_ref::() { + Some(d) if d.__len__() == 0 => vm.ctx.none(), + _ => state, + }; + Ok(vm + .ctx + .new_tuple(vec![ + zelf.class().to_owned().into(), + vm.ctx.empty_tuple.clone().into(), + state, + vm.ctx.none(), + items_iter.into(), + ]) + .into()) + } + + #[pymethod] + fn __sizeof__(&self) -> usize { + self.dict.__sizeof__() + + core::mem::size_of::() + + self.links.lock().nodes.capacity() * core::mem::size_of::>() + } + + #[pyclassmethod] + fn fromkeys(cls: PyTypeRef, args: ODictFromKeysArgs, vm: &VirtualMachine) -> PyResult { + let value = args.value.unwrap_or_none(vm); + let inst = cls.as_object().call((), vm)?; + let iter = args.iterable.get_iter(vm)?; + for key in iter.iter::(vm)? { + inst.set_item(&*key?, value.clone(), vm)?; + } + Ok(inst) + } + + #[pyclassmethod] + fn __class_getitem__( + cls: PyTypeRef, + args: PyObjectRef, + vm: &VirtualMachine, + ) -> PyResult { + PyGenericAlias::from_args(cls, args, vm) + } + } + + impl DefaultConstructor for PyOrderedDict {} + + impl Initializer for PyOrderedDict { + type Args = FuncArgs; + + fn init(zelf: PyRef, args: Self::Args, vm: &VirtualMachine) -> PyResult<()> { + if args.args.len() > 1 { + return Err(vm.new_type_error(format!( + "OrderedDict expected at most 1 argument, got {}", + args.args.len() + ))); + } + Self::update(zelf, args, vm) + } + } + + impl Representable for PyOrderedDict { + fn repr_str(zelf: &Py, vm: &VirtualMachine) -> PyResult { + if zelf.dict.__len__() == 0 { + return Ok(format!("{}()", zelf.class().name())); + } + let Some(_guard) = ReprGuard::enter(vm, zelf.as_object()) else { + return Ok("...".to_owned()); + }; + let tmp = PyDict::default(); + let mut current = zelf.first_or_last_key(false); + while let Some(key) = current { + let value = zelf + .dict + .inner_getitem_opt(&*key, vm)? + .ok_or_else(|| vm.new_key_error(key.clone()))?; + tmp.inner_setitem(&*key, value, vm)?; + current = zelf.links.lock().next_after(&key, false); + } + let dcopy = tmp.into_ref(&vm.ctx); + let dict_repr = dcopy.as_object().repr(vm)?; + Ok(format!("{}({dict_repr})", zelf.class().name())) + } + } + + impl AsMapping for PyOrderedDict { + fn as_mapping() -> &'static PyMappingMethods { + static AS_MAPPING: PyMappingMethods = PyMappingMethods { + length: atomic_func!(|mapping, _vm| Ok(PyOrderedDict::mapping_downcast(mapping) + .dict + .__len__())), + subscript: atomic_func!(|mapping, needle, vm| { + let zelf = PyOrderedDict::mapping_downcast(mapping); + if let Some(value) = zelf.dict.inner_getitem_opt(needle, vm)? { + return Ok(value); + } + if let Some(missing) = + vm.get_method(zelf.to_owned().into(), identifier!(vm, __missing__)) + { + return missing?.call((needle.to_owned(),), vm); + } + Err(vm.new_key_error(needle.to_owned())) + }), + ass_subscript: atomic_func!(|mapping, needle, value, vm| { + let zelf = PyOrderedDict::mapping_downcast(mapping); + if let Some(value) = value { + zelf.setitem_impl(needle.to_owned(), value, vm) + } else { + zelf.delitem_impl(needle.to_owned(), vm) + } + }), + }; + &AS_MAPPING + } + } + + impl AsSequence for PyOrderedDict { + fn as_sequence() -> &'static PySequenceMethods { + static AS_SEQUENCE: PySequenceMethods = PySequenceMethods { + contains: atomic_func!(|seq, target, vm| { + PyOrderedDict::sequence_downcast(seq) + .dict + .inner_getitem_opt(target, vm) + .map(|v| v.is_some()) + }), + ..PySequenceMethods::NOT_IMPLEMENTED + }; + &AS_SEQUENCE + } + } + + impl AsNumber for PyOrderedDict { + fn as_number() -> &'static PyNumberMethods { + static AS_NUMBER: PyNumberMethods = PyNumberMethods { + or: Some(|a, b, vm| { + if !a.fast_isinstance(vm.ctx.types.dict_type) + || !b.fast_isinstance(vm.ctx.types.dict_type) + { + return Ok(vm.ctx.not_implemented()); + } + let cls = if let Some(od) = a.downcast_ref::() { + od.class().to_owned() + } else if let Some(od) = b.downcast_ref::() { + od.class().to_owned() + } else { + return Ok(vm.ctx.not_implemented()); + }; + let new = cls.as_object().call((a.to_owned(),), vm)?; + odict_update_arg(&new, b.to_owned(), vm)?; + Ok(new) + }), + inplace_or: Some(|a, b, vm| { + let Some(od) = a.downcast_ref::() else { + return Ok(vm.ctx.not_implemented()); + }; + odict_update_arg(od.as_object(), b.to_owned(), vm)?; + Ok(od.as_object().to_owned()) + }), + ..PyNumberMethods::NOT_IMPLEMENTED + }; + &AS_NUMBER + } + } + + impl Comparable for PyOrderedDict { + fn cmp( + zelf: &Py, + other: &PyObject, + op: PyComparisonOp, + vm: &VirtualMachine, + ) -> PyResult { + if !matches!(op, PyComparisonOp::Eq | PyComparisonOp::Ne) { + return Ok(PyArithmeticValue::NotImplemented); + } + if !other.fast_isinstance(vm.ctx.types.dict_type) { + return Ok(PyArithmeticValue::NotImplemented); + } + let dict_eq = if let Some(other_od) = other.downcast_ref::() { + let eq = Self::dicts_equal(&zelf.dict, &other_od.dict, vm)?; + if (op == PyComparisonOp::Eq && !eq) || (op == PyComparisonOp::Ne && !eq) { + return Ok(PyArithmeticValue::Implemented(op == PyComparisonOp::Ne)); + } + let keys_eq = zelf.keys_equal(other_od, vm)?; + return Ok(PyArithmeticValue::Implemented( + if op == PyComparisonOp::Eq { + keys_eq + } else { + !keys_eq + }, + )); + } else if let Some(other_d) = other.downcast_ref::() { + Self::dicts_equal(&zelf.dict, other_d, vm)? + } else { + return Ok(PyArithmeticValue::NotImplemented); + }; + Ok(PyArithmeticValue::Implemented( + if op == PyComparisonOp::Eq { + dict_eq + } else { + !dict_eq + }, + )) + } + } + + impl Iterable for PyOrderedDict { + fn iter(zelf: PyRef, vm: &VirtualMachine) -> PyResult { + Ok(Self::iter_kind(zelf, ODICT_ITER_KEYS, vm)) + } + } + + fn odict_update_arg(zelf: &PyObject, arg: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> { + if let Some(dict) = arg.downcast_ref_if_exact::(vm) { + for (key, value) in dict { + zelf.set_item(&*key, value, vm)?; + } + return Ok(()); + } + match arg.get_attr("keys", vm) { + Ok(keys_fn) => { + let keys = keys_fn.call((), vm)?; + let iter = keys.get_iter(vm)?; + for key in iter.iter::(vm)? { + let key = key?; + let value = arg.get_item(&*key, vm)?; + zelf.set_item(&*key, value, vm)?; + } + return Ok(()); + } + Err(e) if e.fast_isinstance(vm.ctx.exceptions.attribute_error) => {} + Err(e) => return Err(e), + } + let iter = arg.get_iter(vm)?; + for (index, element) in iter.iter::(vm)?.enumerate() { + let (key, value) = PyDict::update_sequence_pair(element?, index, vm)?; + zelf.set_item(&*key, value, vm)?; + } + Ok(()) + } + + #[pyattr] + #[pyclass(name = "odict_iterator", traverse = "manual")] + #[derive(Debug, PyPayload)] + struct PyODictIter { + od: Option>, + kind: u8, + size: usize, + state: usize, + current: PyMutex>, + } + + impl PyODictIter { + fn new(od: PyRef, kind: u8) -> Self { + let reversed = kind & ODICT_ITER_REVERSED != 0; + let (state, current) = { + let links = od.links.lock(); + let idx = if reversed { links.last } else { links.first }; + (links.state, idx.and_then(|i| links.key_at(i))) + }; + Self { + size: od.dict.__len__(), + state, + current: PyMutex::new(current), + od: Some(od), + kind, + } + } + + fn od(&self) -> Option<&Py> { + self.od.as_deref() + } + + fn project(&self, key: PyObjectRef, vm: &VirtualMachine) -> PyResult { + let Some(od) = self.od() else { + return Err(vm.new_runtime_error("OrderedDict iterator is cleared")); + }; + let want_key = self.kind & ODICT_ITER_KEYS != 0; + let want_value = self.kind & ODICT_ITER_VALUES != 0; + if want_key && want_value { + let value = od + .dict + .inner_getitem_opt(&*key, vm)? + .ok_or_else(|| vm.new_key_error(key.clone()))?; + Ok(vm.ctx.new_tuple(vec![key, value]).into()) + } else if want_value { + od.dict + .inner_getitem_opt(&*key, vm)? + .ok_or_else(|| vm.new_key_error(key)) + } else { + Ok(key) + } + } + } + + // SAFETY: visits each owned reference at most once. + unsafe impl Traverse for PyODictIter { + fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) { + self.od.traverse(tracer_fn); + if let Some(cur) = self.current.try_lock() + && let Some(key) = cur.as_ref() + { + key.traverse(tracer_fn); + } + } + + fn clear(&mut self, out: &mut Vec) { + if let Some(key) = self.current.get_mut().take() { + out.push(key); + } + if let Some(od) = self.od.take() { + out.push(od.into()); + } + } + } + + #[pyclass(with(IterNext, Iterable), flags(DISALLOW_INSTANTIATION))] + impl PyODictIter { + #[pymethod] + fn __reduce__(&self, vm: &VirtualMachine) -> PyResult { + let remaining = self.collect_remaining(vm)?; + Ok(vm + .ctx + .new_tuple(vec![ + builtins_iter(vm)?, + vm.ctx + .new_tuple(vec![vm.ctx.new_list(remaining).into()]) + .into(), + ]) + .into()) + } + } + + impl PyODictIter { + fn collect_remaining(&self, vm: &VirtualMachine) -> PyResult> { + let Some(od) = self.od() else { + return Ok(Vec::new()); + }; + let mut out = Vec::new(); + let mut current = self.current.lock().clone(); + let reversed = self.kind & ODICT_ITER_REVERSED != 0; + while let Some(key) = current { + out.push(self.project(key.clone(), vm)?); + current = od.links.lock().next_after(&key, reversed); + } + Ok(out) + } + } + + impl SelfIter for PyODictIter {} + + impl IterNext for PyODictIter { + fn next(zelf: &Py, vm: &VirtualMachine) -> PyResult { + let Some(od) = zelf.od() else { + return Ok(PyIterReturn::StopIteration(None)); + }; + let mut current_guard = zelf.current.lock(); + let Some(key) = current_guard.clone() else { + return Ok(PyIterReturn::StopIteration(None)); + }; + let links = od.links.lock(); + if links.state != zelf.state { + return Err(PyOrderedDict::mutated_error(vm)); + } + if od.dict.__len__() != zelf.size { + return Err(vm.new_runtime_error("OrderedDict changed size during iteration")); + } + let reversed = zelf.kind & ODICT_ITER_REVERSED != 0; + let next = links.next_after(&key, reversed); + drop(links); + *current_guard = next; + drop(current_guard); + Ok(PyIterReturn::Return(zelf.project(key, vm)?)) + } + } + + fn odict_view_eq( + od: &PyRef, + kind: u8, + other: &PyObject, + vm: &VirtualMachine, + ) -> PyResult { + if kind == ODICT_ITER_VALUES { + return Ok(PyArithmeticValue::NotImplemented); + } + let set_like = other.fast_isinstance(vm.ctx.types.set_type) + || other.fast_isinstance(vm.ctx.types.frozenset_type) + || other.class().is(vm.ctx.types.dict_keys_type) + || other.class().is(vm.ctx.types.dict_items_type) + || other.downcast_ref::().is_some() + || other.downcast_ref::().is_some(); + if !set_like { + return Ok(PyArithmeticValue::NotImplemented); + } + let Ok(other_len) = other.length(vm) else { + return Ok(PyArithmeticValue::NotImplemented); + }; + if od.dict.__len__() != other_len { + return Ok(PyArithmeticValue::Implemented(false)); + } + let iter_obj = PyODictIter::new(od.clone(), kind).to_pyobject(vm); + let a_iter = iter_obj.get_iter(vm)?; + for item in a_iter.iter::(vm)? { + let item = item?; + if !other.sequence_unchecked().contains(&item, vm)? { + return Ok(PyArithmeticValue::Implemented(false)); + } + } + Ok(PyArithmeticValue::Implemented(true)) + } + + fn ordered_view_number_and(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult { + let a_is_view = a.downcast_ref::().is_some() + || a.downcast_ref::().is_some(); + let (view, other) = if a_is_view { (a, b) } else { (b, a) }; + set_view_number_and(view, other, vm) + } + + fn ordered_item_view_number_xor(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult { + let is_item_view = |obj: &PyObject| { + obj.downcast_ref::().is_some() + || obj.downcast_ref::().is_some() + }; + set_item_view_number_xor(a, b, is_item_view(a) && is_item_view(b), vm) + } + + #[pyattr] + #[pyclass(name = "odict_keys", traverse)] + #[derive(Debug, PyPayload)] + struct PyODictKeys { + od: PyRef, + } + + #[pyclass(with(Iterable, Comparable, AsMapping, AsSequence, AsNumber))] + impl PyODictKeys { + #[pymethod] + fn __reversed__(&self, vm: &VirtualMachine) -> PyObjectRef { + PyODictIter::new(self.od.clone(), ODICT_ITER_KEYS | ODICT_ITER_REVERSED).to_pyobject(vm) + } + + #[pygetset] + fn mapping(&self, vm: &VirtualMachine) -> PyResult { + PyMappingProxy::from_object(self.od.as_object().to_owned(), vm) + } + + #[pymethod] + fn isdisjoint(&self, other: ArgIterable, vm: &VirtualMachine) -> PyResult { + for item in other.iter(vm)? { + if self.od.dict.inner_getitem_opt(&*item?, vm)?.is_some() { + return Ok(false); + } + } + Ok(true) + } + } + + impl AsMapping for PyODictKeys { + fn as_mapping() -> &'static PyMappingMethods { + static AS_MAPPING: PyMappingMethods = PyMappingMethods { + length: atomic_func!(|mapping, _vm| Ok(PyODictKeys::mapping_downcast(mapping) + .od + .dict + .__len__())), + ..PyMappingMethods::NOT_IMPLEMENTED + }; + &AS_MAPPING + } + } + + impl AsSequence for PyODictKeys { + fn as_sequence() -> &'static PySequenceMethods { + static AS_SEQUENCE: PySequenceMethods = PySequenceMethods { + length: atomic_func!(|seq, _vm| Ok(PyODictKeys::sequence_downcast(seq) + .od + .dict + .__len__())), + contains: atomic_func!(|seq, target, vm| { + PyODictKeys::sequence_downcast(seq) + .od + .dict + .inner_getitem_opt(target, vm) + .map(|v| v.is_some()) + }), + ..PySequenceMethods::NOT_IMPLEMENTED + }; + &AS_SEQUENCE + } + } + + impl Iterable for PyODictKeys { + fn iter(zelf: PyRef, vm: &VirtualMachine) -> PyResult { + Ok(PyODictIter::new(zelf.od.clone(), ODICT_ITER_KEYS).to_pyobject(vm)) + } + } + + impl Comparable for PyODictKeys { + fn cmp( + zelf: &Py, + other: &PyObject, + op: PyComparisonOp, + vm: &VirtualMachine, + ) -> PyResult { + op.eq_only(|| odict_view_eq(&zelf.od, ODICT_ITER_KEYS, other, vm)) + } + } + + impl AsNumber for PyODictKeys { + fn as_number() -> &'static PyNumberMethods { + static AS_NUMBER: PyNumberMethods = PyNumberMethods { + subtract: Some(set_inner_number_subtract), + and: Some(ordered_view_number_and), + xor: Some(set_inner_number_xor), + or: Some(set_inner_number_or), + ..PyNumberMethods::NOT_IMPLEMENTED + }; + &AS_NUMBER + } + } + + #[pyattr] + #[pyclass(name = "odict_values", traverse)] + #[derive(Debug, PyPayload)] + struct PyODictValues { + od: PyRef, + } + + #[pyclass(with(Iterable, Comparable, AsMapping))] + impl PyODictValues { + #[pymethod] + fn __reversed__(&self, vm: &VirtualMachine) -> PyObjectRef { + PyODictIter::new(self.od.clone(), ODICT_ITER_VALUES | ODICT_ITER_REVERSED) + .to_pyobject(vm) + } + + #[pygetset] + fn mapping(&self, vm: &VirtualMachine) -> PyResult { + PyMappingProxy::from_object(self.od.as_object().to_owned(), vm) + } + } + + impl AsMapping for PyODictValues { + fn as_mapping() -> &'static PyMappingMethods { + static AS_MAPPING: PyMappingMethods = PyMappingMethods { + length: atomic_func!(|mapping, _vm| Ok(PyODictValues::mapping_downcast(mapping) + .od + .dict + .__len__())), + ..PyMappingMethods::NOT_IMPLEMENTED + }; + &AS_MAPPING + } + } + + impl Iterable for PyODictValues { + fn iter(zelf: PyRef, vm: &VirtualMachine) -> PyResult { + Ok(PyODictIter::new(zelf.od.clone(), ODICT_ITER_VALUES).to_pyobject(vm)) + } + } + + impl Comparable for PyODictValues { + fn cmp( + zelf: &Py, + other: &PyObject, + op: PyComparisonOp, + vm: &VirtualMachine, + ) -> PyResult { + op.eq_only(|| odict_view_eq(&zelf.od, ODICT_ITER_VALUES, other, vm)) + } + } + + #[pyattr] + #[pyclass(name = "odict_items", traverse)] + #[derive(Debug, PyPayload)] + pub(crate) struct PyOrderedDictItems { + od: PyRef, + } + + #[pyclass(with(Iterable, Comparable, AsMapping, AsSequence, AsNumber))] + impl PyOrderedDictItems { + #[pymethod] + fn __reversed__(&self, vm: &VirtualMachine) -> PyObjectRef { + PyODictIter::new(self.od.clone(), ODICT_ITER_ITEMS | ODICT_ITER_REVERSED) + .to_pyobject(vm) + } + + #[pygetset] + fn mapping(&self, vm: &VirtualMachine) -> PyResult { + PyMappingProxy::from_object(self.od.as_object().to_owned(), vm) + } + + #[pymethod] + fn isdisjoint(&self, other: ArgIterable, vm: &VirtualMachine) -> PyResult { + for item in other.iter(vm)? { + let item = item?; + let Some(needle) = item.downcast_ref::() else { + continue; + }; + if needle.len() != 2 { + continue; + } + let Some(found) = self.od.dict.inner_getitem_opt(&*needle[0], vm)? else { + continue; + }; + if vm.identical_or_equal(&found, &needle[1])? { + return Ok(false); + } + } + Ok(true) + } + } + + impl AsMapping for PyOrderedDictItems { + fn as_mapping() -> &'static PyMappingMethods { + static AS_MAPPING: PyMappingMethods = PyMappingMethods { + length: atomic_func!(|mapping, _vm| Ok(PyOrderedDictItems::mapping_downcast( + mapping + ) + .od + .dict + .__len__())), + ..PyMappingMethods::NOT_IMPLEMENTED + }; + &AS_MAPPING + } + } + + impl AsSequence for PyOrderedDictItems { + fn as_sequence() -> &'static PySequenceMethods { + static AS_SEQUENCE: PySequenceMethods = PySequenceMethods { + length: atomic_func!(|seq, _vm| Ok(PyOrderedDictItems::sequence_downcast(seq) + .od + .dict + .__len__())), + contains: atomic_func!(|seq, target, vm| { + let needle: &Py = match target.downcast_ref() { + Some(needle) => needle, + None => return Ok(false), + }; + if needle.len() != 2 { + return Ok(false); + } + let zelf = PyOrderedDictItems::sequence_downcast(seq); + let key = &needle[0]; + let Some(found) = zelf.od.dict.inner_getitem_opt(&**key, vm)? else { + return Ok(false); + }; + vm.identical_or_equal(&found, &needle[1]) + }), + ..PySequenceMethods::NOT_IMPLEMENTED + }; + &AS_SEQUENCE + } + } + + impl Iterable for PyOrderedDictItems { + fn iter(zelf: PyRef, vm: &VirtualMachine) -> PyResult { + Ok(PyODictIter::new(zelf.od.clone(), ODICT_ITER_ITEMS).to_pyobject(vm)) + } + } + + impl Comparable for PyOrderedDictItems { + fn cmp( + zelf: &Py, + other: &PyObject, + op: PyComparisonOp, + vm: &VirtualMachine, + ) -> PyResult { + op.eq_only(|| odict_view_eq(&zelf.od, ODICT_ITER_ITEMS, other, vm)) + } + } + + impl AsNumber for PyOrderedDictItems { + fn as_number() -> &'static PyNumberMethods { + static AS_NUMBER: PyNumberMethods = PyNumberMethods { + subtract: Some(set_inner_number_subtract), + and: Some(ordered_view_number_and), + xor: Some(ordered_item_view_number_xor), + or: Some(set_inner_number_or), + ..PyNumberMethods::NOT_IMPLEMENTED + }; + &AS_NUMBER + } + } +} diff --git a/crates/vm/src/stdlib/_functools.rs b/crates/vm/src/stdlib/_functools.rs index 54d908e03..1c7b45bb4 100644 --- a/crates/vm/src/stdlib/_functools.rs +++ b/crates/vm/src/stdlib/_functools.rs @@ -7,13 +7,13 @@ mod _functools { builtins::{ PyBoundMethod, PyDict, PyDictRef, PyGenericAlias, PyTuple, PyType, PyTypeRef, object, }, - common::lock::PyRwLock, - function::{FuncArgs, KwArgs, OptionalOption, PySetterValue}, + common::{hash::PyHash, lock::PyRwLock}, + function::{Either, FuncArgs, KwArgs, OptionalOption, PyComparisonValue, PySetterValue}, object::AsObject, protocol::PyIter, pyclass, recursion::ReprGuard, - types::{Callable, Constructor, GetDescriptor, Representable}, + types::{Callable, Constructor, GetDescriptor, PyComparisonOp, Representable}, }; use core::sync::atomic::{AtomicU64, Ordering}; use parking_lot::lock_api::RawReentrantMutex as GenericRawReentrantMutex; @@ -62,6 +62,95 @@ mod _functools { Ok(accumulator) } + #[derive(FromArgs)] + struct CmpToKeyArgs { + mycmp: PyObjectRef, + } + + #[pyfunction] + fn cmp_to_key(args: CmpToKeyArgs) -> PyKeyWrapper { + PyKeyWrapper { + cmp: args.mycmp, + object: PyRwLock::new(None), + } + } + + #[derive(FromArgs)] + struct KeyWrapperCallArgs { + obj: PyObjectRef, + } + + #[pyclass(no_attr, name = "KeyWrapper", module = "functools", unhashable = true)] + #[derive(Debug, PyPayload)] + struct PyKeyWrapper { + cmp: PyObjectRef, + object: PyRwLock>, + } + + #[pyclass(with(Callable), flags(IMMUTABLETYPE, DISALLOW_INSTANTIATION))] + impl PyKeyWrapper { + #[pygetset] + fn obj(&self, vm: &VirtualMachine) -> PyObjectRef { + self.object.read().clone().unwrap_or_else(|| vm.ctx.none()) + } + + #[pygetset(setter)] + fn set_obj(&self, value: PySetterValue) { + *self.object.write() = match value { + PySetterValue::Assign(v) => Some(v), + PySetterValue::Delete => None, + }; + } + + #[pygetset] + fn __text_signature__(&self) -> &'static str { + "(obj)" + } + + #[pyslot] + fn slot_richcompare( + zelf: &PyObject, + other: &PyObject, + op: PyComparisonOp, + vm: &VirtualMachine, + ) -> PyResult> { + let Some(zelf) = zelf.downcast_ref::() else { + return Err(vm.new_type_error(format!( + "unexpected payload for {}", + op.method_name(&vm.ctx).as_str() + ))); + }; + let Some(other) = other.downcast_ref::() else { + return Err(vm.new_type_error("other argument must be K instance")); + }; + let x = zelf + .object + .read() + .clone() + .ok_or_else(|| vm.new_attribute_error("object"))?; + let y = other + .object + .read() + .clone() + .ok_or_else(|| vm.new_attribute_error("object"))?; + let res = zelf.cmp.call((x, y), vm)?; + res.rich_compare(vm.ctx.new_int(0).into(), op, vm) + .map(Either::A) + } + } + + impl Callable for PyKeyWrapper { + type Args = KeyWrapperCallArgs; + + fn call(zelf: &Py, args: Self::Args, vm: &VirtualMachine) -> PyResult { + Ok(Self { + cmp: zelf.cmp.clone(), + object: PyRwLock::new(Some(args.obj)), + } + .into_pyobject(vm)) + } + } + // Placeholder singleton for partial arguments // The singleton is stored as _instance on the type class #[pyattr] @@ -558,6 +647,16 @@ mod _functools { } } + #[pyclass(no_attr, name = "_lru_list_elem", module = "functools")] + #[derive(Debug, PyPayload)] + struct PyLruListElem { + hash: PyHash, + result: PyObjectRef, + } + + #[pyclass(flags(IMMUTABLETYPE, DISALLOW_INSTANTIATION))] + impl PyLruListElem {} + /// Native implementation of `functools._lru_cache_wrapper`, mirroring CPython's /// `_functools` accelerator so `functools.lru_cache` doesn't fall back to the much /// slower pure-Python implementation in `Lib/functools.py`. @@ -661,16 +760,17 @@ mod _functools { } /// Refresh `key`'s recency by moving it to the end of `cache`'s insertion - /// order (removing then reinserting it). `cache` itself provides the - /// hashing/equality, so this needs no separate key-comparison machinery. + /// order (removing then reinserting it). `hash` is the already-computed + /// hash of `key`, so `__hash__` is not called again. fn touch_key( key: &PyObject, + hash: PyHash, value: &PyObject, cache: &Py, vm: &VirtualMachine, ) -> PyResult<()> { - cache.del_item(key.as_object(), vm)?; - cache.set_item(key.as_object(), value.to_owned(), vm)?; + cache.del_item_known_hash(key, hash, vm)?; + cache.set_item_known_hash(key, hash, value.to_owned(), vm)?; Ok(()) } @@ -681,8 +781,11 @@ mod _functools { return Ok(()); } let oldest = cache.into_iter().next(); - if let Some((oldest_key, _)) = oldest { - cache.del_item(oldest_key.as_object(), vm)?; + if let Some((oldest_key, oldest_val)) = oldest { + let Some(elem) = oldest_val.downcast_ref::() else { + return Err(vm.new_type_error("lru cache entry is corrupted")); + }; + cache.del_item_known_hash(oldest_key.as_object(), elem.hash, vm)?; } Ok(()) } @@ -743,16 +846,21 @@ mod _functools { } let key = zelf.make_key(&args, vm); + let hash = key.hash(vm)?; { let _guard = RMutexGuard::acquire(&zelf.lock); let cache = zelf.cache.read().clone(); - if let Some(value) = cache.get_item_opt(key.as_object(), vm)? { + if let Some(value) = cache.get_item_known_hash(key.as_object(), hash, vm)? { + let Some(elem) = value.downcast_ref::() else { + return Err(vm.new_type_error("lru cache entry is corrupted")); + }; + let result = elem.result.clone(); zelf.hits.fetch_add(1, Ordering::Relaxed); if zelf.maxsize.is_some() { - Self::touch_key(&key, &value, &cache, vm)?; + Self::touch_key(&key, hash, &value, &cache, vm)?; } - return Ok(value); + return Ok(result); } zelf.misses.fetch_add(1, Ordering::Relaxed); // Lock released here (guard dropped) before calling into user code, so @@ -768,8 +876,13 @@ mod _functools { // (e.g. recursive lookups, or another thread finishing first); in that // case keep the existing entry instead of creating orphan eviction // links (see CPython issue gh-35780). - if !cache.contains_key(key.as_object(), vm) { - cache.set_item(key.as_object(), result.clone(), vm)?; + if !cache.contains_known_hash(key.as_object(), hash, vm)? { + let entry = PyLruListElem { + hash, + result: result.clone(), + } + .into_pyobject(vm); + cache.set_item_known_hash(key.as_object(), hash, entry, vm)?; if let Some(maxsize) = zelf.maxsize { Self::evict_if_full(maxsize, &cache, vm)?; } diff --git a/crates/vm/src/stdlib/_interpchannels.rs b/crates/vm/src/stdlib/_interpchannels.rs index 216669b0e..2936f0b26 100644 --- a/crates/vm/src/stdlib/_interpchannels.rs +++ b/crates/vm/src/stdlib/_interpchannels.rs @@ -16,7 +16,7 @@ pub(crate) mod _interpchannels { function::{ArgSpec, Either, FuncArgs, PyComparisonValue}, protocol::{PyNumber, PyNumberMethods}, types::{AsNumber, Constructor, Hashable, PyComparisonOp, PyStructSequence, Representable}, - vm::crossinterp::{self, Fallback, SharedValue, UNBOUND_REPLACE}, + vm::crossinterp::{Fallback, SharedValue, UnboundOp}, }; use alloc::collections::BTreeMap; use alloc::{collections::VecDeque, sync::Arc}; @@ -25,9 +25,28 @@ pub(crate) mod _interpchannels { use parking_lot::{Condvar, Mutex}; use std::{sync::OnceLock, time::Instant}; - const CHANNEL_SEND: i32 = 1; - const CHANNEL_BOTH: i32 = 0; - const CHANNEL_RECV: i32 = -1; + #[repr(i32)] + #[derive(Copy, Clone, Debug, PartialEq, Eq)] + enum ChannelEnd { + Recv = -1, + Both = 0, + Send = 1, + } + + impl ChannelEnd { + const fn from_i32(v: i32) -> Option { + match v { + -1 => Some(Self::Recv), + 0 => Some(Self::Both), + 1 => Some(Self::Send), + _ => None, + } + } + + const fn as_i32(self) -> i32 { + self as i32 + } + } #[pyattr] #[pyexception(name = "ChannelError", base = crate::exceptions::types::PyRuntimeError)] @@ -173,7 +192,7 @@ pub(crate) mod _interpchannels { interpid: i64, /// `None` once the owning interpreter is gone (the item is "unbound"). data: Option, - unboundop: i32, + unboundop: UnboundOp, waiting: Option>, } @@ -279,7 +298,7 @@ pub(crate) mod _interpchannels { struct ChannelState { queue: VecDeque, ends: ChannelEnds, - unboundop: i32, + unboundop: UnboundOp, fallback: i32, open: bool, /// Send is closed and the queue is draining. @@ -322,7 +341,7 @@ pub(crate) mod _interpchannels { Ok(chan.clone()) } - fn channel_create(unboundop: i32, fallback: i32) -> i64 { + fn channel_create(unboundop: UnboundOp, fallback: i32) -> i64 { let mut table = channels().lock(); let cid = table.next_id; table.next_id += 1; @@ -420,14 +439,14 @@ pub(crate) mod _interpchannels { let entry = table.refs.get_mut(&cid).ok_or(ChanErr::NotFound)?; let chan = entry.chan.as_ref().ok_or(ChanErr::Closed)?.clone(); let mut state = chan.state.lock(); - if !force && end == CHANNEL_SEND && state.closing { + if !force && end == ChannelEnd::Send.as_i32() && state.closing { return Err(ChanErr::Closed); } if !state.open { return Err(ChanErr::Closed); } if !force && !state.queue.is_empty() { - if end != CHANNEL_SEND { + if end != ChannelEnd::Send.as_i32() { return Err(ChanErr::NotEmpty); } if state.closing { @@ -476,7 +495,7 @@ pub(crate) mod _interpchannels { i += 1; continue; } - if item.unboundop == crossinterp::UNBOUND_REMOVE { + if item.unboundop == UnboundOp::Remove { let item = state.queue.remove(i).expect("index checked"); if let Some(w) = item.waiting { waiters.push((w, false)); @@ -495,16 +514,12 @@ pub(crate) mod _interpchannels { } /// `resolve_unboundop`. - fn resolve_unboundop(arg: i32, default: i32, vm: &VirtualMachine) -> PyResult { + fn resolve_unboundop(arg: i32, default: UnboundOp, vm: &VirtualMachine) -> PyResult { if arg < 0 { return Ok(default); } - match arg { - crossinterp::UNBOUND_REMOVE - | crossinterp::UNBOUND_ERROR - | crossinterp::UNBOUND_REPLACE => Ok(arg), - _ => Err(vm.new_value_error(format!("unsupported unboundop {arg}"))), - } + UnboundOp::from_i32(arg) + .ok_or_else(|| vm.new_value_error(format!("unsupported unboundop {arg}"))) } /// `resolve_fallback`. @@ -517,7 +532,7 @@ pub(crate) mod _interpchannels { /// `channel_id_converter`. fn parse_cid(obj: &PyObject, vm: &VirtualMachine) -> PyResult<(i64, i32)> { if let Some(cid) = obj.downcast_ref::() { - return Ok((cid.cid, cid.end)); + return Ok((cid.cid, cid.end.as_i32())); } if !obj.number().is_index() { return Err(vm.new_type_error(format!( @@ -536,7 +551,7 @@ pub(crate) mod _interpchannels { vm.new_value_error(format!("channel ID must be a non-negative int, got {repr}")) ); } - Ok((id, CHANNEL_BOTH)) + Ok((id, ChannelEnd::Both.as_i32())) } /// The single-`cid` signature shared by several module functions. @@ -605,7 +620,7 @@ pub(crate) mod _interpchannels { #[derive(Debug, PyPayload)] pub(crate) struct ChannelID { cid: i64, - end: i32, + end: ChannelEnd, resolve: bool, } @@ -620,9 +635,9 @@ pub(crate) mod _interpchannels { fn repr_str(zelf: &Py, _vm: &VirtualMachine) -> PyResult { let name = zelf.class().name().to_string(); Ok(match zelf.end { - CHANNEL_SEND => format!("{name}({}, send=True)", zelf.cid), - CHANNEL_RECV => format!("{name}({}, recv=True)", zelf.cid), - _ => format!("{name}({})", zelf.cid), + ChannelEnd::Send => format!("{name}({}, send=True)", zelf.cid), + ChannelEnd::Recv => format!("{name}({}, recv=True)", zelf.cid), + ChannelEnd::Both => format!("{name}({})", zelf.cid), }) } } @@ -690,20 +705,20 @@ pub(crate) mod _interpchannels { #[pygetset] fn end(&self) -> String { match self.end { - CHANNEL_SEND => "send".to_owned(), - CHANNEL_RECV => "recv".to_owned(), - _ => "both".to_owned(), + ChannelEnd::Send => "send".to_owned(), + ChannelEnd::Recv => "recv".to_owned(), + ChannelEnd::Both => "both".to_owned(), } } #[pygetset(name = "send")] fn send_end(&self, vm: &VirtualMachine) -> PyResult { - channel_id_from_parts(self.cid, CHANNEL_SEND, true, self.resolve, vm) + channel_id_from_parts(self.cid, ChannelEnd::Send.as_i32(), true, self.resolve, vm) } #[pygetset(name = "recv")] fn recv_end(&self, vm: &VirtualMachine) -> PyResult { - channel_id_from_parts(self.cid, CHANNEL_RECV, true, self.resolve, vm) + channel_id_from_parts(self.cid, ChannelEnd::Recv.as_i32(), true, self.resolve, vm) } #[pymethod] @@ -725,11 +740,13 @@ pub(crate) mod _interpchannels { Err(ChanErr::NotFound) if force => {} Err(e) => return Err(e.into_py(cid, vm)), } + let end = ChannelEnd::from_i32(end).unwrap_or(ChannelEnd::Both); Ok(ChannelID { cid, end, resolve }.to_pyobject(vm)) } pub(crate) fn channel_id_parts(obj: &PyObject) -> Option<(i64, i32)> { - obj.downcast_ref::().map(|c| (c.cid, c.end)) + obj.downcast_ref::() + .map(|c| (c.cid, c.end.as_i32())) } /// `_channelid_new`. @@ -760,9 +777,9 @@ pub(crate) mod _interpchannels { (Some(false), Some(false)) => { return Err(vm.new_value_error("'send' and 'recv' cannot both be False")); } - (Some(true), Some(true)) => end = CHANNEL_BOTH, - (Some(true), _) => end = CHANNEL_SEND, - (_, Some(true)) => end = CHANNEL_RECV, + (Some(true), Some(true)) => end = ChannelEnd::Both.as_i32(), + (Some(true), _) => end = ChannelEnd::Send.as_i32(), + (_, Some(true)) => end = ChannelEnd::Recv.as_i32(), _ => {} } channel_id_from_parts(cid, end, force, resolve, vm) @@ -818,10 +835,10 @@ pub(crate) mod _interpchannels { .parse(&args, vm)?; let unboundarg = int_arg(parsed[0].as_deref(), vm)?.unwrap_or(-1); let fallbackarg = int_arg(parsed[1].as_deref(), vm)?.unwrap_or(-1); - let unboundop = resolve_unboundop(unboundarg, UNBOUND_REPLACE, vm)?; + let unboundop = resolve_unboundop(unboundarg, UnboundOp::Replace, vm)?; let fallback = resolve_fallback(fallbackarg, Fallback::Full.as_i32(), vm)?; let cid = channel_create(unboundop, fallback.as_i32()); - channel_id_from_parts(cid, CHANNEL_BOTH, false, false, vm) + channel_id_from_parts(cid, ChannelEnd::Both.as_i32(), false, false, vm) } #[pyfunction] @@ -840,7 +857,7 @@ pub(crate) mod _interpchannels { args.args.len() + args.kwargs.len() ))); } - let chans: Vec<(i64, i32, i32)> = channels() + let chans: Vec<(i64, UnboundOp, i32)> = channels() .lock() .refs .iter() @@ -854,8 +871,8 @@ pub(crate) mod _interpchannels { items.push( vm.ctx .new_tuple(vec![ - channel_id_from_parts(cid, CHANNEL_BOTH, false, false, vm)?, - vm.ctx.new_int(unboundop).into(), + channel_id_from_parts(cid, ChannelEnd::Both.as_i32(), false, false, vm)?, + vm.ctx.new_int(unboundop.as_i32()).into(), vm.ctx.new_int(fallback).into(), ]) .into(), @@ -924,7 +941,7 @@ pub(crate) mod _interpchannels { chan: &Channel, cid: i64, value: SharedValue, - unboundop: i32, + unboundop: UnboundOp, blocking: bool, timeout: Option, vm: &VirtualMachine, @@ -1001,7 +1018,14 @@ pub(crate) mod _interpchannels { args: &FuncArgs, func: &'static str, vm: &VirtualMachine, - ) -> PyResult<(i64, PyObjectRef, i32, Fallback, bool, Option)> { + ) -> PyResult<( + i64, + PyObjectRef, + UnboundOp, + Fallback, + bool, + Option, + )> { let parsed = ArgSpec { fname: func, keywords: &["cid", "obj", "unboundop", "fallback", "blocking", "timeout"], @@ -1032,7 +1056,7 @@ pub(crate) mod _interpchannels { let state = chan.state.lock(); (state.unboundop, state.fallback) } else { - (-1, -1) + (UnboundOp::Replace, -1) }; let unboundop = resolve_unboundop(unboundarg, default_unboundop, vm)?; let fallback = resolve_fallback(fallbackarg, default_fallback, vm)?; @@ -1104,7 +1128,10 @@ pub(crate) mod _interpchannels { // The item was unbound. return Ok(vm .ctx - .new_tuple(vec![vm.ctx.none(), vm.ctx.new_int(unboundop).into()]) + .new_tuple(vec![ + vm.ctx.none(), + vm.ctx.new_int(unboundop.as_i32()).into(), + ]) .into()); }; let obj = data.into_object(vm)?; @@ -1119,7 +1146,7 @@ pub(crate) mod _interpchannels { fn channel_next( cid: i64, interpid: i64, - ) -> ChanResult<(Option, i32, Option>)> { + ) -> ChanResult<(Option, UnboundOp, Option>)> { let chan = channels_lookup(cid)?; let mut state = chan.state.lock(); if !state.open { @@ -1271,7 +1298,7 @@ pub(crate) mod _interpchannels { Ok(vm .ctx .new_tuple(vec![ - vm.ctx.new_int(unboundop).into(), + vm.ctx.new_int(unboundop.as_i32()).into(), vm.ctx.new_int(fallback).into(), ]) .into()) diff --git a/crates/vm/src/stdlib/_interpqueues.rs b/crates/vm/src/stdlib/_interpqueues.rs index e4075f888..b91c78966 100644 --- a/crates/vm/src/stdlib/_interpqueues.rs +++ b/crates/vm/src/stdlib/_interpqueues.rs @@ -14,7 +14,7 @@ pub(crate) mod _interpqueues { builtins::{PyBaseExceptionRef, PyModule, PyType, PyTypeRef}, function::{ArgSpec, FuncArgs}, types::Constructor, - vm::crossinterp::{self, Fallback, SharedValue, UNBOUND_REMOVE, UNBOUND_REPLACE}, + vm::crossinterp::{Fallback, SharedValue, UnboundOp}, }; use alloc::{collections::BTreeMap, collections::VecDeque, sync::Arc}; use core::cell::Cell; @@ -160,14 +160,14 @@ pub(crate) mod _interpqueues { struct QueueItem { interpid: i64, data: Option, - unboundop: i32, + unboundop: UnboundOp, } struct Queue { alive: bool, maxsize: isize, items: VecDeque, - unboundop: i32, + unboundop: UnboundOp, fallback: Fallback, } @@ -206,7 +206,7 @@ pub(crate) mod _interpqueues { .ok_or(QueueErr::NotFound) } - fn queue_create(maxsize: isize, unboundop: i32, fallback: Fallback) -> QueueResult { + fn queue_create(maxsize: isize, unboundop: UnboundOp, fallback: Fallback) -> QueueResult { let mut queues = queues().lock(); let qid = queues.next_id; if qid < 0 { @@ -244,7 +244,7 @@ pub(crate) mod _interpqueues { Ok(()) } - fn queue_defaults(qid: i64) -> QueueResult<(i32, Fallback)> { + fn queue_defaults(qid: i64) -> QueueResult<(UnboundOp, Fallback)> { let queue = queue_lookup(qid)?; let state = queue.lock(); if !state.alive { @@ -281,16 +281,12 @@ pub(crate) mod _interpqueues { Ok(()) } - fn resolve_unboundop(arg: i32, default: i32, vm: &VirtualMachine) -> PyResult { + fn resolve_unboundop(arg: i32, default: UnboundOp, vm: &VirtualMachine) -> PyResult { if arg < 0 { return Ok(default); } - match arg { - crossinterp::UNBOUND_REMOVE - | crossinterp::UNBOUND_ERROR - | crossinterp::UNBOUND_REPLACE => Ok(arg), - _ => Err(vm.new_value_error(format!("unsupported unboundop {arg}"))), - } + UnboundOp::from_i32(arg) + .ok_or_else(|| vm.new_value_error(format!("unsupported unboundop {arg}"))) } fn resolve_fallback(arg: i32, default: i32, vm: &VirtualMachine) -> PyResult { @@ -393,7 +389,7 @@ pub(crate) mod _interpqueues { }, vm, )?; - let unboundop = resolve_unboundop(unboundarg.get().unwrap_or(-1), UNBOUND_REPLACE, vm)?; + let unboundop = resolve_unboundop(unboundarg.get().unwrap_or(-1), UnboundOp::Replace, vm)?; let fallback = resolve_fallback(fallbackarg.get().unwrap_or(-1), Fallback::Full.as_i32(), vm)?; queue_create(maxsize.get().unwrap(), unboundop, fallback).map_err(|err| err.into_py(-1, vm)) @@ -431,7 +427,7 @@ pub(crate) mod _interpqueues { vm.ctx .new_tuple(vec![ vm.ctx.new_int(qid).into(), - vm.ctx.new_int(state.unboundop).into(), + vm.ctx.new_int(state.unboundop.as_i32()).into(), vm.ctx.new_int(state.fallback.as_i32()).into(), ]) .into(), @@ -474,7 +470,7 @@ pub(crate) mod _interpqueues { let (unboundop, fallback) = queue_defaults(qid).map_err(|err| err.into_py(qid, vm))?; (unboundop, fallback.as_i32()) } else { - (-1, -1) + (UnboundOp::Replace, -1) }; let unboundop = resolve_unboundop(unboundarg, default_unboundop, vm)?; let fallback = resolve_fallback(fallbackarg, default_fallback, vm)?; @@ -515,7 +511,10 @@ pub(crate) mod _interpqueues { }; let (obj, unboundop) = match item.data { Some(data) => (data.into_object(vm)?, vm.ctx.none()), - None => (vm.ctx.none(), vm.ctx.new_int(item.unboundop).into()), + None => ( + vm.ctx.none(), + vm.ctx.new_int(item.unboundop.as_i32()).into(), + ), }; Ok(vm.ctx.new_tuple(vec![obj, unboundop]).into()) } @@ -550,7 +549,7 @@ pub(crate) mod _interpqueues { Ok(vm .ctx .new_tuple(vec![ - vm.ctx.new_int(unboundop).into(), + vm.ctx.new_int(unboundop.as_i32()).into(), vm.ctx.new_int(fallback.as_i32()).into(), ]) .into()) @@ -665,7 +664,7 @@ pub(crate) mod _interpqueues { index += 1; continue; } - if state.items[index].unboundop == UNBOUND_REMOVE { + if state.items[index].unboundop == UnboundOp::Remove { dropped.push(state.items.remove(index).and_then(|item| item.data)); } else { dropped.push(state.items[index].data.take()); diff --git a/crates/vm/src/stdlib/_io.rs b/crates/vm/src/stdlib/_io.rs index 3bec5b299..ac8bef74c 100644 --- a/crates/vm/src/stdlib/_io.rs +++ b/crates/vm/src/stdlib/_io.rs @@ -5713,7 +5713,7 @@ mod fileio { return Err(e); } - if mode.contains(host_io::FileMode::APPENDING) { + if mode.is_superset(&host_io::FileMode::APPENDING) { let _ = host_io::seek_to_end(fd); } @@ -5795,7 +5795,7 @@ mod fileio { if self.fd.load() < 0 { return Err(io_closed_error(vm)); } - Ok(self.mode.load().contains(host_io::FileMode::READABLE)) + Ok(self.mode.load().is_superset(&host_io::FileMode::READABLE)) } #[pymethod] @@ -5803,7 +5803,7 @@ mod fileio { if self.fd.load() < 0 { return Err(io_closed_error(vm)); } - Ok(self.mode.load().contains(host_io::FileMode::WRITABLE)) + Ok(self.mode.load().is_superset(&host_io::FileMode::WRITABLE)) } #[pygetset] @@ -5817,7 +5817,7 @@ mod fileio { read_byte: OptionalSize, vm: &VirtualMachine, ) -> PyResult>> { - if !zelf.mode.load().contains(host_io::FileMode::READABLE) { + if !zelf.mode.load().is_superset(&host_io::FileMode::READABLE) { return Err(new_unsupported_operation( "File or stream is not readable", vm, @@ -5895,7 +5895,7 @@ mod fileio { obj: ArgMemoryBuffer, vm: &VirtualMachine, ) -> PyResult> { - if !zelf.mode.load().contains(host_io::FileMode::READABLE) { + if !zelf.mode.load().is_superset(&host_io::FileMode::READABLE) { return Err(new_unsupported_operation( "File or stream is not readable", vm, @@ -5934,7 +5934,7 @@ mod fileio { obj: ArgBytesLike, vm: &VirtualMachine, ) -> PyResult> { - if !zelf.mode.load().contains(host_io::FileMode::WRITABLE) { + if !zelf.mode.load().is_superset(&host_io::FileMode::WRITABLE) { return Err(new_unsupported_operation( "File or stream is not writable", vm, diff --git a/crates/vm/src/stdlib/_stat.rs b/crates/vm/src/stdlib/_stat.rs index b8533555b..df7acef0d 100644 --- a/crates/vm/src/stdlib/_stat.rs +++ b/crates/vm/src/stdlib/_stat.rs @@ -92,140 +92,100 @@ mod _stat { #[pyattr] pub const S_IFWHT: Mode = rustpython_host_env::os::S_IFWHT as Mode; - // Permission bits + bitflagset::bitflag! { + #[derive(Copy, Clone, Debug, PartialEq, Eq)] + #[allow(non_camel_case_types)] + #[repr(u8)] + enum StatPerm { + S_IXOTH = 0, + S_IWOTH = 1, + S_IROTH = 2, + S_IXGRP = 3, + S_IWGRP = 4, + S_IRGRP = 5, + S_IXUSR = 6, + S_IWUSR = 7, + S_IRUSR = 8, + S_ISVTX = 9, + S_ISGID = 10, + S_ISUID = 11, + } + } + + bitflagset::bitflagset! { + #[derive(Copy, Clone, PartialEq, Eq)] + struct StatPerms(u32): StatPerm + } + + impl StatPerms { + const S_IRWXO: Self = + Self::from_slice(&[StatPerm::S_IXOTH, StatPerm::S_IWOTH, StatPerm::S_IROTH]); + const S_IRWXG: Self = + Self::from_slice(&[StatPerm::S_IXGRP, StatPerm::S_IWGRP, StatPerm::S_IRGRP]); + const S_IRWXU: Self = + Self::from_slice(&[StatPerm::S_IXUSR, StatPerm::S_IWUSR, StatPerm::S_IRUSR]); + } + + const fn perm(flag: StatPerm) -> Mode { + StatPerms::from_element(flag).bits() as Mode + } #[pyattr] - pub const S_ISUID: Mode = libc_const!( - #[cfg(unix)] - S_ISUID, - 0o4000 - ); - + pub const S_IXOTH: Mode = perm(StatPerm::S_IXOTH); #[pyattr] - pub const S_ISGID: Mode = libc_const!( - #[cfg(unix)] - S_ISGID, - 0o2000 - ); - + pub const S_IWOTH: Mode = perm(StatPerm::S_IWOTH); #[pyattr] - pub const S_ENFMT: Mode = libc_const!( - #[cfg(unix)] - S_ISGID, - 0o2000 - ); - + pub const S_IROTH: Mode = perm(StatPerm::S_IROTH); #[pyattr] - pub const S_ISVTX: Mode = libc_const!( - #[cfg(unix)] - S_ISVTX, - 0o1000 - ); - + pub const S_IRWXO: Mode = StatPerms::S_IRWXO.bits() as Mode; #[pyattr] - pub const S_IRWXU: Mode = libc_const!( - #[cfg(unix)] - S_IRWXU, - 0o0700 - ); - + pub const S_IXGRP: Mode = perm(StatPerm::S_IXGRP); #[pyattr] - pub const S_IRUSR: Mode = libc_const!( - #[cfg(unix)] - S_IRUSR, - 0o0400 - ); - + pub const S_IWGRP: Mode = perm(StatPerm::S_IWGRP); #[pyattr] - pub const S_IREAD: Mode = libc_const!( - #[cfg(unix)] - S_IRUSR, - 0o0400 - ); - + pub const S_IRGRP: Mode = perm(StatPerm::S_IRGRP); #[pyattr] - pub const S_IWUSR: Mode = libc_const!( - #[cfg(unix)] - S_IWUSR, - 0o0200 - ); - + pub const S_IRWXG: Mode = StatPerms::S_IRWXG.bits() as Mode; #[pyattr] - pub const S_IXUSR: Mode = libc_const!( - #[cfg(unix)] - S_IXUSR, - 0o0100 - ); - + pub const S_IXUSR: Mode = perm(StatPerm::S_IXUSR); #[pyattr] - pub const S_IRWXG: Mode = libc_const!( - #[cfg(unix)] - S_IRWXG, - 0o0070 - ); - + pub const S_IWUSR: Mode = perm(StatPerm::S_IWUSR); #[pyattr] - pub const S_IRGRP: Mode = libc_const!( - #[cfg(unix)] - S_IRGRP, - 0o0040 - ); - + pub const S_IRUSR: Mode = perm(StatPerm::S_IRUSR); #[pyattr] - pub const S_IWGRP: Mode = libc_const!( - #[cfg(unix)] - S_IWGRP, - 0o0020 - ); - + pub const S_IRWXU: Mode = StatPerms::S_IRWXU.bits() as Mode; #[pyattr] - pub const S_IXGRP: Mode = libc_const!( - #[cfg(unix)] - S_IXGRP, - 0o0010 - ); - + pub const S_ISVTX: Mode = perm(StatPerm::S_ISVTX); #[pyattr] - pub const S_IRWXO: Mode = libc_const!( - #[cfg(unix)] - S_IRWXO, - 0o0007 - ); - + pub const S_ISGID: Mode = perm(StatPerm::S_ISGID); #[pyattr] - pub const S_IROTH: Mode = libc_const!( - #[cfg(unix)] - S_IROTH, - 0o0004 - ); - + pub const S_ISUID: Mode = perm(StatPerm::S_ISUID); #[pyattr] - pub const S_IWOTH: Mode = libc_const!( - #[cfg(unix)] - S_IWOTH, - 0o0002 - ); - + pub const S_ENFMT: Mode = S_ISGID; #[pyattr] - pub const S_IXOTH: Mode = libc_const!( - #[cfg(unix)] - S_IXOTH, - 0o0001 - ); - + pub const S_IREAD: Mode = S_IRUSR; #[pyattr] - pub const S_IWRITE: Mode = libc_const!( - #[cfg(all(unix, not(target_os = "android"), not(target_os = "redox")))] - S_IWRITE, - 0o0200 - ); - + pub const S_IWRITE: Mode = S_IWUSR; #[pyattr] - pub const S_IEXEC: Mode = libc_const!( - #[cfg(all(unix, not(target_os = "android"), not(target_os = "redox")))] - S_IEXEC, - 0o0100 - ); + pub const S_IEXEC: Mode = S_IXUSR; + + const _: () = { + assert!(S_IXOTH == 0o0001 as Mode); + assert!(S_IWOTH == 0o0002 as Mode); + assert!(S_IROTH == 0o0004 as Mode); + assert!(S_IRWXO == 0o0007 as Mode); + assert!(S_IXGRP == 0o0010 as Mode); + assert!(S_IWGRP == 0o0020 as Mode); + assert!(S_IRGRP == 0o0040 as Mode); + assert!(S_IRWXG == 0o0070 as Mode); + assert!(S_IXUSR == 0o0100 as Mode); + assert!(S_IWUSR == 0o0200 as Mode); + assert!(S_IRUSR == 0o0400 as Mode); + assert!(S_IRWXU == 0o0700 as Mode); + assert!(S_ISVTX == 0o1000 as Mode); + assert!(S_ISGID == 0o2000 as Mode); + assert!(S_ISUID == 0o4000 as Mode); + }; // Windows file attributes (if on Windows) @@ -246,8 +206,6 @@ mod _stat { IO_REPARSE_TAG_APPEXECLINK, IO_REPARSE_TAG_MOUNT_POINT, IO_REPARSE_TAG_SYMLINK, }; - // Unix file flags (if on Unix) - #[pyattr] pub use rustpython_host_env::os::{ SF_APPEND, SF_ARCHIVED, SF_DATALESS, SF_FIRMLINK, SF_IMMUTABLE, SF_NOUNLINK, SF_SETTABLE, diff --git a/crates/vm/src/stdlib/_testinternalcapi.rs b/crates/vm/src/stdlib/_testinternalcapi.rs index ffb3862ef..a20774938 100644 --- a/crates/vm/src/stdlib/_testinternalcapi.rs +++ b/crates/vm/src/stdlib/_testinternalcapi.rs @@ -1361,7 +1361,7 @@ fn name_already_seen( return Ok(true); } if let Some(set) = user_set { - return set.__contains__(name.as_object(), vm); + return set.contains(name.as_object(), vm); } Ok(false) } diff --git a/crates/vm/src/stdlib/_types.rs b/crates/vm/src/stdlib/_types.rs index 385ecf8cf..d1d02f720 100644 --- a/crates/vm/src/stdlib/_types.rs +++ b/crates/vm/src/stdlib/_types.rs @@ -106,8 +106,7 @@ mod _types { #[pyattr] fn ClassMethodDescriptorType(vm: &VirtualMachine) -> PyObjectRef { - // TODO: implement as separate type - vm.ctx.types.method_descriptor_type.to_owned().into() + vm.ctx.types.classmethod_descriptor_type.to_owned().into() } #[pyattr] diff --git a/crates/vm/src/stdlib/_warnings.rs b/crates/vm/src/stdlib/_warnings.rs index b8ef0aea2..99d8fc72d 100644 --- a/crates/vm/src/stdlib/_warnings.rs +++ b/crates/vm/src/stdlib/_warnings.rs @@ -28,39 +28,38 @@ mod _warnings { #[pyattr] fn filters(vm: &VirtualMachine) -> PyListRef { - vm.state.warnings.filters.clone() + vm.state.warnings.filters.to_owned() } #[pyattr] fn _defaultaction(vm: &VirtualMachine) -> PyStrRef { - vm.state.warnings.default_action.clone() + vm.state.warnings.default_action.to_owned() } #[pyattr] fn _onceregistry(vm: &VirtualMachine) -> PyDictRef { - vm.state.warnings.once_registry.clone() + vm.state.warnings.once_registry.to_owned() } #[pyattr] fn _warnings_context(vm: &VirtualMachine) -> PyObjectRef { - vm.state - .warnings - .context_var - .get_or_init(|| { - // Try to create a real ContextVar if _contextvars is available. - // During early startup it may not be importable yet, in which - // case we fall back to None. This is safe because - // context_aware_warnings defaults to False. - if let Ok(contextvars) = vm.import("_contextvars", 0) - && let Ok(cv_cls) = contextvars.get_attr("ContextVar", vm) - && let Ok(cv) = cv_cls.call(("_warnings_context",), vm) - { - cv - } else { - vm.ctx.none() - } - }) - .clone() + if let Some(ctx) = vm.state.warnings.context_var.get() { + return ctx.clone(); + } + // _warnings is initialized before _contextvars may be importable. + // Retry until ContextVar can be created; do not cache a None fallback. + let created = vm + .import("_contextvars", 0) + .ok() + .and_then(|m| m.get_attr("ContextVar", vm).ok()) + .and_then(|cv_cls| cv_cls.call(("_warnings_context",), vm).ok()); + match created { + Some(cv) => match vm.state.warnings.context_var.set(cv.clone()) { + Ok(()) => cv, + Err(_) => vm.state.warnings.context_var.get().cloned().unwrap_or(cv), + }, + None => vm.ctx.none(), + } } #[pyfunction] @@ -181,13 +180,20 @@ mod _warnings { let module = args.module.into_option(); - // Validate module_globals: must be None or a dict - if let Some(ref mg) = args.module_globals.into_option() - && !vm.is_none(mg) - && !mg.class().is(vm.ctx.types.dict_type) - { - return Err(vm.new_type_error("module_globals must be a dict")); - } + let source_line = if let Some(mg) = args.module_globals.into_option() { + if vm.is_none(&mg) { + None + } else if !mg.class().is(vm.ctx.types.dict_type) { + return Err(vm.new_type_error(format!( + "module_globals must be a dict, not '{}'", + mg.class().name() + ))); + } else { + crate::warn::get_source_line(&mg, args.lineno, vm)? + } + } else { + None + }; let category = if vm.is_none(&args.category) { None @@ -205,7 +211,7 @@ mod _warnings { args.lineno, module, registry, - None, // source_line + source_line, args.source.into_option(), vm, ) diff --git a/crates/vm/src/stdlib/builtins.rs b/crates/vm/src/stdlib/builtins.rs index 4ffa8f8d0..2f7145047 100644 --- a/crates/vm/src/stdlib/builtins.rs +++ b/crates/vm/src/stdlib/builtins.rs @@ -31,10 +31,7 @@ mod builtins { readline::{Readline, ReadlineResult}, stdlib::sys, types::PyComparisonOp, - vm::compile_mode::{ - CompilerFlags, PY_EVAL_INPUT, PY_FILE_INPUT, PY_FUNC_TYPE_INPUT, PY_SINGLE_INPUT, - compile_future_features_from_flags, - }, + vm::compile_mode::{CompileStart, CompilerFlags, compile_future_features_from_flags}, }; use itertools::Itertools; use num_traits::{Signed, ToPrimitive}; @@ -221,18 +218,18 @@ mod builtins { let future_features = merge_compile_future_features(flags, dont_inherit, vm); let start = if mode_str == "exec" { - PY_FILE_INPUT + CompileStart::File } else if mode_str == "eval" { - PY_EVAL_INPUT + CompileStart::Eval } else if mode_str == "single" { - PY_SINGLE_INPUT + CompileStart::Single } else if mode_str == "func_type" { if !is_ast_only { return Err(vm.new_value_error( "compile() mode 'func_type' requires flag PyCF_ONLY_AST", )); } - PY_FUNC_TYPE_INPUT + CompileStart::FuncType } else { let msg = if is_ast_only { "compile() mode must be 'exec', 'eval', 'single' or 'func_type'" @@ -332,7 +329,7 @@ mod builtins { vm.compile_string_object_with_flags( source, &filename.to_string_lossy(), - start, + start.as_i32(), compile_flags, feature_version, optimize as i32, @@ -766,18 +763,37 @@ mod builtins { .is_ok_and(|fd| fd == expected) }; - // Check if we should use rustyline (interactive terminal, not PTY child) - let use_rustyline = fd_matches(&stdin, 0) + let tty = fd_matches(&stdin, 0) && fd_matches(&stdout, 1) && std::io::stdin().is_terminal() - && !is_pty_child(); + && std::io::stdout().is_terminal(); - // Disable rustyline if prompt contains surrogates (not valid UTF-8 for terminal) + // Encode the prompt with stdout's encoding and reject embedded NULs + // (`strlen` on the encoded bytes). + if tty && let OptionalArg::Present(prompt) = &prompt { + let _ = vm.call_method(&stdout, "flush", ()); + let encoding = stdout.get_attr("encoding", vm)?; + let errors = stdout.get_attr("errors", vm)?; + if !vm.is_none(&encoding) { + let encoded = vm.call_method(prompt.as_object(), "encode", (encoding, errors))?; + let bytes = encoded + .downcast::() + .map_err(|_| vm.new_type_error("encode() must return bytes".to_owned()))?; + if bytes.as_bytes().contains(&0) { + return Err(vm.new_value_error( + "input: prompt string cannot contain null characters".to_owned(), + )); + } + } + } + + // rustyline is the interactive tty reader. Skip it in a PTY child + // (`pty.fork` + `setsid`): raw mode hangs there. let prompt_str = match &prompt { OptionalArg::Present(s) => s.to_str(), OptionalArg::Missing => Some(""), }; - let use_rustyline = use_rustyline && prompt_str.is_some(); + let use_rustyline = tty && !is_pty_child() && prompt_str.is_some(); if use_rustyline { let prompt = prompt_str.unwrap(); diff --git a/crates/vm/src/stdlib/itertools.rs b/crates/vm/src/stdlib/itertools.rs index 24e69b770..cb29cebe8 100644 --- a/crates/vm/src/stdlib/itertools.rs +++ b/crates/vm/src/stdlib/itertools.rs @@ -37,6 +37,13 @@ mod decl { impl PyItertoolsChain { #[pyslot] fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { + let args = + crate::types::drop_kwargs_if_init_overridden(&cls, Self::class(&vm.ctx), args); + if !args.kwargs.is_empty() { + return Err( + vm.new_type_error(format!("{}() takes no keyword arguments", Self::NAME)) + ); + } let args_list = PyList::from(args.args); Self { source: PyRwLock::new(Some(args_list.to_pyobject(vm).get_iter(vm)?)), @@ -248,6 +255,7 @@ mod decl { impl Constructor for PyItertoolsCycle { type Args = PyIter; + const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true; fn py_new(_cls: &Py, iter: Self::Args, _vm: &VirtualMachine) -> PyResult { Ok(Self { @@ -388,6 +396,7 @@ mod decl { impl Constructor for PyItertoolsStarmap { type Args = StarmapNewArgs; + const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true; fn py_new( _cls: &Py, @@ -437,6 +446,7 @@ mod decl { impl Constructor for PyItertoolsTakewhile { type Args = TakewhileNewArgs; + const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true; fn py_new( _cls: &Py, @@ -500,6 +510,7 @@ mod decl { impl Constructor for PyItertoolsDropwhile { type Args = DropwhileNewArgs; + const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true; fn py_new( _cls: &Py, @@ -545,8 +556,7 @@ mod decl { struct GroupByState { current_value: Option, current_key: Option, - #[pytraverse(skip)] - next_group: bool, + tgtkey: Option, #[pytraverse(skip)] grouper: Option>, } @@ -556,7 +566,7 @@ mod decl { f.debug_struct("GroupByState") .field("current_value", &self.current_value) .field("current_key", &self.current_key) - .field("next_group", &self.next_group) + .field("tgtkey", &self.tgtkey) .finish() } } @@ -632,41 +642,43 @@ mod decl { impl IterNext for PyItertoolsGroupBy { fn next(zelf: &Py, vm: &VirtualMachine) -> PyResult { - let mut state = zelf.state.lock(); - state.grouper = None; + { + let mut state = zelf.state.lock(); + state.grouper = None; + } - if !state.next_group { - // FIXME: unnecessary clone. current_key always exist until assigning new - let current_key = state.current_key.clone(); - drop(state); - - let (value, key) = if let Some(old_key) = current_key { - loop { - let (value, new_key) = raise_if_stop!(zelf.advance(vm)?); - if !vm.bool_eq(&new_key, &old_key)? { - break (value, new_key); + loop { + let (tgtkey, currkey) = { + let state = zelf.state.lock(); + (state.tgtkey.clone(), state.current_key.clone()) + }; + match (tgtkey, currkey) { + (_, None) => {} + (None, Some(_)) => break, + (Some(tgtkey), Some(currkey)) => { + if !vm.bool_eq(&tgtkey, &currkey)? { + break; } } - } else { - raise_if_stop!(zelf.advance(vm)?) - }; - - state = zelf.state.lock(); + } + let (value, key) = raise_if_stop!(zelf.advance(vm)?); + let mut state = zelf.state.lock(); state.current_value = Some(value); state.current_key = Some(key); } - state.next_group = false; + let mut state = zelf.state.lock(); + let currkey = state.current_key.clone().unwrap(); + state.tgtkey = Some(currkey.clone()); let grouper = PyItertoolsGrouper { groupby: zelf.to_owned(), + tgtkey: currkey.clone(), } .into_ref(&vm.ctx); state.grouper = Some(grouper.downgrade(None, vm).unwrap()); - Ok(PyIterReturn::Return( - (state.current_key.as_ref().unwrap().clone(), grouper).to_pyobject(vm), - )) + Ok(PyIterReturn::Return((currkey, grouper).to_pyobject(vm))) } } @@ -675,6 +687,7 @@ mod decl { #[derive(Debug, PyPayload)] struct PyItertoolsGrouper { groupby: PyRef, + tgtkey: PyObjectRef, } #[pyclass(with(IterNext, Iterable), flags(HAS_WEAKREF))] @@ -684,30 +697,38 @@ mod decl { impl IterNext for PyItertoolsGrouper { fn next(zelf: &Py, vm: &VirtualMachine) -> PyResult { - let old_key = { - let mut state = zelf.groupby.state.lock(); + if !zelf.groupby.state.lock().is_current(zelf) { + return Ok(PyIterReturn::StopIteration(None)); + } - if !state.is_current(zelf) { - return Ok(PyIterReturn::StopIteration(None)); - } - - // check to see if the value has already been retrieved from the iterator - if let Some(val) = state.current_value.take() { - return Ok(PyIterReturn::Return(val)); - } - - state.current_key.as_ref().unwrap().clone() - }; - let (value, key) = raise_if_stop!(zelf.groupby.advance(vm)?); - if vm.bool_eq(&key, &old_key)? { - Ok(PyIterReturn::Return(value)) - } else { + if zelf.groupby.state.lock().current_value.is_none() { + let (value, key) = raise_if_stop!(zelf.groupby.advance(vm)?); let mut state = zelf.groupby.state.lock(); state.current_value = Some(value); state.current_key = Some(key); - state.next_group = true; - state.grouper = None; - Ok(PyIterReturn::StopIteration(None)) + } + + let currkey = { + let state = zelf.groupby.state.lock(); + if !state.is_current(zelf) { + return Ok(PyIterReturn::StopIteration(None)); + } + state.current_key.clone().unwrap() + }; + let tgtkey = zelf.tgtkey.clone(); + if !vm.bool_eq(&tgtkey, &currkey)? { + return Ok(PyIterReturn::StopIteration(None)); + } + + let mut state = zelf.groupby.state.lock(); + if !state.is_current(zelf) { + return Ok(PyIterReturn::StopIteration(None)); + } + let value = state.current_value.take(); + state.current_key = None; + match value { + Some(v) => Ok(PyIterReturn::Return(v)), + None => Ok(PyIterReturn::StopIteration(None)), } } } @@ -716,7 +737,7 @@ mod decl { #[pyclass(name = "islice", traverse)] #[derive(Debug, PyPayload)] struct PyItertoolsIslice { - iterable: PyIter, + iterable: PyMutex>, #[pytraverse(skip)] cur: AtomicCell, #[pytraverse(skip)] @@ -727,24 +748,26 @@ mod decl { step: usize, } - // Restrict obj to ints with value 0 <= val <= sys.maxsize - // On failure (out of range, non-int object) a ValueError is raised. fn pyobject_to_opt_usize( obj: &PyObject, name: &'static str, vm: &VirtualMachine, ) -> PyResult { - let is_int = obj.fast_isinstance(vm.ctx.types.int_type); - if is_int { - let value = int::get_value(obj).to_usize(); - if let Some(value) = value { - // Only succeeds for values for which 0 <= value <= sys.maxsize - if value <= sys::MAXSIZE as usize { - return Ok(value); - } + let value = match obj.try_index(vm) { + Ok(i) => int::get_value(i.as_object()).to_usize(), + Err(e) + if e.fast_isinstance(vm.ctx.exceptions.type_error) + || e.fast_isinstance(vm.ctx.exceptions.overflow_error) => + { + None } + Err(e) => return Err(e), + }; + if let Some(value) = value + && value <= sys::MAXSIZE as usize + { + return Ok(value); } - // We don't have an int or value was < 0 or > sys.maxsize Err(vm.new_value_error(format!( "{name} argument for islice() must be None or an integer: 0 <= x <= sys.maxsize." ))) @@ -754,6 +777,8 @@ mod decl { impl PyItertoolsIslice { #[pyslot] fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { + let args = + crate::types::drop_kwargs_if_init_overridden(&cls, Self::class(&vm.ctx), args); let (iter, start, stop, step) = match args.args.len() { 0 | 1 => { return Err(vm.new_arity_type_error(Self::NAME, 2..=4, args.args.len())); @@ -776,7 +801,13 @@ mod decl { ) = args.bind_for(vm, Self::NAME)?; let step = if !vm.is_none(&step) { - pyobject_to_opt_usize(&step, "Step", vm)? + let step = pyobject_to_opt_usize(&step, "Step", vm)?; + if step == 0 { + return Err(vm.new_value_error( + "Step for islice() must be a positive integer or None.", + )); + } + step } else { 1usize }; @@ -801,7 +832,7 @@ mod decl { let iter = iter.get_iter(vm)?; Self { - iterable: iter, + iterable: PyMutex::new(Some(iter)), cur: AtomicCell::new(0), next: AtomicCell::new(start), stop, @@ -816,24 +847,38 @@ mod decl { impl IterNext for PyItertoolsIslice { fn next(zelf: &Py, vm: &VirtualMachine) -> PyResult { + let Some(iterable) = zelf.iterable.lock().clone() else { + return Ok(PyIterReturn::StopIteration(None)); + }; + let stop = zelf.stop.unwrap_or(usize::MAX); + while zelf.cur.load() < zelf.next.load() { - zelf.iterable.next(vm)?; + raise_if_stop!({ + let result = iterable.next(vm)?; + if matches!(result, PyIterReturn::StopIteration(_)) { + *zelf.iterable.lock() = None; + } + result + }); zelf.cur.fetch_add(1); } - - if let Some(stop) = zelf.stop - && zelf.cur.load() >= stop - { + if zelf.cur.load() >= stop { + *zelf.iterable.lock() = None; return Ok(PyIterReturn::StopIteration(None)); } - let obj = raise_if_stop!(zelf.iterable.next(vm)?); + let obj = raise_if_stop!({ + let result = iterable.next(vm)?; + if matches!(result, PyIterReturn::StopIteration(_)) { + *zelf.iterable.lock() = None; + } + result + }); zelf.cur.fetch_add(1); - - // TODO is this overflow check required? attempts to copy CPython. - let (next, ovf) = zelf.next.load().overflowing_add(zelf.step); - zelf.next.store(if ovf { zelf.stop.unwrap() } else { next }); - + let oldnext = zelf.next.load(); + let (newnext, ovf) = oldnext.overflowing_add(zelf.step); + zelf.next + .store(if ovf || newnext > stop { stop } else { newnext }); Ok(PyIterReturn::Return(obj)) } } @@ -856,6 +901,7 @@ mod decl { impl Constructor for PyItertoolsFilterFalse { type Args = FilterFalseNewArgs; + const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true; fn py_new( _cls: &Py, diff --git a/crates/vm/src/stdlib/marshal.rs b/crates/vm/src/stdlib/marshal.rs index b13724645..066c3d740 100644 --- a/crates/vm/src/stdlib/marshal.rs +++ b/crates/vm/src/stdlib/marshal.rs @@ -10,8 +10,8 @@ mod decl { PyObject, PyObjectRef, PyResult, TryFromObject, VirtualMachine, builtins::{ PyBaseExceptionRef, PyBool, PyByteArray, PyBytes, PyCode, PyComplex, PyDict, - PyEllipsis, PyFloat, PyFrozenSet, PyInt, PyList, PyNone, PySet, PyStopIteration, PyStr, - PyTuple, + PyEllipsis, PyFloat, PyFrozenSet, PyInt, PyList, PyMemoryView, PyNone, PySet, + PyStopIteration, PyStr, PyTuple, }, convert::ToPyObject, function::{ArgBytesLike, OptionalArg}, @@ -386,18 +386,10 @@ mod decl { buf.write_u8(b'0'); // TYPE_NULL terminator } else if let Some(s) = obj.downcast_ref::() { buf.write_u8(b'<'); - let elems = s.elements(); - buf.write_u32(elems.len() as u32); - for elem in &elems { - write_object_depth(buf, elem, refs, version, allow_code, vm, depth - 1)?; - } + write_set_elements(buf, &s.elements(), refs, version, allow_code, vm, depth)?; } else if let Some(s) = obj.downcast_ref::() { buf.write_u8(b'>'); - let elems = s.elements(); - buf.write_u32(elems.len() as u32); - for elem in &elems { - write_object_depth(buf, elem, refs, version, allow_code, vm, depth - 1)?; - } + write_set_elements(buf, &s.elements(), refs, version, allow_code, vm, depth)?; } else if let Some(co) = obj.downcast_ref::() { if !allow_code { return Err(vm.new_value_error("marshalling code objects is disallowed")); @@ -454,6 +446,32 @@ mod decl { Ok(()) } + /// Serialize set elements in `sorted(v, key=marshal.dumps)` order. + fn write_set_elements( + buf: &mut Vec, + elems: &[PyObjectRef], + refs: &mut Option, + version: i32, + allow_code: bool, + vm: &VirtualMachine, + depth: usize, + ) -> PyResult<()> { + use marshal::Write; + buf.write_u32(elems.len() as u32); + let mut pairs = Vec::with_capacity(elems.len()); + for elem in elems { + let mut dumped = Vec::new(); + let mut inner_refs = (version >= 3).then(WriterRefTable::new); + write_object(&mut dumped, elem, &mut inner_refs, version, allow_code, vm)?; + pairs.push((dumped, elem.clone())); + } + pairs.sort_by(|a, b| a.0.cmp(&b.0)); + for (_, elem) in &pairs { + write_object_depth(buf, elem, refs, version, allow_code, vm, depth - 1)?; + } + Ok(()) + } + #[derive(FromArgs)] struct DumpArgs { value: PyObjectRef, @@ -771,30 +789,87 @@ mod decl { #[pyfunction] fn load(args: LoadArgs, vm: &VirtualMachine) -> PyResult { - // Read from file object into a buffer, one object at a time. - // We read all available data, deserialize one object, then seek - // back to just after the consumed bytes. - let tell_before = vm - .call_method(&args.f, "tell", ())? - .try_into_value::(vm)?; - let read_res = vm.call_method(&args.f, "read", ())?; - let bytes = ArgBytesLike::try_from_object(vm, read_res)?; - - // The borrow ends here: seek() below is the caller's, and reaching the - // same buffer from it would deadlock on a borrow still held. - let (result, consumed) = { - let buf = bytes.borrow_buf(); - let mut rdr: &[u8] = &buf; - let len_before = rdr.len(); - let result = deserialize_value(&mut rdr, args.allow_code, vm)?; - (result, len_before - rdr.len()) + let mut rdr = ReadableFile { + file: args.f, + vm, + buf: Vec::new(), + error: None, }; + let pending_error = RefCell::new(None); + let result = marshal::deserialize_value( + &mut rdr, + PyMarshalBag::new(vm, &pending_error, args.allow_code), + ); + if let Some(err) = rdr.error.take() { + return Err(err); + } + match result { + Ok(value) => Ok(value), + Err(error) => Err(pending_error.into_inner().unwrap_or_else(|| match error { + marshal::MarshalError::Eof => vm.new_eof_error("EOF read where not expected"), + marshal::MarshalError::EofObject => { + vm.new_eof_error("EOF read where object expected") + } + marshal::MarshalError::DataTooShort => vm.new_eof_error("marshal data too short"), + error @ marshal::MarshalError::NullObject => vm.new_type_error(error.to_string()), + error @ (marshal::MarshalError::BadSize(_) + | marshal::MarshalError::UnknownType + | marshal::MarshalError::InvalidRef) => { + vm.new_value_error(format!("bad marshal data ({error})")) + } + _ => vm.new_value_error("bad marshal data"), + })), + } + } - // Seek file to just after the consumed bytes - let new_pos = tell_before + consumed as i64; - vm.call_method(&args.f, "seek", (new_pos,))?; + /// File-backed marshal reader. `r_string` fills a scratch buffer via + /// `readinto` on a contiguous writable memoryview. + struct ReadableFile<'a> { + file: PyObjectRef, + vm: &'a VirtualMachine, + buf: Vec, + error: Option, + } - Ok(result) + impl ReadableFile<'_> { + fn r_string(&mut self, n: usize) -> PyResult<()> { + self.buf.clear(); + let bytearray = PyByteArray::from(vec![0u8; n]).into_ref(&self.vm.ctx); + let memoryview = PyMemoryView::from_object_with_flags( + bytearray.as_object(), + crate::protocol::BufferFlags::CONTIG, + self.vm, + )? + .into_ref(&self.vm.ctx); + let nread_obj = self.vm.call_method(&self.file, "readinto", (memoryview,))?; + let nread = nread_obj + .try_index(self.vm)? + .try_to_primitive::(self.vm)?; + let n_isize = isize::try_from(n).unwrap_or(isize::MAX); + if nread != n_isize { + if nread > n_isize { + return Err(self.vm.new_value_error(format!( + "read() returned too much data: {n} bytes requested, {nread} returned" + ))); + } + return Err(self.vm.new_eof_error("EOF read where not expected")); + } + self.buf.extend_from_slice(&bytearray.borrow_buf()); + Ok(()) + } + } + + impl marshal::Read for ReadableFile<'_> { + fn read_slice(&mut self, n: u32) -> Result<&[u8], marshal::MarshalError> { + if self.error.is_some() { + return Err(marshal::MarshalError::Eof); + } + if let Err(e) = self.r_string(n as usize) { + self.error = Some(e); + return Err(marshal::MarshalError::Eof); + } + Ok(&self.buf) + } } /// Reject subclasses of marshallable types (int, float, complex, tuple, etc.). diff --git a/crates/vm/src/stdlib/mod.rs b/crates/vm/src/stdlib/mod.rs index db5dfea1c..73b816c57 100644 --- a/crates/vm/src/stdlib/mod.rs +++ b/crates/vm/src/stdlib/mod.rs @@ -3,6 +3,7 @@ mod _abc; pub(crate) mod _ast; mod _codecs; mod _collections; +pub(crate) use _collections::ordered_dict::ordered_dict::PyOrderedDictItems; mod _functools; mod _imp; pub mod _io; diff --git a/crates/vm/src/stdlib/sys.rs b/crates/vm/src/stdlib/sys.rs index dcee2aaa0..d15e65603 100644 --- a/crates/vm/src/stdlib/sys.rs +++ b/crates/vm/src/stdlib/sys.rs @@ -983,6 +983,10 @@ pub mod sys { #[pyfunction] fn getprofile(vm: &VirtualMachine) -> PyObjectRef { + #[cfg(feature = "threading")] + if let Some(slot) = crate::vm::thread::current_thread_slot() { + return slot.profile_func.lock().clone(); + } vm.profile_func.borrow().clone() } @@ -1082,6 +1086,10 @@ pub mod sys { #[pyfunction] fn gettrace(vm: &VirtualMachine) -> PyObjectRef { + #[cfg(feature = "threading")] + if let Some(slot) = crate::vm::thread::current_thread_slot() { + return slot.trace_func.lock().clone(); + } vm.trace_func.borrow().clone() } @@ -1251,6 +1259,10 @@ pub mod sys { #[pyfunction] fn setprofile(function: PyObjectRef, vm: &VirtualMachine) { + #[cfg(feature = "threading")] + if let Some(slot) = crate::vm::thread::current_thread_slot() { + *slot.profile_func.lock() = function.clone(); + } vm.profile_func.replace(function); update_use_tracing(vm); } @@ -1274,6 +1286,10 @@ pub mod sys { #[pyfunction] fn settrace(function: PyObjectRef, vm: &VirtualMachine) { + #[cfg(feature = "threading")] + if let Some(slot) = crate::vm::thread::current_thread_slot() { + *slot.trace_func.lock() = function.clone(); + } vm.trace_func.replace(function); update_use_tracing(vm); // The rest of the current line already started before tracing was @@ -1288,6 +1304,17 @@ pub mod sys { fn _settraceallthreads(function: PyObjectRef, vm: &VirtualMachine) { let func = (!vm.is_none(&function)).then(|| function.clone()); *vm.state.global_trace_func.lock() = func; + #[cfg(feature = "threading")] + { + let registry = vm.state.thread_frames.lock(); + #[expect( + clippy::iter_over_hash_type, + reason = "every thread slot, order is irrelevant" + )] + for slot in registry.values() { + *slot.trace_func.lock() = function.clone(); + } + } vm.trace_func.replace(function); update_use_tracing(vm); } @@ -1296,6 +1323,17 @@ pub mod sys { fn _setprofileallthreads(function: PyObjectRef, vm: &VirtualMachine) { let func = (!vm.is_none(&function)).then(|| function.clone()); *vm.state.global_profile_func.lock() = func; + #[cfg(feature = "threading")] + { + let registry = vm.state.thread_frames.lock(); + #[expect( + clippy::iter_over_hash_type, + reason = "every thread slot, order is irrelevant" + )] + for slot in registry.values() { + *slot.profile_func.lock() = function.clone(); + } + } vm.profile_func.replace(function); update_use_tracing(vm); } diff --git a/crates/vm/src/stdlib/sys/monitoring.rs b/crates/vm/src/stdlib/sys/monitoring.rs index 87fb6cb7e..8166af9c0 100644 --- a/crates/vm/src/stdlib/sys/monitoring.rs +++ b/crates/vm/src/stdlib/sys/monitoring.rs @@ -12,76 +12,64 @@ const EVENTS_COUNT: usize = 19; const LOCAL_EVENTS_COUNT: usize = 11; const UNGROUPED_EVENTS_COUNT: usize = 18; -// Event bit positions -bitflags::bitflags! { - #[derive(Copy, Clone, Debug, PartialEq, Eq)] - pub struct MonitoringEvents: u32 { - const PY_START = 1 << 0; - const PY_RESUME = 1 << 1; - const PY_RETURN = 1 << 2; - const PY_YIELD = 1 << 3; - const CALL = 1 << 4; - const LINE = 1 << 5; - const INSTRUCTION = 1 << 6; - const JUMP = 1 << 7; - const BRANCH_LEFT = 1 << 8; - const BRANCH_RIGHT = 1 << 9; - const STOP_ITERATION = 1 << 10; - const RAISE = 1 << 11; - const EXCEPTION_HANDLED = 1 << 12; - const PY_UNWIND = 1 << 13; - const PY_THROW = 1 << 14; - const RERAISE = 1 << 15; - const C_RETURN = 1 << 16; - const C_RAISE = 1 << 17; - const BRANCH = 1 << 18; +/// Event identifier (`PY_MONITORING_EVENT_*`), stored in `tstate->what_event`. +/// `None` on the VM field is the `< 0` sentinel (not in a monitoring callback). +#[repr(i32)] +#[derive(Copy, Clone, Debug, PartialEq, Eq)] +pub(crate) enum MonitoringEvent { + PyStart = 0, + PyResume = 1, + PyReturn = 2, + PyYield = 3, + Call = 4, + Line = 5, + Instruction = 6, + Jump = 7, + BranchLeft = 8, + BranchRight = 9, + StopIteration = 10, + Raise = 11, + ExceptionHandled = 12, + PyUnwind = 13, + PyThrow = 14, + Reraise = 15, + CReturn = 16, + CRaise = 17, + Branch = 18, +} + +impl MonitoringEvent { + pub(crate) const fn mask(self) -> u32 { + 1 << (self as u32) + } + + pub(crate) const fn from_id(id: usize) -> Option { + match id { + 0 => Some(Self::PyStart), + 1 => Some(Self::PyResume), + 2 => Some(Self::PyReturn), + 3 => Some(Self::PyYield), + 4 => Some(Self::Call), + 5 => Some(Self::Line), + 6 => Some(Self::Instruction), + 7 => Some(Self::Jump), + 8 => Some(Self::BranchLeft), + 9 => Some(Self::BranchRight), + 10 => Some(Self::StopIteration), + 11 => Some(Self::Raise), + 12 => Some(Self::ExceptionHandled), + 13 => Some(Self::PyUnwind), + 14 => Some(Self::PyThrow), + 15 => Some(Self::Reraise), + 16 => Some(Self::CReturn), + 17 => Some(Self::CRaise), + 18 => Some(Self::Branch), + _ => None, + } } } -// Re-export as plain u32 constants for use in frame.rs hot-path checks -pub(crate) const EVENT_PY_START: u32 = MonitoringEvents::PY_START.bits(); -pub(crate) const EVENT_PY_RESUME: u32 = MonitoringEvents::PY_RESUME.bits(); -pub(crate) const EVENT_PY_RETURN: u32 = MonitoringEvents::PY_RETURN.bits(); -pub(crate) const EVENT_PY_YIELD: u32 = MonitoringEvents::PY_YIELD.bits(); -pub(crate) const EVENT_CALL: u32 = MonitoringEvents::CALL.bits(); -pub(crate) const EVENT_LINE: u32 = MonitoringEvents::LINE.bits(); -pub(crate) const EVENT_INSTRUCTION: u32 = MonitoringEvents::INSTRUCTION.bits(); -pub(crate) const EVENT_JUMP: u32 = MonitoringEvents::JUMP.bits(); -pub(crate) const EVENT_BRANCH_LEFT: u32 = MonitoringEvents::BRANCH_LEFT.bits(); -pub(crate) const EVENT_BRANCH_RIGHT: u32 = MonitoringEvents::BRANCH_RIGHT.bits(); -pub(crate) const EVENT_RAISE: u32 = MonitoringEvents::RAISE.bits(); -pub(crate) const EVENT_EXCEPTION_HANDLED: u32 = MonitoringEvents::EXCEPTION_HANDLED.bits(); -pub(crate) const EVENT_PY_UNWIND: u32 = MonitoringEvents::PY_UNWIND.bits(); -pub(crate) const EVENT_C_RETURN: u32 = MonitoringEvents::C_RETURN.bits(); -const EVENT_C_RAISE: u32 = MonitoringEvents::C_RAISE.bits(); -pub(crate) const EVENT_STOP_ITERATION: u32 = MonitoringEvents::STOP_ITERATION.bits(); -pub(crate) const EVENT_PY_THROW: u32 = MonitoringEvents::PY_THROW.bits(); -const EVENT_BRANCH: u32 = MonitoringEvents::BRANCH.bits(); -pub(crate) const EVENT_RERAISE: u32 = MonitoringEvents::RERAISE.bits(); - -/// `tstate->what_event` values, the event currently being monitored. -pub(crate) const WHAT_PY_START: i32 = 0; -pub(crate) const WHAT_PY_RESUME: i32 = 1; -pub(crate) const WHAT_PY_RETURN: i32 = 2; -pub(crate) const WHAT_PY_YIELD: i32 = 3; -pub(crate) const WHAT_CALL: i32 = 4; -pub(crate) const WHAT_LINE: i32 = 5; -pub(crate) const WHAT_INSTRUCTION: i32 = 6; -pub(crate) const WHAT_JUMP: i32 = 7; -pub(crate) const WHAT_BRANCH_LEFT: i32 = 8; -pub(crate) const WHAT_BRANCH_RIGHT: i32 = 9; -#[allow(dead_code)] -pub(crate) const WHAT_STOP_ITERATION: i32 = 10; -pub(crate) const WHAT_RAISE: i32 = 11; -pub(crate) const WHAT_EXCEPTION_HANDLED: i32 = 12; -pub(crate) const WHAT_PY_UNWIND: i32 = 13; -pub(crate) const WHAT_PY_THROW: i32 = 14; -#[allow(dead_code)] -pub(crate) const WHAT_RERAISE: i32 = 15; -pub(crate) const WHAT_C_RETURN: i32 = 16; -pub(crate) const WHAT_C_RAISE: i32 = 17; -pub(crate) const WHAT_BRANCH: i32 = 18; -const EVENT_C_RETURN_MASK: u32 = EVENT_C_RETURN | EVENT_C_RAISE; +const C_RETURN_MASK: u32 = MonitoringEvent::CReturn.mask() | MonitoringEvent::CRaise.mask(); const EVENT_NAMES: [&str; EVENTS_COUNT] = [ "PY_START", @@ -210,7 +198,7 @@ fn parse_single_event(event: i32, vm: &VirtualMachine) -> PyResult { return Err(vm.new_value_error("The callback can only be set for one event at a time")); } let event_id = event.trailing_zeros() as usize; - if event_id >= EVENTS_COUNT { + if MonitoringEvent::from_id(event_id).is_none() { return Err(vm.new_value_error(format!("invalid event {event}"))); } Ok(event_id) @@ -231,15 +219,17 @@ fn normalize_event_set(event_set: i32, local: bool, vm: &VirtualMachine) -> PyRe return Err(vm.new_value_error(format!("invalid {kind} 0x{event_set:x}"))); } - if (event_set & EVENT_C_RETURN_MASK) != 0 && (event_set & EVENT_CALL) != EVENT_CALL { + if (event_set & C_RETURN_MASK) != 0 + && (event_set & MonitoringEvent::Call.mask()) != MonitoringEvent::Call.mask() + { return Err(vm.new_value_error("cannot set C_RETURN or C_RAISE events independently")); } - event_set &= !EVENT_C_RETURN_MASK; + event_set &= !C_RETURN_MASK; - if (event_set & EVENT_BRANCH) != 0 { - event_set &= !EVENT_BRANCH; - event_set |= EVENT_BRANCH_LEFT | EVENT_BRANCH_RIGHT; + if (event_set & MonitoringEvent::Branch.mask()) != 0 { + event_set &= !MonitoringEvent::Branch.mask(); + event_set |= MonitoringEvent::BranchLeft.mask() | MonitoringEvent::BranchRight.mask(); } if local && event_set >= (1 << LOCAL_EVENTS_COUNT) { @@ -251,29 +241,32 @@ fn normalize_event_set(event_set: i32, local: bool, vm: &VirtualMachine) -> PyRe /// Event that causes this opcode to be rewritten to INSTRUMENTED_*. /// RESUME uses `oparg`: 0 → PY_START, nonzero → PY_RESUME. -fn event_for_opcode(op: rustpython_compiler_core::bytecode::Instruction, oparg: u8) -> Option { +fn event_for_opcode( + op: rustpython_compiler_core::bytecode::Instruction, + oparg: u8, +) -> Option { use rustpython_compiler_core::bytecode::Opcode; match op.deoptimize().as_opcode() { - Opcode::ReturnValue => Some(EVENT_PY_RETURN), + Opcode::ReturnValue => Some(MonitoringEvent::PyReturn), Opcode::Call | Opcode::CallKw | Opcode::CallFunctionEx | Opcode::LoadSuperAttr => { - Some(EVENT_CALL) + Some(MonitoringEvent::Call) } - Opcode::YieldValue => Some(EVENT_PY_YIELD), - Opcode::JumpForward | Opcode::JumpBackward => Some(EVENT_JUMP), + Opcode::YieldValue => Some(MonitoringEvent::PyYield), + Opcode::JumpForward | Opcode::JumpBackward => Some(MonitoringEvent::Jump), Opcode::PopJumpIfFalse | Opcode::PopJumpIfTrue | Opcode::PopJumpIfNone - | Opcode::PopJumpIfNotNone => Some(EVENT_BRANCH_RIGHT), - Opcode::ForIter => Some(EVENT_BRANCH_LEFT), - Opcode::PopIter => Some(EVENT_BRANCH_RIGHT), - Opcode::EndFor | Opcode::EndSend => Some(EVENT_STOP_ITERATION), - Opcode::NotTaken => Some(EVENT_BRANCH_LEFT), - Opcode::EndAsyncFor => Some(EVENT_BRANCH_RIGHT), + | Opcode::PopJumpIfNotNone => Some(MonitoringEvent::BranchRight), + Opcode::ForIter => Some(MonitoringEvent::BranchLeft), + Opcode::PopIter => Some(MonitoringEvent::BranchRight), + Opcode::EndFor | Opcode::EndSend => Some(MonitoringEvent::StopIteration), + Opcode::NotTaken => Some(MonitoringEvent::BranchLeft), + Opcode::EndAsyncFor => Some(MonitoringEvent::BranchRight), Opcode::Resume => { if oparg != 0 { - Some(EVENT_PY_RESUME) + Some(MonitoringEvent::PyResume) } else { - Some(EVENT_PY_START) + Some(MonitoringEvent::PyStart) } } _ => None, @@ -285,7 +278,7 @@ fn opcode_event_is_active( oparg: u8, events: u32, ) -> bool { - event_for_opcode(op, oparg).is_some_and(|ev| events & ev != 0) + event_for_opcode(op, oparg).is_some_and(|ev| events & ev.mask() != 0) } /// Walk real instructions, skipping specialization CACHE payloads. @@ -366,8 +359,9 @@ pub(crate) fn instrument_code(code: &Py, events: u32) { let caches = base_op.cache_entries(); // LINE/INSTRUCTION wrap every instruction, so specialized // CACHE payloads must be cleared first. - let want_instrumented = events & (EVENT_LINE | EVENT_INSTRUCTION) != 0 - || opcode_event_is_active(base_op, oparg, events); + let want_instrumented = + events & (MonitoringEvent::Line.mask() | MonitoringEvent::Instruction.mask()) != 0 + || opcode_event_is_active(base_op, oparg, events); let restore_base = if want_instrumented { !op.is_instrumented() && u8::from(base_op) != u8::from(op) } else { @@ -432,7 +426,7 @@ pub(crate) fn instrument_code(code: &Py, events: u32) { let op = code.code.instructions.read_op(i); let oparg = code.code.instructions.read_arg(i).as_u8(); let caches = op.deoptimize().cache_entries(); - if (events & (EVENT_LINE | EVENT_INSTRUCTION) != 0 + if (events & (MonitoringEvent::Line.mask() | MonitoringEvent::Instruction.mask()) != 0 || opcode_event_is_active(op, oparg, events)) && let Some(instrumented) = op.to_instrumented() { @@ -444,9 +438,9 @@ pub(crate) fn instrument_code(code: &Py, events: u32) { } } - // Phase 5: Place INSTRUMENTED_INSTRUCTION (if EVENT_INSTRUCTION is active) + // Phase 5: Place INSTRUMENTED_INSTRUCTION when Instruction is active // LINE/INSTRUCTION restore specialized opcodes first, so CACHE slots are valid. - if events & EVENT_INSTRUCTION != 0 { + if events & MonitoringEvent::Instruction.mask() != 0 { for i in first_traceable..len { let op = code.code.instructions[i].op; // Skip ExtendedArg @@ -471,10 +465,10 @@ pub(crate) fn instrument_code(code: &Py, events: u32) { } } - // Phase 6: Place INSTRUMENTED_LINE (if EVENT_LINE is active) + // Phase 6: Place INSTRUMENTED_LINE when Line is active // Mirrors CPython's initialize_lines: first determine which positions // are line starts, then mark branch/jump targets, then place opcodes. - if events & EVENT_LINE != 0 { + if events & MonitoringEvent::Line.mask() != 0 { // is_line_start[i] = true if position i should have INSTRUMENTED_LINE let mut is_line_start = vec![false; len]; let line_locations = rustpython_compiler_core::marshal::linetable_to_locations( @@ -728,9 +722,9 @@ fn register_callback( .callbacks .remove(&(tool, event_id)) .unwrap_or_else(|| vm.ctx.none()); - let branch_id = EVENT_BRANCH.trailing_zeros() as usize; - let branch_left_id = EVENT_BRANCH_LEFT.trailing_zeros() as usize; - let branch_right_id = EVENT_BRANCH_RIGHT.trailing_zeros() as usize; + let branch_id = MonitoringEvent::Branch as usize; + let branch_left_id = MonitoringEvent::BranchLeft as usize; + let branch_right_id = MonitoringEvent::BranchRight as usize; if !vm.is_none(&func) { state.callbacks.insert((tool, event_id), func.clone()); // BRANCH is a composite event: also register for BRANCH_LEFT/RIGHT @@ -856,7 +850,7 @@ thread_local! { /// `cb_extra` contains the callback arguments after the code object. fn fire( vm: &VirtualMachine, - event: u32, + event: MonitoringEvent, code: &Py, offset: u32, cb_extra: &[PyObjectRef], @@ -866,14 +860,15 @@ fn fire( return Ok(()); } - let event_id = event.trailing_zeros() as usize; + let event_id = event as usize; + let event_mask = event.mask(); let code_id = code.get_id(); // C_RETURN and C_RAISE are implicitly enabled when CALL is set. - let check_bit = if event & EVENT_C_RETURN_MASK != 0 { - event | EVENT_CALL + let check_bit = if matches!(event, MonitoringEvent::CReturn | MonitoringEvent::CRaise) { + event_mask | MonitoringEvent::Call.mask() } else { - event + event_mask }; // Collect callbacks and snapshot the DISABLE sentinel under a single lock. @@ -911,7 +906,7 @@ fn fire( FIRING.with(|f| f.set(true)); vm.enter_tracing(); - let old_what = vm.what_event.replace(event_id as i32); + let old_what = vm.what_event.replace(Some(event)); let result = (|| { for (tool, cb) in callbacks { let result = cb.call(args.clone(), vm)?; @@ -945,7 +940,7 @@ fn fire( pub(crate) fn fire_py_start(vm: &VirtualMachine, code: &Py, offset: u32) -> PyResult<()> { fire( vm, - EVENT_PY_START, + MonitoringEvent::PyStart, code, offset, &[vm.ctx.new_int(offset).into()], @@ -955,7 +950,7 @@ pub(crate) fn fire_py_start(vm: &VirtualMachine, code: &Py, offset: u32) pub(crate) fn fire_py_resume(vm: &VirtualMachine, code: &Py, offset: u32) -> PyResult<()> { fire( vm, - EVENT_PY_RESUME, + MonitoringEvent::PyResume, code, offset, &[vm.ctx.new_int(offset).into()], @@ -970,7 +965,7 @@ pub(crate) fn fire_py_return( ) -> PyResult<()> { fire( vm, - EVENT_PY_RETURN, + MonitoringEvent::PyReturn, code, offset, &[vm.ctx.new_int(offset).into(), retval.to_owned()], @@ -985,7 +980,7 @@ pub(crate) fn fire_py_yield( ) -> PyResult<()> { fire( vm, - EVENT_PY_YIELD, + MonitoringEvent::PyYield, code, offset, &[vm.ctx.new_int(offset).into(), retval.to_owned()], @@ -1001,7 +996,7 @@ pub(crate) fn fire_call( ) -> PyResult<()> { fire( vm, - EVENT_CALL, + MonitoringEvent::Call, code, offset, &[vm.ctx.new_int(offset).into(), callable.to_owned(), arg0], @@ -1017,7 +1012,7 @@ pub(crate) fn fire_c_return( ) -> PyResult<()> { fire( vm, - EVENT_C_RETURN, + MonitoringEvent::CReturn, code, offset, &[vm.ctx.new_int(offset).into(), callable.to_owned(), arg0], @@ -1033,7 +1028,7 @@ pub(crate) fn fire_c_raise( ) -> PyResult<()> { fire( vm, - EVENT_C_RAISE, + MonitoringEvent::CRaise, code, offset, &[vm.ctx.new_int(offset).into(), callable.to_owned(), arg0], @@ -1046,7 +1041,13 @@ pub(crate) fn fire_line( offset: u32, line: u32, ) -> PyResult<()> { - fire(vm, EVENT_LINE, code, offset, &[vm.ctx.new_int(line).into()]) + fire( + vm, + MonitoringEvent::Line, + code, + offset, + &[vm.ctx.new_int(line).into()], + ) } pub(crate) fn fire_instruction( @@ -1056,7 +1057,7 @@ pub(crate) fn fire_instruction( ) -> PyResult<()> { fire( vm, - EVENT_INSTRUCTION, + MonitoringEvent::Instruction, code, offset, &[vm.ctx.new_int(offset).into()], @@ -1071,7 +1072,7 @@ pub(crate) fn fire_raise( ) -> PyResult<()> { fire( vm, - EVENT_RAISE, + MonitoringEvent::Raise, code, offset, &[vm.ctx.new_int(offset).into(), exception.to_owned()], @@ -1092,7 +1093,7 @@ pub(crate) fn fire_reraise( RERAISE_PENDING.with(|f| f.set(true)); let result = fire( vm, - EVENT_RERAISE, + MonitoringEvent::Reraise, code, offset, &[vm.ctx.new_int(offset).into(), exception.to_owned()], @@ -1112,7 +1113,7 @@ pub(crate) fn fire_exception_handled( RERAISE_PENDING.with(|f| f.set(false)); fire( vm, - EVENT_EXCEPTION_HANDLED, + MonitoringEvent::ExceptionHandled, code, offset, &[vm.ctx.new_int(offset).into(), exception.to_owned()], @@ -1128,7 +1129,7 @@ pub(crate) fn fire_py_unwind( RERAISE_PENDING.with(|f| f.set(false)); fire( vm, - EVENT_PY_UNWIND, + MonitoringEvent::PyUnwind, code, offset, &[vm.ctx.new_int(offset).into(), exception.to_owned()], @@ -1143,7 +1144,7 @@ pub(crate) fn fire_py_throw( ) -> PyResult<()> { fire( vm, - EVENT_PY_THROW, + MonitoringEvent::PyThrow, code, offset, &[vm.ctx.new_int(offset).into(), exception.to_owned()], @@ -1165,7 +1166,7 @@ pub(crate) fn fire_stop_iteration( }; fire( vm, - EVENT_STOP_ITERATION, + MonitoringEvent::StopIteration, code, offset, &[vm.ctx.new_int(offset).into(), exc], @@ -1180,7 +1181,7 @@ pub(crate) fn fire_jump( ) -> PyResult<()> { fire( vm, - EVENT_JUMP, + MonitoringEvent::Jump, code, offset, &[ @@ -1198,7 +1199,7 @@ pub(crate) fn fire_branch_left( ) -> PyResult<()> { fire( vm, - EVENT_BRANCH_LEFT, + MonitoringEvent::BranchLeft, code, offset, &[ @@ -1216,7 +1217,7 @@ pub(crate) fn fire_branch_right( ) -> PyResult<()> { fire( vm, - EVENT_BRANCH_RIGHT, + MonitoringEvent::BranchRight, code, offset, &[ diff --git a/crates/vm/src/types/slot.rs b/crates/vm/src/types/slot.rs index e95da0e82..182efcc66 100644 --- a/crates/vm/src/types/slot.rs +++ b/crates/vm/src/types/slot.rs @@ -250,6 +250,7 @@ bitflags! { const HEAPTYPE = 1 << 9; const BASETYPE = 1 << 10; const METHOD_DESCRIPTOR = 1 << 17; + const IS_ABSTRACT = 1 << 20; // For built-in types that match the subject itself in pattern matching // (bool, int, float, str, bytes, bytearray, list, tuple, dict, set, frozenset) // This is not a stable API @@ -1123,6 +1124,9 @@ impl PyType { // Check if it's a Python function (not a native descriptor) !attr.class().is(ctx.types.wrapper_descriptor_type) && !attr.class().is(ctx.types.method_descriptor_type) + && !attr + .class() + .is(ctx.types.classmethod_descriptor_type) }) }) }) @@ -1806,6 +1810,9 @@ impl PyType { pub trait Constructor: PyPayload + core::fmt::Debug { type Args: FromArgs; + /// When true, extra keywords are dropped if a subclass replaced `tp_init`. + const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = false; + /// The type slot for `__new__`. Override this only when you need special /// behavior beyond simple payload creation. #[inline] @@ -1813,6 +1820,11 @@ pub trait Constructor: PyPayload + core::fmt::Debug { fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult { // The name is the type the slot was written for, not the subclass being // constructed, so a subclass reports what its base declares. + let args = if Self::DROP_KWARGS_WHEN_INIT_OVERRIDDEN { + drop_kwargs_if_init_overridden(&cls, Self::class(&vm.ctx), args) + } else { + args + }; let args: Self::Args = args.bind_for(vm, Callee::of::(vm))?; let payload = Self::py_new(&cls, args, vm)?; payload.into_ref_with_type(vm, cls).map(Into::into) @@ -1823,6 +1835,22 @@ pub trait Constructor: PyPayload + core::fmt::Debug { fn py_new(cls: &Py, args: Self::Args, vm: &VirtualMachine) -> PyResult; } +/// Extra keywords are ignored when a subclass has replaced `tp_init`. +pub(crate) fn drop_kwargs_if_init_overridden( + cls: &Py, + base: &Py, + mut args: FuncArgs, +) -> FuncArgs { + if args.kwargs.is_empty() { + return args; + } + let uses_base_init = cls.slots.init.load().map(fn_addr) == base.slots.init.load().map(fn_addr); + if !(cls.is(base) || uses_base_init) { + args.kwargs = Default::default(); + } + args +} + pub trait DefaultConstructor: PyPayload + Default + core::fmt::Debug { fn construct_and_init(args: Self::Args, vm: &VirtualMachine) -> PyResult> where diff --git a/crates/vm/src/types/structseq.rs b/crates/vm/src/types/structseq.rs index ae379ab5d..19b8a9c14 100644 --- a/crates/vm/src/types/structseq.rs +++ b/crates/vm/src/types/structseq.rs @@ -1,5 +1,4 @@ use crate::common::lock::LazyLock; -use crate::common::wtf8::Wtf8; use crate::{ AsObject, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, atomic_func, builtins::{ @@ -340,10 +339,15 @@ pub trait PyStructSequence: StaticType + PyClassImpl + Sized + 'static { } } - // Check for unexpected keyword arguments if !kwargs.is_empty() { - let names: Vec<&Wtf8> = kwargs.keys().map(|k| k.as_ref()).collect(); - return Err(vm.new_type_error(format!("Got unexpected field name(s): {names:?}"))); + let names = vm.ctx.new_list( + kwargs + .keys() + .map(|k| vm.ctx.new_str(k.to_owned()).into()) + .collect(), + ); + let names_repr = names.as_object().repr(vm)?; + return Err(vm.new_type_error(format!("Got unexpected field name(s): {names_repr}"))); } PyTuple::new_unchecked(items.into_boxed_slice()) @@ -368,28 +372,19 @@ pub trait PyStructSequence: StaticType + PyClassImpl + Sized + 'static { fn extend_pyclass(ctx: &Context, class: &'static Py) { // Getters for named visible fields (indices 0 to REQUIRED_FIELD_NAMES.len() - 1) for (i, &name) in Self::Data::REQUIRED_FIELD_NAMES.iter().enumerate() { - // cast i to a u8 so there's less to store in the getter closure. - // Hopefully there's not struct sequences with >=256 elements :P - let i = i as u8; class.set_attr( ctx.intern_str(name), - ctx.new_readonly_getset(name, class, move |zelf: &Py| { - zelf[i as usize].to_owned() - }) - .into(), + ctx.new_readonly_tuple_member(name, class, i).into(), ); } // Getters for hidden/skipped fields (indices after visible fields) let visible_count = Self::Data::REQUIRED_FIELD_NAMES.len() + Self::Data::UNNAMED_FIELDS_LEN; for (i, &name) in Self::Data::OPTIONAL_FIELD_NAMES.iter().enumerate() { - let idx = (visible_count + i) as u8; class.set_attr( ctx.intern_str(name), - ctx.new_readonly_getset(name, class, move |zelf: &Py| { - zelf[idx as usize].to_owned() - }) - .into(), + ctx.new_readonly_tuple_member(name, class, visible_count + i) + .into(), ); } diff --git a/crates/vm/src/types/zoo.rs b/crates/vm/src/types/zoo.rs index 3cbca59d3..0d3c5d0fb 100644 --- a/crates/vm/src/types/zoo.rs +++ b/crates/vm/src/types/zoo.rs @@ -79,6 +79,7 @@ pub struct TypeZoo { pub builtin_function_or_method_type: &'static Py, pub builtin_method_type: &'static Py, pub method_descriptor_type: &'static Py, + pub classmethod_descriptor_type: &'static Py, pub property_type: &'static Py, pub getset_type: &'static Py, pub module_type: &'static Py, @@ -202,6 +203,7 @@ impl TypeZoo { weakproxy_type: weakproxy::PyWeakProxy::init_builtin_type(), weakcallableproxy_type: weakproxy::PyWeakCallableProxy::init_builtin_type(), method_descriptor_type: descriptor::PyMethodDescriptor::init_builtin_type(), + classmethod_descriptor_type: descriptor::PyClassMethodDescriptor::init_builtin_type(), none_type: singletons::PyNone::init_builtin_type(), typing_no_default_type: crate::stdlib::_typing::NoDefault::init_builtin_type(), not_implemented_type: singletons::PyNotImplemented::init_builtin_type(), diff --git a/crates/vm/src/vm/compile.rs b/crates/vm/src/vm/compile.rs index f12e8b3ef..804e6bbd4 100644 --- a/crates/vm/src/vm/compile.rs +++ b/crates/vm/src/vm/compile.rs @@ -8,10 +8,7 @@ use crate::{ AsObject, PyObjectRef, PyRef, PyResult, VirtualMachine, builtins::{PyBaseExceptionRef, PyCode}, compiler::{self, CompileError, CompileOpts}, - vm::compile_mode::{ - CompilerFlags, PY_EVAL_INPUT, PY_FILE_INPUT, PY_FUNC_TYPE_INPUT, PY_SINGLE_INPUT, - compile_future_features_from_flags, - }, + vm::compile_mode::{CompileStart, CompilerFlags, compile_future_features_from_flags}, }; #[derive(Debug)] @@ -388,6 +385,9 @@ impl VirtualMachine { use crate::stdlib::_ast; let cf = CompilerFlags::from_bits_retain(flags); + let Some(start) = CompileStart::from_i32(start) else { + return Err(self.new_system_error("Invalid start argument passed to Py_CompileString")); + }; let source = self.decode_source_bytes(source, filename, cf.contains(CompilerFlags::IGNORE_COOKIE))?; let source = source.as_str(); @@ -412,19 +412,15 @@ impl VirtualMachine { }; if is_ast_only { - if start == PY_FUNC_TYPE_INPUT { + if start == CompileStart::FuncType { return _ast::parse_func_type(self, source, filename, optimize, target_version) .map_err(|e| (e, Some(source), allow_incomplete).to_pyexception(self)); } let (parser_mode, interactive) = match start { - PY_SINGLE_INPUT => (ruff_python_parser::Mode::Module, true), - PY_FILE_INPUT => (ruff_python_parser::Mode::Module, false), - PY_EVAL_INPUT => (ruff_python_parser::Mode::Expression, false), - _ => { - return Err( - self.new_system_error("Invalid start argument passed to Py_CompileString") - ); - } + CompileStart::Single => (ruff_python_parser::Mode::Module, true), + CompileStart::File => (ruff_python_parser::Mode::Module, false), + CompileStart::Eval => (ruff_python_parser::Mode::Expression, false), + CompileStart::FuncType => unreachable!(), }; let parsed = _ast::parse( self, @@ -440,7 +436,7 @@ impl VirtualMachine { dont_imply_dedent, ) .map_err(|e| (e, Some(source), allow_incomplete).to_pyexception(self))?; - if start == PY_SINGLE_INPUT { + if start == CompileStart::Single { return _ast::wrap_interactive(self, &parsed); } return Ok(parsed); @@ -448,13 +444,9 @@ impl VirtualMachine { if type_comments { let parser_mode = match start { - PY_SINGLE_INPUT | PY_FILE_INPUT => ruff_python_parser::Mode::Module, - PY_EVAL_INPUT => ruff_python_parser::Mode::Expression, - _ => { - return Err( - self.new_system_error("Invalid start argument passed to Py_CompileString") - ); - } + CompileStart::Single | CompileStart::File => ruff_python_parser::Mode::Module, + CompileStart::Eval => ruff_python_parser::Mode::Expression, + CompileStart::FuncType => ruff_python_parser::Mode::Module, }; _ast::parse( self, @@ -465,7 +457,7 @@ impl VirtualMachine { None, type_comments, false, - start == PY_SINGLE_INPUT, + start == CompileStart::Single, future_features, dont_imply_dedent, ) @@ -473,15 +465,10 @@ impl VirtualMachine { } let mode = match start { - PY_SINGLE_INPUT => compiler::Mode::Single, - PY_FILE_INPUT => compiler::Mode::Exec, - PY_EVAL_INPUT => compiler::Mode::Eval, - PY_FUNC_TYPE_INPUT => compiler::Mode::BlockExpr, - _ => { - return Err( - self.new_system_error("Invalid start argument passed to Py_CompileString") - ); - } + CompileStart::Single => compiler::Mode::Single, + CompileStart::File => compiler::Mode::Exec, + CompileStart::Eval => compiler::Mode::Eval, + CompileStart::FuncType => compiler::Mode::BlockExpr, }; let mut opts = self.compile_opts(); opts.optimize = optimize; @@ -1650,7 +1637,7 @@ mod escape_warnings { vm.compile_string_object_with_flags( b"from __future__ import barry_as_FLUFL\n2 <> 3\n", "", - PY_FILE_INPUT, + CompileStart::File.as_i32(), flags, -1, -1, @@ -1661,7 +1648,7 @@ mod escape_warnings { .compile_string_object_with_flags( b"from __future__ import barry_as_FLUFL\n2 != 3\n", "", - PY_FILE_INPUT, + CompileStart::File.as_i32(), flags, -1, -1, @@ -1686,7 +1673,7 @@ mod escape_warnings { vm.compile_string_object_with_flags( b"2 <> 3\n", "", - PY_FILE_INPUT, + CompileStart::File.as_i32(), flags, -1, -1, diff --git a/crates/vm/src/vm/compile_mode.rs b/crates/vm/src/vm/compile_mode.rs index 6da531bbc..85967e0e6 100644 --- a/crates/vm/src/vm/compile_mode.rs +++ b/crates/vm/src/vm/compile_mode.rs @@ -1,9 +1,30 @@ use crate::bytecode; -pub(crate) const PY_SINGLE_INPUT: i32 = 256; -pub(crate) const PY_FILE_INPUT: i32 = 257; -pub(crate) const PY_EVAL_INPUT: i32 = 258; -pub(crate) const PY_FUNC_TYPE_INPUT: i32 = 345; +/// `Py_single_input` / `Py_file_input` / `Py_eval_input` / `Py_func_type_input`. +#[repr(i32)] +#[derive(Copy, Clone, Debug, PartialEq, Eq)] +pub(crate) enum CompileStart { + Single = 256, + File = 257, + Eval = 258, + FuncType = 345, +} + +impl CompileStart { + pub(crate) const fn from_i32(v: i32) -> Option { + match v { + 256 => Some(Self::Single), + 257 => Some(Self::File), + 258 => Some(Self::Eval), + 345 => Some(Self::FuncType), + _ => None, + } + } + + pub(crate) const fn as_i32(self) -> i32 { + self as i32 + } +} bitflags::bitflags! { /// `PyCF_*` compiler flags together with the `__future__` `CO_FUTURE_*` diff --git a/crates/vm/src/vm/context.rs b/crates/vm/src/vm/context.rs index 76a4b44f6..e6583b2fa 100644 --- a/crates/vm/src/vm/context.rs +++ b/crates/vm/src/vm/context.rs @@ -361,7 +361,9 @@ impl Context { names.__new__.as_str(), PyType::__new__, PyMethodFlags::METHOD, - None, + Some( + "__new__($type, /, *args, **kwargs)\n--\n\nCreate and return a new object. See help(type) for accurate signature.", + ), ); let empty_str = unsafe { string_pool.intern("", types.str_type.to_owned()) }; let empty_bytes = create_object(PyBytes::from(Vec::new()), types.bytes_type); @@ -680,6 +682,30 @@ impl Context { member_descriptor.into_ref(self) } + pub fn new_readonly_tuple_member( + &self, + name: &str, + class: &'static Py, + index: usize, + ) -> PyRef { + let member_def = PyMemberDef { + name: name.to_owned(), + kind: MemberKind::Object, + getter: MemberGetter::TupleItem(index), + setter: MemberSetter::Setter(None), + doc: None, + }; + let member_descriptor = PyMemberDescriptor { + common: PyDescriptorOwned { + typ: class.to_owned(), + name: self.intern_str(name), + qualname: PyRwLock::new(None), + }, + member: member_def, + }; + member_descriptor.into_ref(self) + } + pub fn new_readonly_getset( &self, name: &str, diff --git a/crates/vm/src/vm/crossinterp.rs b/crates/vm/src/vm/crossinterp.rs index c2bd72ab9..56bc7696d 100644 --- a/crates/vm/src/vm/crossinterp.rs +++ b/crates/vm/src/vm/crossinterp.rs @@ -33,9 +33,30 @@ fn memoryview_registered() -> bool { } /// Unbound-item ops (`UNBOUND_REMOVE` / `UNBOUND_ERROR` / `UNBOUND_REPLACE`). -pub const UNBOUND_REMOVE: i32 = 1; -pub const UNBOUND_ERROR: i32 = 2; -pub const UNBOUND_REPLACE: i32 = 3; +#[repr(i32)] +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum UnboundOp { + Remove = 1, + Error = 2, + Replace = 3, +} + +impl UnboundOp { + #[must_use] + pub const fn from_i32(v: i32) -> Option { + Some(match v { + 1 => Self::Remove, + 2 => Self::Error, + 3 => Self::Replace, + _ => return None, + }) + } + + #[must_use] + pub const fn as_i32(self) -> i32 { + self as i32 + } +} /// `xidata_fallback_t`: what to do with an object that has no XI data support. #[derive(Debug, Clone, Copy, PartialEq, Eq)] diff --git a/crates/vm/src/vm/interpreter.rs b/crates/vm/src/vm/interpreter.rs index 7c51c465f..b4b51a9ad 100644 --- a/crates/vm/src/vm/interpreter.rs +++ b/crates/vm/src/vm/interpreter.rs @@ -669,6 +669,7 @@ impl Interpreter { // Wait for thread shutdown - call threading._shutdown() if available. // This waits for all non-daemon threads to complete. // threading module may not be imported, so ignore import errors. + #[cfg(feature = "threading")] if let Ok(threading) = vm.import("threading", 0) && let Ok(shutdown) = threading.get_attr("_shutdown", vm) && let Err(e) = shutdown.call((), vm) @@ -1859,6 +1860,7 @@ for _ in range(40): fn eval_breaker_tripped_when_stop_requested() { let interp = Interpreter::without_stdlib(Default::default()); interp.enter(|vm| { + crate::signal::clear_eval_breaker_for_test(); assert!( crate::vm::thread::set_stop_requested_for_current_thread(true), "current thread has no stop_requested flag" @@ -1945,7 +1947,7 @@ for _ in range(40): #[cfg(feature = "threading")] #[test] fn a_thread_blocked_on_a_lock_does_not_stall_stop_the_world() { - use super::super::thread::THREAD_DETACHED; + use super::super::thread::ThreadState; use crate::common::lock::PyDetachingRwLock; use alloc::sync::Arc; use core::{ @@ -1986,11 +1988,9 @@ for _ in range(40): // hanging it, as the timeout on the stop below does. let deadline = std::time::Instant::now() + Duration::from_secs(10); let blocked_detached = |ident| { - state - .thread_frames - .lock() - .get(&ident) - .is_some_and(|slot| slot.state.load(Ordering::Acquire) == THREAD_DETACHED) + state.thread_frames.lock().get(&ident).is_some_and(|slot| { + slot.state.load(Ordering::Acquire) == ThreadState::Detached as i32 + }) }; loop { match worker_ident.load(Ordering::Acquire) { diff --git a/crates/vm/src/vm/mod.rs b/crates/vm/src/vm/mod.rs index a46141db0..c2b9f6bdd 100644 --- a/crates/vm/src/vm/mod.rs +++ b/crates/vm/src/vm/mod.rs @@ -96,8 +96,9 @@ pub struct VirtualMachine { pub profile_func: RefCell, pub trace_func: RefCell, pub use_tracing: Cell, - /// Event currently being monitored, or -1 when not in a callback. - pub(crate) what_event: Cell, + /// Event currently being monitored (`tstate->what_event`). + /// `None` when not in a monitoring callback. + pub(crate) what_event: Cell>, tracing_depth: Cell, pub recursion_limit: Cell, pub(crate) signal_handlers: OnceCell, @@ -299,7 +300,8 @@ impl StopTheWorldState { .iter() .filter(|(thread_id, slot)| { **thread_id != requester - && slot.state.load(Ordering::Relaxed) != thread::THREAD_SHUTTING_DOWN + && slot.state.load(Ordering::Relaxed) + != thread::ThreadState::ShuttingDown as i32 }) .count(); let count = (count.min(i64::MAX as usize)) as i64; @@ -323,7 +325,7 @@ impl StopTheWorldState { /// Try to CAS detached threads directly to SUSPENDED and check whether /// stop countdown reached zero after parking detached threads. fn park_detached_threads(&self, state: &PyGlobalState) -> bool { - use thread::{THREAD_ATTACHED, THREAD_DETACHED, THREAD_SHUTTING_DOWN, THREAD_SUSPENDED}; + use thread::ThreadState; let requester = self.requester.load(Ordering::Relaxed); let registry = state.thread_frames.lock(); let mut attached_seen = 0u64; @@ -339,11 +341,11 @@ impl StopTheWorldState { } let state = slot.state.load(Ordering::Relaxed); - if state == THREAD_DETACHED { + if state == ThreadState::Detached as i32 { // CAS DETACHED → SUSPENDED (park without thread cooperation) match slot.state.compare_exchange( - THREAD_DETACHED, - THREAD_SUSPENDED, + ThreadState::Detached as i32, + ThreadState::Suspended as i32, Ordering::AcqRel, Ordering::Relaxed, ) { @@ -351,38 +353,40 @@ impl StopTheWorldState { slot.stop_requested.store(false, Ordering::Release); forced_parks = forced_parks.saturating_add(1); } - Err(THREAD_ATTACHED) => { - // Set per-thread stop bit (_PY_EVAL_PLEASE_STOP_BIT). - slot.stop_requested.store(true, Ordering::Release); - crate::signal::set_stop_bit(); - // Raced with a thread re-attaching; it will self-suspend. - attached_seen = attached_seen.saturating_add(1); - } - Err(THREAD_DETACHED) => { - // Extremely unlikely race; next poll will handle it. - } - Err(THREAD_SUSPENDED) => { - slot.stop_requested.store(false, Ordering::Release); - // Another path parked it first. - } - Err(THREAD_SHUTTING_DOWN) => { - slot.stop_requested.store(false, Ordering::Release); - } - Err(other) => { - debug_assert!( - false, - "unexpected thread state in park_detached_threads: {other}" - ); - } + Err(actual) => match ThreadState::from_i32(actual) { + Some(ThreadState::Attached) => { + // Set per-thread stop bit (_PY_EVAL_PLEASE_STOP_BIT). + slot.stop_requested.store(true, Ordering::Release); + crate::signal::set_stop_bit(); + // Raced with a thread re-attaching; it will self-suspend. + attached_seen = attached_seen.saturating_add(1); + } + Some(ThreadState::Detached) => { + // Extremely unlikely race; next poll will handle it. + } + Some(ThreadState::Suspended) => { + slot.stop_requested.store(false, Ordering::Release); + // Another path parked it first. + } + Some(ThreadState::ShuttingDown) => { + slot.stop_requested.store(false, Ordering::Release); + } + None => { + debug_assert!( + false, + "unexpected thread state in park_detached_threads: {actual}" + ); + } + }, } - } else if state == THREAD_ATTACHED { + } else if state == ThreadState::Attached as i32 { // Set per-thread stop bit (_PY_EVAL_PLEASE_STOP_BIT). slot.stop_requested.store(true, Ordering::Release); crate::signal::set_stop_bit(); // Thread is in bytecode — it will see `requested` and self-suspend attached_seen = attached_seen.saturating_add(1); } - // THREAD_SUSPENDED / THREAD_SHUTTING_DOWN → already parked + // Suspended / ShuttingDown → already parked } if attached_seen != 0 { self.stats_attached_seen @@ -525,7 +529,7 @@ impl StopTheWorldState { /// Resume all suspended threads (`start_the_world`). pub fn start_the_world(&self, state: &PyGlobalState) { - use thread::{THREAD_DETACHED, THREAD_SHUTTING_DOWN, THREAD_SUSPENDED}; + use thread::ThreadState; let requester = self.requester.load(Ordering::Relaxed); stw_trace(format_args!("start begin requester={requester}")); let registry = state.thread_frames.lock(); @@ -549,17 +553,18 @@ impl StopTheWorldState { slot.stop_requested.store(false, Ordering::Release); let state = slot.state.load(Ordering::Relaxed); - if state == THREAD_SHUTTING_DOWN { + if state == ThreadState::ShuttingDown as i32 { // `_PyThreadState_RemoveExcept` + SetShuttingDown already // took this thread off the resume path. Leave it hanging. continue; } debug_assert!( - state == THREAD_SUSPENDED, + state == ThreadState::Suspended as i32, "non-requester thread not suspended at start-the-world: id={id} state={state}" ); - if state == THREAD_SUSPENDED { - slot.state.store(THREAD_DETACHED, Ordering::Release); + if state == ThreadState::Suspended as i32 { + slot.state + .store(ThreadState::Detached as i32, Ordering::Release); slot.thread.unpark(); } } @@ -660,7 +665,7 @@ impl StopTheWorldState { /// already SUSPENDED when a new stop counts it neither notifies nor is /// force-parked again, so an edge-based countdown could never reach zero. fn all_non_requester_suspended(&self, state: &PyGlobalState) -> bool { - use thread::{THREAD_SHUTTING_DOWN, THREAD_SUSPENDED}; + use thread::ThreadState; let requester = self.requester.load(Ordering::Relaxed); let registry = state.thread_frames.lock(); @@ -673,7 +678,9 @@ impl StopTheWorldState { continue; } let slot_state = slot.state.load(Ordering::Acquire); - if slot_state != THREAD_SUSPENDED && slot_state != THREAD_SHUTTING_DOWN { + if slot_state != ThreadState::Suspended as i32 + && slot_state != ThreadState::ShuttingDown as i32 + { return false; } } @@ -682,7 +689,7 @@ impl StopTheWorldState { #[cfg(debug_assertions)] fn debug_assert_all_non_requester_suspended(&self, state: &PyGlobalState) { - use thread::{THREAD_SHUTTING_DOWN, THREAD_SUSPENDED}; + use thread::ThreadState; let requester = self.requester.load(Ordering::Relaxed); let registry = state.thread_frames.lock(); @@ -697,7 +704,7 @@ impl StopTheWorldState { let state = slot.state.load(Ordering::Relaxed); debug_assert!( - state == THREAD_SUSPENDED || state == THREAD_SHUTTING_DOWN, + state == ThreadState::Suspended as i32 || state == ThreadState::ShuttingDown as i32, "non-requester thread not suspended during stop-the-world: id={id} state={state}" ); } @@ -705,7 +712,7 @@ impl StopTheWorldState { #[cfg(debug_assertions)] fn debug_assert_all_non_requester_detached(&self, state: &PyGlobalState) { - use thread::THREAD_SUSPENDED; + use thread::ThreadState; let requester = self.requester.load(Ordering::Relaxed); let registry = state.thread_frames.lock(); @@ -720,7 +727,7 @@ impl StopTheWorldState { let state = slot.state.load(Ordering::Relaxed); debug_assert!( - state != THREAD_SUSPENDED, + state != ThreadState::Suspended as i32, "non-requester thread still suspended after start-the-world: id={id} state={state}" ); } @@ -1230,7 +1237,7 @@ impl VirtualMachine { profile_func, trace_func, use_tracing: Cell::new(false), - what_event: Cell::new(-1), + what_event: Cell::new(None), tracing_depth: Cell::new(0), recursion_limit: Cell::new(if cfg!(debug_assertions) { 256 } else { 1000 }), signal_handlers, @@ -1325,13 +1332,13 @@ impl VirtualMachine { // Use dotted names when freeze-stdlib is enabled (modules come from Lib/encodings/), // otherwise use underscored names (modules come from core_modules/). - let (ascii_module_name, utf8_module_name) = if cfg!(feature = "freeze-stdlib") { - ("encodings.ascii", "encodings.utf_8") - } else { - ("encodings_ascii", "encodings_utf_8") - }; + let (ascii_module_name, utf8_module_name, latin1_module_name) = + if cfg!(feature = "freeze-stdlib") { + ("encodings.ascii", "encodings.utf_8", "encodings.latin_1") + } else { + ("encodings_ascii", "encodings_utf_8", "encodings_latin_1") + }; - // Register ascii encoding // __import__("encodings.ascii") returns top-level "encodings", so // look up the actual submodule in sys.modules. self.import(ascii_module_name, 0)?; @@ -1357,19 +1364,17 @@ impl VirtualMachine { .codec_registry .register_manual("utf8", utf8_codec); - // Register latin-1 / iso8859-1 aliases needed very early for stdio - // bootstrap (e.g. PYTHONIOENCODING=latin-1). - if cfg!(feature = "freeze-stdlib") { - self.import("encodings.latin_1", 0)?; - let latin1_module = sys_modules.get_item("encodings.latin_1", self)?; - let getregentry = latin1_module.get_attr("getregentry", self)?; - let codec_info = getregentry.call((), self)?; - let latin1_codec: crate::codecs::PyCodec = codec_info.try_into_value(self)?; - for name in ["latin-1", "latin_1", "latin1", "iso8859-1", "iso8859_1"] { - self.state - .codec_registry - .register_manual(name, latin1_codec.clone()); - } + // latin-1 is a built-in codec (needed very early for stdio bootstrap, + // e.g. PYTHONIOENCODING=latin-1). + self.import(latin1_module_name, 0)?; + let latin1_module = sys_modules.get_item(latin1_module_name, self)?; + let getregentry = latin1_module.get_attr("getregentry", self)?; + let codec_info = getregentry.call((), self)?; + let latin1_codec: crate::codecs::PyCodec = codec_info.try_into_value(self)?; + for name in ["latin-1", "latin_1", "latin1", "iso8859-1", "iso8859_1"] { + self.state + .codec_registry + .register_manual(name, latin1_codec.clone()); } Ok(()) } @@ -2976,7 +2981,7 @@ impl VirtualMachine { { let ret_result = self.trace_event_what( TraceEvent::Return, - crate::stdlib::sys::monitoring::WHAT_PY_UNWIND, + crate::stdlib::sys::monitoring::MonitoringEvent::PyUnwind, None, ); // call_trace_protected: if trace function raises, its error @@ -3842,6 +3847,7 @@ pub fn resolve_frozen_alias(name: &str) -> &str { "_frozen_importlib_external" => "importlib._bootstrap_external", "encodings_ascii" => "encodings.ascii", "encodings_utf_8" => "encodings.utf_8", + "encodings_latin_1" => "encodings.latin_1", "__hello_alias__" | "__phello_alias__" | "__phello_alias__.spam" => "__hello__", "__phello__.__init__" => "<__phello__", "__phello__.ham.__init__" => "<__phello__.ham", diff --git a/crates/vm/src/vm/thread.rs b/crates/vm/src/vm/thread.rs index f83f609dc..edbd1ebdf 100644 --- a/crates/vm/src/vm/thread.rs +++ b/crates/vm/src/vm/thread.rs @@ -1,9 +1,13 @@ #[cfg(all(not(unix), feature = "threading"))] use super::FramePtr; #[cfg(feature = "threading")] +use crate::PyObjectRef; +#[cfg(feature = "threading")] use crate::builtins::PyBaseExceptionRef; #[cfg(feature = "threading")] use alloc::sync::Arc; +#[cfg(feature = "threading")] +use rustpython_common::lock::PyMutex; use crate::frame::InterpreterFrame; #[cfg(feature = "threading")] @@ -23,20 +27,36 @@ use itertools::Itertools; use std::collections::HashMap; use std::thread_local; -// Thread states for stop-the-world support (`_Py_THREAD_*`). -// DETACHED: not executing Python bytecode (in native code, or idle) -// ATTACHED: actively executing Python bytecode -// SUSPENDED: parked by a stop-the-world request -// SHUTTING_DOWN: interpreter is finalizing; the OS thread must hang -// (`_PyThreadState_HangThread`) and must not look done to `_ThreadHandle`. +/// Thread states for stop-the-world support (`_Py_THREAD_*`). +/// +/// DETACHED: not executing Python bytecode (in native code, or idle) +/// ATTACHED: actively executing Python bytecode +/// SUSPENDED: parked by a stop-the-world request +/// SHUTTING_DOWN: interpreter is finalizing; the OS thread must hang +/// (`_PyThreadState_HangThread`) and must not look done to `_ThreadHandle`. #[cfg(feature = "threading")] -pub const THREAD_DETACHED: i32 = 0; +#[repr(i32)] +#[derive(Copy, Clone, Debug, PartialEq, Eq)] +pub enum ThreadState { + Detached = 0, + Attached = 1, + Suspended = 2, + ShuttingDown = 3, +} + #[cfg(feature = "threading")] -pub const THREAD_ATTACHED: i32 = 1; -#[cfg(feature = "threading")] -pub const THREAD_SUSPENDED: i32 = 2; -#[cfg(feature = "threading")] -pub const THREAD_SHUTTING_DOWN: i32 = 3; +impl ThreadState { + #[must_use] + pub const fn from_i32(v: i32) -> Option { + match v { + 0 => Some(Self::Detached), + 1 => Some(Self::Attached), + 2 => Some(Self::Suspended), + 3 => Some(Self::ShuttingDown), + _ => None, + } + } +} /// Per-thread shared state for sys._current_frames() and sys._current_exceptions(). /// The exception field uses atomic operations for lock-free cross-thread reads. @@ -62,6 +82,12 @@ pub struct ThreadSlot { #[cfg(not(unix))] pub frames: parking_lot::Mutex>, pub exception: crate::PyAtomicRef>, + /// `tstate->c_traceobj`. Cross-thread source of truth for sys.gettrace / + /// sys._settraceallthreads. + pub trace_func: PyMutex, + /// `tstate->c_profileobj`. Cross-thread source of truth for sys.getprofile / + /// sys._setprofileallthreads. + pub profile_func: PyMutex, /// Thread state for stop-the-world: DETACHED / ATTACHED / SUSPENDED / SHUTTING_DOWN pub state: core::sync::atomic::AtomicI32, /// Per-thread stop request bit (eval breaker equivalent). @@ -416,13 +442,27 @@ fn ensure_thread_slot(vm: &VirtualMachine) -> CurrentFrameSlot { #[cfg(not(unix))] frames: parking_lot::Mutex::new(Vec::new()), exception: crate::PyAtomicRef::from(None::), + trace_func: PyMutex::new( + vm.state + .global_trace_func + .lock() + .clone() + .unwrap_or_else(|| vm.ctx.none()), + ), + profile_func: PyMutex::new( + vm.state + .global_profile_func + .lock() + .clone() + .unwrap_or_else(|| vm.ctx.none()), + ), state: core::sync::atomic::AtomicI32::new( if vm.state.stop_the_world.requested.load(Ordering::Acquire) { // Match init_threadstate(): new thread-state starts // suspended while stop-the-world is active. - THREAD_SUSPENDED + ThreadState::Suspended as i32 } else { - THREAD_DETACHED + ThreadState::Detached as i32 }, ), stop_requested: core::sync::atomic::AtomicBool::new(false), @@ -436,6 +476,13 @@ fn ensure_thread_slot(vm: &VirtualMachine) -> CurrentFrameSlot { }) } +/// The current thread's `ThreadSlot` for the entered interpreter, if any. +#[cfg(feature = "threading")] +#[must_use] +pub fn current_thread_slot() -> Option { + CURRENT_THREAD_SLOT.with(|slot| slot.borrow().clone()) +} + /// Make `slot` the current thread slot (and the cached top-frame pointer). #[cfg(feature = "threading")] fn set_current_thread_slot(slot: CurrentFrameSlot) { @@ -456,7 +503,7 @@ fn current_slot_is_attached() -> bool { CURRENT_THREAD_SLOT.with(|slot| { slot.borrow() .as_ref() - .is_some_and(|s| s.state.load(Ordering::Acquire) == THREAD_ATTACHED) + .is_some_and(|s| s.state.load(Ordering::Acquire) == ThreadState::Attached as i32) }) } @@ -515,7 +562,7 @@ fn end_interpreter_section(switched: bool) { #[cfg(feature = "threading")] fn wait_while_suspended(slot: &ThreadSlot) -> u64 { let mut wait_yields = 0u64; - while slot.state.load(Ordering::Acquire) == THREAD_SUSPENDED { + while slot.state.load(Ordering::Acquire) == ThreadState::Suspended as i32 { wait_yields = wait_yields.saturating_add(1); std::thread::park(); } @@ -539,8 +586,10 @@ fn hang_thread() -> ! { pub fn hang_current_thread(state: &PyGlobalState) -> ! { CURRENT_THREAD_SLOT.with(|slot| { if let Some(s) = slot.borrow().as_ref() { - let prev = s.state.swap(THREAD_SHUTTING_DOWN, Ordering::AcqRel); - if prev == THREAD_ATTACHED { + let prev = s + .state + .swap(ThreadState::ShuttingDown as i32, Ordering::AcqRel); + if prev == ThreadState::Attached as i32 { crate::object::qsbr::QSBR.offline(&s.qsbr); } s.stop_requested.store(false, Ordering::Release); @@ -570,7 +619,8 @@ pub fn set_other_threads_shutting_down(state: &PyGlobalState) { continue; } slot.stop_requested.store(false, Ordering::Release); - slot.state.store(THREAD_SHUTTING_DOWN, Ordering::Release); + slot.state + .store(ThreadState::ShuttingDown as i32, Ordering::Release); slot.thread.unpark(); } } @@ -582,8 +632,8 @@ fn attach_thread(vm: &VirtualMachine) { super::stw_trace(format_args!("attach begin")); loop { match s.state.compare_exchange( - THREAD_DETACHED, - THREAD_ATTACHED, + ThreadState::Detached as i32, + ThreadState::Attached as i32, Ordering::AcqRel, Ordering::Relaxed, ) { @@ -592,21 +642,23 @@ fn attach_thread(vm: &VirtualMachine) { super::stw_trace(format_args!("attach DETACHED->ATTACHED")); break; } - Err(THREAD_SUSPENDED) => { - // Parked by stop-the-world — wait until released to DETACHED - super::stw_trace(format_args!("attach wait-suspended")); - let wait_yields = wait_while_suspended(s); - vm.state.stop_the_world.add_attach_wait_yields(wait_yields); - // Retry CAS - } - Err(THREAD_SHUTTING_DOWN) => { - super::stw_trace(format_args!("attach hang shutting-down")); - hang_thread(); - } - Err(state) => { - debug_assert!(false, "unexpected thread state in attach: {state}"); - break; - } + Err(state) => match ThreadState::from_i32(state) { + Some(ThreadState::Suspended) => { + // Parked by stop-the-world — wait until released to DETACHED + super::stw_trace(format_args!("attach wait-suspended")); + let wait_yields = wait_while_suspended(s); + vm.state.stop_the_world.add_attach_wait_yields(wait_yields); + // Retry CAS + } + Some(ThreadState::ShuttingDown) => { + super::stw_trace(format_args!("attach hang shutting-down")); + hang_thread(); + } + _ => { + debug_assert!(false, "unexpected thread state in attach: {state}"); + break; + } + }, } } } @@ -628,20 +680,19 @@ fn detach_thread() { CURRENT_THREAD_SLOT.with(|slot| { if let Some(s) = slot.borrow().as_ref() { match s.state.compare_exchange( - THREAD_ATTACHED, - THREAD_DETACHED, + ThreadState::Attached as i32, + ThreadState::Detached as i32, Ordering::AcqRel, Ordering::Acquire, ) { Ok(_) => { crate::object::qsbr::QSBR.offline(&s.qsbr); } - Err(THREAD_DETACHED) => { - debug_assert!(false, "detach called while already DETACHED"); - return; - } Err(state) => { - debug_assert!(false, "unexpected thread state in detach: {state}"); + debug_assert!( + matches!(ThreadState::from_i32(state), Some(ThreadState::Detached)), + "unexpected thread state in detach: {state}" + ); return; } } @@ -663,7 +714,7 @@ pub fn allow_threads(vm: &VirtualMachine, f: impl FnOnce() -> R) -> R { let should_transition = CURRENT_THREAD_SLOT.with(|slot| { slot.borrow() .as_ref() - .is_some_and(|s| s.state.load(Ordering::Acquire) == THREAD_ATTACHED) + .is_some_and(|s| s.state.load(Ordering::Acquire) == ThreadState::Attached as i32) }); if !should_transition { return f(); @@ -700,7 +751,7 @@ pub fn attach_for_callback(vm: &VirtualMachine, f: impl FnOnce() -> R) -> R { let should_transition = CURRENT_THREAD_SLOT.with(|slot| { slot.borrow() .as_ref() - .is_some_and(|s| s.state.load(Ordering::Acquire) != THREAD_ATTACHED) + .is_some_and(|s| s.state.load(Ordering::Acquire) != ThreadState::Attached as i32) }); if !should_transition { return f(); @@ -795,8 +846,8 @@ fn do_suspend(state: &PyGlobalState) { let _registry = state.thread_frames.lock(); if stw.requested.load(Ordering::Acquire) { Some(s.state.compare_exchange( - THREAD_ATTACHED, - THREAD_SUSPENDED, + ThreadState::Attached as i32, + ThreadState::Suspended as i32, Ordering::AcqRel, Ordering::Acquire, )) @@ -816,26 +867,28 @@ fn do_suspend(state: &PyGlobalState) { // Consumed this thread's stop request bit. s.stop_requested.store(false, Ordering::Release); } - Some(Err(THREAD_DETACHED)) => { - // Leaving VM; caller will re-check on next entry. - super::stw_trace(format_args!("suspend skip DETACHED")); - return; - } - Some(Err(THREAD_SUSPENDED)) => { - // Already parked by another path. - s.stop_requested.store(false, Ordering::Release); - super::stw_trace(format_args!("suspend skip already-suspended")); - return; - } - Some(Err(THREAD_SHUTTING_DOWN)) => { - s.stop_requested.store(false, Ordering::Release); - super::stw_trace(format_args!("suspend hang shutting-down")); - hang_thread(); - } - Some(Err(state)) => { - debug_assert!(false, "unexpected thread state in suspend: {state}"); - return; - } + Some(Err(state)) => match ThreadState::from_i32(state) { + Some(ThreadState::Detached) => { + // Leaving VM; caller will re-check on next entry. + super::stw_trace(format_args!("suspend skip DETACHED")); + return; + } + Some(ThreadState::Suspended) => { + // Already parked by another path. + s.stop_requested.store(false, Ordering::Release); + super::stw_trace(format_args!("suspend skip already-suspended")); + return; + } + Some(ThreadState::ShuttingDown) => { + s.stop_requested.store(false, Ordering::Release); + super::stw_trace(format_args!("suspend hang shutting-down")); + hang_thread(); + } + _ => { + debug_assert!(false, "unexpected thread state in suspend: {state}"); + return; + } + }, } super::stw_trace(format_args!("suspend ATTACHED->SUSPENDED")); @@ -853,25 +906,27 @@ fn do_suspend(state: &PyGlobalState) { // Re-attach (DETACHED → ATTACHED), tstate_wait_attach CAS loop. loop { match s.state.compare_exchange( - THREAD_DETACHED, - THREAD_ATTACHED, + ThreadState::Detached as i32, + ThreadState::Attached as i32, Ordering::AcqRel, Ordering::Acquire, ) { Ok(_) => break, - Err(THREAD_SUSPENDED) => { - let extra_wait = wait_while_suspended(s); - stw.add_suspend_wait_yields(extra_wait); - } - Err(THREAD_ATTACHED) => break, - Err(THREAD_SHUTTING_DOWN) => { - super::stw_trace(format_args!("suspend resume hang shutting-down")); - hang_thread(); - } - Err(state) => { - debug_assert!(false, "unexpected post-suspend state: {state}"); - break; - } + Err(state) => match ThreadState::from_i32(state) { + Some(ThreadState::Suspended) => { + let extra_wait = wait_while_suspended(s); + stw.add_suspend_wait_yields(extra_wait); + } + Some(ThreadState::Attached) => break, + Some(ThreadState::ShuttingDown) => { + super::stw_trace(format_args!("suspend resume hang shutting-down")); + hang_thread(); + } + _ => { + debug_assert!(false, "unexpected post-suspend state: {state}"); + break; + } + }, } } s.stop_requested.store(false, Ordering::Release); @@ -941,7 +996,7 @@ pub(crate) fn debug_assert_current_thread_attached() { if let Some(s) = slot.borrow().as_ref() { debug_assert_eq!( s.state.load(Ordering::Relaxed), - THREAD_ATTACHED, + ThreadState::Attached as i32, "type cache read while thread not ATTACHED" ); } @@ -1075,8 +1130,8 @@ pub fn cleanup_current_thread_frames(vm: &VirtualMachine) { // thread-state slot is being removed. if let Some(slot) = &slot_to_clean { let _ = slot.state.compare_exchange( - THREAD_ATTACHED, - THREAD_DETACHED, + ThreadState::Attached as i32, + ThreadState::Detached as i32, Ordering::AcqRel, Ordering::Acquire, ); @@ -1097,7 +1152,7 @@ pub fn cleanup_current_thread_frames(vm: &VirtualMachine) { if let Some(slot) = &_removed && vm.state.stop_the_world.requested.load(Ordering::Acquire) && thread_id != vm.state.stop_the_world.requester_ident() - && slot.state.load(Ordering::Relaxed) != THREAD_SUSPENDED + && slot.state.load(Ordering::Relaxed) != ThreadState::Suspended as i32 { // A non-requester thread disappeared while stop-the-world is pending. // Unblock requester countdown progress. @@ -1175,7 +1230,9 @@ pub fn reinit_frame_slot_after_fork(vm: &VirtualMachine) { #[cfg(not(unix))] frames: parking_lot::Mutex::new(current_frames), exception: crate::PyAtomicRef::from(vm.topmost_exception()), - state: core::sync::atomic::AtomicI32::new(THREAD_ATTACHED), + trace_func: PyMutex::new(vm.trace_func.borrow().clone()), + profile_func: PyMutex::new(vm.profile_func.borrow().clone()), + state: core::sync::atomic::AtomicI32::new(ThreadState::Attached as i32), stop_requested: core::sync::atomic::AtomicBool::new(false), thread: std::thread::current(), qsbr: crate::object::qsbr::QSBR.register(), @@ -1412,7 +1469,7 @@ impl VirtualMachine { profile_func: RefCell::new(global_profile.unwrap_or_else(|| self.ctx.none())), trace_func: RefCell::new(global_trace.unwrap_or_else(|| self.ctx.none())), use_tracing: Cell::new(use_tracing), - what_event: Cell::new(-1), + what_event: Cell::new(None), tracing_depth: Cell::new(0), recursion_limit: self.recursion_limit.clone(), signal_handlers: core::cell::OnceCell::new(), diff --git a/crates/vm/src/warn.rs b/crates/vm/src/warn.rs index f7336bbd8..83c940427 100644 --- a/crates/vm/src/warn.rs +++ b/crates/vm/src/warn.rs @@ -1,8 +1,9 @@ use crate::{ - AsObject, Context, Py, PyObject, PyObjectRef, PyResult, VirtualMachine, + AsObject, Context, Py, PyAtomicRef, PyObject, PyObjectRef, PyResult, VirtualMachine, + anystr::SplitLinesArgs, builtins::{ - PyBaseExceptionRef, PyDictRef, PyListRef, PyStr, PyStrInterned, PyStrRef, PyTuple, - PyTupleRef, PyType, PyTypeRef, + PyBaseExceptionRef, PyDict, PyDictRef, PyList, PyListRef, PyStr, PyStrInterned, PyStrRef, + PyTuple, PyTupleRef, PyType, PyTypeRef, }, convert::TryFromObject, }; @@ -10,9 +11,9 @@ use core::sync::atomic::{AtomicUsize, Ordering}; use rustpython_common::lock::OnceCell; pub struct WarningsState { - pub filters: PyListRef, - pub once_registry: PyDictRef, - pub default_action: PyStrRef, + pub filters: PyAtomicRef, + pub once_registry: PyAtomicRef, + pub default_action: PyAtomicRef, pub filters_version: AtomicUsize, pub context_var: OnceCell, lock_count: AtomicUsize, @@ -70,9 +71,9 @@ impl WarningsState { pub fn init_state(ctx: &Context) -> Self { Self { - filters: Self::create_default_filters(ctx), - once_registry: ctx.new_dict(), - default_action: ctx.new_str("default"), + filters: Self::create_default_filters(ctx).into(), + once_registry: ctx.new_dict().into(), + default_action: ctx.new_str("default").into(), filters_version: AtomicUsize::new(0), context_var: OnceCell::new(), lock_count: AtomicUsize::new(0), @@ -114,7 +115,7 @@ fn get_warnings_attr( vm: &VirtualMachine, attr_name: &'static PyStrInterned, try_import: bool, -) -> Option { +) -> PyResult> { let module = if try_import && !vm .state @@ -123,60 +124,152 @@ fn get_warnings_attr( { match vm.import("warnings", 0) { Ok(module) => module, - Err(_) => return None, + Err(e) if e.fast_isinstance(vm.ctx.exceptions.import_error) => return Ok(None), + Err(e) => return Err(e), } } else { - match vm.sys_module.get_attr(identifier!(vm, modules), vm) { - Ok(modules) => match modules.get_item(vm.ctx.intern_str("warnings"), vm) { - Ok(module) => module, - Err(_) => return None, - }, - Err(_) => return None, + let Ok(modules) = vm.sys_module.get_attr(identifier!(vm, modules), vm) else { + return Ok(None); + }; + match modules.get_item(vm.ctx.intern_str("warnings"), vm) { + Ok(module) => module, + Err(_) => return Ok(None), } }; - module.get_attr(attr_name, vm).ok() + vm.get_attribute_opt(module, attr_name) } /// Get the warnings filters list from `sys.modules['warnings'].filters`, -/// falling back to vm.state.warnings.filters. -fn get_warnings_filters(vm: &VirtualMachine) -> PyListRef { - if let Some(filters_obj) = get_warnings_attr(vm, identifier!(&vm.ctx, filters), false) - && let Ok(filters) = filters_obj.try_into_value::(vm) - { - return filters; +/// falling back to the interpreter cache (`st->filters`). +fn get_warnings_filters(vm: &VirtualMachine) -> PyResult { + if let Some(filters_obj) = get_warnings_attr(vm, identifier!(vm, filters), false)? { + let filters = PyListRef::try_from_object(vm, filters_obj) + .map_err(|_| vm.new_value_error("_warnings.filters must be a list"))?; + vm.state + .warnings + .filters + .swap_to_temporary_refs(filters, vm); } - vm.state.warnings.filters.clone() + Ok(vm.state.warnings.filters.to_owned()) } -/// Get the default action from `sys.modules['warnings']._defaultaction`, +fn get_warnings_context_filters(vm: &VirtualMachine) -> PyResult> { + let Some(ctx_var) = vm.state.warnings.context_var.get() else { + return Ok(None); + }; + if vm.is_none(ctx_var) { + return Ok(None); + } + let ctx = vm.call_method(ctx_var, "get", (vm.ctx.none(),))?; + if vm.is_none(&ctx) { + return Ok(None); + } + let filters = ctx.get_attr("_filters", vm)?; + PyListRef::try_from_object(vm, filters) + .map(Some) + .map_err(|_| vm.new_value_error("_filters of warnings._warnings_context must be a list")) +} + +fn bless_my_loader( + module_globals: &PyObject, + vm: &VirtualMachine, +) -> PyResult> { + let external = vm + .importlib + .get_attr("_bootstrap_external", vm) + .or_else(|_| vm.import("importlib._bootstrap_external", 0))?; + if vm.is_none(&external) { + return Ok(None); + } + let bless = external.get_attr("_bless_my_loader", vm)?; + let loader = bless.call((module_globals.to_owned(),), vm)?; + if vm.is_none(&loader) { + Ok(None) + } else { + Ok(Some(loader)) + } +} + +pub(crate) fn get_source_line( + module_globals: &PyObject, + lineno: usize, + vm: &VirtualMachine, +) -> PyResult> { + let Some(loader) = bless_my_loader(module_globals, vm)? else { + return Ok(None); + }; + + let module_name = if let Some(dict) = module_globals.downcast_ref::() { + match dict.get_item_opt(identifier!(vm, __name__), vm)? { + Some(name) => name, + None => return Ok(None), + } + } else { + match module_globals.get_item(identifier!(vm, __name__), vm) { + Ok(name) => name, + Err(_) => return Ok(None), + } + }; + + let Some(get_source) = vm.get_attribute_opt(loader, vm.ctx.intern_str("get_source"))? else { + return Ok(None); + }; + let source = get_source.call((module_name,), vm)?; + if vm.is_none(&source) { + return Ok(None); + } + + let Some(source_str) = source.downcast_ref::() else { + return Err(vm.new_type_error(format!("expected str, not {}", source.class().name()))); + }; + let lines = source_str.splitlines(SplitLinesArgs { keepends: false }, vm); + if lineno == 0 { + return Err(vm.new_index_error("list index out of range")); + } + lines + .get(lineno - 1) + .cloned() + .ok_or_else(|| vm.new_index_error("list index out of range")) + .map(Some) +} + +/// Get the default action from `sys.modules['warnings'].defaultaction`, /// falling back to vm.state.warnings.default_action. fn get_default_action(vm: &VirtualMachine) -> PyResult { - if let Some(action) = get_warnings_attr(vm, identifier!(&vm.ctx, defaultaction), false) { + if let Some(action) = get_warnings_attr(vm, identifier!(vm, defaultaction), false)? { if !action.class().is(vm.ctx.types.str_type) { return Err(vm.new_type_error(format!( "_warnings.defaultaction must be a string, not '{}'", action.class().name() ))); } - return Ok(action); + let action = PyStrRef::try_from_object(vm, action)?; + vm.state + .warnings + .default_action + .swap_to_temporary_refs(action, vm); } - Ok(vm.state.warnings.default_action.clone().into()) + Ok(vm.state.warnings.default_action.to_owned().into()) } -/// Get the once registry from `sys.modules['warnings']._onceregistry`, +/// Get the once registry from `sys.modules['warnings'].onceregistry`, /// falling back to vm.state.warnings.once_registry. fn get_once_registry(vm: &VirtualMachine) -> PyResult { - if let Some(registry) = get_warnings_attr(vm, identifier!(&vm.ctx, onceregistry), false) { + if let Some(registry) = get_warnings_attr(vm, identifier!(vm, onceregistry), false)? { if !registry.class().is(vm.ctx.types.dict_type) { return Err(vm.new_type_error(format!( "_warnings.onceregistry must be a dict, not '{}'", registry.class().name() ))); } - return Ok(registry); + let registry = PyDictRef::try_from_object(vm, registry)?; + vm.state + .warnings + .once_registry + .swap_to_temporary_refs(registry, vm); } - Ok(vm.state.warnings.once_registry.clone().into()) + Ok(vm.state.warnings.once_registry.to_owned().into()) } fn already_warned( @@ -253,20 +346,18 @@ fn normalize_module(filename: &Py, vm: &VirtualMachine) -> PyObjectRef { } } -/// Search the global filters list for a matching action. -// TODO: split into filter_search() + get_filter() and support -// context-aware filters (get_warnings_context_filters). -fn get_filter( +/// Search a filters list for a matching action. +fn filter_search( category: &PyObject, text: &PyObject, lineno: usize, module: &PyObject, + filters: &PyListRef, + list_name: &str, vm: &VirtualMachine, -) -> PyResult { - let filters = get_warnings_filters(vm); - +) -> PyResult> { // filters could change while we are iterating over it. - // Re-check list length each iteration (matches C behavior). + // Re-check list length each iteration. let mut i = 0; while i < filters.borrow_vec().len() { let Some(tmp_item) = filters.borrow_vec().get(i).cloned() else { @@ -276,22 +367,56 @@ fn get_filter( .ok() .filter(|t| t.len() == 5) .ok_or_else(|| { - vm.new_value_error(format!("_warnings.filters item {i} isn't a 5-tuple")) + vm.new_value_error(format!("warnings.{list_name} item {i} isn't a 5-tuple")) })?; /* action, msg, cat, mod, ln = item */ let action = &tmp_item[0]; + if !action.class().is(vm.ctx.types.str_type) { + return Err(vm.new_type_error(format!( + "action must be a string, not '{}'", + action.class().name() + ))); + } let good_msg = check_matched(&tmp_item[1], text, vm)?; let is_subclass = category.is_subclass(&tmp_item[2], vm)?; let good_mod = check_matched(&tmp_item[3], module, vm)?; let ln: usize = tmp_item[4].try_int(vm).map_or(0, |v| v.as_u32_mask() as _); if good_msg && is_subclass && good_mod && (ln == 0 || lineno == ln) { - return Ok(action.to_owned()); + return Ok(Some(action.to_owned())); } i += 1; } + Ok(None) +} +fn get_filter( + category: &PyObject, + text: &PyObject, + lineno: usize, + module: &PyObject, + vm: &VirtualMachine, +) -> PyResult { + if let Some(context_filters) = get_warnings_context_filters(vm)? { + if let Some(action) = filter_search( + category, + text, + lineno, + module, + &context_filters, + "_warnings_context _filters", + vm, + )? { + return Ok(action); + } + return get_default_action(vm); + } + + let filters = get_warnings_filters(vm)?; + if let Some(action) = filter_search(category, text, lineno, module, &filters, "filters", vm)? { + return Ok(action); + } get_default_action(vm) } @@ -320,6 +445,8 @@ pub fn warn_unawaited_coroutine(coro: &PyObject, qualname: &Py, vm: &Virt let mut warned = false; if let Some(helper) = get_warnings_attr(vm, vm.ctx.intern_str("_warn_unawaited_coroutine"), true) + .ok() + .flatten() { match helper.call((coro.to_owned(),), vm) { Ok(_) => warned = true, @@ -491,7 +618,7 @@ fn call_show_warning( source: Option, vm: &VirtualMachine, ) -> PyResult<()> { - let Some(show_fn) = get_warnings_attr(vm, identifier!(&vm.ctx, _showwarnmsg), source.is_some()) + let Some(show_fn) = get_warnings_attr(vm, identifier!(vm, _showwarnmsg), source.is_some())? else { show_warning(&filename, lineno, &text, &category, source_line, vm); return Ok(()); @@ -501,8 +628,7 @@ fn call_show_warning( return Err(vm.new_type_error("warnings._showwarnmsg() must be set to a callable")); } - let Some(warnmsg_cls) = get_warnings_attr(vm, identifier!(&vm.ctx, WarningMessage), false) - else { + let Some(warnmsg_cls) = get_warnings_attr(vm, identifier!(vm, WarningMessage), false)? else { return Err(vm.new_runtime_error("unable to get warnings.WarningMessage")); }; @@ -528,7 +654,7 @@ fn show_warning( lineno: usize, text: &Py, category: &Py, - _source_line: Option, + source_line: Option, vm: &VirtualMachine, ) { let stderr = crate::stdlib::sys::PyStderr(vm); @@ -540,6 +666,11 @@ fn show_warning( category.name(), text ); + if let Some(source_line) = source_line + && let Ok(line) = source_line.str(vm) + { + writeln!(stderr, "{line}"); + } } /// Check if a frame's filename starts with any of the given prefixes. diff --git a/src/shell.rs b/src/shell.rs index 8e6d6a069..3ef093716 100644 --- a/src/shell.rs +++ b/src/shell.rs @@ -44,7 +44,6 @@ fn shell_exec( *future_features |= code.code.flags & CodeFlags::FUTURE_MASK; let _ = vm.register_code_in_linecache(&code, source); if empty_line_given || !continuing_block { - // We want to execute the full code match vm.run_code_obj(code, scope) { Ok(_val) => ShellExecResult::Ok, Err(err) => ShellExecResult::PyErr(err), @@ -54,20 +53,21 @@ fn shell_exec( ShellExecResult::Ok } } - Err(VmCompileError::Compile(CompileError::Parse(ParseError { - error: ParseErrorType::Lexical(LexicalErrorType::Eof), - .. - }))) => ShellExecResult::ContinueLine, - Err(VmCompileError::Compile(CompileError::Parse(ParseError { - error: - ParseErrorType::Lexical(LexicalErrorType::FStringError( - InterpolatedStringErrorType::UnterminatedTripleQuotedString, - )), - .. - }))) => ShellExecResult::ContinueLine, Err(err) => { - // Check if the error is from an unclosed triple quoted string (which should always - // continue) + if matches!( + &err, + VmCompileError::Compile(CompileError::Parse(ParseError { + error: ParseErrorType::Lexical( + LexicalErrorType::Eof + | LexicalErrorType::FStringError( + InterpolatedStringErrorType::UnterminatedTripleQuotedString, + ) + ), + .. + })) + ) { + return ShellExecResult::ContinueLine; + } if let VmCompileError::Compile(CompileError::Parse(ParseError { error: ParseErrorType::Lexical(LexicalErrorType::UnclosedStringError), raw_location, @@ -82,33 +82,14 @@ fn shell_exec( { return ShellExecResult::ContinueLine; } - }; + } - // bad_error == true if we are handling an error that should be thrown even if we are continuing - // if its an indentation error, set to true if we are continuing and the error is on column 0, - // since indentations errors on columns other than 0 should be ignored. - // if its an unrecognized token for dedent, set to false - - let bad_error = match &err { - VmCompileError::Compile(CompileError::Parse(p)) => { - match &p.error { - ParseErrorType::Lexical(LexicalErrorType::IndentationError) => { - continuing_block - } // && p.location.is_some() - ParseErrorType::OtherError(msg) => { - !msg.starts_with("Expected an indented block") - } - _ => true, // !matches!(p, ParseErrorType::UnrecognizedToken(Tok::Dedent, _)) - } - } - _ => true, // It is a bad error for everything else - }; - - // If we are handling an error on an empty line or an error worthy of throwing - if empty_line_given || bad_error { - ShellExecResult::PyErr(err.into_pyexception(vm, Some(source))) - } else { + // An unfinished suite is _IncompleteInputError, not IndentationError. + let exc = err.into_pyexception_maybe_incomplete(vm, Some(source), !empty_line_given); + if !empty_line_given && exc.fast_isinstance(vm.ctx.exceptions.incomplete_input_error) { ShellExecResult::ContinueBlock + } else { + ShellExecResult::PyErr(exc) } } }