mirrors/RustPython
0
0 Fork 0
mirror of https://github.com/RustPython/RustPython.git synced 2026-10-28 10:44:36 +08:00
Code Activity

Add OrderedDict, marshal, inspect, and chflags bitflagset (#8780)

Browse Source 
* Implement _collections.OrderedDict

Extracted from #6799: native OrderedDict as a dict subclass,
iterator mutation tracking, and mapping-view set operations.

* Add OrderedDict, warnings source lines, and socket sendmsg

Implement _collections.OrderedDict as a dict subclass with a
linked list of insertion-order nodes.

Pack struct F/D as native float/double complex pairs, and report
unexpected __replace__ field names as a list repr.

Register encodings.latin_1 at startup. Reject NUL in input()
prompts. Copy non-exact type dicts in mapping order.

Fetch warning source lines through _bless_my_loader and unicode
splitlines. Keep the last warnings.filters list after deletion.

Snapshot sendmsg sequences before converting buffers, parse
ancillary ints with __index__, and dispatch AF_CAN by protocol.

Store sys.trace and sys.profile on each ThreadSlot so
_settraceallthreads/_setprofileallthreads are visible to
sys.gettrace/getprofile on other threads.

Use <frozen importlib._bootstrap> as frozen importlib
co_filename. Keep HEAPTYPE socket objects GC-tracked.

Assisted-by: Grok:grok-4.6

* Load marshal via readinto and dump sets in dumps order

marshal.load fills a scratch buffer with readinto on a
contiguous writable memoryview, and rejects a returned length
other than the requested size.

set and frozenset dumps write elements in sorted(v,
key=marshal.dumps) order.

Assisted-by: Grok:grok-4.6

* Add classmethod_descriptor and ABC IS_ABSTRACT flags

Give METH_CLASS methods a distinct classmethod_descriptor type.
Store Py_TPFLAGS_IS_ABSTRACT from __abstractmethods__ and look
up that attribute only in the type's own dict.

Expose defaultdict.default_factory as a member descriptor.
Inherit bool's unused number slots from int. Mark set and
frozenset __contains__ as METH_COEXIST.

Parse classmethod, staticmethod, object.__new__, and heap type
text signatures. Register interactive source in linecache.

Assisted-by: Grok:grok-4.6

* address review: odict reuse, descr call, new wrapper

Reuse odict node slots and walk the live list in next_after.
Require both | operands to be dicts. Drop the odict iterator
on GC clear. Bind classmethod descriptors through descr_get
and update IS_ABSTRACT under the type lock. Give odict keys
and items sq_contains. Walk tp_new_wrapper past new_wrapper
so HEAPTYPE socket subclasses can call socket.__new__.
Split FreeBSD Bluetooth protocol constants from Linux values.

Assisted-by: Grok:grok-4.6

* Represent chflags bits with bitflagset

Assisted-by: Grok:grok-4.6

* Represent access and mode bits with bitflagset

Assisted-by: Grok:grok-4.6

* Continue unfinished REPL suites as incomplete input

Assisted-by: Grok:grok-4.6

* Add pickle and os.stat inspect signatures

Give _pickle.Pickler the clinic constructor text signature and
accept fix_imports and buffer_callback as positional-or-keyword.
Allow #[pyfunction(text_signature)] and use it so os.stat reports
(path, *, dir_fd=None, follow_symlinks=True). Uncomment import
_pickle and drop the pickle inspect expectedFailures.

Assisted-by: Grok:grok-4.6

* Add Unpickler, dump/load, and lstat signatures

Give _pickle Unpickler, dump, dumps, load, and loads clinic
text signatures, and os.lstat (path, *, dir_fd=None).

Assisted-by: Grok:grok-4.6

* Complete OrderedDict with insertion-order links

Move the native OrderedDict into ordered_dict.rs and track order
with a linked list of nodes, keeping mapping-view set operations.

Assisted-by: Grok:grok-4.6

* address CI: stat casts, HCI_DATA_DIR, passing xfails

Compare stat permission bits as Mode so Linux clippy does not
flag a u32-to-u32 cast. Export HCI_DATA_DIR with the Bluetooth
socket option values. Drop expectedFailure from tests that now
pass.

Assisted-by: Grok:grok-4.6

* Fix wasm teardown deadlock and bluetooth constants

Assisted-by: Grok:grok-4.6

* Clear process-wide eval breaker bits in STW test

Assisted-by: Grok:grok-4.6

* Drop exclusive-held panic that tripped wasm clippy

Assisted-by: Grok:grok-4.6

* Fix OrderedDict contains and export Windows BDADDR

Assisted-by: Grok:grok-4.6

* Add cmp_to_key and hash-once LRU cache

_functools.cmp_to_key now returns KeyWrapper. LRU lookups store the
key hash on _lru_list_elem so __hash__ is not called again on hits
or eviction.

Assisted-by: Grok:grok-4.6

* Allow subclass keywords on list, tuple, and float

list.__init__ rejects keywords only when tp_new is still list's.
tuple/float.__new__ reject them only when tp_init is still the base's.

Assisted-by: Grok:grok-4.6

* Fix CI lint, test re-export, and structseq members

Drop the duplicate clear_eval_breaker_for_test re-export.
Format structseq imports. Expose struct sequence fields as
readonly members so assignment raises "readonly attribute".

Assisted-by: Grok:grok-4.6

* Allow subclass keywords on itertools and filter

Positional-only constructors ignore extra keywords when a subclass
has replaced tp_init, matching list/tuple/float.

Assisted-by: Grok:grok-4.6

* Remove text_signature derive parameter

Drop the string override on #[pyfunction] and #[pyclass]. Signatures
are generated from the Rust parameter list only. Restore inspect
expectedFailures that depended on the override.

Assisted-by: Grok:grok-4.6

* Rename set contains_method to __contains__

Keep a borrowed contains helper for slots and C-API. The coexist
pymethod is named __contains__.

Assisted-by: Grok:grok-4.6

* Use MonitoringEvent instead of EVENT_* aliases

Drop the unused MonitoringEvents bitflags and the EVENT_PY_START
style mask constants. Call sites use the enum and .mask().

Assisted-by: Grok:grok-4.6

* Store warnings filters as PyListRef

The list already serializes its elements. Drop the extra mutex around
the cached list pointer.

Assisted-by: Grok:grok-4.6

* Re-export clear_eval_breaker_for_test after rebase

Keep the STW test helper visible without bringing back stop_bit_set.

Assisted-by: Grok:grok-4.6

* Fix groupby reentrancy and islice arguments

Give _grouper its own tgtkey and take currvalue after the key
comparison. Accept __index__ on islice bounds, reject step 0, and
drain the source up to stop.

Assisted-by: Grok:grok-4.6

* SETREF warnings filters, onceregistry, and defaultaction

Keep the last list, dict, or str after the module attributes are
deleted by swapping the interpreter cache pointer. Do not store a
None fallback for _warnings_context.

Assisted-by: Grok:grok-4.6

* Iterate OrderedDict inner dict via next_entry

&PyDict is not an iterator after the borrowed-ref signature change.

Assisted-by: Grok:grok-4.6

* Remove expectedFailure from testBluetoothConstants

Linux now exports BTPROTO_HCI and the rest of the set.

Assisted-by: Grok:grok-4.6

* Borrow get_filter arguments as PyObject

Match the warn_explicit call site that passes borrowed
handles after the internal-helper borrow change.

Assisted-by: Grok:grok-4.6

* Fix bool slots and SemHandle safety docs

Leave unused bool number slots unset so inspect reports
__add__ on int. Document SemHandle::from_raw safety.

Assisted-by: Grok:grok-4.6

* Keep sendmsg errors and drop duplicate linecache

Replace sendmsg iterable TypeError only. Register REPL
source in linecache once per compiled input.

Assisted-by: Grok:grok-4.6

* Nest REPL parse or-patterns for clippy

Assisted-by: Grok:grok-4.6

* Mark pydoc os.stat signature test expectedFailure

Assisted-by: Grok:grok-4.6

---------

Co-authored-by: Lee Dogeon <dev.moreal@gmail.com>
This commit is contained in:
Jeong, YunWonandLee Dogeon authored and GitHub committed 2026-09-24 12:21:26 +09:00
1 parent 86c7fa20c4
commit afad92bf1b
88 files changed
+4034 -1307
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.cspell.dict/cpython.txt
+3
View File
@@ -153,6 +153,7 @@ mimalloc
miscompiles
mult
multibytecodec
mycmp
mystricmp
mystrnicmp
nameobj
@@ -187,6 +188,8 @@ numkwonly
numposorkw
nvars
obmalloc
odict
ODICT
opname
opnames
orelse
Cargo.lock
Generated
+2
View File
@@ -3867,6 +3867,7 @@ name = "rustpython-host_env"
version = "0.6.0"
dependencies = [
"bitflags 2.13.1",
"bitflagset",
"cc",
"der 0.8.2",
"dns-lookup",
@@ -4108,6 +4109,7 @@ version = "0.6.0"
dependencies = [
"ascii",
"bitflags 2.13.1",
"bitflagset",
"bstr",
"constant_time_eq 0.6.0",
"crossbeam-utils",
Lib/test/test_abc.py
Vendored
-1
View File
@@ -168,7 +168,6 @@ def test_factory(abc_ABCMeta, abc_get_cache_token):
msg = r"class C without an implementation for abstract methods 'method_one', 'method_two'"
self.assertRaisesRegex(TypeError, msg, C)
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: False is not true
def test_abstractmethod_integration(self):
for abstractthing in [abc.abstractmethod, abc.abstractproperty,
abc.abstractclassmethod,
Lib/test/test_builtin.py
Vendored
-9
View File
@@ -2711,43 +2711,35 @@ class PtyTests(unittest.TestCase):
else:
yield
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux
def test_input_tty(self):
# Test input() functionality when wired to a tty
self.check_input_tty("prompt", b"quux")
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux
def test_input_tty_non_ascii(self):
# Check stdin/stdout encoding is used when invoking PyOS_Readline()
self.check_input_tty("prompté", b"quux\xc3\xa9", "utf-8")
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux
def test_input_tty_non_ascii_unicode_errors(self):
# Check stdin/stdout error handler is used when invoking PyOS_Readline()
self.check_input_tty("prompté", b"quux\xe9", "ascii")
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux
def test_input_tty_null_in_prompt(self):
self.check_input_tty("prompt\0", b"",
expected='ValueError: input: prompt string cannot contain '
'null characters')
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux
def test_input_tty_nonencodable_prompt(self):
self.check_input_tty("prompté", b"quux", "ascii", stdout_errors='strict',
expected="UnicodeEncodeError: 'ascii' codec can't encode "
"character '\\xe9' in position 6: ordinal not in "
"range(128)")
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux
def test_input_tty_nondecodable_input(self):
self.check_input_tty("prompt", b"quux\xe9", "ascii", stdin_errors='strict',
expected="UnicodeDecodeError: 'ascii' codec can't decode "
"byte 0xe9 in position 4: ordinal not in "
"range(128)")
@unittest.skip("TODO: RUSTPYTHON; FAILURE, WORKER BUG")
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: got 0 lines in pipe but expected 2, child output was: quux
def test_input_no_stdout_fileno(self):
# Issue #24402: If stdin is the original terminal but stdout.fileno()
# fails, do not use the original stdout file descriptor
@@ -3036,7 +3028,6 @@ class TestType(unittest.TestCase):
with self.assertRaises(TypeError):
type('A', (B,), {'__slots__': '__weakref__'})
@unittest.skip("TODO: RUSTPYTHON; random failure")
def test_namespace_order(self):
# bpo-34320: namespace should preserve order
od = collections.OrderedDict([('a', 1), ('b', 2)])
Lib/test/test_dataclasses/__init__.py
Vendored
-1
View File
@@ -5000,7 +5000,6 @@ class TestAbstract(unittest.TestCase):
self.assertFalse(inspect.isabstract(Date))
self.assertGreater(Date(2020,12,25), Date(2020,8,31))
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_maintain_abc(self):
class A(abc.ABC):
@abc.abstractmethod
Lib/test/test_float.py
Vendored
-1
View File
@@ -255,7 +255,6 @@ class GeneralFloatCases(unittest.TestCase):
with self.assertRaisesRegex(TypeError, 'keyword argument'):
float(x='3.14')
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Unexpected keyword argument newarg
def test_keywords_in_subclass(self):
class subclass(float):
pass
Lib/test/test_functools.py
Vendored
+8 -33
View File
@@ -1238,6 +1238,10 @@ class TestCmpToKeyC(TestCmpToKey, unittest.TestCase):
if c_functools:
cmp_to_key = c_functools.cmp_to_key
@unittest.expectedFailure # TODO: RUSTPYTHON; inspect.signature reports (args, /) until FromArgs fields generate __text_signature__
def test_cmp_to_signature(self):
super().test_cmp_to_signature()
@support.cpython_only
def test_disallow_instantiation(self):
# Ensure that the type disallows instantiation (bpo-43916)
@@ -1245,38 +1249,6 @@ class TestCmpToKeyC(TestCmpToKey, unittest.TestCase):
self, type(c_functools.cmp_to_key(None))
)
@unittest.expectedFailure # TODO: RUSTPYTHON; + (mycmp)
def test_cmp_to_signature(self):
return super().test_cmp_to_signature()
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() got multiple values for argument 'mycmp'
def test_cmp_to_key_arguments(self):
return super().test_cmp_to_key_arguments()
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() got multiple values for argument 'mycmp'
def test_obj_field(self):
return super().test_obj_field()
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given
def test_bad_cmp(self):
return super().test_bad_cmp()
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given
def test_cmp_to_key(self):
return super().test_cmp_to_key()
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given
def test_hash(self):
return super().test_hash()
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given
def test_sort_int(self):
return super().test_sort_int()
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: cmp_to_key() takes 1 positional argument but 2 were given
def test_sort_int_str(self):
return super().test_sort_int_str()
class TestCmpToKeyPy(TestCmpToKey, unittest.TestCase):
cmp_to_key = staticmethod(py_functools.cmp_to_key)
@@ -1763,7 +1735,6 @@ class TestLRU:
f(0, **{})
self.assertEqual(f.cache_info().hits, 1)
@unittest.expectedFailure # TODO: RUSTPYTHON; Python lru_cache impl doesn't cache hash like C impl
def test_lru_hash_only_once(self):
# To protect against weird reentrancy bugs and to improve
# efficiency when faced with slow __hash__ methods, the
@@ -2266,6 +2237,10 @@ class TestLRUPy(TestLRU, unittest.TestCase):
def cached_staticmeth(x, y):
return 3 * x + y
@unittest.expectedFailure # TODO: RUSTPYTHON; Python lru_cache impl doesn't cache hash like C impl
def test_lru_hash_only_once(self):
return super().test_lru_hash_only_once()
@unittest.skipUnless(c_functools, 'requires the C _functools module')
class TestLRUC(TestLRU, unittest.TestCase):
Lib/test/test_inspect/test_inspect.py
Vendored
+5 -14
View File
@@ -14,7 +14,7 @@ import linecache
import os
import dis
from os.path import normcase
# import _pickle # TODO: RUSTPYTHON
import _pickle
import pickle
import shutil
import stat
@@ -123,7 +123,6 @@ class IsTestBase(unittest.TestCase):
else:
self.assertFalse(other(obj), 'not %s(%s)' % (other.__name__, exp))
@unittest.expectedFailure # TODO: RUSTPYTHON; First has 0, Second has 1: 'iskeyword'
def test__all__(self):
support.check__all__(self, inspect, not_exported=("modulesbyfile",), extra=("get_annotations",))
@@ -176,7 +175,7 @@ class custom_descriptor:
class TestPredicates(IsTestBase):
@unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: type object 'defaultdict' has no attribute 'default_factory'
@unittest.expectedFailure # TODO: RUSTPYTHON; datetime.timedelta.days is a property until _datetime members exist
def test_excluding_predicates(self):
global tb
self.istest(inspect.isbuiltin, 'sys.exit')
@@ -480,7 +479,6 @@ class TestPredicates(IsTestBase):
self.assertIn('a', members)
self.assertNotIn('b', members)
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: False is not true
def test_isabstract(self):
from abc import ABCMeta, abstractmethod
@@ -503,7 +501,6 @@ class TestPredicates(IsTestBase):
self.assertFalse(inspect.isabstract(int))
self.assertFalse(inspect.isabstract(5))
@unittest.expectedFailure # TODO: RUSTPYTHON; + [True, False]
def test_isabstract_during_init_subclass(self):
from abc import ABCMeta, abstractmethod
isabstract_checks = []
@@ -1366,7 +1363,7 @@ class TestClassesAndFunctions(unittest.TestCase):
spec = inspect.getfullargspec(test)
self.assertEqual(test.__annotations__, spec.annotations)
@unittest.expectedFailure # TODO: RUSTPYTHON; NameError: name '_pickle' is not defined. Did you mean: 'pickle'? Or did you forget to import '_pickle'?
@unittest.expectedFailure # TODO: RUSTPYTHON; os.stat signature is generated as (path, dir_fd, follow_symlinks, /)
@unittest.skipIf(MISSING_C_DOCSTRINGS,
"Signature information for builtins requires docstrings")
def test_getfullargspec_builtin_methods(self):
@@ -1531,7 +1528,6 @@ class TestClassesAndFunctions(unittest.TestCase):
self.assertIn(('md', 'method', A), attrs, 'missing method descriptor')
self.assertIn(('dd', 'data', A), attrs, 'missing data descriptor')
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ('to_bytes', 'method', <class 'int'>) not found in [('__abs__', 'method', <class 'bool'>), ('__add__', 'method', <class 'bool'>), ('__and__', 'method', <class 'bool'>), ('__bool__', 'method', <class 'bool'>), ('__ceil__', 'class method', <class 'int'>), ('__class__', 'data', <class 'object'>), ('__delattr__', 'class method', <class 'object'>), ('__dir__', 'class method', <class 'object'>), ('__divmod__', 'method', <class 'bool'>), ('__doc__', 'data', <class 'bool'>), ('__eq__', 'class method', <class 'int'>), ('__float__', 'method', <class 'bool'>), ('__floor__', 'class method', <class 'int'>), ('__floordiv__', 'method', <class 'bool'>), ('__format__', 'class method', <class 'bool'>), ('__ge__', 'class method', <class 'int'>), ('__getattribute__', 'class method', <class 'object'>), ('__getnewargs__', 'class method', <class 'int'>), ('__getstate__', 'class method', <class 'object'>), ('__gt__', 'class method', <class 'int'>), ('__hash__', 'method', <class 'int'>), ('__index__', 'method', <class 'bool'>), ('__init__', 'method', <class 'object'>), ('__init_subclass__', 'class method', <class 'object'>), ('__int__', 'method', <class 'bool'>), ('__invert__', 'method', <class 'bool'>), ('__le__', 'class method', <class 'int'>), ('__lshift__', 'method', <class 'bool'>), ('__lt__', 'class method', <class 'int'>), ('__mod__', 'method', <class 'bool'>), ('__mul__', 'method', <class 'bool'>), ('__ne__', 'class method', <class 'int'>), ('__neg__', 'method', <class 'bool'>), ('__new__', 'static method', <class 'bool'>), ('__or__', 'method', <class 'bool'>), ('__pos__', 'method', <class 'bool'>), ('__pow__', 'method', <class 'bool'>), ('__radd__', 'method', <class 'bool'>), ('__rand__', 'method', <class 'bool'>), ('__rdivmod__', 'method', <class 'bool'>), ('__reduce__', 'class method', <class 'object'>), ('__reduce_ex__', 'class method', <class 'object'>), ('__repr__', 'method', <class 'bool'>), ('__rfloordiv__', 'method', <class 'bool'>), ('__rlshift__', 'method', <class 'bool'>), ('__rmod__', 'method', <class 'bool'>), ('__rmul__', 'method', <class 'bool'>), ('__ror__', 'method', <class 'bool'>), ('__round__', 'class method', <class 'int'>), ('__rpow__', 'method', <class 'bool'>), ('__rrshift__', 'method', <class 'bool'>), ('__rshift__', 'method', <class 'bool'>), ('__rsub__', 'method', <class 'bool'>), ('__rtruediv__', 'method', <class 'bool'>), ('__rxor__', 'method', <class 'bool'>), ('__setattr__', 'class method', <class 'object'>), ('__sizeof__', 'class method', <class 'int'>), ('__str__', 'method', <class 'object'>), ('__sub__', 'method', <class 'bool'>), ('__subclasshook__', 'class method', <class 'object'>), ('__truediv__', 'method', <class 'bool'>), ('__trunc__', 'class method', <class 'int'>), ('__xor__', 'method', <class 'bool'>), ('as_integer_ratio', 'class method', <class 'int'>), ('bit_count', 'class method', <class 'int'>), ('bit_length', 'class method', <class 'int'>), ('conjugate', 'class method', <class 'int'>), ('denominator', 'data', <class 'int'>), ('from_bytes', 'class method', <class 'int'>), ('imag', 'data', <class 'int'>), ('is_integer', 'class method', <class 'int'>), ('numerator', 'data', <class 'int'>), ('real', 'data', <class 'int'>), ('to_bytes', 'class method', <class 'int'>)] : missing plain method
def test_classify_builtin_types(self):
# Simple sanity check that all built-in types can have their
# attributes classified.
@@ -4210,7 +4206,6 @@ class TestSignatureObject(unittest.TestCase):
self.assertEqual(self.signature(C.__call__, follow_wrapped=False),
varargs_signature)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_signature_on_class_with_wrapped_init(self):
class C:
@identity_wrapper
@@ -4460,7 +4455,6 @@ class TestSignatureObject(unittest.TestCase):
((('a', ..., ..., "positional_or_keyword"),),
...))
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_signature_on_class_with_new(self):
with self.subTest('FunctionType'):
class C:
@@ -4621,7 +4615,7 @@ class TestSignatureObject(unittest.TestCase):
with self.assertRaisesRegex(ValueError, "callable.*is not supported"):
self.assertEqual(inspect.signature(D), None)
@unittest.expectedFailure # TODO: RUSTPYTHON; NameError: name '_pickle' is not defined. Did you mean: 'pickle'? Or did you forget to import '_pickle'?
@unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: no signature found for builtin type <class '_pickle.Pickler'>
@unittest.skipIf(MISSING_C_DOCSTRINGS,
"Signature information for builtins requires docstrings")
def test_signature_on_builtin_class(self):
@@ -4647,7 +4641,6 @@ class TestSignatureObject(unittest.TestCase):
pass
self.assertEqual(str(inspect.signature(P4)), '(foo, bar)')
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_signature_on_callable_objects(self):
class Foo:
def __call__(self, a):
@@ -5164,7 +5157,7 @@ class TestSignatureObject(unittest.TestCase):
foo_sig = MySignature.from_callable(foo)
self.assertIsInstance(foo_sig, MySignature)
@unittest.expectedFailure # TODO: RUSTPYTHON; NameError: name '_pickle' is not defined. Did you mean: 'pickle'? Or did you forget to import '_pickle'?
@unittest.expectedFailure # TODO: RUSTPYTHON; ValueError: no signature found for builtin type <class '_pickle.Pickler'>
@unittest.skipIf(MISSING_C_DOCSTRINGS,
"Signature information for builtins requires docstrings")
def test_signature_from_callable_builtin_obj(self):
@@ -6335,7 +6328,6 @@ class TestSignatureDefinitions(unittest.TestCase):
with self.assertRaises(ValueError):
inspect.signature(func)
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: None != '(raw, buffer_size=DEFAULT_BUFFER_SIZE)'
@support.requires_docstrings
def test_base_class_have_text_signature(self):
# see issue 43118
@@ -6567,7 +6559,6 @@ class TestRepl(unittest.TestCase):
raise ValueError("Process didn't exit properly.")
return output
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 'The source is: <<<def f():\n print(0)\n return 1 + 2\n>>>' not found in 'Traceback (most recent call last):\n File "<stdin>", line 1, in <module>\n File "crates/pylib/Lib/inspect.py", line 1161, in getsource\n lines, lnum = getsourcelines(object)\n ~~~~~~~~~~~~~~^^^^^^^^\n File "crates/pylib/Lib/inspect.py", line 1143, in getsourcelines\n lines, lnum = findsource(object)\n ~~~~~~~~~~^^^^^^^^\n File "crates/pylib/Lib/inspect.py", line 978, in findsource\n raise OSError(\'could not get source code\')\nOSError: could not get source code\n'
@unittest.skipIf(not has_subprocess_support, "test requires subprocess")
def test_getsource(self):
output = self.run_on_interactive_mode(textwrap.dedent("""\
Lib/test/test_itertools.py
Vendored
-3
View File
@@ -754,7 +754,6 @@ class TestBasicOps(unittest.TestCase):
next(g)
next(g) # must pass with address sanitizer
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 4 != 1
def test_grouper_reentrant_eq_does_not_crash(self):
# regression test for gh-146613
grouper_iter = None
@@ -1167,7 +1166,6 @@ class TestBasicOps(unittest.TestCase):
self.assertRaises(ValueError, next, starmap(errfunc, [(4,5)]))
self.assertRaises(TypeError, next, starmap(onearg, [(4,5)]))
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_islice(self):
for args in [ # islice(args) should agree with range(args)
(10, 20, 3),
@@ -2493,7 +2491,6 @@ class RegressionTests(unittest.TestCase):
class SubclassWithKwargsTest(unittest.TestCase):
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_keywords_in_subclass(self):
# count is not subclassable...
testcases = [
Lib/test/test_list.py
Vendored
-1
View File
@@ -49,7 +49,6 @@ class ListTest(list_tests.CommonTest):
with self.assertRaisesRegex(TypeError, 'keyword argument'):
list(sequence=[])
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_keywords_in_subclass(self):
class subclass(list):
pass
Lib/test/test_marshal.py
Vendored
+1 -2
View File
@@ -493,7 +493,6 @@ class BugsTestCase(unittest.TestCase):
unicode_string = 'T'
self.assertRaises(TypeError, marshal.loads, unicode_string)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_bad_reader(self):
class BadReader(io.BytesIO):
def readinto(self, buf):
@@ -510,7 +509,7 @@ class BugsTestCase(unittest.TestCase):
for i in range(len(data)):
self.assertRaises(EOFError, marshal.loads, data[0: i])
@unittest.expectedFailure # TODO: RUSTPYTHON
@unittest.expectedFailure # TODO: RUSTPYTHON; set iteration is insertion order, so PYTHONHASHSEED does not scramble repr
def test_deterministic_sets(self):
# bpo-37596: To support reproducible builds, sets and frozensets need to
# have their elements serialized in a consistent order (even when they
Lib/test/test_ordered_dict.py
Vendored
-9
View File
@@ -805,7 +805,6 @@ class CPythonOrderedDictSideEffects:
msg = re.escape("OrderedDict mutated during iteration")
self.assertRaisesRegex(RuntimeError, msg, operator.eq, dict1, dict2)
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeError not raised by eq
def test_issue119004_change_size_by_clear(self):
class Key(_TriggerSideEffectOnEqual):
def side_effect(self):
@@ -818,7 +817,6 @@ class CPythonOrderedDictSideEffects:
self.assertDictEqual(dict1, {})
self.assertDictEqual(dict2, dict.fromkeys((0, Key(), 4.2)))
@unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'NoneType' object has no attribute 'key'
def test_issue119004_change_size_by_delete_key(self):
class Key(_TriggerSideEffectOnEqual):
def side_effect(self):
@@ -832,7 +830,6 @@ class CPythonOrderedDictSideEffects:
self.assertDictEqual(dict1, dict.fromkeys((0, 4.2)))
self.assertDictEqual(dict2, dict.fromkeys((0, Key(), 4.2)))
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeError not raised by eq
def test_issue119004_change_linked_list_by_clear(self):
class Key(_TriggerSideEffectOnEqual):
def side_effect(self):
@@ -846,7 +843,6 @@ class CPythonOrderedDictSideEffects:
self.assertDictEqual(dict1, dict.fromkeys(('a', 'b'), 'c'))
self.assertDictEqual(dict2, dict.fromkeys((0, Key(), 4.2)))
@unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'NoneType' object has no attribute 'key'
def test_issue119004_change_linked_list_by_delete_key(self):
class Key(_TriggerSideEffectOnEqual):
def side_effect(self):
@@ -861,7 +857,6 @@ class CPythonOrderedDictSideEffects:
self.assertDictEqual(dict1, {0: None, 'a': 'c', 4.2: None})
self.assertDictEqual(dict2, dict.fromkeys((0, Key(), 4.2)))
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 1 != 2
def test_issue119004_change_size_by_delete_key_in_dict_eq(self):
class Key(_TriggerSideEffectOnEqual):
trigger = 0
@@ -878,7 +873,6 @@ class CPythonOrderedDictSideEffects:
self.assertDictEqual(dict1, dict.fromkeys((0, 4.2)))
self.assertDictEqual(dict2, dict.fromkeys((0, Key(), 4.2)))
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeError not raised by copy
def test_issue148660_copy_clear_in_key_eq(self):
# gh-148660: od.copy() must not crash when a key's __eq__ clears od
# while copy() is inserting into the new dict.
@@ -901,7 +895,6 @@ class CPythonOrderedDictSideEffects:
msg = "OrderedDict mutated during iteration"
self.assertRaisesRegex(RuntimeError, msg, od.copy)
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeError not raised by copy
def test_issue148660_copy_clear_in_subclass_getitem(self):
# gh-148660: od.copy() must not crash when a subclass __getitem__
# clears od.
@@ -956,7 +949,6 @@ class CPythonOrderedDictTests(OrderedDictTests,
check(iter(od.items()), itersize)
check(iter(od.values()), itersize)
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: RuntimeError not raised
def test_key_change_during_iteration(self):
OrderedDict = self.OrderedDict
@@ -974,7 +966,6 @@ class CPythonOrderedDictTests(OrderedDictTests,
del od['c']
self.assertEqual(list(od), list('bdeaf'))
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_iterators_pickling(self):
OrderedDict = self.OrderedDict
pairs = [('c', 1), ('b', 2), ('a', 3), ('d', 4), ('e', 5), ('f', 6)]
Lib/test/test_pydoc/test_pydoc.py
Vendored
+1 -3
View File
@@ -1573,7 +1573,7 @@ class TestDescriptions(unittest.TestCase):
"dump(obj, /) method of _pickle.Pickler instance")
# this should *never* include self!
@unittest.expectedFailure # TODO: RUSTPYTHON
@unittest.expectedFailure # TODO: RUSTPYTHON; os.stat signature is generated as (path, dir_fd, follow_symlinks, /)
@requires_docstrings
def test_module_level_callable(self):
self.assertEqual(self._get_summary_line(os.stat),
@@ -1994,7 +1994,6 @@ class PydocFodderTest(unittest.TestCase):
self.assertIn(' | B_classmethod(x)', lines)
self.assertIn(' | B_classmethod_alias = B_classmethod(x)', lines)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_html_doc_routines_in_class(self, cls=pydocfodder.B):
doc = pydoc.HTMLDoc()
result = doc.docclass(cls)
@@ -2033,7 +2032,6 @@ class PydocFodderTest(unittest.TestCase):
def test_text_doc_inherited_routines_in_class(self):
self.test_text_doc_routines_in_class(pydocfodder.D)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_html_doc_inherited_routines_in_class(self):
self.test_html_doc_routines_in_class(pydocfodder.D)
Lib/test/test_re.py
Vendored
-1
View File
@@ -2939,7 +2939,6 @@ class ImplementationTest(unittest.TestCase):
pat = re.compile("")
check_disallow_instantiation(self, type(pat.scanner("")))
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_deprecated_modules(self):
deprecated = {
'sre_compile': ['compile', 'error',
Lib/test/test_repl.py
Vendored
-2
View File
@@ -207,7 +207,6 @@ class TestInteractiveInterpreter(unittest.TestCase):
]
self.assertEqual(traceback_lines, expected_lines)
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 101 != 0
def test_interactive_traceback_reporting_multiple_input(self):
user_input1 = dedent("""
def foo(x):
@@ -308,7 +307,6 @@ class TestInteractiveInterpreter(unittest.TestCase):
]
self.assertEqual(output.splitlines()[4:-1], expected_lines)
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 101 != 0
def test_interactive_source_is_in_linecache(self):
user_input = dedent("""
def foo(x):
Lib/test/test_socket.py
Vendored
-11
View File
@@ -943,7 +943,6 @@ def requireSocket(*args):
class GeneralModuleTests(unittest.TestCase):
@unittest.expectedFailure # TODO: RUSTPYTHON; gc.is_tracked not implemented
@unittest.skipUnless(_socket is not None, 'need _socket module')
def test_socket_type(self):
self.assertTrue(gc.is_tracked(_socket.socket))
@@ -1007,7 +1006,6 @@ class GeneralModuleTests(unittest.TestCase):
with self.assertRaises(OSError, msg=msg % 'socket.gaierror'):
raise socket.gaierror
@unittest.expectedFailure # TODO: RUSTPYTHON; error message format differs
def testSendtoErrors(self):
# Testing that sendto doesn't mask failures. See #10169.
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
@@ -2230,7 +2228,6 @@ class GeneralModuleTests(unittest.TestCase):
source=_socket)
enum._test_simple_enum(CheckedAddressInfo, socket.AddressInfo)
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: Expected type 'int' but 'Mut' found
@unittest.skipUnless(hasattr(socket.socket, "sendmsg"),"sendmsg not supported")
def test_sendmsg_reentrant_ancillary_mutation(self):
@@ -2448,14 +2445,12 @@ class ISOTPTest(unittest.TestCase):
with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_ISOTP) as s:
pass
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: AF_CAN address must be a tuple (interface,) or (interface, addr)
def testTooLongInterfaceName(self):
# most systems limit IFNAMSIZ to 16, take 1024 to be sure
with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_ISOTP) as s:
with self.assertRaisesRegex(OSError, 'interface name too long'):
s.bind(('x' * 1024, 1, 2))
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: AF_CAN address must be a tuple (interface,) or (interface, addr)
def testBind(self):
try:
with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_ISOTP) as s:
@@ -2477,7 +2472,6 @@ class J1939Test(unittest.TestCase):
super().__init__(*args, **kwargs)
self.interface = "vcan0"
@unittest.expectedFailure # TODO: RUSTPYTHON; - J1939 constants not fully implemented
@unittest.skipUnless(hasattr(socket, "CAN_J1939"),
'socket.CAN_J1939 required for this test.')
def testJ1939Constants(self):
@@ -2519,7 +2513,6 @@ class J1939Test(unittest.TestCase):
with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_J1939) as s:
pass
@unittest.expectedFailure # TODO: RUSTPYTHON; - AF_CAN J1939 address format not fully implemented
def testBind(self):
try:
with socket.socket(socket.PF_CAN, socket.SOCK_DGRAM, socket.CAN_J1939) as s:
@@ -2698,9 +2691,6 @@ class BasicVSOCKTest(unittest.TestCase):
'Bluetooth sockets required for this test.')
class BasicBluetoothTest(unittest.TestCase):
# Windows publishes the names below. Other platforms still miss
# BTPROTO_HCI and the rest of the non-Windows set.
@(unittest.expectedFailure if sys.platform != "win32" else (lambda func: func))
def testBluetoothConstants(self):
socket.BDADDR_ANY
socket.BDADDR_LOCAL
@@ -7519,7 +7509,6 @@ class ReentrantMutationTests(unittest.TestCase):
See: https://github.com/python/cpython/issues/143988
"""
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: b'Hello' != b'HelloWorldTest'
@unittest.skipUnless(hasattr(socket.socket, "sendmsg"),
"sendmsg not supported")
def test_sendmsg_reentrant_data_mutation(self):
Lib/test/test_struct.py
Vendored
-2
View File
@@ -847,7 +847,6 @@ class StructTest(ComplexesAreIdenticalMixin, unittest.TestCase):
s = struct.Struct('=i2H')
self.assertEqual(repr(s), f'Struct({s.format!r})')
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_c_complex_round_trip(self):
values = [complex(*_) for _ in combinations([1, -1, 0.0, -0.0, 2,
-3, INF, -INF, NAN], 2)]
@@ -874,7 +873,6 @@ class StructTest(ComplexesAreIdenticalMixin, unittest.TestCase):
results = executor.map(exec, [code] * 5)
self.assertListEqual(list(results), [None] * 5)
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: expected at least 1 arguments, got 0
def test_operations_on_half_initialized_Struct(self):
S = struct.Struct.__new__(struct.Struct)
Lib/test/test_structseq.py
Vendored
-1
View File
@@ -286,7 +286,6 @@ class StructSeqTest(unittest.TestCase):
with self.assertRaisesRegex(TypeError, 'unexpected field name'):
copy.replace(t, user=1, error=-1)
@unittest.expectedFailure # TODO: RUSTPYTHON; Wrong error message
def test_copy_replace_with_invisible_fields(self):
assert time.struct_time.n_unnamed_fields == 0
assert time.struct_time.n_sequence_fields < time.struct_time.n_fields
Lib/test/test_sys.py
Vendored
-3
View File
@@ -209,7 +209,6 @@ class SysModuleTest(unittest.TestCase):
def tearDown(self):
test.support.reap_children()
@unittest.expectedFailure # TODO: RUSTPYTHON; latin-1 codec not registered
def test_exit(self):
# call with two arguments
self.assertRaises(TypeError, sys.exit, 42, 42)
@@ -1054,12 +1053,10 @@ class SysModuleTest(unittest.TestCase):
'stdout: surrogateescape\n'
'stderr: backslashreplace\n')
@unittest.expectedFailure # TODO: RUSTPYTHON; iso8859_1 codec not registered
@support.requires_subprocess()
def test_c_locale_surrogateescape(self):
self.check_locale_surrogateescape('C')
@unittest.expectedFailure # TODO: RUSTPYTHON; iso8859_1 codec not registered
@support.requires_subprocess()
def test_posix_locale_surrogateescape(self):
self.check_locale_surrogateescape('POSIX')
Lib/test/test_threading.py
Vendored
-2
View File
@@ -1018,7 +1018,6 @@ class ThreadTests(BaseTestCase):
finally:
threading.settrace(old_trace)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_gettrace_all_threads(self):
def fn(*args): pass
old_trace = threading.gettrace()
@@ -1057,7 +1056,6 @@ class ThreadTests(BaseTestCase):
finally:
threading.setprofile(old_profile)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_getprofile_all_threads(self):
def fn(*args): pass
old_profile = threading.getprofile()
Lib/test/test_trace.py
Vendored
-2
View File
@@ -375,7 +375,6 @@ class TestCoverage(unittest.TestCase):
r = tracer.results()
r.write_results(show_missing=True, summary=True, coverdir=TESTFN)
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 'pprint.py' not found in ''
@requires_resource('cpu')
def test_coverage(self):
tracer = trace.Trace(trace=0, count=1)
@@ -400,7 +399,6 @@ class TestCoverage(unittest.TestCase):
files = os.listdir(TESTFN)
self.assertEqual(files, ['_importlib.cover']) # Ignore __import__
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 'test.tracedmodules.testmod' not found in {}
def test_issue9936(self):
tracer = trace.Trace(trace=0, count=1)
modname = 'test.tracedmodules.testmod'
Lib/test/test_tuple.py
Vendored
-1
View File
@@ -42,7 +42,6 @@ class TupleTest(seq_tests.CommonTest):
with self.assertRaisesRegex(TypeError, 'keyword argument'):
tuple(sequence=())
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_keywords_in_subclass(self):
class subclass(tuple):
pass
Lib/test/test_warnings/__init__.py
Vendored
-35
View File
@@ -607,7 +607,6 @@ class WarnTests(BaseTest):
self.assertEqual(os.path.basename(w[-1].filename),
"<sys>")
@unittest.expectedFailure # TODO: RUSTPYTHON; + crates/pylib/Lib/test/test_warnings/__init__.py
def test_stacklevel_import(self):
# Issue #24305: With stacklevel=2, module-level warnings should work.
import_helper.unload('test.test_warnings.data.import_warning')
@@ -853,38 +852,6 @@ class CWarnTests(WarnTests, unittest.TestCase):
self.assertIsNot(original_warnings, self.module)
self.assertNotHasAttr(self.module.warn, '__code__')
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 1 != 2
def test_gh86298_loader_and_spec_loader_disagree(self):
return super().test_gh86298_loader_and_spec_loader_disagree()
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 1 != 2
def test_gh86298_no_spec(self):
return super().test_gh86298_no_spec()
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 1 != 2
def test_gh86298_no_spec_loader(self):
return super().test_gh86298_no_spec_loader()
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: 1 != 2
def test_gh86298_spec_is_none(self):
return super().test_gh86298_spec_is_none()
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: AttributeError not raised
def test_gh86298_no_loader_and_no_spec_loader(self):
return super().test_gh86298_no_loader_and_no_spec_loader()
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ValueError not raised
def test_gh86298_loader_is_none_and_spec_is_none(self):
return super().test_gh86298_loader_is_none_and_spec_is_none()
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ValueError not raised
def test_gh86298_loader_is_none_and_spec_loader_is_none(self):
return super().test_gh86298_loader_is_none_and_spec_loader_is_none()
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ValueError not raised
def test_gh86298_no_loader_and_spec_is_none(self):
return super().test_gh86298_no_loader_and_spec_is_none()
class PyWarnTests(WarnTests, unittest.TestCase):
module = py_warnings
@@ -960,7 +927,6 @@ class _WarningsTests(BaseTest, unittest.TestCase):
module = c_warnings
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: UserWarning not raised by warn
def test_filter(self):
# Everything should function even if 'filters' is not in warnings.
with self.module.catch_warnings() as w:
@@ -1130,7 +1096,6 @@ class _WarningsTests(BaseTest, unittest.TestCase):
self.assertNotIn(b'Warning!', stderr)
self.assertNotIn(b'Error', stderr)
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: 'int' object is not iterable
def test_issue31285(self):
# warn_explicit() should neither raise a SystemError nor cause an
# assertion failure, in case the return value of get_source() has a
Lib/test/test_zipfile/test_core.py
Vendored
-2
View File
@@ -3551,7 +3551,6 @@ class TestExecutablePrependedZip(unittest.TestCase):
def test_read_zip64_with_exe_prepended(self):
self._test_zip_works(self.exe_zip64)
@unittest.expectedFailure # TODO: RUSTPYTHON
@unittest.skipUnless(sys.executable, 'sys.executable required.')
@unittest.skipUnless(os.access('/bin/bash', os.X_OK),
'Test relies on #!/bin/bash working.')
@@ -3560,7 +3559,6 @@ class TestExecutablePrependedZip(unittest.TestCase):
output = subprocess.check_output([self.exe_zip, sys.executable])
self.assertIn(b'number in executable: 5', output)
@unittest.expectedFailure # TODO: RUSTPYTHON
@unittest.skipUnless(sys.executable, 'sys.executable required.')
@unittest.skipUnless(os.access('/bin/bash', os.X_OK),
'Test relies on #!/bin/bash working.')
Lib/test/test_zoneinfo/test_zoneinfo.py
Vendored
-1
View File
@@ -1866,7 +1866,6 @@ class TzPathTest(TzPathUserMixin, ZoneInfoTestBase):
with self.subTest("filtered", path_var=path_var):
self.assertSequenceEqual(tzpath, expected_paths)
@unittest.expectedFailure # TODO: RUSTPYTHON; + /home/runner/work/RustPython/RustPython/crates/pylib/Lib/test/test_zoneinfo/test_zoneinfo.py
def test_env_variable_relative_paths_warning_location(self):
path_var = "path/to/somewhere"
crates/capi/src/setobject.rs
+3 -3
View File
@@ -66,9 +66,9 @@ pub unsafe extern "C" fn PySet_Contains(anyset: *mut PyObject, key: *mut PyObjec
let key = unsafe { &*key };
if let Some(set) = anyset.downcast_ref::<PySet>() {
set.__contains__(key, vm)
set.contains(key, vm)
} else if let Some(frozenset) = anyset.downcast_ref::<PyFrozenSet>() {
frozenset.__contains__(key, vm)
frozenset.contains(key, vm)
} else {
Err(vm.new_type_error(format!(
"expected set or frozenset, got '{}'",
@@ -83,7 +83,7 @@ pub unsafe extern "C" fn PySet_Discard(set: *mut PyObject, key: *mut PyObject) -
with_vm(|vm| {
let set = unsafe { &*set }.try_downcast_ref::<PySet>(vm)?;
let key = unsafe { &*key };
let had_item = set.__contains__(key, vm)?;
let had_item = set.contains(key, vm)?;
if had_item {
set.discard(key.to_owned(), vm)?;
}
crates/common/src/encodings/cjk/iso2022.rs
+22 -5
View File
@@ -36,12 +36,29 @@ const CHARSET_JISX0213_2: u8 = b'P' | CHARSET_DBCS;
const CHARSET_JISX0213_2004_1: u8 = b'Q' | CHARSET_DBCS;
const CHARSET_JISX0208_O: u8 = b'@' | CHARSET_DBCS;
const F_SHIFTED: u8 = 0x01;
const F_ESCTHROUGHOUT: u8 = 0x02;
bitflags::bitflags! {
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
struct Iso2022StateFlags: u8 {
const SHIFTED = 0x01;
const ESCTHROUGHOUT = 0x02;
}
}
const NO_SHIFT: u8 = 0x01;
const USE_G2: u8 = 0x02;
const USE_JISX0208_EXT: u8 = 0x04;
bitflags::bitflags! {
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
struct Iso2022ConfigFlags: u8 {
const NO_SHIFT = 0x01;
const USE_G2 = 0x02;
const USE_JISX0208_EXT = 0x04;
}
}
const F_SHIFTED: u8 = Iso2022StateFlags::SHIFTED.bits();
const F_ESCTHROUGHOUT: u8 = Iso2022StateFlags::ESCTHROUGHOUT.bits();
const NO_SHIFT: u8 = Iso2022ConfigFlags::NO_SHIFT.bits();
const USE_G2: u8 = Iso2022ConfigFlags::USE_G2.bits();
const USE_JISX0208_EXT: u8 = Iso2022ConfigFlags::USE_JISX0208_EXT.bits();
#[derive(Clone, Copy)]
struct Designation {
crates/compiler-core/src/bytecode.rs
+34 -9
View File
@@ -425,15 +425,40 @@ impl<T> IndexMut<oparg::VarNum> for [T] {
}
}
/// Per-slot kind flags for localsplus (co_localspluskinds).
pub const CO_FAST_ARG_POS: u8 = 0x02;
pub const CO_FAST_ARG_KW: u8 = 0x04;
pub const CO_FAST_ARG_VAR: u8 = 0x08;
pub const CO_FAST_ARG: u8 = CO_FAST_ARG_POS | CO_FAST_ARG_KW | CO_FAST_ARG_VAR;
pub const CO_FAST_HIDDEN: u8 = 0x10;
pub const CO_FAST_LOCAL: u8 = 0x20;
pub const CO_FAST_CELL: u8 = 0x40;
pub const CO_FAST_FREE: u8 = 0x80;
bitflagset::bitflag! {
/// Per-slot kind flags for localsplus (`co_localspluskinds`).
/// Values are bit *positions* (`Local` is bit 5 → mask `0x20`).
#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]
#[repr(u8)]
pub enum CoFastFlag {
ArgPos = 1,
ArgKw = 2,
ArgVar = 3,
Hidden = 4,
Local = 5,
Cell = 6,
Free = 7,
}
}
bitflagset::bitflagset! {
#[derive(Copy, Clone, PartialEq, Eq)]
pub struct CoFastFlags(u8): CoFastFlag
}
impl CoFastFlags {
pub const ARG: Self =
Self::from_slice(&[CoFastFlag::ArgPos, CoFastFlag::ArgKw, CoFastFlag::ArgVar]);
}
pub const CO_FAST_ARG_POS: u8 = CoFastFlags::from_element(CoFastFlag::ArgPos).bits();
pub const CO_FAST_ARG_KW: u8 = CoFastFlags::from_element(CoFastFlag::ArgKw).bits();
pub const CO_FAST_ARG_VAR: u8 = CoFastFlags::from_element(CoFastFlag::ArgVar).bits();
pub const CO_FAST_ARG: u8 = CoFastFlags::ARG.bits();
pub const CO_FAST_HIDDEN: u8 = CoFastFlags::from_element(CoFastFlag::Hidden).bits();
pub const CO_FAST_LOCAL: u8 = CoFastFlags::from_element(CoFastFlag::Local).bits();
pub const CO_FAST_CELL: u8 = CoFastFlags::from_element(CoFastFlag::Cell).bits();
pub const CO_FAST_FREE: u8 = CoFastFlags::from_element(CoFastFlag::Free).bits();
/// Primary container of a single code object. Each python function has
/// a code object. Also a module has a code object.
crates/compiler-core/src/marshal.rs
+8 -10
View File
@@ -1616,10 +1616,6 @@ pub fn read_pstring<R: Read>(rdr: &mut R) -> Result<&[u8]> {
rdr.read_slice(n as u32)
}
const CO_FAST_LOCAL: u8 = 0x20;
const CO_FAST_CELL: u8 = 0x40;
const CO_FAST_FREE: u8 = 0x80;
pub struct LocalsPlusResult<S> {
pub varnames: Vec<S>,
pub cellvars: Vec<S>,
@@ -1645,10 +1641,11 @@ pub fn split_localplus<S: Clone>(
// First pass: collect varnames (LOCAL entries) and freevars
for (name, &kind) in names.iter().zip(kinds.iter()) {
if kind & CO_FAST_LOCAL != 0 {
let kind = CoFastFlags::from_bits_retain(kind);
if kind.contains(&CoFastFlag::Local) {
varnames.push(name.clone());
}
if kind & CO_FAST_FREE != 0 {
if kind.contains(&CoFastFlag::Free) {
freevars.push(name.clone());
}
}
@@ -1659,8 +1656,9 @@ pub fn split_localplus<S: Clone>(
// This preserves the original ordering from localsplusnames.
let mut arg_cell_positions = Vec::new(); // (cell_idx, localplus_idx)
for (i, (name, &kind)) in names.iter().zip(kinds.iter()).enumerate() {
let is_local = kind & CO_FAST_LOCAL != 0;
let is_cell = kind & CO_FAST_CELL != 0;
let kind = CoFastFlags::from_bits_retain(kind);
let is_local = kind.contains(&CoFastFlag::Local);
let is_cell = kind.contains(&CoFastFlag::Cell);
if is_cell {
let cell_idx = cellvars.len();
cellvars.push(name.clone());
@@ -1701,7 +1699,7 @@ pub fn split_localplus<S: Clone>(
let mut deref_map = alloc::vec![u32::MAX; names.len()];
let mut cell_idx = 0u32;
for (i, &kind) in kinds.iter().enumerate() {
if kind & CO_FAST_CELL != 0 {
if CoFastFlags::from_bits_retain(kind).contains(&CoFastFlag::Cell) {
deref_map[i] = cell_idx;
cell_idx += 1;
}
@@ -1709,7 +1707,7 @@ pub fn split_localplus<S: Clone>(
let ncells = cellvars.len();
let mut free_idx = 0u32;
for (i, &kind) in kinds.iter().enumerate() {
if kind & CO_FAST_FREE != 0 {
if CoFastFlags::from_bits_retain(kind).contains(&CoFastFlag::Free) {
deref_map[i] = ncells as u32 + free_idx;
free_idx += 1;
}
crates/host_env/Cargo.toml
+1
View File
@@ -29,6 +29,7 @@ rustls = [
rustpython-wtf8 = { workspace = true }
bitflags = { workspace = true }
bitflagset = { workspace = true }
der = { workspace = true, optional = true }
getrandom = { workspace = true }
libc = { workspace = true }
crates/host_env/src/io.rs
+32 -19
View File
@@ -9,16 +9,29 @@ use rustix::{fs::FileType, io::Errno};
use crate::fileutils;
use crate::{crt_fd, os};
bitflags::bitflags! {
bitflagset::bitflag! {
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub struct FileMode: u8 {
const CREATED = 0b0001;
const READABLE = 0b0010;
const WRITABLE = 0b0100;
const APPENDING = 0b1000;
#[repr(u8)]
pub enum FileModeFlag {
Created = 0,
Readable = 1,
Writable = 2,
Appending = 3,
}
}
bitflagset::bitflagset! {
#[derive(Copy, Clone, PartialEq, Eq)]
pub struct FileMode(u8): FileModeFlag
}
impl FileMode {
pub const CREATED: Self = Self::from_element(FileModeFlag::Created);
pub const READABLE: Self = Self::from_element(FileModeFlag::Readable);
pub const WRITABLE: Self = Self::from_element(FileModeFlag::Writable);
pub const APPENDING: Self = Self::from_element(FileModeFlag::Appending);
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum FileModeError {
Invalid,
@@ -45,20 +58,20 @@ pub struct ParsedFileMode {
impl FileMode {
pub const fn raw_mode(self) -> &'static str {
if self.contains(Self::CREATED) {
if self.contains(Self::READABLE) {
if self.contains(&FileModeFlag::Created) {
if self.contains(&FileModeFlag::Readable) {
"xb+"
} else {
"xb"
}
} else if self.contains(Self::APPENDING) {
if self.contains(Self::READABLE) {
} else if self.contains(&FileModeFlag::Appending) {
if self.contains(&FileModeFlag::Readable) {
"ab+"
} else {
"ab"
}
} else if self.contains(Self::READABLE) {
if self.contains(Self::WRITABLE) {
} else if self.contains(&FileModeFlag::Readable) {
if self.contains(&FileModeFlag::Writable) {
"rb+"
} else {
"rb"
@@ -81,7 +94,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result<ParsedFileMode, FileModeError
return Err(FileModeError::BadRwa);
}
rwa = true;
mode.insert(FileMode::WRITABLE | FileMode::CREATED);
mode |= FileMode::WRITABLE | FileMode::CREATED;
flags |= os::O_EXCL | os::O_CREAT;
}
b'r' => {
@@ -89,14 +102,14 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result<ParsedFileMode, FileModeError
return Err(FileModeError::BadRwa);
}
rwa = true;
mode.insert(FileMode::READABLE);
mode |= FileMode::READABLE;
}
b'w' => {
if rwa {
return Err(FileModeError::BadRwa);
}
rwa = true;
mode.insert(FileMode::WRITABLE);
mode |= FileMode::WRITABLE;
flags |= os::O_CREAT | os::O_TRUNC;
}
b'a' => {
@@ -104,7 +117,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result<ParsedFileMode, FileModeError
return Err(FileModeError::BadRwa);
}
rwa = true;
mode.insert(FileMode::WRITABLE | FileMode::APPENDING);
mode |= FileMode::WRITABLE | FileMode::APPENDING;
flags |= os::O_APPEND | os::O_CREAT;
}
b'+' => {
@@ -112,7 +125,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result<ParsedFileMode, FileModeError
return Err(FileModeError::BadRwa);
}
plus = true;
mode.insert(FileMode::READABLE | FileMode::WRITABLE);
mode |= FileMode::READABLE | FileMode::WRITABLE;
}
b'b' => {}
_ => return Err(FileModeError::Invalid),
@@ -123,9 +136,9 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result<ParsedFileMode, FileModeError
return Err(FileModeError::BadRwa);
}
if mode.contains(FileMode::READABLE | FileMode::WRITABLE) {
if mode.is_superset(&(FileMode::READABLE | FileMode::WRITABLE)) {
flags |= os::O_RDWR;
} else if mode.contains(FileMode::READABLE) {
} else if mode.contains(&FileModeFlag::Readable) {
flags |= os::O_RDONLY;
} else {
flags |= os::O_WRONLY;
crates/host_env/src/io_unsupported.rs
+32 -19
View File
@@ -5,16 +5,29 @@ use crate::crt_fd;
use crate::errno::errors::{EAGAIN, EBADF, EINTR, EISDIR};
use crate::os::{O_APPEND, O_CREAT, O_EXCL, O_RDONLY, O_RDWR, O_TRUNC, O_WRONLY};
bitflags::bitflags! {
bitflagset::bitflag! {
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub struct FileMode: u8 {
const CREATED = 0b0001;
const READABLE = 0b0010;
const WRITABLE = 0b0100;
const APPENDING = 0b1000;
#[repr(u8)]
pub enum FileModeFlag {
Created = 0,
Readable = 1,
Writable = 2,
Appending = 3,
}
}
bitflagset::bitflagset! {
#[derive(Copy, Clone, PartialEq, Eq)]
pub struct FileMode(u8): FileModeFlag
}
impl FileMode {
pub const CREATED: Self = Self::from_element(FileModeFlag::Created);
pub const READABLE: Self = Self::from_element(FileModeFlag::Readable);
pub const WRITABLE: Self = Self::from_element(FileModeFlag::Writable);
pub const APPENDING: Self = Self::from_element(FileModeFlag::Appending);
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub enum FileModeError {
Invalid,
@@ -41,20 +54,20 @@ pub struct ParsedFileMode {
impl FileMode {
pub const fn raw_mode(self) -> &'static str {
if self.contains(Self::CREATED) {
if self.contains(Self::READABLE) {
if self.contains(&FileModeFlag::Created) {
if self.contains(&FileModeFlag::Readable) {
"xb+"
} else {
"xb"
}
} else if self.contains(Self::APPENDING) {
if self.contains(Self::READABLE) {
} else if self.contains(&FileModeFlag::Appending) {
if self.contains(&FileModeFlag::Readable) {
"ab+"
} else {
"ab"
}
} else if self.contains(Self::READABLE) {
if self.contains(Self::WRITABLE) {
} else if self.contains(&FileModeFlag::Readable) {
if self.contains(&FileModeFlag::Writable) {
"rb+"
} else {
"rb"
@@ -78,7 +91,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result<ParsedFileMode, FileModeError
return Err(FileModeError::BadRwa);
}
rwa = true;
mode.insert(FileMode::WRITABLE | FileMode::CREATED);
mode |= FileMode::WRITABLE | FileMode::CREATED;
flags |= O_EXCL | O_CREAT;
}
b'r' => {
@@ -86,14 +99,14 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result<ParsedFileMode, FileModeError
return Err(FileModeError::BadRwa);
}
rwa = true;
mode.insert(FileMode::READABLE);
mode |= FileMode::READABLE;
}
b'w' => {
if rwa {
return Err(FileModeError::BadRwa);
}
rwa = true;
mode.insert(FileMode::WRITABLE);
mode |= FileMode::WRITABLE;
flags |= O_CREAT | O_TRUNC;
}
b'a' => {
@@ -101,7 +114,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result<ParsedFileMode, FileModeError
return Err(FileModeError::BadRwa);
}
rwa = true;
mode.insert(FileMode::WRITABLE | FileMode::APPENDING);
mode |= FileMode::WRITABLE | FileMode::APPENDING;
flags |= O_APPEND | O_CREAT;
}
b'+' => {
@@ -109,7 +122,7 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result<ParsedFileMode, FileModeError
return Err(FileModeError::BadRwa);
}
plus = true;
mode.insert(FileMode::READABLE | FileMode::WRITABLE);
mode |= FileMode::READABLE | FileMode::WRITABLE;
}
b'b' => {
if binary {
@@ -125,9 +138,9 @@ pub fn parse_fileio_mode(mode_str: &str) -> Result<ParsedFileMode, FileModeError
return Err(FileModeError::BadRwa);
}
if mode.contains(FileMode::READABLE | FileMode::WRITABLE) {
if mode.is_superset(&(FileMode::READABLE | FileMode::WRITABLE)) {
flags |= O_RDWR;
} else if mode.contains(FileMode::READABLE) {
} else if mode.contains(&FileModeFlag::Readable) {
flags |= O_RDONLY;
} else {
flags |= O_WRONLY;
crates/host_env/src/multiprocessing.rs
+4 -3
View File
@@ -177,11 +177,12 @@ impl SemHandle {
self.raw as isize
}
/// A handle rebuilt from the integer a `SemLock` stored.
/// Rebuild a handle from the integer a `SemLock` stored.
///
/// # Safety
/// `Drop` closes the semaphore. A caller that still owns that close
/// must forget this value.
///
/// `raw` must be a live `sem_open` handle. `Drop` closes it, so a
/// caller that still owns that close must forget this value.
#[inline]
pub const unsafe fn from_raw(raw: *mut sem_t) -> Self {
Self { raw }
crates/host_env/src/os.rs
+101 -46
View File
@@ -30,10 +30,25 @@ use {
},
};
pub const F_OK: u8 = 0;
pub const R_OK: u8 = 4;
pub const W_OK: u8 = 2;
pub const X_OK: u8 = 1;
bitflagset::bitflag! {
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[repr(u8)]
pub enum AccessFlag {
X = 0,
W = 1,
R = 2,
}
}
bitflagset::bitflagset! {
#[derive(Copy, Clone, PartialEq, Eq)]
pub struct AccessMode(u8): AccessFlag
}
pub const F_OK: u8 = AccessMode::empty().bits();
pub const X_OK: u8 = AccessMode::from_element(AccessFlag::X).bits();
pub const W_OK: u8 = AccessMode::from_element(AccessFlag::W).bits();
pub const R_OK: u8 = AccessMode::from_element(AccessFlag::R).bits();
#[cfg(any(unix, target_os = "wasi"))]
pub use libc::AT_FDCWD;
@@ -158,51 +173,91 @@ mod wasm_oflag {
#[cfg(all(target_arch = "wasm32", target_os = "unknown"))]
pub use wasm_oflag::*;
/// BSD `chflags` bits the `libc` crate does not bind. Published by `_stat`
/// on every platform.
pub const UF_SETTABLE: u32 = 0x0000ffff;
pub const UF_NOUNLINK: u32 = 0x00000010;
pub const UF_TRACKED: u32 = 0x00000040;
pub const UF_DATAVAULT: u32 = 0x00000080;
pub const SF_NOUNLINK: u32 = 0x00100000;
pub const SF_SNAPSHOT: u32 = 0x00200000;
pub const SF_FIRMLINK: u32 = 0x00800000;
pub const SF_DATALESS: u32 = 0x40000000;
pub const SF_SETTABLE: u32 = if cfg!(target_os = "macos") {
0x3fff0000
} else {
0xffff0000
};
#[cfg(target_os = "macos")]
pub const SF_SUPPORTED: u32 = 0x009f0000;
#[cfg(target_os = "macos")]
pub const SF_SYNTHETIC: u32 = 0xc0000000;
bitflagset::bitflag! {
/// BSD `chflags` bits. Values are bit *positions*; masks are `1 << pos`.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[repr(u8)]
pub enum ChFlag {
UfNodump = 0,
UfImmutable = 1,
UfAppend = 2,
UfOpaque = 3,
UfNounlink = 4,
UfCompressed = 5,
UfTracked = 6,
UfDatavault = 7,
UfHidden = 15,
SfArchived = 16,
SfImmutable = 17,
SfAppend = 18,
SfNounlink = 20,
SfSnapshot = 21,
SfFirmlink = 23,
SfDataless = 30,
}
}
/// BSD `chflags` bits libc binds on macOS and `_stat.c` falls back to
/// elsewhere. Published by `_stat` on every platform.
bitflagset::bitflagset! {
#[derive(Copy, Clone, PartialEq, Eq)]
pub struct ChFlags(u32): ChFlag
}
impl ChFlags {
/// Owner-settable bits, including reserved user-flag positions.
pub const UF_SETTABLE: Self = Self::from_bits_retain(0x0000ffff);
pub const SF_SETTABLE: Self = Self::from_bits_retain(if cfg!(target_os = "macos") {
0x3fff0000
} else {
0xffff0000
});
#[cfg(target_os = "macos")]
pub const SF_SUPPORTED: Self = Self::from_bits_retain(0x009f0000);
#[cfg(target_os = "macos")]
pub const SF_SYNTHETIC: Self = Self::from_bits_retain(0xc0000000);
}
pub const UF_NODUMP: u32 = ChFlags::from_element(ChFlag::UfNodump).bits();
pub const UF_IMMUTABLE: u32 = ChFlags::from_element(ChFlag::UfImmutable).bits();
pub const UF_APPEND: u32 = ChFlags::from_element(ChFlag::UfAppend).bits();
pub const UF_OPAQUE: u32 = ChFlags::from_element(ChFlag::UfOpaque).bits();
pub const UF_NOUNLINK: u32 = ChFlags::from_element(ChFlag::UfNounlink).bits();
pub const UF_COMPRESSED: u32 = ChFlags::from_element(ChFlag::UfCompressed).bits();
pub const UF_TRACKED: u32 = ChFlags::from_element(ChFlag::UfTracked).bits();
pub const UF_DATAVAULT: u32 = ChFlags::from_element(ChFlag::UfDatavault).bits();
pub const UF_HIDDEN: u32 = ChFlags::from_element(ChFlag::UfHidden).bits();
pub const UF_SETTABLE: u32 = ChFlags::UF_SETTABLE.bits();
pub const SF_ARCHIVED: u32 = ChFlags::from_element(ChFlag::SfArchived).bits();
pub const SF_IMMUTABLE: u32 = ChFlags::from_element(ChFlag::SfImmutable).bits();
pub const SF_APPEND: u32 = ChFlags::from_element(ChFlag::SfAppend).bits();
pub const SF_NOUNLINK: u32 = ChFlags::from_element(ChFlag::SfNounlink).bits();
pub const SF_SNAPSHOT: u32 = ChFlags::from_element(ChFlag::SfSnapshot).bits();
pub const SF_FIRMLINK: u32 = ChFlags::from_element(ChFlag::SfFirmlink).bits();
pub const SF_DATALESS: u32 = ChFlags::from_element(ChFlag::SfDataless).bits();
pub const SF_SETTABLE: u32 = ChFlags::SF_SETTABLE.bits();
#[cfg(target_os = "macos")]
pub use libc::{
SF_APPEND, SF_ARCHIVED, SF_IMMUTABLE, UF_APPEND, UF_COMPRESSED, UF_HIDDEN, UF_IMMUTABLE,
UF_NODUMP, UF_OPAQUE,
pub const SF_SUPPORTED: u32 = ChFlags::SF_SUPPORTED.bits();
#[cfg(target_os = "macos")]
pub const SF_SYNTHETIC: u32 = ChFlags::SF_SYNTHETIC.bits();
const _: () = {
assert!(UF_NODUMP == 0x00000001);
assert!(UF_IMMUTABLE == 0x00000002);
assert!(UF_APPEND == 0x00000004);
assert!(UF_OPAQUE == 0x00000008);
assert!(UF_NOUNLINK == 0x00000010);
assert!(UF_COMPRESSED == 0x00000020);
assert!(UF_TRACKED == 0x00000040);
assert!(UF_DATAVAULT == 0x00000080);
assert!(UF_HIDDEN == 0x00008000);
assert!(UF_SETTABLE == 0x0000ffff);
assert!(SF_ARCHIVED == 0x00010000);
assert!(SF_IMMUTABLE == 0x00020000);
assert!(SF_APPEND == 0x00040000);
assert!(SF_NOUNLINK == 0x00100000);
assert!(SF_SNAPSHOT == 0x00200000);
assert!(SF_FIRMLINK == 0x00800000);
assert!(SF_DATALESS == 0x40000000);
};
#[cfg(not(target_os = "macos"))]
pub const UF_NODUMP: u32 = 0x00000001;
#[cfg(not(target_os = "macos"))]
pub const UF_IMMUTABLE: u32 = 0x00000002;
#[cfg(not(target_os = "macos"))]
pub const UF_APPEND: u32 = 0x00000004;
#[cfg(not(target_os = "macos"))]
pub const UF_OPAQUE: u32 = 0x00000008;
#[cfg(not(target_os = "macos"))]
pub const UF_COMPRESSED: u32 = 0x00000020;
#[cfg(not(target_os = "macos"))]
pub const UF_HIDDEN: u32 = 0x00008000;
#[cfg(not(target_os = "macos"))]
pub const SF_ARCHIVED: u32 = 0x00010000;
#[cfg(not(target_os = "macos"))]
pub const SF_IMMUTABLE: u32 = 0x00020000;
#[cfg(not(target_os = "macos"))]
pub const SF_APPEND: u32 = 0x00040000;
/// Solaris door/port and BSD whiteout. `_stat` publishes 0 where the
/// platform has no such file type.
crates/host_env/src/posix.rs
+7 -7
View File
@@ -903,7 +903,7 @@ impl From<std::io::Error> for AccessError {
}
}
pub use crate::os::{F_OK, R_OK, W_OK, X_OK};
pub use crate::os::{AccessMode, F_OK, R_OK, W_OK, X_OK};
fn get_permissions(mode: u32) -> Permissions {
Permissions {
@@ -971,24 +971,24 @@ pub fn getgroups() -> std::io::Result<Vec<u32>> {
pub fn check_access(path: &Path, mode: u8) -> Result<bool, AccessError> {
use std::os::unix::fs::MetadataExt;
if mode & !(R_OK | W_OK | X_OK) != 0 {
let Some(mode) = AccessMode::from_bits(mode) else {
return Err(AccessError::InvalidMode);
}
};
let metadata = match crate::fs::metadata(path) {
Ok(m) => m,
Err(_) => return Ok(false),
};
if mode == F_OK {
if mode.is_empty() {
return Ok(true);
}
let perm = get_right_permission(metadata.mode(), metadata.uid(), metadata.gid())?;
let r_ok = (mode & R_OK == 0) || perm.is_readable;
let w_ok = (mode & W_OK == 0) || perm.is_writable;
let x_ok = (mode & X_OK == 0) || perm.is_executable;
let r_ok = !mode.contains(&crate::os::AccessFlag::R) || perm.is_readable;
let w_ok = !mode.contains(&crate::os::AccessFlag::W) || perm.is_writable;
let x_ok = !mode.contains(&crate::os::AccessFlag::X) || perm.is_executable;
Ok(r_ok && w_ok && x_ok)
}
crates/stdlib/src/pickle.rs
+2 -2
View File
@@ -1945,9 +1945,9 @@ mod _pickle {
file: PyObjectRef,
#[pyarg(any, optional)]
protocol: OptionalArg<PyObjectRef>,
#[pyarg(named, optional)]
#[pyarg(any, optional)]
fix_imports: OptionalArg<bool>,
#[pyarg(named, optional)]
#[pyarg(any, optional)]
buffer_callback: OptionalArg<PyObjectRef>,
}
crates/stdlib/src/pystruct.rs
+6
View File
@@ -365,6 +365,12 @@ pub(crate) mod _struct {
let spec = self.ready(vm)?.spec.clone();
UnpackIterator::with_buffer(vm, spec, buffer)
}
#[pymethod]
fn __sizeof__(&self, vm: &VirtualMachine) -> PyResult<usize> {
let inner = self.ready(vm)?;
Ok(core::mem::size_of::<Self>() + inner.spec.codes_sizeof())
}
}
impl Representable for PyStruct {
crates/stdlib/src/socket.rs
+419 -61
View File
@@ -22,7 +22,7 @@ mod _socket {
convert::{IntoPyException, ToPyObject, TryFromBorrowedObject, TryFromObject},
function::{
ArgBytesLike, ArgIntoFloat, ArgMemoryBuffer, ArgStrOrBytesLike, Either, FsPath,
OptionalArg, OptionalOption,
FuncArgs, OptionalArg, OptionalOption,
},
types::{Constructor, DefaultConstructor, Destructor, Initializer, Representable},
utils::ToCString,
@@ -135,6 +135,16 @@ mod _socket {
SOL_CAN_RAW,
};
#[cfg(target_os = "linux")]
#[pyattr]
const J1939_EE_INFO_NONE: i32 = 0;
#[cfg(target_os = "linux")]
#[pyattr]
const J1939_EE_INFO_TX_ABORT: i32 = 1;
#[cfg(target_os = "linux")]
#[pyattr]
const J1939_FILTER_MAX: i32 = 512;
#[cfg(target_os = "linux")]
#[pyattr]
use host_socket::{
@@ -652,6 +662,260 @@ mod _socket {
))]
#[pyattr]
use host_socket::{BDADDR_ANY, BDADDR_LOCAL};
#[cfg(any(
target_os = "android",
target_os = "fuchsia",
target_os = "linux",
target_os = "openbsd"
))]
#[pyattr]
const BTPROTO_L2CAP: i32 = 0;
#[cfg(target_os = "freebsd")]
#[pyattr]
const BTPROTO_L2CAP: i32 = 135;
#[cfg(any(
target_os = "android",
target_os = "fuchsia",
target_os = "linux",
target_os = "openbsd"
))]
#[pyattr]
const BTPROTO_HCI: i32 = 1;
#[cfg(target_os = "freebsd")]
#[pyattr]
const BTPROTO_HCI: i32 = 134;
#[cfg(any(
target_os = "android",
target_os = "fuchsia",
target_os = "linux",
target_os = "openbsd"
))]
#[pyattr]
const BTPROTO_SCO: i32 = 2;
#[cfg(target_os = "freebsd")]
#[pyattr]
const BTPROTO_SCO: i32 = 137;
#[cfg(any(
target_os = "android",
target_os = "fuchsia",
target_os = "linux",
target_os = "openbsd"
))]
#[pyattr]
const BTPROTO_RFCOMM: i32 = 3;
#[cfg(target_os = "freebsd")]
#[pyattr]
const BTPROTO_RFCOMM: i32 = 136;
#[cfg(any(
target_os = "android",
target_os = "fuchsia",
target_os = "linux",
target_os = "openbsd"
))]
#[pyattr]
const SOL_HCI: i32 = 0;
#[cfg(target_os = "freebsd")]
#[pyattr]
const SOL_HCI: i32 = 0x0802;
#[cfg(any(
target_os = "android",
target_os = "fuchsia",
target_os = "linux",
target_os = "openbsd"
))]
#[pyattr]
const HCI_DATA_DIR: i32 = 1;
#[cfg(target_os = "freebsd")]
#[pyattr]
const HCI_DATA_DIR: i32 = 2;
#[cfg(any(
target_os = "android",
target_os = "fuchsia",
target_os = "linux",
target_os = "openbsd"
))]
#[pyattr]
const SOL_L2CAP: i32 = 6;
#[cfg(target_os = "freebsd")]
#[pyattr]
const SOL_L2CAP: i32 = 0x1609;
#[cfg(any(
target_os = "android",
target_os = "fuchsia",
target_os = "linux",
target_os = "openbsd"
))]
#[pyattr]
const SOL_SCO: i32 = 17;
#[cfg(target_os = "freebsd")]
#[pyattr]
const SOL_SCO: i32 = 0x0209;
#[cfg(any(
target_os = "android",
target_os = "fuchsia",
target_os = "linux",
target_os = "openbsd"
))]
#[pyattr]
const SOL_RFCOMM: i32 = 18;
#[cfg(target_os = "freebsd")]
#[pyattr]
const SOL_RFCOMM: i32 = 0x0816;
// Linux <bluetooth/bluetooth.h> / <bluetooth/hci.h> / <bluetooth/l2cap.h>
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const SOL_BLUETOOTH: i32 = 274;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const HCI_DEV_NONE: i32 = 0xffff;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const HCI_CHANNEL_RAW: i32 = 0;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const HCI_CHANNEL_USER: i32 = 1;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const HCI_CHANNEL_MONITOR: i32 = 2;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const HCI_CHANNEL_CONTROL: i32 = 3;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const HCI_CHANNEL_LOGGING: i32 = 4;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const HCI_TIME_STAMP: i32 = 3;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_SECURITY: i32 = 4;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_SECURITY_SDP: i32 = 0;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_SECURITY_LOW: i32 = 1;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_SECURITY_MEDIUM: i32 = 2;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_SECURITY_HIGH: i32 = 3;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_SECURITY_FIPS: i32 = 4;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_FLUSHABLE: i32 = 8;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_FLUSHABLE_OFF: i32 = 0;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_FLUSHABLE_ON: i32 = 1;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_POWER: i32 = 9;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_POWER_FORCE_ACTIVE_OFF: i32 = 0;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_POWER_FORCE_ACTIVE_ON: i32 = 1;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_CHANNEL_POLICY: i32 = 10;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_CHANNEL_POLICY_BREDR_ONLY: i32 = 0;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_CHANNEL_POLICY_BREDR_PREFERRED: i32 = 1;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_CHANNEL_POLICY_AMP_PREFERRED: i32 = 2;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_VOICE: i32 = 11;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_VOICE_TRANSPARENT: i32 = 0x0003;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_VOICE_CVSD_16BIT: i32 = 0x0060;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_SNDMTU: i32 = 12;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_RCVMTU: i32 = 13;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_PHY: i32 = 14;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_PHY_BR_1M_1SLOT: i32 = 0x0001;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_MODE: i32 = 16;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const BT_MODE_BASIC: i32 = 0x00;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const L2CAP_LM: i32 = 0x03;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const L2CAP_LM_MASTER: i32 = 0x0001;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const L2CAP_LM_AUTH: i32 = 0x0002;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const L2CAP_LM_ENCRYPT: i32 = 0x0004;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const L2CAP_LM_TRUSTED: i32 = 0x0008;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const L2CAP_LM_RELIABLE: i32 = 0x0010;
#[cfg(any(target_os = "android", target_os = "fuchsia", target_os = "linux"))]
#[pyattr]
const L2CAP_LM_SECURE: i32 = 0x0020;
#[cfg(any(
target_os = "android",
target_os = "freebsd",
target_os = "fuchsia",
target_os = "linux"
))]
#[pyattr]
const BDADDR_BREDR: i32 = 0x00;
#[cfg(any(
target_os = "android",
target_os = "freebsd",
target_os = "fuchsia",
target_os = "linux"
))]
#[pyattr]
const BDADDR_LE_PUBLIC: i32 = 0x01;
#[cfg(any(
target_os = "android",
target_os = "freebsd",
target_os = "fuchsia",
target_os = "linux"
))]
#[pyattr]
const BDADDR_LE_RANDOM: i32 = 0x02;
#[cfg(any(
target_os = "android",
target_os = "freebsd",
target_os = "fuchsia",
target_os = "linux"
))]
#[pyattr]
const HCI_FILTER: i32 = 2;
// HERE IS WHERE THE BLUETOOTH CONSTANTS END
#[cfg(windows)]
@@ -1183,43 +1447,88 @@ mod _socket {
obj.class().name()
))
})?;
if tuple.is_empty() || tuple.len() > 2 {
return Err(vm
.new_type_error(
"AF_CAN address must be a tuple (interface,) or (interface, addr)",
)
.into());
}
let interface: PyStrRef = tuple[0].clone().downcast().map_err(|obj| {
vm.new_type_error(format!(
"{}(): AF_CAN interface must be str, not {}",
caller,
obj.class().name()
))
})?;
let proto = self.proto.load();
let interface: PyStrRef = tuple
.first()
.cloned()
.ok_or_else(|| {
vm.new_type_error("AF_CAN address must be a tuple (interface, )")
})?
.downcast()
.map_err(|obj| {
vm.new_type_error(format!(
"{}(): AF_CAN interface must be str, not {}",
caller,
obj.class().name()
))
})?;
let interface = interface.try_into_utf8(vm).map_err(IoOrPyException::from)?;
let ifname = interface.as_str();
// Get interface index
let ifindex = if ifname.is_empty() {
0 // Bind to all CAN interfaces
0
} else if ifname.len() >= 16 {
return Err(vm.new_os_error("AF_CAN interface name too long").into());
} else {
// Check interface name length (IFNAMSIZ is typically 16)
if ifname.len() >= 16 {
return Err(vm.new_os_error("interface name too long").into());
}
let cstr = alloc::ffi::CString::new(ifname)
.map_err(|_| vm.new_os_error("invalid interface name"))?;
host_socket::if_nametoindex_checked(cstr.as_c_str())? as i32
};
// Create sockaddr_can
let mut storage: c::sockaddr_storage = unsafe { core::mem::zeroed() };
let can_addr = &mut storage as *mut c::sockaddr_storage as *mut c::sockaddr_can;
unsafe {
(*can_addr).can_family = c::AF_CAN as c::sa_family_t;
(*can_addr).can_ifindex = ifindex;
}
match proto {
c::CAN_RAW | c::CAN_BCM => {
if tuple.len() != 1 {
return Err(vm
.new_type_error("AF_CAN address must be a tuple (interface, )")
.into());
}
}
c::CAN_ISOTP => {
if tuple.len() != 3 {
return Err(vm
.new_type_error(
"AF_CAN ISOTP address must be a tuple (interface, rx_id, tx_id)",
)
.into());
}
let rx_id = tuple[1].try_index(vm)?.try_to_primitive::<u32>(vm)?;
let tx_id = tuple[2].try_index(vm)?.try_to_primitive::<u32>(vm)?;
unsafe {
(*can_addr).can_addr.tp.rx_id = rx_id;
(*can_addr).can_addr.tp.tx_id = tx_id;
}
}
c::CAN_J1939 => {
if tuple.len() != 4 {
return Err(vm
.new_type_error(
"AF_CAN J1939 address must be a tuple (interface, name, pgn, addr)",
)
.into());
}
let name = tuple[1].try_index(vm)?.try_to_primitive::<u64>(vm)?;
let pgn = tuple[2].try_index(vm)?.try_to_primitive::<u32>(vm)?;
let jaddr = tuple[3].try_index(vm)?.try_to_primitive::<u8>(vm)?;
unsafe {
(*can_addr).can_addr.j1939.name = name;
(*can_addr).can_addr.j1939.pgn = pgn;
(*can_addr).can_addr.j1939.addr = jaddr;
}
}
_ => {
return Err(vm
.new_os_error(format!("{caller}(): unsupported CAN protocol"))
.into());
}
}
let storage: host_socket::raw::SockAddrStorage =
unsafe { core::mem::transmute(storage) };
Ok(unsafe {
@@ -1398,7 +1707,7 @@ mod _socket {
#[pyclass(
with(Constructor, Initializer, Representable, Destructor),
flags(BASETYPE)
flags(BASETYPE, HEAPTYPE)
)]
impl PySocket {
fn _init(
@@ -1579,7 +1888,7 @@ mod _socket {
self.sock_snapshot()?.accept_raw()
})?;
let fd = into_sock_fileno(sock);
Ok((fd, get_addr_tuple(&addr, vm)))
Ok((fd, get_addr_tuple(&addr, self.proto.load(), vm)))
}
#[pymethod]
@@ -1652,7 +1961,7 @@ mod _socket {
.recv_from_with_flags(buffer.spare_capacity_mut(), flags)
})?;
unsafe { buffer.set_len(n) };
Ok((buffer, get_addr_tuple(&addr, vm)))
Ok((buffer, get_addr_tuple(&addr, self.proto.load(), vm)))
}
#[pymethod]
@@ -1685,7 +1994,7 @@ mod _socket {
})?;
unsafe { scratch.set_len(n) };
buf.borrow_buf_mut()[..n].copy_from_slice(&scratch);
Ok((n, get_addr_tuple(&addr, vm)))
Ok((n, get_addr_tuple(&addr, self.proto.load(), vm)))
}
#[pymethod]
@@ -1731,25 +2040,26 @@ mod _socket {
}
#[pymethod]
fn sendto(
&self,
bytes: ArgBytesLike,
arg2: PyObjectRef,
arg3: OptionalArg<PyObjectRef>,
vm: &VirtualMachine,
) -> Result<usize, IoOrPyException> {
// signature is bytes[, flags], address
let (flags, address) = match arg3 {
OptionalArg::Present(arg3) => {
// should just be i32::try_from_obj but tests check for error message
let int = arg2
.try_index_opt(vm)
.unwrap_or_else(|| Err(vm.new_type_error("an integer is required")))?;
let flags = int.try_to_primitive::<i32>(vm)?;
(flags, arg3)
fn sendto(&self, args: FuncArgs, vm: &VirtualMachine) -> Result<usize, IoOrPyException> {
if !args.kwargs.is_empty() {
return Err(vm
.new_type_error("sendto() takes no keyword arguments")
.into());
}
let n = args.args.len();
let (bytes, flags, address) = match n {
2 => (args.args[0].clone(), 0, args.args[1].clone()),
3 => {
let flags = args.args[1].try_index(vm)?.try_to_primitive::<i32>(vm)?;
(args.args[0].clone(), flags, args.args[2].clone())
}
_ => {
return Err(vm
.new_type_error(format!("sendto() takes 2 or 3 arguments ({n} given)"))
.into());
}
OptionalArg::Missing => (0, arg2),
};
let bytes = ArgBytesLike::try_from_object(vm, bytes)?;
let addr = self.extract_address(address, "sendto", vm)?;
let buf = bytes.borrow_buf_unlocked(vm)?;
let buf = &*buf;
@@ -1762,8 +2072,8 @@ mod _socket {
#[pymethod]
fn sendmsg(
&self,
buffers: Vec<ArgBytesLike>,
ancdata: OptionalArg,
buffers: PyObjectRef,
ancdata: OptionalArg<PyObjectRef>,
flags: OptionalArg<i32>,
addr: OptionalOption,
vm: &VirtualMachine,
@@ -1779,6 +2089,23 @@ mod _socket {
msg = msg.with_addr(&sockaddr);
}
let collect_iterable =
|obj: &PyObject, msg: &'static str| -> PyResult<Vec<PyObjectRef>> {
match vm.extract_elements_with(obj, Ok) {
Ok(items) => Ok(items),
Err(e) if e.fast_isinstance(vm.ctx.exceptions.type_error) => {
Err(vm.new_type_error(msg))
}
Err(e) => Err(e),
}
};
let buffer_items =
collect_iterable(&buffers, "sendmsg() argument 1 must be an iterable")?;
let buffers = buffer_items
.into_iter()
.map(|obj| ArgBytesLike::try_from_object(vm, obj))
.collect::<PyResult<Vec<_>>>()?;
let buffers = buffers
.iter()
.map(|buf| buf.borrow_buf_unlocked(vm))
@@ -1791,20 +2118,22 @@ mod _socket {
let control_buf;
if let OptionalArg::Present(ancdata) = ancdata {
let cmsgs = vm.extract_elements_with(
&ancdata,
|obj| -> PyResult<(i32, i32, ArgBytesLike)> {
let seq: Vec<PyObjectRef> = obj.try_into_value(vm)?;
let cmsg_fast =
collect_iterable(&ancdata, "sendmsg() argument 2 must be an iterable")?;
let cmsgs = cmsg_fast
.into_iter()
.map(|item| -> PyResult<(i32, i32, ArgBytesLike)> {
let seq: Vec<PyObjectRef> = item.try_into_value(vm)?;
let [lvl, typ, data]: [PyObjectRef; 3] = seq
.try_into()
.map_err(|_| vm.new_type_error("expected a sequence of length 3"))?;
.map_err(|_| vm.new_type_error("[sendmsg() ancillary data items]"))?;
Ok((
lvl.try_into_value(vm)?,
typ.try_into_value(vm)?,
data.try_into_value(vm)?,
lvl.try_index(vm)?.try_to_primitive::<i32>(vm)?,
typ.try_index(vm)?.try_to_primitive::<i32>(vm)?,
ArgBytesLike::try_from_object(vm, data)?,
))
},
)?;
})
.collect::<PyResult<Vec<_>>>()?;
control_buf = Self::pack_cmsgs_to_send(&cmsgs, vm)?;
if !control_buf.is_empty() {
msg = msg.with_control(&control_buf);
@@ -1895,7 +2224,7 @@ mod _socket {
let storage: host_socket::raw::SockAddrStorage =
unsafe { core::mem::transmute(address.storage) };
let addr = unsafe { host_socket::raw::SockAddr::new(storage, address.len as _) };
get_addr_tuple(&addr, vm)
get_addr_tuple(&addr, self.proto.load(), vm)
} else {
vm.ctx.none()
};
@@ -1979,14 +2308,14 @@ mod _socket {
fn getsockname(&self, vm: &VirtualMachine) -> std::io::Result<PyObjectRef> {
let addr = self.sock()?.local_addr()?;
Ok(get_addr_tuple(&addr, vm))
Ok(get_addr_tuple(&addr, self.proto.load(), vm))
}
#[pymethod]
fn getpeername(&self, vm: &VirtualMachine) -> std::io::Result<PyObjectRef> {
let addr = self.sock()?.peer_addr()?;
Ok(get_addr_tuple(&addr, vm))
Ok(get_addr_tuple(&addr, self.proto.load(), vm))
}
#[pymethod]
@@ -2256,7 +2585,11 @@ mod _socket {
}
}
fn get_addr_tuple(addr: &host_socket::raw::SockAddr, vm: &VirtualMachine) -> PyObjectRef {
fn get_addr_tuple(
addr: &host_socket::raw::SockAddr,
#[cfg_attr(not(target_os = "linux"), allow(unused_variables))] proto: i32,
vm: &VirtualMachine,
) -> PyObjectRef {
if let Some(addr) = addr.as_socket() {
return get_ip_addr_tuple(&addr, vm);
}
@@ -2277,7 +2610,6 @@ mod _socket {
{
let family = addr.family();
if family == c::AF_CAN as c::sa_family_t {
// AF_CAN address: (interface_name,) or (interface_name, can_id)
let can_addr = unsafe { &*(addr.as_ptr() as *const c::sockaddr_can) };
let ifindex = can_addr.can_ifindex;
let ifname = if ifindex == 0 {
@@ -2285,7 +2617,33 @@ mod _socket {
} else {
host_socket::if_indextoname_checked(ifindex as u32).unwrap_or_default()
};
return vm.ctx.new_tuple(vec![vm.ctx.new_str(ifname).into()]).into();
let ifname_obj: PyObjectRef = vm.ctx.new_str(ifname).into();
return match proto {
c::CAN_ISOTP => vm
.ctx
.new_tuple(vec![
ifname_obj,
vm.ctx.new_int(unsafe { can_addr.can_addr.tp.rx_id }).into(),
vm.ctx.new_int(unsafe { can_addr.can_addr.tp.tx_id }).into(),
])
.into(),
c::CAN_J1939 => vm
.ctx
.new_tuple(vec![
ifname_obj,
vm.ctx
.new_int(unsafe { can_addr.can_addr.j1939.name })
.into(),
vm.ctx
.new_int(unsafe { can_addr.can_addr.j1939.pgn })
.into(),
vm.ctx
.new_int(unsafe { can_addr.can_addr.j1939.addr })
.into(),
])
.into(),
_ => vm.ctx.new_tuple(vec![ifname_obj]).into(),
};
}
if family == c::AF_ALG as c::sa_family_t {
// AF_ALG address: (type, name)
crates/vm/Cargo.toml
+1
View File
@@ -47,6 +47,7 @@ rustpython-unicode = { workspace = true }
ascii = { workspace = true }
bitflags = { workspace = true }
bitflagset = { workspace = true }
bstr = { workspace = true }
crossbeam-utils = { workspace = true }
constant_time_eq = { workspace = true }
crates/vm/Lib/core_modules/encodings_latin_1.py
+50
View File
@@ -0,0 +1,50 @@
""" Python 'latin-1' Codec
Written by Marc-Andre Lemburg (mal@lemburg.com).
(c) Copyright CNRI, All Rights Reserved. NO WARRANTY.
"""
import codecs
### Codec APIs
class Codec(codecs.Codec):
# Note: Binding these as C functions will result in the class not
# converting them to methods. This is intended.
encode = codecs.latin_1_encode
decode = codecs.latin_1_decode
class IncrementalEncoder(codecs.IncrementalEncoder):
def encode(self, input, final=False):
return codecs.latin_1_encode(input,self.errors)[0]
class IncrementalDecoder(codecs.IncrementalDecoder):
def decode(self, input, final=False):
return codecs.latin_1_decode(input,self.errors)[0]
class StreamWriter(Codec,codecs.StreamWriter):
pass
class StreamReader(Codec,codecs.StreamReader):
pass
class StreamConverter(StreamWriter,StreamReader):
encode = codecs.latin_1_decode
decode = codecs.latin_1_encode
### encodings module API
def getregentry():
return codecs.CodecInfo(
name='iso8859-1',
encode=Codec.encode,
decode=Codec.decode,
incrementalencoder=IncrementalEncoder,
incrementaldecoder=IncrementalDecoder,
streamreader=StreamReader,
streamwriter=StreamWriter,
)
crates/vm/src/buffer.rs
+58 -1
View File
@@ -4,7 +4,7 @@ use crate::{
common::{static_cell, str::wchar_t},
convert::ToPyObject,
exceptions,
function::{ArgBytesLike, ArgIntoBool, ArgIntoFloat},
function::{ArgBytesLike, ArgIntoBool, ArgIntoComplex, ArgIntoFloat},
};
use rustpython_common::wtf8::Wtf8Buf;
@@ -13,6 +13,7 @@ use core::{fmt, iter::Peekable, mem};
use half::f16;
use itertools::Itertools;
use malachite_bigint::BigInt;
use num_complex::Complex64;
use num_traits::{PrimInt, ToPrimitive};
use std::os::raw;
@@ -157,6 +158,8 @@ pub(crate) enum FormatType {
Float = b'f',
Double = b'd',
LongDouble = b'g',
FloatComplex = b'F',
DoubleComplex = b'D',
VoidP = b'P',
PyObject = b'O',
}
@@ -221,6 +224,8 @@ impl FormatType {
Self::Float => nonnative_info!(f32, $end),
Self::Double => nonnative_info!(f64, $end),
Self::LongDouble => nonnative_info!(f64, $end), // long double same as double
Self::FloatComplex => nonnative_info!(PackFloatComplex, $end),
Self::DoubleComplex => nonnative_info!(PackDoubleComplex, $end),
Self::PyObject => nonnative_info!(usize, $end), // pointer size
_ => unreachable!(), // size_t or void*
}
@@ -260,6 +265,8 @@ impl FormatType {
Self::Float => native_info!(raw::c_float),
Self::Double => native_info!(raw::c_double),
Self::LongDouble => native_info!(raw::c_double), // long double same as double for now
Self::FloatComplex => native_info!(PackFloatComplex),
Self::DoubleComplex => native_info!(PackDoubleComplex),
Self::VoidP => native_info!(*mut raw::c_void),
Self::PyObject => native_info!(*mut raw::c_void), // pointer to PyObject
},
@@ -624,6 +631,11 @@ impl FormatSpec {
pub const fn size(&self) -> usize {
self.size
}
#[must_use]
pub fn codes_sizeof(&self) -> usize {
core::mem::size_of::<FormatCode>() * (self.codes.len() + 1)
}
}
trait Packable {
@@ -761,6 +773,51 @@ macro_rules! make_pack_float {
make_pack_float!(f32, "f");
make_pack_float!(f64, "d");
#[repr(C)]
struct PackFloatComplex(f32, f32);
#[repr(C)]
struct PackDoubleComplex(f64, f64);
macro_rules! make_pack_complex {
($T:ty, $Elem:ty, $Bits:ty) => {
impl Packable for $T {
fn pack<E: ByteOrder>(
vm: &VirtualMachine,
_code: FormatType,
arg: PyObjectRef,
data: &mut [u8],
) -> Result<(), PackError> {
let c = ArgIntoComplex::try_from_object(vm, arg)
.map_err(|_| {
PackError::new(
PackErrorKind::Type,
vm,
"required argument is not a complex",
)
})?
.into_complex();
let half = size_of::<$Elem>();
(c.re as $Elem).to_bits().pack_int::<E>(&mut data[..half]);
(c.im as $Elem)
.to_bits()
.pack_int::<E>(&mut data[half..half * 2]);
Ok(())
}
fn unpack<E: ByteOrder>(vm: &VirtualMachine, rdr: &[u8]) -> PyObjectRef {
let half = size_of::<$Elem>();
let re = <$Elem>::from_bits(<$Bits>::unpack_int::<E>(&rdr[..half])) as f64;
let im = <$Elem>::from_bits(<$Bits>::unpack_int::<E>(&rdr[half..half * 2])) as f64;
vm.ctx.new_complex(Complex64::new(re, im)).into()
}
}
};
}
make_pack_complex!(PackFloatComplex, f32, u32);
make_pack_complex!(PackDoubleComplex, f64, u64);
impl Packable for f16 {
fn pack<E: ByteOrder>(
vm: &VirtualMachine,
crates/vm/src/builtins/bool.rs
+1 -1
View File
@@ -172,7 +172,7 @@ impl AsNumber for PyBool {
and: Some(|a, b, vm| PyBool::__and__(a, b.to_owned(), vm).to_pyresult(vm)),
xor: Some(|a, b, vm| PyBool::__xor__(a, b.to_owned(), vm).to_pyresult(vm)),
or: Some(|a, b, vm| PyBool::__or__(a, b.to_owned(), vm).to_pyresult(vm)),
..PyInt::AS_NUMBER
..PyNumberMethods::NOT_IMPLEMENTED
};
&AS_NUMBER
}
crates/vm/src/builtins/code.rs
+35 -1
View File
@@ -655,7 +655,9 @@ impl PyCode {
vm: &VirtualMachine,
code: frozen::FrozenCodeObject<B>,
) -> PyRef<Self> {
Self::new_ref_with_bag(vm, code.decode(PyVmBag(vm)))
let py_code = Self::new_ref_with_bag(vm, code.decode(PyVmBag(vm)));
apply_frozen_co_filename(&py_code, vm);
py_code
}
#[cfg(feature = "host_env")]
@@ -703,6 +705,38 @@ impl PyCode {
}
}
fn frozen_co_filename(path: &str) -> Option<&'static str> {
match path {
"_frozen_importlib" => Some("<frozen importlib._bootstrap>"),
"_frozen_importlib_external" => Some("<frozen importlib._bootstrap_external>"),
_ => None,
}
}
fn apply_frozen_co_filename(code: &PyCode, vm: &VirtualMachine) {
let Some(new) = frozen_co_filename(code.source_path().as_str()) else {
return;
};
set_source_path_tree(code, vm.ctx.intern_str(new));
}
fn set_source_path_tree(code: &PyCode, interned: &'static PyStrInterned) {
code.set_source_path(interned);
for constant in code.constants.iter() {
set_source_path_in_const(&constant.0, interned);
}
}
fn set_source_path_in_const(obj: &PyObject, interned: &'static PyStrInterned) {
if let Some(inner) = obj.downcast_ref::<PyCode>() {
set_source_path_tree(inner, interned);
} else if let Some(tup) = obj.downcast_ref::<super::PyTuple>() {
for item in tup {
set_source_path_in_const(item, interned);
}
}
}
impl fmt::Debug for PyCode {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, "code: {:?}", self.code)
crates/vm/src/builtins/descriptor.rs
+164 -1
View File
@@ -1,4 +1,4 @@
use super::{PyStr, PyStrInterned, PyType};
use super::{PyStr, PyStrInterned, PyTuple, PyType};
use crate::{
AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine,
builtins::{PyTypeRef, builtin_func::PyNativeMethod, type_},
@@ -178,6 +178,157 @@ impl Representable for PyMethodDescriptor {
}
}
/// METH_CLASS descriptors. Same layout as method_descriptor; a distinct type.
#[pyclass(name = "classmethod_descriptor", module = false)]
pub struct PyClassMethodDescriptor {
pub common: PyDescriptor,
pub method: &'static PyMethodDef,
pub objclass: &'static Py<PyType>,
pub(crate) _method_def_owner: Option<PyObjectRef>,
}
impl PyClassMethodDescriptor {
pub fn new(method: &'static PyMethodDef, typ: &'static Py<PyType>, ctx: &Context) -> Self {
Self {
common: PyDescriptor {
typ,
name: ctx.intern_str(method.name),
qualname: PyRwLock::new(None),
},
method,
objclass: typ,
_method_def_owner: None,
}
}
pub fn bind(&self, obj: PyObjectRef, ctx: &Context) -> PyRef<PyNativeMethod> {
self.method.build_bound_method(ctx, obj, self.common.typ)
}
}
impl PyPayload for PyClassMethodDescriptor {
fn class(ctx: &Context) -> &'static Py<PyType> {
ctx.types.classmethod_descriptor_type
}
}
impl core::fmt::Debug for PyClassMethodDescriptor {
fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result {
write!(f, "classmethod descriptor for '{}'", self.common.name)
}
}
impl GetDescriptor for PyClassMethodDescriptor {
fn descr_get(
zelf: PyObjectRef,
obj: Option<PyObjectRef>,
cls: Option<PyObjectRef>,
vm: &VirtualMachine,
) -> PyResult {
let descr = Self::_as_pyref(&zelf, vm).unwrap();
let type_obj = match cls {
Some(typ) => typ,
None => match &obj {
Some(o) => o.class().to_owned().into(),
None => {
return Err(vm.new_type_error(format!(
"descriptor '{}' for type '{}' needs either an object or a type",
descr.common.name,
descr.common.typ.name()
)));
}
},
};
if !type_obj.fast_isinstance(vm.ctx.types.type_type) {
return Err(vm.new_type_error(format!(
"descriptor '{}' for type '{}' needs a type, not '{}' as arg 2",
descr.common.name,
descr.common.typ.name(),
type_obj.class().name()
)));
}
let typ = type_obj.downcast::<PyType>().map_err(|obj| {
vm.new_type_error(format!(
"descriptor '{}' for type '{}' needs a type, not '{}' as arg 2",
descr.common.name,
descr.common.typ.name(),
obj.class().name()
))
})?;
if !typ.fast_issubclass(descr.common.typ) {
return Err(vm.new_type_error(format!(
"descriptor '{}' requires a subtype of '{}' but received '{}'",
descr.common.name,
descr.common.typ.name(),
typ.name()
)));
}
Ok(descr.bind(typ.into(), &vm.ctx).into())
}
}
impl Callable for PyClassMethodDescriptor {
type Args = FuncArgs;
fn call(zelf: &Py<Self>, mut args: FuncArgs, vm: &VirtualMachine) -> PyResult {
let Some(owner) = args.args.first().cloned() else {
return Err(vm.new_type_error(format!(
"descriptor '{}' of '{}' object needs an argument",
zelf.method.name,
zelf.common.typ.name()
)));
};
let bound = Self::descr_get(zelf.to_owned().into(), None, Some(owner), vm)?;
args.args.remove(0);
bound.call(args, vm)
}
}
#[pyclass(
with(GetDescriptor, Callable, Representable),
flags(DISALLOW_INSTANTIATION)
)]
impl PyClassMethodDescriptor {
#[pygetset]
const fn __name__(&self) -> &'static PyStrInterned {
self.common.name
}
#[pygetset]
fn __qualname__(&self) -> String {
format!("{}.{}", self.common.typ.name(), self.common.name)
}
#[pygetset]
fn __doc__(&self) -> Option<&'static str> {
let doc = self.method.doc?;
type_::get_doc_from_internal_doc(self.method.name, doc)
}
#[pygetset]
fn __text_signature__(&self) -> Option<String> {
self.method.doc.and_then(|doc| {
type_::get_text_signature_from_internal_doc(self.method.name, doc)
.map(|signature| signature.to_string())
})
}
#[pygetset]
fn __objclass__(&self) -> PyTypeRef {
self.objclass.to_owned()
}
}
impl Representable for PyClassMethodDescriptor {
#[inline]
fn repr_str(zelf: &Py<Self>, _vm: &VirtualMachine) -> PyResult<String> {
Ok(format!(
"<method '{}' of '{}' objects>",
zelf.method.name,
zelf.common.typ.name()
))
}
}
#[derive(Debug)]
pub enum MemberKind {
Object = 6,
@@ -191,6 +342,7 @@ pub(crate) type MemberSetterFunc =
pub enum MemberGetter {
Getter(fn(&VirtualMachine, PyObjectRef) -> PyResult),
Offset(usize),
TupleItem(usize),
}
pub enum MemberSetter {
@@ -211,6 +363,16 @@ impl PyMemberDef {
match self.getter {
MemberGetter::Getter(getter) => (getter)(vm, obj),
MemberGetter::Offset(offset) => get_slot_from_object(&obj, offset, self, vm),
MemberGetter::TupleItem(index) => {
let tuple = obj.downcast_ref::<PyTuple>().ok_or_else(|| {
vm.new_type_error("unexpected payload for struct sequence member")
})?;
tuple
.as_slice()
.get(index)
.cloned()
.ok_or_else(|| vm.new_index_error(format!("tuple index {index} out of range")))
}
}
}
@@ -497,6 +659,7 @@ pub(crate) fn init(ctx: &'static Context) {
.slots
.vectorcall
.store(Some(vectorcall_method_descriptor));
PyClassMethodDescriptor::extend_class(ctx, ctx.types.classmethod_descriptor_type);
PyWrapper::extend_class(ctx, ctx.types.wrapper_descriptor_type);
ctx.types
.wrapper_descriptor_type
crates/vm/src/builtins/dict.rs
+103 -11
View File
@@ -4,6 +4,7 @@ use super::{
};
use crate::common::lock::LazyLock;
use crate::object::{Traverse, TraverseFn};
use crate::stdlib::PyOrderedDictItems;
use crate::{
AsObject, Context, Py, PyExact, PyObject, PyObjectRef, PyPayload, PyRef, PyRefExact, PyResult,
TryFromObject, atomic_func,
@@ -213,7 +214,7 @@ impl PyDict {
Ok((key.clone(), value.clone()))
}
fn update_sequence_pair(
pub(crate) fn update_sequence_pair(
element: PyObjectRef,
index: usize,
vm: &VirtualMachine,
@@ -422,7 +423,7 @@ impl PyDict {
}
#[pymethod]
fn __sizeof__(&self) -> usize {
pub(crate) fn __sizeof__(&self) -> usize {
core::mem::size_of::<Self>() + self.entries.sizeof()
}
@@ -492,8 +493,9 @@ impl PyDict {
}
fn __or__(&self, other: PyObjectRef, vm: &VirtualMachine) -> PyResult {
let other_dict: Result<PyDictRef, _> = other.downcast();
if let Ok(other) = other_dict {
// Accept dict subclasses that inherit this implementation. Subclasses with
// their own reflected slot, such as OrderedDict, take precedence in dispatch.
if let Ok(other) = other.downcast::<Self>() {
let self_cp = self.copy();
self_cp.merge_dict(&other, true, vm)?;
return Ok(self_cp.into_pyobject(vm));
@@ -538,7 +540,7 @@ impl PyDict {
#[pyclass]
impl Py<PyDict> {
fn inner_cmp(
pub(crate) fn inner_cmp(
&self,
other: &Self,
op: PyComparisonOp,
@@ -820,6 +822,38 @@ impl Py<PyDict> {
self.entries.get_known_hash(vm, key, hash)
}
pub(crate) fn set_item_known_hash(
&self,
key: &PyObject,
hash: crate::common::hash::PyHash,
value: PyObjectRef,
vm: &VirtualMachine,
) -> PyResult<()> {
self.entries.insert_known_hash(vm, key, hash, value)
}
pub(crate) fn del_item_known_hash(
&self,
key: &PyObject,
hash: crate::common::hash::PyHash,
vm: &VirtualMachine,
) -> PyResult<()> {
if self.entries.delete_if_exists_known_hash(vm, key, hash)? {
Ok(())
} else {
Err(vm.new_key_error(key.to_owned()))
}
}
pub(crate) fn contains_known_hash(
&self,
key: &PyObject,
hash: crate::common::hash::PyHash,
vm: &VirtualMachine,
) -> PyResult<bool> {
self.entries.contains_known_hash(vm, key, hash)
}
pub fn get_item_opt<K: DictKey + ?Sized>(
&self,
key: &K,
@@ -1663,7 +1697,7 @@ impl AsNumber for PyDictItems {
static AS_NUMBER: PyNumberMethods = PyNumberMethods {
subtract: Some(set_inner_number_subtract),
and: Some(set_inner_number_and),
xor: Some(set_inner_number_xor),
xor: Some(dict_item_view_number_xor),
or: Some(set_inner_number_or),
..PyNumberMethods::NOT_IMPLEMENTED
};
@@ -1704,19 +1738,77 @@ where
Ok(PySet { inner: f(a, b)? }.into_pyobject(vm))
}
fn set_inner_number_subtract(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult {
pub(crate) fn set_inner_number_subtract(
a: &PyObject,
b: &PyObject,
vm: &VirtualMachine,
) -> PyResult {
set_inner_number_op(a, b, |a, b| a.difference(b, vm), vm)
}
fn set_inner_number_and(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult {
set_inner_number_op(a, b, |a, b| a.intersection(b, vm), vm)
pub(crate) fn set_inner_number_and(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult {
let a_is_view =
a.downcast_ref::<PyDictKeys>().is_some() || a.downcast_ref::<PyDictItems>().is_some();
let (view, other) = if a_is_view { (a, b) } else { (b, a) };
set_view_number_and(view, other, vm)
}
fn set_inner_number_xor(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult {
pub(crate) fn set_view_number_and(
view: &PyObject,
other: &PyObject,
vm: &VirtualMachine,
) -> PyResult {
let other = ArgIterable::<PyObjectRef>::try_from_object(vm, other.to_owned())?;
let result = PySet::default();
for item in other.iter(vm)? {
let item = item?;
if view.sequence_unchecked().contains(&item, vm)? {
result.add(item, vm)?;
}
}
Ok(result.into_pyobject(vm))
}
pub(crate) fn set_inner_number_xor(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult {
set_inner_number_op(a, b, |a, b| a.symmetric_difference(b, vm), vm)
}
fn set_inner_number_or(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult {
pub(crate) fn set_item_view_number_xor(
a: &PyObject,
b: &PyObject,
both_are_item_views: bool,
vm: &VirtualMachine,
) -> PyResult {
if !both_are_item_views {
return set_inner_number_xor(a, b, vm);
}
let result = PySet::default();
let a_iterable = ArgIterable::<PyObjectRef>::try_from_object(vm, a.to_owned())?;
for item in a_iterable.iter(vm)? {
let item = item?;
if !b.sequence_unchecked().contains(&item, vm)? {
result.add(item, vm)?;
}
}
let b_iterable = ArgIterable::<PyObjectRef>::try_from_object(vm, b.to_owned())?;
for item in b_iterable.iter(vm)? {
let item = item?;
if !a.sequence_unchecked().contains(&item, vm)? {
result.add(item, vm)?;
}
}
Ok(result.into_pyobject(vm))
}
fn dict_item_view_number_xor(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult {
let is_native_item_view = |obj: &PyObject| {
obj.downcast_ref::<PyDictItems>().is_some()
|| obj.downcast_ref::<PyOrderedDictItems>().is_some()
};
set_item_view_number_xor(a, b, is_native_item_view(a) && is_native_item_view(b), vm)
}
pub(crate) fn set_inner_number_or(a: &PyObject, b: &PyObject, vm: &VirtualMachine) -> PyResult {
set_inner_number_op(a, b, |a, b| a.union(b, vm), vm)
}
crates/vm/src/builtins/filter.rs
+1
View File
@@ -23,6 +23,7 @@ impl PyPayload for PyFilter {
impl Constructor for PyFilter {
type Args = (PyObjectRef, PyIter);
const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true;
fn py_new(
_cls: &Py<PyType>,
crates/vm/src/builtins/float.rs
+19 -2
View File
@@ -176,9 +176,26 @@ impl Constructor for PyFloat {
type Args = OptionalArg<PyObjectRef>;
fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult {
let float_type = vm.ctx.types.float_type;
let uses_float_init = {
let cls_init = cls.slots.init.load().map(crate::types::fn_addr);
let float_init = float_type.slots.init.load().map(crate::types::fn_addr);
cls_init == float_init
};
// Bind before the fast path so FromArgs::arity decides how many arguments
// are acceptable, rather than a count repeated here.
let arg: Self::Args = args.bind_for(vm, Self::NAME)?;
// are acceptable, rather than a count repeated here. Extra keywords are
// accepted only when a subclass has replaced tp_init.
let arg: Self::Args = if cls.is(float_type) || uses_float_init {
args.bind_for(vm, Self::NAME)?
} else {
match args.args.as_slice() {
[] => OptionalArg::Missing,
[value] => OptionalArg::Present(value.clone()),
slice => {
return Err(vm.new_arity_type_error(Self::NAME, 0..=1, slice.len()));
}
}
};
// Optimization: return exact float as-is
if cls.is(vm.ctx.types.float_type)
crates/vm/src/builtins/frame.rs
+20 -21
View File
@@ -523,42 +523,41 @@ impl FrameObject {
}
};
let what_event = vm.what_event.get();
if what_event < 0 {
let Some(what_event) = vm.what_event.get() else {
return Err(
vm.new_value_error("f_lineno can only be set in a trace function".to_owned())
);
}
};
{
use crate::stdlib::sys::monitoring as mon;
use crate::stdlib::sys::monitoring::MonitoringEvent as Ev;
match what_event {
mon::WHAT_PY_RESUME
| mon::WHAT_JUMP
| mon::WHAT_BRANCH
| mon::WHAT_BRANCH_LEFT
| mon::WHAT_BRANCH_RIGHT
| mon::WHAT_LINE
| mon::WHAT_PY_YIELD => {}
mon::WHAT_PY_START => {
Ev::PyResume
| Ev::Jump
| Ev::Branch
| Ev::BranchLeft
| Ev::BranchRight
| Ev::Line
| Ev::PyYield => {}
Ev::PyStart => {
return Err(vm.new_value_error(
"can't jump from the 'call' trace event of a new frame".to_owned(),
));
}
mon::WHAT_CALL | mon::WHAT_C_RETURN => {
Ev::Call | Ev::CReturn => {
return Err(vm.new_value_error("can't jump during a call".to_owned()));
}
mon::WHAT_PY_RETURN
| mon::WHAT_PY_UNWIND
| mon::WHAT_PY_THROW
| mon::WHAT_RAISE
| mon::WHAT_C_RAISE
| mon::WHAT_INSTRUCTION
| mon::WHAT_EXCEPTION_HANDLED => {
Ev::PyReturn
| Ev::PyUnwind
| Ev::PyThrow
| Ev::Raise
| Ev::CRaise
| Ev::Instruction
| Ev::ExceptionHandled => {
return Err(
vm.new_value_error("can only jump from a 'line' trace event".to_owned())
);
}
_ => {
Ev::StopIteration | Ev::Reraise => {
return Err(vm.new_system_error("unexpected event type".to_owned()));
}
}
crates/vm/src/builtins/list.rs
+23 -1
View File
@@ -11,7 +11,7 @@ use crate::object::{Traverse, TraverseFn};
use crate::{
AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult,
builtins::{PyFloat, PyInt, PyStr, PyTuple},
class::PyClassImpl,
class::{PyClassDef, PyClassImpl},
convert::ToPyObject,
function::{ArgSize, Either, FuncArgs, OptionalArg, PyComparisonValue},
protocol::{PyIterReturn, PyMappingMethods, PySequenceMethods},
@@ -484,6 +484,28 @@ impl Constructor for PyList {
impl Initializer for PyList {
type Args = OptionalArg<PyObjectRef>;
fn slot_init(zelf: PyObjectRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult<()> {
let list_type = vm.ctx.types.list_type;
let cls = zelf.class().to_owned();
let uses_list_new = {
let cls_new = cls.slots.new.load().map(crate::types::fn_addr);
let list_new = list_type.slots.new.load().map(crate::types::fn_addr);
cls_new == list_new
};
let iterable = if cls.is(list_type) || uses_list_new {
args.bind_for(vm, Self::NAME)?
} else {
match args.args.as_slice() {
[] => OptionalArg::Missing,
[iterable] => OptionalArg::Present(iterable.clone()),
slice => {
return Err(vm.new_arity_type_error(Self::NAME, 0..=1, slice.len()));
}
}
};
Self::init(zelf.try_into_value(vm)?, iterable, vm)
}
fn init(zelf: PyRef<Self>, iterable: Self::Args, vm: &VirtualMachine) -> PyResult<()> {
let mut elements = if let OptionalArg::Present(iterable) = iterable {
vm.extract_elements_sized(&iterable, &|| 0, Ok)?
crates/vm/src/builtins/set.rs
+14 -4
View File
@@ -655,10 +655,15 @@ impl PySet {
self.inner.len()
}
pub fn __contains__(&self, needle: &PyObject, vm: &VirtualMachine) -> PyResult<bool> {
pub fn contains(&self, needle: &PyObject, vm: &VirtualMachine) -> PyResult<bool> {
self.inner.contains(needle, vm)
}
#[pymethod(coexist)]
fn __contains__(&self, needle: PyObjectRef, vm: &VirtualMachine) -> PyResult<bool> {
self.contains(&needle, vm)
}
#[pymethod]
fn __sizeof__(&self) -> usize {
core::mem::size_of::<Self>() + self.inner.sizeof()
@@ -914,7 +919,7 @@ impl AsSequence for PySet {
static AS_SEQUENCE: LazyLock<PySequenceMethods> = LazyLock::new(|| PySequenceMethods {
length: atomic_func!(|seq, _vm| Ok(PySet::sequence_downcast(seq).__len__())),
contains: atomic_func!(
|seq, needle, vm| PySet::sequence_downcast(seq).__contains__(needle, vm)
|seq, needle, vm| PySet::sequence_downcast(seq).contains(needle, vm)
),
..PySequenceMethods::NOT_IMPLEMENTED
});
@@ -1148,10 +1153,15 @@ impl PyFrozenSet {
self.inner.len()
}
pub fn __contains__(&self, needle: &PyObject, vm: &VirtualMachine) -> PyResult<bool> {
pub fn contains(&self, needle: &PyObject, vm: &VirtualMachine) -> PyResult<bool> {
self.inner.contains(needle, vm)
}
#[pymethod(coexist)]
fn __contains__(&self, needle: PyObjectRef, vm: &VirtualMachine) -> PyResult<bool> {
self.contains(&needle, vm)
}
#[pymethod]
fn __sizeof__(&self) -> usize {
core::mem::size_of::<Self>() + self.inner.sizeof()
@@ -1309,7 +1319,7 @@ impl AsSequence for PyFrozenSet {
static AS_SEQUENCE: LazyLock<PySequenceMethods> = LazyLock::new(|| PySequenceMethods {
length: atomic_func!(|seq, _vm| Ok(PyFrozenSet::sequence_downcast(seq).__len__())),
contains: atomic_func!(
|seq, needle, vm| PyFrozenSet::sequence_downcast(seq).__contains__(needle, vm)
|seq, needle, vm| PyFrozenSet::sequence_downcast(seq).contains(needle, vm)
),
..PySequenceMethods::NOT_IMPLEMENTED
});
crates/vm/src/builtins/str.rs
+5 -1
View File
@@ -1170,7 +1170,11 @@ impl PyStr {
}
#[pymethod]
fn splitlines(&self, args: anystr::SplitLinesArgs, vm: &VirtualMachine) -> Vec<PyObjectRef> {
pub(crate) fn splitlines(
&self,
args: anystr::SplitLinesArgs,
vm: &VirtualMachine,
) -> Vec<PyObjectRef> {
let into_wrapper = |s: &Wtf8| self.new_substr(s.to_owned()).to_pyobject(vm);
let mut elements = Vec::new();
let mut last_i = 0;
crates/vm/src/builtins/tuple.rs
+17 -1
View File
@@ -221,7 +221,23 @@ impl Constructor for PyTuple {
type Args = Vec<PyObjectRef>;
fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult {
let iterable: OptionalArg<PyObjectRef> = args.bind_for(vm, Self::NAME)?;
let tuple_type = vm.ctx.types.tuple_type;
let uses_tuple_init = {
let cls_init = cls.slots.init.load().map(crate::types::fn_addr);
let tuple_init = tuple_type.slots.init.load().map(crate::types::fn_addr);
cls_init == tuple_init
};
let iterable: OptionalArg<PyObjectRef> = if cls.is(tuple_type) || uses_tuple_init {
args.bind_for(vm, Self::NAME)?
} else {
match args.args.as_slice() {
[] => OptionalArg::Missing,
[iterable] => OptionalArg::Present(iterable.clone()),
slice => {
return Err(vm.new_arity_type_error(Self::NAME, 0..=1, slice.len()));
}
}
};
// Optimizations for exact tuple type
if cls.is(vm.ctx.types.tuple_type) {
crates/vm/src/builtins/type.rs
+122 -40
View File
@@ -24,8 +24,9 @@ use crate::{
object::{Traverse, TraverseFn},
protocol::{PyIterReturn, PyNumberMethods},
types::{
AsNumber, Callable, Constructor, GetAttr, Initializer, PyTypeFlags, PyTypeSlots,
Representable, SLOT_DEFS, SetAttr, TypeDataRef, TypeDataRefMut, TypeDataSlot,
AsNumber, Callable, Constructor, GetAttr, Initializer, NewFunc, PyTypeFlags, PyTypeSlots,
Representable, SLOT_DEFS, SetAttr, TypeDataRef, TypeDataRefMut, TypeDataSlot, fn_addr,
new_wrapper,
},
};
use core::{
@@ -244,7 +245,10 @@ unsafe impl crate::object::Traverse for PyType {
if let Some(base) = self.base.deref() {
tracer_fn(base.as_object());
}
tracer_fn(self.bases.read_recursive().as_untyped().as_object());
// Skip when a writer holds `bases` (same rule as `Traverse for PyRwLock`).
if let Some(bases) = self.bases.try_read_recursive() {
tracer_fn(bases.as_untyped().as_object());
}
self.mro.traverse(tracer_fn);
self.subclasses.traverse(tracer_fn);
self.attributes.traverse(tracer_fn);
@@ -259,8 +263,11 @@ unsafe impl crate::object::Traverse for PyType {
if let Some(base) = unsafe { self.base.swap(None) } {
out.push(base.into());
}
// Clone the empty tuple before taking the write lock: `object`'s
// `bases` is this same lock, and `.read()` while exclusive panics
// on CellRwLock (and hangs on parking_lot).
let empty = object::PyBaseObject::static_type().bases.read().clone();
if let Some(mut bases) = self.bases.try_write() {
let empty = object::PyBaseObject::static_type().bases.read().clone();
let old_bases = core::mem::replace(&mut *bases, empty);
out.push(old_bases.into_untyped().into());
}
@@ -410,7 +417,11 @@ pub enum TypeNamespace {
unsafe impl Traverse for TypeNamespace {
fn traverse(&self, tracer_fn: &mut TraverseFn<'_>) {
match self {
Self::Attributes(attrs) => attrs.read_recursive().traverse(tracer_fn),
Self::Attributes(attrs) => {
if let Some(attrs) = attrs.try_read_recursive() {
attrs.traverse(tracer_fn);
}
}
Self::Dict(dict) => tracer_fn(dict.as_object()),
}
}
@@ -474,18 +485,7 @@ impl TypeNamespace {
/// Bind `name` to `value`, dropping whatever it displaced.
pub fn set(&self, name: &'static PyStrInterned, value: PyObjectRef) {
match self {
Self::Attributes(attrs) => {
attrs.write().insert(name, value);
}
Self::Dict(dict) => {
if let Some(Err(_)) | None =
crate::vm::thread::try_with_current_vm(|vm| dict.set_item(name, value, vm))
{
debug_assert!(false, "type namespace write without a running VM");
}
}
}
drop(self.insert(name, value));
}
/// Bind `name` to `value` and hand back what it displaced, so the caller
@@ -495,7 +495,11 @@ impl TypeNamespace {
Self::Attributes(attrs) => attrs.write().insert(name, value),
Self::Dict(dict) => {
let previous = dict_get(dict, name);
self.set(name, value);
if let Some(Err(_)) | None =
crate::vm::thread::try_with_current_vm(|vm| dict.set_item(name, value, vm))
{
debug_assert!(false, "type namespace write without a running VM");
}
previous
}
}
@@ -881,6 +885,16 @@ impl PyType {
PyTypeFlags::from_bits_truncate(self.abc_tpflags.load(Ordering::Acquire)).contains(flag)
}
pub fn set_is_abstract(&self, is_abstract: bool) {
const MASK: u64 = PyTypeFlags::IS_ABSTRACT.bits();
let _ = self
.abc_tpflags
.try_update(Ordering::AcqRel, Ordering::Acquire, |old| {
Some(if is_abstract { old | MASK } else { old & !MASK })
});
self.modified();
}
pub fn set_abc_collection_flags_recursive(&self, flags: PyTypeFlags) {
const COLLECTION_FLAGS: PyTypeFlags = PyTypeFlags::from_bits_truncate(
PyTypeFlags::SEQUENCE.bits() | PyTypeFlags::MAPPING.bits(),
@@ -1115,11 +1129,14 @@ impl PyType {
// Static types are not tracked by GC.
// They are immortal and never participate in collectable cycles.
unsafe {
crate::gc_state::gc_state()
.untrack_object(core::ptr::NonNull::from(new_type.as_object()));
// Heap types from PyType_FromSpec stay tracked.
if !new_type.slots.flags.has_feature(PyTypeFlags::HEAPTYPE) {
unsafe {
crate::gc_state::gc_state()
.untrack_object(core::ptr::NonNull::from(new_type.as_object()));
}
new_type.as_object().clear_gc_tracked();
}
new_type.as_object().clear_gc_tracked();
new_type.mro.write().insert(0, new_type.clone());
@@ -1889,8 +1906,50 @@ impl PyType {
}
#[pygetset]
const fn __flags__(&self) -> u64 {
fn __flags__(&self) -> u64 {
self.slots.flags.bits()
| (self.abc_tpflags.load(Ordering::Acquire) & PyTypeFlags::IS_ABSTRACT.bits())
}
#[pygetset]
fn __abstractmethods__(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult {
if zelf.is(vm.ctx.types.type_type) {
return Err(vm.new_attribute_error("__abstractmethods__"));
}
zelf.get_direct_attr(identifier!(vm, __abstractmethods__))
.ok_or_else(|| vm.new_attribute_error("__abstractmethods__"))
}
#[pygetset(setter)]
fn set___abstractmethods__(&self, value: PySetterValue, vm: &VirtualMachine) -> PyResult<()> {
let key = identifier!(vm, __abstractmethods__);
let is_abstract = match &value {
PySetterValue::Assign(val) => val.try_to_bool(vm)?,
PySetterValue::Delete => false,
};
match value {
PySetterValue::Assign(val) => {
Self::with_type_lock(vm, || {
self.modified_inner();
let _prev = self.attributes.insert(key, val);
self.set_is_abstract(is_abstract);
});
}
PySetterValue::Delete => {
let removed = Self::with_type_lock(vm, || {
self.modified_inner();
let removed = self.attributes.remove(key);
if removed.is_some() {
self.set_is_abstract(false);
}
removed
});
if removed.is_none() {
return Err(vm.new_attribute_error("__abstractmethods__"));
}
}
}
Ok(())
}
#[pygetset]
@@ -2292,11 +2351,22 @@ impl PyType {
}
#[pygetset]
fn __text_signature__(&self) -> Option<String> {
self.slots
.doc
.and_then(|doc| get_text_signature_from_internal_doc(&self.name(), doc))
.map(|signature| signature.to_string())
fn __text_signature__(&self, vm: &VirtualMachine) -> Option<String> {
let name = self.name();
if let Some(doc) = self.slots.doc
&& let Some(signature) = get_text_signature_from_internal_doc(&name, doc)
{
return Some(signature.to_string());
}
if self.slots.flags.has_feature(PyTypeFlags::HEAPTYPE)
&& let Some(doc_attr) = self.get_direct_attr(identifier!(vm, __doc__))
&& let Some(doc) = doc_attr.downcast_ref::<PyStr>()
&& let Some(doc) = doc.to_str()
&& let Some(signature) = get_text_signature_from_internal_doc(&name, doc)
{
return Some(signature.to_string());
}
None
}
#[pygetset]
@@ -2356,6 +2426,17 @@ impl Constructor for PyType {
let (name, bases, dict, kwargs): (PyStrRef, PyTupleRef, PyDictRef, KwArgs) =
args.clone().bind_for(vm, Self::NAME)?;
// A mapping that is not an exact dict (e.g. OrderedDict) keeps its
// own iteration order; copy via the mapping protocol so that order
// lands in the type dict.
let dict = if args.args[2].class().is(vm.ctx.types.dict_type) {
dict
} else {
let copied = vm.ctx.new_dict();
copied.merge_object(args.args[2].clone(), vm)?;
copied
};
if name.as_bytes().contains(&0) {
return Err(vm.new_value_error("type name must not contain null characters"));
}
@@ -3358,24 +3439,25 @@ pub(crate) fn call_slot_new(
return Err(vm.new_type_error(format!("cannot create '{}' instances", subtype.slot_name())));
}
// "is not safe" check (tp_new_wrapper logic)
// Check that the user doesn't do something silly and unsafe like
// object.__new__(dict). To do this, we check that the most derived base
// that's not a heap type is this type.
// "is not safe" check (tp_new_wrapper). Walk past bases whose tp_new is
// new_wrapper so a Python subclass of a native heap type still reaches
// that type's tp_new; reject object.__new__(dict) and similar.
let mut staticbase = subtype.clone();
while staticbase.slots.flags.has_feature(PyTypeFlags::HEAPTYPE) {
if let Some(base) = staticbase.base.to_owned() {
staticbase = base;
} else {
break;
while staticbase
.slots
.new
.load()
.is_some_and(|f| fn_addr(f) == fn_addr(new_wrapper as NewFunc))
{
match staticbase.base.to_owned() {
Some(base) => staticbase = base,
None => break,
}
}
// Check if staticbase's tp_new differs from typ's tp_new
let typ_new = typ.slots.new.load();
let staticbase_new = staticbase.slots.new.load();
if typ_new.map(|f| crate::types::fn_addr(f)) != staticbase_new.map(|f| crate::types::fn_addr(f))
{
if typ_new.map(fn_addr) != staticbase_new.map(fn_addr) {
return Err(vm.new_type_error(format!(
"{}.__new__({}) is not safe, use {}.__new__()",
typ.slot_name(),
crates/vm/src/frame.rs
+46 -39
View File
@@ -29,7 +29,10 @@ use crate::{
protocol::{PyIter, PyIterReturn},
scope::Scope,
sliceable::SliceableSequenceOp,
stdlib::{_typing, builtins, sys::monitoring},
stdlib::{
_typing, builtins,
sys::monitoring::{self, MonitoringEvent},
},
types::{PyComparisonOp, PyTypeFlags},
vm::{Context, PyMethod},
};
@@ -2675,7 +2678,7 @@ pub(crate) fn trampoline_handle_exception(
exec.fire_exception_trace(exception, vm)?;
let exception = {
let mon_events = vm.state.monitoring_events.load();
if mon_events & monitoring::EVENT_RAISE != 0 {
if mon_events & MonitoringEvent::Raise.mask() != 0 {
let offset = idx as u32 * 2;
let exc_obj: PyObjectRef = exception.to_owned().into();
match monitoring::fire_raise(vm, exec.code, offset, &exc_obj) {
@@ -3327,9 +3330,9 @@ impl ExecutingFrame<'_> {
}
// RESUME oparg 0 is PY_START; nonzero is PY_RESUME.
let what = if resume_type == 0 {
monitoring::WHAT_PY_START
monitoring::MonitoringEvent::PyStart
} else {
monitoring::WHAT_PY_RESUME
monitoring::MonitoringEvent::PyResume
};
let trace_result = vm.trace_event_what(crate::protocol::TraceEvent::Call, what, None)?;
if let Some(local_trace) = trace_result {
@@ -3449,7 +3452,7 @@ impl ExecutingFrame<'_> {
let value = self.top_value().to_owned();
vm.trace_event_what(
crate::protocol::TraceEvent::Return,
monitoring::WHAT_PY_YIELD,
monitoring::MonitoringEvent::PyYield,
Some(value),
)?;
if self.lasti() != lasti_before {
@@ -3457,7 +3460,7 @@ impl ExecutingFrame<'_> {
return Ok(true);
}
}
if self.monitoring_mask & monitoring::EVENT_PY_YIELD != 0 {
if self.monitoring_mask & MonitoringEvent::PyYield.mask() != 0 {
let value = self.top_value().to_owned();
let offset = (self.lasti() - 1) * 2;
monitoring::fire_py_yield(vm, self.code, offset, &value)?;
@@ -3852,7 +3855,7 @@ impl ExecutingFrame<'_> {
let exception = {
let mon_events = vm.state.monitoring_events.load();
if is_reraise {
if mon_events & monitoring::EVENT_RERAISE != 0 {
if mon_events & MonitoringEvent::Reraise.mask() != 0 {
let offset = idx as u32 * 2;
let exc_obj: PyObjectRef = exception.clone().into();
match monitoring::fire_reraise(vm, self.code, offset, &exc_obj) {
@@ -3862,7 +3865,7 @@ impl ExecutingFrame<'_> {
} else {
exception
}
} else if mon_events & monitoring::EVENT_RAISE != 0 {
} else if mon_events & MonitoringEvent::Raise.mask() != 0 {
let offset = idx as u32 * 2;
let exc_obj: PyObjectRef = exception.clone().into();
match monitoring::fire_raise(vm, self.code, offset, &exc_obj) {
@@ -3889,7 +3892,7 @@ impl ExecutingFrame<'_> {
Err(exception) => {
// Fire PY_UNWIND: exception escapes this frame
let exception = if vm.state.monitoring_events.load()
& monitoring::EVENT_PY_UNWIND
& MonitoringEvent::PyUnwind.mask()
!= 0
{
let offset = idx as u32 * 2;
@@ -4113,7 +4116,7 @@ impl ExecutingFrame<'_> {
// Fire PY_THROW and RAISE events before raising the exception.
// If a monitoring callback fails, its exception replaces the original.
if vm.use_tracing.get() {
let what = monitoring::WHAT_PY_THROW;
let what = monitoring::MonitoringEvent::PyThrow;
if let Some(local_trace) =
vm.trace_event_what(crate::protocol::TraceEvent::Call, what, None)?
{
@@ -4123,7 +4126,7 @@ impl ExecutingFrame<'_> {
}
let exception = {
let mon_events = vm.state.monitoring_events.load();
let exception = if mon_events & monitoring::EVENT_PY_THROW != 0 {
let exception = if mon_events & MonitoringEvent::PyThrow.mask() != 0 {
let offset = idx as u32 * 2;
let exc_obj: PyObjectRef = exception.clone().into();
match monitoring::fire_py_throw(vm, self.code, offset, &exc_obj) {
@@ -4133,7 +4136,7 @@ impl ExecutingFrame<'_> {
} else {
exception
};
if mon_events & monitoring::EVENT_RAISE != 0 {
if mon_events & MonitoringEvent::Raise.mask() != 0 {
let offset = idx as u32 * 2;
let exc_obj: PyObjectRef = exception.clone().into();
match monitoring::fire_raise(vm, self.code, offset, &exc_obj) {
@@ -4171,7 +4174,7 @@ impl ExecutingFrame<'_> {
Err(exception) => {
// Fire PY_UNWIND: exception escapes the generator frame.
let exception =
if vm.state.monitoring_events.load() & monitoring::EVENT_PY_UNWIND != 0 {
if vm.state.monitoring_events.load() & MonitoringEvent::PyUnwind.mask() != 0 {
let offset = idx as u32 * 2;
let exc_obj: PyObjectRef = exception.clone().into();
match monitoring::fire_py_unwind(vm, self.code, offset, &exc_obj) {
@@ -5256,7 +5259,7 @@ impl ExecutingFrame<'_> {
)));
}
};
if seen_attrs.__contains__(attr_name.as_object(), vm)? {
if seen_attrs.contains(attr_name.as_object(), vm)? {
let type_name = type_name();
let attr_repr = attr_name.as_object().repr(vm)?;
return Err(vm.new_type_error(format!(
@@ -5309,7 +5312,7 @@ impl ExecutingFrame<'_> {
// Extract keyword attributes
for name in kwd_attrs {
let name_str = name.downcast_ref::<PyStr>().unwrap();
if seen_attrs.__contains__(name_str.as_object(), vm)? {
if seen_attrs.contains(name_str.as_object(), vm)? {
let type_name = type_name();
let attr_repr = name.as_object().repr(vm)?;
return Err(vm.new_type_error(format!(
@@ -5364,7 +5367,7 @@ impl ExecutingFrame<'_> {
.new_base_object(vm.ctx.types.object_type.to_owned(), None);
for key in keys {
if seen_keys.__contains__(key.as_object(), vm)? {
if seen_keys.contains(key.as_object(), vm)? {
return Err(vm.new_value_error(format!(
"mapping pattern checks duplicate key ({})",
key.as_object().repr(vm)?
@@ -5386,7 +5389,7 @@ impl ExecutingFrame<'_> {
} else {
// Fallback if .get() method is not available (shouldn't happen for mappings)
for key in keys {
if seen_keys.__contains__(key.as_object(), vm)? {
if seen_keys.contains(key.as_object(), vm)? {
return Err(vm.new_value_error(format!(
"mapping pattern checks duplicate key ({})",
key.as_object().repr(vm)?
@@ -5556,7 +5559,7 @@ impl ExecutingFrame<'_> {
if vm.use_tracing.get() {
vm.trace_event_what(
crate::protocol::TraceEvent::Return,
monitoring::WHAT_PY_RETURN,
monitoring::MonitoringEvent::PyReturn,
Some(value.clone()),
)?;
}
@@ -7876,7 +7879,8 @@ impl ExecutingFrame<'_> {
Ok(PyIterReturn::StopIteration(value)) => {
// FOR_ITER_GEN returns through END_FOR, which
// fires STOP_ITERATION rather than RAISE.
if vm.state.monitoring_events.load() & monitoring::EVENT_STOP_ITERATION
if vm.state.monitoring_events.load()
& MonitoringEvent::StopIteration.mask()
!= 0
{
let offset = (self.lasti() - 1) * 2;
@@ -8015,10 +8019,10 @@ impl ExecutingFrame<'_> {
let resume_type = u32::from(arg);
let offset = (self.lasti() - 1) * 2;
if resume_type == 0 {
if self.monitoring_mask & monitoring::EVENT_PY_START != 0 {
if self.monitoring_mask & MonitoringEvent::PyStart.mask() != 0 {
monitoring::fire_py_start(vm, self.code, offset)?;
}
} else if self.monitoring_mask & monitoring::EVENT_PY_RESUME != 0 {
} else if self.monitoring_mask & MonitoringEvent::PyResume.mask() != 0 {
monitoring::fire_py_resume(vm, self.code, offset)?;
}
self.trace_call_from_resume(vm, resume_type)?;
@@ -8029,11 +8033,11 @@ impl ExecutingFrame<'_> {
if vm.use_tracing.get() {
vm.trace_event_what(
crate::protocol::TraceEvent::Return,
monitoring::WHAT_PY_RETURN,
monitoring::MonitoringEvent::PyReturn,
Some(value.clone()),
)?;
}
if self.monitoring_mask & monitoring::EVENT_PY_RETURN != 0 {
if self.monitoring_mask & MonitoringEvent::PyReturn.mask() != 0 {
let offset = (self.lasti() - 1) * 2;
monitoring::fire_py_return(vm, self.code, offset, &value)?;
}
@@ -8070,7 +8074,7 @@ impl ExecutingFrame<'_> {
let oparg = bytecode::LoadSuperAttr::from(u32::from(arg));
let offset = (self.lasti() - 1) * 2;
// Fire CALL event before super() call
let call_args = if self.monitoring_mask & monitoring::EVENT_CALL != 0 {
let call_args = if self.monitoring_mask & MonitoringEvent::Call.mask() != 0 {
let global_super: PyObjectRef = self.nth_value(2).to_owned();
let arg0 = if oparg.has_class() {
self.nth_value(1).to_owned()
@@ -8115,7 +8119,7 @@ impl ExecutingFrame<'_> {
let target_idx = self.lasti() + u32::from(arg);
let target = bytecode::Label::from_u32(target_idx);
self.jump(target);
if self.monitoring_mask & monitoring::EVENT_JUMP != 0 {
if self.monitoring_mask & MonitoringEvent::Jump.mask() != 0 {
monitoring::fire_jump(vm, self.code, src_offset, target.as_u32() * 2)?;
}
Ok(None)
@@ -8126,7 +8130,7 @@ impl ExecutingFrame<'_> {
let target_idx = self.lasti() + 1 - u32::from(arg);
let target = bytecode::Label::from_u32(target_idx);
self.jump(target);
if self.monitoring_mask & monitoring::EVENT_JUMP != 0 {
if self.monitoring_mask & MonitoringEvent::Jump.mask() != 0 {
monitoring::fire_jump(vm, self.code, src_offset, target.as_u32() * 2)?;
}
self.trace_backward_same_line(from_idx, vm)?;
@@ -8137,11 +8141,11 @@ impl ExecutingFrame<'_> {
let target = bytecode::Label::from_u32(self.lasti() + 1 + u32::from(arg));
let continued = self.execute_for_iter(vm, target)?;
if continued {
if self.monitoring_mask & monitoring::EVENT_BRANCH_LEFT != 0 {
if self.monitoring_mask & MonitoringEvent::BranchLeft.mask() != 0 {
let dest_offset = (self.lasti() + 1) * 2; // after caches
monitoring::fire_branch_left(vm, self.code, src_offset, dest_offset)?;
}
} else if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 {
} else if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 {
// INSTRUMENTED_POP_ITER: dest is the instruction after
// POP_ITER (FOR_ITER jumps over END_FOR onto POP_ITER).
let dest_offset = (self.lasti() + 1) * 2;
@@ -8157,7 +8161,7 @@ impl ExecutingFrame<'_> {
.downcast_ref::<crate::builtins::PyGenerator>()
.is_some();
let value = self.pop_value();
if is_gen && self.monitoring_mask & monitoring::EVENT_STOP_ITERATION != 0 {
if is_gen && self.monitoring_mask & MonitoringEvent::StopIteration.mask() != 0 {
let offset = (self.lasti() - 1) * 2;
monitoring::fire_stop_iteration(vm, self.code, offset, &value)?;
}
@@ -8173,7 +8177,9 @@ impl ExecutingFrame<'_> {
|| receiver
.downcast_ref::<crate::builtins::PyCoroutine>()
.is_some();
if is_gen_or_coro && self.monitoring_mask & monitoring::EVENT_STOP_ITERATION != 0 {
if is_gen_or_coro
&& self.monitoring_mask & MonitoringEvent::StopIteration.mask() != 0
{
let offset = (self.lasti() - 1) * 2;
monitoring::fire_stop_iteration(vm, self.code, offset, &value)?;
}
@@ -8187,7 +8193,7 @@ impl ExecutingFrame<'_> {
let value = obj.try_to_bool(vm)?;
if value {
self.jump(bytecode::Label::from_u32(target_idx));
if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 {
if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 {
monitoring::fire_branch_right(vm, self.code, src_offset, target_idx * 2)?;
}
}
@@ -8200,7 +8206,7 @@ impl ExecutingFrame<'_> {
let value = obj.try_to_bool(vm)?;
if !value {
self.jump(bytecode::Label::from_u32(target_idx));
if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 {
if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 {
monitoring::fire_branch_right(vm, self.code, src_offset, target_idx * 2)?;
}
}
@@ -8212,7 +8218,7 @@ impl ExecutingFrame<'_> {
let value = self.pop_value();
if vm.is_none(&value) {
self.jump(bytecode::Label::from_u32(target_idx));
if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 {
if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 {
monitoring::fire_branch_right(vm, self.code, src_offset, target_idx * 2)?;
}
}
@@ -8224,14 +8230,14 @@ impl ExecutingFrame<'_> {
let value = self.pop_value();
if !vm.is_none(&value) {
self.jump(bytecode::Label::from_u32(target_idx));
if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 {
if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 {
monitoring::fire_branch_right(vm, self.code, src_offset, target_idx * 2)?;
}
}
Ok(None)
}
Instruction::InstrumentedNotTaken => {
if self.monitoring_mask & monitoring::EVENT_BRANCH_LEFT != 0 {
if self.monitoring_mask & MonitoringEvent::BranchLeft.mask() != 0 {
let not_taken_idx = self.lasti() as usize - 1;
// Scan backwards past CACHE entries to find the branch instruction
let mut branch_idx = not_taken_idx.saturating_sub(1);
@@ -8255,7 +8261,7 @@ impl ExecutingFrame<'_> {
Ok(None)
}
Instruction::InstrumentedEndAsyncFor => {
if self.monitoring_mask & monitoring::EVENT_BRANCH_RIGHT != 0 {
if self.monitoring_mask & MonitoringEvent::BranchRight.mask() != 0 {
let oparg_val = u32::from(arg);
// src = next_instr - oparg (END_SEND position)
let src_offset = (self.lasti() - oparg_val) * 2;
@@ -8607,7 +8613,8 @@ impl ExecutingFrame<'_> {
// Fire EXCEPTION_HANDLED before setting up handler.
// If the callback raises, the handler is NOT set up and the
// new exception propagates instead.
if vm.state.monitoring_events.load() & monitoring::EVENT_EXCEPTION_HANDLED != 0
if vm.state.monitoring_events.load() & MonitoringEvent::ExceptionHandled.mask()
!= 0
{
let byte_offset = offset * 2;
let exc_obj: PyObjectRef = exception.clone().into();
@@ -9011,7 +9018,7 @@ impl ExecutingFrame<'_> {
let is_python_call = callable.downcast_ref_if_exact::<PyFunction>(vm).is_some();
// Fire CALL event
let call_arg0 = if self.monitoring_mask & monitoring::EVENT_CALL != 0 {
let call_arg0 = if self.monitoring_mask & MonitoringEvent::Call.mask() != 0 {
let arg0 = final_args
.args
.first()
@@ -9324,7 +9331,7 @@ impl ExecutingFrame<'_> {
{
return Ok(());
}
let need_raise = vm.state.monitoring_events.load() & monitoring::EVENT_RAISE != 0;
let need_raise = vm.state.monitoring_events.load() & MonitoringEvent::Raise.mask() != 0;
let need_trace = vm.use_tracing.get() && self.trace_is_set(vm);
if !need_raise && !need_trace {
return Ok(());
crates/vm/src/function/method.rs
+4 -7
View File
@@ -3,7 +3,7 @@ use crate::{
builtins::{
PyType,
builtin_func::{PyNativeFunction, PyNativeMethod},
descriptor::PyMethodDescriptor,
descriptor::{PyClassMethodDescriptor, PyMethodDescriptor},
},
class::PyClassDef,
function::{IntoPyNativeFn, PyNativeFn},
@@ -202,12 +202,9 @@ impl PyMethodDef {
&'static self,
ctx: &Context,
class: &'static Py<PyType>,
) -> PyRef<PyMethodDescriptor> {
PyRef::new_ref(
self.to_method(class, ctx),
ctx.types.method_descriptor_type.to_owned(),
None,
)
) -> PyRef<PyClassMethodDescriptor> {
debug_assert!(self.flags.contains(PyMethodFlags::CLASS));
PyClassMethodDescriptor::new(self, class, ctx).into_ref(ctx)
}
pub fn build_staticmethod(
crates/vm/src/import.rs
+5 -1
View File
@@ -233,7 +233,11 @@ fn remove_importlib_frames_inner(
let (inner_tb, mut now_in_importlib) =
remove_importlib_frames_inner(vm, traceback.next.lock().clone(), always_trim);
if file_name == "_frozen_importlib" || file_name == "_frozen_importlib_external" {
if file_name == "<frozen importlib._bootstrap>"
|| file_name == "<frozen importlib._bootstrap_external>"
|| file_name == "_frozen_importlib"
|| file_name == "_frozen_importlib_external"
{
if traceback.frame.iframe().code().obj_name.as_str() == "_call_with_frames_removed" {
now_in_importlib = true;
}
crates/vm/src/intern.rs
+4 -4
View File
@@ -54,8 +54,9 @@ impl StringPool {
#[cold]
fn miss(zelf: &StringPool, s: PyRefExact<PyStr>) -> &'static PyStrInterned {
let cache = CachedPyStrRef { inner: s };
let inserted = zelf.inner.write().insert(cache.clone());
if inserted {
let mut inner = zelf.inner.write();
if inner.insert(cache.clone()) {
drop(inner);
let interned = unsafe { cache.as_interned_str() };
// `mark_intern` also makes the object immortal: the pool
// never gives an entry up and its refcount could already never
@@ -67,8 +68,7 @@ impl StringPool {
interned
} else {
unsafe {
zelf.inner
.read()
inner
.get(cache.as_ref())
.expect("inserted is false")
.as_interned_str()
crates/vm/src/protocol/callable.rs
+12 -12
View File
@@ -179,17 +179,17 @@ impl TraceEvent {
/// Default `what_event` for this legacy event.
#[must_use]
const fn default_what(self) -> i32 {
use crate::stdlib::sys::monitoring as mon;
const fn default_what(self) -> crate::stdlib::sys::monitoring::MonitoringEvent {
use crate::stdlib::sys::monitoring::MonitoringEvent as Ev;
match self {
Self::Call => mon::WHAT_PY_START,
Self::Return => mon::WHAT_PY_RETURN,
Self::Exception => mon::WHAT_RAISE,
Self::Line => mon::WHAT_LINE,
Self::Opcode => mon::WHAT_INSTRUCTION,
Self::CCall => mon::WHAT_CALL,
Self::CReturn => mon::WHAT_C_RETURN,
Self::CException => mon::WHAT_C_RAISE,
Self::Call => Ev::PyStart,
Self::Return => Ev::PyReturn,
Self::Exception => Ev::Raise,
Self::Line => Ev::Line,
Self::Opcode => Ev::Instruction,
Self::CCall => Ev::Call,
Self::CReturn => Ev::CReturn,
Self::CException => Ev::CRaise,
}
}
}
@@ -235,11 +235,11 @@ impl VirtualMachine {
pub(crate) fn trace_event_what(
&self,
event: TraceEvent,
what: i32,
what: crate::stdlib::sys::monitoring::MonitoringEvent,
arg: Option<PyObjectRef>,
) -> PyResult<Option<PyObjectRef>> {
if self.use_tracing.get() && !self.tracing_is_suppressed() {
let old = self.what_event.replace(what);
let old = self.what_event.replace(Some(what));
let result = self._trace_event_inner(event, arg);
self.what_event.set(old);
result
crates/vm/src/signal.rs
+65 -51
View File
@@ -13,13 +13,40 @@ use crate::{PyObjectRef, PyResult, TryFromBorrowedObject, TryFromObject, Virtual
pub(crate) const NSIG: usize = 64;
/// Eval-breaker word: bit flags checked once per bytecode instruction.
#[cfg(not(feature = "threading"))]
bitflagset::bitflag! {
/// Eval-breaker bits checked once per bytecode instruction.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[repr(u8)]
enum EvalBreakerFlag {
Signal = 0,
}
}
#[cfg(feature = "threading")]
bitflagset::bitflag! {
/// Eval-breaker bits checked once per bytecode instruction.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[repr(u8)]
enum EvalBreakerFlag {
Signal = 0,
Qsbr = 1,
Gc = 2,
Stop = 3,
Finalizing = 4,
}
}
bitflagset::bitflagset! {
#[derive(Copy, Clone, PartialEq, Eq)]
struct EvalBreakerBits(u8): EvalBreakerFlag
}
bitflagset::atomic_bitflagset!(struct EvalBreaker(AtomicU8) on EvalBreakerBits);
/// Signal handlers and QSBR set bits with fetch_or (async-signal-safe,
/// lock-free); consumers clear only their own bit with fetch_and.
static EVAL_BREAKER: AtomicU8 = AtomicU8::new(0);
/// A signal handler recorded a pending signal.
const SIGNAL_BIT: u8 = 1 << 0;
static EVAL_BREAKER: EvalBreaker = EvalBreaker::new();
#[expect(
clippy::declare_interior_mutable_const,
@@ -55,12 +82,12 @@ pub fn check_signals(vm: &VirtualMachine) -> PyResult<()> {
// Read-only check first: avoids cache-line invalidation on every
// instruction when no signal is pending (the common case).
if EVAL_BREAKER.load(Ordering::Relaxed) & SIGNAL_BIT == 0 {
if !EVAL_BREAKER.contains(&EvalBreakerFlag::Signal) {
return Ok(());
}
// Atomic RMW only when a signal is actually pending.
if EVAL_BREAKER.fetch_and(!SIGNAL_BIT, Ordering::Acquire) & SIGNAL_BIT == 0 {
if !EVAL_BREAKER.remove(EvalBreakerFlag::Signal) {
return Ok(());
}
@@ -115,86 +142,70 @@ fn trigger_signals(vm: &VirtualMachine) -> PyResult<()> {
pub(crate) fn set_triggered() {
// fetch_or (not store) so a signal handler never clobbers the QSBR bit;
// this compiles to a lock-free RMW, safe to call from a signal handler.
EVAL_BREAKER.fetch_or(SIGNAL_BIT, Ordering::Release);
EVAL_BREAKER.insert(EvalBreakerFlag::Signal);
}
/// Any eval-breaker bit pending? One relaxed load; checked per instruction.
#[inline(always)]
pub(crate) fn eval_breaker_pending() -> bool {
EVAL_BREAKER.load(Ordering::Relaxed) != 0
!EVAL_BREAKER.is_empty()
}
/// Extra eval-breaker bits used only when more than one thread can run.
#[cfg(feature = "threading")]
mod mt {
use super::EVAL_BREAKER;
use core::sync::atomic::Ordering;
use super::{EVAL_BREAKER, EvalBreakerFlag};
/// QSBR has retired allocations pending reclamation.
const QSBR_BIT: u8 = 1 << 1;
/// An automatic collection was scheduled by `maybe_collect` and must run
/// at the next bytecode safepoint rather than synchronously inside the
/// allocation that tripped the threshold.
const GC_BIT: u8 = 1 << 2;
/// At least one thread's `stop_requested` flag may be set (stop-the-world
/// in progress). This bit is shared by every thread rather than being
/// per-thread: the fast (common) path checked once per bytecode
/// instruction becomes a single relaxed load of this word instead of a
/// thread-local lookup plus atomic load of that thread's own flag. Once
/// set, the bit is sticky until `start_the_world`/`reset_after_fork`
/// clears it (both run under the single stop-the-world exclusion, so
/// clearing cannot race a new requester) — every thread takes the slow
/// path for the (short) duration of a stop-the-world request, which only
/// affects programs with more than one live thread; single-threaded
/// programs never set this bit at all, since `stop_the_world` never sets
/// any per-thread `stop_requested` flag when there are no other threads
/// to stop.
const STOP_BIT: u8 = 1 << 3;
/// The interpreter has begun finalizing (see `Interpreter::finalize`).
/// Set once, near process/interpreter shutdown, and never cleared — from
/// that point on every thread takes the slow path once per instruction,
/// which is fine because finalization is a one-time, short-lived,
/// non-performance-sensitive window.
const FINALIZING_BIT: u8 = 1 << 4;
pub(crate) fn set_qsbr_bit() {
EVAL_BREAKER.fetch_or(QSBR_BIT, Ordering::Release);
EVAL_BREAKER.insert(EvalBreakerFlag::Qsbr);
}
pub(crate) fn clear_qsbr_bit() {
EVAL_BREAKER.fetch_and(!QSBR_BIT, Ordering::Release);
EVAL_BREAKER.remove(EvalBreakerFlag::Qsbr);
}
pub(crate) fn qsbr_bit_set() -> bool {
EVAL_BREAKER.load(Ordering::Relaxed) & QSBR_BIT != 0
EVAL_BREAKER.contains(&EvalBreakerFlag::Qsbr)
}
/// Record that some thread's `stop_requested` flag was (or may have been)
/// set. See `STOP_BIT`.
/// Record that some thread's `stop_requested` flag may be set. Shared by
/// every thread rather than being per-thread: the fast path checked once
/// per bytecode instruction becomes a single relaxed load of this word.
/// Sticky until `start_the_world`/`reset_after_fork` clears it.
pub(crate) fn set_stop_bit() {
EVAL_BREAKER.fetch_or(STOP_BIT, Ordering::Release);
EVAL_BREAKER.insert(EvalBreakerFlag::Stop);
}
/// Clear the shared stop-the-world bit. Only safe to call from
/// `start_the_world`/`reset_after_fork`, which run under the single
/// stop-the-world exclusion (see `STOP_BIT`).
/// `start_the_world`/`reset_after_fork`.
pub(crate) fn clear_stop_bit() {
EVAL_BREAKER.fetch_and(!STOP_BIT, Ordering::Release);
EVAL_BREAKER.remove(EvalBreakerFlag::Stop);
}
/// Record that finalization has begun. See `FINALIZING_BIT`.
/// Record that finalization has begun (`Interpreter::finalize`). Set once
/// and never cleared.
pub(crate) fn set_finalizing_bit() {
EVAL_BREAKER.fetch_or(FINALIZING_BIT, Ordering::Release);
EVAL_BREAKER.insert(EvalBreakerFlag::Finalizing);
}
/// Schedule an automatic collection to run at the next bytecode safepoint.
pub(crate) fn schedule_gc() {
EVAL_BREAKER.fetch_or(GC_BIT, Ordering::Release);
EVAL_BREAKER.insert(EvalBreakerFlag::Gc);
}
/// Clear the scheduled-GC bit, returning whether it had been set.
pub(crate) fn take_gc_scheduled() -> bool {
EVAL_BREAKER.fetch_and(!GC_BIT, Ordering::Acquire) & GC_BIT != 0
EVAL_BREAKER.remove(EvalBreakerFlag::Gc)
}
/// Drop every process-wide eval-breaker bit. Tests that assert a single
/// thread's `stop_requested` must not trip `eval_breaker_pending` have to
/// start from a clean word: cargo's Windows runner shares the process
/// across `#[test]` functions, so a sibling can leave SIGNAL/QSBR/GC/STOP.
#[cfg(test)]
pub(crate) fn clear_eval_breaker_for_test() {
EVAL_BREAKER.clear();
}
}
@@ -204,11 +215,14 @@ pub(crate) use mt::{
set_stop_bit, take_gc_scheduled,
};
#[cfg(all(test, feature = "threading"))]
pub(crate) use mt::clear_eval_breaker_for_test;
/// Reset all signal trigger state after fork in child process.
/// Stale triggers from the parent must not fire in the child.
#[cfg(all(unix, feature = "host_env"))]
pub(crate) fn clear_after_fork() {
EVAL_BREAKER.fetch_and(!SIGNAL_BIT, Ordering::Release);
EVAL_BREAKER.remove(EvalBreakerFlag::Signal);
for trigger in &TRIGGERS {
trigger.store(false, Ordering::Relaxed);
}
crates/vm/src/stdlib/_abc.rs
+5
View File
@@ -206,6 +206,11 @@ mod _abc {
let data = AbcData::new();
cls.set_attr("_abc_impl", data.to_pyobject(vm), vm)?;
if let Some(cls_type) = cls.downcast_ref::<PyType>() {
let abstracts = cls.get_attr("__abstractmethods__", vm)?;
cls_type.set_is_abstract(abstracts.try_to_bool(vm)?);
}
Ok(())
}
crates/vm/src/stdlib/_collections.rs
+30 -16
View File
@@ -1,6 +1,10 @@
// cspell:ignore odict
pub(crate) use _collections::module_def;
#[pymodule]
pub(crate) mod ordered_dict;
#[pymodule(with(ordered_dict::ordered_dict))]
mod _collections {
use crate::{
AsObject, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine,
@@ -12,7 +16,7 @@ mod _collections {
},
common::lock::{PyMutex, PyRwLock, PyRwLockReadGuard, PyRwLockWriteGuard},
convert::ToPyObject,
function::{FuncArgs, KwArgs, OptionalArg, PyComparisonValue},
function::{FuncArgs, KwArgs, OptionalArg, PyComparisonValue, PySetterValue},
object::{Traverse, TraverseFn},
protocol::{PyIterReturn, PyMappingMethods, PyNumberMethods, PySequenceMethods},
recursion::ReprGuard,
@@ -885,23 +889,33 @@ mod _collections {
flags(BASETYPE, MAPPING, HAS_DICT)
)]
impl PyDefaultDict {
#[pygetset]
fn default_factory(&self) -> Option<PyObjectRef> {
self.default_factory.read().clone()
#[pymember(type = "object")]
fn default_factory(vm: &VirtualMachine, zelf: PyObjectRef) -> PyResult {
let zelf: &Py<Self> = zelf.try_to_value(vm)?;
Ok(zelf
.default_factory
.read()
.clone()
.unwrap_or_else(|| vm.ctx.none()))
}
#[pygetset(name = "default_factory", setter)]
fn default_factory_setter(&self, value: PyObjectRef, vm: &VirtualMachine) {
*self.default_factory.write() = if value.is(&vm.ctx.none()) {
None
} else {
Some(value)
#[pymember(type = "object", setter)]
fn set_default_factory(
vm: &VirtualMachine,
zelf: PyObjectRef,
value: PySetterValue,
) -> PyResult<()> {
let zelf: &Py<Self> = zelf.try_to_value(vm)?;
*zelf.default_factory.write() = match value {
PySetterValue::Assign(v) if !v.is(&vm.ctx.none()) => Some(v),
_ => None,
};
Ok(())
}
#[pymethod]
fn __missing__(&self, key: PyObjectRef, vm: &VirtualMachine) -> PyResult {
let factory = self.default_factory();
let factory = self.default_factory.read().clone();
if let Some(f) = factory {
let value = f.call((), vm)?;
@@ -914,7 +928,7 @@ mod _collections {
#[pymethod]
#[pymethod(name = "__copy__")]
fn copy(&self) -> Self {
let default_factory = self.default_factory();
let default_factory = self.default_factory.read().clone();
Self {
dict: self.dict.copy(),
@@ -926,7 +940,7 @@ mod _collections {
fn __reduce__(zelf: PyRef<Self>, vm: &VirtualMachine) -> PyResult {
let cls = zelf.class().to_owned();
let default_factory = zelf.default_factory();
let default_factory = zelf.default_factory.read().clone();
let factory_tuple_elements =
default_factory.map_or_else(Vec::new, |factory| vec![factory]);
let factory_tuple = vm.ctx.new_tuple(factory_tuple_elements);
@@ -958,13 +972,13 @@ mod _collections {
return not_implemented();
}
(zelf.default_factory(), zelf.dict.copy())
(zelf.default_factory.read().clone(), zelf.dict.copy())
} else if let Some(zelf) = rhs.downcast_ref::<Self>() {
let Some(dict) = lhs.downcast_ref::<PyDict>() else {
return not_implemented();
};
(zelf.default_factory(), dict.copy())
(zelf.default_factory.read().clone(), dict.copy())
} else {
return Err(vm.new_type_error(format!(
"unsupported operand type(s) for |: '{}' and '{}'",
crates/vm/src/stdlib/_collections/ordered_dict.rs
+1237
View File
File diff suppressed because it is too large. Load diff
crates/vm/src/stdlib/_functools.rs
+127 -14
View File
@@ -7,13 +7,13 @@ mod _functools {
builtins::{
PyBoundMethod, PyDict, PyDictRef, PyGenericAlias, PyTuple, PyType, PyTypeRef, object,
},
common::lock::PyRwLock,
function::{FuncArgs, KwArgs, OptionalOption, PySetterValue},
common::{hash::PyHash, lock::PyRwLock},
function::{Either, FuncArgs, KwArgs, OptionalOption, PyComparisonValue, PySetterValue},
object::AsObject,
protocol::PyIter,
pyclass,
recursion::ReprGuard,
types::{Callable, Constructor, GetDescriptor, Representable},
types::{Callable, Constructor, GetDescriptor, PyComparisonOp, Representable},
};
use core::sync::atomic::{AtomicU64, Ordering};
use parking_lot::lock_api::RawReentrantMutex as GenericRawReentrantMutex;
@@ -62,6 +62,95 @@ mod _functools {
Ok(accumulator)
}
#[derive(FromArgs)]
struct CmpToKeyArgs {
mycmp: PyObjectRef,
}
#[pyfunction]
fn cmp_to_key(args: CmpToKeyArgs) -> PyKeyWrapper {
PyKeyWrapper {
cmp: args.mycmp,
object: PyRwLock::new(None),
}
}
#[derive(FromArgs)]
struct KeyWrapperCallArgs {
obj: PyObjectRef,
}
#[pyclass(no_attr, name = "KeyWrapper", module = "functools", unhashable = true)]
#[derive(Debug, PyPayload)]
struct PyKeyWrapper {
cmp: PyObjectRef,
object: PyRwLock<Option<PyObjectRef>>,
}
#[pyclass(with(Callable), flags(IMMUTABLETYPE, DISALLOW_INSTANTIATION))]
impl PyKeyWrapper {
#[pygetset]
fn obj(&self, vm: &VirtualMachine) -> PyObjectRef {
self.object.read().clone().unwrap_or_else(|| vm.ctx.none())
}
#[pygetset(setter)]
fn set_obj(&self, value: PySetterValue) {
*self.object.write() = match value {
PySetterValue::Assign(v) => Some(v),
PySetterValue::Delete => None,
};
}
#[pygetset]
fn __text_signature__(&self) -> &'static str {
"(obj)"
}
#[pyslot]
fn slot_richcompare(
zelf: &PyObject,
other: &PyObject,
op: PyComparisonOp,
vm: &VirtualMachine,
) -> PyResult<Either<PyObjectRef, PyComparisonValue>> {
let Some(zelf) = zelf.downcast_ref::<Self>() else {
return Err(vm.new_type_error(format!(
"unexpected payload for {}",
op.method_name(&vm.ctx).as_str()
)));
};
let Some(other) = other.downcast_ref::<Self>() else {
return Err(vm.new_type_error("other argument must be K instance"));
};
let x = zelf
.object
.read()
.clone()
.ok_or_else(|| vm.new_attribute_error("object"))?;
let y = other
.object
.read()
.clone()
.ok_or_else(|| vm.new_attribute_error("object"))?;
let res = zelf.cmp.call((x, y), vm)?;
res.rich_compare(vm.ctx.new_int(0).into(), op, vm)
.map(Either::A)
}
}
impl Callable for PyKeyWrapper {
type Args = KeyWrapperCallArgs;
fn call(zelf: &Py<Self>, args: Self::Args, vm: &VirtualMachine) -> PyResult {
Ok(Self {
cmp: zelf.cmp.clone(),
object: PyRwLock::new(Some(args.obj)),
}
.into_pyobject(vm))
}
}
// Placeholder singleton for partial arguments
// The singleton is stored as _instance on the type class
#[pyattr]
@@ -558,6 +647,16 @@ mod _functools {
}
}
#[pyclass(no_attr, name = "_lru_list_elem", module = "functools")]
#[derive(Debug, PyPayload)]
struct PyLruListElem {
hash: PyHash,
result: PyObjectRef,
}
#[pyclass(flags(IMMUTABLETYPE, DISALLOW_INSTANTIATION))]
impl PyLruListElem {}
/// Native implementation of `functools._lru_cache_wrapper`, mirroring CPython's
/// `_functools` accelerator so `functools.lru_cache` doesn't fall back to the much
/// slower pure-Python implementation in `Lib/functools.py`.
@@ -661,16 +760,17 @@ mod _functools {
}
/// Refresh `key`'s recency by moving it to the end of `cache`'s insertion
/// order (removing then reinserting it). `cache` itself provides the
/// hashing/equality, so this needs no separate key-comparison machinery.
/// order (removing then reinserting it). `hash` is the already-computed
/// hash of `key`, so `__hash__` is not called again.
fn touch_key(
key: &PyObject,
hash: PyHash,
value: &PyObject,
cache: &Py<PyDict>,
vm: &VirtualMachine,
) -> PyResult<()> {
cache.del_item(key.as_object(), vm)?;
cache.set_item(key.as_object(), value.to_owned(), vm)?;
cache.del_item_known_hash(key, hash, vm)?;
cache.set_item_known_hash(key, hash, value.to_owned(), vm)?;
Ok(())
}
@@ -681,8 +781,11 @@ mod _functools {
return Ok(());
}
let oldest = cache.into_iter().next();
if let Some((oldest_key, _)) = oldest {
cache.del_item(oldest_key.as_object(), vm)?;
if let Some((oldest_key, oldest_val)) = oldest {
let Some(elem) = oldest_val.downcast_ref::<PyLruListElem>() else {
return Err(vm.new_type_error("lru cache entry is corrupted"));
};
cache.del_item_known_hash(oldest_key.as_object(), elem.hash, vm)?;
}
Ok(())
}
@@ -743,16 +846,21 @@ mod _functools {
}
let key = zelf.make_key(&args, vm);
let hash = key.hash(vm)?;
{
let _guard = RMutexGuard::acquire(&zelf.lock);
let cache = zelf.cache.read().clone();
if let Some(value) = cache.get_item_opt(key.as_object(), vm)? {
if let Some(value) = cache.get_item_known_hash(key.as_object(), hash, vm)? {
let Some(elem) = value.downcast_ref::<PyLruListElem>() else {
return Err(vm.new_type_error("lru cache entry is corrupted"));
};
let result = elem.result.clone();
zelf.hits.fetch_add(1, Ordering::Relaxed);
if zelf.maxsize.is_some() {
Self::touch_key(&key, &value, &cache, vm)?;
Self::touch_key(&key, hash, &value, &cache, vm)?;
}
return Ok(value);
return Ok(result);
}
zelf.misses.fetch_add(1, Ordering::Relaxed);
// Lock released here (guard dropped) before calling into user code, so
@@ -768,8 +876,13 @@ mod _functools {
// (e.g. recursive lookups, or another thread finishing first); in that
// case keep the existing entry instead of creating orphan eviction
// links (see CPython issue gh-35780).
if !cache.contains_key(key.as_object(), vm) {
cache.set_item(key.as_object(), result.clone(), vm)?;
if !cache.contains_known_hash(key.as_object(), hash, vm)? {
let entry = PyLruListElem {
hash,
result: result.clone(),
}
.into_pyobject(vm);
cache.set_item_known_hash(key.as_object(), hash, entry, vm)?;
if let Some(maxsize) = zelf.maxsize {
Self::evict_if_full(maxsize, &cache, vm)?;
}
crates/vm/src/stdlib/_interpchannels.rs
+70 -43
View File
@@ -16,7 +16,7 @@ pub(crate) mod _interpchannels {
function::{ArgSpec, Either, FuncArgs, PyComparisonValue},
protocol::{PyNumber, PyNumberMethods},
types::{AsNumber, Constructor, Hashable, PyComparisonOp, PyStructSequence, Representable},
vm::crossinterp::{self, Fallback, SharedValue, UNBOUND_REPLACE},
vm::crossinterp::{Fallback, SharedValue, UnboundOp},
};
use alloc::collections::BTreeMap;
use alloc::{collections::VecDeque, sync::Arc};
@@ -25,9 +25,28 @@ pub(crate) mod _interpchannels {
use parking_lot::{Condvar, Mutex};
use std::{sync::OnceLock, time::Instant};
const CHANNEL_SEND: i32 = 1;
const CHANNEL_BOTH: i32 = 0;
const CHANNEL_RECV: i32 = -1;
#[repr(i32)]
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
enum ChannelEnd {
Recv = -1,
Both = 0,
Send = 1,
}
impl ChannelEnd {
const fn from_i32(v: i32) -> Option<Self> {
match v {
-1 => Some(Self::Recv),
0 => Some(Self::Both),
1 => Some(Self::Send),
_ => None,
}
}
const fn as_i32(self) -> i32 {
self as i32
}
}
#[pyattr]
#[pyexception(name = "ChannelError", base = crate::exceptions::types::PyRuntimeError)]
@@ -173,7 +192,7 @@ pub(crate) mod _interpchannels {
interpid: i64,
/// `None` once the owning interpreter is gone (the item is "unbound").
data: Option<SharedValue>,
unboundop: i32,
unboundop: UnboundOp,
waiting: Option<Arc<Waiting>>,
}
@@ -279,7 +298,7 @@ pub(crate) mod _interpchannels {
struct ChannelState {
queue: VecDeque<ChannelItem>,
ends: ChannelEnds,
unboundop: i32,
unboundop: UnboundOp,
fallback: i32,
open: bool,
/// Send is closed and the queue is draining.
@@ -322,7 +341,7 @@ pub(crate) mod _interpchannels {
Ok(chan.clone())
}
fn channel_create(unboundop: i32, fallback: i32) -> i64 {
fn channel_create(unboundop: UnboundOp, fallback: i32) -> i64 {
let mut table = channels().lock();
let cid = table.next_id;
table.next_id += 1;
@@ -420,14 +439,14 @@ pub(crate) mod _interpchannels {
let entry = table.refs.get_mut(&cid).ok_or(ChanErr::NotFound)?;
let chan = entry.chan.as_ref().ok_or(ChanErr::Closed)?.clone();
let mut state = chan.state.lock();
if !force && end == CHANNEL_SEND && state.closing {
if !force && end == ChannelEnd::Send.as_i32() && state.closing {
return Err(ChanErr::Closed);
}
if !state.open {
return Err(ChanErr::Closed);
}
if !force && !state.queue.is_empty() {
if end != CHANNEL_SEND {
if end != ChannelEnd::Send.as_i32() {
return Err(ChanErr::NotEmpty);
}
if state.closing {
@@ -476,7 +495,7 @@ pub(crate) mod _interpchannels {
i += 1;
continue;
}
if item.unboundop == crossinterp::UNBOUND_REMOVE {
if item.unboundop == UnboundOp::Remove {
let item = state.queue.remove(i).expect("index checked");
if let Some(w) = item.waiting {
waiters.push((w, false));
@@ -495,16 +514,12 @@ pub(crate) mod _interpchannels {
}
/// `resolve_unboundop`.
fn resolve_unboundop(arg: i32, default: i32, vm: &VirtualMachine) -> PyResult<i32> {
fn resolve_unboundop(arg: i32, default: UnboundOp, vm: &VirtualMachine) -> PyResult<UnboundOp> {
if arg < 0 {
return Ok(default);
}
match arg {
crossinterp::UNBOUND_REMOVE
| crossinterp::UNBOUND_ERROR
| crossinterp::UNBOUND_REPLACE => Ok(arg),
_ => Err(vm.new_value_error(format!("unsupported unboundop {arg}"))),
}
UnboundOp::from_i32(arg)
.ok_or_else(|| vm.new_value_error(format!("unsupported unboundop {arg}")))
}
/// `resolve_fallback`.
@@ -517,7 +532,7 @@ pub(crate) mod _interpchannels {
/// `channel_id_converter`.
fn parse_cid(obj: &PyObject, vm: &VirtualMachine) -> PyResult<(i64, i32)> {
if let Some(cid) = obj.downcast_ref::<ChannelID>() {
return Ok((cid.cid, cid.end));
return Ok((cid.cid, cid.end.as_i32()));
}
if !obj.number().is_index() {
return Err(vm.new_type_error(format!(
@@ -536,7 +551,7 @@ pub(crate) mod _interpchannels {
vm.new_value_error(format!("channel ID must be a non-negative int, got {repr}"))
);
}
Ok((id, CHANNEL_BOTH))
Ok((id, ChannelEnd::Both.as_i32()))
}
/// The single-`cid` signature shared by several module functions.
@@ -605,7 +620,7 @@ pub(crate) mod _interpchannels {
#[derive(Debug, PyPayload)]
pub(crate) struct ChannelID {
cid: i64,
end: i32,
end: ChannelEnd,
resolve: bool,
}
@@ -620,9 +635,9 @@ pub(crate) mod _interpchannels {
fn repr_str(zelf: &Py<Self>, _vm: &VirtualMachine) -> PyResult<String> {
let name = zelf.class().name().to_string();
Ok(match zelf.end {
CHANNEL_SEND => format!("{name}({}, send=True)", zelf.cid),
CHANNEL_RECV => format!("{name}({}, recv=True)", zelf.cid),
_ => format!("{name}({})", zelf.cid),
ChannelEnd::Send => format!("{name}({}, send=True)", zelf.cid),
ChannelEnd::Recv => format!("{name}({}, recv=True)", zelf.cid),
ChannelEnd::Both => format!("{name}({})", zelf.cid),
})
}
}
@@ -690,20 +705,20 @@ pub(crate) mod _interpchannels {
#[pygetset]
fn end(&self) -> String {
match self.end {
CHANNEL_SEND => "send".to_owned(),
CHANNEL_RECV => "recv".to_owned(),
_ => "both".to_owned(),
ChannelEnd::Send => "send".to_owned(),
ChannelEnd::Recv => "recv".to_owned(),
ChannelEnd::Both => "both".to_owned(),
}
}
#[pygetset(name = "send")]
fn send_end(&self, vm: &VirtualMachine) -> PyResult {
channel_id_from_parts(self.cid, CHANNEL_SEND, true, self.resolve, vm)
channel_id_from_parts(self.cid, ChannelEnd::Send.as_i32(), true, self.resolve, vm)
}
#[pygetset(name = "recv")]
fn recv_end(&self, vm: &VirtualMachine) -> PyResult {
channel_id_from_parts(self.cid, CHANNEL_RECV, true, self.resolve, vm)
channel_id_from_parts(self.cid, ChannelEnd::Recv.as_i32(), true, self.resolve, vm)
}
#[pymethod]
@@ -725,11 +740,13 @@ pub(crate) mod _interpchannels {
Err(ChanErr::NotFound) if force => {}
Err(e) => return Err(e.into_py(cid, vm)),
}
let end = ChannelEnd::from_i32(end).unwrap_or(ChannelEnd::Both);
Ok(ChannelID { cid, end, resolve }.to_pyobject(vm))
}
pub(crate) fn channel_id_parts(obj: &PyObject) -> Option<(i64, i32)> {
obj.downcast_ref::<ChannelID>().map(|c| (c.cid, c.end))
obj.downcast_ref::<ChannelID>()
.map(|c| (c.cid, c.end.as_i32()))
}
/// `_channelid_new`.
@@ -760,9 +777,9 @@ pub(crate) mod _interpchannels {
(Some(false), Some(false)) => {
return Err(vm.new_value_error("'send' and 'recv' cannot both be False"));
}
(Some(true), Some(true)) => end = CHANNEL_BOTH,
(Some(true), _) => end = CHANNEL_SEND,
(_, Some(true)) => end = CHANNEL_RECV,
(Some(true), Some(true)) => end = ChannelEnd::Both.as_i32(),
(Some(true), _) => end = ChannelEnd::Send.as_i32(),
(_, Some(true)) => end = ChannelEnd::Recv.as_i32(),
_ => {}
}
channel_id_from_parts(cid, end, force, resolve, vm)
@@ -818,10 +835,10 @@ pub(crate) mod _interpchannels {
.parse(&args, vm)?;
let unboundarg = int_arg(parsed[0].as_deref(), vm)?.unwrap_or(-1);
let fallbackarg = int_arg(parsed[1].as_deref(), vm)?.unwrap_or(-1);
let unboundop = resolve_unboundop(unboundarg, UNBOUND_REPLACE, vm)?;
let unboundop = resolve_unboundop(unboundarg, UnboundOp::Replace, vm)?;
let fallback = resolve_fallback(fallbackarg, Fallback::Full.as_i32(), vm)?;
let cid = channel_create(unboundop, fallback.as_i32());
channel_id_from_parts(cid, CHANNEL_BOTH, false, false, vm)
channel_id_from_parts(cid, ChannelEnd::Both.as_i32(), false, false, vm)
}
#[pyfunction]
@@ -840,7 +857,7 @@ pub(crate) mod _interpchannels {
args.args.len() + args.kwargs.len()
)));
}
let chans: Vec<(i64, i32, i32)> = channels()
let chans: Vec<(i64, UnboundOp, i32)> = channels()
.lock()
.refs
.iter()
@@ -854,8 +871,8 @@ pub(crate) mod _interpchannels {
items.push(
vm.ctx
.new_tuple(vec![
channel_id_from_parts(cid, CHANNEL_BOTH, false, false, vm)?,
vm.ctx.new_int(unboundop).into(),
channel_id_from_parts(cid, ChannelEnd::Both.as_i32(), false, false, vm)?,
vm.ctx.new_int(unboundop.as_i32()).into(),
vm.ctx.new_int(fallback).into(),
])
.into(),
@@ -924,7 +941,7 @@ pub(crate) mod _interpchannels {
chan: &Channel,
cid: i64,
value: SharedValue,
unboundop: i32,
unboundop: UnboundOp,
blocking: bool,
timeout: Option<Duration>,
vm: &VirtualMachine,
@@ -1001,7 +1018,14 @@ pub(crate) mod _interpchannels {
args: &FuncArgs,
func: &'static str,
vm: &VirtualMachine,
) -> PyResult<(i64, PyObjectRef, i32, Fallback, bool, Option<Duration>)> {
) -> PyResult<(
i64,
PyObjectRef,
UnboundOp,
Fallback,
bool,
Option<Duration>,
)> {
let parsed = ArgSpec {
fname: func,
keywords: &["cid", "obj", "unboundop", "fallback", "blocking", "timeout"],
@@ -1032,7 +1056,7 @@ pub(crate) mod _interpchannels {
let state = chan.state.lock();
(state.unboundop, state.fallback)
} else {
(-1, -1)
(UnboundOp::Replace, -1)
};
let unboundop = resolve_unboundop(unboundarg, default_unboundop, vm)?;
let fallback = resolve_fallback(fallbackarg, default_fallback, vm)?;
@@ -1104,7 +1128,10 @@ pub(crate) mod _interpchannels {
// The item was unbound.
return Ok(vm
.ctx
.new_tuple(vec![vm.ctx.none(), vm.ctx.new_int(unboundop).into()])
.new_tuple(vec![
vm.ctx.none(),
vm.ctx.new_int(unboundop.as_i32()).into(),
])
.into());
};
let obj = data.into_object(vm)?;
@@ -1119,7 +1146,7 @@ pub(crate) mod _interpchannels {
fn channel_next(
cid: i64,
interpid: i64,
) -> ChanResult<(Option<SharedValue>, i32, Option<Arc<Waiting>>)> {
) -> ChanResult<(Option<SharedValue>, UnboundOp, Option<Arc<Waiting>>)> {
let chan = channels_lookup(cid)?;
let mut state = chan.state.lock();
if !state.open {
@@ -1271,7 +1298,7 @@ pub(crate) mod _interpchannels {
Ok(vm
.ctx
.new_tuple(vec![
vm.ctx.new_int(unboundop).into(),
vm.ctx.new_int(unboundop.as_i32()).into(),
vm.ctx.new_int(fallback).into(),
])
.into())
crates/vm/src/stdlib/_interpqueues.rs
+17 -18
View File
@@ -14,7 +14,7 @@ pub(crate) mod _interpqueues {
builtins::{PyBaseExceptionRef, PyModule, PyType, PyTypeRef},
function::{ArgSpec, FuncArgs},
types::Constructor,
vm::crossinterp::{self, Fallback, SharedValue, UNBOUND_REMOVE, UNBOUND_REPLACE},
vm::crossinterp::{Fallback, SharedValue, UnboundOp},
};
use alloc::{collections::BTreeMap, collections::VecDeque, sync::Arc};
use core::cell::Cell;
@@ -160,14 +160,14 @@ pub(crate) mod _interpqueues {
struct QueueItem {
interpid: i64,
data: Option<SharedValue>,
unboundop: i32,
unboundop: UnboundOp,
}
struct Queue {
alive: bool,
maxsize: isize,
items: VecDeque<QueueItem>,
unboundop: i32,
unboundop: UnboundOp,
fallback: Fallback,
}
@@ -206,7 +206,7 @@ pub(crate) mod _interpqueues {
.ok_or(QueueErr::NotFound)
}
fn queue_create(maxsize: isize, unboundop: i32, fallback: Fallback) -> QueueResult<i64> {
fn queue_create(maxsize: isize, unboundop: UnboundOp, fallback: Fallback) -> QueueResult<i64> {
let mut queues = queues().lock();
let qid = queues.next_id;
if qid < 0 {
@@ -244,7 +244,7 @@ pub(crate) mod _interpqueues {
Ok(())
}
fn queue_defaults(qid: i64) -> QueueResult<(i32, Fallback)> {
fn queue_defaults(qid: i64) -> QueueResult<(UnboundOp, Fallback)> {
let queue = queue_lookup(qid)?;
let state = queue.lock();
if !state.alive {
@@ -281,16 +281,12 @@ pub(crate) mod _interpqueues {
Ok(())
}
fn resolve_unboundop(arg: i32, default: i32, vm: &VirtualMachine) -> PyResult<i32> {
fn resolve_unboundop(arg: i32, default: UnboundOp, vm: &VirtualMachine) -> PyResult<UnboundOp> {
if arg < 0 {
return Ok(default);
}
match arg {
crossinterp::UNBOUND_REMOVE
| crossinterp::UNBOUND_ERROR
| crossinterp::UNBOUND_REPLACE => Ok(arg),
_ => Err(vm.new_value_error(format!("unsupported unboundop {arg}"))),
}
UnboundOp::from_i32(arg)
.ok_or_else(|| vm.new_value_error(format!("unsupported unboundop {arg}")))
}
fn resolve_fallback(arg: i32, default: i32, vm: &VirtualMachine) -> PyResult<Fallback> {
@@ -393,7 +389,7 @@ pub(crate) mod _interpqueues {
},
vm,
)?;
let unboundop = resolve_unboundop(unboundarg.get().unwrap_or(-1), UNBOUND_REPLACE, vm)?;
let unboundop = resolve_unboundop(unboundarg.get().unwrap_or(-1), UnboundOp::Replace, vm)?;
let fallback =
resolve_fallback(fallbackarg.get().unwrap_or(-1), Fallback::Full.as_i32(), vm)?;
queue_create(maxsize.get().unwrap(), unboundop, fallback).map_err(|err| err.into_py(-1, vm))
@@ -431,7 +427,7 @@ pub(crate) mod _interpqueues {
vm.ctx
.new_tuple(vec![
vm.ctx.new_int(qid).into(),
vm.ctx.new_int(state.unboundop).into(),
vm.ctx.new_int(state.unboundop.as_i32()).into(),
vm.ctx.new_int(state.fallback.as_i32()).into(),
])
.into(),
@@ -474,7 +470,7 @@ pub(crate) mod _interpqueues {
let (unboundop, fallback) = queue_defaults(qid).map_err(|err| err.into_py(qid, vm))?;
(unboundop, fallback.as_i32())
} else {
(-1, -1)
(UnboundOp::Replace, -1)
};
let unboundop = resolve_unboundop(unboundarg, default_unboundop, vm)?;
let fallback = resolve_fallback(fallbackarg, default_fallback, vm)?;
@@ -515,7 +511,10 @@ pub(crate) mod _interpqueues {
};
let (obj, unboundop) = match item.data {
Some(data) => (data.into_object(vm)?, vm.ctx.none()),
None => (vm.ctx.none(), vm.ctx.new_int(item.unboundop).into()),
None => (
vm.ctx.none(),
vm.ctx.new_int(item.unboundop.as_i32()).into(),
),
};
Ok(vm.ctx.new_tuple(vec![obj, unboundop]).into())
}
@@ -550,7 +549,7 @@ pub(crate) mod _interpqueues {
Ok(vm
.ctx
.new_tuple(vec![
vm.ctx.new_int(unboundop).into(),
vm.ctx.new_int(unboundop.as_i32()).into(),
vm.ctx.new_int(fallback.as_i32()).into(),
])
.into())
@@ -665,7 +664,7 @@ pub(crate) mod _interpqueues {
index += 1;
continue;
}
if state.items[index].unboundop == UNBOUND_REMOVE {
if state.items[index].unboundop == UnboundOp::Remove {
dropped.push(state.items.remove(index).and_then(|item| item.data));
} else {
dropped.push(state.items[index].data.take());
crates/vm/src/stdlib/_io.rs
+6 -6
View File
@@ -5713,7 +5713,7 @@ mod fileio {
return Err(e);
}
if mode.contains(host_io::FileMode::APPENDING) {
if mode.is_superset(&host_io::FileMode::APPENDING) {
let _ = host_io::seek_to_end(fd);
}
@@ -5795,7 +5795,7 @@ mod fileio {
if self.fd.load() < 0 {
return Err(io_closed_error(vm));
}
Ok(self.mode.load().contains(host_io::FileMode::READABLE))
Ok(self.mode.load().is_superset(&host_io::FileMode::READABLE))
}
#[pymethod]
@@ -5803,7 +5803,7 @@ mod fileio {
if self.fd.load() < 0 {
return Err(io_closed_error(vm));
}
Ok(self.mode.load().contains(host_io::FileMode::WRITABLE))
Ok(self.mode.load().is_superset(&host_io::FileMode::WRITABLE))
}
#[pygetset]
@@ -5817,7 +5817,7 @@ mod fileio {
read_byte: OptionalSize,
vm: &VirtualMachine,
) -> PyResult<Option<Vec<u8>>> {
if !zelf.mode.load().contains(host_io::FileMode::READABLE) {
if !zelf.mode.load().is_superset(&host_io::FileMode::READABLE) {
return Err(new_unsupported_operation(
"File or stream is not readable",
vm,
@@ -5895,7 +5895,7 @@ mod fileio {
obj: ArgMemoryBuffer,
vm: &VirtualMachine,
) -> PyResult<Option<usize>> {
if !zelf.mode.load().contains(host_io::FileMode::READABLE) {
if !zelf.mode.load().is_superset(&host_io::FileMode::READABLE) {
return Err(new_unsupported_operation(
"File or stream is not readable",
vm,
@@ -5934,7 +5934,7 @@ mod fileio {
obj: ArgBytesLike,
vm: &VirtualMachine,
) -> PyResult<Option<usize>> {
if !zelf.mode.load().contains(host_io::FileMode::WRITABLE) {
if !zelf.mode.load().is_superset(&host_io::FileMode::WRITABLE) {
return Err(new_unsupported_operation(
"File or stream is not writable",
vm,
crates/vm/src/stdlib/_stat.rs
+74 -116
View File
@@ -92,140 +92,100 @@ mod _stat {
#[pyattr]
pub const S_IFWHT: Mode = rustpython_host_env::os::S_IFWHT as Mode;
// Permission bits
bitflagset::bitflag! {
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
#[allow(non_camel_case_types)]
#[repr(u8)]
enum StatPerm {
S_IXOTH = 0,
S_IWOTH = 1,
S_IROTH = 2,
S_IXGRP = 3,
S_IWGRP = 4,
S_IRGRP = 5,
S_IXUSR = 6,
S_IWUSR = 7,
S_IRUSR = 8,
S_ISVTX = 9,
S_ISGID = 10,
S_ISUID = 11,
}
}
bitflagset::bitflagset! {
#[derive(Copy, Clone, PartialEq, Eq)]
struct StatPerms(u32): StatPerm
}
impl StatPerms {
const S_IRWXO: Self =
Self::from_slice(&[StatPerm::S_IXOTH, StatPerm::S_IWOTH, StatPerm::S_IROTH]);
const S_IRWXG: Self =
Self::from_slice(&[StatPerm::S_IXGRP, StatPerm::S_IWGRP, StatPerm::S_IRGRP]);
const S_IRWXU: Self =
Self::from_slice(&[StatPerm::S_IXUSR, StatPerm::S_IWUSR, StatPerm::S_IRUSR]);
}
const fn perm(flag: StatPerm) -> Mode {
StatPerms::from_element(flag).bits() as Mode
}
#[pyattr]
pub const S_ISUID: Mode = libc_const!(
#[cfg(unix)]
S_ISUID,
0o4000
);
pub const S_IXOTH: Mode = perm(StatPerm::S_IXOTH);
#[pyattr]
pub const S_ISGID: Mode = libc_const!(
#[cfg(unix)]
S_ISGID,
0o2000
);
pub const S_IWOTH: Mode = perm(StatPerm::S_IWOTH);
#[pyattr]
pub const S_ENFMT: Mode = libc_const!(
#[cfg(unix)]
S_ISGID,
0o2000
);
pub const S_IROTH: Mode = perm(StatPerm::S_IROTH);
#[pyattr]
pub const S_ISVTX: Mode = libc_const!(
#[cfg(unix)]
S_ISVTX,
0o1000
);
pub const S_IRWXO: Mode = StatPerms::S_IRWXO.bits() as Mode;
#[pyattr]
pub const S_IRWXU: Mode = libc_const!(
#[cfg(unix)]
S_IRWXU,
0o0700
);
pub const S_IXGRP: Mode = perm(StatPerm::S_IXGRP);
#[pyattr]
pub const S_IRUSR: Mode = libc_const!(
#[cfg(unix)]
S_IRUSR,
0o0400
);
pub const S_IWGRP: Mode = perm(StatPerm::S_IWGRP);
#[pyattr]
pub const S_IREAD: Mode = libc_const!(
#[cfg(unix)]
S_IRUSR,
0o0400
);
pub const S_IRGRP: Mode = perm(StatPerm::S_IRGRP);
#[pyattr]
pub const S_IWUSR: Mode = libc_const!(
#[cfg(unix)]
S_IWUSR,
0o0200
);
pub const S_IRWXG: Mode = StatPerms::S_IRWXG.bits() as Mode;
#[pyattr]
pub const S_IXUSR: Mode = libc_const!(
#[cfg(unix)]
S_IXUSR,
0o0100
);
pub const S_IXUSR: Mode = perm(StatPerm::S_IXUSR);
#[pyattr]
pub const S_IRWXG: Mode = libc_const!(
#[cfg(unix)]
S_IRWXG,
0o0070
);
pub const S_IWUSR: Mode = perm(StatPerm::S_IWUSR);
#[pyattr]
pub const S_IRGRP: Mode = libc_const!(
#[cfg(unix)]
S_IRGRP,
0o0040
);
pub const S_IRUSR: Mode = perm(StatPerm::S_IRUSR);
#[pyattr]
pub const S_IWGRP: Mode = libc_const!(
#[cfg(unix)]
S_IWGRP,
0o0020
);
pub const S_IRWXU: Mode = StatPerms::S_IRWXU.bits() as Mode;
#[pyattr]
pub const S_IXGRP: Mode = libc_const!(
#[cfg(unix)]
S_IXGRP,
0o0010
);
pub const S_ISVTX: Mode = perm(StatPerm::S_ISVTX);
#[pyattr]
pub const S_IRWXO: Mode = libc_const!(
#[cfg(unix)]
S_IRWXO,
0o0007
);
pub const S_ISGID: Mode = perm(StatPerm::S_ISGID);
#[pyattr]
pub const S_IROTH: Mode = libc_const!(
#[cfg(unix)]
S_IROTH,
0o0004
);
pub const S_ISUID: Mode = perm(StatPerm::S_ISUID);
#[pyattr]
pub const S_IWOTH: Mode = libc_const!(
#[cfg(unix)]
S_IWOTH,
0o0002
);
pub const S_ENFMT: Mode = S_ISGID;
#[pyattr]
pub const S_IXOTH: Mode = libc_const!(
#[cfg(unix)]
S_IXOTH,
0o0001
);
pub const S_IREAD: Mode = S_IRUSR;
#[pyattr]
pub const S_IWRITE: Mode = libc_const!(
#[cfg(all(unix, not(target_os = "android"), not(target_os = "redox")))]
S_IWRITE,
0o0200
);
pub const S_IWRITE: Mode = S_IWUSR;
#[pyattr]
pub const S_IEXEC: Mode = libc_const!(
#[cfg(all(unix, not(target_os = "android"), not(target_os = "redox")))]
S_IEXEC,
0o0100
);
pub const S_IEXEC: Mode = S_IXUSR;
const _: () = {
assert!(S_IXOTH == 0o0001 as Mode);
assert!(S_IWOTH == 0o0002 as Mode);
assert!(S_IROTH == 0o0004 as Mode);
assert!(S_IRWXO == 0o0007 as Mode);
assert!(S_IXGRP == 0o0010 as Mode);
assert!(S_IWGRP == 0o0020 as Mode);
assert!(S_IRGRP == 0o0040 as Mode);
assert!(S_IRWXG == 0o0070 as Mode);
assert!(S_IXUSR == 0o0100 as Mode);
assert!(S_IWUSR == 0o0200 as Mode);
assert!(S_IRUSR == 0o0400 as Mode);
assert!(S_IRWXU == 0o0700 as Mode);
assert!(S_ISVTX == 0o1000 as Mode);
assert!(S_ISGID == 0o2000 as Mode);
assert!(S_ISUID == 0o4000 as Mode);
};
// Windows file attributes (if on Windows)
@@ -246,8 +206,6 @@ mod _stat {
IO_REPARSE_TAG_APPEXECLINK, IO_REPARSE_TAG_MOUNT_POINT, IO_REPARSE_TAG_SYMLINK,
};
// Unix file flags (if on Unix)
#[pyattr]
pub use rustpython_host_env::os::{
SF_APPEND, SF_ARCHIVED, SF_DATALESS, SF_FIRMLINK, SF_IMMUTABLE, SF_NOUNLINK, SF_SETTABLE,
crates/vm/src/stdlib/_testinternalcapi.rs
+1 -1
View File
@@ -1361,7 +1361,7 @@ fn name_already_seen(
return Ok(true);
}
if let Some(set) = user_set {
return set.__contains__(name.as_object(), vm);
return set.contains(name.as_object(), vm);
}
Ok(false)
}
crates/vm/src/stdlib/_types.rs
+1 -2
View File
@@ -106,8 +106,7 @@ mod _types {
#[pyattr]
fn ClassMethodDescriptorType(vm: &VirtualMachine) -> PyObjectRef {
// TODO: implement as separate type
vm.ctx.types.method_descriptor_type.to_owned().into()
vm.ctx.types.classmethod_descriptor_type.to_owned().into()
}
#[pyattr]
crates/vm/src/stdlib/_warnings.rs
+35 -29
View File
@@ -28,39 +28,38 @@ mod _warnings {
#[pyattr]
fn filters(vm: &VirtualMachine) -> PyListRef {
vm.state.warnings.filters.clone()
vm.state.warnings.filters.to_owned()
}
#[pyattr]
fn _defaultaction(vm: &VirtualMachine) -> PyStrRef {
vm.state.warnings.default_action.clone()
vm.state.warnings.default_action.to_owned()
}
#[pyattr]
fn _onceregistry(vm: &VirtualMachine) -> PyDictRef {
vm.state.warnings.once_registry.clone()
vm.state.warnings.once_registry.to_owned()
}
#[pyattr]
fn _warnings_context(vm: &VirtualMachine) -> PyObjectRef {
vm.state
.warnings
.context_var
.get_or_init(|| {
// Try to create a real ContextVar if _contextvars is available.
// During early startup it may not be importable yet, in which
// case we fall back to None. This is safe because
// context_aware_warnings defaults to False.
if let Ok(contextvars) = vm.import("_contextvars", 0)
&& let Ok(cv_cls) = contextvars.get_attr("ContextVar", vm)
&& let Ok(cv) = cv_cls.call(("_warnings_context",), vm)
{
cv
} else {
vm.ctx.none()
}
})
.clone()
if let Some(ctx) = vm.state.warnings.context_var.get() {
return ctx.clone();
}
// _warnings is initialized before _contextvars may be importable.
// Retry until ContextVar can be created; do not cache a None fallback.
let created = vm
.import("_contextvars", 0)
.ok()
.and_then(|m| m.get_attr("ContextVar", vm).ok())
.and_then(|cv_cls| cv_cls.call(("_warnings_context",), vm).ok());
match created {
Some(cv) => match vm.state.warnings.context_var.set(cv.clone()) {
Ok(()) => cv,
Err(_) => vm.state.warnings.context_var.get().cloned().unwrap_or(cv),
},
None => vm.ctx.none(),
}
}
#[pyfunction]
@@ -181,13 +180,20 @@ mod _warnings {
let module = args.module.into_option();
// Validate module_globals: must be None or a dict
if let Some(ref mg) = args.module_globals.into_option()
&& !vm.is_none(mg)
&& !mg.class().is(vm.ctx.types.dict_type)
{
return Err(vm.new_type_error("module_globals must be a dict"));
}
let source_line = if let Some(mg) = args.module_globals.into_option() {
if vm.is_none(&mg) {
None
} else if !mg.class().is(vm.ctx.types.dict_type) {
return Err(vm.new_type_error(format!(
"module_globals must be a dict, not '{}'",
mg.class().name()
)));
} else {
crate::warn::get_source_line(&mg, args.lineno, vm)?
}
} else {
None
};
let category = if vm.is_none(&args.category) {
None
@@ -205,7 +211,7 @@ mod _warnings {
args.lineno,
module,
registry,
None, // source_line
source_line,
args.source.into_option(),
vm,
)
crates/vm/src/stdlib/builtins.rs
+30 -14
View File
@@ -31,10 +31,7 @@ mod builtins {
readline::{Readline, ReadlineResult},
stdlib::sys,
types::PyComparisonOp,
vm::compile_mode::{
CompilerFlags, PY_EVAL_INPUT, PY_FILE_INPUT, PY_FUNC_TYPE_INPUT, PY_SINGLE_INPUT,
compile_future_features_from_flags,
},
vm::compile_mode::{CompileStart, CompilerFlags, compile_future_features_from_flags},
};
use itertools::Itertools;
use num_traits::{Signed, ToPrimitive};
@@ -221,18 +218,18 @@ mod builtins {
let future_features = merge_compile_future_features(flags, dont_inherit, vm);
let start = if mode_str == "exec" {
PY_FILE_INPUT
CompileStart::File
} else if mode_str == "eval" {
PY_EVAL_INPUT
CompileStart::Eval
} else if mode_str == "single" {
PY_SINGLE_INPUT
CompileStart::Single
} else if mode_str == "func_type" {
if !is_ast_only {
return Err(vm.new_value_error(
"compile() mode 'func_type' requires flag PyCF_ONLY_AST",
));
}
PY_FUNC_TYPE_INPUT
CompileStart::FuncType
} else {
let msg = if is_ast_only {
"compile() mode must be 'exec', 'eval', 'single' or 'func_type'"
@@ -332,7 +329,7 @@ mod builtins {
vm.compile_string_object_with_flags(
source,
&filename.to_string_lossy(),
start,
start.as_i32(),
compile_flags,
feature_version,
optimize as i32,
@@ -766,18 +763,37 @@ mod builtins {
.is_ok_and(|fd| fd == expected)
};
// Check if we should use rustyline (interactive terminal, not PTY child)
let use_rustyline = fd_matches(&stdin, 0)
let tty = fd_matches(&stdin, 0)
&& fd_matches(&stdout, 1)
&& std::io::stdin().is_terminal()
&& !is_pty_child();
&& std::io::stdout().is_terminal();
// Disable rustyline if prompt contains surrogates (not valid UTF-8 for terminal)
// Encode the prompt with stdout's encoding and reject embedded NULs
// (`strlen` on the encoded bytes).
if tty && let OptionalArg::Present(prompt) = &prompt {
let _ = vm.call_method(&stdout, "flush", ());
let encoding = stdout.get_attr("encoding", vm)?;
let errors = stdout.get_attr("errors", vm)?;
if !vm.is_none(&encoding) {
let encoded = vm.call_method(prompt.as_object(), "encode", (encoding, errors))?;
let bytes = encoded
.downcast::<PyBytes>()
.map_err(|_| vm.new_type_error("encode() must return bytes".to_owned()))?;
if bytes.as_bytes().contains(&0) {
return Err(vm.new_value_error(
"input: prompt string cannot contain null characters".to_owned(),
));
}
}
}
// rustyline is the interactive tty reader. Skip it in a PTY child
// (`pty.fork` + `setsid`): raw mode hangs there.
let prompt_str = match &prompt {
OptionalArg::Present(s) => s.to_str(),
OptionalArg::Missing => Some(""),
};
let use_rustyline = use_rustyline && prompt_str.is_some();
let use_rustyline = tty && !is_pty_child() && prompt_str.is_some();
if use_rustyline {
let prompt = prompt_str.unwrap();
crates/vm/src/stdlib/itertools.rs
+115 -69
View File
@@ -37,6 +37,13 @@ mod decl {
impl PyItertoolsChain {
#[pyslot]
fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult {
let args =
crate::types::drop_kwargs_if_init_overridden(&cls, Self::class(&vm.ctx), args);
if !args.kwargs.is_empty() {
return Err(
vm.new_type_error(format!("{}() takes no keyword arguments", Self::NAME))
);
}
let args_list = PyList::from(args.args);
Self {
source: PyRwLock::new(Some(args_list.to_pyobject(vm).get_iter(vm)?)),
@@ -248,6 +255,7 @@ mod decl {
impl Constructor for PyItertoolsCycle {
type Args = PyIter;
const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true;
fn py_new(_cls: &Py<PyType>, iter: Self::Args, _vm: &VirtualMachine) -> PyResult<Self> {
Ok(Self {
@@ -388,6 +396,7 @@ mod decl {
impl Constructor for PyItertoolsStarmap {
type Args = StarmapNewArgs;
const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true;
fn py_new(
_cls: &Py<PyType>,
@@ -437,6 +446,7 @@ mod decl {
impl Constructor for PyItertoolsTakewhile {
type Args = TakewhileNewArgs;
const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true;
fn py_new(
_cls: &Py<PyType>,
@@ -500,6 +510,7 @@ mod decl {
impl Constructor for PyItertoolsDropwhile {
type Args = DropwhileNewArgs;
const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true;
fn py_new(
_cls: &Py<PyType>,
@@ -545,8 +556,7 @@ mod decl {
struct GroupByState {
current_value: Option<PyObjectRef>,
current_key: Option<PyObjectRef>,
#[pytraverse(skip)]
next_group: bool,
tgtkey: Option<PyObjectRef>,
#[pytraverse(skip)]
grouper: Option<PyWeakRef<PyItertoolsGrouper>>,
}
@@ -556,7 +566,7 @@ mod decl {
f.debug_struct("GroupByState")
.field("current_value", &self.current_value)
.field("current_key", &self.current_key)
.field("next_group", &self.next_group)
.field("tgtkey", &self.tgtkey)
.finish()
}
}
@@ -632,41 +642,43 @@ mod decl {
impl IterNext for PyItertoolsGroupBy {
fn next(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<PyIterReturn> {
let mut state = zelf.state.lock();
state.grouper = None;
{
let mut state = zelf.state.lock();
state.grouper = None;
}
if !state.next_group {
// FIXME: unnecessary clone. current_key always exist until assigning new
let current_key = state.current_key.clone();
drop(state);
let (value, key) = if let Some(old_key) = current_key {
loop {
let (value, new_key) = raise_if_stop!(zelf.advance(vm)?);
if !vm.bool_eq(&new_key, &old_key)? {
break (value, new_key);
loop {
let (tgtkey, currkey) = {
let state = zelf.state.lock();
(state.tgtkey.clone(), state.current_key.clone())
};
match (tgtkey, currkey) {
(_, None) => {}
(None, Some(_)) => break,
(Some(tgtkey), Some(currkey)) => {
if !vm.bool_eq(&tgtkey, &currkey)? {
break;
}
}
} else {
raise_if_stop!(zelf.advance(vm)?)
};
state = zelf.state.lock();
}
let (value, key) = raise_if_stop!(zelf.advance(vm)?);
let mut state = zelf.state.lock();
state.current_value = Some(value);
state.current_key = Some(key);
}
state.next_group = false;
let mut state = zelf.state.lock();
let currkey = state.current_key.clone().unwrap();
state.tgtkey = Some(currkey.clone());
let grouper = PyItertoolsGrouper {
groupby: zelf.to_owned(),
tgtkey: currkey.clone(),
}
.into_ref(&vm.ctx);
state.grouper = Some(grouper.downgrade(None, vm).unwrap());
Ok(PyIterReturn::Return(
(state.current_key.as_ref().unwrap().clone(), grouper).to_pyobject(vm),
))
Ok(PyIterReturn::Return((currkey, grouper).to_pyobject(vm)))
}
}
@@ -675,6 +687,7 @@ mod decl {
#[derive(Debug, PyPayload)]
struct PyItertoolsGrouper {
groupby: PyRef<PyItertoolsGroupBy>,
tgtkey: PyObjectRef,
}
#[pyclass(with(IterNext, Iterable), flags(HAS_WEAKREF))]
@@ -684,30 +697,38 @@ mod decl {
impl IterNext for PyItertoolsGrouper {
fn next(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<PyIterReturn> {
let old_key = {
let mut state = zelf.groupby.state.lock();
if !zelf.groupby.state.lock().is_current(zelf) {
return Ok(PyIterReturn::StopIteration(None));
}
if !state.is_current(zelf) {
return Ok(PyIterReturn::StopIteration(None));
}
// check to see if the value has already been retrieved from the iterator
if let Some(val) = state.current_value.take() {
return Ok(PyIterReturn::Return(val));
}
state.current_key.as_ref().unwrap().clone()
};
let (value, key) = raise_if_stop!(zelf.groupby.advance(vm)?);
if vm.bool_eq(&key, &old_key)? {
Ok(PyIterReturn::Return(value))
} else {
if zelf.groupby.state.lock().current_value.is_none() {
let (value, key) = raise_if_stop!(zelf.groupby.advance(vm)?);
let mut state = zelf.groupby.state.lock();
state.current_value = Some(value);
state.current_key = Some(key);
state.next_group = true;
state.grouper = None;
Ok(PyIterReturn::StopIteration(None))
}
let currkey = {
let state = zelf.groupby.state.lock();
if !state.is_current(zelf) {
return Ok(PyIterReturn::StopIteration(None));
}
state.current_key.clone().unwrap()
};
let tgtkey = zelf.tgtkey.clone();
if !vm.bool_eq(&tgtkey, &currkey)? {
return Ok(PyIterReturn::StopIteration(None));
}
let mut state = zelf.groupby.state.lock();
if !state.is_current(zelf) {
return Ok(PyIterReturn::StopIteration(None));
}
let value = state.current_value.take();
state.current_key = None;
match value {
Some(v) => Ok(PyIterReturn::Return(v)),
None => Ok(PyIterReturn::StopIteration(None)),
}
}
}
@@ -716,7 +737,7 @@ mod decl {
#[pyclass(name = "islice", traverse)]
#[derive(Debug, PyPayload)]
struct PyItertoolsIslice {
iterable: PyIter,
iterable: PyMutex<Option<PyIter>>,
#[pytraverse(skip)]
cur: AtomicCell<usize>,
#[pytraverse(skip)]
@@ -727,24 +748,26 @@ mod decl {
step: usize,
}
// Restrict obj to ints with value 0 <= val <= sys.maxsize
// On failure (out of range, non-int object) a ValueError is raised.
fn pyobject_to_opt_usize(
obj: &PyObject,
name: &'static str,
vm: &VirtualMachine,
) -> PyResult<usize> {
let is_int = obj.fast_isinstance(vm.ctx.types.int_type);
if is_int {
let value = int::get_value(obj).to_usize();
if let Some(value) = value {
// Only succeeds for values for which 0 <= value <= sys.maxsize
if value <= sys::MAXSIZE as usize {
return Ok(value);
}
let value = match obj.try_index(vm) {
Ok(i) => int::get_value(i.as_object()).to_usize(),
Err(e)
if e.fast_isinstance(vm.ctx.exceptions.type_error)
|| e.fast_isinstance(vm.ctx.exceptions.overflow_error) =>
{
None
}
Err(e) => return Err(e),
};
if let Some(value) = value
&& value <= sys::MAXSIZE as usize
{
return Ok(value);
}
// We don't have an int or value was < 0 or > sys.maxsize
Err(vm.new_value_error(format!(
"{name} argument for islice() must be None or an integer: 0 <= x <= sys.maxsize."
)))
@@ -754,6 +777,8 @@ mod decl {
impl PyItertoolsIslice {
#[pyslot]
fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult {
let args =
crate::types::drop_kwargs_if_init_overridden(&cls, Self::class(&vm.ctx), args);
let (iter, start, stop, step) = match args.args.len() {
0 | 1 => {
return Err(vm.new_arity_type_error(Self::NAME, 2..=4, args.args.len()));
@@ -776,7 +801,13 @@ mod decl {
) = args.bind_for(vm, Self::NAME)?;
let step = if !vm.is_none(&step) {
pyobject_to_opt_usize(&step, "Step", vm)?
let step = pyobject_to_opt_usize(&step, "Step", vm)?;
if step == 0 {
return Err(vm.new_value_error(
"Step for islice() must be a positive integer or None.",
));
}
step
} else {
1usize
};
@@ -801,7 +832,7 @@ mod decl {
let iter = iter.get_iter(vm)?;
Self {
iterable: iter,
iterable: PyMutex::new(Some(iter)),
cur: AtomicCell::new(0),
next: AtomicCell::new(start),
stop,
@@ -816,24 +847,38 @@ mod decl {
impl IterNext for PyItertoolsIslice {
fn next(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<PyIterReturn> {
let Some(iterable) = zelf.iterable.lock().clone() else {
return Ok(PyIterReturn::StopIteration(None));
};
let stop = zelf.stop.unwrap_or(usize::MAX);
while zelf.cur.load() < zelf.next.load() {
zelf.iterable.next(vm)?;
raise_if_stop!({
let result = iterable.next(vm)?;
if matches!(result, PyIterReturn::StopIteration(_)) {
*zelf.iterable.lock() = None;
}
result
});
zelf.cur.fetch_add(1);
}
if let Some(stop) = zelf.stop
&& zelf.cur.load() >= stop
{
if zelf.cur.load() >= stop {
*zelf.iterable.lock() = None;
return Ok(PyIterReturn::StopIteration(None));
}
let obj = raise_if_stop!(zelf.iterable.next(vm)?);
let obj = raise_if_stop!({
let result = iterable.next(vm)?;
if matches!(result, PyIterReturn::StopIteration(_)) {
*zelf.iterable.lock() = None;
}
result
});
zelf.cur.fetch_add(1);
// TODO is this overflow check required? attempts to copy CPython.
let (next, ovf) = zelf.next.load().overflowing_add(zelf.step);
zelf.next.store(if ovf { zelf.stop.unwrap() } else { next });
let oldnext = zelf.next.load();
let (newnext, ovf) = oldnext.overflowing_add(zelf.step);
zelf.next
.store(if ovf || newnext > stop { stop } else { newnext });
Ok(PyIterReturn::Return(obj))
}
}
@@ -856,6 +901,7 @@ mod decl {
impl Constructor for PyItertoolsFilterFalse {
type Args = FilterFalseNewArgs;
const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = true;
fn py_new(
_cls: &Py<PyType>,
crates/vm/src/stdlib/marshal.rs
+108 -33
View File
@@ -10,8 +10,8 @@ mod decl {
PyObject, PyObjectRef, PyResult, TryFromObject, VirtualMachine,
builtins::{
PyBaseExceptionRef, PyBool, PyByteArray, PyBytes, PyCode, PyComplex, PyDict,
PyEllipsis, PyFloat, PyFrozenSet, PyInt, PyList, PyNone, PySet, PyStopIteration, PyStr,
PyTuple,
PyEllipsis, PyFloat, PyFrozenSet, PyInt, PyList, PyMemoryView, PyNone, PySet,
PyStopIteration, PyStr, PyTuple,
},
convert::ToPyObject,
function::{ArgBytesLike, OptionalArg},
@@ -386,18 +386,10 @@ mod decl {
buf.write_u8(b'0'); // TYPE_NULL terminator
} else if let Some(s) = obj.downcast_ref::<PySet>() {
buf.write_u8(b'<');
let elems = s.elements();
buf.write_u32(elems.len() as u32);
for elem in &elems {
write_object_depth(buf, elem, refs, version, allow_code, vm, depth - 1)?;
}
write_set_elements(buf, &s.elements(), refs, version, allow_code, vm, depth)?;
} else if let Some(s) = obj.downcast_ref::<PyFrozenSet>() {
buf.write_u8(b'>');
let elems = s.elements();
buf.write_u32(elems.len() as u32);
for elem in &elems {
write_object_depth(buf, elem, refs, version, allow_code, vm, depth - 1)?;
}
write_set_elements(buf, &s.elements(), refs, version, allow_code, vm, depth)?;
} else if let Some(co) = obj.downcast_ref::<PyCode>() {
if !allow_code {
return Err(vm.new_value_error("marshalling code objects is disallowed"));
@@ -454,6 +446,32 @@ mod decl {
Ok(())
}
/// Serialize set elements in `sorted(v, key=marshal.dumps)` order.
fn write_set_elements(
buf: &mut Vec<u8>,
elems: &[PyObjectRef],
refs: &mut Option<WriterRefTable>,
version: i32,
allow_code: bool,
vm: &VirtualMachine,
depth: usize,
) -> PyResult<()> {
use marshal::Write;
buf.write_u32(elems.len() as u32);
let mut pairs = Vec::with_capacity(elems.len());
for elem in elems {
let mut dumped = Vec::new();
let mut inner_refs = (version >= 3).then(WriterRefTable::new);
write_object(&mut dumped, elem, &mut inner_refs, version, allow_code, vm)?;
pairs.push((dumped, elem.clone()));
}
pairs.sort_by(|a, b| a.0.cmp(&b.0));
for (_, elem) in &pairs {
write_object_depth(buf, elem, refs, version, allow_code, vm, depth - 1)?;
}
Ok(())
}
#[derive(FromArgs)]
struct DumpArgs {
value: PyObjectRef,
@@ -771,30 +789,87 @@ mod decl {
#[pyfunction]
fn load(args: LoadArgs, vm: &VirtualMachine) -> PyResult<PyObjectRef> {
// Read from file object into a buffer, one object at a time.
// We read all available data, deserialize one object, then seek
// back to just after the consumed bytes.
let tell_before = vm
.call_method(&args.f, "tell", ())?
.try_into_value::<i64>(vm)?;
let read_res = vm.call_method(&args.f, "read", ())?;
let bytes = ArgBytesLike::try_from_object(vm, read_res)?;
// The borrow ends here: seek() below is the caller's, and reaching the
// same buffer from it would deadlock on a borrow still held.
let (result, consumed) = {
let buf = bytes.borrow_buf();
let mut rdr: &[u8] = &buf;
let len_before = rdr.len();
let result = deserialize_value(&mut rdr, args.allow_code, vm)?;
(result, len_before - rdr.len())
let mut rdr = ReadableFile {
file: args.f,
vm,
buf: Vec::new(),
error: None,
};
let pending_error = RefCell::new(None);
let result = marshal::deserialize_value(
&mut rdr,
PyMarshalBag::new(vm, &pending_error, args.allow_code),
);
if let Some(err) = rdr.error.take() {
return Err(err);
}
match result {
Ok(value) => Ok(value),
Err(error) => Err(pending_error.into_inner().unwrap_or_else(|| match error {
marshal::MarshalError::Eof => vm.new_eof_error("EOF read where not expected"),
marshal::MarshalError::EofObject => {
vm.new_eof_error("EOF read where object expected")
}
marshal::MarshalError::DataTooShort => vm.new_eof_error("marshal data too short"),
error @ marshal::MarshalError::NullObject => vm.new_type_error(error.to_string()),
error @ (marshal::MarshalError::BadSize(_)
| marshal::MarshalError::UnknownType
| marshal::MarshalError::InvalidRef) => {
vm.new_value_error(format!("bad marshal data ({error})"))
}
_ => vm.new_value_error("bad marshal data"),
})),
}
}
// Seek file to just after the consumed bytes
let new_pos = tell_before + consumed as i64;
vm.call_method(&args.f, "seek", (new_pos,))?;
/// File-backed marshal reader. `r_string` fills a scratch buffer via
/// `readinto` on a contiguous writable memoryview.
struct ReadableFile<'a> {
file: PyObjectRef,
vm: &'a VirtualMachine,
buf: Vec<u8>,
error: Option<PyBaseExceptionRef>,
}
Ok(result)
impl ReadableFile<'_> {
fn r_string(&mut self, n: usize) -> PyResult<()> {
self.buf.clear();
let bytearray = PyByteArray::from(vec![0u8; n]).into_ref(&self.vm.ctx);
let memoryview = PyMemoryView::from_object_with_flags(
bytearray.as_object(),
crate::protocol::BufferFlags::CONTIG,
self.vm,
)?
.into_ref(&self.vm.ctx);
let nread_obj = self.vm.call_method(&self.file, "readinto", (memoryview,))?;
let nread = nread_obj
.try_index(self.vm)?
.try_to_primitive::<isize>(self.vm)?;
let n_isize = isize::try_from(n).unwrap_or(isize::MAX);
if nread != n_isize {
if nread > n_isize {
return Err(self.vm.new_value_error(format!(
"read() returned too much data: {n} bytes requested, {nread} returned"
)));
}
return Err(self.vm.new_eof_error("EOF read where not expected"));
}
self.buf.extend_from_slice(&bytearray.borrow_buf());
Ok(())
}
}
impl marshal::Read for ReadableFile<'_> {
fn read_slice(&mut self, n: u32) -> Result<&[u8], marshal::MarshalError> {
if self.error.is_some() {
return Err(marshal::MarshalError::Eof);
}
if let Err(e) = self.r_string(n as usize) {
self.error = Some(e);
return Err(marshal::MarshalError::Eof);
}
Ok(&self.buf)
}
}
/// Reject subclasses of marshallable types (int, float, complex, tuple, etc.).
crates/vm/src/stdlib/mod.rs
+1
View File
@@ -3,6 +3,7 @@ mod _abc;
pub(crate) mod _ast;
mod _codecs;
mod _collections;
pub(crate) use _collections::ordered_dict::ordered_dict::PyOrderedDictItems;
mod _functools;
mod _imp;
pub mod _io;
crates/vm/src/stdlib/sys.rs
+38
View File
@@ -983,6 +983,10 @@ pub mod sys {
#[pyfunction]
fn getprofile(vm: &VirtualMachine) -> PyObjectRef {
#[cfg(feature = "threading")]
if let Some(slot) = crate::vm::thread::current_thread_slot() {
return slot.profile_func.lock().clone();
}
vm.profile_func.borrow().clone()
}
@@ -1082,6 +1086,10 @@ pub mod sys {
#[pyfunction]
fn gettrace(vm: &VirtualMachine) -> PyObjectRef {
#[cfg(feature = "threading")]
if let Some(slot) = crate::vm::thread::current_thread_slot() {
return slot.trace_func.lock().clone();
}
vm.trace_func.borrow().clone()
}
@@ -1251,6 +1259,10 @@ pub mod sys {
#[pyfunction]
fn setprofile(function: PyObjectRef, vm: &VirtualMachine) {
#[cfg(feature = "threading")]
if let Some(slot) = crate::vm::thread::current_thread_slot() {
*slot.profile_func.lock() = function.clone();
}
vm.profile_func.replace(function);
update_use_tracing(vm);
}
@@ -1274,6 +1286,10 @@ pub mod sys {
#[pyfunction]
fn settrace(function: PyObjectRef, vm: &VirtualMachine) {
#[cfg(feature = "threading")]
if let Some(slot) = crate::vm::thread::current_thread_slot() {
*slot.trace_func.lock() = function.clone();
}
vm.trace_func.replace(function);
update_use_tracing(vm);
// The rest of the current line already started before tracing was
@@ -1288,6 +1304,17 @@ pub mod sys {
fn _settraceallthreads(function: PyObjectRef, vm: &VirtualMachine) {
let func = (!vm.is_none(&function)).then(|| function.clone());
*vm.state.global_trace_func.lock() = func;
#[cfg(feature = "threading")]
{
let registry = vm.state.thread_frames.lock();
#[expect(
clippy::iter_over_hash_type,
reason = "every thread slot, order is irrelevant"
)]
for slot in registry.values() {
*slot.trace_func.lock() = function.clone();
}
}
vm.trace_func.replace(function);
update_use_tracing(vm);
}
@@ -1296,6 +1323,17 @@ pub mod sys {
fn _setprofileallthreads(function: PyObjectRef, vm: &VirtualMachine) {
let func = (!vm.is_none(&function)).then(|| function.clone());
*vm.state.global_profile_func.lock() = func;
#[cfg(feature = "threading")]
{
let registry = vm.state.thread_frames.lock();
#[expect(
clippy::iter_over_hash_type,
reason = "every thread slot, order is irrelevant"
)]
for slot in registry.values() {
*slot.profile_func.lock() = function.clone();
}
}
vm.profile_func.replace(function);
update_use_tracing(vm);
}
crates/vm/src/stdlib/sys/monitoring.rs
+122 -121
View File
@@ -12,76 +12,64 @@ const EVENTS_COUNT: usize = 19;
const LOCAL_EVENTS_COUNT: usize = 11;
const UNGROUPED_EVENTS_COUNT: usize = 18;
// Event bit positions
bitflags::bitflags! {
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub struct MonitoringEvents: u32 {
const PY_START = 1 << 0;
const PY_RESUME = 1 << 1;
const PY_RETURN = 1 << 2;
const PY_YIELD = 1 << 3;
const CALL = 1 << 4;
const LINE = 1 << 5;
const INSTRUCTION = 1 << 6;
const JUMP = 1 << 7;
const BRANCH_LEFT = 1 << 8;
const BRANCH_RIGHT = 1 << 9;
const STOP_ITERATION = 1 << 10;
const RAISE = 1 << 11;
const EXCEPTION_HANDLED = 1 << 12;
const PY_UNWIND = 1 << 13;
const PY_THROW = 1 << 14;
const RERAISE = 1 << 15;
const C_RETURN = 1 << 16;
const C_RAISE = 1 << 17;
const BRANCH = 1 << 18;
/// Event identifier (`PY_MONITORING_EVENT_*`), stored in `tstate->what_event`.
/// `None` on the VM field is the `< 0` sentinel (not in a monitoring callback).
#[repr(i32)]
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub(crate) enum MonitoringEvent {
PyStart = 0,
PyResume = 1,
PyReturn = 2,
PyYield = 3,
Call = 4,
Line = 5,
Instruction = 6,
Jump = 7,
BranchLeft = 8,
BranchRight = 9,
StopIteration = 10,
Raise = 11,
ExceptionHandled = 12,
PyUnwind = 13,
PyThrow = 14,
Reraise = 15,
CReturn = 16,
CRaise = 17,
Branch = 18,
}
impl MonitoringEvent {
pub(crate) const fn mask(self) -> u32 {
1 << (self as u32)
}
pub(crate) const fn from_id(id: usize) -> Option<Self> {
match id {
0 => Some(Self::PyStart),
1 => Some(Self::PyResume),
2 => Some(Self::PyReturn),
3 => Some(Self::PyYield),
4 => Some(Self::Call),
5 => Some(Self::Line),
6 => Some(Self::Instruction),
7 => Some(Self::Jump),
8 => Some(Self::BranchLeft),
9 => Some(Self::BranchRight),
10 => Some(Self::StopIteration),
11 => Some(Self::Raise),
12 => Some(Self::ExceptionHandled),
13 => Some(Self::PyUnwind),
14 => Some(Self::PyThrow),
15 => Some(Self::Reraise),
16 => Some(Self::CReturn),
17 => Some(Self::CRaise),
18 => Some(Self::Branch),
_ => None,
}
}
}
// Re-export as plain u32 constants for use in frame.rs hot-path checks
pub(crate) const EVENT_PY_START: u32 = MonitoringEvents::PY_START.bits();
pub(crate) const EVENT_PY_RESUME: u32 = MonitoringEvents::PY_RESUME.bits();
pub(crate) const EVENT_PY_RETURN: u32 = MonitoringEvents::PY_RETURN.bits();
pub(crate) const EVENT_PY_YIELD: u32 = MonitoringEvents::PY_YIELD.bits();
pub(crate) const EVENT_CALL: u32 = MonitoringEvents::CALL.bits();
pub(crate) const EVENT_LINE: u32 = MonitoringEvents::LINE.bits();
pub(crate) const EVENT_INSTRUCTION: u32 = MonitoringEvents::INSTRUCTION.bits();
pub(crate) const EVENT_JUMP: u32 = MonitoringEvents::JUMP.bits();
pub(crate) const EVENT_BRANCH_LEFT: u32 = MonitoringEvents::BRANCH_LEFT.bits();
pub(crate) const EVENT_BRANCH_RIGHT: u32 = MonitoringEvents::BRANCH_RIGHT.bits();
pub(crate) const EVENT_RAISE: u32 = MonitoringEvents::RAISE.bits();
pub(crate) const EVENT_EXCEPTION_HANDLED: u32 = MonitoringEvents::EXCEPTION_HANDLED.bits();
pub(crate) const EVENT_PY_UNWIND: u32 = MonitoringEvents::PY_UNWIND.bits();
pub(crate) const EVENT_C_RETURN: u32 = MonitoringEvents::C_RETURN.bits();
const EVENT_C_RAISE: u32 = MonitoringEvents::C_RAISE.bits();
pub(crate) const EVENT_STOP_ITERATION: u32 = MonitoringEvents::STOP_ITERATION.bits();
pub(crate) const EVENT_PY_THROW: u32 = MonitoringEvents::PY_THROW.bits();
const EVENT_BRANCH: u32 = MonitoringEvents::BRANCH.bits();
pub(crate) const EVENT_RERAISE: u32 = MonitoringEvents::RERAISE.bits();
/// `tstate->what_event` values, the event currently being monitored.
pub(crate) const WHAT_PY_START: i32 = 0;
pub(crate) const WHAT_PY_RESUME: i32 = 1;
pub(crate) const WHAT_PY_RETURN: i32 = 2;
pub(crate) const WHAT_PY_YIELD: i32 = 3;
pub(crate) const WHAT_CALL: i32 = 4;
pub(crate) const WHAT_LINE: i32 = 5;
pub(crate) const WHAT_INSTRUCTION: i32 = 6;
pub(crate) const WHAT_JUMP: i32 = 7;
pub(crate) const WHAT_BRANCH_LEFT: i32 = 8;
pub(crate) const WHAT_BRANCH_RIGHT: i32 = 9;
#[allow(dead_code)]
pub(crate) const WHAT_STOP_ITERATION: i32 = 10;
pub(crate) const WHAT_RAISE: i32 = 11;
pub(crate) const WHAT_EXCEPTION_HANDLED: i32 = 12;
pub(crate) const WHAT_PY_UNWIND: i32 = 13;
pub(crate) const WHAT_PY_THROW: i32 = 14;
#[allow(dead_code)]
pub(crate) const WHAT_RERAISE: i32 = 15;
pub(crate) const WHAT_C_RETURN: i32 = 16;
pub(crate) const WHAT_C_RAISE: i32 = 17;
pub(crate) const WHAT_BRANCH: i32 = 18;
const EVENT_C_RETURN_MASK: u32 = EVENT_C_RETURN | EVENT_C_RAISE;
const C_RETURN_MASK: u32 = MonitoringEvent::CReturn.mask() | MonitoringEvent::CRaise.mask();
const EVENT_NAMES: [&str; EVENTS_COUNT] = [
"PY_START",
@@ -210,7 +198,7 @@ fn parse_single_event(event: i32, vm: &VirtualMachine) -> PyResult<usize> {
return Err(vm.new_value_error("The callback can only be set for one event at a time"));
}
let event_id = event.trailing_zeros() as usize;
if event_id >= EVENTS_COUNT {
if MonitoringEvent::from_id(event_id).is_none() {
return Err(vm.new_value_error(format!("invalid event {event}")));
}
Ok(event_id)
@@ -231,15 +219,17 @@ fn normalize_event_set(event_set: i32, local: bool, vm: &VirtualMachine) -> PyRe
return Err(vm.new_value_error(format!("invalid {kind} 0x{event_set:x}")));
}
if (event_set & EVENT_C_RETURN_MASK) != 0 && (event_set & EVENT_CALL) != EVENT_CALL {
if (event_set & C_RETURN_MASK) != 0
&& (event_set & MonitoringEvent::Call.mask()) != MonitoringEvent::Call.mask()
{
return Err(vm.new_value_error("cannot set C_RETURN or C_RAISE events independently"));
}
event_set &= !EVENT_C_RETURN_MASK;
event_set &= !C_RETURN_MASK;
if (event_set & EVENT_BRANCH) != 0 {
event_set &= !EVENT_BRANCH;
event_set |= EVENT_BRANCH_LEFT | EVENT_BRANCH_RIGHT;
if (event_set & MonitoringEvent::Branch.mask()) != 0 {
event_set &= !MonitoringEvent::Branch.mask();
event_set |= MonitoringEvent::BranchLeft.mask() | MonitoringEvent::BranchRight.mask();
}
if local && event_set >= (1 << LOCAL_EVENTS_COUNT) {
@@ -251,29 +241,32 @@ fn normalize_event_set(event_set: i32, local: bool, vm: &VirtualMachine) -> PyRe
/// Event that causes this opcode to be rewritten to INSTRUMENTED_*.
/// RESUME uses `oparg`: 0 → PY_START, nonzero → PY_RESUME.
fn event_for_opcode(op: rustpython_compiler_core::bytecode::Instruction, oparg: u8) -> Option<u32> {
fn event_for_opcode(
op: rustpython_compiler_core::bytecode::Instruction,
oparg: u8,
) -> Option<MonitoringEvent> {
use rustpython_compiler_core::bytecode::Opcode;
match op.deoptimize().as_opcode() {
Opcode::ReturnValue => Some(EVENT_PY_RETURN),
Opcode::ReturnValue => Some(MonitoringEvent::PyReturn),
Opcode::Call | Opcode::CallKw | Opcode::CallFunctionEx | Opcode::LoadSuperAttr => {
Some(EVENT_CALL)
Some(MonitoringEvent::Call)
}
Opcode::YieldValue => Some(EVENT_PY_YIELD),
Opcode::JumpForward | Opcode::JumpBackward => Some(EVENT_JUMP),
Opcode::YieldValue => Some(MonitoringEvent::PyYield),
Opcode::JumpForward | Opcode::JumpBackward => Some(MonitoringEvent::Jump),
Opcode::PopJumpIfFalse
| Opcode::PopJumpIfTrue
| Opcode::PopJumpIfNone
| Opcode::PopJumpIfNotNone => Some(EVENT_BRANCH_RIGHT),
Opcode::ForIter => Some(EVENT_BRANCH_LEFT),
Opcode::PopIter => Some(EVENT_BRANCH_RIGHT),
Opcode::EndFor | Opcode::EndSend => Some(EVENT_STOP_ITERATION),
Opcode::NotTaken => Some(EVENT_BRANCH_LEFT),
Opcode::EndAsyncFor => Some(EVENT_BRANCH_RIGHT),
| Opcode::PopJumpIfNotNone => Some(MonitoringEvent::BranchRight),
Opcode::ForIter => Some(MonitoringEvent::BranchLeft),
Opcode::PopIter => Some(MonitoringEvent::BranchRight),
Opcode::EndFor | Opcode::EndSend => Some(MonitoringEvent::StopIteration),
Opcode::NotTaken => Some(MonitoringEvent::BranchLeft),
Opcode::EndAsyncFor => Some(MonitoringEvent::BranchRight),
Opcode::Resume => {
if oparg != 0 {
Some(EVENT_PY_RESUME)
Some(MonitoringEvent::PyResume)
} else {
Some(EVENT_PY_START)
Some(MonitoringEvent::PyStart)
}
}
_ => None,
@@ -285,7 +278,7 @@ fn opcode_event_is_active(
oparg: u8,
events: u32,
) -> bool {
event_for_opcode(op, oparg).is_some_and(|ev| events & ev != 0)
event_for_opcode(op, oparg).is_some_and(|ev| events & ev.mask() != 0)
}
/// Walk real instructions, skipping specialization CACHE payloads.
@@ -366,8 +359,9 @@ pub(crate) fn instrument_code(code: &Py<PyCode>, events: u32) {
let caches = base_op.cache_entries();
// LINE/INSTRUCTION wrap every instruction, so specialized
// CACHE payloads must be cleared first.
let want_instrumented = events & (EVENT_LINE | EVENT_INSTRUCTION) != 0
|| opcode_event_is_active(base_op, oparg, events);
let want_instrumented =
events & (MonitoringEvent::Line.mask() | MonitoringEvent::Instruction.mask()) != 0
|| opcode_event_is_active(base_op, oparg, events);
let restore_base = if want_instrumented {
!op.is_instrumented() && u8::from(base_op) != u8::from(op)
} else {
@@ -432,7 +426,7 @@ pub(crate) fn instrument_code(code: &Py<PyCode>, events: u32) {
let op = code.code.instructions.read_op(i);
let oparg = code.code.instructions.read_arg(i).as_u8();
let caches = op.deoptimize().cache_entries();
if (events & (EVENT_LINE | EVENT_INSTRUCTION) != 0
if (events & (MonitoringEvent::Line.mask() | MonitoringEvent::Instruction.mask()) != 0
|| opcode_event_is_active(op, oparg, events))
&& let Some(instrumented) = op.to_instrumented()
{
@@ -444,9 +438,9 @@ pub(crate) fn instrument_code(code: &Py<PyCode>, events: u32) {
}
}
// Phase 5: Place INSTRUMENTED_INSTRUCTION (if EVENT_INSTRUCTION is active)
// Phase 5: Place INSTRUMENTED_INSTRUCTION when Instruction is active
// LINE/INSTRUCTION restore specialized opcodes first, so CACHE slots are valid.
if events & EVENT_INSTRUCTION != 0 {
if events & MonitoringEvent::Instruction.mask() != 0 {
for i in first_traceable..len {
let op = code.code.instructions[i].op;
// Skip ExtendedArg
@@ -471,10 +465,10 @@ pub(crate) fn instrument_code(code: &Py<PyCode>, events: u32) {
}
}
// Phase 6: Place INSTRUMENTED_LINE (if EVENT_LINE is active)
// Phase 6: Place INSTRUMENTED_LINE when Line is active
// Mirrors CPython's initialize_lines: first determine which positions
// are line starts, then mark branch/jump targets, then place opcodes.
if events & EVENT_LINE != 0 {
if events & MonitoringEvent::Line.mask() != 0 {
// is_line_start[i] = true if position i should have INSTRUMENTED_LINE
let mut is_line_start = vec![false; len];
let line_locations = rustpython_compiler_core::marshal::linetable_to_locations(
@@ -728,9 +722,9 @@ fn register_callback(
.callbacks
.remove(&(tool, event_id))
.unwrap_or_else(|| vm.ctx.none());
let branch_id = EVENT_BRANCH.trailing_zeros() as usize;
let branch_left_id = EVENT_BRANCH_LEFT.trailing_zeros() as usize;
let branch_right_id = EVENT_BRANCH_RIGHT.trailing_zeros() as usize;
let branch_id = MonitoringEvent::Branch as usize;
let branch_left_id = MonitoringEvent::BranchLeft as usize;
let branch_right_id = MonitoringEvent::BranchRight as usize;
if !vm.is_none(&func) {
state.callbacks.insert((tool, event_id), func.clone());
// BRANCH is a composite event: also register for BRANCH_LEFT/RIGHT
@@ -856,7 +850,7 @@ thread_local! {
/// `cb_extra` contains the callback arguments after the code object.
fn fire(
vm: &VirtualMachine,
event: u32,
event: MonitoringEvent,
code: &Py<PyCode>,
offset: u32,
cb_extra: &[PyObjectRef],
@@ -866,14 +860,15 @@ fn fire(
return Ok(());
}
let event_id = event.trailing_zeros() as usize;
let event_id = event as usize;
let event_mask = event.mask();
let code_id = code.get_id();
// C_RETURN and C_RAISE are implicitly enabled when CALL is set.
let check_bit = if event & EVENT_C_RETURN_MASK != 0 {
event | EVENT_CALL
let check_bit = if matches!(event, MonitoringEvent::CReturn | MonitoringEvent::CRaise) {
event_mask | MonitoringEvent::Call.mask()
} else {
event
event_mask
};
// Collect callbacks and snapshot the DISABLE sentinel under a single lock.
@@ -911,7 +906,7 @@ fn fire(
FIRING.with(|f| f.set(true));
vm.enter_tracing();
let old_what = vm.what_event.replace(event_id as i32);
let old_what = vm.what_event.replace(Some(event));
let result = (|| {
for (tool, cb) in callbacks {
let result = cb.call(args.clone(), vm)?;
@@ -945,7 +940,7 @@ fn fire(
pub(crate) fn fire_py_start(vm: &VirtualMachine, code: &Py<PyCode>, offset: u32) -> PyResult<()> {
fire(
vm,
EVENT_PY_START,
MonitoringEvent::PyStart,
code,
offset,
&[vm.ctx.new_int(offset).into()],
@@ -955,7 +950,7 @@ pub(crate) fn fire_py_start(vm: &VirtualMachine, code: &Py<PyCode>, offset: u32)
pub(crate) fn fire_py_resume(vm: &VirtualMachine, code: &Py<PyCode>, offset: u32) -> PyResult<()> {
fire(
vm,
EVENT_PY_RESUME,
MonitoringEvent::PyResume,
code,
offset,
&[vm.ctx.new_int(offset).into()],
@@ -970,7 +965,7 @@ pub(crate) fn fire_py_return(
) -> PyResult<()> {
fire(
vm,
EVENT_PY_RETURN,
MonitoringEvent::PyReturn,
code,
offset,
&[vm.ctx.new_int(offset).into(), retval.to_owned()],
@@ -985,7 +980,7 @@ pub(crate) fn fire_py_yield(
) -> PyResult<()> {
fire(
vm,
EVENT_PY_YIELD,
MonitoringEvent::PyYield,
code,
offset,
&[vm.ctx.new_int(offset).into(), retval.to_owned()],
@@ -1001,7 +996,7 @@ pub(crate) fn fire_call(
) -> PyResult<()> {
fire(
vm,
EVENT_CALL,
MonitoringEvent::Call,
code,
offset,
&[vm.ctx.new_int(offset).into(), callable.to_owned(), arg0],
@@ -1017,7 +1012,7 @@ pub(crate) fn fire_c_return(
) -> PyResult<()> {
fire(
vm,
EVENT_C_RETURN,
MonitoringEvent::CReturn,
code,
offset,
&[vm.ctx.new_int(offset).into(), callable.to_owned(), arg0],
@@ -1033,7 +1028,7 @@ pub(crate) fn fire_c_raise(
) -> PyResult<()> {
fire(
vm,
EVENT_C_RAISE,
MonitoringEvent::CRaise,
code,
offset,
&[vm.ctx.new_int(offset).into(), callable.to_owned(), arg0],
@@ -1046,7 +1041,13 @@ pub(crate) fn fire_line(
offset: u32,
line: u32,
) -> PyResult<()> {
fire(vm, EVENT_LINE, code, offset, &[vm.ctx.new_int(line).into()])
fire(
vm,
MonitoringEvent::Line,
code,
offset,
&[vm.ctx.new_int(line).into()],
)
}
pub(crate) fn fire_instruction(
@@ -1056,7 +1057,7 @@ pub(crate) fn fire_instruction(
) -> PyResult<()> {
fire(
vm,
EVENT_INSTRUCTION,
MonitoringEvent::Instruction,
code,
offset,
&[vm.ctx.new_int(offset).into()],
@@ -1071,7 +1072,7 @@ pub(crate) fn fire_raise(
) -> PyResult<()> {
fire(
vm,
EVENT_RAISE,
MonitoringEvent::Raise,
code,
offset,
&[vm.ctx.new_int(offset).into(), exception.to_owned()],
@@ -1092,7 +1093,7 @@ pub(crate) fn fire_reraise(
RERAISE_PENDING.with(|f| f.set(true));
let result = fire(
vm,
EVENT_RERAISE,
MonitoringEvent::Reraise,
code,
offset,
&[vm.ctx.new_int(offset).into(), exception.to_owned()],
@@ -1112,7 +1113,7 @@ pub(crate) fn fire_exception_handled(
RERAISE_PENDING.with(|f| f.set(false));
fire(
vm,
EVENT_EXCEPTION_HANDLED,
MonitoringEvent::ExceptionHandled,
code,
offset,
&[vm.ctx.new_int(offset).into(), exception.to_owned()],
@@ -1128,7 +1129,7 @@ pub(crate) fn fire_py_unwind(
RERAISE_PENDING.with(|f| f.set(false));
fire(
vm,
EVENT_PY_UNWIND,
MonitoringEvent::PyUnwind,
code,
offset,
&[vm.ctx.new_int(offset).into(), exception.to_owned()],
@@ -1143,7 +1144,7 @@ pub(crate) fn fire_py_throw(
) -> PyResult<()> {
fire(
vm,
EVENT_PY_THROW,
MonitoringEvent::PyThrow,
code,
offset,
&[vm.ctx.new_int(offset).into(), exception.to_owned()],
@@ -1165,7 +1166,7 @@ pub(crate) fn fire_stop_iteration(
};
fire(
vm,
EVENT_STOP_ITERATION,
MonitoringEvent::StopIteration,
code,
offset,
&[vm.ctx.new_int(offset).into(), exc],
@@ -1180,7 +1181,7 @@ pub(crate) fn fire_jump(
) -> PyResult<()> {
fire(
vm,
EVENT_JUMP,
MonitoringEvent::Jump,
code,
offset,
&[
@@ -1198,7 +1199,7 @@ pub(crate) fn fire_branch_left(
) -> PyResult<()> {
fire(
vm,
EVENT_BRANCH_LEFT,
MonitoringEvent::BranchLeft,
code,
offset,
&[
@@ -1216,7 +1217,7 @@ pub(crate) fn fire_branch_right(
) -> PyResult<()> {
fire(
vm,
EVENT_BRANCH_RIGHT,
MonitoringEvent::BranchRight,
code,
offset,
&[
crates/vm/src/types/slot.rs
+28
View File
@@ -250,6 +250,7 @@ bitflags! {
const HEAPTYPE = 1 << 9;
const BASETYPE = 1 << 10;
const METHOD_DESCRIPTOR = 1 << 17;
const IS_ABSTRACT = 1 << 20;
// For built-in types that match the subject itself in pattern matching
// (bool, int, float, str, bytes, bytearray, list, tuple, dict, set, frozenset)
// This is not a stable API
@@ -1123,6 +1124,9 @@ impl PyType {
// Check if it's a Python function (not a native descriptor)
!attr.class().is(ctx.types.wrapper_descriptor_type)
&& !attr.class().is(ctx.types.method_descriptor_type)
&& !attr
.class()
.is(ctx.types.classmethod_descriptor_type)
})
})
})
@@ -1806,6 +1810,9 @@ impl PyType {
pub trait Constructor: PyPayload + core::fmt::Debug {
type Args: FromArgs;
/// When true, extra keywords are dropped if a subclass replaced `tp_init`.
const DROP_KWARGS_WHEN_INIT_OVERRIDDEN: bool = false;
/// The type slot for `__new__`. Override this only when you need special
/// behavior beyond simple payload creation.
#[inline]
@@ -1813,6 +1820,11 @@ pub trait Constructor: PyPayload + core::fmt::Debug {
fn slot_new(cls: PyTypeRef, args: FuncArgs, vm: &VirtualMachine) -> PyResult {
// The name is the type the slot was written for, not the subclass being
// constructed, so a subclass reports what its base declares.
let args = if Self::DROP_KWARGS_WHEN_INIT_OVERRIDDEN {
drop_kwargs_if_init_overridden(&cls, Self::class(&vm.ctx), args)
} else {
args
};
let args: Self::Args = args.bind_for(vm, Callee::of::<Self>(vm))?;
let payload = Self::py_new(&cls, args, vm)?;
payload.into_ref_with_type(vm, cls).map(Into::into)
@@ -1823,6 +1835,22 @@ pub trait Constructor: PyPayload + core::fmt::Debug {
fn py_new(cls: &Py<PyType>, args: Self::Args, vm: &VirtualMachine) -> PyResult<Self>;
}
/// Extra keywords are ignored when a subclass has replaced `tp_init`.
pub(crate) fn drop_kwargs_if_init_overridden(
cls: &Py<PyType>,
base: &Py<PyType>,
mut args: FuncArgs,
) -> FuncArgs {
if args.kwargs.is_empty() {
return args;
}
let uses_base_init = cls.slots.init.load().map(fn_addr) == base.slots.init.load().map(fn_addr);
if !(cls.is(base) || uses_base_init) {
args.kwargs = Default::default();
}
args
}
pub trait DefaultConstructor: PyPayload + Default + core::fmt::Debug {
fn construct_and_init(args: Self::Args, vm: &VirtualMachine) -> PyResult<PyRef<Self>>
where
crates/vm/src/types/structseq.rs
+11 -16
View File
@@ -1,5 +1,4 @@
use crate::common::lock::LazyLock;
use crate::common::wtf8::Wtf8;
use crate::{
AsObject, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine, atomic_func,
builtins::{
@@ -340,10 +339,15 @@ pub trait PyStructSequence: StaticType + PyClassImpl + Sized + 'static {
}
}
// Check for unexpected keyword arguments
if !kwargs.is_empty() {
let names: Vec<&Wtf8> = kwargs.keys().map(|k| k.as_ref()).collect();
return Err(vm.new_type_error(format!("Got unexpected field name(s): {names:?}")));
let names = vm.ctx.new_list(
kwargs
.keys()
.map(|k| vm.ctx.new_str(k.to_owned()).into())
.collect(),
);
let names_repr = names.as_object().repr(vm)?;
return Err(vm.new_type_error(format!("Got unexpected field name(s): {names_repr}")));
}
PyTuple::new_unchecked(items.into_boxed_slice())
@@ -368,28 +372,19 @@ pub trait PyStructSequence: StaticType + PyClassImpl + Sized + 'static {
fn extend_pyclass(ctx: &Context, class: &'static Py<PyType>) {
// Getters for named visible fields (indices 0 to REQUIRED_FIELD_NAMES.len() - 1)
for (i, &name) in Self::Data::REQUIRED_FIELD_NAMES.iter().enumerate() {
// cast i to a u8 so there's less to store in the getter closure.
// Hopefully there's not struct sequences with >=256 elements :P
let i = i as u8;
class.set_attr(
ctx.intern_str(name),
ctx.new_readonly_getset(name, class, move |zelf: &Py<PyTuple>| {
zelf[i as usize].to_owned()
})
.into(),
ctx.new_readonly_tuple_member(name, class, i).into(),
);
}
// Getters for hidden/skipped fields (indices after visible fields)
let visible_count = Self::Data::REQUIRED_FIELD_NAMES.len() + Self::Data::UNNAMED_FIELDS_LEN;
for (i, &name) in Self::Data::OPTIONAL_FIELD_NAMES.iter().enumerate() {
let idx = (visible_count + i) as u8;
class.set_attr(
ctx.intern_str(name),
ctx.new_readonly_getset(name, class, move |zelf: &Py<PyTuple>| {
zelf[idx as usize].to_owned()
})
.into(),
ctx.new_readonly_tuple_member(name, class, visible_count + i)
.into(),
);
}
crates/vm/src/types/zoo.rs
+2
View File
@@ -79,6 +79,7 @@ pub struct TypeZoo {
pub builtin_function_or_method_type: &'static Py<PyType>,
pub builtin_method_type: &'static Py<PyType>,
pub method_descriptor_type: &'static Py<PyType>,
pub classmethod_descriptor_type: &'static Py<PyType>,
pub property_type: &'static Py<PyType>,
pub getset_type: &'static Py<PyType>,
pub module_type: &'static Py<PyType>,
@@ -202,6 +203,7 @@ impl TypeZoo {
weakproxy_type: weakproxy::PyWeakProxy::init_builtin_type(),
weakcallableproxy_type: weakproxy::PyWeakCallableProxy::init_builtin_type(),
method_descriptor_type: descriptor::PyMethodDescriptor::init_builtin_type(),
classmethod_descriptor_type: descriptor::PyClassMethodDescriptor::init_builtin_type(),
none_type: singletons::PyNone::init_builtin_type(),
typing_no_default_type: crate::stdlib::_typing::NoDefault::init_builtin_type(),
not_implemented_type: singletons::PyNotImplemented::init_builtin_type(),
crates/vm/src/vm/compile.rs
+21 -34
View File
@@ -8,10 +8,7 @@ use crate::{
AsObject, PyObjectRef, PyRef, PyResult, VirtualMachine,
builtins::{PyBaseExceptionRef, PyCode},
compiler::{self, CompileError, CompileOpts},
vm::compile_mode::{
CompilerFlags, PY_EVAL_INPUT, PY_FILE_INPUT, PY_FUNC_TYPE_INPUT, PY_SINGLE_INPUT,
compile_future_features_from_flags,
},
vm::compile_mode::{CompileStart, CompilerFlags, compile_future_features_from_flags},
};
#[derive(Debug)]
@@ -388,6 +385,9 @@ impl VirtualMachine {
use crate::stdlib::_ast;
let cf = CompilerFlags::from_bits_retain(flags);
let Some(start) = CompileStart::from_i32(start) else {
return Err(self.new_system_error("Invalid start argument passed to Py_CompileString"));
};
let source =
self.decode_source_bytes(source, filename, cf.contains(CompilerFlags::IGNORE_COOKIE))?;
let source = source.as_str();
@@ -412,19 +412,15 @@ impl VirtualMachine {
};
if is_ast_only {
if start == PY_FUNC_TYPE_INPUT {
if start == CompileStart::FuncType {
return _ast::parse_func_type(self, source, filename, optimize, target_version)
.map_err(|e| (e, Some(source), allow_incomplete).to_pyexception(self));
}
let (parser_mode, interactive) = match start {
PY_SINGLE_INPUT => (ruff_python_parser::Mode::Module, true),
PY_FILE_INPUT => (ruff_python_parser::Mode::Module, false),
PY_EVAL_INPUT => (ruff_python_parser::Mode::Expression, false),
_ => {
return Err(
self.new_system_error("Invalid start argument passed to Py_CompileString")
);
}
CompileStart::Single => (ruff_python_parser::Mode::Module, true),
CompileStart::File => (ruff_python_parser::Mode::Module, false),
CompileStart::Eval => (ruff_python_parser::Mode::Expression, false),
CompileStart::FuncType => unreachable!(),
};
let parsed = _ast::parse(
self,
@@ -440,7 +436,7 @@ impl VirtualMachine {
dont_imply_dedent,
)
.map_err(|e| (e, Some(source), allow_incomplete).to_pyexception(self))?;
if start == PY_SINGLE_INPUT {
if start == CompileStart::Single {
return _ast::wrap_interactive(self, &parsed);
}
return Ok(parsed);
@@ -448,13 +444,9 @@ impl VirtualMachine {
if type_comments {
let parser_mode = match start {
PY_SINGLE_INPUT | PY_FILE_INPUT => ruff_python_parser::Mode::Module,
PY_EVAL_INPUT => ruff_python_parser::Mode::Expression,
_ => {
return Err(
self.new_system_error("Invalid start argument passed to Py_CompileString")
);
}
CompileStart::Single | CompileStart::File => ruff_python_parser::Mode::Module,
CompileStart::Eval => ruff_python_parser::Mode::Expression,
CompileStart::FuncType => ruff_python_parser::Mode::Module,
};
_ast::parse(
self,
@@ -465,7 +457,7 @@ impl VirtualMachine {
None,
type_comments,
false,
start == PY_SINGLE_INPUT,
start == CompileStart::Single,
future_features,
dont_imply_dedent,
)
@@ -473,15 +465,10 @@ impl VirtualMachine {
}
let mode = match start {
PY_SINGLE_INPUT => compiler::Mode::Single,
PY_FILE_INPUT => compiler::Mode::Exec,
PY_EVAL_INPUT => compiler::Mode::Eval,
PY_FUNC_TYPE_INPUT => compiler::Mode::BlockExpr,
_ => {
return Err(
self.new_system_error("Invalid start argument passed to Py_CompileString")
);
}
CompileStart::Single => compiler::Mode::Single,
CompileStart::File => compiler::Mode::Exec,
CompileStart::Eval => compiler::Mode::Eval,
CompileStart::FuncType => compiler::Mode::BlockExpr,
};
let mut opts = self.compile_opts();
opts.optimize = optimize;
@@ -1650,7 +1637,7 @@ mod escape_warnings {
vm.compile_string_object_with_flags(
b"from __future__ import barry_as_FLUFL\n2 <> 3\n",
"<test>",
PY_FILE_INPUT,
CompileStart::File.as_i32(),
flags,
-1,
-1,
@@ -1661,7 +1648,7 @@ mod escape_warnings {
.compile_string_object_with_flags(
b"from __future__ import barry_as_FLUFL\n2 != 3\n",
"<test>",
PY_FILE_INPUT,
CompileStart::File.as_i32(),
flags,
-1,
-1,
@@ -1686,7 +1673,7 @@ mod escape_warnings {
vm.compile_string_object_with_flags(
b"2 <> 3\n",
"<test>",
PY_FILE_INPUT,
CompileStart::File.as_i32(),
flags,
-1,
-1,
crates/vm/src/vm/compile_mode.rs
+25 -4
View File
@@ -1,9 +1,30 @@
use crate::bytecode;
pub(crate) const PY_SINGLE_INPUT: i32 = 256;
pub(crate) const PY_FILE_INPUT: i32 = 257;
pub(crate) const PY_EVAL_INPUT: i32 = 258;
pub(crate) const PY_FUNC_TYPE_INPUT: i32 = 345;
/// `Py_single_input` / `Py_file_input` / `Py_eval_input` / `Py_func_type_input`.
#[repr(i32)]
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub(crate) enum CompileStart {
Single = 256,
File = 257,
Eval = 258,
FuncType = 345,
}
impl CompileStart {
pub(crate) const fn from_i32(v: i32) -> Option<Self> {
match v {
256 => Some(Self::Single),
257 => Some(Self::File),
258 => Some(Self::Eval),
345 => Some(Self::FuncType),
_ => None,
}
}
pub(crate) const fn as_i32(self) -> i32 {
self as i32
}
}
bitflags::bitflags! {
/// `PyCF_*` compiler flags together with the `__future__` `CO_FUTURE_*`
crates/vm/src/vm/context.rs
+27 -1
View File
@@ -361,7 +361,9 @@ impl Context {
names.__new__.as_str(),
PyType::__new__,
PyMethodFlags::METHOD,
None,
Some(
"__new__($type, /, *args, **kwargs)\n--\n\nCreate and return a new object. See help(type) for accurate signature.",
),
);
let empty_str = unsafe { string_pool.intern("", types.str_type.to_owned()) };
let empty_bytes = create_object(PyBytes::from(Vec::new()), types.bytes_type);
@@ -680,6 +682,30 @@ impl Context {
member_descriptor.into_ref(self)
}
pub fn new_readonly_tuple_member(
&self,
name: &str,
class: &'static Py<PyType>,
index: usize,
) -> PyRef<PyMemberDescriptor> {
let member_def = PyMemberDef {
name: name.to_owned(),
kind: MemberKind::Object,
getter: MemberGetter::TupleItem(index),
setter: MemberSetter::Setter(None),
doc: None,
};
let member_descriptor = PyMemberDescriptor {
common: PyDescriptorOwned {
typ: class.to_owned(),
name: self.intern_str(name),
qualname: PyRwLock::new(None),
},
member: member_def,
};
member_descriptor.into_ref(self)
}
pub fn new_readonly_getset<F, T>(
&self,
name: &str,
crates/vm/src/vm/crossinterp.rs
+24 -3
View File
@@ -33,9 +33,30 @@ fn memoryview_registered() -> bool {
}
/// Unbound-item ops (`UNBOUND_REMOVE` / `UNBOUND_ERROR` / `UNBOUND_REPLACE`).
pub const UNBOUND_REMOVE: i32 = 1;
pub const UNBOUND_ERROR: i32 = 2;
pub const UNBOUND_REPLACE: i32 = 3;
#[repr(i32)]
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum UnboundOp {
Remove = 1,
Error = 2,
Replace = 3,
}
impl UnboundOp {
#[must_use]
pub const fn from_i32(v: i32) -> Option<Self> {
Some(match v {
1 => Self::Remove,
2 => Self::Error,
3 => Self::Replace,
_ => return None,
})
}
#[must_use]
pub const fn as_i32(self) -> i32 {
self as i32
}
}
/// `xidata_fallback_t`: what to do with an object that has no XI data support.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
crates/vm/src/vm/interpreter.rs
+6 -6
View File
@@ -669,6 +669,7 @@ impl Interpreter {
// Wait for thread shutdown - call threading._shutdown() if available.
// This waits for all non-daemon threads to complete.
// threading module may not be imported, so ignore import errors.
#[cfg(feature = "threading")]
if let Ok(threading) = vm.import("threading", 0)
&& let Ok(shutdown) = threading.get_attr("_shutdown", vm)
&& let Err(e) = shutdown.call((), vm)
@@ -1859,6 +1860,7 @@ for _ in range(40):
fn eval_breaker_tripped_when_stop_requested() {
let interp = Interpreter::without_stdlib(Default::default());
interp.enter(|vm| {
crate::signal::clear_eval_breaker_for_test();
assert!(
crate::vm::thread::set_stop_requested_for_current_thread(true),
"current thread has no stop_requested flag"
@@ -1945,7 +1947,7 @@ for _ in range(40):
#[cfg(feature = "threading")]
#[test]
fn a_thread_blocked_on_a_lock_does_not_stall_stop_the_world() {
use super::super::thread::THREAD_DETACHED;
use super::super::thread::ThreadState;
use crate::common::lock::PyDetachingRwLock;
use alloc::sync::Arc;
use core::{
@@ -1986,11 +1988,9 @@ for _ in range(40):
// hanging it, as the timeout on the stop below does.
let deadline = std::time::Instant::now() + Duration::from_secs(10);
let blocked_detached = |ident| {
state
.thread_frames
.lock()
.get(&ident)
.is_some_and(|slot| slot.state.load(Ordering::Acquire) == THREAD_DETACHED)
state.thread_frames.lock().get(&ident).is_some_and(|slot| {
slot.state.load(Ordering::Acquire) == ThreadState::Detached as i32
})
};
loop {
match worker_ident.load(Ordering::Acquire) {
crates/vm/src/vm/mod.rs
+70 -64
View File
@@ -96,8 +96,9 @@ pub struct VirtualMachine {
pub profile_func: RefCell<PyObjectRef>,
pub trace_func: RefCell<PyObjectRef>,
pub use_tracing: Cell<bool>,
/// Event currently being monitored, or -1 when not in a callback.
pub(crate) what_event: Cell<i32>,
/// Event currently being monitored (`tstate->what_event`).
/// `None` when not in a monitoring callback.
pub(crate) what_event: Cell<Option<crate::stdlib::sys::monitoring::MonitoringEvent>>,
tracing_depth: Cell<usize>,
pub recursion_limit: Cell<usize>,
pub(crate) signal_handlers: OnceCell<SignalHandlers>,
@@ -299,7 +300,8 @@ impl StopTheWorldState {
.iter()
.filter(|(thread_id, slot)| {
**thread_id != requester
&& slot.state.load(Ordering::Relaxed) != thread::THREAD_SHUTTING_DOWN
&& slot.state.load(Ordering::Relaxed)
!= thread::ThreadState::ShuttingDown as i32
})
.count();
let count = (count.min(i64::MAX as usize)) as i64;
@@ -323,7 +325,7 @@ impl StopTheWorldState {
/// Try to CAS detached threads directly to SUSPENDED and check whether
/// stop countdown reached zero after parking detached threads.
fn park_detached_threads(&self, state: &PyGlobalState) -> bool {
use thread::{THREAD_ATTACHED, THREAD_DETACHED, THREAD_SHUTTING_DOWN, THREAD_SUSPENDED};
use thread::ThreadState;
let requester = self.requester.load(Ordering::Relaxed);
let registry = state.thread_frames.lock();
let mut attached_seen = 0u64;
@@ -339,11 +341,11 @@ impl StopTheWorldState {
}
let state = slot.state.load(Ordering::Relaxed);
if state == THREAD_DETACHED {
if state == ThreadState::Detached as i32 {
// CAS DETACHED → SUSPENDED (park without thread cooperation)
match slot.state.compare_exchange(
THREAD_DETACHED,
THREAD_SUSPENDED,
ThreadState::Detached as i32,
ThreadState::Suspended as i32,
Ordering::AcqRel,
Ordering::Relaxed,
) {
@@ -351,38 +353,40 @@ impl StopTheWorldState {
slot.stop_requested.store(false, Ordering::Release);
forced_parks = forced_parks.saturating_add(1);
}
Err(THREAD_ATTACHED) => {
// Set per-thread stop bit (_PY_EVAL_PLEASE_STOP_BIT).
slot.stop_requested.store(true, Ordering::Release);
crate::signal::set_stop_bit();
// Raced with a thread re-attaching; it will self-suspend.
attached_seen = attached_seen.saturating_add(1);
}
Err(THREAD_DETACHED) => {
// Extremely unlikely race; next poll will handle it.
}
Err(THREAD_SUSPENDED) => {
slot.stop_requested.store(false, Ordering::Release);
// Another path parked it first.
}
Err(THREAD_SHUTTING_DOWN) => {
slot.stop_requested.store(false, Ordering::Release);
}
Err(other) => {
debug_assert!(
false,
"unexpected thread state in park_detached_threads: {other}"
);
}
Err(actual) => match ThreadState::from_i32(actual) {
Some(ThreadState::Attached) => {
// Set per-thread stop bit (_PY_EVAL_PLEASE_STOP_BIT).
slot.stop_requested.store(true, Ordering::Release);
crate::signal::set_stop_bit();
// Raced with a thread re-attaching; it will self-suspend.
attached_seen = attached_seen.saturating_add(1);
}
Some(ThreadState::Detached) => {
// Extremely unlikely race; next poll will handle it.
}
Some(ThreadState::Suspended) => {
slot.stop_requested.store(false, Ordering::Release);
// Another path parked it first.
}
Some(ThreadState::ShuttingDown) => {
slot.stop_requested.store(false, Ordering::Release);
}
None => {
debug_assert!(
false,
"unexpected thread state in park_detached_threads: {actual}"
);
}
},
}
} else if state == THREAD_ATTACHED {
} else if state == ThreadState::Attached as i32 {
// Set per-thread stop bit (_PY_EVAL_PLEASE_STOP_BIT).
slot.stop_requested.store(true, Ordering::Release);
crate::signal::set_stop_bit();
// Thread is in bytecode — it will see `requested` and self-suspend
attached_seen = attached_seen.saturating_add(1);
}
// THREAD_SUSPENDED / THREAD_SHUTTING_DOWN → already parked
// Suspended / ShuttingDown → already parked
}
if attached_seen != 0 {
self.stats_attached_seen
@@ -525,7 +529,7 @@ impl StopTheWorldState {
/// Resume all suspended threads (`start_the_world`).
pub fn start_the_world(&self, state: &PyGlobalState) {
use thread::{THREAD_DETACHED, THREAD_SHUTTING_DOWN, THREAD_SUSPENDED};
use thread::ThreadState;
let requester = self.requester.load(Ordering::Relaxed);
stw_trace(format_args!("start begin requester={requester}"));
let registry = state.thread_frames.lock();
@@ -549,17 +553,18 @@ impl StopTheWorldState {
slot.stop_requested.store(false, Ordering::Release);
let state = slot.state.load(Ordering::Relaxed);
if state == THREAD_SHUTTING_DOWN {
if state == ThreadState::ShuttingDown as i32 {
// `_PyThreadState_RemoveExcept` + SetShuttingDown already
// took this thread off the resume path. Leave it hanging.
continue;
}
debug_assert!(
state == THREAD_SUSPENDED,
state == ThreadState::Suspended as i32,
"non-requester thread not suspended at start-the-world: id={id} state={state}"
);
if state == THREAD_SUSPENDED {
slot.state.store(THREAD_DETACHED, Ordering::Release);
if state == ThreadState::Suspended as i32 {
slot.state
.store(ThreadState::Detached as i32, Ordering::Release);
slot.thread.unpark();
}
}
@@ -660,7 +665,7 @@ impl StopTheWorldState {
/// already SUSPENDED when a new stop counts it neither notifies nor is
/// force-parked again, so an edge-based countdown could never reach zero.
fn all_non_requester_suspended(&self, state: &PyGlobalState) -> bool {
use thread::{THREAD_SHUTTING_DOWN, THREAD_SUSPENDED};
use thread::ThreadState;
let requester = self.requester.load(Ordering::Relaxed);
let registry = state.thread_frames.lock();
@@ -673,7 +678,9 @@ impl StopTheWorldState {
continue;
}
let slot_state = slot.state.load(Ordering::Acquire);
if slot_state != THREAD_SUSPENDED && slot_state != THREAD_SHUTTING_DOWN {
if slot_state != ThreadState::Suspended as i32
&& slot_state != ThreadState::ShuttingDown as i32
{
return false;
}
}
@@ -682,7 +689,7 @@ impl StopTheWorldState {
#[cfg(debug_assertions)]
fn debug_assert_all_non_requester_suspended(&self, state: &PyGlobalState) {
use thread::{THREAD_SHUTTING_DOWN, THREAD_SUSPENDED};
use thread::ThreadState;
let requester = self.requester.load(Ordering::Relaxed);
let registry = state.thread_frames.lock();
@@ -697,7 +704,7 @@ impl StopTheWorldState {
let state = slot.state.load(Ordering::Relaxed);
debug_assert!(
state == THREAD_SUSPENDED || state == THREAD_SHUTTING_DOWN,
state == ThreadState::Suspended as i32 || state == ThreadState::ShuttingDown as i32,
"non-requester thread not suspended during stop-the-world: id={id} state={state}"
);
}
@@ -705,7 +712,7 @@ impl StopTheWorldState {
#[cfg(debug_assertions)]
fn debug_assert_all_non_requester_detached(&self, state: &PyGlobalState) {
use thread::THREAD_SUSPENDED;
use thread::ThreadState;
let requester = self.requester.load(Ordering::Relaxed);
let registry = state.thread_frames.lock();
@@ -720,7 +727,7 @@ impl StopTheWorldState {
let state = slot.state.load(Ordering::Relaxed);
debug_assert!(
state != THREAD_SUSPENDED,
state != ThreadState::Suspended as i32,
"non-requester thread still suspended after start-the-world: id={id} state={state}"
);
}
@@ -1230,7 +1237,7 @@ impl VirtualMachine {
profile_func,
trace_func,
use_tracing: Cell::new(false),
what_event: Cell::new(-1),
what_event: Cell::new(None),
tracing_depth: Cell::new(0),
recursion_limit: Cell::new(if cfg!(debug_assertions) { 256 } else { 1000 }),
signal_handlers,
@@ -1325,13 +1332,13 @@ impl VirtualMachine {
// Use dotted names when freeze-stdlib is enabled (modules come from Lib/encodings/),
// otherwise use underscored names (modules come from core_modules/).
let (ascii_module_name, utf8_module_name) = if cfg!(feature = "freeze-stdlib") {
("encodings.ascii", "encodings.utf_8")
} else {
("encodings_ascii", "encodings_utf_8")
};
let (ascii_module_name, utf8_module_name, latin1_module_name) =
if cfg!(feature = "freeze-stdlib") {
("encodings.ascii", "encodings.utf_8", "encodings.latin_1")
} else {
("encodings_ascii", "encodings_utf_8", "encodings_latin_1")
};
// Register ascii encoding
// __import__("encodings.ascii") returns top-level "encodings", so
// look up the actual submodule in sys.modules.
self.import(ascii_module_name, 0)?;
@@ -1357,19 +1364,17 @@ impl VirtualMachine {
.codec_registry
.register_manual("utf8", utf8_codec);
// Register latin-1 / iso8859-1 aliases needed very early for stdio
// bootstrap (e.g. PYTHONIOENCODING=latin-1).
if cfg!(feature = "freeze-stdlib") {
self.import("encodings.latin_1", 0)?;
let latin1_module = sys_modules.get_item("encodings.latin_1", self)?;
let getregentry = latin1_module.get_attr("getregentry", self)?;
let codec_info = getregentry.call((), self)?;
let latin1_codec: crate::codecs::PyCodec = codec_info.try_into_value(self)?;
for name in ["latin-1", "latin_1", "latin1", "iso8859-1", "iso8859_1"] {
self.state
.codec_registry
.register_manual(name, latin1_codec.clone());
}
// latin-1 is a built-in codec (needed very early for stdio bootstrap,
// e.g. PYTHONIOENCODING=latin-1).
self.import(latin1_module_name, 0)?;
let latin1_module = sys_modules.get_item(latin1_module_name, self)?;
let getregentry = latin1_module.get_attr("getregentry", self)?;
let codec_info = getregentry.call((), self)?;
let latin1_codec: crate::codecs::PyCodec = codec_info.try_into_value(self)?;
for name in ["latin-1", "latin_1", "latin1", "iso8859-1", "iso8859_1"] {
self.state
.codec_registry
.register_manual(name, latin1_codec.clone());
}
Ok(())
}
@@ -2976,7 +2981,7 @@ impl VirtualMachine {
{
let ret_result = self.trace_event_what(
TraceEvent::Return,
crate::stdlib::sys::monitoring::WHAT_PY_UNWIND,
crate::stdlib::sys::monitoring::MonitoringEvent::PyUnwind,
None,
);
// call_trace_protected: if trace function raises, its error
@@ -3842,6 +3847,7 @@ pub fn resolve_frozen_alias(name: &str) -> &str {
"_frozen_importlib_external" => "importlib._bootstrap_external",
"encodings_ascii" => "encodings.ascii",
"encodings_utf_8" => "encodings.utf_8",
"encodings_latin_1" => "encodings.latin_1",
"__hello_alias__" | "__phello_alias__" | "__phello_alias__.spam" => "__hello__",
"__phello__.__init__" => "<__phello__",
"__phello__.ham.__init__" => "<__phello__.ham",
crates/vm/src/vm/thread.rs
+145 -88
View File
@@ -1,9 +1,13 @@
#[cfg(all(not(unix), feature = "threading"))]
use super::FramePtr;
#[cfg(feature = "threading")]
use crate::PyObjectRef;
#[cfg(feature = "threading")]
use crate::builtins::PyBaseExceptionRef;
#[cfg(feature = "threading")]
use alloc::sync::Arc;
#[cfg(feature = "threading")]
use rustpython_common::lock::PyMutex;
use crate::frame::InterpreterFrame;
#[cfg(feature = "threading")]
@@ -23,20 +27,36 @@ use itertools::Itertools;
use std::collections::HashMap;
use std::thread_local;
// Thread states for stop-the-world support (`_Py_THREAD_*`).
// DETACHED: not executing Python bytecode (in native code, or idle)
// ATTACHED: actively executing Python bytecode
// SUSPENDED: parked by a stop-the-world request
// SHUTTING_DOWN: interpreter is finalizing; the OS thread must hang
// (`_PyThreadState_HangThread`) and must not look done to `_ThreadHandle`.
/// Thread states for stop-the-world support (`_Py_THREAD_*`).
///
/// DETACHED: not executing Python bytecode (in native code, or idle)
/// ATTACHED: actively executing Python bytecode
/// SUSPENDED: parked by a stop-the-world request
/// SHUTTING_DOWN: interpreter is finalizing; the OS thread must hang
/// (`_PyThreadState_HangThread`) and must not look done to `_ThreadHandle`.
#[cfg(feature = "threading")]
pub const THREAD_DETACHED: i32 = 0;
#[repr(i32)]
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub enum ThreadState {
Detached = 0,
Attached = 1,
Suspended = 2,
ShuttingDown = 3,
}
#[cfg(feature = "threading")]
pub const THREAD_ATTACHED: i32 = 1;
#[cfg(feature = "threading")]
pub const THREAD_SUSPENDED: i32 = 2;
#[cfg(feature = "threading")]
pub const THREAD_SHUTTING_DOWN: i32 = 3;
impl ThreadState {
#[must_use]
pub const fn from_i32(v: i32) -> Option<Self> {
match v {
0 => Some(Self::Detached),
1 => Some(Self::Attached),
2 => Some(Self::Suspended),
3 => Some(Self::ShuttingDown),
_ => None,
}
}
}
/// Per-thread shared state for sys._current_frames() and sys._current_exceptions().
/// The exception field uses atomic operations for lock-free cross-thread reads.
@@ -62,6 +82,12 @@ pub struct ThreadSlot {
#[cfg(not(unix))]
pub frames: parking_lot::Mutex<Vec<FramePtr>>,
pub exception: crate::PyAtomicRef<Option<crate::exceptions::types::PyBaseException>>,
/// `tstate->c_traceobj`. Cross-thread source of truth for sys.gettrace /
/// sys._settraceallthreads.
pub trace_func: PyMutex<PyObjectRef>,
/// `tstate->c_profileobj`. Cross-thread source of truth for sys.getprofile /
/// sys._setprofileallthreads.
pub profile_func: PyMutex<PyObjectRef>,
/// Thread state for stop-the-world: DETACHED / ATTACHED / SUSPENDED / SHUTTING_DOWN
pub state: core::sync::atomic::AtomicI32,
/// Per-thread stop request bit (eval breaker equivalent).
@@ -416,13 +442,27 @@ fn ensure_thread_slot(vm: &VirtualMachine) -> CurrentFrameSlot {
#[cfg(not(unix))]
frames: parking_lot::Mutex::new(Vec::new()),
exception: crate::PyAtomicRef::from(None::<PyBaseExceptionRef>),
trace_func: PyMutex::new(
vm.state
.global_trace_func
.lock()
.clone()
.unwrap_or_else(|| vm.ctx.none()),
),
profile_func: PyMutex::new(
vm.state
.global_profile_func
.lock()
.clone()
.unwrap_or_else(|| vm.ctx.none()),
),
state: core::sync::atomic::AtomicI32::new(
if vm.state.stop_the_world.requested.load(Ordering::Acquire) {
// Match init_threadstate(): new thread-state starts
// suspended while stop-the-world is active.
THREAD_SUSPENDED
ThreadState::Suspended as i32
} else {
THREAD_DETACHED
ThreadState::Detached as i32
},
),
stop_requested: core::sync::atomic::AtomicBool::new(false),
@@ -436,6 +476,13 @@ fn ensure_thread_slot(vm: &VirtualMachine) -> CurrentFrameSlot {
})
}
/// The current thread's `ThreadSlot` for the entered interpreter, if any.
#[cfg(feature = "threading")]
#[must_use]
pub fn current_thread_slot() -> Option<CurrentFrameSlot> {
CURRENT_THREAD_SLOT.with(|slot| slot.borrow().clone())
}
/// Make `slot` the current thread slot (and the cached top-frame pointer).
#[cfg(feature = "threading")]
fn set_current_thread_slot(slot: CurrentFrameSlot) {
@@ -456,7 +503,7 @@ fn current_slot_is_attached() -> bool {
CURRENT_THREAD_SLOT.with(|slot| {
slot.borrow()
.as_ref()
.is_some_and(|s| s.state.load(Ordering::Acquire) == THREAD_ATTACHED)
.is_some_and(|s| s.state.load(Ordering::Acquire) == ThreadState::Attached as i32)
})
}
@@ -515,7 +562,7 @@ fn end_interpreter_section(switched: bool) {
#[cfg(feature = "threading")]
fn wait_while_suspended(slot: &ThreadSlot) -> u64 {
let mut wait_yields = 0u64;
while slot.state.load(Ordering::Acquire) == THREAD_SUSPENDED {
while slot.state.load(Ordering::Acquire) == ThreadState::Suspended as i32 {
wait_yields = wait_yields.saturating_add(1);
std::thread::park();
}
@@ -539,8 +586,10 @@ fn hang_thread() -> ! {
pub fn hang_current_thread(state: &PyGlobalState) -> ! {
CURRENT_THREAD_SLOT.with(|slot| {
if let Some(s) = slot.borrow().as_ref() {
let prev = s.state.swap(THREAD_SHUTTING_DOWN, Ordering::AcqRel);
if prev == THREAD_ATTACHED {
let prev = s
.state
.swap(ThreadState::ShuttingDown as i32, Ordering::AcqRel);
if prev == ThreadState::Attached as i32 {
crate::object::qsbr::QSBR.offline(&s.qsbr);
}
s.stop_requested.store(false, Ordering::Release);
@@ -570,7 +619,8 @@ pub fn set_other_threads_shutting_down(state: &PyGlobalState) {
continue;
}
slot.stop_requested.store(false, Ordering::Release);
slot.state.store(THREAD_SHUTTING_DOWN, Ordering::Release);
slot.state
.store(ThreadState::ShuttingDown as i32, Ordering::Release);
slot.thread.unpark();
}
}
@@ -582,8 +632,8 @@ fn attach_thread(vm: &VirtualMachine) {
super::stw_trace(format_args!("attach begin"));
loop {
match s.state.compare_exchange(
THREAD_DETACHED,
THREAD_ATTACHED,
ThreadState::Detached as i32,
ThreadState::Attached as i32,
Ordering::AcqRel,
Ordering::Relaxed,
) {
@@ -592,21 +642,23 @@ fn attach_thread(vm: &VirtualMachine) {
super::stw_trace(format_args!("attach DETACHED->ATTACHED"));
break;
}
Err(THREAD_SUSPENDED) => {
// Parked by stop-the-world — wait until released to DETACHED
super::stw_trace(format_args!("attach wait-suspended"));
let wait_yields = wait_while_suspended(s);
vm.state.stop_the_world.add_attach_wait_yields(wait_yields);
// Retry CAS
}
Err(THREAD_SHUTTING_DOWN) => {
super::stw_trace(format_args!("attach hang shutting-down"));
hang_thread();
}
Err(state) => {
debug_assert!(false, "unexpected thread state in attach: {state}");
break;
}
Err(state) => match ThreadState::from_i32(state) {
Some(ThreadState::Suspended) => {
// Parked by stop-the-world — wait until released to DETACHED
super::stw_trace(format_args!("attach wait-suspended"));
let wait_yields = wait_while_suspended(s);
vm.state.stop_the_world.add_attach_wait_yields(wait_yields);
// Retry CAS
}
Some(ThreadState::ShuttingDown) => {
super::stw_trace(format_args!("attach hang shutting-down"));
hang_thread();
}
_ => {
debug_assert!(false, "unexpected thread state in attach: {state}");
break;
}
},
}
}
}
@@ -628,20 +680,19 @@ fn detach_thread() {
CURRENT_THREAD_SLOT.with(|slot| {
if let Some(s) = slot.borrow().as_ref() {
match s.state.compare_exchange(
THREAD_ATTACHED,
THREAD_DETACHED,
ThreadState::Attached as i32,
ThreadState::Detached as i32,
Ordering::AcqRel,
Ordering::Acquire,
) {
Ok(_) => {
crate::object::qsbr::QSBR.offline(&s.qsbr);
}
Err(THREAD_DETACHED) => {
debug_assert!(false, "detach called while already DETACHED");
return;
}
Err(state) => {
debug_assert!(false, "unexpected thread state in detach: {state}");
debug_assert!(
matches!(ThreadState::from_i32(state), Some(ThreadState::Detached)),
"unexpected thread state in detach: {state}"
);
return;
}
}
@@ -663,7 +714,7 @@ pub fn allow_threads<R>(vm: &VirtualMachine, f: impl FnOnce() -> R) -> R {
let should_transition = CURRENT_THREAD_SLOT.with(|slot| {
slot.borrow()
.as_ref()
.is_some_and(|s| s.state.load(Ordering::Acquire) == THREAD_ATTACHED)
.is_some_and(|s| s.state.load(Ordering::Acquire) == ThreadState::Attached as i32)
});
if !should_transition {
return f();
@@ -700,7 +751,7 @@ pub fn attach_for_callback<R>(vm: &VirtualMachine, f: impl FnOnce() -> R) -> R {
let should_transition = CURRENT_THREAD_SLOT.with(|slot| {
slot.borrow()
.as_ref()
.is_some_and(|s| s.state.load(Ordering::Acquire) != THREAD_ATTACHED)
.is_some_and(|s| s.state.load(Ordering::Acquire) != ThreadState::Attached as i32)
});
if !should_transition {
return f();
@@ -795,8 +846,8 @@ fn do_suspend(state: &PyGlobalState) {
let _registry = state.thread_frames.lock();
if stw.requested.load(Ordering::Acquire) {
Some(s.state.compare_exchange(
THREAD_ATTACHED,
THREAD_SUSPENDED,
ThreadState::Attached as i32,
ThreadState::Suspended as i32,
Ordering::AcqRel,
Ordering::Acquire,
))
@@ -816,26 +867,28 @@ fn do_suspend(state: &PyGlobalState) {
// Consumed this thread's stop request bit.
s.stop_requested.store(false, Ordering::Release);
}
Some(Err(THREAD_DETACHED)) => {
// Leaving VM; caller will re-check on next entry.
super::stw_trace(format_args!("suspend skip DETACHED"));
return;
}
Some(Err(THREAD_SUSPENDED)) => {
// Already parked by another path.
s.stop_requested.store(false, Ordering::Release);
super::stw_trace(format_args!("suspend skip already-suspended"));
return;
}
Some(Err(THREAD_SHUTTING_DOWN)) => {
s.stop_requested.store(false, Ordering::Release);
super::stw_trace(format_args!("suspend hang shutting-down"));
hang_thread();
}
Some(Err(state)) => {
debug_assert!(false, "unexpected thread state in suspend: {state}");
return;
}
Some(Err(state)) => match ThreadState::from_i32(state) {
Some(ThreadState::Detached) => {
// Leaving VM; caller will re-check on next entry.
super::stw_trace(format_args!("suspend skip DETACHED"));
return;
}
Some(ThreadState::Suspended) => {
// Already parked by another path.
s.stop_requested.store(false, Ordering::Release);
super::stw_trace(format_args!("suspend skip already-suspended"));
return;
}
Some(ThreadState::ShuttingDown) => {
s.stop_requested.store(false, Ordering::Release);
super::stw_trace(format_args!("suspend hang shutting-down"));
hang_thread();
}
_ => {
debug_assert!(false, "unexpected thread state in suspend: {state}");
return;
}
},
}
super::stw_trace(format_args!("suspend ATTACHED->SUSPENDED"));
@@ -853,25 +906,27 @@ fn do_suspend(state: &PyGlobalState) {
// Re-attach (DETACHED → ATTACHED), tstate_wait_attach CAS loop.
loop {
match s.state.compare_exchange(
THREAD_DETACHED,
THREAD_ATTACHED,
ThreadState::Detached as i32,
ThreadState::Attached as i32,
Ordering::AcqRel,
Ordering::Acquire,
) {
Ok(_) => break,
Err(THREAD_SUSPENDED) => {
let extra_wait = wait_while_suspended(s);
stw.add_suspend_wait_yields(extra_wait);
}
Err(THREAD_ATTACHED) => break,
Err(THREAD_SHUTTING_DOWN) => {
super::stw_trace(format_args!("suspend resume hang shutting-down"));
hang_thread();
}
Err(state) => {
debug_assert!(false, "unexpected post-suspend state: {state}");
break;
}
Err(state) => match ThreadState::from_i32(state) {
Some(ThreadState::Suspended) => {
let extra_wait = wait_while_suspended(s);
stw.add_suspend_wait_yields(extra_wait);
}
Some(ThreadState::Attached) => break,
Some(ThreadState::ShuttingDown) => {
super::stw_trace(format_args!("suspend resume hang shutting-down"));
hang_thread();
}
_ => {
debug_assert!(false, "unexpected post-suspend state: {state}");
break;
}
},
}
}
s.stop_requested.store(false, Ordering::Release);
@@ -941,7 +996,7 @@ pub(crate) fn debug_assert_current_thread_attached() {
if let Some(s) = slot.borrow().as_ref() {
debug_assert_eq!(
s.state.load(Ordering::Relaxed),
THREAD_ATTACHED,
ThreadState::Attached as i32,
"type cache read while thread not ATTACHED"
);
}
@@ -1075,8 +1130,8 @@ pub fn cleanup_current_thread_frames(vm: &VirtualMachine) {
// thread-state slot is being removed.
if let Some(slot) = &slot_to_clean {
let _ = slot.state.compare_exchange(
THREAD_ATTACHED,
THREAD_DETACHED,
ThreadState::Attached as i32,
ThreadState::Detached as i32,
Ordering::AcqRel,
Ordering::Acquire,
);
@@ -1097,7 +1152,7 @@ pub fn cleanup_current_thread_frames(vm: &VirtualMachine) {
if let Some(slot) = &_removed
&& vm.state.stop_the_world.requested.load(Ordering::Acquire)
&& thread_id != vm.state.stop_the_world.requester_ident()
&& slot.state.load(Ordering::Relaxed) != THREAD_SUSPENDED
&& slot.state.load(Ordering::Relaxed) != ThreadState::Suspended as i32
{
// A non-requester thread disappeared while stop-the-world is pending.
// Unblock requester countdown progress.
@@ -1175,7 +1230,9 @@ pub fn reinit_frame_slot_after_fork(vm: &VirtualMachine) {
#[cfg(not(unix))]
frames: parking_lot::Mutex::new(current_frames),
exception: crate::PyAtomicRef::from(vm.topmost_exception()),
state: core::sync::atomic::AtomicI32::new(THREAD_ATTACHED),
trace_func: PyMutex::new(vm.trace_func.borrow().clone()),
profile_func: PyMutex::new(vm.profile_func.borrow().clone()),
state: core::sync::atomic::AtomicI32::new(ThreadState::Attached as i32),
stop_requested: core::sync::atomic::AtomicBool::new(false),
thread: std::thread::current(),
qsbr: crate::object::qsbr::QSBR.register(),
@@ -1412,7 +1469,7 @@ impl VirtualMachine {
profile_func: RefCell::new(global_profile.unwrap_or_else(|| self.ctx.none())),
trace_func: RefCell::new(global_trace.unwrap_or_else(|| self.ctx.none())),
use_tracing: Cell::new(use_tracing),
what_event: Cell::new(-1),
what_event: Cell::new(None),
tracing_depth: Cell::new(0),
recursion_limit: self.recursion_limit.clone(),
signal_handlers: core::cell::OnceCell::new(),
crates/vm/src/warn.rs
+178 -47
View File
@@ -1,8 +1,9 @@
use crate::{
AsObject, Context, Py, PyObject, PyObjectRef, PyResult, VirtualMachine,
AsObject, Context, Py, PyAtomicRef, PyObject, PyObjectRef, PyResult, VirtualMachine,
anystr::SplitLinesArgs,
builtins::{
PyBaseExceptionRef, PyDictRef, PyListRef, PyStr, PyStrInterned, PyStrRef, PyTuple,
PyTupleRef, PyType, PyTypeRef,
PyBaseExceptionRef, PyDict, PyDictRef, PyList, PyListRef, PyStr, PyStrInterned, PyStrRef,
PyTuple, PyTupleRef, PyType, PyTypeRef,
},
convert::TryFromObject,
};
@@ -10,9 +11,9 @@ use core::sync::atomic::{AtomicUsize, Ordering};
use rustpython_common::lock::OnceCell;
pub struct WarningsState {
pub filters: PyListRef,
pub once_registry: PyDictRef,
pub default_action: PyStrRef,
pub filters: PyAtomicRef<PyList>,
pub once_registry: PyAtomicRef<PyDict>,
pub default_action: PyAtomicRef<PyStr>,
pub filters_version: AtomicUsize,
pub context_var: OnceCell<PyObjectRef>,
lock_count: AtomicUsize,
@@ -70,9 +71,9 @@ impl WarningsState {
pub fn init_state(ctx: &Context) -> Self {
Self {
filters: Self::create_default_filters(ctx),
once_registry: ctx.new_dict(),
default_action: ctx.new_str("default"),
filters: Self::create_default_filters(ctx).into(),
once_registry: ctx.new_dict().into(),
default_action: ctx.new_str("default").into(),
filters_version: AtomicUsize::new(0),
context_var: OnceCell::new(),
lock_count: AtomicUsize::new(0),
@@ -114,7 +115,7 @@ fn get_warnings_attr(
vm: &VirtualMachine,
attr_name: &'static PyStrInterned,
try_import: bool,
) -> Option<PyObjectRef> {
) -> PyResult<Option<PyObjectRef>> {
let module = if try_import
&& !vm
.state
@@ -123,60 +124,152 @@ fn get_warnings_attr(
{
match vm.import("warnings", 0) {
Ok(module) => module,
Err(_) => return None,
Err(e) if e.fast_isinstance(vm.ctx.exceptions.import_error) => return Ok(None),
Err(e) => return Err(e),
}
} else {
match vm.sys_module.get_attr(identifier!(vm, modules), vm) {
Ok(modules) => match modules.get_item(vm.ctx.intern_str("warnings"), vm) {
Ok(module) => module,
Err(_) => return None,
},
Err(_) => return None,
let Ok(modules) = vm.sys_module.get_attr(identifier!(vm, modules), vm) else {
return Ok(None);
};
match modules.get_item(vm.ctx.intern_str("warnings"), vm) {
Ok(module) => module,
Err(_) => return Ok(None),
}
};
module.get_attr(attr_name, vm).ok()
vm.get_attribute_opt(module, attr_name)
}
/// Get the warnings filters list from `sys.modules['warnings'].filters`,
/// falling back to vm.state.warnings.filters.
fn get_warnings_filters(vm: &VirtualMachine) -> PyListRef {
if let Some(filters_obj) = get_warnings_attr(vm, identifier!(&vm.ctx, filters), false)
&& let Ok(filters) = filters_obj.try_into_value::<PyListRef>(vm)
{
return filters;
/// falling back to the interpreter cache (`st->filters`).
fn get_warnings_filters(vm: &VirtualMachine) -> PyResult<PyListRef> {
if let Some(filters_obj) = get_warnings_attr(vm, identifier!(vm, filters), false)? {
let filters = PyListRef::try_from_object(vm, filters_obj)
.map_err(|_| vm.new_value_error("_warnings.filters must be a list"))?;
vm.state
.warnings
.filters
.swap_to_temporary_refs(filters, vm);
}
vm.state.warnings.filters.clone()
Ok(vm.state.warnings.filters.to_owned())
}
/// Get the default action from `sys.modules['warnings']._defaultaction`,
fn get_warnings_context_filters(vm: &VirtualMachine) -> PyResult<Option<PyListRef>> {
let Some(ctx_var) = vm.state.warnings.context_var.get() else {
return Ok(None);
};
if vm.is_none(ctx_var) {
return Ok(None);
}
let ctx = vm.call_method(ctx_var, "get", (vm.ctx.none(),))?;
if vm.is_none(&ctx) {
return Ok(None);
}
let filters = ctx.get_attr("_filters", vm)?;
PyListRef::try_from_object(vm, filters)
.map(Some)
.map_err(|_| vm.new_value_error("_filters of warnings._warnings_context must be a list"))
}
fn bless_my_loader(
module_globals: &PyObject,
vm: &VirtualMachine,
) -> PyResult<Option<PyObjectRef>> {
let external = vm
.importlib
.get_attr("_bootstrap_external", vm)
.or_else(|_| vm.import("importlib._bootstrap_external", 0))?;
if vm.is_none(&external) {
return Ok(None);
}
let bless = external.get_attr("_bless_my_loader", vm)?;
let loader = bless.call((module_globals.to_owned(),), vm)?;
if vm.is_none(&loader) {
Ok(None)
} else {
Ok(Some(loader))
}
}
pub(crate) fn get_source_line(
module_globals: &PyObject,
lineno: usize,
vm: &VirtualMachine,
) -> PyResult<Option<PyObjectRef>> {
let Some(loader) = bless_my_loader(module_globals, vm)? else {
return Ok(None);
};
let module_name = if let Some(dict) = module_globals.downcast_ref::<PyDict>() {
match dict.get_item_opt(identifier!(vm, __name__), vm)? {
Some(name) => name,
None => return Ok(None),
}
} else {
match module_globals.get_item(identifier!(vm, __name__), vm) {
Ok(name) => name,
Err(_) => return Ok(None),
}
};
let Some(get_source) = vm.get_attribute_opt(loader, vm.ctx.intern_str("get_source"))? else {
return Ok(None);
};
let source = get_source.call((module_name,), vm)?;
if vm.is_none(&source) {
return Ok(None);
}
let Some(source_str) = source.downcast_ref::<PyStr>() else {
return Err(vm.new_type_error(format!("expected str, not {}", source.class().name())));
};
let lines = source_str.splitlines(SplitLinesArgs { keepends: false }, vm);
if lineno == 0 {
return Err(vm.new_index_error("list index out of range"));
}
lines
.get(lineno - 1)
.cloned()
.ok_or_else(|| vm.new_index_error("list index out of range"))
.map(Some)
}
/// Get the default action from `sys.modules['warnings'].defaultaction`,
/// falling back to vm.state.warnings.default_action.
fn get_default_action(vm: &VirtualMachine) -> PyResult<PyObjectRef> {
if let Some(action) = get_warnings_attr(vm, identifier!(&vm.ctx, defaultaction), false) {
if let Some(action) = get_warnings_attr(vm, identifier!(vm, defaultaction), false)? {
if !action.class().is(vm.ctx.types.str_type) {
return Err(vm.new_type_error(format!(
"_warnings.defaultaction must be a string, not '{}'",
action.class().name()
)));
}
return Ok(action);
let action = PyStrRef::try_from_object(vm, action)?;
vm.state
.warnings
.default_action
.swap_to_temporary_refs(action, vm);
}
Ok(vm.state.warnings.default_action.clone().into())
Ok(vm.state.warnings.default_action.to_owned().into())
}
/// Get the once registry from `sys.modules['warnings']._onceregistry`,
/// Get the once registry from `sys.modules['warnings'].onceregistry`,
/// falling back to vm.state.warnings.once_registry.
fn get_once_registry(vm: &VirtualMachine) -> PyResult<PyObjectRef> {
if let Some(registry) = get_warnings_attr(vm, identifier!(&vm.ctx, onceregistry), false) {
if let Some(registry) = get_warnings_attr(vm, identifier!(vm, onceregistry), false)? {
if !registry.class().is(vm.ctx.types.dict_type) {
return Err(vm.new_type_error(format!(
"_warnings.onceregistry must be a dict, not '{}'",
registry.class().name()
)));
}
return Ok(registry);
let registry = PyDictRef::try_from_object(vm, registry)?;
vm.state
.warnings
.once_registry
.swap_to_temporary_refs(registry, vm);
}
Ok(vm.state.warnings.once_registry.clone().into())
Ok(vm.state.warnings.once_registry.to_owned().into())
}
fn already_warned(
@@ -253,20 +346,18 @@ fn normalize_module(filename: &Py<PyStr>, vm: &VirtualMachine) -> PyObjectRef {
}
}
/// Search the global filters list for a matching action.
// TODO: split into filter_search() + get_filter() and support
// context-aware filters (get_warnings_context_filters).
fn get_filter(
/// Search a filters list for a matching action.
fn filter_search(
category: &PyObject,
text: &PyObject,
lineno: usize,
module: &PyObject,
filters: &PyListRef,
list_name: &str,
vm: &VirtualMachine,
) -> PyResult {
let filters = get_warnings_filters(vm);
) -> PyResult<Option<PyObjectRef>> {
// filters could change while we are iterating over it.
// Re-check list length each iteration (matches C behavior).
// Re-check list length each iteration.
let mut i = 0;
while i < filters.borrow_vec().len() {
let Some(tmp_item) = filters.borrow_vec().get(i).cloned() else {
@@ -276,22 +367,56 @@ fn get_filter(
.ok()
.filter(|t| t.len() == 5)
.ok_or_else(|| {
vm.new_value_error(format!("_warnings.filters item {i} isn't a 5-tuple"))
vm.new_value_error(format!("warnings.{list_name} item {i} isn't a 5-tuple"))
})?;
/* action, msg, cat, mod, ln = item */
let action = &tmp_item[0];
if !action.class().is(vm.ctx.types.str_type) {
return Err(vm.new_type_error(format!(
"action must be a string, not '{}'",
action.class().name()
)));
}
let good_msg = check_matched(&tmp_item[1], text, vm)?;
let is_subclass = category.is_subclass(&tmp_item[2], vm)?;
let good_mod = check_matched(&tmp_item[3], module, vm)?;
let ln: usize = tmp_item[4].try_int(vm).map_or(0, |v| v.as_u32_mask() as _);
if good_msg && is_subclass && good_mod && (ln == 0 || lineno == ln) {
return Ok(action.to_owned());
return Ok(Some(action.to_owned()));
}
i += 1;
}
Ok(None)
}
fn get_filter(
category: &PyObject,
text: &PyObject,
lineno: usize,
module: &PyObject,
vm: &VirtualMachine,
) -> PyResult {
if let Some(context_filters) = get_warnings_context_filters(vm)? {
if let Some(action) = filter_search(
category,
text,
lineno,
module,
&context_filters,
"_warnings_context _filters",
vm,
)? {
return Ok(action);
}
return get_default_action(vm);
}
let filters = get_warnings_filters(vm)?;
if let Some(action) = filter_search(category, text, lineno, module, &filters, "filters", vm)? {
return Ok(action);
}
get_default_action(vm)
}
@@ -320,6 +445,8 @@ pub fn warn_unawaited_coroutine(coro: &PyObject, qualname: &Py<PyStr>, vm: &Virt
let mut warned = false;
if let Some(helper) =
get_warnings_attr(vm, vm.ctx.intern_str("_warn_unawaited_coroutine"), true)
.ok()
.flatten()
{
match helper.call((coro.to_owned(),), vm) {
Ok(_) => warned = true,
@@ -491,7 +618,7 @@ fn call_show_warning(
source: Option<PyObjectRef>,
vm: &VirtualMachine,
) -> PyResult<()> {
let Some(show_fn) = get_warnings_attr(vm, identifier!(&vm.ctx, _showwarnmsg), source.is_some())
let Some(show_fn) = get_warnings_attr(vm, identifier!(vm, _showwarnmsg), source.is_some())?
else {
show_warning(&filename, lineno, &text, &category, source_line, vm);
return Ok(());
@@ -501,8 +628,7 @@ fn call_show_warning(
return Err(vm.new_type_error("warnings._showwarnmsg() must be set to a callable"));
}
let Some(warnmsg_cls) = get_warnings_attr(vm, identifier!(&vm.ctx, WarningMessage), false)
else {
let Some(warnmsg_cls) = get_warnings_attr(vm, identifier!(vm, WarningMessage), false)? else {
return Err(vm.new_runtime_error("unable to get warnings.WarningMessage"));
};
@@ -528,7 +654,7 @@ fn show_warning(
lineno: usize,
text: &Py<PyStr>,
category: &Py<PyType>,
_source_line: Option<PyObjectRef>,
source_line: Option<PyObjectRef>,
vm: &VirtualMachine,
) {
let stderr = crate::stdlib::sys::PyStderr(vm);
@@ -540,6 +666,11 @@ fn show_warning(
category.name(),
text
);
if let Some(source_line) = source_line
&& let Ok(line) = source_line.str(vm)
{
writeln!(stderr, "{line}");
}
}
/// Check if a frame's filename starts with any of the given prefixes.
src/shell.rs
+20 -39
View File
@@ -44,7 +44,6 @@ fn shell_exec(
*future_features |= code.code.flags & CodeFlags::FUTURE_MASK;
let _ = vm.register_code_in_linecache(&code, source);
if empty_line_given || !continuing_block {
// We want to execute the full code
match vm.run_code_obj(code, scope) {
Ok(_val) => ShellExecResult::Ok,
Err(err) => ShellExecResult::PyErr(err),
@@ -54,20 +53,21 @@ fn shell_exec(
ShellExecResult::Ok
}
}
Err(VmCompileError::Compile(CompileError::Parse(ParseError {
error: ParseErrorType::Lexical(LexicalErrorType::Eof),
..
}))) => ShellExecResult::ContinueLine,
Err(VmCompileError::Compile(CompileError::Parse(ParseError {
error:
ParseErrorType::Lexical(LexicalErrorType::FStringError(
InterpolatedStringErrorType::UnterminatedTripleQuotedString,
)),
..
}))) => ShellExecResult::ContinueLine,
Err(err) => {
// Check if the error is from an unclosed triple quoted string (which should always
// continue)
if matches!(
&err,
VmCompileError::Compile(CompileError::Parse(ParseError {
error: ParseErrorType::Lexical(
LexicalErrorType::Eof
| LexicalErrorType::FStringError(
InterpolatedStringErrorType::UnterminatedTripleQuotedString,
)
),
..
}))
) {
return ShellExecResult::ContinueLine;
}
if let VmCompileError::Compile(CompileError::Parse(ParseError {
error: ParseErrorType::Lexical(LexicalErrorType::UnclosedStringError),
raw_location,
@@ -82,33 +82,14 @@ fn shell_exec(
{
return ShellExecResult::ContinueLine;
}
};
}
// bad_error == true if we are handling an error that should be thrown even if we are continuing
// if its an indentation error, set to true if we are continuing and the error is on column 0,
// since indentations errors on columns other than 0 should be ignored.
// if its an unrecognized token for dedent, set to false
let bad_error = match &err {
VmCompileError::Compile(CompileError::Parse(p)) => {
match &p.error {
ParseErrorType::Lexical(LexicalErrorType::IndentationError) => {
continuing_block
} // && p.location.is_some()
ParseErrorType::OtherError(msg) => {
!msg.starts_with("Expected an indented block")
}
_ => true, // !matches!(p, ParseErrorType::UnrecognizedToken(Tok::Dedent, _))
}
}
_ => true, // It is a bad error for everything else
};
// If we are handling an error on an empty line or an error worthy of throwing
if empty_line_given || bad_error {
ShellExecResult::PyErr(err.into_pyexception(vm, Some(source)))
} else {
// An unfinished suite is _IncompleteInputError, not IndentationError.
let exc = err.into_pyexception_maybe_incomplete(vm, Some(source), !empty_line_given);
if !empty_line_given && exc.fast_isinstance(vm.ctx.exceptions.incomplete_input_error) {
ShellExecResult::ContinueBlock
} else {
ShellExecResult::PyErr(exc)
}
}
}