17 Commits
Author SHA1 Message Date
youknowdot d8b8e47d74 Sync canonical asyncio, marshal and offline profiling libraries (#9315)
Migrate nine substantial stdlib and canonical test modules from CPython
v3.15.0 through scripts/update_lib, preserving RustPython adaptations.
Include offline stack formatting without live process inspection or
changes to the held profiling frontend and test-support closure.

Assisted-by: Codex:unknown
2026-10-23 17:56:04 +09:00
Jeong, YunWonandLee Dogeon afad92bf1b Add OrderedDict, marshal, inspect, and chflags bitflagset (#8780)
* Implement _collections.OrderedDict

Extracted from #6799: native OrderedDict as a dict subclass,
iterator mutation tracking, and mapping-view set operations.

* Add OrderedDict, warnings source lines, and socket sendmsg

Implement _collections.OrderedDict as a dict subclass with a
linked list of insertion-order nodes.

Pack struct F/D as native float/double complex pairs, and report
unexpected __replace__ field names as a list repr.

Register encodings.latin_1 at startup. Reject NUL in input()
prompts. Copy non-exact type dicts in mapping order.

Fetch warning source lines through _bless_my_loader and unicode
splitlines. Keep the last warnings.filters list after deletion.

Snapshot sendmsg sequences before converting buffers, parse
ancillary ints with __index__, and dispatch AF_CAN by protocol.

Store sys.trace and sys.profile on each ThreadSlot so
_settraceallthreads/_setprofileallthreads are visible to
sys.gettrace/getprofile on other threads.

Use <frozen importlib._bootstrap> as frozen importlib
co_filename. Keep HEAPTYPE socket objects GC-tracked.

Assisted-by: Grok:grok-4.6

* Load marshal via readinto and dump sets in dumps order

marshal.load fills a scratch buffer with readinto on a
contiguous writable memoryview, and rejects a returned length
other than the requested size.

set and frozenset dumps write elements in sorted(v,
key=marshal.dumps) order.

Assisted-by: Grok:grok-4.6

* Add classmethod_descriptor and ABC IS_ABSTRACT flags

Give METH_CLASS methods a distinct classmethod_descriptor type.
Store Py_TPFLAGS_IS_ABSTRACT from __abstractmethods__ and look
up that attribute only in the type's own dict.

Expose defaultdict.default_factory as a member descriptor.
Inherit bool's unused number slots from int. Mark set and
frozenset __contains__ as METH_COEXIST.

Parse classmethod, staticmethod, object.__new__, and heap type
text signatures. Register interactive source in linecache.

Assisted-by: Grok:grok-4.6

* address review: odict reuse, descr call, new wrapper

Reuse odict node slots and walk the live list in next_after.
Require both | operands to be dicts. Drop the odict iterator
on GC clear. Bind classmethod descriptors through descr_get
and update IS_ABSTRACT under the type lock. Give odict keys
and items sq_contains. Walk tp_new_wrapper past new_wrapper
so HEAPTYPE socket subclasses can call socket.__new__.
Split FreeBSD Bluetooth protocol constants from Linux values.

Assisted-by: Grok:grok-4.6

* Represent chflags bits with bitflagset

Assisted-by: Grok:grok-4.6

* Represent access and mode bits with bitflagset

Assisted-by: Grok:grok-4.6

* Continue unfinished REPL suites as incomplete input

Assisted-by: Grok:grok-4.6

* Add pickle and os.stat inspect signatures

Give _pickle.Pickler the clinic constructor text signature and
accept fix_imports and buffer_callback as positional-or-keyword.
Allow #[pyfunction(text_signature)] and use it so os.stat reports
(path, *, dir_fd=None, follow_symlinks=True). Uncomment import
_pickle and drop the pickle inspect expectedFailures.

Assisted-by: Grok:grok-4.6

* Add Unpickler, dump/load, and lstat signatures

Give _pickle Unpickler, dump, dumps, load, and loads clinic
text signatures, and os.lstat (path, *, dir_fd=None).

Assisted-by: Grok:grok-4.6

* Complete OrderedDict with insertion-order links

Move the native OrderedDict into ordered_dict.rs and track order
with a linked list of nodes, keeping mapping-view set operations.

Assisted-by: Grok:grok-4.6

* address CI: stat casts, HCI_DATA_DIR, passing xfails

Compare stat permission bits as Mode so Linux clippy does not
flag a u32-to-u32 cast. Export HCI_DATA_DIR with the Bluetooth
socket option values. Drop expectedFailure from tests that now
pass.

Assisted-by: Grok:grok-4.6

* Fix wasm teardown deadlock and bluetooth constants

Assisted-by: Grok:grok-4.6

* Clear process-wide eval breaker bits in STW test

Assisted-by: Grok:grok-4.6

* Drop exclusive-held panic that tripped wasm clippy

Assisted-by: Grok:grok-4.6

* Fix OrderedDict contains and export Windows BDADDR

Assisted-by: Grok:grok-4.6

* Add cmp_to_key and hash-once LRU cache

_functools.cmp_to_key now returns KeyWrapper. LRU lookups store the
key hash on _lru_list_elem so __hash__ is not called again on hits
or eviction.

Assisted-by: Grok:grok-4.6

* Allow subclass keywords on list, tuple, and float

list.__init__ rejects keywords only when tp_new is still list's.
tuple/float.__new__ reject them only when tp_init is still the base's.

Assisted-by: Grok:grok-4.6

* Fix CI lint, test re-export, and structseq members

Drop the duplicate clear_eval_breaker_for_test re-export.
Format structseq imports. Expose struct sequence fields as
readonly members so assignment raises "readonly attribute".

Assisted-by: Grok:grok-4.6

* Allow subclass keywords on itertools and filter

Positional-only constructors ignore extra keywords when a subclass
has replaced tp_init, matching list/tuple/float.

Assisted-by: Grok:grok-4.6

* Remove text_signature derive parameter

Drop the string override on #[pyfunction] and #[pyclass]. Signatures
are generated from the Rust parameter list only. Restore inspect
expectedFailures that depended on the override.

Assisted-by: Grok:grok-4.6

* Rename set contains_method to __contains__

Keep a borrowed contains helper for slots and C-API. The coexist
pymethod is named __contains__.

Assisted-by: Grok:grok-4.6

* Use MonitoringEvent instead of EVENT_* aliases

Drop the unused MonitoringEvents bitflags and the EVENT_PY_START
style mask constants. Call sites use the enum and .mask().

Assisted-by: Grok:grok-4.6

* Store warnings filters as PyListRef

The list already serializes its elements. Drop the extra mutex around
the cached list pointer.

Assisted-by: Grok:grok-4.6

* Re-export clear_eval_breaker_for_test after rebase

Keep the STW test helper visible without bringing back stop_bit_set.

Assisted-by: Grok:grok-4.6

* Fix groupby reentrancy and islice arguments

Give _grouper its own tgtkey and take currvalue after the key
comparison. Accept __index__ on islice bounds, reject step 0, and
drain the source up to stop.

Assisted-by: Grok:grok-4.6

* SETREF warnings filters, onceregistry, and defaultaction

Keep the last list, dict, or str after the module attributes are
deleted by swapping the interpreter cache pointer. Do not store a
None fallback for _warnings_context.

Assisted-by: Grok:grok-4.6

* Iterate OrderedDict inner dict via next_entry

&PyDict is not an iterator after the borrowed-ref signature change.

Assisted-by: Grok:grok-4.6

* Remove expectedFailure from testBluetoothConstants

Linux now exports BTPROTO_HCI and the rest of the set.

Assisted-by: Grok:grok-4.6

* Borrow get_filter arguments as PyObject

Match the warn_explicit call site that passes borrowed
handles after the internal-helper borrow change.

Assisted-by: Grok:grok-4.6

* Fix bool slots and SemHandle safety docs

Leave unused bool number slots unset so inspect reports
__add__ on int. Document SemHandle::from_raw safety.

Assisted-by: Grok:grok-4.6

* Keep sendmsg errors and drop duplicate linecache

Replace sendmsg iterable TypeError only. Register REPL
source in linecache once per compiled input.

Assisted-by: Grok:grok-4.6

* Nest REPL parse or-patterns for clippy

Assisted-by: Grok:grok-4.6

* Mark pydoc os.stat signature test expectedFailure

Assisted-by: Grok:grok-4.6

---------

Co-authored-by: Lee Dogeon <dev.moreal@gmail.com>
2026-09-24 12:21:26 +09:00
Luan Taraschi 0c9fa4219d Stop skipping six tests whose reason no longer reproduces (#8583)
Three skips in test_array say "Hangs", one in test_marshal carries no
reason at all, and two in test_thread say "panic" on Linux. None of the
six does any of that now. Each ran three to five times in a row and
passed, and the array trio was also checked case by case against CPython
3.14, including the mutation-during-index-conversion shapes the tests
were written for.

The neighbouring markers that say "Flaky", "Flakey on CI" and "flaky
EOFError" are left alone. One green run says nothing about those.

Assisted-by: Claude Code:claude-opus-5
2026-08-23 08:33:33 +09:00
Jeong, YunWon 1d6c6aece0 marshal: refuse a back-reference to an object still being written (#8519)
`w_ref` marks a code or slice entry incomplete until `w_complete`, because
the reader rebuilds both from their fields and a `TYPE_REF` issued while
those fields are still on the wire names an object that does not exist
yet.  `WriterRefTable` carries that marker and `write_object_depth`
raises `cannot marshal recursion <type> objects` instead of emitting the
reference.

`test_reference_loop_code`, `test_unmarshallable` and
`test_reference_loop_slice` lose their RustPython markers; `test_marshal`
is 75 run, 16 skipped.

Assisted-by: Claude
2026-08-14 12:58:42 +09:00
Jeong, YunWon 7108ff0f69 marshal: round-trip code constants through the runtime bag (#8516)
* Keep abnormal marshal loop test aligned with CPython

Assisted-by: OpenAI Codex:gpt-5.5

* marshal: retain runtime code constants while loading

Decode code fields through the runtime MarshalBag so co_consts values that do not fit the compiler constant enum remain available to VM code wrappers while the compiler table receives shape placeholders.

Assisted-by: OpenAI Codex:GPT-5.4

* marshal: write and read co_consts through the runtime bag

serialize_code gains a serialize_code_with variant that writes each
co_consts entry through a caller-supplied writer; serialize_code keeps
the BorrowedConstant writer as its default.  The VM writer passes its own
write_object_depth, so a code constant that Literal holds but
BorrowedConstant cannot describe reaches the stream instead of panicking
in borrow_obj_constant, and a constant shared with the enclosing object
takes an entry in the writer's reference table.

PyMarshalBag implements constant_ref_from_value, bytes_from_value,
str_from_value and tuple_elements_from_value, which
deserialize_code_value_inner requires to decode code fields through the
runtime bag.

Assisted-by: Claude
2026-08-14 03:30:51 +09:00
Jeong, YunWon 84a0ba7864 Fix marshal recursive reference loading (#8501)
* Fix marshal recursive reference loading

Create reference-tracked containers before reading their children so recursive list, dict, set, and tuple graphs can be unmarshaled. Preserve interned string markers through the runtime bag and add an initialization-only tuple construction path.

Assisted-by: Codex:gpt-5

* Preserve marshal container insertion errors

Keep Python exceptions raised while constructing unmarshaled sets, frozensets, and dictionaries instead of collapsing them into ValueError. This makes abnormal recursive hash-container streams report TypeError like CPython and removes the remaining test_marshal expected failure.

Assisted-by: Codex:gpt-5

* Enable full abnormal marshal reference loop test

* Test direct marshal tuple reference loop

Assisted-by: Codex:gpt-5
2026-08-12 16:28:09 +09:00
Shahar Naveh c90a1a9644 Update test_marshal.py to 3.14.5 (#7865)
* Update `test_marshal.py` to 3.14.5

* mark failing tests
2026-05-12 20:53:54 +09:00
Jeong, YunWon 98dba36af5 marshal (#7467)
* CPython-compatible marshal format

Unify marshal to a single CPython-compatible format. No separate
"cpython_marshal" reader — one format for frozen modules, .pyc
files, and the Python-level marshal module.

- ComparisonOperator: `(cmp_index << 5) | mask` matching COMPARE_OP
- MakeFunctionFlag: bit-position matching SET_FUNCTION_ATTRIBUTE
- Exception table varint: big-endian (matching Python/assemble.c)
- Linetable varint: little-endian (unchanged)
- Integer: TYPE_INT (i32) / TYPE_LONG (base-2^15 digits)
- Code objects: CPython field order (argcount, posonlyargcount, ...,
  co_localsplusnames, co_localspluskinds, ..., co_exceptiontable)

- FLAG_REF / TYPE_REF for object deduplication (version >= 3)
- allow_code keyword argument on dumps/loads/dump/load
- Subclass rejection (int/float/complex/tuple/list/dict/set/frozenset)
- Slice serialization (version >= 5)
- Buffer protocol fallback for memoryview/array
- Recursion depth limit (2000) for both reads and writes
- Streaming load (reads one object, seeks file position)
- TYPE_INT64, TYPE_FLOAT (text), TYPE_COMPLEX (text) for compat

serialize_code writes co_localsplusnames/co_localspluskinds from
split varnames/cellvars/freevars. deserialize_code splits them back.
Cell variable DEREF indices are translated between flat (wire) and
cell-relative (internal) representations in both directions.

Replace bitwise trick with match for new ComparisonOperator values.

21 -> 3 expected failures. Remaining: test_bad_reader (IO layer),
test_deterministic_sets (PYTHONHASHSEED), testIntern (string interning).

* Address code review: preserve CO_FAST_HIDDEN, fix varint overflow

- Use original localspluskinds from marshal data instead of
  rebuilding, preserving CO_FAST_HIDDEN and other flags
- Fix write_varint_be to handle values >= 2^30 (add 6th chunk)
- Remove unused build_localspluskinds_from_split

* Add depth guard to deserialize_value_typed

Prevents usize underflow when dict key deserialization path calls
deserialize_value_typed with depth=0 on composite types.
2026-03-23 13:10:51 +09:00
Jeong, YunWon 8a05c5ca94 Fix test_code: compiler and code object improvements
- Add CO_NESTED flag (0x10) for nested function scopes
- Emit LOAD_SMALL_INT for integers 0..=255 instead of LOAD_CONST
- Eliminate dead constant expression statements (no side effects)
- Ensure None in co_consts for functions with no other constants
- Add code.__replace__() for copy.replace() support
- Mark test_co_lnotab and test_invalid_bytecode as expectedFailure
2026-03-17 10:54:12 +09:00
Shahar NavehandLee Dogeon 692c777e48 Update test_marshal.py from 3.14.3 (#7409)
---------

Co-authored-by: Lee Dogeon <dev.moreal@gmail.com>
2026-03-14 09:45:12 +09:00
Terry Tianlin LuanandJeong, YunWon 609dbb1439 Downgraded skips in tests (#6716)
* Downgraded skips in tests

* Fixed failing tests

* Fixed test_ftplib + test_socket + test_ssl + test_threaded_import failures

* Removed comments about which tests are run in which environment

* Addressed PR comments

* Annotated skips on failing tests

* Removed unneeded tests

* Removed unneeded sys import from test_ftplib

* Added annotation to test_ftplib

* Readded skipIf to test_cleanup_with_symlink_modes with a more general ENV_POLLUTING_TESTS_WINDOWS

* Addressed PR comments

* Made changes to minimize diff in PR

* Apply suggestion from @youknowone

---------

Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
2026-01-15 21:43:22 +09:00
Kim, YeonWoo b5aa8c8284 Add StopIteration type to marshal 2022-10-25 20:48:14 +09:00
Jeong YunWon ec9fa505e3 marshal.version 2022-08-07 04:15:21 +09:00
Jeong YunWon 6489ef153a bytes, none, ellipsis 2022-08-07 04:13:21 +09:00
Jeong YunWon a353d5e27a redesign marshal 2022-08-07 02:40:40 +09:00
Jeong Yunwon 6ff41a24c2 mark failing tests 2022-05-03 23:58:42 +09:00
Jeong Yunwon 66afb64f79 Add Lib/test/test_marshal.py from CPython 3.10.4 2022-05-03 23:56:41 +09:00