* Add deterministic profiler and canonical Python 3.15 tools
Use the merged operation critical sections with interpreter-bound profiler
entry checks and short payload borrows. Trace owned code references for GC.
Import the canonical tracing and offline profiling output closure via the
repository updater, retaining existing test markers.
Assisted-by: dot:GPT-6.1
* Clarify standalone profiling license cross-reference
Preserve license terms and copyright notices; point to the reproduced
Apache text below instead of CPython's original OpenSSL section above.
Assisted-by: dot:GPT-6.1
* Defer unsupported sampling package from profiler extraction
Keep the deterministic tracing profiler and its canonical tools together.
Defer the entirely new sampling production, tests, assets and their license
as one closure until native record and binary-collector support is ready.
Retain canonical pstats for deterministic empty-profile handling; its
upstream sampled-file sorting issue is not patched here.
Assisted-by: dot:GPT-6.1
* Stage profiling root as a namespace until sampling is available
Omit the whole newly added canonical package initializer rather than
advertise an absent sampling module or hand-edit canonical __all__.
Explicit profiling.tracing and cProfile imports remain available.
The namespace root intentionally lacks canonical metadata/automatic exports.
This addresses test___all__ failures in the prior Linux, macOS and Windows
CI runs. No test assertions, skips or canonical file contents are changed.
Assisted-by: dot:GPT-6.1
* Match canonical static builtin profiler labels and metadata
Normalize the raw builtin receiver instead of its call-argument accessor.
Snapshot static method module metadata at construction, matching CPython,
so explicit module mutation and deletion remain observable. Avoid a
profiler-time fallback that would obscure deleted metadata.
Correct float.__getformat__ to CPython's classmethod calling convention,
including subclass binding, rather than special-casing its profiler label.
Assisted-by: dot:GPT-6.1
* Support collision-safe frozen package containers
Generate empty frozen packages only for source namespace directories with
real frozen descendants. Preserve genuine modules and packages across
frozen-library registrations while retaining real-real replacement order.
Record the container marker in FrozenModule and its library envelope;
legacy module/package encodings remain unchanged. This is closed-tree
frozen packaging, not full PEP 420 namespace merging.
Add structural codec, mock generator, and registration-priority checks.
* Revert frozen package containers before restoring canonical profiling root
The container implementation changed filesystem namespace precedence.
Restore the existing importer and frozen envelope unchanged; the following
offline sampling closure restores the canonical regular package root.
This reverses d5a8049234.
Assisted-by: dot:GPT-6.1
* Restore canonical profiling package with offline sampling codec
Provide native records, status constants and genuine uncompressed binary
reader/writer APIs for canonical offline tools. Keep the unsupported live
process capability absent and preserve existing canonical test gates.
Restore the complete selected sampling production/test/assets closure via
update_lib, including its asyncio frame-schema dependency. The prior frozen
container extension is reverted separately; canonical packages work with
both filesystem and frozen imports without changing namespace semantics.
Assisted-by: dot:GPT-6.1
* Keep platform-dependent C long normalization explicit
Allow only the c_long-to-i32 conversion lint: Windows uses the same type,
while LP64 requires the existing truncation to match CPython normalization.
No runtime behavior or test assertions change.
Assisted-by: dot:GPT-6.1
* Include profiling notices in library and desktop packages
Bundle a byte-identical copy of the root profiling notice in rustpython-pylib
and include the root notice in configured desktop resources. Avoid a symlink
so Windows checkouts preserve the actual notice text. Terms are unchanged.
Assisted-by: dot:GPT-6.1
* Keep profiling asset license notice in pylib crate
* Serve frozen profiling assets through native resource readers
* Preserve canonical profiling asset line endings on checkout
---------
Co-authored-by: youknowdot <dot@youknowone.org>
* Factor shared interpreter stop-the-world guard
Move the existing collector barrier into the VM without changing its stop, restart or ownership behavior, preparing reuse by fork.
Assisted-by: dot:GPT-6.1
* Add interpreter-bound operation section primitive
Prepare attached-only ownership acquisition and detached observer parking with private per-interpreter scope stacks. Lifecycle wiring follows separately; no profiler consumer is enabled by this commit.
Assisted-by: dot:GPT-6.1
* Suspend operation sections at VM lifecycle boundaries
Integrate detach, stop-the-world, shutdown and interpreter-switch boundaries. Reuse the all-interpreter barrier for fork repair and add focused native ownership invariants.
Assisted-by: dot:GPT-6.1
* Replace operation sections with PyMutex critical sections
Add PyMutex with the lock.c protocol: LOCKED/HAS_PARKED bits, a bounded
spin, direct handoff to waiters parked longer than 1 ms, and detaching
while parked.
Rebuild critical sections on PyMutex with the stack kept as a tagged
pointer in ThreadSlot: single and two-mutex sections, recursive
top-most skipping, locking skipped while the world is stopped,
SuspendAll on detach/suspend/shutdown and Resume of the top-most
section on attach. Fork carries the stacks into the replacement slots.
Assisted-by: Claude Code:claude-opus-5-5
* Check attachment around critical-section suspension
Assert that a thread entering a critical section is attached, and
suspend sections in do_suspend only while the slot is still ATTACHED.
Assisted-by: Claude Code:claude-opus-5-5
---------
Co-authored-by: Jeong YunWon <jeong@youknowone.org>
* Use the RustPython Ruff fork 0.17.1 from git and place EOF errors by kind
- The ruff crates come from the RustPython/ruff `rustpython-0.17.1`
branch at 93967836c8 instead of the 0.16.10 crates.io release.
- `EofToken.known_location` is replaced by `EofErrorLocation`
(Tokenizer, Token, StartingFromToken), chosen from the parse error kind.
The keyword scan in `clause_keyword` is removed, since the parser now
reports a forced `:` as `ExpectedForcedToken`.
- An empty import at EOF ends at the end of the line read, and an empty
import before a newline is an empty range at the newline.
Assisted-by: Claude Code:claude-opus-5-5
* Report the obsolete `<>` from the parser outside Barry mode
- Bump the RustPython/ruff fork to 9290a88495, where the parser reports an
adjacent `<` and `>` in a comparison as "invalid syntax. Maybe you meant
'!=' instead of '<>'?" over both tokens.
- Outside Barry mode, `BarrySource` no longer searches the source text for
`<>`, so the parser's error is reported unchanged. A `<>` that starts a
statement is reported at the `<` only, and `<<>` at the `>`.
Assisted-by: Claude Code:claude-opus-5-5
Replaces: #9166
unicode_names2 fixed a bug with its aliases. Ironically, the bug
caused a test in test_snippets.py to pass. Bumping unicode_names2 to
4.0.0 fixes the bug and thus the extra 3.2.0 test fails. I fixed the
logic for name lookup for 3.2.0. Now the test correctly passes.
* Add Python 3.15 math APIs using pymath
Pin pymath to merged dd1e9a4cf26d68caf3b549c09476f52e356e16a3
and expose isnormal, issubnormal, signbit, fmax and fmin through thin
wrappers. Share the existing integer functions through math.integer
and _math_integer, matching CPython module identity and metadata.
Migrate test_math.py and test_math_integer.py via scripts/update_lib
from CPython d220409a13b77de7a6c30abe16ea90db01528edc, preserving the
existing Windows remainder marker. Add bounded Python snippet coverage.
AI assistance: OpenAI Codex implemented and source-reviewed this change,
ran update_lib, and performed focused automated validation. No human
manual testing is claimed.
Assisted-by: OpenAI Codex:model version unavailable
* Track integer math tests in library updates
Assisted-by: OpenAI Codex:model version unavailable
pymath requires malachite-bigint ^0, so the example projects, which
have no lock file, resolve it to 0.14.0 while the workspace pinned
0.13.0, and fail to build.
Assisted-by: Claude Code:claude-opus-5-5
Represent PyObjectRef through PyRef<PyObject> while preserving typed
ownership, payload projection, and drop behavior. Use the sealed target
selector for borrowed set/frozenset union views and typed tuple elements.
Keep tuple storage erased behind invariant typed slice views, with checked
conversions and reference-count ownership tests. Preserve exact tuple
identity for normalized nonempty full slices. Integrate the ownership
contract into the public object module documentation and retain focused
Rust ownership/layout tests without modifying bundled Lib sources.
Spell the set/frozenset union owner directly as PyRef<PyAnySet> without
a second owned alias; retain the sealed internal view registration.
Assisted-by: OpenAI Codex:model identifier unavailable
* Implement Python 3.15 base64 encoding and decoding
Follow CPython 3.15.0 final d220409a13b77de7a6c30abe16ea90db01528edc for base64 options, alphabets, padding and canonical decoding. Keep reverse-table caches in traced module-native state and preserve module-bound callable semantics. Add bounded Python regressions without editing Lib.
Assisted-by: OpenAI Codex:model identifier unavailable
* Complete binascii codecs for canonical Python 3.15 tests
Migrate Lib/test/test_binascii.py unchanged from CPython 3.15.0 final using scripts/update_lib and replace the superseded custom base64 snippet. Implement the base32, Base85, Ascii85 and hex ignorechars APIs required by the canonical suite. Share the existing module-bound original-key alphabet cache across base32/base64/base85 and preserve argument and buffer cleanup semantics. Keep the existing base64 kernels and VM state integration unchanged.
Assisted-by: OpenAI Codex:model identifier unavailable
* Match CPython's SyntaxError range for unparenthesized `except` types
CPython's `invalid_except_stmt` rule raises the error only once the whole
clause has matched, and reports a range that starts at the first exception
type and ends at the `:` closing the clause, so it covers the `as NAME`
part as well:
invalid_except_stmt:
| 'except' a=expression ',' expressions 'as' NAME ':' {
RAISE_SYNTAX_ERROR_STARTING_FROM(a, "multiple exception types
must be parenthesized when using 'as'") }
The parser reports the exception types alone, so look up that `:` in the
source and widen the range to match. Only `as NAME` may follow the
exception types, which is why the first `:` after them is the one closing
the clause.
try:
pass
except A, B, C as e:
pass
CPython 3.14: ('x.py', 3, 8, 'except A, B, C as e:\n', 3, 20)
before: ('x.py', 3, 8, 'except A, B, C as e:\n', 3, 15)
Fixes#8496
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Clarify that the `except` error range stops before the closing colon
`invalid_except_stmt_end` returns the exclusive end of the range, so the
column it yields is the one the `:` sits on even though the `:` is not part
of the range. The previous "ending at the `:`" wording read as if the colon
were included, which invites an off-by-one "fix".
Comments only; no behavior change.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Fix the `except ... as` error column for non-ASCII names and line joins
`invalid_except_stmt_end` recovers the end of CPython's range by scanning the
source, which has to redo by hand what CPython gets from its tokenizer:
- The column was the byte distance from the line start, but `end_offset` is a
character column. `except Ä, B as e:` reported 18 instead of 17. CPython
converts explicitly, in `_PyPegen_byte_offset_to_character_offset`.
- The colon scan already spans explicit line joins, but the `as` check ran on
the raw slice, so a `\` before `as` left the range unextended.
`except A, B \` + `as exc:` reported (3, 12) instead of (4, 7).
Add regression cases to `syntax_invalid.py` covering both, plus the whitespace,
`except*` and line-join placements that already worked. All values are verified
against CPython 3.14.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Update the Ruff fork to 0.16.10
Point the rustpython-ruff_* dependencies at the 0.16.10-rustpython tag
of the RustPython Ruff fork and port to its AST changes:
`ExprCompare` stores all operands in `operands`, call arguments use
`ThinVec`, dict comprehension generators and f-string elements are boxed
slices, `FStringValue` iterates `FStringPartRef`, and `TokenKind::Name`
is now `TokenKind::Identifier`.
Assisted-by: Grok:grok-4.6
Assisted-by: Claude Code:claude-opus-5-5
* Drop the source scan for the `except ... as` error range
The parser now reports the unparenthesized exception types error with a
range ending at the `:` that closes the clause, so
`invalid_except_stmt_end` and the message match that widened the range
are removed.
Assisted-by: Claude Code:claude-opus-5-5
* Take parse error messages from the parser as they are
The Ruff fork now emits the final message texts, so `vm_new.rs` no
longer rewrites parse error messages by matching their text and no
longer lowercases the first character of every message.
`analyze_compile_error` keeps only the caret narrowing for starred
expressions and the line number in the unterminated string message.
The compiler matches the new lowercase texts for indented block and
mixed bytes literal errors.
The fork is referenced by revision until it is published.
Assisted-by: Claude Code:claude-opus-5-5
* Take starred and bytes literal error ranges from the parser
The Ruff fork now reports misplaced starred expressions at the `*` and
non-ASCII bytes literal errors over the whole literal. `vm_new.rs` drops
`narrow_caret` together with `SyntaxErrorInfo`, which only carried the
message after that, and the compiler drops `bytes_literal_span`.
Assisted-by: Claude Code:claude-opus-5-5
* Take indentation error kinds and positions from the parser
The Ruff fork now reports `TabError` and `TooDeepIndentation` as their
own lexer errors and places indentation and line continuation errors
itself. `vm_new.rs` picks TabError or IndentationError from the error
kind instead of scanning the source for mixed indentation, drops the
TabError message override, and sets the end offset of tokenizer errors
to 0 or -1 by kind. The compiler drops the line-end relocation of
unindent errors and the line continuation rewrite. `_tokenize` raises
TabError and depth errors for the token they replace, also for sources
containing tabs.
Remove expectedFailure from test_tokenize.test_max_indent and four
test_tabnanny tests.
Assisted-by: Claude Code:claude-opus-5-5
* Take unterminated string messages from the parser
The Ruff fork's unterminated string errors now include the detected
line and are reported at the start of the string. `vm_new.rs` drops its
message override for `UnclosedStringError`, and the shell continues a
line on an unterminated triple-quoted string by the error's
`triple_quoted` flag instead of reading the quotes from the source.
Assisted-by: Claude Code:claude-opus-5-5
* Take bracket errors and tokenizer error priority from the parser
Bump the ruff fork, which now reports unmatched, mismatched, unclosed
and too deeply nested brackets and decides whether a tokenizer error
replaces the parser's first error.
Remove the bracket, unterminated string and nesting depth source
scanners and `pre_parse_source_error`. A tokenizer error from the
parser now wins over source scanner diagnostics found after it. An
unclosed bracket is incomplete input only when the parser reports it
as such, and other tokenizer errors end where they start.
Remove the expected failure markers from test_unicode_identifiers
test_invalid and the `import ä £` doctest in test_syntax.
Assisted-by: Claude Code:claude-opus-5-5
* Take number literal, string prefix and non-printable errors from the parser
Bump the ruff fork, which now reports malformed number literals,
incompatible string prefixes and non-printable characters itself.
Remove the number literal, string prefix and non-printable character
source scanners and the override classes that ranked them. Leading zero
and string prefix errors keep their range instead of ending where they
start.
Assisted-by: Claude Code:claude-opus-5-5
* Take the missing indented block error from the parser
Bump the ruff fork, which now reports `ExpectedIndentedBlock` with the
clause and the line of its header keyword.
Remove the message rewrite that derived the clause and line from the
source. The VM chooses IndentationError or incomplete input by the
error variant instead of its message.
Assisted-by: Claude Code:claude-opus-5-5
* Take binding target errors from the parser
Bump the ruff fork, which now names the invalid expression in
assignment, augmented assignment, delete, `for`, `with`, import,
pattern and `except` target errors.
Remove the source scanners for those targets and the message rewrites
of `InvalidAssignmentTarget` and `InvalidNamedAssignmentTarget`.
Remove the expected failure markers from test_syntax doctests that now
pass.
Assisted-by: Claude Code:claude-opus-5-5
* Take parameter, argument, star, comma and dictionary errors from the parser
Bump the ruff fork to 36919b62fa. Remove the source scanners for type
parameters, parameter lists, star annotations, call arguments, def type
parameters, parenthesized groups, missing commas and dictionary items.
Set end_offset 0 for ExpectedColonAfterDictionaryKey. Remove 14
EXPECTED_FAILURE markers in test_syntax and test_unpack_ex.
Assisted-by: Claude Code:claude-opus-5-5
* Take expression errors from the parser
Bump the ruff fork to 4f922d2124. Remove the source scanners for
comprehensions, collection and condition assignments, expressions
between strings, named expression targets, missing `in` after for-loop
variables and statements in `if` expressions. Remove the now passing
expected failure markers in test_syntax and test_named_expressions.
Assisted-by: Claude Code:claude-opus-5-5
* Take statement errors from the parser
Bump the ruff fork to c7f6594ed6. Remove the source scanners for
standalone except, `import ... from`, `**_` mapping rests, `elif` after
`else` and mixed except handlers, and the post-parse scanners for call
arguments, match targets, yield after a comma and parenthesized star
imports.
Assisted-by: Claude Code:claude-opus-5-5
* Take string and f-string errors from the parser
Bump the ruff fork to 6b46d0dfc8. Remove the source scanners for
malformed `\N` escapes, f-string and t-string replacement fields, mixed
t-string literals and `print`/`exec` statements, and the override
plumbing they used. Drop the `f'{x'; '` case from the compiler test,
whose expected message differs from CPython 3.14.
Assisted-by: Claude Code:claude-opus-5-5
* Restore the bracket depth check and fix incomplete unclosed strings
Restore `pre_parse_source_error`, which rejects bracket nesting past 200
levels before parsing. The parser reports the same error but still
builds the full tree, which overflows the native stack when dropped.
Remove the VM's unclosed string scanner, which ignored the mode and
treated `'abc` in exec mode as incomplete input. The compiler's check
now skips escaped newlines.
Assisted-by: Claude Code:claude-opus-5-5
* Report multiple statements before later errors in single mode
In single mode, when the first statement is a simple statement, report
code after its line as multiple statements before any error found
later, at the newline token or the comment before it. This replaces the
multiple statements check on the parsed body and also applies to
`ast.parse(mode='single')`.
An unclosed string that follows an earlier syntax error is no longer
incomplete input. The bracket depth error range is now empty.
Assisted-by: Claude Code:claude-opus-5-5
* Update the Ruff fork for parser speed and starred annotations
Bump the ruff fork to 8b59b85344, which skips the tokenizer error pass
when parsing finds no errors, stops allocating for every identifier's
string prefix check, and reports `def f(*args: *b = ...)` as invalid
syntax.
Remove expected failure markers from test_code_module
test_indentation_error and test_sysexcepthook_indentation_error, and
from test_dictcomps test_illegal_assignment.
Assisted-by: Claude Code:claude-opus-5-5
* Use the published Ruff fork 0.16.10 crates
Switch the rustpython-ruff_* dependencies from the git revision to the
0.16.10 crates.io release, published from the same commit (8b59b85344).
Assisted-by: Claude Code:claude-opus-5-5
---------
Co-authored-by: zzarbttoo <qodbwls70@naver.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* Use AtomicPyTypeFlags in PyType_FromSlots
PyType_FromSlots still imported the old PyAtomicTypeFlags name.
Assisted-by: Grok:4.7
* Pass CPython test args as a bash array
Assisted-by: Grok:4.7
* Bump workspace version to 0.6.1
Assisted-by: Grok:4.7
* Restore the CPython test step in ci.yaml
Assisted-by: Grok:4.7
* Use bitflagset for type flags
PyTypeFlags is defined with bitflagset! and the atomic tp_flags word with
atomic_bitflagset!, replacing the hand-written PyAtomicTypeFlags.
Assisted-by: Grok:4.7
Assisted-by: Claude:claude-opus-5-5
* Use bitflagset 0.0.4 from its git main branch
bitflagset 0.0.4 is not on crates.io yet. Patch it to the git main
branch in the workspace and in the example projects that depend on
rustpython-vm, since each of them is a separate workspace.
Assisted-by: Claude:claude-opus-5-5
* Depend on crates.io bitflagset 0.0.4
bitflagset 0.0.4 is published. Drop the git patches from the workspace
and the separate example-project workspaces.
Assisted-by: Grok:4.7
* Keep empty patch tables in example workspaces
The example workspaces already have an empty [patch.crates-io] table.
Dropping the bitflagset git line leaves that table in place.
Assisted-by: Grok:4.7
* Call bitflagset methods directly for type flags
Drop the plain-set has_feature wrapper and the intersects,
is_created_with_flags, DEFAULT, and heap_type_flags aliases.
Single-bit updates use insert, and flag combinations use from_slice.
Assisted-by: Grok:4.7
* OR paired type flags in one update
HAS_WEAKREF with MANAGED_WEAKREF, and HAS_DICT with MANAGED_DICT
(and INLINE_VALUES when present), are applied with one |=.
Assisted-by: Grok:4.7
* Store heap type flag sets as constants
HEAPTYPE | BASETYPE, and the dict and immutable variants built from
it, are const values instead of from_slice calls at each use.
Assisted-by: Grok:4.7
* Test SEQUENCE and MAPPING in one load
PyTypeFlags::COLLECTION is the two-bit mask. Callers check it with
one is_disjoint instead of two contains loads.
Assisted-by: Grok:4.7
* Fix clippy 1.99 lints in the vm
Pass closure arguments by value where clippy reports a needless
borrow, drop the unfulfilled redundant_else expectation on sum, and
return the HRESULT after todo!().
Assisted-by: Grok:4.7
