17 Commits
Author SHA1 Message Date
youknowdot 1cd8f44f1a Sync remaining call, buffer and annotation tests for Python 3.15 (#9433)
Run update_lib against final CPython 3.15.0 for four disjoint test sources. Preserve the existing Proactor sendfile override and capability gates.

Assisted-by: OpenAI Codex: model version unavailable
2026-10-28 15:17:41 +00:00
Jeong, YunWon a7981dccc7 Align compiler futures, annotations, and symtable with CPython (#8550)
* Align compiler futures and annotations with CPython

Handle Barry parsing consistently across AST, type-comment, REPL, and WASM paths. Preserve deferred annotation source and scope metadata, and make _symtable conversion cached and linear-time.

Assisted-by: Codex:gpt-5.4

* Fix f-string debug ranges and update Barry/annotate test expectations

Compute the debug f-string source range from the raw leading and
trailing text instead of the comment-stripped text, so a comment inside
a multi-line replacement field no longer shifts the emitted LOAD_CONST
location.

Restrict the obsolete-operator location shift to `<>` by also requiring
a `>` at the reported offset; without it any `ExpectedExpression` that
followed a `<` (`2 <;`) was reported one column to the left.

Add `CodeFlags::FUTURE_MASK` and use it in builtins, the shell, and the
WASM VM instead of repeating the eight-flag union, and collapse the
duplicated Barry diagnostic sequences into `BarrySource::diagnostic`.

Update the `__annotate__` disassembly snapshot to LOAD_FAST_BORROW and
replace the RustPython-only `barry_as_FLUFL` co_flags assertion in
extra_tests with the CPython-matching one.

Assisted-by: Claude Code:claude-opus-5

* Restore the annotated function in __annotate__ qualnames

Signature annotation scopes are compiled in the scope enclosing the
annotated function, so the function name has to be folded back into the
qualname: `f.__annotate__`, `C.m.__annotate__`,
`outer.<locals>.inner.__annotate__`. test_type_annotations
test_annotate_qualname covers this.

test_pyrepl test_future_barry_as_flufl now passes, so drop its
expectedFailure marker.

Assisted-by: Claude Code:claude-opus-5

* Pass safepoints in the attached-thread interpreter tests

`main_and_subinterpreter_run_sections_overlap` parked its workers on a
condvar while attached, and `busy_main_interpreter_does_not_block_subinterpreter`
looped over protocol calls that never reach a safepoint. An attached
thread that never reaches a safepoint cannot be suspended, so a
concurrent process-wide stop-the-world (from the gc test running in
parallel) never completes and the already-stopped sibling interpreter
stays stopped; the second worker then waits in `wait_while_suspended`
until the 30s deadline.

Both loops now call `vm.check_signals()` each iteration, and the condvar
wait uses `wait_timeout` so the lock is released between safepoints.

Reproduced on Linux (4 CPUs): the two tests with `--test-threads=2`
failed 2/40 before and 0/40 after; the full `rustpython-vm` lib binary
failed 1/30 before and 0/30 after.

Assisted-by: Claude Code:claude-opus-5

* Report the obsolete `<>` operator from one place

`<>` was located twice: `BarrySource` recorded the token ranges it
rewrote to `!=`, and `cpython_parse_diagnostic_override` separately
peeked at the bytes around an `ExpectedExpression` location. Both
produced "invalid syntax" spanning the whole operator.

`prepare_barry_as_flufl_source` now records the operator ranges in the
non-Barry case too -- by plain text search, since nothing is rewritten
there -- so `BarrySource::invalid_legacy_operator` covers both, and the
byte-peeking helper is gone. It also defers to an unclosed bracket
earlier in the source, which the diagnostic chain it used to sit in did
for it.

A `<>` that starts a statement now spans both characters: the parser
stops at the `<` rather than one character in, which the byte peek did
not handle.

Assisted-by: Claude Code:claude-opus-5

* Drop a py object's payload before its class

`PyInner` declares `typ` before `payload`, so `drop_in_place` dropped the
class first and `PyAtomicRef::drop` left it null. A `PyWeak` payload is
still linked into the weakref list of the object it points at until its
own `Drop` unlinks it under the stripe lock, which happens after that.
`WeakRefList::add` walks the same list looking for a proxy to reuse and
reads `node.class()` off every node it passes, so it could dereference
the null class of a node on its way out.

`PyInner::dealloc` now runs the two destructors in the other order at all
three of its exit paths, with a compile-time check that no other field
has one.

Reproduced by the new test in about 0.1s, as the panic CI hit in the
c-api `weakrefobject::tests::new_proxy` run:

  panicked at crates/vm/src/object/ext.rs:312:
  unsafe precondition(s) violated: hint::unreachable_unchecked must
  never be reached

Assisted-by: Claude Code:claude-opus-5
2026-08-20 12:47:36 +09:00
Shahar Naveh 0664e38e5b Set python args to be like CPython in CI (#8181)
* Set python args to be like CPython in CI (ish)

* Use `--slow-ci`

* `--dont-add-python-opts`

* unmark passing test
2026-07-06 23:52:33 +09:00
Jeong, YunWon a1ba6a662c Align compiler and AST behavior with CPython (#8138)
* Align compiler and AST behavior with CPython

Update the compiler pipeline, AST module support, and related VM compile/eval paths toward CPython 3.14 behavior.

Use upstream Ruff parser crates directly and keep RustPython-specific syntax preflight handling in the compile path.

Refresh opcode metadata, snapshots, and targeted tests for the aligned bytecode and AST behavior.

* fix

* Fix PR 8138 CI failures

* Fix match mapping wildcard scan complexity

* fixes

* Use Ruff runtime AST fields

* Minimize residual patch diff

* refactor error handling

* Remove _ast conversion context wrappers

* Clean up _ast conversion glue and compiler warning escalation

Rename the Node conversion parameter `ctx` to `vm` throughout the _ast
module.

Remove the unused compile_program, compile_program_single,
compile_block_expression, and compile_expression forwarders; compile_top
stays as the entry point.

Stash an escalated compiler SyntaxWarning so a non-SyntaxWarning category
propagates unchanged instead of being rewritten to SyntaxError, matching
PyErr_ExceptionMatches(SyntaxWarning) in compiler_warn. Drop the now-unused
CompileWarningError::into_codegen_error.

Replace the PositionalArguments two-variant enum with a struct holding a
shared range and a PositionalArgumentsKind enum.

Assisted-by: Claude

* Represent compiler flags as a CompilerFlags bitflags

Replace the loose PY_CF_*/CO_* i32 constants in compile_mode with a
CompilerFlags bitflags covering the PyCF_* flags and __future__
CO_FUTURE_* bits. The Python-visible ast.PyCF_* attribute values are
derived from it via bits(); parser input modes stay plain consts.

builtins::compile and the compile_string flag handling test flags
through CompilerFlags::contains instead of raw bit math.

Assisted-by: Claude

* Fix PR 8138 test regressions

* Fix PR 8138 CI failures

* Avoid shell interpolation in Linux deps action

* Fix review findings in compiler/AST changes

- frame.rs: compute the MATCH_CLASS class name lazily so a successful
  match no longer takes the type name lock or allocates; the
  __match_args__-not-a-tuple branch reuses it instead of recomputing.
- _ast/constant.rs: add the missing leading space to the with_recursion
  label ("maximum recursion depth exceeded during compilation").
- sys.set_int_max_str_digits: restore the "maxdigits must be 0 or larger
  than {n}" error text.
- codegen: set CO_MAXBLOCKS to 20 in both ir.rs and compile.rs.
- _ast/exception.rs: extract the duplicated excepthandler node guard
  into ensure_excepthandler_node.
- codegen: correct the validate_keywords doc comment.
- Drop redundant "CPython" qualifiers from comments added in this branch.

Assisted-by: Claude
2026-06-26 18:22:08 +09:00
ShaharNaveh 223cedb49d Update test_future_stmt from 3.14.3 2026-02-13 13:43:29 +09:00
Jeong, YunWonandCPython Developers 3df178298a Fix traceback, syntax errors, and exception handling (#7015)
* Update codeop from v3.14.3

* Fix traceback, syntax errors, and exception handling

- Improve unclosed bracket detection with "'(' was never closed" message
- Fix IndentationError location to point to end of line
- Implement frame.clear() with proper checks for executing/suspended frames
- Fix exception context chaining for propagated exceptions
- Add traceback.__dir__() and prevent tb_next deletion
- Fix subscript operation source range restoration in compiler
- Change "duplicate parameter" to "duplicate argument" error message
- Refactor duplicate code in asyncgenerator.rs and frame.rs

---------

Co-authored-by: CPython Developers <>

Assisted-by: Codex:gpt-6
2026-02-07 13:41:45 +09:00
Jeong, YunWon 6df3753229 Py 3.14 changes fix (#6755)
* marshal v5

* conditional blclk

* fix jit don't use lossy string

* add varname

* symboltable takes responsibility of __debug__
2026-01-18 01:02:40 +09:00
Terry Tianlin LuanandJeong, YunWon 609dbb1439 Downgraded skips in tests (#6716)
* Downgraded skips in tests

* Fixed failing tests

* Fixed test_ftplib + test_socket + test_ssl + test_threaded_import failures

* Removed comments about which tests are run in which environment

* Addressed PR comments

* Annotated skips on failing tests

* Removed unneeded tests

* Removed unneeded sys import from test_ftplib

* Added annotation to test_ftplib

* Readded skipIf to test_cleanup_with_symlink_modes with a more general ENV_POLLUTING_TESTS_WINDOWS

* Addressed PR comments

* Made changes to minimize diff in PR

* Apply suggestion from @youknowone

---------

Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
2026-01-15 21:43:22 +09:00
Terry Tianlin Luan f274164dcd Add __future__.py tests (#6585)
* Added tests for __future__.py from v3.13.10

* * Deleted futures not in v3.13.10
* "Fixed" issues in test_future.py
2025-12-30 12:28:03 +09:00
Shahar Naveh 153d0eef51 Revert "Use ruff for Expr unparsing (#6124)"
This reverts commit 0fb7d0fae2.
2025-10-22 22:34:52 +09:00
Shahar Naveh 0fb7d0fae2 Use ruff for Expr unparsing (#6124)
* Use ruff for unparse backend

* Update `test_future_stmt/*.py` from 3.13.7

* Mark failing tests

* Mark failing test

* Merge remote-tracking branch 'upstream/main' into ruff-unparse

* Reapply ruff code

* remove git symbols

* Unmark passing test
2025-10-22 20:29:50 +09:00
Ashwin Naren d46bcd9291 typing upgrade to 3.13.2 (#5590) 2025-04-27 15:26:37 +09:00
Noa d3d92bbb6f Update unparse to work with ruff & remove ruff_python_codegen 2025-03-27 22:14:58 -05:00
a6b4ef7f5d Replace Python parser with ruff parser (#5494)
* stage1

* compiler pass build

* introduce rustpython-compiler-source

* stage2

* fixup

* pass compile

* Fix hello world compiler test

* Fix code generation for if-elif-else statement

* Fix code generation for lambda expression

* Fix code generation for integers

* Fix code generation for fstrings

* Fix code generation for if statement

* Fix code generation for if statement

* Fix code generation for if statement

* Fix code generation for fstring

* Fix code generation for class definition

* Replace feature flags

* Initialize frozen core modules

* Allow __future__ import after module doc comment

* Disable ast module

* Commit remaining fixes for compile errors in examples

* Fix some warnings

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Split ast stdlib module into files

* Fix codegen for positional arguments with defaults

* Update ast stdlib module

* Update ast stdlib module

* Extract string and constant handling from expression.rs

* Always add required fields to AST nodes

* Compile doc strings correctly again

* Enable "ast" Cargo feature by default

* Refactor compilation of big integer literal

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Reset barebones example

* Fix some left-over warnings

* Undo accidential change

* Adapt shell to ruff parser

* Pin parser to v0.4.10

* fix clippy

* Add TODO about interactive mode

* Fix compilation of complex number expression

* Remove moved code

* Update test case to ruff v0.4.10

* Apply suggestion

Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>

* Apply suggestion

Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>

* Fix compilation of fstring expression

* Fix compilation of fstring expression

* Fix wasm compile errors

* Attach correct source locations to ast objects

* Fix some more wasm compile errors

* Consider compile mode and enable AST stdlib module again

* Fix incorrect AST source location end column

* Fix compile error if "compiler" feature is not enabled

* Fix regrtests

* Fix some test_ast tests

* Add source range to type ignore

* Fix incompatibility with Rust 2024 edition

* Fix todos by implementing missing ast conversions and deleting unused code

* Appease clippy

* Fix remaining ast tests

* Fix remaining ast tests

* Mark/fix remaining tests

* Fix more

* Hacky windows fix

---------

Co-authored-by: Kangzhi Shi <shikangzhi@gmail.com>
Co-authored-by: Jeong YunWon <jeong@youknowone.org>
Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
Co-authored-by: Noa <coolreader18@gmail.com>
2025-03-19 21:06:03 -05:00
Jonathan Rotter 07fdcb6160 Fix for test future.py test annotations (#5319)
* enable test fixed in parser

* update parser dependency
2024-05-13 08:40:14 +09:00
Jonathan Rotter a8ea67178d adjusted SyntaxError args and __str__() to match CPython (#5294) 2024-05-08 12:57:03 +09:00
CPython Developers 153ec2823d Update __future__ from CPython 3.12.2 2024-04-23 11:57:46 +09:00