Commit Graph
37 Commits
Author SHA1 Message Date
Jeong, YunWon f56bfc9a5c Show stdin traceback lines; omit StopIteration None args (#8778)
* Show stdin traceback lines; omit StopIteration None args

Register interactive <stdin> source in linecache so REPL
tracebacks include the source line and caret. Build
StopIteration with empty args when the value is None, and
wrap monitoring STOP_ITERATION the same way.

Unskip REPL stderr flush tests and pdb Flaky doctests.

Assisted-by: Grok:4.6

* Take &Py<PyCode> in register_code_in_linecache

Assisted-by: Grok:4.6

* Count Unknown closing braces in tokenize EOF check

`$` inside an f-string interpolation is TokenKind::Unknown, so
the matching `}` is Unknown rather than Rbrace and generate_tokens
raised TokenError. Also convert &Py<PyCode> to PyObjectRef
explicitly when registering linecache.

Assisted-by: Grok:4.6

* Unskip test_pdb.PdbTestCase.test_zipimport

The test now passes, so the expectedFailure is an unexpected
success under CI.

Assisted-by: Grok:4.6

* Park STW threads; poll sockets after timeout deadline

Check per-thread stop_requested in eval_breaker_tripped so
workers still park when another interpreter clears STOP_BIT.

Start socket timeout after the fd snapshot, and poll once with
timeout 0 when the deadline has already passed, so a readable
socket is not dropped as timed out.

Assisted-by: Grok:4.6
2026-09-22 14:53:52 +09:00
Jeong, YunWon 8fda0eccee Fix fork-child SIGSEGV and yield-from throw() args (#8759)
* Unskip passing flaky CPython tests

Remove RUSTPYTHON skip overlays after Linux release repeats passed
for enum, threading, logging, subprocess, socket, int, re, strtod,
and regrtest cases.

Assisted-by: Grok:4.6

* Fix fork-child SIGSEGV and yield-from throw args

Forget JoinHandle of threads that did not survive fork instead
of dropping them. Zero-reinit the import lock and condvars in
the child rather than unlock/notify. Resume the world if
os.fork() fails.

Call a yield-from subiterator throw() without None fillers so
a 1-arg throw stays 1-arg.

Drop the stale Windows expectedFailure on logging queue config
and unskip test_reinit_tls_after_fork plus the asyncio
local_addr family test.

Assisted-by: Grok:4.6

* Restore import lock depth after fork; add WASI clock info

Zeroing IMP_LOCK in the child discarded its recursion count, so
imp.release_lock() raised and test_fork1 exited 1. Re-acquire the
saved depth, then drop the extra pre-fork hold.

Expose time.get_clock_info on WASI via clock_getres so test_int
Stopwatch can run.

Unskip lock_tests test_acquire_contended and test_thread_leak.

Assisted-by: Grok:4.6

* Honor chmod follow_symlinks=False via fchmodat

Linux chmod followed the symlink, so dangling links raised
FileNotFoundError instead of NotImplementedError. Use fchmodat
with AT_SYMLINK_NOFOLLOW and map ENOTSUP to NotImplementedError.

Unskip httpservers truncated Content-Length (CPython already
skips as root) and tempfile symlink-mode cleanup on Linux.

Assisted-by: Grok:4.6

* Unskip httplib local bad-hostname HTTPS test

The skip was leftover. With default ssl-rustls the test passes
under -W error on Linux.

Assisted-by: Grok:4.6

* Fix list cmp deadlock, wasm digit overflow, REPL prompt

Do not hold list/deque locks across item comparison callbacks.
Compute int-to-str digit bounds in u64 so the check cannot wrap
on wasm32. Snapshot set contents when list.extend copies a set.

For a non-TTY stdin, write the REPL prompt to stderr and read
lines from stdin. Flush stdout/stderr after each interactive
command. Set sys.ps1/ps2 to ">>> " and "... ".

Unskip list mutation compares, asyncio all_tasks race, importlib
deadlock/parallel/circular-import tests, and REPL stdout flush.

Assisted-by: Grok:4.6

* Avoid unnecessary u64 cast in int-to-str digit check

bits() is already u64, so `bits as u64` fails clippy
unnecessary_cast under -Dwarnings.

Assisted-by: Grok:4.6
2026-09-21 20:53:53 +09:00
Jeong, YunWon cfee8a7ffa Fix -m argv[0] and file SyntaxError display (#8684)
* Fix -m argv[0] and file SyntaxError display

Set sys.argv[0] to -m for module execution. Convert an escalated
invalid-escape SyntaxWarning into a SyntaxError that spans just
the `\X` pair. Report null bytes in source files with filename,
lineno, and the truncated line.

Enable the four corresponding test_cmd_line_script cases.

Assisted-by: Grok:4.6

* Pass SyntaxError text when escalating escape warnings

Keep the warning and SyntaxError messages as separate strings at
the invalid-escape site. CompileWarningError carries an optional
replacement instead of rewriting the warning text later.

Assisted-by: Grok:4.6

* Remove expectedFailure from bytes invalid-escape test

The invalid-escape SyntaxError path now satisfies
test_eval_bytes_invalid_escape.

Assisted-by: Grok:4.6
2026-09-11 09:54:59 +09:00
Jeong, YunWon eee9ad6420 Align contextvars, CLI startup, and stat validation with CPython (#8657)
* Align contextvars, CLI startup, and stat validation with CPython

Validate Context and ContextVar arguments and mapping keys, hash names through the Python protocol, and preserve string contents without invoking __str__. Set the main module loader for command and stdin execution, report missing scripts with exit status 2, and reject file descriptors combined with follow_symlinks=False.

Enable the nine corresponding stdlib regression tests.

Assisted-by: Codex:gpt-6

* Move set_main_builtin_importer off the compiler feature

Define the method in vm_new.rs so builds without rustpython-compiler,
such as example_projects/barebone, can compile. Also set BuiltinImporter
on an existing __main__ when its loader is missing or None.

Assisted-by: Grok:4.6
2026-09-06 23:54:30 +09:00
Jeong, YunWon 25b5dfe5ca Fix /dev/fd, rlimit, spawn scheduler, and context cache (#8651)
* Fix /dev/fd, rlimit, spawn scheduler, and context cache

Honor PYTHONEXECUTABLE when computing sys.executable.
Rewind the file offset after a pyc magic probe.
Stop masking rlimit values with RLIM_INFINITY.
Print the pyrepl-unavailable warning before the rustyline shell.
Parse posix_spawn scheduler and apply it where the host supports it.
Update ContextVar cache fields under one lock.
Remove expectedFailure and skip markers for tests that now pass.

Assisted-by: Grok:4.6

* address review: check rlim_t overflow on convert

Replace the u64-to-rlim_t as-cast in py2rlim_limit with
a checked try_from, matching py2rlim.

Assisted-by: Grok:4.6
2026-09-05 17:57:29 +09:00
Jeong, YunWon 7c6c78f955 Implement CPython 3.14 subinterpreter APIs (#8605)
* Add _interpreters and _interpchannels modules

Apply the patch baseline for the low-level subinterpreter modules:

- crates/vm/src/stdlib/_interpreters.rs: create/destroy/list_all/get_main/
  get_current/is_running/exec/call/run_string/run_func/set___main___attrs/
  is_shareable/whence and the Interpreter*Error exceptions
- crates/vm/src/stdlib/_interpchannels.rs: channel create/destroy/send/recv/
  list_all/list_interpreters/release/close and the Channel*Error exceptions
- crates/vm/src/vm/crossinterp.rs: cross-interpreter data protocol
- crates/vm/src/vm/{interpreter,mod,runtime,vm_new}.rs: interpreter registry
  and lifecycle plumbing the modules need
- Lib/concurrent/interpreters/: high-level PEP 734 package

Assisted-by: Claude:claude-opus-5

* Correct _interpreters and _interpchannels against CPython 3.14

Argument matching:
- Add `function::ArgSpec`, a `PyArg_ParseTupleAndKeywords` equivalent that
  applies the `|`, `$` and `:` markers of a format string to a `kwlist`, and
  route every module function of both modules through it. This enforces
  keyword-only parameters, rejects unexpected keywords, accepts keyword forms
  that were previously positional-only (`whence(id=)`, `get_config(id=)`,
  `set___main___attrs(id=, updates=)`, `capture_exception(exc=)`,
  `_register_end_types(send=, recv=)`), and reproduces the arity messages.
- `O!` slots (`shared`, `updates`, `call`'s `args`/`kwargs`) now reject None,
  and `_PyArg_BadArgument` renders None as "None" rather than "NoneType".
- `new_config` takes at most one positional `str`; `list_all` of
  `_interpchannels` is argument-less.
- Convert arguments in `kwlist` order, so an argument's own error is raised
  ahead of the checks that follow it.

_interpreters:
- Add the `CrossInterpreterBufferView` type and build the received memoryview
  on it, keeping the sending interpreter's exporter out of the destination.
- Build `excinfo.errdisplay` from `traceback.TracebackException`.

Feature flags:
- `os.fork` checks finalization before `allow_fork`, matching `os_fork_impl`.
- `_thread.start_new_thread` and `start_joinable_thread` reject an interpreter
  without `allow_threads`.

Lib/concurrent/interpreters/_crossinterp.py: restore the stripped docstrings
and comments.

Assisted-by: Claude:claude-opus-5

* Correct the stateless checks and error shapes of _interpreters against CPython 3.14

verify_stateless_function follows _PyFunction_VerifyStateless: it rejects
non-dict builtins, a non-empty __defaults__, __kwdefaults__ or __closure__,
and code whose LOAD_GLOBAL names are held by the function's globals or
absent from its builtins.

code_returns_only_none follows _PyCode_ReturnsOnlyNone: generator,
coroutine and async-generator code is rejected up front, the instruction
walk skips inline caches and maps specialized and instrumented opcodes
back, and the LOAD_CONST preceding each RETURN_VALUE is compared against
the index of None in co_consts.

ExcInfo::capture keeps an empty exception message instead of dropping it,
and builds `formatted` as `module.qualname: msg`, leaving out the builtins
and __main__ modules.

_interpreters.call packs the callable through _PyFunction_GetXIData before
pickle and re-raises the stateless-check failure when both fail. A func,
args or kwargs that cannot be rebuilt in the target, and a result without
cross-interpreter data, now raise NotShareableError with the snapshot as
the cause rather than being returned as excinfo.

_interpreters.capture_exception() without an argument returns None.

pickle_loads and _PyFunction_GetXIData attach the underlying failure as
__cause__.

channel_send converts the object to cross-interpreter data after the
channel's `closing` check.

ChannelID rich comparison returns the result of comparing its id with the
other number instead of coercing that result to bool.

PyMemoryView_FromObjectAndFlags raises "memoryview: a bytes-like object is
required, not 'X'" for an object that is not a buffer.

Assisted-by: Claude:claude-opus-5

* Correct cross-interpreter data conversion and channel argument parsing

Bound shareable ints by isize, raising OverflowError("try sending as
bytes") outside that range and keeping it as the NotShareableError cause.

Report an unshareable object by its repr, and attach a failing conversion
as __cause__ of the NotShareableError raised for the object it was called
for, so each level of a nested tuple appears in the chain.

Guard each tuple item conversion with with_recursion("while sharing a
tuple").

Gate the memoryview getdata function on `_interpreters` having been
imported, and route channel_send_buffer through a memoryview so it uses
that function and the resolved fallback.

Create the cross-interpreter exception types from the module_exec of
either module that raises them, instead of only `_interpreters`.

parse_cid raises OverflowError("int too big to convert"); int_arg
reproduces the `i` converter's three overflow messages and is also used
while parsing channel_send arguments.

Add BASETYPE to _queue.Empty, and gate nt.execv/nt.execve on allow_exec.

Assisted-by: Claude:claude-opus-5

* Correct interpreter ID parsing and the channel_send defaults lookup

_PyInterpreterState_ObjectToID accepts any object with __index__, raises
OverflowError("int too big to convert") outside the int64 range, and
reports a negative ID with the repr of the original object.

channelsmod_send reads the channel's default unboundop and fallback only
when one of the two arguments is negative, so an explicit pair is
validated before the channel is looked up.

Assisted-by: Claude:claude-opus-5

* Add the _interpqueues module and concurrent.interpreters._queues

_interpqueues holds a process-wide queue table and exposes create,
destroy, list_all, put, get, bind, release, get_maxsize,
get_queue_defaults, is_full, get_count and _register_heap_types, raising
QueueError and QueueNotFoundError.  QueueEmpty and QueueFull, which
subclass queue.Empty and queue.Full, are registered per interpreter by
concurrent.interpreters._queues.

Cross-interpreter data carries a queue as a refcounted queue ID that
binds when the data is captured and releases when it is dropped, and
is_shareable reports a registered Queue as shareable.

Queue and channel items owned by an interpreter are now cleared from
Interpreter::finalize, after module finalization, rather than from
destroy_owned_interpreter, so values sent by an atexit callback also
become unbound.

is_shareable gates memoryview on the same registration flag as the
getdata lookup.

Assisted-by: Claude:claude-opus-5

* Add the high-level concurrent.interpreters API

Interpreter, ExecutionFailed and the queue aliases, copied verbatim from
CPython 3.14.

Assisted-by: Claude:claude-opus-5

* Destroy leftover subinterpreters when the main interpreter finalizes

Interpreter::finalize now runs finalize_subinterpreters between the atexit
handlers and the finalizing flag: when the main interpreter still owns
subinterpreters it emits the RuntimeWarning "remaining subinterpreters;
close them with Interpreter.close()" and then finalizes each of them.

runtime::owned_interpreter_ids lists the runtime-owned interpreters.

Assisted-by: Claude:claude-opus-5

* Absolutize the script path in run_file

_Py_abspath and _PyPathConfig_ComputeSysPath0 arrive as abs_path and
script_sys_path0. run_file hands the absolutized path to get_importer
and to the runner, so __file__ and co_filename no longer stay relative,
and inserts the symlink-resolved script directory as sys.path[0]
instead of the argument's parent.

test_cmd_line_script.test_script_abspath no longer expects a failure.

Assisted-by: Claude:claude-opus-5

* Name types in error messages the way tp_name does

The object, number, sequence and mapping protocols, the
unsupported-operand helpers, and the type-specific messages fed from
them now read PyType::slot_name() where they read PyType::name(),
so a type declared with a module keeps it.

Message texts corrected along the way:
- PySequence_Size, PyMapping_Size, PySequence_GetItem and
  PySequence_SetItem/DelItem split their merged wording in two, and
  PyObject_SetItem/DelItem say "object".
- PyObject_GenericSetAttr reports a read-only attribute and the
  missing __dict__.
- check_class drops the type name it appended.
- __int__ and __index__ drop the class prefix; __float__ keeps it.
- float(), complex(), round(), next(), str subscripting and list
  subscripting take their own wording.
- str.join and bytes.join number the item they turn down and name what
  they wanted, and report a non-iterable as "can only join an
  iterable".

test_class.testObjectAttributeAccessErrorMessages,
string_tests.test_subscript and the test_descrtut doctest no longer
expect a failure.

Assisted-by: Claude:claude-opus-5

* Gate script_code's source compilation on the compiler feature

crossinterp::script_code called vm.compile unconditionally, which only
exists with the rustpython-compiler feature. Builds without it, such as
example_projects/barebone, now raise a TypeError instead of failing to
compile.

Assisted-by: Claude:claude-opus-5

* Skip test_pyinit_function_raises_exception

The test imports _testsinglephase in its body without a decorator
guarding it, and now that _interpreters exists requires_subinterpreters
no longer skips it.

Assisted-by: Claude:claude-opus-5
2026-08-30 13:45:54 +09:00
Jeong, YunWon a1ba6a662c Align compiler and AST behavior with CPython (#8138)
* Align compiler and AST behavior with CPython

Update the compiler pipeline, AST module support, and related VM compile/eval paths toward CPython 3.14 behavior.

Use upstream Ruff parser crates directly and keep RustPython-specific syntax preflight handling in the compile path.

Refresh opcode metadata, snapshots, and targeted tests for the aligned bytecode and AST behavior.

* fix

* Fix PR 8138 CI failures

* Fix match mapping wildcard scan complexity

* fixes

* Use Ruff runtime AST fields

* Minimize residual patch diff

* refactor error handling

* Remove _ast conversion context wrappers

* Clean up _ast conversion glue and compiler warning escalation

Rename the Node conversion parameter `ctx` to `vm` throughout the _ast
module.

Remove the unused compile_program, compile_program_single,
compile_block_expression, and compile_expression forwarders; compile_top
stays as the entry point.

Stash an escalated compiler SyntaxWarning so a non-SyntaxWarning category
propagates unchanged instead of being rewritten to SyntaxError, matching
PyErr_ExceptionMatches(SyntaxWarning) in compiler_warn. Drop the now-unused
CompileWarningError::into_codegen_error.

Replace the PositionalArguments two-variant enum with a struct holding a
shared range and a PositionalArgumentsKind enum.

Assisted-by: Claude

* Represent compiler flags as a CompilerFlags bitflags

Replace the loose PY_CF_*/CO_* i32 constants in compile_mode with a
CompilerFlags bitflags covering the PyCF_* flags and __future__
CO_FUTURE_* bits. The Python-visible ast.PyCF_* attribute values are
derived from it via bits(); parser input modes stay plain consts.

builtins::compile and the compile_string flag handling test flags
through CompilerFlags::contains instead of raw bit math.

Assisted-by: Claude

* Fix PR 8138 test regressions

* Fix PR 8138 CI failures

* Avoid shell interpolation in Linux deps action

* Fix review findings in compiler/AST changes

- frame.rs: compute the MATCH_CLASS class name lazily so a successful
  match no longer takes the type name lock or allocates; the
  __match_args__-not-a-tuple branch reuses it instead of recomputing.
- _ast/constant.rs: add the missing leading space to the with_recursion
  label ("maximum recursion depth exceeded during compilation").
- sys.set_int_max_str_digits: restore the "maxdigits must be 0 or larger
  than {n}" error text.
- codegen: set CO_MAXBLOCKS to 20 in both ir.rs and compile.rs.
- _ast/exception.rs: extract the duplicated excepthandler node guard
  into ensure_excepthandler_node.
- codegen: correct the validate_keywords doc comment.
- Drop redundant "CPython" qualifiers from comments added in this branch.

Assisted-by: Claude
2026-06-26 18:22:08 +09:00
Shahar Naveh 620faab734 Update some tests to 3.14.5 (#7990)
* Update `test_embed.py` to 3.14.5

* Update `test_cmd_line*` to 3.14.5

* Update unicode related tests

* Update `test_script_helper.py`

* Update `test_gc.py`
2026-05-30 20:16:10 +09:00
Jeong, YunWonandCPython Developers 3df178298a Fix traceback, syntax errors, and exception handling (#7015)
* Update codeop from v3.14.3

* Fix traceback, syntax errors, and exception handling

- Improve unclosed bracket detection with "'(' was never closed" message
- Fix IndentationError location to point to end of line
- Implement frame.clear() with proper checks for executing/suspended frames
- Fix exception context chaining for propagated exceptions
- Add traceback.__dir__() and prevent tb_next deletion
- Fix subscript operation source range restoration in compiler
- Change "duplicate parameter" to "duplicate argument" error message
- Refactor duplicate code in asyncgenerator.rs and frame.rs

---------

Co-authored-by: CPython Developers <>

Assisted-by: Codex:gpt-6
2026-02-07 13:41:45 +09:00
Jeong, YunWon fea26e456f impl more nt (#6984)
* mpl new features

* windows encodings

* impl nt functions

* revert

* codecs

* fix codecs
2026-02-04 09:53:02 +09:00
Jeong, YunWon bdf3b3654e fix syntaxerror ast end_offset (#6646) 2026-01-05 08:01:41 +09:00
Jeong, YunWon 7f4d308efc Implement maybe_pyc_file and .pyc file execution (#6539)
* Implement maybe_pyc_file and .pyc file execution

* unmark failing tests
2025-12-26 22:13:00 +09:00
Jeong, YunWon 57b4b4ae45 fix sys path (#6537) 2025-12-26 21:43:00 +09:00
Copilotandyouknowone 6dbc8f0cfa Set SourceFileLoader for script execution in run_simple_file (#6496)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>
2025-12-25 10:33:22 +09:00
Noa a86126419c Fix remaining tests 2025-03-25 19:05:12 -05:00
Noa ba1b5811ee Update encoding to use wtf8 2025-03-25 19:05:11 -05:00
NakanoMiku 9a61716f74 Edit test_cmd_line_script.py 2023-11-30 19:52:25 +08:00
NakanoMiku 700f2b9c12 Update test_cmd_line_script.py from CPython v3.12.0 2023-11-30 19:49:27 +08:00
Andrey MaltsevandJim Fasarakis-Hilliard 94dc7bb279 Apply suggestions from code review
Co-authored-by: Jim Fasarakis-Hilliard <d.f.hilliard@gmail.com>
2023-04-02 18:25:21 +03:00
Andrey Maltsev cde5527c9f fix mac os 2023-04-02 15:19:02 +00:00
Andrey Maltsev e169d97713 Update test_cmd_line_script from Cpython v3.11.2 2023-04-02 12:51:55 +00:00
Jeong YunWon 6ee827bd8a Fix CR/LF of python libraries 2023-02-28 17:11:41 +09:00
Jeong YunWon 53ef2b9a0a remove failure markers from test_imp,test_zipimport,test_cmd_line_script 2022-08-15 20:09:34 +09:00
CPython Developers 628c4c0edf Update runpy from CPython 3.10.6 2022-08-15 01:08:58 +09:00
Jeong YunWon 2fbd01f23e Fix surrogateescape_errors 2022-07-25 04:10:59 +09:00
Myeongseon Choi 44004935b0 Specify expectedFailure of test_non_ascii in Lib/test/test_cmd_line_script.py 2022-07-14 05:50:00 +09:00
Myeongseon Choi 5ab55dc7ce Add expectedFailure tag to test_non_ascii 2022-07-14 05:50:00 +09:00
Myeongseon Choi 9f29dc558a Update Lib/test/test_cmd_line_script.py to be runable 2022-07-14 05:50:00 +09:00
Myeongseon Choi e86a1bc92f Copy Lib/test/test_cmd_line_script.py from CPython v3.10.5 2022-07-14 05:50:00 +09:00
Dean Li 6f98288e84 test: use import_helper 2021-11-29 21:03:02 +08:00
Dean Li 5ee4fb899b test: use os_helper 2021-11-28 20:51:32 +08:00
Jeong YunWon 913b78ca44 Revert "Merge pull request #3433 from deantvv/test-update"
This reverts commit 9fa5c5ac66, reversing
changes made to e7fa32c687.
2021-11-17 17:06:51 +09:00
Dean Li 49a5805d11 test: use os_helper 2021-11-13 02:18:33 +00:00
Padraic Fanning 3865af44d1 Clean up skips in test_cmd_line_script 2021-10-10 19:11:23 -04:00
Padraic Fanning c3221a0d7b Unmark passing tests 2021-10-01 19:39:57 -04:00
snowapril 4014b91780 add decorators on failed & error tests
Signed-off-by: snowapril <sinjihng@gmail.com>
2021-10-01 14:10:58 +09:00
snowapril 2432c784c3 add test_cmd_line_script.py from cpython 3.8
Signed-off-by: snowapril <sinjihng@gmail.com>
2021-10-01 14:10:58 +09:00