Commit Graph
90 Commits
Author SHA1 Message Date
Jeong, YunWon a1ba6a662c Align compiler and AST behavior with CPython (#8138)
* Align compiler and AST behavior with CPython

Update the compiler pipeline, AST module support, and related VM compile/eval paths toward CPython 3.14 behavior.

Use upstream Ruff parser crates directly and keep RustPython-specific syntax preflight handling in the compile path.

Refresh opcode metadata, snapshots, and targeted tests for the aligned bytecode and AST behavior.

* fix

* Fix PR 8138 CI failures

* Fix match mapping wildcard scan complexity

* fixes

* Use Ruff runtime AST fields

* Minimize residual patch diff

* refactor error handling

* Remove _ast conversion context wrappers

* Clean up _ast conversion glue and compiler warning escalation

Rename the Node conversion parameter `ctx` to `vm` throughout the _ast
module.

Remove the unused compile_program, compile_program_single,
compile_block_expression, and compile_expression forwarders; compile_top
stays as the entry point.

Stash an escalated compiler SyntaxWarning so a non-SyntaxWarning category
propagates unchanged instead of being rewritten to SyntaxError, matching
PyErr_ExceptionMatches(SyntaxWarning) in compiler_warn. Drop the now-unused
CompileWarningError::into_codegen_error.

Replace the PositionalArguments two-variant enum with a struct holding a
shared range and a PositionalArgumentsKind enum.

Assisted-by: Claude

* Represent compiler flags as a CompilerFlags bitflags

Replace the loose PY_CF_*/CO_* i32 constants in compile_mode with a
CompilerFlags bitflags covering the PyCF_* flags and __future__
CO_FUTURE_* bits. The Python-visible ast.PyCF_* attribute values are
derived from it via bits(); parser input modes stay plain consts.

builtins::compile and the compile_string flag handling test flags
through CompilerFlags::contains instead of raw bit math.

Assisted-by: Claude

* Fix PR 8138 test regressions

* Fix PR 8138 CI failures

* Avoid shell interpolation in Linux deps action

* Fix review findings in compiler/AST changes

- frame.rs: compute the MATCH_CLASS class name lazily so a successful
  match no longer takes the type name lock or allocates; the
  __match_args__-not-a-tuple branch reuses it instead of recomputing.
- _ast/constant.rs: add the missing leading space to the with_recursion
  label ("maximum recursion depth exceeded during compilation").
- sys.set_int_max_str_digits: restore the "maxdigits must be 0 or larger
  than {n}" error text.
- codegen: set CO_MAXBLOCKS to 20 in both ir.rs and compile.rs.
- _ast/exception.rs: extract the duplicated excepthandler node guard
  into ensure_excepthandler_node.
- codegen: correct the validate_keywords doc comment.
- Drop redundant "CPython" qualifiers from comments added in this branch.

Assisted-by: Claude
2026-06-26 18:22:08 +09:00
Shahar Naveh 7da696661b Reduce number of string clones when compiling python source code (#8103) 2026-06-15 11:20:48 +09:00
Joshua MegnauthandJeong, YunWon f347ae4fbb Support custom rustls providers (#7657)
* Modularize rustls as work towards providers

`rustls`'s architecture is very clean and trait-driven. There are many
providers for `rustls` including the built-in `aws-lc-rs` and `ring` as
well as backends for `boringssl`, `graviola`, `openssl`, `mbedtls`, etc.

This commit removes the hard dependency on `aws-lc-rs` and adds support
for `ring`. It works towards #7059 as well.

* Clean up rustls features

* Remove ring as an explicit feature
* ssl-rustls is the default and implies aws-lc

* Support custom rustls crypto providers

The new feature, `ssl-rustls-no-provider`, enables custom rustls
providers. By default, `aws-lc-rs` is enabled which matches the old
behavior and keeps backward compatibility.

I wrote a new type that abstracts what we need from crypto providers.
CryptoExt encapsulates the ticketer as well as cipher suites and KX
groups. I wrote fallbacks to help select a reasonable default if a
provider is missing features (they all seem to support the same things
though).

I also wrote an example to show how to actually use custom providers.

* Fix duplicate VERIFY_X509 constants and unused imports in ssl module

Remove duplicate VERIFY_X509_STRICT/VERIFY_X509_PARTIAL_CHAIN definitions
from compat.rs (already defined in _ssl module with #[pyattr]).
Remove unused imports: ClientConnection, ServerConnection.

* no-provider as default

* Fix CI failures: openssl build, wasm target, cargo-shear

- Update openssl.rs to use renamed sock_wait/SockWaitKind and add vm parameter
- Add skip_ssl for wasm32-wasip2 target (aws-lc-sys cannot build for wasm)
- Remove unused workspace dependency aws-lc-rs
- Fix foreign-types-shared version to match openssl's dependency (0.1)
- Restore Cargo.lock from upstream/main

---------

Co-authored-by: Jeong, YunWon <jeong@youknowone.org>
2026-05-28 11:07:09 +09:00
Shahar Naveh 76128d183f Remove unused rust impl for formatting dis output (#7660)
* Remove unused rust impl for formatting dis output

* remove `examples/dis.rs`

* Added tests

* Update lock

* Try to set snapshot dir

* Remove verbose flag

* Regenerate snapshots after #7711

* Revert "Bump insta from 1.46.3 to 1.47.2 (#7706)"

This reverts commit e6d9ea6bfe.

* Debug info

* Show diff as well

* Show debug faster

* CI: true env

* Recert CI

* Add `CI: true` in ci emv

* Reapply "Bump insta from 1.46.3 to 1.47.2 (#7706)"

This reverts commit 693ca8cbe4d7885a81162a9be31e8bb567db885a.

* simplify macro

* trim on function side

* Force insta workspace root

* fix merge
2026-05-05 08:09:35 +00:00
f51f27c949 Create rustpython-host-env (#7582)
* Create rustpython-host-env crate; move host abstractions out of common

Move os, crt_fd, fileutils, windows, macros modules from
rustpython-common into the new rustpython-host-env crate.
This isolates host OS API wrappers behind a crate boundary
with zero Python runtime dependency.

- Add crates/host_env to workspace
- Drop nix, windows-sys, widestring deps from common
- Wire vm and stdlib to depend on rustpython-host-env
- Migrate all imports from common::{os,crt_fd,fileutils,windows}
  to rustpython_host_env::

* refactor: extract host helpers

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/48d1e64d-37ce-409f-b511-8e61a349665c

Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>

* lint: enforce direct host API boundaries

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/97225fb7-7b3d-4197-a77c-eb44aead5b13

Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>

* refactor: extract remaining host env helpers

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/d96f57e1-b196-4460-9983-97d5ff118835

Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>

* fix: clean extracted host env follow-up

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/d96f57e1-b196-4460-9983-97d5ff118835

Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>

* fix: document env mutation safety

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/d96f57e1-b196-4460-9983-97d5ff118835

Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>

* refactor: split host fs helpers from fileutils

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/c57424c5-0e1d-490a-82b3-2d2f6c8cf2cd

Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>

* fix: resolve latest host env ci regressions

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/899eb717-ebc6-4a4a-870c-2a15c5f33e02

Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>

* fix: resolve remaining windows clippy host fs calls

Agent-Logs-Url: https://github.com/RustPython/RustPython/sessions/12f32740-8173-4b10-a1d6-00b29e90a8ec

Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>

* host_env

---------

Co-authored-by: Jeong, YunWon <jeong@youknowone.org>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: youknowone <69878+youknowone@users.noreply.github.com>
2026-04-21 00:13:25 +09:00
Noa 276a4bdc80 Make paths in macro arguments relative to the source file, not MANIFEST_DIR (#6873) 2026-01-30 22:37:18 +09:00
ShaharNaveh 2c2b13784e Clippy 2026-01-23 16:41:24 +02:00
Jeong YunWon 20376451eb Implement Py_mod_create slot support in multi-phase init 2026-01-22 11:21:42 +09:00
Jeong YunWon bc02b2318c module_exec 2026-01-21 22:39:40 +09:00
Jeong, YunWon e79a1a1a66 Fix traceback (#6569)
* Fix traceback

* Update traceback from CPython 3.13.11

* unmark test_traceback

* fix code

* fix debug range

* fix tests
2025-12-30 16:58:53 +09:00
Terry Tianlin Luan 1464d5ca43 Adding + Fixing Clippy rules to better align with #[no_std] (#6570)
* * Added alloc_instead_of_core, std_instead_of_alloc, and std_instead_of_core clippy rules
* Manually changed part of the code to use core/alloc

* use clippy --fix to fix issues in stdlib

* * Used clippy --fix to fix issues in vm
* Imported Range in vm/src/anystr.rs

* * Used clippy --fix to fix issues in common
2025-12-30 13:10:14 +09:00
Jeong, YunWon 4bec0ad1c6 Fix test_runpy (#6409) 2025-12-12 15:26:00 +09:00
Shahar Naveh 4bf32a04f4 Apply some clipy lints (#6045) 2025-07-30 12:16:02 +09:00
Jeong, YunWon c529c247bb PyRef::into_ref() (#3744) 2025-07-03 14:07:30 +09:00
Jeong YunWon 3673372d3d Fix cspell warnings 2025-06-27 12:32:59 +09:00
Jeong, YunWon fe2c9bf361 Fix stable clippy (#5843) 2025-06-27 12:12:21 +09:00
Rex Ledesma 4e2e0b41c6 chore: add ruff format --check (#5774)
* chore: add `ruff format --check`

* fix tests
2025-05-12 14:20:01 +09:00
Noa 6b72d2ef5d Check+lint examples, tests, and benches in CI 2025-03-28 11:26:12 +09:00
a6b4ef7f5d Replace Python parser with ruff parser (#5494)
* stage1

* compiler pass build

* introduce rustpython-compiler-source

* stage2

* fixup

* pass compile

* Fix hello world compiler test

* Fix code generation for if-elif-else statement

* Fix code generation for lambda expression

* Fix code generation for integers

* Fix code generation for fstrings

* Fix code generation for if statement

* Fix code generation for if statement

* Fix code generation for if statement

* Fix code generation for fstring

* Fix code generation for class definition

* Replace feature flags

* Initialize frozen core modules

* Allow __future__ import after module doc comment

* Disable ast module

* Commit remaining fixes for compile errors in examples

* Fix some warnings

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Split ast stdlib module into files

* Fix codegen for positional arguments with defaults

* Update ast stdlib module

* Update ast stdlib module

* Extract string and constant handling from expression.rs

* Always add required fields to AST nodes

* Compile doc strings correctly again

* Enable "ast" Cargo feature by default

* Refactor compilation of big integer literal

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Reset barebones example

* Fix some left-over warnings

* Undo accidential change

* Adapt shell to ruff parser

* Pin parser to v0.4.10

* fix clippy

* Add TODO about interactive mode

* Fix compilation of complex number expression

* Remove moved code

* Update test case to ruff v0.4.10

* Apply suggestion

Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>

* Apply suggestion

Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>

* Fix compilation of fstring expression

* Fix compilation of fstring expression

* Fix wasm compile errors

* Attach correct source locations to ast objects

* Fix some more wasm compile errors

* Consider compile mode and enable AST stdlib module again

* Fix incorrect AST source location end column

* Fix compile error if "compiler" feature is not enabled

* Fix regrtests

* Fix some test_ast tests

* Add source range to type ignore

* Fix incompatibility with Rust 2024 edition

* Fix todos by implementing missing ast conversions and deleting unused code

* Appease clippy

* Fix remaining ast tests

* Fix remaining ast tests

* Mark/fix remaining tests

* Fix more

* Hacky windows fix

---------

Co-authored-by: Kangzhi Shi <shikangzhi@gmail.com>
Co-authored-by: Jeong YunWon <jeong@youknowone.org>
Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
Co-authored-by: Noa <coolreader18@gmail.com>
2025-03-19 21:06:03 -05:00
Noa a596568151 Use lexopt for argument parsing (#5602) 2025-03-19 12:28:58 -05:00
Ashwin Naren b870b0e1b5 2024 edition formatting
Signed-off-by: Ashwin Naren <arihant2math@gmail.com>
2025-02-26 11:48:22 -08:00
James Clarke 1bd143027a Switched to system implementation of libffi and fixed minor lint errors 2024-06-30 13:29:04 +01:00
Jeong, YunWon 6eed8f42c7 split vm.import and vm.import_from (#5212) 2024-04-08 00:07:38 +09:00
rrupy 2f8e5189d3 Use ast::Suite::parse instead of deprecated parse_program. (#5186) 2024-03-05 22:37:42 +09:00
Jeong, YunWon bda7c5cf06 Fix examples/package_embed (#5096) 2023-10-18 15:27:51 -07:00
Jeong YunWon bac1cdc55f Remove compiler_core::CompileError 2023-05-06 18:22:25 +09:00
Jeong YunWon 98eef0804e Method overhaul with PyMethodDef 2023-04-30 17:33:23 +09:00
Jeong YunWon f314328294 lifetime bounded TryFromBorrwoedObject 2023-03-19 04:57:06 +09:00
Jeong YunWon 310a83427c cspell derive 2023-03-16 22:42:03 +09:00
Jeong YunWon 21afc1933a Update cspell for compiler 2023-03-16 22:39:09 +09:00
Jeong YunWon b60271a6cf deprecate vm.invoke 2023-03-08 01:35:30 +09:00
Jeong YunWon f078f79c90 Flatten rustpython_parser interface 2023-02-22 20:32:31 +09:00
Jeong YunWon 3942a08e0c Simplify examples/call_between_rust_and_python 2023-02-09 21:31:32 +09:00
Jan Krecke a442056e24 changed unraisable exception type to base exception
- print exception type to screen to match cpython behaviour
- included sample script to illustrate behaviour at sys.exit()
2022-12-25 13:58:15 +09:00
Jeong YunWon d935fbc445 Fix nightly clippy warnings 2022-12-05 12:18:16 +09:00
gelox 340816d435 Add direct member access to example
In the example demonstrating how to call between rust and python add
direct member access.
2022-11-21 21:50:31 +01:00
gelox 445cb6cc8b Format 2022-11-21 21:50:22 +01:00
gelox 88299ef0bb Fix review comments 2022-11-21 21:50:09 +01:00
gelox 357cf7863c Add example for calling between rust and python
Add an example which illustrates how to call between rust and python.
Most importantly there no examples which illustrate how to call rust
from python and it is not obvious how to do this.
2022-11-21 21:49:36 +01:00
Jeong YunWon d94f638784 Fix nightly clippy warnings 2022-10-16 02:38:50 +09:00
Jeong YunWon 55061c7b6a new_syntax_error working without full compiler
Fix #4100
2022-08-23 05:08:24 +09:00
Jeong YunWon fc6d614bf4 rustpython-bytecode -> rustpython-compiler-core 2022-08-22 08:42:20 +09:00
Jeong YunWon d45f3da192 Add source_path to ParseError 2022-08-22 08:42:20 +09:00
joohongpark 9cac7a36dd replace exit with Exitcode (#3956) 2022-08-04 15:46:17 +09:00
Jeong Yunwon bae68873dd Adapt BlockExpr to example/generator.rs 2022-06-16 01:07:26 +09:00
Jeong Yunwon eb8c640518 generator sample 2022-06-16 01:07:26 +09:00
Jeong Yunwon fcd564ad97 Replace kybra_like to project-neutral example 2022-06-16 01:07:26 +09:00
Jeong Yunwon f3de183607 Add example kybra_like 2022-06-16 01:07:26 +09:00
Jeong Yunwon 93d1438345 PyObjBag refers Context instead of vm 2022-05-16 04:45:13 +09:00
Jeong Yunwon 6fd5094c05 Remove Interpreter::default() not to trap users init without stdlib 2022-04-30 05:09:33 +09:00