Commit Graph
20 Commits
Author SHA1 Message Date
Jeong, YunWon 3b4c5f65dd Align compiler and AST behavior with CPython (#8138)
* Align compiler and AST behavior with CPython

Update the compiler pipeline, AST module support, and related VM compile/eval paths toward CPython 3.14 behavior.

Use upstream Ruff parser crates directly and keep RustPython-specific syntax preflight handling in the compile path.

Refresh opcode metadata, snapshots, and targeted tests for the aligned bytecode and AST behavior.

* fix

* Fix PR 8138 CI failures

* Fix match mapping wildcard scan complexity

* fixes

* Use Ruff runtime AST fields

* Minimize residual patch diff

* refactor error handling

* Remove _ast conversion context wrappers

* Clean up _ast conversion glue and compiler warning escalation

Rename the Node conversion parameter `ctx` to `vm` throughout the _ast
module.

Remove the unused compile_program, compile_program_single,
compile_block_expression, and compile_expression forwarders; compile_top
stays as the entry point.

Stash an escalated compiler SyntaxWarning so a non-SyntaxWarning category
propagates unchanged instead of being rewritten to SyntaxError, matching
PyErr_ExceptionMatches(SyntaxWarning) in compiler_warn. Drop the now-unused
CompileWarningError::into_codegen_error.

Replace the PositionalArguments two-variant enum with a struct holding a
shared range and a PositionalArgumentsKind enum.

Assisted-by: Claude

* Represent compiler flags as a CompilerFlags bitflags

Replace the loose PY_CF_*/CO_* i32 constants in compile_mode with a
CompilerFlags bitflags covering the PyCF_* flags and __future__
CO_FUTURE_* bits. The Python-visible ast.PyCF_* attribute values are
derived from it via bits(); parser input modes stay plain consts.

builtins::compile and the compile_string flag handling test flags
through CompilerFlags::contains instead of raw bit math.

Assisted-by: Claude

* Fix PR 8138 test regressions

* Fix PR 8138 CI failures

* Avoid shell interpolation in Linux deps action

* Fix review findings in compiler/AST changes

- frame.rs: compute the MATCH_CLASS class name lazily so a successful
  match no longer takes the type name lock or allocates; the
  __match_args__-not-a-tuple branch reuses it instead of recomputing.
- _ast/constant.rs: add the missing leading space to the with_recursion
  label ("maximum recursion depth exceeded during compilation").
- sys.set_int_max_str_digits: restore the "maxdigits must be 0 or larger
  than {n}" error text.
- codegen: set CO_MAXBLOCKS to 20 in both ir.rs and compile.rs.
- _ast/exception.rs: extract the duplicated excepthandler node guard
  into ensure_excepthandler_node.
- codegen: correct the validate_keywords doc comment.
- Drop redundant "CPython" qualifiers from comments added in this branch.

Assisted-by: Claude
2026-06-26 18:22:08 +09:00
Jeong YunWon 3673372d3d Fix cspell warnings 2025-06-27 12:32:59 +09:00
Jeong, YunWon fe2c9bf361 Fix stable clippy (#5843) 2025-06-27 12:12:21 +09:00
Noa 6b72d2ef5d Check+lint examples, tests, and benches in CI 2025-03-28 11:26:12 +09:00
a6b4ef7f5d Replace Python parser with ruff parser (#5494)
* stage1

* compiler pass build

* introduce rustpython-compiler-source

* stage2

* fixup

* pass compile

* Fix hello world compiler test

* Fix code generation for if-elif-else statement

* Fix code generation for lambda expression

* Fix code generation for integers

* Fix code generation for fstrings

* Fix code generation for if statement

* Fix code generation for if statement

* Fix code generation for if statement

* Fix code generation for fstring

* Fix code generation for class definition

* Replace feature flags

* Initialize frozen core modules

* Allow __future__ import after module doc comment

* Disable ast module

* Commit remaining fixes for compile errors in examples

* Fix some warnings

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Split ast stdlib module into files

* Fix codegen for positional arguments with defaults

* Update ast stdlib module

* Update ast stdlib module

* Extract string and constant handling from expression.rs

* Always add required fields to AST nodes

* Compile doc strings correctly again

* Enable "ast" Cargo feature by default

* Refactor compilation of big integer literal

* Update ast stdlib module

* Update ast stdlib module

* Update ast stdlib module

* Reset barebones example

* Fix some left-over warnings

* Undo accidential change

* Adapt shell to ruff parser

* Pin parser to v0.4.10

* fix clippy

* Add TODO about interactive mode

* Fix compilation of complex number expression

* Remove moved code

* Update test case to ruff v0.4.10

* Apply suggestion

Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>

* Apply suggestion

Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>

* Fix compilation of fstring expression

* Fix compilation of fstring expression

* Fix wasm compile errors

* Attach correct source locations to ast objects

* Fix some more wasm compile errors

* Consider compile mode and enable AST stdlib module again

* Fix incorrect AST source location end column

* Fix compile error if "compiler" feature is not enabled

* Fix regrtests

* Fix some test_ast tests

* Add source range to type ignore

* Fix incompatibility with Rust 2024 edition

* Fix todos by implementing missing ast conversions and deleting unused code

* Appease clippy

* Fix remaining ast tests

* Fix remaining ast tests

* Mark/fix remaining tests

* Fix more

* Hacky windows fix

---------

Co-authored-by: Kangzhi Shi <shikangzhi@gmail.com>
Co-authored-by: Jeong YunWon <jeong@youknowone.org>
Co-authored-by: Jeong, YunWon <69878+youknowone@users.noreply.github.com>
Co-authored-by: Noa <coolreader18@gmail.com>
2025-03-19 21:06:03 -05:00
Noa a596568151 Use lexopt for argument parsing (#5602) 2025-03-19 12:28:58 -05:00
Ashwin Naren b870b0e1b5 2024 edition formatting
Signed-off-by: Ashwin Naren <arihant2math@gmail.com>
2025-02-26 11:48:22 -08:00
James Clarke 1bd143027a Switched to system implementation of libffi and fixed minor lint errors 2024-06-30 13:29:04 +01:00
rrupy 2f8e5189d3 Use ast::Suite::parse instead of deprecated parse_program. (#5186) 2024-03-05 22:37:42 +09:00
Jeong YunWon f078f79c90 Flatten rustpython_parser interface 2023-02-22 20:32:31 +09:00
Jeong YunWon d935fbc445 Fix nightly clippy warnings 2022-12-05 12:18:16 +09:00
Jeong YunWon d94f638784 Fix nightly clippy warnings 2022-10-16 02:38:50 +09:00
Jeong YunWon d45f3da192 Add source_path to ParseError 2022-08-22 08:42:20 +09:00
Noah bcf58d9091 Misc fixes 2021-01-14 12:37:37 -06:00
Jeong YunWon 1bac582921 &str::to_string -> &str::to_owned for literals 2020-02-05 22:55:33 +09:00
Windel Bouwman dbc562c613 Implement clippy hint. 2019-08-11 09:57:40 +02:00
Windel Bouwman a400f6631c Improve lexing of numbers with underscores. 2019-08-11 09:29:21 +02:00
Windel Bouwman 2fdfcb96a5 Add parser support for async for loop. Add yield expression inside parentheses. 2019-08-07 20:11:27 +02:00
Windel Bouwman 1fddce461f Implement review comments and improve parsing a bit. 2019-07-25 23:54:31 +02:00
Windel Bouwman a06bd3ecea Add an example usecase of the parser. 2019-07-24 20:13:30 +02:00