* common: add PyPy CJK codec engines
Assisted-by: Codex:GPT-5
* common: mark CJK mapping tables generated
Assisted-by: Codex:GPT-5
* common: add reusable byte and text engines
Assisted-by: Codex:GPT-5
* common: record opt-in engine dependencies
Assisted-by: Codex:GPT-5
* stdlib: add `_multibytecodec` and the six `_codecs_*` modules
Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec`
holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`,
`MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and
the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the
encodings of its region.
`rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`.
Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from
building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the
escape-throughout flag, and resets only G0 and the shifted flag.
Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`,
`test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec.
`test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected
failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping.
Assisted-by: Claude Code:claude-opus-5
* stdlib: use the byte and text engines from `rustpython-common`
`binascii`, `_json` and `_socket`'s address converters now call the engines in
`rustpython-common` instead of carrying their own copies, and `rustpython-stdlib`
enables the `binascii`, `inet` and `json` features for them. That leaves
`base64` and `crc32fast` with no caller in `rustpython-stdlib`, and
`crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code,
goes away with its last caller.
The address converters change behaviour. `inet_aton` reads the lenient forms
(`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a
redundant leading zero and a group of five hex digits, `inet_ntop` writes the
IPv4-compatible form as `::c000:201`, and an address family with no converter
raises `EAFNOSUPPORT` instead of a bare `OSError`.
Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX`
escape it needs to carry `_json`'s workload. Against the code it replaces,
`json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and
`json.loads` 1.2x.
`binascii.a2b_base64` now reports the messages `binascii.c` does: no
`error decoding base64: ` prefix, and the data-character count in parentheses.
Assisted-by: Claude Code:claude-opus-5
* test: enable the Windows cp932 codec case
The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure.
Assisted-by: Codex:GPT-5
* Align CFG cleanup bytecode with CPython
* Bytecode parity: fblock unwind, fstring join, folding, scope
- compile.rs: unwind_fblock_stack returns whether a finally ran so
return-statement emission can adjust location handling; restructure
try/except/finally cleanup to preserve or drop boundary NOPs based on
whether the body falls through; rework f-string lowering with
count/join helpers; remove the per-collection-type heuristic for
AST-level folding and defer to flowgraph passes; add several folding
helpers and a ComprehensionLoopControl enum.
- ir.rs: re-run unary/binop folding around tuple folding, add
reorder_conditional_scope_exit_and_jump_back_blocks and several block
classification helpers, add MAX_STR_SIZE, change is_exit_without_lineno
to take the block list.
- symboltable.rs: in analyze_cells, remove names owned as cells in
function-like scopes from the parent's free set; mark lambda scope
type explicitly.
* Refine CFG scope-exit backedge ordering
* Convert parsing logic to python
* Trigger job. (DROP ME BEFORE MERGE!)
* Missing EOF
* Add found modules to set
* Suggestion by @fanninpm
* Fix missing trailing slash
* Ensure no `.py` at module name
* Strip any file auffix
* Revert "Trigger job. (DROP ME BEFORE MERGE!)"
This reverts commit 8cf9651a24.
* Handle quoted args
* Update `test_unicodedata.py` to 3.14.4
* Add test asset
* Mark failing tests
* Add `expectedSuccess` docorator
* Add support for nee decorator in patch_spec
* Reapply patches
* Align bytecode codegen structure with CPython 3.14
* Bytecode parity - constant folding, annotation ordering, superinstruction alignment
- Add BoolOp constant folding with short-circuit semantics in compile_expression
- Add constant truthiness evaluation for assert statement optimization
- Disable const collection/boolop folding in starred unpack and assignment contexts
- Move annotation block generation after body with AnnotationsPlaceholder splicing
- Reorder insert_superinstructions to run before push_cold_blocks (matching flowgraph.c)
- Lower LOAD_CLOSURE after superinstructions to avoid false LOAD_FAST_LOAD_FAST
- Add ToBool before PopJumpIf in comparisons and chained compare cleanup blocks
- Unify annotation dict building to always use incremental BuildMap + StoreSubscr
- Add TrueDivide constant folding for integer operands
- Fold constant sets to Frozenset (not Tuple) in try_fold_constant_collection
- Add PyVmBag for frozenset constant materialization in code objects
- Add remove_redundant_const_pop_top_pairs pass and peephole const+branch folding
- Emit Nop for skipped constant expressions and constant-true asserts
- Preserve comprehension local ordering by source-order bound name collection
- Simplify annotation scanning in symboltable (remove simple-name gate)
* Fix CI regressions in marshal and fast-local ops
* impl more
* Align bytecode codegen with CPython structure
* Bytecode parity - comprehension/except scope ordering, load_fast_borrow fixes
- Reorder comprehension symbol-table walk so the outermost iterator
registers its sub_tables in the enclosing scope before the comp
scope, and rescan elt/ifs in CPython's order. Codegen peeks past the
outermost iterator's nested scopes to find the comprehension table.
- For plain try/except, emit handler sub_tables before the else block
so codegen's linear sub_table cursor stays aligned.
- Rename `collect_simple_annotations` to `collect_annotations` and
evaluate non-simple annotations during __annotate__ compilation to
preserve source-order side effects while keeping the simple-name
index stable.
- Dedupe equivalent code constants in `arg_constant` and add a
structural equality check on `CodeObject`.
- Disable LOAD_FAST_BORROW for the tail end block when a try has a
bare `except:` clause, and have `new_block` inherit the flag from
the current block.
- Remove `cfg!(debug_assertions)` guard around the
`optimize_load_fast_borrow` start-depth check so mismatches are
handled (return instead of assert) in release builds.
- Collapse nop-only blocks that precede a return epilogue and hoist
the prior line number into the next real instruction so the
line table matches.
- Unmark now-passing `test_consts_in_conditionals`,
`test_load_fast_unknown_simple`,
`test_load_fast_known_because_already_loaded`, and PEP 646 f3/f4
annotation checks.
* Bytecode parity - try/except line tracking, assert 0 shape
- In `compile_try_except`, drop the leading Nop and set the end
block's source range from the last orelse/body statement so line
events after the try fall on the right line.
- Recognise constant-false asserts as the direct-raise shape (no
ToBool/PopJumpIfFalse) and flip the test assertion accordingly.
- Extend `remove_redundant_nops_in_blocks` to also look through a
trailing nop before a return-epilogue pair (LoadConst/ReturnValue
or LoadSmallInt/ReturnValue) so the epilogue keeps the correct
line number.
- Rename `preds` to `predecessor_blocks` in the LOAD_FAST_BORROW
disable pass and add a test-only `debug_late_cfg_trace` helper.
- Regenerate the `nested_double_async_with` snapshot: the tail
reference to `stop_exc` now emits LOAD_FAST instead of
LOAD_FAST_BORROW.
* Bytecode parity - iter folding, break/continue line, cold inlining
- Fold a constant list iterable into a constant tuple in for-loop
iterable position, matching the CPython optimizer, and strip a
redundant LIST_TO_TUPLE immediately before GET_ITER in the IR
peephole pass.
- Emit a Nop at the break/continue source range before unwinding
so line events land on the break/continue statement instead of
the following instruction.
- Drop `propagate_disable_load_fast_borrow`; the forward propagation
was over-zealous and the per-block inheritance in `new_block` plus
the bare-except marker are enough.
- Relax `inline_small_or_no_lineno_blocks` so small exit blocks at
the tail of a cold block are always inlined, not just return
epilogues.
- Add codegen tests covering the LIST_TO_TUPLE/GET_ITER peephole and
the late-CFG trace helper for a for-loop list-literal iterable.
- Use POP_TOP instead of POP_ITER for for-loop break/return cleanup
- Expand duplicate_end_returns to clone final return for jump predecessors
- Restrict late jump threading pass to unconditional jumps only
- Skip exception blocks in inline/reorder passes
- Simplify threaded_jump_instr NoInterrupt handling
* Fix exception handling: except* chaining, finally cleanup, RERAISE
- Align except* bytecode chaining
- Fix exception state model and finally handler cleanup
- Fix RERAISE to only pop exception, preserve values below
* Port IR optimization passes from flowgraph.c
- BUILD_TUPLE n + UNPACK_SEQUENCE n elimination
- Dead store elimination within basic blocks
- apply_static_swaps for SWAP reduction
* Add bytecode comparison and disassembly dump scripts
- compare_bytecode.py: compare CPython vs RustPython bytecode output
- dis_dump.py: extract disassembly in normalized JSON format
* use `prek` for unified linting
* Fix actionlint error
* Generate metadata when specific files change
* `check_redundant_patches.py` to accept glob path
* Test
* revert defective changes
* use `rustfmt` over `cargo fmt` for individual files
* debug reviewdog
* rustfmt
* Move comment to correct location
* defevtive fmt test
* Fail with reviewdog
* fix reviewdog perms
* Try to use present token
* without checks oerms
* put normal perms
* fmt
* [update_lib] fast date lookup for todo
* add deps
* Auto-format: ruff format
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
* Make auto-mark output deterministic and fix blank line leak
Sort set iteration in build_patches and dict iteration in
_iter_patch_lines Phase 2 so expectedFailure markers are
always added in alphabetical order.
Include preceding blank line in _method_removal_range so
removing a super-call override doesn't leave behind the
blank line that was added with the method.
* deps code
* auto-mark handles crash better
* Auto-format: ruff format
---------
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Bug: update_lib quick ast was not committing _ast_unparse.py
because git_commit() only added lib_path and test_paths,
but not hard_deps.
Fixed by:
- Add hard_deps parameter to git_commit()
- Collect hard_deps from DEPENDENCIES in main()
- Add hard_deps to paths_to_add in git_commit()
Jeong, YunWon