17 Commits
Author SHA1 Message Date
Jeong, YunWon 322f294a21 common: add VM-independent codec and text engines (#8622)
* common: add PyPy CJK codec engines

Assisted-by: Codex:GPT-5

* common: mark CJK mapping tables generated

Assisted-by: Codex:GPT-5

* common: add reusable byte and text engines

Assisted-by: Codex:GPT-5

* common: record opt-in engine dependencies

Assisted-by: Codex:GPT-5

* stdlib: add `_multibytecodec` and the six `_codecs_*` modules

Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec`
holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`,
`MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and
the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the
encodings of its region.

`rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`.

Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from
building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the
escape-throughout flag, and resets only G0 and the shifted flag.

Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`,
`test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec.
`test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected
failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping.

Assisted-by: Claude Code:claude-opus-5

* stdlib: use the byte and text engines from `rustpython-common`

`binascii`, `_json` and `_socket`'s address converters now call the engines in
`rustpython-common` instead of carrying their own copies, and `rustpython-stdlib`
enables the `binascii`, `inet` and `json` features for them. That leaves
`base64` and `crc32fast` with no caller in `rustpython-stdlib`, and
`crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code,
goes away with its last caller.

The address converters change behaviour. `inet_aton` reads the lenient forms
(`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a
redundant leading zero and a group of five hex digits, `inet_ntop` writes the
IPv4-compatible form as `::c000:201`, and an address family with no converter
raises `EAFNOSUPPORT` instead of a bare `OSError`.

Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX`
escape it needs to carry `_json`'s workload. Against the code it replaces,
`json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and
`json.loads` 1.2x.

`binascii.a2b_base64` now reports the messages `binascii.c` does: no
`error decoding base64: ` prefix, and the data-character count in parentheses.

Assisted-by: Claude Code:claude-opus-5

* test: enable the Windows cp932 codec case

The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure.

Assisted-by: Codex:GPT-5
2026-08-31 15:44:55 +09:00
Shahar Naveh c3e1f0fa37 Get all builtin types dynamiclly (#8095) 2026-06-15 11:11:32 +09:00
Shahar Naveh 26dfeae512 Autogen opcodes metadata (#7983) 2026-05-27 16:44:58 +09:00
Jeong, YunWon 3aeecc5751 Add gh-aw agentic workflow for upgrading Python libraries (#7117)
* Add gh-aw agentic workflow for upgrading Python libraries

- upgrade-pylib.md: Claude-based workflow triggered via workflow_dispatch
- Runs `scripts/update_lib quick <name>` to sync a library from CPython
- Creates a PR with the result, referencing issue #6839
2026-02-14 11:26:03 +09:00
Jeong, YunWon d90792258d import .gitattributes from CPython (#6939) 2026-02-02 00:27:21 +09:00
Shahar Naveh db1adaa2c2 Move __doc__ crate to crates/doc (#6234)
* Add `__doc__` crate

* Base auto-generate ci

* Add dummy files

* Update docs

* Set codegen-units to 1 for doc db

* Mark `*.inc.rs` as auto generated

* Disable doctest

* Reset docs
2025-11-14 23:47:51 +09:00
Noa bfe72689fc Remove -merge from Cargo.lock gitattributes 2025-03-20 14:28:03 +09:00
Noa 2721f2de3f Fix a bunch of random tests (#5533) 2025-02-25 08:41:54 +09:00
Jeong YunWon dc569e9921 Refactoring 2023-05-10 19:35:27 +09:00
Jeong YunWon 721d124848 *.snap linguist-generated -merge 2022-08-22 04:43:03 +09:00
Jeong YunWon fb5ac9e79b reorganize compiler crates 2022-08-22 04:42:42 +09:00
Jeong YunWon c3cab9e8cc remove generated lalrpop result from repository 2022-08-18 08:20:35 +09:00
Jeong YunWon 79f985c256 fix *.{rs,lalrpop} and Lib/*.py to eol=lf 2022-08-18 07:44:50 +09:00
Noa 07ab9737cd Mark generated lalrpop as -merge -diff 2022-04-26 20:39:16 -05:00
Noa ceb59abbd9 Speedup compile time for rustpython-parser 2022-04-24 10:36:02 -05:00
Nikita Sobolev 353ccf1f3f Make this a Rust project once again
Subfolders were not ignored.
Docs: https://github.com/github/linguist/blob/0c7f82f88a2fb482db3a02812cbb4b8aab7308d1/docs/overrides.md#vendored-code
2021-08-30 16:01:37 +03:00
Noah 25013169f7 Fix Github language statistics to make this a Rust project 2019-09-16 18:01:24 +00:00