mirror of
https://github.com/RustPython/RustPython.git
synced 2026-10-28 10:44:36 +08:00
Jeong, YunWon
322f294a21
* common: add PyPy CJK codec engines
Assisted-by: Codex:GPT-5
* common: mark CJK mapping tables generated
Assisted-by: Codex:GPT-5
* common: add reusable byte and text engines
Assisted-by: Codex:GPT-5
* common: record opt-in engine dependencies
Assisted-by: Codex:GPT-5
* stdlib: add `_multibytecodec` and the six `_codecs_*` modules
Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec`
holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`,
`MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and
the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the
encodings of its region.
`rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`.
Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from
building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the
escape-throughout flag, and resets only G0 and the shifted flag.
Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`,
`test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec.
`test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected
failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping.
Assisted-by: Claude Code:claude-opus-5
* stdlib: use the byte and text engines from `rustpython-common`
`binascii`, `_json` and `_socket`'s address converters now call the engines in
`rustpython-common` instead of carrying their own copies, and `rustpython-stdlib`
enables the `binascii`, `inet` and `json` features for them. That leaves
`base64` and `crc32fast` with no caller in `rustpython-stdlib`, and
`crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code,
goes away with its last caller.
The address converters change behaviour. `inet_aton` reads the lenient forms
(`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a
redundant leading zero and a group of five hex digits, `inet_ntop` writes the
IPv4-compatible form as `::c000:201`, and an address family with no converter
raises `EAFNOSUPPORT` instead of a bare `OSError`.
Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX`
escape it needs to carry `_json`'s workload. Against the code it replaces,
`json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and
`json.loads` 1.2x.
`binascii.a2b_base64` now reports the messages `binascii.c` does: no
`error decoding base64: ` prefix, and the data-character count in parentheses.
Assisted-by: Claude Code:claude-opus-5
* test: enable the Windows cp932 codec case
The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure.
Assisted-by: Codex:GPT-5
74 lines
2.3 KiB
Plaintext
74 lines
2.3 KiB
Plaintext
Lib/** linguist-vendored
|
|
Cargo.lock linguist-generated
|
|
*.snap linguist-generated -merge
|
|
Lib/*.py text working-tree-encoding=UTF-8 eol=LF
|
|
**/*.rs text working-tree-encoding=UTF-8 eol=LF
|
|
crates/vm/src/stdlib/ast/gen.rs linguist-generated -merge
|
|
crates/doc/src/data.inc.rs generated
|
|
crates/compiler-core/src/bytecode/opcode_metadata.rs generated
|
|
crates/common/src/encodings/cjk/mappings_*.rs generated
|
|
|
|
.github/workflows/*.lock.yml linguist-generated=true merge=ours
|
|
|
|
# Binary data types
|
|
*.aif binary
|
|
*.aifc binary
|
|
*.aiff binary
|
|
*.au binary
|
|
*.bmp binary
|
|
*.exe binary
|
|
*.icns binary
|
|
*.gif binary
|
|
*.ico binary
|
|
*.jpg binary
|
|
*.pck binary
|
|
*.pdf binary
|
|
*.png binary
|
|
*.psd binary
|
|
*.tar binary
|
|
*.wav binary
|
|
*.whl binary
|
|
*.zip binary
|
|
|
|
# Text files that should not be subject to eol conversion
|
|
[attr]noeol -text
|
|
|
|
Lib/test/cjkencodings/* noeol
|
|
Lib/test/tokenizedata/coding20731.py noeol
|
|
Lib/test/decimaltestdata/*.decTest noeol
|
|
Lib/test/test_email/data/*.txt noeol
|
|
Lib/test/xmltestdata/* noeol
|
|
|
|
# Shell scripts should have LF even on Windows because of Cygwin
|
|
Lib/venv/scripts/common/activate text eol=lf
|
|
Lib/venv/scripts/posix/* text eol=lf
|
|
|
|
# CRLF files
|
|
[attr]dos text eol=crlf
|
|
|
|
# Language aware diff headers
|
|
# https://tekin.co.uk/2020/10/better-git-diff-output-for-ruby-python-elixir-and-more
|
|
# https://gist.github.com/tekin/12500956bd56784728e490d8cef9cb81
|
|
*.css diff=css
|
|
*.html diff=html
|
|
*.py diff=python
|
|
*.md diff=markdown
|
|
*.inc.rs diff=rust
|
|
|
|
# Generated files
|
|
# https://github.com/github/linguist/blob/master/docs/overrides.md
|
|
#
|
|
# To always hide generated files in local diffs, mark them as binary:
|
|
# $ git config diff.generated.binary true
|
|
#
|
|
[attr]generated linguist-generated=true diff=generated
|
|
|
|
Lib/_opcode_metadata.py generated
|
|
Lib/keyword.py generated
|
|
Lib/idlelib/help.html generated
|
|
Lib/test/certdata/*.pem generated
|
|
Lib/test/certdata/*.0 generated
|
|
Lib/test/levenshtein_examples.json generated
|
|
Lib/test/test_stable_abi_ctypes.py generated
|
|
Lib/token.py generated
|