Jeong, YunWon
322f294a21
common: add VM-independent codec and text engines ( #8622 )
...
* common: add PyPy CJK codec engines
Assisted-by: Codex:GPT-5
* common: mark CJK mapping tables generated
Assisted-by: Codex:GPT-5
* common: add reusable byte and text engines
Assisted-by: Codex:GPT-5
* common: record opt-in engine dependencies
Assisted-by: Codex:GPT-5
* stdlib: add `_multibytecodec` and the six `_codecs_*` modules
Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec`
holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`,
`MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and
the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the
encodings of its region.
`rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`.
Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from
building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the
escape-throughout flag, and resets only G0 and the shifted flag.
Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`,
`test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec.
`test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected
failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping.
Assisted-by: Claude Code:claude-opus-5
* stdlib: use the byte and text engines from `rustpython-common`
`binascii`, `_json` and `_socket`'s address converters now call the engines in
`rustpython-common` instead of carrying their own copies, and `rustpython-stdlib`
enables the `binascii`, `inet` and `json` features for them. That leaves
`base64` and `crc32fast` with no caller in `rustpython-stdlib`, and
`crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code,
goes away with its last caller.
The address converters change behaviour. `inet_aton` reads the lenient forms
(`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a
redundant leading zero and a group of five hex digits, `inet_ntop` writes the
IPv4-compatible form as `::c000:201`, and an address family with no converter
raises `EAFNOSUPPORT` instead of a bare `OSError`.
Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX`
escape it needs to carry `_json`'s workload. Against the code it replaces,
`json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and
`json.loads` 1.2x.
`binascii.a2b_base64` now reports the messages `binascii.c` does: no
`error decoding base64: ` prefix, and the data-character count in parentheses.
Assisted-by: Claude Code:claude-opus-5
* test: enable the Windows cp932 codec case
The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure.
Assisted-by: Codex:GPT-5
2026-08-31 15:44:55 +09:00
..
2025-08-01 22:39:00 +09:00
2026-02-17 21:33:54 +09:00
2026-03-01 11:36:46 +09:00
2026-02-15 14:30:45 +09:00
2025-08-04 20:08:17 +09:00
2025-08-23 19:45:53 +03:00
2025-07-25 19:04:49 +09:00
2025-08-23 19:45:53 +03:00
2022-08-15 16:09:32 +09:00
2026-06-15 11:19:01 +09:00
2026-01-13 22:05:32 +09:00
2026-01-21 20:44:26 +09:00
2025-08-24 10:36:07 +03:00
2026-08-22 20:56:18 +09:00
2026-08-25 22:01:24 +09:00
2026-08-03 01:38:44 +09:00
2026-03-10 12:57:29 +09:00
2026-05-17 20:54:56 +09:00
2026-08-26 14:38:10 +00:00
2026-03-03 18:01:18 +09:00
2026-08-31 15:44:55 +09:00
2026-08-20 12:47:36 +09:00
2026-08-30 13:45:54 +09:00
2026-06-02 18:06:36 +09:00
2026-08-25 22:01:24 +09:00
2026-08-23 14:32:56 +09:00
2026-07-15 23:36:43 +09:00
2026-06-02 00:35:17 +09:00
2026-01-31 23:09:10 +09:00
2026-01-31 23:09:10 +09:00
2026-02-13 22:15:19 +09:00
2026-05-19 13:20:00 +09:00
2026-08-31 14:23:29 +09:00
2026-08-31 14:23:29 +09:00
2026-08-16 20:13:10 +09:00
2026-08-25 22:01:24 +09:00
2026-06-15 11:20:13 +09:00
2026-05-19 13:20:00 +09:00
2026-07-28 10:20:04 +09:00
2026-05-12 20:52:20 +09:00
2026-08-31 15:44:55 +09:00
2026-07-15 23:36:43 +09:00
2026-01-30 19:24:33 +09:00
2023-02-28 17:11:41 +09:00
2025-08-23 19:45:53 +03:00
2026-02-11 09:39:04 +09:00
2023-04-20 15:36:28 +09:00
2026-01-06 21:35:21 +09:00
2023-03-02 15:01:12 +09:00
2026-01-10 22:10:35 -05:00
2026-03-16 15:25:29 +09:00
2026-08-22 20:56:18 +09:00
2026-03-11 22:20:37 +09:00
2026-03-25 14:02:38 +09:00
2026-05-30 20:16:10 +09:00
2026-03-12 20:48:22 +09:00
2026-07-27 21:36:27 +09:00
2025-07-20 10:31:30 +09:00
2025-12-30 13:42:47 +09:00
2024-05-08 12:52:33 +09:00
2024-05-08 12:52:33 +09:00
2026-05-24 19:56:35 +09:00
2026-01-10 22:10:35 -05:00
2022-08-15 02:36:46 +09:00
2022-08-15 04:24:58 +09:00
2022-08-15 04:24:58 +09:00
2022-08-15 04:24:58 +09:00
2022-08-15 04:24:58 +09:00
2022-08-15 04:24:58 +09:00
2026-08-23 00:13:15 +09:00
2026-02-27 09:07:23 +09:00
2023-03-01 18:06:50 +09:00
2026-05-24 19:58:16 +09:00
2025-07-25 10:49:05 +09:00
2026-08-26 14:38:10 +00:00
2026-02-05 00:21:34 +09:00
2026-08-26 14:38:10 +00:00
2026-08-23 00:13:15 +09:00
2026-08-23 00:13:15 +09:00
2022-10-18 03:26:29 +09:00
2026-01-03 00:14:43 +09:00
2026-01-03 00:14:43 +09:00
2026-06-02 00:35:17 +09:00
2025-10-22 10:57:47 +03:00
2026-06-02 00:35:17 +09:00
2026-01-03 00:14:43 +09:00
2026-02-15 14:30:45 +09:00
2026-05-01 19:48:41 +09:00
2026-05-17 19:49:42 +09:00
2026-08-25 22:01:24 +09:00
2026-02-27 04:36:46 +09:00
2026-05-31 21:47:03 +09:00
2025-07-25 10:50:17 +09:00
2025-08-23 19:45:53 +03:00
2026-05-12 20:51:33 +09:00
2026-06-15 11:19:01 +09:00
2023-03-01 18:06:50 +09:00
2026-08-20 07:08:03 +09:00
2026-06-26 18:24:20 +09:00
2026-03-01 11:36:46 +09:00
2026-06-25 22:33:13 +09:00
2026-08-30 13:45:54 +09:00
2026-02-15 20:33:58 +09:00
2026-01-18 20:00:15 +09:00
2026-02-02 02:02:12 +09:00
2026-03-18 22:09:00 +09:00
2026-03-02 01:24:15 +09:00
2026-02-13 22:15:52 +09:00
2026-02-11 17:42:16 +09:00
2026-06-02 18:00:54 +09:00
2026-05-13 19:53:18 +09:00
2026-01-25 15:20:05 +09:00
2026-08-23 00:13:15 +09:00
2026-08-23 08:33:33 +09:00
2026-05-19 13:20:00 +09:00
2026-02-02 10:03:42 +09:00
2026-08-22 20:56:18 +09:00
2026-06-26 18:22:08 +09:00
2026-04-19 09:14:46 +09:00
2026-06-25 22:33:51 +09:00
2026-05-24 19:56:35 +09:00
2023-11-23 15:40:57 +08:00
2026-08-22 20:52:01 +09:00
2026-08-22 12:43:05 +09:00
2026-02-08 10:00:50 +09:00
2026-04-15 14:17:48 +09:00
2026-02-13 13:36:33 +02:00
2026-08-16 21:32:29 +09:00
2026-02-13 14:13:30 +09:00
2026-06-26 18:22:08 +09:00
2026-08-20 07:08:03 +09:00
2026-07-21 00:30:10 +09:00
2026-02-21 14:12:18 +09:00
2026-08-26 14:38:10 +00:00
2026-04-30 04:40:44 +09:00
2026-05-10 00:30:26 +09:00
2026-08-30 13:45:54 +09:00
2026-05-19 13:20:00 +09:00
2026-04-19 09:14:35 +09:00
2026-08-30 13:45:54 +09:00
2026-06-16 15:58:12 +09:00
2026-01-24 11:37:32 +09:00
2026-08-14 00:57:10 +09:00
2026-08-22 20:52:01 +09:00
2026-08-29 22:43:11 +09:00
2026-08-31 15:44:55 +09:00
2026-08-31 15:44:55 +09:00
2026-08-31 15:44:55 +09:00
2026-08-31 15:44:55 +09:00
2026-08-31 15:44:55 +09:00
2026-08-31 15:44:55 +09:00
2026-08-31 15:44:55 +09:00
2026-08-31 15:44:55 +09:00
2026-08-31 15:44:55 +09:00
2026-08-31 15:44:55 +09:00
2026-08-31 15:44:55 +09:00
2026-08-31 15:44:55 +09:00
2026-08-26 14:38:10 +00:00
2026-08-16 21:32:29 +09:00
2026-08-22 20:55:49 +09:00
2023-11-30 20:01:48 +08:00
2026-08-22 20:56:18 +09:00
2026-08-26 14:38:10 +00:00
2026-01-27 10:49:05 +09:00
2026-01-27 10:49:05 +09:00
2026-08-29 22:43:11 +09:00
2026-08-23 00:13:15 +09:00
2026-04-29 18:29:31 +09:00
2026-05-10 00:28:30 +09:00
2026-05-12 00:41:05 +09:00
2026-05-12 00:40:32 +09:00
2026-08-25 22:01:24 +09:00
2024-06-24 19:00:54 +09:00
2026-05-05 14:04:12 +09:00
2026-05-31 21:47:03 +09:00
2026-08-23 00:13:15 +09:00
2026-05-12 00:42:10 +09:00
2026-01-01 14:44:53 +09:00
2026-01-19 02:45:47 +09:00
2026-02-11 16:50:23 +09:00
2026-08-18 22:24:02 +09:00
2026-04-19 22:33:27 +09:00
2026-06-02 18:00:54 +09:00
2026-06-12 23:50:07 +09:00
2026-08-26 14:38:10 +00:00
2026-08-30 13:45:54 +09:00
2023-04-02 13:42:27 +00:00
2026-08-22 20:56:18 +09:00
2026-02-13 13:36:33 +02:00
2026-08-26 14:38:10 +00:00
2026-02-08 20:36:35 +09:00
2026-08-23 00:13:15 +09:00
2026-08-23 14:32:56 +09:00
2022-08-09 03:36:26 +09:00
2026-02-13 18:07:25 +09:00
2026-06-25 22:33:51 +09:00
2023-02-28 17:11:41 +09:00
2023-02-28 17:11:41 +09:00
2026-03-01 10:39:06 +09:00
2026-03-01 19:27:33 +09:00
2023-11-30 21:01:17 +08:00
2026-05-30 20:16:10 +09:00
2026-05-12 00:39:37 +09:00
2026-06-12 23:49:15 +09:00
2026-08-26 14:38:10 +00:00
2026-08-09 14:58:53 +09:00
2023-11-30 21:06:09 +08:00
2026-02-10 00:48:22 +09:00
2026-01-31 05:49:16 +09:00
2026-08-22 23:54:46 +00:00
2026-01-31 05:49:16 +09:00
2026-01-31 05:49:16 +09:00
2026-08-23 14:32:56 +09:00
2026-02-12 01:01:35 +09:00
2026-07-02 16:32:33 +09:00
2026-02-22 21:31:42 +09:00
2026-06-15 11:20:13 +09:00
2026-06-01 00:08:40 +09:00
2026-04-03 00:49:55 +09:00
2026-01-03 08:27:18 +09:00
2026-05-12 20:52:08 +09:00
2026-02-16 00:31:57 +09:00
2026-06-25 22:33:51 +09:00
2026-05-10 00:28:50 +09:00
2026-07-26 14:31:34 +09:00
2026-08-20 12:47:36 +09:00
2026-02-13 22:15:19 +09:00
2026-03-07 20:20:16 +09:00
2026-08-09 14:58:07 +09:00
2026-04-19 09:15:12 +09:00
2026-07-08 15:21:41 +09:00
2026-02-09 01:40:26 +09:00
2026-08-23 14:32:56 +09:00
2026-06-12 23:49:15 +09:00
2026-08-08 02:25:21 +09:00
2026-05-12 20:54:18 +09:00
2026-05-30 20:16:10 +09:00
2026-05-19 13:20:00 +09:00
2026-04-29 18:52:22 +09:00
2026-08-01 00:59:41 +09:00
2026-06-12 23:50:07 +09:00
2026-02-03 14:58:43 +09:00
2026-02-03 22:10:31 +09:00
2026-06-26 18:22:08 +09:00
2026-08-22 20:55:49 +09:00
2026-02-08 16:35:56 +09:00
2026-03-29 00:52:34 +09:00
2026-02-13 22:15:19 +09:00
2026-06-26 18:22:08 +09:00
2026-06-30 23:32:30 +09:00
2026-01-18 01:23:34 +09:00
2026-06-12 23:50:07 +09:00
2026-02-06 00:15:18 +09:00
2026-05-19 13:31:33 +09:00
2026-06-10 11:14:05 +09:00
2026-01-18 01:23:34 +09:00
2026-03-01 10:51:39 +09:00
2026-08-22 20:55:49 +09:00
2026-02-27 05:44:37 +00:00
2026-06-12 23:49:47 +09:00
2026-05-17 19:49:19 +09:00
2026-02-27 05:44:37 +00:00
2026-06-15 11:21:28 +09:00
2026-08-29 17:59:22 +09:00
2026-08-18 22:24:02 +09:00
2026-08-31 15:44:55 +09:00
2026-06-15 11:20:13 +09:00
2026-01-18 01:23:34 +09:00
2026-08-22 20:56:18 +09:00
2026-08-20 07:08:03 +09:00
2022-08-15 04:24:58 +09:00
2026-08-14 16:05:30 +09:00
2024-06-21 22:37:15 +09:00
2026-02-14 21:40:04 +09:00
2025-08-31 10:35:23 +03:00
2026-04-03 00:49:55 +09:00
2026-02-15 22:11:48 +09:00
2026-02-09 12:56:53 +09:00
2026-08-22 20:56:18 +09:00
2026-08-30 17:20:33 +09:00
2026-08-23 00:13:15 +09:00
2026-05-12 00:42:54 +09:00
2026-02-13 13:55:05 +09:00
2026-07-15 23:16:42 +09:00
2026-02-14 18:19:57 +02:00
2026-08-23 08:33:33 +09:00
2025-08-23 22:09:11 +03:00
2026-08-23 00:13:15 +09:00
2026-08-26 14:38:10 +00:00
2026-08-29 23:00:31 +09:00
2026-05-20 08:59:59 +09:00
2026-08-23 00:13:15 +09:00
2026-07-18 16:46:57 +09:00
2026-07-02 16:32:33 +09:00
2026-08-09 15:06:48 +09:00
2026-07-08 15:21:41 +09:00
2026-07-02 16:32:33 +09:00
2026-08-31 15:44:55 +09:00
2024-04-24 01:15:34 +09:00
2026-08-16 20:13:54 +09:00
2025-08-20 14:08:05 +09:00
2026-01-25 00:18:55 +09:00
2026-01-25 00:18:55 +09:00
2024-07-29 19:53:44 +09:00
2026-04-03 01:33:34 +09:00
2026-01-26 10:40:22 +09:00
2026-02-14 09:17:41 +09:00
2026-04-03 01:33:50 +09:00
2026-01-30 20:48:57 +09:00
2026-08-26 14:38:10 +00:00
2026-08-25 22:01:24 +09:00
2026-01-31 19:56:55 -05:00
2026-06-30 23:32:30 +09:00
2026-07-28 10:20:04 +09:00
2026-06-26 18:22:08 +09:00
2026-06-30 23:32:30 +09:00
2026-04-03 00:50:25 +09:00
2026-08-25 22:01:24 +09:00
2026-02-05 19:27:10 +09:00
2026-02-05 19:27:10 +09:00
2026-05-19 13:31:33 +09:00
2026-07-07 09:19:25 +09:00
2026-05-12 00:43:49 +09:00
2026-05-12 20:49:48 +09:00
2026-06-11 13:36:51 +09:00
2026-02-03 22:10:31 +09:00
2026-03-12 18:08:56 +09:00
2026-05-20 08:59:59 +09:00
2026-06-02 18:00:54 +09:00
2026-02-13 20:24:57 +09:00
2025-08-31 11:58:21 +03:00
2026-02-06 01:14:55 +09:00
2026-02-13 13:36:33 +02:00
2026-05-31 21:47:03 +09:00
2026-02-13 13:42:52 +09:00
2026-06-15 11:20:13 +09:00
2026-07-26 13:30:48 +09:00
2026-07-09 20:24:16 +09:00
2026-06-12 23:50:07 +09:00
2026-08-25 22:01:24 +09:00
2026-05-22 03:53:35 +09:00
2026-07-13 18:40:18 +09:00
2026-02-04 18:32:45 +09:00
2025-07-17 00:03:25 +09:00
2026-03-03 19:57:50 +09:00
2026-08-26 14:38:10 +00:00
2026-04-15 14:15:27 +09:00
2026-08-29 22:43:11 +09:00
2026-07-18 16:46:57 +09:00
2026-06-26 18:22:08 +09:00
2026-08-08 02:25:21 +09:00
2026-07-15 23:16:42 +09:00
2026-02-16 01:19:57 +09:00
2026-06-12 23:49:47 +09:00
2026-08-26 14:38:10 +00:00
2026-05-12 20:51:46 +09:00
2026-07-09 20:24:16 +09:00
2025-04-21 21:11:42 -07:00
2026-03-31 15:45:18 +09:00
2026-05-30 20:16:10 +09:00
2026-01-31 20:51:53 -05:00
2025-12-12 10:37:40 +09:00
2026-08-20 07:08:03 +09:00
2026-02-03 13:41:58 +09:00
2026-01-01 17:15:00 +09:00
2026-08-22 20:55:49 +09:00
2026-06-12 23:49:47 +09:00
2026-06-15 11:12:39 +09:00
2026-05-12 00:43:34 +09:00
2026-05-18 00:23:06 +09:00
2026-02-14 21:40:04 +09:00
2026-05-12 20:51:57 +09:00
2026-08-25 22:01:24 +09:00
2026-05-22 20:10:49 +09:00
2026-05-19 13:31:33 +09:00
2026-08-10 00:43:44 +09:00
2026-02-18 03:09:55 +09:00
2026-02-14 09:18:09 +09:00
2026-02-11 18:01:41 +09:00
2026-08-20 07:08:03 +09:00
2026-02-14 09:13:06 +09:00
2026-03-01 10:42:37 +09:00
2025-03-26 23:12:21 -05:00
2026-05-12 00:42:10 +09:00
2026-05-23 20:16:03 +09:00
2026-08-22 20:55:49 +09:00
2026-08-14 00:57:10 +09:00
2026-02-02 08:11:26 +09:00
2026-06-26 18:23:54 +09:00
2026-02-13 13:42:38 +09:00
2026-08-20 12:47:36 +09:00
2026-08-22 20:56:18 +09:00
2026-08-20 12:47:36 +09:00
2026-08-23 14:32:56 +09:00
2026-06-26 18:22:08 +09:00
2026-06-03 22:51:41 +09:00
2026-03-21 22:49:09 +09:00
2026-05-12 00:43:18 +09:00
2026-06-03 22:51:41 +09:00
2026-03-10 11:28:16 +09:00
2026-08-29 17:59:22 +09:00
2026-06-25 22:32:57 +09:00
2026-07-26 13:30:48 +09:00
2026-01-30 20:48:57 +09:00
2026-06-03 09:52:32 +00:00
2026-08-23 08:33:33 +09:00
2026-06-03 09:52:32 +00:00
2026-07-26 13:47:35 +09:00
2026-06-03 09:52:32 +00:00
2026-02-02 08:08:29 +09:00
2026-02-15 14:37:48 +09:00
2026-05-12 20:53:11 +09:00
2026-05-22 20:11:27 +09:00
2026-08-09 14:58:53 +09:00
2026-03-04 05:38:58 +09:00
2026-08-31 15:44:55 +09:00
2026-05-31 21:47:03 +09:00
2026-04-18 15:17:56 +09:00
2026-06-11 12:15:21 +09:00
2026-03-03 21:58:46 +09:00
2026-06-03 18:36:37 +09:00
2026-02-11 09:39:04 +09:00
2026-08-12 16:22:05 +09:00
2026-05-19 13:20:00 +09:00
2026-06-26 18:22:08 +09:00
2026-08-29 21:12:40 +09:00
2026-08-30 22:54:22 +09:00
2026-08-29 21:12:40 +09:00
2026-05-30 20:16:10 +09:00
2025-01-17 18:04:25 -08:00
2026-05-30 20:16:10 +09:00
2025-07-12 20:44:34 +09:00
2026-05-30 20:16:10 +09:00
2026-08-26 22:34:32 +09:00
2025-08-31 12:13:09 +03:00
2026-06-26 18:22:08 +09:00
2026-06-11 13:36:51 +09:00
2026-02-05 16:20:22 +09:00
2026-04-14 08:58:51 +02:00
2026-08-26 14:38:10 +00:00
2026-01-25 00:18:55 +09:00
2026-01-16 23:03:49 +09:00
2026-08-26 14:38:10 +00:00
2026-01-25 00:18:55 +09:00
2026-08-26 14:38:10 +00:00
2026-04-29 18:16:27 +09:00
2026-05-10 00:29:27 +09:00
2026-05-03 00:15:03 +09:00
2026-06-11 13:36:51 +09:00
2026-03-03 21:27:11 +09:00
2026-06-02 18:00:54 +09:00
2026-06-22 16:46:53 +09:00
2025-07-25 10:49:05 +09:00
2025-07-25 10:49:05 +09:00
2026-02-13 13:42:22 +09:00
2026-08-30 17:28:46 +09:00
2026-04-30 04:45:43 +09:00
2026-05-13 19:53:31 +09:00
2026-01-25 00:18:55 +09:00
2026-08-30 23:35:08 +09:00
2026-02-15 22:11:48 +09:00
2026-06-22 16:46:53 +09:00
2026-05-12 22:14:24 +09:00
2026-06-22 16:46:53 +09:00
2026-05-12 00:44:05 +09:00
2024-06-22 15:24:42 +09:00
2025-12-30 00:36:52 +09:00
2026-03-03 20:00:30 +09:00
2026-07-18 16:46:57 +09:00
2026-06-15 11:19:40 +09:00
2026-05-18 00:23:34 +09:00
2026-05-19 13:20:00 +09:00
2026-05-19 13:20:00 +09:00
2026-04-29 18:52:22 +09:00
2026-03-05 17:25:59 +09:00
2025-12-17 00:28:06 +09:00
2026-06-25 22:33:51 +09:00
2026-01-24 16:11:36 +09:00
2026-01-19 00:38:51 +09:00
2026-01-19 00:38:51 +09:00
2023-06-01 15:29:16 +09:00
2023-04-20 15:36:28 +09:00
2026-05-18 00:23:34 +09:00