common: add VM-independent codec and text engines (#8622)

* common: add PyPy CJK codec engines

Assisted-by: Codex:GPT-5

* common: mark CJK mapping tables generated

Assisted-by: Codex:GPT-5

* common: add reusable byte and text engines

Assisted-by: Codex:GPT-5

* common: record opt-in engine dependencies

Assisted-by: Codex:GPT-5

* stdlib: add `_multibytecodec` and the six `_codecs_*` modules

Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec`
holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`,
`MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and
the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the
encodings of its region.

`rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`.

Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from
building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the
escape-throughout flag, and resets only G0 and the shifted flag.

Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`,
`test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec.
`test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected
failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping.

Assisted-by: Claude Code:claude-opus-5

* stdlib: use the byte and text engines from `rustpython-common`

`binascii`, `_json` and `_socket`'s address converters now call the engines in
`rustpython-common` instead of carrying their own copies, and `rustpython-stdlib`
enables the `binascii`, `inet` and `json` features for them. That leaves
`base64` and `crc32fast` with no caller in `rustpython-stdlib`, and
`crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code,
goes away with its last caller.

The address converters change behaviour. `inet_aton` reads the lenient forms
(`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a
redundant leading zero and a group of five hex digits, `inet_ntop` writes the
IPv4-compatible form as `::c000:201`, and an address family with no converter
raises `EAFNOSUPPORT` instead of a bare `OSError`.

Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX`
escape it needs to carry `_json`'s workload. Against the code it replaces,
`json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and
`json.loads` 1.2x.

`binascii.a2b_base64` now reports the messages `binascii.c` does: no
`error decoding base64: ` prefix, and the data-character count in parentheses.

Assisted-by: Claude Code:claude-opus-5

* test: enable the Windows cp932 codec case

The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure.

Assisted-by: Codex:GPT-5

scripts/port_cjk_mappings.py

@@ -0,0 +1,221 @@
+#!/usr/bin/env python3
+"""Mechanically translate PyPy cjkcodecs mapping headers to Rust arrays.
+
+The codec state machines stay hand-ported line by line.  This script only
+changes the spelling of the generated static data and its page indexes, so the
+Rust port keeps PyPy's ``decode_map`` / ``encode_map`` storage shape.
+"""
+
+from __future__ import annotations
+
+import argparse
+import re
+from pathlib import Path
+
+ARRAY_RE = re.compile(
+    r"static const (?P<type>ucs2_t|ucs4_t|DBCHAR|unsigned char) "
+    r"(?P<name>[A-Za-z0-9_]+)\[(?P<size>\d*)\] = \{(?P<body>.*?)\n\};",
+    re.S,
+)
+INDEX_RE = re.compile(
+    r"static const struct (?P<type>dbcs_index|widedbcs_index|unim_index) "
+    r"(?P<name>[A-Za-z0-9_]+)\[256\] = \{(?P<body>.*?)\n\};",
+    re.S,
+)
+GB18030_RANGE_RE = re.compile(
+    r"static const struct _gb18030_to_unibmp_ranges \{.*?\} "
+    r"gb18030_to_unibmp_ranges\[\] = \{(?P<body>.*?)\};",
+    re.S,
+)
+PAIR_ENCODE_RE = re.compile(
+    r"static const struct pair_encodemap (?P<name>[A-Za-z0-9_]+)"
+    r"\[(?P<size>[A-Za-z0-9_]+)\] = \{(?P<body>.*?)\n\};",
+    re.S,
+)
+
+RUST_TYPE = {
+    "ucs2_t": "u16",
+    "ucs4_t": "u32",
+    "DBCHAR": "u16",
+    "unsigned char": "u8",
+}
+SENTINEL = {"U": "UNIINV", "N": "NOCHAR", "M": "MULTIC", "D": "DBCINV"}
+
+
+def rust_name(name: str) -> str:
+    return name.lstrip("_").upper()
+
+
+def wrap(items: list[str], width: int = 100) -> list[str]:
+    lines: list[str] = []
+    line = "    "
+    for item in items:
+        token = item + ","
+        if len(line) > 4 and len(line) + len(token) + 1 > width:
+            lines.append(line.rstrip())
+            line = "    "
+        line += token + " "
+    if len(line) > 4:
+        lines.append(line.rstrip())
+    return lines
+
+
+def translate(source: Path) -> str:
+    text = source.read_text()
+    output = [
+        "// Generated mechanically from PyPy's cjkcodecs mapping header by",
+        "// `scripts/port_cjk_mappings.py`.  Do not change the page layout:",
+        "// `cjkcodecs.h::_TRYMAP_DEC` / `_TRYMAP_ENC` index these exact arrays.",
+        "",
+        "pub(super) const UNIINV: u16 = 0xfffe;",
+        "#[allow(dead_code)]",
+        "pub(super) const NOCHAR: u16 = 0xffff;",
+        "#[allow(dead_code)]",
+        "pub(super) const MULTIC: u16 = 0xfffe;",
+        "#[allow(dead_code)]",
+        "pub(super) const DBCINV: u16 = 0xfffd;",
+        "",
+        "#[derive(Clone, Copy)]",
+        "pub(super) struct MapIndex {",
+        "    pub(super) offset: usize,",
+        "    pub(super) bottom: u8,",
+        "    pub(super) top: u8,",
+        "    pub(super) present: bool,",
+        "}",
+        "",
+        "const EMPTY_INDEX: MapIndex = MapIndex {",
+        "    offset: 0,",
+        "    bottom: 0,",
+        "    top: 0,",
+        "    present: false,",
+        "};",
+        "",
+    ]
+
+    arrays: dict[str, int] = {}
+    for match in ARRAY_RE.finditer(text):
+        c_name = match.group("name")
+        name = rust_name(c_name) + "_DATA"
+        values = [
+            part.strip() for part in match.group("body").replace("\n", "").split(",")
+        ]
+        rust_type = RUST_TYPE[match.group("type")]
+        values = [
+            (
+                f"{SENTINEL[value]} as {rust_type}"
+                if value in SENTINEL and rust_type != "u16"
+                else SENTINEL.get(value, value)
+            )
+            for value in values
+            if value
+        ]
+        size = int(match.group("size")) if match.group("size") else len(values)
+        if len(values) != size:
+            raise ValueError(f"{c_name}: declared {size}, parsed {len(values)}")
+        arrays[c_name] = size
+        output.append(f"pub(super) static {name}: [{rust_type}; {size}] = [")
+        output.extend(wrap(values))
+        output.extend(["];", ""])
+
+    for match in INDEX_RE.finditer(text):
+        c_name = match.group("name")
+        name = c_name.upper()
+        entries = re.findall(r"\{([^{}]*)\}", match.group("body"))
+        if len(entries) != 256:
+            raise ValueError(f"{c_name}: expected 256 indexes, parsed {len(entries)}")
+        translated: list[str] = []
+        for entry in entries:
+            compact = re.sub(r"\s+", "", entry)
+            pointer, bottom, top = compact.split(",")
+            if pointer == "0":
+                translated.append("EMPTY_INDEX")
+                continue
+            array_name, offset = pointer.split("+")
+            if array_name not in arrays:
+                raise ValueError(f"{c_name}: unknown array {array_name}")
+            translated.append(
+                "MapIndex { "
+                f"offset: {offset}, bottom: {bottom}, top: {top}, present: true "
+                "}"
+            )
+        output.append(f"pub(super) static {name}: [MapIndex; 256] = [")
+        output.extend(wrap(translated, 120))
+        output.extend(["];", ""])
+
+    if match := GB18030_RANGE_RE.search(text):
+        entries = re.findall(
+            r"\{\s*(\d+)\s*,\s*(\d+)\s*,\s*(\d+)\s*\}", match.group("body")
+        )
+        output.extend(
+            [
+                "#[derive(Clone, Copy)]",
+                "pub(super) struct Gb18030Range {",
+                "    pub(super) first: u32,",
+                "    pub(super) last: u32,",
+                "    pub(super) base: u32,",
+                "}",
+                "",
+                f"pub(super) static GB18030_TO_UNIBMP_RANGES: [Gb18030Range; {len(entries)}] = [",
+            ]
+        )
+        output.extend(
+            wrap(
+                [
+                    f"Gb18030Range {{ first: {first}, last: {last}, base: {base} }}"
+                    for first, last, base in entries
+                ],
+                120,
+            )
+        )
+        output.extend(["];", ""])
+
+    if match := PAIR_ENCODE_RE.search(text):
+        entries = re.findall(
+            r"\{\s*(0x[0-9a-fA-F]+)\s*,\s*(0x[0-9a-fA-F]+)\s*\}", match.group("body")
+        )
+        size_name = match.group("size")
+        size_match = re.search(rf"#define\s+{re.escape(size_name)}\s+(\d+)", text)
+        if size_match is None:
+            raise ValueError(f"{match.group('name')}: unknown size {size_name}")
+        size = int(size_match.group(1))
+        if len(entries) != size:
+            raise ValueError(
+                f"{match.group('name')}: declared {size}, parsed {len(entries)}"
+            )
+        output.extend(
+            [
+                "#[derive(Clone, Copy)]",
+                "pub(super) struct PairEncodeMap {",
+                "    pub(super) uniseq: u32,",
+                "    pub(super) code: u16,",
+                "}",
+                "",
+                f"pub(super) static {match.group('name').upper()}: [PairEncodeMap; {size}] = [",
+            ]
+        )
+        output.extend(
+            wrap(
+                [
+                    f"PairEncodeMap {{ uniseq: {uniseq}, code: {code} }}"
+                    for uniseq, code in entries
+                ],
+                120,
+            )
+        )
+        output.extend(["];", ""])
+
+    if not arrays:
+        raise ValueError(f"{source}: no mapping arrays found")
+    return "\n".join(output)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("source", type=Path)
+    parser.add_argument("output", type=Path)
+    args = parser.parse_args()
+    args.output.write_text(translate(args.source))
+
+
+if __name__ == "__main__":
+    main()