18 Commits
Author SHA1 Message Date
Jeong, YunWonandyouknowdot f0b1e82652 Add tracing profiler and Python 3.15 offline profiling tools (#9271)
* Add deterministic profiler and canonical Python 3.15 tools

Use the merged operation critical sections with interpreter-bound profiler
entry checks and short payload borrows. Trace owned code references for GC.
Import the canonical tracing and offline profiling output closure via the
repository updater, retaining existing test markers.

Assisted-by: dot:GPT-6.1

* Clarify standalone profiling license cross-reference

Preserve license terms and copyright notices; point to the reproduced
Apache text below instead of CPython's original OpenSSL section above.

Assisted-by: dot:GPT-6.1

* Defer unsupported sampling package from profiler extraction

Keep the deterministic tracing profiler and its canonical tools together.
Defer the entirely new sampling production, tests, assets and their license
as one closure until native record and binary-collector support is ready.
Retain canonical pstats for deterministic empty-profile handling; its
upstream sampled-file sorting issue is not patched here.

Assisted-by: dot:GPT-6.1

* Stage profiling root as a namespace until sampling is available

Omit the whole newly added canonical package initializer rather than
advertise an absent sampling module or hand-edit canonical __all__.
Explicit profiling.tracing and cProfile imports remain available.
The namespace root intentionally lacks canonical metadata/automatic exports.

This addresses test___all__ failures in the prior Linux, macOS and Windows
CI runs. No test assertions, skips or canonical file contents are changed.

Assisted-by: dot:GPT-6.1

* Match canonical static builtin profiler labels and metadata

Normalize the raw builtin receiver instead of its call-argument accessor.
Snapshot static method module metadata at construction, matching CPython,
so explicit module mutation and deletion remain observable. Avoid a
profiler-time fallback that would obscure deleted metadata.

Correct float.__getformat__ to CPython's classmethod calling convention,
including subclass binding, rather than special-casing its profiler label.

Assisted-by: dot:GPT-6.1

* Support collision-safe frozen package containers

Generate empty frozen packages only for source namespace directories with
real frozen descendants. Preserve genuine modules and packages across
frozen-library registrations while retaining real-real replacement order.

Record the container marker in FrozenModule and its library envelope;
legacy module/package encodings remain unchanged. This is closed-tree
frozen packaging, not full PEP 420 namespace merging.

Add structural codec, mock generator, and registration-priority checks.

* Revert frozen package containers before restoring canonical profiling root

The container implementation changed filesystem namespace precedence.
Restore the existing importer and frozen envelope unchanged; the following
offline sampling closure restores the canonical regular package root.

This reverses d5a8049234.

Assisted-by: dot:GPT-6.1

* Restore canonical profiling package with offline sampling codec

Provide native records, status constants and genuine uncompressed binary
reader/writer APIs for canonical offline tools. Keep the unsupported live
process capability absent and preserve existing canonical test gates.
Restore the complete selected sampling production/test/assets closure via
update_lib, including its asyncio frame-schema dependency. The prior frozen
container extension is reverted separately; canonical packages work with
both filesystem and frozen imports without changing namespace semantics.

Assisted-by: dot:GPT-6.1

* Keep platform-dependent C long normalization explicit

Allow only the c_long-to-i32 conversion lint: Windows uses the same type,
while LP64 requires the existing truncation to match CPython normalization.
No runtime behavior or test assertions change.

Assisted-by: dot:GPT-6.1

* Include profiling notices in library and desktop packages

Bundle a byte-identical copy of the root profiling notice in rustpython-pylib
and include the root notice in configured desktop resources. Avoid a symlink
so Windows checkouts preserve the actual notice text. Terms are unchanged.

Assisted-by: dot:GPT-6.1

* Keep profiling asset license notice in pylib crate

* Serve frozen profiling assets through native resource readers

* Preserve canonical profiling asset line endings on checkout

---------

Co-authored-by: youknowdot <dot@youknowone.org>
2026-10-22 11:18:03 +09:00
Jeong, YunWon 322f294a21 common: add VM-independent codec and text engines (#8622)
* common: add PyPy CJK codec engines

Assisted-by: Codex:GPT-5

* common: mark CJK mapping tables generated

Assisted-by: Codex:GPT-5

* common: add reusable byte and text engines

Assisted-by: Codex:GPT-5

* common: record opt-in engine dependencies

Assisted-by: Codex:GPT-5

* stdlib: add `_multibytecodec` and the six `_codecs_*` modules

Wire the CJK codec engines in `rustpython-common` up to the VM. `_multibytecodec`
holds `MultibyteCodec`, `MultibyteIncrementalEncoder`, `MultibyteIncrementalDecoder`,
`MultibyteStreamReader` and `MultibyteStreamWriter`, the encode/decode drivers and
the error-handler protocol; each `_codecs_*` module exposes `getcodec` for the
encodings of its region.

`rustpython-stdlib` now enables the `cjk-codecs` feature of `rustpython-common`.

Add `cjk::decode_reset`, which the decoder `reset()` methods need. It differs from
building a fresh decoder state: ISO 2022 keeps its G1..G3 designations and the
escape-throughout flag, and resets only G0 and the shifted flag.

Drop the markers from the CJK codec tests and from the `test_codecs`, `test_io`,
`test_email`, `test_zipfile` and `test_traceback` cases that needed a CJK codec.
`test_codecs.BasicUnicodeTest.{test_basics,test_decoder_state}` stay expected
failures for an unrelated reason: `codecs.charmap_encode` rejects a `None` mapping.

Assisted-by: Claude Code:claude-opus-5

* stdlib: use the byte and text engines from `rustpython-common`

`binascii`, `_json` and `_socket`'s address converters now call the engines in
`rustpython-common` instead of carrying their own copies, and `rustpython-stdlib`
enables the `binascii`, `inet` and `json` features for them. That leaves
`base64` and `crc32fast` with no caller in `rustpython-stdlib`, and
`crates/stdlib/src/json/machinery.rs`, the last of the json_in_type-derived code,
goes away with its last caller.

The address converters change behaviour. `inet_aton` reads the lenient forms
(`127.1`, `16909060`, `0x7f.1`, `0177.0.0.1`), `inet_pton` rejects an octet with a
redundant leading zero and a group of five hex digits, `inet_ntop` writes the
IPv4-compatible form as `::c000:201`, and an address family with no converter
raises `EAFNOSUPPORT` instead of a bare `OSError`.

Give the JSON engine the closing-quote scan and the allocation-free `\uXXXX`
escape it needs to carry `_json`'s workload. Against the code it replaces,
`json.dumps` runs 2.3x faster with `ensure_ascii=True` and 1.7x without it, and
`json.loads` 1.2x.

`binascii.a2b_base64` now reports the messages `binascii.c` does: no
`error decoding base64: ` prefix, and the data-character count in parentheses.

Assisted-by: Claude Code:claude-opus-5

* test: enable the Windows cp932 codec case

The shared CJK engine now satisfies the Windows code-page adapter, so keeping this test marked as an expected failure turns its success into a CI failure.

Assisted-by: Codex:GPT-5
2026-08-31 15:44:55 +09:00
Shahar Naveh c3e1f0fa37 Get all builtin types dynamiclly (#8095) 2026-06-15 11:11:32 +09:00
Shahar Naveh 26dfeae512 Autogen opcodes metadata (#7983) 2026-05-27 16:44:58 +09:00
Jeong, YunWon 3aeecc5751 Add gh-aw agentic workflow for upgrading Python libraries (#7117)
* Add gh-aw agentic workflow for upgrading Python libraries

- upgrade-pylib.md: Claude-based workflow triggered via workflow_dispatch
- Runs `scripts/update_lib quick <name>` to sync a library from CPython
- Creates a PR with the result, referencing issue #6839
2026-02-14 11:26:03 +09:00
Jeong, YunWon d90792258d import .gitattributes from CPython (#6939) 2026-02-02 00:27:21 +09:00
Shahar Naveh db1adaa2c2 Move __doc__ crate to crates/doc (#6234)
* Add `__doc__` crate

* Base auto-generate ci

* Add dummy files

* Update docs

* Set codegen-units to 1 for doc db

* Mark `*.inc.rs` as auto generated

* Disable doctest

* Reset docs
2025-11-14 23:47:51 +09:00
Noa bfe72689fc Remove -merge from Cargo.lock gitattributes 2025-03-20 14:28:03 +09:00
Noa 2721f2de3f Fix a bunch of random tests (#5533) 2025-02-25 08:41:54 +09:00
Jeong YunWon dc569e9921 Refactoring 2023-05-10 19:35:27 +09:00
Jeong YunWon 721d124848 *.snap linguist-generated -merge 2022-08-22 04:43:03 +09:00
Jeong YunWon fb5ac9e79b reorganize compiler crates 2022-08-22 04:42:42 +09:00
Jeong YunWon c3cab9e8cc remove generated lalrpop result from repository 2022-08-18 08:20:35 +09:00
Jeong YunWon 79f985c256 fix *.{rs,lalrpop} and Lib/*.py to eol=lf 2022-08-18 07:44:50 +09:00
Noa 07ab9737cd Mark generated lalrpop as -merge -diff 2022-04-26 20:39:16 -05:00
Noa ceb59abbd9 Speedup compile time for rustpython-parser 2022-04-24 10:36:02 -05:00
Nikita Sobolev 353ccf1f3f Make this a Rust project once again
Subfolders were not ignored.
Docs: https://github.com/github/linguist/blob/0c7f82f88a2fb482db3a02812cbb4b8aab7308d1/docs/overrides.md#vendored-code
2021-08-30 16:01:37 +03:00
Noah 25013169f7 Fix Github language statistics to make this a Rust project 2019-09-16 18:01:24 +00:00