Merge remote-tracking branch 'origin/main' into copilot/fix-isalnum-function-behavior

# Conflicts:
#	crates/sre_engine/src/string.rs
#	extra_tests/snippets/builtin_str_unicode.py

.agents/skills/apple-container/SKILL.md

@@ -0,0 +1,63 @@
+---
+name: apple-container
+description: Run RustPython tests in Apple's `container` CLI Linux environment. Use when user asks to run or compare Linux test results on macOS.
+allowed-tools: Bash(container:*) Bash(cargo:*) Read Grep Glob
+---
+
+# Run Tests in Linux Container (Apple `container` CLI)
+
+Run RustPython tests inside a Linux container using Apple's `container` CLI.
+NEVER use Docker, Podman, or any other container runtime. Only use the `container` command.
+
+## Arguments
+
+- Test command to run (examples: `test_io`, `test_codecs -v`, `test_io -v -m "test_errors"`)
+
+## Prerequisites
+
+- The `container` CLI is installed via `brew install container`.
+- The dev image `rustpython-dev` is already built.
+
+## Workflow
+
+1. Check whether the test container is already running:
+
+   ```shell
+   container list 2>/dev/null | grep rustpython-test
+   ```
+
+2. Start the container if it is not running:
+
+   ```shell
+   container run -d --name rustpython-test -m 8G -c 4 \
+       --mount type=bind,source="$(pwd)",target=/workspace \
+       -w /workspace rustpython-dev sleep infinity
+   ```
+
+3. Run the requested test command inside the container:
+
+   ```shell
+   container exec rustpython-test cargo run --release -- -m test <test-args>
+   ```
+
+4. Report results:
+
+- Show pass/fail summary and expected failures / unexpected successes.
+- Highlight new failures compared to macOS results, if available.
+- Do not stop or remove the container after testing.
+
+## Notes
+
+- The workspace is bind-mounted, so local code changes are immediately available in the container.
+- Use `container exec rustpython-test sh -c "..."` for any command inside the container.
+- Rebuild after code changes with:
+
+  ```shell
+  container exec rustpython-test sh -c "cargo build --release"
+  ```
+
+- Stop the container when explicitly requested:
+
+  ```shell
+  container rm -f rustpython-test
+  ```

.agents/skills/investigate-test-failure/SKILL.md

@@ -0,0 +1,53 @@
+---
+name: investigate-test-failure
+description: Investigate a failing RustPython test, compare with CPython behavior, and decide whether to implement a fix or prepare incompatibility issue details.
+allowed-tools: Bash(python3:*) Bash(cargo run:*) Bash(gh issue create:*) Read Grep Glob Bash(git add:*) Bash(git commit:*) Bash(cargo fmt:*) Bash(git diff:*) Task
+---
+
+# Investigate Test Failure
+
+Investigate why a specific test is failing and determine whether it can be fixed now or should be reported as an incompatibility issue.
+
+## Arguments
+
+- Failed test identifier (example: `test_inspect.TestGetSourceBase.test_getsource_reload`)
+
+## Workflow
+
+1. Analyze the failure cause:
+
+- Read the test code.
+- Analyze failure message and traceback.
+- Check related RustPython implementation code.
+
+2. Verify behavior in CPython:
+
+- Run the test with `python3 -m unittest`.
+- Document expected behavior/output.
+
+3. Determine fix feasibility:
+
+- Simple fix (import issues, small logic bugs): implement fix, run formatting and checks, review changes, and commit.
+- Complex fix (major missing feature): gather issue information and report to user.
+
+Pre-commit review process:
+
+- Run `git diff` to review local changes.
+- Use a focused subagent review to compare implementation against CPython behavior and check for missed edge cases.
+- Commit only after review passes.
+
+4. For complex issues, collect issue information using `.github/ISSUE_TEMPLATE/report-incompatibility.md`:
+
+- Feature: missing or broken Python feature.
+- Minimal reproduction code.
+- CPython behavior (`python3`).
+- RustPython behavior (`cargo run`).
+- Python documentation reference link.
+
+Report the collected information to the user. Create an issue only when explicitly requested.
+
+Example issue command:
+
+```shell
+gh issue create --template report-incompatibility.md --title "..." --body "..."
+```

.agents/skills/rustpython-capi-expansion/SKILL.md

@@ -0,0 +1,60 @@
+---
+name: rustpython-capi-expansion
+description: Implement missing RustPython C-API functions in crates/capi using the pyo3-ffi header split mapping (`pyo3-ffi/src/*.rs`, mirroring CPython C API headers). Use this whenever the user asks to add or port C-API functions (for example from setobject.h, dictobject.h, unicodeobject.h) or add capi tests.
+---
+
+# RustPython C-API Expansion
+
+Use this workflow for adding missing C-API functions to RustPython.
+
+## Source of truth for target files
+
+- Use this mapping source: `pyo3-ffi/src/*.rs`, which mirrors the CPython header split used by the C API.
+- Map requested header APIs to `crates/capi/src/<header_basename>.rs` using that split. Examples:
+  - `setobject.h` -> `crates/capi/src/setobject.rs`
+  - `dictobject.h` -> `crates/capi/src/dictobject.rs`
+  - `unicodeobject.h` -> `crates/capi/src/unicodeobject.rs`
+- Do not invent alternate target modules when the header split implies a direct target.
+- If the target file is not present yet, create it and wire it in `crates/capi/src/lib.rs`.
+
+## Implementation workflow
+
+1. Identify requested missing APIs from the user request and their originating C API header.
+2. Open nearby capi modules in `crates/capi/src/` and follow existing style and patterns.
+3. Implement only the requested functions in the mapped target file.
+4. Keep behavior aligned with CPython C-API contracts.
+5. Prefer using existing `rustpython-vm` functionality as much as possible instead of re-implementing behavior in capi.
+6. If a needed `rustpython-vm` helper exists but is private, make it public with a minimal, focused visibility change.
+7. Prefer direct contract assumptions over defensive null checks unless required by the established local style.
+8. Add basic tests only; do not overfit with very specific edge-case clutter.
+9. In tests, use `pyo3` only as a safe wrapper over the API. Avoid raw pointer-heavy direct FFI-style tests.
+10. Run tests from `crates/capi`.
+
+## Testing rules
+
+- Run test commands with working directory set to `crates/capi`.
+- Prefer targeted tests first (module/function filter), then broader capi tests if needed.
+- Keep test names concise (no required `test_` prefix).
+
+## Style rules
+
+- Follow existing RustPython capi coding style in neighboring files.
+- Reuse `rustpython-vm` methods and types first; avoid duplicating VM logic in capi wrappers.
+- When exposing previously private VM helpers, keep the API surface minimal and avoid unrelated refactors.
+- Only expose and implement ABI-stable C-API surface needed for `abi3` / `abi3t`.
+- Add comments only when they explain non-obvious behavior.
+- Keep edits minimal and focused on requested API expansion.
+
+## Completion checklist
+
+- [ ] All requested functions implemented in mapped target file.
+- [ ] New module exported in `crates/capi/src/lib.rs` when applicable.
+- [ ] Basic safe-wrapper `pyo3` tests added/updated.
+- [ ] Tests executed from `crates/capi` and passing for changed area.
+- [ ] Final response includes changed file paths and test command summary.
+
+## Example prompts this skill should handle
+
+- "Implement these missing functions from `dictobject.h`."
+- "Add `setobject.h` C-API functions in RustPython and include basic tests."
+- "Port the listed `unicodeobject.h` APIs in capi, follow existing style, and run tests from `crates/capi`."

.agents/skills/upgrade-pylib-next/SKILL.md

@@ -0,0 +1,34 @@
+---
+name: upgrade-pylib-next
+description: Pick the next CPython stdlib module ready for upgrade based on update_lib todo output and skip modules with open upgrade PRs.
+allowed-tools: Skill(upgrade-pylib) Bash(gh pr list:*) Bash(cargo run:*)
+---
+
+# Upgrade Next Python Library
+
+Find the next Python library module ready for upgrade and then run the `upgrade-pylib` workflow for it.
+
+## Workflow
+
+1. Get current TODO status:
+
+```shell
+cargo run --release -- scripts/update_lib todo 2>/dev/null
+```
+
+2. Get open upgrade PRs:
+
+```shell
+gh pr list --search "Update in:title" --json number,title --template '{{range .}}#{{.number}} {{.title}}{{"\\n"}}{{end}}'
+```
+
+3. From TODO output, select modules in this priority order:
+
+- `[ ] [no deps]`
+- `[ ] [0/n]` where dependencies are all upgraded (for example `[0/3]`, `[0/5]`)
+
+4. Skip modules that already have an open upgrade PR.
+
+5. Run upgrade for selected module using the `upgrade-pylib` workflow.
+
+If no modules match, report that eligible modules are blocked by dependencies.

.agents/skills/upgrade-pylib/SKILL.md

@@ -0,0 +1,102 @@
+---
+name: upgrade-pylib
+description: Upgrade a Python standard library module from CPython into RustPython using scripts/update_lib and then triage and mark remaining failures.
+allowed-tools: Bash(git add:*) Bash(git commit:*) Bash(git diff:*) Bash(cargo run:*) Bash(python3 scripts/update_lib quick:*) Bash(python3 scripts/update_lib auto-mark:*)
+---
+
+# Upgrade Python Library from CPython
+
+Upgrade a Python standard library module from CPython to RustPython.
+
+## Arguments
+
+- Library name or path (examples: `inspect`, `asyncio`, `json`, `cpython/Lib/inspect.py`, `cpython/Lib/json/`)
+
+## Important: Report Tool Issues First
+
+If `scripts/update_lib` shows a bug, missing automation, or unexpected behavior, stop the upgrade and report the tooling issue first, including:
+
+1. What you were trying to do:
+
+- Library name.
+- Full command used.
+
+2. What went wrong or what is missing.
+3. Expected vs actual behavior.
+
+## Workflow
+
+1. Run quick upgrade with update_lib:
+
+- `python3 scripts/update_lib quick <module>`
+- or `python3 scripts/update_lib quick cpython/Lib/<module>.py`
+- or `python3 scripts/update_lib quick cpython/Lib/<module>/`
+
+This step copies library files, patches tests while preserving markers where possible, runs tests, auto-marks new failures, removes stale `@unittest.expectedFailure`, and creates a commit.
+
+If warnings like `WARNING: TestCFoo does not exist in remote file` appear, class structure changed and some markers must be restored manually.
+
+2. Review diff and restore RustPython-specific changes:
+
+- Run `git diff Lib/test/test_<module>`.
+- Restore only changes with explicit `RUSTPYTHON` comments.
+- Do not restore unrelated upstream CPython changes.
+
+3. Investigate failing dependent tests:
+
+- Get dependencies:
+
+  ```shell
+  cargo run --release -- scripts/update_lib deps <module>
+  ```
+
+- Find direct dependent test modules from the `- [ ] <module>:` line.
+- Run dependent tests and collect failures:
+
+  ```shell
+  cargo run --release -- -m test <test_modules...> 2>&1 | grep -E "^(FAIL|ERROR):"
+  ```
+
+- For each failing test identifier, investigate with the `investigate-test-failure` workflow.
+
+4. Mark remaining failures with auto-mark:
+
+- `python3 scripts/update_lib auto-mark Lib/test/test_<module>.py --mark-failure`
+- or `python3 scripts/update_lib auto-mark Lib/test/test_<module>/ --mark-failure`
+
+Note: `--mark-failure` marks all current failures including regressions; review carefully.
+
+5. Handle panics manually:
+
+- For tests that panic/crash/hang, use `@unittest.skip("TODO: RUSTPYTHON; ...")` rather than expectedFailure.
+
+6. Handle class-specific failures:
+
+- If only one subclass fails (for example `TestCFoo` but not `TestPyFoo`), move marker to the failing subclass override.
+
+7. Commit test-fix markers as a separate commit:
+
+```shell
+git add -u && git commit -m "Mark failing tests"
+```
+
+## Example usage
+
+```text
+upgrade-pylib inspect
+upgrade-pylib json
+upgrade-pylib asyncio
+upgrade-pylib cpython/Lib/inspect.py
+upgrade-pylib cpython/Lib/json/
+```
+
+## Notes
+
+- The `cpython/` directory should contain the CPython source used for sync.
+- `scripts/update_lib` modes:
+  - `quick`: patch + auto-mark
+  - `migrate`: patch only
+  - `auto-mark`: mark failures only
+  - `copy-lib`: copy library files only
+- Patching transfers `@unittest.expectedFailure` and `@unittest.skip` with `TODO: RUSTPYTHON` markers where possible.
+- The tool does not preserve every RustPython-specific change; always review and restore explicit RUSTPYTHON-marked logic.

.cargo/config.toml

@@ -6,3 +6,16 @@ rustflags = "-C link-args=-Wl,--stack,8000000"
 
 [target.wasm32-unknown-unknown]
 rustflags = ["--cfg=getrandom_backend=\"wasm_js\""]
+
+# Enforce a 1 MB stack limit for WASI targets.
+# (Runaway heap growth is fixed by the smaller DataStack chunk
+# and the size‑optimized wasm‑release profile.)
+[target.wasm32-wasip1]
+rustflags = [
+    "-C", "link-arg=-zstack-size=1048576",
+]
+
+[target.wasm32-wasip2]
+rustflags = [
+    "-C", "link-arg=-zstack-size=1048576",
+]

.claude/commands/apple-container.md

@@ -1,46 +0,0 @@
----
-allowed-tools: Bash(container *), Bash(cargo *), Read, Grep, Glob
----
-
-# Run Tests in Linux Container (Apple `container` CLI)
-
-Run RustPython tests inside a Linux container using Apple's `container` CLI.
-**NEVER use Docker, Podman, or any other container runtime.** Only use the `container` command.
-
-## Arguments
-- `$ARGUMENTS`: Test command to run (e.g., `test_io`, `test_codecs -v`, `test_io -v -m "test_errors"`)
-
-## Prerequisites
-
-The `container` CLI is installed via `brew install container`.
-The dev image `rustpython-dev` is already built.
-
-## Steps
-
-1. **Check if the container is already running**
-   ```shell
-   container list 2>/dev/null | grep rustpython-test
-   ```
-
-2. **Start the container if not running**
-   ```shell
-   container run -d --name rustpython-test -m 8G -c 4 \
-       --mount type=bind,source=.,target=/workspace \
-       -w /workspace rustpython-dev sleep infinity
-   ```
-
-3. **Run the test inside the container**
-   ```shell
-   container exec rustpython-test sh -c "cargo run --release -- -m test $ARGUMENTS"
-   ```
-
-4. **Report results**
-   - Show test summary (pass/fail counts, expected failures, unexpected successes)
-   - Highlight any new failures compared to macOS results if available
-   - Do NOT stop or remove the container after testing (keep it for reuse)
-
-## Notes
-- The workspace is bind-mounted, so local code changes are immediately available
-- Use `container exec rustpython-test sh -c "..."` for any command inside the container
-- To rebuild after code changes, run: `container exec rustpython-test sh -c "cargo build --release"`
-- To stop the container when done: `container rm -f rustpython-test`

.claude/commands/investigate-test-failure.md

@@ -1,49 +0,0 @@
----
-allowed-tools: Bash(python3:*), Bash(cargo run:*), Read, Grep, Glob, Bash(git add:*), Bash(git commit:*), Bash(cargo fmt:*), Bash(git diff:*), Task
----
-
-# Investigate Test Failure
-
-Investigate why a specific test is failing and determine if it can be fixed or needs an issue.
-
-## Arguments
-- `$ARGUMENTS`: Failed test identifier (e.g., `test_inspect.TestGetSourceBase.test_getsource_reload`)
-
-## Steps
-
-1. **Analyze failure cause**
-   - Read the test code
-   - Analyze failure message/traceback
-   - Check related RustPython code
-
-2. **Verify behavior in CPython**
-   - Run the test with `python3 -m unittest` to confirm expected behavior
-   - Document the expected output
-
-3. **Determine fix feasibility**
-   - **Simple fix** (import issues, small logic bugs): Fix code → Run `cargo fmt --all` → Pre-commit review → Commit
-   - **Complex fix** (major unimplemented features): Collect issue info and report to user
-
-   **Pre-commit review process**:
-   - Run `git diff` to see the changes
-   - Use Task tool with `general-purpose` subagent to review:
-     - Compare implementation against cpython/ source code
-     - Verify the fix aligns with CPython behavior
-     - Check for any missed edge cases
-   - Proceed to commit only after review passes
-
-4. **For complex issues - Collect issue information**
-   Following `.github/ISSUE_TEMPLATE/report-incompatibility.md` format:
-
-   - **Feature**: Description of missing/broken Python feature
-   - **Minimal reproduction code**: Smallest code that reproduces the issue
-   - **CPython behavior**: Result when running with python3
-   - **RustPython behavior**: Result when running with cargo run
-   - **Python Documentation link**: Link to relevant CPython docs
-
-   Report collected information to the user. Issue creation is done only upon user request.
-
-   Example issue creation command:
-   ```
-   gh issue create --template report-incompatibility.md --title "..." --body "..."
-   ```

.claude/commands/upgrade-pylib-next.md

@@ -1,33 +0,0 @@
----
-allowed-tools: Skill(upgrade-pylib), Bash(gh pr list:*)
----
-
-# Upgrade Next Python Library
-
-Find the next Python library module ready for upgrade and run `/upgrade-pylib` for it.
-
-## Current TODO Status
-
-!`cargo run --release -- scripts/update_lib todo 2>/dev/null`
-
-## Open Upgrade PRs
-
-!`gh pr list --search "Update in:title" --json number,title --template '{{range .}}#{{.number}} {{.title}}{{"\n"}}{{end}}'`
-
-## Instructions
-
-From the TODO list above, find modules matching these patterns (in priority order):
-
-1. `[ ] [no deps]` - Modules with no dependencies (can be upgraded immediately)
-2. `[ ] [0/n]` - Modules where all dependencies are already upgraded (e.g., `[0/3]`, `[0/5]`)
-
-These patterns indicate modules that are ready to upgrade without blocking dependencies.
-
-**Important**: Skip any modules that already have an open PR in the "Open Upgrade PRs" list above.
-
-**After identifying a suitable module**, run:
-```
-/upgrade-pylib <module_name>
-```
-
-If no modules match these criteria, inform the user that all eligible modules have dependencies that need to be upgraded first.

.claude/commands/upgrade-pylib.md

@@ -1,157 +0,0 @@
----
-allowed-tools: Bash(git add:*), Bash(git commit:*), Bash(python3 scripts/update_lib quick:*), Bash(python3 scripts/update_lib auto-mark:*)
----
-
-# Upgrade Python Library from CPython
-
-Upgrade a Python standard library module from CPython to RustPython.
-
-## Arguments
-- `$ARGUMENTS`: Library name to upgrade (e.g., `inspect`, `asyncio`, `json`)
-
-## Important: Report Tool Issues First
-
-If during the upgrade process you encounter any of the following issues with `scripts/update_lib`:
-- A feature that should be automated but isn't supported
-- A bug or unexpected behavior in the tool
-- Missing functionality that would make the upgrade easier
-
-**STOP the upgrade and report the issue first.** Describe:
-1. What you were trying to do
-    - Library name
-    - The full command executed (e.g. python scripts/update_lib quick cpython/Lib/$ARGUMENTS.py)
-2. What went wrong or what's missing
-3. Expected vs actual behavior
-
-This helps improve the tooling for future upgrades.
-
-## Steps
-
-1. **Run quick upgrade with update_lib**
-   - Run: `python3 scripts/update_lib quick $ARGUMENTS` (module name)
-   - Or: `python3 scripts/update_lib quick cpython/Lib/$ARGUMENTS.py` (library file path)
-   - Or: `python3 scripts/update_lib quick cpython/Lib/$ARGUMENTS/` (library directory path)
-   - This will:
-     - Copy library files (delete existing `Lib/$ARGUMENTS.py` or `Lib/$ARGUMENTS/`, then copy from `cpython/Lib/`)
-     - Patch test files preserving existing RustPython markers
-     - Run tests and auto-mark new test failures (not regressions)
-     - Remove `@unittest.expectedFailure` from tests that now pass
-     - Create a git commit with the changes
-   - **Handle warnings**: If you see warnings like `WARNING: TestCFoo does not exist in remote file`, it means the class structure changed and markers couldn't be transferred automatically. These need to be manually restored in step 2 or added in step 3.
-
-2. **Review git diff and restore RUSTPYTHON-specific changes**
-   - Run `git diff Lib/test/test_$ARGUMENTS` to review all changes
-   - **Only restore changes that have explicit `RUSTPYTHON` comments**. Look for:
-     - `# XXX: RUSTPYTHON` or `# XXX RUSTPYTHON` - Comments marking RustPython-specific code modifications
-     - `# TODO: RUSTPYTHON` - Comments marking tests that need work
-     - Code changes with inline `# ... RUSTPYTHON` comments
-   - **Do NOT restore other diff changes** - these are likely upstream CPython changes, not RustPython-specific modifications
-   - When restoring, preserve the original context and formatting
-
-3. **Investigate test failures with subagent**
-   - First, get dependent tests using the deps command:
-     ```
-     cargo run --release -- scripts/update_lib deps $ARGUMENTS
-     ```
-   - Look for the line `- [ ] $ARGUMENTS: test_xxx test_yyy ...` to get the direct dependent tests
-   - Run those tests to collect failures:
-     ```
-     cargo run --release -- -m test test_xxx test_yyy ... 2>&1 | grep -E "^(FAIL|ERROR):"
-     ```
-   - For example, if deps output shows `- [ ] linecache: test_bdb test_inspect test_linecache test_traceback test_zipimport`, run:
-     ```
-     cargo run --release -- -m test test_bdb test_inspect test_linecache test_traceback test_zipimport 2>&1 | grep -E "^(FAIL|ERROR):"
-     ```
-   - For each failure, use the Task tool with `general-purpose` subagent to investigate:
-     - Subagent should follow the `/investigate-test-failure` skill workflow
-     - Pass the failed test identifier as the argument (e.g., `test_inspect.TestGetSourceBase.test_getsource_reload`)
-   - If subagent can fix the issue easily: fix and commit
-   - If complex issue: subagent collects issue info and reports back (issue creation on user request only)
-   - Using subagent prevents context pollution in the main conversation
-
-4. **Mark remaining test failures with auto-mark**
-   - Run: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS.py --mark-failure`
-   - Or for directory: `python3 scripts/update_lib auto-mark Lib/test/test_$ARGUMENTS/ --mark-failure`
-   - This will:
-     - Run tests and mark ALL failing tests with `@unittest.expectedFailure`
-     - Remove `@unittest.expectedFailure` from tests that now pass
-   - **Note**: The `--mark-failure` flag marks all failures including regressions. Review the changes before committing.
-
-5. **Handle panics manually**
-   - If any tests cause panics/crashes (not just assertion failures), they need `@unittest.skip` instead:
-     ```python
-     @unittest.skip("TODO: RUSTPYTHON; panics with 'index out of bounds'")
-     def test_crashes(self):
-         ...
-     ```
-   - auto-mark cannot detect panics automatically - check the test output for crash messages
-
-6. **Handle class-specific failures**
-   - If a test fails only in the C implementation (TestCFoo) but passes in the Python implementation (TestPyFoo), or vice versa, move the marker to the specific subclass:
-     ```python
-     # Base class - no marker here
-     class TestFoo:
-         def test_something(self):
-             ...
-
-     class TestPyFoo(TestFoo, PyTest): pass
-
-     class TestCFoo(TestFoo, CTest):
-         # TODO: RUSTPYTHON
-         @unittest.expectedFailure
-         def test_something(self):
-             return super().test_something()
-     ```
-
-7. **Commit the test fixes**
-   - Run: `git add -u && git commit -m "Mark failing tests"`
-   - This creates a separate commit for the test markers added in steps 2-6
-
-## Example Usage
-```
-# Using module names (recommended)
-/upgrade-pylib inspect
-/upgrade-pylib json
-/upgrade-pylib asyncio
-
-# Using library paths (alternative)
-/upgrade-pylib cpython/Lib/inspect.py
-/upgrade-pylib cpython/Lib/json/
-```
-
-## Example: Restoring RUSTPYTHON changes
-
-When git diff shows removed RUSTPYTHON-specific code like:
-```diff
--# XXX RUSTPYTHON: we don't import _json as fresh since...
--cjson = import_helper.import_fresh_module('json') #, fresh=['_json'])
-+cjson = import_helper.import_fresh_module('json', fresh=['_json'])
-```
-
-You should restore the RustPython version:
-```python
-# XXX RUSTPYTHON: we don't import _json as fresh since...
-cjson = import_helper.import_fresh_module('json') #, fresh=['_json'])
-```
-
-## Notes
-- The cpython/ directory should contain the CPython source that we're syncing from
-- `scripts/update_lib` package handles patching and auto-marking:
-  - `quick` - Combined patch + auto-mark (recommended)
-  - `migrate` - Only migrate (patch), no test running
-  - `auto-mark` - Only run tests and mark failures
-  - `copy-lib` - Copy library files (not tests)
-- The patching:
-  - Transfers `@unittest.expectedFailure` and `@unittest.skip` decorators with `TODO: RUSTPYTHON` markers
-  - Adds `import unittest # XXX: RUSTPYTHON` if needed for the decorators
-  - **Limitation**: If a class was restructured (e.g., method overrides removed), update_lib will warn and skip those markers
-- The smart auto-mark:
-  - Marks NEW test failures automatically (tests that didn't exist before)
-  - Does NOT mark regressions (existing tests that now fail) - these are warnings
-  - Removes `@unittest.expectedFailure` from tests that now pass
-- The script does NOT preserve all RustPython-specific changes - you must review `git diff` and restore them
-- Common RustPython markers to look for:
-  - `# XXX: RUSTPYTHON` or `# XXX RUSTPYTHON` - Inline comments for code modifications
-  - `# TODO: RUSTPYTHON` - Test skip/failure markers
-  - Any code with `RUSTPYTHON` in comments that was removed in the diff
-- **Important**: Not all changes in the git diff need to be restored. Only restore changes that have explicit `RUSTPYTHON` comments. Other changes are upstream CPython updates.

.cspell.dict/cpython.txt

@@ -4,6 +4,7 @@ argdefs
 argtypes
 asdl
 asname
+ASNATIVEBYTES
 atopen
 atext
 attro
@@ -139,6 +140,8 @@ metavars
 miscompiles
 mult
 multibytecodec
+mystricmp
+mystrnicmp
 nameobj
 nameop
 nargsf
@@ -169,6 +172,7 @@ nvars
 opname
 opnames
 orelse
+osmodule
 outparam
 outparm
 paramfunc
@@ -189,11 +193,14 @@ pycore
 pyinner
 pydecimal
 pyerrors
+pyframe
 Pyfunc
 pylifecycle
 pymain
+pymem
 pyrepl
 pystate
+pystrcmp
 PYTHONTRACEMALLOC
 PYTHONUTF8
 pythonw

.cspell.dict/rust-more.txt

@@ -50,6 +50,7 @@ modpow
 msvc
 muldiv
 nanos
+noalias
 nonoverlapping
 objclass
 peekable

.cspell.dict/rustpython.txt

@@ -27,6 +27,7 @@ pystr
 pystruct
 pystructseq
 pytype
+qsbr
 rustix
 struc
 zelf

.cspell.json

@@ -59,6 +59,7 @@
     "alnum",
     "csock",
     "coro",
+    "Crnl",
     "dedentations",
     "dedents",
     "deduped",
@@ -66,8 +67,13 @@
     "deoptimize",
     "emscripten",
     "excs",
+    "fdigits",
+    "flufl",
     "fnfe",
+    "fsdefault",
     "ifexp",
+    "implicits",
+    "inity",
     "interps",
     "jitted",
     "jitting",
@@ -76,8 +82,12 @@
     "mcache",
     "oparg",
     "opargs",
+    "pointee",
     "pyc",
+    "qmark",
     "reborrow",
+    "reborrows",
+    "reparenting",
     "reraises",
     "reraising",
     "significand",

.gitattributes

@@ -1,10 +1,13 @@
 Lib/**      linguist-vendored
 Cargo.lock  linguist-generated
 *.snap      linguist-generated -merge
-vm/src/stdlib/ast/gen.rs    linguist-generated -merge
 Lib/*.py    text working-tree-encoding=UTF-8 eol=LF
 **/*.rs     text working-tree-encoding=UTF-8 eol=LF
-crates/rustpython_doc_db/src/*.inc.rs   linguist-generated=true
+crates/vm/src/stdlib/ast/gen.rs    linguist-generated -merge
+crates/doc/src/data.inc.rs                              generated
+crates/compiler-core/src/bytecode/opcode_metadata.rs    generated
+
+.github/workflows/*.lock.yml linguist-generated=true merge=ours
 
 # Binary data types
 *.aif binary
@@ -45,10 +48,11 @@ Lib/venv/scripts/posix/* text eol=lf
 # Language aware diff headers
 # https://tekin.co.uk/2020/10/better-git-diff-output-for-ruby-python-elixir-and-more
 # https://gist.github.com/tekin/12500956bd56784728e490d8cef9cb81
-*.css   diff=css
-*.html  diff=html
-*.py    diff=python
-*.md    diff=markdown
+*.css       diff=css
+*.html      diff=html
+*.py        diff=python
+*.md        diff=markdown
+*.inc.rs    diff=rust
 
 # Generated files
 # https://github.com/github/linguist/blob/master/docs/overrides.md
@@ -66,6 +70,3 @@ Lib/test/certdata/*.0                                   generated
 Lib/test/levenshtein_examples.json                      generated
 Lib/test/test_stable_abi_ctypes.py                      generated
 Lib/token.py                                            generated
-crates/compiler-core/src/bytecode/opcode_metadata.rs    generated
-
-.github/workflows/*.lock.yml linguist-generated=true merge=ours

.github/actions/install-linux-deps/action.yml

@@ -29,6 +29,10 @@ inputs:
     description: Install gcc-aarch64-linux-gnu (gcc-aarch64-linux-gnu)
     required: false
     default: "false"
+  gcc-mingw-w64-x86-64:
+    description: Install gcc-mingw-w64-x86-64 (gcc-mingw-w64-x86-64)
+    required: false
+    default: "false"
   clang:
     description: Install clang (clang)
     required: false
@@ -39,11 +43,40 @@ runs:
     - name: Install Linux dependencies
       shell: bash
       if: ${{ runner.os == 'Linux' }}
-      run: >
-        sudo apt-get update
+      env:
+        GCC_MULTILIB: ${{ inputs.gcc-multilib }}
+        MUSL_TOOLS: ${{ inputs.musl-tools }}
+        CLANG: ${{ inputs.clang }}
+        GCC_AARCH64_LINUX_GNU: ${{ inputs.gcc-aarch64-linux-gnu }}
+        GCC_MINGW_W64_X86_64: ${{ inputs.gcc-mingw-w64-x86-64 }}
+      run: |
+        if ! sudo apt-get update; then
+          echo "::warning::apt-get update failed; disabling nonessential Microsoft apt sources and retrying"
+          for source in /etc/apt/sources.list.d/*microsoft* /etc/apt/sources.list.d/*azure-cli*; do
+            if [ -e "$source" ]; then
+              sudo mv "$source" "$source.disabled"
+            fi
+          done
+          sudo apt-get update
+        fi
 
-        sudo apt-get install --no-install-recommends
-        ${{ fromJSON(inputs.gcc-multilib) && 'gcc-multilib' || '' }}
-        ${{ fromJSON(inputs.musl-tools) && 'musl-tools' || '' }}
-        ${{ fromJSON(inputs.clang) && 'clang' || '' }}
-        ${{ fromJSON(inputs.gcc-aarch64-linux-gnu) && 'gcc-aarch64-linux-gnu linux-libc-dev-arm64-cross libc6-dev-arm64-cross' || '' }}
+        packages=()
+        if [[ "$GCC_MULTILIB" == "true" ]]; then
+          packages+=(gcc-multilib)
+        fi
+        if [[ "$MUSL_TOOLS" == "true" ]]; then
+          packages+=(musl-tools)
+        fi
+        if [[ "$CLANG" == "true" ]]; then
+          packages+=(clang)
+        fi
+        if [[ "$GCC_AARCH64_LINUX_GNU" == "true" ]]; then
+          packages+=(gcc-aarch64-linux-gnu linux-libc-dev-arm64-cross libc6-dev-arm64-cross)
+        fi
+        if [[ "$GCC_MINGW_W64_X86_64" == "true" ]]; then
+          packages+=(gcc-mingw-w64-x86-64)
+        fi
+
+        if ((${#packages[@]})); then
+          sudo apt-get install --no-install-recommends "${packages[@]}"
+        fi

.github/dependabot.yml

@@ -24,6 +24,7 @@ updates:
           - "sha1"
           - "sha2"
           - "sha3"
+          - "shake"
           - "blake2"
           - "hmac"
           - "pbkdf2"
@@ -88,6 +89,7 @@ updates:
           - "getrandom"
           - "mt19937"
           - "rand*"
+          - "chacha20"
       rayon:
         patterns:
           - "rayon*"

.github/workflows/ci.yaml

@@ -33,8 +33,8 @@ env:
   CARGO_PROFILE_DEV_DEBUG: 0
   CARGO_PROFILE_RELEASE_DEBUG: 0
   CARGO_TERM_COLOR: always
-  FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true # TODO: Remove on 2026/06/02
   CI: true
+  FORCE_COLOR: 1
 
 jobs:
   determine_changes:
@@ -44,7 +44,7 @@ jobs:
       # Flag that is raised when any rust code is changed.
       rust_code: ${{ steps.check_rust_code.outputs.changed }}
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           fetch-depth: 0
           persist-credentials: false
@@ -88,14 +88,14 @@ jobs:
         os: [macos-latest, ubuntu-latest, windows-2025]
       fail-fast: false
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
       - uses: dtolnay/rust-toolchain@stable
 
       - name: Restore cache
-        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: |
             ~/.cargo/bin/
@@ -198,13 +198,17 @@ jobs:
           target: aarch64-unknown-linux-gnu
           dependencies:
             gcc-aarch64-linux-gnu: true
+        - os: ubuntu-latest
+          target: x86_64-pc-windows-gnu
+          dependencies:
+            gcc-mingw-w64-x86-64: true
         - os: macos-latest
           target: aarch64-apple-ios
         - os: macos-latest
           target: x86_64-apple-darwin
       fail-fast: false
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
@@ -216,9 +220,10 @@ jobs:
           gcc-multilib: ${{ matrix.dependencies.gcc-multilib || false }}
           musl-tools: ${{ matrix.dependencies.musl-tools || false }}
           gcc-aarch64-linux-gnu: ${{ matrix.dependencies.gcc-aarch64-linux-gnu || false }}
+          gcc-mingw-w64-x86-64: ${{ matrix.dependencies.gcc-mingw-w64-x86-64 || false }}
 
       - name: Restore cache
-        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: |
             ~/.cargo/bin/
@@ -295,6 +300,8 @@ jobs:
         - os: macos-latest
           extra_test_args:
             - '-u all'
+            - '--timeout 600'
+            - '--dont-add-python-opts'
           env_polluting_tests:
             - test_set
           skips: []
@@ -302,30 +309,31 @@ jobs:
         - os: ubuntu-latest
           extra_test_args:
             - '-u all'
+            - '--timeout 600'
+            - '--dont-add-python-opts'
           env_polluting_tests:
             - test_set
           skips: []
           timeout: 60
         - os: windows-2025
-          extra_test_args: [] # TODO: Enable '-u all'
+          extra_test_args:
+            - '-u all'
+            - '--timeout 600'
+            - '--dont-add-python-opts'
           env_polluting_tests:
             - test_set
-          skips:
-            - test_rlcompleter
-            - test_pathlib # panic by surrogate chars
-            - test_posixpath # OSError: (22, 'The filename, directory name, or volume label syntax is incorrect. (os error 123)')
-            - test_venv # couple of failing tests
+          skips: []
           timeout: 50
       fail-fast: false
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
       - uses: dtolnay/rust-toolchain@stable
 
       - name: Restore cache
-        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: |
             ~/.cargo/bin/
@@ -340,13 +348,14 @@ jobs:
         # Windows runners randomly crashes, https://github.com/actions/cache/issues/1754
         continue-on-error: true
 
-      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
 
       - name: Install macOS dependencies
         uses: ./.github/actions/install-macos-deps
         with:
           openssl: true
 
+      # Keep features in sync with update-caches.yml CARGO_ARGS.
       - name: build rustpython
         run: cargo build --release --verbose --features=threading,jit ${{ env.CARGO_ARGS }}
 
@@ -363,7 +372,7 @@ jobs:
 
       - name: Run CPython tests
         run: |
-          target/release/rustpython -m test -j ${{ steps.cores.outputs.cores }} ${{ join(matrix.extra_test_args, ' ') }} --slowest --fail-env-changed --timeout 600 -v -x ${{ env.FLAKY_MP_TESTS }} ${{ join(matrix.skips, ' ') }}
+          target/release/rustpython -u -m test --slow-ci -j ${{ steps.cores.outputs.cores }} ${{ join(matrix.extra_test_args, ' ') }} -x ${{ env.FLAKY_MP_TESTS }} ${{ join(matrix.skips, ' ') }}
         timeout-minutes: ${{ matrix.timeout }}
         env:
           RUSTPYTHON_SKIP_ENV_POLLUTERS: true
@@ -374,7 +383,7 @@ jobs:
             echo "::group::Attempt ${attempt}"
 
             set +e
-            target/release/rustpython -m test -j 1 ${{ join(matrix.extra_test_args, ' ') }} --slowest --fail-env-changed --timeout 600 -v ${{ env.FLAKY_MP_TESTS }}
+            target/release/rustpython -u -m test --slow-ci -j 1 ${{ join(matrix.extra_test_args, ' ') }} ${{ env.FLAKY_MP_TESTS }}
             status=$?
             set -e
 
@@ -399,7 +408,7 @@ jobs:
           for thing in "${target_array[@]}"; do
             for i in $(seq 1 10); do
               set +e
-              target/release/rustpython -m test -j 1 --slowest --fail-env-changed --timeout 600 -v "${thing}"
+              target/release/rustpython -u -m test --slow-ci -u all -j 1 --timeout 600 --dont-add-python-opts "${thing}"
               exit_code=$?
               set -e
               if [ "${exit_code}" -eq 3 ]; then
@@ -434,6 +443,17 @@ jobs:
           target/release/rustpython -m ensurepip
           target/release/rustpython -c "import pip"
 
+      - if: runner.os == 'Windows'
+        name: Check pip HTTPS with the Windows trust store
+        run: >-
+          target/release/rustpython -m pip download
+          --disable-pip-version-check
+          --no-cache-dir
+          --no-deps
+          --only-binary=:all:
+          --dest "$env:RUNNER_TEMP\rustpython-pip-smoke"
+          six
+
       - if: runner.os != 'Windows'
         name: Check if pip inside venv is functional
         run: |
@@ -462,7 +482,7 @@ jobs:
           - ubuntu-latest
           - windows-latest
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
@@ -471,7 +491,7 @@ jobs:
           components: clippy
 
       - name: Restore cache
-        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: |
             ~/.cargo/bin/
@@ -500,13 +520,13 @@ jobs:
       needs.determine_changes.outputs.rust_code == 'true' ||
       github.ref == 'refs/heads/main'
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
       - uses: dtolnay/rust-toolchain@stable
 
-      - uses: cargo-bins/cargo-binstall@aaa84a43aec4955a42c5ffc65d258961e39f276e # v1.19.1
+      - uses: cargo-bins/cargo-binstall@ead08b90bd7b2e6d81963fb9cf0b7239f66d5db4 # v1.21.0
 
       - name: cargo shear
         run: |
@@ -523,11 +543,11 @@ jobs:
       pull-requests: write
       security-events: write # for zizmor
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
-      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
 
       - uses: dtolnay/rust-toolchain@stable
         with:
@@ -537,23 +557,17 @@ jobs:
         uses: reviewdog/action-actionlint@6fb7acc99f4a1008869fa8a0f09cfca740837d9d # v1.72.0
 
       - name: zizmor
-        uses: zizmorcore/zizmor-action@5f14fd08f7cf1cb1609c1e344975f152c7ee938d # v0.5.6
+        uses: zizmorcore/zizmor-action@192e21d79ab29983730a13d1382995c2307fbcaa # v0.5.7
 
       - name: restore prek cache
-        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           key: prek-${{ hashFiles('.pre-commit-config.yaml') }}
           path: ~/.cache/prek
 
-      # TODO: Remove on 2026/06/02 when node24 is the default
-      - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
-        with:
-          package-manager-cache: false
-          node-version: "24"
-
       - name: install prek
         id: prek
-        uses: j178/prek-action@bdca6f102f98e2b4c7029491a53dfd366469e33d # v2.0.4
+        uses: j178/prek-action@e98a699c41eb69ab013a45817a0406469a748f8d # v2.0.5
         with:
           cache: false
           show-verbose-logs: false
@@ -569,7 +583,7 @@ jobs:
           echo "version=${version}" >> "$GITHUB_OUTPUT"
 
       - name: Clone CPython
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           repository: python/cpython
           path: cpython
@@ -583,7 +597,7 @@ jobs:
 
       - name: save prek cache
         if: ${{ github.ref == 'refs/heads/main' }} # only save on main
-        uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           key: prek-${{ hashFiles('.pre-commit-config.yaml') }}
           path: ~/.cache/prek
@@ -594,7 +608,7 @@ jobs:
 
       - name: reviewdog
         if: ${{ !cancelled() }}
-        uses: reviewdog/action-suggester@aa38384ceb608d00f84b4690cacc83a5aba307ff # v1.24.0
+        uses: reviewdog/action-suggester@2558ba17e65a9039e73764a73009fc05fef28a46 # v1.24.3
         with:
           level: warning
           fail_level: error
@@ -607,7 +621,7 @@ jobs:
     env:
       NIGHTLY_CHANNEL: nightly
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
@@ -617,7 +631,7 @@ jobs:
           components: miri
 
       - name: Restore cache
-        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: |
             ~/.cargo/bin/
@@ -642,7 +656,7 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
@@ -651,7 +665,7 @@ jobs:
           components: clippy
 
       - name: Restore cache
-        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: |
             ~/.cargo/bin/
@@ -676,12 +690,12 @@ jobs:
           mkdir geckodriver
           tar -xzf geckodriver-v0.36.0-linux64.tar.gz -C geckodriver
 
-      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
 
       - run: python -m pip install -r requirements.txt
         working-directory: ./wasm/tests
 
-      - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
           package-manager-cache: false
 
@@ -691,7 +705,7 @@ jobs:
         run: echo "dir=$(npm config get cache)" >> "$GITHUB_OUTPUT"
 
       - name: Restore npm cache
-        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         # don't restore on main or release
         if: github.ref != 'refs/heads/main' && github.ref != 'refs/heads/release'
         with:
@@ -746,7 +760,7 @@ jobs:
       - name: Save npm cache
         # Save only on main or release
         if: github.ref == 'refs/heads/main' || github.ref == 'refs/heads/release'
-        uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: ${{ steps.npm-cache-dir.outputs.dir }}
           key: node-${{ runner.os }}-wasm-demo-${{ hashFiles('wasm/demo/package-lock.json') }}
@@ -757,7 +771,7 @@ jobs:
     runs-on: ubuntu-latest
     timeout-minutes: 30
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
@@ -766,7 +780,7 @@ jobs:
           target: wasm32-wasip1
 
       - name: Restore cache
-        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: |
             ~/.cargo/bin/
@@ -790,11 +804,13 @@ jobs:
           clang: true
 
       - name: build rustpython
-        run: cargo build --release --target wasm32-wasip1 --no-default-features --features freeze-stdlib,stdlib,stdio,importlib,host_env --verbose
+        run: cargo build --profile wasm-release --target wasm32-wasip1 --no-default-features --features freeze-stdlib,stdlib,stdio,importlib,host_env --verbose
       - name: run snippets
-        run: wasmer run --dir "$(pwd)" target/wasm32-wasip1/release/rustpython.wasm -- "$(pwd)/extra_tests/snippets/stdlib_random.py"
+        run: |
+          wasmer run --dir "$(pwd)" target/wasm32-wasip1/wasm-release/rustpython.wasm -- "$(pwd)/extra_tests/snippets/stdlib_random.py"
+          wasmer run --dir "$(pwd)" target/wasm32-wasip1/wasm-release/rustpython.wasm -- "$(pwd)/extra_tests/snippets/stdlib_time.py"
       - name: run cpython unittest
-        run: wasmer run --dir "$(pwd)" target/wasm32-wasip1/release/rustpython.wasm -- "$(pwd)/Lib/test/test_int.py"
+        run: wasmer run --dir "$(pwd)" target/wasm32-wasip1/wasm-release/rustpython.wasm -- "$(pwd)/Lib/test/test_int.py"
 
   cargo_doc:
     needs:
@@ -809,14 +825,14 @@ jobs:
     name: cargo doc
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
       - uses: dtolnay/rust-toolchain@stable
 
       - name: Restore cache
-        uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: |
             ~/.cargo/bin/

.github/workflows/cron-ci.yaml

@@ -15,7 +15,6 @@ on:
 
 env:
   CARGO_ARGS: --no-default-features --features stdlib,importlib,stdio,encodings,ssl-rustls-aws-lc,jit,host_env
-  FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: 'true' # TODO: Remove on 2026/06/02
 
 jobs:
   # codecov collects code coverage data from the rust tests, python snippets and python test suite.
@@ -25,18 +24,20 @@ jobs:
     runs-on: ubuntu-latest
     # Disable this scheduled job when running on a fork.
     if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
+    env:
+      INSTA_WORKSPACE_ROOT: ${{ github.workspace }}
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
       - uses: dtolnay/rust-toolchain@stable
 
-      - uses: taiki-e/install-action@b550161ef8a7bc4f2a671c0b03a18ac9ccedea1e # v2.79.1
+      - uses: taiki-e/install-action@a6b2e2dcd845ddd7f509ce4f3ed3d922b80cc5d9 # v2.84.0
         with:
           tool: cargo-llvm-cov
 
-      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
 
       - run: sudo apt-get update && sudo apt-get -y install lcov
 
@@ -56,7 +57,7 @@ jobs:
 
       - name: Upload to Codecov
         if: ${{ github.event_name != 'pull_request' }}
-        uses: codecov/codecov-action@57e3a136b779b570ffcdbf80b3bdc90e7fab3de2 # v6.0.0
+        uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
         with:
           files: ./codecov.lcov
 
@@ -66,7 +67,7 @@ jobs:
     # Disable this scheduled job when running on a fork.
     if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: true
 
@@ -84,7 +85,6 @@ jobs:
         if: ${{ github.event_name != 'pull_request' }}
         env:
           SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }}
-          GITHUB_ACTOR: ${{ github.actor }}
         run: |
           echo "$SSHKEY" >~/github_key
           chmod 600 ~/github_key
@@ -94,7 +94,7 @@ jobs:
           cd website
           cp ../extra_tests/cpython_tests_results.json ./_data/regrtests_results.json
           git add ./_data/regrtests_results.json
-          if git -c user.name="Github Actions" -c user.email="actions@github.com" commit -m "Update regression test results" --author="$GITHUB_ACTOR"; then
+          if git -c user.name="Github Actions" -c user.email="actions@github.com" commit -m "Update regression test results"; then
             git push
           fi
 
@@ -104,13 +104,13 @@ jobs:
     # Disable this scheduled job when running on a fork.
     if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: true
 
       - uses: dtolnay/rust-toolchain@stable
 
-      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
 
       - name: build rustpython
         run: cargo build --release --verbose
@@ -126,7 +126,6 @@ jobs:
         if: ${{ github.event_name != 'pull_request' }}
         env:
           SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }}
-          GITHUB_ACTOR: ${{ github.actor }}
         run: |
           echo "$SSHKEY" >~/github_key
           chmod 600 ~/github_key
@@ -157,7 +156,7 @@ jobs:
           }
           EOF
           git add -A
-          if git -c user.name="Github Actions" -c user.email="actions@github.com" commit -m "Update what is left results" --author="$GITHUB_ACTOR"; then
+          if git -c user.name="Github Actions" -c user.email="actions@github.com" commit -m "Update what is left results"; then
             git push
           fi
 
@@ -167,13 +166,13 @@ jobs:
     # Disable this scheduled job when running on a fork.
     if: ${{ github.repository == 'RustPython/RustPython' || github.event_name != 'schedule' }}
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: true
 
       - uses: dtolnay/rust-toolchain@stable
 
-      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
 
       - run: cargo install cargo-criterion
 
@@ -203,6 +202,8 @@ jobs:
         if: ${{ github.event_name != 'pull_request' }}
         env:
           SSHKEY: ${{ secrets.ACTIONS_TESTS_DATA_DEPLOY_KEY }}
+          COMMIT_SHA: ${{ github.sha }}
+          REF_NAME: ${{ github.ref_name }}
         run: |
           echo "$SSHKEY" >~/github_key
           chmod 600 ~/github_key
@@ -214,8 +215,8 @@ jobs:
           cp -r ../target/criterion ./assets/criterion
           printf '{\n  "generated_at": "%s",\n  "rustpython_commit": "%s",\n  "rustpython_ref": "%s"\n}\n' \
             "$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
-            "${{ github.sha }}" \
-            "${{ github.ref_name }}" > ./_data/criterion-metadata.json
+            "$COMMIT_SHA" \
+            "$REF_NAME" > ./_data/criterion-metadata.json
           git add ./assets/criterion ./_data/criterion-metadata.json
           if git -c user.name="Github Actions" -c user.email="actions@github.com" commit -m "Update benchmark results"; then
             git push

.github/workflows/lib-deps-check.yaml

@@ -6,6 +6,8 @@ on:
     paths:
       - "Lib/**"
 
+permissions: {}
+
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref_name }}-${{ github.event.pull_request.number }}
   cancel-in-progress: true
@@ -18,7 +20,7 @@ jobs:
     timeout-minutes: 10
     steps:
       - name: Checkout base branch
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           # Use base branch for scripts (security: don't run PR code with elevated permissions)
           ref: ${{ github.event.pull_request.base.ref }}
@@ -26,13 +28,17 @@ jobs:
           persist-credentials: false
 
       - name: Fetch PR head
+        env:
+          PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
         run: |
-          git fetch origin ${{ github.event.pull_request.head.sha }}
+          git fetch origin "$PR_HEAD_SHA"
 
       - name: Checkout PR Lib files
+        env:
+          PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
         run: |
           # Checkout only Lib/ directory from PR head for accurate comparison
-          git checkout ${{ github.event.pull_request.head.sha }} -- Lib/
+          git checkout "$PR_HEAD_SHA" -- Lib/
 
       - name: Get target CPython version
         id: cpython-version
@@ -41,7 +47,7 @@ jobs:
           echo "version=${version}" >> "$GITHUB_OUTPUT"
 
       - name: Checkout CPython
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           repository: python/cpython
           path: cpython
@@ -51,14 +57,17 @@ jobs:
 
       - name: Get changed Lib files
         id: all-changed-files
+        env:
+          PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
+          PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
         run: |
           # Get the list of changed files under Lib/
           {
             echo 'changed<<EOF'
 
-            git diff --name-only ${{ github.event.pull_request.base.sha }} ${{ github.event.pull_request.head.sha }} -- 'Lib/*.py' 'Lib/**/*.py'
+            git diff --name-only "$PR_BASE_SHA" "$PR_HEAD_SHA" -- 'Lib/*.py' 'Lib/**/*.py'
 
-            echo 'EOF' 
+            echo 'EOF'
           } >> "$GITHUB_OUTPUT"
 
       - name: Parse changed files
@@ -98,7 +107,7 @@ jobs:
 
       - name: Setup Python
         if: steps.changed-files.outputs.modules != ''
-        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
 
       - name: Run deps check
         if: steps.changed-files.outputs.modules != ''
@@ -114,7 +123,7 @@ jobs:
 
       - name: Post comment
         if: steps.deps-check.outputs.deps_output != ''
-        uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0 # v3.0.4
+        uses: marocchino/sticky-pull-request-comment@5770ad5eb8f42dd2c4f34da00c94c5381e49af88 # v3.0.5
         with:
           header: lib-deps-check
           number: ${{ github.event.pull_request.number }}
@@ -131,7 +140,7 @@ jobs:
 
       - name: Remove comment if no Lib changes
         if: steps.changed-files.outputs.modules == ''
-        uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0 # v3.0.4
+        uses: marocchino/sticky-pull-request-comment@5770ad5eb8f42dd2c4f34da00c94c5381e49af88 # v3.0.5
         with:
           header: lib-deps-check
           number: ${{ github.event.pull_request.number }}

.github/workflows/release.yml

@@ -52,7 +52,7 @@ jobs:
           #   target: aarch64-pc-windows-msvc
       fail-fast: false
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
@@ -91,7 +91,7 @@ jobs:
     permissions:
       contents: read
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 
@@ -114,7 +114,7 @@ jobs:
       - name: install wasm-pack
         run: curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh
 
-      - uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
+      - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
         with:
           package-manager-cache: false
 
@@ -156,7 +156,7 @@ jobs:
     permissions:
       contents: write # for creating a release
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
 

.github/workflows/update-caches.yml

@@ -19,6 +19,7 @@ env:
   CARGO_PROFILE_TEST_DEBUG: 0
   CARGO_PROFILE_DEV_DEBUG: 0
   CARGO_PROFILE_RELEASE_DEBUG: 0
+  # Keep feature list in sync with CI's release build in .github/workflows/ci.yaml.
   CARGO_ARGS: --workspace --no-default-features --features stdlib,importlib,stdio,encodings,sqlite,ssl-rustls-aws-lc,host_env,threading,jit --exclude rustpython_wasm --exclude rustpython-compiler-source --exclude rustpython-venvlauncher
 
 jobs:
@@ -39,14 +40,14 @@ jobs:
             target: ""
     steps:
       - name: Checkout RustPython main branch
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           repository: RustPython/RustPython
           ref: main
           persist-credentials: false
 
       - name: Setup Rust
-        uses: dtolnay/rust-toolchain@3c5f7ea28cd621ae0bf5283f0e981fb97b8a7af9
+        uses: dtolnay/rust-toolchain@stable
         with:
           toolchain: ${{ matrix.toolchain }}
           target: ${{ matrix.target }}
@@ -66,7 +67,7 @@ jobs:
         run: cargo build --profile release ${{ env.CARGO_ARGS }}
 
       - name: Save cache
-        uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           path: |
             ~/.cargo/bin/

.github/workflows/update-doc-db.yml

@@ -30,13 +30,13 @@ jobs:
           - windows-latest
           - macos-latest
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
           sparse-checkout: |
             crates/doc
 
-      - uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
         with:
           python-version: ${{ inputs.python-version }}
 
@@ -58,7 +58,7 @@ jobs:
       contents: write
       pull-requests: write
     steps:
-      - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: true
           ref: ${{ inputs.base-ref }}

.github/workflows/update-libs-status.yaml

@@ -21,7 +21,7 @@ jobs:
     if: ${{ github.repository == 'RustPython/RustPython' }}
     steps:
       - name: Clone RustPython
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           path: rustpython
           persist-credentials: false
@@ -37,7 +37,7 @@ jobs:
           echo "version=${version}" >> "$GITHUB_OUTPUT"
 
       - name: Clone CPython
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           repository: python/cpython
           path: cpython

.github/workflows/upgrade-pylib.lock.yml

@@ -58,7 +58,7 @@ jobs:
       comment_repo: ""
     steps:
       - name: Setup Scripts
-        uses: github/gh-aw/actions/setup@2c1a237d2048b0e2412e7d7528892ea1257840e2 # v0.74.4
+        uses: github/gh-aw/actions/setup@8b820ae1073f301991aaf2f307f7e271f618bb9f # v0.82.14
         with:
           destination: /opt/gh-aw/actions
       - name: Check workflow file timestamps
@@ -99,22 +99,22 @@ jobs:
       secret_verification_result: ${{ steps.validate-secret.outputs.verification_result }}
     steps:
       - name: Setup Scripts
-        uses: github/gh-aw/actions/setup@2c1a237d2048b0e2412e7d7528892ea1257840e2 # v0.74.4
+        uses: github/gh-aw/actions/setup@8b820ae1073f301991aaf2f307f7e271f618bb9f # v0.82.14
         with:
           destination: /opt/gh-aw/actions
       - name: Checkout repository
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           persist-credentials: false
       - name: Setup Python
-        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
+        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
         with:
           python-version: '3.14'
       - name: Create gh-aw temp directory
         run: bash /opt/gh-aw/actions/create_gh_aw_tmp_dir.sh
       # Cache configuration from frontmatter processed below
       - name: Cache (cpython-lib-${{ env.PYTHON_VERSION }})
-        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
+        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
         with:
           key: cpython-lib-${{ env.PYTHON_VERSION }}
           path: cpython
@@ -806,7 +806,7 @@ jobs:
       total_count: ${{ steps.missing_tool.outputs.total_count }}
     steps:
       - name: Setup Scripts
-        uses: github/gh-aw/actions/setup@2c1a237d2048b0e2412e7d7528892ea1257840e2 # v0.74.4
+        uses: github/gh-aw/actions/setup@8b820ae1073f301991aaf2f307f7e271f618bb9f # v0.82.14
         with:
           destination: /opt/gh-aw/actions
       - name: Download agent output artifact
@@ -927,7 +927,7 @@ jobs:
       success: ${{ steps.parse_results.outputs.success }}
     steps:
       - name: Setup Scripts
-        uses: github/gh-aw/actions/setup@2c1a237d2048b0e2412e7d7528892ea1257840e2 # v0.74.4
+        uses: github/gh-aw/actions/setup@8b820ae1073f301991aaf2f307f7e271f618bb9f # v0.82.14
         with:
           destination: /opt/gh-aw/actions
       - name: Download agent artifacts
@@ -1039,7 +1039,7 @@ jobs:
       process_safe_outputs_temporary_id_map: ${{ steps.process_safe_outputs.outputs.temporary_id_map }}
     steps:
       - name: Setup Scripts
-        uses: github/gh-aw/actions/setup@2c1a237d2048b0e2412e7d7528892ea1257840e2 # v0.74.4
+        uses: github/gh-aw/actions/setup@8b820ae1073f301991aaf2f307f7e271f618bb9f # v0.82.14
         with:
           destination: /opt/gh-aw/actions
       - name: Download agent output artifact
@@ -1061,7 +1061,7 @@ jobs:
           path: /tmp/gh-aw/
       - name: Checkout repository
         if: ((!cancelled()) && (needs.agent.result != 'skipped')) && (contains(needs.agent.outputs.output_types, 'create_pull_request'))
-        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
         with:
           token: ${{ github.token }}
           persist-credentials: false

.github/workflows/upgrade-pylib.md

@@ -52,7 +52,7 @@ cache:
     - cpython-lib-
 
 env:
-  PYTHON_VERSION: "v3.14.4"
+  PYTHON_VERSION: "v3.14.6"
   ISSUE_ID: "6839"
 ---
 

.github/zizmor.yml

@@ -1,4 +1,9 @@
 rules:
+  excessive-permissions:
+    ignore:
+      # pull_request_target is needed to post PR comments with pull-requests: write.
+      # Workflow-level permissions: {} restricts defaults; only the job has write access.
+      - lib-deps-check.yaml:3
   unpinned-uses:
     config:
       policies:

.gitignore

@@ -8,7 +8,9 @@ __pycache__/
 .repl_history.txt
 .vscode/
 wasm-pack.log
-.idea/
+.idea/*
+!.idea/icon.svg
+!.idea/vcs.xml
 .envrc
 
 flame-graph.html
@@ -27,4 +29,5 @@ Lib/site-packages/*
 Lib/test/data/*
 !Lib/test/data/README
 cpython/
-.claude/scheduled_tasks.lock
+.claude/
+docs/superpowers/

.idea/icon.svg

@@ -0,0 +1,3 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="512" height="512" viewBox="0 0 512 512">
+  <image href="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAIAAAACACAYAAADDPmHLAAAAAXNSR0IArs4c6QAAAJ5lWElmTU0AKgAAAAgABAEaAAUAAAABAAAAPgEbAAUAAAABAAAARgEoAAMAAAABAAIAAIdpAAQAAAABAAAATgAAAAAAAAEsAAAAAQAAASwAAAABAASShgAHAAAAGQAAAISgAQADAAAAAQABAACgAgAEAAAAAQAAAICgAwAEAAAAAQAAAIAAAAAAQVNDSUkAAABDcmVhdGVkIHdpdGggR0lNUADcIp6GAAAACXBIWXMAAC4jAAAuIwF4pT92AAABaWlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNi4wLjAiPgogICA8cmRmOlJERiB4bWxuczpyZGY9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkvMDIvMjItcmRmLXN5bnRheC1ucyMiPgogICAgICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgICAgICAgICB4bWxuczpleGlmPSJodHRwOi8vbnMuYWRvYmUuY29tL2V4aWYvMS4wLyI+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PkNyZWF0ZWQgd2l0aCBHSU1QPC9leGlmOlVzZXJDb21tZW50PgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KLa3NoAAAHSZJREFUeAHtXQtgVMXVPnPv3VcSQhDkJaAIiDZAAmilgBosAiLa2haK9rf8WgsERdu/Pn7tX01/sb9FW62Wd/vbap9Sq/IIlFZFAa0PIIBR8VF5+EAeEkIem929d/qdu9llEzZmc+/d7G6yA5O9j5m5M+ecOXPmzJkzRNmQhUAWAlkIZCHQSSEgOmm7U9rso1+lggJJ/WuDdCivnA6ksjJKKj/e2b4tiYT/MprfRaetAYMqNIW2+y+nFbVXUN9UwSLLAdoJ8nI6nRJsoEUuopk6vhkyiFRAX8MDf4Ce9dbQFLGRQu1UnehntOhV9sI2BD4oIW+ffDpdSBomJZ2JAs+UgnwkSfP7aahXoXMbgH0X+K5HBeKZEoL0Fuhgm+2PWywgywEsAo6zoVergQYqFAZdYAi6CIgejscD3ArliGaQNcD/g+j1eF6Py3/i9QbdoBd9ObRdrKR6G9WwlTXLAdoIvs8mUtcuHjoPCJ/kr6cSdOYRLpU8XAwQStypG/AbG5gWmCAURLyqBetfj/F/uVhNVbHpUnHNdcuGBCBQN5XO11T6Nlj7FCDzTGbjkV6NHm0GHtM5MqI58Hs9HDlJHWKuhnccQSSHcb9DVWgd0q11r6bdeBwpCq/aJzRWtX0+lolfOT6Jerq8tECRNAtId7PwxkhlTDHw0JNNpHPvD0o6gkfv48UuyAEV+N2DLu8Hgo8j2VFS6Tq3oNs5LwiJgHwUAFmggTZCCLwkFUJglgCAg5ZCzVQqdqv0OBA/LADezpyde7cLkVk6BLoQft4GLp/Fqw26QhU5q+gTPGMcxw310+hacInvgyjOQFn8b2dQp5ty19LWuBmS/BB1zYZ4EKi+jMb6FFqJHt6XpXUGFEvuIIQ6IP+1kKRyvPu7qxpS/EbyxyujpWcfYLbQy0c9fD6i9+rp0JB11NBS2mQ/zxJAHAjXAfmQ5J8Gok9lgY57PLq0BNt+HGzgAU857YqTLSMfZQmgGdqglRvphlAG9tyHke8JC3tHoaG50bea/tAsecbfZgkgBoVyIg3QffQcxuhBPOazpI+uv08EaIbYQK/EJO0wl51GDzB4/sOerg0er6KFfEFD8SqKSzN0luHCgSfylYcfLuslqgfVobsrGO+hxat9yXXmnXNPv/1A8cD9Qzil1BSMDkZ3SPk9pCHdoBMzoCCWEY/hd3fF0tI94afp/zcKgPSvauI1LJy3KE9TPUMUPTSKDFEsFTkUPbkPSsgHCiF6CTeugWJTtotI7KJA6rmRG/5akJRgtaLValKPdBSGF197BFMITwVigzRYUKjCn81CqCt1I7R+59LSg7FJ0u26WQvSrXqJ12fY9Q/1Ut3eEiHFNCDmi0DCQEXjpRZhSm8E5DA/D/+PRfOJb7DipnmIKHWaPDfTxUnMiUAUiqLxZ8AT9D0GGf+9Y/HcP/NtOoaMJgBm610M93iSytWS5GShaKcJIEAaOpAOhLO2pd0DCAB0J3WMI1wXKRtIhq6rWDIvLQXIjCSA0bOXDQBTngncXoVYrKhhgJtIb3eEx3wQCDeCATq063nqNng0efJPDQ8TUj+iG7IEw8EbManT4jIytqVFZVqrxIj5K/opujFfJzFLUbVejHDuaUYo0FrWdnnPvUkaIao98D516X8OBIUwN1I0d3dFBm7F61ntUpE2fCQjOAALdS6plQpFfA9svi8jPeW9vUUgQ7vLiDeFjcYhSCggDONTkMfoHUtLP2oxa/wXp+PxeYjPIh6Nn8T6U57ppnUYNW/p19zCtRnj6kJUtC/39vRFPoMSqG8uf+AexNtLy8k/KwbY5mwC9zwDjY0QXKOBucYOxJWIY6JPHbxI2yFg1LxFgzFb+wnAOR3QSxs2bxX2PDP4dPv6W5D/q4hsAzgU0YsYK6kyRw4ibkbMQ7wacSPizxCZAzge0pIARs5dMtMQ6v2KovaT5vjOU7gMDxgWGqoOTW1sBVsA7UVkG4HYYZiJgYniuzHP2WqYl5mTEmI/npQPtKXQ0bMf6KGrXe4VimICgKdzmRhQf57+oW/HdG5wsZr9b13/3uoHt6BNxxFZQRSPsllBdQbiWMRZiCWIHO5CvMe8cvBP2hDAiNnLzlc1sVSoWrERYi4YAzwHG9weRfmrDpLmzSXNBy7ORACCIMP4JCe3Z/FLD3ytrZrBYtSZOcdziP90uv5MbSkPI+cs/YZwq38SQhkodUZ+JgdJe/7xGzB2hfJ6n8nSP9YVXMB/aO3rD3/7MQst4yFgE+KHFvK2miXlMkDx3CXfEqr6K3R4L8+hMz8IGjDhGoJaGoLrifYohngyHduWUgIoLl02HdOjFZD0gfx4w2E6gqz1OnnyuvFk0GT/UALxDGZDUNXXt56z/VOkTAYowvxeESpYosztSMg3UQj2r6gq2D8Tgb460KD+Z+Wvr/+s/dHb+hdTwgFGli6+AmPkb1G9dkI+tHO8rCdUU0sXDyxs/cPWvrGBrYHYzPtzA4S8qGKKBb6w+veoocsXsDK5XNHpH0B+2go2rTXvc9tu5eXI0uUXEeztAKmCZE/zoDYG4lmJFIRth2Qh6l189yNw52NSyKghJ/BM02u3zehOdQPY7Bs4hDZG+Nd4C3+/z93zoNcInNDOSaEiL1uD1xlSBpB0P6jkoNSlrphlai4p9bcyxSikXQlgVOniEYbQyoGU08zlUisU1FoeYM+UusNTye1o4NNSKP9wi8DuVxfd2KJCRU6i66CQ/XV0Zs6QkdDIVdMNYhPtbO2zmfq+3QigcPYvBrg133pg55ykTPUiiNeDVdCtPQNu/NtQz25bKstmJLRUyMwbNvv3+VS6jTdw8miAzZy82aMKQ/lPPAFaDLvA2kxFdEv1bhcCGHHLY7lqbd0qobkvdn7pNmqAcQzat0dVnZZtXT7n7ZYa3NpzEME9MPr7n8YtXaYMwMahIIQK7A76JRQnz2BPH2/r6hChXQiguHTJYkXzlDqNfNMuj+dbUv7ZUGjBjkWzK53ACpw4zMa2rfsx0OebW7hRKIjC3BUEYXE/bjfAKHQ9Nohu9X5MH4qt5gJOq5+uv5wGQNXRHXsDQ9Ua7Tl1lakSbjVfMhMknQCK5yyah56/KDzVayZm22iZOb/Wg/8Shrht+9LZjitZApfTGIwqCzFNuoBXJHhrNwdzcyeIgWd4eFaN4WYvSPAVpF2PrWGv5jxjEkg4cePf6ql0lkeje8FBvowdRV2QVoKTvA+ucmfOGnqqSeJ2vkkqAYy8YemXpFQ24CN50amSzQZCXYzRmhdbjD8aqrht5yPfZek+KeHdS8nT30WzgLjvA/FnM7ACQHzEeJRnljxN5MjPgNwj+NmB2xdgAvASCOIz1aD/AMKvwh7D3nALYy4NcGWZoyCPH0QwzbcmOUu9iQAFdU1OKPzOr05xu/XnSVVHOCXxmyyfZDUULLdWLJmzPDk1P7nUI5dSfp5G0/DmGsTxQB6v1UeQHiUIJgTmEEC4yTEgR+helVTmHixT8LuIlTEuSWByCQcTP/KsoQVcXioCOFxyguYOlYH1j3Bq3GfkQ8g7igF/5o4lczYkp9bxS+2+DqyezG1hf/CDnUOrMwGzjLFQLgzH8yFAskkQrEMwezkemggGc4jIEHwPDvEOhgvmEm4QSbUrRHtRzna8SlngejkeoOmbSIqrHGwa5rpors0QFvZktdCD39y27Ia00amzi5iGWhoE2pyAVjJRfBFNPcMDhSM3mZsOpJu9n7kDBMeXsBX8BznrnF/WtQpixwmg+HuPFlCDf5NQXcOcYP2syUOoEdK4etviuautNrQ98kkMFfUqFYLNjwHyR+Kbg0AKAyHw9eEhgFXLIIgaAP2X9XX0QP6zybP0SbS9zhPA3CX3Ki7PnY6wfgh7sLBt0EP6tTuXzf1joo1Kp3SY+l2E3v934N7FsgDLACwAQvh7B15F5qVSAGQ4md3LKYDBsOML6LE3Gk4YdWCQBPIxihg/yFTkM1yxpfwF9PpZ4AiHvJC4GPk8NLg0OgvywPlOwd5qOU4KgQLKmLtU1ZUfNuS0WqVwPraiBRf5PyyqLLJXUupz56ymP/qvoNdg4vhNIH80uEA32LruPJJDD6a6do4RAAS/C2H79vWwPZ+9ZoWVPIEnCnp/ere9ktInt3cVvYfa3Js+NQrXxBkCKCLine truncated
+</svg>

.idea/vcs.xml

@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="IssueNavigationConfiguration">
+    <option name="links">
+      <list>
+        <IssueNavigationLink>
+          <option name="issueRegexp" value="#(\d+)" />
+          <option name="linkRegexp" value="https://github.com/RustPython/RustPython/pull/$1" />
+        </IssueNavigationLink>
+      </list>
+    </option>
+  </component>
+  <component name="VcsDirectoryMappings">
+    <mapping directory="$PROJECT_DIR$" vcs="Git" />
+  </component>
+</project>

.pre-commit-config.yaml

@@ -10,7 +10,7 @@ repos:
         priority: 0
 
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.15.12
+    rev: v0.15.22
     hooks:
       - id: ruff-format
         priority: 0
@@ -63,7 +63,7 @@ repos:
           - manual
 
   - repo: https://github.com/streetsidesoftware/cspell-cli
-    rev: v10.0.0
+    rev: v10.0.1
     hooks:
       - id: cspell
         types: [rust]
@@ -77,7 +77,7 @@ repos:
         priority: 0
 
   - repo: https://github.com/rbubley/mirrors-prettier
-    rev: v3.8.3
+    rev: v3.9.5
     hooks:
       - id: prettier
         files: '^wasm/.*$'

.python-version

@@ -1 +1 @@
-3.14.5
+3.14.6

AGENTS.md

@@ -1,6 +1,6 @@
-# GitHub Copilot Instructions for RustPython
+# AI Agent Instructions for RustPython
 
-This document provides guidelines for working with GitHub Copilot when contributing to the RustPython project.
+This document provides guidelines for AI coding agents (GitHub Copilot, Claude Code, Gemini, etc.) contributing to the RustPython project.
 
 ## Project Overview
 
@@ -13,22 +13,7 @@ RustPython is a Python 3 interpreter written in Rust, implementing Python 3.14.0
 
 ## Repository Structure
 
-- `src/` - Top-level code for the RustPython binary
-- `vm/` - The Python virtual machine implementation
-  - `builtins/` - Python built-in types and functions
-  - `stdlib/` - Essential standard library modules implemented in Rust, required to run the Python core
-- `compiler/` - Python compiler components
-  - `parser/` - Parser for converting Python source to AST
-  - `core/` - Bytecode representation in Rust structures
-  - `codegen/` - AST to bytecode compiler
-- `Lib/` - CPython's standard library in Python (copied from CPython). **IMPORTANT**: Do not edit this directory directly; The only allowed operation is copying files from CPython.
-- `derive/` - Rust macros for RustPython
-- `common/` - Common utilities
-- `extra_tests/` - Integration tests and snippets
-- `stdlib/` - Non-essential Python standard library modules implemented in Rust (useful but not required for core functionality)
-- `wasm/` - WebAssembly support
-- `jit/` - Experimental JIT compiler implementation
-- `pylib/` - Python standard library packaging (do not modify this directory directly - its contents are generated automatically)
+See the "Code organization" section in [CONTRIBUTING.md](CONTRIBUTING.md#code-organization) for the current directory layout.
 
 ## AI Agent Rules
 
@@ -265,12 +250,10 @@ cargo run --features jit
 
 ### Linux Build and Debug on macOS
 
-See the "Testing on Linux from macOS" section in [DEVELOPMENT.md](DEVELOPMENT.md#testing-on-linux-from-macos).
+See the "Testing on Linux from macOS" section in [CONTRIBUTING.md](CONTRIBUTING.md#testing-on-linux-from-macos).
 
 ### Building venvlauncher (Windows)
 
-See DEVELOPMENT.md "CPython Version Upgrade Checklist" section.
-
 **IMPORTANT**: All 4 venvlauncher binaries use the same source code. Do NOT add multiple `[[bin]]` entries to Cargo.toml. Build once and copy with different names.
 
 ## Test Code Modification Rules
@@ -301,7 +284,7 @@ If you modify any file under `.github/workflows/`, the change must pass a [zizmo
 ## Documentation
 
 - Check the [architecture document](/architecture/architecture.md) for a high-level overview
-- Read the [development guide](/DEVELOPMENT.md) for detailed setup instructions
+- Read the [development guide](/CONTRIBUTING.md) for detailed setup instructions
 - Generate documentation with `cargo doc --no-deps --all`
 - Online documentation is available at [docs.rs/rustpython](https://docs.rs/rustpython/)
 - [How to update test files](https://github.com/RustPython/RustPython/wiki/How-to-update-test-files#checkout-cpython-source-code-initial-setup) — guide for syncing test cases from upstream CPython into the `Lib/` directory

CONTRIBUTING.md

@@ -89,7 +89,15 @@ $ pytest -v
 Rust unit tests can be run with `cargo`:
 
 ```shell
-$ cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher
+$ cargo test --workspace --exclude rustpython_wasm --exclude rustpython-venvlauncher --exclude rustpython-capi
+```
+
+`rustpython-capi` needs to be tested from inside its own directory, since it has a
+separate `cargo` config that only applies there:
+
+```shell
+$ cd crates/capi
+$ cargo test
 ```
 
 Python unit tests can be run by compiling RustPython and running the test module:

Cargo.toml

@@ -49,7 +49,6 @@ env_logger = "0.11"
 flamescope = { version = "0.1.2", optional = true }
 
 rustls = { workspace = true, optional = true }
-rustls-graviola = { workspace = true, optional = true }
 
 [target.'cfg(windows)'.dependencies]
 libc = { workspace = true }
@@ -60,6 +59,7 @@ rustyline = { workspace = true }
 [dev-dependencies]
 criterion = { workspace = true }
 pyo3 = { workspace = true, features = ["auto-initialize"] }
+rustls-graviola = { workspace = true }
 rustpython-stdlib = { workspace = true }
 ruff_python_parser = { workspace = true }
 
@@ -79,7 +79,6 @@ path = "src/main.rs"
 name = "custom_tls_providers"
 path = "examples/custom_tls_providers.rs"
 required-features = [
-    "rustls-graviola",
     "rustls/ring",
     "rustpython-pylib/freeze-stdlib",
     "rustpython-stdlib/ssl-rustls",
@@ -117,6 +116,14 @@ opt-level = 3
 [profile.release]
 lto = "thin"
 
+[profile.wasm-release]
+inherits = "release"
+opt-level = "s"
+lto = true
+codegen-units = 1
+strip = true
+panic = "abort"
+
 [patch.crates-io]
 parking_lot_core = { git = "https://github.com/youknowone/parking_lot", branch = "rustpython" }
 # REDOX START, Uncomment when you want to compile/check with redoxer
@@ -172,45 +179,38 @@ rustpython-vm = { path = "crates/vm", default-features = false, version = "0.5.0
 rustpython-pylib = { path = "crates/pylib", version = "0.5.0" }
 rustpython-stdlib = { path = "crates/stdlib", default-features = false, version = "0.5.0" }
 rustpython-sre_engine = { path = "crates/sre_engine", version = "0.5.0" }
+rustpython-unicode = { path = "crates/unicode", version = "0.5.0" }
 rustpython-wtf8 = { path = "crates/wtf8", version = "0.5.0" }
 rustpython-doc = { path = "crates/doc", version = "0.5.0" }
 
-# Use RustPython-packaged Ruff crates from the published fork while keeping
-# existing crate names in the codebase.
-ruff_python_parser = { package = "rustpython-ruff_python_parser", version = "0.15.8" }
-ruff_python_ast = { package = "rustpython-ruff_python_ast", version = "0.15.8" }
-ruff_text_size = { package = "rustpython-ruff_text_size", version = "0.15.8" }
-ruff_source_file = { package = "rustpython-ruff_source_file", version = "0.15.8" }
-# To update ruff crates, comment out the above lines and uncomment the following lines to pull directly from the Ruff repository at the specified commit hash.
-# Ruff tag 0.15.8 is based on commit c2a8815842f9dc5d24ec19385eae0f1a7188b0d9
-# at the time of this capture. We use the commit hash to ensure reproducible builds.
-# ruff_python_parser = { git = "https://github.com/astral-sh/ruff.git", rev = "c2a8815842f9dc5d24ec19385eae0f1a7188b0d9" }
-# ruff_python_ast = { git = "https://github.com/astral-sh/ruff.git", rev = "c2a8815842f9dc5d24ec19385eae0f1a7188b0d9" }
-# ruff_text_size = { git = "https://github.com/astral-sh/ruff.git", rev = "c2a8815842f9dc5d24ec19385eae0f1a7188b0d9" }
-# ruff_source_file = { git = "https://github.com/astral-sh/ruff.git", rev = "c2a8815842f9dc5d24ec19385eae0f1a7188b0d9" }
+# Use the RustPython Ruff fork for RustPython public `_ast` metadata.
+ruff_python_parser = { package = "rustpython-ruff_python_parser", git = "https://github.com/RustPython/ruff.git", tag = "0.15.19-rustpython" }
+ruff_python_ast = { package = "rustpython-ruff_python_ast", git = "https://github.com/RustPython/ruff.git", tag = "0.15.19-rustpython" }
+ruff_text_size = { package = "rustpython-ruff_text_size", git = "https://github.com/RustPython/ruff.git", tag = "0.15.19-rustpython" }
+ruff_source_file = { package = "rustpython-ruff_source_file", git = "https://github.com/RustPython/ruff.git", tag = "0.15.19-rustpython" }
 
 der = { version = "0.8", features = ["alloc", "oid", "pem", "zeroize"] }
-phf = { version = "0.13.1", default-features = false, features = ["macros"]}
+phf = { version = "0.14.0", default-features = false, features = ["macros"]}
 adler32 = "1.2.0"
 approx = "0.5.1"
-ascii = "1.1"
+ascii = { version = "1.1", default-features = false }
 base64 = "0.22"
-blake2 = "0.10.4"
+blake2 = "0.11.0-rc.6"
 bitflags = "2.11.0"
 bitflagset = "0.0.3"
-bstr = "1"
+bstr = { version = "1", default-features = false, features = ["unicode"] }
 bzip2 = "0.6"
 chrono = { version = "0.4.44", default-features = false, features = ["clock", "std"] }
 console_error_panic_hook = "0.1"
-constant_time_eq = "0.4"
-cranelift = "0.131.2"
-cranelift-jit = "0.131.2"
-cranelift-module = "0.131.0"
+constant_time_eq = "0.5"
+cranelift = "0.132.0"
+cranelift-jit = "0.132.0"
+cranelift-module = "0.132.0"
 crc32fast = "1.3.2"
 criterion = { version = "0.8", features = ["html_reports"] }
 crossbeam-utils = "0.8.21"
 csv-core = "0.1.11"
-digest = "0.10.7"
+digest = "0.11"
 dns-lookup = "3.0"
 dyn-clone = "1.0.10"
 exitcode = "1.1.2"
@@ -220,38 +220,39 @@ flate2 = { version = "1.1.9", default-features = false }
 # Bump only when the openssl crate bumps it
 foreign-types-shared = "0.1"
 gethostname = "1.0.2"
-getrandom = { version = "0.3", features = ["std"] }
+getrandom = { version = "0.4", features = ["std"] }
 glob = "0.3"
 half = "2"
 hex = "0.4.3"
 hexf-parse = "0.2.1"
-hmac = "0.12"
+hmac = "0.13"
 indexmap = { version = "2.14.0", features = ["std"] }
 insta = "1.47"
-itertools = "0.14.0"
+itertools = { version = "0.15.0", default-features = false, features = ["use_alloc"] }
+itoa = "1"
 is-macro = "0.3.7"
 js-sys = "0.3"
-junction = "1.4.2"
+junction = "2.0.0"
 lexical-parse-float = "1.0.6"
 libc = "0.2.186"
 libffi = "5"
 libloading = "0.9"
-liblzma = "0.4"
-liblzma-sys = "0.4"
-libsqlite3-sys = "0.37"
+xz = "0.4.6-rc.0"
+xz-sys = "0.4.6-rc.0"
+libsqlite3-sys = "0.38"
 libz-rs-sys = "0.6"
 lock_api = "0.4"
-log = "0.4.29"
+log = "0.4.30"
 lz4_flex = "0.13"
 nix = { version = "0.31", features = ["fs", "user", "process", "term", "time", "signal", "ioctl", "socket", "sched", "zerocopy", "dir", "hostname", "net", "poll"] }
 mac_address = "1.1.3"
-malachite-bigint = "0.9.1"
-malachite-q = "0.9.1"
-malachite-base = "0.9.1"
-md-5 = "0.10.1"
-memchr = "2.8.0"
+malachite-bigint = "0.10.0"
+malachite-q = "0.10.0"
+malachite-base = "0.10.0"
+md-5 = "0.11"
+memchr = { version = "2.8.1", default-features = false, features = ["alloc"] }
 memmap2 = "0.9.10"
-mt19937 = "<=3.2"  # upgrade it once rand is upgraded
+mt19937 = "3.3"
 num-complex = "0.4.6"
 num-integer = "0.1.46"
 num-traits = "0.2"
@@ -264,22 +265,21 @@ openssl-probe = "0.2.1"
 optional = "0.5"
 parking_lot = "0.12.3"
 paste = "1.0.15"
-pbkdf2 = "0.12"
+pbkdf2 = "0.13"
 pem-rfc7468 = "1.0"
 pkcs8 = "0.11"
 proc-macro2 = "1.0.105"
 psm = "0.1"
 pymath = { version = "0.2.0", features = ["mul_add", "malachite-bigint", "complex"] }
-pyo3 = "0.28"
+pyo3 = "0.29"
 quote = "1.0.45"
 radium = "1.1.1"
-rand = "0.9"
-rand_core = { version = "0.9", features = ["os_rng"] }
+rand = "0.10"
 rapidhash = "4.4.1"
 result-like = "0.5.0"
-rustix = { version = "1.1", features = ["event", "param", "system"] }
+rustix = { version = "1.1", features = ["event", "fs", "param", "system"] }
 rustls = { version = "0.23.39", default-features = false }
-rustls-graviola = "0.3"
+rustls-graviola = "0.4"
 rustls-native-certs = "0.8"
 rustls-pemfile = "2.2"
 rustls-platform-verifier = "0.7"
@@ -288,11 +288,12 @@ serde = { package = "serde_core", version = "1.0.225", default-features = false,
 schannel = "0.1.29"
 scopeguard = "1"
 serde-wasm-bindgen = "0.6.5"
-sha-1 = "0.10.0"
-sha2 = "0.10.2"
-sha3 = "0.10.1"
+sha1 = "0.11"
+sha2 = "0.11"
+sha3 = "0.12"
+shake = "0.1"
 siphasher = "1"
-socket2 = "0.6.3"
+socket2 = "0.6.4"
 static_assertions = "1.1"
 strum = "0.28"
 strum_macros = "0.28"
@@ -303,16 +304,14 @@ tcl-sys = { git = "https://github.com/arihant2math/tkinter.git", tag = "v0.2.0"
 textwrap = { version = "0.16.2", default-features = false }
 termios = "0.3.3"
 thiserror = "2.0"
-timsort = "0.1.2"
+thin-vec = "0.2.14"
 tk-sys = { git = "https://github.com/arihant2math/tkinter.git", tag = "v0.2.0" }
 icu_casemap = "2"
 icu_locale = "2"
 icu_properties = "2"
 icu_normalizer = "2"
-uuid = "1.23.1"
-ucd = "0.1.1"
-unic-ucd-age = "0.9.0"
-unicode_names2 = "2.0.0"
+uuid = "1.23.2"
+unicode_names2 = { version = "3", default-features = false, features = ["no_std"] }
 widestring = "1.2.0"
 windows-sys = "0.61.2"
 wasm-bindgen = "0.2.106"
@@ -320,9 +319,9 @@ wasm-bindgen-futures = "0.4"
 web-sys = "0.3"
 webpki-roots = "1.0"
 which = "8"
-x509-cert = "0.2.5"
+x509-cert = "0.3.0"
 x509-parser = "0.18"
-xml = "1.2"
+xml = "1.3"
 writeable = "0.6"
 
 # Lints
@@ -351,29 +350,49 @@ similar_names = "allow"
 # restriction lints
 alloc_instead_of_core = "warn"
 cfg_not_test = "warn"
+iter_over_hash_type = "warn"
 redundant_test_prefix = "warn"
 std_instead_of_alloc = "warn"
 std_instead_of_core = "warn"
 tests_outside_test_module = "warn"
 
 # nursery lints to enforce gradually
+collection_is_never_read = "warn"
 debug_assert_with_mut_call = "warn"
 derive_partial_eq_without_eq = "warn"
 imprecise_flops = "warn"
+iter_on_empty_collections = "warn"
+iter_on_single_items = "warn"
+literal_string_with_formatting_args = "warn"
+needless_collect = "warn"
 or_fun_call = "warn"
 redundant_clone = "warn"
 search_is_some = "warn"
+significant_drop_in_scrutinee = "warn"
 single_option_map = "warn"
 trait_duplication_in_bounds = "warn"
+tuple_array_conversions = "warn"
+type_repetition_in_bounds = "warn"
+unnecessary_struct_initialization = "warn"
 unused_peekable = "warn"
 unused_rounding = "warn"
 use_self = "warn"
 useless_let_if_seq = "warn"
+while_float = "warn"
 
 # pedantic lints to enforce gradually
+assigning_clones = "warn"
+bool_to_int_with_if = "warn"
+checked_conversions = "warn"
 cloned_instead_of_copied = "warn"
 collapsible_else_if = "warn"
 comparison_chain = "warn"
+copy_iterator = "warn"
+doc_link_with_quotes = "warn"
+duration_suboptimal_units = "warn"
+elidable_lifetime_names = "warn"
+enum_glob_use = "warn"
+explicit_deref_methods = "warn"
 explicit_into_iter_loop = "warn"
 explicit_iter_loop = "warn"
 filter_map_next = "warn"
@@ -381,11 +400,37 @@ flat_map_option = "warn"
 format_collect = "warn"
 from_iter_instead_of_collect = "warn"
 inconsistent_struct_constructor = "warn"
+index_refutable_slice = "warn"
 inefficient_to_string = "warn"
+ip_constant = "warn"
+iter_filter_is_ok = "warn"
+iter_filter_is_some = "warn"
+large_futures = "warn"
+large_types_passed_by_value = "warn"
+manual_assert = "warn"
+manual_instant_elapsed = "warn"
 manual_is_variant_and = "warn"
 map_unwrap_or = "warn"
+match_bool = "warn"
+mismatching_type_param_order = "warn"
 must_use_candidate = "warn"
+mut_mut = "warn"
+needless_bitwise_bool = "warn"
+needless_for_each = "warn"
+non_std_lazy_statics = "warn"
+option_as_ref_cloned = "warn"
+ptr_offset_by_literal = "warn"
+range_minus_one = "warn"
+range_plus_one = "warn"
 redundant_else = "warn"
+ref_option = "warn"
+return_self_not_must_use = "warn"
+same_functions_in_if_condition = "warn"
+single_char_pattern = "warn"
+trivially_copy_pass_by_ref = "warn"
+unchecked_time_subtraction = "warn"
 uninlined_format_args = "warn"
+unnecessary_box_returns = "warn"
+unnecessary_join = "warn"
 unnecessary_wraps = "warn"
 unnested_or_patterns = "warn"

Lib/_collections_abc.py

@@ -461,8 +461,8 @@ class Buffer(metaclass=ABCMeta):
 class _CallableGenericAlias(GenericAlias):
     """ Represent `Callable[argtypes, resulttype]`.
 
-    This sets ``__args__`` to a tuple containing the flattened ``argtypes``
-    followed by ``resulttype``.
+    This sets ``__args__`` to a tuple containing the flattened
+    ``argtypes`` followed by ``resulttype``.
 
     Example: ``Callable[[int, str], float]`` sets ``__args__`` to
     ``(int, str, float)``.
@@ -927,8 +927,9 @@ class MutableMapping(Mapping):
     __marker = object()
 
     def pop(self, key, default=__marker):
-        '''D.pop(k[,d]) -> v, remove specified key and return the corresponding value.
-          If key is not found, d is returned if given, otherwise KeyError is raised.
+        '''D.pop(k[,d]) -> v, remove specified key and return the corresponding
+        value.  If key is not found, d is returned if given, otherwise
+        KeyError is raised.
         '''
         try:
             value = self[key]
@@ -962,9 +963,12 @@ class MutableMapping(Mapping):
 
     def update(self, other=(), /, **kwds):
         ''' D.update([E, ]**F) -> None.  Update D from mapping/iterable E and F.
-            If E present and has a .keys() method, does:     for k in E.keys(): D[k] = E[k]
-            If E present and lacks .keys() method, does:     for (k, v) in E: D[k] = v
-            In either case, this is followed by: for k, v in F.items(): D[k] = v
+            If E present and has a .keys() method, does:
+                for k in E.keys(): D[k] = E[k]
+            If E present and lacks .keys() method, does:
+                for (k, v) in E: D[k] = v
+            In either case, this is followed by:
+                for k, v in F.items(): D[k] = v
         '''
         if isinstance(other, Mapping):
             for key in other:
@@ -1029,8 +1033,8 @@ class Sequence(Reversible, Collection):
             yield self[i]
 
     def index(self, value, start=0, stop=None):
-        '''S.index(value, [start, [stop]]) -> integer -- return first index of value.
-           Raises ValueError if the value is not present.
+        '''S.index(value, [start, [stop]]) -> integer -- return first index of
+           value.  Raises ValueError if the value is not present.
 
            Supporting start and stop arguments is optional, but
            recommended.
@@ -1138,15 +1142,16 @@ class MutableSequence(Sequence):
             self[i], self[n-i-1] = self[n-i-1], self[i]
 
     def extend(self, values):
-        'S.extend(iterable) -- extend sequence by appending elements from the iterable'
+        """S.extend(iterable) -- extend sequence by appending elements from the
+        iterable"""
         if values is self:
             values = list(values)
         for v in values:
             self.append(v)
 
     def pop(self, index=-1):
-        '''S.pop([index]) -> item -- remove and return item at index (default last).
-           Raise IndexError if list is empty or index is out of range.
+        '''S.pop([index]) -> item -- remove and return item at index (default
+        last).  Raise IndexError if list is empty or index is out of range.
         '''
         v = self[index]
         del self[index]

Lib/_pydecimal.py

@@ -104,8 +104,8 @@ class DecimalException(ArithmeticError):
     anything, though.
 
     handle  -- Called when context._raise_error is called and the
-               trap_enabler is not set.  First argument is self, second is the
-               context.  More arguments can be given, those being after
+               trap_enabler is not set.  First argument is self, second is
+               the context.  More arguments can be given, those being after
                the explanation in _raise_error (For example,
                context._raise_error(NewError, '(-x)!', self._sign) would
                call NewError().handle(context, self._sign).)
@@ -222,11 +222,12 @@ class InvalidContext(InvalidOperation):
     """Invalid context.  Unknown rounding, for example.
 
     This occurs and signals invalid-operation if an invalid context was
-    detected during an operation.  This can occur if contexts are not checked
-    on creation and either the precision exceeds the capability of the
-    underlying concrete representation or an unknown or unsupported rounding
-    was specified.  These aspects of the context need only be checked when
-    the values are required to be used.  The result is [0,qNaN].
+    detected during an operation.  This can occur if contexts are not
+    checked on creation and either the precision exceeds the capability of
+    the underlying concrete representation or an unknown or unsupported
+    rounding was specified.  These aspects of the context need only be
+    checked when the values are required to be used.  The result is
+    [0,qNaN].
     """
 
     def handle(self, context, *args):
@@ -319,8 +320,9 @@ class FloatOperation(DecimalException, TypeError):
     Decimal.from_float() or context.create_decimal_from_float() do not
     set the flag.
 
-    Otherwise (the signal is trapped), only equality comparisons and explicit
-    conversions are silent. All other mixed operations raise FloatOperation.
+    Otherwise (the signal is trapped), only equality comparisons and
+    explicit conversions are silent.  All other mixed operations raise
+    FloatOperation.
     """
 
 # List of public traps and flags
@@ -2898,8 +2900,8 @@ class Decimal(object):
         """Compares self to other using the abstract representations.
 
         This is not like the standard compare, which use their numerical
-        value. Note that a total ordering is defined for all possible abstract
-        representations.
+        value. Note that a total ordering is defined for all possible
+        abstract representations.
         """
         other = _convert_other(other, raiseit=True)
 
@@ -2970,7 +2972,8 @@ class Decimal(object):
     def compare_total_mag(self, other, context=None):
         """Compares self to other using abstract repr., ignoring sign.
 
-        Like compare_total, but with operand's sign ignored and assumed to be 0.
+        Like compare_total, but with operand's sign ignored and assumed to
+        be 0.
         """
         other = _convert_other(other, raiseit=True)
 
@@ -4107,9 +4110,9 @@ class Context(object):
     def abs(self, a):
         """Returns the absolute value of the operand.
 
-        If the operand is negative, the result is the same as using the minus
-        operation on the operand.  Otherwise, the result is the same as using
-        the plus operation on the operand.
+        If the operand is negative, the result is the same as using the
+        minus operation on the operand.  Otherwise, the result is the same
+        as using the plus operation on the operand.
 
         >>> ExtendedContext.abs(Decimal('2.1'))
         Decimal('2.1')
@@ -4165,16 +4168,17 @@ class Context(object):
     def compare(self, a, b):
         """Compares values numerically.
 
-        If the signs of the operands differ, a value representing each operand
-        ('-1' if the operand is less than zero, '0' if the operand is zero or
-        negative zero, or '1' if the operand is greater than zero) is used in
-        place of that operand for the comparison instead of the actual
-        operand.
+        If the signs of the operands differ, a value representing each
+        operand ('-1' if the operand is less than zero, '0' if the operand
+        is zero or negative zero, or '1' if the operand is greater than
+        zero) is used in place of that operand for the comparison instead of
+        the actual operand.
 
-        The comparison is then effected by subtracting the second operand from
-        the first and then returning a value according to the result of the
-        subtraction: '-1' if the result is less than zero, '0' if the result is
-        zero or negative zero, or '1' if the result is greater than zero.
+        The comparison is then effected by subtracting the second operand
+        from the first and then returning a value according to the result of
+        the subtraction: '-1' if the result is less than zero, '0' if the
+        result is zero or negative zero, or '1' if the result is greater
+        than zero.
 
         >>> ExtendedContext.compare(Decimal('2.1'), Decimal('3'))
         Decimal('-1')
@@ -4237,8 +4241,8 @@ class Context(object):
         """Compares two operands using their abstract representation.
 
         This is not like the standard compare, which use their numerical
-        value. Note that a total ordering is defined for all possible abstract
-        representations.
+        value.  Note that a total ordering is defined for all possible
+        abstract representations.
 
         >>> ExtendedContext.compare_total(Decimal('12.73'), Decimal('127.9'))
         Decimal('-1')
@@ -4265,7 +4269,8 @@ class Context(object):
     def compare_total_mag(self, a, b):
         """Compares two operands using their abstract representation ignoring sign.
 
-        Like compare_total, but with operand's sign ignored and assumed to be 0.
+        Like compare_total, but with operand's sign ignored and assumed to
+        be 0.
         """
         a = _convert_other(a, raiseit=True)
         return a.compare_total_mag(b)
@@ -4923,8 +4928,8 @@ class Context(object):
 
         If either operand is a special value then the general rules apply.
         Otherwise, the operands are multiplied together
-        ('long multiplication'), resulting in a number which may be as long as
-        the sum of the lengths of the two operands.
+        ('long multiplication'), resulting in a number which may be as long
+        as the sum of the lengths of the two operands.
 
         >>> ExtendedContext.multiply(Decimal('1.20'), Decimal('3'))
         Decimal('3.60')
@@ -5200,19 +5205,19 @@ class Context(object):
         """Returns a value equal to 'a' (rounded), having the exponent of 'b'.
 
         The coefficient of the result is derived from that of the left-hand
-        operand.  It may be rounded using the current rounding setting (if the
-        exponent is being increased), multiplied by a positive power of ten (if
-        the exponent is being decreased), or is unchanged (if the exponent is
-        already equal to that of the right-hand operand).
+        operand.  It may be rounded using the current rounding setting (if
+        the exponent is being increased), multiplied by a positive power of
+        ten (if the exponent is being decreased), or is unchanged (if the
+        exponent is already equal to that of the right-hand operand).
 
         Unlike other operations, if the length of the coefficient after the
         quantize operation would be greater than precision then an Invalid
-        operation condition is raised.  This guarantees that, unless there is
-        an error condition, the exponent of the result of a quantize is always
-        equal to that of the right-hand operand.
+        operation condition is raised.  This guarantees that, unless there
+        is an error condition, the exponent of the result of a quantize is
+        always equal to that of the right-hand operand.
 
-        Also unlike other operations, quantize will never raise Underflow, even
-        if the result is subnormal and inexact.
+        Also unlike other operations, quantize will never raise Underflow,
+        even if the result is subnormal and inexact.
 
         >>> ExtendedContext.quantize(Decimal('2.17'), Decimal('0.001'))
         Decimal('2.170')
@@ -5266,13 +5271,13 @@ class Context(object):
         """Returns the remainder from integer division.
 
         The result is the residue of the dividend after the operation of
-        calculating integer division as described for divide-integer, rounded
-        to precision digits if necessary.  The sign of the result, if
-        non-zero, is the same as that of the original dividend.
+        calculating integer division as described for divide-integer,
+        rounded to precision digits if necessary.  The sign of the result,
+        if non-zero, is the same as that of the original dividend.
 
-        This operation will fail under the same conditions as integer division
-        (that is, if integer division on the same two operands would fail, the
-        remainder cannot be calculated).
+        This operation will fail under the same conditions as integer
+        division (that is, if integer division on the same two operands
+        would fail, the remainder cannot be calculated).
 
         >>> ExtendedContext.remainder(Decimal('2.1'), Decimal('3'))
         Decimal('2.1')
@@ -5306,9 +5311,9 @@ class Context(object):
         is chosen).  If the result is equal to 0 then its sign will be the
         sign of a.
 
-        This operation will fail under the same conditions as integer division
-        (that is, if integer division on the same two operands would fail, the
-        remainder cannot be calculated).
+        This operation will fail under the same conditions as integer
+        division (that is, if integer division on the same two operands
+        would fail, the remainder cannot be calculated).
 
         >>> ExtendedContext.remainder_near(Decimal('2.1'), Decimal('3'))
         Decimal('-0.9')
@@ -5366,8 +5371,8 @@ class Context(object):
     def same_quantum(self, a, b):
         """Returns True if the two operands have the same exponent.
 
-        The result is never affected by either the sign or the coefficient of
-        either operand.
+        The result is never affected by either the sign or the coefficient
+        of either operand.
 
         >>> ExtendedContext.same_quantum(Decimal('2.17'), Decimal('0.001'))
         False
@@ -5439,8 +5444,8 @@ class Context(object):
     def sqrt(self, a):
         """Square root of a non-negative number to context precision.
 
-        If the result must be inexact, it is rounded using the round-half-even
-        algorithm.
+        If the result must be inexact, it is rounded using the
+        round-half-even algorithm.
 
         >>> ExtendedContext.sqrt(Decimal('0'))
         Decimal('0')

Lib/_pyio.py

@@ -83,27 +83,28 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     wrapped. (If a file descriptor is given, it is closed when the
     returned I/O object is closed, unless closefd is set to False.)
 
-    mode is an optional string that specifies the mode in which the file is
-    opened. It defaults to 'r' which means open for reading in text mode. Other
-    common values are 'w' for writing (truncating the file if it already
-    exists), 'x' for exclusive creation of a new file, and 'a' for appending
-    (which on some Unix systems, means that all writes append to the end of the
-    file regardless of the current seek position). In text mode, if encoding is
-    not specified the encoding used is platform dependent. (For reading and
-    writing raw bytes use binary mode and leave encoding unspecified.) The
-    available modes are:
+    mode is an optional string that specifies the mode in which the file
+    is opened.  It defaults to 'r' which means open for reading in text
+    mode.  Other common values are 'w' for writing (truncating the file if
+    it already exists), 'x' for exclusive creation of a new file, and
+    'a' for appending (which on some Unix systems, means that all writes
+    append to the end of the file regardless of the current seek position).
+    In text mode, if encoding is not specified the encoding used is platform
+    dependent.  (For reading and writing raw bytes use binary mode and leave
+    encoding unspecified.)  The available modes are:
 
-    ========= ===============================================================
+    ========= ==========================================================
     Character Meaning
-    --------- ---------------------------------------------------------------
+    --------- ----------------------------------------------------------
     'r'       open for reading (default)
     'w'       open for writing, truncating the file first
     'x'       create a new file and open it for writing
-    'a'       open for writing, appending to the end of the file if it exists
+    'a'       open for writing, appending to the end of the file if it
+              exists
     'b'       binary mode
     't'       text mode (default)
     '+'       open a disk file for updating (reading and writing)
-    ========= ===============================================================
+    ========= ==========================================================
 
     The default mode is 'rt' (open for reading text). For binary random
     access, the mode 'w+b' opens and truncates the file to 0 bytes, while
@@ -111,23 +112,23 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     raises an `FileExistsError` if the file already exists.
 
     Python distinguishes between files opened in binary and text modes,
-    even when the underlying operating system doesn't. Files opened in
+    even when the underlying operating system doesn't.  Files opened in
     binary mode (appending 'b' to the mode argument) return contents as
-    bytes objects without any decoding. In text mode (the default, or when
+    bytes objects without any decoding.  In text mode (the default, or when
     't' is appended to the mode argument), the contents of the file are
     returned as strings, the bytes having been first decoded using a
     platform-dependent encoding or using the specified encoding if given.
 
     buffering is an optional integer used to set the buffering policy.
-    Pass 0 to switch buffering off (only allowed in binary mode), 1 to select
-    line buffering (only usable in text mode), and an integer > 1 to indicate
-    the size of a fixed-size chunk buffer.  When no buffering argument is
-    given, the default buffering policy works as follows:
+    Pass 0 to switch buffering off (only allowed in binary mode), 1 to
+    select line buffering (only usable in text mode), and an integer > 1 to
+    indicate the size of a fixed-size chunk buffer.  When no buffering
+    argument is given, the default buffering policy works as follows:
 
-   * Binary files are buffered in fixed-size chunks; the size of the buffer
-     is max(min(blocksize, 8 MiB), DEFAULT_BUFFER_SIZE)
-     when the device block size is available.
-     On most systems, the buffer will typically be 128 kilobytes long.
+    * Binary files are buffered in fixed-size chunks; the size of the buffer
+      is max(min(blocksize, 8 MiB), DEFAULT_BUFFER_SIZE) when the device
+      block size is available.
+      On most systems, the buffer will typically be 128 kilobytes long.
 
     * "Interactive" text files (files for which isatty() returns True)
       use line buffering.  Other text files use the policy described above
@@ -147,8 +148,8 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
     encoding error strings.
 
     newline is a string controlling how universal newlines works (it only
-    applies to text mode). It can be None, '', '\n', '\r', and '\r\n'.  It works
-    as follows:
+    applies to text mode). It can be None, '', '\n', '\r', and '\r\n'.  It
+    works as follows:
 
     * On input, if newline is None, universal newlines mode is
       enabled. Lines in the input can end in '\n', '\r', or '\r\n', and
@@ -164,17 +165,17 @@ def open(file, mode="r", buffering=-1, encoding=None, errors=None,
       other legal values, any '\n' characters written are translated to
       the given string.
 
-    closedfd is a bool. If closefd is False, the underlying file descriptor will
-    be kept open when the file is closed. This does not work when a file name is
-    given and must be True in that case.
+    closedfd is a bool. If closefd is False, the underlying file descriptor
+    will be kept open when the file is closed. This does not work when
+    a file name is given and must be True in that case.
 
     The newly created file is non-inheritable.
 
     A custom opener can be used by passing a callable as *opener*. The
-    underlying file descriptor for the file object is then obtained by calling
-    *opener* with (*file*, *flags*). *opener* must return an open file
-    descriptor (passing os.open as *opener* results in functionality similar to
-    passing None).
+    underlying file descriptor for the file object is then obtained by
+    calling *opener* with (*file*, *flags*).  *opener* must return an open
+    file descriptor (passing os.open as *opener* results in functionality
+    similar to passing None).
 
     open() returns a file object whose type depends on the mode, and
     through which the standard file operations such as reading and writing
@@ -351,10 +352,12 @@ class IOBase(metaclass=abc.ABCMeta):
         interpreted relative to the position indicated by whence.  Values
         for whence are ints:
 
-        * 0 -- start of stream (the default); offset should be zero or positive
+        * 0 -- start of stream (the default); offset should be zero or
+          positive
         * 1 -- current stream position; offset may be negative
         * 2 -- end of stream; offset is usually negative
-        Some operating systems / file systems could provide additional values.
+        Some operating systems / file systems could provide additional
+        values.
 
         Return an int indicating the new absolute position.
         """
@@ -367,8 +370,8 @@ class IOBase(metaclass=abc.ABCMeta):
     def truncate(self, pos=None):
         """Truncate file to size bytes.
 
-        Size defaults to the current IO position as reported by tell().  Return
-        the new size.
+        Size defaults to the current IO position as reported by tell().
+        Return the new size.
         """
         self._unsupported("truncate")
 
@@ -492,7 +495,8 @@ class IOBase(metaclass=abc.ABCMeta):
     def fileno(self):
         """Returns underlying file descriptor (an int) if one exists.
 
-        An OSError is raised if the IO object does not use a file descriptor.
+        An OSError is raised if the IO object does not use a file
+        descriptor.
         """
         self._unsupported("fileno")
 
@@ -1485,17 +1489,22 @@ class FileIO(RawIOBase):
     _closefd = True
 
     def __init__(self, file, mode='r', closefd=True, opener=None):
-        """Open a file.  The mode can be 'r' (default), 'w', 'x' or 'a' for reading,
-        writing, exclusive creation or appending.  The file will be created if it
-        doesn't exist when opened for writing or appending; it will be truncated
-        when opened for writing.  A FileExistsError will be raised if it already
-        exists when opened for creating. Opening a file for creating implies
-        writing so this mode behaves in a similar way to 'w'. Add a '+' to the mode
-        to allow simultaneous reading and writing. A custom opener can be used by
-        passing a callable as *opener*. The underlying file descriptor for the file
-        object is then obtained by calling opener with (*name*, *flags*).
-        *opener* must return an open file descriptor (passing os.open as *opener*
-        results in functionality similar to passing None).
+        """Open a file.
+
+        The mode can be 'r' (default), 'w', 'x' or 'a' for reading,
+        writing, exclusive creation or appending.  The file will be created
+        if it doesn't exist when opened for writing or appending; it will be
+        truncated when opened for writing.  A FileExistsError will be raised
+        if it already exists when opened for creating.  Opening a file for
+        creating implies writing so this mode behaves in a similar way to
+        'w'.  Add a '+' to the mode to allow simultaneous reading and
+        writing.
+
+        A custom opener can be used by passing a callable as *opener*.
+        The underlying file descriptor for the file object is then obtained
+        by calling opener with (*name*, *flags*).  *opener* must return
+        an open file descriptor (passing os.open as *opener* results in
+        functionality similar to passing None).
         """
         if self._fd >= 0:
             # Have to close the existing file first.
@@ -1733,8 +1742,8 @@ class FileIO(RawIOBase):
         """Write bytes b to file, return number written.
 
         Only makes one system call, so not all of the data may be written.
-        The number of bytes actually written is returned.  In non-blocking mode,
-        returns None if the write would block.
+        The number of bytes actually written is returned.  In non-blocking
+        mode, returns None if the write would block.
         """
         self._checkClosed()
         self._checkWritable()
@@ -1746,11 +1755,12 @@ class FileIO(RawIOBase):
     def seek(self, pos, whence=SEEK_SET):
         """Move to new file position.
 
-        Argument offset is a byte count.  Optional argument whence defaults to
-        SEEK_SET or 0 (offset from start of file, offset should be >= 0); other values
-        are SEEK_CUR or 1 (move relative to current position, positive or negative),
-        and SEEK_END or 2 (move relative to end of file, usually negative, although
-        many platforms allow seeking beyond the end of a file).
+        Argument offset is a byte count.  Optional argument whence defaults
+        to SEEK_SET or 0 (offset from start of file, offset should be >= 0);
+        other values are SEEK_CUR or 1 (move relative to current position,
+        positive or negative), and SEEK_END or 2 (move relative to end of
+        file, usually negative, although many platforms allow seeking beyond
+        the end of a file).
 
         Note that not all file objects are seekable.
         """
@@ -1783,8 +1793,8 @@ class FileIO(RawIOBase):
     def close(self):
         """Close the file.
 
-        A closed file cannot be used for further I/O operations.  close() may be
-        called more than once without error.
+        A closed file cannot be used for further I/O operations.
+        close() may be called more than once without error.
         """
         if not self.closed:
             self._stat_atopen = None
@@ -1879,8 +1889,8 @@ class TextIOBase(IOBase):
     def read(self, size=-1):
         """Read at most size characters from stream, where size is an int.
 
-        Read from underlying buffer until we have size characters or we hit EOF.
-        If size is negative or omitted, read until EOF.
+        Read from underlying buffer until we have size characters or we hit
+        EOF.  If size is negative or omitted, read until EOF.
 
         Returns a string.
         """

Lib/_pyrepl/__main__.py

@@ -1,6 +1,10 @@
 # Important: don't add things to this module, as they will end up in the REPL's
 # default globals.  Use _pyrepl.main instead.
 
+# Avoid caching this file by linecache and incorrectly report tracebacks.
+# See https://github.com/python/cpython/issues/129098.
+__spec__ = __loader__ = None
+
 if __name__ == "__main__":
     from .main import interactive_console as __pyrepl_interactive_console
     __pyrepl_interactive_console()

Lib/_pyrepl/_minimal_curses.py

@@ -1,68 +0,0 @@
-"""Minimal '_curses' module, the low-level interface for curses module
-which is not meant to be used directly.
-
-Based on ctypes.  It's too incomplete to be really called '_curses', so
-to use it, you have to import it and stick it in sys.modules['_curses']
-manually.
-
-Note that there is also a built-in module _minimal_curses which will
-hide this one if compiled in.
-"""
-
-import ctypes
-import ctypes.util
-
-
-class error(Exception):
-    pass
-
-
-def _find_clib() -> str:
-    trylibs = ["ncursesw", "ncurses", "curses"]
-
-    for lib in trylibs:
-        path = ctypes.util.find_library(lib)
-        if path:
-            return path
-    raise ModuleNotFoundError("curses library not found", name="_pyrepl._minimal_curses")
-
-
-_clibpath = _find_clib()
-clib = ctypes.cdll.LoadLibrary(_clibpath)
-
-clib.setupterm.argtypes = [ctypes.c_char_p, ctypes.c_int, ctypes.POINTER(ctypes.c_int)]
-clib.setupterm.restype = ctypes.c_int
-
-clib.tigetstr.argtypes = [ctypes.c_char_p]
-clib.tigetstr.restype = ctypes.c_ssize_t
-
-clib.tparm.argtypes = [ctypes.c_char_p] + 9 * [ctypes.c_int]  # type: ignore[operator]
-clib.tparm.restype = ctypes.c_char_p
-
-OK = 0
-ERR = -1
-
-# ____________________________________________________________
-
-
-def setupterm(termstr, fd):
-    err = ctypes.c_int(0)
-    result = clib.setupterm(termstr, fd, ctypes.byref(err))
-    if result == ERR:
-        raise error("setupterm() failed (err=%d)" % err.value)
-
-
-def tigetstr(cap):
-    if not isinstance(cap, bytes):
-        cap = cap.encode("ascii")
-    result = clib.tigetstr(cap)
-    if result == ERR:
-        return None
-    return ctypes.cast(result, ctypes.c_char_p).value
-
-
-def tparm(str, i1=0, i2=0, i3=0, i4=0, i5=0, i6=0, i7=0, i8=0, i9=0):
-    result = clib.tparm(str, i1, i2, i3, i4, i5, i6, i7, i8, i9)
-    if result is None:
-        raise error("tparm() returned NULL")
-    return result

Lib/_pyrepl/_module_completer.py

@@ -0,0 +1,426 @@
+from __future__ import annotations
+
+import importlib
+import os
+import pkgutil
+import sys
+import token
+import tokenize
+from importlib.machinery import FileFinder
+from io import StringIO
+from contextlib import contextmanager
+from dataclasses import dataclass
+from itertools import chain
+from tokenize import TokenInfo
+
+TYPE_CHECKING = False
+
+if TYPE_CHECKING:
+    from typing import Any, Iterable, Iterator, Mapping
+
+
+HARDCODED_SUBMODULES = {
+    # Standard library submodules that are not detected by pkgutil.iter_modules
+    # but can be imported, so should be proposed in completion
+    "collections": ["abc"],
+    "os": ["path"],
+    "xml.parsers.expat": ["errors", "model"],
+}
+
+
+def make_default_module_completer() -> ModuleCompleter:
+    # Inside pyrepl, __package__ is set to None by default
+    return ModuleCompleter(namespace={'__package__': None})
+
+
+class ModuleCompleter:
+    """A completer for Python import statements.
+
+    Examples:
+        - import <tab>
+        - import foo<tab>
+        - import foo.<tab>
+        - import foo as bar, baz<tab>
+
+        - from <tab>
+        - from foo<tab>
+        - from foo import <tab>
+        - from foo import bar<tab>
+        - from foo import (bar as baz, qux<tab>
+    """
+
+    def __init__(self, namespace: Mapping[str, Any] | None = None) -> None:
+        self.namespace = namespace or {}
+        self._global_cache: list[pkgutil.ModuleInfo] = []
+        self._curr_sys_path: list[str] = sys.path[:]
+        self._stdlib_path = os.path.dirname(importlib.__path__[0])
+
+    def get_completions(self, line: str) -> list[str] | None:
+        """Return the next possible import completions for 'line'."""
+        result = ImportParser(line).parse()
+        if not result:
+            return None
+        try:
+            return self.complete(*result)
+        except Exception:
+            # Some unexpected error occurred, make it look like
+            # no completions are available
+            return []
+
+    def complete(self, from_name: str | None, name: str | None) -> list[str]:
+        if from_name is None:
+            # import x.y.z<tab>
+            assert name is not None
+            path, prefix = self.get_path_and_prefix(name)
+            modules = self.find_modules(path, prefix)
+            return [self.format_completion(path, module) for module in modules]
+
+        if name is None:
+            # from x.y.z<tab>
+            path, prefix = self.get_path_and_prefix(from_name)
+            modules = self.find_modules(path, prefix)
+            return [self.format_completion(path, module) for module in modules]
+
+        # from x.y import z<tab>
+        return self.find_modules(from_name, name)
+
+    def find_modules(self, path: str, prefix: str) -> list[str]:
+        """Find all modules under 'path' that start with 'prefix'."""
+        modules = self._find_modules(path, prefix)
+        # Filter out invalid module names
+        # (for example those containing dashes that cannot be imported with 'import')
+        return [mod for mod in modules if mod.isidentifier()]
+
+    def _find_modules(self, path: str, prefix: str) -> list[str]:
+        if not path:
+            # Top-level import (e.g. `import foo<tab>`` or `from foo<tab>`)`
+            builtin_modules = [name for name in sys.builtin_module_names
+                               if self.is_suggestion_match(name, prefix)]
+            third_party_modules = [module.name for module in self.global_cache
+                                   if self.is_suggestion_match(module.name, prefix)]
+            return sorted(builtin_modules + third_party_modules)
+
+        if path.startswith('.'):
+            # Convert relative path to absolute path
+            package = self.namespace.get('__package__', '')
+            path = self.resolve_relative_name(path, package)  # type: ignore[assignment]
+            if path is None:
+                return []
+
+        modules: Iterable[pkgutil.ModuleInfo] = self.global_cache
+        imported_module = sys.modules.get(path.split('.')[0])
+        if imported_module:
+            # Filter modules to those who name and specs match the
+            # imported module to avoid invalid suggestions
+            spec = imported_module.__spec__
+            if spec:
+                modules = [mod for mod in modules
+                           if mod.name == spec.name
+                           and mod.module_finder.find_spec(mod.name, None) == spec]
+            else:
+                modules = []
+
+        is_stdlib_import: bool | None = None
+        for segment in path.split('.'):
+            modules = [mod_info for mod_info in modules
+                       if mod_info.ispkg and mod_info.name == segment]
+            if is_stdlib_import is None:
+                # Top-level import decide if we import from stdlib or not
+                is_stdlib_import = all(
+                    self._is_stdlib_module(mod_info) for mod_info in modules
+                )
+            modules = self.iter_submodules(modules)
+
+        module_names = [module.name for module in modules]
+        if is_stdlib_import:
+            module_names.extend(HARDCODED_SUBMODULES.get(path, ()))
+        return [module_name for module_name in module_names
+                if self.is_suggestion_match(module_name, prefix)]
+
+    def _is_stdlib_module(self, module_info: pkgutil.ModuleInfo) -> bool:
+        return (isinstance(module_info.module_finder, FileFinder)
+                and module_info.module_finder.path == self._stdlib_path)
+
+    def is_suggestion_match(self, module_name: str, prefix: str) -> bool:
+        if prefix:
+            return module_name.startswith(prefix)
+        # For consistency with attribute completion, which
+        # does not suggest private attributes unless requested.
+        return not module_name.startswith("_")
+
+    def iter_submodules(self, parent_modules: list[pkgutil.ModuleInfo]) -> Iterator[pkgutil.ModuleInfo]:
+        """Iterate over all submodules of the given parent modules."""
+        specs = [info.module_finder.find_spec(info.name, None)
+                 for info in parent_modules if info.ispkg]
+        search_locations = set(chain.from_iterable(
+            getattr(spec, 'submodule_search_locations', [])
+            for spec in specs if spec
+        ))
+        return pkgutil.iter_modules(search_locations)
+
+    def get_path_and_prefix(self, dotted_name: str) -> tuple[str, str]:
+        """
+        Split a dotted name into an import path and a
+        final prefix that is to be completed.
+
+        Examples:
+            'foo.bar' -> 'foo', 'bar'
+            'foo.' -> 'foo', ''
+            '.foo' -> '.', 'foo'
+        """
+        if '.' not in dotted_name:
+            return '', dotted_name
+        if dotted_name.startswith('.'):
+            stripped = dotted_name.lstrip('.')
+            dots = '.' * (len(dotted_name) - len(stripped))
+            if '.' not in stripped:
+                return dots, stripped
+            path, prefix = stripped.rsplit('.', 1)
+            return dots + path, prefix
+        path, prefix = dotted_name.rsplit('.', 1)
+        return path, prefix
+
+    def format_completion(self, path: str, module: str) -> str:
+        if path == '' or path.endswith('.'):
+            return f'{path}{module}'
+        return f'{path}.{module}'
+
+    def resolve_relative_name(self, name: str, package: str) -> str | None:
+        """Resolve a relative module name to an absolute name.
+
+        Example: resolve_relative_name('.foo', 'bar') -> 'bar.foo'
+        """
+        # taken from importlib._bootstrap
+        level = 0
+        for character in name:
+            if character != '.':
+                break
+            level += 1
+        bits = package.rsplit('.', level - 1)
+        if len(bits) < level:
+            return None
+        base = bits[0]
+        name = name[level:]
+        return f'{base}.{name}' if name else base
+
+    @property
+    def global_cache(self) -> list[pkgutil.ModuleInfo]:
+        """Global module cache"""
+        if not self._global_cache or self._curr_sys_path != sys.path:
+            self._curr_sys_path = sys.path[:]
+            self._global_cache = list(pkgutil.iter_modules())
+        return self._global_cache
+
+
+class ImportParser:
+    """
+    Parses incomplete import statements that are
+    suitable for autocomplete suggestions.
+
+    Examples:
+        - import foo          -> Result(from_name=None, name='foo')
+        - import foo.         -> Result(from_name=None, name='foo.')
+        - from foo            -> Result(from_name='foo', name=None)
+        - from foo import bar -> Result(from_name='foo', name='bar')
+        - from .foo import (  -> Result(from_name='.foo', name='')
+
+    Note that the parser works in reverse order, starting from the
+    last token in the input string. This makes the parser more robust
+    when parsing multiple statements.
+    """
+    _ignored_tokens = {
+        token.INDENT, token.DEDENT, token.COMMENT,
+        token.NL, token.NEWLINE, token.ENDMARKER
+    }
+    _keywords = {'import', 'from', 'as'}
+
+    def __init__(self, code: str) -> None:
+        self.code = code
+        tokens = []
+        try:
+            for t in tokenize.generate_tokens(StringIO(code).readline):
+                if t.type not in self._ignored_tokens:
+                    tokens.append(t)
+        except tokenize.TokenError as e:
+            if 'unexpected EOF' not in str(e):
+                # unexpected EOF is fine, since we're parsing an
+                # incomplete statement, but other errors are not
+                # because we may not have all the tokens so it's
+                # safer to bail out
+                tokens = []
+        except SyntaxError:
+            tokens = []
+        self.tokens = TokenQueue(tokens[::-1])
+
+    def parse(self) -> tuple[str | None, str | None] | None:
+        if not (res := self._parse()):
+            return None
+        return res.from_name, res.name
+
+    def _parse(self) -> Result | None:
+        with self.tokens.save_state():
+            return self.parse_from_import()
+        with self.tokens.save_state():
+            return self.parse_import()
+
+    def parse_import(self) -> Result:
+        if self.code.rstrip().endswith('import') and self.code.endswith(' '):
+            return Result(name='')
+        if self.tokens.peek_string(','):
+            name = ''
+        else:
+            if self.code.endswith(' '):
+                raise ParseError('parse_import')
+            name = self.parse_dotted_name()
+        if name.startswith('.'):
+            raise ParseError('parse_import')
+        while self.tokens.peek_string(','):
+            self.tokens.pop()
+            self.parse_dotted_as_name()
+        if self.tokens.peek_string('import'):
+            return Result(name=name)
+        raise ParseError('parse_import')
+
+    def parse_from_import(self) -> Result:
+        stripped = self.code.rstrip()
+        if stripped.endswith('import') and self.code.endswith(' '):
+            return Result(from_name=self.parse_empty_from_import(), name='')
+        if stripped.endswith('from') and self.code.endswith(' '):
+            return Result(from_name='')
+        if self.tokens.peek_string('(') or self.tokens.peek_string(','):
+            return Result(from_name=self.parse_empty_from_import(), name='')
+        if self.code.endswith(' '):
+            raise ParseError('parse_from_import')
+        name = self.parse_dotted_name()
+        if '.' in name:
+            self.tokens.pop_string('from')
+            return Result(from_name=name)
+        if self.tokens.peek_string('from'):
+            return Result(from_name=name)
+        from_name = self.parse_empty_from_import()
+        return Result(from_name=from_name, name=name)
+
+    def parse_empty_from_import(self) -> str:
+        if self.tokens.peek_string(','):
+            self.tokens.pop()
+            self.parse_as_names()
+        if self.tokens.peek_string('('):
+            self.tokens.pop()
+        self.tokens.pop_string('import')
+        return self.parse_from()
+
+    def parse_from(self) -> str:
+        from_name = self.parse_dotted_name()
+        self.tokens.pop_string('from')
+        return from_name
+
+    def parse_dotted_as_name(self) -> str:
+        self.tokens.pop_name()
+        if self.tokens.peek_string('as'):
+            self.tokens.pop()
+        with self.tokens.save_state():
+            return self.parse_dotted_name()
+
+    def parse_dotted_name(self) -> str:
+        name = []
+        if self.tokens.peek_string('.'):
+            name.append('.')
+            self.tokens.pop()
+        if (self.tokens.peek_name()
+            and (tok := self.tokens.peek())
+            and tok.string not in self._keywords):
+            name.append(self.tokens.pop_name())
+        if not name:
+            raise ParseError('parse_dotted_name')
+        while self.tokens.peek_string('.'):
+            name.append('.')
+            self.tokens.pop()
+            if (self.tokens.peek_name()
+                and (tok := self.tokens.peek())
+                and tok.string not in self._keywords):
+                name.append(self.tokens.pop_name())
+            else:
+                break
+
+        while self.tokens.peek_string('.'):
+            name.append('.')
+            self.tokens.pop()
+        return ''.join(name[::-1])
+
+    def parse_as_names(self) -> None:
+        self.parse_as_name()
+        while self.tokens.peek_string(','):
+            self.tokens.pop()
+            self.parse_as_name()
+
+    def parse_as_name(self) -> None:
+        self.tokens.pop_name()
+        if self.tokens.peek_string('as'):
+            self.tokens.pop()
+            self.tokens.pop_name()
+
+
+class ParseError(Exception):
+    pass
+
+
+@dataclass(frozen=True)
+class Result:
+    from_name: str | None = None
+    name: str | None = None
+
+
+class TokenQueue:
+    """Provides helper functions for working with a sequence of tokens."""
+
+    def __init__(self, tokens: list[TokenInfo]) -> None:
+        self.tokens: list[TokenInfo] = tokens
+        self.index: int = 0
+        self.stack: list[int] = []
+
+    @contextmanager
+    def save_state(self) -> Any:
+        try:
+            self.stack.append(self.index)
+            yield
+        except ParseError:
+            self.index = self.stack.pop()
+        else:
+            self.stack.pop()
+
+    def __bool__(self) -> bool:
+        return self.index < len(self.tokens)
+
+    def peek(self) -> TokenInfo | None:
+        if not self:
+            return None
+        return self.tokens[self.index]
+
+    def peek_name(self) -> bool:
+        if not (tok := self.peek()):
+            return False
+        return tok.type == token.NAME
+
+    def pop_name(self) -> str:
+        tok = self.pop()
+        if tok.type != token.NAME:
+            raise ParseError('pop_name')
+        return tok.string
+
+    def peek_string(self, string: str) -> bool:
+        if not (tok := self.peek()):
+            return False
+        return tok.string == string
+
+    def pop_string(self, string: str) -> str:
+        tok = self.pop()
+        if tok.string != string:
+            raise ParseError('pop_string')
+        return tok.string
+
+    def pop(self) -> TokenInfo:
+        if not self:
+            raise ParseError('pop')
+        tok = self.tokens[self.index]
+        self.index += 1
+        return tok

Lib/_pyrepl/base_eventqueue.py

@@ -0,0 +1,110 @@
+#   Copyright 2000-2008 Michael Hudson-Doyle <micahel@gmail.com>
+#                       Armin Rigo
+#
+#                        All Rights Reserved
+#
+#
+# Permission to use, copy, modify, and distribute this software and
+# its documentation for any purpose is hereby granted without fee,
+# provided that the above copyright notice appear in all copies and
+# that both that copyright notice and this permission notice appear in
+# supporting documentation.
+#
+# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
+# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
+# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
+# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
+# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
+# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
+# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
+
+"""
+OS-independent base for an event and VT sequence scanner
+
+See unix_eventqueue and windows_eventqueue for subclasses.
+"""
+
+from collections import deque
+
+from . import keymap
+from .console import Event
+from .trace import trace
+
+class BaseEventQueue:
+    def __init__(self, encoding: str, keymap_dict: dict[bytes, str]) -> None:
+        self.compiled_keymap = keymap.compile_keymap(keymap_dict)
+        self.keymap = self.compiled_keymap
+        trace("keymap {k!r}", k=self.keymap)
+        self.encoding = encoding
+        self.events: deque[Event] = deque()
+        self.buf = bytearray()
+
+    def get(self) -> Event | None:
+        """
+        Retrieves the next event from the queue.
+        """
+        if self.events:
+            return self.events.popleft()
+        else:
+            return None
+
+    def empty(self) -> bool:
+        """
+        Checks if the queue is empty.
+        """
+        return not self.events
+
+    def flush_buf(self) -> bytearray:
+        """
+        Flushes the buffer and returns its contents.
+        """
+        old = self.buf
+        self.buf = bytearray()
+        return old
+
+    def insert(self, event: Event) -> None:
+        """
+        Inserts an event into the queue.
+        """
+        trace('added event {event}', event=event)
+        self.events.append(event)
+
+    def push(self, char: int | bytes) -> None:
+        """
+        Processes a character by updating the buffer and handling special key mappings.
+        """
+        assert isinstance(char, (int, bytes))
+        ord_char = char if isinstance(char, int) else ord(char)
+        char = ord_char.to_bytes()
+        self.buf.append(ord_char)
+
+        if char in self.keymap:
+            if self.keymap is self.compiled_keymap:
+                # sanity check, buffer is empty when a special key comes
+                assert len(self.buf) == 1
+            k = self.keymap[char]
+            trace('found map {k!r}', k=k)
+            if isinstance(k, dict):
+                self.keymap = k
+            else:
+                self.insert(Event('key', k, bytes(self.flush_buf())))
+                self.keymap = self.compiled_keymap
+
+        elif self.buf and self.buf[0] == 27:  # escape
+            # escape sequence not recognized by our keymap: propagate it
+            # outside so that i can be recognized as an M-... key (see also
+            # the docstring in keymap.py
+            trace('unrecognized escape sequence, propagating...')
+            self.keymap = self.compiled_keymap
+            self.insert(Event('key', '\033', b'\033'))
+            for _c in self.flush_buf()[1:]:
+                self.push(_c)
+
+        else:
+            try:
+                decoded = bytes(self.buf).decode(self.encoding)
+            except UnicodeError:
+                return
+            else:
+                self.insert(Event('key', decoded, bytes(self.flush_buf())))
+            self.keymap = self.compiled_keymap

Lib/_pyrepl/commands.py

@@ -21,6 +21,7 @@
 
 from __future__ import annotations
 import os
+import time
 
 # Categories of actions:
 #  killing
@@ -31,6 +32,7 @@ import os
 #  finishing
 # [completion]
 
+from .trace import trace
 
 # types
 if False:
@@ -368,6 +370,13 @@ class self_insert(EditCommand):
         r = self.reader
         text = self.event * r.get_arg()
         r.insert(text)
+        if r.paste_mode:
+            data = ""
+            ev = r.console.getpending()
+            data += ev.data
+            if data:
+                r.insert(data)
+                r.last_refresh_cache.invalidated = True
 
 
 class insert_nl(EditCommand):
@@ -411,14 +420,17 @@ class delete(EditCommand):
     def do(self) -> None:
         r = self.reader
         b = r.buffer
-        if (
-            r.pos == 0
-            and len(b) == 0  # this is something of a hack
-            and self.event[-1] == "\004"
-        ):
-            r.update_screen()
-            r.console.finish()
-            raise EOFError
+        if self.event[-1] == "\004":
+            if b and b[-1].endswith("\n"):
+                self.finish = True
+            elif (
+                r.pos == 0
+                and len(b) == 0  # this is something of a hack
+            ):
+                r.update_screen()
+                r.console.finish()
+                raise EOFError
+
         for i in range(r.get_arg()):
             if r.pos != len(b):
                 del b[r.pos]
@@ -437,7 +449,7 @@ class help(Command):
         import _sitebuiltins
 
         with self.reader.suspend():
-            self.reader.msg = _sitebuiltins._Helper()()  # type: ignore[assignment, call-arg]
+            self.reader.msg = _sitebuiltins._Helper()()  # type: ignore[assignment]
 
 
 class invalid_key(Command):
@@ -456,7 +468,7 @@ class invalid_command(Command):
 class show_history(Command):
     def do(self) -> None:
         from .pager import get_pager
-        from site import gethistoryfile  # type: ignore[attr-defined]
+        from site import gethistoryfile
 
         history = os.linesep.join(self.reader.history[:])
         self.reader.console.restore()
@@ -471,19 +483,23 @@ class show_history(Command):
 
 
 class paste_mode(Command):
-
     def do(self) -> None:
         self.reader.paste_mode = not self.reader.paste_mode
         self.reader.dirty = True
 
 
-class enable_bracketed_paste(Command):
+class perform_bracketed_paste(Command):
     def do(self) -> None:
-        self.reader.paste_mode = True
-        self.reader.in_bracketed_paste = True
-
-class disable_bracketed_paste(Command):
-    def do(self) -> None:
-        self.reader.paste_mode = False
-        self.reader.in_bracketed_paste = False
-        self.reader.dirty = True
+        done = "\x1b[201~"
+        data = ""
+        start = time.time()
+        while done not in data:
+            ev = self.reader.console.getpending()
+            data += ev.data
+        trace(
+            "bracketed pasting of {l} chars done in {s:.2f}s",
+            l=len(data),
+            s=time.time() - start,
+        )
+        self.reader.insert(data.replace(done, ""))
+        self.reader.last_refresh_cache.invalidated = True

Lib/_pyrepl/completing_reader.py

@@ -91,7 +91,7 @@ def build_menu(
         #          D E F                       B E
         #          G                           C F
         #
-        # "fill" the table with empty words, so we always have the same amout
+        # "fill" the table with empty words, so we always have the same amount
         # of rows for each column
         missing = cols*rows - len(wordlist)
         wordlist = wordlist + ['']*missing
@@ -293,3 +293,7 @@ class CompletingReader(Reader):
 
     def get_completions(self, stem: str) -> list[str]:
         return []
+
+    def get_line(self) -> str:
+        """Return the current line until the cursor position."""
+        return ''.join(self.buffer[:self.pos])

Lib/_pyrepl/console.py

@@ -19,11 +19,12 @@
 
 from __future__ import annotations
 
-import _colorize  # type: ignore[import-not-found]
+import _colorize
 
 from abc import ABC, abstractmethod
 import ast
 import code
+import linecache
 from dataclasses import dataclass, field
 import os.path
 import sys
@@ -152,6 +153,8 @@ class Console(ABC):
 
 
 class InteractiveColoredConsole(code.InteractiveConsole):
+    STATEMENT_FAILED = object()
+
     def __init__(
         self,
         locals: dict[str, object] | None = None,
@@ -159,7 +162,7 @@ class InteractiveColoredConsole(code.InteractiveConsole):
         *,
         local_exit: bool = False,
     ) -> None:
-        super().__init__(locals=locals, filename=filename, local_exit=local_exit)  # type: ignore[call-arg]
+        super().__init__(locals=locals, filename=filename, local_exit=local_exit)
         self.can_colorize = _colorize.can_colorize()
 
     def showsyntaxerror(self, filename=None, **kwargs):
@@ -173,6 +176,16 @@ class InteractiveColoredConsole(code.InteractiveConsole):
                 limit=traceback.BUILTIN_EXCEPTION_LIMIT)
         self.write(''.join(lines))
 
+    def runcode(self, code):
+        try:
+            exec(code, self.locals)
+        except SystemExit:
+            raise
+        except BaseException:
+            self.showtraceback()
+            return self.STATEMENT_FAILED
+        return None
+
     def runsource(self, source, filename="<input>", symbol="single"):
         try:
             tree = self.compile.compiler(
@@ -193,6 +206,7 @@ class InteractiveColoredConsole(code.InteractiveConsole):
             item = wrapper([stmt])
             try:
                 code = self.compile.compiler(item, filename, the_symbol)
+                linecache._register_code(code, source, filename)
             except SyntaxError as e:
                 if e.args[0] == "'await' outside function":
                     python = os.path.basename(sys.executable)
@@ -209,5 +223,7 @@ class InteractiveColoredConsole(code.InteractiveConsole):
             if code is None:
                 return True
 
-            self.runcode(code)
+            result = self.runcode(code)
+            if result is self.STATEMENT_FAILED:
+                break
         return False

Lib/_pyrepl/curses.py

@@ -1,33 +0,0 @@
-#   Copyright 2000-2010 Michael Hudson-Doyle <micahel@gmail.com>
-#                       Armin Rigo
-#
-#                        All Rights Reserved
-#
-#
-# Permission to use, copy, modify, and distribute this software and
-# its documentation for any purpose is hereby granted without fee,
-# provided that the above copyright notice appear in all copies and
-# that both that copyright notice and this permission notice appear in
-# supporting documentation.
-#
-# THE AUTHOR MICHAEL HUDSON DISCLAIMS ALL WARRANTIES WITH REGARD TO
-# THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY
-# AND FITNESS, IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL,
-# INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER
-# RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF
-# CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
-# CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
-
-
-try:
-    import _curses
-except ImportError:
-    try:
-        import curses as _curses  # type: ignore[no-redef]
-    except ImportError:
-        from . import _minimal_curses as _curses  # type: ignore[no-redef]
-
-setupterm = _curses.setupterm
-tigetstr = _curses.tigetstr
-tparm = _curses.tparm
-error = _curses.error

Lib/_pyrepl/fancy_termios.py

@@ -20,19 +20,25 @@
 import termios
 
 
-class TermState:
-    def __init__(self, tuples):
-        (
-            self.iflag,
-            self.oflag,
-            self.cflag,
-            self.lflag,
-            self.ispeed,
-            self.ospeed,
-            self.cc,
-        ) = tuples
+TYPE_CHECKING = False
 
-    def as_list(self):
+if TYPE_CHECKING:
+    from typing import cast
+else:
+    cast = lambda typ, val: val
+
+
+class TermState:
+    def __init__(self, attrs: list[int | list[bytes]]) -> None:
+        self.iflag = cast(int, attrs[0])
+        self.oflag = cast(int, attrs[1])
+        self.cflag = cast(int, attrs[2])
+        self.lflag = cast(int, attrs[3])
+        self.ispeed = cast(int, attrs[4])
+        self.ospeed = cast(int, attrs[5])
+        self.cc = cast(list[bytes], attrs[6])
+
+    def as_list(self) -> list[int | list[bytes]]:
         return [
             self.iflag,
             self.oflag,
@@ -45,32 +51,32 @@ class TermState:
             self.cc[:],
         ]
 
-    def copy(self):
+    def copy(self) -> "TermState":
         return self.__class__(self.as_list())
 
 
-def tcgetattr(fd):
+def tcgetattr(fd: int) -> TermState:
     return TermState(termios.tcgetattr(fd))
 
 
-def tcsetattr(fd, when, attrs):
+def tcsetattr(fd: int, when: int, attrs: TermState) -> None:
     termios.tcsetattr(fd, when, attrs.as_list())
 
 
 class Term(TermState):
     TS__init__ = TermState.__init__
 
-    def __init__(self, fd=0):
+    def __init__(self, fd: int = 0) -> None:
         self.TS__init__(termios.tcgetattr(fd))
         self.fd = fd
-        self.stack = []
+        self.stack: list[list[int | list[bytes]]] = []
 
-    def save(self):
+    def save(self) -> None:
         self.stack.append(self.as_list())
 
-    def set(self, when=termios.TCSANOW):
+    def set(self, when: int = termios.TCSANOW) -> None:
         termios.tcsetattr(self.fd, when, self.as_list())
 
-    def restore(self):
+    def restore(self) -> None:
         self.TS__init__(self.stack.pop())
         self.set()

Lib/_pyrepl/keymap.py

@@ -30,20 +30,20 @@ sequence.
 pyrepl uses its own keyspec format that is meant to be a strict superset of
 readline's KEYSEQ format. This means that if a spec is found that readline
 accepts that this doesn't, it should be logged as a bug. Note that this means
-we're using the `\\C-o' style of readline's keyspec, not the `Control-o' sort.
+we're using the '\\C-o' style of readline's keyspec, not the 'Control-o' sort.
 
 The extension to readline is that the sequence \\<KEY> denotes the
 sequence of characters produced by hitting KEY.
 
 Examples:
-`a'      - what you get when you hit the `a' key
-`\\EOA'  - Escape - O - A (up, on my terminal)
-`\\<UP>' - the up arrow key
-`\\<up>' - ditto (keynames are case-insensitive)
-`\\C-o', `\\c-o'  - control-o
-`\\M-.'  - meta-period
-`\\E.'   - ditto (that's how meta works for pyrepl)
-`\\<tab>', `\\<TAB>', `\\t', `\\011', '\\x09', '\\X09', '\\C-i', '\\C-I'
+'a'      - what you get when you hit the 'a' key
+'\\EOA'  - Escape - O - A (up, on my terminal)
+'\\<UP>' - the up arrow key
+'\\<up>' - ditto (keynames are case-insensitive)
+'\\C-o', '\\c-o'  - control-o
+'\\M-.'  - meta-period
+'\\E.'   - ditto (that's how meta works for pyrepl)
+'\\<tab>', '\\<TAB>', '\\t', '\\011', '\\x09', '\\X09', '\\C-i', '\\C-I'
    - all of these are the tab character.
 """
 

Lib/_pyrepl/main.py

@@ -1,6 +1,7 @@
 import errno
 import os
 import sys
+import types
 
 
 CAN_USE_PYREPL: bool
@@ -29,12 +30,10 @@ def interactive_console(mainmodule=None, quiet=False, pythonstartup=False):
             print(FAIL_REASON, file=sys.stderr)
         return sys._baserepl()
 
-    if mainmodule:
-        namespace = mainmodule.__dict__
-    else:
-        import __main__
-        namespace = __main__.__dict__
-        namespace.pop("__pyrepl_interactive_console", None)
+    if not mainmodule:
+        mainmodule = types.ModuleType("__main__")
+
+    namespace = mainmodule.__dict__
 
     # sys._baserepl() above does this internally, we do it here
     startup_path = os.getenv("PYTHONSTARTUP")

Lib/_pyrepl/mypy.ini

@@ -4,8 +4,9 @@
 
 [mypy]
 files = Lib/_pyrepl
+mypy_path = $MYPY_CONFIG_FILE_DIR/../../Misc/mypy
 explicit_package_bases = True
-python_version = 3.12
+python_version = 3.13
 platform = linux
 pretty = True
 

Lib/_pyrepl/reader.py

@@ -22,15 +22,13 @@
 from __future__ import annotations
 
 import sys
+import _colorize
 
 from contextlib import contextmanager
 from dataclasses import dataclass, field, fields
-import unicodedata
-from _colorize import can_colorize, ANSIColors  # type: ignore[import-not-found]
-
 
 from . import commands, console, input
-from .utils import ANSI_ESCAPE_SEQUENCE, wlen, str_width
+from .utils import wlen, unbracket, disp_str, gen_colors, THEME
 from .trace import trace
 
 
@@ -39,38 +37,7 @@ Command = commands.Command
 from .types import Callback, SimpleContextManager, KeySpec, CommandName
 
 
-def disp_str(buffer: str) -> tuple[str, list[int]]:
-    """disp_str(buffer:string) -> (string, [int])
-
-    Return the string that should be the printed representation of
-    |buffer| and a list detailing where the characters of |buffer|
-    get used up.  E.g.:
-
-    >>> disp_str(chr(3))
-    ('^C', [1, 0])
-
-    """
-    b: list[int] = []
-    s: list[str] = []
-    for c in buffer:
-        if c == '\x1a':
-            s.append(c)
-            b.append(2)
-        elif ord(c) < 128:
-            s.append(c)
-            b.append(1)
-        elif unicodedata.category(c).startswith("C"):
-            c = r"\u%04x" % ord(c)
-            s.append(c)
-            b.extend([0] * (len(c) - 1))
-        else:
-            s.append(c)
-            b.append(str_width(c))
-    return "".join(s), b
-
-
-# syntax classes:
-
+# syntax classes
 SYNTAX_WHITESPACE, SYNTAX_WORD, SYNTAX_SYMBOL = range(3)
 
 
@@ -136,8 +103,7 @@ default_keymap: tuple[tuple[KeySpec, CommandName], ...] = tuple(
         (r"\M-9", "digit-arg"),
         (r"\M-\n", "accept"),
         ("\\\\", "self-insert"),
-        (r"\x1b[200~", "enable_bracketed_paste"),
-        (r"\x1b[201~", "disable_bracketed_paste"),
+        (r"\x1b[200~", "perform-bracketed-paste"),
         (r"\x03", "ctrl-c"),
     ]
     + [(c, "self-insert") for c in map(chr, range(32, 127)) if c != "\\"]
@@ -175,20 +141,21 @@ class Reader:
     Instance variables of note include:
 
       * buffer:
-        A *list* (*not* a string at the moment :-) containing all the
-        characters that have been entered.
+        A per-character list containing all the characters that have been
+        entered. Does not include color information.
       * console:
         Hopefully encapsulates the OS dependent stuff.
       * pos:
-        A 0-based index into `buffer' for where the insertion point
+        A 0-based index into 'buffer' for where the insertion point
         is.
       * screeninfo:
-        Ahem.  This list contains some info needed to move the
-        insertion point around reasonably efficiently.
+        A list of screen position tuples. Each list element is a tuple
+        representing information on visible line length for a given line.
+        Allows for efficient skipping of color escape sequences.
       * cxy, lxy:
         the position of the insertion point in screen ...
       * syntax_table:
-        Dictionary mapping characters to `syntax class'; read the
+        Dictionary mapping characters to 'syntax class'; read the
         emacs docs to see what this means :-)
       * commands:
         Dictionary mapping command names to command classes.
@@ -234,7 +201,6 @@ class Reader:
     dirty: bool = False
     finished: bool = False
     paste_mode: bool = False
-    in_bracketed_paste: bool = False
     commands: dict[str, type[Command]] = field(default_factory=make_default_commands)
     last_command: type[Command] | None = None
     syntax_table: dict[str, int] = field(default_factory=make_default_syntax_table)
@@ -252,7 +218,6 @@ class Reader:
     ## cached metadata to speed up screen refreshes
     @dataclass
     class RefreshCache:
-        in_bracketed_paste: bool = False
         screen: list[str] = field(default_factory=list)
         screeninfo: list[tuple[int, list[int]]] = field(init=False)
         line_end_offsets: list[int] = field(default_factory=list)
@@ -266,7 +231,6 @@ class Reader:
                          screen: list[str],
                          screeninfo: list[tuple[int, list[int]]],
             ) -> None:
-            self.in_bracketed_paste = reader.in_bracketed_paste
             self.screen = screen.copy()
             self.screeninfo = screeninfo.copy()
             self.pos = reader.pos
@@ -279,8 +243,7 @@ class Reader:
                 return False
             dimensions = reader.console.width, reader.console.height
             dimensions_changed = dimensions != self.dimensions
-            paste_changed = reader.in_bracketed_paste != self.in_bracketed_paste
-            return not (dimensions_changed or paste_changed)
+            return not dimensions_changed
 
         def get_cached_location(self, reader: Reader) -> tuple[int, int]:
             if self.invalidated:
@@ -310,7 +273,7 @@ class Reader:
         self.screeninfo = [(0, [])]
         self.cxy = self.pos2xy()
         self.lxy = (self.pos, 0)
-        self.can_colorize = can_colorize()
+        self.can_colorize = _colorize.can_colorize()
 
         self.last_refresh_cache.screeninfo = self.screeninfo
         self.last_refresh_cache.pos = self.pos
@@ -348,13 +311,17 @@ class Reader:
 
         prompt_from_cache = (offset and self.buffer[offset - 1] != "\n")
 
+        if self.can_colorize:
+            colors = list(gen_colors(self.get_unicode()))
+        else:
+            colors = None
+        trace("colors = {colors}", colors=colors)
         lines = "".join(self.buffer[offset:]).split("\n")
-
         cursor_found = False
         lines_beyond_cursor = 0
         for ln, line in enumerate(lines, num_common_lines):
-            ll = len(line)
-            if 0 <= pos <= ll:
+            line_len = len(line)
+            if 0 <= pos <= line_len:
                 self.lxy = pos, ln
                 cursor_found = True
             elif cursor_found:
@@ -368,34 +335,33 @@ class Reader:
                 prompt_from_cache = False
                 prompt = ""
             else:
-                prompt = self.get_prompt(ln, ll >= pos >= 0)
+                prompt = self.get_prompt(ln, line_len >= pos >= 0)
             while "\n" in prompt:
                 pre_prompt, _, prompt = prompt.partition("\n")
                 last_refresh_line_end_offsets.append(offset)
                 screen.append(pre_prompt)
                 screeninfo.append((0, []))
-            pos -= ll + 1
-            prompt, lp = self.process_prompt(prompt)
-            l, l2 = disp_str(line)
-            wrapcount = (wlen(l) + lp) // self.console.width
-            if wrapcount == 0:
-                offset += ll + 1  # Takes all of the line plus the newline
+            pos -= line_len + 1
+            prompt, prompt_len = self.process_prompt(prompt)
+            chars, char_widths = disp_str(line, colors, offset)
+            wrapcount = (sum(char_widths) + prompt_len) // self.console.width
+            if wrapcount == 0 or not char_widths:
+                offset += line_len + 1  # Takes all of the line plus the newline
                 last_refresh_line_end_offsets.append(offset)
-                screen.append(prompt + l)
-                screeninfo.append((lp, l2))
+                screen.append(prompt + "".join(chars))
+                screeninfo.append((prompt_len, char_widths))
             else:
-                i = 0
-                while l:
-                    prelen = lp if i == 0 else 0
+                pre = prompt
+                prelen = prompt_len
+                for wrap in range(wrapcount + 1):
                     index_to_wrap_before = 0
                     column = 0
-                    for character_width in l2:
-                        if column + character_width >= self.console.width - prelen:
+                    for char_width in char_widths:
+                        if column + char_width + prelen >= self.console.width:
                             break
                         index_to_wrap_before += 1
-                        column += character_width
-                    pre = prompt if i == 0 else ""
-                    if len(l) > index_to_wrap_before:
+                        column += char_width
+                    if len(chars) > index_to_wrap_before:
                         offset += index_to_wrap_before
                         post = "\\"
                         after = [1]
@@ -404,11 +370,14 @@ class Reader:
                         post = ""
                         after = []
                     last_refresh_line_end_offsets.append(offset)
-                    screen.append(pre + l[:index_to_wrap_before] + post)
-                    screeninfo.append((prelen, l2[:index_to_wrap_before] + after))
-                    l = l[index_to_wrap_before:]
-                    l2 = l2[index_to_wrap_before:]
-                    i += 1
+                    render = pre + "".join(chars[:index_to_wrap_before]) + post
+                    render_widths = char_widths[:index_to_wrap_before] + after
+                    screen.append(render)
+                    screeninfo.append((prelen, render_widths))
+                    chars = chars[index_to_wrap_before:]
+                    char_widths = char_widths[index_to_wrap_before:]
+                    pre = ""
+                    prelen = 0
         self.screeninfo = screeninfo
         self.cxy = self.pos2xy()
         if self.msg:
@@ -421,42 +390,15 @@ class Reader:
 
     @staticmethod
     def process_prompt(prompt: str) -> tuple[str, int]:
-        """Process the prompt.
+        r"""Return a tuple with the prompt string and its visible length.
 
-        This means calculate the length of the prompt. The character \x01
-        and \x02 are used to bracket ANSI control sequences and need to be
-        excluded from the length calculation.  So also a copy of the prompt
-        is returned with these control characters removed."""
-
-        # The logic below also ignores the length of common escape
-        # sequences if they were not explicitly within \x01...\x02.
-        # They are CSI (or ANSI) sequences  ( ESC [ ... LETTER )
-
-        # wlen from utils already excludes ANSI_ESCAPE_SEQUENCE chars,
-        # which breaks the logic below so we redefine it here.
-        def wlen(s: str) -> int:
-            return sum(str_width(i) for i in s)
-
-        out_prompt = ""
-        l = wlen(prompt)
-        pos = 0
-        while True:
-            s = prompt.find("\x01", pos)
-            if s == -1:
-                break
-            e = prompt.find("\x02", s)
-            if e == -1:
-                break
-            # Found start and end brackets, subtract from string length
-            l = l - (e - s + 1)
-            keep = prompt[pos:s]
-            l -= sum(map(wlen, ANSI_ESCAPE_SEQUENCE.findall(keep)))
-            out_prompt += keep + prompt[s + 1 : e]
-            pos = e + 1
-        keep = prompt[pos:]
-        l -= sum(map(wlen, ANSI_ESCAPE_SEQUENCE.findall(keep)))
-        out_prompt += keep
-        return out_prompt, l
+        The prompt string has the zero-width brackets recognized by shells
+        (\x01 and \x02) removed.  The length ignores anything between those
+        brackets as well as any ANSI escape sequences.
+        """
+        out_prompt = unbracket(prompt, including_content=False)
+        visible_prompt = unbracket(prompt, including_content=True)
+        return out_prompt, wlen(visible_prompt)
 
     def bow(self, p: int | None = None) -> int:
         """Return the 0-based index of the word break preceding p most
@@ -525,7 +467,7 @@ class Reader:
 
     def get_arg(self, default: int = 1) -> int:
         """Return any prefix argument that the user has supplied,
-        returning `default' if there is None.  Defaults to 1.
+        returning 'default' if there is None.  Defaults to 1.
         """
         if self.arg is None:
             return default
@@ -533,10 +475,10 @@ class Reader:
 
     def get_prompt(self, lineno: int, cursor_on_line: bool) -> str:
         """Return what should be in the left-hand margin for line
-        `lineno'."""
+        'lineno'."""
         if self.arg is not None and cursor_on_line:
             prompt = f"(arg: {self.arg}) "
-        elif self.paste_mode and not self.in_bracketed_paste:
+        elif self.paste_mode:
             prompt = "(paste) "
         elif "\n" in self.buffer:
             if lineno == 0:
@@ -549,7 +491,8 @@ class Reader:
             prompt = self.ps1
 
         if self.can_colorize:
-            prompt = f"{ANSIColors.BOLD_MAGENTA}{prompt}{ANSIColors.RESET}"
+            t = THEME()
+            prompt = f"{t.prompt}{prompt}{t.reset}"
         return prompt
 
     def push_input_trans(self, itrans: input.KeymapTranslator) -> None:
@@ -564,9 +507,9 @@ class Reader:
         pos = 0
         i = 0
         while i < y:
-            prompt_len, character_widths = self.screeninfo[i]
-            offset = len(character_widths) - character_widths.count(0)
-            in_wrapped_line = prompt_len + sum(character_widths) >= self.console.width
+            prompt_len, char_widths = self.screeninfo[i]
+            offset = len(char_widths)
+            in_wrapped_line = prompt_len + sum(char_widths) >= self.console.width
             if in_wrapped_line:
                 pos += offset - 1  # -1 cause backslash is not in buffer
             else:
@@ -577,6 +520,7 @@ class Reader:
         cur_x = self.screeninfo[i][0]
         while cur_x < x:
             if self.screeninfo[i][1][j] == 0:
+                j += 1  # prevent potential future infinite loop
                 continue
             cur_x += self.screeninfo[i][1][j]
             j += 1
@@ -586,29 +530,33 @@ class Reader:
 
     def pos2xy(self) -> tuple[int, int]:
         """Return the x, y coordinates of position 'pos'."""
-        # this *is* incomprehensible, yes.
-        p, y = 0, 0
-        l2: list[int] = []
+
+        prompt_len, y = 0, 0
+        char_widths: list[int] = []
         pos = self.pos
         assert 0 <= pos <= len(self.buffer)
+
+        # optimize for the common case: typing at the end of the buffer
         if pos == len(self.buffer) and len(self.screeninfo) > 0:
             y = len(self.screeninfo) - 1
-            p, l2 = self.screeninfo[y]
-            return p + sum(l2) + l2.count(0), y
+            prompt_len, char_widths = self.screeninfo[y]
+            return prompt_len + sum(char_widths), y
+
+        for prompt_len, char_widths in self.screeninfo:
+            offset = len(char_widths)
+            in_wrapped_line = prompt_len + sum(char_widths) >= self.console.width
+            if in_wrapped_line:
+                offset -= 1  # need to remove line-wrapping backslash
 
-        for p, l2 in self.screeninfo:
-            l = len(l2) - l2.count(0)
-            in_wrapped_line = p + sum(l2) >= self.console.width
-            offset = l - 1 if in_wrapped_line else l  # need to remove backslash
             if offset >= pos:
                 break
 
-            if p + sum(l2) >= self.console.width:
-                pos -= l - 1  # -1 cause backslash is not in buffer
-            else:
-                pos -= l + 1  # +1 cause newline is in buffer
+            if not in_wrapped_line:
+                offset += 1  # there's a newline in buffer
+
+            pos -= offset
             y += 1
-        return p + sum(l2[:pos]), y
+        return prompt_len + sum(char_widths[:pos]), y
 
     def insert(self, text: str | list[str]) -> None:
         """Insert 'text' at the insertion point."""
@@ -619,6 +567,7 @@ class Reader:
     def update_cursor(self) -> None:
         """Move the cursor to reflect changes in self.pos"""
         self.cxy = self.pos2xy()
+        trace("update_cursor({pos}) = {cxy}", pos=self.pos, cxy=self.cxy)
         self.console.move_cursor(*self.cxy)
 
     def after_command(self, cmd: Command) -> None:
@@ -670,6 +619,16 @@ class Reader:
                 setattr(self, arg, prev_state[arg])
             self.prepare()
 
+    @contextmanager
+    def suspend_colorization(self) -> SimpleContextManager:
+        try:
+            old_can_colorize = self.can_colorize
+            self.can_colorize = False
+            yield
+        finally:
+            self.can_colorize = old_can_colorize
+
+
     def finish(self) -> None:
         """Called when a command signals that we're finished."""
         pass
@@ -685,9 +644,7 @@ class Reader:
 
     def refresh(self) -> None:
         """Recalculate and refresh the screen."""
-        if self.in_bracketed_paste and self.buffer and not self.buffer[-1] == "\n":
-            return
-
+        self.console.height, self.console.width = self.console.getheightwidth()
         # this call sets up self.cxy, so call it first.
         self.screen = self.calc_screen()
         self.console.refresh(self.screen, self.cxy)

Lib/_pyrepl/readline.py

@@ -32,20 +32,22 @@ import warnings
 from dataclasses import dataclass, field
 
 import os
-from site import gethistoryfile   # type: ignore[attr-defined]
+from site import gethistoryfile
 import sys
 from rlcompleter import Completer as RLCompleter
 
 from . import commands, historical_reader
 from .completing_reader import CompletingReader
 from .console import Console as ConsoleType
+from ._module_completer import ModuleCompleter, make_default_module_completer
 
 Console: type[ConsoleType]
 _error: tuple[type[Exception], ...] | type[Exception]
-try:
-    from .unix_console import UnixConsole as Console, _error
-except ImportError:
+
+if os.name == "nt":
     from .windows_console import WindowsConsole as Console, _error
+else:
+    from .unix_console import UnixConsole as Console, _error
 
 ENCODING = sys.getdefaultencoding() or "latin1"
 
@@ -89,6 +91,7 @@ __all__ = [
     # "set_pre_input_hook",
     "set_startup_hook",
     "write_history_file",
+    "append_history_file",
     # ---- multiline extensions ----
     "multiline_input",
 ]
@@ -99,7 +102,7 @@ __all__ = [
 class ReadlineConfig:
     readline_completer: Completer | None = None
     completer_delims: frozenset[str] = frozenset(" \t\n`~!@#$%^&*()-=+[{]}\\|;:'\",<>/?")
-
+    module_completer: ModuleCompleter = field(default_factory=make_default_module_completer)
 
 @dataclass(kw_only=True)
 class ReadlineAlikeReader(historical_reader.HistoricalReader, CompletingReader):
@@ -132,6 +135,9 @@ class ReadlineAlikeReader(historical_reader.HistoricalReader, CompletingReader):
         return "".join(b[p + 1 : self.pos])
 
     def get_completions(self, stem: str) -> list[str]:
+        module_completions = self.get_module_completions()
+        if module_completions is not None:
+            return module_completions
         if len(stem) == 0 and self.more_lines is not None:
             b = self.buffer
             p = self.pos
@@ -161,6 +167,10 @@ class ReadlineAlikeReader(historical_reader.HistoricalReader, CompletingReader):
             result.sort()
         return result
 
+    def get_module_completions(self) -> list[str] | None:
+        line = self.get_line()
+        return self.config.module_completer.get_completions(line)
+
     def get_trimmed_history(self, maxlength: int) -> list[str]:
         if maxlength >= 0:
             cut = len(self.history) - maxlength
@@ -268,10 +278,6 @@ class maybe_accept(commands.Command):
         r = self.reader  # type: ignore[assignment]
         r.dirty = True  # this is needed to hide the completion menu, if visible
 
-        if self.reader.in_bracketed_paste:
-            r.insert("\n")
-            return
-
         # if there are already several lines and the cursor
         # is not on the last one, always insert a new \n.
         text = r.get_unicode()
@@ -446,6 +452,7 @@ class _ReadlineWrapper:
                         del buffer[:]
                     if line:
                         history.append(line)
+        self.set_history_length(self.get_current_history_length())
 
     def write_history_file(self, filename: str = gethistoryfile()) -> None:
         maxlength = self.saved_history_length
@@ -457,6 +464,19 @@ class _ReadlineWrapper:
                 entry = entry.replace("\n", "\r\n")  # multiline history support
                 f.write(entry + "\n")
 
+    def append_history_file(self, filename: str = gethistoryfile()) -> None:
+        reader = self.get_reader()
+        saved_length = self.get_history_length()
+        length = self.get_current_history_length() - saved_length
+        history = reader.get_trimmed_history(length)
+        f = open(os.path.expanduser(filename), "a",
+                 encoding="utf-8", newline="\n")
+        with f:
+            for entry in history:
+                entry = entry.replace("\n", "\r\n")  # multiline history support
+                f.write(entry + "\n")
+        self.set_history_length(saved_length + length)
+
     def clear_history(self) -> None:
         del self.get_reader().history[:]
 
@@ -526,6 +546,7 @@ set_history_length = _wrapper.set_history_length
 get_current_history_length = _wrapper.get_current_history_length
 read_history_file = _wrapper.read_history_file
 write_history_file = _wrapper.write_history_file
+append_history_file = _wrapper.append_history_file
 clear_history = _wrapper.clear_history
 get_history_item = _wrapper.get_history_item
 remove_history_item = _wrapper.remove_history_item
@@ -587,6 +608,7 @@ def _setup(namespace: Mapping[str, Any]) -> None:
     # set up namespace in rlcompleter, which requires it to be a bona fide dict
     if not isinstance(namespace, dict):
         namespace = dict(namespace)
+    _wrapper.config.module_completer = ModuleCompleter(namespace)
     _wrapper.config.readline_completer = RLCompleter(namespace).complete
 
     # this is not really what readline.c does.  Better than nothing I guess

Lib/_pyrepl/simple_interact.py

@@ -26,18 +26,14 @@ allowing multiline input and multiline history entries.
 from __future__ import annotations
 
 import _sitebuiltins
-import linecache
 import functools
 import os
 import sys
 import code
+import warnings
+import errno
 
-from .readline import _get_reader, multiline_input
-
-TYPE_CHECKING = False
-
-if TYPE_CHECKING:
-    from typing import Any
+from .readline import _get_reader, multiline_input, append_history_file
 
 
 _error: tuple[type[Exception], ...] | type[Exception]
@@ -115,6 +111,10 @@ def run_multiline_interactive_console(
     more_lines = functools.partial(_more_lines, console)
     input_n = 0
 
+    _is_x_showrefcount_set = sys._xoptions.get("showrefcount")
+    _is_pydebug_build = hasattr(sys, "gettotalrefcount")
+    show_ref_count = _is_x_showrefcount_set and _is_pydebug_build
+
     def maybe_run_command(statement: str) -> bool:
         statement = statement.strip()
         if statement in console.locals or statement not in REPL_COMMANDS:
@@ -125,12 +125,12 @@ def run_multiline_interactive_console(
         command = REPL_COMMANDS[statement]
         if callable(command):
             # Make sure that history does not change because of commands
-            with reader.suspend_history():
+            with reader.suspend_history(), reader.suspend_colorization():
                 command()
             return True
         return False
 
-    while 1:
+    while True:
         try:
             try:
                 sys.stdout.flush()
@@ -148,20 +148,34 @@ def run_multiline_interactive_console(
                 continue
 
             input_name = f"<python-input-{input_n}>"
-            linecache._register_code(input_name, statement, "<stdin>")  # type: ignore[attr-defined]
             more = console.push(_strip_final_indent(statement), filename=input_name, _symbol="single")  # type: ignore[call-arg]
             assert not more
+            try:
+                append_history_file()
+            except (FileNotFoundError, PermissionError, OSError) as e:
+                warnings.warn(f"failed to open the history file for writing: {e}")
+
             input_n += 1
         except KeyboardInterrupt:
             r = _get_reader()
+            r.cmpltn_reset()
             if r.input_trans is r.isearch_trans:
                 r.do_cmd(("isearch-end", [""]))
             r.pos = len(r.get_unicode())
             r.dirty = True
             r.refresh()
-            r.in_bracketed_paste = False
             console.write("\nKeyboardInterrupt\n")
             console.resetbuffer()
         except MemoryError:
             console.write("\nMemoryError\n")
             console.resetbuffer()
+        except SystemExit:
+            raise
+        except:
+            console.showtraceback()
+            console.resetbuffer()
+        if show_ref_count:
+            console.write(
+                f"[{sys.gettotalrefcount()} refs,"
+                f" {sys.getallocatedblocks()} blocks]\n"
+            )

Lib/_pyrepl/terminfo.py

@@ -0,0 +1,488 @@
+"""Pure Python curses-like terminal capability queries."""
+
+from dataclasses import dataclass, field
+import errno
+import os
+from pathlib import Path
+import re
+import struct
+
+
+# Terminfo constants
+MAGIC16 = 0o432  # Magic number for 16-bit terminfo format
+MAGIC32 = 0o1036  # Magic number for 32-bit terminfo format
+
+# Special values for absent/cancelled capabilities
+ABSENT_BOOLEAN = -1
+ABSENT_NUMERIC = -1
+CANCELLED_NUMERIC = -2
+ABSENT_STRING = None
+CANCELLED_STRING = None
+
+
+# Standard string capability names from ncurses Caps file
+# This matches the order used by ncurses when compiling terminfo
+# fmt: off
+_STRING_NAMES: tuple[str, ...] = (
+    "cbt", "bel", "cr", "csr", "tbc", "clear", "el", "ed", "hpa", "cmdch",
+    "cup", "cud1", "home", "civis", "cub1", "mrcup", "cnorm", "cuf1", "ll",
+    "cuu1", "cvvis", "dch1", "dl1", "dsl", "hd", "smacs", "blink", "bold",
+    "smcup", "smdc", "dim", "smir", "invis", "prot", "rev", "smso", "smul",
+    "ech", "rmacs", "sgr0", "rmcup", "rmdc", "rmir", "rmso", "rmul", "flash",
+    "ff", "fsl", "is1", "is2", "is3", "if", "ich1", "il1", "ip", "kbs", "ktbc",
+    "kclr", "kctab", "kdch1", "kdl1", "kcud1", "krmir", "kel", "ked", "kf0",
+    "kf1", "kf10", "kf2", "kf3", "kf4", "kf5", "kf6", "kf7", "kf8", "kf9",
+    "khome", "kich1", "kil1", "kcub1", "kll", "knp", "kpp", "kcuf1", "kind",
+    "kri", "khts", "kcuu1", "rmkx", "smkx", "lf0", "lf1", "lf10", "lf2", "lf3",
+    "lf4", "lf5", "lf6", "lf7", "lf8", "lf9", "rmm", "smm", "nel", "pad", "dch",
+    "dl", "cud", "ich", "indn", "il", "cub", "cuf", "rin", "cuu", "pfkey",
+    "pfloc", "pfx", "mc0", "mc4", "mc5", "rep", "rs1", "rs2", "rs3", "rf", "rc",
+    "vpa", "sc", "ind", "ri", "sgr", "hts", "wind", "ht", "tsl", "uc", "hu",
+    "iprog", "ka1", "ka3", "kb2", "kc1", "kc3", "mc5p", "rmp", "acsc", "pln",
+    "kcbt", "smxon", "rmxon", "smam", "rmam", "xonc", "xoffc", "enacs", "smln",
+    "rmln", "kbeg", "kcan", "kclo", "kcmd", "kcpy", "kcrt", "kend", "kent",
+    "kext", "kfnd", "khlp", "kmrk", "kmsg", "kmov", "knxt", "kopn", "kopt",
+    "kprv", "kprt", "krdo", "kref", "krfr", "krpl", "krst", "kres", "ksav",
+    "kspd", "kund", "kBEG", "kCAN", "kCMD", "kCPY", "kCRT", "kDC", "kDL",
+    "kslt", "kEND", "kEOL", "kEXT", "kFND", "kHLP", "kHOM", "kIC", "kLFT",
+    "kMSG", "kMOV", "kNXT", "kOPT", "kPRV", "kPRT", "kRDO", "kRPL", "kRIT",
+    "kRES", "kSAV", "kSPD", "kUND", "rfi", "kf11", "kf12", "kf13", "kf14",
+    "kf15", "kf16", "kf17", "kf18", "kf19", "kf20", "kf21", "kf22", "kf23",
+    "kf24", "kf25", "kf26", "kf27", "kf28", "kf29", "kf30", "kf31", "kf32",
+    "kf33", "kf34", "kf35", "kf36", "kf37", "kf38", "kf39", "kf40", "kf41",
+    "kf42", "kf43", "kf44", "kf45", "kf46", "kf47", "kf48", "kf49", "kf50",
+    "kf51", "kf52", "kf53", "kf54", "kf55", "kf56", "kf57", "kf58", "kf59",
+    "kf60", "kf61", "kf62", "kf63", "el1", "mgc", "smgl", "smgr", "fln", "sclk",
+    "dclk", "rmclk", "cwin", "wingo", "hup","dial", "qdial", "tone", "pulse",
+    "hook", "pause", "wait", "u0", "u1", "u2", "u3", "u4", "u5", "u6", "u7",
+    "u8", "u9", "op", "oc", "initc", "initp", "scp", "setf", "setb", "cpi",
+    "lpi", "chr", "cvr", "defc", "swidm", "sdrfq", "sitm", "slm", "smicm",
+    "snlq", "snrmq", "sshm", "ssubm", "ssupm", "sum", "rwidm", "ritm", "rlm",
+    "rmicm", "rshm", "rsubm", "rsupm", "rum", "mhpa", "mcud1", "mcub1", "mcuf1",
+    "mvpa", "mcuu1", "porder", "mcud", "mcub", "mcuf", "mcuu", "scs", "smgb",
+    "smgbp", "smglp", "smgrp", "smgt", "smgtp", "sbim", "scsd", "rbim", "rcsd",
+    "subcs", "supcs", "docr", "zerom", "csnm", "kmous", "minfo", "reqmp",
+    "getm", "setaf", "setab", "pfxl", "devt", "csin", "s0ds", "s1ds", "s2ds",
+    "s3ds", "smglr", "smgtb", "birep", "binel", "bicr", "colornm", "defbi",
+    "endbi", "setcolor", "slines", "dispc", "smpch", "rmpch", "smsc", "rmsc",
+    "pctrm", "scesc", "scesa", "ehhlm", "elhlm", "elohlm", "erhlm", "ethlm",
+    "evhlm", "sgr1", "slength", "OTi2", "OTrs", "OTnl", "OTbc", "OTko", "OTma",
+    "OTG2", "OTG3", "OTG1", "OTG4", "OTGR", "OTGL", "OTGU", "OTGD", "OTGH",
+    "OTGV", "OTGC","meml", "memu", "box1"
+)
+# fmt: on
+
+
+def _get_terminfo_dirs() -> list[Path]:
+    """Get list of directories to search for terminfo files.
+
+    Based on ncurses behavior in:
+    - ncurses/tinfo/db_iterator.c:_nc_next_db()
+    - ncurses/tinfo/read_entry.c:_nc_read_entry()
+    """
+    dirs = []
+
+    terminfo = os.environ.get("TERMINFO")
+    if terminfo:
+        dirs.append(terminfo)
+
+    try:
+        home = Path.home()
+        dirs.append(str(home / ".terminfo"))
+    except RuntimeError:
+        pass
+
+    # Check TERMINFO_DIRS
+    terminfo_dirs = os.environ.get("TERMINFO_DIRS", "")
+    if terminfo_dirs:
+        for d in terminfo_dirs.split(":"):
+            if d:
+                dirs.append(d)
+
+    dirs.extend(
+        [
+            "/etc/terminfo",
+            "/lib/terminfo",
+            "/usr/lib/terminfo",
+            "/usr/share/terminfo",
+            "/usr/share/lib/terminfo",
+            "/usr/share/misc/terminfo",
+            "/usr/local/lib/terminfo",
+            "/usr/local/share/terminfo",
+        ]
+    )
+
+    return [Path(d) for d in dirs if Path(d).is_dir()]
+
+
+def _validate_terminal_name_or_raise(terminal_name: str) -> None:
+    if not isinstance(terminal_name, str):
+        raise TypeError("`terminal_name` must be a string")
+
+    if not terminal_name:
+        raise ValueError("`terminal_name` cannot be empty")
+
+    if "\x00" in terminal_name:
+        raise ValueError("NUL character found in `terminal_name`")
+
+    t = Path(terminal_name)
+    if len(t.parts) > 1:
+        raise ValueError("`terminal_name` cannot contain path separators")
+
+
+def _read_terminfo_file(terminal_name: str) -> bytes:
+    """Find and read terminfo file for given terminal name.
+
+    Terminfo files are stored in directories using the first character
+    of the terminal name as a subdirectory.
+    """
+    _validate_terminal_name_or_raise(terminal_name)
+    first_char = terminal_name[0].lower()
+    filename = terminal_name
+
+    for directory in _get_terminfo_dirs():
+        path = directory / first_char / filename
+        if path.is_file():
+            return path.read_bytes()
+
+        # Try with hex encoding of first char (for special chars)
+        hex_dir = "%02x" % ord(first_char)
+        path = directory / hex_dir / filename
+        if path.is_file():
+            return path.read_bytes()
+
+    raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), filename)
+
+
+# Hard-coded terminal capabilities for common terminals
+# This is a minimal subset needed by PyREPL
+_TERMINAL_CAPABILITIES = {
+    # ANSI/xterm-compatible terminals
+    "ansi": {
+        # Bell
+        "bel": b"\x07",
+        # Cursor movement
+        "cub": b"\x1b[%p1%dD",  # Move cursor left N columns
+        "cud": b"\x1b[%p1%dB",  # Move cursor down N rows
+        "cuf": b"\x1b[%p1%dC",  # Move cursor right N columns
+        "cuu": b"\x1b[%p1%dA",  # Move cursor up N rows
+        "cub1": b"\x08",  # Move cursor left 1 column
+        "cud1": b"\n",  # Move cursor down 1 row
+        "cuf1": b"\x1b[C",  # Move cursor right 1 column
+        "cuu1": b"\x1b[A",  # Move cursor up 1 row
+        "cup": b"\x1b[%i%p1%d;%p2%dH",  # Move cursor to row, column
+        "hpa": b"\x1b[%i%p1%dG",  # Move cursor to column
+        # Clear operations
+        "clear": b"\x1b[H\x1b[2J",  # Clear screen and home cursor
+        "el": b"\x1b[K",  # Clear to end of line
+        # Insert/delete
+        "dch": b"\x1b[%p1%dP",  # Delete N characters
+        "dch1": b"\x1b[P",  # Delete 1 character
+        "ich": b"\x1b[%p1%d@",  # Insert N characters
+        "ich1": b"",  # Insert 1 character
+        # Cursor visibility
+        "civis": b"\x1b[?25l",  # Make cursor invisible
+        "cnorm": b"\x1b[?12l\x1b[?25h",  # Make cursor normal (visible)
+        # Scrolling
+        "ind": b"\n",  # Scroll up one line
+        "ri": b"\x1bM",  # Scroll down one line
+        # Keypad mode
+        "smkx": b"\x1b[?1h\x1b=",  # Enable keypad mode
+        "rmkx": b"\x1b[?1l\x1b>",  # Disable keypad mode
+        # Padding (not used in modern terminals)
+        "pad": b"",
+        # Function keys and special keys
+        "kdch1": b"\x1b[3~",  # Delete key
+        "kcud1": b"\x1bOB",  # Down arrow
+        "kend": b"\x1bOF",  # End key
+        "kent": b"\x1bOM",  # Enter key
+        "khome": b"\x1bOH",  # Home key
+        "kich1": b"\x1b[2~",  # Insert key
+        "kcub1": b"\x1bOD",  # Left arrow
+        "knp": b"\x1b[6~",  # Page down
+        "kpp": b"\x1b[5~",  # Page up
+        "kcuf1": b"\x1bOC",  # Right arrow
+        "kcuu1": b"\x1bOA",  # Up arrow
+        # Function keys F1-F20
+        "kf1": b"\x1bOP",
+        "kf2": b"\x1bOQ",
+        "kf3": b"\x1bOR",
+        "kf4": b"\x1bOS",
+        "kf5": b"\x1b[15~",
+        "kf6": b"\x1b[17~",
+        "kf7": b"\x1b[18~",
+        "kf8": b"\x1b[19~",
+        "kf9": b"\x1b[20~",
+        "kf10": b"\x1b[21~",
+        "kf11": b"\x1b[23~",
+        "kf12": b"\x1b[24~",
+        "kf13": b"\x1b[1;2P",
+        "kf14": b"\x1b[1;2Q",
+        "kf15": b"\x1b[1;2R",
+        "kf16": b"\x1b[1;2S",
+        "kf17": b"\x1b[15;2~",
+        "kf18": b"\x1b[17;2~",
+        "kf19": b"\x1b[18;2~",
+        "kf20": b"\x1b[19;2~",
+    },
+    # Dumb terminal - minimal capabilities
+    "dumb": {
+        "bel": b"\x07",  # Bell
+        "cud1": b"\n",  # Move down 1 row (newline)
+        "ind": b"\n",  # Scroll up one line (newline)
+    },
+    # Linux console
+    "linux": {
+        # Bell
+        "bel": b"\x07",
+        # Cursor movement
+        "cub": b"\x1b[%p1%dD",  # Move cursor left N columns
+        "cud": b"\x1b[%p1%dB",  # Move cursor down N rows
+        "cuf": b"\x1b[%p1%dC",  # Move cursor right N columns
+        "cuu": b"\x1b[%p1%dA",  # Move cursor up N rows
+        "cub1": b"\x08",  # Move cursor left 1 column (backspace)
+        "cud1": b"\n",  # Move cursor down 1 row (newline)
+        "cuf1": b"\x1b[C",  # Move cursor right 1 column
+        "cuu1": b"\x1b[A",  # Move cursor up 1 row
+        "cup": b"\x1b[%i%p1%d;%p2%dH",  # Move cursor to row, column
+        "hpa": b"\x1b[%i%p1%dG",  # Move cursor to column
+        # Clear operations
+        "clear": b"\x1b[H\x1b[J",  # Clear screen and home cursor (different from ansi!)
+        "el": b"\x1b[K",  # Clear to end of line
+        # Insert/delete
+        "dch": b"\x1b[%p1%dP",  # Delete N characters
+        "dch1": b"\x1b[P",  # Delete 1 character
+        "ich": b"\x1b[%p1%d@",  # Insert N characters
+        "ich1": b"\x1b[@",  # Insert 1 character
+        # Cursor visibility
+        "civis": b"\x1b[?25l\x1b[?1c",  # Make cursor invisible
+        "cnorm": b"\x1b[?25h\x1b[?0c",  # Make cursor normal
+        # Scrolling
+        "ind": b"\n",  # Scroll up one line
+        "ri": b"\x1bM",  # Scroll down one line
+        # Keypad mode
+        "smkx": b"\x1b[?1h\x1b=",  # Enable keypad mode
+        "rmkx": b"\x1b[?1l\x1b>",  # Disable keypad mode
+        # Function keys and special keys
+        "kdch1": b"\x1b[3~",  # Delete key
+        "kcud1": b"\x1b[B",  # Down arrow
+        "kend": b"\x1b[4~",  # End key (different from ansi!)
+        "khome": b"\x1b[1~",  # Home key (different from ansi!)
+        "kich1": b"\x1b[2~",  # Insert key
+        "kcub1": b"\x1b[D",  # Left arrow
+        "knp": b"\x1b[6~",  # Page down
+        "kpp": b"\x1b[5~",  # Page up
+        "kcuf1": b"\x1b[C",  # Right arrow
+        "kcuu1": b"\x1b[A",  # Up arrow
+        # Function keys
+        "kf1": b"\x1b[[A",
+        "kf2": b"\x1b[[B",
+        "kf3": b"\x1b[[C",
+        "kf4": b"\x1b[[D",
+        "kf5": b"\x1b[[E",
+        "kf6": b"\x1b[17~",
+        "kf7": b"\x1b[18~",
+        "kf8": b"\x1b[19~",
+        "kf9": b"\x1b[20~",
+        "kf10": b"\x1b[21~",
+        "kf11": b"\x1b[23~",
+        "kf12": b"\x1b[24~",
+        "kf13": b"\x1b[25~",
+        "kf14": b"\x1b[26~",
+        "kf15": b"\x1b[28~",
+        "kf16": b"\x1b[29~",
+        "kf17": b"\x1b[31~",
+        "kf18": b"\x1b[32~",
+        "kf19": b"\x1b[33~",
+        "kf20": b"\x1b[34~",
+    },
+}
+
+# Map common TERM values to capability sets
+_TERM_ALIASES = {
+    "xterm": "ansi",
+    "xterm-color": "ansi",
+    "xterm-256color": "ansi",
+    "screen": "ansi",
+    "screen-256color": "ansi",
+    "tmux": "ansi",
+    "tmux-256color": "ansi",
+    "vt100": "ansi",
+    "vt220": "ansi",
+    "rxvt": "ansi",
+    "rxvt-unicode": "ansi",
+    "rxvt-unicode-256color": "ansi",
+    "unknown": "dumb",
+}
+
+
+@dataclass
+class TermInfo:
+    terminal_name: str | bytes | None
+    fallback: bool = True
+
+    _capabilities: dict[str, bytes] = field(default_factory=dict)
+
+    def __post_init__(self) -> None:
+        """Initialize terminal capabilities for the given terminal type.
+
+        Based on ncurses implementation in:
+        - ncurses/tinfo/lib_setup.c:setupterm() and _nc_setupterm()
+        - ncurses/tinfo/lib_setup.c:TINFO_SETUP_TERM()
+
+        This version first attempts to read terminfo database files like ncurses,
+        then, if `fallback` is True, falls back to hardcoded capabilities for
+        common terminal types.
+        """
+        # If termstr is None or empty, try to get from environment
+        if not self.terminal_name:
+            self.terminal_name = os.environ.get("TERM") or "ANSI"
+
+        if isinstance(self.terminal_name, bytes):
+            self.terminal_name = self.terminal_name.decode("ascii")
+
+        try:
+            self._parse_terminfo_file(self.terminal_name)
+        except (OSError, ValueError):
+            if not self.fallback:
+                raise
+
+            term_type = _TERM_ALIASES.get(
+                self.terminal_name, self.terminal_name
+            )
+            if term_type not in _TERMINAL_CAPABILITIES:
+                term_type = "dumb"
+            self._capabilities = _TERMINAL_CAPABILITIES[term_type].copy()
+
+    def _parse_terminfo_file(self, terminal_name: str) -> None:
+        """Parse a terminfo file.
+
+        Populate the _capabilities dict for easy retrieval
+
+        Based on ncurses implementation in:
+        - ncurses/tinfo/read_entry.c:_nc_read_termtype()
+        - ncurses/tinfo/read_entry.c:_nc_read_file_entry()
+        - ncurses/tinfo/lib_ti.c:tigetstr()
+        """
+        data = _read_terminfo_file(terminal_name)
+        too_short = f"TermInfo file for {terminal_name!r} too short"
+        offset = 12
+        if len(data) < offset:
+            raise ValueError(too_short)
+
+        magic, name_size, bool_count, num_count, str_count, str_size = (
+            struct.unpack("<Hhhhhh", data[:offset])
+        )
+
+        if magic == MAGIC16:
+            number_size = 2
+        elif magic == MAGIC32:
+            number_size = 4
+        else:
+            raise ValueError(
+                f"TermInfo file for {terminal_name!r} uses unknown magic"
+            )
+
+        # Skip data than PyREPL doesn't need:
+        # - names (`|`-separated ASCII strings)
+        # - boolean capabilities (bytes with value 0 or 1)
+        # - numbers (little-endian integers, `number_size` bytes each)
+        offset += name_size
+        offset += bool_count
+        if offset % 2:
+            # Align to even byte boundary for numbers
+            offset += 1
+        offset += num_count * number_size
+        if offset > len(data):
+            raise ValueError(too_short)
+
+        # Read string offsets
+        end_offset = offset + 2 * str_count
+        if offset > len(data):
+            raise ValueError(too_short)
+        string_offset_data = data[offset:end_offset]
+        string_offsets = [
+            off for [off] in struct.iter_unpack("<h", string_offset_data)
+        ]
+        offset = end_offset
+
+        # Read string table
+        if offset + str_size > len(data):
+            raise ValueError(too_short)
+        string_table = data[offset : offset + str_size]
+
+        # Extract strings from string table
+        capabilities = {}
+        for cap, off in zip(_STRING_NAMES, string_offsets):
+            if off < 0:
+                # CANCELLED_STRING; we do not store those
+                continue
+            elif off < len(string_table):
+                # Find null terminator
+                end = string_table.find(0, off)
+                if end >= 0:
+                    capabilities[cap] = string_table[off:end]
+            # in other cases this is ABSENT_STRING; we don't store those.
+
+        # Note: we don't support extended capabilities since PyREPL doesn't
+        # need them.
+
+        self._capabilities = capabilities
+
+    def get(self, cap: str) -> bytes | None:
+        """Get terminal capability string by name.
+        """
+        if not isinstance(cap, str):
+            raise TypeError(f"`cap` must be a string, not {type(cap)}")
+
+        return self._capabilities.get(cap)
+
+
+def tparm(cap_bytes: bytes, *params: int) -> bytes:
+    """Parameterize a terminal capability string.
+
+    Based on ncurses implementation in:
+    - ncurses/tinfo/lib_tparm.c:tparm()
+    - ncurses/tinfo/lib_tparm.c:tparam_internal()
+
+    The ncurses version implements a full stack-based interpreter for
+    terminfo parameter strings. This pure Python version implements only
+    the subset of parameter substitution operations needed by PyREPL:
+    - %i (increment parameters for 1-based indexing)
+    - %p[1-9]%d (parameter substitution)
+    - %p[1-9]%{n}%+%d (parameter plus constant)
+    """
+    if not isinstance(cap_bytes, bytes):
+        raise TypeError(f"`cap` must be bytes, not {type(cap_bytes)}")
+
+    result = cap_bytes
+
+    # %i - increment parameters (1-based instead of 0-based)
+    increment = b"%i" in result
+    if increment:
+        result = result.replace(b"%i", b"")
+
+    # Replace %p1%d, %p2%d, etc. with actual parameter values
+    for i in range(len(params)):
+        pattern = b"%%p%d%%d" % (i + 1)
+        if pattern in result:
+            value = params[i]
+            if increment:
+                value += 1
+            result = result.replace(pattern, str(value).encode("ascii"))
+
+    # Handle %p1%{1}%+%d (parameter plus constant)
+    # Used in some cursor positioning sequences
+    pattern_re = re.compile(rb"%p(\d)%\{(\d+)\}%\+%d")
+    matches = list(pattern_re.finditer(result))
+    for match in reversed(matches):  # reversed to maintain positions
+        param_idx = int(match.group(1))
+        constant = int(match.group(2))
+        value = params[param_idx] + constant
+        result = (
+            result[: match.start()]
+            + str(value).encode("ascii")
+            + result[match.end() :]
+        )
+
+    return result

Lib/_pyrepl/trace.py

@@ -1,6 +1,7 @@
 from __future__ import annotations
 
 import os
+import sys
 
 # types
 if False:
@@ -12,10 +13,22 @@ if trace_filename := os.environ.get("PYREPL_TRACE"):
     trace_file = open(trace_filename, "a")
 
 
-def trace(line: str, *k: object, **kw: object) -> None:
-    if trace_file is None:
-        return
-    if k or kw:
-        line = line.format(*k, **kw)
-    trace_file.write(line + "\n")
-    trace_file.flush()
+
+if sys.platform == "emscripten":
+    from posix import _emscripten_log
+
+    def trace(line: str, *k: object, **kw: object) -> None:
+        if "PYREPL_TRACE" not in os.environ:
+            return
+        if k or kw:
+            line = line.format(*k, **kw)
+        _emscripten_log(line)
+
+else:
+    def trace(line: str, *k: object, **kw: object) -> None:
+        if trace_file is None:
+            return
+        if k or kw:
+            line = line.format(*k, **kw)
+        trace_file.write(line + "\n")
+        trace_file.flush()

Lib/_pyrepl/types.py

@@ -1,8 +1,10 @@
 from collections.abc import Callable, Iterator
 
-Callback = Callable[[], object]
-SimpleContextManager = Iterator[None]
-KeySpec = str  # like r"\C-c"
-CommandName = str  # like "interrupt"
-EventTuple = tuple[CommandName, str]
-Completer = Callable[[str, int], str | None]
+type Callback = Callable[[], object]
+type SimpleContextManager = Iterator[None]
+type KeySpec = str  # like r"\C-c"
+type CommandName = str  # like "interrupt"
+type EventTuple = tuple[CommandName, str]
+type Completer = Callable[[str, int], str | None]
+type CharBuffer = list[str]
+type CharWidths = list[int]

Lib/_pyrepl/unix_console.py

@@ -29,31 +29,41 @@ import signal
 import struct
 import termios
 import time
+import types
 import platform
 from fcntl import ioctl
 
-from . import curses
+from . import terminfo
 from .console import Console, Event
-from .fancy_termios import tcgetattr, tcsetattr
+from .fancy_termios import tcgetattr, tcsetattr, TermState
 from .trace import trace
 from .unix_eventqueue import EventQueue
 from .utils import wlen
 
+# declare posix optional to allow None assignment on other platforms
+posix: types.ModuleType | None
+try:
+    import posix
+except ImportError:
+    posix = None
 
 TYPE_CHECKING = False
 
 # types
 if TYPE_CHECKING:
-    from typing import IO, Literal, overload
+    from typing import AbstractSet, IO, Literal, overload, cast
 else:
     overload = lambda func: None
+    cast = lambda typ, val: val
 
 
 class InvalidTerminal(RuntimeError):
-    pass
+    def __init__(self, message: str) -> None:
+        super().__init__(errno.EIO, message)
 
 
-_error = (termios.error, curses.error, InvalidTerminal)
+_error = (termios.error, InvalidTerminal)
+_error_codes_to_ignore = frozenset([errno.EIO, errno.ENXIO, errno.EPERM])
 
 SIGWINCH_EVENT = "repaint"
 
@@ -110,7 +120,7 @@ delayprog = re.compile(b"\\$<([0-9]+)((?:/|\\*){0,2})>")
 try:
     poll: type[select.poll] = select.poll
 except AttributeError:
-    # this is exactly the minumum necessary to support what we
+    # this is exactly the minimum necessary to support what we
     # do with poll objects
     class MinimalPoll:
         def __init__(self):
@@ -118,12 +128,13 @@ except AttributeError:
 
         def register(self, fd, flag):
             self.fd = fd
+
         # note: The 'timeout' argument is received as *milliseconds*
         def poll(self, timeout: float | None = None) -> list[int]:
             if timeout is None:
                 r, w, e = select.select([self.fd], [], [])
             else:
-                r, w, e = select.select([self.fd], [], [], timeout/1000)
+                r, w, e = select.select([self.fd], [], [], timeout / 1000)
             return r
 
     poll = MinimalPoll  # type: ignore[assignment]
@@ -150,19 +161,28 @@ class UnixConsole(Console):
 
         self.pollob = poll()
         self.pollob.register(self.input_fd, select.POLLIN)
-        self.input_buffer = b""
-        self.input_buffer_pos = 0
-        curses.setupterm(term or None, self.output_fd)
+        self.terminfo = terminfo.TermInfo(term or None)
         self.term = term
+        self.is_apple_terminal = (
+            platform.system() == "Darwin"
+            and os.getenv("TERM_PROGRAM") == "Apple_Terminal"
+        )
+
+        try:
+            self.__input_fd_set(tcgetattr(self.input_fd), ignore=frozenset())
+        except _error as e:
+            raise RuntimeError(f"termios failure ({e.args[1]})")
 
         @overload
-        def _my_getstr(cap: str, optional: Literal[False] = False) -> bytes: ...
+        def _my_getstr(
+            cap: str, optional: Literal[False] = False
+        ) -> bytes: ...
 
         @overload
         def _my_getstr(cap: str, optional: bool) -> bytes | None: ...
 
         def _my_getstr(cap: str, optional: bool = False) -> bytes | None:
-            r = curses.tigetstr(cap)
+            r = self.terminfo.get(cap)
             if not optional and r is None:
                 raise InvalidTerminal(
                     f"terminal doesn't have the required {cap} capability"
@@ -196,26 +216,19 @@ class UnixConsole(Console):
 
         self.__setup_movement()
 
-        self.event_queue = EventQueue(self.input_fd, self.encoding)
+        self.event_queue = EventQueue(
+            self.input_fd, self.encoding, self.terminfo
+        )
         self.cursor_visible = 1
 
-    def more_in_buffer(self) -> bool:
-        return bool(
-            self.input_buffer
-            and self.input_buffer_pos < len(self.input_buffer)
-        )
+        signal.signal(signal.SIGCONT, self._sigcont_handler)
+
+    def _sigcont_handler(self, signum, frame):
+        self.restore()
+        self.prepare()
 
     def __read(self, n: int) -> bytes:
-        if not self.more_in_buffer():
-            self.input_buffer = os.read(self.input_fd, 10000)
-
-        ret = self.input_buffer[self.input_buffer_pos : self.input_buffer_pos + n]
-        self.input_buffer_pos += len(ret)
-        if self.input_buffer_pos >= len(self.input_buffer):
-            self.input_buffer = b""
-            self.input_buffer_pos = 0
-        return ret
-
+        return os.read(self.input_fd, n)
 
     def change_encoding(self, encoding: str) -> None:
         """
@@ -238,8 +251,9 @@ class UnixConsole(Console):
         if not self.__gone_tall:
             while len(self.screen) < min(len(screen), self.height):
                 self.__hide_cursor()
-                self.__move(0, len(self.screen) - 1)
-                self.__write("\n")
+                if self.screen:
+                    self.__move(0, len(self.screen) - 1)
+                    self.__write("\n")
                 self.posxy = 0, len(self.screen)
                 self.screen.append("")
         else:
@@ -328,6 +342,8 @@ class UnixConsole(Console):
         """
         Prepare the console for input/output operations.
         """
+        self.__buffer = []
+
         self.__svtermstate = tcgetattr(self.input_fd)
         raw = self.__svtermstate.copy()
         raw.iflag &= ~(termios.INPCK | termios.ISTRIP | termios.IXON)
@@ -339,17 +355,15 @@ class UnixConsole(Console):
         raw.lflag |= termios.ISIG
         raw.cc[termios.VMIN] = 1
         raw.cc[termios.VTIME] = 0
-        tcsetattr(self.input_fd, termios.TCSADRAIN, raw)
+        self.__input_fd_set(raw)
 
         # In macOS terminal we need to deactivate line wrap via ANSI escape code
-        if platform.system() == "Darwin" and os.getenv("TERM_PROGRAM") == "Apple_Terminal":
+        if self.is_apple_terminal:
             os.write(self.output_fd, b"\033[?7l")
 
         self.screen = []
         self.height, self.width = self.getheightwidth()
 
-        self.__buffer = []
-
         self.posxy = 0, 0
         self.__gone_tall = 0
         self.__move = self.__move_short
@@ -371,13 +385,18 @@ class UnixConsole(Console):
         self.__disable_bracketed_paste()
         self.__maybe_write_code(self._rmkx)
         self.flushoutput()
-        tcsetattr(self.input_fd, termios.TCSADRAIN, self.__svtermstate)
+        self.__input_fd_set(self.__svtermstate)
 
-        if platform.system() == "Darwin" and os.getenv("TERM_PROGRAM") == "Apple_Terminal":
+        if self.is_apple_terminal:
             os.write(self.output_fd, b"\033[?7h")
 
         if hasattr(self, "old_sigwinch"):
-            signal.signal(signal.SIGWINCH, self.old_sigwinch)
+            try:
+                signal.signal(signal.SIGWINCH, self.old_sigwinch)
+            except ValueError as e:
+                import threading
+                if threading.current_thread() is threading.main_thread():
+                    raise e
             del self.old_sigwinch
 
     def push_char(self, char: int | bytes) -> None:
@@ -410,6 +429,8 @@ class UnixConsole(Console):
                             return self.event_queue.get()
                         else:
                             continue
+                    elif err.errno == errno.EIO:
+                        raise SystemExit(errno.EIO)
                     else:
                         raise
                 else:
@@ -422,7 +443,6 @@ class UnixConsole(Console):
         """
         return (
             not self.event_queue.empty()
-            or self.more_in_buffer()
             or bool(self.pollob.poll(timeout))
         )
 
@@ -525,6 +545,7 @@ class UnixConsole(Console):
                 e.raw += e.raw
 
             amount = struct.unpack("i", ioctl(self.input_fd, FIONREAD, b"\0\0\0\0"))[0]
+            trace("getpending({a})", a=amount)
             raw = self.__read(amount)
             data = str(raw, self.encoding, "replace")
             e.data += data
@@ -566,11 +587,9 @@ class UnixConsole(Console):
 
     @property
     def input_hook(self):
-        try:
-            import posix
-        except ImportError:
-            return None
-        if posix._is_inputhook_installed():
+        # avoid inline imports here so the repl doesn't get flooded
+        # with import logging from -X importtime=2
+        if posix is not None and posix._is_inputhook_installed():
             return posix._inputhook
 
     def __enable_bracketed_paste(self) -> None:
@@ -602,14 +621,14 @@ class UnixConsole(Console):
         if self._dch1:
             self.dch1 = self._dch1
         elif self._dch:
-            self.dch1 = curses.tparm(self._dch, 1)
+            self.dch1 = terminfo.tparm(self._dch, 1)
         else:
             self.dch1 = None
 
         if self._ich1:
             self.ich1 = self._ich1
         elif self._ich:
-            self.ich1 = curses.tparm(self._ich, 1)
+            self.ich1 = terminfo.tparm(self._ich, 1)
         else:
             self.ich1 = None
 
@@ -634,7 +653,7 @@ class UnixConsole(Console):
 
         # reuse the oldline as much as possible, but stop as soon as we
         # encounter an ESCAPE, because it might be the start of an escape
-        # sequene
+        # sequence
         while (
             x_coord < minlen
             and oldline[x_pos] == newline[x_pos]
@@ -706,7 +725,7 @@ class UnixConsole(Console):
         self.__buffer.append((text, 0))
 
     def __write_code(self, fmt, *args):
-        self.__buffer.append((curses.tparm(fmt, *args), 1))
+        self.__buffer.append((terminfo.tparm(fmt, *args), 1))
 
     def __maybe_write_code(self, fmt, *args):
         if fmt:
@@ -757,7 +776,6 @@ class UnixConsole(Console):
         self.__write_code(self._cup, y - self.__offset, x)
 
     def __sigwinch(self, signum, frame):
-        self.height, self.width = self.getheightwidth()
         self.event_queue.insert(Event("resize", None))
 
     def __hide_cursor(self):
@@ -790,9 +808,9 @@ class UnixConsole(Console):
         will never do anyone any good."""
         # using .get() means that things will blow up
         # only if the bps is actually needed (which I'm
-        # betting is pretty unlkely)
+        # betting is pretty unlikely)
         bps = ratedict.get(self.__svtermstate.ospeed)
-        while 1:
+        while True:
             m = prog.search(fmt)
             if not m:
                 os.write(self.output_fd, fmt)
@@ -808,3 +826,17 @@ class UnixConsole(Console):
                 os.write(self.output_fd, self._pad * nchars)
             else:
                 time.sleep(float(delay) / 1000.0)
+
+    def __input_fd_set(
+        self,
+        state: TermState,
+        ignore: AbstractSet[int] = _error_codes_to_ignore,
+    ) -> bool:
+        try:
+            tcsetattr(self.input_fd, termios.TCSADRAIN, state)
+        except termios.error as te:
+            if te.args[0] not in ignore:
+                raise
+            return False
+        else:
+            return True

Lib/_pyrepl/unix_eventqueue.py

@@ -18,12 +18,9 @@
 # CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
 # CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 
-from collections import deque
-
-from . import keymap
-from .console import Event
-from . import curses
+from .terminfo import TermInfo
 from .trace import trace
+from .base_eventqueue import BaseEventQueue
 from termios import tcgetattr, VERASE
 import os
 
@@ -57,96 +54,24 @@ CTRL_ARROW_KEYCODES= {
     b'\033Oc': 'ctrl right',
 }
 
-def get_terminal_keycodes() -> dict[bytes, str]:
+def get_terminal_keycodes(ti: TermInfo) -> dict[bytes, str]:
     """
     Generates a dictionary mapping terminal keycodes to human-readable names.
     """
     keycodes = {}
     for key, terminal_code in TERMINAL_KEYNAMES.items():
-        keycode = curses.tigetstr(terminal_code)
+        keycode = ti.get(terminal_code)
         trace('key {key} tiname {terminal_code} keycode {keycode!r}', **locals())
         if keycode:
             keycodes[keycode] = key
     keycodes.update(CTRL_ARROW_KEYCODES)
     return keycodes
 
-class EventQueue:
-    def __init__(self, fd: int, encoding: str) -> None:
-        self.keycodes = get_terminal_keycodes()
+
+class EventQueue(BaseEventQueue):
+    def __init__(self, fd: int, encoding: str, ti: TermInfo) -> None:
+        keycodes = get_terminal_keycodes(ti)
         if os.isatty(fd):
             backspace = tcgetattr(fd)[6][VERASE]
-            self.keycodes[backspace] = "backspace"
-        self.compiled_keymap = keymap.compile_keymap(self.keycodes)
-        self.keymap = self.compiled_keymap
-        trace("keymap {k!r}", k=self.keymap)
-        self.encoding = encoding
-        self.events: deque[Event] = deque()
-        self.buf = bytearray()
-
-    def get(self) -> Event | None:
-        """
-        Retrieves the next event from the queue.
-        """
-        if self.events:
-            return self.events.popleft()
-        else:
-            return None
-
-    def empty(self) -> bool:
-        """
-        Checks if the queue is empty.
-        """
-        return not self.events
-
-    def flush_buf(self) -> bytearray:
-        """
-        Flushes the buffer and returns its contents.
-        """
-        old = self.buf
-        self.buf = bytearray()
-        return old
-
-    def insert(self, event: Event) -> None:
-        """
-        Inserts an event into the queue.
-        """
-        trace('added event {event}', event=event)
-        self.events.append(event)
-
-    def push(self, char: int | bytes) -> None:
-        """
-        Processes a character by updating the buffer and handling special key mappings.
-        """
-        ord_char = char if isinstance(char, int) else ord(char)
-        char = bytes(bytearray((ord_char,)))
-        self.buf.append(ord_char)
-        if char in self.keymap:
-            if self.keymap is self.compiled_keymap:
-                #sanity check, buffer is empty when a special key comes
-                assert len(self.buf) == 1
-            k = self.keymap[char]
-            trace('found map {k!r}', k=k)
-            if isinstance(k, dict):
-                self.keymap = k
-            else:
-                self.insert(Event('key', k, self.flush_buf()))
-                self.keymap = self.compiled_keymap
-
-        elif self.buf and self.buf[0] == 27:  # escape
-            # escape sequence not recognized by our keymap: propagate it
-            # outside so that i can be recognized as an M-... key (see also
-            # the docstring in keymap.py
-            trace('unrecognized escape sequence, propagating...')
-            self.keymap = self.compiled_keymap
-            self.insert(Event('key', '\033', bytearray(b'\033')))
-            for _c in self.flush_buf()[1:]:
-                self.push(_c)
-
-        else:
-            try:
-                decoded = bytes(self.buf).decode(self.encoding)
-            except UnicodeError:
-                return
-            else:
-                self.insert(Event('key', decoded, self.flush_buf()))
-            self.keymap = self.compiled_keymap
+            keycodes[backspace] = "backspace"
+        BaseEventQueue.__init__(self, encoding, keycodes)

Lib/_pyrepl/utils.py

@@ -1,25 +1,391 @@
-import re
-import unicodedata
+from __future__ import annotations
+import builtins
 import functools
+import keyword
+import re
+import token as T
+import tokenize
+import unicodedata
+import _colorize
+
+from collections import deque
+from io import StringIO
+from tokenize import TokenInfo as TI
+from typing import Iterable, Iterator, Match, NamedTuple, Self
+
+from .types import CharBuffer, CharWidths
+from .trace import trace
 
 ANSI_ESCAPE_SEQUENCE = re.compile(r"\x1b\[[ -@]*[A-~]")
+ZERO_WIDTH_BRACKET = re.compile(r"\x01.*?\x02")
+ZERO_WIDTH_TRANS = str.maketrans({"\x01": "", "\x02": ""})
+IDENTIFIERS_AFTER = {"def", "class"}
+KEYWORD_CONSTANTS = {"True", "False", "None"}
+BUILTINS = {str(name) for name in dir(builtins) if not name.startswith('_')}
+
+
+def THEME(**kwargs):
+    # Not cached: the user can modify the theme inside the interactive session.
+    return _colorize.get_theme(**kwargs).syntax
+
+
+class Span(NamedTuple):
+    """Span indexing that's inclusive on both ends."""
+
+    start: int
+    end: int
+
+    @classmethod
+    def from_re(cls, m: Match[str], group: int | str) -> Self:
+        re_span = m.span(group)
+        return cls(re_span[0], re_span[1] - 1)
+
+    @classmethod
+    def from_token(cls, token: TI, line_len: list[int]) -> Self:
+        end_offset = -1
+        if (token.type in {T.FSTRING_MIDDLE, T.TSTRING_MIDDLE}
+            and token.string.endswith(("{", "}"))):
+            # gh-134158: a visible trailing brace comes from a double brace in input
+            end_offset += 1
+
+        return cls(
+            line_len[token.start[0] - 1] + token.start[1],
+            line_len[token.end[0] - 1] + token.end[1] + end_offset,
+        )
+
+
+class ColorSpan(NamedTuple):
+    span: Span
+    tag: str
 
 
 @functools.cache
 def str_width(c: str) -> int:
     if ord(c) < 128:
         return 1
+    # gh-139246 for zero-width joiner and combining characters
+    if unicodedata.combining(c):
+        return 0
+    category = unicodedata.category(c)
+    if category == "Cf" and c != "\u00ad":
+        return 0
     w = unicodedata.east_asian_width(c)
-    if w in ('N', 'Na', 'H', 'A'):
+    if w in ("N", "Na", "H", "A"):
         return 1
     return 2
 
 
 def wlen(s: str) -> int:
-    if len(s) == 1 and s != '\x1a':
+    if len(s) == 1 and s != "\x1a":
         return str_width(s)
     length = sum(str_width(i) for i in s)
     # remove lengths of any escape sequences
     sequence = ANSI_ESCAPE_SEQUENCE.findall(s)
-    ctrl_z_cnt = s.count('\x1a')
+    ctrl_z_cnt = s.count("\x1a")
     return length - sum(len(i) for i in sequence) + ctrl_z_cnt
+
+
+def unbracket(s: str, including_content: bool = False) -> str:
+    r"""Return `s` with \001 and \002 characters removed.
+
+    If `including_content` is True, content between \001 and \002 is also
+    stripped.
+    """
+    if including_content:
+        return ZERO_WIDTH_BRACKET.sub("", s)
+    return s.translate(ZERO_WIDTH_TRANS)
+
+
+def gen_colors(buffer: str) -> Iterator[ColorSpan]:
+    """Returns a list of index spans to color using the given color tag.
+
+    The input `buffer` should be a valid start of a Python code block, i.e.
+    it cannot be a block starting in the middle of a multiline string.
+    """
+    sio = StringIO(buffer)
+    line_lengths = [0] + [len(line) for line in sio.readlines()]
+    # make line_lengths cumulative
+    for i in range(1, len(line_lengths)):
+        line_lengths[i] += line_lengths[i-1]
+
+    sio.seek(0)
+    gen = tokenize.generate_tokens(sio.readline)
+    last_emitted: ColorSpan | None = None
+    try:
+        for color in gen_colors_from_token_stream(gen, line_lengths):
+            yield color
+            last_emitted = color
+    except SyntaxError:
+        return
+    except tokenize.TokenError as te:
+        yield from recover_unterminated_string(
+            te, line_lengths, last_emitted, buffer
+        )
+
+
+def recover_unterminated_string(
+    exc: tokenize.TokenError,
+    line_lengths: list[int],
+    last_emitted: ColorSpan | None,
+    buffer: str,
+) -> Iterator[ColorSpan]:
+    msg, loc = exc.args
+    if loc is None:
+        return
+
+    line_no, column = loc
+
+    if msg.startswith(
+        (
+            "unterminated string literal",
+            "unterminated f-string literal",
+            "unterminated t-string literal",
+            "EOF in multi-line string",
+            "unterminated triple-quoted f-string literal",
+            "unterminated triple-quoted t-string literal",
+        )
+    ):
+        start = line_lengths[line_no - 1] + column - 1
+        end = line_lengths[-1] - 1
+
+        # in case FSTRING_START was already emitted
+        if last_emitted and start <= last_emitted.span.start:
+            trace("before last emitted = {s}", s=start)
+            start = last_emitted.span.end + 1
+
+        span = Span(start, end)
+        trace("yielding span {a} -> {b}", a=span.start, b=span.end)
+        yield ColorSpan(span, "string")
+    else:
+        trace(
+            "unhandled token error({buffer}) = {te}",
+            buffer=repr(buffer),
+            te=str(exc),
+        )
+
+
+def gen_colors_from_token_stream(
+    token_generator: Iterator[TI],
+    line_lengths: list[int],
+) -> Iterator[ColorSpan]:
+    token_window = prev_next_window(token_generator)
+
+    is_def_name = False
+    bracket_level = 0
+    for prev_token, token, next_token in token_window:
+        assert token is not None
+        if token.start == token.end:
+            continue
+
+        match token.type:
+            case (
+                T.STRING
+                | T.FSTRING_START | T.FSTRING_MIDDLE | T.FSTRING_END
+                | T.TSTRING_START | T.TSTRING_MIDDLE | T.TSTRING_END
+            ):
+                span = Span.from_token(token, line_lengths)
+                yield ColorSpan(span, "string")
+            case T.COMMENT:
+                span = Span.from_token(token, line_lengths)
+                yield ColorSpan(span, "comment")
+            case T.NUMBER:
+                span = Span.from_token(token, line_lengths)
+                yield ColorSpan(span, "number")
+            case T.OP:
+                if token.string in "([{":
+                    bracket_level += 1
+                elif token.string in ")]}":
+                    bracket_level -= 1
+                span = Span.from_token(token, line_lengths)
+                yield ColorSpan(span, "op")
+            case T.NAME:
+                if is_def_name:
+                    is_def_name = False
+                    span = Span.from_token(token, line_lengths)
+                    yield ColorSpan(span, "definition")
+                elif keyword.iskeyword(token.string):
+                    span_cls = "keyword"
+                    if token.string in KEYWORD_CONSTANTS:
+                        span_cls = "keyword_constant"
+                    span = Span.from_token(token, line_lengths)
+                    yield ColorSpan(span, span_cls)
+                    if token.string in IDENTIFIERS_AFTER:
+                        is_def_name = True
+                elif (
+                    keyword.issoftkeyword(token.string)
+                    and bracket_level == 0
+                    and is_soft_keyword_used(prev_token, token, next_token)
+                ):
+                    span = Span.from_token(token, line_lengths)
+                    yield ColorSpan(span, "soft_keyword")
+                elif (
+                    token.string in BUILTINS
+                    and not (prev_token and prev_token.exact_type == T.DOT)
+                ):
+                    span = Span.from_token(token, line_lengths)
+                    yield ColorSpan(span, "builtin")
+
+
+keyword_first_sets_match = {"False", "None", "True", "await", "lambda", "not"}
+keyword_first_sets_case = {"False", "None", "True"}
+
+
+def is_soft_keyword_used(*tokens: TI | None) -> bool:
+    """Returns True if the current token is a keyword in this context.
+
+    For the `*tokens` to match anything, they have to be a three-tuple of
+    (previous, current, next).
+    """
+    trace("is_soft_keyword_used{t}", t=tokens)
+    match tokens:
+        case (
+            None | TI(T.NEWLINE) | TI(T.INDENT) | TI(string=":"),
+            TI(string="match"),
+            TI(T.NUMBER | T.STRING | T.FSTRING_START | T.TSTRING_START)
+            | TI(T.OP, string="(" | "*" | "[" | "{" | "~" | "...")
+        ):
+            return True
+        case (
+            None | TI(T.NEWLINE) | TI(T.INDENT) | TI(string=":"),
+            TI(string="match"),
+            TI(T.NAME, string=s)
+        ):
+            if keyword.iskeyword(s):
+                return s in keyword_first_sets_match
+            return True
+        case (
+            None | TI(T.NEWLINE) | TI(T.INDENT) | TI(T.DEDENT) | TI(string=":"),
+            TI(string="case"),
+            TI(T.NUMBER | T.STRING | T.FSTRING_START | T.TSTRING_START)
+            | TI(T.OP, string="(" | "*" | "-" | "[" | "{")
+        ):
+            return True
+        case (
+            None | TI(T.NEWLINE) | TI(T.INDENT) | TI(T.DEDENT) | TI(string=":"),
+            TI(string="case"),
+            TI(T.NAME, string=s)
+        ):
+            if keyword.iskeyword(s):
+                return s in keyword_first_sets_case
+            return True
+        case (TI(string="case"), TI(string="_"), TI(string=":")):
+            return True
+        case (
+            None | TI(T.NEWLINE) | TI(T.INDENT) | TI(T.DEDENT) | TI(string=":"),
+            TI(string="type"),
+            TI(T.NAME, string=s)
+        ):
+            return not keyword.iskeyword(s)
+        case _:
+            return False
+
+
+def disp_str(
+    buffer: str,
+    colors: list[ColorSpan] | None = None,
+    start_index: int = 0,
+    force_color: bool = False,
+) -> tuple[CharBuffer, CharWidths]:
+    r"""Decompose the input buffer into a printable variant with applied colors.
+
+    Returns a tuple of two lists:
+    - the first list is the input buffer, character by character, with color
+      escape codes added (while those codes contain multiple ASCII characters,
+      each code is considered atomic *and is attached for the corresponding
+      visible character*);
+    - the second list is the visible width of each character in the input
+      buffer.
+
+    Note on colors:
+    - The `colors` list, if provided, is partially consumed within. We're using
+      a list and not a generator since we need to hold onto the current
+      unfinished span between calls to disp_str in case of multiline strings.
+    - The `colors` list is computed from the start of the input block. `buffer`
+      is only a subset of that input block, a single line within. This is why
+      we need `start_index` to inform us which position is the start of `buffer`
+      actually within user input. This allows us to match color spans correctly.
+
+    Examples:
+    >>> utils.disp_str("a = 9")
+    (['a', ' ', '=', ' ', '9'], [1, 1, 1, 1, 1])
+
+    >>> line = "while 1:"
+    >>> colors = list(utils.gen_colors(line))
+    >>> utils.disp_str(line, colors=colors)
+    (['\x1b[1;34mw', 'h', 'i', 'l', 'e\x1b[0m', ' ', '1', ':'], [1, 1, 1, 1, 1, 1, 1, 1])
+
+    """
+    chars: CharBuffer = []
+    char_widths: CharWidths = []
+
+    if not buffer:
+        return chars, char_widths
+
+    while colors and colors[0].span.end < start_index:
+        # move past irrelevant spans
+        colors.pop(0)
+
+    theme = THEME(force_color=force_color)
+    pre_color = ""
+    post_color = ""
+    if colors and colors[0].span.start < start_index:
+        # looks like we're continuing a previous color (e.g. a multiline str)
+        pre_color = theme[colors[0].tag]
+
+    for i, c in enumerate(buffer, start_index):
+        if colors and colors[0].span.start == i:  # new color starts now
+            pre_color = theme[colors[0].tag]
+
+        if c == "\x1a":  # CTRL-Z on Windows
+            chars.append(c)
+            char_widths.append(2)
+        elif ord(c) < 128:
+            chars.append(c)
+            char_widths.append(1)
+        elif unicodedata.category(c).startswith("C"):
+            c = r"\u%04x" % ord(c)
+            chars.append(c)
+            char_widths.append(len(c))
+        else:
+            chars.append(c)
+            char_widths.append(str_width(c))
+
+        if colors and colors[0].span.end == i:  # current color ends now
+            post_color = theme.reset
+            colors.pop(0)
+
+        chars[-1] = pre_color + chars[-1] + post_color
+        pre_color = ""
+        post_color = ""
+
+    if colors and colors[0].span.start < i and colors[0].span.end > i:
+        # even though the current color should be continued, reset it for now.
+        # the next call to `disp_str()` will revive it.
+        chars[-1] += theme.reset
+
+    return chars, char_widths
+
+
+def prev_next_window[T](
+    iterable: Iterable[T]
+) -> Iterator[tuple[T | None, ...]]:
+    """Generates three-tuples of (previous, current, next) items.
+
+    On the first iteration previous is None. On the last iteration next
+    is None. In case of exception next is None and the exception is re-raised
+    on a subsequent next() call.
+
+    Inspired by `sliding_window` from `itertools` recipes.
+    """
+
+    iterator = iter(iterable)
+    window = deque((None, next(iterator)), maxlen=3)
+    try:
+        for x in iterator:
+            window.append(x)
+            yield tuple(window)
+    except Exception:
+        raise
+    finally:
+        window.append(None)
+        yield tuple(window)

Lib/_pyrepl/windows_console.py

@@ -22,11 +22,9 @@ from __future__ import annotations
 import io
 import os
 import sys
-import time
-import msvcrt
 
-from collections import deque
 import ctypes
+import types
 from ctypes.wintypes import (
     _COORD,
     WORD,
@@ -42,9 +40,10 @@ from ctypes import Structure, POINTER, Union
 from .console import Event, Console
 from .trace import trace
 from .utils import wlen
+from .windows_eventqueue import EventQueue
 
 try:
-    from ctypes import GetLastError, WinDLL, windll, WinError  # type: ignore[attr-defined]
+    from ctypes import get_last_error, GetLastError, WinDLL, windll, WinError  # type: ignore[attr-defined]
 except:
     # Keep MyPy happy off Windows
     from ctypes import CDLL as WinDLL, cdll as windll
@@ -52,11 +51,20 @@ except:
     def GetLastError() -> int:
         return 42
 
+    def get_last_error() -> int:
+        return 42
+
     class WinError(OSError):  # type: ignore[no-redef]
         def __init__(self, err: int | None, descr: str | None = None) -> None:
             self.err = err
             self.descr = descr
 
+# declare nt optional to allow None assignment on other platforms
+nt: types.ModuleType | None
+try:
+    import nt
+except ImportError:
+    nt = None
 
 TYPE_CHECKING = False
 
@@ -94,7 +102,9 @@ VK_MAP: dict[int, str] = {
     0x83: "f20",  # VK_F20
 }
 
-# Console escape codes: https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences
+# Virtual terminal output sequences
+# Reference: https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences#output-sequences
+# Check `windows_eventqueue.py` for input sequences
 ERASE_IN_LINE = "\x1b[K"
 MOVE_LEFT = "\x1b[{}D"
 MOVE_RIGHT = "\x1b[{}C"
@@ -102,10 +112,25 @@ MOVE_UP = "\x1b[{}A"
 MOVE_DOWN = "\x1b[{}B"
 CLEAR = "\x1b[H\x1b[J"
 
+# State of control keys: https://learn.microsoft.com/en-us/windows/console/key-event-record-str
+ALT_ACTIVE = 0x01 | 0x02
+CTRL_ACTIVE = 0x04 | 0x08
+
+WAIT_TIMEOUT = 0x102
+WAIT_FAILED = 0xFFFFFFFF
+
+# from winbase.h
+INFINITE = 0xFFFFFFFF
+
 
 class _error(Exception):
     pass
 
+def _supports_vt():
+    try:
+        return nt._supports_virtual_terminal()
+    except AttributeError:
+        return False
 
 class WindowsConsole(Console):
     def __init__(
@@ -117,17 +142,29 @@ class WindowsConsole(Console):
     ):
         super().__init__(f_in, f_out, term, encoding)
 
+        self.__vt_support = _supports_vt()
+
+        if self.__vt_support:
+            trace('console supports virtual terminal')
+
+        # Save original console modes so we can recover on cleanup.
+        original_input_mode = DWORD()
+        GetConsoleMode(InHandle, original_input_mode)
+        trace(f'saved original input mode 0x{original_input_mode.value:x}')
+        self.__original_input_mode = original_input_mode.value
+
         SetConsoleMode(
             OutHandle,
             ENABLE_WRAP_AT_EOL_OUTPUT
             | ENABLE_PROCESSED_OUTPUT
             | ENABLE_VIRTUAL_TERMINAL_PROCESSING,
         )
+
         self.screen: list[str] = []
         self.width = 80
         self.height = 25
         self.__offset = 0
-        self.event_queue: deque[Event] = deque()
+        self.event_queue = EventQueue(encoding)
         try:
             self.out = io._WindowsConsoleIO(self.output_fd, "w")  # type: ignore[attr-defined]
         except ValueError:
@@ -146,8 +183,9 @@ class WindowsConsole(Console):
 
         while len(self.screen) < min(len(screen), self.height):
             self._hide_cursor()
-            self._move_relative(0, len(self.screen) - 1)
-            self.__write("\n")
+            if self.screen:
+                self._move_relative(0, len(self.screen) - 1)
+                self.__write("\n")
             self.posxy = 0, len(self.screen)
             self.screen.append("")
 
@@ -204,11 +242,9 @@ class WindowsConsole(Console):
 
     @property
     def input_hook(self):
-        try:
-            import nt
-        except ImportError:
-            return None
-        if nt._is_inputhook_installed():
+        # avoid inline imports here so the repl doesn't get flooded
+        # with import logging from -X importtime=2
+        if nt is not None and nt._is_inputhook_installed():
             return nt._inputhook
 
     def __write_changed_line(
@@ -232,7 +268,7 @@ class WindowsConsole(Console):
 
         # reuse the oldline as much as possible, but stop as soon as we
         # encounter an ESCAPE, because it might be the start of an escape
-        # sequene
+        # sequence
         while (
             x_coord < minlen
             and oldline[x_pos] == newline[x_pos]
@@ -247,18 +283,13 @@ class WindowsConsole(Console):
             self._erase_to_end()
 
         self.__write(newline[x_pos:])
-        if wlen(newline) == self.width:
-            # If we wrapped we want to start at the next line
-            self._move_relative(0, y + 1)
-            self.posxy = 0, y + 1
-        else:
-            self.posxy = wlen(newline), y
+        self.posxy = min(wlen(newline), self.width - 1), y
 
-            if "\x1b" in newline or y != self.posxy[1] or '\x1a' in newline:
-                # ANSI escape characters are present, so we can't assume
-                # anything about the position of the cursor.  Moving the cursor
-                # to the left margin should work to get to a known position.
-                self.move_cursor(0, y)
+        if "\x1b" in newline or y != self.posxy[1] or '\x1a' in newline:
+            # ANSI escape characters are present, so we can't assume
+            # anything about the position of the cursor.  Moving the cursor
+            # to the left margin should work to get to a known position.
+            self.move_cursor(0, y)
 
     def _scroll(
         self, top: int, bottom: int, left: int | None = None, right: int | None = None
@@ -291,6 +322,12 @@ class WindowsConsole(Console):
     def _disable_blinking(self):
         self.__write("\x1b[?12l")
 
+    def _enable_bracketed_paste(self) -> None:
+        self.__write("\x1b[?2004h")
+
+    def _disable_bracketed_paste(self) -> None:
+        self.__write("\x1b[?2004l")
+
     def __write(self, text: str) -> None:
         if "\x1a" in text:
             text = ''.join(["^Z" if x == '\x1a' else x for x in text])
@@ -320,8 +357,15 @@ class WindowsConsole(Console):
         self.__gone_tall = 0
         self.__offset = 0
 
+        if self.__vt_support:
+            SetConsoleMode(InHandle, self.__original_input_mode | ENABLE_VIRTUAL_TERMINAL_INPUT)
+            self._enable_bracketed_paste()
+
     def restore(self) -> None:
-        pass
+        if self.__vt_support:
+            # Recover to original mode before running REPL
+            self._disable_bracketed_paste()
+            SetConsoleMode(InHandle, self.__original_input_mode)
 
     def _move_relative(self, x: int, y: int) -> None:
         """Moves relative to the current posxy"""
@@ -342,7 +386,7 @@ class WindowsConsole(Console):
             raise ValueError(f"Bad cursor position {x}, {y}")
 
         if y < self.__offset or y >= self.__offset + self.height:
-            self.event_queue.insert(0, Event("scroll", ""))
+            self.event_queue.insert(Event("scroll", ""))
         else:
             self._move_relative(x, y)
             self.posxy = x, y
@@ -371,14 +415,7 @@ class WindowsConsole(Console):
 
         return info.srWindow.Bottom  # type: ignore[no-any-return]
 
-    def _read_input(self, block: bool = True) -> INPUT_RECORD | None:
-        if not block:
-            events = DWORD()
-            if not GetNumberOfConsoleInputEvents(InHandle, events):
-                raise WinError(GetLastError())
-            if not events.value:
-                return None
-
+    def _read_input(self) -> INPUT_RECORD | None:
         rec = INPUT_RECORD()
         read = DWORD()
         if not ReadConsoleInput(InHandle, rec, 1, read):
@@ -386,15 +423,26 @@ class WindowsConsole(Console):
 
         return rec
 
+    def _read_input_bulk(
+        self, n: int
+    ) -> tuple[ctypes.Array[INPUT_RECORD], int]:
+        rec = (n * INPUT_RECORD)()
+        read = DWORD()
+        if not ReadConsoleInput(InHandle, rec, n, read):
+            raise WinError(GetLastError())
+
+        return rec, read.value
+
     def get_event(self, block: bool = True) -> Event | None:
         """Return an Event instance.  Returns None if |block| is false
         and there is no event pending, otherwise waits for the
         completion of an event."""
-        if self.event_queue:
-            return self.event_queue.pop()
 
-        while True:
-            rec = self._read_input(block)
+        if not block and not self.wait(timeout=0):
+            return None
+
+        while self.event_queue.empty():
+            rec = self._read_input()
             if rec is None:
                 return None
 
@@ -407,31 +455,47 @@ class WindowsConsole(Console):
                     continue
                 return None
 
-            key = rec.Event.KeyEvent.uChar.UnicodeChar
+            key_event = rec.Event.KeyEvent
+            raw_key = key = key_event.uChar.UnicodeChar
 
-            if rec.Event.KeyEvent.uChar.UnicodeChar == "\r":
-                # Make enter make unix-like
-                return Event(evt="key", data="\n", raw=b"\n")
-            elif rec.Event.KeyEvent.wVirtualKeyCode == 8:
+            if key == "\r":
+                # Make enter unix-like
+                return Event(evt="key", data="\n")
+            elif key_event.wVirtualKeyCode == 8:
                 # Turn backspace directly into the command
-                return Event(
-                    evt="key",
-                    data="backspace",
-                    raw=rec.Event.KeyEvent.uChar.UnicodeChar,
-                )
-            elif rec.Event.KeyEvent.uChar.UnicodeChar == "\x00":
+                key = "backspace"
+            elif key == "\x00":
                 # Handle special keys like arrow keys and translate them into the appropriate command
-                code = VK_MAP.get(rec.Event.KeyEvent.wVirtualKeyCode)
-                if code:
-                    return Event(
-                        evt="key", data=code, raw=rec.Event.KeyEvent.uChar.UnicodeChar
-                    )
+                key = VK_MAP.get(key_event.wVirtualKeyCode)
+                if key:
+                    if key_event.dwControlKeyState & CTRL_ACTIVE:
+                        key = f"ctrl {key}"
+                    elif key_event.dwControlKeyState & ALT_ACTIVE:
+                        # queue the key, return the meta command
+                        self.event_queue.insert(Event(evt="key", data=key))
+                        return Event(evt="key", data="\033")  # keymap.py uses this for meta
+                    return Event(evt="key", data=key)
                 if block:
                     continue
 
                 return None
+            elif self.__vt_support:
+                # If virtual terminal is enabled, scanning VT sequences
+                for char in raw_key.encode(self.event_queue.encoding, "replace"):
+                    self.event_queue.push(char)
+                continue
 
-            return Event(evt="key", data=key, raw=rec.Event.KeyEvent.uChar.UnicodeChar)
+            if key_event.dwControlKeyState & ALT_ACTIVE:
+                # Do not swallow characters that have been entered via AltGr:
+                # Windows internally converts AltGr to CTRL+ALT, see
+                # https://learn.microsoft.com/en-us/windows/win32/api/winuser/nf-winuser-vkkeyscanw
+                if not key_event.dwControlKeyState & CTRL_ACTIVE:
+                    # queue the key, return the meta command
+                    self.event_queue.insert(Event(evt="key", data=key))
+                    return Event(evt="key", data="\033")  # keymap.py uses this for meta
+
+            return Event(evt="key", data=key)
+        return self.event_queue.get()
 
     def push_char(self, char: int | bytes) -> None:
         """
@@ -446,7 +510,7 @@ class WindowsConsole(Console):
         """Wipe the screen"""
         self.__write(CLEAR)
         self.posxy = 0, 0
-        self.screen = [""]
+        self.screen = []
 
     def finish(self) -> None:
         """Move the cursor to the end of the display and otherwise get
@@ -472,18 +536,53 @@ class WindowsConsole(Console):
     def getpending(self) -> Event:
         """Return the characters that have been typed but not yet
         processed."""
-        return Event("key", "", b"")
+        e = Event("key", "", b"")
+
+        while not self.event_queue.empty():
+            e2 = self.event_queue.get()
+            if e2:
+                e.data += e2.data
+
+        recs, rec_count = self._read_input_bulk(1024)
+        for i in range(rec_count):
+            rec = recs[i]
+            # In case of a legacy console, we do not only receive a keydown
+            # event, but also a keyup event - and for uppercase letters
+            # an additional SHIFT_PRESSED event.
+            if rec and rec.EventType == KEY_EVENT:
+                key_event = rec.Event.KeyEvent
+                if not key_event.bKeyDown:
+                    continue
+                ch = key_event.uChar.UnicodeChar
+                if ch == "\x00":
+                    # ignore SHIFT_PRESSED and special keys
+                    continue
+                if ch == "\r":
+                    ch += "\n"
+                e.data += ch
+        return e
+
+    def wait_for_event(self, timeout: float | None) -> bool:
+        """Wait for an event."""
+        if timeout is None:
+            timeout = INFINITE
+        else:
+            timeout = int(timeout)
+        ret = WaitForSingleObject(InHandle, timeout)
+        if ret == WAIT_FAILED:
+            raise WinError(get_last_error())
+        elif ret == WAIT_TIMEOUT:
+            return False
+        return True
 
     def wait(self, timeout: float | None) -> bool:
-        """Wait for an event."""
-        # Poor man's Windows select loop
-        start_time = time.time()
-        while True:
-            if msvcrt.kbhit(): # type: ignore[attr-defined]
-                return True
-            if timeout and time.time() - start_time > timeout / 1000:
-                return False
-            time.sleep(0.01)
+        """
+        Wait for events on the console.
+        """
+        return (
+            not self.event_queue.empty()
+            or self.wait_for_event(timeout)
+        )
 
     def repaint(self) -> None:
         raise NotImplementedError("No repaint support")
@@ -553,6 +652,13 @@ MENU_EVENT = 0x08
 MOUSE_EVENT = 0x02
 WINDOW_BUFFER_SIZE_EVENT = 0x04
 
+ENABLE_PROCESSED_INPUT = 0x0001
+ENABLE_LINE_INPUT = 0x0002
+ENABLE_ECHO_INPUT = 0x0004
+ENABLE_MOUSE_INPUT = 0x0010
+ENABLE_INSERT_MODE = 0x0020
+ENABLE_VIRTUAL_TERMINAL_INPUT = 0x0200
+
 ENABLE_PROCESSED_OUTPUT = 0x01
 ENABLE_WRAP_AT_EOL_OUTPUT = 0x02
 ENABLE_VIRTUAL_TERMINAL_PROCESSING = 0x04
@@ -584,6 +690,10 @@ if sys.platform == "win32":
     ]
     ScrollConsoleScreenBuffer.restype = BOOL
 
+    GetConsoleMode = _KERNEL32.GetConsoleMode
+    GetConsoleMode.argtypes = [HANDLE, POINTER(DWORD)]
+    GetConsoleMode.restype = BOOL
+
     SetConsoleMode = _KERNEL32.SetConsoleMode
     SetConsoleMode.argtypes = [HANDLE, DWORD]
     SetConsoleMode.restype = BOOL
@@ -592,14 +702,15 @@ if sys.platform == "win32":
     ReadConsoleInput.argtypes = [HANDLE, POINTER(INPUT_RECORD), DWORD, POINTER(DWORD)]
     ReadConsoleInput.restype = BOOL
 
-    GetNumberOfConsoleInputEvents = _KERNEL32.GetNumberOfConsoleInputEvents
-    GetNumberOfConsoleInputEvents.argtypes = [HANDLE, POINTER(DWORD)]
-    GetNumberOfConsoleInputEvents.restype = BOOL
 
     FlushConsoleInputBuffer = _KERNEL32.FlushConsoleInputBuffer
     FlushConsoleInputBuffer.argtypes = [HANDLE]
     FlushConsoleInputBuffer.restype = BOOL
 
+    WaitForSingleObject = _KERNEL32.WaitForSingleObject
+    WaitForSingleObject.argtypes = [HANDLE, DWORD]
+    WaitForSingleObject.restype = DWORD
+
     OutHandle = GetStdHandle(STD_OUTPUT_HANDLE)
     InHandle = GetStdHandle(STD_INPUT_HANDLE)
 else:
@@ -610,9 +721,10 @@ else:
     GetStdHandle = _win_only
     GetConsoleScreenBufferInfo = _win_only
     ScrollConsoleScreenBuffer = _win_only
+    GetConsoleMode = _win_only
     SetConsoleMode = _win_only
     ReadConsoleInput = _win_only
-    GetNumberOfConsoleInputEvents = _win_only
     FlushConsoleInputBuffer = _win_only
+    WaitForSingleObject = _win_only
     OutHandle = 0
     InHandle = 0

Lib/_pyrepl/windows_eventqueue.py

@@ -0,0 +1,42 @@
+"""
+Windows event and VT sequence scanner
+"""
+
+from .base_eventqueue import BaseEventQueue
+
+
+# Reference: https://learn.microsoft.com/en-us/windows/console/console-virtual-terminal-sequences#input-sequences
+VT_MAP: dict[bytes, str] = {
+    b'\x1b[A': 'up',
+    b'\x1b[B': 'down',
+    b'\x1b[C': 'right',
+    b'\x1b[D': 'left',
+    b'\x1b[1;5D': 'ctrl left',
+    b'\x1b[1;5C': 'ctrl right',
+
+    b'\x1b[H': 'home',
+    b'\x1b[F': 'end',
+
+    b'\x7f': 'backspace',
+    b'\x1b[2~': 'insert',
+    b'\x1b[3~': 'delete',
+    b'\x1b[5~': 'page up',
+    b'\x1b[6~': 'page down',
+
+    b'\x1bOP':   'f1',
+    b'\x1bOQ':   'f2',
+    b'\x1bOR':   'f3',
+    b'\x1bOS':   'f4',
+    b'\x1b[15~': 'f5',
+    b'\x1b[17~': 'f6',
+    b'\x1b[18~': 'f7',
+    b'\x1b[19~': 'f8',
+    b'\x1b[20~': 'f9',
+    b'\x1b[21~': 'f10',
+    b'\x1b[23~': 'f11',
+    b'\x1b[24~': 'f12',
+}
+
+class EventQueue(BaseEventQueue):
+    def __init__(self, encoding: str) -> None:
+        BaseEventQueue.__init__(self, encoding, VT_MAP)

Lib/asyncio/__main__.py

@@ -86,22 +86,27 @@ class REPLThread(threading.Thread):
         global return_code
 
         try:
-            banner = (
-                f'asyncio REPL {sys.version} on {sys.platform}\n'
-                f'Use "await" directly instead of "asyncio.run()".\n'
-                f'Type "help", "copyright", "credits" or "license" '
-                f'for more information.\n'
-            )
+            if not sys.flags.quiet:
+                banner = (
+                    f'asyncio REPL {sys.version} on {sys.platform}\n'
+                    f'Use "await" directly instead of "asyncio.run()".\n'
+                    f'Type "help", "copyright", "credits" or "license" '
+                    f'for more information.\n'
+                )
 
-            console.write(banner)
+                console.write(banner)
 
-            if startup_path := os.getenv("PYTHONSTARTUP"):
+            if not sys.flags.isolated and (startup_path := os.getenv("PYTHONSTARTUP")):
                 sys.audit("cpython.run_startup", startup_path)
-
-                import tokenize
-                with tokenize.open(startup_path) as f:
-                    startup_code = compile(f.read(), startup_path, "exec")
+                try:
+                    import tokenize
+                    with tokenize.open(startup_path) as f:
+                        startup_code = compile(f.read(), startup_path, "exec")
                     exec(startup_code, console.locals)
+                except SystemExit:
+                    raise
+                except BaseException:
+                    console.showtraceback()
 
             ps1 = getattr(sys, "ps1", ">>> ")
             if CAN_USE_PYREPL:
@@ -236,4 +241,5 @@ if __name__ == '__main__':
             break
 
     console.write('exiting asyncio REPL...\n')
+    loop.close()
     sys.exit(return_code)

Lib/asyncio/base_events.py

@@ -483,10 +483,10 @@ class BaseEventLoop(events.AbstractEventLoop):
         If factory is None the default task factory will be set.
 
         If factory is a callable, it should have a signature matching
-        '(loop, coro, **kwargs)', where 'loop' will be a reference to the active
-        event loop, 'coro' will be a coroutine object, and **kwargs will be
-        arbitrary keyword arguments that should be passed on to Task.
-        The callable must return a Task.
+        '(loop, coro, **kwargs)', where 'loop' will be a reference to the
+        active event loop, 'coro' will be a coroutine object, and **kwargs
+        will be arbitrary keyword arguments that should be passed on to
+        Task.  The callable must return a Task.
         """
         if factory is not None and not callable(factory):
             raise TypeError('task factory must be a callable or None')
@@ -721,8 +721,8 @@ class BaseEventLoop(events.AbstractEventLoop):
     def stop(self):
         """Stop running the event loop.
 
-        Every callback already scheduled will still run.  This simply informs
-        run_forever to stop looping after a complete iteration.
+        Every callback already scheduled will still run.  This simply
+        informs run_forever to stop looping after a complete iteration.
         """
         self._stopping = True
 
@@ -963,7 +963,7 @@ class BaseEventLoop(events.AbstractEventLoop):
             f"and file {file!r} combination")
 
     async def _sock_sendfile_fallback(self, sock, file, offset, count):
-        if offset:
+        if hasattr(file, 'seek'):
             file.seek(offset)
         blocksize = (
             min(count, constants.SENDFILE_FALLBACK_READBUFFER_SIZE)
@@ -1070,12 +1070,12 @@ class BaseEventLoop(events.AbstractEventLoop):
 
         Create a streaming transport connection to a given internet host and
         port: socket family AF_INET or socket.AF_INET6 depending on host (or
-        family if specified), socket type SOCK_STREAM. protocol_factory must be
-        a callable returning a protocol instance.
+        family if specified), socket type SOCK_STREAM. protocol_factory must
+        be a callable returning a protocol instance.
 
-        This method is a coroutine which will try to establish the connection
-        in the background.  When successful, the coroutine returns a
-        (transport, protocol) pair.
+        This method is a coroutine which will try to establish the
+        connection in the background.  When successful, the coroutine
+        returns a (transport, protocol) pair.
         """
         if server_hostname is not None and not ssl:
             raise ValueError('server_hostname is only meaningful with ssl')
@@ -1278,7 +1278,6 @@ class BaseEventLoop(events.AbstractEventLoop):
             raise RuntimeError(
                 f"fallback is disabled and native sendfile is not "
                 f"supported for transport {transport!r}")
-
         return await self._sendfile_fallback(transport, file,
                                              offset, count)
 
@@ -1287,7 +1286,7 @@ class BaseEventLoop(events.AbstractEventLoop):
             "sendfile syscall is not supported")
 
     async def _sendfile_fallback(self, transp, file, offset, count):
-        if offset:
+        if hasattr(file, 'seek'):
             file.seek(offset)
         blocksize = min(count, 16384) if count else 16384
         buf = bytearray(blocksize)
@@ -1345,6 +1344,17 @@ class BaseEventLoop(events.AbstractEventLoop):
         # have a chance to get called before "ssl_protocol.connection_made()".
         transport.pause_reading()
 
+        # gh-142352: move buffered StreamReader data to SSLProtocol
+        if server_side:
+            from .streams import StreamReaderProtocol
+            if isinstance(protocol, StreamReaderProtocol):
+                stream_reader = getattr(protocol, '_stream_reader', None)
+                if stream_reader is not None:
+                    buffer = stream_reader._buffer
+                    if buffer:
+                        ssl_protocol._incoming.write(buffer)
+                        buffer.clear()
+
         transport.set_protocol(ssl_protocol)
         conmade_cb = self.call_soon(ssl_protocol.connection_made, transport)
         resume_cb = self.call_soon(transport.resume_reading)
@@ -1532,11 +1542,11 @@ class BaseEventLoop(events.AbstractEventLoop):
         The host parameter can be a string, in that case the TCP server is
         bound to host and port.
 
-        The host parameter can also be a sequence of strings and in that case
-        the TCP server is bound to all hosts of the sequence. If a host
-        appears multiple times (possibly indirectly e.g. when hostnames
-        resolve to the same IP address), the server is only bound once to that
-        host.
+        The host parameter can also be a sequence of strings and in that
+        case the TCP server is bound to all hosts of the sequence.  If
+        a host appears multiple times (possibly indirectly e.g. when
+        hostnames resolve to the same IP address), the server is only bound
+        once to that host.
 
         Return a Server object which can be used to stop the service.
 

Lib/asyncio/base_subprocess.py

@@ -265,7 +265,7 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
             # to avoid hanging forever in self._wait as otherwise _exit_waiters
             # would never be woken up, we wake them up here.
             for waiter in self._exit_waiters:
-                if not waiter.cancelled():
+                if not waiter.done():
                     waiter.set_result(self._returncode)
         if all(p is not None and p.disconnected
                for p in self._pipes.values()):
@@ -278,7 +278,7 @@ class BaseSubprocessTransport(transports.SubprocessTransport):
         finally:
             # wake up futures waiting for wait()
             for waiter in self._exit_waiters:
-                if not waiter.cancelled():
+                if not waiter.done():
                     waiter.set_result(self._returncode)
             self._exit_waiters = None
             self._loop = None

Lib/asyncio/coroutines.py

@@ -18,8 +18,8 @@ _is_coroutine = object()
 
 
 def iscoroutinefunction(func):
-    import warnings
     """Return True if func is a decorated coroutine function."""
+    import warnings
     warnings._deprecated("asyncio.iscoroutinefunction",
                          f"{warnings._DEPRECATED_MSG}; "
                          "use inspect.iscoroutinefunction() instead",

Lib/asyncio/events.py

@@ -374,8 +374,8 @@ class AbstractEventLoop:
 
         If host is an empty string or None all interfaces are assumed
         and a list of multiple sockets will be returned (most likely
-        one for IPv4 and another one for IPv6). The host parameter can also be
-        a sequence (e.g. list) of hosts to bind to.
+        one for IPv4 and another one for IPv6). The host parameter can also
+        be a sequence (e.g. list) of hosts to bind to.
 
         family can be set to either AF_INET or AF_INET6 to force the
         socket to use IPv4 or IPv6. If not set it will be determined
@@ -415,8 +415,9 @@ class AbstractEventLoop:
 
         start_serving set to True (default) causes the created server
         to start accepting connections immediately.  When set to False,
-        the user should await Server.start_serving() or Server.serve_forever()
-        to make the server to start accepting connections.
+        the user should await Server.start_serving() or
+        Server.serve_forever() to make the server to start accepting
+        connections.
         """
         raise NotImplementedError
 
@@ -479,8 +480,9 @@ class AbstractEventLoop:
 
         start_serving set to True (default) causes the created server
         to start accepting connections immediately.  When set to False,
-        the user should await Server.start_serving() or Server.serve_forever()
-        to make the server to start accepting connections.
+        the user should await Server.start_serving() or
+        Server.serve_forever() to make the server to start accepting
+        connections.
         """
         raise NotImplementedError
 
@@ -511,8 +513,8 @@ class AbstractEventLoop:
 
         protocol_factory must be a callable returning a protocol instance.
 
-        socket family AF_INET, socket.AF_INET6 or socket.AF_UNIX depending on
-        host (or family if specified), socket type SOCK_DGRAM.
+        socket family AF_INET, socket.AF_INET6 or socket.AF_UNIX depending
+        on host (or family if specified), socket type SOCK_DGRAM.
 
         reuse_address tells the kernel to reuse a local socket in
         TIME_WAIT state, without waiting for its natural timeout to
@@ -552,7 +554,8 @@ class AbstractEventLoop:
     async def connect_write_pipe(self, protocol_factory, pipe):
         """Register write pipe in event loop.
 
-        protocol_factory should instantiate object with BaseProtocol interface.
+        protocol_factory should instantiate object with BaseProtocol
+        interface.
         Pipe is file-like object already switched to nonblocking.
         Return pair (transport, protocol), where transport support
         WriteTransport interface."""

Lib/asyncio/futures.py

@@ -392,7 +392,7 @@ def _chain_future(source, destination):
 
     def _call_check_cancel(destination):
         if destination.cancelled():
-            if source_loop is None or source_loop is dest_loop:
+            if source_loop is None or source_loop is events._get_running_loop():
                 source.cancel()
             else:
                 source_loop.call_soon_threadsafe(source.cancel)
@@ -401,7 +401,7 @@ def _chain_future(source, destination):
         if (destination.cancelled() and
                 dest_loop is not None and dest_loop.is_closed()):
             return
-        if dest_loop is None or dest_loop is source_loop:
+        if dest_loop is None or dest_loop is events._get_running_loop():
             _set_state(destination, source)
         else:
             if dest_loop.is_closed():

Lib/asyncio/graph.py

@@ -112,13 +112,13 @@ def capture_call_graph(
     optional keyword-only 'depth' argument can be used to skip the specified
     number of frames from top of the stack.
 
-    If the optional keyword-only 'limit' argument is provided, each call stack
-    in the resulting graph is truncated to include at most ``abs(limit)``
-    entries. If 'limit' is positive, the entries left are the closest to
-    the invocation point. If 'limit' is negative, the topmost entries are
-    left. If 'limit' is omitted or None, all entries are present.
-    If 'limit' is 0, the call stack is not captured at all, only
-    "awaited by" information is present.
+    If the optional keyword-only 'limit' argument is provided, each call
+    stack in the resulting graph is truncated to include at most
+    ``abs(limit)`` entries.  If 'limit' is positive, the entries left are
+    the closest to the invocation point.  If 'limit' is negative, the
+    topmost entries are left.  If 'limit' is omitted or None, all entries
+    are present.  If 'limit' is 0, the call stack is not captured at all,
+    only "awaited by" information is present.
     """
 
     loop = events._get_running_loop()

Lib/asyncio/locks.py

@@ -158,10 +158,10 @@ class Lock(_ContextManagerMixin, mixins._LoopBoundMixin):
 class Event(mixins._LoopBoundMixin):
     """Asynchronous equivalent to threading.Event.
 
-    Class implementing event objects. An event manages a flag that can be set
-    to true with the set() method and reset to false with the clear() method.
-    The wait() method blocks until the flag is true. The flag is initially
-    false.
+    Class implementing event objects.  An event manages a flag that can be
+    set to true with the set() method and reset to false with the clear()
+    method.  The wait() method blocks until the flag is true.  The flag is
+    initially false.
     """
 
     def __init__(self):
@@ -353,9 +353,9 @@ class Semaphore(_ContextManagerMixin, mixins._LoopBoundMixin):
     """A Semaphore implementation.
 
     A semaphore manages an internal counter which is decremented by each
-    acquire() call and incremented by each release() call. The counter
-    can never go below zero; when acquire() finds that it is zero, it blocks,
-    waiting until some other thread calls release().
+    acquire() call and incremented by each release() call.  The counter
+    can never go below zero; when acquire() finds that it is zero, it
+    blocks, waiting until some other thread calls release().
 
     Semaphores also support the context management protocol.
 
@@ -511,8 +511,8 @@ class Barrier(mixins._LoopBoundMixin):
     async def wait(self):
         """Wait for the barrier.
 
-        When the specified number of tasks have started waiting, they are all
-        simultaneously awoken.
+        When the specified number of tasks have started waiting, they are
+        all simultaneously awoken.
         Returns an unique and individual index number from 0 to 'parties-1'.
         """
         async with self._cond:

Lib/asyncio/proactor_events.py

@@ -756,8 +756,7 @@ class BaseProactorEventLoop(base_events.BaseEventLoop):
                 offset += blocksize
                 total_sent += blocksize
         finally:
-            if total_sent > 0:
-                file.seek(offset)
+            file.seek(offset)
 
     async def _sendfile_native(self, transp, file, offset, count):
         resume_reading = transp.is_reading()

Lib/asyncio/queues.py

@@ -33,11 +33,11 @@ class QueueShutDown(Exception):
 class Queue(mixins._LoopBoundMixin):
     """A queue, useful for coordinating producer and consumer coroutines.
 
-    If maxsize is less than or equal to zero, the queue size is infinite. If it
-    is an integer greater than 0, then "await put()" will block when the
-    queue reaches maxsize, until an item is removed by get().
+    If maxsize is less than or equal to zero, the queue size is infinite.
+    If it is an integer greater than 0, then "await put()" will block when
+    the queue reaches maxsize, until an item is removed by get().
 
-    Unlike the standard library Queue, you can reliably know this Queue's size
+    Unlike queue.Queue, you can reliably know this Queue's size
     with qsize(), since your single-threaded asyncio application won't be
     interrupted between calling qsize() and doing an operation on the Queue.
     """
@@ -174,8 +174,8 @@ class Queue(mixins._LoopBoundMixin):
 
         If queue is empty, wait until an item is available.
 
-        Raises QueueShutDown if the queue has been shut down and is empty, or
-        if the queue has been shut down immediately.
+        Raises QueueShutDown if the queue has been shut down and is empty,
+        or if the queue has been shut down immediately.
         """
         while self.empty():
             if self._is_shutdown and self.empty():
@@ -203,10 +203,11 @@ class Queue(mixins._LoopBoundMixin):
     def get_nowait(self):
         """Remove and return an item from the queue.
 
-        Return an item if one is immediately available, else raise QueueEmpty.
+        Return an item if one is immediately available, else raise
+        QueueEmpty.
 
-        Raises QueueShutDown if the queue has been shut down and is empty, or
-        if the queue has been shut down immediately.
+        Raises QueueShutDown if the queue has been shut down and is empty,
+        or if the queue has been shut down immediately.
         """
         if self.empty():
             if self._is_shutdown:
@@ -223,12 +224,12 @@ class Queue(mixins._LoopBoundMixin):
         a subsequent call to task_done() tells the queue that the processing
         on the task is complete.
 
-        If a join() is currently blocking, it will resume when all items have
-        been processed (meaning that a task_done() call was received for every
-        item that had been put() into the queue).
+        If a join() is currently blocking, it will resume when all items
+        have been processed (meaning that a task_done() call was received
+        for every item that had been put() into the queue).
 
-        Raises ValueError if called more times than there were items placed in
-        the queue.
+        Raises ValueError if called more times than there were items placed
+        in the queue.
         """
         if self._unfinished_tasks <= 0:
             raise ValueError('task_done() called too many times')
@@ -239,10 +240,11 @@ class Queue(mixins._LoopBoundMixin):
     async def join(self):
         """Block until all items in the queue have been gotten and processed.
 
-        The count of unfinished tasks goes up whenever an item is added to the
-        queue. The count goes down whenever a consumer calls task_done() to
-        indicate that the item was retrieved and all work on it is complete.
-        When the count of unfinished tasks drops to zero, join() unblocks.
+        The count of unfinished tasks goes up whenever an item is added to
+        the queue.  The count goes down whenever a consumer calls
+        task_done() to indicate that the item was retrieved and all work on
+        it is complete.  When the count of unfinished tasks drops to zero,
+        join() unblocks.
         """
         if self._unfinished_tasks > 0:
             await self._finished.wait()

Lib/asyncio/runners.py

@@ -35,7 +35,8 @@ class Runner:
     with asyncio.Runner(debug=True) as runner:
         runner.run(main())
 
-    The run() method can be called multiple times within the runner's context.
+    The run() method can be called multiple times within the runner's
+    context.
 
     This can be useful for interactive console (e.g. IPython),
     unittest runners, console tools, -- everywhere when async code

Lib/asyncio/selector_events.py

@@ -530,11 +530,12 @@ class BaseSelectorEventLoop(base_events.BaseEventLoop):
     async def sock_sendall(self, sock, data):
         """Send data to the socket.
 
-        The socket must be connected to a remote socket. This method continues
-        to send data from data until either all data has been sent or an
-        error occurs. None is returned on success. On error, an exception is
-        raised, and there is no way to determine how much data, if any, was
-        successfully processed by the receiving end of the connection.
+        The socket must be connected to a remote socket.  This method
+        continues to send data from data until either all data has been
+        sent or an error occurs.  None is returned on success.  On error,
+        an exception is raised, and there is no way to determine how much
+        data, if any, was successfully processed by the receiving end of
+        the connection.
         """
         base_events._check_ssl_socket(sock)
         if self._debug and sock.gettimeout() != 0:
@@ -583,11 +584,12 @@ class BaseSelectorEventLoop(base_events.BaseEventLoop):
     async def sock_sendto(self, sock, data, address):
         """Send data to the socket.
 
-        The socket must be connected to a remote socket. This method continues
-        to send data from data until either all data has been sent or an
-        error occurs. None is returned on success. On error, an exception is
-        raised, and there is no way to determine how much data, if any, was
-        successfully processed by the receiving end of the connection.
+        The socket must be connected to a remote socket.  This method
+        continues to send data from data until either all data has been
+        sent or an error occurs.  None is returned on success.  On error,
+        an exception is raised, and there is no way to determine how much
+        data, if any, was successfully processed by the receiving end of
+        the connection.
         """
         base_events._check_ssl_socket(sock)
         if self._debug and sock.gettimeout() != 0:
@@ -698,10 +700,11 @@ class BaseSelectorEventLoop(base_events.BaseEventLoop):
     async def sock_accept(self, sock):
         """Accept a connection.
 
-        The socket must be bound to an address and listening for connections.
-        The return value is a pair (conn, address) where conn is a new socket
-        object usable to send and receive data on the connection, and address
-        is the address bound to the socket on the other end of the connection.
+        The socket must be bound to an address and listening for
+        connections.  The return value is a pair (conn, address) where
+        conn is a new socket object usable to send and receive data on the
+        connection, and address is the address bound to the socket on the
+        other end of the connection.
         """
         base_events._check_ssl_socket(sock)
         if self._debug and sock.gettimeout() != 0:

Lib/asyncio/streams.py

@@ -541,17 +541,17 @@ class StreamReader:
             self._waiter = None
 
     async def readline(self):
-        """Read chunk of data from the stream until newline (b'\n') is found.
+        r"""Read chunk of data from the stream until newline (b'\n') is found.
 
-        On success, return chunk that ends with newline. If only partial
+        On success, return chunk that ends with newline.  If only partial
         line can be read due to EOF, return incomplete line without
-        terminating newline. When EOF was reached while no bytes read, empty
-        bytes object is returned.
+        terminating newline.  When EOF was reached while no bytes read,
+        empty bytes object is returned.
 
-        If limit is reached, ValueError will be raised. In that case, if
+        If limit is reached, ValueError will be raised.  In that case, if
         newline was found, complete line including newline will be removed
-        from internal buffer. Else, internal buffer will be cleared. Limit is
-        compared against part of the line without newline.
+        from internal buffer.  Else, internal buffer will be cleared.
+        Limit is compared against part of the line without newline.
 
         If stream was paused, this function will automatically resume it if
         needed.

Lib/asyncio/tasks.py

@@ -624,10 +624,11 @@ def as_completed(fs, *, timeout=None):
     Run the supplied awaitables concurrently. The returned object can be
     iterated to obtain the results of the awaitables as they finish.
 
-    The object returned can be iterated as an asynchronous iterator or a plain
-    iterator. When asynchronous iteration is used, the originally-supplied
-    awaitables are yielded if they are tasks or futures. This makes it easy to
-    correlate previously-scheduled tasks with their results:
+    The object returned can be iterated as an asynchronous iterator or
+    a plain iterator.  When asynchronous iteration is used, the
+    originally-supplied awaitables are yielded if they are tasks or
+    futures.  This makes it easy to correlate previously-scheduled tasks
+    with their results:
 
         ipv4_connect = create_task(open_connection("127.0.0.1", 80))
         ipv6_connect = create_task(open_connection("::1", 80))
@@ -643,26 +644,27 @@ def as_completed(fs, *, timeout=None):
             else:
                 print("IPv4 connection established.")
 
-    During asynchronous iteration, implicitly-created tasks will be yielded for
-    supplied awaitables that aren't tasks or futures.
+    During asynchronous iteration, implicitly-created tasks will be
+    yielded for supplied awaitables that aren't tasks or futures.
 
-    When used as a plain iterator, each iteration yields a new coroutine that
-    returns the result or raises the exception of the next completed awaitable.
-    This pattern is compatible with Python versions older than 3.13:
+    When used as a plain iterator, each iteration yields a new coroutine
+    that returns the result or raises the exception of the next completed
+    awaitable.  This pattern is compatible with Python versions older than
+    3.13:
 
         ipv4_connect = create_task(open_connection("127.0.0.1", 80))
         ipv6_connect = create_task(open_connection("::1", 80))
         tasks = [ipv4_connect, ipv6_connect]
 
         for next_connect in as_completed(tasks):
-            # next_connect is not one of the original task objects. It must be
-            # awaited to obtain the result value or raise the exception of the
-            # awaitable that finishes next.
+            # next_connect is not one of the original task objects. It must
+            # be awaited to obtain the result value or raise the exception
+            # of the awaitable that finishes next.
             reader, writer = await next_connect
 
-    A TimeoutError is raised if the timeout occurs before all awaitables are
-    done. This is raised by the async for loop during asynchronous iteration or
-    by the coroutines yielded during plain iteration.
+    A TimeoutError is raised if the timeout occurs before all awaitables
+    are done.  This is raised by the async for loop during asynchronous
+    iteration or by the coroutines yielded during plain iteration.
     """
     if inspect.isawaitable(fs):
         raise TypeError(
@@ -1030,21 +1032,22 @@ def run_coroutine_threadsafe(coro, loop):
 def create_eager_task_factory(custom_task_constructor):
     """Create a function suitable for use as a task factory on an event-loop.
 
-        Example usage:
+    Example usage:
 
-            loop.set_task_factory(
-                asyncio.create_eager_task_factory(my_task_constructor))
+        loop.set_task_factory(
+            asyncio.create_eager_task_factory(my_task_constructor))
 
-        Now, tasks created will be started immediately (rather than being first
-        scheduled to an event loop). The constructor argument can be any callable
-        that returns a Task-compatible object and has a signature compatible
-        with `Task.__init__`; it must have the `eager_start` keyword argument.
+    Now, tasks created will be started immediately (rather than being first
+    scheduled to an event loop).  The constructor argument can be any
+    callable that returns a Task-compatible object and has a signature
+    compatible with `Task.__init__`; it must have the `eager_start`
+    keyword argument.
 
-        Most applications will use `Task` for `custom_task_constructor` and in
-        this case there's no need to call `create_eager_task_factory()`
-        directly. Instead the  global `eager_task_factory` instance can be
-        used. E.g. `loop.set_task_factory(asyncio.eager_task_factory)`.
-        """
+    Most applications will use `Task` for `custom_task_constructor` and in
+    this case there's no need to call `create_eager_task_factory()`
+    directly. Instead the  global `eager_task_factory` instance can be
+    used. E.g. `loop.set_task_factory(asyncio.eager_task_factory)`.
+    """
 
     def factory(loop, coro, *, eager_start=True, **kwargs):
         return custom_task_constructor(

Lib/asyncio/threads.py

@@ -17,7 +17,8 @@ async def to_thread(func, /, *args, **kwargs):
     allowing context variables from the main thread to be accessed in the
     separate thread.
 
-    Return a coroutine that can be awaited to get the eventual result of *func*.
+    Return a coroutine that can be awaited to get the eventual result of
+    *func*.
     """
     loop = events.get_running_loop()
     ctx = contextvars.copy_context()

Lib/asyncio/timeouts.py

@@ -25,7 +25,8 @@ class _State(enum.Enum):
 class Timeout:
     """Asynchronous context manager for cancelling overdue coroutines.
 
-    Use `timeout()` or `timeout_at()` rather than instantiating this class directly.
+    Use `timeout()` or `timeout_at()` rather than instantiating this class
+    directly.
     """
 
     def __init__(self, when: float | None) -> None:

Lib/asyncio/unix_events.py

@@ -54,7 +54,8 @@ def waitstatus_to_exitcode(status):
 class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop):
     """Unix event loop.
 
-    Adds signal handling and UNIX Domain Socket support to SelectorEventLoop.
+    Adds signal handling and UNIX Domain Socket support to
+    SelectorEventLoop.
     """
 
     def __init__(self, selector=None):
@@ -384,12 +385,12 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop):
             # order to simplify the common case.
             self.remove_writer(registered_fd)
         if fut.cancelled():
-            self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+            self._sock_sendfile_update_filepos(fileno, offset)
             return
         if count:
             blocksize = count - total_sent
             if blocksize <= 0:
-                self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+                self._sock_sendfile_update_filepos(fileno, offset)
                 fut.set_result(total_sent)
                 return
 
@@ -423,20 +424,20 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop):
                 # plain send().
                 err = exceptions.SendfileNotAvailableError(
                     "os.sendfile call failed")
-                self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+                self._sock_sendfile_update_filepos(fileno, offset)
                 fut.set_exception(err)
             else:
-                self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+                self._sock_sendfile_update_filepos(fileno, offset)
                 fut.set_exception(exc)
         except (SystemExit, KeyboardInterrupt):
             raise
         except BaseException as exc:
-            self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+            self._sock_sendfile_update_filepos(fileno, offset)
             fut.set_exception(exc)
         else:
             if sent == 0:
                 # EOF
-                self._sock_sendfile_update_filepos(fileno, offset, total_sent)
+                self._sock_sendfile_update_filepos(fileno, offset)
                 fut.set_result(total_sent)
             else:
                 offset += sent
@@ -447,9 +448,9 @@ class _UnixSelectorEventLoop(selector_events.BaseSelectorEventLoop):
                                 fd, sock, fileno,
                                 offset, count, blocksize, total_sent)
 
-    def _sock_sendfile_update_filepos(self, fileno, offset, total_sent):
-        if total_sent > 0:
-            os.lseek(fileno, offset, os.SEEK_SET)
+    def _sock_sendfile_update_filepos(self, fileno, offset):
+        # After this helper runs, the source fd's lseek pointer is at offset."
+        os.lseek(fileno, offset, os.SEEK_SET)
 
     def _sock_add_cancellation_callback(self, fut, sock):
         def cb(fut):

Lib/asyncio/windows_events.py

@@ -610,6 +610,9 @@ class IocpProactor:
         ov = _overlapped.Overlapped(NULL)
         offset_low = offset & 0xffff_ffff
         offset_high = (offset >> 32) & 0xffff_ffff
+        # TransmitFile ignores OVERLAPPED.Offset for handles not opened with
+        # FILE_FLAG_OVERLAPPED, so seek the CRT file pointer to match.
+        file.seek(offset)
         ov.TransmitFile(sock.fileno(),
                         msvcrt.get_osfhandle(file.fileno()),
                         offset_low, offset_high,

Lib/asyncio/windows_utils.py

@@ -10,7 +10,6 @@ import itertools
 import msvcrt
 import os
 import subprocess
-import tempfile
 import warnings
 
 
@@ -24,6 +23,7 @@ BUFSIZE = 8192
 PIPE = subprocess.PIPE
 STDOUT = subprocess.STDOUT
 _mmap_counter = itertools.count()
+_MAX_PIPE_ATTEMPTS = 20
 
 
 # Replacement for os.pipe() using handles instead of fds
@@ -31,10 +31,6 @@ _mmap_counter = itertools.count()
 
 def pipe(*, duplex=False, overlapped=(True, True), bufsize=BUFSIZE):
     """Like os.pipe() but with overlapped support and using handles not fds."""
-    address = tempfile.mktemp(
-        prefix=r'\\.\pipe\python-pipe-{:d}-{:d}-'.format(
-            os.getpid(), next(_mmap_counter)))
-
     if duplex:
         openmode = _winapi.PIPE_ACCESS_DUPLEX
         access = _winapi.GENERIC_READ | _winapi.GENERIC_WRITE
@@ -56,9 +52,20 @@ def pipe(*, duplex=False, overlapped=(True, True), bufsize=BUFSIZE):
 
     h1 = h2 = None
     try:
-        h1 = _winapi.CreateNamedPipe(
-            address, openmode, _winapi.PIPE_WAIT,
-            1, obsize, ibsize, _winapi.NMPWAIT_WAIT_FOREVER, _winapi.NULL)
+        for attempts in itertools.count():
+            address = r'\\.\pipe\python-pipe-{:d}-{:d}-{}'.format(
+                os.getpid(), next(_mmap_counter), os.urandom(8).hex())
+            try:
+                h1 = _winapi.CreateNamedPipe(
+                    address, openmode, _winapi.PIPE_WAIT,
+                    1, obsize, ibsize, _winapi.NMPWAIT_WAIT_FOREVER, _winapi.NULL)
+                break
+            except OSError as e:
+                if attempts >= _MAX_PIPE_ATTEMPTS:
+                    raise
+                if e.winerror not in (_winapi.ERROR_PIPE_BUSY,
+                                      _winapi.ERROR_ACCESS_DENIED):
+                    raise
 
         h2 = _winapi.CreateFile(
             address, access, 0, _winapi.NULL, _winapi.OPEN_EXISTING,
@@ -104,8 +111,9 @@ class PipeHandle:
 
     def close(self, *, CloseHandle=_winapi.CloseHandle):
         if self._handle is not None:
-            CloseHandle(self._handle)
+            handle = self._handle
             self._handle = None
+            CloseHandle(handle)
 
     def __del__(self, _warn=warnings.warn):
         if self._handle is not None:

Lib/base64.py

@@ -325,17 +325,20 @@ def a85encode(b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False):
 
     foldspaces is an optional flag that uses the special short sequence 'y'
     instead of 4 consecutive spaces (ASCII 0x20) as supported by 'btoa'. This
-    feature is not supported by the "standard" Adobe encoding.
+    feature is not supported by the standard encoding used in PDF.
 
-    wrapcol controls whether the output should have newline (b'\\n') characters
-    added to it. If this is non-zero, each output line will be at most this
-    many characters long, excluding the trailing newline.
+    If wrapcol is non-zero, insert a newline (b'\\n') character after at most
+    every wrapcol characters.
 
-    pad controls whether the input is padded to a multiple of 4 before
-    encoding. Note that the btoa implementation always pads.
+    pad controls whether zero-padding applied to the end of the input
+    is fully retained in the output encoding, as done by btoa,
+    producing an exact multiple of 5 bytes of output.
+
+    adobe controls whether the encoded byte sequence is framed with <~
+    and ~>, as in a PostScript base-85 string literal.  Note that
+    while ASCII85Decode streams in PDF documents must be terminated
+    with ~>, they must not use a leading <~.
 
-    adobe controls whether the encoded byte sequence is framed with <~ and ~>,
-    which is used by the Adobe implementation.
     """
     global _a85chars, _a85chars2
     # Delay the initialization of tables to not waste memory
@@ -364,12 +367,14 @@ def a85encode(b, *, foldspaces=False, wrapcol=0, pad=False, adobe=False):
 def a85decode(b, *, foldspaces=False, adobe=False, ignorechars=b' \t\n\r\v'):
     """Decode the Ascii85 encoded bytes-like object or ASCII string b.
 
-    foldspaces is a flag that specifies whether the 'y' short sequence should be
-    accepted as shorthand for 4 consecutive spaces (ASCII 0x20). This feature is
-    not supported by the "standard" Adobe encoding.
+    foldspaces is a flag that specifies whether the 'y' short sequence
+    should be accepted as shorthand for 4 consecutive spaces (ASCII
+    0x20).  This feature is not supported by the standard Ascii85
+    encoding used in PDF and PostScript.
 
-    adobe controls whether the input sequence is in Adobe Ascii85 format (i.e.
-    is framed with <~ and ~>).
+    adobe controls whether the <~ and ~> markers are present. While
+    the leading <~ is not required, the input must end with ~>, or a
+    ValueError is raised.
 
     ignorechars should be a byte string containing characters to ignore from the
     input. This should only contain whitespace characters, and by default
@@ -442,8 +447,10 @@ _b85dec = None
 def b85encode(b, pad=False):
     """Encode bytes-like object b in base85 format and return a bytes object.
 
-    If pad is true, the input is padded with b'\\0' so its length is a multiple of
-    4 bytes before encoding.
+    The input is padded with b'\0' so its length is a multiple of 4
+    bytes before encoding.  If pad is true, all the resulting
+    characters are retained in the output, which will always be a
+    multiple of 5 bytes.
     """
     global _b85chars, _b85chars2
     # Delay the initialization of tables to not waste memory

Lib/collections/__init__.py

@@ -57,8 +57,7 @@ except ImportError:
 try:
     from _collections import defaultdict
 except ImportError:
-    # TODO: RUSTPYTHON - implement defaultdict in Rust
-    from ._defaultdict import defaultdict
+    pass
 
 heapq = None  # Lazily imported
 

Lib/collections/_defaultdict.py

@@ -1,62 +0,0 @@
-from reprlib import recursive_repr as _recursive_repr
-
-class defaultdict(dict):
-    def __init__(self, *args, **kwargs):
-        if len(args) >= 1:
-            default_factory = args[0]
-            if default_factory is not None and not callable(default_factory):
-                raise TypeError("first argument must be callable or None")
-            args = args[1:]
-        else:
-            default_factory = None
-        super().__init__(*args, **kwargs)
-        self.default_factory = default_factory
-
-    def __missing__(self, key):
-        if self.default_factory is not None:
-            val = self.default_factory()
-        else:
-            raise KeyError(key)
-        # CPython parity: a recursive __missing__ via factory() may have
-        # already populated key; preserve that value instead of overwriting.
-        if key in self:
-            return self[key]
-        self[key] = val
-        return val
-
-    @_recursive_repr()
-    def __repr_factory(factory):
-        return repr(factory)
-
-    def __repr__(self):
-        return f"{type(self).__name__}({defaultdict.__repr_factory(self.default_factory)}, {dict.__repr__(self)})"
-
-    def copy(self):
-        return type(self)(self.default_factory, self)
-
-    __copy__ = copy
-
-    def __reduce__(self):
-        if self.default_factory is not None:
-            args = self.default_factory,
-        else:
-            args = ()
-        return type(self), args, None, None, iter(self.items())
-
-    def __or__(self, other):
-        if not isinstance(other, dict):
-            return NotImplemented
-
-        new = defaultdict(self.default_factory, self)
-        new.update(other)
-        return new
-
-    def __ror__(self, other):
-        if not isinstance(other, dict):
-            return NotImplemented
-
-        new = defaultdict(self.default_factory, other)
-        new.update(self)
-        return new
-
-defaultdict.__module__ = 'collections'

Lib/compression/zstd/__init__.py

@@ -61,8 +61,9 @@ class FrameInfo:
 def get_frame_info(frame_buffer):
     """Get Zstandard frame information from a frame header.
 
-    *frame_buffer* is a bytes-like object. It should start from the beginning
-    of a frame, and needs to include at least the frame header (6 to 18 bytes).
+    *frame_buffer* is a bytes-like object. It should start from the
+    beginning of a frame, and needs to include at least the frame header
+    (6 to 18 bytes).
 
     The returned FrameInfo object has two attributes.
     'decompressed_size' is the size in bytes of the data in the frame when
@@ -103,16 +104,17 @@ def finalize_dict(zstd_dict, /, samples, dict_size, level):
     finalize *zstd_dict* by adding headers and statistics according to the
     Zstandard dictionary format.
 
-    You may compose an effective dictionary content by hand, which is used as
-    basis dictionary, and use some samples to finalize a dictionary. The basis
-    dictionary may be a "raw content" dictionary. See *is_raw* in ZstdDict.
+    You may compose an effective dictionary content by hand, which is used
+    as basis dictionary, and use some samples to finalize a dictionary.  The
+    basis dictionary may be a "raw content" dictionary.  See *is_raw* in
+    ZstdDict.
 
-    *samples* is an iterable of samples, where a sample is a bytes-like object
-    representing a file.
+    *samples* is an iterable of samples, where a sample is a bytes-like
+    object representing a file.
     *dict_size* is the dictionary's maximum size, in bytes.
     *level* is the expected compression level. The statistics for each
-    compression level differ, so tuning the dictionary to the compression level
-    can provide improvements.
+    compression level differ, so tuning the dictionary to the compression
+    level can provide improvements.
     """
 
     if not isinstance(zstd_dict, ZstdDict):
@@ -140,8 +142,8 @@ def compress(data, level=None, options=None, zstd_dict=None):
     COMPRESSION_LEVEL_DEFAULT ('3').
     *options* is a dict object that contains advanced compression
     parameters. See CompressionParameter for more on options.
-    *zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary. See
-    the function train_dict for how to train a ZstdDict on sample data.
+    *zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary.
+    See the function train_dict for how to train a ZstdDict on sample data.
 
     For incremental compression, use a ZstdCompressor instead.
     """
@@ -152,8 +154,8 @@ def compress(data, level=None, options=None, zstd_dict=None):
 def decompress(data, zstd_dict=None, options=None):
     """Decompress one or more frames of Zstandard compressed *data*.
 
-    *zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary. See
-    the function train_dict for how to train a ZstdDict on sample data.
+    *zstd_dict* is a ZstdDict object, a pre-trained Zstandard dictionary.
+    See the function train_dict for how to train a ZstdDict on sample data.
     *options* is a dict object that contains advanced compression
     parameters. See DecompressionParameter for more on options.
 

Lib/compression/zstd/_zstdfile.py

@@ -36,9 +36,9 @@ class ZstdFile(_streams.BaseStream):
 
         *file* can be either an file-like object, or a file name to open.
 
-        *mode* can be 'r' for reading (default), 'w' for (over)writing, 'x' for
-        creating exclusively, or 'a' for appending. These can equivalently be
-        given as 'rb', 'wb', 'xb' and 'ab' respectively.
+        *mode* can be 'r' for reading (default), 'w' for (over)writing, 'x'
+        for creating exclusively, or 'a' for appending.  These can
+        equivalently be given as 'rb', 'wb', 'xb' and 'ab' respectively.
 
         *level* is an optional int specifying the compression level to use,
         or COMPRESSION_LEVEL_DEFAULT if not given.
@@ -296,26 +296,27 @@ def open(file, /, mode='rb', *, level=None, options=None, zstd_dict=None,
          encoding=None, errors=None, newline=None):
     """Open a Zstandard compressed file in binary or text mode.
 
-    file can be either a file name (given as a str, bytes, or PathLike object),
-    in which case the named file is opened, or it can be an existing file object
-    to read from or write to.
+    file can be either a file name (given as a str, bytes, or PathLike
+    object), in which case the named file is opened, or it can be
+    an existing file object to read from or write to.
 
-    The mode parameter can be 'r', 'rb' (default), 'w', 'wb', 'x', 'xb', 'a',
-    'ab' for binary mode, or 'rt', 'wt', 'xt', 'at' for text mode.
+    The mode parameter can be 'r', 'rb' (default), 'w', 'wb', 'x', 'xb',
+    'a', 'ab' for binary mode, or 'rt', 'wt', 'xt', 'at' for text mode.
 
-    The level, options, and zstd_dict parameters specify the settings the same
-    as ZstdFile.
+    The level, options, and zstd_dict parameters specify the settings the
+    same as ZstdFile.
 
     When using read mode (decompression), the options parameter is a dict
-    representing advanced decompression options. The level parameter is not
-    supported in this case. When using write mode (compression), only one of
-    level, an int representing the compression level, or options, a dict
-    representing advanced compression options, may be passed. In both modes,
-    zstd_dict is a ZstdDict instance containing a trained Zstandard dictionary.
+    representing advanced decompression options.  The level parameter is not
+    supported in this case.  When using write mode (compression), only one
+    of level, an int representing the compression level, or options, a dict
+    representing advanced compression options, may be passed.  In both
+    modes, zstd_dict is a ZstdDict instance containing a trained Zstandard
+    dictionary.
 
-    For binary mode, this function is equivalent to the ZstdFile constructor:
-    ZstdFile(filename, mode, ...). In this case, the encoding, errors and
-    newline parameters must not be provided.
+    For binary mode, this function is equivalent to the ZstdFile
+    constructor: ZstdFile(filename, mode, ...).  In this case, the encoding,
+    errors and newline parameters must not be provided.
 
     For text mode, an ZstdFile object is created, and wrapped in an
     io.TextIOWrapper instance with the specified encoding, error handling

Lib/email/_header_value_parser.py

@@ -1463,6 +1463,16 @@ def get_phrase(value):
         else:
             try:
                 token, value = get_word(value)
+                if (token[0].token_type == 'encoded-word'
+                    and phrase
+                    and phrase[-1].token_type == 'atom'
+                    and len(phrase[-1]) > 1
+                    and phrase[-1][-2].token_type == 'encoded-word'
+                    and phrase[-1][-1].token_type == 'cfws'
+                    and not phrase[-1][-1].comments
+                ):
+                    # linear ws between ews needs special handing...
+                    phrase[-1][-1] = EWWhiteSpaceTerminal(phrase[-1], 'fws')
             except errors.HeaderParseError:
                 if value[0] in CFWS_LEADER:
                     token, value = get_cfws(value)

Lib/email/_parseaddr.py

@@ -59,7 +59,7 @@ def _parsedate_tz(data):
 
     The last (additional) element is the time zone offset in seconds, except if
     the timezone was specified as -0000.  In that case the last element is
-    None.  This indicates a UTC timestamp that explicitly declaims knowledge of
+    None.  This indicates a UTC timestamp that explicitly disclaims knowledge of
     the source timezone, as opposed to a +0000 timestamp that indicates the
     source timezone really was UTC.
 

Lib/email/charset.py

@@ -93,8 +93,6 @@ ALIASES = {
 
 # Map charsets to their Unicode codec strings.
 CODEC_MAP = {
-    'gb2312':      'eucgb2312_cn',
-    'big5':        'big5_tw',
     # Hack: We don't want *any* conversion for stuff marked us-ascii, as all
     # sorts of garbage might be sent to us in the guise of 7-bit us-ascii.
     # Let that stuff pass through without conversion to/from Unicode.

Lib/encodings/aliases.py

@@ -71,6 +71,10 @@ aliases = {
 
     # cp1140 codec
     '1140'               : 'cp1140',
+    'cp01140'            : 'cp1140',
+    'csibm01140'         : 'cp1140',
+    'ebcdic_us_37_euro'  : 'cp1140',
+    'ibm01140'           : 'cp1140',
     'ibm1140'            : 'cp1140',
 
     # cp1250 codec
@@ -159,8 +163,12 @@ aliases = {
 
     # cp858 codec
     '858'                : 'cp858',
+    'cp00858'            : 'cp858',
+    'csibm00858'         : 'cp858',
     'csibm858'           : 'cp858',
+    'ibm00858'           : 'cp858',
     'ibm858'             : 'cp858',
+    'pc_multilingual_850_euro' : 'cp858',
 
     # cp860 codec
     '860'                : 'cp860',

Lib/ensurepip/__init__.py

@@ -10,7 +10,7 @@ from shutil import copy2
 
 
 __all__ = ["version", "bootstrap"]
-_PIP_VERSION = "26.1.1"
+_PIP_VERSION = "26.1.2"
 
 # Directory of system wheel packages. Some Linux distribution packaging
 # policies recommend against bundling dependencies. For example, Fedora

Lib/enum.py

@@ -678,9 +678,9 @@ class EnumType(type):
         """
         Either returns an existing member, or creates a new enum class.
 
-        This method is used both when an enum class is given a value to match
-        to an enumeration member (i.e. Color(3)) and for the functional API
-        (i.e. Color = Enum('Color', names='RED GREEN BLUE')).
+        This method is used both when an enum class is given a value to
+        match to an enumeration member (i.e. Color(3)) and for the
+        functional API (i.e. Color = Enum('Color', names='RED GREEN BLUE')).
 
         The value lookup branch is chosen if the enum is final.
 
@@ -688,16 +688,17 @@ class EnumType(type):
 
         `value` will be the name of the new class.
 
-        `names` should be either a string of white-space/comma delimited names
-        (values will start at `start`), or an iterator/mapping of name, value pairs.
+        `names` should be either a string of white-space/comma delimited
+        names (values will start at `start`), or an iterator/mapping of
+        name, value pairs.
 
         `module` should be set to the module this class is being created in;
-        if it is not set, an attempt to find that module will be made, but if
-        it fails the class will not be picklable.
+        if it is not set, an attempt to find that module will be made, but
+        if it fails the class will not be picklable.
 
-        `qualname` should be set to the actual location this class can be found
-        at in its module; by default it is set to the global scope.  If this is
-        not correct, unpickling will fail in some circumstances.
+        `qualname` should be set to the actual location this class can be
+        found at in its module; by default it is set to the global scope.
+        If this is not correct, unpickling will fail in some circumstances.
 
         `type`, if set, will be mixed in as the first base class.
         """
@@ -791,8 +792,8 @@ class EnumType(type):
         """
         Returns a mapping of member name->value.
 
-        This mapping lists all enum members, including aliases. Note that this
-        is a read-only view of the internal mapping.
+        This mapping lists all enum members, including aliases.  Note that
+        this is a read-only view of the internal mapping.
         """
         return MappingProxyType(cls._member_map_)
 

Lib/ftplib.py

@@ -883,7 +883,16 @@ def ftpcp(source, sourcename, target, targetname = '', type = 'I'):
     type = 'TYPE ' + type
     source.voidcmd(type)
     target.voidcmd(type)
-    sourcehost, sourceport = parse227(source.sendcmd('PASV'))
+    # Don't trust the IPv4 address the source server advertises in its PASV
+    # reply: a malicious source could otherwise point the target's data
+    # connection at an arbitrary host (SSRF).  A caller that needs the old
+    # behavior can set trust_server_pasv_ipv4_address on the source FTP
+    # object.  See FTP.makepasv(), which applies the same rule.
+    untrusted_host, sourceport = parse227(source.sendcmd('PASV'))
+    if source.trust_server_pasv_ipv4_address:
+        sourcehost = untrusted_host
+    else:
+        sourcehost = source.sock.getpeername()[0]
     target.sendport(sourcehost, sourceport)
     # RFC 959: the user must "listen" [...] BEFORE sending the
     # transfer request.

Lib/functools.py

@@ -556,16 +556,16 @@ def lru_cache(maxsize=128, typed=False):
     If *maxsize* is set to None, the LRU features are disabled and the cache
     can grow without bound.
 
-    If *typed* is True, arguments of different types will be cached separately.
-    For example, f(decimal.Decimal("3.0")) and f(3.0) will be treated as
-    distinct calls with distinct results. Some types such as str and int may
-    be cached separately even when typed is false.
+    If *typed* is True, arguments of different types will be cached
+    separately.  For example, f(decimal.Decimal("3.0")) and f(3.0) will be
+    treated as distinct calls with distinct results.  Some types such as
+    str and int may be cached separately even when typed is false.
 
     Arguments to the cached function must be hashable.
 
     View the cache statistics named tuple (hits, misses, maxsize, currsize)
-    with f.cache_info().  Clear the cache and statistics with f.cache_clear().
-    Access the underlying function with f.__wrapped__.
+    with f.cache_info().  Clear the cache and statistics with
+    f.cache_clear().  Access the underlying function with f.__wrapped__.
 
     See:  https://en.wikipedia.org/wiki/Cache_replacement_policies#Least_recently_used_(LRU)
 

Lib/glob.py

@@ -25,17 +25,17 @@ def glob(pathname, *, root_dir=None, dir_fd=None, recursive=False,
     The order of the returned list is undefined. Sort it if you need a
     particular order.
 
-    If `root_dir` is not None, it should be a path-like object specifying the
-    root directory for searching. It has the same effect as changing the
-    current directory before calling it (without actually
-    changing it). If pathname is relative, the result will contain
-    paths relative to `root_dir`.
+    If `root_dir` is not None, it should be a path-like object specifying
+    the root directory for searching.  It has the same effect as changing
+    the current directory before calling it (without actually changing it).
+    If pathname is relative, the result will contain paths relative to
+    `root_dir`.
 
     If `dir_fd` is not None, it should be a file descriptor referring to a
     directory, and paths will then be relative to that directory.
 
-    If `include_hidden` is true, the patterns '*', '?', '**'  will match hidden
-    directories.
+    If `include_hidden` is true, the patterns '*', '?', '**'  will match
+    hidden directories.
 
     If `recursive` is true, the pattern '**' will match any files and
     zero or more directories and subdirectories.
@@ -56,16 +56,16 @@ def iglob(pathname, *, root_dir=None, dir_fd=None, recursive=False,
     particular order.
 
     If `root_dir` is not None, it should be a path-like object specifying
-    the root directory for searching. It has the same effect as changing
-    the current directory before calling it (without actually
-    changing it). If pathname is relative, the result will contain
-    paths relative to `root_dir`.
+    the root directory for searching.  It has the same effect as changing
+    the current directory before calling it (without actually changing it).
+    If pathname is relative, the result will contain paths relative to
+    `root_dir`.
 
     If `dir_fd` is not None, it should be a file descriptor referring to a
     directory, and paths will then be relative to that directory.
 
-    If `include_hidden` is true, the patterns '*', '?', '**'  will match hidden
-    directories.
+    If `include_hidden` is true, the patterns '*', '?', '**'  will match
+    hidden directories.
 
     If `recursive` is true, the pattern '**' will match any files and
     zero or more directories and subdirectories.
@@ -294,15 +294,15 @@ _no_recurse_symlinks = object()
 def translate(pat, *, recursive=False, include_hidden=False, seps=None):
     """Translate a pathname with shell wildcards to a regular expression.
 
-    If `recursive` is true, the pattern segment '**' will match any number of
-    path segments.
+    If `recursive` is true, the pattern segment '**' will match any number
+    of path segments.
 
     If `include_hidden` is true, wildcards can match path segments beginning
     with a dot ('.').
 
     If a sequence of separator characters is given to `seps`, they will be
-    used to split the pattern into segments and match path separators. If not
-    given, os.path.sep and os.path.altsep (where available) are used.
+    used to split the pattern into segments and match path separators.  If
+    not given, os.path.sep and os.path.altsep (where available) are used.
     """
     if not seps:
         if os.path.altsep:

Lib/gzip.py

@@ -34,16 +34,16 @@ def open(filename, mode="rb", compresslevel=_COMPRESS_LEVEL_BEST,
          encoding=None, errors=None, newline=None):
     """Open a gzip-compressed file in binary or text mode.
 
-    The filename argument can be an actual filename (a str or bytes object), or
-    an existing file object to read from or write to.
+    The filename argument can be an actual filename (a str or bytes object),
+    or an existing file object to read from or write to.
 
-    The mode argument can be "r", "rb", "w", "wb", "x", "xb", "a" or "ab" for
-    binary mode, or "rt", "wt", "xt" or "at" for text mode. The default mode is
-    "rb", and the default compresslevel is 9.
+    The mode argument can be "r", "rb", "w", "wb", "x", "xb", "a" or "ab"
+    for binary mode, or "rt", "wt", "xt" or "at" for text mode.  The default
+    mode is "rb", and the default compresslevel is 9.
 
-    For binary mode, this function is equivalent to the GzipFile constructor:
-    GzipFile(filename, mode, compresslevel). In this case, the encoding, errors
-    and newline arguments must not be provided.
+    For binary mode, this function is equivalent to the GzipFile
+    constructor: GzipFile(filename, mode, compresslevel).  In this case,
+    the encoding, errors and newline arguments must not be provided.
 
     For text mode, a GzipFile object is created, and wrapped in an
     io.TextIOWrapper instance with the specified encoding, error handling
@@ -148,8 +148,8 @@ class GzipFile(_streams.BaseStream):
     """The GzipFile class simulates most of the methods of a file object with
     the exception of the truncate() method.
 
-    This class only supports opening files in binary mode. If you need to open a
-    compressed file in text mode, use the gzip.open() function.
+    This class only supports opening files in binary mode.  If you need to
+    open a compressed file in text mode, use the gzip.open() function.
 
     """
 
@@ -165,31 +165,33 @@ class GzipFile(_streams.BaseStream):
         non-trivial value.
 
         The new class instance is based on fileobj, which can be a regular
-        file, an io.BytesIO object, or any other object which simulates a file.
-        It defaults to None, in which case filename is opened to provide
-        a file object.
+        file, an io.BytesIO object, or any other object which simulates
+        a file.  It defaults to None, in which case filename is opened to
+        provide a file object.
 
         When fileobj is not None, the filename argument is only used to be
         included in the gzip file header, which may include the original
         filename of the uncompressed file.  It defaults to the filename of
         fileobj, if discernible; otherwise, it defaults to the empty string,
-        and in this case the original filename is not included in the header.
+        and in this case the original filename is not included in the
+        header.
 
-        The mode argument can be any of 'r', 'rb', 'a', 'ab', 'w', 'wb', 'x', or
-        'xb' depending on whether the file will be read or written.  The default
-        is the mode of fileobj if discernible; otherwise, the default is 'rb'.
-        A mode of 'r' is equivalent to one of 'rb', and similarly for 'w' and
-        'wb', 'a' and 'ab', and 'x' and 'xb'.
+        The mode argument can be any of 'r', 'rb', 'a', 'ab', 'w', 'wb',
+        'x', or 'xb' depending on whether the file will be read or written.
+        The default is the mode of fileobj if discernible; otherwise, the
+        default is 'rb'.  A mode of 'r' is equivalent to one of 'rb', and
+        similarly for 'w' and 'wb', 'a' and 'ab', and 'x' and 'xb'.
 
-        The compresslevel argument is an integer from 0 to 9 controlling the
-        level of compression; 1 is fastest and produces the least compression,
-        and 9 is slowest and produces the most compression. 0 is no compression
-        at all. The default is 9.
+        The compresslevel argument is an integer from 0 to 9 controlling
+        the level of compression; 1 is fastest and produces the least
+        compression, and 9 is slowest and produces the most compression.
+        0 is no compression at all. The default is 9.
 
-        The optional mtime argument is the timestamp requested by gzip. The time
-        is in Unix format, i.e., seconds since 00:00:00 UTC, January 1, 1970.
-        If mtime is omitted or None, the current time is used. Use mtime = 0
-        to generate a compressed stream that does not depend on creation time.
+        The optional mtime argument is the timestamp requested by gzip.
+        The time is in Unix format, i.e., seconds since 00:00:00 UTC,
+        January 1, 1970.  If mtime is omitted or None, the current time
+        is used.  Use mtime = 0 to generate a compressed stream that does
+        not depend on creation time.
 
         """
 
@@ -575,10 +577,10 @@ class _GzipReader(_streams.DecompressReader):
             # Read a chunk of data from the file
             if self._decompressor.needs_input:
                 buf = self._fp.read(READ_BUFFER_SIZE)
-                uncompress = self._decompressor.decompress(buf, size)
             else:
-                uncompress = self._decompressor.decompress(b"", size)
+                buf = b""
 
+            uncompress = self._decompressor.decompress(buf, size)
             if self._decompressor.unused_data != b"":
                 # Prepend the already read bytes to the fileobj so they can
                 # be seen by _read_eof() and _read_gzip_header()

Lib/http/cookies.py

@@ -391,18 +391,18 @@ class Morsel(dict):
         return '<%s: %s>' % (self.__class__.__name__, self.OutputString())
 
     def js_output(self, attrs=None):
-        import base64
+        import urllib.parse
         # Print javascript
         output_string = self.OutputString(attrs)
         if _has_control_character(output_string):
             raise CookieError("Control characters are not allowed in cookies")
         # Base64-encode value to avoid template
         # injection in cookie values.
-        output_encoded = base64.b64encode(output_string.encode('utf-8')).decode("ascii")
+        output_encoded = urllib.parse.quote(output_string, safe='', encoding='utf-8')
         return """
         <script type="text/javascript">
         <!-- begin hiding
-        document.cookie = atob(\"%s\");
+        document.cookie = decodeURIComponent(\"%s\");
         // end hiding -->
         </script>
         """ % (output_encoded,)

Lib/idlelib/CREDITS.txt

@@ -0,0 +1,48 @@
+Guido van Rossum, as well as being the creator of the Python language, is the
+original creator of IDLE.  Other contributors prior to Version 0.8 include
+Mark Hammond, Jeremy Hylton, Tim Peters, and Moshe Zadka.
+
+Until Python 2.3, IDLE's development was carried out in the SF IDLEfork project. The
+objective was to develop a version of IDLE which had an execution environment
+which could be initialized prior to each run of user code.
+IDLefork was merged into the Python code base in 2003.
+
+The IDLEfork project was initiated by David Scherer, with some help from Peter
+Schneider-Kamp and Nicholas Riley.  David wrote the first version of the RPC
+code and designed a fast turn-around environment for VPython.  Guido developed
+the RPC code and Remote Debugger currently integrated in IDLE.  Bruce Sherwood
+contributed considerable time testing and suggesting improvements.
+
+Besides David and Guido, the main developers who were active on IDLEfork
+are Stephen M. Gava, who implemented the configuration GUI, the new
+configuration system, and the About dialog, and Kurt B. Kaiser, who completed
+the integration of the RPC and remote debugger, implemented the threaded
+subprocess, and made a number of usability enhancements.
+
+Other contributors include Raymond Hettinger, Tony Lownds (Mac integration),
+Neal Norwitz (code check and clean-up), Ronald Oussoren (Mac integration),
+Noam Raphael (Code Context, Call Tips, many other patches), and Chui Tey (RPC
+integration, debugger integration and persistent breakpoints).
+
+Scott David Daniels, Tal Einat, Hernan Foffani, Christos Georgiou,
+Jim Jewett, Martin v. Löwis, Jason Orendorff, Guilherme Polo, Josh Robb,
+Nigel Rowe, Bruce Sherwood, Jeff Shute, and Weeble have submitted useful
+patches.  Thanks, guys!
+
+Major contributors since 2005:
+
+- 2005: Tal Einat
+- 2010: Terry Jan Reedy (current maintainer)
+- 2013: Roger Serwy
+- 2014: Saimadhav Heblikar
+- 2015: Mark Roseman
+- 2017: Louie Lu, Cheryl Sabella, and Serhiy Storchaka
+- 2025: Stan Ulbrych
+
+For additional details refer to NEWS.txt and Changelog.
+
+If we missed you, feel free to submit a PR with a summary of
+contributions (for instance, at least 5 merged PRs).
+
+
+