Jeong, YunWon
c603eb7c9a
* Implement PEP 688 __buffer__ and __release_buffer__ A Python class could not export a buffer: the slot machinery had no bf_getbuffer or bf_releasebuffer, and every consumer acquired buffers as PyBUF_FULL_RO through a module of PyBUF_* constants. Add both slots. PyBuffer::release now runs a Python __release_buffer__ before the exporter's own release, once per acquisition, which PyBuffer tracks with an `acquired` flag that clones do not inherit. An export made by a Python __buffer__ is held by a _buffer_wrapper payload that counts its exports and drops the returned memoryview with the last one, and the view handed to __release_buffer__ is a _buffer_window that owns no export, so releasing it inside the hook is inert instead of re-entering it. Replace the PyBUF_* constants with a BufferFlags bitflags type whose composite requests are supersets of the simpler ones, so `contains` answers the REQ_* questions, and pass the request to PyBuffer::from_object. Each consumer now asks for what its counterpart asks for: y* arguments for SIMPLE, w* for WRITABLE, BytesIO.write for CONTIG_RO, bytes(), bytearray() and memoryview() for FULL_RO. memoryview checks the request in memory_getbuf, and array.array and mmap.mmap expose __release_buffer__. Test buffer support with PyObject::check_buffer (PyObject_CheckBuffer) instead of attempting an acquisition, so an exception raised by __buffer__ is no longer reported as the object not being bytes-like, and a __buffer__ with side effects runs once. PyBytesInner becomes a y* conversion as a result: bytes and bytearray methods no longer accept iterables of ints, and find, index, count and __contains__ take the arguments parse_args_finds_byte and bytes_contains describe. A view exports its start offset in the descriptor rather than in its window, which fixes a panic when collecting from a negative-stride view. BytesIO.write rechecks closed after acquiring its buffer, which __buffer__ can close in between. Assisted-by: Claude Code:claude-opus-5 * Rework the buffer protocol around managed exports and view offsets Give `PyBuffer` the `_PyManagedBufferObject` shape: one `bf_getbuffer` acquisition is shared by every handle taken from it, cloning takes another share instead of re-acquiring, and the exporter's release runs once when the last share goes away. Remove `retain`, the unsafe `drop_without_release`, the three `impl Drop`s and the `ManuallyDrop` that stood in for this. Add `abort_acquisition` so a failed request does not run `bf_releasebuffer`. Move the view start into `BufferDescriptor::offset`, the `Py_buffer.buf` analogue, and drop the separate `start` fields on `PyMemoryView` and `PyBufferWrapper`. Slicing goes through `SaturatedSlice::adjust_indices_start`, which reproduces `PySlice_AdjustIndices` and keeps the adjusted start. Fix `zip_eq` to take its contiguous fast path only when both last dimensions are contiguous, and make `for_each_segment` and `zip_eq` handle zero-length and zero-dimensional views. Add `BufferDescriptor::projected` so a request without `PyBUF_ND`, `PyBUF_STRIDES` or `PyBUF_FORMAT` receives a correspondingly reduced descriptor, and reject a request without `PyBUF_INDIRECT` against an exporter that has suboffsets. Copy the source first in `memoryview` slice assignment when both sides reach the same root exporter. Hold the export across the resize in `bytearray.extend`, take `y*` in `marshal.loads`, stop probing the buffer protocol in `FsPath`, rewrite `ord` over the concrete string types, fold `array`'s buffer slot into one `slot_as_buffer`, take `w*`/`y*` in `_overlapped`, and thread the new `offset` field through the `_ctypes` descriptors. Assisted-by: Claude * Re-check a memoryview after a conversion and tighten cast and release `pack_single` and `unpack_single` addressed the buffer with a position taken before `__index__` ran, so releasing the view from that conversion read or wrote outside the exporter's storage, panicking when it had also shrunk. Check the released flag again once the conversion is done, as `CHECK_RELEASED_AGAIN` does. Reject a cast to a format that is not a single native format character with an optional `@` in front of it. An empty format reached a division by its item size of zero. get_native_fmtchar Report a second `__release_buffer__` on the same view as a `ValueError` rather than accepting it, and check that the view belongs to the object first; the silent case is a view that exports nothing. wrap_releasebuffer Compare against another memoryview by reading its view where it lies instead of acquiring a buffer from it, so the restricted view handed to `__release_buffer__` compares equal rather than unequal in one direction only. memory_richcompare Name the type in the unraisable an exception from `__release_buffer__` reports, as `releasebuffer_call_python` does, instead of reporting the exporter object. Assisted-by: Claude * Re-check the destination of a memoryview slice assignment Acquiring the source runs `__buffer__`, which can release the destination view, so check the released flag again once the source is in hand and before the structures are compared, as `copy_single` does. Build the sliced destination as a view that counts as no export, the way a `Py_buffer dest = *view` copy does. Holding one kept the exporter unresizable for the length of the assignment, so a source that released the view and then resized the exporter met a `BufferError` instead of the assignment reporting the released view. Assisted-by: Claude * Answer the memoryview contiguity getsets with the layout `f_contiguous` reported the row-major answer for one dimension and False for any more, so a view laid out both ways, such as one of shape (1, 8), reported False while `check_buffer_request` accepted a `F_CONTIGUOUS` request on it. `contiguous` answered row-major order alone. Move the Fortran-order test from `PyMemoryView` to `BufferDescriptor`, next to the row-major one it differs from only in iteration order, and answer all three getsets with it. Assisted-by: Claude
RustPython
A Python-3 (CPython >= 3.14.0) Interpreter written in Rust 🐍 😱 🤘.
Usage
Check out our online demo running on WebAssembly.
RustPython requires Rust latest stable version (e.g 1.67.1 at February 7th 2023). If you don't currently have Rust installed on your system you can do so by following the instructions at rustup.rs.
To check the version of Rust you're currently running, use rustc --version. If you wish to update,
rustup update stable will update your Rust installation to the most recent stable release.
To build RustPython locally, first, clone the source code:
git clone https://github.com/RustPython/RustPython
RustPython uses symlinks to manage python libraries in Lib/. If on windows, running the following helps:
git config core.symlinks true
Then you can change into the RustPython directory and run the demo (Note: --release is
needed to prevent stack overflow on Windows):
$ cd RustPython
$ cargo run --release -- -c 'print("Hello, RustPython!")'
Hello, RustPython!
Or use the interactive shell:
$ cargo run --release
Welcome to rustpython
>>>>> 2+2
4
NOTE: For windows users, please set RUSTPYTHONPATH environment variable as Lib path in project directory.
(e.g. When RustPython directory is C:\RustPython, set RUSTPYTHONPATH as C:\RustPython\Lib)
You can also install and run RustPython with the following:
$ cargo install --git https://github.com/RustPython/RustPython rustpython
$ rustpython
Welcome to the magnificent Rust Python interpreter
>>>>>
venv
Because RustPython currently doesn't provide a well-packaged installation, using venv helps to use pip easier.
$ rustpython -m venv <your_env_name>
$ . <your_env_name>/bin/activate
$ python # now `python` is the alias of the RustPython for the new env
PIP
If you'd like to make https requests, you can enable the ssl feature, which
also lets you install the pip package manager. Note that on Windows, you may
need to install OpenSSL, or you can enable the ssl-openssl-vendor feature instead,
which compiles OpenSSL for you but requires a C compiler, perl, and make.
OpenSSL version 3 is expected and tested in CI. Older versions may not work.
Once you've installed rustpython with SSL support, you can install pip by running:
cargo install --git https://github.com/RustPython/RustPython
rustpython --install-pip
You can also install RustPython through the conda package manager, though
this isn't officially supported and may be out of date:
conda install rustpython -c conda-forge
rustpython
SSL provider
For HTTPS requests, ssl-rustls-aws-lc is enabled by default for the RustPython binary. Embedders can use rustpython-stdlib's provider-agnostic ssl-rustls feature and install their own rustls crypto provider, or replace rustls with ssl-openssl if their environment requires OpenSSL.
Note that to use OpenSSL on Windows, you may need to install OpenSSL, or you can enable the ssl-openssl-vendor feature instead,
which compiles OpenSSL for you but requires a C compiler, perl, and make.
OpenSSL version 3 is expected and tested in CI. Older versions may not work.
WASI
You can compile RustPython to a standalone WebAssembly WASI module so it can run anywhere.
Build
cargo build --target wasm32-wasip1 --no-default-features --features freeze-stdlib,stdlib --release
Run by wasmer
wasmer run --dir `pwd` -- target/wasm32-wasip1/release/rustpython.wasm `pwd`/extra_tests/snippets/stdlib_random.py
Run by wapm
$ wapm install rustpython
$ wapm run rustpython
>>>>> 2+2
4
Building the WASI file
You can build the WebAssembly WASI file with:
cargo build --release --target wasm32-wasip1 --features="freeze-stdlib"
Note: we use the
freeze-stdlibto include the standard library inside the binary. You also have to run oncerustup target add wasm32-wasip1.
JIT (Just in time) compiler
RustPython has a very experimental JIT compiler that compile python functions into native code.
Building
By default the JIT compiler isn't enabled, it's enabled with the jit cargo feature.
cargo run --features jit
This requires autoconf, automake, libtool, and clang to be installed.
Using
To compile a function, call __jit__() on it.
def foo():
a = 5
return 10 + a
foo.__jit__() # this will compile foo to native code and subsequent calls will execute that native code
assert foo() == 15
Embedding RustPython into your Rust Applications
Interested in exposing Python scripting in an application written in Rust,
perhaps to allow quickly tweaking logic where Rust's compile times would be inhibitive?
Then examples/hello_embed.rs and examples/mini_repl.rs may be of some assistance.
Disclaimer
RustPython is in development, and while the interpreter certainly can be used in interesting use cases like running Python in WASM and embedding into a Rust project, do note that RustPython is not totally production-ready.
Contribution is more than welcome! See our contribution section for more information on this.
Conference videos
Checkout those talks on conferences:
Use cases
Although RustPython is a fairly young project, a few people have used it to make cool projects:
- GreptimeDB: an open-source, cloud-native, distributed time-series database. Using RustPython for embedded scripting.
- pyckitup: a game engine written in rust.
- Robot Rumble: an arena-based AI competition platform
- Ruff: an extremely fast Python linter, written in Rust
Goals
- Full Python-3 environment entirely in Rust (not CPython bindings)
- A clean implementation without compatibility hacks
Documentation
Currently along with other areas of the project, documentation is still in an early phase.
You can read the online documentation for the latest release, or the user guide.
You can also generate documentation locally by running:
cargo doc # Including documentation for all dependencies
cargo doc --no-deps --all # Excluding all dependencies
Documentation HTML files can then be found in the target/doc directory or you can append --open to the previous commands to
have the documentation open automatically on your default browser.
For a high level overview of the components, see the architecture document.
Contributing
Contributions are welcome and highly appreciated. To get started, check out the contributing guidelines.
You can also join us on Discord.
Compiling to WebAssembly
Community
Chat with us on Discord.
Code of conduct
Our code of conduct can be found here.
Credit
The initial work was based on windelbouwman/rspython and shinglyu/RustPython
Links
These are some useful links to related projects:
- https://github.com/ProgVal/pythonvm-rust
- https://github.com/shinglyu/RustPython
- https://github.com/windelbouwman/rspython
License
This project is licensed under the MIT license. Please see the LICENSE file for more details.
The project logo is licensed under the CC-BY-4.0 license. Please see the LICENSE-logo file for more details.