Implement PEP 688 and rework the buffer protocol around managed exports (#8523)

* Implement PEP 688 __buffer__ and __release_buffer__

A Python class could not export a buffer: the slot machinery had no
bf_getbuffer or bf_releasebuffer, and every consumer acquired buffers as
PyBUF_FULL_RO through a module of PyBUF_* constants.

Add both slots. PyBuffer::release now runs a Python __release_buffer__
before the exporter's own release, once per acquisition, which PyBuffer
tracks with an `acquired` flag that clones do not inherit. An export made
by a Python __buffer__ is held by a _buffer_wrapper payload that counts its
exports and drops the returned memoryview with the last one, and the view
handed to __release_buffer__ is a _buffer_window that owns no export, so
releasing it inside the hook is inert instead of re-entering it.

Replace the PyBUF_* constants with a BufferFlags bitflags type whose
composite requests are supersets of the simpler ones, so `contains` answers
the REQ_* questions, and pass the request to PyBuffer::from_object. Each
consumer now asks for what its counterpart asks for: y* arguments for
SIMPLE, w* for WRITABLE, BytesIO.write for CONTIG_RO, bytes(), bytearray()
and memoryview() for FULL_RO. memoryview checks the request in
memory_getbuf, and array.array and mmap.mmap expose __release_buffer__.

Test buffer support with PyObject::check_buffer (PyObject_CheckBuffer)
instead of attempting an acquisition, so an exception raised by __buffer__
is no longer reported as the object not being bytes-like, and a __buffer__
with side effects runs once. PyBytesInner becomes a y* conversion as a
result: bytes and bytearray methods no longer accept iterables of ints, and
find, index, count and __contains__ take the arguments
parse_args_finds_byte and bytes_contains describe.

A view exports its start offset in the descriptor rather than in its
window, which fixes a panic when collecting from a negative-stride view.

BytesIO.write rechecks closed after acquiring its buffer, which __buffer__
can close in between.

Assisted-by: Claude Code:claude-opus-5

* Rework the buffer protocol around managed exports and view offsets

Give `PyBuffer` the `_PyManagedBufferObject` shape: one `bf_getbuffer`
acquisition is shared by every handle taken from it, cloning takes another
share instead of re-acquiring, and the exporter's release runs once when the
last share goes away. Remove `retain`, the unsafe `drop_without_release`, the
three `impl Drop`s and the `ManuallyDrop` that stood in for this. Add
`abort_acquisition` so a failed request does not run `bf_releasebuffer`.

Move the view start into `BufferDescriptor::offset`, the `Py_buffer.buf`
analogue, and drop the separate `start` fields on `PyMemoryView` and
`PyBufferWrapper`. Slicing goes through `SaturatedSlice::adjust_indices_start`,
which reproduces `PySlice_AdjustIndices` and keeps the adjusted start.

Fix `zip_eq` to take its contiguous fast path only when both last dimensions
are contiguous, and make `for_each_segment` and `zip_eq` handle zero-length and
zero-dimensional views.

Add `BufferDescriptor::projected` so a request without `PyBUF_ND`,
`PyBUF_STRIDES` or `PyBUF_FORMAT` receives a correspondingly reduced
descriptor, and reject a request without `PyBUF_INDIRECT` against an exporter
that has suboffsets.

Copy the source first in `memoryview` slice assignment when both sides reach
the same root exporter.

Hold the export across the resize in `bytearray.extend`, take `y*` in
`marshal.loads`, stop probing the buffer protocol in `FsPath`, rewrite `ord`
over the concrete string types, fold `array`'s buffer slot into one
`slot_as_buffer`, take `w*`/`y*` in `_overlapped`, and thread the new `offset`
field through the `_ctypes` descriptors.

Assisted-by: Claude

* Re-check a memoryview after a conversion and tighten cast and release

`pack_single` and `unpack_single` addressed the buffer with a position taken
before `__index__` ran, so releasing the view from that conversion read or wrote
outside the exporter's storage, panicking when it had also shrunk. Check the
released flag again once the conversion is done, as `CHECK_RELEASED_AGAIN` does.

Reject a cast to a format that is not a single native format character with an
optional `@` in front of it. An empty format reached a division by its item size
of zero. get_native_fmtchar

Report a second `__release_buffer__` on the same view as a `ValueError` rather
than accepting it, and check that the view belongs to the object first; the
silent case is a view that exports nothing. wrap_releasebuffer

Compare against another memoryview by reading its view where it lies instead of
acquiring a buffer from it, so the restricted view handed to
`__release_buffer__` compares equal rather than unequal in one direction only.
memory_richcompare

Name the type in the unraisable an exception from `__release_buffer__` reports,
as `releasebuffer_call_python` does, instead of reporting the exporter object.

Assisted-by: Claude

* Re-check the destination of a memoryview slice assignment

Acquiring the source runs `__buffer__`, which can release the destination view,
so check the released flag again once the source is in hand and before the
structures are compared, as `copy_single` does.

Build the sliced destination as a view that counts as no export, the way a
`Py_buffer dest = *view` copy does. Holding one kept the exporter unresizable
for the length of the assignment, so a source that released the view and then
resized the exporter met a `BufferError` instead of the assignment reporting the
released view.

Assisted-by: Claude

* Answer the memoryview contiguity getsets with the layout

`f_contiguous` reported the row-major answer for one dimension and False
for any more, so a view laid out both ways, such as one of shape (1, 8),
reported False while `check_buffer_request` accepted a `F_CONTIGUOUS`
request on it. `contiguous` answered row-major order alone.

Move the Fortran-order test from `PyMemoryView` to `BufferDescriptor`,
next to the row-major one it differs from only in iteration order, and
answer all three getsets with it.

Assisted-by: Claude
This commit is contained in:
Jeong, YunWon authored and GitHub committed 2026-08-16 21:32:29 +09:00
1 parent 5c13fa1e25
commit f24b2570a9
45 files changed
+1973 -468

No files matched your search

+1
View File
@@ -59,6 +59,7 @@
"alnum",
"csock",
"coro",
"contig",
"Crnl",
"dedentations",
"dedents",
-11
View File
@@ -4471,7 +4471,6 @@ class TestBufferProtocol(unittest.TestCase):
class TestPythonBufferProtocol(unittest.TestCase):
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_basic(self):
class MyBuffer:
def __buffer__(self, flags):
@@ -4500,7 +4499,6 @@ class TestPythonBufferProtocol(unittest.TestCase):
self.assertRaises(TypeError, memoryview, WrongArity())
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_release_buffer(self):
class WhatToRelease:
def __init__(self):
@@ -4523,7 +4521,6 @@ class TestPythonBufferProtocol(unittest.TestCase):
self.assertEqual(mv.tobytes(), b"hello")
self.assertFalse(wr.held)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_same_buffer_returned(self):
class WhatToRelease:
def __init__(self):
@@ -4549,7 +4546,6 @@ class TestPythonBufferProtocol(unittest.TestCase):
self.assertEqual(mv.tobytes(), b"hello")
self.assertFalse(wr.held)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_buffer_flags(self):
class PossiblyMutable:
def __init__(self, data, mutable) -> None:
@@ -4589,7 +4585,6 @@ class TestPythonBufferProtocol(unittest.TestCase):
mv[0] = ord(b'x')
self.assertEqual(mv.tobytes(), b"hello")
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_call_builtins(self):
ba = bytearray(b"hello")
mv = ba.__buffer__(0)
@@ -4651,7 +4646,6 @@ class TestPythonBufferProtocol(unittest.TestCase):
mv = memoryview(a)
self.assertEqual(mv.tobytes(), b"hello")
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_inheritance_releasebuffer(self):
rb_call_count = 0
class B(bytearray):
@@ -4668,7 +4662,6 @@ class TestPythonBufferProtocol(unittest.TestCase):
self.assertEqual(rb_call_count, 0)
self.assertEqual(rb_call_count, 1)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_inherit_but_return_something_else(self):
class A(bytearray):
def __buffer__(self, flags):
@@ -4708,7 +4701,6 @@ class TestPythonBufferProtocol(unittest.TestCase):
with memoryview(c) as mv:
self.assertEqual(mv.tobytes(), b"hello")
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_release_saves_reference(self):
smuggled_buffer = None
@@ -4736,7 +4728,6 @@ class TestPythonBufferProtocol(unittest.TestCase):
with self.assertRaises(ValueError):
smuggled_buffer.tobytes()
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_release_saves_reference_no_subclassing(self):
ba = bytearray(b"hello")
@@ -4757,7 +4748,6 @@ class TestPythonBufferProtocol(unittest.TestCase):
c.buffer.release()
ba.clear()
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_multiple_inheritance_buffer_last(self):
class A:
def __buffer__(self, flags):
@@ -4817,7 +4807,6 @@ class TestPythonBufferProtocol(unittest.TestCase):
c.clear()
self.assertIs(c.buffer, None)
@unittest.expectedFailure # TODO: RUSTPYTHON
def test_release_buffer_with_exception_set(self):
class A:
def __buffer__(self, flags):
-1
View File
@@ -1956,7 +1956,6 @@ class TestCollectionABCs(ABCTestCase):
# No metaclass conflict
class Z(ByteString, Awaitable): pass
@unittest.expectedFailure # TODO: RUSTPYTHON; Need to implement __buffer__ and __release_buffer__ (https://docs.python.org/3.13/reference/datamodel.html#emulating-buffer-types)
def test_Buffer(self):
for sample in [bytes, bytearray, memoryview]:
self.assertIsInstance(sample(b"x"), Buffer)
-8
View File
@@ -587,7 +587,6 @@ class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin, unittest.TestCase):
self.ioclass(initial_bytes=buf)
self.assertRaises(TypeError, self.ioclass, buf, foo=None)
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: a bytes-like object is required, not 'B'
def test_write_concurrent_close(self):
class B:
def __buffer__(self, flags):
@@ -601,7 +600,6 @@ class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin, unittest.TestCase):
# concurrently mutates (e.g., closes or exports) 'memio'.
# See: https://github.com/python/cpython/issues/143378.
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: a bytes-like object is required, not 'B'
def test_writelines_concurrent_close(self):
class B:
def __buffer__(self, flags):
@@ -611,7 +609,6 @@ class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin, unittest.TestCase):
memio = self.ioclass()
self.assertRaises(ValueError, memio.writelines, [B()])
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: a bytes-like object is required, not 'B'
def test_write_concurrent_export(self):
class B:
buf = None
@@ -622,7 +619,6 @@ class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin, unittest.TestCase):
memio = self.ioclass()
self.assertRaises(BufferError, memio.write, B())
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: a bytes-like object is required, not 'B'
def test_writelines_concurrent_export(self):
class B:
buf = None
@@ -633,7 +629,6 @@ class PyBytesIOTest(MemoryTestMixin, MemorySeekTestMixin, unittest.TestCase):
memio = self.ioclass()
self.assertRaises(BufferError, memio.writelines, [B()])
@unittest.expectedFailure # TODO: RUSTPYTHON; TypeError: a bytes-like object is required, not 'B'
def test_write_mutating_buffer(self):
# Test that buffer is exported only once during write().
# See: https://github.com/python/cpython/issues/143602.
@@ -930,9 +925,6 @@ class CBytesIOTest(PyBytesIOTest):
def test_flags(self):
return super().test_flags()
@unittest.expectedFailure # TODO: RUSTPYTHON; AssertionError: ValueError not raised by write
def test_write(self):
return super().test_write()
class CStringIOTest(PyStringIOTest):
ioclass = io.StringIO
-1
View File
@@ -797,7 +797,6 @@ class OtherTest(unittest.TestCase):
m[0] = MyBool()
self.assertEqual(ba[:8], b'\0'*8)
@unittest.expectedFailure # TODO: RUSTPYTHON; AttributeError: 'memoryview' object has no attribute '__buffer__'
def test_buffer_reference_loop(self):
m = memoryview(b'abc').__buffer__(0)
o = MyObject()
-2
View File
@@ -498,12 +498,10 @@ class StructTest(ComplexesAreIdenticalMixin, unittest.TestCase):
with self.assertRaises((IndexError, OverflowError)):
pack_into(writable_buf, -2**1000, test_string)
@unittest.expectedFailure # TODO: RUSTPYTHON; BufferError: non-contiguous buffer is not a bytes-like object
def test_pack_into(self):
s = struct.Struct('21s')
self._test_pack_into(s.pack_into)
@unittest.expectedFailure # TODO: RUSTPYTHON; BufferError: non-contiguous buffer is not a bytes-like object
def test_pack_into_fn(self):
pack_into = lambda *args: struct.pack_into('21s', *args)
self._test_pack_into(pack_into)
+6 -3
View File
@@ -1162,13 +1162,16 @@ where
let slot_ident = Ident::new(&slot_ident.to_string().to_lowercase(), slot_ident.span());
let slot_name = slot_ident.to_string();
let tokens = {
const NON_ATOMIC_SLOTS: &[&str] = &["as_buffer"];
const POINTER_SLOTS: &[&str] = &["as_sequence", "as_mapping"];
const STATIC_GEN_SLOTS: &[&str] = &["as_number"];
if NON_ATOMIC_SLOTS.contains(&slot_name.as_str()) {
if slot_name == "as_buffer" {
// bf_releasebuffer is not a separate function in RustPython; the
// exporter's BufferMethods already release. Only its presence is
// observable, and AsBuffer declares that.
quote_spanned! { span =>
slots.#slot_ident = Some(Self::#ident as _);
slots.#slot_ident.store(Some(Self::#ident as _));
slots.has_release_buffer.store(Self::RELEASE_BUFFER);
}
} else if POINTER_SLOTS.contains(&slot_name.as_str()) {
quote_spanned! { span =>
+37 -15
View File
@@ -27,8 +27,8 @@ pub mod array {
ArgBytesLike, ArgIntoFloat, ArgIterable, KwArgs, OptionalArg, PyComparisonValue,
},
protocol::{
BufferDescriptor, BufferMethods, BufferResizeGuard, PyBuffer, PyIterReturn,
PyMappingMethods, PySequenceMethods,
BufferDescriptor, BufferFlags, BufferMethods, BufferResizeGuard, PyBuffer,
PyIterReturn, PyMappingMethods, PySequenceMethods,
},
sequence::{OptionalRangeArgs, SequenceExt, SequenceMutExt},
sliceable::{
@@ -732,12 +732,12 @@ pub mod array {
}
} else if init.downcastable::<PyBytes>() || init.downcastable::<PyByteArray>() {
init.try_bytes_like(vm, |x| array.frombytes(x))?;
} else if let Ok(iter) = ArgIterable::try_from_object(vm, init.clone()) {
} else {
// Everything else is taken item by item, buffer or not.
let iter = ArgIterable::try_from_object(vm, init)?;
for obj in iter.iter(vm)? {
array.push(obj?, vm)?;
}
} else {
init.try_bytes_like(vm, |x| array.frombytes(x))?;
}
}
@@ -1291,20 +1291,42 @@ pub mod array {
}
}
impl PyArray {
fn buffer_desc(&self) -> BufferDescriptor {
let array = self.read();
BufferDescriptor::format(
array.len() * array.itemsize(),
false,
array.itemsize(),
array.typecode_str().into(),
)
}
}
impl AsBuffer for PyArray {
const RELEASE_BUFFER: bool = true;
// array_buffer_getbuf, which reports the type code only when the request
// asked for a format.
fn slot_as_buffer(
zelf: &PyObject,
flags: BufferFlags,
vm: &VirtualMachine,
) -> PyResult<PyBuffer> {
let zelf = zelf
.downcast_ref::<Self>()
.ok_or_else(|| vm.new_type_error("unexpected payload for as_buffer"))?;
let desc = zelf.buffer_desc().projected(flags);
flags.check_writable(desc.readonly, "Object is not writable.", vm)?;
Ok(PyBuffer::new(zelf.to_owned().into(), desc, &BUFFER_METHODS))
}
fn as_buffer(zelf: &Py<Self>, _vm: &VirtualMachine) -> PyResult<PyBuffer> {
let array = zelf.read();
let buf = PyBuffer::new(
Ok(PyBuffer::new(
zelf.to_owned().into(),
BufferDescriptor::format(
array.len() * array.itemsize(),
false,
array.itemsize(),
array.typecode_str().into(),
),
zelf.buffer_desc(),
&BUFFER_METHODS,
);
Ok(buf)
))
}
}
+2
View File
@@ -611,6 +611,8 @@ mod mmap {
};
impl AsBuffer for PyMmap {
const RELEASE_BUFFER: bool = true;
fn as_buffer(zelf: &Py<Self>, _vm: &VirtualMachine) -> PyResult<PyBuffer> {
let readonly = matches!(zelf.access, AccessMode::Read);
let buf = PyBuffer::new(
+19 -7
View File
@@ -12,7 +12,7 @@ mod _overlapped {
builtins::{PyBaseExceptionRef, PyBytesRef, PyModule, PyStrRef, PyTupleRef, PyType},
common::lock::PyMutex,
convert::{ToPyException, ToPyObject},
function::OptionalArg,
function::{ArgBytesLike, ArgMemoryBuffer, OptionalArg},
object::{Traverse, TraverseFn},
protocol::PyBuffer,
types::{Constructor, Destructor},
@@ -428,12 +428,14 @@ mod _overlapped {
fn ReadFileInto(
zelf: &Py<Self>,
handle: isize,
buf: PyBuffer,
// w*, as _overlapped.Overlapped.ReadFileInto takes
buf: ArgMemoryBuffer,
vm: &VirtualMachine,
) -> PyResult {
use host_winapi::{
ERROR_BROKEN_PIPE, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_SUCCESS,
};
let buf: PyBuffer = buf.into();
let mut inner = zelf.inner.lock();
if !matches!(inner.data, OverlappedData::None) {
@@ -530,13 +532,15 @@ mod _overlapped {
fn WSARecvInto(
zelf: &Py<Self>,
handle: isize,
buf: PyBuffer,
// w*, as _overlapped.Overlapped.WSARecvInto takes
buf: ArgMemoryBuffer,
flags: u32,
vm: &VirtualMachine,
) -> PyResult {
use host_winapi::{
ERROR_BROKEN_PIPE, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_SUCCESS,
};
let buf: PyBuffer = buf.into();
let mut inner = zelf.inner.lock();
if !matches!(inner.data, OverlappedData::None) {
@@ -583,10 +587,12 @@ mod _overlapped {
fn WriteFile(
zelf: &Py<Self>,
handle: isize,
buf: PyBuffer,
// y*, as _overlapped.Overlapped.WriteFile takes
buf: ArgBytesLike,
vm: &VirtualMachine,
) -> PyResult {
use host_winapi::{ERROR_IO_PENDING, ERROR_SUCCESS};
let buf: PyBuffer = buf.into();
let mut inner = zelf.inner.lock();
if !matches!(inner.data, OverlappedData::None) {
@@ -629,11 +635,13 @@ mod _overlapped {
fn WSASend(
zelf: &Py<Self>,
handle: isize,
buf: PyBuffer,
// y*, as _overlapped.Overlapped.WSASend takes
buf: ArgBytesLike,
flags: u32,
vm: &VirtualMachine,
) -> PyResult {
use host_winapi::{ERROR_IO_PENDING, ERROR_SUCCESS};
let buf: PyBuffer = buf.into();
let mut inner = zelf.inner.lock();
if !matches!(inner.data, OverlappedData::None) {
@@ -870,12 +878,14 @@ mod _overlapped {
fn WSASendTo(
zelf: &Py<Self>,
handle: isize,
buf: PyBuffer,
// y*, as _overlapped.Overlapped.WSASendTo takes
buf: ArgBytesLike,
flags: u32,
address: PyTupleRef,
vm: &VirtualMachine,
) -> PyResult {
use host_winapi::{ERROR_IO_PENDING, ERROR_SUCCESS};
let buf: PyBuffer = buf.into();
let mut inner = zelf.inner.lock();
if !matches!(inner.data, OverlappedData::None) {
@@ -1001,7 +1011,8 @@ mod _overlapped {
fn WSARecvFromInto(
zelf: &Py<Self>,
handle: isize,
buf: PyBuffer,
// w*, as _overlapped.Overlapped.WSARecvFromInto takes
buf: ArgMemoryBuffer,
size: u32,
flags: OptionalArg<u32>,
vm: &VirtualMachine,
@@ -1009,6 +1020,7 @@ mod _overlapped {
use host_winapi::{
ERROR_BROKEN_PIPE, ERROR_IO_PENDING, ERROR_MORE_DATA, ERROR_SUCCESS,
};
let buf: PyBuffer = buf.into();
let mut inner = zelf.inner.lock();
if !matches!(inner.data, OverlappedData::None) {
+19 -17
View File
@@ -1158,19 +1158,19 @@ mod _ssl {
let pwd_result = callable.call((), vm)?;
// Convert callable result to string
let password_from_callable = if let Ok(pwd_str) =
PyUtf8StrRef::try_from_object(vm, pwd_result.clone())
{
pwd_str.as_str().to_owned()
} else if let Ok(pwd_bytes_like) = ArgBytesLike::try_from_object(vm, pwd_result) {
String::from_utf8(pwd_bytes_like.borrow_buf().to_vec()).map_err(|_| {
vm.new_type_error("password callback returned invalid UTF-8 bytes")
})?
} else {
return Err(
vm.new_type_error("password callback must return a string or bytes")
);
};
let password_from_callable =
if let Ok(pwd_str) = PyUtf8StrRef::try_from_object(vm, pwd_result.clone()) {
pwd_str.as_str().to_owned()
} else if pwd_result.check_buffer() {
let pwd_bytes_like = ArgBytesLike::try_from_object(vm, pwd_result)?;
String::from_utf8(pwd_bytes_like.borrow_buf().to_vec()).map_err(|_| {
vm.new_type_error("password callback returned invalid UTF-8 bytes")
})?
} else {
return Err(
vm.new_type_error("password callback must return a string or bytes")
);
};
// Validate callable password length
if password_from_callable.len() > PEM_BUFSIZE {
@@ -1808,7 +1808,8 @@ mod _ssl {
// Validate filepath is str or bytes
let path_str = if let Ok(s) = PyUtf8StrRef::try_from_object(vm, filepath.clone()) {
s.as_str().to_owned()
} else if let Ok(b) = ArgBytesLike::try_from_object(vm, filepath) {
} else if filepath.check_buffer() {
let b = ArgBytesLike::try_from_object(vm, filepath)?;
String::from_utf8(b.borrow_buf().to_vec())
.map_err(|_| vm.new_value_error("Invalid path encoding"))?
} else {
@@ -1863,7 +1864,8 @@ mod _ssl {
// Validate name is str or bytes
let curve_name = if let Ok(s) = PyUtf8StrRef::try_from_object(vm, name.clone()) {
s.as_str().to_owned()
} else if let Ok(b) = ArgBytesLike::try_from_object(vm, name) {
} else if name.check_buffer() {
let b = ArgBytesLike::try_from_object(vm, name)?;
String::from_utf8(b.borrow_buf().to_vec())
.map_err(|_| vm.new_value_error("Invalid curve name encoding"))?
} else {
@@ -2106,8 +2108,8 @@ mod _ssl {
Ok((Some(pwd_str.as_str().to_owned()), None))
}
// Try bytes-like
else if let Ok(pwd_bytes_like) = ArgBytesLike::try_from_object(vm, p.clone())
{
else if p.check_buffer() {
let pwd_bytes_like = ArgBytesLike::try_from_object(vm, p.clone())?;
let pwd = String::from_utf8(pwd_bytes_like.borrow_buf().to_vec())
.map_err(|_| vm.new_type_error("password bytes must be valid UTF-8"))?;
Ok((Some(pwd), None))
+18 -12
View File
@@ -4,7 +4,7 @@ use num_traits::{cast::ToPrimitive, sign::Signed};
use rustpython_unicode::case;
use crate::{
Py, PyObject, PyObjectRef, PyResult, TryFromObject, VirtualMachine,
AsObject, PyObject, PyObjectRef, PyResult, TryFromObject, VirtualMachine,
builtins::{PyIntRef, PyTuple},
convert::TryFromBorrowedObject,
function::OptionalOption,
@@ -485,19 +485,25 @@ where
F: Fn(T) -> PyResult<bool>,
M: Fn(&PyObject) -> String,
{
if let Ok(single) = obj.try_to_value::<T>(vm) {
(predicate)(single)
} else {
let tuple: &Py<PyTuple> = obj
.try_to_value(vm)
.map_err(|_| vm.new_type_error((message)(obj)))?;
for obj in tuple {
if single_or_tuple_any(obj, predicate, message, vm)? {
// _Py_bytes_tailmatch: a tuple is taken apart before anything is converted, and
// each item is converted on its own terms, so a tuple of tuples is not an affix.
if let Some(tuple) = obj.downcast_ref::<PyTuple>() {
for item in tuple {
if (predicate)(item.try_to_value::<T>(vm)?)? {
return Ok(true);
}
}
Ok(false)
return Ok(false);
}
// Only the argument simply being the wrong kind of object is reported as such;
// whatever the conversion itself raised belongs to the caller.
let single = obj.try_to_value::<T>(vm).map_err(|exc| {
if exc.fast_isinstance(vm.ctx.exceptions.type_error) {
vm.new_type_error((message)(obj))
} else {
exc
}
})?;
(predicate)(single)
}
+50 -18
View File
@@ -1,7 +1,7 @@
//! Implementation of the python bytearray object.
use super::{
PositionIterInternal, PyBytes, PyDictRef, PyGenericAlias, PyIntRef, PyStrRef, PyTuple,
PyTupleRef, PyType, PyTypeRef, iter::builtins_iter,
PositionIterInternal, PyBytes, PyDictRef, PyGenericAlias, PyStrRef, PyTuple, PyTupleRef,
PyType, PyTypeRef, iter::builtins_iter,
};
use crate::{
AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, TryFromObject,
@@ -11,7 +11,8 @@ use crate::{
byte::{bytes_from_object, value_from_object},
bytes_inner::{
ByteInnerFindOptions, ByteInnerHexOptions, ByteInnerNewOptions, ByteInnerPaddingOptions,
ByteInnerSplitOptions, ByteInnerTranslateOptions, DecodeArgs, PyBytesInner, bytes_decode,
ByteInnerSplitOptions, ByteInnerSub, ByteInnerTranslateOptions, DecodeArgs, PyBytesInner,
bytes_decode,
},
class::PyClassImpl,
common::{
@@ -23,10 +24,10 @@ use crate::{
},
convert::{ToPyObject, ToPyResult},
function::{
ArgBytesLike, ArgIterable, ArgSize, Either, OptionalArg, OptionalOption, PyComparisonValue,
ArgBytesLike, ArgIterable, ArgSize, OptionalArg, OptionalOption, PyComparisonValue,
},
protocol::{
BufferDescriptor, BufferMethods, BufferResizeGuard, PyBuffer, PyIterReturn,
BufferDescriptor, BufferFlags, BufferMethods, BufferResizeGuard, PyBuffer, PyIterReturn,
PyMappingMethods, PyNumberMethods, PySequenceMethods,
},
sliceable::{SequenceIndex, SliceableSequenceMutOp, SliceableSequenceOp},
@@ -228,11 +229,8 @@ impl PyByteArray {
self.inner().add(&other.borrow_buf()).into()
}
fn __contains__(
&self,
needle: Either<PyBytesInner, PyIntRef>,
vm: &VirtualMachine,
) -> PyResult<bool> {
fn __contains__(&self, needle: PyObjectRef, vm: &VirtualMachine) -> PyResult<bool> {
let needle = ByteInnerSub::from_contains_arg(needle, vm)?;
self.inner().contains(needle, vm)
}
@@ -613,12 +611,34 @@ impl Py<PyByteArray> {
#[pymethod]
fn extend(&self, object: PyObjectRef, vm: &VirtualMachine) -> PyResult<()> {
if self.is(&object) {
PyByteArray::irepeat(self, 2, vm)
} else {
let items = bytes_from_object(vm, &object)?;
self.try_resizable(vm)?.elements.extend(items);
Ok(())
return PyByteArray::irepeat(self, 2, vm);
}
// bytearray_setslice keeps the export alive across the resize, so a value
// looking at this bytearray is what stops it from growing.
let buffer = object
.check_buffer()
.then(|| {
PyBuffer::from_object(vm, &object, BufferFlags::SIMPLE).map_err(|_| {
// What an exporter refuses to hand out leaves the value simply
// not usable here, whatever the exporter's own complaint was.
vm.new_type_error(format!(
"can't set bytearray slice from {}",
object.class().name()
))
})
})
.transpose()?;
let items = match &buffer {
Some(buffer) => buffer
.as_contiguous()
.ok_or_else(|| {
vm.new_buffer_error("non-contiguous buffer is not a bytes-like object")
})?
.to_vec(),
None => bytes_from_object(vm, &object)?,
};
self.try_resizable(vm)?.elements.extend(items);
Ok(())
}
#[pymethod]
@@ -731,6 +751,20 @@ static BUFFER_METHODS: BufferMethods = BufferMethods {
};
impl AsBuffer for PyByteArray {
const RELEASE_BUFFER: bool = true;
fn slot_as_buffer(
zelf: &PyObject,
flags: BufferFlags,
vm: &VirtualMachine,
) -> PyResult<PyBuffer> {
let zelf = zelf
.downcast_ref::<Self>()
.ok_or_else(|| vm.new_type_error("unexpected payload for as_buffer"))?;
flags.fill_info_check(false, vm)?;
Self::as_buffer(zelf, vm)
}
fn as_buffer(zelf: &Py<Self>, _vm: &VirtualMachine) -> PyResult<PyBuffer> {
Ok(PyBuffer::new(
zelf.to_owned().into(),
@@ -801,9 +835,7 @@ impl AsSequence for PyByteArray {
}
}),
contains: atomic_func!(|seq, other, vm| {
let other =
<Either<PyBytesInner, PyIntRef>>::try_from_object(vm, other.to_owned())?;
PyByteArray::sequence_downcast(seq).__contains__(other, vm)
PyByteArray::sequence_downcast(seq).__contains__(other.to_owned(), vm)
}),
inplace_concat: atomic_func!(|seq, other, vm| {
let other = ArgBytesLike::try_from_object(vm, other.to_owned())?;
+23 -15
View File
@@ -1,27 +1,28 @@
use super::{
PositionIterInternal, PyDictRef, PyGenericAlias, PyIntRef, PyStrRef, PyTuple, PyTupleRef,
PyType, PyTypeRef, iter::builtins_iter,
PositionIterInternal, PyDictRef, PyGenericAlias, PyStrRef, PyTuple, PyTupleRef, PyType,
PyTypeRef, iter::builtins_iter,
};
use crate::common::lock::LazyLock;
use crate::{
AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult,
TryFromBorrowedObject, TryFromObject, VirtualMachine,
TryFromBorrowedObject, VirtualMachine,
anystr::{self, AnyStr},
atomic_func,
bytes_inner::{
ByteInnerFindOptions, ByteInnerHexOptions, ByteInnerNewOptions, ByteInnerPaddingOptions,
ByteInnerSplitOptions, ByteInnerTranslateOptions, DecodeArgs, PyBytesInner, bytes_decode,
ByteInnerSplitOptions, ByteInnerSub, ByteInnerTranslateOptions, DecodeArgs, PyBytesInner,
bytes_decode,
},
class::PyClassImpl,
common::{hash::PyHash, lock::PyMutex},
convert::{ToPyObject, ToPyResult},
function::{
ArgBytesLike, ArgIndex, ArgIterable, Either, FuncArgs, OptionalArg, OptionalOption,
ArgBytesLike, ArgIndex, ArgIterable, FuncArgs, OptionalArg, OptionalOption,
PyComparisonValue,
},
protocol::{
BufferDescriptor, BufferMethods, PyBuffer, PyIterReturn, PyMappingMethods, PyNumberMethods,
PySequenceMethods,
BufferDescriptor, BufferFlags, BufferMethods, PyBuffer, PyIterReturn, PyMappingMethods,
PyNumberMethods, PySequenceMethods,
},
sliceable::{SequenceIndex, SliceableSequenceOp},
types::{
@@ -246,11 +247,8 @@ impl PyBytes {
self.inner.add(&other.borrow_buf())
}
fn __contains__(
&self,
needle: Either<PyBytesInner, PyIntRef>,
vm: &VirtualMachine,
) -> PyResult<bool> {
fn __contains__(&self, needle: PyObjectRef, vm: &VirtualMachine) -> PyResult<bool> {
let needle = ByteInnerSub::from_contains_arg(needle, vm)?;
self.inner.contains(needle, vm)
}
@@ -627,6 +625,18 @@ static BUFFER_METHODS: BufferMethods = BufferMethods {
};
impl AsBuffer for PyBytes {
fn slot_as_buffer(
zelf: &PyObject,
flags: BufferFlags,
vm: &VirtualMachine,
) -> PyResult<PyBuffer> {
let zelf = zelf
.downcast_ref::<Self>()
.ok_or_else(|| vm.new_type_error("unexpected payload for as_buffer"))?;
flags.fill_info_check(true, vm)?;
Self::as_buffer(zelf, vm)
}
fn as_buffer(zelf: &Py<Self>, _vm: &VirtualMachine) -> PyResult<PyBuffer> {
let buf = PyBuffer::new(
zelf.to_owned().into(),
@@ -673,9 +683,7 @@ impl AsSequence for PyBytes {
.map(|x| vm.ctx.new_bytes(vec![x]).into())
}),
contains: atomic_func!(|seq, other, vm| {
let other =
<Either<PyBytesInner, PyIntRef>>::try_from_object(vm, other.to_owned())?;
PyBytes::sequence_downcast(seq).__contains__(other, vm)
PyBytes::sequence_downcast(seq).__contains__(other.to_owned(), vm)
}),
..PySequenceMethods::NOT_IMPLEMENTED
});
+37
View File
@@ -542,6 +542,10 @@ pub enum SlotFunc {
NumBinaryRight(PyNumberBinaryFunc), // __radd__, __rsub__, etc. (swapped args)
NumTernary(PyNumberTernaryFunc), // __pow__
NumTernaryRight(PyNumberTernaryFunc), // __rpow__ (swapped first two args)
// Buffer protocol
GetBuffer(crate::types::AsBufferFunc), // __buffer__
ReleaseBuffer, // __release_buffer__
}
impl core::fmt::Debug for SlotFunc {
@@ -582,6 +586,8 @@ impl core::fmt::Debug for SlotFunc {
Self::NumBinaryRight(_) => write!(f, "SlotFunc::NumBinaryRight(...)"),
Self::NumTernary(_) => write!(f, "SlotFunc::NumTernary(...)"),
Self::NumTernaryRight(_) => write!(f, "SlotFunc::NumTernaryRight(...)"),
Self::GetBuffer(_) => write!(f, "SlotFunc::GetBuffer(...)"),
Self::ReleaseBuffer => write!(f, "SlotFunc::ReleaseBuffer"),
}
}
}
@@ -758,10 +764,41 @@ impl SlotFunc {
let z = z.unwrap_or_else(|| vm.ctx.none());
func(&y, &obj, &z, vm) // Swapped: y ** obj % z
}
// Buffer protocol
Self::GetBuffer(func) => {
let (flags_obj,): (PyObjectRef,) = args.bind(vm)?;
let buffer = func(&obj, parse_buffer_flags(flags_obj, vm)?, vm)?;
crate::builtins::PyMemoryView::from_buffer(buffer, vm)
.map(|mv| mv.into_pyobject(vm))
}
Self::ReleaseBuffer => {
let (mv_obj,): (PyObjectRef,) = args.bind(vm)?;
let mv = mv_obj
.downcast::<crate::builtins::PyMemoryView>()
.map_err(|_| vm.new_type_error("expected a memoryview object"))?;
crate::builtins::memory::release_buffer_from_python(&obj, mv, vm)?;
Ok(vm.ctx.none())
}
}
}
}
/// Parse the `flags` argument of `__buffer__`. wrap_buffer
fn parse_buffer_flags(
arg: PyObjectRef,
vm: &VirtualMachine,
) -> PyResult<crate::protocol::BufferFlags> {
use num_traits::ToPrimitive;
let idx = arg.try_index(vm)?;
let flags = idx
.as_bigint()
.to_isize()
.ok_or_else(|| vm.new_overflow_error("cannot fit 'int' into an index-sized integer"))?;
let flags =
i32::try_from(flags).map_err(|_| vm.new_overflow_error("buffer flags out of range"))?;
Ok(crate::protocol::BufferFlags::from_bits_retain(flags as u32))
}
/// wrapper_descriptor: wraps a slot function as a Python method
// = PyWrapperDescrObject
#[pyclass(name = "wrapper_descriptor", module = false)]
+8 -8
View File
@@ -3,7 +3,7 @@ use crate::{
AsObject, Context, Py, PyObject, PyObjectRef, PyPayload, PyRef, PyRefExact, PyResult,
TryFromBorrowedObject, VirtualMachine,
builtins::PyUtf8StrRef,
bytes_inner::PyBytesInner,
byte::bytes_from_object,
class::PyClassImpl,
common::{
format::FormatSpec,
@@ -572,13 +572,13 @@ impl PyInt {
vm: &VirtualMachine,
) -> PyResult<PyRef<Self>> {
let signed = args.signed.map_or(false, Into::into);
// PyObject_Bytes, so an iterable of ints is as good as a buffer
let bytes = bytes_from_object(vm, &args.bytes)?;
let value = match (args.byteorder, signed) {
(ArgByteOrder::Big, true) => BigInt::from_signed_bytes_be(args.bytes.as_bytes()),
(ArgByteOrder::Big, false) => BigInt::from_bytes_be(Sign::Plus, args.bytes.as_bytes()),
(ArgByteOrder::Little, true) => BigInt::from_signed_bytes_le(args.bytes.as_bytes()),
(ArgByteOrder::Little, false) => {
BigInt::from_bytes_le(Sign::Plus, args.bytes.as_bytes())
}
(ArgByteOrder::Big, true) => BigInt::from_signed_bytes_be(&bytes),
(ArgByteOrder::Big, false) => BigInt::from_bytes_be(Sign::Plus, &bytes),
(ArgByteOrder::Little, true) => BigInt::from_signed_bytes_le(&bytes),
(ArgByteOrder::Little, false) => BigInt::from_bytes_le(Sign::Plus, &bytes),
};
Self::with_value(cls, value, vm)
}
@@ -802,7 +802,7 @@ pub(crate) struct IntOptions {
#[derive(FromArgs)]
struct IntFromByteArgs {
bytes: PyBytesInner,
bytes: PyObjectRef,
#[pyarg(any, default = ArgByteOrder::Big)]
byteorder: ArgByteOrder,
#[pyarg(named, optional)]
File diff suppressed because it is too large. Load diff
+20 -9
View File
@@ -23,7 +23,9 @@ use crate::{
function::{ArgIterable, ArgSize, FuncArgs, OptionalArg, OptionalOption, PyComparisonValue},
intern::PyInterned,
object::{MaybeTraverse, Traverse, TraverseFn},
protocol::{PyIterReturn, PyMappingMethods, PyNumberMethods, PySequenceMethods},
protocol::{
BufferFlags, PyBuffer, PyIterReturn, PyMappingMethods, PyNumberMethods, PySequenceMethods,
},
sequence::SequenceExt,
sliceable::{SequenceIndex, SliceableSequenceOp},
types::{
@@ -441,15 +443,24 @@ impl Constructor for PyStr {
if input.fast_isinstance(vm.ctx.types.str_type) {
return Err(vm.new_type_error("decoding str is not supported"));
}
if !input.fast_isinstance(vm.ctx.types.bytes_type)
&& !input.fast_isinstance(vm.ctx.types.bytearray_type)
&& crate::protocol::PyBuffer::try_from_borrowed_object(vm, &input).is_err()
let input = if input.fast_isinstance(vm.ctx.types.bytes_type)
|| input.fast_isinstance(vm.ctx.types.bytearray_type)
{
return Err(vm.new_type_error(format!(
"decoding to str: need a bytes-like object, {} found",
input.class().name()
)));
}
input
} else {
// PyUnicode_FromEncodedObject: whatever an exporter
// complains about, the argument is simply not bytes-like.
let buffer = PyBuffer::from_object(vm, &input, BufferFlags::SIMPLE)
.map_err(|_| {
vm.new_type_error(format!(
"decoding to str: need a bytes-like object, {} found",
input.class().name()
))
})?;
vm.ctx
.new_bytes(buffer.contiguous_or_collect(<[u8]>::to_vec))
.into()
};
let enc_str = encoding.as_ref().map_or("utf-8", |e| e.as_str());
let s = vm
.state
-13
View File
@@ -821,8 +821,6 @@ impl PyType {
slots.basicsize = base.slots.basicsize;
}
Self::inherit_readonly_slots(&mut slots, &base);
// Normalize: any type with HAS_WEAKREF gets MANAGED_WEAKREF
if slots.flags.has_feature(PyTypeFlags::HAS_WEAKREF) {
slots.flags |= PyTypeFlags::MANAGED_WEAKREF;
@@ -891,8 +889,6 @@ impl PyType {
slots.basicsize = base.slots.basicsize;
}
Self::inherit_readonly_slots(&mut slots, &base);
// Normalize: any type with HAS_WEAKREF gets MANAGED_WEAKREF
if slots.flags.has_feature(PyTypeFlags::HAS_WEAKREF) {
slots.flags |= PyTypeFlags::MANAGED_WEAKREF;
@@ -1018,18 +1014,9 @@ impl PyType {
}
}
/// Inherit readonly slots from base type at creation time.
/// These slots are not AtomicCell and must be set before the type is used.
fn inherit_readonly_slots(slots: &mut PyTypeSlots, base: &Self) {
if slots.as_buffer.is_none() {
slots.as_buffer = base.slots.as_buffer;
}
}
/// Inherit slots from base type. inherit_slots
pub(crate) fn inherit_slots(&self, base: &Self) {
// Use SLOT_DEFS to iterate all slots
// Note: as_buffer is handled in inherit_readonly_slots (not AtomicCell)
for def in SLOT_DEFS {
def.accessor.copyslot_if_none(self, base);
}
+8 -3
View File
@@ -2,11 +2,16 @@
use num_traits::ToPrimitive;
use crate::{AsObject, PyObject, PyResult, VirtualMachine};
use crate::{
AsObject, PyObject, PyResult, VirtualMachine,
protocol::{BufferFlags, PyBuffer},
};
// PyBytes_FromObject
pub fn bytes_from_object(vm: &VirtualMachine, obj: &PyObject) -> PyResult<Vec<u8>> {
if let Ok(elements) = obj.try_bytes_like(vm, |bytes| bytes.to_vec()) {
return Ok(elements);
if obj.check_buffer() {
let buffer = PyBuffer::from_object(vm, obj, BufferFlags::FULL_RO)?;
return Ok(buffer.contiguous_or_collect(|bytes| bytes.to_vec()));
}
if !obj.fast_isinstance(vm.ctx.types.str_type)
+58 -21
View File
@@ -1,6 +1,7 @@
// spell-checker:ignore unchunked
use crate::{
AsObject, PyObject, PyObjectRef, PyResult, TryFromBorrowedObject, VirtualMachine,
AsObject, PyObject, PyObjectRef, PyResult, TryFromBorrowedObject, TryFromObject,
VirtualMachine,
anystr::{self, AnyStr, AnyStrContainer, AnyStrWrapper},
builtins::{
PyBaseExceptionRef, PyByteArray, PyBytes, PyBytesRef, PyInt, PyIntRef, PyStr, PyStrRef,
@@ -12,7 +13,7 @@ use crate::{
common::wtf8::is_py_ascii_whitespace,
function::{ArgIterable, Either, OptionalArg, OptionalOption, PyComparisonValue},
literal::escape::Escape,
protocol::PyBuffer,
protocol::{BufferFlags, PyBuffer},
sequence::{SequenceExt, SequenceMutExt},
types::PyComparisonOp,
};
@@ -35,9 +36,10 @@ impl From<Vec<u8>> for PyBytesInner {
}
}
/// "y*": any bytes-like object, and nothing else.
impl<'a> TryFromBorrowedObject<'a> for PyBytesInner {
fn try_from_borrowed_object(vm: &VirtualMachine, obj: &'a PyObject) -> PyResult<Self> {
bytes_from_object(vm, obj).map(Self::from)
obj.try_bytes_like(vm, <[u8]>::to_vec).map(Self::from)
}
}
@@ -137,10 +139,50 @@ impl ByteInnerNewOptions {
}
}
/// What is searched for: a bytes-like object, or a single byte given as an
/// integer. parse_args_finds_byte
pub enum ByteInnerSub {
Buffer(PyBytesInner),
Byte(PyIntRef),
}
impl TryFromObject for ByteInnerSub {
fn try_from_object(vm: &VirtualMachine, obj: PyObjectRef) -> PyResult<Self> {
if obj.check_buffer() {
return PyBytesInner::try_from_object(vm, obj).map(Self::Buffer);
}
match obj.try_index_opt(vm) {
Some(int) => int.map(Self::Byte),
None => Err(vm.new_type_error(format!(
"argument should be integer or bytes-like object, not '{}'",
obj.class().name()
))),
}
}
}
impl ByteInnerSub {
/// The needle of a containment test, which is an integer if it is one at
/// all and a bytes-like object otherwise. bytes_contains
pub fn from_contains_arg(obj: PyObjectRef, vm: &VirtualMachine) -> PyResult<Self> {
match obj.try_index_opt(vm) {
Some(int) => int.map(Self::Byte),
None => PyBytesInner::try_from_object(vm, obj).map(Self::Buffer),
}
}
fn into_vec(self, vm: &VirtualMachine) -> PyResult<Vec<u8>> {
Ok(match self {
Self::Buffer(buffer) => buffer.elements,
Self::Byte(int) => vec![int.as_bigint().byte_or(vm)?],
})
}
}
#[derive(FromArgs)]
pub struct ByteInnerFindOptions {
#[pyarg(positional)]
sub: Either<PyBytesInner, PyIntRef>,
sub: ByteInnerSub,
#[pyarg(positional, default)]
start: Option<PyIntRef>,
#[pyarg(positional, default)]
@@ -153,10 +195,7 @@ impl ByteInnerFindOptions {
len: usize,
vm: &VirtualMachine,
) -> PyResult<(Vec<u8>, core::ops::Range<usize>)> {
let sub = match self.sub {
Either::A(v) => v.elements.to_vec(),
Either::B(int) => vec![int.as_bigint().byte_or(vm)?],
};
let sub = self.sub.into_vec(vm)?;
let range = anystr::adjust_indices(self.start, self.end, len);
Ok((sub, range))
}
@@ -203,14 +242,11 @@ impl ByteInnerTranslateOptions {
let table = self.table.map_or_else(
|| Ok((0..=u8::MAX).collect::<Vec<u8>>()),
|v| {
let bytes = v
.try_into_value::<PyBytesInner>(vm)
.ok()
.filter(|v| v.elements.len() == 256)
.ok_or_else(|| {
vm.new_value_error("translation table must be 256 characters long")
})?;
Ok(bytes.elements.to_vec())
let bytes: PyBytesInner = v.try_into_value(vm)?;
if bytes.elements.len() != 256 {
return Err(vm.new_value_error("translation table must be 256 characters long"));
}
Ok(bytes.elements)
},
)?;
@@ -363,10 +399,10 @@ impl PyBytesInner {
self.elements.py_add(other)
}
pub fn contains(&self, needle: Either<Self, PyIntRef>, vm: &VirtualMachine) -> PyResult<bool> {
pub fn contains(&self, needle: ByteInnerSub, vm: &VirtualMachine) -> PyResult<bool> {
Ok(match needle {
Either::A(byte) => self.elements.contains_str(byte.elements.as_slice()),
Either::B(int) => self.elements.contains(&int.as_bigint().byte_or(vm)?),
ByteInnerSub::Buffer(sub) => self.elements.contains_str(sub.elements.as_slice()),
ByteInnerSub::Byte(int) => self.elements.contains(&int.as_bigint().byte_or(vm)?),
})
}
@@ -522,7 +558,8 @@ impl PyBytesInner {
pub fn fromhex_object(string: PyObjectRef, vm: &VirtualMachine) -> PyResult<Vec<u8>> {
if let Some(s) = string.downcast_ref::<PyStr>() {
Self::fromhex(s.as_bytes(), vm)
} else if let Ok(buffer) = PyBuffer::try_from_borrowed_object(vm, &string) {
} else if string.check_buffer() {
let buffer = PyBuffer::from_object(vm, &string, BufferFlags::SIMPLE)?;
let borrowed = buffer
.as_contiguous()
.ok_or_else(|| vm.new_buffer_error("fromhex() requires a contiguous buffer"))?;
@@ -974,7 +1011,7 @@ impl PyBytesInner {
}
pub fn concat(&self, other: &PyObject, vm: &VirtualMachine) -> PyResult<Vec<u8>> {
let buffer = PyBuffer::try_from_borrowed_object(vm, other)?;
let buffer = PyBuffer::from_object(vm, other, BufferFlags::SIMPLE)?;
let borrowed = buffer.as_contiguous();
if let Some(other) = borrowed {
let mut v = Vec::with_capacity(self.elements.len() + other.len());
+22 -17
View File
@@ -22,7 +22,7 @@ use crate::{
wtf8::{CodePoint, Wtf8, Wtf8Buf},
},
function::ArgIntoFloat,
protocol::PyBuffer,
protocol::{BufferFlags, PyBuffer},
stdlib::builtins,
};
@@ -39,24 +39,29 @@ fn spec_format_bytes(
let b = builtins::ascii(obj, vm)?.as_bytes().to_vec();
Ok(b)
}
// format_obj
CFormatConversion::Str | CFormatConversion::Bytes => {
if let Ok(buffer) = PyBuffer::try_from_borrowed_object(vm, &obj) {
Ok(buffer.contiguous_or_collect(|bytes| spec.format_bytes(bytes)))
} else {
let bytes = vm
.get_special_method(&obj, identifier!(vm, __bytes__))?
.ok_or_else(|| {
let msg = format!(
"%b requires a bytes-like object, or an object that \
implements __bytes__, not '{}'",
obj.class().name()
);
vm.new_type_error(msg)
})?
.invoke((), vm)?;
let bytes = PyBytes::try_from_borrowed_object(vm, &bytes)?;
Ok(spec.format_bytes(bytes.as_bytes()))
if let Some(bytes) = obj.downcast_ref::<PyBytes>() {
return Ok(spec.format_bytes(bytes.as_bytes()));
}
if let Some(bytearray) = obj.downcast_ref::<PyByteArray>() {
return Ok(spec.format_bytes(&bytearray.borrow_buf()));
}
if let Some(method) = vm.get_special_method(&obj, identifier!(vm, __bytes__))? {
let bytes = method.invoke((), vm)?;
let bytes = PyBytes::try_from_borrowed_object(vm, &bytes)?;
return Ok(spec.format_bytes(bytes.as_bytes()));
}
if obj.check_buffer() {
let buffer = PyBuffer::from_object(vm, &obj, BufferFlags::FULL_RO)?;
return Ok(buffer.contiguous_or_collect(|bytes| spec.format_bytes(bytes)));
}
let msg = format!(
"%b requires a bytes-like object, or an object that \
implements __bytes__, not '{}'",
obj.class().name()
);
Err(vm.new_type_error(msg))
}
},
CFormatType::Number(number_type) => match number_type {
+40 -7
View File
@@ -3,7 +3,7 @@ use crate::{
VirtualMachine,
builtins::{PyStr, PyStrRef},
common::borrow::{BorrowedValue, BorrowedValueMut},
protocol::PyBuffer,
protocol::{BufferFlags, PyBuffer},
};
// Python/getargs.c
@@ -17,7 +17,7 @@ impl PyObject {
where
F: FnOnce(&[u8]) -> R,
{
let buffer = PyBuffer::try_from_borrowed_object(vm, self)?;
let buffer = PyBuffer::from_object(vm, self, BufferFlags::SIMPLE)?;
buffer
.as_contiguous()
.map(|x| f(&x))
@@ -28,7 +28,7 @@ impl PyObject {
where
F: FnOnce(&mut [u8]) -> R,
{
let buffer = PyBuffer::try_from_borrowed_object(vm, self)?;
let buffer = PyBuffer::from_object(vm, self, BufferFlags::WRITABLE)?;
buffer
.as_contiguous_mut()
.map(|mut x| f(&mut x))
@@ -77,9 +77,9 @@ impl From<ArgBytesLike> for PyObjectRef {
}
}
impl<'a> TryFromBorrowedObject<'a> for ArgBytesLike {
fn try_from_borrowed_object(vm: &VirtualMachine, obj: &'a PyObject) -> PyResult<Self> {
let buffer = PyBuffer::try_from_borrowed_object(vm, obj)?;
impl ArgBytesLike {
fn from_request(vm: &VirtualMachine, obj: &PyObject, flags: BufferFlags) -> PyResult<Self> {
let buffer = PyBuffer::from_object(vm, obj, flags)?;
if buffer.desc.is_contiguous() {
Ok(Self(buffer))
} else {
@@ -88,6 +88,31 @@ impl<'a> TryFromBorrowedObject<'a> for ArgBytesLike {
}
}
impl<'a> TryFromBorrowedObject<'a> for ArgBytesLike {
fn try_from_borrowed_object(vm: &VirtualMachine, obj: &'a PyObject) -> PyResult<Self> {
Self::from_request(vm, obj, BufferFlags::SIMPLE)
}
}
/// A bytes-like object asked for as `PyBUF_CONTIG_RO`, which is what a shape is
/// requested with rather than assumed.
#[derive(Debug, Traverse)]
pub struct ArgContiguousBytesLike(ArgBytesLike);
impl core::ops::Deref for ArgContiguousBytesLike {
type Target = ArgBytesLike;
fn deref(&self) -> &Self::Target {
&self.0
}
}
impl<'a> TryFromBorrowedObject<'a> for ArgContiguousBytesLike {
fn try_from_borrowed_object(vm: &VirtualMachine, obj: &'a PyObject) -> PyResult<Self> {
ArgBytesLike::from_request(vm, obj, BufferFlags::CONTIG_RO).map(Self)
}
}
/// A memory buffer, read-write access. Like the `w*` format code for `PyArg_Parse` in CPython.
#[derive(Debug, Traverse)]
pub struct ArgMemoryBuffer(PyBuffer);
@@ -124,7 +149,15 @@ impl From<ArgMemoryBuffer> for PyBuffer {
impl<'a> TryFromBorrowedObject<'a> for ArgMemoryBuffer {
fn try_from_borrowed_object(vm: &VirtualMachine, obj: &'a PyObject) -> PyResult<Self> {
let buffer = PyBuffer::try_from_borrowed_object(vm, obj)?;
let buffer = PyBuffer::from_object(vm, obj, BufferFlags::WRITABLE).map_err(|exc| {
if obj.check_buffer() {
// An exporter that cannot serve the request leaves the argument
// simply the wrong kind of object, as `PyArg_Parse` reports it.
vm.new_type_error("buffer is not a read-write bytes-like object")
} else {
exc
}
})?;
if !buffer.desc.is_contiguous() {
Err(vm.new_buffer_error("non-contiguous buffer is not a bytes-like object"))
} else if buffer.desc.readonly {
+2 -10
View File
@@ -3,7 +3,6 @@ use crate::{
builtins::{PyBytes, PyBytesRef, PyStrRef},
convert::{IntoPyException, ToPyObject},
function::PyStr,
protocol::PyBuffer,
};
use alloc::borrow::Cow;
use core::hint::cold_path;
@@ -147,16 +146,9 @@ impl ToPyObject for FsPath {
}
impl TryFromObject for FsPath {
// PyUnicode_FSDecoder in CPython
// PyUnicode_FSDecoder, which takes what PyOS_FSPath takes: str, bytes, or an
// object with __fspath__, and nothing that merely exports a buffer.
fn try_from_object(vm: &VirtualMachine, obj: PyObjectRef) -> PyResult<Self> {
let obj = match obj.try_to_value::<PyBuffer>(vm) {
Ok(buffer) => {
let mut bytes = vec![];
buffer.append_to(&mut bytes);
vm.ctx.new_bytes(bytes).into()
}
Err(_) => obj,
};
Self::try_from_path_like(obj, true, vm)
}
}
+3 -1
View File
@@ -15,7 +15,9 @@ pub use argument::{
OptionalArg, OptionalOption, PosArgs,
};
pub use arithmetic::{PyArithmeticValue, PyComparisonValue};
pub use buffer::{ArgAsciiBuffer, ArgBytesLike, ArgMemoryBuffer, ArgStrOrBytesLike};
pub use buffer::{
ArgAsciiBuffer, ArgBytesLike, ArgContiguousBytesLike, ArgMemoryBuffer, ArgStrOrBytesLike,
};
pub use builtin::{IntoPyNativeFn, PyNativeFn, static_func, static_raw_func};
pub use either::Either;
pub use fspath::FsPath;
+381 -30
View File
@@ -6,14 +6,101 @@ use crate::{
common::{
borrow::{BorrowedValue, BorrowedValueMut},
lock::{MapImmutable, PyMutex, PyMutexGuard},
rc::PyRc,
},
object::PyObjectPayload,
sliceable::SequenceIndexOp,
};
use alloc::borrow::Cow;
use bitflags::bitflags;
use core::{fmt::Debug, ops::Range};
use crossbeam_utils::atomic::AtomicCell;
use itertools::Itertools;
bitflags! {
/// Capabilities a consumer asks a buffer exporter for, the `flags` argument of
/// `bf_getbuffer` and of `__buffer__` (`PyBUF_*`).
///
/// The composite requests are supersets of the simpler ones, so
/// [`contains`](Self::contains) answers the `REQ_*` questions an exporter asks:
/// `flags.contains(BufferFlags::C_CONTIGUOUS)` is `REQ_C_CONTIGUOUS(flags)`.
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub struct BufferFlags: u32 {
const WRITABLE = 0x0001;
const FORMAT = 0x0004;
const ND = 0x0008;
const STRIDES = 0x0010 | Self::ND.bits();
const C_CONTIGUOUS = 0x0020 | Self::STRIDES.bits();
const F_CONTIGUOUS = 0x0040 | Self::STRIDES.bits();
const ANY_CONTIGUOUS = 0x0080 | Self::STRIDES.bits();
const INDIRECT = 0x0100 | Self::STRIDES.bits();
}
}
impl BufferFlags {
/// `PyBUF_SIMPLE`: a plain read-only block of bytes.
pub const SIMPLE: Self = Self::empty();
/// `PyBUF_CONTIG`
pub const CONTIG: Self = Self::ND.union(Self::WRITABLE);
/// `PyBUF_CONTIG_RO`
pub const CONTIG_RO: Self = Self::ND;
/// `PyBUF_STRIDED`
pub const STRIDED: Self = Self::STRIDES.union(Self::WRITABLE);
/// `PyBUF_STRIDED_RO`
pub const STRIDED_RO: Self = Self::STRIDES;
/// `PyBUF_RECORDS`
pub const RECORDS: Self = Self::STRIDED.union(Self::FORMAT);
/// `PyBUF_RECORDS_RO`
pub const RECORDS_RO: Self = Self::STRIDED_RO.union(Self::FORMAT);
/// `PyBUF_FULL`: everything an exporter can describe, writable.
pub const FULL: Self = Self::INDIRECT.union(Self::WRITABLE).union(Self::FORMAT);
/// `PyBUF_FULL_RO`: everything an exporter can describe, read-only.
pub const FULL_RO: Self = Self::INDIRECT.union(Self::FORMAT);
/// `PyBUF_READ`. Belongs to `PyMemoryView_FromMemory`, not to `bf_getbuffer`.
const MEMORY_READ: Self = Self::from_bits_retain(0x100);
/// `PyBUF_WRITE`. Belongs to `PyMemoryView_FromMemory`, not to `bf_getbuffer`.
const MEMORY_WRITE: Self = Self::from_bits_retain(0x200);
/// Whether this request is really a `PyMemoryView_FromMemory` access mode,
/// which no exporter can serve.
#[must_use]
pub const fn is_memory_access_mode(self) -> bool {
self.bits() == Self::MEMORY_READ.bits() || self.bits() == Self::MEMORY_WRITE.bits()
}
/// Whether the consumer demands a writable buffer.
#[must_use]
pub const fn is_writable(self) -> bool {
self.intersects(Self::WRITABLE)
}
/// The argument checks `PyBuffer_FillInfo` performs, for exporters that hand
/// out a flat block of bytes.
pub fn fill_info_check(self, readonly: bool, vm: &VirtualMachine) -> PyResult<()> {
if self == Self::SIMPLE {
return Ok(());
}
if self.is_memory_access_mode() {
return Err(vm.new_system_error("bad argument to internal function"));
}
self.check_writable(readonly, "Object is not writable.", vm)
}
/// Reject a writable request against a read-only export.
pub fn check_writable(
self,
readonly: bool,
message: &str,
vm: &VirtualMachine,
) -> PyResult<()> {
if self.is_writable() && readonly {
return Err(vm.new_buffer_error(message.to_owned()));
}
Ok(())
}
}
pub struct BufferMethods {
pub obj_bytes: fn(&PyBuffer) -> BorrowedValue<'_, [u8]>,
pub obj_bytes_mut: fn(&PyBuffer) -> BorrowedValueMut<'_, [u8]>,
@@ -32,13 +119,46 @@ impl Debug for BufferMethods {
}
}
#[derive(Debug, Clone, Traverse)]
/// One acquisition from an exporter: the state a single `bf_getbuffer` set up,
/// shared by every handle taken from it. _PyManagedBufferObject
#[derive(Debug)]
struct BufferExport {
/// Handles and raw shares that have not been given up yet. mbuf->exports
shares: AtomicCell<usize>,
/// Whether the exporter's release has already run.
/// _Py_MANAGED_BUFFER_RELEASED
released: AtomicCell<bool>,
}
#[derive(Debug, Traverse)]
pub struct PyBuffer {
pub obj: PyObjectRef,
#[pytraverse(skip)]
pub desc: BufferDescriptor,
#[pytraverse(skip)]
methods: &'static BufferMethods,
#[pytraverse(skip)]
export: PyRc<BufferExport>,
/// Whether this handle still holds its share of `export`.
#[pytraverse(skip)]
owns_share: AtomicCell<bool>,
}
/// Cloning takes another share of the same acquisition rather than asking the
/// exporter for a new one, and the exporter's release waits for the last share.
/// mbuf_add_view
impl Clone for PyBuffer {
fn clone(&self) -> Self {
debug_assert!(!self.export.released.load());
self.export.shares.fetch_add(1);
Self {
obj: self.obj.clone(),
desc: self.desc.clone(),
methods: self.methods,
export: self.export.clone(),
owns_share: AtomicCell::new(true),
}
}
}
impl PyBuffer {
@@ -47,8 +167,17 @@ impl PyBuffer {
#[cfg(debug_assertions)]
let desc = desc.validate();
let zelf = Self { obj, desc, methods };
zelf.retain();
let zelf = Self {
obj,
desc,
methods,
export: PyRc::new(BufferExport {
shares: AtomicCell::new(1),
released: AtomicCell::new(false),
}),
owns_share: AtomicCell::new(true),
};
(zelf.methods.retain)(&zelf);
zelf
}
@@ -78,14 +207,16 @@ impl PyBuffer {
/// assume the buffer is contiguous
#[must_use]
pub unsafe fn contiguous_unchecked(&self) -> BorrowedValue<'_, [u8]> {
self.obj_bytes()
let range = self.desc.contiguous_range();
BorrowedValue::map(self.obj_bytes(), |x| &x[range])
}
/// # Safety
/// assume the buffer is contiguous and writable
#[must_use]
pub unsafe fn contiguous_mut_unchecked(&self) -> BorrowedValueMut<'_, [u8]> {
self.obj_bytes_mut()
let range = self.desc.contiguous_range();
BorrowedValueMut::map(self.obj_bytes_mut(), |x| &mut x[range])
}
pub fn append_to(&self, buf: &mut Vec<u8>) {
@@ -113,6 +244,18 @@ impl PyBuffer {
f(v)
}
/// A copy of these bytes in C order, keeping shape and format. The copy
/// borrows nothing from the exporter, so it can be read while the exporter is
/// borrowed for writing.
#[must_use]
pub fn to_contiguous(&self, vm: &VirtualMachine) -> Self {
let mut data = vec![];
self.append_to(&mut data);
VecBuffer::from(data)
.into_ref(&vm.ctx)
.into_pybuffer_with_descriptor(self.desc.contiguous())
}
#[must_use]
pub fn obj_as<T: PyObjectPayload>(&self) -> &Py<T> {
unsafe { self.obj.downcast_unchecked_ref() }
@@ -128,31 +271,87 @@ impl PyBuffer {
(self.methods.obj_bytes_mut)(self)
}
/// Give up this handle's share of the acquisition. PyBuffer_Release
///
/// Idempotent: a handle that has already been released owns nothing, so
/// dropping it afterwards does nothing, like a `Py_buffer` whose `obj` was
/// cleared.
///
/// This can run arbitrary Python through `__release_buffer__`, so no borrow
/// of the exporter may be held while a buffer is released or dropped.
pub fn release(&self) {
if self.owns_share.swap(false) {
self.drop_share();
}
}
/// Take a share of this acquisition that no handle owns. An exporter that
/// forwards a consumer's export onto a buffer it holds itself keeps the
/// acquisition alive this way. memory_getbuf
pub(crate) fn retain_share(&self) {
self.export.shares.fetch_add(1);
}
/// Give back a share taken by [`Self::retain_share`]. memory_releasebuf
pub(crate) fn release_share(&self) {
self.drop_share();
}
fn drop_share(&self) {
if self.export.shares.fetch_sub(1) == 1 {
self.finalize();
}
}
/// The exporter learns its export is gone, once per acquisition. mbuf_release
fn finalize(&self) {
// Latched before the hook runs, so a release re-entered from Python is
// inert.
if self.export.released.swap(true) {
return;
}
// slot_bf_releasebuffer: a Python-level `__release_buffer__` runs first,
// then the exporter's own release so export counts stay balanced.
if self.obj.class().slots.python_release_buffer.load() {
crate::builtins::memory::release_buffer_call_python(self);
}
(self.methods.release)(self)
}
pub fn retain(&self) {
(self.methods.retain)(self)
/// Undo an acquisition the exporter had already handed out but that could not
/// be served, without telling Python: `bf_releasebuffer` does not run when
/// `bf_getbuffer` fails.
pub(crate) fn abort_acquisition(self) {
debug_assert_eq!(self.export.shares.load(), 1);
self.owns_share.store(false);
self.export.released.store(true);
(self.methods.release)(&self);
}
// drop PyBuffer without calling release
// after this function, the owner should use forget()
// or wrap PyBuffer in the ManuallyDrop to prevent drop()
pub(crate) unsafe fn drop_without_release(&mut self) {
// SAFETY: requirements forwarded from caller
unsafe {
core::ptr::drop_in_place(&mut self.obj);
core::ptr::drop_in_place(&mut self.desc);
/// A copy that owns no share: it reads the same memory, but releasing it is
/// inert and it never finalizes the acquisition. A `Py_buffer` whose `obj` is
/// NULL.
#[must_use]
pub fn detached(&self) -> Self {
Self {
obj: self.obj.clone(),
desc: self.desc.clone(),
methods: self.methods,
export: self.export.clone(),
owns_share: AtomicCell::new(false),
}
}
}
impl<'a> TryFromBorrowedObject<'a> for PyBuffer {
fn try_from_borrowed_object(vm: &VirtualMachine, obj: &'a PyObject) -> PyResult<Self> {
impl PyBuffer {
/// Acquire a buffer from `obj`. PyObject_GetBuffer
pub fn from_object(vm: &VirtualMachine, obj: &PyObject, flags: BufferFlags) -> PyResult<Self> {
if flags.is_memory_access_mode() {
return Err(vm.new_system_error("bad argument to internal function"));
}
let cls = obj.class();
if let Some(f) = cls.slots.as_buffer {
return f(obj, vm);
if let Some(f) = cls.slots.as_buffer.load() {
return f(obj, flags, vm);
}
Err(vm.new_type_error(format!(
"a bytes-like object is required, not '{}'",
@@ -161,6 +360,26 @@ impl<'a> TryFromBorrowedObject<'a> for PyBuffer {
}
}
impl PyObject {
/// Whether this object's type exports the buffer protocol. PyObject_CheckBuffer
///
/// A consumer that falls back to something else for non-buffer objects asks
/// this instead of attempting an acquisition, so that an error raised by
/// `__buffer__` is not mistaken for "not a buffer".
#[must_use]
pub fn check_buffer(&self) -> bool {
self.class().slots.as_buffer.load().is_some()
}
}
/// The request a conversion makes when the consumer has no say in it: describe
/// the export as fully as possible, read-only.
impl<'a> TryFromBorrowedObject<'a> for PyBuffer {
fn try_from_borrowed_object(vm: &VirtualMachine, obj: &'a PyObject) -> PyResult<Self> {
Self::from_object(vm, obj, BufferFlags::FULL_RO)
}
}
impl Drop for PyBuffer {
fn drop(&mut self) {
self.release();
@@ -172,10 +391,20 @@ pub struct BufferDescriptor {
/// product(shape) * itemsize
/// bytes length, but not the length for obj_bytes() even is contiguous
pub len: usize,
/// Byte position of the element at index `[0, .., 0]` within
/// [`PyBuffer::obj_bytes`], which always yields the exporter's whole memory.
/// `Py_buffer.buf`
///
/// A view that walks backwards begins at the far end of its data, so this is
/// where addressing starts rather than a lower bound. A view with no elements
/// addresses nothing and may sit outside the exporter altogether, which is why
/// this is signed.
pub offset: isize,
pub readonly: bool,
pub itemsize: usize,
pub format: Cow<'static, str>,
/// (shape, stride, suboffset) for each dimension
/// (shape, stride, suboffset) for each dimension. A non-zero suboffset means
/// the dimension is reached through a pointer; slicing never introduces one.
pub dim_desc: Vec<(usize, isize, isize)>,
// TODO: flags
}
@@ -185,6 +414,7 @@ impl BufferDescriptor {
pub fn simple(bytes_len: usize, readonly: bool) -> Self {
Self {
len: bytes_len,
offset: 0,
readonly,
itemsize: 1,
format: Cow::Borrowed("B"),
@@ -201,6 +431,7 @@ impl BufferDescriptor {
) -> Self {
Self {
len: bytes_len,
offset: 0,
readonly,
itemsize,
format,
@@ -208,9 +439,48 @@ impl BufferDescriptor {
}
}
/// The descriptor an exporter hands to a consumer that asked for `flags`:
/// everything the request did not ask for is dropped.
///
/// A `Py_buffer` drops a field by setting it to NULL and leaves the consumer to
/// reconstruct it. A descriptor has no NULL, so a dropped field is filled in
/// here with what that reconstruction would produce: `"B"` for a format, C-order
/// strides for strides, and a single dimension of `len / itemsize` items for a
/// shape. `itemsize` is never touched, so `calcsize(format)` and `itemsize` can
/// disagree on a projected descriptor — the format governs an element's width,
/// the item size governs the step — and `product(shape) * itemsize == len`
/// continues to hold.
#[must_use]
pub fn projected(&self, flags: BufferFlags) -> Self {
let mut desc = self.clone();
if !flags.contains(BufferFlags::FORMAT) {
desc.format = Cow::Borrowed("B");
}
if !flags.contains(BufferFlags::ND) {
// A request this flat is refused unless the layout is C-contiguous, so
// one dimension addresses the same bytes.
let shape = desc.len.checked_div(desc.itemsize).unwrap_or(0);
desc.dim_desc = vec![(shape, desc.itemsize as isize, 0)];
} else if !flags.contains(BufferFlags::STRIDES) {
// Shape survives but strides do not, which means C order.
let mut stride = desc.itemsize as isize;
for (shape, dim_stride, suboffset) in desc.dim_desc.iter_mut().rev() {
*dim_stride = stride;
*suboffset = 0;
stride *= *shape as isize;
}
}
desc
}
#[cfg(debug_assertions)]
#[must_use]
pub fn validate(self) -> Self {
// Only a view with nothing to address is allowed to start outside the
// exporter.
if self.len != 0 {
debug_assert!(self.offset >= 0);
}
// ndim=0 is valid for scalar types (e.g., ctypes Structure)
if self.ndim() == 0 {
// Empty structures (len=0) can have itemsize=0
@@ -239,6 +509,7 @@ impl BufferDescriptor {
self.dim_desc.len()
}
/// Whether the elements are laid out in row-major order. _IsCContiguous
#[must_use]
pub fn is_contiguous(&self) -> bool {
if self.len == 0 {
@@ -254,11 +525,76 @@ impl BufferDescriptor {
true
}
/// Whether the elements are laid out in column-major order. A view whose
/// dimensions are all but one of length 1 is laid out both ways at once.
/// _IsFortranContiguous
#[must_use]
pub fn is_fortran_contiguous(&self) -> bool {
if self.len == 0 {
return true;
}
let mut sd = self.itemsize;
for (shape, stride, _) in self.dim_desc.iter().copied() {
if shape > 1 && stride != sd as isize {
return false;
}
sd *= shape;
}
true
}
/// The byte range this view occupies in [`PyBuffer::obj_bytes`], for a
/// contiguous view.
///
/// A view with no bytes maps to the empty range at zero: its offset is
/// wherever slicing left it and need not be a position that exists.
#[must_use]
pub fn contiguous_range(&self) -> Range<usize> {
if self.len == 0 {
return 0..0;
}
debug_assert!(self.offset >= 0);
let start = self.offset as usize;
start..start + self.len
}
/// The same shape, format and item size, laid out in C order from byte zero.
#[must_use]
pub fn contiguous(&self) -> Self {
let itemsize = self.itemsize;
let mut dim_desc = self.dim_desc.clone();
if let Some((_, stride, suboffset)) = dim_desc.last_mut() {
*stride = itemsize as isize;
*suboffset = 0;
}
for i in (1..dim_desc.len()).rev() {
dim_desc[i - 1].1 = dim_desc[i].1 * dim_desc[i].0 as isize;
dim_desc[i - 1].2 = 0;
}
Self {
len: self.len,
offset: 0,
readonly: self.readonly,
itemsize: self.itemsize,
format: self.format.clone(),
dim_desc,
}
}
/// Whether any dimension is reached through a pointer rather than by
/// stepping, the layout `PyBUF_INDIRECT` describes.
#[must_use]
pub fn has_suboffsets(&self) -> bool {
self.dim_desc
.iter()
.any(|(_, _, suboffset)| *suboffset != 0)
}
/// this function do not check the bound
/// panic if indices.len() != ndim
#[must_use]
pub fn fast_position(&self, indices: &[usize]) -> isize {
let mut pos = 0;
let mut pos = self.offset;
for (i, (_, stride, suboffset)) in indices
.iter()
.copied()
@@ -271,7 +607,7 @@ impl BufferDescriptor {
/// panic if indices.len() != ndim
pub fn position(&self, indices: &[isize], vm: &VirtualMachine) -> PyResult<isize> {
let mut pos = 0;
let mut pos = self.offset;
for (i, (shape, stride, suboffset)) in indices
.iter()
.copied()
@@ -289,14 +625,19 @@ impl BufferDescriptor {
where
F: FnMut(Range<isize>),
{
// A view with no bytes reaches nothing, and its offset need not be a
// position that exists, so it yields no segment at all.
if self.len == 0 {
return;
}
if self.ndim() == 0 {
f(0..self.itemsize as isize);
f(self.offset..self.offset + self.itemsize as isize);
return;
}
if try_contiguous && self.is_last_dim_contiguous() {
self._for_each_segment::<_, true>(0, 0, &mut f);
self._for_each_segment::<_, true>(self.offset, 0, &mut f);
} else {
self._for_each_segment::<_, false>(0, 0, &mut f);
self._for_each_segment::<_, false>(self.offset, 0, &mut f);
}
}
@@ -328,14 +669,24 @@ impl BufferDescriptor {
where
F: FnMut(Range<isize>, Range<isize>) -> bool,
{
if self.ndim() == 0 {
f(0..self.itemsize as isize, 0..other.itemsize as isize);
if self.len == 0 {
return;
}
if try_contiguous && self.is_last_dim_contiguous() {
self._zip_eq::<_, true>(other, 0, 0, 0, &mut f);
if self.ndim() == 0 {
f(
self.offset..self.offset + self.itemsize as isize,
other.offset..other.offset + other.itemsize as isize,
);
return;
}
// last_dim_is_contiguous: the whole-run path walks both sides at once, so
// both have to be laid out that way.
let run_at_once =
try_contiguous && self.is_last_dim_contiguous() && other.is_last_dim_contiguous();
if run_at_once {
self._zip_eq::<_, true>(other, self.offset, other.offset, 0, &mut f);
} else {
self._zip_eq::<_, false>(other, 0, 0, 0, &mut f);
self._zip_eq::<_, false>(other, self.offset, other.offset, 0, &mut f);
}
}
+3 -1
View File
@@ -6,7 +6,9 @@ mod number;
mod object;
mod sequence;
pub use buffer::{BufferDescriptor, BufferMethods, BufferResizeGuard, PyBuffer, VecBuffer};
pub use buffer::{
BufferDescriptor, BufferFlags, BufferMethods, BufferResizeGuard, PyBuffer, VecBuffer,
};
pub use callable::PyCallable;
pub(crate) use callable::TraceEvent;
pub use iter::{PyIter, PyIterIter, PyIterReturn};
+44
View File
@@ -419,6 +419,50 @@ impl SaturatedSlice {
(range, self.step, slice_len)
}
// PySlice_AdjustIndices, keeping the adjusted start rather than a range.
/// The index the slice begins at, clamped into `0..=len` for a positive step
/// and into `-1..=len-1` for a negative one, together with its length.
///
/// Unlike [`Self::adjust_indices`] this stays meaningful for an empty slice,
/// where it is still the position a strided view moves to.
#[must_use]
pub fn adjust_indices_start(&self, len: usize) -> (isize, usize) {
let len = len as isize;
let clamp = |i: isize| {
if i < 0 {
let i = i.saturating_add(len);
if i < 0 {
if self.step.is_negative() { -1 } else { 0 }
} else {
i
}
} else if i >= len {
if self.step.is_negative() {
len - 1
} else {
len
}
} else {
i
}
};
let start = clamp(self.start);
let stop = clamp(self.stop);
let step = self.step.unsigned_abs();
let slice_len = if self.step.is_negative() {
if stop < start {
(start - stop - 1) as usize / step + 1
} else {
0
}
} else if start < stop {
(stop - start - 1) as usize / step + 1
} else {
0
};
(start, slice_len)
}
#[must_use]
pub fn iter(&self, len: usize) -> SaturatedSliceIter {
SaturatedSliceIter::new(self, len)
+1
View File
@@ -1061,6 +1061,7 @@ impl AsBuffer for PyCArray {
dim_desc.reverse();
BufferDescriptor {
offset: 0,
len: buffer_len,
readonly: false,
itemsize,
+3 -1
View File
@@ -624,7 +624,9 @@ impl PyCData {
// Get buffer pointer - the memory is owned by source
let ptr = {
let bytes = buffer.obj_bytes();
// Contiguity is checked above, so this is the view's own bytes rather
// than the whole exporter's.
let bytes = unsafe { buffer.contiguous_unchecked() };
bytes.as_ptr().wrapping_add(offset)
};
+1
View File
@@ -1655,6 +1655,7 @@ impl AsBuffer for PyCFuncPtr {
(Cow::Borrowed(pointer_format()), pointer_size())
};
let desc = BufferDescriptor {
offset: 0,
len: itemsize,
readonly: false,
itemsize,
+3 -1
View File
@@ -712,7 +712,8 @@ impl PyCPointer {
}
// Try bytes
if let Ok(bytes) = value.try_bytes_like(vm, |b| b.to_vec()) {
if value.check_buffer() {
let bytes = value.try_bytes_like(vm, |b| b.to_vec())?;
rustpython_host_env::ctypes::write_value_to_address(
addr,
size,
@@ -776,6 +777,7 @@ impl AsBuffer for PyCPointer {
let itemsize = stg_info.size;
// Pointer types are scalars with ndim=0, shape=()
let desc = BufferDescriptor {
offset: 0,
len: itemsize,
readonly: false,
itemsize,
+1
View File
@@ -1283,6 +1283,7 @@ impl AsBuffer for PyCSimple {
let itemsize = stg_info.size;
// Simple types are scalars with ndim=0, shape=()
let desc = BufferDescriptor {
offset: 0,
len: itemsize,
readonly: false,
itemsize,
@@ -822,6 +822,7 @@ impl AsBuffer for PyCStructure {
let buf = PyBuffer::new(
zelf.to_owned().into(),
BufferDescriptor {
offset: 0,
len: buffer_len,
readonly: false,
itemsize: buffer_len,
+1
View File
@@ -685,6 +685,7 @@ impl AsBuffer for PyCUnion {
let buf = PyBuffer::new(
zelf.to_owned().into(),
BufferDescriptor {
offset: 0,
len: buffer_len,
readonly: false,
itemsize: buffer_len,
+8 -3
View File
@@ -178,7 +178,6 @@ mod _imp {
use crate::{
PyObjectRef, PyPayload, PyRef, PyResult, VirtualMachine,
builtins::{PyBytesRef, PyCode, PyMemoryView, PyModule, PyStrRef, PyUtf8StrRef},
convert::TryFromBorrowedObject,
function::OptionalArg,
import, version,
};
@@ -270,8 +269,14 @@ mod _imp {
name.clone().into_wtf8(),
)
};
// A non-buffer is a TypeError, not invalid frozen data.
crate::protocol::PyBuffer::try_from_borrowed_object(vm, &data)?;
// A non-buffer is a TypeError, not invalid frozen data. The request
// is the one marshal.loads() makes, so that what passes here is
// exactly what it accepts.
crate::protocol::PyBuffer::from_object(
vm,
&data,
crate::protocol::BufferFlags::SIMPLE,
)?;
// The data is a marshalled code object: a whole marshal value, which
// deserialize_code() does not read — it takes the code body alone,
// without the type byte the writer puts in front of it.
+7 -3
View File
@@ -135,8 +135,8 @@ mod _io {
convert::ToPyObject,
exceptions::nul_char_error,
function::{
ArgBytesLike, ArgIterable, ArgMemoryBuffer, ArgSize, Either, FsPath, FuncArgs,
IntoFuncArgs, OptionalArg, OptionalOption, PySetterValue,
ArgBytesLike, ArgContiguousBytesLike, ArgIterable, ArgMemoryBuffer, ArgSize, Either,
FsPath, FuncArgs, IntoFuncArgs, OptionalArg, OptionalOption, PySetterValue,
},
protocol::{
BufferDescriptor, BufferMethods, BufferResizeGuard, PyBuffer, PyIterReturn, VecBuffer,
@@ -4785,8 +4785,12 @@ mod _io {
}
#[pymethod]
fn write(&self, data: ArgBytesLike, vm: &VirtualMachine) -> PyResult<u64> {
fn write(&self, data: ArgContiguousBytesLike, vm: &VirtualMachine) -> PyResult<u64> {
let mut buffer = self.try_resizable(vm)?;
// Acquiring the buffer can run `__buffer__`, which may have closed us.
if self.closed.load() {
return Err(io_closed_error(vm));
}
data.with_ref(|b| buffer.write(b))
.ok_or_else(|| vm.new_type_error("Error Writing Bytes"))
}
+4 -4
View File
@@ -3,8 +3,8 @@ pub(crate) use _sre::module_def;
#[pymodule]
mod _sre {
use crate::{
Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, TryFromBorrowedObject,
TryFromObject, VirtualMachine, atomic_func,
Py, PyObject, PyObjectRef, PyPayload, PyRef, PyResult, TryFromObject, VirtualMachine,
atomic_func,
builtins::{
PyCallableIterator, PyDictRef, PyGenericAlias, PyInt, PyList, PyListRef, PyStr,
PyStrRef, PyTuple, PyTupleRef, PyTypeRef,
@@ -13,7 +13,7 @@ mod _sre {
common::{ascii, hash::PyHash},
convert::ToPyObject,
function::{ArgCallable, OptionalArg, PosArgs, PyComparisonValue},
protocol::{PyBuffer, PyCallable, PyMappingMethods},
protocol::{BufferFlags, PyBuffer, PyCallable, PyMappingMethods},
stdlib::sys,
types::{AsMapping, Comparable, Hashable, Representable},
};
@@ -386,7 +386,7 @@ mod _sre {
where
F: FnOnce(&[u8]) -> PyResult<R>,
{
PyBuffer::try_from_borrowed_object(vm, string)?.contiguous_or_collect(f)
PyBuffer::from_object(vm, string, BufferFlags::SIMPLE)?.contiguous_or_collect(f)
}
#[pymethod(name = "match")]
+24 -16
View File
@@ -21,9 +21,9 @@ mod builtins {
bytecode,
common::hash::PyHash,
function::{
ArgBytesLike, ArgCallable, ArgIndex, ArgIntoBool, ArgIterable, ArgMapping,
ArgPrimitiveIndex, ArgStrOrBytesLike, Either, FsPath, FuncArgs, KwArgs, OptionalArg,
OptionalOption, PosArgs,
ArgCallable, ArgIndex, ArgIntoBool, ArgIterable, ArgMapping, ArgPrimitiveIndex,
ArgStrOrBytesLike, Either, FsPath, FuncArgs, KwArgs, OptionalArg, OptionalOption,
PosArgs,
},
protocol::{PyIter, PyIterReturn},
py_io,
@@ -997,18 +997,10 @@ mod builtins {
}
#[pyfunction]
fn ord(string: Either<ArgBytesLike, PyStrRef>, vm: &VirtualMachine) -> PyResult<u32> {
match string {
Either::A(bytes) => bytes.with_ref(|bytes| {
let bytes_len = bytes.len();
if bytes_len != 1 {
return Err(vm.new_type_error(format!(
"ord() expected a character, but string of length {bytes_len} found"
)));
}
Ok(u32::from(bytes[0]))
}),
Either::B(string) => match string.as_wtf8().code_points().exactly_one() {
// builtin_ord
fn ord(c: PyObjectRef, vm: &VirtualMachine) -> PyResult<u32> {
let bytes = if let Some(string) = c.downcast_ref::<PyStr>() {
return match string.as_wtf8().code_points().exactly_one() {
Ok(character) => Ok(character.to_u32()),
Err(_) => {
let string_len = string.char_len();
@@ -1016,8 +1008,24 @@ mod builtins {
"ord() expected a character, but string of length {string_len} found"
)))
}
},
};
} else if let Some(bytes) = c.downcast_ref::<PyBytes>() {
bytes.as_bytes().to_vec()
} else if let Some(bytearray) = c.downcast_ref::<PyByteArray>() {
bytearray.borrow_buf().to_vec()
} else {
return Err(vm.new_type_error(format!(
"ord() expected string of length 1, but {} found",
c.class().name()
)));
};
let bytes_len = bytes.len();
if bytes_len != 1 {
return Err(vm.new_type_error(format!(
"ord() expected a character, but string of length {bytes_len} found"
)));
}
Ok(u32::from(bytes[0]))
}
#[derive(FromArgs)]
+4 -9
View File
@@ -16,7 +16,6 @@ mod decl {
convert::ToPyObject,
function::{ArgBytesLike, OptionalArg},
object::{AsObject, PyPayload},
protocol::PyBuffer,
};
use core::cell::RefCell;
use malachite_bigint::BigInt;
@@ -651,20 +650,16 @@ mod decl {
#[derive(FromArgs)]
struct LoadsArgs {
#[pyarg(any)]
data: PyBuffer,
// marshal_loads_impl takes `bytes: Py_buffer`, a y* argument.
data: ArgBytesLike,
#[pyarg(named, default = true)]
allow_code: bool,
}
#[pyfunction]
fn loads(args: LoadsArgs, vm: &VirtualMachine) -> PyResult<PyObjectRef> {
let LoadsArgs {
data: pybuffer,
allow_code,
} = args;
let buf = pybuffer.as_contiguous().ok_or_else(|| {
vm.new_buffer_error("Buffer provided to marshal.loads() is not contiguous")
})?;
let LoadsArgs { data, allow_code } = args;
let buf = data.borrow_buf();
let result = deserialize_value(&mut &buf[..], vm)?;
if !allow_code {
+3 -3
View File
@@ -6,10 +6,10 @@ pub(crate) use winsound::module_def;
#[pymodule]
mod winsound {
use crate::builtins::{PyBaseExceptionRef, PyBytes, PyStr};
use crate::convert::{IntoPyException, ToPyException, TryFromBorrowedObject};
use crate::convert::{IntoPyException, ToPyException};
use crate::exceptions;
use crate::host_env::windows::ToWideString;
use crate::protocol::PyBuffer;
use crate::protocol::{BufferFlags, PyBuffer};
use crate::{AsObject, PyObjectRef, PyResult, VirtualMachine};
use rustpython_host_env::winsound::{PlaySoundError, PlaySoundSource, play_sound};
@@ -90,7 +90,7 @@ mod winsound {
}
if flags & SND_MEMORY != 0 {
let buffer = PyBuffer::try_from_borrowed_object(vm, &sound)?;
let buffer = PyBuffer::from_object(vm, &sound, BufferFlags::SIMPLE)?;
let buf = buffer
.as_contiguous()
.ok_or_else(|| vm.new_type_error("a bytes-like object is required, not 'str'"))?;
+88 -6
View File
@@ -9,7 +9,7 @@ use crate::{
convert::ToPyObject,
function::{Either, FromArgs, FuncArgs, PyComparisonValue, PyMethodDef, PySetterValue},
protocol::{
PyBuffer, PyIterReturn, PyMapping, PyMappingMethods, PyMappingSlots, PyNumber,
BufferFlags, PyBuffer, PyIterReturn, PyMapping, PyMappingMethods, PyMappingSlots, PyNumber,
PyNumberMethods, PyNumberSlots, PySequence, PySequenceMethods, PySequenceSlots,
},
types::slot_defs::{SlotAccessor, find_slot_defs_by_name},
@@ -149,7 +149,12 @@ pub struct PyTypeSlots {
pub setattro: AtomicCell<Option<SetattroFunc>>,
// Functions to access object as input/output buffer
pub as_buffer: Option<AsBufferFunc>,
pub as_buffer: AtomicCell<Option<AsBufferFunc>>,
/// bf_releasebuffer: releasing an export of this type is observable, so the
/// type exposes `__release_buffer__`.
pub has_release_buffer: AtomicCell<bool>,
/// True when a Python-level `__release_buffer__` must be invoked on release.
pub python_release_buffer: AtomicCell<bool>,
// Assigned meaning in release 2.1
// rich comparisons
@@ -296,7 +301,8 @@ pub(crate) type StringifyFunc = fn(&PyObject, &VirtualMachine) -> PyResult<PyRef
pub(crate) type GetattroFunc = fn(&PyObject, &Py<PyStr>, &VirtualMachine) -> PyResult;
pub(crate) type SetattroFunc =
fn(&PyObject, &Py<PyStr>, PySetterValue, &VirtualMachine) -> PyResult<()>;
pub(crate) type AsBufferFunc = fn(&PyObject, &VirtualMachine) -> PyResult<PyBuffer>;
/// bf_getbuffer
pub(crate) type AsBufferFunc = fn(&PyObject, BufferFlags, &VirtualMachine) -> PyResult<PyBuffer>;
pub(crate) type RichCompareFunc = fn(
&PyObject,
&PyObject,
@@ -329,6 +335,15 @@ pub(crate) type MapSubscriptFunc = fn(PyMapping<'_>, &PyObject, &VirtualMachine)
pub(crate) type MapAssSubscriptFunc =
fn(PyMapping<'_>, &PyObject, Option<PyObjectRef>, &VirtualMachine) -> PyResult<()>;
// slot_bf_getbuffer
pub(crate) fn python_as_buffer(
obj: &PyObject,
flags: BufferFlags,
vm: &VirtualMachine,
) -> PyResult<PyBuffer> {
crate::builtins::memory::buffer_from_python_getbuffer(obj, flags, vm)
}
// slot_sq_length
pub(crate) fn len_wrapper(obj: &PyObject, vm: &VirtualMachine) -> PyResult<usize> {
let ret = vm.call_special_method(obj, identifier!(vm, __len__), ())?;
@@ -1579,6 +1594,58 @@ impl PyType {
}
}
// === Buffer protocol ===
SlotAccessor::BfGetBuffer => {
if ADD {
match self.lookup_slot_in_mro(name, ctx, |sf| {
if let SlotFunc::GetBuffer(f) = sf {
Some(*f)
} else {
None
}
}) {
SlotLookupResult::NativeSlot(func) => {
self.slots.as_buffer.store(Some(func));
}
SlotLookupResult::PythonMethod => {
self.slots.as_buffer.store(Some(python_as_buffer));
}
SlotLookupResult::NotFound => {
accessor.inherit_from_mro(self);
}
}
} else {
accessor.inherit_from_mro(self);
}
}
SlotAccessor::BfReleaseBuffer => {
// Which of the two implementations `__release_buffer__` resolves to
// decides whether buffer release has to call back into Python.
if ADD {
match self.lookup_slot_in_mro(name, ctx, |sf| {
if matches!(sf, SlotFunc::ReleaseBuffer) {
Some(())
} else {
None
}
}) {
SlotLookupResult::NativeSlot(()) => {
self.slots.python_release_buffer.store(false);
self.slots.has_release_buffer.store(true);
}
SlotLookupResult::PythonMethod => {
self.slots.python_release_buffer.store(true);
self.slots.has_release_buffer.store(true);
}
SlotLookupResult::NotFound => {
accessor.inherit_from_mro(self);
}
}
} else {
accessor.inherit_from_mro(self);
}
}
// Reserved slots - no-op
_ => {}
}
@@ -2070,14 +2137,29 @@ pub trait SetAttr: PyPayload {
#[pyclass]
pub trait AsBuffer: PyPayload {
// TODO: `flags` parameter
/// bf_releasebuffer: set when releasing an export of this type is observable,
/// i.e. the exporter counts exports. Such types expose `__release_buffer__`.
const RELEASE_BUFFER: bool = false;
#[inline]
#[pyslot]
fn slot_as_buffer(zelf: &PyObject, vm: &VirtualMachine) -> PyResult<PyBuffer> {
fn slot_as_buffer(
zelf: &PyObject,
flags: BufferFlags,
vm: &VirtualMachine,
) -> PyResult<PyBuffer> {
let zelf = zelf
.downcast_ref()
.ok_or_else(|| vm.new_type_error("unexpected payload for as_buffer"))?;
Self::as_buffer(zelf, vm)
let buffer = Self::as_buffer(zelf, vm)?;
if let Err(exc) = flags.check_writable(buffer.desc.readonly, "Object is not writable.", vm)
{
// An acquisition that cannot be served never happened, so the
// exporter's release is undone without running the Python hook.
buffer.abort_acquisition();
return Err(exc);
}
Ok(buffer)
}
fn as_buffer(zelf: &Py<Self>, vm: &VirtualMachine) -> PyResult<PyBuffer>;
+60 -4
View File
@@ -71,7 +71,7 @@ pub struct SlotDef {
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
#[repr(u8)]
pub enum SlotAccessor {
// Buffer protocol (1-2) - Reserved, not used in RustPython
// Buffer protocol (1-2)
BfGetBuffer = 1,
BfReleaseBuffer = 2,
@@ -173,9 +173,7 @@ impl SlotAccessor {
pub fn is_reserved(&self) -> bool {
matches!(
self,
Self::BfGetBuffer
| Self::BfReleaseBuffer
| Self::TpAlloc
Self::TpAlloc
| Self::TpBase
| Self::TpBases
| Self::TpClear
@@ -411,6 +409,10 @@ impl SlotAccessor {
)
}
// Buffer protocol
Self::BfGetBuffer => matches!(slot_func, SlotFunc::GetBuffer(_)),
Self::BfReleaseBuffer => matches!(slot_func, SlotFunc::ReleaseBuffer),
// New and reserved slots
Self::TpNew => false,
_ => false, // Reserved slots
@@ -539,6 +541,18 @@ impl SlotAccessor {
Self::MpSubscript => inherit_mapping!(subscript),
Self::MpAssSubscript => inherit_mapping!(ass_subscript),
// Buffer protocol
Self::BfGetBuffer => {
let inherited = mro.iter().find_map(|cls| cls.slots.as_buffer.load());
typ.slots.as_buffer.store(inherited);
}
Self::BfReleaseBuffer => {
let has_release = mro.iter().any(|cls| cls.slots.has_release_buffer.load());
typ.slots.has_release_buffer.store(has_release);
let py_release = mro.iter().any(|cls| cls.slots.python_release_buffer.load());
typ.slots.python_release_buffer.store(py_release);
}
// Reserved slots - no-op
_ => {}
}
@@ -677,6 +691,25 @@ impl SlotAccessor {
Self::MpSubscript => copy_mapping!(subscript),
Self::MpAssSubscript => copy_mapping!(ass_subscript),
// Buffer protocol
Self::BfGetBuffer => {
if typ.slots.as_buffer.load().is_none()
&& let Some(base_val) = base.slots.as_buffer.load()
{
typ.slots.as_buffer.store(Some(base_val));
}
}
Self::BfReleaseBuffer => {
if !typ.slots.has_release_buffer.load() && base.slots.has_release_buffer.load() {
typ.slots.has_release_buffer.store(true);
}
if !typ.slots.python_release_buffer.load()
&& base.slots.python_release_buffer.load()
{
typ.slots.python_release_buffer.store(true);
}
}
// Reserved slots - no-op
_ => {}
}
@@ -816,6 +849,16 @@ impl SlotAccessor {
.load()
.map(SlotFunc::MapSetSubscript),
// Buffer protocol
Self::BfGetBuffer => slots.as_buffer.load().map(SlotFunc::GetBuffer),
Self::BfReleaseBuffer => {
if slots.has_release_buffer.load() || slots.python_release_buffer.load() {
Some(SlotFunc::ReleaseBuffer)
} else {
None
}
}
// Reserved slots
_ => None,
}
@@ -973,6 +1016,19 @@ pub const SLOT_DEFS_COUNT: usize = SLOT_DEFS.len();
/// All slot definitions
pub static SLOT_DEFS: &[SlotDef] = &[
// Buffer protocol (bf_*)
SlotDef {
name: "__buffer__",
accessor: SlotAccessor::BfGetBuffer,
op: None,
doc: "Return a buffer object that exposes the underlying memory of the object.",
},
SlotDef {
name: "__release_buffer__",
accessor: SlotAccessor::BfReleaseBuffer,
op: None,
doc: "Release the buffer object that exposes the underlying memory of the object.",
},
// Type slots (tp_*)
SlotDef {
name: "__init__",
+2
View File
@@ -103,6 +103,7 @@ declare_const_name! {
__await__,
__bases__,
__bool__,
__buffer__,
__build_class__,
__builtins__,
__bytes__,
@@ -205,6 +206,7 @@ declare_const_name! {
__rdivmod__,
__reduce__,
__reduce_ex__,
__release_buffer__,
__repr__,
__reversed__,
__rfloordiv__,
+424
View File
@@ -90,3 +90,427 @@ def test_delitem():
test_delitem()
def test_empty_view_offset():
# An empty view keeps the offset slicing left it, which can sit outside the
# exporter, and reaches no byte through it.
ba = bytearray(range(17))
assert bytes(memoryview(ba)[::-9][-30::-9]) == b""
assert bytes(memoryview(ba)[-30::-1]) == b""
v = memoryview(ba)[::-9][-30::-9]
assert v.shape == (0,)
assert v.strides == (81,)
assert v.suboffsets == ()
b24 = bytearray(range(24))
assert bytes(memoryview(b24).cast("B", [4, 6])[-30::-1]) == b""
test_empty_view_offset()
def test_exported_suboffsets():
mv = memoryview(bytearray(b"abcdef"))[::-1]
exported = mv.__buffer__(284)
assert exported.suboffsets == ()
assert bytes(exported) == b"fedcba"
assert (
bytes(memoryview(memoryview(bytearray(b"abcdefg"))[::2].__buffer__(284)))
== b"aceg"
)
test_exported_suboffsets()
def test_setitem_slice_strided_source():
src = bytearray(b"abcdef")
dst = bytearray(b"......")
memoryview(dst)[:] = memoryview(src)[::-1]
assert bytes(dst) == b"fedcba"
dst = bytearray(b"...")
memoryview(dst)[:] = memoryview(src)[::2]
assert bytes(dst) == b"ace"
test_setitem_slice_strided_source()
def test_zero_dim_position():
z = memoryview(bytearray(range(8)))[4:5].cast("B", [])
assert z[()] == 4
assert z.tolist() == 4
w = bytearray(range(8))
memoryview(w)[4:5].cast("B", [])[()] = 99
assert w[4] == 99
assert w[0] == 0
test_zero_dim_position()
def test_cast_zero_dim_size():
assert_raises(TypeError, lambda: memoryview(bytearray(range(8))).cast("B", []))
assert memoryview(bytearray(b"a")).cast("B", []).nbytes == 1
test_cast_zero_dim_size()
def test_hash_format():
assert_raises(ValueError, lambda: hash(memoryview(b"abcd").cast("I")))
hash(memoryview(b"abcd").cast("b"))
hash(memoryview(b"abcdef")[::2])
hash(memoryview(b"a").cast("B", []))
test_hash_format()
def test_cast_keeps_exports():
ba = bytearray(b"abc")
mv = memoryview(ba)
cast = mv.cast("B")
mv.release()
assert_raises(BufferError, lambda: ba.clear())
cast.release()
ba.clear()
assert bytes(ba) == b""
test_cast_keeps_exports()
def test_setitem_converts_before_writing():
ba = bytearray(b"abc")
mv = memoryview(ba)
class Idx:
def __index__(self):
return len(bytes(ba))
mv[0] = Idx()
assert bytes(ba) == b"\x03bc"
test_setitem_converts_before_writing()
def test_pep688_exporter_aliasing():
def exporter(view_factory):
class C:
def __buffer__(self, flags):
return view_factory()
def __release_buffer__(self, view):
pass
return C()
ba = bytearray(b"abc")
memoryview(ba)[:] = exporter(lambda: memoryview(ba))
assert bytes(ba) == b"abc"
ba = bytearray(b"abcdef")
memoryview(ba)[0:3] = exporter(lambda: memoryview(ba)[3:6])
assert bytes(ba) == b"defdef"
ba = bytearray(b"abcdef")
memoryview(ba)[3:6] = exporter(lambda: memoryview(ba)[0:3])
assert bytes(ba) == b"abcabc"
ba = bytearray(b"abcdef")
memoryview(ba)[:] = exporter(lambda: memoryview(ba)[::-1])
assert bytes(ba) == b"fedcba"
ba = bytearray(b"abcdef")
memoryview(ba)[::2] = exporter(lambda: memoryview(ba)[0:3])
assert bytes(ba) == b"abbdcf"
ba = bytearray(b"abcdef")
mv = memoryview(exporter(lambda: memoryview(ba)))
mv[:] = exporter(lambda: memoryview(ba))
assert bytes(ba) == b"abcdef"
mv[:] = ba
assert bytes(ba) == b"abcdef"
test_pep688_exporter_aliasing()
def test_release_buffer_waits_for_last_view():
class C(bytearray):
calls = 0
def __release_buffer__(self, view):
type(self).calls += 1
super().__release_buffer__(view)
c = C(b"abcdef")
a = memoryview(c)
b = memoryview(a)
a.release()
assert C.calls == 0
assert b.tobytes() == b"abcdef"
b.release()
assert C.calls == 1
class D:
n = 0
def __init__(self):
self.b = bytearray(b"abcdef")
def __buffer__(self, flags):
return memoryview(self.b)
def __release_buffer__(self, view):
type(self).n += 1
d = D()
m = memoryview(d)
m2 = memoryview(m)
m3 = m.cast("B")
m.release()
m2.release()
assert D.n == 0
m3.release()
assert D.n == 1
# Two acquisitions are two exports, each released on its own.
D.n = 0
d = D()
a1 = memoryview(d)
a2 = memoryview(d)
a1.release()
assert D.n == 1
a2.release()
assert D.n == 2
test_release_buffer_waits_for_last_view()
def test_failed_request_does_not_release():
import inspect
import mmap
class M(mmap.mmap):
calls = 0
def __release_buffer__(self, view):
type(self).calls += 1
super().__release_buffer__(view)
m = M(-1, 10, access=mmap.ACCESS_READ)
assert_raises(BufferError, lambda: m.__buffer__(inspect.BufferFlags.WRITABLE))
assert M.calls == 0
test_failed_request_does_not_release()
def test_request_shapes_exported_descriptor():
import array
a = array.array("I", [1, 2, 3])
assert a.__buffer__(0).format == "B"
assert a.__buffer__(28).format == "I"
m = memoryview(a)
b = m.__buffer__(0)
assert (b.format, b.itemsize, b.ndim, b.shape, b.strides) == ("B", 4, 1, (3,), (4,))
assert m.__buffer__(28).format == "I"
b = a.__buffer__(0)
assert b[0] == 1
assert b.tolist() == [1, 2, 3]
assert len(b.tobytes()) == 12
b[0] = 9
assert a[0] == 9
n = memoryview(bytearray(b"abcdef" * 4)).cast("I", (2, 3))
assert n.__buffer__(0).ndim == 1
assert n.__buffer__(0).shape == (6,)
assert n.__buffer__(8).ndim == 2
assert n.__buffer__(8).format == "B"
test_request_shapes_exported_descriptor()
def test_release_during_index_conversion():
# CHECK_RELEASED_AGAIN: the conversion that produces the value, and the one
# that produced the index, both run Python that can release the view.
ba = bytearray(b"abcdefgh")
mv = memoryview(ba)
class Writer:
def __index__(self):
mv.release()
ba.clear()
return 7
try:
mv[7] = Writer()
raise AssertionError("write into a released view")
except ValueError as e:
assert "released memoryview" in str(e), e
ba = bytearray(b"abcdefgh")
mv = memoryview(ba)
class Reader:
def __index__(self):
mv.release()
ba.clear()
return 7
try:
mv[Reader()]
raise AssertionError("read from a released view")
except ValueError as e:
assert "released memoryview" in str(e), e
# A release that does not resize still forbids the write.
ba = bytearray(b"abcd")
mv = memoryview(ba)
class Quiet:
def __index__(self):
mv.release()
return 65
try:
mv[0] = Quiet()
raise AssertionError("write into a released view")
except ValueError as e:
assert "released memoryview" in str(e), e
assert bytes(ba) == b"abcd"
test_release_during_index_conversion()
def test_cast_rejects_non_native_format():
# get_native_fmtchar
for fmt in ["", "ii", "<i", "z"]:
try:
memoryview(b"").cast(fmt)
raise AssertionError(f"cast accepted {fmt!r}")
except ValueError as e:
assert "native single character format" in str(e), e
assert memoryview(bytes(8)).cast("@i").itemsize == 4
test_cast_rejects_non_native_format()
def test_release_buffer_called_twice():
# wrap_releasebuffer
ba = bytearray(b"abc")
mv = memoryview(ba)
assert ba.__release_buffer__(mv) is None
try:
ba.__release_buffer__(mv)
raise AssertionError("second release accepted")
except ValueError as e:
assert "already been released" in str(e), e
mv = memoryview(bytearray(b"abc"))
try:
bytearray(b"abc").__release_buffer__(mv)
raise AssertionError("release by a foreign object accepted")
except ValueError as e:
assert "not this object" in str(e), e
test_release_buffer_called_twice()
def test_restricted_view_compares():
# memory_richcompare reads another view where it lies rather than acquiring
# it, so the restricted view handed to __release_buffer__ still compares.
seen = []
class K(bytearray):
def __release_buffer__(self, view):
seen.append(memoryview(b"hello") == view)
seen.append(view == memoryview(b"hello"))
memoryview(K(b"hello")).release()
assert seen == [True, True], seen
test_restricted_view_compares()
def test_release_during_slice_assignment():
# copy_single: acquiring the source runs __buffer__, which can release the
# destination view.
ba = bytearray(b"abcd")
mv = memoryview(ba)
class Src:
def __buffer__(self, flags):
mv.release()
return memoryview(b"WXYZ")
try:
mv[:] = Src()
raise AssertionError("wrote through a released view")
except ValueError as e:
assert "released memoryview" in str(e), e
assert bytes(ba) == b"abcd"
# The destination of a slice assignment counts as no export, so the source
# may resize the exporter once it has released the view.
ba = bytearray(b"abcd")
mv = memoryview(ba)
class Shrink:
def __buffer__(self, flags):
mv.release()
ba.clear()
return memoryview(b"WXYZ")
try:
mv[:] = Shrink()
raise AssertionError("wrote through a released view")
except ValueError as e:
assert "released memoryview" in str(e), e
assert bytes(ba) == b""
test_release_during_slice_assignment()
def test_fortran_contiguity():
# A view whose dimensions are all but one of length 1 is laid out both in
# row-major and in column-major order.
mv = memoryview(bytearray(range(8)))
for shape in [(1, 8), (8, 1), (1, 1, 8), (8,)]:
view = mv.cast("B", shape)
assert view.c_contiguous, shape
assert view.f_contiguous, shape
assert view.contiguous, shape
for shape in [(2, 4), (1, 2, 4), (2, 1, 4), (2, 2, 2)]:
view = mv.cast("B", shape)
assert view.c_contiguous, shape
assert not view.f_contiguous, shape
assert view.contiguous, shape
# A view with no elements is laid out both ways whatever its shape.
empty = memoryview(b"").cast("B")
assert empty.c_contiguous and empty.f_contiguous and empty.contiguous
scalar = memoryview(b"a").cast("B", ())
assert scalar.c_contiguous and scalar.f_contiguous and scalar.contiguous
strided = memoryview(bytearray(b"abcdefgh"))[::2]
assert not strided.c_contiguous
assert not strided.f_contiguous
assert not strided.contiguous
test_fortran_contiguity()