Commit Graph
31 Commits
Author SHA1 Message Date
Jeong, YunWonandLee Dogeon 358f6a875a native _json, _pickle, pyexpat, and _elementtree (#8741)
* _json: add native make_encoder (c_make_encoder)

json.encoder fell back to the pure-Python _make_iterencode because _json
had no make_encoder, making json.dumps ~25x slower than CPython. Implement
the C encoder semantics natively: str/None/bool/int/float/list/tuple/dict
ordering, key coercion and skipkeys, sort_keys via the VM's fallible sort
(comparison errors propagate instead of panicking), circular reference
detection, indent handling, allow_nan, default() fallback with recursion
guard, and CPython's add_note annotations. Removes the now-passing
expectedFailure markers in test_json.

Assisted-by: Claude Code:claude-fable-5-1

* pyexpat: add UseForeignDTD, fix silently swallowed handler exceptions

genshi's XML template loader (bm_genshi) called xmlparser.UseForeignDTD(),
which pyexpat.rs didn't implement, raising AttributeError. Add it as an
accepted no-op: the pyexpat module here wraps the pure-Rust xml-rs crate
(not real libexpat), which has no hook for splicing in a foreign DTD.

Root-causing the resulting bug also uncovered why genshi's template
generate()/render() produced empty output instead of erroring: do_parse's
invoke_handler swallowed every exception raised by a Python callback
(handler.call(...).ok()), and genshi's expat callbacks fail immediately
by reading `parser.CurrentLineNumber`/`CurrentColumnNumber`, which
pyexpat.rs never exposed. The AttributeError was silently discarded,
so no start/end/text events ever reached genshi's queue.

Fix both: propagate handler exceptions out of Parse()/ParseFile() (matching
libexpat/CPython semantics), and add CurrentLineNumber/CurrentColumnNumber/
CurrentByteIndex as read-only attributes backed by xml-rs's per-event
TextPosition (byte index is derived from it and is exact for ASCII
documents). This also fixes test_minidom's
testExceptionOnSpacesInXMLNSValue, previously an expected failure because
the ValueError from an invalid xmlns value was swallowed the same way.

PositionTest in test_pyexpat.py checks byte-exact expat semantics (token
*start* position) that xml-rs's *current parse cursor* position can't
reproduce exactly, so that one test is now marked an expected failure.

Assisted-by: Claude Code:claude-fable-5-1

* pyexpat: support multi-chunk Parse()/ParseFile() feeds

bm_xml_etree failed with AssertionError: missing end tags from
TreeBuilder.close(), because xml.etree.ElementTree reads and feeds large
documents to XMLParser.feed() in ~64KB chunks (and xml.sax's
IncrementalParser and ElementTree's XMLPullParser do the same). Each
Parse()/ParseFile() call built a brand-new xml-rs EventReader from only
the bytes passed in that call, discarding all parser state between calls.
When a chunk boundary fell in the middle of an element, the fragment
parsed by itself was invalid/incomplete XML, so xml-rs silently errored
out (matching this module's existing "ignore stricter xml-rs errors"
behavior) and the corresponding start/end events were dropped entirely
-- not just delayed.

xml-rs has no API to resume a parser across separate Read sources, so
instead each call now appends the new chunk to a persistent buffer and
re-parses the buffer as a whole, replaying only the events at or beyond
the count already dispatched in previous calls (dispatched_events). This
also makes position tracking (CurrentLineNumber/Column/ByteIndex) correct
regardless of how the input was chunked. Trade-off: each Parse() call is
now O(size fed so far), so a document delivered in many small chunks is
parsed several times over; acceptable for correctness but noticeably
slower than a true incremental parser for pathological chunk sizes.

This also fixes several previously-xfailed tests that depend on
incremental feeding: test_xml_etree.py's XMLPullParserTest (test_events,
test_feed_while_iterating, test_simple_xml, test_simple_xml_chunk_22,
test_simple_xml_with_ns) and ElementTreeTest.test_parseliteral, plus
test_sax.py's ExpatReaderTest (test_expat_entityresolver_default,
test_expat_incremental, test_expat_incremental_reset) -- their
expectedFailure markers are removed. test_simple_xml_chunk_1/_chunk_5
(1- and 5-byte chunks) still fail and remain xfail: at that granularity
xml-rs's stricter-than-expat tokenizer errors on some intermediate
prefixes that libexpat tolerates.

Assisted-by: Claude Code:claude-fable-5-1

* test: drop stale expectedFailure markers fixed by the pyexpat changes

test_plistlib's four invalid-document tests and test_xmlrpc's
test_loads_unsupported pass now that pyexpat no longer swallows
handler exceptions and keeps parser state across chunked feeds.

Assisted-by: Claude Code:claude-fable-5-1

* pyexpat: drive a single background EventReader for true incremental Parse()

8802451f3 fixed multi-chunk Parse()/ParseFile() feeds by accumulating every
fed byte and re-parsing the whole buffer from scratch on each call, which is
O(total bytes) per chunk -- quadratic in document size. xml-rs's EventReader
has no API to pause/resume across separate Read sources, so instead one
EventReader is now driven to completion on a dedicated background thread fed
through a condvar-backed queue (FeedBuffer/FeedReader); Parse()/ParseFile()
push their chunk, wake the thread, and drain (on the calling thread only)
whatever events that unblocks, stopping once the thread reports it is
blocked waiting for more input. Handlers always run on the calling thread.
A handler exception stops dispatch without tearing down the stream, so the
next Parse() resumes where it left off, mirroring libexpat. CurrentByteIndex
is now computed incrementally per line (with an amortized per-query cursor,
needed since real documents are often a single very long line) instead of
scanning the whole buffer. ParserCreate() never spawns a thread; one is
spawned lazily by the first Parse()/ParseFile() call, and dropping the
parser always stops and joins it.

Fixes test_xml_etree{,_c}'s test_events_comment, previously xfailed because
the old whole-buffer replay lost top-level comment events across feeds.

Measured (M4 Pro, release build):
- Scaling (ElementTree.parse, newline-separated doc): new backend 1/4/16 MB
  = 0.15/0.61/2.35s (linear); CPython 3.14 same file = 0.015/0.055/0.23s.
- pyperformance's bm_xml_etree doc (156KB, effectively one line): old
  backend single etree.parse() = 1.99s, single iterparse() = 3.53s; new
  backend = 0.025s / 0.031s (~80-110x). Full bench_parse/iterparse/generate/
  process (30/10/10/1 reps) now finish in ~1.5s/0.6s/0.4s/0.3s instead of
  timing out past pyperformance's 400s limit.
- 10k ParserCreate() (never fed / partially fed / fully fed and dropped):
  0.32s, no residual native threads, no RSS growth.

test -j4 test_pyexpat test_xml_etree test_xml_etree_c test_sax test_minidom
test_xml_dom_minicompat test_pulldom test_plistlib test_xmlrpc test_html
test_htmlparser test_threading: all pass (1350 tests, 89 skipped).

Assisted-by: Claude Code:claude-fable-5-1

* pyexpat: add a Sync fallback backend for targets without threads

2f58e9531 made pyexpat always drive its EventReader on a background
thread (ParserStream::spawn), panicking via .expect() if the thread
couldn't be spawned. pyexpat is registered unconditionally, including
on wasm32, where std::thread::spawn always fails at runtime, so the
first Parse() call panicked there.

Introduce a Backend enum: Backend::Threaded (the existing ParserStream)
on targets that can spawn threads, and a new Backend::Sync, the
accumulate-and-reparse implementation from 8802451f3, used on wasm32
and as a runtime fallback if spawning a thread ever fails. Both share
LineTracker for identical CurrentLineNumber/CurrentColumnNumber/
CurrentByteIndex and handler/error semantics.

Fixed two correctness bugs while porting the Sync backend forward:
marking the parser permanently "finished" as soon as a from-scratch
reparse happened to reach EndDocument (which can be a false positive
on a not-yet-complete buffer, e.g. right before epilog content
arrives) rather than only on the caller's real isfinal signal; and
counting that same EndDocument towards the dispatched-event index,
which could make a real event land on an index a later, longer reparse
had already skipped past. Verified by force-selecting Backend::Sync
under a test-only env var (removed) and diffing its output against
Backend::Threaded for the xml/sax/etree/plistlib/xmlrpc test suites
and a 2MB document fed in 1KB chunks.

Assisted-by: Claude Code:claude-fable-5-1

* _pickle: native Pickler, Unpickler and PickleBuffer

Adds `crates/stdlib/src/pickle.rs`, a native `_pickle` implementing
Pickler, Unpickler, PickleBuffer, PicklerMemoProxy/UnpicklerMemoProxy,
dump/dumps/load/loads and the PickleError family, so `Lib/pickle.py`
stops falling back to `_Pickler`/`_Unpickler`.

Protocols 0-5 are supported and produce byte-identical output to the
pure-Python pickler, including framing for protocol >= 4, memoization,
`fast` mode, dispatch_table, reducer_override, persistent_id /
persistent_load, find_class with `_compat_pickle` fix_imports, buffer
callbacks and out-of-band protocol 5 buffers. This also turns on
`pickle.PickleBuffer`, which RustPython did not have before.

Speedups over the pure-Python implementation (best of 5, us/op):
pickle 835 -> 21 (39x), pickle_list 525 -> 4.4 (119x),
pickle_dict 4018 -> 32 (125x), unpickle 630 -> 24 (26x),
unpickle_list 317 -> 5.0 (63x). CPython 3.14's C accelerator runs the
same workloads 1.5-2.9x faster than this.

test_pickle now runs 1008 tests (was 470) since the C-implementation
test classes are enabled; test_pickle, test_pickletools,
test_picklebuffer are green and 22 obsolete expectedFailure markers are
removed.

Assisted-by: Claude Code:claude-fable-5-1

* _pickle: raise the pickle.find_class audit event

Unpickler.find_class now raises `sys.audit('pickle.find_class', module,
name)` like CPython, so audit hooks can veto global lookups
(test_audit.test_pickle). Also drops four expectedFailure markers that
the native module makes pass: test_pydoc's builtin-method description
tests, test_memoryview.test_picklebuffer_reference_loop and
test_pyclbr.test_others.

Full suite: 456 test files / 42,648 tests green.

Assisted-by: Claude Code:claude-fable-5-1

* _pickle: avoid temporary buffers on the save fast paths

Opcode plus payload is now appended straight to the output buffer
(`write_pair`) instead of building a `Vec` per int, str, bytes and memo
opcode, `save_tuple` no longer copies the element slice, and the
`when serializing ...` note text is only formatted when an error is
actually raised.

bm_pickle "pickle" workload: 21.4 -> 17.0 us/op.

Assisted-by: Claude Code:claude-fable-5-1

* pyexpat: make the threaded backend fork-safe

fork() only keeps the calling thread. test_platform's
test_mac_ver_with_fork crashed the child with SIGTRAP: plistlib parses
a plist through pyexpat (spawning a background parser thread) before
os.fork(), then parses again in the child. Bisecting away every other
explanation (a plain threading.Thread survives the same fork fine,
since it goes through this VM's own fork-safe thread bookkeeping)
showed the crash is not from touching the *inherited* ParserStream
(already joined by refcounting before the fork in this repro) but from
spawning a *brand new* parser thread from the child, which reliably
crashed during the new thread's own startup.

Register a pthread_atfork child hook (lazily, the first time a parser
thread is spawned) that sets a permanent, process-wide flag the
instant this process is ever the child of a fork(). Once set:
- ParserStream::try_spawn refuses to spawn any further thread and
  Backend::new falls back to Backend::Sync instead, exactly as on
  targets that cannot spawn threads at all.
- Feeding an inherited Backend::Threaded raises a clear ExpatError
  instead of touching its FeedBuffer/channel (whose mutex/condvar may
  look held by a thread that no longer exists), and dropping it leaks
  its JoinHandle/buffer/channel via ManuallyDrop rather than joining or
  destroying them, since even destroying a POSIX mutex that looks held
  is undefined behavior.

The parent is unaffected: the pthread_atfork hook only runs in the
child. Verified with a 200-iteration fork stress test (fresh parsers
and parsers forked mid-parse) and the full test_platform/xml/fork/
subprocess/multiprocessing_fork test modules.

Assisted-by: Claude Code:claude-fable-5-1

* pyexpat: implement name interning via self.intern, matching libexpat

self.intern (an always-present dict by default) was stored but never
consulted; every StartElement/EndElement/attribute name allocated a
fresh PyStr even when the same tag or attribute name repeated across
many sibling events. Wire it up the way libexpat does
(PyDict_SetDefault(self->intern, name, name)): element and attribute
*names* are now memoized so a repeated name reuses the same PyStr
object (and its cached hash) instead of reallocating and rehashing.
Attribute values are left out of the cache, same as libexpat.

This fixes test_pyexpat's previously-xfailed InterningTest.test, which
asserts object identity across repeated StartElementHandler/
EndElementHandler calls.

Measured on the pyperformance xml_etree parse/iterparse/generate/
process variants: no measurable wall-clock change (the dominant cost
there is Python-level TreeBuilder/Element bytecode execution and
thread hand-off, not name allocation), so this is a correctness/
feature-completion fix rather than a throughput win for that
benchmark; kept because it is a real, previously-inert compatibility
gap and is otherwise free.

Assisted-by: Claude Code:claude-fable-5-1

* _elementtree: native Element, SubElement and ParseError

Adds crates/stdlib/src/elementtree.rs providing the accelerator names
xml/etree/ElementTree.py picks up at import time. Element carries the
full API (tag/text/tail/attrib, indexing and slicing, find*/iter/
itertext, copy/deepcopy/pickle state) with CPython's lazily created
attrib dict and deferred text join, is subclassable, weak-referenceable
and GC-traversable, and drops every displaced object after releasing its
lock so a __del__ can re-enter safely.

xml_etree, best of 3 (s): parse 1.64 -> 1.14, iterparse 0.68 -> 0.54,
generate 0.44 -> 0.25, process 0.27 -> 0.16.

Assisted-by: Claude Code:claude-fable-5-1

* _elementtree: native TreeBuilder and XMLParser

TreeBuilder keeps CPython's element/data/tail bookkeeping in Rust,
including the pull-parser event hooks _setevents installs. XMLParser
drives a pyexpat parser whose handlers are this type's own native
methods, so a start or end tag reaches the builder without a Python
frame: the universal-name cache, attribute dict and TreeBuilder call all
happen in Rust when the target is our own builder, while foreign targets
still go through the documented start/end/data/comment/pi protocol.
ElementTree.parse() also picks up the _parse_whole fast path.

xml_etree, best of 3 (s): parse 1.14 -> 0.53, iterparse 0.54 -> 0.28,
generate 0.25 -> 0.24, process 0.16 -> 0.16. Against the pure-Python
baseline that is parse 3.1x, iterparse 2.4x, generate 1.8x, process
1.7x.

Assisted-by: Claude Code:claude-fable-5-1

* pyexpat: stop rescanning the whole line per position query

LineTracker converted a TextPosition column to a byte offset by
validating and decoding its entire buffered line tail on every query,
and appended fed bytes to that tail one at a time. On a document that is
one long line -- what xml.etree.tostring() writes, and what the
xml_etree benchmark parses -- the tail is the whole read-ahead, so a
per-event lookup became quadratic. Walk only the characters the query
actually asks for, over the deque's two halves in place, and record fed
bytes a line at a time. Byte indices are unchanged.

xml_etree, best of 3 (s): parse 0.52 -> 0.38, iterparse 0.28 -> 0.25.

Assisted-by: Claude Code:claude-fable-5-1

* _elementtree: charge Element.__deepcopy__ to the recursion budget

A chain of elements is copied by native recursion, so a deep enough tree
overflowed the real stack instead of raising. Wrap the copy in
with_recursion, as CPython does, and drop the skip on
test_deeply_nested_deepcopy for the accelerated run.

Assisted-by: Claude Code:claude-fable-5-1

* pyexpat: don't let a stale NeedMore end a Parse() early

The parser thread reports NeedMore when it finds the feed buffer empty
and parks. If it reached that point before the chunk Parse() was about
to push became visible -- which it does whenever the thread happens to
be scheduled first, so more often under load -- pump() consumed that
report as if it answered the new chunk and handed control back to Python
having dispatched nothing, silently losing the whole document.

Stamp each NeedMore with the push count observed when the buffer was
found empty and ignore reports older than the push being pumped. The
push notifies the condvar under the same mutex the report was sent
from, so the parser thread is guaranteed to wake and report again.

Fixes an intermittent failure of test_pyexpat's InterningTest under
parallel load (about 5 losses per 20000 parses before, none after).

Assisted-by: Claude Code:claude-fable-5-1

* Restore xml.etree test markers instead of skipTest in bodies

Assisted-by: Grok:4.6

* Clone state before try_to_bool in pickle load_build

try_to_bool takes the object by value, and load_build still needs the
state dict afterwards.

Assisted-by: Grok:4.6

* Drop decorative section-separator comments

Assisted-by: Grok:4.6

* Represent pickle opcodes as an enum

Assisted-by: Grok:4.6

* Use full paths for thread-only types in pyexpat

Assisted-by: Grok:4.6

---------

Co-authored-by: Lee Dogeon <dev.moreal@gmail.com>
2026-09-19 19:21:32 +09:00
Jeong, YunWon 6658ffa717 memoryview: traverse the view for the collector, and name types as tp_name does (#8608)
* memoryview: walk the view and the buffer wrapper for the collector

Neither `memoryview` nor the `_buffer_wrapper` a `__buffer__` produces was
traversed, so the object a view looks at was never reported and a cycle
running through one was classified as reachable:

    class Src:
        def __buffer__(self, flags): return memoryview(b'abcdef')
    o = Src(); o.keep = memoryview(o)   # o is never freed

Both now report what they hold, as `memory_traverse` and `mbuf_traverse` do.

Assisted-by: Claude

* Name the type in an unhashable-type message the way `tp_name` does

The message took the short name, so a type defined in a module lost the
module: `hash(array.array('B', b'a'))` said `unhashable type: 'array'`
where the name is `array.array`, and `collections.deque` said `'deque'`.
`slot_name()` is what `tp_name` holds, and it leaves a class defined in
Python reported by its own name as before.

Assisted-by: Claude
2026-08-29 23:00:31 +09:00
Jeong, YunWon adf7da508b memoryview: count exports, hash the exporter, and index through __index__ (#8553)
* memoryview: index through `__index__` and name the dimension

A tuple key was taken apart only when every item was already an `int`, so
`m[I(), 2]` with an `__index__` on `I` was reported as an invalid slice key
rather than indexed. What kind of key it is now follows from the types in
it, the way `is_multiindex` decides, and each item is converted where it
sits; an item that answers `__index__` but raises on the way reports that
rather than making the whole key invalid.

That conversion runs Python, which can release the view, so the
multi-dimensional read goes through `unpack_single` and its re-check
rather than reading the bytes itself.

`IndexError` named the index that was out of bounds where
`lookup_dimension` names the dimension it was out of bounds on, counted
from one, and the one-dimensional path had a message of its own.

Assisted-by: Claude

* memoryview: count what a view has exported, and hash the exporter

The view kept no count of the buffers taken from it, so `release()` always
succeeded, even while something was still reading through an export. It now
answers `BufferError` the way `_memory_release` does, and `__exit__` reports
the same refusal.

`memory_hash` asks the exporter for its hash and throws the answer away, so
a view is no more hashable than what it looks at; a read-only view over a
`bytearray` hashed here where CPython refuses it. Asking runs Python, so the
view counts as exported for the duration and a release attempted from inside
that hash is refused rather than obeyed.

Assisted-by: Claude

* bytes, bytearray, memoryview: measure the hex separator as `len()` does

`hex()` took the separator's length from its bytes rather than from the
object, so a `bytes` subclass that answers `__len__` was measured by what it
holds instead of what it says: `b'abcd'.hex(S(b'::'))` with `__len__`
returning 1 was refused where `_Py_strhex_impl` accepts it, and a two-byte
answer was accepted where it refuses. The check also came after the
`bytes_per_sep == 0` and empty-data shortcuts, so `b'abcd'.hex('::', 0)`
never reached it at all.

Measuring runs Python, and the bytes to be written out must not be borrowed
while it does, so the separator is now resolved from the arguments before
the buffer is reached. For a memoryview that also means the view counts as
exported for the duration, so a release attempted from inside `__len__` is
refused.

Assisted-by: Claude

* array: refuse to hash an array

`array.array` is mutable but inherited `object.__hash__`, so an array could
be put in a set or used as a dict key and then changed underneath it. It is
unhashable, as `tp_hash = PyObject_HashNotImplemented` makes it.

A read-only memoryview over an array now reports the exporter as unhashable
too, which is what `memory_hash` asks it for.

Assisted-by: Claude

* Remove expectedFailure from three memoryview tests

test_hash_use_after_free, test_hex_use_after_free and
test_use_released_memory pass.

Assisted-by: Claude

* memoryview: bound the dimensions a cast can name

`cast()` accepted a shape of any length, so `mv.cast('B', (1,)*64 + (8,))`
built a 65-dimensional view where `memoryview_cast_impl` refuses anything
past `PyBUF_MAX_NDIM`. The limit is answered before the shape is looked at
any further, as it is there.

Assisted-by: Claude
2026-08-18 22:19:08 +09:00
Jeong, YunWon c603eb7c9a Implement PEP 688 and rework the buffer protocol around managed exports (#8523)
* Implement PEP 688 __buffer__ and __release_buffer__

A Python class could not export a buffer: the slot machinery had no
bf_getbuffer or bf_releasebuffer, and every consumer acquired buffers as
PyBUF_FULL_RO through a module of PyBUF_* constants.

Add both slots. PyBuffer::release now runs a Python __release_buffer__
before the exporter's own release, once per acquisition, which PyBuffer
tracks with an `acquired` flag that clones do not inherit. An export made
by a Python __buffer__ is held by a _buffer_wrapper payload that counts its
exports and drops the returned memoryview with the last one, and the view
handed to __release_buffer__ is a _buffer_window that owns no export, so
releasing it inside the hook is inert instead of re-entering it.

Replace the PyBUF_* constants with a BufferFlags bitflags type whose
composite requests are supersets of the simpler ones, so `contains` answers
the REQ_* questions, and pass the request to PyBuffer::from_object. Each
consumer now asks for what its counterpart asks for: y* arguments for
SIMPLE, w* for WRITABLE, BytesIO.write for CONTIG_RO, bytes(), bytearray()
and memoryview() for FULL_RO. memoryview checks the request in
memory_getbuf, and array.array and mmap.mmap expose __release_buffer__.

Test buffer support with PyObject::check_buffer (PyObject_CheckBuffer)
instead of attempting an acquisition, so an exception raised by __buffer__
is no longer reported as the object not being bytes-like, and a __buffer__
with side effects runs once. PyBytesInner becomes a y* conversion as a
result: bytes and bytearray methods no longer accept iterables of ints, and
find, index, count and __contains__ take the arguments
parse_args_finds_byte and bytes_contains describe.

A view exports its start offset in the descriptor rather than in its
window, which fixes a panic when collecting from a negative-stride view.

BytesIO.write rechecks closed after acquiring its buffer, which __buffer__
can close in between.

Assisted-by: Claude Code:claude-opus-5

* Rework the buffer protocol around managed exports and view offsets

Give `PyBuffer` the `_PyManagedBufferObject` shape: one `bf_getbuffer`
acquisition is shared by every handle taken from it, cloning takes another
share instead of re-acquiring, and the exporter's release runs once when the
last share goes away. Remove `retain`, the unsafe `drop_without_release`, the
three `impl Drop`s and the `ManuallyDrop` that stood in for this. Add
`abort_acquisition` so a failed request does not run `bf_releasebuffer`.

Move the view start into `BufferDescriptor::offset`, the `Py_buffer.buf`
analogue, and drop the separate `start` fields on `PyMemoryView` and
`PyBufferWrapper`. Slicing goes through `SaturatedSlice::adjust_indices_start`,
which reproduces `PySlice_AdjustIndices` and keeps the adjusted start.

Fix `zip_eq` to take its contiguous fast path only when both last dimensions
are contiguous, and make `for_each_segment` and `zip_eq` handle zero-length and
zero-dimensional views.

Add `BufferDescriptor::projected` so a request without `PyBUF_ND`,
`PyBUF_STRIDES` or `PyBUF_FORMAT` receives a correspondingly reduced
descriptor, and reject a request without `PyBUF_INDIRECT` against an exporter
that has suboffsets.

Copy the source first in `memoryview` slice assignment when both sides reach
the same root exporter.

Hold the export across the resize in `bytearray.extend`, take `y*` in
`marshal.loads`, stop probing the buffer protocol in `FsPath`, rewrite `ord`
over the concrete string types, fold `array`'s buffer slot into one
`slot_as_buffer`, take `w*`/`y*` in `_overlapped`, and thread the new `offset`
field through the `_ctypes` descriptors.

Assisted-by: Claude

* Re-check a memoryview after a conversion and tighten cast and release

`pack_single` and `unpack_single` addressed the buffer with a position taken
before `__index__` ran, so releasing the view from that conversion read or wrote
outside the exporter's storage, panicking when it had also shrunk. Check the
released flag again once the conversion is done, as `CHECK_RELEASED_AGAIN` does.

Reject a cast to a format that is not a single native format character with an
optional `@` in front of it. An empty format reached a division by its item size
of zero. get_native_fmtchar

Report a second `__release_buffer__` on the same view as a `ValueError` rather
than accepting it, and check that the view belongs to the object first; the
silent case is a view that exports nothing. wrap_releasebuffer

Compare against another memoryview by reading its view where it lies instead of
acquiring a buffer from it, so the restricted view handed to
`__release_buffer__` compares equal rather than unequal in one direction only.
memory_richcompare

Name the type in the unraisable an exception from `__release_buffer__` reports,
as `releasebuffer_call_python` does, instead of reporting the exporter object.

Assisted-by: Claude

* Re-check the destination of a memoryview slice assignment

Acquiring the source runs `__buffer__`, which can release the destination view,
so check the released flag again once the source is in hand and before the
structures are compared, as `copy_single` does.

Build the sliced destination as a view that counts as no export, the way a
`Py_buffer dest = *view` copy does. Holding one kept the exporter unresizable
for the length of the assignment, so a source that released the view and then
resized the exporter met a `BufferError` instead of the assignment reporting the
released view.

Assisted-by: Claude

* Answer the memoryview contiguity getsets with the layout

`f_contiguous` reported the row-major answer for one dimension and False
for any more, so a view laid out both ways, such as one of shape (1, 8),
reported False while `check_buffer_request` accepted a `F_CONTIGUOUS`
request on it. `contiguous` answered row-major order alone.

Move the Fortran-order test from `PyMemoryView` to `BufferDescriptor`,
next to the row-major one it differs from only in iteration order, and
answer all three getsets with it.

Assisted-by: Claude
2026-08-16 21:32:29 +09:00
Yubin Kim 79f7da3b47 bytes: support keyword arguments in hex() (#8312)
* bytes: support keyword arguments in hex()

* bytes: unmark test_memoryview_hex_separator
2026-07-22 20:31:31 +09:00
Shahar Naveh 292f66e051 Update some tests to 3.14.5 (#7909)
* Align patches for `test_class.py`

* Update more tests to 3.14.5

* Update `test_math`

* `test_hash.py`

* Update `test_grammar.py`

* Fix failing test

* Restore `test_grammar.py`
2026-05-19 13:31:33 +09:00
CPython Developers 7ef9e5ebff Update test_memoryview from v3.14.3 2026-02-10 22:27:46 +09:00
Jeong YunWon 3073947a8b mark failing test_memoryview tests 2022-08-15 04:24:58 +09:00
CPython Developers b5dcd61b58 Update test_memoryview from CPython 3.10.6 2022-08-15 04:24:58 +09:00
CPython developers c7a5b960be Update test_memoryview from CPython 3.10.5 2022-07-18 04:07:35 +09:00
Dean Li 6f98288e84 test: use import_helper 2021-11-29 21:03:02 +08:00
Noa e51cb82ff0 Unskip tests 2021-11-15 16:08:46 -06:00
Steve ShiandJeong YunWon 742ea0c4fa Reimpl Buffer Protocol and memoryview support ndarray with shape, stride and suboffset (#3340)
* Fix buffer protocol and memoryview

* Fix io and array the right way to use buffer protocol

* fix rebase

* fix memoryview and introduce VecBuffer

* fix deadlock

* fix memoryview avoid double release buffer

* impl ndim buffer support

* first implement nd array for buffer and memoryview

* fix slice adjust_indices

* fix adjust_indices introduct SaturatedSliceIterator

* fix memoryview cmp

* fix adjust_indices positive_order

* fix bug mark passed test

* fix clippy

* fix set_item_slice deadlock, optimize buffer

* Delete @test_113974_tmp

* Remove vec_buffer_type

* rusty names

* impl memoryview multi index

* add comments

Co-authored-by: Jeong YunWon <jeong@youknowone.org>
2021-11-13 07:17:11 +02:00
Padraic Fanning e25941a5f8 Unskip test(s) in test_memoryview 2021-02-25 21:46:00 -05:00
Padraic Fanning 5996bafae0 Explain test_gc skip 2021-02-07 15:49:42 -05:00
Noah f428fcc692 unskip tests 2021-01-25 18:28:18 -06:00
Kangzhi Shi 44048f60cb Implement memoryview cast 2020-12-27 19:03:05 +02:00
hyperbora 928fbcda50 fix memoryview compare 2020-12-07 21:58:54 +09:00
Jeong YunWon b230045677 Merge pull request #2299 from qingshi163/memoryview_hash
Implement memoryview hash
2020-10-25 13:51:09 +09:00
Kangzhi Shi b38d278f91 memoryview constructor args 2020-10-22 16:45:30 +02:00
Kangzhi Shi 101d8c8dc1 memoryview hash 2020-10-22 13:17:54 +02:00
Kangzhi Shi ce097617f7 fix memoryview is not pickleable 2020-10-17 13:35:29 +02:00
Kangzhi Shi 6058d653a8 memoryview hex 2020-10-10 17:01:00 +09:00
Kangzhi Shi b48c5a2b52 BytesIO getbuffer; no more hack for pickle.py 2020-10-10 17:01:00 +09:00
Kangzhi Shi 3fe8a30ca0 bytesinner from pytuple 2020-10-10 17:01:00 +09:00
Kangzhi Shi 275b727e9d memoryview __setitem__ 2020-10-10 17:01:00 +09:00
Kangzhi Shi cf3572385c Fix more to pass unittest 2020-10-10 17:00:57 +09:00
Kangzhi Shi 499d99794c Implement Buffer Protocol 2020-10-10 16:57:19 +09:00
Noah b245712949 Fix some 32-bit only oom tests + others 2020-06-06 17:14:32 -05:00
Jeong YunWon b9f2b87758 Mark failing tests from test_memoryview.py 2020-03-08 05:53:30 +09:00
CPython dev 4a329bdc65 Lib/test/test_memoryview.py from CPython 3.8.2 2020-03-08 05:53:30 +09:00