4 Commits

Include renames

Author	SHA1	Message	Date
Jeong, YunWonandLee Dogeon	358f6a875a	native _json, _pickle, pyexpat, and _elementtree (#8741) ... * _json: add native make_encoder (c_make_encoder) json.encoder fell back to the pure-Python _make_iterencode because _json had no make_encoder, making json.dumps ~25x slower than CPython. Implement the C encoder semantics natively: str/None/bool/int/float/list/tuple/dict ordering, key coercion and skipkeys, sort_keys via the VM's fallible sort (comparison errors propagate instead of panicking), circular reference detection, indent handling, allow_nan, default() fallback with recursion guard, and CPython's add_note annotations. Removes the now-passing expectedFailure markers in test_json. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: add UseForeignDTD, fix silently swallowed handler exceptions genshi's XML template loader (bm_genshi) called xmlparser.UseForeignDTD(), which pyexpat.rs didn't implement, raising AttributeError. Add it as an accepted no-op: the pyexpat module here wraps the pure-Rust xml-rs crate (not real libexpat), which has no hook for splicing in a foreign DTD. Root-causing the resulting bug also uncovered why genshi's template generate()/render() produced empty output instead of erroring: do_parse's invoke_handler swallowed every exception raised by a Python callback (handler.call(...).ok()), and genshi's expat callbacks fail immediately by reading `parser.CurrentLineNumber`/`CurrentColumnNumber`, which pyexpat.rs never exposed. The AttributeError was silently discarded, so no start/end/text events ever reached genshi's queue. Fix both: propagate handler exceptions out of Parse()/ParseFile() (matching libexpat/CPython semantics), and add CurrentLineNumber/CurrentColumnNumber/ CurrentByteIndex as read-only attributes backed by xml-rs's per-event TextPosition (byte index is derived from it and is exact for ASCII documents). This also fixes test_minidom's testExceptionOnSpacesInXMLNSValue, previously an expected failure because the ValueError from an invalid xmlns value was swallowed the same way. PositionTest in test_pyexpat.py checks byte-exact expat semantics (token *start* position) that xml-rs's *current parse cursor* position can't reproduce exactly, so that one test is now marked an expected failure. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: support multi-chunk Parse()/ParseFile() feeds bm_xml_etree failed with AssertionError: missing end tags from TreeBuilder.close(), because xml.etree.ElementTree reads and feeds large documents to XMLParser.feed() in ~64KB chunks (and xml.sax's IncrementalParser and ElementTree's XMLPullParser do the same). Each Parse()/ParseFile() call built a brand-new xml-rs EventReader from only the bytes passed in that call, discarding all parser state between calls. When a chunk boundary fell in the middle of an element, the fragment parsed by itself was invalid/incomplete XML, so xml-rs silently errored out (matching this module's existing "ignore stricter xml-rs errors" behavior) and the corresponding start/end events were dropped entirely -- not just delayed. xml-rs has no API to resume a parser across separate Read sources, so instead each call now appends the new chunk to a persistent buffer and re-parses the buffer as a whole, replaying only the events at or beyond the count already dispatched in previous calls (dispatched_events). This also makes position tracking (CurrentLineNumber/Column/ByteIndex) correct regardless of how the input was chunked. Trade-off: each Parse() call is now O(size fed so far), so a document delivered in many small chunks is parsed several times over; acceptable for correctness but noticeably slower than a true incremental parser for pathological chunk sizes. This also fixes several previously-xfailed tests that depend on incremental feeding: test_xml_etree.py's XMLPullParserTest (test_events, test_feed_while_iterating, test_simple_xml, test_simple_xml_chunk_22, test_simple_xml_with_ns) and ElementTreeTest.test_parseliteral, plus test_sax.py's ExpatReaderTest (test_expat_entityresolver_default, test_expat_incremental, test_expat_incremental_reset) -- their expectedFailure markers are removed. test_simple_xml_chunk_1/_chunk_5 (1- and 5-byte chunks) still fail and remain xfail: at that granularity xml-rs's stricter-than-expat tokenizer errors on some intermediate prefixes that libexpat tolerates. Assisted-by: Claude Code:claude-fable-5-1 * test: drop stale expectedFailure markers fixed by the pyexpat changes test_plistlib's four invalid-document tests and test_xmlrpc's test_loads_unsupported pass now that pyexpat no longer swallows handler exceptions and keeps parser state across chunked feeds. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: drive a single background EventReader for true incremental Parse() 8802451f3 fixed multi-chunk Parse()/ParseFile() feeds by accumulating every fed byte and re-parsing the whole buffer from scratch on each call, which is O(total bytes) per chunk -- quadratic in document size. xml-rs's EventReader has no API to pause/resume across separate Read sources, so instead one EventReader is now driven to completion on a dedicated background thread fed through a condvar-backed queue (FeedBuffer/FeedReader); Parse()/ParseFile() push their chunk, wake the thread, and drain (on the calling thread only) whatever events that unblocks, stopping once the thread reports it is blocked waiting for more input. Handlers always run on the calling thread. A handler exception stops dispatch without tearing down the stream, so the next Parse() resumes where it left off, mirroring libexpat. CurrentByteIndex is now computed incrementally per line (with an amortized per-query cursor, needed since real documents are often a single very long line) instead of scanning the whole buffer. ParserCreate() never spawns a thread; one is spawned lazily by the first Parse()/ParseFile() call, and dropping the parser always stops and joins it. Fixes test_xml_etree{,_c}'s test_events_comment, previously xfailed because the old whole-buffer replay lost top-level comment events across feeds. Measured (M4 Pro, release build): - Scaling (ElementTree.parse, newline-separated doc): new backend 1/4/16 MB = 0.15/0.61/2.35s (linear); CPython 3.14 same file = 0.015/0.055/0.23s. - pyperformance's bm_xml_etree doc (156KB, effectively one line): old backend single etree.parse() = 1.99s, single iterparse() = 3.53s; new backend = 0.025s / 0.031s (~80-110x). Full bench_parse/iterparse/generate/ process (30/10/10/1 reps) now finish in ~1.5s/0.6s/0.4s/0.3s instead of timing out past pyperformance's 400s limit. - 10k ParserCreate() (never fed / partially fed / fully fed and dropped): 0.32s, no residual native threads, no RSS growth. test -j4 test_pyexpat test_xml_etree test_xml_etree_c test_sax test_minidom test_xml_dom_minicompat test_pulldom test_plistlib test_xmlrpc test_html test_htmlparser test_threading: all pass (1350 tests, 89 skipped). Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: add a Sync fallback backend for targets without threads 2f58e9531 made pyexpat always drive its EventReader on a background thread (ParserStream::spawn), panicking via .expect() if the thread couldn't be spawned. pyexpat is registered unconditionally, including on wasm32, where std::thread::spawn always fails at runtime, so the first Parse() call panicked there. Introduce a Backend enum: Backend::Threaded (the existing ParserStream) on targets that can spawn threads, and a new Backend::Sync, the accumulate-and-reparse implementation from 8802451f3, used on wasm32 and as a runtime fallback if spawning a thread ever fails. Both share LineTracker for identical CurrentLineNumber/CurrentColumnNumber/ CurrentByteIndex and handler/error semantics. Fixed two correctness bugs while porting the Sync backend forward: marking the parser permanently "finished" as soon as a from-scratch reparse happened to reach EndDocument (which can be a false positive on a not-yet-complete buffer, e.g. right before epilog content arrives) rather than only on the caller's real isfinal signal; and counting that same EndDocument towards the dispatched-event index, which could make a real event land on an index a later, longer reparse had already skipped past. Verified by force-selecting Backend::Sync under a test-only env var (removed) and diffing its output against Backend::Threaded for the xml/sax/etree/plistlib/xmlrpc test suites and a 2MB document fed in 1KB chunks. Assisted-by: Claude Code:claude-fable-5-1 * _pickle: native Pickler, Unpickler and PickleBuffer Adds `crates/stdlib/src/pickle.rs`, a native `_pickle` implementing Pickler, Unpickler, PickleBuffer, PicklerMemoProxy/UnpicklerMemoProxy, dump/dumps/load/loads and the PickleError family, so `Lib/pickle.py` stops falling back to `_Pickler`/`_Unpickler`. Protocols 0-5 are supported and produce byte-identical output to the pure-Python pickler, including framing for protocol >= 4, memoization, `fast` mode, dispatch_table, reducer_override, persistent_id / persistent_load, find_class with `_compat_pickle` fix_imports, buffer callbacks and out-of-band protocol 5 buffers. This also turns on `pickle.PickleBuffer`, which RustPython did not have before. Speedups over the pure-Python implementation (best of 5, us/op): pickle 835 -> 21 (39x), pickle_list 525 -> 4.4 (119x), pickle_dict 4018 -> 32 (125x), unpickle 630 -> 24 (26x), unpickle_list 317 -> 5.0 (63x). CPython 3.14's C accelerator runs the same workloads 1.5-2.9x faster than this. test_pickle now runs 1008 tests (was 470) since the C-implementation test classes are enabled; test_pickle, test_pickletools, test_picklebuffer are green and 22 obsolete expectedFailure markers are removed. Assisted-by: Claude Code:claude-fable-5-1 * _pickle: raise the pickle.find_class audit event Unpickler.find_class now raises `sys.audit('pickle.find_class', module, name)` like CPython, so audit hooks can veto global lookups (test_audit.test_pickle). Also drops four expectedFailure markers that the native module makes pass: test_pydoc's builtin-method description tests, test_memoryview.test_picklebuffer_reference_loop and test_pyclbr.test_others. Full suite: 456 test files / 42,648 tests green. Assisted-by: Claude Code:claude-fable-5-1 * _pickle: avoid temporary buffers on the save fast paths Opcode plus payload is now appended straight to the output buffer (`write_pair`) instead of building a `Vec` per int, str, bytes and memo opcode, `save_tuple` no longer copies the element slice, and the `when serializing ...` note text is only formatted when an error is actually raised. bm_pickle "pickle" workload: 21.4 -> 17.0 us/op. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: make the threaded backend fork-safe fork() only keeps the calling thread. test_platform's test_mac_ver_with_fork crashed the child with SIGTRAP: plistlib parses a plist through pyexpat (spawning a background parser thread) before os.fork(), then parses again in the child. Bisecting away every other explanation (a plain threading.Thread survives the same fork fine, since it goes through this VM's own fork-safe thread bookkeeping) showed the crash is not from touching the *inherited* ParserStream (already joined by refcounting before the fork in this repro) but from spawning a *brand new* parser thread from the child, which reliably crashed during the new thread's own startup. Register a pthread_atfork child hook (lazily, the first time a parser thread is spawned) that sets a permanent, process-wide flag the instant this process is ever the child of a fork(). Once set: - ParserStream::try_spawn refuses to spawn any further thread and Backend::new falls back to Backend::Sync instead, exactly as on targets that cannot spawn threads at all. - Feeding an inherited Backend::Threaded raises a clear ExpatError instead of touching its FeedBuffer/channel (whose mutex/condvar may look held by a thread that no longer exists), and dropping it leaks its JoinHandle/buffer/channel via ManuallyDrop rather than joining or destroying them, since even destroying a POSIX mutex that looks held is undefined behavior. The parent is unaffected: the pthread_atfork hook only runs in the child. Verified with a 200-iteration fork stress test (fresh parsers and parsers forked mid-parse) and the full test_platform/xml/fork/ subprocess/multiprocessing_fork test modules. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: implement name interning via self.intern, matching libexpat self.intern (an always-present dict by default) was stored but never consulted; every StartElement/EndElement/attribute name allocated a fresh PyStr even when the same tag or attribute name repeated across many sibling events. Wire it up the way libexpat does (PyDict_SetDefault(self->intern, name, name)): element and attribute *names* are now memoized so a repeated name reuses the same PyStr object (and its cached hash) instead of reallocating and rehashing. Attribute values are left out of the cache, same as libexpat. This fixes test_pyexpat's previously-xfailed InterningTest.test, which asserts object identity across repeated StartElementHandler/ EndElementHandler calls. Measured on the pyperformance xml_etree parse/iterparse/generate/ process variants: no measurable wall-clock change (the dominant cost there is Python-level TreeBuilder/Element bytecode execution and thread hand-off, not name allocation), so this is a correctness/ feature-completion fix rather than a throughput win for that benchmark; kept because it is a real, previously-inert compatibility gap and is otherwise free. Assisted-by: Claude Code:claude-fable-5-1 * _elementtree: native Element, SubElement and ParseError Adds crates/stdlib/src/elementtree.rs providing the accelerator names xml/etree/ElementTree.py picks up at import time. Element carries the full API (tag/text/tail/attrib, indexing and slicing, find*/iter/ itertext, copy/deepcopy/pickle state) with CPython's lazily created attrib dict and deferred text join, is subclassable, weak-referenceable and GC-traversable, and drops every displaced object after releasing its lock so a __del__ can re-enter safely. xml_etree, best of 3 (s): parse 1.64 -> 1.14, iterparse 0.68 -> 0.54, generate 0.44 -> 0.25, process 0.27 -> 0.16. Assisted-by: Claude Code:claude-fable-5-1 * _elementtree: native TreeBuilder and XMLParser TreeBuilder keeps CPython's element/data/tail bookkeeping in Rust, including the pull-parser event hooks _setevents installs. XMLParser drives a pyexpat parser whose handlers are this type's own native methods, so a start or end tag reaches the builder without a Python frame: the universal-name cache, attribute dict and TreeBuilder call all happen in Rust when the target is our own builder, while foreign targets still go through the documented start/end/data/comment/pi protocol. ElementTree.parse() also picks up the _parse_whole fast path. xml_etree, best of 3 (s): parse 1.14 -> 0.53, iterparse 0.54 -> 0.28, generate 0.25 -> 0.24, process 0.16 -> 0.16. Against the pure-Python baseline that is parse 3.1x, iterparse 2.4x, generate 1.8x, process 1.7x. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: stop rescanning the whole line per position query LineTracker converted a TextPosition column to a byte offset by validating and decoding its entire buffered line tail on every query, and appended fed bytes to that tail one at a time. On a document that is one long line -- what xml.etree.tostring() writes, and what the xml_etree benchmark parses -- the tail is the whole read-ahead, so a per-event lookup became quadratic. Walk only the characters the query actually asks for, over the deque's two halves in place, and record fed bytes a line at a time. Byte indices are unchanged. xml_etree, best of 3 (s): parse 0.52 -> 0.38, iterparse 0.28 -> 0.25. Assisted-by: Claude Code:claude-fable-5-1 * _elementtree: charge Element.__deepcopy__ to the recursion budget A chain of elements is copied by native recursion, so a deep enough tree overflowed the real stack instead of raising. Wrap the copy in with_recursion, as CPython does, and drop the skip on test_deeply_nested_deepcopy for the accelerated run. Assisted-by: Claude Code:claude-fable-5-1 * pyexpat: don't let a stale NeedMore end a Parse() early The parser thread reports NeedMore when it finds the feed buffer empty and parks. If it reached that point before the chunk Parse() was about to push became visible -- which it does whenever the thread happens to be scheduled first, so more often under load -- pump() consumed that report as if it answered the new chunk and handed control back to Python having dispatched nothing, silently losing the whole document. Stamp each NeedMore with the push count observed when the buffer was found empty and ignore reports older than the push being pumped. The push notifies the condvar under the same mutex the report was sent from, so the parser thread is guaranteed to wake and report again. Fixes an intermittent failure of test_pyexpat's InterningTest under parallel load (about 5 losses per 20000 parses before, none after). Assisted-by: Claude Code:claude-fable-5-1 * Restore xml.etree test markers instead of skipTest in bodies Assisted-by: Grok:4.6 * Clone state before try_to_bool in pickle load_build try_to_bool takes the object by value, and load_build still needs the state dict afterwards. Assisted-by: Grok:4.6 * Drop decorative section-separator comments Assisted-by: Grok:4.6 * Represent pickle opcodes as an enum Assisted-by: Grok:4.6 * Use full paths for thread-only types in pyexpat Assisted-by: Grok:4.6 --------- Co-authored-by: Lee Dogeon <dev.moreal@gmail.com>	2026-09-19 19:21:32 +09:00
Shahar Naveh	bbb6c66a29	Update pickle.py from 3.14.2 (#6982) ... * Update `_compat_pickle.py` from 3.14.2 * Update `pickle.py` from 3.14.2 * Update pickletools and tests * Update all other pickle related files * Make `test_extcall` to use modified doctest checker	2026-02-05 19:27:10 +09:00
ShaharNaveh	0d1a68dfab	mark failing tests in test_picklebuffer.py	2025-07-25 15:41:37 +02:00
ShaharNaveh	68cd33f37e	Update pickle tests from 3.13.5	2025-07-25 14:53:03 +02:00