_PyObject_LookupSpecial needs a PyErr_Occurred() check to handle
failures while during the lookup, or else it can fail with a "returned
a result with an exception set" error.
Add some missing ones in bytes and weakref/
PyBytesWriter_Finish() doesn't need to check the size. It's known to
be valid.
PyBytesWriter_Create() now only calls byteswriter_resize() if the
size is larger than the small buffer.
Remove byteswriter_data() function: call static inline
_PyBytesWriter_GetData() function directly.
Add bytes_alloc() helper function. bytes_alloc() and
_PyBytes_FromStringAndSize() have less conditional branches than
previous code, and so should be a little bit faster.
* Rename existing bytes_alloc() to bytes_type_alloc().
* Replace _PyBytes_FromSize(calloc=1) with _PyBytes_FromSizeZero().
* Add set_ob_shash_unsafe(): similar to set_ob_shash() but don't use
an atomic operation on Free Threading. Use this new function
on newly allocated bytes objects and in bytes_resize_inplace().
* Replace PyBytes_FromStringAndSize(NULL, size) with
bytes_alloc(size).
* Use bytes_alloc() in PyBytes_FromString() and _PyBytes_Repeat().
* Add _PyBytes_FromStringAndSize(): similar to
PyBytes_FromStringAndSize(), but str must not be NULL.
* Replace PyBytes_FromStringAndSize() with
_PyBytes_FromStringAndSize().
* Test that PyBytes_FromString() and PyBytes_FromStringAndSize()
return singletons for 0 or 1 bytes.
* Fix test_tracemalloc.test_set_traceback_limit().
_PyBytes_Resize() now creates a new bytes object if the hash value
was already computed.
bytes_resize_inplace() no longer sets the hash value to -1, since
this function must not be called if the hash value was already
computed.
Add tests on PyUnicode_Resize().
Add specialized bytes_resize_inplace() and inline
PyBytesWriter_Discard() in PyBytesWriter_FinishWithSize().
No longer copy the buffer if _PyObject_IsUniquelyReferenced() is false.
Add also bytes_get_char() function.
Fix PyBytesWriter_Format() error handling: restore the writer to its
previous state on error. Add an unit test injecting MemoryError.
Fix PyBytesWriter_FinishWithSize() to return an empty bytearray when
use_bytearray is set and size is 0, instead of always returning an
empty bytes object.
Fix test_capi.test_bytes: always check writer finish result type, to
make sure that _PyBytesWriter_CreateByteArray() always create
bytearray objects.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Do no read output_start pointer after PyBytesWriter_Finish() since
the pointer became a dangling pointer.
Co-authored-by: Vyron Vasileiadis <hi@fedonman.com>
* Add _PyBytes_GET_CACHED_HASH() static inline function.
* _PyBytes_IsMutable() makes sure that the hash value is not cached
yet.
* bytearray_reinit_from_bytes() checks that the bytes object is
mutable.
Co-authored-by: Cody Maloney <cmaloney@theoreticalchaos.com>
PyBytesWriter_FinishWithSize() and PyUnicodeWriter_Finish() now
discard the output string instead of resizing it if the single
byte/character singleton is used.
Add byteswriter_check_consistency() to check that the bytes/bytearray
object can be mutated. It is called by all PyBytesWriter functions.
* Add test_thread to PyBytesWriter tests.
* _PyBytes_IsMutable() now checks _PyObject_IsUniquelyReferenced().
* Clarify thread safety in PyBytesWriter documentation.
Adjust the logic to set newly allocated bytes to a known byte pattern
(PyBytesWrite_NEW_BYTE). Only copy 'size' bytes from the small buffer
to the new bytes/bytearray object.
When Python is built in debug mode, bytes an bytearray destructors
now check if the trailing null byte has been overridden to detect
overflow. Add bytes_dealloc() to implement the check.
Add _PyBytes_CheckOverflow() to share code
Reserve one byte in PyBytesWriter used as a canary byte: set it to a
special value. PyBytesWriter_Finish() checks if the canary byte has
been overriden to detect a buffer overflow. Add tests for this feature.
Update buffer overflow check in fcntl: allocate extra guard bytes in
the writer and then truncate these bytes.
Elaborate on mutability in _PyBytes_Resize() and PyBytesWriter
documentation.
Check _PyBytes_IsMutable() in functions which require a mutable bytes
object like PyBytesWriter_Resize().
* Fix PyBytesWriter_Grow() check on the size argument.
* Fix undefined behavior in _PyBytes_ResizeKeepOnError(): raise OverflowError
on integer overflow.
* Add write(), grow() and get_data() methods to the _testcapi wrapper of
PyBytesWriter.
* Remove the string argument from _testcapi.PyBytesWriter() and
resize() method.
* Add tests on PyBytesWriter_Resize() invalid sizes.
If PyBytesWriter_Resize() fails, leave the writer unchanged.
Add a new internal _PyBytes_ResizeKeepOnError() function similar to
_PyBytes_Resize() but leaves the bytes object unchanged on error.
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
Reject size larger than the allocated size.
Check also negative size in PyBytesWriter_FinishWithSize() to always
raise ValueError. Previously, the function raised SystemError or
ValueError depending on the code path.
Replace _PyBytesWriter_GetAllocated() optimization with
_PyBytesWriter_ResizeToAllocated() to update the writer size to its
allocated size.
test_capi: only run the 3 "example" tests in BytesWriterTest. There
is no need to rerun the same tests in ByteArrayWriterTest.
The intent is to test "resize and overallocate", even if in practice
"resize & overallocate" is the same (both variables are either 0 or
1). Use "&&" to better describe the intent of the test.
Introduce a bytearray_new() function to ensure that
ob_bytes_object is always set on a bytearray.
Resizing a bytearray to 0 length now explicitly sets
the ob_bytes_object to the empty constant immortal.
Add a check in the 'bytearray init from string'
fast path to ensure there are no active exports.
This fixes asserts/crashes on the following:
- bytearray(1).__init__()
- bytearray().__new__(bytearray).append(1)
- a = bytearray(); b = memoryview(a); a.__init__('x', 'ascii')
Co-authored-by: Cody Maloney <cmaloney@users.noreply.github.com>
* Add FOR_ITER_VIRTUAL to specialize FOR_ITER for virtual iterators
* Add GET_ITER_SELF to specialize GET_ITER for iterators (including generators)
* Add GET_ITER_VIRTUAL to specialize GET_ITER for iterables as virtual iterators
* Add new (internal) _tp_iteritem function slot to PyTypeObject
* Put limited RESUME at start of genexpr for free-threading. Fix up exception handling in genexpr
Accepted range for the bytes_per_sep argument of bytes.hex(),
bytearray.hex(), memoryview.hex(), and binascii.b2a_hex()
is now increased, so passing sys.maxsize and -sys.maxsize is now
valid.
This affects string formatting as well as bytes and bytearray formatting.
* For errors in the format string, always include the position of the
start of the format unit.
* For errors related to the formatted arguments, always include the number
or the name of the formatted argument.
* Suggest more probable causes of errors in the format string (stray %,
unsupported format, unexpected character).
* Provide more information when the number of arguments does not match
the number of format units.
* Raise more specific errors when access of arguments by name is mixed with
sequential access and when * is used with a mapping.
* Add tests for some uncovered cases.
Optimize bytes.translate() by deferring change detection
Move the equality check out of the hot loop to allow better compiler
optimization. Instead of checking each byte during translation, perform
a single memcmp at the end to determine if the input can be returned
unchanged.
This allows compilers to unroll and pipeline the loops, resulting in ~2x
throughput improvement for medium-to-large inputs (tested on an AMD zen2).
No change observed on small inputs.
It will also be faster for bytes subclasses as those do not need change
detection.
Update `bytearray` to contain a `bytes` and provide a zero-copy path to
"extract" the `bytes`. This allows making several code paths more efficient.
This does not move any codepaths to make use of this new API. The documentation
changes include common code patterns which can be made more efficient with
this API.
---
When just changing `bytearray` to contain `bytes` I ran pyperformance on a
`--with-lto --enable-optimizations --with-static-libpython` build and don't see
any major speedups or slowdowns with this; all seems to be in the noise of
my machine (Generally changes under 5% or benchmarks that don't touch
bytes/bytearray).
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Maurycy Pawłowski-Wieroński <5383+maurycy@users.noreply.github.com>
The previous `Py_REFCNT(x) == 1` checks can have data races in the free
threaded build. `_PyObject_IsUniquelyReferenced(x)` is a more conservative
check that is safe in the free threaded build and is identical to
`Py_REFCNT(x) == 1` in the default GIL-enabled build.
Replace PyBytes_FromStringAndSize() and _PyBytes_Resize() with the
PyBytesWriter API.
Add _PyBytesWriter_GetSize() and _PyBytesWriter_GetData() static
inline functions.
