test_https_client_non_tls_response_ignored() now uses a context
manager for the server and always closes the connection to avoid
ResourceWarning.
Also add a timeout to SingleConnectionTestServerThread.__exit__() to
not hang forever if something goes wrong.
fd_count() can now use /dev/fd/ fdescfs file system if it's mounted,
which is not the case by default. Otherwise, call os.dup(fd) in a
loop which is slow since os.sysconf("SC_OPEN_MAX") is larger than
200,000 by default.
Fix test_is_python_process_non_python_subprocess(). Wait until the
forked Python process calls exec() before calling is_python_process().
Previously, is_python_build() returned True if exec() was not called
yet in the forked process.
test_strings() gets gdb.host_charset(), but fails if gdb writes
anything to stderr. For example, it fails if gdb logs the warning:
"/usr/bin/gdb: warning: Couldn't determine a path for the index
cache directory."
Even if stdout contains the expected encoding ("UTF-8").
Change test_gdb to only fail if stdout is empty, ignore stderr
otherwise to ignore gdb warnings.
Always call pthread_attr_init() in hardware_stack_limits(), not only
on FreeBSD (if pthread_attr_get_np() is available).
Emscripten: add pthread_attr_init() workaround.
Use pthread_attr_get_np() on FreeBSD to get the stack size (and guard
size) of the current thread.
Check if pthread_attr_get_np() function and <pthread_np.h> header are
available in configure.
Update test_json.test_recursion. On FreeBSD 16 with a default stack size of
512 MB, the json C implementation no longer raises RecursionError in
test_highly_nested_objects_encoding(). Reduce also max_depth from 5,000 to
2,000 so test_highly_nested_objects_encoding() of the Python implementation
only takes 1.6 seconds, instead of 12 seconds.
Use the decorator on test methods, not on the whole class.
Previously, test_json ran no tests:
$ ./python -m test -v test_free_threading.test_json -v
...
Ran 0 tests in 0.000s
NO TESTS RAN
Add also a check in @reap_threads to detect misuse (on a class).
If Py_MIN/MAX/ABS() macros are called on an existing "_x" or "_y"
variable name, the external variable is used instead of macro local
variable. So use more unique names in the macros to avoid
the issue.
Add test_min_max_abs_macros() to _testlimitedcapi.
* Add 7 INCREF/DECREF macros
* distinguish between 'const' and 'macro' in stable_abi.dat
The file now has the 'kind' from the toml, not the ReST role.
* Fix kind of PY_VECTORCALL_ARGUMENTS_OFFSET
* Make it explicit that macros aren't part of the ABI
* Add is_static_inline_function to the TOML file (these are treated as macros in the docs)
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Add a new internal _Py_strerror() function to Python/fileutils.c. It
uses strerror_r() if available, or use strerror() otherwise.
Replace all strerror(code) calls with _Py_strerror(code).
Optimize io.BufferedReader.readline() when lines are close to the
buffer size (128 kB by default) or longer than the buffer size.
Replace the C loop searching for the newline byte in the buffer with
a memchr() call which is more efficient. In the good case,
it makes readline() up to 4x faster!
Initialize all fields in the macro. Otherwise, g++ -Wall -Wextra
complains that some fields are not initialized
[-Werror=missing-field-initializers]:
error: missing initializer for member ‘PySlot::sl_flags’
error: missing initializer for member ‘PySlot::<anonymous>’
error: missing initializer for member ‘PySlot::<anonymous>’
Replace ctypes with _testlimitedcapi. ctypes is not always available,
and _testlimitedcapi is a thin wrapper to the C API.
Skip the test if _testlimitedcapi is missing.
Make os.strerror() thread-safe: use the reentrant strerror_r()
function if available.
* The configure script now checks if strerror_r() is supported.
* Add a stress test to test_free_threading.test_os (new module).
* Add a comment on decode_current_locale() assertion which fails if
the input string is mutated.
* Add an assertion to _Py_DecodeLocale() to detect if the input
string was mutated during the function call.
Fix the compiler warning "extension used" when building test_cext
with clang.
* test_cext no longer overrides the CC environment variable if it's
already defined.
* test_cext now also logs __STRICT_ANSI__ macro.
Soft deprecate PyUnicode_New(), PyUnicode_CopyCharacters(),
PyUnicode_Fill(), PyUnicode_Resize(), PyUnicode_WRITE() and
PyUnicode_WriteChar() functions. Use the PyUnicodeWriter API instead.
* Add tests on PyUnicode_New() and PyUnicode_Resize(). Check that the
result is either a mutable string or the empty string singleton.
* Check that PyUnicode_Fill(), PyUnicode_CopyCharacters() and
PyUnicode_WriteChar() fail to modify a string with 2 references.
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
PyBytesWriter_Finish() doesn't need to check the size. It's known to
be valid.
PyBytesWriter_Create() now only calls byteswriter_resize() if the
size is larger than the small buffer.
Remove byteswriter_data() function: call static inline
_PyBytesWriter_GetData() function directly.
Add output_length to _Py_EncodeLocale(), _Py_EncodeUTF8(),
encode_current_locale() and encode_ascii(). So unicode_encode_locale() and
wstr_to_utf8() can now use the output_length, instead of having to compute
strlen().
* Rename functions:
* _Py_DecodeUTF8Ex() => _Py_DecodeUTF8()
* _Py_EncodeUTF8Ex() => _Py_EncodeUTF8()
* _Py_EncodeLocaleEx() => _Py_EncodeLocale()
* _Py_DecodeLocaleEx() => _Py_DecodeLocale()
* Add tests on Py_DecodeLocale() and Py_EncodeLocale() in test_codecs.
Clarify also their documentation.
* _Py_EncodeLocale() and _Py_DecodeLocale() now sets all output parameters on
error. Add more tests on these output parameters.
* Add encode_current_locale_impl() to simplify encode_current_locale().
* Remove reason parameter of _Py_EncodeLocale() and _Py_DecodeLocale() to
simplify the code. Only the UTF-8 decoder used specific reasons.
* _Py_EncodeUTF8() now uses size_t instead of Py_ssize_t to iterate
on the input string.
* On error, encode_current_locale_impl() now returns ENCODE_ERROR (new
constant) instead of DECODE_ERROR.
* Mark _Py_EncodeLocaleRaw() as static (it's not used outside fileutils.c).
Add bytes_alloc() helper function. bytes_alloc() and
_PyBytes_FromStringAndSize() have less conditional branches than
previous code, and so should be a little bit faster.
* Rename existing bytes_alloc() to bytes_type_alloc().
* Replace _PyBytes_FromSize(calloc=1) with _PyBytes_FromSizeZero().
* Add set_ob_shash_unsafe(): similar to set_ob_shash() but don't use
an atomic operation on Free Threading. Use this new function
on newly allocated bytes objects and in bytes_resize_inplace().
* Replace PyBytes_FromStringAndSize(NULL, size) with
bytes_alloc(size).
* Use bytes_alloc() in PyBytes_FromString() and _PyBytes_Repeat().
* Add _PyBytes_FromStringAndSize(): similar to
PyBytes_FromStringAndSize(), but str must not be NULL.
* Replace PyBytes_FromStringAndSize() with
_PyBytes_FromStringAndSize().
* Test that PyBytes_FromString() and PyBytes_FromStringAndSize()
return singletons for 0 or 1 bytes.
* Fix test_tracemalloc.test_set_traceback_limit().
_PyBytes_Resize() now creates a new bytes object if the hash value
was already computed.
bytes_resize_inplace() no longer sets the hash value to -1, since
this function must not be called if the hash value was already
computed.
Add tests on PyUnicode_Resize().
Py_GetVersion() now allocates memory on the heap, instead of using a
static buffer, to no longer truncate the version if it's longer than
299 bytes.
Update Py_GetCompiler() and Py_GetBuildInfo() tests: they are now
always a part of sys.version.
If Python is built in release mode with assertions,
PyUnicode_FromKindAndData() and PyUnicodeWriter_Finish() fail with an
assertion error on invalid characters.
Add specialized bytes_resize_inplace() and inline
PyBytesWriter_Discard() in PyBytesWriter_FinishWithSize().
No longer copy the buffer if _PyObject_IsUniquelyReferenced() is false.
Add also bytes_get_char() function.
Replace global variables with a new _PyOS_GetOpt structure:
* _PyOS_opterr => getopt->error
* _PyOS_optind => getopt->index
* _PyOS_optarg => getopt->arg
Changes:
* Add tests to test_cmd_line.
* Rename _PyOS_ResetGetOpt() to _PyOS_GetOpt_Init(). Add argc and
argv parameters.
* Don't expose longindex in the structure, it's only used internally.
* Move _PyOS_LongOption structure from the internal C API to
getopt.c.
In debug mode, PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND) and
PyUnicodeWriter_WriteUCS4() now raise a SystemError if a character is
not in range [U+0000; U+10ffff], instead of creating an invalid str
object.
In release mode, PyUnicode_FromKindAndData(PyUnicode_4BYTE_KIND)
raises a SystemError if the size is 1.
* In debug mode, unicode_char() no longer fails with an assertion
error if the character is invalid. Instead, raise SystemError.
* Add tests on invalid UCS-4 characters.
* Add _testinternalcapi._Py_MAX_UNICODE.
* Add unicode_invalid_character() helper function.
python-gdb.py is now able to format an invalid Unicode string:
render invalid characters as "\Uhhhhhhhh" instead of raising an
exception.
The PyUnicodeWriter API fills a UCS-4 buffer with 0xff byte pattern
to detect usage of uninitialized characters. It produces invalid
characters '\Uffffffff'.
PyBytesObjectPtr now iterates on bytes (list of integers), instead of
creating a temporary Unicode string. Add more bytes tests in test_gdb
pretty printer.
Remove the unused 'ch2' variable.
Add assertion to _PyUnicodeWriter_Prepare() to check that length is
not negative.
Add some assertions to check that _PyUnicodeWriter_Prepare() is
called with len >= 1.
Remove special case for int in PyUnicode_Format(). Instead, reuse
PyUnicodeWriter_WriteStr() and PyUnicodeWriter_WriteRepr() in
PyUnicode_Format().
Add PyUnicode_Format() tests on integer formats.
Enable the PyUnicodeWriter read-only optimization for large integer
(more than 1000 digits) in PyUnicodeWriter_WriteStr() and
PyUnicodeWriter_WriteRepr().
If PyUnicodeWriter_WriteUTF8() or PyUnicodeWriter_Format() fails,
PyUnicodeWriter_Finish() now checks if the string uses the most
efficient storage. Previously, PyUnicodeWriter_Finish() could return
a string with an inefficient storage.
* Add _PyUnicodeWriter.recheck_maxchar. PyUnicodeWriter_WriteUTF8()
and PyUnicodeWriter_Format() now set recheck_maxchar to 1 on error.
* Move error handling from caller to _PyUnicode_DecodeUTF8Writer().
* PyUnicodeWriter_DecodeUTF8Stateful(): rename length parameter to
size and rename string parameter to str.
Fix PyBytesWriter_Format() error handling: restore the writer to its
previous state on error. Add an unit test injecting MemoryError.
Fix PyBytesWriter_FinishWithSize() to return an empty bytearray when
use_bytearray is set and size is 0, instead of always returning an
empty bytes object.
Fix test_capi.test_bytes: always check writer finish result type, to
make sure that _PyBytesWriter_CreateByteArray() always create
bytearray objects.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Remove a spurious Py_DECREF(writer_type) in _PyTestCapi_Init_Bytes()
that over-decremented the reference after it was already released.
Also fix writer_discard method flags (METH_VARARGS -> METH_NOARGS),
a typo, and PEP 7 spacing in for-loop initializers.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Do no read output_start pointer after PyBytesWriter_Finish() since
the pointer became a dangling pointer.
Co-authored-by: Vyron Vasileiadis <hi@fedonman.com>
* Add _PyBytes_GET_CACHED_HASH() static inline function.
* _PyBytes_IsMutable() makes sure that the hash value is not cached
yet.
* bytearray_reinit_from_bytes() checks that the bytes object is
mutable.
Co-authored-by: Cody Maloney <cmaloney@theoreticalchaos.com>
On C++, do not use decltype() in _Py_TYPEOF since it produces invalid
code in Py_CLEAR() and Py_SETREF(). Instead, implement Py_CLEAR(),
Py_SETREF() and Py_XSETREF() using "auto" on C++11 and newer.
Add Py_CLEAR(), Py_SETREF() and Py_XSETREF() tests on an array.
test_cext now discovers and calls all "test" functions. On a debug
build, it also checks for reference leaks.
Fix a reference leak in test_virtual_object(): use a dealloc function
which calls Py_DECREF() on the type, and trigger an explicit garbage
collection.
* test_cext now uses a single virtual environment for all tests.
It makes the test way faster since creating a virtual environment
is quite slow.
* Add Py_TARGET_ABI3T test on C++.
* Move macro tests from module exec function to test_macros().
* Call Py_BEGIN_CRITICAL_SECTION() on a fresh dictionary object,
rather than on the module.
* Rename some environment variables to always use "CPYTHON_TEST_"
prefix.
* Replace support.MS_WINDOWS with MSVC to be more explicit.
* Move macro tests to test_macros().
* On macOS, use C++11 by default.
If a command line option wants to exit Python, move this option
processing from PyConfig_Read() to Py_Main(). So
Py_InitializeFromInitConfig() and Py_InitializeFromConfig() can no
longer return an exit code.
Deprecate PyInitConfig_GetExitCode(), PyStatus_Exit(), and PyStatus_IsExit()
functions, and PyStatus.exitcode member.
Renumber getopt.c long options to start at 1, instead of 0.
Keep 4 variables in the stable ABI:
* Py_FileSystemDefaultEncodeErrors
* Py_FileSystemDefaultEncoding
* Py_HasFileSystemDefaultEncoding
* Py_UTF8Mode
Mark the 4 variables as ABI only in Misc/stable_abi.toml.
Remove Py_SetProgramName(), Py_SetPythonHome(), PySys_SetArgv() and
PySys_SetArgvEx() functions, deprecated since Python 3.11.
Mark the 4 functions as abi_only in stable_abi.toml.
PyConfig_Read() and _PyPreConfig_Read() no longer read global config
variables (such as Py_BytesWarningFlag). Instead, PyConfig_Read() now
copies PyPreConfig members (isolated, use_environment and dev_mode).
_PyPreConfig_Read() still reads the last global configuration
variable: Py_UTF8Mode.
No longer use deprecated Py_IgnoreEnvironmentFlag in Py_GETENV().
Instead, use PyConfig.use_environment when Python is initialized.
Modify also PySys_SetArgv() to use PyConfig.isolated when Python is
initialized.
PyUnicodeWriter_Create(length) no longer allocates 'length'
characters immediately. The allocation of the buffer is now done
lazily at the first write, except if the read-only optimization is
used. So the read-only optimization can also be used even if length
is greater than 0.
No longer overallocate the first buffer (at the first write).
Only overallocate when the buffer is resized (at the second write).
It avoids the need to truncate in PyUnicodeWriter_Finish() when
PyUnicodeWriter_Create(length) used the exact output length.
Add get_buffer() method to writer tests.
configure.ac: Add PY_CHECK_FUNC_PRIVATE() function, similar to
PY_CHECK_FUNC() but defines macro _Py_HAVE_xxx instead of HAVE_xxx.
Use PY_CHECK_FUNC_PRIVATE() for functions: acospi, asinpi, atanpi,
atan2pi, cospi, sinpi and tanpi. Update check_capi_macros_ignored.txt.
Add Tools/build/check_capi_macros.py. Ignore existing macros which
don't start with "Py", only prevent adding new names which don't
respect this convention.
Run check_capi_macros.py in pre-commit.
Co-authored-by: Peter Bierma <zintensitydev@gmail.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
If the first and only write to a PyUnicodeWriter is a Python str
object, PyUnicodeWriter_Finish() returns the object unchanged.
PyUnicodeWriter_WriteChar() uses a singleton if no buffer was
allocated yet.
Move test_unicode_equal() to the correct test case (CAPITest).
PyBytesWriter_FinishWithSize() and PyUnicodeWriter_Finish() now
discard the output string instead of resizing it if the single
byte/character singleton is used.
Document also that PyUnicode_READ() and PyUnicode_READ_CHAR()
accept reading the trailing null character, and that PyUnicode_GetLength()
doesn't count the trailing null character.
Replace soft deprecated PyBytes_FromStringAndSize() with
PyBytesWriter.
Remove 'pseudo' parameter of PySSL_RAND(): it's no longer needed since
ssl.RAND_pseudo_bytes() has been removed (in 2022,
commit d435a18c53).
Add a test on ssl.RAND_bytes(0).
Replace soft deprecated PyBytes_FromStringAndSize() with
PyBytesWriter.
Replace also PyBytes_FromStringAndSize(NULL, 0) with
Py_GetConstant(Py_CONSTANT_EMPTY_BYTES) in other functions.
Replace also PyBytes_FromStringAndSize(NULL, 0) with
PyBytes_FromStringAndSize("", 0) in _Py_GetConstant_Init().
Add byteswriter_check_consistency() to check that the bytes/bytearray
object can be mutated. It is called by all PyBytesWriter functions.
* Add test_thread to PyBytesWriter tests.
* _PyBytes_IsMutable() now checks _PyObject_IsUniquelyReferenced().
* Clarify thread safety in PyBytesWriter documentation.
Check if the trailing null character has been modified to detect
buffer overflow in C extensions.
_PyUnicode_CheckConsistency() now always check if the trailing null
character has been overridden to detect buffer overflow. Previously,
it was only been checked if check_content parameter was non-zero.
Make sure that a writer can be modified before writing into it.
* Document that PyUnicodeWriter is not thread safe.
* Add singletons tests to test_capi.test_unicode.
* Check PyUnicode_CheckExact() earlier in _PyUnicode_IsModifiable().
Use free() instead of PyMem_RawFree() in pythread_wrapper() to avoid
a data race if another thread calls PyMem_SetAllocator() in parallel.
Reenable @nomemtest tests on TSAN.
Co-authored-by: Nathan Goldbaum <nathan.goldbaum@gmail.com>
In debug mode at Python exit, check if immutable singleton objects
have been modified by mistake to detect bugs in C extensions.
Add tests corrupting bytes, str, bool and int singleton objects.
Add pycore_global_objects_fini.h internal header file.
pycore_global_objects_fini_generated.h is now fully generated by the
script.
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Victor Stinner