charset_loc_ignore() tested the whole set once per locale case and took
the disjunction, which complements a set before closing it under case
instead of after: [bc] matched b'B', but so did [^bc]. Match both cases
of the character against every set member instead.
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replace the state->repeat checks that guard mark saving and restoring
with an explicit save_marks counter maintained where repeat and
possessive contexts are entered and left. Each push is now paired
with a pop decided by the same condition.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Add support for \p{property} and \P{property} escapes in Unicode (str)
regular expressions, for the properties the engine can resolve without
the unicodedata database. They are matched as CATEGORY opcodes or as
fixed sets of character ranges.
Supported in this change: many General_Category values (the groups L, N,
Z, C and the values Lu, Lt, Lm, Nd, Nl, No, Zs, Zl, Zp, Cc, Cf, Cs, Co
and Cn); the binary properties Alphabetic, Lowercase, Uppercase, Numeric,
Printable, XID_Start, XID_Continue, Cased and Case_Ignorable; the POSIX
compatibility classes; the code-point classes ASCII, Any, Assigned,
Noncharacter_Code_Point, Join_Control, Pattern_Syntax and
Pattern_White_Space; and Regional_Indicator, ASCII_Hex_Digit and
Hex_Digit.
Property and value names use loose matching (UAX #44 UAX44-LM3), so a
property may be spelled \p{Lu}, \p{gc=Lu} or \p{name=yes}.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Character class escapes (``\d``, ``\D``, ``\s``, ``\S``, ``\w`` and
``\W``) that occur outside a character set are now compiled directly to a
single CATEGORY opcode instead of being wrapped in an IN block. This
removes the IN wrapper (three code words) and an indirect charset() call,
and makes such an escape a simple repeatable unit so that, for example,
``\d+`` uses the REPEAT_ONE fast path; a CATEGORY case is added to
SRE(count).
The transformation preserves behaviour exactly. For category-heavy
patterns the compiled byte code is about 20% smaller and matching is up
to ~2x faster, with no effect on patterns that do not use bare category
escapes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Replace all documentation which says "See PEP 585"
The following classes in the stdlib get simple updates:
- array.array
- asyncio.Future
- asyncio.Task
- collections.defaultdict
- collections.deque
- contextvars.ContextVar
- contextvars.Token
- ctypes.Array
- os.DirEntry
- re.Match
- re.Pattern
- string.templatelib.Interpolation
- string.templatelib.Template
- types.MappingProxyType
- queue.SimpleQueue
- weakref.ref
The following classes are documented publicly as functions, and are
therefore updated internally (`__class_getitem__.__doc__`) but not in the
public docs:
- functools.partial
- itertools.chain
The following builtin types have updates to `__class_getitem__.__doc__`
but not to any documentation pages:
- BaseExceptionGroup
- coroutines (from generators)
- dict
- enumerate
- frozendict
- frozenset
- generators (and async generators)
- list
- memoryview
- set
- slice
- tuple
Special cases:
- union objects are now documented as "supporting class-level []",
rather than anything to do with generics.
- Templates might be generic over a single type (union, in theory) or
over a TypeVarTuple. As this is not currently fully settled, it is
marked with a comment and a mild hint that it is a single type is used
(namely, "type" is singular rather than "types", plural)
* Apply suggestions from code review
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
* Correct several class getitem docs
And expand the text for tuples.
Co-authored-by: Jelle Zijlstra <906600+JelleZijlstra@users.noreply.github.com>
* Add notes on generic typing of builtins
* Fix typo in tuple.__class_getitem__ docstring
* Typo fix: malformed refs
Fix `generic` links which weren't marked as `:ref:`.
* Strike unnecessary docs on generic-ness
Co-authored-by: Jelle Zijlstra <906600+JelleZijlstra@users.noreply.github.com>
* Apply suggestions from code review
These are applied at both the originally indicated locations and in the
corresponding docstring definitions.
Co-authored-by: Alex Waygood <66076021+AlexWaygood@users.noreply.github.com>
* Update Doc/library/re.rst
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* Update Objects/enumobject.c
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
* Remove tuple generic doc in 'stdtypes' page
This is covered in more detail in the cross-linked typing documentation.
The other copy of this documentation -- in the docstring for
`tuple.__class_getitem__` -- is left in place.
* Fix whitespace around new doc of generics
Per review, do not introduce or remove whitespace such that section
breaks are altered by the introduction of doc on various generic types.
In most cases, this is a removal of an extra line.
In one case (Arrays), it is the reintroduction of a line.
Additionally, two other minor fixes are included:
- incorrect indent on 'defaultdicts'
- make `mappingproxy.__class_getitem__.__doc__` consistent with other
mapping type generic docs
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* Move placement of memoryview generic note
Previous placement was at the end of the main docstring, which is
consistent with other types but places it after a section on various
methods (which makes it read somewhat inconsistently). Moving it up
helps resolve.
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* Ensure sphinxdoc does not start sentences lowercase
Lowercase class names at the start of sentences are marked out with the
`class` role. In the case of `deque`, documentation already refers to
these as `Deques`, so this form is preferred.
* Apply suggestions from code review
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
* Fix line endings and wrap more tightly
Line endings fixed by pre-commit ; also re-wrapped the MappingProxyType
text which was too long.
* Use 'ContextVars' style in sphinx doc
---------
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Jelle Zijlstra <906600+JelleZijlstra@users.noreply.github.com>
Co-authored-by: Alex Waygood <66076021+AlexWaygood@users.noreply.github.com>
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
state_init() always did PyMem_New(state->mark, groups*2), which for a
pattern with no capturing groups is PyMem_Malloc(0) -- a real allocation
(plus matching free) on every match/search/fullmatch call, for an array
that is never read: groupless patterns emit no MARK opcodes and group 0's
span is taken from state->start/ptr.
Guard the allocation with `if (pattern->groups)`. state->mark stays NULL
(set by the preceding memset), and both the error path and state_fini
already PyMem_Free(NULL) safely.
Rename from _Py_INTERNAL_ABI_SLOT to _Py_ABI_SLOT
and define the macro using _PyABIInfo_DEFAULT.
Use the ABI slot in stdlib extension modules to enable running
a check of ABI version compatibility.
_tkinter, _tracemalloc and readline don't use the slots, hence they need
explicit handling.
Co-authored-by: Victor Stinner <vstinner@python.org>
Adds `prefixmatch` APIs to the re module as an alternate name for our long existing `match` APIs to help alleviate a common Python confusion for those coming from other languages regular expression libraries.
These alleviate common confusion around what "match" means as Python is different than other popular languages regex libraries in our use of the term as an API name. The original `match` names are **NOT being deprecated**. Source tooling like linters, IDEs, and LLMs could suggest using `prefixmatch` instead of match to improve code health and reduce cognitive burden of understanding the intent of code when configured for a modern minimum Python version.
See the documentation changes for a better description.
Discussions took place in the PR, in the issue, and finally at https://discuss.python.org/t/add-re-prefixmatch-deprecate-re-match/105927
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Shantanu <12621235+hauntsaninja@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Added atomic operations to `scanner_begin()` and `scanner_end()` to prevent
race conditions on the `executing` flag in free-threaded builds. Also added
tests for concurrent usage of the `re` module.
Without the atomic operations, `test_scanner_concurrent_access()` triggers
`assert(self->executing)` failures, or a thread sanitizer run emits errors.
If SRE(match) function terminates abruptly, either because of a signal
or because memory allocation fails, allocated SRE_REPEAT blocks might
be never released.
Co-authored-by: <wjssz@users.noreply.github.com>
This makes the following macros public as part of the non-limited C-API for
locking a single object or two objects at once.
* `Py_BEGIN_CRITICAL_SECTION(op)` / `Py_END_CRITICAL_SECTION()`
* `Py_BEGIN_CRITICAL_SECTION2(a, b)` / `Py_END_CRITICAL_SECTION2()`
The supporting functions and structs used by the macros are also exposed for
cases where C macros are not available.
This PR adds the ability to enable the GIL if it was disabled at
interpreter startup, and modifies the multi-phase module initialization
path to enable the GIL when loading a module, unless that module's spec
includes a slot indicating it can run safely without the GIL.
PEP 703 called the constant for the slot `Py_mod_gil_not_used`; I went
with `Py_MOD_GIL_NOT_USED` for consistency with gh-104148.
A warning will be issued up to once per interpreter for the first
GIL-using module that is loaded. If `-v` is given, a shorter message
will be printed to stderr every time a GIL-using module is loaded
(including the first one that issues a warning).
Starts adding thread safety to dict objects.
Use @critical_section for APIs which are exposed via argument clinic and don't directly correlate with a public C API which needs to acquire the lock
Use a _lock_held suffix for keeping changes to complicated functions simple and just wrapping them with a critical section
Acquire and release the lock in an existing function where it won't be overly disruptive to the existing logic
Move the following private functions and structures to
pycore_modsupport.h internal C API:
* _PyArg_BadArgument()
* _PyArg_CheckPositional()
* _PyArg_NoKeywords()
* _PyArg_NoPositional()
* _PyArg_ParseStack()
* _PyArg_ParseStackAndKeywords()
* _PyArg_Parser structure
* _PyArg_UnpackKeywords()
* _PyArg_UnpackKeywordsWithVararg()
* _PyArg_UnpackStack()
* _Py_ANY_VARARGS()
Changes:
* Python/getargs.h now includes pycore_modsupport.h to export
functions.
* clinic.py now adds pycore_modsupport.h when one of these functions
is used.
* Add pycore_modsupport.h includes when a C extension uses one of
these functions.
* Define Py_BUILD_CORE_MODULE in C extensions which now include
directly or indirectly (via code generated by Argument Clinic)
pycore_modsupport.h:
* _csv
* _curses_panel
* _dbm
* _gdbm
* _multiprocessing.posixshmem
* _sqlite.row
* _statistics
* grp
* resource
* syslog
* _testcapi: bad_get() no longer uses METH_FASTCALL calling
convention but METH_VARARGS. Replace _PyArg_UnpackStack() with
PyArg_ParseTuple().
* _testcapi: add PYTESTCAPI_NEED_INTERNAL_API macro which is defined
by _testcapi sub-modules which need the internal C API
(pycore_modsupport.h): exceptions.c, float.c, vectorcall.c,
watchers.c.
* Remove Include/cpython/modsupport.h header file.
Include/modsupport.h no longer includes the removed header file.
* Fix mypy clinic.py
Now re.error is raised instead of OverflowError or RuntimeError for
too large width of look-behind pattern.
The limit is increased to 2**32-1 (was 2**31-1).
Remove <ctype.h> in C files which don't use it; only sre.c and
_decimal.c still use it.
Remove _PY_PORT_CTYPE_UTF8_ISSUE code from pyport.h:
* Code added by commit b5047fd019
in 2004 for MacOSX and FreeBSD.
* Test removed by commit 52ddaefb6b
in 2007, since Python str type now uses locale independent
functions like Py_ISALPHA() and Py_TOLOWER() and the Unicode
database.
Modules/_sre/sre.c replaces _PY_PORT_CTYPE_UTF8_ISSUE with new
functions: sre_isalnum(), sre_tolower(), sre_toupper().
Remove unused includes:
* _localemodule.c: remove <stdio.h>.
* getargs.c: remove <float.h>.
* dynload_win.c: remove <direct.h>, it no longer calls _getcwd()
since commit fb1f68ed7c (in 2001).
Argument Clinic now only includes pycore_gc.h if PyGC_Head is needed,
and only includes pycore_runtime.h if _Py_ID() is needed.
* Add 'condition' optional argument to Clinic.add_include().
* deprecate_keyword_use() includes pycore_runtime.h when using
the _PyID() function.
* Fix rendering of includes: comments start at the column 35.
* Mark PC/clinic/_wmimodule.cpp.h and
"Objects/stringlib/clinic/*.h.h" header files as generated in
.gitattributes.
Effects:
* 42 header files generated by AC no longer include the internal C
API, instead of 4 header files before. For example,
Modules/clinic/_abc.c.h no longer includes the internal C API.
* Fix _testclinic_depr.c.h: it now always includes pycore_runtime.h
to get _Py_ID().
Move these private functions to the internal C API
(pycore_abstract.h):
* _Py_convert_optional_to_ssize_t()
* _PyNumber_Index()
Argument Clinic now emits #include "pycore_abstract.h" when these
functions are used.
The parser of the c-analyzer tool now uses a list of files which use
the limited C API, rather than a list of files using the internal C
API.
Serhiy Storchaka