* Docs: split builtins to their own page from library
* review feedback
* addressed Hugo's feedback
* update the What's Next page
* one more wording tweak
* move builtins to their own directory. fix the reference name
* moved pages need to be noted in tools/removed-ids.txt
* add Python to the builtins reference title
* add sphinxext-rediraffe for the library->builtins split
* update check-html-ids to handle rediraffe redirects
* now we don't need (page missing) for the redirected pages
* update other references to moved pages
* A seealso from builtins to library
* make a nice section for rediraffe settings
* move the builtins note to the end, as a seealso
* address merwok's comments
* cache looking for ids in files
* simplify the intro paragraphs
mkstringprep uses things like str.lower(), so it generates the wrong
result if run in an interpreter with a different Unicode data version
than the target.
This means that updating the Unicode version is a two-step process:
run makeunicodedata.py, then compile, then run mkstringprep.py.
The two steps can (and should) be combined when re-running
regen-unicodedata to verify that the data is up to date.
The GH-155292 fix only considered that case.
Change makeunicodedata.py to only run mkstringprep.py when the
current interpreter is up to it. Otherwise, show a reminder.
As an extra complication, download the input (RFC 3454) in
the "first step", since an out-of-date stringprep.py's freshness
assertion may prevent downloads.
Co-authored-by: Maurycy Pawłowski-Wieroński <maurycy@maurycy.com>
Due to a bug, some Unicode codepoint attributes were considered
for characters not yet defined in Unicode 3.2.0 or attributes
which changed in later Unicode versions. RFC 3454 (StringPrep)
requires using Unicode 3.2.0 strictly.
Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Add support for \p{property} and \P{property} escapes in Unicode (str)
regular expressions, for the properties the engine can resolve without
the unicodedata database. They are matched as CATEGORY opcodes or as
fixed sets of character ranges.
Supported in this change: many General_Category values (the groups L, N,
Z, C and the values Lu, Lt, Lm, Nd, Nl, No, Zs, Zl, Zp, Cc, Cf, Cs, Co
and Cn); the binary properties Alphabetic, Lowercase, Uppercase, Numeric,
Printable, XID_Start, XID_Continue, Cased and Case_Ignorable; the POSIX
compatibility classes; the code-point classes ASCII, Any, Assigned,
Noncharacter_Code_Point, Join_Control, Pattern_Syntax and
Pattern_White_Space; and Regional_Indicator, ASCII_Hex_Digit and
Hex_Digit.
Property and value names use loose matching (UAX #44 UAX44-LM3), so a
property may be spelled \p{Lu}, \p{gc=Lu} or \p{name=yes}.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Scan the nfc_first/nfc_last reindex tables comparing only .start, range-check
the candidate once, and terminate on a sentinel above every codepoint, so each
entry costs a single comparison. ~2x faster on non-Latin and combining-heavy
NFC/NFKC input; no new data tables.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Since the code for rule NR2 is already here, to support CJK unified
ideographs and Tangut ideographs, it can also be used for other names
derived by rule NR2.
Add the unicodedata.iter_graphemes() function to iterate over grapheme
clusters according to rules defined in Unicode Standard Annex #29.
Add unicodedata.grapheme_cluster_break(), unicodedata.indic_conjunct_break()
and unicodedata.extended_pictographic() functions to get the properties
of the character which are related to the above algorithm.
Co-authored-by: Guillaume "Vermeille" Sanchez <guillaume.v.sanchez@gmail.com>
Now the fields have names! Much easier to keep straight as a
reader than the elements of an 18-tuple.
Runs about 10-15% slower: from 10.8s to 12.3s, on my laptop.
Fortunately that's perfectly fine for this maintenance script.
Since PEP 393 in Python 3.3, this value is always 0x10ffff, the
maximum codepoint in Unicode; there's no longer such a thing as a
UCS-2 build of Python, which couldn't properly represent some
characters.
There are a couple of spots left where we still condition on the value
of this constant. Take them out.
Much like the lower-level logic in commit ef2af1ad4, we had
4 copies of this logic, written in a couple of different ways.
They're all implementing the same standard, so write it just once.
The `expand` option was introduced in 2000 in commit fad27aee1.
It appears to have been always set since it was committed, and
what it does is tell the code to do something essential. So,
just always do that, and cut the option.
Also cut the `linebreakprops` option, which isn't consulted anymore.
There were 10 copies of this, and almost as many distinct versions of
exactly how it was written. They're all implementing the same
standard. Pull them out to the top, so the more interesting logic
that remains becomes easier to read.
Benjamin Peterson