UnicodeNormalize.normalize and .normalized? read three constants that are
not shareable, so any call from a non-main Ractor raised
Ractor::IsolationError:
* NF_HASH_C / NF_HASH_D, the memoization caches. They cannot be made
shareable because a shareable Hash cannot be written to. Keep them per
Ractor instead: normalization is deterministic, so recomputing an entry
in another Ractor always gives the same string, and the cache is only a
cache. The DoS bound of MAX_HASH_LENGTH now applies per Ractor.
* UNICODE_ENCODINGS, which is splatted in a `when` clause on the path for
UTF-16, UTF-32, GB18030 and UCS-2/4BE. Freeze it.
The other constants are already shareable on master: tables.rb is
frozen_string_literal with frozen tables, and Regexp instances are frozen
since cb03bf43eb.
Keeping the caches keeps main Ractor performance: dropping them instead
costs 2.7x to 11x on real text, and asking the main Ractor for an entry
costs 19.4us per round trip against 3.8us to just compute it. The per
Ractor lookup adds about 60ns per call; normalized? fetches its cache
lazily so a string without a match does not pay it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Unicode Version 12.1.0 adds one single character, U+32FF SQUARE ERA NAME REIWA,
for the new Japanese era starting on May 1st. 12.1.0 will be finalized only on
May 7th, so we go with the beta version because further changes in the data we
need are highly unlikely, and we want to make sure Ruby is ready for the new era.
* common.mk: change UNICODE_VERSION to 12.1.0, UNICODE_BETA to YES
* enc/unicode/12.1.0, enc/unicode/12.1.0/casefold.h, enc/unicode/12.1.0/name2ctype.h:
add directory and generated data files for new version
* lib/unicode_normalize/tables.rb: update for new character
* test/ruby/test_regexp.rb: add test for character property age=12.1
* test/test_unicode_normalize.rb: add test for NFKC decomposition of new character
This (mostly) completes issue #15195.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@67441 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* lib/unicode_normalize/normalize.rb: Fix the range check for trailing
Hangul jamo characters in Unicode normalization. Different from
leading or vowel jamos, where LBASE and VBASE are actual characters,
a value equal to TBASE expresses the absence of a trailing jamo.
This fix is technically correct, but there was no bug because
the regular expressions in lib/unicode_normalize/tables.rb
eliminate jamos equal to TBASE from normalization processing.
* test/test_unicode_normalize.rb: Add preventive test
test_no_trailing_jamo based on
https://github.com/python/cpython/commit/d134809cd3764c6a634eab7bb8995e3e2eff14d5
just for the case we ever get a regression.
This closes issue #14934, thanks to MaLin (Lin Ma) for reporting.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64087 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/test_unicode_normalize.rb: extract tests that do not depend
on NormalizationTest.txt data file from conditionally constructed
part of TestUnicodeNormalize class, to always run them even if
the data file isn't found.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64086 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/test_unicode_normalize.rb: search ucd directory
first if it exists. This change follows r61415.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@64085 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
test/test_unicode_normalize.rb: Replace UNICODE_VERSION from
UnicodeNormalize with RbConfig::CONFIG['UNICODE_VERSION'] from
feature 12460.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55567 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/ruby/enc/test_case_comprehensive.rb: noting to test if
Unicode data files are available.
[ruby-core:76160] [Bug #12433]
* test/test_unicode_normalize.rb: ditto.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55521 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
Unicode data file; refactoring; fix an error with tests for destructive
method (unicode_normalize!).
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@55196 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
When you change this to true, you may need to add more tests.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@53141 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/test_unicode_normalize.rb (setup): skip only once, and
silently ignore other tests.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48388 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* test/test_unicode_normalize.rb (read_tests): do nothing if the
test data file is not found.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48387 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
* template/unicode_norm_gen.tmpl (UnicodeNormalize): embed the
version of Unicode data files used to generate.
* test/test_unicode_normalize.rb (TestUnicodeNormalize): use the
embedded version to load the test data.
git-svn-id: svn+ssh://ci.ruby-lang.org/ruby/trunk@48357 b2dd03c8-39d4-4d8f-98ff-823fe69b080e
