These still held String and Array lengths and indices in long, or cast them to long. rb_len_t is still long everywhere, so the generated code does not change.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once rb_len_t widens this changes the `long *` output parameters of rb_range_beg_len(), rb_str_subpos(), rb_arithmetic_sequence_beg_len_step(), ruby_enc_find_basename() and ruby_enc_find_extname(), so extensions passing `long *` to them will need to be updated. The extension guides list these signatures verbatim and follow along.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
[Bug #22330]
The following script demonstrates a use-afer-free where we see corruption:
s = "あ" * 10_000
s.encode("US-ASCII", fallback: proc { |c| s.clear; "?" })
Raises may different errors such as:
"\xCB" followed by "\xF5" on UTF-8 (Encoding::InvalidByteSequenceError)
"\xF4" followed by "r" on UTF-8 (Encoding::InvalidByteSequenceError)
"\x8B" on UTF-8 (Encoding::InvalidByteSequenceError)
rb_econv_insert_output() grew the insertion buffer with
ruby_xrealloc_sized(*buf_start_p, s, buf_end_p - buf_start_p);
The old-size argument subtracts the two local pointer variables instead
of the pointers they point at, so the allocator's size accounting gets a
small constant rather than the buffer's size. Present since the sized
variant was introduced; the memory itself was never corrupted.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
transcode_loop() passed the fallback result to rb_econv_insert_output()
as a bare pointer and length. Nothing roots the string during that call:
args.rep still holds the pre-fallback error-bytes string, and with
clang -O3 the VALUE itself lives in no stack slot the conservative scan
can see -- only the raw data pointer survives in a register, and for an
embedded string that is an interior pointer the scanner rejects.
rb_econv_insert_output() can run a GC: when the fallback string's
encoding differs from the converter's insert encoding it calls
allocate_converted_string(), which opens a fresh econv and allocates the
destination buffer. The fallback string is then swept mid-call and the
sub-conversion reads its freed bytes.
On an ASAN build this reports use-after-poison at the 1-byte input read
in transcode_restartable0(); CI hit it about once in two hundred runs of
test_fallback_proc, and GC.stress reproduces it in a few hundred
iterations on a clang -O3 ASAN build. With RB_GC_GUARD the same loop is
clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Set RUBY_TYPED_THREAD_SAFE_FREE on TypedData types whose dfree function
is trivially safe, and only frees its own memory
Types not yet marked as THREAD_SAFE_FREE:
- id2ref_tbl_type: sets the process-global id2ref_tbl to NULL.
- mutex_data_type: unlinks itself from another thread's keeping_mutexes list.
- autoload_data_type: deletes nodes from a shared intrusive list.
- autoload_const_type: deletes itself from a shared list.
- rb_cont_data_type: mutates the shared fiber pool and the global first_jit_cont list.
- rb_fiber_data_type: delegates to cont_free.
- FiberPoolDataType: mutates shared fiber pool free-lists.
- ractor_data_type: reconfigures global VM event-hook flags and call caches.
- exported_object_registry: frees a global table under the VM lock.
- rb_box_data_type (Box::Entry): unlinks classext from other live class and module objects.
- box_ext_cleanup_type: dereferences another String object and calls unlink.
- monitor_data_type: uses the default free, so the flag has no effect.
Co-authored-by: Luke Gruber <luke.gruber@shopify.com>
rb_econv_close freed ec->replacement_str using replacement_len, but when
the replacement is converted to the destination encoding its allocated
size is replacement_bufsize, which can be larger. Free it with bufsize,
as rb_econv_set_replacement already does.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When the fallback function in transcode_loop raises, it will leak the memory
in rb_econv_t. The following script reproduces the leak:
10.times do
100_000.times do
"\ufffd".encode(Encoding::US_ASCII, fallback: proc { raise })
rescue
end
puts `ps -o rss= -p #{$$}`
end
Before:
451196
889980
1328508
1767676
2206460
2645372
3083900
3522428
3960956
4399484
After:
12508
12636
12892
12892
13148
13404
13532
13788
13916
13916
Make sure VM lock is not held when calling `load_transcoder_entry`, as
that causes deadlock inside ractors. `String#encode` now works inside
ractors, among others.
Atomic load the rb_encoding_list
Without this, wbcheck would sometimes hit a missing write barrier.
Co-authored-by: John Hawthorn <john.hawthorn@shopify.com>
Hold VM lock when iterating over global_enc_table.names
This st_table can be inserted into at runtime when autoloading
encodings.
minor optimization when calling Encoding.list
Ex: `str.encode` and `str.encode!` work across ractors now.
The global table `transcoder_table` needs a lock around each st_lookup/st_insert, and it's a two-level
table so the second level also needs to be locked around insertion/deletion. In addition to this, the
transcoder entries (values in the second-level hash table) need to be locked around retrieving them and
loading them as they are loaded lazily.
The transcoding objects (`Encoding::Converter`) can't be made shareable, so their operations don't need to be locked.
Delete duplicated `Document-const:` directives
Those directives are attached to `rb_define_const` and `rb_file_const` where we don’t need them because the functions are supported by RDoc.
when the RUBY_FREE_ON_SHUTDOWN environment variable is set, manually free memory at shutdown.
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Co-authored-by: Peter Zhu <peter@peterzhu.ca>
According to the C99 specification section 7.20.3.2 paragraph 2:
> If ptr is a null pointer, no action occurs.
So we do not need to check that the pointer is a null pointer.
Remove !USE_RVARGC code
[Feature #19579]
The Variable Width Allocation feature was turned on by default in Ruby
3.2. Since then, we haven't received bug reports or backports to the
non-Variable Width Allocation code paths, so we assume that nobody is
using it. We also don't plan on maintaining the non-Variable Width
Allocation code, so we are going to remove it.
Previously, newline: :lf was accepted but ignored. Where it
should have been used was commented out code that didn't work,
but unlike all other invalid values, using newline: :lf did
not raise an error.
This adds support for newline: :lf and :lf_newline, for consistency
with newline: :cr and :cr_newline. This is basically the same as
universal_newline, except that it only affects writing and not
reading due to RUBY_ECONV_NEWLINE_DECORATOR_WRITE_MASK.
Add tests for the File.open :newline option while here.
Fixes [Bug #12436]
Instead of relying on setting an unsetting ruby_verbose, which is
not thread-safe, restructure require_internal and load_lock to
accept a warn argument for whether to warn, and add
rb_require_internal_silent to require without warnings. Use
rb_require_internal_silent when loading encoding.
Note this does not modify ruby_debug and errinfo handling, those
remain thread-unsafe.
Also silent requires when loading transcoders.
When using a non-ASCII compatible source and destination encoding
and xml escaping (the :xml option to String#encode), the resulting
string was broken, as it used the correct non-ASCII compatible
encoding, but contained data that was ASCII-compatible instead of
compatible with the string's encoding.
Work around this issue by detecting the case where both the
source and destination encoding are non-ASCII compatible, and
transcoding the source string from the non-ASCII compatible
encoding to UTF-8. The xml escaping code will correctly handle
the UTF-8 source string and the return the correctly encoded
and escaped value.
Fixes [Bug #12052]
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Has been deprecated since 684bdf6171.
Matz says in [ruby-core:83954] that Data should be an alias of Object.
Because rb_cData has not been deprecated, let us deprecate the constant
to make it a C-level synonym of rb_cObject.
RARRAY_AREF has been a macro for reasons. We might not be able to
change that for public APIs, but why not relax the situation internally
to make it an inline function.
Nobuyoshi Nakada