iseq.h does not reference any of the pm_* types or functions declared in
prism_compile.h, so include the latter from the files that actually use
it instead. Only 6 of the 19 files including iseq.h need the prism
declarations, so this keeps prism/prism.h out of the include closure of
ast.c, box.c, builtin.c, cont.c, eval.c, jit.c, proc.c, thread.c,
vm_dump.c, vm_trace.c, yjit.c, zjit.c and internal/jit.h.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The file, dir, load and ruby.c changes go together because rb_get_expanded_load_path() returns the maximum path length through a pointer. Lengths passed to the Windows string conversion helpers that take `long *` stay `long`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The test TestRequire#test_loading_fifo_threading_raise fails with the
following error in LSAN because pm_load_parse_file could raise which
will cause the pm_parse_result_t to leak.
Direct leak of 104 byte(s) in 1 object(s) allocated from:
#0 0x64ee1a50385d in calloc build-llvm/tools/clang/stage2-bins/runtimes/runtimes-bins/compiler-rt/lib/asan/asan_malloc_linux.cpp:74:3
#1 0x64ee1a578206 in calloc1 gc/default/default.c:2000:12
#2 0x64ee1a578206 in rb_gc_impl_calloc gc/default/default.c:11069:5
#3 0x64ee1a578206 in ruby_xcalloc_body gc.c:6146:12
#4 0x64ee1a578206 in ruby_xcalloc gc.c:6140:34
#5 0x64ee1acfaed1 in pm_parse_result_init prism_compile.c:10639:23
#6 0x64ee1a67bbd1 in load_iseq_eval load.c:748:13
#7 0x64ee1a674cd4 in rb_load_internal load.c:866:9
#8 0x64ee1a6754d6 in rb_load_entrypoint load.c:908:5
#9 0x64ee1a6790a5 in rb_f_load load.c:950:12
#10 0x64ee1a97043b in vm_call_cfunc_with_frame_ vm_insnhelper.c:3899:11
#11 0x64ee1a957d0b in vm_call_method_each_type vm_insnhelper.c:4891:16
#12 0x64ee1a95779b in vm_call_method vm_insnhelper.c
#13 0x64ee1a91a9ad in vm_sendish vm_insnhelper.c:6213:15
#14 0x64ee1a91a9ad in vm_exec_core insns.def:909:11
#15 0x64ee1a9083bd in rb_vm_exec vm.c:2875:22
#16 0x64ee1a54bccc in rb_ec_exec_node eval.c:299:9
#17 0x64ee1a54b993 in ruby_run_node eval.c:337:30
#18 0x64ee1a547f50 in rb_main main.c:42:12
#19 0x64ee1a547f50 in main main.c:62:12
#20 0x7f8507a2a1c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#21 0x7f8507a2a28a in __libc_start_main csu/../csu/libc-start.c:360:3
#22 0x64ee1a45fb74 in _start (ruby+0x197b74) (BuildId: 5b18a734908cec0a5f93267f1f350850b0f3fb6d)
Direct leak of 16 byte(s) in 1 object(s) allocated from:
#0 0x64ee1a50385d in calloc build-llvm/tools/clang/stage2-bins/runtimes/runtimes-bins/compiler-rt/lib/asan/asan_malloc_linux.cpp:74:3
#1 0x64ee1a578206 in calloc1 gc/default/default.c:2000:12
#2 0x64ee1a578206 in rb_gc_impl_calloc gc/default/default.c:11069:5
#3 0x64ee1a578206 in ruby_xcalloc_body gc.c:6146:12
#4 0x64ee1a578206 in ruby_xcalloc gc.c:6140:34
#5 0x64ee1acfaeb9 in pm_parse_result_init prism_compile.c:10638:21
#6 0x64ee1a67bbd1 in load_iseq_eval load.c:748:13
#7 0x64ee1a674cd4 in rb_load_internal load.c:866:9
#8 0x64ee1a6754d6 in rb_load_entrypoint load.c:908:5
#9 0x64ee1a6790a5 in rb_f_load load.c:950:12
#10 0x64ee1a97043b in vm_call_cfunc_with_frame_ vm_insnhelper.c:3899:11
#11 0x64ee1a957d0b in vm_call_method_each_type vm_insnhelper.c:4891:16
#12 0x64ee1a95779b in vm_call_method vm_insnhelper.c
#13 0x64ee1a91a9ad in vm_sendish vm_insnhelper.c:6213:15
#14 0x64ee1a91a9ad in vm_exec_core insns.def:909:11
#15 0x64ee1a9083bd in rb_vm_exec vm.c:2875:22
#16 0x64ee1a54bccc in rb_ec_exec_node eval.c:299:9
#17 0x64ee1a54b993 in ruby_run_node eval.c:337:30
#18 0x64ee1a547f50 in rb_main main.c:42:12
#19 0x64ee1a547f50 in main main.c:62:12
#20 0x7f8507a2a1c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#21 0x7f8507a2a28a in __libc_start_main csu/../csu/libc-start.c:360:3
#22 0x64ee1a45fb74 in _start (ruby+0x197b74) (BuildId: 5b18a734908cec0a5f93267f1f350850b0f3fb6d)
Flattening the full source path into a single file name made the copy
fail with ENAMETOOLONG when the extension lives under a deep path such
as a long BUNDLE_PATH, since the name exceeds NAME_MAX. Name each copy
with a short per-process sequence instead, in a 0700 directory with an
unpredictable name so that other local users cannot occupy the copy
destination in a shared TMPDIR.
https://bugs.ruby-lang.org/issues/22110
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
box_entry_free could run before other T_DATA objects are finalized in
rb_objspace_call_finalizer, and FreeLibrary there unmapped the
rb_data_type_t and dfree functions that later finalization still
dereferences, crashing any boxed process that leaves extension objects
alive at exit. Keep the copied DLLs loaded until ruby_vm_destruct, then
unload and delete them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
It generally end up frozen in `$LOADED_FEATURES` so it's preferable
to predict its size and only deal with an embedded string most of
the time.
It saves on having to allocate a `MAXPATHLEN + 2` (1026) bytes long
buffer and then realloc it later to shink it.
`rb_load_entrypoint` is used only by `rb_box_load`; it checks that the
argument has two-elements (implicitly assuming it is an array),
extracts its contents, and then simply discards the argument.
C23 has qualifier-preserving standard library functions, so calling
strchr with a `const char *` will return a `const char *`. We can change
the type of the local variables because we don't mutate the strings.
Prism changed structure quite a bit. Most of the previously-public
structs are now opaque. This requires quite a bit of changes
internally.
It also triggered some unrelated changes, which were necessary
because Prism's main header used to pull in standard headers. So
box.c and jit.c are now no longer transitively getting all the
headers they needed.
Top level constants are defined on the Object class's constant table,
and those constants can be referred as box::CONST_NAME from outside box.
So load_wrapping() is not needed now.
When multiple files with the same name are required, the features_index
hash stores the indexes in `$LOADED_FEATURES` array into a darray.
The dest and src arguments for `MEMMOVE` were wrongly reversed when
inserting a new index in the darray.
[Bug #21568]
to fix inconsistent and wrong current namespace detections.
This includes:
* Moving load_path and related things from rb_vm_t to rb_namespace_t to simplify
accessing those values via namespace (instead of accessing either vm or ns)
* Initializing root_namespace earlier and consolidate builtin_namespace into root_namespace
* Adding VM_FRAME_FLAG_NS_REQUIRE for checkpoints to detect a namespace to load/require files
* Removing implicit refinements in the root namespace which was used to determine
the namespace to be loaded (replaced by VM_FRAME_FLAG_NS_REQUIRE)
* Removing namespaces from rb_proc_t because its namespace can be identified by lexical context
* Starting to use ep[VM_ENV_DATA_INDEX_SPECVAL] to store the current namespace when
the frame type is MAGIC_TOP or MAGIC_CLASS (block handlers don't exist in this case)
The variable `prev_ext_config` is modified by `ext_config_push` between
`setjmp` and `longjmp` calls. Since `ext_config_push` and `ext_config_pop`
are small and likely to be inlined, `prev_ext_config` can be allocated on
a register and get clobbered. Fix by making it `volatile`.
This bug can be observed by adding a check for values greater than 1 in
`th2->ext_config.ractor_safe` after `ext_config_pop` and building with Clang.
None of the datastructures involved in the require process are
safe to call on a secondary ractor, however when autoloading
encodings, we do so from the current ractor.
So all sorts of corruption can happen when using an autoloaded
encoding for the first time from a secondary ractor.
Ref: https://github.com/fxn/zeitwerk/pull/308
```ruby
require 'benchmark'
$LOAD_PATH << 'relative-path'
autoload :FOO, '/tmp/foo.rb'
puts Benchmark.realtime {
500_000.times do
Object.autoload?(:FOO)
end
}
```
The above script takes 2.5 seconds on `master`, and only
50ms on this branch.
When we're looking for a feature with an absolute path, we don't
need to call the expensive `get_expanded_load_path`.
When searching for native extensions, if the name does not end in ".so"
then we create a new string and append ".so" so it. If the native extension
is in static_ext_inits, then we could trigger a GC in the rb_filesystem_str_new_cstr.
This could cuase the GC to free lookup_name since we don't use the local
variable anymore.
This bug was caught in this ASAN build:
http://ci.rvm.jp/results/trunk_asan@ruby-sp1/5479182
==435614==ERROR: AddressSanitizer: use-after-poison on address 0x715a63022da0 at pc 0x5e7463873e4e bp 0x7fff383c8b00 sp 0x7fff383c82c0
READ of size 14 at 0x715a63022da0 thread T0
#0 0x5e7463873e4d in __asan_memcpy (/tmp/ruby/build/trunk_asan/ruby+0x214e4d) (BuildId: 607411c0626a2f66b4c20c02179b346aace20898)
#1 0x5e7463b50a82 in memcpy /usr/include/x86_64-linux-gnu/bits/string_fortified.h:29:10
#2 0x5e7463b50a82 in ruby_nonempty_memcpy /tmp/ruby/src/trunk_asan/include/ruby/internal/memory.h:671:16
#3 0x5e7463b50a82 in str_enc_new /tmp/ruby/src/trunk_asan/string.c:1035:9
#4 0x5e74639b97dd in search_required /tmp/ruby/src/trunk_asan/load.c:1126:21
#5 0x5e74639b97dd in require_internal /tmp/ruby/src/trunk_asan/load.c:1274:17
#6 0x5e74639b83c1 in rb_require_string_internal /tmp/ruby/src/trunk_asan/load.c:1401:22
#7 0x5e74639b83c1 in rb_require_string /tmp/ruby/src/trunk_asan/load.c:1387:12
If there's a syntax error during iseq compilation then prism would leak
memory because it would not free the pm_parse_result_t.
This commit changes pm_iseq_new_with_opt to have a rb_protect to catch
when an error is raised, and return NULL and set error_state to a value
that can be raised by calling rb_jump_tag after memory has been freed.
For example:
10.times do
10_000.times do
eval("/[/=~s")
rescue SyntaxError
end
puts `ps -o rss= -p #{$$}`
end
Before:
39280
68736
99232
128864
158896
188208
217344
246304
275376
304592
After:
12192
13200
14256
14848
16000
16000
16000
16064
17232
17952
Many libraries should be loaded on the main ractor because of
setting constants with unshareable objects and so on.
This patch allows to call `requore` on non-main Ractors by
asking the main ractor to call `require` on it. The calling ractor
waits for the result of `require` from the main ractor.
If the `require` call failed with some reasons, an exception
objects will be deliverred from the main ractor to the calling ractor
if it is copy-able.
Same on `require_relative` and `require` by `autoload`.
Now `Ractor.new{pp obj}` works well (the first call of `pp` requires
`pp` library implicitly).
[Feature #20627]
While working on a separate issue we found that in some cases
`ary_heap_realloc` was being called on frozen arrays. To fix this, this
change does the following:
1) Updates `rb_ary_freeze` to assert the type is an array, return if
already frozen, and shrink the capacity if it is not embedded, shared
or a shared root.
2) Replaces `rb_obj_freeze` with `rb_ary_freeze` when the object is
always an array.
3) In `ary_heap_realloc`, ensure the new capa is set with
`ARY_SET_CAPA`. Previously the change in capa was not set.
4) Adds an assertion to `ary_heap_realloc` that the array is not frozen.
Some of this work was originally done in
https://github.com/ruby/ruby/pull/2640, referencing this issue
https://bugs.ruby-lang.org/issues/16291. There didn't appear to be any
objections to this PR, it appears to have simply lost traction.
The original PR made changes to arrays and strings at the same time,
this PR only does arrays. Also it was old enough that rather than revive
that branch I've made a new one. I added Lourens as co-author in addtion
to Aaron who helped me with this patch.
The original PR made this change for performance reasons, and while
that's still true for this PR, the goal of this PR is to avoid
calling `ary_heap_realloc` on frozen arrays. The capacity should be
shrunk _before_ the array is frozen, not after.
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Co-Authored-By: methodmissing <lourens@methodmissing.com>
This patch removes the `VALUE flags` member from the `rb_ast_t` structure making `rb_ast_t` no longer an IMEMO object.
## Background
We are trying to make the Ruby parser generated from parse.y a universal parser that can be used by other implementations such as mruby.
To achieve this, it is necessary to exclude VALUE and IMEMO from parse.y, AST, and NODE.
## Summary (file by file)
- `rubyparser.h`
- Remove the `VALUE flags` member from `rb_ast_t`
- `ruby_parser.c` and `internal/ruby_parser.h`
- Use TypedData_Make_Struct VALUE which wraps `rb_ast_t` `in ast_alloc()` so that GC can manage it
- You can retrieve `rb_ast_t` from the VALUE by `rb_ruby_ast_data_get()`
- Change the return type of `rb_parser_compile_XXXX()` functions from `rb_ast_t *` to `VALUE`
- rb_ruby_ast_new() which internally `calls ast_alloc()` is to create VALUE vast outside ruby_parser.c
- `iseq.c` and `vm_core.h`
- Amend the first parameter of `rb_iseq_new_XXXX()` functions from `rb_ast_body_t *` to `VALUE`
- This keeps the VALUE of AST on the machine stack to prevent being removed by GC
- `ast.c`
- Almost all change is replacement `rb_ast_t *ast` with `VALUE vast` (sorry for the big diff)
- Fix `node_memsize()`
- Now it includes `rb_ast_local_table_link`, `tokens` and script_lines
- `compile.c`, `load.c`, `node.c`, `parse.y`, `proc.c`, `ruby.c`, `template/prelude.c.tmpl`, `vm.c` and `vm_eval.c`
- Follow-up due to the above changes
- `imemo.{c|h}`
- If an object with `imemo_ast` appears, considers it a bug
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
Because Kernel#autoload? uses the current namespace, it can lead to
potentially confusing results. We should make it clearer that modules
count as separate namespaces to lookup in.
Co-authored-by: Jeremy Evans <code@jeremyevans.net>
Co-authored-by: Nobuyoshi Nakada <nobu@ruby-lang.org>
