[PATCH] [Bug #22191] Report `EXIT` handler exceptions after `END` blocks
Keep exceptions raised by `EXIT` traps until `END` handlers finish.
That leaves them available for reporting during cleanup.
Print `END` handler exceptions without replaying causes already
reported from earlier exit handlers. This keeps the previous handler
exception visible while avoiding duplicate diagnostics.
This change separates the master box from the root box, as the single
master copy of boxes.
Before this change, the root box is the source of copies, and also
it runs builtin classes' code. Builtin code makes changes by
requiring files, thus the source of copies is mutable, be different
from the past one. This causes different internal states of boxes,
depending on when it is created.
After this change, all boxes are created from the master box, and
the master box never runs code, so it realizes the immutable source
of copies.
This also makes is possible to separate RubyGems (and other default
gems) from each boxes. All boxes will have their own copies of RubyGems.
RubyGems has its internal state, but it'll be separated between boxes
after this change.
Ruby::Box fix stale cached values for exception-related global variables ($! and $@)
The exception-related virtual variables $! (current exception) and
$@ (its backtrace) are stored on the execution context (ec->errinfo
and the rescue/ensure frame's local slot accessed via errinfo_place),
not in box->gvar_tbl. Caching their values in box->gvar_tbl makes the
second read return a stale value from the previous raise/rescue:
```
begin; raise "first"; rescue; p $!; end
begin; raise "second"; rescue; p $!; end
# before: #<RuntimeError: first> / #<RuntimeError: first>
# after: #<RuntimeError: first> / #<RuntimeError: second>
```
```
begin; raise "first"; rescue; p $@.first; end
begin; raise "second"; rescue; p $@.first; end
# before: same backtrace returned for both
# after: distinct backtrace per raise
```
Fixes [Bug #21991](https://bugs.ruby-lang.org/issues/21991)
Related PR: https://github.com/ruby/ruby/pull/16303
Currently we maintain the subclasses list for two separate purposes (we essentially have to different relationships we're putting into the same list):
1. On a T_MODULE, we track the T_ICLASSes created to include it into
other classes. Used for method invalidation and propagating includes
on the module that happen after it's been used
2. On a T_CLASS/T_ICLASS, we track the T_CLASS/T_ICLASS which are the
immediate children of the class. We use this for method invalidation,
some cvar things, and to iterate through subclasses.
Purpose 1 does not have any issues with box, the T_ICLASS always belongs
to one specific module and that's immutable. This list can be box-global
(always use the prime classext or hoist it out) and only needs to be
pruned during free. If we care about behaviour under a particular box
(ie. the propagating includes), we should look up the current box being
modified on the ICLASS itself.
Purpose 2 is more complicated. It currently tracks the immediate
children, the T_CLASS or T_ICLASS whose super points back. Because super
is per-box and is mutable (include/prepend insert ICLASSes into the
chain) we need to update the list on include/prepend, entries must be
per-box, and we can have multiple entries per-box. *I propose we
simplify this by no longer tracking the immediate subclass*, but instead
tracking the T_CLASS -> ... -> T_CLASS relationship, ie. the inverse of
rb_class_superclass. That relationship is the same across all boxes and
immutable after Class creation.
As a special case the ICLASS for refinements are also added to the
purpose 2 list (on T_CLASS). As those ICLASS do not chain to an eventual
leaf T_CLASS.
When we need to find the classes which have included a module, we can
use the module subclasses list to find the ICLASS and then use
RCLASS_INCLUDER. If we needed to iterate all T_ICLASS, we could then
walk up the CLASS_SUPER chain, but I didn't find anywhere we needed to
do that.
* Document Range#to_set
* Update Thread#raise and Fiber#raise signatures and docs
* Add reference to String#strip to character_selectors.rdoc
* Update *nil docs when calling methods
* Enhance Array#find and #rfind docs
* Add a notice to Kernel#raise about cause:
to fix inconsistent and wrong current namespace detections.
This includes:
* Moving load_path and related things from rb_vm_t to rb_namespace_t to simplify
accessing those values via namespace (instead of accessing either vm or ns)
* Initializing root_namespace earlier and consolidate builtin_namespace into root_namespace
* Adding VM_FRAME_FLAG_NS_REQUIRE for checkpoints to detect a namespace to load/require files
* Removing implicit refinements in the root namespace which was used to determine
the namespace to be loaded (replaced by VM_FRAME_FLAG_NS_REQUIRE)
* Removing namespaces from rb_proc_t because its namespace can be identified by lexical context
* Starting to use ep[VM_ENV_DATA_INDEX_SPECVAL] to store the current namespace when
the frame type is MAGIC_TOP or MAGIC_CLASS (block handlers don't exist in this case)
* Add support for `cause:` argument to `Fiber#raise` and `Thread#raise`.
The implementation behaviour is consistent with `Kernel#raise` and
`Exception#initialize` methods, allowing the `cause:` argument to be
passed to `Fiber#raise` and `Thread#raise`. This change ensures that
the `cause:` argument is handled correctly, providing a more consistent
and expected behavior when raising exceptions in fibers and threads.
[Feature #21360]
* Shared specs for Fiber/Thread/Kernel raise.
---------
Co-authored-by: Samuel Williams <samuel.williams@shopify.com>
Rather than to lazily check the env using a trinary
value, we can more straightforwardly check for the
env during the VM boot.
This allow `rb_namespace_available` to just be a pointer
dereference.
Previously we used a flag to set whether a module was uninitialized.
When checked whether a class was initialized, we first had to check that
it had a non-zero superclass, as well as that it wasn't BasicObject.
With the advent of namespaces, RCLASS_SUPER is now an expensive
operation, and though we could just check for the prime superclass, we
might as well take this opportunity to use a flag so that we can perform
the initialized check with as few instructions as possible.
It's possible in the future that we could prevent uninitialized classes
from being available to the user, but currently there are a few ways to
do that.
This implements a hash set which is wait-free for lookup and lock-free
for insert (unless resizing) to use for fstring de-duplication.
As highlighted in https://bugs.ruby-lang.org/issues/19288, heavy use of
fstrings (frozen interned strings) can significantly reduce the
parallelism of Ractors.
I tried a few other approaches first: using an RWLock, striping a series
of RWlocks (partitioning the hash N-ways to reduce lock contention), and
putting a cache in front of it. All of these improved the situation, but
were unsatisfying as all still required locks for writes (and granular
locks are awkward, since we run the risk of needing to reach a vm
barrier) and this table is somewhat write-heavy.
My main reference for this was Cliff Click's talk on a lock free
hash-table for java https://www.youtube.com/watch?v=HJ-719EGIts. It
turns out this lock-free hash set is made easier to implement by a few
properties:
* We only need a hash set rather than a hash table (we only need keys,
not values), and so the full entry can be written as a single VALUE
* As a set we only need lookup/insert/delete, no update
* Delete is only run inside GC so does not need to be atomic (It could
be made concurrent)
* I use rb_vm_barrier for the (rare) table rebuilds (It could be made
concurrent) We VM lock (but don't require other threads to stop) for
table rebuilds, as those are rare
* The conservative garbage collector makes deferred replication easy,
using a T_DATA object
Another benefits of having a table specific to fstrings is that we
compare by value on lookup/insert, but by identity on delete, as we only
want to remove the exact string which is being freed. This is faster and
provides a second way to avoid the race condition in
https://bugs.ruby-lang.org/issues/21172.
This is a pretty standard open-addressing hash table with quadratic
probing. Similar to our existing st_table or id_table. Deletes (which
happen on GC) replace existing keys with a tombstone, which is the only
type of update which can occur. Tombstones are only cleared out on
resize.
Unlike st_table, the VALUEs are stored in the hash table itself
(st_table's bins) rather than as a compact index. This avoids an extra
pointer dereference and is possible because we don't need to preserve
insertion order. The table targets a load factor of 2 (it is enlarged
once it is half full).
The current doc is partially wrong since `refinements` method
returns an array of `Refinement` class, not `Module`.
I'm not sure if it's right to link to `Refinement`, but it has
some documentation so it should be useful.
https://docs.ruby-lang.org/en/3.2/Refinement.html
28a1c4f33e seems to call an improper
ensure clause. [Bug #20655]
Than fixing it properly, I bet it would be much better to simply revert
that commit. It reduces the unneeded complexity. Jumping into a block
called by a C function like Hash#each with callcc is user's fault.
It does not need serious support.
This commit splits gc.c into two files:
- gc.c now only contains code not specific to Ruby GC. This includes
code to mark objects (which the GC implementation may choose not to
use) and wrappers for internal APIs that the implementation may need
to use (e.g. locking the VM).
- gc_impl.c now contains the implementation of Ruby's GC. This includes
marking, sweeping, compaction, and statistics. Most importantly,
gc_impl.c only uses public APIs in Ruby and a limited set of functions
exposed in gc.c. This allows us to build gc_impl.c independently of
Ruby and plug Ruby's GC into itself.
This frees FL_USER0 on both T_MODULE and T_CLASS.
Note: prior to this, FL_SINGLETON was never set on T_MODULE,
so checking for `FL_SINGLETON` without first checking that
`FL_TYPE` was `T_CLASS` was valid. That's no longer the case.
In the following code, the iclass tree of refinements in cref should be <iclass of Kernel@M2> -> <iclass of Kernel@M1> -> Kernel.
However, the iclass tree was broken because of code for included modules of refinements in rb_using_refinement().
Refinement#include is now removed, so this commit removes such unnecessary code.
```ruby
module M1
refine(Kernel) do
def f1 = :f1
end
end
module M2
refine(Kernel) do
def f2 = :f2
end
end
class Foo
using M1
using M2
def test
p f2 #=> :f2
p f1 # expected => :f1
# actual => undefined local variable or method 'f1' for an instance of Foo
end
end
Foo.new.test
```
This commit changes how stack extents are calculated for both the main
thread and other threads. Ruby uses the address of a local variable as
part of the calculation for machine stack extents:
* pthreads uses it as a lower-bound on the start of the stack, because
glibc (and maybe other libcs) can store its own data on the stack
before calling into user code on thread creation.
* win32 uses it as an argument to VirtualQuery, which gets the extent of
the memory mapping which contains the variable
However, the local being used for this is actually too low (too close to
the leaf function call) in both the main thread case and the new thread
case.
In the main thread case, we have the `INIT_STACK` macro, which is used
for pthreads to set the `native_main_thread->stack_start` value. This
value is correctly captured at the very top level of the program (in
main.c). However, this is _not_ what's used to set the execution context
machine stack (`th->ec->machine_stack.stack_start`); that gets set as
part of a call to `ruby_thread_init_stack` in `Init_BareVM`, using the
address of a local variable allocated _inside_ `Init_BareVM`. This is
too low; we need to use a local allocated closer to the top of the
program.
In the new thread case, the lolcal is allocated inside
`native_thread_init_stack`, which is, again, too low.
In both cases, this means that we might have VALUEs lying outside the
bounds of `th->ec->machine.stack_{start,end}`, which won't be marked
correctly by the GC machinery.
To fix this,
* In the main thread case: We already have `INIT_STACK` at the right
level, so just pass that local var to `ruby_thread_init_stack`.
* In the new thread case: Allocate the local one level above the call to
`native_thread_init_stack` in `call_thread_start_func2`.
[Bug #20001]
fix
The implementation of `native_thread_init_stack` for the various
threading models can use the address of a local variable as part of the
calculation of the machine stack extents:
* pthreads uses it as a lower-bound on the start of the stack, because
glibc (and maybe other libcs) can store its own data on the stack
before calling into user code on thread creation.
* win32 uses it as an argument to VirtualQuery, which gets the extent of
the memory mapping which contains the variable
However, the local being used for this is actually allocated _inside_
the `native_thread_init_stack` frame; that means the caller might
allocate a VALUE on the stack that actually lies outside the bounds
stored in machine.stack_{start,end}.
A local variable from one level above the topmost frame that stores
VALUEs on the stack must be drilled down into the call to
`native_thread_init_stack` to be used in the calculation. This probably
doesn't _really_ matter for the win32 case (they'll be in the same
memory mapping so VirtualQuery should return the same thing), but
definitely could matter for the pthreads case.
[Bug #20001]
This patch introduce M:N thread scheduler for Ractor system.
In general, M:N thread scheduler employs N native threads (OS threads)
to manage M user-level threads (Ruby threads in this case).
On the Ruby interpreter, 1 native thread is provided for 1 Ractor
and all Ruby threads are managed by the native thread.
From Ruby 1.9, the interpreter uses 1:1 thread scheduler which means
1 Ruby thread has 1 native thread. M:N scheduler change this strategy.
Because of compatibility issue (and stableness issue of the implementation)
main Ractor doesn't use M:N scheduler on default. On the other words,
threads on the main Ractor will be managed with 1:1 thread scheduler.
There are additional settings by environment variables:
`RUBY_MN_THREADS=1` enables M:N thread scheduler on the main ractor.
Note that non-main ractors use the M:N scheduler without this
configuration. With this configuration, single ractor applications
run threads on M:1 thread scheduler (green threads, user-level threads).
`RUBY_MAX_CPU=n` specifies maximum number of native threads for
M:N scheduler (default: 8).
This patch will be reverted soon if non-easy issues are found.
[Bug #19842]
From Ruby 3.0, refined method invocations are slow because
resolved methods are not cached by inline cache because of
conservertive strategy. However, `using` clears all caches
so that it seems safe to cache resolved method entries.
This patch caches resolved method entries in inline cache
and clear all of inline method caches when `using` is called.
fix [Bug #18572]
```ruby
# without refinements
class C
def foo = :C
end
N = 1_000_000
obj = C.new
require 'benchmark'
Benchmark.bm{|x|
x.report{N.times{
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
}}
}
_END__
user system total real
master 0.362859 0.002544 0.365403 ( 0.365424)
modified 0.357251 0.000000 0.357251 ( 0.357258)
```
```ruby
# with refinment but without using
class C
def foo = :C
end
module R
refine C do
def foo = :R
end
end
N = 1_000_000
obj = C.new
require 'benchmark'
Benchmark.bm{|x|
x.report{N.times{
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
}}
}
__END__
user system total real
master 0.957182 0.000000 0.957182 ( 0.957212)
modified 0.359228 0.000000 0.359228 ( 0.359238)
```
```ruby
# with using
class C
def foo = :C
end
module R
refine C do
def foo = :R
end
end
N = 1_000_000
using R
obj = C.new
require 'benchmark'
Benchmark.bm{|x|
x.report{N.times{
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
obj.foo; obj.foo; obj.foo; obj.foo; obj.foo;
}}
}
[Bug #19593]
rb_ec_setup_exception did not check if errinfo is a throw_data. This can
cause crashes in code since it is assumed that id_cause is an object.
We saw a crash in show_cause due to id_cause of errinfo being a
throw_data. It crashes on rb_obj_is_kind_of since it cannot be called on
T_IMEMO objects.
Unfortunately, we couldn't find a reproduction script, however we
debugged the core dump and rb_ec_setup_exception is the only place where
id_cause is assigned from errinfo without checking if it is a
throw_data.
```
0x0000556c5708e6dd in sigsegv (sig=11, info=0x7f301befa3f0, ctx=0x7f301befa2c0) at signal.c:964
0x00007f301d046420 in <signal handler called> () at /lib/x86_64-linux-gnu/libpthread.so.0
class_search_class_ancestor (c=139844586301760, cl=<optimized out>) at object.c:810
rb_obj_is_kind_of (obj=obj@entry=139839221734880, c=139844586301760) at object.c:861
0x0000556c56f2f00f in show_cause
(errinfo=errinfo@entry=139838840645160, str=str@entry=139839221730520, opt=139839221730480, highlight=0, reverse=reverse@entry=0, backtrace_limit=backtrace_limit@entry=-1, shown_causes=0x7ffe9d1a2d68) at ./include/ruby/internal/special_consts.h:175
```
Co-Authored-By: Jean Boussier <byroot@ruby-lang.org>
Right now the attached object is stored as an instance variable
and all the call sites that either get or set it have to know how it's
stored.
It's preferable to hide this implementation detail behind accessors
so that it is easier to change how it's stored.
Takashi Kokubun