rb_shape_verify_consistency asserted that every checked object has room
for at least one embedded field.
The default GC always satisfies this because its smallest slot is 32
bytes so this assertion never fires. MMTk gives an empty imemo_fields a
16-byte slot, so its shape carries zero capacity bits, and boot failed
on the first class freeze.
Instead of forcing MMTk to have a minimum object size, we should just
allow shapes to support a capacity of 0.
If we allocate objects larger than 1024 bytes, the capacity in the shape
ID will overflow. This commit fixes it set the capacity to SHAPE_ID_CAPACITY_MAX
in that case and falls back to rb_gc_obj_slot_size instead.
THe downside is that we now always reserve one 8B reference for the
IMEMO/fields, even for structs that never have any ivars.
But it is an acceptable tradeoff given that ivars on structs aren't
rare.
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Followup: https://github.com/ruby/ruby/pull/17631
Now that complex `RObject` always have an IMEMO/fields, we no
longer need to have an external `st_table` to replicate the
memory layout of `RObject`.
Instead of spilling into a raw buffer, `RObject` now spills
into an `IMEMO/fields` object like other types.
From an instance variable layout standpoint, an extended `RObject`
is now identical to a `RTypedData`, as in they both store the
reference to their `IMEMO/fields` at the same offset (`VALUE * 2`).
One positive consequence of this is that the only case where a `T_OBJECT`
needs sweeping is if a finalizer was registered.
YJIT now side exit when it need to write a new ivar into a `RObject`
that is out of space. This is unlikely to cause a performance regression
as this codepath isn't supposed to happen after warmup given `max_iv_count`
is recorded on classes, so future objects should be large enough.
Hence it's best not to waste executable memory for such codepath.
ZJIT lost support for writting into extended `RObject`.
Most of the code to support it is there it's just missing an
implementation of `RBASIC_SET_SHAPE_ID`, which recently changed
to strip some bits out of the shape (see comments).
I will leave it to the ZJIT team to implement it (sorry).
There are a few future cleanups planned that I keep for followups:
- We can now get rid of the `ROBJECT_HEAP` flag, it's redundant with
the shape layout bits.
- `imemo_fields` can now embed the `st_table` when complex.
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Co-authored-by: Randy Stauner <randy@r4s6.net>
SHAPE_MAX_CAPACITY needed to be below or equal to the maximum capacity
allocatable by the GC otherwise it would crash. This commit decouples that
by adding back max_capacity in the rb_shape_tree_t and adding a new GC
API function rb_gc_impl_max_allocation_size to report the maximum size
allocatable by the GC.
Set RUBY_TYPED_THREAD_SAFE_FREE on TypedData types whose dfree function
is trivially safe, and only frees its own memory
Types not yet marked as THREAD_SAFE_FREE:
- id2ref_tbl_type: sets the process-global id2ref_tbl to NULL.
- mutex_data_type: unlinks itself from another thread's keeping_mutexes list.
- autoload_data_type: deletes nodes from a shared intrusive list.
- autoload_const_type: deletes itself from a shared list.
- rb_cont_data_type: mutates the shared fiber pool and the global first_jit_cont list.
- rb_fiber_data_type: delegates to cont_free.
- FiberPoolDataType: mutates shared fiber pool free-lists.
- ractor_data_type: reconfigures global VM event-hook flags and call caches.
- exported_object_registry: frees a global table under the VM lock.
- rb_box_data_type (Box::Entry): unlinks classext from other live class and module objects.
- box_ext_cleanup_type: dereferences another String object and calls unlink.
- monitor_data_type: uses the default free, so the flag has no effect.
Co-authored-by: Luke Gruber <luke.gruber@shopify.com>
rb_shape_transition_object_id skipped the max_capacity guard that the
instance variable path applies, so an object filled to exactly
SHAPE_MAX_FIELDS would assert when assigning an object_id.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reserve 2 bits for expressing object layout
We would like to make instance variable reads in the JIT compiler faster
(as well as simplify the JIT implementation). Currently, in order to
read an instance variable, we have to:
1. Test for heap object
2. Load object to a 64 bit register
3. Mask the object header
4. Bit test against the masked header
5. JNE
6. Load field
We would like to:
1. Test for heap object
2. Load object shape to a 32 bit register
3. Bit test against the shape
4. JNE
5. Load field
The way we fetch instance variables is not consistent across objects.
In order to realize our goal, we need to encode object layout inside the
shape. If we encode object layout inside the shape, then the shape
itself will guarantee that the access pattern generated by the JIT
compiler is correct.
We should encode the following load patterns into the shape tag bits.
This way we can share shapes on transitions, but be able to
differentiate the access patterns for the JIT compiler. In other words,
two objects can have an `@a -> @b -> @c` transition and share the same
shape, but the tag bits can differentiate the access pattern so that the
JIT compiler can be confident that the machine code is correct.
Here are the patterns:
1. Embedded/Extended T_OBJECT Instance Variables
Objects with direct references to instance variables or via malloc
buffer
2. Objects with fields_objects fields
These are Data and TypedData objects. They have an associated axillary
imemo/fields object that stores the instance variables. The access
pattern is `object[2] + 2`. The fields object is the 3rd field, and the
instance variables start at +2 inside the fields object. The fields
object itself is a Ruby object, so it contains the usual header bits +
class headers.
3. Non Boxable Classes / Modules
This is similar to Objects with fields_objects, but the fields object is
stored at a different offset. We’re differentiating this from boxable
classes and modules because those are harder to support.
4. Other
"Other" pattern is for objects that are rare, or have
difficult-to-implement access patterns. This includes:
* Boxable classes and modules
* Structs (for now)
* Objects that use the geniv table
Proposed shape bit layout:
```
Current shape_id_t is 32 bits:
31 28 27 26 25 24 23 22 19 18 0
+-----------+--+--+--+--+--+------------+----------------------------+
| unused |L1|L0|OI|FR|CX| heap index | shape tree offset |
+-----------+--+--+--+--+--+------------+----------------------------+
| | | | | | |
| | | | | | +-- bits 0-18: SHAPE_ID_OFFSET_MASK
| | | | | +--------------- bits 19-22: SHAPE_ID_HEAP_INDEX_MASK
| | | | +------------------ bit 23: SHAPE_ID_FL_COMPLEX
| | | +--------------------- bit 24: SHAPE_ID_FL_FROZEN
| | +------------------------ bit 25: SHAPE_ID_FL_HAS_OBJECT_ID
+--+--------------------------- bits 26-27: SHAPE_ID_LAYOUT_MASK
```
The important part about these layout patterns is that they do not
reflect the _type_ of object, only how the object is laid out in memory.
For example, we currently treat structs as "other", but we can refactor
them to have the same layout as "Objects with fields_objects", and when
we do that they should get a different bit in the shape header.
This commit only reserves the two bits, it doesn't use them in the JIT
compiler yet.
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Co-Authored-By: Max Bernstein <tekknolagi@gmail.com>
* Update gc.c
Co-authored-by: Nobuyoshi Nakada <nobu.nakada@gmail.com>
* Update shape.h
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
* fix function name
* Update shape.c
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
* fix function name
* Revert "Update shape.c"
This reverts commit 900711defc6c541a93f3393a350819ae88cf87f1.
* add comment
---------
Co-authored-by: John Hawthorn <john@hawthorn.email>
Co-authored-by: Max Bernstein <tekknolagi@gmail.com>
Co-authored-by: Nobuyoshi Nakada <nobu.nakada@gmail.com>
Co-authored-by: Jean Boussier <jean.boussier@gmail.com>
Now that we have 1024B slots, we can store up to 126 fields inline.
Objects larger than this are rare if not non-existent, hence we can
get rid of the `malloc` path for imemo/fields and simply transition
to `TOO_COMPLEX`.
This additionally allows to shrink `attr_index_t` from 16 to 8B.
Note: the ZJIT "ivar on extended" tests are renamed as "complex" because
"extended" AKA malloc allocated imemo/fields no longer exists.
They're now complex fields, AKA st tables.
rb_class_allocate_instance: start as complex when over max_fields
If `RCLASS_MAX_IV_COUNT` is over `max_fields`, allocating a large
slot to end up transitioning to `TOO_COMPLEX` is wasteful.
We might as well start as complex directly.
Given we only store the shape offset in ivar caches, and the
offset is only 19bits, we should be able to store two shape offsets
and an attribute index in a single uint64_t.
Which is perfect for `setivar` caches.
However when initializing caches we set them to `INVALID_SHAPE_ID`
which is a full 32bits.
By making `INVALID_SHAPE_ID` only as large as a shape offset
we open the door to packing more information in ivar caches.
When the shape doesn't have a SHAPE_OBJ_ID edge, we don't
really have to rebuild it, we can simply combine the old shape
offset with the new shape flags.
Expose both `rb_obj_shape_` functions that take a `VALUE`
and `rb_shape_` functions that take a `shape_id`.
Make common transition functions such as `complex` and `frozen`
inlineable.
Also get rid of RB_SET_SHAPE_ID and rb_set_boxed_class_shape_id.
We don't need to call into `shape_transition_object_id`, we know
we're transitioning to TOO_COMPLEX so the offset part of the id
can only be `ROOT_TOO_COMPLEX_SHAPE_ID` or `ROOT_TOO_COMPLEX_WITH_OBJ_ID`.
So there's no need to lookup the shape tree.
Embed and shrink the capacities array so that it can be queried
more efficiently.
Move `next_shape_id` and `cache_size` outside of the struct on their
own cache line, as they're expected to be incremented atomically from
concurrent threads, so we want to avoid false sharing.
The also don't need to be exposed to the rest of the VM.
Get rid of `root_shape` as it's always equal to `shape_list`.
Otherwise some code defining ivars on a naked object early during
program boot can cause all objects to be larger than needed.
This seem like it was always the intention, but wasn't quite
properly prevented.
Also stop updating `max_iv_count` during GC marking as it is
redundant.
When pool slot sizes can be smaller than sizeof(struct RBasic) (e.g. a
32-byte pool on 64-bit where RBasic is 16 bytes), the capacity
calculation would underflow. Guard against this by setting capacity to
0 for pools too small to hold fields.
This should never be true. I added an `rb_bug` in case it was and it
wasn't true in any of btest or test-all.
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
[Bug #21710]
- struct.c: `struct_alloc`
It is possible for a `NEWOBJ` tracepoint call back to write fields
into a newly allocated object before `struct_alloc` had the time
to set the `RSTRUCT_GEN_FIELDS` flags and such.
Hence we can't blindly initialize the `fields_obj` reference to `0`
we first need to check no fields were added yet.
- object.c: `rb_class_allocate_instance`
Similarly, if a `NEWOBJ` tracepoint tries to set fields on the object,
the `shape_id` must already be set, as it's required on T_OBJECT to
know where to write fields.
`NEWOBJ_OF` had to be refactored to accept a `shape_id`.
When we dup a complex object we need to issue a writebarrier_remember
on the new object. This was caught by wbcheck inside the rubygems test
suite.
Co-authored-by: Aaron Patterson <tenderlove@ruby-lang.org>
Matt Valentine-House