The constant pool starts at source size / 120 entries, 4 at the least,
and doubles at a load of 3/4; each outgrown pair of arrays stays in the
metadata arena. An embedder that interns a fixed set of names into the
pool at every parse (mruby's compiler puts its 58 presyms there, as its
symbol table) grows a short program's pool from 4 to 128 and leaves
about 3 KB behind, as much as the pool itself. With the minimum under
#ifndef it can build with -DPM_CONSTANT_POOL_MIN_CAPACITY=128: measured
on a 32-bit build with 2 KB arena blocks, parsing one REPL line then
holds 6 KB instead of 10 KB. The default stays 4.
https://github.com/ruby/prism/commit/7ed8c71c59
61ebe5123 made defines.h include <sys/types.h> ahead of <inttypes.h>:
with an arm-none-eabi toolchain whose <stdint.h> is GCC's own, newlib's
<inttypes.h> never sees the marker it tests before defining the 64-bit
PRI macros, and PRIu64 goes undefined while PRIu32 does not. 400b21739
("Remove defines") took the header out and the workaround with it, so
a cross build of integer.c and errors_format.c, the two files that
include <inttypes.h> now, fails on such a toolchain (Debian's
gcc-arm-none-eabi 13.2.1 with newlib). Put the include back in those
two files, with the note.
https://github.com/ruby/prism/commit/ff42521e49
The arena's first block is 8 KB, sized for a host. An embedder on a
small heap parses short programs (a REPL line uses about 6 KB in two
arenas) and pays two 8 KB blocks for each; with the constant under
#ifndef it can build with -DPM_ARENA_INITIAL_SIZE=2048 instead.
https://github.com/ruby/prism/commit/f7a062efd4
When formatting error messages, first try to format into a fixed
buffer instead of formatting once to get the length then a second
time to get the actual format.
https://github.com/ruby/prism/commit/557cffe23e
There are some constructs that only last for the duration of a
parse, and should no longer be held once the parse is through. For
those we should use a scratch arena, freed at the end of parsing.
https://github.com/ruby/prism/commit/30a255539e
Instead of the half-baked murmur hash we were using. Also since
it now nicely spreads out the bits, keep all of the nodes in one
hash/allocation, and keep a 4-pointer buffer on the stack for the
majority case where there aren't that many keys.
https://github.com/ruby/prism/commit/cde8d1183c
The C++ interface is intentionally minimal: it provides only a
couple of niceties in order to largely stay out of the way. It is
mostly used to provide:
* RAII for opaque resources
* Safe up and down casting for nodes
* string_view for various resources
* Iteration over resource lists
Note that if consumers want to use it, they must be on C++17 at
minimum.
https://github.com/ruby/prism/commit/57b8dbe0e9
`__MINGW_PRINTF_FORMAT` is defined in stdio.h.
```
prism/buffer.c: In function 'pm_buffer_append_format':
prism/buffer.c:114:5: warning: function 'pm_buffer_append_format' might be a candidate for 'gnu_printf' format attribute [-Wsuggest-attribute=format]
114 | int result = vsnprintf(NULL, 0, format, arguments);
| ^~~
prism/buffer.c:123:9: warning: function 'pm_buffer_append_format' might be a candidate for 'gnu_printf' format attribute [-Wsuggest-attribute=format]
123 | vsnprintf(buffer->value + cursor, length, format, arguments);
| ^~~~~~~~~
```
https://github.com/ruby/prism/commit/72da47973e
We do not want strings to have the ability or the contract of
having to be freed. This makes the API simpler and keeps us away
from dynamic allocation.
https://github.com/ruby/prism/commit/c9b06230ec
So that consumers don't unintentionally encode the fact that a
constant id is actually 1 + its index in the constants table.
Ideally this makes it so that we can change the underlying
implementation if we want to.
https://github.com/ruby/prism/commit/3a7f3cb6d5
CRuby is now hashing the contents of a file and attaching that to
iseqs in order to properly determine if a file has changed since
it was compiled. This is used in a couple places, but namely it is
used for error_highlight to make sure it isn't highlighting stale
locations.
The hashing algorithm doesn't actually matter since it's just used
to compare across parses, so as long as it is consistent it's okay.
I wanted something fast and standard, so this commit implements the
XXH3_64 algorithm from the spec. Noteably it uses the default
secret and a seed of 0 to make things even simpler. With NEON and
SSE2 implementations, this thing flies quite a bit faster than the
upstream FNV-1a implementation.
I figured this could be useful for other consumers as well, in
particular for tools that generate some kind of metadata around
files (type checkers, linters, etc). Eventually a nice extension
may be to hash a minimal version of the contents (i.e., with
context-insensitive whitespace removed) so that changes to docs or
indentation do not change the hash value.
If you're using the C API, you are only charged for this function
if you actually call it since it is computed lazily. For everyone
else it is automatically called, but truthfully it is so fast that
the difference is completely negligible.
https://github.com/ruby/prism/commit/6b7c5773eb
The current version is too dynamic for rdoc
and so C methods like `Ruby::Prism.lex` are not doccumented.
Also tweak `Ruby::Prism` introduction. Ruby makes breaking changes
in minor versions all the time, no need to call it out.
long is narrower than the String length on mswin, so a gets result of 2GiB or more could turn negative, pass the clamp and reach memcpy as a huge size.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Introduce a perfect hash for keyword checking so we do not have to
do as many comparisons. This results in about a 1% speed up.
Also, introduce a function called by the C extension and the FFI
backend that can check if something is a keyword.
https://github.com/ruby/prism/commit/9d6a9e7d5f
Previously checking for a command write was the same concept as
checking for a command call, but in reality (and since 4.0) they
are different things.
https://github.com/ruby/prism/commit/e9cdac6b98
Unparenthesized lambdas with do...end inside a while loop need to
change ever so slightly. No one is using this syntax, but a fuzzer
found it.
https://github.com/ruby/prism/commit/fe09f5ac08
Because of the `^^^` hints, it's not valid ruby and so rdoc
gives it no syntax highlighting.
We can convince it otherwise and syntax-highlight syntax-invalid code.
Although it is syntax-invalid, the results look very good.
https://github.com/ruby/prism/commit/338b012386
Kevin Newton