There are some constructs that only last for the duration of a
parse, and should no longer be held once the parse is through. For
those we should use a scratch arena, freed at the end of parsing.
https://github.com/ruby/prism/commit/30a255539e
Instead of the half-baked murmur hash we were using. Also since
it now nicely spreads out the bits, keep all of the nodes in one
hash/allocation, and keep a 4-pointer buffer on the stack for the
majority case where there aren't that many keys.
https://github.com/ruby/prism/commit/cde8d1183c
We do not want strings to have the ability or the contract of
having to be freed. This makes the API simpler and keeps us away
from dynamic allocation.
https://github.com/ruby/prism/commit/c9b06230ec
CRuby is now hashing the contents of a file and attaching that to
iseqs in order to properly determine if a file has changed since
it was compiled. This is used in a couple places, but namely it is
used for error_highlight to make sure it isn't highlighting stale
locations.
The hashing algorithm doesn't actually matter since it's just used
to compare across parses, so as long as it is consistent it's okay.
I wanted something fast and standard, so this commit implements the
XXH3_64 algorithm from the spec. Noteably it uses the default
secret and a seed of 0 to make things even simpler. With NEON and
SSE2 implementations, this thing flies quite a bit faster than the
upstream FNV-1a implementation.
I figured this could be useful for other consumers as well, in
particular for tools that generate some kind of metadata around
files (type checkers, linters, etc). Eventually a nice extension
may be to hash a minimal version of the contents (i.e., with
context-insensitive whitespace removed) so that changes to docs or
indentation do not change the hash value.
If you're using the C API, you are only charged for this function
if you actually call it since it is computed lazily. For everyone
else it is automatically called, but truthfully it is so fast that
the difference is completely negligible.
https://github.com/ruby/prism/commit/6b7c5773eb
Introduce a perfect hash for keyword checking so we do not have to
do as many comparisons. This results in about a 1% speed up.
Also, introduce a function called by the C extension and the FFI
backend that can check if something is a keyword.
https://github.com/ruby/prism/commit/9d6a9e7d5f
Pre-compilation could potentially enable SSSE3, when the target
the code is being installed on might not actually have it. In
order to use more modern instruction sets, we need runtime
detection. It's simpler just to rely on the baseline for now.
https://github.com/ruby/prism/commit/1bab98e441
There are times when we don't need explicit order in a list of
constants, namely when looking at names of captures in a pattern
match and names of captures in a regular expression. In these cases
it's better to be linear with respect to the number of names, so
using a hash makes a little more sense.
https://github.com/ruby/prism/commit/d1c23d4956
The parse.y grammar has the concept of a words separator that must
delimit words in a word list literal. Prism previously didn't emit
these in the token stream because they don't actually correspond
to any source code.
However, in order to better compare against the vendored grammars,
it's much easier if we do emit them. This commit adds them into
the token stream without actually touching the parser, so it
barely effects anything except conformance.
Since we have split up all those other tokens now and we have this
in place, we now don't need any special handling: our tokens stream
is pretty much exactly a superset of parse.y's token stream,
meaning it's very easy to drive a parse.y grammar using prism's
token stream provided you have a mapping of token names.
https://github.com/ruby/prism/commit/f941f7beb2
_BitScanForward64 is not available for 32-bit x86, so the i386-mswin32 build failed at prism.c. Use the portable fallback there, as ntz_int64 in internal/bits.h does.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Previously we were just checking the token, which meant something
like `{ a: 1, **{a: 2}.select { false } }` would warn. As esoteric
as this is, it was possible. This also happily gets rid of the
current_hash_keys field on parser, making it just a tad bit slimmer.
https://github.com/ruby/prism/commit/3783ab0d31
* In file included from /usr/include/string.h:535,
from include/prism/internal/arena.h:12,
from src/prism.c:6:
In function 'memset',
inlined from 'lex_mode_push_regexp' at src/prism.c:290:5:
.../string_fortified.h:59:10: error: '__builtin_memset' offset [26, 34] from the object at 'lex_mode' is out of the bounds of referenced subobject 'breakpoints' with type 'uint8_t[7]' {aka 'unsigned char[7]'} at offset 18 [-Werror=array-bounds]
https://github.com/ruby/prism/commit/2908c1c6c5
I've been seeing this for a while now:
```
../../../../ext/prism/../../src/memchr.c: In function ‘pm_memchr’:
../../../../ext/prism/../../src/memchr.c:35:16: warning: return discards ‘const’ qualifier from pointer target type [-Wdiscarded-qualifiers]
35 | return memchr(memory, character, number);
| ^~~~~~
```
https://github.com/ruby/prism/commit/a3b1f10dbc
Kevin Newton