When formatting error messages, first try to format into a fixed
buffer instead of formatting once to get the length then a second
time to get the actual format.
https://github.com/ruby/prism/commit/557cffe23e
Instead of the half-baked murmur hash we were using. Also since
it now nicely spreads out the bits, keep all of the nodes in one
hash/allocation, and keep a 4-pointer buffer on the stack for the
majority case where there aren't that many keys.
https://github.com/ruby/prism/commit/cde8d1183c
CRuby is now hashing the contents of a file and attaching that to
iseqs in order to properly determine if a file has changed since
it was compiled. This is used in a couple places, but namely it is
used for error_highlight to make sure it isn't highlighting stale
locations.
The hashing algorithm doesn't actually matter since it's just used
to compare across parses, so as long as it is consistent it's okay.
I wanted something fast and standard, so this commit implements the
XXH3_64 algorithm from the spec. Noteably it uses the default
secret and a seed of 0 to make things even simpler. With NEON and
SSE2 implementations, this thing flies quite a bit faster than the
upstream FNV-1a implementation.
I figured this could be useful for other consumers as well, in
particular for tools that generate some kind of metadata around
files (type checkers, linters, etc). Eventually a nice extension
may be to hash a minimal version of the contents (i.e., with
context-insensitive whitespace removed) so that changes to docs or
indentation do not change the hash value.
If you're using the C API, you are only charged for this function
if you actually call it since it is computed lazily. For everyone
else it is automatically called, but truthfully it is so fast that
the difference is completely negligible.
https://github.com/ruby/prism/commit/6b7c5773eb
Introduce a perfect hash for keyword checking so we do not have to
do as many comparisons. This results in about a 1% speed up.
Also, introduce a function called by the C extension and the FFI
backend that can check if something is a keyword.
https://github.com/ruby/prism/commit/9d6a9e7d5f
Previously checking for a command write was the same concept as
checking for a command call, but in reality (and since 4.0) they
are different things.
https://github.com/ruby/prism/commit/e9cdac6b98
This is the output of
ruby tool/sync_default_gems.rb prism
git rm -r lib/prism.rb lib/prism
at https://github.com/ruby/prism/commit/0a5f9c9, with the new mapping:
lib/prism is now lib/ruby/prism, the namespace is rewritten to
Ruby::Prism, and lib/prism/prism.gemspec is no longer imported, so
prism is no longer installed as a default gem.
prism/srcs.mk.in (excluded from the sync) follows the template engine,
now Ruby::Prism::Template: it defines the Ruby module, which does not
exist in BASERUBY older than Ruby 4.0, before loading
templates/template.rb.
Ruby::Prism is the very parser that parsed the running code, no
matter which version of the prism gem is installed; the two can be
loaded side by side (the ruby binary is built with -fvisibility=hidden,
so the gem's pm_* and rb_cPrism* symbols do not collide with the
built-in ones).
[Bug #22379]
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* This is more accurate than using the line and is a portable API between Ruby implementations.
* I also tried to use #syntax_tree but that fails 2 tests as it returns the CallNode instead of BlockNode for blocks.
Additionally, #syntax_tree emits many warnings when running the test suite:
syntax_tree: a prism gem other than the default gem is loaded; the result may not correspond exactly to the compiled code
https://github.com/ruby/prism/commit/3c2855741e
There are times when we don't need explicit order in a list of
constants, namely when looking at names of captures in a pattern
match and names of captures in a regular expression. In these cases
it's better to be linear with respect to the number of names, so
using a hash makes a little more sense.
https://github.com/ruby/prism/commit/d1c23d4956
When a variable is captured anywhere in a pattern, the operands of each
alternation are visited to report captures inside them. That visitor
treated every local variable target as a capture, including
underscore-prefixed names and targets that belong to a lambda, a string
interpolation, or a pinned expression:
case foo
in [bar, 1 | _baz] # error
in [bar, "#{case 1; in baz; end}" | 2] # error
end
Without `bar` both are accepted. Skip underscore-prefixed names, like
parse_pattern_capture does, and only descend into the nodes that make up
the pattern itself. Nested alternations are skipped too, their operands
were already checked when they were parsed. Before this, a capture like
`1 => [x, (1 | y) | 2]` was reported twice.
The new fixture is excluded from the tests that compare against parse.y,
which currently rejects these forms because of the capture of `bar`:
https://bugs.ruby-lang.org/issues/22310https://github.com/ruby/prism/commit/33ef692741
(https://github.com/ruby/prism/pull/4221)
Ref https://bugs.ruby-lang.org/issues/22294
pm_check_value_expression descended the left operand of every and/or
node it visited. Because pm_and_node_create and pm_or_node_create
already assert the value of their left operand when the node is built,
walking a left-associative chain such as `a && a && ... && a`
re-checked the whole left spine once per operator, which is quadratic
in the length of the chain: 16k operators took seconds.
The left operand of an existing and/or node was therefore already
checked, so stop at the node instead of descending. This makes the
check linear. The only observable change is that a void value on the
left spine of a chain is now reported once, at the innermost node
where it is created, rather than once per enclosing operator; the
errors fixture that pinned the duplicate is updated to match.
https://github.com/ruby/prism/commit/b19a0a5bc0
When escaped newlines are encountered, they are added to the newline
list. We need to check that we aren't backing up to read the rest of
the line because of a heredoc, in which case the newline would have
already have been added to the list.
https://github.com/ruby/prism/commit/a3f8ba3d4a
When prism encounter invalid bytes and still switches over to parsing comments,
`strip` will raise
`syntax_suggest` might run into this via `ruby/spec`
https://github.com/ruby/prism/commit/c72817c58b
If the two preceeding instructions are known not to be dependent
on the state of the stack, then we can eliminate the `swap` and inverse
the two previous instructions.
This mostly eliminate instruction in two cases:
```ruby
Object.new
```
```
0000 opt_getconstant_path <ic:0 Object> ( 1)[Li]
0002 putnil
0003 swap
0004 opt_new <calldata!mid:new, argc:0, ARGS_SIMPLE>, 11
```
And:
```ruby
{**@a, b: 2}
```
```
0000 getinstancevariable :@a, <is:0> ( 1)[Li]
0003 putspecialobject 1
0005 swap
0006 putobject :b
0008 putobject 2
0010 opt_send_without_block <calldata!mid:core#hash_merge_ptr, argc:3, ARGS_SIMPLE>
```
Co-Authored-By: John Hawthorn <john@hawthorn.email>
Previously we were just checking the token, which meant something
like `{ a: 1, **{a: 2}.select { false } }` would warn. As esoteric
as this is, it was possible. This also happily gets rid of the
current_hash_keys field on parser, making it just a tad bit slimmer.
https://github.com/ruby/prism/commit/3783ab0d31
Instead of a 2-element array. Nobody should care about the state, and nobody seems to.
A code search on Github shows that it only gets discarded (like via `value.map(&:first)`, or similar things.
But because our own ripper translator still needs it for compatibility, we have to keep it around in some way.
I played around with `Prism.lex_internal` that keeps the same return value but that duplicates much
logic like serialization/deserialization.
This achieves the same thing and is much smaller in scope.
https://github.com/ruby/prism/commit/32455d191c
There are actually 3 tokens in the Ruby grammar: the newline in a
whitespace insensitive position that is ignored, a newline that is
in a whitespace sensitive position that acts as a statement
terminator, and a newline in a whitespace sensisitive position that
acts as the terminator for an expression. The third one doesn't
exist in our grammar at the moment, but I want to add it, because
it makes working with translating the lex output easier.
https://github.com/ruby/prism/commit/3dbe592a76
This reverts commit 0017dea395,
5e36630fc0, and
f3ec056c9f.
We are trying to bump to latest IRB, which has "Accomodate new prism
tokens", but that runs into another CI failure. While we figure out a
path forward at <https://github.com/ruby/ruby/pull/18207>, let's get
back to a green CI.
There's some special handling in regards to empty symbols and those containing newlines
that should apply here. Makes it produce the correct amount of sym/dsym nodes
https://github.com/ruby/prism/commit/8b567cc403
This was previously an allow-list of the various tokens that were
considered the start of a pattern expression. Instead, use our
existing expression checking stuff and avoid having to list out
each token.
https://github.com/ruby/prism/commit/a139586ca0
Co-authored-by: kai-matsudate <kai.matsudate@techouse.jp>
Package precompiled native gems for Ruby 3.3, 3.4, and 4.0 across 10
platforms: aarch64-linux-gnu, aarch64-linux-musl, aarch64-mingw-ucrt,
arm-linux-gnu, arm-linux-musl, arm64-darwin, x64-mingw-ucrt,
x86_64-darwin, x86_64-linux-gnu, and x86_64-linux-musl.
- Add Ruby-version-aware extension loader in lib/prism.rb with GLIBC
error message for musl/glibc mismatches
- Add cross-compilation support to Rakefile using rake-compiler-dock
- Add a reusable `build-gems.yml` workflow called by both the CI
workflow (`test-gems.yml`) and the publish
workflow (`publish-gem.yml`), building source and native gems across
all platforms and Ruby versions
- Add scripts for building, verifying, and testing gems
Update `publish-gem.yml` to handle multi-gem publishing. Note that
`rubygems/release-gem` only supports a single gem, so this uses
`rubygems/configure-rubygems-credentials` and loops `gem push` over
each built `.gem`. Add a `workflow_dispatch` trigger alongside the
existing tag-push trigger, and create/update a GitHub release for the
tag with all gems plus a `CHECKSUMS.txt` manifest.
Move testing of the "ruby" platform gem (compile-from-source) out of
`main.yml` into `test-gems.yml` so we're testing the gem built by the
workflow tooling.
https://github.com/ruby/prism/commit/6cc5aa2d98
A couple of things to fix, notably that we relied on the stream
returning strings and relied on it not lying about how much was
being written. Now we no longer trust either and we verify
instead.
https://github.com/ruby/prism/commit/70147e5b44
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
It is nice to be able to slice out a location in pattern matching
without having to explicitly get the location object and call
slice manually.
Also, construct the response for deconstruct_keys lazily in order
to avoid reifying location objects if we don't have to.
https://github.com/ruby/prism/commit/3024cb55f2
Kevin Newton