Consume the leading byte before calling tokadd_mbchar in the regexp Unicode escape fast path. The helper reads the character length from pcur - 1; passing an unconsumed byte corrupted multibyte characters and skipped following bytes.
Add regression coverage for two-, three-, and four-byte UTF-8 characters in invalid Unicode escapes inside extended regexp comments.
Token columns are int, so a token past column INT_MAX got a negative column, and the excerpt shown for a syntax error there was read from before the line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rb_long2int is a plain cast where int and long have the same width, so on 64-bit mswin a column past INT_MAX wrapped instead of raising RangeError as it does on LP64.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Where rb_len_t is wider than long, a heredoc opener past column LONG_MAX made the long offset negative, and the identifier and the restored lexer position were read from before the line.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rb_parser_string_t kept its length in a long, so rb_str_to_parser_string had to reject a String longer than LONG_MAX where rb_len_t is wider than long. Now the parser string holds the whole length and the check is gone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
rb_parser_string_t holds its length as a long, which is narrower than a String length on mswin once rb_len_t widens, so a line or literal of 2GiB or more would be truncated.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
These hold String and Array lengths or offsets into them, so they would truncate on mswin once rb_len_t widens. A build with rb_len_t set to long long reported them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
For `Const += value`, parse.y builds a NODE_CDECL whose value is an
OPCALL on a NODE_CONST read node, and that read node only spanned the
constant name. The NameError for an undefined constant is raised by that
read, so Thread::Backtrace::Location#source_range reported only the
constant name. Prism has no separate read node and reports the whole
ConstantOperatorWriteNode, and the same holds for ConstantAndWriteNode.
Give the NODE_CONST read node the location of the whole operator
assignment (`+=`, `&&=` and `||=`) so both parsers report the same range.
Other variable reads keep their own location as error_highlight locates
the operator from the end of the receiver node.
See [Bug #22235] for more details.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
reduce_nodes eliminated a bare `nil` in a method's tail (value) position
down to an empty method body, so parse.y emitted no :line event and
recorded no line coverage for that line, even though it is executed:
def m
nil # <- no :line event, no coverage under parse.y
end
The Prism compiler keeps the explicit nil and correctly emits the event.
Keep the nil in reduce_nodes when it carries a newline flag (i.e. it is
on its own line) so it still produces a line event, while the implicit
nil of an empty body (no newline flag) is still dropped. This makes
parse.y match Prism for TracePoint :line events and line coverage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Avoid quadratic parse time on long and/or chains
logop() builds a left-associative and/or chain into a right-leaning
tree, and found the insertion point by walking the whole right spine
from the top on every operator. For a chain such as
`a && a && ... && a` that walk is O(n) per operator, so parsing the
chain is O(n^2): 32k operators took seconds.
Cache the tail of the chain most recently built by logop(), so an
operator that extends the same chain reaches the insertion point in
constant time, making the parse linear. The cached tail is validated
before use (it must still be a node of the chain's type whose nd_2nd
is not another such node), so a stale entry from an earlier parse
falls back to the scan. The resulting tree is unchanged: AST and
compiled bytecode are byte-identical to before across left-associative,
parenthesized, and mixed and/or chains.
value_expr_check descended into the left operand of an and/or node,
so a chain such as `x = (return) && (return) && a` reported the same
void value once per operator instead of once. The left operand is
already checked for a value when logop() builds the node, so stop at
the node rather than descending.
This matches the existing test_void_value_in_rhs expectation that the
same warning is emitted just once, and mirrors the same change in
ruby/prism#4221.
Introduce a streaming source hash API (rb_source_hash_init/update/
finalize) in ruby_parser.c, and use it in the lexer of parse.y to
accumulate a hash of the source as each line is read. The hash is
stored in the AST, and will be used to check whether a file still
contains the same source code when it is re-parsed later.
[Feature #21795]
The hash algorithm (currently FNV-1a) is hidden behind the API as an
implementation detail of the interpreter, so it can be changed freely
between releases.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tokspace stops doubling as soon as the buffer reaches the token's length,
so a copy that lands exactly on the end leaves tokfix writing its
terminator one byte past it. No caller passes a large enough n to reach
that today, but the invariant tokfix relies on is worth holding.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Parametrize `block_param` and `f_args` into `arg-list` and
`tail-only-args`. Since the pattern for `pre_args` alone differs
between `opt_comma` and `excessed_comma`, retain it within each rule.
* Consistent with plain `blocks` and `for` blocks and methods
where the source_location covers their entire definition.
* Matches the documentation which mentions
"where the definition starts/ends".
* Partially reverts d357d50f0a
which was a workaround to be compatible with parse.y.
Nobuyoshi Nakada