`spe` is the string end, so if the escape is incomplete, it doesn't
mean we reached the end of stream, hence it's a syntax error, not
an EOS error.
https://github.com/ruby/json/commit/c1a4372837
The column of a parse error is a character position in the source, so it follows them while the line number stays long. It falls back to long on Ruby versions that do not define HAVE_RB_LEN_T.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Convert non-Proc on_load callbacks through to_proc and validate the result in the shared parser configuration. Previously, arbitrary values were stored and passed directly to rb_proc_call_with_block.
Cover Method callbacks in JSON.parse, JSON::Coder, and ResumableParser, along with invalid types, invalid conversion results, and nil or false callbacks.
https://github.com/ruby/json/commit/672da19fd5
The C parser guards leading-first surrogate pairs but silently accepts
a lone trailing surrogate, encoding U+DC00..U+DFFF as three UTF-8 bytes
into the output buffer. The returned Ruby String is tagged UTF-8 but
fails valid_encoding? and raises misleading errors from downstream
String operations (upcase, split, regex, encode, JSON.generate).
Add a symmetric branch alongside the leading-surrogate check to raise
JSON::ParserError at the parse boundary. Extend test_invalid_surogates
with the new cases and move the three JSONTestSuite fixtures that this
fix newly rejects from INVALID_ENCODING_TESTS into UNDEFINED_FAILING,
closing the CRuby/JRuby parser parity gap documented in the file.
https://github.com/ruby/json/commit/82bf6bedcf
When this code was refactored in 9c4db31, the mantissa_digits captured
only the length of the mantissa and the negative sign was captured
separately. Additionally, the paths diverged between signed and
unsigned values. However, MAX_FAST_INTEGER_SIZE stayed at 18 which limited
the range of uint64_t and int64_t that took the fast path.
This change expands the range of uint64_t and int64_t's that can be
returned by the fast paths without falling back to
json_decode_large_number.
https://github.com/ruby/json/commit/0238abcaf3
Same reason we don't for errors, since the resumable parser doesn't
keep the entire stream in memory, it can't always be accurate, and
the buffer may even have been freed.
Anyway, for a long lived stream, it wouldn't be very helpful either.
Also adds some assertions to ensure the buffer liveness.
https://github.com/ruby/json/commit/2c332bfe2b
Allow extension projects to update marked dependency sections after
`create_makefile` has resolved their source and object lists.
Share the dependency scanner with `tool/mkdepend.rb` so bundled and
upstream extensions can generate the same dependency files.
Replace `tool/update-deps` with a Ruby source scanner so dependency
updates no longer require GNU make, a configured build, or compiler
preprocessor output.
Keep compact source mappings and scanner declarations in dependency
files, and expand complete rules into the build directory. Avoid the
temporary fixture headers required by the compiler-driven prototype.
Use the generated rules from GNU make, BSD make, and NMake builds.
Support out-of-tree and read-only source trees, and retain expanded
dependencies shipped in release archives when baseruby is unavailable.
`JSON::ResumableParser` re-decodes an incomplete number on every chunk it
receives, so a long number delivered in small chunks is disproportionately slow
to parse -- the work is quadratic (a bit worse in practice) in the number's
length.
## Background
When the parser reads a number that reaches the end of the buffer, it cannot
tell whether more digits will arrive in a later chunk, so it rewinds the cursor
and returns `false` to wait for more input. Until now it fully decoded the
number before that check -- and then discarded the result. For a long run of
digits that decode builds a bignum, whose cost grows faster than linearly with
the digit count, and it was repeated on every chunk that extended the number.
The total cost is therefore quadratic (a bit worse in practice) in the length
of the number. The same "rewind and re-process" happens for incomplete strings
and comments, but those only re-scan (no value construction), so only numbers
pay this decode cost.
## Benchmark
Feeding an incomplete token in 128-byte chunks, one iteration being one full
chunked feed (`benchmark-ips`; macOS 15.6 arm64; ruby 4.0.0):
```ruby
require "json"
require "benchmark/ips"
def feed_incomplete(token, chunk_size)
parser = JSON::ResumableParser.new
i = 0
while i < token.bytesize
parser << token.byteslice(i, chunk_size)
i += chunk_size
parser.parse
end
end
CHUNK = 128
sizes = [100, 16_000, 32_000, 64_000, 128_000]
Benchmark.ips do |x|
x.config(time: 3, warmup: 1)
sizes.each do |n|
number = "1" * n
string = '"' + ("a" * n)
x.report("number #{n} digits") { feed_incomplete(number, CHUNK) }
x.report("string #{n} chars") { feed_incomplete(string, CHUNK) }
end
end
```
Per-iteration time, before vs. after this change:
| Input (128-byte chunks) | Before | After | Change |
|-------------------------|---------:|---------:|-----------------|
| number, 100 digits | 949 ns | 414 ns | negligible |
| number, 16,000 digits | 16.4 ms | 0.138 ms | ~119x faster |
| number, 32,000 digits | 94.8 ms | 0.531 ms | ~178x faster |
| number, 64,000 digits | 540.8 ms | 2.05 ms | ~264x faster |
| number, 128,000 digits | 3.07 s | 8.13 ms | ~378x faster |
| string, 100 chars | 411 ns | 391 ns | ~1x (unchanged) |
| string, 16,000 chars | 49.4 μs | 47.7 μs | ~1x (unchanged) |
| string, 32,000 chars | 169.7 μs | 165.8 μs | ~1x (unchanged) |
| string, 64,000 chars | 621 μs | 616 μs | ~1x (unchanged) |
| string, 128,000 chars | 2.38 ms | 2.35 ms | ~1x (unchanged) |
The string rows are unchanged, confirming the change is number-specific.
## Fix
Defer the decode. `json_parse_number` still scans the digits (cheap, and enough
to advance the cursor so the caller can detect an incomplete number), but when
the scan reaches the end of the buffer in resumable mode it returns `Qundef`
without decoding. The caller already rewinds and returns `false` in exactly that
case, so the decoded value was never used there. The number is decoded once,
when a following byte proves it complete.
Non-resumable `JSON.parse` is unaffected: it passes `resumable = false`, so the
early return is never taken and the number is decoded as before. A test checks
that large integers and floats split across feeds still decode to the correct
value.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PpeQSEFkF1X1Uuzjx9BsYehttps://github.com/ruby/json/commit/d1299d42ac
When a feed boundary fell right after a consumed token -- a `,` or an opening
`[` / `{` -- whose effect was not yet committed to the parser's persistent
state, the token was dropped on resume, producing parse errors or silently
wrong values.
## Reproduction
```ruby
require "json"
def parse_in_two(first, second, **opts)
parser = JSON::ResumableParser.new(**opts)
parser << first
parser.parse
parser << second
parser.parse
parser.value
rescue JSON::ParserError => e
e.message
end
# a ',' and its closing bracket split across the boundary
p parse_in_two("[1,", "]", allow_trailing_comma: true)
# a comment right after '[' split across the boundary
p parse_in_two("[/*", "*/1]", allow_comments: true)
```
## Expected Behavior
```
[1]
[1]
```
Feeding `[1,]` or `[/**/1]` in a single chunk parses to `[1]`, so splitting it
at the comma or inside the comment should make no difference.
## Actual Behavior
```
"unexpected character: ']'"
1
```
For the comma, the closing bracket that arrived in the later chunk was rejected
because the comma had been forgotten. For the bracket, the array's first
element leaked out as a bare top-level document (`1` instead of `[1]`), and a
following `parse` would raise `unexpected character: ']'` on the orphaned
closing bracket. The same shapes affect objects (`{"a":1,` + `}`, `{/*` +
`*/"a":1}`), line comments (`[//` + `"x\n1]"`), and comments right after a
comma. It happens with the default configuration too, since comments are
accepted (with a deprecation warning) unless disabled.
## Description
Both bugs share one root cause: the parser consumed a token, then hit a
suspension point (end of buffer, possibly inside a comment) before recording
the token's effect in the persistent frame/phase state, so resuming had no
memory of the token. They differ in where that record lives, so the fix has
two parts.
On `,`, the comma phases used to eat whitespace and comments before
committing the frame's phase, in order to peek for a trailing-comma close.
The fix advances the phase to `JSON_PHASE_VALUE` (array) or
`JSON_PHASE_OBJECT_KEY` (object) immediately after consuming the comma.
Trailing-comma detection moves into those phases: an element/key position
that finds `]` / `}` (only reachable after a comma, since an empty container
closes inline) hands off to the comma phase to close.
On `[` / `{`, the bracket's frame is only pushed after the empty-container
check, and pushing it earlier is not an option: an empty container is decoded
inline without a frame, which is what keeps `[]` from consuming a nesting
level (`[[]]` parses with `max_nesting: 1`). Committing a frame first would
break that property or require reworking nesting accounting, affecting the
non-resumable parser as well. Instead, widen the rewind: the suspension
already rewinds the cursor, it just rewound to the start of the comment,
which is after the bracket. `json_eat_comments` gains a `resume_pos` that,
in resumable mode only, overrides the rewind target for all three suspension
sites (unterminated block comment, unterminated line comment, and a comment
marker split at the boundary), and the `[` / `{` cases pass the bracket's own
position via the new `json_eat_whitespace_resume_at` variant. Re-reading the
bracket and the comment on resume mirrors how strings and numbers already
rewind to `value_start` when they hit the end of the buffer. Non-resumable
parsing is unaffected in both parts: the same tokens are consumed in the same
order, and error positions still point at the comment.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PpeQSEFkF1X1Uuzjx9BsYehttps://github.com/ruby/json/commit/cc010c4688
There was no single API answering "does the stream end in the middle of
a document?" once all parseable fed bytes have been consumed. Callers
had to combine two complementary APIs:
!parser.rest.empty? || !parser.partial_value.nil?
`rest` only reflects unconsumed tokenizer bytes, so it is empty when the
stream is truncated exactly on a token boundary (right after a ':' or
','), while `partial_value` is nil when truncation happens mid-token
before any container is registered. Neither alone covers all shapes,
and `partial_value` materializes the partially built Ruby objects just
to test for nil.
`partial_value?` answers the same question as `!partial_value.nil?` by
looking at the parser's internal value stack directly, without building
the partial Ruby object graph.
`empty?` is strict: true only when the buffer is fully consumed, no
document is under construction and no parsed value awaits retrieval
with `value`. It is defined in Ruby as the composition of the three
underlying predicates so its definition doubles as documentation:
def empty?
eos? && !partial_value? && !value?
end
https://github.com/ruby/json/commit/0864e83701
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A `//` line comment split across a feed boundary was treated as fully
consumed, so comment body delivered in a later chunk leaked out as parsed
JSON values.
## Reproduction
```ruby
require "json"
parser = JSON::ResumableParser.new(allow_comments: true)
documents = []
"[1]//[999]\n[3]".each_char do |char|
parser << char
documents << parser.value while parser.parse
end
p documents
```
## Expected Behavior
```
[[1], [3]]
```
The `[999]` is inside the `//` comment (which runs until the newline), so it
is ignored, leaving the two documents `[1]` and `[3]`. This is what feeding
the whole string in a single chunk produces.
## Actual Behavior
```
[[1], [999], [3]]
```
The commented-out `[999]` leaked out as a real value. When the comment body
does not form valid JSON the split feed raises a ParserError instead. This
happens with the default configuration too, since comments are accepted
(with a deprecation warning) unless disabled.
## Description
A `//` line comment is only terminated by a newline. `json_eat_comments`
searched for that newline and, when none was present in the buffer, moved
the cursor to the end of the buffer and returned as if the comment had been
fully consumed.
For the one-shot `JSON.parse` this is correct: the buffer holds the whole
document, so a `//` comment with no trailing newline genuinely runs to the
end of input. For `ResumableParser`, however, the end of the buffer is only
a chunk boundary, not the end of input. Reaching it does not mean the comment
is over -- the newline (and possibly more comment body) may still arrive in a
later `<<`. Swallowing the buffered bytes as a finished comment loses the
"still inside a comment" state, so any comment body delivered in the next
chunk was parsed as JSON instead of being ignored. Because the result then
depended on where the input happened to be split, the same byte stream could
produce different values.
The block-comment branch already handles this correctly: when it cannot find
the closing `*/` it rewinds to the comment start and raises an EOS-tagged
error, which `ResumableParser#parse` swallows into a `false` return so the
comment is retried once more input is available. This change makes the
line-comment branch behave the same way, but only in resumable mode
(detected via `state->parser`, the same discriminator `raise_parse_error`
already uses). In non-resumable mode the previous behaviour is preserved: a
`//` comment with no trailing newline is still consumed to the end of input.
As a consequence, a stream that ends with an unterminated line comment now
stays incomplete (`parse` keeps returning `false`) until a newline arrives,
which mirrors how a trailing bare number is only considered complete once a
following separator is seen.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PpeQSEFkF1X1Uuzjx9BsYehttps://github.com/ruby/json/commit/9dbfeb83df
partial_value rebuilds the open containers on a scratch copy of the value
stack. Folding an empty innermost container pops nothing and pushes its
result, so head climbs one past the live size; the scratch buffer was sized
to exactly the live size, so the push triggered rvalue_stack_grow which
reallocated the ALLOCV buffer (a non-malloc pointer) and crashed.
A parent fold always reclaims its child's slot, so head exceeds the live
size by at most one -- either for the missing-value placeholder or for an
empty innermost fold, never both. Size the buffer to capa + 1.
https://github.com/ruby/json/commit/94c1af25b5
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
When parsing JSON floats with extremely negative exponents (like
123.456e-789 or 123e-10000000), the parser would fall back to
rb_cstr_to_dbl which internally calls strtod. When strtod returns
ERANGE due to underflow to 0.0, Ruby emits a "Float out of range"
warning, causing noise in the test output.
Fix: when mantissa_digits + exponent < -324, the effective value is
less than 10^(-324) < DBL_TRUE_MIN/2, so it must round to 0.0 in
IEEE 754 round-to-nearest. Return 0.0 directly without going through
rb_cstr_to_dbl, avoiding the spurious warning.
This fixes warnings introduced by JSONMinefieldParserTest tests
(test_i_number_double_huge_neg_exp and test_i_number_real_underflow)
added in commit https://github.com/ruby/json/commit/6507a836c5.
https://github.com/ruby/json/commit/724eddaeaa
`JSON::ResumableParser#<<` raises `Encoding::CompatibilityError` when the
first chunk is a frozen multibyte UTF-8 string and more data is fed after a
partial parse.
## Reproduction
```ruby
require "json"
parser = JSON::ResumableParser.new
parser << '{"message":"日本'.freeze # frozen + multibyte (UTF-8)
parser.parse # => false (incomplete)
parser << '語のつづき"}'.freeze # => Encoding::CompatibilityError: BINARY and UTF-8
```
The error only surfaces when the first chunk is both frozen and multibyte.
A frozen ASCII-only first chunk stays ASCII-compatible and a mutable first
chunk keeps its encoding, so neither triggers it. This is common in
streaming use cases where chunks arrive as frozen UTF-8 (e.g. gRPC/protobuf
string fields), and is easily hit with multibyte (e.g. Japanese) payloads.
## Root cause
In `cResumableParser_feed`, the first feed adopts a frozen input string
directly as the buffer (`parser->buffer = str`), keeping its UTF-8 encoding.
When a later feed needs to reallocate that frozen buffer, it allocates the
new buffer with `rb_str_buf_new()`, which returns an ASCII-8BIT (BINARY)
string and does not carry over the original UTF-8 encoding. The subsequent
`rb_str_append(parser->buffer, str)` then appends a multibyte UTF-8 chunk to
a BINARY buffer that already holds non-ASCII bytes, and the two
encodings are incompatible.
`convert_encoding` always normalizes the input to UTF-8, so the buffer is
guaranteed to be UTF-8. Associating the freshly allocated buffer with UTF-8
restores that invariant. This mirrors the existing idiom used for the
generator buffer, and also fixes the latent case where a frozen ASCII-only
first chunk would silently leave the buffer as BINARY.
## Test
Added a regression test feeding frozen multibyte chunks across a partial
parse. The byte-by-byte `assert_resumed_parsing` helper cannot cover this:
it feeds single ASCII-8BIT bytes via `byte.chr`, which `convert_encoding`
turns into a mutable dup, so the buffer never becomes frozen and the
reallocation path is never taken. The `.freeze` calls are kept explicit so
the test still exercises the frozen path even if the file's
`frozen_string_literal` magic comment is ever removed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PpeQSEFkF1X1Uuzjx9BsYehttps://github.com/ruby/json/commit/532065c259
Fix: https://github.com/ruby/json/pull/1016#issuecomment-4744487710
`json` takes option hashes across the board, mostly because its API
predates the introduction of keyword arguments.
I'd like to change that to only take keyword arguments and error
when an unknown argument is passed, but I'm not yet sure of the
backward compatibility consequences, so it might wait for the next
major.
But in the meantime, `ResumableParser` being a new API, it can
safely use keyword arguments.
https://github.com/ruby/json/commit/f08c66338c
Fix: https://github.com/ruby/json/issues/1022
They can't always be accurate because we don't always keep the
full document in the buffer. As such it's better never to compute
them than to sometimes provide wrong coordinates.
In theory we could keep the number of lines since the start
of the parse, but that's more book keeping for little utility.
Anyway, these are useful to find a syntax error in a file,
not so much in a stream of documents.
https://github.com/ruby/json/commit/b2372115af
Stop updating stack handles, we don't actually need them as we're
starting heap allocated. Add an assertion to prove it.
Use a dedicated field for storing the parser reference to be
used for marking EOS errors. Also stop marking EOS errors when
not in resumable mode.
https://github.com/ruby/json/commit/004a5c1faa
An incomplete document is buffered in full with no size limit, so reading from an
untrusted source can grow memory without bound. Note in the rdoc that bounding the
input is the caller's responsibility.
https://github.com/ruby/json/commit/3650e2bf41
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cursor_position consumed the newline ending the previous line (post-decrement)
before counting lines, so any error past the first line was reported one line
too low (e.g. "[1,\n@" reported '@' at line 1 instead of line 2). Count that
newline when the column loop breaks on it.
https://github.com/ruby/json/commit/82b6b21354
Co-Authored-By: Jean Boussier <jean.boussier@gmail.com>
Jean Boussier