While it's not allowed by the spec, some parsers like Oj do
accept it, and it can be blocking a transition.
Having this feature can help people migrate.
https://github.com/ruby/json/commit/3459499cb3
When serializing an Array, and one of the elements of the Array requires
calling `to_json`, if the depth is changed, it will be used for the next
entries, which wasn't the case before
https://github.com/ruby/json/commit/5abd43490714, and is not the case with
TruffleRuby and JRuby.
Additionally, with TruffleRuby and JRuby the state's depth after the
`to_json` call is used to close the Array, which isn't the case with
CRuby.
https://github.com/ruby/json/commit/386b36fde5
This prevent from freezing and sharing state instances.
If you needs some sort of arguments or extra state to the generator
methods, consider using `JSON::Coder` instead.
https://github.com/ruby/json/commit/e9fbc8937f
The caller already know if the string contains escape sequences
so this check is redundant.
Also stop calling `rstring_cache_fetch` from `json_string_unescape`
as we know it won't match anyways.
```
== Parsing twitter.json (567916 bytes)
ruby 3.4.6 (2025-09-16 revision https://github.com/ruby/json/commit/dbd83256b1) +YJIT +PRISM [arm64-darwin24]
Warming up --------------------------------------
after 122.000 i/100ms
Calculating -------------------------------------
after 1.226k (± 0.3%) i/s (815.85 μs/i) - 6.222k in 5.076282s
Comparison:
before: 1206.2 i/s
after: 1225.7 i/s - 1.02x faster
```
https://github.com/ruby/json/commit/b8cdf3282d
Co-Authored-By: Scott Myron <samyron@gmail.com>
Overview
This PR uses the [jdk.incubator.vector module](https://docs.oracle.com/en/java/javase/20/docs/api/jdk.incubator.vector/jdk/incubator/vector/package-summary.html) as mentioned in [issue #739](https://github.com/ruby/json/issues/739) to accelerate generating JSON with the same algorithm as the C extension.
The PR as it exists right now, it will attempt to build the `json.ext.VectorizedEscapeScanner` class with a target release of `16`. This is the first version of Java with support for the `jdk.incubator.vector` module. The remaining code is built for Java 1.8. The code will attempt to load the `json.ext.VectorizedEscapeScanner` only if the `json.enableVectorizedEscapeScanner` system property is set to `true` (or `1`).
I'm not entirely sure how this is packaged / included with JRuby so I'd love @byroot and @headius's (and others?) thought about how to potential package and/or structure the JARs. I did consider adding the `json.ext.VectorizedEscapeScanner` to a separate `generator-vectorized.jar` but I thought I'd solicit feedback before spending any more time on the build / package process.
Benchmarks
Machine M1 Macbook Air
Note: I've had trouble modifying the `compare.rb` I was using for the C extension to work reliability with the Java extension. I'll probably spend more time trying to get it to work, but as of right now these are pretty raw benchmarks.
Below are two sample runs of the real-world benchmarks. The benchmarks are much more variable then the C extension for some reason. I'm not sure if HotSpot is doing something slightly different per execution.
Vector API Enabled
```
scott@Scotts-MacBook-Air json % ONLY=json JAVA_OPTS='--add-modules jdk.incubator.vector -Djson.enableVectorizedEscapeScanner=true' ruby -I"lib" benchmark/encoder-realworld.rb
WARNING: Using incubator modules: jdk.incubator.vector
== Encoding activitypub.json (52595 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 1.384k i/100ms
Calculating -------------------------------------
json 15.289k (± 0.8%) i/s (65.41 μs/i) - 153.624k in 10.048481s
== Encoding citm_catalog.json (500298 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 76.000 i/100ms
Calculating -------------------------------------
json 753.787 (± 3.6%) i/s (1.33 ms/i) - 7.524k in 9.997059s
== Encoding twitter.json (466906 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 173.000 i/100ms
Calculating -------------------------------------
json 1.751k (± 1.1%) i/s (571.24 μs/i) - 17.646k in 10.081260s
== Encoding ohai.json (20147 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 2.390k i/100ms
Calculating -------------------------------------
json 23.829k (± 0.8%) i/s (41.97 μs/i) - 239.000k in 10.030503s
```
Vector API Disabled
```
scott@Scotts-MacBook-Air json % ONLY=json JAVA_OPTS='--add-modules jdk.incubator.vector -Djson.enableVectorizedEscapeScanner=false' ruby -I"lib" benchmark/encoder-realworld.rb
WARNING: Using incubator modules: jdk.incubator.vector
VectorizedEscapeScanner disabled.
== Encoding activitypub.json (52595 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 1.204k i/100ms
Calculating -------------------------------------
json 12.937k (± 1.1%) i/s (77.30 μs/i) - 130.032k in 10.052234s
== Encoding citm_catalog.json (500298 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 80.000 i/100ms
Calculating -------------------------------------
json 817.378 (± 1.0%) i/s (1.22 ms/i) - 8.240k in 10.082058s
== Encoding twitter.json (466906 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 147.000 i/100ms
Calculating -------------------------------------
json 1.499k (± 1.3%) i/s (667.08 μs/i) - 14.994k in 10.004181s
== Encoding ohai.json (20147 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 2.269k i/100ms
Calculating -------------------------------------
json 22.366k (± 5.7%) i/s (44.71 μs/i) - 224.631k in 10.097069s
```
`master` as of commit `https://github.com/ruby/json/commit/c5af1b68c582`
```
scott@Scotts-MacBook-Air json % ONLY=json ruby -I"lib" benchmark/encoder-realworld.rb
== Encoding activitypub.json (52595 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 886.000 i/100ms
Calculating -------------------------------------
json^C%
scott@Scotts-MacBook-Air json % ONLY=json ruby -I"lib" benchmark/encoder-realworld.rb
== Encoding activitypub.json (52595 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 1.031k i/100ms
Calculating -------------------------------------
json 10.812k (± 1.3%) i/s (92.49 μs/i) - 108.255k in 10.014260s
== Encoding citm_catalog.json (500298 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 82.000 i/100ms
Calculating -------------------------------------
json 824.921 (± 1.0%) i/s (1.21 ms/i) - 8.282k in 10.040787s
== Encoding twitter.json (466906 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 141.000 i/100ms
Calculating -------------------------------------
json 1.421k (± 0.7%) i/s (703.85 μs/i) - 14.241k in 10.023979s
== Encoding ohai.json (20147 bytes)
jruby 9.4.12.0 (3.1.4) 2025-02-11 https://github.com/ruby/json/commit/f4ab75096a Java HotSpot(TM) 64-Bit Server VM 21.0.7+8-LTS-245 on 21.0.7+8-LTS-245 +jit [arm64-darwin]
Warming up --------------------------------------
json 2.274k i/100ms
Calculating -------------------------------------
json 22.612k (± 0.9%) i/s (44.22 μs/i) - 227.400k in 10.057516s
```
Observations
`activitypub.json` and `twitter.json` seem to be consistently faster with the Vector API enabled. `citm_catalog.json` seems consistently a bit slower and `ohai.json` is fairly close to even.
https://github.com/ruby/json/commit/d40b2703a8
Fix: https://github.com/rails/rails/commit/90616277e3d8fc46c9cf35d6a7470ff1ea0092f7#r168784389
Because the `depth` counter is inside `JSON::State` it can't be used
concurrently, and in case of a circular reference the counter may be
left at the max value.
The depth counter should be moved outside `JSON_Generator_State` and
into `struct generate_json_data`, but it's a larger refactor.
In the meantime, `JSON::Coder` calls `State#generate_new` so I changed
that method so that it first copy the state on the stack.
https://github.com/ruby/json/commit/aefa671eca
Because both strings and symbols keys are serialized the same,
it always has been possible to generate documents with duplicated
keys:
```ruby
>> puts JSON.generate({ foo: 1, "foo" => 2 })
{"foo":1,"foo":2}
```
This is pretty much always a mistake and can cause various
issues because it's not guaranteed how various JSON parsers
will handle this.
Until now I didn't think it was possible to catch such case without
tanking performance, hence why I only made the parser more strict.
But I finally found a way to check for duplicated keys cheaply enough.
Étienne Barrié