Promote prism to a bundled gem

prism is no longer a default gem (the built-in parser is Ruby::Prism
now), so add the prism gem to gems/bundled_gems.  `require "prism"`
keeps providing the gem, which can be any version.  lib/bundled_gems.rb
warns when it is required under bundler without a Gemfile entry, like
other gems promoted to bundled gems.

The gem is pinned to the upstream revision that ruby/ruby is synced
with, because the tests of the released 1.9.0 compare with the parser
of the running Ruby, whose grammar has moved on in 4.1.

prism does not keep its templated files in git, so BundledGem.build
(used by both common.mk and defs/gmake.mk) generates them before
building the gem from the checkout.  Since the gem built from the
revision has the same version as the released one, BundledGem.unpack
now clears the target directory first; otherwise files removed upstream
(ext/prism/api_pack.c) remain from the released gem and break the
build.

tool/test-bundled-gems.rb: several bundled gems (irb, rdoc, rbs,
typeprof, ...) require prism, and their tests spawn ruby with
--disable=gems, where the prism gem is not loadable anymore.  Preset its
load paths in RUBYLIB, as is already done for debug.

[Bug #22379]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

gems/bundled_gems

@@ -38,6 +38,7 @@ pstore              0.2.1   https://github.com/ruby/pstore
 benchmark           0.5.0   https://github.com/ruby/benchmark
 logger              1.7.0   https://github.com/ruby/logger
 rdoc                8.1.0   https://github.com/ruby/rdoc
+prism               1.9.0   https://github.com/ruby/prism 0a5f9c956cb62b371ab680868d08c82ae0d49201
 win32ole            1.9.3   https://github.com/ruby/win32ole
 irb                 1.18.0  https://github.com/ruby/irb ac42eaaa88e6399384c1a56cc948c4b2528a9cc2
 reline              0.7.0   https://github.com/ruby/reline

lib/bundled_gems.rb

@@ -31,6 +31,7 @@ module Gem::BUNDLED_GEMS # :nodoc:
     "reline" => "4.0.0",
     # "readline" => "4.0.0", # This is wrapper for reline. We don't warn for this.
     "tsort" => "4.1.0",
+    "prism" => "4.1.0",
   }.freeze
 
   EXACT = {

tool/lib/bundled_gem.rb

@@ -28,6 +28,9 @@ module BundledGem
   def unpack(file, *rest)
     pkg = Gem::Package.new(file)
     prepare_test(pkg.spec, *rest) do |dir|
+      # Remove stale files of another build of the same version (e.g. the
+      # released gem replaced with one built from a pinned revision).
+      FileUtils.rm_rf(dir)
       pkg.extract_files(dir)
       FileUtils.rm_rf(Dir.glob(".git*", base: dir).map {|n| File.join(dir, n)})
     end
@@ -44,6 +47,11 @@ module BundledGem
     outdir = File.expand_path(outdir)
     gemdir, gemfile = File.split(gemspec)
     Dir.chdir(gemdir) do
+      if gemfile == "prism.gemspec" and File.exist?("templates/template.rb")
+        # prism does not keep its templated files in git; generate them so
+        # that the gem can be built from the checkout.
+        system(RbConfig.ruby, "templates/template.rb", exception: true)
+      end
       spec = Gem::Specification.load(gemfile)
       abort "Failed to load #{gemspec}" unless spec
       output = File.join(outdir, spec.file_name)

tool/test-bundled-gems.rb

@@ -63,6 +63,13 @@ rubylib = [gem_dir+'/lib', ENV["RUBYLIB"]].compact.join(File::PATH_SEPARATOR)
 run_opts = ENV["RUN_OPTS"]&.shellsplit
 exit_code = 0
 ruby = ENV['RUBY'] || RbConfig.ruby
+
+# prism is a bundled gem, not a default gem loadable without RubyGems.
+# Several bundled gems (irb, rdoc, rbs, typeprof, ...) require it, and
+# their tests spawn ruby with --disable=gems (assert_in_out_err etc.), so
+# preset the load paths of the prism gem in RUBYLIB for every gem.
+prism_libs = IO.popen([ruby, "-e", "old = $:.dup; require 'prism'; puts $:-old"], &:read)
+rubylib = [*prism_libs.split("\n"), rubylib].join(File::PATH_SEPARATOR) if $?.success?
 failed = []
 
 max = ENV['TEST_BUNDLED_GEMS_NPROCS']&.to_i || [Etc.nprocessors, 8].min