Files
Edouard CHIN ac3b237782 [ruby/rubygems] Split the download and install process of a gem:
- ### Problem

  Bundler awaits for the dependencies of a gem to be download and installed
  before it proceeds to downloading and installing the dependency itself.
  This creates a bottleneck at the end of the installation process and
  block a thread unnecessarily.

  ### Details

  The installation strategy in Bundler is to await for "leaf gems" to
  be download/installed before the "root gem" is processed.

  For instance, in this scenario:

  - Gem "foo" has a dependency on "bar" (We can call this the "root gem")
  - Gem "bar" has no dependency (We call cal this the "leaf gems")
  - A Gemfile adds "gem 'foo'"

  In this case, only a single thread will have work to do, that is
  because Bundler will queue the gem "bar" to be downloaded and
  installed, and only when "bar" is finished, then Bundler will queue
  work for "foo".

  For **pure ruby gems**, this strategy is a waste of time because
  during the installation, a gem's code is not evaluated and no
  "require" statement is evaluated.

  For gems with native extensions, this strategy make sense. When the
  `extconf.rb` of a gem is evaluated, it's possible that the extconf
  requires a dependency and therefore Bundler needs to wait for those
  dependencies to be installed before it can execute the extconf.
  A typical example is a native extension that require 'mini_portile2'.

  ### Solution

  From the explanation above, I'd like to split the download from
  the installation of a gem.

  The tricky aspect is that there is no RubyGems API to know whether
  a gem has a native extension. The only way to know is after we
  download the gem and read the `metadata` from the tarball.

  So the solution in this patch is as follow:

  1. We download all gems without doing any checks.
  2. Once the gems are downloaded, we now know whether a gem has a
     native extensions.
  3. If a gem is a pure ruby gems, we install it without waiting.
  4. If a gem has a native extension, we check whether its dependencies
     are installed. If a dependency is not yet installed, we put back
     the gem in the queue. It will be dequeued later on and we'll redo
     this logic.

  ### Performance gain

  The speed gain highly depends on how deep the dependency tree is.
  E.g. bar depends on foo which depends on baz which depends on jane ...
  Previously we'd proceed to installing gems just one by one and only
  a single thread would be used.

  In a freshly generated Rails application, the speed gain is not that
  important. And the reason is because we are having a "tail latency"
  issue. Around the end of the installation process, the remaining
  gems to be installed are the one with native extensions, since we
  had to wait for for their dependencies to be installed. Compiling
  them is the slowest part, and since we are doing it at the end then
  the speed gain is not that noticeable.

  ### Misc

  Another advantage of this change is to be able to more easily
  implement this kind of feature https://github.com/ruby/rubygems/issues/9138
  to let users solely download gems, without installing them .

https://github.com/ruby/rubygems/commit/e5e8c0a507
2026-03-12 11:46:30 +00:00

121 lines
3.1 KiB
Ruby

# frozen_string_literal: true
module Bundler
class Source
autoload :Gemspec, File.expand_path("source/gemspec", __dir__)
autoload :Git, File.expand_path("source/git", __dir__)
autoload :Metadata, File.expand_path("source/metadata", __dir__)
autoload :Path, File.expand_path("source/path", __dir__)
autoload :Rubygems, File.expand_path("source/rubygems", __dir__)
autoload :RubygemsAggregate, File.expand_path("source/rubygems_aggregate", __dir__)
attr_accessor :dependency_names
attr_reader :checksum_store
def unmet_deps
specs.unmet_dependency_names
end
def version_message(spec, locked_spec = nil)
message = "#{spec.name} #{spec.version}"
message += " (#{spec.platform})" if spec.platform != Gem::Platform::RUBY && !spec.platform.nil?
if locked_spec
locked_spec_version = locked_spec.version
if locked_spec_version && spec.version != locked_spec_version
message += Bundler.ui.add_color(" (was #{locked_spec_version})", version_color(spec.version, locked_spec_version))
end
end
message
end
def download(*); end
def can_lock?(spec)
spec.source == self
end
def prefer_local!; end
def local!; end
def local_only!; end
def cached!; end
def remote!; end
def add_dependency_names(names)
@dependency_names = Array(dependency_names) | Array(names)
end
# it's possible that gems from one source depend on gems from some
# other source, so now we download gemspecs and iterate over those
# dependencies, looking for gems we don't have info on yet.
def double_check_for(*); end
def dependency_names_to_double_check
specs.dependency_names
end
def spec_names
specs.spec_names
end
def include?(other)
other == self
end
def inspect
"#<#{self.class}:0x#{object_id} #{self}>"
end
def identifier
to_s
end
def path?
instance_of?(Bundler::Source::Path)
end
def extension_cache_path(spec)
return unless Bundler.settings[:global_gem_cache]
return unless source_slug = extension_cache_slug(spec)
Bundler.user_cache.join(
"extensions", Gem::Platform.local.to_s, Bundler.ruby_scope,
source_slug, spec.full_name
)
end
private
def version_color(spec_version, locked_spec_version)
if Gem::Version.correct?(spec_version) && Gem::Version.correct?(locked_spec_version)
# display yellow if there appears to be a regression
earlier_version?(spec_version, locked_spec_version) ? :yellow : :green
else
# default to green if the versions cannot be directly compared
:green
end
end
def earlier_version?(spec_version, locked_spec_version)
Gem::Version.new(spec_version) < Gem::Version.new(locked_spec_version)
end
def print_using_message(message)
if !message.include?("(was ")
Bundler.ui.debug message
else
Bundler.ui.info message
end
end
def extension_cache_slug(_)
nil
end
end
end