mirror of
https://github.com/BurntSushi/ripgrep.git
synced 2026-10-28 11:44:37 +08:00
ignore: benchmark Python workspace discovery and matching (#3554)
We add benchmarks for the ignore operations used by Ruff, uv, and ty: serial and parallel walks, early-exit workspace discovery, and incremental path matching. The 39 cases cover different project layouts, nested ignore rules, multiple roots, and cached directories. We create fixtures before timing and check their expected file counts during walks. The README links the consumer call sites and describes how to compare profiles. These measure library work with a warm filesystem cache, not end-to-end application performance. We also test these benchmarks in CI. PR #3554
This commit is contained in:
1 parent
7155f6408d
commit
fada8f9386
3 files changed
+688
No files matched your search
@@ -252,6 +252,19 @@ jobs:
|
||||
RUSTDOCFLAGS: -D warnings
|
||||
run: cargo doc --no-deps --document-private-items --workspace
|
||||
|
||||
benches:
|
||||
name: Test benchmarks
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
- name: Install Rust
|
||||
uses: dtolnay/rust-toolchain@master
|
||||
with:
|
||||
toolchain: nightly
|
||||
- name: Test ignore benchmarks
|
||||
run: cargo +nightly test -p ignore --benches
|
||||
|
||||
fuzz_testing:
|
||||
name: Compile Fuzz Test Targets
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
# Workspace discovery benchmarks
|
||||
|
||||
`workloads.rs` measures the `ignore` operations used by Python development tools:
|
||||
|
||||
| Consumer | Operation | Benchmarks |
|
||||
| --- | --- | --- |
|
||||
| [Ruff file discovery](https://github.com/astral-sh/ruff/blob/ae8ec3753192debd310e7b3d93255a19a5a2af85/crates/ruff_workspace/src/resolver.rs#L530) and [ty project indexing](https://github.com/astral-sh/ruff/blob/ae8ec3753192debd310e7b3d93255a19a5a2af85/crates/ruff_db/src/system/os.rs#L312) | Parallel walks, hidden files included, up to 12 workers | `walk_*_parallel_*`, `walk_explicit_files` |
|
||||
| [uv script discovery](https://github.com/astral-sh/uv/blob/83d556d712c7e316932b239c49e86a5de95d124a/crates/uv/src/commands/workspace/list.rs#L137) | Serial traversal with ignore rules and hidden files included | `walk_*_serial`, `uv_scripts_source_archive` |
|
||||
| [uv workspace discovery](https://github.com/astral-sh/uv/blob/83d556d712c7e316932b239c49e86a5de95d124a/crates/uv-workspace/src/workspace.rs#L1931) | Stop at the first non-directory entry, or exhaust an ignored member | `uv_*_workspace_member` |
|
||||
| [ty incremental indexing](https://github.com/astral-sh/ruff/blob/ae8ec3753192debd310e7b3d93255a19a5a2af85/crates/ruff_db/src/system/os/ignore.rs#L10) | Repeated relative-path checks with cached ancestor matchers | `incremental_*` |
|
||||
|
||||
The synthetic trees contain Python modules, package metadata, non-Python files,
|
||||
ignored bytecode directories, a virtual environment, and build outputs. The
|
||||
layouts vary independently: a small project, many sibling packages, deeply
|
||||
nested packages, and packages with their own ignore rules and negations. The
|
||||
parallel visitor counts files locally and merges once per worker, avoiding an
|
||||
artificial shared counter on every entry. One- and four-worker cases supplement
|
||||
the 12-worker configuration used by Ruff and ty. The unfiltered walk models
|
||||
disabling ignore rules and provides a control for traversal overhead.
|
||||
The source-archive case honors `.gitignore` without a `.git` directory, matching
|
||||
uv's `require_git(false)` configuration.
|
||||
|
||||
Fixtures are created before timing and removed afterward. Every walk checks its
|
||||
Python file count against the fixture's expected count. The benchmarks retain
|
||||
parent ignore discovery, but disable global Git ignore configuration so that
|
||||
results do not depend on a user's Git configuration. Runs use a warm filesystem
|
||||
cache. They measure `ignore`, excluding parsing, configuration resolution, and
|
||||
other application work; they are not end-to-end application speedups.
|
||||
|
||||
`incremental_cold` includes building matchers and loading directory rules on
|
||||
first use. `incremental_warm` checks 256 paths after their directories have been
|
||||
cached. `incremental_warm_noncanonical` checks the same paths with an internal
|
||||
`./` component, exercising normalization during cached queries.
|
||||
`incremental_ignored` checks a path beneath a pruned virtual environment.
|
||||
`incremental_warm_deep` checks 768 paths in a 12-level package tree, and
|
||||
`incremental_ignored_parent` directly hits an ignored directory's cache entry.
|
||||
`incremental_ignored_nested` checks a deeper descendant of a cached ignored
|
||||
virtual environment inside an otherwise allowed package.
|
||||
The `walk_dense_ignore_*` cases put rules in every directory of a deep tree,
|
||||
providing a control for optimizations that skip directories without rules.
|
||||
`incremental_dense_ignore_cold` and `incremental_dense_ignore_warm` use the same
|
||||
tree to check incremental matching when nearly every cached directory has rules.
|
||||
The `walk_multiple_roots_*` cases pass 64 sibling package directories as roots,
|
||||
covering parent-matcher reuse when Ruff receives multiple directory arguments.
|
||||
`walk_with_overrides_and_types` exercises explicitly configured override and
|
||||
file-type matchers as a control for their more common empty configuration.
|
||||
The `build_*` and `match_*` microbenchmarks help distinguish compilation and
|
||||
matching costs from filesystem traversal.
|
||||
The single-pattern cases cover bytecode exclusions, the directory whitelist
|
||||
used in Ruff's and uv's root `.ignore` files, and ty's vendored-typeshed whitelist.
|
||||
`build_single_directory_whitelist` builds that single-rule matcher from an
|
||||
already populated builder.
|
||||
The `*gitignore_only*` controls remove `.ignore` and `.git/info/exclude`, covering
|
||||
projects that rely solely on a `.gitignore` with multiple patterns.
|
||||
`walk_selected_python_files` uses two positive override patterns, exercising
|
||||
file-selection rules that match most visited files.
|
||||
|
||||
Like the `globset` benchmarks, these use Rust's nightly `test` harness:
|
||||
|
||||
```console
|
||||
cargo +nightly bench -p ignore --bench workloads
|
||||
cargo +nightly bench -p ignore --bench workloads incremental
|
||||
```
|
||||
|
||||
For comparisons, use the same compiler, profile, fixture parameters, and CPU
|
||||
affinity for both revisions. Alternate baseline and candidate runs, and retain
|
||||
the per-run results instead of relying on a single measurement. Parallel walks
|
||||
also include worker startup and shutdown, which can dominate small projects.
|
||||
|
||||
CPU affinity also affects match-index scratch-pool sizing: available parallelism
|
||||
is cached when the first pool is constructed. Pinning to one CPU therefore
|
||||
changes pool construction costs as well as scheduling. Report affinity with
|
||||
results, especially when comparing construction or cached matching costs.
|
||||
|
||||
Ruff, uv, and ty enable fat LTO in release builds, while their profiling builds
|
||||
omit cross-crate LTO. Check promising changes in both configurations, especially
|
||||
when small matching benchmarks give inconsistent results:
|
||||
|
||||
```console
|
||||
CARGO_PROFILE_BENCH_LTO=fat CARGO_PROFILE_BENCH_CODEGEN_UNITS=16 cargo +nightly bench -p ignore --bench workloads
|
||||
```
|
||||
|
||||
For focused comparisons, pass one benchmark name with `--exact` so that each
|
||||
case runs in a separate process:
|
||||
|
||||
```console
|
||||
cargo +nightly bench -p ignore --bench workloads -- --exact incremental_ignored_parent
|
||||
```
|
||||
@@ -0,0 +1,587 @@
|
||||
//! File discovery and incremental matching workloads used by Python tools.
|
||||
#![feature(test)]
|
||||
|
||||
extern crate test;
|
||||
|
||||
use std::{
|
||||
fs,
|
||||
hint::black_box,
|
||||
path::{Path, PathBuf},
|
||||
sync::{
|
||||
Mutex,
|
||||
atomic::{AtomicUsize, Ordering},
|
||||
},
|
||||
time::{SystemTime, UNIX_EPOCH},
|
||||
};
|
||||
|
||||
use ignore::{
|
||||
DirEntry, Error, ParallelVisitor, ParallelVisitorBuilder, WalkBuilder,
|
||||
WalkState, gitignore::GitignoreBuilder, overrides::OverrideBuilder,
|
||||
types::TypesBuilder,
|
||||
};
|
||||
|
||||
const RULES: &str = "\
|
||||
.venv/
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*.egg-info/
|
||||
/build/
|
||||
/dist/
|
||||
*.log
|
||||
";
|
||||
|
||||
/// A deterministic source tree, created before timing starts.
|
||||
struct Workspace {
|
||||
temp: PathBuf,
|
||||
root: PathBuf,
|
||||
paths: Vec<PathBuf>,
|
||||
python_files: usize,
|
||||
}
|
||||
|
||||
impl Workspace {
|
||||
fn new(packages: usize, depth: usize, files: usize, nested: bool) -> Self {
|
||||
static NEXT: AtomicUsize = AtomicUsize::new(0);
|
||||
let temp = std::env::temp_dir().join(format!(
|
||||
"ignore-bench-{:x}-{}",
|
||||
SystemTime::now().duration_since(UNIX_EPOCH).unwrap().as_nanos(),
|
||||
NEXT.fetch_add(1, Ordering::Relaxed),
|
||||
));
|
||||
fs::create_dir(&temp).unwrap();
|
||||
let root = temp.join("workspace");
|
||||
let mut ws = Workspace { temp, root, paths: vec![], python_files: 0 };
|
||||
ws.write(".git/info/exclude", "private/\n");
|
||||
ws.write(".gitignore", RULES);
|
||||
ws.write(".ignore", ".git/\n*.tmp\n");
|
||||
ws.write(".venv/lib/python3.13/site-packages/dependency.py", "");
|
||||
ws.write("private/internal.py", "");
|
||||
ws.write("build/generated.py", "");
|
||||
for package in 0..packages {
|
||||
let base = format!("packages/pkg{package}");
|
||||
ws.write(&format!("{base}/pyproject.toml"), "[project]\n");
|
||||
if nested {
|
||||
ws.write(
|
||||
&format!("{base}/.gitignore"),
|
||||
"generated_*.py\n!generated_keep.py\n",
|
||||
);
|
||||
}
|
||||
let mut dir = base;
|
||||
for level in 0..depth {
|
||||
dir.push_str(&format!("/level{level}"));
|
||||
for file in 0..files {
|
||||
let path = format!("{dir}/module{file}.py");
|
||||
ws.write(&path, "");
|
||||
ws.paths.push(PathBuf::from(path));
|
||||
ws.python_files += 1;
|
||||
}
|
||||
ws.write(&format!("{dir}/__pycache__/module.pyc"), "");
|
||||
ws.write(&format!("{dir}/notes.txt"), "");
|
||||
if nested {
|
||||
ws.write(&format!("{dir}/generated_drop.py"), "");
|
||||
ws.write(&format!("{dir}/generated_keep.py"), "");
|
||||
ws.python_files += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
ws
|
||||
}
|
||||
|
||||
fn write(&self, relative: &str, contents: &str) {
|
||||
let path = self.root.join(relative);
|
||||
fs::create_dir_all(path.parent().unwrap()).unwrap();
|
||||
fs::write(path, contents).unwrap();
|
||||
}
|
||||
|
||||
fn builder(&self) -> WalkBuilder {
|
||||
let mut builder = WalkBuilder::new(&self.root);
|
||||
// Keep parent discovery, but do not depend on the user's Git config.
|
||||
builder.current_dir(&self.root).hidden(false).git_global(false);
|
||||
builder
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for Workspace {
|
||||
fn drop(&mut self) {
|
||||
fs::remove_dir_all(&self.temp).unwrap();
|
||||
}
|
||||
}
|
||||
|
||||
fn is_python(entry: &DirEntry) -> bool {
|
||||
entry.file_type().is_some_and(|ft| ft.is_file())
|
||||
&& entry.path().extension().is_some_and(|ext| ext == "py")
|
||||
}
|
||||
|
||||
/// Accumulate per worker, as Ruff does, instead of contending on every file.
|
||||
struct CountBuilder<'a>(&'a Mutex<usize>);
|
||||
|
||||
impl<'s> ParallelVisitorBuilder<'s> for CountBuilder<'s> {
|
||||
fn build(&mut self) -> Box<dyn ParallelVisitor + 's> {
|
||||
Box::new(CountVisitor { total: self.0, count: 0 })
|
||||
}
|
||||
}
|
||||
|
||||
struct CountVisitor<'a> {
|
||||
total: &'a Mutex<usize>,
|
||||
count: usize,
|
||||
}
|
||||
|
||||
impl ParallelVisitor for CountVisitor<'_> {
|
||||
fn visit(&mut self, entry: Result<DirEntry, Error>) -> WalkState {
|
||||
let entry = entry.unwrap();
|
||||
assert!(entry.error().is_none());
|
||||
self.count += usize::from(is_python(&entry));
|
||||
WalkState::Continue
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for CountVisitor<'_> {
|
||||
fn drop(&mut self) {
|
||||
*self.total.lock().unwrap() += self.count;
|
||||
}
|
||||
}
|
||||
|
||||
fn walk(builder: &WalkBuilder, threads: usize) -> usize {
|
||||
if threads == 0 {
|
||||
builder
|
||||
.build()
|
||||
.map(|entry| {
|
||||
let entry = entry.unwrap();
|
||||
assert!(entry.error().is_none());
|
||||
usize::from(is_python(&entry))
|
||||
})
|
||||
.sum()
|
||||
} else {
|
||||
let total = Mutex::new(0);
|
||||
let mut counter = CountBuilder(&total);
|
||||
builder.clone().threads(threads).build_parallel().visit(&mut counter);
|
||||
total.into_inner().unwrap()
|
||||
}
|
||||
}
|
||||
|
||||
macro_rules! walk_bench {
|
||||
($name:ident, $packages:expr, $depth:expr, $files:expr, $nested:expr, $threads:expr) => {
|
||||
#[bench]
|
||||
fn $name(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new($packages, $depth, $files, $nested);
|
||||
let builder = ws.builder();
|
||||
assert_eq!(walk(&builder, $threads), ws.python_files);
|
||||
b.iter(|| {
|
||||
assert_eq!(
|
||||
walk(black_box(&builder), $threads),
|
||||
ws.python_files
|
||||
)
|
||||
});
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
walk_bench!(walk_small_serial, 4, 2, 8, false, 0);
|
||||
walk_bench!(walk_small_parallel_12, 4, 2, 8, false, 12);
|
||||
walk_bench!(walk_wide_serial, 128, 1, 32, false, 0);
|
||||
walk_bench!(walk_wide_parallel_1, 128, 1, 32, false, 1);
|
||||
walk_bench!(walk_wide_parallel_4, 128, 1, 32, false, 4);
|
||||
walk_bench!(walk_wide_parallel_12, 128, 1, 32, false, 12);
|
||||
walk_bench!(walk_deep_serial, 16, 12, 16, false, 0);
|
||||
walk_bench!(walk_deep_parallel_12, 16, 12, 16, false, 12);
|
||||
walk_bench!(walk_nested_serial, 64, 3, 16, true, 0);
|
||||
walk_bench!(walk_nested_parallel_12, 64, 3, 16, true, 12);
|
||||
|
||||
#[bench]
|
||||
fn walk_unfiltered_parallel_12(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(128, 1, 32, false);
|
||||
let mut builder = ws.builder();
|
||||
builder.standard_filters(false);
|
||||
let expected = ws.python_files + 3;
|
||||
assert_eq!(walk(&builder, 12), expected);
|
||||
b.iter(|| assert_eq!(walk(black_box(&builder), 12), expected));
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn walk_explicit_files(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(64, 1, 8, false);
|
||||
let mut builder = WalkBuilder::from_iter(
|
||||
ws.paths.iter().map(|relative| ws.root.join(relative)),
|
||||
);
|
||||
builder.current_dir(&ws.root).hidden(false).git_global(false).threads(12);
|
||||
b.iter(|| assert_eq!(walk(black_box(&builder), 12), ws.python_files));
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn uv_ignored_workspace_member(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(1, 1, 0, false);
|
||||
let member = ws.root.join("packages/pkg0/level0");
|
||||
fs::remove_file(member.join("notes.txt")).unwrap();
|
||||
let mut builder = WalkBuilder::new(&member);
|
||||
builder.hidden(false).git_global(false).current_dir(&ws.root);
|
||||
b.iter(|| {
|
||||
// uv stops as soon as it encounters a non-directory entry.
|
||||
assert!(!builder.build().any(|entry| !entry.unwrap().path().is_dir()));
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn uv_nonempty_workspace_member(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(1, 1, 32, false);
|
||||
let member = ws.root.join("packages/pkg0/level0");
|
||||
let mut builder = WalkBuilder::new(&member);
|
||||
builder.hidden(false).git_global(false).current_dir(&ws.root);
|
||||
b.iter(|| {
|
||||
assert!(builder.build().any(|entry| !entry.unwrap().path().is_dir()));
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn uv_scripts_source_archive(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(64, 3, 16, true);
|
||||
fs::remove_dir_all(ws.root.join(".git")).unwrap();
|
||||
let mut builder = ws.builder();
|
||||
builder.require_git(false);
|
||||
// Without .git/info/exclude, private/internal.py is also visible.
|
||||
let expected = ws.python_files + 1;
|
||||
assert_eq!(walk(&builder, 0), expected);
|
||||
b.iter(|| assert_eq!(walk(black_box(&builder), 0), expected));
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn incremental_cold(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(16, 4, 4, true);
|
||||
let builder = ws.builder();
|
||||
b.iter(|| {
|
||||
let mut matchers = builder.build_matchers();
|
||||
for path in &ws.paths {
|
||||
assert!(!matchers[0].matched(black_box(path), false).is_ignore());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn incremental_warm(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(16, 4, 4, true);
|
||||
let mut matchers = ws.builder().build_matchers();
|
||||
for path in &ws.paths {
|
||||
assert!(!matchers[0].matched(path, false).is_ignore());
|
||||
}
|
||||
b.iter(|| {
|
||||
for path in &ws.paths {
|
||||
assert!(!matchers[0].matched(black_box(path), false).is_ignore());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn incremental_warm_noncanonical(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(16, 4, 4, true);
|
||||
let paths: Vec<_> = ws
|
||||
.paths
|
||||
.iter()
|
||||
.map(|path| {
|
||||
Path::new("packages/.")
|
||||
.join(path.strip_prefix("packages").unwrap())
|
||||
})
|
||||
.collect();
|
||||
let mut matchers = ws.builder().build_matchers();
|
||||
for path in &paths {
|
||||
assert!(!matchers[0].matched(path, false).is_ignore());
|
||||
}
|
||||
b.iter(|| {
|
||||
for path in &paths {
|
||||
assert!(!matchers[0].matched(black_box(path), false).is_ignore());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn incremental_ignored(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(0, 0, 0, false);
|
||||
let mut matchers = ws.builder().build_matchers();
|
||||
let path = Path::new(".venv/lib/python3.13/site-packages/dependency.py");
|
||||
assert!(matchers[0].matched(path, false).is_ignore());
|
||||
b.iter(|| {
|
||||
assert!(matchers[0].matched(black_box(path), false).is_ignore())
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn build_empty_gitignore(b: &mut test::Bencher) {
|
||||
let builder = GitignoreBuilder::new("workspace/packages/example");
|
||||
b.iter(|| black_box(builder.build().unwrap()));
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn build_single_directory_whitelist(b: &mut test::Bencher) {
|
||||
let mut builder = GitignoreBuilder::new("workspace");
|
||||
builder.add_line(None, "!/.github/").unwrap();
|
||||
b.iter(|| black_box(builder.build().unwrap()));
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn build_python_gitignore(b: &mut test::Bencher) {
|
||||
let mut builder = GitignoreBuilder::new("workspace");
|
||||
for line in RULES.lines() {
|
||||
builder.add_line(None, line).unwrap();
|
||||
}
|
||||
b.iter(|| black_box(builder.build().unwrap()));
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn match_python_gitignore(b: &mut test::Bencher) {
|
||||
let mut builder = GitignoreBuilder::new("workspace");
|
||||
for line in RULES.lines() {
|
||||
builder.add_line(None, line).unwrap();
|
||||
}
|
||||
let matcher = builder.build().unwrap();
|
||||
b.iter(|| {
|
||||
assert!(
|
||||
matcher
|
||||
.matched(black_box("workspace/pkg/mod.py"), false)
|
||||
.is_none()
|
||||
);
|
||||
assert!(
|
||||
matcher
|
||||
.matched(black_box("workspace/pkg/mod.pyc"), false)
|
||||
.is_ignore()
|
||||
);
|
||||
assert!(
|
||||
matcher
|
||||
.matched(black_box("workspace/pkg/__pycache__"), true)
|
||||
.is_ignore()
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn incremental_warm_deep(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(16, 12, 4, true);
|
||||
let mut matchers = ws.builder().build_matchers();
|
||||
for path in &ws.paths {
|
||||
assert!(!matchers[0].matched(path, false).is_ignore());
|
||||
}
|
||||
b.iter(|| {
|
||||
for path in &ws.paths {
|
||||
assert!(!matchers[0].matched(black_box(path), false).is_ignore());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn incremental_ignored_nested(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(1, 1, 1, false);
|
||||
let path = Path::new(
|
||||
"packages/pkg0/level0/.venv/lib/python3.13/site-packages/dependency.py",
|
||||
);
|
||||
ws.write(path.to_str().unwrap(), "");
|
||||
let mut matchers = ws.builder().build_matchers();
|
||||
assert!(matchers[0].matched(path, false).is_ignore());
|
||||
b.iter(|| {
|
||||
assert!(matchers[0].matched(black_box(path), false).is_ignore())
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn incremental_ignored_parent(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(0, 0, 0, false);
|
||||
let mut matchers = ws.builder().build_matchers();
|
||||
let path = Path::new(".venv/dependency.py");
|
||||
assert!(matchers[0].matched(path, false).is_ignore());
|
||||
b.iter(|| {
|
||||
assert!(matchers[0].matched(black_box(path), false).is_ignore())
|
||||
});
|
||||
}
|
||||
|
||||
/// Create a deep tree with ignore rules at every package level.
|
||||
fn dense_ignore_workspace() -> Workspace {
|
||||
let ws = Workspace::new(16, 12, 16, true);
|
||||
for package in 0..16 {
|
||||
let mut dir = format!("packages/pkg{package}");
|
||||
for level in 0..12 {
|
||||
dir.push_str(&format!("/level{level}"));
|
||||
ws.write(
|
||||
&format!("{dir}/.gitignore"),
|
||||
"generated_*.py\n!generated_keep.py\n",
|
||||
);
|
||||
}
|
||||
}
|
||||
ws
|
||||
}
|
||||
|
||||
fn dense_ignore_walk(b: &mut test::Bencher, threads: usize) {
|
||||
let ws = dense_ignore_workspace();
|
||||
let builder = ws.builder();
|
||||
assert_eq!(walk(&builder, threads), ws.python_files);
|
||||
b.iter(|| assert_eq!(walk(black_box(&builder), threads), ws.python_files));
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn walk_dense_ignore_serial(b: &mut test::Bencher) {
|
||||
dense_ignore_walk(b, 0);
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn walk_dense_ignore_parallel_12(b: &mut test::Bencher) {
|
||||
dense_ignore_walk(b, 12);
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn incremental_dense_ignore_cold(b: &mut test::Bencher) {
|
||||
let ws = dense_ignore_workspace();
|
||||
let builder = ws.builder();
|
||||
b.iter(|| {
|
||||
let mut matchers = builder.build_matchers();
|
||||
for path in &ws.paths {
|
||||
assert!(!matchers[0].matched(black_box(path), false).is_ignore());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn incremental_dense_ignore_warm(b: &mut test::Bencher) {
|
||||
let ws = dense_ignore_workspace();
|
||||
let mut matchers = ws.builder().build_matchers();
|
||||
for path in &ws.paths {
|
||||
assert!(!matchers[0].matched(path, false).is_ignore());
|
||||
}
|
||||
b.iter(|| {
|
||||
for path in &ws.paths {
|
||||
assert!(!matchers[0].matched(black_box(path), false).is_ignore());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
fn multiple_roots_walk(b: &mut test::Bencher, threads: usize) {
|
||||
let ws = Workspace::new(64, 1, 32, false);
|
||||
let mut builder = WalkBuilder::new(ws.root.join("packages/pkg0"));
|
||||
builder.current_dir(&ws.root).hidden(false).git_global(false);
|
||||
for package in 1..64 {
|
||||
builder.add(ws.root.join(format!("packages/pkg{package}")));
|
||||
}
|
||||
assert_eq!(walk(&builder, threads), ws.python_files);
|
||||
b.iter(|| assert_eq!(walk(black_box(&builder), threads), ws.python_files));
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn walk_multiple_roots_parallel_1(b: &mut test::Bencher) {
|
||||
multiple_roots_walk(b, 1);
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn walk_multiple_roots_parallel_12(b: &mut test::Bencher) {
|
||||
multiple_roots_walk(b, 12);
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn walk_with_overrides_and_types(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(64, 3, 16, true);
|
||||
let mut overrides = OverrideBuilder::new(&ws.root);
|
||||
overrides.add("!*.txt").unwrap();
|
||||
let mut types = TypesBuilder::new();
|
||||
types.add("python", "*.py").unwrap();
|
||||
types.select("python");
|
||||
let mut builder = ws.builder();
|
||||
builder
|
||||
.overrides(overrides.build().unwrap())
|
||||
.types(types.build().unwrap());
|
||||
assert_eq!(walk(&builder, 12), ws.python_files);
|
||||
b.iter(|| assert_eq!(walk(black_box(&builder), 12), ws.python_files));
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn match_single_file_glob(b: &mut test::Bencher) {
|
||||
let mut builder = GitignoreBuilder::new("");
|
||||
builder.add_line(None, "*.pyc").unwrap();
|
||||
let matcher = builder.build().unwrap();
|
||||
let paths = [
|
||||
("module.py", false),
|
||||
("module.pyc", true),
|
||||
("src/module.py", false),
|
||||
("src/__pycache__/module.pyc", true),
|
||||
];
|
||||
b.iter(|| {
|
||||
for (path, ignored) in paths {
|
||||
assert_eq!(
|
||||
matcher.matched(black_box(path), false).is_ignore(),
|
||||
ignored
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn match_single_directory_whitelist(b: &mut test::Bencher) {
|
||||
let mut builder = GitignoreBuilder::new("");
|
||||
builder.add_line(None, "!/.github/").unwrap();
|
||||
let matcher = builder.build().unwrap();
|
||||
let paths = [
|
||||
(".github", true, true),
|
||||
(".github", false, false),
|
||||
(".github/workflows/check.yml", false, false),
|
||||
("src/module.py", false, false),
|
||||
("src", true, false),
|
||||
];
|
||||
b.iter(|| {
|
||||
for (path, is_dir, whitelisted) in paths {
|
||||
assert_eq!(
|
||||
matcher.matched(black_box(path), is_dir).is_whitelist(),
|
||||
whitelisted
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn match_single_path_whitelist(b: &mut test::Bencher) {
|
||||
let mut builder = GitignoreBuilder::new("");
|
||||
builder.add_line(None, "!/vendor/typeshed/**/*").unwrap();
|
||||
let matcher = builder.build().unwrap();
|
||||
let paths = [
|
||||
("vendor/typeshed/stdlib/builtins.pyi", true),
|
||||
("vendor/typeshed/stubs/pkg/__init__.pyi", true),
|
||||
("src/lib.rs", false),
|
||||
("vendor/other/module.py", false),
|
||||
];
|
||||
b.iter(|| {
|
||||
for (path, whitelisted) in paths {
|
||||
assert_eq!(
|
||||
matcher.matched(black_box(path), false).is_whitelist(),
|
||||
whitelisted
|
||||
);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn walk_gitignore_only_parallel_12(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(128, 1, 32, false);
|
||||
fs::remove_file(ws.root.join(".ignore")).unwrap();
|
||||
fs::remove_file(ws.root.join(".git/info/exclude")).unwrap();
|
||||
let builder = ws.builder();
|
||||
// The private module is visible without .git/info/exclude.
|
||||
let expected = ws.python_files + 1;
|
||||
assert_eq!(walk(&builder, 12), expected);
|
||||
b.iter(|| assert_eq!(walk(black_box(&builder), 12), expected));
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn incremental_gitignore_only(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(16, 4, 4, false);
|
||||
fs::remove_file(ws.root.join(".ignore")).unwrap();
|
||||
fs::remove_file(ws.root.join(".git/info/exclude")).unwrap();
|
||||
let mut matchers = ws.builder().build_matchers();
|
||||
for path in &ws.paths {
|
||||
assert!(!matchers[0].matched(path, false).is_ignore());
|
||||
}
|
||||
b.iter(|| {
|
||||
for path in &ws.paths {
|
||||
assert!(!matchers[0].matched(black_box(path), false).is_ignore());
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
#[bench]
|
||||
fn walk_selected_python_files(b: &mut test::Bencher) {
|
||||
let ws = Workspace::new(128, 1, 32, false);
|
||||
let mut overrides = OverrideBuilder::new(&ws.root);
|
||||
overrides.add("*.py").unwrap().add("*.pyi").unwrap();
|
||||
let mut builder = ws.builder();
|
||||
builder.overrides(overrides.build().unwrap());
|
||||
assert_eq!(walk(&builder, 12), ws.python_files);
|
||||
b.iter(|| assert_eq!(walk(black_box(&builder), 12), ws.python_files));
|
||||
}
|
||||
Reference in new issue
Block a user