Files
zed/crates/streaming_diff/benches/streaming_diff.rs
T
Anthony Eid e683fd7b46 gpui: Add bench_metrics for multi-metric Criterion benchmarks (#64753)
GPUI benchmarks measured only wall time, which is noisy enough on shared
CI runners that a regression must be large before Criterion flags it.
This PR adds a `bench_metrics` crate that records several metrics per
Criterion sample and reports them alongside Criterion's own analysis,
and wires GPUI's benchmark harness to it. Nothing in `bench_metrics`
depends on GPUI, so any Criterion benchmark in the workspace can adopt
it.

**What a benchmark now prints** (Linux, hardware counters available):

```
closed_inspector_grid/case/100-nodes-0-cycles
                        time:   [273.99 µs 274.78 µs 276.04 µs]
GPUI bench report (all observed iterations): closed_inspector_grid/case/100-nodes-0-cycles
  instructions per iteration: median 5.791 M instructions (min 5.788, max 5.880, samples 22)
  foreground instructions per iteration: median 5.791 M instructions (...)
  cycles per iteration: median 1.453 M cycles (...)
  branch misses per iteration: median 1.856 K branch misses (...)
  cache misses per iteration: median 123.008 cache misses (...)
  foreground context switches per iteration: median 0.000 context switches (...)
  page faults per iteration: median 0.000 page faults (...)
  IPC: median 3.996 (min 3.349, max 4.063, samples 22)
  foreground share of instructions: median 1.000 (...)
```

Criterion analyzes one metric (the *primary*; wall time by default) and
the rest are *secondaries* taken over the same iterations.
`BENCH_MEASUREMENT=instructions` makes process-wide retired instructions
the primary, and `BENCH_MEASUREMENT=foreground-instructions` the
benchmark thread's alone; both are near-deterministic, so a ~0.01%
interval is typical and small regressions are detectable, with wall time
demoted to the report. `BENCH_MEASUREMENT=wall-time` disables all
counters. Counters that can't be opened (no `CAP_PERFMON`, Docker
seccomp, VMs without a PMU, Intel Macs) are skipped with one note, so
`cargo bench` never fails by default; the instruction modes fail fast so
CI can't silently measure the wrong thing.

**Sources.** `HardwareCounter` is platform-neutral over per-OS backends.
On Linux it uses `perf_event_open`: counters are opened once per process
with `inherit` so every later thread is covered, read as deltas per
sample (~150 ns per counter), with hybrid-CPU encodings parsed from
sysfs and multiplexing scaled with a one-time warning. On Apple Silicon
it reads `proc_pid_rusage` and `thread_selfcounts`, which are
unprivileged, for instructions and cycles; branch and cache events need
`kperf` and stay Linux-only. `ResourceCounter` reads `getrusage` for
voluntary context switches on the foreground thread (the waits
instructions can't see) and minor page faults, with no privileges.

Foreground-thread instructions are counted separately from the process
total, so work shifting between GPUI's foreground and background threads
is visible even when the total is flat; with a headless renderer the gap
is driver-thread submission work (about 7% of a small Metal frame).

**Design.** `BenchMeasurement` type-erases the primary so
`#[gpui::bench]` functions and `BenchAppContext` need no type parameter;
`with_secondary`/`with_ratio` add metrics; `bench_group!` still
delegates to `criterion_group!`. Plain Criterion benches opt in with a
`config =` line and `MetricReport::iter` (documented in the gpui-bench
skill; wiring `rope` etc. is a follow-up). Reports are now emitted per
input for `inputs = ...` benchmarks, since per-iteration metrics blended
across inputs are meaningless. Criterion is bumped to 0.8.2, which also
randomizes stack alignment per sample.

**Verified** on Linux (hybrid Alder Lake) and an M-series Mac: every
mode on `inspector_render` and `editor_render`; counter denial via
seccomp on Linux; `bench_metrics` and `gpui` `bench_context` tests
including thread-scope inclusion/exclusion through real counters on both
platforms.

**Follow-ups:** counting-allocator metric (peak heap, allocations),
wiring non-GPUI benches, per-secondary baseline comparison.

Release Notes:

- [GPUI] Improved benchmarks to report retired instructions, cycles,
IPC, foreground-thread share, cache and branch misses, context switches,
and page faults per iteration alongside wall time, with
`BENCH_MEASUREMENT=instructions` or `=foreground-instructions` selecting
near-deterministic instruction counts as the Criterion-analyzed metric
for CI regression detection
2026-09-27 17:37:29 +00:00

321 lines
10 KiB
Rust

use criterion::{BatchSize, BenchmarkId, Criterion, Throughput, criterion_group, criterion_main};
use rand::{Rng as _, SeedableRng as _, rngs::StdRng};
use std::hint::black_box;
use streaming_diff::StreamingDiff;
const SEED: u64 = 0x5EED_5EED;
const CHUNK_SIZE: usize = 512;
#[derive(Clone)]
struct EditFixture {
name: &'static str,
old_text: String,
new_text: String,
}
fn streaming_diff_push_new(criterion: &mut Criterion) {
let fixtures = fixtures();
let mut group = criterion.benchmark_group("streaming_diff_push_new");
group.sample_size(10);
for fixture in fixtures {
group.throughput(Throughput::Bytes(fixture.new_text.len() as u64));
group.bench_with_input(
BenchmarkId::new(fixture.name, fixture.old_text.len()),
&fixture,
|bench, fixture| {
bench.iter_batched(
|| StreamingDiff::new(fixture.old_text.clone()),
|mut diff| {
let mut operation_count = 0;
for chunk in chunk_text(&fixture.new_text, CHUNK_SIZE) {
operation_count += black_box(diff.push_new(chunk)).len();
}
black_box(operation_count);
},
BatchSize::SmallInput,
);
},
);
}
group.finish();
}
fn streaming_diff_finish(criterion: &mut Criterion) {
let fixtures = fixtures();
let mut group = criterion.benchmark_group("streaming_diff_finish");
group.sample_size(10);
for fixture in fixtures {
group.throughput(Throughput::Bytes(fixture.new_text.len() as u64));
group.bench_with_input(
BenchmarkId::new(fixture.name, fixture.old_text.len()),
&fixture,
|bench, fixture| {
bench.iter_batched(
|| {
let mut diff = StreamingDiff::new(fixture.old_text.clone());
for chunk in chunk_text(&fixture.new_text, CHUNK_SIZE) {
black_box(diff.push_new(chunk));
}
diff
},
|diff| {
black_box(diff.finish());
},
BatchSize::SmallInput,
);
},
);
}
group.finish();
}
fn fixtures() -> Vec<EditFixture> {
// Keep fixtures modest because `StreamingDiff` is intentionally stressed here and
// can become very slow on tens of kilobytes of replacement text. These sizes still
// represent realistic `edit_file` old/new text blocks and are large enough to cross
// frame-budget-sized CPU work.
vec![
make_fixture(
"tiny_function_rewrite",
2,
EditPattern::LocalizedRewrite {
start_line: 12,
line_count: 6,
},
SEED,
),
make_fixture(
"small_function_rewrite",
5,
EditPattern::LocalizedRewrite {
start_line: 22,
line_count: 12,
},
SEED + 1,
),
make_fixture(
"medium_many_small_changes",
8,
EditPattern::ManySmallChanges { every_nth_line: 7 },
SEED + 2,
),
make_fixture(
"medium_insertions",
8,
EditPattern::InsertHelperBlocks { every_nth_line: 9 },
SEED + 3,
),
]
}
enum EditPattern {
LocalizedRewrite {
start_line: usize,
line_count: usize,
},
ManySmallChanges {
every_nth_line: usize,
},
InsertHelperBlocks {
every_nth_line: usize,
},
}
fn make_fixture(
name: &'static str,
function_count: usize,
pattern: EditPattern,
seed: u64,
) -> EditFixture {
let mut rng = StdRng::seed_from_u64(seed);
let mut lines = random_rust_module(&mut rng, function_count);
let old_text = lines.join("\n");
match pattern {
EditPattern::LocalizedRewrite {
start_line,
line_count,
} => rewrite_local_block(&mut lines, start_line, line_count, &mut rng),
EditPattern::ManySmallChanges { every_nth_line } => {
rewrite_many_small_lines(&mut lines, every_nth_line, &mut rng)
}
EditPattern::InsertHelperBlocks { every_nth_line } => {
insert_helper_blocks(&mut lines, every_nth_line, &mut rng)
}
}
EditFixture {
name,
old_text,
new_text: lines.join("\n"),
}
}
fn random_rust_module(rng: &mut StdRng, function_count: usize) -> Vec<String> {
let mut lines = vec![
"use anyhow::{Context as _, Result};".to_string(),
"use collections::HashMap;".to_string(),
"".to_string(),
"#[derive(Clone, Debug)]".to_string(),
"pub struct WorkspaceSnapshot {".to_string(),
" buffers: HashMap<String, usize>,".to_string(),
" version: usize,".to_string(),
"}".to_string(),
"".to_string(),
"impl WorkspaceSnapshot {".to_string(),
];
for function_index in 0..function_count {
let function_name = identifier(rng, function_index);
let argument_name = identifier(rng, function_index + 1_000);
let local_name = identifier(rng, function_index + 2_000);
let branch_name = identifier(rng, function_index + 3_000);
let multiplier = rng.random_range(2..17);
let offset = rng.random_range(1..128);
lines.extend([
format!(
" pub fn {function_name}(&mut self, {argument_name}: usize) -> Result<usize> {{"
),
format!(" let mut {local_name} = {argument_name}.saturating_mul({multiplier});"),
format!(" if {local_name} % 2 == 0 {{"),
format!(
" {local_name} = {local_name}.saturating_add(self.version + {offset});"
),
" } else {".to_string(),
format!(" {local_name} = {local_name}.saturating_sub({offset});"),
" }".to_string(),
format!(" let {branch_name} = self.buffers.len().saturating_add({local_name});"),
format!(" self.version = self.version.saturating_add({branch_name});"),
format!(" Ok({branch_name})"),
" }".to_string(),
"".to_string(),
]);
}
lines.push("}".to_string());
lines.push("".to_string());
lines.push("pub fn normalize_path(path: &str) -> String {".to_string());
lines.push(" path.replace('\\\\', \"/\")".to_string());
lines.push("}".to_string());
lines
}
fn rewrite_local_block(
lines: &mut [String],
start_line: usize,
line_count: usize,
rng: &mut StdRng,
) {
let end_line = (start_line + line_count).min(lines.len());
for (relative_index, line) in lines[start_line..end_line].iter_mut().enumerate() {
let suffix = identifier(rng, relative_index + 10_000);
if line.contains("saturating_add") {
*line = format!(
" let {suffix} = self.version.checked_add({relative_index}).context(\"version overflow\")?;"
);
} else if line.contains("saturating_sub") {
*line = format!(
" {suffix}.saturating_sub({});",
rng.random_range(8..256)
);
} else if line.trim().is_empty() {
*line = format!(
" tracing::trace!(target: \"agent_bench\", value = {relative_index});"
);
} else {
*line = format!("{line} // updated {suffix}");
}
}
}
fn rewrite_many_small_lines(lines: &mut [String], every_nth_line: usize, rng: &mut StdRng) {
for (line_index, line) in lines.iter_mut().enumerate() {
if line_index % every_nth_line != 0 || line.trim().is_empty() {
continue;
}
if line.contains("let mut") {
*line = line.replace("let mut", "let mut updated");
} else if line.contains("Ok(") {
*line = line.replace("Ok(", "Ok(black_box_value(");
} else if line.ends_with('{') {
*line = format!("{line} // scenario {}", identifier(rng, line_index));
} else {
*line = format!("{line} // touched {}", identifier(rng, line_index));
}
}
}
fn insert_helper_blocks(lines: &mut Vec<String>, every_nth_line: usize, rng: &mut StdRng) {
let mut line_index = every_nth_line;
while line_index < lines.len() {
if lines[line_index].trim() == "}" {
let helper_name = identifier(rng, line_index + 20_000);
lines.splice(
line_index..line_index,
[
format!(" let {helper_name} = self.buffers.len();"),
format!(" tracing::trace!(target: \"agent_bench\", {helper_name});"),
],
);
line_index += 2;
}
line_index += every_nth_line;
}
}
fn identifier(rng: &mut StdRng, salt: usize) -> String {
const WORDS: &[&str] = &[
"buffer",
"workspace",
"snapshot",
"version",
"project",
"entry",
"path",
"cursor",
"anchor",
"edit",
"thread",
"message",
"context",
"store",
"diff",
"range",
"token",
"parser",
"semantic",
"format",
"completion",
"diagnostic",
"terminal",
"channel",
];
let first = WORDS[(rng.random_range(0..WORDS.len()) + salt) % WORDS.len()];
let second = WORDS[(rng.random_range(0..WORDS.len()) + salt / 3) % WORDS.len()];
format!("{first}_{second}_{salt}")
}
fn chunk_text(text: &str, max_chunk_size: usize) -> Vec<&str> {
let mut chunks = Vec::new();
let mut start = 0;
while start < text.len() {
let mut end = (start + max_chunk_size).min(text.len());
while end < text.len() && !text.is_char_boundary(end) {
end += 1;
}
chunks.push(&text[start..end]);
start = end;
}
chunks
}
criterion_group!(benches, streaming_diff_push_new, streaming_diff_finish);
criterion_main!(benches);