mirror of
https://github.com/zed-industries/zed.git
synced 2026-11-19 06:59:22 +01:00
gpui: Add seeded benchmarks and heap allocation metrics (#64843)
Two additions to GPUI's benchmark harness, split out of #64842 so the randomized element tree harness can build on them. ## Seeded `#[gpui::bench]` benchmarks A benchmark that takes a `StdRng` parameter runs once per seed, and each seed is its own Criterion benchmark named `<input>/seed-<n>` (or `<function>/seed-<n>` without inputs). Seeds come from `calculate_seeds`, moved out of `test.rs` into `seeds.rs` so it compiles under `bench-support` and is shared with `#[gpui::test]`. `seed = N`, `seeds(...)`, `iterations = N`, `SEED` and `ITERATIONS` therefore mean the same in tests and benchmarks, and one `SEED` reproduces either. The RNG is rebuilt on every Criterion routine call, so warm-up and every sample measure the same workload. The generated code seeds it through `gpui::private::rand`, so a bench crate needs `rand` only to name `StdRng` in its signature. ```rust #[gpui::bench(inputs = families(), input_name = "tree", iterations = 6)] fn full_refresh(family: &Family, mut rng: StdRng, cx: &mut BenchAppContext) { ... } ``` The macro now also checks the benchmark's parameters, and reports a wrong parameter count, a `&mut StdRng`, or `seed`/`seeds`/`iterations` without a `StdRng` parameter as clear errors. **Behavior change in `gpui::test`:** with `SEED` set and `iterations > 1`, `calculate_seeds` returned `SEED` and then `SEED..SEED+iterations`, so `SEED=7 ITERATIONS=2` ran 7, 7, 8. Criterion rejects the duplicate benchmark ID, and the function's comment already described `SEED..SEED+iterations`, so it now runs 7, 8. ## Heap allocation metrics in `bench_metrics` - `CountingAllocator<A = System>` wraps any global allocator and counts allocations, allocated bytes and freed bytes in sharded relaxed atomics, with no backtraces. Each thread also counts its own allocations in plain thread-locals. - `AllocationCounter` measures allocations or allocated bytes per iteration over a `MetricScope`, like `HardwareCounter`. When the allocator is installed, `allocations`, `foreground allocations` and `allocated bytes` are default secondaries. `BENCH_MEASUREMENT=allocations` or `foreground-allocations` makes one of them the primary. Without the allocator they're skipped with a one-time note, like unavailable hardware counters. - `foreground allocations` leaves out background workers' timing noise. It can still vary between samples when the workload does, for example when frames coalesce or caches grow in early samples, so a CI gate on it needs a small tolerance. - `allocation_stats()` exposes the totals and `live_bytes()` for heap held around setup and teardown. - `gpui::bench_main!` installs `CountingAllocator` over the system allocator. `gpui::bench_main!(allocator = mimalloc::MiMalloc; benches)` wraps another allocator instead, and `allocator = none;` installs nothing. Example, `inspector_render` smallest case: `allocations per iteration: median 558`, `allocated bytes per iteration: median 390 K`. ## Notes for reviewers - The counting allocator is installed in every GPUI bench binary. It adds a few relaxed atomic increments per allocation, so wall-time baselines saved before this PR shift slightly. - Placements in GPUI's element arena aren't heap allocations, so they aren't counted once the arena is warm. Arena bytes per iteration could be a GPUI secondary in a follow-up. ## Validation - Throwaway seeded bench (inputs `a`/`b`, `iterations = 2`, `seeds(40)`): correct seed order by default and with `SEED=7`, `SEED=7 ITERATIONS=2`, `ITERATIONS=3`; every routine call drew the same first value for a seed; the allocations primary reported exactly 1 allocation / 64 bytes per iteration for `vec![0u8; 64]`. A second throwaway bench imported `StdRng` through `gpui::private::rand` with no `rand` dependency of its own and ran `probe/seed-0` and `probe/seed-1`. - `inspector_render` ran with the default allocator and with `allocator = none;` (which prints the note and reports no allocation lines), and built with `allocator = std::alloc::System;`. - New unit tests for seed order, process-wide allocation counting, and exact calling-thread counts that exclude other threads; `bench_metrics` and `bench_context` tests pass. - `./script/clippy` and `cargo fmt` clean; `--no-run` builds of `gpui_platform` and `benchmarks` benches. Release Notes: - [GPUI] Added seeded `#[gpui::bench]` benchmarks that take a `StdRng`, and heap allocation counts in benchmark reports
This commit is contained in:
@@ -132,9 +132,19 @@ The attribute supports options such as:
|
||||
|
||||
Inspect the current macro before depending on an option because the API is evolving.
|
||||
|
||||
Register benchmark functions with `gpui::bench_group!` and `gpui::bench_main!` (drop-in for Criterion's `criterion_group!`/`criterion_main!`, including the `name = ...; config = ...; targets = ...` form). The generated functions take `&mut criterion::Criterion<gpui::BenchMeasurement>`; `bench_group!` supplies that measurement from the environment.
|
||||
For randomized workloads, take a `StdRng` parameter (by value) and draw the workload from it instead of hand-rolling seeds. The macro then runs one Criterion benchmark per seed, named `<input>/seed-<n>`, with `seed = N`, `seeds(...)`, `iterations = N`, `SEED`, and `ITERATIONS` meaning exactly what they do for `#[gpui::test]`, so `SEED=3` reproduces the same workload in a test and a benchmark. The RNG is rebuilt for every Criterion routine call, so warm-up and every sample see the same workload.
|
||||
|
||||
### Measurements: wall time, hardware counters, and rusage
|
||||
```rust
|
||||
#[gpui::bench(inputs = tree_families(), input_name = "tree", iterations = 6)]
|
||||
fn full_refresh(family: &TreeFamily, mut rng: StdRng, cx: &mut gpui::BenchAppContext) {
|
||||
let tree = family.sample(&mut rng);
|
||||
// ...
|
||||
}
|
||||
```
|
||||
|
||||
Register benchmark functions with `gpui::bench_group!` and `gpui::bench_main!` (drop-in for Criterion's `criterion_group!`/`criterion_main!`, including the `name = ...; config = ...; targets = ...` form). The generated functions take `&mut criterion::Criterion<gpui::BenchMeasurement>`; `bench_group!` supplies that measurement from the environment. `bench_main!` also installs `gpui::CountingAllocator` as the binary's global allocator, so a GPUI bench file must not declare its own `#[global_allocator]`; to measure with another allocator, write `gpui::bench_main!(allocator = mimalloc::MiMalloc; benches)`, which wraps it. `allocator = none;` installs no allocator and drops the allocation metrics.
|
||||
|
||||
### Measurements: wall time, hardware counters, rusage, and allocations
|
||||
|
||||
`gpui::BenchMeasurement` (from the gpui-independent `bench_metrics` crate) records several metrics per Criterion sample. Criterion analyzes one of them (the *primary*, shown on its `time:` line with confidence intervals and baseline comparison); the rest are *secondaries*, printed per iteration in the GPUI bench report as median/min/max. `BENCH_MEASUREMENT` selects the arrangement:
|
||||
|
||||
@@ -143,6 +153,8 @@ Register benchmark functions with `gpui::bench_group!` and `gpui::bench_main!` (
|
||||
| unset (default) | wall time | every counter the machine supports (below) |
|
||||
| `instructions` | process-wide retired instructions; fails fast if counters are unavailable | the rest, plus wall time |
|
||||
| `foreground-instructions` | the benchmark thread's retired instructions; fails fast if unavailable | the rest, plus wall time |
|
||||
| `allocations` | process-wide heap allocations; fails fast without `CountingAllocator` | the rest, plus wall time |
|
||||
| `foreground-allocations` | the benchmark thread's heap allocations, without background workers' timing noise; fails fast without `CountingAllocator` | the rest, plus wall time |
|
||||
| `wall-time` | wall time | nothing; no counters are opened |
|
||||
|
||||
For a CI gate on frame cost prefer `foreground-instructions`: it excludes background parsing and GPU-driver submission threads, whose work varies run to run, and on Apple Silicon the process-wide count includes driver jitter that widens the interval (about ±0.6% versus ±0.01% for the foreground thread).
|
||||
@@ -154,6 +166,7 @@ Default secondaries, each skipped with a one-time note when unavailable:
|
||||
- **branch misses** and **cache misses** (last-level), for diagnosing an IPC drop. Linux only.
|
||||
- **foreground context switches**: voluntary switches on the benchmark thread per iteration, i.e. how many times a frame blocked on a lock, channel, or fence. This is the metric that catches the waits instructions cannot see. Linux only.
|
||||
- **page faults**: minor faults per iteration, a privilege-free proxy for fresh memory touched.
|
||||
- **allocations**, **foreground allocations**, and **allocated bytes**: heap allocations (reallocations count as one each) across every thread and on the benchmark thread alone, and the bytes requested across every thread, per iteration. Background workers allocate on their own schedule, so the foreground count excludes that noise. It can still vary between samples when the workload does, e.g. frames coalescing or caches growing in early samples, so a CI gate needs a small tolerance. Arena placements (GPUI's element arena) are not heap allocations and are not counted once the arena is warm. Needs `bench_metrics::CountingAllocator` as the global allocator, which `gpui::bench_main!` installs; it wraps any allocator (`CountingAllocator<MiMalloc>`) and costs a few relaxed atomics per allocation. For heap held between iterations, read `bench_metrics::allocation_stats()` before and after setup; `live_bytes()` is allocated minus freed.
|
||||
|
||||
Run one benchmark process per machine at a time. Hardware counters are per task, so instruction counts stay correct next to a concurrent `cargo bench`, but wall time, IPC, and cache misses share the hardware and degrade, and two processes requesting counters can exceed a core's counter slots, at which point the kernel multiplexes and the counts become scaled estimates (a warning is printed once). Within one process, benchmarks already run sequentially; process-wide counters can still attribute a previous benchmark's trailing background work to the next benchmark's first sample, which the foreground-thread metrics are immune to.
|
||||
|
||||
@@ -182,7 +195,7 @@ group.bench_function("append", |b| report.iter(b, || rope.append(small.clone()))
|
||||
report.print(" ");
|
||||
```
|
||||
|
||||
Criterion's `Throughput` lines keep working alongside. Page faults per iteration are a useful allocation-pressure signal for data-structure crates even without a counting allocator.
|
||||
Criterion's `Throughput` lines keep working alongside. Page faults per iteration are a useful allocation-pressure signal even without a counting allocator; for exact counts, install `bench_metrics::CountingAllocator` as the binary's `#[global_allocator]`.
|
||||
|
||||
### `bench_iter`: synchronous functions and compute work
|
||||
|
||||
|
||||
@@ -6,8 +6,9 @@
|
||||
//! measurements taken over the same iterations, whose per-iteration values a
|
||||
//! benchmark harness records into a [`MetricReport`]. The measurements here
|
||||
//! are process- or thread-scoped counters: Linux hardware performance
|
||||
//! counters ([`HardwareCounter`]) and `getrusage` statistics
|
||||
//! ([`ResourceCounter`]). Nothing in this crate depends on GPUI, so any
|
||||
//! counters ([`HardwareCounter`]), `getrusage` statistics
|
||||
//! ([`ResourceCounter`]), and heap allocations counted by
|
||||
//! [`CountingAllocator`] ([`AllocationCounter`]). Nothing in this crate depends on GPUI, so any
|
||||
//! Criterion benchmark can use it:
|
||||
//!
|
||||
//! ```ignore
|
||||
@@ -992,6 +993,319 @@ impl Measurement for ResourceCounter {
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Heap allocations
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/// A global allocator that forwards to `A` and counts what passes through it,
|
||||
/// for [`AllocationCounter`] and [`allocation_stats`].
|
||||
///
|
||||
/// Counting records no backtraces, so it adds a few relaxed atomic increments
|
||||
/// per allocation and is cheap enough to install in every benchmark binary.
|
||||
/// `A` can be any allocator, so a binary that wants to measure the allocator it
|
||||
/// ships with wraps that one:
|
||||
///
|
||||
/// ```ignore
|
||||
/// #[global_allocator]
|
||||
/// static ALLOCATOR: bench_metrics::CountingAllocator<mimalloc::MiMalloc> =
|
||||
/// bench_metrics::CountingAllocator::new(mimalloc::MiMalloc);
|
||||
/// ```
|
||||
///
|
||||
/// A binary has exactly one global allocator, so this can't be installed where a
|
||||
/// dependency already sets one (such as `ztracing` built with
|
||||
/// `ztracing_with_memory`); allocation metrics are then skipped.
|
||||
pub struct CountingAllocator<A = std::alloc::System> {
|
||||
inner: A,
|
||||
}
|
||||
|
||||
impl<A> CountingAllocator<A> {
|
||||
/// Wraps `inner`, which serves every allocation.
|
||||
pub const fn new(inner: A) -> Self {
|
||||
Self { inner }
|
||||
}
|
||||
}
|
||||
|
||||
// SAFETY: every method forwards to `inner` with the caller's arguments and
|
||||
// returns its result unchanged; counting never touches the memory.
|
||||
unsafe impl<A: std::alloc::GlobalAlloc> std::alloc::GlobalAlloc for CountingAllocator<A> {
|
||||
unsafe fn alloc(&self, layout: std::alloc::Layout) -> *mut u8 {
|
||||
// SAFETY: the caller upholds `GlobalAlloc::alloc`'s contract.
|
||||
let pointer = unsafe { self.inner.alloc(layout) };
|
||||
if !pointer.is_null() {
|
||||
allocation_counts::record_allocation(layout.size());
|
||||
}
|
||||
pointer
|
||||
}
|
||||
|
||||
unsafe fn alloc_zeroed(&self, layout: std::alloc::Layout) -> *mut u8 {
|
||||
// SAFETY: the caller upholds `GlobalAlloc::alloc_zeroed`'s contract.
|
||||
let pointer = unsafe { self.inner.alloc_zeroed(layout) };
|
||||
if !pointer.is_null() {
|
||||
allocation_counts::record_allocation(layout.size());
|
||||
}
|
||||
pointer
|
||||
}
|
||||
|
||||
unsafe fn dealloc(&self, pointer: *mut u8, layout: std::alloc::Layout) {
|
||||
// SAFETY: the caller upholds `GlobalAlloc::dealloc`'s contract.
|
||||
unsafe { self.inner.dealloc(pointer, layout) };
|
||||
allocation_counts::record_free(layout.size());
|
||||
}
|
||||
|
||||
unsafe fn realloc(
|
||||
&self,
|
||||
pointer: *mut u8,
|
||||
layout: std::alloc::Layout,
|
||||
new_size: usize,
|
||||
) -> *mut u8 {
|
||||
// SAFETY: the caller upholds `GlobalAlloc::realloc`'s contract.
|
||||
let new_pointer = unsafe { self.inner.realloc(pointer, layout, new_size) };
|
||||
// A reallocation is counted as a new allocation replacing the old one,
|
||||
// as dhat and heaptrack count it, so growing a `Vec` shows up as churn.
|
||||
if !new_pointer.is_null() {
|
||||
allocation_counts::record_free(layout.size());
|
||||
allocation_counts::record_allocation(new_size);
|
||||
}
|
||||
new_pointer
|
||||
}
|
||||
}
|
||||
|
||||
/// Totals counted by [`CountingAllocator`] since the process started.
|
||||
#[derive(Clone, Copy, Debug, Default, Eq, PartialEq)]
|
||||
pub struct AllocationStats {
|
||||
/// Allocations served, including reallocations.
|
||||
pub allocations: u64,
|
||||
/// Bytes requested by those allocations.
|
||||
pub allocated_bytes: u64,
|
||||
/// Bytes returned by frees and by the old side of reallocations.
|
||||
pub freed_bytes: u64,
|
||||
}
|
||||
|
||||
impl AllocationStats {
|
||||
/// Bytes allocated and not yet freed.
|
||||
pub fn live_bytes(&self) -> i64 {
|
||||
self.allocated_bytes as i64 - self.freed_bytes as i64
|
||||
}
|
||||
}
|
||||
|
||||
/// Returns the process's allocation totals, or `None` when this binary's
|
||||
/// global allocator is not a [`CountingAllocator`]. Differences between two
|
||||
/// calls give the allocations, bytes, and live-heap change of the code between
|
||||
/// them, across every thread.
|
||||
pub fn allocation_stats() -> Option<AllocationStats> {
|
||||
allocation_counts::installed().then(allocation_counts::read)
|
||||
}
|
||||
|
||||
/// Counters shared by every thread, plus each thread's own. Each thread
|
||||
/// increments one of several cache-line-aligned shards, so threads allocating
|
||||
/// concurrently rarely contend on a line; readers sum the shards. The
|
||||
/// per-thread counts are plain cells only their own thread reads.
|
||||
mod allocation_counts {
|
||||
use super::AllocationStats;
|
||||
use std::{
|
||||
cell::Cell,
|
||||
sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering},
|
||||
};
|
||||
|
||||
const SHARD_COUNT: usize = 32;
|
||||
|
||||
#[repr(align(128))]
|
||||
struct Shard {
|
||||
allocations: AtomicU64,
|
||||
allocated_bytes: AtomicU64,
|
||||
freed_bytes: AtomicU64,
|
||||
}
|
||||
|
||||
static SHARDS: [Shard; SHARD_COUNT] = [const {
|
||||
Shard {
|
||||
allocations: AtomicU64::new(0),
|
||||
allocated_bytes: AtomicU64::new(0),
|
||||
freed_bytes: AtomicU64::new(0),
|
||||
}
|
||||
}; SHARD_COUNT];
|
||||
static INSTALLED: AtomicBool = AtomicBool::new(false);
|
||||
static NEXT_SHARD: AtomicUsize = AtomicUsize::new(0);
|
||||
|
||||
thread_local! {
|
||||
// `const` and without `Drop`, so accessing these never allocates and
|
||||
// still works while the thread's other thread-locals are being destroyed.
|
||||
static SHARD_INDEX: Cell<Option<usize>> = const { Cell::new(None) };
|
||||
static THREAD_ALLOCATIONS: Cell<u64> = const { Cell::new(0) };
|
||||
static THREAD_ALLOCATED_BYTES: Cell<u64> = const { Cell::new(0) };
|
||||
}
|
||||
|
||||
fn shard() -> &'static Shard {
|
||||
let index = SHARD_INDEX
|
||||
.try_with(|index| match index.get() {
|
||||
Some(index) => index,
|
||||
None => {
|
||||
let next = NEXT_SHARD.fetch_add(1, Ordering::Relaxed) % SHARD_COUNT;
|
||||
index.set(Some(next));
|
||||
next
|
||||
}
|
||||
})
|
||||
.unwrap_or(0);
|
||||
&SHARDS[index % SHARD_COUNT]
|
||||
}
|
||||
|
||||
pub(super) fn record_allocation(size: usize) {
|
||||
// Loaded first so the flag's cache line stays shared across threads
|
||||
// instead of being written on every allocation.
|
||||
if !INSTALLED.load(Ordering::Relaxed) {
|
||||
INSTALLED.store(true, Ordering::Relaxed);
|
||||
}
|
||||
let shard = shard();
|
||||
shard.allocations.fetch_add(1, Ordering::Relaxed);
|
||||
shard
|
||||
.allocated_bytes
|
||||
.fetch_add(size as u64, Ordering::Relaxed);
|
||||
// Fails only while the thread is exiting, when nothing reads its counts.
|
||||
THREAD_ALLOCATIONS
|
||||
.try_with(|count| count.set(count.get() + 1))
|
||||
.ok();
|
||||
THREAD_ALLOCATED_BYTES
|
||||
.try_with(|bytes| bytes.set(bytes.get() + size as u64))
|
||||
.ok();
|
||||
}
|
||||
|
||||
pub(super) fn record_free(size: usize) {
|
||||
shard()
|
||||
.freed_bytes
|
||||
.fetch_add(size as u64, Ordering::Relaxed);
|
||||
}
|
||||
|
||||
pub(super) fn installed() -> bool {
|
||||
INSTALLED.load(Ordering::Relaxed)
|
||||
}
|
||||
|
||||
/// The calling thread's allocations and allocated bytes; frees are not
|
||||
/// tracked per thread, since a block is often freed by another thread.
|
||||
pub(super) fn read_calling_thread() -> AllocationStats {
|
||||
AllocationStats {
|
||||
allocations: THREAD_ALLOCATIONS.try_with(Cell::get).unwrap_or(0),
|
||||
allocated_bytes: THREAD_ALLOCATED_BYTES.try_with(Cell::get).unwrap_or(0),
|
||||
freed_bytes: 0,
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn read() -> AllocationStats {
|
||||
SHARDS
|
||||
.iter()
|
||||
.fold(AllocationStats::default(), |total, shard| AllocationStats {
|
||||
allocations: total.allocations + shard.allocations.load(Ordering::Relaxed),
|
||||
allocated_bytes: total.allocated_bytes
|
||||
+ shard.allocated_bytes.load(Ordering::Relaxed),
|
||||
freed_bytes: total.freed_bytes + shard.freed_bytes.load(Ordering::Relaxed),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
/// What [`AllocationCounter`] counts.
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub enum AllocationMetric {
|
||||
/// Allocations served, including reallocations: the churn put through the
|
||||
/// allocator, and nearly deterministic for the same work.
|
||||
Allocations,
|
||||
/// Bytes requested by those allocations.
|
||||
AllocatedBytes,
|
||||
}
|
||||
|
||||
impl AllocationMetric {
|
||||
/// The unit noun used in reports.
|
||||
pub fn unit(self) -> &'static str {
|
||||
match self {
|
||||
Self::Allocations => "allocations",
|
||||
Self::AllocatedBytes => "bytes",
|
||||
}
|
||||
}
|
||||
|
||||
fn read(self, stats: &AllocationStats) -> u64 {
|
||||
match self {
|
||||
Self::Allocations => stats.allocations,
|
||||
Self::AllocatedBytes => stats.allocated_bytes,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Criterion measurement for heap allocations. Needs [`CountingAllocator`]
|
||||
/// installed as the binary's global allocator.
|
||||
///
|
||||
/// [`MetricScope::Process`] counts every thread. [`MetricScope::CallingThread`]
|
||||
/// counts only the thread that runs the measurement, which in GPUI benchmarks
|
||||
/// is the foreground thread. Background workers allocate on their own
|
||||
/// schedule, so the calling thread's count is the steadier regression gate.
|
||||
pub struct AllocationCounter {
|
||||
metric: AllocationMetric,
|
||||
scope: MetricScope,
|
||||
formatter: CountFormatter,
|
||||
}
|
||||
|
||||
impl AllocationCounter {
|
||||
/// Creates a counter for `metric` over `scope`, failing when the binary's
|
||||
/// global allocator is not a [`CountingAllocator`].
|
||||
pub fn new(metric: AllocationMetric, scope: MetricScope) -> Result<Self> {
|
||||
if !allocation_counts::installed() {
|
||||
return Err(anyhow!(
|
||||
"this binary's global allocator is not bench_metrics::CountingAllocator \
|
||||
(GPUI benchmarks install it through gpui::bench_main! unless they pass \
|
||||
`allocator = none`)"
|
||||
));
|
||||
}
|
||||
Ok(Self {
|
||||
metric,
|
||||
scope,
|
||||
formatter: CountFormatter::new(metric.unit()),
|
||||
})
|
||||
}
|
||||
|
||||
/// The counted statistic.
|
||||
pub fn metric(&self) -> AllocationMetric {
|
||||
self.metric
|
||||
}
|
||||
|
||||
/// The covered threads.
|
||||
pub fn scope(&self) -> MetricScope {
|
||||
self.scope
|
||||
}
|
||||
|
||||
fn read(&self) -> u64 {
|
||||
let stats = match self.scope {
|
||||
MetricScope::Process => allocation_counts::read(),
|
||||
MetricScope::CallingThread => allocation_counts::read_calling_thread(),
|
||||
};
|
||||
self.metric.read(&stats)
|
||||
}
|
||||
}
|
||||
|
||||
impl Measurement for AllocationCounter {
|
||||
type Intermediate = u64;
|
||||
type Value = f64;
|
||||
|
||||
fn start(&self) -> Self::Intermediate {
|
||||
self.read()
|
||||
}
|
||||
|
||||
fn end(&self, start: Self::Intermediate) -> Self::Value {
|
||||
self.read().saturating_sub(start) as f64
|
||||
}
|
||||
|
||||
fn add(&self, first: &Self::Value, second: &Self::Value) -> Self::Value {
|
||||
first + second
|
||||
}
|
||||
|
||||
fn zero(&self) -> Self::Value {
|
||||
0.0
|
||||
}
|
||||
|
||||
fn to_f64(&self, value: &Self::Value) -> f64 {
|
||||
*value
|
||||
}
|
||||
|
||||
fn formatter(&self) -> &dyn ValueFormatter {
|
||||
&self.formatter
|
||||
}
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Composite measurement
|
||||
// ---------------------------------------------------------------------------
|
||||
@@ -1090,31 +1404,41 @@ impl BenchMeasurement {
|
||||
/// supports is a secondary: process and calling-thread instructions,
|
||||
/// cycles, and their ratios where [`HardwareCounter`] has a backend
|
||||
/// (Linux, Apple Silicon); branch and cache misses on Linux; voluntary
|
||||
/// context switches and minor page faults from `getrusage`. Counters
|
||||
/// that fail to open are skipped with a note printed once.
|
||||
/// context switches and minor page faults from `getrusage`; heap
|
||||
/// allocations (process-wide and calling-thread) and allocated bytes
|
||||
/// where the binary installs [`CountingAllocator`]. Counters that fail to open are skipped with a
|
||||
/// note printed once.
|
||||
/// * `wall-time`: wall time only, no counters.
|
||||
/// * `instructions`: Criterion analyzes process-wide retired instructions.
|
||||
/// * `foreground-instructions`: Criterion analyzes the benchmark thread's
|
||||
/// retired instructions, which excludes background and driver threads
|
||||
/// and is the tighter regression gate for frame cost.
|
||||
/// * `allocations`: Criterion analyzes process-wide heap allocations.
|
||||
/// * `foreground-allocations`: Criterion analyzes the benchmark thread's
|
||||
/// heap allocations, which excludes background workers' timing noise.
|
||||
///
|
||||
/// In the instruction modes wall time becomes a secondary, and construction
|
||||
/// fails when counters are unavailable so CI does not silently measure
|
||||
/// something else.
|
||||
/// In the instruction and allocation modes wall time becomes a secondary,
|
||||
/// and construction fails when the primary is unavailable so CI does not
|
||||
/// silently measure something else.
|
||||
pub fn from_env() -> Result<Self> {
|
||||
match std::env::var(BENCH_MEASUREMENT_ENV_VAR).as_deref() {
|
||||
Err(std::env::VarError::NotPresent) => {
|
||||
let measurement = Self::new(criterion::measurement::WallTime);
|
||||
Ok(measurement.with_default_secondaries(None))
|
||||
Ok(measurement.with_default_secondaries(Primary::WallTime))
|
||||
}
|
||||
Ok("wall-time") => Ok(Self::new(criterion::measurement::WallTime)),
|
||||
Ok("instructions") => Self::with_instructions_primary(MetricScope::Process),
|
||||
Ok("foreground-instructions") => {
|
||||
Self::with_instructions_primary(MetricScope::CallingThread)
|
||||
}
|
||||
Ok("allocations") => Self::with_allocations_primary(MetricScope::Process),
|
||||
Ok("foreground-allocations") => {
|
||||
Self::with_allocations_primary(MetricScope::CallingThread)
|
||||
}
|
||||
Ok(value) => Err(anyhow!(
|
||||
"unsupported {BENCH_MEASUREMENT_ENV_VAR} value {value:?}; expected \"wall-time\", \
|
||||
\"instructions\", or \"foreground-instructions\""
|
||||
\"instructions\", \"foreground-instructions\", \"allocations\", or \
|
||||
\"foreground-allocations\""
|
||||
)),
|
||||
Err(error) => Err(anyhow!(
|
||||
"{BENCH_MEASUREMENT_ENV_VAR} is not valid Unicode: {error}"
|
||||
@@ -1141,14 +1465,25 @@ impl BenchMeasurement {
|
||||
};
|
||||
Ok(Self::new(instructions)
|
||||
.named(name)
|
||||
.with_default_secondaries(Some(scope))
|
||||
.with_default_secondaries(Primary::Instructions(scope))
|
||||
.with_secondary("wall time", criterion::measurement::WallTime))
|
||||
}
|
||||
|
||||
/// Adds every counter this machine supports, leaving out the instruction
|
||||
/// count for `primary_instructions` when that scope is already the
|
||||
/// primary.
|
||||
fn with_default_secondaries(mut self, primary_instructions: Option<MetricScope>) -> Self {
|
||||
fn with_allocations_primary(scope: MetricScope) -> Result<Self> {
|
||||
let allocations = AllocationCounter::new(AllocationMetric::Allocations, scope)?;
|
||||
Ok(Self::new(allocations)
|
||||
.named(allocations_name(scope))
|
||||
.with_default_secondaries(Primary::Allocations(scope))
|
||||
.with_secondary("wall time", criterion::measurement::WallTime))
|
||||
}
|
||||
|
||||
/// Adds every counter this machine supports, leaving out the one that is
|
||||
/// already the primary.
|
||||
fn with_default_secondaries(mut self, primary: Primary) -> Self {
|
||||
let primary_instructions = match primary {
|
||||
Primary::Instructions(scope) => Some(scope),
|
||||
Primary::WallTime | Primary::Allocations(_) => None,
|
||||
};
|
||||
// Every hardware counter needs the same access, so a denial of the
|
||||
// first one is reported once for all of them rather than once per
|
||||
// event. Events a platform lacks are still noted individually.
|
||||
@@ -1217,7 +1552,45 @@ impl BenchMeasurement {
|
||||
ResourceMetric::MinorPageFaults,
|
||||
MetricScope::Process,
|
||||
);
|
||||
self
|
||||
let allocation =
|
||||
|this: Self, name, metric, scope| match AllocationCounter::new(metric, scope) {
|
||||
Ok(counter) => this.with_secondary(name, counter),
|
||||
Err(error) => {
|
||||
note_unavailable("heap allocations", &error);
|
||||
this
|
||||
}
|
||||
};
|
||||
for scope in [MetricScope::Process, MetricScope::CallingThread] {
|
||||
if primary != Primary::Allocations(scope) {
|
||||
self = allocation(
|
||||
self,
|
||||
allocations_name(scope),
|
||||
AllocationMetric::Allocations,
|
||||
scope,
|
||||
);
|
||||
}
|
||||
}
|
||||
allocation(
|
||||
self,
|
||||
"allocated bytes",
|
||||
AllocationMetric::AllocatedBytes,
|
||||
MetricScope::Process,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
/// The metric Criterion analyzes in a [`BenchMeasurement::from_env`] mode.
|
||||
#[derive(Clone, Copy, PartialEq)]
|
||||
enum Primary {
|
||||
WallTime,
|
||||
Instructions(MetricScope),
|
||||
Allocations(MetricScope),
|
||||
}
|
||||
|
||||
fn allocations_name(scope: MetricScope) -> &'static str {
|
||||
match scope {
|
||||
MetricScope::Process => "allocations",
|
||||
MetricScope::CallingThread => "foreground allocations",
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1597,6 +1970,66 @@ mod tests {
|
||||
atomic::{AtomicU64, Ordering},
|
||||
};
|
||||
|
||||
#[global_allocator]
|
||||
static ALLOCATOR: CountingAllocator = CountingAllocator::new(std::alloc::System);
|
||||
|
||||
#[test]
|
||||
fn counting_allocator_counts_allocations_bytes_and_frees() {
|
||||
// Other tests allocate concurrently and the counts are process-wide, so
|
||||
// these assert lower bounds.
|
||||
let allocations =
|
||||
AllocationCounter::new(AllocationMetric::Allocations, MetricScope::Process)
|
||||
.expect("this test binary installs CountingAllocator");
|
||||
let bytes = AllocationCounter::new(AllocationMetric::AllocatedBytes, MetricScope::Process)
|
||||
.expect("this test binary installs CountingAllocator");
|
||||
let before = allocation_stats().expect("CountingAllocator is installed");
|
||||
let allocations_start = allocations.start();
|
||||
let bytes_start = bytes.start();
|
||||
|
||||
let mut buffers = Vec::with_capacity(10);
|
||||
for _ in 0..10 {
|
||||
buffers.push(std::hint::black_box(vec![0_u8; 1 << 16]));
|
||||
}
|
||||
let during = allocation_stats().expect("CountingAllocator is installed");
|
||||
drop(std::hint::black_box(buffers));
|
||||
|
||||
assert!(allocations.end(allocations_start) >= 11.0);
|
||||
assert!(bytes.end(bytes_start) >= (10 << 16) as f64);
|
||||
let after = allocation_stats().expect("CountingAllocator is installed");
|
||||
assert!(after.freed_bytes - during.freed_bytes >= 10 << 16);
|
||||
assert!(during.allocated_bytes - before.allocated_bytes >= 10 << 16);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn calling_thread_allocations_are_exact_and_exclude_other_threads() {
|
||||
let allocations =
|
||||
AllocationCounter::new(AllocationMetric::Allocations, MetricScope::CallingThread)
|
||||
.expect("this test binary installs CountingAllocator");
|
||||
let bytes =
|
||||
AllocationCounter::new(AllocationMetric::AllocatedBytes, MetricScope::CallingThread)
|
||||
.expect("this test binary installs CountingAllocator");
|
||||
let allocations_start = allocations.start();
|
||||
let bytes_start = bytes.start();
|
||||
let buffers: Vec<Vec<u8>> = (0..10).map(|_| vec![0_u8; 100]).collect();
|
||||
std::hint::black_box(&buffers);
|
||||
assert_eq!(allocations.end(allocations_start), 11.0);
|
||||
assert_eq!(
|
||||
bytes.end(bytes_start),
|
||||
(10 * 100 + 10 * std::mem::size_of::<Vec<u8>>()) as f64
|
||||
);
|
||||
|
||||
let start = allocations.start();
|
||||
std::thread::spawn(|| std::hint::black_box(vec![vec![0_u8; 100]; 1000]))
|
||||
.join()
|
||||
.expect("allocating thread should finish");
|
||||
let spawn_overhead = allocations.end(start);
|
||||
assert!(
|
||||
spawn_overhead < 100.0,
|
||||
"another thread's 1001 allocations should not count toward the calling thread: \
|
||||
{spawn_overhead}"
|
||||
);
|
||||
}
|
||||
|
||||
fn fast_criterion(measurement: BenchMeasurement) -> criterion::Criterion<BenchMeasurement> {
|
||||
criterion::Criterion::default()
|
||||
.with_measurement(measurement)
|
||||
|
||||
@@ -22,8 +22,8 @@ use slotmap::SlotMap;
|
||||
pub use async_context::*;
|
||||
#[cfg(feature = "bench-support")]
|
||||
pub use bench_context::{
|
||||
BenchAppContext, BenchMeasurement, BenchReport, BenchWindowContext, MetricReport,
|
||||
bench_platform,
|
||||
BenchAppContext, BenchMeasurement, BenchReport, BenchWindowContext, CountingAllocator,
|
||||
MetricReport, bench_platform,
|
||||
};
|
||||
use collections::{FxHashMap, FxHashSet, HashMap, TypeIdHashMap, TypeIdHashSet, VecDeque};
|
||||
pub use context::*;
|
||||
|
||||
@@ -10,7 +10,7 @@ use std::{
|
||||
};
|
||||
|
||||
use anyhow::{Result, anyhow};
|
||||
pub use bench_metrics::{BenchMeasurement, MetricReport};
|
||||
pub use bench_metrics::{BenchMeasurement, CountingAllocator, MetricReport};
|
||||
use hdrhistogram::Histogram;
|
||||
|
||||
use crate::{
|
||||
|
||||
+29
-3
@@ -47,6 +47,8 @@ pub mod profiler;
|
||||
#[expect(missing_docs)]
|
||||
pub mod queue;
|
||||
mod scene;
|
||||
#[cfg(any(test, feature = "test-support", feature = "bench-support"))]
|
||||
mod seeds;
|
||||
mod shared_uri;
|
||||
mod spring;
|
||||
mod style;
|
||||
@@ -64,6 +66,8 @@ mod window;
|
||||
|
||||
#[cfg(any(test, feature = "test-support"))]
|
||||
pub use proptest;
|
||||
#[cfg(any(test, feature = "test-support", feature = "bench-support"))]
|
||||
pub use seeds::calculate_seeds;
|
||||
|
||||
#[cfg(doc)]
|
||||
pub mod _accessibility;
|
||||
@@ -75,6 +79,7 @@ pub mod _ownership_and_data_flow;
|
||||
pub mod private {
|
||||
pub use anyhow;
|
||||
pub use inventory;
|
||||
pub use rand;
|
||||
pub use schemars;
|
||||
pub use serde;
|
||||
pub use serde_json;
|
||||
@@ -147,11 +152,32 @@ macro_rules! bench_group {
|
||||
/// Defines the entry point for GPUI Criterion benchmark groups.
|
||||
///
|
||||
/// This mirrors `criterion::criterion_main!` so GPUI benchmark files can keep the
|
||||
/// same shape as ordinary Criterion benchmarks.
|
||||
/// same shape as ordinary Criterion benchmarks. It also installs
|
||||
/// [`CountingAllocator`] as the binary's global allocator, so reports include
|
||||
/// heap allocations per iteration. It wraps the system allocator by default;
|
||||
/// `allocator = Path;` wraps another unit-struct allocator instead, e.g. to
|
||||
/// compare allocators, which pay the same small counting cost:
|
||||
///
|
||||
/// ```ignore
|
||||
/// gpui::bench_main!(allocator = mimalloc::MiMalloc; benches);
|
||||
/// ```
|
||||
///
|
||||
/// `allocator = none;` installs no global allocator, leaving Rust's default and
|
||||
/// dropping the allocation metrics from reports.
|
||||
#[macro_export]
|
||||
macro_rules! bench_main {
|
||||
($($tokens:tt)*) => {
|
||||
criterion::criterion_main!($($tokens)*);
|
||||
(allocator = none; $($groups:tt)*) => {
|
||||
criterion::criterion_main!($($groups)*);
|
||||
};
|
||||
(allocator = $allocator:path; $($groups:tt)*) => {
|
||||
#[global_allocator]
|
||||
static GPUI_BENCH_ALLOCATOR: $crate::CountingAllocator<$allocator> =
|
||||
$crate::CountingAllocator::new($allocator);
|
||||
|
||||
criterion::criterion_main!($($groups)*);
|
||||
};
|
||||
($($groups:tt)*) => {
|
||||
$crate::bench_main!(allocator = ::std::alloc::System; $($groups)*);
|
||||
};
|
||||
}
|
||||
pub use gpui_shared_string::*;
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
use std::env;
|
||||
|
||||
/// Returns the seeds a randomized `#[gpui::test]` or `#[gpui::bench]` runs with, in
|
||||
/// order, and whether there is more than one.
|
||||
///
|
||||
/// With `$SEED` set, runs `$SEED..$SEED + iterations` and ignores `explicit_seeds`.
|
||||
/// Otherwise runs `0..iterations` followed by `explicit_seeds`, except that the
|
||||
/// default single iteration is dropped when explicit seeds are given.
|
||||
/// `$ITERATIONS` overrides `iterations`.
|
||||
/// Tests and benchmarks share this so that one `SEED` reproduces either.
|
||||
///
|
||||
/// This is intended for use by the `gpui::test` and `gpui::bench` macros and generally
|
||||
/// should not be used directly.
|
||||
#[doc(hidden)]
|
||||
pub fn calculate_seeds(
|
||||
iterations: u64,
|
||||
explicit_seeds: &[u64],
|
||||
) -> (impl Iterator<Item = u64> + '_, bool) {
|
||||
let iterations = env::var("ITERATIONS")
|
||||
.ok()
|
||||
.map(|var| var.parse().expect("invalid ITERATIONS variable"))
|
||||
.unwrap_or(iterations);
|
||||
|
||||
let env_seed = env::var("SEED")
|
||||
.map(|seed| seed.parse().expect("invalid SEED variable as integer"))
|
||||
.ok();
|
||||
|
||||
let iter = seeds(iterations, explicit_seeds, env_seed);
|
||||
let is_multiple_runs = iter.clone().nth(1).is_some();
|
||||
(iter, is_multiple_runs)
|
||||
}
|
||||
|
||||
fn seeds(
|
||||
iterations: u64,
|
||||
explicit_seeds: &[u64],
|
||||
env_seed: Option<u64>,
|
||||
) -> impl Iterator<Item = u64> + Clone + '_ {
|
||||
let (iterations_range, explicit_seeds) = match env_seed {
|
||||
Some(env_seed) => (env_seed..env_seed + iterations, &[][..]),
|
||||
None if iterations == 1 && !explicit_seeds.is_empty() => (0..0, explicit_seeds),
|
||||
None => (0..iterations, explicit_seeds),
|
||||
};
|
||||
iterations_range.chain(explicit_seeds.iter().copied())
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::seeds;
|
||||
|
||||
fn collect(iterations: u64, explicit_seeds: &[u64], env_seed: Option<u64>) -> Vec<u64> {
|
||||
seeds(iterations, explicit_seeds, env_seed).collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn seeds_follow_iterations_explicit_seeds_and_seed_variable() {
|
||||
assert_eq!(collect(1, &[], None), [0]);
|
||||
assert_eq!(collect(3, &[], None), [0, 1, 2]);
|
||||
assert_eq!(collect(1, &[10, 20], None), [10, 20]);
|
||||
assert_eq!(collect(2, &[10], None), [0, 1, 10]);
|
||||
assert_eq!(collect(1, &[10], Some(7)), [7]);
|
||||
assert_eq!(
|
||||
collect(2, &[10], Some(7)),
|
||||
[7, 8],
|
||||
"`SEED` starts the iterations once, without repeating itself"
|
||||
);
|
||||
}
|
||||
}
|
||||
+1
-49
@@ -25,7 +25,7 @@
|
||||
//! assert!(true)
|
||||
//! }
|
||||
//! ```
|
||||
use crate::{Entity, Subscription, TestAppContext, TestDispatcher};
|
||||
use crate::{Entity, Subscription, TestAppContext, TestDispatcher, calculate_seeds};
|
||||
use futures::StreamExt as _;
|
||||
use proptest::prelude::{Just, Strategy, any};
|
||||
use std::{
|
||||
@@ -141,54 +141,6 @@ pub fn run_test(
|
||||
}
|
||||
}
|
||||
|
||||
fn calculate_seeds(
|
||||
iterations: u64,
|
||||
explicit_seeds: &[u64],
|
||||
) -> (impl Iterator<Item = u64> + '_, bool) {
|
||||
let iterations = env::var("ITERATIONS")
|
||||
.ok()
|
||||
.map(|var| var.parse().expect("invalid ITERATIONS variable"))
|
||||
.unwrap_or(iterations);
|
||||
|
||||
let env_num = env::var("SEED")
|
||||
.map(|seed| seed.parse().expect("invalid SEED variable as integer"))
|
||||
.ok();
|
||||
|
||||
let empty_range = || 0..0;
|
||||
|
||||
let iter = {
|
||||
let env_range = if let Some(env_num) = env_num {
|
||||
env_num..env_num + 1
|
||||
} else {
|
||||
empty_range()
|
||||
};
|
||||
|
||||
// if `iterations` is 1 and !(`explicit_seeds` is non-empty || `SEED` is set), then add the run `0`
|
||||
// if `iterations` is 1 and (`explicit_seeds` is non-empty || `SEED` is set), then discard the run `0`
|
||||
// if `iterations` isn't 1 and `SEED` is set, do `SEED..SEED+iterations`
|
||||
// otherwise, do `0..iterations`
|
||||
let iterations_range = match (iterations, env_num) {
|
||||
(1, None) if explicit_seeds.is_empty() => 0..1,
|
||||
(1, None) | (1, Some(_)) => empty_range(),
|
||||
(iterations, Some(env)) => env..env + iterations,
|
||||
(iterations, None) => 0..iterations,
|
||||
};
|
||||
|
||||
// if `SEED` is set, ignore `explicit_seeds`
|
||||
let explicit_seeds = if env_num.is_some() {
|
||||
&[]
|
||||
} else {
|
||||
explicit_seeds
|
||||
};
|
||||
|
||||
env_range
|
||||
.chain(iterations_range)
|
||||
.chain(explicit_seeds.iter().copied())
|
||||
};
|
||||
let is_multiple_runs = iter.clone().nth(1).is_some();
|
||||
(iter, is_multiple_runs)
|
||||
}
|
||||
|
||||
/// A test struct for converting an observation callback into a stream.
|
||||
pub struct Observation<T> {
|
||||
rx: Pin<Box<async_channel::Receiver<T>>>,
|
||||
|
||||
+177
-43
@@ -1,6 +1,9 @@
|
||||
use proc_macro::TokenStream;
|
||||
use quote::{format_ident, quote};
|
||||
use syn::{Expr, ItemFn, LitStr, parse::Parser, spanned::Spanned};
|
||||
use syn::{
|
||||
Expr, FnArg, ItemFn, LitInt, LitStr, Token, Type, parse::Parser, punctuated::Punctuated,
|
||||
spanned::Spanned,
|
||||
};
|
||||
|
||||
pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {
|
||||
let mut fps: Option<u64> = None;
|
||||
@@ -8,9 +11,40 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {
|
||||
let mut input_name: Option<LitStr> = None;
|
||||
let mut group_name: Option<LitStr> = None;
|
||||
let mut sample_size: Option<usize> = None;
|
||||
let mut explicit_seeds: Option<Vec<u64>> = None;
|
||||
let mut iterations: Option<u64> = None;
|
||||
if !args.is_empty() {
|
||||
let parser = syn::meta::parser(|meta| {
|
||||
if meta.path.is_ident("fps") {
|
||||
if meta.path.is_ident("seed") || meta.path.is_ident("seeds") {
|
||||
if explicit_seeds.is_some() {
|
||||
return Err(meta
|
||||
.error("#[gpui::bench] accepts one of `seed = N` or `seeds(...)`, once"));
|
||||
}
|
||||
let seeds = if meta.path.is_ident("seed") {
|
||||
vec![meta.value()?.parse::<LitInt>()?]
|
||||
} else {
|
||||
let content;
|
||||
syn::parenthesized!(content in meta.input);
|
||||
Punctuated::<LitInt, Token![,]>::parse_terminated(&content)?
|
||||
.into_iter()
|
||||
.collect()
|
||||
};
|
||||
explicit_seeds = Some(
|
||||
seeds
|
||||
.iter()
|
||||
.map(|seed| seed.base10_parse::<u64>())
|
||||
.collect::<syn::Result<_>>()?,
|
||||
);
|
||||
Ok(())
|
||||
} else if meta.path.is_ident("iterations") {
|
||||
let value: LitInt = meta.value()?.parse()?;
|
||||
let value = value.base10_parse::<u64>()?;
|
||||
if value == 0 {
|
||||
return Err(meta.error("#[gpui::bench] `iterations` must be greater than zero"));
|
||||
}
|
||||
iterations = Some(value);
|
||||
Ok(())
|
||||
} else if meta.path.is_ident("fps") {
|
||||
let value: syn::LitInt = meta.value()?.parse()?;
|
||||
let value = value.base10_parse::<u64>()?;
|
||||
if value == 0 {
|
||||
@@ -39,7 +73,7 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {
|
||||
Ok(())
|
||||
} else {
|
||||
Err(meta.error(
|
||||
"#[gpui::bench] only accepts `fps = N`, `inputs = EXPR`, `input_name = \"...\"`, `group = \"...\"`, and `sample_size = N`",
|
||||
"#[gpui::bench] only accepts `fps = N`, `inputs = EXPR`, `input_name = \"...\"`, `group = \"...\"`, `sample_size = N`, `seed = N`, `seeds(...)`, and `iterations = N`",
|
||||
))
|
||||
}
|
||||
});
|
||||
@@ -71,6 +105,105 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {
|
||||
let inner_fn_name = format_ident!("__gpui_bench_{}", outer_fn_name);
|
||||
inner_fn.sig.ident = inner_fn_name.clone();
|
||||
|
||||
// A `StdRng` parameter may appear anywhere; the others are, in order, the input
|
||||
// (when `inputs` is given) and the benchmark context.
|
||||
let expected_signature = if inputs.is_some() {
|
||||
"`(input: &Input, cx: &mut BenchAppContext)`"
|
||||
} else {
|
||||
"`(cx: &mut BenchAppContext)`"
|
||||
};
|
||||
let mut call_arguments = Vec::new();
|
||||
let mut takes_rng = false;
|
||||
let mut positional_count = 0;
|
||||
for argument in &inner_fn.sig.inputs {
|
||||
let FnArg::Typed(argument) = argument else {
|
||||
return error_to_stream(syn::Error::new(
|
||||
argument.span(),
|
||||
"#[gpui::bench] functions cannot take `self`",
|
||||
));
|
||||
};
|
||||
if is_std_rng(&argument.ty) {
|
||||
if takes_rng {
|
||||
return error_to_stream(syn::Error::new(
|
||||
argument.span(),
|
||||
"#[gpui::bench] provides one `StdRng`",
|
||||
));
|
||||
}
|
||||
takes_rng = true;
|
||||
call_arguments.push(quote! { rng });
|
||||
continue;
|
||||
}
|
||||
if let Type::Reference(reference) = &*argument.ty
|
||||
&& is_std_rng(&reference.elem)
|
||||
{
|
||||
return error_to_stream(syn::Error::new(
|
||||
argument.span(),
|
||||
"#[gpui::bench] provides `StdRng` by value; take `rng: StdRng`",
|
||||
));
|
||||
}
|
||||
call_arguments.push(match (inputs.is_some(), positional_count) {
|
||||
(true, 0) => quote! { input },
|
||||
(true, 1) | (false, 0) => quote! { &mut cx },
|
||||
_ => {
|
||||
return error_to_stream(syn::Error::new(
|
||||
argument.span(),
|
||||
format!(
|
||||
"#[gpui::bench] expected {expected_signature}, optionally with a `StdRng`"
|
||||
),
|
||||
));
|
||||
}
|
||||
});
|
||||
positional_count += 1;
|
||||
}
|
||||
if positional_count != if inputs.is_some() { 2 } else { 1 } {
|
||||
return error_to_stream(syn::Error::new(
|
||||
inner_fn.sig.inputs.span(),
|
||||
format!("#[gpui::bench] expected {expected_signature}, optionally with a `StdRng`"),
|
||||
));
|
||||
}
|
||||
if !takes_rng && (explicit_seeds.is_some() || iterations.is_some()) {
|
||||
return error_to_stream(syn::Error::new(
|
||||
inner_fn.sig.span(),
|
||||
"#[gpui::bench] `seed`, `seeds`, and `iterations` require a `StdRng` parameter",
|
||||
));
|
||||
}
|
||||
|
||||
// Criterion calls the routine several times (warm-up, then each sample), so the
|
||||
// context and the RNG are rebuilt on every call: each one measures the same seed.
|
||||
let rng = takes_rng
|
||||
.then(|| quote! { let rng = gpui::private::rand::SeedableRng::seed_from_u64(seed); });
|
||||
let routine = quote! {
|
||||
#rng
|
||||
let mut cx = gpui::BenchAppContext::new_with_platform_and_report(
|
||||
gpui::bench_platform(
|
||||
Some(Box::new(|| {
|
||||
gpui_platform::current_headless_renderer()
|
||||
})),
|
||||
gpui_platform::current_platform(true).text_system(),
|
||||
),
|
||||
Some(stringify!(#outer_fn_name)),
|
||||
bencher,
|
||||
report.clone(),
|
||||
);
|
||||
#inner_fn_name(#(#call_arguments),*);
|
||||
cx.teardown();
|
||||
};
|
||||
// Each seed is its own benchmark, named after it, so a baseline recorded under
|
||||
// `SEED=n` compares against the same tree next time.
|
||||
let for_each_seed = |run: proc_macro2::TokenStream| {
|
||||
if takes_rng {
|
||||
let iterations = iterations.unwrap_or(1);
|
||||
let explicit_seeds = explicit_seeds.clone().unwrap_or_default();
|
||||
quote! {
|
||||
for seed in gpui::calculate_seeds(#iterations, &[#(#explicit_seeds),*]).0 {
|
||||
#run
|
||||
}
|
||||
}
|
||||
} else {
|
||||
run
|
||||
}
|
||||
};
|
||||
|
||||
let benchmark = if let Some(inputs) = inputs {
|
||||
let input_name = match input_name {
|
||||
Some(input_name) => quote! { #input_name },
|
||||
@@ -82,33 +215,30 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {
|
||||
};
|
||||
let sample_size =
|
||||
sample_size.map(|sample_size| quote! { group.sample_size(#sample_size); });
|
||||
let parameter = if takes_rng {
|
||||
quote! { format!("{}/seed-{}", input, seed) }
|
||||
} else {
|
||||
quote! { input.to_string() }
|
||||
};
|
||||
let run = for_each_seed(quote! {
|
||||
// One report per benchmark: per-iteration metrics differ across
|
||||
// inputs and seeds, so blending them would make the summary meaningless.
|
||||
let report = #report_expr;
|
||||
let parameter = #parameter;
|
||||
let report_name = format!("{}/{}/{}", #group_name, #input_name, parameter);
|
||||
group.bench_with_input(criterion::BenchmarkId::new(#input_name, ¶meter), &input, {
|
||||
let report = report.clone();
|
||||
move |bencher, input| {
|
||||
#routine
|
||||
}
|
||||
});
|
||||
report.print(&report_name);
|
||||
});
|
||||
quote! {
|
||||
let mut group = criterion.benchmark_group(#group_name);
|
||||
#sample_size
|
||||
for input in #inputs {
|
||||
// One report per input: per-iteration metrics differ across
|
||||
// inputs, so blending them would make the summary meaningless.
|
||||
let report = #report_expr;
|
||||
let report_name = format!("{}/{}/{}", #group_name, #input_name, input);
|
||||
group.bench_with_input(criterion::BenchmarkId::new(#input_name, &input), &input, {
|
||||
let report = report.clone();
|
||||
move |bencher, input| {
|
||||
let mut cx = gpui::BenchAppContext::new_with_platform_and_report(
|
||||
gpui::bench_platform(
|
||||
Some(Box::new(|| {
|
||||
gpui_platform::current_headless_renderer()
|
||||
})),
|
||||
gpui_platform::current_platform(true).text_system(),
|
||||
),
|
||||
Some(stringify!(#outer_fn_name)),
|
||||
bencher,
|
||||
report.clone(),
|
||||
);
|
||||
#inner_fn_name(input, &mut cx);
|
||||
cx.teardown();
|
||||
}
|
||||
});
|
||||
report.print(&report_name);
|
||||
#run
|
||||
}
|
||||
group.finish();
|
||||
}
|
||||
@@ -131,28 +261,22 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {
|
||||
"#[gpui::bench] `sample_size` requires `inputs`",
|
||||
));
|
||||
}
|
||||
quote! {
|
||||
let name = if takes_rng {
|
||||
quote! { format!("{}/seed-{}", stringify!(#outer_fn_name), seed) }
|
||||
} else {
|
||||
quote! { stringify!(#outer_fn_name).to_string() }
|
||||
};
|
||||
for_each_seed(quote! {
|
||||
let report = #report_expr;
|
||||
criterion.bench_function(stringify!(#outer_fn_name), {
|
||||
let name = #name;
|
||||
criterion.bench_function(&name, {
|
||||
let report = report.clone();
|
||||
move |bencher| {
|
||||
let mut cx = gpui::BenchAppContext::new_with_platform_and_report(
|
||||
gpui::bench_platform(
|
||||
Some(Box::new(|| {
|
||||
gpui_platform::current_headless_renderer()
|
||||
})),
|
||||
gpui_platform::current_platform(true).text_system(),
|
||||
),
|
||||
Some(stringify!(#outer_fn_name)),
|
||||
bencher,
|
||||
report.clone(),
|
||||
);
|
||||
#inner_fn_name(&mut cx);
|
||||
cx.teardown();
|
||||
#routine
|
||||
}
|
||||
});
|
||||
report.print(stringify!(#outer_fn_name));
|
||||
}
|
||||
report.print(&name);
|
||||
})
|
||||
};
|
||||
|
||||
TokenStream::from(quote! {
|
||||
@@ -165,6 +289,16 @@ pub fn bench(args: TokenStream, function: TokenStream) -> TokenStream {
|
||||
})
|
||||
}
|
||||
|
||||
fn is_std_rng(ty: &Type) -> bool {
|
||||
let Type::Path(path) = ty else {
|
||||
return false;
|
||||
};
|
||||
path.path
|
||||
.segments
|
||||
.last()
|
||||
.is_some_and(|segment| segment.ident == "StdRng")
|
||||
}
|
||||
|
||||
fn error_to_stream(error: syn::Error) -> TokenStream {
|
||||
TokenStream::from(error.into_compile_error())
|
||||
}
|
||||
|
||||
@@ -197,6 +197,24 @@ pub fn test(args: TokenStream, function: TokenStream) -> TokenStream {
|
||||
/// `bench_with_input`. `group`, `input_name`, and `sample_size` can customize
|
||||
/// the generated input benchmark group.
|
||||
///
|
||||
/// A benchmark that takes a `StdRng` parameter (by value, in any position) is run
|
||||
/// once per seed, with the RNG seeded from it. Seeds are chosen exactly as for
|
||||
/// `#[gpui::test]`: `seed = N`, `seeds(...)`, and `iterations = N` take the same
|
||||
/// meaning, and the `SEED` and `ITERATIONS` environment variables override them the
|
||||
/// same way, so one `SEED` reproduces a test or a benchmark. Each seed is its own
|
||||
/// Criterion benchmark, named `<input>/seed-<seed>` (or `<function>/seed-<seed>`
|
||||
/// without inputs), and the RNG is rebuilt for every Criterion routine call so all
|
||||
/// samples measure the same seed. The generated code seeds it through gpui's own
|
||||
/// `rand`, so the crate needs `rand` only to name `StdRng` in the signature.
|
||||
///
|
||||
/// ```ignore
|
||||
/// #[gpui::bench(inputs = families(), iterations = 6)]
|
||||
/// fn full_refresh(family: &Family, mut rng: StdRng, cx: &mut BenchAppContext) {
|
||||
/// let tree = family.sample(&mut rng);
|
||||
/// // ...
|
||||
/// }
|
||||
/// ```
|
||||
///
|
||||
/// The benchmark crate must add `criterion` and `gpui_platform` (with its
|
||||
/// `test-support` feature) to its dev-dependencies and enable gpui's `bench`
|
||||
/// feature, since the generated code references all three.
|
||||
|
||||
Reference in New Issue
Block a user