mirror of
https://github.com/zed-industries/zed.git
synced 2026-11-19 06:59:22 +01:00
agent_ui: Improve model and configuration option search (#64147)
# Objective Fixes #64144. Improve search in agent model and configuration option pickers. Previously, searches could fail when punctuation differed between the query and model name, such as `GPT 5.6` versus `GPT-5.6`. Configuration options also could not be found by provider/group, description, or value identifier, and results were limited to 100 entries. ## Solution Use the existing `fuzzy_nucleo` matcher for agent model and configuration option searches. Include relevant metadata in each search candidate: - Model name and provider/group - Configuration option name, group, description, and value identifier Return all matching configuration options while preserving their existing ordering when scores are equal. Rank matches against option/model names independently of their metadata, then append metadata-only matches without duplicates. This keeps descriptions and identifiers searchable without letting their lengths demote exact name matches or reorder equally relevant names. Run the shared pure-data search pipeline on the background executor, including candidate construction, ranking, and result cloning. This prevents large catalogs and descriptions from monopolizing foreground task polls without changing the ranking rule. ## Testing - Added and ran automated tests with the following: - Searches with punctuation differences, such as `GPT 5.6` - Searches by provider/group - Searches across configuration option names, descriptions, groups, and identifiers - Searches returning more than 100 configuration options - Existing fuzzy-match ordering behavior - Exact option-name matches ahead of partial-name and metadata-only matches - Stable name ordering despite different metadata lengths - Combined queries spanning names and metadata Follow-up validation on the ranking fix: - `cargo test -p agent_ui --lib`: all 521 tests passed. Both new configuration-option ranking regression tests failed before the fix and passed afterward. - `./script/clippy -p agent_ui`: passed, including the available cargo-shear and typo checks. - `rustfmt --check --edition 2024 crates/agent_ui/src/config_options.rs crates/agent_ui/src/model_selector.rs`: passed. - `git diff --check`: passed. Additionally compiled the app with `cargo run`, launched OpenCode agent in ACP mode, and searched `GPT 5.6`, which produces `OpenAI/GPT-5.6 Sol` and `OpenRouter/GPT-5.6 Sol`. See video below. ## Self-Review Checklist: - [x] I've reviewed my own diff for quality, security, and reliability - [x] Unsafe blocks (if any) have justifying comments - [x] The content adheres to Zed's UI standards ([UX/UI](https://github.com/zed-industries/zed/blob/main/CONTRIBUTING.md#uiux-checklist) and [icon](https://github.com/zed-industries/zed/blob/main/crates/icons/README.md) guidelines) - [x] Tests cover the new/changed behavior - [x] Performance impact has been considered and is acceptable ## Background-search validation - `cargo test -p agent_ui --lib`: all 522 tests passed. The new background-preparation test failed on the foreground implementation and now passes across 20 scheduler seeds. - `./script/clippy -p agent_ui -p agent_ui_benchmarks`: passed, including cargo-shear and typo checks. - `cargo fmt -p agent_ui -p agent_ui_benchmarks --check`, `git diff --check`, and `./script/check-licenses`: passed. - `./script/bench-agent-model-search --test`: all 20 workloads passed before and after the change. The runner rejects any `test-support` feature in the benchmark package graph. - Ran quick mode before measured runs. Recorded `--save-baseline foreground-preparation` on `b6157587dbfe96`, then ran the identical benchmark with `--baseline foreground-preparation` on `4617d47567401e`. Apple Silicon, Rust 1.98.1, `release-fast`, 15 Criterion samples, 200 ms warm-up, 500 ms measurement. Representative broad-query results: | Synthetic catalog | Total completion before → after | Foreground task-poll p95 before → after | | --- | --- | --- | | 10,000 options, 128-byte descriptions | 10.35 → 12.49 ms | 7.36 → 0.46 ms | | 10,000 options, 1,024-byte descriptions | 42.16 → 43.95 ms | 30.26 → 0.66 ms | For the larger-description workload, foreground p99/max fell from 33.10/33.10 ms to 1.35/1.35 ms. Median foreground instructions fell from 879.9 million to 11.8 million, while total instructions stayed approximately 2.14 billion. Observed synthetic 120 FPS budget overruns fell from 66 to zero for this workload. This improves foreground responsiveness rather than reducing total work. Other Rust builds were active on the machine, so completion-time estimates are noisy and no throughput speedup is claimed. The benchmark measures the production search module with synthetic ACP-like options, not provider requests, picker rendering, or live keystrokes. Its foreground report includes warm-up/calibration, and budget overruns are not literal display-dropped frames. ## Showcase https://github.com/user-attachments/assets/04c1a59e-8454-49f8-af6c-47e1107f9416 --- Release Notes: - Improved fuzzy searching for agent model and configuration option pickers across Zed Agent and third-party ACP-compatible agents, preserving name relevance and keeping search preparation off the UI thread. --------- Co-authored-by: Ben Brandt <benjamin.j.brandt@gmail.com>
This commit is contained in:
Generated
+14
@@ -484,6 +484,7 @@ dependencies = [
|
||||
"fs",
|
||||
"futures 0.3.34",
|
||||
"fuzzy",
|
||||
"fuzzy_nucleo",
|
||||
"git",
|
||||
"git_ui_core",
|
||||
"gpui",
|
||||
@@ -553,6 +554,19 @@ dependencies = [
|
||||
"zed_actions",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "agent_ui_benchmarks"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"collections",
|
||||
"criterion",
|
||||
"futures 0.3.34",
|
||||
"fuzzy_nucleo",
|
||||
"gpui",
|
||||
"gpui_platform",
|
||||
"ordered-float 5.5.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ahash"
|
||||
version = "0.8.12"
|
||||
|
||||
@@ -10,6 +10,7 @@ members = [
|
||||
"crates/agent_settings",
|
||||
"crates/agent_skills",
|
||||
"crates/agent_ui",
|
||||
"crates/agent_ui_benchmarks",
|
||||
"crates/ai_onboarding",
|
||||
"crates/anthropic",
|
||||
"crates/askpass",
|
||||
|
||||
@@ -58,6 +58,7 @@ file_icons.workspace = true
|
||||
fs.workspace = true
|
||||
futures.workspace = true
|
||||
fuzzy.workspace = true
|
||||
fuzzy_nucleo.workspace = true
|
||||
git.workspace = true
|
||||
gpui.workspace = true
|
||||
gpui_tokio.workspace = true
|
||||
|
||||
@@ -21,6 +21,7 @@ mod language_model_selector;
|
||||
mod mention_set;
|
||||
mod message_editor;
|
||||
mod mode_selector;
|
||||
mod model_search;
|
||||
mod model_selector;
|
||||
mod model_selector_popover;
|
||||
mod profile_selector;
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use std::{cmp::Reverse, rc::Rc, sync::Arc};
|
||||
use std::{rc::Rc, sync::Arc};
|
||||
|
||||
use acp_thread::AgentSessionConfigOptions;
|
||||
use agent_client_protocol::schema::v2 as acp;
|
||||
@@ -6,11 +6,10 @@ use agent_servers::AgentServer;
|
||||
|
||||
use collections::HashSet;
|
||||
use fs::Fs;
|
||||
use fuzzy::StringMatchCandidate;
|
||||
use fuzzy_nucleo::StringMatchCandidate;
|
||||
use gpui::{
|
||||
App, BackgroundExecutor, Context, DismissEvent, Entity, Subscription, Task, Window, prelude::*,
|
||||
};
|
||||
use ordered_float::OrderedFloat;
|
||||
use picker::popover_menu::PickerPopoverMenu;
|
||||
use picker::{Picker, PickerDelegate};
|
||||
use settings::{AgentConfigOptionValue, SettingsStore};
|
||||
@@ -22,6 +21,7 @@ use unicode_segmentation::UnicodeSegmentation;
|
||||
use util::ResultExt as _;
|
||||
use zed_actions::agent::ToggleModelSelector;
|
||||
|
||||
use crate::model_search;
|
||||
use crate::ui::documentation_aside_side;
|
||||
use crate::{
|
||||
CycleFavoriteModels, CycleModeSelector, CycleThinkingEffort, ToggleProfileSelector,
|
||||
@@ -1074,32 +1074,22 @@ async fn fuzzy_search_options(
|
||||
query: &str,
|
||||
executor: BackgroundExecutor,
|
||||
) -> Vec<ConfigOptionValue> {
|
||||
let candidates = options
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(ix, opt)| StringMatchCandidate::new(ix, &opt.name))
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
let mut matches = fuzzy::match_strings(
|
||||
&candidates,
|
||||
query,
|
||||
false,
|
||||
true,
|
||||
100,
|
||||
&Default::default(),
|
||||
executor,
|
||||
)
|
||||
.await;
|
||||
|
||||
matches.sort_unstable_by_key(|mat| {
|
||||
let candidate = &candidates[mat.candidate_id];
|
||||
(Reverse(OrderedFloat(mat.score)), candidate.id)
|
||||
});
|
||||
|
||||
matches
|
||||
.into_iter()
|
||||
.map(|mat| options[mat.candidate_id].clone())
|
||||
.collect()
|
||||
model_search::search(options, query, executor, |index, option| {
|
||||
(
|
||||
StringMatchCandidate::new(index, option.name.clone()),
|
||||
StringMatchCandidate::new(
|
||||
index,
|
||||
format!(
|
||||
"{} {} {} {}",
|
||||
option.name,
|
||||
option.group.as_deref().unwrap_or_default(),
|
||||
option.description.as_deref().unwrap_or_default(),
|
||||
option.value.0,
|
||||
),
|
||||
),
|
||||
)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
fn find_option_name(
|
||||
@@ -1145,6 +1135,119 @@ mod tests {
|
||||
use project::{AgentId, Project};
|
||||
use std::{any::Any, cell::RefCell};
|
||||
|
||||
fn option_names(options: Vec<ConfigOptionValue>) -> Vec<String> {
|
||||
options.into_iter().map(|option| option.name).collect()
|
||||
}
|
||||
|
||||
#[gpui::test]
|
||||
async fn config_option_search_matches_all_searchable_fields(executor: BackgroundExecutor) {
|
||||
let options = vec![
|
||||
ConfigOptionValue {
|
||||
value: acp::SessionConfigValueId::new("openai/gpt-5.6-sol"),
|
||||
name: "GPT-5.6 Sol".to_string(),
|
||||
description: Some("Flagship reasoning model".to_string()),
|
||||
group: Some("OpenAI".to_string()),
|
||||
},
|
||||
ConfigOptionValue {
|
||||
value: acp::SessionConfigValueId::new("anthropic/claude-sonnet-4"),
|
||||
name: "Claude Sonnet 4".to_string(),
|
||||
description: None,
|
||||
group: Some("Anthropic".to_string()),
|
||||
},
|
||||
];
|
||||
|
||||
for query in [
|
||||
"GPT 5.6",
|
||||
"GPT-5.6",
|
||||
"gpt5.6 sol",
|
||||
"openai",
|
||||
"openai/gpt-5.6-sol",
|
||||
"flagship",
|
||||
"openai reasoning",
|
||||
] {
|
||||
assert_eq!(
|
||||
option_names(fuzzy_search_options(options.clone(), query, executor.clone()).await),
|
||||
["GPT-5.6 Sol"],
|
||||
"query {query:?} should match the GPT model",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[gpui::test]
|
||||
async fn config_option_search_ranks_names_before_metadata(executor: BackgroundExecutor) {
|
||||
let options = vec![
|
||||
ConfigOptionValue {
|
||||
value: acp::SessionConfigValueId::new("reasoner"),
|
||||
name: "Reasoner".to_string(),
|
||||
description: Some("High reasoning effort".to_string()),
|
||||
group: None,
|
||||
},
|
||||
ConfigOptionValue {
|
||||
value: acp::SessionConfigValueId::new("xhigh"),
|
||||
name: "Extra High".to_string(),
|
||||
description: None,
|
||||
group: None,
|
||||
},
|
||||
ConfigOptionValue {
|
||||
value: acp::SessionConfigValueId::new("high"),
|
||||
name: "High".to_string(),
|
||||
description: Some(
|
||||
"High effort with advanced reasoning for complex tasks.".to_string(),
|
||||
),
|
||||
group: None,
|
||||
},
|
||||
];
|
||||
|
||||
assert_eq!(
|
||||
option_names(fuzzy_search_options(options, "high", executor).await),
|
||||
["High", "Extra High", "Reasoner"],
|
||||
);
|
||||
}
|
||||
|
||||
#[gpui::test]
|
||||
async fn config_option_search_preserves_name_order_with_different_metadata(
|
||||
executor: BackgroundExecutor,
|
||||
) {
|
||||
let options = vec![
|
||||
ConfigOptionValue {
|
||||
value: acp::SessionConfigValueId::new("model-a"),
|
||||
name: "Model A".to_string(),
|
||||
description: Some("Advanced reasoning for complex tasks".to_string()),
|
||||
group: Some("Provider A".to_string()),
|
||||
},
|
||||
ConfigOptionValue {
|
||||
value: acp::SessionConfigValueId::new("b"),
|
||||
name: "Model B".to_string(),
|
||||
description: None,
|
||||
group: None,
|
||||
},
|
||||
];
|
||||
|
||||
assert_eq!(
|
||||
option_names(fuzzy_search_options(options, "model", executor).await),
|
||||
["Model A", "Model B"],
|
||||
);
|
||||
}
|
||||
|
||||
#[gpui::test]
|
||||
async fn config_option_search_returns_more_than_one_hundred_matches(
|
||||
executor: BackgroundExecutor,
|
||||
) {
|
||||
let options = (0..101)
|
||||
.map(|index| ConfigOptionValue {
|
||||
value: acp::SessionConfigValueId::new(format!("model-{index}")),
|
||||
name: format!("Model {index}"),
|
||||
description: None,
|
||||
group: None,
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
assert_eq!(
|
||||
fuzzy_search_options(options, "model", executor).await.len(),
|
||||
101
|
||||
);
|
||||
}
|
||||
|
||||
#[gpui::test]
|
||||
fn config_defaults_wait_for_authoritative_response(cx: &mut TestAppContext) {
|
||||
for boolean in [false, true] {
|
||||
|
||||
@@ -0,0 +1,86 @@
|
||||
use std::cmp::Reverse;
|
||||
|
||||
use collections::HashSet;
|
||||
use fuzzy_nucleo::StringMatchCandidate;
|
||||
use gpui::BackgroundExecutor;
|
||||
use ordered_float::OrderedFloat;
|
||||
|
||||
pub(crate) async fn search<T, BuildCandidates>(
|
||||
values: Vec<T>,
|
||||
query: &str,
|
||||
executor: BackgroundExecutor,
|
||||
build_candidates: BuildCandidates,
|
||||
) -> Vec<T>
|
||||
where
|
||||
T: Clone + Send + 'static,
|
||||
BuildCandidates: Fn(usize, &T) -> (StringMatchCandidate, StringMatchCandidate) + Send + 'static,
|
||||
{
|
||||
let query = query.to_owned();
|
||||
// Preparing and cloning candidates can block input even though matching is already asynchronous.
|
||||
executor
|
||||
.clone()
|
||||
.spawn(async move {
|
||||
let (names, candidates): (Vec<_>, Vec<_>) = values
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, value)| build_candidates(index, value))
|
||||
.unzip();
|
||||
|
||||
match_names_with_metadata(&names, &candidates, &query, executor)
|
||||
.await
|
||||
.into_iter()
|
||||
.map(|index| values[index].clone())
|
||||
.collect()
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
async fn match_names_with_metadata(
|
||||
names: &[StringMatchCandidate],
|
||||
candidates: &[StringMatchCandidate],
|
||||
query: &str,
|
||||
executor: BackgroundExecutor,
|
||||
) -> Vec<usize> {
|
||||
let cancel_flag = Default::default();
|
||||
let (mut name_matches, mut metadata_matches) = futures::join!(
|
||||
fuzzy_nucleo::match_strings_async(
|
||||
names,
|
||||
query,
|
||||
fuzzy_nucleo::Case::Ignore,
|
||||
fuzzy_nucleo::LengthPenalty::On,
|
||||
names.len(),
|
||||
&cancel_flag,
|
||||
executor.clone(),
|
||||
),
|
||||
fuzzy_nucleo::match_strings_async(
|
||||
candidates,
|
||||
query,
|
||||
fuzzy_nucleo::Case::Ignore,
|
||||
fuzzy_nucleo::LengthPenalty::On,
|
||||
candidates.len(),
|
||||
&cancel_flag,
|
||||
executor,
|
||||
),
|
||||
);
|
||||
|
||||
// Metadata should expand the results without changing the relevance of name matches.
|
||||
let sorting_key = |matched: &fuzzy_nucleo::StringMatch| {
|
||||
(Reverse(OrderedFloat(matched.score)), matched.candidate_id)
|
||||
};
|
||||
name_matches.sort_unstable_by_key(sorting_key);
|
||||
metadata_matches.sort_unstable_by_key(sorting_key);
|
||||
|
||||
let matched_name_ids: HashSet<_> = name_matches
|
||||
.iter()
|
||||
.map(|matched| matched.candidate_id)
|
||||
.collect();
|
||||
name_matches
|
||||
.into_iter()
|
||||
.chain(
|
||||
metadata_matches
|
||||
.into_iter()
|
||||
.filter(|matched| !matched_name_ids.contains(&matched.candidate_id)),
|
||||
)
|
||||
.map(|matched| matched.candidate_id)
|
||||
.collect()
|
||||
}
|
||||
@@ -1,4 +1,4 @@
|
||||
use std::{cmp::Reverse, rc::Rc, sync::Arc};
|
||||
use std::{rc::Rc, sync::Arc};
|
||||
|
||||
use acp_thread::{
|
||||
AgentModelIcon, AgentModelId, AgentModelInfo, AgentModelList, AgentModelSelector,
|
||||
@@ -7,19 +7,19 @@ use acp_thread::{
|
||||
use anyhow::Result;
|
||||
use collections::{HashSet, IndexMap};
|
||||
use futures::FutureExt;
|
||||
use fuzzy::{StringMatchCandidate, match_strings};
|
||||
use fuzzy_nucleo::StringMatchCandidate;
|
||||
use gpui::{
|
||||
Action, AsyncWindowContext, BackgroundExecutor, DismissEvent, FocusHandle, Subscription, Task,
|
||||
TaskExt, WeakEntity,
|
||||
};
|
||||
use itertools::Itertools;
|
||||
use ordered_float::OrderedFloat;
|
||||
use picker::{Picker, PickerDelegate};
|
||||
use settings::SettingsStore;
|
||||
use ui::{DocumentationAside, IntoElement, prelude::*};
|
||||
use util::ResultExt;
|
||||
use zed_actions::agent::OpenSettings;
|
||||
|
||||
use crate::model_search;
|
||||
use crate::ui::{
|
||||
ModelSelectorFooter, ModelSelectorHeader, ModelSelectorListItem, documentation_aside_side,
|
||||
};
|
||||
@@ -441,44 +441,34 @@ async fn fuzzy_search(
|
||||
) -> AgentModelList {
|
||||
async fn fuzzy_search_list(
|
||||
model_list: Vec<AgentModelInfo>,
|
||||
group_name: Option<SharedString>,
|
||||
query: &str,
|
||||
executor: BackgroundExecutor,
|
||||
) -> Vec<AgentModelInfo> {
|
||||
let candidates = model_list
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(ix, model)| StringMatchCandidate::new(ix, model.name.as_ref()))
|
||||
.collect::<Vec<_>>();
|
||||
let mut matches = match_strings(
|
||||
&candidates,
|
||||
query,
|
||||
false,
|
||||
true,
|
||||
100,
|
||||
&Default::default(),
|
||||
executor,
|
||||
)
|
||||
.await;
|
||||
|
||||
matches.sort_unstable_by_key(|mat| {
|
||||
let candidate = &candidates[mat.candidate_id];
|
||||
(Reverse(OrderedFloat(mat.score)), candidate.id)
|
||||
});
|
||||
|
||||
matches
|
||||
.into_iter()
|
||||
.map(|mat| model_list[mat.candidate_id].clone())
|
||||
.collect()
|
||||
model_search::search(model_list, query, executor, move |index, model| {
|
||||
(
|
||||
StringMatchCandidate::new(index, model.name.clone()),
|
||||
StringMatchCandidate::new(
|
||||
index,
|
||||
format!(
|
||||
"{} {}",
|
||||
model.name,
|
||||
group_name.as_deref().unwrap_or_default()
|
||||
),
|
||||
),
|
||||
)
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
match model_list {
|
||||
AgentModelList::Flat(model_list) => {
|
||||
AgentModelList::Flat(fuzzy_search_list(model_list, &query, executor).await)
|
||||
AgentModelList::Flat(fuzzy_search_list(model_list, None, &query, executor).await)
|
||||
}
|
||||
AgentModelList::Grouped(index_map) => {
|
||||
let groups =
|
||||
futures::future::join_all(index_map.into_iter().map(|(group_name, models)| {
|
||||
fuzzy_search_list(models, &query, executor.clone())
|
||||
fuzzy_search_list(models, Some(group_name.0.clone()), &query, executor.clone())
|
||||
.map(|results| (group_name, results))
|
||||
}))
|
||||
.await;
|
||||
@@ -698,6 +688,53 @@ mod tests {
|
||||
// Fuzzy search - test with specific model name
|
||||
let results = fuzzy_search(models.clone(), "mistral".into(), cx.executor()).await;
|
||||
assert_models_eq(results, vec![("ollama", vec!["mistral"])]);
|
||||
|
||||
let models = create_model_list(vec![
|
||||
("OpenAI", vec!["GPT-5.6 Sol"]),
|
||||
("Anthropic", vec!["Claude Sonnet 4"]),
|
||||
]);
|
||||
let results = fuzzy_search(models.clone(), "GPT 5.6".into(), cx.executor()).await;
|
||||
assert_models_eq(results, vec![("OpenAI", vec!["GPT-5.6 Sol"])]);
|
||||
|
||||
let results = fuzzy_search(models.clone(), "openai 5.6".into(), cx.executor()).await;
|
||||
assert_models_eq(results, vec![("OpenAI", vec!["GPT-5.6 Sol"])]);
|
||||
|
||||
let results = fuzzy_search(models, "openai".into(), cx.executor()).await;
|
||||
assert_models_eq(results, vec![("OpenAI", vec!["GPT-5.6 Sol"])]);
|
||||
}
|
||||
|
||||
#[gpui::test]
|
||||
async fn test_fuzzy_match_prioritizes_names_over_group(cx: &mut TestAppContext) {
|
||||
let models = create_model_list(vec![(
|
||||
"High Provider",
|
||||
vec!["Reasoner", "Extra High", "High"],
|
||||
)]);
|
||||
|
||||
let results = fuzzy_search(models, "high".into(), cx.executor()).await;
|
||||
assert_models_eq(
|
||||
results,
|
||||
vec![("High Provider", vec!["High", "Extra High", "Reasoner"])],
|
||||
);
|
||||
}
|
||||
|
||||
#[gpui::test(iterations = 20)]
|
||||
async fn test_model_search_builds_candidates_in_background(executor: BackgroundExecutor) {
|
||||
let background_executor = executor.clone();
|
||||
let results = model_search::search(
|
||||
vec!["High".to_string()],
|
||||
"high",
|
||||
executor,
|
||||
move |index, name| {
|
||||
assert!(!background_executor.is_main_thread());
|
||||
(
|
||||
StringMatchCandidate::new(index, name.clone()),
|
||||
StringMatchCandidate::new(index, name.clone()),
|
||||
)
|
||||
},
|
||||
)
|
||||
.await;
|
||||
|
||||
assert_eq!(results, ["High"]);
|
||||
}
|
||||
|
||||
#[gpui::test]
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
[package]
|
||||
name = "agent_ui_benchmarks"
|
||||
version = "0.1.0"
|
||||
edition.workspace = true
|
||||
publish.workspace = true
|
||||
license = "GPL-3.0-or-later"
|
||||
|
||||
[lints]
|
||||
workspace = true
|
||||
|
||||
[package.metadata.cargo-shear]
|
||||
# Used by the path-included production module and the gpui::bench expansion.
|
||||
ignored = ["futures", "gpui_platform"]
|
||||
|
||||
[dev-dependencies]
|
||||
collections.workspace = true
|
||||
criterion.workspace = true
|
||||
futures.workspace = true
|
||||
fuzzy_nucleo.workspace = true
|
||||
gpui = { workspace = true, features = ["bench-support"] }
|
||||
gpui_platform = { workspace = true, features = ["bench-support"] }
|
||||
ordered-float.workspace = true
|
||||
|
||||
[[bench]]
|
||||
name = "model_search"
|
||||
harness = false
|
||||
+1
@@ -0,0 +1 @@
|
||||
../../LICENSE-GPL
|
||||
@@ -0,0 +1,47 @@
|
||||
# Agent model search benchmark
|
||||
|
||||
This benchmark measures the search pipeline shared by the agent model and ACP
|
||||
configuration-option pickers, including candidate construction, matching, ranking,
|
||||
and result cloning. It records both completion latency and foreground task-poll
|
||||
durations, so work moved to a background task is not mistaken for eliminated work.
|
||||
|
||||
The benchmark is a separate package because `agent_ui`'s dev dependencies enable
|
||||
`test-support`. It compiles the production `model_search` module directly rather
|
||||
than maintaining a benchmark-only copy of the search algorithm. The runner checks
|
||||
the complete package feature graph and refuses to run if `test-support` leaks in.
|
||||
|
||||
From the repository root:
|
||||
|
||||
```sh
|
||||
# Smoke: exercise every workload once and validate the results.
|
||||
./script/bench-agent-model-search --test
|
||||
|
||||
# Quick timing check before a measured run.
|
||||
./script/bench-agent-model-search --quick '1000-options-128-byte-descriptions-selective'
|
||||
|
||||
# Record a baseline before changing the production search implementation.
|
||||
./script/bench-agent-model-search --save-baseline before
|
||||
|
||||
# Run the identical benchmark after the change.
|
||||
./script/bench-agent-model-search --baseline before
|
||||
```
|
||||
|
||||
Run measurements sequentially on an otherwise idle machine. Each measured case
|
||||
uses 15 samples, 200 ms warm-up, and 500 ms measurement time; filter to a single
|
||||
workload for longer local investigation. Use `BENCH_MEASUREMENT=wall-time` to
|
||||
disable counters, or leave it unset to report supported hardware and allocation
|
||||
counters alongside wall time.
|
||||
|
||||
Fixtures contain 100, 1,000, or 10,000 synthetic ACP-like options, with 128-byte or
|
||||
1,024-byte descriptions. They cover broad, selective, combined name/metadata,
|
||||
missing, and exact-name ranking queries. Workers and matcher pools are warm.
|
||||
Each iteration includes one foreground clone of the input snapshot and the full
|
||||
production search; fixture generation and result validation are outside timing.
|
||||
The synchronous reference checks all returned value identifiers and their order.
|
||||
|
||||
The benchmark measures the search pipeline, not the full picker: it does not
|
||||
measure provider requests, layout, painting, presentation, or live keystrokes.
|
||||
No window is rendered. Foreground frame-budget overruns use GPUI's synthetic
|
||||
120 FPS budget (8.33 ms); they are not display-dropped frames. The foreground
|
||||
report includes warm-up and calibration, whereas Criterion's latency estimate
|
||||
uses its measurement phase.
|
||||
@@ -0,0 +1,209 @@
|
||||
use std::{cmp::Reverse, fmt, sync::Arc, time::Duration};
|
||||
|
||||
use collections::HashSet;
|
||||
use fuzzy_nucleo::{Case, LengthPenalty, StringMatch, StringMatchCandidate};
|
||||
use gpui::{BenchAppContext, SharedString};
|
||||
use ordered_float::OrderedFloat;
|
||||
|
||||
// Compile the production search without pulling in the rest of agent_ui.
|
||||
#[path = "../../agent_ui/src/model_search.rs"]
|
||||
mod model_search;
|
||||
|
||||
#[derive(Clone)]
|
||||
struct OptionValue {
|
||||
value: SharedString,
|
||||
name: String,
|
||||
description: Option<String>,
|
||||
group: Option<String>,
|
||||
}
|
||||
|
||||
struct SearchInput {
|
||||
options: usize,
|
||||
description_bytes: usize,
|
||||
query_name: &'static str,
|
||||
query: &'static str,
|
||||
}
|
||||
|
||||
impl fmt::Display for SearchInput {
|
||||
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
write!(
|
||||
formatter,
|
||||
"{}-options-{}-byte-descriptions-{}",
|
||||
self.options, self.description_bytes, self.query_name,
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
fn inputs() -> Vec<SearchInput> {
|
||||
[(100, 128), (1_000, 128), (10_000, 128), (10_000, 1_024)]
|
||||
.into_iter()
|
||||
.flat_map(|(options, description_bytes)| {
|
||||
[
|
||||
("broad", "gpt"),
|
||||
("selective", "GPT 5.6 000042"),
|
||||
("combined", "openai reasoning"),
|
||||
("missing", "nonexistent-zzzzzz"),
|
||||
("name-ranking", "high"),
|
||||
]
|
||||
.into_iter()
|
||||
.map(move |(query_name, query)| SearchInput {
|
||||
options,
|
||||
description_bytes,
|
||||
query_name,
|
||||
query,
|
||||
})
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn options(input: &SearchInput) -> Vec<OptionValue> {
|
||||
let description = "Flagship reasoning model for coding, analysis, and complex tasks. ";
|
||||
let mut description = description.repeat(input.description_bytes.div_ceil(description.len()));
|
||||
description.truncate(input.description_bytes);
|
||||
|
||||
(0..input.options)
|
||||
.map(|index| {
|
||||
let (name, group, identifier) = match index % 4 {
|
||||
0 => ("Claude Sonnet 4", "Anthropic", "anthropic/claude-sonnet-4"),
|
||||
1 => ("Gemini 2.5 Pro", "Google", "google/gemini-2.5-pro"),
|
||||
2 => ("GPT-5.6 Sol", "OpenAI", "openai/gpt-5.6-sol"),
|
||||
_ => ("DeepSeek V3", "OpenRouter", "openrouter/deepseek-v3"),
|
||||
};
|
||||
let mut option = OptionValue {
|
||||
value: format!("{identifier}/{index:06}").into(),
|
||||
name: format!("{name} {index:06}"),
|
||||
description: Some(description.clone()),
|
||||
group: Some(group.to_string()),
|
||||
};
|
||||
match index {
|
||||
0 => {
|
||||
option.name = "High".to_string();
|
||||
option.description =
|
||||
Some("High effort with advanced reasoning for complex tasks.".to_string());
|
||||
}
|
||||
1 => {
|
||||
option.name = "Extra High".to_string();
|
||||
option.description = None;
|
||||
}
|
||||
2 => {
|
||||
option.name = "Reasoner".to_string();
|
||||
option.description = Some("High reasoning effort".to_string());
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
option
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn candidates(index: usize, option: &OptionValue) -> (StringMatchCandidate, StringMatchCandidate) {
|
||||
(
|
||||
StringMatchCandidate::new(index, option.name.clone()),
|
||||
StringMatchCandidate::new(
|
||||
index,
|
||||
format!(
|
||||
"{} {} {} {}",
|
||||
option.name,
|
||||
option.group.as_deref().unwrap_or_default(),
|
||||
option.description.as_deref().unwrap_or_default(),
|
||||
option.value,
|
||||
),
|
||||
),
|
||||
)
|
||||
}
|
||||
|
||||
fn expected_values(options: &[OptionValue], query: &str) -> Vec<SharedString> {
|
||||
let (names, metadata): (Vec<_>, Vec<_>) = options
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(index, option)| candidates(index, option))
|
||||
.unzip();
|
||||
let mut names = fuzzy_nucleo::match_strings(
|
||||
&names,
|
||||
query,
|
||||
Case::Ignore,
|
||||
LengthPenalty::On,
|
||||
options.len(),
|
||||
);
|
||||
let mut metadata = fuzzy_nucleo::match_strings(
|
||||
&metadata,
|
||||
query,
|
||||
Case::Ignore,
|
||||
LengthPenalty::On,
|
||||
options.len(),
|
||||
);
|
||||
let sorting_key =
|
||||
|matched: &StringMatch| (Reverse(OrderedFloat(matched.score)), matched.candidate_id);
|
||||
names.sort_unstable_by_key(sorting_key);
|
||||
metadata.sort_unstable_by_key(sorting_key);
|
||||
let name_ids: HashSet<_> = names.iter().map(|matched| matched.candidate_id).collect();
|
||||
names
|
||||
.into_iter()
|
||||
.chain(
|
||||
metadata
|
||||
.into_iter()
|
||||
.filter(|matched| !name_ids.contains(&matched.candidate_id)),
|
||||
)
|
||||
.map(|matched| options[matched.candidate_id].value.clone())
|
||||
.collect()
|
||||
}
|
||||
|
||||
struct ValidatedResults {
|
||||
results: Vec<OptionValue>,
|
||||
expected: Arc<Vec<SharedString>>,
|
||||
}
|
||||
|
||||
impl Drop for ValidatedResults {
|
||||
fn drop(&mut self) {
|
||||
// bench_task drops its output after timing, so validation doesn't inflate latency.
|
||||
assert!(
|
||||
self.results
|
||||
.iter()
|
||||
.map(|option| &option.value)
|
||||
.eq(self.expected.iter()),
|
||||
"search dropped, duplicated, or reordered options",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[gpui::bench(
|
||||
inputs = inputs(),
|
||||
input_name = "case",
|
||||
group = "agent_model_search",
|
||||
fps = 120,
|
||||
sample_size = 15
|
||||
)]
|
||||
fn model_search(input: &SearchInput, cx: &mut BenchAppContext) {
|
||||
let options = Arc::new(options(input));
|
||||
let expected = Arc::new(expected_values(&options, input.query));
|
||||
match input.query_name {
|
||||
"missing" => assert!(expected.is_empty()),
|
||||
"name-ranking" => {
|
||||
assert_eq!(
|
||||
expected.first(),
|
||||
options.first().map(|option| &option.value)
|
||||
);
|
||||
assert_eq!(expected.get(1), options.get(1).map(|option| &option.value));
|
||||
}
|
||||
_ => assert!(!expected.is_empty()),
|
||||
}
|
||||
let query = input.query;
|
||||
cx.bench_task(|cx| {
|
||||
let options = options.clone();
|
||||
let expected = expected.clone();
|
||||
let executor = cx.background_executor().clone();
|
||||
cx.foreground_executor().spawn(async move {
|
||||
let results = model_search::search(options.to_vec(), query, executor, candidates).await;
|
||||
ValidatedResults { results, expected }
|
||||
})
|
||||
});
|
||||
}
|
||||
|
||||
gpui::bench_group! {
|
||||
name = benches;
|
||||
config = criterion::Criterion::default()
|
||||
.warm_up_time(Duration::from_millis(200))
|
||||
.measurement_time(Duration::from_millis(500));
|
||||
targets = model_search
|
||||
}
|
||||
gpui::bench_main!(benches);
|
||||
Executable
+12
@@ -0,0 +1,12 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
feature_tree="$(cargo tree --locked -p agent_ui_benchmarks -e normal,build,dev,features)"
|
||||
if grep -F 'feature "test-support"' <<< "$feature_tree"; then
|
||||
echo "The model search benchmark must not enable test-support." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cargo bench --locked -p agent_ui_benchmarks --bench model_search --profile release-fast -- "$@"
|
||||
Reference in New Issue
Block a user