Files
agent-cost-estimator/SPEC.md
T
user 17239e26b3 Add comprehensive SPEC.md for project recreation
- Document architecture, data flow, and module structure
- Define harness adapter interface contract
- Specify pricing table format (provider-based)
- Document hook integration requirements
- List all feature requirements and configuration files
- Include glossary, examples, and file index

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
2026-10-24 02:45:45 +02:00

30 KiB

name, version, title, type, created, updated
name version title type created updated
Agent Cost Estimator Specification 1.0.0 Project Spec Sheet specification 2026-10-24 2026-10-24

Agent Cost Estimator Specification

This document is the build contract for Agent Cost Estimator. A reader with this file, and no source tree, must be able to produce the same observable software. Status values are Implemented and Planned.

1. Product Overview

1.1. Purpose

Agent Cost Estimator provides real-time visibility into AI agent session costs by reading usage data directly from harness log files and calculating costs based on configurable pricing models (USD per million tokens).

1.2. Scope

In Scope:

  • Reading token usage from AI harness session logs
  • Calculating costs based on configurable pricing
  • Supporting multiple AI harnesses (extensible adapter system)
  • Supporting multiple AI providers with different pricing models
  • CLI interface for manual cost queries
  • Hook-based integration for real-time cost tracking

Out of Scope:

  • Actual token generation or AI model interaction
  • Authentication with AI providers
  • Billing system integration
  • Historical cost analysis across multiple sessions (future enhancement)

1.3. Non-Goals

  • Providing official pricing data (prices are user-configured)
  • Modifying AI harness behavior
  • Storing session data long-term
  • User authentication or authorization

1.4. Document Conventions

  • File paths are relative to the repository root unless stated otherwise
  • MUST / SHALL = mandatory requirement
  • SHOULD = strong recommendation
  • MAY = optional feature
  • Code examples use Python 3.9+ syntax

2. Architecture

2.1. Component Diagram

┌─────────────────────────────────────────────────────────────────┐
│                         Agent Cost Estimator                        │
├─────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────────────────────────────────────────────────────┐│
│  │                         CLI (cli.py)                              ││
│  └──────────────────────┬──────────────────────────────────────┘│
│                          │                                          │
│         ┌────────────────┴────────────────┐                        │
│         │                                 │                        │
│  ┌──────▼──────┐              ┌────────▼──────┐                  │
│  │ core.py     │              │ pricing.py    │                  │
│  │ (calculations)│              │ (pricing data)│                  │
│  └──────┬──────┘              └────────┬──────┘                  │
│         │                                 │                        │
│         └────────────────────┼────────────────┘                        │
│                          │                                          │
│         ┌────────────────────▼────────────────┐                        │
│         │              Harness Adapters            │                        │
│         │   (extensible interface: base.py)       │                        │
│         └──────────────────┬──────────────────┘                        │
│                            │                                      │
│              ┌─────────────────┼─────────────────┐                  │
│              ▼                 ▼                 ▼                  │
│       ┌──────────────┐   ┌──────────────┐   ┌──────────────┐        │
│       │ vibe.py      │   │ openai.py    │   │ ...          │        │
│       │ (Mistral)    │   │ (Planned)    │   │ (Future)      │        │
│       └──────────────┘   └──────────────┘   └──────────────┘        │
│                                                                     │
└─────────────────────────────────────────────────────────────────┘
                          │
                          ▼
            Session Data (read-only)
            ├── meta.json
            ├── journal/*.jsonl
            └── generations/*/projection-state.json

2.2. Data Flow

Vibe Session
    │
    ▼ (hook trigger: post_agent)
┌─────────────────────┐
│ post_turn.py Hook    │◄── Receives: session_id, hook_event_name
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ agent-cost-estimator │◄── Args: --session-id <path> --json
│ CLI (cli.py)         │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ VibeHarness         │◄── Reads: meta.json, journal/*.jsonl,
│ (vibe.py)           │    projection-state.json
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ collect_usage()     │◄── Returns: SessionUsage (tokens, calls, turns)
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ load_pricing()      │◄── Reads: pricing.json (provider-based)
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ get_model_prices()  │◄── Returns: (rates, key, currency, is_default)
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ compute_cost()      │◄── Returns: cost breakdown (input, cached, output)
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│ Output Formatter    │◄── Formats: default, details, JSON
└─────────────────────┘
           │
           ▼
    Hook Output: "mistral:mistral-medium | session:$X | turn:$Y"

2.3. Module Structure

agent-cost-estimator/
├── agent_cost_estimator/
│   ├── __init__.py           # Public API exports
│   ├── __main__.py           # Entry point for python -m
│   ├── cli.py                # CLI interface and main()
│   ├── core.py               # Cost computation logic
│   ├── pricing.py            # Pricing table handling
│   │
│   ├── harnesses/
│   │   ├── __init__.py       # Harness registry
│   │   ├── base.py           # HarnessAdapter base class (ABC)
│   │   └── vibe.py           # Vibe-specific implementation
│   │
│   ├── hooks/
│   │   └── post_turn.py      # Vibe CLI hook script
│   │
│   └── skill/
│       └── SKILL.md          # LLM usage instructions
│
├── pyproject.toml            # Package metadata and dependencies
├── pricing.json              # Default pricing configuration
├── README.md                 # Human-readable documentation
└── tests/
    └── test_agent_cost_estimator.py  # Test suite

3. Core Contracts

3.1. Harness Adapter Interface

File: agent_cost_estimator/harnesses/base.py

Purpose: Define the interface that all AI harness adapters MUST implement.

Contract:

from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from typing import Protocol

@dataclass
class SessionMeta:
    """Session metadata extracted from harness logs."""
    title: str | None
    model_name: str | None
    alias: str | None

@dataclass
class TurnUsage:
    """Token usage for a single turn/assistant response."""
    label: str
    calls: int
    input_tokens: int
    cached_input_tokens: int
    output_tokens: int
    fresh_input_tokens: int

@dataclass
class SessionUsage:
    """Aggregate token usage for a session."""
    input_tokens: int
    cached_input_tokens: int
    output_tokens: int
    fresh_input_tokens: int
    calls: int
    turns: list[TurnUsage]
    usage_source: str  # e.g., "projection-state", "journal"

class HarnessAdapter(Protocol):
    """Interface for AI harness adapters.
    
    To add a new harness:
    1. Implement this interface in a new file under harnesses/
    2. Register via entry point in pyproject.toml
    """
    
    name: str  # e.g., "vibe", "openai"
    home_env_var: str  # e.g., "VIBE_HOME", "OPENAI_HOME"
    home_dir_name: str  # e.g., ".vibe", ".openai"
    
    def default_home(self) -> Path:
        """Return the default home directory for this harness."""
        ...
    
    def resolve_session_dir(self, session_arg: str | None, home: Path) -> Path:
        """Resolve session directory from argument or default discovery.
        
        Args:
            session_arg: User-provided session ID or path, or None for auto-discovery
            home: Harness home directory
            
        Returns:
            Absolute path to the session directory
            
        Raises:
            FileNotFoundError: If session not found
        """
        ...
    
    def read_meta(self, session_dir: Path) -> SessionMeta:
        """Read session metadata from harness logs.
        
        Args:
            session_dir: Session directory path
            
        Returns:
            SessionMeta with title, model_name, and alias
        """
        ...
    
    def collect_usage(self, session_dir: Path) -> SessionUsage:
        """Collect token usage data from harness logs.
        
        Args:
            session_dir: Session directory path
            
        Returns:
            SessionUsage with aggregate and per-turn token counts
        """
        ...

Registration:

To register a new harness adapter, add to pyproject.toml:

[project.entry-points."agent_cost_estimator.harnesses"]
my-harness = "my_package.harnesses:MyHarnessAdapter"

3.2. Pricing Table Format

File: agent_cost_estimator/pricing.json (bundled default)

Purpose: Define pricing models for different AI providers and models.

Format: Provider-based structure (version 2.0+)

{
  "providers": {
    "<provider_name>": {
      "currency": "<currency_code>",
      "models": {
        "<model_name>": {
          "input_price": <number>,
          "cached_input_price": <number>,
          "output_price": <number>,
          "note": "<optional description>",
          "currency": "<optional override>"
        },
        "default": {
          "input_price": <number>,
          "cached_input_price": <number>,
          "output_price": <number>
        }
      }
    }
  }
}

Semantics:

  • providers: REQUIRED. Dictionary of provider configurations
  • provider_name: Provider identifier (e.g., "mistral", "openai", "anthropic")
  • currency: Provider-level currency (default: "USD" if not specified)
  • models: REQUIRED. Dictionary of model pricing for this provider
  • model_name: Model identifier (e.g., "mistral-medium", "gpt-4")
  • input_price: USD per million input tokens (prompt tokens)
  • cached_input_price: USD per million cached input tokens (context cache)
  • output_price: USD per million output tokens (completion tokens)
  • Model-level currency: Optional override for specific model

Lookup Order:

  1. Exact match on model_name (passed as first argument to get_model_prices)
  2. Exact match on alias (passed as second argument)
  3. Provider's default model
  4. First provider's default model (fallback)

Example:

{
  "providers": {
    "default": {
      "currency": "USD",
      "models": {
        "default": {
          "input_price": 1.5,
          "cached_input_price": 0.15,
          "output_price": 7.5,
          "note": "Global fallback rates"
        }
      }
    },
    "mistral": {
      "currency": "USD",
      "models": {
        "default": {
          "input_price": 1.5,
          "cached_input_price": 0.15,
          "output_price": 7.5
        },
        "mistral-small": {
          "input_price": 1.0,
          "cached_input_price": 0.1,
          "output_price": 3.0
        },
        "mistral-medium": {
          "input_price": 1.5,
          "cached_input_price": 0.15,
          "output_price": 7.5
        },
        "mistral-large": {
          "input_price": 2.5,
          "cached_input_price": 0.25,
          "output_price": 12.5
        }
      }
    }
  }
}

Validation Rules:

  • All price values MUST be non-negative numbers
  • Each provider MUST have a models dictionary
  • Each provider's models MUST contain at least one model
  • The default provider MUST exist (for fallback)

3.3. Hook Integration

Purpose: Enable real-time cost tracking in AI harness sessions.

Hook Types:

  • post_agent: Fires after assistant turns (RECOMMENDED for cost tracking)

Hook Contract:

  1. Hook receives JSON payload on stdin:

    {
      "session_id": "<uuid>",
      "hook_event_name": "post_agent"
    }
    
  2. Hook MUST output JSON with system_message field:

    {
      "system_message": "-- <formatted cost output>"
    }
    
  3. Hook MUST be executable and return exit code 0 on success

Hook Installation:

# Automatic (recommended)
agent-cost-estimator --integrate mistral

# Manual
mkdir -p ~/.vibe/hooks
cp agent_cost_estimator/hooks/post_turn.py ~/.vibe/hooks/
chmod +x ~/.vibe/hooks/post_turn.py

4. Package and Runtime

4.1. Dependencies

Runtime Dependencies:

  • Python >= 3.9
  • Standard library only (no external dependencies for core functionality)

Development Dependencies:

  • pytest >= 7.0.0
  • pytest-cov >= 4.0.0

Packaging:

# pyproject.toml
[build-system]
requires = ["setuptools>=77"]
build-backend = "setuptools.build_meta"

4.2. Entry Points

CLI:

[project.scripts]
agent-cost-estimator = "agent_cost_estimator.cli:main"

Harness Adapters:

[project.entry-points."agent_cost_estimator.harnesses"]
vibe = "agent_cost_estimator.harnesses.vibe:VibeHarness"

4.3. Installation

From Source:

cd agent-cost-estimator
pip install .

Verify:

agent-cost-estimator --list-harnesses  # Should output: vibe
agent-cost-estimator --skill          # Should output SKILL.md

5. Feature Requirements

5.1. CLI Interface

Status: Implemented

Command: agent-cost-estimator [OPTIONS]

Options:

Option Type Required Default Description
--harness str No vibe Harness to read (vibe, etc.)
--session-id str No auto Session ID or directory path
--home str No harness default Harness data root directory
--pricing str No bundled Path to pricing.json file
--model str No from session Force pricing lookup key
--input-price float No from pricing Override input price
--cached-input-price float No from pricing Override cached input price
--output-price float No from pricing Override output price
-d, --details flag No False Full report table
-s, --session flag No False Session cost only (bare number)
-t, --turn flag No False Latest turn cost only (bare number)
--turns flag No False Per-turn breakdown (with -d)
--json flag No False JSON output
--list-harnesses flag No False List available harnesses
--skill flag No False Print SKILL.md
--integrate HARNESS str No None Install hooks for harness
--update-pricing flag No False Persist pricing to file
--dry-run flag No False Dry run (with --update-pricing)

Exit Codes:

  • 0: Success
  • 1: Error (invalid arguments, session not found, etc.)

5.2. Vibe Harness Adapter

Status: Implemented

Purpose: Read session data from Mistral Vibe CLI logs.

File: agent_cost_estimator/harnesses/vibe.py

Implementation:

  • default_home(): Returns ~/.vibe (or $VIBE_HOME)
  • resolve_session_dir(): Discovers latest session or uses provided ID
  • read_meta(): Reads meta.json for title, model_name, active_model
  • collect_usage(): Reads from projection-state.json (preferred) or journal files

Session Discovery:

  1. If --session-id provided: Use directly
  2. Otherwise: Find newest directory in ~/.vibe/logs/session/unified/

Usage Source Priority:

  1. projection-state.json (survives journal rotation, has cumulative totals)
  2. Journal files (current window on disk)

Cached Token Handling:

  • Cached tokens are clamped to input tokens (matching Vibe's behavior)
  • Fresh tokens = input_tokens - cached_input_tokens

5.3. Pricing Configuration

Status: Implemented

Features:

  • Provider-based pricing structure
  • Model-level currency overrides
  • Default fallback (provider default → global default)
  • Warning when using default pricing ( **model price not available**)

Lookup Priority:

  1. Passed model name (now uses active_model first)
  2. Passed alias (model_name)
  3. Provider's default model
  4. First provider's default model

CLI Overrides:

agent-cost-estimator --input-price 2.0 --output-price 10.0 -s

Persist Overrides:

agent-cost-estimator --model my-model --input-price 2.0 --update-pricing

5.4. Cost Calculation

Status: Implemented

File: agent_cost_estimator/core.py

Formula:

input_cost_usd = (fresh_input_tokens * input_price + cached_input_tokens * cached_input_price) / 1_000_000
output_cost_usd = (output_tokens * output_price) / 1_000_000
total_cost_usd = input_cost_usd + output_cost_usd

Function:

def compute_cost(tokens: dict[str, int], rates: dict[str, float]) -> dict[str, float]:
    """Compute cost from token counts and pricing rates.
    
    Args:
        tokens: {"fresh": int, "cached": int, "output": int}
        rates: {"input_price": float, "cached_input_price": float, "output_price": float}
        
    Returns:
        {"input_usd": float, "cached_usd": float, "output_usd": float, "total_usd": float}
    """

5.5. Hook System

Status: Implemented

Purpose: Display costs after each assistant turn in Vibe CLI.

Hook Script: agent_cost_estimator/hooks/post_turn.py

Behavior:

  1. Receives session_id from hook payload
  2. Resolves session directory
  3. Calls agent-cost-estimator --session-id <path> --json
  4. Formats output as provider:model | session:$X | turn:$Y
  5. Returns JSON with system_message field

Output Examples:

-- mistral:mistral-medium | session:$3.40 | turn:$0.92
-- mistral:mistral-medium-3.5 | session:$0.00 | turn:$0.00

Error Handling:

  • Session not found: -- [cost: session dir not found]
  • Timeout: -- [cost: timeout]
  • Other errors: -- [cost: error - <message>]

5.6. Integration Command

Status: Implemented

Command: agent-cost-estimator --integrate <harness>

Supported Harnesses:

  • mistral (Vibe CLI)

Modular Design:

The --integrate command accepts a harness name argument, allowing future support for additional harnesses without breaking changes.

Actions:

  1. Creates harness-specific hook directory
  2. Installs hook script
  3. Creates hooks.toml configuration
  4. Updates bash allowlist in config.toml

Example:

agent-cost-estimator --integrate mistral

Future Extension:

To add a new harness openai:

  1. Add to supported_harnesses list in integrate_hooks()
  2. Add harness-specific directory logic if needed
  3. Update documentation

5.7. Output Formatting

Status: Implemented

Default Format:

provider:model | session:$X.XX | turn:$Y.YY

Details Format (-d):

harness:  vibe
session:  <session-id>
model:    provider:model
<warning if default pricing>
model calls:        N
usage source:       projection-state
input tokens:       N,NNN
output tokens:      N,NNN
pricing:           provider:model from bundled pricing.json
currency:          USD
prices (USD/M):  input X cached Y output Z
input cost:         $X.XXXX
cached input cost:  $X.XXXX
output cost:        $X.XXXX
TOTAL:              $X.XXXX

JSON Format (--json):

{
  "harness": "vibe",
  "session_id": "<uuid>",
  "title": null,
  "model": "mistral-vibe-cli-latest",
  "provider": "mistral",
  "provider_model": "mistral:mistral-medium-3.5",
  "model_price_unavailable": false,
  "calls": N,
  "input_tokens": N,
  "cached_input_tokens": N,
  "fresh_input_tokens": N,
  "output_tokens": N,
  "pricing": {
    "file": "bundled",
    "model_key": "mistral:mistral-medium-3.5",
    "currency": "USD",
    "rates_usd_per_million": {
      "input_price": X,
      "cached_input_price": Y,
      "output_price": Z
    }
  },
  "usage_source": "projection-state",
  "cost_usd": {
    "input_usd": X,
    "cached_usd": Y,
    "output_usd": Z,
    "total_usd": T
  },
  "last_turn": {...},
  "turns": [...]
}

Warning Indicator:

When a model falls back to default pricing, all output modes include:

 **model price not available**

6. Configuration Files

6.1. pricing.json

Location: agent_cost_estimator/pricing.json (bundled)

Purpose: Default pricing configuration for all supported providers/models.

Structure: See 3.2. Pricing Table Format

Bundled Version: Contains Mistral, Zai, and default provider configurations.

Customization:

Users can:

  • Provide custom pricing file via --pricing flag
  • Update pricing for a model via --update-pricing
  • Override prices via CLI flags (--input-price, etc.)

6.2. hooks/post_turn.py

Location: agent_cost_estimator/hooks/post_turn.py

Purpose: Vibe CLI hook for real-time cost tracking.

Features:

  • Session directory resolution (session_id → path)
  • Error handling with graceful degradation
  • JSON output format for Vibe CLI compatibility
  • Logging for debugging

Installation:

Automatic via --integrate command, or manual copy to ~/.vibe/hooks/

6.3. hooks/hooks.toml

Location: Created at ~/.vibe/hooks.toml during integration

Purpose: Vibe CLI hook configuration

Content:

[[hooks]]
name = "cost-tracker"
type = "post_agent"
command = "python ~/.vibe/hooks/post_turn.py"
timeout = 30
strict = false
description = "Display cost after each assistant turn"

7. Error Handling

Principle: Fail gracefully, never break the user's session.

CLI Errors:

  • Invalid arguments: Exit 1 with error message to stderr
  • Session not found: Exit 1 with error message
  • Pricing lookup failure: Exit 1 with error message

Hook Errors:

  • Session directory not found: Return error message in JSON format
  • agent-cost-estimator fails: Return error message with stdout/stderr
  • Timeout: Return timeout message
  • Unexpected errors: Return error message with exception

Pricing Errors:

  • Model not in pricing.json: Falls back to default, displays warning
  • Pricing file invalid: Exit 1 with validation error
  • Pricing file missing: Exit 1 with FileNotFoundError

8. Testing

Framework: pytest

Coverage: All core functionality tested

Test Files:

  • tests/test_agent_cost_estimator.py (30 tests)

Test Categories:

  • Core cost calculation
  • Pricing table loading/lookup/validation
  • Vibe harness adapter (journal parsing, projection, dedupe, session discovery)
  • Harness registry
  • CLI (pricing flags, -t, -s, --json, --update-pricing, --integrate, error exits)

Running Tests:

cd agent-cost-estimator
python -m pytest tests/test_agent_cost_estimator.py -v

9. Deployment

Packaging:

# Build
python -m build

# Install
pip install dist/agent_cost_estimator-*.tar.gz

Distribution:

  • PyPI (future)
  • Direct installation from Git repository

Versioning:

  • Follows semantic versioning (MAJOR.MINOR.PATCH)
  • Bump version in pyproject.toml for each release
  • Maintain CHANGELOG (future enhancement)

Appendix A: Glossary

Term Definition
Session A single interaction with an AI agent, with its own UUID and log directory
Turn A single assistant response within a session
Harness An AI agent framework (e.g., Vibe CLI, OpenAI SDK)
Provider An AI model provider (e.g., Mistral, OpenAI, Anthropic)
Model A specific AI model (e.g., mistral-medium, gpt-4)
Token A unit of text that AI models process (input/output)
Cached Tokens Tokens that were already in the model's context cache
Fresh Tokens Tokens that are new in the current request (input_tokens - cached_input_tokens)
Hook A callback that runs at specific events in the harness lifecycle

Appendix B: Example Usage

Basic Usage:

# Quick summary of current session
agent-cost-estimator

# Full details
agent-cost-estimator -d

# Session cost only
agent-cost-estimator -s

# Turn cost only
agent-cost-estimator -t

# JSON output
agent-cost-estimator --json

# Specific session
agent-cost-estimator --session-id ~/.vibe/logs/session/unified/<uuid>

# Custom pricing file
agent-cost-estimator --pricing /path/to/pricing.json

# Override pricing for this run
agent-cost-estimator --input-price 2.0 --output-price 10.0

# Update pricing file with new rates
agent-cost-estimator --model my-model --input-price 2.0 --update-pricing

Integration:

# Install hooks for Vibe CLI
agent-cost-estimator --integrate mistral

# After installation, start a Vibe session
vibe "Your prompt"
# Output after each turn: -- mistral:mistral-medium | session:$X.XX | turn:$Y.YY

Configuration:

# List available harnesses
agent-cost-estimator --list-harnesses

# Print usage instructions for LLMs
agent-cost-estimator --skill

Appendix C: File Index

Core Modules:

File Purpose
agent_cost_estimator/__init__.py Public API exports
agent_cost_estimator/__main__.py Entry point
agent_cost_estimator/cli.py CLI interface
agent_cost_estimator/core.py Cost calculation
agent_cost_estimator/pricing.py Pricing data handling

Harness Adapters:

File Harness Status
harnesses/__init__.py Registry Implemented
harnesses/base.py Base interface Implemented
harnesses/vibe.py Mistral Vibe CLI Implemented

Hooks:

File Purpose
hooks/post_turn.py Vibe post-agent hook

Configuration:

File Purpose
pricing.json Default pricing
pyproject.toml Package metadata

Documentation:

File Purpose
SPEC.md This specification
README.md User documentation
skill/SKILL.md LLM usage instructions

Tests:

File Purpose
tests/test_agent_cost_estimator.py Test suite