The Scientific Method for Prompt Engineering

A local-first CLI evaluation harness to benchmark prompt styles with statistical rigor. No cloud, no bias, just data.

$ npm run cli run --config configs/experiments/default.yaml
$ npm run cli analyze exp-mrml9htj
┌──────────────────┬────────┬──────────┬─────────┐
│ Style            │ Score  │ Latency  │ Cost    │
├──────────────────┼────────┼──────────┼─────────┤
│ Declarative      │ 0.92   │ 1.2s     │ $0.004  │
│ Policy-Based     │ 0.89   │ 1.4s     │ $0.005  │
│ Example-Driven   │ 0.87   │ 0.9s     │ $0.003  │
│ Procedural       │ 0.83   │ 1.1s     │ $0.004  │
│ Minimal          │ 0.74   │ 0.8s     │ $0.001  │
│ Verbose          │ 0.68   │ 2.1s     │ $0.008  │
└──────────────────┴────────┴──────────┴─────────┘
git clone https://github.com/rmax-ai/promptbench.git Docs on GitHub →

Engineering, not guesswork.

The Problem
  • ✗ Opaque quality scores hide real failures
  • ✗ Evaluator bias contaminates results
  • ✗ Results that can't be reproduced
The Solution
  • ✓ 8 deterministic graders for objective facts
  • ✓ Blinded model judge eliminates bias
  • ✓ Hashed manifests for complete reproducibility

Local-First Execution

Define
Load JSONL dataset, validate schema
Render
Render 6 prompt styles from templates
Execute
Run API calls with bounded concurrency
Grade
8 deterministic graders + blinded judge
Analyze
Aggregates, paired tests, bootstrap CIs

Battle-test every strategy.

Six prompt styles, same task, controlled comparison.

Minimal
Lower bound

Goal + evidence + schema. Smallest viable baseline.

You will decide the outcome.
Evidence:
- Refund window: 30 days
Return: {decision, explanation}
Procedural
Imperative prompting

Explicit step-by-step workflow.

Step 1: Inspect the request.
Step 2: Inspect each evidence item.
Step 3: Compare with decision criteria.
Declarative Contract
Contract-oriented

Role, goal, success criteria, constraints, stopping condition.

Role: Decision analyst.
Success: Correct outcome + cite evidence.
Stopping: Return schema after decision.
Verbose Scaffolded
Dilution & overhead

Detailed role, repeated reminders, extensive guidance.

Remember: verify every fact.
Important: cross-reference all evidence.
Caution: do not extrapolate.
Policy-Based
Policy classification

Invariants, decision policies, fallback rules.

INVARIANTS:
- Never claim evidence you do not have.
DECISION POLICY:
- Deny if past deadline without defect.
Example-Driven
Few-shot guidance

Compact instruction + 2–3 examples.

Example 1: Past deadline, no defect → Deny.
Example 2: Defect documented → Approve.
Now evaluate the case below.

Zero-Bias Evaluation.

Deterministic graders handle the facts. A blinded LLM judge evaluates nuance.

Deterministic Graders
✓ JSON parse validity
✓ Schema validity
✓ Decision match
✓ Required fact coverage
✓ Forbidden claim detection
✓ Output completeness
✓ Refusal detection
✓ Explanation length
Model Judge
BLINDED
  • Cannot see prompt style name
  • Cannot see model identity
  • Cannot see latency, tokens, or cost
  • Only sees: case, evidence, rubric, response

CLI-native workflow.

Every command is inspectable, scriptable, and reproducible.

$ npm run cli init # Create .env from .env.example
$ npm run cli validate --config configs/experiments/default.yaml # Validate config, dataset, prompts
$ npm run cli run --config configs/experiments/default.yaml --dry-run # Estimate 1,200 requests, ~$40 max
$ npm run cli run --config configs/experiments/default.yaml # Execute full benchmark (resumable)
$ npm run cli analyze exp-mrml9htj ┌──────────────────┬────────┬──────────┬─────────┐ │ Style │ Score │ Latency │ Cost │ ├──────────────────┼────────┼──────────┼─────────┤ │ Declarative │ 0.92 │ 1.2s │ $0.004 │ │ Policy-Based │ 0.89 │ 1.4s │ $0.005 │ │ Example-Driven │ 0.87 │ 0.9s │ $0.003 │ │ Procedural │ 0.83 │ 1.1s │ $0.004 │ │ Minimal │ 0.74 │ 0.8s │ $0.001 │ │ Verbose │ 0.68 │ 2.1s │ $0.008 │ └──────────────────┴────────┴──────────┴─────────┘

Quality and Cost, unbundled.

0.844
Mean Quality Score
2,152 ms
Avg Latency
100%
Pass Rate
100%
Schema Validity

Built for the Long View.

Every run generates a signed manifest. Past experiments remain interpretable even if the library changes.

Run Manifest
├── experiment_id: exp-mrml9htj
├── config_hash: 3b4da5ad...
├── dataset_hash: 5d22b030...
├── prompt_hashes: {minimal: 66260043...}
├── model: gpt-5.6-luna
├── seed: 481516
├── sdk_version: 5.23.2
└── node_version: v24.15.0

Ready to benchmark?

$ git clone https://github.com/rmax-ai/promptbench.git
$ cd promptbench && npm install
$ cp .env.example .env  # add your OpenAI API key
$ npm run cli validate --config configs/experiments/default.yaml
$ npm run cli run --config configs/experiments/default.yaml --dry-run
GitHub Roadmap