Skip to content

ADR-002: Candidate-001 - Source Context for Error Parsing

Status: IMPLEMENTED
Date: 2026-02-12
Decision Makers: Jeff Mosley, Brian Moore

Context

The runner loop (THE KILLER FEATURE) executes build/test commands, parses errors, generates fixes via LLM, and retries. The current objective is to reduce average iterations from 3.1 → 2.0.

Problem: The LLM receives only the error message and line number, lacking surrounding context to understand the code structure and generate accurate fixes.

Decision

Implement Candidate-001: Source Context - add 2 lines of context before and after each error to improve LLM fix generation quality.

Single Variable Change

Add SourceContext and ContextStart fields to StructuredError:

// CANDIDATE CHANGE: Add context lines for better LLM fix generation
SourceContext []string `json:"source_context,omitempty"` // 2 lines before + error line + 2 lines after
ContextStart  int      `json:"context_start,omitempty"`  // Line number of first context line

Mode-Based Activation

Context extraction is enabled only in candidate mode:

if GetCurrentMode() == ModeCandidate {
    parser = NewMultiParserWithContext(true, 2)
} else {
    parser = NewMultiParser()
}

Implementation

Files Modified

File Changes
runner_loop.go Added SourceContext/ContextStart fields to StructuredError, mode-based parser selection
error_parsers.go Added GoErrorParser.EnableContext, ParseWithSource(), extractContext() helper
mode.go Added GetCurrentMode() helper function
metrics.go Added runner loop Prometheus metrics (THE KEY METRIC)

New Prometheus Metrics

Metric Type Description
cortex_runner_loop_iterations Histogram THE KEY METRIC - iterations per execution
cortex_runner_loop_total Counter Total executions by mode/outcome
cortex_runner_loop_duration_seconds Histogram Execution duration
cortex_runner_loop_errors_parsed Histogram Errors parsed per execution
cortex_runner_loop_fixes_applied Histogram Fixes applied per execution

Evaluation Results

Simulated Baseline (Stable Mode)

  • Total Tasks: 28
  • Pass Rate: 100%
  • Avg Iterations: 2.11

Simulated Candidate (With SourceContext)

  • Total Tasks: 28
  • Pass Rate: 100%
  • Avg Iterations: 2.07

Pass Criteria

  • ✅ avg_iterations decreased: 2.11 → 2.07
  • ✅ success_rate unchanged: 100% → 100%

Production Deployment

Servers

  • 192.168.11.132 (cortex-api-1)
  • 192.168.11.133 (cortex-api-2)

Activation

Set environment variable to enable candidate mode: 

export CORTEX_MODE=candidate

Monitoring

Query Prometheus for THE KEY METRIC: 

# Average iterations by mode
histogram_quantile(0.5, sum(rate(cortex_runner_loop_iterations_bucket[1h])) by (le, mode))

# Compare stable vs candidate
avg(cortex_runner_loop_iterations_sum / cortex_runner_loop_iterations_count) by (mode)

Rollback Plan

If candidate shows regression: 1. Set CORTEX_MODE=stable on both servers 2. Restart services: systemctl restart cortex-api 3. Document findings in this ADR

Next Steps

  1. Enable candidate mode on ONE server (192.168.11.133)
  2. Collect metrics for 1-2 weeks
  3. Compare avg_iterations between stable and candidate
  4. If improvement confirmed (target: 20% reduction), promote to stable

References