t.a.c AI Challenge — Q1 2026

When a conversation exceeds 80% of the token budget, the compaction engine fires. It is supposed to summarize the conversation so the agent can continue with context intact.

The model continues responding but it's flying blind. It re-reads files it already analyzed. It forgets constraints. It contradicts decisions it already made.

Additionally, the token estimator uses len(text) // 4 — a heuristic that underestimates code tokens by 25-40%, causing compaction to trigger later than it should.

1. Replace regex-based compaction with LLM-powered semantic summarization. Use an async Haiku call to produce a structured summary preserving: decisions, code changes, files modified, constraints discovered, open questions, and reasoning chains.
2. Design the summary schema. The compacted output must be structured (not free-form prose) so the agent loop can inject it reliably.
3. Fix the token estimator. Replace len(text) // 4 with accurate token counting.
4. Build a quality benchmark. 10+ real multi-turn sessions, compare old vs new, measure accuracy post-compaction.
5. Wire it in. Integrate into both agent_loop() and agent_loop_stream().

• ✓
  Semantic fidelity: Post-compaction, agent answers 5 session questions with ≥90% accuracy. Current scores <20%.
• ✓
  No regression: Token count reduced by ≥60%.
• ✓
  Token counting accuracy: Within ±10% of actual.
• ✓
  Latency: Haiku summarization adds <2 seconds.
• ✓
  Tests: Unit tests, integration tests, benchmark suite.

Extraction is regex: 18 hardcoded patterns decide what's "worth remembering." When a pattern fires, the entire raw message (500+ words) is saved — no extraction of the actual insight.

Retrieval is keyword Jaccard: "prod database connection" won't find "production postgresql" because exact match fails.

Eviction is blind LRU: Hard cap of 500 memories. Critical architectural decisions evicted because nobody accessed them recently.

1. LLM-powered memory extraction: Replace regex with Haiku calls that extract clean, atomic memory statements.
2. Vector-based semantic retrieval: Replace Jaccard with ChromaDB embedding search.
3. Importance-weighted eviction: Architectural decisions > preferences > casual mentions.
4. Team memory scope: Project-scoped memories visible to all team members.
5. Memory curation UI: /memories page — see, edit, pin, delete memories.

• ✓
  Extraction precision: ≥80% of memory-worthy statements correctly identified from 50 messages.
• ✓
  Retrieval recall: Correct memory in top-3 results ≥85% of the time (current: <30%).
• ✓
  Importance eviction: Memories with importance ≥0.8 preserved regardless of recency.
• ✓
  Team visibility: Two users on same project see each other's memories.
• ✓
  Web UI: /memories page renders with search, importance scores, pin/delete.

agent_loop.py is 1,200+ lines with three implementations: sync (~358 lines), streaming (~373 lines), and architect (sync only). They share zero code.

The AG-UI event system (13 event types, SSE, EventBus) exists but the agent loop emits zero events.

1. Unify into a single AgentLoop class with one pipeline and streaming/non-streaming output adapter.
2. Fix the three divergence bugs.
3. Wire AG-UI events — emit tool_call.start/end, text.delta, step events.
4. Build a live agent run viewer in the web console.
5. Zero regressions on existing tests.

• ✓
  Single implementation: No duplicated logic.
• ✓
  Bugs fixed: Hooks in streaming, checkpoints with summaries, architect in both modes.
• ✓
  Events emitted: Every tool call and text generation emits AG-UI events.
• ✓
  Web viewer: Real-time tool calls and streaming text visible in console.
• ✓
  Zero regressions: All existing tests pass.

9 agents exist as YAML files. Creating a new one means: open a text editor, write valid YAML, understand MCP extensions, know model tiers, define output schemas. A consultant can't do this.

1. Fully functional Agent Builder UI — guided form with live YAML preview and test sandbox.
2. Agent Marketplace page — browse all agents, fork & customize, categories, star/favorite.
3. Wire the Virtual Keys page for API access.

• ✓
  End-to-end: Non-technical user creates, tests, publishes an agent in <10 minutes.
• ✓
  Valid output: Generated YAML passes the validator.
• ✓
  Test sandbox: "Try this agent" sends a real request and shows output.
• ✓
  Marketplace: All 9 built-in agents browsable with Fork button.

"why is auth broken?" (22 chars) → routes to Haiku. Needs deep reasoning. Routing logic is duplicated between platform API and CLI with diverging keywords.

1. Semantic routing classifier using a Haiku meta-call.
2. Feedback loop from implicit user signals.
3. Unify routing logic — single implementation.
4. Routing analytics dashboard.

• ✓
  Accuracy: ≥80% agreement with human labels (current: ~50%).
• ✓
  Cost: ≥20% reduction from Opus misroutes.
• ✓
  Latency: Pre-flight adds <800ms.
• ✓
  Unified: CLI and API use same routing logic.

PII detection is detection-only (no redaction). Only 4 PII patterns. No prompt injection detection. No jailbreak detection. Regex-only.

1. Wire guardrails into the request pipeline — input & output checking.
2. PII redaction with typed placeholders before LLM call.
3. Prompt injection detection.
4. Fire guardrail.violation webhooks.
5. Violation dashboard in web console.

• ✓
  Runtime enforcement: 100% of requests checked. PII never reaches the LLM.
• ✓
  PII redaction: Typed placeholders. Optional restore-on-response.
• ✓
  Injection blocked: ≥90% detection on 20 known patterns, <5% false positives.
• ✓
  Webhooks fire: Every violation fires the event.

1. Connect Runner to Store — save state after every step.
2. Resume-after-interrupt — skip completed steps, continue from gate.
3. Parallel step execution — fan-out with asyncio.gather().
4. Emit AG-UI events at every lifecycle point.
5. Extended conditions — numeric, boolean, list membership, AND/OR.
6. Step-level retry and timeout.

• ✓
  Crash recovery: Kill mid-workflow. Restart. Resumes from last completed step.
• ✓
  Resume-after-interrupt: Paused workflow continues with human input.
• ✓
  Parallel execution: Fan-out runs concurrently (timing proves it).
• ✓
  Events: step.start, step.end, state.delta, interrupt, run.end in correct order.

1. Fix the auth bypass chain — auth enabled by default, API key users get identity.
2. Fix response cache — user_id + system_prompt_hash in cache key.
3. Add tenant isolation — org_id throughout the pipeline.
4. Distributed rate limiting — Redis for production.
5. Tamper-evident audit logging — hash-chain rows.

• ✓
  No anonymous access: All callers get identity with RBAC + rate limits.
• ✓
  Cache isolation: User A's response never served to User B.
• ✓
  Tenant isolation: Org A admin can't see Org B's data.
• ✓
  Tamper evidence: Deleted audit row breaks the hash chain.

• No reranking pass after RRF fusion.
• No entity-to-chunk linking — can't query "show all code using PostgreSQL."
• ACL filtering post-retrieval — unauthorized chunks briefly in memory.
• No recency signal — 2-year-old file ranks same as yesterday's.
• No query expansion or HyDE — vocabulary gap between natural language and code.

1. Add reranking — cross-encoder or Haiku reranking call.
2. Wire entity extraction into ingestion.
3. Push ACL filtering into SQL.
4. Recency-weighted ranking.
5. Query expansion + optional HyDE.

• ✓
  Reranking: ≥15% higher NDCG@5 than RRF-only on 50-query benchmark.
• ✓
  Entity queries: search_by_entity("PostgreSQL") finds psycopg2/pg_connection chunks.
• ✓
  ACL in SQL: Unauthorized chunks never appear in raw query results.
• ✓
  Recency: Recently modified chunk ranks higher when equally relevant.

1. No retry on transient errors — 429/503 terminates the session.
2. Hook modified_args built but never consumed — 90% built, 100% broken.
3. Background task failures swallowed — user never knows.
4. Empty LLM responses are terminal — no retry.
5. No circuit breaker — 5-tool run waits 10 minutes before failing.

1. Retry with exponential backoff — 429, 502, 503, 504. Max 3 retries. Respect Retry-After.
2. Connect hook modified_args to tool execution.
3. Surface background task failures in the REPL.
4. Retry empty responses.
5. Circuit breaker — fast-fail after N consecutive failures.

• ✓
  Retry works: Mock 503→503→200 — agent succeeds on 3rd attempt.
• ✓
  Hook transforms: modified_args applied to tool execution.
• ✓
  Failure notification: Background task failure surfaces within 5 seconds.
• ✓
  Circuit breaker: 4th request fast-fails after 3 consecutive failures.