Tracecheck

Why I built it

When I saw Jev released, I started thinking about where I'd actually use it. A model built around focused decisions raised a different set of possibilities than another model I could ask to write code. I wanted to build something with it and find out what the surrounding application needed to do to make those decisions useful. Code review gave me a problem worth testing. A coding agent can explain a suspected bug convincingly and still be wrong. It might miss a caller that validates the input, assume a requirement that doesn't exist, or keep defending its first interpretation after the evidence changes. Asking it to review its own work again can leave those assumptions untouched. I built Tracecheck to give that agent a separate check on a specific concern. The agent investigates the code, records what it thinks is wrong, and gathers the evidence. Tracecheck checks the references and asks Jev whether the concern holds up. The agent then has to account for that result before deciding what to do. This was also a way for me to test building with Jev beyond a standalone API example. A useful result needs source locations, enough surrounding context, a way to represent uncertainty, and a next step when the answer is inconclusive. Tracecheck puts those pieces around the model through a CLI, a local MCP server, and an agent review skill.

What Jev does

Jev is TypeSafe's model for decision tasks. Its API returns typed decisions and probability distributions rather than generated prose. You give it context and questions with defined answer types. The application can then use the answers without extracting a verdict from a written explanation. That fits questions such as whether an excerpt supports a claim, which category a request belongs in, or how an item scores against an ordered rubric. These are still semantic questions. A regular expression cannot decide whether a caller satisfies a function's contract. But the application often needs a bounded answer, not an essay. Jev exposes three question types. Tracecheck uses them for different parts of a review:

Choice selects among named alternatives. Focused verification uses it for support, potential impact, and missing-evidence categories.
Noul expresses a yes/no judgment as a probability. The broader quality assessment uses it to ask whether a dimension is relevant and whether the evidence is sufficient.
Score returns an ordered assessment. Tracecheck maps these judgments to per-dimension quality scores when its evidence and confidence gates permit a score.

Several questions can share the same source context in one request. The return values have known shapes, so Tracecheck can validate them, retain uncertainty, and compare compatible checkpoints in code. Jev supplies the judgment; it doesn't generate the repair or execute a test. The distinction matters more to me than receiving another review paragraph. I can design an explicit response to missing context. I can keep applicability separate from quality. I can show that a concern was rejected without turning that into a claim that the entire change is safe. None of that makes the judgment infallible. Type safety tells me how to read an answer, not whether to trust it.

Where Tracecheck fits in a code review

A linter is useful when a rule can be checked mechanically. Tests are useful when I can execute behavior and compare it with an expectation. An agent is useful for following code across files, interpreting requirements, and deciding which concerns deserve attention. Tracecheck adds a focused semantic check between that investigation and the decision to act. It doesn't send a second agent off to explore the repository. In the primary verification path, Jev sees the hypothesis, contract, and evidence the first agent selected. It cannot discover the missing caller on its own. The agent records its provisional verdict before the call, then compares that verdict with Jev's response. Tracecheck doesn't need to send the provisional verdict as an input field for that comparison to happen. When the agent and Jev disagree, the agent can investigate the disputed assumption or gather the missing evidence. Two model verdicts do not settle the question by themselves. A supported concern warrants investigation. A rejected concern doesn't certify the rest of the repository. The coding agent still owns the patch and the project's checks.

Evidence before a verdict

The primary workflow starts with a hypothesis, not a request to find something wrong. The agent records the suspected defect, its provisional verdict, the relevant contract, and both supporting and contradicting evidence. That gives the independent check something concrete to disagree with. Each hypothesis points to an exact source quote and original line range. Tracecheck checks that the quote matches the supplied evidence. When the caller supplies a repository, it also checks referenced files against the local checkout before and after inference. With a repository supplied, stale or fabricated excerpts fail validation. Without that repository check, the report identifies the evidence as caller-supplied. That boundary matters. A local-file match establishes that the excerpt came from the cited checkout. It doesn't prove the agent found every caller, understood an external service, or supplied the right requirements. Missing evidence stays visible in the result.

The agent review loop

The bundled skill makes the process repeatable while keeping investigation and edits with the calling agent. These are instructions for the calling agent, not a background loop that the MCP server runs autonomously. The agent first establishes the requested behavior and review scope. For each material concern, it records a falsifiable hypothesis, trigger, source location, and provisional verdict. It then looks for counterevidence before asking Jev. An enclosing guard, an intentional error contract, or a caller that constrains inputs can change the answer. After verification, the agent compares the judgments. If Jev identifies a missing contract or the agent disagrees with its interpretation, the agent goes back to the relevant code. It expands the evidence only when doing so answers a specific question, then can retry at most twice for that hypothesis. Repeating the same packet until a preferred answer appears is not a useful review strategy. If the needed evidence is unavailable, the case stays uncertain. When the concern warrants a repair, the agent demonstrates the behavior with a suitable reproducer or regression test where practical, makes the change, and runs the normal project checks. It re-reads the evidence after the repair and can verify the revised behavior. The final report distinguishes model judgments from checks that actually ran, and preserves material disagreements and unresolved questions.

A concrete example

Consider a function that divides by an input value. The agent suspects the divisor can be zero. That suspicion alone is not enough to report a bug. The relevant evidence includes the function, the caller that supplies the value, and the contract for invalid input. If the packet leaves out the caller, missing context is a useful result. The agent should inspect how the value reaches the function. A guard may rule out zero on that path, or the call chain may reveal an unguarded input. The next verification uses that additional evidence. This is an illustrative case, not a benchmark result; the point is that the loop improves the question before it changes the code. A repair still needs an execution check. An updated excerpt that looks correct to Jev is not proof that the application behaves correctly.

Architecture

A local tool with an explicit provider boundary

Tracecheck runs locally through a CLI or MCP over stdio. There is no hosted Tracecheck application backend. Live assessments send selected code context to Jev through TypeSafe or OpenRouter, using the caller's provider key. Local execution does not mean local inference. Four MCP tools expose the workflows:

tracecheck_verify checks an agent-selected hypothesis against supplied evidence.
tracecheck_preview shows repository collection, packet membership, and coverage gaps without inference.
tracecheck_review assesses the collected repository changes.
tracecheck_assess evaluates supplied context across the quality dimensions.

The same underlying implementation supports terminal use and agent clients. Reports retain source references, uncertainty, and provider usage rather than returning only a prose review.

Bounded repository collection

For a repository review, Tracecheck compares the working tree with a Git baseline. It collects changed source, baseline excerpts, and supporting context such as callers and related tests. JS/TS and Python import discovery helps find that context, but remains heuristic. Dynamic behavior and unresolved imports can still be missing. Large changes are split into packets. Each packet has room for up to 16 collected files, with at most eight primary changed files so supporting evidence has space too. Large files use bounded excerpts with original line anchors and visible omissions. Assigning every eligible changed file to a packet does not mean every changed line fits in the review. MCP preview returns a snapshot token. Review recollects the context and rejects a changed snapshot, then checks freshness again after inference. CLI review collects its own context and marks a report stale if the reviewed evidence changes during the request. Those checks stop a result from silently referring to an earlier version of the code.

Broader quality without one overall grade

Focused verification is the main agent workflow. An optional broader assessment considers 19 quality dimensions, including correctness, readability, test quality, security, and maintainability. Performance, scalability, compatibility, and observability depend on the context of the change. Each dimension keeps its own relevance, evidence sufficiency, score, and confidence. Insufficient evidence can leave it unscored. Tracecheck does not average those judgments into one overall grade that could hide a correctness concern behind good formatting. Quality scores never justify unrelated refactoring or expanding the review scope. Parser-derived checks add exact source candidates for a narrower set of JS/TS patterns, including division and remainder boundaries, swallowed failures, and JSON parsing boundaries. A matching pattern is a hypothesis for assessment, not an automatic bug report. Agent-selected verification accepts evidence in any language, but that is not a claim of equal accuracy across languages.

Comparing checkpoints without claiming a fix

Saved reports let an agent compare review checkpoints locally. Tracecheck checks whether the scope, model, and assessment policies are compatible before computing quality deltas or finding history. Earlier scores are not sent to Jev as evidence about the current implementation. A finding can disappear because the code changed, because the available evidence changed, or because a model judgment changed. The comparison does not label that disappearance as a verified fix. The agent still needs to inspect the change and run the project's checks. This complements the source-retrieval work in Obsidian Vault RAG. Retrieval gives an agent relevant context with citations. Tracecheck asks whether a particular conclusion follows from the evidence the agent selected.

Request limits and trust

Preview lets the caller inspect what collection found before a live review. Request budgets, context limits, timeouts, and bounded concurrency constrain provider use. A committed .tracecheck.json can supply task context and tighten default limits, but cannot redirect the provider or raise those limits. It rejects recognized credential fields and values. Expanding the review belongs to the person or client running it. Secret screening catches known credential patterns, not every possible secret. Reports contain source excerpts and repository metadata, so they need the same care as the source itself. Tracecheck does not execute the reviewed project's tests or build, though Git collection can invoke configured clean filters in an existing checkout. The security model explains that distinction.

Tech stack

TypeScript and Node.js for the CLI, collection, review orchestration, and comparisons
MCP over stdio for agent integration
Jev through TypeSafe or OpenRouter for typed semantic judgments
Babel for parser-derived JS/TS candidates and Zod for schema validation
Git for baseline source and change collection
esbuild for the bundled runtime, with GitHub Actions for offline validation and release checks

What I still need to establish

Building the tool is not the same as proving that the extra check improves an agent's reviews. The project has offline tests, live validation records, and a protocol for comparing an agent's initial findings with its findings after Tracecheck. It has not completed a fresh, blinded agent-only versus agent-assisted study. That is the next question I care about. Does the extra judgment catch mistakes, prevent unnecessary edits, or identify missing evidence often enough to justify the additional work? The comparison needs to count missed defects and cases where the assistance hurts, not only examples where a second check looks helpful. More investigation time can improve a review by itself, so a fair comparison also needs an agent-only continuation with the same budget. The evaluation protocol records those distinctions. For now, I can describe how the loop works and what has been tested without claiming a general accuracy improvement.

Open source

Tracecheck is MIT licensed on GitHub. Version 0.4.0 is available as @bmccarn/tracecheck on npm and through the project's agent plugin marketplace. The installation guide covers the CLI and plugin paths, supported Node.js versions, and provider configuration. The validation record separates automated checks from observed live results. Broader source checks and executable fix verification remain future work. The check asks whether the selected evidence supports a concern and identifies what the agent needs to investigate next.

Tracecheck

Why I built it

What Jev does

Where Tracecheck fits in a code review

Evidence before a verdict

The agent review loop

A concrete example

Architecture

A local tool with an explicit provider boundary

Bounded repository collection

Broader quality without one overall grade

Comparing checkpoints without claiming a fix

Request limits and trust

Tech stack

What I still need to establish

Open source

Related projects

Rollo Label Printer

Garmin MCP Server

blakemccarn.dev

Charlotte Wire & Cable

Paperless OCR Enhanced

Paperless Knowledge Graph

RapidEPR