Docs

How it keeps working while you don’t.

How Nap Works

You describe an app in a sentence. An agent builds it on a machine of its own. The point of everything below is that you do not have to watch it happen.

One session, uncut: a sentence, a sandbox waking up, the agent reading and writing files, the project's own checks run against what it claimed, and the app it left behind.

A turn is one exchange: your prompt, whatever the agent does about it, and exactly one terminal event saying it finished or failed. A job is the objective behind it, and it outlives the turn — a trivial request is a job that opens and closes in a single turn, and a large one is a job that spans six, without anything having had to decide in advance which it was going to be.

you                 "a todo list with add, complete and delete"
  │
API                 admits it, writes it to the queue, answers — and runs none of it
  │
Worker              claims the request and holds a lease on your session
  │
Runtime             opens a job, says something, acquires a sandbox
  │
ContextEngine       assembles the prompt within a token budget
  │
AgentService        drives the model loop, one tool call at a time
  │                   read_file · write_file · edit_file
  │                   list_files · search_files · run_command
  │
sandbox             files land, the dev server reloads, the preview updates
  │
Runtime             persists → publishes → commits → verifies → snapshots
  │
you                 come back to it running

One turn, from a sentence to a running preview.

The last line of that diagram is where most of the engineering is. A turn that changed files is committed and then verified against the project’s own checks, because a model saying it is finished is a claim rather than a finding. A project nobody has touched for a while is committed, bundled and destroyed, because a sandbox is billed by the second. And every step of it is written to a durable, ordered event log, because the story of what happened has to survive you closing the tab.

Leaving costs nothing: An idle project is snapshotted to object storage and its sandbox destroyed. Your next message restores it — files and git history intact.
The work is not in your request: A worker claims the turn and runs it; the socket you were watching from is not the process doing anything. Closing the tab stops the watching and nothing else.
Coming back has a place to start: The transcript opens at the seam your reading stopped at, and one card above it says what was decided in your absence — if anything was.
Rejoining is one question: The transcript is a fold over an append-only log, so catching up is everything after seq. A reconnect an hour later is the same operation as a reconnect a second later.
Finishing is not the model's call: A completed turn is committed, then checked. Passing makes that commit a checkpoint; failing opens a repair turn carrying the failure.

Architecture

A thin vertical slice through five planes, with one component owning each thing and the boundaries between them doing the real work.

Browser (Next.js)                       ← presentation
   │ HTTPS + WebSocket
API server (Hono on Bun)                ← gateway · sessions · streaming hub
   │   admits and enqueues; executes nothing
   ▼
turn_requests (Postgres)                ← the queue; one leased turn per session
   │
Worker  claim ─► renew ─► settle        ← executes; serves nothing
   └── Runtime  (turn orchestration)    ← intelligence
         ├── ContextEngine ──► MemoryProvider
         ├── AgentService  ──► LLMProvider
         ├── SandboxManager ────────────► E2B sandbox      ← execution
         ├── EventStore (Postgres)         /workspace (git repo)
         ├── Verifier (@nap/verify)        vite dev :5173 → preview URL
         └── EventBus (in-process, or Postgres NOTIFY)

Reaper (exactly one)                    ← idle sweep · capacity · the janitor

The planes, and what sits in each.

Three processes, one image, and no call between them. The API serves and executes nothing; the worker executes and serves nothing; the reaper does neither and runs as a single replica. All three are built by one bootNap and differ only in the role they pass, so a new dependency is one edit rather than three that drift apart. A turn reaches a worker through the turn_requests table and nothing else. Scale is what that buys, and what it cost.

Who owns what

Component	Owns	Never does
TurnQueue	The durable queue of turn requests and the per-session leases that make one exclusive. Also the one answer to “is this session busy?”, so close, delete and the idle sweep all mean the same thing by it.	Running anything. Holding a credential — it records whether the asker pays, never their key.
TurnWorker	Claiming a request, renewing its lease, running it through the Runtime, settling the row — and aborting the moment a renewal says the lease is gone. Draining on shutdown.	Admission, ceilings, model access. Deciding what a turn does. Serving anything.
Runtime	The turn lifecycle: acquire sandbox, build context, run agent, persist, publish, commit, verify, snapshot, photograph. Budgets, cancellation, recovery. Opening and closing the job a turn belongs to.	Prompt content, model parameters, tool implementations. Deciding which checks exist.
ContextEngine	Assembling context, and owning the token budget and the order things are truncated in.	Calling the model; deciding when a turn ends.
AgentService	Driving the model loop for one turn, executing the proxy tools, emitting typed events.	Persistence, git, sandbox lifecycle, prompt assembly.
LLMProvider	Model policy — effort, thinking configuration, refusal and fallback, retries, usage accounting.	Deciding which models a caller may reach. That is the route's.
SandboxManager	Sandbox lifecycle, filesystem, exec, preview URL.	Knowing what an agent or a turn is.
EventStore / EventBus	Durable append, then fanout — in that order.	Business logic.

The dependency direction is enforced by a test, not by discipline

runtime ──► context · agent · sandbox · storage · capture · db · verify ──► shared

bench ──► verify · shared          (NapBench's pure half — tasks, scoring, reports)
apps/napbench ──► bench            (the shell: Playwright, real infrastructure)

the edge that must never exist:    runtime ──► bench

Which package may import which.

agent imports the SandboxManager interface and never the E2B adapter, which is what makes swapping E2B for something else a one-package change rather than an audit. verify sits below both the runtime and the benchmark: the runtime uses it to arbitrate a turn’s claim, the benchmark uses it to build a score, and the edge that must never exist is the system under test importing the thing that grades it.

None of that holds by vigilance. A test reads every package’s manifest and then the specifiers its source actually imports — type-only ones included, because the runtime hoists workspace packages and an undeclared import would otherwise resolve, typecheck and ship. Adding a violating import turns the suite red; adding a new workspace package fails it until its rule is declared. test/architecture.ts

Runtime & the Event Model

Every durable fact about a session is an event with a sequence number, written to Postgres before it is published to anybody watching.

Publishing first is faster, and it means a client can see an event that a crash then loses — after which the browser and the database disagree and nothing can say which of them is right. Fixing the order costs a write on the hot path and buys the property the rest of the system is built on: the log is authoritative, and what a client holds is a copy of it.

append to Postgres  (seq assigned, monotonic per session)
        │
        ▼
publish to the bus  (whoever is listening, if anybody is)

never the other way round.

The order that makes catching up decidable.

Because the order is fixed, catching up is a single question — everything after seq. A reconnect a second later and a reconnect an hour later are the same operation, so there is no separate resume path to get wrong, and joining mid-turn needs no special case.

The session log, and the views over it

A session log is one reader’s copy of the events: one socket, one seq, many derived views. There is exactly one per workspace — a second is two clients that can disagree about what the newest event was.

The transcript is the conversation you actually see, and it is a fold over that log rather than state of its own. It is much shorter than the log it comes from, because one tool call, everything it printed, the files it touched and how it ended are four kinds of event and a single thing on screen. Nothing is ever written into it: it is recomputed from the log every frame, which is why joining mid-turn, reloading the page and watching from a second tab all land on the same picture without anything having to be reconciled.

Three speakers, not two: The user, the agent, and the verifier — which is a fact about the log rather than a third party to the conversation.
A second view is another fold: Not another copy. Anything that wanted a different reading of the same session derives it from the log rather than keeping its own.
The notification is not the event: Across several server processes, publishing is a Postgres NOTIFY carrying a session and a seq and nothing else; each process then reads the events themselves out of the log. So a wake-up that never arrives costs latency rather than an event — a poll asks the same question every two seconds anyway — and the socket you are on need not be the process running your turn.

Two cursors, and why they must not share a word

The seq above is a replay cursor: per-connection, held in memory, and gone when the page closes. It answers “what have I been sent?” The second cursor answers a different question — “what has this browser ever displayed?” — and it is per-browser and durable, kept in localStorage against the session. The events after it are unseen, and where they begin is the seam: a line through the transcript, and where the transcript opens rather than at the bottom.

The seen cursor advances only while the document is visible. A background tab keeps its socket open and the worker keeps working, so counting what arrives there as displayed would make the feature fire in every case except the one it exists for. And having received nothing is not a cursor of zero: writing that first zero down turns “never opened” into “seen nothing of it” and puts the seam above the first thing anybody said.

Unseen is deliberately not away. Away names the user’s state, which nothing can observe; what is computed is a property of the log against a cursor. The copy on screen may well say “while you were away” — copy is allowed to be warmer than the concept, so long as the concept keeps its name in the source.

What sits above the seam is one card, and it fires on a conclusion — a job completed, checkpointed or failed among the unseen events — rather than on elapsed time or volume. Both of those fire on activity: gone four hours with nothing decided, and you would be told “47 events”, which is true and worthless. Most returns show no card, and that is what earns the one that appears its interruption. It is worked out once, when reading resumes, and then held still, because a card recomputed every frame would announce “while you were away” about the turn its reader is sitting there watching.

ADR-0008 — The transcript is a derived view, not a chat client

Durable Jobs

A job is one objective and the durable unit of work that outlives a turn: what was asked, what phase it is in, what has been verified, and how many repair attempts remain.

It has no table and no file behind it. A job is a fold over the session’s events, exactly as a turn is — so there is one source of truth rather than two that can disagree, and resuming is replaying. Every turn belongs to one. A job opens on a prompt and stays open until verification agrees it is satisfied or its attempts run out, which is why a trivial request and a six-turn build need no decision in advance about which they are.

open        working ──► verifying ──► repairing ──┐
                  ▲                            │
                  └────────────────────────────┘   up to 3 attempts

closed      verified     checks passed — the commit is a checkpoint
            unverified   the turn changed no files, so there was nothing to check
            exhausted    3 repairs spent, checks still red
            abandoned    the turn it was riding on was cancelled or refused

Three phases while open, four ways to close.

Unverified is the one worth pausing on: a turn that changed no files has not failed its checks, because there was nothing to check. Calling that a failure would put every conversational turn into a repair loop it cannot leave. It is not a success either, and it gets its own word rather than being folded into one of the neighbours.

Continuing, which is not resuming

A process restart leaves a job open rather than failing it. When the project is next opened, the open job is continued — and nothing continues a job while nobody is watching, which is deliberate: an unattended sandbox is a sandbox being paid for.

That is a different word from resume, which already means bringing a put-away project’s sandbox back up. The two are separate operations on separate things, and the glossary keeps them apart on purpose.

What you actually see of it

A strip above the transcript says where the current job stands, and it does not scroll away with the conversation — a status somebody goes looking for during the exact minute it matters should not be a thing they have to scroll to find. It is mirrored into the workspace bar, so collapsing the chat to give the preview the whole window does not take the one signal saying whether the project works off the screen with it.

Behind it is a history of jobs, and that word is the decision worth reading. job.checkpointed is written on the success path only, so a history built from checkpoints is a history with every failure deleted from it: green ticks, and no record that the thing asked for at 14:32 was attempted, repaired three times and abandoned. A panel whose job is to answer “what happened?” cannot be built that way. So Checkpoint keeps its strong meaning on the verified-commit line inside an entry, and the entries themselves include the ones that ended badly.

One function decides all of it: foldJobs in @nap/shared — pure, over the event list, with no I/O and nothing to mock.
A job is not a benchmark task: NapBench's task is a specification of work to be repeated; a job is one actual piece of work being done once. The words collide and both keep their names.

ADR-0006 — A completed turn is a claim, not a fact

Verification & Repair

turn.completed is the model’s claim that the work is done, not the system’s finding that it is. Verification is what turns one into the other.

turn.completed          the model's claim
      │
      ├── changed no files ──────────────► job closes: unverified
      │
      ▼
commit                  every completed turn commits
      │
      ▼
run the project's checks, cheapest first, stopping at the first failure
      │
      ├── passed  ──► the commit becomes a checkpoint, job closes: verified
      ├── failed  ──► a repair turn opens, carrying the failure
      └── errored ──► nothing was learned; job closes: abandoned

What happens after the model says it has finished.

A check has three outcomes, and the third is load-bearing

A check is one command, run in the sandbox, that passed, failed, or was absent. Absent is not failure, and the gap matters in both directions: a project with no test script has not failed its tests, and treating a missing script as a failure would put every fresh project into a repair loop it cannot leave.

Which checks exist is discovered from the project rather than declared by the model — read out of its manifest, so an agent cannot pass by claiming a check it never had. They run cheapest first and stop at the first failure, because the second failure teaches nothing the first has not already earned a repair turn for.

A verdict has three outcomes too, and they are not the same three

Passed: The run is sound. The commit becomes a checkpoint.
Failed: The project's own problem, and the thing a repair turn is for.
Errored: Nothing was learned about the project — the sandbox refused the command, or the preview listens inside and is unreachable from outside. A repair turn on that would ask a model to fix a machine it cannot see, so an errored run is never written as a verification and the job ends abandoned instead.

Repair is a turn, not a smaller thing

A failed verification opens a repair turn, and it is an ordinary turn whose prompt happens to come from the failure rather than from you. That is what makes it inherit budgets, cancellation, event ordering and commit-on-completion without any of them being rebuilt. The bound is attempts rather than a token ledger: three, after which the job closes exhausted with the last good checkpoint still intact.

Each repair carries a job brief — the objective, and every verification failure already seen on this job, oldest first. That second half is procedural memory done deterministically, and it exists because a transcript shows the model confidently finishing and never that the finish was rejected. Without it, each repair is free to make the last one’s mistake again. It is near-unevictable from the context window on purpose: the situation it exists for is a long repair with a full window, which is exactly when the turn that stated the objective has fallen out of it.

What a long job costs, and the ceiling that actually fires

Four turns on one project is where that stops being theoretical. Two ceilings exist and they are not the same ceiling: one caps a single request, the other caps the sum over a turn’s round trips — and a turn re-sends its whole transcript on every round trip, so its bill is roughly the assembled size times its step count. Both factors grow with the session. A real funded session died on the second ceiling while sitting at a fifth of the first, which means the truncation ladder was perfectly correct and had never run.

So tool traffic from any turn older than the most recent one is emptied unconditionally, before the budget is consulted. The call keeps its shape — which tool, against which path — so the turn still reads as something that happened; what goes is any argument big enough to be a file’s contents, and everything the call printed. Prose on both sides is never touched. That is a different question from truncation, which is still the only answer to this does not fit: fitting was never the test, being worth ten to forty copies is.

The failing session’s event log is committed to the repository unedited, so the measurement behind that reproduces for nothing rather than being a number in a paragraph.

The whole loop is confined to sandbox commands and a preview probe. Driving a browser stays the benchmark’s alone — nothing that ships carries Playwright.

ADR-0006 · ADR-0007 — The check primitive moves below both the runtime and the benchmark · ADR-0011 — An old turn’s tool traffic is not worth carrying

Sandbox & Snapshots

A sandbox is the isolated machine a project’s code is written into and served from. It is reclaimable at any time — which is a design constraint, not a caveat.

No agent harness, and the six tools are the reason

A batteries-included agent SDK ships built-in file and shell tools, and those act on the filesystem of the process running the harness — which here is one of our own server processes, not the user’s sandbox. So AgentService drives the model loop itself over an LLMProvider port, and the only tools that exist are these six.

read_file      write_file     edit_file
list_files     search_files   run_command

every one of them proxies to SandboxManager.
there is no seventh, and no filesystem but the sandbox's.

The whole tool surface.

That is stronger than disabling built-ins, because there is no toggle to get wrong. It is also what makes an unattended agent something you can walk away from: there is no reachable filesystem but its own.

An idle project is snapshotted, not paused

Keeping a sandbox alive so a project stays openable means paying for a machine nobody is using. So a reaper commits the workspace, bundles the git repository to object storage and destroys the sandbox. Restore is the inverse, and takes seconds. That turns “come back tomorrow” from a billing problem into a cold start.

A project in that state is put away: not an error and not an empty project, but the state a project spends most of its life in. The bytes and the bookkeeping are separate ports because they fail independently, and teardown ordering is only expressible if they do.

A checkpoint and a snapshot are different things: A checkpoint is about whether the work is sound; a snapshot is about where the work is kept. One is a verified commit, the other an archived filesystem.
A preview is identified by an event, not a URL: The seq of the preview.ready that announced it. A project put away and restarted has two announcements and only one live sandbox — so no view may read “there is a preview.ready in the log” as “something is running”.
The ceiling is claimed where the sandbox is made: A queued turn may not create anything for a minute, so counting sandboxes at admission counts the wrong moment. Capacity is reserved at creation, and the reaper reconciles it against what the provider says it is actually running — which is how a slot no path gave back comes back in minutes rather than never.
Somebody else's project answers 404: Not 403 — a 403 confirms the row exists, which is itself a fact about someone else's data. The authorization filter lives in the query rather than in a handler that might forget it.

packages/agent/src/tools

Scale

A turn used to run inside the request that asked for it. That is a shape with a hard ceiling and one expensive failure, and getting out of it is most of what the last stretch of work was.

The ceiling first: while the API process was the worker, a request’s lifetime was the turn’s lifetime. A deploy was a lost turn, a crash was a lost turn, and there was nothing to scale on its own — sockets and model loops shared one event loop, so the thing you would add capacity for and the thing you would add it because of could not be separated.

The failure is worse and is the one that actually decided it. Two API replicas accept two turns for one session, each calls acquireSandbox, and the project ends up holding two E2B sandboxes — one of which nothing references and nothing stops paying for. Keeping turns apart with a map of promises inside a process is correct in that process and enforces nothing across two.

one image, three commands, no call between them

API      bun apps/api/src/index.ts    serves HTTP + sockets    executes nothing
Worker   bun apps/api/src/worker.ts   executes turns           serves nothing
Reaper   bun apps/api/src/reaper.ts   sweeps, reconciles       exactly one replica

the only thing between them:  turn_requests, and the event log

The binary is split by role, not by codebase.

The queue is a Postgres table, deliberately

turn_requests, claimed with for update skip locked. Per-session exclusivity is a partial unique index — one leased row per session, enforced by the database rather than by every caller remembering — so busy means the same thing in every process, and the close, delete and idle-sweep paths all ask one question to find out.

Redis, JetStream and a hosted queue are all better queues than a table. Every feature they bring over one — retry policy, backoff, dead-lettering, a visibility timeout — is aimed at redelivery, and redelivery is the single thing this system must not do: a turn delivered twice is a model run somebody pays for twice. Recovery here is the event log’s job, not the queue’s, because a job is a fold over that log and already knows how far it got.

queued ──claim──► leased ──settle──► succeeded
                   │                     failed
                   │                     cancelled
                   └──lease expires──► the janitor closes it out

  no path back to queued — a redelivered turn is a second model run somebody pays for

A turn request has no path back to queued.

A worker renews its lease while it works and aborts the turn the moment a renewal says the lease is gone — a worker that has been declared dead must stop being alive. Shutdown is a drain rather than a stop: it quits claiming, keeps renewing what it holds, and aborts only what is left at the deadline.

Then the events have to cross too

Once a turn runs on a worker, every socket watching it is on an API pod, and an in-process bus reaches none of them. That failure is silent in the worst way — every turn executes perfectly and every chat pane sits still — so the two processes that publish refuse to boot without the cross-process bus rather than letting it be discovered from a browser.

The notification carries no payload: pg_notify sends a session and a seq; the receiving pod reads the events out of the log. Postgres caps a notification at 8,000 bytes, and the events that would exceed it are exactly the interesting ones — a build log, a written file. A payload-carrying design works in every test and then drops those.
A missed wake-up costs latency, not an event: A catch-up poll asks the same question every two seconds regardless, so the notification is an optimisation over a loop that would have got there anyway.
One delivery path, not two: The live path and the replay path read the same rows. A message and a row would be two copies of one event that can disagree, and only one of them would be the tested one.

What the ramp says

The same k6 script was run against one process and then against a Kubernetes cluster — three API pods, two-to-four workers, one reaper, KEDA scaling the workers on queue depth and an HPA on open sockets. Only the architecture underneath differs; the model and the sandbox are the same fakes at the same recorded speeds, which is why the whole thing costs nothing and can be run again.

At 100 concurrent turns the cluster ran 2,310 turns with 100% job, turn and verification completion — zero sequence gaps, zero duplicates, zero WebSocket failures, zero 5xx — including 219 mid-turn reconnects that each asked for the gap and got exactly it. Admission and delivery both got faster as load rose. The workers scaled on queue depth in both directions, and the API pods, sitting well under their socket target, were correctly left alone.

Two of the fifteen thresholds failed on the first run and both are written up rather than smoothed over: one was the fake measuring itself, and the other was real — the runtime acquired a sandbox before emitting anything, so a project’s first turn showed its author nothing for the length of a cold start. That is fixed, and the prompt is now drained before the acquisition. The cold start moved rather than vanished, into queue_wait, which is honestly green at two-thirds of its threshold and is named as the number to watch.

scaling-design.md — the semantics and the invariants · scaling-baseline.md — one process · scaling-cluster.md — nine pods, compared stage by stage · ADR-0009 · ADR-0010

NapBench

NapBench is the harness that measures Nap’s agent: a task is a reproducible unit of work put to it, a run is one execution of one task against one configuration, and a run ends as a report with a score and a trajectory.

Tasks are data, validated by a schema as they load, and independent of how Nap is built — so the same task can be pointed at a different model, prompt or context engine without being edited. A custom check kind was specified and deliberately not built: a check that was code is one no schema can validate and no sandbox can be handed.

What a score is made of

A score has two halves. The objective half asks whether the application does what was asked, and is a weighted mean over checks that a machine ran. The product half asks whether anybody would want to use what was built, and is a judge’s grades over screenshots. They are combined geometrically.

overall = √(objective × product)

  correct 95, beautiful 90  →  92
  correct 95, ugly      25  →  49
  broken  30, beautiful 90  →  52, then capped at 40 by the build gate
  broken  30, ugly      25  →  27

neither half can carry the other. under a weighted mean,
the second line lands in the eighties.

Why the two halves multiply rather than average.

Under a weighted mean, correctness buys the rest: an application that does exactly what was asked and looks terrible still lands in the eighties, which is not a result anybody shipping to a real user would call good. Multiplied, a weak half drags a strong one down towards it instead of being averaged away. The opposite direction was already covered — a preview that never serves fails the run outright, and a failed build caps it.

functional   50    does it do what was asked
browser      25    does it behave when driven
visual       15    v1, superseded — appearance is now the product half
code         10    typecheck and the accessibility audit

a category that produced no results is not scored zero.
the remaining weights renormalise over what was actually measured.

The objective half: the default category weights.

Functional dominates deliberately: an application that does not do what was asked is a failure however well it is written or laid out. The renormalisation is the part that keeps it honest — a category that produced no results is scored over what was measured rather than docked its weight for something nobody ran. An unjudged run is treated the same way and is scored on its objective half alone, never on a product half of zero.

How the product half is graded

excellent 95   good 78   moderate 55   weak 35   poor 12

nine dimensions, equally weighted: hierarchy, typography, spacing,
color, layout, components, interaction, responsiveness, restraint.
a tenth, polish, is reported and never scored.

not_assessable carries no number at all. it renormalises.

The ordinal scale, and the dimensions it is applied to.

The judge is shown screenshots and one neutral sentence about what the application is for, and nothing else — no source, no prompts. Screenshots-only is what stops it rewarding a stack it recognises rather than a product that is good; withholding the prompts is what stops it grading feature completion, which the checks already measure and measure better, because a check cannot be talked round.

And it is never asked for a number. A judge asked for 73 invents precision it does not have, and the same screenshots come back 68 the next run; asked whether typography is weak or moderate, it is making a judgement a reader can check against the evidence it cited. Every graded dimension has to carry that evidence, naming the screenshot it came from — enforced by the schema, because a prompt is a request and a schema is a refusal.

One judge carries one judge’s taste, and that is a disclosed limitation rather than a solved problem. What is done about it: the grades are ordinal so the bias is at least stable, every judgement records which judge and which rubric version produced it, and nine hand-written fixtures — the same application designed nine ways — exist to check that the judge can tell them apart at all. An evaluator nobody has watched discriminate is a check that has never been observed failing.

Sitting above all of it are gates: an ordered list of pure functions that constrain the outcome regardless of what the checks summed to. A preview that never serves fails the run. A failed required check fails the run. A build failure fails it and caps the score. Gates exist so a broken application cannot score well by being good at everything except working.

When there is no score

A run ends passed, failed, errored or cancelled. The first two are results and both have a score. Errored means no result was obtained, so there is no number to give — and an errored run is attributed to one of seven kinds, in four groups: the system under test, what it depends on, the instrument, and the operator.

That split is what keeps the benchmark honest. An agent that refused and a provider outage both produce no score, and only the first is evidence about the agent. What is measured is the model, with Nap held fixed, so the question is not whose code was at fault but whether the failure says anything about a model.

A run repeated has a spread; a run once does not: Mean, median, sample standard deviation and range — reported per task, never across a suite. A deviation over different tasks measures how much the tasks differ in difficulty, which is a fact about the benchmark rather than about the model.
A comparison is two runs, never three: Baseline and candidate, and what moved per category and per check. Refused outright when the two effective weight vectors differ, because a renormalised score is only meaningful relative to the categories that produced it — and refused across the two arithmetics, because both land on 0–100 and that is exactly what makes them dangerous side by side.
An unmeasured run yields no reward at all: Projected into an external harness, a report becomes named metrics on a 0–1 scale. An errored or cancelled run gets none of them: the format has no null, so the only alternatives are zero or nothing, and zero would convert our bad afternoon into the model's bad result. The full report is written either way — the reward is a lossy projection of a lossless artefact.
Route is not duration: Tool calls, tool failures, commands run and files touched — the claim 'same score, different route' is about two runs doing different things, so time and token counts are reported beside it and excluded from deciding it.

docs/NAPBENCH.md · ADR-0004 — NapBench measures the model · ADR-0012 — two halves, combined geometrically · ADR-0013 — graded ordinally, from screenshots

Scores from a run against fakes mean nothing and are not published as though they did; a funded run against real infrastructure gets a write-up of its own, in docs/napbench-*.md.

Decisions

Thirteen decisions that would be expensive to reverse, each recorded where it was made rather than reconstructed afterwards.

0001
NapBench splits into a pure package and an app
Tasks, scoring and reports are written against ports; Playwright belongs to the shell alone and to nothing that ships.
0002
Absent scoring categories renormalise rather than score zero
An unmeasured category is not a failed one. Scoring it zero would punish a run for a question nobody got round to asking.
0003
Metrics the event log cannot supply stay absent
Anything the event stream cannot answer is reported as missing rather than inferred, because an inferred metric is indistinguishable from a measured one once it is in a table.
0004
NapBench measures the model, and Nap's own faults are infrastructure
Fixes the frame of the whole benchmark: an agent that refused is evidence, and a sandbox that died is not.
0005
A navigation that never arrives is not a broken application
The preview gate has already proven the URL serves, so a check that never got there observed nothing and must not record a failure against the agent.
0006
A completed turn is a claim, not a fact
The decision the whole verification and repair loop is built on, and the reason a job exists at all.
0007
The check primitive moves below both the runtime and the benchmark
Running one check and saying whether it passed is shared; the edge that must never exist is the system under test importing the thing that grades it.
0008
The transcript is a derived view, not a chat client
Nothing is written into what you read; it is recomputed from the log, which is why three tabs cannot disagree.
0009
Turns execute on workers, behind a Postgres queue
A table beats a better queue here, because everything a better queue buys is aimed at redelivery — and a redelivered turn is a model run somebody pays for twice.
0010
Event fanout is notify-then-read, and the log is the delivery
The notification carries a session and a seq and never a payload, so the events it cannot fit are not the ones it silently drops.
0011
An old turn's tool traffic is not worth carrying, even when it fits
A funded session died on a budget while using a fifth of its context window. Fitting is not the test; being worth ten to forty copies is.
0012
The score becomes two halves, combined geometrically
Under a weighted mean, correctness buys the rest and an application that works and looks terrible lands in the eighties. Multiplied, neither half can carry the other.
0013
Product quality is graded ordinally, from screenshots alone
Screenshots-only is what stops the judge rewarding a stack it recognises; ordinal grades are what stop the same images coming back 68 one run and 79 the next.

The rest of the reasoning lives beside the code: what the constraints are and why the code is shaped around them is in docs/GOTCHAS.md, and what each concept is called is in CONTEXT.md — one concept, one name, which is why the words on this page are the words in the source.