Give AI agents a metric. Keep only the wins that hold up.
Thermite is open-source autoresearch: AI agents that run experiments in a loop, for hours or days, on your own machines. You set the goal: a number to push (tokens a second, accuracy, latency, memory, cost, a benchmark score), a list of checks to pass, or a question to answer. A judge that the agents can't see or change measures every attempt, and keeps only what it confirms. One dashboard, with a REST API and MCP, shows and steers every campaign on every machine.
Quick start · Documentation · Architecture · Changelog · Roadmap · Changelog · Discord
Warning
Thermite is under heavy development. It changes every day, and some parts will break. If you don't want to take every update as it comes, pin a release: install from a tagged release's bundle, whose compose file pins every image by digest, and update when you choose. Most of Thermite can be driven from the dashboard, and its REST API and MCP server work well with coding agents, which can help you set it up, test it and run it.
Status: v0.2.4 is out: install it from its release bundle. A v0.1.1 to v0.2.3 stack updates to it in place; v0.1.0 breaks from them, so install fresh from it (changelog). Each fix to it is a release of its own (v0.2.5, v0.2.6), and new features go on toward v0.3.0: see the roadmap.
A lot of engineering is a loop. Change something, measure it, keep it if it is better, undo it if it isn't, and try the next idea. Autoresearch hands that loop to AI agents, so it runs through the night, on many ideas at once, without you.
The hard part is trusting the results. An agent asked to make a number go up will sometimes make it only look higher: it skips the slow case, moves the timer, or just reports a win. So in Thermite, agents only propose. A separate program, the judge, builds each change, runs it against the best version so far on the same hardware, checks that its output is still right, and decides. The agents can neither read the judge nor change it.
Each campaign is one of three kinds, called modes:
| Mode | Use it to | It runs until | For example |
|---|---|---|---|
| Hill climb | Push one number, such as speed, size or accuracy, as far as it goes, with guards so that nothing else gets worse | You stop it | Make a matmul kernel faster without breaking its tests |
| Goal | Meet a list of checks: it builds, it loads the model, its answers match | Every required goal is met, or the evidence shows that they cannot be | Port a library to a new platform: it builds, matches its goldens and meets a speed target |
| Research | Answer a question from sources. Code checks every quote, word for word, against the page it came from | A review panel finds every question answered, its research stops adding, or its longest time is up | How do modern memory allocators keep tail latency low? |
The modes combine: a hill climb or a goal campaign can read what a research campaign found. Choose a campaign mode helps you pick, campaign modes explains each, and the contract reference has every field.
- You set the goal. On the dashboard, or from a script or your own AI assistant, say what to improve and how to measure it: where the project is, which files the agents may change, and a kit that builds, runs and checks it. The New campaign page can write a simple kit from your answers.
- Agents try ideas side by side. Each agent works in its own lane, on its own copy of the code. A stateless planner gives each lane a different experiment from one shared queue, so the search stays wide. Every agent session starts fresh from the run's records, so a crash loses nothing.
- The judge decides. It builds each change and measures it against the best version so far, on the same device. A change that looks better must hold up again before it counts: on fresh runs, and on test cases the agents never saw when the campaign has them.
- Wins become the new starting point. The best confirmed version is the champion, and the next ideas build on it. A change that loses is kept too, with its code and the judge's reasons.
- You watch, steer and take the result. Follow each agent live, add ideas, send the agents a message or pause the run. When you are happy, download the champion as a patch for your project.
flowchart LR
Planner["Planner<br/>(gives each lane an experiment)"] --> Workers["Workers<br/>(a fresh agent for each trial)"]
Workers -- a change and a claim --> Judge["Judge<br/>(code the agents cannot change)"]
Judge -- confirmed win --> Champion[(Champion)]
Judge -- every result, with its reasons --> Ledger[(Ledger)]
Champion --> Planner
Ledger --> Planner
- Agents never grade their own work. They submit a change and what they expect it to do. "The agent says it's faster" and "it is faster" stay separate: only the judge's measurement counts.
- The judge holds the clock. It times each change and the champion in alternating blocks on the same device, decides with statistics, and times the champion against itself once a day to learn how noisy the machine is.
- A surprising win gets a second look. On a rig with an auditor, an unusually large gain is audited before it is kept. You can also ask to approve every new champion yourself.
- You can watch it catch cheats. Each project comes with a kit, the judge's half of the project: how a change is built, run, timed and checked. Agents never see it. Thermite's CPU smoke kit ships 12 changes whose right verdicts are known: a real speed-up, a no-op, a timer edit, a fallback to the reference code, NaN output, memory and disk exhaustion, and an attempt to overwrite the champion. The judge acceptance test requires each one to get its expected verdict, and the quick start shows it on your computer. Kits and judging explain the mechanism.
- It keeps going. There are no default caps on turns, submissions, failures or wall time. A campaign runs until its goals are met or its questions are answered, its research stops adding, its time is up, or you stop it; only the limits you configure apply.
- Your machines, your models. Run it on your own computer, on machines with GPUs over SSH, on a Kubernetes cluster, or on a RunPod GPU by the hour. The agents use a free model, a model server of your own, an API key, or a Claude or ChatGPT subscription.
- Nothing useful is thrown away. Every losing attempt is kept with its code and its verdict. Agents keep field notes as they work, and each campaign's records go into a knowledge base that later runs read.
- People stay in charge. One dashboard shows every campaign on every machine, live. You can pause, redirect, message, reset and reassign any run from the dashboard, the REST API or an MCP client, and decide which wins wait for your approval.
You need Linux or macOS, and Podman or Docker with compose. Thermite runs from published images: you build nothing. A model is optional at first: the judge works without one.
Before you give Thermite a model, read the known limits. Agents run the code that a model writes on your computer, and under compose that code can reach your local network. The security model says what each part can reach and what it trusts.
mkdir thermite && cd thermite
curl -fsSLO https://github.com/iammrduncan/thermite/releases/latest/download/compose.yaml
docker compose run --rm init # writes local tokens to .dev/
docker compose up -dWith Podman, write podman compose in place of docker compose. The first start downloads the images. To check the machine before the first up, run npx thermite doctor from the command line, built from the source with Node.js and git: it checks Docker or Podman, compose, the port, disk and the clock, and says how to fix each problem (Troubleshoot). Then:
- Open http://127.0.0.1:8800/ui and sign in with the token in
.dev/secrets/control-admin-token. - Open Rigs › local-1, and under Checks press Run it beside Judge smoke. The rig builds a small benchmark and submits known changes to its judge: a real speed-up, a no-op, broken builds and attempts to cheat. Each one gets its expected verdict within a few minutes on a CPU.
- Add a model under Settings › Providers (any OpenAI-compatible endpoint with tool calling, Anthropic, or OpenCode), then start a campaign from Campaigns › New campaign: on the sample project in
examples/matmul-cpu, a folder of this repository, or on your own project.
The tutorials walk through every step, up to taking the winning patch back to your project (Your own project). To build Thermite from its source code instead, see Contributing.
Just want to look around? npm ci && node --experimental-strip-types scripts/dev/dashboard-demo/main.ts serves the dashboard with synthetic campaigns and rigs at http://127.0.0.1:8810/ui, with no containers. It prints the token to sign in with.
flowchart TB
subgraph Control[Control service]
UI[Dashboard]
API[REST API and MCP]
end
subgraph Rig[Rig, repeated across the fleet]
S[Supervisor]
H[Harness: judge, ledger, planner, workers]
end
People[People and agents] --> UI
People --> API
S -- reports, signed commands --> API
S --> H
H --> M[Model endpoint]
S --> Net[Git, the internet, frontier models]
- A rig is two containers. The harness does the research. It needs only DNS, the internet and its model endpoint; on Kubernetes, network rules keep it to those, and under compose or on a machine it can also reach the local network (known limits). The supervisor does everything that needs more reach: git, reporting and relaying signed commands from control.
- Rigs run under docker-compose or podman-compose on one machine. Control can also set rigs up on your own machines over SSH, or on a Kubernetes cluster.
- Target drivers connect the judge to what is measured: CUDA GPUs and CPUs today.
More in architecture.
| Read this | To |
|---|---|
| Quick start | Go from the release's compose file to a first verdict, then to agents on your own project |
| Write a kit | Teach the judge how to build, run and check your project |
| Judging and integrity | See how results are accepted without trusting the agents |
| Known limits | Know the risks and what is not built yet, before you run it |
| The security model | See what each part can reach and what it trusts |
| Settings reference | Configure control, the harness and the supervisor |
| Contributing | Set up a copy, run the checks and the acceptance tests, and send a change |
| Glossary | Look up a term |
Thermite's community meets on the Hackers in the Loop Discord: questions, ideas and results go there. Bugs and feature requests go in GitHub issues.
- Contributing: CONTRIBUTING.md has the essentials. There is no contributor agreement and no sign-off rule. Read CONVENTIONS.md before you change the code, and run
npm run check(typecheck, lint, tests and the public-content check) before you open a pull request. Contributing lists the compose and acceptance tests for changes to containers, the judge or control. - Code of conduct: the Contributor Covenant 2.1.
- Security: report a problem privately, as SECURITY.md says, never in a public issue.
Thermite builds on other people's research and software. Credits lists the papers, systems, programs, libraries, standards and services it draws on, runs or ships, each with its licence.
Apache License 2.0. Version 0.1.0 was released under the MIT License.
