fault

Download one file and run it

fault is released as a standalone executable. There is no archive to unpack and no service to install. Download the file into your user-local bin directory and make it executable.

Linux · x86-64

curl -fL https://github.com/fault-project/fault/releases/latest/download/fault-linux-x86_64 -o "$HOME/.local/bin/fault"
chmod +x "$HOME/.local/bin/fault"
"$HOME/.local/bin/fault" --version

macOS · Apple Silicon

curl -fL https://github.com/fault-project/fault/releases/latest/download/fault-macos-aarch64 -o "$HOME/.local/bin/fault"
chmod +x "$HOME/.local/bin/fault"
xattr -d com.apple.quarantine "$HOME/.local/bin/fault" 2>/dev/null || true
"$HOME/.local/bin/fault" --version

macOS attaches the com.apple.quarantine attribute to downloaded executables. Removing it allows this unsigned CLI to run without Gatekeeper treating it as a newly quarantined download.

Windows · x86-64

Invoke-WebRequest `
  https://github.com/fault-project/fault/releases/latest/download/fault-windows-x86_64.exe `
  -OutFile "$HOME\.local\bin\fault.exe"
& "$HOME\.local\bin\fault.exe" --version

curl -fLO https://github.com/fault-project/fault/releases/latest/download/SHA256SUMS
expected=$(grep '  fault-linux-x86_64$' SHA256SUMS | cut -d ' ' -f1)
echo "$expected  $HOME/.local/bin/fault" | sha256sum --check -

curl -fLO https://github.com/fault-project/fault/releases/latest/download/SHA256SUMS
expected=$(grep '  fault-macos-aarch64$' SHA256SUMS | cut -d ' ' -f1)
echo "$expected  $HOME/.local/bin/fault" | shasum -a 256 --check -

Invoke-WebRequest https://github.com/fault-project/fault/releases/latest/download/SHA256SUMS -OutFile SHA256SUMS
$expected = (Select-String '  fault-windows-x86_64.exe$' SHA256SUMS).Line.Split()[0]
if ((Get-FileHash "$HOME\.local\bin\fault.exe" -Algorithm SHA256).Hash -ne $expected) { throw 'fault checksum mismatch' }

Add $HOME/.local/bin on Linux or macOS, or %USERPROFILE%\.local\bin on Windows, to PATH to invoke the installed executable as fault.

Run your first fault

Let’s start with a complete example. We will listen locally, forward the connection to Google, and add 250 milliseconds to response traffic. fault forwards the TLS bytes without trying to understand HTTP.

schema_version: 1
name: slow Google connection
proxies:
  - name: google
    protocol: tcp
    listen: 127.0.0.1:18080
    upstream: www.google.com:443
phases:
  - name: slow responses
    proxies:
      - proxy: google
        faults:
          - type: latency
            flow: to-client
            distribution:
              type: uniform
              min_ms: 250.0
              max_ms: 250.0

fault run google-proxy.yaml

curl --connect-to www.google.com:443:127.0.0.1:18080 https://www.google.com/

--connect-to sends the TCP connection through fault while preserving www.google.com as the TLS server name and HTTP host.

At this point curl still speaks TLS directly with Google. The only difference is that every response byte crosses fault’s listener and therefore experiences the configured delay. The dashboard shows the active fault and the number of streams it affected.

Understand where the bytes go

There is no hidden interception. Your client connects to fault’s listen address and fault opens a second connection to the real upstream. A named proxy joins those addresses to a transport and a current chain of faults.

Directions are always named from the client’s point of view:

• to-upstream — requests or bytes sent by the client.
• to-client — responses or bytes returned by the upstream.
• both — both directions.

Faults are chainable and evaluated in declaration order. TCP fault state follows the stream. UDP latency, jitter, and blackhole operate at the selected datagram boundary. DNS faults are the only feature that inspects payloads.

Choose what should go wrong

Start with the failure you want the application to experience, not with a protocol feature. The table below shows the small vocabulary fault uses and the engineering questions each fault is useful for.

Bandwidth and connection reset are rejected for UDP. There is no honest UDP connection to reset, and a bandwidth scope would be ambiguous across independent exchanges.

Failures seen in real systems

Production incidents are usually combinations rather than isolated primitives. These patterns provide realistic starting points; the application’s metrics and behavior determine whether it handled them well.

Try a DNS failure

schema_version: 1
name: DNS failure
proxies:
  - name: dns
    protocol: udp
    listen: 127.0.0.1:15353
    upstream: 1.1.1.1:53
phases:
  - name: resolver failure
    proxies:
      - proxy: dns
        faults:
          - type: dns
            case: serv-fail

fault run dns-proxy.yaml

dig @127.0.0.1 -p 15353 example.com

Reproduce failures you actually see

Each recipe is a complete entry to paste beneath a run document’s phases: key. The examples assume named database, payments, and dns proxies are declared at the top level. A phase replaces the complete chain for every proxy it names, so faults meant to coexist appear together.

- name: resolver outage
  duration: 30s
  proxies:
    - proxy: dns
      faults:
        - type: dns
          case: timeout

- name: incorrect service discovery
  duration: 30s
  proxies:
    - proxy: dns
      faults:
        - type: dns
          case: random-a
          delay_ms: 100

- name: slow database completion
  duration: 2m
  proxies:
    - proxy: database
      faults:
        - type: latency
          flow: to-client
          distribution:
            type: uniform
            min_ms: 250.0
            max_ms: 500.0
        - type: bandwidth
          flow: to-client
          bytes_per_second: 32768

- name: extremely slow payment responses
  duration: 2m
  proxies:
    - proxy: payments
      faults:
        - type: bandwidth
          flow: to-client
          bytes_per_second: 8192

- name: congested remote region
  duration: 3m
  proxies:
    - proxy: payments
      faults:
        - type: latency
          flow: both
          distribution:
            type: uniform
            min_ms: 80.0
            max_ms: 140.0
        - type: jitter
          flow: both
          min_delay_ms: 20.0
          max_delay_ms: 250.0
          probability: 0.15
        - type: bandwidth
          flow: both
          bytes_per_second: 131072

- name: asymmetric partition
  duration: 45s
  proxies:
    - proxy: database
      faults:
        - type: blackhole
          flow: to-client

- name: load balancer rotation
  duration: 45s
  proxies:
    - proxy: database
      faults:
        - type: connection-reset
          flow: to-client
          probability: 0.3

- name: unstable payment dependency
  duration: 90s
  proxies:
    - proxy: payments
      faults:
        - type: jitter
          flow: both
          min_delay_ms: 100.0
          max_delay_ms: 1000.0
          probability: 0.5
        - type: connection-reset
          flow: to-client
          probability: 0.2

- name: checkout dependency incident
  duration: 60s
  proxies:
    - proxy: database
      faults:
        - type: latency
          flow: to-client
          distribution:
            type: uniform
            min_ms: 200.0
            max_ms: 450.0
    - proxy: payments
      faults:
        - type: connection-reset
          flow: to-client
          probability: 0.2
    - proxy: dns
      faults:
        - type: dns
          case: serv-fail
          delay_ms: 250

- name: database partition
  duration: 60s
  proxies:
    - proxy: database
      faults:
        - type: blackhole
          flow: both

- name: network recovered
  duration: 2m
  proxies: []

Let conditions change during the run

A useful experiment rarely ends with “the network is slow”. It asks what happens when a healthy dependency becomes slow and then recovers. Put those conditions into ordered phases. Each phase replaces the named proxy chains, runs for its duration, then yields to the next one.

schema_version: 1
name: database instability

proxies:
  - name: database
    protocol: tcp
    listen: 127.0.0.1:15432
    upstream: 127.0.0.1:5432

phases:
  - name: healthy
    duration: 5s
    proxies: []

  - name: slow
    duration: 10s
    proxies:
      - proxy: database
        faults:
          - type: latency
            flow: both
            distribution:
              type: uniform
              min_ms: 100.0
              max_ms: 250.0

  - name: recovered
    duration: 5s
    proxies: []

fault run scheduled-run.yaml

Programmatic schedules can add, modify, delete, start, or stop future phases. A phase becomes immutable as soon as it starts; invalid mutation is an error, never a silent no-op.

Put fault between client and dependency

fault does not provision cloud resources, rewrite routes, or inject itself into workloads. Deployment means running the proxy somewhere reachable and explicitly pointing the client at its listener.

Local or CI

fault run database-latency.json --journal fault-run.ndjson

Bind to loopback when only processes on the same host need access. Keep the journal as a CI artifact if the individual stream or exchange records matter.

Container or Kubernetes sidecar

command: ["fault"]
args: ["run", "/config/run.json", "--journal", "/records/run.ndjson"]

{
  "listen": "0.0.0.0:15432",
  "upstream": "database.default.svc.cluster.local:5432"
}

Mount configuration read-only. Mount a writable volume only when using --journal; otherwise let the dashboard or JSON output go to the container logs. Configure the application—not the cluster—to use the sidecar listener.

Drive fault from your own code

The same engine is available directly from Rust and through a thin PyO3 binding. Rust always owns lifecycle, fault semantics, schedules, events, validation, and errors.

Start the engine from Rust

use fault_engine::FaultEngine;
use fault_model::{
    DelayDistribution, FaultSpec, Phase, Proxy, ProxyFaults, Run,
    TrafficFlow, TransportProtocol, SCHEMA_VERSION,
};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let run = Run {
        schema_version: SCHEMA_VERSION,
        name: "permanent database latency".into(),
        proxies: vec![Proxy {
            name: "database".into(),
            protocol: TransportProtocol::Tcp,
            listen: "127.0.0.1:15432".into(),
            upstream: "127.0.0.1:5432".into(),
        }],
        phases: vec![Phase {
            name: "degraded".into(),
            duration: None,
            proxies: vec![ProxyFaults {
                proxy: "database".into(),
                faults: vec![FaultSpec::Latency {
                    flow: TrafficFlow::Both,
                    distribution: DelayDistribution::Uniform {
                        min_ms: 100.0,
                        max_ms: 250.0,
                    },
                }],
            }],
        }],
    };

    let running = FaultEngine::from_run(&run).start().await?;
    println!("listening on {}", running.endpoints().tcp[0]);
    tokio::signal::ctrl_c().await?;

    let summary = running.shutdown().await?;
    println!("{} streams impacted", summary.status.tcp.impacted);
    Ok(())
}

Start the engine from Python 3.14+

Python supplies familiar orchestration around the Rust capabilities without becoming a second domain API.

import asyncio
from faultlib import Engine, Run

run: Run = {
    "schema_version": 1,
    "name": "slow DNS",
    "proxies": [{
        "name": "dns",
        "protocol": "udp",
        "listen": "127.0.0.1:15353",
        "upstream": "1.1.1.1:53",
    }],
    "phases": [{
        "name": "degraded",
        "proxies": [{
            "proxy": "dns",
            "faults": [{
                "type": "dns", "case": "delay", "delay_ms": 500
            }],
        }],
    }],
}

async def main() -> None:
    async with Engine(run) as engine:
        async with asyncio.TaskGroup() as tasks:
            tasks.create_task(engine.run())
            print(engine.endpoints.udp[0])
            await asyncio.Event().wait()

asyncio.run(main())

Read connection records as they arrive

from faultlib import StatusEvent, TcpStreamEvent, UdpExchangeEvent

async def observe(engine: Engine) -> None:
    while engine.alive():
        match event := await engine.next_event():
            case StatusEvent(status):
                print(status.tcp.active, status.udp.active)
            case TcpStreamEvent(stream):
                print(stream.stream_id, stream.outcome)
            case UdpExchangeEvent(exchange):
                print(exchange.exchange_id, exchange.outcome)
            case None:
                return

Change future phases from Python

async with engine.schedule() as schedule:
    phase = await schedule.add_phase(
        "database latency",
        {"database": [{
            "type": "latency",
            "flow": "both",
            "distribution": {
                "type": "uniform",
                "min_ms": 100.0,
                "max_ms": 250.0,
            },
        }]},
        duration="30s",
    )
    await schedule.start_phase(phase)

    while transition := await schedule.next_transition():
        print(transition.phase.name, transition.phase.state)

This is deliberately ordinary Python. It can sit beside Kubernetes clients, test runners, cloud SDKs, or application-specific checks while the network hot path remains entirely in Rust.

See what happened to each connection

When an experiment surprises you, a counter saying “some traffic was affected” is not enough. fault can identify an individual TCP stream or UDP exchange and record what it did. It deliberately does not decide whether your application passed the experiment.

• Status is a cheap live aggregate: active, completed, impacted, failed, traffic averages, and applied delay.
• Transport records identify one TCP stream or UDP exchange and describe bytes, faults, outcome, and failure.
• The journal writes run metadata, completed transport records, and the final status as NDJSON.

fault run proxy.json --journal run.ndjson
fault --output json run scheduled-run.json --journal run.ndjson

Use fault with a real system

The examples now move beyond a single proxy. They show how to route an existing application, preserve TLS identity, retain records from a load test, and let a larger Python experiment control the network.

1. Slow database responses

Add 200–350 ms only to traffic returning from a PostgreSQL server. Client requests reach the database normally.

schema_version: 1
name: slow database responses
proxies:
  - name: database
    protocol: tcp
    listen: 127.0.0.1:15432
    upstream: 127.0.0.1:5432
phases:
  - name: degraded
    proxies:
      - proxy: database
        faults:
          - type: latency
            flow: to-client
            distribution:
              type: uniform
              min_ms: 200.0
              max_ms: 350.0

fault run database-latency.yaml

psql postgresql://localhost:15432/app

2. Reset a TCP connection

The first selected client-bound traffic resets the established connection.

schema_version: 1
name: reset database connections
proxies:
  - name: database
    protocol: tcp
    listen: 127.0.0.1:15432
    upstream: 127.0.0.1:5432
phases:
  - name: resetting
    proxies:
      - proxy: database
        faults:
          - type: connection-reset
            flow: to-client
            probability: 1.0

fault run connection-reset.yaml

3. Return a delayed NXDOMAIN

Reply locally after 750 ms. No query reaches the configured upstream.

schema_version: 1
name: delayed NXDOMAIN
proxies:
  - name: dns
    protocol: udp
    listen: 127.0.0.1:15353
    upstream: 1.1.1.1:53
phases:
  - name: missing record
    proxies:
      - proxy: dns
        faults:
          - type: dns
            case: nx-domain
            delay_ms: 750

fault run dns-nxdomain.yaml

dig @127.0.0.1 -p 15353 example.com

4. Degrade several dependencies in phases

One engine can own several named proxies. This run begins healthy, slows database responses, introduces a compound database/payments/DNS failure, then restores every proxy. A phase replaces the complete fault chain for each proxy it names; unnamed proxies are healthy in that phase.

schema_version: 1
name: checkout dependency degradation

proxies:
  - name: database
    protocol: tcp
    listen: 127.0.0.1:15432
    upstream: database:5432
  - name: payments
    protocol: tcp
    listen: 127.0.0.1:18081
    upstream: payments:8080
  - name: dns
    protocol: udp
    listen: 127.0.0.1:15353
    upstream: 1.1.1.1:53

phases:
  - name: baseline
    duration: 15s
    proxies: []

  - name: slow database reads
    duration: 45s
    proxies:
      - proxy: database
        faults:
          - type: latency
            flow: to-client
            distribution:
              type: uniform
              min_ms: 180.0
              max_ms: 320.0

  - name: cascading dependency failure
    duration: 30s
    proxies:
      - proxy: database
        faults:
          - type: latency
            flow: both
            distribution:
              type: uniform
              min_ms: 300.0
              max_ms: 600.0
          - type: bandwidth
            flow: to-client
            bytes_per_second: 32768

      - proxy: payments
        faults:
          - type: jitter
            flow: both
            min_delay_ms: 50.0
            max_delay_ms: 400.0
            probability: 0.4
          - type: connection-reset
            flow: to-client
            probability: 0.2

      - proxy: dns
        faults:
          - type: dns
            case: serv-fail
            delay_ms: 250

  - name: recovered
    duration: 15s
    proxies: []

fault run full-stack.yaml --journal full-stack.ndjson

5. Route an existing application through fault

No application code needs to know about fault. Keep the real dependency addresses in the proxy configuration, then override the environment variables the application already uses for remote endpoints.

DATABASE_URL=postgresql://database:5432/checkout
PAYMENTS_BASE_URL=http://payments:8080

DATABASE_URL=postgresql://127.0.0.1:15432/checkout
PAYMENTS_BASE_URL=http://127.0.0.1:18081

fault run full-stack.yaml --journal full-stack.ndjson

env $(cat .env.fault) ./checkout-service

6. Route an HTTPS request with HTTPX

fault forwards TLS without terminating it. Connect HTTPX to fault’s listener, but retain the upstream hostname for both HTTP and TLS identity. Certificate verification remains enabled.

import httpx

async with httpx.AsyncClient() as client:
    response = await client.get(
        "https://127.0.0.1:18080/",
        headers={"Host": "www.google.com"},
        extensions={"sni_hostname": "www.google.com"},
    )
    response.raise_for_status()

The URL selects the TCP destination. The Host header selects the HTTP authority, and HTTPX’s sni_hostname extension selects the TLS server name used for certificate validation. Do not replace this with verify=False.

7. Keep connection records from a load test

The dashboard remains light while the journal records completed streams or exchanges as NDJSON.

fault run api.json --journal fault-run.ndjson

oha -z 30s -c 10 http://127.0.0.1:18080/

jq -c 'select(.type == "tcp-stream-completed")' fault-run.ndjson

8. Coordinate several faults with a broader Python experiment

External orchestration stays ordinary Python. Here, pod deletion and application recovery checks belong to the experiment—not to fault—while one phase changes two dependency proxies atomically.

async with Engine(config) as engine:
    async with engine.schedule() as schedule:
        compound_failure = await schedule.add_phase(
            "database slow while payments restart",
            {
                "database": [{
                    "type": "latency",
                    "flow": "both",
                    "distribution": {
                        "type": "uniform",
                        "min_ms": 150.0,
                        "max_ms": 300.0,
                    },
                }],
                "payments": [{
                    "type": "connection-reset",
                    "flow": "to-client",
                    "probability": 0.25,
                }],
            },
            duration="30m",
        )

        await schedule.start_phase(compound_failure)
        await delete_test_pod("payments-0")
        await assert_checkout_remains_available()

        while (status := await engine.status()).tcp.impacted == 0:
            await asyncio.sleep(0.5)

        await wait_for_replacement_pod("payments-0")
        await schedule.stop_phase(compound_failure)

Teach an agent how to use fault

The repository includes a small fault-network-injection skill for coding agents. Give the skill to an agent when you want it to design or run a network experiment as part of a larger task.

The skill starts with the failure you are investigating. It guides the agent through routing, transport choice, fault chains, phases, and useful connection records. It also tells the agent when to use the CLI, the Rust engine, or the Python binding, and points it back to the generated schemas instead of asking it to guess field names.

Install the bundled skill into the user-level directory for your agent:

fault skill install --target codex
fault skill install --target claude
fault skill install --target opencode

fault asks whether to install for the current workspace or your home directory. Agents and scripts can answer directly with --scope workspace or --scope home. The command will not replace a modified skill unless you add --force. Use fault skill show to inspect or pipe the exact skill bundled with your version of fault.

For a smaller context window, start with the agent reference. For the complete workflow and safety guidance, use the SKILL.md file.

Use the reference when field names matter

The CLI accepts JSON and YAML, and both become the same Rust types. The human reference explains every field, fault variant, constraint, event, and result. Use the raw schemas when generating a run or checking an exact field. AI agents can begin with the shorter agent reference.

Build from source with current stable Rust. The Python binding requires Python 3.14 or newer.

cargo build --release -p fault-cli
uv sync --project fault-python --python 3.14 --reinstall-package faultlib