ferret-scan Find and redact sensitive data before it leaks.

A single-binary Go CLI (plus embedded web UI and Go library — pkg/scan for detection, pkg/redact for redaction) that detects PII, secrets, and IP markers in your files and streams — then redacts them in place, format-preserving, with context-aware confidence scoring. No runtime dependencies. No data leaves your host.

A customer SSN pasted into a log line. An AWS access key committed in a diff. A support transcript archived to S3. A photo with EXIF location data attached to a ticket. Sensitive data leaks through the seams between systems — and once it lands in a log store or an object bucket, it is expensive to get back out. ferret-scan is the control you put in front of those seams: it finds the sensitive values, scores how likely each one is real, and redacts them so the rest of the data keeps flowing.

# 1. Install
pip install ferret-scan

# 2. Scan a file (values are HIDDEN by default — findings are safe to share)
ferret-scan --file secrets.env

# 3. Redact a stream in one pipe (redacted text -> stdout, findings -> stderr)
echo "card 5500-0000-0000-0004 from [email protected]" \
  | ferret-scan --stdin --enable-redaction

That last command turns sensitive values into masked-but-shaped output:

Sensitive values are masked while the shape of the data survives — so downstream tooling, tests, and log pipelines keep working. (Examples here use reserved documentation values.)

• Confidence you can act on. Scoring is context-aware, not just regex. A credit card in a financial document scores higher than the same digits in a test fixture, because the engine re-weights on document type and surrounding domain. The document type (CSV, TSV, JSON, configuration, code, ...) is read from the whole document, not from its first screen: every line up to 32 KiB and 2,000 lines, and evenly spaced windows of a longer one. Findings land in three bands: HIGH (90–100), MEDIUM (60–89), LOW (0–59).
• Redaction that composes. The stdin gateway streams redacted bytes to stdout and findings to stderr, so it drops into any Unix pipe or CI step. Three strategies: simple, format_preserving (default), and synthetic (realistic fakes for building test datasets).
• Explainable, offline. --explain attaches a plain-language rationale, a verdict (likely_real / likely_test / uncertain), and a drafted suppression reason to every finding. Fully deterministic — no network, no LLM.
• Ships everywhere. One static binary, no runtime deps, a scratch-based Docker image (~5–10 MB), a pip package, a Homebrew tap, a pre-commit hook, and a stable Go library (pkg/scan + pkg/redact).
• Safe by design. Matched values are hidden unless you pass --show-match. In-memory redaction produces payload-free audit records. The web UI binds to 127.0.0.1 with CSRF and CSP protections.

Pick whichever fits your workflow — all are first-class.

Docker one-liner (mount the current directory and scan it):

docker run --rm -v "$PWD:/data" public.ecr.aws/awslabs/ferret-scan:latest --file /data --recursive

Nineteen validators, each purpose-built. Enable a subset with --checks CREDIT_CARD,SECRETS,SSN or run them all (the default).

By default, matched values are hidden — the report is safe to paste into a ticket or share with a teammate.

LEVEL    VALIDATOR    TYPE                 CONF%    LINE       MATCH      FILE
[HIGH  ] ssn          SSN                   100.00% line     3 [HIDDEN]   demo.txt
[HIGH  ] email        BUSINESS              100.00% line     1 [HIDDEN]   demo.txt
[HIGH  ] secrets      AWS_ACCESS_KEY        100.00% line     4 [HIDDEN]   demo.txt
[MEDIUM] phone        PHONE                  75.00% line     3 [HIDDEN]   demo.txt

Pass --show-match to reveal the underlying values, and --explain to see why each was flagged.

Choose a format with --format:

text (default) · json · csv · yaml · junit · gitlab-sast · sarif

The sarif, gitlab-sast, and junit formats slot directly into GitHub code scanning, GitLab SAST reports, and CI test dashboards.

Scan files, directories, and globs

ferret-scan --file ./src --recursive
ferret-scan --file "logs/*.txt" --checks SECRETS,CREDIT_CARD
ferret-scan --file report.pdf --explain --format json --output findings.json

Redact a stream (composes in pipes; redacted bytes to stdout, findings to stderr)

cat customer-export.csv | ferret-scan --stdin --enable-redaction --redaction-strategy synthetic > safe-export.csv

Pre-commit hook — block secrets before they land

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/awslabs/ferret-scan
    rev: v2.5.5
    hooks:
      - id: ferret-scan

CI/CD — emit a SARIF or GitLab SAST report

ferret-scan --file . --recursive --format sarif --output ferret.sarif --quiet

Container — scan a mounted directory with no local install

docker run --rm -v "$PWD:/data" public.ecr.aws/awslabs/ferret-scan:latest --file /data --recursive

Web UI — folder drag-and-drop and bulk suppression management

ferret-scan --web --port 8080

Go library — embed redaction in-process (see below).

Launch a local, CloudScape-styled interface for interactive scanning, folder drag-and-drop, and bulk suppression management:

ferret-scan --web

It binds to 127.0.0.1 by default (auto-detecting 0.0.0.0 only inside containers, where the port publish is the boundary: publish it as -p 127.0.0.1:8080:8080), enforces CSRF/Origin checks, and sends CSP headers. The UI has no authentication, so to use it from another machine, forward the port over SSH (ssh -L 8080:127.0.0.1:8080 <host>) rather than widening the bind; --bind 0.0.0.0 exposes it to every host that can reach this one, and prints a warning at startup.

Two public packages let you embed ferret-scan in your Go application — no subprocess, no CLI, no payload leakage:

Detection and redaction are separate concerns — use one or both:

import "github.com/awslabs/ferret-scan/v2/pkg/scan"

// In-memory text detection (no disk, no temp files)
result, err := scan.ScanText(ctx, "card 5500-0000-0000-0004", scan.TextOptions{
    Checks:  []string{"CREDIT_CARD", "SSN"},
    Explain: true,  // attach "why flagged" rationale
})
if err != nil {
    return err // includes ctx's own error when it fired mid-scan; result then holds the partial findings
}
if result.Incomplete {
    log.Printf("not fully examined: %+v", result.NotExamined) // e.g. a validator size cap fired
}
for _, f := range result.Findings {
    fmt.Printf("%s (line %d, %s, %s)\n", f.Type, f.LineNumber, f.Band(), f.Rationale)
}

// File-based detection (PDF, DOCX, XLSX, images, text — 90+ types)
result, err = scan.ScanFile(ctx, "report.docx", scan.FileOptions{})
if err != nil {
    return err
}

// Check if a file type is supported before scanning
ok, reason := scan.CanProcessFile("archive.zip")  // false, "Unsupported file type"

// Redact text in-place using pre-computed findings (no re-detection)
redacted, err := scan.RedactText(text, result.Findings, scan.StrategyFormatPreserving)
if err != nil {
    return err
}

// Redact a file (writes a redacted copy of the same type: .docx→.docx, .jpg→.jpg)
// PDF, TIFF, GIF, BMP and WEBP are reported but cannot be rewritten — see docs/redaction-support.md
fileResult, err := scan.RedactFileContext(ctx, "report.docx", scan.RedactFileOptions{
    OutputDir: "/tmp/redacted",
    Strategy:  scan.StrategyFormatPreserving,
})
if errors.Is(err, scan.ErrIncomplete) {
    // Not fully examined (cut short, unparseable, no text): fileResult.Coverage says why,
    // and nothing at fileResult.RedactedFilePath is safe to share.
    return err
} else if err != nil {
    return err
}

pkg/scan exposes: ScanText, ScanFile, RedactText, RedactFile, RedactFileContext, CanProcessFile, CheckNames, ConfidenceOf, ParseStrategy. All delegate to the internal engine with zero duplication. Check the error before using any result: a redaction that could not examine its whole input returns an error wrapping ErrIncomplete (redact.ErrIncomplete below), never a quiet success.

The METADATA validator requires filesystem access — available via ScanFile but not ScanText.

pkg/redact is the higher-level API when you want both detection and redaction in one step with a reusable, pre-warmed engine. An Engine is built once and reused; it is safe for concurrent use.

import "github.com/awslabs/ferret-scan/v2/pkg/redact"

engine, err := redact.NewEngine(redact.EngineOptions{
    Checks:   []string{"CREDIT_CARD", "EMAIL"},
    Strategy: redact.FormatPreserving,
})
if err != nil {
    log.Fatal(err)
}
defer engine.Close()

result, err := engine.Redact(ctx, redact.Request{
    Text:  "card 5500-0000-0000-0004 from [email protected]",
    Label: "req-abc-123",
})
if err != nil {
    // errors.Is(err, redact.ErrIncomplete): the scan was cut short (ctx's deadline, a size cap)
    // and result.Redacted may still hold values in clear. Log result.AuditRecord(); never release it.
    return err
}
log.Println(result.Redacted)              // ****-****-****-0004 from j*****@example.com
log.Printf("%+v", result.AuditRecord())   // payload-free audit summary

Result.AuditRecord() returns a payload-free record — per-type finding counts, byte counts, duration — and never the matched bytes. LogWriter defaults to io.Discard, so the no-leak property is enforced by construction. Matched substrings are only reachable via explicit opt-in (Result.FindingsWithMatchText()). Input is capped at 100 MB.

Because both packages run entirely in memory — no subprocess, no filesystem, and payload-free audit records — you can build a single-tenant redaction service without ferret-scan ever writing sensitive bytes to disk or logs. A representative shape:

• Embed the Engine in an AWS Lambda (provided.al2023, arm64), constructed once in init() and reused across invocations.
• Front it with an API Gateway HTTP API using IAM (SigV4) auth.
• Callers POST JSON { "text": "...", "strategy": "format_preserving" } and receive { "redacted": "...", "request_id": "...", "duration_ms": <n> }.
• CloudWatch records audit counts only — never payload bytes — because AuditRecord carries no matched substrings.
• Gateway throttling bounds abuse and DoS.

This is an architecture the public API enables; ferret-scan itself ships the CLI, web UI, and library — not the gateway. Note that Lambda synchronous invokes cap request bodies at ~6 MB.

Pre-commit mode (--pre-commit-mode) maps findings onto the exit code instead, so a commit can be blocked; 3 still overrides a 0 there.

The distinction that matters for CI: 0 means the tool worked, not that the file was clean. Parse the output for findings, and treat 1 as "this report is not trustworthy" — a formatting failure now exits 1 rather than writing the error where the document should be.

• Values hidden by default. Findings never include matched text unless you pass --show-match (CLI) or opt into Result.FindingsWithMatchText() (library).
• Payload-free audit records. The library's AuditRecord reports counts, sizes, and timing — never the sensitive bytes.
• Memory scrubbing. Sensitive buffers use a SecureString with multi-pass zeroing (a MEDIUM security posture, bounded by what the Go runtime allows).
• Suppression system. Rule-based false-positive management, with bulk operations in the web UI, so you tune signal without editing source.
• Hardened web server. 127.0.0.1 binding by default, CSRF/Origin checks, and CSP headers.

For the full model — trust boundaries, threats, and mitigations — see THREAT_MODEL.md.

Apache-2.0. Copyright Amazon.com, Inc. or its affiliates. An awslabs open-source project — contributions welcome. See LICENSE.txt.