Intelligence WatchDog

Overview

The problem. Operations leaders run on KPIs such as claim turnaround or invoice-to-pay time. When a KPI slips, finding the cause takes analysts days of slicing data. A language model asked "why" will produce a fluent answer with invented numbers and plausible but wrong causes.

The approach. WatchDog keeps the model for what it is good at (forming hypotheses, reading evidence, writing) and leaves every calculation to code.

Who it is for. Operations and process-excellence leaders, and the consultants who serve them.

Why the results can be trusted.

• Synthetic data with planted root causes and decoys, so "did it find the real causes, and did it fall for the decoys?" is a measured number.
• Two different processes (insurance claims, procure-to-pay) on one engine, which shows the engine is horizontal.
• Governance in the backend: agents can propose and execute, never approve; every step lands in a tamper-evident audit log.
• Zero spend: free model tiers, local embeddings, local-first deployment.

Live demo

WatchDog runs publicly at intelligence-watchdog.duckdns.org, live since 12 November 2026, on a free Oracle Cloud server and Groq's free model tier.

• Browse without signing in. Visitors have read-only access (D58).
• Replay recorded investigations. Showcase runs are real investigations recorded ahead of time. Replaying one costs no model tokens.
• Run new investigations. Sign in with GitHub. Only accounts an admin created can sign in (D56, D61).

How it works

WatchDog closes the loop from a KPI alert to a verified fix. The investigation stage is where it differs from a dashboard or a chatbot.

• Watch. Monitors are configuration, with thresholds taken from the client's own policies ("critical if above 24 hours for two consecutive months"). New critical alerts are ranked and the top one is investigated automatically, within a daily budget.
• Investigate. A LangGraph workflow. Code scans the data for every significant pattern. The model turns patterns into hypotheses and picks checks from a fixed menu. Code runs every check, including confounder controls. The model judges each hypothesis, and rule overrides correct it in both directions. The executive summary's numbers are verified against the computed facts.
• Recommend. Each corrective action in the catalogue names the policy clause that permits it and a lever. A deterministic what-if on the event log values it (breaches avoided, illustrative annual ROI).
• Act. Propose, then approval by a person with the right business role, then execution through the Operations MCP server against the system of record. Agents can never approve, and the proposer cannot approve their own request.

Around the loop: the platform an enterprise would ask for

Two processes, one engine

Both processes run on the same engine, each defined by a "domain pack" of configuration: 18,000 cases per pack over 18 months (April 2025 to September 2026), five planted root causes, and decoys that look like causes but are explained by something else. Both companies are fictional.

Insurance claims · Kestrel General Insurance

Motor and health reimbursement claims, a five-day turnaround SLA, 20 policy and SOP documents, and a post-closure customer survey. Breach rate 26%.

Procure-to-pay · Varaha Auto Components

Supplier invoices at a five-plant manufacturer, a ten-day invoice-to-pay target, 13 documents. Breach rate 34%.

The data also carries realistic seasonality (monsoon motor surge, March fiscal year-end rush) and change points that match the dated policy documents. Every case draws its randomness up front and is re-simulated with each cause switched off, so "this breach happened because of RC1" is an exact counterfactual, not an estimate.

Results

Insurance claims · six-month data, 3 seeds × 3 questions per model

Incomplete runs were refused by Groq's free tier (quota or per-minute size cap), not model failures, and are reported separately.

Procure-to-pay (six-month data, local 7B, 9 investigations): 100% recall, 0 decoys, remedy recommended for 17 of 18 causes, policy answers grounded 100%, retrieval hit@5 100%.

Other evidence

• Policy Q&A: 96% to 100% of answers cite a relevant passage; citations are re-checked by a cross-encoder, not trusted.
• Retrieval on the larger 18-month corpus (hybrid search): the right passage is in the top five for 97% (insurance) and 100% (procure-to-pay) of benchmark questions.
• What-if estimates land at 0.5 to 0.99 of the exact counterfactual. The engine tends to understate, which is the safe direction.
• Unverified numbers in executive summaries: 0 across all completed runs.
• In one procure-to-pay evaluation the model called a confounded segment "supported" 25 times; the confounder rule refuted every one, so none reached the summary.

Limitations

• The data is synthetic and the policies author-written. The results show the system recovers planted patterns and resists planted decoys, not that it would on a messy real event log.
• Samples are small (three seeds, three to five questions per pack and model).
• Number verification checks numbers, not claims: a small model can attach a correct number to the wrong statement.
• The what-if engine understates some effects (calendar effects, mechanisms beyond waiting time).
• The full evaluation on the 18-month data, with five causes per pack, is not yet complete; the results above are mainly from the six-month data.

Architecture and stack

A modular monolith with separate processes for the watcher and three MCP servers, all on one PostgreSQL database and driven by per-process configuration. The investigation graph is the product. Everything else makes it trustworthy (governance, audit, observability) or affordable (local embeddings, free model tiers, caching). A new process is a configuration pack, not new code.

Deployment

The plan was Oracle's free 4-core ARM machine. It was out of capacity, so the demo runs on the free 1 GB AMD micro instance with 4 GB of swap (D60). The running stack uses about 0.5 GB of RAM, so responses are slower while parts of it page in from swap.

• One public entry point. Caddy gets and renews the HTTPS certificate, adds security headers, serves the built frontend and forwards /api/* to the backend without buffering, so investigation events stream live. Postgres, the backend, the Operations MCP server and the mock enterprise API have no published ports.
• Builds happen off the server. Images are built on a laptop with the same CPU architecture and streamed to the server over SSH. The first transfer took under three minutes. Building on a one-eighth-core machine would take about an hour or run out of memory.
• Data is regenerated, not copied. The generators are deterministic, so the server rebuilds the identical 18-month datasets from seed 42 and ingests the 13 policy documents.
• Replays are checked. A showcase run loads only if a fingerprint of the server's data matches the data it was recorded on, so a replay cannot quote numbers that are not there. Runs are recorded locally on the free model (D59).
• Free-tier limits. Groq allows 8,000 tokens a minute and 200,000 a day per model (D62). Calls run one at a time with up to eight retries that wait as long as the provider asks. A test investigation hit 21 per-minute refusals and still finished in 176 seconds. The app allows 4 fresh investigations per person per hour and 12 a day overall.
• Security. Only ports 22, 80 and 443 are open. Sessions use a Secure, HttpOnly cookie with a hashed token and a 12-hour sliding expiry. GitHub sign-in asks for identity scopes only, uses the token for two API calls and discards it, and a state cookie blocks login CSRF.
• Resilience. Every service restarts automatically after a crash or reboot. Nightly database dumps keep the newest 14.

Known limits: about ten fresh investigations a day, slower responses than the planned ARM shape, and Oracle can reclaim free instances it considers idle. The hourly watcher keeps the server lightly busy.

Key design decisions

The project's credibility rests on a handful of decisions. The full decision log is kept in the repository.

Build story

Nine milestones, each demoable on its own.

1. M0–M3 · Foundations
   Repo, Docker Compose, data generator, metrics layer, process mining, document retrieval with citations.
2. M4 · Investigate
   LangGraph investigation (plan, evidence, judge, recommend, summarise), what-if ROI.
3. M5 · Act
   Operations MCP server, approvals, audit log, mock enterprise API.
4. M6 · Watch
   KPI monitors with policy thresholds, alerts, alert to investigation, follow-up metrics.
5. M7 · Evaluation
   Ground-truth benchmark, retrieval metrics, multi-model comparison, evaluation report.
6. M8 · Second pack
   Procure-to-pay on the same engine; case study and demo script.
7. M8 · Platform
   Accounts and two-layer RBAC, built-in observability, policy-answer cache, light model tier, model comparison page; then investigation cache and showcase replay, rate limits, Analytics and Knowledge MCP servers, exploratory SQL, automatic alert triage.
8. M8 · Richer data
   18 months per pack, seasonality, change points, a fifth cause and a new decoy per pack, 10 new documents.
9. M8 · Deployment
   Public HTTPS deployment on a free Oracle instance, GitHub sign-in, showcase runs re-recorded on the new data, free-tier pacing for the cloud models.

Problems found and fixed along the way

• Scoped investigations were blind to their own cause. Asking "why are broker claims breaching?" compared brokers only with brokers. Fix: scoped checks also compare with the rest of the business.
• Tool calling failed on two cloud models. JSON-schema output succeeded 3/3 where tool calls did not; it became the default, with a fallback.
• A slow model server stalled runs for minutes. Per-read timeouts reset with every token; every call now has a wall-clock deadline, and output tokens are capped.
• Over-caution cost recall. The 120B model left real causes inconclusive (79% recall). Code-worded upgrades with strict evidence criteria took it to 100% without ever accepting a decoy.
• The second pack exposed hidden assumptions. Most of the work of adding procure-to-pay was moving process constants out of code into configuration.
• Observability found bugs on day one. Parallel assessment lost its context; cache hits were never recorded.
• Automatic triage, run live, failed honestly. The watcher container could not reach the models; the alert recorded "failed" with the reason, and the fix took one compose change.
• Richer data stressed the engine. A queue's month-by-month signal was blurred by another change, decoy signals crowded out a real cause, and a hypothesis with no significant effect stayed "supported". Three generic rule fixes (D54) resolved all three.
• The free ARM server was not available. Oracle's free Ampere capacity is shared and often exhausted. The demo moved to the 1 GB micro and the build moved off the server (D60).
• Port 80 timed out. The subnet's security list had no ingress rule for 80 and 443. A timeout means a silently dropped packet, while "No route to host" means the server's own firewall refused it, so the two point at different layers.
• A healthy app failed its health check. Six containers booting on an eighth of a core pushed the check past its 3-second timeout, and Caddy waits for a healthy backend. A 20-second timeout and a 3-minute start period fixed it.
• Parallel model calls hit the per-minute token limit. Four simultaneous calls used a minute's allowance at once. One call at a time with more retries finishes reliably (D62).
• A day's token quota was spent before launch. Re-recording showcase runs on Groq burned it. They are now recorded locally, as D59 intended.
• GitHub sign-in looked broken. The flow worked, but the accounts table was empty until the seed ran, and the app refuses unknown GitHub users by design.

Roadmap

The remaining work turns the live demo into a finished showpiece.

1. OpenComplete the 18-month evaluation: local 7B model on both packs, three seeds, then rebuild the evaluation report.
2. DoneRe-record showcase runs on the new data, so visitors can replay investigations without spending model quota.
3. DonePublic deployment: HTTPS, production settings, a cloud model protected by the rate limits, and showcase runs shipped with it.
4. DoneGitHub sign-in for visitors, with read-only access by default.
5. OpenRe-run the cloud models on the 18-month data, paced for free-tier limits.
6. OpenDemo video (about 4½ minutes) recorded against the live URL.

Deliberately out of scope: multi-tenancy, Kubernetes, streaming platforms, a third process pack.

Glossary