We repurposed coding agents like Claude Code and Codex for end-to-end detection validation. A custom plugin teaches the agent to write TTPForge attack simulations, ship them to a lab host over SSH, detonate them, and query Splunk to confirm that telemetry arrived and the detection fired. When a step fails, the agent rewrites and retries until the technique lands or it runs out of options. We ran the loop across a sample of our custom detections (mostly Windows, some Linux) and reviewed every TTP by hand. Most TTPs were fit for the detection they targeted. More usefully, the loop surfaced small, mechanical issues across the detection pipeline that had slipped past code review, without making a human drive each validation run manually.

The rest of this post walks through how we built the loop: the TTP framework, the agent plugin, the Splunk knowledge base, the verification workflow, and the isolation needed to let an agent run attack simulations unattended.

Detection validation

Validating a detection means different things depending on how much of the system you put under test. Regression tests replay labelled events against the rule in isolation. Synthetic log injection extends the coverage to ingestion and parsing. Regression tests are fast and fit naturally in CI. Synthetic injection is heavier, because you need somewhere to inject events into the pipeline, but it can still reach parsing and detection logic without detonating anything on an endpoint.

Running real techniques on real endpoints, and letting the logs flow through the production pipeline, puts the full path under test: endpoint configuration, sensor behaviour, ingestion, the SIEM, scheduled detections, and the response system that picks up the alert.

This post is about the last one: testing the detection as close to end to end as we can, by running the behaviour on a real endpoint and following the evidence all the way to the alert.

Lab environment and TTP framework

Before the agent comes in, two prerequisites have to be in place: somewhere safe to detonate attacks that mirrors production closely enough for the signal you are testing, and a way to describe those attacks step by step.

For endpoint detections, that means Windows and Linux VMs built from the same golden images as production, with the same EDR configuration, policies, and log forwarding. Our lab can provision ad-hoc, personal environments with multiple Windows and Linux clients joined to a domain-controlled setup, so a validation run can get its own hosts instead of sharing a long-lived machine. For cloud detections, the same idea applies to the subscription or account where you detonate: an Azure subscription should have the same policy assignments and logging configuration as a normal production subscription, and an AWS account should sit under the same organization controls and forwarding path as production accounts. The ingestion path has to match, because a detection passing in the lab only tells you something if the data reached it the same way real data does. The lab build is the largest upfront investment in the whole effort, but we won’t cover it here. Credit for that work belongs to my former teammate Øystein Skonseng Landsverk, who built the lab infrastructure and automated provisioning this work depends on.

The goal is not only ad-hoc validation. In the CDC, a GitHub Actions workflow runs the reviewed TTP catalogue on a schedule and records which TTPs launched cleanly, which failed on the endpoint, and whether the detection arrived in the downstream system after the telemetry reached Splunk. The results feed an Observable Framework dashboard in the office landscape, giving the team an at-a-glance view of whether the detection pipelines are still healthy or something drifted overnight.

For the attack framework we picked TTPForge (Meta). Its YAML schema has first-class support for reusable building blocks, remote steps, tests, checks, cleanup, and arguments. Tests define safe automated run configurations, often with specific argument values. Per-step checks let the agent verify that each action actually worked before moving on. The same signals make scheduled runs diagnosable: when a TTP fails on the endpoint, you can see which step broke instead of treating the whole run as a black box. We looked at the other mature options as well. Any of them would have been workable. TTPForge just matched the shape of this project best.

A TTP is a YAML file with a sequence of steps. Three features make it suitable for repeatable automation rather than a one-shot manual run:

• Cleanup undoes what each successful step did, in reverse. That makes repeated runs practical without accumulating stale artifacts on the lab host.
• Checks run per step and verify that the action actually worked, not just that the command exited 0. Did the scheduled task get created? Did the DLL load? That feedback is useful both while the agent is developing the TTP and later when a scheduled run fails.
• Args parameterize the YAML through Go templates, so one TTP can cover different hosts, paths, payload names, or variants without being copied.

A complete example, kept simple for the shape rather than realism: a scheduled task TTP that creates a task with schtasks.exe.

---
api_version: 2.0
uuid: a3e8c1f2-5d47-4b9e-8c36-7f2a9b0e4d51
name: new-scheduled-task
description: |
  Creates a scheduled task with schtasks.exe /Create. MDE logs this as a
  ScheduledTaskCreated ActionType event in DeviceEvents.

mitre:
  tactics:
    - "TA0003 Persistence"
  techniques:
    - "T1053 Scheduled Task/Job"
  subtechniques:
    - "T1053.005 Scheduled Task"

requirements:
  platforms:
    - os: windows

args:
  - name: task_name
    type: string
    default: TTPForgeValidation

steps:
  - name: create_scheduled_task
    executor: cmd.exe
    inline: |
      schtasks /Create ^
        /TN "{{ .Args.task_name }}" ^
        /TR "cmd.exe /c echo validation" ^
        /SC ONCE /ST 23:59 /F
    checks:
      - msg: "schtasks /Create did not report SUCCESS"
        output_contains: "SUCCESS"
    cleanup:
      executor: cmd.exe
      inline: |
        schtasks /Delete /TN "{{ .Args.task_name }}" /F >nul 2>&1

TTPForge also supports remote execution over SSH, but each step opens a fresh connection with a new working directory. We found it simpler to copy the TTP to the lab host and invoke ttpforge run there inside a single SSH session.

The agent plugin

Writing TTPs by hand does not scale. Each one needs to be realistic enough to produce the right telemetry, safe enough to clean up, and structured enough to run repeatedly. We instructed the agent to do that work instead. The plumbing is a Claude Code plugin: a directory of skills (the markdown workflows the agent follows), hooks (shell scripts that fire on lifecycle events), and reference material that gives the agent the TTPForge, telemetry, and detection-engineering context it needs. Plain markdown and shell scripts all the way down.

Skills are markdown workflows the agent loads when a task matches their description. The detectionkit plugin has two: validation (the TTP lifecycle) and detection (working with SPL and telemetry). Each skill’s SKILL.md is a dispatcher: its frontmatter describes when the agent should load the skill at all, and the body routes the agent to the right workflow inside it. So “write a TTP for this detection” loads the validation skill and sends the agent into workflows/create.md. “Run it on the lab” sends it into workflows/remote-run.md. Skills load progressively, so only a one-line description sits in context until one is invoked. You can keep dozens around without bloating the prompt.

Hooks are the guardrails. They fire on lifecycle events (file written, tool used), and the agent cannot choose to skip them. Prompting “always validate after writing” costs tokens on every iteration and the model can sidestep it anyway. A hook runs outside the reasoning loop, so a failed validation blocks progress and the agent has to fix the problem before it can proceed.

One practical lesson from debugging plugins and skills: the description field matters more than it looks. It is the routing hint the agent sees before the full skill is loaded, so vague descriptions either fail to trigger or pull the skill into unrelated work.

---
name: validation
description: "Authors, runs, and reviews TTPForge attack-simulation YAML
  against an owned validation lab. Use when writing or generating a TTP,
  deploying/running a TTP on a lab VM, verifying telemetry reaches Splunk,
  or reviewing an existing TTP YAML..."
allowed-tools: Bash, Read, Write, Agent
---

# TTP Validation

**Create** - User wants a new TTP from a detection rule or attack description.
-> Read `workflows/create.md`

**Run / verify** - A TTP exists and the user wants it run on the lab.
-> Read `workflows/remote-run.md`

**Review** - User hands over an existing TTP YAML for critique.
-> Read `workflows/review-ttp.md`

## Rationalizations to Reject

- "The detection only fires on the command line, so the payload doesn't
  matter" - Wrong. A TTP that runs `rundll32.exe javascript:...` with no
  actual payload is not a real attack. It generates the right command-line
  telemetry but proves nothing about whether the technique works in this
  environment. Always make the technique DO something (drop a file, load a
  DLL, execute code) so checks can verify the outcome.

- "I'll check with an echo sentinel - it proves the step ran" - No.
  `echo DONE` followed by `output_contains: "DONE"` proves the shell
  continued executing. It doesn't prove the attack worked. Check for the
  real artifact: the file the injected code dropped, the registry key the
  exploit wrote, the process the persistence mechanism spawned.

- "The technique is close enough" - T1059 when T1059.001 applies is a
  miss. Vague mappings hide coverage gaps and mislead triage. Always use
  the most specific subtechnique that genuinely applies.

{
  "hooks": {
    "PostToolUse": [
      {
        "matcher": "Write|Edit",
        "hooks": [
          {
            "type": "command",
            "command": "${CLAUDE_PLUGIN_ROOT}/scripts/validate-ttp.sh",
            "timeout": 30
          }
        ]
      }
    ]
  }
}

#!/usr/bin/env bash
set -euo pipefail

FILE=$(jq -r '.tool_input.file_path // .tool_input.path // empty' 2>/dev/null || true)

# Only validate .yaml files under ttps/
[[ "$FILE" == *.yaml ]] || exit 0
[[ "$FILE" == *ttps/* ]] || exit 0

# Find ttpforge binary
TTPFORGE="${CLAUDE_PROJECT_DIR:-}/ttpforge"
[[ -x "$TTPFORGE" ]] || TTPFORGE=$(command -v ttpforge 2>/dev/null || true)
[[ -z "$TTPFORGE" || ! -x "$TTPFORGE" ]] && exit 0

# Validate - exit 2 blocks the write
OUTPUT=$("$TTPFORGE" validate "$FILE" 2>&1) || { echo "$OUTPUT" >&2; exit 2; }
exit 0

## Executor Choice (Windows)

Use `executor: cmd.exe` for tools that are simple command-line invocations:
net.exe, netsh.exe, sc.exe, schtasks.exe, bitsadmin.exe, wevtutil.exe,
reg.exe, attrib.exe. Use `executor: powershell` when you need PowerShell
capabilities: Set-MpPreference, New-ItemProperty, Invoke-WebRequest,
try/catch, loops, objects.

## When to Use Non-Inline Action Types

`create_file` - Use when a step's sole purpose is writing a file to disk.
`cleanup: default` auto-removes the file. Parent directories created
automatically.

`edit_file` - Use when appending to or replacing in an existing config file.
`backup_file` + `cleanup: default` = automatic restore. Pattern-not-found
is a hard error (not silent like sed).

`fetch_uri` - Use for payload staging from HTTP/HTTPS URLs. Native download
with `cleanup: default`.

Those pieces turn TTP authoring into a feedback loop rather than a single generation step.

This is a good fit for coding agents because the task gives them something to check. The YAML validates or it does not. The TTP runs or it does not. The step checks pass or they fail. Most importantly, telemetry arrives and the detection matches, or it does not. The agent can use those outcomes to revise its own work instead of relying only on a plausible-looking first draft.

A single iteration looks like this:

1. Agent writes or edits the TTP YAML
2. PostToolUse hook validates it automatically
3. Agent deploys to the lab host over SSH
4. TTPForge runs and reports check results
5. On a failed check, the agent diagnoses and loops back to step 1

The reference material pushes the agent toward a defanged payload: real enough to exercise the attack path and generate the telemetry the detection looks for, harmless enough that cleanup can fully reverse it. A marker file on disk, a registry key set to a known value, a process spawned with a known argument. TTPForge’s checks confirm the technique actually worked, not just that the command exited cleanly. That tells us the attack ran on the host. The next question is whether the telemetry made it through the SIEM pipeline and matched the rule.

The knowledge base

To answer that, we built a second plugin that wraps Splunk’s REST API. The wrapper is the easy part. The useful part is the knowledge base behind it: one markdown file per sourcetype, documenting field names, value formats, known gaps, and query templates.

When the agent starts from an existing SPL detection, most of the fields and predicates are already in front of it. In that mode the knowledge base is mostly a sanity check and a debugging aid: does this field exist in this sourcetype, what hostname form should we query, is this category known to arrive late or not at all?

The knowledge base matters more when the agent is exploring telemetry or drafting a new detection from a TTP. Splunk returns empty results with no error when you query a nonexistent field, an empty time range, or a hostname in the wrong format. Without knowing what to expect, the agent cannot tell a bad query from a real pipeline failure. The rule the agent follows is simple: read the schema for the target data source before writing exploratory SPL.

At runtime, the agent walks a progressive loading sequence:

That is the runtime path: route to the right schema, read the field notes, then write or debug the SPL.

Building those schema files is a separate workflow. Splunk’s fieldsummary gives you field names and cardinality, but it can’t tell you that DeviceName stores FQDN on domain-joined machines and short name on WORKGROUP, or that SHA256 is usually empty and you should join on SHA1 instead. That kind of context lives in vendor documentation, so each schema file is built in two passes:

The first pass, profile-index, runs fieldsummary and learns field names and cardinality, but not what a field means or which values quietly break a query. The second pass, enrich-schema, fills the rest in from vendor docs (Microsoft Learn, xdrinternals.com, AWS documentation) and writes it between <!-- EXTERNAL --> markers. Re-profiling regenerates everything above the markers and leaves the enriched section untouched.

Two schema-file excerpts to show the shape, the routing table at the top and the leaf schema underneath:

# Splunk Index Catalog

Known indexes and what they contain.
Full field profiles are in schemas/.

| Data | Index | Sourcetype | Search by |
|---|---|---|---|
| MDE endpoint (process, file, net) | azure | mde:advancedhunting | TERM(<hostname>) |
| Azure AD sign-ins | azure | azure:aad:signin | TERM(<upn>) |
| Windows security events | winevent_sec | WinEventLog | TERM(<user>) |
| Network / Zeek | corelight | corelight_conn | TERM(<ip>) |
| Firewall | checkpoint | cp_log | TERM(<ip>) |
| Web proxy | zscaler | zscalernss-web | TERM(<upn>) |
| AWS CloudTrail | aws_* | aws:cloudtrail | TERM(<action>) |
| Email filtering | proofpoint | pps_messagelog | TERM(<upn>) |
| M365 activity | o365 | o365:management | TERM(<user>) |
| ... | ... | ... | ... |

# Index: azure / Sourcetype: mscs:azure:eventhub:defender:advancedhunting

## What this data is
Device table family from Microsoft Defender for Endpoint.
Events cover process creation, file events, network connections,
registry changes, and scheduled task operations. All events
share one sourcetype, differentiated by the category field
(stored with an `AdvancedHunting-` prefix).

## Key fields

| Field | Notes |
|-------|-------|
| category | MDE table name: DeviceProcessEvents, DeviceRegistryEvents, etc. |
| properties.DeviceName | FQDN on domain-joined, short on WORKGROUP. Discover stored form first. |
| properties.ProcessCommandLine | Full command line of spawned process |
| properties.InitiatingProcessFileName | Parent process name |
| properties.SHA256 | Hash of process binary (usually empty - join on SHA1 instead) |
| properties.AccountUpn | UPN of the acting user |

## Categories

| Category | Description |
|----------|-------------|
| DeviceProcessEvents | Process creation with full command lines |
| DeviceNetworkEvents | Outbound/inbound connections |
| DeviceFileEvents | File create/modify/delete |
| DeviceRegistryEvents | Registry modifications |
| DeviceImageLoadEvents | DLL loads (selective, not exhaustive) |
| DeviceEvents | Security controls, ASR, scheduled tasks |

The same idea has shown up elsewhere. Anthropic described the pattern for data analytics, where per-table markdown documentation gets loaded before the model writes a query. Google’s Open Knowledge Format standardizes the same shape with YAML frontmatter. OKF is what we would adopt if we rebuilt the knowledge base today.

The schema files are living documents, not generated once and forgotten. Parsers change, fields get renamed, and new event types appear. When that happens, the schema needs to be updated before the agent mistakes stale documentation for a detection failure.

Plugin composition

The detectionkit plugin does not query Splunk itself. Its remote-run workflow carries one line about the other plugin (“Telemetry: verification requires the splunk plugin (provides splunk:search)”) and that is the entire integration. The two plugins share no code. The primary session keeps the orchestration small and spawns short-lived sub-agents for anything slow or risky: a verification agent that polls Splunk, and a debug-detection agent that takes over when verification times out.

Verification runs only when the user signals it: “did the detection fire,” “end-to-end,” “verify.” A plain “run it” stops once the TTP has executed. When verification is on, the remote-run workflow adds one extra step, written directly into the markdown:

5. Verify telemetry. Spawn a background agent that invokes splunk:search
   (data dictionary -> schema -> query), polls every 2 minutes for raw
   events plus detection match, hard timeout 10 minutes. On timeout,
   hand off to debug-detection.

That single line is the entire integration between the two plugins.

The verification loop

Verification is not a single query. The agent has to wait for telemetry to land, confirm it arrived, and only then check whether the detection matched. We run it as a background agent with a polling loop and a timeout:

sequenceDiagram
  participant V as Validation Agent
  participant S as Verification Agent
  participant Q as Splunk
  participant D as Debug Agent

  V->>S: spawn (hostname, timestamp, indicators)
  loop Every 2 min (max 10 min)
      S->>Q: raw events present?
      Q-->>S: events / empty
  end
  alt Events found
      S->>Q: detection SPL matches?
      Q-->>S: results / empty
      S-->>V: FOUND
  else 10-min timeout
      S-->>V: NOT FOUND
      V->>D: spawn debug-detection
      D->>Q: diagnostic queries
      D-->>V: diagnosis + recommendation
  end

The first query looks for raw telemetry from the host, using TERM() bloom-filter tokens to skip irrelevant buckets. If nothing arrives, the agent waits two minutes and retries. In this setup, MDE events usually land in Splunk in under four minutes. Once raw events appear, the same window is re-queried with the detection’s SPL. Both queries returning rows is the success condition: technique landed, telemetry arrived, detection matched.

When the 10-minute window expires without events, the workflow hands off to debug-detection, a diagnose-only agent that cannot edit TTPs or re-run anything. It walks four failure modes and returns a verdict:

• The host was silent. The TTP exited cleanly but nothing arrived in the SIEM. The agent first checks whether the host is forwarding any telemetry at all in the window. If it is silent, the lab build is the problem (a stuck agent, a forwarder that needs restarting), not the TTP.

• The right event never arrived. The host is forwarding, but the specific event the detection keys on is missing. The agent compares what categories arrived against what the rule expects. The common verdicts here are MDE telemetry caps and filtering (FalconForce has a good write-up) or a category that exists in the EDR’s timeline view but never reaches the hunting tables.

• Events arrived too late. The right events came through, but after the verification window had closed. The agent measures ingestion lag against the detonation timestamp. If the p95 is well above the polling window, the technique was fine and the window was too tight. The fix is patience, not a rewrite.

• The rule rejected a matching event. Events arrived on time, the right category is present, but the detection returns nothing. The agent walks the rule predicate by predicate, looking for the clause that rejects the otherwise-matching event. The common verdicts are field-name mismatches (FQDN vs short hostname), the technique landing under an adjacent event category, or a preparatory step in the TTP diverting the technique to a different surface than the detection expected.

Worked example

The run in the video at the top of the post is a real validation against a Netsh helper DLL persistence detection (T1546.007). The rule watches for registry writes under HKLM\SOFTWARE\Microsoft\Netsh, where attackers register a malicious DLL that gets loaded whenever netsh runs. What follows is what the agent actually did, step by step. This worked example is scoped to the registry-write signal the rule keys on. The fuller DLL-loading chain appears later as a separate observation about agent behaviour, not as this validation run.

The agent starts by reading the rule. The relevant clauses:

index=azure TERM(AdvancedHunting-DeviceRegistryEvents) TERM(Netsh)
sourcetype="mscs:azure:eventhub:defender:advancedhunting"
category="AdvancedHunting-DeviceRegistryEvents"
properties.ActionType IN ("RegistryValueSet", "RegistryKeyCreated")
(properties.RegistryKey="HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Netsh"
 OR properties.RegistryKey="HKEY_LOCAL_MACHINE\\SOFTWARE\\WOW6432Node\\Microsoft\\Netsh")
properties.InitiatingProcessFileName!="msiexec.exe"
properties.InitiatingProcessFileName!="TiWorker.exe"
properties.InitiatingProcessFileName!="DismHost.exe"
properties.InitiatingProcessFileName!="svchost.exe"
| rename properties.RegistryValueData AS registry_value
| eval suspicious_path=if(like(registry_value, "%\\Temp\\%")
    OR like(registry_value, "%\\AppData\\%")
    OR like(registry_value, "%\\Public\\%")
    OR like(registry_value, "%\\Downloads\\%"), "true", "false")

The full version of T1546.007 is a four-step kill chain: cross-compile a DLL, drop it on disk, register it in the Netsh key, run netsh to load it. The agent considered that path and rejected it. Its reasoning, captured in the transcript: the rule fires on the registry write event, not on the DLL load. The full kill chain validates the technique end to end, but adds nothing to validating this detection. A single reg.exe invocation writing a value pointing at a fake DLL path hits the exact event the rule keys on. Using the path C:\Users\Public\ttpforge_netsh_helper.dll also exercises the suspicious_path flag because of the \Public\ substring.

This is what the reference material is for: the agent has read enough about the rule and the framework to know which corners it can cut without invalidating the test.

A single step, a reg add writing the fake DLL path:

api_version: 2.0
uuid: 2da9e404-1d0d-477f-b9f5-ecd64b6fefe5
name: win-netsh-regmod
description: |
  Writes a Netsh helper registry value pointing at a fake DLL. This is the
  registry-write half of T1546.007 persistence. The detection fires on the
  RegistryValueSet event regardless of whether the DLL actually exists.
mitre:
  tactics:
    - "TA0003 Persistence"
  techniques:
    - "T1546 Event Triggered Execution"
  subtechniques:
    - "T1546.007 Netsh Helper DLL"
requirements:
  platforms:
    - os: windows
  superuser: true
detections:
  - MDE_NetshRegmod
steps:
  - name: register_netsh_helper_dll
    executor: cmd.exe
    inline: |
      reg add "HKLM\SOFTWARE\Microsoft\Netsh" /v ttpforge_helper /t REG_SZ ^
        /d "C:\Users\Public\ttpforge_netsh_helper.dll" /f
    checks:
      - msg: "reg add must report success - Defender may strip the key
              afterward but the RegistryValueSet event is already emitted"
        output_contains: "completed successfully"
    cleanup:
      executor: cmd.exe
      inline: |
        reg delete "HKLM\SOFTWARE\Microsoft\Netsh" /v ttpforge_helper /f >nul 2>&1
        exit /b 0

The agent ships the TTP to the lab host and runs it under TTPForge. Both Defender and the EDR react. On the first attempt, the TTP writes the registry value successfully, but the follow-up reg query check fails because Defender has already stripped the value. The agent changes the check to validate the reg add output instead (completed successfully), because the RegistryValueSet event is emitted at write time. That is a weaker local check, so the downstream verification matters: the run is only useful if MDE actually records the write and the detection matches it. The second attempt passes, cleanup runs without error, and the agent records the post-execution timestamp before spawning the verification agent.

The verification agent first discovers the device-name format for this host. MDE stores lab-dc-3783 as a short hostname because the lab is a WORKGROUP machine, unlike domain-joined hosts that show up as hostname.cdclab.com. With the right TERM(lab-dc-3783) filter in hand, it queries Splunk for raw DeviceRegistryEvents in the detonation window.

Ten events come back: five RegistryValueSet and five RegistryValueDeleted (the write and Defender’s cleanup). The right category arrived. The agent then re-runs the detection’s full SPL against the same window. Five matches, all on the Netsh key, all with suspicious_path=true, all from reg.exe, none of them in the exclusion list. The RegistryValueDeleted events are correctly filtered out by the rule’s ActionType IN clause.

The verdict the agent writes back: telemetry arrived after about a three-minute ingestion lag, the detection matched on five events, end-to-end confirmed. One incidental discovery from this run, captured in the schema knowledge base afterwards: MDE stores the key with mixed case (NetSh) while the SPL queries it as Netsh. Splunk field matching is case-insensitive so it worked, but it is the kind of trap that future debugging sessions need to know about.

Isolation and execution

The agent writes attack code, runs shell commands, installs cross-compilers, and SSHes into lab hosts. None of that should run directly on the laptop it launches from. For now we run the whole thing locally on a Mac. The agent runs inside a devcontainer based on Trail of Bits’ setup. The devcontainer runs in Docker, and Docker is hosted by a colima Linux VM. Inside the container the agent has broad permissions. Outside it, access is limited to mounted project files and the network paths we explicitly allow.

The mounted volume survives across runs, so a session can pick up where the previous one left off. The devcontainer and VM keep the blast radius local to the mounted files and explicitly allowed network paths. Workflow-specific scope boundaries still live mostly in plugin instructions and review, as the forwarder incident below shows.

Results and observations

This was a limited round of experimentation, not a full audit. We ran the loop across a broad sample of custom SPL detections: mostly Windows, plus a smaller set of Linux rules. The findings below are selected examples from that process. Some are ordinary detection or pipeline issues. Others are issues with the agent workflow itself.

Detection, SPL, and pipeline findings

A separate static-analysis pass over the rules surfaced more of the same flavour. A missing equals sign that had turned a field comparison into an unanchored full-text search on _raw. An index reference left as a literal placeholder. A typo that narrowed coverage to attackers who happened to make the same typo. A category filter copy-pasted from a different sourcetype. The kind of thing that survives code review because the query still parses. Running the attack is what makes it visible.

Agent workflow findings

The agent also surfaced issues in the validation approach itself: places where the TTP changed the signal being tested, where SSH could not represent the right user session, or where the agent tried to fix infrastructure instead of reporting the gap.

Two runs from this round illustrate the scope question especially clearly.

Clever: netsh helper DLL Too far: forwarder fix

Asked to validate a netsh helper DLL persistence detection, the agent reasoned through the full chain (registry persistence, netsh loads DLL, code executes, ImageLoad event fires), proposed exporting InitHelperDll, added DNS resolution to generate network telemetry, and set up Windows cross-compilation from macOS.

This screenshot is from an earlier run, not the worked example from the video. The worked example validates the registry-write detection with reg add and does not compile a DLL. This earlier run is included because it shows an interesting behaviour: the agent started from the same kind of detection and reasoned toward the fuller Netsh helper DLL implementation.

The equivalent defanged payload is small: a DLL that exports InitHelperDll and, when loaded by netsh.exe, writes a marker file or makes a harmless DNS lookup. The full-chain version would need a compiled DLL or a prebuilt defanged binary. The registry-only validation used later in this post does not. The agent-driven part is the end-to-end setup: write or stage the payload, compile it if needed, copy the DLL into place, register it under the Netsh helper key, run netsh, and check that the marker appeared. Same target, different path through the environment: nondeterminism in action.

Three payload options, the correct registry key and entry point, and a cross-compilation setup. The useful part was the reasoning: the agent moved from a registry-only trigger toward the full Netsh helper DLL chain.

The DLL work is the kind of reasoning we wanted. The forwarder fix is where we drew the line: the agent can install tools and disable Defender on the lab host, but it should not touch logging, forwarding, or SIEM configuration. The same boundary is why debug-detection is diagnose-only, and why a separate review agent inspects the lab after each TTP run.

Summary, limitations, and next steps

We built a closed-loop pipeline for end-to-end detection validation from two composable plugins. The detectionkit plugin owns the TTP lifecycle: YAML authoring, schema validation, lab deployment, and remote execution over SSH. The splunk plugin owns SIEM access and is backed by a per-sourcetype schema knowledge base that documents which fields are reliable for each event type. The plugins share no code. The validation workflow invokes the splunk plugin through a single skill reference when it needs telemetry verification.

Two pieces of infrastructure make the loop practical to run unattended. The local devcontainer and Colima VM limit the agent to mounted project files and explicitly allowed network paths, which makes it acceptable to give it cross-compilers, SSH sessions, and broad bash permissions inside the container. The schema knowledge base keeps Splunk queries grounded in parser-level field behaviour that official documentation does not capture.

Every stage produces a checkable signal. The technique executed or it didn’t. The telemetry arrived or it didn’t. The detection matched or it didn’t. A failure localizes to one stage, so the agent knows whether to rewrite the TTP, wait longer, or hand off to a human. The small issues the loop surfaced were of the kind that survive static review but struggle to survive a live detonation. That is the operational value of the loop: it turns detection validation from a manual exercise into a repeatable way to find drift, parser changes, and operational breaks in the path from endpoint to alert.

This is still an early version of the workflow. The schema knowledge base has to keep up with parser changes, or the agent will mistake stale documentation for a detection failure. Scope is enforced mostly through plugin instructions and review rather than hard technical guardrails, and command-level hooks are difficult to make strict without blocking legitimate validation work. The worked example also shows a subtler failure mode: the agent can satisfy a local check without proving the richer technique. For the Netsh rule that was acceptable because the rule fires on the registry write, but a stronger T1546.007 TTP would compile or stage a helper DLL, run netsh, and verify a marker from InitHelperDll. Interactive development runs are useful exactly because they expose those shortcuts while they are still cheap to correct, and because they help the analyst understand the attack and the telemetry it produces. The traces and reasoning logs then become input to the next plugin iteration, where bad behaviour turns into review checks or rationalizations to reject. We also experimented with Raindrop Workshop for this kind of meta-loop: trace a Claude Code session, have another agent inspect the trace, change the skill files, rerun, and compare what changed. We only used the local Workshop setup and ran into some early-release quirks, but the workflow was useful.

The same setup hints at workflows we have only started to explore. The post so far has described an agent that validates an existing detection against an existing technique, but the same plumbing covers two more modes that run the loop forward instead of backward. The first is test-initiated detection engineering (TIDE), where the agent writes the TTP first, runs it, then drafts the detection against the telemetry that actually arrives. The attack becomes the test, and the detection has to pass it. The second is telemetry exploration, which answers “what shows up when I do X?” by running the technique and looking, instead of guessing which fields are populated or what format they take.

A related direction is more exploratory detection engineering: automated threat hunting from threat intelligence, tailored to your own environment. Given a vendor report, a CVE writeup, or a short prose description of an attack, the agent can spin up the behaviour in an isolated lab, record the execution window, and use the knowledge base to explore the data sources that should have seen it. The output does not have to be a finished detection. It can be analyst-facing intel: which events appeared, which fields were useful, which sources were silent, and which searches are worth trying against the wider dataset. From there the agent can draft candidate SPL, hunting queries, or detection logic and fire them off quickly to see whether anything similar is already present. That still needs a human in the loop, but the lab and the agent workflows make the early exploration much faster.

References and related work

• TTPForge - Meta’s attack-emulation framework, the YAML schema and runtime used throughout this post.
• Atomic Red Team, Stratus Red Team, Cortado - the framework alternatives compared in the post.
• Test-Initiated Detection Engineering (TIDE) - the methodology referenced in the closing section.
• How Anthropic enables self-service data analytics with Claude - the per-table schema pattern adapted here for SIEM sourcetypes.
• Open Knowledge Format - Google’s portable schema spec, the shape we would adopt if we rebuilt the knowledge base today.
• Raindrop Workshop - local tracing and agent-eval tooling we experimented with for debugging Claude Code sessions and plugin changes.
• Claude Code plugins and hooks - the plugin and hook systems the detectionkit and splunk plugins are built on.

• #detection-engineering
• #detection-validation
• #ttpforge
• #agents
• #claude-code
• #splunk