LLMs in SAST: Good, Bad, Costly

Static Application Security Testing (SAST) has been a fixture of the secure-development toolchain for two decades. The mechanics are well understood: parse source into an intermediate representation, build a control- and data-flow graph, and match it against rules that encode known vulnerability patterns. Fast, deterministic, runs without executing the code.

It also has a chronic flaw — noise. Depending on the tool and codebase, the false-positive rate of traditional SAST commonly lands between 30% and 100%. Security engineers burn most of their time triaging findings that turn out to be unreachable, already sanitized, or simply wrong. The result is alert fatigue, ignored scanners, and real vulnerabilities lost in the backlog.

Large language models change the economics of this — but not evenly. They are genuinely transformative in some places, dangerous in others, and quietly expensive in ways the vendor demos don’t mention. This post sorts the reality into three buckets: the good, the bad, and the costly.

A note on framing before we start: the systems that work do not ask an LLM to find vulnerabilities from scratch. They run a deterministic engine first, then use the model to reason about its output. That ordering is the thread running through everything below.

The Good

LLMs are excellent at the triage problem — the most expensive part of SAST

The single highest-ROI use of an LLM in security is not finding bugs. It’s reading a SAST finding plus its code context and deciding whether it’s real. This is a bounded reasoning task — “is this specific flagged path actually exploitable?” — and it’s exactly what models are good at, because triage is fundamentally a code-reading-and-intent problem.

An LLM can recognize that a “vulnerable” string concatenation is actually safe because the framework parameterizes it, or that a tainted data-flow path is dead because an auth check upstream blocks it. Pattern-matching rules have no model of intent; they see structure and fire. The model has an implicit model of what the code is trying to do.

The production numbers are striking. Semgrep reports that its Assistant feature filters roughly one in five SAST findings as a false positive safe to ignore, agrees with its own security researchers on true-positive triage decisions about 96% of the time, and now handles around 60% of incoming triage work for customers. Independent academic work goes further: one OWASP-benchmark study found LLM-based post-filtering could cut a tool’s false-positive rate from over 92% down to roughly 6%.

LLMs turn findings into something a developer can act on

A raw SAST finding is a file, a line number, and a rule ID. What a developer needs is the data flow narrative, why it’s exploitable, the blast radius, and whether it matters for this deployment. LLMs are good at generating that explanation and at reasoning about severity in context — distinguishing a reachable, externally-controlled sink from a theoretical one buried behind layers of internal-only code. This is what moves a tool from “here are 4,000 findings” to “here are the 40 that matter this sprint.”

The hybrid architecture genuinely cancels each side’s weaknesses

The reason this all works is the SAST-first, LLM-second pipeline:

Source code
   │
   ▼
[Deterministic SAST engine]  →  raw findings (high recall, low precision)
   │
   ▼
[Context assembly]           →  snippet, call graph, surrounding functions,
   │                             triggered rule, CWE reference
   ▼
[LLM reasoning layer]        →  triage verdict + explanation + fix
   │
   ▼
Prioritized, explained findings

The deterministic engine maximizes recall — catch everything plausible, cheaply and reproducibly. The LLM layer maximizes precision — discard what isn’t real and explain what is. You get the strengths of both. Retrieval-augmented generation can inject up-to-date CVE knowledge at inference time so the model reasons about today’s threats, and a locally hosted model keeps proprietary source out of third-party APIs. The newest layer — Copilot Autofix, Snyk’s agent fix, GitLab’s agentic resolution — closes the loop by drafting the patch as a reviewable pull request.

That’s the good. It’s real, it’s measurable, and it attacks the most painful part of the workflow.

The Bad

The capabilities that look like semantic reasoning are still, at bottom, sophisticated pattern matching driven by training data. That has consequences you cannot paper over.

Hallucination

Models invent vulnerabilities that don’t exist and, worse, confidently explain non-issues. In a security gate, a confident wrong answer is more dangerous than a missing one, because it gets trusted. This is why mature systems take a verify-don’t-trust posture: every finding must carry the exact snippet, the triggered rule, and traceable reasoning so a human can spot-check it fast.

Non-determinism

The same code can yield different verdicts across runs. For a deterministic security gate, that’s a genuine operational problem — and a strong argument for keeping the deterministic engine as the source of truth for what to look at, with the LLM only reasoning about it.

Coverage gaps that look like clean scans

A model that finds nothing is not the same as a codebase with nothing to find. Run an LLM solo over a 200-file repo and it will miss most of the real vulnerabilities while inventing a few. Independent benchmarks of LLMs scanning on their own show detection rates nowhere near production-acceptable. The danger is the false sense of security: solo-LLM scanning under-reports, and silence reads as safety.

A brand-new attack surface

As more code is itself AI-generated and as applications embed LLMs, SAST now has to reason about prompt injection, insecure handling of model output, and related classes. The tooling is racing to catch up, and the rules are far less mature than for, say, SQL injection.

The takeaway: an LLM is a bad replacement for static analysis and a bad autonomous security authority. It’s only safe as a reasoning layer on top of a deterministic engine, with a human owning the hard calls.

The Costly

This is the part the demos skip. Even where LLMs work, they’re expensive in ways that don’t show up until you operate them at scale.

Inference cost scales with finding volume, not codebase size

A deterministic scan is essentially free per finding. An LLM triage layer costs tokens per finding — and a noisy engine producing tens of thousands of raw findings means tens of thousands of model calls, each loading a fat context window of surrounding code. The irony: the noisier your SAST engine, the more the LLM layer costs to clean up after it. Context assembly (the thing that drives quality) is also the thing that inflates token count.

Latency in the pipeline

Developers expect SAST feedback in seconds. LLM reasoning adds real latency, especially with rich context. Push it into a blocking CI/CD gate and you slow every pull request; push it async and you lose the fast-feedback property that made SAST useful in the IDE. Either way, there’s a cost.

The privacy tax of self-hosting

Many enterprises can’t send proprietary source to a third-party API, so the privacy-conscious architecture is a locally hosted model. That means GPUs, model-ops, and the engineering to run and update a retrieval layer in-house. The privacy requirement converts a per-call API bill into a standing infrastructure line item.

The accuracy cliff is an economic cliff

Here’s the most important cost, and it’s subtle. A skilled human triages at ~95%. An agent that matches it feels indispensable. But drop to even 85% and trust collapses — security teams stop believing the suppressions and re-review everything, which erases the entire ROI while you keep paying the inference bill. Accuracy isn’t a slope you slide down gracefully; it’s a cliff. This is why mature systems act only when confident, never auto-close a finding outright, and keep suppressed findings grouped for fast bulk audit — so a single missed true positive is recoverable rather than catastrophic.

Residual human cost

LLMs shrink the triage workload; they don’t eliminate it. Borderline findings still route to people, suppressions still need periodic audit, and auto-generated fixes still need review. The metrics that actually tell you whether you’re saving money are fix acceptance rate and regression rate — a patch that silently changes behavior costs far more than the one it replaced.

Where this leaves you

The defensible position is hybrid, and the order matters: deterministic engine for recall, LLM layer for precision, human for the hard calls.

• Good — LLMs automate the triage and explanation work that was always the bottleneck, with production-validated accuracy when bounded to reasoning-about-findings.
• Bad — they hallucinate, they’re non-deterministic, and solo they under-report dangerously. Never let one be the source of truth or the final authority.
• Costly — inference scales with finding volume, latency and self-hosting add real overhead, and the accuracy cliff means sub-par precision burns money instead of saving it.

The LLM isn’t replacing your static analyzer. It’s doing the job a senior security engineer used to do by hand at 2 a.m. — reading each finding, asking “is this actually exploitable here?”, and either writing it up or throwing it out. That job was always the bottleneck. Automating most of it is the real shift — as long as you keep the deterministic engine anchoring the ground truth and stay honest about what it costs.