Your app ships every week. Is your pen test keeping up?

AWS’s new Reimagine report says AI systems should be tested continuously, not once. The same now applies to the apps they help you build. Here’s why a once-a-year pen test can’t keep up, and what testing at the speed you ship looks like.

Most teams we work with ship something every week. Most of them get a penetration test once a year. That gap used to be an acceptable trade-off. With AI writing more of the code and attackers using AI to find flaws faster, it isn’t any more.

The 60-second version

  1. 01

    Software now changes in days, not months.

    Every release, new dependency, API and AI feature is new attack surface. A yearly pen test describes one version of your app, not the one in production.

  2. 02

    Attackers got faster too.

    Mandiant measured the average time from disclosure to exploitation falling from 63 days (2018–19) to 5 days in 2023.[2] AI makes finding and chaining flaws cheaper for everyone.

  3. 03

    Test at the speed you ship.

    AI pen testing makes a real attack-and-prove test cheap enough to run every quarter and after every major release, not once a year.

THE GAP

A pen test describes one version of your app.

A traditional pen test is a snapshot. Skilled testers spend a week or two attacking the version of your app that exists that week, then hand over a report. It’s valuable, and it starts going out of date the moment your next release ships.

One year of a typical SaaS team

52 weeks of releases. 1 pen test.

ReleaseMajor releasePen test
Pen test · week 1
51releases nobody attacked
JANJULDEC
Illustrative. A team shipping weekly, with a major release every six weeks and one annual pen test. Everything in the shaded area reached production untested.

That isn’t a criticism of manual testing. It’s arithmetic. If a test takes weeks to book and costs tens of thousands of dollars, you do it once a year, and every release in between reaches customers without anyone trying to break it.

WHAT AWS FOUND

Test continuously, not once.

AWS’s Executives in Residence spent nine months interviewing 154 executives at 128 organisations for their Reimagine report.[1] The chapter on governing AI sets out four principles, drawn partly from Amazon’s experience running more than 26,000 AI agents internally. The fourth is the one that matters here.

PRINCIPLE 01

Get the basics right first.

Identity, access and encryption still do the protective work, now at machine speed.

PRINCIPLE 02

Set boundaries outside the agent.

Real limits live in infrastructure the system can’t talk its way around.

PRINCIPLE 03

Let agents earn their autonomy.

Start with human approval and expand it only after proven reliability.

PRINCIPLE 04

Test continuously, not once.

Models update, prompts evolve, tools change. Each change can bring failure modes the last review never saw.

The report’s point is about AI systems: a test that passed yesterday tells you less than it used to, because the system keeps changing underneath it. The same is now true of the apps those AI systems help you build. When code ships in days, governance and testing that run once a year can’t keep up.[1]

Every release is new attack surface. Your testing should move at the same speed.

WHAT CHANGES BETWEEN TESTS

Six things your last pen test never saw.

Each of these can introduce a vulnerability, and most teams change all six in a normal quarter.

New code

Features, fixes and refactors every sprint

New dependencies

Libraries and their own vulnerabilities

New endpoints

APIs, webhooks and integrations

New AI features

Models, prompts, tools and agents

New access

Roles, permissions and test users

New infrastructure

Config, networking and storage

THE OTHER SIDE

Attackers got faster too.

The same AI that helps your developers ship faster helps attackers find and chain vulnerabilities faster and more cheaply. The window between a flaw becoming known and being exploited has been shrinking for years.

Average time-to-exploit, Mandiant

The window between a flaw and an attack keeps shrinking.

2018–1963 days
2020–2144 days
2021–2232 days
20235 days
Average days between a vulnerability being disclosed and exploited, for vulnerabilities Mandiant analysed. The 2023 figure excludes 15 outliers; Mandiant reports 47 days with them included.[2]

If attackers can move in days, a test you run in twelve months’ time isn’t protecting the release you shipped last Tuesday.

WHAT CHANGES WITH AI

Testing at the speed you ship.

AI pen testing changes the economics. AWS Security Agent is AWS’s own AI penetration testing agent, generally available since March 2026.[3] A team of specialised agents reads your code and design docs, runs real multi-step attacks against a staging copy of your app, and only reports what it can prove by exploiting it. Each finding comes with a CVSS severity score, steps to reproduce and a recommended fix.[4]

Because a run is quick and repeatable, testing stops being an annual event and becomes part of how you ship.

EVERY MAJOR RELEASETest → Fix → ShipShipReadAttackProveFixRe-test

How the loop runs

  1. 01

    Ship

    A major release goes out

  2. 02

    Read

    The agent reads your code and docs

  3. 03

    Attack

    It runs real attack chains on staging

  4. 04

    Prove

    Findings are confirmed by exploiting them

  5. 05

    Fix

    Developers fix, ranked by severity

  6. 06

    Re-test

    The agent confirms each fix holds

A CADENCE THAT WORKS

When to test.

This is the rhythm we set up for teams that ship often. Each run is repeatable, and we agree the run time and budget before it starts.

  1. 01

    Every quarter

    A full baseline test of the app as it stands.

  2. 02

    After every major release

    New features and flows get attacked before customers rely on them.

  3. 03

    When you add an AI feature or integration

    New models, prompts, tools and third-party connections change how your app can fail.

  4. 04

    After fixing a finding

    Re-test, so you know it’s closed rather than hoping it is.

YOUR NEXT STEP

Five questions before your next release.

If you can’t answer the first two with confidence, your testing is behind your shipping.

  • When did someone last attack the version of your app that’s in production today?

  • How many releases, dependencies and endpoints have shipped since then?

  • Have you added AI features, prompts or tool integrations since your last test?

  • Do you have a staging environment that mirrors production, so testing never touches customers?

  • If a test found something serious tomorrow, could you re-test the fix the same week?

Live webinar · Mon 23 Nov

See AWS Security Agent run, live.

11:00 am AEST · 12:00 pm AEDT on Zoom, with Will Garcia from AWS and Rodrigo Genaro from Assembly. Free.

Save my seat

Offer · until 30 Nov 2026

A complimentary AI pen test.

For technology companies in production spending $3k+ a month on cloud. Funded through the AWS Proof of Concept program, subject to AWS approval.

Check my eligibility

References.

  1. [1]
    AWS Executives in Residence (2026). Reimagine: Turning AI into Value. Amazon Web Services. Chapter 05, “The governed path”. aws.amazon.com/executive-insights/reimagine ↗
  2. [2]
    Charrier, C. & Weiner, R. (15 Oct 2024). How Low Can You Go? An Analysis of 2023 Time-to-Exploit Trends. Mandiant, Google Cloud Blog. cloud.google.com/blog/topics/threat-intelligence ↗
  3. [3]
    Amazon Web Services (31 Mar 2026). AWS Security Agent on-demand penetration testing is now generally available. What’s New with AWS. aws.amazon.com/about-aws/whats-new ↗
  4. [4]
    Amazon Web Services. AWS Security Agent features. aws.amazon.com/security-agent/features ↗
  5. [5]
    Assembly (2026). Kamal.ai: staying ahead of AI-powered attackers with AWS Security Agent. Case study. assembly.cloud/case-studies/kamal-ai ↗

Updates, without the noise.

New events, case studies and announcements, once a month.

Send your subscription request by email.