Your app ships every week. Is your pen test keeping up?
AWS’s new Reimagine report says AI systems should be tested continuously, not once. The same now applies to the apps they help you build. Here’s why a once-a-year pen test can’t keep up, and what testing at the speed you ship looks like.
Most teams we work with ship something every week. Most of them get a penetration test once a year. That gap used to be an acceptable trade-off. With AI writing more of the code and attackers using AI to find flaws faster, it isn’t any more.
The 60-second version
- 01
Software now changes in days, not months.
Every release, new dependency, API and AI feature is new attack surface. A yearly pen test describes one version of your app, not the one in production.
- 02
Attackers got faster too.
Mandiant measured the average time from disclosure to exploitation falling from 63 days (2018–19) to 5 days in 2023.[2] AI makes finding and chaining flaws cheaper for everyone.
- 03
Test at the speed you ship.
AI pen testing makes a real attack-and-prove test cheap enough to run every quarter and after every major release, not once a year.
THE GAP
A pen test describes one version of your app.
A traditional pen test is a snapshot. Skilled testers spend a week or two attacking the version of your app that exists that week, then hand over a report. It’s valuable, and it starts going out of date the moment your next release ships.
One year of a typical SaaS team
52 weeks of releases. 1 pen test.
That isn’t a criticism of manual testing. It’s arithmetic. If a test takes weeks to book and costs tens of thousands of dollars, you do it once a year, and every release in between reaches customers without anyone trying to break it.
WHAT AWS FOUND
Test continuously, not once.
AWS’s Executives in Residence spent nine months interviewing 154 executives at 128 organisations for their Reimagine report.[1] The chapter on governing AI sets out four principles, drawn partly from Amazon’s experience running more than 26,000 AI agents internally. The fourth is the one that matters here.
PRINCIPLE 01
Get the basics right first.
Identity, access and encryption still do the protective work, now at machine speed.
PRINCIPLE 02
Set boundaries outside the agent.
Real limits live in infrastructure the system can’t talk its way around.
PRINCIPLE 03
Let agents earn their autonomy.
Start with human approval and expand it only after proven reliability.
PRINCIPLE 04
Test continuously, not once.
Models update, prompts evolve, tools change. Each change can bring failure modes the last review never saw.
The report’s point is about AI systems: a test that passed yesterday tells you less than it used to, because the system keeps changing underneath it. The same is now true of the apps those AI systems help you build. When code ships in days, governance and testing that run once a year can’t keep up.[1]
Every release is new attack surface. Your testing should move at the same speed.
WHAT CHANGES BETWEEN TESTS
Six things your last pen test never saw.
Each of these can introduce a vulnerability, and most teams change all six in a normal quarter.
New code
Features, fixes and refactors every sprint
New dependencies
Libraries and their own vulnerabilities
New endpoints
APIs, webhooks and integrations
New AI features
Models, prompts, tools and agents
New access
Roles, permissions and test users
New infrastructure
Config, networking and storage
THE OTHER SIDE
Attackers got faster too.
The same AI that helps your developers ship faster helps attackers find and chain vulnerabilities faster and more cheaply. The window between a flaw becoming known and being exploited has been shrinking for years.
Average time-to-exploit, Mandiant
The window between a flaw and an attack keeps shrinking.
If attackers can move in days, a test you run in twelve months’ time isn’t protecting the release you shipped last Tuesday.
WHAT CHANGES WITH AI
Testing at the speed you ship.
AI pen testing changes the economics. AWS Security Agent is AWS’s own AI penetration testing agent, generally available since March 2026.[3] A team of specialised agents reads your code and design docs, runs real multi-step attacks against a staging copy of your app, and only reports what it can prove by exploiting it. Each finding comes with a CVSS severity score, steps to reproduce and a recommended fix.[4]
Because a run is quick and repeatable, testing stops being an annual event and becomes part of how you ship.
How the loop runs
- 01
Ship
A major release goes out
- 02
Read
The agent reads your code and docs
- 03
Attack
It runs real attack chains on staging
- 04
Prove
Findings are confirmed by exploiting them
- 05
Fix
Developers fix, ranked by severity
- 06
Re-test
The agent confirms each fix holds
A CADENCE THAT WORKS
When to test.
This is the rhythm we set up for teams that ship often. Each run is repeatable, and we agree the run time and budget before it starts.
- 01
Every quarter
A full baseline test of the app as it stands.
- 02
After every major release
New features and flows get attacked before customers rely on them.
- 03
When you add an AI feature or integration
New models, prompts, tools and third-party connections change how your app can fail.
- 04
After fixing a finding
Re-test, so you know it’s closed rather than hoping it is.
YOUR NEXT STEP
Five questions before your next release.
If you can’t answer the first two with confidence, your testing is behind your shipping.
When did someone last attack the version of your app that’s in production today?
How many releases, dependencies and endpoints have shipped since then?
Have you added AI features, prompts or tool integrations since your last test?
Do you have a staging environment that mirrors production, so testing never touches customers?
If a test found something serious tomorrow, could you re-test the fix the same week?
References.
- [1]AWS Executives in Residence (2026). Reimagine: Turning AI into Value. Amazon Web Services. Chapter 05, “The governed path”. aws.amazon.com/executive-insights/reimagine ↗
- [2]Charrier, C. & Weiner, R. (15 Oct 2024). How Low Can You Go? An Analysis of 2023 Time-to-Exploit Trends. Mandiant, Google Cloud Blog. cloud.google.com/blog/topics/threat-intelligence ↗
- [3]Amazon Web Services (31 Mar 2026). AWS Security Agent on-demand penetration testing is now generally available. What’s New with AWS. aws.amazon.com/about-aws/whats-new ↗
- [4]Amazon Web Services. AWS Security Agent features. aws.amazon.com/security-agent/features ↗
- [5]Assembly (2026). Kamal.ai: staying ahead of AI-powered attackers with AWS Security Agent. Case study. assembly.cloud/case-studies/kamal-ai ↗
