The Shadow Traffic Problem: Combating AI-Driven Automated Threats to Websites
Day in and day out, web applications are subject to unwanted automated usage. These events often relate to misuse of inherent valid functionality, rather than the attempted exploitation of unmitigated vulnerabilities. Examples of these events include click fraud, comment spamming, content scraping, password cracking, and many more.
An OWASP project on Automated Threats to Web Applications has produced an ontology providing a common language to facilitate clear communication and help tackle these threats. The project identifies symptoms of these issues and discusses countermeasures against them.
Nowadays, AI is turbocharging the problem into overdrive. Residential proxies, full browser engines, and LLM-driven agents now produce automation that looks human. When combined with emerging Scraping-as-a-Service providers, the cost of doing business on the web goes up significantly.
An early 2026 study of web traffic by AI shows that 51% of web traffic is human and 1% is wanted bots; the rest is unwanted automation. By September, human traffic accounted for less than half of all web interactions. This shift is further highlighted by the arrival of AI agents that are chattier than humans.
This talk will discuss both technical and non-technical countermeasures for combating unwanted automation. We will explore regulatory frameworks and financial incentives to realign human-machine co-existence on the web.
In addition, we will discuss emerging mechanics such as cryptographic agent identity via HTTP message signatures, machine-readable terms and pricing, pay-per-crawl programs, HTTP 402 Payment Required as a settlement primitive, and intermediaries at natural metering points.
AI Summary
Video chapters
0:00Introduction and speaker background2:39Origins of the OWASP automated threats project5:02The automated threats ontology8:50Countermeasures across the development lifecycle10:44Today's bots and scraping as a service13:48How AI is increasing bot traffic17:30Continuous defense and the legality of scraping20:29Bot traffic as an economic problem23:10Questions and answers
Key facts
- The OWASP Automated Threats to Web Applications ontology, started in 2015, lists 21 threats; ad fraud was recently broadened to cost inflation fraud to cover attacks such as SMS pumping.
- Fastly data from around January showed 51% human and 49% bot traffic, with only 1% of all traffic being wanted automation.
- In more recent data from around August and September, Fastly observed more bot traffic than human traffic, with traffic linked to AI code generation up 34 times.
- A joint study by Fastly and GreyNoise found that a large number of user agents impersonate AI crawlers, apparently hoping operators will not block them.
- Residential proxy networks often rely on cheap consumer devices, such as picture frames or televisions, and click-through end-user agreements to route automated requests through users' connections.
Ideas
- Automated threats abuse features that work as intended, so secure development lifecycle (SDLC) controls and vulnerability management alone do not stop them; the problem surfaces in operations instead.
- Naming threats by mechanism rather than attacker goal matters: credential cracking and credential stuffing both lead to account takeover but produce different signals and need different countermeasures.
- Countermeasures should span the whole transaction: prevent before it, detect, block or slow down during it, and recover afterwards by canceling bookings or refusing to ship suspicious orders.
- Old bot assumptions such as data-center IP addresses, no JavaScript, failed CAPTCHAs and repetitive bursts no longer hold, as scraping-as-a-service providers offer paced, residential, browser-based automation.
- Treating unwanted automation as an economic problem suggests charging heavy automated consumers, while leaving allowance for occasional, low-volume lookers.
Keywords
- OWASP
- OWASP Automated Threats
- bot mitigation
- scraping
Quotes
- “So the problem is that you cannot touch a feature that is working as intended, right?”
- “AI agents tend to be a lot chattier than humans, right? Before they buy something, they ask a lot of questions.”
- “What I really think this should be framed is is an economic problem.”
- “There should be some allowance for occasional lookers, right? And only when somebody becomes too aggressive that you start asking like did you pay for it?”
Recommendations
- Use the OWASP automated threats ontology to identify which specific threats affect your business functions, and combine technical controls with business measures such as terms and conditions, inventory limits and post-transaction cancellation.
- Treat bot defense as a continuous process rather than a set-and-forget rule set, monitoring and adapting controls as automation becomes more human-like.
- Consider verifying signed agent requests and returning HTTP 402 Payment Required to aggressive automated consumers, instead of only rate limiting with HTTP 429, while tolerating occasional low-volume crawlers.
Resources
- OWASP Automated Threats to Web Applications — Framework — Gives defenders a common language to classify bot abuse of legitimate functionality and select appropriate countermeasures.
- RFC 9421: HTTP Message Signatures — Standard — Enables cryptographic agent identity so operators can distinguish and meter legitimate automated clients.
- Fastly Threat Insights Report — Report — Provides CDN-scale data on the share and growth of bot and AI traffic.
About the speaker
Tin Zaw
Read more …