Every machine learning model shares the same vulnerability profile. A decision tree scoring loan applications, a random forest flagging fraudulent transactions, and a trillion-parameter language model writing your emails are all vulnerable to the same six categories of attack. Not similar vulnerabilities. The same six, for the same underlying reason, regardless of architecture.
Here's the part that should bother you: none of it goes away as the models get better. There's no future version of this problem where enough compute and enough data make it disappear. This isn't a claim about the current state of AI security research catching up to AI capability. It's a claim about what these models fundamentally are, at the moment you decide to build one at all.
By the end of this post you'll understand why, and you'll have a framework for sorting every AI attack you ever read about into one of three tiers, based on when in a model's life the attack actually happens. Let's start by understanding what a model actually is, because most explanations of AI security skip straight to the attacks and skip the vocabulary to understand why they work.
Boundless possibilities, but which to choose
Say you're building a model to catch fraudulent credit card transactions. Every transaction gets turned into a list of numbers: the amount, the time of day, the distance from the cardholder's home, how many transactions happened in the last hour, and so on. That list of numbers is called a vector. It's just a point in space, the same way (3, 7) is a point on a two-dimensional graph, except your fraud vector might have a few hundred dimensions instead of just two.
Training an AI model means feeding it thousands of these vectors, each one labeled "fraud" or "not fraud," and asking it to find a way to divide that space so fraud vectors land on one side and legitimate ones land on the other. That dividing line, or in higher dimensions, that dividing surface, is called the decision boundary. Once it's drawn, classifying a new transaction is just a matter of checking which side of the boundary it falls on. That is what the model is, at its core. A decision boundary.
A neural network draws a wildly complicated, curved boundary using layers of weighted connections. A decision tree draws a boundary made of right-angle splits, one variable at a time. A support vector machine tries to draw the boundary with the widest possible margin between the two classes. Different methods, same underlying job: turn things into vectors, draw a line, decide which side something's on.
And this isn't just a classification story. It's easy to picture a boundary when the output is a label like "fraud" or "not fraud," but the exact same mechanism drives every decision a model makes, including the ones that don't look like classification at all.
A game-playing AI deciding whether to move left or right is picking the highest-scoring option among a handful of vectors representing possible next states, which is a boundary decision with many regions instead of just two. A language model deciding which word comes next is doing the same thing at a staggering scale: it turns the current context into a vector, and effectively draws a boundary across the entire vocabulary, tens of thousands of possible next tokens, to decide which region that vector falls into.
"Choose an action," "choose a word," and "choose a label" are the same operation wearing different clothes. Somewhere underneath any AI decision you can name, something got turned into a vector and something got decided based on which side of a boundary it landed on.
Everything that follows in this piece comes back to that line.
It's all heuristics, baby!
Every decision boundary you'll ever encounter is a heuristic, not a perfect answer; that's true by design, not by accident. Machine learning algorithms don't produce perfect answers. They were never built to. If a perfect answer existed, we'd just write an algorithm to compute it directly, every time, and skip the whole business of training an approximator on examples. Nobody's done that for the kinds of problems AI gets used for. There's no known perfect algorithm that can tell you, with certainty, whether a given email is spam in the same way there's a perfect algorithm for sorting a list of numbers. Spam isn't a problem with a formula. It's a moving target designed by people actively trying to guess what your filter will let through, which is a very different kind of problem than the ones exact algorithms are good at.
Let's start with the data problem. When you train a fraud detection model, for example, you're not showing it every credit card transaction that will ever happen. You can't; the overwhelming majority of them haven't happened yet. You're showing it a finite sample and asking it to draw a boundary that generalizes to transactions it's never seen. The error rate on your training sample is called empirical risk. The error rate on the true, complete population of every transaction that could ever occur is called true risk. Every model you've ever used is minimizing the first one as a stand-in for the second, and the gap between them never fully closes. It shrinks with more data, but it's a structural feature of learning from a sample rather than a bug someone forgot to patch.
It gets worse. In 1996, David Wolpert proved something that sounds almost like a magic trick: averaged across every possible target function a learning algorithm could be asked to approximate, no algorithm outperforms any other, including random guessing [1]. This is the No Free Lunch theorem for supervised learning, and it means a model only works well on real-world problems because it encodes assumptions, baked in by its architecture and training procedure, about what the real world looks like. A model that assumes nearby points tend to share a label will do well on problems where that's true and badly on problems where it isn't. There's no such thing as a bias-free learner that's good at everything. Every model's usefulness comes from the specific assumptions it makes, and every one of those assumptions is a place an adversary can look for where the real world doesn't match the assumption.
And then there's the part that should really bother you: for many of these problems, finding the exact best answer isn't just hard, it's provably intractable. In 1976, Hyafil and Rivest proved that constructing the optimal binary decision tree for a given dataset is NP-complete [2], computer science's term for a problem where the time required to find the perfect answer explodes so fast as the problem grows that no computer, then or now, could finish it for a dataset of any real size. Not slow. Not "needs a better processor." Actually infeasible, for any dataset worth building a model over.
In 1992, Blum and Rivest proved that exactly training even a trivial three-node neural network to fit a given set of examples is NP-complete as well [3]. These aren't results about deep learning specifically or about some particularly unlucky architecture. They're results about the exact versions of the underlying optimization problems, and they hold regardless of which method you use to attack them.
This is why every model you'll ever use is trained with a heuristic instead of an exact solver. Gradient descent doesn't guarantee it finds the global best set of weights; it finds a decent local one and stops when it runs out of patience. Greedy decision tree splitting doesn't guarantee the optimal tree; it makes the locally best choice at each node and never looks back. Nobody does this because they're being lazy. They do it because that exact version of the problem is often computationally out of reach for any amount of hardware you're willing to buy, and a decent heuristic answer today beats a perfect answer that would take longer than the heat death of the universe to compute.
Put those three things together and you realize that the approximation is the product. It's not an unfortunate side effect of current AI technology. It's the entire reason these models are useful at all, since they're solving problems that don't have a practical exact solution. And a system built around an approximation, by definition, has cases where the approximation is wrong. Those cases are exactly what the following six threats exploit.
The six threats, in three tiers
The National Institute of Standards and Technology maintains an official taxonomy of adversarial machine learning attacks, and it sorts them along similar lines to what follows [4]. I'd group the same threats slightly differently, by when in a model's life the attack actually happens, because that's what determines whether "just build a better model" would help at all.
Tier 1: Attacks that shape the boundary before it exists
These happen during training, before the final decision boundary has even been drawn.
Data poisoning corrupts the training data itself, shifting where the boundary ends up. Biggio, Nelson, and Laskov demonstrated this against support vector machines back in 2012, showing that carefully crafted poison points inserted into the training set could shift the learned boundary enough to significantly increase the model's error rate on legitimate data [5]. It's worth making the point that this predates the deep learning boom by several years. This isn't a neural network quirk. Any model that learns its boundary from data can have that boundary bent by corrupting the data.
Model trojans, sometimes called backdoors, plant a hidden behavior during training that only activates when the model sees a specific trigger. Gu, Dolan-Gavitt, and Garg's BadNets work showed that a model could be trained to perform normally on every input except ones containing a small, almost invisible trigger pattern, at which point it would output whatever the attacker wanted [6]. A stop sign classifier that works perfectly except when a small sticker is present, at which point it reports "speed limit sign," is the canonical example. I'll be honest about a limitation here: the trojan literature is overwhelmingly deep-learning-specific. I don't have a strong classical machine learning precedent to show you the way I do for poisoning. The underlying mechanism, a trigger correlated with a hidden shortcut in a large flexible model, isn't fundamentally exclusive to deep networks, but the extensive demonstration of it mostly is.
Tier 2: Attacks that exploit the boundary once it exists
These treat a trained model as a black box (or occasionally a gray box) and probe its statistical residue from the outside, after training is finished.
Adversarial examples craft an input that crosses the boundary while looking unchanged to a human observer. If you want the concrete version of this, I already walked through it in detail: changing a single pixel out of 50,000 can flip a state-of-the-art image classifier's output entirely [7]. The model didn't get worse at its job. The boundary always had a soft spot nearby; nobody had gone looking for it until someone did.
Membership inference doesn't try to fool the model at all. It tries to determine whether a specific record was part of the training set in the first place, which matters enormously for privacy when the training data includes medical records, financial histories, or anything else sensitive. Shokri, Stronati, Song, and Shmatikov showed in 2017 that this works because models are systematically more confident on data they've memorized than on data they've only generalized to [8]. Feed the model a record and check how suspiciously confident it is; that confidence gap alone leaks membership.
Model extraction reconstructs a working approximation of the boundary purely through query access, no internal access required. Tramèr, Zhang, Juels, Reiter, and Ristenpart demonstrated this against models exposed as prediction APIs, including decision trees, logistic regression, and neural networks, showing an attacker could rebuild a functionally equivalent model just by asking it enough well-chosen questions and watching the answers [9]. This one deserves particular attention if your business model depends on a trained model being expensive to reproduce, because the paper's whole point is that expense doesn't protect you the way you'd hope.
Tier 3: Attacks that skip the boundary entirely
This is the tier that breaks the 'perfect boundary' scenario wide open.
Direct manipulation means altering a model's learned parameters directly: the weights and biases of a neural network, the split thresholds of a decision tree, the coefficients of a regression model. Not the training data. Not queries sent to a deployed API. The parameters themselves, after the fact.
This sounds abstract until you see how concretely it's been demonstrated. Rakin, He, and Fan's Bit-Flip Attack showed that flipping as few as three bits in a quantized neural network's stored weights, selected by how much each bit's corruption would increase the model's loss, is enough to catastrophically wreck its accuracy [10]. Three bits, out of millions. And this isn't confined to a lab thought experiment with privileged hardware access: Yao, Rakin, and Fan's DeepHammer showed those bit flips can be induced from unprivileged code, using the Rowhammer hardware vulnerability in commodity DRAM, which repeatedly hammers memory rows until electrical interference flips bits in physically adjacent rows storing the model's weights [11]. No access to the training data. No queries to an API. Just memory access and a well-chosen set of bits to target.
Direct manipulation is fundamentally an integrity and access-control problem wearing an ML costume. It doesn't care whether the boundary was drawn well or drawn badly. It works identically well against a state-of-the-art model as it does a mediocre one, because it isn't attacking the statistics at all. It's attacking the artifact.
Why there's no free lunch
Here's where the two halves of this equation come together.
Tiers 1 and 2 exist because of the argument from the second section: every model is a finite-sample approximation carrying baked-in assumptions about the world, trained by a heuristic instead of an exact solver, because the exact version of the problem is often computationally out of reach. That gap between the model you trained and the model you wanted is structurally guaranteed, not a temporary state of the art. It doesn't close because a bigger model came out. It shrinks, sometimes, but the underlying reason attacks in these tiers are possible at all never goes away.
It's worth pushing this even further than "the model isn't good enough yet," because there's a genuinely uncomfortable result in the adversarial examples literature that a lot of AI security writing glosses over. Shafahi, Huang, Studer, Feizi, and Goldstein proved that for high-dimensional data under fairly general assumptions, adversarial examples exist even for a hypothetically perfect classifier, meaning one with zero error across the entire true population, not just the training sample [12]. The mechanism traces back to Gilmer and colleagues' earlier work studying classifiers on high-dimensional spheres, and it comes down to concentration of measure: in high-dimensional space, a tiny expansion around almost any region tends to swallow up nearly the whole space [13]. A boundary that's correct everywhere can still have correct points sitting adjacent to incorrect ones, close enough that a human can't perceive the difference between them. This tracks with a related argument, that adversarial vulnerability may not be a bug at all but a direct consequence of models learning genuinely predictive statistical patterns that simply don't align with human perception [14]. Fixing the statistical-approximation problem entirely, in other words, might not be sufficient to close this gap. The geometry of high-dimensional data may guarantee it regardless.
That's the first reason, and it applies even in the best case you could hope for.
Tier 3 is the second, independent reason, and it doesn't need any of the above to be true. Direct manipulation doesn't ask whether your boundary is good, bad, or theoretically perfect. It doesn't touch the statistics. A model trained on flawless data, validated against every edge case imaginable, running with zero generalization error, is exactly as vulnerable to having three of its weight bits flipped as a model somebody trained badly over a weekend. The attack never goes through the boundary's quality at all, which means improving the boundary's quality does nothing to stop it.
You can't close the gap that Tiers 1 and 2 exploit, because it's a structural consequence of learning from finite data using necessarily heuristic methods, and recent theoretical work suggests high-dimensional geometry alone may guarantee some version of it regardless of how good the model gets. And even in the counterfactual world where you somehow did close that gap completely, it wouldn't matter, because Tier 3 proves a perfect model is exactly as exposed as an imperfect one to an attacker who can just rewrite its parameters directly.
There's no no escape. Not because AI security research hasn't caught up yet, but because two entirely separate lines of argument both point at the same conclusion, for different reasons, from different directions.
No perfect defense doesn't mean no defense at all
Everything above is true, and none of it is an argument for giving up.
It might feel that way, reading a piece that just spent several paragraphs proving a negative result. But "there's no perfect defense" isn't a discovery specific to AI, and it isn't evidence that security engineering doesn't work. It's the permanent condition of the entire field. Bruce Schneier made essentially this argument about traditional computer security back in 2000, pointing out that buffer overflow attacks had already been understood since 1960s time-sharing systems, and that decades of security literature and enormous amounts of tooling still hadn't made them go away [15]. His conclusion wasn't that security is impossible. It was that security is a process, not a product, something you manage continuously rather than something you install once and cross off a list.
That's the reframe we have been building toward. The goal was never a system with zero risk. That system doesn't exist for software in general, and the arguments in the last section show it can't exist for machine learning specifically, for reasons that run deeper than "we haven't gotten around to fixing it yet." The actual goal of security engineering, in AI or anywhere else, is risk management: reduce the likelihood an attack succeeds, reduce the damage when one does anyway, and make the whole exercise expensive and detectable enough that an attacker goes looking for an easier target instead. None of the six threats in this piece need to be solved. They need to be made not worth the trouble, which is a completely different, and entirely achievable, goal.
That reframing is also the plan for the rest of this series. Each tier has its own defense literature, its own tradeoffs, and its own open questions. Future posts here will work through all six threats in turn: data provenance and training-time anomaly detection against poisoning, trigger reconstruction and activation clustering against trojans, adversarial training and certified robustness against evasion, differential privacy against membership inference, query throttling and watermarking against extraction, and integrity verification, cryptographic signing, and hardware attestation against direct manipulation.
None of those defenses are perfect either. And at this point, that really shouldn't surprise you. What they buy you isn't certainty. It's an attacker who has to work harder, get luckier, and risk getting caught, instead of walking in through a door nobody bothered to lock. That's not everything. It's not nothing, either.
References
[1] D. H. Wolpert, "The lack of a priori distinctions between learning algorithms," Neural Computation, vol. 8, no. 7, pp. 1341-1390, Oct. 1996.
[2] L. Hyafil and R. L. Rivest, "Constructing optimal binary decision trees is NP-complete," Information Processing Letters, vol. 5, no. 1, pp. 15-17, May 1976.
[3] A. L. Blum and R. L. Rivest, "Training a 3-node neural network is NP-complete," Neural Networks, vol. 5, no. 1, pp. 117-127, 1992.
[4] A. Vassilev, A. Oprea, A. Fordyce, H. Anderson, X. Davies, and M. Hamin, "Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations," National Institute of Standards and Technology, NIST AI 100-2e2025, Mar. 2025. [Online]. Available: https://doi.org/10.6028/NIST.AI.100-2e2025
[5] B. Biggio, B. Nelson, and P. Laskov, "Poisoning attacks against support vector machines," in Proc. 29th Int. Conf. Machine Learning (ICML), 2012.
[6] T. Gu, B. Dolan-Gavitt, and S. Garg, "BadNets: Identifying vulnerabilities in the machine learning model supply chain," arXiv:1708.06733, 2017.
[7] J. Su, D. V. Vargas, and K. Sakurai, "One pixel attack for fooling deep neural networks," IEEE Transactions on Evolutionary Computation, vol. 23, no. 5, pp. 828-841, Oct. 2019.
[8] R. Shokri, M. Stronati, C. Song, and V. Shmatikov, "Membership inference attacks against machine learning models," in Proc. IEEE Symp. Security and Privacy, 2017.
[9] F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart, "Stealing machine learning models via prediction APIs," in Proc. 25th USENIX Security Symposium, 2016.
[10] A. S. Rakin, Z. He, and D. Fan, "Bit-Flip Attack: Crushing neural network with progressive bit search," in Proc. IEEE/CVF Int. Conf. Computer Vision (ICCV), 2019.
[11] F. Yao, A. S. Rakin, and D. Fan, "DeepHammer: Depleting the intelligence of deep neural networks through targeted chain of bit flips," in Proc. 29th USENIX Security Symposium, 2020.
[12] A. Shafahi, W. R. Huang, C. Studer, S. Feizi, and T. Goldstein, "Are adversarial examples inevitable?," in Proc. Int. Conf. Learning Representations (ICLR), 2019.
[13] J. Gilmer, L. Metz, F. Faghri, S. S. Schoenholz, M. Raghu, M. Wattenberg, and I. Goodfellow, "Adversarial spheres," arXiv:1801.02774, 2018.
[14] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, "Adversarial examples are not bugs, they are features," in Proc. 33rd Conf. Neural Information Processing Systems (NeurIPS), 2019.
[15] B. Schneier, "The Process of Security," Information Security, Apr. 2000. [Online]. Available: https://www.schneier.com/essays/archives/2000/04/the_process_of_secur.html