Hidden bias in the DUD-E dataset leads to misleading performance of deep learning in structure-based virtual screening

A convolutional neural network trained to find drug candidates scores an area under the receiver operating characteristic curve — a ranking quality measure where 1.0 is perfect and 0.5 is random — of 0.98 across one hundred two protein targets. That is a stunning number. Then Chen and colleagues did something simple: they deleted the protein. They fed the model only the drug molecule, replaced the binding site with a single dummy atom, and ran the test again. The score was still 0.98. The model had never needed the protein at all. That result is the center of gravity of this paper, and understanding it requires knowing what these models were built on. Virtual screening is the computational funnel that drug discovery teams use to narrow millions of candidate molecules down to a manageable list worth testing in the lab. Experimental assays are slow and expensive, so a better computational filter can dramatically reduce cost and time in early drug development. The recent approach Chen and colleagues examined treats protein-ligand complexes as three-dimensional images. The binding site is converted into a three-dimensional grid of atom-type densities, forty-eight by forty-eight by forty-eight voxels, twenty-four angstroms per side. A convolutional neural network learns to distinguish binders from non-binders from that grid. Studies using this approach reported striking results, often exceeding 0.9 area under the curve on the benchmark the field had converged on, the Directory of Useful Decoys: Enhanced, or DUD-E. DUD-E contains one hundred two protein targets, twenty-two thousand eight hundred eighty-six experimentally verified active molecules, and over one million decoys. It became the community's standard for training and comparing virtual screening methods. The question Chen and colleagues asked was pointed: when these networks score so well on DUD-E, are they learning the physics of how molecules bind proteins, or are they learning something hidden in the dataset itself? To understand how the bias got in, you need to understand how DUD-E was built. Each active ligand was paired with decoy molecules chosen to match it on simple physical properties, such as molecular size and formal charge, while being topologically dissimilar by fingerprint. Specifically, the top seventy-five percent of candidate decoys most similar to actives by fingerprint were removed, leaving only the most dissimilar twenty-five percent. The intention was sensible: prevent models from trivially separating actives from decoys by gross physical differences. But this design choice created a different problem entirely. Because medicinal chemists optimize drug candidates iteratively — testing one scaffold, modifying it, testing again — many of the active molecules for a given target come from the same chemical series. They are structurally similar to each other. This is what Chen and colleagues call analogue bias, and it appears in two flavors. Intra-target analogue bias means training and test actives for the same protein look alike. Inter-target analogue bias means actives across different protein families can share structural features. Decoy bias means that the conservative fingerprint-based selection made decoys as a class systematically different from actives — not because of anything about binding, but because of how decoys were chosen. A model that learns these patterns can achieve excellent benchmark scores without understanding binding at all. The ligand-only experiment confirmed exactly that. Chen and colleagues trained convolutional neural network models on all one hundred two DUD-E targets in two conditions: one with full receptor-ligand structural input, and one with the protein replaced by a dummy atom. Both models averaged area under the curve of 0.98 across all targets. The per-target scores were correlated with an R-squared of 0.98 and a slope of essentially 1.0. The average absolute area under the curve difference between the two models across one hundred two targets was 0.001. To check whether a receptor-trained model was at least using receptor information at prediction time, the team fed receptor-trained models ligand inputs with and without receptor channels. The average area under the curve difference was 0.0006, and the largest single-target change was 0.027 for a target called fatty acid binding protein four. Inspection of the first-layer convolution weights confirmed it: receptor atom-type channels carried far less weight than ligand channels, and ligand scores predicted with or without receptor information correlated at an R-squared of 0.998. The convolutional neural networks were solving DUD-E almost entirely from ligand features. The protein was decorative. Simple fingerprint-based nearest-neighbor models underscored just how much information the ligand alone carries in DUD-E. A K-nearest-neighbor model using standard molecular fingerprints achieved an area under the curve greater than 0.82 for every one of the one hundred two targets, and ninety-seven targets exceeded 0.90. The optimal neighbor count was one or two for ninety-four percent of targets. That is not a sophisticated model. That is pattern matching on molecular structure — and it nearly matches the deep learning benchmark. To isolate analogue bias from decoy bias, Chen and colleagues built an Actives-as-Decoys dataset — replacing the DUD-E decoys with actual active molecules from other targets, which removes the decoy selection artifact. They then split cross-target predictions into three groups: same target, similar function, and different function. On standard DUD-E decoys, the ligand-only model averaged an area under the curve of 0.983 for same-target predictions, 0.835 for similar-function targets, and 0.618 for different-function targets. When they swapped in the Actives-as-Decoys decoys, those numbers fell sharply: same target dropped to 0.927, similar function to 0.682, and different function landed exactly at 0.500 — random chance. When neither analogue nor decoy bias is available, the models do not generalize at all. Multi-target models made the same point. Convolutional neural networks trained on ten targets and tested on the remaining ninety-two achieved an average area under the curve of 0.80 on standard DUD-E decoys. On the Actives-as-Decoys dataset, that dropped to 0.53 for ligand-only models and 0.54 for receptor-ligand models — essentially coin flips. The bias wasn't a minor confound. It was carrying most of the performance. Which brings us to what happens when you train on better data. Chen and colleagues evaluated two convolutional neural networks trained on PDBbind — a database of actual experimentally determined protein-ligand crystal structures with measured binding affinities — and compared them to AutoDock Vina, a traditional physics-based docking program that uses a hand-crafted energy function rather than learned weights. Vina achieved an average area under the curve of 0.725 across DUD-E targets. The convolutional neural network called Gnina, trained on PDBbind docked poses, averaged 0.709. Pafnucy, trained on PDBbind crystal structures, averaged 0.632. Counting strong performances, Vina exceeded an area under the curve of 0.8 on twenty-four targets and 0.9 on three targets; Gnina on twenty-eight and ten targets respectively; Pafnucy on twelve targets, and never on 0.9. The headline claim — that deep learning beats classical docking — does not survive a clean test. Vina performed comparably to Gnina and better than Pafnucy. Matching a decades-old, parameterized scoring function is a deflating result for systems with millions of learned parameters. Chen and colleagues reinforced this with a concrete example: two chemically similar factor Xa ligands with very different measured binding affinities. All three methods — Vina, Gnina, and Pafnucy — failed to reliably distinguish them by predicted score. The physics the models were supposed to learn apparently wasn't there. The problem Chen and colleagues identify isn't deep learning as a method. It's that the field adopted a benchmark that was never stress-tested for the kinds of shortcuts complex models can take, and then built a literature of claimed advances on top of it. When a benchmark's decoy selection procedure leaves detectable fingerprint patterns, when training and test actives come from the same chemical series, and when models have tens of channels of input and millions of parameters, the conditions for learning artifacts instead of physics are essentially guaranteed. The fixes are straightforward to describe. Chen and colleagues propose three controls that should be standard. First, receptor-stripping: always run a ligand-only version of any receptor-ligand model to measure how much signal comes from the protein versus the molecule alone. Second, decoy-bias-free evaluation: test on datasets where decoys weren't selected by topological dissimilarity from actives. Third, inter-target validation: hold out proteins that are functionally or structurally similar to training targets so analogue bias can't bleed across the split. None of these controls require new experimental data. They require asking harder questions of the data already in hand. A benchmark that passes those three tests would mean something. A model that scores well on it would be worth trusting. The field doesn't need to abandon machine learning in drug discovery — it needs benchmarks that machine learning can't game. Knowing the difference is where the real progress starts. This lecture was created by ennepō. Go to https://ennepo.ai to Discover, Create and Follow the latest research in your field. Read when you can. Listen when you want to.