Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation

Picture the inferior temporal cortex, or IT, as the brain's high-level clearinghouse for objects. By the time a visual signal gets there, shapes and edges have been turned into meanings. In both humans and monkeys, that meaning space has a distinct layout: animates separated from inanimates, with faces and bodies forming tight little neighborhoods within the animate side. The obvious, almost irresistible question is: can our computer vision models land in that same space? Do they carve the world the way IT does, or do they exist in a different geometry entirely? Khaligh-Razavi and Kriegeskorte tackled that with a simple yet powerful idea called Representational Similarity Analysis. Think of it as building a map of confusion. You take a set of images, measure how similar the brain's response patterns are for every pair, and write those dissimilarities into a square matrix. That's the representational dissimilarity matrix, or RDM: zeroes along the diagonal, larger values when two images provoke very different responses, defined here as one minus the Pearson correlation of the patterns. Do the same for a model. Now compare the two matrices. If the model's RDM tracks the brain's RDM, then the model exists in a brain-like geometry. The comparison uses Kendall's tau-A, a rank-based metric that works well when many pairs tie. Crucially, they frame all of this by a "noise ceiling," an estimate of how well any model could hope to perform given measurement noise. You can't exceed the ceiling, but if you're under it, there's still explainable structure left on the table. The test bed is pleasingly broad. On the brain side, human inferior temporal cortex from functional MRI and monkey IT from single-unit recordings, both probed with the same images. On the model side, a variety of options: classic hand-crafted features like Gist and SIFT, biologically inspired stacks like HMAX and VisNet, and a deep supervised convolutional network trained on about one point two million labeled photographs. Since the brain's categories loom large in IT, they went beyond raw features. They trained simple linear classifiers—support vector machines or SVMs—on three splits that matter for IT: animate versus inanimate, face versus nonface, and body versus nonbody, using eight hundred eighty-four labeled images. Those decision values were treated as new features. Along with model features, they were linearly combined with non-negative least squares to best approximate human IT's RDM. Weights were learned through cross-validation—training on a random subset of images and predicting the held-out chunk—so the fit didn't just memorize noise. Behavioral readouts were evaluated with twelve-fold cross-validation. To keep comparisons fair, model outputs were noise-matched to the brain by adding Gaussian noise until their variability matched that of humans. Let's start with what happens if you don't rely on category labels. The not strongly supervised models—hand-crafted descriptors and biologically inspired pipelines with little or no category supervision—do relate to IT, but only modestly. In human IT, their Kendall tau-A correlations top out below about zero point seventeen; for monkey IT, the best land under roughly zero point twenty-six. A carefully blended combination of twenty-seven such models, dubbed "combi27," performs slightly better than any single one, achieving around zero point seventeen for human IT. However, the signature IT structure—animates split from inanimates, with faces and bodies nestled within animates—doesn't emerge cleanly. You can see bits of face clustering, for example, but human and animal faces don't merge into one tight IT-like cluster, and inanimate objects remain dispersed. Deep supervision moves the needle. If you take the deep network's raw features and compute their RDMs, the later layers come closer to IT than the hand-crafted set, but they still fall short of IT's categorical geometry. Layer seven—the penultimate layer—hits around zero point twenty-four with human IT and about zero point twenty-nine with monkey IT. Better, yes. IT-like, not yet. The real change occurs when you let the model explicitly emphasize the brain's major categorical axes. Khaligh-Razavi and Kriegeskorte remixed the deep network's features across layers and added those three linear classifiers—animate versus inanimate, face versus nonface, and body versus nonbody—as additional dimensions to weight. With this supervised IT-geometry fit, the alignment jumps. The combined representation correlates with human IT at about zero point thirty-eight and with monkey IT at roughly zero point forty. That's not just a bump; it lives within the estimated noise ceiling for human IT, meaning it explains essentially all the reliable, noise-free structure in the measurements. It even surpasses the correlation you get by comparing a human IT map to a monkey IT map on the same images, which underscores that noise is doing a lot of work in limiting cross-subject and cross-species ceiling performance. What's going on here is simple to state and gratifying to see. When you train a model rigorously on category labels, it tends to organize images along behaviorally significant lines—what the object is—rather than lower-level details like texture or contour statistics. When you allow those category boundaries to flexibly reweight the model's space, that organization ends up mirroring IT. Now, does that representational similarity actually matter for behavior? It does. On a straightforward task—animate versus inanimate categorization—the deep network's final layer achieves about ninety-six percent accuracy on the image set. The best of the not strongly supervised models, combi27, comes in at approximately seventy-six percent. That's a significant gap. More intriguingly, the relationship across models reveals itself. The closer a representation is to IT, the better it typically categorizes. Across twenty-eight not strongly supervised models and the seven deep layers, the correlation between "how IT-like your RDM is" and categorization accuracy is strong: roughly zero point seventy-five for human IT and around zero point sixty-eight for monkey IT. That's a neat connection between geometry and performance. Zooming in further, you observe that it's not just the broad between-category splits that matter. How well a model captures the within-category structure—the fine-grained layout among, say, different animals or different tools—also predicts how accurately it categorizes. That within-category IT similarity correlates with categorization at about zero point forty-five for human IT and roughly zero point sixty-seven for monkey IT, both statistically robust. So, yes, the animate versus inanimate distinction is crucial. But so is the nuanced internal structure on each side of that division. There's a broader mapping here too, beyond IT. Early visual cortex is concerned with edges, orientations, and local contrasts—the types of features classic models were designed to capture. And that is evident. Among the hand-crafted and biologically inspired set, the HMAX model's C2 stage shows the strongest link to early visual cortex, with a Kendall tau-A around zero point twenty-two. Interestingly, a broader HMAX feature pool best matches the fusiform face area, at about zero point thirteen. For object-selective cortex and the parahippocampal place area, the combi27 blend rises to the top, reaching roughly zero point fourteen for object-selective cortex and about zero point three for the place area, a small but statistically reliable effect. On the deep side, there's a graceful shift along the network: layers two and three nearly reach the noise ceiling for early visual cortex. As you move forward, alignment with early visual cortex diminishes while alignment with object-selective regions grows, with layer seven edging close to the object-selective ceiling. For the face area, layer six matches the ceiling. The place area remains an outlier—barely explained by any model here—which likely reflects more about the stimulus set than the models. As Konkle and Oliva have argued, parahippocampal responses depend on real-world size and contextual cues, which were limited in these centered, relatively uncluttered images. The cross-species story ties everything together neatly. The same images affected both human and monkey IT in similar ways: the same models rose and fell, and the same supervised remix peaked in both. That convergence matters. It suggests we're capturing something about the computation, not just idiosyncrasies of a scanner or a training pipeline. The behavioral link—the way IT resemblance predicts categorization—held across species as well, connecting brain geometry, model space, and performance. A quick note about how meticulously this was measured, because that's part of the confidence. The RDM comparisons used Kendall's tau-A specifically to handle tied ranks when models placed many image pairs as equally dissimilar. Statistical significance and uncertainty were derived from resampling—bootstrap across conditions and randomization tests that permuted labels. The noise ceiling was estimated in two ways: an upper bound that essentially overfits the average single-subject RDMs, and a lower bound using leave-one-subject-out comparisons, which places a conservative limit on what any model could match. To avoid a common pitfall—models appearing better due to less noise—the team added Gaussian noise to the model RDMs until their variability matched that of the brain's, then compared like with like. None of these steps are glamorous, but all of them make the conclusions stronger. If you're curious about what, behind the scenes, that supervised remix actually accomplished, it's refreshingly linear. The deep network yields layer-wise features. The three support vector machines yield decision values that indicate how much an image resembles animate objects, faces, or bodies in that feature space. The team then learned non-negative weights on those components to align the resulting RDM with human IT's RDM, using cross-validation to avoid overfitting. Essentially, they let the data inform them: emphasize this axis more, ease off on that one, until the model's representational map aligns with the brain's shape. It's a straightforward recipe with a strong flavor: prioritize the three categorical splits most salient in IT, and the rest of the structure aligns closer to place. There are limitations, and the authors are transparent about them. The models examined are feedforward. Real ventral streams are recurrent and attentive, and magnetoencephalography work suggests that those crisp category boundaries in IT may emerge slightly later than a single pass through a feedforward network would indicate. The images were simplified—centered and relatively uncluttered—which likely downplays regions like the parahippocampal place area that thrive on scenes and context. While supervised labels are a convenient learning signal for machines, the brain's teaching signal is multifaceted: it blends attention, temporal continuity, action outcomes, and multimodal context. That's part of why the authors describe IT as visuo-semantic. What IT encodes isn't just what hits the retina; it's what that object means for a behaving animal. What should we take away from this? First, there's a resolute pattern: scale and supervision matter. When you train on large labeled datasets, you achieve internal geometries that resemble IT, especially when you allow some linear adjustments to amplify the animate versus inanimate and face versus body axes that dominate in IT. Second, representational similarity isn't merely an aesthetic match—it predicts behavior. The models that resemble IT categorize more accurately, not just by adhering to the broad boundaries but by capturing the nuances within each category. Third, the connection from models to the brain is regional and graded: early layers map to early visual cortex; deeper layers relate to object- and face-selective regions; and a supervised, category-tuned blend can align with IT at its own geometry. Looking ahead, the optimistic yet grounded view is this: we can already match a substantial portion of IT's representational space with supervised deep networks, and we can achieve this in a way that's cross-validated, noise-capped, and predictive of behavior. To fully bridge the gap, we'll likely need elements beyond static feedforward vision—recurrent loops that refine categories over time, attention that selects task-relevant features, and learning signals that resemble real-world scenarios more than the ImageNet dataset. The path is clear enough to traverse, and the first steps, as Khaligh-Razavi and Kriegeskorte demonstrated, are already underway.