Shared computational principles for language processing in humans and deep language models
Ariel Goldstein, Zaid Zada, Eliav Buchnik et al. · Nature Neuroscience · 2022
Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.
Cited by 443 works. doi:10.1038/s41593-022-01026-4 Open-access PDF
Authors
- Ariel Goldstein
- Zaid Zada
- Eliav Buchnik
- Mariano Schain
- Amy Rose Price
- Bobbi Aubrey
- Samuel A. Nastase
- Amir Feder
- Dotan Emanuel
- Alon Cohen
- Aren Jansen
- Harshvardhan Gazula
- Gina Choe
- Catherine Kim
- Colton Casto
- Lora Fanda
- Werner Doyle
- Daniel J. Friedman
- Patricia C. Dugan
References
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
- Glove: Global Vectors for Word Representation
- ASPECTS OF THE THEORY OF SYNTAX
- Syntactic Structures
- “Cloze Procedure”: A New Tool for Measuring Readability
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
- FieldTrip: Open Source Software for Advanced Analysis of MEG, EEG, and Invasive Electrophysiological Data
- Reading Senseless Sentences: Brain Potentials Reflect Semantic Incongruity
- Receiver Operating Characteristic Curve in Diagnostic Test Assessment
- Whatever next? Predictive brains, situated agents, and the future of cognitive science