A new transformer architecture emulates imagination and higher-level human mental states

A research team has developed a novel transformer architecture named Co4, designed to emulate human-like imagination and higher-level mental states. Drawing inspiration from neuroscience, Co4 incorporates triadic modulation loops among queries, keys, and values, mirroring interactions between external sensory inputs and internal contextual information in the brain. This design enables the model to perform parallel, multi-perspective reasoning akin to the brain’s cortico-thalamic modulation, facilitating high-level perceptual processing and imaginative thought.

Unlike traditional transformer models that rely on sequential token-level processing, Co4 operates at the representation level, allowing for more efficient and flexible reasoning. By integrating mechanisms analogous to those found in layer 5 pyramidal neurons, the architecture supports transitions between different mental states, such as wakefulness and sleep, enhancing its ability to model complex cognitive functions. This advancement represents a significant step toward AI systems that more closely replicate human cognitive processes.