AI tool generates video from brain activity

“Alexa, play back that dream I had about Kirsten last week.” That’s a command that may not be too far off in the future, as researchers close in on technology that can tap into our minds and retrieve the imagery of our thoughts.

Researchers at the National University of Singapore and the Chinese University of Hong Kong reported last week that they have developed a process capable of generating video from brain scans. The research is published on the arXiv preprint server.

Using a process called functional magnetic resonance imaging (fMRI), researchers Jiaxin Qing, Zijiao Chen and Juan Helen Zhou coupled data retrieved through imaging with the deep learning model Stable Diffusion to create smooth, high quality videos.

Successful recreations of still imagery gleaned from brain activity through AI-assisted Stable Diffusion, commonly used for image generation, have been reported in recent months. But, as Qing and his team reported, retrieving continuous visual images poses a particular challenge.

The process of fMRI generally taps blood oxygenation level-dependent (BOLD) signals and captures images of brain activity once every few seconds. This would yield poor-quality video images. A standard video capture rate is 30 frames per second.

“Understanding the information hidden within our complex brain activities is a big puzzle in cognitive neuroscience,” Qing said. “The task of recreating human vision from brain recordings, especially using non-invasive tools like fMRI, is an exciting but difficult task.”

His team achieved high-caliber video with their Mind-Video model. Described as “a two-module pipeline designed to bridge the gap between image and video brain decoding,” its fMRI decoder progressively learns from acquired brain signals, training with image databases and fine tuning.

The results were high-quality videos, they said, complete with motion and scene dynamics at an accuracy rate of 85%.

Qing says their work shows promise for future large model applications “from neuroscience to brain-computer interfaces.”

The use of AI along with MRI and EGM (electromyogram) to study imagery, brain activity and muscle movement is opening new vistas into the workings of the mind. Shinji Nishimoto, a neuroscientist at Osaka University, says these processes may one day be used to capture thoughts and dreams.

Dream researcher Daniel Oldis, working with a colleague at the Cognitive Neuroscience Lab at the University of Texas, Austin, is working on dream-recording MRI technology that tracks brain activity and nerve impulses in muscles to define images, speech and movement in dreams.

“This is like the early years of the space race,” he said recently. “But in this case, we’re going into the dream space.”

The average person has up to six dreams a night, but forgets 90% of them within minutes after awakening. The prospect of capturing a permanent record of dreams is exhilarating.

Dreams have mystified us for centuries. In early cultures, dreams were believed to be messages from God. In the 20th century, Freud introduced the idea of repressed sexual and emotional impulses behind our dreams.

Some of our greatest inspirations come from those nocturnal journeys we all take each night of our lives.

The melody to the Beatles’ “Yesterday,” named the best song of the 20th century by a BBC Radio poll, came to Paul McCartney in a dream. The famous three-note opening riff on the Rolling Stones’ giant hit “Satisfaction” came to Keith Richards in the middle of the night; he awoke to record a rough version of the song, sleepily mumbled “I can’t get no satisfaction,” went back to sleep, and only realized what he had jotted down when he listened to the tape that morning.

Salvator Dali referred to fantastical surrealist works such as his “The Persistence of Memory” as “hand-painted dream photographs.”

And a nervous student fearful that his admission to Stanford University was an error had an anxiety dream in which he downloaded the entire web onto his computer. When he awakened, he embarked on a project that one day would draw 89 billion viewers per month and answer 99,000 inquiries per second. His name: Larry Page. His creation: Google.

Recording and streaming our dreams—from nightmares to game-changing inspirational ideas—may be only a dream right now.

But as the works by Qing, Oldis and others indicate, not for long.

© 2023 Science X Network