Connect with us

Computer Graphics

3D Streaming Gets Leaner: Predicting Visible Content for Immersive Experiences

A new approach to streaming technology may significantly improve how users experience virtual reality and augmented reality environments, according to a new study. The research describes a method for directly predicting visible content in immersive 3D environments, potentially reducing bandwidth requirements by up to 7-fold while maintaining visual quality.

Avatar photo

Published

on

A groundbreaking new approach to 3D streaming has emerged, poised to revolutionize how users experience virtual reality (VR) and augmented reality (AR) environments. Researchers at NYU Tandon School of Engineering have developed a method for directly predicting visible content in immersive 3D environments, potentially reducing bandwidth requirements by up to 7-fold while maintaining visual quality.

This innovative technology addresses the fundamental challenge of streaming immersive content: the massive amount of data required to render high-quality 3D experiences. Traditional video streaming sends everything within a frame, but this new approach is more like having your eyes follow you around a room – it only processes what you’re actually looking at.

The system works by dividing 3D space into “cells” and treating each cell as a node in a graph network. It uses transformer-based graph neural networks to capture spatial relationships between neighboring cells, and recurrent neural networks to analyze how visibility patterns evolve over time. This approach reduces error accumulation and improves prediction accuracy, allowing the system to predict what will be visible for a user 2-5 seconds ahead – a significant improvement over previous systems that could only accurately predict a user’s field of view (FoV) a fraction of a second ahead.

The research team’s approach has been applied in an ongoing project to bring point cloud video to dance education, making 3D dance instruction streamable on standard devices with lower bandwidth requirements. This technology has the potential to transform the way people experience immersive content, enabling more responsive AR/VR experiences with reduced data usage and allowing developers to create more complex environments without requiring ultra-fast internet connections.

“We’re seeing a transition where AR/VR is moving from specialized applications to consumer entertainment and everyday productivity tools,” said Yong Liu, professor in the Electrical and Computer Engineering Department at NYU Tandon. “Bandwidth has been a constraint. This research helps address that limitation.”

Computer Graphics

Unlocking Next-Generation Particle Physics Experiments with Quantum Sensors

Researchers have developed a novel high-energy particle detection instrumentation approach that leverages the power of quantum sensors — devices capable of precisely detecting single particles.

Avatar photo

Published

on

By

The quest to understand the fundamental nature of matter, energy, space, and time has led physicists to create powerful particle accelerators that collide high-energy particles at incredible speeds. These collisions produce a massive number of subatomic particles per second, making it challenging for researchers to detect and analyze them accurately.

To overcome this challenge, scientists have developed quantum sensors, specifically designed to precisely detect single particles. Researchers from the Fermi National Accelerator Laboratory (Fermilab), Caltech, NASA’s Jet Propulsion Laboratory (JPL), and other collaborating institutions have successfully tested these novel high-energy particle detection instruments at Fermilab.

The research team, led by Maria Spiropulu, used superconducting microwire single-photon detectors (SMSPDs) to detect charged particles for the first time. These sensors can precisely track particles in both space and time, achieving better spatial and time resolution simultaneously.

According to Si Xie, a scientist at Fermilab, “This is just the beginning. We have the potential to detect particles lower in mass than we could before as well as exotic particles like those that may constitute dark matter.” The quantum sensors used in this study are similar to superconducting nanowire single-photon detectors (SNSPDs), which have applications in quantum networks and astronomy experiments.

The researchers demonstrated that the SMSPD sensors were highly efficient at detecting high-energy beams of protons, electrons, and pions. This breakthrough has significant implications for future particle physics experiments, such as those planned for the Future Circular Collider or a muon collider.

“We are very excited to work on cutting-edge detector R&D like SMSPDs because they may play a vital role in capstone projects in the field,” said Fermilab scientist and Caltech alumnus Cristián Peña. The study, titled “High energy particle detection with large area superconducting microwire array,” was funded by the US Department of Energy, Fermilab, the National Agency for Research and Development (ANID) in Chile, and the Federico Santa María Technical University.

The success of this research has paved the way for further advancements in particle physics experiments, utilizing quantum sensors to optimize next-generation searches for new particles and dark matter.

Continue Reading

Artificial Intelligence

Engineering a Robot that Can Leap Like a Nematode

Inspired by the movements of a tiny parasitic worm, engineers have created a 5-inch soft robot that can jump as high as a basketball hoop. Their device, a silicone rod with a carbon-fiber spine, can leap 10 feet high even though it doesn’t have legs. The researchers made it after watching high-speed video of nematodes pinching themselves into odd shapes to fling themselves forward and backward.

Avatar photo

Published

on

By

The tiny parasitic worm, nematode, has long been a subject of fascination for scientists. These creatures can jump as high as 20 times their body length, which is an incredible feat considering they don’t have legs. Inspired by this remarkable ability, researchers at Georgia Tech have created a soft robot that can leap 10 feet high without any legs.

The robot’s design is based on the unique way nematodes move. They can bend their bodies into different shapes to propel themselves forward and backward. By watching high-speed videos of these creatures, the researchers were able to develop simulations of their jumping behavior. This led them to create soft robots that could replicate the leaping worms’ movement.

The key to the robot’s success lies in its ability to store energy when it kinks its body. This stored energy is then rapidly released to propel the robot forward or backward. The researchers found that by reinforcing the robot with carbon fibers, they could accelerate the jumps and make them more efficient.

This breakthrough has significant implications for robotics and engineering. With the ability to create simple elastic systems made of carbon fiber or other materials, engineers can design robots that can hop across various terrain. This technology could be used in search and rescue missions where robots need to traverse unpredictable terrain and obstacles.

Lead researcher Sunny Kumar said, “We’re not aware of any other organism at this tiny scale that can efficiently leap in both directions at the same height.” The researchers are continuing to study the unique way nematodes use their bodies to move and build robots to mimic them. This research has the potential to lead to innovative solutions for robotics and engineering.

Associate Professor Saad Bhamla’s lab collaborated on this project with researchers from the University of California, Berkeley, and the University of California, Riverside. The study was published in Science Robotics on April 23.

Continue Reading

Brain Injury

Unlocking the Secrets of the Brain with Digital Twins

In a new study, researchers created an AI model of the mouse visual cortex that predicts neuronal responses to visual images.

Avatar photo

Published

on

By

The researchers used a combination of AI and neuroscience techniques to create the digital twin. They first recorded the brain activity of real mice as they watched movies made for people, which provided a realistic representation of what the mice might see in natural settings. The films were action-packed and had a lot of movement, which strongly activated the visual system of the mice.

The researchers then used this data to train a core model, which could be customized into a digital twin of any individual mouse with additional training. These digital twins were able to closely simulate the neural activity of their biological counterparts in response to a variety of new visual stimuli, including videos and static images.

The large quantity of aggregated training data was key to the success of the digital twins, allowing them to make accurate predictions about the brain’s response to new situations. The researchers verified these predictions against high-resolution imaging of the mouse visual cortex, which provided unprecedented detail.

This technology has significant implications for the field of neuroscience. By creating a digital twin of the mouse brain, scientists can perform experiments on a realistic simulation of the brain, allowing them to gain insights into how the brain processes information and the principles of intelligence.

The researchers plan to extend their modeling into other brain areas and to animals, including primates, with more advanced cognitive capabilities. This could ultimately lead to the creation of digital twins of at least parts of the human brain, which would be a major breakthrough in the field of neuroscience.

Content:

Unlocking the Secrets of the Brain with Digital Twins

A group of researchers have created a digital twin of the mouse brain, which can predict the response of tens of thousands of neurons to new videos and images. This AI model has been trained on large datasets of brain activity collected from real mice watching movie clips.

The digital twin is an example of a foundation model, capable of learning from large datasets and applying that knowledge to new tasks and new types of data. This technology has the potential to revolutionize the field of neuroscience, allowing scientists to perform experiments on a realistic simulation of the mouse brain and gaining insights into how the brain processes information.

The researchers used a combination of AI and neuroscience techniques to create the digital twin. They first recorded the brain activity of real mice as they watched movies made for people, which provided a realistic representation of what the mice might see in natural settings. The films were action-packed and had a lot of movement, which strongly activated the visual system of the mice.

The researchers then used this data to train a core model, which could be customized into a digital twin of any individual mouse with additional training. These digital twins were able to closely simulate the neural activity of their biological counterparts in response to a variety of new visual stimuli, including videos and static images.

The large quantity of aggregated training data was key to the success of the digital twins, allowing them to make accurate predictions about the brain’s response to new situations. The researchers verified these predictions against high-resolution imaging of the mouse visual cortex, which provided unprecedented detail.

This technology has significant implications for the field of neuroscience. By creating a digital twin of the mouse brain, scientists can perform experiments on a realistic simulation of the brain, allowing them to gain insights into how the brain processes information and the principles of intelligence.

The researchers plan to extend their modeling into other brain areas and to animals, including primates, with more advanced cognitive capabilities. This could ultimately lead to the creation of digital twins of at least parts of the human brain, which would be a major breakthrough in the field of neuroscience.

Funding:

The study received funding from the Intelligence Advanced Research Projects Activity, a National Science Foundation NeuroNex grant, the National Institute of Mental Health, the National Institute of Neurological Disorders and Stroke (grant U19MH114830), the National Eye Institute (grant R01 EY026927 and Core Grant for Vision Research T32-EY-002520-37), the European Research Council and the Deutsche Forschungsgemeinschaft.

Continue Reading

Trending