Project Starline: A high-fidelity telepresence system
Jason Lawrence
(Google)
Please LOG IN to view the video.
Date: January 12, 2022
Description:
We present a real-time bidirectional communication system that lets two people, separated by distance, experience a face-to-face conversation as if they were copresent. It is the first telepresence system that is demonstrably better than 2D videoconferencing, as measured using participant ratings (e.g., presence, attentiveness, reaction-gauging, engagement), meeting recall, and observed nonverbal behaviors (e.g., head nods, eyebrow movements). This milestone is reached by maximizing audiovisual fidelity and the sense of copresence in all design elements, including physical layout, lighting, face tracking, multi-view capture, microphone array, multi-stream compression, loudspeaker output, and lenticular display. Our system achieves key 3D audiovisual cues (stereopsis, motion parallax, and spatialized audio) and enables the full range of communication cues (eye contact, hand gestures, and body language), yet does not require special glasses or body-worn microphones/headphones. The system consists of a head-tracked autostereoscopic display, high-resolution 3D capture and rendering subsystems, and network transmission using compressed color and depth video streams. Other contributions include a novel image-based geometry fusion algorithm, free-space dereverberation, and talker localization.
Further Information:
Jason is a research scientist at Google in Seattle. He co-founded and currently leads the research and engineering team behind Project Starline (video). Prior to joining Google, he was associate professor of computer science at the University of Virginia. He holds a PhD in computer science from Princeton University and a BS in electrical engineering from Carnegie Mellon University.
Jason’s research interests span computer graphics, computer vision, and machine learning. He has worked on a wide range of topics including physically-based rendering, real-time rendering, material appearance modeling and representation, computational fabrication, and systems for acquiring dense accurate measurements of 3D geometry and material appearance. His recent work includes high-fidelity real-time 3D capture, 3D display technologies, digital relighting, and real-time communications.
Jason’s research interests span computer graphics, computer vision, and machine learning. He has worked on a wide range of topics including physically-based rendering, real-time rendering, material appearance modeling and representation, computational fabrication, and systems for acquiring dense accurate measurements of 3D geometry and material appearance. His recent work includes high-fidelity real-time 3D capture, 3D display technologies, digital relighting, and real-time communications.
Created: Wednesday, January 12th, 2022