"Augmented TV" is a new image representation technology which creates the appearance of connecting the onscreen world, captured by the TV camera, to the real world in front of the TV screen. The user can enjoy effects such as TV characters leaping out from the screen. We developed an AR (augmented reality) system for TV video images, a versatile technology which is characterized by the smoothness of the “leaping out” representation. The TV screen is viewed with a camera built into a mobile device, on which 3DCG animation is superimposed at the calculated position of TV video images captured by the camera. The animation is synchronized with the TV video images, enabling characters to seem to leap out from the screen. We developed a synchronization method for the system that superimposes a moving marker showing the playback time on TV video images; the marker is captured by the mobile device camera and used to synchronize the 3DCG animation. We achieved a synchronization error time of about 0.03 seconds, allowing frame-accurate synchronization at 30fps. We also developed a method for estimating the position and rotation of the TV using image processing and mobile device gyro sensors, and achieved a highly flexible representation in which the character leaping out can go right around the user. Communication between the TV and the mobile device is not necessary, making the system highly versatile and applicable to areas such as digital signage.
Movies
【Exhibition content】
“If only characters on TV could leap out into our living rooms…” Haven’t we all dreamed about this when we were children? A new technology called Augmented TV is working to make this dream a reality. You can watch characters leaping out from the TV screen through the camera on a tablet by using AR (augmented reality) technology. In our booth, the characters are ancient animals introduced in NHK nature programs.
