Industry Insights: Virtual and AR experts open up about new realities

As we navigate a year full with major sporting events and significant elections worldwide, integrating augmented reality (AR) and virtual production into broadcasting opens new avenues of storytelling. 

In this roundtable discussion, part of our Industry Insights series, we gather insights from leading broadcast vendors and suppliers to explore how these technologies enhance viewer engagement and bring new dimensions to broadcast content.

Industry Insights: Virtual production professionals talk challenges, advancements

From overlaying dynamic 3D graphics to creating immersive virtual environments, AR and virtual production offer innovative solutions for presenting complex data and enriching the viewing experience. This first installment delves into how these advancements are reshaping the industry, with real-world applications in sports and election coverage.

Key takeaways from the Industry Insights roundtable

• Enhanced Visual Storytelling: AR and virtual production provide broadcasters with tools to create dynamic and interactive visual stories, making complex information more accessible and engaging for viewers.
• Increased Viewer Engagement: AR applications, such as virtual set extensions and immersive graphics, can draw viewers into the broadcast, creating memorable and impactful experiences.
• Real-Time Data Visualization: Broadcasters use AR to present real-time data in visually compelling ways, enhancing the understanding of complex information during live events like elections and sports.
• New Advertising Opportunities: AR enables broadcasters to integrate advertising seamlessly within the content, creating new revenue streams while maintaining viewer engagement.
• Cost-Effective Production: Virtual production techniques offer cost savings by allowing multiple environments to be created within a studio, reducing the need for location shoots.

How is augmented reality changing how broadcasters engage with their audiences?

Martin Klampferer, product owner and R&D manager, Vizrt: The ability to overlay 3D virtual graphics and information onto real-world images or video is a powerful way to enhance visual storytelling. AR graphics have numerous applications for broadcasters in news and entertainment and are an expected part of modern sports productions. They can be highly dynamic and interactive to help explain complex stories to a large audience and increase viewer engagement.

Dan Pack, managing director, Silver Spoon: When executed effectively, AR acts like a masterful magic trick or seamless visual effects. It creates the illusion that the graphic is a tangible part of the scene — whether it’s a virtual set extension making a small studio feel expansive, or a giant Yeti showering snow over a crowd at SoFi Stadium. AR has the power to transform ordinary broadcasts into extraordinary experiences, drawing viewers into the narrative in ways that were previously unimaginable.

Robert DeFranco, CEO, Sequin AR: Audiences have a more emotional response and a better recall with AR. Broadcasters are capitalizing on this by using AR to create more memorable moments throughout the entire broadcast. Broadcasters are finding new ways to tell stories through 3D data visualization and demonstrations.

Marcus Brodersen, CEO, Pixotope: By overlaying digital information onto the real world, AR allows broadcasters to present content in a more dynamic way, making storytelling more impactful and memorable. Additionally, AR offers new opportunities for advertising revenue streams by enabling broadcasters to integrate and create new advertising space that exists within the programming, acting as an extension of the production and keeping viewers engaged, rather than a distinctive and separate commercial break.

Advertisement

Ofir Benovici, CEO, Zero Density: Virtual production technologies such as virtual studios, extended reality and augmented reality open brand-new avenues and approaches for broadcasters to engage with their audience, and augmented reality is the easiest to embrace and brings high value when compared to the required overall effort. This is because AR can utilize the already existing physical set by overlaying real-time virtual graphics like 3D objects or visualized data, on top of a video image of a physical set. That makes AR the best option for getting into the virtual production scene and proceeding to more advanced technologies based on positive feedback from the viewers.

Miguel Churruca, marketing and communications director, Brainstorm: Augmented reality allows anchors to use in-context graphics to enhance information and storytelling, making it possible to tell stories or explain concepts by displaying information graphics in a different way. This makes storytelling easier and more engaging, as presenters do not have to rely on pre-recorded videos with a voice over, but allows them to interact with the graphics.

How can graphics enhance the viewer’s understanding of complex data and information?

Chris Izatt, director of innovation and virtual, AE Live: Complex data can be difficult to digest, so it’s important to grab the viewer’s attention and hold onto it long enough for them to fully comprehend the message that is being conveyed. Virtual graphics use camera tracking technology to embed themselves within the real-world physical environment that is captured through the lens which means that the viewer does not need to disengage from the real-world viewing perspective when consuming virtual data graphics. This also means that we can utilize a fully 3-dimensional canvas when designing how complex data is visualized, working with producers, directors, talent, and camera operators to create engaging and immersive sequences for the viewer to consume.

Robert DeFranco: The old axiom “show don’t tell,” are words to live by for complex data. We process pictures twice in our brain, and words only once. Leveraging enhanced graphics pulls in the audience compared to simple talking head data presentations.

Ofir Benovici: Graphics revolutionize how data is interpreted and experienced by the viewers, and also make them accessible and engaging for everyone. Our virtual production solutions transform, for example, complex election data into dynamic, stunning visuals in real-time, integrating with third-party sources to create viewer-friendly graphics that support fact-based discussions and analysis.

Patrick Twomey, director of graphics product management, Ross Video: Great graphics visualizations can turn a casual viewer into an avid fan. The average viewer watching a sporting or news event often won’t know how to interpret complex statistical data and apply it to their understanding, but graphics can bridge that gap, so the viewer feels they understand what they’re watching better. That understanding can take them from just watching their local team to following the whole league because it gives them the confidence to have more enriched, better conversations with their peers about what they’re watching.

Miguel Churruca: Graphics, especially those that are data driven, can display large amounts of data, so it is up to the design department to come up with ways to display them in an attractive, engaging, manner, making such data accessible to the audience. Information graphics are essential to explain complex stories, illustrate concepts or making data rich graphics more accessible; augmented reality helps with making this information more accessible by displaying data interactively and in context, so the anchors can interact with the graphics making the story more compelling.

How are broadcasters leveraging data-rich graphics to create more informative and engaging content?

Adam Callaway, global lead of virtual production and broadcast, Brompton Technology: Whether it be sports or news broadcasting, data-rich graphics are becoming increasingly important for viewer engagement. In a world trending towards short format pieces, graphics can be used to quickly display information on complex subjects and help the viewer easily digest what is being presented. During the upcoming 2024 elections, graphics will play a big part in relaying information from around the country quickly and effectively.

Martin Klampferer: Data-rich graphics are a tool to convey more information in real time. This is helpful for use cases like market data analysis or election coverage and becomes very effective when dealing with real-time data, such as in sports. Data aggregation tools, in combination with modern render engines and dedicated control applications, are used to show statistics and individual player data and enable in-depth analysis of relevant scenes.

Marcus Brodersen: Graphics turn complex data into visual stories that are easier to understand and retain — from weather updates to sports statistics, election results, and more — and with the ability to easily make adjustments live on-air, broadcasters can provide real-time updates to the data to ensure ongoing accuracy. They can highlight trends, compare and analyze data, and illustrate concepts in a straightforward manner, ultimately enhancing comprehension and making information more accessible to a wider audience. For example, we previously assisted the Weather Channel in creating immersive and engaging weather presentations highlighting the impact of climate change.

Patrick Twomey: Ironically, some of the most engaging ways I’ve seen data leveraged are the most straightforward graphics but used very effectively: things like pie charts to display the results of phone-in or online polls, social media sentiment, or just to show live voting results in stacked bar charts as they develop in real-time. The sports world is also developing many interesting ways to bring game statistics to life for viewers, such as scatter charts to show basketball players’ success rates from different court quadrants, which can make data tangible for viewers. What might have taken several hours to produce offline just five years ago now just happens without people noticing and really improves the experience for viewers.

What are the key benefits of incorporating virtual production techniques in broadcast storytelling?

Adam Callaway: Using virtual production within a broadcast environment allows you to immerse your audiences that much more — your presenting team can cover an event happening thousands of miles away and they can interact with dynamic augmented reality elements in the studio environment. Virtual production gives studios the tools and flexibility to go the extra mile.

Dan Pack: Incorporating virtual production techniques in broadcast storytelling unlocks creative and technical possibilities that were previously out of reach, often achieving these results more efficiently. For instance, during the 2023 NBA Finals on ESPN, we utilized virtual production for the opening sequence. By shooting an actor against an LED wall and employing Unreal Engine, we were able to transport the actor through multiple environments in a single shoot day. With set extensions, we placed him inside a stadium, a trophy hall, and a showcase room—all captured directly in-camera.

Advertisement

Robert DeFranco: Virtual production allows broadcasters to do more with less, which in this current market helps everyone stretch their budgets. Being able to shoot five locations in one studio in one day is just not possible with traditional production, but can be achieved with virtual production.

Marcus Brodersen: Modern viewers are coming to expect high-quality content from all sources, and incorporating virtual production into broadcast storytelling opens up a world of unparalleled flexibility and creativity, allowing for the design of vast, detailed sets without the constraints of physical locations. It can reduce production costs, enhance visual appeal and enable real-time interaction with virtual elements, making stories more visually stunning and captivating, as seen at the SuperBowl where CBS and our creative partner Silver Spoon used Pixotope for live AR graphics and attracted a record-breaking 123 million viewers. These benefits can also be seen with the utilization of virtual production for advertising, in which a more engaged audience sees a greater ROI.

Ofir Benovici: There are lots of benefits to talk about but maybe the most important of them is “real-time”. Virtual production techniques allow high-quality visual effects and enhanced storytelling to be applied to broadcasts in real-time with small to no delay as every second matters. Other than real-time, we can list important benefits such as cost-effectiveness, efficiency, intuitiveness, endless creativity, live audience engagement and sustainability.

Miguel Churruca: Firstly, virtual production takes storytelling to a completely new level, with more possibilities for integrating anchors and actors in any kind of environment, existing or created, at any time; secondly, this technology saves time but makes production more cost effective, better using the broadcasters resources and anchors’ time, as everything can be done in the studio with no need to send people to remote locations.

What are the potential drawbacks or limitations of relying heavily on augmented reality and virtual environments?

Adam Callaway: Virtual production and AR both require time spent in pre-production, so time-pressured environments such as rolling news coverage may not be able to deploy AR assets as effectively compared to scripted pieces.

Chris Izatt: The design of physical set elements within a green screen studio becomes more challenging as certain materials and shapes must be avoided to ensure high quality keying can be achieved. Disaster recovery planning typically becomes more complex. It can be difficult for talent to adjust to the heavy use of virtual or augmented graphics.

How can broadcasters ensure that the use of these technologies enhances, rather than distracts from, the storytelling?

Adam Callaway: Using high quality virtual production environments and great planning is the best way to ensure these techniques enhance the production rather than distract the audience. This is especially important when planning immersive virtual production environments where you want the audience to believe that the production is on a location and not on a soundstage. If virtual production is not used correctly, or one part of the chain is low quality, then the illusion will be broken.

Dan Pack: The key to ensuring that these technologies enhance rather than distract from storytelling lies in a single principle: purpose. We constantly ask ourselves: Why are we doing this? How does it benefit the viewer? How does it fit into the broadcast? Does it align with the network or brand strategy? Does it elevate the overall broadcast experience? By keeping these questions at the forefront, we ensure that this technology is used thoughtfully and meaningfully, enhancing the storytelling and creating a more engaging and cohesive viewer experience.

Any key considerations given it’s an election year in many regions of the world?

Ofir Benovici: With a record number, about 3.7 billion people are handing out ballots to vote in 70 countries in 2024. Therefore, channels have the opportunity to become the place to be on election nights, with immersive election coverage that brings the drama of democracy to life in exciting and impactful new ways. Our real-time virtual production and real-time motion graphics solutions open up infinite possibilities for election coverage, turning dry facts into fascinating features, and adding dynamic new dimensions to discussion and analysis.

Patrick Twomey: Recently, I’ve seen some exciting things happening with all types of election graphics and data. Things like bar or sliver charts updating in the lower third in real-time seem simple but can keep viewers engaged and feeling part of the story. At the other end of the scale, I recently saw a broadcaster using volumetric scans of political candidates during an analysis breakdown. Every time they brought up a key policy, the candidates who were for it would step forward, and those against it would step back, making for a clear way to communicate a candidate’s position; they were visibly taking a stance on it on the screen.

Miguel Churruca: Elections are about displaying large amounts of complex data in a way that the audience can understand, while trying to build engagement with the viewers. Advanced virtual tools and AR help showing data-rich graphics in context, such as maps, tables, comparisons, and many more, so the anchors can enhance their storytelling by interacting with such graphics in real time. Elections are perfect examples of how journalists and designers combine their creativity and expertise to push the technology forward to provide the audiences with a show that, in essence, is about displaying data for several hours, which is always a challenge!

Industry Insights: Virtual production professionals talk challenges, advancements