Avoiding errors and maintaining quality when adopting AI

AI is already delivering value across the media industry enabling media companies and broadcasters to produce, manage and distribute content much more efficiently. From speeding up post-production editing through automating time-consuming tasks, to enhancing live production with automated camera tracking and real-time QC, through to improving efficiency in content management with automated tagging at scale using object and scene detection, AI is being adopted across a wide range of use cases. It’s also being used by video services providers to help deliver a more engaging and personalized user experience, and to enhance quality and efficiency through using analytics and observability tools to drive real-time service adjustments.

Yet without the right guardrails in place, AI adoption can lead to errors and undesirable consequences. At the steep end of the scale, it can also lead to potential model breakdown where AI is checking and learning from AI, leading to unreliable and false or made-up outputs, also known as hallucinations. With all this in mind, what can video service providers do to avoid errors, maintain quality, and extract maximum value from AI?

Human-in-the-loop design approach

AI clearly enables speed and scale, but to ensure accuracy, it’s vital that human oversight is maintained at critical decision points. What this ‘human-in-the-loop’ approach means in practice, in terms of the level of human input and the point at which it occurs, will depend on the nature of the process or task in question.

For low risk, repetitive tasks that follow a set pattern with little deviation, AI may be able operate almost autonomously once outputs are verified, whereas other tasks may require human oversight and approval for critical decisions. More complex tasks may need human-AI collaboration, with the strengths of each being leveraged, and the most nuanced, complicated tasks will rely on AI for recommendations and analysis but need humans to make the final decisions.

No matter the level of complexity, what is critical here is that there is human involvement at some point in the process, usually at the beginning and the end. Even with the simplest tasks that can be largely automated with minimal human input, a human still needs to design the process, set expected outputs, then review outputs before the process can be AI-automated.

Incremental implementation

Alongside a human-in-the-loop approach, it’s also essential that AI is implemented into a workflow incrementally rather than introducing masses of AI-enabled processes all at the same time. This allows for controlled testing and validation, and prevents small errors from turning into major service wide failures. An incremental approach allows video services to identify any mistakes in outputs, and then isolate and correct before rolling out AI in additional workflows or processes.

If video services go for a big bang approach where all human led processes are replaced by AI all at the same time, you can end up in a system where AI is checking AI. Without human validation, this will quickly lead to bad outputs and hallucinations which is obviously what needs to be avoided. For example, if a video service software engineering team starts using AI generated coding to drive a new UI feature, and then at the same time also uses AI-driven quality assurance to validate said coding and outputs, this does not allow for any human led testing and validation to ensure outputs are correct. That way is the fast road to errorsville. A phased, measured approach on the other hand allows video providers to verify accuracy of outputs within an isolated process before expanding to implement additional layers of AI.

Fine tuning and iteration

Initial usage of foundational AI models in any new workflow is likely to generate errors and inaccuracies. AI models are powerful and smart, but they still have limitations such as a lack of knowledge past the date when they were trained, as well as knowledge gaps that result in false outputs in response to complex queries, and potential bias because of initial training data. Additionally, AI models also need to be customized so they preform optimally for a specific use case.

Advertisement

Fine tuning helps to overcome these limitations and improve and control the AI output, so it is optimized for a specific environment and workflow. Video providers therefore need to fine tune and calibrate AI models to meet their specific needs and then validate output and reiterate to reduce the error rates. This process can be used to improve the quality of outputs until an acceptable level of error rates is achieved.
Final thoughts

AI implementation shouldn’t be all or nothing but instead should start small, then gradually increase as reliability is proved and the desired results are achieved. By taking a phased, measured approach, with a human-in-the-loop design, video services can stay in control of their AI implementation while scaling up operational impact.

AI models are improving all the time, and we’re seeing major leaps in quality of output and their ability to be more accurate. Additionally, as models improve, they’re also becoming less prone to generating bad output when they receive a less-than-ideal prompt. As models continue to improve, video service providers must refine how and where they apply AI, and learn through experience how to shape it around their workflows so it delivers meaningful value rather than added complexity and errors.

Michael Chan, AccedoMichael Chan is VP of delivery and operations for Accedo.