Enhancing manual HDMI camera inspection of medical devices through the integration of augmented reality graphics and machine vision systems

In medical device manufacture, manual inspection and adjustment of submillimetre components is often done using HDMI-based vision cameras. This digital enhancement provides a visual aid to operators in quality inspection. While machine vision systems can be used to further aid operators in the inspection process, these systems are expensive to retrofit, and require operator retraining, system integration, and data management. In this present work, an investigation is conducted into integrating machine vision technology with HDMI cameras to improve operator inspection and adjustment. However, HDMI is not an industrial machine vision communication interface standard.

Here we show the development of a machine vision system with a HDMI camera imaging source and capabilities of displaying real time augmented reality graphics which aid an operator with visual inspection. This system was developed through iterations of three phases of experiments, achieving increasing levels of sophistication and functionality at each stage. In Phase 1 a standard machine vision system using GigE interface was built to develop and test a vision application capable of carrying out the inspection. In Phase 2 HDMI video was captured and successfully converted for the machine vision software to process. For Phase 3 the graphics created in the machine vision software were extracted and placed as augmented reality operator visual aids over a HDMI live video on a UHD monitor.

The results showed this process of converting HDMI does incur lag between real time and displayed image. For phase 2 displaying the processed image and associated graphics together would function adequately for static image application, however when using this as a system to guide an operator in real time, likelihood of damage to the inspected product would occur. In phase 3 it was observed that extracting these machine vision graphics from the processed digital image and placing them over the existing real time HDMI video negated the incurred lag as the real time video was synchronised with the operator reactions.

For a selected case study involving enhancing five inspection stations with augmented reality guidance, cost savings of €33,600 per machine is estimated. Further developments using this method of integrating HDMI equipment are encouraging. At present there are numerous HDMI inspection devices at manual visual inspection stations in every manufacturing site in the world and vital image data required for future development of machine automation systems is being lost. The findings from this body of work show that this image data can be harnessed at relatively low cost and disruption to the process.