Embedded vision for driver safety on Ambarella CV25

What was the project?

• We worked with a manufacturer of an aftermarket compact driver monitoring (DMS) camera.
• The goal was to provide real-time signals related to head position and facial landmarks as a basis for attentiveness and drowsiness detection.
• The algorithms needed to run on Ambarella CV25 SoC, using Ambarella’s inference accelerators (DPU).
• In-cabin camera setup included the use of a near-infrared sensor and several different positions of the camera.

What did we deliver?

• In order to optimize our vision model for the use case, we applied transfer learning with specific in-cabin NIR images, which we collected and annotated ourselves.
• Quantization-aware training was required to prepare the model for the fixed-point architecture of Ambarella’s inference engine.
• We achieved high accuracy on the NIR real-time stream up to full-HD resolution, exceeding customer expectations and matching state-of-the-art performance.
• The total size (code and data) was less than 13MB.
• The CPU usage with the frame resolution of 640×480 on Ambarella CV25 (Quad-core 1 GHz Arm®Cortex®A53) was only 26%.

What was the secret ingredient?

To achieve the required performance, we needed to handle all specifics of the use case proactively. So, here’s what we did.

• We collected a dataset using NIR and RGB cameras from three in-cabin positions with carefully designed scenarios and annotated the data set.
• Then, we developed image preparation and processing steps needed to implement our algorithms on Ambarella CV25.
• Finally, transfer learning and model quantization were necessary to optimize the model for the selected SoC.
• Integration with Ambarella’s inference engine was efficient due to a previously in-house-built inference engine wrapper (ViNNIE), which unifies different inference engines with a simple API.