Hand gesture recognition is an increasingly practical form of Human-Machine Interaction (HMI) achievable through a range of technologies, including wearable sensors, radar, time-of-flight, and/or vision systems. RGB cameras offer perhaps the greatest potential for recognizing the widest range of intricate, subtle, and ambiguous gestures, though the sheer volume of high-dimensional data they provide makes gesture interpretation an exceptionally difficult problem. Many image processing techniques have been used to detect hand gestures from said data, but recent advances in AI have made the task far more feasible. Furthermore, the introduction of edge devices capable of running more and more complex models has made it possible to utilize hand gestures as an input mechanism for everyday devices like thermostats, infotainment systems, vending machines, etc.
The Game Controller Hardware
To demonstrate how effectively AI preprocessing can enable vision-based hand gesture recognition using RGB cameras, we put together an interactive demo (Figure 1) in which multiple games running on a Raspberry Pi can be launched and controlled using hand gestures. The game controller is based on the STM32N6 microcontroller (MCU) from STMicroelectronics, which is their first product featuring a Neural Processing Unit (NPU). It enables larger AI models, such as those for computer vision and audio processing, to be run at the edge in real time. The resulting applications benefit from lower latency, enhanced privacy, and reduced hardware/bandwidth costs compared to those which rely on remote servers for AI inference. Top-level highlights of the MCU include:
- Arm Cortex-M55 with a 800 MHz max operating frequency, Arm MVE (M-Profile vector extension), and an FPU
- ST Neural-ART Accelerator clocked at 1 GHz providing 600 Gops and 288 MAC/cycle
- 4.2 MB SRAM plus several high-speed external memory interfaces
- Neo-Chrome 2.5D GPU plus other peripherals supporting rich graphics
- A video pipeline including CSI-2 camera interfaces, ISP, and H264 encoding acceleration
One of the easiest ways to get started with the STM32N6 is the STM32N6570-DK discovery kit. The board includes several development aids, including a 5" touch LCD, camera module, external memories, digital microphone, and board connectors (USB Type-C, Ethernet, microSD card, Arduino expansion headers, etc.). As shown in Figure 2, it is integrated directly into the demo to showcase many of these onboard features as well as the advanced capabilities of the STM32N6.
The custom gaming application designed for this demo could be run on any Linux-based system, but a Raspberry Pi was used here for convenience. This application receives game controls from the STM32N6570-DK board over USB and reports state information back to the controller to indicate what is currently running (e.g. Main Menu, Game 1, Game 2, etc.). USB Type-C to Type-C cables and a suitable power supply are used to power both the Raspberry Pi and the STM32N6570-DK board to ensure their supply requirements are met.
The Game Controller Application
As part of their support for the STM32N6570-DK board, ST provides several example AI models which perform tasks like object detection, segmentation, and pose estimation. This demo is based on the x-cube-n6-ai-hand-landmarks application, which estimates hand landmarks using two cascaded models derived from the MediaPipe Hand Landmarker. The first model in the pipeline is a palm detector, which takes the entire image from the camera as input and searches for a hand. If a hand is found, the image is cropped to isolate that region and fed into the second model, which identifies the hand landmarks. Inference is performed on the STM32N6 in real time, allowing the pipeline to maintain a frame rate of 30 FPS. Figure 3 shows the 21 hand landmarks which are detected by this model and how they are enumerated. This final output is often referred to as skeleton data.
Figure 3: The 21 enumerated hand landmarks returned by the 2-stage AI pipeline. Source: https://developers.google.com/edge/mediapipe/solutions/vision/hand_landmarker
Skeleton data is the distillation of high-dimensional image data into a low-dimensional set of points representing joints or other body features. This demo project directly uses the 21 data points produced by the reference application’s computer vision pipeline to recognize gestures via rule-based decision logic. For example, consider the pseudo-code shown in Listing 1 used to determine whether the “#1” gesture (Figure 4) is being made. It simply compares the x and y coordinates of each finger’s joints to see if the index finger is pointing straight up and the other fingers are not. The complete code used to detect this and other supported gestures can be found in the app.c file in the game controller source code.
Listing 1: Pseudo code for detecting “#1” 
gesture using skeleton data.
FUNCTION check_num1_gesture(landmark)
// If the thumb is bent inward (works for left and right hands)
IF abs(landmark[4].x - landmark[17].x) < abs(landmark[3].x - landmark[17].x)
// If the index finger is not tilted either left or right by more than 33.7 degrees
IF (landmark[8].y - landmark[5].y) > (abs(landmark[8].x - landmark[5].x) * 1.5)
// If the index finger is pointed up
IF (landmark[6].y > landmark[5].y) AND (landmark[7].y > landmark[6].y) AND (landmark[8].y > landmark[7].y)
// If the middle finger is not pointed up
IF (landmark[10].y < landmark[9].y) OR (landmark[11].y < landmark[10].y) OR (landmark[12].y < landmark[11].y)
// If the ring finger is not pointed up
IF (landmark[14].y < landmark[13].y) OR (landmark[15].y < landmark[14].y) OR (landmark[16].y < landmark[15].y)
// If the pinky finger is not pointed up
IF (landmark[18].y < landmark[17].y) OR (landmark[19].y < landmark[18].y) OR (landmark[20].y < landmark[19].y)
RETURN True
RETURN False
END FUNCTION
This straightforward approach to gesture recognition offers several advantages including being simple to implement, efficient to execute, and deterministic in how it behaves. It’s an acceptable approach for an application as simple as ours, which only considers a handful of gestures. However, hand-crafting the features for each gesture is an error-prone process and the result often behaves poorly when presented with edge cases. For instance, while the pseudo-code in Listing 1 works well for recognizing the #1 gesture when presented with a palm-facing view, it would likely fail to do so if the user were to present a side view. Furthermore, scaling the application by adding more gestures becomes quite tedious and the code base quickly becomes difficult to manage. A more sophisticated approach would be to train a deep learning model on a curated dataset of skeleton data representing the desired gestures and deploy it as a third model in the image processing pipeline. Of course, doing so would require additional development and implementation effort, with no guarantee the resulting model performs any better.
To avoid comparing each frame’s detected hand landmarks against the entire set of known gestures, the gaming application on the Raspberry Pi communicates its current state to the controller application on the STM32N6570-DK. That way, the controller only tests for gestures relevant to that state. For instance, when Game 1 is being played, the controller will only test for the gestures used to control Game 1.
Referring back to Figure 2, we see that the game controls and the state information are exchanged over the same USB interface. This was achieved by adding and configuring the USB Device middleware so the USB1 port on the discovery kit enumerates as a USB composite device. That is, it combines two USB Device classes: the HID (Human Interface Device) class to function as an HID keyboard and the CDC (Communications Device Class) class to provide a virtual COM port. The HID keyboard controls the gaming application by sending key press events in response to recognized gestures. For example, to select Game 1 from the menu screen, the user makes the “#1” gesture, prompting the STM32N6 to send a ‘1’ key press event. The virtual COM port enables the exchange of serial messages between the boards, which the gaming application uses to transmit state information to the controller (e.g., “state:0”, “state:1”).
At the time of writing, the x-cube-n6-ai-hand-landmarks application used as the starting point for this demo runs FreeRTOS whereas the middleware components provided in ST’s STM32CubeN6 package are based on Eclipse ThreadX RTOS (formerly Azure RTOS). Therefore, ST’s classic core middleware package was used to add the USB Device middleware to the controller application as it does not rely on ThreadX. It supports several USB Device classes, but only the HID, CDC, and CompositeBuilder classes were utilized for the project. After the middleware files were added, the HID descriptors were modified to define the device as a keyboard (based on the tutorial in this ST community post), the CDC interface was configured to parse the state messages from the gaming application, and the CompositeBuilder class was tailored to combine the HID and CDC classes into a single composite device.
The complete source code for the STM32N6570-DK game controller application can be found in its GitHub repository. See the commit history to review all of the changes made to the starting application.
The Game Launcher and Games
While the focus of this demo is on the STM32N6570-DK and its AI acceleration capabilities, a brief overview of the custom gaming application is provided here for the sake of completeness. The game launcher is simply a Python script using the PyGame library to create the GUI and handle keypress events. The games themselves are launched in separate web browser windows and the script brings them to the foreground as they are selected from the main menu. Currently, the launcher menu appears as shown in Figure 5(a) and supports two games: Zig Zag and Fishy. Instructions for each game are shown to the user before gameplay begins (Figures 5(b) and 5(c)).
Figure 5: (a) the Main Menu screen, (b) the instructions for Game 1 (Zig Zag), (c) the instructions for Game 2 (Fishy)
For reference, the gaming application files are available in its GitHub repository along with instructions for running the demo on a Raspberry Pi.
Conclusions and Future Work
As provided, this project demonstrates how the STM32N6 line of MCUs can run increasingly capable AI models, making it practical to deploy advanced applications at the edge. Of course, there are still opportunities for improvement and expansion. As mentioned previously, a third model trained on skeleton data could be added to the pipeline to predict hand gestures rather than rely on hand-crafted features. Assuming this results in the controller recognizing a wider variety of gestures more reliably, additional games could easily be added. Beyond gaming, the application could be tailored to enable gesture recognition for practical applications such as robot control or drone navigation. The performance offered by the STM32N6 allows plenty of room to explore what else can be done at the edge.