Setting up a Single-Board Computer for Computer Vision

[mohnish.nanthakumar]

Introduction

This article will record progress on the second part of the robotic sorting system project: computer vision. The progress on the first part is recorded in Setting up a Single-Board Computer for Robotics. To recall, the project uses a single-board computer (BeagleY-AI, 2820-102991834-ND) to control an industrial robot arm (igus ReBeL 6DOF; 4903-REBEL-6DOF-02-ND) and camera module (Arducam; 4679-B0292-ND). The three are intended to interact together to pick and place electronic components, spread out across a table, into bins. In this post, we’ll focus on the process for implementing computer vision on the Beagle. This article will be divided into three sections, camera control, image processing and machine learning. All prerequisite theory will be quickly explained if necessary.

Part I: Gaining Sight

The first step for getting any sort of computer vision running is having a camera configured. As mentioned before, an Arducam is being used for this. We will connect it to the Beagle with a USB-A cable. For the programming part, we will make use of OpenCV.

Start by creating a virtual environment in the filesystem of the Beagle.

python3 -m venv venv

Activate it.

source venv/bin/activate

Finally, install OpenCV, NumPy, and pillow

python3 -m pip install opencv-python numpy pillow

Now, we’ll create a script to contain our computer vision code. The Arducam Wiki is an excellent resource for getting started.

import cv2

# open video0 for camera capture
cap = cv2.VideoCapture(0)

# Configure camera parameters
cap.set(cv2.CAP_PROP_FOURCC, cv2.VideoWriter_fourcc(*'MJPG')) # specify video codec
cap.set(cv2.CAP_PROP_BRIGHTNESS, 32)
cap.set(cv2.CAP_PROP_CONTRAST, 0)
cap.set(cv2.CAP_PROP_AUTOFOCUS, 0) # turn off autofocus
cap.set(cv2.CAP_PROP_FOCUS,110) # set our own custom focus
cap.set(cv2.CAP_PROP_BUFFERSIZE, 1) # internal buffer will store 1 frame at a time

# set to maximum resolution as given by datasheet
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 3264)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 2448)

while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()
    # Display the resulting frame
    cv2.imshow('frame', frame)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

# When loop exits, release the capture
cap.release()
cv2.destroyAllWindows()

The above program is a simple live camera feed program, and exits when ‘q’ is pressed. It can be run in the activated virtual environment.

python3 camera_capture.py

If issues arise, check that the camera is plugged into the right USB port. Additionally, you can debug your camera with V4L2 with the following package:

apt install v4l2-utils

You can list out all connected V4L2 devices like so:

v4l2-ctl --list-devices

Sample output:

e5010 (platform:fd20000.jpeg-encoder):
        /dev/video0

You can also check the supported video formats and change your code to reflect them:

v4l2-ctl --list-formats -d 0 # 0 is video0

Sample output:

ioctl: VIDIOC_ENUM_FMT
    Type: Video Capture Multiplanar

    [0]: 'JPEG' (JFIF JPEG, compressed)

Now that the camera setup is complete, we can move on to image processing.

Part II: Distinguishing Parts from Paper

In order to make our system somewhat intelligent about where it picks up components, we need to tell it how to recognize electronic components against a background. For our project, we positioned the camera above a table and covered it with white printer paper, so that camera captures would have a clean, white background. As for the component recognition part, we’ll use basic image processing. After playing around with various transformations, smoothing operations, edge-detection algorithms, and more, we determined a sufficiently capable sequence of image processing operations: adaptive thresholding, morphological erosion and contouring.

Thresholding includes converting a grayscale image black-and-white at certain “thresholds” of intensity. Any pixel above the threshold is set to 255 (white) and any below is set to 0 (black). In our project, we use Adaptive Thresholding, where rather than having a constant threshold for the entire image, we have multiple thresholds being calculated for multiple regions within the image. This helps to account for subtle differences in the background in multiple parts of the image. This is good for our use case, because at portions where the papers overlap, there is a different color than in the non-overlapping parts.

Erosion is where the size of the white areas, after thresholding, of course, are reduced. The outer layers of the white shapes that appear after thresholding get stripped and become skinnier. This is especially useful for removing noise and, for our specific application, electronic pins and leads. The inverse of erosion is dilation, where the white areas increase in size from all angles.

Contouring draws outlines around all of these shapes. For every boundary point (points between white and black pixels), a curve is drawn along continuous points. We can draw bounding boxes around these shapes, and, taking the center, we can get the center coordinate to pick up a component at.

Source: https://medium.com/thedeephub/mastering-contouring-in-opencv-a-comprehensive-guide-10e6fe2a069a

To implement these operations in code, OpenCV provides functions that take in the input image and return the output image.

You’ll need to read in frames and convert them to grayscale.

while (True):
    ret, frame = cap.read()
    img = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

After that, you can apply operations on top of each previous one.

    # adaptive thresholding -- calculates thresholds, at regions of size 100x100, minus 30
    img = cv2.adaptiveThreshold(img, 255, cv2.ADAPTIVE_THRESH_MEAN_C, cv2.THRESH_BINARY_INV, 100, 30)

    # morphological erosion -- applies erosion 5 times, with a kernel of size 2x2
    kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (2, 2))
    img = cv2.morphologyEx(img, cv2.MORPH_ERODE, kernel, 5)

    # contouring -- approximates contours in simple mode, and retrieves them in external mode
    contours, hierarchy = cv2.findContours(img, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    cv2.drawContours(img, contours[:len(contours)], -1, (0, 255, 0), 3)

Some links to useful documentation:

• OpenCV: Image Thresholding
• OpenCV: Morphological Transformations
• OpenCV: Contours : Getting Started

We can also clean up the list of contours.

    # get list of contour areas
    cnt_areas = np.fromiter((cv2.contourArea(cnt) for cnt in contours), dtype=np.float64, count=len(contours)
    # keep all contours with areas greater than 100 pixels
    order = np.argsort(cnt_areas)[::-1]
    keep = cnt_areas[order] >= 100
    order = order[keep]
    contours = tuple(contours[i] for i in order)
    # draw these contours onto the image
    cv2.drawContours(img, contours[:len(contours)], -1, (0, 255, 0), 3)

Bounding boxes can also be drawn.

    rects = tuple(cv2.minAreaRect(cnt) for cnt in contours[:len(contours)])
    boxes = tuple(np.round(cv2.boxPoints(r)).astype(int) for r in rects)
    cv2.drawContours(img, boxes, -1, (0, 0, 255), 3)

From these rectangles, we can determine the coordinates of each component in the camera’s coordinate space, translate them into the robotic arm’s coordinate space and move the robotic arm.

Note: I will be using both the terms “pixel” and “camera” interchangeably, but both refer to the way that OpenCV stores images as a matrix of color vectors for each pixel, thereby giving coordinates for each pixel.

From the first rectangle in the image, for example, we can extract its center coordinates and convert them.

    # rectangle center coordinates
    r_center_x, r_center_y = rects[0]
    comp_robot_x, comp_robot_y = pixel_to_robot_coords(r_center_x, r_center_y)

pixel_to_robot_coords() is an arbitrary function to convert between the two coordinate spaces, and looks something like this:

def get_robot_coords(r_center_x, r_center_y):
    robot_y = -0.24 * r_center_x + 228.52
    robot_x = -0.2528 * r_center_y + 421.39
    return robot_x, robot_y

The function does a simple computation for each axis by using parameters retrieved from a linear regression mapping of pixel/camera-coordinates to robot-coordinates.

The procedure for this mapping required moving the robotic arm to certain positions (e.g. (x: 300, y: -250)), marking those points with a colored sticker, using the color detection script to determine what coordinates the camera sees at those points, and recording those data. Since we assumed our camera was positioned straight (rotation square with the table), the robot’s y-coordinate was determined to be a function of the pixel’s x-coordinate, and the robot’s x-coordinate a function of the pixel’s y-coordinate.

After applying image processing to determine pixel-coordinates of electronic components on the table and converting these into robot-relative coordinates, we were able to move the robot arm to pick up components autonomously.

Part III: Distinguishing Parts from Parts

The next step after picking is placing, so we need to teach the Beagle where to place components based on what each one is. For this, we will use Machine Learning and deploy an AI model directly onto the Beagle. The way image classification, specifically, works is through a Convolutional Neural Network (CNN). CNNs work by taking a set of input images and identifying, over the course of computationally-expensive training sessions, the patterns that make up an image class (e.g. cat, dog, mouse). During training, it applies a series of filters onto the base image and tries to determine which combinations of these patterns (also called features) help determine the image class accurately.

Source: https://medium.com/dataseries/visualizing-the-feature-maps-and-filters-by-convolutional-neural-networks-e1462340518e

For our project, we will use a pre-trained model that can already pick out common shapes and patterns. This is what’s called Transfer Learning. We won’t have to teach the model how to see, but what to see. For this, we’ll use MobileNet V2, an efficient CNN architecture designed for computer vision in mobile and embedded applications. Google provides a cloud-based platform called Colab, which we’ll also use, that allows users to access and use various hardware accelerators (CPUs, GPUs, and TPUs) online for free, up to a certain point.

Google Colab uses Jupyter Notebooks, so we can use a Python-based Machine Learning framework like TensorFlow to train and test our model.

Our first order of business is to collect data. We’ll want to use the coordinates given by the image processing on the Beagle to crop out a small section of the larger image to center in on any specific component before inferring its type.

We set up a script to save camera captures on key presses using OpenCV, and produced 24 images, each of which contained many components. We then made crops around individual components using GNU Image Manipulation Program (GIMP) to rapidly save 300x300 boxes around components as images.

In the end, we had 1300+ images to train our model with. Because it would be unlikely that no components would overlap into each other’s bounding boxes, we decided to keep other components in the image, since the model should only classify the component in the center of the image even if there are other one’s present. We also wanted to account for the image not being exactly centered on the component of interest, so we have 300x300 crops from which we’ll take a 160x160 sub-crop about a random point to allow for variance created by the image processing script and the pixel-coordinates it returns (it gives both pixel- and robot-coordinates!).

From here, we’ll move on to Google Colab by making use of TensorFlow’s Transfer Learning Notebook to use the pre-trained model mentioned before.

Since the Jupyter Notebook is comprehensively laid out, I’ll only provide a list of high-level descriptions for each major step in the Notebook. For reference, here is my version of the notebook: robot_arm_transfer_learning.ipynb (1.2 MB)
And a Python script just in case!: robot_arm_transfer_learning.py (24.7 KB)

• Save image dataset to Google Drive
• Mount Google Drive filesystem into the notebook
• Create training, validation and test datasets (75%, 12.5%, 12.5%, respectively)
• Take every image in training, validation and test datasets, crop at center to 192x192, then crop at random point to 160x160
• Use Data Augmentation to “generate” new data by creating copies of training images with variations in rotation, brightness, contrast, hue, Gaussian blur and sharpness
• Rescale pixel values in preprocessing of inputs (images are in [0,255] but MobileNet V2 expects [-1,1])
• Create the base model from MobileNet V2 (don’t include the top layer, as it’s task-specific, and does not have as much generality as the bottleneck layer)
• Freeze the base model, so that we can build on top of it without updating the base’s weights during training
• Create a Global Average Layer and Prediction Layer to generate predictions from our block of features
• Build a model by chaining together the inputs, data augmentation, rescaling, the base model, global average layer, prediction layer, and outputs
• Compile the model with a learning rate of 0.001 and a SparseCategoricalCrossEntropy loss function
• Train the model for 100 epochs using the training and validation datasets
  
  
• Unfreeze the base model and freeze all layers from 1 to 99 to fine-tune the model
• Compile the model with a learning rate of 0.0001 and a SparseCategoricalCrossEntropy loss function (we want a lower learning rate to avoid overfitting)
• Train the model for 1000 epochs using the training and validation datasets (we used early-stopping, which caused training to cease at around epoch 300)
  
  

Trying out the test dataset on this model showed promising results, and we moved on to deploying it on the Beagle. First, we converted the model to TensorFlow Lite (TFLite). Next, we created a simple script to read input images and print predictions to the console:

import tensorflow as tf
import numpy as np

# create a TFLite Interpreter
interpreter = tf.lite.Interpreter(model_path="model.tflite")
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

# infer on frame1.jpg - frame5.jpg
for i in range(1, 6):
	pic_name = "test_pics/frame" + str(i) + ".jpg"
	print("NAME: " + str(i))
	input_data = tf.keras.preprocessing.image.load_img(pic_name)
	input_data = tf.keras.preprocessing.image.img_to_array(input_data)

	interpreter.set_tensor(input_details[0]['index'], input_data[np.newaxis, :])
	interpreter.invoke()
	output_data = interpreter.get_tensor(output_details[0]['index'])

	print("Predictions: ", output_data)

Of course, we had to create a virtual environment first.

python3 -m venv venv
source venv/bin/activate
python3 -m pip install numpy tensorflow

Output:

NAME: 1
Predictions:  [[9.9994671e-01 9.4633997e-06 3.7777522e-06 3.9879771e-05 1.8353434e-07]]
NAME: 2
Predictions:  [[1.2622441e-07 1.0861262e-06 9.9999881e-01 3.1261608e-11 6.7432733e-11]]
NAME: 3
Predictions:  [[5.1979964e-06 1.9225517e-07 1.9193716e-05 9.9996650e-01 8.9024006e-06]]
NAME: 4
Predictions:  [[3.1221173e-08 1.0000000e+00 2.7385351e-12 3.1222172e-09 1.4687450e-09]]
NAME: 5
Predictions:  [[5.46070169e-06 8.02626730e-08 4.71836330e-08 1.10886255e-07
  9.99994278e-01]]

Using the greatest value and its index, the image class can be determined. For me, the classes are ordered like so: capacitors, chips, resistors, switches, transistors. Now, by taking camera captures and cropping them to the right size at the coordinates determined by the image processing, we can feed images into our model to infer component types.

Conclusion

This part of the project was more software-heavy than the last, but it was definitely important. We implemented computer vision on the Beagle in two stages, image processing and machine learning, and laid the groundwork for a fully autonomous electronic component sorting system. We were able to isolate different parts in development and demonstrate their functionalities in these independent environments. The next step will be to integrate these parts and prove they can work together. For now, the robotic sorting system project is almost ready, we just need to connect the subsystems.