Introduction
This post serves as a top-level overview for a robotic sorting system project, in which we attempt (and succeed in!) sorting electronic components spread out across a table with a robotic arm. The parts used are a single-board computer (BeagleY-AI, 2820-102991834-ND), an industrial robotic arm (igus ReBeL 6DOF; 4903-REBEL-6DOF-02-ND) and a camera module (Arducam; 4679-B0292-ND), all of which can be bought from DigiKey. A high-level synopsis of the obstacles encountered and design choices made will be given. We’ll walk through each stage of the project and talk about where progress sped up and stagnated. With that, let’s get started.
Part I: It’s Alive!
The Beagle controls our system, but the star of the show is the robotic arm. Without it working, the whole project fails or an expensive alternative must be used. Therefore, the first step in our project was to prove that our robotic arm could move as intended. From here, we broke the problem down and defined requirements. We needed the system to detect components, pick one up and place it into its corresponding bin.
It could do the detecting before picking one up, so that the camera is given time to make a camera capture without the robotic arm obstructing its field-of-view. Regardless of how we want to arrange this action sequence, the robotic arm needs to be able to move to specified coordinates on-command. For this, we tried running ROS2 on the Beagle. Many Docker images and debug messages later, we found that this software was causing trouble. Attempting to move to a pair of coordinates led the CPU usage to spike and movement was inaccurate and imprecise. ROS2 is a very comprehensive framework for designing complex robotic software and provides useful abstractions intended to make interfacing with the robotic arm simple, so that advanced behavior can be implemented. Learning how to use these abstractions in the first place revealed a mountain of required prerequisite knowledge that would’ve extended the implementation of the project significantly. Fortunately, we found a replacement: CRI-Python-Lib, which adds a small layer of Python abstraction to the robotic arm’s messaging protocol to make development as barebones as possible without being overly tedious.
Figuring out how to initialize the robot in our new lower-level code seemed to be its own endeavor, however. But eventually, by modeling the code after the sequence performed in the official Windows-based robotic control software (iRC) and figuring out the values of the parameters used, we created a minimal, repeatable way to start our robotic arm in code. After this, getting the arm to move to input coordinates was straightforward. A leaner substitute to ROS2, CRI-Python-Lib allowed us to move along either cartesian axes or joint axes, which is all we needed for our project. To simulate picking up objects, we 3D-printed fingers to attach to our arm’s external end-effector.
Throughout the project, these fingers evolved, eventually becoming spring-loaded in the end. We had the robotic control we needed, now on to the next phase.
Part II: Visual Intelligence
Since our system is supposed to recognize electronic components and determine their types, we’ll begin this section by laying out the plan for the computer vision. We’ll split the computer vision pipeline into two parts: image processing and machine learning. Image processing will be used to quickly determine where parts are located on the table and machine learning, specifically deep learning, will be used to determine what type each part is. Image filtering and image classification will be the major ideas in this section of the article and a more in-depth walkthrough of our implementation process can be found in Setting up a Single-Board Computer for Computer Vision. As mentioned before, the system should take a photo, analyze it, then send the coordinates to the robotic arm. Implementing image processing included using OpenCV to apply filters to our camera’s base image in a script and adjusting arguments to these function calls. After trying out techniques like blurring, thresholding, edge detection and more, it was determined that the minimal implementation required Thresholding, Erosion and Contouring. We were able to produce a list of the center coordinates (in camera/image coordinates) of the components in the image. To use these camera-relative coordinates, we needed to convert them to a robot-relative form. A table of coordinate-pairs was made to map known robot-coordinates (x) to camera-coordinates f(x). To test this conversion function, a small script was written to pick up objects marked with a pink dot sticker. The script detected the pink color, determined the center coordinate-pair of that dot, converted it to robot-relative coords and sent it to the robotic control script.
Moving onto the machine learning portion of this section, the big question involved figuring out how to develop an entire deep learning model for image classification, test it and deploy it onto our embedded system. Luckily for us, this isn’t an uncommon problem to have, and tools have been developed for our application. TensorFlow is a common machine learning framework that can be used with Python. Google Colab is a cloud-based platform for accessing Google’s hardware accelerators by running Python libraries like TensorFlow. As for the AI model development part, we’ll use a technique called transfer learning, where a pre-trained model is retrained with application-specific data but to a degree significantly less than training from scratch. As a result, less data and compute are used to “fine-tune” the model. We used TensorFlow’s Transfer Learning Notebook to accomplish this. After 1300+ images and over 100 epochs of initial training, our model achieved ~91% accuracy. Over ~300 epochs of fine-tuning, that got bumped up to 99%.
This was a good point to move on to testing the model. A minimal script was created to take in pre-saved image crops around components and infer using our new model. From the few images we tested, the model was able to deduce all of types correctly, so we figured it was time to try integrating of the subsystems implemented so far.
Part III: Tying it all Together
To complete the project, we needed to connect our three subsystems: the Beagle, robotic arm and AI model. We had been running scripts for each subsystem independently during implementation, so in order to test the whole system together, we needed communication channels weaved between them. For this, we chose a basic file read-write scheme, where if, for example, a new set of camera coordinates and robot coordinates were already used, the corresponding files would be cleared. This required us to poll the states of these files continuously. During testing, however, an issue became apparent. The camera-to-robot-coordinates conversion was able to neither accurately nor precisely calculate the positions of the intended electronic components. The robotic arm would move the fingers to the correct region, but there was always an offset in both the x- and y-axes preventing it from gripping the components at the right spot. We figured that somewhere along the way, the camera orientation was involuntarily adjusted and needed to be realigned. Our previous approach assumed the camera would remain straight relative to the table, which allowed us to create a simple conversion function. Since our axes were now mistakenly misaligned, we needed a solution that could account for the changes in two axes rather than in one. We needed to use matrices, more specifically, a concept called 2D Homography, where a coordinate space of dimension n is transformed to another coordinate space of dimension n by multiplying the coordinate vector by a homography matrix H.
Source: 2D homography matrix describing the mapping between the target plane
All we needed to do was use OpenCV’s findHomography() function on a set of points in two coordinate spaces. This included placing colored stickers at regular intervals relative to the robot’s coordinates, then using a color detection script to print out the camera/pixel coordinates of each of these color blobs. Two sets of points were extracted from this: the robot-relative coordinates, and their corresponding camera/pixel coordinates given by OpenCV.
Using this, we created a homography matrix and were able to move the robotic arm to electronic parts, grab them while inferring their types and drop them into their respective bins.
Conclusion
By taking these subsystems into isolated development, incremental testing and integration, we developed a complex project with real-world applications in various industries. One major theme of this project was simplicity. To keep development quick and consistent, a minimal approach was taken, which saved us from unnecessary headaches. Switching from ROS2 to CRI-Python-Lib prevented taking on needless complexity. Training our model on 5 types of components struck a good balance between proving robustness and minimizing nuisances with image processing filters. Overall, this project aimed to explore how robotics can be used in conjunction with deep learning to complete basic tasks autonomously. We built a system that can act differently on objects according to their types, a functionality can be adapted easily to other tasks. Hopefully, the robotic sorting system shows that combining open-source software and common embedded hardware can create intelligent, useful physical systems using engineering, abstraction and the Internet.