Students: Gianni Meesters, Demi Smeets, René Kuijvenhoven, Timotheüs van den Berg
Original situation
A Sigma line is a weighing line that weighs trays with fresh mushrooms. De sigma line weighs the trays and divides them in three groups: overweight, underweight, and the correct weight. Each group is transported over a different track. This means one track for the overweight trays, one track for the underweight trays, and one track for the trays with the correct weight.
The over- and underweight tracks contain loadcells. When a tray is detected on a loadcell the conveyer is stopped. At this point the mushrooms are manually sorted until each tray contains the correct weight. During this process it is important that the top layer in the tray is positioned with the mushroom caps pointing upwards.
The problem
Producers want to reduce their cost within the mushroom production. A great way to do this is to automate certain parts of the process. Viscon was wondering if the Sigma line could be one of these processes. Therefore, us, students at SMR, where asked to do a feasibility study on this.
They decided that they wanted a proof of concept. If the proof of concept showed that it is indeed achievable to automate this process, and more importantly save costs, the company would go ahead and start automating the Sigma weighing line.
our solution
To be able to show Viscon what was possible we build a setup around a real Sigmaline. This setup contains the Sigmaline, an UR10 cobot, two intel realsense depth cameras, an end of arm tool with all the necessary air components, the frame that connects all the parts and of course quite a lot of software. The entire (hardware) setup is shown in below picture. The hardest part during this project was creating vision software that would be able to function just as good as a human. We decided to use machine learning to detect the mushrooms. Machine learning would also make it easy for us to detect the position of the mushroom.
However, after a couple of weeks during in which we trained our machine learning algoritm, we found out that our camera’s do not have a good enough image quality to be able to properly detect mushrooms inside a tray. And we were therefore not able to correctly train the machine learning algoritm.
Because the camera’s needed would cost a lot of money, we decided to take a different approach, to use python openCV. This is open source software that, with the right filters, would detect all the mushrooms in a tray using circle detection. The false positives are filtered out using the dominant color within the cirlce. If the dominant color is white, a mushroom is detected. If the dominant color is not white, the circle detected is a false positive.
Furthermore, a SQL database is used to store the data collected. This way data can be accesed at any time, which can provide valuable information about the process. Also, a SCADA systems is used to store and display the data in a way the costumer would understand.
The process
- The trays are placed on the right tracks (underweight, overweight, correct weight) and arrive at the loadcells.
- The loadcells measure the weight of the tray, this data is then send to our database.
- The python programm then calculates the amount of mushrooms that need to be taken out, or placed in a tray.
- The trays are detected on the loadcells using rectangle detection on the image provided by the camera. Within these rectangles, mushrooms are detected using circle detection.
- The coordinates of these mushrooms are then send to the robot.
- The robot picks the right amount of mushrooms out of the overweight trays and places them in to the underweight trays. When there are more mushrooms needed then the overweight trays can provide, the robot moves to the buffertray to pick mushrooms.
- When the weight is correct, the trays are approved and transported to the next part of the process.
The major decisions made during the project
During this project major decision were made to create a successful result. The following list provides some of these decisions with short explanations.
- Using OpenCV circle detection, after trying lots of options through vision we concluded that our way to go was OpenCV. With this we were able to detect the mushrooms
- Using an UR10 and not an UR5. Our university provided two different robots, an UR5 and an UR10. We chose the UR10 since it has more range and is therefore better capable of reaching everything.
- Making end of arm tool that can pick up three mushrooms instead of one. We made this tool to majorly improve our picks and places. Otherwise it would have been impossible to reach the target amount.
- Using a SCADA system for data analysing. To process and collect data of our system and of the process we used a SCADA system. From there we were able to save information on a SQL database.
- To only handle mushrooms that have the cap upwards and not handle mushrooms with the stem sidewards or upwards. Detecting the stem or the side of the mushroom was not possible with OpenCV and the capabilities of our cameras. Therefore we decided to only handle mushrooms with the cap upwards.
- To encase the frame to block light influences from external sources. External light sources caused reflections and created problems within our vision. To block this we added a roof and a piece of wall.