The code for the live detection of the ball in the camera feed is located in the files detection.cpp and detection.h and uses constants from magic.h that are being determined from calibration.
The purpose of this code is:
There are three so called regions of interest (ROIs) defined in the algorithm.
The main ROI is the important region for keeping the ball in the game. The purpose of the top region is to be able to use the top flipper to deflect the ball into parts of the playfield that are otherwise hard to reach.
The ball detection algorithm can be run on any defined ROI. However, the ROIs currently in use are chosen to include as few lamps as possible (which must be masked or the detection algorithm will be confused) and fit the time window of at most 33.3 ms that is available well. Too many or too large regions will become problematic at some point.
The following is a step by step description and visualization of the actual detection algorithm for the ball.
The given ROI is cropped out of the full frame:
Three color channels are not required, reduce to single grey channel for more efficient processing.
Ignore lamps and areas outside of the playfield using polygon mask:
Ignore high frequency noise.
Compute pixel wise difference to fixed reference frame, makes movement visible.
Areas with no movement are black, areas with movement are grey/white:
Set all pixels with gray value above certain threshold to full white, rest to full black
→ Binary image of moving areas:
Widen contours by several pixels. Fill in holes, join close areas together.
Narrow contours by several pixels. Roughly restores original size of contours but holes remain filled, small gaps remain closed.
Operation is called “Closing”:
Apart from ball detection in the main and top ROI, the algorithm does the following:
The algorithm detects if the two bottom flippers are up and will mask them if that's the case. This is done to avoid that the contour of the flipper merges with the ball contour so that the ball isn't detectable anymore.
The launch ROI is a small area at the red lamp that is flashing when a ball is ready to be launched into the game. This is detected by looking at how many pixels in that region are white, if the number is above a certain threshold, the lamp is assumed to be on.
We tried to observe the whole camera frame with the ball detection algorithm (not only regions) and mask flashing lamps automatically. This has been tried by observing the flashing lamps for a while before the actual game starts and saving the information about the lamps (based on their purple color in the no-IR filter image) in a buffer image that then was used to compute a reference frame free of blinking lamps and also to mask lamps from the current frame. While that did work, it introduced too many blind spots due to the sheer amount of lamps on the playfield, so that not much was gained compared to the approach of just looking at the bottom part of the pinball machine that has a black cover for the lamps. Also the ball was colored in the same purple color as the lamps itself when it passed a flashing lamp, making it almost impossible to detect.