Planning

The general idea is that there’s a projector vertically mounted overhead that projects the game onto the ground, then there’s a webcam also placed over the projected area that is used to track the player’s location on the field, which will translate into the location of the cursor on the laptop. Lastly, the player will be holding a bluetooth mouse, used only for left-clicking. This means that the player has to actually walk and step on game objects that are projected on the ground.

Code

The principle behind translating player position into cursor location is simple. Just look for a red blob, find the center of the red blob, set that value as the location of the screen. I guess here’s the code for the color detection algorithm? (btw I’m using OpenCV)

whileTrue:_,frame=cap.read()# The webcam I'm using inverts everything, # I also oriented it the wrong way when attaching it to the ceiling.# So I'm compensating for that in code because I'm lazy.frame=cv2.flip(frame,-1)# Cropping because the projection does not fill up the webcam FOVframe=frame[int(r[1]):int(r[1]+r[3]),int(r[0]):int(r[0]+r[2])]#HSV works better than thresholding Rhsv=cv2.cvtColor(frame,cv2.COLOR_BGR2HSV)# lower red rangelower_red=np.array([0,120,70])upper_red=np.array([10,255,255])mask1=cv2.inRange(hsv,lower_red,upper_red)# upper red rangelower_red=np.array([170,120,70])upper_red=np.array([180,255,255])mask=mask1+cv2.inRange(hsv,lower_red,upper_red)kernel=np.ones((5,5),np.uint8)# Get only red blobsmask=cv2.erode(mask,kernel)_,contours,_=cv2.findContours(mask,cv2.RETR_TREE,cv2.CHAIN_APPROX_SIMPLE)ifcontours:# Only use the largest red blobcnt=max(contours,key=lambdax:cv2.contourArea(x))cv2.drawContours(frame,[cnt],0,(0,0,0),5)# Calculate centroid of red blobM=cv2.moments(cnt)# Scale the two values based on screen resolutioncenter_x=int((M['m10']/M['m00'])*x_mult)center_y=int((M['m01']/M['m00'])*y_mult)# Set cursor location using centroid of red blobpyautogui.moveTo(center_x,center_y)cv2.imshow("Frame",frame)cv2.imshow("Mask",mask)# Use esc to stop programkey=cv2.waitKey(1)ifkey==27:breakcap.release()cv2.destroyAllWindows()

Since the projection didn’t fill the webcam FOV, I added some code to set ROI.

Hardware

So with the code working, it was now time to set up this whole system. The webcam was easily attached to the ceiling using hook and loop tape. The projector was another story. My parents were very against me drilling holes in the ceiling, thus, I had to get creative. Experimentation lead me to create a zip-tie monstrosity that hangs from a clamp fastened to a beam on the ceiling (the clamp was later moved to the skylight).

Then I just connected everything and hoped that it would work. To my surprise, it did. After some cable management, I got to playing. Sadly, the projected image is really small, which means that the cursor position isn’t very accurate. Also, the latency for the cursor position is pretty bad, which is a death sentence in the world of rhythm games. So my dream of playing life sized osu! has to wait until another day.