The three smaller robots, named Shimi, perform "dance movies" to the music and do not actually play instruments. Two of them perform percussion parts that were composed ahead of time, but one has the ability to compose an original accompaniment on the spot. Then the larger robot, named Shimon, actually improvises on the marimba, much like a human jazz musician. It generates the musical notes it's supposed to play using artificial intelligence, and then uses a path-planning algorithm in order to position its appendages to hit the right notes. Path-planning algorithms are notoriously tricky in the field of robotics, as it's extremely difficult for a robot to focus on generating the intelligence, guiding its body, and avoiding obstacles simultaneously.

"In addition to figuring out what to play, Shimon has to figure out how to play it," said Bretan. "You might have generated the most beautiful music ever created by the musical algorithms, but if it's corrupted by the path-planning algorithm, you'll lose that original beauty."