In a small bowl, whisk the olive oil, olives, lemon juice, garlic, thyme, black pepper, and cayenne. Bring a large pot of well-salted water to a boil and cook the cavatelli until al dente. Drain it well and return it to the pan. Add the arugula and the olive-oil mixture, folding gently until the ingredients are combined. Let sit until the arugula has wilted, about 3 minutes. Sprinkle with the ricotta salata just before serving.