Many studies about social question and answer (Social Q&A) platforms have focused on the recommendation algorithms designed to find the right person for the question. But very little literature has investigated how to motivate the selected potential answerers to answer a question, especially in an enterprise setting. In this work, we designed an in-situ experiment in an enterprise social Q&A system to understand how different design aspects (e.g., exposing relationship information, directly asking the answerer, indicating question’s importance and urgency, and using virtual points as incentives) could influence answerers' engagement behaviors. We found that two design features examined in the experiment can affect answerers’ viewing and answering behaviors. These findings lead to specific design recommendations, which are also discussed in the paper.