Incorporate domain knowledge into predictive model: As a data scientist / consultant, in many cases we are bei...

As a data scientist / consultant, in many cases we are being called in to work with domain experts who has in-depth business knowledge of industry settings. The main objective is to help our clients to validate and quantify the intuition of existing domain knowledge based on empirical data, and remove any judgement bias. In many cases, customers will also want to build a predictive model to automate their business decision making process. To create a predictive model, feature engineering (defining the set of input) is a key part if not the most important. In this post, I'd like to share my experience in how to come up with the initial set of features and how to evolve it as we learn more. Firstly, we need to acknowledge two forces in this setting Domain experts tends to be narrowly focused (and potentially biased) towards their prior experience. Their domain knowledge can ...