This paper gives an insight into the design considerations like functional partitioning and optimization techniques across accelerator cores of typical ADAS SoCs for porting RealTime ADAS applications, such as object detection algorithm. It also addresses design considerations for efficient usage of deep learning techniques such as Convolutional Neural Network (CNN) in an entry level ADAS video system such as Texas Instruments TDA3x ADAS SoC.