Anonymous.The high accuracy for both small and large motion estimation are mainly cause by two contributions: firstly, we present and implement an edge preserve patch match (EPM) layer that propagates self-similarity patterns in addition to offsets. The accuracy of optical flow prediction has greatly improved by this method. Secondly, we develop a course-to-fine network architecture to tackle large displacement estimation and introduce a residual flow method to solve small displacement estimation.