Motion Estimation (ME), which is composed of integer motion estimation (IME) and fractional motion estimation (FME), is the most computational intensive module in HEVC encoding procedure. In this paper, a new fast fractional pixel motion search method, that is based on a six-parameter two-dimension error surface model, is proposed. In our proposal, by solving the over-determined equations, nine integer-pixel rate-distortion costs (RDC), including the best integer-pixel search candidate and its eight neighboring integer pixels, are used to estimate the six parameters in the model. Then, we can obtain the minimal position on the fitted error surface equation, which is the quarter-pixel accurate search center. We provided three kinds of search patterns in the quarter-pixel search stage, which could take a tradeoff between the computational complexity and the prediction accuracy. Experimental results demonstrate that, as compared with HM reference software (HM-15.0), the three proposed FME patterns could reduce 35.1%, 29.4%, and 22.5% encoding time, while the corresponding compression efficiency losses in terms of BDBR are 3.04%, 0.79%, and 0.43%, respectively.