We propose a new optical one-time password(OTP) authentication method by using digital holography, which enhances security strength in the cryptosystem compared to the conventional electronic OTP method. In this paper, a challenge-response optical OTP authentication based on two-factor authentication is presented by 2-step quadrature phase-shifting digital holography using orthogonal polarization, and two-way authentication is also performed using the challenge-response handshake in both directions. The ID (identification), PW (password) and OTP information are encrypted with a shared key by applying phase-shifting digital holography, and these encrypted information are verified each other by the shared key. Because the encrypted digital holograms which are transmitted to the other party are expressed as random distribution, it guards against a replay attack and results in higher security level. Optically, encrypted digital hologram in our method is Fourier transform hologram and is recorded on CCD with 256 gray-level quantized intensities. The proposed method has an advantage that it does not need a time-synchronized OTP and can be applied to various security services. Computer experiments show that the proposed method is suitable for high secure OTP authentication.