Coupling of light to and from integrated optical circuits has been recognized as a major practical challenge since the early years of photonics. The coupling is particularly difficult for high index contrast waveguides such as silicon-on-insulator, since the cross-sectional area of silicon wire waveguides is more than two orders of magnitude smaller than that of a standard single-mode fiber. Here, we experimentally demonstrate unprecedented control over the light coupling between the optical fiber and silicon chip by constructing the nanophotonic coupler with ultra-high coupling efficiency simultaneously for both transverse electric and transverse magnetic polarizations. We specifically demonstrate a subwavelength refractive index engineered nanostructure to mitigate loss and wavelength resonances by suppressing diffraction effects, enabling a coupling efficiency over 92% (0.32 dB) and polarization independent operation for a broad spectral range exceeding 100 nm.