In the context of a warped extra dimension with standard model fields in the bulk, we obtain the general flavor structure of the Higgs couplings to fermions. These couplings will be generically misaligned with respect to the fermion mass matrix, producing large and potentially dangerous flavor changing neutral currents. As recently pointed out [K. Agashe and R. Contino, arXiv:0906.1542.], a similar effect is expected from the point of view of a composite Higgs sector, which corresponds to a four-dimensional theory dual to the five-dimensional setup by the AdS/CFT correspondence. We also point out that the effect is independent of the geographical nature of the Higgs (bulk or brane localized), and specifically that it does not go away as the Higgs is pushed towards the IR boundary. The flavor changing neutral currents mediated by a light enough Higgs (especially their contribution to epsilon(K)) could become of comparable size as the ones coming from the exchange of Kaluza-Klein gluons. Moreover, both sources of flavor violation are complementary since they have inverse dependence on the five-dimensional Yukawa couplings, such that we cannot decouple the flavor violation effects by increasing or decreasing these couplings. We also find that for Kaluza-Klein scales of a few TeV, the Higgs couplings to third generation fermions could experience suppressions of up to 40% while the rest of diagonal couplings would suffer much milder corrections. Potential LHC signatures like the Higgs flavor violating decays h -> mu tau or h -> tc, or the exotic top decay channel t -> ch, are finally addressed.