We generalize the recently introduced dual fermion (DF) formalism for disordered fermion systems by including the effect of interactions. For an interacting disordered system the contributions to the full vertex function have to be separated into elastic and inelastic scattering processes, and addressed differently when constructing the DF diagrams. By applying our approach to the Anderson-Falicov-Kimball model and systematically restoring the nonlocal correlations in the DF lattice calculation, we show a significant improvement over the Dynamical Mean-Field Theory and the Coherent Potential Approximation for both one-particle and two-particle quantities.