Perception of visual stimuli is modulated by their context. The effect of context can be facilitatory or suppressive in a manner that is highly sensitive to stimulus conditions. To gain a better insight into mechanisms of contextual modulation, we asked what patterns of modulation are invariant of the nuance of stimulation. We uncover the invariants by measuring maps of modulation across the full range of modulating parameters and then compare the empirical maps with predictions of models of neural interactions. We studied contextual modulation using two luminance gratings ("flankers") and a "probe" positioned between the flankers. The probe was either distributed (luminance grating) or localized (line). We measured the probe visibility for a wide range of flanker contrasts (C) and spatial frequencies (SF). First, we used distributed probes and obtained a bivariate map of probe contrast threshold in the coordinates of C and SF. The facilitatory effect of context formed well-defined "islands" in the map, i.e., facilitation was tuned to both C and SF. The nonmonotonic effect of flanker contrast could only arise in a nonlinear system, for example in the process of stimulus encoding or when the distributed neural activity is collapsed to the binary decision variable. Second, we attempted to bypass the decision nonlinearity using a localized probe (line). Line contrast threshold varied as a function of location between the flankers and it depended on flanker contrast, similar to the results with distributed probes. The results suggest the encoding origin of the nonlinearity. We use results of empirical mapping to constrain a model of contextual modulation in terms of the canonical inhibition-stabilized neural network (ISN). We show that a chain of canonical ISN nodes is tuned to SF (arXiv:1410.4237) producing lateral interactions that are also tuned to flanker SF and C, similar to our results.