Sensoryweighting and realignment: independentcompensatoryprocesses.

1Department of Neuroscience, The Johns Hopkins School of Medicine, Baltimore, Maryland, USA.

Abstract

When estimating the position of one hand for the purpose of reaching to it with the other, humans have visual and proprioceptive estimates of the target hand's position. These are thought to be weighted and combined to form an integrated estimate in such a way that variance is minimized. If visual and proprioceptive estimates are in disagreement, it may be advantageous for the nervous system to bring them back into register by spatially realigning one or both. It is possible that realignment is determined by weights, in which case the lower-weighted modality should always realign more than the higher-weighted modality. An alternative possibility is that realignment and weightingprocesses are controlled independently, and either can be used to compensate for a sensory misalignment. Here, we imposed a misalignment between visual and proprioceptive estimates of target hand position in a reaching task designed to allow simultaneous, independent measurement of weights and realignment. In experiment 1, we used endpoint visual feedback to create a situation where task success could theoretically be achieved with either a weighting or realignment strategy, but vision had to be regarded as the correctly aligned modality to achieve success. In experiment 2, no endpoint visual feedback was given. We found that realignment operates independently of weights in the former case but not in the latter case, suggesting that while weighting and realignment may operate in conjunction in some circumstances, they are biologically independentprocesses that give humans behavioral flexibility in compensating for sensory perturbations.

Sensoryweighting and realignment could each be used to compensate for a visuoproprioceptive misalignment. A: when reaching into a fountain to pick up a coin, for example, the visual estimate of hand position (ŶV) will be offset from the proprioceptive estimate (ŶP). B: the brain is thought to weight and combine the available estimates into a single, integrated estimate of hand position (ŶVP) (). However, relying on this integrated estimate will result in missing the coin, which is most closely lined up with ŶV. C: to bring the integrated estimate closer to ŶV, the brain could increase the contribution of ŶV (i.e., up weight vision), yielding ŶVP1. D–F: alternatively, the brain could realign proprioception, bringing ŶP closer to ŶV (E). When the brain integrates ŶP1 and ŶV (F), ŶVP1 is obtained, and the person can accurately reach the coin.

Views of the relationship between sensory weights and realignment. A: possibility 1. The sensory modality with greater variance will be weighted less in an integrated sensory estimate (i) and will realign more if realignment takes place (ii). The connection between variance and weights (i) has been demonstrated in a number of human behaviors (e.g., ; ; ), but the relationship between sensoryrealignment and weights (ii) has been studied only indirectly (). B: possibility 2. In the present study, we found evidence to support an alternative view, in which weighting and realignment are two independently controlled processes, either of which can be used to compensate for a sensory misalignment. It should be noted that while weights (; ; ) and realignment (; ; ) have each been linked to the relative variances in sensory estimates, we do not suggest that either weights or realignment are wholly dependent on variance; other influences, such as attention/conscious effort, have been shown to affect both weights () and realignment (). Accuracy may also be important (). C and D: experimental predictions. C: if possibility 1 is correct and sensoryrealignment depends on weights, we predicted that in both experiments, subjects with a higher weight of vision (wV) would show a greater realignment of proprioception (ΔŷP). D: if possibility 2 is correct and sensoryrealignment is controlled independently of weights, we expected wv and ΔŷP to be unrelated in at least one of the two experiments.

Experimental setup for experiments 1 and 2. A: the subject looked down into a horizontal mirror (middle) and saw targets and cursors indicating hand position reflected from a horizontal rear projection screen (top). The (dominant) reaching hand rested on a hard acrylic reaching surface (bottom) below the mirror. The (nondominant) target hand remained below the reaching surface at all times. The mirror was positioned midway between screen and reaching surface, such that images in the mirror appeared to be in the plane of the reaching surface. Not pictured: black drapes that obscured the subject's vision of his/her arms and the room outside the apparatus. B: timeline of a single reach in the baseline block and bird's eye view of the display. Dashed lines were not visible to the subject. The total display area was 75 × 100 cm. Schematic is not shown to scale. V, visual target; P, proprioception target; VP, combination of V and P targets. Screen 1: subjects placed their reaching finger (white dashed line) in the yellow start box, which appeared in one of five possible positions (yellow dashed squares; the central position was 20 mm from each of the other positions), with the aid of an 8-mml:diameter blue cursor indicating reaching finger position [veridical to minimize proprioceptive drift between reaches ()]. Screen 2: a red fixation cross appeared in a random location within an invisible zone (gray), and subjects were instructed to fixate on it for the duration of the reach. Screens 3–5: subjects positioned their target finger (dashed gray line) as instructed, on one of the two tactile markers stuck to the bottom of the reaching surface ∼40 mm apart (green dashed circles) for a P or VP target or down in their lap for a V target, which appeared as a white box in one of the two possible target locations. For VP reaches, the V target was projected on the P target during the baseline block but was gradually offset in the y direction during the adaptation block. Once both hands were correctly positioned, subjects reached toward the target, with the cursor disappearing at movement initiation. Movement speed was not restricted, and subjects were permitted to make adjustments. Screens 6–8: when the reaching finger had not moved more than 1 mm for 2 consecutive seconds, the endpoint position was recorded and the red fixation cross disappeared. In experiment 2, subjects were immediately instructed to lower their target hand to rest in their lap and return the reaching finger to the start box. In experiment 1 only, endpoint visual feedback (blue dot) was displayed for 2 s at the movement endpoint location after reaches to VP targets. After this, subjects were instructed to lower their target hand and return the reaching finger to the start box. C: bird's eye view of an unsuccessful (screen 1) and successful (screen 2) reach to a late adaptation VP target in experiment 1. The illustration is to scale. The V component (white box) of VP targets was gradually displaced up to 70 mm away from the P component (green dashed circle) in the positive y direction. Subjects were not aware of this manipulation, but reaching endpoints close to the V component (screen 2) counted as a hit: subjects viewed an animated explosion, heard sound effects, and were awarded a point (top right corner). Reaching endpoints farther than 10 mm from the V component (screen 1) resulted in a negative sound effect and no point was awarded. In other words, the visual estimate of target hand position had to be regarded as the “good” estimate (and proprioception the “bad” estimate) to succeed in experiment 1. In experiment 2, no endpoint visual feedback was given (no blue dot, no explosions, and no points awarded). *Endpoint visual feedback was given in experiment 1 only and only for VP targets.

Individuals used different strategies in experiment 1. When visual-proprioceptive alignment is perturbed and vision is constrained to be the “good” modality, subjects compensate by weighting vision heavily, shifting P reach endpoints in the direction of the perturbation, or both. A: subject who used a combined weighting and P endpoint strategy. i, Targets and reach endpoints in the adaptation block. V targets (solid gray line) were gradually displaced from P targets (dashed gray line) in the positive y direction. This displacement applied to both V and VP targets and reached a maximum of 70 mm by the end of the 84 reaches. V endpoints (solid blue line) tended to follow V targets. Because endpoint visual feedback was given only for VP targets, VP endpoints (dashed purple line) approximate the subject's success: if VP endpoints closely followed the V component of the target (solid gray line), subjects were scoring more points. This subject was 79% “successful.” A change in the position of P endpoints (ΔŷP = 57.4 mm, dotted red line) suggests that proprioceptive realignment may have taken place, but motor adaptation of the reaching hand could also have contributed. ii, wV in the adaptation block. A separate wV value was calculated for each VP reach. The line represents the best fit but was not used in any calculations. This subject relied more on vision by the end of the adaptation block (wV = 0.55). iii and iv, Example wV calculations. wV was calculated by comparing the mean of three VP endpoints (purple circle) to the means of the four V endpoints and four P endpoints (blue and red circles) occurring closest in time. Two-dimensional distances were used. P targets were at the origin. iii, It is apparent that early in adaptation (fourth wV is illustrated in this case), VP endpoints are closer to the P estimate than the V estimate, reflecting a high weight of proprioception and low weight of vision. iv, The situation was reversed later in adaptation (second-last wV is illustrated in this case). B: subject who used the P endpoint strategy alone. i, P endpoints shifted substantially during adaptation (38.2 mm). Success was 66%. ii, However, vision was down-weighted, and the subject relied more on proprioception by late adaptation (wV = 0.29). C: subject who used the weighting strategy alone. i, No P endpoint shift in the direction of the misalignment occurred (−7.3 mm, not significantly different from zero; P > 0.4 by rank sum test). Nonetheless, success was 79%, because vision was weighted heavily throughout (late adaptation wV = 0.94; ii). D: subject who did not use either strategy. i and ii, Little P endpoint shift took place (9.9 mm; i), and the subject did not rely heavily on vision (late adaptation wv = 0.30; ii). This subject had limited success (44%). E: group data (n = 39 subjects). For the purpose of dividing subjects into categories, we counted P endpoint shift ≥ 35 mm as using the P endpoint strategy and late adaptation wV ≥ 0.5 as using the weighting strategy.

wV and ΔŷP are each related to success in experiment 1. A: each square represents a single subject. Color reflects success. Subjects who rely heavily on vision are not more or less inclined to shift P reach endpoints in the direction of the perturbation (no significant correlation between wV and ΔŷP: r = −0.15, P = 0.37, n = 39 subjects). Subjects could succeed at the task by shifting P endpoints in the direction of the misalignment (ΔŷP, reflecting some combination of proprioceptive realignment and motor adaptation), weighting vision high, or both. Together, wV and ΔŷP account for 62% of the variance in success (stepwise multiple regression R2 = 0.62, F = 29.3, P < 0.0001). B: ΔŷP is related to success for subjects in experiment 1 (r = 0.49, P = 0.0014). C: wV is related to success for subjects in experiment 1 (r = 0.53, P = 0.0005). Overall, these results suggest P endpoint and weighting strategies operate independently in this task and, furthermore, that either strategy can lead to success at hitting targets. This implies greater behavioral flexibility than if only one strategy led to success.

Experiment 1sensory subexperiment. A: sensory subexperiment example subject. Before and after the adaptation block, 18 subjects estimated the location of the P target by pressing buttons on a keypad rather than reaching to it. The change in these estimates (filled circle, ΔŷP) reflects sensoryrealignment of the proprioceptive estimate of target hand position (ΔŶP) rather than adaptation of the motor command to the reaching hand. B: sensory subexperiment group data. For the 11 subjects with a significant P endpoint shift (first two bars), mean subexperiment ΔŷP (solid bars) was 18.6 mm, or 52% of the P endpoint shift (open bars), which includes motor adaptation. This difference was significant (P = 0.002 by Wilcoxon rank sum test), suggesting that the P endpoint shift we measured in experiment 1 had approximately equal sensory and motor components. To verify that subexperiment ΔŷP is not the same no matter what the P endpoint shift is, we also looked at the seven subjects whose P endpoint change was not significant (second two bars) and found that subexperiment ΔŷP was not significantly different from zero or from P endpoint change. Furthermore, we found a significant correlation between P endpoint shift ΔŷP and subexperiment ΔŷP for these 18 subjects (correlation r = 0.54, P = 0.021, slope of the fit line: 0.77), supporting the idea that the P endpoint shift in experiment 1 comprised both sensory and motor components across subjects. Error bars represent SEs. *Includes a motor adaptation component in addition to sensoryrealignment.

Sensoryrealignment can operate independently of sensory weights. A: in experiment 1, there was no relationship between P endpoint shift (ΔŷP, includes motor adaptation; open circles and dashed line) or proprioceptive realignment (ΔŷP from the sensory subexperiment; solid circles and solid line) and wV early in the baseline block, suggesting that sensoryweighting and realignment are separable processes that can operate independently. B; in experiment 2, without endpoint visual feedback to necessitate that proprioception be regarded as the misaligned modality to succeed at the task, P endpoint shift (ΔŷP, motor adaptation component = 0, i.e., proprioceptive realignment) was strongly correlated with early baseline wV (correlation r = 0.55, P = 0.01), suggesting that weighting and realignment do operate in conjunction (high wV associated with large ΔŷP) in this situation. *Includes both motor adaptation and proprioceptive realignment.