The brain combines information from multiple sensory modalities to build a consistent representation of the world. The principles by which multimodal stimuli are integrated in cortical hierarchies are well studied, but it is less clear whether and how unimodal inputs shape the processing of signals carried by a different modality. In rodents, for instance, direct connections from primary auditory cortex reach visual cortex, but studies disagree on the impact of these projections on visual cortical processing. Both enhancement and suppression of visually evoked responses by auditory inputs have been reported, as well as sharpening of orientation tuning and improvement in the coding of visual information. Little is known, however, about the functional impact of auditory signals on rodent visual perception. Here we trained a group of rats in a visual temporal frequency (TF) classification task, where the visual stimuli to categorize were paired with simultaneous but task-irrelevant auditory stimuli, to prevent high-level multisensory integration and investigate instead the spontaneous, direct impact of auditory signals on the perception of visual stimuli. Rat classification of visual TF was strongly and systematically altered by the presence of sounds, in a way that was determined by sound intensity but not by its temporal modulation. To investigate the mechanisms underlying this phenomenon, we developed a Bayesian ideal observer model, combined with a neural coding scheme where neurons linearly encode visual TF but are inhibited by concomitant sounds by a measure that depends on their intensity. This model captured very precisely the full spectrum of rat perceptual choices we observed, supporting the hypothesis that auditory inputs induce an effective compression of the visual perceptual space. This suggests an important role for inhibition as the key mediator of auditory-visual interactions and provides clear, mechanistic hypotheses to be tested by future work on visual cortical codes.

In the brain, information from different senses converges on specialized regions where it is combined in a multimodal representation of the environment. However, inputs from a given modality can directly affect processing in a different modality via direct projections from a given unimodal region (e.g., auditory) to a different one (e.g., visual). Here, we investigated the impact of these auditory-mediated influences on rat visual perception. By combining behavioral experiments with computational modeling, we found that pairing task-irrelevant sounds to drifting gratings systematically altered how these visual stimuli were perceived by the animals, in a way that is consistent with an effective compression of the visual perceptual space induced by the auditory inputs. This, in turn, suggests that inhibition mediates auditory-visual interactions at the neural representation level.

A. Rendering of a rat extending its head through a viewing hole in the wall of a sound-isolated operant box, equipped with a display and a speaker for presentation of the audio-visual stimuli. The hole was placed in front of the stimulus display and the speaker (in cyan). The hole also allowed access to three nose-poke ports used to trigger stimulus presentation and report the temporal frequency of the visual stimuli. B. Schematic representation of a behavioral trial. As in previous studies [39,40], the rats learned to protrude their head through the viewing hole and trigger stimulus presentation by touching the central nose-poke port. The visual stimuli were circular, outward-moving sinusoidal gratings that rats classified according to their temporal frequency (TF). Gratings with TF < 2.12 HZ had to be classified as "Low TF"; those with TF > 2.12 Hz as "High TF". Each class was associated with one of the two lateral nose-poke ports. Auditory stimuli did not provide any information about the correct classification of the visual stimuli. C. Stimulus conditions presented to the rats during the test phase of the experiment. These included: 1) visual gratings paired with fixed amplitude sounds (green); 2) visual gratings paired with AM sounds (cells of the square matrix: blue, gray and red refer, respectively, to congruent, incongruent and anti-congruent audiovisual conditions; the shades of gray, from light to dark, indicate AM sounds with progressively larger intensity; see Methods and S2 Fig for details); 3) unimodal purely visual gratings (purple); and 4) unimodal purely auditory AM sounds (cyan). D. Details on the auditory stimuli. Green dot: white noise burst with a fixed maximal amplitude. Gray dots: amplitude-modulated white noise bursts, each with a different temporal frequency of the sinusoidal envelope. Purple dot: absence of an auditory stimulus. See Methods and S2 Fig for details on estimation of average perceived sound intensity. Error bars: s.e.m. across rats. For the modulated noise bursts, different shades of gray are used to group stimuli based on their average sound intensity (same as in C). Insets: sound amplitude control signal for white noise burst (green), the absence of auditory stimulus (purple) and one example of modulated noise burst (gray).

https://doi.org/10.1371/journal.pcbi.1013608.g001

A. The line plot (top) shows the percentage of correct choices of Example Rat 1 in classifying the temporal frequency of the drifting gratings across more than 70 daily training sessions. The color of the dots indicates the training phase (from 2 to 5; see caption), while the intensity of the gray patches denotes the various stages of contrast reduction that were performed during the training (see legend and the Materials and Methods). The dashed horizontal line represents the criterion accuracy (70%) required to progress to the next training phase. To advance, rats had to achieve performance above criterion for at least two consecutive days within the same training phase. The bar plot (bottom) shows the number of behavioral trials that were collected for the rat in each daily session. B. Same as A, but for Example Rat 2.

https://doi.org/10.1371/journal.pcbi.1013608.g002

Task-irrelevant sounds affect rat perception of drifting gratings in a way that is independent of the temporal congruency between the auditory and visual stimuli

Once the animals had become proficient in the TF classification task with 25% contrast gratings, they were moved to the testing phase, where they faced a rich variety of audiovisual stimuli (Fig 1C). The gratings were paired not only with the fixed amplitude burst used during training (Fig 1C, green cells), but also with AM white noise stimuli. These sounds were obtained by modulating the amplitude of a white noise burst with sinusoids having 9 possible TFs (the same of the visual gratings). The rats were presented with all possible pairwise combinations of visual and auditory TFs (cells in the squared matrix of Fig 1C). This yielded trials where the auditory and visual stimuli had either the same TF (blue cells) or different TFs (gray and red cells). The latter included a special pool of anti-congruent conditions (red cells), where progressively larger visual TFs were paired with progressively smaller auditory TFs. Additionally, we also tested unimodal, purely visual trials (purple cells) and unimodal, purely auditory trials with AM sounds (cyan cells). Since the rats were never trained to classify the TFs of the auditory stimuli, with the latter trials we simply measured the spontaneous responses of the animals to the AM sounds, without providing any feedback about their choices (see Materials and Methods). The rats did not spontaneously classify the AM bursts according to their TF in the auditory only conditions, thus confirming the task-irrelevance of the sounds (see S1 Fig).

Overall, this design allowed probing the impact of task-irrelevant auditory stimuli on visual perception along two distinct dimensions: 1) whether sound alters the sensitivity of visual discrimination or, rather, introduces a bias in visual classification; and 2) whether the impact of sound depends on the temporal consistency of audio-visual stimuli or, rather, on the intensity of the auditory stimuli.

When the gratings were paired with the fixed amplitude noise, the group average psychometric curve reporting the proportion of "High TF" choices as a function of the grating frequency was sharp and symmetrical, with the point of subjective equality sitting squarely on the ambiguous TF (Fig 3A, green). Interestingly, pairing the gratings with either the congruent (blue) or anti-congruent (red) AM sounds produced an identical increase in the proportion of "High TF" choices, which was particularly prominent for the highest TFs. This observation was confirmed by a 2-way ANOVA having as factors the audiovisual experimental condition and the grating TF (see the Materials and Methods). Both factors significantly modulated rat choices and so did their interaction (Table 1), thus confirming that the increase of "High TF" choices yielded by the modulated sounds was not homogeneous along the TF axis. A post-hoc Tukey test confirmed that the curves observed for the two conditions featuring AM sounds were statistically indistinguishable. This indicated that the congruency between the TF of the AM sounds and the TF of the gratings did not affect the impact of the auditory stimuli on the visual discrimination task.

This observation led us to group together all the trials where the visual stimuli were paired with an AM sound, regardless of the congruency of their TFs. The resulting psychometric curve (Fig 3B, black) featured an asymmetrical vertical shift, with respect to the reference curve measured with the fixed amplitude sounds (green dots/line), that was equivalent to the one previously observed with the congruent and anti-congruent AM sounds (compare to the blue and red curves in Fig 3A). Strikingly, a similar but even more prominent shift of the psychometric curve was observed for the unimodal visual conditions, when the gratings were presented without any concomitant sound (purple dots/line). Again, a two-way ANOVA showed a significant main effect of experimental condition, TF of the gratings and their interaction, and a post-hoc Tukey test revealed a significant difference between the curve obtained with the unimodal visual stimuli and the curves measured with the paired sounds, either AM or fixed amplitude (Table 2).

These results showed that rat perception of visual temporal frequencies was systematically shifted towards reporting the "High TF" category if the noise bursts became amplitude-modulated and, even more so, when the sounds disappeared altogether. Wondering about the possible cause of these progressively larger shifts, we realized that one key attribute of the sound did decrease from the fixed amplitude to the AM bursts and then further to the purely visual conditions: its intensity. In fact, because of the sinusoidal modulation, there were moments during the AM bursts where the instantaneous sound intensity dropped to zero (S2B Fig). As a result, the average sound intensity (as computed over the inferred reaction time of the animals) was lower for the AM bursts than for the fixed amplitude burst (see Figs 1D and S2C), besides being minimal for the purely visual stimuli (corresponding, in this case, to the environmental background noise). In a previous study [29], noise bursts were reported to hyperpolarize V1 neurons, with the magnitude of this inhibition increasing monotonically with their intensity. Given this observation, we hypothesized that the perceptual effect of sound is to compress visual cortical representations in a way that is proportional to its intensity.

A computational model accounts for the behavioral impact of task-irrelevant sounds as the result of visual perceptual space compression

To test the hypothesis that response suppression is the key mediator of auditory influences on visual representations, we developed a Bayesian ideal observer model of rat perceptual choices. The model rests on three assumptions, which are illustrated in the cartoons of Fig 4 (the full mathematical derivation of the model can be found under the "Ideal observer model" section in the Materials and Methods). The first is that V1 neurons respond to progressively larger TFs with an approximately linear increase of firing rates, at least within the TF range tested in our study (Fig 4A). Although no systematic studies exist about the tuning of rat visual cortical neurons for temporal frequency, this hypothesis is grounded on the periodic activity bursts evoked in rat V1 simple cells by the succession of luminance waves in drifting gratings [43–45]. It is therefore reasonable to assume that gratings with a larger number of cycles (i.e., higher TF) will tend to drive V1 simple cells more vigorously. The second assumption is that sound dampens neuronal responses, thus lowering the slope of the linear dependency of neuronal firing from the TF of the gratings, with this decrease being stronger for higher intensity sounds (Fig 4A; compare the purple line, corresponding to the unimodal, purely visual condition, to the gray and green lines, corresponding to conditions where the visual stimuli are paired with sounds with progressively larger intensity). This hypothesis is consistent with the sound-induced (and intensity-dependent) hyperpolarization of V1 pyramidal neurons reported by [29], as well as with the inhibitory impact of sounds on V1 responses reported by several authors under certain stimulus conditions [30,35,36] (see the Discussion for details). In the representational space defined by the activity of a population of V1 neurons, such response suppression results in a compression of the population vectors encoding the visual TFs towards the origin of the space (Fig 4B; compare the green to the purple distributions). Finally, the third hypothesis underlying our model is that downstream decision neurons learn an optimal discrimination boundary in the representational space (Fig 4B; dashed line) under the sensory conditions they were repeatedly exposed to during training (i.e., with the visual gratings being paired with the fixed amplitude sound; green distributions in Fig 4B). This optimality is what gives our model its character as an ideal observer [46,47]. We also assume that the boundary remains unchanged when new conditions (e.g., the purely visual gratings; purple distributions in Fig 4B) are interleaved with those used for training.

In our model, on each trial, the combination of a visual and an auditory stimulus generates an internal representation sampled from a Normal distribution with fixed standard deviation σ, centered on some average representation of the TF value of the visual stimulus s. Following from the assumptions above, the internal representation is then scaled by a factor γ (see the Ideal observer model section in the Materials and Methods), and the decision rule is the one that would be Bayes-optimal in the training condition (i.e., with the gratings paired with the fixed amplitude sound). We can thus derive the probability for a rat of classifying a given stimulus s as belonging to the H (i.e., "High TFs") category as

where Φ is the standard Normal cumulative function and s₀ is the boundary 2.12 Hz visual stimulus that separates the "Low TFs" class from the "High TFs" class (see the Ideal observer model section in the Materials and Methods). In the model, the σ parameter controls the slope of the psychometric function and measures the internal noise of the representation (i.e., rat sensitivity over the visual TF axis), while the γ parameter controls the gain of the internal perceptual representation.

This model allowed testing alternative hypotheses about the impact of sounds on visual representations, depending on the number of distinct values that the σ and γ parameters were free to take. At one extreme, allowing a different value of σ for each sound condition would represent the hypothesis that sound affects the sensitivity of the visual representation (i.e., the sharpness of the psychometric curves). At the other extreme, keeping a constant σ but allowing a different value of γ for each level of sound intensity would represent the hypothesis that sound compresses the visual representation according to its intensity, but leaves visual perceptual sensitivity unaltered.

Using the model to test for the impact of sound intensity on visual perception required estimating the sound intensity experienced by the rats in the time interval between stimulus onset and each animal’s estimated reaction time (see the Materials and Methods and S2B Fig, second to last row). Analyzing the resulting, trial-averaged intensity levels revealed a finer-grain grouping of stimulus conditions (see Figs 1D and S2C) compared to the one shown in Fig 3B. In addition to the conditions with maximal and minimal intensity (i.e., 74.8 dB for the fixed amplitude sounds and 32 dB for the purely visual stimuli, respectively), we found that the AM conditions could be grouped into three main clusters with increasingly larger intensity, based on the TF at which the noise bursts were modulated: TF = 0.25 Hz (48.5 ± 2.4 dB; mean ± s.e.m. across rats); TF = 0.82 Hz (64.2 ± 2.3 dB); and 1.32 Hz ≤ TF ≤ 4 Hz (70.7 ± 0.5 dB; these conditions are shown with progressively darker shades of gray in the stimulus matrix of Figs 1C, 1D and S2).

We inferred the parameters of the ideal observer using a hierarchical Bayesian approach (see the section Ideal observer model of the Materials and Methods, S3 and S4 Figs for details on the model and our inference approach), which yielded a posterior probability distribution over the values σ and γ that characterized each specific rat (referred to as rat-level parameters), as well as the values that characterized all rats taken together (referred to as group-level parameters). The group-level estimates obtained for a model with a single σ and 5 different γ per rat (i.e., one γ for each of the 5 levels of sound intensity in our stimulus set) are summarized by the psychometric curves shown in Fig 5A.

The curves fitted tightly the measured classification accuracies and captured well the way the proportion of "High TF" choices progressively increased, while sound intensity dropped from maximal (green curve; fixed amplitude sound) to minimal (purple curve; no sound), passing through the three intermediate levels in which the AM sounds were clustered (gray curves) (Fig 5; see Materials and Methods and S5 Fig for a posterior predictive analysis illustrating the tightness of the model fit to the data, and S1 Table for details on fitted model parameters and inference diagnostics). This means that an ideal observer with constant sensitivity (i.e., a single σ) could describe well the impact of sound on rat visual perceptual choices, provided that the internal representation of visual TFs was appropriately scaled by a gain factor (γ). Importantly, the magnitude of this gain decreased monotonically across the 5 levels of increasing sound intensity in which our stimulus conditions could be grouped (Fig 5B).

To verify that this was the best description of the data, we compared the predictive power of this "reference" model to that of other variants, with a different number of values that σ and γ were allowed to take. The predictive power of each model was measured using the expected log predictive density (ELPD), that is, the average, expected log-likelihood of the model on new data. This metric is estimated by leave-one-out cross-validation using a standard approach and automatically penalizes overly complex models that tend to overfit (see section Model comparison in the Materials and Methods). In our comparison, we computed the difference in ELPD between the reference model with 1 σ and 5 γ and each other model. We refer to this metric as ΔELPD.

The reference model with a fixed σ and 5 γ (one for each of the 5 levels of sound intensity) was the best (i.e., the one with the lowest ΔELPD) of a broad range of models we tested (Fig 5A, inset). In particular, allowing a separate γ for each of the 9 AM conditions, thus obtaining a total of 11 possible γ values (while still keeping σ fixed to a single value), yielded a worse model (an indication of overfitting). This confirmed that noise bursts modulated within the [1.32 Hz, 4 Hz] range were all equivalent in terms of their impact on visual perceptual choices, as expected given their very similar intensity (see Figs 1D and S2). A model with 5 σ and 1 γ also performed much worse than the reference model, thus confirming that the different sound stimuli did not impact rat sensitivity to the visual TFs, but, rather, the dynamic range of the internal representation of the TFs. This conclusion was further reinforced by noticing that not even a model with 5 σ and 5 γ outperformed the reference model with only 1 σ and 5 γ. Finally, in order to describe more quantitatively the relationship between perceptual gain and sound intensity, we also compared the models just described with one that puts a linear constraint on the dependence between these two quantities (while keeping σ fixed as in the best-performing model seen so far). This model performed just as well as the () model, confirming quantitatively that γ decreases for larger values of sound intensity (slope: , intercept: , posterior mean ± standard deviation; S6 Fig).

In summary, our Bayesian ideal observer analysis supports the hypothesis that the effect of sound on the representation of temporally modulated visual gratings is to compress the visual perceptual space rather than increase or decrease perceptual uncertainty. Mechanistically, this finding suggests that sound acts as a powerful modulator of the responses of visual cortical neurons (see Fig 4) but leaves their tuning for temporal frequency unaltered.

Activation of A1 (in cyan) by sound bursts leads to an inhibition of the responses evoked by circular drifting gratings in V1 (in green) via cortico-cortical projections (orange line). This produces a decrease in the gain of the tuning curves of V1 neurons over the temporal frequencies of the gratings.

https://doi.org/10.1371/journal.pcbi.1013608.g006

Subjects

All animal procedures followed international and institutional standards for animal care and use in research and were approved by the Italian Ministry of Health (project approval 940/2015-PR on September 4, 2015; project approval 1183/2020-PR on December 4, 2020).

We trained 12 adult Long Evans rats (Charles River Laboratories) in a visual temporal frequency (TF) classification task. The animals, seven weeks old on arrival, weighed approximately 240 g each. They were housed three per cage in a climate-controlled chamber (23 ± 1° C, 40-50% humidity). Training began at 10 weeks, following two weeks of handling to accustom them to human interaction. Rats always had free access to food (20 g/day) but their access to water was restricted in the days of the behavioral training/test (5 days a week). Water was provided as a reward for correct responses in the discrimination task and for about 30 minutes after each session, ensuring at least the recommended 50 ml kg⁻¹ daily intake. During the course of the study, their body weight increased to about 500 g. The duration of the training phase (see below) varied depending on the animal. Five rats completed it in about 3 months and proceeded to the test phase (see below), while for the remaining rats it took about 6 months. The test phase lasted approximately 12 months for all the rats.

Behavioral apparatus

Rats were trained in a rig made of two identical racks, each with three vertically stacked slots, allowing six animals to be trained simultaneously. Each slot contained a custom-built black operant chamber (×ばつ30 cm) with a 4 cm-diameter viewing hole for the rats to extend their heads and face the stimulus display (Fig 1A). The visual stimuli were presented on a 21.5" LCD monitor (ASUS VE228) placed at 30 cm from the viewing hole, while the auditory stimuli were presented from a full-range speaker (Visaton BF37, 4 Ohm, 1.5" diameter, frequency response 100–20000 Hz) positioned above the monitor, tilted 45° towards the viewing hole, and located 35 cm from it. Each speaker received a pre-amplified signal (Q-BAIHE TDA7377PRO). A 3D-printed block with three equidistant response ports was positioned at 3 cm from the viewing hole. Each port had a stainless-steel feeding needle and a LED-photodiode pair that worked as a licking sensor to detect interactions of the rats with the port. Signals from the photodiodes were digitized with a microcontroller (Phidgets 1203 input/output device) and sent to a computer running the open-source MWorks software (https://mworks.github.io/). This system automatically presented the visual and auditory stimuli when the animal’s nose was detected in the central port, while the lateral ports were used to record the responses of the animals to the visual stimuli and decide whether to deliver the reward, along with reinforcement sounds (Fig 1B). Correct responses were rewarded with a 4% stevia-water solution from the lateral feeding needles connected to computer-controlled syringe pumps (New Era Pump Systems; NE-500). Each animal was individually monitored with a camera (AXIS M1033-W Network Camera) positioned 14 cm above the chamber viewing hole. The operant chambers were acoustically insulated from each other and the environment with sound-proof foam panels.

Visual and auditory stimuli

Rats were presented with grayscale outward-moving sinusoidal gratings (30 fps) with a spatial frequency of 0.05 cycles deg⁻¹ and temporal frequencies of 0.25, 0.82, 1.32, 1.75, 2.12, 2.48, 2.91, 3.41, and 4 Hz (the temporal frequency of the gratings can be defined as the number of grating cycles moving through a point of the screen in 1 second). These frequencies were symmetrically distributed around 2.12 Hz on a logarithmic scale (Fig 1C, x axis). The gratings, displayed on a middle-gray background, were enveloped with a Gaussian circular aperture with diameter corresponding to the monitor height (see examples in Fig 1A and 1B).

Two classes of auditory stimuli were used in our study: a fixed amplitude sound (Fig 1D, green curve) and amplitude-modulated sounds (see Fig 1D, gray curve, for an example; see S2A Fig, gray curves, for the complete set of stimuli). In both cases, the stimuli were generated in Matlab (https://www.mathworks.com/) from a full-range white noise sampled at 44.1 kHz. In the case of the AM sounds, the white noise time series was used as a carrier and multiplied with positive sinusoid envelopes (the modulators), having the same temporal frequencies of the visual stimuli (Fig 1C, y axis). More in detail, sound stimuli were generated as digital control signals, sampled at 44.1 KHz in arbitrary units, which were then converted in WAV format (16 bits per sample). The WAV files were then passed through our custom-built amplifier and loudspeaker (see "Behavioral Apparatus" and Fig 1A).

The control signal was generated starting from a white noise sequence. We used two distinct white noise sequences, one for the fixed amplitude stimulus and one for the AM stimuli. In the fixed amplitude condition, the white noise sequence (denoted as ) was composed of independent and identically distributed random variables sampled from a normal distribution with a sampling rate matching that of the control signal ( kHz). The distribution had parameters and . Before conversion to WAV format the sequence was normalized to have values in the interval [–1,1], hence giving the distribution an effective standard deviation . The same sequence was used in all the "V + fixed amplitude sound" trials.

The white noise carrier of the amplitude-modulated sounds, denoted as a_AM(t), was composed of independent and identically distributed random variables sampled from a uniform distribution in an interval [–1,1], again with a sampling rate of kHz. The same sequence was used in all the trials of the modulated conditions. The sequence a_AM(t) was multiplied by a modulation function , where is the angular frequency of modulation and is the modulation frequency of condition n (S2A Fig, gray curves). The modulation was non-negative and bounded in [0,1], ensuring that the modulated signal remained within [–1,1].

Estimation of sound intensity

We measured the intensity of the sound stimuli with a phonometer (Tadeto SL720). At any point in time, the phonometer provides an instantaneous estimate of the sound intensity based on a characteristic sampling window (0.125 s). To avoid any interactions between the temporal modulation of our sound stimuli and the finite sampling rate of the phonometer, we devised and calibrated a transformation that estimated the instantaneous sound intensity as a function of the nominal control signal. This transformation takes into account that the control signal a(t) is converted into a sound pressure wave p(t) by the amplifier/speaker, assuming minimal distortion (i.e., that there is a linear relationship between a and p), and that the measurements of sound intensities L(t) returned by the phonometer are in dB - i.e., they are a nonlinear function of p(t). We note here that our dB measurements done using "A" frequency weighting, as in previous studies [29].

To carry out this computation, we first verified that the relationship between the nominal control signal a(t) and p(t) as measured by the phonometer was linear. To do so, we measured with the phonometer the intensity L of a uniform white noise sequence lasting 10 seconds. The long duration ensured that we got a precise estimate of the intensity in dB. We repeated the measurements 10 times, each time dimming the control signal a by a factor of 0.8. All these measures were taken by placing the phonometer inside the sound-proof operant chambers, in the same position where the head of the rats was located during the task. Under the same settings, we also measured the intensity of the ambient noise inside the operant chambers (32 dB). The relationship between the sound intensity L (in decibels) of a white noise sequence and its sound pressure level p (in Pascals) is given by:

where is the root-mean-square (RMS) of p(t) over a time window τ, p₀ = 20 μP is the reference pressure, and is the pressure of the ambient noise. Inverting the equation, we can compute the sound pressure from the intensity measurements. The ambient sound pressure is given by:

The sound pressure of the white noise sequences is, therefore:

We could thus compute p from the measured values of L, finding that, as expected under the assumption of a linear mapping from a to p, the average pressure for each sequence was times as intense as the previous one. This confirmed that for some value of f and allowed us to estimate the factor f of the linear relationship as f = 0.39 P. We repeated the same procedure with gaussian white noise sequences, confirming the result and obtaining a compatible estimate for f. We could then use the estimated value of f to calculate the sound intensity (in dB) of all sound stimuli used in the experiment. For the fixed amplitude, white noise sound , this results in dB. For the amplitude-modulated stimuli, their sound intensity profile is in principle influenced by the duration τ of the time window over which the RMS of is computed. For short values of τ, close to the white-noise-sampling period (), noise fluctuations dominate the temporal profile. By contrast, long τs, comparable to the period of the modulation, will effectively smooth the modulation, resulting in values that do not accurately represent the dynamic characteristics of the signal. For a wide range of intermediate time windows, however, the computed sound intensity is independent of the details of the fluctuation of the carrier and its temporal profile correctly tracks the temporal modulation.

More specifically, the RMS of over a time window τ is defined as

If τ is smaller than the temporal scale of the modulation, then m(t) is approximately constant inside the integration window and

On the other hand, if τ is larger than the sampling rate of the white noise () there will be enough samples in the window for the RMS of to be equal to the uniform distribution average:

Therefore, the resulting temporal intensity profile of the AM sounds (in dB), when using one of the intermediate values of τ can be computed as:

The sound intensity profiles for each modulation frequency (computed with ms ), as well as the profiles of and (with the latter corresponding to the "visual only" condition) are shown in S2B Fig.

To estimate the sound intensity experienced by a rat during each trial, we computed the average intensity of the trial condition n throughout a specific time window with duration T. This duration was equal to the reaction time (RcT) of the animal in that given trial, i.e., to the period during which the rat was exposed to the sound stimulus before initiating a motor response. The reaction time is defined as:

where RT is the response time (i.e., the time from the trial onset to the moment a response port is reached), and MoRsT is the motor response time (i.e., the time taken by the animal to reach the selected response port, after leaving the central port for the first time). In the recorded data, only the response time was directly measured for each trial. To estimate the reaction time, we assigned to MoRsT a fixed value across all trials, as determined in one of our previous studies, which carefully mapped the trajectories and durations of rat head movements in a similar experimental setup [53]. Specifically, we used MoRsT = 0.3 s, which corresponds to the duration of a direct ballistic movement from the central port to either response port. This type of movement was the most frequently observed in that study (1149 out of 1359 recordings). Hence, for each trial i, we computed the average perceived intensity as − . An example of the computation is shown on the "0.25 Hz AM sound" row of S2B Fig.

The resulting values for each trial are shown as scatter plots in S2C Fig, along with violin plots to illustrate their distribution in each sound condition for every individual rat (the dashed lines report the rat group average intensities). The averages of these sound intensities obtained for each rat are those reported on the x axis of Fig 5B (squares, triangles, and crosses). Note, however, that, in that figure, the AM conditions with modulation frequencies 1.32 Hz TF Hz have been grouped together, given that their average intensity was virtually identical (as shown in Fig 1D).

Behavioral task

The rats learned to initiate a behavioral trial by nose-poking the central port, and to approach the lateral ports to classify the visual stimuli based on their temporal frequency (Fig 1B). The "low temporal frequency" class (0.25-1.75 Hz) and the "high temporal frequency" class (2.48-4 Hz) had to be reported by half of the rats by poking, respectively, the left and right port, while the other half was trained with the opposite association. To initiate a trial, rats had to keep their nose in the central port for 300 ms, followed by a 100 ms tone (500 Hz pure tone) signaling the stimulus presentation, occurring 200 ms later. After stimulus onset, rats were given 2 seconds to make a response; responses within 300 ms were aborted to prevent impulsivity. Correct responses were rewarded with 4% stevia-water solution and a concomitant reinforcement sound, while incorrect responses were followed by an unpleasant sound (400 ms, 300 Hz square-wave) and a 1-3 second timeout period. A middle-grey background was displayed during non-stimulus periods. In a given trial, every stimulus had an equal probability of being presented, but with the constraint that a maximum of three consecutive presentations of the same visual class was allowed to prevent the development of a bias towards one of the response ports. This trial structure was the same for all the training and test phases of the experiment, described in the next section. Stimulus presentation, response collection, and reward delivery were controlled by the MWorks software (https://mworks.github.io).

Experimental design

Statistical analyses

The classification accuracies achieved by the rats in the test conditions described in the previous section were statistically compared using a repeated measures, two-way ANOVA (see Fig 3 and Tables 1 and 2). To perform the ANOVA, we first calculated the probabilities of each rat to classify a visual stimulus as belonging to the "High TF" category across all experimental conditions. This resulted in 10 data points for each of the stimulus conditions analyzed in Fig 3A and 3B, corresponding to the 10 rats in our study. The ANOVA analysis requires two key assumptions for the data distributions:

1. Normality, i.e., the data should follow a Gaussian distribution. This is particularly important for proportions of correct choices, as distributions near 0 and 1 can saturate and deviate from normality. To address this issue, we transformed the psychometric data using the arcsine of the square root of each value. After the transformation, we tested the normality of the distributions with the Shapiro-Wilk test [54] and we verified that it was satisfied by each data point distribution.
2. Homoscedasticity, i.e., the variance should be equal across experimental conditions. The arcsine transform also helps stabilize the variance for proportional data. This was assessed using Levene’s test. Following the transform, the test could not reject the null hypothesis that all distributions had equal variance (p = 0.3421), indicating that the assumption of homoscedasticity was satisfied.

We then proceeded to calculate the two-way ANOVA with repeated measures. Our experimental design involved two independent variables: the TF of the visual stimuli, with 9 levels, and the experimental condition, with 3 levels. The latter were as follows. For the analysis carried out in Fig 3A: the "V + fixed amplitude sound", the "V + AM sounds (congruent)" and the "V + AM sounds (anti-congruent)"; for the analysis carried out in Fig 3B: the "V + fixed amplitude sound", the "V + AM sounds" and the "V only". The dependent variable was rat response, measured as the probability of classifying a visual stimulus in the "High TF" category for each experimental condition, which was recorded for the same 10 animals across all the different levels of the independent variables. Tables 1A and 2A present the results of this analysis for the data shown, respectively, in Fig 3A and 3B, reporting, in both cases, significant effects for both the visual TF and the experimental condition variables, as well as a significant interaction between them. This indicates that the dependent variable changed across different visual TFs and experimental conditions, and that there was a combined influence of visual TF and experimental condition on the dependent variable.

Finally, to determine which specific levels of the experimental conditions contributed to the main effect reported in the ANOVA table, we conducted post-hoc tests. We performed pairwise comparisons of the different levels of the experimental conditions and evaluated their effects using the Tukey test to control for multiple comparisons, as summarized in Tables 1B and 2B.

Ideal observer model

In this section we derive the ideal observer model used to predict the responses of the rats in the visual rate classification task. The ideal observer follows a standard structure for a sensory classification task, based on the assumption that an internal, abstract representation of the visual stimulus, called the measurement and indicated by x, can be modeled as a Gaussian random variable (an approach that is firmly established in psychophysical modeling, going back to its roots in signal detection theory, see e.g. [46,47,55–57]). The main nonstandard element of our analysis is the added hypothesis that x is scaled by a gain factor γ which can depend on the intensity of the auditory stimulus. This additional hypothesis stands on its own on an abstract level, but we show below that it can be derived as the consequence of a putative suppressive effect of sound on the activity of visual cortical neurons.

On each trial of the two-alternative forced choice (2AFC) task, the visual stimuli were assigned a temporal frequency which was randomly chosen from a pool of 9 possible values (0.25, 0.82, 1.32, 1.75, 2.12, 2.48, 2.91, 3.41 and 4 Hz). In the following, we will denote this visual temporal frequency by s, for stimulus. The goal of the animal was to accurately classify each visual stimulus as belonging to the low frequency class (L, for gratings with frequencies from 0.25 to 2.12 Hz) or to the high frequency class (H, from 2.12 to 4 Hz). Each of the 9 frequency values had an equal chance of being presented in a trial. The central value was randomly assigned to the H or L class, and therefore also H and L had an equal chance of being presented. The probabilistic structure of the task can also be restated as follows. On each trial, the stimulus class C is chosen with equal chance to be H or L, that is

Based on the class, the visual stimulus s presented to the animal is selected with probability

where

and for convenience with our derivations below we order the index k symmetrically around s₀, such that , , etc and , , etc. Our task here is to compute the probability that the observer chooses the class "high visual frequency" (), when the frequency of the visual stimulus is s and the sound condition is n.

We start by modeling the process that on each trial gives rise to the measurement x, given the visual stimulus s and the sound condition n. We initially disregard the effect of the sound, which we will add back later. We assume that a stimulus s gives rise to a certain pattern of firing rates in V1, where r_i is the rate of the i-th neuron. The population response is normally distributed around a stimulus-dependent value : , where Σ is a generic (but stimulus-independent) covariance matrix.

The subject generates an internal representation (the measurement) x of the external stimulus s based on the activation pattern . We assume that this measurement is a projection of the vector of activities on some measurement axis ; in other words,

This is biologically plausible if, for example, the measurement x is the activity of a readout neuron with connectivity strength with each neuron coding s. Ideally, the measurement axis y used by the subject is the one that allows to best discriminate the stimulus classes, therefore allowing for the best performance, but for the purpose of developing the model we do not need to make any particular assumptions on and . Because x is an affine transformation of a normally-distributed random variable, it is itself a normally distributed random variable:

If the projection operation is itself noisy and introduces additional Gaussian noise of standard deviation τ, the distribution of the measurement becomes simply

In the following, we will lump both sources of variability into a single parameter ρ,

We will conceptualize ρ as the general level of sensory noise present in the system for the purposes of this task. The distribution of the measurement will therefore be

where we have defined

for notational convenience.

To decide if a stimulus has high frequency (H) or low frequency (L), we assume that the subject performs Bayesian inference to compute a posterior probability over the classes H and L given the measurement and reports the class with highest posterior. In our case (see below for a detailed justification), this will be equivalent to finding a decision boundary x* such that , and reporting H if x > x* and L if x < x*. On average over the distribution of the measurement given the stimulus, the probability of reporting a given class given the stimulus (which we may also call the psychometric function) will then be

and − . We will now calculate x* and consequently we will derive .

Let’s consider the simplest possible scenario for , namely the case where is linear in s:

In this case, the mapping between s and x also becomes linear:

with and .

We will now show that when is linear the symmetry of the problem and the shape of the response distributions imply that x* is unique and .

If we compute the posteriors explicitly from the class-conditional stimulus distributions in Eq (11) we get

Note now that the symmetry of the class-conditional stimulus distributions in Eq (11) is such that

Also, for any value of x and s,

which means that, if b > 0 (and simply reversing the inequalities if b < 0),

Therefore,

Thus as a consequence of the symmetry of the problem the decision boundary x* is simply

Hence:

with

As shown in Fig 4, we model the effect of an external sound stimulus n as a suppression of the activity of V1 neurons.This suppression may result from direct inhibition of V1 neurons by auditory neurons, but it could arise from indirect effects as well. We model it by multiplying the mean response of the neurons by a sound-dependent factor , while keeping fixed the decision boundary x* learned during the training sessions. By substituting with in the integral above, we get:

with

Now, if the circuit computing x is adapted to extract the maximum amount of information from in this task, at least in the case where the noise for is isotropic, will be parallel to (this may not be true in presence of nontrivial noise correlation structure). On the other hand, if is a random high dimensional vector with no particular relation to , we can assume that with high probability ⋅ . Therefore, under these simple assumptions, ⋅ ⋅ . In this case, the expression for the probability of reporting H in presence of auditory stimulus n reduces to

where and σ are free parameters of the model. Finally, we note that while the linearity assumption for made in this section can seem restrictive, the derivations above remain valid as long as ⋅ is (approximately) linear over the range of values of s that are relevant in the task.

Lapse rates. We extended the ideal observer to account for lapses by defining two lapse rates and , such that on each trial the observer has a probability 1 − − of actually engaging in the task, a probability of reporting H independently of the stimulus, and a probability of reporting L independently of the stimulus. Therefore,

and

Inferring the model parameters from experimental data

We used the ideal observer model derived above to describe the data from our experiment. According to the formula for , the behavior of each rat r is characterized by some parameters . We inferred the value of these parameters using a hierarchical Bayesian approach [58,59], for which we now give a brief conceptual overview (refer to S3 Fig for a graphical depiction).

Denote by D the set of recorded behavioral data, with being the choice (H or L) of rat number on trial number . We start by defining, for any rat, a prior probability for the parameters:

We also define a likelihood function for the parameters . The likelihood is simply the probability of observing the rat choices , given a certain value of and of the visual and auditory stimuli provided to the rat on all trials (respectively, and ), seen as a function of :

where and are given by Eq (49) and Eq (50). The likelihood function therefore encapsulates the ideal observer model, developed in the previous section. By applying Bayes’ theorem, we compute a joint posterior probability distribution for the parameters:

(from now on, for convenience, we will omit the explicit dependency on s and n in the likelihood). However, since the rats belong to the same population and are subjected to the same experimental conditions, we assume that the parameters describing different animals will be related. In other words, gathering information about one rat tells us something about that specific rat, but should also inform us on what to expect about other rats, or rat behavior in general. We can build this assumption into the model by imposing that the parameters associated to different rats are all sampled from the same distribution, leaving the parameters of this higher-level probability distribution to be also inferred from the data. Besides extracting "population-level" information from the data of each rat, this procedure will also help constraining the rat-level parameters to reasonable values given what we observed in the other rats, thus acting as a regularizer for the inference process. More formally, our posterior will be:

where are the rat specific parameters for all R rats, and η are the common (population) parameters.

More concretely, the priors we assign to the parameters ( in Eq (56)) are:

It follows that our population parameters η are the mean and the standard deviation of the parameters, and and for the perceptual noise. Following standard practice [58,60], we assign weakly informative priors to the population-level parameters:

Finally, in this hierarchical scheme we also need a global likelihood function ( in Eq (56)) which we obtain simply as the product of the rat-level likelihoods:

Prior predictive check. We conducted a prior predictive check [58] to evaluate whether the model’s prior distributions were consistent with plausible observations, even before any actual data was considered. The check was performed by sampling 1000 sets of model parameters from their prior distributions and using them to generate synthetic data. The results showed that our assumed priors are flexible enough to allow a full range of behavior for the psychometric curves, and have no bias towards supporting the central claim of our paper, namely the dependence of the gain on the sound intensity (S4 Fig).

Sampling the posterior distribution. We performed inference with a Markov Chain Monte Carlo (MCMC) method, using the NUTS algorithm [61], as implemented in PyMC version 5.2.0 [62]. This method samples parameters from the posterior distribution : with a high enough number of samples, the distribution of the samples matches closely the distribution of the posterior. We sampled 4 independent chains of 2000 draws each, 1000 of which we used for tuning the algorithm and subsequently discarded, and 1000 as actual samples of the target posterior distribution. The target acceptance probability parameter for NUTS was set at 0.99, and no divergences were detected during sampling.

Posterior predictive check. Following the model’s inference, we performed a posterior predictive check [58] to evaluate the goodness-of-fit to the observed data. The check was performed by sampling a set of synthetic observations from each of the 1000 parameter posterior draws saved in the Markov chain. The results showed that the simulated data was consistent with the observed data, indicating that the model provides a robust description of the observed rats’ behavior (S5 Fig).

Model comparison

To verify that the derived model describes well the experimental data, we compared it with other models that could also reproduce the psychometric curves of the rats. To perform the comparison, we used a Leave-One-Out Cross-Validation (LOO) estimate of the Expected Log pointwise Predictive Density (ELPD)[63]. LOO estimates how well, on average, the model describes data it hasn’t been trained on. To do so, it computes how well a model trained on all the trials (across all animals) except one predicts the outcome of that trial, and averages this value over all the trials. The ELPD therefore automatically balances goodness of fit with model complexity, penalizing overly complex models that tend to overfit the noise in the data. More formally, the LOO estimate of the ELPD for a model for which we can sample the posterior distribution with MCMC is defined as:

where is the total number of trials in the data, S is the number of posterior samples in the Markov chain for model , is the likelihood function of the model, d_i is the data from the i-th trial and is the j-th posterior sample of a Markov chain generated from all data except d_i (note that each posterior distribution sample, including , is a high-dimensional vector containing one value for each parameter in the model). In Eq (66), the argument of the logarithm represents the likelihood of the model for the datapoint d_i, averaged over the posterior distribution over θ (where this posterior is obtained without looking at datapoint d_i). Indeed, since the training of results in a posterior distribution rather than a single parameter vector, all quantities related to must be averaged over this distribution. For MCMC training, this average is computed as a sum over all sampled parameters, given that the samples follow the posterior distribution:

In practice, computing Eq (66) requires training the model N times, one per trial: given the large number of trials in our experiment, we approximated the leave-one-out procedure with pareto-smoothed importance sampling (PSIS-LOO) [63].

Computing the for different models gives us a relative measure of how well they describe the data. As shown in the inset of Fig 5A, the model with the highest is the one with 5 distinct γ parameters, one for each sound condition whose power is in a certain range (see main text for details).

We also compared this model with a model that incorporates the additional λ parameter discussed in the ideal observer derivation above (Eq (47)). This parameter is included with an analogous hierarchical structure to that of σ in the main model discussed above, such that each rat has its own , with

This comparison allowed us to test the hypothesis that, to the extent that the assumptions about stimulus encoding made in that section are valid, the vector of resting activities of neurons has no effect on the projection of measurement. The resulting model performance is indeed comparable to the model performance ( ; posterior mean ± std. err.). Moreover, the inferred population value of λ in the model is consistent with 0: the posterior estimate is (mean ± std. err.), and the probability of direction is . The probability of direction is the proportion of the posterior distribution for a parameter that has the same sign as the median, and in simple regression scenarios it is related to a frequentist two-sided p-value () [64].

S1 Fig. Rats are not sensitive to the temporal frequency of the unimodal, purely auditory stimuli.

Group average proportion of "High TF" choices (n = 10 rats) as a function of the TF of the unimodal auditory stimuli. Error bars are SEM.

https://doi.org/10.1371/journal.pcbi.1013608.s001

(PNG)

S2 Fig. The auditory stimuli and their intensity.

Each row refers to a distinct auditory stimulus used in the experiment: 1) the white noise burst with fixed maximal amplitude (top row; green); 2) the amplitude-modulated white noise bursts, with the 9 different temporal frequencies of the sinusoidal envelops (middle rows; gray); and 3) the absence of the auditory stimulus (bottom row; purple). A. Waveforms of the auditory stimuli used to drive the speakers that delivered the sounds to the rats (note that, for the sake of visualization, the sounds are not shown at their actual sampling frequency of 44.1 KHz but they have been downsampled to 500 Hz). B. Intensities of the auditory stimuli in dB as a function of time, as computed based on the waveforms shown in A and the measured intensity of 10 s long white noise bursts with different amplitudes (see the Materials and Methods for details). The panel referring to the TF of 0.25 Hz illustrates how the average sound intensity experienced by a rat in a given trial was computed, based on the reaction time RcT of the animal. The latter was obtained by subtracting the estimated motor response time MoReT to the measured response time RT. C. Distributions and estimates of the average sound intensities experienced by each rat across all trials recorded for any given stimulus condition. Scatter plots show the estimated sound intensity experienced by the rats in each individual trial. Violin plots represent the full probability density distributions of these estimates while box plots show their median and interquartile range. The dashed line indicates the across-rat average intensity per stimulus condition.

https://doi.org/10.1371/journal.pcbi.1013608.s002

(PNG)

S3 Fig. Schematic of Bayesian hierarchical inference for our ideal observer model.

On the left, each row is labeled as representing the population parameters priors (), the subject-level parameters priors () or the likelihood (). On the right, each bubble represents a parameter of the model. The name of the parameter and its distribution are reported. The rectangular plates indicate that multiple iid variables have been grouped. The group name and the corresponding number of variables are indicated on the bottom left of each plate. Each arrow represents the dependencies of the model. Gray coloring indicates that the variable is conditioned to observations.

https://doi.org/10.1371/journal.pcbi.1013608.s003

(PNG)

S4 Fig. Prior predictive simulation of the model.

In black the average proportion of answers "High" of the rats. Error bars denote the s.e.m. across all rats. Each colored line is the average predicted psychometric curve obtained for one of the five noise condition, generated by parameters sampled from its respective prior distributions. The γ and σ parameters are drawn from the population average parameters; the ε parameters, given the absence of a hierarchical structure, are averaged across all rats. The shaded areas represent one (dark area) and two (dim area) standard deviations from the mean. The five curves all look almost identical (up to sampling noise) as the priors for the noise conditions were the same.

https://doi.org/10.1371/journal.pcbi.1013608.s004

(PNG)

S5 Fig. Posterior predictive simulation of the model.

In black the average proportion of answers "High" of the rats. Error bars denote the s.e.m. across all rats. Each colored line is the average predicted psychometric curve obtained for one of the five noise condition, generated by the parameters sampled in the MCMC chain, following the respective posterior distributions (same as the solid lines displayed in Fig 5A). The γ and σ parameters are drawn from the population average parameters; the ε parameters, given the absence of a hierarchical structure, are averaged across all rats. The shaded areas represent one (dark area) and two (dim area) standard deviations from the mean of the posterior.

https://doi.org/10.1371/journal.pcbi.1013608.s005

(PNG)

S6 Fig. Posterior prediction of the linear relationship between sound intensity and perceptual gain factor in an ideal observer model.

The solid line represents the population-level gain parameter , modeled as a linear function of the average sound pressure level I. The slope of the line is , the intercept (posterior mean ± st. dev.). The color gradient along the line maps the specific sound conditions: green for the fixed amplitude condition, grays for the V+AM conditions, and purple for the V-only condition. The dashed lines represent one standard deviation of . The shaded area around the regression line represents the posterior mean of the between-subject standard deviation, . This area illustrates the estimated population-level variability. The symbols (triangle, square, cross) denote the relative average perceived sound pressure (x axis) and the posterior means (y axis) for individual rats. The inset shows the relative distance in Expected Log-Predictive Density (ELPD) between the linear model and our best model (, error bars denote the standard deviation of the difference.

https://doi.org/10.1371/journal.pcbi.1013608.s006

(PNG)

S1 Table.

Posterior summary statistics for the key population-level parameters of the model. The columns represent the posterior mean (mean), posterior standard deviation (sd), the 3% and 97% boundaries of the Highest Density Interval (hdi_3%, hdi_97%), as well as a number of standard diagnostics for MCMC-based Bayesian inference, namely the Monte Carlo Standard Error for the mean and standard deviation (mcse_mean, mcse_sd) [58], the bulk and tail Effective Sample Size (ess_bulk, ess_tail) [65], and the R-hat convergence diagnostic (r_hat) [65]. All diagnostics were computed with ArviZ version 0.15.1 [66]. Note that the values are not population parameters in the same hierarchical sense as the others; instead, they represent the simple average of the individual rat ε values estimated, as the ε parameters were modeled without a group-level distribution.

https://doi.org/10.1371/journal.pcbi.1013608.s007

(CSV)