Phase inference from point process event timing (PIPPET)

PIPPET is the problem of dynamically estimating a hidden noisy phase variable based on the timing of events generated as a point process whose rate is modulated as a specific function of phase. The generative model consists of a phase that advances as a drift-diffusion process: (1) and an inhomogeneous point process that generates events with probability λ(ϕ). This function is known to the observer. We will refer to λ(ϕ) as an “expectation template” because it describes the temporal structure of the observer’s event expectations, though it can also be understood as a hazard rate for events. To achieve both analytical tractability and flexible descriptive power, we assume that λ(ϕ) is a sum of a constant λ₀ and a set of scaled Gaussian peaks indexed by i = 1, 2, … etc. Each Gaussian peak i is centered at a mean phase ϕ_i with variance v_i and scale λ_i: (2) where φ(⋅|m, v) denotes the pdf of a Gaussian distribution with mean m and variance v.

• Each Gaussian mean ϕ_i represents a phase at which an event is expected;
• λ_i represents the strength of that expectation;
• and is the temporal precision of that expectation.
• λ₀ > 0 represents the rate of events being generated as part of a uniform noise background unrelated to phase.

The point process with rate described by (2) can be understood as a sum of independent point processes i, one for each expectation peak and one for the uniform background process with rate λ₀, whose events are indistinguishable. The mathematics of updating a phase estimate at an event can be understood to involve a causal inference on which of these processes caused each event.

λ(ϕ)dt is the likelihood function over ϕ associated with the occurrence of an event, so λ(ϕ) is a rescaled likelihood function. See Fig 1A for illustration.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Illustration of the PIPPET filter.

A) In the PIPPET generative model, λ(ϕ) represents the instantaneous rate of events occurring when the underlying temporal process is at phase ϕ. This is assumed to be a sum of Gaussian-shaped functions with means ϕ_i representing the phases at which specific events are expected, variances v_i representing (the inverse of) the temporal precision of the expectations, and scales λ_i representing the strength of the expectations. A constant λ₀ is also added, representing the instantaneous rate of events unrelated to phase. B) At any time t, the filter’s estimate of current phase p_t(ϕ) is forced to be a Gaussian with mean μ_t (the estimated phase at time t) and variance V_t (the level of uncertainty about the phase estimate). C) These allow us to define a subjective hazard rate Λ (implicitly a function of time) representing the degree to which an event is anticipated at t, and conditional subjective hazard rates Λ_i representing the degree to which an event is anticipated from peak i. These hazard rates become less precise as phase uncertainty V_t increases. D) Each peak i of λ is associated with a “candidate posterior” with mean and variance —this would be the posterior on phase if the event were known to come from peak i. E) At an event, the phase distribution resets to a Gaussian with mean and variance . These incorporate the influences of each candidate posterior, and can increase if the cause of the event is ambiguous (as dramatically illustrated above). F) Between events, μ_t increases at rate 1 and V_t grows at rate σ². Additionally, μ and V are pushed away from and with a strength proportionate to subjective hazard rate Λ.

https://doi.org/10.1371/journal.pcbi.1009025.g001

Note that ϕ is assumed to be on the real line, not the circle. This design decision allows PIPPET to entrain to temporally patterned expectations with or without periodic structure by choosing a periodic or aperiodic expectation template λ.

Given a series of event times {t_n} tallied by an event-counting function , an expectation template λ(ϕ), and a prior distribution p₀(ϕ) describing the distribution of phase at time t = 0, the observer’s goal is to infer a posterior distribution p_t(ϕ) = p(ϕ|N_τ<t) describing an estimate of phase ϕ at any time t based on the event history up to t.

In [23], Snyder derives an exact PDE for the evolution of this posterior distribution over time. Following the predictive processing ansatz of maintaining Gaussian posterior distributions (the Laplace assumption), which provides both computational tractability and neurophysiological plausibility by reducing the representation of the posterior to a mean and a variance, we project the posterior onto a Gaussian at each dt time-step. We do this by moment-matching: we use Snyder’s solution to determine the evolution of the mean and variance of the posterior, and then replace the true posterior with a Gaussian with the same mean and variance. This choice of Gaussian is the choice with minimum KL divergence from the true posterior [24], and therefore also minimizes the free energy of the solution within the family of possible Gaussian posteriors in accordance with the Free Energy Principle [25].

The result of this derivation is a generalization of a Kalman-Bucy filter with Poisson observation noise. Eden and Brown [26] have derived an explicit form for this filter, but it relies on a local approximation of the rate function λ that hides some of the interesting effects of events expected at nearby time points. For λ a mixture of Gaussians, we derive a filter directly from Snyder’s solution in [23] that more accurately approximates the optimal (Bayesian) solution. The derivation is presented in S2 Text.

Phase and tempo inference from point process event timing (PATIPPET)

PATIPPET extends PIPPET by making the rate of phase advancement itself a noisy dynamic variable subject to ongoing inference. The dynamic state of the system is now a two-dimensional vector , where ϕ is the phase as above, θ is the rate of phase advancement (or tempo), and σ and σ_θ are the levels of phase and tempo noise, respectively: (4)

As above, an inhomogeneous point process generates events with probability based on an expectation template λ, which in this case is a function of both phase ϕ and tempo θ. In this formulation, we want events to occur with a certain probability in each dϕ phase bin regardless of tempo, which we can accomplish by scaling the event rate by θ: (5)

Note that this is the same as the PIPPET expression for event rate if we set θ = 1.

As before, the observer’s goal is to infer a posterior distribution at any time t using preceding event times; now the distribution p_t(x) describes an estimate of both phase and tempo. A similar derivation provides a point-process Kalman-Bucy filter that optimally serves this function within the constraint of Gaussian posteriors, providing a running estimate of a mean phase and tempo μ_t and a phase/tempo covariance matrix V_t. The solution is presented in S1 Text and its derivation is presented in S2 Text.

The resulting PATIPPET filter generalizes the PIPPET filter, and is identical if the initial tempo distribution is set to a delta distribution at θ = 1 and σ_θ is set to zero. At each event, the distribution of phase and tempo is discontinuously updated to a 2D Gaussian posterior, which evolves continuously between events. This scheme is similar to [27], which estimates phase and tempo by updating a 2D Gaussian posterior, but is updated in continuous time and is significantly more flexible in its capacity to track phase based on arbitrary expectation templates.

PIPPET with multiple event streams (mPIPPET)

Finally, we generalize PIPPET to include multiple types of events (indexed by j), each generated as point processes with rates determined by functions λ^j(ϕ) of a single underlying phase: (6) (7)

The Kalman-Bucy estimate of phase for this model is described by mean μ and variance V evolving according to the ODE (8) and resetting to and when an event occurs in stream j, where we define Λ^j, , and as we defined Λ, , and above but in reference only to event stream j.

The same adjustment can be made to the PATIPPET generative model, and the PATIPPET filter can be similarly generalized to account for multiple event streams.

Computational simulations

In the “Results” section, we conduct a series of simulations to illustrate how the novel terms representing dynamic tracking of uncertainty and the influence of expectations in the absence of events allow the PIPPET and PATIPPET filters to reproduce perceptual and behavioral observations during human entrainment to auditory rhythms. Parameters for these simulations are listed in S3 Text. Simulations were conducted in MATLAB [28] using code available at https://github.com/joncannon/PIPPET.

Updating posterior in response to events

We simulated the PIPPET filter with a single expectation peak and varied parameters to illustrate its basic behavior (Fig 2). Fig 2 column i illustrates the effect of an event on the phase estimate as a function of initial estimated phase μ_t. Events occurring when μ_t is near an expected event phase ϕ₁ caused μ to shift linearly toward ϕ₁. When we set the uniform rate of background events λ₀ > 0, events occurring far from the expected event phase ϕ₁ were attributed to the background and therefore caused negligible adjustment to the phase estimate. Phase uncertainty V_t decreased at events except when λ₀ was positive and μ was not sufficiently close to ϕ₁; in this case, V_t increased due to causal ambiguity, or stayed the same if the cause was unambiguously the uniform background source.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Characterizing PIPPET’s behavior at events.

A) An event is expected at phase ϕ₁ = 0.5 with variance v₁ and expectation strength λ₁. The expected background event rate is set to λ₀ = 0. An event occurs when the phase estimate is at μ_t with uncertainty V_t. Panels in columns i-iv show the resulting mean μ_t+ and variance V_t+ of the posterior on phase as the parameters μ_t, V_t, λ₁, and v₁ are varied. i) μ is corrected linearly toward ϕ₁, while V decreases uniformly regardless of initial phase. ii) Corrections to μ are more thorough when V_t is large. iii) These corrections do not depend on λ₁. iv) These corrections are more thorough for smaller v₁. B) The same simulations are carried out with background event rate λ₀ = 0.5. i) If μ_t is close to ϕ₁, it is linearly corrected toward ϕ₁ and V_t decreases; if it is far, no correction is made. In the liminal zone, V_t increases due to the ambiguity of whether the event was related to the expectation peak or due to the background source. ii) V_t+ is larger due to the effect of ambiguity as to whether the event is associated with ϕ₁ or with the background rate. iii) Now the correction depends on λ₁: stronger expectations make this peak the favored cause relative to the background source. iv) Note that if the expectation peak is extremely narrow, V_t+ may still be large after the event and μ_t may not fully reset to ϕ₁ due to the aforementioned causal ambiguity.

https://doi.org/10.1371/journal.pcbi.1009025.g002

Tracking complex rhythms with uneven subdivision

The PIPPET framework describes entrainment to rhythms in which each expected event phase may or may not be populated by an event. It is formulated in sufficient generality to describe entrainment to rhythms based on timing expectations with complex, non-isochronous stress patterns [29] and with non-integer duration ratios using suitably constructed (presumably learned) expectation templates λ(ϕ). Such rhythmic patterns have been shown to support highly precise synchronization in musicians with appropriate training and enculturated expectations [30], and should therefore be accounted for by models of human entrainment.

As an example of entrainment to a complex rhythm based on a temporal structure with non-integer duration ratios, we simulated entrainment to a swing rhythm. The rhythm is based on an underlying grid of “swung” eighth notes, where the first event of every pair is followed by a slightly longer inter-event duration than the second. Though the “swing” feel is often caricatured using eighth note pairs with a 2:1 duration ratio, this value has been shown to vary by with style and tempo and is certainly not limited to small integer ratios [31]. We used an expectation template with a swing ratio of 3:2 (though the exact ratio is not important) and associated the first eighth note in each pair with a stronger expectation than the second. The PIPET filter entrained to a complex, syncopated rhythm based on this template, drawing on the timing of both strongly and weakly expected events (Fig 3A). It corrected its phase estimate when an event timing shift or a phase shift was introduced into the rhythm (Fig 3B and 3C).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Tracking phase through swung rhythms.

PIPPET is given a pattern of expectations representing “swung” eighth notes, with alternating longer and shorter inter-event durations and stronger, more precise expectations on the first of every pair. Dotted lines correspond to weaker expectations and solid lines correspond to stronger expectations. A) Phase is successfully tracked over the course of a rhythmic stimulus, with phase uncertainty growing between events and contracting at events. B) One event in the rhythm is shifted earlier in time. Estimated phase μ_t adjusts partially to compensate for the timing shift, and then adjusts back at the subsequent event. Uncertainty V_t is not as effectively reined in by these unpredictably-timed events, but decreases as later events corroborate the corrected phase estimate. C) A phase shift is introduced into the rhythm, moving all subsequent events earlier in time. When the first early event arrives, uncertainty increases. Estimated phase is corrected over the first few events after the shift, and V_t decreases most substantially when the estimate μ_t is corroborated by a strongly expected event happening at the appropriate estimated phase.

https://doi.org/10.1371/journal.pcbi.1009025.g003

Failure mode: Too much syncopation

The phase inference framework can account for human failures to track perfectly timed rhythms, i.e., rhythms in which every event falls at a peak of the expectation template. A prime example of this failure mode in human rhythm tracking is tracking overly syncopated rhythms (rhythms with a predominance of events at time points with weaker expectations). Listeners tend to “re-hear” such rhythms by attributing events to metrical positions where events are more strongly expected [32, 33].

In PIPPET, these failures consist of inferring the presence of phase noise where none actually occurred. Such behavior is a necessary consequence of Bayesian optimality: a given stimulus may be generated by different combinations of phase noise and point process event generation noise, and the inference process is concerned only with the most likely explanation for the stimulus, which may include phase noise even if the stimulus was actually generated without it.

Using the expectation template with a swing grid as in the previous section, we simulated a strongly syncopated rhythm (Fig 4A). The rhythm’s phase was not tracked successfully due to a convergence of two factors: the disproportionate influence of the higher peaks of the expectation template, and the accumulation of phase uncertainty V_t. Phase uncertainty was only slightly reduced by events occurring at weakly expected phases, so it accumulated over the course of the rhythm, and especially during the long silence. Once V_t was large, indicating the possibility of substantial phase noise having accumulated, the higher expectation peaks ϕ_i became the most likely explanations for events that were actually perfectly timed to coincide with nearby lower peaks—since precise event timing was no longer a reliable indicator of the source of an event, local peak height became the best indicator, and higher peaks won out. Thus, at each event, the estimated phase was adjusted to better align the higher peaks with the events.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Too much syncopation causes rhythm tracking failure.

A predominance of events associated with weak expectations combined with accumulated phase uncertainty can lead to a failure to track phase accurately. A) In this example, phase uncertainty V increases over a long silence. At the next event, this high uncertainty leads the model to partially attribute a weakly expected event to the nearby phase at which an event is strongly expected. As a result, the model ends up aligning the fifth event with a strong phase rather than a weak one, and overestimating phase at the final event (correct phase marked with yellow dot). B) When the rate of accumulation of phase uncertainty (i.e., the expected phase noise σ²) is decreased, phase is tracked correctly. C) Alternatively, phase can be tracked successfully by inserting an isochronous stream of finger taps and a suitable template for the alignment between expected auditory feedback from the taps and phase. We use mPIPPET to simulate an expectation for isochronous taps (green notes and trace on the left). For simplicity, taps are placed every 0.5 sec; however, even noisy taps generated based on estimated phase could serve to reduce phase uncertainty and avoid a total phase tracking failure.

https://doi.org/10.1371/journal.pcbi.1009025.g004

The same rhythm could be successfully tracked in two alternate conditions. First, it was successfully tracked when we decreasing the rate of accumulation of phase uncertainty σ² (Fig 4B), demonstrating the key role of uncertainty in making the system susceptible to the disruptive effect of syncopation. Second, it was successfully tracked when an additional stream of sensory input was added by simulating an isochronous finger tap (Fig 4C). We used mPIPPET to create a second expectation template for tapping. As phase tracking was simulated, we planned new tap events just before μ reached expected tap phases by extrapolating μ forward. When taps occurred, phase uncertainty decreased, reducing the disruptive effects of syncopation. Note that planning actions specifically to fulfill sensory expectations and using this sensory feedback to inform inference about the outside world is an example of “active inference”, the principle framework for understanding action in the literature on predictive processing [25].

Tempo inference

We simulated the PATIPPET filter with basic metronomic expectations to observe its capacity to infer phase and tempo at once. We gave the model a wide initial range of possible tempi and a simple metronomic stimulus with actual tempo near the upper end of that range. In these conditions and with the parameter set we chose, the model established the appropriate tempo and phase to within a tight range over the course of the first two events (Fig 5).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. The PATIPPET filter estimates phase and tempo.

PATIPPET is initialized with high tempo uncertainty. The first event occurs relatively early, causing the estimated tempo to increase. Each subsequent event occurs close to the time expected based on the estimated phase and tempo, causing the posterior to contract in both the phase and tempo direction as its prediction of event time is fulfilled and its phase and tempo estimates are corroborated. Ultimately, PATIPPET settles on a narrow distribution around the appropriate tempo as it continues to accurately estimate phase.

https://doi.org/10.1371/journal.pcbi.1009025.g005

In addition to its value as a model of human rhythmic cognition, the PATIPPET filter shows promise as a general-purpose tempo tracking algorithm for musical applications. This would require a principled method of choosing values for the various free parameters of the generative model, which might be done a priori based on a labeled corpus, adaptively over the course of listening, or through some combination of the two. We leave a more thorough exploration of the relative performance of this model to future work.

Period-dependent corrections

In sensorimotor entrainment literature, finger taps entrained to a metronome generally shift to correct a certain fraction of an event timing perturbation on the next tap. This fraction is called α. In human subjects, α has repeatedly been observed to increase linearly with metronome period (“inter-onset interval,” or IOI), exceeding 1 (i.e., over-correction) for sufficiently long IOIs [34, 35].

The phase inference framework offers a principled explanation for α increasing with IOI. During an event-free interval, phase uncertainty increases over time. When an event does occur, the precision of the prior distribution on phase and tempo is weighed against the precision of the likelihood function associated with the expectation of that event. If the prior is less precise due to accumulated uncertainty, the precision of the likelihood weighs more heavily against it and the adjustment in phase is more thorough. Thus, all else being equal, events spaced more widely apart in time induce more extensive phase corrections.

Since the strongest phase correction PIPPET can make at an event is to fully update the phase estimate to the expected event time, it cannot account for α values above 1. However, it has been previously suggested that α may exceed 1 for long metronome periods due to some period correction occurring in addition to phase correction [34]. We were therefore curious to see whether PATIPPET could reproduce the linear increase of α with increasing IOI up to and beyond α = 1.

In Fig 6, we show that with appropriate parameters, PATIPPET can indeed reproduce the experimental observation of a near-linear increase in α from below to above 1 as IOI increases. In PATIPPET, this phenomenon is a natural consequence of optimal inference in the context of phase and tempo uncertainty that accumulates between observed events.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. PATIPPET reproduces human tapping data showing stronger error correction for longer inter-onset intervals.

A and B) The distribution on phase and tempo leading up to and following a phase shift at the fourth event in an isochronous sequence for two different metronome tempi, i.e., two different inter-onset intervals. (Same color key as Fig 5, but with phase/tempo distribution contours strobed every.05 sec). Note that when the IOI is short, PATIPPET arrives at the phase-shifted event with a high degree of phase and tempo certainty. C) PATIPPET makes a proportionally larger correction to phase and tempo for long IOIs than for short IOIs due to the greater degree of uncertainty preceding each event. D) Alpha (α) is the proportion of a phase shift that is corrected at the next tap time. With this set of parameters, PATIPPET reproduces the empirical observation from [35] that the phase shift is undercorrected when IOIs are short and overcorrected α > 1 when IOIs are long.

https://doi.org/10.1371/journal.pcbi.1009025.g006

Time warping in the absence of expected events

When an event in a rhythmic stimulus is strongly expected but no event occurs, an optimal Bayesian observer should initially be biased to believe that in spite of their current phase estimate, the stimulus may not have reached the expected event phase yet. The result should be that a perfectly timed event later in the stimulus will seem to be arriving earlier than expected: in other words, the tempo of the stimulus will seem to accelerate. The degree of this effect will depend on the observer’s degree of phase and tempo uncertainty.

There is evidence of such an effect in human rhythm perception. The “filled duration” illusion is the impression that an isochronous sequence has changed tempo when it is initially subdivided by additional predictable events and then subdivisions are eliminated. According to multiple reports, the magnitude of this effect is reduced or eliminated if the empty intervals precede the filled intervals [36–39] (though there is some disagreement about this [40]), suggesting that it is indeed the established expectation of continuing subdivision that interferes with the perceived passage of time when the subdivisions cease. A second result that could be similarly accounted for is the surprising finding in [41] that a participant tapping along with a subdivided beat delays their tap following the omission of an expected subdivision. If taps are planned to coincide with the arrival of a specific mean estimated phase, then the slowing of estimated phase induced by an omission of a strongly expected event should indeed delay the subsequent tap.

We stimulated PATIPPET with a strong isochronous expectation template by scaling up λ and presented it with a “filled duration” in which all expected events occurred and an “empty duration” in which events occurred only at the beginning and end of the interval (Fig 7). PATIPPET loyally tracked phase through the filled duration; however, when strongly expected events were omitted, the mean phase estimate slowed down at each expected event phase, leading to an overall slowing in estimated phase advance and an unexpectedly early onset of the event marking the end of the empty duration (Fig 7A).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. The filled duration illusion: Time warping by the omission of strongly expected events.

(Same image key as 4, with shading displaying PATIPPET phase variance.) A) PATIPPET is simulated with strong expectations for isochronous events. Left: When a set of strongly expected events occur as expected (a filled duration), estimated phase stays on track, advancing (on average) at a rate of 1. Right: When the duration is empty, estimated phase deviates from steady progression (red diagonal) by dragging as each expected event point approaches and passes, leading to the illusion that the event marking the end of the interval has arrived earlier than expected. B) PATIPPET is simulated with a high expected background rate of events λ₀, but no phase-specific event expectations ϕ_i. In this case, too, an empty duration leads to dragging estimated phase and an unexpectedly early final event.

https://doi.org/10.1371/journal.pcbi.1009025.g007

Specifically timed event expectations are not necessary to produce a filled duration illusion: random raindrop sounds were sufficient to lengthen produced intervals during audiomotor synchronization task [42]. In PATIPPET, a filled duration effect was also produced when the expectation template consisted only of a high expected background rate of events λ₀. In this case, estimated phase advance slowed during the empty interval because estimated tempo dropped. The PATIPPET filter effectively noted that not as many events were occurring as expected, and in response it lowered estimated tempo because a lower event rate is expected at a lower tempo. This type of explanation could be invoked to offer a normative account for other non-rhythmic filled interval illusions, though doing so is beyond the scope of this work.

Relationship to other models of timing

The dynamics of PIPPET and PATIPPET in response to sensory events are similar to dynamics of other entrainment models that correct phase and period based on event timing, e.g., [43, 44]. Models based on Dynamic Attending Theory, e.g., [11, 12], are also similar in explicitly modeling timing expectations and their effect on phase and period adjustment. The phase inference framework differ from these existing models in four key ways. First, they are derived as optimal solutions to specific inference problems, and therefore all modeling decisions can be justified within a normative framework. Second, they are formulated in sufficient generality to describe entrainment based on non-isochronous and even aperiodic temporal expectations, an area that has lately received increasing experimental attention [6, 45, 46] but has been largely neglected in entrainment modeling. Third, they allow expectations to influence the inferred phase even in the absence of sensory events, creating the time-warping effect of disappointed expectations evidenced in humans by the “filled duration” illusion. Finally and most critically, they explicitly track uncertainty in phase and tempo, providing a system for moderating between assimilation of new timing data and loyalty to an internal sense of time.

Bayesian methods have been used elsewhere to analyze rhythmic structure as time series of point events. Some of these are application-focused methods that require offline analyses [47, 48] and therefore do not serve as satisfying models of real-time behavior. Cemgil et al (2000) [27] use a Kalman filter that tracks a distribution on phase and tempo similarly to PATIPPET. However, this model is structured to infer phase and tempo event-by-event rather than in continuous time, and is not equipped to handle complex rhythms or temporal structures more complex than approximate isochrony.

Bayesian inference has also been used to model timing estimation in the brain (e.g., [20, 21]), but it is generally used to describe inferences about discrete variables like interval durations and event times, whereas PIPPET describes a continuous inference process underlying predictions about event times. One such model leading to particularly PIPPET-like results was presented in Elliot et al 2014 [22]. The authors created a Bayesian model to explain the results of an experiment that had participants tap along to a stimulus consisting of two jittered metronomes. The model behaves similarly to PIPPET in that it estimates the next event time using a weighted average of previous event times and prior beliefs, with weights informed by expected timing precision. However, like [27], their model infers the anticipated timing of discrete, metronomic events, whereas PIPPET predicts and updates an underlying phase in continuous time and can therefore generalize to non-isochronous and complex rhythms and account for the effects of event omissions. Additionally, in order to account for participants ignoring events far from predicted time points, they introduce the assumption that participants repeatedly test the hypotheses that events come from one or two separate streams, whereas PIPPET naturally accounts for this phenomenon by attributing stray events to a uniform background event rate λ₀.

Interpreting the generative model

The PIPPET generative model is formulated as though it implements perfect variational Bayesian inference on inherently stochastic stimuli. However, Bayesian computations in the brain are often invoked to compensate for internal as well as external sources of stochasticity [49], and in the case of PIPPET the most reasonable interpretation may be a combination of the two possibilities. In reality, we do not often listen to musical rhythms with random timing and phase jitter; however, neural noise and interaction with other ongoing processes may introduce timing variability into the processing of sensory events and give rise to variability in the process of tracking estimated phase. This interpretation also allows for changes in generative model parameters based on internal states that might affect internal noise levels, e.g., attentiveness (which has been shown to affect tempo correction but not phase correction [50], and which therefore might be modeled through its effect on σ_θ). Ideally, the phase inference framework could be reconstructed based on assumptions of a combination of internal and external noise; however, that is beyond the scope of the current work.

Given this ambiguity, the generative model parameters may ultimately reflect some combination of the empirical statistics of rhythmic stimuli and internal factors. We briefly discuss the precision parameters v_i as an example. First, an upper bound on the precision of expected event timing is the precision of sensory timing perception, which is, for example, high for human audition and significantly lower for human vision (An event can only be experienced after it occurs, so (as pointed out in [21]) the likelihood function on underlying phase associated with this type of uncertainty should be asymmetrical. The analytically tractable incarnation of our framework presented here uses Gaussian likelihood peaks, so cannot account for the effect of asymmetrical likelihoods; however, we could posit a λ function with asymmetrical peaks and use numerical methods rather than the explicit solution derived here to estimate underlying phase at each time step). Second, expected event timing precision may further reflect the observed relative timing distributions of event streams. These observations may inform expectations on time scales ranging from a single sitting to a lifetime of listening. Expected timing may be learned separately for different sensory modalities, different musical genres (e.g., techno vs. funk), or even different instruments (e.g., kick drum, snare, hi-hat, as discussed below). The precision of a beat-based temporal expectation is closely related to the width of a “beat bin,” the window of time (rather than a single time point) that is proposed to constitute the “beat” in [51], and to the width of the temporal “expectancy region” described in dynamic attending theory [11]; in both cases, this width is increased by imprecision in the immediately preceding stimulus.

Testable behavioral predictions

Given the ambiguous interpretation of the generative model discussed above, the question of whether human expectation-based entrainment is truly described by a normative framework may be ill-posed. However, two key qualitative elements of this framework can be tested directly: the tracking of phase uncertainty and the influence of expectations in the absence of events. Seeking further experimental evidence of these two phenomena would help determine the value of phase-inference-based models in describing human entrainment behavior.

The phase inference framework predicts that the accumulation of uncertainty over the course of empty time has a critical effect on the perceptual interpretation of subsequent events. In Fig 4, we show a rhythm that is perceptually misinterpreted due in part to empty time preceding syncopation. An experiment could be designed along the lines of [33] to test this aspect of the phase inference framework by measuring the effect of empty time on the interpretation of rhythmic stimuli that follow.

A second prediction along these lines is that various measurable perceptual phenomena, including period-dependent error correction in motor entrainment, perceptual parsing of ambiguous rhythms, and susceptibility to temporal illusions such as the filled duration illusion, should depend critically on levels of phase and tempo uncertainty. Assuming that the parameters of uncertainty tracking vary across individuals, the PIPPET/PATIPPET framework would predict correlations in measurements across these domains: certain individuals should show increased sensitivity to temporal illusion, misleading rhythms, and the effect of period on error correction. Further, stimulus manipulations that affect phase and tempo uncertainty, including the temporal precision of the auditory events and the length of the click train establishing an initial tempo estimate, should have direct and predictable effects on these perceptual and behavioral measures.

Third, the phase inference framework predicts that omissions of strongly expected events should systematically distort estimates of phase and tempo, or, perhaps indistinguishably, of elapsed time. These effects could be explored by parametrically manipulating event expectations through priming stimuli and then measuring distortions induced by event omissions through perceptual report or timed motor response.

If we find situations in which human behavior qualitatively differs from solutions to the inference problems posed by PIPPET and PATIPPET, these can be interpreted in two perfectly valid ways: either human behavior has not been optimally tuned for the task at hand, or we have not correctly identified and encapsulated the task and its survival-relevant objective. If we follow the latter interpretation, we might attempt to refine the generative model, e.g., by introducing the belief that tempo changes occur in jumps or ramps rather than as random drift, or to modify the objective of the task, e.g., by including additional cost functions or priors associated with perceptual report or motor output as discussed above.

Application to analysis of behavioral data

The phase inference framework offers a predictive processing lens for understanding the results of rhythm perception and production experiments. Given a perceptual or behavioral task, we can suppose that motor or perceptual human entrainment behavior is optimally solving an inference problem, and determine the parameters of that problem by fitting them with appropriate methods. These parameters come with natural interpretations in the language of prediction and precision. We can then study the changes in these parameters over the course of an experiment, over different variations on the same experiment, over the human lifespan, across cultures, etc.

For some experimental data, the many parameters available in PIPPET may prove redundant. For example, the observation of weak error correction in entrained tapping could be explained by imprecise auditory timing expectations (high v_i), an overly precise internal model of phase (low V_t, caused perhaps by low σ), or overly precise tap feedback timing expectations (as discussed below). However, we believe these to be meaningful distinctions that call for disambiguation through carefully designed experiments—for example, skipping taps to separate out the precision effects of tapping feedback or varying silent durations within the stimulus to separate the accumulating effects of phase uncertainty V_t from the history-independent effects of timing expectation uncertainty v_i. For experiments that do not take such measures, redundant parameter sets that fit the data may be interpreted as meaningfully different possible interpretations of the results.

Limitations and possible extensions of the phase inference framework

PIPPET in the brain

Though PIPPET and PATIPPET are abstract models not committed to a particular brain-based implementation, advances in the brain basis of timing and beat-keeping combined with the hypothesized neural bases of predictive processing suggest the beginnings of a plausible approximation of PIPPET in the brain, described below.

The essential aspect of the PIPPET framework that qualitatively differentiates its behavior from previous models is the explicit tracking of uncertainty over time for the purpose of informing the relative weights of sensory event timing and internal state estimates. There have been various proposals of how uncertainty is represented and utilized in the brain, and the system likely differs by task and type of uncertainty [49, 73]. One proposal is of particular interest in relation to timing: uncertainty about a hidden state may be computed in medial frontal cortex and signalled via dopaminergic neurons in the ventral tegmental area [74]. In this case, the hidden state would be the phase and tempo of the stimulus. This proposal is consistent with the observations that dopaminergic neurons encode of certainty in the temporal expectation of sensory cues [75] and that dopamine receptor antagonism in humans causes increased timing uncertainty [76].

In the predictive processing literature, dopamine is often given the role of signaling certainty (“expected precision”) across levels of hierarchical processing [77]. In this framework, it participates in probabilistic computations by weighting the input to error-calculating neural populations, causing these errors to be weighted more heavily in the ongoing process of error-minimization that implements variational Bayesian estimation of hidden states. Different dopaminergic populations may signal precision at different levels of processing; in particular, dopamine may signal precision of both higher-level state estimates and lower-level sensory expectations. Thus, phase certainty and expected timing precision may both influence computation through dopaminergic signalling.

Experiments with non-human primates have shown neural trajectories in medial premotor cortex (MPC, encompassing the supplementary and pre-supplementary motor areas) that represent progress through self-generated behavioral processes. The author hypothesizes in [78] that similar trajectories represent rhythmic phase in human MPC. A representation of a linear phase ϕ, used in the phase inference framework for flexibility and mathematical tractability, would seem to be a limiting factor for implementation in the brain. For shorter, aperiodic learned patterns of temporal expectation, phase could be represented by short, aperiodic trajectories [79], as observed in primates in timed response tasks; for simple periodic patterns, phase could be represented circularly [80], as observed in isochronous tapping tasks; and for longer, hierarchical patterns, phase could be represented by hierarchically structured trajectories that loop but also evolve in other dimensions, as observed in cyclic behaviors whose sensory components change from one cycle to the next [81].

Guided by the “Action Simulation for Auditory Prediction” (ASAP) hypothesis presented in [82] and further developed in [78], the theory of hierarchical predictive processing [83], and the predictive functions proposed for the dorsal auditory pathway [84, 85], we propose a neural implementation of PIPPET’s phase estimation in Fig 8. An essential aspect of this account is that it does not insist on the mathematical convenience of instantaneous phase updates, which are obviously implausible in the brain. Instead, precise timing predictions are issued with appropriate timing to intercept rising sensory signals, and the resulting timing errors are then be used to update phase through an error minimization process over the next few hundred milliseconds.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. A possible implementation of PIPPET in the brain.

This diagram embeds a formal predictive coding error minimization scheme is embedded within an informal information-passing schematic to outline how estimated phase μ_t might be calculated and updated on each dt time step by a network of interacting brain regions. Estimated phase μ_t and phase uncertainty V_t are represented in medial premotor cortex (MPC). These estimates are used to calculate instantaneous subjective hazard rate Λ with the help of basal ganglia, which has selected an expectation template λ based on recent rhythmic context. The hazard rate is sent to parietal cortex, where it acts as a prediction of pulses rising from the event-based auditory pathway. An “event prediction error” signal comparing pulses to their prediction is sent back up to MPC, where it pushes μ_t+ in the direction that reduces prediction error—strongly toward local expectancy peaks when events occur, and weakly away from them when there are no events. (Note that phase updating at events is assumed to be rapid but not instantaneous as represented in the PIPPET filter.) The event prediction error is counterbalanced by a local “phase prediction error” signal generated through local interactions within MPC that pushes μ_t+ to continue its steady forward progress. Phase prediction error is weighted by dopaminergic signaling of state precision through VTA.

https://doi.org/10.1371/journal.pcbi.1009025.g008

Briefly, phase is represented by stereotyped trajectories of population firing rates in MPC, and phase uncertainty is also represented locally in medial frontal cortex [74]. Basal ganglia selects and activates an expectation template appropriate to the context. This template is combined with phase and phase uncertainty estimates in MPC to compute a momentary subjective hazard rate Λ. The hazard rate is sent to parietal cortex as a prediction of event-based input, where it meets ascending pulses from the auditory system associated with auditory events (which may be relayed rapidly from the dorsal cochlear nucleus via cerebellum [19]). “Event prediction error” from parietal cortex returns to MPC, where it pushes μ in the direction that reduces error: toward expected event phases at events and away from them between events. This influence is opposed by a “phase prediction error” signal within MPC that pulls μ to progress steadily at tempo θ. This error signal is weighted by phase precision V⁻¹.

Note that this is not a full, formal error-minimization scheme for implementing PIPPET, which is beyond the scope of this manuscript. In particular, it leaves out an updating scheme for V; see [86] for a discussion of the neurophysiology of precision updating. Further, it does not yet include an appropriate scheme for weighting event prediction error.

Although it would be difficult to directly test this neurophysiological setting of PIPPET in humans, it may be possible to indirectly observe a PIPPET-like process in neural data. At the scalp level and in intracortical electrodes, slow electrical oscillations do seem to anticipatorily track the structure of periodic auditory stimuli [87, 88], and this tracking is associated with the subjective passage of time [89]; these oscillations could be explored as possible estimates of mean underlying phase, with particular focus on those in motor areas. Ideally, timing prediction errors could be observed in the evoked EEG response to events (the ERP), allowing a direct measurement of event expectancy at each event time, and there are indeed indicators that the ERP is sensitive to temporal predictability (e.g., [90, 91]); however, the sensitivity of the ERP to recent stimulus history makes this approach unpromising. However, timing prediction errors may be observable in EEG/MEG through their effect on gamma oscillations [92, 93]. Further, the subjective hazard rate Λ itself may be observable by using techniques recently applied to decode the temporal hazard function from EEG data [94], or through its correlation with beta oscillations [95].

Although human-like beat-based perceptual and audio-motor entrainment seems to be unique to humans, other primates do show rudimentary rhythmic timing abilities, especially in the visual modality [96], and represent phase of self-generated cyclic behavioral processes in MPC [80, 81]. Experimental paradigms appropriately modified to engage mechanisms of self-action tracking might activate in non-human primates the same mechanisms of uncertainty-informed event-timing-based phase tracking that we hypothesize for auditory rhythm tracking in humans. Thus, primate neurophysiology in MPC and the dopaminergic system may be a promising avenue for indirectly testing the phase inference framework as a description of the human faculty of rhythmic entrainment.