TAPESTREA : Transformation + Synthesis mechanisms

version: 0.1.x.x (tap tap)

home: http://taps.cs.princeton.edu


Each desired component extracted from the analysis can be saved as a reusable and independently transformable template. In addition to these basic templates, there are some special synthesis templates to control the placement and distribution of sounds in a scene or composition. Here is the list of available templates.

  • Deterministic : Deterministic events extracted during analysis.
  • Transient : Transient events extracted during analysis.
  • Background : Stochastic background extracted during analysis.
  • Raw : Raw template extracted during analysis.
  • Loop : Parametric repetition of a single event.
  • Timeline : Explicit placement of templates in time.
  • MixedBag : Control over the relative synthesis of multiple templates.
  • Script : Control over synthesis using the ChucK programming language.

Naturally, each template comes with a different set of transformations, parameters, and capabilities.


Deterministic Template

A deterministic event is synthesized from its tracks using sinusoidal resynthesis. The sinusoidal track representation makes it easy to time-stretch (or -shrink) and frequency-warp the event by huge factors, in real-time. The frequency warp factors can also be quantized to a user-specified table, to obtain specific pitch ratios.


Transient Template

A transient event is stored as time-domain samples that are played back when needed. A transient can also be time-stretched and frequency-warped using a phase vocoder. Frequency warp factors can be quantized in the same way as for deterministic templates.


Background Template

A background template includes the original extracted background, from which similar background is continuously generated using a modified wavelet tree learning algorithm (Dubnov). Transformations on this include control over the segmentation and randomness in the synthesized background.


Raw Template

Like a transient event, a raw template is stored as time-domain samples. It can therefore be played back and transformed in the same way as a transient event.


Loop Template

A loop repeatedly synthesizes a single event, randomizing the time and frequency transformations as well as gain and pan at each iteration. Apart from the amount of randomness, one can also control the average frequency-warp, time-stretch amount, gain and pan, as well the mean density and the periodicity of the repetition. This is useful for generating crowd sounds from a single instance of an event. Once again, the frequency-warps can be quantized to a pitch table. The density can also be quantized to a time table, such that the interval between two consecutive occurrences can only equal a value entered in the table, for a rhythmic effect.


Timeline

A timeline allows different templates (including other timelines and itself) to be explicitly positioned in time relative to each other. A template can be placed anywhere between the beginning and end of the timeline. When the timeline is played, each template on it is synthesized at the specified time and stops either when its synthesis finishes or when the timeline ends. A timeline can also be quantized in time and frequency according to user-specified tables. Time quantization lets a template be placed only at points in time corresponding to table entries. Pitch quantization frequency-warps consecutive templates on the timeline by consecutive pitch table entries.


Mixed Bag

A mixed bag offers control over the relative density of multiple, possibly repeating events. Each template in the mixed bag has an individual randomness range for transormations, a synthesis likelihood, and the option of repeating. The mixed bag itself has periodicity and density controls comparable to a loop's, but each time it synthesizes an event, it randomly picks one of the repeating templates based on their likelihoods. This is useful for synthesizing textures with many repeating components, as well as for generating crowd sounds from a few event instances. Pitch and time can be quantized similarly to a loop.


ChucK Script

A ChucK script is, interestingly, a script written in the audio programming language ChucK. API bindings between taps and the ChucK virtual machine allow ChucK programs to access and control templates and automate tasks such as loading a group of related templates. Scripts themselves are also templates, and can be played or placed on timelines, as well as being synchronized to each other. Since ChucK allows the user to specify events and actions precisely and concurrently in time, it is straightforward to write scores to dynamically evolve a sound tapestry by controlling synthesis parameters over time.




taps | soundlab | cs | music