How does Muse work?

Muse is creating a platform to express what you see, hear, think, and feel in the moment. It has several parts:


The best way of exploring how to express the moment is to create an instrument and put it in peoples hands. We played with quite a few, including keyboards, tablets, joysticks, even a virtual puppet controller (a separate collaboration with thanks to Carl Page, Roger Nelson, and others):

In 2002, we rewrote Sky from the ground up. By 2006, the final tablet version look like this:

And then in 2007, the iPhone was announced, which changed everything. It had everything needed to create a new performance medium: the ability to record visuals, sound, and touch and playback on millions of connected devices. So, we started to rewrite everything again.


Underlying each instrument is a patch. Each patch translates gestures – be it a touch, drag, swipe, shake, or what have you – into a way of manipulating visuals or sound. To support patches and performances in a way that may be distributed over the net, we wrote special purpose ontology, called Tr3. Here is a white paper describing Tr3.

To visualize the Tr3 ontology, we wrote a viewer. Here is the Tr3 ontology looks like for the tablet based synthesizer, shown above:

Pass alongs

Every patch can be tweaked and saved as a new patch. Patches can be passed along to your friends to experience, tweak, and pass back.