tools for telematics

People often ask about the software we are using for telematic music and what is possible with newer technologies. The bandwidth required for networking uncompressed audio and video is hard to access outside of universities, but here is information on the tools we have used:

Audio

There are several good options for networking with uncompressed audio, but the software that we use in my circles is a fantastic open source tool called JackTrip. It was developed by the SoundWIRE group at Stanford University’s CCRMA. As described by its authors, it is “a Linux and Mac OS X-based system used for multi-machine network performance over the Internet. It supports any number of channels (as many as the computer/network can handle) of bidirectional, high quality, uncompressed audio signal steaming. You can use it between any combination of Linux and Mac OS X (i.e., one end using Linux can connect to the other using Mac OS X).”

JackTrip is essentially a plugin for the open source JACK audio server platform that allows remote machines to be included in the JACK routing environment, so you need JACK to use JackTrip. Two other useful tools are QJackCtl, which provides a graphical routing interface for JACK and other features, and JMess, a utility for saving routing configurations. I’m very grateful to Chris Chafe and Juan-Pablo Caceres, the developers of JackTrip, for all their work over the years on this excellent software.

I’ve made two videos on the basics of using JackTrip for a graduate course Mark Dresser and I teach on telematics. If you’re a programmer, you might find the pace to be slow, but for everyone else, they’re here:
JackTrip Basics, Part 1
JackTrip Basics, Part 2

We’re also working on a JackTrip manual (in progress), and the JackTrip Users Group is a great resource for expert-level advice.

Latency

(Because everyone asks…) With sufficient bandwidth, jacktrip allows us to perform rhythmically complex music together in tight synchrony at distances of a few hundred miles. The perceptual threshold where audio latency becomes a problem for musicians seeking tight synchrony is roughly around 25-30 ms, which would be equivalent to standing around 25-30 feet away from someone in the same room. The total latency we experience includes not only the signal’s travel time through through the internet but also the latency added by audio gear in each location. We’ve managed to achieve total latency of 25-30 ms or lower between two universities with 10 Gb ethernet networks located 500 miles apart (e.g. Berkeley and San Diego, CA), with 8 channels of 44.1k audio. Put another way, we can exchange multi-channel, studio quality audio 500 miles apart with no timing problems and with total latency lower than what two people sitting at opposite sides of a symphony orchestra would experience.

At longer distances, the latency is definitely perceivable and presents creative challenges. For distances of a few thousand miles, such as NY-CA, Zurich-CA or even Korea-CA, we have generally been able to achieve total latency in the range of 200-300 ms, meaning that the delay we hear in the room as performers feels roughly like a 16th note at quarter note = 60 bpm. It’s also important to understand that many factors contribute to latency and sound quality in networked audio contexts, so getting good results requires a lot of experimentation, often in consultation with network administrators. Video latency is of course higher.

Video

Here are some of the tools we’ve explored for video networking:

UltraGrid: High quality open source video networking software. Although we found it challenging to use in 2011, it’s evolved a lot since then and in early 2016 we used it on a 3-site telematic concert with excellent results. Members of the UltraGrid development team were super helpful, so you might want to contact them if you plan to use it for a major project.

AccessGrid: Open source platform, but in my experience not so useful for performances.

VIPR: Software created by Benjamin Day Smith that is “a port of ffmpeg’s libavcodec for cycling74’s Max environment.” Still under development but very useful if you want to use Jitter objects to exchange video, since you can experiment with different compression settings to suit your bandwidth/needs.

Lifesize and “Softphone”: Not ideal for performance, and not free, but it does provide several advantages over consumer options like Skype (networking with h.323 protocol, and IP to IP). Used for Virtual Tour in 2013 and many other projects and classes since.

Since most of my collaborators and I use Macs, we have not tested Windows options, but there you could try ConferenceXP and LOLA. LOLA is one of the few options that provides an integrated audio-video networking platform, but it requires expensive frame-grabbers at each performance node.