5. Deep Leap

A dancer with long braids falls back onto a white floor with one arm raised, under a fixture hanging on wires. A lit white wall behind. — Rebecca Livaniou performing Deep Leap at MDT in April 2026. Photograph: Senay Berhe.

This page presents a Mac application that I built for Lisa Janbell’s dance performance work Deep Leap. The software uses cameras to detect the dancers’ bodies and translates movement into sound through parameters accessible in audio production software. A camera looks at the stage, the software detects the dancer’s body, and the position of the dancer becomes a number that is sent to the audio production software, in this case Ableton Live. The final version uses the dancer’s hand as a control point, so they can essentially modulate effect units in Ableton by moving their hands.

The production wanted to explore how the dancers can shape the sound during the performance with their movement, giving the dancers agency over the sound in a concrete way. The Deep Leap team did research about how to achieve this and had explored various tools, through audio triggers and ready made tracking projects, but they were not working in the intended way. I was already working on something similar in a project of my own, so I adapted my designs to the production of Deep Leap and built the software for them between November 2025 and April 2026.

In April 2026 Deep Leap played at MDT, Moderna Dansteatern in Stockholm. The two dancers shaped the sound with their hands, and Sade Kasiri, the production’s sound artist and technician, ran the app from the sound desk.

For: Deep Leap, a piece by Lisa Janbell
Shown at: MDT, Stockholm, 23 to 26 April 2026
Role: Software engineer
Built with: Swift, SwiftUI, Apple Vision, AVFoundation, OSC over UDP
Plays: Ableton Live, through a Max for Live receiver
Code: github.com/strm0/deep-leap
Dates: November 2025 to April 2026

5.1 What it is

A dancer moves, a camera sees it, the program turns the hand's position into a number, and the number changes the sound in Ableton Live while the dancer is still moving.

The app that I made makes no sound of its own. It’s more of a detection suite that watches the dancers, measures where their hands are and sends one number for each dancer to the music software. What the number does is decided there. For example this number can be mapped to the feedback amount of a delay, the dry / wet on a reverb or simply a gain knob, so that the dancers can control the volume of the sounds that are playing. Nothing has to be worn for it. The cameras are ordinary ones on tripods. Various cameras were used during production and rehearsal, but the final production used two iPhones on stage.

The main component in the app is Apple’s Vision framework, a tool built into and accessible through the macOS Swift programming environment. This tool recognises people in pictures, and was modified with custom code to find the dancers’ bodies and turn the distance between two body parts into OSC messages that Ableton Live can read. The production wanted the dancers to be able to control effects with their movement, and the OSC message is what carries this possibility. The message is always a floating point number between 0 and 1, and can be mapped to any knob on all effect units in Ableton Live.

1. A camera watches

A camera on a tripod looks at the stage. The tracking needs nothing on the dancer's body: no sensor, no marker, no suit.

2. The program finds the hand

Apple's Vision framework finds the body and the hands in every frame of the video. One click tells the program which hand is to play, and a ring follows it.

3. The hand becomes a number

A square lies over the picture. Where the hand is in it becomes one number from 0 to 1: how far it is from the centre, or in the other mode how high it is.

4. The number moves a slider

The number goes to Ableton Live 30 times a second. There it is tied to something in the music: a filter, an effect, a volume.

5. The sound follows the hand

So when the hand moves, the sound moves with it, at once, and the dancer hears their own movement.

6. Two of everything

The window has two zones. Each has its own camera, its own lock and its own number, so two dancers can play at the same time.

A laptop screen in a dark room. The app's window is split in two. On the left a dancer in a grey hoodie with a red ring around one hand, and rows of small controls above. On the right a mirrored no signal card from a capture device. — The app on the laptop in February 2026. In the left zone a red ring marks the dancer’s locked hand, and the right zone has no camera connected yet. The controls above the picture choose the camera and the mode, set how quickly the number follows the hand, and move the lock to another hand.

In the theatre

A stage rehearsal at MDT, 16 April 2026. The monitor at the sound desk shows what the app sees, and on the floor the two dancers each have a hand in the air.

5.2 Why it was built

The production had already started when I came into it. They needed a way for the dancers to control the sound with their bodies, and the ready made projects they had tried for the tracking, made in TouchDesigner, had not worked.

I met Lisa Janbell while I was working on soundSpeedGradient, a piece of my own in which a webcam reads a hand, and the hand controls sound in various ways. I told her that I was working on exactly what her production was missing, and I made some early prototypes and tested them with her.

The first prototypes ran in a web browser and were made of three parts that had to talk to each other. The version for the stage, which I started in November 2025, is a single Mac app that opens the cameras, finds the hands and sends the numbers by itself.

In February 2026 I spent a full week at a residency in Luleå focusing on the development of the software that would be used for the performance, moving from prototype to an actual application. Here many different approaches were tested, but we ended up going for the most robust and simple approach that still covered all of the functionality that was envisioned for the performance.

An early version

15 December 2025, my living room, as the app saw it. An early version of the Mac app reads the whole body, and in the mode named at the top left the distance from the middle of the body to the fingers of each hand sets the two numbers at the top right.

First studio test

The first test in a dance studio.

In one picture

February 2026. Rebecca Livaniou holds an arm out, an iPhone watches from the tripod, and the laptop in front shows what the app sees.

The ring

The red ring of the lock stays on the dancer’s hand while the dancer moves.

5.3 How it was used

The vision and idea for the Deep Leap performance came from Lisa Janbell, was danced by Cajsa Godée and Rebecca Livaniou, with sound from Adde Huumonen and Sade Kasiri. MDT describes it as a choreographed soundscape in which the dancers, with microphones worn on the body and a responsive technical solution that captures their movements, modulate and generate the sound in real time. This app is that solution.

It played at MDT on Skeppsholmen in Stockholm from 23 to 26 April 2026. In the theatre two cameras stood at the edge of the stage, and at the sound desk a monitor showed the app’s two zones, one for each dancer, next to the mixer and the laptop with the music.

Sade Kasiri ran the app during the shows, which meant clicking on each dancer’s hand in the picture before the show started. A red ring appears on the hand, and the app follows it from then on. The manual I wrote for the app gives two pieces of advice for a show: lock the hands before it starts, and if a lock has to be moved, move it in a quiet moment of the music.

To the stage

A week at a residency in Luleå in February 2026, rehearsals with both dancers in April, and then the shows at MDT in Stockholm from 23 to 26 April.

In the room

Two cameras face the stage, and their pictures run to the sound desk, where the app was run during the shows next to the mixer and the music.

Before the show

The person running the app clicks each dancer’s hand in the picture. A red ring appears on it, and from then on the app follows that hand and nothing else.

Rehearsal

A rehearsal on 7 April 2026, with one dancer in each zone on the monitor and the music on the laptop in front of it.

On stage

At MDT on 16 April 2026, the monitor at the sound desk while the two dancers move as one shape on the floor.

Two people sit on cushions at a low desk in a dark theatre, seen from behind. On the desk a monitor shows the app's two camera pictures with a dancer in each, next to a mixing console and a laptop. The stage curtain is behind. — The sound desk at MDT, with the monitor showing both zones, the mixer and the laptop with the music.

A dancer with short bleached hair and a small microphone at the cheek holds one thumb up high and one at the hip, in pale light. Another dancer's arm is a blur in the foreground. — Cajsa Godée performing Deep Leap at MDT, April 2026. Photograph: Senay Berhe.

Two dancers lean back on their arms on a white floor with their eyes closed, one behind the other, in a pale blue light. — Cajsa Godée and Rebecca Livaniou on the white floor of the stage. Photograph: Senay Berhe.

5.4 Made for the stage

One thing mattered more than anything else: the dancers have to feel that the sound is theirs. When the app lost track of which hand was which, the error was small for the tracking but large for the dancer, because the feeling that their movement was causing the sound was gone.

The lock solved this. A person points out the hand once, and after that the app only has to find the hand nearest to where that hand just was, instead of guessing whose hand it is.

In the end the lock became the whole instrument. In February the app still had six ways of reading the body, four of them without the lock: the hands measured from the centre, from the shoulders or from the feet, and the distance between two dancers’ heads. The final version keeps only the two that are built on the lock.

A show also lasts far longer than a test, and two faults only appeared after a long run. The app slowly used more and more memory, and the part of the music software that receives the numbers went silent after 30 to 60 minutes. Both are now handled by quietly restarting that part while the app keeps running, so that it lasts a whole show.

Two dancers bend forward close together, one with long braids and one with short bleached hair, both looking down, in a pale blue haze. — Cajsa Godée and Rebecca Livaniou at MDT, April 2026. Photograph: Senay Berhe.

Hands that swap

Vision finds the hands again in every frame and keeps no memory of which is which. When a dancer turned, or brought the hands together, the sound jumped to the other hand.

The nearest hand

The lock does not ask whose hand it is. In each new frame it takes the hand nearest to where the locked one was, looking within a tighter distance when two people are close and a looser one for large movements.

A relock is eased

When the lock is moved to another hand the number does not jump. It eases to the new value over 3 seconds, and the manual's advice is to relock in a quiet moment of the score.

Made to run a whole show

Two faults only showed after a long run. The app now quietly restarts the parts that failed, about once a minute, while it keeps running.

A cable comes loose

If a camera is unplugged its zone says so and stops sending. When the cable is back the zone reconnects by itself.

Two modes and a manual

In February the app had six modes. The final version has two, both built on the lock, and a manual for the person who runs it.

5.5 Credits

The app, its design and its manual: Oscar Ström, software engineer on the production, November 2025 to April 2026. The tracking is Apple's Vision framework.

Deep Leap, as MDT credits it. Idea, concept and choreography: Lisa Janbell. Co-creating dancers: Cajsa Godée and Rebecca Livaniou. Sound design and composition: Adde Huumonen. Sound art and sound engineering: Sade Kasiri, who also ran the app. Light design and technician: Jonathan Winbo. Costume design: Amanda Hedman Hägerström. Dramaturgical and artistic advice: Karina Sarkissova. Producer: Puma Lagos Parra. Production assistant: Ela Villanueva. Consultation: Magnus Nordberg, Nordberg Movement.

Co-produced by Lisa Janbells verksamhet, Dansens Hus, Dansinitiativet, MARC, MDT Moderna Dansteatern and Riksteatern, with support from Kulturrådet and the City of Stockholm.

The photographs of the performance are by Senay Berhe. The other pictures are phone recordings from February to April 2026, a recording of the app from December 2025 and a clip from the development video of my own soundSpeedGradient.