With over 5000 stars on GitHub, Pedalboard is one of the most popular open-source projects based on JUCE, speeding up Spotify’s machine learning data processing by 300x. Peter talks us through how it came about and what it’s like working at Spotify’s Audio Intelligence Lab.
I was just watching your Pedalboard talk. I do some prototyping in Python, so I was like “Oh, I can use this as well!”. I can load files with it instead of Librosa, right?
Yeah, absolutely.
The MP3 stuff is convenient. Pedalboard is much more of a swiss army knife than I expected. I thought it was “an awesome machine learning audio tool” but it’s more “making Python audio first class.”
That’s a really good way to put it. It started out as a machine learning (ML) thing, but a lot of people are using Python to do audio nowadays and the tools in Python that exist for audio are…. academic.
I understand.
I work with folks who write academic Python every day. I know many of the Librosa authors. We’re all friendly with each other, but there’s a different goal for Pedalboard, which is speed and efficiency and developer ergonomics. Whereas all the other software is mostly made to accomplish some scientific task and learn something new.
Going over its feature set, it runs and automates VST effects. It has built-in effects. You can now play software instruments and specify automation for VSTs. What’s the deal with live audio? I saw you recently added the AudioStream class.
People have asked about live audio in Python for a long time. They want to do things they can hear in their speakers. It’s super interactive if you can play with the VST and hear the parameters change in real-time instead of having to render and hit playback.
People have been using PyAudio for a long time, which is great. There’s a couple other libraries. They all work. But they all go through Python itself, which means they take the GIL, the global interpreter lock, and that means there can be a garbage collection at any time. And anybody who’s written audio code, you know you want your audio loop to not garbage collect and pause.
A big no-no.
Exactly, I had a lot of people actually write third-party libraries that used Pedalboard. And they would be like, “Why is it stuttering? Why is Pedalboard broken?” And I’m like “Well, that’s not Pedalboard, that’s a fundamental limitation of how Python works. So AudioStream was written to put all of the audio processing, the entire audio callback into C++. And then you can manipulate it through Python, but the audio callback never waits for Python.
I saw that you are using Rubberband and Lame for MP3. As far as built-in effects, did you get to have some fun and build those yourself? Or did you pull them off the shelf somewhere?
I must confess, they’re mostly pulled off the shelf. A lot of them are DSP widgets out of JUCE. So if anybody’s developing a plugin with JUCE, you can just take the Pedalboard plugins that are there and use those as building blocks as well. You can chain them together in a way that you might want to do in your own plugin and then know exactly how to write your JUCE code.
I was thinking that because Pedalboard has automation, I could do some end-to-end testing. Do you know if anyone is doing null testing or end-to-end testing with it?
Yes, absolutely. I know people in the community are using Pedalboard for testing. I have a lot of people reach out from major plugin vendors, opening GitHub issues, saying “we’re using this for testing.” Antares is working with Pedalboard for sure. I think they actually had a job ad out that says you must know how to use Pedalboard, which is wild to me.
It works super well. Because you can use it to host a plugin, but you can script that hosting. You can load and save the plugin state, you can change all the parameters, can’t take screenshots of the UI just yet, but maybe that’ll be something someone adds in the future. And then you can use PyTest, which in my opinion is the best testing framework around, even though it’s in Python.
Python tests are very readable! Could someone write a DAW with Pedalboard?
Great question. So technically you could write a DAW with Pedalboard. It has all the components necessary to host the VSTs. You can get audio out of them in the appropriate order and at the right time. It’s a bit difficult to do so. Like, there isn’t a built in wrapper or framework that allows you to do it super easily, but it would probably be only a couple hundred lines of Python to wire everything together.
The hard part would be visual layout, the data structures to represent when you want things to play and all that kind of stuff, but all the tools are there and it’s really made for flexibility.
from pedalboard import Pedalboard, Chorus, Reverb
from pedalboard.io import AudioFile
# Make a Pedalboard object, containing multiple audio plugins:
board = Pedalboard([Chorus(), Reverb(room_size=0.25)])
# Open an audio file for reading, just like a regular file:
with AudioFile('some-file.wav') as f:
# Open an audio file to write to:
with AudioFile('output.wav', 'w', f.samplerate, f.num_channels) as o:
# Read one second of audio at a time, until the file is empty:
while f.tell() < f.frames:
chunk = f.read(f.samplerate)
# Run the audio through our pedalboard:
effected = board(chunk, f.samplerate, reset=False)
# Write the output to our output file:
o.write(effected)It seems like Pedalboard could also be like the foundational layer for something like — oh, what’s the name of that live music coding software written in Ruby? It’s really cool.
Is it Sonic Pi?
Yes!
That would be pretty neat. It would take a lot of effort to translate between that coding environment and the actual VSTs and stuff, but it would be doable.
Here’s something that I’m personally curious about: How do I get this kind of job? How did you end up making Pedalboard and doing open-source for Spotify?
It was strange to me! My team does audio machine learning stuff. And we needed something to apply audio effects at scale. Imagine we have a whole bunch of audio examples to train a machine learning model. We need thousands or tens of thousands of them. How can we script that so we can apply reverb and add compression and stuff like that?
I had been playing with JUCE on my own a little while and realized JUCE works perfectly for this. If only we could script it nicely so I could give a Python interface to some academic researchers. So I wrote Python bindings for it, put it all together. It accelerated our data processing by about 300x. It was just so, so, so much faster.
And then I realized, well, if we need this, what else does JUCE have? JUCE has really good file I/O. JUCE has really good streaming and you know, all these other features that are in Pedalboard now.
So, “How do you get this job…”
Yeah, I really do want to know!
This is a really interesting story. This is going to dive a little bit further into my own history than I expected. Way back, I flipped a coin between going into software engineering as my profession and becoming a professional drummer. I flipped a coin, went into engineering and always wanted to do something in music. Every chance I got, every internship opportunity, I looked for a company that was somewhere in between the two. And I found a little company called The Echo Nest. This was around 2009.
Wow, you worked for The Echo Nest?
I never properly worked for The Echo Nest, but I found their APIs, used them as an excuse to learn Python. I got to know their people, I went to their hack days, I did some collaboration with them.
Wait — which Music Hack Day did you go to?
Music Hack Day, Montreal 2011 and then Toronto 2013, I think.
I went to the one in Stockholm in 2010! I loved that company. When they were bought by Spotify, I found it really interesting.
I really credit them with kickstarting my career in music and tech. I didn’t think it was possible for a long time. I went off and worked in not-music fields for a long time. And then one day some folks over there came knocking on my door and said “Hey, we have open positions at Spotify. We got bought by Spotify, don’t know if you’ve heard of them. We’d love it if you came to interview.”
Fast forward a couple years, now I lead the engineering for the team that owns a significant portion of the software that The Echo Nest built. Now it’s my job to keep that running.
That’s awesome. You’re the curator. So all those Music Information Retrieval (MIR) type things, but then I guess it grew into stuff like Spotify’s “Discover Weekly” as well?
Yeah, it kind of split into two a little bit. There’s the MIR piece, which my team owns. We’re about 10 people, mostly research scientists and a couple engineers. And then there’s the music recommendation and discovery which grew dramatically from I think 100 people when the Echo Nest got bought. The MIR stuff is super important, but needs less investment.
Let’s talk about MIR a bit. Can you describe what it is? Also, is it still a thing? I feel like ever since AI became so hyped up maybe things have changed a bit.
I’m probably going to ruffle some feathers by saying the wrong thing here. What I describe as MIR is “We teach computers how to listen to music.” We ask computers to tell us things like where the beats are, what the tempo is, what the key is, stuff like that.
It’s still hugely useful, but modern machine learning techniques have a new approach to it entirely, where instead of getting low level features, it’s now possible to pass entire audio files into large ML models and get the answers we want, instead of using smaller, more interpretable models.
But I have to stress, I’m definitely not an MIR researcher. There’s a huge, very deep academic field. And I realize how little I know in that field.
Makes sense! I saw a recent ADC talk from JOS from CCRMA, lamenting that modern AI can sometimes forget MIR audio fundamentals and was advocating for feeding more spectral and MIR information to ML algorithms.
We never actually show customers the MIR data in the Spotify apps. It’s not like there’s a view where you can just go see the beats. It’s in the API. But we do show it to you indirectly as a part of the recommendation features.
During Covid lockdowns, I looked into MIR and did some Coursera courses. One thing I always found surprising is that for audio, they’re always like “First, convert it to a spectrogram to make an image.” And I’m like, “Yeah, I understand it’s a way of compressing the audio data, but why do we always do this?” I guess because raw audio data is so abundant and unstructured, just a stream of numbers? And an image representation is 2 dimensional?
When you say it’s “just a stream of numbers” — which is true — that confused me for a long time as well when I started to get into this. Because I thought “Ok, the waveform is the most low level representation of audio that you could have, it’s coming right off the microphone, it’s gotta be what you need to feed into the model”.
And then over time, my intuition shifted about that. I worked with more and more different types of machine learning models for audio and I realized there isn’t really a correct or incorrect representation of audio. If you change a waveform into a spectrogram, you can do that in such a way that you get the exact same waveform back when you invert that. And so we actually do that in a lot of places.
The spectrogram to me is now a more useful canonical, correct ground truth representation of the audio. I no longer think of it as a transform you have to do to your audio. I think of it as what our ears are hearing.
Peter Sobot
A couple of years ago, we released a denoising ML model that ran on your phone for creating podcasts (see: Spotify for Podcasters app). That’s one where we went from waveform into spectrogram, did our ML on the spectrogram, and then went back into waveform again. And it was one-to-one. You would get the exact same floating point numbers at the other end.
I realized for ML, having that two dimensional representation is much more valuable, because similar frequencies are close to each other, and similar time points are close to each other. Whereas with the waveform, you don’t get that locality. And the model has to do a lot more work to try and infer across longer time periods there. So the spectrogram to me is now a more useful canonical, correct ground truth representation of the audio. I no longer think of it as a transform you have to do to your audio. I think of it as what our ears are hearing. Waveform is the weird one.
Exactly! That’s a great insight. I wasn’t trying to throw the frequency domain under the bus, I just found it funny that throughout the history of audio ML, it’s like “let’s make images!” Frequency vs. time domain is always mind-bending…
It’s just super hard. Especially in real-time audio you can’t really do things in the frequency domain nearly as easily. Like if you’re writing a plugin, you’re going to end up with tiny slices of the waveform. If you need a spectrogram, it’s harder to do in real time, it’s harder to reason about more computationally difficult things. But I do think our ears “think” more in the frequency domain than they do in the waveform domain.
I really enjoy hearing these intuitions and insights. Just having a casual conversation is enough to make me feel more enlightened on the topic. I appreciate it!
I’m flattered. A lot of this is intuition driven. If you had to ask me to write out why any of this stuff is true or why I believe this, I would have to talk to one of the PhDs that I sit with and say like “can you actually write down the math for this?” You really just work with it for a while, and you realize “Oh, this is true. If I believe this about how the system works, my code will work better. I can build a system that will actually function instead of needing to go back to first principles all the time.”
Are you translating a lot of math to code? Or are the researchers prototyping stuff in MATLAB or Python or something?
The research scientists I work with are very broad in their scope. So they will go all the way from writing math on a whiteboard to writing some Python code in whatever framework of their choosing to validate ideas and test models. They will often also write the machine learning code. So they’ll write PyTorch code or TensorFlow code or whatever so that they can train a model on this kind of stuff.
Where we, on my team, segment between research and engineering is when there’s an engineering need that researchers can’t really handle. So let’s say they want to train on a whole bunch of GPUs or they want to scale up to millions and millions of training data points or they want to put a model into production and serve that to hundreds of millions of Spotify users, that’s where engineering really comes into play.
So it’s a lot of collaborating really closely. It’s pretty rare for them to send a bunch of math over the wall. In fact, there’s no wall. We sit next to each other. It’s often like, “Hey, I’m working on PyTorch code, do you think we could put this into production easily?”
That’s slightly different from academic environments I know of, where the most programming they’ll do is a prototype in MATLAB.
When I was first on this team 6 years ago or so, there were folks who were not writing Python at all. There were folks who were just doing MATLAB or some writing R.
So… jazz drumming. Who is your favorite jazz drummer?
I’m going to be really offensive to some jazz heads and say Louis Cole.
Louis Cole is pretty damn fun, no matter how you cut it. Plus he’s doing everything.
He’s doing everything. He’s much more than a drummer. But people would say he’s not a jazz drummer.
He’s got chops, doesn’t he?
He’s got chops for sure. Chops for days.
If I had to go a little bit further into traditional jazz drumming, I love Mark Guiliana. I’ve seen him many, many times here in New York. Brian Blade, classic.
But I’m like a newer jazz kind of guy. I’m into the modern stuff. A lot of which comes out of Vienna, surprisingly.
What, really?
Yeah, there’s a whole bunch of like, I want to call them like jazz nerd kids, in the most positive and friendly way, who come up with really cool new stuff that you would only see on Instagram and then YouTube and then eventually on Spotify. And a lot of them are from Vienna.
That’s cool, I’ll have to look into that! How’s “maintainer life?” I guess that’s kind of new for you with Pedalboard? It’s a pretty big public project.
Pedalboard is definitely the biggest open-source project that I’ve ever maintained. We’ve had a lot of community interest. A lot of people bring their bug reports and often pull requests as well. There haven’t been any major contributions of huge features. It’s often someone has a feature request or bug, suggests some code to fix it, and then it’s up to the maintenance team to come in and get it across the last step.
It’s interesting because Pedalboard is used by people who know Python, but it’s written in C++. So the user base is not usually the kind of programmer who would be able to make a change to Pedalboard in the first place. They lack that cross language skill, which puts a bit of extra burden on the maintenance team. But it’s still something that has a lot of activity, regardless.
I never thought of that. The Python community is so much larger than C++.
I’ll give you an example. I was at PyCon in Pittsburgh this year. Really great conference. A lot of people wanted to talk about music and audio and stuff. PyCon has sprints where you work on an open-source project together. I had some people come up and offer to help. And most of the time was really just talking about how the build system works and how to get acquainted and how to set up your development environment and stuff like that. Very little time was actually spent building a feature or fixing a bug. Which I guess is to be expected.
Yeah, you’re kind of bootstrapping people on just getting C++ compiling. I noticed you guys are still on JUCE 6. Should I submit a Pull Request? [laughs].
You’re right. It’s really just because the features used by Pedalboard haven’t changed that much between 6 and 7. And there aren’t any major bugs that would be fixed by upgrading. It’ll probably just work out of the box. I just haven’t updated and tested it out yet.
If I find some extra time, I’ll take a look!
That would be a pull request I would instantly merge. You’d get committer access right away. [laughs]
Peter Sobot was interviewed by Sudara, who is building an additive synth with JUCE.
Have comments on this interview? Ideas for who we should talk to next? Let us know on the JUCE forum.