here's an idea using AI

All your LV2 and LADSPA goodness and more.

Moderators: MattKingUSA, khz

User avatar
funkmuscle
Established Member
Posts: 2801
Joined: Mon Jun 02, 2008 2:30 pm
Has thanked: 129 times
Been thanked: 31 times

here's an idea using AI

Post by funkmuscle »

Since we're in the age of Artificial Intelligence, any plugins or standalone software out there that can analyze mixes?
I mean, you mix a song then run it through this software and it can tell you what's too hot, lower this frequency, raise the vocals, etc.

Code: Select all

before anyone says use your ears or anything like that, I'm thinking of this for the bedroom engineer who cannot set up the proper environment for mixing/mastering
After all, we're here to make music so since AI are created to make life easier, why not intelligent software to help mix/master a lot easier?
Someone thought of that already(LANDR, etc.) but how about software on our own PC's. Something that give say a rough mix as a starting point and maybe give suggestions on EQ'ing, compressing, etc., and then it can master the track but we still have the ability to tweak.
Remember, I'm thinking of folks like myself who doesn't have the money or rights to fix up my bedroom for mixing/mastering and most of us don't have pro studios.

Y'all think something like that is possible? I think Waves Audio maybe working on this.
User avatar
sysrqer
Established Member
Posts: 2520
Joined: Thu Nov 14, 2013 11:47 pm
Has thanked: 319 times
Been thanked: 148 times
Contact:

Re: here's an idea using AI

Post by sysrqer »

Izotope's Neuron and Ozone have an assistant that does an analysis and sets eq, comp etc accordingly. I've been using it in ozone for mastering and it is really good, it actually sets up a dynamic eq just before the limiter to lessen the load on the limiter.

I really want to try out Neuron to use it on individual tracks, it has the assistant but also has a masking/eq clash feature which looks very useful. Unfortunately, it doesn't work at all in linvst for me although ozone is fine.

I would love to see something comparable natively in linux but I think they are many years ahead of what we have.
CrocoDuck
Established Member
Posts: 1133
Joined: Sat May 05, 2012 6:12 pm
Been thanked: 17 times

Re: here's an idea using AI

Post by CrocoDuck »

funkmuscle wrote:Since we're in the age of Artificial Intelligence, any plugins or standalone software out there that can analyze mixes?
I mean, you mix a song then run it through this software and it can tell you what's too hot, lower this frequency, raise the vocals, etc.
Yes, and it is actually a research topic in many universities. If I am not wrong, there are actually websites the offer this exact service for masters. I remember I stumbled across one, but I quite don't remember how it was called.

I think there was also an article on Sound on Sound about "auto-master", if we want to call it like that.

A very fascinating topic indeed, one of the many reasons I am attempting to study Machine Learning in my spare time.
sysrqer wrote:I would love to see something comparable natively in linux but I think they are many years ahead of what we have.
Not too much. There are already open source speech recognition systems, which are essentially a trained machine learning program (for example, Julius). Doing a mix / master is not the same problem, of course, but it is in the same domain. I think it will perhaps boil to how much time talented developers can find to develop the thing.
User avatar
sysrqer
Established Member
Posts: 2520
Joined: Thu Nov 14, 2013 11:47 pm
Has thanked: 319 times
Been thanked: 148 times
Contact:

Re: here's an idea using AI

Post by sysrqer »

Indeed, it's not that is not possible but perhaps is a tall order for a lone programmer in their spare time.
User avatar
funkmuscle
Established Member
Posts: 2801
Joined: Mon Jun 02, 2008 2:30 pm
Has thanked: 129 times
Been thanked: 31 times

Re: here's an idea using AI

Post by funkmuscle »

yes the online services is stuff like LANDR.
that's amazing that it's already out there as a plugin or a piece of software to be installed and used.
Awesome if we do have something native for sure!
User avatar
loxstep
Established Member
Posts: 37
Joined: Thu Jan 11, 2018 12:40 am

Re: here's an idea using AI

Post by loxstep »

Wonder how that would work. If it was anything like other AI projects I've encountered, an AI for audio would be trained on a corpus of existing songs. You could put in thousands of songs and it would look for patterns in how songs tend to be mixed / mastered. In the best case scenario, an AI could compare your track with similar commercial examples, and point out differences in the dynamic range, frequencies, etc. I bet a lot of people would find that useful. But I would be hesitant to use such a tool. I don't want my mixes to sound like everyone else's.

I think mixing is too subjective to be automated. There is no right level something should be at. It depends on what you want your audience to pay attention to.
User avatar
funkmuscle
Established Member
Posts: 2801
Joined: Mon Jun 02, 2008 2:30 pm
Has thanked: 129 times
Been thanked: 31 times

Re: here's an idea using AI

Post by funkmuscle »

loxstep wrote:Wonder how that would work. If it was anything like other AI projects I've encountered, an AI for audio would be trained on a corpus of existing songs. You could put in thousands of songs and it would look for patterns in how songs tend to be mixed / mastered. In the best case scenario, an AI could compare your track with similar commercial examples, and point out differences in the dynamic range, frequencies, etc. I bet a lot of people would find that useful. But I would be hesitant to use such a tool. I don't want my mixes to sound like everyone else's.

I think mixing is too subjective to be automated. There is no right level something should be at. It depends on what you want your audience to pay attention to.
Yep I fully agree but remember I said for us bedroom engineers? I was thinking the AI would on initial set up be able to analyze your mic, then you play a song you think is perfectly mixed at the max volume you can in your room.
Then the AI will analyze the results of the environment so it will let you know during mixing if something is too hot or needs boosting etc.
That avoids the frequent 'car test'. :). I'm on the 14th floor and my car is 2 levels underground. Just when I'd think I have the perfect mix, I head down to the car just to find out the bottom end is very weak in the mix.
With this AI, it will have all that taken care of with suggestions.
Once final mix is ready, the AI will analyze the final mix for mastering frequencies and suggest or automatically master the song like LANDR.
Love your idea of the AI analyzing many different genre of music for learning.
Wish I could code or program or whatever these geniuses do that create plugins, etc.
CrocoDuck
Established Member
Posts: 1133
Joined: Sat May 05, 2012 6:12 pm
Been thanked: 17 times

Re: here's an idea using AI

Post by CrocoDuck »

loxstep wrote:Wonder how that would work. If it was anything like other AI projects I've encountered, an AI for audio would be trained on a corpus of existing songs. You could put in thousands of songs and it would look for patterns in how songs tend to be mixed / mastered. In the best case scenario, an AI could compare your track with similar commercial examples, and point out differences in the dynamic range, frequencies, etc. I bet a lot of people would find that useful. But I would be hesitant to use such a tool. I don't want my mixes to sound like everyone else's.
Another idea could be to learn an AI algorithm for every genre, maybe only the major ones. You would have a training set of couples of unmastered-mastered songs for each music genre. Then, the AI would learn a sort of unmaster-to-master relationship, perhaps as parameters for a generic bank of mastering plugins. Then, given a new unmastered song, the learned law would derive from it the parameters for the mastering chain.
loxstep wrote: I think mixing is too subjective to be automated. There is no right level something should be at. It depends on what you want your audience to pay attention to.
I would bet it is not as subjective as most people think. However, I think this sort of applications would be good to get good starting points rather than perfect solutions. Just like painters buy pre-made canvas instead of making their owns.
User avatar
loxstep
Established Member
Posts: 37
Joined: Thu Jan 11, 2018 12:40 am

Re: here's an idea using AI

Post by loxstep »

CrocoDuck wrote:I would bet it is not as subjective as most people think. However, I think this sort of applications would be good to get good starting points rather than perfect solutions. Just like painters buy pre-made canvas instead of making their owns.
You're not wrong. If you're working in an established style and the fans expect a certain sound, the mixing is hardly subjective at all. For example, you could rely on AI to tell you that 99% of modern pop songs have the vocals, drums, and bass predominant in the mix, and adjust accordingly. There's probably a market for such an AI.

If you're not working in a set style, and there are not many examples to go by, I don't think an AI would help.
Likewise, if you're the type of composer/producer who treats the studio like an instrument, an AI wouldn't be much help there either.

Side note: Not sure about mixing, but I know there are already automated programs that do mastering in set configurations (select: rock / pops / jazz / bossa nova). Probably not the same as hiring a mastering engineer, but some people like them. My experience with them is that they overcompress everything.

There's also this service which does automated mastering. Does this count as AI?
https://www.wavemod.com/

I'll stick with JAMin
User avatar
funkmuscle
Established Member
Posts: 2801
Joined: Mon Jun 02, 2008 2:30 pm
Has thanked: 129 times
Been thanked: 31 times

Re: here's an idea using AI

Post by funkmuscle »

loxstep wrote:
Side note: Not sure about mixing, but I know there are already automated programs that do mastering in set configurations (select: rock / pops / jazz / bossa nova). Probably not the same as hiring a mastering engineer, but some people like them. My experience with them is that they overcompress everything.

There's also this service which does automated mastering. Does this count as AI?
https://www.wavemod.com/

I'll stick with JAMin
yes services like LANDRhttps://www.landr.com/en
That's what I mean. They're using something with intelligence that masters the music. I'm now suggesting an AI kinda software plugin or standalone like JAMin but when you feed your music to it, it will do what LANDR does but you will have the final say(tweaking, etc.)

SO in other words, bring the 'online mastering services' to your desktop with you having the final say.
CrocoDuck
Established Member
Posts: 1133
Joined: Sat May 05, 2012 6:12 pm
Been thanked: 17 times

Re: here's an idea using AI

Post by CrocoDuck »

loxstep wrote:There's also this service which does automated mastering. Does this count as AI?
It might do. I think that any kind of program is a kind of AI. Maybe a very simple kind, but a kind nevertheless. Single ants, for example, are pretty much a binary chooser. It is like they were programmed with a switch statement. In terms of intelligence, a single ant is as good as the earliest spell checkers ever designed.

The interaction of thousands of ants propagates all their simple decisions in a vast network though, and the end result is that the hive is actually remarkably intelligent, much more significantly than a single ant.

This for saying that usually there are programs we don't think as AI (like a brutal spell checker from the 90s) that are actually as intelligent as real animals, in which we typically recognize at least a level of intelligence.
Last edited by CrocoDuck on Tue Feb 06, 2018 4:45 pm, edited 2 times in total.
CrocoDuck
Established Member
Posts: 1133
Joined: Sat May 05, 2012 6:12 pm
Been thanked: 17 times

Re: here's an idea using AI

Post by CrocoDuck »

funkmuscle wrote:That's what I mean. They're using something with intelligence that masters the music. I'm now suggesting an AI kinda software plugin or standalone like JAMin but when you feed your music to it, it will do what LANDR does but you will have the final say(tweaking, etc.)
If the problem needs to be formulated in terms of supervised learning one big obstacle will be getting a large (enough) bank of data from which the algorithm can learn. Companies can have funds and time to make large surveys, even involving high profile partners. If the problem can be unsupervised AI, then maybe it is not too much out of reach for not corporate developers.
CrocoDuck
Established Member
Posts: 1133
Joined: Sat May 05, 2012 6:12 pm
Been thanked: 17 times

Re: here's an idea using AI

Post by CrocoDuck »

I think that probably the simplest auto-mixer engine will have to work this way:

The session is a parallel arrangement of tracks. To each we assign a final gain stage and a filter. These two additional plugins are controlled by the algorithm. We will start from flat filters and neutral gains.

The goal of the algorithm is to use a masking and loudness model to predict, for each track, which frequency bands will not be audible in the complete mix. Then it tweaks the filters and gains in order to cut, from each track, the not audible bands. Repeat until the whole mix scores good perceptual neutrality in the frequency domain, maybe by assessing the energy in the bins of a parallel gamma tone filter bank (maybe in series with an outer ear filter simulator. Gamma tone filters simulate the inner ear behaviour).

Why? Well, this is how I mix stuff. I just filter out the parts of each track I cannot quite hear in the whole mix. Doing so increases the overall clarity of the mix, as each single instruments has its own "frequency window" in which it lives without being "disturbed" by the tails of the other instruments. Then, I also try to make it feel "complete", without large holes in the frequency domain, and balanced.

More than an AI in its most common sense, it would be a scripted version of what I do that uses "digitally simulated ears" instead of my own ears.
User avatar
funkmuscle
Established Member
Posts: 2801
Joined: Mon Jun 02, 2008 2:30 pm
Has thanked: 129 times
Been thanked: 31 times

Re: here's an idea using AI

Post by funkmuscle »

even something like thishttps://www.youtube.com/watch?v=dzgqT1SsJGw
I know we have a version of this in Linux but it doesn't do what this does.
User avatar
funkmuscle
Established Member
Posts: 2801
Joined: Mon Jun 02, 2008 2:30 pm
Has thanked: 129 times
Been thanked: 31 times

Re: here's an idea using AI

Post by funkmuscle »

Post Reply