Distro Comparison, Benchmarking, Latency

[i2productions]

In recent weeks, a certain video has come onto the scene comparing side by side the abilities of the top 5 current studio distros available for our linux consumer choice. Some have questioned the validity of said video. I would like you to all know that I'm currently putting together my own "side-by-side" video comparing at least 3 of those 5. I will be omiting Tango studio, as I feel it's unfair to compare a distro based on a 3 year old version of Ubuntu. Nothing against tango, it's a lovely product, just not one I feel is worth comparing. For sure I will be comparing KXStudio, AV Linux, and Dream Studio(All 32-bit). I will compare there uses both as live-media and installed to a hard drive. I have put out a call to l0wt3ch to provide a copy of Studio 13.37 (32-bit), to be included in this comparison as well. I am posting this a week ahead of the video for a very important reason. I would like input from the linux musician community to determine what tests you feel are important in making a fair comparison of these distros. My plans are to test:Live media boot time, live latency(from external source), live xruns, live stability, ease of install, install time, installed boot time, installed latency(from external source), and installed xruns.

[brummer]

You can do what ever you wont, compare this ore that, it's your opinion. Sorry, but I don't believe that if it will be worse more then any other vid of this subject. On the linux audio market we are not in a fight, more over, it is important to work together, to get the best out of the free open source available world wide in the net.
But, to be honest, you speak from top 5 studio distros?
At least one of them you mention is for sure not under the top 5, if you at least will call it a distro at all.
I wouldn’t naming it, but I guess all here know which one I mean.

[Capoeira]

In recent weeks, a certain video has come onto the scene comparing side by side the abilities of the top 5 current studio distros available for our linux consumer choice. Some have questioned the validity of said video. I would like you to all know that I'm currently putting together my own "side-by-side" video comparing at least 3 of those 5. I will be omiting Tango studio, as I feel it's unfair to compare a distro based on a 3 year old version of Ubuntu. Nothing against tango, it's a lovely product, just not one I feel is worth comparing. For sure I will be comparing KXStudio, AV Linux, and Dream Studio(All 32-bit). I will compare there uses both as live-media and installed to a hard drive. I have put out a call to l0wt3ch to provide a copy of Studio 13.37 (32-bit), to be included in this comparison as well. I am posting this a week ahead of the video for a very important reason. I would like input from the linux musician community to determine what tests you feel are important in making a fair comparison of these distros. My plans are to test:Live media boot time, live latency(from external source), live xruns, live stability, ease of install, install time, installed boot time, installed latency(from external source), and installed xruns.

nice one.

My thought is, we gotta find a way to proof the results a reliable and not faked.
First ideias: no screencapturing but a camera filming uncut. besides the monitor there could be put an analogic big watch which shows minutes and seconds.


I also would like to encourage IOwitch heavily to provide a testing media of his distro.

[Capoeira]

On the linux audio market we are not in a fight, more over, it is important to work together, to get the best out of the free open source available world wide in the net.

yes, but it happens that we got a rebel who loves to fight against our comunity.
Since we are liberal and we don't bann people, someone gotta "fill his mouth".

[brummer]

yes, but it happens that we got a rebel who loves to fight against our comunity.
Since we are liberal and we don't bann people, someone gotta "fill his mouth".

The best beat is nothing without "silence".
I guess that is what we have to give him, may we can enjoy then "his" beats.

[i2productions]

My thought is, we gotta find a way to proof the results a reliable and not faked.
First ideias: no screencapturing but a camera filming uncut. besides the monitor there could be put an analogic big watch which shows minutes and seconds.

I had this thought as well. Thinking of keep a running stopwatch app going on my android phone, and place it just at the bottom of the screen, and have it captured as well.

So that I make this clear up front, this is not as much about disproving the other video out there as it is about having an semi-accurate metric for potential new users to gauge a choice on. While I do have a bias towards KX Studio I will attempt to keep this experiment as objective as possible. I haven't even tried AV Linux 6, or the 12.04 version of Dream studio, so this is also testing for me. I have all weekend as my wife is housesitting,so I have ample time to complete this task over the next few days. I don't want this to turn into any kind of flame war, or make any lovers of any particular distro feel put out. Simply a benchmark!

[brummer]

So that I make this clear up front, this is not as much about disproving the other video out there as it is about having an semi-accurate metric for potential new users to gauge a choice on. While I do have a bias towards KX Studio I will attempt to keep this experiment as objective as possible. I haven't even tried AV Linux 6, or the 12.04 version of Dream studio, so this is also testing for me. I have all weekend as my wife is housesitting,so I have ample time to complete this task over the next few days. I don't want this to turn into any kind of flame war, or make any lovers of any particular distro feel put out. Simply a benchmark!

Anyhow, like I said, do what ever you wont, but you ask here, so you get a answer, even if you don’t like it.
Attempts like this, are just another stone in the wall

my (bold) 2cents on this

[Capoeira]

well, we got a obiously manipulated test here. It is here on the page. We don't believe in it, but others visiting the page do.
So i think a "real" test would be a good thing to have


on the other hand it can show the devs where to start to make their distros better

[raboof]

I would like input from the linux musician community to determine what tests you feel are important in making a fair comparison of these distros. My plans are to test:Live media boot time, live latency(from external source), live xruns, live stability, ease of install, install time, installed boot time, installed latency(from external source), and installed xruns.

I don't care much about boot and start up times

I do care about latency (both from internal and external sources), xruns and stability.

It would be cool if the tests would be as reproducable as possible, and if it would be easy re-run the tests with new/tweaked versions of the distro's, so we can find out what causes the differences (kernel version? kernel configuration? userspace versions? userspace configuration? etc) and work on making all the distro's equally awesome.

[l0wt3ch]

well, we got a obiously manipulated test here.

It is not a manipulated test, as much as you would like it to be. But feel free to conduct your own experiments, after all I was the one who suggested it.

Don't look to me for a contribution, however. I've got zero confidence in your ability to do a fair comparison.

But I do see all of you as quite comical!

"His video must be manipulated. His distro can't be that good. I mean, I've never tried it, but KX Studio has the best latencies and stuff, so it can't be true!"

"Ya that's right. Let's admit up front that we're biased, and then do our own comparison!"

"And we'll call him names for daring to speak the truth!"

What a bunch of silly children you are.

In case anyone is here from Google, this is the video that has everybody's panties in such a bunch. And, yes, the software is that good.

[l0wt3ch]

In case people don't feel like clicking the links, here are the quotes:

The nice thing about the JACK audio server is that you actually can compare latencies on different systems. It's not just opinion anymore at that point. Anyone who doesn't believe I was accurate with the video can easily download the other four distros and do the same tests, and they will get the same results. KXStudio and Dream Studio will be the slowest, with the worst latencies.

While I do have a bias towards KX Studio I will attempt to keep this experiment as objective as possible.

I also would like to encourage IOwitch heavily to provide a testing media of his distro.

[StudioDave]

... this is the video that has everybody's panties in such a bunch. And, yes, the software is that good.

The music is certainly annoying enough, which is of course its intention. It was annoying enough for me to shut off the video after two minutes. I get your point though - your distro is better than theirs. Okay, if you say so. Myself, I won't know until I try it, and I truly don't care how you represent your work or yourself here. I'm interested in your work, not you, so at some point I'll give your distro a fair turn here at Studio DLP.

Btw, Linux Weekly News will be running my review of KXStudio in a few weeks. I've added a brief note regarding other available systems, including yours.

Best,

dp

[brummer]

well, we got a obiously manipulated test here. It is here on the page. We don't believe in it, but others visiting the page do.
So i think a "real" test would be a good thing to have

Anyone who watch the vid could clearly see that it is a marketing stray. So what. It is completely meaningless, no mater if it is manipulated or not. Just no-one will purchase his product here. So what, maybe it is as good as he try to imply, who cares?
He just grep the work from other people and try to sell it, more people before have try that, all of them be history now.
As you see, he just get more food by this thread. Otherwise, nobody will care about his product.

on the other hand it can show the devs where to start to make their distros better

The work is still done all the time in background, kernel is moving forward, init systems be develop, audio-cart-drivers, alsa, fado, jack, ladish, dbus, . . . . all of that is under heavy development. Also apps and plug-in-systems. Do you really believe that dev's need such a vid to compare systems? Or to improve there own systems? Or at least that they care about?
Well, usually developers read the source and run test-cases by them-self, if they like to compare some-thing, then, they get a meaningful output for them-self.

[Capoeira]

As you see, he just get more food by this thread.

Well, I posted many times "do not feed him", but he is very talented in ofending people in a tricky way, so someone always feeds him.
anyway, ignoring him is always the best way, but as you see......

[l0wt3ch]

The music is certainly annoying enough, which is of course its intention. It was annoying enough for me to shut off the video after two minutes.

Can't please everybody, I guess. A lot of people like Ween, though. And I love that song.

I'm interested in your work, not you, so at some point I'll give your distro a fair turn here at Studio DLP.

Btw, Linux Weekly News will be running my review of KXStudio in a few weeks. I've added a brief note regarding other available systems, including yours.

Good to hear.

As for how I represent myself, it's unfortunate that I have to get into these silly arguments every time I try to make claims about my product. It's not just me, though; look how they treated the good Open Octave folks.

Sometimes I think the "Linux community" is about as "enlightened" as a bag of hungry rats.