LinuxMusicians

We're in the process of rebuilding everything on alternate servers as the forensics procedure at Virginia Tech simply takes too long. So hopefully tomorrow or beginning of next week we can flick the DNS switch.

Short recap, someone or something (this was probably an automated attack) probably got a reverse shell and exploited a local privilege escalation vulnerability, in this case Dirty COW. That's a somewhat older vulnerability which we could've mitigated by rebooting the server more often. The server was updated regularly but we were simply too sloppy with rebooting it as the linuxaudio.org is a hardware server sitting in some server room and there was some concern it wouldn't come back properly after a reboot.

The alternate servers are VM's so rebooting shouldn't be an issue anymore. They're also located in the EU on a fully open source cloud solution (OpenStack).

Regarding Twitter, unfortunately I have no access to that account. And we already have good backups and after the move that part is covered too.

Jeremy

OpenStack!!! That is something I'd like to learn in near future. This could be interesting for setting this up in my home environment even if people would tell me that is overkill as if an Ubiquiti UniFi 48-Port 500W PoE is overkill (48-port will be used for 4 video cameras, in-wall tablets, and a Doorbird powered by PoE, so I will have use of it when building a house in the near future).

OpenStack and Ubiquiti products (excluding AmpliFi) aren't designed for consumers in a home environment, but I'm more of a guy who likes having industrial-type products such as 1.5U custom-built servers and a 1U switch.

Anyway, good luck on getting the websites back online.

And yes, data forensics does take a while. It's important to preserve the data at all times for investigation and make sure all the access times are not updated upon touching the files in the filesystem.

PS: And yes, OpenStack does make sense for a large business environments as it's more for those who are looking to setup a hybrid cloud. I'm not certain if there are businesses out there that are using OpenStack internally as a private IaaS (Infrastructure as a Service) cloud.

A big "Thank you!" to everyone involved in helping with this. I was glad to hear there were backups!


brian

GraysonPeddie wrote:I'm not certain if there are businesses out there that are using OpenStack internally as a private IaaS (Infrastructure as a Service) cloud.

it is usable as a private IaaS cloud, but requires some knowledge how to set it up with simpler network infrastructure, because that one in official openstack documentation is certainly overcomplicated.

Even if I use conjure-up in Ubuntu?

Hi everyone, please stay on topic, thanks in advance!

Just like to add my thanks for all your hard work. Its often the case that we don’t fully appreciate what we have until its not there.

Cheers

I read in IRC the server was hacked ( my layman's term) but at that time I didn't realize kx and lmp depended on that server too.

Anyway, many thanks for taking care of this!
Let us all be patient and let the guys do their work

Thank you for the hard work and great resources. I'd like to support your work with a small donation... Where does one go for that? It would be great to be able to do that on Liberapay!

chtfn wrote:Thank you for the hard work and great resources. I'd like to support your work with a small donation... Where does one go for that? It would be great to be able to do that on Liberapay!

I would also be happy to support you through a small liberapay donation.

Thanks to all who contribute.

So far the progress is slow. We have to deal with a timezone difference, I'm in CET while the current server and the Virginia Tech department hosting the server are in EST, and also the communication itself is not optimal. And then there's a another time issue, I can't put all my available time into restoring the server, I have a responsible day job, a family with two kids and several bands I rehearse with. We also lost some time over discussing whether linuxaudio.org should move away from the VT server or not.

Luckily I got some help for the mail services and the owner of the linuxaudio.org domain is standing by to change the DNS. And your kind words certainly help too!!! Many thanks for the support!

Jeremy

First sites are starting to work again:

• kxstudio.linuxaudio.org
• kokkinizita.linuxaudio.org
• download.linuxaudio.org
• lac.linuxaudio.org/2018

Great news!

There might be some issues with SSL certificates, I revoked them all and renewed a few ones. We weren't using HSTS yet so the sites that are up should be accessible to everyone.

Tomorrow we'll move on and hopefully we can also get the mailing lists back online again.