Ultralite AVB

[wrl]

Anybody else testing this out? Any feedback?

-w

[ahellquist]

Anybody else testing this out? Any feedback?

-w

I just downloaded 5.11rc2 tarball to give it a go.

I am running firmware 1.3.4+558 on my device (Utralite AVB non-ESS) right now..
Do I need to upgrade to newer firmware to get the usb protocol up to date ??

I will try to get my system running with the new kernel as soon as I applied the patch and built it successfully

/Anders

[wrl]

I am running firmware 1.3.4+558 on my device (Utralite AVB non-ESS) right now..
Do I need to upgrade to newer firmware to get the usb protocol up to date ??

I will try to get my system running with the new kernel as soon as I applied the patch and built it successfully

/Anders

If that's still before they added implicit feedback support then probably hold off testing this until we're sure it's stable enough. I wouldn't want you to lose a working device to help test a WIP patchset. Not that the patch is going to brick your device, but if it's already working, maybe don't mess with it until we're 100% confident in this patch.

-w

[ahellquist]

I am running firmware 1.3.4+558 on my device (Utralite AVB non-ESS) right now..
Do I need to upgrade to newer firmware to get the usb protocol up to date ??

I will try to get my system running with the new kernel as soon as I applied the patch and built it successfully

/Anders

If that's still before they added implicit feedback support then probably hold off testing this until we're sure it's stable enough. I wouldn't want you to lose a working device to help test a WIP patchset. Not that the patch is going to brick your device, but if it's already working, maybe don't mess with it until we're 100% confident in this patch.

-w

My firmware is a test build that is not published on motu.com anymore but i have audio glitches from time to time but no channel drifting and never had.

I think it is fairly safe to test and rolling back to this firmware is supported and booting my previous kernel is also not a big problem. I will give it a try to add feedback to this thread.

My kernel and modules are compiling right now and it would be a waste of CPU cycles not to test them.

[wrl]

Alright, sounds good. Yeah, I wouldn't want you to be in a situation where you had upgraded to a new firmware in anticipation of this patch, found it didn't work, and then couldn't downgrade. If you're able to downgrade then I think you're in the clear.

-w

[ahellquist]

Alright, sounds good. Yeah, I wouldn't want you to be in a situation where you had upgraded to a new firmware in anticipation of this patch, found it didn't work, and then couldn't downgrade. If you're able to downgrade then I think you're in the clear.

-w

Ok, yes, downgrading is fully supported. Which is great for those who need 64/64 usb channels in CC mode which is only supported in the very first firmware versions..

[shellwalker]

Yeah, I've also considered this and acknowledge that it's a possibility, but I think in practice it's unlikely to happen. I've been using my 828es during testing for hours-long production sessions without issue, then just left it running for days after that, checking in several times a day to make sure everything was kosher.

The approaches I've tried which were unsuccessful tended to very obviously drift after 18-24 hours and then "stayed" drifted or just gradually drifted even further forward. Sure, selection bias (and your sample rate analogy is apt) but I think that the scenario where a device is drifting but only enough to consistently appear like nothing is wrong... I mean, Schrödinger's USB device, no?

Regardless – I've got a patched 5.11rc2 kernel smoke testing right now. ~24 hours in, no drifting yet.

-w

I had a strange evening yesterday - experiencing a lot of drifting, even with the patched linux-next_20201223!!
Today, installed pending updates, powered down the system and have started a new test.

My observations so far:
- at the beginning of the test I had to search around to find which channel is actually working. Initially I tried "playback_1" and "playback_2", but there was no signal on them. So I tried higher numbered ones until I found the signal.
- strangely after finding the signal on higher numbered ones, I then could also find it on playback_1+2!!

I'll let Ardour continue playback in a loop for a while and come back later to see what's happened.

Another thing: in your case, have you observed the kernel messages about clock source not being valid and also about the "duplicate endpoints"?

Code: Select all

[Fr Jan  8 09:34:45 2021] usb 1-5: new high-speed USB device number 8 using xhci_hcd
[Fr Jan  8 09:34:45 2021] usb 1-5: config 1 interface 6 altsetting 1 has a duplicate endpoint with address 0x9, skipping
[Fr Jan  8 09:34:45 2021] usb 1-5: config 1 interface 7 altsetting 1 has a duplicate endpoint with address 0x87, skipping
[Fr Jan  8 09:34:45 2021] usb 1-5: New USB device found, idVendor=07fd, idProduct=0005, bcdDevice= 2.00
[Fr Jan  8 09:34:45 2021] usb 1-5: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[Fr Jan  8 09:34:45 2021] usb 1-5: Product: 828ES_LEXXY
[Fr Jan  8 09:34:45 2021] usb 1-5: Manufacturer: MOTU
[Fr Jan  8 09:34:45 2021] usb 1-5: SerialNumber: xxxxxxxxxxxxxxxxx
[Fr Jan  8 09:34:50 2021] usb 1-5: clock source 1 is not valid, cannot use
[Fr Jan  8 09:34:51 2021] usb 1-5: 1:1: freq mismatch (RO clock): req 192000, clock runs @88200
[Fr Jan  8 09:34:56 2021] usb 1-5: 2:1: freq mismatch (RO clock): req 192000, clock runs @176400
[Fr Jan  8 09:34:57 2021] usb 1-5: clock source 1 is not valid, cannot use
[Fr Jan  8 09:34:58 2021] usb 1-5: 2:1: freq mismatch (RO clock): req 44100, clock runs @192000
[Fr Jan  8 09:34:59 2021] usb 1-5: 2:1: freq mismatch (RO clock): req 44100, clock runs @192000
[Fr Jan  8 09:34:59 2021] usb 1-5: clock source 1 is not valid, cannot use
[Fr Jan  8 09:35:00 2021] usb 1-5: 2:1: freq mismatch (RO clock): req 44100, clock runs @192000
[Fr Jan  8 09:35:00 2021] usb 1-5: clock source 1 is not valid, cannot use
[Fr Jan  8 09:35:00 2021] usb 1-5: clock source 1 is not valid, cannot use
[Fr Jan  8 09:35:00 2021] usb 1-5: clock source 1 is not valid, cannot use
[Fr Jan  8 09:35:00 2021] usb 1-5: clock source 1 is not valid, cannot use

[shellwalker]

Sadly my test has confirmed bad behaviour. Here is my test log:
Start: 09:38 on playback_1 + playback_2 (though initially there was no signal on them, after trying out other channels the signal appeared also on these...)
End: 10:15 . Signal no longer on playback_1+2 -> Drifting has occurred Since I wasn't observing this all the time, I can't say how long it actually stayed stable.

Now just to be sure we are on the same page:
1. The patch should work without any special kernel command-line options, right?
2. It should work without any USB tweaks (like those discussed with Drumfix), right?
3. It should work without any CPU affinity tweaks, right?

My current kernel-command-line:

Code: Select all

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-5.10.0-next-20201223auh-2+ root=/dev/mapper/vgkubuntusys-root ro rootflags=subvol=@ acpi_osi=! "acpi_osi=Windows 2009" quiet splash psmouse.synaptics_intertouch=1

I'll retry the test with additional parameters: "acpi_irq_nobalance nowatchdog nohz=off intel_idle.max_cstate=1"

[shellwalker]

Sad news!

I retested with additional kernel parameters:

Code: Select all

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-5.10.0-next-20201223auh-2+ root=/dev/mapper/vgkubuntusys-root ro rootflags=subvol=@ acpi_osi=! "acpi_osi=Windows 2009" quiet splash psmouse.synaptics_intertouch=1 acpi_irq_nobalance nowatchdog nohz=off intel_idle.max_cstate=1

Results:
Start: 10:33 on playback_1 + playback_2 (this time the signal was immediately here)
End: 11:26: Signal no longer on playback_1+2 -> Drifting has occurred Again, since I wasn't observing this all the time, I can't say how long it actually stayed stable.

@wrl: any ideas what may be going wrong at my end? Any ideas what may be different at your end?

[fbacker]

Just ordered one of these as an upgrade from my Scarlett 2i4 that I couldn't get working, only to come across this.

How can I tell when that patch gets merged into the kernel?

[wrl]

@wrl: any ideas what may be going wrong at my end? Any ideas what may be different at your end?

Man, this sucks. I'm observing the same things you are about the duplicate endpoints and clock source:

Code: Select all

[    1.177019] usb 3-2: new high-speed USB device number 2 using xhci_hcd
[    1.324255] usb 3-2: config 1 interface 6 altsetting 1 has a duplicate endpoint with address 0x9, skipping
[    1.324258] usb 3-2: config 1 interface 7 altsetting 1 has a duplicate endpoint with address 0x87, skipping
[    1.336245] usb 3-2: New USB device found, idVendor=07fd, idProduct=0005, bcdDevice= 2.00
[    1.336250] usb 3-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    1.336252] usb 3-2: Product: 828ES
[    1.336253] usb 3-2: Manufacturer: MOTU
[    1.336254] usb 3-2: SerialNumber: 0001f2fffe00f347
[   19.642951] usb 3-2: clock source 1 is not valid, cannot use
[   24.530030] usb 3-2: clock source 1 is not valid, cannot use
[   39.878845] usb 3-2: clock source 1 is not valid, cannot use

Unfortunately, if this isn't already working, then it's not going to work. This patch deals with one hypothesis and one hypothesis only: that the drifting is due to scheduling delays between receiving an inbound (capture) transfer and submitting an outbound (playback) transfer. This patch eliminates that delay entirely – there's no further we can go on this path.

If this isn't working for folks, there is a slim to zero chance that my hypothesis is correct at all, and I'll need to take a different approach. I'm going to edit the post that has the patch in it to indicate this but then it's back to the drawing board.

-w

[shellwalker]

Unfortunately, if this isn't already working, then it's not going to work. This patch deals with one hypothesis and one hypothesis only: that the drifting is due to scheduling delays between receiving an inbound (capture) transfer and submitting an outbound (playback) transfer. This patch eliminates that delay entirely – there's no further we can go on this path.

If this isn't working for folks, there is a slim to zero chance that my hypothesis is correct at all, and I'll need to take a different approach. I'm going to edit the post that has the patch in it to indicate this but then it's back to the drawing board.

-w

ok, but before we give it up entirely, just let me ask a few pendantic questions (as a novice!!), just to be sure:
1. Is the dmesg about the "clock source 1 is not valid, cannot use" at all related to the drifting?
2. I just checked which USB port I'm using and it appears to be a USB3.2 Gen 1 port. Is that also covered by xhci in the kernel or would the patch need to be applied elsewhere? I'm asking because you mentioned you would need to port the patch to EHCI as well, so I read between the lines that the various USB standards require separate implementation in the kernel...
3. Is there a possibility to activate some sort of debug trace feature in order to track where things are going wrong?
4. What could be the difference between your system and mine? Why was yours so stable?
5. What about the suggestions made by Drumfix, where he stated following:

You need both flags.

URB_FAST_COMPLETION must be set in all submitted ISO URBs, no matter if they are submitted from inside and outside of the completion handler.
USB_HOST_LOCK_IS_ACQUIRED must be set in ISO urbs submitted from inside the completion handler only.

Analyzing Drumfix's statement, I could see the following:
a. Maybe your patch is actually NOT setting URB_FAST_COMPLETION in _all_ ISO URBs (whatever that is...), because maybe you missed some code somewhere?
b. Maybe setting URB_FAST_COMPLETION on all ISO URBs is having some side-effect elsewhere?

Just throwing in a few random thoughts just to make sure we don't close this path prematurely..

[shellwalker]

Adding to my latest post:
I just ran a pure "next-20201223" without the patched applied:
1. also here I'm seeing the dmesges about the "not valid " clock source. So I guess that is not related to the patch.
2. I'm seeing issues with my thunderbolt setup (network card in the docking station is not being detected). This is with as well as without patch. Now, I guess Thunderbolt is related to USB in some way, so maybe there is anyhow some strange things going on in "next-20201223"?

-> would it be feasible to have your patch applied to a stable kernel just to rule out that we are stumbling over some kind of "cross-talk" here?

[wrl]

ok, but before we give it up entirely, just let me ask a few pendantic questions (as a novice!!), just to be sure:
1. Is the dmesg about the "clock source 1 is not valid, cannot use" at all related to the drifting?

Not from what I can tell. That also shows up on my system and I don't have the drifting issue with this patch.

2. I just checked which USB port I'm using and it appears to be a USB3.2 Gen 1 port. Is that also covered by xhci in the kernel or would the patch need to be applied elsewhere? I'm asking because you mentioned you would need to port the patch to EHCI as well, so I read between the lines that the various USB standards require separate implementation in the kernel...

USB3 is XHCI, so you should be good. The patch does also cover EHCI.

3. Is there a possibility to activate some sort of debug trace feature in order to track where things are going wrong?

Not sure yet, need to dive in more to figure out how to instrument this stuff. This patch was a really early test of the hypothesis made without me fully understanding the implications, but, hey, "worked on my system" and I was curious.

4. What could be the difference between your system and mine? Why was yours so stable?

That's kind of the big question here, and I really have no idea. Might be down to the USB controller.

Could you paste your lspci?

5. What about the suggestions made by Drumfix, where he stated following:

You need both flags.

URB_FAST_COMPLETION must be set in all submitted ISO URBs, no matter if they are submitted from inside and outside of the completion handler.
USB_HOST_LOCK_IS_ACQUIRED must be set in ISO urbs submitted from inside the completion handler only.

Analyzing Drumfix's statement, I could see the following:
a. Maybe your patch is actually NOT setting URB_FAST_COMPLETION in _all_ ISO URBs (whatever that is...), because maybe you missed some code somewhere?
b. Maybe setting URB_FAST_COMPLETION on all ISO URBs is having some side-effect elsewhere?

Just throwing in a few random thoughts just to make sure we don't close this path prematurely..

URB_FAST_COMPLETION is set on all of the snd-usb-audio playback/capture URBs in this patch, so I think that's unlikely to be the case. The USB_HOST_LOCK_IS_ACQUIRED flag is an improvement I didn't get around to but which wouldn't have made a difference (perhaps a small one, but very unlikely to be the cause of the patch just not working on other systems).

Also, a message from further up in the thread – my kernel commandline has nothing except for initrd=, root=, and cryptdevice=. None of the IRQ stuff or nohz.

-w

[shellwalker]

Not sure yet, need to dive in more to figure out how to instrument this stuff. This patch was a really early test of the hypothesis made without me fully understanding the implications, but, hey, "worked on my system" and I was curious.

An idea I had regarding debugging: maybe it is possible to inject "known data" as an audio stream which can be checked in the kernel. Something like a known series of changing values. That way within the kernel you could detect at what point in time the data starts wandering away into the wrong channels.
This could also be the basis for drift detection in user-space as well - route back the signal within the motu and check the values in user space.

That's kind of the big question here, and I really have no idea. Might be down to the USB controller.

Could you paste your lspci?

1. One new thought on the differences: how is your motu 828es configured? In my case I have:
- 6 "Computer to MOTU" channels ("From USB").
- 14 "Motu to Computer" USB channels.

2. Here come my system details:

Code: Select all

# lspci 
00:00.0 Host bridge: Intel Corporation Device 3e20 (rev 0d)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 0d)
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (Mobile) (rev 02)
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:14.3 Network controller: Intel Corporation Wireless-AC 9560 [Jefferson Peak] (rev 10)
00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10)
00:15.2 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #2 (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17 (rev f0)
00:1b.4 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1d.4 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #13 (rev f0)
00:1f.0 ISA bridge: Intel Corporation HM470 Chipset LPC/eSPI Controller (rev 10)
00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
01:00.0 VGA compatible controller: NVIDIA Corporation TU106M [GeForce RTX 2070 Mobile] (rev a1)
01:00.1 Audio device: NVIDIA Corporation TU106 High Definition Audio Controller (rev a1)
01:00.2 USB controller: NVIDIA Corporation TU106 USB 3.1 Host Controller (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU106 USB Type-C UCSI Controller (rev a1)
02:00.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02)
03:00.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02)
03:01.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02)
03:02.0 PCI bridge: Intel Corporation JHL6340 Thunderbolt 3 Bridge (C step) [Alpine Ridge 2C 2016] (rev 02)
04:00.0 System peripheral: Intel Corporation JHL6340 Thunderbolt 3 NHI (C step) [Alpine Ridge 2C 2016] (rev 02)
05:00.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02)
06:01.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02)
06:02.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02)
06:03.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02)
06:04.0 PCI bridge: Intel Corporation JHL6540 Thunderbolt 3 Bridge (C step) [Alpine Ridge 4C 2016] (rev 02)
07:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller
08:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)
09:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)
3a:00.0 USB controller: Intel Corporation JHL6340 Thunderbolt 3 USB 3.1 Controller (C step) [Alpine Ridge 2C 2016] (rev 02)
3b:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
3c:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
3d:00.0 Ethernet controller: Qualcomm Atheros QCA8171 Gigabit Ethernet (rev 10)

Code: Select all

# lsusb
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 012 Device 002: ID 0bda:8153 Realtek Semiconductor Corp. RTL8153 Gigabit Ethernet Adapter
Bus 012 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 011 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 009 Device 002: ID 0572:1a05 Conexant Systems (Rockwell), Inc. CONEXANT USB AUDIO
Bus 009 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 010 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 002: ID 05e3:0612 Genesys Logic, Inc. Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 003: ID 06cb:009b Synaptics, Inc. 
Bus 001 Device 008: ID 07fd:0005 Mark of the Unicorn 828ES_LEXXY
Bus 001 Device 006: ID 04f3:0103 Elan Microelectronics Corp. ActiveJet K-2024 Multimedia Keyboard
Bus 001 Device 004: ID 1017:6002 Speedy Industrial Supplies, Pte., Ltd 
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 007: ID 8087:0aaa Intel Corp. 
Bus 001 Device 005: ID 5986:211c Acer, Inc HD Webcam
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Code: Select all

dmesg output when connecting:
[Sa Jan  9 09:09:27 2021] usb 1-5: new high-speed USB device number 8 using xhci_hcd
[Sa Jan  9 09:09:27 2021] usb 1-5: config 1 interface 6 altsetting 1 has a duplicate endpoint with address 0x9, skipping
[Sa Jan  9 09:09:27 2021] usb 1-5: config 1 interface 7 altsetting 1 has a duplicate endpoint with address 0x87, skipping
[Sa Jan  9 09:09:27 2021] usb 1-5: New USB device found, idVendor=07fd, idProduct=0005, bcdDevice= 2.00
[Sa Jan  9 09:09:27 2021] usb 1-5: New USB device strings: Mfr=1, Product=2, SerialNumber=3