Realtime group scheduling doesn't know JACK

From the article.

Keep up the good work Jon, I like it this way.

Speaking up is like sitting in the audience where a decision is being discussed. You can say your piece, but have little leverage in the active decision.

however if you are trying to do things in real-time, you can't afford to have large buffers as you need the real-world output to happen as close to the input as possible.

so to minimize the buffers, they need to be able to be sure of having CPU time when it's needed.

Right, so you're saying it's latency. But why is this such a big deal for audio? The latency between a user's input and the response comes up everywhere in an interactive system (moving the mouse and seeing the arrow move, clicking to close a browser tab, pressing spacebar to pause your mplayer video, shooting a gun in a video game, etc) and yet you don't hear every other developer complaining about latency. Why is the latency between audio input -> audio output be any more difficult than the latency between USB input -> video output?

when you are in a game and there is a slight delay between hitting the trigger and hearing the sound, it's not really that big a deal, but if the sound from the speakers is delayed from the sound directly from the audio source, it is very jarring.

The eye can see very short delays too. Certainly moving the mouse or typing on your screen would be a lot less enjoyable if you had 200ms lag in there. And lag alone can't be that big of a problem with audio -- just standing 20 feet away from the speaker will add 20ms on its own.

> when you are in a game and there is a slight delay between hitting the trigger and hearing the sound, it's not really that big a deal, but if the sound from the speakers is delayed from the sound directly from the audio source, it is very jarring.

OK, so it sounds like that concern isn't directly lag, but rather having two sources of audio that are out of sync. And this can often be accounted for by just delaying the faster source slightly to match the delay of the slower (laggier) one. That's how e.g. video games like Rock Band deal with the fact that modern TVs can have quite large lag -- they let you delay the audio so that the audio and video reach the user at the same time.

Can you provide a more specific example setup where you'd see the problem you describe?

however, 90ms of lag in mixing live audio is very noticable.

the difference is what you are comparing to.

your eyes can detect latency, yes, but they detect it by comparing different visual clues, finger (keyboard) to screen is not all visual.

The speed of sound in a low humidity environment is going to be something like 350 to 400 meters per second. I've heard that humans can't distinguish 10msec from instantaneous. That would be somebody standing about 4 meters away from you making a sound.

Probably 100msec is acceptable, I figure. From end to end. It'll sound like your standing a ways away in terms of response, but you can compensate.

But if you tune your system to deliver 30msec performance and Linux will randomly have latency spikes past 150 msec every time a disk is accessed then it's worthless to you.

Realtime is about being able to provide deterministic latency.... Linux can't deliver reliable performance without realtime configurations. You can't really tell what latencies you can rely on because the system is not configured to behave in that way. It will be optimized for batch processing and server workloads and in that you can latencies all over the place as long as that will deliver the best performance in the long run.

Human ear can distinguish about 500 microsecond difference. I.e. a difference in distance of about 20 centimeters. Humans use this for directional hearing.

That's why smallest possible latencies are so important.

But most of what I've seen on the topic seems to be complaining about other problems. The JACK FAQ says "JACK requires real-time scheduling privileges for reliable, dropout-free operation". I guess maybe that's just because they set their buffers so small to try to reduce latency. For users who aren't interested in the use case you described, it would be much easier to just increase the buffers and allow 50ms latency rather than have to go through all this trouble to try to get the kernel to do realtime.

Of course JACK is just meant for music production, you don't need it for everyday sound.

For audio applications what matters much more is that everything is predictable and you keep end to end latencies controllable. Imagine a concert pianist with a piano that will produce a sound randomly from instantaneous to quarter of a second to produce sounds when they press the keys. Some sounds may even come out of order. This is fine with TCP/IP, but Linux-based digital audio workstations with midi controllers and such that sort of performance would be unacceptable.

Linux under common configurations you will latencies vary by huge amounts. It is simply not capable of processing audio in a reliable enough manner that it can be suitable for audio production. You will see latencies randomly within a range of 10msec to 500 or a 1000 msecs depending on system loads. (probably 150-250msecs is very common) And system loads can change quite a bit... like double clicking on a fill to open a new MP3 sample, or aptd daemon kicking off and checking for security updates, or user alt-tab'ng between applications... Any disk activity is usually pretty damning.

What makes it worse for Linux (from a development perspective) is that high performance workloads often require a system with extremely lousy realtime performance. Think of server-style/database/batch workload. That way you take advantage of cache and all that happy stuff to make your processes go fast. Nice long, continuous reads from disk into memory, nice long lasting level3 caches to make your loops go fast, etc etc.

With a realtime-style workload you are going to be more then willing to flush those buffers and take page hits and all that stuff all over the place in exchange for the ability to say "X needs to get done every 10msec no matter what". So when Linux is configured for good realtime performance you will often see a dramatic loss in system efficiency if there is a realtime process actually running.

For audio work it's really important that you keep feeding the sound card and other devices audio data at a predictable rate. The hardware is effectively a "First In First Out" buffer that is always going to produce output regardless of whether or not you provide it information. Even if your system does not process the audio quick enough it is not going to stop your hardware from playing or recording audio information. As a result a system with poor realtime performance will create a large number of audio artifacts when loaded. You'll get echos, pops, squeeks, pauses, out of sync audio, out of sync video, stuttering video, scratches, etc etc and all other type of really bad things in your recordings and performances.

To solve the problem of buffer underruns, you need to increase the size of your buffers. If you cannot increase the size of your buffers due to latency concerns, or if you have hardware limitations, _then_ I agree you need realtime. But it seems to me that the vast majority of "I'm getting xruns with jack, I need realtime!!" type postings that I see can be solved better by increasing software buffer size, because they are use cases that don't also require low latency (e.g. playing one track while recording another, or playing and live mixing multiple tracks at once, etc). Maybe my perception is just wrong, and low latency is really a requirement more often than I think. Or maybe JACK does need to better handle the case where latency is OK and expected, and account for it rather than trying to push it under the carpet.

In some of my research work, we deal with streaming data from various capture cards at relatively high rate (up to 2Mbit/s). We write it directly to disk on a relatively busy system that can go out to lunch for 500ms or more. The only time I ever had to invoke SCHED_FIFO was with some poorly-designed USB hardware that only had a 384 byte buffer. Now we design our hardware with at least 1MB buffer and the kernel scheduler is no longer a concern.

when you are building and configuring a system, how do you know if the output is going to a recorder (where latency is acceptable), or to speakers (where it's not)

since there are some cases where you need the real time response, why not use that setting all the time rather than having one set of settings if you are going from live to live and a different set of settings if you are doing something where latency is acceptable?

No, it doesn't corrupt. And it's real world data coming in. And I also didn't have any corruption or artifacts when I had 500msec latency when I was running a MythTV backend recording two streams of live television simultaneously. And I can play DVD video just fine even though my old DVD-ROM drive has seek times on the order of 250ms. Etc, etc.

That's my whole point -- there is a whole world of other systems that (to me) seem much more demanding than audio processing, like video input and output, and gaming, that don't seem to have the realtime requirements or problems of JACK. I'm trying to understand exactly what the audio use cases are that make processing 48 KHz data so difficult. And it seems like the answer is the specific case of live looping back audio from input->output which needs low latency.

Or consider DJing. Suppose we just have a 2 second buffer. So I hear the end of track A, there's a boring instrumental lead out, I want to play track B - must I wait 2 seconds before it begins? That's a long time. OK, so maybe we provide the DJ a separate mix. Now I can hear where track B will start playing, but it takes 2 seconds before I hear my own decisions. This is very difficult to work with.

You can play with this, if you have lab audio equipment, create a 2 second delay audio loop, where what you say is recorded and then played back quietly through headphones (so it doesn't feed back). You won't be able to hold a conversation properly, because you must concentrate very hard just to override a built-in audio-cognitive mechanism that tells you you're being interrupted. Reduce the loop to 15ms. Now it's fine. The much shorter delay resembles environmental echo, which was present when our audio cognition evolved. Latency matters.

192000 samples/second * 8 movie channels * 4 bytes/sample * 128 streams / 1048576

That's a whole 750MB/s ... my desktop computer has 30GB/s dram bandwidth. For the highly more common 48K * 2 ch * 2 b/sample * 32 streams, the bandwidth is a pathetically small 6MB/s.

The fact that the Linux kernel can't natively handle mixing a few audio streams for the average desktop user without trotting out the anti-DoS argument is silly.

Ouch. My only experience with lowish-latency audio on Linux was back in 2005 when I was trying to make WoW in Wine as responsive as possible. At the time, the buffer could be squeezed down to about 90ms, but any lower would risk very occasional underruns when under heavy load (and always caused by disk activity, not CPU) - becoming quite frequent underruns by the time you got to a target latency of 50ish. Using rt could push it a couple of 10s of ms lower but wasn't really necessary for gaming and came at a cost in throughput.

If we now need rt scheduling to avoid being several times worse than we were on cheap hardware years ago, then something is surely wrong. I wonder if this difference is due to software, or the continued decline in quality of audio hardware (eg. all mixing is done in software nowadays because the hardware can't do it).

That's assuming that Linux is in charge of measuring the timing between MIDI keypresses, and therefore a delay in Linux will add a delay in the music. Why is that the case? Is MIDI hardware really that bad? I would hope the MIDI hardware sends data to the computer that looks like:
1. Pressed C at 12:00:00.102
2. Pressed D at 12:00:00.112
3. Pressed A at 12:00:00.122
4. Pressed B at 12:00:00.132

If the computer is busy for 25ms and misses #2 and #3, that's fine, it will read those events out of a buffer when it returns. The data will still show a keypress every 10ms as intended.

If MIDI hardware is exceptionally stupid and requires the computer to perform all timings, then it sounds like there's a market for a very simple hardware enhancement that would avoid this sort of issue.

> You have to realize that a lot of compositions are done in layers so
> that you have something playing, and you add another track to it. In
> those cases, it's critical that the time between pressing the key on the
> piano keyboard and registering it be kept to a minimum. Buffering will
> not help you there.

In this case I'd argue that the output and input streams need only to be _synchronized_, not that there is zero-latency between them. Assume you have a huge buffer for audio output, and a huge buffer for the MIDI input. The computer should be able to know that "output sample #12345 made it to the speaker at 12:00:00.123" and "keypress C on the piano was registered at 12:00:00.123" based on either measured latencies, reported timestamps from the relevant devices, or sample rate info plus manual adjustment (like Rock Band's calibration screen). Even if the computer is busy and doesn't actually register the keypress until 12:00:00.456, you can still with 100% accuracy match the keypress to the proper place in the music, so that when you go to play it back later you can put it exactly where the musician intended.

It really seems to me like the approach of audio guys is to try to get latency to zero, rather than accepting variable latency and simply accounting for it with larger buffers and proper timestamping. Zero latency seems like a losing battle because you'll never reach it, so for use cases like you described where you can correct for the effects of latency, that seems like a better solution.

Uh, no. In fact, you can't do jackd with 0 latency and that isn't the goal. RTFS. Your scenario is FAIL and it is clear you have never used it in production. Why not actually listen to the people that actually have done this?

I think my only real error was underestimating how frequently people need to either apply live digital effects or otherwise play live synthesized audio from live external triggers, which sounds like "always" from what people are saying here.

From what it says on Wikipedia, it's played as a duet - i.e. the music you play is still in time with itself; your part is in phase with the other part.

Still hell to play, yeah, but it would be pretty much impossible if the sound from your own piano were to come at you with an audible delay. It would be easier to play if you were deaf - at least then nothing would interfere with the rhythm in your head.

increasingly things are digital for large portions of the path, and may only become analog when they go to the amplifiers.

My main point was that low latency isn't the only reason for requiring RT privileges.

but I agree that low latency isn't the only reason.

More to the point, even with large buffers I can't get JACK to work properly _at all_ without RT scheduling privileges, so low latency is a bit of a distraction in this argument IMO. JACK needs RT privileges just to work reliably without being pushed aside by anything else the system might be doing.

What does 'at all' mean? Like it won't even start? Or it runs but suffers constant underruns? Presumably not the latter since larger buffers should eliminate that problem. Or maybe you mean that without RT you occasionally get such tremendous latency spikes that the buffer size would need to be absurd to avoid them.

Just interested really since the behaviour you see doesn't seem intuitive.

That's what I meant.

Still don't see why JACK would be much worse than common desktop audio systems in this scenario though.

that means that the total latency that can be tolorated is less than for a normal desktop app, so the buffers can't be as large.

There exists at least one choice of buffer size which is sufficient for at least one possible purpose, with which it is possible to get skip-free audio using other sound systems on a normal kernel, but only on a real-time kernel using JACK.