Missing the AF_BUS

Do keep in mind that there are a number of avenues to improve D-Bus in userspace as well; I'm mystified why so much effort has gone into kernel changes and near zero into userspace:
http://lists.freedesktop.org/archives/dbus/2012-March/015...

but in any case the opportunity is there. My belief is that nobody is chasing this because for the vast majority of people D-Bus performance is not an actual problem. When I've asked for concrete examples of when it was a problem, things like Nokia N900 (iPhone 3G era hardware right?) come up, and poorly coded applications aren't ruled out and seem likely to be involved even in that case.

basically there is just no need to performance tune the kind of stuff dbus is normally used for on a stock Linux desktop... if something is only 1% of user-visible speed, making it double fast isn't perceptible.

people do show up on the mailing list using dbus on low resource embedded systems and needing ultra low latency or something, but in those cases dbus was pretty clearly a poor choice of hammer for the nail at hand.

I don't think Alan is wrong though. the dbus semantics and guarantees that make it slow are also what make it convenient, and app developers generally want those guarantees and apps are less buggy if they have them. So it might be nice to make this genre of thing fast, even if the simple notifications etc. used by the desktop aren't performance critical, there are other domains that might benefit from ordered, reliable delivery, lifecycle tracking, etc. there's no question a faster implementation of dbus would be more broadly useful beyond just the desktop.

Concerning your comments on userspace D-Bus improvements. Collabora employ one of the upstream D-Bus maintainers and pay for some of their time to work on D-Bus.

We most definitely are committed to improving the userspace side of D-Bus in addition to the kernel work (which was a project for the GENIVI alliance)

Our eventual aim using all the solutions is for a tripling in throughput and a significant reduction of latency for the general case.

> poorly coded applications aren't ruled out

On the system bus, which is a trust boundary, poorly- or even maliciously-coded applications can never be ruled out, unfortunately.

> in those cases dbus was pretty clearly a poor choice of hammer for the nail at hand

People consider D-Bus to be a suitable transport for all sorts of things, desktop or not. The first sentence of the specification describes it as "a system for low-latency, low-overhead, easy to use interprocess communication", which probably contributes to the view that it's the right hammer for every nail - in practice, its current design tradeoffs tend to prioritize "easy to use" over low-latency.

Improving its latency, and avoiding priority inversion between the dbus-daemon and its clients, certainly increases the number of situations where D-Bus can be used. They might not be necessary for "the desktop bus", but that's by no means the only thing people use D-Bus for.

Improving the kernel-level transport is orthogonal to improving the user-space part (of which message (de)serialization is indeed likely to be the lowest-hanging fruit), and there's no reason they can't both happen.

> the dbus semantics and guarantees that make it slow are also what make it convenient

I absolutely agree that the convenient semantics - multicast signals, total ordering, conventions for lifecycle tracking and so on - are what make D-Bus valuable, and if you're willing to sacrifice those convenient semantics for performance, that's a sign that D-Bus is not right for you. Having said that, given the constraints of those semantics, the more efficient the better, and AF_BUS proves that there is room for improvement.

Apparently people now try to do some things across DBus that require a higher data rate and lower latency than the traditional "huh, somebody plugged in a USB stick / wants root for aptitude/yum / left fpr lunch" events DBus handled in the past.

Newfangled stuff like multi-touch. Or keyboard input (for Chinese and whatnot).

You don't want that stuff to go through more context switches (and processes) than strictly necessary. So AF_BUS seems to be a Good Thing.

It would be nice to be also able to use DBUS over the network. If you stick to DBUS-only protocol then it's not a problem.

But the minute you start using socketpairs - it becomes impossible.

socketpair-ing the sender and recipient would probably work, but it has a few downsides.

* you lose the easy debugging and monitoring you now get with DBus (presumably, with AF_BUS you could use something like wireshark),

* the client now has to juggle multiple file descriptors, that requires an API change

* multiple file descriptors and reliable message ordering don't mix

Too many downsides if you ask me.

I think I was unclear: I'm not suggesting that DBus do some sort of socketpair thing to provide high-speed dbus links; I'm suggesting that use cases that don't really care about DBus, but need to negotiate with another program for their input, could get back to their normal operation (input from a fd) by having the other side send a fd back when it's all set up. That is, user input doesn't normally go over DBus, so applications are already designed around a non-DBus input path; even if you need IPC to set up your input device appropriately, it doesn't make too much sense to get user input over DBus, particularly if you can't get corresponding user input over DBus if the system doesn't have Chinese input or multi-touch.

Of course, it's certainly possible that people will want high-speed IPC with DBus properties also, and it makes sense for DBus to be efficient regardless of whether it's running into performance constraints. But it doesn't make sense to use DBus for all communication, even if its performance could be made good enough.

> When I've asked for concrete examples of when it was a problem, things like Nokia N900 (iPhone 3G era hardware right?) come up

One of the worst issues is D-BUS message delivery reliability. All it needs is an app that subscribes for some frequent message (like device orientation) and then doesn't read its messages either because it was supended or just buggy. As message delivery needs to be reliable, D-BUS will then just buffer the messages and get all the time slower and slower as it starts to swap.

Second issue is too complicated D-BUS setup. I think e.g. the N900 call handling goes through half a dozen daemons before the call UI pops up. Each of these steps adds it's own socket buffering and process scheduling overhead in addition to other overheads (e.g. paging the processes in to RAM if they were swapped out etc).

Then there's the D-BUS daemon code itself. Ever wondered why something that's "just" supposed to read and write data from sockets is CPU bound instead of IO bound? D-BUS daemon spends a lot of CPU on message content marshaling.

How easy? can you just give me a starter, please?

Would you mind providing (links to) more information, for people interested in learning about the purported inefficiencies in Linux's TCP stack?

I'll have to tell you about it, because the actual code is buried deep in somebody's trading engine and they would likely take issue with me posting it on the web. Profiling turned up some really bad CPU bumps in places you would not immediately suspect, like UDP send, which was taking nearly a microsecond per packet more than it should. I thought there would actually be some deep reason for that, but when I dug in I found that the reason was just sloppy, rambling code, pure and simple. I straightened it all out and cut the CPU overhead in half, consequently reducing the hop latency by that amount. I went on to analyze the rest of the stack to some extent and found it was all like that. You can too, all you need to do is go look at the code.

Here's a lovely bit:

http://lxr.linux.no/#linux+v3.4.4/net/ipv4/tcp_output.c#L796

This is part of a call chain that goes about 20 levels deep. There is much worse in there. See, that stuff looks plausible and if you listen to the folklore it sounds fast. But it actually isn't, which I know beyond a shadow of a doubt.

>This code just kills efficiency by a thousand cuts. There is no single culprit, it is just that all that twisting and turning, calling lots of little helpers...

Much of the complexity of that function has to do with kernel support for fragmented skbs, which is required for packets that are larger than the page size. That is the sort of thing that would go away if the kernel adopted a kernel page size larger than the hardware page size in cases where the latter is ridiculously small.

I am not sure what the real benefits are of managing everything in terms of 4K pages is on a system with modern memory sizes. Perhaps the idea of managing everything in terms of 64K pages (i.e. in groups of 16 hardware pages) could be revisited. That would dramatically simplify much of the networking code, because support for fragmented skbs could be dropped. No doubt it would have other benefits as well.

> I straightened it all out and cut the CPU overhead in half, consequently reducing the hop latency by that amount. I went on to analyze the rest of the stack to some extent and found it was all like that. You can too, all you need to do is go look at the code.

> This is part of a call chain that goes about 20 levels deep. There is much worse in there. See, that stuff looks plausible and if you listen to the folklore it sounds fast. But it actually isn't, which I know beyond a shadow of a doubt.

Don't you have some notes, implementation ideas or performance tests that you want to share with the rest of the kernel community? I'm pretty sure that they would love to hear how to cut in half the CPU overhead of UDP messages without regressions in functionalities.

This kind of impact would surely reduce the battery consumption of mobile applications, so, maybe the main developers will not interested, but devs of mobile-oriented forks like Android will surely be.

I should add that fragmented skbs are used for zero copy support too, so if the idea is to simplify the networking stack by dropping them, zero copy would be out. On the other hand, zero copy seems to be usable for sendfile() and not much else, so that doesn't sound like much of a loss if it improves the much more common case.

> and that reliable delivery of multicast messages cannot be done, even in the limited manner needed by D-Bus

I, too, was wondering how they (D-Bus) [expect to] achieve this...

In kernel dbus, when the sender wants to send a message to N recipients, it first checks that there is space in the queue for each recipient. If there is, it puts the message there. If there isn't, then it returns an error. It's that simple.

Couldn't it still be a kernel module that dbus could use optionally?

> It works, but the routing daemon adds context switches, overhead, and latency to each message it handles.

Why not instead convert the dbus daemon into a kernel module, like has been done in the past with the http daemon? It would avoid having to context switch to and from the daemon, and need no changes to the networking subsystem.

Note: I am joking.

"so there may be a significant amount of multicast interpersonal messaging required before we have multicast interprocess messaging in the kernel."

It's there already, with reliable delivery.. modprobe tipc.

It does not pass SCM_RIGHTS or FD's, but a patchset that does this for node-local TIPC messaging will probably gain more acceptance than a new AF..

I asked on netdev if they had considered this, but i never saw a reply why they didn't choose it.

It is rather depressing that the kernel people refuse to include in-kernel support for either of the extremely widely used Linux IPC systems: binder and d-bus.

Why is a bus even needed?

Make it so that the server creates a UNIX socket with the same it wants to take, and the client connects to it.

One of those could be an enumeration/activation/etc. server (but not a message router!).

For multicast, do the same and connect to all message broadcasters, using inotify to notice when new ones come up; the publisher just sends to all connected clients.

ZeroMQ can automate most of this, if desired.

The only kernel support that might be needed is support for having unlimited size Unix socket buffers and charging that memory to the receiver, so that the OOM killer/rlimit/etc. kills a non-responsive multicast receiver rather than the sender.

A more sophisticated solution that doesn't duplicate the socket buffer for each subscriber would be even better, but probably doesn't matter for normal usage cases.

Alternatively, get rid of signals, and instead have a key/value store abstraction where you can subscribe to value updates: this way, if the buffer space is full, you can just end an "overflow" packet and the client manually asks for the values of all its watched keys.