Xiph.org's "Monty" on codecs and patents

While the talks at the 2011 GStreamer conference mostly focused on the multimedia framework itself—not surprising—there were also some that looked at the wider multimedia ecosystem. One of those was Christopher "Monty" Montgomery's presentation about Xiph.org, and its work to promote free and open source multimedia. Xiph is known for its work on the Ogg container format (and the Vorbis and Theora codecs), but the organization has worked on much more than just those. In addition, Montgomery outlined a new strategy that Xiph is trying out to combat one of the biggest problems in the free multimedia world: codec patents.

Xiph was founded in 1994, originally as a for-profit company (Xiph.com) that was set up to sell codecs. These days, it is a non-profit that consists of various "loosely grouped" codec projects. All of the members are volunteers, and various FOSS companies pay the salaries of some of the members as donations to Xiph.org. For example, Red Hat pays Montgomery's salary to allow him to work on Xiph projects. The organization is "like a coffee shop where skilled codec developers hang out", Montgomery said.

Beyond Ogg, Vorbis, and Theora, there are a number of different projects under the Xiph umbrella, Montgomery said. The cdparanoia compact disc ripper program and library was something he wrote as a student that is now part of Xiph. The Icecast streaming media server is another Xiph project, he said, as are various codecs including Speex, FLAC, the new Opus audio codec, and "a whole bunch of codecs that no one remembers".

Xiph does hold "intellectual property", Montgomery said, and that is one of the reasons it exists. Non-profits have an advantage when it comes to patents because the board gets to decide what happens to the patents if the organization goes out of business. That's different from for-profit companies that go bankrupt, he said, because whoever buys the assets gets the patents free of any promises or other entanglements (at least those that aren't legally binding, like licenses). If the original company promised not to assert some patents (e.g. for free software implementations or to implement a standard), a new owner may not be bound by that promise. A non-profit's board can ensure that any patents end up with a like-minded organization, he said.

Codec news

The biggest Xiph news in the recent past is that Google chose Vorbis as the audio codec for WebM. Montgomery said that he is very happy to see Vorbis included into WebM, but is also glad to see that Google is stepping up to help the cause of free codecs. Xiph has been trying to "hold the line on free codecs", mostly by themselves, he said. He is hopeful that Google picking up some of that will allow Xiph to "go back to what we are actually good at", which is codec development.

Xiph will be continuing to do more codec development because the members enjoy doing so, Montgomery said. Revising the Ogg container format is one thing that's on the plate now. That is not something that Xiph wanted to do while Ogg was part of its effort to hold the free codec line. With the advent of WebM, which uses the Matroska container format, some of the "legitimate complaints" about Ogg can now be addressed.

FLAC is now finished, he said. It is stable and mature with good penetration; it is essentially the standard for lossless audio codecs, and one that Apple has been unable to overturn, Montgomery said. He also noted that there were plans for a Theora 1.2 release that never happened, partly because "everyone went to work on VP8 and Opus". He believes that the release will still happen at some point, but that the pressure is off because of the existence of WebM.

Opus is a new audio codec that incorporates pieces from Xiph's CELT codec and Skype's SILK codec. Opus is designed for streaming voice or other audio over the internet, and is the subject of an IETF Internet-draft. As is usual for such documents, Intellectual Property Rights (IPR) disclosures were made by various parties who believed they had IP (e.g. patents) that are required to implement the proposed standard. Qualcomm has filed such a disclosure for Opus, but, unlike the other disclosing organizations, Qualcomm has not offered its patents under a royalty-free license.

Patent strategy

Montgomery was clear that he wasn't singling out Qualcomm in his talk, because what it has done is "business as usual" in the industry, and Qualcomm is "not in any sense alone" in making these kinds of claims. But it has led Xiph to spend almost as much time on patent strategy as it has in writing code recently. Part of the problem is that these IPR disclosures are immediately assumed to be valid by everyone, whether they know something about patents in that space or not. The presumption is that Qualcomm would never have made the claims without doing a great deal of research.

But Montgomery is not convinced that there is much of substance to Qualcomm's claims. The patent game is essentially a protection racket, he said, and those who are trying to do things royalty-free are messing things up for those who want to collect tolls. "The industry is pissed at Google because they won't play the protection racket game", he said. Qualcomm and others just list some patents that look like they could plausibly read on a royalty-free codec, because it doesn't cost them anything.

That leaves Xiph with few options, though. There is the "thermonuclear option" of going to court and getting a declaratory judgement, but there are some major downsides to pursuing that strategy. It will take a lot of time and money to do so and "no one will use it while the litigation is going on". Montgomery's original inclination was to pursue a declaratory judgement, to『bash in some teeth』and "show that Xiph.org is not to be trifled with". But even if Xiph won, it would only impact those few patents listed by Qualcomm. What is needed is a way to "change 'business as usual'", he said.

Companies "have figured out how to fight 'free'", Montgomery said, by making it illegal. In order to fight back through the courts, there would be an endless series of cases that would have to be won, and each of those wins would not hurt the companies at all. There is a "presumption of credibility" when a patent holder makes a claim of infringement, and the press "plays along with that", he said. But Eben Moglen has pointed out that an accusation of infringement has no legal weight, so there is no real downside to making such a claim.

One way to combat that is to document why the patents don't apply. Basically, Xiph did enough research to show why the Qualcomm patents don't apply to Opus and it is planning to release that information. It is a dangerous strategy at some level because it gives away some of the defense strategy, he said, but Xiph has to try something. By publishing the results of the research, Xiph will be『giving away detailed knowledge of the patents』and may be called to testify if those patents ever do get litigated, but it should counter the belief that the Qualcomm patents cover Opus.

Qualcomm could respond to the research in several different ways. It could ignore it, respond to it, or come back with more patents. It could also formally abandon the claim. If Qualcomm doesn't respond, Montgomery said, that does have some legal weight. One advantage of this approach is that regardless of how Qualcomm responds, Xiph has something concrete (i.e. the research) for the money that it has spent, which is not really the case when taking the declaratory judgement route.

New codecs

Montgomery called Opus a "best in class codec" that Xiph would like to see widely used. Hardware implementations of Opus have been considered, but have not been done yet, he said. Finishing the Opus rollout and "responding to patent claims" have been higher on the list, but they will get to it eventually.

He mentioned two other codecs that Xiph will be working on, including Ghost, which splits audio into two components: strong tones and everything else. Each of the components will be processed separately, much like what the ears do, he said. Both can be represented compactly, but the same transforms don't work on them, so representing them separately may make sense. There was a need to "invent some amount of math for all of this", he said. In addition, Xiph will be working on a new video codec that is being done as part of a『friendly rivalry with On2』(makers of the VP8 codec in WebM).

Montgomery painted a picture of an organization that is doing a great deal to further the cause of free multimedia formats. There are lots of technical and political battles to fight, but Xiph.org seems to be up to the task. It will be interesting to see how Qualcomm responds to the Opus research, and generally how the codec patent landscape plays out over the next few years. The battle is truly just beginning ...

[ I'd like to thank the Linux Foundation for helping with travel expenses so that I could attend the GStreamer conference. ]

The interplay between standards bodies and patent-miners like QTL is a increasingly complicated and hot area.

In many cases a standard depends on low barriers to adoption in order to provide value— especially for software and protocols which have no intrinsic marginal cost to hide licensing fees in— and it's actually pretty easy to find inventors who are willing to give away infrastructure technology simply because it'll enable other business. Which is also why you see groups like Mozilla or Google sponsoring the development of unencumbered web technology— in order to build the next generation of awesome stuff (commercial or otherwise) we need quality common infrastructure and we need it to be as widely available as possible.

The existence of patents themselves aren't really a barrier to this but the patent system and some of its users manage to create significant barriers as collateral damage.

In addition to the innocent patent holder who would just prefer not to give away their inventions, who's inventions must be tediously discovered and avoided, there is also a backdrop of parasites. Things like trolls buying up shaky patents and extorting people who can't afford to litigate or the risk of non-participating patent holders sneaking some trivial but essential patented technique into a standard, or following the procedings and filing applications around the groups' work, and thereby "capturing" the standard and allowing them to extract a rent from all its users. (Both are inequitable conduct which would bar their patent claims… if you can prove it and wait the two technology generations it would take to go through the courts!)

No one seems to know how this tension should be resolved, not the regulators, the standards bodies, or the patent holders. This results in some rather schizophrenic behavior from all involved as they try to act based on their interpretation of what the law is (or will be!).

Earlier this year the FTC recently put out a call for comments on the interaction of patents and standards bodies and Xiph.Org was one of the respondents and offered a number of thoughts on improving the situation.

or following the procedings and filing applications around the groups' work,