LCA: Linux gaming at last

At the beginning of his talk on Linux game development at linux.conf.au 2013, Eric Anholt noted that his original motivation for getting into graphics driver development was that he wanted to be able to play games on alternative operating systems. At that time, there were a few games available, but the graphics drivers were in such a poor state that he ended up working on drivers instead of playing games. Thus, he has now been working in Intel's graphics driver team for seven years; currently, he works mainly on Mesa's OpenGL graphics drivers.

Eric's talk took the form of a short review of some recent history in Linux game development followed by a description of his experiences in the Intel open source graphics driver team working with Valve Software to port the Steam game platform to Linux.

Recent history of game development on Linux

Eric started with a summary of significant changes that have taken place in the graphics stack over the last seven years. These changes include kernel mode setting and improvements to memory management so that multiple processes can reliably employ memory on the graphics card. On the OpenGL side, things have improved considerably. "Back when I started, we were about ten years behind". The then-current OpenGL version on Linux was 2.1, no modern games ran on Linux, and there was no OpenGLES on Linux. However, by now, Linux supports OpenGL 3.1 and has reached Khronos-certified OpenGLES 2.0 conformance, and Open GLES 3.0 certification seems to be quite close.

Development of open source games seems to have lagged. Eric suggested a number of reasons for this. One of these is that creating a game requires building a multi-talented team composed of artists, developers, and designers. It's difficult to put a team like that together. And then, even if one does manage to assemble such a team, it's hard to agree on a direction: when it comes to game design, the design space is so large that it can be difficult to agree on what you are creating. Finally, the move into open source game development means that you spend less time doing the thing you want to do: playing games.

Nevertheless, there have been a few open source games, such as etracer (Extreme Tux Racer), Neverball, Xonotic, Foobillard, and Wesnoth. In addition, there were closed source games such as Quake (later open sourced), Unreal Tournament 2004, Loki, Minecraft, and whatever users could force to run with Wine.

In May 2010, the Humble Indie Bundle appeared. The concept was a package of games made available DRM-free, with the user choosing the price that they would pay. "They've actually released some surprisingly good games for Linux." One of those games was Braid, and Eric noted that the developers who participated in Humble Bundle learned an important lesson from an earlier attempt to port that game to Linux.

The developer of Braid, Jonathan Blow, made a blog post asking for help on how to port Braid to Linux, asking questions such as『how do I deal with mouse grabs, so that mouse clicks only go to the game window?』and "how do I output sound?" The community did try to help: in all, the blog post got 251 responses, many of them containing directly conflicting advice. In response, the developer gave up: he couldn't justify spending the time to determine the correct way to do the port for what was a small target market.

The lesson that the Humble Bundle developers learned from the Braid experience was that game developers should not be burdened with the task of porting games. So instead, they employed Ryan C. Gordon, a developer who had already ported a number of games to Linux, to port all of their games. This approach has been surprisingly successful at quickly getting games to run on Linux.

Working with Valve on the Linux port of Steam

There have been petitions for Valve Software to port Steam to Linux for as long as Steam has been around, Eric said. The Intel graphics driver team started working with Valve Software in July 2012. During the porting project, the Intel team had access to the Steam source code and worked with Valve on both tuning Steam to work better with the Mesa graphics library and tuning Mesa to work better with Steam. The closed beta test for the port was started in November 2012, and the open beta started in December. The port included the Steam's Source engine and the game Left For Dead 2.

The cooperative work between the graphics developers and Valve proved to be quite productive, but in the process, the graphics developers learned about a number of things that Valve found really disappointing. The first of these disappointments concerned ARB_debug_output. This is an extension in the Windows development environment for OpenGL that provides a general event logging infrastructure between the graphics layer on one side and middleware and applications on the other side. This is an important debugging tool in the Windows environment, "where you don't really have the console so much". There is an implementation of ARB_debug_output in Mesa, but it is rudimentary, supporting only two event types.

Another major disappointment concerned bad debugging and performance-measurement tools. The Valve developers described the tools they used for debugging on Windows, and wanted to know what equivalents existed on Linux. In response, the graphics developers were excited to show Valve an API-tracing tool that they had developed. API tracing is a technique that is extremely useful for driver developers because it allows them to obtain a reproducible trace of the graphics operations requested by the application. This means driver developers can get the application out of the way while they replay the trace in order to debug problems in the driver or improve the driver's performance. However, this sort of low-level tool provides little assistance for analyzing performance at the application level.

By contrast, Windows has some good tools in this area that allow the application programmer to insert tracepoints to track application steps such "starting to draw the world", "starting to draw a figure", and "starting to draw a helmet". This enables the application developer to obtain a hierarchical view of performance and determine if (say) drawing helmets takes a long time. Linux has to do a lot to catch up in this area.

The Valve developers also complained that Mesa was simply too slow. Many of the code paths in Mesa are not well optimized, with enormous switch statements containing nested conditionals. One possible solution is to offload some of the graphics work onto a separate thread running on a different core. Some work has been done in this area, but, so far, performance is no better than for the existing single-threaded approach. More work is required.

Notwithstanding the disappointments, there were other aspects of working on Linux that the Valve developers loved. For example, they greatly appreciated the short development cycles that were made possible when using open source drivers. Although the support for ARB_debug_output was poor on Linux, the Valve developers were impressed when Eric was able to quickly implement some instrumentation of driver hotspots, so that the driver would log messages when the application asked it to carry out operations that were known to perform poorly. The Valve developers were also surprised that the logging could be controlled by environment variables, so that the same driver would be used in both quiet and "logging" mode.

A final pleasant surprise for the Valve developers was that drivers could be debugged using breakpoints. That possibility is unavailable with closed source Windows drivers. More generally, the Valve developers don't have much insight into the closed source drivers that they use on Windows (or, for that matter, closed source drivers on Linux). Thus, they have to resort to experimentation to form mental models about performance problems and how to get around them.

Concluding remarks

For gaming enthusiasts, the announcement by Valve—one of the largest producers of game software—that Steam would be ported to Linux was something of a watershed moment. The Linux gaming landscape is poised to grow much bigger, and even those of us who are not gamers can reflect that a much-improved games ecosystem will at the very least widen interest in Linux as a platform for desktop computer systems.

Another major disappointment concerned bad debugging and performance-measurement tools.