LWN.net Weekly Edition for May 2, 2013

At the 2013 Linux Foundation Collaboration Summit, GNOME Foundation Executive Director Karen Sandler spoke about bringing more women into open source projects. GNOME has produced measurable results in recent years with its Outreach Program for Women (OPW), which has now expanded to include numerous other free software projects.

Sandler started off by reminding the audience why it is important to talk about the subject of increasing participation from women in open source. If you have not experienced incidents of sexism directly, she said, you are not necessarily aware of what it can be like. She noted that when she herself started out in software development, she did not want to talk about the issue. Instead, she said, she just wanted to concentrate on doing her own work. When she went to law school, however, the percentage of female participants was considerably greater, and once she returned to the free software community, she could not help but notice the distinctly different corporate environment.

At present, 25% of all software developers are women, Sandler said, and 18% of currently graduating computer science students are women, but only 3% of free software developers are women. However, the Google Summer of Code (GSoC) program experiences a distinctly higher percentage, 8.3% in 2012, which Sandler said reflects hard work being done on the GSoC team's part. Rather than diving into the motivators and reasons for open source software's low percentages, she said, she wanted to take a practical look at how GNOME has improved its own participation statistics through OPW.

In 2006, GNOME had 181 applicants to its GSoC projects, out of which zero were women. The project realized that it had a real problem, so it launched its Outreach Program For Women, which offers paid internships working with GNOME software projects. The program was "moderately successful" that year, Sandler said, but none of the women who participated as interns stuck around after their internships ended. In 2010, the program was relaunched, with the addition of Marina Zhurakhinskaya, who currently co-leads the program with Sandler.

This incarnation of the program started with an effort to look systematically at all of the reasons why women are not participating in the project. Sandler noted that in "every discussion about women in free software we're always saying 'well, I think the reason why they're not here is because ...'" followed by any one of a handful of reasons. Rather than so assuming the answer (or answers), however, the relaunched program systematically looked at all of the possible barriers to participation, and it tries to address each of them in turn.

The program runs two rounds every year, one concurrent with GSoC, which Sandler said inspired the overall mentoring and internship process (the other round begins in January and runs through April). OPW produces advertising that specifically targets women, which Sandler said produces better results than did more generic marketing materials. In fact, OPW periodically hears from an applicant that she did not know she could apply for GSoC (or thinks that she is underqualified for GSoC), and OPW sometimes helps applicants apply to GSoC instead, especially if the code project in question makes more sense as a GSoC internship.

Successes and moving beyond GNOME

The effort has produced measurable results. Prior to the relaunched OPW, GNOME's GSoC participation garnered zero or one female student each year (or "binary numbers," Sandler quipped), out of the 20 to 30 total GNOME students. But in 2011, seven of GNOME's 27 GSoC participants were women, followed by five of its 29 participants in GSoC 2012. Similarly, 4% of GUADEC attendees in 2009 were women; at GUADEC 2012, 17% of attendees were women.

The program proved successful enough that GNOME decided it should not keep the program a GNOME-only affair. Sandler first reached out to the Software Freedom Conservancy (SFC), and SFC member project Twisted agreed to run its own internship program in conjunction with OPW's on a test basis. The test went well, so GNOME concluded it that should expand OPW to include as many projects as possible.

The marketing and branding materials were revised to highlight free software in general, and a broader OPW was launched for the January 2013 round, with ten other software projects participating—such as Fedora, Tor, JBoss, Perl, and Mozilla.

How (and why) it works

Sandler attributes the success of OPW to the systematic approach it takes. The outreach addresses women directly, which has been shown to be more effective. It also accepts non-students and non-developers, which has attracted a more diverse set of applicants, and it connects the interns with mentors, which helps participants jump in and ask for help when they otherwise might not.

One of the differences between OPW and other internship programs is that OPW requires applicants to make a contribution (code or otherwise) with their application. Zhurakhinskaya came up with the idea, Sandler said, and it has two beneficial effects. First, not everyone who applies for an internship can be accepted, but helping applicants get connected with a project during the application process allows them to get involved and make contacts. Second, the contribution requirement gives the OPW program managers a far better idea of what the applicant is like than would filling out a form alone.

During the internships, the focus is on encouraging participants to take on manageable tasks that can land during the internship period; it is sometimes tempting to want to take on a large, ambitious contribution, Sandler said, but the downside is that not being able to deliver on such a project is not a positive experience. However, mentors are encouraged to stay in close contact with the interns, and can ramp up the project if the intern is getting things done. Interns are also required to post bi-weekly blog updates on their work, and the interns' posts are aggregated on Planet GNOME, which helps raise the profile of the program.

There is a specific IRC channel for OPW participants, plus regular meetings for interns. GNOME has historically tried to provide travel assistance for GUADEC to the interns, and is now building a specific travel stipend into program. All of these mechanisms help the interns build ongoing relationships with community members, Sandler said, which is important since (as the organizers recognize) building mentoring relationships is the most important aspect of the program. OPW makes a point of encouraging mentors to continue to keep in touch with their interns after the program ends, and of encouraging former interns to stay involved by (among other things) presenting at conferences.

A large number of interns have returned as mentors, Sandler observed, which is one of the best indicators of success. But even those interns who never return to open source software development leave the program with a better understanding of software freedom, the community that produces it, and of its potential to change the world, and they become advocates for the movement.

Similarly, implementing OPW has helped GNOME in ways that extend to participation by men as well as women. The process of finding mentors identifies people who will be good mentors for anyone, the process of identifying smaller tasks for OPW applicants' initial contributions helps the project find good starting points for all newcomers, and so on.

Up next

The next round of OPW internships is slated to begin in June (concurrent with GSoC); most applications are due on May 1, although applications for the kernel internships are due May 8. But there are many other ways to help get more women involved in open source software, Sandler said, such as the Ada Initiative, which runs the AdaCamp unconference for women in open source and open culture projects. The next AdaCamp is planned for June 8 and 9, in San Francisco. AdaCamp is invitation-only, she noted, so it is important to apply as soon as possible.

Some people feel like they must choose to support one initiative or another, Sandler added, but in reality all of the different efforts fit together, and they address different aspects of participation. She encouraged everyone to help out in some way, whether by spreading the word, inviting potential participants to apply, or by persuading your company or free software project to participate as a mentor. Participating in OPW as a mentoring organization requires just two things, she said: funding to pay the stipend for at least one intern, and mentors willing to work with newcomers. Zhurakhinskaya has set a tentative goal of 12% participation from women in GSoC 2013. That may be a little ambitious, Sandler said, but only by setting lofty goals can we get there.

Comments (5 posted)

A report from the Google Test Automation Conference

This article was contributed by David Röthlisberger

The focus of 2013's Google Test Automation Conference, held April 23 and 24 in New York City, was "Testing Media and Mobile". A major theme was WebDriver, which is an API for automating web browsers. Mozilla presented its work on WebDriver support in Gecko and extensions to WebDriver to allow automated testing of FirefoxOS beyond just the Gecko-powered content layer. Google talked about WebDriver support in Chrome/Chromium, including Chrome on Android. Others demonstrated FOSS software that re-purposes the WebDriver API for testing native mobile applications on Android and iOS. WebDriver seems to be gathering a lot of momentum in the world of automated user interface (UI) testing.

WebDriver

Selenium is a commonly used tool for web browser automation that is primarily used to automate UI-level tests for web pages and rich web applications. Selenium controls the browser by sending mouse and keyboard events, and gathers results by querying the state of particular document object model (DOM) elements. So, for example, a test script for an online contact-management application might click an "add contact" button (that loads a dialog via Ajax), then check that certain DOM elements have appeared, and that they contain a particular text. You target DOM elements using XPath locators. Selenium also comes with an IDE (implemented as a Firefox plugin) that "records" test scripts by watching the mouse and keyboard.

This type of testing is not a replacement for unit testing or isolated component testing; but as Ken Kania pointed out in his talk [video], you still want some guarantees that what the user sees in the UI is correct.

Selenium initially relied on JavaScript run in the browser to simulate user interactions; but since version 2.0 (released in 2011) Selenium incorporates WebDriver. WebDriver fires events natively at the OS level, thus overcoming many limitations in the JavaScript-based approach.

WebDriver has three parts: A protocol (using JSON over HTTP and now a W3C working draft); a "local" implementation, which is a library used by the automated test scripts (implementations exist for a variety of programming languages); and a "remote" implementation, called the "browser driver" that is implemented as a browser plugin (Firefox, Safari) or as a separate executable that communicates with the browser's own debugging and inspection interfaces (Internet Explorer, Chrome, Opera). The Chrome and Opera implementations are officially supported by the respective browser vendor. There are also WebDriver implementations for mobile browsers. These are typically implemented as an app that uses a WebView object — i.e. the WebDriver app contains a browser in-process. These implementations extend the WebDriver protocol with mobile-specific APIs (finger gestures, screen orientation, etc.).

A WebDriver test script in Python might look like this (taken from a slide from Kania's talk):

    driver = webdriver.Chrome('chromedriver.exe')
    driver.get('http://www.google.com/xhtml')
    search_box = driver.find_element_by_name('q')
    search_box.send_keys('ChromeDriver')
    search_box.submit()
    self.assertTrue('ChromeDriver' in driver.title)
    driver.quit()

This script starts up a WebDriver "browser driver" targeting the Chrome browser (apparently on Windows). It then has the browser make an HTTP GET request, waits for the response (which contains an HTML form), and fills in and submits the form. Finally, the script does the actual test, which is to assert that the submitted text is present in the title of the HTML page sent as a response to the form submission. Real-world test frameworks would decouple the details of which particular browser is being tested (in this case, Chrome) from the test script itself, but this example gives the idea.

ChromeDriver

Kania, who is from the Google Chromium team, talked about Google's work on ChromeDriver (WebDriver for Chrome). ChromeDriver has been available for the desktop browser on Linux, OS X, and Windows for over a year; it is available in the Chromium source repository.

Recently ChromeDriver was re-architected into "ChromeDriver2" to support more platforms. Currently, the only addition is Android, and still at alpha-level quality; if you want support for Chrome on Chrome OS or iOS, "you have to be patient... or contribute", Kania said.

ChromeDriver is a stand-alone executable that speaks with WebDriver on one side, and communicates with the browser on the other side. Since ChromeDriver2, it uses the Chrome DevTools protocol to control the browser (see Kania's ChromeDriver architecture slide). Kania noted that Chrome is a multi-process browser, and the main browser process already uses this DevTools protocol to communicate with the renderer processes.

Michael Klepikov, from Google's "make the web faster" team, talked [video] about Google's work to make the profiling information from Chrome Developer Tools programmatically available to WebDriver scripts. This slide shows the specific API to enable this profiling. Klepikov demonstrated a test script that launches Chrome, goes to news.google.com, searches for "GTAC 2013", and switches to web (as opposed to news) results, all via the UI; it (presumably) also asserts that certain results are displayed. He demonstrated this test both on the desktop Chrome browser (video), and on Chrome running on a real Android phone (video). When the test completes, it prints the URL of a web server on localhost (powered by WebPagetest) serving an interactive display of the timing measurements, which can also be viewed within the Chrome Developer Tools interface.

Marionette: FirefoxOS meets WebDriver

Malini Das and David Burns from Mozilla presented [video] the work done on automated testing for FirefoxOS. The FirefoxOS runtime engine is based on Gecko (Firefox-the-browser's rendering engine), so the phone's user interface, as well as the bundled apps and third-party apps, are written using HTML5, JavaScript, and CSS.

The existing Firefox WebDriver implementation (a browser plugin) works for the desktop browser, but not for FirefoxOS. So Mozilla has built WebDriver support directly into Gecko, under the name Marionette. Why WebDriver? The Firefox test suite contains thousands of Selenium WebDriver-based tests, and there are thousands of web developers already familiar with WebDriver.

Marionette allows full control of the phone (including telephony functions), and not just within an individual application. Marionette scripts can even control multiple devices at once (to test a call from one device to another, say). Mozilla has written client-side libraries (what the WebDriver spec calls the "local" side: the automated test script) for JavaScript and Python.

WebDriver isn't fully specified for gesture-based input, so Mozilla has added its own extensions. You can define gestures by chaining together actions (a "pinch" could be implemented as press-move-release with one finger, plus press-hold-release on another finger). Another Mozilla extension to WebDriver is the setContext function, to switch between the "content" and the privileged "chrome" contexts. Third party application developers only need to test at the "content" level, but testing FirefoxOS itself often requires access to privileged functions.

Marionette supports (non-UI) unit tests too, and can run the hundreds of thousands of tests that already exist for Firefox. Marionette can run these tests against Firefox on the desktop, the Boot to Gecko simulator, FirefoxOS device emulators, and real FirefoxOS devices. David demonstrated some geolocation unit tests running on an emulator, though there wasn't much to see other than the emulator booting up, plus some text output on a console. Unfortunately the demonstration of an actual UI test running on a real phone encountered technical difficulties. "Let me demo how quickly the phone resets", David said to much laughter.

Of course there are security concerns with allowing a phone to be controlled remotely. For now, Marionette is only available in engineering builds; Mozilla is working on a way to deploy it for third-party developers.

Appium: automation for native apps

Jonathan Lipps from Sauce Labs started his talk [video] with a complaint: while "mobile is taking over the world", test automation for mobile apps is extremely underdeveloped.

Lipps is the lead developer of Appium, a test automation framework for native (and hybrid) mobile apps; it is released under the Apache 2.0 license. Appium consists of a server that listens for WebDriver commands, and translates them into implementation-specific commands for the particular device under test (Android or iPhone, real device or emulator). Re-using the WebDriver protocol (originally intended for in-browser web applications) means that users can write tests in any existing WebDriver-based framework.

Appium extends WebDriver with mobile-specific features by hijacking WebDriver's executeScript command (you pass in a string starting with "mobile:" instead of actual JavaScript). Presumably a more official mechanism will be worked out eventually.

Behind the scenes, Appium uses the operating system's official automation frameworks: UI Automation and Instruments on iOS; uiautomator on Android. Appium also has an optional backend for the community project Selendroid which, unlike uiautomator, works on versions of Android prior to Jelly Bean.

Lipps demonstrated [video] Appium's brand-new support for FirefoxOS by running a test script against the FirefoxOS simulator. The test launched the Contacts app, asserted that Lipps's name wasn't present in the list of contacts, clicked the "add contact" button, added him as a contact, and finally asserted that the new contact was present in the list view. Lipps had added FirefoxOS support during the conference, since Burns and Das's talk the previous day, which really shows the advantages of open standards like WebDriver.

Android uiautomator

Guang Zhu (朱光) and Adam Momtaz from Google covered [video] in some detail the Android uiautomator framework. Much like Selenium WebDriver tests, a test for a mobile application needs a way to control the mobile device, and a way to receive feedback on the state of the device; Zhu provided a survey of the approaches taken by existing Android automation frameworks (slide 1, slide 2, slide 3). uiautomator uses the InputManager system service for control, and the Accessibility service for feedback. This means it will only work if the application's widgets meet accessibility guidelines — which will hopefully encourage more developers to make their apps accessible.

Set-top box testing with GStreamer and OpenCV

I (David Röthlisberger) demonstrated [video] how YouView uses GStreamer and OpenCV for video capture and image processing to run black-box automated tests against YouView's set-top box product. Using GStreamer's gst-launch command-line utility, I showed how easy it is to capture video and insert image-processing elements into the GStreamer media pipeline. (Incidentally, the ability to create a prototype so quickly using GStreamer's command-line tools was crucial to obtaining management buy-in for this project at YouView.)

Using these technologies, YouView built stb-tester, released publicly under the LGPL v2.1+. stb-tester scripts are written in Python, and have two primitive operations available: press to send an infrared signal to the system under test (pretending to be a human user pressing buttons on a remote control), and wait_for_match to search for a given image in the system-under-test's video output.

I also showed some examples of "model-based testing" built on top of stb-tester: First [video], a script that knows how to navigate within a menu arranged as two rows of thumbnail images (aka a double carousel), so instead of saying "press UP, then LEFT, etc.", the script says "navigate to entry X". Second [video], a state machine of YouView's "setup wizard", that allows test scripts to generate random walks through the possible user choices during the setup wizard.

In spite of the name, stb-tester can be used to test any consumer electronic device, not just a set-top box, as long as it outputs video and can be controlled via infrared. See stb-tester.com for documentation and introductory material.

Who's responsible for testing?

"Your developers — or worst case, test organization — produce tests."
—Michael Klepikov [video]

Another major theme of the conference, which was more on the people and process side, was that the responsibility for testing is shifting from a separate test team to the developers themselves. In any organization following modern development practices, developers will write unit tests for their code; but end-to-end integration testing, performance testing, and security testing have traditionally been the domains of separate teams. Even Google has separate positions for "Software Engineer" and "Software Engineer in Test", and there seems to be a difference in status between those two positions — though judging from conversations with various Googlers, that difference is decreasing.

Claudio Criscione, who works on web security at Google, gave a talk [video] on an internal tool his team developed to find cross-site scripting vulnerabilities; he said they put a lot of effort into making the tool easy to use, so that development teams could run the security tests against their own applications, taking the load off the security team. Similarly, James Waldrop pointed out in his talk [video] that his performance testing team at Twitter can't possibly scale up to test all the systems being produced by the various development teams, so instead his team provides tools to make it easy for the developers to write and run performance tests themselves. Specifically, he presented Iago, which is a load generator developed at Twitter and released under the Apache 2.0 license.

Simon Stewart, who created WebDriver, leads the Selenium project, and now works at Facebook, said in his talk [video] that Facebook has no such thing as a "Software Engineer in Test", only "Software Engineers". Furthermore, Facebook has no test or QA departments. The developers are responsible for testing.

About the conference

The conference is in its 7th year (though it skipped a year in 2012). Attendance is free of charge, but by invitation and limited to around 200 technical participants; applications are announced each year on the Google Testing Blog. Live streaming was available to the public, along with a Google Moderator system for the remote audience to submit questions to the speakers.

It was a very polished conference — you can tell Google has done this before. It was held in Google's New York City office; meals were catered by Google's famous cafeteria. A sign-language interpreter was on site, and stenographers provided live transcripts of the talks. Videos and slides are available here.

Comments (5 posted)

LFCS: The value of FOSS fiscal sponsorship

This article was contributed by Martin Michlmayr

As open source becomes more popular and mature, questions of formalizing the governance and corporate structures of projects are becoming of increasing importance, as can been seen by the rising visibility of various FOSS foundations. At the Linux Foundation Collaboration Summit in San Francisco, Tony Sebro shared his insights about the value that fiscal sponsors bring as umbrella organizations for FOSS projects. Sebro is the General Counsel of Software Freedom Conservancy, which is the home of about 30 free and open source projects, including Samba, Git, and BusyBox.

Sebro kicked off his talk (slides [PDF]) by explaining some of the problems that unincorporated open source projects face. The Internet makes it easy for developers to get together and collaborate on a project. Unfortunately, there are several potential downsides in such a loose organization, especially when a project grows larger. While individuals and corporations can contribute time and code, it's difficult for unincorporated projects to raise and manage funds. Of course, many projects use PayPal, Flattr, and other means to obtain funding, but an unincorporated project has no central place to hold and manage the assets. What happens in practice is that the project founder or a similarly trusted developer will hold these assets.

Fortunately, this arrangement works out in most cases, but it can lead to severe problems in the case of a dispute—the person who registered the domain name owns it, rather than the project, and it's unclear who holds trademark rights. Sebro noted that it's possible to hold common-law trademark rights even if a trademark has not formally been registered. This is something few developers are aware of and it can lead to uncertainties as to the ownership if there is a dispute. More generally, unincorporated projects suffer from a lack of administrative support and expertise. Most developers want to write code and tend not to worry about administrative tasks.

Sebro further explained that developers working on an unincorporated project face potential risk and legal exposure. For example, if a project intends to organize a conference they may have to sign a contract with a venue. Since an unincorporated project cannot sign contracts, one of the developers would have to do so. The person who signed the contract would be on the hook if there were any serious problems, for example if the project had to cancel the conference due to lack of registrations or unforeseen events. Finally, Sebro argued that there's little insulation from corporate influence. An unincorporated project could be taken over by a company quite easily by employing the core developers or by spending significant engineering resources on the project.

Advantages of forming a legal entity

The solution for these problems, according to Sebro, is to provide a corporate home for the project. For most projects, a 501(c)(3) non-profit is the most appropriate structure, he said. 501(c)(3) refers to the US tax code that defines a charity. A 501(c)(3) charity can receive charitable donations on behalf of the project, manage and spend funds according to its mission, and enter into contracts. A non-profit can also provide administrative support, such as bookkeeping, legal services, and technical infrastructure for the project. The non-profit can act as the central place to hold project assets, so there are no uncertainties as to the ownership of trademarks, domain names, and physical assets (such as servers or t-shirts and other merchandise).

Sebro suggested that a non-profit may potentially limit the exposure to risk for individual developers. He cited the US Volunteer Protection Act, which limits the liability of volunteers acting on behalf of a public charity. Sebro noted that developers would have to act under the auspices of the charity in order to benefit from the protection and they cannot be grossly negligent. While Sebro seemed fairly confident that the Volunteer Protection Act would limit the risk for developers, "this has not been tested", he said. However, a non-profit home can certainly protect individual developers from risk associated with contracts, conferences, and similar activities.

Another advantage of a formal structure is that it insulates a project from corporate influence. A 501(c)(3) charity must work toward meeting its charitable mission and must act in the interests of the public. A charity may not tailor their activities toward one or more individuals or corporations. Furthermore, while corporations can make donations, they cannot dictate how their donations are to be used. Answering a question from the audience, Sebro explained that this is a fine line to draw. He said that a company cannot donate to a non-profit in order to get a specific feature implemented that is of primary interest to the company, as that would provide a direct benefit to the company. However, it can certainly make donations so the non-profit can add specific features on its roadmap—even if the company benefits from these features. At this point, Sebro briefly explained the difference between 501(c)(3) organizations, like Software in the Public Interest or Software Freedom Conservancy, and 501(c)(6) organizations, like the Linux Foundation and the Eclipse Foundation. While the former is a charity which has to act in the interest of the public, the latter is a trade association—companies working together to further a common goal. According to Sebro, a 501(c)(6) insulates the group from imbalanced corporate influence, as the organization has to balance the interests of competing members.

(Don't) do it yourself

Having explained the advantages of incorporating as a 501(c)(3) organization, Sebro explained different ways that projects can go about obtaining the benefits of such a structure. The one that may seem most obvious is to create your own. The process in the US would be to create a corporate and governance structure, such as officers and a board of directors, and to incorporate at the state level. The tricky part, however, is to attain 501(c)(3) status, which is required so that donors get tax deductions for their contributions. Unfortunately, new applications for 501(c)(3) undergo close scrutiny where the mission involves free and open source software, he said. Any newly formed FOSS organization will be handicapped, according to Sebro, as it will be difficult to obtain 501(c)(3) status.

A new organization would also have to answer many questions regarding its service plan: would it handle bookkeeping and legal services in-house or via a third party, and most importantly, does it have enough funding to cover that overhead? Summarizing pros and cons, Sebro said that the organization would benefit from the single brand and complete autonomy that comes from having its own non-profit. However, it comes with significant drawbacks—high overhead, a lack of clarity about who is actually going to do the work and provide the required business and legal skills, and the major uncertainty regarding attaining 501(c)(3) status.

The alternative solution is to create a relationship with an existing FOSS foundation that can act as a fiscal sponsor. It's important to note that a fiscal sponsor does not sponsor a project through monetary contributions. Its function is to provide a legal structure and environment in which projects can operate and raise funds. Many fiscal sponsors also offer a range of services to projects, typically for a fee or by taking a certain percentage from a project's donations. Sebro's talk explained two types of fiscal sponsors: comprehensive fiscal sponsors and grantor/grantee fiscal sponsors.

Comprehensive fiscal sponsors

When a project joins a comprehensive fiscal sponsor, they sign a fiscal sponsorship agreement (FSA) and become fully integrated into the sponsor—in a sense, it's like a merger and acquisition in which the fiscal sponsor takes over the open source project. The fiscal sponsor supervises the project to ensure that it follows rules of the IRS (Internal Revenue Service, the US tax agency) and is in compliance with the mission of the fiscal sponsor (which is broadly defined to encompass many projects). In the case of Software Freedom Conservancy, technical decisions and questions of infrastructure are decided by the project. The Apache Software Foundation (ASF), on the other hand, dictates parts of the development process (the Apache Way). In return for joining a comprehensive fiscal sponsor, projects will benefit from the service portfolio offered by the organization. This can include legal services, bookkeeping, conference organization, and more. Software Freedom Conservancy, for example, offers a service plan from which members can choose à la carte. Furthermore, developers are shielded from risk because they are acting as volunteers for the non-profit.

There are a number of advantages of this type of structure: overhead is shared, the fiscal sponsor has 501(c)(3) status already, and it is likely to have a lot of experience running a non-profit so it can offer reliable services and solid advice. A drawback is a certain loss of autonomy: since a fiscal sponsor is responsible for several projects, they are likely to be more conservative than a single project may be on its own. Another risk is that pooled resources imply pooled risk: if member A owes funds, would the assets of members B and C be at risk? Sebro cited the Charitable Trust Doctrine which says that donations have to be spent for the declared charitable purposes under which the donations were solicited. Spending member B's resources to pay for debts caused by member A may be in conflict with this doctrine, which could therefore provide some protection. Sebro mentioned a case that held "New York's long-standing policy honoring donors' restrictions on the use of the property they donate has greater weight than the claims of creditors." While there is some uncertainty in this area, Sebro argued that the best way to manage this problem is through good governance.

Grantor/grantee fiscal sponsors

An alternative arrangement is for projects to enter into a grantor/grantee relationship with a fiscal sponsor. This model is more lightweight: projects don't join the fiscal sponsor but remain autonomous. Sponsors can receive and manage funds on behalf of the project and provide these funds to the project in form of a grant. The sponsoring organization has a duty to ensure that the funds are spent in an appropriate way. In a sense, the fiscal sponsor acts as a bank for the project, but it can also offer a range of services to projects. Software in the Public Interest (SPI) follows this model.

This structure also allows the sharing of overhead burden and provides administrative functions to the project. While the project can stay autonomous, it won't benefit from potential risk protection at it would have if the work was done under the umbrella of a non-profit. The project and organization also have to be careful that grants are spent according to the purpose of the grant. If a fiscal sponsor takes money and gives it to a project without supervision on how it is being spent, it may run into problems with the IRS.

Advantages for developers and corporations

Sebro summarized the advantages of joining an existing non-profit foundation: it's a shortcut to benefit from 501(c)(3) status, there are economies of scale and lower administrative costs, and the administration can be outsourced to people who have experience running a non-profit organization.

In addition to these direct benefits, projects may also find it easier to obtain funding and other contributions from companies. Compared to an unincorporated entity, a 501(c)(3) fiscal sponsor provides better governance, better structure, and mentorship. In other words, it provides stability to an open source project, which is something corporations are seeking before they invest in a project. Non-profits also provide a neutral territory: a company is not able to control how the project is operating but neither can its competitors. However, corporations can still "vote" by allocating employee time: if a project has different sub-projects or tasks, it can decide which areas to focus on.

Role and sustainability of fiscal sponsors

Sebro explained that there are different roles that fiscal sponsors can play for projects. He asked whether a fiscal sponsor should be seen as a playpen, an apartment, or a nursing home. In fact, these roles are not mutually exclusive and a fiscal sponsor can play all of them. In the playpen role, the fiscal sponsor essentially provides an environment for incubation until a project reaches a certain maturity and decides to go its own way. Sebro cited former Conservancy member jQuery which decided to form its own trade association. Many projects view a fiscal sponsor as an apartment—a long term living solution. Sebro mentioned that a comprehensive fiscal sponsor could hire staff for a specific project, although Conservancy is far away from being able to do that. Finally, a nursing home, or mausoleum, can provide a home for projects that are essentially dying off. There is some value in that context as the copyright and other assets can be assigned to the fiscal sponsor. This makes it easier to the revive the project or relicense its code in the future.

Finally, Sebro raised the question of sustainability. Fiscal sponsors usually take a cut from donations to member projects in order to fund their operations (Conservancy: 10%; SPI: 5%). In the case of Conservancy, this is not sufficient to cover its operations and it relies on donations to the general fund from corporations and individuals. Sebro mentioned examples from other areas (e.g. the arts) where fiscal sponsors have achieved a sustainable model, but many FOSS fiscal sponsors still have a long way to go in that respect.

Comments (8 posted)

A goodbye note from Michael Kerrisk

This article was contributed by Michael Kerrisk.

Comments (11 posted)

Page editor: Jonathan Corbet

Security

Code authenticity checking

Cryptographically signed binary packages are mainstay of modern Linux distributions, used for verifying packages against both tampering and accidental corruption. But as ever fewer users need to compile their software from source, it is possible to forget that verifying the authenticity and integrity of binary packages depends first on the packagers' ability to verify the upstream source code. In April, Allan McRae posted an entry on his blog that looked at the verifiability of the source code releases made by a number of well-known projects, with less-than-perfect results.

McRae is a core Arch Linux developer, and he undertook his verifiability survey because Arch's packaging tool makepkg recently gained support for checking the PGP signatures of source files. Naturally, such verification is only possible when the source bundle includes a PGP signature, but McRae also looked at four other interrelated factors: whether or not the signing key can be easily verified as authentic, whether checksums for the source files are available, whether the checksums are signed, and whether the checksums are available on a different server than the one hosting the source files. This last criterion is a guard against a server compromise; if an attacker can replace the source files, he or she can also replace the checksum with one that matches the replacement source.

To provide a meaningful sample population for his survey, McRae looked at 63 "core" packages common to most Linux distributions.『For me, that basically means the packages required to build a fairly minimal booting system. This is essentially the package list from Linux From Scratch with a few additions that I see as needed …』 His results are presented in a color-coded table; green cells indicate that a package meets all of the verification criteria, red indicates that no verification criteria pass, and yellow indicate some but not all criteria are met. Of the 63 packages examined, ten offered no means of verification and 14 others offered only partial verification.

The packages offering no verification mechanism are file, flex, iana-etc, isl, kbd, libarchive, pkg-config, procps, psmisc, and which. Those packages that partially meet the measured criteria fall into two basic categories, those that provide PGP signatures but have a difficult-to-verify key (gawk, groff, patch, sed, sudo, sysvinit, texinfo, and xz), and those that provide their checksum data on the same server as their files (bzip2, perl, tzdata, and zlib). In addition, gmp and less fell somewhere in between, providing a PGP signature, but making the public key or key ID available only on the same server as the source release.

To be sure, McRae's specific criteria and red/yellow/green assessments draw some rather arbitrary lines—as he observes, several of the projects have other means of verification available, and he admits that the definition of a "readily verifiable" key includes keys signed by keys that he trusts. But the aggregate picture is the important one: most of the packages are green (which is good news), but roughly 15% of them offer no source verification whatsoever (which is far from good). He also notes that the picture seems to rapidly deteriorate "as you move further away from this core subset of packages needed for a fairly standard Linux system."

Best practices

McRae's post was picked up on the oss-security mailing list, where the talk turned to how to establish a set of common guidelines for releasing source code with verifiable authenticity. Alan Coopersmith commented that X.org has received complaints asking it to do more, but without concrete suggestions.『If there was a common standard, with instructions, we'd be far more likely to spend the time to adopt it, than just a 'make signatures appear somewhere, in an unspecified format'.』 Eric H. Christensen concurred, saying he was interested in establishing a "best practices" recommendation for Red Hat—but, he asked, what really constitutes the best way to disseminate releases? A recommendation would advise against using MD5 for checksumming, he said, and although he favors PGP for signatures, perhaps it has its drawbacks as well.

Indeed, Alistair Crooks replied with a lengthy list of questions one might ask about a PGP-signed release, addressing everything from the key management practices employed by the signing entity to the options specified when generating the key itself (such as whether it is an RSA or DSA key and whether or not it requires a passphrase). A PGP signature proves only that a person with access to the signing key attests that the signed message had a particular value at a particular time, he argued, which does not provide much authentication:

But the majority seemed to feel that PGP in fact provides quite a few gains. Nicolas Vigier and Florian Weimer both commented that key-continuity over multiple releases safeguards against a man-in-the-middle attacker replacing a release. Weimer noted that『hosting sites have been compromised, or serve their content exclusively over a mirror network which literally anyone can join.』Kurt Seifried agreed, but acknowledged that the『real problem』with PGP is the cost of implementing it:

Daniel Kahn Gillmor responded that even a simple "the release manager has the OpenPGP key for the project on the keyring of her personal development machine" workflow raises the bar for would-be attackers and would constitute an improvement for the numerous projects that do not currently sign their releases at all. But he still advocated producing guidelines:

He pointed to the existing OpenPGP best practices page at the Debian Grimoire wiki as an example.

For his part, McRae had comparatively simple goals in mind:

Identity cleft

Although a general consensus developed around the idea of crafting a "best practices" recommendation, Crooks's questions about the limitations of PGP signatures raised some valuable points—such as the importance of distinguishing between identity and trust. Some mistook his original email for a call to ditch PGP signatures on source releases, since they do not offer absolute security, but Crooks said that was a misinterpretation.『It's a bit disappointing that my advice (in pointing out ways that PGP can be worked around in order to diminish integrity and security) was categorised as an attack on PGP itself - I shall take that as a reminder that I should be more clear in what I write.』 Rather, he said, he hoped to "warn against the magic 'it's signed, so it's gospel' myth by pointing out the problems of key management."

The crux of Crooks's argument is that a PGP signature should not be trusted just because it is associated with a known identity (person or team of developers). As he elaborated in one message in the ensuing thread:

Elsewhere in the same message, he noted:

In essence, this is an argument that the signer only earns trust by demonstrating their reliability over time. For individuals, the issue might be the quality of the software (as Crooks discussed above), but to trust the signature on a project's releases, more stringent requirements may be necessary. As he added later: "You actually know very little about the key before and after the signing took place; so you have no way of ascertaining whether the key has been used to sign other things fraudulently."

Add in Crooks's earlier questions about the key options and security on the machines used to sign releases, and proper key management, it would seem, remains an area where there is still work to be done. Crooks did reiterate his support for encouraging PGP signatures on all source releases; he simply cautioned that blind trust in the presence of a signature can lead to a false sense of security.

Then again, a campaign to persuade more upstream software projects to integrate easily-verified PGP signatures and out-of-band checksums into their release process has to start somewhere. There are practical challenges to consider, such as the role that popular hosting providers play in the build and release processes (Stuart Henderson noted that Github and Bitbucket dynamically generate tar archives, which complicates signing releases).

But convincing individual developers and release managers might be the best way to convince hosting services to adapt. Presumably few people enjoy seeing their project marked with the red "No verification available" label. It would certainly be informative to conduct a large-scale examination of McRae's criteria on other popular open source projects. Just over 38% of the core packages McRae examined could not be "readily validated"—a sizable minority, but a minority nonetheless; that gives one hope that the community in general takes source verifiability as a matter worth addressing.

Comments (11 posted)

Brief items

Security quotes of the week

Comments (15 posted)

Mozilla: Protecting our brand from a global spyware provider

Comments (1 posted)

New vulnerabilities

clamav: multiple vulnerabilities

Comments (none posted)

glibc: denial of service

Comments (none posted)

kernel: multiple vulnerabilities

Comments (none posted)

libxml2: multiple unspecified vulnerabilities

Comments (none posted)

mantis: two vulnerabilities

Comments (none posted)

mediawiki: multiple vulnerabilities

Comments (none posted)

mysql: multiple vulnerabilities

Comments (none posted)

pdns-recursor: ghost domain name resolving flaw

Comments (none posted)

php-twig-Twig: file disclosure

Comments (none posted)

qemu: host file disclosure

Comments (none posted)

strongswan: authentication bypass

Comments (none posted)

Page editor: Jake Edge

Kernel development

Brief items

Kernel release status

The 3.10 merge window is open; see the separate summary below for an overview of what has been merged so far.

Stable updates: 3.8.9, 3.4.42, 3.2.44 and 3.0.75 all came out on April 25; 3.8.10 followed one day later with a fix for a build problem. 3.8.11, 3.4.43, and 3.0.76 were released on May 1.

Comments (none posted)

Quotes of the week

Comments (8 posted)

Deloget: The SoC GPU driver interview

Comments (3 posted)

Output redirection vulnerabilities in recent kernels

Full Story (comments: 12)

Three Outreach Program for Women kernel internships available

Comments (1 posted)

Kernel development news

What's coming in 3.10, part 1

User-visible changes merged at the beginning of the 3.10 cycle include:

• The "ftrace" tracing facility has seen a number of improvements. At the top of the list, arguably, is the ability to establish multiple buffers for tracing information and to direct specific events to different buffers. Additionally, it is now possible set up a trigger that will enable or disable specific events when a given kernel function is called, and it is possible to get a stack trace when a given function is called. See the improved ftrace.txt document for lots of details.

• Control groups have seen a fair amount of work in this release, some of which is implementing the improvements planned over one year ago. The device and perf_event groups now offer full hierarchical support. There is a new mount option (with the unwieldy name "__DEVEL__sane_behavior") that tries to ensure more consistent hierarchy behavior across all groups; it is obviously meant for development rather than production, but it gives some clues about what is coming.

• Applications within control groups can now request memory pressure notifications when the system is running low on available memory. See the new section 11 at the end of Documentation/cgroups/memory.txt for details on how to use this feature.

• "Return probes" are now supported in user-space probing; they allow the activation of a breakpoint when a target function returns to its caller.

• The generation of POSIX timer IDs has changed; IDs are no longer guaranteed to be unique across the system. A process's timers can now be queried by reading /proc/PID/timers. These changes make it possible for the checkpoint/restart feature to restore active timers without changing their IDs.

• POSIX and high-resolution timers support a new clock (CLOCK_TAI) which operates in international atomic time.

• The perf command (along with the perf_events subsystem) can now do memory access profiling.

• The iSCSI subsystem has gained support for the iSCSI extensions for RDMA (iSER) protocol.

• CPU frequency governors can now exist in multiple instances with different tuning parameters. This feature is needed in heterogeneous multiprocessing systems (ARM big.LITTLE, for example) to allow different CPUs to run with different parameters.

• The ability to run scripts as executables (with the interpreter specified using the "#!" sequence) can now be built as a module — or left out altogether for systems that run no scripts.

• New hardware support includes:
  • Display and graphics: Microsoft Hyper-V synthetic video devices and ILI Technology ILI9221/ILI9222 backlight controllers.

  • Hardware monitoring: Analog Devices ADT7310/ADT7320 temperature monitors, Nuvoton NCT6779D hardware monitoring chips, National Semiconductor LM95234 temperature sensors, and ST-Ericsson AB8500 thermal monitors.

  • Input: Apple infrared receivers.

  • Miscellaneous: Advantech PCI-1724U analog output cards, Analog Devices AD7923 analog to digital interfaces, Qualcomm single-wire serial bus interfaces, Broadcom BCM2835 SPI controllers, Aeroflex Gaisler GRLIB SPI controllers, NVIDIA Tegra114 SPI controllers, Silicon Labs 5351A/B/C programmable clock generators, on-chip static RAM regions, TI TPS65090 battery chargers, Samsung EXYNOS5440 CPU frequency controllers, and ARM big.LITTLE CPU frequency controllers.

  • Networking: Netlogic XLR/XLS network interfaces.

  • USB: DesignWare USB2 USB controllers.

  • Video4Linux: Rafael Micro R820T silicon tuners, ITE Tech IT913x silicon tuners, OmniVision OV7640 sensors, Philips UDA1342 audio codecs, Techwell TW9903, TW9906 and TW2804 video decoders, Silicon Laboratories 4761/64/68 AM/FM radios, Silicon Laboratories Si476x I2C FM radios, and Samsung EXYNOS4x12 FIMC-IS imaging subsystems.

  Note also that the Android "configurable composite gadget" driver has been removed from the staging tree. It is apparently difficult to maintain and no current hardware uses it.

Changes visible to kernel developers include:

• The devtmpfs filesystem now provides drivers with the ability to specify which user and group ID should own a given device. This capability is somewhat controversial — there is resistance to encoding user and group ID policy in the kernel — but it will be useful for systems like Android.

• The staging tree has gained a new "sync" driver (from Android) that can be used for synchronization between other drivers.

• There is a new "dummy-irq" driver that does nothing other than register an interrupt handler. It exists to debug IRQ sharing problems by forcing the enabling of a specific interrupt line.

• A lot of the low-level USB PHY access functions have been changed to GPL-only exports.

• The new idr_alloc_cyclic() function allocates ID numbers in a cyclic fashion: when the given range is exhausted, allocations will start again at the beginning of that range.

• The workqueue subsystem has seen some substantial reworking which, among other things, should make it perform better on NUMA systems. There is also a new sysfs interface that can be used to tweak some workqueue parameters.

If the usual pattern holds, this merge window should remain open until around May 12 and the 3.10 kernel can be expected in early July. As usual, LWN will follow the mainline as the merge window progresses.

Comments (none posted)

Wait/wound mutexes

A conceptual overview

Situations where multiple locks must be held simultaneously pose a risk of deadlocks: if the order in which those locks are acquired is not always the same, there will eventually come a time when two threads find themselves blocked, each waiting for the other to release a lock. Kernel code tends to be careful about lock ordering, and the "lockdep" checking tool has gotten quite good about finding code that violates the rules. So deadlocks are quite rare, despite the huge number of locks used by the kernel.

But what about situations where the ordering of lock acquisition cannot be specified in advance, or, even worse, is controlled by user space? Maarten's patch describes one such scenario: a chain of buffers used with the system's graphical processing unit (GPU). These buffers must, at various times, be "owned" by the GPU itself, the GPU driver, user space, and, possibly, another driver completely, such as for a video frame grabber. User space can submit the buffers for processing in an arbitrary order, and the GPU may complete them in a different order. If locking is used to control the ownership of the buffers, and if multiple buffers must be manipulated at once, avoiding deadlocks could become difficult.

Imagine a simple situation where there are two buffers of interest:

Imagine further that we have two threads (we'll call them T1 and T2) that attempt to lock both buffers in the opposite order: T1 starts with Buffer A, while T2 starts with Buffer B. As long as they do not both try to grab the buffers at the same time, things will work. But, someday, each will succeed in locking one buffer and a situation like this will develop:

The kernel's existing locking primitives have no answer to a situation like this other than "don't do that." The wait/wound mutex, instead, is designed for just this case. In general terms, what will happen in this situation is:

• The thread that "got there first" will simply sleep until the remaining buffer becomes available. If T1 started the process of locking the buffers first, it will be the thread that waits.

• The other thread will be "wounded," meaning that it will be told it must release any locks it holds and start over from scratch.

So if T2 is wounded, the deadlock will be resolved by telling T2 to release Buffer B; it must then wait until that buffer becomes available again and start over. So the situation will look something like this:

Once T1 has released the buffers, T2 will be able to retry and, presumably, make forward progress on its task.

The details

The first step toward using a set of locks within the wait/wound mechanism is to define a "class"; this class is essentially a context within which the locks are to be acquired. When multiple threads contend for the same locks, they must do so using the same class. A wait/wound class is defined with:

    #include <linux/mutex.h>

    static DEFINE_WW_CLASS(my_class);

As far as users of the system are concerned, the class needs to exist, but it is otherwise opaque; there is no explicit initialization required. Internally, the main purpose for the class's existence is to hold a sequence number (an atomic counter) used to answer the "who got there first" question; it also contains some information used by lockdep to verify correct use of wait/wound locks.

The acquisition of a specific set of locks must be done within a "context" that tracks the specific locks held. Before acquiring the first lock, a call should be made to:

    void ww_acquire_init(struct ww_acquire_ctx *ctx, struct ww_class *ww_class);

This call will assign a sequence number to the context and do a bit of record keeping. Once that has been done, it is possible to start acquiring locks:

    int ww_mutex_lock(struct ww_mutex *lock, struct ww_acquire_ctx *ctx);

If the lock has been successfully acquired, the return value will be zero. When all goes well, the thread will manage to acquire all of the locks it needs. Once that process is complete, that fact should be signaled with:

    void ww_acquire_done(struct ww_acquire_ctx *ctx);

This function is actually a no-op in the current implementation, but that could change in the future. After this call, the processing of the locked data can proceed normally. Once the job is done, it is time to release the locks and clean up:

    void ww_mutex_unlock(struct ww_mutex *lock);
    void ww_acquire_fini(struct ww_acquire_ctx *ctx);

Each held lock should be released with ww_mutex_unlock(); once all locks have been released, the context should be cleaned up with ww_acquire_fini().

The above description describes what happens when all goes well, but it has left out an important case that all wait/wound mutex users must handle: the detection of a potential deadlock. That case comes about whenever an attempt is made to lock a ww_mutex that is already locked; in this case, there are three possible outcomes.

The first of these comes about if the locking thread already holds that ww_mutex and is attempting to lock it for a second time. With ordinary mutexes, this would be an error, but the wait/wound mechanism is designed for this case. Evidently, sometimes, the ordering of the locking is so poorly defined that multiple locking attempts can happen. In such cases, ww_mutex_lock() will return -EALREADY. The locking thread, assuming it knows how to respond to -EALREADY, can continue about its business.

The second possibility is that the sequence number in the context for the locking process is higher than the number associated with thread already holding the lock. In this case, the new caller gets "wounded"; ww_mutex_lock() will return -EDEADLK to signal that fact. The wounded thread is expected to clean up and get out of the way. "Cleaning up" means releasing all locks held under the relevant context with calls to ww_mutex_unlock(). Once all of the locks are free, the wounded thread can try again, but only when the contended lock is released by the victorious thread; waiting for that to happen is done with:

    void ww_mutex_lock_slow(struct ww_mutex *lock, struct ww_acquire_ctx *ctx);

This function will block the calling thread until lock becomes free; once it returns, the thread can try again to acquire all of the other locks it needs. It is entirely possible that this thread could, once again, fail to acquire all of the needed locks. But, since the sequence number increases monotonically, a once-wounded thread must eventually reach a point where it has the highest priority and will win out.

The final case comes about when the new thread's sequence number is lower than that of the thread currently holding the lock. In this case, the new thread will simply block in ww_mutex_lock() until the lock is freed. If the thread holding the contended lock attempts to acquire another lock that is already held by the new thread, it will get the -EDEADLK status at that point; it will then release the contended lock and let the new thread proceed. Going back to the example above:

Thread T1, holding the lower sequence number, will wait for Buffer B to be unlocked, while thread T2 will see -EDEADLK when it attempts to lock Buffer A.

The documentation in the patch does not describe what happens if the holding process never calls ww_mutex_lock() again. In this case, it will never know that it is supposed to back off. But, in this case, the holder must necessarily already have acquired all of the locks it needs, so there should be no reason why it cannot simply finish its work and release the locks normally. So the end result will be the same.

Conclusion

Needless to say, there are plenty of details that have not been covered here; see the ww-mutex-design.txt document included with the patch set for more information.

In that document, there are code examples for three different ways of working with wait/wound mutexes. One need not read for long to conclude that the API looks a bit complex and tricky to use; it will be far harder to write correct locking code using this facility than it would be with normal mutexes. Perhaps that complexity is necessary, and it seems certain that this mechanism will not be needed in many places in the kernel, so the complexity should not spread too far. But an API like this can be expected to raise some eyebrows.

What is missing at this point is any real code that uses wait/wound mutexes. Kernel developers will certainly want to see some examples of where this kind of locking mechanism is needed. After all, the kernel has made it through its first two decades without this kind of complex locking; convincing the community that this feature is now necessary is going to take a strong sales effort. That is best done by showing how wait/wound mutexes solve a problem that cannot be easily addressed otherwise. Until that is done, wait/wound mutexes are likely to remain an interesting bit of code on the sidelines.

Comments (14 posted)

LSFMM Summit coverage complete

Since the initial posting, the following sessions have been added:

Combined Filesystem/Storage sessions

• dm-cache and bcache: the future of two storage-caching technologies for Linux.

• Error returns: filesystems could use better error information from the storage layer.

• Storage management: how do we ease the task of creating and managing filesystems on Linux systems?

• O_DIRECT: the kernel's direct I/O code is complicated, fragile, and hard to change. Is it time to start over?

• Reducing io_submit() latency: submitting asynchronous I/O operations can potentially block for long periods of time, which is not what callers want. Various ways of addressing this problem were discussed, but easy solutions are not readily at hand.

Filesystem-only sessions

• NFS status: what is going on in the NFS subsystem.

• Btrfs status: what has happened with the next-generation Linux filesystem, and when will it be ready for production use?

• User-space filesystem servers: what can the kernel do to support user-space servers like Samba and NFS-GANESHA?

• Range locking: a proposal to lock portions of files within the kernel.

• FedFS: where things stand with the creation of a Federated Filesystem implementation for Linux.

Storage-only sessions

• Reducing SCSI latency. The SCSI stack is having a hard time keeping up with the fastest drives; what can be done to speed things up?

• SCSI testing. It would be nice to have a test suite for SCSI devices; after this session, one may well be in the works.

• Error handling and firmware updates: some current problems with handling failing drives, and how can we do online firmware updates on SATA devices?

Thanks are due to those who helped with the creation of these writeups. We would like to thank Elena Zannoni in particular for providing comprehensive notes from the Storage track.

Comments (none posted)

Patches and updates

Kernel trees

• Linus Torvalds: Linux 3.9 released . (April 29, 2013)

• Alexandre Oliva: GNU Linux-libre 3.9-gnu . (April 30, 2013)

• Greg KH: Linux 3.8.11 . (May 1, 2013)

• Greg KH: Linux 3.8.10 . (April 26, 2013)

• Sebastian Andrzej Siewior: 3.8.10-rt6 . (April 30, 2013)

• Greg KH: Linux 3.8.9 . (April 26, 2013)

• Sebastian Andrzej Siewior: 3.8.9-rt4 . (April 28, 2013)

• Greg KH: Linux 3.4.43 . (May 1, 2013)

• Greg KH: Linux 3.4.42 . (April 26, 2013)

• Steven Rostedt: 3.4.42-rt56 . (May 1, 2013)

• Steven Rostedt: 3.4.41-rt55-feat3 . (April 30, 2013)

• Ben Hutchings: Linux 3.2.44 . (April 26, 2013)

• Steven Rostedt: 3.2.44-rt64 . (May 1, 2013)

• Steven Rostedt: 3.2.43-rt63-feat2 . (April 30, 2013)

• Greg KH: Linux 3.0.76 . (May 1, 2013)

• Greg KH: Linux 3.0.75 . (April 26, 2013)

• Steven Rostedt: 3.0.75-rt102 . (May 1, 2013)

Build system

• Yann E. MORIN: kconfig-frontends-3.9.0.0 released . (April 30, 2013)

Core kernel code

• Vincent Guittot: sched: packing small tasks . (April 26, 2013)

• Alex Shi: sched: use runnable load based balance . (April 28, 2013)

• Thomas Gleixner: clocksource/events: Overhaul (un)registration . (April 26, 2013)

• Maarten Lankhorst: Wait/wound mutex implementation, v3 . (April 29, 2013)

Development tools

• Alexandru Copot: selftests: Basic framework for tests . (April 25, 2013)

• Sasha Levin: liblockdep: userspace lockdep . (April 30, 2013)

Device drivers

• Hayes Wang: net/usb: new driver for RTL8152 . (April 26, 2013)

• Amit Daniel Kachhap: thermal: exynos: Add thermal driver for exynos5440 . (April 26, 2013)

• Qiaowei Ren: TXT driver . (April 28, 2013)

• Jon Arne Jørgensen: Add a driver for Somagic smi2021 . (April 29, 2013)

• Tony Prisk: Add support for velocity network driver on platform devices . (April 29, 2013)

• Kishon Vijay Abraham I: Generic PHY Framework . (April 29, 2013)

• Rafael J. Wysocki: Driver core / ACPI: Add offline/online for graceful hot-removal of devices . (April 29, 2013)

• Zoran Markovic: drivers: power: Add watchdog timer to catch drivers which lockup during suspend. . (May 1, 2013)

Filesystems and block I/O

• Pavel Shilovsky: Add O_DENY* support for VFS and CIFS/NFS . (April 26, 2013)

• Philippe De Muyter: Add aix lvm partitions support . (April 29, 2013)

Memory management

• Minchan Kim: Per process reclaim . (April 25, 2013)

• Glauber Costa: kmemcg shrinkers . (April 29, 2013)

• Mel Gorman: Obey mark_page_accessed hint given by filesystems . (April 29, 2013)

• Pavel Emelyanov: mm: Ability to monitor task memory changes (v4) . (April 30, 2013)

Architecture-specific

• Xi Wang: seccomp filter JIT . (April 26, 2013)

• Steve Capper: HugeTLB and THP support for ARM64. . (April 30, 2013)

• Aneesh Kumar K.V: THP support for PPC64 (Patchset 1) . (April 29, 2013)

Security-related

• Kees Cook: kernel ASLR . (April 26, 2013)

• H. Peter Anvin: random: Account for entropy loss due to overwrites . (April 28, 2013)

• Steve Dickson: lnfs: 3.9-rc8 release . (April 29, 2013)

• Jari Ruusu: Announce loop-AES-v3.6h file/swap crypto package . (April 29, 2013)

Miscellaneous

• Karel Zak: util-linux 2.23 . (April 25, 2013)

• Stephen Hemminger: iproute2 3.9.0 . (April 30, 2013)

• Pavel Emelyanov: Checkpoint-restore tool v0.5 . (April 30, 2013)

Page editor: Jonathan Corbet

Distributions

x32 ABI support by distributions

This article was contributed by Andrew Shewmaker

The x32 ABI uses the same registers and floating point hardware as the x86-64 ABI, but with 32-bit pointers and long variables as on ia32 (32-bit x86). On the down side, fully supporting x32 means having a third set of libraries. However, its assembly code is more efficient than ia32. Additionally, if an application uses many pointers and requires less than 4GB of RAM, then x32 can be a big win over x86-64. It can theoretically double the number of pointers held in cache, reducing misses, and halve the amount of system memory used to store pointers, reducing page faults.

Benchmark results

The x32 project site lists two SPEC benchmarks that show particularly good benefits. MCF, a network simplex algorithm making heavy use of pointers, shows x32 with a 40% advantage over x86-64. Crafty, a chess program made up predominantly of integer code, shows x32 with a 40% advantage over ia32. In H. Peter Anvin's 2011 Linux Plumbers Conference talk x32 — a native 32-bit ABI for x86-64 [PPT], he lists embedded devices as one of the primary use cases. He reports 5-10% improvement for the SPEC CPU 2K/2006 INT geomean over ia32 and x86-64, and 5-11% advantage for SPEC CPU 2K/2006 FP geomean over ia32, but none against x86-64. The MCF results have been called out as the most impressive, but the majority of users may care more about video codec performance. The news is good there too. H.264 performed 35% better on x32 than on ia32 and 15-20% better than on x86-64.

Intel gives additional SPEC and Embedded Microprocessor Benchmark Consortium (EEMBC) performance numbers in a paper describing the work done to support x32 in GCC and their own compiler. Parser, a syntactic parser of English, shows an approximately 17% improvement for x32 over both ia32 and x86-64. And Eon, a probabilistic ray tracer, performs almost 30% better than ia32. In general, GCC's floating-point code generation isn't as good on x32 as on x86-64. However, Intel's compiler does at least as well for x32 as x86-64. Take Facerec, a facial recognition algorithm which compares graph structures with vertices containing information computed from fast Fourier transforms, where x32 performs almost 40% better than x86-64.

Scientific simulations are often limited by memory latency and capacity. Nathalie Rauschmayr has presented results showing how x32 benefits some of CERN's applications [PDF]. These applications use millions of pointers and, when compiled as x32, see a reduction in physical memory use of 10-30%. While some ran no faster, a couple sped up by 10-15%. In those cases, the underlying cause for the improvement is a 10-38% reduction in page faults. Loop unrolling is an important optimization for scientific code and x32 could use some work in that area, but H.J. Lu, the driving force behind x32, doesn't plan to do that work himself.

Distribution support

As Nathalie notes, a working x32 environment doesn't come out of the box in a Red Hat– or Debian–based Linux distribution just yet. Currently, a person must build glibc with an intermediate GCC build before building a full GCC. Fortunately, distributions are beginning to provide an easier way to experiment with x32. Two of the first projects to feature a release with official x32 support are Gentoo and the Yocto meta-distribution, and a couple of traditional distributions are following to one degree or another.

Gentoo's first x32 release candidate came out last June; support is now integrated into the main tree. You can download and install it like any other port. When I installed it, I ran into a problem creating an initramfs due to a BusyBox linking error. Configuring the kernel with everything built-in (eliminating the need for an initramfs altogether) worked around this problem, resulting in a working x32 environment. The remaining x32 issues are largely in multimedia packages' hand-optimized assembly routines. As soon as Gentoo's infrastructure team upgrades the kernel on a development box, Mike Frysinger plans on announcing the availability of an x32 environment for upstream multimedia projects to port their optimized assembly to the x32 ABI.

I have not tried the Linux Foundation's Yocto project. It is used for creating custom embedded distributions and has integrated x32 support into its core, as well as fixing most of its recipes.

x32 won't be an official ABI for Debian until after the next release, Wheezy. Daniel Schepler, who oversees the new port, told me that it is in reasonable shape. Grub2 doesn't currently build, so an x32-only install is not possible yet. But Daniel provides directions to install into a chroot environment from his private archive on the Debian x32 Port wiki. Newer packages are available from Debian Ports. Simpler desktop environments like XFCE are installable, and KDE is almost ready. However, more work needs to be done before GNOME is installable. Its biggest blocker is that Iceweasel currently doesn't build. Also, LibreOffice and PHP are not ready yet.

My experience with Debian went well. After a minimal installation from the stable repository and upgrading to Wheezy, I installed one of Daniel's kernels and used the debootstrap tool to quickly create an x32 chroot from Daniel's personal archive. After that, I was able to point to the most recent packages on the Debian Ports archive. Installing a chroot directly from those involves learning how to use the multistrap tool, and I stopped short of that.

Ubuntu shipped with limited x32 support in 13.04. However, it isn't planning on providing a full port of x32 packages like Debian. Ubuntu is simply providing a kernel with the x32 syscalls and a functional tri-arch toolchain with GCC and eglibc.

x32 hasn't found as much acceptance in other open-source projects. Fedora has no plans for x32, which means it's unlikely for RHEL too. Arch Linux has some unofficial packages. LLVM users can apply patches from June 2012. But merging of x32 support looks unlikely at this point. A team from Google inquired about it earlier this year, and they were told that the patch submitter had not addressed the maintainers' concerns yet.

Some experiments

After reading Diego Elio "Flameeyes" Pettenò's series of posts on debunking x32 myths, I decided someone ought to compare x32 and x86-64 performance of the B-heap simulator mentioned in Poul-Henning Kamp's ACM article "You're Doing It Wrong." This example is interesting because Kamp created a data structure called a "B-heap" to take advantage of the large address space provided by 64-bit pointers. That would seem to argue against using x32's smaller address space, but in his article he also says, "The order of magnitude of difference obviously originates with the number of levels of heap inside each VM page, so the ultimate speedup will be on machines with small pointer sizes and big page sizes."

Kamp's B-heap simulator compares the virtual memory performance of a traditional binary heap to his B-heap. A port from BSD to Linux went smoothly, and the results are identical. No changes were needed to compile the simulator in my Debian x32 chroot. Its virtual memory footprint was reduced by 19% and its resident set size was 88% smaller. Performance of x32 always exceeded x86-64, especially for larger problem sizes. In particular, 500K ran 15% faster. It's important to note that the simulator itself doesn't use much memory, so it would be interesting to see how a real-world use of a B-heap, as in the Varnish Cache, would compare.

It will be interesting to see which other applications will benefit from x32. If it shows as much promise for the embedded space as its creators suggest, high performance computing as CERN shows, and web servers as my experiment indicates, then desktop users in the middle of the computing spectrum might also benefit. The key to that segment will likely be whether multimedia codecs (other than the H.264 reference implementation tested by SPEC CPU 2006) see a substantial enough improvement to justify developing and maintaining another assembly code implementation of their core routines. I predict they will. Perhaps x32 will prove its value in such a broad number of cases that it becomes the default for many software packages. Perhaps we'll even see developers for other 64-bit architectures such as IBM's Power or upcoming ARM server chips propose similar ABIs.

Comments (32 posted)

Brief items

Distribution quote of the week

Comments (5 posted)

DragonFly BSD 3.4 released

Comments (none posted)

GNU Linux-libre 3.9-gnu

Full Story (comments: none)

OpenBSD 5.3 released

Full Story (comments: 8)

Ubuntu 13.04 (Raring Ringtail) released

Full Story (comments: 22)

Distribution News

Debian GNU/Linux

Debian Installer 7.0 RC2 release

Full Story (comments: none)

Bits from the 6th Debian Groupware Meeting

Full Story (comments: none)

Ubuntu family

Next Ubuntu now open for development

Full Story (comments: none)

Newsletters and articles of interest

Distribution newsletters

• Debian Project News (April 29)
• DistroWatch Weekly, Issue 505 (April 29)
• Maemo Weekly News (April 29)
• SprezzOS Weekly News (April 25)
• Ubuntu Weekly Newsletter, Issue 314 (April 28)

Comments (none posted)

PCLinuxOS "So Cool Ice Cubes are Jealous" (Linux.com)

Comments (none posted)

Shuttleworth: The Supercalifragilisticexpialidocious Scorpionfish. Not.

Comments (35 posted)

Ward: So long Fuduntu, hello FuSE

Comments (none posted)

Page editor: Rebecca Sobol

Development

Go and Rust — objects without class

This article was contributed by Neil Brown

Since the advent of object-oriented programming languages around the time of Smalltalk in the 1970s, inheritance has been a mainstay of the object-oriented vision. It is therefore a little surprising that both "Go" and "Rust" — two relatively new languages which support object-oriented programming — manage to avoid mentioning it. Both the Rust Reference Manual and The Go Programming Language Specification contain the word "inherit" precisely once and the word "inheritance" not at all. Methods are quite heavily discussed, but inheritance is barely more than a "by the way".

This may be just an economy of expression, or it may be an indication of a sea change in attitudes towards object orientation within the programming language community. It is this second possibility which this article will consider while exploring and contrasting the type systems of these two languages.

The many faces of inheritance

While inheritance is a core concept in object-oriented programming, it is not necessarily a well-defined concept. It always involves one thing getting some features by association with some previously defined things, but beyond that languages differ. The thing is typically a "class", but sometimes an "interface" or even (in prototype inheritance) an "object" that borrows some behavior and state from some other "prototypical" object.

The features gained are usually fields (for storing values) and methods (for acting on those values), but the extent to which the inheriting thing can modify, replace, or extend these features is quite variable.

Inheriting from a single ancestor is common. Inheriting from multiple ancestors is sometimes possible, but is an even less well-defined concept than single inheritance. Whether multiple inheritance really means anything useful, how it should be implemented, and how to approach the so-called diamond problem all lead to substantial divergence among approaches to inheritance.

If we clear away these various peripheral details (important though they are), inheritance boils down to two, or possibly three, core concepts. It is the blurring of these concepts that is created by using one word ("inheritance"), which, it would seem, results in the wide variance among languages. And it is this blurring that is completely absent from Go and Rust.

Data embedding

The possible third core concept provided by inheritance is data embedding. This mechanism allows a data structure to be defined that includes a previously defined data structure in the same memory allocation. This is trivially achieved in C as seen in:

    struct kobject {
 char  *name;
 struct list_head entry;
 ...
    };

where a struct list_head is embedded in a struct kobject. It can sometimes be a little more convenient if the members of the embedded structure (next and prev in this case) can be accessed in the embedding object directly rather than being qualified as, in this case entry.next and entry.prev. This is possible in C11 and later using "anonymous structures".

While this is trivial in C, it is not possible in this form in a number of object-oriented languages, particularly languages that style themselves as "pure" object oriented. In such languages, another structure (or object) can only be included by reference, not directly (i.e. a pointer can be included in the new structure, but the old structure itself cannot).

Where structure embedding is not possible directly, it can often be achieved by inheritance, as the fields in the parent class (or classes) are directly available in objects of the child class. While structure embedding may not be strong motivation to use inheritance, it is certainly an outcome that can be achieved through using it, so it does qualify (for some languages at least) as one of the faces of inheritance.

Subtype polymorphism

Subtype polymorphism is a core concept that is almost synonymous with object inheritance. Polymorphic code is code that will work equally well with values from a range of different types. For subtype polymorphism, the values' types must be subtypes of some specified super-type. One of the best examples of this, which should be familiar to many, is the hierarchy of widgets provided by various graphical user interface libraries such as GTK+ or Qt.

At the top of this hierarchy for GTK+ is the GtkWidget which has several subtypes including GtkContainer and GtkEditable. The leaves of the hierarchy are the widgets that can be displayed, such as GtkEntry and GtkRadioButton.

GtkContainer is an ancestor of all widgets that can serve to group other widgets together in some way, so GtkHBox and GtkVBox — which present a list of widgets in a horizontal or vertical arrangement — are two subtypes of GtkContainer. Subtype polymorphism allows code that is written to handle a GtkContainer to work equally well with the subtypes GtkHBox and GtkVBox.

Subtype polymorphism can be very powerful and expressive, but is not without its problems. One of the classic examples that appears in the literature involves "Point and ColorPoint" and exactly how the latter can be made a subtype of the former — which intuitively seems obvious, but practically raises various issues.

A real-world example of a problem with polymorphism can be seen with the GtkMenuShell widget in the GTK+ widget set. This widget is used to create drop-down and pop-up menus. It does this in concert with GtkMenuItem which is a separate widget that displays a single item in a menu. GtkMenuShell is declared as a subtype of GtkContainer so that it can contain a collection of different GtkMenuItems, and can make use of the methods provided by GtkContainer to manage this collection.

The difficulty arises because GtkMenuShellisonly allowed to contain GtkMenuItem widgets, no other sort of child widget is permitted. So, while it is permitted to add a GtkButton widget to a GtkContainer, it is not permitted to add that same widget to a GtkMenuShell.

If this restriction were to be encoded in the type system, GtkMenuShell would not be a true subtype of GtkContainer as it cannot be used in every place that a GtkContainer could be used — specifically it cannot be the target of gtk_container_add(myButton).

The simple solution to this is to not encode the restriction into the type system. If the programmer tries to add a GtkButton to a GtkMenuShell, that is caught as a run-time error rather than a compile-time error. To the pragmatist, this is a simple and effective solution. To the purist, it seems to defeat the whole reason we have static typing in the first place.

This example seems to give the flavor of subtype polymorphism quite nicely. It can be express a lot of type relationships well, but there are plenty of relationships it cannot express properly; cases where you need to fall back on run-time type checking. As such, it can be a reason to praise inheritance, and a reason to despise it.

Code reuse

The remaining core concept in inheritance is code reuse. When one class inherits from another, it not only gets to include fields from that class and to appear to be a subtype of that class, but also gets access to the implementation of that class and can usually modify it in interesting ways.

Code reuse is, of course, quite possible without inheritance, as we had libraries long before we had objects. Doing it with inheritance seems to add an extra dimension. This comes from the fact that when some code in the parent class calls a particular method on the object, that method might have been replaced in the child object. This provides more control over the behavior of the code being reused, and so can make code reuse more powerful. A similar thing can be achieved in a C-like language by explicitly passing function pointers to library functions as is done with qsort(). That might feel a bit clumsy, though, which would discourage frequent use.

This code reuse may seem as though it is just the flip-side of subtype inheritance, which was, after all, motivated by the value of using code from an ancestor to help implement a new class. In many cases, there is a real synergy between the two, but it is not universal. The classic examination of this issue is a paper by William R. Cook that examines the actual uses of inheritance in the Smalltalk-80 class library. He found that the actual subtype hierarchy (referred to in the paper as protocol conformance) is quite different from the inheritance hierarchy. For this code base at least, subtypes and code reuse are quite different things.

As different languages have experimented with different perspectives on object-oriented programming, different attitudes to these two or three different faces have resulted in widely different implementations of inheritance. Possibly the place that shows this most clearly is multiple inheritance. When considering subtypes, multiple inheritance makes perfect sense as it is easy to understand how one object can have two orthogonal sets of behaviors which make it suitable to be a member of two super-types. When considering implementation inheritance for code reuse, multiple inheritance doesn't make as much sense because the different ancestral implementations have more room to trip over each other. It is probably for this reason that languages like Java only allow a single ancestor for regular inheritance, but allow inheritance of multiple "interfaces" which provide subtyping without code reuse.

In general, having some confusion over the purpose of inheritance can easily result in confusion over the use of inheritance in the mind of the programmer. This confusion can appear in different ways, but perhaps the most obvious is in the choice between "is-a" relationships and "has-a" relationships that is easy to find being discussed on the Internet. "is-a" reflects subtyping, "has-a" can provide code reuse. Which is really appropriate is not always obvious, particularly if the language uses the same syntax for both.

Is inheritance spent?

Having these three very different concepts all built into the one concept of "inheritance" can hardly fail to result in people developing very different understandings. It can equally be expected to result in people trying to find a way out of the mess. That is just what we see in Go and Rust.

While there are important differences, there are substantial similarities between the type systems of the two languages. Both have the expected scalars (integers, floating point numbers, characters, booleans) in various sizes where appropriate. Both have structures and arrays and pointers and slices (which are controlled pointers into arrays). Both have functions, closures, and methods.

But, importantly, neither have classes. With inheritance largely gone, the primary tool for inheritance — the class — had to go as well. The namespace control provided by classes is left up to "package" (in Go) or "module" (in Rust). The data declarations are left up to structures. The use of classes to store a collection of methods has partly been handed over to "interfaces" (Go) or "traits" (Rust), and partly been discarded.

In Go, a method can be defined anywhere that a function can be defined — there is simply an extra bit of syntax to indicate what type the method belongs to — the "receiver" of the method. So:

    func (p *Point) Length() float64 {
 return math.Sqrt(p.x * p.x + p.y * p.y)
    }

is a method that applies to a Point, while:

    func Length(p *Point) float64 {
 return math.Sqrt(p.x * p.x + p.y * p.y)
    }

would be a function that has the same result. These compile to identical code and when called as "p.Length()" and "Length(&p)" respectively, identical code is generated at the call sites.

Rust has a somewhat different syntax with much the same effect:

    impl Point {
 fn Length(&self) -> float {
     sqrt(self.x * self.x + self.y * self.y)
 }
    }

A single impl section can define multiple methods, but it is perfectly legal for a single type to have multiple impl sections. So while an impl may look a bit like a class, it isn't really.

The "receiver" type on which the method operates does not need to be a structure — it can be any type though it does need to have a name. You could even define methods for int were it not for rules about method definitions being in the same package (or crate) as the definition of the receiver type.

So in both languages, methods have managed to escape from existing only in classes and can exist on their own. Every type can simply have some arbitrary collection of methods associated with it. There are times though when it is useful to collect methods together into groups. For this, Go provides "interfaces" and Rust provides "traits".

    type file interface {
 Read(b Buffer) bool
 Write(b Buffer) bool
 Close()
    }

    trait file {
 fn Read(&self, b: &Buffer) -> bool;
 fn Write(&self, b: &Buffer) -> bool;
 fn Close(&self);
    }

These two constructs are extremely similar and are the closest either language gets to "classes". They are however completely "virtual". They (mostly) don't contain any implementation or any fields for storing data. They are just sets of method signatures. Other concrete types can conform to an interface or a trait, and functions or methods can declare parameters in terms of the interface or traits they must conform to.

Traits and interfaces can be defined with reference to other traits or interfaces, but it is a simple union of the various sets of methods.

    type seekable interface {
 file
 Seek(offset u64) u64
    }



    trait seekable : file {
 fn Seek(&self, offset: u64) -> u64;
    }

No overriding of parameter or return types is permitted.

Both languages allow pointers to be declared with interface or trait types. These can point to any value of any type that conforms to the given interface or trait. This is where the real practical difference between the Length() function and the Length() method defined earlier becomes apparent. Having the method allows a Point to be assigned to a pointer with the interface type:

    type measurable interface {
        Length() float64
    }

Exploring the new inheritance

Here we see the brave new world of inheritance. It is nothing more or less than simply sharing a collection of method signatures. It provides simple subtyping and doesn't even provide suggestions of code reuse or structure embedding. Multiple inheritance is perfectly possible and has a simple well-defined meaning. The diamond problem has disappeared because implementations are not inherited. Each method needs to be explicitly implemented for each concrete type so the question of conflicts between multiple inheritance paths simply does not arise.

This requirement to explicitly implement every method for every concrete type may seem a little burdensome. Whether it is in practice is hard to determine without writing a substantial amount of code — an activity that current time constraints don't allow. It certainly appears that the developers of both languages don't find it too burdensome, though each has introduced little shortcuts to reduce the burden somewhat.

The "mostly" caveat above refers to the shortcut that Rust provides. Rust traits can contain a "default" implementation for each method. As there are no data fields to work with, such a default cannot really do anything useful and can only return a constant, or call other methods in the trait. It is largely a syntactic shortcut, without providing any really inheritance-like functionality. An example from the Numeric Traits bikeshedis

    trait Eq {
        fn eq(&self, other: &Self) -> bool { return !self.ne(other) };
        fn ne(&self, other: &Self) -> bool { return !self.eq(other) };
    }

In this example it is clear that the defaults by themselves do not provide a useful implementation. The real implementation is expected to define at least one of these methods to something meaningful for the final type. The other could then usefully remain as a default. This is very different from traditional method inheritance, and is really just a convenience to save some typing.

In Go, structures can have anonymous members much like those in C11 described earlier. The methods attached to those embedded members are available on the embedding structure as delegates: if a method is not defined on a structure it will be delegated to an anonymous member value which does define the method, providing such a value can be chosen uniquely.

While this looks a bit more like implementation inheritance, it is still quite different and much simpler. The delegated method can only access the value it is defined for and can only call the methods of that value. If it calls methods which have been redefined for the embedding object, it still gets the method in the embedded value. Thus the "extra dimension" of code reuse mentioned earlier is not present.

Once again, this is little more than a syntactic convenience — undoubtedly useful but not one that adds new functionality.

Besides these little differences in interface declarations, there are a couple of significant differences in the two type systems. One is that Rust supports parameterized types while Go does not. This is probably the larger of the differences and would have a pervasive effect on the sort of code that programmers write. However, it is only tangentially related to the idea of inheritance and so does not fit well in the present discussion.

The other difference may seem trivial by comparison — Rust provides a discriminated union type while Go does not. When understood fully, this shows an important difference in attitudes towards inheritance exposed by the different languages.

A discriminated union is much like a C "union" combined with an enum variable — the discriminant. The particular value of the enum determines which of the fields in the union is in effect at a particular time. In Rust this type is called an enum:

    enum Shape {
        Circle(Point, float),
        Rectangle(Point, Point)
    }

So a "Shape" is either a Circle with a point and a length (center and radius) or a Rectangle with two points (top left and bottom right). Rust provides a match statement to access whichever value is currently in effect:

    match myshape {
 Circle(center, radius) => io::println("Nice circle!");
 Rectangle(tl, br) => io::println("What a boring rectangle");
    }

Go relies on interfaces to provide similar functionality. A variable of interface type can point to any value with an appropriate set of methods. If the types to go in the union have no methods in common, the empty interface is suitable:

    type void interface {
    }

Avoid variable can now point to a circle or a rectangle.

    type Circle struct {
 center Point
 radius float
    }
    type Rectangle struct {
 top_left, bottom_right Point
    }

Of course it can equally well point to any other value too.

The value stored in a void pointer can only be accessed following a "type assertion". This can take several forms. A nicely illustrative one for comparison with Rust is the type switch.

    switch s := myshape.(type) {
    case Circle:
 printString("Nice circle!")
    case Rectangle:
 printString("What a boring rectangle")
    }

While Rust can equally create variables of empty traits and can assign a wide variety of pointers to such variables, it cannot copy Go's approach to extracting the actual value. There is no Rust equivalent of the "type assertion" used in Go. This means that the approaches to discriminated union in Rust and Go are disjoint — Go has nothing like "enum" and Rust has nothing like a "type assertion".

While a lot could be said about the comparative wisdom and utility of these different choices (and, in fact, much has been said) there is one particular aspect which relates to the topic of this article. It is that Go uses inheritance to provide discriminated unions, while Rust provides explicit support.

Are we moving forward?

The history of programming languages in recent years seems to suggest that blurring multiple concepts into "inheritance" is confusing and probably a mistake. The approach to objects and methods taken by both Rust and Go seem to suggest an acknowledgment of this and a preference for separate, simple, well-defined concepts. It is then a little surprising that Go chooses to still blend two separate concepts — unions and subtyping — into one mechanism: interfaces.

This analysis only provides a philosophical objection to that blend and as such it won't and shouldn't carry much weight. The important test is whether any practical complications or confusions arise. For that we'll just have to wait and see.

One thing that is clear though is that the story of the development of the object-oriented programming paradigm is a story that has not yet been played out — there are many moves yet to make. Both Rust and Go add some new and interesting ideas which, like languages before them, will initially attract programmers, but will ultimately earn both languages their share of derision, just as there are plenty of detractors for C++ and Java today. They nonetheless serve to advance the art and we can look forward to the new ideas that will grow from the lessons learned today.

Comments (31 posted)

Brief items

Quotes of the Week

Comments (5 posted)

GDB 7.6 released

Full Story (comments: none)

Wayland development plans posted

Full Story (comments: 1)

Open Build Service version 2.4 released

Comments (10 posted)

Newsletters and articles

Development newsletters from the past week

• Caml Weekly News (April 30)
• What's cooking in git.git (April 26)
• What's cooking in git.git (April 29)
• Openstack Community Weekly Newsletter (April 25)
• Perl Weekly (April 29)
• PostgreSQL Weekly News (April 28)
• Ruby Weekly (April 25)

Comments (none posted)

Development Update, Schedule, and Funds (OpenShot blog)

Comments (20 posted)

Packard: Shared Memory Fences

Keith Packard has posted another update on the ongoing work to revise the Direct Rendering Infrastructure in X. This post covers the GPU serialization mechanism implemented in SyncFences. "SyncFences are the same as Events as provided by Python and other systems. Of course all of the names have been changed to keep things interesting. I’ll call them Fences here, to be consistent with the current X usage."

Comments (none posted)

Mozilla: Push notifications for the Open Web

Mozilla has unveiled Simple Push, a new web API for "push" notifications. A blog post explaining the API notes several important design choices, such as the fact that developers do not need to pre-register their applications in order to receive notifications, and that users do not need to log in to a third-party service in order to receive notifications. "Web applications are freed from repeatedly polling for updates, leading to a better experience for everybody. Users get better battery life and more responsive applications. Developers don’t have to re-implement polling logic in every application they write. Mobile devices benefit from intelligent scheduling of notifications to reduce network usage and further improve battery life."

Comments (none posted)

Page editor: Nathan Willis

Announcements

Brief items

The SFC aims to create better non-profit accounting software

Comments (8 posted)

FSF certifies ThinkPenguin USB Wifi adapter

Full Story (comments: none)

Upcoming Events

Workshop on Essential Abstractions in GCC, 2013

Full Story (comments: none)

Events: May 2, 2013 to July 1, 2013

If your event does not appear here, please tell us about it.

Page editor: Rebecca Sobol