Testing Blog
Tuesday, January 25, 2011
This is the first in a series of posts on this topic.
The one question I get more than any other is "How does Google test?" It's been explained in bits and pieces on this blog but the explanation is due an update. The Google testing strategy has never changed but the tactical ways we execute it has evolved as the company has evolved. We're now a search, apps, ads, mobile, operating system, and so on and so forth company. Each of these Focus Areas (as we call them) have to do things that make sense for their problem domain. As we add new FAs and grow the existing ones, our testing has to expand and improve. What I am documenting in this series of posts is a combination of what we are doing today and the direction we are trending toward in the foreseeable future.
Let's begin with organizational structure and it's one that might surprise you. There isn't an actual testing organization at Google. Test exists within a Focus Area called Engineering Productivity. Eng Prod owns any number of horizontal and vertical engineering disciplines, Test is the biggest. In a nutshell, Eng Prod is made of:
1. A product team that produces internal and open source productivity tools that are consumed by all walks of engineers across the company. We build and maintain code analyzers, IDEs, test case management systems, automated testing tools, build systems, source control systems, code review schedulers, bug databases... The idea is to make the tools that make engineers more productive. Tools are a very large part of the strategic goal of prevention over detection.
2. A services team that provides expertise to Google product teams on a wide array of topics including tools, documentation, testing, release management, training and so forth. Our expertise covers reliability, security, internationalization, etc., as well as product-specific functional issues that Google product teams might face. Every other FA has access to Eng Prod expertise.
3. Embedded engineers that are effectively loaned out to Google product teams on an as-needed basis. Some of these engineers might sit with the same product teams for years, others cycle through teams wherever they are needed most. Google encourages all its engineers to change product teams often to stay fresh, engaged and objective. Testers are no different but the cadence of changing teams is left to the individual. I have testers on Chrome that have been there for several years and others who join for 18 months and cycle off. Keeping a healthy balance between product knowledge and fresh eyes is something a test manager has to pay close attention to.
So this means that testers report to Eng Prod managers but identify themselves with a product team, like Search, Gmail or Chrome. Organizationally they are part of both teams. They sit with the product teams, participate in their planning, go to lunch with them, share in ship bonuses and get treated like full members of the team. The benefit of the separate reporting structure is that it provides a forum for testers to share information. Good testing ideas migrate easily within Eng Prod giving all testers, no matter their product ties, access to the best technology within the company.
This separation of project and reporting structures has its challenges. By far the biggest is that testers are an external resource. Product teams can't place too big a bet on them and must keep their quality house in order. Yes, that's right: at Google it's the product teams that own quality, not testers. Every developer is expected to do their own testing. The job of the tester is to make sure they have the automation infrastructure and enabling processes that support this self reliance. Testers enable developers to test.
What I like about this strategy is that it puts developers and testers on equal footing. It makes us true partners in quality and puts the biggest quality burden where it belongs: on the developers who are responsible for getting the product right. Another side effect is that it allows us a many-to-one dev-to-test ratio. Developers outnumber testers. The better they are at testing the more they outnumber us. Product teams should be proud of a high ratio!
Ok, now we're all friends here right? You see the hole in this strategy I am sure. It's big enough to drive a bug through. Developers can't test! Well, who am I to deny that? No amount of corporate kool-aid could get me to deny it, especially coming off my GTAC talk last year where I pretty much made a game of developer vs. tester (spoiler alert: the tester wins).
Google's answer is to split the role. We solve this problem by having two types of testing roles at Google to solve two very different testing problems. In my next post, I'll talk about these roles and how we split the testing problem into two parts.
Google
Labels:
James Whittaker
12 comments
:
LaurentJanuary 26, 2011 at 1:15:00 AM PST
Thanks for the interesting post.
2 questions about : "So this means that testers report to Eng Prod managers but identify themselves with a product team"
1) does this also apply to developpers ?
2) Where are Eng Prod Managers sitting ? Meaning : are they embeded in projects also or they stay outsite ?
Thanks,
Laurent
Replies
AnonymousJanuary 26, 2011 at 5:33:00 AM PST
Surreal!
Replies
Nick DreweJanuary 26, 2011 at 5:41:00 AM PST
Thanks for the insight. I can fully understand how testing is a very tricky area for a behemoth as large as Google, especially with the diversity of products you have.
Our company, Market Dojo, finds testing challenging enough, even though we have just the one piece of software!
Replies
AnonymousJanuary 27, 2011 at 2:20:00 AM PST
Hi James,
Enjoy reading these posts. Keen to hear more about how Google tests, particularly, what methods/processes are used in estimations.
With dev's doing the lions share of the testing, how are estimates gathered for testing? Is there a particular process used for estimation (or one you could recommend)?
Thanks.
Replies
Darren McMillanJanuary 27, 2011 at 2:30:00 AM PST
Hi James,
This setup is very similar to how our company works and like at Google it works very well for us.
One line though I'd have probably have been more careful about is this:
"The better they are at testing the more they outnumber us. Product teams should be proud of a high ratio!"
I think you'll find you'll have upset a lot of your testers at Google with that.
Cem Kaner wrote a very good paper around metrics, I’d say this falls into an unwritten management metric which is probably best left unspoken.
http://www.kaner.com/pdfs/measurement_segue.pdf
Cheers,
Darren
Replies
Manoj NarayananJanuary 27, 2011 at 7:11:00 AM PST
Would like to add few comments.
1. Looks like there is a de-facto matrix structure for the testers here with dual reporting to product manager and core Eng Prod Mgr. Works well if the tester also has a long term career option linked to the Eng Prod team - basically some one to take care of the career, skill set development etc.
2. There could also be problems if the product line suffers budget constraints - who insists on minimum quality then? Is there a minimum RMI that each product team has to sign up for before release?
3. As we split test engineering and testing roles - eagerly looking forward to this explanation in the next post - it might be worthwhile to look at engaging testing service vendors as a lower cost option for the testing portion. Might be a good bridge between dog fooding and crowd testing, especially if you can throw in few SLA adherence requirements.
Replies
UnknownJanuary 27, 2011 at 8:29:00 AM PST
Hi James,
Very interesting post. Looking forward to reading the rest of the series.
It would be interesting to see your Test Case and defect management systems and see how they stack up against ones that are commonly used throughout the industry.
I bet they're slicker and sexier than the majority of the ones available on the market!
Regards,
Adam Brown
http://www.gobanana.co.uk
Replies
zecarreraJanuary 28, 2011 at 6:17:00 AM PST
Great post, already anxious for the others...
Replies
MikeJanuary 28, 2011 at 6:19:00 AM PST
Really interested to know whether Google promotes TDD/BDD or whether they use a test-after approach. If TDD/BDD, how is it enforced/maintained?
Replies
Kiran BadiFebruary 9, 2011 at 11:25:00 PM PST
Exactly the reason I have always felt that IT organisation structure are disfunctional in nature.Google is pretty much the same structure.
So basically governance lies with product teams and testing belongs to service lines.Huge pie of success if it comes by goes to product teams and small pie comes to service lines whereas in real sense the real execution of those products is done by service lines.Just because product team conceives a great product idea does not assure you a success, it needs that greats ideas needs to have strong implementation strategies and to do this we need to ensure that we are making everyone accountable for their role and share the burden equally.
I do not have right solution now but problem statement is pretty much clear, you have operational issue to deal with in your plate.
Your structure is one of the reasons why I feel google lacks quality now a days.
Hope you are not reading my comments in wrong spirit,sometimes I often research on the ways organisations are structured.
Replies
UnknownMarch 7, 2012 at 7:29:00 AM PST
Thanks for the interesting post
100 points
Replies
UnknownMay 22, 2017 at 5:29:00 PM PDT
Hello James,
I'm reading the book "How Google Tests Software" and I'm going to share the information in this book with my friends in a workshop.
So could you please confirm if the information in that book (like roles, testing process, testing philosophy) is still correct and is till using at Google now?
Thanks in advance,
Long Lee
Replies
Load more...
The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.
Labels
●
TotT
99
●
GTAC
61
●
James Whittaker
42
●
Misko Hevery
32
●
Code Health
28
●
Anthony Vallone
27
●
Patrick Copeland
23
●
Jobs
18
●
Andrew Trenk
12
●
C++
11
●
Patrik Höglund
8
●
JavaScript
7
●
Allen Hutchison
6
●
George Pirocanac
6
●
Zhanyong Wan
6
●
Harry Robinson
5
●
Java
5
●
Julian Harty
5
●
Alberto Savoia
4
●
Ben Yu
4
●
Erik Kuefler
4
●
Philip Zembrod
4
●
Shyam Seshadri
4
●
Adam Bender
3
●
Chrome
3
●
Dillon Bly
3
●
John Thomas
3
●
Lesley Katzen
3
●
Marc Kaplan
3
●
Markus Clermont
3
●
Max Kanat-Alexander
3
●
Sonal Shah
3
●
APIs
2
●
Abhishek Arya
2
●
Alan Myrvold
2
●
Alek Icev
2
●
Android
2
●
April Fools
2
●
Chaitali Narla
2
●
Chris Lewis
2
●
Chrome OS
2
●
Diego Salas
2
●
Dori Reuveni
2
●
Jason Arbon
2
●
Jochen Wuttke
2
●
Kostya Serebryany
2
●
Marc Eaddy
2
●
Marko Ivanković
2
●
Mobile
2
●
Oliver Chang
2
●
Simon Stewart
2
●
Stefan Kennedy
2
●
Test Flakiness
2
●
Titus Winters
2
●
Tony Voellm
2
●
WebRTC
2
●
Yiming Sun
2
●
Yvette Nameth
2
●
Zuri Kemp
2
●
Aaron Jacobs
1
●
Adam Porter
1
●
Adam Raider
1
●
Adel Saoud
1
●
Alan Faulkner
1
●
Alex Eagle
1
●
Anantha Keesara
1
●
Antoine Picard
1
●
App Engine
1
●
Ari Shamash
1
●
Arif Sukoco
1
●
Benjamin Pick
1
●
Bob Nystrom
1
●
Bruce Leban
1
●
Carlos Arguelles
1
●
Carlos Israel Ortiz García
1
●
Cathal Weakliam
1
●
Christopher Semturs
1
●
Clay Murphy
1
●
Dagang Wei
1
●
Dan Maksimovich
1
●
Dan Shi
1
●
Dan Willemsen
1
●
Dave Chen
1
●
Dave Gladfelter
1
●
David Mandelberg
1
●
Derek Snyder
1
●
Diego Cavalcanti
1
●
Dmitry Vyukov
1
●
Eduardo Bravo Ortiz
1
●
Ekaterina Kamenskaya
1
●
Elliott Karpilovsky
1
●
Elliotte Rusty Harold
1
●
Espresso
1
●
Felipe Sodré
1
●
Francois Aube
1
●
Gene Volovich
1
●
Google+
1
●
Goran Petrovic
1
●
Goranka Bjedov
1
●
Hank Duan
1
●
Havard Rast Blok
1
●
Hongfei Ding
1
●
Jason Elbaum
1
●
Jason Huggins
1
●
Jay Han
1
●
Jeff Hoy
1
●
Jeff Listfield
1
●
Jessica Tomechak
1
●
Jim Reardon
1
●
Joe Allan Muharsky
1
●
Joel Hynoski
1
●
John Micco
1
●
John Penix
1
●
Jonathan Rockway
1
●
Jonathan Velasquez
1
●
Josh Armour
1
●
Julie Ralph
1
●
Kai Kent
1
●
Karin Lundberg
1
●
Kaue Silveira
1
●
Kevin Bourrillion
1
●
Kevin Graney
1
●
Kirkland
1
●
Kurt Alfred Kluever
1
●
Manjusha Parvathaneni
1
●
Marek Kiszkis
1
●
Marius Latinis
1
●
Mark Ivey
1
●
Mark Manley
1
●
Mark Striebeck
1
●
Matt Lowrie
1
●
Meredith Whittaker
1
●
Michael Bachman
1
●
Michael Klepikov
1
●
Mike Aizatsky
1
●
Mike Wacker
1
●
Mona El Mahdy
1
●
Noel Yap
1
●
Palak Bansal
1
●
Patricia Legaspi
1
●
Per Jacobsson
1
●
Peter Arrenbrecht
1
●
Peter Spragins
1
●
Phil Norman
1
●
Phil Rollet
1
●
Pooja Gupta
1
●
Project Showcase
1
●
Radoslav Vasilev
1
●
Rajat Dewan
1
●
Rajat Jain
1
●
Rich Martin
1
●
Richard Bustamante
1
●
Roshan Sembacuttiaratchy
1
●
Ruslan Khamitov
1
●
Sam Lee
1
●
Sean Jordan
1
●
Sharon Zhou
1
●
Shiva Garg
1
●
Siddartha Janga
1
●
Simran Basi
1
●
Stan Chan
1
●
Stephen Ng
1
●
Tejas Shah
1
●
Test Analytics
1
●
Test Engineer
1
●
Tim Lyakhovetskiy
1
●
Tom O'Neill
1
●
Vojta Jína
1
●
automation
1
●
dead code
1
●
iOS
1
●
mutation testing
1
Archive
●
►
2024
(9)
●
►
Jul
(1)
●
►
May
(3)
●
►
Apr
(3)
●
►
Mar
(1)
●
►
Feb
(1)
●
►
2023
(14)
●
►
Dec
(2)
●
►
Nov
(2)
●
►
Oct
(5)
●
►
Sep
(3)
●
►
Aug
(1)
●
►
Apr
(1)
●
►
2022
(2)
●
►
Feb
(2)
●
►
2021
(3)
●
►
Jun
(1)
●
►
Apr
(1)
●
►
Mar
(1)
●
►
2020
(8)
●
►
Dec
(2)
●
►
Nov
(1)
●
►
Oct
(1)
●
►
Aug
(2)
●
►
Jul
(1)
●
►
May
(1)
●
►
2019
(4)
●
►
Dec
(1)
●
►
Nov
(1)
●
►
Jul
(1)
●
►
Jan
(1)
●
►
2018
(7)
●
►
Nov
(1)
●
►
Sep
(1)
●
►
Jul
(1)
●
►
Jun
(2)
●
►
May
(1)
●
►
Feb
(1)
●
►
2017
(17)
●
►
Dec
(1)
●
►
Nov
(1)
●
►
Oct
(1)
●
►
Sep
(1)
●
►
Aug
(1)
●
►
Jul
(2)
●
►
Jun
(2)
●
►
May
(3)
●
►
Apr
(2)
●
►
Feb
(1)
●
►
Jan
(2)
●
►
2016
(15)
●
►
Dec
(1)
●
►
Nov
(2)
●
►
Oct
(1)
●
►
Sep
(2)
●
►
Aug
(1)
●
►
Jun
(2)
●
►
May
(3)
●
►
Apr
(1)
●
►
Mar
(1)
●
►
Feb
(1)
●
►
2015
(14)
●
►
Dec
(1)
●
►
Nov
(1)
●
►
Oct
(2)
●
►
Aug
(1)
●
►
Jun
(1)
●
►
May
(2)
●
►
Apr
(2)
●
►
Mar
(1)
●
►
Feb
(1)
●
►
Jan
(2)
●
►
2014
(24)
●
►
Dec
(2)
●
►
Nov
(1)
●
►
Oct
(2)
●
►
Sep
(2)
●
►
Aug
(2)
●
►
Jul
(3)
●
►
Jun
(3)
●
►
May
(2)
●
►
Apr
(2)
●
►
Mar
(2)
●
►
Feb
(1)
●
►
Jan
(2)
●
►
2013
(16)
●
►
Dec
(1)
●
►
Nov
(1)
●
►
Oct
(1)
●
►
Aug
(2)
●
►
Jul
(1)
●
►
Jun
(2)
●
►
May
(2)
●
►
Apr
(2)
●
►
Mar
(2)
●
►
Jan
(2)
●
►
2012
(11)
●
►
Dec
(1)
●
►
Nov
(2)
●
►
Oct
(3)
●
►
Sep
(1)
●
►
Aug
(4)
●
▼
2011
(39)
●
►
Nov
(2)
●
►
Oct
(5)
●
►
Sep
(2)
●
►
Aug
(4)
●
►
Jul
(2)
●
►
Jun
(5)
●
►
May
(4)
●
►
Apr
(3)
●
►
Mar
(4)
●
►
Feb
(5)
●
▼
Jan
(3)
●
How Google Tests Software - Part One
●
Google Innovation and The Pretotyping Manifesto
●
New Year's Resolutions
●
►
2010
(37)
●
►
Dec
(3)
●
►
Nov
(3)
●
►
Oct
(4)
●
►
Sep
(8)
●
►
Aug
(3)
●
►
Jul
(3)
●
►
Jun
(2)
●
►
May
(2)
●
►
Apr
(3)
●
►
Mar
(3)
●
►
Feb
(2)
●
►
Jan
(1)
●
►
2009
(54)
●
►
Dec
(3)
●
►
Nov
(2)
●
►
Oct
(3)
●
►
Sep
(5)
●
►
Aug
(4)
●
►
Jul
(15)
●
►
Jun
(8)
●
►
May
(3)
●
►
Apr
(2)
●
►
Feb
(5)
●
►
Jan
(4)
●
►
2008
(75)
●
►
Dec
(6)
●
►
Nov
(8)
●
►
Oct
(9)
●
►
Sep
(8)
●
►
Aug
(9)
●
►
Jul
(9)
●
►
Jun
(6)
●
►
May
(6)
●
►
Apr
(4)
●
►
Mar
(4)
●
►
Feb
(4)
●
►
Jan
(2)
●
►
2007
(41)
●
►
Oct
(6)
●
►
Sep
(5)
●
►
Aug
(3)
●
►
Jul
(2)
●
►
Jun
(2)
●
►
May
(2)
●
►
Apr
(7)
●
►
Mar
(5)
●
►
Feb
(5)
●
►
Jan
(4)
Feed
●
Google
●
Privacy
●
Terms