Testing Blog
Wednesday, February 09, 2011
In order for the “you build it, you break it” motto to be real, there are roles beyond the traditional developer that are necessary. Specifically, engineering roles that enable developers to do testing efficiently and effectively have to exist. At Google we have created roles in which some engineers are responsible for making others more productive. These engineers often identify themselves as testers but their actual mission is one of productivity. They exist to make developers more productive and quality is a large part of that productivity. Here's a summary of those roles:
The SWEorSoftware Engineer is the traditional developer role. SWEs write functional code that ships to users. They create design documentation, design data structures and overall architecture and spend the vast majority of their time writing and reviewing code. SWEs write a lot of test code including test driven design, unit tests and, as we explain in future posts, participate in the construction of small, medium and large tests. SWEs own quality for everything they touch whether they wrote it, fixed it or modified it.
The SETorSoftware Engineer in Test is also a developer role except their focus is on testability. They review designs and look closely at code quality and risk. They refactor code to make it more testable. SETs write unit testing frameworks and automation. They are a partner in the SWE code base but are more concerned with increasing quality and test coverage than adding new features or increasing performance.
The TEorTest Engineer is the exact reverse of the SET. It is a a role that puts testing first and development second. Many Google TEs spend a good deal of their time writing code in the form of automation scripts and code that drives usage scenarios and even mimics a user. They also organize the testing work of SWEs and SETs, interpret test results and drive test execution, particular in the late stages of a project as the push toward release intensifies. TEs are product experts, quality advisers and analyzers of risk.
From a quality standpoint, SWEs own features and the quality of those features in isolation. They are responsible for fault tolerant designs, failure recovery, TDD, unit tests and in working with the SET to write tests that exercise the code for their feature.
SETs are developers that provide testing features. A framework that can isolate newly developed code by simulating its dependencies with stubs, mocks and fakes and submit queues for managing code check-ins. In other words, SETs write code that allows SWEs to test their features. Much of the actual testing is performed by the SWEs, SETs are there to ensure that features are testable and that the SWEs are actively involved in writing test cases.
Clearly SETs primary focus is on the developer. Individual feature quality is the target and enabling developers to easily test the code they write is the primary focus of the SET. This development focus leaves one large hole which I am sure is already evident to the reader: what about the user?
User focused testing is the job of the Google TE. Assuming that the SWEs and SETs performed module and feature level testing adequately, the next task is to understand how well this collection of executable code and data works together to satisfy the needs of the user. TEs act as a double-check on the diligence of the developers. Any obvious bugs are an indication that early cycle developer testing was inadequate or sloppy. When such bugs are rare, TEs can turn to their primary task of ensuring that the software runs common user scenarios, is performant and secure, is internationalized and so forth. TEs perform a lot of testing and test coordination tasks among TEs, contract testers, crowd sourced testers, dog fooders, beta users, early adopters. They communicate among all parties the risks inherent in the basic design, feature complexity and failure avoidance methods. Once TEs get engaged, there is no end to their mission.
Ok, now that the roles are better understood, I'll dig into more details on how we choreograph the work items among them. Until next time...thanks for your interest.
Google
Labels:
James Whittaker
21 comments
:
UnknownFebruary 10, 2011 at 2:52:00 AM PST
Interesting. I had never heard of Test Engineer being used as a job title before. Does the addition of the word engineer really represent what the job entails? Or do you think it is a reaction to the typically pejorative title 'tester' or 'QA'.
Note: I write test automation code for a living, and have always wondered what a suitable job title should be.
Replies
Eudes CostaFebruary 10, 2011 at 3:31:00 AM PST
good set of posts, thanks for sharing with the community
Replies
UnknownFebruary 10, 2011 at 11:43:00 AM PST
love you james
Replies
kengccFebruary 10, 2011 at 1:55:00 PM PST
Interesting, sounds like all 3 roles require engineers that have coding skills.
Is there any place for exploratory testers without scripting/coding skills?
Replies
UnknownFebruary 10, 2011 at 10:24:00 PM PST
sounds like the key function performed by Google SET is enabling testability. I think a lot of software companies have SET perform the functions of both SET and TE described here, minus the expectation of code refactoring for the system under test. in my opinion, refactoring code is probably best performed by one closest to code, developers who are more aware of all the implementation subtleties since they coded the subtleties in the first place.
also, testability can be a hard quality to define and measure in some situations. i wonder how Google assesses contributions from SET, thru deltas in code cyclomatic complexity before and after the refactoring?
Replies
UnknownFebruary 11, 2011 at 7:43:00 AM PST
I wonder how you feel a smaller development shop should approach testing, say a smaller engineering R&D group part of a larger organization. Since we have fewer engineering resources most of them are focused on development and QA is a smaller group staffed mostly with a few testers with less of a computer science background who do more functional and acceptance testing. Do you think this is the most effective way to approach testing or would we benefit from investing resources in order to make our testing department more like Google's?
Replies
Shrini KulkarniFebruary 11, 2011 at 7:44:00 AM PST
James - the role of TE is new one I believe. Was this new? Late cycle testing, risk quality advising and lots of testing work (not testing by writing code).
Shrini
Replies
Erich von HauskeFebruary 11, 2011 at 12:11:00 PM PST
How is the mix? How many TE per SWE per SET?
How does this change in the differente Focus Areas?
Replies
UnknownFebruary 11, 2011 at 5:57:00 PM PST
My experience as a titled Test Engineer at an aerospace company aligns well with James' description with the exception of the statement that I spent most of my time writing automation. A majority of my time was spent working with the customers on how they use the system, researching test methodology & learning/testing the system I was working on in the lab. I typically performed those tasks in that order, over & over again, week by week until the project was complete. The automation I used as part of my testing was to simulate a random input generating live system that communicated with our system via an API. I needed to be able to control the input so that I could test different user scenarios & that's where automation became useful. However, I did not write the base test code, the code was written by an experienced developer who modified an existing script in a matter of minutes to do the basics of what I need. It was a fairly simple program so I was able to expend it to simulate exactly what I needed. The customers were impressed that I was able to not only create scenarios to test individual events & functions but that I was also able to create interesting real world scenarios to show how the functions integrated. I was able to design a large volume of user scenario tests because I was I had the benefit of a development expert modifying a piece of code for my own testing use that I could further expand as my test ideas expanded. A TE should be focused on how the customers use their system first & foremost. Automation is a tool to help them achieve the best possible scenarios in the smallest amount of time.
Replies
KazumatanFebruary 12, 2011 at 9:58:00 AM PST
I have the position equivalent to a SET. We work with "SWEs" in our designs but target users are typically "TEs".
Replies
Big 40wt SvetlyakFebruary 12, 2011 at 11:48:00 AM PST
Такое разделение труда наталкивает меня на мысль, что Гугл практикует создание "индусских" dev фабрик.
Это же наверное медленно, когда ты пишешь код, потом ждешь пока кто-то напишет для него тесты, потом окажется что тесты надо подкорректировать и ты опять кого-то ждешь. Куда продуктивнее все делать самому, советуясь с более сведущими коллегами в случае необходимости.
BTW, can't share this post in the Google Reader. Go write better tests ;-)
Replies
AnonymousFebruary 12, 2011 at 6:33:00 PM PST
Big 40 translated: This division of labor pushes me to think that Google is practicing the creation of "Hindu" dev factories. This is also probably slow, when you write code, then wait until someone writes tests to prove that the tests then have to correct you again and waiting for someone. Much more productive to do everything myself, consulting with more knowledgeable colleagues, if necessary.
Replies
UnknownFebruary 13, 2011 at 8:35:00 PM PST
Very interesting, SET and SWE must work very very closely and SET must understand SWE code very well. SET is like a white-box tester. So, what's the ratio of SWE, SET and TE in Google, like 3:2:1?
Replies
deniskoFebruary 14, 2011 at 11:15:00 PM PST
Hi, who is dog fooder?
What does it means?
Thanks.
Replies
Tomasz OstrowskiFebruary 24, 2011 at 4:12:00 AM PST
@denisko: see Eating your own dog food
Replies
TheronMarch 11, 2011 at 1:54:00 PM PST
Deeply grateful for this! You are my Rockstar of the Month. Thanks!
Replies
ctrisnoMay 26, 2011 at 3:21:00 PM PDT
James,
Thanks for sharing this great post. I'm just curious that if the SETs and SWEs are partnered together, then how do they deal w/ personality or other conflicts should they arise? The reason I ask is becuase in most companies especially the big ones, there's usually some individuals who are harder to get along and work with than others. So I'd just like to know how does Google handle this type of situation? Thanks.
Replies
AnandJuly 27, 2011 at 2:21:00 PM PDT
Hi,
I am very keen to know how does Google perform its UAT?
Considering the shear size of the search engine, how does it actually perform its UAT?
Or, does it perform some other tests instead of UAT?
Can you please let me know?
Thank you,
Anand
Replies
bigyellowJanuary 19, 2012 at 4:32:00 AM PST
what is the common ratio of SWE and SET in a delopement team?
Replies
Manoj TahilianiNovember 21, 2015 at 3:36:00 PM PST
Ratio of SWE and SET depends on team to team and also depends on type of project.
Generally it can be between 1:1 ratio to 1:3 ration (1 Test on 3 Dev).
Replies
Anthony ValloneNovember 23, 2015 at 8:22:00 AM PST
The Google average is about 1:10 (SWE:SETI), but it varies based on testability, complexity, and criticality. Some teams have no SETIs (test automation is straight forward), and the highest ratio I've seen is 1:5 (very complex automation required and/or mission critical project).
And of course, there are teams that are just SETIs, because they are working on Google-wide development/test infrastructure/tooling.
Replies
Load more...
The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.
Labels
●
TotT
99
●
GTAC
61
●
James Whittaker
42
●
Misko Hevery
32
●
Code Health
28
●
Anthony Vallone
27
●
Patrick Copeland
23
●
Jobs
18
●
Andrew Trenk
12
●
C++
11
●
Patrik Höglund
8
●
JavaScript
7
●
Allen Hutchison
6
●
George Pirocanac
6
●
Zhanyong Wan
6
●
Harry Robinson
5
●
Java
5
●
Julian Harty
5
●
Alberto Savoia
4
●
Ben Yu
4
●
Erik Kuefler
4
●
Philip Zembrod
4
●
Shyam Seshadri
4
●
Adam Bender
3
●
Chrome
3
●
Dillon Bly
3
●
John Thomas
3
●
Lesley Katzen
3
●
Marc Kaplan
3
●
Markus Clermont
3
●
Max Kanat-Alexander
3
●
Sonal Shah
3
●
APIs
2
●
Abhishek Arya
2
●
Alan Myrvold
2
●
Alek Icev
2
●
Android
2
●
April Fools
2
●
Chaitali Narla
2
●
Chris Lewis
2
●
Chrome OS
2
●
Diego Salas
2
●
Dori Reuveni
2
●
Jason Arbon
2
●
Jochen Wuttke
2
●
Kostya Serebryany
2
●
Marc Eaddy
2
●
Marko Ivanković
2
●
Mobile
2
●
Oliver Chang
2
●
Simon Stewart
2
●
Stefan Kennedy
2
●
Test Flakiness
2
●
Titus Winters
2
●
Tony Voellm
2
●
WebRTC
2
●
Yiming Sun
2
●
Yvette Nameth
2
●
Zuri Kemp
2
●
Aaron Jacobs
1
●
Adam Porter
1
●
Adam Raider
1
●
Adel Saoud
1
●
Alan Faulkner
1
●
Alex Eagle
1
●
Anantha Keesara
1
●
Antoine Picard
1
●
App Engine
1
●
Ari Shamash
1
●
Arif Sukoco
1
●
Benjamin Pick
1
●
Bob Nystrom
1
●
Bruce Leban
1
●
Carlos Arguelles
1
●
Carlos Israel Ortiz García
1
●
Cathal Weakliam
1
●
Christopher Semturs
1
●
Clay Murphy
1
●
Dagang Wei
1
●
Dan Maksimovich
1
●
Dan Shi
1
●
Dan Willemsen
1
●
Dave Chen
1
●
Dave Gladfelter
1
●
David Mandelberg
1
●
Derek Snyder
1
●
Diego Cavalcanti
1
●
Dmitry Vyukov
1
●
Eduardo Bravo Ortiz
1
●
Ekaterina Kamenskaya
1
●
Elliott Karpilovsky
1
●
Elliotte Rusty Harold
1
●
Espresso
1
●
Felipe Sodré
1
●
Francois Aube
1
●
Gene Volovich
1
●
Google+
1
●
Goran Petrovic
1
●
Goranka Bjedov
1
●
Hank Duan
1
●
Havard Rast Blok
1
●
Hongfei Ding
1
●
Jason Elbaum
1
●
Jason Huggins
1
●
Jay Han
1
●
Jeff Hoy
1
●
Jeff Listfield
1
●
Jessica Tomechak
1
●
Jim Reardon
1
●
Joe Allan Muharsky
1
●
Joel Hynoski
1
●
John Micco
1
●
John Penix
1
●
Jonathan Rockway
1
●
Jonathan Velasquez
1
●
Josh Armour
1
●
Julie Ralph
1
●
Kai Kent
1
●
Karin Lundberg
1
●
Kaue Silveira
1
●
Kevin Bourrillion
1
●
Kevin Graney
1
●
Kirkland
1
●
Kurt Alfred Kluever
1
●
Manjusha Parvathaneni
1
●
Marek Kiszkis
1
●
Marius Latinis
1
●
Mark Ivey
1
●
Mark Manley
1
●
Mark Striebeck
1
●
Matt Lowrie
1
●
Meredith Whittaker
1
●
Michael Bachman
1
●
Michael Klepikov
1
●
Mike Aizatsky
1
●
Mike Wacker
1
●
Mona El Mahdy
1
●
Noel Yap
1
●
Palak Bansal
1
●
Patricia Legaspi
1
●
Per Jacobsson
1
●
Peter Arrenbrecht
1
●
Peter Spragins
1
●
Phil Norman
1
●
Phil Rollet
1
●
Pooja Gupta
1
●
Project Showcase
1
●
Radoslav Vasilev
1
●
Rajat Dewan
1
●
Rajat Jain
1
●
Rich Martin
1
●
Richard Bustamante
1
●
Roshan Sembacuttiaratchy
1
●
Ruslan Khamitov
1
●
Sam Lee
1
●
Sean Jordan
1
●
Sharon Zhou
1
●
Shiva Garg
1
●
Siddartha Janga
1
●
Simran Basi
1
●
Stan Chan
1
●
Stephen Ng
1
●
Tejas Shah
1
●
Test Analytics
1
●
Test Engineer
1
●
Tim Lyakhovetskiy
1
●
Tom O'Neill
1
●
Vojta Jína
1
●
automation
1
●
dead code
1
●
iOS
1
●
mutation testing
1
Archive
●
►
2024
(9)
●
►
Jul
(1)
●
►
May
(3)
●
►
Apr
(3)
●
►
Mar
(1)
●
►
Feb
(1)
●
►
2023
(14)
●
►
Dec
(2)
●
►
Nov
(2)
●
►
Oct
(5)
●
►
Sep
(3)
●
►
Aug
(1)
●
►
Apr
(1)
●
►
2022
(2)
●
►
Feb
(2)
●
►
2021
(3)
●
►
Jun
(1)
●
►
Apr
(1)
●
►
Mar
(1)
●
►
2020
(8)
●
►
Dec
(2)
●
►
Nov
(1)
●
►
Oct
(1)
●
►
Aug
(2)
●
►
Jul
(1)
●
►
May
(1)
●
►
2019
(4)
●
►
Dec
(1)
●
►
Nov
(1)
●
►
Jul
(1)
●
►
Jan
(1)
●
►
2018
(7)
●
►
Nov
(1)
●
►
Sep
(1)
●
►
Jul
(1)
●
►
Jun
(2)
●
►
May
(1)
●
►
Feb
(1)
●
►
2017
(17)
●
►
Dec
(1)
●
►
Nov
(1)
●
►
Oct
(1)
●
►
Sep
(1)
●
►
Aug
(1)
●
►
Jul
(2)
●
►
Jun
(2)
●
►
May
(3)
●
►
Apr
(2)
●
►
Feb
(1)
●
►
Jan
(2)
●
►
2016
(15)
●
►
Dec
(1)
●
►
Nov
(2)
●
►
Oct
(1)
●
►
Sep
(2)
●
►
Aug
(1)
●
►
Jun
(2)
●
►
May
(3)
●
►
Apr
(1)
●
►
Mar
(1)
●
►
Feb
(1)
●
►
2015
(14)
●
►
Dec
(1)
●
►
Nov
(1)
●
►
Oct
(2)
●
►
Aug
(1)
●
►
Jun
(1)
●
►
May
(2)
●
►
Apr
(2)
●
►
Mar
(1)
●
►
Feb
(1)
●
►
Jan
(2)
●
►
2014
(24)
●
►
Dec
(2)
●
►
Nov
(1)
●
►
Oct
(2)
●
►
Sep
(2)
●
►
Aug
(2)
●
►
Jul
(3)
●
►
Jun
(3)
●
►
May
(2)
●
►
Apr
(2)
●
►
Mar
(2)
●
►
Feb
(1)
●
►
Jan
(2)
●
►
2013
(16)
●
►
Dec
(1)
●
►
Nov
(1)
●
►
Oct
(1)
●
►
Aug
(2)
●
►
Jul
(1)
●
►
Jun
(2)
●
►
May
(2)
●
►
Apr
(2)
●
►
Mar
(2)
●
►
Jan
(2)
●
►
2012
(11)
●
►
Dec
(1)
●
►
Nov
(2)
●
►
Oct
(3)
●
►
Sep
(1)
●
►
Aug
(4)
●
▼
2011
(39)
●
►
Nov
(2)
●
►
Oct
(5)
●
►
Sep
(2)
●
►
Aug
(4)
●
►
Jul
(2)
●
►
Jun
(5)
●
►
May
(4)
●
►
Apr
(3)
●
►
Mar
(4)
●
▼
Feb
(5)
●
This Code is CRAP
●
How Google Tests Software - A Brief Interlude
●
Who reads this blog?
●
How Google Tests Software - Part Three
●
How Google Tests Software - Part Two
●
►
Jan
(3)
●
►
2010
(37)
●
►
Dec
(3)
●
►
Nov
(3)
●
►
Oct
(4)
●
►
Sep
(8)
●
►
Aug
(3)
●
►
Jul
(3)
●
►
Jun
(2)
●
►
May
(2)
●
►
Apr
(3)
●
►
Mar
(3)
●
►
Feb
(2)
●
►
Jan
(1)
●
►
2009
(54)
●
►
Dec
(3)
●
►
Nov
(2)
●
►
Oct
(3)
●
►
Sep
(5)
●
►
Aug
(4)
●
►
Jul
(15)
●
►
Jun
(8)
●
►
May
(3)
●
►
Apr
(2)
●
►
Feb
(5)
●
►
Jan
(4)
●
►
2008
(75)
●
►
Dec
(6)
●
►
Nov
(8)
●
►
Oct
(9)
●
►
Sep
(8)
●
►
Aug
(9)
●
►
Jul
(9)
●
►
Jun
(6)
●
►
May
(6)
●
►
Apr
(4)
●
►
Mar
(4)
●
►
Feb
(4)
●
►
Jan
(2)
●
►
2007
(41)
●
►
Oct
(6)
●
►
Sep
(5)
●
►
Aug
(3)
●
►
Jul
(2)
●
►
Jun
(2)
●
►
May
(2)
●
►
Apr
(7)
●
►
Mar
(5)
●
►
Feb
(5)
●
►
Jan
(4)
Feed
●
Google
●
Privacy
●
Terms