In Google’s early days, a small handful of software engineers built, tested, and released software. But as the user-base grew and products proliferated, engineers started specializing in roles, creating more scale in the development process:

This story focuses on the evolution of quality assurance and the roles of the engineers behind it at Google. The REs and SREs also evolved, but we’ll leave that for another day.

Initially, teams relied heavily on manual operations. When we attempted to automate testing, we largely focused on the frontends, which worked, because Google was small and our products had fewer integrations. However, as Google grew, longer and longer manual test cycles bogged down iterations and delayed feature launches. Also, since we identified bugs later in the development cycle, it took us longer and longer to fix them. We determined that pushing testing upstream via automation would help address these issues and accelerate velocity.

As manual testing transitioned to automated processes, two separate testing roles began to emerge at Google:

Test Engineers (TEs) -- With their deep product knowledge and test/quality domain expertise, TEs focused on what should be tested.

Software Engineers in Test (SETs) -- Originally software engineers with deep infrastructure and tooling expertise, SETs built the frameworks and packages required to implement automation.

The impact was significant:

Automated tests became more efficient and deterministic (e.g. by improving runtimes, eliminating sources of flakiness, etc.)

Manual operations were reduced to manual verification on new features, and typically only in end-to-end, cross product integration boundaries. TEs developed extreme depth of knowledge for the products they supported. They became go-to engineers for product teams that needed expertise in test automation and integration. Their role evolved into a broad spectrum of responsibilities: writing scripts to automate testing, creating tools so developers could test their own code, and constantly designing better and more creative ways to identify weak spots and break software.

SETs (in collaboration with TEs and other engineers) built a wide array of test automation tools and developed best practices that were applicable across many products. Release velocity accelerated for products. All was good, and there was much rejoicing!

Through these efforts, the testing cycle time decreased dramatically, but interestingly the overall velocity did not increase proportionately, since other phases in the development cycle became the bottleneck. SETs started building tools to accelerate all other aspects of product development, including:

Automating measurement of developer productivity, helping understand what’s working and what isn’t.

In summary, the work done by the SETs naturally progressed from supporting only product testing efforts to include supporting product development efforts as well. Their role now encompassed a much broader Engineering Productivity agenda.

Given the expanded SET charter, we wanted the title of the role to reflect the work. But what should the new title be? We empowered the SETs to choose a new title, and they overwhelmingly (91%) selected Software Engineer, Tools & Infrastructure (abbreviated to SETI).

Today, SETIs and TEs still collaborate very closely on optimizing the entire development life cycle with a goal of eliminating all friction from getting features into production. Interested in building next generation tools and infrastructure? Join us (SETI, TE)!

31 comments
:

Would be able to provide more detail with regards to expectations of the TE when it comes to their technical abilities, is it similar to SETI. Would TEs at google say that they are automation experts, or are they more focused and specialized on manual testing?

I believe the article is saying the TEs are good at making test plans, risk analysis, and manual testing while SEIT/SETI excel at automating tests, builds, releases, deployment, gathering metrics, and finding ways to reduce cycle time.

I find this to be a useful distinction for job descriptions and it's worth noting that both roles are very valuable to product teams.

There is an older, but still accurate, post about the expectations and responsibilities of test engineers at Google.

http://googletesting.blogspot.com/2013/01/test-engineers-google.html

As for should they be separate people, it depends on the team (and, thinking outside of Google, the company).

For a smaller team/company, it wouldn't make sense to have distinction between the roles because usually there'd only be one person responsible for everything. However, when products get larger and more complex, the team grows, and people tend to gravitate toward the responsibilities that interest them most.

It's not that SETIs can't do the work we primarily attribute to TEs, or vice versa. Many engineers find the primary responsibilities of one role or the other more interesting and more in line with what they want to do. Having separate roles makes things simpler versus having a combined role with a laundry list of responsibilities, many of which an individual engineer might not be doing.

To clarify, the purpose of this post was to describe the transition from Software Engineer in Test (SET) to Software Engineer, Tools & Infrastructure (SETI) and explain the new SETI role. Look for a future post on the evolution of the TE role at Google, but also refer to past post such as http://googletesting.blogspot.com/2013/01/test-engineers-google.html .

To answer some of the immediate questions:- The TE role does have a coding bar candidates need to meet when they interview for a TE position, and automated testing is part of the TE role. The technical bar is quite high for all individuals at Google: All engineers, and many others (e.g. people managers, program managers, etc.) also contribute code.

- As people's passions change over time, they can (and do) learn new skills, leading to a mobility within teams and roles. Via rotations, engineers can also try out other roles before committing to a change.

All engineers at Google are expected to contribute test cases for code that they write, not just TEs. We actively avoid situations where TEs end up writing all the test cases for code written by others.

I think it's possible that you could have one of each. On an Agile Scrum team I think you would always want to have a TE. Having a SETI is the goal but it may not always be fiscally possible or needed for a mature team. More frugal organizations may have 'floating' SETI engineers which move from team to team as needed.

It is very interesting and excellent article showing the reality and evolution that has the area of testing. What I like most is that unlike the 2 roles (SETI and TE) and integrated work will allow a response time faster for the attention of the SDLC.

I agree with you, Alvaro, many times it takes disparate skills and team work to solve a problem. In an ideal world, the same person would have all the necessary skills, but that rarely happens, especially for complex issues.

I read this article recently: http://www.wired.com/2016/04/google-ensures-services-almost-never-go/

It talks about the SRE discipline at Google. What I find very interesting is the paragraph that talks about the two types of engineers that are hired, e.g. “set of technical skills that is useful to SRE but is rare for most software engineers"..

This post could not have come at a more appropriate time for my team. We are in the middle of the evolution from a more manual approach to an ever increasing set of automation and tooling. We have also recognized the necessity of our "Manual" testers, but it has been hard putting our finger on it in relation to the automation teams. This article helped to define that line a little better for us. I would love if you could provide any examples, allegorical or otherwise, about the two groups working together on a project? Maybe fodder for a future blog post?

"Their role evolved into a broad spectrum of responsibilities: writing scripts to automate testing, creating tools so developers could test their own code..."

So the TEs write scripts and create tools for the devs to use - not the SETs? Is this because the scripts and tools require domain / product knowledge which the TEs possess?

This distinction between the two roles is very interesting, as is the strong point made that engineers in general have coding skills. We are working towards this in my organisation but it is quite daunting, mainly due to the vastness of applications, products languages etc.

Regarding the sentence, "Their role evolved into a broad spectrum of responsibilities: writing scripts to automate testing, creating tools so developers could test their own code..."

We added that sentence to indicate that TEs, if they weren't already, started contributing tools and code in a big way as part of their evolution at Google. In fact, contributing technically is part of the job description for TEs. TEs at Google cannot be successful w/out technical contributions.

This task is by no means exclusive to TEs only. Every engineer within Google (TE or otherwise) ends up contributing tools and code during their career.

I hope this clarifies, feel free to follow up if you have any further questions.

Ari, Good article on your journey from SETs to SETI. Would like to know how the whole organization is now structured at Google specifically product development for various verticals and different/one productivity organization.

Thanks for sharing your eloquent story about Google evolution of quality assurance and the roles of the engineers , this is same experience we went through in our company. In order to close the test gap we improved our team cohesion and operational agility by redesign our organization away for a hierarchy organization to network organization. In addition be began merit badge system to encourage our QA to shift left by allowing them examine other tools and methods to determine if want to pursue them as a vocation. Looking foward to sharing experience and discussing idea about ways to increas engineering productivity.

Hi Ari, It's really great article! Just one quick question here, as you mentioned SETI are building tools to accelerate all other aspects of product development, now container technique are becoming mainstream. So is there any overlap between SETIs and REs? Will these two roles combine together in future.

Do the role SET's support cross platform organization? Meaning, SET's working for Software QA initiatives are expected to support Hardware QA initiatives as well? I am curious to understand how Google handles this after the new roles are coined, generally the Software QA engineers primarily focus on the productivity and efficiency measurements and they write and perform unit test using this as a basis whereas the Hardware engineers (in the testing organization) should have skills in the reliability level like Mean time between failure, Mean cycle between failure etc...