Why Buying a Commercial Selenium Grid Solution is (Typically) Better Than Building Your Own

For the past decade we've been working closely with a variety of companies who have created (or thought about creating) their own Selenium Grid solutions in-house.

Although from an engineering perspective these always look like exciting projects, the reality is that, in most cases, they end up not being such a good idea - especially if you plan to use that solution in the long term.

There are multiple aspects that come to play, and that you’ll need to consider (very carefully):

• Cost
• Business Continuity
• Skills & Talent
• Productivity
• Functionality and Features

Let’s look at each of these in detail:

Cost

Depending on the size and complexity of your organization, the cost of building and maintaining your own Selenium Grid will obviously depend. 

However, the drivers of cost will be common between organizations.

First of all, building and maintaining your own enterprise grade Grid requires significant resources. To do it well, the required staff will need to include senior level software engineers with deep know-how in Selenium, system administrators, project manager, security engineers, and network architects. If those resources are not readily available, you’ll need to invest time (and resources) in recruiting. So you should consider not only the cost of the actual resources that you’ll allocate to the project, but also the cost of the recruiting itself.

From our experience, the typical timeframe for building such a system can be anywhere from 9 and 24 months, where this team will be fully focused on the development effort. Maintenance and further development will also require a team of developers, system admins and security personnel allocated close to full time. 

If mobile testing is in scope, these figures can get significantly higher. Appium/mobile is extremely complex to set up in an enterprise grade fashion especially if you want it to work in a combined Grid for browsers and mobile.

When compared with available off-the shelf solutions, the cost of building your own Selenium Grid will probably not make economical sense from the get go. 

Business Continuity

As described above, a homegrown testing Grid requires a highly specialized internal team (with backups) for continuous support, maintenance and further development.

In case of a company restructuring, attrition, or if key people leave the team, the entire test automation infrastructure can be jeopardized. We all know that documentation in internal projects is usually non-existing and that knowledge transfer is difficult (to say the least!)

These resource related challenges are a serious hazard for business continuity!

When buying a commercially supported solution like SBOX, there are clear SLAs in place and you always opt for 24/7 support - so even if your company changes priorities and shuffles resources, your enterprise testing grid will always be out of harm’s way.

Skills and Talent

As mentioned in the Cost considerations, building and maintaining your own Grid requires a team that has in-depth knowledge of Selenium (down to the protocol level) as well as advanced skills in server operations, proxies, firewalls, etc. 

These skills are typically quite hard to find, let alone bringing them together into one team. 

Pulling together the right team to implement your Selenium Grid might be the biggest show stopper for your project. 

Productivity

Building and maintaining test automation infrastructure is (most likely) not your core business. 

With engineering resources being scarce, organizations should focus on tasks and topics that are their core business and lie within their core competencies. 

For everything else, they should use commercial solutions - just like everybody uses tools like Microsoft Office rather than building their own text editors or spreadsheet software.

If you jump into a build project, you will see your resources spending considerable time and energy keeping the lights on on a supporting system, instead of being productive executing tests on other business applications that might have a bigger impact on your company’s bottom line.

Functionality and Features

The open source Selenium Grid is fundamentally not enterprise ready as there are many key features missing. Also, the architecture does not easily allow for building these features without considerable investment. The open source Selenium Grid was never meant to be used in enterprises.

A commercial solution will provide a comprehensive set of features out of the box, that will immediately kick start your testing team’s productivity and results.

As an example, below is a high-level overview of SBOX features in comparison to an open source grid:
‍

On the maintenance side, browsers are usually updated once a month, with no advanced warning. Since cross-browser testing is a key reason for having a test grid, you’ll want to have your maintenance team available and ready to jump-in to ensure that your grid is ready to support new browser versions as they come out. This level of maintenance effort is considerable, and if not addressed will have a tremendous impact on your ability to test your applications in the latest browser versions that your users have on their devices.

Conclusion

A secure, reliable and scalable Selenium Grid cross-browser testing infrastructure is crucial for successful continuous testing and DevOps setup. 

Setting up and maintaining a mature enterprise grade Selenium Grid infrastructure requires a significant investment and is a complex undertaking. While a homegrown solution can be a good starting point, for most enterprise scenarios it will quickly become unmanageable.

Homegrown solutions require a big up-front investment in terms of engineering resources, deep Selenium Grid know-how, and ongoing maintenance which is time consuming and prone to error due to frequent new release of browsers and the Selenium ecosystem.

Unless your company has very specific and unique needs that are not covered by existing solutions (and has the skills and budget to embark in a build project), the best option will be to select a proven commercial solution that can cover your needs immediately, at a fraction of the cost of building it.