Enterprise Selenium Grid with Michael Palotas - Test Guild Podcast

‍

Joe

Welcome to another episode of the Test Guild Automation podcast. Today we'll be talking with Michael Palotas, on Enterprise Selenium Grids. Michael currently works as the CEO of Element34 Solutions (now Element34), the market leader for Enterprise Selenium Grid infrastructure solutions that run inside the corporate firewall. Before Element34, he was the managing director at Grid Fusion, an engineering consulting firm for software organizations that specialize in lean and agile software development. Hopefully we'll also dive into this a little bit more: Michael was the Head of Test Engineering at eBay, where he was instrumental in the design, development, and open sourcing of Selenium Grid, Selendroid and iOS driver. Michael also worked as a software engineer at Intel and Nortel, so he really knows his stuff. I’m really excited to have him on the show. If you do anything with Selenium and Selenium Grid, you don't want to miss it.

Michael, I'm really curious to know… you actually developed the original implementation of Selenium Grid?

Michael

Yes, but not just myself. I was working at eBay at the time and we were tinkering with Selenium and had this need to run a lot of tests in a very short timeframe. So we said, okay, let's just build it ourselves. So a few folks on my team went off and built something that was the initial version of Selenium Grid and we used it inside the company. Then we started talking to some folks on the Selenium project, like Simon Stewart. We said, okay, why don't we open source it and make it available for everybody? Then of course from that point, a lot of other folks jumped in and added very useful features to it. But yes, I think it's fair to say that we were heavily involved in the creation and open sourcing of Selenium Grid.

Joe

Awesome. So what was the genesis of you developing Selenium Grid then? What was it about the software development lifecycle at eBay that you thought, Wow, we really need this solution. Let’s build it. Now that it’s built we have the solution. Then after that thinking, Wow, this solution is not only going to work for us, but it's going to work for other engineers.

Michael

Yeah, at eBay we had a requirement to scale our automated tests and we had a lot of tests. We were looking after a lot of different platforms on my team. We had this need that we couldn't just run our tests sequentially anymore. I think if we had done that, it would have taken seven days or more to run everything sequentially. We needed a solution that allows us to parallelize our tests and to bring it down to a matter of minutes and ideally to run the entire suite. That was the key driver for us to go and develop a solution for it. Then we said, all right, now it works for us. We were in contact with the folks in the Selenium project. We just started talking to them and they said, it would be nice if you could open source this.

So we went through a bit of a process inside the company to figure out how we can open source things without giving away any eBay specifics. But then in the end we managed to open source it, make it available to everybody, and I think this is one of those nice examples where you can see that when somebody open sources something, then other people come in, add more stuff, and then it becomes what it is today.

Joe

So Michael, before we go any further, I assume people listening to this are pretty hip to testing and tools, but just in case they aren’t let's set the stage a little: What is a Selenium Grid?

Michael

The Selenium Grid is effectively a piece of infrastructure that allows you to run tests across different browsers and across different operating systems at scale. It provides a central entry point for all your tests that you have in the Selenium space. It provides you a central entry point for your test execution to run that across different browsers and mobile via Appium. I think essentially what Selenium Grid allows you to do as an organization is to shift left, or to help shift left, by speeding up the feedback cycles and by shortening those feedback cycles within the organization. Because typically as test automation matures, there's more and more tests and if you don't have a way to run all these tests in parallel, those test runs will take longer and longer and that's not something you want. That's where Selenium Grid comes in and allows you to run a lot of tests in a short time.

Joe

Nice. So if someone's thinking, this sounds great, but why would I need Selenium Grid? It sounds like one main reasons is to be able to run your tests in parallel. Are there other main reasons you usually provide for why an organization should be using a Selenium Grid?

Michael

I think one other main factor is the cross browser aspect. Usually when you look into dev organizations, a lot of testing is happening on Chrome only. Depending on who's using your application that you're developing. But typically, especially for publicly facing websites, you can't really control what kind of browsers people are using. So you have to make sure that all the different browsers are covered in your testing. That's also something where Selenium Grid helps. So you can have any browser, any browser version, any kind of operating system, browser and version combination at scale in a Selenium Grid.

Joe

I've used Selenium Grid in-house for certain companies and I found sometimes the maintenance of the grid is really difficult. Are there any known difficulties you've seen companies, or organizations, have when they try to create their own Selenium Grid?

Michael

Yes, getting started with Selenium Grid is typically fairly simple. You download the jar, spin it up, attach a couple of notes to it, so you can show results very, very early on. It maybe takes you half a day to get something up and running. What becomes very labor intensive (and also very expensive) is the whole maintenance of the grid as it grows and scales. Having a couple of browsers supported, that's easy. But once you have a Selenium Grid that serves an entire enterprise, for example, where you potentially run hundreds or thousands of tests in parallel, you need all those machines. You need all that infrastructure to support that. Then it can become very time consuming and very expensive to maintain that.

Sometimes it's an entire team that does that. Sometimes it's also just a one man show, or a one woman show, who does that in a company. Such a central piece of the CI pipeline, like Selenium Grid, is sometimes completely dependent on one person in the company, whether that person is there or whether that person is on vacation. When the person is on vacation, then this piece of infrastructure doesn't get maintained. There’s typically a cost aspect to not maintaining. When you talk about building and maintaining your own grid, there's also scalability issues in the open source Selenium Grid version today. You can only scale to a certain point. That's also what we hear from our customers. It's also from a business continuity perspective, if you want to make sure that this piece of your CI pipeline is up to date, or up to speed, and up and running all the time. Having that relying on one person can prove difficult.

Joe

I absolutely agree, and that's why I think a lot of folks go to cloud-based solutions but that’s not always going to be the right option for companies especially when it comes to privacy. When I spoke with you before you said not only is there your own grid and cloud-based, but there's another kind, and that’s where you came up with Selenium Box (SBOX). Could you tell us what Selenium Box is? What makes it different between a cloud vendor option or an in-house type of solution?

Michael

In general when we look at the Selenium Grid space, there's two main paths you can choose. One is you have it running inside your firewall, inside your network, and the other one is the outside. With that, the SaaS option. I think for any enterprise, building a Selenium Grid on your own, is not really an option for them. Because they typically get to a point where they say, okay, this is too much, we need something vendor managed with proper SLAs around it. So as you said, there are numerous cloud, or SaaS solutions out there, which are, by the way, are fantastic products. They’re feature rich products and have all the bells and whistles that an open source homegrown Selenium Grid doesn't have. Then there's this other path that you can also take.

If you want to stay inside your corporate firewall for, and there's various reasons why you would want to do that, but if you decide that you want to stay inside your corporate firewall, then there is, Selenium Box. That's the product that we develop and we bring that experience that customers have with the SaaS providers. We bring all that inside the corporate firewall. With that there's a lot of aspects, specifically around security. Because if you go out to a SaaS solution, you're sending out all your data, all your test data to the SaaS solutions, they can potentially see that data. For some organizations that may be okay, and for others it isn't. For those where it is not okay to send things outside their corporate firewall, that's exactly where we come in, where we bring that rich feature set inside your firewall without having to do any of the maintenance that we mentioned earlier in the homegrown and self maintained Selenium Grid space.

Joe

I used to work for a very large, large company healthcare division, and it was very difficult to get any type of cloud-based solution that we could use. Even if we did, it would take forever to find out who we would even go to for getting permission to open up a security port or something that the utility might need to use. So a lot of times what they end up doing is they take the SaaS solution, they do an on-prem implementation of it. So is Selenium Box just an on-prem solution of a cloud provider, or is it something else as well?

Michael

Yes, so from a feature perspective, you can look at all the features that you are used to from the SaaS solutions, such as: live view video and getting all the log details. That's also something that we provide. All the features that you see in SaaS solutions, we take that and we bring that in-house. Selenium Box itself is not a SaaS product, we provide all the bells and whistles that people expect with the added security that everything is running inside your firewall. Not only security, but with running tests inside your firewall, you also have aspects like performance. For example, you don't have to cross a big pond or go from the East Coast to the West Coast and around for every Selenium command. If it runs inside your network, the latency times are very short, and with that, you will see a massive increase in performance.

Joe

I would think that'd be a key feature because that's one of the main complaints and concerns folks have when they run in these cloud-based solutions. It does tend to be slower. So it sounds like that's another main benefit is if you have an in-house, it runs faster and is more secure.

Michael

Absolutely. Of course that will depend a little bit on where you are and your servers are and where that SaaS provider is. If you're next door, then the latency is not as big of a deal. But let's not forget in an enterprise environment in order to get outside of your network to a SaaS provider and back in, you have to navigate several proxies, several firewalls which typically slows things down. It’s very common that we get the feedback from our customers, especially the ones who were with SaaS solutions before, where they say, wow, this is a magnitude faster than before.

Joe

That’s very cool. So are there other features that differentiate you from any other type of solution out there in this space?

Michael

Just because we are running inside the customer network, we have a lot tighter integration points into the existing enterprise infrastructure. Certain authentication mechanisms that we can support by being inside the network. There's a lot of enterprise level features that come because you are inside the network and cannot be exposed to the outside. So as a SaaS provider, you are not in a position that you can offer those services because you don't have access to them. Whereas when we're inside the network, we can actually provide a lot of those additional features.

Joe

So another popular option is Selenium, which I believe uses like Docker or Kubernetes in the background and puts up things that makes it quicker, and faster, to tear up and tear down environments for your tests that are supposed to help with maintenance. So what's the difference between say, Selenium Box vs Selenium?

Michael

So, one of the key differences with Selenium Box and Selenium is that Selenium is an open source project and we are a commercial product. There were no SLAs around open source projects. If you have a problem, you may or may not get a response on an issue you’re having. All our customers are large enterprises that need SLAs around if something goes wrong. They need a one hour response time for us to get back to them. When we look at a project like Selenium that supports Chrome and Firefox. So everything that you can spin up in a container, whereas we go way beyond that, we can do Internet Explorer, we can do Edge, we can do Safari, we can do mobile emulators for iOS and Android, and we can also do real devices. It is really the entire spectrum of what you may need as an enterprise rather than just the Firefox and Chrome containers.

Joe

So I know I asked you about Zalenium but I just realized that it actually has been discontinued. So I guess I was just using Zalenium as an example, but whatever you said about Zalenium would apply to any other solution now or after that is open source. Where you talk about, I think a lot of people don't appreciate is when you work for an enterprise security and being able to get tech support is very important because those SLAs is usually mission critical for certain functionality for sure.

Michael

It’s not just about Zalenium, but there’s various other open source Selenium Grid solutions out there. I think one thing with any open source tool, when you want to bring that into the enterprise, I find it always makes sense to look at how active that project is. When has the last commit been made? If it's five years ago, then there's a pretty good chance that either the product is, or the project, is really mature or it's dead <laugh>. I think what's super important to look at is how many people are behind it. I think specifically in the Selenium Grid space, there's really cool solutions out there, but it's commonly maintained by one person. If that person moves on and you've brought that into your enterprise tool chain you will have a problem. That's what we're seeing. That's why companies are looking toward commercial solutions like the SaaS providers, if that's okay with them from a security perspective or of course, and also at solutions like what we provide.

Joe

So once again, not a small point people should dwell on this a little bit. I've been blogging for 10 years and I just looked at my broken links, and a bunch of them are two projects that are no longer support on GitHub because, like you said, they had one or two maintainers. That's a great point that whenever you invest in any type of technology, especially open source, you want to make sure how active the community is. That's a really great tip. So the next question that folks would have then is say for some reason someone does invest in Selenium Box, is there anything they they need to do in order to make their tests work with Selenium Box? Then if they no longer use Selenium Box, do they have to make any changes to the code for it to work normally again in a normal Selenium Grid.

Michael

No, there are no changes required at all to the tests. Just like the SaaS solutions out there, they support the W3C standard. So do we. The web driver protocol is W3C and we fully are compliant to that. If you are running your tests on your homegrown Selenium Grid, either on the SaaS solution or on ours, there's absolutely no changes required. That said the change that you have to make is where you send it to and do you send it to us? Do you send it to one of the SaaS solutions? That's a one-liner in a configuration file that you need to change. From a Selenium command and from a Selenium test perspective, there's zero changes that are required. That's the beauty of open source technology. For a customer there is no vendor lock-in. If they can choose the solution that works best for them, and it's not because you bought Selenium Box, that you're going to be locked in for the next 10 years to use it. If there's something better out there you can go there. Which is also one of the key drivers for us to provide a fantastic product and for everybody else as well. It is easy to switch if needed.

Joe

That's awesome. You don't need to change your test to run against Selenium Box. Another question would be, you've been dealing with Selenium Grid since the beginning, are there any best practices folks still need to think about as they're creating automation tests that allow them to run better, or more optimally, in a Selenium Grid type environment?

Michael

Yeah, I can particularly think of one thing that nothing to do specifically with Selenium Grid, but what we are seeing quite frequently is that companies who start out running tests sequentially, they come to that point when they say, all right, we need something that is able to parallelize our tests to run them faster. Then they plug in a Selenium Grid and everything explodes, <laugh>, so their tests explode, nothing works, they get flaky, sometimes they pass, sometimes they fail, and oftentimes the blame is on the tool itself. So Sauce Labs, BrowserStack or US (Selenium Grid) or on Selenium as a whole. Deep down, what typically happens is that tests are not designed with parallelizing them in mind. Those tests are designed to be run sequentially. When you take a thousand tests that were run sequentially before, and then say, okay, now run them in parallel, and they're not independent from each other. Let's say they share the users, they share the test data that they're using, if you have a situation like that, then, of course, you will get a horrible outcome.

Michael

So one piece of advice that I can give (and this is also just from working on the side where we created a lot of tests) is design your tests with parallelization in mind from day one. Which means, ideally, you are able to have test data that is atomic to each test. Each test should have their own data. And if that's production data or production-like data, that's fine. It's just that every test should have their own set of data rather than sharing the five golden user accounts across a thousand tests that you're going to run in parallel. I think, that's one of the key success factors to run tests at scale and to get those short feedback cycles.

Joe

Great advice. So for Selenium Box is it comparable to a cloud-based solution or any other price benefits of going with a in-house solution?

Michael

That's a very good question. As I mentioned before our Selenium Box solution is an on-premise solution. It's designed for use across a whole enterprise. If you are looking for maybe one or two users as an introduction to commercial Selenium Grid, then we would be outside of your price point until up to a certain level. We'll be equivalent to other commercial SaaS vendors. The beauty of our solution is we allow companies to scale to an infinite amount, and when you can scale to that amount using our product with an infinite license, then we become very, very commercially attractive relative to other solutions. So really there's an initial price point where we get into the game, and then the more you scale, the more the our solution works in your favor.

Joe

I guess the question should have been build vs buy. It sounds like you're saying if you're a small team and you're using a Selenium Grid, I know a lot of small teams are happy with it, that's fine. As I said, I worked for an enterprise and I worked across multiple verticals against multiple teams doing multiple things. A solution like this might be the perfect kind of option for 'em, so definitely they should check it out. I guess one other thing I want to cover before I ask my last question is updates. There's pros and cons to having things on-prem but one thing SaaS products are good at is if there's an update, they handle it for you usually, but when it's on-prem, you have to coordinate with a bunch of different teams and make a plan when you are going to do the update. How hard is it to update a solution that that is in-house in general like yours?

Michael

Thank you for that question because that is actually one of our key strengths because the update is exactly as transparent to the end user as it is on any SaaS solution. So what we do is when there's, let's say a new browser that releases, we of course pick that up, we test everything, we make sure that everything works just like how we expect it to work, and then we make that binary available in a private repository for the customer. Then the customer just pulls it automatically from there. So the customer has actually no interaction whatsoever with the fact that there is a new browser that released or that anything changed in the sell any new ecosystem. From that experience perspective, it's exactly the same as using a SaaS solution. That's also what we want. We want to bring that SaaS experience inside the firewall.

Joe

Okay, Michael, before we go, is there one piece of actionable advice you can give to someone to help them with their automation Selenium Grid testing efforts? And what's the best way to contact you or learn more about Selenium Box?

Michael

So you can find us on either SeleniumBox.com or Element34.com. If you have any questions you can reach out and find us there. There's also a lot of resources out there that you can go through. If you want to learn more, a piece of advice that we give to any kind of customer that we start interacting with is to think about is “Do you want to build or do you want to buy?” If you want buy, do you want your testing data to go outside of your firewall with all the implications that come with that? Or you do want to keep that all  inside your firewall? That will guide you in terms of which solution is the best one for you. From there I think it's important to understand that none of the Selenium Grid providers make it easier to write your tests.
‍
So you still have to write your tests in a much more robust way than if you just run everything sequentially. So the Selenium Grid gives you that opportunity to run a lot of tests in a short timeframe but writing the test is still on you and I always say test automation is software development. You develop software in the end and you should apply all the techniques, all the patterns that you would apply in a regular software application and in the development process. You should never say, oh, this is just a piece of test code and that's why we can get sloppy. I think that's when we look at how to write good tests and that's typically what I take as the baseline.

Joe

Thank you, Michael, for your automation awesomeness. As always, test everything and keep the good. 

‍