TestVagrant completed more than 2 years in business and during the last 2 years we have worked on a variety of mobile apps. Apart from giving us a good dose of exposure across domains, these projects helped us better understand the mobile testing landscape. The mobile ecosystem and its challenges are way different from web based applications of the past. I am saying “past” because there is no point re-iterating the fact that has been universally accepted “Mobile is the future”.
So here is to the future. Let’s discuss in detail the challenges to mobile test automation and the intelligent solutions that could come to the rescue.
Here is a quick gist of some of the mobile apps that we worked under TestVagrant:
- A fastest growing billion-dollar valued B2C enterprise app dealing with logistics domain
- Any end to end scenario in this space meant interaction between 2 apps, one which consumer is using and the other the corresponding driver app.
- Competing with some of the globally established player like Uber, one of the prime requirement was to build solution that would be fast, intelligent and reliable.
- Multiple products lead to lot of end to end scenarios and the massive user base led to the requirement of running tests in parallel as well as addressing the device fragmentation.
- UK based cab aggregator app dealing with a large volume of cab booking on a regular basis.
- Given the geographical spread, majority of their user base was on iOS.
- They had just started, so focus was a lot more on stability and faster feedback.
- Integration with SauceLabs was strategic given the variety of iOS devices needed for test run.
- A fitness app targeted towards millennials.
- Given their user spread, a lot of mid to high end android devices needed to be covered.
- The inherent nature of the app needed a geo spoofing to be able to test the scenarios.
- Addressing device fragmentation was the key in this case.
All of above plus a few more apps that we came across were in tricky B2C space, where the following needed to be solved:
- Covering a good number of devices based on the user spread. Though it might sound simple in case of iOS, this could be really tricky in android space.
- Covering test scenarios in logistics space where 2 or more apps and their respective users could interact. e.g. A cab booking app like Uber would need 2 apps (Driver and Customer) and respective user profile to communicate for an end to end test to work.
- Dealing with a large number of test scenarios, running them multiple times a day given the pace of feedback required. So the need was of a very reliable mechanism to run tests in parallel.
- Spoofing a lot of device level data to simulate run related scenarios. e.g spoofing gps location to simulate a 5km run, integrating with on device stored music to play while the run is finished, sharing the geographic map on social media etc.
Over a period of time we could identify patterns across different projects, majority of which had lot of common problems to solve in terms of the ecosystem challenges. This collective learning resulted in a smart and intelligent solution “Optimus”.
Before we discuss further details on Optimus, let’s understand where we are in the industry with respect to the mobile automation solutions :
- Currently there are no freely available solutions (Uber built Octopus but never made it public) to deal with inter app available, whether it is “android to android” or “android to iOS”.
- Running tests in parallel, distributed functionally across multiple devices is a situation that is sparingly discussed on different forums but no code to help the folks out there.
- Running all tests across all devices in another flavour of parallelism but is really tricky to achieve.
- Getting an insight into hardware performance snapshot like CPU, memory or battery usage for a particular step in your functional test scenario.
Optimus does all of the above and a lot more. The philosophy behind is – Simplify mobile testing for the community at large.
How does Optimus work ?
- Optimus gets all connected devices and prepares a device matrix that it uses intelligently to run tests on.
- Based on the capabilities specified, it creates driver instances to use while authoring the test.
- Everything which is needed as part of mobile ecosystem is built in and can be readily used, there by reducing the effort to develop just the functional test.
This summarises how optimus works, in the upcoming blogs we will be talking about Optimus project template and internals of optimus.