Webinar: Test Orchestration using HyperExecute

The speed at which tests are executed and the “dearth of smartness” in testing are the two major problems developers and testers encounter.

Traditional cloud architecture is slower than a local infrastructure. On the other hand, local configurations lack the features that the cloud can provide. A trade-off has always existed. We were certain that there had to be a way to combine the best capabilities of both and improve it.

And here, we introduce HyperExecute, a cutting-edge smart test orchestration platform that enables companies to conduct end-to-end automation tests at the fastest speed possible. It is up to 70% faster than any online Selenium Grid. Businesses may now orchestrate tests rather than execute them.

Businesses can reduce test execution time using HyperExecute to achieve the shortest time-to-market. With HyperExecute, our ultimate goal is for customers to concentrate on creating test cases while leaving the rest to us.

How to accelerate faster feedback from the developer end? Let’s deep-dive into the webinar to know more about the latest offering by LambdaTest, HyperExecute.

Hey! Do you know? random-data-from-regexp — Free online tool to generate random data as per the written regular expression.

About the Webinar

In this webinar, our special guests, MAYANK BHOLA, Co-founder & Head of Engineering at LambdaTest, and HARSHIT PAUL, Product & Growth Manager at LambdaTest, discuss HyperExecute’s world-class features, test orchestration and how is it different from test execution.

You can visit our LambdaTest YouTube channel for more videos on LambdaTest Webinars, responsive testing, mobile testing, and Cypress testing.

Let’s have a look at the webinar agenda before we get into the details of the webinar.

Introduction to LambdaTest

Harshit Paul kickstarted the webinar by pitching in the overview and features of the LambdaTest platform.

LambdaTest is one of the leading cloud platforms that allow you to perform browser compatibility testing on an online browser farm of 3000+ browsers and operating systems. LambdaTest also provides a real device cloud to perform real-time app and automation testing over real devices running on real operating systems.

LambdaTest Offerings

Currently, LambdaTest provides the below offerings:

Live Testing: With LambdaTest, you can perform Real-time Cross Browser compatibility testing of your website by entering the required URL. The key features include:

• 3000+ Live test environments (Desktop and Mobile Browsers)

• Native App Testing on Emulators and Simulators

• Fastest responsive debugging with LT browser

Automation Cloud: You can run Selenium, Cypress, and Appium automated tests on reliable, scalable, and secure cloud-based infrastructure by cutting down your test execution time significantly.

• Fastest and highly reliable Selenium Grid on cloud

• Run tests on mobile and desktop browsers

• Headless Testing (Chrome and Firefox)

• Advanced reporting and analytics

Real Devices Cloud: You can perform Real Device testing of your apps on Android and iOS devices:

• Manual and automation app testing

• Manual and automation browser testing

• Support for Appium Framework

• Support for PlayStore, AppStore, TestFlight

• Camera and GPS access

• Debugging logs

HyperExecute: HyperExecute is a blazingly fast test orchestration cloud that helps businesses run end-to-end Selenium tests at the fastest speed possible.

• Run tests at the speed of your local grid

• Windows, macOS, and containers support

• 60+ regions supported

• Matrix-based multiplexing

• YAML-based workflow

• Auto-split of tests

Supported Frameworks

The following are some of the major frameworks supported on the LambdaTest platform:

• Selenium

• Cypress

• Playwright

• Puppeteer

• Appium

• TestCafe

Integrations Landscape

You can synchronize your LambdaTest account with various project and bug management tools and streamline your workflow. LambdaTest lets you move your bug directly to your project management dashboard with just one click and understand the complete project execution timeline from updating requirements to software development completion.

The following integrations are supported by the LambdaTest Platform:

HyperExecute — Blazing fast test execution platform

Mayank then starts deep-diving into the HyperExecute. He claims that HyperExecute is an end-to-end test orchestration platform supporting the major operating systems (Windows, Mac, and Linux containers) that allow running end-to-end test cases over the LambdaTest platform.

The main objective of HyperExecute is to meet the local execution speed and provide the best features of the cloud without the need to maintain or manage any of the infra resources.

HyperExecute provides a similar kind of convenience without compromising on any of the features you were getting before. On top of that, it has a smart CI feature layer and a fast, blazing speed that will meet your local execution. Before getting into detail, Mayank explained why traditional end-to-end testing platforms(solutions) are slow?

If you’re using a cloud-based test execution platform, it takes several hops for your web browser and the command to reach out to the target website while performing Selenium automation testing.

Example: If you are in India and wish to test a website hosted in Australia, the script running on your local system will send some HTTP-based requests known as Selenium APIs. To make the services much more horizontally scalable and regionally accessible, these APIs are being handled by a component called a hub that receives all commands and schedules a virtual machine running a web browser geographically separated from this hub component.

All the commands received from the client script will be transmitted by the hub back to the virtual machine hosting the web browser. And finally, that web browser is going to access your target website. For every single command, it has to traverse through multiple hops. There are hundreds of commands in a typical end-to-end Selenium test case and hundreds of test cases in a standard test suite.

If you have to run multiple test cases at once, waiting for the network response increases the flakiness of these test cases. If any one of the connections between these two hubs is degraded, the entire test suite can fail. The code, website, or platform are the three possible factors for connection interruption. This results in test execution over the standard solutions much slower when compared to local systems.

Let us discuss the local systems. As a developer or tester executing an end-to-end Selenium test case, all components like your script, the hub, the Selenium components, and the web browser reside in a single machine (laptop). Once you execute the test case, everything happens natively. This is why we decided to simulate the exact similar execution environment over the cloud. In HyperExecute, all the components required to run the test case were part of a single virtual environment. This results in three-fourths of the hops being collapsed into one. The only network going out is the website. This results in high-performance benefits of up to 60%.

HyperExecute allows you to use and optimize the localhost environment with utmost ease. It was not just adding up everything on the cloud but being smart and intelligent in our execution. The orchestration layer was added on the top. Compared to traditional clouds, the components are re-engineered to have 70 to 80 percent of performance benefits. But in reality, traditional clouds are no competition.

We want to meet the local execution speeds that can be achieved using HyperExecute. Considering the optimizations done over the system, where we try to run multiple test cases sequentially on a local system, the time between consecutive test cases can range from four to five seconds. The reason is that the browser used to execute the test case that needs to shut down and start a new instance. During this time, four or five seconds are available on your local system. Still, with HyperExecute, we have broken this wall and achieved the latency between stopping the last test case and starting a new test case up to 300 milliseconds.

HyperExecute allows you to reduce the end-to-end feedback time. You can write the test cases and leave the rest to HyperExecute. Let us consider the cycle that a developer or QA does when trying to test out something. They author the test case and then orchestrate those test cases. Orchestration is about deciding how many test cases and which test cases you want to run on which concurrent workers. So, in today’s world, nobody runs the test cases sequentially. If they have hundreds of test cases, they most probably do a parallel execution of their test cases.

But during the parallel execution, either you rely on an automation testing framework to distribute your test cases, or if you want to take manual control, then you have to write your own code to orchestrate it. After orchestration, you must execute a test case and fire off that command to a remote cloud provider like the language standard grid to maintain the entire infrastructure. Finally, once the test cases are executed over that infrastructure, you need to deprovision that infrastructure if you are manually doing it or you need to call the cleanup scripts and finally collect the data.

Mayank further claims that, apart from writing the test cases, there are other activities that people are forced to spend their time on. This is where he explains that HyperExecute wants to take away all the control. He wants their users to focus on only crafting their test cases. He does not want the users to focus on infra provisioning, how to orchestrate, how to execute them efficiently, and how to collect back all the data because that will prove to be a waste of time.

Hyperexecute provides a very simple onboarding for the end customers, achieved using a YAML-based workflow. A YAML file guides what needs to be run, what needs to be tested, and then they download our CLI binary. When they hit our CLI binary, almost everything is abstracted. So, when you write a test case, you just create a YAML once, and for every change, you just hit the CLI boundary. CLI allows you to create and provision the fresh infra, orchestrate, execute the test case, and bring the final results back to you. This is how it promises to change the market related to end-to-end execution where it wants the users to focus on writing better.

Do you know? difference-checker — Free online tool to compare and check the difference between two text files.

HyperExecute Execution Demo

Mayank further explains about the execution. He claims that they have created an excellent onboarding page in which the creation of YAML is also simplified for the end-users. They have also put up various examples of popular testing frameworks.

Let’s say that you want to generate a YAML file for existing test cases, you could choose the language in the framework, and then you have to answer some questions on the left. Your YAML is getting ready on the right with every change on the left-hand side. You can just copy this YAML file and put it inside the root directory of a test folder and you are done. It supports Windows and Linux machines for trial accounts, while Mac is available only for paid accounts. As a free user, you can run over test cases where you will be assigned two virtual machines or containers or a mix of both based on your preference.

HyperExecute provides very fine-grain control over your testing durations. Let’s say that an organization has a policy that no test should run for more than 10 minutes. In that case, a timeout can be set up in HyperExecute. Any test case that takes more time in execution (more than 10 minutes) will be aborted. You get quick feedback on test cases that breach your SLA.

It provides retry functionality where you can retry only fail test scenarios. If you are not using the standard plugins or framework and want to roll out your testing framework, that can also be integrated. HyperExecute provides no tight coupling with any language or framework. It is very language and platform agnostic. It supports Selenium, Cypress, Playwright, Puppeteer, and Taiko.

HyperExecute provides the complete and underlying infrastructure. You do not have to focus on writing web test cases if there are other scenarios in which you want access to underlying system properties like registry settings in Windows. Once the YAML is ready, you need to download the YAML into your root file and then trigger our CLI.

There are directives listed in the YAML file, such as autosplit, concurrency, etc. These parameters allow you to customize the test execution. On hitting the CLI, a unique link is generated to track the progress of that particular job. So if you love the command line, this CLI provides excellent insights in real-time.

This is the main screen of the HyperExecute dashboard. The HyperExecute test cases are segregated into two virtual machines. All of these logs are segregated similarly. The logs are channelized according to different stages.

In this section, you can only find logs for the pre-steps; there are no other logs. If your pre-step fails, you know these are the limited lines you need to look at, and you do not have to focus and worry about all the lines of code. Secondly, for every single test execution, the logs are segregated. The logs are provided in real-time for every test case. The browser-related logs are not complete input for your pricing or debugging session, and you need to correlate these with the terminal logs you used on your system or CI system. With HyperExecute, you do not have to jump between different platforms, and you have everything in a single place. You can view all the browser-related logs in one place.

More About HyperExecute

HyperExecute provides you with the complete underlying infra. You can run any process, access any operating system setting, and run any command you want, just like CI systems, making it easy for the end customers to customize your testing environment. There are three steps followed in the YAML file that allows users to run some commands before and after the test cases.

Users can install certificates at run time, run MySQL instances, generate analytics after test case execution, and install a desktop application while the test cases are running. Apart from this, smart dependency caching is available to understand the languages and has its package managers for Python and npm for node JS. The runtime packages take a longer time to download. So, the platform caches them according to manifest files. When executing the test case again, you have the same manifest file, which is not changed, and a pattern will automatically cache all the dependencies.

Automatic Report Generation in HyperExecute

Customers spend a lot of time customizing their reports. The platform should be intelligent enough to provide the most insightful reports. HyperExecute has its native reporting that is insightful.

The above page screenshot is a downloadable HTML report that is generated by HyperExecute for every single job. The Feature Summary, Test Summary, and Scenario Summary are visualized in detail. Task analytics also shows how your test cases were distributed between different machines. At the end of every job, you can download this HTML report via CLI or the dashboard.

Artifice Management

All of the test execution creates some form of output files. Sometimes, these are videos, JSON files, or CSV files. Since the splitting of the execution is done across different nodes, customers can generate and manage these artifacts automatically. So, if your test scripts generate some artifacts, the HyperExecute platform will automatically pack them in a consumable package that can be downloaded using CLI or APIs.

Deep Analytics

This feature ensures that if any test case fails, you are not required to read the log files. The machines and platforms will automatically parse the logs and provide insightful details without the user spending any effort. The type of error is visible at the top-right without reading the logs. This provides the next level of user experience.

Orchestration Features

There is a difference between orchestration and execution platforms. HyperExecute dynamically distributes all the test cases across the available nodes in the most efficient manner. The entire history of test executions is known where the exact time and parameters are given. By using the deterministic analysis, scheduling these test cases on the nodes is done most efficiently.

Running the test cases makes the system more intelligent. There is a scenario where you need not run every test case or rerun the job. So HyperExecute provides the feature to run the failed test case and failure scenarios that can be specified in the YAML file.

HyperExecute provides this functionality or feature across all the languages and frameworks. This helps you to test the flakiness of your test cases. Automatic re-ordering can be explained by an example where out of 100 test cases if your 99th test case fails, you go back to fix it. The job is executed again.

HyperExecute smartly reorders the test cases so that the 99th test case runs first and then is followed by other test cases. The previously failed test cases are scheduled to the top automatically to get faster feedback. HyperExecute matrix strategy is all about defining all the permutations and combinations. It runs each of the permutations as a single machine.

HyperExecute can be easily integrated with any CI platform because it takes only two commands:

• Download CLI from a public URL

• Hit the CLI

This is all it takes to integrate HyperExecute with any of the existing CI platforms.

Deployment Options

The modern architecture of HyperExecute allows it to ship and run on any of the major cloud platforms. There are three different options:

• Shared Cloud

• Dedicated Cloud: Whole infra will be dedicated to your organization

• On-Prem: Supporting Azure GCP

HyperExecute is much more compact than other solutions, and we can ship it to various other cloud providers.

Click on the link to get A Definitive Guide To Regression Testing.

Wrap Up — About LambdaTest

LambdaTest is a cross-browser testing platform. There are different offerings as part of LambdaTest:

• You can select from 3000 browsers or operating systems. Choose the required browser and add an extension into your browser if it is part of your testing scenario.

• You can switch environments easily.

• You can mark a bug.

• We have a variety of integrations supported on the platform.

• You can change geographies and screen resolutions as well.

• App testing provides emulators and simulators that allow you to upload your app or test using a URL.

• Live interactive testing on real devices or automate them using the preferred framework.

• Analytics and archives help you to analyze huge test suites and build in the past.

• The handy utility is known as the LT browser. This is a free utility that helps to test your mobile website in a side-by-side view over different mobile devices.

• Automated screenshot testing where you can capture full-page screenshots of your website or mobile app.

• Smart comparison testing.

If you are working on an open-source project, make sure you claim your free access to HyperExecute or LambdaTest as a whole.

At the end, Mayank answered a few questions from the audience.

Q&A Session

How reliable is the Taiko Testing Tool?

Mayank: Taiko is not used by many people, but it is a simple CBD-based testing tool. HyperExecute supports it because of its simplistic nature. Taiko lacks some interesting features but has a huge opportunity and simplifies various different use cases. We are trying to support as many use cases as possible with HyperExecute.

Is it possible to download the produced Cucumber JSON files and XMLs using HyperExecute API?

Mayank: Yes, this is what is done by comprehensive artifact management. You can download all the generated files using an API and also use your own custom reporting logic. Many customers use it for the same purpose already.

Will HyperExecute be available in LambdaTest automation products in the future, or is it planned to keep it a completely separate product?

Mayank: HyperExecute is a completely different product, but it is tightly integrated with the entire LambdaTest. So using it as a separate product will not cause any downsides. HyperExecute license provides access to a standard platform as well.

Does HyperExecute support app automation?

Mayank: Soon, not now. But as we launch the support from emulators, simulators in mobile devices, then definitely yes.

While running the Salesforce scripts with any automation tools, usually, the browser opens a new incognito window to execute the script. In this case, Salesforce will send an OTP for an email to log in, but there is an option in Salesforce to add a safe or whitelist an IP address so that it won’t ask for an OTP further. How can the OTP method be skipped in LambdaTest, and what is the IP address for LambdaTest execution to add to Salesforce for whitelisting?

Mayank: Depends on the platform that is being used. If you are using standard automation, we do have a way to create a dedicated proxy for you that will have a single IP address. So, you can ask customer support to create a dedicated proxy and then provide the IP address. In HyperExecute, we provide standard netting solutions for every different organization. So, once you are on board and sign up, three different IP addresses can be safely whitelisted. This allows you to skip the OTP method.

I hope you enjoyed the Webinar!!

We hope you learned something from this webinar. In case you missed it, the complete recording is accessible on the LambdaTest YouTube channel. To anyone who is in need of HyperExecute, forward this post. Keep informed about the latest events in the test automation industry. Sign up for Coding Jag and our weekly newsletter.