Our Thoughts
Anonymous
22 June, 2021
4 MIN READ

Cypress timeout – uncovering website performance issues

We were working on building a site that was a bridge between interested customers and the car dealers/OEMs. All the data related to a vehicle which will be useful for a customer was displayed on the website. The data from interested buyers was also captured through a form and shared with the relevant dealers/OEMs.

As the project was growing day by day, new features were getting added. So, manual regression testing was becoming a lengthy process before every release which happened twice a week.

The website was on ReactJS. We were using Cypress for automation because it is purely based on JavaScript and uses a unique DOM manipulation technique and operates directly in the browser. Cypress also provides a unique interactive test runner in which it executes all commands. We started automating the regression test cases according to the priority of the module. The regression suite was integrated with CI. Before every release, it was mandatory for the regression suite to pass in the CI.


Problem Statement 

There were two releases in a week, which meant that new features were added every time. Suddenly we started facing issues on CI and test cases started failing randomly with timeout errors. 

We tried running it multiple times but test cases that were passed earlier start failing, on every run we were seeing various sets of test cases failing which were passing before, the result was not the same in every run. 

We suspected that there was some issue with the Cypress tests, so we tried to replicate the issue on our local machine. But many test cases randomly started giving timeout errors on our local machines as well.


What is a timeout?

All the commands in Cypress have a default timeout that means Cypress will wait for a definite time for the command to get executed, all the DOM-based commands have a default timeout of 4 seconds.

Cypress timeout

For example cy.get(‘#firstname’), in this case Cypress waits up to 4 seconds for id to be present in DOM. Similarly for every assert command Cypress waits up to 4 seconds by default. It looks for the element up to 4 seconds and if it is not present, Cypress throws an error 'Timed out retrying after 4000ms: Expected to find element: #firstname, but never found it'.

We can modify the default timeout for a single command by passing a timeout parameter, for the above example if we want to add a 10 seconds timeout we can modify it as below.

cy.get('#firstname', { timeout: 10000 })

Else, we can provide a default timeout by adding "defaultCommandTimeout": 10000 in cypress.json which will by default add a 10-second timeout for all the commands.


Our Solution 

On looking at the timeout errors, our first approach was to increase the default timeout to 20 seconds, just to check if the test cases started passing. Yes, all the test cases started passing after we increased the default timeout.

How did we decide the timeout for our application?

Now, the next question was what should be the value of default timeout? We thought of checking the performance of our website. We found that the average time taken by all the pages to be interactive was around 8.5 seconds. 

Cypress timeout

When we increased the default timeout to 10 seconds in cypress.json our test cases started working on CI.

 

Conclusion

We were continuously adding new features to the site, due to which performance of the site was impacted, and our Cypress tests started failing because the time required for rendering the elements on the pages was high. The real issue was with the performance of the page. These reports were shared with the development team and it was decided to improve the performance of the websites in the next sprints. 

It's not always that your test needs to be fixed. We need to identify the root cause and provide feedback to the team. 

 

Anonymous