“Sorry Cypress” – a free (partial) alternative to Cypress Dashboard

It is almost 2022, and Cypress does not need an introduction anymore. Today, it is a well-known, mature e2e testing framework. With ~16000 commits and ~400 releases, Cypress is widespread and massively adopted by testers and developers across the industry. On the other side, a lot of development teams are still struggling with the implementation and optimization of their Continuous Integration pipelines. This is not the problem specific to Cypress in particular, but teams are often trying to avoid additional costs (of Cypress Dashboard) and trying to implement their own custom-made solutions for test parallelization. For this reason, I will give an insight into the capabilities of Sorry Cypress, and hopefully, make a – whether I want or not to invest in the Cypress Dashboard – decision a bit easier.

Deploying Sorry Cypress to AWS

The official Sorry Cypress documentation described the following installation methods:

• Docker/docker-compose
• AWS
• Google Cloud
• Microsoft Azure
• Heroku
• Kubernetes

Unfortunately, Microsoft Azure and Kubernetes installation methods are still “under construction”, so will have to wait for the official guideline. Out of the remaining four installation methods, I have chosen the AWS deployment for this demo. AWS deployment is done via provided Cloud Formation template, which makes the whole process quite simple. In the dedicated part of the documentation, you can find the “Launch Stack” button, that will take you to the AWS Cloud Formation wizard with the pre-loaded sorry-cypress-stack.yml file. Based on the documentation, there isn’t any additional configuration needed to deploy the stack, but in reality, be aware of the following:

• Stack name (not StackName!) needs to be unique, S3 bucket name will be the same so the default one won’t be acceptable (it already exists)
• the target group name cannot be longer than 32 characters, and since the name is created as stack name + “-director-tg”, your stack name can’t be longer than 20 characters

If you avoid these (or any other) issues, after ~5 minutes your Sorry Cypress instance should be up and running. As a compute engine, it uses AWS ECS on AWS Fargate with all necessary SC components load-balanced, and the S3 bucket is used to store test execution screenshots and videos:

A full SC setup contains the following components

• sorry-cypress-director service
• sorry-cypress-api
• sorry-cypress-dashboard
• MongoDB (to persist execution data)
• storage (to persist screenshots and videos)

Relevant URLs for the director, API, and a dashboard that we will need in the next section can be found in the Outputs section of the Cloud Formation stack:

Sorry Cypress in action

For the demonstration of the parallel test execution with SC, I will be using the default tests that are coming together with the Cypress installation. When it comes to client-side installation of the SC it is done with the following command:

npm install cy2 --save-dev

cy2 basically intercepts the Cypress commands and sets the API URL (director URL) that we provide through the environment variable:

export CYPRESS_API_URL="https://sorry.yourdomain.com/"

That’s it, now let’s run tests (without parallelization) and send results to SC just to see if it’s working fine:

npx cy2 run --record --key XXX --parallel --ci-build-id build_serial

If we go to the dashboard URL we should now see the demo_project (this needs to be set in cypress.json), and within that project, we should see our build (build_serial) with the real-time progress of our test execution:

If we want to inspect a failing spec file we can click on it, and we’ll get a report for a particular spec file with all test cases within, video recording of the execution, status, error, duration, attempts, and retries:

(note: I intentionally changed one test so it can fail for the purpose of this demonstration)

In order to troubleshoot further, we can click on a failed test, where detailed error and the fail screenshot can be found for every attempt:

Good, now let’s finally run our tests in parallel:

Now we can see which test was taken by which machine (Machine #), and we can see real-time both machines executing tests. Also, if we take a look at the total execution times for both jobs, we can see the expected difference:

In terms of settings, there isn’t a lot to be configured in SC. On the project level, the run timeout can be configured or GitHub/Bitbucket/Slack/Custom hook can be added.

Conclusion

If you are not a fan of on-premise solutions, there is a SaaS version of the Sorry Cypress called Currents.dev as well. Feature-wise, you will get the exact same set of functionalities, for the $US65/month (for 10 users and 50.000 test results per month package). I tried to make a pricing comparison between Sorry Cypress and Cypress Dashboard, but it was just not possible due to feature differences as well as user/test results ratio. These two tools are definitely having a different audience, and since Sorry Cypress will most likely target teams that want cheap, self-hosted Cypress dashboard and parallelization, Cypress Dashboard will target an audience that needs a couple more advanced features (analytics, flake management, etc.), but they’re willing to pay more as well.

Sorry Cypress is a decent, production-ready tool, that can serve basic dashboard/parallelization needs. If you’re willing to mess with your own infrastructure (or pay?), it will give you enough options and probably fit into your existing setup. The only thing that got me frustrated quite often during usage of SC, was the documentation. With a lot of “Under construction” areas and a lot of stingy sections, it can get hard to understand certain parts. If it’s still under the WIP flag, I would like the SC team to finish it and make the last step towards a great Cypress Dashboard alternative.

Links

• https://docs.sorry-cypress.dev/
• https://currents.dev/
• https://www.cypress.io/dashboard/