e2e testing framework in mako

[benjaminpaige]

As you may know we are going to be ramping up our e2e testing in mako. Currently in onemac cypress is implemented heavily, used in the pipeline and very familiar to the developers. In mako we have added playwright into the pipeline with some, but far fewer tests.

As we are deciding which framework to use this discussions is meant to serve as a conversation about the pros/cons of the two options. i.e have you had any experiences, positive or negative, with either Cypress or Playwright? What factors influenced your choice, and in what contexts did one tool outshine the other?

Our good friend Haywood has put together a very thoughtful analysis documenting some of the features and benefits of the two and offered his proposed solution here

i encourage you to give it a read.

Please feel free to share your thoughts, anecdotes, or even your personal preferences here in this discussion.

Looking forward to hearing your thoughts and experiences!

[charmcitygavin]

That's an excellent analysis of those platforms. Thank you, Haywood! I haven't used Playwright before, but have used Cypress on many projects over the years. I do agree that its strong community support is a strong selling point, and I agree that sticking with it is a good choice unless there's a Playwright feature that would improve our process.

[aalousi]

Thank you Haywood for putting this amazing analysis together and thank you Ben for getting this conversation started. I do agree that we should stick with cypress framework for the time being as i dont see any specific feature that can be done in Playwright and can not be done in Cypress. Strong community, and current QE skill sets aligns well with cypress as well. As far as Maco, i think we should keep what we have in playwright instead of redoing it in cypress but build cypres as part of the framework and write future test cases and scenarios in cypress. We may benefit from keeping playwright as part of the framework in the future if we ever run into specific scenarios that cypress does not solve and can be solved using playwright then we can utilize the foundation that we have. What are your thoughts?

[13bfrancis]

I would like to throw my name in the hat as it pertains to using playwright. I have used both and they are both solid choices, but would like to pitch for Playwright for the following reasons.

• Playwright selector syntax. Playwright has a syntax for writing selectors that very closely mirrors that of react-testing-library which we use for writing component test. Consistency across how we write test I theorize will make for less of a learning curve as it pertains to onboarding. Reference to selector docs here: https://playwright.dev/docs/locators#introduction
• Ability to use async - await syntax in Playwright. I find this to be a ergonomic improvement when it comes to working with selectors and makes it easier to write less flaky test. For example if I had just clicked a button and new text appears on the page I can write this statement and it will wait intelligently for the text to appear

await screen.findByText("The text has appeared");

• Reduction in flaky tests. Playwright has an out of the box solution to handle flaky tests and reduce the amount of "false positive failures" that happen. Reference to docs here: https://playwright.dev/docs/test-retries#introduction

[jdinh8124]

I'll throw my opinion in here. To me Playwright shines with its:

1. Multi-language compatibility

Supports multiple languages such as JavaScript, Java, Python, and .NET C#

I know we've shifted towards typescript/aws direction, but this leaves the door open for us to add Playwright to applications we inherit written in other languages and we will have transferable skills to those applications.

2. Support for multiple test runners

Mocha, Jest, Jasmine

3. Parallel execution & testing on multiple browsers at the same time:

Playwright’s architecture allows for parallel test execution and can execute tests in multiple browsers simultaneously, leading to faster test runs, especially when conducting cross-browser testing.

This will lead to faster builds in our CI/CD pipelines

4. Support for multiple tabs and multiple page testing scenarios

Each browser context can host multiple pages (tabs)

At the end of the day I can see arguments for either tool, just wanted to offer some points that could be helpful.

[benjaminpaige]

We have come to a decision regarding the e2e framework. Thank you all for your weighing in here, all of your input was considered when making this decision. We will continue to use the implementation of playwright that exists in mako today. This means there will be a learning curve for those who might be writing some of these tests as the framework is unfamiliar. But i know that folks like myself and the other front end developers that have experience with playwright will help to make the transition as smooth as possible by offering guidance, support and hands on assistance.

The plan is, lets define the 3-5 most valuable tests to have and get those scoped out and written. these would prob test features like a11y, new submission, rai withdrawal, and some of the other actions to ensure data flow integrity. This would be a huge value add and also give folks a chance to work with the patterns.

I look forward to working together to making this application a great one.

[mdial89f]

Recapping a meeting just held with most people in this thread:

Decision: Let's build a dev environment off master, provide the branch and url to @aalousi , and let @aalousi poke around, see how the site drives, trial moving things over, keeping Cypress. Then let's reconvene soon... I think we said possibly Thursday.