[HN Gopher] Playwright: Automate Chromium, WebKit and Firefox
___________________________________________________________________
Playwright: Automate Chromium, WebKit and Firefox
Author : cyrusmg
Score : 164 points
Date : 2022-01-26 08:12 UTC (1 days ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| brimstedt wrote:
| Does anyone know how it compares to NightwatchJs?
|
| Br
| tw20212021 wrote:
| Is there an open source web testing tool which also integrates a
| dashboard, keeps track of test runs, creates reports, something
| that I can just install on a vm and run to test a web app?
| hoten wrote:
| Give Lighthouse CI a shot.
|
| https://github.com/GoogleChrome/lighthouse-ci
| syspec wrote:
| Interesting tidbit:
|
| One of the main contributors of this project[0], was the core
| contributor (creator?) of Puppeteer[1], but then I guess left
| Google to join Microsoft and work on this[2][3].
|
| [0] - https://github.com/aslushnikov
|
| [1] - https://github.com/puppeteer/puppeteer/
|
| [2] - https://github.com/microsoft/playwright/graphs/contributors
|
| [3] - https://github.com/microsoft/playwright/graphs/contributors
| vlunkr wrote:
| I wonder what the story is there. Why wouldn't MS just have him
| continue to work on Puppeteer? They're both open source, so
| there's not much point in "owning" their own clone of it.
| dzhiurgis wrote:
| I'm exaggerating but Playwright vs Puppeteer is a bit like
| comparing Puppeteer with Selenium
| fikama wrote:
| From what I know puppeter works only with chromium, this
| could be deal breaker for microsoft
| abraham wrote:
| Puppeteer has experimental Firefox support.
|
| https://pptr.dev/#faq:~:text=What%20is%20the%20status%20of%
| 2...
| jahewson wrote:
| Edge is chromium though.
| zamadatix wrote:
| Microsoft's web products run on more than Edge.
| ithrow wrote:
| For generating PDFs like invoices in a webapp, is libraries like
| this the way to go these days or is still using a pdf lib the
| norm?
|
| Pros of Playwright/Puppeteer:
|
| Reuse existing HTML/CSS knowledge
|
| Cons: Requires an external service or shelling out to an external
| process
|
| Pros of using a pdf lib:
|
| Probably better performance, simpler architecture by being in-
| process.
|
| Cons: Ad-hoc language for designing the PDF.
| [deleted]
| rudasn wrote:
| It's very simple to use either, there are loads of example
| implementations on GitHub.
|
| I used one based on docker, and the bottleneck was actually
| sending the html, css you want to print (if it's not already
| served over http). I used a shared docker volume to write to
| from one process (python) and read from another (the node
| pupetter).
|
| It all comes down to, load html, wait to load, save to pdf.
| Very simple, fast, and reliable. More so than weasyprint for
| example.
| robstain wrote:
| Anyone tried BotCity?
|
| https://botcity.dev
| petermd wrote:
| kundi wrote:
| Does it support screencast - video recording of the browser with
| audio?
| mxschmitt wrote:
| It supports video recording (without audio), screenshots, and
| post mortem recording which is called Tracing.
| hoten wrote:
| Having worked with these folks back in Chrome, it's been great
| seeing this project continue to be successful. Great job!
| naasking wrote:
| Anyone have experience with Playwright compared to Selenium? I
| have a fairly large test suite and Selenium produces constant
| false positive errors, typically due to various timeouts that
| seem fundamentally unsolvable when running it from .NET. It's
| just very finicky.
|
| I don't know if it's Selenium specifically or some problem with
| the .NET binding, but I figure Microsoft must have better .NET
| integration so it will at least eliminate that possible source of
| problems.
| Bilal_io wrote:
| I tried selenium then playwright for a .Net project, selenium
| wasn't easy to work with. Playwright was good but for some
| reason which I don't recall exactly (could have been because it
| had to redownload chromium everytime we deployed). I ended up
| switching to puppeteer and I ended up very happy with it.
| simonw wrote:
| On paper Playwright should be a LOT better - it's taken a
| similar approach to Cypress, where everything is designed
| around the need to reduce flaky tests.
|
| In Playwright that manifests itself as the "auto-wait" feature:
| https://playwright.dev/docs/actionability
|
| You can do this kind of thing with Selenium too but it wasn't
| designed in from the very start of that project.
| tmcneal wrote:
| I'm not sure if any of these are pertinent to your tests, but
| these are the issues I see most often that cause flaky tests:
|
| - Hard-coded waits in your code, like "Thread.sleep(1000)". A
| better alternative is to replace hard-coded waits with
| something that waits for an element or value to appear on the
| page. i.e. click on a button and wait for a 'Success' message
| to appear. Puppeteer and Playwright both have good constructs
| for doing this.
|
| - Needless complexity in the tests. Conditionals in particular
| are a code-smell and indicate there's something needlessly
| complex about the test.
|
| - No test data management strategy. The more assumptions you
| can make about the state of your application, the simpler your
| tests become. Ideally tests are running in an environment that
| nothing else is touching, and you're seeding data into that
| environment before tests run. I personally don't believe in
| mocking data in regression tests since that quickly becomes
| hard to manage.
|
| We spend a lot of time thinking about these issues at my
| company and wrote a guide that covers other common regression
| testing issues in more detail here:
| https://reflect.run/regression-testing-guide/
| naasking wrote:
| > - Hard-coded waits in your code, like "Thread.sleep(1000)".
| A better alternative is to replace hard-coded waits with
| something that waits for an element or value to appear on the
| page.
|
| We don't do any timed waits, all of our waits are for an
| element or value to appear, but these waits never complete
| sometimes, non-deterministically. We then added a long 5 min
| timeout on these waits because we know the test will never
| complete at that point. It's always fine in manual testing
| though, and if we don't run the browser in headless mode and
| watch it work. Very frustrating.
|
| Sometimes the HTTP requests themselves timeout after a few
| minutes, but this never happens in manual testing either.
| That's actually the most common issue these days, and this
| happens non-deterministically too. This is what I meant by
| "flaky".
| mattlondon wrote:
| I have had similar issues with selenium via other languages too
| - it is generally pretty flaky. E.g. saying a button or some
| other element doesn't exist when it clearly does.
|
| With great care and effort you can make your tests reliable
| (especially if you are happy to allow a "best of 3" type test
| strategy to allow for 1 flake and 2 passes) though. Prodigious
| use of the wait (i.e. stdlib polling) primitives seems to give
| you the most bang for your buck.
|
| I am note sure if this is just the nature of web automation, or
| if selenium is just crap? My gut is to say it is selenium's
| fault since we never get the same issues when using javascript
| in the DOM or in an extension)...maybe it is the browser APIs I
| guess? U have no idea but if this playwright is any better than
| that would be superb.
| i_like_apis wrote:
| While I have not used Playwright (but have a lot of experience
| with Se), I would say the code style is refreshing:
| // Expect an element "to be visible". await
| expect(page.locator('text=Learn more').first()).toBeVisible();
|
| Writing await for every action makes the timeout of the action
| seem more explicitly declared. There seems to be more granular
| control of timeouts as well https://playwright.dev/docs/test-
| timeouts
|
| > I don't know if it's Selenium specifically or some problem
| with the .NET binding
|
| If the execution in .NET is slow then I suppose it could be
| .NET. But it could be (and often is) the suite design. You must
| wait for /everything/ before interacting with it because the
| code execution is quicker than the page.
|
| Large Se/Webdriver suites are often a PIA. I find it's nice to
| write them with Python or Ruby so they can be debugged
| interactively with the an interactive shell.
| naasking wrote:
| > If the execution in .NET is slow then I suppose it could be
| .NET. But it could be (and often is) the suite design. You
| must wait for /everything/ before interacting with it
|
| That's what I do, but the wait for an element in certain
| tests times out after a few minutes, even though the elements
| are clearly visible, and manual use never has an issue.
|
| From other comments it sounds like Puppeteer and Playwright
| are better on this, so will look into switching.
| gmokki wrote:
| When I fixed many similar selenium/webdriver tests the root
| cause was always the same: You grab reference to an element
| and for example wait it to become enabled or some text to
| appear. But your ui framework actually replaces the element
| in the dom while doing its thing and your reference to
| stale element will never change. Fix is to loop searching
| the element with selector and check if the element fills
| the conditions. If not, retry from search again. We had
| nice helpers for those and had very stable selenium tests.
| londons_explore wrote:
| I think it's possible to write tests in selinium which are
| time-independant... Eg. "Wait for element #foo to exist".
|
| You can also give the browser a virtual clock so that you can
| use time based timeouts and give every test a timeout of 3
| hours, but those 3 hours only take milliseconds in real time.
| That approach gets CPU expensive if your site has any
| background polling scripts or animation, because obviously the
| animation will end up animating a lot during the test!
| naasking wrote:
| > I think it's possible to write tests in selinium which are
| time-independant... Eg. "Wait for element #foo to exist".
|
| Yes, this is what I've done but the elements non-
| deterministically do or do not appear according to selenium,
| and then the wait times out. This happens for dozens of tests
| across dozens of pages with no issue with manual use, so
| something fishy is going on.
| apatheticonion wrote:
| The only thing I wish we had was remote browser access - so I
| could run my tests on a VM (like within a docker image) and use a
| browser on the host.
|
| We use TestCafe at work for this purpose. I personally hate
| TestCafe as it's is an absurd unfocused mess of a browser remote,
| but it lets me control my browser by navigating to a URL which no
| other browser remote system does.
| defied wrote:
| I'm working on a project that provides remote browsers, running
| on VMs/containers, capable of running Playwright tests (and
| Puppeteer scripts): https://headlesstesting.com/
|
| We've seen a consistent growth of interest in people wanting to
| use Playwright for browser automation (and testing).
| headlessvictim2 wrote:
| Off-topic, but our freemium website is under attack by headless
| browsers.
|
| The freemium service provides access to compute-heavy machine
| learning models running on GPUs.
|
| Hackers blast 50-100 requests in the same second, which clog the
| servers and block legitimate users.
|
| We reported IPs to AWS and use Cloudflare "Super Bot Fight Mode"
| to thwart attacks, but the hackers still break through.
|
| We don't require accounts, but could impose account requirements
| if this helps.
|
| Any suggestions?
| rabi_molar wrote:
| Perhaps Captcha?
| headlessvictim2 wrote:
| Thanks for the suggestion.
|
| It is possible, but this degrades the experience for
| legitimate users.
|
| We prefer solving this without impacting/taxing normal users
| if possible.
| slig wrote:
| Just block the AWS ASN on CF, it's nor worth fighting.
| rob-olmos wrote:
| +1 and GCP, and many other hosting ASNs
| forgotmyoldacc wrote:
| Why not ReCAPTCHA?
| headlessvictim2 wrote:
| Thanks for the suggestion.
|
| It is possible, but this degrades the experience for
| legitimate users.
|
| We prefer solving this without impacting/taxing normal users
| if possible.
| readyplayeremma wrote:
| Just add the captcha only for requests coming from the
| problematic ASNs, like AWS.
|
| edit: Actually, since you use CF, just make a firewall rule
| that forces the captcha for those ASNs before it even gets
| to your app. They have a field named "ip.geoip.asnum" for
| that, and an action called "challenge" which will force a
| captcha.
| [deleted]
| synergy20 wrote:
| what's your site? would like to play with it
| cmeacham98 wrote:
| 100 requests/second isn't that much, especially if you're
| fronting your website with Cloudflare. Do you have some
| unauthenticated endpoint(s) that eat up a ton of server CPU?
| headlessvictim2 wrote:
| Thanks for the reply!
|
| The freemium service provides access to machine learning
| models on GPU instances, served with FastAPI.
|
| Each request invokes a compute-intensive ML model, but
| perhaps there is something wrong with the FastAPI
| configuration as well?
| tempest_ wrote:
| It could be.
|
| I watch the FastAPI repos a lot and tones of people do not
| understand how async python works and put their models with
| sync code in an async context.
| headlessvictim2 wrote:
| Consider us one. :)
|
| We tried removing "async" -- thinking it would force
| sequential processing -- but it unexpectedly seemed to
| cause parallel processing of requests, which caused CUDA
| memory errors.
|
| Before removing "async", this is the weird behavior we
| observed:
|
| * Hacker blasts 50-100 requests.
|
| * Our ML model processes each request in normal time and
| sequentially.
|
| * But instead of returning individual responses
| immediately, the server holds onto all responses --
| sending responses only when the last request finishes (or
| a bunch of requests finish).
|
| * Normally, request 1 should return in N seconds, request
| 2 in 2 _N seconds, but with this, all requests returned
| in about N_ 50 seconds (assuming batch size of 50).
|
| 1. Any suggestions on this?
|
| 2. Mind clarifying how sync vs aync works? The FastAPI
| docs are unclear.
|
| Any help would be much appreciated.
|
| This has been extremely frustrating.
| azinman2 wrote:
| What are the bots goals? Curious
| austincheney wrote:
| Browser automation will occur by executing events in the DOM or
| by calling properties of the page/window. It's all JavaScript
| designed for user interaction executed by a bot.
|
| The one event that cannot be automated is cursor
| movement/position. Put a check into your event handlers that
| check that the cursor is actually over the event target.
| darkstar999 wrote:
| That sounds like an accessibility problem.
| austincheney wrote:
| Use an alternate control for keyboard navigation that is
| visually hidden and is accessed only by tab focus.
| headlessvictim2 wrote:
| This is interesting. Thanks for sharing.
|
| Are you saying block form submission unless the cursor is
| over the event target?
|
| If so:
|
| * How to handle legitimate requests from mobile users?
|
| * How to handle form submissions with the "return" key?
| austincheney wrote:
| Mobile users will use touch events instead of click events
| and likely your interface will be different and the screen
| width will be different. Check for these things along with
| keywords from the user agent string to determine mobile
| users from other users.
|
| Return key on a control in a form will fire a submit event.
| Check for cursor position in your submit handler.
| forgotmyoldacc wrote:
| Benefits of Playwright over Puppeteer - official support for
| languages outside of JavaScript, and official codegen/record
| support. Great!
| vthommeret wrote:
| Reposting my previous notes on Playwright
| (https://news.ycombinator.com/item?id=30060135):
|
| I just want to plug Playwright by Microsoft as I've been using it
| over the past month and have had a really great experience with
| it: https://playwright.dev It's built by the founders of
| Puppeteer which came out of the Chrome team. Some things I like
| about it:
|
| 1. It's reliable and implements auto-waiting as described in the
| article. You can use modern async/await syntax and it ensures
| elements are a) attached to the DOM, visible, stable (not
| animating), can receive events, and are enabled:
| https://playwright.dev/docs/actionability
|
| 2. It's fast -- It creates multiple processes and runs tests in
| parallel, unlike e.g. Cypress.
|
| 3. It's cross-browser -- supports Chrome, Safari, and Firefox
| out-of-the-box. 4. The tracing tools are incredible, you can step
| through the entire test execution and get a live DOM that you can
| inspect with your browser's existing developer tools, see all
| console.logs, etc...
|
| 5. The developers and community are incredibly responsive. This
| is one of the biggest ones -- issues are quickly responded to and
| addressed often by the founders, pull requests are welcomed and
| Slack is highly active and respectful.
|
| My prior experience with end-to-end tests was that they were
| highly buggy and unreliable and so Playwright was a welcome
| surprise and inspired me to fully test all the variations of our
| checkout flow.
| [deleted]
| johnnypangs wrote:
| What do people thing of playwright vs cypress? I've been
| considering using playwright instead as it supports more browsers
| and I feel like it's easier to do production monitoring (by
| putting it in a aws lambda or using checkly)
|
| - Cypress: https://www.cypress.io
|
| - Playwright aws lambda:
| https://github.com/PauloGoncalvesBH/running-playwright-on-aw...
|
| - Checkly: https://www.checklyhq.com
| machiaweliczny wrote:
| Playwright is definitely better IMO. Cypress is overengineered.
| felipellrocha wrote:
| I wouldn't say cypress is over engineered. Just a byproduct
| of its time.
| CSDude wrote:
| - Cypress does not run on M1 natively.
|
| - Playwright is more lightweight. Can be good or bad on what
| you expect. But I definitely prefer Playwright.
| cebert wrote:
| On my team we evaluated Cypress and Playwright and landed on
| Playwright. Some features it has that Cypress didn't was Safari
| support, support for cross domain tests, and support for tests
| that need to open multiple browser windows (for testing
| collaborative editing).
| cornedor wrote:
| We recently picked playwright for a project because we could do
| more with the "mouse". With playwight you can click coordinates
| on the page. I did not find an easy way to do this using
| cypress. Aside from that, playwright seems to be so much
| faster.
| tnolet wrote:
| We see a strong adoption of Playwright. It's the default we
| recommend to users now. We also support Puppeteer, but its
| development is lagging.
|
| Having said that, I would love to support Cypress if I had
| infinite time and focus.
|
| Side note: Selenium is not on the menu, even with its large
| install base.
|
| We are aiming for where the puck is going and it's going to
| Playwright.
|
| Full disclaimer: I'm CTO at Checkly.
| hugs wrote:
| Why was Selenium off the menu?
| cebert wrote:
| Selenium is slow and based on webdriver. I think most
| consider it legacy at this point. Most new projects tend to
| use Playwright, Cypress, or Puppeteer for E2E tests. All
| three options are much more performant and reliable.
| synergy20 wrote:
| I saw puppeteer is actively developed, wonder where the
| 'lagging' implies.
|
| doesn't playwright use puppeteer for chromium-based
| browsers(even edge-based), I thought it's just a wrapper for
| puppeteer with extra support for firefox.
| djbusby wrote:
| The checkly folk recently published this
|
| https://blog.checklyhq.com/cypress-vs-selenium-vs-
| playwright...
|
| Cypress vs Selenium vs Playwright vs Puppeteer speed
| comparison
| twohaibei wrote:
| I highly recommend codeceptjs. After 4 years of using test cafe
| for e2e testing, codecept has proven to be much more pleasant
| to use.
| dvngnt_ wrote:
| I love Cypress. there are definitely limitations though like
| iframe support or visiting two separate domains, tab support
|
| https://docs.cypress.io/guides/references/trade-offs#Permane...
| [deleted]
| simonw wrote:
| Electron appear to have dropped support for their previous
| automated testing framework Spectron -
| https://github.com/electron-userland/spectron/blob/master/RE... -
| and now suggest Playwright as an alternative:
| https://www.electronjs.org/docs/latest/tutorial/automated-te...
| and https://playwright.dev/docs/api/class-electronapplication/
| eatonphil wrote:
| WebDriver or Playwright. I switched from Spectron to selenium-
| webdriver.
| Karupan wrote:
| Playwright is great, especially if you are dealing with test
| cases that span multiple domains/contexts. I had to test some
| user flows which involved logging into two apps, each with three
| different users to perform and validate various actions.
| Playwright's context switching made it a breeze. Also, it offers
| a nice separation of browser automation and test runner API, so
| it can be used outside of E2E testing as well.
___________________________________________________________________
(page generated 2022-01-27 23:00 UTC)