Scrapy middleware using SeleniumBase's pure CDP mode

[nyg]

Hello all,

I've written a Scrapy downloader middleware that delegates requests to SeleniumBase. It only uses the pure CDP mode, allowing it to run on platforms without dedicated WebDriver builds, e.g., Raspberry Pi / AArch64.

The repository is nyg/scrapy-seleniumbase-cdp. Let me know what you think!

It is a fork of an existing repository (Quartz-Core/scrapy-seleniumbase) which did not use the CDP mode.

Usage example

class MySpider(Spider):
    # …

    async def start(self):
        # …

        async def maximize_window(browser: Browser):
            await browser.main_tab.maximize()
    
        yield SeleniumBaseRequest(url=url,
                                  callback=self.parse_result,
                                  wait_for='h1.some-class', # wait for this element to appear
                                  browser_callback=maximize_window, # interact with the Browser
                                  script={ # execute some JavaScript code
                                      'await_promise': True,
                                      'script': '''
                                          document.getElementById('onetrust-accept-btn-handler').click()
                                          new Promise(resolve => setTimeout(resolve, 1000))
                                      '''
                                  },
                                  screenshot=True) # take a screenshot of the page

Installation

pip install scrapy-seleniumbase-cdp

[mdmintz]

Very cool! Thanks for showing off the integration.