Skip to content

feat!: align enqueue_links URL filters and limit with Crawlee JS - #2375

Open
vdusek wants to merge 1 commit into
v2from
feat/enqueue-links-js-alignment
Open

vdusek wants to merge 1 commit into
v2from
feat/enqueue-links-js-alignment

Conversation

@vdusek

@vdusek vdusek commented Oct 31, 2026

Copy link
Copy Markdown
Collaborator

Aligns include, exclude and limit in enqueue_links, extract_links and add_requests with Crawlee JS:

  • Glob patterns match case-insensitively.
  • Regular expressions match anywhere in the URL (re.search). Use ^ to anchor them.
  • limit is applied after all filters, transform_request_function and the max_crawl_depth check, so links the transform skips don't use it up.

SitemapRequestLoader uses the same matching through a shared check_url_patterns helper in _utils/globs.py. The upgrade guide describes each change.

Closes: #2122

✍️ Drafted by Claude Code

@vdusek vdusek added the t-tooling Issues with this label are in the ownership of the tooling team. label Oct 31, 2026
@vdusek vdusek self-assigned this Oct 31, 2026
@codecov

codecov Bot commented Oct 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (v2@a65cd99). Learn more about missing BASE report.

Additional details and impacted files
@@          Coverage Diff          @@
##             v2    #2375   +/-   ##
=====================================
  Coverage      ?   94.40%           
=====================================
  Files         ?      187           
  Lines         ?    13825           
  Branches      ?        0           
=====================================
  Hits          ?    13052           
  Misses        ?      773           
  Partials      ?        0           
Flag Coverage Δ
unit 94.40% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@vdusek
vdusek marked this pull request as ready for review October 31, 2026 11:48
@vdusek
vdusek requested review from Mantisus and Pijukatel October 31, 2026 11:48

@Mantisus Mantisus left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@apify-approval-mirror apify-approval-mirror Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mirroring approval from @Mantisus.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

t-tooling Issues with this label are in the ownership of the tooling team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants