Strategy to crawl robots from ftp scheme #13

Open

opened

Like ftp://ftp.example.com/robots.txt

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests