🤖 Downloader Crawler
Information about the Downloader web crawler
What is Downloader?
Downloader is a website downloading service that allows users to create offline copies of publicly accessible web pages. Our crawler retrieves pages and assets at the explicit request of our users.
User-Agent String
Our crawler identifies itself with the following User-Agent:
Crawler Behavior
- On-demand only: We only crawl pages when explicitly requested by a user
- Respectful rate limiting: We limit request frequency to avoid overloading servers
- Robots.txt compliance: By default, we respect robots.txt directives
- No deep crawling by default: Most requests are for single pages only
We do not: Bypass authentication, access password-protected content, circumvent access controls, or crawl content that is not publicly accessible.
Whitelisting Our Crawler
If you’d like to whitelist our crawler, you can allow the User-Agent string above or add this to your robots.txt:
Blocking Our Crawler
To block our crawler, add this to your robots.txt:
Contact Us
If you have questions or concerns about our crawler, please contact us at:
support@dynomapper.com
Learn more about Downloader at: downloader.dynomapper.com