What is the redirection.io crawler?
The redirection.io crawler is a tool that allows you to scan your website and find all the URLs that are available on it. It is a key component of the redirection.io solution, as it allows you to monitor the URLs that are available on a website, and to detect the ones that are not working properly.
The crawler is now in private beta. Please contact us if you want to try it.
The crawler gives a global overview of your website, allowing to analyse each resource in depth.
The main components of a crawl are:
- The crawl dashboard, a global overview of the crawl, with some key metrics;
- An alerts list, which reports if there are issues that should be fixed;
- Some reports to drill down into the crawl results;
- The list of all the URLs found on the website, with a powerful filters system;
- A details page for each URL that has been found during the crawl.
Why should you use the crawler?
Thanks to the crawler, you can find all 404 errors on your website, and fix them before
they are reported by your users. This will help you improve the user experience
on your website, and avoid losing potential customers.
But the redirection.io crawler is not only about 404! It also allows to find all the configuration errors and frontend optimizations that should be set up on your website.
You can read more information about the data redirection.io collects about your website URLs in the "Columns reference" page. For each page that is discovered, redirection.io computes a serie of metrics that are used to rate the page - read the Metrics reference page for more information.
How to use the crawler?
Once the crawler is enabled in your project, click on the "Crawler" tab in the left menu of the manager to access the crawler interface. From there, you can start a new crawl.