Using traffic logs

One of the major features of redirection.io is to collect all the traffic logs for your website and display them in real time in the redirection.io manager graphical interface.

Using these logs, it is quite easy to spot issues and react by creating redirection rules. We will not develop here how much important it is to use your website's logs to understand if it works well - read more about our log features and why using your traffic logs is a good idea!

The logs explorer

When logged in the manager, open a project and head to the logs screen by hitting on the "Logs" menu item.

The interface of the logs explorer is divided into three main parts:

• a timeline, that allows to "zoom" on a specific time frame;
• the logs list itself. This list is highly configurable. For example, you can customize the displayed columns (by using the columns button ), their order, the sort direction, etc. The list also displays the grand total of requests matching the filters during the selected period;
• the filters bar, which allows searching for specific log entries.

Filtering the logs

The logs explorer provides several filters, that allow to narrow the requests handled by your website. The following filters are available:

• Status code: filter the requests based on the status code of the response that is sent back to the user.
• HTTP method: filter based on the HTTP method used for the request.
• URL: filter based on the full URL
• Redirection: if a redirection was performed, filter on the value of the Location of the redirection
• User agent: filter requests based on the specified User-Agent.
• User agent type: filter on the category of User-Agent. We categorize User-Agents under 4 types: Desktop clients, Mobile clients, Search engines, Tools and miscellaneous.
• Referrer: if the Referrer was provided in the request, this filter allows you to search your logs based on this referrer.
• Content type: as redirection.io logs all the traffic of your website, you may want to filter based on the Content-Type of the response, to display only web pages (text/html), for example.
• Has rule allows to filter the requests that triggered the execution of a redirection.io rule.
• Instance and Proxy are two technical filters, which allow users to understand which redirection.io agent instances logged this request / response.

It is possible to combine filters to efficiently and precisely isolate requests. For example, you can check which 4xx errors Googlebot found during the last day, check which pages are the most hit with a success status under the /blog path, etc.

Each filter can be used in two ways:

1. either by selecting one or more specific values.
2. or by defining conditions.

Note the red filter symbol that appears on the top right part of each filter when this filter is enabled: you can click this filter symbol to disable this filter in a quick way.

Logs rate and retention

Logs rate and quota

When redirection.io is deployed on a website, it collects all the traffic logs. However, the traffic data that we save is rate-limited with a soft limit of 2M log lines per day per project. We allow temporary bursts, but we will drop log lines if your traffic consistently exceeds this limit.

When your project generates more logs than what it is allowed to, we will still count the associated requests and allow to display those in the traffic chart, but we will not store their detailed information. You will end up with a grey traffic chart:

Analytics data displayed on the project dashboard are computed based on all the logged requests, even those that are not stored due to daily overquota.

If you need an exhaustive view of your traffic data, we can higher the logging rate-limit for your project. Please contact us to get your daily logs quota increased.

Logs retention

The stored logs are kept during a period called the "logs retention" period. The duration of retention depends on the plan of the project:

• logs for "Free" projects are kept for 1 day, which means that you can get and inspect your detailed traffic logs for one day.
• logs for "Starter" projects are kept for 1 week (7 days).
• logs for "Pro" projects are kept for 2 weeks (14 days).

Starter and Pro projects can however have the logs retention period extended on demand - usually, SEO audits recommend to analyze logs over a complete search engine crawling window, which can extend up to two months.

We allow extending logs to several duration steps:

• 2 weeks
• 4 weeks
• 8 weeks
• 12 weeks

Starter project administrators can extend the logs retention period for their project directly in the manager, under the project billing section:

1. Head to the "billing" section of your project;
2. Click on the "Add logs retention" button and validate the retention upgrade;
3. The billing screen is updated and the logs retention of the project is updated. From now on, new logs will be available for the defined duration.

Pro projects administrators can simply contact their redirection.io dedicated account manager to discuss the logs retention extension. We will be happy to help!

Logs quota and retention pricing

If your project needs a higher daily quota, or a longer logs retention, or both, we can scale the limits of your project plan. Storing more logs, or for a longer period, is a project addon that is billed as an option to your subscription plan:

• longer logs retention can be configured for all paid plans (Starter and Pro)
• higher daily quota can be configured for Pro plans.

Please contact us to get your quota and retention modified. We don't have a public price list for additonal logs, as it decreases according to the maximum number of log lines stored in your project at a given time. As an indication, the monthly prices for additional logs ranges from €0.54 to €1.25 per million log lines stored. For example:

• 10M daily logs, stored for 1 month, cost around €0.83 per million log lines.
• 2M daily logs, stored for 2 months, are billed at approximately €1.07 per million log lines.

Grouping the logs

It is also possible to count the requests that match given criterions, using the "Group by" feature. It is possible to group requests by:

• URL: this will display the top-100 URLs that match the selected filters, along with the number of times such requests occurred
• HTTP method: see which HTTP methods were used on your website, and how many times
• Status Code: see the distribution of status codes across the responses sent by your web platform
• User Agent: see which user agents are hitting your website most
• Instance and Proxy allow to see the distribution of the requests handled by your redirection.io agents and proxies.

Like with filters, it is possible to group requests using multiple groups. For example, grouping both by HTTP Method and Status Code will show which pairs (method / status code) are the most frequent.

Exporting traffic log entries

Once you have narrowed the logs list using filters, and arranged the displayed columns as you wish, you can export this list as a CSV file using the "export" button, on the top right corner of the logs list:

We do not set a limit on the amount of log lines that can be exported in this way, so you can use external tools to process and analyze the traffic statistics.

Googlebot requests validation

The categorization of the user agents and bot names that perform requests against your web platform is based on the User-Agent HTTP header, which is very often faked. Many bots, spiders, crawlers or automated tools, for example, use the Googlebot user agent, in order to avoid being blocked, instead of using their real name.

By default, we strongly validate requests that come with the "Googlebot" User-Agent, and categorize such requests into three groups:

• Googlebot: requests that are really performed by Googlebot are marked with the Googlebot user agent
• Googlebot (other): requests that are performed by Google systems, but not by Googlebot, are tagged Googlebot (other). This can be for example Adsense, Adsbot, Feedfetcher, etc. Google provides a list of its bots
• Googlebot (unsure): requests that are performed on your website and announced as "Googlebot", but not performed by Google, are marked as made by Googlebot (unsure). This is a convenient euphemism to avoid tagging such requests as "fake". If you receive many requests with such a user agent, it might be possible that some crawler spoofs the Googlebot user-agent to crawl your website.

This Googlebot validation feature is not designed for security reasons, but to help you analyze more precisely how Google crawls your website, and make sure that an analysis focused on the requests made by Googlebot is perfectly precise, exhaustive, and not "polluted" by other bot requests.

Log views

While the redirection.io logs explorer is snappy and offers many options, it can become annoying to repeat the same filters configuration to get a specific report. For example, if your weekly website monitoring routine includes checking the HTTP errors found by search engines, you could end up being a bit fed up with configuring again and again the "Status code" and "User agent type" filters.

For such cases, we offer the "Logs views" feature. Logs views are a way to save a specific columns, filters and groups configuration and to be able to reload this configuration with as few clicks as possible.

redirection.io proposes by default several predefined "Logs views" that address common analysis use cases, but you can define your own log views, that will be saved at the project level so you and your colleagues can use them.

To load an existing logs view:

1. Click the small gear wheel on the top right corner of the filters bar
2. then, hit the "Load another view" button.

This opens the "Logs views" lateral panel, where you can see all the available log views.

Each log view comes with several information:

• the name of the view
• a description
• an occurrence chart
• the properties of the view (columns displayed, sort criteria, filters, groups, etc.)
• an actions button bar, which allows to:
  • select and load the view using the check button
  • mark the view as a "favorite" using the start button
  • custom views, that have been created in your project specifically, can be edited and deleted using the appropriate buttons.

Views marked as "favorite" are displayed at the top of the list, in alphabetical order. They can also be loaded directly by hovering the gear wheel at the top of the filters bar.

The log entry detail panel

When no group is active, log lines can be clicked to display all the request details in a dedicated lateral panel:

This panel contains some helpful shortcuts:

• the "open" button allows to open the link in a new window:
• the "create rule" button opens the rule creation form, with the "Source URL" already filled with this URL:
• hover on the log properties: this allows to enable a filter on this value, to display all the similar requests: