When it comes to collecting HTTP traffic logs, redirection.io is one of the best solutions on the market: once installed, you get all the HTTP traffic logs of your website in real time, and you can use them to analyze user behavior, monitor website performance, and debug issues.

Collecting logs has been one of redirection.io's core features since the beginning and we've been working hard to make it as efficient and reliable as possible. To achieve this, we have been using a lot of different technologies and experimenting with a lot of solutions. As a result, we have recently taken some steps towards making our logs feature the most efficient and reliable it can be and we thought it would be interesting to share the improvements with our community.

Introducing Log View Counters

Two years ago, we added the concept of "Logs views", which was an innovative way to save individual logs screen configurations as bookmarks, so that users could easily access their preffered view at all times. This feature became very popular, and many of our users utilize it to monitor specific parts of their website, or only a range of HTTP status codes, user agents, etc.

Since then, we've been thinking of interesting ways to go further and to add some kind of "counters" to the logs views allowing users to see the number of logs in each view more easily.

Today, we are enabling, for all our users, the new "Log View Counters" feature, which is a way to "pin" several clickable log view counters at the top of the logs screen, so that you can see the number of logs that match the log view. This is a very simple feature, but it is very useful, and we hope that it will help you to better understand your HTTP traffic logs.

Logs timeline navigation

Also, it is a shame that our logs GUI did not easily allow you to navigate in the logs history - this is now fixed and we have added "back" and "forward" buttons to the logs screen, so that you can easily navigate in the timeline.

New filters and aggregations

Some new filters and aggregations are available in the logs explorer, so that you can better understand your HTTP traffic logs. Of course, all these filters can be used in combination with log views, so you can prepare specific traffic views that match your business and your website structure.

For example, we have added the ability to filter logs by "Path" (see only the requests for which the path starts with /blog), "Referrer Host" (which domains to the visitors come from?) or "Query params" (did the URL of the request contain a query parameter?).

Customers using our fully managed instances or the redirection.io agent in reverse proxy mode can also display and filter requests by their "Duration", which is basically the time it took for the backend to process the request. This duration is measured at the redirection.io agent level, from the moment the agent receives the request to the moment it begins to send the response to the client. This is a very useful metric to monitor the performance of your website, and to detect slow pages. This metric can be used to inspect the performance of the redirection.io agent when it comes to performing redirections: most of the time, you should see a 0 ms. duration of such requests, as the agent is able to perform redirections almost instantly.

Faster logs exports

Since day one, we have allowed the export of traffic logs in CSV format. Recently, we've made several changes in our logs storage infrastructure which have resulted in a faster logs export feature. You can now export logs in a matter of seconds, even very large logs sets.

More logs per day

Historically, there has been a limit of 2 million rows per day (even for free projects), in order to avoid performance issues and offer a quality service to all our users. We've even allowed temporary bursts in order to accept traffic peaks. In the current situation, we collect and process every day several billions of HTTP requests.

Since September 2023, we allow to extend the logs retention duration, so our customers can have an insight on their detailed logs for up to 12 weeks. This has been a great step forward for SEO consultants, as this allows them to cover a complete Googlebot exploration window and get clear insights of the traffic fixes priorities: which 404 errors did Googlebot encounter, which pages are the most crawled, where is the crawl budget consumed, etc.

However, we were still missing the ability to upper the amount of logs that can be loaded in a project each day. This was a bit problematic for some of our customers, who have very high traffic websites and who wanted to collect all the logs of their website - the over quota logs were dropped, and this was a bit frustrating as it translated into a flat traffic chart and eventually a loss in the traffic errors collection.

This is now a thing of the past, and "Pro" plan customers can now request higher daily traffic logs limits. We will not detail why collecting an exhaustive view of traffic logs is important from a SEO perspective, as this has already been covered in many other articles, but we are happy to offer this new feature to our customers.

👉 Please contact us if you wish to increase your daily logs quota.

We always love to hear from our customers, and we are always happy to help you to better understand your HTTP traffic logs. Please contact us if you have any questions, if you need some help to extract information from your logs or if you have ideas of features that you would like to see in redirection.io. We are working hard to improve the logs explorer and have many new improvements coming in the next months, so stay tuned!

In today's fast-paced digital landscape, safeguarding your online assets is of paramount importance. Whether you're protecting sensitive data, ensuring the privacy of your users, or simply maintaining control over your web services, having robust security measures in place is essential. That's why we're thrilled to introduce a new feature in our SaaS platform - Multiple Project Keys - designed to give you greater control and flexibility over your projects in redirection.io.

The Project Key: The Project Configuration Secret

Before diving into the new feature, let's first understand the significance of the "project key." This unique string is the linchpin of your redirection.io project. It's the secret sauce that binds an instance to a specific project, allowing it to access and interact with the associated rules and logs. By default, every project receives a project key, which you can directly use to configure your redirection.io agent instance. However, there are scenarios where having multiple project keys can be helpful.

Why Multiple Project Keys Matter

Here are a few compelling reasons why you might want to utilize multiple project keys:

• Enhanced Security: If your project key becomes public knowledge, it could pose a significant risk. With multiple project keys, you can rotate or change keys as needed to maintain the utmost security.
• Development-Production Segregation: To maintain a clear separation between development and production environments, you can use specific project keys for each. This way, your development team can work without access to the production project key, ensuring data integrity.
• Key Expiration: Sometimes, you might want a project key to have a limited lifespan. With the option to set expiration dates on project keys, you can control when they stop working.

Introducing Multiple Project Keys

redirection.io now offers the capability to create as many project keys as you require. Each project key can be customized with a name, helping you clarify why it was created, and an optional expiration date, ensuring it ceases to work after a set time.

Key Rotation for Improved Security

If you suspect that your project key has been compromised or simply want to enhance security, our platform makes it easy to rotate project keys. Here's how you can do it:

1. Create a new perpetual project key that will serve as the replacement.
2. Add an expiration date to the old project key, ensuring it will no longer function after that date.
3. Update your instance configurations with the new project key.

The graphical interface also displays the last time a project key was used, allowing you to determine when it's safe to remove the old key. This way, you can rest assured that all your instances have been updated, and the old project key is no longer in use.

The introduction of Multiple Project Keys in redirection.io is a significant step forward. With this feature, you can enhance the security of your projects, maintain clear separations between environments, and have full control over the lifespan of your project keys. To learn more about this feature, check out our documentation about instances and project keys.

We have recently improved the the redirection.io rules system by introducing negation support for the "Method" and "Backend Response Status Code" triggers, providing you with greater flexibility and control over rule execution.

In a redirection.io rule, Triggers define the conditions that incoming HTTP requests must meet for the rule to be applied. For example, the "Method" trigger filters requests based on the HTTP method used, while the "Backend Response Status Code" trigger allows you to restrict triggering based on the status code generated by your backend application.

With our latest update, you can now utilize negation for these triggers. For example, you can define rules that trigger when the HTTP method is not GET or POST, or when the backend response status code is not 200 or 301. This new capability enables you to refine your rule logic and execute specific actions based on negated conditions.

The feature is immedialtely available for the customers who use our "hosted instances" or the managed Cloudflare workers. Other deployment modes (nginx or Apache modules, agent reverse proxy, manually added Cloudflare worker, Fastly Compute@Edge, etc.) needs to be updated to use at least the redirection.io agent 2.7.0 or libredirectionio 2.8.0.

To learn more about negation support and explore other features of the redirection.io rules system, refer to our documentation about triggers. We value your feedback and are continuously working to improve our services based on your needs. Thank you for choosing redirection.io as your trusted solution for efficient traffic management!

We're pleased to announce the addition of the Aggregated Logs Details Panel to the redirection.io manager. This new feature brings convenience and efficiency to your analysis of HTTP traffic logs, allowing you to delve deeper into your most frequent requests.

In the logs screen of the manager, you can already group requests by various criteria like URL, HTTP method, response status code, and user agent. However, we've now made it even better by enabling the display of details for grouped requests.

For example, if you want to examine the most frequently requested URLs, simply group the requests by URL and click on a specific URL. You'll instantly access a comprehensive list of matching requests, complete with a preview of the most recent ones. You can easily navigate through these requests using the intuitive "next" and "previous" buttons.

This small yet handy feature provides valuable insights into your traffic logs, allowing you to quickly identify any issues or patterns. It's perfect for troubleshooting or conducting in-depth analysis.

We strive to make your experience with the redirection.io manager as seamless as possible, and Aggregated Logs Details is another step towards achieving that goal.

If you have any questions or need assistance, our support team is ready to help!

We are thrilled to announce a new feature in redirection.io that will make managing your sitemap.xml files easier than ever before. Our new "Sitemap management" feature is now available for "Pro" plan subscribers and allows you to define the content of your sitemap files directly within redirection.io.

Creating and managing sitemap files is an essential part of website management. However, it can often be a time-consuming and manual process, especially for larger websites. With our Sitemap management feature, you can avoid the need for developers to deploy a new version of the website every time a sitemap.xml file needs to be updated. This feature is useful in many professional contexts when the sitemap.xml management is not fully automated.

With the new redirection.io Sitemap action, you can easily define the content of your website's sitemap.xml file(s) directly from redirection.io, and serve it at the location specified in the "Source URL" trigger.

By using redirection.io to manage your sitemap files, you can improve your productivity and focus on more important tasks. This feature, combined with the redirection.io crawler, guarantees better reactivity and the ability to cover all the pages of your website, which in turn improves your website's SEO. Additionally, you can configure several sitemaps using this action, which is perfect for larger websites or for keeping sitemaps properly sorted.

To get started with Sitemap management in redirection.io, upgrade to our "Pro" plan today. And as always, let us know your thoughts and feedback! We're excited to see how this feature will help you better manage your website!

While the redirection.io logs explorer is snappy and offers many options, it can become annoying to repeat the same filters configuration to get a specific report. For example, if your weekly website monitoring routine includes checking the HTTP errors found by search engines, you could end up being a bit fed up with configuring again and again the "Status code" and "User agent type" filters.

For such cases, we offer the "Logs views" feature. Logs views are a way to save a specific columns, filters and aggregation configuration, and to be able to reload this configuration with as few clicks as possible.

We already several predefined "Logs views" that address common analysis use cases:

• All HTTP errors: This report displays the list of all HTTP errors (both 4xx and 5xx). Use this view to spot issues peaks during the last logging period.
• Googlebot verified requests: This report lists all the requests that have been performed by Googlebot or any other Google-operated bot. We have validated this request and filtered out third-party bots pretending to be Googlebot.
• Googlebot most common errors: This report shows the most common HTTP errors encountered by verified Googlebot on your website. Whenever possible, you should try to solve these issues by setting up a new redirect rule.
• Fake Googlebot: Lists all requests that have been sent using the "Googlebot" User-Agent, but that we identified as obviously not initiated by Google services. It may be that these requests are performed by third-party bots that try to scrape some content from your website.
• Most common HTTP errors: This report lists the top 100 most common HTTP errors (both 4xx and 5xx), grouped by URL and status code. The errors are sorted by occurrence count.
• Top hit pages: Displays the most frequently hit URLs that serve html content.

You can also define your own log views, that will be saved at the project level, so you and your colleagues can use them.

Loading an existing "logs view" is a one-click operation:

1. Hover the small gear wheel on the top right corner of the filters bar
2. Then, choose one of the displayed views, or hit the "Load another view" button.

📖 You can read the full documentation about log views in the dedicated part of our user documentation.

A few weeks ago, we have enabled two-factor authentication for all redirection.io customer accounts. While this feature has been fairly successful, we have found that helping to improve the security of user accounts was not enough. Some redirection.io customers have organization with dozens of users, and one single compromised account or password could result in the leakage of company sensitive data - traffic statistics, configured rules, etc.

So starting today, we're giving organization administrators a way to strengthen access to their organization's projects, by making a new security setting available. Organization administrators can now require all organization users to enable Two-factor authentication on their personal accounts.

A picture is worth a thousand words, so here is what it looks like:

You can find detailed explanations in the documentation about Organizations.

In the 1.6.0 release of the agent, we have introduced the test mode, a new way to help you test and try your redirect rules before they are published to your production servers. Historicaly, it was not possible to log the traffic from instances configured with this "test mode", which was meant for pre-production or testing servers - we estimated that it did not make much sense to log traffic from these platforms.

However, due to several requests on this topic, we have changed the behavior in the recent 2.1.0 release of the agent, and we have made it possible to enable HTTP traffic logging from instances using the "test mode". From now on, you can get a view of your live HTTP traffic logs even on "test" instances.

Another much-demanded feature about instances was the ability to configure their behavior directly within the agent.yml configuration file. We have introduced, also in the 2.1.0 release of the agent, the new test_mode and logging configuration, which can be used to statically set the configuration of agents server-side. If those keys are defined, you will not be able anymore to change this setting in the instances management interface:

This morning, we have launched an improved way for managing roles within organizations and projects. In particular, these enhancements allow for the creation of "contributor" and "financial" users, with restricted permissions.

financial users can access billing information and invoices - this role is designed to fit the requirements of financial departments, which usually do not require access to all the project functionnality.

contributor users can create new rules or modify existing ones, but they can not publish rules, which means that they do not have direct access to the redirection rules that are executed on your public website. This is especially useful to allow an external SEO consulting company to contribute new redirect rules for your website, while having a way to control upstream what is going to be changed on your website.

Our documentation about permissions has been updated to reflect these changes.

Some times ago, we have been made aware that several customers found it difficult to figure out the concept of rules "rank".

The "rank" of a rule is helpful to elect the redirection to execute when several rules match a given request. However, many users were hesitating when configuring the rank: was the lower rank executed first, or after? Several other concerns about the "rank" have been raised, among others the fact that the rank could not be lower than 0, thus forbidding to insert new rules infront of already existing 0-level rules.

Therefore, we have renamed and migrated the "rank" model to a "priority" property, that can be attached to the redirect rules of your projects just like the rank did before. The legacy rank was somehow limited in the range 0-32768, so we have transformed it into a "priority" property, that can range from -32768 to 32767.

From now, if two redirection rules match a request, then the one with the higher priority will be applied. We have migrated all the existing redirection plans, so you do not notice any issue with existing rulesets.

As before, you can change the priority of each rule inside the rule creation/edition form. You can also add example requests to your rules, and have the relative priorities of rules be auto-defined when tweaking the rules order for a given example. Please read our guide about creating rules to learn more!