A series of improvements on the logs explorer
When it comes to collecting HTTP traffic logs, redirection.io is one of the best solutions on the market: once installed, you get all the HTTP traffic logs of your website in real time, and you can use them to analyze user behavior, monitor website performance, and debug issues.
Collecting logs has been one of redirection.io's core features since the beginning and we've been working hard to make it as efficient and reliable as possible. To achieve this, we have been using a lot of different technologies and experimenting with a lot of solutions. As a result, we have recently taken some steps towards making our logs feature the most efficient and reliable it can be and we thought it would be interesting to share the improvements with our community.
Introducing Log View Counters
Two years ago, we added the concept of "Logs views", which was an innovative way to save individual logs screen configurations as bookmarks, so that users could easily access their preffered view at all times. This feature became very popular, and many of our users utilize it to monitor specific parts of their website, or only a range of HTTP status codes, user agents, etc.
Since then, we've been thinking of interesting ways to go further and to add some kind of "counters" to the logs views allowing users to see the number of logs in each view more easily.
Today, we are enabling, for all our users, the new "Log View Counters" feature, which is a way to "pin" several clickable log view counters at the top of the logs screen, so that you can see the number of logs that match the log view. This is a very simple feature, but it is very useful, and we hope that it will help you to better understand your HTTP traffic logs.
Logs timeline navigation
Also, it is a shame that our logs GUI did not easily allow you to navigate in the logs history - this is now fixed and we have added "back" and "forward" buttons to the logs screen, so that you can easily navigate in the timeline.
New filters and aggregations
Some new filters and aggregations are available in the logs explorer, so that you can better understand your HTTP traffic logs. Of course, all these filters can be used in combination with log views, so you can prepare specific traffic views that match your business and your website structure.
For example, we have added the ability to filter logs by "Path" (see only the requests for which the path starts with /blog), "Referrer Host" (which domains to the visitors come from?) or "Query params" (did the URL of the request contain a query parameter?).
Customers using our fully managed instances or the redirection.io agent in reverse proxy mode can also display and filter requests by their "Duration", which is basically the time it took for the backend to process the request. This duration is measured at the redirection.io agent level, from the moment the agent receives the request to the moment it begins to send the response to the client. This is a very useful metric to monitor the performance of your website, and to detect slow pages. This metric can be used to inspect the performance of the redirection.io agent when it comes to performing redirections: most of the time, you should see a 0 ms. duration of such requests, as the agent is able to perform redirections almost instantly.
Faster logs exports
Since day one, we have allowed the export of traffic logs in CSV format. Recently, we've made several changes in our logs storage infrastructure which have resulted in a faster logs export feature. You can now export logs in a matter of seconds, even very large logs sets.
More logs per day
Historically, there has been a limit of 2 million rows per day (even for free projects), in order to avoid performance issues and offer a quality service to all our users. We've even allowed temporary bursts in order to accept traffic peaks. In the current situation, we collect and process every day several billions of HTTP requests.
Since September 2023, we allow to extend the logs retention duration, so our customers can have an insight on their detailed logs for up to 12 weeks. This has been a great step forward for SEO consultants, as this allows them to cover a complete Googlebot exploration window and get clear insights of the traffic fixes priorities: which 404 errors did Googlebot encounter, which pages are the most crawled, where is the crawl budget consumed, etc.
However, we were still missing the ability to upper the amount of logs that can be loaded in a project each day. This was a bit problematic for some of our customers, who have very high traffic websites and who wanted to collect all the logs of their website - the over quota logs were dropped, and this was a bit frustrating as it translated into a flat traffic chart and eventually a loss in the traffic errors collection.
This is now a thing of the past, and "Pro" plan customers can now request higher daily traffic logs limits. We will not detail why collecting an exhaustive view of traffic logs is important from a SEO perspective, as this has already been covered in many other articles, but we are happy to offer this new feature to our customers.
👉 Please contact us if you wish to increase your daily logs quota.
We always love to hear from our customers, and we are always happy to help you to better understand your HTTP traffic logs. Please contact us if you have any questions, if you need some help to extract information from your logs or if you have ideas of features that you would like to see in redirection.io. We are working hard to improve the logs explorer and have many new improvements coming in the next months, so stay tuned!