Real-time log monitoring for SEO migration tracking
Have you just carried out a migration or a redesign of your website, or have you deployed major changes to production? As an SEO manager or SEO consultant, you have naturally performed your usual checks:
- Crawl of the new version of your site to verify that all SEO elements are in place and optimized (title, meta description, markup, internal linking, etc.) and that your pages do not return 4XX or 5XX errors
- Crawl of your redirection plan to ensure that your old URLs are properly redirected to the equivalent new URLs
If necessary, you have created the required tickets so that the identified problems are corrected as soon as possible.
And now? You are preparing to wait for the first data that will indicate if your migration or redesign is a success: organic traffic indicators to identify a trend, Google Search Console data (clicks and impressions, processing and indexing of your new URLs), evolution of the visibility index of your favorite SEO tool, etc.
Yet, without prejudging the success or failure of your migration, it is possible from the moment it goes live to gather initial data that can indicate if you are on the right track:
- Are your users and/or search engine crawlers hitting error pages?
- Is Googlebot successfully crawling your new pages? Is it taking your redirections into account?
This data comes from real-time log analysis. Having it live allows you to immediately correct course if your migration takes a wrong turn - without waiting several days for information to appear in Google Search Console or your analytics tool. Discover how to leverage this, and implement necessary corrections if needed, without being dependent on your IT team.
What is log analysis?
Log analysis consists of exploiting the log files generated by your web server. Concretely, for every request - whether it comes from a user or a crawler like Google -, a line is recorded. This line contains information such as the requested URL, the HTTP response code obtained (200, 301, 404, etc.), the date and time, the user-agent, or even the IP address and response time.
Logs thus offer raw data and show you everything that actually passes through your server, whereas a tool like Google Analytics offers a user-oriented view (sessions, conversions, etc.), with potentially partial data (lack of consent, poor implementation) and processed data.
Why log analysis is important in SEO
Monitoring site exploration by Googlebot and AI crawlers
In SEO, analyzing your server logs is the only way to verify that Googlebot - Google's crawler - is properly exploring your website pages, and more specifically:
- To ensure it crawls all your URLs (and which ones are ignored)
- To check that it does not encounter error response codes (notably 404 or 500 errors)
- To verify the frequency at which it explores each of your URLs, different sections of your site, or your different page templates
This is of course valid for crawlers from other search engines (Bingbot for Bing, for example), but also for AI crawlers (ChatGPT, Perplexity, Mistral, etc.). In fact, analyzing your logs is currently the only reliable way to identify which pages of your site served as sources in LLM responses.
Monitoring and optimizing the crawl budget
Log analysis is also important in SEO to ensure that the crawl budget is well optimized, particularly for sites with high page volumes. Indeed, given the considerable number of pages online, Google cannot explore or index everything. The crawl budget therefore refers to the time and resources Google has to explore a site. It is determined by:
- The crawl capacity limit, calculated by Google to ensure it explores your site without overloading your servers (maximum number of simultaneous parallel connections, delay between two explored pages)
- The crawl demand, which varies depending on your site's size, update frequency, content quality, relevance, page popularity, etc.
To optimize this crawl budget, your role as an SEO manager is to guide Googlebot toward your site's useful pages and prevent it from wasting resources exploring useless URLs (duplicate content, unimportant content, etc.). Log analysis allows you to see exactly which URLs are explored by Googlebot, and therefore if your exploration budget is well optimized.
Crossing log and crawl data
Finally, crossing your logs with your crawl data provides a wealth of useful information for your SEO:
- Orphan pages (explored by Google but not linked to your site structure through internal linking)
- Exploration rate according to page depth
- Exploration rate by section and/or page template
- Active and inactive page rates by section and/or page template
- Gaps between crawl and organic traffic
- Etc.
Generally, log data is analyzed over the last 30 days and compared to a recent crawl over the period.
Why use real-time log analysis
In the case of real-time log analysis, the data is studied live - in other words, over a much shorter time frame (a few minutes or a few hours). You are not looking at what happened yesterday or in previous days, but what is happening on your site right now.
You can use a tool like redirection.io to analyze your traffic logs in real time, with one advantage: you don't have to import your log files via FTP (File Transfer Protocol). Once redirection.io is installed, the tool records all your HTTP traffic and makes it exploitable live.
Live log analysis is particularly useful in many cases:
- During a website migration or redesign
- During the production launch of a new site or new features on a site
- When publishing new content
In the specific case of a migration, real-time log data allows you:
- To verify that your users or search engine crawlers are not accessing 4XX or 5XX error URLs - unless they are pages for which you intentionally returned a 404 or 410 response code, for example
- To see if Googlebot is exploring your new URLs and taking your redirections into account
- To ensure that old URLs have not been forgotten in your redirection plan by monitoring the response code they return
The benefit? Reacting immediately if a problem is identified. Indeed, while it will take at least several hours to see the first click and impression data in Google Search Console, and even several days for data on your explored or indexed URLs, live log analysis allows you to immediately see the first error response codes appearing. And thus to implement the necessary corrections as soon as possible, to minimize the impact on your organic traffic and revenue.
Analyzing your logs live with redirection.io
The real-time HTTP traffic log analysis feature of redirection.io perfectly meets this need. Moreover, redirection.io's features generally allow you to implement the necessary corrections if a problem is identified, autonomously, without being dependent on IT, since you can directly from the interface:
- Set up redirections
- Modify your robots.txt file
- Return a 410 response code (for example) on certain URLs
- Set certain pages to noindex
- Etc.
A significant asset when you would otherwise have to wait several weeks for the next production release to include the fix for the identified problems, while your organic traffic begins to decline in the meantime...
Domain change
Have you migrated from one domain to another? redirection.io allows you:
- To set up the necessary redirections on your old domain
- To monitor live their correct implementation, especially by Googlebot
- To monitor traffic and response codes on your new domain in real time - particularly Googlebot's crawl
For these last two points, simply go to the interface, to Logs > Logs list, and then create a filtered view for each domain (for example by filtering on the host).
In the case of the old domain, you can then filter by the Googlebot user-agent to ensure it is taking your redirections into account - if necessary, also filtering by response code to isolate 301 URLs or others.
In the case of the new domain, after filtering by host, you can also use the response code filter to exclude all URLs that do not return a 200, check all your URLs that are in error (and in redirection), and thus ensure that this behavior is expected.
Migration or redesign
Similarly, in the event of a site migration or redesign, you can create multiple log views to filter by your old URLs and your new URLs, ensuring in real time that some of your URLs do not return an error, that Googlebot is exploring your new URLs, etc.
You can, for example, filter by your new URLs, then by those returning only 4XX or 5XX, and group by URL to prioritize those most consulted (by your users and/or by crawlers).
You can then react immediately to fix your error pages or in case of another identified problem by creating a redirection rule in one click via the redirection.io interface.
Practical for identifying pages that might have escaped your vigilance during migration and fixing the problem to avoid any loss of organic traffic or poor user experience. All this autonomously, without having to wait for the ticket created with your IT team to be processed.
Tracking Googlebot's crawl
Have you just migrated your website, done a production release, or published new content, and want to see how Googlebot reacts to the changes made? Nothing could be simpler: by filtering on the Googlebot user-agent, you see in real time the pages explored by Google's crawler.
You can thus create a dedicated log view and save it, and check if Googlebot is exploring your new content - without waiting several days for Google Search Console data to update.
Similarly, you can filter the data again to focus on certain sections (or page templates) of your site, certain response codes, or group by URL.
Monitoring your GEO optimizations
Are you optimizing your content to be cited in responses from LLMs like ChatGPT or Perplexity? Want to see if your latest changes have paid off? Your server logs allow you to identify the URLs of your site that served as sources for these LLMs, by filtering specifically on "user-triggered" user-agents, i.e., those that visit a page in response to a user action.
This includes, for example, ChatGPT-User (for ChatGPT), Perplexity-User (for Perplexity), Claude-User (for Claude), or MistralAI-User (for Mistral AI).
A filter on these user-agents, combined with a filter on URLs, allows you to identify if the content of your choice (either a specific page or a section of the site - depending on your filtering) that you have just optimized or published is being visited by these "user-triggered" crawlers, and thus if they are serving as a source for ChatGPT and other LLMs.
Detection of SEO or technical incidents
Combining real-time traffic logs and notifications on redirection.io also allows you to be alerted in case of an incident or traffic anomaly. You simply need to create the log view of your choice, save it, and then configure a notification if this log view contains a certain number of elements over the selected time period (or in case of an abnormal variation).
You can, for example:
- Create a log view filtering by response code on 500 errors to be notified if their number spikes
- Create a log view with examples of URLs to monitor that should return a 301 (redirections to monitor), and be notified if any of these URLs returns another response code
- Etc.
This feature is particularly useful for being warned of an incident or anomaly on the site, whether or not it is related to a production release.
Conclusion
Carrying out a migration or launching a redesign always involves a degree of risk for your SEO. How will it be perceived by Google? How will your organic traffic evolve?
With real-time log analysis, you can - from the production launch of your migration or redesign - monitor response codes live, see how Googlebot reacts to changes by looking at how it crawls your URLs, etc. And thus react to avoid or limit the loss of organic traffic, without having to wait for data to appear in your usual tools (notably Google Search Console).
With redirection.io, there is no need to import your logs via FTP. Once the tool is installed, all your HTTP traffic is recorded and can be consulted in real time. Most importantly, you can directly from the interface create rules to fix any problems. Autonomously, without being dependent on your IT team, and without having to wait several weeks for your bug tickets to be processed.