If you’re not already familiar with the Log File Analyser tool, it allows you to upload your raw server log files, verify search engine bots, and analyse search bot data for invaluable SEO insight and understanding about how search engines are really crawling a site.
While this update is fairly small, it includes some significant new features and upgrades to the Log File Analyser based upon user requests and feedback. Let’s take a look at what’s new in version 3.0.
1) Configurable User-agents
You’re now able to completely configure the user-agents you wish to import into a project. You can choose from our pre-defined list of common search engine bot user-agents, or de-select those that are not relevant to you.
This helps improve performance and reduces disk usage by focusing only on bots of interest. You can also add your own custom user-agents, which are then stored and can be selected for projects.
Previously the Log File Analyser only analysed bots for Google, Bing, Yandex and Baidu, so now this allows users to monitor bots from other popular search engines, such as Seznam in the Czech Republic. It also allows users to analyse and monitor other specific user-agents of interest, such as Google-News, or Adsbot etc.
2) Include Functionality
Similar to the SEO Spider include feature, you’re able to supply a list of regular expressions for URLs to import into a project. So if you only wanted to analyse certain domains, or paths, like the /blog/, or /products/ pages on a huge site, then you can now do that to save time and resource – and more granular analysis.
3) New Log File Format Support
The Log File Analyser now supports Application Load Balancing log file format and Verizon Edge Cast format.
All you need to do is drag and drop in log files (or folders) as usual, and the Log File Analyser will automatically detect their format and start to analyse them.
4) JSON Support
You can now upload log files in JSON format to the Log File Analyser. There isn’t a common standard, so we have utilised customer provided JSON formats, and provided support for them all. We’ve found these to cover most cases, but do let us know if you have any issues when importing.