Auditing Your SEO with Screaming Frog
Screaming Frog is the best SEO audit tool. Period.
One of the most important aspects of auditing your SEO is crawling your website. Screaming Frog isn’t the only web crawler on the market, but it’s definitely the leader in the industry.
In this guide we’ll run through a number of configuration options, tips and tricks to make your next SEO audit smooth. Performing an audit of your overall SEO isn’t always straight-forward, but following these steps can remove much of the ambiguity.
What are your goals?
First, determine what the goal of your SEO audit is before diving in. Are you simply exploring your SEO at a high-level to look for issues and opportunities? Or do you know of an existing issue that needs to be investigated further? Having goals upfront for your SEO audit will make it much easier to decide on configuration options.
How to configure your crawl settings
Before configuring your crawl it’s important to decide which crawling mode you’ll be using. There’re two primary modes: Spider and List (we’ll come back to SERP and Compare modes later). Spider mode uses a starting URL and then crawls subsequent pages using links it discovers. List mode on the other hand is where you give a specific list of URLs to Screaming Frog to crawl. The former is better for exploring opportunities and issues whereas the latter is better suited for validation.
With your crawling mode set and SEO audit goals in mind, you can now setup your crawl configuration. Begin by clicking Configuration > Crawl Config > Spider > Crawl. Here’s four important considerations to make at this step:
- Do we care about crawling external links? (not doing so can reduce time and noise if it’s not needed)
- Is there a need to include images, CSS, JavaScript and/or SWF resources in our crawl? (crawling CSS and JavaScript might be necessary to properly render the page, but storing it may still be optional)
- Do we want the crawl to extend beyond the current sub-domain/start folder?
- Are there any XML sitemaps we should manually include in our crawl?
Speeding up your crawl
Assuming you have deadlines and don’t have all day (or multiple days) to perform a crawl, it’s good to know how to speed your crawl up. There are two settings to consider when speeding up your crawl.
First, allocating additional memory (RAM) to Screaming Frog can be done by clicking File > Settings > Settings > Memory Allocation. Warning: increasing that increasing the memory allocation to the maximum allowed may cause some computers to experience issues (i.e., crash).
Next, increasing the number of requests made by Screaming Frog per second can be done by clicks Crawl Config > Speed and then increasing the Max Threads. Warning: increasing your crawl speed can increase the chances of being denied access to certain sites.
Issues with bot detection
If you’re noticing 403 and/or connection timeout errors while crawling you might be experiencing issues with bot detection. Most popular websites use special software to limit scraping and traffic from nefarious sources. This can limit your ability to crawl when performing an SEO audit.
The first step to working around these issues is to try changing the User-Agent to Chrome or another browser (Configuration > User-Agent and select from the present dropdown). The next step is to enable JavaScript rendering. In theory, this should make your crawler appear more like a normal user.
JavaScript rendering
Enabling JavaScript rendering in Screaming Frog can be done by clicking Configuration > Spider > Rendering and selecting JavaScript from the dropdown.
Using JavaScript rendering offers a number of pros and cons in addition to being used to bypass bot detection. JavaScript rendering is slower and more resource intensive. But it’s also required to accurately crawl websites that use JavaScript frameworks like React & Angular.
Crawling large websites
Sometimes speeding up your crawl and using text only rendering isn’t enough to crawl a large website within a reasonable amount of time. If the website you’re attempting to crawl is larger than a thousand pages there’s a good chance you’ll benefit from enhancing your crawl efficiency. If the website is over 5,000 pages it’s practically essential.
Only crawling what’s necessary for your audit and excluding everything else is often the best way to improve the performance of the crawler. Sometimes the best way to improve your crawl speed is simply by not storing unnecessary information (such as external links, CSS, JS, rendered screenshots of pages etc.).
For websites larger than 10,000 pages we recommend planning to break it up into pieces (e.g., by folder path), or use a server-based crawler (e.g., running Screaming Frog in the Cloud). We don’t recommend saving a crawl and revisiting it over the course of multiple days. The audit results could prove misleading if numerous changes are being made during the course of the crawl.
Crawl analysis
Being able to analyze crawl data and drawing meaningful insights is a potent skill to possess. Being able to distinguish opportunities and emergencies from all the noise takes a lot of experience and a nuanced approach. There isn’t a perfect website in existence so even the cleanest crawl data can look problematic at first glance to the uninitiated. For example, unindexable content isn’t always a bad thing. Some pages should have noindex tags on them. And every large website has a few broken links, no matter how diligent and talented the team is.
We recommend exporting crawl data from Screaming Frog to CSV, Excel or Google Sheets. This can be particularly helpful for large crawls that need to be segmented into digestible chucks or if there is a lot of “noise” in the data (i.e., irrelevant files that aren’t hurting or helping your SEO). After you clear out any noise (i.e., removed pages from your crawl data that don’t matter for SEO), here’s some things we recommend checking:
- URLs that aren’t indexable (but should be)
- Missing title tags and/or meta descriptions
- Meta data that doesn’t follow best practices (e.g., title tags over 60 characters)
- Pages with missing or duplicate canonical tags
- Pages with missing, duplicate or multiple H1s
- The source of 4xx and 5xx errors (using the inlinks tab)
Compare mode, as the name implies, allows you to compare one crawl file against another. Comparing crawls is a great way to see changes being made to a website over time. This can be very helpful for confirming updates have been made, aka SEO validation.
Lastly, SERP mode allows you to edit meta data in Screaming Frog’s SERP emulator. This will allow you to see how title tag and meta descriptions could look on Google. While not necessarily a form of crawl analysis, it can be helpful to use when making updates.
Preparing updates
The last step to auditing your SEO with Screaming Frog is to prepare recommendations and/or updates. After you’ve identified issues through your crawl analysis you can then begin preparing recommendations to address them. From our experience it’s best to use Excel (or an equivalent like Google Sheets) to organize SEO recommendations. Using spreadsheets makes it easy to both make changes to the recommendations and copy them into the content management system (CMS) for implementation. Trying to use copy docs (Word documents used to create web page copy) or implement recommendations directly into the CMS is not advisable.
Conclusion
We hope you’re able to use the tips and tricks in this article to perfect your next SEO audit with Screaming Frog! You can use Screaming Frog for free to crawl and analyze your website SEO if you’re website is under 500 pages. Alternatively, you can invest in paid license if your website is larger or if you need access to the configuration options. Check out Screaming Frog’s pricing page for a full breakdown of features available with the free and paid versions of the tool.
Still need help? Contact us today for your next SEO audit or to learn more about our training services – an SEO expert is standing by to assist you!