Screaming Frog has been my go-to resource prior to planning any sort of SEO strategy for about 2 years now. Unfortunately, its ugly and somewhat unintuitive design has been a turn off for many people I know, which prevents them from realizing its true value. I’m going to go step-by-step through the process of using the tool, while detailing situations in which you would need it. This post will be easier to follow if you download the latest version of the tool, located here:
At the top of the Screaming Frog (henceforth SF) window, you’ll see a form that says Enter URL to spider. When you enter the URL, make sure you put the correct version (www vs non-www). If you just copy the homepage URL of the site you’ll be fine.
The unlicensed version of SF will crawl up to 500 URI, which means everything from HTML pages to images to CSS and JS scripts and more (a URI is basically an item on a website, including URLs). If you only want to see HTML pages, click the box that says Filter and select HTML. Note that this won’t allow you to crawl more URI—you’ll just see fewer items.
The Main Crawl Window
The bulk of the SF window is made up of crawl information, which includes everything you could want to know about the site in question and more, including:
- Address – the location of the URI
- Content – what the URI is (HTML file, CSS file, image, etc.)
- Status Code – the code the URI returns. Useful for finding information about redirects, broken links, etc.
- Status – what the Status Code means (404= Not Found, 301=Moved Permanently, 200=OK, etc.)
- Title 1 – the text of the <title> tag on your page
- Title 1 Length – length, in characters, of the <title> tag
- Title 1 Pixel Width – length, in pixels, of the <title> tag. We’ve typically estimated <title> tags in the SERPs to cut off after around 70 characters, but estimates put it closer to 584 px. This measure could be useful in determining the ideal length of your <title> tags.
It goes on to list meta description, headings, page inlinks and outlinks, etc. They’re easy enough to figure out, so I won’t go into any more detail (though if someone wants to add them in the comments that would probably be helpful).
Above the main window you’ll see tabs that will allow you to filter by information, so if you only want to see Response Code information, for example, click this tab and get even more detailed information about this. This can be especially useful for the next topic, exporting data.
Exporting Screaming Frog Crawl Data
Exporting data from SF can make a number of tasks much, much easier. This is obviously not even close to an exhaustive list—be creative and find new ways to use the tool:
- Updating title / meta
- Checking for duplicate headings
- Finding word count for all your pages
The URL Info Window
Below the main crawl window, you’ll find the URL window info. There are 5 tabs:
- URL Info –displays relevant URL information, including status code, title and meta information, headings, and more. Mostly information you can get from the primary crawl window.
- In Links –lists every internal page that links to the page you’ve selected, along with the anchor text, alt text, and whether it’s followed or not. This is extremely useful for finding out the location of broken links. You can export a list of all 4XX-code in links, which we’ll get to in a minute.
Right clicking anywhere in this tab will give you four options: Copy FROM URL, Copy TO URL, Open FROM URL in Browser, Open TO URL in Browser. Should be pretty obvious what each of these does.
- Out Links –shows every page the selected page links out to, including external pages.
- Image Info –shows all pages linking to an image, as well as the image alt text.
- SERP Snippet – mocks up a Google SERP snippet so you can see how altering your title / meta, adding rich snippet information, etc. will affect your site’s appearance in the SERP.
This is a new feature of SF 2.5 and up, and allows you to export the following data:
- All In Links
- No Response In Links
- Success (2XX) In Links
- Redirection (3XX) In Links
- Client Error (4XX) In Links
- Server Error (5XX) In Links
- All Out Links
- All Anchor Text
- All Image Alt Text
- Images Missing Alt Text
Newer SF versions allow for the easy creation of XML sitemaps. Just click Create XML Sitemap under the Sitemaps tab.