Screaming Frog for Beginners: A Practical Audit Walkthrough

This Screaming Frog tutorial shows you how to run a real technical audit with the SEO Spider, from the first crawl to the spreadsheet you hand to a developer. Screaming Frog is a desktop crawler that visits your pages the way a search engine does and reports what it finds: broken links, redirects, missing titles, blocked URLs and more. The free version crawls up to 500 URLs, which is enough for many small sites. Below you will find the crawl configuration settings worth checking before you press Start, a step-by-step method for broken link detection, an XML sitemap crawl, and an export routine that turns raw data into fixes. Version details reflect 24.3, the latest release on Screaming Frog’s release history at the time of writing.

What Is Screaming Frog SEO Spider?#

Screaming Frog SEO Spider is a desktop website crawler for Windows, macOS and Linux. It follows the links on a site, records each URL’s status code, metadata, directives and links, and flags technical SEO problems. The company says it can identify over 300 SEO issues, warnings and opportunities.

Because it runs on your own computer, you control when it crawls, how fast, and how much it covers. Your machine’s memory limits what it can handle.

Source: Screaming Frog’s pricing page and crawling guide. Prices can change, so confirm before buying.

How Do You Run Your First Crawl?#

Download the SEO Spider from Screaming Frog, install it, type your homepage URL into the “Enter URL to spider” box, and click Start. Results fill in while the crawl runs. Wait for the progress bar to reach 100% before exporting anything.

1. Enter the full URL including https://.
2. Choose your scope. If you enter a subdomain such as www, other subdomains are treated as external links. If you start from the root domain, the tool crawls all subdomains by default.
3. Crawl a section only if needed. Enter the subfolder with a trailing slash (for example /blog/) and select Subfolder in the top switcher. The start URL must return a 200 response and link to other pages in that folder, otherwise there is nothing to discover.
4. Watch the right-hand overview pane. It updates during the crawl, so you can start reading response codes before it finishes.

Which Crawl Configuration Settings Matter Most?#

Before a serious crawl, check mode, scope, resource types, speed, rendering and robots.txt handling. These crawl configuration settings decide what you see and how much load you put on the server. Some options require a licence, so check the pricing page for the current split.

For large crawls, Screaming Frog recommends database storage (File > Settings > Storage Mode) and a 4GB memory allocation for up to 2 million URLs. By default the tool obeys robots.txt, and its Blocked URLs view shows what a disallow rule is hiding. If the audit reveals a bad file, a free robots.txt generator can help you draft a clean replacement.

How Do You Find Broken Links With Screaming Frog?#

Crawl the site, open the Response Codes tab, and apply the Client Error (4xx) filter. Click a broken URL, then open the Inlinks tab at the bottom to see which pages link to it. Export everything with Bulk Export > Response Codes > Client Error (4XX) Inlinks.

For broken link detection, these details matter:

• Read the Inlinks columns. “From” is the page containing the broken link, “To” is the dead URL, and you also get anchor text and whether the link is followed.
• Prioritise by type. The Issues tab treats broken internal links as high priority and broken external links as low priority warnings.
• Watch for redirected sources. The source in the export is the last page before the 404. If its type says HTTP Redirect, open Reports > Redirects > All Redirects, filter the final status code to 404, and trace the original source.
• Check jump links separately. Broken bookmarks do not return a 404. Enable Crawl Fragment Identifiers under Config > Spider > Advanced, crawl, then use the URL tab’s Broken Bookmark filter.

Screaming Frog’s broken link tutorial shows each click. To fix, update the link to a live equivalent, or remove it. Add a redirect only when a genuinely relevant replacement page exists.

How Do You Crawl an XML Sitemap in Screaming Frog?#

Enable Crawl Linked XML Sitemaps under Configuration > Spider > Crawl. Then either auto-discover sitemaps through robots.txt or paste the sitemap URLs. When the crawl ends, run Crawl Analysis > Start so the Sitemaps tab filters populate.

The XML sitemap crawl compares what your sitemap claims against what the crawler found. Screaming Frog’s sitemap audit guide covers the process. The Sitemaps tab offers these filters:

• URLs in Sitemap: every URL listed in a sitemap.
• URLs Not in Sitemap: pages found in the crawl but missing from the sitemap.
• Orphan URLs: pages that appear only in the sitemap and have no internal links.
• Non-Indexable URLs in Sitemap: listed pages that are noindexed, redirected or otherwise not indexable. Remove them or fix the cause.
• URLs in Multiple Sitemaps, plus flags for any sitemap with over 50,000 URLs or over 50MB.

Those last two flags exist because Google limits a single sitemap to 50MB uncompressed or 50,000 URLs, with sitemap index files for larger sites. If cleanup means rebuilding the file, the SEO Spider can generate sitemaps, and Search Savvy’s free sitemap XML generator is another option for smaller sites.

What Does a Repeatable Technical Audit Workflow Look Like?#

A good technical audit workflow moves from crawl access to indexability to on-page issues, then ends with prioritised exports. A fixed order keeps you from chasing minor warnings while serious errors sit unread.

1. Set scope and settings. Decide whether you are crawling the whole site, a subfolder or a URL list, and note the configuration you used.
2. Triage the Issues tab. Start with high-priority items such as broken internal links.
3. Review status codes. Look at 4xx errors, 5xx server errors and redirects in the Response Codes tab.
4. Check indexability. Review the Directives tab for noindex rules, the Canonicals tab for missing or conflicting canonicals, and Blocked URLs for robots.txt rules.
5. Audit on-page elements. Use the Page Titles and Meta Description tabs to find missing, duplicate or overlong entries.
6. Reconcile the sitemap. Run the XML sitemap crawl and compare it with internal links.
7. Export, assign, re-crawl. Send fixes to owners, then crawl again to confirm they worked.

For example, fix broken navigation links first because they appear on every page, then clean the sitemap, then rewrite duplicate titles. Teams that would rather hand this analysis to specialists can look at Search Savvy’s website audit services and technical SEO services.

Which Screaming Frog Export Reports Should You Use?#

Most audits need four or five exports. Match each to the person who will act on it, and keep the original file untouched.

Add owner, priority and status columns before sharing, so a developer knows what to fix first.

What Changed in Screaming Frog During 2026?#

Version 24.0, released on 19 May 2026, was the biggest update this year. According to Screaming Frog’s version 24 announcement, it added an SEO Spider MCP so you can run crawls and analyse data through AI chat assistants such as Claude, along with Auto Compare Crawls, crawl-change summaries in email notifications, crawl export attachments by email, and a way to find uncrawlable link types. To see uncrawlable links, enable Store in Config > Spider > Crawl. Versions 24.1 to 24.3 were bug-fix releases.

Screaming Frog stresses that the MCP is not a replacement for an experienced SEO professional, and not every feature is supported through it.

What Mistakes Should Beginners Avoid?#

• Crawling too fast. The default is 5 threads for a reason. Screaming Frog advises agreeing a crawl rate with the webmaster and watching response times.
• Omitting the trailing slash on a subfolder. /blog is not recognised; /blog/ is.
• Skipping Crawl Analysis. Without it, the Orphan URLs filter stays empty.
• Treating the Issues list as a strategy. It shows what the crawler detected, not what matters most for your traffic.
• Ignoring the cap. A free crawl stops at 500 URLs, so larger sites are only partly covered.

Frequently Asked Questions#

Is Screaming Frog really free?#

Yes, within limits. The free version crawls up to 500 URLs. A licence costs £199 per year and removes that limit while unlocking advanced features.

Is Screaming Frog hard for beginners?#

The interface looks dense, but the first workflow is simple: crawl, open Response Codes, filter to 4xx, and export.

Can Screaming Frog crawl JavaScript websites?#

Yes. In JavaScript rendering mode it renders pages in a headless Chrome browser and extracts content and links from the rendered HTML. This feature requires a licence.

How do I crawl only part of a website?#

Enter the subfolder with a trailing slash and select Subfolder. You can also narrow a crawl with the include and exclude functions or by uploading a URL list in List mode.

Does Screaming Frog replace Google Search Console?#

No. Search Console shows how Google sees your site, while Screaming Frog shows what a crawler finds on demand. The licensed version can pull in Search Console and Google Analytics data.

How often should I run a crawl?#

Crawl after migrations, redesigns or template changes, and on a regular schedule for active sites. Scheduling is a licensed feature.

Your Next Step#

Start small. Download the free version, crawl your most important 500 URLs, and fix every broken internal link and erroring sitemap URL. Then crawl again to confirm. When you want help turning crawl data into a roadmap, or want to unpack terms like canonical and crawl depth, the SEO glossary and the team at Search Savvy are good places to continue.