Looking for ways to take advantage of Screaming Frog’s new ChatGPT integration? You’re in the right place! Learn how to get started below or jump right to the good stuff.

Screaming Frog Version 20

In case you missed it, Screaming Frog’s release of version 20 on May 7^th was a huge deal in the SEO world. This release added a handful of incredible features. But the one that stood out the most is their integration with ChatGPT’s API. But first let’s get a ChatGPT API account and key setup.

Please note: all the stuff I talk about in this article requires a paid license for Scream Frog.

How to setup a ChatGPT API account & key

ChatGPT and API accounts are separate. You can use the same email for both, but you will need to setup an API account even if you already use ChatGPT’s conversational interface.

Start by going to https://platform.openai.com/ and clicking sign up in the top right. Enter an email and password or use an existing Google, Microsoft or Apple account. Verify your email address and provide your name, organization (optional) and your birthday. Next you will see a screen with two account options – ChatGPT and API. Select API and you’re in.

ChatGPT prompt ideas for Screaming Frog

Check out these seven custom ChatGPT prompt ideas, with examples, to use with Screaming Frog’s custom JavaScript snippets.

Top keywords

One of my favorite uses for crawling with ChatGPT so far is extracting keywords from pages. But what I found particularly interesting and helpful is the ability to have ChatGPT include the context for how the keywords were being used. The most effective way I found for doing this is using the prompt below with the gpt-4-turbo model.

List the top 10 keywords with the number of mentions and context included for each, but do not include a summary

I often mention NOT including a summary when using the gpt-4-turbo model because it has a weird penchant for over explaining responses. Verbose responses aren’t only an annoyance when using the API – they actually cost more too. Furthermore, if you’re getting too much contextual “fluff” it can help to turn the temperature down (I personally like 0.4 or 0.5).

Here’s an example of a response from the prompt above when crawling screamingfrog.co.uk/seo-spider/:

[1. **SEO Spider Tool (23 mentions)**
   - Context: Described as a comprehensive website crawler that helps improve onsite SEO by auditing for common issues. It offers features like crawling JavaScript-rich websites, finding broken links, auditing redirects, and integrating with analytics tools.

2. **Crawl Configuration (12 mentions)**
   - Context: Refers to the various settings and options available within the SEO Spider Tool that allow users to customize their crawling strategies, including adjusting crawl depth, obeying or ignoring specific directives, and managing how JavaScript is rendered.

3. **Integration with Analytics (9 mentions)**
   - Context: Highlights the capability of the SEO Spider Tool to integrate with Google Analytics, Google Search Console, and Google PageSpeed Insights to fetch performance data directly during crawls for enhanced analysis.

4. **Custom Extraction (8 mentions)**
   - Context: Discusses the feature within the SEO Spider Tool that allows users to extract data using XPath, CSS Path selectors, or regex from the HTML of the pages crawled.

5. **JavaScript Rendering (7 mentions)**
   - Context: Points to the SEO Spider Tool's ability to render web pages using Chromium WRS to accurately crawl dynamic, JavaScript-driven websites and frameworks like Angular, React, and Vue.js.

6. **XML Sitemap Generation (6 mentions)**
   - Context: Focuses on the functionality within the SEO Spider Tool to quickly generate XML Sitemaps and Image XML Sitemaps, providing advanced configuration for URLs to include, which helps in optimizing site indexing.

7. **Duplicate Content Detection (6 mentions)**
   - Context: Describes how the SEO Spider Tool can discover exact and near duplicate pages using advanced algorithmic checks, which is crucial for SEO to avoid content redundancy issues.

8. **Visualise Site Architecture (5 mentions)**
   - Context: Explains the feature of visualizing internal linking and URL structure through interactive crawl and directory force-directed diagrams and tree graph site visualizations, helping users understand site layout better.

9. **Crawl Comparison and Staging (5 mentions)**
   - Context: Details how users can compare different crawls to track progress on resolving SEO issues and see changes between different stages of the website, providing a thorough auditing mechanism.

10. **Scheduled Audits (5 mentions)**
   - Context: Mentions the ability to schedule crawls at chosen intervals, automate them via command line, and export crawl data to locations like Google Sheets, facilitating ongoing SEO monitoring and reporting.]

Writing titles & meta descriptions

I’ve tested writing titles and meta descriptions out while crawling with ChatGPT two different ways. At first, I tried having ChatGPT analyze existing meta data and rewrite it to be better. While the responses did follow instructions for the most part, I didn’t find them particularly helpful (or any better than human written meta data).

Next, I tested having ChatGPT write title tags and meta descriptions based on the body text. This I found much more useful since ChatGPT was actually analyzing the page copy instead of trying to guess what might make a better title tag based solely on the tag itself.

I wouldn’t suggest using ChatGPT to write all your website title tags and meta descriptions. But it can be a great brainstorming tool and a way to fill gaps. For example, if you have an ecommerce website with thousands of missing meta descriptions for product pages – why not just have ChatGPT write them? In most cases it will be better than leaving them blank/missing.

Rewriting duplicate content

Do you have tons of duplicate content and not enough time to make it unique? I found prompts like the one below particularly helpful when optimizing geographic specific landing pages:

Write an introduction based on this text to make it more specific about the city mentioned and include references to local landmarks and fun things to do

Below is a portion of the response (because it was super long) I got when using the prompt above on this page homedepot.com/services/l/mi/detroit/heating-air-conditioning-repair/673dcc2c8 with gpt-4-turbo:

Welcome to Detroit, the vibrant heart of Michigan, where the blend of historical richness and modern innovation creates a unique urban experience. If you're a resident or planning to settle in this dynamic city, maintaining a comfortable home environment is essential, especially given Detroit's varied climate. Fortunately, The Home Depot in Detroit is your go-to destination for top-notch heating and air conditioning repair services.

Similar to meta data, I wouldn’t recommend using generative AI copy on your website from an SEO perspective. But if you’re website is all duplicate content anyhow then using ChatGPT can be a great starting point for making it unique. I encourage you to have a person proofread and edit anything from ChatGPT prior to publishing (but like, do whatever you want lol).

Proofreading

Sitting on a mountain of subpar content that needs to be combed through? Ask ChatGPT to proofread your text and call our misspellings and grammatical errors with the prompt below:

List any typos, misspellings and grammatical errors you can find in this text

Personally, I don’t like asking ChatGPT to fix the errors because the larger responses will take more credits/cost more money. Instead, I like to have ChatGPT call out the issues for a person to go back and fix.

Writing schema

I’ve tested a few different prompts for writing schema, but the two I think were the most useful were prompts for Product and Article schema. For example:

Write product schema in JSON-LD format for any products you find in this test

From my experience ChatGPT struggles a little when it comes to writing schema. Some of the issues I’ve observed are:

• Not including the @context or @type fields
• Including the word ‘json’ unnecessarily within the code
• Making up information (often reviews and ratings)
• Mistaking non-product content as a product
• It has a difficult time selecting URLs and image URLs

Social media post

If you’re short on hands and have multiple hats to wear – crawling with ChatGPT can be a huge help. Need help writing social media posts to drive traffic to various pages on your website? Crawling with ChatGPT makes it easy. Using a prompt like:

Write a social media post summarizing this text in a 1 to 3 sentences and entice people to visit the website to learn more

is a great starting point. Here’s a response to the prompt above for this article using the 3.5 model:

[Are you considering blocking ChatGPT from crawling your website? Learn how to do it and the reasons for and against in this insightful article by Ken Freel. Visit the Nuaveu website to read more! #ChatGPT #AI #WebsiteCrawling #NuaveuWebsite]

From here, you can custom the prompt to the social media platform(s) you want ChatGPT to write for. You can add character limits, suggest using/not using emojis and other parameters to meet your needs. For example, the updated prompt below:

Write a social media post summarizing this text in a 1 to 3 sentences and entice people to visit the website to learn more. Make sure to include at least 1 emoji and 1 hashtag.

Produced this response for the same article using gpt-4-turbo:

[🤖 Wondering if you should block ChatGPT from crawling your website? There are pros and cons to consider, from protecting your content to maximizing your SEO and exposure. Check out the latest guide by Ken Freel on how to do it and why you might want to! Dive in for more details ➡️ [Link to website] #AI #SEO]

Email marketing/sales messaging

Email marketing and other sales messaging is easy to create when crawling with ChatGPT. Similar to the social media post prompts above, simply explain what type of messaging you’re looking for to drive people to visit the page. For example:

Write a sales email enticing a local business owner to visit this web page and purchase the services mentioned in this text

Produced the response below when tested on Mailchimp’s Email Marketing page using model gpt-3.5-turbo:

[Dear [Local Business Owner],

Are you looking to take your email marketing to the next level and drive more sales for your business? Look no further than Mailchimp, the industry-leading email marketing platform trusted by over 12 million users globally.

With up to 39x ROI seen by Mailchimp users, it's no wonder we're the #1 email marketing and automations platform*. Our platform is easy for beginners to use and powerful for sophisticated marketers, offering everything you need to send professional, engaging marketing emails.

Unlike other platforms, Mailchimp ensures that your marketing emails get delivered with top-ranked email deliverability, meaning more of your emails reach your customers' inboxes and not their junk folders.

With all your marketing data in one place, Mailchimp becomes your single source of truth for marketing activities. Plus, our powerful AI and data tools can help you craft better-performing emails and optimize your campaigns for maximum impact.

We offer personalized onboarding services, 24/7 customer support, and a community of experts ready to help you with your campaigns. And with a 1-month risk-free trial, there's no reason not to give Mailchimp a try today.

Visit [Web Page Link] to start your free trial and see why millions of users trust us with their email marketing. Don't miss out on the opportunity to boost your ROI and drive sales with Mailchimp.

Best regards,

[Your Name]

*Based on December 2023 publicly available data on competitors' number of customers.]

Summary

ChatGPT and its API are incredibly powerful tools and Screaming Frog’s integration has made the latter much more accessible. With absolutely no coding knowledge and a minimal budget you have the power to crawl the web with the power of AI at your side. I hope you found this prompt ideas helpful and I look forward to sharing more soon!

---

Fun fact: the API calls made to provide the examples in this article only costed me 9 cents.

• Mail
• LinkedIn
• YouTube
• Instagram