ChatGPT Prompt Ideas for Screaming Frog

Looking for ways to take advantage of Screaming Frog’s new ChatGPT integration? You’re in the right place! Learn how to get started below or jump right to the good stuff.

Screaming Frog Version 20

In case you missed it, Screaming Frog’s release of version 20 on May 7th was a huge deal in the SEO world. This release added a handful of incredible features. But the one that stood out the most is their integration with ChatGPT’s API. But first let’s get a ChatGPT API account and key setup.

Please note: all the stuff I talk about in this article requires a paid license for Scream Frog.

How to setup a ChatGPT API account & key

ChatGPT and API accounts are separate. You can use the same email for both, but you will need to setup an API account even if you already use ChatGPT’s conversational interface.

Start by going to and clicking sign up in the top right. Enter an email and password or use an existing Google, Microsoft or Apple account. Verify your email address and provide your name, organization (optional) and your birthday. Next you will see a screen with two account options – ChatGPT and API. Select API and you’re in.

New accounts receive $5 in free API credits, but only if the phone number used for verification hasn’t been used before. Based on my testing, $5 will actually get your pretty far. But if you’ve already used your free credits (or unknowingly let them expire like me) click on Settings and then over to Billing. From here you can add a payment method and prepay for credits. Personally, I found the prepay feature really helpful since it can be difficult to forecast usage and costs while testing.

To setup an API key, click API keys in the menu on the left and then + Create new secret key. Leave everything as default for now and just give the key a name. Make sure to keep a copy of your secret key as you will not be able to view it again. I recommend just throwing it into a text file somewhere safe, but accessible. If you lose your secret key, no worries – you can always delete the API key and create a new one.

OpenAI account options
Open AI account creation options
Surreal design of a frog with spider webs generated by ImageFX

Configuring your custom JS snippets

 Instead of me rewriting Screaming Frog’s user guide section on Custom JavaScript I’m just going to give you the TL;DR version:

  1. Enable JavaScript rendering by going to Spider > Crawl > Rendering
  2. Click Configuration > Custom > Custom JavaScript
  3. Click + Add from Library
  4. Select (ChatGPT) Template*
  5. Click the JS button with the pen icon to the right of the green checkmark
  6. Copy and paste your API key secret onto line 25
  7. Adjust the prompt on line 26 as needed

From here I encourage you to test your heart out with individual URLs using the JavaScript tester in the editor. By default, ChatGPT will use the body text of the page when responding to prompts. But check out the comments in the JavaScript Editor from Screaming Frog for details on how to analyze the title tag, meta description and other HTML tags.

* You should also check out the 6 default ChatGPT snippets provided by Screaming Frog under the system tab. I found that image alt text one particularly useful myself.

ChatGPT prompt ideas for Screaming Frog

Check out these seven custom ChatGPT prompt ideas, with examples, to use with Screaming Frog’s custom JavaScript snippets.

Top keywords

One of my favorite uses for crawling with ChatGPT so far is extracting keywords from pages. But what I found particularly interesting and helpful is the ability to have ChatGPT include the context for how the keywords were being used. The most effective way I found for doing this is using the prompt below with the gpt-4-turbo model.

I often mention NOT including a summary when using the gpt-4-turbo model because it has a weird penchant for over explaining responses. Verbose responses aren’t only an annoyance when using the API – they actually cost more too. Furthermore, if you’re getting too much contextual “fluff” it can help to turn the temperature down (I personally like 0.4 or 0.5).

Here’s an example of a response from the prompt above when crawling

[1. **SEO Spider Tool (23 mentions)**
   - Context: Described as a comprehensive website crawler that helps improve onsite SEO by auditing for common issues. It offers features like crawling JavaScript-rich websites, finding broken links, auditing redirects, and integrating with analytics tools.

2. **Crawl Configuration (12 mentions)**
   - Context: Refers to the various settings and options available within the SEO Spider Tool that allow users to customize their crawling strategies, including adjusting crawl depth, obeying or ignoring specific directives, and managing how JavaScript is rendered.

3. **Integration with Analytics (9 mentions)**
   - Context: Highlights the capability of the SEO Spider Tool to integrate with Google Analytics, Google Search Console, and Google PageSpeed Insights to fetch performance data directly during crawls for enhanced analysis.

4. **Custom Extraction (8 mentions)**
   - Context: Discusses the feature within the SEO Spider Tool that allows users to extract data using XPath, CSS Path selectors, or regex from the HTML of the pages crawled.

5. **JavaScript Rendering (7 mentions)**
   - Context: Points to the SEO Spider Tool's ability to render web pages using Chromium WRS to accurately crawl dynamic, JavaScript-driven websites and frameworks like Angular, React, and Vue.js.

6. **XML Sitemap Generation (6 mentions)**
   - Context: Focuses on the functionality within the SEO Spider Tool to quickly generate XML Sitemaps and Image XML Sitemaps, providing advanced configuration for URLs to include, which helps in optimizing site indexing.

7. **Duplicate Content Detection (6 mentions)**
   - Context: Describes how the SEO Spider Tool can discover exact and near duplicate pages using advanced algorithmic checks, which is crucial for SEO to avoid content redundancy issues.

8. **Visualise Site Architecture (5 mentions)**
   - Context: Explains the feature of visualizing internal linking and URL structure through interactive crawl and directory force-directed diagrams and tree graph site visualizations, helping users understand site layout better.

9. **Crawl Comparison and Staging (5 mentions)**
   - Context: Details how users can compare different crawls to track progress on resolving SEO issues and see changes between different stages of the website, providing a thorough auditing mechanism.

10. **Scheduled Audits (5 mentions)**
   - Context: Mentions the ability to schedule crawls at chosen intervals, automate them via command line, and export crawl data to locations like Google Sheets, facilitating ongoing SEO monitoring and reporting.]

Writing titles & meta descriptions

I’ve tested writing titles and meta descriptions out while crawling with ChatGPT two different ways. At first, I tried having ChatGPT analyze existing meta data and rewrite it to be better. While the responses did follow instructions for the most part, I didn’t find them particularly helpful (or any better than human written meta data).

Next, I tested having ChatGPT write title tags and meta descriptions based on the body text. This I found much more useful since ChatGPT was actually analyzing the page copy instead of trying to guess what might make a better title tag based solely on the tag itself.

I wouldn’t suggest using ChatGPT to write all your website title tags and meta descriptions. But it can be a great brainstorming tool and a way to fill gaps. For example, if you have an ecommerce website with thousands of missing meta descriptions for product pages – why not just have ChatGPT write them? In most cases it will be better than leaving them blank/missing.

Rewriting duplicate content

Do you have tons of duplicate content and not enough time to make it unique? I found prompts like the one below particularly helpful when optimizing geographic specific landing pages:

Below is a portion of the response (because it was super long) I got when using the prompt above on this page with gpt-4-turbo:

Welcome to Detroit, the vibrant heart of Michigan, where the blend of historical richness and modern innovation creates a unique urban experience. If you're a resident or planning to settle in this dynamic city, maintaining a comfortable home environment is essential, especially given Detroit's varied climate. Fortunately, The Home Depot in Detroit is your go-to destination for top-notch heating and air conditioning repair services.

Similar to meta data, I wouldn’t recommend using generative AI copy on your website from an SEO perspective. But if you’re website is all duplicate content anyhow then using ChatGPT can be a great starting point for making it unique. I encourage you to have a person proofread and edit anything from ChatGPT prior to publishing (but like, do whatever you want lol).


Sitting on a mountain of subpar content that needs to be combed through? Ask ChatGPT to proofread your text and call our misspellings and grammatical errors with the prompt below:

Personally, I don’t like asking ChatGPT to fix the errors because the larger responses will take more credits/cost more money. Instead, I like to have ChatGPT call out the issues for a person to go back and fix.

Writing schema

I’ve tested a few different prompts for writing schema, but the two I think were the most useful were prompts for Product and Article schema. For example:

From my experience ChatGPT struggles a little when it comes to writing schema. Some of the issues I’ve observed are:

  • Not including the @context or @type fields
  • Including the word ‘json’ unnecessarily within the code
  • Making up information (often reviews and ratings)
  • Mistaking non-product content as a product
  • It has a difficult time selecting URLs and image URLs

Social media post

If you’re short on hands and have multiple hats to wear – crawling with ChatGPT can be a huge help. Need help writing social media posts to drive traffic to various pages on your website? Crawling with ChatGPT makes it easy. Using a prompt like:

is a great starting point. Here’s a response to the prompt above for this article using the 3.5 model:

[Are you considering blocking ChatGPT from crawling your website? Learn how to do it and the reasons for and against in this insightful article by Ken Freel. Visit the Nuaveu website to read more! #ChatGPT #AI #WebsiteCrawling #NuaveuWebsite]

From here, you can custom the prompt to the social media platform(s) you want ChatGPT to write for. You can add character limits, suggest using/not using emojis and other parameters to meet your needs. For example, the updated prompt below:

Produced this response for the same article using gpt-4-turbo:

[🤖 Wondering if you should block ChatGPT from crawling your website? There are pros and cons to consider, from protecting your content to maximizing your SEO and exposure. Check out the latest guide by Ken Freel on how to do it and why you might want to! Dive in for more details ➡️ [Link to website] #AI #SEO]

Email marketing/sales messaging

Email marketing and other sales messaging is easy to create when crawling with ChatGPT. Similar to the social media post prompts above, simply explain what type of messaging you’re looking for to drive people to visit the page. For example:

Produced the response below when tested on Mailchimp’s Email Marketing page using model gpt-3.5-turbo:

[Dear [Local Business Owner],

Are you looking to take your email marketing to the next level and drive more sales for your business? Look no further than Mailchimp, the industry-leading email marketing platform trusted by over 12 million users globally.

With up to 39x ROI seen by Mailchimp users, it's no wonder we're the #1 email marketing and automations platform*. Our platform is easy for beginners to use and powerful for sophisticated marketers, offering everything you need to send professional, engaging marketing emails.

Unlike other platforms, Mailchimp ensures that your marketing emails get delivered with top-ranked email deliverability, meaning more of your emails reach your customers' inboxes and not their junk folders.

With all your marketing data in one place, Mailchimp becomes your single source of truth for marketing activities. Plus, our powerful AI and data tools can help you craft better-performing emails and optimize your campaigns for maximum impact.

We offer personalized onboarding services, 24/7 customer support, and a community of experts ready to help you with your campaigns. And with a 1-month risk-free trial, there's no reason not to give Mailchimp a try today.

Visit [Web Page Link] to start your free trial and see why millions of users trust us with their email marketing. Don't miss out on the opportunity to boost your ROI and drive sales with Mailchimp.

Best regards,

[Your Name]

*Based on December 2023 publicly available data on competitors' number of customers.]


ChatGPT and its API are incredibly powerful tools and Screaming Frog’s integration has made the latter much more accessible. With absolutely no coding knowledge and a minimal budget you have the power to crawl the web with the power of AI at your side. I hope you found this prompt ideas helpful and I look forward to sharing more soon!

Fun fact: the API calls made to provide the examples in this article only costed me 9 cents.

Scroll to Top