I previously connected Gemini to Screaming Frog to review how clearly pages explain their topics and to identify structured data opportunities. The same workflow works with OpenAI and Anthropic: Screaming Frog sends selected page data to a model during a crawl, then displays the response beside its SEO data.
Here is a practical way to set it up and use it.
Understand the connection
This guide uses Screaming Frog’s direct AI API integration. You write a prompt inside SEO Spider, choose what page data to send, and receive a response for each crawled URL. Screaming Frog supports Gemini, OpenAI, and Anthropic for this workflow. Screaming Frog
When I say “ChatGPT” below, I mean an OpenAI model accessed through the OpenAI API. A ChatGPT subscription does not pay for OpenAI API usage. Screaming Frog
What you need
- Screaming Frog SEO Spider
- An API key for each provider you want to use
- A small set of test URLs
- One clear SEO question to ask about those URLs
| AI service | Create an API key | Menu in Screaming Frog |
|---|---|---|
| Gemini | Google AI Studio | Config > API Access > AI > Gemini |
| ChatGPT models | OpenAI Platform | Config > API Access > AI > OpenAI |
| Claude models | Anthropic Console | Config > API Access > AI > Anthropic |
Each provider has its own API billing and rate limits. Check those in your account before running prompts across a large site. Screaming Frog
Step 1: Connect a provider
In Screaming Frog, open Config > API Access > AI and select a provider. Paste its key into the Account Information tab, then click Connect.
Start with one provider. Once you have a useful prompt and a successful test crawl, you can repeat the setup with the other two. Sending every URL to all three models immediately creates more API calls and more results to review. Screaming Frog
Step 2: Choose the page data
Open Prompt Configuration and click Add. Select a conversational model and choose the data it should receive.
For content analysis, start with Page Text. Use HTML when the model needs to inspect markup. You can also use custom extraction or combine several inputs in an advanced prompt. If you select Page Text or HTML, enable Store HTML under Config > Spider > Extraction. Check the extracted text in a test, since the Page Text setting uses Screaming Frog’s content area rules. Screaming Frog
Name the prompt for the task, such as Missing page facts or Schema evidence check. That name will make its output easier to find.
Step 3: Write one focused prompt
Here is a prompt based on the kind of page clarity check I used with Gemini. Select Page Text as the input:
Review the supplied page text as an SEO editor. In no more than 70 words, state: (1) the main entity or topic; (2) the user question this page answers; (3) one important fact a reader might still need; (4) whether the page has a clear definition and a direct answer. Use only the supplied page. If evidence is missing, say “not found in supplied text.” Do not claim that this page appears in, or is cited by, an AI search product.
This prompt helps find pages that may need clearer facts and answers. Its result is an editorial assessment of the page, not a measurement of whether ChatGPT or another AI service mentions it.
For a structured data review, select HTML and try this:
Examine the supplied HTML. Identify the visible main content and any JSON-LD types you can actually see. Suggest at most two relevant schema types supported by the visible content. For each, name the on-page evidence needed. If that evidence is absent, say “do not add yet.” Do not invent ratings, prices, FAQs, authors, or business details. Return a short recommendation, not finished JSON-LD.
Treat the response as a task to verify. Before adding markup, check the live page and the relevant eligibility rules.
Step 4: Test it on two URLs
Click the play icon beside the prompt, enter a URL, and press Test. Screaming Frog shows both the extracted input and the model’s response. If the input is empty, check Store HTML and the content area settings. If the response invents facts, make the prompt more restrictive. Screaming Frog
Test a second page type too. A prompt that works on an article may give poor answers on a product page.
Step 5: Run a small crawl
Crawl a short URL list or one section of the site. When the crawl and API processing finish, review the AI tab. The prompt results also appear alongside crawl data in the Internal tab, where you can filter and export them. Screaming Frog
For a larger project, use Config > Segments to limit a prompt to relevant URLs or issues. For example, run a meta description prompt only where descriptions are missing. You can attach the segment through the cog beside the prompt. The provider’s Advanced tab lets you adjust requests per minute if you hit a rate limit. Screaming Frog
My review process is simple: sort the responses, check examples from each page template against the actual pages, and turn confirmed findings into tasks. For checks such as status codes, canonicals, or whether a tag exists, use Screaming Frog’s standard reports first.
Three more prompts to try
Find unanswered customer questions
Based only on the supplied page text, list up to three concrete questions a prospective customer may still have. Do not include questions already answered. If the page answers the likely core questions, write “no clear gap found.”
Classify page intent
Classify the page’s primary intent as informational, commercial investigation, transactional, or navigational. Return the label and one sentence citing a phrase or feature from the supplied text. If the page mixes intents, mention the main conflict.
Draft a meta description for review
Write one accurate meta description based only on the supplied page text. Aim for no more than 155 characters including spaces. Mention a real benefit or distinguishing detail when available. Do not invent offers, prices, or guarantees. Return only the description.
Screaming Frog recommends reviewing generated descriptions before publishing. For straightforward extraction, its custom extraction tools can also avoid API costs. Screaming Frog
What about connecting Claude to control Screaming Frog?
Screaming Frog also has an MCP server. This is a separate workflow: an assistant such as Claude Desktop can access crawls, reports, and other Spider functions. It requires a paid SEO Spider licence and database storage mode. Screaming Frog documents both a headless setup and one that keeps the Spider interface visible. Use the direct Anthropic API integration above when you simply want Claude to answer a prompt for each page. Screaming Frog
My recommended first run
Choose one provider and 10 to 20 URLs from the same page template. Run one prompt, test two pages first, and compare the results with what those pages actually say. When the answers are useful and repeatable, save the prompt and expand the crawl to a relevant segment.
The goal is a short, verifiable list of SEO improvements. An AI response for every URL is only useful when you can act on what it finds.