How To Create FAQs At Scale With Screaming Frog + OpenAI API - Nectiv

So one of the biggest initiatives that a lot of SEOs are starting to utilize to optimize for LLMs is creating FAQ content to add to their landing pages. This is one of the tactics a lot of people are seeing some early success with.FAQs are generally seen to be a very strong element to utilize is because they contain extremely “dense” content. Basically FAQs can self-contain entire pages in very few words. From a technical standpoint, these sections generally have very strong cosine similarity to a give search query/prompt.

Probably the best research around this topic was done by Chris Green in this article. He basically took different content types (dense prose, FAQs, structured content) and used different models to measure the similarity of each content type against five different search queries.

You can see the results below:

The overall takeaways was that in almost all models, FAQ content had the highest degrees of similarity. While not definite, AI engines might gravitate to these sections more. AI systems might have a “grounding budget” (Google’s is estimated to be 2K words per query) which means they need to understand a whole piece of content with a relatively small amount of text.

As it turns out, FAQ content is amazing at doing just this.

Creating FAQs At Scale With Screaming Frog

So we’re getting asked a lot about how to create FAQ content for pages. While I’m personally still not completely onboard the “scale AI content at all costs” train, it certainly does have its time and place when used responsibly + human edited. FAQs feel like a pretty safe space for AI content as it’s meant to summarize and quickly answer as opposed to add unique thoughts or perspectives. For this reason, I feel pretty OK with using AI to create this type of content at a fast pace.

So how do you go about creating FAQs for the different landing pages on your site? Well the good news is that our old friend Screaming Frog provides a FANTASTIC way to be able to do this. Screaming Frog has direct integrations with the OpenAI API. This means that you can push your entire content through ChatGPT at scale.

Let’s get started with our Screaming Frog configuration:

First you’re going to open up Screaming Frog and navigate to Configuration > Extraction. Here you’ll ensure that both “Stored HTML” and “Store Rendered HTML” are checked on.

Next, we’ll need to connect Screaming Frog to our LLM of choice. I tested a few different models for this process and OpenAI consistently gave me the best results so I went with that. To do this, navigate to Configuration > API Access > AI > OpenAI. You’ll want to grab an API key from OpenAI and plug it in the “API Key” field.

Once connected, you’ll then go to “Prompt Configuration”. Here’s where you’re going to add in the prompt you want to feed into OpenAI. Select “Add” to configure it. I used the following attributes for my prompt:

Name: FAQs
Category: ChatGPT
Model: gpt-5-mi
Content Type: HTML
Prompt Target: Page Text
Prompt: Read the Page Text of the page. After that do two different things: “1. Create five questions someone might have about this content 2. Write an answer to each question. Keep the answers below five sentences“

Now, click the “Play” button next to the prompt field. This opens up the “Prompt Tester” dialog box. You can copy/paste any URL from the site and test out what the response will look like before running this at scale:

Finally, go ahead and paste in your domain into the address bar and perform a crawl in Spider mode. As always, Screaming Frog will begin to crawl your site. However, now it will start pushing your content through the API and provide responses to the prompts you give it.

You can navigate to the right-hand sidebar and find the “AI” report. Navigate to the FAQ tab and you’ll be able to see the FAQs that Screaming Frog was able to create. In just 5 minutes, Screaming Frog was able to create 150+ FAQs that we can then choose to add to the page or not:

From there you can directly export the FAQs by hitting the “Export” button. You’ll then be able to see a spreadsheet with all of your FAQs from OpenAI.

Is This More AI Slop?

Fair question. To me the answer is a resounding “no” for several reasons. Probably the biggest one is that FAQs by nature are more summary based as opposed to educational-based. Oftentimes, FAQs are putting the information already available on a page in a condensed, easy to read format that gives users a TLDR of the whole page. Basically you’re using AI as a summarization agent as opposed to something that regurgitating the same content throughout the web.

Here’s an example on the Kit FAQs that we created here. On the Forms Feature page, it created several different FAQs such as “How do I add Kit forms to my website or landing pages?” and “Will I be charged multiple times if a subscriber signs up through different forms?”.

This information is available throughout the page but harder to parse out as questions someone might have.

However, these are nicely translated into FAQs that someone might actually be asking about.

Now that being said, there are certain limitations and rules that I would apply to this:

As with any generative AI content, it needs to be reviewed by a human. This is a starting point, not the ending point.
This process works much better on longer pages. The FAQs that it draws from are pretty limited on pages with low word counts as there isn’t a lot of source content
FAQs will be more valuable on some page types than others. Guides, features, use cases etc are all good examples of where FAQs could help.