SearchFlow documentation

Overview

SearchFlow is workflow infrastructure for SEO intelligence. Instead of stitching crawlers, rank trackers, and scrapers together by hand, you call a single API that orchestrates the right data providers, applies expert methodology, and returns structured results.

Workflows are self-contained units with clear inputs and outputs. You can chain them in your own pipelines, schedule them, or hand them to AI agents that need SEO context.

• API-first — plain HTTP, JSON in and out.
• Async by design — long-running work never blocks the request.
• Observable progress — poll a run for status, step, and percentage.
• Partial resilience — per-keyword errors are collected without failing the whole run.

For AI agents

If you are an AI agent reading this to call SearchFlow — through the HTTP API, an MCP server, or any other tool — the rules in this section are part of the contract, not a suggestion.

Every workflow run debits credits from the user's balance the moment it is created. Never create a run before telling the user exactly how many credits it will cost and getting an explicit confirmation. This is the same confirmation step the dashboard playground shows before it starts a run, and it applies identically to you.

• Compute the cost from the payload first — the formulas below are exact, so you never have to guess.
• State the workflow, the number of billed items, the billing mode, and the resulting credit cost, then ask.
• Wait for an explicit yes. An earlier 'go ahead', silence, or a confirmation given for a different run is not consent.
• Re-confirm whenever the payload changes — more keywords, thematic clustering switched on, a longer URL list — because the cost changes with it.
• Never split one job into several smaller runs to make each one look cheaper.
• Report the actual cost back to the user once the run is created.

Ready to run keyword clustering on 2,400 keywords with thematic
clustering enabled. This will debit 108 credits from your balance
when the run is created (45 credits per 1,000 keywords, semantic
clustering). Confirm?

Reading a run you already created is free — only creating a run costs credits, so polling GET /runs/{run_id} needs no confirmation.

Credit cost

The cost of a run is fully determined by its payload, so it can be computed before the request is sent. The API is the source of truth: it debits the same amount when the run is created, and refunds it automatically if the run fails.

A run that completes with partial errors is not refunded — the work was done. A run that fails outright is refunded in full.

• Keyword clustering — 40 credits per 1,000 keywords, 45 per 1,000 when thematic_clustering.enabled is true. Prorated: 250 keywords with thematic clustering cost 11.25 credits.
• Indexation check — 40 credits per 1,000 URLs, 45 per 1,000 when categorization.enabled is true. Prorated the same way.
• E-commerce category content — 2 credits per query, not prorated. 40 queries cost 80 credits.
• Product content — 1 credit per product plus 0.5 per field written. Reading the SERP costs the same whether one field is written or five, so it is billed once; generation is billed per field. A full five-field sheet is 3.5 credits per product, a titles-only run is 1.5. Switch off the fields you do not need.
• Keyword research — 80 credits per study, plus 10 per batch of 20 seeds, plus 20 per ranked source, plus 55 per 1,000 keywords of max_keywords. A study is one country and one language; two markets are two runs. A seed batch is what the Google Keyword Planner takes in one task, so 1 to 20 seeds is one batch and 21 to 40 is two — and a run that declares no seed still pays for one, because the model writes them. A source is a competitor or the studied site itself when you pass a domain and leave include_site_keywords on; discovery only fills the seats you leave empty, so naming 4 competitors and asking for 2 more still bills 4. So: a domain, 2 competitors, 20 seeds, max_keywords 2000 = 80 + 10 + 20x3 + 55x2 = 240 credits.
• Rank or create — 2 credits per query and per market, not prorated. A site declared with one market bills 2 credits per query; a site declared with a French and a Dutch market bills 4, because every query is analysed once per market. 20 queries on a single-market site cost 40 credits.
• Coverage gap analysis — 45 credits per 1,000 keyword slots, prorated. A slot is one keyword row the run may read, and the count is markets × (competitors + 1) × keywords_per_domain — the site's own domain is the +1. Named competitors replace discovery, so a run with 3 named competitors and keywords_per_domain 500 on one market is 4 × 500 = 2,000 slots = 90 credits. A discovery run is billed on competitor_discovery.limit, because what discovery will find is not knowable before the call.
• Local SEO pages — 3 credits per city, not prorated. One city is one page. 20 cities cost 60 credits. The rate is flat: it covers the local pack, the Google Maps listing, the competitor extraction and the generation, and it does not change with the number of sections, FAQ entries or businesses requested.
• Informational content — 4 credits per article plus 1 per 1,000 words of the length.max_words ceiling, rounded up. Omit the length block and the ceiling is 1,400 words, so an article costs 6 credits; set max_words to 1,000 or less and it costs 5. The detected article format never changes the price — only what you send does.
• Technical crawl — 4 credits per run, plus 20 per 1,000 pages of crawl.max_pages, plus 0.2 per JavaScript-rendered page. The rendered sample is ceil(max_pages × javascript_rendering.percent / 100), capped at 200. A 1,000-page crawl with the default 5% sample is 4 + 20 + 10 = 34 credits.
• Technical crawl bills the crawl BUDGET, not the pages found: crawl.max_pages is charged even if the site turns out to be smaller. Set it to what you actually want read. The same applies to the rendered sample in mode 'auto' — it is billed up front and skipped when no JavaScript framework is detected, so send mode 'off' if you know the site is server-rendered and want to pay nothing for it.
• Discover topics — 4 credits per theme plus 0.6 per exploration keyword. Expanding the theme and analysing the coverage cost the same at any depth, so they are billed once per theme; the Google News reads follow exploration.keywords. The default of 10 keywords is 10 credits per theme; the range is 3 to 30, so 5.8 to 22 credits. exploration.topics does not change the price.
• Discover article — a flat 5 credits per article, plus 9 more per article when image.generate is true. The image prompt is always returned and is never billed; the 9 credits buy the rendered 2048×1152 cover. Nothing else in the payload changes the price — length, articles_read and source_urls only ever lower the cost the flat rate was set on.
• Cannibalization analysis — 1 credit per analysed query, 1.5 when recommendations.enabled is true (the default). One query is one SERP read for one market, so the same query submitted for two markets is two readings and costs two.
• Indexation check, category content, product content, rank or create, local SEO pages and informational content and both Discover workflows and cannibalization analysis collapse duplicates before billing; keyword clustering does not, so every keyword you send is charged. De-duplicate the list yourself.
• Keyword research is billed on max_keywords, not on what it returns. It is the ceiling on the merged sheet — the only list that gets one SERP read per keyword — so a run that finds fewer relevant keywords still costs what you asked for. Set it to the size of study you want. The two form helpers are billed separately and on delivery: POST /keyword-research/site-description costs 1 credit, POST /keyword-research/brand-variants costs 0.5, and a call that fails costs nothing.

If the balance is too low, the API answers 402 and nothing is queued and nothing is charged — but check the cost with the user before you get there, not after.

Quick start

Create an API key from the dashboard under Usage & billing, then call a workflow endpoint. Every workflow follows the same pattern: create a run, poll until the status is completed or failed, then read the result.

• Generate an API key at /app/usage.
• POST to the workflow runs endpoint with your payload.
• Receive 202 Accepted with a run_id.
• GET the run until status is completed or failed.

curl -X POST https://api.searchflow.app/api/v1/workflows/keyword-clustering/runs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "keywords": [
      { "keyword": "agence seo", "volume": 1000 },
      { "keyword": "agence seo paris", "volume": 900 }
    ],
    "locale": { "country": "fr", "language": "fr" },
    "serp_similarity_threshold": 0.4,
    "thematic_clustering": { "enabled": true, "label_with_ai": true },
    "domain_tracking": { "enabled": false }
  }'

Authentication

Authenticate with a Bearer token using your SearchFlow API key. Keys are created in the dashboard and can be rotated at any time.

When a valid API key is provided, requests are attributed to your account and logged for usage tracking. Workflow runs created with a key are scoped to your user — other authenticated users cannot read them.

Authorization: Bearer sf_live_xxxxxxxx

The dashboard playground authenticates with your Supabase session (Bearer JWT). External integrations should use an API key (sf_live_…).

MCP server

SearchFlow is available as a hosted MCP server, so an AI assistant can run workflows directly. It is remote — nothing to install — and authenticates with the same API key as the HTTP API. Add the URL below to any MCP client that supports remote servers with bearer authentication.

Five tools cover every workflow, because they all share one contract. The usual sequence is searchflow_list_workflows, then searchflow_get_workflow_schema for the one you want, then searchflow_estimate_cost on the payload you built, then searchflow_start_run.

• searchflow_list_workflows — every workflow your account can run.
• searchflow_get_workflow_schema — the JSON Schema of one workflow's payload.
• searchflow_estimate_cost — prices a payload and reports your balance. Charges nothing.
• searchflow_start_run — creates the run. This debits credits.
• searchflow_get_run — status, progress and result. Free, and safe to poll.

{
  "mcpServers": {
    "searchflow": {
      "url": "https://api.searchflow.app/mcp",
      "headers": {
        "Authorization": "Bearer sf_live_xxxxxxxx"
      }
    }
  }
}

searchflow_estimate_cost validates through the same request model the run uses, so the figure it returns is the cost of the run that will actually execute — duplicates already collapsed. Quote it to the user and get a yes before calling searchflow_start_run.

How workflows run

Workflows are designed for work that can take seconds to hours. A POST request validates your input, enqueues the job, and immediately returns 202 Accepted with a run_id.

Execution happens in background workers backed by a Redis queue. State is persisted in Supabase so you can poll reliably across restarts.

• queued — accepted and waiting for a worker.
• running — pipeline in progress; check progress and message fields.
• completed — result is available on the run object.
• failed — the workflow could not finish; see errors for details.

Keyword clustering

Groups keywords by SERP overlap, then optionally builds thematic clusters with embeddings and AI-generated labels. This is the first production workflow available on SearchFlow.

The pipeline fetches SERP data (with caching when available), clusters keywords by URL overlap similarity, and — when thematic clustering is enabled — generates embeddings and groups SERP clusters into higher-level themes.

• SERP fetch — retrieves search results per keyword for the given locale.
• SERP clustering — groups keywords sharing overlapping ranking URLs.
• Embeddings — OpenAI text-embedding-3-small on unique keywords (thematic mode).
• Semantic clustering — HDBSCAN groups SERP clusters by embedding similarity.
• AI labeling — optional human-readable labels per thematic cluster.

Domain tracking adds your site's ranking position and URL per keyword inside each SERP cluster when enabled.

Keyword clustering inputs

Send a JSON body to POST /api/v1/workflows/keyword-clustering/runs. Keywords are normalized to lowercase; at least one non-empty keyword is required (up to 100,000).

• serp_similarity_threshold — float between 0.1 and 1.0 (default 0.4). Higher values require more SERP overlap to cluster together.
• thematic_clustering.enabled — when true, runs embedding + semantic grouping on top of SERP clusters.
• thematic_clustering.label_with_ai — when true, generates a short label per thematic cluster.
• domain_tracking — when enabled, domain is required; adds keyword_position and keyword_url per keyword.

{
  "keywords": [
    { "keyword": "seo audit", "volume": 1200 },
    { "keyword": "technical seo checklist", "volume": 480 }
  ],
  "locale": {
    "country": "us",
    "language": "en",
    "geolocation": null
  },
  "serp_similarity_threshold": 0.4,
  "thematic_clustering": {
    "enabled": true,
    "label_with_ai": true
  },
  "domain_tracking": {
    "enabled": true,
    "domain": "example.com"
  }
}

Keyword clustering outputs

On completion, the run result contains SERP clusters and, when enabled, semantic clusters. Partial errors per keyword or stage are returned in errors without necessarily failing the run.

{
  "workflow": "keyword_clustering",
  "locale": { "country": "fr", "language": "fr" },
  "serp_clusters": [
    {
      "id": "serp_1",
      "centroid": "agence seo",
      "keywords": [
        {
          "keyword": "agence seo",
          "volume": 1000,
          "keyword_position": 4,
          "keyword_url": "https://example.com/seo"
        }
      ]
    }
  ],
  "semantic_clusters": [
    {
      "id": "semantic_1",
      "label": "SEO agencies",
      "serp_cluster_ids": ["serp_1"],
      "keywords": ["agence seo", "agence seo paris"]
    }
  ],
  "errors": [],
  "meta": {
    "keyword_count": 42,
    "serp_cluster_count": 8,
    "semantic_cluster_count": 3,
    "serp_similarity_threshold": 0.4,
    "domain_tracking_enabled": false,
    "completed_at": "2026-07-04T10:15:00+00:00"
  }
}

Progress and polling

Poll GET /api/v1/workflows/{workflow}/runs/{run_id} until status is completed or failed. Large keyword lists can take several minutes — use a 2-second interval and a generous timeout.

While running, the response includes progress (0–100), current_step, and message with human-readable status such as fetching SERPs or labeling clusters.

• starting — workflow accepted by a worker.
• serp_fetch — SERP retrieval in progress (keyword clustering).
• serp_clustering — grouping keywords by SERP overlap (keyword clustering).
• indexation_check — site: queries in progress (indexation check).
• embeddings — generating embeddings.
• semantic_clustering / categorization — building groups.
• semantic_labeling / categorization_labeling — AI labels for each group.
• content_extraction — reading the competitor pages (category content).
• content_generation — writing the category texts (category content).
• competitor_discovery / ranked_keywords — reading the competitive set (coverage gap).
• relevance / themes / theme_labeling — filtering and grouping the gaps (coverage gap).

Indexation check

Checks whether Google has indexed each page of a list, then optionally groups the checked URLs into labeled categories. Runs on the short-task workers: expect minutes, not hours.

Each URL is verified with its own site: query. A non-empty result is not proof on its own — site:example.com/blog also returns deeper pages — so a page counts as indexed only when Google returns that exact URL. Scheme, www and trailing-slash variants are treated as the same page.

• Indexation check — one site: query per URL, never served from cache.
• Embeddings — OpenAI text-embedding-3-small on the page title, description and URL path (categorization mode).
• Categorization — HDBSCAN groups the URLs; outliers join their nearest category so every URL is categorized.
• AI labeling — each category gets a broad category and a precise theme.

Duplicate URLs are collapsed before billing: submitting http and https variants of the same page costs one check.

Indexation check inputs

Send a JSON body to POST /api/v1/workflows/indexation-check/runs. At least one valid URL is required (up to 5,000). URLs without a public hostname are rejected with 422.

• urls — list of absolute or bare URLs, 1 to 5,000 entries.
• locale — the Google property queried; indexation is mostly global but the local index is the one your visitors see.
• categorization.enabled — when true, embeds and groups the checked URLs. Adds 5 credits per 1,000 URLs.
• categorization.label_with_ai — when true, names each category with a category and a theme.

{
  "urls": [
    "https://example.com/",
    "https://example.com/blog/seo-audit"
  ],
  "locale": { "country": "fr", "language": "fr" },
  "categorization": { "enabled": true, "label_with_ai": true }
}

Indexation check outputs

The result holds one row per URL and, when categorization is enabled, the list of categories. A URL whose check failed at the provider is reported with status error and is excluded from categorization, without failing the run.

{
  "workflow": "indexation_check",
  "locale": { "country": "fr", "language": "fr" },
  "results": [
    {
      "url": "https://example.com/blog/seo-audit",
      "status": "indexed",
      "matched_url": "https://example.com/blog/seo-audit",
      "title": "SEO audit — the complete guide",
      "description": "How to audit a website.",
      "site_results_count": 1,
      "category_id": "category_1",
      "category": "Blog articles",
      "theme": "Technical SEO audits",
      "error": null
    }
  ],
  "categories": [
    {
      "id": "category_1",
      "category": "Blog articles",
      "theme": "Technical SEO audits",
      "url_count": 12,
      "urls": ["https://example.com/blog/seo-audit"]
    }
  ],
  "errors": [],
  "meta": {
    "url_count": 20,
    "indexed_count": 17,
    "not_indexed_count": 3,
    "error_count": 0,
    "indexation_rate": 0.85,
    "categorization_enabled": true,
    "category_count": 4,
    "completed_at": "2026-08-02T10:15:00+00:00"
  }
}

indexation_rate is computed over the URLs that were actually checked — URLs in error are excluded from the denominator.

E-commerce category content

Writes the two texts a category page needs: a short introduction shown above the product grid, and a long-form text shown at the bottom of the page or behind a 'read more'. It is built for standard retail verticals — fashion, home, garden, sport, pets, DIY — not for YMYL topics.

Each query is researched against the live SERP before anything is written: the top organic results are extracted for the passages that answer the query, and the 'people also ask' and related searches are used to widen the coverage. One run accepts up to 100 queries and processes them in parallel.

• SERP fetch — one batch scrape for every query in the run, cached across runs.
• Content extraction — the top 4 ranking pages are highlighted with Exa, cropped to 4,000 characters each.
• Content generation — gpt-5.6-luna writes both texts under a strict JSON schema.
• Defaults — 50–100 words for the short text, 300–400 words for the long text.
• Output — minimal HTML (h2, h3, p, ul, ol, li, strong, em, br) with no attributes, plain text, or both.

curl -X POST https://api.searchflow.app/api/v1/workflows/ecommerce-category-content/runs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "queries": [
      { "query": "chaussures de randonnée femme" },
      { "query": "sac à dos randonnée 30l", "category_name": "Sacs à dos 30L" }
    ],
    "locale": { "country": "fr", "language": "fr" },
    "output_format": "html",
    "brand": {
      "name": "Alpine Store",
      "tone_of_voice": "Expert, direct, no marketing fluff",
      "banned_words": ["imbattable"]
    }
  }'

Brand context is optional but changes the result substantially: brand name, tone of voice, audience, selling points, constraints, banned words and free-form instructions are all injected into the prompt as customer-supplied data.

Coverage gap analysis

Compares a site with its competitors keyword by keyword, then answers the question a ranking diff cannot: which of those gaps are worth working on. The ranked keywords of your domain and of every competitor are read from DataForSEO Labs, merged into one keyword universe per market, and each keyword is labelled absent, behind or leading from the positions alone.

The business context is what makes the result usable. Declared product lines, services, priorities and excluded topics are embedded, every candidate keyword is scored against them, and anything too far is set aside — a parapharmacy that does not sell mobility equipment gets no wheelchair keywords, however well a competitor ranks for them. Nothing is removed silently: every discarded keyword comes back with its score and the line it was closest to.

• Competitors — the domains you name, or DataForSEO Labs competitors_domain when you name none.
• Ranked keywords — one capped read per domain per market, ordered by search volume so a truncated read is predictable.
• Relevance — text-embedding-3-small scores each keyword against your declared context; an excluded topic beats an included one.
• Themes — HDBSCAN groups the remaining keywords, gpt-5.6-luna names each group.
• Priority — volume, difficulty, how many competitors are already there, and closeness to what you sell. A theme four competitors rank for is a safer bet than one a single competitor holds.
• Output — themes with a status and one recommended action (create, improve, defend, monitor), plus a flat export row per keyword.

curl -X POST https://api.searchflow.app/api/v1/workflows/coverage-gap-analysis/runs \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "site": { "domain": "example.fr" },
    "competitors": ["rival.fr", "other-rival.fr"],
    "keywords_per_domain": 500,
    "business_context": {
      "product_lines": ["soins du visage", "compléments alimentaires"],
      "excluded_topics": ["matériel médical"]
    }
  }'

A site whose market cannot be inferred — a .com, a .be — is refused with 422 rather than analysed against a guess. Declare the country and language, or list your markets, and the analysis is run once per market.

Errors

SearchFlow distinguishes HTTP errors (bad request, not found, service unavailable) from workflow-level errors collected during execution.

Workflow errors are objects with a stage, a message, and the item they concern — keyword for keyword clustering, item for indexation check. A run can complete with a partial result and a non-empty errors array — for example, a single keyword whose SERP fetch failed.

• 401 — missing or invalid API key. Every workflow endpoint requires authentication.
• 402 — insufficient credits for the requested run. Nothing is queued and nothing is charged.
• 404 — run not found, or not accessible to the authenticated user.
• 503 — Supabase or Redis not configured on the API.
• 422 — request validation failed (empty keywords, missing domain when tracking is enabled).

Base URL

All endpoints are prefixed with /api/v1. The current public API base URL is https://api.searchflow.app.

Set NEXT_PUBLIC_API_URL in your environment to match your deployment. The dashboard playground proxies requests through the web app to avoid CORS during browser testing.

When ENABLE_DOCS is enabled on the API, interactive OpenAPI documentation is available at /docs on the API host.