Social listening with an API: a working pipeline

November 8, 2026

Social listening means collecting what people say about a brand, a product or a topic in public, then turning it into numbers a team can act on. With an API, it is a scheduled job: run saved searches, keep each new mention once, enrich the important ones, and count. This page is the full pipeline, with the calls, the cost and the formulas.

Who does this and why

• Brand and comms teams want to see complaints and praise before they spread, and to report share of voice against competitors.
• Product teams mine feature requests and bug reports from Reddit threads, YouTube comments and X replies.
• Agencies run the same pipeline for many clients and need the cost per client to stay predictable.

If you want a no-code version in Google Sheets, read social listening on a budget. For the different kinds of monitoring APIs and a polling loop, see social media monitoring APIs. This page covers the pipeline design and the analysis.

The data you need, by platform

Most video mentions never type the brand name. They say it out loud. Transcripts on the top videos catch mentions that caption search misses.

The pipeline in five stages

1. Queries. Write one query per brand spelling, product name and campaign hashtag. Add the common misspellings. On X, use search operators to cut noise; our post on X search operators lists the useful ones.
2. Collect. Run every query on every platform on a schedule, sorted by latest where the endpoint allows it. Take the first page each run.
3. Dedupe. Key each mention on platform plus post id. A mention you have seen is skipped, so a second run only adds new rows.
4. Enrich. For the top mentions by engagement, pull one page of comments and, for videos, the transcript.
5. Classify and count. Tag each mention with sentiment and topic (a small LLM prompt or a keyword list works), then compute the metrics below once a day.

Worked example: the monthly cost

A coffee brand tracks 6 queries on 6 platforms (X, TikTok, Instagram, YouTube, Reddit, LinkedIn), 4 times a day, one page each time.

• Searches on the five 1-credit platforms: 6 queries x 5 platforms x 4 runs = 120 credits a day.
• YouTube search at 2 credits: 6 x 4 x 2 = 48 credits a day.
• Comments on the 20 biggest new mentions: 20 credits a day.
• Transcripts for the 10 biggest new videos: 10 credits a day.

That is 198 credits a day, or 5,940 credits a month. At $1.88 per 1,000 on the $47 pack, the month costs $11.17, and one pack lasts a little over 4 months. On the $497 pack ($0.99 per 1,000) the same month is about $5.90. Failed calls and not-found answers cost nothing, and a repeat of the same search inside its cache window is free, so the real bill is usually a bit lower.

Code: collect and dedupe

import requests, sqlite3

KEY = "YOUR_KEY"
BASE = "https://scrapeio.com"
SEARCH = {
    "x": ("/v1/twitter/search", {"type": "latest"}),
    "tiktok": ("/v1/tiktok/search/keyword", {"sort_by": "latest"}),
    "instagram": ("/v1/instagram/search/posts", {}),
    "youtube": ("/v1/youtube/search", {"sort_by": "date"}),
}
QUERIES = ["acme coffee", "acmecoffee", "acme cold brew"]

db = sqlite3.connect("mentions.db")
db.execute("""create table if not exists mentions (
  platform text, id text, url text, text text, created_at text,
  likes int, comments int, query text, seen_at text,
  primary key (platform, id))""")

for q in QUERIES:
    for platform, (path, extra) in SEARCH.items():
        r = requests.get(BASE + path, params={"query": q, **extra},
                         headers={"x-api-key": KEY}, timeout=60).json()
        if not r.get("success"):
            continue
        for p in r["data"]["items"]:
            db.execute(
                "insert or ignore into mentions values (?,?,?,?,?,?,?,?,datetime('now'))",
                (platform, p["id"], p.get("url"), p.get("caption"),
                 p.get("created_at"), p.get("likes"), p.get("comments"), q))
db.commit()

Because every platform returns the same field names, one insert handles all four. Run it from cron every 6 hours.

The metrics that matter

• Share of voice = your mentions / (your mentions + competitor mentions) over the same queries, platforms and days.
• Engagement-weighted share of voice = the sum of likes + comments + shares on your mentions / the same sum for all brands. One viral video can outweigh a hundred quiet posts, and this version shows it.
• Net sentiment = (positive - negative) / (positive + negative). Leave neutral out, or it drags every score toward zero.
• Mention velocity = mentions today / average daily mentions over the last 7 days. A value above 2 is worth a look the same day.
• Top topics: count tags per week and watch which ones grow, not which ones are biggest.

Pitfalls

• Common-word brands. If your brand is also an ordinary word, add a required second term or the feed fills with noise.
• Search is a ranked sample. Platform search returns what the platform ranks, not every post. Compare trends over time inside one platform, and do not add raw counts across platforms as if they were a census.
• Reposts and quotes. One complaint quoted 40 times is one issue. Group by the original post before you count.
• Time zones. Store created_at in UTC and convert only for the report.
• Prompt injection. If an LLM tags your mentions, treat post text as untrusted input. A post can contain instructions aimed at the model.

For scoring mentions so the right person reads them, see brand monitoring tools. For Reddit-only alerts, see Reddit keyword monitoring.

Run it from an AI agent

Connect the hosted MCP server at https://scrapeio.com/mcp to Claude Code, Cursor or Codex (setup is on /agents). The search endpoints above appear as tools such as twitter_search, tiktok_search_keyword and reddit_search. A prompt that works:

Search X, TikTok and Reddit for "acme cold brew" from the last 7 days.
Group the mentions into complaints, praise and questions.
Give share of voice against "bolt cold brew" on the same queries,
and link the 5 posts with the most engagement.

The agent is good for a weekly read and for one-off questions. Keep the cron job for the daily numbers, so the history is complete and the cost is fixed.

Doing it with ScrapeIO

One key covers search on X, TikTok, Instagram, YouTube, Reddit, LinkedIn and Threads, with the same fields on every platform. Most calls cost 1 credit, credits never expire, and failed or cached calls are free. Start with the 100 free credits and run the script above on one query to see what your feed looks like.