Common Web Scraping Mistakes That Quietly Ruin Your Data

November 8, 2026

Most scraping failures are not crashes. The job runs, the table fills, and the numbers are wrong. Here are the mistakes we see most often in social media and web data projects, and how to catch each one before it reaches a report.

1. Storing numbers as text

"1.2M", "1,204" and "1204" are the same number in three forms. If you store whatever the page shows, sorting and sums break. Parse to integers at ingest, keep the raw string in a second column, and use null when a platform does not report a value. Never use 0 for unknown: a video with unknown plays is not a video with zero plays.

2. Reading big ids as numbers

Some platform ids are larger than 2^53, the largest integer JavaScript and many JSON parsers hold exactly. Instagram media ids are one example. Parse them as numbers and the last digits silently change, so two posts can collide. Always store ids as strings.

3. Paging the wrong way

• Offset paging on a live feed. New posts push items down between requests, so you get duplicates and gaps. Use cursors when the source offers them.
• No stop condition. Stop when has_more is false, when the cursor repeats, when a page is empty, or when you reach a date or count limit. Without these, a loop can run forever.
• Restarting from page one after a crash. Save the last cursor after each page so you can resume.

4. Ignoring rate limits

Rate limits are part of the contract with whoever serves the data. Respect them. When you get a 429, wait: use the Retry-After header if present, otherwise back off exponentially with jitter. Set your own concurrency below the published limit, and spread scheduled jobs so they do not all start at the top of the hour.

5. Schema drift

Sources change fields without telling you. A field gets renamed, a number becomes a string, a list becomes an object. Your parser keeps running and writes nulls.

• Validate every record against a schema at ingest.
• Track the null rate per field per day. A jump from 2% to 90% is a broken parser, not a trend.
• Keep raw responses for at least a week so you can re-parse after a fix.

6. Mixing time zones and fetch times

Store created_at in UTC, ISO 8601. Store when you fetched the record separately, because counts change. A post's like count is only meaningful next to the time you read it.

7. Treating search as a census

Platform search returns a ranked sample, not every matching post. "Top" results favor popular posts, and some platforms limit how far back search goes. If you report "mentions of X", say how you collected them and over what period. For true counts over time, poll with "latest" sorting on a fixed schedule.

8. Not deduplicating

The same post shows up in two queries, on two pages, or in two runs. Use platform plus id as the primary key, and upsert so counts update instead of creating new rows.

9. Collecting more than you need

Every extra field is storage, parsing work, and privacy exposure. Decide your fields before you start. If you do not need commenter names for your analysis, do not store them.

A pagination loop that avoids most of this

async function fetchAll(path, params, maxPages = 20) {
  const items = [];
  let cursor = null;
  for (let page = 0; page < maxPages; page++) {
    const qs = new URLSearchParams({ ...params, ...(cursor ? { cursor } : {}) });
    const res = await fetch("https://scrapeio.com" + path + "?" + qs, { headers: { "x-api-key": process.env.SCRAPEIO_API_KEY } });
    if (res.status === 429) { await new Promise((r) => setTimeout(r, 2000 * 2 ** page)); page--; continue; }
    if (!res.ok) break;
    const { data } = await res.json();
    items.push(...data.items);
    if (!data.has_more || !data.cursor || data.cursor === cursor) break;
    cursor = data.cursor; // save this to resume after a crash
  }
  return items;
}

Cap retries in production so a persistent 429 does not loop forever.

Pre-flight checklist

1. Ids stored as strings.
2. Numbers parsed, unknowns are null.
3. Timestamps in UTC, fetch time stored separately.
4. Cursor paging with four stop conditions.
5. Backoff on 429 and a concurrency cap.
6. Schema validation and a daily null-rate check.
7. Deduplication on platform plus id.
8. A written note on how the sample was collected.

Doing it with ScrapeIO

ScrapeIO handles several of these for you. Ids come back as strings, counts are numbers or null, and every list endpoint, from the TikTok comments API to the YouTube channel videos API, returns items, cursor and has_more in the same shape. The limit is 300 requests a minute per key, and 429 answers are not charged. See one schema for social media data for the field list.