API Caching: Why Free Cache Hits Cut Your Scraping Costs

November 8, 2026

A surprising share of data API calls ask for something the system already fetched a few minutes ago. A dashboard refreshes. Two teammates look up the same creator. An AI agent retries a tool call after a timeout. If every repeat costs full price, caching is the cheapest saving you will ever make.

Where repeats come from

• Dashboards and reports that reload on every page view.
• AI agents that call the same tool twice in one session, or restart a task.
• Several people or services asking for the same popular profile.
• Development loops: you run the same script 30 times while fixing a parser.
• Paging restarts after a crash, re-reading pages 1 to 9 to reach page 10.

The cost formula

monthly cost = calls × price × (1 - hit rate × hit discount)

With 100,000 calls a month at $1.88 per 1,000 and a 30% repeat rate, you pay $188 if repeats are charged, and $132 if they are free. At a 60% repeat rate, which is common for dashboards, the bill drops to $75. Measure your own hit rate before you assume: log a hash of each request for a week and count duplicates within your TTL.

Some vendors already build this in. X's pay-per-use pricing deduplicates resources within a 24-hour UTC window. Zyte bills per successful response and Bright Data bills only for successful results, which removes the cost of failures but not of repeats.

Measure your repeat rate in a week

1. Log every outgoing request as a hash of its path and sorted parameters, with a timestamp.
2. After a week, count requests whose hash was already seen within the last hour, the last day and the last week.
3. Split the counts by endpoint. Profiles and transcripts usually repeat most; search repeats least.
4. Multiply each repeat count by its price to see what caching is worth per endpoint.

Pick a TTL for each type of data

The last row is negative caching. If a handle does not exist, remember that briefly so a loop does not ask 500 times, but not for long, since accounts get created and renamed.

Build a good cache key

1. Use the endpoint path plus parameters, sorted by name.
2. Normalize values: lowercase handles, strip @, remove tracking parameters from URLs.
3. Leave out anything that does not change the answer, such as a request id.
4. Include the cursor for paginated calls, so each page caches on its own.

A small cache wrapper

const cache = new Map(); // swap for Redis or KV in production
function key(path, params) {
  const sorted = Object.keys(params).sort().map((k) => k + "=" + String(params[k]).toLowerCase());
  return path + "?" + sorted.join("&");
}
async function cachedGet(path, params, ttlSeconds) {
  const k = key(path, params);
  const hit = cache.get(k);
  if (hit && hit.expires > Date.now()) return hit.body;
  const res = await fetch("https://scrapeio.com" + k, { headers: { "x-api-key": process.env.SCRAPEIO_API_KEY } });
  const body = await res.json();
  if (res.ok) cache.set(k, { body, expires: Date.now() + ttlSeconds * 1000 });
  return body;
}

Only cache successful answers for the full TTL. Do not cache rate-limit or server errors at all. If you serve HTTP yourself, the standard Cache-Control rules in RFC 9111 let browsers and CDNs do part of this work for you.

When not to cache

• When you are measuring change over time, such as hourly play counts for a launch. Use fresh params or bypass the cache on purpose.
• When the user explicitly asks for live data.
• When terms or privacy rules limit how long you may keep the data.

Doing it with ScrapeIO

ScrapeIO caches on its side and does not charge for cached repeats. A response tells you which one you got: the envelope has cached and credits_used, so a repeat of the TikTok profile API or YouTube video API within the cache window shows cached: true and costs 0. Transcripts from the YouTube transcript API rarely change, so they are the easiest place to save. You can still add your own cache in front for speed.

Sources

Checked on 8 November 2026 from each vendor's own pages. Prices and limits change; follow the links before you decide.

• https://docs.x.com/x-api/getting-started/pricing
• https://www.zyte.com/pricing/
• https://brightdata.com/pricing/web-scraper
• https://www.rfc-editor.org/rfc/rfc9111