Stretch 2,500 Free SERP Credits Into 10,000 Lookups: A Python Cache Layer

Free tiers are bait in the best sense. Serper.dev gives you 2,500 searches a month, which sounds enormous — until your pipeline grows a second tool, and the second tool asks the API the exact questions the first tool asked yesterday.

This article is about the gap between what you ask and what you need to ask. It ends with a ~40-line, zero-dependency cache layer, and the real numbers from running it.

Where Duplicate Calls Hide

Nobody sets out to waste credits. The duplication is structural:

• Difficulty scoring pulls the SERP for serp scraping python.
• Intent classification on the same keyword list pulls the same SERP again — it needs the same top-10 the difficulty scorer just looked at.
• Competitor gap runs over your tracked keywords — many of which scored last week.
• Weekly rank tracking re-queries everything, because tracking must be fresh.

One keyword, four consumers, four identical requests on different days. Multiply by a 500-keyword list and a month has 2,000 API calls of which maybe 700 carried information you didn’t already have.

The naive fix is deduplicating keyword lists by hand. The correct fix is making the request layer the only thing that talks to the API — and giving it a memory.

The Cache Key Is the Whole Game

Before any code: what makes two SERP requests equivalent? Not the query string. We measured this directly — same keyword, same API, only the market parameters changed, and the top-10 shuffled:

keyword: 'python seo tools'  — domain positions by market (gl/hl)
domain                              us    gb    de    in
adver.tools/python/seo               5     6     4     8
searchengineland.com/python-...      -     9     -     4
searchwilderness.com/free-seo-tools  8     7     7     9

A US response is not a German response. Full methodology in the localization variance test — the takeaway here: a cache key without gl/hl is a data corruption bug that looks like a performance optimization.

The canonical key:

import hashlib, json

def cache_key(endpoint, q, gl, hl, num):
    raw = json.dumps(
        {"e": endpoint, "q": q, "gl": gl, "hl": hl, "n": num},
        sort_keys=True, ensure_ascii=False,
    )
    return hashlib.sha1(raw.encode()).hexdigest()

Sorted keys, explicit fields, deterministic serialization. The hash is the key; the JSON is your debug log when something looks cached-wrong.

The Layer, Whole

Stdlib only — this drops into the zens_ink toolchain or any script:

import hashlib, json, os, sqlite3, time, urllib.request

DB = sqlite3.connect("serp_cache.db")
DB.execute("""CREATE TABLE IF NOT EXISTS serp_cache (
    k TEXT PRIMARY KEY, endpoint TEXT, q TEXT, gl TEXT, hl TEXT,
    payload TEXT, fetched_at REAL)""")

def serp(q, gl="us", hl="en", num=10, ttl=86400):
    endpoint = "search"
    key = cache_key(endpoint, q, gl, hl, num)
    row = DB.execute(
        "SELECT payload, fetched_at FROM serp_cache WHERE k=?", (key,)
    ).fetchone()

    if row and time.time() - row[1] < ttl:
        return json.loads(row[0])          # cache hit: no credits, no network

    body = json.dumps({"q": q, "gl": gl, "hl": hl, "num": num}).encode()
    req = urllib.request.Request(
        f"https://google.serper.dev/{endpoint}",
        data=body,
        headers={"X-API-KEY": os.environ["SERPER_API_KEY"],
                 "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=20) as r:
        payload = json.load(r)

    DB.execute(
        "INSERT OR REPLACE INTO serp_cache VALUES (?,?,?,?,?,?,?)",
        (key, endpoint, q, gl, hl, json.dumps(payload), time.time()),
    )
    DB.commit()
    return payload

Every tool — kd scoring, intent classification, competitor gap — calls serp(), never the API. That single indirection is the entire architecture.

The Real Numbers

Three consecutive live calls, timed (this is the same session as the scraping failure logs):

call 1: 200, 2.043979s
call 2: 200, 2.342513s
call 3: 200, 3.060339s

And the sqlite point lookup, averaged over 1,000 hits:

sqlite cache hit avg: 0.005 ms

That’s roughly 2,500 ms vs 0.005 ms — about 500,000× — and more importantly, credits: 1 on one path, zero on the other. Speed is a nice side effect; the credit budget is the point.

TTLs Are Use-Case Decisions, Not Constants

How stale is too stale depends on why you’re asking:

Use case	TTL	Why
Keyword discovery / difficulty scoring	7 days	SERP structure changes slowly; scores tolerate it
Intent classification	7 days	Page types in the top-10 are stable
Content audits on live pages	24 hours	Fresh enough for editorial decisions
Rank tracking	no cache	You are measuring change; caching cancels the measurement

That last row is the discipline that keeps the cache honest. Rank tracking must hit the API fresh every scheduled run — if the budget gets tight, cut tracked keywords, never cache ranks. Everything upstream of tracking is fair game.

Single-Flight: The Last Duplicate Killer

One race remains: two tools in the same process ask for the same cold keyword in the same second. Both miss the cache, both hit the API. At pipeline scale, a simple in-flight dict closes it:

_inflight = {}

def serp_singleflight(*args, **kwargs):
    key = cache_key(kwargs.get("endpoint","search"), kwargs.get("q",""),
                    kwargs.get("gl","us"), kwargs.get("hl","en"), kwargs.get("num",10))
    if key in _inflight:
        return _inflight[key].result()   # join the running call
    fut = THREAD_POOL.submit(serp, *args, **kwargs)
    _inflight[key] = fut
    try:
        return fut.result()
    finally:
        _inflight.pop(key, None)

Same idea as request coalescing in CDNs: one actual fetch, N callers.

Where This Is Wrong

1. Cache hit rate is a guess until measured. The 3-5x figure is what our pipelines see; yours depends on how much your keyword lists overlap between stages. Instrument it — store fetched_at and count hits vs misses for a week before trusting any multiplier.

2. Sqlite is single-writer. For one process on one machine it’s perfect. The day you fan out to parallel workers, move to per-worker DBs merged nightly, or accept write contention. Don’t reach for Redis — that’s a server for a problem you don’t have yet.

3. TTLs are opinions. A 7-day TTL on difficulty SERPs means you can score a keyword against a SERP that changed mid-week. For scoring that’s noise; for a launch decision it might matter. Drop the TTL for anything feeding a one-time irreversible decision.

4. The API response is already the compressed truth. You’re caching the API’s normalized SERP, not the live SERP — which is the right thing to cache, but remember what localization testing showed: even fresh API responses are market-pinned samples, not universal truth.

The Math That Pays For It

Concrete scenario, because multipliers without arithmetic are marketing:

• 500 tracked keywords
• Weekly rank tracking, 4 weeks: 2,000 calls — uncached, always (that’s the rule)
• Monthly difficulty re-score of the same 500: 500 calls → 0 with 7d TTL (weekly overlap)
• Intent pass over 1,500 candidate keywords, 40% previously seen: 1,500 → ~900
• Competitor gap over 300 keywords, 70% overlap with tracked set: 300 → ~90

Naive: 4,300 credits (you’re out of the free tier). With the layer: 2,090 — still inside 2,500, at $0. That’s the difference between “free tier covers me” and “free tier covers me until Tuesday.”

Fitting It Into a Workflow

1. Acquire SERPs through one cached entry point — this article’s serp()
2. Score what you fetched: difficulty scoring, SERP structure analysis
3. Discover new keywords to feed the machine: Autocomplete mining (free, unmetered — the API budget belongs to SERPs)
4. Track weekly, fresh, never cached

pip install git+https://github.com/ZensInk/zens-ink-seo-package.git
export SERPER_API_KEY="your_key"

The Point

A free tier isn’t a budget, it’s a rate of information. Most pipelines spend it re-learning what they already knew yesterday. One indirection — every tool asks the cache, the cache alone asks the API — and the same tier quietly covers several times the work.

FAQ

How do I make a free SERP API tier last longer?

Cache responses locally keyed by the full request — query, gl, hl, num, and endpoint — in sqlite, and route every tool through the same wrapper. In practice most pipelines see 3-5x overlap between keyword lists used by difficulty scoring, intent checks, and rank tracking. A 2,500/month free tier commonly covers effective demand of 8,000-12,000 lookups with a 24h TTL.

What should the cache key for a SERP response include?

At minimum: endpoint, query, gl, hl, and num. Omitting gl/hl is the classic bug — the same keyword returns different rankings per market (we measured position swings across four markets through the same API), so an English-US response cached under a bare query will poison a German-market lookup. Hash the canonical key string with sha1 and store the market params alongside for debugging.

Should I cache rank-tracking SERP responses?

No — not on the tracking path. Rank tracking exists to measure change, so caching the measurement defeats the purpose. Cache everything upstream (keyword research, difficulty scoring, intent classification) but let the weekly rank check hit the API fresh, pinned to the same gl/hl every run. If budget is tight, reduce keyword count rather than caching ranks.

Is sqlite fast enough for a SERP cache?

Yes, by about five orders of magnitude. A measured sqlite point lookup on a primary key averaged 0.005 ms per hit, against 2,000-3,000 ms for a real Serper.dev round trip. Even accounting for JSON deserialization, the cache path is ~500,000x faster than the network path — and it costs zero API credits.