SERP Scraping With Python: Real Failure Logs and What Actually Works

Every SERP scraper has the same biography. It works on day one. You build the parser, you feel clever. Two weeks later the keyword counts drift to zero and nobody notices, because the requests still return 200.

This article is the autopsy. Real commands, real responses, from a real datacenter IP — not a hypothetical “Google might block you” warning. Then the ladder of what actually works, with costs.

The Experiment: One curl Against Google

The setup is the smallest possible scraper — no Python even, just curl with a normal desktop user agent:

curl -s -o /tmp/google.html \
  -w "HTTP %{http_code}, %{size_download} bytes, ip=%{remote_ip}\n" \
  -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" \
  "https://www.google.com/search?q=serp+scraping+python&num=10"

Response:

HTTP 302, 420 bytes, ip=198.18.0.63

No CAPTCHA. No 429. No “unusual traffic” page. A 302 redirect — 420 bytes of “please go somewhere else.” Where? Let’s look:

curl -s -o /dev/null -w "%{redirect_url}\n" \
  -A "Mozilla/5.0 ..." \
  "https://www.google.com/search?q=serp+scraping+python&num=10"

https://www.google.com.hk/url?sa=p&hl=zh-CN&pref=hkredirect&pval=yes&q=https://www.google.com.hk/search%3Fq%3Dserp%2Bscraping%2Bpython...

The exit IP for this request resolves to Hong Kong, so Google redirects to google.com.hk with hl=zh-CN bolted on. A scraper that doesn’t follow redirects sees a 302 and gives up. A scraper that does follow them gets the real trap.

The Trap: 200 OK With Nothing Inside

Following the redirect:

curl -sL -o /tmp/google2.html -w "HTTP %{http_code}, %{size_download} bytes\n" \
  -A "Mozilla/5.0 ..." \
  "https://www.google.com/search?q=serp+scraping+python&num=10"

HTTP 200, 92366 bytes

Success, right? 92 KB of HTML. Now check what’s actually in it:

grep -o '<title>[^<]*</title>' /tmp/google2.html
# <title>Google Search</title>

grep -o -c 'href="/url?' /tmp/google2.html
# 0

grep -o -iE 'enablejs|jserror' /tmp/google2.html | sort | uniq -c
#    2 enablejs
#    1 jserror

Zero organic result links. The page title is the generic “Google Search,” and the body contains enablejs markers — this is the JavaScript-required shell. Google served a full-size, perfectly valid HTML page that contains no results and never will, because it wants a real browser.

This is the worst failure mode in scraping: silent success. Your requests.get() returns 200. Your .status_code == 200 check passes. Your parser finds zero h3 elements, your pipeline records “no results for keyword,” and your keyword counts quietly rot. Weeks later you’re making decisions on empty data.

The fix isn’t a better parser. It’s assertions:

resp = session.get(url, headers=HEADERS, timeout=15)
resp.raise_for_status()
results = parse_organic(resp.text)
if len(results) < 3:
    raise SerpShellError(
        f"got 200 but only {len(results)} results — "
        f"likely a JS shell or consent page, not a real SERP"
    )

Assert on result count, not status code. Every SERP fetcher we ship does this.

Meanwhile, at Bing

Same request, same IP, same user agent:

curl -s -o /tmp/bing.html \
  -w "HTTP %{http_code}, %{size_download} bytes\n" \
  -A "Mozilla/5.0 ..." \
  "https://www.bing.com/search?q=serp+scraping+python"

HTTP 200, 124893 bytes

Full page, real results, b_algo result blocks right there in the HTML. Bing is dramatically easier to scrape than Google from a datacenter IP.

So just scrape Bing? Here’s the catch we hit in our own keyword difficulty tooling: the rankings you scrape from a datacenter IP are not the rankings a real user sees. We validated Bing positions against what actual users in the target market got, and they didn’t match — same keyword, different orders, some domains missing entirely. Location and IP reputation are ranking inputs, and your datacenter exit is a very unusual “user.”

Easy to scrape and subtly wrong is worse than hard to scrape. At least the Google block is loud.

The Ladder That Actually Works

Three rungs, in ascending order of reliability:

Rung 1: Raw requests with manners. Pin your locale (gl, hl parameters), keep one session, sleep between calls, back off exponentially on any non-200. Works for tens of queries a day. Dies at scale or on bad IP-reputation days. Fine for experiments, wrong for pipelines.

Rung 2: Headless browser. Playwright with a real fingerprint solves the JS shell. It also multiplies your cost per query by 10-50× (browser startup, page weight), and you’ve now entered the fingerprint arms race — headless Chrome has tells, and Google looks for them. Use this when you need rendered page content, not when you need SERP structure.

Rung 3: SERP API on a free tier. Someone else runs the IP pool, the browsers, and the parsing. You get structured JSON:

curl -s -X POST "https://google.serper.dev/search" \
  -H "X-API-KEY: $SERPER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"q":"serp scraping python","num":10}'

{
  "searchParameters": {"q": "serp scraping python", "gl": "us", "hl": "en"},
  "organic": [
    {"title": "Web Scraping in Python (Complete Tutorial 2026)", "link": "https://serpapi.com/blog/python-web-scraping-tutorial/", "position": 1},
    {"title": "How to Scrape Google Search Results: Python Tutorial", "link": "https://github.com/oxylabs/scrape-google-python", "position": 2}
  ],
  "peopleAlsoAsk": [...],
  "relatedSearches": [...],
  "credits": 1
}

Real timings from three consecutive calls: 2.04s, 2.34s, 3.06s. Free tier: 2,500 calls a month. That’s a full keyword research pipeline for $0 — and since the response is already parsed, the “silent empty page” failure mode is structurally impossible. No organic array means the API is telling you something, not hiding it.

Our entire keyword research stack — difficulty scoring, SERP intent checks, competitor gap — runs on this rung.

If You Must Scrape: The Pattern

Sometimes the free tier isn’t enough and you’re back to raw scraping. The minimum viable discipline:

import time, random, requests

def fetch_with_backoff(url, session, max_retries=4):
    for attempt in range(max_retries):
        resp = session.get(url, timeout=15)
        if resp.status_code == 200 and parse_organic(resp.text):
            return resp
        if resp.status_code in (429, 503) or resp.status_code == 200:
            # 200-with-no-results is treated as a soft block
            sleep = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(sleep)
            continue
        resp.raise_for_status()
    raise RuntimeError("soft-blocked after retries")

# between keywords, not just between retries
time.sleep(random.uniform(4, 9))

Three rules that matter more than the code: treat 200-with-zero-results as a block (that’s the whole lesson above), rate-limit between keywords not just between retries, and cache every response so you never re-ask the same question — a local cache layer turns 2,500 free credits into several times that.

Where This Is Wrong

1. Your IP will behave differently. These logs are from one datacenter exit via one proxy route. Residential IPs get friendlier treatment from Google; other datacenter ranges get outright CAPTCHAs instead of the polite redirect. Your failure mode will differ — the lesson (assert on results, not status codes) transfers regardless.

2. SERP APIs are a dependency. Rung 3 means someone else’s uptime, someone else’s pricing, and a hard meter. When your pipeline outgrows 2,500/month, you’re paying. That’s still cheaper than running your own browser fleet, but it’s not free forever.

3. API SERPs are normalized SERPs. A SERP API returns a deterministic, location-pinned snapshot — which is exactly what you want for scoring, and not identical to what any specific real user sees. We measured how much markets differ even through the same API in the localization variance test.

Fitting It Into a Workflow

Scraping is the data-acquisition layer of a pipeline, not the pipeline:

1. Acquire — this article’s rungs: raw requests, browser, or API
2. Stretch the budget — cache every SERP response before you spend another credit
3. Score — feed SERPs into difficulty scoring and SERP structure analysis
4. Track — re-run weekly through the same pinned gl/hl, with rank tracking that knows its own limits

The whole stack installs in one line and runs on free tiers:

pip install git+https://github.com/ZensInk/zens-ink-seo-package.git
export SERPER_API_KEY="your_key"

The Point

The hard part of SERP scraping was never getting HTML — it’s knowing when the HTML you got is a lie. Google’s most effective anti-scraping weapon isn’t the CAPTCHA; it’s the 200 OK with nothing inside. Assert on what you parsed, not what the server said, and the rest of the ladder falls into place.

FAQ

Why does my Google scraper get HTTP 200 but no search results?

Because Google frequently serves a JavaScript-required shell or a regional redirect instead of a block page. In the test replayed here, google.com/search returned 302 to google.com.hk, and following it produced a 200 OK page of ~92 KB whose title was just 'Google Search' with zero organic result links inside. A scraper that only checks the status code records a success and parses nothing — the failure is silent unless you assert on result count.

Is it legal to scrape Google search results?

This article doesn't give legal advice, and the practical answer varies by jurisdiction and scale. What's safe to say: Google's terms disallow automated scraping, enforcement is mostly rate-based, and at meaningful volume you will hit consent walls, CAPTCHAs, or IP blocks long before anyone talks to you. SERP APIs exist precisely because they operate that gray zone as a service — you pay (or use free tiers) and stay out of the IP-reputation business.

Is Bing easier to scrape than Google?

From a datacenter IP, yes — far easier. The same request that got redirected and emptied by Google returned a full 124 KB Bing results page with organic listings in plain HTML. The catch is that Bing's rankings themselves vary by IP and location, so what you scrape from a datacenter is not what a real user in your target market sees. Easy to scrape and subtly wrong is a worse failure mode than hard to scrape.

What's the cheapest way to get structured Google SERP data?

A SERP API on a free tier. Serper.dev gives 2,500 searches/month for free and returns parsed JSON — organic results, people-also-ask, related searches — in about 2-3 seconds per call. For a keyword research pipeline of a few thousand keywords a month, that's $0. Add a local cache layer and the same budget stretches several times further.