Reddit Comment Scrapers: How to Get Full Comment Threads

November 8, 2026

Reddit posts get the clicks, but the comments hold the detail: product complaints, workarounds, comparisons and the replies that correct them. Collecting comments is harder than collecting posts. This guide explains why, compares the tools that handle it, and gives you code to turn any thread into clean rows.

Why comments are harder than posts

• They are a tree. Each comment has a parent, which is either the post or another comment. Analysis usually needs that structure kept.
• Large threads are cut. Reddit returns part of the tree and leaves "more comments" stubs for the rest. A naive scraper stops there and misses most replies on busy threads.
• Sort order changes what you see. Top, new and controversial give different first pages. Pick one and record it.
• Scores move. A comment's score on day one is not its score on day seven. Store the time you collected it.
• Content disappears. Deleted and removed comments leave placeholders. Keep the row, mark it, and do not try to recover the text.

Tools that handle comment trees

The PRAW way

The PRAW comment tutorial shows the core pattern. By default replace_more expands up to 32 stubs; limit=None keeps going until none are left. Each replacement is one network request, and its response can contain more stubs, so the count grows on big threads.

submission = reddit.submission(url="https://www.reddit.com/r/.../comments/abc123/...")
submission.comment_sort = "top"
submission.comments.replace_more(limit=None)
for c in submission.comments.list():
    print(c.id, c.parent_id, c.score, c.body[:80])

If you only need the conversation's main points, cap it with limit=10 or set a threshold. You trade completeness for speed, and for most sentiment work the top of the tree is enough.

Flatten any thread into rows

Whatever tool you use, you will end up with nested JSON. This function walks a tree without recursion, so deep threads cannot overflow the stack. It assumes each comment has an id and a replies list; rename the keys to match your source.

def flatten(comments, post_id):
    rows, stack = [], [(c, post_id, 0) for c in reversed(comments)]
    while stack:
        c, parent, depth = stack.pop()
        rows.append({
            "id": c["id"],
            "parent_id": parent,
            "depth": depth,
            "author": c.get("author"),
            "text": c.get("body") or c.get("text"),
            "likes": c.get("likes") or c.get("score"),
            "created_at": c.get("created_at"),
        })
        for child in reversed(c.get("replies") or []):
            stack.append((child, c["id"], depth + 1))
    return rows

Rows keep their parent, so you can rebuild the tree later, count replies per comment, or find the most argued-over branch.

A data checklist for comment projects

1. Store id, parent_id, depth, score, timestamp and the time you collected it.
2. Deduplicate by comment ID. Pagination and re-runs overlap.
3. Record the sort order and whether the tree was fully expanded.
4. Mark deleted or removed comments instead of dropping them, so reply counts still add up.
5. Keep usernames out of your analysis tables unless you need them. Aggregate where you can.
6. For sentiment or topic work, weight by score, and look at depth 0 and depth 1 separately. Top-level comments react to the post; replies react to each other.

Estimating cost

Comment pulls scale with thread size, not post count. A simple estimate: calls per post = 1 for the post + ceil(comments you want ÷ comments per page). For 500 posts where you want the first 200 comments each, at a page size near 100 that is about 500 × 3 = 1,500 calls. Run one real thread first and check its page size before you budget.

Doing it with ScrapeIO

Send a post URL to the Reddit post comments API and follow data.cursor while data.has_more is true. Use the Reddit post API for the score, title and comment count first, so you know how deep to go. Video posts also have a transcript endpoint.

import requests
KEY = "YOUR_KEY"
params = {"url": "https://www.reddit.com/r/.../comments/abc123/..."}
items = []
while True:
    r = requests.get("https://scrapeio.com/v1/reddit/post/comments",
                     params=params, headers={"x-api-key": KEY}).json()
    items += r["data"]["items"]
    if not r["data"].get("has_more"):
        break
    params["cursor"] = r["data"]["cursor"]

Sources

Checked on 8 November 2026 from each vendor's own pages. Prices and limits change; follow the links before you decide.

• https://praw.readthedocs.io/en/stable/tutorials/comments.html
• https://brightdata.com/products/web-scraper/reddit
• https://github.com/ArthurHeitmann/arctic_shift