I have been working on a project where we need to extract schema org structured data from the full CC-MAIN-2026-25 Common Crawl corpus (100,000 WARC files, ~88TB compressed, June 2026 crawl).

Been using extruct (Python) which parses JSON-LD, Microdata, and RDFa from raw HTML. I went with extruct over regex because regex only picks up ~35 schema types while extruct gets 3,000+. The files are streamed directly from S3 to EC2 so no storage costs.

The script processes each WARC file record by record, applies a byte-level pre-filter to skip pages with no schema markup, then runs extruct on pages that pass. Been using a shared multiprocessing pool with a 30 second per-page timeout to handle pages where extruct hangs. Checkpoints every 50 files so it can resume if interrupted.

Results so far on a c6i.xlarge with 4 workers:

• ~1,320 seconds per file
• ~20,800 pages per file
• ~45.8% schema coverage
• 0 file errors, 1 page timeout across ~83,000 pages

At that speed the full corpus would take ~382 days on that instance. I am not quite sure how this will scale and what is the realistic time and cost for the full corpus.

Questions:

1. Has anyone run extruct at this scale? Is 1,320s/file reasonable or are we leaving performance on the table somewhere?
2. Is there a smarter way to parallelize this, spot fleet, Lambda, ECS? The job is resumable so spot interruptions are manageable.
3. Any experience with extruct slowness on specific page types? We're seeing 0 timeouts almost everywhere but the overall speed still feels slow.
4. Would lxml or another parser swap inside extruct make a meaningful difference at this scale?

Happy to share more details on the setup if useful.

What is the best way to limit search on selected Reddit Communities at once?

Hello all, I am attempting to create a scraper that will collect data for LNG freights, pipelines, and storage facilities. A lot of this stuff is blocked off by paywalls amongst other things. I am a college student and am do not have access to exclusive commodity sites, much less have the funds. I have never built a scraper really don't know where to begin. Any advice would be greatly appreciated.

TLDR:A federal court dismissed Google's DMCA claims against SerpApi, ruling that blocking scrapers from public search results isn't copyright circumvention when no copyrighted content is involved. Those claims are gone for good, but the part about licensed Knowledge Panel images was dismissed with leave to amend — Google has 21 days to try again by proving copyright owners authorized SearchGuard. A win for SERP scraping tools generally.

So i am scraping a site, its got like 900k pages, i boiled down to like just over 300k. Its taking way too long to scrape. I am doing asyncio.semaphore too it still takes way too long with inbuilt retry feature. Can anyone suggest me what to do?

Manager came to me with a list of 100 competitor websites, and asked me to scrape their data daily with a scheduler. I hit several blockers and think this professional community could offer some advices lol.

I am currently writing the scraping app that will eventually be a microservice. The marketing team could get on my service, add a website and it will be automatically scraped daily. Questions I have so far:

1) Is this a viable/launchable idea? It seems like every website has its own structure. Some are simple HTML pages, while others have protections like Cloudflare or other anti-bot systems. When a user adds a website, is it possible to automatically identify what type of website it is and determine the best scraping approach?

2) Some websites require users to log in before accessing the promotional pages that I want to scrape. In this case, is manual login the only solution? For example, would users need to manually log in to the website every morning before my scraping service can run?

3) Is there a way to automate opening a browser and logging in from the backend for scraping purposes? If a website requires SMS OTP verification during login, how can this process be automated?

4) What are some practice you would do to avoid getting banned from websites for automated data scraping?

I’m investigating something interesting and was wondering if anyone has seen this before.
A friend of mine can consistently find the URLs of newly created TeePublic stores while they’re still in the Pending review stage. Some of these stores have no designs yet, and some are later rejected, but he can still find their store URLs shortly after they’re’re created.
He doesn’t have access to the accounts. My current theory is that he’s monitoring some public source (an index, feed, search endpoint, or other metadata) rather than the stores directly.
Has anyone researched TeePublic’s public-facing infrastructure or seen anything that could explain this? I’m trying to understand the mechanism, not exploit anything.

Guys,

. would be interesting to hear what's everyone's tool stack looks like that you use which you are very happy with

Also what are the top challenges?

What happened to eBay’s frontend recently? Somehow, scraping via proxies has stopped working whereas my home residential IP is okay. That aside, it also became necessary that the scraping was headful. Is it a matter of blacklisted ip addresses and I am not able to work around this?

Hey everyone,
I’m doing a personal deep dive into the travel experiences space and trying to understand how professionals analyse a destination like Rome or Paris.
What tools or methods would you use to quickly compare:

OTA listings, pricing and ratings
Review sentiment and recurring complaints
Top local operators and their contact details
Potential supply gaps

Since booking and conversion data are private, which public signals are actually useful? Review growth, rankings, availability, sold-out dates, number of listings?
I’d also be curious to hear what your step-by-step workflow looks like and which tools you use for scraping, analysis and dashboards.

Thanks!

Hey r/webscraping,

I’ve been building Vintrack, an open-source monitoring platform for Vinted. It continuously checks listings based on configurable filters and shows newly discovered items in a live dashboard. Alerts can also be sent through Discord or Telegram.

The project is completely free to self-host. To make testing easier, Vintrack includes shared free proxy pools for supported Vinted regions. The system imports public proxy candidates, validates them against each region, tracks their health, and only rotates proxies that are currently working into the pool.

This means you can create your first monitor without immediately purchasing proxies. Public proxies are obviously best-effort and can be slow or disappear, but they’re useful for testing the project before moving to private proxies.

Some technical details:

  - Go-based scraping workers

  - Next.js control center

  - PostgreSQL for persistent data

  - Redis for deduplication and live state

  - Per-region proxy health checks and rotation

  - TLS fingerprint rotation using tls-client

  - Configurable polling intervals

  - Filters for keywords, categories, brands, sizes, colors, prices, condition, seller country, and Vinted region

  - Server-Sent Events for the live dashboard feed

  - Discord webhooks and Telegram notifications

  - Docker Compose setup for self-hosting

  - Optional browser extension for linking a Vinted account and keeping its session synchronized

Vintrack also supports bringing your own proxy groups when you need more reliable or dedicated capacity. The shared free pools are intended as starter infrastructure, not as a replacement for good private proxies.

Repository:

https://github.com/JakobAIOdev/Vintrack-Vinted-Monitor

Live demo:

https://vintrack.jakobaio.dev

This is self-promotion—I’m the developer—but the project is open source, and I thought the proxy health and monitoring architecture might be interesting to this community.

Feedback and contributions are very welcome.

Hey everyone,

Using cloudscraper with python, its super solid for HTTP, but from my knowledge doesn't do anything for WS. WS connections keep getting flagged as bot, which is really annoying since im connecting to a public facing WS api endpoint. Any advice appreciated, thanks!

I run a bunch of scrapers for a few clients, and I already got that sites change their layouts without warning and a parser that worked yesterday quietly returns empty fields, or half the data. The script runs fine and exits cleanly so nothing seems to be wrong until someone looks at the output.

Now I have some basic checks, like if a scrape returns zero rows or way less than normal, it alerts me. But that only catches the obvious cases. I don't want to be caught out like that, where a section of a site changes and I don't notice it until the client rings me about it.

So what do you do with this when you have more than a handful of scrapers running? How often do you check data by hand if you do that at all?

Tengo una consulta, ahora mismo tengo una web que realiza Web Scraping a Tioanime y AnimeFLV, pero por que no tienen cloudfire o bloqueo de IP por bot, hay formas y maneras de evitar o ingresar? el proyecto es en github, desplegado en vercel y conectado por Variables

Welcome to the weekly discussion thread!

This is a space for web scrapers of all skill levels—whether you're a seasoned expert or just starting out. Here, you can discuss all things scraping, including:

• Hiring and job opportunities
• Industry news, trends, and insights
• Frequently asked questions, like "How do I scrape LinkedIn?"
• Marketing and monetization tips

If you're new to web scraping, make sure to check out the Beginners Guide 🌱

Commercial products may be mentioned in replies. If you want to promote your own products and services, continue to use the monthly thread

I've been quite succesfull scraping hard to crack sites individually (bot detection, proxy rotation, internal api etc.)

I'm now running a project to scrape a few 100 sites daily.

Except for a few which are very popular in the niche and have bot detection, I can access, say, 95% of the sites on a daily basis without issues HOWEVER the output I am getting is garbage:

My workflow is:

- get HTML, convert to markdown, have AI parse it for the information I need, save specific internal links, access those, parse, save information. I'm just not getting the quality of the output - halucinating URLs in the output, skipping sections etc.

Is there a better way to do this scale of a daily scrape? Again, my issue isnt actually accessing the sites, its 'processing' them.

Would I need to build a specific parser per site?

I have a small extension that compares prices across a few selected webshops.

To avoid scraping I built it, so that ot uses the webshops own search function in real time to find products. One of these shops has now deprecated the gtin search on-site.

I hoped to find their xml for google shopping somewhere but i haven't been able to.

Back to scraping.

The site i currently want to scrape is bike discount dot de.

My current plan to do it as gentle as possible is to setup a dedicated scrape computer at home and have it go thru every product page and log the price + gtin. Slow, but thats OK for my use case.

What would you do?

Hi ive been scraping for months now with a script. Using sb.solve_captcha() and the standard method of activating cdp mode before.

It worked for months. But for 3 days now, I see it ticking the captcha box, its loading, then its kinda rejected, unticked again.

Anyone else? Whats going on?

I wanted to work on my own Music Player for KMP Desktop... Until I wanted to teach UI Material 3 to AI (Gemini). So I went to the website to write a .md files but the amout of pages was wholesome!

> This is not a commercial project and is completely opensource!

So i made this Utility that will Fetch the images and .html from the website and put it all in .md file!

Features

• Concurrent Crawling: Downloads many pages at the same time.
• Smart Extraction: Removes headers, footers, sidebars, and menus.
• Markdown Conversion: Converts HTML into standard Markdown formats.
• Image Downloading: Saves images locally and updates links.
• Auto Language Detection: Detects code languages like Python, Java, and Bash.
• Cache and Resume: Saves progress using SQLite so it can resume after stopping.
• Fast Parsing: Uses a C++ module in the background for fast XML parsing and URL checking.

Architecture

The project is split into three main parts.

• C++ Extension: Handles fast sitemap parsing and checking for duplicate URLs.
• Python Crawler: Manages network requests, retries, and caching.
• Python Extractor: Analyzes the HTML, cleans it, and converts it to Markdown.

Anyone crack the Everest?

Our business is finally starting to take off, which is great, but it is bringing new challenges. In the last week we've gotten several cease and desist letters from website owners.

All of the data we scrape is 100% publicly available, so our team is thinking we could technically tell them to pound sand, but we don't want to be assholes. So far we have just complied and removed them from our platform.

We do use business names on our site and we attempt to show "live" data, which ends up being a bit off due to the time delay from our scraping schedule and realtime events.

We still have plenty of other sites we scrape, so the current volume is not too concerning yet, but we're fearful of where this could go if there is a domino affect of websites communicating with each other.

I know it's a bit of a gray area, but our business operates entirely off of the data we scrape (we make it easier for consumers to digest and navigate the data) and I know there are plenty of other business that have a very similar model. We're currently collect about 2-3M product data points per day from a few thousand sites. I think we total about 1.1M requests per day distributed to those sites.

Ideally I would like our business to not be so fickle and have the risk of crumbling if we get blocked or asked to stop.

We are currently seeking legal counsel, but I'm curious - how would you guys handle this (aside from just shitting your pants like I have)?

Is there any way to scrape reddit communities with ai like groq or something free model. I m trying to find problems worth solving for from reddit if anyone has built this scraper or knows how to lmk

Hello,

Begginer here, with only very small projects so far mostly through vibe coding.

For work (indepent compliant auditor) I was asked by my client to retrieve this whole public table with reported support (courses, international congresses, etc) by Pharma to doctors in Portugal.

It will take me a whole day to manually copy this page by page. Is there a way to do it immediately through web scraping?

LINK: https://extranet.infarmed.pt/pmro/Publico/ListagemPublica.aspx

I’m aware there is not a public api.