So i am scraping a site, its got like 900k pages, i boiled down to like just over 300k. Its taking way too long to scrape. I am doing asyncio.semaphore too it still takes way too long with inbuilt retry feature. Can anyone suggest me what to do?

Hello all, I am attempting to create a scraper that will collect data for LNG freights, pipelines, and storage facilities. A lot of this stuff is blocked off by paywalls amongst other things. I am a college student and am do not have access to exclusive commodity sites, much less have the funds. I have never built a scraper really don't know where to begin. Any advice would be greatly appreciated.

TLDR:A federal court dismissed Google's DMCA claims against SerpApi, ruling that blocking scrapers from public search results isn't copyright circumvention when no copyrighted content is involved. Those claims are gone for good, but the part about licensed Knowledge Panel images was dismissed with leave to amend — Google has 21 days to try again by proving copyright owners authorized SearchGuard. A win for SERP scraping tools generally.

What is the best way to limit search on selected Reddit Communities at once?