Hi everyone. I am a Transportation Engineering student (Civil and Environmental Engineering department) from Bangladesh, still learning, so please forgive me if I say something wrong.
I am working on a research project involving NLP and sentiment analysis on Bangladeshi newspaper articles. The project requires collecting news articles from around 10 newspapers — both English and Bangla language publications — covering the period from 2015 to 2026. I need the article headline, full text, publication date, and source name, ideally stored as CSV or JSON files, one per newspaper or per year.
I have already tried Python with BeautifulSoup and Selenium, GDELT API, Archive.org CDX API, and pre-scraped Kaggle datasets. For two newspapers I managed to collect solid datasets. For the rest I keep running into bot detection, CAPTCHA blocks, anti-scraping measures, or very uneven temporal coverage where certain years return zero articles at all.
I cannot offer payment, but I can offer co-authorship on the research paper I am working toward publishing. If you have experience collecting large-scale news archives from South Asian or Bangladeshi news websites and are interested in collaborating, I would love to hear from you. Any general advice is also very welcome even without a collaboration.
Thank you for your time.