Hi everyone,
I'm honestly feeling overwhelmed and could really use some guidance.
I'm a fresher who recently joined a startup as a Data Analyst intern. I expected to work on SQL, Python, dashboards, and analysis, but my very first task is to design and build a complete production data pipeline.
The requirement is to combine webhooks for real-time data capture with API-based data pulling for historical data.
The problem is that the company has 10 years of historical data, and the third-party API they use has strict rate limits. Because of that, a full data pull takes around 3–4 days. New data is also being generated every day, so the pipeline has to handle both historical backfills and ongoing real-time updates.
The biggest issue is that I've never built anything like this before.
I don't come from a data engineering background. I've spent the last few days reading blogs, watching videos, and using AI tools like Claude and ChatGPT to understand how these systems work and to draft an architecture. But I don't want to build something just because AI suggested it—I want to understand why it's designed that way.
Right now I feel like I'm trying to solve a senior-level problem with beginner-level knowledge.
I'm willing to put in the work. I just don't know what the right learning path is.
If you were in my position:
Where would you start?
What concepts should I learn first?
Are there any books, courses, YouTube playlists, or GitHub projects that helped you understand production data pipelines?
How would an experienced data engineer approach a problem like this?
What mistakes should I avoid?
I know this is a broad question, but even if you could point me toward the right resources or learning roadmap, it would mean a lot.
Thank you for taking the time to read this.