Currently I use Excel files and manually capture data in these Excel files. I have other peers who also do a similar procedure and we all have different formats. Is a streamlit app a good solution to standardize the excel format ? Is it even feasible as something that can be deployed to multiple users through tablets so that they can register the data via a streamlit app ?

Trying to do cross-ISO work (PJM/MISO/ERCOT/CAISO/SPP/NYISO/ISO-NE) and I'm losing my mind reconciling seven different schemas and update cadences - the congestion component especially (NYISO's sign convention alone…). Right now it's a pile of per-ISO scrapers held together with tape. Is everyone just using gridstatus / rolling their own, or is there something that already normalizes all of this? Curious what SPP/MISO historical depth people actually get.

I may not be able to make you understand the problem in a better way, but below is the issue.

Has anybody built a trigger/job, say a destination in Airflow, that would run on the 1st of every month, and this job should depend on another job, say a source, which is running every day, and the destination job should only run after that source one is finished on the last day of every month. Let's say if source one failed on the 31st and the team got it correct, then ran it on the 1st, then also the destination should run (edge case).

The source itself has a stage, a silver, and an archive layer (to store the data as backup).

The destination job (it has only one sub-job, which was used earlier, so no change in this) should be triggered after the above activity.

Both the source and destination are based on a YAML-based job.

Can we connect these two jobs/yaml to make this work? Or how can we work around it?

Probably one for the Duck DB nerds. I came across a ETL/ELT open source (for now) tool called Duckle which look pretty solid if you just need a light weight ETL/ELT tool for smaller workloads on a laptop or local box. Anyone used it?

What cheaper ETL tools are you using in 2026? I don't want AI in them and don't want to be forced to use AI. However, I have basic requirements:

1. Able to connect to SharePoint for files, lists, etc.

2. Able to connect to AWS S3.

3. Able to connect to Snowflake databases.

The developers at the biotech company don't want to heavily use SQL for ETL but want to use a drag-and-drop UI to save time and focus more on analysis. We explored Alteryx, but it was expensive. dbt doesn't have an intuitive UI.

They also don't want to use AI since the biotech company is very new and wants to save money as much as possible.

EDIT : processing 100k rows as if now every day. But it will grow to more than 500k or 1 million plus rows after few years. Need a scalable solution with headless or client based etl tool with option to schedule the pipelines in the background.

Curious to learn what checks or validation techniques experienced data engineers use in production.

I work at Estuary, and next week we’re doing a live technical demo of our new Agent Skills.

We’ll show an AI coding agent:

• Creating a real-time capture and materialization
• Working from the same pipeline specifications an engineer would edit
• Checking task health and pipeline status
• Inspecting logs and troubleshooting failures
• Handling routine configuration without hiding what it changed

The interesting question for us is where agents genuinely reduce data engineering work today, versus where an engineer still needs to step in.

We’ll build the pipeline live and leave time for technical questions.

Registration: https://zoom.us/webinar/register/4517840417866/WN_1oZszaPUQ_Sz0uEGM_p-zA

Task: We operate a toll road network with 200 sensor-equipped lanes across 15 locations. Each sensor captures license plate reads, timestamps, and lane metadata. We need to compute real-time traffic volume, average transit times between checkpoints, and flag anomalies for toll evasion detection. Design the data model.

I modeled it as a single accumulating snapshot fact table, grain = one row per vehicle crossing (entry -> exit). The row is inserted at entry with status = OPEN and all exit columns NULL, then updated in place when the exit read comes in – no join needed for evasion detection, just WHERE status = 'OPEN' AND entry_time < now() - window.

I have a confusion: does update-in-place on an accumulating snapshot actually hold up under real-time write load (concurrent OPEN -> COMPLETED updates), or does this call for an append-only/streaming pattern instead with the snapshot table only as a serving layer?

Hi everyone,

I have an upcoming interview for an ETL/Data Engineering (Fresher) role at Ameriprise Financial.

JD-

• Candidate should have a strong engineering foundation, preferably in Computer Science or related disciplines.
• Demonstrate eagerness to learn new technologies and adapt to evolving data integration tools.
• Support ETL operations by assisting with monitoring, troubleshooting, and maintaining workflows on platforms like Informatica PowerCenter, Informatica Cloud, and AWS Glue.
• Help ensure smooth execution of data pipelines and assist in resolving job failures and performance issues.
• Work with team members to optimize ETL processes and improve data quality and reliability.
• Basic understanding of SQL, databases, and data warehousing concepts along with strong problem-solving skills is desirable.

If anyone has interviewed there, could you please share:

• Number of interview rounds
• SQL questions
• Python coding questions (Easy/Medium?)
• ETL/Data Warehousing questions
• Informatica/AWS Glue questions (conceptual or hands-on?)
• Project-based questions
• Any tips or things I should prepare for?

My background: Python, SQL, Pandas, ETL projects using Python + SQL Server, but no hands-on experience with Informatica or AWS Glue.

Hello everyone, would like to know if anyone did the ETL pipelines from denodo to fabric migration? How tough it is or what do I need to know before starting the migration.

Like person I should be looking for expertise? Or any migration documentation I can follow.

TIA

Anybody out there using Transalis or Celtrino for EDI

Would be great to get some real world opinions - the good the bad the ugly

Thank you

Looking for practical lessons and real-world experiences from ETL professionals.

Ik it's an odd question to ask strangers on the internet, I have a friend who is an ETL developer for 3.5 years now and looking to switch jobs, and is there any possibility of y'all to hire him or refer him to your company?

Ps- He is super skilled and currently working in a major IT consulting and outsourcing company.

About Me

Senior Data Engineer with 5+ years of experience specializing in ETL/ELT pipeline design, Data Engineering, and Applied AI. Based in Bangalore, India. Available for remote work globally.

Rate: $25 - $50/hr depending on project scope and complexity.

ETL/ELT & Data Engineering Stack

Python, SQL, PySpark, Spark

Apache Airflow (DAG design, orchestration)

Databricks (Delta Lake, Unity Catalog)

AWS (S3, Glue, Athena, Lambda, EMR)

Snowflake, Redshift, BigQuery

dbt (data build tool)

Data Quality & Testing Frameworks

FastAPI, REST APIs

What I Can Help With

Design and build scalable ETL/ELT pipelines (batch & streaming)

Migrate on-prem pipelines to cloud

Orchestrate workflows with Airflow

Optimize slow pipelines and reduce costs

Data quality testing and validation frameworks

Build AI applications using LLMs, RAG, and agent-based workflows

Training & Mentorship

ETL/ELT Design & Data Engineering fundamentals

Apache Airflow & Orchestration

PySpark & Databricks

Snowflake & Cloud Data Warehousing

AWS for Data Engineering

In-person weekend sessions available in Bangalore. Remote sessions available globally.

Availability

Freelance projects & consulting

Part-time remote roles

Weekend training & mentorship

DM me with a brief description of your requirements!

We're in the middle of moving our ETL stack from Alteryx to Databricks and I wanted to see how others have approached this without it turning into a multi-year slog.

A few things we're figuring out:

• Workflow inventory first — cataloging every Alteryx workflow (inputs, tools used, macros, outputs, schedules) before touching any code, so we know what's actually in scope and can flag the "weird" ones early
• Mapping Alteryx tools → PySpark/SQL equivalents — most joins, filters, formulas, and summarize tools translate pretty cleanly to PySpark or Spark SQL, but some things (fuzzy match, certain spatial tools, batch macros) need custom logic
• Prioritizing by complexity + business value — starting with simple, high-value workflows to build momentum and reusable patterns, then tackling the gnarly ones
• Building reusable templates/notebooks — instead of one-off conversions, creating parameterized notebooks that mimic common Alteryx patterns (data blending, prep, output routing) so future workflows go faster
• Validation strategy — running old and new pipelines in parallel and diffing outputs row-by-row before cutting over, since silent transformation bugs are the scariest part
• Orchestration — moving from Alteryx Server schedules to Databricks Workflows/Jobs, and rethinking dependencies since Alteryx's visual flow doesn't map 1:1 to DAGs

Curious if anyone has:

1. Used or built a tool to auto-convert .yxmd files into PySpark scaffolding (even partial conversion would save time)
2. Tips for handling Alteryx macros (especially batch macros) in Databricks
3. War stories on what took way longer than expected

Appreciate any lessons learned.

Looking for real-world experiences and lessons learned from production ETL pipelines.

was running a handfull of python scripts as cron jobs and it got annoying fast. no logs when something failed, dependency conflicts between scripts, no resource limits. looked at airflow, immediately closed the tab. a postgres db, redis broker, scheduler and webserver for 10 scripts felt insane.

so i built Fast-Flow. a pipeline is just a folder with a main.py. you git push, it syncs, it runs. no DAGs, no decorators, no image builds. dependencies get installed on the fly with uv and cached, so runs are fast even without pre-building anythign. each run happens in its own docker container or kubernetes job, your choice.

also has live logs, oauth login, encrypted secrets, cron scheduling. self hosted, no telemetry.

not trying to replace airflow for real DAG workflows, this is more for "cron but actually managable"

GPL-3.0, docs included: https://github.com/ttuhin03/fastflow

i made it, happy to answer question

Hello to all,

I have curently changed my team and they only have MsSql as storage solution and PBI as reporting solution.

I want to create a proper architecture,that will cover ingestion,etl ,storage ,reporting and later on data science stuff (with keeping previous two stuff for storage and reporting)...

I do not know what softwares are the best for the job ,and some of the options that I know might pass in our company are as follows:

-MS Fabric

-Knime analytics(i have used this before) heavily

-Databricks ( one i am most interested in)

I am talking here about only ETL part that I can set automatic schedules....

I am still a junior so, I am in need of some advice which is the best ...or maybe even get some other advice that would help me!

Thanks upfront

Hi everyone,

I recently built an Amazon Data Analytics project that analyzes Amazon sales data and provides insights through data cleaning, visualization, and exploratory analysis.

Features:

Data preprocessing and cleaning

Exploratory Data Analysis (EDA)

Visualizations using Python

Sales and product trend analysis

I'd really appreciate any feedback on the code structure, analysis, or suggestions for improvement.

I burned through 6 million tokens on Claude Code last quarter. It started as a productivity boost, I was shipping faster than ever, crushing dbt models and ingestion scripts in hours instead of days.

Then Claude went down for about half an hour during a pretty routine ETL fix. I just sat there staring at the terminal. Couldn't remember how to structure a simple window function without it suggesting the whole thing. That's when it hit me: I've outsourced my working memory to an autocomplete tool.

Coding assistants are great for finishing your sentences, but they don't manage the full pipeline. They don't understand your schema drift, your monitoring, or your deployment steps. You still have to hold all that context yourself and stitch the pieces together. That's the real bottleneck, not writing the code but knowing what to write and where it fits.

I've been looking at platforms that try to automate more of the end-to-end data engineering workflow, Databricks, AI workflows, Genesis Data Agents, and Snowflake Cortex AI, but they all introduce their own failure modes. Debugging a multi-step agent decision is even harder than debugging a bad SQL query.

The tool doesn't fix the fundamental need to understand your data and your architecture. I'm now forcing myself to do one raw coding session per week without any AI help, just to keep the muscle memory alive. Feels like studying for a test I already passed, but the alternative is being useless when the API goes down.