Duckle is a local-first ETL/ELT framework powered by DuckDB.
You can now define pipelines in Python. DuckDB executes them as optimized, vectorized SQL. Your data stays on your machine from start to finish.

Why Duckle?

✅ No Python bottleneck
Pipelines are compiled into SQL before execution.
No rows flow through the Python interpreter.
No hidden to_pandas() conversions.

✅ Minimal setup
~20 MB install
Bundles the DuckDB CLI
No JVM
No Docker
No server
No account required

✅ Python-first API
import duckle
from duckle import col
(duckle.read_csv("orders.csv")
.where(col.amount >= 20)
.derive(total="round(amount * 1.2, 2)")
.write_parquet("out.parquet")
.run())

Write familiar Python expressions while Duckle translates them into efficient DuckDB SQL.

More than file transformations
Duckle includes 359 built-in components:
-104 Sources
-66 Sinks
-138 Transforms

Supporting databases and services including PostgreSQL, MySQL, SQL Server, Oracle, Snowflake, Databricks, Kafka, Salesforce, SAP OData, S3, SFTP, WebSocket, IMAP, LanceDB, dbt and many more.

Built for automation
Validate pipelines without connecting to data sources or requiring credentials.
duckle validate
duckle validate --json
duckle --pipeline my.json

Perfect for CI/CD, containers, cron jobs, and local development.

Code ↔ Visual Studio
The same pipeline can be authored in Python or opened directly in the Duckle visual studio because both use the same JSON format.

Open source. Local first. Built on DuckDB.
⭐ GitHub: https://github.com/slothflowlabs/duckle/
📦 PyPI: https://pypi.org/project/duckle/
🔗 Links: https://github.com/slothflowlabs/duckle#quick-links

Currently I use Excel files and manually capture data in these Excel files. I have other peers who also do a similar procedure and we all have different formats. Is a streamlit app a good solution to standardize the excel format ? Is it even feasible as something that can be deployed to multiple users through tablets so that they can register the data via a streamlit app ?

Trying to do cross-ISO work (PJM/MISO/ERCOT/CAISO/SPP/NYISO/ISO-NE) and I'm losing my mind reconciling seven different schemas and update cadences - the congestion component especially (NYISO's sign convention alone…). Right now it's a pile of per-ISO scrapers held together with tape. Is everyone just using gridstatus / rolling their own, or is there something that already normalizes all of this? Curious what SPP/MISO historical depth people actually get.

I may not be able to make you understand the problem in a better way, but below is the issue.

Has anybody built a trigger/job, say a destination in Airflow, that would run on the 1st of every month, and this job should depend on another job, say a source, which is running every day, and the destination job should only run after that source one is finished on the last day of every month. Let's say if source one failed on the 31st and the team got it correct, then ran it on the 1st, then also the destination should run (edge case).

The source itself has a stage, a silver, and an archive layer (to store the data as backup).

The destination job (it has only one sub-job, which was used earlier, so no change in this) should be triggered after the above activity.

Both the source and destination are based on a YAML-based job.

Can we connect these two jobs/yaml to make this work? Or how can we work around it?

Probably one for the Duck DB nerds. I came across a ETL/ELT open source (for now) tool called Duckle which look pretty solid if you just need a light weight ETL/ELT tool for smaller workloads on a laptop or local box. Anyone used it?

What cheaper ETL tools are you using in 2026? I don't want AI in them and don't want to be forced to use AI. However, I have basic requirements:

1. Able to connect to SharePoint for files, lists, etc.

2. Able to connect to AWS S3.

3. Able to connect to Snowflake databases.

The developers at the biotech company don't want to heavily use SQL for ETL but want to use a drag-and-drop UI to save time and focus more on analysis. We explored Alteryx, but it was expensive. dbt doesn't have an intuitive UI.

They also don't want to use AI since the biotech company is very new and wants to save money as much as possible.

EDIT : processing 100k rows as if now every day. But it will grow to more than 500k or 1 million plus rows after few years. Need a scalable solution with headless or client based etl tool with option to schedule the pipelines in the background.

Curious to learn what checks or validation techniques experienced data engineers use in production.

medium : https://medium.com/@abdul.haris.djafar/apitap-0-6-0-2277630a6a25

I work with data pipelines and kept running into the same problem.

Production data was unavailable or unsafe to reuse. Staging data was incomplete or broken. The fixtures I could create manually were too small to properly test the workflow.

So I built \*\*RowScale\*\*.

You upload a small valid sample or paste the data directly. RowScale analyzes it, infers the schema and generation rules, and lets you review or adjust ambiguous fields before generation.

Then you choose the number of rows and the output format.

It currently supports CSV, JSON, SQL INSERT, XLSX and Parquet as both input and output.

The raw sample is temporary and deleted after the run. Generated files are private, expire automatically, and every job includes a deletion receipt.

RowScale does not currently support direct database connections, public APIs, multi table relational generation or enterprise data anonymization.

The current focus is simple.

Start with a file or payload your system already understands and generate a much larger test dataset from it.

I launched it today and would really appreciate blunt feedback from people who test ETL jobs, imports, migrations or data pipelines.

\*\*What would RowScale need to handle before you would use it instead of writing a custom script?\*\*
\[https://rowscale.dev\\\](https://rowscale.dev/)

I work at Estuary, and next week we’re doing a live technical demo of our new Agent Skills.

We’ll show an AI coding agent:

• Creating a real-time capture and materialization
• Working from the same pipeline specifications an engineer would edit
• Checking task health and pipeline status
• Inspecting logs and troubleshooting failures
• Handling routine configuration without hiding what it changed

The interesting question for us is where agents genuinely reduce data engineering work today, versus where an engineer still needs to step in.

We’ll build the pipeline live and leave time for technical questions.

Registration: https://zoom.us/webinar/register/4517840417866/WN_1oZszaPUQ_Sz0uEGM_p-zA

Task: We operate a toll road network with 200 sensor-equipped lanes across 15 locations. Each sensor captures license plate reads, timestamps, and lane metadata. We need to compute real-time traffic volume, average transit times between checkpoints, and flag anomalies for toll evasion detection. Design the data model.

I modeled it as a single accumulating snapshot fact table, grain = one row per vehicle crossing (entry -> exit). The row is inserted at entry with status = OPEN and all exit columns NULL, then updated in place when the exit read comes in – no join needed for evasion detection, just WHERE status = 'OPEN' AND entry_time < now() - window.

I have a confusion: does update-in-place on an accumulating snapshot actually hold up under real-time write load (concurrent OPEN -> COMPLETED updates), or does this call for an append-only/streaming pattern instead with the snapshot table only as a serving layer?

Hi everyone,

I have an upcoming interview for an ETL/Data Engineering (Fresher) role at Ameriprise Financial.

JD-

• Candidate should have a strong engineering foundation, preferably in Computer Science or related disciplines.
• Demonstrate eagerness to learn new technologies and adapt to evolving data integration tools.
• Support ETL operations by assisting with monitoring, troubleshooting, and maintaining workflows on platforms like Informatica PowerCenter, Informatica Cloud, and AWS Glue.
• Help ensure smooth execution of data pipelines and assist in resolving job failures and performance issues.
• Work with team members to optimize ETL processes and improve data quality and reliability.
• Basic understanding of SQL, databases, and data warehousing concepts along with strong problem-solving skills is desirable.

If anyone has interviewed there, could you please share:

• Number of interview rounds
• SQL questions
• Python coding questions (Easy/Medium?)
• ETL/Data Warehousing questions
• Informatica/AWS Glue questions (conceptual or hands-on?)
• Project-based questions
• Any tips or things I should prepare for?

My background: Python, SQL, Pandas, ETL projects using Python + SQL Server, but no hands-on experience with Informatica or AWS Glue.

Hello everyone, would like to know if anyone did the ETL pipelines from denodo to fabric migration? How tough it is or what do I need to know before starting the migration.

Like person I should be looking for expertise? Or any migration documentation I can follow.

TIA

Anybody out there using Transalis or Celtrino for EDI

Would be great to get some real world opinions - the good the bad the ugly

Thank you

So we benchmarked Duckle against the ETL tools everyone already
uses, with every tool tuned to its BEST configuration:

The task is deliberately boring: read one CSV(TPC-H Width and style), land it as a table.
It is the single most common job in ETL.

Three things we want to be upfront about, because benchmarks are
easy to rig 🫣:

1) Duckle sits right on raw DuckDB
own load (CREATE TABLE lineitem AS SELECT * FROM read_csv('out/lineitem_20m.csv') floor (~16s to fully parse and write 20M typed rows to disk). Duckle wraps the engine with pipelines, connectors and
then gets out of its way. That is the whole design goal.

2) Talend and Informatica used their bulk output connectors at max
config, not the slow default row-by-row sink. On defaults they
5-7x slower. We did not want to strawman them.

3) Airbyte's number is derived from real 2M and 5M runs, and it needs
an always-on 8 GB platform just to start.

Duckle is free and open source.
Try it: https://github.com/slothflowlabs/duckle

I'm finding that while most of our ETL tests can be automated, a few complex validations still need manual checks. It's slowing down our testing process. How are others handling this?

Looking for practical lessons and real-world experiences from ETL professionals.

Ik it's an odd question to ask strangers on the internet, I have a friend who is an ETL developer for 3.5 years now and looking to switch jobs, and is there any possibility of y'all to hire him or refer him to your company?

Ps- He is super skilled and currently working in a major IT consulting and outsourcing company.

About Me

Senior Data Engineer with 5+ years of experience specializing in ETL/ELT pipeline design, Data Engineering, and Applied AI. Based in Bangalore, India. Available for remote work globally.

Rate: $25 - $50/hr depending on project scope and complexity.

ETL/ELT & Data Engineering Stack

Python, SQL, PySpark, Spark

Apache Airflow (DAG design, orchestration)

Databricks (Delta Lake, Unity Catalog)

AWS (S3, Glue, Athena, Lambda, EMR)

Snowflake, Redshift, BigQuery

dbt (data build tool)

Data Quality & Testing Frameworks

FastAPI, REST APIs

What I Can Help With

Design and build scalable ETL/ELT pipelines (batch & streaming)

Migrate on-prem pipelines to cloud

Orchestrate workflows with Airflow

Optimize slow pipelines and reduce costs

Data quality testing and validation frameworks

Build AI applications using LLMs, RAG, and agent-based workflows

Training & Mentorship

ETL/ELT Design & Data Engineering fundamentals

Apache Airflow & Orchestration

PySpark & Databricks

Snowflake & Cloud Data Warehousing

AWS for Data Engineering

In-person weekend sessions available in Bangalore. Remote sessions available globally.

Availability

Freelance projects & consulting

Part-time remote roles

Weekend training & mentorship

DM me with a brief description of your requirements!

We're in the middle of moving our ETL stack from Alteryx to Databricks and I wanted to see how others have approached this without it turning into a multi-year slog.

A few things we're figuring out:

• Workflow inventory first — cataloging every Alteryx workflow (inputs, tools used, macros, outputs, schedules) before touching any code, so we know what's actually in scope and can flag the "weird" ones early
• Mapping Alteryx tools → PySpark/SQL equivalents — most joins, filters, formulas, and summarize tools translate pretty cleanly to PySpark or Spark SQL, but some things (fuzzy match, certain spatial tools, batch macros) need custom logic
• Prioritizing by complexity + business value — starting with simple, high-value workflows to build momentum and reusable patterns, then tackling the gnarly ones
• Building reusable templates/notebooks — instead of one-off conversions, creating parameterized notebooks that mimic common Alteryx patterns (data blending, prep, output routing) so future workflows go faster
• Validation strategy — running old and new pipelines in parallel and diffing outputs row-by-row before cutting over, since silent transformation bugs are the scariest part
• Orchestration — moving from Alteryx Server schedules to Databricks Workflows/Jobs, and rethinking dependencies since Alteryx's visual flow doesn't map 1:1 to DAGs

Curious if anyone has:

1. Used or built a tool to auto-convert .yxmd files into PySpark scaffolding (even partial conversion would save time)
2. Tips for handling Alteryx macros (especially batch macros) in Databricks
3. War stories on what took way longer than expected

Appreciate any lessons learned.

Hi everyone,

I’ve been a Test Lead for over a decade, mostly focused on backend database and ETL validation in banking, insurance, and retail. On paper, it looks like a solid career. In reality, I feel like the ground is shifting under me, and I don't know how to keep up.

My strength is SQL and data logic, but even my confidence there has taken a hit lately. I’ve found myself leaning on AI tools to write scripts for me instead of doing it manually, which makes me doubt how sharp I actually still am. Meanwhile, job postings want Selenium, Playwright, APIs, BDD, CI/CD, and AI certifications... I have surface-level exposure to some of it, but no real depth.

Every time I sit down to try to plan how to fix this, I completely freeze. There are too many directions, too little time, and life doesn't pause for upskilling. I don't even know where to start or what a realistic plan looks like without burning out completely.

I'm posting because I need help getting out of this rut:

1. If you transitioned from ETL/Data testing into modern automation, how did you do it? What did you focus on first, and what tools actually matter vs. what is just keyword hype?
2. How do you structure learning when you're already exhausted from work? Any tips for breaking the paralysis?
3. Does anyone want to tackle this together? I don't have a concrete plan yet—I want to build one based on the feedback here. If you are also a mid-to-senior QA feeling lost and wants to connect, build a roadmap together, and keep each other accountable, please let me know.

I’d genuinely love any suggestions, course recommendations, or just to connect with anyone else who is trying to reinvent themselves right now. Would mean a lot to know I'm not doing this alone.