I recently saw a video where they were talking about Fabric Apps and how that would basically replace Power BI. They had it connected to some AI, would provide it with a link to a data model and it would spit out a fully rendered Fabric App page. The demo was decently impressive, as most demos are, but demos always lack that "grit" you see in normal day-to-day development - those scenarios where true experience and problem solving skills saves the day.

The app thing mostly caught my attention because I have some experience with Power BI specifically, but the topic is also about Fabric in general. Those of you who have or are working with Fabric - how is it? What problems does it solve? What are the downsides?

I did some googling and people from ~1 year ago would say that it's mostly half-baked Microslop crap. Has that changed at at?

Hello guys I'm a junior data engineer tasked to like fix a business logic for one of our tables. The thing is I'm worried that making additional cte's would make it costly and slower. The table that I would be sourcing from contains around >30gb. so this is just the basic gist of it

With temp as(
  SELECT
        *,
        (some transfomration here) as converted_date
  FROM source_table
),
temp2 as (
  SELECT
        *,
        (using converted_date column) as converted_date1 
  FROM temp),
temp3 as (
  SELECT
        *,
        (using converted_date1 column) as converted_date2
  FROM temp2)

SELECT * from temp3

So, I already did try to see how much the query will cost in Bigquery and it seems that it does not increase that much like just couple of hundred mb or <5gb. My question is that does anyone have experience doing things like this and does the cost really not change even if I used additional three cte? Like what are the potential problems that might occur if I proceed doing it like this?

Hello,

Has anyone worked with TrinoDB ? What are your opinions ?

I'm currently working on a possible migration out of BQ and TrinoDB looks like one of the solutions. What other alternatives do you work with ?

My basic needs are to connect to a BI, the product backend db, Elastic, GCS and S3 buckets and BigQuery. Some tools will write data directly to TrinoDB.

I will need a UI to expose this data to business to query and analyse.

Have you worked on similar setups ?

How do you set up the data layers (staging, facts, datamarts) ?

How would you make data available in real time analytics ?

Happy to discuss and share opinions.

I am currently in Service based company A (np 60days) & earlier (till last week) was working in a project mainly based on Informatica Power Center ,Azure ecosystem, some datalake , Power Bi, OBIEE .It was mostly a support and encouragement project .The project got over and went to a different service based company B (NP 90 days).

I am already allocated to a project(again support and enhancement) in my current company but this one is more ETL oriented. It uses AWS to ingest data in Redshift and then uses Informatica/ Datalake to transform the data.There are more Pyspark scripts in Glue job to analyze compared to previous one.

Today I recieved offer from the service based company B that they are willing to take me in the project (as they are looking for resource to stabilize the BI area).They are offering atleast 30% hike ( can be more based on negotiation with HR)

I am confused as to which way to go?Both are support and enhancement project (not pure devlopment).Should I stay back in the current company as the tech stack looks good or go to the other company where my technical skill might stagnate(may be there will be more datalake opportunities but that's for later as I have to support everything).Can someone suggest me?

YoE:5

Never switched before

I am looking for some good job portals in Australia for Data Engineering.

Trying to switch from production support to data engineer is a tough road I am looking at. Applied for a few jobs but don't seem to get any response and I am starting to think with so many people applying on LinkedIn, I feel LinkedIn in not a great portal right now.

Can you guys suggest what should I look at for data engineering roles.

Although seek is popular, I always thoughts it's for non-tech roles and I could be wrong.

Hello everyone.

I will being going into my 3rd year of college this Fall, and am currently wrapping up a "Data Systems Engineering" internship at a large municipal utilities company. All I heard before I started this job was how great a company "ABC Utility Co" was to work for.

I have now been given a return offer for the Fall, and would be able to continue working remotely ~20 hrs/week for ~$20/hr. In theory this sounds like an utter DREAM. I knew at the start of the internship there was an opportunity to continue in the Fall, and it was one of the things I was most excited about. Now I feel differently...

The problem(s):
- I have coded probably less than 10% of my active hours worked. My main intern project deliverable is a long written report detailing the gaps I have identified with their current architecture. If this seems like a weird intern deliverable, it probably is. I was told on the second day of my internship I was supposed to "act like a consultant".

- I spent all day today crafting a Google Form to be used for data quality incident reporting. I spent all day. Making a very detailed Google Form. I wasn't specifically told to do this, but since I am basically not allowed to code (see below), I couldn't come up with anything better to do.

- I have zero access to our main ELT tool (Qlik Talend). The licenses are too expensive and they won't spend that money on an intern.

- I have zero access to our change data capture tool (IBM CDC), also in theory due to cost.

- "All these things are too expensive," and yet I'm not allowed to use free software either. Getting approval for a new open-source package takes FOREVER. In the sense that it has not happened the whole summer.

- This company probably only built 10% of its own infrastructure, and the other 90% of things were built by various contractors over the years. What this means is that when I ask questions like "How does X work?" or "Why is X done Y way?", nobody has an answer for me. I have tried reading the contractors' documentation, but it is vague and doesn't give me the answers I need. This is obviously hindering my report, but I am doing my best to piece together tribal knowledge scattered across departments, at times self-contradicting vendor documentation from two years ago, source code, and physical tables in the data warehouse.

- On that note, everyone keeps insisting that "we can't track deletes." I want to jump off a cliff, because -- correct me if I'm wrong -- CDC can **absolutely** track deletes. I have tried politely to express this, but we've had like three meetings over the past two weeks where our inability to track deletes (when ingesting from on-prem to the data warehouse) has gotten brought up.

- I have twice had to explain to people who are supposed to be senior employees that git is different from GitHub, and that BitBucket is not dependent on GitHub. I was also asked "Why would we want to use git?"

- There are also SQL scripts stored as .txt files in the company BitBucket (which just irks me), and two different 8000-line .sql files "to recreate all our tables if we lose them". But that seems like a load of nonsense to me because you could run 400 modular .sql scripts too. They don't need to be in one 8000-line file...

- My manager talks big game about AI-readiness and how they want to move away from Talend and modernize, but everything here moves like a snail through molassess.

My concern:

I want to actually learn things. I want to code and sharpen my technical skills (especially since I didn't become a CS major until sophomore year of college, which was this past year). I don't need to be hand-held, but I want to be supported by a knowledgable mentor. I do not think that exists here. Am I delusional in thinking that somehow I can still gain value from continuing here in the fall?

I think I am perhaps falling prey to sunk-cost fallacy. I spent the whole summer here with nothing worthwhile technical/code-related to put on my resumé, and think I have the notion that if I stay for fall I could make up for it. Or I feel like I've worked so, so hard this summer -- even if not on writing any code :( -- and if I stay, maybe I can finally put some of my ideas (from all the gaps I've identified) into practice and make an amazing story out of it for an intervew about how I turned things around.

I feel like reddit is a cynical place, but reddit also is aware how much internship experience matters, and how cooked the job market is right now. I guess my main question is if the title matters more, or the actual job duties. I just want advice/opinions, I suppose.

If it matters, I do not hope to return here next summer. If possible, I would like an internship at a "real" tech company, but I need to crack down on LeetCode for that (which is a separate convo). I am not married to data engineering as a future career, although it is interesting, and I am also interested in general software engineering, as well as DevOps, but I don't think anyone hires fresh grads for DevOps roles.

For context, the intervew gave the impression this would be a much more technical internship that it actually has been.

If you read this far, bless you, and any advice is appreciated <3

I built ssbt, a dbt inspired tool for engineers who deal with Excel Spreadsheets that don't want to deal with some of the overhead involved with dbt and you want to output as another xlsx file.

You can turn spreadsheets into a version-controlled, testable build pipeline using SQL and YAML. No database required due to duckdb being the main engine.

Chriscrpntr/ssbt: dbt inspired tool for engineers who deal with Excel Spreadsheets.

Disclosure: The platform offers a free starting experience, with additional premium practice content available for purchase, most exams cost 5 bucks, max 10.

Hi everyone!
I’ve been building Upskillz to help people prepare for data engineering certifications with realistic practice questions, exam simulations, detailed explanations, and progress tracking.
It currently supports:

dbt Analytics Engineer
Snowflake SnowPro Core
Snowflake SnowPro Data Engineer
Databricks Data Engineer Associate

https://upskillzlab.com

Best of luck to those studying🫶

Saw some rumblings of people 'looking for new positions' on LinkedIn; did Starburst just do another round of layoffs (4th? 5th? at this point)? Wondering about the potential impact to our enterprise Trino support and if they're going to shut down operations at some point, and if we should start hunting for another alternative to use. Figured this was the best community to ask.

I wanted to write this for probably a year now, finally sat down and did it. I've been using mmap in two append-only databases I've built and this is everything I learned actually understanding it, including why it works in our favor even though Andy Pavlo's paper says mostly don't use it.

Shared some content on data engineering based on my experiences. If you are also new to this learn from the article.

I work on the insights team at TouchBistro (a Canadian restaurant software company), and I led a project to pipe restaurant data from a sharded postgres operational database into Redshift using Debezium.

I found articles on the web which talked about Debezium and CDC more generally but not much on how to best use it with Redshift. We tried several options, and after settling on a solution and running it in production for a while we wanted to share our findings with the community in case others face similar challenges.

Hi all, I pivoted to DE 3 years ago as a junior and moved to a new DE role at a consultancy. I've had essentially no onboarding which I guess is somewhat normal as I've not been staffed on any project yet.

I was told to prep for certifications since they help sell to clients and I picked Fabric because I've had no exposure to it before. 3 days ago I was told I'll help as the sole DE on a Fabric IQ project which is mostly about workspace/roles configuration, reviewing semantic models, creating ontology and the Copilot agent.

I'm more than happy to take on new things but I don't want to become the fall guy or make a mess while I'm under probation too. I moved to get more exposure to AI since my first job was 90% spark programming on a Databricks lakehouse. Any advice on how to navigate this and succeed?

P.S. the guy who interviewed and hired me has already resigned! I am now reporting to his replacement.

Hey guys. We use postgres rn for the web app. I have got like 200gb of read only referential data. And planning to add more soon. Was wondering what's a good db that can store it in a compressed format. Good balance of storage and speed for analytics reports. Postgres we anyways needed for the oltp stuff so it was simpler to add it in there but it's easier to work with a smaller db.

I was seeing parquet can be queried by duck db directly but it's sync for the webapp. The only other option that seemed suitable was clickhouse which had lz4 and zstd compressions which were comparable to parquet. Also there's pg columnar extensions. What would you guys suggest.

I am a newly-hired data analyst in a company. I thought I will be assigned in data cleaning and data visualization but I was tasked to consolidate data from different social media platforms and create a pipeline from extracting data from supermetrics up to the visualization.

Now, I am worried that I do not have an experience to this. They do not have any system in handling data and I will solely handle all of this. I feel really overwhelmed and I just use claude in creating ETL pipelines. I still examine the python scripts so that I can assure that it is correct. I am still worried and I might just flipped due to stress.

My plan is to create a python script in extracting the data from supermetrics, then I will load it to bigquery for consolidation. Then, another script for cleaning before importing it into looker studio.

Can you help me guys navigate in this task? I do not have anyone from the company to ask if I am doing it right. I hope you could help me.

Hi, Guys! No so long time ago I had to start working with geo data and this was a whole new topic to cover for me. So I though to write short series for folks who might also work for the first time:

• Geospatial Data in Databricks — Part 0: Theory
• Geospatial Data in Databricks — Part 1: ST_* Functions
• Geospatial Data in Databricks — Part 2: H3_* functions
• Geospatial Data in Databricks — Part 3: Visualisation

Your feedback is highly appreciated.

My background is in data migration which somewhat overlaps with governance but not in great detail

I have been developing a version 1 of a data governance application. In essence, it scans your systems via Database connectors/APIs flagging data that does not match your SQL validation rules. These flags are emailed to Team A. If not resolved in x number of days, it is escalated to the Governance Team.

Is there any glaring features or processes I am missing?

I am writing a website called The Ledger where I talk about interesting data structures that I have read about (or will read about). This article talks about LSM trees, I am also in the progress of writing my own lsm-based database engine, you can checkout the codebase for that here: github. And the crate here: crates.io .

Note: This website is free, it just serves as personal learnings I want to share with others.

Nothing quite beats the pain of being stuck in an hour-and-a-half-long meeting where the quasi- or non-technical executives brag about technical accomplishments and debate future choices.

I work in clinical research and a lot of the stuff we do is SQL scripting. I really want to move away from that and take my career to the next level, and as such have worked on my own projects to sharpen my skillset and create a portfolio for job hunting. Unfortunately, every time I get close to a potential offer, a better candidate comes along. I’ve asked for feedback and many have told me it’s due to my lack of real world experience in cloud technology. As such, I was wondering if getting a DE certification would help my chances of finding a job in this space. If so, what platform and certification should I prioritize?

I'm looking for advice on how small teams triage data-quality issues.

We have a small startup, ~60 people. The data team is 1 data engineer and 2 data analysts. They have struggled to establish a consistent triaging process, saying the other side isnt doing enough to support.

By way of example, here's today's incident involving a core operational system that is shared across the entire organization:

DA discovered that report had stale data, traced the lineage back to the raw tables, and confirmed the raw table hadnt received data new data since 6/30.

DA shared the query and issue and asked DE for support.

DE identified that the ingestion was running and has not thrown any errors. Recommends to DA to check the source.

DA says they're not trained on the source and dont know how to confirm.

DE says its not their job to check the source, citing "principles of data governance" and the person that raises the issue must confirm data present in the source.

At this point, we're effectively deadlocked. My observation is that the data in question is an backend abstraction and not something I would expect the business owner to understand or be able to support through triage.

How do you divide responsibility during triage? When does handoff happen and what does it include? What accommodations would you make on a small, understaffed team?

Before anyone mentions it: yes, we're working on getting freshness tests in place.