Over the past year I have been working primarily in the Palantir Foundry system. My CEO unilaterally decided to pursue an enterprise agreement after being sold the AI dream. Palantir sales engineers did ‘analyses’ which suggested that the multimillion dollar price tag would result in 10x savings due to process and decision optimization. Our IT team cautioned no, but were steamrolled.

The project I am on was estimated to take four months and require 5 additional contract engineers. 2 of those were directly employed by Palantir as part of an additional contract (read: more $$) and the other 3 were a separate agency because Palantir said they don’t do XYZ work (again, more $$$). It took the externals plus me 15 months to deliver an MVP. This is primarily because we were building a complex enterprise grade app (which we previously subscribed to) on a low code platform. The Palantir engineers left as soon as MVP was deemed complete with just 30 days notice and since then I have taken on their SOW.

The work completed by the “brilliant” Palantir FDEs has been consistently failing. I’m finding they hardcoded dates. They hardcoded accounts. They used different inputs for the same business concepts. They ‘fixed’ issues that were earlier pointed out with hardcoded logic. They used AI FDE to code spaghetti mess logic. This has been a freaking nightmare.

My company had received 0 ROI to date and the CEO blames our IT team for the lack of delivery.

If you have the ability to run away from working with this god awful company and their charlatans RUN.

I recently saw a video where they were talking about Fabric Apps and how that would basically replace Power BI. They had it connected to some AI, would provide it with a link to a data model and it would spit out a fully rendered Fabric App page. The demo was decently impressive, as most demos are, but demos always lack that "grit" you see in normal day-to-day development - those scenarios where true experience and problem solving skills saves the day.

The app thing mostly caught my attention because I have some experience with Power BI specifically, but the topic is also about Fabric in general. Those of you who have or are working with Fabric - how is it? What problems does it solve? What are the downsides?

I did some googling and people from ~1 year ago would say that it's mostly half-baked Microslop crap. Has that changed at at?

Hello guys I'm a junior data engineer tasked to like fix a business logic for one of our tables. The thing is I'm worried that making additional cte's would make it costly and slower. The table that I would be sourcing from contains around >30gb. so this is just the basic gist of it

With temp as(
  SELECT
        *,
        (some transfomration here) as converted_date
  FROM source_table
),
temp2 as (
  SELECT
        *,
        (using converted_date column) as converted_date1 
  FROM temp),
temp3 as (
  SELECT
        *,
        (using converted_date1 column) as converted_date2
  FROM temp2)

SELECT * from temp3

So, I already did try to see how much the query will cost in Bigquery and it seems that it does not increase that much like just couple of hundred mb or <5gb. My question is that does anyone have experience doing things like this and does the cost really not change even if I used additional three cte? Like what are the potential problems that might occur if I proceed doing it like this?

Hello,

Has anyone worked with TrinoDB ? What are your opinions ?

I'm currently working on a possible migration out of BQ and TrinoDB looks like one of the solutions. What other alternatives do you work with ?

My basic needs are to connect to a BI, the product backend db, Elastic, GCS and S3 buckets and BigQuery. Some tools will write data directly to TrinoDB.

I will need a UI to expose this data to business to query and analyse.

Have you worked on similar setups ?

How do you set up the data layers (staging, facts, datamarts) ?

How would you make data available in real time analytics ?

Happy to discuss and share opinions.

I was offered a position as Data Analytics Engineer I at Checkout.com in London. The interviewers seemed nice, and I didn't notice any major red flags throughout the process. Three things make me hesitate taking the offer:

• The Glassdoor reviews are pretty bad, especially when it comes to their hybrid working policy. I don't mind having into come to the office, but some reviews mention a toxic culture across the company
• The people I spoke to mentioned that the team is heavily invested in agentic coding. Guess that is just the reality of how bigger companies write code now, so not necessarily a red flag? People I talked to seemed very competent though
• I am currently working as a Junior Data Engineer. I built my orgs data stack (deploying the stack on AWS, building some small pipelines, data modeling), though they decided to go all-in on AI now and there is little to no data work left for me right now. I enjoyed the infra work at my current org (though the largest data set is only a few 100k rows), so going from Data Engineer to Data Analytics Engineer feels a bit like a downgrade? From the job description and from what I heared during my interviews the role itself seems to be more on the technical side.

Anybody currently working at Checkout and willing to share some insights/experience on how it is like working there? Taking the role would mean a 50% salary bump, so I am very tempted.

Hi everyone, I have a problem where I keep getting distracted when developing an AI data project. I would make plans in one conversation, execute code in another, return, and then lose track of my whereabouts. Every mistake required a fresh explanation of the entire project.

The code wasn't the issue. It was that my choices, the reasons behind them, and the things that are still broken were dispersed throughout my thoughts and chat windows.

I created devtrail as a result. A compact CLI that stores everything in a single file. Every line is sorted by a keyword when you log things as you go:

devtrail log "decision picked ClickHouse over Postgres"
devtrail log "error pipeline breaks on validation"
devtrail context   # prints everything, paste into any AI

It monitors the logic rather than the code, so it's not a "dump my codebase into markdown" tool. Works with any AI. Since Arabic is my first language, keywords also function in that language.

First package, so it's simple. Install with pip install devtrail

Currently, how do you all keep track of decisions made throughout sessions? I'm trying to figure out whether this is just a personal issue.

Repo: https://github.com/abdelrahman-essam-d/devtrail

Hi everyone,

I’d like to ask how you approach AI-driven spec-based development (SDD) for pipelines and ETL/ELT workflows, particularly in Databricks.

Based on my observations and what I’ve learned so far, the two simplest and most common approaches are short-lived specs and long-lived specs.

The short-lived approach is straightforward: you create a specification for a specific task and optionally store it in the repository. The long-lived approach is based on maintaining project documentation (for example, in Markdown files and other supporting documents), which is continuously expanded and updated as new tasks come up.

There are also frameworks that support this workflow, such as Superpowers or Spec Kit. Of course, the AI also needs the right context - an architecture description, perhaps a reference repository, coding standards, and anything else that helps it generate a good specification.

As for interacting with AI agents, the process is usually through an IDE or CLI. You iterate with the agent, discussing and refining the solution until you arrive at a complete specification.

My question is: what’s your approach to this in your projects? Do you use SDD only up until the point where you have a branch with the implementation and pipeline ready, or do you continue using it all the way through deployment?

I’m also curious about testing. In my projects, we write unit tests for our Python and Spark code, and then hand everything over to the QA team, which performs end-to-end testing.

Do you think the QA team should maintain a separate specification within their own testing framework (it is a custom tool we wrote to make testing components such as functions and SQLs reusable, written in Python), or how do you handle this in your projects?
End-to-end tests are time-consuming and expensive, so I’m wondering how you optimize them when adopting an SDD-based workflow.

Is Cybersecurity Data Engineer a real position or role in the corporate world? Curious to know if anyone has heard of such a position or role that blends data engineering and cyber security.

I am currently in Service based company A (np 60days) & earlier (till last week) was working in a project mainly based on Informatica Power Center ,Azure ecosystem, some datalake , Power Bi, OBIEE .It was mostly a support and encouragement project .The project got over and went to a different service based company B (NP 90 days).

I am already allocated to a project(again support and enhancement) in my current company but this one is more ETL oriented. It uses AWS to ingest data in Redshift and then uses Informatica/ Datalake to transform the data.There are more Pyspark scripts in Glue job to analyze compared to previous one.

Today I recieved offer from the service based company B that they are willing to take me in the project (as they are looking for resource to stabilize the BI area).They are offering atleast 30% hike ( can be more based on negotiation with HR)

I am confused as to which way to go?Both are support and enhancement project (not pure devlopment).Should I stay back in the current company as the tech stack looks good or go to the other company where my technical skill might stagnate(may be there will be more datalake opportunities but that's for later as I have to support everything).Can someone suggest me?

YoE:5

Never switched before

I am looking for some good job portals in Australia for Data Engineering.

Trying to switch from production support to data engineer is a tough road I am looking at. Applied for a few jobs but don't seem to get any response and I am starting to think with so many people applying on LinkedIn, I feel LinkedIn in not a great portal right now.

Can you guys suggest what should I look at for data engineering roles.

Although seek is popular, I always thoughts it's for non-tech roles and I could be wrong.

Hello everyone.

I will being going into my 3rd year of college this Fall, and am currently wrapping up a "Data Systems Engineering" internship at a large municipal utilities company. All I heard before I started this job was how great a company "ABC Utility Co" was to work for.

I have now been given a return offer for the Fall, and would be able to continue working remotely ~20 hrs/week for ~$20/hr. In theory this sounds like an utter DREAM. I knew at the start of the internship there was an opportunity to continue in the Fall, and it was one of the things I was most excited about. Now I feel differently...

The problem(s):
- I have coded probably less than 10% of my active hours worked. My main intern project deliverable is a long written report detailing the gaps I have identified with their current architecture. If this seems like a weird intern deliverable, it probably is. I was told on the second day of my internship I was supposed to "act like a consultant".

- I spent all day today crafting a Google Form to be used for data quality incident reporting. I spent all day. Making a very detailed Google Form. I wasn't specifically told to do this, but since I am basically not allowed to code (see below), I couldn't come up with anything better to do.

- I have zero access to our main ELT tool (Qlik Talend). The licenses are too expensive and they won't spend that money on an intern.

- I have zero access to our change data capture tool (IBM CDC), also in theory due to cost.

- "All these things are too expensive," and yet I'm not allowed to use free software either. Getting approval for a new open-source package takes FOREVER. In the sense that it has not happened the whole summer.

- This company probably only built 10% of its own infrastructure, and the other 90% of things were built by various contractors over the years. What this means is that when I ask questions like "How does X work?" or "Why is X done Y way?", nobody has an answer for me. I have tried reading the contractors' documentation, but it is vague and doesn't give me the answers I need. This is obviously hindering my report, but I am doing my best to piece together tribal knowledge scattered across departments, at times self-contradicting vendor documentation from two years ago, source code, and physical tables in the data warehouse.

- On that note, everyone keeps insisting that "we can't track deletes." I want to jump off a cliff, because -- correct me if I'm wrong -- CDC can **absolutely** track deletes. I have tried politely to express this, but we've had like three meetings over the past two weeks where our inability to track deletes (when ingesting from on-prem to the data warehouse) has gotten brought up.

- I have twice had to explain to people who are supposed to be senior employees that git is different from GitHub, and that BitBucket is not dependent on GitHub. I was also asked "Why would we want to use git?"

- There are also SQL scripts stored as .txt files in the company BitBucket (which just irks me), and two different 8000-line .sql files "to recreate all our tables if we lose them". But that seems like a load of nonsense to me because you could run 400 modular .sql scripts too. They don't need to be in one 8000-line file...

- My manager talks big game about AI-readiness and how they want to move away from Talend and modernize, but everything here moves like a snail through molassess.

My concern:

I want to actually learn things. I want to code and sharpen my technical skills (especially since I didn't become a CS major until sophomore year of college, which was this past year). I don't need to be hand-held, but I want to be supported by a knowledgable mentor. I do not think that exists here. Am I delusional in thinking that somehow I can still gain value from continuing here in the fall?

I think I am perhaps falling prey to sunk-cost fallacy. I spent the whole summer here with nothing worthwhile technical/code-related to put on my resumé, and think I have the notion that if I stay for fall I could make up for it. Or I feel like I've worked so, so hard this summer -- even if not on writing any code :( -- and if I stay, maybe I can finally put some of my ideas (from all the gaps I've identified) into practice and make an amazing story out of it for an intervew about how I turned things around.

I feel like reddit is a cynical place, but reddit also is aware how much internship experience matters, and how cooked the job market is right now. I guess my main question is if the title matters more, or the actual job duties. I just want advice/opinions, I suppose.

If it matters, I do not hope to return here next summer. If possible, I would like an internship at a "real" tech company, but I need to crack down on LeetCode for that (which is a separate convo). I am not married to data engineering as a future career, although it is interesting, and I am also interested in general software engineering, as well as DevOps, but I don't think anyone hires fresh grads for DevOps roles.

For context, the intervew gave the impression this would be a much more technical internship that it actually has been.

If you read this far, bless you, and any advice is appreciated <3

I built ssbt, a dbt inspired tool for engineers who deal with Excel Spreadsheets that don't want to deal with some of the overhead involved with dbt and you want to output as another xlsx file.

You can turn spreadsheets into a version-controlled, testable build pipeline using SQL and YAML. No database required due to duckdb being the main engine.

Chriscrpntr/ssbt: dbt inspired tool for engineers who deal with Excel Spreadsheets.

Saw some rumblings of people 'looking for new positions' on LinkedIn; did Starburst just do another round of layoffs (4th? 5th? at this point)? Wondering about the potential impact to our enterprise Trino support and if they're going to shut down operations at some point, and if we should start hunting for another alternative to use. Figured this was the best community to ask.

Disclosure: The platform offers a free starting experience, with additional premium practice content available for purchase, most exams cost 5 bucks, max 10.

Hi everyone!
I’ve been building Upskillz to help people prepare for data engineering certifications with realistic practice questions, exam simulations, detailed explanations, and progress tracking.
It currently supports:

dbt Analytics Engineer
Snowflake SnowPro Core
Snowflake SnowPro Data Engineer
Databricks Data Engineer Associate

https://upskillzlab.com

Best of luck to those studying🫶

I wanted to write this for probably a year now, finally sat down and did it. I've been using mmap in two append-only databases I've built and this is everything I learned actually understanding it, including why it works in our favor even though Andy Pavlo's paper says mostly don't use it.

Shared some content on data engineering based on my experiences. If you are also new to this learn from the article.

I work on the insights team at TouchBistro (a Canadian restaurant software company), and I led a project to pipe restaurant data from a sharded postgres operational database into Redshift using Debezium.

I found articles on the web which talked about Debezium and CDC more generally but not much on how to best use it with Redshift. We tried several options, and after settling on a solution and running it in production for a while we wanted to share our findings with the community in case others face similar challenges.

Hi all, I pivoted to DE 3 years ago as a junior and moved to a new DE role at a consultancy. I've had essentially no onboarding which I guess is somewhat normal as I've not been staffed on any project yet.

I was told to prep for certifications since they help sell to clients and I picked Fabric because I've had no exposure to it before. 3 days ago I was told I'll help as the sole DE on a Fabric IQ project which is mostly about workspace/roles configuration, reviewing semantic models, creating ontology and the Copilot agent.

I'm more than happy to take on new things but I don't want to become the fall guy or make a mess while I'm under probation too. I moved to get more exposure to AI since my first job was 90% spark programming on a Databricks lakehouse. Any advice on how to navigate this and succeed?

P.S. the guy who interviewed and hired me has already resigned! I am now reporting to his replacement.

Hey guys. We use postgres rn for the web app. I have got like 200gb of read only referential data. And planning to add more soon. Was wondering what's a good db that can store it in a compressed format. Good balance of storage and speed for analytics reports. Postgres we anyways needed for the oltp stuff so it was simpler to add it in there but it's easier to work with a smaller db.

I was seeing parquet can be queried by duck db directly but it's sync for the webapp. The only other option that seemed suitable was clickhouse which had lz4 and zstd compressions which were comparable to parquet. Also there's pg columnar extensions. What would you guys suggest.

I am a newly-hired data analyst in a company. I thought I will be assigned in data cleaning and data visualization but I was tasked to consolidate data from different social media platforms and create a pipeline from extracting data from supermetrics up to the visualization.

Now, I am worried that I do not have an experience to this. They do not have any system in handling data and I will solely handle all of this. I feel really overwhelmed and I just use claude in creating ETL pipelines. I still examine the python scripts so that I can assure that it is correct. I am still worried and I might just flipped due to stress.

My plan is to create a python script in extracting the data from supermetrics, then I will load it to bigquery for consolidation. Then, another script for cleaning before importing it into looker studio.

Can you help me guys navigate in this task? I do not have anyone from the company to ask if I am doing it right. I hope you could help me.

Hi, Guys! No so long time ago I had to start working with geo data and this was a whole new topic to cover for me. So I though to write short series for folks who might also work for the first time:

• Geospatial Data in Databricks — Part 0: Theory
• Geospatial Data in Databricks — Part 1: ST_* Functions
• Geospatial Data in Databricks — Part 2: H3_* functions
• Geospatial Data in Databricks — Part 3: Visualisation

Your feedback is highly appreciated.

My background is in data migration which somewhat overlaps with governance but not in great detail

I have been developing a version 1 of a data governance application. In essence, it scans your systems via Database connectors/APIs flagging data that does not match your SQL validation rules. These flags are emailed to Team A. If not resolved in x number of days, it is escalated to the Governance Team.

Is there any glaring features or processes I am missing?