I was offered a position as Data Analytics Engineer I at Checkout.com in London. The interviewers seemed nice, and I didn't notice any major red flags throughout the process. Three things make me hesitate taking the offer:

• The Glassdoor reviews are pretty bad, especially when it comes to their hybrid working policy. I don't mind having into come to the office, but some reviews mention a toxic culture across the company
• The people I spoke to mentioned that the team is heavily invested in agentic coding. Guess that is just the reality of how bigger companies write code now, so not necessarily a red flag? People I talked to seemed very competent though
• I am currently working as a Junior Data Engineer. I built my orgs data stack (deploying the stack on AWS, building some small pipelines, data modeling), though they decided to go all-in on AI now and there is little to no data work left for me right now. I enjoyed the infra work at my current org (though the largest data set is only a few 100k rows), so going from Data Engineer to Data Analytics Engineer feels a bit like a downgrade? From the job description and from what I heared during my interviews the role itself seems to be more on the technical side.

Anybody currently working at Checkout and willing to share some insights/experience on how it is like working there? Taking the role would mean a 50% salary bump, so I am very tempted.

Hello guys I'm a junior data engineer tasked to like fix a business logic for one of our tables. The thing is I'm worried that making additional cte's would make it costly and slower. The table that I would be sourcing from contains around >30gb. so this is just the basic gist of it

With temp as(
  SELECT
        *,
        (some transfomration here) as converted_date
  FROM source_table
),
temp2 as (
  SELECT
        *,
        (using converted_date column) as converted_date1 
  FROM temp),
temp3 as (
  SELECT
        *,
        (using converted_date1 column) as converted_date2
  FROM temp2)

SELECT * from temp3

So, I already did try to see how much the query will cost in Bigquery and it seems that it does not increase that much like just couple of hundred mb or <5gb. My question is that does anyone have experience doing things like this and does the cost really not change even if I used additional three cte? Like what are the potential problems that might occur if I proceed doing it like this?

Is Cybersecurity Data Engineer a real position or role in the corporate world? Curious to know if anyone has heard of such a position or role that blends data engineering and cyber security.

I recently saw a video where they were talking about Fabric Apps and how that would basically replace Power BI. They had it connected to some AI, would provide it with a link to a data model and it would spit out a fully rendered Fabric App page. The demo was decently impressive, as most demos are, but demos always lack that "grit" you see in normal day-to-day development - those scenarios where true experience and problem solving skills saves the day.

The app thing mostly caught my attention because I have some experience with Power BI specifically, but the topic is also about Fabric in general. Those of you who have or are working with Fabric - how is it? What problems does it solve? What are the downsides?

I did some googling and people from ~1 year ago would say that it's mostly half-baked Microslop crap. Has that changed at at?

I am looking for some good job portals in Australia for Data Engineering.

Trying to switch from production support to data engineer is a tough road I am looking at. Applied for a few jobs but don't seem to get any response and I am starting to think with so many people applying on LinkedIn, I feel LinkedIn in not a great portal right now.

Can you guys suggest what should I look at for data engineering roles.

Although seek is popular, I always thoughts it's for non-tech roles and I could be wrong.

Hello,

Has anyone worked with TrinoDB ? What are your opinions ?

I'm currently working on a possible migration out of BQ and TrinoDB looks like one of the solutions. What other alternatives do you work with ?

My basic needs are to connect to a BI, the product backend db, Elastic, GCS and S3 buckets and BigQuery. Some tools will write data directly to TrinoDB.

I will need a UI to expose this data to business to query and analyse.

Have you worked on similar setups ?

How do you set up the data layers (staging, facts, datamarts) ?

How would you make data available in real time analytics ?

Happy to discuss and share opinions.

Over the past year I have been working primarily in the Palantir Foundry system. My CEO unilaterally decided to pursue an enterprise agreement after being sold the AI dream. Palantir sales engineers did ‘analyses’ which suggested that the multimillion dollar price tag would result in 10x savings due to process and decision optimization. Our IT team cautioned no, but were steamrolled.

The project I am on was estimated to take four months and require 5 additional contract engineers. 2 of those were directly employed by Palantir as part of an additional contract (read: more $$) and the other 3 were a separate agency because Palantir said they don’t do XYZ work (again, more $$$). It took the externals plus me 15 months to deliver an MVP. This is primarily because we were building a complex enterprise grade app (which we previously subscribed to) on a low code platform. The Palantir engineers left as soon as MVP was deemed complete with just 30 days notice and since then I have taken on their SOW.

The work completed by the “brilliant” Palantir FDEs has been consistently failing. I’m finding they hardcoded dates. They hardcoded accounts. They used different inputs for the same business concepts. They ‘fixed’ issues that were earlier pointed out with hardcoded logic. They used AI FDE to code spaghetti mess logic. This has been a freaking nightmare.

My company had received 0 ROI to date and the CEO blames our IT team for the lack of delivery.

If you have the ability to run away from working with this god awful company and their charlatans RUN.

Hello everyone.

I will being going into my 3rd year of college this Fall, and am currently wrapping up a "Data Systems Engineering" internship at a large municipal utilities company. All I heard before I started this job was how great a company "ABC Utility Co" was to work for.

I have now been given a return offer for the Fall, and would be able to continue working remotely ~20 hrs/week for ~$20/hr. In theory this sounds like an utter DREAM. I knew at the start of the internship there was an opportunity to continue in the Fall, and it was one of the things I was most excited about. Now I feel differently...

The problem(s):
- I have coded probably less than 10% of my active hours worked. My main intern project deliverable is a long written report detailing the gaps I have identified with their current architecture. If this seems like a weird intern deliverable, it probably is. I was told on the second day of my internship I was supposed to "act like a consultant".

- I spent all day today crafting a Google Form to be used for data quality incident reporting. I spent all day. Making a very detailed Google Form. I wasn't specifically told to do this, but since I am basically not allowed to code (see below), I couldn't come up with anything better to do.

- I have zero access to our main ELT tool (Qlik Talend). The licenses are too expensive and they won't spend that money on an intern.

- I have zero access to our change data capture tool (IBM CDC), also in theory due to cost.

- "All these things are too expensive," and yet I'm not allowed to use free software either. Getting approval for a new open-source package takes FOREVER. In the sense that it has not happened the whole summer.

- This company probably only built 10% of its own infrastructure, and the other 90% of things were built by various contractors over the years. What this means is that when I ask questions like "How does X work?" or "Why is X done Y way?", nobody has an answer for me. I have tried reading the contractors' documentation, but it is vague and doesn't give me the answers I need. This is obviously hindering my report, but I am doing my best to piece together tribal knowledge scattered across departments, at times self-contradicting vendor documentation from two years ago, source code, and physical tables in the data warehouse.

- On that note, everyone keeps insisting that "we can't track deletes." I want to jump off a cliff, because -- correct me if I'm wrong -- CDC can **absolutely** track deletes. I have tried politely to express this, but we've had like three meetings over the past two weeks where our inability to track deletes (when ingesting from on-prem to the data warehouse) has gotten brought up.

- I have twice had to explain to people who are supposed to be senior employees that git is different from GitHub, and that BitBucket is not dependent on GitHub. I was also asked "Why would we want to use git?"

- There are also SQL scripts stored as .txt files in the company BitBucket (which just irks me), and two different 8000-line .sql files "to recreate all our tables if we lose them". But that seems like a load of nonsense to me because you could run 400 modular .sql scripts too. They don't need to be in one 8000-line file...

- My manager talks big game about AI-readiness and how they want to move away from Talend and modernize, but everything here moves like a snail through molassess.

My concern:

I want to actually learn things. I want to code and sharpen my technical skills (especially since I didn't become a CS major until sophomore year of college, which was this past year). I don't need to be hand-held, but I want to be supported by a knowledgable mentor. I do not think that exists here. Am I delusional in thinking that somehow I can still gain value from continuing here in the fall?

I think I am perhaps falling prey to sunk-cost fallacy. I spent the whole summer here with nothing worthwhile technical/code-related to put on my resumé, and think I have the notion that if I stay for fall I could make up for it. Or I feel like I've worked so, so hard this summer -- even if not on writing any code :( -- and if I stay, maybe I can finally put some of my ideas (from all the gaps I've identified) into practice and make an amazing story out of it for an intervew about how I turned things around.

I feel like reddit is a cynical place, but reddit also is aware how much internship experience matters, and how cooked the job market is right now. I guess my main question is if the title matters more, or the actual job duties. I just want advice/opinions, I suppose.

If it matters, I do not hope to return here next summer. If possible, I would like an internship at a "real" tech company, but I need to crack down on LeetCode for that (which is a separate convo). I am not married to data engineering as a future career, although it is interesting, and I am also interested in general software engineering, as well as DevOps, but I don't think anyone hires fresh grads for DevOps roles.

For context, the intervew gave the impression this would be a much more technical internship that it actually has been.

If you read this far, bless you, and any advice is appreciated <3

Hi everyone, I have a problem where I keep getting distracted when developing an AI data project. I would make plans in one conversation, execute code in another, return, and then lose track of my whereabouts. Every mistake required a fresh explanation of the entire project.

The code wasn't the issue. It was that my choices, the reasons behind them, and the things that are still broken were dispersed throughout my thoughts and chat windows.

I created devtrail as a result. A compact CLI that stores everything in a single file. Every line is sorted by a keyword when you log things as you go:

devtrail log "decision picked ClickHouse over Postgres"
devtrail log "error pipeline breaks on validation"
devtrail context   # prints everything, paste into any AI

It monitors the logic rather than the code, so it's not a "dump my codebase into markdown" tool. Works with any AI. Since Arabic is my first language, keywords also function in that language.

First package, so it's simple. Install with pip install devtrail

Currently, how do you all keep track of decisions made throughout sessions? I'm trying to figure out whether this is just a personal issue.

Repo: https://github.com/abdelrahman-essam-d/devtrail

Hi everyone,

I’d like to ask how you approach AI-driven spec-based development (SDD) for pipelines and ETL/ELT workflows, particularly in Databricks.

Based on my observations and what I’ve learned so far, the two simplest and most common approaches are short-lived specs and long-lived specs.

The short-lived approach is straightforward: you create a specification for a specific task and optionally store it in the repository. The long-lived approach is based on maintaining project documentation (for example, in Markdown files and other supporting documents), which is continuously expanded and updated as new tasks come up.

There are also frameworks that support this workflow, such as Superpowers or Spec Kit. Of course, the AI also needs the right context - an architecture description, perhaps a reference repository, coding standards, and anything else that helps it generate a good specification.

As for interacting with AI agents, the process is usually through an IDE or CLI. You iterate with the agent, discussing and refining the solution until you arrive at a complete specification.

My question is: what’s your approach to this in your projects? Do you use SDD only up until the point where you have a branch with the implementation and pipeline ready, or do you continue using it all the way through deployment?

I’m also curious about testing. In my projects, we write unit tests for our Python and Spark code, and then hand everything over to the QA team, which performs end-to-end testing.

Do you think the QA team should maintain a separate specification within their own testing framework (it is a custom tool we wrote to make testing components such as functions and SQLs reusable, written in Python), or how do you handle this in your projects?
End-to-end tests are time-consuming and expensive, so I’m wondering how you optimize them when adopting an SDD-based workflow.

I am currently in Service based company A (np 60days) & earlier (till last week) was working in a project mainly based on Informatica Power Center ,Azure ecosystem, some datalake , Power Bi, OBIEE .It was mostly a support and encouragement project .The project got over and went to a different service based company B (NP 90 days).

I am already allocated to a project(again support and enhancement) in my current company but this one is more ETL oriented. It uses AWS to ingest data in Redshift and then uses Informatica/ Datalake to transform the data.There are more Pyspark scripts in Glue job to analyze compared to previous one.

Today I recieved offer from the service based company B that they are willing to take me in the project (as they are looking for resource to stabilize the BI area).They are offering atleast 30% hike ( can be more based on negotiation with HR)

I am confused as to which way to go?Both are support and enhancement project (not pure devlopment).Should I stay back in the current company as the tech stack looks good or go to the other company where my technical skill might stagnate(may be there will be more datalake opportunities but that's for later as I have to support everything).Can someone suggest me?

YoE:5

Never switched before