I recently saw a video where they were talking about Fabric Apps and how that would basically replace Power BI. They had it connected to some AI, would provide it with a link to a data model and it would spit out a fully rendered Fabric App page. The demo was decently impressive, as most demos are, but demos always lack that "grit" you see in normal day-to-day development - those scenarios where true experience and problem solving skills saves the day.

The app thing mostly caught my attention because I have some experience with Power BI specifically, but the topic is also about Fabric in general. Those of you who have or are working with Fabric - how is it? What problems does it solve? What are the downsides?

I did some googling and people from ~1 year ago would say that it's mostly half-baked Microslop crap. Has that changed at at?

Hello,

Has anyone worked with TrinoDB ? What are your opinions ?

I'm currently working on a possible migration out of BQ and TrinoDB looks like one of the solutions. What other alternatives do you work with ?

My basic needs are to connect to a BI, the product backend db, Elastic, GCS and S3 buckets and BigQuery. Some tools will write data directly to TrinoDB.

I will need a UI to expose this data to business to query and analyse.

Have you worked on similar setups ?

How do you set up the data layers (staging, facts, datamarts) ?

How would you make data available in real time analytics ?

Happy to discuss and share opinions.

Hello guys I'm a junior data engineer tasked to like fix a business logic for one of our tables. The thing is I'm worried that making additional cte's would make it costly and slower. The table that I would be sourcing from contains around >30gb. so this is just the basic gist of it

With temp as(
  SELECT
        *,
        (some transfomration here) as converted_date
  FROM source_table
),
temp2 as (
  SELECT
        *,
        (using converted_date column) as converted_date1 
  FROM temp),
temp3 as (
  SELECT
        *,
        (using converted_date1 column) as converted_date2
  FROM temp2)

SELECT * from temp3

So, I already did try to see how much the query will cost in Bigquery and it seems that it does not increase that much like just couple of hundred mb or <5gb. My question is that does anyone have experience doing things like this and does the cost really not change even if I used additional three cte? Like what are the potential problems that might occur if I proceed doing it like this?

I am looking for some good job portals in Australia for Data Engineering.

Trying to switch from production support to data engineer is a tough road I am looking at. Applied for a few jobs but don't seem to get any response and I am starting to think with so many people applying on LinkedIn, I feel LinkedIn in not a great portal right now.

Can you guys suggest what should I look at for data engineering roles.

Although seek is popular, I always thoughts it's for non-tech roles and I could be wrong.

Hello everyone.

I will being going into my 3rd year of college this Fall, and am currently wrapping up a "Data Systems Engineering" internship at a large municipal utilities company. All I heard before I started this job was how great a company "ABC Utility Co" was to work for.

I have now been given a return offer for the Fall, and would be able to continue working remotely ~20 hrs/week for ~$20/hr. In theory this sounds like an utter DREAM. I knew at the start of the internship there was an opportunity to continue in the Fall, and it was one of the things I was most excited about. Now I feel differently...

The problem(s):
- I have coded probably less than 10% of my active hours worked. My main intern project deliverable is a long written report detailing the gaps I have identified with their current architecture. If this seems like a weird intern deliverable, it probably is. I was told on the second day of my internship I was supposed to "act like a consultant".

- I spent all day today crafting a Google Form to be used for data quality incident reporting. I spent all day. Making a very detailed Google Form. I wasn't specifically told to do this, but since I am basically not allowed to code (see below), I couldn't come up with anything better to do.

- I have zero access to our main ELT tool (Qlik Talend). The licenses are too expensive and they won't spend that money on an intern.

- I have zero access to our change data capture tool (IBM CDC), also in theory due to cost.

- "All these things are too expensive," and yet I'm not allowed to use free software either. Getting approval for a new open-source package takes FOREVER. In the sense that it has not happened the whole summer.

- This company probably only built 10% of its own infrastructure, and the other 90% of things were built by various contractors over the years. What this means is that when I ask questions like "How does X work?" or "Why is X done Y way?", nobody has an answer for me. I have tried reading the contractors' documentation, but it is vague and doesn't give me the answers I need. This is obviously hindering my report, but I am doing my best to piece together tribal knowledge scattered across departments, at times self-contradicting vendor documentation from two years ago, source code, and physical tables in the data warehouse.

- On that note, everyone keeps insisting that "we can't track deletes." I want to jump off a cliff, because -- correct me if I'm wrong -- CDC can **absolutely** track deletes. I have tried politely to express this, but we've had like three meetings over the past two weeks where our inability to track deletes (when ingesting from on-prem to the data warehouse) has gotten brought up.

- I have twice had to explain to people who are supposed to be senior employees that git is different from GitHub, and that BitBucket is not dependent on GitHub. I was also asked "Why would we want to use git?"

- There are also SQL scripts stored as .txt files in the company BitBucket (which just irks me), and two different 8000-line .sql files "to recreate all our tables if we lose them". But that seems like a load of nonsense to me because you could run 400 modular .sql scripts too. They don't need to be in one 8000-line file...

- My manager talks big game about AI-readiness and how they want to move away from Talend and modernize, but everything here moves like a snail through molassess.

My concern:

I want to actually learn things. I want to code and sharpen my technical skills (especially since I didn't become a CS major until sophomore year of college, which was this past year). I don't need to be hand-held, but I want to be supported by a knowledgable mentor. I do not think that exists here. Am I delusional in thinking that somehow I can still gain value from continuing here in the fall?

I think I am perhaps falling prey to sunk-cost fallacy. I spent the whole summer here with nothing worthwhile technical/code-related to put on my resumé, and think I have the notion that if I stay for fall I could make up for it. Or I feel like I've worked so, so hard this summer -- even if not on writing any code :( -- and if I stay, maybe I can finally put some of my ideas (from all the gaps I've identified) into practice and make an amazing story out of it for an intervew about how I turned things around.

I feel like reddit is a cynical place, but reddit also is aware how much internship experience matters, and how cooked the job market is right now. I guess my main question is if the title matters more, or the actual job duties. I just want advice/opinions, I suppose.

If it matters, I do not hope to return here next summer. If possible, I would like an internship at a "real" tech company, but I need to crack down on LeetCode for that (which is a separate convo). I am not married to data engineering as a future career, although it is interesting, and I am also interested in general software engineering, as well as DevOps, but I don't think anyone hires fresh grads for DevOps roles.

For context, the intervew gave the impression this would be a much more technical internship that it actually has been.

If you read this far, bless you, and any advice is appreciated <3

I am currently in Service based company A (np 60days) & earlier (till last week) was working in a project mainly based on Informatica Power Center ,Azure ecosystem, some datalake , Power Bi, OBIEE .It was mostly a support and encouragement project .The project got over and went to a different service based company B (NP 90 days).

I am already allocated to a project(again support and enhancement) in my current company but this one is more ETL oriented. It uses AWS to ingest data in Redshift and then uses Informatica/ Datalake to transform the data.There are more Pyspark scripts in Glue job to analyze compared to previous one.

Today I recieved offer from the service based company B that they are willing to take me in the project (as they are looking for resource to stabilize the BI area).They are offering atleast 30% hike ( can be more based on negotiation with HR)

I am confused as to which way to go?Both are support and enhancement project (not pure devlopment).Should I stay back in the current company as the tech stack looks good or go to the other company where my technical skill might stagnate(may be there will be more datalake opportunities but that's for later as I have to support everything).Can someone suggest me?

YoE:5

Never switched before