r/dataanalysis • u/ian_the_data_dad • 22h ago
r/dataanalysis • u/cool_beans_and_goats • 5h ago
Excel question…
The company I work for distributes products.
We created our item numbers based off of theirs but with our own identifiers.
Example : ABC-1234 (ABC = Identifier and 1234 = manufacturer part number)
The manufacturer has recently renumbered their products and now we have to add the new numbers to the existing descriptions so that they match up to the old ones.
I have an excel sheet with one column (A) showing our part number and the one next to it (B) showing their new number for the same product.
I have a different file that I exported all of our numbers and descriptions to and now I need to take our part number, and their new one, and add it to the export so I can upload to our system.
What is the most efficient way to get this done?
Thank you!!!
r/dataanalysis • u/RealPunk99 • 1d ago
Project Feedback The dashboard I did to win a small competition at my academy
I know it's an extremely basic one.
My teacher taught me Power BI basics and asked me to incorporate all the things into a project.
At the same time,she said that there's a project and I need to submit it within a week or so.
So i spent 2-3 days on this Spotify dataset,
Tried to incorporate all possible charts and stuff.
Please give me your feedback fellow redditors.
What should be the aim?
Communicate clearly even if charts,colour themes are simple,or such glitzy slides should be there in a presentation to elevate it?
For those who wanna know what I won,
I just won 500 Rs lol(But I was happy that my teacher noticed my efforts and appreciated my dashboard)
If any of you have done any nice dashboards, please share in comments section.
I might take some inspiration from it.
r/dataanalysis • u/Rajiv_2002 • 11h ago
Guidance needed
I am learning Python from Youtube till now l am done with basics like logical operators,if-else,Time, Maths, Dictionary, Random, Indexing,Loops, Collections, Functions, Arguments and return.
Should I move to Numpy or there are still things left if I am preparing for Data analyst job.
r/dataanalysis • u/Sh_HolmesB211 • 13h ago
Data Question How to explain the findings
Hello everyone, hope you're doing well.
I don't have a problem with the technical stuff, I'm still learning but I know I can be good at it, my problem is after I collect,clean and analyze the data and visualize it, Idk what I should write for example in the README on GitHub? How can I explain why the sells dropped on March for example ? It just dropped, how can I know why ? And how can I help in decision making and give advice ??
I only see tutorials on how to use the tools not on how to think and understanding the findings..
Any advice ?
r/dataanalysis • u/Effective_Ocelot_445 • 1d ago
What’s one data analysis habit that significantly improved the accuracy of your insights?
I'd love to hear the small practices or workflows that made the biggest difference in your day-to-day analysis.
r/dataanalysis • u/Previous-Mixture-456 • 17h ago
Guys, what is currently trending in buisness intelligence
I need a new trending tool or solution in business intelligence and data analysis. I'm planning a brown bag session on that.
r/dataanalysis • u/Exact_Entertainer600 • 1d ago
Data Question How do you handle messy date formats when merging datasets from different sources?
Working on a project that pulls data from a few different internal systems and every single one stores dates differently. One uses MM/DD/YYYY, another stores them as Unix timestamps, one is just plain text like Jan 5 2026 with no consistency in spacing or zeropadding. Merging them into a single clean table has been way more painful than expected.
The actual analysis part is straightforward once everything lines up, but getting there is eating up most of the time. Right now I'm converting everything to ISO 8601 before any joins, which works, but the transformation logic is getting long and brittle. If a new source comes in with a slightly different format it tends to break something upstream.
Curious how others structure this. Do you build format detection into the pipeline or just document the expected input format and enforce it at ingestion? Also wondering if this is one of those things where a more rigid schema at the source would have saved a ton of pain downstream, or if messy inputs are just the reality and you build around them.
The specific tools here are Python and SQL but I'm more interested in the general approach than syntax.
r/dataanalysis • u/Altruistic-Nature583 • 2d ago
Project Feedback Review my first BI daashboard
I made this today, its not a unique one by any means and was meant as something to have practice for my portfolio. How bad/good is it? I mean, is it worthy of anything or should i never show it to anyone or post iit anywhere? It def took me some time too finish, but it was a good first dashboard acc to me. Do roast me if u feel like its sht :(
r/dataanalysis • u/stifenahokinga • 1d ago
Data Tools Help with classification of email contents
I have many emails that pretty much answered a question that I formulated to the addressees. I predict that these replies gave a specific type of answer to that question: So imagine that the question overall was "do you think that ice cream is the best dessert that exists?" and I want to see how many if them answered something like "yes it is!", so that no matter how the reply is formulated, it basically answers something along these lines
I would like to use an AI to see the degree of accuracy of this prediction, but in this case the test has to be blinded
I have transformed all emails into a big pdf document with many pages (although I have not seen the contents of these pdfs of course) and I have joined them into a single pdf (about 100 emails in total).
I did another pdf document of "expected" answers. In this document I posted the original question that I asked to all the addressees (the questions are overall the same, but the details change in each case, so there is pretty much a unique question by email) and then I actually wrote the type of answer that I expect. Then I my intention is to ask the AI model to check the degree of accuracy or similarity that my written "expected" answers have with the actual ones, and then ask it to give me a number like a percentage or score.
However, I have a couple of problems:
The first is the prompt, I don't know how a good prompt could be. I have written the following prompt, but I think it looks very noobie, so I don't know if the AI model could completely understand the task:
>Now could you check the answers from the previous PDF and compare with this other PDF with the expected answers for the same emails? So, in this new PDF each email from the previouse PDF is collected and here we have put the answers that we expect to get from the scientists. The details of each expected answer do not matter too much, it does not have to coincide perfectly with the actual answers, but the overall question has to be answered in the same way (for example, if I asked someone "Do you think that ice cream is the best dessert in the world?" they may reply "yes, it is indeed" or "well it is one of my favourites but not my single one", so these replies are not exactly the same, but they have basically answered affirmatively so it would count as the same type of answer). Therefore, can you give a percentage of coincidence between the expected and actual answers, so that if the actual answer is basically and overall the same as the expected ones, the percentage would be higher and higher?
Another problem is the AI model to use. I have used perplexity to do some tests with a very small set of emails that I made up (5 emails where the questions and the answers were known and compared with "expected answers" that were written after seeing the actual ones), and it is pretty good, but the test has just 5 emails (compared to the near 100 of the actual task) and some people have told me in reddit that perplexity would not be a good option for this task... I have tried ChatGPT as well but it is somewhat worse than perplexity in my experience... So I don't know what to do
And finally, another big problem is that in many cases the AI themselves comment the results even thought I told them not to, so I fear that, when doing the real test, they may start revealing the actual answers when presenting the results
I was thinking as another option that perhaps someone here with a lot more of knowledge in these kinds of tasks could contact me so that I can send them the PDF (via private message) and do the test for me if they now how to wirte prompts and perform these kinds of tasks properly. In principle the task is simple, just tell the AI to compare the PDFs with the actual and expected answers somehow and then send me the percentage of coincidence (without telling me the details that the AI may comment about the documents). I am even willing to pay whoever does this, if the price is reasonable!
So if you could help me this wold be greatly appreciated
r/dataanalysis • u/Tryhard_314 • 1d ago
Data Tools Python script for LLM-Based Hierarchical Topic Modeling / Data Normalization
Hello! A couple months ago I asked on this sub for ways to do Hierarchical Topic Modeling (mapping strings to categories and sub-categories) standard sentences embeddings approaches such as BERTopic couldn't follow the instructions I wanted to give them.
I decided to implement a custom LLM-Based script that maps raw variations to categories and sub-categories by following the users instruction and looping until all values are mapped. It's not perfect but through some iteration it reached a step where I feel like I can share it.
If you've faced a similar problem with Hierarchical Topic Modeling, this may be helpful for you.
How it works:
- The user starts the script and give the path to a jsonl file with unnormalized values.
- The user sees a sample of values that are currently unmapped, he gives an instruction to specify how to normalize the fields, he can also set some fields to be ignored.
- The user enters how many unmapped values to give to the LLM for mapping in this iteration
- The script sends the user instruction, a sample of unmapped values and a sample of the values in the current config to the LLM and asks for the new mappings.
- The user can also put the LLM in autocomplete mode (it loops with the same previous instruction until all values are mapped)
- The script outputs 2 files: a normalization_config.json file (with the raw variations to categories mappings) and a normalized_data.jsonl file with the same schema as the original data file expected raw variations are replaced by the new categories (following this format: Category|SubCategory)
Limitations:
1- Dataset size: Best for datasets with less than 30k rows (if you want more consider using a cheaper model such as deepseek-v4).
2- Some manual tunning is needed, the script generates a config file, for optimal results you might want to tune it in an LLM chat session (use one that can support long copy-pastes, I personally use deepseek here).
3- It can't map a value to multiple categories for now.
Honestly it works pretty well for my use case, I use it frequently for normalizing data extracted out of reddit with LLMs, it helps me not set a strict taxonomy for the LLM at the extraction phase and get more unbiased results.
If you tried it let me know if it helped you or if you faced some problems with it send me a message and I'll fix them right away, I isolated it from a larger script so it might not work perfectly out of the box.
https://github.com/Tryhard-cs/LLM-Hierarchical-Topic-Modeling
r/dataanalysis • u/Sad_Acanthaceae_2718 • 1d ago
analyzing negative values
how do i analyse a continuos dependent variable with negative values. The negative values are indicative of direction rather than magnitude
r/dataanalysis • u/IndependenceFit3935 • 1d ago
Looking for Honest Feedback on my Football Intelligence App
Past couple of months I have been working on an end-to-end data engineering project. the idea was a demo but i became so fond of the project that i have decided to turn it into a live product.
Basically it is a BI dashboard on football data. Data is updated daily via a fully automated pipeline. I ingest data from Sportmonks API, transform it with dbt and serve BI on evidence.dev. I think it is worth noting that the budget for this project is $0.00 which shaped the technical choices along the way.
Feedback im looking for is about the ease of use, value of insights it provides, or any other ideas that could add value to it.
Have a look at it here: Superliga Analytics — Danish football intelligence
It is also available as PWA so that it sits on your mobile home screen, providing an app experience.
r/dataanalysis • u/helovesbreakfast • 1d ago
Project Feedback Feedback on dashboard - Bike Equity
Hi all, I'm looking for feedback on the dashboard for a learning and portfolio project. I'm hoping to know if it is easy to draw conclusions without consulting a write up to learn the methodology behind the metrics. Any and all questions and critiques are welcome as I'm very new to data analytics, from a CS background.
Made in Tableau, data sourced from ACS and People For Bikes, cleaned and interpolated with Python (geopandas, numpy, tobler).
The first pic is without a selection, and the others are with a community selected.
r/dataanalysis • u/FixelSmith • 2d ago
How I backtest a fraud rule before it ships
Most fraud rules ship on intuition. Somebody sees a pattern in last week’s confirmed cases, writes a threshold that would have caught them, and pushes it live. Then the team spends the next month discovering what else the rule catches, one false positive at a time.
If you keep alert history and disposition labels, you already have most of what you need to test a rule against the past before it touches production.
The backtest is four queries: replay the rule, compare it with known outcomes, check how much it overlaps with existing rules, and sweep the threshold.
I run all four on anything I propose. The overlap query kills more rule ideas than the other three combined.
What you need before any of this works
Three tables:
• transactions — the events themselves, with the attributes the rule uses.
• alerts — every alert the existing rules fired, ideally linked to a transaction or case.
• dispositions — the analyst’s verdict: confirmed fraud, confirmed legitimate, or unresolved.
Without disposition data, you can estimate volume. You can’t estimate whether the alerts would be useful. Disposition discipline was the first of the five wins in the noisy-alerts post, and this is where it pays off a second time: the labels you recorded to tune old rules are what let you test new ones for free.
Chargebacks, customer reports, and manual referrals make the test better because they include fraud that didn’t come through the current rules. Pull those outcomes in too. Otherwise a new rule gets evaluated almost entirely against cases the old rules already found.
I usually combine those sources into a known outcomes view with one row per transaction and an outcome of fraud or legitimate. The one-row rule matters. A transaction confirmed through two different channels shouldn’t get counted twice.
Query 1: replay the rule against history
Write the candidate rule as a plain SELECT over historical transactions. No deployment, no rules engine, no feature branch.
If the rule is “flag transactions over $200 where the account used a new device in the last 24 hours,” the replay is:
-- Candidate rule replayed over the last 6 months
SELECT
t.transaction_id,
t.account_id,
t.amount,
t.created_at
FROM transactions t
WHERE t.amount > 200
AND EXISTS (
SELECT 1
FROM device_events d
WHERE d.account_id = t.account_id
AND d.first_seen_at >= t.created_at - INTERVAL '24 hours'
AND d.first_seen_at <= t.created_at
)
AND t.created_at >= CURRENT_DATE - INTERVAL '6 months';
Six months is my starting point, not a requirement.
If the business is seasonal, the window needs to include the relevant season. If the checkout flow or device-tracking logic changed three months ago, six months of history may be worse than three because half the test no longer represents production.
The row count is the first sanity check. If the replay returns 40,000 transactions over roughly six months, the rule would generate about 220 alerts a day. If the team clears 500 alerts a day across the entire rule set, this one rule would take almost half its capacity.
There’s no reason to debate precision until the volume problem has an answer.
Query 2: compare the replay with known outcomes
I report two rates here because they answer different questions:
WITH replay AS (
-- Query 1 goes here
)
SELECT
COUNT(*) AS would_fire,
COUNT(*) FILTER (
WHERE o.outcome = 'fraud'
) AS known_fraud,
COUNT(*) FILTER (
WHERE o.outcome IN ('fraud', 'legitimate')
) AS labeled_events,
ROUND(
100.0 * COUNT(*) FILTER (WHERE o.outcome = 'fraud')
/ NULLIF(
COUNT(*) FILTER (
WHERE o.outcome IN ('fraud', 'legitimate')
),
0
),
2
) AS precision_among_labeled_pct,
ROUND(
100.0 * COUNT(*) FILTER (WHERE o.outcome = 'fraud')
/ NULLIF(COUNT(*), 0),
2
) AS known_fraud_rate_pct
FROM replay r
LEFT JOIN known_outcomes o
ON o.transaction_id = r.transaction_id;
There’s a problem with both numbers.
Most labels exist because an old rule fired and an analyst looked at the transaction. Transactions the old rules missed are much less likely to have a disposition at all.
precision_among_labeled_pct ignores transactions with no outcome. It’s the cleaner precision calculation, but it inherits the bias in the labeling process.
known_fraud_rate_pct includes every replayed transaction in the denominator, even the unlabeled ones. I treat it as a conservative known-fraud rate, not as measured production precision.
Unlabeled doesn’t mean legitimate. It means you don’t know.
I still use 2 percent as a first-pass bar, applied to precision_among_labeled_pct, when I’m comparing rules evaluated through the same label pipeline. A candidate that can’t clear that bar against known outcomes usually doesn’t get better after it starts creating work. But the percentage only means something when the rules were measured the same way.
Query 3: incremental catch, the query that kills most rules
A candidate can match plenty of known fraud and still add almost no coverage.
The cleanest overlap check uses the transaction or case that caused the alert:
WITH replay AS (
-- Query 1 again
),
candidate_fraud AS (
SELECT
r.transaction_id,
r.account_id,
r.created_at
FROM replay r
JOIN known_outcomes o
ON o.transaction_id = r.transaction_id
AND o.outcome = 'fraud'
),
coverage AS (
SELECT
c.*,
EXISTS (
SELECT 1
FROM alerts a
WHERE a.transaction_id = c.transaction_id
) AS already_alerted
FROM candidate_fraud c
)
SELECT
COUNT(*) AS known_fraud_matched,
COUNT(*) FILTER (
WHERE NOT already_alerted
) AS known_fraud_not_previously_alerted
FROM coverage;
If alerts are linked at the case level, use case_id instead. That’s often closer to the operational question: did another rule already bring this case to the team’s attention?
An account-and-time-window comparison is a workable fallback when neither identifier exists. It’s still a proxy. An unrelated alert on the same account can make the candidate look redundant when it wasn’t.
This is where a lot of proposed rules fall apart. They catch fraud, but almost every case was already surfaced by velocity, geography, device risk, or another existing signal.
A candidate that matches 60 known-fraud transactions sounds useful. If 57 of those transactions already generated an alert, the actual proposal is all of the candidate’s alert volume in exchange for three additional historical matches.
That may still be worthwhile. It’s a different decision from the one suggested by the original count.
When overlap is high, I look at the existing rule before adding another one. The better change may be adjusting its threshold or its place in the scoring model.
Query 4: sweep the threshold
The original proposal used $200. There’s nothing inherently correct about $200.
Remove the amount threshold from the replay and group the results into buckets:
WITH replay AS (
-- Query 1 with the amount threshold removed
),
bucketed AS (
SELECT
FLOOR(r.amount / 100) * 100 AS threshold,
COUNT(*) AS bucket_volume,
COUNT(*) FILTER (
WHERE o.outcome = 'fraud'
) AS bucket_known_fraud
FROM replay r
LEFT JOIN known_outcomes o
ON o.transaction_id = r.transaction_id
GROUP BY 1
),
sweep AS (
SELECT
threshold,
SUM(bucket_volume) OVER (
ORDER BY threshold DESC
) AS cumulative_volume,
SUM(bucket_known_fraud) OVER (
ORDER BY threshold DESC
) AS cumulative_known_fraud
FROM bucketed
)
SELECT
threshold,
cumulative_volume,
cumulative_known_fraud,
ROUND(
100.0 * cumulative_known_fraud
/ NULLIF(cumulative_volume, 0),
2
) AS cumulative_known_fraud_rate_pct
FROM sweep
ORDER BY threshold DESC;
Each row approximates the results of setting the rule at that threshold.
Moving the threshold downward should increase both volume and known-fraud matches. The useful question is how quickly each one increases. Moving from $300 to $200 might add 8,000 alerts and one known-fraud match. Moving from $400 to $300 might add 200 alerts and 20 matches.
That’s the threshold decision. Not which number looks clean in a meeting, but how much analyst work the next bucket buys.
There usually isn’t one mathematically perfect cutoff. The threshold has to fit the team’s capacity, the cost of the fraud being detected, and the confidence you have in the labels.
Then run it in shadow mode
Passing the backtest doesn’t mean the rule should begin creating analyst work immediately.
Historical SQL can’t reproduce everything that happens in production. Fields arrive late. Streaming logic behaves differently from a batch query. Device identifiers change. A rule that looks manageable against six months of data can behave very differently during a promotion or a fraud spike.
I start new rules in log-only mode. They evaluate production traffic and write would-be alerts to a table, but they create no analyst work.
Two weeks is usually enough to compare volume and catch obvious production-data problems. It may not be enough to measure quality. That depends on how quickly analyst dispositions, chargebacks, and other outcomes arrive.
Once enough outcomes have matured, run the same volume, outcome, and overlap checks against the shadow results. If they look materially different from the backtest, find out why before turning on the queue.
The habit
None of this is sophisticated. Four queries, then shadow mode.
The point is to make a proposed rule prove that it deserves analyst time before it gets any.
Find the problem in SQL, not in the analyst queue.
Next post: matching accounts that belong to the same person when that person would rather you didn’t. Entity resolution in plain SQL.
https://analytics.fixelsmith.com/posts/backtesting-fraud-rules/
r/dataanalysis • u/mgrusin • 2d ago
Does a CSV to dashboard tool exist?
Hey folks, I'm looking for a simple, general-purpose CSV to text-based (runs in a terminal) dashboard tool. I imagine something like this already exists (it always does!) but my searches all end up at other things.
My use case is we have microcontrollers on COM/TTY ports periodically outputting lines of CSV text with the values of a dozen or so sensors. While one can look at these scrolling by on a screen, it's not easy for humans to parse. I would like something that can ingest the CSV and render a very simple "dashboard" in the terminal. I'll write it if I have to but I can't imagine someone hasn't already done this.
Here's what I'm imagining:
- Takes in a line of CSV from a pipe, TTY port, >, etc.
- Consults a definitions file that tells it what to do with each CSV field (put it at an x,y location on the screen, type a prefix label and/or postfix units, color it red/green/etc. based on value, ideally could do some simple math transforms, etc.
- Uses ANSI or just line-by-line to output a "pretty" version of the data to the terminal, allowing humans to more easily interpret it.
- Linux seems like the most likely place to find this but I'm not picky.
I know there are much more comprehensive (and complex) systems out there like Grafana; but I feel like that's huge overkill when a simple text-based tool would work great.
Thanks for any ideas!
r/dataanalysis • u/helovesbreakfast • 2d ago
Project Feedback Feedback on Dashboard - Bike Equity
Hi all, I'd love some feedback on the Tableau dashboard for the largest of my first portfolio/learning projects before I begin my write-up. My methodology section will describe the calculations of 'bike score', 'need', and 'service rating' in depth, but I'm hoping the audience can still deduce conclusions without being presented all the technical details.
Unfortunately anonymity is lost in sharing a Tableau Public link, but users can select a community by clicking its representation on any chart. The first image shows the dash before selection, and the second and third show after.
One thing I have not been able to figure out is how to use a community parameter dropdown to highlight the respective community on the charts. Hence, you can only select a community via the map and charts. If anyone has advice on how to create a highlight action using a parameter, please let me know!
I used Python (geopandas, numpy, tobler) for data wrangling, cleaning, and interpolation. Data from ACS and People For Bikes' Bike Network Analysis.
Any and all feedback and questions are appreciated. Thanks!
r/dataanalysis • u/HoleInTheMirror • 2d ago
Comparing tables - Help?
Hello guys, I need your help! How to compare 2 tables on easiest way?
I have 2 tables from 2 different systems, but with the same columns. I need to check if every row from the first table exist in another, and if not to return the difference. Can you help me? Should I use excel formulas, python, something else?
r/dataanalysis • u/Zealousideal_Pie7079 • 2d ago
Project Feedback Please review this
github.comI have started a data analysis project after learning for few days. It is not complete but would appreciate your feedback
r/dataanalysis • u/Not_A_Doc74 • 3d ago
Data Tools I made a scaleable Rock Paper Scissors (up to RPS-15) solvers with my own modifiers!
My coworker and I (both not in the tech/data field) have gotten really into RPS recently. We started of by learning RPSLS, which we got down pretty quickly. Since we both have fun making spreadsheets, I made a sheet to keep track of our games. This kind of escalated into a full on game with custom modifiers and stats. I also added a dashboard to automatically generate new games. (https://limewire.com/d/iUICg#IiTgJh7IJe)

This has been going for about 1.5 months now, and I have spend a handful of late nights learning a lot of new Excel tricks to make the sheet more stable. I also made a way to "recap" the games (we call them Seasons) using VBA.

We are currently up to RPS-7, but we are planning to keep upping our games until at least RPS-25 in the future. This is also why i made the sheet scaleable. Since we didn't want to get bored of the game too soon, I decided to make modifiers. I came up with the following:
| Modifier name | Description |
|---|---|
| 🚫 No Modifier | No modifiers are added, just your "classic" game of Rock, Paper, Scissors (and whatever other stuff you want) |
| ♠️ Blackjack Gambit | If a round ends in a tie, it is declared a "Split Round." In this scenario, two games are played simultaneously, with one hand designated for each game. Scoring is adjusted as follows: winning both games earns 3 points, while winning only one game earns 1 point. If a tie occurs during a split round, it is disregarded and cannot be split further. When this modifier is active, the spreadsheet automatically calculates these scores. |
| 🎲 Let it Ride | Enables a player to double the stakes of a round before it begins. As a result, the outcome—whether a win or a loss—is worth 2 points instead of the standard 1. Check the box in the players column to up the stakes. |
| 🪞 Mirror Mirror | Receive a bonus point when winning the way you just lost! That’s it. If you lose rock vs paper, and then win with paper against rock immediately after, you get bonus points. |
I love learning new tools, so I'm currently working on a APK mobile app to keep track of the games more easily (obviously with export to excel function).
I'm curious to hear if you guys have any advise on how to make it run better!
r/dataanalysis • u/i_devour_kids • 3d ago
Localized scraping: How accurate is proxy country and city-level targeting in practice?
Hey everyone, I'm working on an ad-verification and localized pricing tool. I need to pull data from specific regions to ensure ads are rendering correctly for users in those exact locations. Geonode boasts coverage in over 190 countries, but I’ve been burned by budget providers before where an advertised "US" proxy turns out to route through an offshore datacenter.
**How reliable is their geo-targeting? Can I trust their targeting filters for sensitive geolocation verification?**
r/dataanalysis • u/JavacLMD • 3d ago
How are Slowly Changing Dimensions (Type 2) actually implemented in production?
I'm taking a Data Science Analysis course and have been learning about dimensional modeling and Slowly Changing Dimensions (SCDs). I've modeled databases quite a bit, but I haven't had much opportunity to design a data warehouse from scratch outside of personal projects, so I'm curious how this is handled in real production environments.
I understand the theory behind SCD Type 2—when a tracked attribute changes (like a customer's address), you create a new dimension record so historical facts continue to reference the correct version of that customer.
What I'm struggling with is the implementation.
For example, if a customer moves from New Hampshire to Texas, I still want to know that an order placed four years ago was made while they lived in New Hampshire. At the same time, I also want the business to know the customer's current address.
How is this typically modeled?
- Does each new dimension row reference the previous version (like a
PreviousCustomerKeyor similar)? - Is there usually an effective start/end date and a current flag, without linking versions together?
- Do some organizations keep a separate history/audit table instead?
- When fact tables need customer information, do they simply point to the correct version of the dimension row?
Another approach I'm familiar with is denormalizing historical data into the transaction itself. For example, when an order is placed, the shipping address is copied into the order record so it never changes regardless of updates to the customer profile. That seems useful even if you're using SCD Type 2.
For those of you working in analytics, BI, or data engineering, what's the most common approach you've seen? Are SCD Type 2 dimensions still the standard, or have other patterns become more common?
I'd appreciate hearing how this is done in real systems rather than just textbook examples.
r/dataanalysis • u/QuantumOdysseyGame • 5d ago
Decade-long project to completely gamify Quantum Computing
Hi
If you are remotely interested in Gate model framework Quantum Computing, oh boy this is for you. I am the Dev behind Quantum Odyssey (AMA! I love taking qs) - worked on it for about 10 years (3+ during PhD, the visual method I developed ended up being my thesis, it is a complete Hilbert space visualizer), the goal was to make a super immersive space for anyone to learn quantum computing through zachlike (open-ended) logic puzzles and compete on leaderboards and lots of community made content on finding the most optimal quantum algorithms. The game has a unique set of visuals capable to represent any sort of quantum dynamics for any number of qubits and this is pretty much what makes it now possible for anybody 12yo+ to actually learn quantum logic without having to worry at all about the mathematics behind.
This is a game super different than what you'd normally expect in a programming/ logic puzzle game, so try it with an open mind.
Stuff you'll play & learn a ton about
- Boolean Logic – bits, operators (NAND, OR, XOR, AND…), and classical arithmetic (adders). Learn how these can combine to build anything classical. You will learn to port these to a quantum computer.
- Quantum Logic – qubits, the math behind them (linear algebra, SU(2), complex numbers), all Turing-complete gates (beyond Clifford set), and make tensors to evolve systems. Freely combine or create your own gates to build anything you can imagine using polar or complex numbers.
- Quantum Phenomena – storing and retrieving information in the X, Y, Z bases; superposition (pure and mixed states), interference, entanglement, the no-cloning rule, reversibility, and how the measurement basis changes what you see.
- Core Quantum Tricks – phase kickback, amplitude amplification, storing information in phase and retrieving it through interference, build custom gates and tensors, and define any entanglement scenario. (Control logic is handled separately from other gates.)
- Famous Quantum Algorithms – explore Deutsch–Jozsa, Grover’s search, quantum Fourier transforms, Bernstein–Vazirani, and more.
- Build & See Quantum Algorithms in Action – instead of just writing/ reading equations, make & watch algorithms unfold step by step so they become clear, visual, and unforgettable. Quantum Odyssey is built to grow into a full universal quantum computing learning platform. If a universal quantum computer can do it, we aim to bring it into the game, so your quantum journey never ends.
Nice to watch:
Khan academy style tutorials in qm/qc: https://www.youtube.com/@MackAttackx
Physics teacher stream with 400hs in https://www.twitch.tv/beardhero
r/dataanalysis • u/unluckiestluckyguy • 5d ago
Data Question Claude for Data Merging and Cleaning
is claude for data cleaning and merging. How accurate is it? Because gpt makes mistakes in data. Asking for a MSc in Data Science and Economics
r/dataanalysis • u/thedifferent_ • 5d ago
Help
Best roadmap \ online courses for healthcare data analysis