r/datamining Jun 21 '26

Is there any way to datamine a DSIware game?

1 Upvotes

There's a piece of DSIware I have on my 3ds that I really enjoy, but there's some things I've never managed to do even though I've put probably over a couple hundred hours into it. Just wondering if anyone here knows how to get it into readable code


r/datamining Jun 09 '26

Call for Papers: ADMA 2026 Special Session

1 Upvotes

Call for Papers: ADMA 2026 Special Session on Data-Efficient Agentic Learning for Data Mining (DEAL-DM)

Hi everyone,
Disclosure: I am part of the organizing team for this special session.

We are inviting submissions to the ADMA 2026 Special Session on Data-Efficient Agentic Learning for Data Mining (DEAL-DM). ADMA 2026 will be held in Hong Kong, China, on November 13–15, 2026.

The session focuses on building effective and reliable LLM agents and agentic data mining systems under limited data, supervision, feedback, and real-world interaction budgets.

Topics include, but are not limited to:

  • Agentic data mining and autonomous data analysis
  • Experience augmentation and simulated interaction
  • Memory, retrieval, knowledge reuse, and external knowledge integration
  • Planning, tool use, verification, and reflection
  • In-context and test-time adaptation
  • Parameter-efficient fine-tuning and budget-efficient reinforcement learning
  • Personalization and recommender systems
  • Agentic systems for scientific discovery, healthcare, and decision support
  • Trustworthy evaluation, robustness, safety, and privacy

Submission deadline: June 26, 2026 (AoE)
Paper length: Up to 15 pages
Review process: Double-blind
Proceedings: Springer LNCS

When submitting through CMT, please select “Special Session Track” and choose the subject area “Data-Efficient Agentic Learning for Data Mining (DEAL-DM)”.

More information:
https://wangyaqing.github.io/adma26-deal-dm/

We welcome original research papers, case studies, and technical reports. Happy to answer any questions in the comments.


r/datamining Jun 08 '26

I'm from India — building a complete broadcast archive of UT

1 Upvotes

I'm a content creator from India, and I'm trying to do something that sounds simple but is actually very hard: build a complete, community-verified archive of every movie that ever aired on UTV Action, an Indian TV channel that shut down in 2023 without any official broadcast records.

The mission:

Reconstruct the complete broadcast history of UTV Action from launch (January 2010) to shutdown/rebranding (March 2023). That's 13 years of daily movie broadcasts.

Why this is impossible right now:

- No official broadcast logs exist publicly

- News archives from 2010 are not digitized

- The Wayback Machine has very few snapshots of Indian TV guide sites

- The channel was rebranded into Star Gold Thrills in 2023 — the old identity is being erased

What I need:

I'm reaching out to international OSINT researchers, digital archivists, and data hoarders who specialize in:

- Wayback Machine / CDX API queries

- Newspaper archive searches

- Forum scraping from defunct websites

- Digital preservation techniques

- Crowdsourcing memory-based data

I already have a community of Indian viewers who want to help — they remember specific movies, scenes, and Hindi dubs. But I need technical help to cross-reference, verify, and build a permanent archive.

What this project is:

This is a non-commercial, educational preservation project. The goal is to create a public database that will never be deleted, for anyone who wants to remember this part of Indian TV history.

Even a single movie title, a screenshot of an old TV guide, or a clip from UTV Action can help fill a gap.

If you have experience in OSINT, digital preservation, or just want to help preserve television history from another country, I'd love your input.

This is a call for help from India to the global archival community.


r/datamining Jun 08 '26

Where is the Visual Novel script file for Quartero Event?

1 Upvotes

Hi everyone,

I am making a lore video for the Quartero Seasonal Event in Dota 2. Using Source 2 Viewer, I extracted all voice lines, character portraits, and the localization text lines.

However, I cannot find the file that links them together. Quartero has many different expressions, and I want to know exactly which portrait/expression triggers with each dialogue line to make my video accurate.

Could anyone point me to the exact file path inside pak01_dir.vpk where the visual novel sequence or expression triggers are stored?

Thanks in advance!


r/datamining Jun 06 '26

Aes keys for Fmodel itsn't working

1 Upvotes

I'm relatively new to fmodel and I'm trying to use it to view some pak files for a game but it doesn't work. There's a lot of tutorials for using fmodel but majority of them are for fortnite or valorant and those seem a bit different. I suspect it's something to do with the aes keys. There's a bunch of reddit and github posts on the aes keys for the game but they don't seem to work so I tried extracting the keys on my own. I did manage to get the keys but for some reason those don't work either. So atp I'm thinking maybe its because the aes dumpster that takes the aes keys from the win64 shipping is outdated but the latest doesn't have the right attachments. It could also be that I'm not selecting the correct directory for fmodel but I doubt it. Does anyone know what could be the issue? I'm genuinely so lost. I can post all the links if anyone needs them to check.


r/datamining Mar 24 '26

idea need feedback: data CLI for data exploration

2 Upvotes

I want to build a Data CLI focused on exploration.

My job has me jumping between postgres, bigbuery, and random json files daily. Exploring data across all of them is already messy. When I started using Claude Code and Gemini CLI, it got worse. The agent needed to explore a dataset, I was either copy-pasting schema manually or leaking credentials I'd rather keep private.

So, I want to build a Data CLI focused on exploration. Define your sources once, then run data query or data schema against any of them from the terminal. Your agent explores freely, credentials stay on your machine.

Would love feedback on the idea before I build further.


r/datamining Mar 14 '26

I have no freaking idea about datamining

1 Upvotes

So What's the most UNNECESSARYLY COMPLEX thing that a Company made against data miners?


r/datamining Feb 25 '26

How to access the UCR time series data set? Can anyone help?

1 Upvotes

I'm not able to find the password of the ucr time series classification data set. Currently working on imbalanced time series classification for my PhD and I need to access it for practicals and i am not able to locate or find the password. Urgent help is needed. Thanks!


r/datamining Feb 25 '26

Would automated web data structuring be useful in your workf

3 Upvotes

I’m working on a system that automatically extracts statistical data from public web pages and converts it into clean, structured JSON.

The core idea isn’t basic scraping — it’s transforming messy, human-readable web content into normalized, machine-ready datasets that can be cached and reused by downstream systems.

The pipeline looks like this:

  • Search public sources
  • Extract statistical tables / metrics
  • Structure everything into consistent JSON
  • Cache results
  • Automatically visualize the structured JSON into charts

So the output becomes both reusable structured data and instant visual analytics.

From a data workflow perspective:

Would automated structuring of public web statistics (with instant visualization) be useful in practice, or do most teams prefer sticking to official APIs and curated datasets?

Trying to understand whether this solves a real pain point or if it overlaps too much with existing data tools.


r/datamining Jan 07 '26

Looking for datasets on the anomaly of satellite on orbit.

2 Upvotes

I am from the background of computer science. And Our team are trying to apply the LLM agents on the automatic analysis and root-cause detection of anomaly of satellite on orbit.

I am dying for some public datasets to start with. Like, some public operation logs to tackle specific anomaly by stuffs at nasa or somewhere else, as an important empirical study materials for large language models.

Greatly appreciate anyone who could share some link below!


r/datamining Dec 27 '25

Applying Data Mining Techniques in RAG Systems

1 Upvotes

I am currently working on a university project which deals with RAG systems in which we are required to apply traditional data mining techniques in order to improve the quality of the retrieved chunks, our initial idea was to apply clustering to the chunks after embedding using the cosine similarity, but we found out that this approach has some negative affects, does anyone know effective data mining approaches that could really come in handy in the pipeline?


r/datamining Nov 24 '25

Do Residential Proxy IP Ranges Perform Better on Linkedin?

1 Upvotes

I’ve been testing residential proxies on LinkedIn for lead generation. Have you noticed that certain IP ranges perform better, or is it more about rotation frequency?


r/datamining Nov 15 '25

How to pull gis table as csv? Table provided

1 Upvotes

Hello. I'm working with an open government dataset:

https://www.arcgis.com/apps/mapviewer/index.html?webmap=d34f3091e0384dbfa98b8b503eb55967

Years ago I'd pulled this whole dataset down successfully - I believe there was just a download button. It may still exist, but I haven't found it. But I CAN still open the full table 15000x10.

Layers (at top left) --> TxDOT Commercial Signs --> ••• --> Show Table.

How can I pull this down?

And while I appreciate if someone succeeds and uploads the csv, I'm interested in how to do this regularly since the data gets updated regularly.

Thanks


r/datamining Nov 06 '25

Any projects trying to parse congress financial disclosures?

3 Upvotes

OpenSource stopped parsing non-stock, non-insider related financial data in 2018. This data is still legally required to be posted, but is being stored in scans of PDFs and static HTML code. It would be very difficult to build and maintain a dataset by myself without some kind of advanced OCR model or going and reading each disclosure one by one.

Is anyone trying to do this? Would it be easier to lobby for machine-readable disclosures instead?


r/datamining Nov 05 '25

Idea for new data mining center design

3 Upvotes

Hi everyone. Not sure if this exactly the right spot for this but I will let the mods figure it out. I have a design for a waste to energy facility that can produce enough energy to run itself plus produce surplus energy to facilitate operations in data mining. The plant I am working with handles up to 70 tons of waste a day. If you set up a few of these say in or near a major landfill site or any other place where there is sufficient waste you could easily power and cool major server banks. All completely off grid while actually removing waste from the local environment and atmosphere. I have the design, the roi, the industry contacts to build the complete base wte system and get it up and running. It isnt super complicated just a different process. Data mining is just one configuration. I thought maybe someone here in the industry might be interested or someone might know who to contact. Ive heard of major plants being built on grid. This is an opprtunity to function fully with very stable power output without draining grid resources. Thanks if you took the time to read this. I look forward to hearing your thoughts and opinions.


r/datamining Oct 24 '25

What tools do you use these days when writing web scrapers?

13 Upvotes

Given how much coding assistants like Cursor/Claude Code/Codex can do, I'm curious how useful they've been to folks that are into web scraping. How are you using them? Where do they fall short for this type of code?


r/datamining Aug 31 '25

Data mining project idea ?

5 Upvotes

I have data mining course in my uni and i have to do a academic project on it, I want to build a proper data mining project which should be deployable and publishable, but I can't seem to get any idea which interests me that much,pls share some unique and interesting data mining projects, so i can take some inspiration from it.
Also I can only use an algorithm from what is mentioned in my syllabus which is:

  1. Basic concepts of clustering, measure of similarity, types of clusters and clustering methods, K means algorithm, measures for cluster validation, determine optimal number of clusters.
  2. Transaction data-set, frequent itemset, support measure, rule generation, confidence of association rule, Apriori algorithm, Apriori principle
  3. Naive Bayes classifier, Nearest Neighbour classifier, decision tree, overfitting, confusion matrix, evaluation metrics and model evaluation.

r/datamining Aug 01 '25

Need info on web scraping proxies. What's your setup on data mining?

10 Upvotes

I’ve been knee-deep in a data mining project lately, pulling data from all sorts of websites for some market research. One thing I’ve learned the hard way is that a solid proxy setup is a real shift when you’re scraping at scale.

I’ve been checking out this option to buy proxies, and it seems like there’s a ton of providers out there offering residential IPs, datacenter proxies, or even mobile ones. Some, like Infatica, seem to have a pretty legit setup with millions of IPs across different countries, which is clutch for avoiding blocks and grabbing geo-specific data. They also talk big about zero CAPTCHAs and high success rates, which sounds dope, but I’m wondering how it holds up in real-world projects.

What’s your proxy setup like for those grinding on web scraping? Are you rolling with residential proxies, datacenter ones, or something else? How do you pick a provider that doesn’t tank your budget but still gets the job done?


r/datamining Jul 29 '25

Website-Crawler: Extract data from websites in LLM ready JSON or CSV format. Crawl or Scrape entire website with Website Crawler

Thumbnail github.com
4 Upvotes

r/datamining Jun 30 '25

US government data has been backed-up, large projects and public archives that serve as alternatives to federal data sources, and subscription-based library databases. Visit these sources in the event that federal data becomes unavailable.

Thumbnail libguides.brown.edu
6 Upvotes

r/datamining Jun 23 '25

Association mining (confidence) - Why are these answers correct?

1 Upvotes

Trying to understand why these should be correct? Isn't H missing on the RHS for all? Else we shouldn't be able to conclude whether the confidence is lower?


r/datamining Jun 17 '25

Help decompiling STRIDE (for the meta quest 2)

1 Upvotes

https://drive.google.com/file/d/1vJvYiB0CPoO6NoDfC8SJhSe_9go-trWB/view?usp=drivesdk

This is as far as I could get- I don't know what to do about anything in the paks folder. I'm trying to put them all into folders sorted by apk and obb, in order to allow for modding


r/datamining May 16 '25

Where to find vin decoded data to use for a dataset?

1 Upvotes

Currently building out a dataset full of vin numbers and their decoded information(Make,Model,Engine Specs, Transmission Details, etc.). What I have so far is the information form NHTSA Api, which works well, but looking if there is even more available data out there. Does anyone have a dataset or any source for this type of information that can be used to expand the dataset?


r/datamining May 02 '25

Am i confused or is there inconsistency in the dataset

2 Upvotes

I feel like the numbers here dont add up, am i understanding the concept wrong or is this dataset faulty, my problem lies in the fact the there is less packets in a second than a nanosecond even though a nanosecond i s much smaller


r/datamining Apr 15 '25

Perform mindful data analysis using Python, NumPy and AI.

3 Upvotes

Hey folks, I’ve noticed a common pattern with beginner data scientists: they often ask LLMs super broad questions like “How do I analyze my data?” or “Which ML model should I use?”

The problem is — the right steps depend entirely on your actual dataset. Things like missing values, dimensionality, and data types matter a lot. For example, you'll often see ChatGPT suggest "remove NaNs" — but that’s only relevant if your data actually has NaNs. And let’s be honest, most of us don’t even read the code it spits out, let alone check if it’s correct.

So, I built NumpyAI — a tool that lets you talk to NumPy arrays in plain English. It keeps track of your data’s metadata, gives tested outputs, and outlines the steps for analysis based on your actual dataset. No more generic advice — just tailored, transparent help.

🔧 Features:

Natural Language to NumPy: Converts plain English instructions into working NumPy code

Validation & Safety: Automatically tests and verifies the code before running it

Transparent Execution: Logs everything and checks for accuracy

Smart Diagnosis: Suggests exact steps for your dataset’s analysis journey

Give it a try and let me know what you think!

👉 GitHub: aadya940/numpyai. 📓 Demo Notebook (Iris dataset).