I was looking through a public dataset yesterday and realized something.

Some images felt like they could reasonably have two different labels depending on who's annotating them.

Do companies just write really detailed guidelines, or is there another process for keeping everyone consistent?

I'm curious how this works in practice because it seems like even small inconsistencies could affect the model later.

I'm the founder of StockFit API - SEC sourced clean fundamentals for all US Companies (delisted or not) - all Point-In-Time data perfect for back testing.

One of my subscribers has pushed me to investigate what it would take to widen the coverage to the pre-XBRL time.

If you don't know what that means:
In 2009, XBRL was mandated my the SEC, which from that time on provides a structured data format to extract the actual fundamentals a company reported.

Before that time, there where 2 distinct data format time frames that you will have to prepare for if you want that data too:
1. Pre 2001: FDS (Financial Data Schedule) - a semi-structured way data was reported, similar to XML
2. 2001 - 2009: No structure at all - a full mix of HTML/ASCII text/tables

FDS is kinda ok to parse, but comes with many exceptions across filers. You can easily get to a coverage of 90%+

The nightmare starts in 2001. There really is no structural mandate whatsoever that you can rely on! Everything has to be tested empirically and your parser needs to be able to handle everything, validate as much as it can, and provide an observation layer for you to ensure data integrity.

This is probably the most challenging piece in my entire stack to get right. But in the end, I would be able to claim that I serve 30+ years of historical fundamentals, and that is absolutely worth it.

This effort is NOT finished yet. Until now, I'm serving 20+ years of fundamentals that are of very high quality/accuracy. Getting that next batch to that same bar is something I'm trying to get to now.

But I'll absolutely take one step at a time to get there. Otherwise this will get ugly very quickly. And there is nothing worse than serving wrong data.

I'm curious if anyone has done this pre-XBRL parsing before? Any lessons you can share after having gone through that?

And from the consumer side: How much do you care in your use case about fundamentals that are pre-2009?

I’m building a creator marketplace and want to build the largest possible database of creators in my niche.

I don’t need influencer analytics—just a list of creators who have posted under specific hashtags.
What’s the cheapest way to collect 100k+ creator

profiles?
Custom scraper?
Apify?
Bright Data?
Other tools?
Hire a freelancer?

Has anyone built something like this before? What did it cost?

There's a popular Duke course "Managing-Big-Data-with-MySQL" that used to use a big MySQL database called "Dillar's database" hosted on Teradata.

You can see a few sample exercises here: https://github.com/hongwai1920/Managing-Big-Data-with-MySQL/blob/master/Week%203/Week-3-Teradata-Practice-Exercises.pdf

I've been looking everywhere for this dataset (as it looks fairly sizable) and I can't find it anywhere.

I've purchased the Specialization on Coursera, and they have switched to a different dataset because their partnership with Teradata finished.

Anyways, if anybody knows where I could find it I'd really appreciate it. Thanks!

I've been building an on-device assistant and, like anyone doing tool-calling SFT, kept shipping subtle junk into my dataset, hallucinated tool names, invented arguments, a tool-call with no result, the model over-calling a calculator for "2+2", near-duplicate examples quietly collapsing my diversity. So I wrote a validator, and I figured it might save someone else the same headaches.

It runs entirely in your browser — you drop in your .jsonl and your tool catalog, and nothing is uploaded anywhere. Your dataset never leaves your machine (that mattered to me, and I assume to some of you too).

What it checks, per example, with a typed verdict (keep / KTO-negative / discard):

- structure & roles, and every tool_call name/args against your catalog (required present, no invented args, types)

- tool-flow (each call gets its result, no orphans, ends with a real answer)
- over-calling, verbosity/filler, prompt-injection defense
- lexical and semantic dedup (the embedding model runs locally too, via WASM)
- diversity (distinct n-grams) + a difficulty breakdown

It spits out clean.jsonl / kto-negatives.jsonl / rejected.jsonl so you can just use the output.
It's free, no account, no catch. Link: nothumanallowed.com/tools/dataset-validator
It's early and opinionated (the heuristics come from my own pipeline), so if it flags something it shouldn't, or misses something it should, I'd genuinely like to hear it — happy to adjust. Hope it's useful to someone.

Disclosure up front: this is my project, so I'm tagging it as self-promotion per the subreddit rules.

A few weeks ago I was trying to build a historical closing line value dataset and realized I couldn't find one that combined de-vigged fair probabilities, closing values, and game outcomes across multiple sports. Everything I found was either limited to one sport, only included raw odds, or couldn't be redistributed because of licensing.

So I started building my own derived dataset.

Each row contains things like:

• Sport
• Market
• Selection
• Event time
• Opening fair probability
• Closing fair probability
• De-vig method
• Whether the fair line came from a sharp reference (Pinnacle) or a consensus fallback
• Edge at entry
• Closing line value (CLV)
• Beat close (yes/no)
• Final result

A couple notes:

• This does not include raw sportsbook odds or sportsbook names since my data provider doesn't allow redistributing that data.
• The dataset only contains derived metrics like fair probabilities and CLV.
• "Beat the close" is meant as a research metric, not proof that a bet was good or profitable.

One thing I think is useful is the anchor field. You can separate observations that were generated from a sharp market reference from ones that used a consensus fallback instead of treating them as the same thing.

I'm planning to keep expanding this over time as more sports and markets are added.

Project: https://edgedesksports.com

I'd really appreciate feedback from people who work with betting or forecasting datasets.

• Are there any columns you'd want added?
• Is there another derived metric that would make this more useful for research?
• Has anyone found a comparable open dataset that covers multiple sports?

Produced by https://www.landrecords.us by sourcing directly from State and County governments, and enriched with public data from USGS, BLM, and EPA.

LifeDive.org

Disclosure: I built DietlyAPI, one of the APIs compared below. I used the providers’ public documentation and included the calculations so people can challenge them. This is not a speed or nutrition-accuracy benchmark.

I compared the closest common operation: searching for a food or looking up a product/barcode. Recipe search, image recognition, natural-language parsing, and other premium operations are not directly equivalent.

Prices and limits checked in July 2026.

Open Food Facts

Price: Free

Limit: 10 search requests/minute/IP and 15 product reads/minute/IP

Monthly allocation: No fixed monthly quota published

Best when you need open, downloadable worldwide product data. The hosted search API is not intended for search-as-you-type at scale. Open Food Facts recommends using its exports or running your own backend for high-volume applications.

USDA FoodData Central

Price: Free

Limit: 1,000 requests/hour/IP

Equivalent sustained average: 16.7 requests/minute

Authentication: Free data.gov API key required

Best for authoritative US reference data. The data is public domain under CC0 and can also be downloaded. Its search behavior and geographic focus differ from a worldwide consumer-product autocomplete API.

FatSecret Basic

Price: Free

Included quota: 5,000 calls/day

Equivalent sustained average: 3.47 requests/minute

Equivalent monthly capacity: 150,000 calls over 30 days

FatSecret also offers Premier Free to qualifying startups, nonprofits, and students. That tier advertises unlimited calls but requires verification, attribution, and is limited to US data. Paid international access is quote-based.

Edamam Basic

Price: $14/month

Peak limit: 50 food and nutrition requests/minute

Included quota: 100,000 calls/month

Equivalent sustained average: 2.31 requests/minute

Edamam Core costs $69/month for 750,000 calls and 100 requests/minute. Plus costs $299/month for 5 million calls and 300 requests/minute.

Edamam is particularly strong when natural-language parsing, serving measures, diet filters, allergy filters, and image recognition matter. Its caching and attribution requirements should be reviewed carefully.

Spoonacular Cook

Price: $29/month

Peak limit: 5 requests/second, or 300/minute

Included quota: 1,500 points/day

Optimistic monthly ceiling: 45,000 one-point calls over 30 days

Equivalent sustained average: at most 1.04 one-point calls/minute

Spoonacular uses points rather than ordinary requests. A call usually costs one point plus an amount based on the number of results, while some endpoints cost more. It is strongest for recipe and ingredient workflows, but its advertised requests-per-second limit should not be confused with the included daily capacity.

Nutritionix

Price: Contact/quote

Published numeric limit: I could not find a current public figure

Published quota: I could not find a current public figure

Nutritionix focuses on natural-language food logging, instant search, barcode lookup, nutrients, and exercise parsing. Its documentation currently describes a database of more than 600,000 foods. I would request a current quote instead of relying on old third-party pricing.

DietlyAPI Pro

Price: €27/month

Peak limit: 500 requests/minute/account

Monthly call allocation: None

Theoretical 30-day ceiling at the RPM limit: 21.6 million calls

Approximate subscription price per theoretical million calls: €1.25

DietlyAPI Scale

Price: €92/month

Peak limit: 3,000 requests/minute/account

Monthly call allocation: None

Theoretical 30-day ceiling at the RPM limit: 129.6 million calls

Approximate subscription price per theoretical million calls: €0.71

Dietly also allows anonymous non-commercial reads at 30 requests/minute/IP with a small fairness delay.

The Dietly numbers are theoretical RPM ceilings, not an SLA or a promise that an application should run continuously at the limit. They show that the plans are rate-capped instead of monthly-credit-capped.

Dietly currently focuses on fuzzy and confidence-aware food search, stable food IDs, barcode lookup, and structured nutrition across more than 4.2 million indexed foods. Its catalog is primarily derived from Open Food Facts and retains the relevant attribution and share-alike obligations.

It does not currently provide USDA data, recipe discovery, food-image recognition, or Edamam-style natural-language parsing.

My conclusion

There is no honest universal winner:

- Open Food Facts is best when open and downloadable worldwide product data matters most.

- USDA is best for authoritative US reference data and permissive reuse.

- FatSecret is compelling for eligible startups or businesses needing verified country-specific datasets.

- Edamam is strong for NLP, measures, and diet/allergy features.

- Spoonacular is strong for recipes and ingredient workflows.

- Nutritionix is worth evaluating when natural-language food logging is central, but you need a current quote.

- DietlyAPI has the strongest published price-to-throughput ratio in this comparison for straightforward food search and barcode lookup.

The biggest lesson was that peak requests per minute can be misleading. A service may advertise hundreds of requests per minute while its daily or monthly quota only supports one or two requests per minute when averaged over the complete billing period.

Disclosure first: I'm one of the maintainers of PortalJS, an open source framework for data portals, so this is my own project.

The problem we kept hitting: you have a folder of CSVs you want to publish properly. "Properly" increasingly means machine readable metadata, DCAT feeds so data.gov or data.europa.eu can harvest you, and lately Croissant so ML tooling can load your datasets. The usual answer is "run CKAN", which is a lot of infrastructure if all you have is fifty files that update quarterly.

Our approach is a static site that does the metadata work at build time. You describe each dataset once (it's a Frictionless Data Package under the hood) and the build emits DCAT catalog feeds in JSON-LD, Turtle and RDF/XML, with application profiles for DCAT 2/3, DCAT-AP, DCAT-US, GeoDCAT-AP and Croissant, plus autodiscovery links and SHACL conformance checks against the EU ITB validator where the profile requires it. Static hosting, no runtime, harvesters just fetch the feeds.

If you use Claude Code there are skills that do the assembly:

/portaljs-new-portal "Auckland Council open data portal"
/portaljs-add-dataset ./air-quality.csv
/portaljs-add-dcat profiles=dcat-ap

But it's plain Next.js you can edit by hand, the skills are a convenience. Repo: https://github.com/datopian/portaljs

What I'd like to hear from people who publish datasets: does anyone actually see harvest or referral traffic that you can attribute to DCAT or Croissant metadata? I have a decent theory of why it matters (national portal harvesting, dataset search, agents reading catalogs) but I'm curious what it looks like in practice versus a clean CSV and a good README.

I published a free sample of 5,000 food records for anyone working on nutrition search, calorie tracking, barcode matching, data cleaning, or food-related ML experiments.

Dataset:

https://www.kaggle.com/datasets/dietly/dietly-food-sample-5000-foods-calories-and-macros

Each row can include:

• - food name and brand
• - barcode
• - category
• - serving size
• - calories
• - protein, fat and carbohydrates
• - fiber and sugar
• - sodium, saturated fat and cholesterol
• - potassium
• - source and confidence metadata

The data is primarily derived from Open Food Facts and is provided with its ODbL provenance and attribution requirements.

Important limitations:

• - nutrition fields can be missing
• - community-contributed labels can be incorrect or outdated
• - coverage varies by product and country
• - this is a 5,000-row sample, not the complete catalog
• - it should not be treated as medical or laboratory-verified data

Disclosure: I created this sample while building DietlyAPI, a hosted search and barcode API over a much larger indexed catalog. The downloadable sample itself is available without an API signup.

I’d especially appreciate feedback about the schema: which additional fields or export format would make the dataset more useful?

GitHub link: here

Licensed under MIT.

Includes:

• States and Union Territories
• Districts
• Sub-districts (Tehsils/Taluks)
• Latitude/Longitude
• Bounding boxes (where available)

Sources:

• iGOD (India Portal) for administrative divisions
• OpenStreetMap Photon for geocoding

The repository also includes scripts to regenerate the dataset. The script can also be incremented to get the block-level data.

Create structured data datasets, infinite possibilities (companies, people, places, products, markets, regulations, the obscure and the everyday): You can define the topic/theme of the desired data, define the output fields, generate a sample record with test lookup values, and then when ready generate an API to it that we host and that you can integrate to or call from anywhere, as well as append the defined dataset to another file [self-promotion]: https://custom-data-wizard.interzoid.com/

Hi everyone! I recently published an open flight dataset on Zenodo containing simulated Antonov An-32 flight data focused on accelerated stall events. The dataset was created as part of my undergraduate research and is intended for machine learning, anomaly detection, time-series forecasting, and flight dynamics research.

It includes:

• Flight state variables and control inputs.
• Normal and accelerated stall scenarios.
• Time-series data suitable for deep learning models.
• DOI and open access via Zenodo.

I'd be happy to answer any questions or receive feedback!

https://zenodo.org/records/20359823

Looking to connect with teams and individuals collecting egocentric data across North America, LATAM, Asia, and Europe.

If you've got data or are mid-collection, let's talk.

Hi community,

I recently published a research-based synthetic Chronic Kidney Disease (CKD) dataset on Kaggle after spending several weeks studying clinical guidelines and epidemiological literature.

The motivation came from a common challenge I encountered: many publicly available CKD datasets contain only a few hundred patient records and a limited number of clinical variables, making them less suitable for building and evaluating modern machine learning models.

Dataset Highlights

• 200,000 synthetic patient records

• 82 clinically meaningful features

• Research-informed design using published clinical guidelines and epidemiological evidence

• Covers demographics, lifestyle, medical history, vital signs, kidney biomarkers, medications, frailty, healthcare utilization, and clinical outcomes

• Includes CKD stage, kidney failure risk, dialysis requirement, and hospitalization risk

• Machine learning and healthcare analytics ready

The dataset is completely synthetic and contains no real patient information. It was created for educational purposes, machine learning experiments, healthcare analytics, and research.

I'd really appreciate feedback from the community.

Some questions I'd love your thoughts on:

• Are there any important CKD-related variables you think are missing?

• What types of ML or analytics projects would you build with this dataset?

• What would you improve in a future version?

Kaggle Dataset:

https://www.kaggle.com/datasets/mohankrishnathalla/chronic-kidney-disease-risk-dataset-2026

Thanks for taking the time to check it out. I'm happy to answer questions about the design process or discuss future improvements.

Hi everyone,

As part of my PhD thesis focusing on the integration of BIM, AI, and IoT for predictive HVAC systems management, I am looking to find a dataset specific to the Moroccan context.

I am searching for historical time-series data that ideally includes:

Building energy consumption (preferably commercial, office, or institutional buildings)

Associated meteorological/weather data

HVAC parameters (indoor temperatures, setpoints, flow rates, fan/pump speeds, etc.)

If you know of any open-source Moroccan repositories (like past projects from IRESEN, AMEE, or local universities) or if you could point me toward professionals, researchers, or facilities management teams who might share anonymized data, I would be incredibly grateful.

Thank you in advance for your help!

Hi everyone,

As part of my PhD thesis focusing on the integration of BIM, AI, and IoT for predictive HVAC systems management, I am looking to find a dataset specific to the Moroccan context.

I am searching for historical time-series data that ideally includes:

Building energy consumption (preferably commercial, office, or institutional buildings)

Associated meteorological/weather data

HVAC parameters (indoor temperatures, setpoints, flow rates, fan/pump speeds, etc.)

If you know of any open-source Moroccan repositories (like past projects from IRESEN, AMEE, or local universities) or if you could point me toward professionals, researchers, or facilities management teams who might share anonymized data, I would be incredibly grateful.

Thank you in advance for your help!

Hello, here's a little python script I made to download data from Reddit community dumps. I isolated it from a larger project, it fetches data for the subreddits you input, output format CSV or JSON (with the actual threads).
https://github.com/Tryhard-cs/reddit-download-tool

Note: It will download all pre-2026 data (for the subreddits) if you include a time interval that has pre2026 data (downloads the data from academic torrent archives)

For post2026 data it will only download that interval, but download speed is very low since it's community APIs (which I am very grateful for and that should be the case to prevent abuse)

Sources: https://academictorrents.com/details/3e3f64dee22dc304cdd2546254ca1f8e8ae542b4 (made by u/Watchful1 I believe)
https://github.com/ArthurHeitmann/arctic_shift

I remember testing the data coverage manually by comparing to the Reddit website and being satisfied, but my main purpose was more for analytics / research and tracking evolution of stuff overtime / by category and not for a precise total amount. I can't do any guarantees honestly.

I may add more categories / PRAW support in the future.

PS: Looking for someone to test it and tell me if it works for them / if the install instructions are clear enough, if this is helpful for you and you tried it and you encountered a problem send me a message I'll fix it right away.

I don't understand scraping infrastructure.

I can make 10 fake YouTube accounts and try to scrape Koala 36M but it's not possible. It takes like 100-1000VMs to actually do this scraping in time

Large companies don't publish anything. They have 10s of millions scale videos and don't even put of 10M.

Does anyone have any advice on this? Im training video models and world models.

The project is primarily intended to help the police in my city monitor situations that could potentially lead to harm to individuals or the public. It would analyze public posts and public comments to identify content that may indicate emerging threats or dangerous situations. In a way, it can be considered a sentiment analysis system designed for law enforcement.

However, the project requires a live data feed. Reddit has effectively stopped providing the level of API access needed for this use case, Meta's APIs (Facebook and Instagram) are too restrictive to be useful, and Twitter (X) API access is paid. At the moment, the only options I can think of are web scraping or using downloaded datasets.

The problem with downloaded datasets is that the project is specific to a single city Surat, India so historical datasets may not contain the localized, real-time information needed for effective monitoring.