I've been working on a BigQuery FinOps suite that I've open-sourced and wanted to share it with the community (it is totally free)

If you run BigQuery at scale , you've probably dealt with some of these:

• You're on Editions but have no idea if your baseline is right-sized, or if you're paying autoscaler premiums for nothing
• Logical vs. physical storage billing — which datasets should switch, and what's the actual delta?
• Cost attribution is broken because GCP dumps idle reservation costs into the admin project
• "Cooldown tax" — 60-second billing minimums silently inflating your bill on high-frequency short queries
• No easy way to see if HBO (History-Based Optimization) is actually saving you anything

What the tool does: It connects to your org-level INFORMATION_SCHEMA views and runs diagnostic queries across your entire GCP Organization. The output is an interactive dashboard with actionable findings — DDL you can copy-paste, capacity simulation matrices, cost attribution breakdowns, etc.

Modules

Key design decisions

• No AI hype: 17 out of 18 modules are pure SQL + Python analytics. The only GenAI module (AI Doctor) is clearly labeled and optional.
• No infrastructure: It's a FastAPI app that uses your ADC credentials. No Terraform, no service accounts, no pub/sub. pip install and run.
• Copy-paste DDL: Storage changes, fluid scaling configs, reservation adjustments — all generate ready-to-execute DDL statements.
• Org-wide scope: Queries JOBS_BY_ORGANIZATION, TABLE_STORAGE_BY_ORGANIZATION, etc. — not just single-project views.
• Focus Projects: Optional scoping to specific project IDs when you don't want to analyze the whole org.
• Security: SQL injection prevention via _safe_ident(), Pydantic validation on every parameter, XSS escaping on all frontend rendering.

Give it a try

• 🔗 GitHub: github.com/mbettan/bq-finops-optimizer
• 🎮 Live Simulator (no GCP needed): Simulator Demo
• 📄 Website: mbettan.github.io/bq-finops-optimizer

Quick start

git clone https://github.com/mbettan/bq-finops-optimizer.git
cd bq-finops-optimizer
pip install -r requirements.txt
AUTH_ENFORCED_UPSTREAM=true uvicorn src.main:app --reload --port 8080

Somehow it disappeared and now I struggle to control the size of the pannel.

The public BQ datasets (bigquery-public-data.crypto_bitcoin, the Stack Overflow one) are great, but they ship as a pile of tables with no diagram and no enforced FKs — you end up holding the joins in your head. I drew them as ERDs on a free canvas so the UTXO flow / posts-graph is actually navigable, and exported the model as Google's new OKF (portable markdown you can keep in git or hand to an LLM). Screenshots + how it was made in the comments.

What other public BQ datasets would be useful to have as a ready-made ERD?

About Me

Senior Data Engineer with 5+ years of experience in Data Engineering, Analytics Engineering, and Applied AI. Based in Bangalore, India. Available for remote work globally.

Rate: $25 - $50/hr depending on project scope and complexity.

Tech Stack & Expertise

Python, SQL, Spark, Databricks, Airflow

BigQuery (data modeling, optimization, cost management)

AWS & Cloud Data Platforms (S3, Glue, Redshift, EMR)

Snowflake, Redshift

ETL/ELT Design & Orchestration

dbt for analytics engineering

FastAPI, REST APIs

LLMs, RAG, AI Agents

Data Quality & Testing Frameworks

What I Can Help With

Build and optimize BigQuery pipelines and data models

Design scalable ETL/ELT architectures on GCP/AWS

Develop backend APIs and automation solutions (FastAPI + Python)

Build AI applications using LLMs, RAG, and agent-based workflows

Support and optimize existing data platforms

Availability

Open to freelance projects, contract work, and part-time engagements.

Available immediately.

Feel free to DM me with project details or questions.

Hey everyone,

I’m currently digging into stream data processing architectures and trying to decide between Databricks (Spark Structured Streaming) and Google BigQuery (Continuous Queries).

While there are a ton of threads comparing these two from a pure data engineering infrastructure standpoint, I want to look at this specifically from a Data Scientist / ML Engineer perspective.

Moving from a standard batch mindset (Pandas, static DataFrames, SQL warehouses) to live, unbounded streams introduces a unique set of challenges. I’m trying to figure out which tool makes life easier—or harder—for an actual production ML/DS workflow.

I'd love to hear from anyone who has used either (or both) of these platforms for streaming. Specifically:

1. Feature Engineering & Time Windows: How painful is it to handle sliding/tumbling windows or manage late data (watermarking) in BigQuery SQL vs. Databricks PySpark?
2. Model Inference in the Stream: If you’re doing real-time scoring/predictions on the fly, how seamless is the integration? (e.g., calling an MLflow model in Databricks vs. using BQML / Vertex AI integrations in BigQuery).
3. The Online-Offline Skew: How do you ensure the feature logic you write for building your models offline matches the streaming logic online? Which ecosystem bridges that gap better (e.g., Feature Stores)?
4. Debugging & DX (Developer Experience): As a data scientist, do you find yourself fighting Databricks cluster configs and JVM errors, or hitting walls with BigQuery’s SQL-first limitations?

If your team had to choose one of these stacks specifically to support real-time data science and production ML pipelines, which way would you lean and why? What are the hidden gotchas you found out the hard way?

Thanks in advance for sharing your real-world experiences! 🙏

For those using Databricks Auto Loader (cloudFiles), how do you handle schema inference and evolution without breaking downstream ML models? If a new feature column drops in or an upstream data type silently widens, do you rely on the _rescued_data column to catch anomalies, or does the automatic stream restart cause unexpected issues for your online serving pipelines? How does BigQuery handle this kind of raw file ingestion drift by comparison?

Hey BigQuery community - here's the June 2026 summary.

🔤 GoogleSQL Language Features & Functions

• Continuous Query Aggregation Support - BigQuery continuous queries now support ARRAY_AGG and STRING_AGG aggregation functions in Preview.
• Remote Functions Custom Paths - Specifying custom endpoint URL paths allows reusing a single Cloud Run service for multiple BigQuery remote functions.

🧠 AI, Machine Learning & Foundation Models

• Autonomous Embedding Generation - Enable autonomous embedding generation on tables to automatically create and update vector embeddings when source data changes.
• AI Functions ObjectRef Support - BigQuery AI functions can now accept ObjectRef values directly as input without calling the OBJ.GET_ACCESS_URL function.
• AI.KEY_DRIVERS Function - Restored support for the AI.KEY_DRIVERS function to identify data segments causing statistically significant metric changes.
• Generative AI Token Cost Controls - Configure daily token quotas to manage and limit costs associated with BigQuery generative AI functions. (See footnote, was temporarily disabled, included for completeness.)

💻 Developer Experience (DX) & BigQuery Tooling

• Table Explorer Transition - Table Explorer behavior will transition to the Reference panel in the console starting in July 2026.
• Gemini Query Performance Recommendations - Use Gemini Cloud Assist to analyze SQL queries and receive optimization recommendations to improve query performance.
• Gemini Troubleshooting in Jobs Explorer - Use Gemini Code Assist within jobs explorer and capacity management pages to troubleshoot performance issues.
• BigQuery Studio Column Resizing - Resize table column widths in BigQuery Studio listings by dragging the divider to your preferred width.
• Gemini Cloud Assist Administration - Use Gemini Cloud Assist to monitor performance, analyze capacity, and optimize costs directly in BigQuery.
• Gemini Query Scheduling - Use Gemini Cloud Assist to schedule query execution directly within the BigQuery interface.
• Gemini Data Lineage Analysis - Use Gemini Cloud Assist to analyze and visualize data lineage directly inside the BigQuery console.
• Java Database Connectivity (JDBC) Driver - Connect Java applications to BigQuery using the newly released Google-developed open-source JDBC driver.

🔌 Data Integration, Pipelines & Ingestion (ELT)

• DTS Facebook Ads Connector Reports - The Facebook Ads connector now supports data transfers from nine additional reports including campaigns and ad insights.

🔒 Security, Governance & Workload Management

• IAM Deny Policies - IAM deny policies are now GA, allowing you to explicitly restrict access to specific BigQuery resources.
• Custom Sharing Constraints - Use custom constraints with Organization Policy to enforce granular control over specific fields in sharing resources.
• Fluid Scaling - BigQuery fluid scaling is now GA, providing per-second slot autoscaling billing with no minimum duration.

⚠️ Breaking Changes, Deprecations & Pricing Updates

• Daily Token Quota Disablement - Support for configuring daily token quotas for generative AI functions has been temporarily disabled.

As always, any feedback is welcome (about the post contents, the post itself, the community, what you want to see from Developer Relations team, etc.) - let us know!

Hi everyone, I’m in the process of interviewing at this AI company and the next step is to use bigquery dialect of SQL where I will cover real-worlds scenarios and build tables.

Problem is I have never used SQL and I am just finding out about what it is, I’ve never heard of it. I will be watching a few YouTube videos but wanted to see if anybody has gone thru this process before?

Been a data engineer for 4 years. Every time I join

a new project, I waste hours understanding what

tables actually mean.

Built a CLI tool that analyzes BigQuery tables and

explains the business context using AI.

Demo: https://www.loom.com/share/af3409be37fa4692bb38b63b9f4a58cc

Happy to share the GitHub link in comments.

I work at a startup and due to the hard economic circumstances, the focus has come back to Bigquery Cost Optimization right now (that or they fire my ass, jk), we do the usual partitioning and clustering tables based on use, althougth the reports are usually not partitioned. We realize BQ is a columnar db and we don't do the `SELECT *` business.

Still, we are trying to figure out new strategies to reduce costs. Any suggestions would be helpful. If you drop resources (on caching results and any other thing) in the chat, that'd be great too.

One bit of extra info is, most of our costs are coming from looker studio querying data everyday from report tables (multiple people using graphs and each selection on looker fires a query)

Curious how much time your team actually spends dealing with BigQuery notebook limitations like session timeouts, isolated runtimes, scheduling through Dataform etc. Like is this a minor annoyance or does it genuinely eat into your week? Trying to gauge if it’s worth pushing for a different setup or if I’m overthinking this

Was going through DB and BQ and found out Unity Catalog has unified UI and thereby saving clicks. But BQ has knowledge catalog but it isn't unified. But got to know from someone in the industry that BQ has a faster processing speed. So, just need to confirm if DB is actually saving the time and cost or is it just a myth?

A deep dive into why traditional Data Vault hash keys don’t align well with BigQuery’s clustering and pruning mechanisms. The article explores how introducing physical locality through structured surrogate keys, dates, and bucketing can significantly improve query performance and reduce scan costs. Based on practical BigQuery architecture considerations.

I'm working through the Derive Insights from BigQuery Data: Challenge Lab and I swear some of the "correct answers" are literally wrong.

For example, the first Q asked you to calculate the total cases/deaths/etc worldwide on a date. The accepted answer is general is:

SELECT sum(cumulative_outcome) as total_outcome_worldwide
FROM `bigquery-public-data.covid19_open_data.covid19_open_data`
WHERE date = 'requested-date'

This will give a much larger number than is true because it's summing over all rows, ignoring the fact that the data is hierarchical/rolled up data and has an aggregation level column that will not be accepted in queries.

A more accurate result is (and i'm realizing even this is flawed):

SELECT sum(cumulative_outcome) as total_outcome_worldwide
FROM `bigquery-public-data.covid19_open_data.covid19_open_data`
WHERE date = 'requested-date' AND aggregation_level = 0

This comes up in several of the later questions and I'm struggling to pass because I do not get how to give them the wrong answer their looking for.

How could a course on "deriving insights" direct students to literally do so in an inaccurate way??? Am I missing something??

I’m looking for help explaining the scope of a major systems migration to a non-technical executive.
Our company has outgrown Google Sheets as the backbone of our operations, so I’m migrating our entire reporting and automation ecosystem to Google Cloud.

The project includes:

• Migrating data architecture from spreadsheets to BigQuery
• Rewriting Apps Script automations into Cloud Run services
• Replacing spreadsheet-driven triggers with Cloud Scheduler and cloud-based workflows
• Moving credentials into Secret Manager
• Rebuilding reporting pipelines and dashboards
• Recreating business logic that currently exists across formulas, scripts, spreadsheets, and manual processes
• Validating that the new system produces identical results to the old system

The challenge is that I can’t shut the business down while I do this is I have to keep the existing production environment running, support users, fix bugs, answer questions, and maintain daily operations while simultaneously rebuilding everything underneath it.

From a leadership perspective, my boss seems to view this as “moving data from A to B” and struggles to understand why it takes months instead of days.
For those of you who have led similar migrations:
How do you explain the true scope of work to non-technical executives?
What analogies have worked well?
What hidden work do executives most commonly underestimate in projects like this?
If you’ve done a spreadsheet-to-data-warehouse migration, how long did it actually take compared to leadership’s original estimate?

I’d love to hear real-world examples!!!

How does your company actually find and track PII?

I'm curious what the reality looks like outside of vendor marketing.

If someone asks:
"Show me everywhere we store emails, phone numbers, names, credit cards, national IDs, etc."

How do you answer?

• Commercial tools?
• Internal scripts?
• Data catalog?
• Manual process?
• Hope for the best?

What's worked well, and what has been painful?

Hey, Google team that lurks this sub. Love the idea behind the new query results UI, but, right now, it's not showing the queries with errors.

You can see here my query -- which was called through a multi-step procedure -- failed, but only the successful steps show up in "Recent", so you have to dig through Log Explorer to figure out what went wrong.

Hi everyone, new member here! I'm writing this post due to a concern of mine on my current job. I work as a Full Stack Developer/Data Engineer/Wizard in the department of finance. What I do is develop multiple microservices that use Pandas as a data processing tool and store all the data in BigQuery (mostly invoices and payments).

Now the thing is that the end-product is visualizing all of this data on a dashboard in my (somewhat) developend frontend. Let's say that my dashboard has 20 graphics with drilldown (visualize all the invoices that compose that sum) and filters(date, currency, specific provider and type of provider), what I do is store each graphic and drilldown as an endpoint on my backend, and my frontend calls (async) every single one. But it comes to my mind, wouldnt it better to store each query on BigQuery as a materialized or normal view??

Even tho I have almost a year in this company, most of peers do not have deep knowledge on BigQuery or even GCP. So, the best thing I could is ask. I hope I made myself clear and sorry for bad english ^_^