Hello :)

I am currently setting up a Cortex Agent with the aim of using the Cortex Analyst to address complicated text-to-SQL/business questions.

At this point, I would like to establish common limitations of the orchestration budget with regards to the following factors:

– Tokens;

– Seconds.

The Snowflake solution has recommended a time budget of 5 minutes, which was unexpected.

At the moment, I am estimating something of around:

Budget: tokens: 16000, seconds: 300

The objective is primarily to protect against any unusual long reasoning or looping that would lead to unnecessary costs while avoiding rejection of valid questions.

To those who work with Cortex Agents on a day-to-day basis:

What limitations do you normally apply when setting tokens and time?

Do you use fixed limitations or determine precise limitations on the basis of the actual usage, such as P95 + certain margin?

Also, what experience do you have regarding reliability of the mechanism when applying time limits below 5 minutes?

Thanks for the feedback !

Every time snowflake cost optimization lands on me it comes down to the same couple of boring things. Credits were up a good bit this quarter, finance flagged it and it wasn't more queries or more users. Usage barely moved.

Went through query history and it was mostly one warehouse that a BI tool kept awake almost all the time because auto-suspend was set to 3600 seconds instead of 60. Whoever set that has left. Two other warehouses were sized a couple of steps bigger than the work ever needed.

Dropped the suspend and resized the two big ones, that part was easy. What I keep going back and forth on is how far down you can size a warehouse before it starts hurting, because too small and the queries just queue and run longer and you give the savings straight back.

Where do you draw the line on warehouse size and are you catching the auto-suspend stuff as it happens or only when the bill jumps?

Disclosure: I'm from Datacoves, we run managed dbt Core, so we have a horse in this race. Wrote this up because the license change is confusing and we had to untangle it for our own customers.

The short version:

One engine now. dbt Core v2 and Fusion share the same Rust runtime. Core v2 is Apache 2.0, fully open source. Fusion is the enhanced proprietary distribution on top of it.

Fusion is the recommended default, and it's a funnel. Free to install, and the new license lets anyone offer it as a managed service, as long as you don't block the path to a dbt Labs account. Free login unlocks advanced features, paid platform account unlocks premium ones. Reasonable model, but go in with eyes open.

Metered pricing is inside the dbt Core now. dbt State (skip models when logic and data haven't changed) bills $0.094 per model reused per day. Orchestra open-sourced a free alternative (Sao Paolo, Apache 2.0).

Migration gotchas: every deprecation warning must be resolved before v2 runs your project. Python models don't work in Fusion yet. Community adapters are constrained (drivers need to be signed by dbt Labs). --models and -m error out, --partial-parse is gone.

The afternoon test: upgrade to v1.12, clear deprecations, run dbt parse --use-v2-parser. Tells you where you stand without committing to anything.

Full writeup with a Core v2 vs Fusion comparison and the four ways to handle state: https://datacoves.com/post/dbt-fusion

Has anyone run the v2 parser against a large project yet? And is anyone planning to use dbt State or trying out Sao Paolo?

Hi all, i am testing out snowflake as a personal development project. I wanted to see how cdc works from a cloud database like azure postgresql. I tried to use openflow but I think it requires a license that free trial doesnt allow. I tried the legacy method of postgresql connector but I think its too complicated to setup (as a cloud newbie). Do you guys have any recommendations for testing out openflow? I dont really want to buy a license because it might cost me a lot bcs I dont really know what im doing here.

Currently I use Excel files and manually capture data in these Excel files. I have other peers who also do a similar procedure and we all have different formats. Is a streamlit app a good solution to standardize the excel format ? Is it even feasible as something that can be deployed to multiple users through tablets so that they can register the data via a streamlit app ?

I think one of my warehouses is overutilized, so I was looking at the view snowflake.account_usage.query_history to see relevant stats during queries. I know the view is documented here in the docs, but I still have questions. Is the field query_load_percent reporting how much resources were used for that specific query in the row? Or is it reporting the resources that were being used across all fields at the time the query was being executed?

Are there any other fields in that table or other views that I should look at to see if my warehouse is being used efficiently and sized correctly?

I have many excel files in sharepoint which keep updating regularly i want to connect them with snowflake tables so that snowflake tables have same data as those excel files ,suggest how to automate this

Update:
I used a JSON based configure file stored in share point which has details for each excel file like
Target table :
Columns:
KeyColumns:
UpdateColumns:
Source:{siteurl: ,folderpath: ,filename: ,sheetName:}

I parsed this sort of json in power automate then passed these variables to a single office script

Which runs for each excel file and creates a sql statement and that sql gets executed in power automate

So now if i need ro fetch data from any new excel i just need to enter its details into json configure file

This method is working good for me tried on many excels stored in different sharepoints

Hey everyone, I’ve been deep in the Snowflake ecosystem for a while, primarily leveraging snowpark, streamlit, and cortex AI for internal workflows. I’m now looking to transition into building and launching a commercial product directly onto the Snowflake Marketplace.

Instead of guessing what the market needs or launching another generic LLM prompt-wrapper, I want to talk directly to the enterprise architects, data product managers, and founders here. Given how frictionless the native app framework is for bypassing grueling 6 month enterprise security reviews (since code executes within the consumer’s perimeter), I’m looking to solve a real, high-value data friction point.

Appreciate any blunt data engineering realities or marketplace wishlists you can throw my way!

I wanted to make Iceberg Tables easier to understand, so I created this Stranger Things-inspired infographic. The idea is simple: what you query is just the tip of the iceberg, while snapshots, metadata, schema evolution, and Parquet files are hidden underneath. Feedback is welcome—especially if I've oversimplified or missed anything

Traditionally, when I needed to upload large files into Snowflake, I relied on SnowSQL, Snowflake CLI, or an external stage such as S3.

Recently, I experimented with a different approach using Streamlit in Snowflake, st.file_uploader, and the Container Runtime. This makes it possible to upload multiple files—including files larger than the standard Snowsight limits—directly into an internal stage without leaving the Snowflake ecosystem.

I wrote up the architecture, limitations, and implementation details here:

👉 Snowflake: Uploading 250MB+ files to Internal Stages using only Snowsight

I'm curious how others handle large file uploads today:

• Snowflake CLI / SnowSQL
• External stages (S3, Azure Blob, GCS)
• Snowpark or Native Apps
• Streamlit in Snowflake

Would love to hear about other approaches and trade-offs.

#snowflake #streamlit #dataengineering

While doing the Snowflake badges, I feel like I'm just copying and pasting code. I'm not going to lie—I have learned the overall UI and gained a basic understanding of how Snowflake works, but not much beyond that.

Should I continue, or not?

It also takes a lot of time just to learn the basics, and I feel like I'm wasting a lot of time doing it.

Hi r/snowflake,

I wrote a technical article about automating Snowflake account provisioning, authentication, and authorization by combining:

- Snowflake

- Microsoft Entra ID

- SSO / SAML

- SCIM provisioning

- Microsoft Forms

- Power Automate

- Microsoft Teams

Article:

[Automating Snowflake authentication, authorization, and account provisioning with Forms + Power Automate + Teams + Entra ID](https://qiita.com/mshdtksk/items/de1e9c730e2aa7c2af78)

The main idea is to use Entra ID groups as the source of authorization, where each group corresponds to a Snowflake role through SCIM provisioning.

The flow looks like this:

1. A user submits a request through Microsoft Forms
2. Power Automate receives the request
3. The user is added to a Microsoft Teams team / Entra ID group
4. SCIM sync provisions the user and group into Snowflake
5. The corresponding Snowflake role is assigned automatically
6. The user logs in to Snowflake through Entra ID SSO

This approach helped me reduce manual account creation and role assignment work in my multi-cloud lab environment.

A few key takeaways:

- SSO handles authentication

- SCIM provisions users and groups

- Entra ID groups can be mapped to Snowflake roles

- Teams membership can become an operational interface for role assignment

- Power Automate can connect the request and approval process

- Snowflake-side GRANTs still need to be designed carefully

- SCIM sync is not always immediate, so some delay should be expected

This is not meant to be a perfect production-ready pattern, but rather a practical example of how existing Microsoft 365 / Entra ID assets can be used to simplify Snowflake access management.

I would be interested to hear how others are handling Snowflake user provisioning and role management.

Are you using SCIM with Entra ID, Okta, or another IdP?

Do you manage Snowflake roles directly, or do you map them from external groups?

With Snowflake, you don’t play around with security: a AuthenticationPolicy lets you control who gets in, how, with what, from where, and under which conditions — all at once.

With a single policy, you define:

• Who can log in — allowed email domains, external accounts blocked
• How they authenticate — SSO for humans, KEYPAIR for services, passwords forbidden

I'm just grateful that I cleared it. Graduated recently and in an extremely panicky stage right now wrt jobs. Someone suggested, I should get this certificate. Most of my experience is in software development and ML and the only previous experience I had working with something related to databases was with IBM Filenet.

anyways here's the study guide I followed and some tips if helpful

- Started with Tom Bailey's videos and hands on

It is considerably easy and might give you false confidence for the actual exam

- Did a few practice tests online, many free tests and some paid ones have very outdated questions, be aware

- While practice is good, some tests are too easy while some are unnecessarily complicated

- purchased a practice test series from udemy which was wayyy outdated but they refused to give me my money back🫠

- Purchased Tom's practice set, while it gives you a vast array of info, some questions might be outdated like the maximum size of a variant row which says 16mb in the test but it's been resized to 128mb

- I was panicking about numbers, I don't think these are important except for a few like time travel or partition size. Memorising every function's time period is kind of overwhelming and unnecessary, me feels

I didn't get even one question about these

- I then tried giving practice tests with chatgpt but even with context or searching it will only give you very obvious questions and options that you can easily eliminate. The actual exam holds all the power in the choices

- I also used chatgpt sol as my final prep stage which was kinda better but exam questions are not as straightforward as exam sets or LLM generated questions are.

I realised that if I practice more tests I'll only end up panicking as there's never-ending information in snowflake so I just scheduled it.

Exam scenario:

Initial questions put me in a state of panic, the video courses and the exam sets didn't prepare me for SCD, DMF, Telemetry, Usage reports (not what you are thinking) etc but from what I noticed from plenty Redditor's posts, snowflake typically does give harder questions at the top. The second half was too straightforward and simple but I kind of panicked at the start and lost a lot of time.

Majority of the questions were related to

- Materialized vs SOS vs QAS vs Warehouse (they try to trick you but mostly the answer is materialized😭)

- result vs query vs metadata caches

- external tables and staging

- sharing and listing (I thought it was only a small part of the syllabus)

- dynamic table was overrepresented

- they are apparently huge fans of Apache iceberg and external staging

- some streaming, privileges and network policy questions but they try to trick you with options

- some easy questions about masking and row policies

- tricky part is roles, priveleges and heirarchy

- Cortex was barely there

I'm honestly surprised about the latter half of the exam, it was absolutely undeniably, a piece of cake. If I hadn't panicked in the first half, I would probably have a late 800 or early 900 score.

Prep time

I took around 10+ days to finish the videos between my depressive days and long unbearable working hours.

Then practiced test sets for 4 days, scheduled the exam the same day, panicked and rescheduled it to the next day but it was actually a good decision, if I had delayed nothing good was going to come out of it. No test series had any questions that I failed the answers for.

I don't know what the future holds and I know I could've done better but I'm just happy right now.

Edit: If the passing criteria was 50%, you would probably clear it by only studying about how queries work

I got access to Snowflake Intelligence, coco and cortex analyst and other AI stuff in snowflake I want to become an expert in this part, I need ideas or any realtime scenario labs... Thanks in advance...

We built this free course because these questions come up constantly from Snowflake users: how to stage properly, when to use SPCS, what Native Apps actually change for ingestion.

Most Snowflake ingestion setups run outside Snowflake — an Airflow box, a Lambda, a SaaS tool, and pipe data in. Snowpark Container Services changed that: you can run the ingestion itself inside your Snowflake perimeter, next to the data, on compute you already govern.

The course covers production-grade ingestion patterns for Snowflake using dlt (the open-source Python library), in this progression:

• Core loading patterns — merge/append/replace, schema evolution, incremental state
• External S3 staging — when volumes make direct loading the wrong tool
• In-Snowflake execution with SPCS — running your pipelines inside your own account
• Snowflake Native Apps — packaging ingestion as an integrated workflow

Python-based and hands-on. If you've been wiring dlt + Snowflake together from docs and guessing at the staging or SPCS setup, this consolidates it into one path.

📖 Course link

We also offer a few other free courses that aim to teach data ingestion and related best practices more broadly

• Agentic Data Engineering with dltHub — building, validating, and deploying pipelines with AI agents: from one prompt to a running system you can trust
• dlt Fundamentals — start here if dlt is new to you: core Pythonic loading concepts before you point anything at Snowflake
• dlt Advanced — custom sources, complex APIs and databases, and controlling exactly how data lands in your warehouse
• Deploy dlt pipelines — if you run ingestion outside Snowflake, this covers deploying on the common orchestrators
• Freecodecamp: Data Engineering with Python and AI — the broadest one: Python data engineering from ingestion basics through transformation, deployment, and automation

disclaimer: I work at dltHub, we're a Snowflake partner

I'm thinking about creating a completely free Snowflake course and uploading everything for the community.

Before I start recording, I wanted to ask:

What do you actually want to learn?

Should I focus on:

- Snowflake from scratch

- Advanced SQL

- Snowpipe & S3

- Streams & Tasks

- Dynamic Tables

- Performance tuning

- Security (RBAC)

- Airflow integration

- Real-world data engineering projects

- Interview questions & mock interviews

Also, what other technologies should I include alongside Snowflake? (AWS, Azure, GCP, Python, dbt, Airflow, Apache NiFi, etc.)

The goal is to build something that's practical, industry-focused, and 100% free.

I'd really appreciate your suggestions. What do you think most Snowflake courses are missing?

When I attempt to run a machine learning model, XGBoost, in Python using the XGBClassifier and GridSearchCV functions I receive the following error message:

The version of package 'xgboost' in the local environment is 3.1.3, which does not fit the criteria for the requirement 'xgboost==3.1.2'. Your UDF might not work when the package version is different between the server and your local environment.

When I attempt to pip install xgboost:

!pip install xgboost==3.1.2

I get the following message:

Looking in links: /opt/wheels
[33mWARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f8134ea7e50>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/xgboost/[0m[33m
[0m[33mWARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f8134ed8190>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/xgboost/[0m[33m
[0m[33mWARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f8134ed8340>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/xgboost/[0m[33m
[0m[33mWARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f8134ed84f0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/xgboost/[0m[33m
[0m[33mWARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f8134ed86a0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/xgboost/[0m[33m
[0m[31mERROR: Could not find a version that satisfies the requirement xgboost==3.1.2 (from versions: none)[0m[31m
[0m[31mERROR: No matching distribution found for xgboost==3.1.2[0m[31m
[0m

What's going on?

Hi,

Few teammates suggesting to set alerting based on the historical Avg. of the application query run time. Want to know from experts, is this something we should really do and will that add real value?

Say for e.g. if the normal AVG respnse time is ~10 minutes and todays its has exceeded ~30minutes, then it means something wrong is there (say it may be a bad plan etc.), so that should get alerted.

And for above to get implemented, I am thinking to refer column query_parameterized_hash and first store the last one month of query_history data from account_usage view as reference and get that refreshed daily once(say the table query_baselines). say mainly query_parameterized_hash and avg response time avg_seconds, p95_seconds etc. And have another task created , which will fetch data from information_schema.query_history and put it in a temp table and then join this data with the query_baselines table to see if any query ran > (avg_seconds * 3) , will throw an alert. This task may be executed once in ~15 minutes or so. (I understand it wont cater adhoc or newly added queries as hash will change).

Is above approach is good and really worth or any other easy and less costly way exists to achieve such alerting?

Hey there!

Got this question on a COF-CO3 Snowpro Core Practice Exam, and I truly have no clue what the answer is (google and ChatGPT don't know either, they disagree with the question).

Which table characteristic will prevent the query acceleration service in Snowflake from accelerating queries?

A.) The table contains a large number of micro-partitions.

B.) The table has the search optimization service configured.

C.) A clustering key is defined on a column with the GEOMETRY data type.

D.) The table has been created using the CREATE TABLE ... AS SELECT command.

Any help would be greatly appreciated!

Hi friends, I'm a co-founder at Greybeam. A few weeks ago we released an open source Snowflake cost observability tool with surprisingly good reception (you know how harsh reddit can be!). We got a lot of feedback on whether we could build a way to reduce idle compute as it's especially relevant for multi-cluster users. Snowflake will aggressively spin up clusters and can often take over 10 minutes to wind them down despite no activity because unfortunately the only levers are `SCALING_POLICY = STANDARD or ECONOMY`.

So today we launched this exact feature and it's free to use either at Greysight or self-hosted. the tldr is we poll Snowflake and if the following criteria below are met then we issue a SUSPEND on the warehouse.

• status is STARTED
• running queries = 0
• queued queries = 0
• resumed on >= 60s ago
• and a few others

Would love for you to try it and share any feedback, the feature is still early and a bit bare bones--really just a enable or disable config but we intend on adding more. Next up among other observability features is finer controls on scaling.

More details here: https://www.greybeam.ai/blog/snowflake-auto-savings

Github: https://github.com/greybeam/greysight

Hello, I just discovered Snowflake via Linkedin Ads. So upon researching it, I still cannot comprehend what I am reading. Can you explain to me what it does? also, is learning Snowflake can be a great skill to learn in the long run of my career (I am a structural steel detailer - creating and drafting drawings) ? Thank you.

For those who have implemented Snowflake and dbt in a regulated environment:

• Did you use one overall change control or separate change controls for the platform, dbt transformations, and stored procedures?
• How did you classify Snowflake, dbt models, macros, stored procedures, and custom SQL under GAMP 5?
• Did you treat the platform as Category 4 and custom-developed code as Category 5?
• What were the core documents in your validation package?
• How do you manage frequent dbt changes without revalidating the full platform?
• Do you validate the platform once and then validate individual data products through risk-based change controls?

I would appreciate practical lessons learned from anyone who has gone through the validation qualification.