r/zapier • u/Novel_Willow_8780 • 9h ago
I ran the same workflows 5,000+ times on Zapier, Make and n8n over 3 weeks and counted silent failures
A silent failure = the platform accepts your event, tells you nothing went wrong, and the work never happens. Everyone who runs automations long enough has a story; nobody publishes a rate. So I measured one.
Setup: identical workflows on every platform (webhook in → HTTP out), with both endpoints under my control,so a third-party app outage can't be blamed for a platform failure. A cron fires ID-tagged events around the clock; every receipt is reconciled against every fired event, one by one. I paid for all the plans myself.
Results after 5,141 monitored runs (3 weeks):
n8n (self-hosted): 3,714 runs,0 silent failures,95% CI on the true rate ≤ 0.10%,median delivery 0.8 s
Make (paid plan): 228 runs,0 silent failures,CI ≤ 1.7%,median delivery 1.0 s
Zapier (paid plan): 231 runs,0 silent failures,CI ≤ 1.6%,median delivery 4.4 s
So under normal operation: nobody silently dropped anything. That's genuinely good news, and the confidence intervals are there because 0-in-231 and 0-in-3,714 are different claims.
The interesting failures were at the edges:
- The quota wall. When Pipedream's free-tier credits ran out mid-test, its webhooks kept returning success while dropping 14 of 14 deliveries. No error, no queue,"accepted" just quietly stopped meaning "delivered". If your automation touches revenue, the failure semantics at the quota boundary matter more than the pricing page.
- One rejected POST. My own n8n box (deliberately tiny 1 GB VM) stalled about 8 seconds under a 10-event burst and refused one POST. The sender saw the error, so it's not a silent failure,but it taught me to record send outcomes separately, because a sender that ignores a non-2xx turns a loud failure into a silent one.
- The meters are honest. Zapier's task meter matched my executed-action count exactly (45 = 45). Filtered runs and editor tests billed $0 on both paid platforms. The "filtered runs still burn tasks" folklore didn't survive contact with the meter.
Honest limitations: paid-platform samples are small (that's what the CIs are for), it's one workload shape, and three weeks is short. The harness keeps running and the numbers tighten every week.
Full method, per-run raw CSV and the always-current numbers are linked in my first comment below (no affiliate links on that page; the site discloses how it makes money elsewhere).
Happy to answer anything about the harness,and if there's a failure mode you want tested, tell me and I'll add it to the queue.
