Back to Blog

Data 360 Zero Copy: How Acceleration Turns Into an Expensive Copy

•

I've seen a zero copy stream in Data 360 run all weekend with three settings: acceleration on, Full Refresh, and a 30 minute schedule. The source table had hundreds of millions of rows. Nobody got hacked and nothing broke, but by Monday the consumption was a lot more than anyone planned for.

Each of those looks reasonable on the setup screen. Together they're the kind of mistake you only make once.

How Zero Copy Works, With and Without Acceleration

Zero copy connects Data 360 to your warehouse (Snowflake, Databricks, BigQuery or Redshift) and creates a data stream that points at a table or view. Nothing lands in Data 360 yet. What happens next depends on one toggle.

Without acceleration, it's Live Query. When something in Data 360 needs the data, like a segment, a calculated insight or identity resolution, Data 360 sends the query to the warehouse and the warehouse runs it. Nothing is stored in Data 360, so there's nothing to refresh.

With acceleration, Data 360 caches a copy of the table and refreshes it on a schedule. Queries hit the cache instead of the warehouse, which is faster on a big table that gets used all day. But now there's a copy, and every refresh reloads it and hands those rows to whatever is built on top: transforms, calculated insights, identity resolution.

Most teams pick zero copy so they don't keep a copy. Acceleration quietly gives them one.

Full Refresh Every 30 Minutes

The refresh mode is what turned an expensive setup into a crazy one. Full Refresh deletes everything in the cached copy and reloads it from the source. Incremental only picks up rows that changed since the last run.

A 30 minute schedule is 48 runs a day. Over a weekend that's well past 100 runs, each one reloading every row, tens of billions of rows in total. Depending on what's built on top of the stream, that can mean the whole table getting reprocessed downstream, over and over. And nobody checks consumption on a Saturday.

Why Self-Built Streams Go Wrong Here

The team that set this up did what most teams would do. They heard "zero copy," took it to mean nothing gets copied so it stays cheap, and set the stream up the way the screen suggested.

None of the risk is obvious from the setup screen. Acceleration is a toggle and Full Refresh is a dropdown. The docs explain each setting on its own, but nothing tells you what they do together on a table that size before you hit save. If it's your first Data 360 build, you've never seen what that combination does to a bill.

I'd bet this happens a lot more than people talk about.

What I Check Before Any External Stream Goes Live

  • Do you need the data inside Data 360 at all? If it's for segmentation, a few calculated insights or an agent lookup, Live Query is usually enough. That's where I start.
  • If you do need acceleration, use incremental refresh with a real cursor field like a last modified timestamp. Full Refresh on a big table should be something you run once, on purpose.
  • Match the schedule to how often the source changes. Refreshing every 30 minutes on a table that loads nightly buys you nothing.
  • Test on a subset. Point the stream at a view with 100k rows, let it run a day, and check consumption before you point it at the full table.
  • Set consumption alerts in Digital Wallet. One alert would have caught this hours in.
  • Don't turn on a new stream before a weekend. Same rule as deployments.

A 30 minute review of the stream config catches all of this. If you're about to connect a warehouse to Data 360, have someone who has done it take a look first. Book a call.

Get in touch

Connecting a Warehouse to Data 360

Send me the stream config before it runs and I'll tell you what I'd change.

Book a free 30-min scoping call

See the Data 360 Launch package