Data Chunking Techniques for Massive Orgs

Hello, and welcome to another X Force Data Summit presentation. Today I have Daniel Peter, who is the Salesforce product lead at a consultancy called Robots and Pencils. And he's talking about data chunking techniques for massive orgs. When he says massive, he means massive.

These are techniques that we've talked a little bit before the presentation. These are techniques that don't, a lot of people don't find out until they really, really need to know them. Like, and they need to fix a problem like right away. So this might not be a presentation that's good for you today but it might be something you might want to keep in the back of your head.

So here's Daniel.

All right, thanks a lot. Yeah, as Leonard said, I'm Salesforce practice leader, robots and pencils. I've also got some other titles, Salesforce MVP and Bay Area Salesforce Developer Group Leader, lot of certifications, don't know, twenty seven ish, something like that.

Yeah, I'm very happy to join you. This is a topic near and dear to my heart. I've worked with this stuff a lot over the years and really enjoy talking about it.

So what do we mean by massive orgs or large data volumes? I think it's hard to just give a single number like a single line to cross over says, okay, now you were a small org or a medium org and now you're a large data org.

I prefer to explain it as this.

To know if hitting large data volume issues or not, the volume of data that will typically cause errors with typical programming patterns. So what I mean by that is this number could move. So if you're using like a standard kind of Apex select statement and you start hitting errors, okay, now you're in a large data volume or versus maybe if you use batch Apex or bulk API, that might be a higher threshold.

Both scenarios are large data volume, but the numbers will be different. So think of it conceptually as the way you query your data starts breaking down for the way that you happen to be accessing it at that time.

So that's what I'm going refer to. I'll throw out some different numbers later, different examples, but just think of it, makes you large data when these things start to break down.

Okay, so there's a couple different kind of buckets I like to use for working with large data in terms of how are you using it.

So the first one is returning large amounts of records. The second one is you still got that large amount of data but you're not trying to return all that data. You're just trying to find some small subset of it.

So in the first example, some problems you might see are like too many query rows, fifty thousand and one. This is a common one. Salesforce only wants you to return fifty thousand like in a standard query in a synchronous context.

So in that context, anything over fifty thousand is going to be considered large data because that programming pattern breaks down.

The other example would be trying to find this needle in a haystack. So maybe you just want to give me the, I don't know, dozen records in this fifty million record database.

You're not returning very much data. But to look at that many records is like a very expensive operation. So these are some of the common places where it'll break down and the common errors you might see. System. QueryException, non selective query against large object type, no response from server, time limit exceeded, your request exceeded the time limit for processing.

And these ones you might start to see around half a million to a million records depending on how you've indexed your database and what fields you have in your query.

Those are the two buckets and the two ways these things can break down.

The first one where you're returning large amounts of records, just to go into some more details there.

I think the first thing is Salesforce puts a limit on this for a reason. It's that fifty thousand record limit or maybe one hundred thousand. You can kind of squeak it up to one hundred thousand if you move to read only context.

Do you really need to return all that data? Salesforce thinks most of the time, probably not. That's probably like a bad programming practice because if you really looking, say you have an Excel spreadsheet with like fifty thousand or one hundred thousand records, can you really understand all that in your brain at once?

What's the need for having that much data?

So just think that through as your first step.

Do you really need to see it all? Or is it more like some summary you could show instead?

But sometimes it is valid. Sometimes you need these hundreds of thousands of records on the screen at once.

But if you do, you're probably not going to want to actually put them in the DOM. Client side pagination, it kind of will keep those, say, million records in the JavaScript and sort of swap them back and forth into the DOM, maybe page at a time.

It's like your browsers can really struggle if you try to put all those records in the DOM at once. You can hold much more data in the JavaScript memory of your browser than you can in the actual DOM.

So in memory you might slice and dice, you might show some of the columns or some of the rows as the user scrolls through. We don't try to show them all at once.

And the other consideration with returning a lot of data to the screen is typically I like Chrome better but in this one use case Firefox will handle it much better. So the Chrome V eight engine goes up to about a gigabyte of heap, whereas the SpiderMonkey Firefox engine will go up to four gigabytes. So just a little trick there. If you do have a client that needs a ton of data to be returned and work with that in the DOM or in the browser, maybe consider Firefox if you're getting issues with Chrome.

Okay, so how do you do this? We talked about there's like a fifty thousand limit for synchronous, one hundred thousand for asynchronous.

So you've got some tools you can use to game the system here.

So the first thing I want to talk about is using a Visualforce is a way whether you're using Lightning or Visualforce, you can kind of use Visualforce as a container almost like an API that you use for your lightning component or LWC to get access to this data.

So what you have to do here is actually use the AJAX toolkit. So Salesforce API will not time out and an easy way to wrap the API's Ajax toolkit in Visualforce.

There's other use cases like off the platform, you might use the straight REST API or something like that, same concept, but I'm just going to use the Ajax toolkit with the visual forces as my example for this talk.

What you would do is you would run some large query, maybe it could be, let's say twenty million records, typical query locator across twenty million records might take, I don't know, ten minutes. Sounds like a long time.

But if you know, Apex will never run anything for ten minutes.

That right there is like the proof that the API does not time out the way Apex does.

So what you do is you'll have a Visualforce page with Ajax Toolkit running your query.

If you're just doing pure visual force, then that's fine. You do everything in your page there. Or if you're doing the lightning component route, you'll have, you'll put the visual force in a hidden iframe and you'll talk back and forth to that visual force hidden iframe.

Essentially, what this is going to do is it's going to give you a query locator, which will have the ability to access all the records. It's like a database cursor.

And you then use that as the keys to unlock those queries that you want. We'll show how we do that later.

I think also Einstein Analytics is an interesting possibility.

This is a database optimized for large data that sits on top of the Salesforce database.

There would still be a chunking concept there, but the chunks would be bigger. Maybe you can get like seventy five thousand records per chunk instead of fifty thousand, something like that.

I think just the other consideration with that query locator that we get with Ajax Toolkit, we have the ability to use it in parallel or in serial.

Think if you have issues with your Internet connection, too many parallel queries at once can be an issue, so you might have to consider a serial approach.

The other use case was the finding the smaller result in the large amount of data. So this is we don't have to worry about returning and holding all that data like in our JavaScript or browser, moving it to the DOM. We don't have these considerations.

But we still have to access all that data to find the needle in the haystack. It's still difficult to do.

And just a few ways we can do this.

We can use the trick that I talked about previously. You can just throw away all the data you don't want.

But I like this cool batch Apex trick, which sometimes meets the requirements. Typically when you think about batch Apex, you think about, I'm going to iterate over all the data or most of the data.

But really that query locator in the start method is one of the few places on the whole platform you can run a query with almost no governor limits.

So it says it supports up to fifty million records.

Essentially, you can put the worst non selective query in there and it'll just sit there until it finishes. So if you have a situation where you have huge data and maybe you only need the result of one execute method, so whatever you say batch size to, if you set your batch size to say two hundred, typical batch size, if you think that result is going to return less than two hundred records, you can just put your horribly inefficient query in the query locator start method of your batch Apex and get that one execution and get the result there. So that's one trick that sometimes works.

It's pretty cool. It's very simple to do. The other one is the reporting API or used to be called the analytics API before Einstein Analytics.

But this thing only returns the first two thousand records. So it's a little bit like the batch Apex, but it can return perhaps more records.

And this has an asynchronous mode, which as you know, the reporting API is different. It doesn't have the Apex governor limits.

And typically this thing can go very large too. I have seen a time out though, have seen it.

Time limit exceeded when you're running a Salesforce report, they'll get that same error when you use the API.

And just some other options I'm going to mention briefly, async SQL.

This is actually built for this, but there's an additional cost. You can use it for a million records for free or you have to pay to add on after a million.

And there's Einstein Analytics Sackwall. This is similar to what we talked about in the previous slide.

But this is a very filtered version of it. You're not returning all the large data, you're just returning the subset.

And then the bulk API. So this is Salesforce's native way to do this. This actually works pretty darn good. It has its own retry built in.

So this is something we'll talk about later about the cache warming effect and how retrying can make something work maybe on the third or fourth time that didn't work on the first time. The bulk API has that retry built And it has parallelism built in also.

Just some more terminology.

This talk is going to be talking about using a term called chunking, PK chunking. So maybe you know what this is, maybe not. I just want to get into this. So chunking in general, just kind of dividing up the data into pieces.

In our case, it's going to be groups of rows. And once we have those groups we can kind of divide and conquer. Can go after all the groups in parallel.

What's really cool about this is Salesforce is a multi tenant platform.

It's made to service a lot of customers at the same time.

And so by chunking and parallelizing, we're kind of aligning our architecture with a multi tenant platform architecture. So in a sense, whether you're a bunch of tenants using it at once or a single tenant breaking up your job into a lot of little things, the architecture is meant to handle that. So we're leveraging the strengths of the multi tenant platform instead of fighting it.

More specifically PK chunking or primary key chunking, that's what the PK stands for.

This is chunking based on the Salesforce ID. So this term is a Salesforce term.

If you look at the bulk API documentation, they reference PK chunking.

But I've actually gotten more in the weeds with PK chunking and figured out how you can implement it in custom ways other than the bulk API. We're going to talk more about that.

But at its most basic level an example of PK chunking is this query here, this third bullet where ID is greater than some ID and ID is less than some ID. What this does, it returns just a chunk of the data based on that primary key.

So we'll look at more specific examples later, but that's what chunking and PK chunking means.

So in order to do PK chunking kind of should understand the Salesforce ID which is the primary key.

So up here we have an example ID which is fifteen characters long. There's also an eighteen character version.

They're essentially the same.

The eighteen character version has the three digit checksum at the end which isn't shown here.

And that one is good for if you're doing like a say a VLOOKUP in Excel.

Excel will match case insensitively and so like a lowercase e will get matched on an uppercase ID and so if you don't use the eighteen digit version you might end up getting an improper vlookup.

So I'm kind of ignoring that extra complexity of the eighteen digit ID is the fifteen digit is what you'll see most often natively on the platform.

So this fifteen digit ID is actually a composite. It's composed of these four pieces.

The first three digits is the object ID. So if you're on the platform a lot you've got these memorized. You know that one is account and three is contact and here this a00 this is some custom object we've created.

So that's the yellow box the next one in purple the two digits is the pod id that this record was created on and this is very important created on Because this may not be the pod that it's actually on now.

And I'm going to talk more about it later, but this can create some problems in the way you do your PK chunking.

Essentially if all your records are always on one pod, all your pod identifiers for all your records will be the same. But let's say you move your production to sandbox. Sandbox is on CS10.

Well, you'll have a mix now of different pod IDs, the one that came from your NA ten production and the ones that are created under CS ten sandbox.

The other use case would be Salesforce. Your pod gets too congested and they do a pod split. Maybe they move you from NA10 to NA11. You might have a mixture of pod IDs.

And this will influence the way you decide to do your PK chunking. We'll talk about that.

The next digit there with the black background is zero is just a reserve digit. You can kind of ignore that. It'll be used for some future things.

The last nine digits in green these are the most important ones for PK chunking and this is essentially the majority of the primary key what this is is a very large base sixty two number And they've got it's alphanumeric. They've got zero through nine and A through Z. They've got A through Z, uppercase and lowercase. So ten numbers, twenty six times two for the lowercase and uppercase, that equals your sixty two potential characters. Base sixty four is more commonly used in programming in general, and that one has a plus sign and a forward slash. But for whatever reason, didn't want the plus sign and the forward slash in here. So we have base sixty two.

So the reason they use base sixty two with the sixty two different digits is because they can reach approximately thirteen point five quadrillion unique values.

If they were just to use base ten, they can only go up to about a billion values. So when you've got a multi tenant platform and you need unique identifiers, you need big big numbers. Something is base sixty two to get all those distinct values.

So what is important about this primary key?

Why do we use this instead of something else like a name or an auto number or a date to chunk our records?

If you look here, we've got the Salesforce query plan tool. This is a great tool. If you're working with big data, I recommend you get very familiar with the query plan tool. This picks apart your query and shows how the query optimizer is interpreting a query so you can figure out what's good or bad about it.

What you can see here is we're querying an object and we're querying it where some name equals.

If you see here, name equals this.

To run a query filtered on name, it incurs a certain cost. This is the cost and we're going to look at this cost relative to another cost.

This cost is if we were to query based on ID instead of name.

In this example, the Query Plan tool is showing us how efficient is this query when we filter on ID.

Here's the results of those two compared.

We can see when we queried on name, we had a certain cost, it was at zero point zero zero zero zero two five.

Then when we did the same similar query but we filtered on ID instead, you can see our cost was like an order of magnitude lower or the other way to put it is the name was like an order of magnitude higher 3x.

You've got a whole another decimal place essentially of efficiency when you use ID because lower is better for the cost. You want the cheapest cost possible to access that data.

So even though name is a standard field, we think it would be very well accessible, very well indexed, and it is pretty good.

ID is way better. So it's the most well indexed field in Salesforce.

It's kind of at that oracle database level it's always present you don't have worry about it adding it as a custom field you know every object's going to have it and the other interesting thing is even though it's this composite key with base sixty two number is one of the components the salesforce database interprets this data is ordinal So when you say things like order by ID or IDs greater than or less than, Salesforce database is actually smart enough to basically treat it like you'd expect like a base ten number.

And this is one of the keys to PK chunking is that the database can do that.

I did put careful there because you can have some gotchas. One of them can be when you have different pod identifiers in your org. There can be big gaps in the IDs.

The other one is Salesforce reporting.

Doesn't really treat it as ordinarily as it should. So you can basically treat it a little bit. I don't know, I'd say ninety percent ordinarily and ten percent like alphanumeric in some cases. So you can't rely on PK chunking with the Salesforce reports reporting. So don't put like ID greater than your Salesforce reports.

So here's a really specific example of how we do PK chunking.

So if you write this query, really all we want to know is we're querying some numbers from a large object and we just want where the number is greater than ten and less than twenty.

But if we were to run that query on millions of records, it would just it would time out or it might even block it from even executing.

But if we say if we add in these extra two where clause conditions and we say ID is greater than this, less than this, something where we're filtering down based on IDs, all of a sudden this becomes the most specific part of our query. So Salesforce has no trouble running a query where ID is greater than this and less than this if the gap between these two IDs is a reasonable sized chunk. Maybe there's fifty thousand records between this and this or maybe it's ten thousand records between this and this. So all of a sudden you've honed in on a really well indexed piece of the database.

So this is what Salesforce uses to get the specificity, to have a good cost in your query plan tool.

And now if it has to do a small table scan through that result from the IDs to get this less specific filter, it doesn't have a problem doing it.

This is the key to PK chunking is that you put the query you actually want up here, then you chunk up your database and you make many copies of this query.

Each one has this id greater than less than filter you can execute these in series or you can execute them in parallel and essentially you have to assemble the results when they come back So maybe some don't even have results. Maybe you're looking through millions of records and none of them have a number greater than ten and less than twenty. So you just throw away those responses and the ones that have results, you stitch those all together. That's the core concept of PK chunking and how you actually write a query that implements this.

So how do you do the PK chunking? How do you get the query locators? How do you come up with those base sixty two numbers? I talked a little bit about the AJAX toolkit earlier, Visualforce.

This is using something I come up with these terms, QLPK, query locator primary key chunking, or base sixty two PK, base sixty two primary key chunking.

The query locator, you can use Ajax Toolkit, you can use batch Apex, but essentially when you get that query locator, it's returning the actual IDs of all the records in the database. So this is very good for situations where you've created a lot of records, deleted a lot of records, created them again, deleted them again because you may have you may be bumping up that auto incrementing number very high over time but the query locator is just going to get what's in there right now also if you move from production to sandbox or you've done one of those pod splits where you've got a variety of pod identifiers and a variety of base sixty two numbers, query locator is very good because it gets the actual numbers. It doesn't care about what pod it's on.

The downside is you're having to ask the database for this information.

So base sixty two primary key is another technique. This doesn't get a query locator. What this does is it gets the first ID in the database. You'd say select ID from object, order by ascending limit one.

And you'd say select ID from object, order by ID descending limit one. You have the first ID in the database, the last ID, and now you can chop off the last nine digits and get those base sixty two numbers.

You can either do base sixty two operations on them to chunk them up or you can convert them to base ten and chunk them up and then recompose them to base sixty two. But essentially what you're doing is you're guessing or synthesizing at all the chunks in that database by looking at the first and the last ID in the database.

The downside here is like I said maybe you create a record it sits there from years ago over time you've created so many records and deleted them that id keeps going higher and higher so what you'll end up with when you when you interpolate the smallest to the largest you'll end up with a lot of things in the middle that don't really exist So it's fine to ask for them, but they're going to return empty results because those IDs just aren't even there.

Also, you've had a pod shift, the original pod may have had a much lower set of numbers than the new ones. So you may get like very expanded ranges there.

So I kind of they both have their pros and cons.

If you're on a production org, you've never had a pod shift, you're not dealing with sandboxes, you might be able to make a case for the base sixty two because it's much faster to calculate all the IDs than it is to actually ask the database for them.

So I've kind of got this matrix prepared here that might give you some guidance as to when to use one or the other.

So heterogeneous pod IDs means basically all of the pod you never left your original pod you never went to a sandbox you never did a pod split if that happens I'm sorry, I got that backwards. It's actually heterogeneous versus homogeneous. So heterogeneous means you might have some difference in there. You did go to a different pod. You did go to a sandbox. It's really binary here.

I would say if you've got everything on the same pod, you can consider base sixty two. If you've got any heterogeneity going on in your pod, you're forced to use query locator.

Okay, so let's move on to the next set of criteria.

So let's say you don't have heterogeneity, do you have fragmentation or not? And so this fragmentation is that process of deleting records and recreating them over time, where you might have like a chunk of records over here at the early set and then a chunk of records over here at the later set and stuff's empty in between. So kind of like if you remember defragging your hard drive or would kind of move everything back and line it all up.

It's kind of that situation. And so I gave some guidelines here. I talked about low, medium, and high fragmentation.

So low fragmentation I said is where your fragmentation is less than one point five x.

So easy way to think about that is if you use query locator based chunking and it returns ten chunks and now you use base sixty two and it returns fifteen because there's five empty ones, That's like a low fragmentation, 1.5x.

So if you've got low fragmentation and you don't have heterogeneous pod IDs, you can choose base sixty two and you'll actually get really good performance.

If you've got medium, you can choose either one.

If you've got high fragmentation, even if you're on the same pod, that's going to push you into using query locator because requesting thirty chunks when you really only need ten, or it could be even worse, requesting one hundred when you only need ten or one thousand when you only need ten, it's just too slow, too inefficient.

So that's some guidelines as to which type of PK chunking to pick. These are the two that I've come up with.

I would say this list possibly there could be another way.

I listed two. There might be as many as four. I haven't really figured them out yet but don't think of this list as exhaustive. You may find out another chunking scheme that I haven't thought of.

There may be a new platform feature that introduces a new way to chunk, but these are the only two I've really figured out for now.

So there's a lot of different when I said in that medium range you can pick either one.

There's a lot of qualitative factors as to why you might pick these different methods. I'm not going to go through this whole list. I'm just going to point out a few things.

Like the first row here, batch Apex. This is really good when it comes to something like security review. This is like a tried and true way to access a large amount of data in Salesforce.

If you produce an app for the AppExchange, this might be the way to go.

This one I've highlighted in red here, this is the one I talked about originally.

This is the Ajax toolkit parallel chunking method. This way it's like it might not pass security review or might have to explain yourself on this one because you're doing some pretty crazy things using an iframe and JavaScript browser to chunk up all the data.

But this way is very fast.

You can see the column here granular data, no. What this means is if you need to return, like if you need to do that needle in a haystack, you need to find some data within some data, this is pretty good. You can return granular data, but you have to do very small chunks.

That sort starts to break down. But essentially what you can do is you can use the PK chunking as your buckets, feed that into your apex layer with visual force remoting, and use those buckets to limit down your query. And your query is not going to return all the records. It's going to return maybe no records or maybe a few.

So in practice this red line is the one I found to be the most useful and the most fun.

I've actually built an open source library on this called Hyperbatch.

So I wrote a blog post for Salesforce developer blog some years back and it's about these PK chunking techniques. It goes into maybe a little more detail on some of the theory behind the base sixty two things like that.

But also I've got this hyperbatch repo. So hyperbatch is an implementation of QLPK and what it does is it gives you an Apex interface you can implement. You can rewrite your batch Apex as Hyperbatch very easily. They're very similar interfaces.

When you implement them as hyperbatch, instead of getting a query locator and chunking through them in series with the batch Apex framework that Salesforce gives you, it's got a Visualforce page that orchestrates running them all in parallel and there's some control over chunk size there's control over whether you run-in synchronous or asynchronous mode and you can run multiple batches in parallel if you want as many as you want.

So I'd encourage you to check this out.

It's pretty good but I would say it's probably time for some refreshes so I encourage people to make pull requests on it. I'd love folks to contribute to it.

When you use Hyperbatch, you've got synchronous and asynchronous mode. Keep in mind synchronous mode is running in this free tier of Salesforce.

So you have to make your chunk sizes smaller.

Would say one, but you get a high SLA, which is great.

One problem with synchronous mode is if your batch size is not small enough, you'll hit too many concurrent long running processes in your org. So there is some speed bumps that Salesforce puts in place for parallelism on their platform.

So you have to stay kind of under the radar when you do a lot of little things on the platform once. If you run-in a synchronous mode, you run-in asynchronous mode, you can go larger on your chunks. But it's an asynchronous queue, which doesn't really have an SLA.

So different trade offs why you might want to run-in one mode or the other. But the nice thing is the framework, you can try it either way and see what works best depending on what you're actually doing in your batch.

All right. So that's all I have for you today. Thank you very much for listening to my talk. And I've got my email up here and my Twitter address if you want to reach out to me.

And with that, I'd love to see if you've got any questions.

Well, Dan, I had never really known the components of a Salesforce ID until now, and now I do.

That's interesting. You look at them all the time and I just thought they were just completely random, but you're right.

Yeah, it's pretty fun to geek out on that stuff once in a while.

Yeah, I enjoyed it.

You know, it's kind of interesting that I just don't understand the use case we want to bring down one hundred thousand records into the browser. Can you expand on that a little bit? It sounds dumb to me, sorry, but.

No, it is. Tried to say that in a little bit nicer way. Think it is usually a dumb thing to do. The use case where we had to do it, we needed to create a real time financial accounting report, a trial balance sheet for a large company.

And they might have had lots of people on the system all the time and they needed to be able to pull all that data out real time, slice and dice you know all their general ledger accounts against all their whatever departments or products or these types of things. So that was a specific use case because when you start talking about that kind of thing you start oftentimes saying, oh well a BI tool is better, like Einstein Analytics is better. Einstein Analytics isn't real time. So if you need lots of data, if you need to analyze lots of data up to the second, this is a technique where you can use it.

But I would also say that the use case I showed was pulling all the data into the browser.

But you can use the exact same concept on a back end server connection.

So you can use the Salesforce REST APIs, for example, with a huge AWS EC2 instance, something that wouldn't be so dumb, something that could handle all that data.

And so I didn't want to go into details on all the different ways you could use chunking, but I wanted to give like maybe one or two examples.

But the concepts apply to all those things. And I would say step one is like think about if you should really even be doing this, maybe push it back and see, well, is it okay if your data is an hour old or something like that?

Yeah, right.

Have you had any experience with the I guess, Saql, S A Q L, the analytics API, do you find that to be more performance and maybe you can run around? There might be some uses where you can kind of do an end run around having to do the chunking with the PK chunking or the prequery chunking?

Yeah it's no doubt more performant and it's you know it's a BI type database instead of like a transactional database so runs circles around SOQL.

I would say there's additional cost and there's kind of that ETL aspect of it. So that's the downside. But on the upside, yeah, it's super fast and you can access it. You could build like a Salesforce app that's running a SACL query.

It can be a seamless part of your Salesforce also. It doesn't have to be like two different worlds. But I would say be careful because if you mash up Salesforce data and Einstein Analytics data, you might end up in a situation where Salesforce is showing you a certain set of values and and analytics is showing you different because the data in analytics is old.

Yeah, it's basically extracted and refreshed, right?

Yeah, so when you do that, be mindful of how you present it to the user. Don't give them a reason to find discrepancies.

Yeah, yeah. And so asynchronous SOCQL, there's a million record limit in terms of the query, what it's querying against can be has to be a million records or less or your result has to be a million records or less.

Yeah, it's what you're querying against. So the results go into a custom object.

Okay.

If you happen to already have a lot of Salesforce data storage, that result is just using storage already paying for. If you don't, you might get hit twice. You might have to now upgrade your data storage for the results for the custom object.

So I would say a million records on your input data depending on what your output data is and how much data storage you already have you may get hit with cost there.

Yeah, well they definitely make you pay to use their platform and I guess that's a perfect example of it.

They do, they do.

I think the biggest key is taking the total cost of ownership into account. Yeah, most of this data storage is premium price but that's all you're paying.

You're not paying for people to reboot your AWS servers or whatever upgrade your SSD drives or it's really a high level of service you get.

So even though you pay a lot for that service it's kind of all you have to pay for which is nice.

That's a good way to look at it. Yeah. Well thanks so much for the talk and an area that I think hardly anybody listening to this talk has probably ever gone here but as I said at the start I think it's something that it's good to have in the back of your mind and it's a little more understanding of the platform so thanks a lot Daniel.

Yeah, thank you for having me.

‍