Transaction basics

There's a lot of great information already out there that can give you a foundational understanding of how to put DynamoDB transactions to work - one of the best is Alex DeBrie's page here: https://www.alexdebrie.com/posts/dynamodb-transactions/

I'll give you what you need to know only so far as understanding the rest of this article. When a set of DynamoDB transaction actions are combined into that atomic unit for all-or-nothing handling, what DynamoDB does under the covers is essentially a 2-phase commit for writes, or for reads just retrieving the item twice and verifying that the item images are unchanged from one result to the next. It makes sense, then, that each read or write is metered twice. It is the price to be paid for multi-item guarantees in a distributed database.

Knowing that DynamoDB developers would use transactions for things like numeric transfers from one item to another (a bank account transfer for example), the DynamoDB team were concerned about these not being idempotent, and the potential for repeated processing for a transaction that's intended to occur only once.

The magic token ride

To address the problem of repeated application of transactions involving non-idempotent actions, the DynamoDB team added a "client token" to the TransactWriteItems API. This parameter accepts a value which is intended to uniquely identify the intent of a transactional write. If the transaction succeeds, the token is stored by DynamoDB for approximately 10 minutes. If the same transaction request is seen again (with same token) within that 10 minute window, the transactional writes are not applied again, but the client will receive a success response as though it had. The idea is to allow for the proper retry operation of DynamoDB clients using transactions, but not allow repeat application of the same transactional writes. If you're using TransactWriteItems via one of the standard SDKs, you're getting this benefit - even if you weren't aware of it. What's really nice about this is that it does not consume any write units - the cost of managing this client token in a retry situation where the transaction has already succeeded on a prior attempt is metered as just one read for each item in the transaction - much cheaper than the writes.

What if you want to extend upon this to cover retries further up your stack? Perhaps in a step function or processing requests from a queue? Just take a unique identifier for each change intent at the source, and supply it as the client token for all associated calls to TransactWriteItems. Now your higher level retries are also covered! But only for that 10 minute window of course. Unfortunately, if you want this functionality for a single item change, the only way to get it is to wrap that one action in a transaction and pay double (for no reason - the two phases are meaningless as a single item change is fundamentally atomic). Yep - it's a rip-off folks (as is the metering of conditional write failures - but I digress). This client token support really should be extended to UpdateItem, and possibly PutItem and DeleteItem too.

You say you want more?

Okay - what if 10 minutes is not long enough? What if there is batch processing involved in addition to online transactions? And if a batch fails you want to be able to just replay and not make a huge mess? You need to persist those unique transaction intent identifiers longer. The answer is to persist them as their own unique items in a separate table that tracks applied transaction intents - use the identifier as the (simple) primary key, and then make their non-existence a condition check when applying any transaction. How long should you store those applied transaction identifiers? Well, that depends how long you think you need to protect against duplicate application of the same intent. You can use TTL on those records to expire them and keep things efficient - maybe a month is suitable for you... or a year?

For a complete example of this and a very simple playground where you can experiment and learn about client tokens and transactions, take a look at the simple online bank demo I've created (using only the AWS CLI):

https://github.com/pete-naylor/ddb_tx-basic-demo

You can use the long-term token storage to protect for longer periods, but still benefit from use of the client token as a means to achieve the same with better efficiency in the short term.

Okay, now I'm really done

Hopefully this blog has made you aware of the importance of handling errors appropriately when making writes to DynamoDB - and an understanding of why the default retry behavior in the SDKs could both help and hurt you. You can harness some of the powerful functionality in DynamoDB to use retries to your advantage without accepting risk to the integrity of your data. DynamoDB's strong leader-based replication makes it different from many other non-relational databases you may hear about - you don't have to accept eventual consistency or divergence, you can easily enforce constraints, and you can make atomic all-or-nothing changes across multiple items based on conditions you choose. If you build something cool using transactions, please drop me a message to tell me about it!

What makes DynamoDB so different, anyway?

The information above about possible outcomes from a request to store a change assumes that the client is talking directly to the database - and for many, that has been the reality of their experience. The client connects to the writer node for a relational database, and those three possibilities (success, fail, timeout) are what need to be handled by the developer. But when your client is talking to DynamoDB, it is not actually connected directly to that writer/leader replica of the data. The client is connected via a proxy called a "Request Router".

It's worth noting that this proxy arrangement is not peculiar to DynamoDB - you see connection poolers and similar in other databases - and there are more distributed relational databases gaining traction in the market these days too (with their own intelligent proxy layer). Here's the key point to observe - if there is a timeout for a request between the proxy and the backing database replica, how will this manifest in the result seen by the client? The proxy does not want to wait forever, so it will give up at some point - and then there is no point keeping the client waiting. In DynamoDB, the proxy (request router) will respond to the client with a 500-level response code - essentially looking the same as a "your request failed and the mutation was not applied" response. But, the change may in fact have already been applied. That's right - unlike other database systems you may have worked with in the past, an error response might actually mean that the write was successful! You should treat system error responses just as you would a timeout.

If you want to be sure that your change was applied in DynamoDB, you must typically keep retrying until you get a success (200-level) response code. If your client sees a timeout or a system error, there is often no way to know if the change was applied or not.

'Tis a lesson you should heed:

Try, try, try again.

If at first you don't succeed,

Try, try, try again.

- Edward Hickson

There are some exceptions. If you can be absolutely certain that there are no other threads that could be writing changes to the same items in your DynamoDB table, and you know the state of that item prior to your attempted write, you can make a consistent read of that item to see if your change was applied. An example might be a workflow which inserts a new item with a UUID as the key - if the request response is an error, you can just go read the item at that UUID key to see if it is present and thus determine whether the write succeeded. This assumes that you are comfortable with the statistical improbability of UUID collision!

DynamoDB SDK default behavior

Be aware that the DynamoDB SDKs automatically retry on 500 level error responses by default. It's important to keep this in mind, particularly if you are making use of non-idempotent operations like the atomic counter pattern. Depending on the use case, you might want to think about customizing this configuration - at least for non-idempotent writes. You may not even be aware of the retries, as generally only the status of the final attempt is reported in logs etc. Most errors in DynamoDB are transient - lasting perhaps a few seconds, so the retries will eventually find success.

For operations which should be retried, how long should you go on retrying? As long as it takes - or as long as you can stand, and then you'll need to figure out what it means and how to deal with this exceptional situation. When I was a kid riding around in the back of my father's car anxious to be done with his errands and move on to the destination I was interested in, I would ask "how long, Dad?". He would answer by posing another question: "how long is a piece of string?" This would drive me crazy! So, I apologize for giving you effectively the same answer on DynamoDB retries.

Making it easier to keep things tidy

We've seen that using idempotent operations in DynamoDB tends to keep things simpler - use them preferentially wherever you can. We've also seen that conditional writes can enforce some constraints to maintain correctness and concurrency control (think optimistic locking, and multi-version concurrency control). And, we've seen that recording a unique identifier for each applied application intent can help to ensure exactly-once processing (paired with a condition on that identifier not having been applied before). Is there a way we can accomplish this for additional updates to an existing item? Yes, and here is how...

For each step that changes an item, add a unique identifier for that intent to the item itself, and include in the condition expression a check to make sure that identifier is not already present. A clean way to accomplish this would be to store these change identifiers in a set attribute - because sets can easily be added to and they maintain a list of unique elements - they can also be searched for presence of an identifier using the "contains" operator in the condition expression. Let me give you a quick (JavaScript) example - in this case we have a non-idempotent operation (atomic increment of a counter). Perhaps we are processing workflows with steps from a queue - each step increments the counter, but we don't want to have any retries causing overcounting. So we maintain a string set with UUIDs to log each successful update - and we check it each time to be sure we're not reapplying a change intent.

function createUpdateItemInput() {
  return {
    "TableName": "ex1p",
    "Key": {
      "p": {
        "S": "process257"
      }
    },
    "ReturnConsumedCapacity": "TOTAL",
    "UpdateExpression": "SET #a6400 = #a6400 + :a6400 ADD #a6401 :a6401",
    "ConditionExpression": "NOT (contains(#a6402, :a6402))",
    "ExpressionAttributeValues": {
      ":a6400": {
        "N": "1"
      },
      ":a6401": {
        "SS": [
          "f9485b28-5cbb-4b96-9717-1e8edf8ee3f5"
        ]
      },
      ":a6402": {
        "S": "f9485b28-5cbb-4b96-9717-1e8edf8ee3f5"
      }
    },
    "ExpressionAttributeNames": {
      "#a6400": "counter",
      "#a6401": "changes",
      "#a6402": "changes"
    }
  }
}

The above UpdateItem operation can safely be retried as many times as you like - the change will only be applied once! This can be a very workable pattern for items which have a known lifecycle that includes a limited number of updates. It is not a good choice when the number of updates in the life of an item is large (or even unbounded). This is because that string set is going to grow and grow, eventually causing each update to become quite costly in write unit consumption because of the item size - and you could also run into the ~400KB item size limit.

Yep - I'm going to retry the "next time" thing...

I'm trying to keep each of my blogs bite-size. Looking at what we've covered so far and knowing what's ahead, I think this is a good point to take a break. So step away for a few minutes and think over what we've covered so far. See you in part 3, where we'll look at the game-changer that is DynamoDB Transactions.

"Great database in the sky, PLEASE make this change"

When you make that request to write a change to the database, you are of course hoping for the happy case - but sometimes things go wrong. Packets can be lost, sessions can drop, network links can flap, nodes and network devices can reboot or fail, your application process can crash. So, what happened?! Let's take stock of the possibilities and what they mean.

1. Database says "YES - YOU ARE LUCKY (THIS TIME) - THE DATA DEITIES HAVE SMILED UPON YOU TODAY". Joy! Move on with your application flow, knowing the database has got your back.

2. Database says "NOPE - TODAY IS NOT YOUR DAY - YOU LOSE!". Well, that seems pretty clear. Better get up off the ground, dust yourself off, and decide what to do about it. You can retry: "Hey - I said make this change and I meant it!" or perhaps "I beg of you - take mercy on me!". Or, you can just forget it and move on - maybe it's not that important anymore - bail out and apologetically take your application user back a little in their experience.

3. Database says "[AWKWARD SILENCE]". General, your messenger has not returned. How long should you wait? Eventually, you have to give up. Maybe assume it didn't happen and try again? But what if it did happen and you'd be repeating a transaction that you shouldn't? What if this results in a customer order being processed twice? Yikes.

Idempotent operations

This is a subtle topic - causes plenty of confusion. I know because it took me years to get it squared away in my head (thanks for your patience, Somu) - and there might yet be more for me to learn. But I'm going to try to explain what I know as best I can, in a way that I hope will be simple for you to apply in your own thinking.

Idempotence is a property describing an operation that can be applied multiple times and the resulting effect is unchanged. Reading a record is idempotent - you just get the view to the record and no change is made. But what about write operations?

Deleting a record is also idempotent - no matter how many times you do it, the result is the same - the record is gone! How about upserting a uniquely keyed record (writing regardless of whether there's already a record for that key)? Yep, if you were to do it multiple times the effect would be the same each time - idempotent operations. Okay, now let's look a bit closer at updating a record. If my request is to "update the record, adding 5 to the counter value X", and I make that call multiple times - is the effect the same? No, because the result might go from 22, to 27, to 32. Not idempotent. How can I make it idempotent? I could say instead "update the record, adding 5 to the counter value X, but only if the current value of X is 22". Now the operation is idempotent. How about if I ask the database to "insert this item, and assign a new unique key for it". You got it - not idempotent, because the end result is in fact different - with retries, you could wind up with multiple entries of the same data and that might have unintended consequences. A similar non-idempotent effect can be seen if your database allows a function for appending members to a list (where members are not required to be unique) - if you retry this operation, you get more and more repeat entries in the list!

Why does idempotency matter? Using idempotent operations helps to simplify things and give repeatable results. If you use idempotent operations it's easier to build in reliability functions such as retries, replaying of batches, resuming of workflows. Great, now let's look at a commonly conflated requirement that is somewhat linked to this idempotence property.

Workflow ordering and exactly-once processing

Idempotent operations sure sound nice, don't they? But let me present a situation that they don't completely address.

1. User A is doing some shopping on your retail site. Working with their shopping cart, they add Item X. The database happily inserts the record, with a unique key for Item X in the User A cart. That's idempotent - yay! But the application code does not receive confirmation - there is something flaky going on. It waits a while to see if the database response is just slow...

2. User A reloads and sees Item X in their cart: the read shows it is present. They change their mind and decide to remove it. So they delete, and again the database happily complies - this time the response is received. Another idempotent operation! The item no longer appears in the cart. User A feels assured that things are working as intended.

3. Now, the original database request to add Item X to the cart times out, and is retried. Item X is back in the cart and User A is very confused - starts to wonder if your company can be trusted.

This is a pretty simple scenario, but I think we can agree it demonstrates some undesirable behaviors. What can be done to address it? We could add some conditions to the insert in step 1, perhaps. How about if we say "only add Item X if there isn't already an Item X record present? That condition does not provide a fix. What if we change things as follows?

1. User A wants to put 3 of Item X in their cart. The application first checks how many are there, finds 0, and submits a request to the database to increase the number of Item X in the cart by 3 - but only if the existing number is still 0. No response received from the database and the application continues waiting, but 3 of Item X are in fact added to the cart.

2. User A reloads their view of the cart and can see there are 3 of Item X in the cart, they change their mind and decide to remove all 3. The application asks the database to remove 3 of Item X, but only if the existing count is still 3. This succeeds and it matches User A's intent. This is an optimistic locking pattern.

3. Now the original database request to add 3 of Item X (if the current count is 0) is retried, and succeeds. But once again, the result is that the cart contents are not what User A reasonably expects.

Idempotent operations are not everything. Sometimes we need more to keep our data true in representing our process intentions. Will multi-version concurrency control (MVCC) help us in this situation? Let's see.

1. User A wants to put 3 of Item X in their cart. The application first checks the existing record and finds that there is an item which has version 1, and the count of Item X is presently 0. A request is submitted to the database to increase the count by 3 - but only if the version number is still 1. The change is committed, but the confirming response never makes it to the application and it keeps waiting.

2. User A gives up and reloads, seeing 3 of Item X in their cart. They change their mind and want to remove that 3 of Item X from their cart. They submit a request to do so, and the application knows the current version of the record is 1. So it asks the database to reduce the count of Item X by 3, but only if the version of the record is still 1 - oh, and bump the version to 2.

3. Now the original request to add 3 of Item X (if the current version number is 0) is retried. And it fails because the version number has changed. What does this mean? It still cannot tell if the user's intention was applied or not. Should it retry by going to get the latest version and adding 3?

MVCC is also not a complete solution - it leaves some unknowns that might be very important for the user experience, or for crucial data correctness (imagine if these were banking transactions or stock trades).

To enforce order, there must be a monotonically increasing timestamp or version applied to each change intent - or an ordered queue to work from - one step at a time. And to ensure exactly-once processing semantics, you must request your changes along with the unique identifier for that intent. For example: User A wants to transfer $27 to User B - I'll assign ID UA-UB-f6812f37-4c3d-4f59-95f5-b068e2f73733 to this intent. On successful processing, the unique identifier is stored for future reference. When applying any change, it is made dependent on the unique identifier not already being present in the store. If the identifier is already present, then a prior attempt succeeded - retries can be discontinued knowing that the intent has been satisfied.

Tune in next time...

So, we're beginning to see that getting all of this right is quite complex - and there has been a lot to absorb already. I'll close out this first part of our exploration of the topic for now. Next time, we'll talk about a behavior of DynamoDB that sometimes surprises developers. And I'll share some tips and techniques for adding idempotency, controlling order, and ensuring exactly-once processing with DynamoDB. Follow me to part 2.

In this blog, I'm going to share serverless insights from my own journey. I know... you might be thinking "oh, great - yet another pundit pushing their personal serverless definition". I hope you'll stick around, as I'm going to talk about this not only from the consumer side, but from the provider side - something for everyone! I'll build the discussion around pricing models and scaling elasticity, and we'll consider unstated business and developer contracts in any serverless product that aims to be more than a fleeting also-ran in the push towards a simpler future. Let's start with a baseline around serverless expectations - for the consumer of the service, and then for the provider.

What the consumer wants (it's a lot!)

1. Simpler and faster path to getting a particular architectural functionality in place and operating efficiently and reliably.

2. Cost which generally scales linearly with consumption, and in small increments.

3. Decoupling of fundamental resource types when it comes to cost and scale: CPU, memory, persistent storage, network throughput.

4. Enough choices to cover the majority of needs, with a few extra options to cover some corner cases. Don't require a bunch of configurables just to get started.

5. Low financial barrier to entry. For experimentation and other non-production environments, give me a starting point that doesn't result in a monthly bill that's hard to justify - and give me automation to avoid expensive accidents.

6. Scaling elasticity - and minimal management of limits (or quotas, as AWS likes to call them these days). Scale out and scale in as required, covering all spikes in load without penalty.

7. Present the product to me as a nice clean API.

8. Minimize disruption from maintenance.

9. Discounts at scale: let me share in the economy of scale my consumption brings.

[Meanwhile, at the serverless pet shop... a customer wants a replacement parrot.]

"Sorry Gov, I'm right out of parrots - but I've got a slug!"

Facing the challenge as a provider (it's hard!)

1. Have to abstract the servers away from the product experience.

2. Need to build around multi-tenancy and achieve economies of scale with oversubscription.

3. Need to build in just enough resource capacity buffer to cover rapid changes in requirements for all customers without requiring onerous planning effort on the customer's part.

4. Have to isolate customer loads and avoid noisy neighbor effects without imposing limits that become onerous enough to ruin the customer experience.

5. Must develop a pricing model which is cost-following: allocates cost fairly to consumers based on the portion of load they place on various component parts of the service infrastructure.

The biggest fish in the bowl might actually be a whale

(Yes, I know whales aren't fish). When you're building a serverless product, you focus closely on the infrastructure costs and you try to track to a multi-tenancy ideal. There are many advantages to multi-tenant designs in the context of serverless products - Khawaja Shams wrote an article titled "the dark art of multi-tenancy" that I'd recommend as a foundation if you are interested in learning more about the pros/cons. The article speaks to the difficult balance of simplified scaling elasticity and protection against noisy neighbors. I want to add some emphasis on the degree of difficulty when economies of scale are a distant hope, or when dealing with variability in tenant size.

When we think about multi-tenancy and oversubscription, we tend to imagine a whole bunch of customers all neatly fitting into a "typical" subscription size. In practice, the distribution might look more like the following plot.

Such a scenario has greater significance as a new provider, or an existing provider branching into a new pool of resources (new region? different cloud provider?). What kind of experience do those "whale" customers get? Well, they aren't getting the same elasticity of scale - they are paying for the large pool of resources that everyone else is happily frolicking in. The smaller customers are living it up, neglecting good practices for smoothing load like caching and queue-based load-leveling, and paying minimally for the benefit of a "no ceiling" experience. Often the whale is asked to: 1/ provide signal on scaling requirements; 2/ spend time tweaking limits; and 3/ make commitments about continued consumption in the future. The whale may begin to wonder if they are really getting a fair deal - oversubscription is not working out so well for them.

A knee-jerk provider reaction may be to offer a separate whale "pod" - you tell us about your load requirements and we'll build you a resource pool that you don't have to share - at a great price! This is not a great long-term outcome for anyone, actually - it is short-sighted. As a provider, you need to focus on scaling out in order to get the best of those multi-tenant benefits for everyone.

A school of herring can outweigh a whale

As a serverless provider wanting to grow your product, you understand that getting broad industry adoption takes time. You have to win the hearts and minds of consumers in small increments - enticing people to experiment with your product with a low barrier to entry is on your 3 year plan! And so you focus on an offering that is low in commitment and high in value. Maybe even a free tier? If your service actually persists some state, be very careful about offering a free tier in perpetuity. It may not seem important at first, but in time the resources associated with consumption in the free tier (often forgotten and unused) can contribute to slowing of your progress towards the efficiency of scale that benefits your customers - and your paying customers end up footing the bill - the free tier isn't free, folks.

"Scale-to-zero" could be a race to the bottom

Calls for scale-to-zero are common and some will criticize services which don't appear to check that box as "not serverless!" It's an unreasonable demand in many cases, and here is why: some services must maintain significant state and/or a certain level of isolated infrastructure capacity to performantly serve the first sporadic request from a customer without an unacceptable noisy neighbor risk. What makes the difference? Persistence and heavy operations. A simple request to retrieve a stored record by key is light and easy to serve. But what if the request is for a complex SQL query that could scan many uncached gigabytes and consume a lot of CPU with JOIN, SORT, etc? If you're familiar with Redis, think about the difference between the GET (@fast) and GEOSEARCH (@slow) commands - vastly different time complexities. Logically, demanding a true scale-to-zero experience (such as that of S3) isn't going to end up well for some services because of their very nature - it will mean comprised quality of service for the consumer, and takes everyone back towards the undesirable whale/herring scenarios discussed above.

There is a very reasonable consumer expectation that we can take from a "scale-to-zero" ask - as usual, one needs to don the product manager hat and read between the lines. In some cases, a consumer of the service is willing to make a trade-off on the experience, and is happy to indicate this through a configuration. A good example is Aurora Serverless v1 - customers could check a box to say that for a particular database, they wanted the resource to go to sleep and stop incurring some charges if it appeared inactive for a while. For experimenters and non-production stages it might be a very reasonable choice - it's okay if the first request takes a while to see a result (30 seconds, a minute, perhaps) - just don't make me manage starting up and shutting down - and I don't want to be at risk of running up a bill on forgotten test environments. This was a terrific solution for both consumer and provider. Unfortunately, it went away with v2 of the product.

Aligning around a contract

How should we bring all of the above together for long term success as a consumer and as a provider? The most crucial element is the pricing model. Get it wrong early, or appear wishy-washy, and your product will face an extended uphill battle on acceptance and growth - that won't help anybody. Tips for pricing model success:

• Make it "cost following" but simple. The cost that is passed to the consumer should track well with the costs incurred in providing service to them. This might seem obvious, but you'd be surprised how many products get this wrong. DynamoDB got the core dimensions of this right: you pay for gigabytes stored, and you pay for requests by size. This works well because the requests have bounded complexity - the metering by size encapsulates the minimal compute cost, the network transfer, and the storage access. It's not perfect, and if there were ever a feature added to respond to complex/aggregates, there might need to be a compute element added. But that's okay - the existing dimensions still fit. Customers will have workload variations that consume these dimensions in different ratios, but the cost is still allocated fairly.

• Keep minimum buy-in low. If you push experimenters away by making it expensive to play with and learn about your service, your product's long term prognosis is poor. Further, don't burn learners with a surprising bill if they accidentally forget to shut down a resource. This doesn't mean "scale-to-zero" is a requirement - it means giving them reasonably priced minimal configurations, and making it easy to have them automatically shut down when idle.

• Don't give away too much, or for too long. Having consumers paying for their consumption encourages good practices that benefit everyone - optimize, clean up unused resources, avoid waste.

• Don't bundle indirectly related resource types as a single unit of spend or scale. An invented "unit" that couples say, storage volume with compute cycles in a service where workloads will run a broad spectrum of consumption ratios is just like going back to servers. As a consumer, you'll have me overspending on one resource or the other, and you might even have me trying to optimize by choosing between different types or fixed configurations of the "unit". If it looks like a server and smells like a server, please don't try to sell it to me as serverless.

• Make scaling and metering highly granular - I want my cost per unit of consumption to be close to linear - if there appear to be step function increases ahead, I'm not interested.

• Offer simple mechanisms to share in the economy of scale that my consumption affords the product. This can be tiered pricing (like S3), or a provisioned/reserved rate in return for a commitment or signal about expected traffic patterns. This is where all of the "expensive at scale" complaints come from - listen to them because there's certainly some truth behind them.

• No penalties for scaling in. If I have temporarily scaled out in consumption of one resource dimension (without signing on for a greater commitment), I want to be able to return to the exact same level of spend for the original consumption level. Scaling elasticity is about cost as well as function.

Some assessments to consider

You may or may not fully agree with me on the following, and I'd love to hear what you think. If nothing else, I hope the below information will give you some fresh perspective.

ElastiCache Serverless - this was long overdue, and the result is really quite a good start! The Memcached variation is most meaningful and achieves the best serverless experience - it even adds multi-AZ resilience where there was no replication before. It's reasonable that "scale-to-zero" may not be a fit for ElastiCache, and the Redis variation complicates things in particular (supporting a range of high time complexity operations). The minimum spend of $90/mo (presented as a minimum billed consumption of $0.125 per GB-hour) is not ideal, but it's also not terrible. Needs work: the variability of unit consumption by time complexity of the operation feels a bit opaque ("commands that require additional vCPU time or transfer more than 1 KB of data will consume proportionally more ECPUs"), and there needs to be an option to allow for automatic "sleep" when consumption truly drops to zero for some period (no data stored and no active operations). There is also no mechanism for discounts at scale.

DynamoDB - a long time standard in the serverless landscape. Started out well with decoupled metering/pricing of storage and requests - truly a groundbreaking product at launch. There's no maintenance window with DynamoDB, and no visible version concept (except for Global Tables!!) - service-side maintenance is minimally disruptive and for the most part passes unnoticed by customers. 12 years in, DynamoDB is showing its age: metering anomalies and billing pain points that force customers to jump through hoops have not been addressed. An on-demand capacity mode was introduced to simplify management of scaling beyond auto scaling of the provisioned capacity mode, but it is comparatively so expensive that it rarely makes sense for customers operating at scale. The choice of capacity modes has actually moved the product in the direction of greater complexity instead of enhancing simplicity. The integrated caching feature is woefully "serverful" and has poor software support. There is a path to discounted rates at scale: provisioned capacity with auto scaling, and capacity reservations - but managing reserved capacity is a miserable experience. DynamoDB's core infrastructure costs must surely be storage, compute and network throughput - all of which have seen massive bang-per-buck improvements over the years, yet DynamoDB has not delivered a price reduction in a decade. Product improvements that move the needle have become few and far between. Sadly, it seems that this product has reached stasis - may no longer be receiving any AWS investment? Customers should expect more.

Lambda - functions as a service! Often held as the genesis of serverless at AWS, but actually preceded by many other services which track well to the serverless target. I think there is a lot to love about the Lambda service - it allows running of your own code in an execution environment you have a lot of control of, without requiring you to deal with hassles at an operating system level. There are terrific ecosystem integrations available. The pricing model checks the "scale-to-zero" box for most people, and you pay for actual consumption. There is a mechanism for savings at scale with a commitment (Compute Savings Plans). In short, Lambda provides an amazing level of abstraction that is a great fit for many needs. Unfortunately, some cracks have appeared in its shiny serverless appearance since launch in 2014. As developers push it harder, they find reasons (like latency, memory configuration vs performance) to dig deeper into the underlying containers and operating details. One particular element that troubles me is the scaling which couples memory configuration with vCPUs - when developers have to choose a configuration and delve deep into tuning this feels too much like selecting an instance size. And dealing with concurrency limits, reserved concurrency, and provisioned concurrency seems to stray quite a ways from the serverless ideal. I can't help thinking that there will be a next level of abstraction to further refine the serverless compute experience.

OpenSearch Serverless - this launch seemed to land with a thud - and I am not surprised. In some ways it is a very helpful step forward in management for OpenSearch on AWS, but I feel (as many others do) that it falls way short of being an acceptable effort for anything that wants to bill itself as serverless. Reading through the pricing page it is readily apparent that the OpenSearch Compute Unit is smoke and mirrors - it includes a fixed ratio of memory, vCPUs and EBS storage - sounds a bit like a server, right? It goes on to explain that a minimum of four such units will be provisioned, because it requires leader and standby for each of two different functionalities. These are details one should not have to be concerned with in a serverless product. The minimum configuration costs around $350/mo (no standby OCUs) - which just doesn't make sense at all for a builder who is starting out with a small project, and scaling out from there is in increments of $175/mo. There's no discount mechanism at scale. It takes 5 or 10 minutes to create a new collection, and from there on some form of auto scaling will occur - there's not much detail on this.

What was I talking about again?

Ah yes, all the churn around this "serverless" term. Has it lost its meaning? Well, I know what it means for me, whether you choose to call it serverless or not.

Serverless is a (newish) tool for our toolbox. Making that tool everything it can be will take relentless (but realistic and clearly stated) consumer demands. And it will also take providers who begin with an earnest attempt and then consistently iterate on the experience while never wavering from the fundamentals: deliver simplicity and the benefits of multi-tenancy by achieving economies of scale. If we get all this right, developers can move more of their applications to the serverless band of the spectrum - that's the progress I want to see.

Perspectives for evaluation

We need to consider the capacity modes from both a financial perspective and an operational perspective. In terms of cost, DynamoDB users don't want to pay for throughput they aren't using, and they also want cost reductions if they provide signal to AWS on the throughput they expect and commit to it over time. In terms of operations, they want to minimize hassles, avoid undesirable throttling of requests, and optionally place limits to control unintended overspend.

Provisioned capacity mode

In provisioned capacity mode, you configure your table (or global secondary index) for a particular read and write throughput. Once in place, DynamoDB will deliver that level of throughput whenever you need it - but you'll pay for that throughput capacity whether you use it or not. The provisioned read and write throughput can also be auto scaled, with a policy which sets a minimum, a maximum, and a target utilization (70% by default). Auto scaling is based on recent consumption metrics in CloudWatch, and can take a few minutes to identify a trend before starting to adjust the provisioned capacity. You need to tune your auto scaling policy to set a minimum and target utilization which maintains enough buffer to cover rapid increases in throughput - otherwise you'll see throttling.

For long term use of DynamoDB, you can purchase a capacity reservation. This is a commitment (one year or three year) to purchase a particular level of provisioned throughput - in return DynamoDB extends you a significant (up to ~70%) discount.

Provisioned capacity gives a lot of control - you can ensure that you have a particular throughput capability before an expected peak event - you'll know that DynamoDB has already built out all the partitions on the backend to cover your needs. You can also cap the throughput so that accidental loops etc in development environments cannot result in crazy levels of spend. But maintaining an auto scaling policy for optimum efficiency takes time and effort - and you might still see throttling from time to time. The minimum provisioned capacity is 1 read unit and 1 write unit - so there's no way to completely avoid throughput cost on an inactive table in this mode.

Provisioned capacity is not an ideal answer for customers in its present form.

On-demand capacity mode

When your DynamoDB table is configured for on-demand capacity mode, you don't need to configure auto scaling or provision any particular level of throughput - and you only pay for the read units and write units which are actually consumed. The service monitors your consumption (near real-time) and splits out partitions as required - it tries to maintain a 50% capability buffer over and above your past needs. In on-demand mode, every partition is allowed to give its full capability in any given second. Each on-demand table starts with 4 partitions (each supporting 3000 read units per second and 1000 write units per second) for a total capability of 12000 read units per second and 4000 write units per second.

An important point is that splitting of partitions takes time. Each split generation (one parent partition being replaced by two child partitions to double the storage and throughput capability of that part of the key space) can take a few minutes, but it can sometimes take much longer. I like to set 30mins for a split as a reasonably safe expectation. There's no way in on-demand mode to tell DynamoDB that you're expecting a peak load that will require 32 partitions to avoid throttling - you have to switch to provisioned mode, configure your expected throughput, wait for the splitting to finish, then convert back to on-demand. Yes, that customer experience is a bit crummy. The only alternative is to actually drive the load, and potentially encounter significant throttling while those 4 partitions split not once, but three times to reach a total of 32.

Another concern with on-demand mode is cost. First, the only constraint on the throughput is the per-table limit on read units and writes units consumed per second (aka "quota", which is configurable only for all tables in a particular account/region). This is 40k reads and 40k writes per second by default - imagine the surprising bill you'd receive if you accidentally created a loop when experimenting with an on-demand table and drove consumption of 40k write units per second for a month!

On a read/write unit-for-unit basis, on-demand throughput costs ~7x provisioned throughput (and that's not allowing for the possibility of reserved capacity - which has no equivalent in on-demand). This assumes 100% utilization of the provisioned capacity, which is difficult if not impossible to achieve. But a utilization of only 14.5% is cost equivalent to on-demand! While there are some workload patterns which are sporadic and unpredictable enough for on-demand to work out as the more cost efficient choice, these are not as common as people might think - and if there is any predictability at all, it can be quite reasonable to schedule auto scaling policy around the requirements. I would argue that many load spikes are only a concern because the solution lacks best practice implementation details like caching of reads and queue-based load leveling for writes. Leaving aside the reserved capacity option, if you can use provisioned capacity with a target utilization of 20% and not see any throttling, you will be saving money over on-demand.

With that 20% target utilization in provisioned mode, you will essentially be having DynamoDB maintain built-out backing partitions which allow you to increase load 5x at any time and not have to wait for splitting to accommodate it. On-demand is equivalent to a target utilization of 50% in this regard - accommodating only a 2x increase before a delay for splitting is required.

On-demand is also not a complete answer for DynamoDB customers.

The one capacity mode to rule them all

Okay, so what would the ideal capacity mode look like? And how might the original provisioned mode have smoothly evolved to take that form?

At a high level, we'd take all the best operational parts of each mode and merge them - then the auto scaled provisioned read and write unit values would be used for billing and as an indicator for determining partitioning requirements - it would not be applied as a rate limiter (for throttling). Let me break it down a little more as the evolutionary path that the original provisioned mode (with auto scaling) could have taken...

1. Provisioned read and write capacity values are allowed to be set as zero, and zero indicates a behavior which is just like today's on-demand.

2. The provisioned read and write capacities are billed as per the existing provisioned pricing (ie, you pay for the throughput capacity whether you use it all or not), and they continue to be eligible for capacity reservations.

3. Auto scaling minimum is used to guide partition requirements - the table or index is always kept at a partitioning level that accommodates that minimum (or more).

4. Auto scaling maximum is used to set a throughput limit beyond which requests will be throttled. The maximum is optional - if not set, the account/region configured per-table limit still applies.

5. Throughput beyond the provisioned value is allowed if the partitions are capable (up to the auto scaling maximum) and is billed at the on-demand rate.

Doesn't this sound dreamy? I'd love to hear your thoughts on this! We live in the real world, so I plan to share some guidance on choosing between provisioned and on-demand in future blogs, and some tips and tricks for auto scaling success, too.

Local Secondary Indexes (LSIs) are one of the least understood features of DynamoDB. In this article, I want to dig into LSIs in depth: differences from Global Secondary Indexes (GSIs), pros/cons, potential gotchas, and practical guidance that can help you get out of a bind. For some helpful background, you can read through my earlier article which offers a high-level comparison of LSIs and GSIs.

Accessing data in your Local Secondary Index

Your DynamoDB base table is your primary index. It lets you read items and item collections using their partition key and optional sort key (as defined for the table). You can also scan the entire table if that makes sense for your needs (no, scanning is not always evil).

Defining a Local Secondary Index for your table allows you to read (Query or Scan) the same data with an optional filter applied and with item collections applying a sort order that differs from the base table.

Let's first talk about the filtering. If you define a sort key for your LSI using an attribute that is not present in all base table items, the LSI will be sparse - it will include only those items that have the attributes that are part of its (composite) key definition. This is a great way to see only items that have a particular property - a flag of sorts. You can use this to efficiently support a "needle in a haystack" pattern, where you want to only see items that are overwhelmingly in the minority among the bulk of the entire table. Additionally, you can set a selective projection - only attributes (beyond the key attributes) that you include in a configured list will be copied through into the separate stored view of the LSI. So far so good - and GSIs can do all this too!

You must define a sort key for an LSI - and it must be different from the sort key of the base table - it's intended to collect and sort items differently from your base table. A good example might be a user interface where you have multiple columns (attributes) and you want to be able to present the list sorted according to the values in any of several different columns. The partition key for the LSI must be the same as that for the base table - the primary index data lives in the same DynamoDB partition as the data for that partition key in the base table. This is different from a GSI, which is essentially a separate table residing on its own partitions (and allowing for a different attribute as the partition key).

Item collection size constraint with LSIs

All the items in the base table and any associated LSIs with the same value of partition key attribute (that is, in the same item collection) must live on the same partition - this is a requirement of strong read-after-write consistency (more on this below). DynamoDB partitions are constrained to approximately 10GB of data. While partitions can be split to accommodate greater storage (and throughput), if you have one or more LSIs defined for your table, the partition must be split between item collections. A given item collection cannot span multiple partitions if you are using LSIs. If you are not using LSIs, item collections in the base table and any GSIs may span as many partitions as required for the data volume and throughput (no bound).

This leads me to the first important point about when not to use an LSI.

Use LSIs only when you know that the size of your item collections will always be bounded by the application at less than 10GB in total size (includes all items with the same value of partition key in the table and all associated LSIs).

You can learn more about this limit (and how to monitor item collection size) in the DynamoDB documentation, here. Lee H. reminded me that there's another little-known limit relating to LSIs - item size. It turns out that the 400KB item size maximum is actually the sum of the base table view to the item and any LSI projections. There is a lot of subtle nuance around this item size limit - watch for a future blog where I'll share the details.

Getting burned by the item collection size limit

Unfortunately, there's no way to delete an LSI without also deleting the table it is associated with. Be very careful to consider the highlighted information above in the context of your DynamoDB data model. I have seen many customers encounter this limitation (including Amazon teams) - either because they didn't know about it or because they never dreamed that any of their item collections would grow to this size. I can assure you that it's a painful place to be. If you find yourself needing to get away from an LSI you now regret, you'll find a discussion on options in a later section of this blog.

Contention for throughput on a single partition key value

As mentioned above, if you have LSIs on your table, all the items in a given collection must live on a single partition. This means that the throughput for any item collection is limited to 1000 write units per second and 3000 read units per second. If you used GSIs instead, you'd be able to scale your item collections out by spanning multiple partitions. Let's imagine that you have 5 LSIs attached to your table. Items in the table are <1KB in size and the LSIs are complete (not sparse). Without the LSIs, you could update any given item (or item collection) 1000 times each second. But with the 5 LSIs, you can only update each item (or item collection) in the table 200 times each second. So, this is another important caution regarding the use of LSIs.

LSIs share partition throughput limitations with their base table, and items in a collection must always be colocated on a single partition. Without LSIs, your constraint is 1000 writes per second per item and 3000 reads per second per item. With LSIs, the same constraint applies to item collections as well. If you envision needing to write to any item collection at a rate exceeding 500 write units per second or read from the item collection at a rate exceeding 3000 read units per second, LSIs are not for you!

Reading your table data via your LSI keys

Okay - enough about the downsides of LSIs. Let's talk about a little-known upside: reading table data via the index keys of an LSI. When you make use of selective projection for a secondary index, some of the attributes are not written into the secondary index storage. Let's consider an example. Imagine you're running a drag racing track. You want to store a history of run times for each car, along with an image from the finish line camera. Drivers typically use your website to look at their run times - most often sorted by date/time, but occasionally sorted by the fastest results, and they want to see all the information for each run (including the finish line photo). The finish line photos are typically around 30KB stored (as an attribute with binary data type). The data model is very simple - below you can see the table with example data, and two secondary indexes: a GSI and an LSI. Each secondary index projects all base table items, but does not project the finish_image attribute, which contains ~30KB of binary data. The projection for these secondary indexes is "KEYS_ONLY". We're using both an LSI and a GSI with the same indexing to demonstrate some differences - hang in there. For the sake of simplicity, I'm not using actual 30KB images in a binary attribute - I just used some random strings - but this is not important to the point I'm trying to make.

A couple of things to point out: 1/ both these secondary indexes use the same key and the same projection - they seem to provide the same indexing capability, and 2/ you'll that that the LSI (third image) is annotated as a GSI by NoSQL Workbench. This is because NoSQL Workbench still doesn't include support for LSIs (what?!). When modeling with this tool, you can pretend your LSI is a GSI to work around this shortcoming. For the rest of this example, know that I did create the table with sample data, one GSI, and one LSI - as shown above.

Why not project "ALL" attributes to the secondary indexes? We want to try to reduce the duplication of storage costs and amplification of write unit consumption in the GSI and LSI by leaving out the big binary attribute!

Now - back to our access patterns. The frequent requirement for a list of results by car (sorted by date/time) with finish line images is easily met using Query to the base table. But what about the occasional list of results by car with sorting by fastest runs? Using the GSI, we can get the list of results with correct sorting, but then we need to go back to the base table separately and request the base table items by key (BatchGetItem perhaps) to include the finish line images. That could take a while, right? How about if we use the LSI? Here's where LSIs get interesting...

aws dynamodb query --table-name dragraces --index-name runs_by_time-lsi \
    --key-condition-expression "car = :mycar" \
    --expression-attribute-values '{":mycar": {"S": "Dyno Dasher"}}' \
    --return-consumed-capacity INDEXES  
{
    "Items": [
        {
            "car": {
                "S": "Dyno Dasher"
            },
            "run_time": {
                "N": "6692"
            },
            "run_timestamp": {
                "N": "1699321175"
            }
        },
        {
            "car": {
                "S": "Dyno Dasher"
            },
            "run_time": {
                "N": "7122"
            },
            "run_timestamp": {
                "N": "1699348916"
            }
        }
    ],
    "Count": 2,
    "ScannedCount": 2,
    "ConsumedCapacity": {
        "TableName": "dragraces",
        "CapacityUnits": 0.5,
        "Table": {
            "CapacityUnits": 0.0
        },
        "LocalSecondaryIndexes": {
            "runs_by_time-lsi": {
                "CapacityUnits": 0.5
            }
        }
    }
}

Okay, so above we can see that we used Query to retrieve all the results for one of our cars - and they were provided in order sorted by the run time using the LSI. Nothing unusual here - we can see that we consumed read units from the LSI and nothing from the table itself. And we only see the attributes that are being projected to the LSI from the table. But, using a capability that's unique to LSIs (doesn't work in a GSI), we can request additional attributes that are present in the table but not projected to the LSI! We use the "ProjectionExpression" parameter in the Query API to achieve this - see below.

aws dynamodb query --table-name dragraces --index-name runs_by_time-lsi \
    --key-condition-expression "car = :mycar" \
    --expression-attribute-values '{":mycar": {"S": "Dyno Dasher"}}' \
    --return-consumed-capacity INDEXES \
    --projection-expression "car,run_timestamp,run_time,finish_image"
{
    "Items": [
        {
            "car": {
                "S": "Dyno Dasher"
            },
            "run_time": {
                "N": "6692"
            },
            "finish_image": {
                "S": "g98df9gh9w9jergw9rg9j8ej9t9oijsdgoijg9e8"
            },
            "run_timestamp": {
                "N": "1699321175"
            }
        },
        {
            "car": {
                "S": "Dyno Dasher"
            },
            "run_time": {
                "N": "7122"
            },
            "finish_image": {
                "S": "fisdfoigfd99re8gjowjo0f0wreoigosdfodfnfgjs9d8gr9sd8fg8hj"
            },
            "run_timestamp": {
                "N": "1699348916"
            }
        }
    ],
    "Count": 2,
    "ScannedCount": 2,
    "ConsumedCapacity": {
        "TableName": "dragraces",
        "CapacityUnits": 1.5,
        "Table": {
            "CapacityUnits": 1.0
        },
        "LocalSecondaryIndexes": {
            "runs_by_time-lsi": {
                "CapacityUnits": 0.5
            }
        }
    }
}

Now we can see that some read unit consumption has shown up for the table - and the finish_image attribute gaps have been filled for us automagically! The Query first worked with the LSI, but then went and grabbed the missing data from the table for us. This does use the additional read units (just as it would if we used a GSI to first Query and then BatchGetItem - the very same number of read units consumed). In theory, it probably adds some latency (I found this to be essentially imperceptible) - but not nearly as much as using the two-request process with the GSI. The DynamoDB request router is already talking to the right storage node for the LSI - and it can continue to talk to the same storage node to get the additional attributes from the table - the data is colocated on the same partition. Unlike the two-step GSI process, there's no way to get a list of items from the secondary index only to find that one or more of the associated items are gone from the table when you go back to get the additional attributes in a separate request. Using this LSI "read-through" you get a valid result every time. This is powerful stuff!!

When should you consider using this feature of LSIs? First, make sure your use case does not run afoul of the constraints mentioned earlier - your item collections must be bounded in size (never going to grow beyond 10GB), and each item collection must demand relatively low throughput (never more than 1000 write units per second and 3000 read units per second). If you're okay with these, the LSI "read-through" approach can benefit you by reducing storage cost and write unit consumption for secondary index projection (if your base table items are larger than 1KB). If the read pattern supported by the LSI "read through" is high velocity, the cost of additional read units could potentially outweigh the benefits of LSI "read through" - but this would require a truly extreme situation because write units are so much more expensive than read units.

Consistent reads - cool - what does it mean to you?

Many of the proponents of the misguided "single table design" bandwagon have declared LSIs to have no value. The LSI feature seems to have been neglected by the DynamoDB team for some years, so perhaps they've been bamboozled by social media hype too? What little information is out there tends to oversimplify the distinction between LSIs and GSIs: "if you really need strong consistency, you can use an LSI". Let's talk about this "strong consistency" a bit, and put it in the context of customer experience within an application.

Here's the key insight to keep in mind as you read through this: changes to items in a table are propagated to LSIs synchronously (and atomically) and to GSIs asynchronously (and non-atomically).

First, let's look at "read-after-write" consistency. Consistent read-after-write means that if you make a change to your data, and then immediately go back to read it, you'll see a view that is inclusive of your write (plus any subsequent changes). Without this property, you might make your write, then go look for it and see a view that is not inclusive of the write that you just made. You cannot have consistent read-after-write with a GSI. You can have consistent read-after-write with an LSI (or the base table) if you specify that you want a consistent read using the optional request parameter. Think carefully about whether you truly need this property - consistent reads cost twice as much as a default eventually consistent read. In most cases, eventual consistency is fine and a great trade-off to accept (for better availability and lower cost).

The second property to consider is "monotonicity". Monotonic reads always provide a view to your DynamoDB item which is moving forward in time. Imagine you are making a series of changes to your item - it moves from state t1 to t2, and then t3. If you're also making a series of reads to that item, a monotonic view will show "t1, t2, t3" or perhaps "t1, t3". But if your reads are not monotonic, it's possible to see "t1, t3, t2" or even "t1, t2, t1, t3". When you specify "consistent read" for your DynamoDB request to a table or an LSI, your reads are guaranteed monotonic. Otherwise (default eventual consistency - always the case for GSIs) they are not. There's a very good chance that this won't matter for you and your application, but it could be important - think it through carefully.

The final property is rarely considered and perhaps most are entirely unaware of the consideration. This property is atomicity. Imagine your secondary index has a sort key which is an attribute that isn't part of the primary key for the base table. When you update an item and change the value of that attribute, the projection to the secondary index involves both deleting the originally projected view and writing a replacement view with a new key (this consumes 2x the write units). In an LSI, you will always see a valid view - but in a GSI, you may see 0 items in the secondary index (neither new nor old), one (new or old), or two (both new and old). Need an example?

Let's say you are recording game scores. Your table uses the game title as partition key, and player id as the sort key. It also has a non-key attribute for the score itself. Player X has been playing game Y for some time and their best score was 732. In the latest game, they set a new high score of 819. You record this by using UpdateItem to modify their entry in the table. You also have a GSI on the game scores table - it uses the game title as the partition key and score as the sort key. The application logic records new scores, then immediately goes to the secondary index to grab the latest scoreboard for presentation to Player X. In this sequence (using a GSI) it's possible for Player X to see their old score (732) ranked - or their new score, but it's also possible for them to be omitted completely - or to see both duplicate scoreboard entries (both old score and new). If this secondary index were an LSI, it would be possible to see the old score (with a default eventually consistent Query) or the new score - but you would never see Player X completely absent from the scoreboard or listed twice (both old and new) in the rankings.

It is important to note that some of these behaviors would be rarely seen in most use cases - however, DynamoDB is a database, and when dealing with data it is important to be very clear about what is guaranteed and what is just fortunate happenstance. As a developer, you need to make informed choices to avoid surprises, and LSIs (and GSIs) have pros and cons you need to be aware of. It may be possible to make adjustments in the application code to counteract many undesirable effects when choosing to make trade-offs in these properties - but first, you have to know about them and consider them.

So, you want out of your LSI?

Bad news - as mentioned earlier, there's no way to delete the LSI without deleting the table. There's also no way to add an LSI to an existing table. GSIs, in contrast, can be added or deleted (with caution) at will. If you just don't need the LSI anymore (or you want to replace it with a GSI), you may have options beyond migrating to a new table. You can change the name of the attribute which is defined as a sort key for the LSI - this will empty the LSI (it becomes perfectly sparse). You will still have the same constraints on the table because of the presence of the LSI (10GB item collection size maximum, item collections constrained to the throughput of a single partition), but at least you won't be paying for the storage for data in the LSI or write units to propagate table data to it. If your use case involves having data roll through the table over time, you can just let this conversion happen naturally. Otherwise, you'll need to plan for a bulk update process (be careful!).

If you're up against the item collection size limit, you can consider deleting some historical data from the table to reduce the size of the item collections. Changing the attribute name and adding a replacement GSI can also help reduce item collection sizes and give you more breathing room. Ultimately, you may just need to migrate the table to a replacement (which does not have the LSI defined). For a live table serving a critical use case, this process can be very complex - and risky. Wouldn't it be nice if the DynamoDB team enabled the deletion of LSIs? I'm not holding my breath!

Closing thoughts on LSIs

LSIs are an extremely valuable and useful feature of DynamoDB, and they can bring big wins when applied to the right use cases. If you want to gain a deeper understanding of DynamoDB and unlock more potential solutions, I highly recommend doing your own exploration and experimentation with LSIs.

You’ve probably heard about “single table design.” It has become a contentious topic, and (as usual) there are two sides to the story. In this fourth (and final) article in a series about DynamoDB data modeling, I’ll talk about the history of “single table design”. Where did it come from? How did it go wrong? And how can you make smart choices about which data belongs in the same table versus separate tables?

‍If you’ve been following along with the series I’d recommend skimming the prior articles to refresh your memory on some of the core concepts like item collections, secondary index projection, and metering. The context will be important to get a full understanding of the discussion below.

‍For the impatient, here is the TLDR: the “single table design” techniques and example models used to push an ideal of exactly one table are contrived - not representative of the reality for DynamoDB customers operating critical loads at scale. The details often do not stand up to the scrutiny of service experts, and focusing too heavily on reducing the number of tables is likely to leave you with a costly, scale-constrained, underperforming design. As engineers, we have learned to take a skeptical view of seasonal bandwagons - and “single table design” is no different. You should consider the many downsides before jumping on board and committing your developers to a DynamoDB experience that’s much bumpier than it needs to be.

The optimal number of tables for a particular DynamoDB data model might be one, but often it will be more.

Oh, you’re in for the whole story? Great - let’s start at the beginning.

Is there something to this “single table design” thing?

Yes! There is definitely something important to understand from the early learnings that first prompted the “single table design” phrase. When Amazon.com teams were trying to wrap their heads around DynamoDB as part of their database modernization goals, they tended to bring their third normal form modeling approach along from their relational database experience: they’d create the same number of DynamoDB tables as they would have done in their relational model. Unknowingly, they were building JOINs into their application code instead of making use of two very powerful concepts in DynamoDB: item collections, and schema flexibility. They were missing out on some core DynamoDB benefits and trying to build part of a database engine into their application code without reason!

Getting the best results from DynamoDB requires that you identify opportunities to denormalize - store fully-built results for easy retrieval with no additional work necessary. You can do this using an item or an item collection - not to point out the obvious, but data in the same item or in the same item collection is also (by necessity) in the same table. DynamoDB’s greatest strength is in scaling via storage rather than compute - so denormalize and take advantage! You might find that some post-processing in your application code is necessary to meet advanced requirements - but the good news is that this scales better than doing all that computation on a centralized database node.

Denormalization as a technique is certainly not unique to DynamoDB, but it is of particular importance in DynamoDB. Why? Because if you don’t store information together where it can be accessed in a minimum of requests, you may see higher failure rates and increased tail latency. This is very different from your familiar RDBMS of old, where the data is all available on the same node that’s processing your request - DynamoDB is a distributed system (in the extreme). If you store the data for a particular data operation across a number of different keys, you’ll need to retrieve them via separate client requests. The more separate requests you make the higher the chance that one will encounter a transient problem - they’re likely to live on different partitions (which reside on different nodes, in different racks, and maybe different AZs) - and they’ll go through a different set of network links, routers, switches etc.

Storing data in the same table does not help to address this - except perhaps for extremely small tables when making a Scan call.

‍What does help is storing related information in the same item or the same item collection (that is, the same partition key value).

Denormalizing in DynamoDB data modeling produced the bonus effect of having less total tables than a standard third normal form pattern. Back when single table design was first being marketed, there was no auto scaling, no on-demand, no adaptive capacity - each table meant more operational burden. So fewer tables held some attraction - however, I argue (and did so from the beginning) that if availability is a high priority, the “fewer tables is always better” claim is bogus. DynamoDB has come a long way, and you definitely want the option to tailor things like capacity mode, auto scaling policy, backups, storage class etc according to the requirements of particular subsets of your data. When you force all your data (even completely unrelated) into one table, you are stuck with just one configuration for all of it.

What went wrong?

The “single table design” phrase was originally intended to encourage Amazon.com teams to take a fresh look at data modeling possibilities when migrating to DynamoDB. In retrospect, it was a choice of words that was too easily misunderstood.

The goal was to convince those Amazon.com teams to denormalize and store related data in an already-JOINed form using schema flexibility and item collections for efficiency. Despite some internal concerns about potential for confusion in the “single table” message, the terminology was presented externally by some as a cure-all. With an AWS marketing assist and the power of social media, it took on a life of its own. Some practitioners took things too literally - making exactly one table (and even “overloaded” global secondary indexes (GSIs) - BAD IDEA) the goal in the designs they evangelized. For promotional purposes, the needless complexity was characterized as wizardry - and everybody on Twitter wanted to say they’d fully grasped it as that was the price of admission to the cool kids club! In the confusion, things like scalability, operability, and complexity were quickly waved away and glossed over with (false) claims of best practice and performance improvement. Blindly applying all of the techniques for pushing all data into just one table for every data model is something that you may get away with in very low throughput use cases - but when scaled out in production it can quickly produce some serious operational regrets.

I’ve worked with hundreds (thousands?) of large scale customers of DynamoDB to review their data models so they reach the fullest levels of success using this powerful database - this includes many AWS service teams who are building critical dependencies for efficiency and scale around DynamoDB. I have seen it work incredibly well, but I’ve also seen all the various modalities of burn when it is misused.

Based on my experience, I would strongly recommend treating the number of tables as an outcome of a good DynamoDB data modeling approach.

Denormalize when possible and advantageous - beyond that think about your operating requirements and standards. Do NOT start with exactly one table as a design constraint or you’ll end up working backwards through a lot of complicated and costly compromises to make it happen. There’s a very strong likelihood that you’d regret those choices in time.

A wise man (hi Krog!) once said, ”just because you can, doesn’t mean you should”. It has always been possible to place any two items in the same DynamoDB table - all they need is the same data type for the primary key and unique values of the key attribute(s). It’s easy to meet these requirements (for a price). But should you choose to? Well, only if it makes it easier and more efficient to manage and make use of that data. Many of the techniques discussed as the “single table design” story gained momentum were aimed at finding ways to cram completely unrelated data (which will never be stored together in an item collection or retrieved together using Query) into the same table.

Reasons to store data in the same table…

• It’s one “record” - that is, it is denormalized into a single item, or into the same item collection (in the table itself or in a secondary index) - this is so that it can be efficiently updated and retrieved - the data is definitely related and you can’t help but keep it in the same table!

• It’s multiple records, but of the same type and authoritatively “owned” by the same service. This is analogous to S3 buckets - if you were running the “invoice” service, you’d probably create a separate S3 bucket for the invoices, right? Why? Isolation reduces risk and you want the flexibility to manage all the data the same way without forcing the same configuration on other data.‍

The price you pay for “exactly one table”...

You make adjustments to ensure key uniqueness - for example, adding a text prefix to every partition key value. This increases the size of your items (recall that item size is the most crucial factor in cost efficiency and smooth scalability). It also adds developer complexity and reduces readability. And you are not getting anything in return if neither of the above reasons to store data in the same table are true.

1. All the data must have the same Streams configuration, the same backup configuration, the same capacity mode and auto scaling policy, the same TTL attribute name, same table class, same global table replication configuration, same table-level metrics (you lose potentially crucial per-data-type observability). Different types of data often have different operating patterns: different timing of load variations and more or less spikiness; and/or a different balance of reads/writes/storage. You are limiting flexibility in a big way - for what?

2. You can’t limit a Scan by type of data and you can’t limit an export to S3 either. These are both very common and useful paths for ETL, and also for occasional bulk updates. As an example, imagine you’ve got an avatar management site with 10k user accounts (1KB items), and 2M 5kB image items. You bundled all those records into the same table (social media told you to do it). Now you want to revise all the account creation timestamps in the account records to store as a number attribute (epoch seconds) instead of a string (ISO 8601). To find those account records, you have to scan or export the entire table, 99.9% of the effort being wasted on records which are irrelevant to your goal.

3. Increased blast radius. Even if you’re following the guidance to isolate data stores by service (shared access only via APIs), mixing unrelated types of data within a service in the same table doesn’t make sense. In the avatar management scenario above, let’s imagine that you made a mistake in your bulk update and you pummel one account item to the point that the partition throttles. Yes, DynamoDB will try to split things out to isolate the hot key (this takes time), but in the meantime you’re impacting a portion of your users whose account items live on the same partition - they may have trouble with the account preferences management part of the site. BUT, you’ll also be affecting the experience of a different bunch of users whose avatar access is also being impacted. Wouldn’t it be nice if the dependency upon these unrelated data types was independent? How about if you make a big mistake with your accounts management workflow and you want to restore those account records using point-in-time recovery? Now you’re dealing with a much bigger volume to be recovered and a longer time to recover. When you have unrelated data supporting separate functionalities, “exactly one table” is not your friend.

4. Complicated access control. For unrelated data, it’s easier to manage policy at a table level - that means less risk of error and better security.‍

But I saw something in a video that said…

1. “It’s more performant/efficient to get multiple items if they’re in the same table”. This is only true if the items are in the same item collection so you can retrieve them using Query. Otherwise, DynamoDB doesn’t care at all - if you BatchGetItem 10 items from 10 tables, the read unit consumption and latency will be the same as if they were all stored in the same table.

2. “Because of some DynamoDB internal stuff, magical things will happen if I just force all my data into one DynamoDB table”. Nope. Does not work that way in real life, sorry.

3. “Multiple data types sharing the same auto-scaled capacity in a table will allow for sharing of underutilized provisioned capacity and I can save money”. There might be rare cases where this could save you a tiny margin on your DynamoDB bill, but I haven’t seen it happen. Given that “exactly one table” strategy typically involves paying for larger items, over-projection to secondary indexes and wasted read/write units, this feels quite dubious - and are you willing to put your future operational success at risk to roll the dice? That would not be my recommendation.‍

The straight scoop

In full disclosure, my work history at Amazon gives me a strong DynamoDB bias. I’m working on different serverless infrastructure services at Momento these days - but I still have deep affection for DynamoDB. It’s an incredible database when applied effectively for the right purpose - and we’re very lucky to have so many choices in the database space these days! The engineering experts who build and operate DynamoDB are phenomenal folks and they have over 10 years of experience continually refining the product for customers. They want you to have success when building around DynamoDB, and I do too.

Unfortunately, the misinterpretation of “single table design” took DynamoDB and some of its customers on a painful tangent. I spoke with countless folks who were about to give up on DynamoDB because the (very reasonable) design they’d come up with had more than one table, and merging them made no sense. I was able to help many of them move past the confusion, but I often wonder how many others have abandoned their interest in a DynamoDB-backed solution because of misguided “single table design” pressure in social media. Until recently, the DynamoDB documentation did not mention “single table design”. Sadly, it was recently updated to include information about the (apparently binary) choice between “single table design” (one heavily misunderstood phrase) and “multi table design” (a truly strange and misguided concept) - so, now it’s clear as mud, right?!

Here’s my advice to you, dear reader: set aside the whole “single table design” debacle. There’s no free lunch in databases and DynamoDB isn’t as different or complicated as some would have you believe. When building your DynamoDB data model, start with your access patterns, recognize that DynamoDB tables allow for schema flexibility, keep items small, and build item collections where appropriate to optimize your read and write unit consumption. Carry the same methodologies through to your secondary indexes. Factor in the operational needs you have. You’ll wind up with N tables. It’s okay - go ahead and prosper.‍

If you want to discuss single table design with me or suggest topics for me to write about in future articles, please reach out to me on Twitter (@pj_naylor)—or email me directly.

This is the third installment in a series of blogs about DynamoDB data modeling, mainly crafted to help DynamoDB developers understand best practice and avoid going down a regrettable path with misguided “single table design” techniques. In the previous episode—is this a show now?—we introduced local and global secondary indexes (LSIs and GSIs). This time we’ll dig deeper to understand which table data gets projected into a secondary index and how that contributes to throughput consumption. We’ll also tease apart one of the more recent twists in “single table design”—GSI overloading—and explain why it is generally a really bad idea with no actual upside. Ready? Off we go…

Which base table changes are relevant for projection into a secondary index?

When you define a secondary index, you get to choose which attributes will be copied (projected) from relevant item changes in the base table:

• ALL - all of the attributes in the base table item will be copied to the secondary index

• INCLUDE - only the named list of attributes will be copied (plus the primary key attributes for the base table and the secondary index—these are always included)

• KEYS_ONLY - only the primary key attributes for the base table and the secondary index will be copied

Here is how metering works for DynamoDB secondary indexes: every write you make to your base table will be considered for projection into each of your secondary indexes. If the change to the item in the base table is relevant to the secondary index, it will be projected, and there will be write unit consumption—metered in accordance with the size of the projected item view. There will be a storage cost for the projected data in the secondary index too. And remember—as explained in the first article in this series, Query and Scan (the only read operations available for secondary indexes) aggregate the size of all the items and then round up to assess read unit metering. If you keep the projected view of your items small, the index will cost less and will scale further before facing any kind of hot key concerns.

Consider projection very carefully for your secondary index. ALL seems simple but can be very costly and may inhibit scalability of your overall design. Only project the attributes that you really need.

‍There’s even more to this efficiency equation for secondary indexes. You’re probably wondering why I keep emphasizing the word relevant. Well, it’s because it is really important! First, because DynamoDB offers schema flexibility, the attributes defined as the key for your secondary index are not required to be present in every item in your table (unless they are part of the primary key for that table). If the index key attributes are not present in an item in the table, it will not be considered relevant for projection into the index. You can use this to create a “sparse” index—a powerful mechanism for filtering—to find the data you need at lowest storage and throughput cost.

‍An example might be a database which stores details of status for millions of jobs—the vast majority of them being completed. How can we make it easy for worker processes to find jobs which are in “submitted” status? For jobs which are in the relevant status, populate an attribute which indicates this, and create a secondary index which is keyed by this attribute. The attribute’s presence effectively becomes a flag—when the job is completed, remove this attribute—the job will no longer be visible in the secondary index for consideration by workers.

[Our base table which tracks job status]

[Sparse secondary index which contains only jobs awaiting assignment]

Now let’s imagine that as part of the workflow for jobs in submitted status, the job item in the table is updated five times to add various details in non-key attributes. If those attributes are projected to our submitted jobs sparse secondary index, the index will also need to be updated five times, consuming a bunch of write units. But our sparse secondary index really has no use for those attributes—they’re not relevant for the access pattern we need the index to serve. So we choose a ^KEYS_ONLY projection for the index—it keeps read unit consumption low, maximizes scalability for the worker processes, saves money on storage, and avoids a bunch of unnecessary writes for index projection.‍

This is worth repeating: only project the attributes that you really need.

[The same sparse secondary index with KEYS_ONLY projection - much more efficient!]‍

What is “GSI overloading” and why is it a bad idea?‍

Back in 2017, Amazon teams were working hard to migrate their most critical workloads from relational databases to DynamoDB. They were learning to think differently about data modeling—looking beyond the familiar third normal form, and denormalizing into DynamoDB items and item collections. At that time, DynamoDB supported a maximum of 5 GSIs per table—hard limit. In some rare cases, there would be a team with a table which required more than 5 GSIs to cover all their access patterns. For those of us working to support those teams, there was a realization that some of the index needs could be sparse, and did not overlap. Maybe with some wonky adaptation we could use the same GSI to cover multiple access pattern requirements—could this help us to cheat on the GSI limit? Yes: it was a last resort, had terrible efficiency and operability implications—but sometimes it worked. We referred to it as “GSI overloading”.

Fast forward to December 2018, and DynamoDB increased the limit on GSIs per table to 20. Rejoice! No need for the ugly overloading hack henceforth… right? Well, the engineering team thought so—but then came some disturbing twists in the “single table design” story.

It turns out that if you do unnatural, unnecessary, complex, and inefficient things to push completely unrelated data into just one DynamoDB table for no reason, you’ll wind up having to create more secondary indexes for that table.

If you are designing with the goal of exactly one table (instead of prioritizing efficiency, flexibility, and scalability), you are more likely to encounter the limit on GSIs per table—and then the same “exactly one table” obsession drives a misguided desire to have exactly one GSI, which leads to overloading. This creates a whole slew of problems. There is actually no benefit to this technique—it’s a bad idea. Here’s why…

• You’ll likely need to make tradeoffs for projection that will cost you money and hurt your design’s scalability. The lowest common denominator is projecting ALL and defaulting to strings for primary key attributes (when numbers would be more efficient). Ouch!

• You lose the flexibility of being able to scan the secondary index and retrieve only the data you care about. That’s a powerful functionality gone—for no reason at all.

• One of the great things about GSIs is that you can delete them and recreate them as necessary. Deleting a GSI costs nothing. If you’ve mixed multiple index requirements into a single GSI and at some point you realize that you don’t need to index one of your patterns, you cannot just delete that GSI—you’ll have to (carefully) go back to the table and make expensive (and potentially risky) bulk updates. You really don’t want to be dealing with that.

Important takeaways

In short, there’s no benefit to “GSI overloading”—only negatives. Don’t do it! Also, if you care about cost optimization and scalability, consider sparseness and minimal projection when you design with DynamoDB secondary indexes.‍

In the next article in this series, I plan to build on what we’ve learned so far to explain where things went wrong with “single table design” and why interpreting it as exactly one table is a terrible mistake (that Amazon teams are not making).

This is the second article in a short series about DynamoDB data modeling (without the “single table design” misconceptions). If you haven’t read the first article, I’d encourage you to check it out. In that one, I discuss the most important concepts in DynamoDB modeling: schema flexibility and item collections. This time, I’ll build upon that foundation to discuss secondary indexes—one of my favorite DynamoDB features. Please note this article assumes some familiarity with core DynamoDB concepts like partitions, items, primary key types and data types. If you need to get up to speed on those, here’s a video playlist I recommend.

Secondary indexes are powerful! You can use them to automatically provide different perspectives for reads of the data in your table. They help you to define and maintain additional relationships (item collections) within the same data, they allow you to sort the related data with a different dimension, and they can be incredibly effective filters.‍

But like any other tool, it’s possible to use DynamoDB secondary indexes poorly.

With great power comes great responsibility—and great need to understand your options! In this blog, I’ll focus on comparing the different types of secondary indexes and offer some tips for choosing between them. I’ll save talk about secondary index projection choices and the consequences for cost and scalability for the next blog—plus some observations about a recent trend in use of DynamoDB global secondary indexes called “overloading” and why it should be avoided in the majority of designs.

So stay tuned! And hold onto your hats—it’s going to be a whirlwind tour.

‍Where do secondary indexes live? And how do they get there?

Data from your table (the primary index) is projected into your secondary indexes by the DynamoDB service based on the index definition you provide. You can’t write directly to a secondary index, but when you write to an item in your base table DynamoDB will project relevant changes into your secondary indexes for you. There are two types of secondary index: Local Secondary Indexes (LSIs) and Global Secondary Indexes (GSIs).

‍Local secondary indexes

LSIs live on the same DynamoDB partitions as the base table—they share the same partition key attribute (but have a different sort key attribute) and they share throughput with the base table. LSIs are local because they offer a different sort order for an item collection within the same partition. LSIs support strongly consistent reads after a write to the base table if specified as a parameter to the request (otherwise the default eventual consistency is used).

In fact, LSIs and base tables share the same item collections—and this constrains each item collection to living on a single partition. When a table has one or more LSIs, each item collection can never grow beyond ~10GB (all data for the same value of the partition key value in the base table and all LSIs combined). The read and write throughput for any given item collection is limited to 3,000 read units per second and 1,000 write units per second within the table and all associated LSIs.

LSIs must be defined when the table is created and cannot be deleted without deleting the associated base table. Think carefully before using LSIs in your data model—you should have a good reason (like a valid requirement for strongly consistent reads), and you must know that you’ll never have an item collection which could grow to require more than 10GB, 3,000 read units per second, or 1,000 write units per second.

Global secondary indexes

GSIs are like a separate table—they can have a different partition key attribute, and they have their own partitions and throughput capability. They can be added (with backfill) as required, and removed (at no cost) when no longer needed. They are global because they allow new relationships (item collections) to be defined across items in all of the base table’s partitions. Item collections in a GSI can span partitions to store more data and deliver greater throughput (also true for the base table as long as there are no LSIs).

One of the biggest differences between LSIs and GSIs is in their behavior during writes to the base table. Because the base table and any LSIs share the same partitions, any updates to the LSIs are handled atomically with the change to the base table item. For GSIs, the change is asynchronously propagated to a different partition. This has read-after-write consistency implications. Reads from an LSI can be requested to be consistent if desired—but a read from a GSI is always eventually consistent.

‍A further consideration for reads from a GSI is monotonicity. GSI reads are not monotonic. If you update an item in your base table to increment an attribute’s value from 7 to 8, you could make three successive reads of the projected data in your GSI and see the value as 8 first, then 7, and finally back to 8. A series of reads for the same data in a GSI can return results that move both forward and backward over time. Strongly consistent reads from the table or an LSI are monotonic.

‍GSIs are more flexible than LSIs, and any LSI could easily be modeled as a GSI instead. Only use LSIs if you are sure the index will need to support consistent/monotonic reads, or if you want to benefit from the “read through” capability (details on this in a future blog).

‍Interesting properties of secondary index keys

First and foremost: the value for the index key is not guaranteed unique like the primary key in your base table. The index can have multiple projected entries which have the same values for the index partition key and sort key! The GetItem API is not supported for secondary indexes because GetItem implies reading a maximum of one item for a particular value of the key. But in a secondary index, even when a specific value of the index key is provided you may see many items returned! You must use Query or Scan to read from an LSI or GSI. LSIs provide alternate sorting, GSIs provide alternate collections and (optional) sorting.

‍As mentioned before, an LSI must have a composite key. The base table the LSI is attached to must have a composite key also. The LSI’s partition key is required to be the same as the base table, and the sort key attribute must be different from that of the table. A simple example might be a graphical user interface which lists a set of entries in tabular form—the entries are grouped (item collections related by the same value of the partition key attribute), and sorted by one of the column’s values (base table sort key). What if the user should need to sort the same group of entries by a different column? This is where an LSI can help you.

GSIs are more flexible: a partition key is required, but it can be different from the base table—you can relate/group your table’s data on a different attribute! A GSI can have a simple key or a composite key—and the same is true for the base table it is attached to. If you don’t need to retrieve item collections from your GSI in sorted order (perhaps the pattern is just to Query for all the items that have a common value of partition key in the index), don’t define a sort key—the GSI can still build collections of items for efficient retrieval. Defining a sort key when it’s not necessary can limit your scalability for the item collections.

Distilling distinctions between secondary index types

There's a lot of nuance here, but it really boils down to some simple differences, so I made a cheat sheet you can use next time you're considering your secondary index options.

Keep your eyes peeled for the next entry in this DynamoDB Data Modeling series (which aims to debunk “single table design”). Coming up in future articles: more on secondary indexing and a summary discussion about where “single table design” went horribly wrong.‍

If you want to discuss this topic with me, get my thoughts on a DynamoDB data modeling question you have, or suggest topics for me to write about in future articles, please reach out to me on Twitter (@pj_naylor)—or email me directly!

When I left the DynamoDB team a couple of months ago, I decided that I needed to share what I’ve learned about designing for DynamoDB and operating it well. Through my six years of working with internal and external DynamoDB customers at AWS, I was fortunate to gain exposure to a wide variety of DynamoDB data models. I’ve also seen the way they evolved in operation: how they scaled (or didn’t) under varying load, their flexibility to accommodate new application requirements, and whether they became cost-prohibitive over time.

This is part one of a short series of articles I’ll publish aimed at correcting some misconceptions about DynamoDB data modeling best practices. In recent years, strange recommendations have evolved through the rumblings of social media (with help from AWS marketing). I’m going to write about the real world. I’ll provide all the same advice the DynamoDB team gives to other Amazon teams for whom the service is mission critical—the same optimization recommendations, the same warnings about “single table” snake oil, the same tips on interpreting metrics, the same detail on metering nuances, and the same operational excellence focus.

Ready to get started? In this first installment, I’m going to set the foundation for the series by sharing the secret sauce to success with DynamoDB. Admittedly, it took me a while to truly grok this stuff. Follow me! I know some shortcuts.

‍What really matters if it’s not all about the number of tables?

The first two concepts you need to understand when you are learning to take DynamoDB data modeling beyond the simplest key-value use cases are 1) schema flexibility and 2) item collections.

Schema flexibility means that the items in a DynamoDB table do not all require the same structure—each item can have its own set of attributes and data types—this opens up many possibilities! In fact, the only attributes for which the schema is enforced are primary key attributes—those used to index the data in the table. Each item must include the key attributes defined for the table with the correct data type.

‍An item collection is the set of all items that have the same partition key attribute value; in other words, they are all related by the partition key’s value. If you’re familiar with relational databases, one way to think about this is to look at an item collection as being a bit like a materialized JOIN, where the common value of the partition key attribute is something like a foreign key. To optimize your DynamoDB data model, you want to look for opportunities to use item collections in the base table or in secondary indexes where you have related items that you’d like to (selectively) retrieve together. This might seem obvious, but it’s worth being clear: to be in the same item collection, items also need to be in the same table—the item collection is a subset of the table.

For example, you might represent a shopping cart as an item collection, where the unique cart identifier (Pete’s Cart) is the partition key value, and the identifier for each product added to the cart is the sort key value. Apparently, Pete likes coffee more than veggies, but not as much as chocolates. The number of each product type in the cart is another (non-key) attribute. The image below depicts such an item collection.

‍Your code has a more rigid definition—like the NoSQL Workbench screenshot below.

Item collections can exist in the primary index (the base table) and/or a secondary index. I’ll dig into the details of secondary indexes in a future blog, so let’s focus on the base table this time. Item collections in the base table require a composite primary key (partition key and sort key). If there are no Local Secondary Indexes (LSIs), the item collection can span multiple DynamoDB partitions—it won’t start out that way, but DynamoDB will automatically split out the item collection across partitions in order to accommodate growing data volume, and sometimes it can also distribute the item collection across partitions in order to provide more overall throughput or to isolate hotter items (or ranges of items).

‍It’s helpful to think about items and item collections as being two different types of records in DynamoDB. In a base table with a simple primary key (no sort key defined), you work entirely with items. But if your table has a composite primary key (partition key + sort key), you work with item collections. The items in the collection are stored in order of the sort key values and can be returned in either forward or reverse order.

An item collection is magical—it allows us to efficiently store and retrieve related items together. The items in the collection might have differing schema (DynamoDB is flexible, remember?)—each representing a part of the overall item collection record.

If you have a 200KB item, then updating even a small part of that item will consume 200 write units. If that item is instead stored as a collection of item parts in a collection, you can update any small portion for minimal write unit consumption. But that’s just the beginning!

You can use Query to retrieve all the items in the collection, or only those with a specific range of sort key values. The specification for the range is called a sort key condition. Within the item collection you can limit items retrieved to be those with a sort key less than a chosen value, or greater than, or even between two values.

For string sort keys, you can also use begins_with (which is really just a variation of between if you think about it). Applying a sort key condition effectively limits the range of items to be returned from the item collection. Importantly, only items in the collection that match the sort key condition will be included in the metering of read units. This is in contrast with a filter expression which applies after the read units have been assessed—so you still pay to read items that are filtered out. When you retrieve multiple items using Query (or Scan), the metering of read unit consumption adds the sizes of all the items and then rounds up to the next 4KB boundary (rather than rounding up for each item like GetItem and BatchGetItem do). So, item collections allow you to store very large records, update selective parts at low cost, and retrieve them with optimal efficiency.

Item collections (Query) and Scan are actually the only compelling cost and performance differentiators that argue for storing two items in the same table. They both allow for retrieval of multiple items with the aggregated item sizes rounded up to the next 4KB boundary. If you’re not going to index two items together in the same item collection (table or secondary index) and you don’t want to return both from every Scan, there’s no advantage to keeping them in the same table. But there are certainly some disadvantages that I’ll cover a little later. For all the other data operations (including multi-item like BatchGetItem, TransactWriteItems, etc.), DynamoDB doesn’t care if the items are in the same table or not—you’ll see the same storage/throughput metering and the same performance either way.

Sometimes, in order to bring data together into the same item collection, you need to make some adjustments to the primary key values. To be stored in the same table, the same primary key definition must be used. Examples of this include storing a numeric value as a string to match the data type of the partition key or sort key, or creating a unique value for the sort key where you didn’t necessarily have something important to store (using the same value as the partition key attribute is common for this when creating a “metadata” item in the collection).

Imagine a table for storing customer orders for shipping. The partition key is a unique numeric order identifier (data type is number), and you want to use a collection to store both a per-item record of items from the cart (identified by a numeric SKU) and tracking events (identified by a sortable UUID such as ksuid) for order processing. You want to retrieve these together for efficiency because the most common pattern is to provide a tracking page with the purchased items shown in addition to the tracking details. In this case, you’ll need to define the sort key with data type string, and convert values to store those numeric SKUs as strings. There’s a cost for this (because storing a number as a string consumes more bytes), but it’s worth it to get the benefit of the item collection.

Since it’s easy enough to accommodate these changes when writing your items into the table, a developer might as well just do it for all items and item collections, right? And put them all in the same table? Would it make sense for all DynamoDB customers to band together and store their data in one giant multi-tenant table? No—of course not. These techniques come with a cost and should only be used when they can be justified—to reap the benefits of item collections.

Closing thoughts

The valuable and valid part of “single table design” guidance is simply this:

Yep—that’s it. There really never was a need to create new terminology. “Single table design” confused some people, and sent many others down a painful, complex, costly path (details on this in an upcoming article) when it was taken too literally and then misrepresented as “best practice”. It’s time to just let this terminology go—stick with schema flexibility and item collections.

If you want to discuss this topic with me, get my thoughts on a DynamoDB data modeling question you have, or suggest topics for me to write about in future articles, please reach out to me on Twitter (@pj_naylor)—or email me directly!

Keep an eye out for a follow-up article where I’ll discuss the nuances of LSIs and GSIs and explain why “GSI overloading” is a bogus modeling pattern.

Appendix: What would I know about DynamoDB anyway?

Until recently, I worked at AWS. I started as a Technical Account Manager in 2016 within the account team that supports Amazon.com as an enterprise customer of AWS. My area of focus was helping Amazon to achieve some ambitious organizational goals: 1) migrate critical transactional database loads from traditional relational databases to distributed purpose-built databases (DynamoDB); 2) “lift and shift” second tier transactional loads from Oracle to Aurora; and 3) move all data warehousing from Oracle to Redshift. This was a fantastic experience all-around, but the part I loved most was witnessing the dramatic reduction in operational load and improvements in availability and latency that DynamoDB delivered. I worked with hundreds of Amazon.com developer teams to review their data models, and teach them about managing DynamoDB limits, scaling, alarms, and dashboards. I sat in the war rooms for peak events like Prime Day as the point of contact for any kind of DynamoDB-related concerns that emerged with rapidly growing event traffic. It wasn’t easy for everyone to make the data model paradigm shift, but when this program was complete, all those developers quickly came to take all the DynamoDB advantages for granted. They got to focus a lot more time on things that made a difference for their customers—seeing this made a deep impression on me.

‍In 2018, I became a DynamoDB specialist solutions architect—part of a small team working with the largest (external) enterprise customers of AWS. I helped them develop effective and efficient DynamoDB data models for a wide variety of use cases and access patterns, and I taught many developers how to operate well with DynamoDB. My last 18 months at AWS was spent as a product manager for the DynamoDB service. I also provided hundreds of consultations to engineers across many AWS service teams as part of an “office hours” program for DynamoDB—reviewing data models, providing broader architectural guidance, and teaching about how DynamoDB works and the insights revealed by metrics.

‍All this is to say that I’ve seen a lot of production use cases for DynamoDB (from hundreds of requests per day to millions of requests per second), and I have a strong sense for the way that developers (at Amazon.com, inside AWS, and at many other companies around the globe) are actually using DynamoDB in practice.