Object Store v2 – rate limiting may be a problem

Last Updated on 30/05/2021 by Patryk Bandurski

Lately, I had to design the solution that would store errors next to the application on CloudHub in Object Store v2. However, the problem arouses when the object store started to contain more than one thousand entries. Eventually, the solution began to fail reads and other operations. What has happened? In this article, you will see what rate limiting does to the Object Store v2 and how to work around that.

Case

My application should collect daily error messages in a persistent Object Store. I should be able, once a day, easily retrieve all entries and generate a single report than clear the Object Store. My application uses Mule 4 Runtime, and it uses by default Object Store version 2.

Object Store can be accessed via Object Store connector and Object Store v2 REST API

In my case, I used the Object Store connector for the simplicity reasons.

Something goes wrong

Everything seemed to works as expected. One day I have noticed that my application throws some bizarre errors.

There was an error while retrieving keys from store APP_persistent-object-store-v2__defaultPersistentObjectStore, status code was 429, response was StatusResponse{status=failure, statusMessage=ip 3.16.69.227 can not perform more requests in this time window, stackTrace=null}

Error retrieved from the Object Store v2

I looked at Object Store and found out that my partition contains around two thousand records in it. It worked like a charm when I had approximately six hundred entries.

Object Store connector

What does the MuleSoft say?

On the MuleSoft web page, you can find the FAQ for Object Store v2. There is a section Are there rate limits on Object Store v2?. Based on this you can find out that Object Store v2 has a separate license that can be obtained. You may send 10 requests per second per deploy application.

Object Store subscription

To verify your subscription you need to log to the Anypoint Platform. In Access Management under Subscription, you will find Object Store.

As you can see on the screenshot below I have API Request Allowed 0.0. This means that I have a free tier assigned. Furthermore, there is a diagram showing you the usage.

According to MuleSoft support, you may purchase up to 100 transactions per second per app. This means up to ten times than the free tier.

You may send 10 requests per second per deploy application in a free tier.

Affected connector operations

I have used the Object Store connector in my application. We may divide its operations into two main categories. Operation related to only single item and operation working on a collection of items.

Manipulating a single item:

• Contains
• Remove
• Retrieve
• Store

Manipulating a collection:

• Clear
• Retrieve All
• Retrieve All Keys
• Store

Each operation is affected by the rate-limiting policy.

Workarounds

So what are the options to retrieve all the records from the Object Store?

Retrieve all

The first, most obvious solution would be to use Retrieve All operation. According to the documentation, it retrieves all the key-value pairs.

Unfortunately, it won’t work as we expect it to. We will receive 429 error indicating that we have faced the calls’ limit. I guess that behind the scene this operation calls records one by one. Not as a single transaction.

Retrieve all keys and Retrieve

The next logical step would be to get all the keys by using Retrieve all keys operation. For each retrieved key using Retrieve operation, we would be able to retrieve the value.

Keys’ retrieval works well even though a large amount of keys is returned. However, at some point, the For Each Scope will rethrow an error 429 because too many Retrieve operations will be invoked.

Until successful

We can make a small improvement in the solution from the previous step. We may add Until Successful Scope like in the picture below.

Although it works, it has degraded performance. Once the limit is breached the application needs to wait.

Object Store v2 REST API

Once I resolve the issue with Object Store REST API I will write a separate article. This may be a reasonable substitution to Object Store connector as an operation to retrieve Object Store items is pageable.

Summary

In short, there are some limitations on Object Store v2 but we have a way to work around it or buy a proper subscription. Object Store connector in comparison to Object Store REST API doesn’t have pagination and we need to retrieve all or nothing. Rest API may come to the rescue, but about it in the next article.