Circuit Breaker Module

Last Updated on 30/05/2021 by Patryk Bandurski

I guess that we more often think about our services in “happy path” term, but we underestimate not so happy paths. What if my API is trying to reach another API/system and it is failing constantly? Can I reach it less often in case of timeout? I thought about that in case of a Mule 4 application and I come up with Circuit Breaker pattern. In this article, I will briefly introduce the pattern and show you how you use it in your Mule application.

Circuit Breaker Pattern

Circuit Breaker monitors API calls. When everything is working as expected it is in the state closed. When the number of fails, like timeout, reach a specified threshold, Circuit Breaker will stop processing further requests. We call it the open state. As a result, API clients will receive instant information that something went wrong without waiting for the timeout to come.

The Circuit is opened for a specified period of time. After timeout occurs, the circuit breaker goes into the half-opened state. Next, the API call will hit the external system/API. After that, the circuit will decide whether to close or open itself.

You can find all states and flow in the diagram below (from Martin Fowler’s site).

Custom Mule 4 module

Mule does not have such functionality. As a result, we need either create it within the flow or as a custom module. I have decided to write a separate module that I would be able to reuse and share with you :). In order to do it, I will use the new Mule SDK.

I have decided to use this functionality to introduce the Circuit Breaker pattern in my flows. How I have prepared Circuit Breaker module I will describe in one of my next articles. Stay tuned!

Design

My module is completely independent of other Mule components. Therefore I have prepared two possible operations:

• filter – to stop processing in case of opened circuit
• record failure – to increase the counter of failures

Apart from that, we have a global configuration that configures circuit breaker behavior.

Global configuration

In order to control when circuit breaker state changes we need to have defined:

• threshold – how many failers mean that the circuit is opened
• timeout [ms] – how many time need to pass before the circuit will be in the closed state

We can introduce one circuit breaker for all HTTP requests or many. That depends on the scenario. For example, when our process API calls two system APIs we may have two different circuits for them.

To differentiate the breaker we will use breaker name property.

You may ask where the state will be kept? The module uses the default persistent Object Store instance. In the future, I plan to separate that logic and give the user the possibility to assign separately defined Object Store.

Filter

This operation blocks execution of any further component placed next to it. In case of an opened circuit breaker, it won’t allow executing external calls by throwing an error OpenedCircuitException.

Filter pass the event to the next event processor only if the circuit breaker is closed. It evaluates whether opened circuit should be closed and pass the traffic further.

Record failure

So far we only change status from open to closed. How to detect if it should be made in the opposite direction?

We need to catch errors thrown by the external call like HTTP Requestor. Our error strategy should use Record failure operation in order to increase the counter of failures.

When the threshold will be met – the counter of failures will be equal or greater than the threshold number from global configuration – the circuit state will be moved to open state.

Module installation

You may use the Circuit Breaker module prepared by me. It is available to download from GitHub.

After you download the project you need to install it in your local maven repository using maven mvn clean install command. Next, you make a reference to the module by adding the following dependency:

<groupId>pl.profit-online.extensions</groupId>
<artifactId>circuit-breaker</artifactId>
<version>1.0.0</version>
<classifier>mule-plugin</classifier>

After a couple of seconds, you should see a new module available on the Mule Palette – like in the picture below.

Circuit Breaker module usage

I have prepared two simple applications depicting Circuit Breaker module usage. You can find it here on the GitHub.

Although in the application I have hardcoded the failure of POST /consents in the Consent-MongoDB-SAPI, you may imagine the success path as below. The caller should receive in a reasonable time a response – let’s say 200 milliseconds.

However, the application is designed to fall. In that case, our process API will call POST /consents but will receive an error. Then that error will be propagated to the caller. As you may imagine, in case of timeouts it maybe 30 seconds or even more.

It could annoy our caller especially if he/she would like to try to make a call a couple of times. However, we have a Circuit Breaker Module embraced. After 5th error, the Circuit will be moved to the opened state and any further call, made by the caller, will be rejected fast without calling failing system API.

We give time the System API to rest before handling new requests.

Below you can find sample filter and record failure usage for one of the operations. In error strategy, we react on all errors and record failures. However, you may pick specific errors like Timeout, etc.

Summary

When I was working with microservices using Spring Boot framework I really liked the idea of Netflix Hystrix Circuit Breaker library. It made it easy to use that pattern in your own code.

My work experience tells me that in Mule happens situations that you would like to use the same pattern in your code. Although it is not available out of the box you may implement such behavior.

Why use Circuit Breaker pattern? In order to shorten the fail response time and give a time an endpoint service to rest and go back into an operational state.

You may use the implementation prepared by me, make it better or give me some comments so I could make it better. Let’s build it for the community :).

I encourage you to leave here a comment. Have a great day!