Protecting your services with a simple fuse..
Michael Nygard, in his book Release It! Design and Deploy Production-Ready Software describes a pattern he called Circuit Breaker. It is based on the idea of fuses, that is, anything that may be dangerous should be put around a safe structure, that may disable the operation requests if it has any chance to do any harm to the application itself or others. It is best described through the image (click to enlarge):
The flow is like that: the dangerous operation has the fuse as a shell. The first state, “closed fuse”, has a counter of failed invocations and a threshold. When the client produces an invocation, it allows the invocation to pass through. If the call succeeds, then resets the counter. If it fails, increment the counter. If the counter reaches the threshold, it disables the fuse, going to the “open fuse” state. This state has a variable that represents an ammount of time and another one representing the moment in time when it has become the active state. Any call to the dangerous operation in this state will cause it to fail without even invoking the operation. When it is in this state for the amount of time specified in the variable, it decides that the call deserves another chance. So the invocation goes to the “half open fuse”. This state tries to invoke the operation again. If it fails, go to the open state again, resetting the timer. If it succeeds, go to the closed state again, resetting the counter of failed invocations.
OK, nice pattern, but what does it have to do with SOA?
The magic in this pattern (and the whole book) is that it brings light to subjects that most developers don´t give enough attention. One of these subjects is that you cannot rely on the network. Final. I have never seen any thrustable network (and I believe you haven´t, too!), so, as long as SOA is a kind of distributed architecture that relies on the network and we cannot rely on the network, so we can´t rely on services either! So, as long as services are unthrustable, we can apply this pattern, to:
- Ensure that we won´t get stuck waiting for services that might never return;
- Ensure that, if the server that is holding the web service is drowning from lots of invocations, at least we are not the ones that are gonna disable it for good;
- And many many other good reasons to do so. Read the book 😉
As Michael himself doesn´t give any hints on the implementation of such a pattern, I decided to implement it, and you can download it from the downloads section. It is very simple, as it doesn´t allow only web services to be invoked via the pattern, but any other kind of dangerous operation too. You can modify the code the way you want to achieve your desire. My hint is that, allied to stuff like AOP and interceptors in general, you may do it the ultimate solution to never, ever have this kind of problem again.