I recently saw this great article by Mark O’Neill, CTO at Vordel about throttling. Take a look here http://www.soatothecloud.com/2011/07/how-to-configure-throttling-on-vordel.html.

Mark assumes you have a degree of familiarity with either the Vordel Gateway.

What I’d like to do in this post is focus a little deeper on request throttling, and show how the gateway can help avoid partners and service consumers from overloading your services not just by blocking requests that exceed a certain threshold, but by giving advanced warning of erroneous excessive service consumption.

It is worth taking a few paragraphs to explain the very real value of request throttling, and perhaps to make a brief comparison with SLA management, quota management, traffic shaping and concurrent transaction processing limits.

In the context of gateways being used to virtualise services for external consumption, throttling is primarily concerned with the number of requests made by consumers to the target services. Restrictions take the form of a message count that is incremented and measured over a specific time period. The Vordel gateway provides both coarse and fine grained control of request counts and perhaps more importantly can track the request count based on virtually any index.

The throttling filter provides a variety of options for measuring and tracking request counts – the time period unit ranges from seconds, minute, hour, day, and week meaning that it can used to implement quota management as part of a pay-per-use solution.

The number of messages, period and period unit can also specified by parameter substitution – enabling complete decoupling of the throttling limits which could be managed in an external provisioning system or database.

Key to the successful implementation of a request throttling solution of enterprise scale is the ability to maintain a consistent count across multiple gateways in an HA / HA-DR deployment – achieved quite simply here by the use of the gateway’s caching mechanism.

Pay special attention to the “Track per key” setting that allows the request count to be applied to different contexts. In the screen shot above, the wildcard attribute of ${http.request.clientaddr} is being used – fairly obviously this will apply the request count restrictions based on unique IP address value of the calling client. The ability to control the “key” with attributes gives enormous flexibility, and because multiple throttling filters can be linked together in a policy (or circuit) limits can be applied to an incredibly low level. You could for example apply a global request rate limit across all services (good practice for mitigating DoS attacks). Legacy applications that have been “service enabled” may be a little fragile (having never been designed for web volumes) – it would make sense to apply a service wide limit in addition to the global restriction, but independent of the calling client – you could even be operation specific. Most of us are also familiar with the pattern that allows restricted service usage for “free”, and a tiered allowance for those signing up to the “pro” version – a good example of metered service consumption by role. Then there is the hard core application – of quota management by identity (which could be at individual user level or partner organisation).

For the record, throttling should not be confused with SLA (Service Level Agreement) monitoring and management. It may be sensible to offer an SLA within restricted bounds (of request volume) but SLA is more concerned with response times, number of failures (typically non 2xx response codes) etc…

The value here is pretty obvious, by implementing a well considered throttling strategy (which incidentally can be designed once and applied globally) you get the following

• Protection from request volume based DoS attacks
• Shielding of vulnerable servers, services (and even specific operations)
• Service Consumption Metering (billable pay per use services)
• Optional and conditional routing of excess traffic (maybe expanding into the cloud at peek times
• Pro-active non-blocking alerting that allows action to be taken before critical limits are reached with built in alert storm protection
• boundary volumes for tightening SLA agreements

Using externally set parameters means that limits can be changed an applied dynamically (or in extreme cases even scripted!!) you could for example automate the increase of a limit on receipt of payment for a particular partner / customer.

The wealth of functionality in just one filter component is impressive in its own right, and… there’s more… Mark’s blog describes a relatively simple design…. once you reach the throttling limit for the service then future requests are “rejected” until such time as the rate calculation falls below the specified threshold. By introducing a soft limit we have the capability to provide non-blocking alerts that give prior warning of an impending breech .

Working through an example….

1. Basic service throttling

The figure above shows rudimentary service throttling (at 100 requests per second is used as an example only). If the number of requests to the service exceed this, then a soap fault will be returned by the gateway and the request is not sent to the target service (blocking).

If we introduce another throttling filter (different key for tracking!) with a lower limit, we can provide an alert to an appropriate party (operations / partner administration contact / account manager or all)

2. Setting a “soft” limit

In this example I’ve chosen to set the warning level at 80 requests per second – and changed the key to track by to avoid a double increment with the hard limit filter.

At first glance, it looks a little odd – basically we don’t change the policy execution path when we exceed the SOFT limit. This will however show in trace files, real time monitoring and the new Traffic Monitor

3. Take some action on exceeding the SOFT limit

a) For demonstration purposes I’ll use a syslog alert, but it should be obvious at this point that the conditional processing that occurs as a result of a SOFT limit breach is not restricted to syslog alerting – or alerting at all! – setting up an alert

Having set up an alert target, it can now be used to flag a soft limit breach… n.b. the alert filter is found in the Monitoring category

The actual alert configuration is outside the scope of what I want to discuss here, I’ll perhaps cover it in greater detail in a different post

So from the top down …..

If the request being processed causes the 80/s count to be exceeded we follow the failure path (the red line) to the “Alert” filter – my alert is configured to send a message to syslog with details of the service soft limit breech.

In the above policy I have chosen not to put the alert on the critical path, in real terms this means that even if the alert fails then message processing will continue. Then the policy checks the HARD limit which is where we started at step 1… meaning that if the request is over the soft limit but below the hard limit then the alert is sent and the request is processed as desired, only when the hard limit is exceeded are requests blocked.

** The brighter readers of this might observe that this means you could have 20 alerts in a second for each request that is >79 but less than 100 ** essentially an alert storm. Fortunately the alerting filter provides the ability to prevent this by configuring a limit for the number of alerts that can be sent in a given time period…. the clever guys at Vordel think of almost everything