How we built efficient load generator in 6 hrs ?
When you lack resources you tend to become creative. My manager recently assigned our team with a task where we had to load test our newly implemented functionalities. We replied …Cool, consider it done! But wait you have just one day to scale test our API’s …figure out our latency, …capacity required to support requests , identify slow performing queries , check fairness of our system .. else we are not going to deliver quality product on time :(
Sure we got this Boss !!…Hmm .. This is gonna be fun !
Ok! Let’s break this down. So here is what we need to do
- Test our new API’s with various loads (traffic) to get a sense of how much capacity we need at different times.
- Load test our slow performing queries, benchmark it and figure out what are the different bottlenecks
- Test our workflows to verify how are workers are picking up new requests are we treating all customers equally ?
- We don’t want to burn our dollars, solution should be cost effective.
- We should have knobs to test at various loads .. so we need some type of configuration.
Thinking .. Thinking .. Thinking ….Viola ! We can trigger all the above with AWS lambda and it scales magically for us at dirt cheap !
Here is the architecture with our team came up with ( yeah i know i work with some awesome folks). Basic idea is an event emitting lambda will publish messages to SNS which will be subscribed by appropriate load test functions and trigger the actual functionality we want to test.
We have configured a cloudwatch event with a cron expression rate(1 minute) which triggers “EventEmitter” lambda.
In the “EventEmitter” environment variables we have configured SNS topic arns and number of events we want to publish
In the event emitter we have written function which will emit required number of events to appropriate topic ARN’s . Here is the pseudo code
Woah ! Who writes such dumb code ? Haha .. remember this is pseudo code !!!
So the above lambda is all set to emit required number of events to configured SNS topics. But wait we have a limitation on the number of lambdas which we can have on AWS account right ?
Yep ! Multithreading to the rescue. We used command pattern to implement our individual load triggering lambda function where API load testing lambda implements ApiCallingCommand , Query load testing lambda implements QueryExecutingCommand, Workflow load testing lambda implements WorkflowTriggeringCommand. These commands are passed to command executor which looks something like this.
Yahoo ! now effectively we have the capability of generating load of
TOTAL LOAD = EVENT_COUNT * THREAD_COUNT * BATCH (If possible for example BatchWrite) Also apart from the executor we also implemented a small decorator which captured the metrics we want and emit to cloudwatch to see how we are performing with various loads.
Okay Boss ! Let us know with how much load you want us to test with ?? Got numbers, did the math, configured required number of event counts in environment variables, thread counts, enabled cloudwatch event to run it overnight, slept, got numbers in the morning, fixed small performance bugs and we shipped yay!!!!
Still not impressed ???? :(
References
}