In Part I of this blog we tested the performance of a Hello World example for 8 different runtimes and got us some very interesting metrics. However, we didn’t stop there. We wanted to simulate a use case common in many applications — CRUD (Create Read Update Delete) operations in a NoSQL database like AWS DynamoDB.

As a recap of Part I, we found that:

• Cold-start time and memory usage vary depending on the runtime
• AWS Lambda execution durations are stable
• API Gateway and AWS Lambda are reliable, no errors
• API Gateway latency accounts for more than just code execution in AWS Lambda
• Package size isn’t always an indicator for faster cold-start, demonstrated by Golang and Ruby
• Python and Node.js have great performance

Benchmarking process

In this case, we reduced the number of runtimes we benchmarked down to 5, Python 3.6, Node.js 8.10, Java 8, C# (.NET 2.1), and Haskell.

For each of these runtimes, we built CRUD operations with products, containing SKU, name and description. We also created a DynamoDB table with autoscaling enabled; the stack was deployed using Serverless Framework.

For testing, we continued to use Serverless Artillery, but created a different script that generated a random SKU, name, and description for each of the requests and then assigned those values to the following sequential HTTP requests:

Create > List > Update > Get > Delete

With the above flow, we could test all the functions and empty the table when the test was completed, win win 😃

Once again, we used AWS CloudWatch to extract all the metrics we were interested in.

You can visit our repository to see the project we created for benchmarking.

Lambda Pre-requisites

AWS Lambda functions interact with other AWS services or APIs through libraries. In this case, we needed a library to serialize and deserialize JSON inputs and outputs, a library for date/time, and another library to interact with AWS DynamoDB. Official AWS SDKs are available for C#, Python, Node.js, and Java and non-official AWS SDKs for Haskell such as Amazonka or AWS SDK.

CRUD — Benchmark Result

For this benchmark, similar to the procedure we used in Part I, we performed 20 requests per second (rps) for a period of 1200 seconds. Making a total of 120,000 requests per runtime.

Cold-start

We were expecting an increase in the cold-start due to the addition of dependencies. However, we were surprised by the dramatic increase for Python and Node.js, by around 30x and 35x respectively in the worst case scenario.

As can be seen in the above table, Java continues to be the slowest to warm up, followed by C# and Haskell.

Execution durations

Once we get over the cold-start period, we start seeing very stable execution times for C#, Python, and Java. However, we got some spikes in Haskell and Node.js for all the CRUD operations that were performed.

Create Product— We can see some changes in the benchmark when creating items in DynamoDB tables in comparison to Part I. C# takes the lead with an average execution duration that’s almost 1 ms faster than Python and Java, and 2x faster than Node.js. Additionally, we can see a maximum duration of 6 seconds in Haskell, which is where our timeout was set.

List Products — The ranking of results doesn’t change for listing all items in a table; as in Part I, C# is the best performer. The execution durations increased for all the runtimes due to the scan on DynamoDB tables, which requires the entire table to be read before a result is returned. Furthermore, scan operations are limited to a 1 MB payload, which means that if there is more than 1 MB of data to be retrieved, it will be paginated through a nextToken key.

Early in the graph below, you can see another timeout in Haskell, when executing for more than 6 seconds.

Get Product — The trend continues for reading items from DynamoDB. C# is the fastest, closely followed by Python and Java. Moreover, we observe a lower average execution duration for all the runtimes, between 1 and 2 ms, in comparison to the write operation.

Update Product — Python has the fastest average execution duration when updating items in DynamoDB, with C# just 0.48 ms behind. In this operation, we need to specify the partition key and attributes to be updated.

Delete Product — The impressive execution durations of C# and Python are reflected one more time in the delete item operation from DynamoDB. Spikes of around 2 seconds in Node.js and Haskell continue, but no timeout errors were recorded.

CRUD — Results overview

It is apparent for all the CRUD operations above that Node.js and Haskell are being penalized in the average execution duration due to a really high delta between the maximum and minimum values. This negative impact is more noticeable in Node.js, where the minimum execution duration is usually no more than 2 ms away from the best performers. Furthermore, according to the data displayed above, C#, Python, and Java have similar durations if we don’t take into consideration cold-starts.

The single digit execution durations of C#, Python, Node.js, and Java are great, bearing in mind that a NoSQL database is being accessed. On the other hand, Haskell, in spite of being slower than other runtimes, has acceptable execution durations that we are very confident will improve in the future.

It can also be seen that Create, Get, Update and Delete operations have similar performance, while the List operation has a slightly slower performance, around a 20 percent difference, due to the scan on the whole table.

Memory usage

Memory usage increases when performing AWS DynamoDB reads or writes. However, it is still below 128 MB for all the runtimes except Java, which demanded a maximum of 147 MB in some of the requests.

Errors

We performed a total of 720,000 requests against 25 different functions, 5 functions per runtime, and we only registered 4 errors in Haskell functions, which were caused by timeouts.

Package vs max cold-start

• Python 5.1 KB 118 ms
• Node.js 6.0 KB 201 ms
• C# 633.5 KB 784 ms
• Haskell 5.8 MB 1300 ms
• Java 10.1 MB 4050 ms

For the runtimes analyzed in this benchmark, we can see that the ones with bigger packages also took longer to initialize. In Part I, we saw that Ruby and Go were the two exceptions to this norm.

API Gateway latency

We continue to be interested in the full duration of the request from the point in which API Gateway receives the request until a response is returned to the client.

Including cold-start, Python is the only runtime with a maximum API Gateway average latency below 1 second — to be more precise, 635 ms. If we subtract the worst cold-start execution duration of 118 ms, we end up with around half a second in which AWS is getting the VMs ready to execute the code.

Note: The data below has been measured without cold-starts

We see some interesting results from C#, Python, and Java: their average latency is 20 to 30 percent faster than Node.js and nearly 10x faster than Haskell. Additionally, the minimum latency experienced is 14 ms, which hints at the best performance that could be achieved.

As we mentioned before, we have to add the time that it takes to transfer data from and to the AWS region, i.e. us-east-1, to the above latencies. This latency could be measured here.

Conclusion

We’re happy with the results and we feel that there’s good potential for improving not only AWS Lambda but also the synergy with AWS API Gateway, for example, when provisioning VMs to execute the code. We will continue to monitor how the different runtimes evolve in this rapidly changing environment.

We found Python’s performance impressive, and the results from C# after cold-start even more impressive. We feel that Node.js and Haskell runtimes will need to continue improving, firstly, by finding more stability during executions, to avoid the spikes we saw in the graphs, and second, by reducing the execution time to be more aligned with the top performers. Additionally, in our benchmark, Haskell is using Amazonka as the AWS SDK. From the performance of the Haskell Hello World example, it seems that some improvements can be made in the non-official AWS SDK in order to reduce the duration when interacting with DynamoDB.

We also believe that there is some more work to be done to improve Java and C# cold-starts, which will make them more appealing when latencies are highly important at any given point in time.

AWS Lambda and AWS API Gateway continue to be highly reliable: only 4 timeout errors were experienced in Haskell out of a total of 600,000 requests to the 5 different runtimes.

To sum up, we are very happy with the results that we’ve obtained for the runtimes analyzed and especially proud of the Haskell runtime, which we will continue working on to achieve great performance. Moreover, we encourage the community to contribute to it and look forward to your ideas.

Thanks to Carlos Domínguez Padrón for working on this with me, and Annie Anderson and to the rest of The Agile Monkeys for proofreading and giving feedback prior to publishing this post. Special thanks to Javier Toledo.

Feedback from everyone else interested in the topic is welcome, I am open to discussion :D

If you have questions, or would like to work with us, please contact us www.theagilemonkeys.com