(Erratum) VPC Endpoint increases DynamoDB latency by 30%

Our reader Tom wrote in to tell me, that the latency for read requests to DynamoDB increased significantly after enabling a VPC endpoint a few weeks ago. Someone else reported a similar problem in the AWS discussion forums as well. Therefore, I started to investigate to write this article. After many hours of benchmarking, I came to the conclusion that using a VPC endpoint to connect to DynamoDB increase latency by 30% compared to connections through a internet gateway or NAT gateway. Shortly, after publishing the article, Petar send me a message via Twitter to tell me that something must be wrong with my benchmark. Unfortunatly, that was correct. I made a mistake when measuring the read latency for DynamoDB from EC2.

Therefore, I have depublished the original blog post. Instead you will find a post mortem analysis in the following.

/images/2020/08/snail.jpg

The problem starts with Node.js and the AWS SDK. I was using the following script to measure the read latency for DynamoDB.

const AWS = require('aws-sdk')
const dynamodb = new AWS.DynamoDB({apiVersion: '2012-08-10'});
AWS.config.maxRetries = 0;

async function benchmark() {
  for (let index = 0; index < 10000; index++) {
    let start = new Date().getTime();
    let result = await dynamodb.getItem({
      Key: {
       "id": {
         S: "1"
        }
      },
      TableName: "benchmark"
    }).promise();
    let end = new Date().getTime();
    console.log(`${index},${end-start}`);
    await new Promise(r => setTimeout(r, 50));
  }
}

benchmark();

With that code, requests through a VPC endpoint took about 10 ms - about 30% - longer than when connecting through an Internet gateway.

The problem with that Node.js code? Each DynamoDB API request create a new TCP connection. Doing so adds latency, and it seems like establishing a new TCP connection is taking about 30% longer when using a VPC endpoint.

However, you should not use the Node.js default in production. Instead, you need to tell the AWS SDK to reuse exsisting connections. To do so, I modified my code as described in Reusing Connections with Keep-Alive in Node.js.

const AWS = require('aws-sdk')
const https = require('https');
const agent = new https.Agent({
  keepAlive: true
});
const dynamodb = new AWS.DynamoDB({httpOptions: {agent}});
AWS.config.maxRetries = 0;

async function benchmark() {
  for (let index = 0; index < 10000; index++) {
    let start = process.hrtime.bigint();
    let result = await dynamodb.getItem({
      Key: {
       "id": {
         S: "1"
        }
      },
      TableName: "benchmark"
    }).promise();
    let end = process.hrtime.bigint();
    console.log(`${index},${end-start}`);
    await new Promise(r => setTimeout(r, 50));
  }
}

benchmark();

With that modification I could no longer measure any significant differences between connecting to DynamoDB through an Internet gateway, NAT gateway, or VPC endpoint.

I’m sorry about my mistake. Michael and I are working hard on publishing high-quality content.

View the full article