Secreta: a little suite for managing configuration secrets

After learning AWS and Lambda, in the past few weeks I wrote Secreta, which is a set of three tools to manage configuration secrets in AWS Lambda functions.

secreta-generate-aws is a command line utility that creates a pair of keys for asymmetric key encryption, using RSA. It does so by

  1. running the forge module directly in an AWS Lambda function
  2. saving the private key directly in an AWS Parameter, encrypted and protected by an access tag
  3. saving the public key to a local file

secreta-encrypt is a command line utility that encrypts to a local .secreta file the secrets referenced in your configuration files (using a public key).

secreta-decrypt-aws is a NodeJS module that you can install into your AWS Lambda function to decrypt .secreta files in memory (using the corresponding private key, retrieved from the AWS Parameter) and merge them into the rest of the configuration, as if they had never been encrypted.

You can share the public key in your project repository. This will allow any other trusted developer (like yourself) to always have a current public key to keep encrypting configuration secrets. These could be obtained by exchanging GPG email messages, for example.

You can share the .secreta files in your project repository. This will allow any other developer to always have current configuration secrets to keep deploying your AWS Lambda function.

Documentation here.

AWS Lambda Invoke Errors

Lately, I’ve been developing a Lambda function to create a pair of keys, store one in a parameter and return the other to the user. Today I got my first clean run, and I’m writing this to celebrate.

As part of learning lots of things along the way, because I hadn’t developed anything on AWS before, I found out that AWS reports errors in many different ways. Here are those that occurred to me in the last few hours.

See: AWS Lambda Invoke Errors (documentation)

Error — InvalidZipFileException

{ InvalidZipFileException: Lambda was not able to unzip the file
       <stack trace...>
     message: 'Lambda was not able to unzip the file',
     code: 'InvalidZipFileException',
     time: 2017-09-16T16:04:04.558Z,
     requestId: '<request id...>',
     statusCode: 502,
     retryable: true } }
  • This error occurred because the zip file I had uploaded was wrong: the files it contained were stored into a directory.
  • This is a Lambda creation error, and it’s quite complete.
  • Unhandled / Handled classification is limited to Lambda execution.

Error — Process exited

{ errorMessage: 'RequestId: <request id...> Process exited before completing request' }
  • This error occurred because I threw a validation error.
  • I could catch this error, but don’t, so it’s classified as Unhandled.
  • The logs show my error and its stack trace.
17:12:39 START RequestId: <request id...> Version: $LATEST
17:12:39 2017-09-16T17:12:39.442Z   <request id...>    Error: Expected a pair ID for your keys. at setPairId (/var/task/createPairOfKeys.js:9832:15) at handler (/var/task/createPairOfKeys.js:9805:9)
17:12:39 END RequestId: <request id...>
17:12:39 REPORT RequestId: <request id...> Duration: 203.72 ms Billed Duration: 300 ms Memory Size: 128 MB Max Memory Used: 78 MB
17:12:39 RequestId: <request id...> Process exited before completing request

Error — Timed out

{ errorMessage: '2017-09-16T17:32:08.746Z <request id...> Task timed out after 10.00 seconds' }
  • This error occurred because generating a pair of keys is not fast (bigint maths involved) and, in my few tests, it sometimes needed more than 40 seconds.
  • Surprisingly, my MacBook is faster than AWS Lambda. A few times it took less than 1 second, most of the times less than 6 seconds, and only once it timed out at 10 seconds.
  • I can’t catch this error, so it’s classified as Unhandled.

Error — AccessDeniedException

{ errorMessage: 'User: <user arn...> is not authorized to perform: ssm:PutParameter on resource: <resource arn...>',
        errorType: 'AccessDeniedException',
        stackTrace: [Object] } } }
  • This error occurred because the role I had assigned to my Lambda function didn’t have the right to write SSM parameters.
  • I catch this error and swallow it, so it’s classified as Handled.

Error — ParameterAlreadyExists

{ errorMessage: 'The parameter already exists. To overwrite this value, set the overwrite option in the request to true.',
        errorType: 'ParameterAlreadyExists',
        stackTrace: [Object] } } }
  • This error occurred because I explicitly create my SSM parameter with overwrite set to false. Not gonna change it, this is by design.
  • Notice how this is an exception whose name doesn’t end with Exception.
  • I catch this error and swallow it, so it’s classified as Handled.

How to terminate AWS Lambda functions

The AWS Lambda, Programming Model document states:

Your Lambda function code must be written in a stateless style, and have no affinity with the underlying compute infrastructure.

The AWS Lambda, Best Practices document recommends:

Take advantage of container re-use to improve the performance of your function.

Those contradictory statements make sense at scale.

When an AWS Lambda function completes (which means it terminates without unhandled exceptions, either with an error (also referred to as a failure) or a result (also referred to as a success)), the AWS Lambda service will automatically freeze its container and probably reuse it for a new invocation.

When an AWS Lambda function takes longer than the specified maximum execution time, the AWS Lambda service times it out by throwing an exception (unhandled by definition) and then manages it accordingly to some policies.

Given that AWS Lambda functions for a NodeJS runtime are NodeJS applications that, in general, add callbacks to the event loop, the AWS Lambda service will, by default, make sure that there is no pending work before considering the AWS Lambda function fully completed, independently from the fact that its callback was already called or not. To call the callback is, in fact, optional.

Unfortunately, if the AWS Lambda function somehow blocks the event loop (for example, a database connection which is being periodically polled), the AWS Lambda service won’t ever find the event loop empty and the AWS Lambda function will time out.

NodeJS V8 has an experimental Async Hooks module to help debug these situations, while V6 (used by AWS Lambda) has these two undocumented functions:

  • process._getActiveHandles() gets you handles that are still alive
  • process._getActiveRequests() gets you info about active libuv requests.

Conveniently, the AWS Lambda service calls the exports.handler function with a second argument set to the container context. In turn the context has a callbackWaitsForEmptyEventLoop property which can be set to false to signal to the AWS Lambda service that the callback call (in a way) marks the end of the AWS Lambda function.

In reality:

AWS Lambda will freeze the process, any state data and the events in the NodeJS event loop (any remaining events in the event loop [will be] processed when the Lambda function is called next and if AWS Lambda chooses to use the frozen process).

let expensiveResult;

exports.handler = (event, context, callback) => {
    endWhenTheCallbackIsCalledOrTheTimeoutIsReached(context);

    process(event, callback);
};

// ---

function process(event, done) {
    try {
        if (! expensiveResult) {
            // set expensiveResult
        }
        // use expensiveResult
        // then
        done(null, result);
    } catch (e) {
        done(JSON.stringify(e));
    }
}

function endWhenTheCallbackIsCalledOrTheTimeoutIsReached(context) {
    context.callbackWaitsForEmptyEventLoop = false;
}

function endWhenTheEventLoopIsEmptyOrTheTimeoutIsReached(context) {
    context.callbackWaitsForEmptyEventLoop = true;
}