Multi Lambda Development Workflow With Webpack

Ideally, a lambda is the glue that ties together a microservice that it’s developed in isolation from other parts of a system and it’s stored and managed in its own repository. Shared functionality between lambdas, if any, might be exposed and consumed as npm packages for further isolation.

For smaller systems, a lambda in combination with an API Gateway to expose it through a REST API, is the whole microservice. To enforce complete isolation through npm packages and independent repositories might be overkill. In this scenario a lambda is used mostly as a means to achieve instant scalability at a low cost. How should then a small team develop and manage the lambdas that their application needs?

Monorepo

The most straightforward simplification is to store all the lambdas, along with any shared functionality, in a single repository (monorepo). Because each lambda will have its own dependencies, will be deployed independently to AWS as they are logically independent resources and I’m writing them in Typescript, my first attempt was to use npm workspaces.

After trying for a while I abandoned this approach because workspaces are optimized for javascript apps that have multiple npm packages that the author wants to publish independently to npm for other apps to consume. My lambdas on the other hand are never going to be published or consumed from npm. They are published to AWS and consumed through a REST API or the AWS SDK.

I found an alternative method that at its core uses webpack to have multiple lambdas written in Typescript, with unit tests also written in Typescript and that can be published to AWS independently with only the dependencies they need and nothing more. This last part is important as AWS imposes a limit on the size of the lambda to 50 MB for the zip file to be uploaded and 250 MB for the uncompressed folder.

Project Structure

Below is a sample project structure of a system written in typescript that has two lambdas and those lambdas have some shared functionality. When compiled, the dist folder will have a javascript file for each lambda bundled with all its dependencies ready to be pushed to AWS as a zip file. The function handler of each lambda will be exposed as a commonjs module so AWS can use it.

.
├── dist
│   ├── lambda-a
│   │   └── index.js
│   └── lambda-b
│       └── index.js
└── src
    ├── lambda-a
    │   ├── index.test.ts
    │   └── index.ts
    ├── lambda-b
    │   ├── index.test.ts
    │   └── index.ts
    └── shared
        └── utils.ts

Below is the content of one lambda (lambda-a), it’s unit test and the shared functionality for context. The other lambda is very similar and it’s there just to show how to handle multiple lambdas in one project. You can see the complete code structure and their files content in the Github repo.

src/lambda-a/index.ts

import { APIGatewayProxyResult } from "aws-lambda";
import { createResponse } from "../shared/utils";

export const handler = async (): Promise<APIGatewayProxyResult> => {
  const name = "David";
  return createResponse(200, name);
};

src/lambda-a/index.test.ts

import { handler } from "./index";

describe("handler", () => {
  it('should return "Hello DAVID"', async () => {
    const response = await handler();
    expect(response.body).toBe("Hello DAVID");
  });
});

src/shared/utils.ts

import { APIGatewayProxyResult } from "aws-lambda";
import { toUpper } from "ramda";

export const createResponse = (
  code: number,
  name: string
): APIGatewayProxyResult => ({
  statusCode: code,
  body: `Hello ${toUpper(name)}`,
});

Configuration Files

Now that we know the file structure and the sample code we can provide all the required configuration to make our project work. 

NPM

As usual we start by creating a package.json file.

$ npm init -y

The default configuration file created by the init command is shown below.

package.json

{
  "name": "lambdas-webpack",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {},
  "dependencies": {}
}

Because our project is not meant to be published to npm we can add set the private property to true and remove some of the boilerplate that doesn’t apply to us. The modified config file is shown below.

package.json

{
  "name": "lambdas-webpack",
  "private": true,
  "license": "MIT",
  "scripts": {},
  "devDependencies": {},
  "dependencies": {}
}

We can then proceed to install all of the project dev dependencies. In a nutshell we will need webpack, typescript, jest and relevant typescript types.

$ npm install -D typescript webpack webpack-cli clean-webpack-plugin ts-loader prettier jest ts-jest @types/{jest,node,aws-lambda,ramda}

For this example project the only runtime dependency is ramda.

$ npm install ramda

Typescript

Next we have to configure typescript for our project and we start by creating a configuration file. Make sure to use the locally installed version of tsc by using npx instead of relying on a global installation of typescript that might be outdated.

$ npx tsc --init

The default typescript configuration file is shown below.

tsconfig.json

{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true
  }
}

Because on AWS I configured my lambdas to use the Node 14 runtime I can safely assume that it supports the latest ECMAScript standard available when it was released in 2020. I’ll then change the target from ES5 to ES2020 and make sure the corresponding libraries for that standard are also loaded and available to typescript using the lib property.

tsconfig.json

{
  "compilerOptions": {
    "target": "ES2020",
    "lib": ["ES2020"],
    "module": "commonjs",
    "esModuleInterop": true,
    "forceConsistentCasingInFileNames": true,
    "strict": true,
    "skipLibCheck": true
  }
}

Jest

Because we want to write our unit tests in typescript without having to compile them to javascript we have to configure jest to use the ts-jest preset. For this we have to create a configuration file provided by ts-jest.

$ npx ts-jest config:init

This command creates the following config file at the root of our project.

jest.config.js

/** @type {import('ts-jest/dist/types').InitialOptionsTsJest} */
module.exports = {
  preset: 'ts-jest',
  testEnvironment: 'node',
};

Webpack

This is the heart of our configuration and we will have to create webpack’s configuration file by hand as I’m not aware of a simple way to create this file using a cli.

webpack.config.js

const path = require("path");
const { CleanWebpackPlugin } = require("clean-webpack-plugin");

module.exports = {
  entry: {
    ["lambda-a"]: "./src/lambda-a/index.ts",
    ["lambda-b"]: "./src/lambda-b/index.ts",
  },
  target: "node",
  output: {
    path: path.resolve(__dirname, "dist"),
    filename: "[name]/index.js",
    library: {
      type: "commonjs",
    },
  },
  module: {
    rules: [
      {
        test: /\.ts$/,
        use: ["ts-loader"],
        exclude: /node_modules/,
      },
    ],
  },
  resolve: {
    extensions: [".ts", ".js", ".json"],
  },
  mode: "production",
  plugins: [new CleanWebpackPlugin()],
};

There’s a lot to unpack here so let’s go bit by bit. First we need to pass an object to the entry property (line 5) as we want to create one bundle per lambda and for each lambda we need to inform webpack where the entry file is. 

The keys of entry object are used as part of the property output.filename (line 12) as the replacement for the token [name]. This way we will have a different path for each lambda within the dist folder as shown below.

dist
├── lambda-a
│   └── index.js
└── lambda-b
    └── index.js

Because webpack can be used to bundle files to be used in a browser (js) or in a server (node) we have to let it know that our lambdas are meant to be used in a server setting the property target to node (line 9).

Also, because AWS expects our handler function from the lambda file to be exported using the commonjs module system, we have to instruct webpack to do so with the property output.library.type set to commonjs (line 14).

Finally to allow for tree shaking and to minimize our bundle as much as possible to avoid hitting the AWS limits on lambdas file size, we set the property mode to production (line 29).

Scripts

The final step is to create all the npm scripts that we need to test, bundle, zip and deploy our lambdas to AWS.

Test and bundle are straightforward as they simply use their corresponding libraries without any extra configuration beyond what their configuration file holds.

package.json

{
  ...
  "scripts": {
    "test": "jest",
    "bundle": "webpack",
    ...
  },
  ...
}

Next we need to be able to create a zip file for each lambda to then upload it to AWS.

package.json

{
  ...
  "scripts": {
    ...
    "zip:lambda-a": "cd dist/lambda-a && zip index.zip index.js",
    "zip:lambda-b": "cd dist/lambda-b && zip index.zip index.js",
    "zip:all": "npm run zip:lambda-a && npm run zip:lambda-b",
  },
  ...
}

To avoid duplicating the command for every lambda in our system, we can make use of environmental variables to make it generic.

package.json

{
  ...
  "scripts": {
    ...
    "zip:lambda": "cd dist/${LAMBDA} && zip index.zip index.js",
    "zip:all": "LAMBDA=lambda-a npm run zip:lambda && LAMBDA=lambda-b npm run zip:lambda",
  },
  ...
}

Now if you want to run the command zip:lambda directly from the command line you will need to provide a value for the environmental variable LAMBDA.

$ LAMBDA=lambda-a npm run zip:lambda

Next we need to define the scripts to upload and update our lambda code on AWS.

package.json

{
  ...
  "scripts": {
    ...
    "update:lambda-a": "aws lambda update-function-code --function-name lambda-a --zip-file fileb://dist/lambda-a/index.zip --no-cli-pager",
    "update:lambda-b": "aws lambda update-function-code --function-name lambda-b --zip-file fileb://dist/lambda-b/index.zip --no-cli-pager",
    "update:all": "npm run update:lambda-a && npm run update:lambda-b",
    ...
  },
  ...
}

We can do the same trick as before and create a generic script with the same environmental variable.

package.json

{
  ...
  "scripts": {
    ...
    "update:lambda": "aws lambda update-function-code --function-name ${LAMBDA} --zip-file fileb://dist/${LAMBDA}/index.zip --no-cli-pager",
    "update:all": "LAMBDA=lambda-a npm run update:lambda && LAMBDA=lambda-b npm run update:lambda",
    ...
  },
  ...
}

We can tie it all together with a deploy script that creates the bundle, the zip file and upload the function to AWS.

package.json

{
  ...
  "scripts": {
    ...
    "deploy:lambda-a": "npm run bundle && npm run zip:lambda-a && npm run update:lambda-a",
    "deploy:lambda-b": "npm run bundle && npm run zip:lambda-b && npm run update:lambda-b",
    "deploy:all": "npm run bundle && npm run zip:all && npm run update:all"
  },
  ...
}

Using the same environmental variable trick we get:

package.json

{
  ...
  "scripts": {
    ...
    "deploy:lambda": "npm run bundle && npm run zip:lambda && npm run update:lambda",
    "deploy:all": "npm run bundle && npm run zip:all && npm run update:all"
  },
  ...
}

Now, if we make a modification on a single lambda and we want to update just that lambda on AWS we can run the command:

$ LAMBDA=<lambda-name> npm run deploy:lambda

But if we modify more than one lambda and we want to update them all at the same time we can do:

$ npm run deploy:all

Conclusion

In this article I have shown you one way to set up a monorepo that allows you to manage multiple lambdas in a single place with the ability to publish them independently or in tandem. This is of course not the only way or even the best way to do it, but it’s a workflow that works well for me.

Granted, I’m more familiar with frontend tools so the choice of webpack might seem odd for a backend developer but it does the trick. This could be one of those cases where the saying “If you only have a hammer everything looks like a nail” applies. 
If there’s a simpler way to achieve this please let me know in the comments section. If you want to see the source code head over to the Github repo.