15 minute read

Why?

Since early days of AWS Lambda I always wanted to have some ways to test functions before uploading code. Ideal case is when local development environment allows that. Of course, it is possible to do some partial code execution as part of unit/integration test setup, but that won’t be ideal close to production setup. Moreover, using java8 runtime environment requires all dependencies to be bundled together, thus resulting upload bundle blows up enormously and it might take some time to upload it. This scenario is less than perfect, mainly because for every test of function in the AWS Lambda environment there is a significant overhead (upload file, execute function, wait for logs to appear, read logs and etc.). Goal of this post is to introduce existing solution allowing local AWS Lambda execution and provide working example.

Configuration

In order to execute functions locally we will need the following tools installed:

To implement example functions we will be using Kotlin. Actual execution will use java8 runtime environment.

Implement and run first function

Create a directory you’d like to use for the example code and put build.gradle file there with the following content:

buildscript {
    ext.kotlin_version = '1.2.30'
}

plugins {
    id 'org.jetbrains.kotlin.jvm' version '1.2.30'
}

apply plugin: 'java'
apply plugin: 'kotlin'
apply plugin: 'idea'

repositories {
    mavenCentral()
    jcenter()
}

dependencies {
    compile(
            'org.jetbrains.kotlin:kotlin-stdlib-jdk8:1.2.30',
            'org.jetbrains.kotlin:kotlin-reflect:1.2.30',
            'org.jetbrains.kotlinx:kotlinx-coroutines-core:0.22.5',
            'com.fasterxml.jackson.module:jackson-module-kotlin:2.9.3',
            'com.amazonaws:aws-lambda-java-core:1.1.0',
            'com.amazonaws:aws-lambda-java-events:1.1.0',
            'com.amazonaws:aws-java-sdk:1.11.275'
    )
}

kotlin {
    experimental {
        coroutines "enable"
    }
}

compileKotlin {
    kotlinOptions {
        jvmTarget = "1.8"
    }
}

compileTestKotlin {
    kotlinOptions {
        jvmTarget = "1.8"
    }
}

idea {
    module {
        downloadJavadoc = true
        downloadSources = true
    }
}

task buildZip(type: Zip) {
    from compileJava
    from compileKotlin
    from processResources
    into('lib') {
        from configurations.runtime
    }
}

task buildDocker(type: Copy) {
    from compileJava
    from compileKotlin
    from processResources
    into('lib') {
        from configurations.runtime
    }
    into 'build/docker'
}

build.dependsOn buildDocker

Build configuration above is a good starting level as it provides configuration for both java and kotlin, enables plugin for Intellij IDEA and configures build to copy all build artifacts to corresponding location suitable for local execution.

Let’s create our first function using Kotlin!

First let’s create some directory where we can store source code and also use some namespace. In this post we will put examples under io.enjapan.examples.lambda namespace.

mkdir -p src/main/kotlin/io/enjapan/examples/lambda

Now we’re ready to create our first function. Create helloHandler.kt in the directory above and add the code below:

package io.enjapan.examples.lambda

import com.amazonaws.services.lambda.runtime.Context
import com.amazonaws.services.lambda.runtime.RequestStreamHandler
import java.io.*
import com.fasterxml.jackson.module.kotlin.*

class Hello : RequestStreamHandler {
    data class HandlerInput(val name: String)
    data class HandlerOutput(val message: String)

    private val mapper = jacksonObjectMapper()

    override fun handleRequest(input: InputStream, output: OutputStream, context: Context) {
        val inputObj = mapper.readValue<HandlerInput>(input)
        mapper.writeValue(output, HandlerOutput("Hello ${inputObj.name}"))
    }
}

Few words on how the code above works and what it does:

  • class Hello implements RequestStreamHandler interface provided by AWS Lambda runtime
  • two data classes used to serialize/deserialize data passed to the function and returned from it
  • jackson object mapper is used to parse data from input stream and deserialize it using HandlerInput
  • jackson object mapper also used to serialize data stored in the instance of HandlerOutput and write it to the output stream

This is a valid AWS lambda function handler. Let’s execute it!

./gradlew build

Get docker image that will be used to run our code:

docker pull lambci/lambda:java8

Running function using local docker environment:

docker run --rm -v "$PWD/build/docker":/var/task lambci/lambda:java8 io.enjapan.examples.lambda.Hello '{"name": "Bob"}'

Should produce the following output:

START RequestId: aa0789b5-6149-4b99-84da-51cfd9a5a5e1 Version: $LATEST

END RequestId: aa0789b5-6149-4b99-84da-51cfd9a5a5e1
REPORT RequestId: aa0789b5-6149-4b99-84da-51cfd9a5a5e1	Duration: 1571.25 ms	Billed Duration: 1600 ms	Memory Size: 1536 MB	Max Memory Used: 8 MB

{"message":"Hello Bob"}

Very good! We were able to implement very simple AWS Lambda function and execute it. Even though it is a bit oversimplifed example, it provides solid foundation on how to implement lambda function, how to pass input data and return output. Armed with that knowledge we can actually implement something interesting.

Price search

Now we are going to implement something more interesting. Since the goal of this blog post is to explain how to execute lambda function locally, i.e. before it could be executed in the real AWS environment, while we at it let’s also create a small function that would allow us to get pricing for given region and list of services.

Create in the same directory a new class priceSearchHandler.kt with the following initial code:

class PriceSearch : RequestStreamHandler {
    companion object {
        const val ENDPOINT = "api.pricing.us-east-1.amazonaws.com"
        const val REGION = "us-east-1"
    }
    
    private val mapper = jacksonObjectMapper()
}

Now, let’s decide how function input and output should look like. Before we said, that search query should include a list of services and location/region, thus input JSON could look like below:

{ "services": ["lambda"], "location": "Asia Pacific (Tokyo)" }

We can use the following data class to store deserialized data for the query above:

data class HandlerInput(val services: List<String>, val location: String)

The result we expect is a list of prices, so it should be also pretty straightforward.

data class HandlerOutput(val result: List<Price>)

Now, how should Price look like? We are interested in the prices for specified services, thus at the very least it should consist of service name or code and a list of prices for that service.

data class Price(val serviceCode: String?, val priceList: MutableList<String>?)

Note that both fields are marked as nullable, because we expect to store there data coming from the Java layer, i.e. AWS SDK, thus it gives us guarantee that at the very least calls won’t cause NPE (NullPointerException).

Below is a method to configure corresponding AWS pricing client, since we are going to need client in two places: search for service code and then with get pricing for that service code.

private fun getAwsPricingClient(): AWSPricing {
    val endPointConfig = AwsClientBuilder.EndpointConfiguration(ENDPOINT, REGION)
    return AWSPricingClientBuilder.standard().withEndpointConfiguration(endPointConfig).build()
}

Finding service by provided search term:

private fun findService(searchTerm: String): Service? {
    val client = getAwsPricingClient()
    val res = client.describeServices(DescribeServicesRequest())
    return res.services.find {
        it.serviceCode.equals(searchTerm, true) || it.serviceCode.contains(searchTerm, true)
    }
}

Fetching prices by provided service code and filter:

private fun genFilter(field: String, value: String, type: String = "TERM_MATCH"): Filter {
    return Filter().withType(type).withField(field).withValue(value)
}

private fun fetchPrices(serviceCode: String?, filter: Filter): MutableList<String>? {
    val client = getAwsPricingClient()
    val req = GetProductsRequest().withServiceCode(serviceCode).withFilters(filter)
    return client.getProducts(req).priceList
}

Okay, so now we are ready to implement the handler method and actually wire everything together.

override fun handleRequest(input: InputStream, output: OutputStream, context: Context) {
    val inputObj = mapper.readValue<HandlerInput>(input)
    val prices = inputObj.services.map {
        val serviceCode = findService(it)?.serviceCode
        val priceList = fetchPrices(serviceCode, genFilter("location", inputObj.location))
        Price(serviceCode, priceList)
    }
    mapper.writerWithDefaultPrettyPrinter().writeValue(output, HandlerOutput(prices))
}

What happens here?

  • input data gets deserialized and saved inside newly created HandlerInput instance
  • for each service corresponding service code is fetched and then using that service code price list is created
  • results of both calls above are stored inside Price instance
  • all resulting prices are stored in HandlerOutput instance which is serialized back to JSON

Before we can actually execute it is important to note, that code is executed inside docker container, thus for code relying on AWS SDK to function properly, it is necessary to pass corresponding credentials.

With small wrapper script (let’s name it run_lambda.sh) like below, it should be enough to run the code above:

#!/usr/bin/env bash

export AWS_DEFAULT_REGION="ap-northeast-1"
TMP_CREDENTIALS=$(aws sts get-session-token --duration-seconds 900) # request temporary credentials for 15 minutes

# uses jq (https://stedolan.github.io/jq/)
export AWS_ACCESS_KEY_ID=$(echo ${TMP_CREDENTIALS} | jq -r '.Credentials.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo ${TMP_CREDENTIALS} | jq -r '.Credentials.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo ${TMP_CREDENTIALS} | jq -r '.Credentials.SessionToken')

docker run --rm \
	-e AWS_DEFAULT_REGION=$AWS_DEFAULT_REGION \
	-e AWS_ACCESS_KEY_ID=$AWS_ACCESS_KEY_ID \
	-e AWS_SECRET_ACCESS_KEY=$AWS_SECRET_ACCESS_KEY \
	-e AWS_SESSION_TOKEN=$AWS_SESSION_TOKEN \
	-v "$PWD/build/docker":/var/task lambci/lambda:java8 "$@"

Finally, let’s see the code in action!

./gradlew build && ./run_lambda.sh io.enjapan.examples.lambda.PriceSearch '{"services": ["lambda"], "location": "Asia Pacific (Tokyo)"}'
START RequestId: 880f7ca6-96ef-4509-8600-574a6bb6164a Version: $LATEST
END RequestId: 880f7ca6-96ef-4509-8600-574a6bb6164a
REPORT RequestId: 880f7ca6-96ef-4509-8600-574a6bb6164a	Duration: 5535.57 ms	Billed Duration: 5600 ms	Memory Size: 1536 MB	Max Memory Used: 10 MB

{
  "result": [
    {
      "serviceCode": "AWSLambda",
      "priceList": [
        "{\"product\":{\"productFamily\":\"Serverless\",\"attributes\":{\"servicecode\":\"AWSLambda\",\"groupDescription\":\"Invocation call for a Lambda function\",\"usagetype\":\"APN1-Request\",\"locationType\":\"AWS Region\",\"location\":\"Asia Pacific (Tokyo)\",\"servicename\":\"AWS Lambda\",\"operation\":\"\",\"group\":\"AWS-Lambda-Requests\"},\"sku\":\"3BE8DYKG4FYSZGDW\"},\"serviceCode\":\"AWSLambda\",\"terms\":{\"OnDemand\":{\"3BE8DYKG4FYSZGDW.JRTCKXETXF\":{\"priceDimensions\":{\"3BE8DYKG4FYSZGDW.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"Requests\",\"endRange\":\"Inf\",\"description\":\"AWS Lambda - Total Requests - Asia Pacific (Tokyo)\",\"appliesTo\":[],\"rateCode\":\"3BE8DYKG4FYSZGDW.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0000002000\"}}},\"sku\":\"3BE8DYKG4FYSZGDW\",\"effectiveDate\":\"2018-04-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20180419235108\",\"publicationDate\":\"2018-04-19T23:51:08Z\"}",
        "{\"product\":{\"productFamily\":\"Serverless\",\"attributes\":{\"servicecode\":\"AWSLambda\",\"groupDescription\":\"Invocation duration weighted by memory assigned to function, measured in GB-s\",\"usagetype\":\"APN1-Lambda-GB-Second\",\"locationType\":\"AWS Region\",\"location\":\"Asia Pacific (Tokyo)\",\"servicename\":\"AWS Lambda\",\"operation\":\"\",\"group\":\"AWS-Lambda-Duration\"},\"sku\":\"FSYUV9NMNDEXRJ5H\"},\"serviceCode\":\"AWSLambda\",\"terms\":{\"OnDemand\":{\"FSYUV9NMNDEXRJ5H.JRTCKXETXF\":{\"priceDimensions\":{\"FSYUV9NMNDEXRJ5H.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"seconds\",\"endRange\":\"Inf\",\"description\":\"AWS Lambda - Total Compute - Asia Pacific (Tokyo)\",\"appliesTo\":[],\"rateCode\":\"FSYUV9NMNDEXRJ5H.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0000166667\"}}},\"sku\":\"FSYUV9NMNDEXRJ5H\",\"effectiveDate\":\"2018-04-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20180419235108\",\"publicationDate\":\"2018-04-19T23:51:08Z\"}",
        "{\"product\":{\"productFamily\":\"Serverless\",\"attributes\":{\"servicecode\":\"AWSLambda\",\"groupDescription\":\"Invocation duration weighted by memory assigned to function, measured in GB-s\",\"usagetype\":\"APN1-Lambda-Edge-GB-Second\",\"locationType\":\"AWS Region\",\"location\":\"Asia Pacific (Tokyo)\",\"servicename\":\"AWS Lambda\",\"operation\":\"\",\"group\":\"AWS-Lambda-Edge-Duration\"},\"sku\":\"JC3XFM9B7WBA85YZ\"},\"serviceCode\":\"AWSLambda\",\"terms\":{\"OnDemand\":{\"JC3XFM9B7WBA85YZ.JRTCKXETXF\":{\"priceDimensions\":{\"JC3XFM9B7WBA85YZ.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"Lambda-GB-Second\",\"endRange\":\"Inf\",\"description\":\"AWS Lambda Edge - Total Compute - Asia Pacific (Tokyo)\",\"appliesTo\":[],\"rateCode\":\"JC3XFM9B7WBA85YZ.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0000500100\"}}},\"sku\":\"JC3XFM9B7WBA85YZ\",\"effectiveDate\":\"2018-04-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20180419235108\",\"publicationDate\":\"2018-04-19T23:51:08Z\"}",
        "{\"product\":{\"productFamily\":\"Serverless\",\"attributes\":{\"servicecode\":\"AWSLambda\",\"groupDescription\":\"Invocation call for a Lambda function\",\"usagetype\":\"APN1-Lambda-Edge-Request\",\"locationType\":\"AWS Region\",\"location\":\"Asia Pacific (Tokyo)\",\"servicename\":\"AWS Lambda\",\"operation\":\"\",\"group\":\"AWS-Lambda-Edge-Requests\"},\"sku\":\"ZXDACKGUCWW6BGB3\"},\"serviceCode\":\"AWSLambda\",\"terms\":{\"OnDemand\":{\"ZXDACKGUCWW6BGB3.JRTCKXETXF\":{\"priceDimensions\":{\"ZXDACKGUCWW6BGB3.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"Request\",\"endRange\":\"Inf\",\"description\":\"AWS Lambda Edge - Total Requests - Asia Pacific (Tokyo)\",\"appliesTo\":[],\"rateCode\":\"ZXDACKGUCWW6BGB3.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0000006000\"}}},\"sku\":\"ZXDACKGUCWW6BGB3\",\"effectiveDate\":\"2018-04-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20180419235108\",\"publicationDate\":\"2018-04-19T23:51:08Z\"}"
      ]
    }
  ]
}

Conclusion

Ability executing lambda functions locally have clear benefits such as testing before actually pushing the code and reducing overall time needed to execute functions (thus less overhead). At the same time, approach described here was meant for the code execution only. If its necessary to test more integration points (for example, testing how API Gateway handles requests and passes them to corresponding underlying lambda functions), there is a tool provided by Amazon AWS SAM. It is worth mentioning that under the hood AWS SAM use runtimes from lambci project, thus points explained in this blog post would be also true for AWS SAM as well.

All code samples in this blog post are also available in Github repository.

Leave a Comment

Your email address will not be published. Required fields are marked *

Loading...