Limitation of Access to Internet-facing Elasticsearch Only from Lambda with IAM Role

Limitation of Access to Internet-facing Elasticsearch Only from Lambda with IAM Role

Takahiro Iwasa
(岩佐 孝浩)
Takahiro Iwasa (岩佐 孝浩)
3 min read
Elasticsearch IAM Lambda

Amazon Elasticsearch users should basically place Elasticsearch clusters within VPCs. However, this requires NAT Gateway or NAT instances, by which you would incur additional costs. When placing your clusters in public, you can use a Lambda function as a proxy to your Elasticsearch domain.

Prerequisites

Install the following on you computer.

Creating SAM Application

Directory Structure

/
|-- es-proxy-lambda/
|   |-- __init__.py
|   |-- lambda_function.py
|   `-- requirements.txt
|-- samconfig.toml
`-- template.yaml

AWS SAM Template

Please note the following:

  • An Elasticsearch domain limits access using AccessPolicies. (lines 8-16)
  • A Lambda function needs appropriate policies to access the Elasticsearch domain. (lines 64-70)
AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Resources:
  Elasticsearch:
    Type: AWS::Elasticsearch::Domain
    Properties:
      AccessPolicies:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS:
                - !GetAtt IamRole.Arn
            Action: es:*
            Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/es-for-lambda/*
      DomainName: es-for-lambda
      EBSOptions:
        EBSEnabled: true
        VolumeSize: 10
        VolumeType: standard
      ElasticsearchClusterConfig:
        DedicatedMasterEnabled: false
        InstanceCount: 1
        InstanceType: t2.small.elasticsearch
      ElasticsearchVersion: 7.4

  Lambda:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: es-proxy-lambda/
      Environment:
        Variables:
          ES_DOMAIN: !GetAtt Elasticsearch.DomainEndpoint
      FunctionName: es_proxy_lambda
      Handler: lambda_function.lambda_handler
      Role: !GetAtt IamRole.Arn
      Runtime: python3.8

  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub
        - /aws/lambda/${name}
        - {name: !Ref Lambda}
      RetentionInDays: 1

  IamRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - es:ESHttpHead
                  - es:DescribeElasticsearchDomain
                  - es:ESHttpGet
                  - es:DescribeElasticsearchDomainConfig
                Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/es-for-lambda
          PolicyName: policy
      RoleName: es-proxy-lambda-role

Python Script

requirements.txt

The AWS Lambda runtime environment has boto3 installed by default, so there is no need to include it in your requirements.txt.

certifi==2019.11.28
chardet==3.0.4
elasticsearch==7.5.1
idna==2.9
requests==2.23.0
requests-aws4auth==0.9
urllib3==1.25.8

lambda_function.py

The following script uses requests_aws4auth to generate an Auth header including AWS credentials. (lines 10-11 and 14)

import os

import boto3
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth


es_domain = os.environ.get('ES_DOMAIN')
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(
    credentials.access_key, credentials.secret_key, 'ap-northeast-1', 'es', session_token=credentials.token)
es = Elasticsearch(
    hosts=[{'host': es_domain, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)


def lambda_handler(event, context):
    response = es.info()
    print(response)

samconfig.toml

Replace <YOUR_S3_BUCKET> with the actual value.

version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
stack_name = "es-proxy-lambda"
s3_bucket = "<YOUR_S3_BUCKET>"
s3_prefix = "es-proxy-lambda"
region = "ap-northeast-1"
capabilities = "CAPABILITY_IAM CAPABILITY_NAMED_IAM"

Build and Deploy

Build and deploy with the following command. Creating the Elasticsearch domain may take 10 to 20 minutes.

sam build
sam deploy

Testing

Testing with Lambda

Execute the Lambda function. It should succeed to access the Elasticsearch domain.

Testing with Terminal

Try to access the Elasticsearch domain using the following command. It should be rejected by the Elasticsearch domain.

$ curl https://search-es-for-lambda-rase3snu6yozl6xhjcuq34cu5m.ap-northeast-1.es.amazonaws.com/
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}

Cleaning Up

Clean up the provisioned AWS resources with the following command.

sam delete --stack-name es-proxy-lambda
Takahiro Iwasa
(岩佐 孝浩)

Takahiro Iwasa (岩佐 孝浩)

Software Developer at iret, Inc.
Architecting and developing cloud native applications mainly with AWS. Japan AWS Top Engineers 2020-2023