Users expect AI responses to appear token by token, like ChatGPT. But AWS Lambda and API Gateway aren't designed for long-lived, streaming responses out of the box.

I wanted Server-Sent Events (SSE): a simple way to stream AI responses into a chat UI without managing servers. Here's how Lambda Response Streaming solved both problems.

The Problem

Staying serverless while delivering real-time UX meant every common approach had deal-breaking limitations:

• API Gateway HTTP + AWS Lambda (proxy)

  • 30-second timeout limit for HTTP APIs

  • No true streaming, responses buffer until function completion

• API Gateway WebSockets

  • Requires complex connection management and keepalive logic

• Async job + polling

  • Poor UX, users stare at loading spinners while clients poll for updates

What I wanted was progressive delivery: send partial LLM streaming responses immediately, without breaking my serverless constraint

The Solution: Lambda Response Streaming

AWS introduced Lambda Response Streaming about two years ago (2023), but the ideal use cases weren't immediately obvious.

Interacting with AI models turned out to be the perfect fit.

Lambda Response Streaming lets functions send data directly to clients instead of buffering everything until completion. Combined with the rise of streaming LLM models like those in AWS Bedrock, this creates exactly what users expect from modern conversational AI interfaces.

My architecture handles privacy compliance while leveraging the best AI models:

The flow is straightforward but powerful:

1. User request flows through CloudFront to the Lambda Function URL

2. Comprehend extracts and anonymizes personal financial data

3. System prompt is read from S3 bucket (if not cached)

4. User message is stored in DynamoDB

5. Anonymized data sent to Bedrock for AI analysis

6. Bedrock streams tokens back through Lambda as Server-Sent Events

7. User gets real-time responses without personal data leaving the EU

Why CloudFront? It provides out-of-the-box country-level access control. Since Creditaki targets a single market, I wanted to prevent non-targeted users from consuming expensive AI tokens.

The timing worked perfectly: Lambda Response Streaming matured just as AI model streaming became the expected UX standard.

Implementation with Node.js and TypeScript

Infrastructure as Code

I’m using AWS CDK to create the Lambda function and Lambda Function URL resources.

The Lambda Function URL requires specific configuration for response streaming:
invokeMode: RESPONSE_STREAM enables the streaming capability. By default, Lambda will buffer the entire response.

const chatFunction = new nodeLambda.NodejsFunction(this, "ChatFunction", {
  runtime: lambda.Runtime.NODEJS_20_X,
  timeout: cdk.Duration.seconds(180), // Extended timeout for AI processing
  memorySize: 256,
  environment: {
    DDB_TABLE: this.ddbTable.tableName,
    S3_BUCKET: this.s3Bucket.bucketName,
    // other env 
  },
});

const chatFunctionUrl = chatFunction.addFunctionUrl({
  authType: lambda.FunctionUrlAuthType.NONE,
  invokeMode: lambda.InvokeMode.RESPONSE_STREAM, // this is the setting
  cors: {
    allowedOrigins: ["*"],
    allowedMethods: [lambda.HttpMethod.GET, lambda.HttpMethod.POST],
  },
});

The 180-second timeout accommodates AI workflows where models use external tools. Bedrock models can call APIs, search knowledge bases, or perform multi-step reasoning that takes time to complete.

Application Code

The core of Lambda Response Streaming is the streamifyResponse wrapper and custom stream processing.

// Lambda handler
exports.handler = awslambda.streamifyResponse(
  async (event, responseStream) => {
    await pipeline(
      Readable.from(sendMessage(event)),
      responseStream
    );
  }
);

Since I wanted to use POST requests for chat messages, I used fetch with manual stream processing instead of EventSource (which only supports GET).

// frontend integration
async function sendChatMessage(message) {
  const response = await fetch('/c', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ message })
  });

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const chunk = decoder.decode(value);
    const lines = chunk.split('\n');

    for (const line of lines) {
      if (line.startsWith('data: ')) {
        const data = JSON.parse(line.slice(6));
        if (data.type === 'token') {
          appendToChat(data.content); // Update UI in real-time
        }
      }
    }
  }
}

This gives users the immediate feedback they expect from AI interactions - tokens appear as they're generated, just like ChatGPT.

Why Manual Stream Processing Instead of EventSource?

The browser's EventSource API is the standard way to consume Server-Sent Events, but it has a critical limitation: it only supports GET requests.

In my chat implementation, I need both methods:

• POST /c - Send new messages with user input

• GET /c - Retrieve chat session history

// This won't work for sending messages - EventSource only supports GET
const eventSource = new EventSource('/c', {
  method: 'POST',  // ❌ Not supported
  body: JSON.stringify({ message: "Hello" })
});

// Solution: Use fetch for both GET and POST with consistent stream processing
const response = await fetch('/c', {
  method: 'POST',  // ✅ Send messages
  body: JSON.stringify({ message: "Hello" })
});

// Same streaming logic works for GET requests too
const sessionResponse = await fetch('/c', {
  method: 'GET'  // ✅ Get chat history
});

Using fetch for both endpoints keeps the streaming implementation consistent across your API, rather than mixing EventSource (GET only) with custom fetch logic (POST).

Many developers miss this nuance when building chat interfaces - they assume EventSource will handle all SSE scenarios.

Key Takeaways

Lambda Response Streaming transformed how I deliver AI experiences at Creditaki:

• Real-time LLM streaming without complex infrastructure

• AWS Bedrock integration with privacy compliance

• Cost control through CloudFront geo-blocking

• Simple Node.js implementation compared to WebSocket alternatives

In short: ChatGPT-style streaming, on a 100% serverless stack.

The pattern works well for any AI application where users expect streaming responses. As more developers build AI workflows, Lambda Response Streaming provides a clean serverless solution that scales automatically.

Check out Creditaki - Credit AI Assistant to see this streaming pattern in action.

I've been working on migrating a client's on-premise shopping cart solution to AWS, and one of the tasks was to refactor the email notifications feature to use AWS SES instead of Mailgun.

For transactional emails such as order confirmations, shipping notifications, or payment receipts the email templates work like magic, you set it and forget it.

Both Mailgun and AWS SES support email templates, they're based on the Handlebars template system and work pretty much the same, or at least they should.

The problem

Post migration all transactional emails are getting sent but not delivered to the inbox. No clue of what's happening, no errors whatsoever.

Debugging steps

I started to investigate the possible causes:

• ✅ the account is moved out of sandbox mode

• ✅ the sender's identity is verified

• ✅ the DKIM is set

• ✅ SendTemplatedEmailCommand returns a successful message

• ✅ sends count against Daily email usage in the SES dashboard

• ✅ checked the Spam folder, nothing

Everything looks fine, yet emails are not reaching the inbox.

I did a quick test with a one-line template and it works. Bingo! 🥳 So there's a problem with the email template rendering.

Ok, great, but how do I debug the AWS SES template? The syntax looks correct!

The solution

I quickly looked for some tools but didn't find anything meaningful, so went to the docs and found this CLI method aws sesv2 test-render-email-template which saved my day!

You can use this command with the --template-name and --template-data
parameters and get a preview of the email content or an error.

Of course, it was an email template rendering error and this is the only way I could figure it out.

An error occurred (BadRequestException) when calling the TestRenderEmailTemplate operation: Attribute 'tax_amount' is not present in the rendering data.

The error is misleading because the tax_amount was present in my --template-data but had a null value for this specific case:

{
    "id": "2cl7Nr9KoU2KvB8ADy0s1hnD111",
    "customer_name": "Stefan Potter",
    "currency": "USD",
    "items":
        [
            {
                "image": "https://path/to/images/product-1.jpg",
                "quantity": 1,
                "price": 89.00,
                "name": "Wireless Headphones",
                "description": "Simple product with available quantity.",
                "id": "HEADPHONESX"
            }
        ],
    "shipping_address": "Behind Corner 5<br />Crater 12345<br />Moon",
    "shipping_method": "flat",
    "shipping_amount": 12.00,
    "billing_method": "stripe",
    "tax_amount": null,
    "total_amount": 101.00
}

Replacing the value with 0 fixed the template rendering.

If the template data contains non-compliant variables (whatever that means) or is missing variables, then Amazon SES won't deliver the email, but it won't tell you specifically.

I later came across this warning buried deeply in the CLI docs:

However, if Amazon SES can't render the email because the template contains errors, it doesn't send the email. Additionally, because it already accepted the message, Amazon SES doesn't return a message stating that it was unable to send the email.

The recommended way to handle these sorts of issues is to configure Amazon SES to send Rendering Failure event notifications through Amazon SNS.

Allowing direct access to the production database was a flat-out "nope" for security reasons and because any direct interaction with the production database poses potential risks.

Now, you might wonder, "Why not just spin up an RDS read replica and call it a day?". Good question!

An RDS read replica could have been a straightforward choice, a sidekick that would allow our non-dev colleagues to run SQL queries without stirring up any trouble in the production setup.

But there was no need for real-time data, most of the queries are done on previous days, if not weeks or months' data, so a 24-hour data replication delay was acceptable, and keeping up an additional RDS instance would be unnecessary.

A principle that resonates with my identity as a cloud architect is to focus on crafting secure and efficient solutions and avoid unnecessary infrastructure. This often leads me towards serverless approaches.

As we have automated backups enabled on our RDS instance, it means we have daily snapshots that are already available, the only thing left is getting that data into S3 so it can be crawled with Glue and queried using Athena.

And this is the architecture I came up with:

I love working with Step Functions, it's like having superpowers to orchestrate various services while having a crystal-clear view of the workflow, and when it comes to event-driven architectures, I find it satisfying to design systems that dynamically respond to specific events or changes.

So, Step Functions and EventBridge together? I'm all in!

The service components

• RDS instance - with automated backups enabled (you probably have it already)

• S3 Bucket - where the RDS data gets exported to and crawled from

• KMS key - to keep the things encrypted

• 2 Lambda functions - one to prepare the export files for crawling, the other to send SendTaskSuccess to resume workflow execution

• 3 EventBridge rules - listen for snapshot/export/crawler events

• Glue Crawler - prepares and organizes the data catalog

• Step Function - orchestrates the workflow

RDS Snapshots

The first step is to find out when an RDS automated snapshot is created, for this we create an EventBridge rule that will match the RDS DB snapshot event.

The automated snapshot names have the following format rds:dbid-yyyy-mm-dd-hh-mm

{
  "detail-type": ["RDS DB Snapshot Event"],
  "source": ["aws.rds"],
  "detail": {
    "Message": [{
      "prefix": "Automated snapshot created"
    }],
    "SourceIdentifier": [{
      "prefix": "rds:YOUR-DB-ID"
    }]
  }
}

When an automated snapshot gets created the event will match the above pattern, and fire up our Step Function execution.

Exporting RDS Snapshot to S3

Once the snapshot is created we need to export it into S3.

So, the first state of our Step Function is StartExportTask which will take care of the export.

"StartExportTask": {
    "Type": "Task",
    "Next": "Wait for Export (TT)",
    "Parameters": {
        "ExportTaskIdentifier.$": "States.ArrayGetItem(States.StringSplit($.detail.SourceIdentifier, ':'),1)",
        "IamRoleArn": "${ExportRoleArn}",
        "KmsKeyId": "${KmsKeyId}",
        "S3BucketName": "${S3Bucket}",
        "SourceArn.$": "$.detail.SourceArn"
    },
    "Resource": "arn:aws:states:::aws-sdk:rds:startExportTask"
}

The ExportTaskIdentifier is the name of the Export to S3 task and will also be the S3 folder where the data is exported. Note each Export Task name needs to be unique and for some reason, an Export to S3 task can't be deleted once created.

We use the Intrinsic Functions to remove the rds: prefix from the SourceIdentifier so we can use it as an S3 folder name.

The data will be exported in S3 in Apache Parquet format, the process is asynchronous and takes quite a bit. It depends on your database type and size, as the export task first restores and scales the entire database before extracting the data to Amazon S3.

In my experience, it takes at least 30 minutes just to get the export task out of STARTING status and a minimum of 5 minutes to COMPLETE.

Waiting for the Export to S3 to complete

Now, we need to wait for the Export to S3 to complete. As mentioned previously, the Export Task is asynchronous, it runs in the background so we need to pause our workflow and for this, we will add a state with a callback.

A state with a callback in Step Functions allows the workflow to pause until an external process signals its completion by calling the SendTaskSuccess (or SendTaskFailure) APIs with a task token.

The Wait for Export (TT) state will store the task token into an object in our S3 bucket and pause the workflow. Note the .waitForTaskToken suffix to the Resource ARN.

{
    "Wait for Export (TT)": {
        "Type": "Task",
        "Next": "Move Exported Objects",
        "Parameters": {
            "Body.$": "$$.Task.Token",
            "Bucket": "${S3Bucket}",
            "Key": "export_task_token"
        },
        "Resource": "arn:aws:states:::aws-sdk:s3:putObject.waitForTaskToken",
        "HeartbeatSeconds": 3600,
        "ResultPath": null
    }
}

To find out when the Export to S3 task gets completed we created another EventBridge rule with the following pattern:

{
  "detail-type": ["RDS DB Snapshot Event"],
  "source": ["aws.rds"],
  "detail": {
    "SourceIdentifier": [{
      "prefix": "rds:YOUR-DB-ID"
    }],
    "Message": [{
      "prefix": "Export task completed"
    }]
  }
}

Matching the above pattern signals the snapshot export is complete and will invoke the Lambda function that's responsible for resuming our workflow. The Lambda reads the task token that was written in the S3 object and calls SendTaskSuccess API.

const { SFNClient, SendTaskSuccessCommand } = require("@aws-sdk/client-sfn"),
    { S3Client, GetObjectCommand } = require("@aws-sdk/client-s3");
const sf = new SFNClient(), s3 = new S3Client();
exports.handler = async (event) => {
    // get the task token from s3
    const taskToken = await s3
        .send(
            new GetObjectCommand({
                Bucket: process.env.S3_BUCKET,
                Key: event.source === "aws.glue" ? "glue_task_token" : "export_task_token",
            })
        )
        .then((data) => data.Body.transformToString("utf-8"))
        .catch((err) => {
            console.log(err);
            return null;
        });
    // send the task token to step function
    await sf.send(
        new SendTaskSuccessCommand({
            output: JSON.stringify(event),
            taskToken: taskToken,
        })
    );
    return;
};

Preparing the data for the Glue Crawler

Now that we have the data exported in S3 in Apache Parquet we need to move it in the folder our Glue Crawler was instructed to look for it.

Every export is made in a different folder uniquely named after the export task identifier, so we need to move the Parquet files into the data folder so it can be crawled. Here's the Lambda function that does the job:

const { S3Client, ListObjectsV2Command, CopyObjectCommand, DeleteObjectCommand } = require("@aws-sdk/client-s3");
const s3 = new S3Client();
exports.handler = async (event) => {
    // list the objects in the bucket
    const objects = await s3
        .send(
            new ListObjectsV2Command({
                Bucket: process.env.S3_BUCKET,
                Prefix: event.prefix,
            })
        )
        .then((data) => data.Contents)
        .catch((err) => {
            console.log(err);
            return null;
        });
    // copy the objects to the data folder
    for (const object of objects) {
        await s3
            .send(
                new CopyObjectCommand({
                    Bucket: process.env.S3_BUCKET,
                    CopySource: `${process.env.S3_BUCKET}/${object.Key}`,
                    Key: `data/${object.Key.split("/").slice(1).join("/")}`,
                })
            )
            .then(() => {
                // delete the old object
                s3.send(
                    new DeleteObjectCommand({
                        Bucket: process.env.S3_BUCKET,
                        Key: object.Key,
                    })
                );
            })
            .catch((err) => {
                console.log(err);
                return null;
            });
    }
};

To be honest, I didn't want to use Lambda function for this but there was no other option. I initially tried using states directly in the Step Functions, unfortunately, the string manipulation is still very limited with Intrinsic Functions, and didn't find a good way to remove the prefix from the S3 Object Key to move the exports to the standardized folder. The flow would have looked as below:

Crawling the data

Once the data is moved we can start the Glue Crawler and prepare it to be queried by Athena.

The Glue Crawler analyzes the content and structure of the data, extracting key details such as table definitions and column information. This metadata is then stored in the AWS Glue Data Catalog, allowing Athena to query the data.

The crawl process is also asynchronous and takes a couple of minutes.

Waiting for the Glue Crawler to complete

In my Step Function, I added another state that waits for the crawl to complete, this is optional. I added it because I want to be notified if the workflow fails (not part of the article/template).

It uses the same technique, writing the task token to an S3 object and pausing the workflow until an event matches the below pattern and the Lambda gets invoked to resume the process.

{
  "detail-type": ["Glue Crawler State Change"],
  "source": ["aws.glue"],
  "detail": {
    "crawlerName": ["my_crawler_name"],
    "state": ["Succeeded"]
  }
}

Note the Standard Workflow for Step Functions can run up to a year, and we pay per state transition, so waiting for the callback won't cost extra.

However, I set a heartbeat to 3600 seconds for both states using a callback to allow the entire workflow to timeout if the Glue Crawled doesn't complete within that timeframe.

And there you have it – a secure, cost-efficient, and hassle-free way to run SQL queries on your RDS data while keeping your production database safe.

Summary

Here's the 1-click Deployment link to get going quickly, or click to view the complete self-contained Cloudformation template that can be deployed into any AWS account.

This becomes a problem when our Lambda function needs to communicate with IP-restricted access resources. I recently needed in a serverless project a Lambda function to connect to an Azure SQL Database that only accepted connections from whitelisted IP addresses.

There's no way to configure the IP address of a Lambda function.

The solution is to run the Lambda inside a VPC and route the outbound connections through a NAT that gets assigned an Elastic IP. This way the external requests done by the Lambda function will have the same IP address on every invocation.

When running inside a VPC, our Lambda function doesn't have access to the internet, even when connected to a public subnet. There's no way around using a NAT.

The standard implementation of the service architecture looks as in the diagram below. NAT is the key piece when we need a fixed IP for our Lambda functions.

What's a NAT?

A Network Address Translation (NAT) is a compute instance that enables traffic between our resources in a private subnet and the Internet.

In AWS we have 2 NAT options: NAT Gateway & NAT Instance.

The NAT Gateway is the official recommendation, but it comes at a premium.

NAT Gateway vs. NAT Instance

NAT Gateway is a highly available and scalable managed service, and it costs $0.045/hr, about ~$33/month before tax.

For personal projects, dev environments, or even in production for low-traffic or non-business critical applications a NAT Gateway is overpriced for its role.

In my case, the Lambda function has to connect just a couple of times a day to the Azure SQL Database and pull some fresh data, it's not a business-critical service, occasional failure can be tolerated. Adding the NAT Gateway into the stack would have 4X'ed the project's monthly AWS bill. Serverless ;)

Instead of a NAT Gateway, I used an EC2 instance that's configured to act as a router between the private subnet where the Lambda is invoked and the outside world, a NAT Instance.

Actually, this is nothing new, for many years the only way to create a NAT in AWS was by creating a NAT Instance.

An EC2 t4g.nano instance costs $0.0042/hr, so the smallest NAT Instance can be 10x cheaper compared to NAT Gateway.

The downside of using a NAT Instance instead of a NAT Gateway is we have to configure and then manage it, we're responsible for replacing the instance if it goes down and for applying software patches.

To solve the availability problem we'll launch the NAT Instance inside an Autoscaling group, there are no additional fees for it, and like this, we make sure to have a healthy NAT Instance running at all times. A self-healing NAT Instance. To patch it we can use AWS Systems Manager.

Worth mentioning that both options fail the serverless litmus test.

Routing Lambda through a NAT Instance

In this article, I'll explain how to run your Lambda functions with a fixed IP using a NAT Instance, which is up to 10x cheaper than a NAT Gateway.

The service architecture diagram remains almost the same, we replace the NAT Gateway with an EC2 instance that will do the NAT job.

Setting up a NAT Instance is a bit more complicated than using a NAT Gateway, but worry not, at the end of the article you'll find a 1-click deployment CloudFormation template that does all the work.

VPC

As mentioned previously, we need to run Lambda from inside a VPC, below are the steps and resources to create the VPC:

• create a VPC with 2 subnets in a single Availability Zone

• attach an Internet Gateway to the VPC

• create 2 Route Tables and associate each with the relevant subnet

• in the Public Subnet route table, we add a default route so the requests to any IP (0.0.0.0/0) to be routed through the Internet Gateway. This gives all resources created in the Public Subnet internet access.

• note the Private Subnet route table has no routes, we'll add that later

VPC:
    Type: AWS::EC2::VPC
    Properties:
        CidrBlock: !Ref VpcCIDR
        EnableDnsSupport: true
        EnableDnsHostnames: true
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName}
InternetGateway:
    Type: AWS::EC2::InternetGateway
    Properties:
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName}
InternetGatewayAttachment:
    Type: AWS::EC2::VPCGatewayAttachment
    Properties:
        InternetGatewayId: !Ref InternetGateway
        VpcId: !Ref VPC
PublicSubnet:
    Type: AWS::EC2::Subnet
    Properties:
        VpcId: !Ref VPC
        AvailabilityZone: !Select [0, !GetAZs ""]
        CidrBlock: !Ref PublicSubnetCIDR
        MapPublicIpOnLaunch: true
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName} Public Subnet
PublicRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
        VpcId: !Ref VPC
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName} Public Routes
DefaultPublicRoute:
    Type: AWS::EC2::Route
    DependsOn: InternetGatewayAttachment
    Properties:
        RouteTableId: !Ref PublicRouteTable
        DestinationCidrBlock: 0.0.0.0/0
        GatewayId: !Ref InternetGateway
PublicSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
        RouteTableId: !Ref PublicRouteTable
        SubnetId: !Ref PublicSubnet
PrivateSubnet:
    Type: AWS::EC2::Subnet
    Properties:
        VpcId: !Ref VPC
        AvailabilityZone: !Select [0, !GetAZs ""]
        CidrBlock: !Ref PrivateSubnetCIDR
        MapPublicIpOnLaunch: true
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName} Private Subnet
PrivateRouteTable:
    Type: AWS::EC2::RouteTable
    Properties:
        VpcId: !Ref VPC
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName} Private Routes
PrivateSubnetRouteTableAssociation:
    Type: AWS::EC2::SubnetRouteTableAssociation
    Properties:
        RouteTableId: !Ref PrivateRouteTable
        SubnetId: !Ref PrivateSubnet

Elastic IP

The Elastic IP will give us the fixed IP that we need to whitelist on the 3rd-party services. It will be associated with the EC2 instance at launch time.

Note the Elastic IP is free as long it is associated with a running instance. If it is not associated, or associated with a stopped instance it will cost $0.005/hr.

ElasticIP:
    Type: AWS::EC2::EIP
    Properties:
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName}-ElasticIP

NAT Instance

NAT Instance IAM Role

First, we have to create an IAM Role that can be assumed by our EC2 Instance in order to do the following actions:

• ec2:AssociateAddress - to associate the Elastic IP

• ec2:ModifyInstanceAttribute - to update the EC2 instance attribute

• ec2:CreateRoute and ec2:ReplaceRoute - to create/replace the route in the Private Subnet route table

Also, note the AmazonSSMManagedInstanceCore managed policy, which allows using AWS Systems Manager to Run Command or Session Manager.

NATInstanceRole:
    Type: AWS::IAM::Role
    Properties:
        Path: /
        RoleName: !Sub ${AWS::StackName}-NATInstanceRole
        AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
                - Effect: Allow
                  Principal:
                      Service:
                          - ec2.amazonaws.com
                  Action: sts:AssumeRole
        ManagedPolicyArns:
            - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        Policies:
            - PolicyName: !Sub ${AWS::StackName}-NATInstancePolicy
              PolicyDocument:
                  Version: "2012-10-17"
                  Statement:
                      - Effect: Allow
                        Action:
                            - ec2:AssociateAddress
                            - ec2:ReplaceRoute
                            - ec2:CreateRoute
                            - ec2:ModifyInstanceAttribute
                        Resource: "*"
NATInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
        Path: /
        InstanceProfileName: !Sub ${AWS::StackName}-NATInstanceProfile
        Roles:
            - !Ref NATInstanceRole

NAT Instance Security Group

The Security Group restricts the inbound connections to our NAT Instance only to resources from our VPC.

NATInstanceSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
        GroupDescription: !Sub ${AWS::StackName} NAT Instance SG
        VpcId: !Ref VPC
        SecurityGroupIngress:
            - IpProtocol: tcp
              FromPort: 0
              ToPort: 65535
              CidrIp: !Ref VpcCIDR
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName} NAT Instance SG

Autoscaling

We're going to launch the NAT Instance in the Public Subnet using EC2 Auto Scaling, there are no additional fees for it, and the service can automatically determine the health status of our instance using EC2 status checks.

If the NAT Instance is unhealthy, it will be terminated and replaced by a new one with minimum downtime. The AutoScalingGroup will make sure we have 1 healthy instance running at all times.

When our instance gets terminated and a new one gets created, the new instance will have a different IP, and that's the reason we didn't create a route in the private route table.

AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
        VPCZoneIdentifier:
            - !Ref PublicSubnet
        LaunchTemplate:
            LaunchTemplateId: !Ref LaunchTemplate
            Version: !GetAtt LaunchTemplate.LatestVersionNumber
        MinSize: 1
        MaxSize: 1
        DesiredCapacity: 1
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName} NAT Instance
              PropagateAtLaunch: true

LaunchTemplate:
    Type: AWS::EC2::LaunchTemplate
    Properties:
        LaunchTemplateName: !Sub ${AWS::StackName}-LaunchTemplate
        LaunchTemplateData:
            ImageId: !Ref EC2InstanceAmi
            InstanceType: !Ref EC2InstanceType
            IamInstanceProfile:
                Arn: !GetAtt NATInstanceProfile.Arn
            SecurityGroupIds:
                - !Ref NATInstanceSecurityGroup
            UserData:
                Fn::Base64: !Sub |
                    #!/bin/sh
                    # ... removed for brevity

UserData

The UserData property allows specifying a shell script that will run each time the instance is launched. If our Autoscaling group terminates an unhealthy instance, the new instance will execute this script at launch and automatically set up the NAT Instance.

Here's what it does:

• updates all existing packages and installs the iptables-services package

• enables IP forwarding

• configures NAT routing

• Associates the Elastic IP to the EC2 instance, the public IP of the instance will be the Elastic IP, and all requests that pass through this instance will appear as coming from this IP address.

• Disables EC2 instance source/destination check. This is enabled by default forcing the instance to be the source or destination of any traffic it sends or receives. For NAT we need the traffic to pass through.

• Creates/updates the route in the Private Subnet route table. We're adding a default route so the requests made by our resources in the Private Subnet (Lambda) to any destination (0.0.0.0/0) to be routed through the NAT Instance.

# update packages
yum update -y
# install iptable
yum install -y iptables-services
# enable iptable
systemctl enable iptables.service
# enable IP forwarding
sysctl -w net.ipv4.ip_forward=1
# configure NAT
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
# save iptable
service iptables save
# start iptable
service iptables start

# get instance-id
EC2_INSTANCE_ID=$(curl -s http://instance-data/latest/meta-data/instance-id)

# associate the Elastic IP address to the instance
aws ec2 associate-address --instance-id $EC2_INSTANCE_ID --allocation-id ${ElasticIP.AllocationId} --allow-reassociation  --region ${AWS::Region}

# disable source/destination check
aws ec2 modify-instance-attribute --instance-id $EC2_INSTANCE_ID --source-dest-check "{\"Value\": false}" --region ${AWS::Region}

# replace or create route in the default private route table
aws ec2 replace-route --route-table-id ${PrivateRouteTable} --destination-cidr-block "0.0.0.0/0" --instance-id $EC2_INSTANCE_ID --region ${AWS::Region} || aws ec2 create-route --route-table-id ${PrivateRouteTable} --destination-cidr-block "0.0.0.0/0" --instance-id $EC2_INSTANCE_ID --region ${AWS::Region}

Lambda Function

To run inside a VPC our Lambda function needs permissions to create and manage network interfaces (ENI) and be associated with a subnet and a security group.

We created the Private Subnet previously with the VPC, so we'll create a new IAM Role that will be assumed by our Lambda and the Security Group.

Lambda Execution Role & Security Group

The Lambda execution role is attached the AWSLambdaVPCAccessExecutionRole managed policy which grants permissions to manage ENIs within an Amazon VPC and write to CloudWatch Logs.

NodeLambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
        Path: /
        RoleName: !Sub ${AWS::StackName}-NodeLambdaExecutionRole
        AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
                - Effect: Allow
                  Principal:
                      Service:
                          - lambda.amazonaws.com
                  Action: sts:AssumeRole
        ManagedPolicyArns:
            - arn:aws:iam::aws:policy/service-role/AWSLambdaVPCAccessExecutionRole

NodeLambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
        GroupDescription: !Sub ${AWS::StackName} Node Lambda SG
        VpcId: !Ref VPC
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName} Node Lambda SG

Lambda Function Code

For demo purposes, I included a simple NodeJS Lambda function in the stack that calls the Ipify API and checks if the request made by Lambda comes from the Elastic IP.

exports.handler = async (event) => {
    const https = require("https");
    const url = "https://api.ipify.org?format=json";
    const response = await new Promise((resolve, reject) => {
        https
            .get(url, (res) => {
                let data = "";
                res.on("data", (chunk) => {
                    data += chunk;
                });
                res.on("end", () => {
                    resolve(JSON.parse(data));
                });
            })
            .on("error", (err) => {
                reject(err);
            });
    });
    return { elastic_ip: process.env.ELASTIC_IP, source_ip: response.ip };
};

Deploying the Lambda Function

As our function code is up to 4096 chars we can deploy it as a CloudFormation resource using inline code. Note that we pass the ELASTIC_IP as an environment variable so our code can reference it.

NodeLambdaFunction:
    Type: AWS::Lambda::Function
    Properties:
        Code:
            ZipFile: |
                # ... function code omitted for brevity
        Handler: index.handler
        Role: !GetAtt NodeLambdaExecutionRole.Arn
        Runtime: nodejs18.x
        Timeout: 30
        Environment:
            Variables:
                ELASTIC_IP: !Ref ElasticIP
        VpcConfig:
            SecurityGroupIds:
                - !Ref NodeLambdaSecurityGroup
            SubnetIds:
                - !Ref PrivateSubnet

Summary

Here's the 1-click Deployment link to get going quickly, or click to view the complete self-contained Cloudformation template that can be deployed into any AWS account.

But API Gateway directly integrates with a handful of other AWS APIs, and SQS is one of them.

We can have an HTTP endpoint send messages directly to SQS without the need for a Lambda function, and we can even perform request validation at the API Gateway level.

This serverless pattern is extremely handy when creating 3rd party integrations for our services. In this article, I will explain how it works and provide a Cloudformation template you can easily adapt to your use case and deploy with one click.

Asynchronous processing

Processing incoming webhook data in real time can be challenging, especially if our service needs to perform additional processing or trigger downstream actions.

In these situations, the producer only needs to know if the data was successfully received, the processing can happen in the background, asynchronously, so we return a successful response and send the request in a queue for later processing.

We often know the schema in advance so we can also validate the request before sending the message to SQS. The backend worker service will consume messages from the queue at its own pace, one by one or in batches, and do the hard lifting.

If the order of processing is important we can use an SQS FIFO queue to process the messages in the exact order they were received. We can also bring an SQS Dead Letter Queue into the mix to collect the processing failures for further investigation or retry.

I used a similar architecture for a service that integrates with Mailgun for marketing email sending and reporting. Each time an action is performed by one of the email recipients (open, click, bounce, unsubscribe) Mailgun triggers the service HTTP endpoint and sends an event notification, the notification is stored in SQS, then a Lambda consumes these events, and creates campaign reports and other actions. It handles millions of events per week.

Main components

• SQS

• API Gateway (REST)

• IAM Role

SQS

In this template, I'm creating two queues, the queue that receives the messages for processing and one that will act as a Dead Letter Queue and receive all the messages that fail to process more than 5 times.

I'm also using a FIFO queue to illustrate how to use the OpenAPI extension to deal with message grouping and content-based deduplication. But for most use cases, the order of processing isn't important and we can default to a normal queue.

# SQS
Queue:
    Type: AWS::SQS::Queue
    Properties:
        QueueName: !Sub ${QueueName}-${StageName}.fifo
        FifoQueue: true
        ContentBasedDeduplication: true
        RedrivePolicy:
            deadLetterTargetArn: !GetAtt QueueDLQ.Arn
            maxReceiveCount: 5
# SQS DLQ
QueueDLQ:
    Type: AWS::SQS::Queue
    Properties:
        QueueName: !Sub ${QueueName}-${StageName}-dlq.fifo
        FifoQueue: true

API Gateway

I'm using the REST API Gateway, not the HTTP type (API Gateway v2). REST API supports more features than HTTP API such as API keys or request validation.

API Gateway will create an /events endpoint that will accept POST requests. It will perform basic validation of the request payload before passing it to SQS.

APIGateway:
    Type: AWS::ApiGateway::RestApi
    Properties:
        Name: !Ref AWS::StackName
        Description: !Sub ${AWS::StackName} API Gateway
        EndpointConfiguration:
            Types:
                - REGIONAL
        Body:
            openapi: 3.0.1
            info:
                title: API Gateway -> SQS
            paths:
                /events:
                    post:
                        description: Send events to SQS
                        operationId: sendEvent
                        requestBody:
                            content:
                                application/json:
                                    schema:
                                        $ref: "#/components/schemas/event"
                        # enable request validator
                        x-amazon-apigateway-request-validator: basic
                        # define request response
                        responses:
                            "200":
                                description: Successful request
                            "400":
                                description: Bad request
                            "500":
                                description: Server Error
                        # enable api gateway integration
                        x-amazon-apigateway-integration:
                            # ... removed for brevity
                            # ... check SQS Integration section

            # define request body schema
            components:
                schemas:
                    event:
                        # ... removed for brevity    
                        # ... check Request Validation section

            # define request validator
            x-amazon-apigateway-request-validators:
                basic:
                    validateRequestParameters: false
                    validateRequestBody: true

SQS Integration

The OpenAPI x-amazon-apigateway-integration extension allows us to configure SQS integration in the Body property for each endpoint. What you need to know is:

• passthroughBehavior - is set to never so the request is always transformed using requestTemplates value

• requestParameters - SQS POST request sends query parameters as a form in the body of the HTTP request, we need to set the Content-Type header accordingly

• requestTemplates - map the request payload, headers, query, or path parameters to the SQS API request, it uses Velocity templates - check API Gateway mapping template & variable reference

• responses - configures the response, in our case returns the SQS MessageId in case of a successful request

x-amazon-apigateway-integration:
    httpMethod: POST
    type: aws
    credentials: !GetAtt AGW2SQSRole.Arn
    uri: !Sub arn:aws:apigateway:${AWS::Region}:sqs:path/${AWS::AccountId}/${Queue.QueueName}
    passthroughBehavior: never
    requestParameters:
        integration.request.header.Content-Type: "'application/x-www-form-urlencoded'"
    requestTemplates:
        application/json: |
            #set($MsgDedupId = $context.requestId)
            #set($GroupId = $input.json('$.data.event_type'))
            Action=SendMessage&MessageGroupId=$GroupId&MessageDeduplicationId=$MsgDedupId&MessageBody=$input.body
    responses:
        default:
            statusCode: 200
            responseTemplates:
                application/json: |
                    #set($output = $input.path('$'))
                    {
                      "id": "$output.SendMessageResponse.SendMessageResult.MessageId"
                    }

Request Validation

API Gateway allows us to perform a basic request validation before sending the message to SQS. When the validation fails, API Gateway immediately fails the request, returns a 400 error response, and publishes the validation results in CloudWatch Logs.

We're using the OpenAPI components property to define our model schema, in our case, we'll only validate the request body.

event:
    required:
        - event
        - recipient
        - timestamp
    type: object
    properties:
        event:
            type: string
            example: opened
            enum:
                - accepted
                - opened
                - clicked
                - failed
                - rejected
                - unsubscribed
                - complained
        recipient:
            type: string
            example: "john@example.com"
            pattern: "^[a-zA-Z0-9_.+-]+@[a-zA-Z0-9-]+\\.[a-zA-Z0-9-.]+$"
        timestamp:
            type: string
            format: date-time
            example: "2023-01-01T23:00:00Z"

IAM Role

The IAM Role will be assumed by the API Gateway service and will allow it to send messages to our SQS queue, it'll also allow API Gateway to write logs into CloudWatch.

Note the IAM policy is only granting the service permission to perform this specific task.

# role allowing AGW to send messages to SQS queue
AGW2SQSRole:
    Type: AWS::IAM::Role
    Properties:
        Path: /
        RoleName: !Sub ${AWS::StackName}-agw2sqs
        AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
                - Effect: Allow
                  Principal:
                      Service:
                          - apigateway.amazonaws.com
                  Action: sts:AssumeRole
        Policies:
            - PolicyName: !Sub ${AWS::StackName}-agw2sqs
              PolicyDocument:
                  Version: "2012-10-17"
                  Statement:
                      # allow AGW to send messages to SQS queue
                      - Effect: Allow
                        Action:
                            - sqs:SendMessage
                        Resource: !GetAtt Queue.Arn
                      # let AGW write to CloudWatch
                      - Effect: Allow
                        Action:
                            - logs:CreateLogGroup
                            - logs:CreateLogStream
                            - logs:PutLogEvents
                        Resource: "*"

If you're having trouble understanding IAM concepts check my article about it - Explain like I'm 5: AWS IAM

One-click Deployment to get going quickly, or click to view the complete self-contained Cloudformation template that can be deployed into any AWS account.

Quite often the underlying database schema is not optimized for our search needs, the relevance of search results is questionable, and running ad-hoc queries on big datasets usually leads to performance issues and slow response.

This is why you should add a dedicated search service into the mix.

Typesense is my favorite, it's simple, fast, typo-tolerant, and open-source. I've been using it on multiple production projects handling searches on datasets of millions of entries.

For production use, it is recommended to run a multi-node Typesense cluster, however, if you're looking for a lighter setup check my other post on how to run a single Typesense node in AWS.

In this article, I will explain how to get a highly available (multi-node) Typesense cluster running in AWS using infrastructure as code.

The setup uses 2 AWS CloudFormation stacks that can be deployed into any AWS account:

• the Network Stack (1-click deployment) - creates the VPC and subnets in 3 AZs

• the Cluster Stack (1-click deployment)- creates all the cluster resources

How it Works

Typesense uses the Raft consensus algorithm to manage the cluster and recover from node failures. In cluster mode, Typesense will automatically replicate the entire dataset to all nodes. Automatically and continuously.

Raft requires a quorum for consensus and we need to run a minimum of 3 nodes to tolerate a 1-node failure, and a 5-node cluster will tolerate failures of up to 2 nodes, but at the expense of slightly higher write latencies.

To start a Typesense node as part of a cluster, each node instance needs to have a definition file with the list of all nodes in the following format, separated by commas:

<peering_address>:<peering_port>:<api_port>

• The peering_address (the EC2 instance private IP address)

• The peering_port (the port used for cluster operations - default 8107)

• The api_port (the port to which clients connect to - default 8108)

We will use an Auto Scaling group to launch a minimum of 3 EC2 Instances distributed across 3 Availability Zones and start a Typesense service on each.

# ASG
AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
        LaunchConfigurationName: !Ref LaunchConfig
        DesiredCapacity: 3
        MinSize: 3
        MaxSize: 6
        NotificationConfigurations:
            - NotificationTypes:
                  - autoscaling:EC2_INSTANCE_TERMINATE
              TopicARN: !Ref EC2EventsTopic
        TargetGroupARNs:
            - !Ref ALBTargetGroup
        VPCZoneIdentifier:
            - !ImportValue typesense-vpc-PublicSubnet1
            - !ImportValue typesense-vpc-PublicSubnet2
            - !ImportValue typesense-vpc-PublicSubnet3

EC2 Instance Type

CPU capacity is important to handle concurrent search traffic and indexing operations, so Typesense requires at least 2 vCPUs of compute capacity to operate.

All the next-gen EC2 T4g instances have at least 2 vCPUs, they're powered by Arm-based AWS Graviton2 processors and are ideal for running applications with moderate CPU usage that experience temporary spikes in usage.

The amount of RAM required is completely dependent on the size of the data you index, but for Typesense to hold the whole index in memory, the instance memory should be 2-3X the size of the data set.

As the Typesense process, itself is quite lightweight (20MB RAM with empty dataset) a 2GB memory instance (t4g.small) will likely handle a data set of up to 1GB.

Data Storage

Typesense stores a copy of the raw data on disk and then builds the in-memory index with the data. Then at search time, after determining the final set of documents to return in the API response, it fetches these documents (only) from the disk and puts them in the API response.

We'll use the Amazon EC2 instance store for the node data, although this type of storage is ephemeral and data will be lost if the instance gets terminated. When a new instance is launched the data will be replicated from the other nodes automatically.

We'll have scheduled backups for our peace of mind.

Load Balancing

AWS Elastic Load Balancing is used to distribute the traffic equally across all nodes, and we're going to add an Application Load Balancer in front of our EC2 instances from all 3 Availability Zones. It will also help us manage the SSL termination.

ALB:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
        Name: !Sub ${AWS::StackName}-${EnvironmentType}
        Subnets:
            - !ImportValue typesense-vpc-PublicSubnet1
            - !ImportValue typesense-vpc-PublicSubnet2
            - !ImportValue typesense-vpc-PublicSubnet3
        SecurityGroups:
            - !GetAtt ALBSecurityGroup.GroupId

The ALB Security Group allows inbound traffic on ports 80 (HTTP) and 443 (HTTPS).

ALBSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
        GroupDescription: allow traffic on port 80
        VpcId: !ImportValue typesense-vpc-VPCID
        SecurityGroupIngress:
            - IpProtocol: tcp
              FromPort: 80
              ToPort: 80
              CidrIp: 0.0.0.0/0
            - IpProtocol: tcp
              FromPort: 443
              ToPort: 443
              CidrIp: 0.0.0.0/0

We have 2 ALB listeners:

• one for HTTPS that will forward traffic to the Target Group

• one for HTTP that will redirect all requests to HTTPS

ALBHttpsListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
        DefaultActions:
            - Type: forward
              TargetGroupArn: !Ref ALBTargetGroup
        LoadBalancerArn: !Ref ALB
        Certificates:
            - CertificateArn: !Ref ALBCertificate
        Port: 443
        Protocol: HTTPS
ALBListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
        DefaultActions:
            - Type: redirect
              RedirectConfig:
                  Host: "#{host}"
                  Path: "/#{path}"
                  Port: 443
                  Protocol: "HTTPS"
                  Query: "#{query}"
                  StatusCode: HTTP_301
        LoadBalancerArn: !Ref ALB
        Port: 80
        Protocol: HTTP

And finally, the Target Group routes requests to our nodes.

The Target Group also performs health checks on our nodes, it uses Typesense's /health endpoint that will return an HTTP code 200 and {"ok": true} if our node search service is in a working state.

ALBTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
        HealthCheckIntervalSeconds: 30
        HealthCheckProtocol: HTTP
        HealthCheckTimeoutSeconds: 15
        HealthyThresholdCount: 5
        HealthCheckPath: /health
        Matcher:
            HttpCode: 200
        Name: !Sub ${AWS::StackName}-${EnvironmentType}
        Port: 8108
        Protocol: HTTP
        UnhealthyThresholdCount: 3
        VpcId: !ImportValue typesense-vpc-VPCID

So we have two health checks in place for our highly available search service:

• Auto Scaling group that checks if our instance is healthy

• Load Balancer health checks will check if our application running on the instance is healthy

Node Launching

The Launch Configuration will be used by the Auto Scaling group to configure our EC2 instances (search nodes).

LaunchConfig:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
        InstanceType: !Ref InstanceType
        ImageId: !Ref LatestLinuxAmiId
        IamInstanceProfile: !GetAtt EC2InstanceProfile.Arn
        SecurityGroups:
            - !Ref EC2SecurityGroup
        UserData:    
            Fn::Base64: !Sub |
                # [...] removed for brevity plese check below

Launch Configuration UserData

UserData allows us to run commands on our instance at launch. When an unhealthy instance is terminated and a new one is created and the UserData will run the commands to ensure the cluster node is reconfigured back to its running state.

1. CloudWatch Agent

We want to monitor our nodes, CloudWatch Agent will collect metrics and logs from our EC2 instances and send them to CloudWatch. The CloudWatch configuration is created by the CloudFormation template and referenced in the commands below.

# install cloudwatch agent
yum -y install amazon-cloudwatch-agent
# auto configure cloudwatch agent
/opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c ssm:${CloudWatchAgentConfig} -s

2. Download & Install Typesense Server

We'll run the service on T4g instances powered by Graviton2 processors running Amazon Linux and there's no package available, so we have to download the arm64 Binary.

# download & unarchive typesense
curl -O https://dl.typesense.org/releases/${TypesenseVersion}/typesense-server-${TypesenseVersion}-linux-arm64.tar.gz
tar -xzf typesense-server-${TypesenseVersion}-linux-arm64.tar.gz -C /opt/typesense
# remove archive
rm typesense-server-${TypesenseVersion}-linux-arm64.tar.gz

3. Configure the Typesense Node

We get the EC2 Instance Private IP, create the Typesense data & log folders and the nodes and server config file. Read more about Typesense server config.

# get instance private IP
EC2_INSTANCE_IP=$(curl -s http://instance-data/latest/meta-data/local-ipv4)

# create typesense folders
mkdir -p /opt/typesense/data
mkdir -p /opt/typesense/log

# create the nodes file
echo "$EC2_INSTANCE_IP:${TypesensePeeringPort}:${TypesenseApiPort}" > /opt/typesense/nodes

# create typesense server config file
echo "[server]
api-key = ${TypesenseApiKey.Value}
data-dir = /opt/typesense/data
log-dir = /opt/typesense/log
enable-cors = true
api-port = ${TypesenseApiPort}
peering-port = ${TypesensePeeringPort}
peering-address = $EC2_INSTANCE_IP
nodes = /opt/typesense/nodes" > /opt/typesense/typesense.ini

4. Create systemd service & enable the daemon

As we installed it from a binary we have to create a systemd service for the Typesense server. This will make the Typesense server service always available.

# create typesense service
echo "[Unit]
Description=Typesense service
After=network.target

[Service]
Type=simple
Restart=always
RestartSec=5
User=ec2-user
ExecStart=/opt/typesense/typesense-server --config=/opt/typesense/typesense.ini

[Install]
WantedBy=default.target" > /etc/systemd/system/typesense.service

# start typesense service
systemctl start typesense
# enable typesense daemon
systemctl enable typesense

5. Notify Cluster

We publish a message using AWS CLI on the SNS topic that a new Typesense node is available.

# notify sns topic of the new instance
aws sns publish --message "New Instance Ready (instanceId:$EC2_INSTANCE_IP)" --topic-arn ${EC2EventsTopic}  --region ${AWS::Region}

Cluster Events

Every Typesense node needs to be aware of the other cluster nodes in order to automatically replicate the data. When a node instance is terminated or a new node is launched we need to update the configuration on the rest of the nodes.

An SNS topic will receive notifications from the Auto Scaling Group in case of an EC2_INSTANCE_TERMINATE event and also from the AWS CLI (UserData) once the new node is running.

The SNS will then trigger a Lambda that will send a command through the Systems Manager to all the healthy instances to update the nodes definition file.

Click here to view the cluster events listener Lambda code.

Systems Manager Run Command

AWS SSM helps us securely manage the configuration of our nodes and allows running a shell script on multiple instances at once. This facilitates updating the nodes definition of our Typesense service.

# SSM Document
UpdateNodesCommand:
    Type: AWS::SSM::Document
    Properties:
        Content:
            schemaVersion: "2.2"
            description: Update Typesense Nodes
            parameters:
                NodeList:
                    type: String
                    description: Nodes string
            mainSteps:
                - action: aws:runShellScript
                  name: updateTypesense
                  inputs:
                      runCommand:
                          - echo "{{NodeList}}" > /opt/typesense/nodes
                          - sudo systemctl restart typesense

        DocumentFormat: YAML
        DocumentType: Command
        Name: !Sub ${AWS::StackName}-${EnvironmentType}-update-nodes
        TargetType: /AWS::EC2::Instance

EC2 IAM Role & Security Groups

The template will create an Instance Profile, an IAM Role, and a Security Group for the EC2 instances.

The IAM Role will be assumed by the EC2 instance and contains policies that will allow only the necessary actions (principle of least privilege).

• CloudWatchAgentServerPolicy - to read instance information and write it to CloudWatch Logs and Metrics

• AmazonSSMManagedInstanceCore - to receive messages for Run Command and read parameters in Parameter Store (CloudWatch agent config)

• sns:Publish - publish SNS message when a new node is ready

EC2InstanceRole:
    Type: AWS::IAM::Role
    Properties:
        RoleName: !Sub ${AWS::StackName}-${EnvironmentType}-instance-role
        AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
                - Effect: Allow
                  Principal:
                      Service:
                          - ec2.amazonaws.com
                  Action:
                      - sts:AssumeRole
        Path: /
        ManagedPolicyArns:
            - arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
            - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
        Policies:
            - PolicyName: !Sub ${AWS::StackName}-${EnvironmentType}-ec2-policy
              PolicyDocument:
                  Version: "2012-10-17"
                  Statement:
                      - Effect: Allow
                        Action:
                            - sns:Publish
                        Resource:
                            - !Ref EC2EventsTopic

The Instance Profile allows passing the IAM role to the EC2 instance.

The Security Group permits the EC2 instances to:

• receive traffic on Typesense API Port 8108 from the Load Balancer

• communicate with the other VPC nodes on API Port 8108 and Peering Port 8107

EC2SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
        GroupDescription: allow connections from ALB and SSH, from other instances within VPC
        VpcId: !ImportValue typesense-vpc-VPCID
        SecurityGroupIngress:
            - IpProtocol: tcp
              FromPort: 8108
              ToPort: 8108
              SourceSecurityGroupId: !GetAtt ALBSecurityGroup.GroupId
            - IpProtocol: tcp
              FromPort: 8107
              ToPort: 8107
              CidrIp: !ImportValue typesense-vpc-VPCCidrBlock
            - IpProtocol: tcp
              FromPort: 8108
              ToPort: 8108
              CidrIp: !ImportValue typesense-vpc-VPCCidrBlock

Typesense API Key Generation

The CloudFormation stack will generate a Typesense admin API key using a CustomResource that references a RandomStringGenerator Lambda function.

The TypesenseApiKeyis exported in the Output so it can be referenced in other stacks. Optionally it can be stored in a AWS::SSM::Parameter to be used by other AWS services.

The admin API key provides full control over the Typesense API, make sure you create Scoped API Keys to be used in your application.

TypesenseApiKey:
    Type: AWS::CloudFormation::CustomResource
    Properties:
        Length: 32
        ServiceToken: !GetAtt RandomStringGenerator.Arn

The random string generator Lambda function is deployed using inline code, this works for simple functions when the code length is up to 4096 chars.

Summary

The setup uses 2 AWS CloudFormation stacks that can be deployed into any AWS account:

• the Network Stack (1-click deployment) - creates the VPC and subnets in 3 AZs

• the Cluster Stack (1-click deployment)- creates all the cluster resources

To get started with the Typesense API once you have the service running, check API Reference

For educational purposes, this post and the included templates feature some common real-life implementations of the AWS Well-Architected Framework concepts such as high availability, elasticity, scalability, resiliency, security, principles of least privilege, etc.

Cover Photo by Samuel Sianipar on Unsplash

Think of your AWS account as an office building, you're the owner.

Your building has great amenities such as a gym, a fancy cafe, an underground garage, an electric car charging station, conference rooms, a technical room, and so on. Those are the AWS resources.

You don't want just anyone to be able to come into your building use the gym, grab a coffee, and mess stuff in the technical room, right?

So, you have locks on the doors and you only give an access card to people you trust, like your company employees or contractors.

AWS IAM works in a similar way, AWS resources have "locks" and require permissions to be accessed.

You can then give users (or programs) "keys" to access your AWS resources.

You can give different keys to different users or programs, and you can also decide what they can do with your resources.

General Concepts:

IAM User - an identity type - single principal (human or an application)

IAM User Group - a collection of users

IAM Role - an identity type - multiple principals (human, application, AWS service)

IAM Policy - a document specifying permissions

Some examples of AWS resources include EC2 instances, S3 buckets, RDS databases, Lambda functions, etc.

IAM User

The IAM User is a user account that is created within the AWS IAM system.

Each IAM User is given a unique username (within your AWS account) and can authenticate themselves using an access key and/or a password.

The access key is for programmatic access - users that will access your resources via AWS API, CLI, or SDK using an access key ID and secret access key.

The access key is usually for technical users such as developers, sys admins, and other applications (e.g. GitHub).

The password is for AWS Management Console - users that will access your resources via a graphical interface.

An IAM User has no permissions by default but can have both access types enabled, access key and password.

IAM User Groups

IAM User Groups are like a club that the IAM Users join and get special privileges to do certain things. For example, all IAM Users in the "DB Managers" group have permission to add and change things in a database.

So, IAM User Groups are a way to manage and organize IAM Users.

You can create groups and assign users to them, and then apply IAM policies to the group to grant or restrict access to AWS resources.

This is a useful way to manage permissions for large numbers of users, as you can make changes to the group policy and have it apply to all users in the group.

It is also a way to simplify the process of granting permissions to new users, as you can simply add them to the appropriate group instead of having to individually assign permissions.

IAM Policy

Okay, so think of IAM Policy as a document that contains all the actions you're allowed to perform, or not.

For example, your HVAC contractor will have a policy that allows access to the building but only to the underground garage, the technical room, and the roof.

Similarly, by creating different IAM policies for different IAM users or groups, you can control who has access to which resources in your AWS account.

This helps you make sure that only the right identities have access to the resources they need, and helps you keep your AWS account secure.

IAM policies are written JSON and tell AWS which actions a user or group of users is allowed to perform on which resources.

In the example below the IAM User that has attached that IAM Policy will only be allowed to view the yearly-report.xlsx file from the reports S3 bucket. They won't be able to view the other files or update the report.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowLatestReport",
            "Effect": "Allow",
            "Action": ["s3:GetObject"],
            "Resource": "arn:aws:s3:::reports/yearly-report.xlsx"
        }
    ]
}

IAM Role

IAM Role is probably the most confusing type of identity, at first.

Remember the cafe in your office building? A catering company supplies it with food and drinks daily and manages the service of the cafe equipment. The number of delivery agents can vary and they're not the same each day, and sometimes they bring in a technician.

You trust the catering company employees to assume the "catering" role and obtain temporary access to the underground parking, the elevator, and the cafe to do their job.

An IAM Role works similarly, it's used to grant specific privileges to specific entities for a set duration of time.

An entity can assume a role only if it's allowed by the Trust Policy.

Each time a role is assumed the entity will get assigned a set of temporary (up to 12 hours) security credentials that will allow to access the AWS resource if that's allowed by the Permissions Policy.

For example, you can have all the EC2 Instances in your account assume an IAM Role that will allow writing logs and metrics into AWS CloudWatch. You don't have to do that individually.

So, IAM User or IAM Role?

Use IAM User when you can identify the single principal to that you want to grant specific privileges for the longer term.

The IAM User can be a human or an application. They will authenticate using an access key id & secret access key or username & password and be authorized by the IAM Policy to perform certain actions on AWS resources.

Use IAM Role when you have multiple principals or when you don't know the number of entities that you want to temporarily grant specific privileges.

The entity assuming an IAM Role can be an IAM User or Group, an AWS service or account, a federated identity, etc.

The entity can assume an IAM Role only if it's specified in the role Trust Policy, and then it will be authorized by the IAM Policy to perform certain actions on AWS resources.

If you're serious about search and need typo tolerance, result rankings, synonyms, filtering, faceting, geo search, and whatnot, you better forget about those sub-optimal DB queries and bring a dedicated search service into the mix.

There are multiple cloud offerings for hosted search such as Elasticsearch, Algolia, Meilisearch, or Typesense cloud. But due to infrastructure architecture requirements, regulatory reasons, cost optimization, or just cost consolidation it often makes more sense to run our search service in AWS.

Note: AWS has its own search service, OpenSearch which is a fork of ElasticSearch. At the time of writing AWS just announced OpenSearch Serverless, which is in beta/preview release.

Typesense is my favorite, it's simple, fast, typo-tolerant, and open-source. I've been using it on multiple production projects handling searches on datasets of millions of entries.

For production use, it is recommended to run a multi-node Typesense cluster, however, there are plenty of scenarios when a single-node service will do the job.

In this article, I will explain how to get Typesense running as a single node using infrastructure as code and make it as resilient and robust as it can be in a single Availability Zone and how to size your instance depending on your data set.

The CloudFormation template will create the following AWS resources:

• EC2 Auto Scaling Group & Launch Configuration

• Instance Profile, IAM Roles & Security Group

• EBS Volume

• Elastic IP

• Lambda Function & CloudFormation Custom Resource

A self-healing search service

We're going to launch the instance using EC2 Auto Scaling, there are no additional fees for it, and the service can automatically determine the health status of an instance using EC2 status checks.

If the search service instance is unhealthy, it will be terminated and replaced by a new one with minimum downtime.

When an instance is terminated and a new one gets created, the data stored in the EC2 instance store is lost and the new instance will have a different IP.

That's why it is critical to have persistent block storage and an elastic IP.

EC2 Auto Scaling will make sure to have 1 healthy instance running at all times.

# ASG
AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
        LaunchConfigurationName: !Ref LaunchConfig
        DesiredCapacity: 1
        MinSize: 0
        MaxSize: 1
        AvailabilityZones:
            - !Select [0, !GetAZs ""]
        Tags:
            - Key: Name
              PropagateAtLaunch: true
              Value: !Sub ${AWS::StackName}-${EnvironmentType}

EC2 Instance Type

CPU capacity is important to handle concurrent search traffic and indexing operations, so Typesense requires at least 2 vCPUs of compute capacity to operate.

All the next-gen EC2 T4g instances have at least 2 vCPUs, they're powered by Arm-based AWS Graviton2 processors and are ideal for running applications with moderate CPU usage that experience temporary spikes in usage.

The amount of RAM required is completely dependent on the size of the data you index, but for Typesense to hold the whole index in memory, the instance memory should be 2-3X the size of the data set.

As the Typesense process, itself is quite lightweight (20MB RAM with empty dataset) a 2GB memory instance (t4g.small) will likely handle a data set of up to 1GB.

Data Storage

Amazon EC2 instance store is temporary, if the instance stops, hibernates, or terminates the data from the instance store is lost, so we need to use a durable data storage such as EBS.

Typesense stores a copy of the raw data on disk and then builds the in-memory index with the data. Then at search time, after determining the final set of documents to return in the API response, it fetches these documents (only) from the disk and puts them in the API response.

We'll need a fast and durable storage, hence we'll go for EBS gp3 volume type (next-gen General Purpose SSD).

The size of the EBS volume should be at least the size of the raw dataset.

Launch Configuration

The Launch Configuration will be used by the Auto Scaling group to configure our EC2 instance.

LaunchConfig:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
        InstanceType: !Ref InstanceType
        ImageId: !Ref LatestLinuxAmiId
        IamInstanceProfile: !GetAtt EC2InstanceProfile.Arn
        SecurityGroups:
            - !Ref EC2SecurityGroup
        UserData:
            Fn::Base64: !Sub |
                # [...] removed for brevity plese check below

Launch Configuration UserData

This is where most of the auto-healing magic happens, UserData allows us to run commands on our instance at launch. When an unhealthy instance is terminated and a new one is created it will be empty.

UserData will run the commands to ensure the search service is reconfigured back to its running state.

1. Associate the IP & attach the EBS volume

#!/usr/bin/env bash
# get instance-id
EC2_INSTANCE_ID=$(curl -s http://instance-data/latest/meta-data/instance-id)

# associate the Elastic IP address to the instance
aws ec2 associate-address --instance-id $EC2_INSTANCE_ID --allocation-id ${ElasticIP.AllocationId} --allow-reassociation  --region ${AWS::Region}

# attach volume
aws ec2 attach-volume --device ${VolumeDevice} --instance-id $EC2_INSTANCE_ID --volume-id ${EBSVolume} --region ${AWS::Region}

# wait for volume to be properly attached
DATA_STATE="unknown"
until [ "${!DATA_STATE}" == "attached" ]; do
    DATA_STATE=$(aws ec2 describe-volumes \
        --region ${AWS::Region} \
        --filters \
            Name=attachment.instance-id,Values=${!EC2_INSTANCE_ID} \
            Name=attachment.device,Values=${VolumeDevice} \
        --query Volumes[].Attachments[].State \
        --output text)

    sleep 5
done

# format the volume if doesn't have file system (on stack creation)
blkid --match-token TYPE=ext4 ${VolumeDevice} || mkfs.ext4 -m0 ${VolumeDevice}

# create mount root folder
mkdir -p ${VolumeMountPath}

# mount the volume to folder
mount ${VolumeDevice} ${VolumeMountPath}

# persist the volume on restart
echo "${VolumeDevice} ${VolumeMountPath} ext4 defaults,nofail 0 2" >> /etc/fstab

2. Configure the Typesense Server

If you're using the CloudFormation template defaults the VolumeMountPath is set to /mnt/ebs hence:

• The config file will be at /mnt/ebs/typesense.ini

• Logs will be under /mnt/ebs/log/

• Data will be stored under /mnt/ebs/data/

# create typesense server config
if [ ! -f ${VolumeMountPath}/typesense.ini ]
then
    echo "[server]
api-key = ${TypesenseApiKey.Value}
data-dir = ${VolumeMountPath}/data
log-dir = ${VolumeMountPath}/log
api-port = ${TypesensePort}" > ${VolumeMountPath}/typesense.ini
    chown ec2-user:ec2-user ${VolumeMountPath}/typesense.ini
fi

# create typesense data folder if not exists
if [ ! -d ${VolumeMountPath}/data ]
then
    mkdir -p ${VolumeMountPath}/data
    chown ec2-user:ec2-user ${VolumeMountPath}/data
fi

# create typesense logs folder if not exists
if [ ! -d ${VolumeMountPath}/log ]
then
    mkdir -p ${VolumeMountPath}/log
    chown ec2-user:ec2-user ${VolumeMountPath}/log
fi

3. Download & Install Typesense Server

As we're running the service on a T4g instance powered by Graviton2 processors we have to download the arm64 Binary.

# download & unarchive typesense
curl -O https://dl.typesense.org/releases/${TypesenseVersion}/typesense-server-${TypesenseVersion}-linux-arm64.tar.gz
tar -xzf typesense-server-${TypesenseVersion}-linux-arm64.tar.gz -C /home/ec2-user
# remove archive
rm typesense-server-${TypesenseVersion}-linux-arm64.tar.gz

4. Create systemd service & enable the daemon

As we installed it from a binary we have to create a systemd service for the Typesense server. This will make the Typesense server service always available.

# create typesense service
echo "[Unit]
Description=Typesense service
After=network.target

[Service]
Type=simple
Restart=always
RestartSec=5
User=ec2-user
ExecStart=/home/ec2-user/typesense-server --config=${VolumeMountPath}/typesense.ini

[Install]
WantedBy=default.target" > /etc/systemd/system/typesense.service

# start typesense service
systemctl start typesense
# enable typesense daemon
systemctl enable typesense

EBS & Elastic IP

Important: the instance needs to be launched in the same Availability Zone as the EBS volume.

Tip: to save costs on dev instances, you can create EC2 Auto Scaling scheduled actions that set autoscaling to 0 at night and back to 1 during work hours.

EBSVolume:
    Type: AWS::EC2::Volume
    Properties:
        Size: !Ref VolumeSize
        VolumeType: gp3
        AvailabilityZone: !Select [0, !GetAZs ""]
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName}-${EnvironmentType}
    DeletionPolicy: Snapshot

ElasticIP:
    Type: AWS::EC2::EIP
    Properties:
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName}-${EnvironmentType}

EC2 IAM Role & Security Groups

The template will create an Instance Profile, an IAM Role, and a Security Group.

The IAM Role will be assumed by the EC2 instance so it can attach the EBS volume and self-associate the Elastic IP. These commands are run using AWS CLI in the Launch Configuration UserData .

The Instance Profile allows passing the IAM role to the EC2 instance.

The Security Group permits the instance to receive traffic on the port used by the Typesense server (8108) and on port 22 for SSH access.

Note: a LambdaExecutionRole is also created to allow the API Key generation function to write logs to AWS CloudWatch, but it's omitted for brevity.

EC2InstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
        InstanceProfileName: !Sub ${AWS::StackName}-${EnvironmentType}-instance-profile
        Path: /
        Roles:
            - !Ref EC2InstanceRole

EC2InstanceRole:
    Type: AWS::IAM::Role
    Properties:
        RoleName: !Sub ${AWS::StackName}-${EnvironmentType}-instance-role
        AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
                - Effect: Allow
                  Principal:
                      Service:
                          - ec2.amazonaws.com
                  Action:
                      - sts:AssumeRole
        Path: /
        Policies:
            - PolicyName: !Sub ${AWS::StackName}-${EnvironmentType}-instance-policy
              PolicyDocument:
                  Version: "2012-10-17"
                  Statement:
                      # allow attaching EBS volumes
                      - Effect: Allow
                        Action:
                            - ec2:AttachVolume
                            - ec2:DescribeVolumes
                        Resource: "*"
                      # allow Elastic IP association
                      - Effect: Allow
                        Action:
                            - ec2:AssociateAddress
                        Resource: "*"
# SG
EC2SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
        GroupDescription: allow connections on HTTP & SSH
        SecurityGroupIngress:
            # Allow HTTP traffic on Typesense port
            - IpProtocol: tcp
              FromPort: !Ref TypesensePort
              ToPort: !Ref TypesensePort
              CidrIp: 0.0.0.0/0
            # Allow SSH - !! narrow CidrIp down, too broad
            - IpProtocol: tcp
              FromPort: 22
              ToPort: 22
              CidrIp: 0.0.0.0/0

Typesense API Key Generation

The CloudFormation stack will generate a Typesense admin API key using a CustomResource that references a RandomStringGenerator Lambda function.

The TypesenseApiKeyis exported in the Output so it can be referenced in other stacks. Optionally it can be stored in a AWS::SSM::Parameter to be used by other AWS services.

The admin API key provides full control over the Typesense API, make sure you create Scoped API Keys to be used in your application.

TypesenseApiKey:
    Type: AWS::CloudFormation::CustomResource
    Properties:
        Length: 32
        ServiceToken: !GetAtt RandomStringGenerator.Arn

The random string generator Lambda function is deployed using inline code, this works for simple functions when the code length is up to 4096 chars.

RandomStringGenerator:
    Type: AWS::Lambda::Function
    Properties:
        FunctionName: !Sub ${AWS::StackName}-${EnvironmentType}-random-string
        Code:
            ZipFile: >
                const response = require("cfn-response"), crypto = require("crypto");
                exports.handler = (event, context) =>{
                    response.send(event, context, response.SUCCESS, {Value:crypto.randomBytes(parseInt(event['ResourceProperties']['Length'])).toString('hex')});
                };
        Handler: index.handler
        Role: !GetAtt LambdaExecutionRole.Arn
        Runtime: nodejs16.x
        MemorySize: 128
        Timeout: 10

One-click Deployment to get going quickly, or click to view the complete self-contained Cloudformation template that can be deployed into any AWS account.

For complete details about Typesense API check API Reference

Summary

Although this CloudFormation stack template is made especially to get a Typesense search service up and running in minutes, it can be repurposed easily, e.g. for installing WordPress instead of Typesense.

For educational purposes, it features some common real-life scenarios and pattern examples such as:

• Create a single self-healing instance with EC2 Auto Scaling

• Use Launch Configuration and UserData to run commands on the instance at launch

• Persistent auto-mounting EBS storage + creating a filesystem on new volumes

• Auto-install a service from a Linux Binary

• Creating a Linux systemd service & daemon

• Use of Metadata to group and sort CloudFormation parameters

• How to deploy a Lambda function with inline code

• Use of Custom Resource to trigger a Lambda and reference the output

• Use of CloudFormation Intrinsic Functions (!Ref, !GetAtt, !Sub, !Select, !GetAZs)

Cover photo by Marten Newhall on Unsplash

Working with DynamoDB requires a different mindset than working with relational databases.

Here are 8 bite-sized tips:

1. Know your access patterns in advance

Spend more time upfront so you won’t regret it later.

Before you design your DynamoDB schema, know the questions it will need to answer, and write down all the access patterns.

Use AWS NoSQL Workbench to model your data.

2. Keep related data together

Maintain as few tables as possible, if modeling relational data, aim for a single-table design.

Use generic attributes for indexes, have fun with index overloading.

If you want to see a real-life example check out my other article on how I'm modeling data for monithor.co.

Also, make sure you watch @alexbdebrie explain single-table design.

3. Avoid Scans

Scan operations are less efficient as they always scan the entire table and then filter out values to return the result.

As a table or index grows, the Scan operation slows. Avoid it on large datasets, but if there are no other options use Parallel Scan.

4. Use ConditionExpression to prevent overwriting

If an item with the same primary key exists in the table, the PutItem operation will replace it.

Use ConditionExpression to avoid overwriting in a single call Instead of doing 2 calls (a GetItem before PutItem).

5. Use GSI Write Sharding to avoid hot partitions

Distribute items evenly across key Aspace to avoid hot partitions. Do that by expanding the space, add a random number or a calculated suffix to the partition key.

Use parallel scatter-gather queries to retrieve the results.

6. Use Transactions to ensure consistency

Group multiple actions together and submit them as a single all-or-nothing operation, if one action fails they’ll all fail.

Don't group unnecessary operations in a transaction, simpler transactions are more likely to succeed.

7. Use TTL to delete expired items

Leverage the TTL (Time-to-Live) to auto delete items. E.g. auto-remove log data older than X days from a table just by setting the TTL to the desired timestamp.

Just pay attention to the items you set the TTL attribute to.

8. Use On-Demand Pricing at first

Yes, you'll read that it can be about 7x more expensive, but unless your DynamoDB bill is $$$ it won’t worth the headache.

Once you know your usage patterns you can switch to Provisioned and fine-tune it.

Bonus

Watch @houlihan_rick deep-dive videos on DynamoDB advanced design patterns AWS reInvent.

Search YouTube and watch them all from 2017 to date, repetition is the mother of all learning.

click here to jump straight to the technical part

Leaving the landing page by the very end, when in my acceptance the project is "ready" to launch is a mistake I did way too many times in the past. Nearly all my abandoned projects were code-first, marketing-later.

The pattern is quite common among fellow developers: buy a domain ⟶ code like crazy for a few weeks ⟶ run out of steam / find a new project idea ⟶ abandon the project ⟶ repeat. Very few projects even have the chance to get in front of potential users.

Although this article is published as the 3rd in the series, the landing page was the first thing I did for Monithor. A landing page helps you look at things from the other side of the fence, and writing the copy gives clarity and helps better define the scope.

I wanted to avoid getting lost in details and spend a lot of time designing & coding the template for the landing page, so I went straight to HTMLRev and got a great-looking template. I highly recommend Lucian's work, crazy attention to detail, and the code quality is great.

The logo and the color palette are from logology, they have a really fun and quick process and the result is great.

I glued the pieces together, wrote the copy, and a day later I had a great-looking landing page that was ready to be deployed. I usually deploy static sites to Netlify, but as Monithor will be built entirely on top of AWS it made more sense to have it all in one place.

Deploy to AWS

Deploying a static website to AWS might sound simple, but if you want a custom domain with an SSL certificate, and a proper non-www to www redirect, things get messy. It's not complicated, but it's a very cumbersome process to set it up from the AWS console, I did it so many times that I got fed up.

As a developer, fan of Infrastructure as Code (IaC), I decided to create a CloudFormation template that will save you time in the future.

The standard way

But first, let me describe the standard way of setting up a static website on AWS to understand why I want to avoid it in the future:

1. create two S3 buckets, one for the naked domain (non-www) and one for the www
2. enable static website hosting on buckets
3. set the non-www bucket to redirect requests to the www domain (or the other way around)
4. request a certificate to cover both the non-www and the www domain
5. add CNAMES in the Route53 hosted zone to validate the domain for the certificate
6. wait for the certificate to be issued
7. create two CloudFront distributions, one for each of the buckets
8. make sure each distribution Origin points to the bucket website endpoint, not the bucket URL
9. go back to Route53 and create alias entries to point the domain to the CloudFront distros
10. done, if we didn't make any mistake

The easy way

The CloudFormation template will save you time and frustration:

• creates a single S3 bucket with the required policy
• creates a single CloudFront distribution and sets the proper Origin
• issues and installs the SSL certificate
• automates all the required Route53 DNS records
• manages redirects with a CloudFront Function

You simply type npm run create-stack command in the terminal and wait for the magic to happen. It will take about 5 minutes to complete though, you can monitor the progress from the AWS Console

You just need to:

• have the domain purchased and pointed to the hosted zone created for it in AWS Route53 before creating the stack
• have AWS CLI installed and a user with AdministratorAccess

Create the stack

The create-stack command is as below:

aws cloudformation create-stack \
--stack-name STACK_NAME \
--template-body file://template.yml \
--profile AWS_PROFILE \
--region us-east-1 \
--parameters ParameterKey=DomainName,ParameterValue=DOMAIN_NAME ParameterKey=HostedZoneId,ParameterValue=HOSTED_ZONE_ID

Where

• STACK_NAME - is your stack name
• AWS_PROFILE - is your credentials profile in the
• DOMAIN_NAME - is your domain name (the naked domain, e.g. example.com)
• HOSTED_ZONE_ID - is your Route53 zone for the domain, should look like Z07414553OA4KK51T5EZA

Deploy the site

The deploy command will deploy your static site to the S3 bucket and invalidate the CloudFront cache. You may update as per your needs, by default the copy command excludes all paths in the current folder, in order to prevent deploying sensitive info.

aws s3 cp ./ s3://S3_BUCKET_NAME \
--acl public-read \
--profile AWS_PROFILE \
--recursive \
--exclude "*" \
--include index.html \
--include 404.html \
&& aws cloudfront create-invalidation \
--profile monithor \
--distribution-id DISTRIBUTION_ID \
--paths "/*"

Where:

• S3_BUCKET_NAME - is your S3 bucket name
• AWS_PROFILE - is your credentials profile
• DISTRIBUTION_ID - is your CloudFront distribution ID

And here's the CloudFormation template:

Description: Deploy a static site

Parameters:
    DomainName:
        Description: Domain name
        Type: String
    HostedZoneId:
        Description: Hosted Zone ID
        Type: String

Resources:
    S3Bucket:
        Type: AWS::S3::Bucket
        DeletionPolicy: Delete
        Properties:
            AccessControl: PublicRead
            BucketName: !Sub "${AWS::StackName}"
            WebsiteConfiguration:
                ErrorDocument: "404.html"
                IndexDocument: "index.html"

    S3BucketPolicy:
        Type: AWS::S3::BucketPolicy
        Properties:
            Bucket: !Ref S3Bucket
            PolicyDocument:
                Version: "2012-10-17"
                Statement:
                    - Effect: "Allow"
                      Action: "s3:GetObject"
                      Principal: "*"
                      Resource: !Sub "${S3Bucket.Arn}/*"
    #
    CertificateManagerCertificate:
        Type: AWS::CertificateManager::Certificate
        Properties:
            # naked domain
            DomainName: !Ref DomainName
            # add www to certificate
            SubjectAlternativeNames:
                - !Sub "www.${DomainName}"
            ValidationMethod: DNS
            DomainValidationOptions:
                # DNS record for the naked domain
                - DomainName: !Ref DomainName
                  HostedZoneId: !Ref HostedZoneId
                # DNS record for the www domain
                - DomainName: !Sub "www.${DomainName}"
                  HostedZoneId: !Ref HostedZoneId
    #
    CloudFrontDistribution:
        Type: AWS::CloudFront::Distribution
        Properties:
            DistributionConfig:
                Aliases:
                    - !Ref DomainName
                    - !Sub "www.${DomainName}"
                CustomErrorResponses:
                    - ErrorCachingMinTTL: 60
                      ErrorCode: 404
                      ResponseCode: 404
                      ResponsePagePath: "/404.html"
                DefaultCacheBehavior:
                    AllowedMethods:
                        - GET
                        - HEAD
                    CachedMethods:
                        - GET
                        - HEAD
                    Compress: true
                    DefaultTTL: 86400
                    ForwardedValues:
                        Cookies:
                            Forward: none
                        QueryString: true
                    MaxTTL: 31536000
                    SmoothStreaming: false
                    TargetOriginId: !Sub "S3-${AWS::StackName}"
                    ViewerProtocolPolicy: "redirect-to-https"
                    FunctionAssociations:
                        - EventType: viewer-request
                          FunctionARN: !GetAtt RedirectFunction.FunctionMetadata.FunctionARN
                DefaultRootObject: "index.html"
                Enabled: true
                HttpVersion: http2
                IPV6Enabled: true
                Origins:
                    - CustomOriginConfig:
                          HTTPPort: 80
                          HTTPSPort: 443
                          OriginKeepaliveTimeout: 5
                          # keep http-only to avoid 504 errors after stack creation
                          OriginProtocolPolicy: "http-only"
                          OriginReadTimeout: 30
                          OriginSSLProtocols:
                              - TLSv1
                              - TLSv1.1
                              - TLSv1.2
                      #Bucket website endpoint without http://
                      DomainName: !Join
                          - ""
                          - - !Ref S3Bucket
                            - ".s3-website-"
                            - !Ref AWS::Region
                            - ".amazonaws.com"
                      Id: !Sub "S3-${AWS::StackName}"
                PriceClass: PriceClass_All
                ViewerCertificate:
                    AcmCertificateArn: !Ref CertificateManagerCertificate
                    MinimumProtocolVersion: TLSv1.1_2016
                    SslSupportMethod: sni-only

    RedirectFunction:
        Type: AWS::CloudFront::Function
        Properties:
            AutoPublish: true
            Name: !Sub "${AWS::StackName}-redirects"
            # add the config, even if optional, the stack creation will thrown InternalFailure error otherwise
            FunctionConfig:
                Comment: !Sub "Redirect to ${DomainName}"
                Runtime: cloudfront-js-1.0
            FunctionCode: !Sub |
                function handler(event) {
                    //
                    var request = event.request;
                    var host = request.headers.host.value;

                    if (!host.startsWith("www.")) {
                        return {
                            statusCode: 301,
                            statusDescription: "Permanently moved",
                            headers: {
                                location: { value: "https://www." + host },
                            },
                        };
                    }
                    return request;
                }

    Route53RecordSetGroup:
        Type: AWS::Route53::RecordSetGroup
        Properties:
            # keep the . suffix
            HostedZoneName: !Sub "${DomainName}."
            # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-route53-aliastarget.html#cfn-route53-aliastarget-hostedzoneid
            RecordSets:
                - Name: !Ref DomainName
                  Type: A
                  AliasTarget:
                      DNSName: !GetAtt CloudFrontDistribution.DomainName
                      EvaluateTargetHealth: false
                      HostedZoneId: Z2FDTNDATAQYW2 # leave hardcoded, don't confuse w/ !Ref HostedZoneId
                - Name: !Sub "www.${DomainName}"
                  Type: A
                  AliasTarget:
                      DNSName: !GetAtt CloudFrontDistribution.DomainName
                      EvaluateTargetHealth: false
                      HostedZoneId: Z2FDTNDATAQYW2 # leave hardcoded, don't confuse w/ !Ref HostedZoneId

Outputs:
    WebsiteURL:
        Value: !GetAtt S3Bucket.WebsiteURL
        Description: URL for website hosted on S3
    CloudfrontDomainName:
        Value: !GetAtt CloudFrontDistribution.DomainName

Truth be told, it looks much more complex than it is. I also created a GitHub repository with the template so you can get started quicker. Check it out.

Credits

• Nickolas Kraus
• AWS CloudFormation Functions
• AWS CloudFormation
• Cover photo by Sebastien Gabriel

About

Monithor is a web service monitoring tool that helps you check websites or APIs' uptime and functionality and be alerted when they don't work as expected. It's built on AWS using the Serverless framework and it's open-source

There are various ways to run cronjobs on a schedule in AWS, I analyzed the options that follow the serverless principles. Although all of them can perform pretty well in the case of a single-tenant system, we'll offer a cloud-hosted plan, which is a multi-tenant system that needs to scale.

Solution requirements:

• precision - run as close to the scheduled time, preferably within seconds, an occasional execution delay up to a couple of seconds is acceptable
• scale - possibly tens of thousands of scheduled monitoring jobs, occasionally thousands to be executed around the same time
• costs - keep the AWS bill within reasonable limits

Now, let's dig into the options that AWS offers out of the box:

1. DynamoDB TTL + Streams

Set a Lambda function to process the DynamoDB Streams and react to data modifications. Each monitor job would have a DynamoDB entry with the expiration time (TTL) equal to the scheduled execution time. Once the entry is deleted would trigger the monitor runner logic.

Problem: The lack of precision.

As per DynamoDB docs TTL typically deletes expired items within 48 hours of expiration. - this means a task could be triggered minutes if not hours after the scheduled time.

2. EventBridge Rules (CloudWatch Events)

Create an EventBridge Rule for every monitor and set the Lambda containing the runner logic as the rule target.

Problem: The service quotas.

At the time of writing EventBridge Rules are limited to 300 per event bus and a maximum of 100 event buses per account. These are soft limits that will suffice for plenty of use cases, and can probably be increased on request. But the low initial limit is deterring.

3. Step Functions

We could make use of the wait state of the Step Functions and pause the workflow until the monitor's scheduled time of execution, it can be delayed even up to an entire year.

Problem: Relatively high cost.

The solution

Given that none of the out-of-the-box solutions tick all the requirements we have to start playing with the building blocks that AWS has to offer.

DynamoDB already stores all our scheduled tasks, including the time when they should run, so we'll create two Lambda functions: a Scheduler and a Runner (or executor). The Scheduler will pull the jobs to be executed in the next interval from DynamoDB and publish them into a SQS queue that will be immediately consumed by Runners.

The Scheduler Function

The Scheduler function will be triggered by an EventBridge Rule once a minute.

The scope of the Scheduler function is to Query all scheduled monitoring jobs in the system (possibly thousands) and select those that are due for execution in the subsequent whole-minute interval. For example, the scheduler function that runs at 6:58 am will get all the monitors that need to run between 6:59:00 and 6:59:59 am. It would then enqueue those monitors into the SQS queue.

SQS allows us to set DelaySeconds parameter that will keep the message published to the queue invisible to consumers until the scheduled execution time.

The message payload contains all the info necessary for the Runner to execute the monitoring job, e.g. monitor config, project variables, and notification settings.

A drawback of this approach is that a monitoring job once sent to the queue to be executed can't be canceled but given the short time interval that's acceptable.

Scaling the Scheduler

The Scheduler function could become a bottleneck in case thousands of jobs have to be executed at the same time. There are intervals of the day that are prone to become hotspots, think the first minute of a day, or the first minute of every hour.

We added a dedicated global secondary index (RUNNER) that will help to Query which items are to be sent to the SQS queue. It uses the write sharding technique to better distribute writes across a partition key space and a scatter-gather query.

The tests show that it has a big impact on the DynamoDB query performance, so we can safely assume that we can get and send thousands, if not tens of thousands of monitoring jobs to the SQS queue in a matter of seconds.

If you want to dig deeper into Monithor's data model, I wrote an extensive post on the matter.

Here's how the Scheduler function code looks and how scatter-gather queries are done:

// get the monitors of this cron interval
    const promises = [],
        sqsMessages = [],
        sqsPromises = [],
        // create shards array starting 0
        shards = [...Array(process.env.SHARDS_COUNT).keys()];

    // loop shards - scatter-gather query
    for (var idx in shards) {
        promises.push(
            ddb
                .query({
                    TableName: process.env.DDB_TABLE,
                    IndexName: "RUNNER",
                    KeyConditionExpression:
                        "#RUNNER = :runner AND #SK BETWEEN :t1 AND :t2",
                    FilterExpression: "#status = :status",
                    ExpressionAttributeNames: {
                        "#RUNNER": "RUNNER",
                        "#SK": "nextrun_at",
                        "#status": "status",
                    },
                    ExpressionAttributeValues: AWS.DynamoDB.Converter.marshall({
                        ":runner": "runner#" + idx,
                        ":status": "ENABLED", // only the enabled monitors
                        ":t1": t1, // next rounded minute start
                        ":t2": t1 + 59,
                    }),
                    ScanIndexForward: false,
                })
                .promise()
        );
    }

    // await all promises to finish // parallel
    await Promise.allSettled(promises)
        .then((res) => {
            res.forEach((v) => {
                if (v.value.Items) {
                    v.value.Items.forEach((item) => {
                        // unmarshal item
                        item = AWS.DynamoDB.Converter.unmarshall(item);
                        // create SQS mesage
                        sqsMessages.push({
                            Id: item.PK.replace("monitor#", ""),
                            MessageBody: JSON.stringify(item),
                            // delay the diff in seconds until scheduled time
                            DelaySeconds:
                                item.nextrun_at -
                                Math.floor(+new Date() / 1000),
                        });
                    });
                }
            });
        })
        .catch((err) => {
            console.log(err.message);
            return [];
        });

    // no monitors to enqueue, stop
    if (!sqsMessages.length) {
        return [];
    }

    // group sqs messages into chunks of 10 for batch sending
    chunkArray(sqsMessages, 10).forEach((chunk) => {
        sqsPromises.push(
            sqs
                .sendMessageBatch({
                    Entries: chunk,
                    QueueUrl: process.env.QUEUE,
                })
                .promise()
        );
    });

    // wait for the sqs messages to be published
    await Promise.allSettled(sqsPromises)
        .then()
        .catch((err) => {
            console.log(err);
        });

The Runner function

The Runner function will be triggered by the SQS queue, will receive up to 10 monitoring jobs and execute those in parallel. The execution of a runner consists in making the monitoring request and asserting the results and writing in the DynamoDB table a log entry and updating the monitor's next execution time and counters.

Depending on the notification settings, the assertion result is further published to an SNS topic that triggers a Lambda fan-out function that handles the notification sending (email, webhooks, etc).

Scaling the Runner

When monitoring jobs are available in the queue, Lambda reads up to 5 batches and sends them to our Runner function. If jobs are still available, Lambda increases the number of processes that are reading batches by up to 60 more instances per minute.

This will allow us to run thousands of monitoring jobs per minute.

Resources

• Yan Cui's post on Using CloudWatch and Lambda to implement ad-hoc scheduling
• Michael Bahr's post on Serverless Scheduler

About

Monithor is a web service monitoring tool that helps you check websites or APIs' uptime and functionality and be alerted when they don't work as expected. It's built on AWS using the Serverless framework and it's open-source.

After more than a decade of dealing mostly with relational databases, using DynamoDB requires looking at things from a different angle, unlike traditional SQL databases, it does not use table joins and other relational database constructs. However, we can model many common relational designs in a single DynamoDB table but the process is different.

DynamoDB Table Basics

Before getting deeper into things let's summarize the DynamoDB table basic concepts:

• Partition (hash) key (PK) - defines in which partition the data is stored into
• Sort (range) key (SK) - defines the sorting of items in the partition, and it's optional. You can query it by condition expression (=, >, <, between, begins_with, contains, in) combined with PK
• Local secondary index (LSI) - allows defining a different sort key for PK. It can only be created when the table is created
• Global secondary index (GSI) - allows defining a new partition and/or sort key. GSI is the central part of most design patterns in a single table design. Both LSI and GSI are implemented as copying all the data to a new table-like structure
• Attributes - can be scalar (single value) or complex (list, maps, sets)

There are three ways of reading data from DynamoDB:

• GetItem - retrieves a single item from a table. This is the most efficient way to read a single item because it provides direct access to the physical location of the item.
• Query - retrieves all of the items that have a specific partition key. Within those items, you can apply a condition to the sort key and retrieve only a subset of the data. Query provides quick, efficient access to the partitions where the data is stored.
• Scan - retrieves all of the items in the specified table. (This operation should not be used with large tables, because it can consume large amounts of system resources.

Key concepts when using DynamoDB

Know your access patterns in advance

You shouldn't start designing the DynamoDB schema until you know the questions it will need to answer. Understanding the business problems and the application use cases upfront is essential. Write down all the access patterns, here's what the access pattern list looks like for Monithor.

Our table can grow quite big so we want to avoid Scan at all costs so we'll create our data model so we can extract what we need using Query and GetItem.

Single table design

You should maintain as few tables as possible in a DynamoDB application, we'll go with a single table design.

Yes, having all the data in a single table might look confusing and hard to read, but it offers a rich set of flexible and efficient design patterns, better utilization of resources, and easier configuration, capacity planning, and monitoring.

When using a single table, use generic attributes for the primary key like PK (partition key), SK (sort key), GS1PK (first global secondary index partition key), GS1SK (first global secondary index sort key).

Also, identify the type of the item by prefixing keys with type. e.g. project#1234 (project = type, 1234 = id)

The DynamoDB single-table design requires a different mindset than relational database table design, I recommend using NoSQL workbench for DynamoDB for data modeling and visualization. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/workbench.html

Monithor's Table Design

• a composite primary key (PK and SK)
• a local secondary index with the sort key created_at - allows querying PK created within an interval of time
• a local secondary index with the sort key nextrun_at - allows querying PK due to be executed within a time interval
• a global secondary index (GSI) with the sort key set as table's sort key (SK)

Here's the CloudFormation template for our DynamoDB single table.

Resources:
    # DynamoDB
    ddbTable:
        Type: AWS::DynamoDB::Table
        # enable table retention on production use
        # DeletionPolicy: Retain
        Properties:
            TableName: ${self:custom.resourceName}
            AttributeDefinitions:
                - AttributeName: PK
                  AttributeType: S
                - AttributeName: SK
                  AttributeType: S
                - AttributeName: created_at
                  AttributeType: N
                - AttributeName: nextrun_at
                  AttributeType: N
            KeySchema:
                - AttributeName: PK
                  KeyType: HASH
                - AttributeName: SK
                  KeyType: RANGE
            LocalSecondaryIndexes:
                - IndexName: pk_created
                  KeySchema:
                      - AttributeName: PK
                        KeyType: HASH
                      - AttributeName: created_at
                        KeyType: RANGE
                  Projection:
                      ProjectionType: ALL
                - IndexName: pk_run
                  KeySchema:
                      - AttributeName: PK
                        KeyType: HASH
                      - AttributeName: nextrun_at
                        KeyType: RANGE
                  Projection:
                      ProjectionType: ALL
            GlobalSecondaryIndexes:
                - IndexName: GSI
                  KeySchema:
                      - AttributeName: GSI
                        KeyType: HASH
                      - AttributeName: SK
                        KeyType: RANGE
                  Projection:
                      ProjectionType: ALL

            # use the On Demand pricing
            BillingMode: PAY_PER_REQUEST
            # autodelete logs after specified time
            TimeToLiveSpecification:
                AttributeName: expires_at
                Enabled: TRUE

Relational data

Modeling relational data in DynamoDB requires reusing or overloading keys and secondary indexes for different purposes. Overly-simplified, the data model for Monithor looks as in the diagram below.

• Organization - the account that owns the projects & cloud subscription, an organization can have multiple projects (one-to-many)
• Project - the monitors are organized into projects for easier monitoring and access control, a project can have multiple monitors (one-to-many), but a project also has and belongs to many users (many-to-many)
• Monitor - the monitor will create a log entry with the result of each execution, so a monitor can have multiple logs (one-to-many)

One-to-Many relationship

Let's take aside the relationship between projects and monitors, a project can have multiple monitors. In the UI we'll have to display all the monitors of a project, so we'll have to create an API endpoint that will query all monitors by project ID.

We rely on the GSI global secondary index, for each monitor entry we'll populate the GSI with the project ID prefixed with the entity type project#.

Then, when we query on the GSI index the key conditions will be GSI=project#1234 AND SK begins_with(monitor#) which will return the information we're looking for.

The query code would look like this:

const monitors = await ddb
        .query({
            TableName: process.env.DDB_TABLE,
            IndexName: "GSI",
            KeyConditionExpression: "#GSI = :gsi AND begins_with(#SK, :prefix)",
            ExpressionAttributeNames: {
                "#GSI": "GSI",
                "#SK": "SK"
            },
            ExpressionAttributeValues: AWS.DynamoDB.Converter.marshall({
                ":gsi": "project#1234",
                ":prefix": "monitor#"
            }),
        });

And here's yet another example of a one-to-many relationship. We need to show a list of all projects within the organization dashboard UI. In this case, we'll fill the organization ID in the GSI of the project item.

Querying the GSI with the key conditions GSI=org#1234 AND SK begins_with(project#) will return all the projects of that organization.

Many-to-Many relationship

The many-to-many relationships may seem a bit difficult at first, in a relational database we sort it out with a relationship table and two JOIN statements, but in DynamoDB this is not possible.

On Monithor, the core many-to-many relationship is between projects & users. We want to have granular access control for every project, so a user can have different access levels on multiple projects. A project can have multiple users.

We rely on the table sort key to store the association between entities, we store a concatenated value of user ID & project ID plus making use of GSI.

In the app UI, once the user logs in we'll want to display a list of the projects they have access to. We know the user ID from the login session so we'll have the query the table index with the following key conditions PK=user#1234 AND SK begins_with user#1234|project#

This will return all the project IDs the user#1234 has access to.

The code will look as follows.

const monitors = await ddb
        .query({
            TableName: process.env.DDB_TABLE,
            KeyConditionExpression: "#PK = :pk AND begins_with(#SK, :prefix)",
            ExpressionAttributeNames: {
                "#PK": "PK",
                "#SK": "SK"
            },
            ExpressionAttributeValues: AWS.DynamoDB.Converter.marshall({
                ":pk": "user#1234",
                ":prefix": "user#1234|project#"
            }),
        });

There's another UI screen where we need to display a list of users that have access to a project. This time we'll query the GSI* with the following key conditions GSI=project#1234 AND SK begins_with user#

This will return all the user IDs that have access to project#1234.

Resources

If you want to get deeper into DynamoDB I can't recommend enough Alex DeBrie's The DynamoDB Book.

Also, have a look at the Best Practices for Designing and Architecting with DynamoDB.

About

Monithor is a web service monitoring tool that helps you check websites or APIs' uptime and functionality and be alerted when they don't work as expected. It's built on AWS using the Serverless framework and it's open-source.

I quickly realized that most of the reported issues are due to missing or miss configured HTTP headers. It was a good opportunity to review and summarize the security headers any website or web app should use.

The HTTP headers can be configured on the server to enhance the overall security of the website or web application, adding them won't literally make the site or app more secure, but will prevent the exploitation of potential vulnerabilities.

There are 2 types of HTTP headers we have to pay attention to:

1) The headers that protect against attacks

• X-Frame-Options
• X-Content-Type-Options
• X-XSS-Protection
• Content-Security-Policy
• Strict-Transport-Security
• Access-Control-Allow-Origin

2) The headers that leak information

• Server
• X-Powered-By

1) The headers that protect against attacks

These headers can be quite easily set using .htaccess, just make sure you have Apache's mod_headers enabled.

X-Frame-Options

Protects against clickjacking.

If a malicious website can embed your site as an iframe it may allow attackers to invoke unintended actions by the user with clickjacking.

To prevent your website from being embedded in an iframe set the X-Frame-Options to DENY.

Header always append X-Frame-Options DENY

X-Content-Type-Options

Protects against MIME sniffing.

When a malicious HTML document is served from your domain (for example, if an image uploaded to a photo service contains valid HTML markup), some browsers will treat it as an active document and allow it to execute scripts in the context of the application.

This is known as content sniffing, the automatic detection of the content type, we need to tell browsers to not sniff content.

Header set X-Content-Type-Options nosniff

X-XSS-Protection (deprecated)

Protects against cross-site scripting (XSS).

The X-XSS-Protection header has been deprecated by modern browsers and its use can introduce additional security issues on the client-side. (Better use Content-Security-Policy instead).

It is recommended to set the header as X-XSS-Protection: 0 in order to disable the XSS Auditor, and not allow it to take the default behavior of the browser handling the response.

Header set X-XSS-Protection 0

Content-Security-Policy (CSP)

Protects against cross-site scripting (XSS), clickjacking and HTML injection attacks.

The Content-Security-Policy provides an added layer to mitigate XSS attacks by restricting which scripts can be executed by the page. It's probably the most important header, but also the most tedious to configure.

Please note there's no one-size-fits-all CSP, you have to generate CSP policy specific to the resources loaded by your website.

Header set Content-Security-Policy "default-src * data: 'unsafe-inline' 'unsafe-eval'"

Strict-Transport-Security (HSTS)

Protects against man-in-the-middle attacks.

HTTP Strict Transport Security instructs the browser to access the webserver over HTTPS only.

Header set Strict-Transport-Security "max-age=10886400; includeSubDomains; preload"

Learn more about HSTS here.

Access-Control-Allow-Origin

Protects against cross-site request forgery (CSRF) attacks.

This is a CORS (Cross-Origin Resource Sharing) header. By default, modern browsers enforce the same-origin policy to prevent a web page from accessing cross-origin resources.

Many developers use a wildcard Access-Control-Allow-Origin header which explicitly opens up cross-origin requests from the attacker’s origin.

Don't set this header unless you need to allow AJAX requests from external origins, if so, be specific.

 Header set Access-Control-Allow-Origin "https://mysite.com"

.htaccess

Enable the headers that protect against attacks using these directives:

<IfModule mod_headers.c>
    # Security Headers
    Header set X-XSS-Protection 0
    Header always append X-Frame-Options DENY
    Header set X-Content-Type-Options nosniff
    Header set Strict-Transport-Security "max-age=10886400; includeSubDomains; preload"
    #Header set Content-Security-Policy "default-src * data: 'unsafe-inline' 'unsafe-eval'"
</IfModule>

2) The headers that leak information

To update these headers you'll likely need to get your hands dirty with server configuration files.

Server

This header contains information about the software used by the backend server that handles the request.

An overly-detailed Server value might make it easier for attackers to exploit known security holes.

Update the server settings to show the least info possible.

In httpd.conf update ServerTokens to "Prod"

ServerTokens Prod

Or, if you have WHM access, go to Service Configuration > Apache Configuration > Global Configuration and set Server Tokens to "Product Only".

X-Powered-By

This header reveals information about the technology used in an application and can be a valuable hint for hackers who can exploit the security weaknesses of the technology, especially when using outdated software.

It's highly recommended to remove this header.

In php.ini set:

expose_php = Off

Or, if you have WHM access, go to Software > MultiPHP INI Editor > Editor Mode tab and look for the expose_php directive and set it to "Off".

Taking a glance at Google trends we're seeing PHP interest is in a downwards spiral.

The quick adoption of client-side rendering, the new shiny tools, plus its bad reputation among developers, seem to have slowly pushed PHP in the back of people’s minds.

PHP is the most widely used server-side language and also the most hated, sometimes viscerally hated. It has been reportedly dying for at least a decade.

Yet, it's still here as the web backbone, powering at least 7 out of 10 websites.

Why so much hate on a scripting language?

• it's technically inconsistent, has some design flaws
• it's easy to learn, doesn't appeal to programming purists
• it's very popular, but lacks the hype, there's nothing sexy in building with PHP
• it’s cool to hate PHP

A scripting language is just a tool that helps web developers solve problems. Getting fanatic about a language is as dumb as getting fanatic about a hammer.

There are only two kinds of languages: the ones people complain about and the ones nobody uses.

Yes, PHP might not be the best or the most modern scripting language, but it does the job for many and has come a long way since the dreaded version 3.

The difference in performance between PHP5.6 and 7.4 is huge, and considering the improvements and capabilities added under PHP8, it looks very alive and kicking, and better yet, faster than Python or Ruby.

There are a plethora of extremely popular tools built on PHP, such as WordPress that literally dominates the CMS market, MediaWiki the software behind Wikipedia, Magento and OpenCart, two popular open-source e-commerce solutions.

There's also a vibrant and lovely community around the Laravel framework, tons of modern web apps and APIs are built on Laravel (using PHP).

Is PHP dead?

Absolutely not.

It seems like it's slowly losing its steam, but it's not going away anytime soon.

Is it still worth it to learn PHP as a new developer?

Yes.

It has a lean learning curve, and the likelihood of having to deal with PHP projects is huge. Learn the fundamentals, if you like it, dig deeper, but don't stop at it.

Continuous learning is a must for software developers, when you stop learning you quickly become irrelevant.

Do I use PHP?

Yes, on a daily basis.

I have to admit that I didn't start any new project on PHP for quite some time. But that's just because the projects were more suitable for another technology, not that I have something with PHP. Looking forward to spinning up a Laravel 8 app in the near future.

What do you think about PHP's future?

The new design features larger images, it will display thumbnails with different proportions and also makes use of "srcset" attribute to display responsive images.

The old logic works like this: the user uploads product photos > the app server creates predefined image sizes using GD Library > uploads into an S3 bucket. It does the job, but wastes server resources.

There are about 300k images to be resized, running a batch process to resize all the images into new dimensions will be time-consuming and quite costly.

Why process and store the assets that may not even be accessed by users? I'm going to process them on the fly (lazily), a resized image will only be created if a user requests that specific size.

Architecture overview

1) The user requests a resized image via Cloudfront (if the asset exists it will be returned and everything stops here)

2) Cloudfront requests the asset from the S3 bucket (using a website endpoint)

3) Because the asset doesn't exist the browser will be temporary redirected (307) to the API Gateway endpoint

4) API Gateway triggers the Lambda function

5) The Lambda function downloads the original image from the S3 bucket, resizes it, and uploads the resized image back into the bucket with the requested key

6) The API Gateway returns a permanent redirect (301) to the newly created asset Cloudfront URL

The Setup

• IAM
• Lambda
• API Gateway
• S3
• CloudFront

IAM User

We'll start by creating an IAM user that will allow our Lambda function to put the resized images on S3.

1. Log in to the AWS account and go to the Identity & Access Management (IAM)

2. Go to Users and click "Add users" > pick a username that'll be easier to remember > check "Programmatic access"

3. Click "Next" to go through to the Permissions page, click on "Attach existing policies" directly, and select AmazonS3FullAccess*

4. View and copy the API Key & Secret to a temporary place, we'll need them for the Lambda config.

Only use AmazonS3FullAccess policy for a quick demonstrative setup. In a production environment, I recommend following the principle of least privilege and set more restrictive access. The Lambda function only requires getObject & putObject permissions to our specific S3 bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject"
            ],
            "Resource": "arn:aws:s3:::crafty-products/*"
        }
    ]
}

Lambda & API Gateway

Using Serverless Framework

I'm using the Serverless framework to build and deploy my services, but it can also be done directly from the AWS Console.

If you're using Serverless this is the Lambda function config from the serverless.yml file:

# ...
functions:
    # on the fly image resizer function
    resizer:
        handler: src/resizer.handler
        memorySize: 1024
        timeout: 15
        environment:
            AWS_KEY: ${self:custom.environment.AWS_KEY}
            AWS_SECRET: ${self:custom.environment.AWS_SECRET}
            BUCKET: ${self:custom.environment.PRODUCTS_BUCKET}
            CDN_URL: ${self:custom.environment.CDN_URL}
        events:
            - http:
                  path: /resizer
                  method: GET
# ...

When the service gets deployed with Serverless (sls deploy) it will create a Lambda function and the API Gateway that will listen for the GET requests and trigger the Lambda to resize the image.

I get into more details about Serverless configs & deploy in another post: how to automatically deploy Serverless applications on AWS using GitHub Actions.

Using AWS Console

If not using Serverless, everything can be done from the AWS console, though it's quite cumbersome, here are the steps:

Creating the Lambda function

1) Go to the Lambda Functions and click "Create function"

2) Select "Author from Scratch", enter the function name and select the Node runtime (14 or newer), create the function.

3) Download the zip from GitHub, it contains the Lambda function

4) In the function's code tab, click "Upload from zip" in the top right and upload the function zip. Note that the Sharp module is quite large so you won't be able to edit the function in the AWS Console, but you can add Sharp as a Lambda Layer and maintain the ability to update your function from the AWS Console editor.

5) In the function's configuration tab we'll add the environment variables

• AWS_KEY (the IAM user access key)
• AWS_SECRET (the IAM user access secret)
• BUCKET (the S3 bucket containing the images)
• CDN_URL (the CloudFront URL, we don't have it at this step, we'll set this up later)

6) In the function's configuration, in the general configuration tab set the Memory to 1024MB and Timeout to at least 15 seconds to allocate enough resources & time for Lambda to process larger images. Default settings (128MB/3s timeout) will likely fail.

The Lambda function code

You can find the Lambda function code on GitHub

It reads the source image from an S3 bucket as a stream, pipes it to Sharp, then writes the resized version as a stream back to the S3 bucket. It uses Node.js streams under the hood to prevent loading large amount of data in the Lambda function's memory and Sharp, a high-performance Node.js module for image processing (https://sharp.pixelplumbing.com/)

The function has a set of predefined image sizes, you will have to adjust to your needs or remove entirely. The goal was to prevent malicious users increase the AWS bill by generating images that won't ever be used.

//
const s3 = new AWS.S3(),
    sizes =[
        "360x270", // larger thumb restricted
        "480h", // height restricted
        "640w",
        "1280w", // largest web image
    ];
//

The Lambda function will be triggered with a GET request like this: ... ?key=produs_227613/360x270/img.AZ70KuRzw9.jpg

The key query string parameter contains the S3 path, the desired output size, and the original asset filename.

The sizes are of 3 types:

• width x height - i.e. 360x270 will generate a width & height constricted thumbnail 360px wide and 270px high
• width only - i.e. 640w will generate an image 640px wide and the height will be automatically adjusted to maintain proportions
• height only - i.e. 640h will generate an image 640px high and the width will be automatically adjusted to maintain proportions

Creating the API Gateway

1) Go to the API Gateway and click "Create API"

2) We need an HTTP API, so click "Build" on that

3) For the integrations select Lambda and the name of the function we just created, then give the new API a name and click "Next"

4) For the routes we'll use a GET method and route "/resize" path to our Lambda integration, click "Next"

5) I'm using "prod" as a stage in the samples below, you can name the stage any way you want

6) Create the API and copy the Invoke URL somewhere, we'll need it for S3

The lambda function will now be triggered by Invoke URL + route, i.e. https://bm8w4yiv1b.execute-api.us-east-1.amazonaws.com/prod/resize

S3

1) Enable "Static website hosting"

Open AWS console go to S3 > select the bucket hosting the images > Properties > Static website hosting.

Copy the bucket website endpoint somewhere, we'll need it for the CloudFront settings.

2) Redirection rules

This will temporarily redirect any S3 object that doesn't exist to the API Gateway URL passing the S3 key as a query string. Make sure you update the hostname with yours.

[
    {
        "Condition": {
            "HttpErrorCodeReturnedEquals": "404",
            "KeyPrefixEquals": ""
        },
        "Redirect": {
            "HostName": "k28ihy1li5.execute-api.us-east-1.amazonaws.com",
            "HttpRedirectCode": "307",
            "Protocol": "https",
            "ReplaceKeyPrefixWith": "prod/resizer?key="
        }
    }
]

In other words, when a visitor tries to load a product thumbnail with this URL: https://d1p41knyaaw8le.cloudfront.net/produs_227613/360x270/img.AZ70KuRzw9.jpg

If the thumbnail doesn't exist in S3, the browser will get redirected to the API gateway URL: https://k28ihy1li5.execute-api.us-east-1.amazonaws.com/prod/resizer?key=produs_227613/360x270/img.AZ70KuRzw9.jpg

3) Bucket Policy

Open the bucket "Permissions" tab and allow GetObject action to any asset in your bucket (update the resource ARN accordingly)

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::crafty-products/*"
        }
    ]
}

CloudFront

Using a CDN to serve static assets is a no-brainer, so we'll create a new CloudFront distribution (if we don't have one already). There are 2 main things to pay attention to when setting CloudFront for lazy image processing.

Origin Domain

Enter the S3 bucket website endpoint URL, don't select the suggested S3 options from the dropdown list, should look something like this:

crafty-products.s3-website-eu-west-1.amazonaws.com

Behaviors

Having CloudFront in front of S3 brings one extra challenge when requesting an image that is not found in S3: the not found response is cached, and the user will end up in a redirect loop.

We need to define a behavior for all the image types that will be resized, in this case, jpg, jpeg & png.

Lambda config

Once the CloudFront distribution is deployed,

• if you created the Lambda using AWS Console you have to go back into the Lambda config and update the CDN_URL environment variable

• if you used Serverless update the CDN_URL variable and deploy the function again sls deploy --f resizer

That's all, it seems a bit complicated than it really is.

I wrote this post to illustrate the pattern and explain the mechanics of on-the-fly image processing. AWS has a Serverless Image Handler solution available which can be deployed via an AWS CloudFormation template.

It helps you avoid those embarrassing situations where a minor change creates a big mess in production, helps you spend less time doing boring stuff like manual testing, helps you create automated tests, visually, with no code.

I started DiffHub as a hobby side project, to scratch my own itch.

It quickly became a challenge, I really wanted to build it, also wanted it to solve problems for my consulting clients, wanted to be scalable, and wanted to hone my Serverless & AWS skills. Why not turn it into a SaaS then?

I put off some consulting projects and got head down, waking up at 4 am (without alarm) to code, and sacrificed many weekends.

I probably made all the possible mistakes that can send a product to the SaaS graveyard:

• started coding without a clearly defined scope
• didn't talk to the users enough
• didn't write documentation
• didn't have pricing so didn't charge anyone
• didn't do much marketing
• thought if I build it they will come (luckily, a bunch did)
• put things on autopilot

I'm really happy I did them, it was a joy to build DiffHub, a very fulfilling experience as a developer. I learned a lot, and not only on the dev side. I also received tons of extremely helpful feedback and got connected with a lot of smart people.

The product didn't do bad, private beta went live in late 2019, just a few weeks before Covid, beta requests started pouring in, I onboarded a dozen users, received feedback (generally positive), implemented, and repeated the process weekly.

I kept a product-building journal from the first commit, reading it now makes me chuckle, building a SaaS app is like a roller coaster, here are a few excerpts.

But 667 hours 😈 and 700+ commits later I was starting to run out of steam.

The motivation slowly faded as the big development challenge was pretty much completed, "just" needed to fix bugs, do marketing, sales, and write docs. The not-so-sexy things for a developer.

My main business is web development & consulting, in early March 2020 times got very turbulent due to Covid, lockdowns, a lot of uncertainty, being quite risk-averse I decided to put the side projects on autopilot and focus on my main biz until things settle.

Autopilot is not suitable for a SaaS business, especially when running a service in this field you need to grind, ship, talk, talk, ship, talk ... Putting things on autopilot may work for a couple of weeks, but it will kill the momentum.

We, indie hackers, have some degree of shiny new object syndrome, better not kill the momentum, eventually procrastinating with other pet projects.

A few months later, the consulting business was thankfully doing very well. I started to think about how to push DiffHub forward, but realized this product is a bigger hat than I could wear by myself, in order to move the needle I had to either jump all-in in the SaaS roller-coaster, bring someone in and look for funding.

Running a SaaS product as a (part-time) solo founder is almost a mission impossible. Very few nail it, hats off to them!

I have to admit that I also thought about shutting it down.

I let it sit for a few more weeks before making a decision, wrote down the reasons I built it, running it as a SaaS business wasn't one of them. Seems I got caught in the SaaS frenzy, again.

So, why make it open source?

I fancy the idea of building in public and give back to the community, is there a better way to do that than releasing and maintaining an open-source project?

By making it open-source:

• it continues to help people save time by automating the boring repetitive tasks
• allows running automated tests in your AWS account for free (depends on usage, but it's easy to fit within AWS free tier, and pay-as-you-go afterwards)
• eliminates the bus factor risk of a side-project ran by a solo founder
• gives users a comfort level as they can have a full picture of how their sensitive data is stored and used
• allows others to contribute to the project, improve & expand it
• allows me to give back to the community, share my learnings and learn from others

I'm very aware that open-source software is aimed at enthusiasts, self-hosting requires some expertise, and it adds additional effort in maintaining, managing, monitoring.

About the stack

The backend is built on the Serverless framework and runs in AWS and makes use of services such as API Gateway, Lambda, DynamoDB, or SES. The tests are run using Axios for the API testing and Puppeteer for the browser tests. The test assertions are using Jest.

The front end is a Vue.js app.

DiffHub wasn't built to run on-premise, so I'll have to adapt a few things, migrate the functionality from the SaaS app into the open-source repo.

The serverless.yml contains a CloudFormation template that will deploy the entire service & resources without headaches. You can have it up and running in less than a minute, it also features a powerful API that can be easily integrated into CI/CD pipelines.

The UI won't be included in the initial open-source release, tests can be managed via API for now.

The code will be available on GitHub, I won't mind if you star the repo :)

If you want to stay in the loop with the progress please request private beta access on DiffHub homepage or DM me on Twitter.

Let's see where all this goes!

Most of the projects I have created lately are serverless, but I also use it to extend existing apps running in classic infrastructures and offload the resource-intensive tasks (i.e. image processing).

I use AWS as the serverless provider and Node.js to code the Lambda functions.

In the early days, I did everything manually from the AWS console, but it's cumbersome, not very scalable, and configuration drifts can happen so subtly that it’s very easy to lose track.

AWS best practices recommend deploying apps using Infrastructure-as-Code (IAC).

Using a framework like Serverless, in combination with version control, makes it really easy to track every change that's made to the app code & infrastructure.

The Serverless framework enables us to build AWS CloudFormation templates to describe the resources to be deployed into the cloud and uses the AWS CLI to deploy them. It makes the deployment as easy as typing:

sls deploy

Ok, I have the app & infra code already in a GitHub repository, why not automate the deployment process?

Serverless overview

The foundation of a serverless application consists of a serverless.yml file and an env.yml file.

The serverless.yml is the heart of a serverless application, it describes the entire infrastructure, all the way from the programming language to resource access.

service: my-service-name

custom:
    defaultStage: dev
    environment: ${file(env.yml):${self:provider.stage}, file(env.yml):dev}

provider:
    name: aws
    stage: ${opt:stage, self:custom.defaultStage}
    runtime: nodejs14.x

functions:
    # on the fly image resizer function
    resizer:
        handler: src/resizer.handler
        memorySize: 1024
        timeout: 15
        environment:
            AWS_KEY: ${self:custom.environment.AWS_KEY}
            AWS_SECRET: ${self:custom.environment.AWS_SECRET}
            BUCKET: ${self:custom.environment.S3_BUCKET}
            CDN_URL: ${self:custom.environment.CDN_URL}
        events:
            - http:
                  path: /resizer
                  method: GET

#optional plugins
plugins:
    - serverless-offline

The env.yml file stores the environment variables that are used in the application code (i.e. API keys)

add env.yml to .gitignore, don't commit it to the repo.

Some developers prefer to store the environment variables into JSON files instead of YML, the workflow is similar.

prod:
    name: "crafty"
    AWS_KEY: "MYAWSKEY" # add your AWS Key
    AWS_SECRET: "MYAWSSECRET" # add your AWS Secret
    S3_BUCKET: "crafty-products"
    CDN_URL: "https://mycdnurl.com"

dev:
    name: "DEV - crafty"
    AWS_KEY: "MYAWSKEY" # add your AWS Key
    AWS_SECRET: "MYAWSSECRET" # add your AWS Secret
    S3_BUCKET: "crafty-products-dev"
    CDN_URL: "https://mycdnurl.com"

Tip: I use the Serverless Offline plugin which emulates AWS Lambda and API Gateway on my local machine, allows me to test stuff locally, and speeds up the development cycles.

We'll assume we have a basic Serverless app ready to be deployed so let's jump right into how to automate this workflow.

Create the IAM user

There are two different IAM entities (users or roles) that you need to worry about when working with the Serverless framework:

1. The IAM user used by the Framework to deploy our Serverless service (the Framework user)

2. The IAM role used by a Lambda function when it's executed (a function role).

Below we'll create the IAM user that's needed by the framework to deploy our app & service, the framework user.

1. Log in to the AWS account and go to the Identity & Access Management (IAM)

2. Go to Users and click "Add users" > pick a username that'll be easier to remember, I prefer to prefix users for GitHub actions with github_sls > check "Programmatic access"

3. Click "Next" to go through to the Permissions page, click on "Attach existing policies" directly, and select AdministratorAccess

4. View and copy the API Key & Secret to a temporary place, we'll need them shortly.

To set more restrictive access and follow the principle of least privilege, I recommend checking this post of Michael Timbs and read more about the minimum IAM roles for deploying a serverless project.

GitHub Actions

First, we have to store the AWS Key and AWS Secret that have AdministratorAccess and will be used by Serverless for deployment.

Go into GitHub repository Settings > Secrets:

1. add the newly created AWS API Key & Secret as AWS_KEY_DEPLOY & AWS_SECRET_DEPLOY

2. add the content of the env.yml file in the ENV secret

Keep in mind that all the secrets stored in Github are encrypted, we won't able to view them after creation, just to update the value or remove them.

Next, we have to create a YAML file to define the action workflow configuration, which can be done in two ways:

1. from the GitHub repo, click Actions and create a workflow
2. define YAML files and store them in .github/workflows in your application root

This is a sample YAML used to describe the GitHub action workflow that deploys the app on every commit in the main branch.

name: Deploy Production
# on main branch updates
on:
    push:
        branches:
            - main
jobs:
    deploy:
        runs-on: ubuntu-latest
        steps:
            - uses: actions/checkout@v1
            - uses: actions/setup-node@v1
              with:
                  node-version: "12.x"
              #install serverless
            - name: Install Serverless Framework
              run: npm install -g serverless
              # authentication
            - name: Serverless AWS authentication
              run: sls config credentials --provider aws --key ${{ secrets.AWS_KEY_DEPLOY }} --secret ${{ secrets.AWS_SECRET_DEPLOY }}
              # create ENV, use the shell `cat` command to create env.yml from secrets
            - name: Create env file
              run: |
                  cat > env.yml << EOF
                  ${{ secrets.ENV }}
                  EOF
              #install node modules
            - name: Install NPM dependencies
              run: npm install
              #serverless deploy
            - name: Deploy
              run: sls deploy -s prod

And this is how the workflow log looks like.

Not that hard, isn't it?

This input is hidden from normal users, humans will leave it empty because they won't see it, so it's easy to tell when a form submission is spam or not.

The honeypot will help you catch the majority of the dumb spam bots, however, bots get smarter and harder to trick, so it's not bullet-proof.

<form method="POST" action="form-handler.php">
    <!--  ... form fields ... -->
    <!-- this is the honeypot field -->
    <div style="position: absolute; left: -9999px">
        <label for="website-url">Your website</label>
        <input type="text" id="website-url" name="url" tabindex="-1" autocomplete="nope" />
    </div>
    <!--  ... form fields ... -->
</form>

How to get it right?

Adding a honeypot requires getting your hands dirty with some code, update the form HTML and backend code. Below are my tips on how to make the honeypot more effective and avoid silly mistakes.

1. Make it juicy

Make the field look as legit as possible so even the smarter bots will be tricked into filling it.

✅ set the input type="text" or type="email"

✅ use a common name attribute like email, phone, address, i.e. name="url"

2. Hide it well

Bots get smarter every day, they can recognize honeypot patterns and we don't want that.

❌ don't use type="hidden"

❌ don't use name="honeypot" or another name that would give a hint it's a trap

❌ don't use "display: none" - most bots won't fill the CSS hidden inputs

❌ don't use "hidden" or similar in the class name if you're using that instead of inline styles

✅ Wrap the input & its label in a div tag and position it outside the viewport.

<!-- honeypot field -->
<div class="url-field-wrapper">
    <label for="website-url">Your website</label>
    <input type="text" id="website-url" name="url" tabindex="-1" autocomplete="nope" />
</div>
<!-- the rest of the form -->

.url-field-wrapper {
    position: absolute; 
    left: -9999px;
}

3. Don't shoot yourself in the foot

Browsers have functionalities that can be easily neglected and make legit users get caught by our spam bait input. Yep, false positives. There are 2 input attributes we have to set to avoid:

🚨 set tabindex="-1", tech-savvy visitors will use the tab key to navigate through form fields, if the tabindex attribute is not set they'll be able to focus and accidentally fill in the honeypot

🚨 set autocomplete="nope", avoid browsers' autocomplete feature to fill in the honeypot field, especially when using a common name attribute

4. Fake happiness

If the backend validation fails because the honeypot was filled, don't throw an error, silently discard the submissions and show a success message. Make the spammer think their junk went through.

// require the honeypot field to be present but empty
if(!isset($_REQUEST['url']) || !empty($_REQUEST['url'])) {
    // hello spammer!
    echo 'Thank you for submission!';
}

Additional techniques

As mentioned previously, the honeypot field won't stop all the SPAM submissions but will help to catch at least half of it. If you struggle with spam submissions you'll want to use the honeypot in conjunction with other methods.

The general go-to solution is to add a CAPTCHA, I recommend against it, until it really hurts.

It's effective, but why punish our users because we have a problem with SPAM? It's our problem, not theirs, let's try to address it.

I've had excellent results filtering the form submissions through Akismet, try that before CAPTCHA.

Shameless plug: I run a service that will help you deal with form processing & SPAM protection

👉 POST2