How to run Typesense search in AWS

Searching datasets and retrieving relevant results blazing-fast is a challenge, no matter if the underlying database of our application is RDBMS or NoSQL.

If you're serious about search and need typo tolerance, result rankings, synonyms, filtering, faceting, geo search, and whatnot, you better forget about those sub-optimal DB queries and bring a dedicated search service into the mix.

There are multiple cloud offerings for hosted search such as Elasticsearch, Algolia, Meilisearch, or Typesense cloud. But due to infrastructure architecture requirements, regulatory reasons, cost optimization, or just cost consolidation it often makes more sense to run our search service in AWS.

Note: AWS has its own search service, OpenSearch which is a fork of ElasticSearch. At the time of writing AWS just announced OpenSearch Serverless, which is in beta/preview release.

Typesense is my favorite, it's simple, fast, typo-tolerant, and open-source. I've been using it on multiple production projects handling searches on datasets of millions of entries.

For production use, it is recommended to run a multi-node Typesense cluster, however, there are plenty of scenarios when a single-node service will do the job.

In this article, I will explain how to get Typesense running as a single node using infrastructure as code and make it as resilient and robust as it can be in a single Availability Zone and how to size your instance depending on your data set.

The CloudFormation template will create the following AWS resources:

• EC2 Auto Scaling Group & Launch Configuration

• Instance Profile, IAM Roles & Security Group

• EBS Volume

• Elastic IP

• Lambda Function & CloudFormation Custom Resource

A self-healing search service

We're going to launch the instance using EC2 Auto Scaling, there are no additional fees for it, and the service can automatically determine the health status of an instance using EC2 status checks.

If the search service instance is unhealthy, it will be terminated and replaced by a new one with minimum downtime.

When an instance is terminated and a new one gets created, the data stored in the EC2 instance store is lost and the new instance will have a different IP.

That's why it is critical to have persistent block storage and an elastic IP.

EC2 Auto Scaling will make sure to have 1 healthy instance running at all times.

# ASG
AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
        LaunchConfigurationName: !Ref LaunchConfig
        DesiredCapacity: 1
        MinSize: 0
        MaxSize: 1
        AvailabilityZones:
            - !Select [0, !GetAZs ""]
        Tags:
            - Key: Name
              PropagateAtLaunch: true
              Value: !Sub ${AWS::StackName}-${EnvironmentType}

EC2 Instance Type

CPU capacity is important to handle concurrent search traffic and indexing operations, so Typesense requires at least 2 vCPUs of compute capacity to operate.

All the next-gen EC2 T4g instances have at least 2 vCPUs, they're powered by Arm-based AWS Graviton2 processors and are ideal for running applications with moderate CPU usage that experience temporary spikes in usage.

The amount of RAM required is completely dependent on the size of the data you index, but for Typesense to hold the whole index in memory, the instance memory should be 2-3X the size of the data set.

As the Typesense process, itself is quite lightweight (20MB RAM with empty dataset) a 2GB memory instance (t4g.small) will likely handle a data set of up to 1GB.

Data Storage

Amazon EC2 instance store is temporary, if the instance stops, hibernates, or terminates the data from the instance store is lost, so we need to use a durable data storage such as EBS.

Typesense stores a copy of the raw data on disk and then builds the in-memory index with the data. Then at search time, after determining the final set of documents to return in the API response, it fetches these documents (only) from the disk and puts them in the API response.

We'll need a fast and durable storage, hence we'll go for EBS gp3 volume type (next-gen General Purpose SSD).

The size of the EBS volume should be at least the size of the raw dataset.

Launch Configuration

The Launch Configuration will be used by the Auto Scaling group to configure our EC2 instance.

LaunchConfig:
    Type: AWS::AutoScaling::LaunchConfiguration
    Properties:
        InstanceType: !Ref InstanceType
        ImageId: !Ref LatestLinuxAmiId
        IamInstanceProfile: !GetAtt EC2InstanceProfile.Arn
        SecurityGroups:
            - !Ref EC2SecurityGroup
        UserData:
            Fn::Base64: !Sub |
                # [...] removed for brevity plese check below

Launch Configuration UserData

This is where most of the auto-healing magic happens, UserData allows us to run commands on our instance at launch. When an unhealthy instance is terminated and a new one is created it will be empty.

UserData will run the commands to ensure the search service is reconfigured back to its running state.

1. Associate the IP & attach the EBS volume

#!/usr/bin/env bash
# get instance-id
EC2_INSTANCE_ID=$(curl -s http://instance-data/latest/meta-data/instance-id)

# associate the Elastic IP address to the instance
aws ec2 associate-address --instance-id $EC2_INSTANCE_ID --allocation-id ${ElasticIP.AllocationId} --allow-reassociation  --region ${AWS::Region}

# attach volume
aws ec2 attach-volume --device ${VolumeDevice} --instance-id $EC2_INSTANCE_ID --volume-id ${EBSVolume} --region ${AWS::Region}

# wait for volume to be properly attached
DATA_STATE="unknown"
until [ "${!DATA_STATE}" == "attached" ]; do
    DATA_STATE=$(aws ec2 describe-volumes \
        --region ${AWS::Region} \
        --filters \
            Name=attachment.instance-id,Values=${!EC2_INSTANCE_ID} \
            Name=attachment.device,Values=${VolumeDevice} \
        --query Volumes[].Attachments[].State \
        --output text)

    sleep 5
done

# format the volume if doesn't have file system (on stack creation)
blkid --match-token TYPE=ext4 ${VolumeDevice} || mkfs.ext4 -m0 ${VolumeDevice}

# create mount root folder
mkdir -p ${VolumeMountPath}

# mount the volume to folder
mount ${VolumeDevice} ${VolumeMountPath}

# persist the volume on restart
echo "${VolumeDevice} ${VolumeMountPath} ext4 defaults,nofail 0 2" >> /etc/fstab

2. Configure the Typesense Server

If you're using the CloudFormation template defaults the VolumeMountPath is set to /mnt/ebs hence:

• The config file will be at /mnt/ebs/typesense.ini

• Logs will be under /mnt/ebs/log/

• Data will be stored under /mnt/ebs/data/

# create typesense server config
if [ ! -f ${VolumeMountPath}/typesense.ini ]
then
    echo "[server]
api-key = ${TypesenseApiKey.Value}
data-dir = ${VolumeMountPath}/data
log-dir = ${VolumeMountPath}/log
api-port = ${TypesensePort}" > ${VolumeMountPath}/typesense.ini
    chown ec2-user:ec2-user ${VolumeMountPath}/typesense.ini
fi

# create typesense data folder if not exists
if [ ! -d ${VolumeMountPath}/data ]
then
    mkdir -p ${VolumeMountPath}/data
    chown ec2-user:ec2-user ${VolumeMountPath}/data
fi

# create typesense logs folder if not exists
if [ ! -d ${VolumeMountPath}/log ]
then
    mkdir -p ${VolumeMountPath}/log
    chown ec2-user:ec2-user ${VolumeMountPath}/log
fi

3. Download & Install Typesense Server

As we're running the service on a T4g instance powered by Graviton2 processors we have to download the arm64 Binary.

# download & unarchive typesense
curl -O https://dl.typesense.org/releases/${TypesenseVersion}/typesense-server-${TypesenseVersion}-linux-arm64.tar.gz
tar -xzf typesense-server-${TypesenseVersion}-linux-arm64.tar.gz -C /home/ec2-user
# remove archive
rm typesense-server-${TypesenseVersion}-linux-arm64.tar.gz

4. Create systemd service & enable the daemon

As we installed it from a binary we have to create a systemd service for the Typesense server. This will make the Typesense server service always available.

# create typesense service
echo "[Unit]
Description=Typesense service
After=network.target

[Service]
Type=simple
Restart=always
RestartSec=5
User=ec2-user
ExecStart=/home/ec2-user/typesense-server --config=${VolumeMountPath}/typesense.ini

[Install]
WantedBy=default.target" > /etc/systemd/system/typesense.service

# start typesense service
systemctl start typesense
# enable typesense daemon
systemctl enable typesense

EBS & Elastic IP

Important: the instance needs to be launched in the same Availability Zone as the EBS volume.

Tip: to save costs on dev instances, you can create EC2 Auto Scaling scheduled actions that set autoscaling to 0 at night and back to 1 during work hours.

EBSVolume:
    Type: AWS::EC2::Volume
    Properties:
        Size: !Ref VolumeSize
        VolumeType: gp3
        AvailabilityZone: !Select [0, !GetAZs ""]
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName}-${EnvironmentType}
    DeletionPolicy: Snapshot

ElasticIP:
    Type: AWS::EC2::EIP
    Properties:
        Tags:
            - Key: Name
              Value: !Sub ${AWS::StackName}-${EnvironmentType}

EC2 IAM Role & Security Groups

The template will create an Instance Profile, an IAM Role, and a Security Group.

The IAM Role will be assumed by the EC2 instance so it can attach the EBS volume and self-associate the Elastic IP. These commands are run using AWS CLI in the Launch Configuration UserData .

The Instance Profile allows passing the IAM role to the EC2 instance.

The Security Group permits the instance to receive traffic on the port used by the Typesense server (8108) and on port 22 for SSH access.

Note: a LambdaExecutionRole is also created to allow the API Key generation function to write logs to AWS CloudWatch, but it's omitted for brevity.

EC2InstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
        InstanceProfileName: !Sub ${AWS::StackName}-${EnvironmentType}-instance-profile
        Path: /
        Roles:
            - !Ref EC2InstanceRole

EC2InstanceRole:
    Type: AWS::IAM::Role
    Properties:
        RoleName: !Sub ${AWS::StackName}-${EnvironmentType}-instance-role
        AssumeRolePolicyDocument:
            Version: "2012-10-17"
            Statement:
                - Effect: Allow
                  Principal:
                      Service:
                          - ec2.amazonaws.com
                  Action:
                      - sts:AssumeRole
        Path: /
        Policies:
            - PolicyName: !Sub ${AWS::StackName}-${EnvironmentType}-instance-policy
              PolicyDocument:
                  Version: "2012-10-17"
                  Statement:
                      # allow attaching EBS volumes
                      - Effect: Allow
                        Action:
                            - ec2:AttachVolume
                            - ec2:DescribeVolumes
                        Resource: "*"
                      # allow Elastic IP association
                      - Effect: Allow
                        Action:
                            - ec2:AssociateAddress
                        Resource: "*"
# SG
EC2SecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
        GroupDescription: allow connections on HTTP & SSH
        SecurityGroupIngress:
            # Allow HTTP traffic on Typesense port
            - IpProtocol: tcp
              FromPort: !Ref TypesensePort
              ToPort: !Ref TypesensePort
              CidrIp: 0.0.0.0/0
            # Allow SSH - !! narrow CidrIp down, too broad
            - IpProtocol: tcp
              FromPort: 22
              ToPort: 22
              CidrIp: 0.0.0.0/0

Typesense API Key Generation

The CloudFormation stack will generate a Typesense admin API key using a CustomResource that references a RandomStringGenerator Lambda function.

The TypesenseApiKeyis exported in the Output so it can be referenced in other stacks. Optionally it can be stored in a AWS::SSM::Parameter to be used by other AWS services.

The admin API key provides full control over the Typesense API, make sure you create Scoped API Keys to be used in your application.

TypesenseApiKey:
    Type: AWS::CloudFormation::CustomResource
    Properties:
        Length: 32
        ServiceToken: !GetAtt RandomStringGenerator.Arn

The random string generator Lambda function is deployed using inline code, this works for simple functions when the code length is up to 4096 chars.

RandomStringGenerator:
    Type: AWS::Lambda::Function
    Properties:
        FunctionName: !Sub ${AWS::StackName}-${EnvironmentType}-random-string
        Code:
            ZipFile: >
                const response = require("cfn-response"), crypto = require("crypto");
                exports.handler = (event, context) =>{
                    response.send(event, context, response.SUCCESS, {Value:crypto.randomBytes(parseInt(event['ResourceProperties']['Length'])).toString('hex')});
                };
        Handler: index.handler
        Role: !GetAtt LambdaExecutionRole.Arn
        Runtime: nodejs16.x
        MemorySize: 128
        Timeout: 10

One-click Deployment to get going quickly, or click to view the complete self-contained Cloudformation template that can be deployed into any AWS account.

For complete details about Typesense API check API Reference

Summary

Although this CloudFormation stack template is made especially to get a Typesense search service up and running in minutes, it can be repurposed easily, e.g. for installing WordPress instead of Typesense.

For educational purposes, it features some common real-life scenarios and pattern examples such as:

• Create a single self-healing instance with EC2 Auto Scaling

• Use Launch Configuration and UserData to run commands on the instance at launch

• Persistent auto-mounting EBS storage + creating a filesystem on new volumes

• Auto-install a service from a Linux Binary

• Creating a Linux systemd service & daemon

• Use of Metadata to group and sort CloudFormation parameters

• How to deploy a Lambda function with inline code

• Use of Custom Resource to trigger a Lambda and reference the output

• Use of CloudFormation Intrinsic Functions (!Ref, !GetAtt, !Sub, !Select, !GetAZs)

Cover photo by Marten Newhall on Unsplash