Search for a command to run...
Get Typesense search running within minutes in AWS using infrastructure as code.
An architect by trade, practicing architecture in the Cloud. 7x AWS Certified, AWS Community Builder, Serverless advocate, Vue enthusiast.
Hi Stefan,
First, thank you for the template. I am a complete newbie to the topic. I created an AWS server instance via the template and upon connecting to it, i have trouble to even find the typsense application itself under the ./typsense-server, the folder doesnt exist. Any ideas why?
Help would be greatly appreciated.
Regards
Yes, using Launch Template is recommended now by AWS, however, at the time of writing, I chose to use LC.
I appreciate your post. Thank you.
I have one concern:
It seems like half of the UserData bash script should be part of the CFN code instead. Specifically these steps should all be in CFN so that CloudFormation can track the resources, no?:
# get instance-id
# associate the Elastic IP address to the instance
# attach volume
# wait for volume to be properly attached
# format the volume if doesn't have file system
# create mount root folder
# mount the volume to folder
That's in UserData to make the service self-healing.
In case the instance becomes unhealthy and Autoscaling terminates it and starts a new one, the new instance will be attached the Elastic IP and EBS volume automatically.
Stefan Olaru Oh, I didn't think of that. It wasn't clear to me from the AWS autoscaling docs what happens with the attached EBS (when set to not delete) upon autoscaling. That's very tricky. Thank you for the quick response.
Stefan Olaru May I also ask why you generate the admin api key using a lambda function instead of using secrets manager?
That’s what I had at hand at time of writing, but as you say, doing it with GenerateSecretString can also work and it’s a more elegant solution ;) I would appreciate if you can share how you end up solving it.
Stefan Olaru Certainly!
Stefan Olaru Took a me a while (converted to cdk and changed/added some things), but here is what I did to generate the bootstrap api key and startup the typesense instance:
Have AWS Secretsmanager generate the bootstrap api key without it ever touching my laptop:
typesense_bootstrap_api_key = secretsmanager.Secret(self, "TypesenseBootstrapAPIKeySecret",
secret_name="TypesenseBootstrapAPIKey",
generate_secret_string=secretsmanager.SecretStringGenerator(
exclude_uppercase=True,
exclude_punctuation=True,
password_length=64,
),
)
Get the Bootstrap API Key from SecretsManager upon execution of the UserData:
sudo yum install jq
# get the api key for bootstrapping typesense
secretsmanager_resp=$(aws secretsmanager get-secret-value \
--secret-id {typesense_bootstrap_api_key.secret_name})
ApiKey=$(echo $secretsmanager_resp | jq -r .SecretString)
The Typesense EC2 Instance IAM Role needs additional permissions to get this secret from SecretsManager. I created a SecretsManagerReadOnlyPolicy and attached it to the role:
secrets_manager_read_only_policy = iam.ManagedPolicy(self, "SecretsManagerReadOnlyPolicy",
managed_policy_name="SecretsManagerReadOnly",
statements=[iam.PolicyStatement(
effect=iam.Effect.ALLOW,
actions=[
"secretsmanager:GetResourcePolicy",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
],
resources=["*"]
)]
)
I do end up using a lambda function to generate api keys. It gets the bootstrap or admin key (depending on what keyname you pass in the payload) from secretsmanager, reaches out to typesense to generate a new key with whatever name and permissions were passed in the payload, and then stores that new key in secretsmanager. Hopefully this can be run to generate the basic keys I'd want (admin, readwrite, readonly, idk what else) upon CDK deploy, but I don't have a great way to ensure it runs after the typesense server actually starts up for the first time without doing something very hacky.
By the way, my UserData script needed this addition to get the EC2_INSTANCE_ID:
# get instance-id
TOKEN=$(curl -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 600" "http://instance-data/latest/api/token")
EC2_INSTANCE_ID=$(curl -H "X-aws-ec2-metadata-token: $TOKEN" -s http://instance-data/latest/meta-data/instance-id)
This is all built into a greater cdk project I have going, but I will pull this out and put it in a public repo when I have time.
Calvin Butler Awesome, thanks for the detailed reply! 👏
I used cloudformation, but it look like the IP that is output does not support SSL. Can you provide some insight on how I could support SSL?
Hi Nick, SSL on IP is tricky but you can point a sub/domain to it and install Certbot to issue a free Let's Encrypt certificate.
Stefan Olaru Thanks for the reply. I've tried that but then I run into issues with the certificate being only readable by root, but the typesense service being run by ec2-user. I get this error on startup:
"An error occurred while trying to load server certificate file: /etc/letsencrypt/live/example.com/fullchain.pem"
I've read that putting a load balancer in front of everything will allow us to install an aws issued certificate. Would you have an example of how to do that?
Hi Nick, you could add an ALB in front of your instance, though for a single instance and just for SSL certificate it might be overkill.
Have a look at my other article about running a multi-node Typesense, it uses a Load Balancer https://blog.stefanolaru.com/how-to-run-highly-available-typesense-cluster-in-aws
Another (free) option would be to install Nginx which will handle the SSL termination and proxy to Typesense. But it's very likely possible to manage this directly from Typesense ...
A clean serverless pattern for streaming real-time AI workflow responses with Server-Sent Events and AWS Lambda
Emails not reaching the inbox due to template Rendering Failure
A secure, efficient, and hassle-free way to run data analytics queries
How to run your AWS Lambda functions with a fixed IP up to 10x cheaper.
Using the API Gateway - SQS integration to create scalable webhook listeners and validate the requests at the API Gateway level
Searching datasets and retrieving relevant results blazing-fast is a challenge, no matter if the underlying database of our application is RDBMS or NoSQL.
If you're serious about search and need typo tolerance, result rankings, synonyms, filtering, faceting, geo search, and whatnot, you better forget about those sub-optimal DB queries and bring a dedicated search service into the mix.
There are multiple cloud offerings for hosted search such as Elasticsearch, Algolia, Meilisearch, or Typesense cloud. But due to infrastructure architecture requirements, regulatory reasons, cost optimization, or just cost consolidation it often makes more sense to run our search service in AWS.
Note: AWS has its own search service, OpenSearch which is a fork of ElasticSearch. At the time of writing AWS just announced OpenSearch Serverless, which is in beta/preview release.
Typesense is my favorite, it's simple, fast, typo-tolerant, and open-source. I've been using it on multiple production projects handling searches on datasets of millions of entries.
For production use, it is recommended to run a multi-node Typesense cluster, however, there are plenty of scenarios when a single-node service will do the job.
In this article, I will explain how to get Typesense running as a single node using infrastructure as code and make it as resilient and robust as it can be in a single Availability Zone and how to size your instance depending on your data set.
The CloudFormation template will create the following AWS resources:
EC2 Auto Scaling Group & Launch Configuration
Instance Profile, IAM Roles & Security Group
EBS Volume
Elastic IP
Lambda Function & CloudFormation Custom Resource
We're going to launch the instance using EC2 Auto Scaling, there are no additional fees for it, and the service can automatically determine the health status of an instance using EC2 status checks.
If the search service instance is unhealthy, it will be terminated and replaced by a new one with minimum downtime.
When an instance is terminated and a new one gets created, the data stored in the EC2 instance store is lost and the new instance will have a different IP.
That's why it is critical to have persistent block storage and an elastic IP.
EC2 Auto Scaling will make sure to have 1 healthy instance running at all times.
# ASG
AutoScalingGroup:
Type: AWS::AutoScaling::AutoScalingGroup
Properties:
LaunchConfigurationName: !Ref LaunchConfig
DesiredCapacity: 1
MinSize: 0
MaxSize: 1
AvailabilityZones:
- !Select [0, !GetAZs ""]
Tags:
- Key: Name
PropagateAtLaunch: true
Value: !Sub ${AWS::StackName}-${EnvironmentType}
CPU capacity is important to handle concurrent search traffic and indexing operations, so Typesense requires at least 2 vCPUs of compute capacity to operate.
All the next-gen EC2 T4g instances have at least 2 vCPUs, they're powered by Arm-based AWS Graviton2 processors and are ideal for running applications with moderate CPU usage that experience temporary spikes in usage.
The amount of RAM required is completely dependent on the size of the data you index, but for Typesense to hold the whole index in memory, the instance memory should be 2-3X the size of the data set.
As the Typesense process, itself is quite lightweight (20MB RAM with empty dataset) a 2GB memory instance (t4g.small) will likely handle a data set of up to 1GB.
Amazon EC2 instance store is temporary, if the instance stops, hibernates, or terminates the data from the instance store is lost, so we need to use a durable data storage such as EBS.
Typesense stores a copy of the raw data on disk and then builds the in-memory index with the data. Then at search time, after determining the final set of documents to return in the API response, it fetches these documents (only) from the disk and puts them in the API response.
We'll need a fast and durable storage, hence we'll go for EBS gp3 volume type (next-gen General Purpose SSD).
The size of the EBS volume should be at least the size of the raw dataset.
The Launch Configuration will be used by the Auto Scaling group to configure our EC2 instance.
LaunchConfig:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
InstanceType: !Ref InstanceType
ImageId: !Ref LatestLinuxAmiId
IamInstanceProfile: !GetAtt EC2InstanceProfile.Arn
SecurityGroups:
- !Ref EC2SecurityGroup
UserData:
Fn::Base64: !Sub |
# [...] removed for brevity plese check below
This is where most of the auto-healing magic happens, UserData allows us to run commands on our instance at launch. When an unhealthy instance is terminated and a new one is created it will be empty.
UserData will run the commands to ensure the search service is reconfigured back to its running state.
#!/usr/bin/env bash
# get instance-id
EC2_INSTANCE_ID=$(curl -s http://instance-data/latest/meta-data/instance-id)
# associate the Elastic IP address to the instance
aws ec2 associate-address --instance-id $EC2_INSTANCE_ID --allocation-id ${ElasticIP.AllocationId} --allow-reassociation --region ${AWS::Region}
# attach volume
aws ec2 attach-volume --device ${VolumeDevice} --instance-id $EC2_INSTANCE_ID --volume-id ${EBSVolume} --region ${AWS::Region}
# wait for volume to be properly attached
DATA_STATE="unknown"
until [ "${!DATA_STATE}" == "attached" ]; do
DATA_STATE=$(aws ec2 describe-volumes \
--region ${AWS::Region} \
--filters \
Name=attachment.instance-id,Values=${!EC2_INSTANCE_ID} \
Name=attachment.device,Values=${VolumeDevice} \
--query Volumes[].Attachments[].State \
--output text)
sleep 5
done
# format the volume if doesn't have file system (on stack creation)
blkid --match-token TYPE=ext4 ${VolumeDevice} || mkfs.ext4 -m0 ${VolumeDevice}
# create mount root folder
mkdir -p ${VolumeMountPath}
# mount the volume to folder
mount ${VolumeDevice} ${VolumeMountPath}
# persist the volume on restart
echo "${VolumeDevice} ${VolumeMountPath} ext4 defaults,nofail 0 2" >> /etc/fstab
If you're using the CloudFormation template defaults the VolumeMountPath is set to /mnt/ebs hence:
The config file will be at /mnt/ebs/typesense.ini
Logs will be under /mnt/ebs/log/
Data will be stored under /mnt/ebs/data/
# create typesense server config
if [ ! -f ${VolumeMountPath}/typesense.ini ]
then
echo "[server]
api-key = ${TypesenseApiKey.Value}
data-dir = ${VolumeMountPath}/data
log-dir = ${VolumeMountPath}/log
api-port = ${TypesensePort}" > ${VolumeMountPath}/typesense.ini
chown ec2-user:ec2-user ${VolumeMountPath}/typesense.ini
fi
# create typesense data folder if not exists
if [ ! -d ${VolumeMountPath}/data ]
then
mkdir -p ${VolumeMountPath}/data
chown ec2-user:ec2-user ${VolumeMountPath}/data
fi
# create typesense logs folder if not exists
if [ ! -d ${VolumeMountPath}/log ]
then
mkdir -p ${VolumeMountPath}/log
chown ec2-user:ec2-user ${VolumeMountPath}/log
fi
As we're running the service on a T4g instance powered by Graviton2 processors we have to download the arm64 Binary.
# download & unarchive typesense
curl -O https://dl.typesense.org/releases/${TypesenseVersion}/typesense-server-${TypesenseVersion}-linux-arm64.tar.gz
tar -xzf typesense-server-${TypesenseVersion}-linux-arm64.tar.gz -C /home/ec2-user
# remove archive
rm typesense-server-${TypesenseVersion}-linux-arm64.tar.gz
As we installed it from a binary we have to create a systemd service for the Typesense server. This will make the Typesense server service always available.
# create typesense service
echo "[Unit]
Description=Typesense service
After=network.target
[Service]
Type=simple
Restart=always
RestartSec=5
User=ec2-user
ExecStart=/home/ec2-user/typesense-server --config=${VolumeMountPath}/typesense.ini
[Install]
WantedBy=default.target" > /etc/systemd/system/typesense.service
# start typesense service
systemctl start typesense
# enable typesense daemon
systemctl enable typesense
Important: the instance needs to be launched in the same Availability Zone as the EBS volume.
Tip: to save costs on dev instances, you can create EC2 Auto Scaling scheduled actions that set autoscaling to 0 at night and back to 1 during work hours.
EBSVolume:
Type: AWS::EC2::Volume
Properties:
Size: !Ref VolumeSize
VolumeType: gp3
AvailabilityZone: !Select [0, !GetAZs ""]
Tags:
- Key: Name
Value: !Sub ${AWS::StackName}-${EnvironmentType}
DeletionPolicy: Snapshot
ElasticIP:
Type: AWS::EC2::EIP
Properties:
Tags:
- Key: Name
Value: !Sub ${AWS::StackName}-${EnvironmentType}
The template will create an Instance Profile, an IAM Role, and a Security Group.
The IAM Role will be assumed by the EC2 instance so it can attach the EBS volume and self-associate the Elastic IP. These commands are run using AWS CLI in the Launch Configuration UserData .
The Instance Profile allows passing the IAM role to the EC2 instance.
The Security Group permits the instance to receive traffic on the port used by the Typesense server (8108) and on port 22 for SSH access.
Note: a LambdaExecutionRole is also created to allow the API Key generation function to write logs to AWS CloudWatch, but it's omitted for brevity.
EC2InstanceProfile:
Type: AWS::IAM::InstanceProfile
Properties:
InstanceProfileName: !Sub ${AWS::StackName}-${EnvironmentType}-instance-profile
Path: /
Roles:
- !Ref EC2InstanceRole
EC2InstanceRole:
Type: AWS::IAM::Role
Properties:
RoleName: !Sub ${AWS::StackName}-${EnvironmentType}-instance-role
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- sts:AssumeRole
Path: /
Policies:
- PolicyName: !Sub ${AWS::StackName}-${EnvironmentType}-instance-policy
PolicyDocument:
Version: "2012-10-17"
Statement:
# allow attaching EBS volumes
- Effect: Allow
Action:
- ec2:AttachVolume
- ec2:DescribeVolumes
Resource: "*"
# allow Elastic IP association
- Effect: Allow
Action:
- ec2:AssociateAddress
Resource: "*"
# SG
EC2SecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: allow connections on HTTP & SSH
SecurityGroupIngress:
# Allow HTTP traffic on Typesense port
- IpProtocol: tcp
FromPort: !Ref TypesensePort
ToPort: !Ref TypesensePort
CidrIp: 0.0.0.0/0
# Allow SSH - !! narrow CidrIp down, too broad
- IpProtocol: tcp
FromPort: 22
ToPort: 22
CidrIp: 0.0.0.0/0
The CloudFormation stack will generate a Typesense admin API key using a CustomResource that references a RandomStringGenerator Lambda function.
The TypesenseApiKeyis exported in the Output so it can be referenced in other stacks. Optionally it can be stored in a AWS::SSM::Parameter to be used by other AWS services.
The admin API key provides full control over the Typesense API, make sure you create Scoped API Keys to be used in your application.
TypesenseApiKey:
Type: AWS::CloudFormation::CustomResource
Properties:
Length: 32
ServiceToken: !GetAtt RandomStringGenerator.Arn
The random string generator Lambda function is deployed using inline code, this works for simple functions when the code length is up to 4096 chars.
RandomStringGenerator:
Type: AWS::Lambda::Function
Properties:
FunctionName: !Sub ${AWS::StackName}-${EnvironmentType}-random-string
Code:
ZipFile: >
const response = require("cfn-response"), crypto = require("crypto");
exports.handler = (event, context) =>{
response.send(event, context, response.SUCCESS, {Value:crypto.randomBytes(parseInt(event['ResourceProperties']['Length'])).toString('hex')});
};
Handler: index.handler
Role: !GetAtt LambdaExecutionRole.Arn
Runtime: nodejs16.x
MemorySize: 128
Timeout: 10
One-click Deployment to get going quickly, or click to view the complete self-contained Cloudformation template that can be deployed into any AWS account.
For complete details about Typesense API check API Reference
Although this CloudFormation stack template is made especially to get a Typesense search service up and running in minutes, it can be repurposed easily, e.g. for installing WordPress instead of Typesense.
For educational purposes, it features some common real-life scenarios and pattern examples such as:
Create a single self-healing instance with EC2 Auto Scaling
Use Launch Configuration and UserData to run commands on the instance at launch
Persistent auto-mounting EBS storage + creating a filesystem on new volumes
Auto-install a service from a Linux Binary
Creating a Linux systemd service & daemon
Use of Metadata to group and sort CloudFormation parameters
How to deploy a Lambda function with inline code
Use of Custom Resource to trigger a Lambda and reference the output
Use of CloudFormation Intrinsic Functions (!Ref, !GetAtt, !Sub, !Select, !GetAZs)
Cover photo by Marten Newhall on Unsplash