SQS Deep dive : cost, performance, management & security
Fajri Abdillah
fajri@[horangi.com, serverless.id, myedisi.com]
AWS Pop-Up Loft Jakarta 29 oktober 2019
Agenda
About SQS
Performance
Cost
Management & Security
About SQS
AWS SQS
Amazon Simple Queue Service (Amazon SQS) offers a secure, durable, and available hosted queue that lets you integrate and decouple distributed software systems and components.
SQS based Queue
Redis based Queue
SQS - Power Features
Serverless - No need to think about server, battle tested since 2006
Unlimited Throughput - Standard queues support a nearly unlimited number of transactions per second
Security - Control who can send messages to and receive messages from an Amazon SQS queue
Durability - Make use of dead letter queue, no data is missing, up to 14 days data retention.
Event Source - Lambda can process message in SQS
SQS - Limitations
Single Consumer - once a consumer processed a message then message is removed
Tricky pricing - Billed per transaction, chunked per 64 KB, but still cheap
Limited payload size - 256 KB, can use S3 for bigger payload, up to 2 GB, but more expensive
Delay queue - Maximum 15 minutes
FIFO Queue - 300 messages per second (without batch)
Performance
Mobile Analytics as a Service
7.5 million messages in SQS
11 million messages in SQS
How big is 10 million messages?
100 KB * 10 = 1,000 KB = 1 MB
100 KB * 100 = 10,000 KB = 10 MB
100 KB * 1,000 = 100,000 KB = 100 MB
100 KB * 10,000 = 1,000,000 KB = 1,000 MB = 1 GB
100 KB * 100,000 = 10,000,000 KB = 10,000 MB = 10 GB
100 KB * 1,000,000 = 100,000,000 KB
= 100,000 MB = 100 GB
100 KB * 10,000,000 = 1,000,000,000 KB
= 1,000,000 MB = 1,000 GB
= 1 TB
Messages in Flight
Processing ~2000 Messages / second
100 KB * 2,000 = 200,000 KB = 200 MB / second
Redis based Queue
How to scale to 10 million?
How many shard needed? replica?
How to autoscale?
How many nodes needed for 1TB of data? Type?
How much is it?
Redis based Queue
| Dev (cache.t2.micro) | Staging (cache.t2.small) | Prod (cache.r4.large) |
|---|---|---|
 |
 |
 |
SQS based Queue
| Dev | Staging | Prod |
|---|---|---|
|
|
|
Cost
SQS - Pricing (ap-southeast-1)
| Price per 1 Million Requests after Free Tier (Monthly) | |
|---|---|
| Standard Queue | $0.40 ($0.00000040 per request) |
| FIFO Queue | $0.50 ($0.00000050 per request) |
SQS - Pricing Estimation* (ap-southeast-1)
Daily Request = 40 million
Total Pricing = 40 * 31 * $0.40
Monthly Pricing = $496
Elasticache - Pricing (ap-southeast-1)
| Reserved instance - No Upfront 1 year (Monthly) | |
|---|---|
| cache.m5.large | $97.09 |
| cache.r5.large | $129.21 |
Management & Security
Avoid using IAM user with AdministratorAccess Policy
Use IAM Role when possible
Limit the policy (IAM or SQS )
Use STS service for temporary access
Management & Security
Use Server-Side Encryption when needed
Use VPC Endpoints when needed
SQS - Demo
Thank You
Join our Facebook Group : Serverless Indonesia
Our Medium Publications : Serverless Indonesia
Our u(n)pdated Blog : blog.serverless.id