Thanks for coming here to answer questions. Much appreciated!
Does each new Lambda cold start pull the entire image from the repo? Or if I derive from Lambda base images am I likely to get some of the layers already cached?
I'm trying to think about how the data transfer costs are going to each time the Lambda is instantiated. I didn't take this into account with Fargate and got burned when trying to trigger images on demand in a similar way.
This could replace my whole SQS / CloudWatch Events / Fargate setup using Lambda if I can figure out cold start costs ($ not time).
There's a few things at play. Functions will still stay warm inbetween invocations and will keep local any data already in the worker. We also maintain a couple different levels of cache so as to not hit ECR often.
I know we've got a few blog posts coming out over the next couple weeks on this new feature, and each tells a few bits and pieces about the story.
Depending on volume you'll probably find that Lambda will be cheaper for that workload, especially with the new 1ms billing.
FWIW, a little experimented I just ran showed me that with simple layers the cold start time of my little 3MB Go app was <100ms, using the Docker image `amazon/aws-lambda-go:1` instead took ~1500ms.
- - - -
REPORT RequestId: f905d5fe-a64e-48c8-b1f2-6535640a6f82 Duration: 7.55 ms Billed Duration: 1309 ms Memory Size: 256 MB Max Memory Used: 49 MB Init Duration: 1301.10 ms
- - - -
REPORT RequestId: 89afb20d-bc49-4d89-91f0-f1ef62ac99aa Duration: 12.20 ms Billed Duration: 13 ms Memory Size: 256 MB Max Memory Used: 35 MB Init Duration: 85.37 ms
I agree in general with the responses encouraging better usage of managed platforms. I've run a SaaS app for a couple of years using a combination of AWS Elasticbeanstalk (Flask and Django) and AWS Lambda. Server resource related downtime has been minimal and recovery is quick/automated. Even hosting on Lambda you can run into issues without layers of redundancy (Lambda may be fine but a Route 53 outage would prevent you from hitting that endpoint if you're using that for DNS).
Before thinking about handing over management of the deployment, I would encourage you to think about what the root cause of the outage is and whether something in the app will create that situation again. I invested in setting up DataDog monitoring for all hosts with alerts on key resource metrics that were causing issues (CPU was biggest issue for me).
The other thing that's worked well for me is just keeping things simple. As a solo founder, time spent with customers is more valuable than time spent on infrastructure (assuming all is running well). It's a little dated, but I still think this is a good path to follow as you're building your customer base. A simple stack will let you spend more time learning how your product can help your customers best.
We are considering Datadog, and nothing else seems to compare to them, but they seem extremely expensive. As a small startup/solo founder, did your implementation justify costs?
Datadog is the best all around tool with APM and log monitoring built in. I highly recommend it as it will be substantially cheaper than paying for separate monitoring, apm, logs, etc.
The support is also a lot more receptive towards small business owners unlike other platforms such as New Relic.
Also if you send custom stats to datadog such as login activity, you can use their Anomaly detection to find suspicious behavior.
Disclaimer: we use datadog at my company and have tried all the other popular options. Hands down datadog is the most feature rich and user friendly.
Yes, it's been an incredibly valuable tool for me. I get the most out of the host monitoring and the logging / alerting. The APM is nice, but I'm not using it nearly as much as the other pieces.
Just getting logging centralized alone has saved me tons of time, which is in turn more time spent on the product. I've been able to use the log parsing to setup metrics that tell me when an outside integration is acting up and isolate which paths. Take a day to really learn how their logs work and you'll be able to generate metrics / advanced event alerting in no time.
I was hesitant to pay the premium, but the peace of mind has been worth it. You can piece the same thing together with open source tooling. But then you've got another thing to manage.
Datadog has ~9000 customers. I imagine the respective open-source tools have many more customers than that, which says something about the perceived value of the two approaches. Also, Datadog wraps many tools into one whereas the open-source solutions have singular purposes. It's an apples to oranges comparison.
$15 per server per month isn't that extreme. I've used Datadog in production for 3+ years (starting from a headcount of 10) and can honestly say they've been worth every penny. Observability is irreplacable.
IBM had the contract to run most of the IT infrastructure for the State of Texas and subsequently lost it. I don't have any facts as to why, but rumor was it was an execution problem.
Web Application Engineer | TicketCity | Austin, TX
We're looking for someone with strong experience building web apps, mostly back-end focused. Someone that's interested in the API to the data store and everything that happens between. The existing stack is .NET but we're writing new stuff in a mix of technologies (Node, RabbitMQ, open to others). You'll be helping build out a microservice architecture and supporting a new set of client applications.
Does each new Lambda cold start pull the entire image from the repo? Or if I derive from Lambda base images am I likely to get some of the layers already cached?
I'm trying to think about how the data transfer costs are going to each time the Lambda is instantiated. I didn't take this into account with Fargate and got burned when trying to trigger images on demand in a similar way.
This could replace my whole SQS / CloudWatch Events / Fargate setup using Lambda if I can figure out cold start costs ($ not time).