Serverless Architecture
Serverless promises no infrastructure management and automatic scaling. The reality has sharp edges (cold starts, statelessness, vendor lock-in) that shape when it fits.
The Concept Explained
Serverless, in the sense of functions as a service, means you deploy a function rather than a server. The platform runs it in response to an event, scales the number of concurrent executions automatically, and charges only for the time it actually runs.
There are still servers. What changes is that you never think about them: no capacity planning, no instance count, no patching, no scaling policy. The unit of deployment is a function and the unit of billing is an invocation.
The execution model is what generates every consequence. The platform maintains an execution environment per concurrent invocation. A request arriving with no environment available triggers the creation of one, which is the cold start. Once created, an environment is typically reused for subsequent invocations, which is a warm start. Idle environments are eventually reclaimed.
Concurrency is the unit of scale, not requests per second. The platform runs one invocation per environment at a time, so a hundred simultaneous requests means a hundred environments. This is why serverless absorbs sudden bursts effortlessly and why sustained high concurrency gets expensive: you are billed for concurrent execution time, and there is no equivalent of an idle server absorbing more work.
How It Works
The Invocation Lifecycle and Where Latency Comes From
Click each step to explore
The Cold Start Problem
Cold start latency varies by an order of magnitude depending on runtime, package size, and whether the function attaches to private networking. Lighter interpreted runtimes with small packages sit at the low end, heavier runtimes with large dependency trees at the high end.
What matters for design is when it hurts, and the answer is uneven.
For a user-facing synchronous API, a cold start is added directly to a user's request. The pain shows up in tail latency rather than in the average: if 2% of requests hit a cold environment, your p99 is a cold start regardless of how fast the warm path is. This is the single most common reason serverless disappoints on interactive workloads.
For asynchronous or batch work, a cold start is almost irrelevant. A queue consumer taking an extra 300 milliseconds occasionally changes nothing.
Traffic pattern matters as much as workload type. Steady traffic keeps environments warm, so cold starts are rare. Sporadic traffic is the worst case: enough requests to matter, spaced far enough apart that environments are reclaimed between them, so a high proportion of requests pay the penalty. A function invoked a few times an hour may cold start nearly every time.
Mitigations exist: keep packages small, move initialization outside the handler, choose a lighter runtime, and use the platform's provisioned concurrency where available. Note that provisioned concurrency means paying for idle capacity, which gives up part of the original premise.
Forced statelessness is a stricter constraint than it first appears. Environments are created and destroyed unpredictably, so nothing in memory survives reliably. In-memory caches, accumulated counters, and background threads all break. Most consequentially, database connection pooling does not work the way it does on a server: each environment holds its own connections, so a thousand concurrent invocations can open a thousand connections and exhaust a database that would comfortably serve a handful of pooled application servers. This is a real production failure, and it is why connection proxies exist.
The Cost Model Inverts
Servers charge for time you hold them, whether busy or not. Serverless charges per invocation and per unit of execution time, so idle costs nothing.
That inversion is decisive at both ends of the traffic range. Something invoked a few thousand times a day costs almost nothing, where the equivalent always-on instance costs the same whether used or not. Something running at sustained high volume can cost considerably more than the instances that would serve the same load, because you are paying for every millisecond of execution with no economy from packing many requests onto one machine.
The crossover depends on your numbers, and the shape is what to remember: serverless wins on low, spiky, or unpredictable traffic and loses on sustained high traffic. A workload running continuously at high concurrency is the case where a reserved instance is dramatically cheaper.
System Design Implications
Good fits. Event-driven processing where an object landing in storage or a message arriving triggers work. Scheduled jobs that run briefly and infrequently, where paying for an idle instance the rest of the time is pure waste. Sporadic APIs such as internal tools and webhook receivers. Highly bursty workloads where provisioning for the peak would mean idle capacity most of the time. Glue code between managed services.
Poor fits. Latency-sensitive user-facing APIs, where cold starts land in your tail latency. Long-running computation, since platforms impose an execution time limit. Anything needing in-process state, persistent connections, or a warm cache. Sustained high-throughput services, on cost. Workloads with heavy database connection requirements, unless a connection proxy sits in between.
On lock-in, be precise rather than alarmed. The function code itself is usually ordinary and portable. What locks you in is the surrounding platform: the event source integrations, the identity and permission model, the managed services the function calls, and the infrastructure definitions. Migrating the handler is easy; migrating everything around it is the project. Keeping business logic in plain functions and confining platform specifics to a thin adapter layer is the practical mitigation, and it is exactly the hexagonal architecture idea from later in this module.
Design for the concurrency model rather than fighting it. Move initialization outside the handler so it amortizes across invocations. Keep packages small. Assume no memory persists. Use a connection proxy for relational databases. And set concurrency limits deliberately, because automatic scaling will happily scale into a downstream dependency until it falls over.
Answer the cold start question with where it lands rather than with a number. It affects tail latency specifically, so if a small percentage of requests hit a cold environment then your p99 is a cold start no matter how fast the warm path is. Then add the pattern that makes it worst: sporadic traffic, frequent enough to matter but spaced far enough apart that environments are reclaimed between requests.
Tradeoffs and Decision Framework
| Dimension | Serverless | Always-on servers |
|---|---|---|
| Idle cost | None | Full |
| Cost at sustained high load | High | Lower |
| Scaling | Automatic, near-instant | Configured, slower |
| Cold start | Yes, hits tail latency | None after startup |
| In-memory state | Unreliable, do not rely on it | Available |
| Connection pooling | Per environment, can exhaust a database | Normal pooling |
| Execution time limit | Yes | None |
| Operational burden | Minimal | Patching, scaling, capacity |
| Portability | Logic yes, integrations no | Higher |
The framework in four questions. Is traffic spiky, sporadic, or low, which is where the cost model wins decisively? Is the work triggered by events rather than serving synchronous user requests? Can the workload tolerate occasional cold start latency in its tail? And does it fit within the execution time limit without needing in-memory state between invocations?
Four yeses and serverless is an excellent choice that removes real operational burden. A no on latency sensitivity or on sustained high throughput is usually decisive against it. Many systems sensibly use both: containers for the steady user-facing path and functions for the event-driven and scheduled edges.
Common Mistakes
Using it for latency-sensitive APIs without accounting for cold starts. The average looks fine and the p99 is a cold start.
Assuming in-memory state persists. Environments are created and destroyed unpredictably. Caches and counters in memory are lost silently.
Opening a database connection per invocation. High concurrency exhausts the connection limit, and this takes down the database rather than the function.
Ignoring the cost curve at high sustained load. The model that is nearly free at low volume can be markedly more expensive than instances at high volume.
Large deployment packages. Package size is paid on every cold start, and it is the part of cold start latency you most directly control.
Initialization inside the handler. Connection setup and configuration loading belong outside it so they run once per environment rather than once per request.
Unbounded automatic scaling. Functions will scale into a downstream dependency until it fails. Concurrency limits are a protection mechanism, not a cost control.
Treating lock-in as all or nothing. The handler is portable, the surrounding integrations are not. Isolating platform specifics behind an adapter is the practical answer.
For which workloads is serverless a great fit, and for which is it a poor one? Explain the cold start problem and its impact.