Serverless Systems

The decomposition is the tool. When a requirement arrives — "we need a portal where teams submit evidence and compliance reviews it" — resist the urge to reach for services, and first ask which of the five jobs the requirement is made of. Submitting is accepting requests. Checking the submission is running logic. The evidence itself is stored state. Telling the reviewer is moving messages. Proving it happened is keeping records. Only then do you cast services into the roles.

The bargain has a precise shape. AWS takes: hardware, operating systems, runtime patching, capacity planning, most of scaling, and the physical security of all of it. You keep: identity (who may do what), data (what you store, for how long), wiring (which piece talks to which), and correctness (idempotency, ordering, failure handling). Notice that everything you keep is design. Nothing you keep is toil. That is the trade, and it is why the design conversation gets more important as the operations conversation gets shorter.

Traditional capacity is a fixed cost you overprovision: you pay for the peak whether it comes or not, and success looks like servers running warm. Serverless inverts this completely. Lambda bills per invocation and per millisecond of execution; DynamoDB per request; S3 per gigabyte-month; Athena per byte scanned. An idle system costs approximately nothing. For internal tools with bursty, unpredictable traffic — most internal tools — this is close to a cheat code: the same design serves ten requests a day or ten thousand a minute without re-architecture, and the bill follows usage down as readily as up.

But an inverted cost model inverts the failure modes too. When cost is linear with use, a runaway loop is not a performance bug — it is a spending bug. A Lambda that retries itself recursively, a poller that wakes every second against an empty queue, a debug log statement inside a hot loop: each is now a line item. Two surprises account for most first-bill shocks: CloudWatch log ingestion (verbose logging at volume costs real money — often more than the compute that produced it) and cross-service chatter (every hop in a chain of six services is billed at every hop).

Nearly everything in this guide is assembled from about a dozen services. Learn them as roles, not as brands — each one is the managed answer to a slice of the five jobs. This table is the cast list for every diagram that follows.

Here is the property that quietly changes your security and audit posture: in a serverless system, every action against every service is an authenticated, loggable API call. There is no SSH, no box to shell into, no path to the data that does not pass through IAM. A Lambda reading a table is an API call with an identity attached. A deploy is a series of API calls. Even an administrator's console click is an API call, recorded by CloudTrail like any other.

Two consequences follow. First, identity is the perimeter. There is no network wall to hide behind; every request is decided by policy. This sounds like a loss until you realize what it buys — the second consequence: your security posture becomes configuration, and configuration is reviewable, diffable, version-controlled, and auditable in a way that patch levels and firewall states never were. The question "is this system secure?" becomes the tractable question "what do these policies permit?" — a question you can answer by reading, and prove by testing.

Managed services trade away one thing you used to own: the ceiling. On your own machines, a slow job runs until it finishes. In serverless, every dimension has a number, and the numbers are design inputs. The ones that shape architectures most often:

The habit that separates smooth projects from painful ones is almost embarrassingly simple: skim the quotas page for each service you adopt before the design review, not during the incident. Half of serverless "outages" are a system meeting a documented number for the first time. The other half are in module 04.

Three parties, four arrows. A browser asks for a page; DNS points at CloudFront, which serves the file from the nearest edge cache — usually without touching the origin at all. On a cache miss, CloudFront fetches from the private S3 bucket and caches the file per its Cache-Control headers. To ship a change, a developer pushes to main; a GitHub Actions workflow syncs the files to S3 and then tells CloudFront to invalidate, so edges refetch the new files.

Read the figure through the module-01 lens: the request path has exactly two hops, both operated by AWS, both cached, both encrypted. Everything that can break lives on the deploy path — which runs perhaps ten times a day, is retryable, and inconveniences only you when it fails. Arranging for failures to land on the party who can fix them is most of what good architecture is.

The whole pattern stands or falls on one question: when a file changes, who finds out, and when? The answer is a two-tier cache policy that you should be able to recite from memory.

Assets get a year; HTML gets nothing. CSS, JavaScript, images, and fonts are published under versioned names — a content hash in the filename (app.3f2a.js) or a version query string — and served with Cache-Control: public, max-age=31536000, immutable. They can be cached forever precisely because they are never updated in place: a new version is a new URL. HTML is the opposite — it is the entry point that names those versioned assets, so it ships with no-cache (cache, but revalidate every time). The browser asks "still fresh?" on each visit, gets a cheap 304 most of the time, and gets the new HTML — pointing at the new asset names — the moment a deploy lands.

# the two-pass deploy this site actually uses
aws s3 sync ./ s3://site --exclude "*.html" \
  --cache-control "public, max-age=31536000, immutable"
aws s3 sync ./ s3://site --include "*.html" \
  --cache-control "no-cache"
aws cloudfront create-invalidation --paths "/*"

The bucket must not be publicly readable — and not because the files are secret (they're a public website). The reason is subtler: every property you care about lives on the CloudFront path. TLS termination, access logging, WAF rules if you add them, geographic restrictions, the cache itself. A publicly readable bucket is a second, unmonitored front door that bypasses all of it — an origin URL that shows up in a search index and serves your site with none of your controls attached.

The mechanism is Origin Access Control: the bucket stays private, and its policy grants read access to your CloudFront distribution specifically — CloudFront signs its origin requests, and S3 verifies the signature. One resource policy, and the invariant "all traffic passes through the front door" becomes something S3 enforces rather than something you hope.

Inventory what an attacker could want from this system. The content? Public. The bucket? Private, and read-only to the world via OAC. The interesting target is the thing that can write: the deploy pipeline. Whoever controls the CI workflow controls the site — and "the site" here means arbitrary JavaScript served to everyone who visits, from your domain, with your TLS padlock. A static site's XSS story is its supply chain.

The defenses are unglamorous and effective. Protect main — the workflow deploys on push, so pushing is deploying; require review before merge. Scope the CI role to this bucket only — s3:PutObject on site-bucket/* plus one invalidation permission, not s3:* on the account; module 01's blast-radius logic applies to robots exactly as it applies to functions. Prefer short-lived credentials — GitHub's OIDC federation gives the workflow a token that expires in minutes, so there is no long-lived secret to leak. A leaked static-site deploy key with broad S3 permissions is how "defacement" escalates to "data breach."

The pattern's boundary is precise: content changes on deploy, not per user. The moment a page needs per-user data, authenticated state, secrets, or writes, that content has left the static tier — and the answer is never to smuggle it back in. The classic smuggle is an API key in client-side JavaScript: every "secret" shipped to a browser is published, full stop. The dashboard that "just needs one little token" to call a third-party API has quietly become a proxy for anyone who opens view-source.

Growing up is graceful, though, and that's the point of the composition: the static tier keeps serving the shell — HTML, styles, scripts, anything cacheable — and the dynamic needs become a small API standing next to it, authenticated per user, holding the secrets server-side. That API is module 03, and Casework crosses the boundary immediately: the portal is static, but "show me my submissions" cannot be. The skill this module leaves you with is recognizing the boundary early — before the workaround, not after it ships.

Follow one request end to end. The portal sends GET /submissions/sub_84h2 with a token obtained by signing in through Cognito. API Gateway validates the token, applies throttling, and rejects malformed requests. The matched route invokes the Lambda function with the request as an event; the function queries DynamoDB and returns the item back up the same path. Logs and metrics land in CloudWatch throughout.

One operational note that earns its place in a design module: Lambda's logs and metrics arrive in CloudWatch automatically, but API Gateway's access logs and tracing are opt-in stage settings. Turn them on when you build the stage. You want the full request picture before the first incident, not after it — retrofitting observability during an outage is how one incident becomes two.

The most common junior mistake in this pattern is treating API Gateway as a pass-through and doing everything in the function: parse the token in code, validate the body in code, rate-limit in code. It all works — and it means every malicious, malformed, or unauthenticated request in the world gets to invoke a function you pay for and maintain.

The gateway can do three jobs before your code runs. Authentication: a Cognito or Lambda authorizer validates the token at the edge; requests without a valid identity never reach your function — they are rejected by managed infrastructure at no compute cost to you. Throttling: per-client rate and burst limits absorb abusive or buggy callers while everyone else proceeds. Request validation: a JSON schema on the route rejects malformed bodies — missing fields, wrong types — with a clean 400, which is free defense-in-depth against injection and a kindness to your own error handling, which now only sees well-formed input.

Three disciplines make Lambda functions boring in the best sense. Small and single-purpose: a function per route or per tightly-related route group, so each has one reason to change and one failure domain. Stateless: a function instance may be reused for many invocations (that's what makes warm calls fast) but may vanish at any time — anything worth keeping goes to the table, never to memory or /tmp. Narrowly armed: one IAM role per function, granting exactly its own actions on its own resources — dynamodb:GetItem and Query on the submissions table, not dynamodb:* on everything. When a function is compromised — dependency, injection, logic flaw — its role is the blast radius, and you decided its size on the day you wrote the policy.

Then there is the cold start, Lambda's most-discussed and most-misunderstood property. A function that hasn't run recently pays initialization on its first invocation — roughly 100 ms to 1 s depending on runtime and dependencies. For an internal tool at internal-tool traffic, this is almost always fine, and the engineering effort spent "solving" it is usually misallocated. Respect it in the tail latencies, mention it honestly in the design review, reach for provisioned concurrency only when a p99 target genuinely demands it.

DynamoDB is the piece that punishes imported habits. In SQL you model entities and relationships, then write whatever queries you need later; the database's job is to answer ad-hoc questions. DynamoDB makes the opposite trade: it will answer a known question in single-digit milliseconds at any scale, and it will make unknown questions painful — there are no joins, and scanning a table is an anti-pattern billed by the gigabyte. The discipline: write down the access patterns first — the exact questions with their exact parameters — and design the table to answer them.

Casework's questions: "a submission by ID," "a user's submissions, newest first," "all submissions pending review, oldest first." That yields a table keyed on PK = USER#{userId}, SK = SUB#{timestamp} — the second question becomes one Query, already sorted — plus a global secondary index keyed on the bare submission id for the first (a lone sub_84h2 isn't in the base key, so it needs its own index), and another keyed on status for the third. The entity diagram never appears; the questions were the design.

# access patterns → key design
Q1  submission by id            → Query    GSI2: PK=id  (bare id isn't in the base key)
Q2  my submissions, newest      → Query    PK=USER#u, ScanIndexForward=false
Q3  pending review, oldest      → Query GSI1: PK=STATUS#pending, SK=submitted_at

The synchronous path has a speed limit, and it is set by the party who can't be configured: the human. Around two seconds of wait, users retry, refresh, or distrust the tool — and their browser, their proxy, and your gateway all have timeouts standing behind theirs (API Gateway cuts sync invocations at 29 seconds by default — raisable on REST APIs via a quota increase, but doing so is usually the wrong move, since user patience is the real ceiling). So the rule: if a request triggers work that takes more than a second or two — PDF rendering, third-party calls, LLM inference, anything with someone else's latency in it — do not make the caller wait. Accept the request, record it, return 202 Accepted with a job ID, and finish the work asynchronously.

POST /submissions          → validate · persist PENDING · enqueue
202 Accepted
{ "id": "sub_84h2", "status": "PENDING" }

GET /submissions/sub_84h2  → { "status": "PROCESSING", … }   # poll, or notify

Notice what the rule really is: a boundary-drawing tool. It splits every feature into the part the user must witness (validation, persistence, the receipt) and the part they must not wait for (everything else). Drawing that line is the design act; the machinery on the far side of it — the queue, the worker, the retries — is module 04's subject, and Casework's document processing is about to need all of it.

Consider the coupling you get without the queue: the API function calls the processing function directly and waits. Now the producer moves at the worker's speed (a slow parse is a slow API), fails when the worker fails (a crashed parser loses the submission), and scales when the worker scales (a burst of submissions is a burst of heavy processing, immediately). Three couplings, three outages waiting.

Insert the queue and all three couplings dissolve. The producer's job shrinks to: validate, persist a PENDING record, enqueue a small message — { submission_id: "sub_84h2" }, a reference, never the document (module 01's 256 KB lesson) — and return. The worker polls at its own pace, processes in small batches, updates the record as it goes. Producer and worker now scale independently, fail independently, and deploy independently.

Three purchases, one insertion. Burst absorption: a spike of 10,000 submissions becomes a backlog that drains smoothly instead of a wall of timeouts — the queue converts a latency catastrophe into a temporary, visible, measurable delay. Failure isolation: if the worker or a downstream dependency is broken, messages wait safely in the queue; an outage of the processing tier becomes a processing delay rather than data loss, and recovery is automatic when the dependency returns. Retries for free: a message that fails processing reappears and is tried again — no retry code, no scheduler; failure handling is a property of the infrastructure rather than a feature of your application.

The backlog itself is an underrated gift: it makes load visible. Queue depth and message age are the two most honest metrics in the system — you can watch pressure build, alarm on it, and scale workers against it. The synchronous system under the same load shows you nothing until it shows you timeouts.

Here is the demand the queue makes in return. SQS guarantees at-least-once delivery: every message arrives, but the same message will occasionally arrive twice. This is not a quality problem to file a ticket about — it is a law of distributed queues. The mechanics make it concrete: when a worker receives a message, SQS doesn't delete it; it hides it for the visibility timeout. The worker processes, then deletes. If the worker crashes mid-processing — or just runs longer than the timeout — the message reappears and another worker receives it. The crash case is the feature (no message lost); the slow case is the trap (two workers, same message, both "succeeding").

So the worker must be idempotent: processing a message twice must equal processing it once. The standing technique is a conditional write keyed on a stable identity — the submission ID the message already carries:

# first delivery wins; the duplicate becomes a no-op
UpdateItem sub_84h2
  SET status = PROCESSING
  ConditionExpression: status = PENDING   # fails on the duplicate → skip

Every side effect needs the same treatment: writes are conditional, emails are deduplicated by a send-record keyed on submission_id, downstream calls carry an idempotency key. And set the visibility timeout deliberately — a common rule is roughly six times the function timeout, so a slow-but-alive worker isn't silently raced by its own retry.

Retries handle transient failure — the dependency that was down, the throttle that passed. But some messages fail every time: the corrupted PDF, the payload from a buggy client version, the edge case your parser never met. Without intervention, a poison message retries forever, burning compute and — in a batch — blocking its neighbors. The mechanism is maxReceiveCount: after N failed receives, SQS moves the message to the dead-letter queue, where it waits, intact, for a human.

Now the operational truth the diagram can't show: a DLQ nobody monitors is a place where work goes to die — with a receipt. The user got a 202; the system accepted responsibility; the message is sitting in a queue no one reads. That is strictly worse than a synchronous failure, which at least told the user. So the DLQ ships with three attachments, or it isn't finished: an alarm on depth > 0 (a dead letter is a sev-worthy event in a system that promised acceptance), a runbook (how to inspect, fix, and redrive messages back to the source queue after the bug is fixed), and a retention check (14 days maximum — the clock is running on your fix).

Two refinements complete the pattern. First, ordering: standard SQS queues don't guarantee it. Messages arrive roughly in order, but "roughly" is not a contract, and retries reshuffle. If Casework's decision.recorded must never be processed before its submission.created, you have three options in descending order of preference: design the workers so order doesn't matter (idempotent, state-checking handlers that re-fetch current truth rather than trusting message sequence); use a FIFO queue with a message group per submission — strict order within each group, at the price of lower throughput and head-of-line blocking within the group; or reach for the orchestration of module 06, where sequence is explicit. Choosing FIFO by reflex is the common error: most "ordering requirements" dissolve when handlers check state instead of assuming sequence.

Second, batching: workers receive a batch per invocation — up to ten by default, raisable to 10,000 on a standard queue — and by default one bad message fails the whole batch — nine innocent messages retried alongside one poison pill, all ten inching toward the DLQ together. Enable partial-batch responses (ReportBatchItemFailures), so the worker reports exactly which messages failed and only those retry. It's a checkbox and three lines of code, and it's the difference between a poison message quarantining itself and a poison message taking hostages.

The mechanics are compact. Something happens — the decision function from module 03 finishes its write — and it publishes an event to the bus: a small JSON document with a source (casework.review), a detail-type (decision.recorded), and a detail payload. Rules on the bus match by pattern — source, type, fields of the payload — and each matching rule forwards a copy to its targets: a Lambda, a queue, a Step Functions execution, another bus.

Contrast the alternative honestly, because it's what most systems do first: the decision function calls the notifier, then the indexer, then starts the workflow — in a row, in code. Every new consumer is a code change to the producer. Any consumer's failure is the producer's failure. Any consumer's latency is the producer's latency. The bus severs all three dependencies at once.

The architectural payoff deserves its own section because it changes how teams work, not just how systems run. When the new requirement arrives — "we also need to notify the review team" — the change is: add a rule, deploy a subscriber. The producing code doesn't change, isn't redeployed, and cannot be broken by the addition. The blast radius of new functionality is the new functionality.

This is the open-closed principle wearing an ops uniform, and its compounding value is organizational: teams ship consumers independently, on their own schedules, without negotiating deploy windows with the producer's owners. The event catalog becomes the integration surface — a new team reads the catalog, subscribes to the facts it needs, and never files a ticket against the producing service. Systems that grow this way stay comprehensible, because every capability is a legible pair: which facts it consumes, which facts it emits.

Here is the bus's hidden invoice. The moment a second consumer subscribes to decision.recorded, its field names, types, and semantics have become a contract with consumers you don't know about — that's the whole point of not knowing your subscribers, applied against you. Rename decision_at to decided_at and somewhere, a dashboard silently reads undefined; there is no compiler, no failing build, no 400 response to tell you. The event just stops meaning what it meant, quietly, in someone else's code.

The disciplines are exactly an API's, because this is an API: additive changes only (new fields are safe; renames and removals are breaking); version explicitly (a version field in the detail, or a versioned detail-type, with old and new published side by side during migrations); document like a public surface (a schema registry or a versioned catalog in the repo — discoverable, not tribal); and carry references, not blobs — { submission_id, decision, decided_by, correlation_id }, not the entire document, both because payloads sprawl into every subscriber's logs and because a fat payload is a snapshot that goes stale while a reference is always current.

In module 03, a request was a call stack: one arrow in, one arrow out, one log stream to read. That's gone now. "What happened to submission sub_84h2?" is answered across the API's logs, the queue worker's logs, the bus, three subscribers' logs, and a workflow execution — each with its own timestamps and request IDs, none aware of the others. Without deliberate design, tracing one submission through the system is an afternoon of grepping in five consoles.

The deliberate design is one field and one habit. The field: a correlation ID — minted once at the edge (the API request that started everything), then propagated unchanged through every queue message, every event payload, every workflow input, every downstream call. The habit: structured logs that always include it. JSON logs with correlation_id, submission_id, and a terse event name, in every service, means CloudWatch Logs Insights can reassemble the entire story with one query:

fields @timestamp, @log, event, detail
| filter correlation_id = "corr_7f3k"
| sort @timestamp asc

That query output — the submission accepted here, queued there, scanned, decided, indexed, notified, archived, in order, with timing — is the call stack you gave up, rebuilt from discipline. It costs one field and a logging convention; it is the difference between event-driven systems that are observable and event-driven systems that are folklore.

Three operational refinements finish the pattern. A queue per consumer: route rules through SQS rather than invoking subscriber Lambdas directly, so each consumer gets module 04's full toolkit — buffering, retries, a DLQ of its own — and one slow consumer never drops events or backs up its neighbors. The bus fans out; the queues absorb. The archive: EventBridge can retain everything that crosses the bus, replayable on demand. This quietly changes what's recoverable: deploy a buggy indexer, fix it, replay the week — the events re-run and the index rebuilds. Stand up a brand-new consumer next quarter and replay history into it, as if it had been subscribed all along. The archive turns "we didn't think of that consumer yet" from a data loss into a delay. (Replay is redelivery — your consumers' idempotency, again, is what makes it safe.)

And eventual consistency, stated honestly: subscribers react in milliseconds-to-seconds, not instantly. The search index trails the decision by a moment; the dashboard trails the index. This is fine — if the interfaces say so. "Decision recorded — search results update shortly" is honest; a search page that implies real-time and silently isn't will generate bug reports forever. Eventual consistency is an architecture fact; whether it's a UX bug is a copywriting decision.

Name the two styles precisely, because the choice between them is the module's real subject. Choreography is what modules 04 and 05 built: services react to each other's messages and events, no one is in charge, and the overall behavior emerges. Orchestration adds a conductor: a central definition that says step one, then step two, branch here, retry there — and tracks each execution through it.

Choreography's strengths are decoupling and evolution — consumers as additions, no central chokepoint. Its weakness is that the process is invisible: no single place shows the sequence, and cross-step concerns (timeouts spanning steps, compensation when step four fails after step two committed) have no natural home. Orchestration inverts the trade: the process is explicit, visible, versioned, and recorded — at the cost of a central definition that every step change touches.

The selector: how much does it matter that the process, as a whole, is legible and provable? Plumbing — thumbnail on upload, index on change, notify on decision — wants choreography; nobody audits the thumbnail pipeline. Processes with stakes — reviews, approvals, escalations, anything someone will later ask "was the procedure followed?" about — want orchestration, because for those, the visibility is the feature. The slogan that compresses this module: orchestrate workflows; choreograph plumbing.

Casework's review process, as a state machine: a Validate task (Lambda) checks the submission and enriches it; a Scan task runs the document check; a choice state branches on the verdict — clean submissions proceed to human review, flagged ones to escalation; after the decision, a Record task writes the outcome and a final task publishes decision.recorded to module 05's bus. Deadlines, retries, and error routes are part of the definition itself.

Notice what the tasks don't contain. Validate declares Retry: 3 attempts, exponential backoff on the state — no retry loop in the function. Scan declares Catch → Quarantine — no try/except-and-route logic in code. The functions shrink to pure business logic; the process concerns — sequence, retry, branching, timeout, compensation — live in the definition, where they are visible, reviewable in a pull request, and uniform across steps. That relocation is the pattern's quiet gift: process logic stops being smeared through application code.

The step that makes orchestration irreplaceable for Casework is the one no queue can model well: wait for a person. Step Functions does it with a callback token: the Human-review state (declared waitForTaskToken) generates a token, hands it to your notification path — the reviewer gets a link — and the execution simply stops. Not polling, not holding a connection: stopped, durable, costing nothing, for hours or days. When the reviewer decides, the decision endpoint calls SendTaskSuccess with the token and the verdict, and the execution resumes exactly where it paused. A timeout on the state (five days, say) routes to an escalation branch, so "the reviewer never responded" is a designed path rather than a stuck execution.

Now the trap. The token proves which execution; it does not prove who. Anyone holding the token can resume the workflow, so if the approval link is the only control, approval-by-forwarded-email is your access model. The decision endpoint must therefore do two checks that the token cannot: authenticate the human (module 03's authorizer — the reviewer signs in) and authorize them for this request (are they an assigned reviewer of sub_84h2, and not, say, its submitter?). Then, and only then, it spends the token. Token proves execution; session proves identity; policy proves entitlement — three different facts, three different mechanisms, all required.

Step Functions comes in two modes, and the choice is architectural, not a pricing detail. Standard workflows run up to a year, bill per state transition, guarantee exactly-once state execution, and record full execution history — built for Casework's review, which waits days for humans. Express workflows cap at five minutes, bill per request and duration, run at enormous volume, and log history only through CloudWatch — built for high-rate, short-lived processing where Standard's per-transition pricing would sting. The tell: humans and days → Standard; volume and seconds → Express.

Then the caveat this guide has been building toward. Standard's execution history — every state, input, output, and timestamp — is queryable for roughly 90 days after completion, and then it is gone. For debugging, 90 days is generous: history is the debugger, letting you inspect exactly what step four received the Tuesday before last. But Casework's compliance story needs "prove the process was followed" answerable in year three, and Express workflows never had rich history to begin with. So: treat execution history as the working record, never the archive. The durable proof is the completion event — the workflow's final step publishes decision.recorded with the decision, actor, and timestamps, and module 08's trail records it durably. The event is the evidence; the history is the debugger.

Every pattern in this guide has a failure mode of enthusiasm, and orchestration's is the most seductive: once the review workflow is visible and provable, everything starts looking like it deserves a state machine. Resist. For simple "A then B," a queue is cheaper, simpler, and has fewer moving parts; wrapping two steps in Standard-workflow ceremony adds per-transition cost, a definition to version, and an execution history nobody will read. The bar from section one stands: orchestration is earned by processes whose legibility matters — stakes, branches, humans, deadlines, or auditors.

Two operational clauses close the module. Payload limits, again: state-to-state data passes through the execution itself, capped at 256 KB — the same lesson as the queue: pass submission_id and S3 references, never documents, or the workflow that ran fine in testing dies on the first large evidence file. The conductor's role: the state machine's IAM role can invoke its steps — its Lambdas, its publish action — and nothing else. It is the conductor, not a superuser; module 03's blast-radius clause applies to the process layer exactly as it applied to functions.

Walk one evidence file through. It arrives in the raw bucket — uploaded through the portal or delivered by another system. The ObjectCreated event triggers processing — through a queue, per module 04, because arrival is bursty and the queue is how bursts become backlogs. The processor validates and parses the file, writes extracted metadata to DynamoDB — type, dates, parties, page count, the fields the portal queries — and writes a normalized, columnar copy to the curated bucket, partitioned by date. The original in raw is never modified. Athena then answers analytical questions with plain SQL over the curated files, paying per query.

The two-product split is the design insight. The portal's questions ("this submission's documents") are known, keyed, and hot — module 03's access-pattern territory. Compliance's questions ("slice by team, month, attestation status") are ad-hoc, relational, and cold — exactly what module 03 told you DynamoDB punishes. One ingest, two representations, each workload served by the store built for it.

Module 01's limits already ruled out files transiting Lambda; the pattern that replaces it deserves precision. The client asks the API (authenticated, module 03) for an upload grant; the API returns a pre-signed URL — a time-limited, signed permission to perform one specific operation against S3. The client PUTs the file directly; bytes flow client-to-S3, never through your compute. The API stays in charge of who may upload and where the object lands (it chooses the key); S3 does the heavy lifting.

Constrain the grant like the credential it is: short expiry (minutes — it's an upload window, not a bearer token for the afternoon), fixed key (the API names the object — never let clients choose paths in your bucket), fixed content type, and — the one everyone learns the hard way — a size cap, which a plain pre-signed PUT cannot express. A pre-signed PUT signs headers, not payload size; to cap size you use the pre-signed POST policy form, whose content-length-range condition S3 enforces at upload time. If uploads are unbounded, your storage bill is a stranger's decision.

The raw bucket has one rule, and the whole compliance story leans on it: nothing modifies an original, ever. The processor reads raw and writes elsewhere; normalization, redaction, format conversion, compression — every transformation produces a new object in the curated bucket, keyed back to its source. The raw object stays byte-for-byte as received, because "what did the submitter actually send?" is a question you will be asked, and a transformed copy is an argument where an original is an answer.

Make the rule structural, not behavioral. Versioning on (an overwrite becomes a new version rather than a destruction); per-stage roles (the upload path can only PutObject to raw; the processor reads raw and writes curated; analysts read curated only — nobody holds write access to raw after landing); and for genuinely evidentiary workloads, Object Lock on the raw bucket, which module 08 covers in anger. Record a checksum (S3 gives you one; store it in the metadata item) so "unchanged since receipt" is demonstrable arithmetic rather than an assurance.

Athena's pricing is its design pressure: you pay per byte scanned. Everything that matters about laying out the curated bucket follows from that one sentence. Partition — lay keys out as dt=2026-07/team=alpha/…, and a query filtered to July scans July, not the archive; unpartitioned, every query scans everything forever, and the "cheap" pattern quietly becomes the expensive one. Go columnar — Parquet stores columns together, so SELECT team, COUNT(*) reads two columns instead of whole JSON rows; with compression, typical scans shrink by an order of magnitude or two versus raw JSON. Compact — millions of tiny files make every query pay per-object overhead; batch small events into fewer, larger files (a scheduled compaction job, or Data Firehose doing the batching on the way in).

-- compliance asks; Athena answers; the scan is one partition, two columns
SELECT team, COUNT(*) AS missing
FROM curated.submissions
WHERE dt = '2026-07' AND attestation = false
GROUP BY team;

Register the schema once in the Glue Data Catalog (a metadata store both Athena and future tools read), and the curated bucket becomes, functionally, a warehouse — with no cluster, no capacity, no idle cost, and the same eleven-nines substrate as everything else. For an internal tool, this is the whole analytics stack.

Three habits keep the pipeline trustworthy at volume. Poison files fail alone: one malformed upload must never stall the line. Files process independently (the queue between event and processor is what makes that true), failures route to the DLQ carrying the object key, and module 04's partial-batch discipline keeps one bad file from taking nine hostages. The DLQ alarm matters more here than anywhere: a submission that silently never processed is a compliance gap wearing a 202.

Idempotency, again: S3 events ride at-least-once machinery, and event replay (module 05) will re-run history on purpose. The processor is safe because both writes tolerate repetition — the metadata item is a full Put keyed on the object (same input, same item), and the curated write is deterministic (same source key → same curated key, overwritten with identical content). Reprocessing is then not merely safe but useful: fix the parser, replay the month, and the curated bucket heals.

Watch the lag: the pipeline's health metric is freshness — the age of the oldest unprocessed object (queue age, module 04's honest metric, plus a periodic raw-vs-processed reconciliation). Dashboards that show throughput but not lag report how fast you're going, not whether you're falling behind — and in an evidence pipeline, behind is the state that becomes a finding.

The design principle first. Don't just store "submission sub_84h2 is approved" — store the append-only sequence of facts that got it there: submitted by whom, at when; scanned with what verdict; reviewed by whom; decided on what basis. Current state is derivable from events; events are not derivable from state. A state row that reads APPROVED answers today's question; the event sequence answers the auditor's questions — who, when, in what order, under which policy version — including the ones nobody has thought to ask yet.

And crucially, the trail has two layers, because "who did what" has two meanings. The application layer records what users did: submissions, decisions, escalations — your events, emitted by your code. The infrastructure layer records what every IAM identity did to the system itself: policy changes, table modifications, bucket configuration — CloudTrail's record, emitted by AWS whether your code cooperates or not. Auditors ask about both, and the second layer is what makes the first believable: the application's trail is only as trustworthy as the answer to "who could have altered it?" — which is exactly the question CloudTrail answers.

"We never update audit rows" is a team convention; conventions don't survive incidents, personnel changes, or subpoenas. The trail is append-only because policy makes the alternatives impossible, and it takes two locks. First, IAM: roles that write the events table get dynamodb:PutItem — no UpdateItem, no DeleteItem. Second — the one everyone misses — a condition expression on every write: a plain PutItem on an existing key silently overwrites it, so "append-only" IAM still permits rewriting history one key at a time. ConditionExpression: attribute_not_exists(PK) closes the loophole: unique event IDs, insert-or-fail, never insert-or-replace.

# both locks, or it isn't append-only
IAM:  Allow dynamodb:PutItem on events-table        # no Update, no Delete
API:  PutItem { PK: "EVT#01J8ZK…", … }
      ConditionExpression: attribute_not_exists(PK)  # overwrite → error

Then get the events somewhere the application can't reach at all. DynamoDB Streams emits every write at the database layer — the application cannot skip it — into an archiver Lambda that batches events into the archive bucket. Two operational honesty notes: Streams orders changes per item, not across the table, so each event carries its own timestamp and sequence and those are the chronology, not stream order. And the stream is a 24-hour buffer, not a guarantee — an archiver broken longer than the retention window drops records into a silent gap. Alarm on the stream's iterator age, give the archiver an on-failure destination, and treat that alarm as a sev-2: it is the sound of history not being written.

The archiver lands events in the archive bucket with S3 Object Lock in compliance mode: write-once-read-many storage where, until the retention period ends, no identity — not an administrator, not the root account — can alter or delete an object. This is the property that upgrades your trail from "we log things" to "we can prove things": immutability enforced by the platform against everyone, including you. Respect its edge: compliance mode is genuinely irreversible — test retention settings on a scratch bucket first, because a misconfigured ten-year lock is a ten-year mistake. And set retention deliberately per record class: regulations set floors (keep at least N years), privacy law sets ceilings (don't keep personal data longer than needed), and the archive should carry references and event facts, not personal-data payloads (module 05's clause, paying off).

In parallel, CloudTrail ships the infrastructure layer to its own locked bucket — including any attempt to tamper with this very pipeline. Know its default honestly: management events (creating, configuring, deleting things) are on; data events — S3 object reads and writes, DynamoDB item operations — are not, and must be enabled explicitly for the buckets and tables that matter. "Who read the evidence file?" is only answerable if you turned that on before the question was asked.

Finally, separate the watcher from the watched: deliver CloudTrail and the archive to a dedicated logging account that workload operators can read but never write. A trail an administrator of the workload can edit is testimony; a trail in an account they cannot touch is evidence. Alarm on the meta-signals — trail configuration changes, Object Lock policy changes, denied writes against the archive — and when the auditor finally asks, Athena (module 07, reused whole) answers with SQL across the archive: every action on sub_84h2, everything one user touched in March, every permission change this quarter — minutes, not weeks.

Look back at seven modules of security notes and see the pattern: serverless security is a small number of moves applied consistently. Under the shared-responsibility model (module 01), AWS handles the layers below you; what remains is configuration — who can do what, to which data, and how you'd know. Configuration is reviewable, diffable, and auditable in a way patch levels never were. The six moves:

If you internalize one sentence from the entire guide, take this one: in serverless systems, the IAM policy review is the architecture review. The diagram tells you what the system is supposed to do; the roles tell you everything it can do. Keep the gap between those two as close to zero as you can — every permission no arrow uses is attack surface with no compensating feature.

Real systems compose these patterns rather than choosing one — Casework ended as a static portal (m2) calling an API (m3) that queues heavy work (m4), publishes events (m5) that start workflows (m6), over an ingestion pipeline (m7), with everything draining into the audit trail (m8). The skill is mapping each requirement to its shape:

Five honest gotchas, each the compressed form of a lesson above: the cost model inverts (billing alarms on day one; log ingestion is the classic surprise); everything has a limit (skim the quotas before the design review); idempotency is not optional (queues redeliver, events duplicate, retries retry — a handler unsafe to run twice is a latent bug); observability is a design input (correlation IDs and structured logs are decided in the design, not bolted on); and start boring — patterns 03 and 04 cover most internal tools, and every hop you add is latency, cost, and a new place for messages to get lost.

A last thought, the same one the original edition of this guide closed on, because it has not stopped being true. Diagrams like these are the easy part — any of them can be assembled in an afternoon. The durable work is in the requirements the boxes encode: what must never be lost, who must approve what, what you'd need to prove a year from now. Get those right, and the architecture mostly draws itself.