Document intelligence pipeline

This little map is the whole project on one screen. Nothing here is a program you run — it shows who calls whom, and the key idea is that nothing polls (nothing sits in a loop asking "are we done yet?"). Each step wakes up the next one with an event, so the pipeline only does work when there is a document to process.

1. A file lands in an S3 bucket (S3 is AWS's file storage). That upload automatically fires an event — think of it as a doorbell.
2. The doorbell wakes the first Lambda (a Lambda is a small function AWS runs for you, no server to manage), start_textract, which hands the file to Textract — AWS's OCR service that reads text out of PDFs and images. async means Textract works in the background and tells us later, so we don't wait.
3. When Textract finishes it sends an SNS message (a notification), which wakes the second Lambda, extract_and_store. That one asks Claude on Bedrock to pull out the useful fields (with a Guardrail checking for unsafe content), then saves the result to DynamoDB, a fast key-value database.
4. "keyed by doc id → idempotent" means each document is stored under its own id. If the same event arrives twice, the second write just overwrites the first — no duplicates, no harm.

Try this: Trace one PDF through the arrows top to bottom out loud. If you can say what each arrow's event is (upload, done-notification) you already understand event-driven architecture — the backbone of serverless.

This is the heart of the pipeline: the second Lambda. It runs after Textract has read the document, and its job is to turn a wall of raw text into clean, structured fields (vendor, total, due date) and save them. Read it in four movements — set-up, gather text, ask Claude, store.

1. Set-up (top). boto3 is the AWS SDK for Python; boto3.client("textract") and client("bedrock-runtime") create objects that talk to those AWS services. os.environ["TABLE"] reads the DynamoDB table name from an environment variable that Terraform will set later — so the code isn't hard-wired to one table.
2. INVOICE_TOOL is a tool schema: a JSON contract that tells Claude exactly what shape of answer you want. It defines four fields and marks vendor and total as required. This is how you get reliable machine-readable output instead of a paragraph of prose.
3. Gather text. The while True: loop calls get_document_analysis to pull Textract's results one page at a time. NextToken is AWS's "there's more — here's your bookmark" marker; when it comes back empty, break ends the loop. It keeps only the LINE blocks and joins them into one text string.
4. Ask Claude. brt.converse(...) sends that text to Claude on Bedrock. toolChoice forces Claude to answer by filling in the record_invoice tool (not free text), and guardrailConfig runs the safety Guardrail (W6) on the exchange. The modelId string is the specific Claude 3.5 Sonnet version on Bedrock.
5. Store. next(b["toolUse"]["input"] for ... if "toolUse" in b) grabs the structured fields Claude filled in. ddb.put_item writes them to DynamoDB using doc_id as the key — so re-running on the same document overwrites rather than duplicates (that's the "idempotent" promise).

What the output means: A dictionary like {"stored": {"vendor": "Acme", "total": 1250.0, ...}}. Those same fields are now a row in DynamoDB, queryable by doc_id.

Try this: Add "currency": {"type": "string"} to the tool's properties. Claude will start returning a currency too — changing the schema is how you teach the pipeline to extract new fields, no prompt rewording needed.

This is Terraform — infrastructure as code. Instead of clicking around the AWS console, you declare the resources you want in these .tf files, and Terraform creates or updates them to match. Here we declare the two pieces that store and process data: the database table and the Lambda function.

1. Each resource "aws_..." "name" { } block describes one AWS thing. The first string is the type (what AWS resource); the second is a local nickname you use to reference it elsewhere in the file.
2. aws_dynamodb_table.invoices defines the table. billing_mode = "PAY_PER_REQUEST" means you pay per read/write (no capacity to plan). hash_key = "doc_id" makes doc_id the primary key — the same key the Lambda writes by, which is what makes storage idempotent.
3. aws_lambda_function.extract defines the Lambda: which file/function to run (handler), the Python runtime, a 120-second timeout, and the filename of the zipped code to deploy.
4. The environment { variables { ... } } block is the payoff: it feeds aws_dynamodb_table.invoices.name and the guardrail id into the Lambda as the TABLE and GUARDRAIL env vars the Python read with os.environ. Terraform wires the two files together automatically.

Try this: Notice the code never types the table name twice. Change the table's name once and both the table and the Lambda's env var update — that reference (aws_dynamodb_table.invoices.name) is why infra-as-code stays consistent.

The final step: prove the whole chain actually works, end to end, from the outside. This tiny script does what a real user would — drop a file in S3 — then checks that a result showed up in the database a moment later. If it does, every arrow in the architecture map fired correctly.

1. s3.upload_file("sample-invoice.pdf", "my-docs", ...) puts a test PDF into the bucket. That single upload is the doorbell that triggers the pipeline — you don't call any Lambda yourself; the S3 event does it for you.
2. time.sleep(30) waits 30 seconds. Textract's OCR and the Claude call happen in the background, so you give the pipeline a moment to finish before you look.
3. ddb.scan(Limit=5)["Items"] reads up to 5 rows straight from DynamoDB and prints them. If the pipeline worked, you'll see the extracted invoice fields there.

What the output means: A list of DynamoDB items, e.g. the vendor/total/due_date Claude pulled out of your sample PDF. An empty list [] means the document hasn't finished processing (wait longer) or a step failed (check the Lambda's CloudWatch logs).

Try this: Run it twice with the same file. Because storage is keyed by doc_id, you still see one row, not two — live proof that the idempotency you built into the design actually holds.