Skip to content
All projects

AI engineering · Python · AWS

Prumo

The real accuracy of an AI in production: a few human labels correct the LLM judge's bias, and the prompt rolls back on its own when quality drops.

View code on github.com(opens in a new tab)
Client
Personal open-source project
Role
Software Engineer — independent project
Year
2026
Technologies
  • Python
  • FastAPI
  • Amazon Bedrock
  • AWS Lambda
  • AWS CDK
  • Step Functions
  • DynamoDB
  • AppConfig
  • CloudWatch
  • Pydantic
  • NumPy
  • pytest
  • GitHub Actions

Context

Open-source Python 3.12 service that measures the real accuracy of an AI in production. The AI (Amazon Nova Lite on Bedrock) classifies leads as QUENTE, MORNO or FRIO (hot, warm or cold), an LLM judge checks 100% of the decisions, a random sample goes to human review and statistics correct the judge's bias. All data is synthetic. The project picks up where Easy Control leaves off: there, the AI pre-qualifies leads and the CEO reviews every hot or warm one before the first contact, because the volume still allows it.

Challenge

At 50 times the volume, reviewing everything is no longer possible. Trusting the LLM judge alone is not enough either: its errors are systematic, and when a new prompt makes the AI wrong more often, nothing breaks, no alert fires and the judge's score barely moves. The challenge was to keep the automation under control with few human labels, know the real accuracy with an error margin that stays honest even when checked every hour, and roll the prompt back without waiting for someone to notice.

Results

simulations in which the drop from 92% to 76% was detected and rolled back while the judge stayed around 91% (prumo bench benchmark)
10/10
wrongful rollbacks of v3, the version that improves the AI, across every benchmark setting
0/10
real accuracy of Nova Lite in the AWS test with 30 synthetic leads, against 70% approved by the Nova Pro judge, which let 4 wrong classifications through
60%
automated tests: 205 in the app, with 94% coverage, and 56 for the CDK infrastructure
261

Key deliverables

  1. 01Implemented Prediction-Powered Inference (PPI and PPI++) in pure Python, with the standard library's math and statistics modules, using a random sample of human labels to remove the LLM judge's bias and report the real accuracy with a confidence band, plus an anytime-valid confidence sequence (Waudby-Smith et al., 2024) for hourly readings.
  2. 02Architected the full flow on AWS with Python CDK: a FastAPI service on Lambda (through Mangum), the classifier and the judge on Bedrock through the Converse API with forced tool use, a single-table DynamoDB design with atomic counters and a 180-day TTL, and human review on Step Functions with waitForTaskToken.
  3. 03Automated the prompt rollback: the hourly estimator publishes the metric as EMF to CloudWatch, and the alarm for 2 consecutive readings below target (85% SLO) triggers, through EventBridge, a Lambda that reverts the version in AppConfig with compare-and-swap, with the estimator itself as a safety net.
  4. 04Built the FastAPI web dashboard with a simulator that runs at no AWS cost and the prumo bench benchmark, which reproduces a prompt version that drops the AI from 92% to 76% while the judge stays around 91%, comparing methods and human-label volumes per hour.
  5. 05Created 261 automated tests (205 in the app, with 94% coverage, and 56 for the infrastructure), including Hypothesis property tests, Monte Carlo simulations in NumPy that validate interval coverage and AWS adapters tested with Stubber and moto, under strict mypy, ruff and CI on GitHub Actions.
  6. 06Secured operations with an admin token generated by Secrets Manager and read at cold start, deploys from GitHub Actions through OIDC with no stored access key, SNS alarms with an SQS dead-letter queue for the rollback, and synthetic data only.

Screens and components

  1. The judge and the real rate

    After the incident, the judge's score barely moves, the estimated real accuracy drops with v2 and the prompt rolls back to v1 on its own. The dashboard comes from the simulator, with fictitious data.

    Prumo dashboard in light theme, in simulated mode, after the incident: prompt v1 back in production with the Healthy status, the judge saying 92.1%, an estimated real accuracy of 91.0% with an 88.3% to 93.7% band, an error-budget burn rate of 0.6× and 1,789 human labels, 9.3% of 19,200 decisions; the chart of the last 48 readings, where the judge's score stays nearly flat around 91% while the estimated real accuracy drops with its band between the v2 and v1 markers, below the dashed 85% target, and returns to its previous level after v1; and the events: v1 in production, v2 published, v2 safely below target in 2 consecutive readings and the rollback from v2 to v1.
    Prumo dashboard in light theme, in simulated mode, after the incident: prompt v1 back in production with the Healthy status, the judge saying 92.1%, an estimated real accuracy of 91.0% with an 88.3% to 93.7% band, an error-budget burn rate of 0.6× and 1,789 human labels, 9.3% of 19,200 decisions; the chart of the last 48 readings, where the judge's score stays nearly flat around 91% while the estimated real accuracy drops with its band between the v2 and v1 markers, below the dashed 85% target, and returns to its previous level after v1; and the events: v1 in production, v2 published, v2 safely below target in 2 consecutive readings and the rollback from v2 to v1.

  2. The incident

    During the incident, the judge stays near 91% while v2's estimated real accuracy falls below the 85% target, and the status turns to Below target. Simulator with fictitious data.

    Prumo dashboard in dark theme, in simulated mode, during the incident: controls to publish prompts v1 and v3, human labels per hour set to 40 and automatic rollback on; indicators with prompt v2 in production and the Below target status, the judge saying 90.8%, an estimated real accuracy of 77.2% with a 70.0% to 84.4% band, an error-budget burn rate of 1.5× and 521 human labels, 10% of 5,200 decisions; on the chart, after the v2 marker, the judge's score stays near 91% while the estimate and the simulator's ground truth fall below the 85% target; and, in the events, the v2 deploy, with no rollback yet.
    Prumo dashboard in dark theme, in simulated mode, during the incident: controls to publish prompts v1 and v3, human labels per hour set to 40 and automatic rollback on; indicators with prompt v2 in production and the Below target status, the judge saying 90.8%, an estimated real accuracy of 77.2% with a 70.0% to 84.4% band, an error-budget burn rate of 1.5× and 521 human labels, 10% of 5,200 decisions; on the chart, after the v2 marker, the judge's score stays near 91% while the estimate and the simulator's ground truth fall below the 85% target; and, in the events, the v2 deploy, with no rollback yet.

  3. Indicators on mobile

    On mobile, the indicators separate what the judge says, 90.8%, from the estimated real accuracy, 77.2%, with the confidence band and the budget burn rate right next to them. Fictitious data.

    • Top of the Prumo dashboard on mobile, in simulated mode, during the incident: buttons to publish prompts v1 and v3, human labels per hour set to 40 and automatic rollback on; indicators with prompt v2 in production and the Below target status, the LLM judge's raw score on the What the judge says card, 90.8%, an estimated real accuracy of 77.2% with a 70.0% to 84.4% band and an error-budget burn rate of 1.5×, which spends the monthly budget in 20 days.
      Top of the Prumo dashboard on mobile, in simulated mode, during the incident: buttons to publish prompts v1 and v3, human labels per hour set to 40 and automatic rollback on; indicators with prompt v2 in production and the Below target status, the LLM judge's raw score on the What the judge says card, 90.8%, an estimated real accuracy of 77.2% with a 70.0% to 84.4% band and an error-budget burn rate of 1.5×, which spends the monthly budget in 20 days.

  4. Human review and events

    In the review queue, the reviewer says whether the AI was right and sees the simulator's answer key; the events log deploys, alarms and rollbacks with date and time. Fictitious data.

    • Human review queue on mobile, with the note that only sampled decisions show up: three synthetic text leads, with profession, age, monthly turnover and financing, each with the AI's decision, MORNO (warm), and the LLM judge's approval; two have The AI was right and The AI was wrong buttons, and the middle one, already labeled, shows Answer key: FRIO (cold). The AI was wrong, and the judge had wrongly approved it. Below, the link to see the rubric used as the answer key.
      Human review queue on mobile, with the note that only sampled decisions show up: three synthetic text leads, with profession, age, monthly turnover and financing, each with the AI's decision, MORNO (warm), and the LLM judge's approval; two have The AI was right and The AI was wrong buttons, and the middle one, already labeled, shows Answer key: FRIO (cold). The AI was wrong, and the judge had wrongly approved it. Below, the link to see the rubric used as the answer key.
    • Prumo event list on mobile, newest first, with date and time: rollback from v2 to v1; v2 safely below target in 2 consecutive readings; prompt v2 published, previously v1; and prompt v1 in production, with Prumo measuring quality every hour. Below, the start of the human review queue with two synthetic leads approved by the judge.
      Prumo event list on mobile, newest first, with date and time: rollback from v2 to v1; v2 safely below target in 2 consecutive readings; prompt v2 published, previously v1; and prompt v1 in production, with Prumo measuring quality every hour. Below, the start of the human review queue with two synthetic leads approved by the judge.

  5. Design system foundations

    Ten color tokens in light and dark, type, radii and spacing support the Jinja2 and framework-free JavaScript dashboard, from buttons in every state to the SLO statuses.

    Prumo design system sheet, foundations and primitives, with the plumb-line symbol beside the title and chips for tokens in prumo.css, Jinja2 with framework-free JavaScript and light + dark. Color: ten tokens as light and dark pairs with hex values, estimate (real rate), judge (LLM judge), slo (target and alarm), warn (watch), focus, ink, muted, paper, surface and rule. Type in Familjen Grotesk and IBM Plex, from the Taxa real (real rate) heading to the 91.0% figure and the 88.3% to 93.7% band in mono. Radii from 4 to 999 px, spacing from 6 to 24 px, a 1 px border and a 2 px focus ring. Primary, default, warn and good buttons in default, focus-visible and disabled states; the panel, kpi and q-item surfaces; the controls for human labels per hour and automatic rollback; the SLO statuses Healthy, Under watch, Below target and Collecting labels; the chart legend; the log event types; and the header, the admin token field, the lead text and the error notice.
    Prumo design system sheet, foundations and primitives, with the plumb-line symbol beside the title and chips for tokens in prumo.css, Jinja2 with framework-free JavaScript and light + dark. Color: ten tokens as light and dark pairs with hex values, estimate (real rate), judge (LLM judge), slo (target and alarm), warn (watch), focus, ink, muted, paper, surface and rule. Type in Familjen Grotesk and IBM Plex, from the Taxa real (real rate) heading to the 91.0% figure and the 88.3% to 93.7% band in mono. Radii from 4 to 999 px, spacing from 6 to 24 px, a 1 px border and a 2 px focus ring. Primary, default, warn and good buttons in default, focus-visible and disabled states; the panel, kpi and q-item surfaces; the controls for human labels per hour and automatic rollback; the SLO statuses Healthy, Under watch, Below target and Collecting labels; the chart legend; the log event types; and the header, the admin token field, the lead text and the error notice.

  6. Domain components

    The domain components put the judge next to the real rate, show the SLO status, the 48 readings and the deploy, alarm and rollback events, and organize the human review, in both themes. Fictitious data.

    Prumo domain components sheet, showing only what is in the dashboard, with fictitious data: the 5 indicators (prompt in production v2 with the Below target status, the judge says 90.8%, estimated real rate 77.2% with a 70.0% to 84.4% band, error budget burn rate 1.5× and 521 human labels); the 4 SLO statuses, Collecting labels, Healthy, Under watch and Below target, and the indicators awaiting labels; the chart of the last 48 readings with the judge, real rate, band and simulator ground-truth lines; the list of deploy, alarm and rollback events; the simulator controls; the human review card with the answer key after the click; the lead classification form, the admin token field and the error notice; and, on a dark panel, the same components with the dark theme tokens.
    Prumo domain components sheet, showing only what is in the dashboard, with fictitious data: the 5 indicators (prompt in production v2 with the Below target status, the judge says 90.8%, estimated real rate 77.2% with a 70.0% to 84.4% band, error budget burn rate 1.5× and 521 human labels); the 4 SLO statuses, Collecting labels, Healthy, Under watch and Below target, and the indicators awaiting labels; the chart of the last 48 readings with the judge, real rate, band and simulator ground-truth lines; the list of deploy, alarm and rollback events; the simulator controls; the human review card with the answer key after the click; the lead classification form, the admin token field and the error notice; and, on a dark panel, the same components with the dark theme tokens.

  7. How the math works

    In a 20-decision example, the judge approves 90%, the human sample shows 75% and a band below target triggers the rollback. The confidence sequence stays valid even when checked every hour.

    Teaching sheet on how the math works. In a 20-decision example, the LLM judge approves 18, or 90%; a person checks against the answer key and only 15 are right, or 75%: 15 right and approved, 3 wrong ones the judge approved, its blind spot, and 2 the judge caught. Step by step: the judge is optimistic by 15 percentage points, the real accuracy is about 75% with a confidence band and, below the 85% target for 2 consecutive readings, the prompt rolls back. Below, the band against the target in the dashboard's three statuses, Healthy, Under watch and Below target, and an illustrative chart of the regular interval (ppi_ci), valid for one look and alarming by chance, against the confidence sequence (ppi_cs), wider and valid at every look, with 92% to 98% coverage for the 95% interval over 400 simulations and an error of at most α + 2% when checked every hour for 30 hours.
    Teaching sheet on how the math works. In a 20-decision example, the LLM judge approves 18, or 90%; a person checks against the answer key and only 15 are right, or 75%: 15 right and approved, 3 wrong ones the judge approved, its blind spot, and 2 the judge caught. Step by step: the judge is optimistic by 15 percentage points, the real accuracy is about 75% with a confidence band and, below the 85% target for 2 consecutive readings, the prompt rolls back. Below, the band against the target in the dashboard's three statuses, Healthy, Under watch and Below target, and an illustrative chart of the regular interval (ppi_ci), valid for one look and alarming by chance, against the confidence sequence (ppi_cs), wider and valid at every look, with 92% to 98% coverage for the 95% interval over 400 simulations and an error of at most α + 2% when checked every hour for 30 hours.

  8. AWS architecture

    The board follows the lead from 1 to 12: API Gateway, Lambda with FastAPI, Bedrock with the judge, DynamoDB, human review in Step Functions, the hourly estimator, the CloudWatch alarm and the prompt rollback on AppConfig by compare-and-swap, all in Python CDK.

    Prumo architecture board in the style of a Miro board, on AWS, from the lead to the prompt rollback: an AWS CDK (Python) stack with 4 Lambdas, inside the AWS Cloud and Region us-east-1 groups, with no VPC and public endpoints protected by IAM and TLS. Outside AWS sit the API client, the human reviewer, the operator and the web dashboard with OpenAPI. The lead's path, numbered 1 to 12, enters through API Gateway (an HTTP API with request limits) to the API Lambda running FastAPI through Mangum, which reads the active prompt version from AppConfig, classifies the lead with Nova Lite on Bedrock as HOT, WARM or COLD, has the LLM judge, Nova Pro, check 100% of the decisions and writes the decision and the sums in one DynamoDB transaction; a random 5% of decisions open a human review in Step Functions with waitForTaskToken and a 72-hour deadline, and the reviewer labels through the API. Every hour, EventBridge Scheduler triggers the estimator Lambda, which reads the sums, computes the confidence sequence and publishes the EMF metric to CloudWatch; the SloBreached alarm fires on 2 out of 2 readings, an EventBridge rule triggers the rollback Lambda and a compare-and-swap on AppConfig rolls back one version, with an SQS DLQ and an SNS alert on failure, while the estimator applies the same rule every hour as a safety net. Delivery runs from A to D: a push to the public repository, CI with 2 jobs, a manual OIDC deploy with no stored key and CloudFormation generated by CDK. The board is rounded out by the security strip (Secrets Manager and least-privilege IAM), the observability strip (SNS, Budgets, X-Ray and JSON logs), sticky notes with decisions and trade-offs and a legend with the lead's 12 steps.
    Prumo architecture board in the style of a Miro board, on AWS, from the lead to the prompt rollback: an AWS CDK (Python) stack with 4 Lambdas, inside the AWS Cloud and Region us-east-1 groups, with no VPC and public endpoints protected by IAM and TLS. Outside AWS sit the API client, the human reviewer, the operator and the web dashboard with OpenAPI. The lead's path, numbered 1 to 12, enters through API Gateway (an HTTP API with request limits) to the API Lambda running FastAPI through Mangum, which reads the active prompt version from AppConfig, classifies the lead with Nova Lite on Bedrock as HOT, WARM or COLD, has the LLM judge, Nova Pro, check 100% of the decisions and writes the decision and the sums in one DynamoDB transaction; a random 5% of decisions open a human review in Step Functions with waitForTaskToken and a 72-hour deadline, and the reviewer labels through the API. Every hour, EventBridge Scheduler triggers the estimator Lambda, which reads the sums, computes the confidence sequence and publishes the EMF metric to CloudWatch; the SloBreached alarm fires on 2 out of 2 readings, an EventBridge rule triggers the rollback Lambda and a compare-and-swap on AppConfig rolls back one version, with an SQS DLQ and an SNS alert on failure, while the estimator applies the same rule every hour as a safety net. Delivery runs from A to D: a push to the public repository, CI with 2 jobs, a manual OIDC deploy with no stored key and CloudFormation generated by CDK. The board is rounded out by the security strip (Secrets Manager and least-privilege IAM), the observability strip (SNS, Budgets, X-Ray and JSON logs), sticky notes with decisions and trade-offs and a legend with the lead's 12 steps.

  9. Documented API

    The FastAPI service documents with OpenAPI the routes for quality, prompts, decisions and human review, including prompt deploy and rollback.

    Prumo OpenAPI documentation generated by FastAPI at /docs: version 0.1.0, OAS 3.1, the summary A taxa real de acerto de uma IA em produção (the real accuracy of an AI in production) and the routes grouped by tag: sistema (GET /health), qualidade (GET /api/v1/quality), prompts (GET /api/v1/prompts, POST /api/v1/prompts/{version_id}/deploy and POST /api/v1/prompts/rollback), decisões (POST /api/v1/decisions) and revisão (GET /api/v1/review/queue and POST /api/v1/review/{decision_id}).
    Prumo OpenAPI documentation generated by FastAPI at /docs: version 0.1.0, OAS 3.1, the summary A taxa real de acerto de uma IA em produção (the real accuracy of an AI in production) and the routes grouped by tag: sistema (GET /health), qualidade (GET /api/v1/quality), prompts (GET /api/v1/prompts, POST /api/v1/prompts/{version_id}/deploy and POST /api/v1/prompts/rollback), decisões (POST /api/v1/decisions) and revisão (GET /api/v1/review/queue and POST /api/v1/review/{decision_id}).

  10. Results

    The benchmark runs 10 simulations per row, and the real AWS test used 30 synthetic leads: the AI was right 60% of the time, and the judge approved 70%.

    Results sheet, with synthetic data. On the left, the prumo bench benchmark, with 10 simulations per row and 400 decisions per hour: a chart shows v2 dropping the AI's real accuracy from 92% to 76% while the judge keeps saying about 91%, and v3 rising to 93%, against the 85% target; the table shows, for ppi_cs, the default, with 10, 20 and 40 labels per hour, v2 detected 10 out of 10 times, rollback in 28, 17 and 11 hours (median) and 0 out of 10 wrongful rollbacks of v3; for ppi_ci, 14, 5 and 4 hours, also with 0 out of 10; and two notes: watching only the judge never alarms, and the default is the slowest because the confidence sequence holds at every hourly look. On the right, the real AWS test with 30 synthetic leads with an answer key: 60% real accuracy for Nova Lite against 70% approved by the Nova Pro judge, on a grid with 17 right and approved, 4 wrong and approved, 1 right but rejected and 8 wrong ones the judge caught, plus 0 HTTP errors, about 2.6 s per lead and a deploy in about 3.5 min. Below, the tests and quality panel: green CI on GitHub Actions, tests for the Python app and the CDK infrastructure, app coverage, end-to-end tests, Monte Carlo, Stubber and moto, strict mypy, ruff, Hypothesis and cdk synth.
    Results sheet, with synthetic data. On the left, the prumo bench benchmark, with 10 simulations per row and 400 decisions per hour: a chart shows v2 dropping the AI's real accuracy from 92% to 76% while the judge keeps saying about 91%, and v3 rising to 93%, against the 85% target; the table shows, for ppi_cs, the default, with 10, 20 and 40 labels per hour, v2 detected 10 out of 10 times, rollback in 28, 17 and 11 hours (median) and 0 out of 10 wrongful rollbacks of v3; for ppi_ci, 14, 5 and 4 hours, also with 0 out of 10; and two notes: watching only the judge never alarms, and the default is the slowest because the confidence sequence holds at every hourly look. On the right, the real AWS test with 30 synthetic leads with an answer key: 60% real accuracy for Nova Lite against 70% approved by the Nova Pro judge, on a grid with 17 right and approved, 4 wrong and approved, 1 right but rejected and 8 wrong ones the judge caught, plus 0 HTTP errors, about 2.6 s per lead and a deploy in about 3.5 min. Below, the tests and quality panel: green CI on GitHub Actions, tests for the Python app and the CDK infrastructure, app coverage, end-to-end tests, Monte Carlo, Stubber and moto, strict mypy, ruff, Hypothesis and cdk synth.

Features and the pains they solve

Every feature was born from a real operational problem.

  1. 01

    Real accuracy with a confidence band

    Pain
    The LLM judge checks 100% of the decisions, but it always errs in the same direction: its score is not the real accuracy, and reviewing everything by hand does not scale.
    Solution
    A random sample labeled by people measures the judge's error, and Prediction-Powered Inference (PPI and PPI++) removes that error from the judge's score, reporting the real accuracy with a confidence band. The estimate only uses sums and sums of squares, so reading the quality costs O(1).

    Impact: The 95% interval contains the truth 92% to 98% of the time (400 Monte Carlo simulations)

  2. 02

    Confidence sequence

    Pain
    A regular confidence interval is valid for a single look; read every hour, the chance of a false alarm grows with each reading.
    Solution
    The default method (ppi_cs) uses an asymptotic confidence sequence (Waudby-Smith et al., 2024), which stays valid even when checked forever, and requires at least 30 labels before any decision. It detects more slowly than the regular interval, in exchange for a real guarantee against false alarms.

    Impact: Checked every hour for 30 hours, it errs at most α + 2%

  3. 03

    Automatic prompt rollback

    Pain
    A prompt version that makes the AI worse raises no exception and no alert: without measurement, the regression stays live until someone notices.
    Solution
    If the whole band falls below target (85% SLO) for 2 consecutive readings, the CloudWatch alarm triggers, through EventBridge, a Lambda that rolls the prompt back to the previous version in AppConfig, with no new code deploy.

    Impact: Rollback in 28, 17 and 11 h (median) with 10, 20 and 40 labels per hour; v2 detected 10/10

  4. 04

    Two triggers, one atomic swap

    Pain
    EventBridge only fires on the transition to ALARM: if that attempt fails, the alarm sits still without a new event, and a redelivered alarm could roll back two versions.
    Solution
    The hourly estimator applies the same rule as a safety net, and the swap is compare-and-swap on the measured version: a redelivered alarm, a simultaneous manual click or both triggers together never roll back two versions. A failed event goes to an SQS dead-letter queue, with an alarm.

    Impact: 0/10 wrongful rollbacks of v3 across every benchmark setting

  5. 05

    Human review on Step Functions

    Pain
    If only the doubtful cases are reviewed, the correction of the judge's bias becomes biased itself, and a label written twice would count double in the estimate.
    Solution
    Sampled decisions enter a Step Functions state machine that waits for the reviewer with waitForTaskToken, for up to 72 h. The API rejects labels on decisions that were not sampled, and the DynamoDB write is conditional, so the same decision can never be labeled twice.

  6. 06

    Classification with Bedrock

    Pain
    Asking the model for JSON and hoping for the right shape breaks the flow on the first off-format answer, and the lead's text may carry injected instructions.
    Solution
    The classifier and the judge call Bedrock through the Converse API with a forced tool (tool use) and temperature 0, and the output is still validated with Pydantic. The prompt tells the model to ignore instructions inside the lead, and Bedrock Guardrails can be turned on through configuration.

    Impact: Real test with 30 synthetic leads: 0 HTTP errors and about 2.6 s per lead

  7. 07

    Dashboard and simulator

    Pain
    A silent quality drop is hard to explain without seeing the judge and the real accuracy side by side, and testing the idea straight on AWS costs money.
    Solution
    The dashboard shows the judge's score, the estimated real accuracy with its band, the error-budget burn rate and the events. In simulated mode, with no AWS and no cost, you can publish v2, change the labels per hour, turn the rollback on or off and label decisions yourself.

    Impact: The same logic runs in the simulator and on AWS, through ports and adapters

  8. 08

    Method benchmark

    Pain
    Without comparing methods, there is no way to know how many labels per hour are enough or how many hours each statistical guarantee costs.
    Solution
    The prumo bench command runs 10 simulations per row: v1 live, v2 dropping the AI from 92% to 76% and v3 improving it to 93%, measuring detection, hours to rollback and wrongful rollbacks for ppi_cs and ppi_ci with 10, 20 and 40 labels per hour.

    Impact: ppi_ci rolls back in 14, 5 and 4 h; a judge-only monitor never alarms

  9. 09

    Infrastructure as code

    Pain
    Setting up Lambda, Step Functions, alarms and AppConfig by hand in the console is not reproducible, and an access key stored in CI is a leak waiting to happen.
    Solution
    A single AWS CDK (Python) stack creates every application resource, with tests that lock the template's invariants. GitHub Actions reruns lint, type checks and tests before assuming, through OIDC, a role bound to the production environment, and deploys only run when triggered manually.

    Impact: Stack deploy in about 3.5 min

  10. 10

    Personal data and secrets

    Pain
    The lead's text is personal data under Brazil's LGPD, and a token in an environment variable is visible to anyone who can read the Lambda configuration.
    Solution
    The decision text expires after 180 days through DynamoDB TTL, while the estimate's sums remain. The admin token is generated by Secrets Manager and read at cold start; without it, the protected routes refuse the request.

    Impact: No customer data: every lead is synthetic