← All work
Case study · 2026

Perception

A multi-tenant brand-perception platform on AWS: reviews, social posts and surveys turned into themes, sentiment and Genie, a tool-calling AI analyst, for a market-research consultancy.

Next.jsGoPythonPostgreSQLpgvectorAWSBedrockDockerLive site ↗
perceptionq.converseglobal.com
Perception — screenshot of the live site

Overview

Perception is a Voice-of-Customer dashboard for a market-research consultancy. It takes what people say about brands (reviews, social posts, creator content, survey answers), sorts it into themes and sentiment, and lets analysts compare brands and export the results straight into client decks. Genie, a chat assistant, answers questions about the data in plain language by calling the same queries the charts use.

It runs as one installation that serves several client workspaces: brand-health studies, a quantitative survey, an influencer roster. Each user sees only the workspaces an admin has granted them, and every row in the database is tagged with the workspace it belongs to.

This is client work under NDA, so this write-up covers the architecture and infrastructure. It leaves out clients, data sources and how the analysis itself works.

My role

A four-person team built it, and I'm the largest contributor by commits (about 60% of the history). Roughly, I owned:

  • Dashboard features: most of the chart tabs analysts use every day (theme analysis, category drivers, prioritisation matrix, trends, word cloud), with filters and period controls that follow the active workspace.
  • Quant survey workspace: funnels, brand imagery as z-scores, heat maps and a biplot, built to match the client's own workbook.
  • A self-serve survey data-file builder: viewers define their own breaks from any question and download counts and column percentages as one styled spreadsheet.
  • Native exports: every tab exports to PowerPoint as real, editable charts, tables and shapes, not screenshots, and to Excel with the full detail behind each chart.
  • CI/CD: GitHub Actions to production through OIDC and SSM, with migrations and health checks on every deploy.
  • Hardening: a security and correctness pass on the Go API, an authenticated ingest endpoint, every test running in CI, and the database moved off the public internet onto private RDS.
  • Cost: an infrastructure cost audit that cut the monthly AWS bill.
  • Docs: a runbook and a CLAUDE.md good enough that a new engineer (or agent) can add a client workspace without asking anyone.

Architecture

Everything runs on AWS. One EC2 host runs the app in Docker Compose behind Caddy, which routes by path so the browser only ever talks to one origin and the session cookie stays first-party. The same host runs the scheduled collection jobs as systemd timers.

flowchart LR
  subgraph client["Browser"]
    ui["Dashboard - Next.js 14"]
  end
  subgraph ec2["EC2 host"]
    caddy["Caddy - TLS and path routing"]
    subgraph compose["Docker Compose"]
      api["Query API - Go Fiber"]
      etl["Data service + Genie - FastAPI"]
      web["Next.js server"]
    end
    jobs["systemd timers - collectors, scoring, rollup refresh"]
  end
  subgraph aws["AWS managed"]
    rds[("RDS PostgreSQL 16 + pgvector - private")]
    bedrock["Bedrock - Claude, Titan embeddings"]
    lambda["Lambda fetchers - 17 regions"]
  end
  ui -->|"HTTPS, one origin"| caddy
  caddy -->|"/api/chat, /api/ingest, /health"| etl
  caddy -->|"/api"| api
  caddy -->|"everything else"| web
  api --> rds
  etl --> rds
  etl -->|"tool-calling agent"| bedrock
  jobs -->|"fetch pages"| lambda
  jobs -->|"classify, embed"| bedrock
  jobs -->|"write results, refresh rollups"| rds
  • Caddy terminates TLS and splits /api traffic by specificity: chat, ingest and health go to the Python service, the rest of /api to Go, and everything else to Next.js. Both backends already serve under /api, so the order of the rules matters.
  • The Go query service handles auth (HTTP-only signed JWT cookies, OTP invites and resets), checks the workspace, then runs the aggregations behind every chart. It keeps serving if the database is down at boot and retries its bootstrap in the background, so a slow RDS never takes the frontend with it.
  • The Python data service takes spreadsheet uploads and runs Genie, an agent on Bedrock with semantic search over pgvector embeddings.
  • Scheduled jobs are systemd timers on the same host. They fetch fresh posts and reviews through a fleet of small Lambda functions in 17 regions, ask Bedrock to classify them, write to RDS and refresh the rollup tables. Some sources vanish after 24 hours, so a missed day can't be backfilled; that's why these are timers with logs, not something someone remembers to run.
  • RDS PostgreSQL 16 holds every workspace. It has no public endpoint; engineers reach it through an SSM port-forwarding tunnel, and every standalone script refuses to touch it without an explicit --target=prod.

System design

Two request paths matter most: a chart load, and a question to Genie. In both, the server enforces workspace isolation from an X-Workspace-Id header checked against the user's grants; nothing from the browser is trusted for it.

sequenceDiagram
  participant U as Analyst
  participant C as Caddy
  participant Q as Query API
  participant E as Genie
  participant DB as RDS Postgres
  participant B as Bedrock
  U->>C: Open a tab with filters
  C->>Q: GET /api chart data with session cookie
  Q->>Q: Verify JWT and workspace grant
  Q->>DB: Aggregate rows for this workspace only
  DB-->>Q: Results
  Q-->>U: Chart series
  U->>C: Ask Genie a question
  C->>E: POST /api/chat
  E->>E: Verify session, load workspace profile and tools
  loop Phase 1 - up to 6 tool rounds
    E->>B: Converse with tool definitions
    B-->>E: Tool calls
    E->>DB: Run tools - KPIs, themes, rollups, pgvector search
    DB-->>E: Rows scoped to workspace
  end
  E->>B: Phase 2 - converse_stream, no tools
  B-->>U: Streamed answer with sources

Genie

Genie is a two-phase agent. Phase one is a non-streaming tool loop: the model picks from about twenty tools (KPI summaries, theme breakdowns, brand comparisons, survey crosstabs, semantic search) for up to six rounds. Phase two is one streaming call with no tools, so the model can only write prose and the slowest part, the synthesis, streams token by token.

Each workspace gets its tools from three layers:

  • A data profile decides which tools exist at all. An influencer roster and a brand-health study have different tables, so they get different tools.
  • A config row holds the brand, its competitors and the starter questions.
  • Live grounding is computed from the data. If a table has no rows for a workspace, its tools are withdrawn automatically.

It fails closed: a missing profile, a database error or an unknown value resolves to no tools, and Genie says it can't answer yet rather than answering confidently from the wrong schema. A test in CI catches any active workspace that would silently lose Genie. The heavier features (a daily rollup table and semantic search) shipped behind feature flags, each turned on only after a parity check against the raw queries.

Deploys

Every push to main builds all three services and runs the tests, and only a green build reaches the server.

flowchart LR
  push["Push to main"] --> build["GitHub Actions: go vet and build, next build, Python import check, pytest per file"]
  build -->|"passed"| oidc["Assume AWS role via OIDC - no stored keys"]
  oidc --> ssm["SSM send-command to the tagged EC2 host"]
  ssm --> deploy["checkout sha, docker compose build"]
  deploy --> migrate["Apply schema migrations"]
  migrate --> up["docker compose up -d"]
  up --> health["Health-check each service, confirm the running sha"]

The deploy role can do exactly four things: describe instances, send one SSM document to the instance with the right tag, and read the result. There's no SSH key in GitHub and no port 22 open to the internet. The health endpoint reports the running git sha, so a deploy is confirmed by what's actually serving, not just an exit code.

Decisions & trade-offs

  • One EC2 host with Docker Compose, not ECS. For a handful of workspaces it's cheaper, simpler to debug and quick to redeploy. The cost is a single point of failure; the runbook already covers moving to ECS Fargate behind an ALB when uptime matters more than cost.
  • One database, every row tagged by workspace. A new client is configuration, not new infrastructure. Every query must filter by workspace, so access checks live in one place in each service, and both services resolve denied access the same way (written up as an ADR).
  • Go for reads, Python for data and AI work. Chart queries need to be fast and predictable; ingestion and model calls are easier in Python. Both share the database, so there's no sync layer.
  • Genie calls tools instead of writing SQL. Slower to extend, since every question type needs a tool, but every number it quotes comes from a query the charts already trust, and nothing it generates can reach another workspace's rows.
  • OIDC and SSM instead of keys and SSH. Nothing long-lived to leak, and nothing listening for the internet to probe.
  • Native editable exports. Screenshots would have been quicker to build, but analysts restyle every chart for their decks, so real charts are what they need.

What I'd do next

  • Embed new opinions incrementally after each ingestion batch, instead of with a backfill script.
  • Extend the response cache that the influencer tabs already have to every workspace, since most tabs are read far more often than the data changes.
  • Write the infrastructure down as code; today the runbook is the source of truth.