Verita AI

A vision-model audit pipeline

The product question is deceptively simple: when a remote contractor bills for eight hours, how do you know what those hours were? The answer had to hold up commercially, since it determines what people get paid, which means "the model thinks it looks fine" is not a sufficient standard.

I built the pipeline that reads session frames from screen recordings with a vision model, evaluates each against the task policy for that specific project, and buckets the time: genuine work, AI-assisted work, off-task activity, and frames the model could not read. The result is a sealed ledger: priced, and recomputable from the underlying rows, so a disputed figure can be traced rather than argued about.

An illustration of the read pass: every session frame is classified before a single hour is billed. The proportions shown are illustrative, not production figures.

Choosing the read model was an evaluation problem, not a preference. I benchmarked candidates across providers on a golden set built from human-labelled sessions, comparing accuracy and cost per session before committing. The pipeline reached 80% precision and recall on that set, and has sealed 264 sessions in production.

An LLM data-annotation platform

A platform where distributed annotators produce training data, and where the client's view, the reviewer's view, and the operator's view must never leak into each other. I built the core: a five-role permission model with an explicit capability matrix, a workflow state machine with audited transitions and calibration routing, screening and qualification gates that annotators pass before they can claim work, and an append-only audit log.

Two decisions here I would defend anywhere. First, the permission model is tested by a table-driven matrix over every endpoint and every persona, with a completeness guard that fails if a new endpoint is added without a corresponding assertion, because permission bugs are found by the endpoint nobody remembered to test. Second, audit coverage is derived from the URL configuration rather than maintained by hand, so the check verifies all 54 endpoints are actually covered instead of trusting a list. Over the work, the test suite grew from 210 to 1,239.

I also spent real time on the ingestion path: fetching remote images safely, filtering private address ranges to prevent server-side request forgery, re-hosting through presigned uploads, and moving enqueueing off the request cycle in batches after thousand-image imports began timing out.

A rubric verifier, and a testing standard

For an AI interviewing product, I delivered the second generation of the verifier that checks whether an interview actually covered what the rubric required: LLM-judge claim validation, shadow-verifier gating so a new version could run against production traffic without affecting outcomes, and hard execution limits bound to the container timeout so a slow evaluation fails predictably instead of hanging.

The lasting contribution was a standard rather than code. A test only counts once a deliberate defect introduced into the source proves the test can fail, and the defect itself has to be verified as landed, because a mutation that silently did not apply produces a test that looks proven and is not. Tests that pass against broken code are worse than no tests: they transfer confidence without earning it.

SOC 2 Type II, during a live audit window

I led readiness while the observation period was already running, which removes the option of fixing things quietly. Control coverage moved from 154 to 165 of 173.

• A database audit-logging and storage-encryption migration across 11 production PostgreSQL databases, with the infrastructure persisted in Terraform across three repositories rather than applied by hand.
• Threat detection, audit trails, and file-integrity monitoring deployed and wired to real alerting, including finding that the file-integrity rules had been installed in a form that never actually fired.
• Over forty evidence artifacts produced for the auditor, plus remediation of findings from authenticated penetration tests.
• A backup and access review that surfaced unprotected buckets and failed backup jobs that had been silently failing on missing role permissions.

The recurring lesson: most compliance failures are not missing controls, they are controls that were configured once and never verified as working. The interesting work is proving the alarm fires, not installing the alarm.

Security tooling

I built a multi-scanner pipeline that runs several static-analysis engines over a codebase and uses a reasoning model to triage the output: deduplicating across engines, filtering false positives, and ranking by exploitability. On a production codebase it reduced 345 raw findings to 33 verified issues, nine critical.

The generalized version is open source, with no findings or customer data in it. Read the case study →