Handshake AI
Evaluation work on frontier models, the human side of the pipeline that my later engineering work automates. Scoring model outputs against calibrated rubrics is where you learn how slippery "better" is as a judgment, and why an evaluator without a rubric is just a preference.
- Role
- AI data & evaluation
- Focus
- Image-generation quality · ML code review
- Feeds
- RLHF preference data
- Recognition
- Selected as a Star Fellow
Rubric-based model evaluation
I evaluated frontier image-generation models against calibrated quality rubrics, producing the preference data that RLHF pipelines train on. The axes that mattered were the ones that resist a quick glance: rendering artifacts that only appear at certain scales, and anatomical inconsistencies that a first impression reads straight past.
Calibration is the whole game. Two evaluators applying "quality" independently produce two different datasets; two evaluators applying the same rubric produce one. That experience is the direct ancestor of how I now build automated evaluation: golden sets, explicit criteria, and measuring the evaluator's own bias rather than assuming it away.
ML code review
I also assessed machine-learning codebases, giving structured feedback on algorithm selection, implementation quality, and adherence to sound practice, the difference between a model that scores well and a modelling approach that is actually defensible.
Star Fellow
Selected as a Star Fellow for a program that evaluated model understanding against frontier research publications, probing systematically for the boundaries of model reasoning, where it stops generalizing and starts pattern-matching.