The question behind the score
An answer can look convincing while relying on a flawed assumption. Evaluation also depends on the task itself: an ambiguous prompt, incomplete rubric, or incorrect reference can distort what a model’s result tells us.
My contribution: reasoning and evaluation
I conduct generative AI research for Meta on contract, evaluating frontier LLM reasoning across mathematics and other STEM domains. I review prompts, rubrics, and reference solutions, examining correctness, assumptions, proof validity, ambiguity, and alignment with the intended task.
The checks connect the reasoning to its conclusion and the evaluation criteria to the question being asked. A different valid approach needs to be distinguished from an actual error; a reference answer needs scrutiny too.
Turning findings into research direction
I partner with Meta researchers on vendor strategy: helping define research questions, structure evaluation workflows, and determine which findings should guide the next steps.
Within a four-person leadership group, I advise three external data vendors. My contribution is to translate research priorities and quality findings into recommendations for annotation workflows and vendor process improvements.
Active research: annotation diagnosis and repair
I lead early-stage research on self-healing annotated datasets: how to diagnose an annotation issue, constrain a proposed repair, and verify that the change preserves the task’s intent.
The work is at the stage of defining methods and evaluation criteria. A central question is what evidence should be required to accept a repair and when a case needs human review.
- Separate errors in a model’s reasoning from problems in the prompt, rubric, or reference solution.
- Determine what a repair may change without changing the problem being evaluated.
- Define checks for correctness, consistency, and the need for further review.
Earlier work at JPMorgan
I led the CIB Finance Model Risk team’s frontier LLM initiative, applying supervised fine-tuning and domain-specific evaluation to model-risk analysis and review workflows. Alongside that work, I built Python challenger models, tested assumptions and stability, and documented and escalated model risks.
Applied AI: earlier feedback on documentation
I developed guidance for an internal LLM-assisted workflow that automatically assessed technical-document readiness before model review. The workflow surfaced potential critical documentation gaps to model developers, helping them address issues earlier and reducing back-and-forth with reviewers.