Sycamore: Exploring Grounded Synthetic Evaluation for Visualization Systems

Abstract

Evaluating visualization systems in specialized domains, such as genomics, is challenging due to the scarcity of domain experts and difficulty recruiting relevant user types. LLM-based synthetic personas provide an appealing workaround, but they face skepticism as their claims often lack verifiable grounding. We present Sycamore, an exploratory approach to evaluating visualization systems with grounded synthetic personas whose responses are restricted to interview evidence. Sycamore enforces provenance with participant citations for each claim and an abstention mechanism when retrieval yields no relevant evidence. A protocol runner guides the synthetic personas through an evaluation of the live visualization system, producing multi-turn transcripts for qualitative coding. We instantiate the approach on Geranium, a multimodal genomics visualization search engine, with seven evaluators grounded in prior interviews on genomics visualization authoring. We report themes in the resulting transcripts, highlight an emergent requirement absent from the source data, and provide a preliminary comparison to a prior human user study. Recognizing that our findings derive from an exploratory perspective and that synthetic proxies cannot replace human evaluation, we discuss their implications for integrating synthetic evaluators alongside expert user studies. The protocol, codebook, extended results, transcripts, and demo video are in the Supplementary Materials and at osf.io/kdfr3; the source code and project page are at huyen-nguyen.github.io/sycamore.