Who's That Player?: Externalizing Query Interpretation in Spoken XR Sports Interaction

Authors

Chunggi Lee (Harvard University), Tica Lin (Dolby Laboratories), Yalong Yang (Georgia Institute of Technology), Hanspeter Pfister (Harvard University)

Presentation

Session

Data really is everywhere

Time

Tuesday, Nov 10, 16:00 – 16:12 (US/Eastern) · session 15:00 – 16:30

Location

Hall Essex north

Keywords

Extended reality (XR), sports analytics, speech interaction, situated visualization

Abstract

XR sports viewing enables spectators to follow play from immersive, spatially anchored perspectives while accessing contextual analytics directly within the scene. In such settings, speech offers a practical interaction modality because text entry and menu navigation can interrupt attention during fast-paced gameplay. However, spoken queries are often underspecified: viewers may omit which player, time period, field location, or metric they intend. When systems resolve these ambiguities implicitly, their assumptions remain hidden, making misinterpretations difficult to notice and correct (repair). We investigate how externalizing a system's interpretation of spoken queries can support inspection and correction of such misunderstandings in XR sports viewing. Through a formative study, we identified four recurring ambiguity types (referential, spatial, temporal, and metric) that characterize ambiguous spoken queries in this context. We develop a design space that organizes externalization along three dimensions (ambiguity type, interpretation state, externalization strategy) and instantiate it in an interactive XR soccer viewing system that combines situated visual cues with supporting analytic views. A within-subjects user study (N=16) comparing externalized interpretation against a voice-only baseline reveals that externalization is associated with higher inspectability on most measured dimensions and increased explicit repair language overall. However, repair occurred in only 38% of misaligned externalization trials, and this visibility-action gap varied by ambiguity type, indicating that transparency and correction affordance are orthogonal design axes.

For Practitioners

XR sports viewing developers, sports analytics platform designers, and designers of speech and LLM based interfaces may benefit from this work. They can use our ambiguity taxonomy and externalization design space to create future sports viewing experiences in which spectators can inspect how their spoken queries are interpreted and easily correct misunderstandings.