Physician-benchmarked multimodal clinical cases
Controlled simulated cases in which the clinically important information may be spoken, written, heard, seen or held in the clinical context, each paired with assessments from board-certified physicians.
The cases
Every case is a staged simulation with standardized patients, recorded in advance. No real patients, patient care or patient records are involved. Because the recording is fixed, different models and physician reviewers can examine exactly the same material, and the same case can be supplied as a transcript, as audio, as video, or as all of them with clinical context, so that a difference in output can be traced to a difference in input.
Cases range from a few seconds with no dialogue, for a first impression or a single sign, to several minutes for a telehealth-style or evolving encounter. The model reviews what was recorded. It does not conduct a live interview, which is a different research question.
Collections
Eight areas of development and custom production, organized in three research families.
Clinical Perception & Examination
Short, fixed recordings of a single examination or observation. They test whether a model recognizes a specific finding, which side is affected and how much certainty the recording allows, before asking what significance it assigns.
-
Neurologic Findings & GCS
Pronator drift, limb weakness, facial asymmetry, gaze, coordination, Romberg testing and abnormal gait, along with eye, verbal and motor responses scored as separate Glasgow Coma Scale components.
A total GCS score is used only when the whole examination it requires is shown. Unequal pupils are included only where the camera can actually resolve them.
-
Physical Examination
Guarding or a visible response to palpation, McBurney’s point and Rovsing’s sign, reflexes and clonus where they are credibly visible, breathing effort, posture and general appearance.
Some responses are portrayed rather than observed: tenderness, a tingling sensation after Tinel’s test, or rigidity. The recording shows the maneuver and what the standardized patient does or says. Cases mark these as simulated responses, separate from findings a camera can record directly.
-
Speech & Behavior
Dysarthria, disorientation, aphasia where it can be portrayed convincingly, strained or broken speech, hesitation, and a companion who answers for the patient.
A glance or a nervous expression does not on its own establish a diagnosis or a safety outcome, and the reference assessments do not treat it as if it did.
Multimodal Encounters & Discrepancies
Fixed audiovisual scenes in which what is seen or heard changes the meaning of what is said or written. They test how information beyond the transcript changes a model’s interpretation.
-
Telehealth Encounters
Longer prerecorded interactions between a clinician and a standardized patient, or a patient speaking to a camera, with spoken history, appearance, movement and clinically relevant sound.
The model reviews the recording. It does not conduct the interview, which is a different research question.
-
Cross-Modal Discrepancies
Reassuring words alongside labored breathing, a monitor reading that conflicts with the written vital signs, or a handoff that contradicts the scene.
Matched versions of one case, presented as text, audio, video or all three, can isolate which modality carried the decisive observation. Sometimes the right response is to flag the discrepancy and ask for verification.
Acute Risk, Triage & Disposition
Cases drawn mainly from emergency medicine, where the safety question is not only what the diagnosis might be but how urgent the situation is and what should happen next.
-
Rapid Visual Triage
Fixed scenes of roughly 5 to 10 seconds: a stretcher arriving with EMS, a patient at the bedside, a patient walking into the room.
Tasks ask about observable concerning features, the need for rapid attention, and whether there is enough information to say. Eventual disposition is not treated as knowable from a few seconds of video.
-
Bedside Assessment & Disposition
A patient says they are ready to leave, but their gait, weakness, speech or surroundings may justify reassessment, assistance or observation.
A safe disposition often depends on more than an isolated clip, and these cases are built with that in mind.
-
Time-Critical Emergencies
Stroke-like findings, altered cognition, concerning respiratory appearance and signs of occult deterioration, where an observed cue should change urgency, workup or escalation.
A single visual sign is treated as a reason for concern, not as proof of one diagnosis.
What a case package can include
The unit of value is a clinically specified, annotated case, or a family of related cases, rather than a video file on its own. Not every case carries every element.
- A controlled audiovisual recording
- Separate text, audio and video versions, where scientifically meaningful
- A timestamped transcript
- A timestamped map of when each finding appears
- Matched controls and scenario variations
- Physician reference assessments
- Recorded uncertainty, disagreement and adjudication
- Scoring criteria written for the case
- Optional time-synchronized physician commentary
- Provenance, intended use and licensing terms
Physician reference assessments
Board-certified physicians review each case independently and record what they observed, how they judged risk, what they would do next, and how certain they were, with timestamps for when the evidence became available. A model’s output can then be compared finding by finding.
These assessments are references, not ground truth. Where reviewers differ, the disagreement is recorded and, where appropriate, adjudicated. Optional commentary, recorded while a physician reviews the case aloud, captures the observations and reasoning they chose to put into words. It is not a complete record of how a decision was reached.
Training and held-out evaluation
A model should not be evaluated on material it was trained on. Cases licensed for training are kept apart from private evaluation sets that are not released, so that benchmark results are not inflated by contamination. A license to evaluate, a license to train, and exclusive use are separate arrangements, agreed separately.
Access and licensing
There is no public catalog. Collections are designed and produced by arrangement, and organizations can commission cases to a specification: setting, findings, variations, length, annotation and physician review. As rights-cleared collections are completed, we can describe what is available and how it is annotated.