Research

What changes when clinical systems can see and hear?

A curated reading of published research that motivates and informs Greyoak’s work: what audiovisual information carries, how consistently physicians read it, what a remote view leaves out, and how hard it is to find the decisive moment in clinical video.

The studies are by other research groups, not by Greyoak, and their authors do not endorse Greyoak. Each figure is our own drawing of values reported in the publication, with the source linked beside it.

Reading the figures

Colors and mark shapes are used the same way in every figure on this page, and every plotted value is also printed as a number. Intervals are drawn only where a paper reports them.

  • Combined or native audio-video input
  • Video or visual input
  • Audio input
  • Text, transcript or structured data
  • Physician judgment and agreement
  • Reported 95% confidence interval

Multimodal clinical perception

When the same clinical material reaches a model as text, audio or video, how much does its assessment change?

Agreement with physician-adjudicated exam scores, by input

  1. Native audio-video Three synchronized cameras [0.784, 0.875]
  2. Audio only
  3. Transcript only

Muted visual-only input: approximately 0.20, as described by the authors. Not plotted.

Across 21 rubric items, native video compared with transcript only

Mean item kappa

Video
0.667
Transcript
0.395

Macro-F1

Video
0.777
Transcript
0.592

Raw agreement

Video
92.3%
Transcript
79.8%
Quadratic weighted kappa for the best-performing model. The 95% interval is shown only for native audio-video, the one condition for which it is reported in the text. The abstract gives 0.830 (95% CI 0.773 to 0.880) for the native audio-video configuration. This figure follows the detailed results of version 1 throughout. Recreated by Greyoak Research from results reported by Kang et al., medRxiv (2026), research preprint. Greyoak did not conduct this study.

Research preprintmedRxiv2026

The same examination, different sensory access

Kang et al. Automated Assessment of OSCE Physical Exams using Multimodal AI. medRxiv, 2026. Not peer reviewed at the time of verification.

Researchers graded 300 recorded student physical examinations with a multimodal model, giving it the same encounters in different forms. With synchronized three-camera audio-video, agreement with physician-adjudicated scores reached a quadratic weighted kappa of 0.832, against 0.649 with audio alone and 0.587 with a transcript. Muted video performed worst, at roughly 0.20 by the authors’ account. What the model could perceive changed its grading even though the encounter did not.

Why it matters for evaluation. The closest published analogue to modality-ablation testing: the case is held fixed and only the model’s sensory access changes, scored against an adjudicated reference.

Limitation. A preprint from one institution. It grades students performing standardized examinations, not diagnosis or disposition in real patients, and spoken narration during the exams may help transcript-based grading.

Inputs

  • Text compared
  • Audio compared
  • Visual compared

The same recorded encounters were given to the model as audio-video, audio, transcript and muted video.

Read the original preprint doi:10.64898/2026.01.09.26343786

Text chat compared with real-time video, same research system

Scored by independent physician evaluators

Case-specific rubric, overall
75 → 83%

Rated by patient actors

Communicative effectiveness
79 → 89%
Ease and convenience
71 → 88%
Feeling understood
81 → 90%
Paired percentages for the text and video versions. The first row is scored by physician evaluators; the remaining rows are patient-actor ratings. Recreated by Greyoak Research from results reported by Nagda et al., arXiv (2026), research preprint. Greyoak did not conduct this study.

Research preprintarXiv2026

Moving a consultation from text to video changed what could be assessed

Nagda et al. Towards Expert-level Medical AI for Real-time Video Consultations. arXiv, 2026. Not peer reviewed at the time of verification.

A research consultation system was tested in simulated visits with professional patient actors, once through text chat and once through real-time video. The main evaluation covered 100 scenarios and 300 consultations, including a comparison arm of primary care physicians consulting by video. Independent primary care physicians scored the video version higher on the case-specific rubric, 83% against 75%, and actors rated it higher on communication, convenience and feeling understood. Targeted perception tests in the same paper showed persistent weaknesses with subtle cues, including nystagmus, tremor and affect.

Why it matters for evaluation. It separates two questions that evaluations often merge: whether richer input improves performance overall, and whether specific findings are actually perceived.

Limitation. Interactive consultations with actors and a research system, not deployment with real patients and not fixed prerecorded cases. Clinician-scored and actor-scored outcomes are different measures.

Inputs

  • Text compared
  • Audio compared
  • Visual compared

A text-chat version of the same research system was compared with a real-time audio-video version.

Read the original preprint

Visual clinical assessment and triage

What does a patient’s visible presentation carry, and how consistently do physicians read it?

Discrimination of eventual hospitalization, by input

  1. Triage data Routine triage features [0.668, 0.687]
  2. Video Sampled clips, no audio [0.689, 0.696]
  3. Video and triage Late fusion of both models [0.709, 0.719]

Video input, per patient

  1. About 5 minutes recorded
  2. Up to five 2-second clips sampled, about 10 seconds in all
  3. Audio not used by the video model

Schematic. Clip widths are to scale against a five-minute recording; their positions are illustrative.

Area under the ROC curve as reported, with 95% confidence intervals in brackets. The axis starts at 0.50, the value for no discrimination. The intervals are narrow, so the shaded range is repeated at a larger scale. Recreated by Greyoak Research from results reported by Ip et al., npj Digital Medicine (2024). Greyoak did not conduct this study.

Peer-reviewed studynpj Digital Medicine2024

Visual presentation carried information beyond routine triage data

Ip et al. Hospitalization prediction from the emergency department using computer vision AI with short patient video clips. npj Digital Medicine, 2024.

Emergency department patients were filmed for about five minutes, and a computer vision model used up to five two-second clips sampled from each recording, without audio. Predicting eventual hospitalization, video alone reached an AUROC of 0.693 and routine triage data 0.678; combining them reached 0.714. Visible presentation held signal that the triage variables did not, although overall discrimination remained moderate.

Why it matters for evaluation. A real-patient demonstration that what a patient looks like can carry information that structured data leaves out, which is the premise behind testing whether a system actually uses it.

Limitation. The outcome was eventual hospitalization, not diagnosis or immediate acuity, and the model sampled about ten seconds from much longer recordings. It is not a study of a ten-second clinical judgment.

Inputs

  • Text compared
  • Audio not used
  • Visual compared

Video compared with structured triage data. The video model used no audio.

Read the original paper doi:10.1038/s41746-024-01375-3

Speed, accuracy and agreement in video-based judgments

  • 10 to 11 s

    Average time to a judgment

    70% rated nearer “knew immediately” than “deliberated”

  • 55%

    Eventual disposition predicted correctly

  • 68%

    Agreement among physicians on disposition

  • 0.44

    Reliability of disposition between physicians

    Intraclass correlation

Four measures with different units, each on its own scale. They are not combined into a single score. Recreated by Greyoak Research from results reported by Sibbald et al., Medical Education (2017). Greyoak did not conduct this study.

Peer-reviewed studyMedical Education2017

Fast visual impressions were not reliably correct

Sibbald et al. Eyeballing: the use of visual appearance to diagnose ‘sick’. Medical Education, 2017.

Fourteen emergency physicians watched 25 patient videos and judged how sick each patient was and where they would go. Judgments took ten to eleven seconds on average, and physicians rated 70% of them as closer to knowing immediately than to deliberating. Yet eventual disposition was predicted correctly 55% of the time, and agreement among the physicians on disposition was 68%. Quick access to an impression did not make the conclusion dependable.

Why it matters for evaluation. Physician judgment is the reference standard for much of this work, and this study shows why it has to be collected from several physicians with their disagreement recorded, not taken as a single answer.

Limitation. Only 25 videos. Eventual disposition is not the same as immediate acuity, and it depends on test results and other information that no video shows.

Inputs

  • Text not used
  • Audio not reported
  • Visual present, not isolated

Physicians judged short patient videos. Channels were not compared.

Read the original paper doi:10.1111/medu.13396

Admission discrimination and triage-level agreement

Discrimination of eventual admission

  1. Computerized triage TTAS
  2. Physician video ratings

Agreement on triage level

  • 0.18

    Weighted kappa

  • 0.30

    Correlation

AUROC for eventual admission as reported, and agreement between physician ratings and triage levels, each on its own scale. Intervals are not drawn; the values are the point estimates given in the abstract. Recreated by Greyoak Research from results reported by Cheng et al., Academic Emergency Medicine (2022). Greyoak did not conduct this study.

Peer-reviewed studyAcademic Emergency Medicine2022

Similar discrimination, weak agreement on categories

Cheng et al. Physician gestalt for emergency department triage: A prospective videotaped study. Academic Emergency Medicine, 2022.

At a Taiwanese emergency department, five physicians assigned triage levels after watching videotaped triage encounters for 656 adults. Their ratings discriminated eventual hospital admission about as well as the computerized Taiwan Triage and Acuity Scale did, with AUROCs of 0.631 and 0.635. Yet the two often disagreed about which level a patient belonged in, with a weighted kappa of 0.18. Matching on an aggregate measure did not mean they classified the same patients the same way.

Why it matters for evaluation. Two assessors can look equivalent on discrimination while disagreeing case by case. Evaluations need to report categorical agreement and individual disagreements, not only a summary score.

Limitation. The recordings were of full triage encounters and may include spoken and contextual information, so this is not a controlled visual-only condition. One hospital.

Inputs

  • Text compared
  • Audio not reported
  • Visual present, not isolated

Physicians rated videotaped triage encounters, which may include speech, and were compared with a computerized triage system.

Read the original paper doi:10.1111/acem.14557

Limits of remote clinical examination

Which findings survive a video connection, and which need the patient in the room or an instrument?

Remote and in-person agreement on the need for imaging

  • 75%

    Agreed on whether imaging was needed

    42 of 56 patients, 95% CI 62% to 86%

  • 0.41

    Agreement beyond chance

    Cohen’s kappa, 95% CI 0.15 to 0.67

Each square is one patient

Remote and in-person physicians agreed (42) They disagreed (14)

Percent agreement and Cohen’s kappa, each with its reported 95% confidence interval, on separate scales. Below, each square is one of the 56 patients. Recreated by Greyoak Research from results reported by Hayden et al., JACEP Open (2023). Greyoak did not conduct this study.

Peer-reviewed studyJACEP Open2023

Remote abdominal assessment left meaningful disagreement

Hayden et al. Can tablet video-based telehealth assessment of the abdomen safely determine the need for abdominal imaging? A pilot study. JACEP Open, 2023.

Fifty-six adults with abdominal pain had their usual in-person assessment and an additional physician-directed history and abdominal examination over tablet video. Remote and in-person physicians agreed on whether imaging was needed in 42 of 56 patients, 75%, with a Cohen’s kappa of 0.41 once chance agreement is accounted for. No timely imaging was missed among the few patients who later needed a procedure.

Why it matters for evaluation. It measures a remote examination against an action, whether to image, and reports disagreement directly rather than asking only whether a sign can be named.

Limitation. A small pilot at one academic emergency department. The reassuring follow-up in seven procedure cases cannot establish a safety rate for rare misses.

Inputs

  • Text not used
  • Audio present, not isolated
  • Visual present, not isolated

A physician-directed history and examination over tablet video, compared with in-person assessment.

Read the original paper doi:10.1002/emp2.12963

Changes after in-person laryngoscopy

  1. Presumptive diagnosis
  2. Treatment plan
Share of presumptive telehealth diagnoses and treatment plans that changed once laryngoscopy was performed, as reported. The study included 38 patients. Recreated by Greyoak Research from results reported by Montalbaron et al., Annals of Otology, Rhinology & Laryngology (2023). Greyoak did not conduct this study.

Peer-reviewed studyAnnals of Otology, Rhinology & Laryngology2023

Some decisive information never enters a video visit

Montalbaron et al. Presumptive Diagnosis in Tele-Health Laryngology: A Multi-Center Observational Study. Annals of Otology, Rhinology & Laryngology, 2023.

Thirty-eight patients with voice complaints were first seen by video and later examined in person, where laryngoscopy was available. After the instrumented examination, 38% of the presumptive diagnoses and 37% of the treatment plans changed. The added information came from looking at the larynx, something an ordinary camera cannot do, rather than from seeing the patient in person.

Why it matters for evaluation. A reminder that a capture setup bounds what any reviewer, human or model, can know. Evaluations should state which findings the recording could contain.

Limitation. The decisive difference was laryngoscopy, so this is not a comparison of webcam video with in-person observation. Small retrospective sample.

Inputs

  • Text not used
  • Audio present, not isolated
  • Visual present, not isolated

A video visit was followed by an in-person visit that added laryngoscopy.

Read the original paper doi:10.1177/00034894231165811

Finding the evidence in clinical video

Before a system can interpret a clinical cue, it has to locate the moment that contains it.

How much of the footage holds the answer

759 hours of clinical procedure video

0.166% of frames hold the evidence for a question, on average

Best evaluated model, 1,253 evidence-grounded questions

Questions answered correctly
41.1%
The highlighted window is drawn to scale for the average evidence share; its position on the timeline is illustrative. Accuracy is shown on a 0 to 100 percent scale. Greyoak Research schematic based on results reported by Du et al., arXiv (2026). Greyoak did not conduct this study.

Research preprintarXiv2026

In long clinical video, finding the evidence comes first

Du et al. MedHorizon: Towards Long-context Medical Video Understanding in the Wild. arXiv, 2026. Not peer reviewed at the time of verification; a revised arXiv version may differ in detail.

MedHorizon keeps clinical procedure recordings at full length, 759 hours in all, and asks 1,253 questions whose answers rest on specific moments in the footage. Those moments are rare: on average, the relevant evidence makes up 0.166% of frames. The best model evaluated answered 41.1% of questions correctly, and the authors report that giving models more frames did not reliably help.

Why it matters for evaluation. Whether a system samples and attends to the right moment is a separate question from whether it interprets that moment correctly, and both need to be measured.

Limitation. A benchmark of long procedural videos, not patient-facing consultations or short clinical clips.

Inputs

  • Text present, not isolated
  • Audio not reported
  • Visual present, not isolated

Text questions about long procedure videos. The benchmark tests whether models find and use visual evidence.

Read the original preprint

Historical foundation

An early study of what remote audiovisual examination can and cannot transmit, item by item.

Agreement on NIH Stroke Scale items, remote compared with bedside

Excellent agreement

  • Orientation
  • Arm motor
  • Leg motor
  • Neglect

Good agreement

  • Language
  • Dysarthria
  • Sensation
  • Visual fields
  • Facial palsy
  • Gaze

Poor agreement

  • Commands
  • Ataxia

Total score correlation r = 0.97 Exam time 9.70 min remote, 6.55 min bedside

Item categories as reported by the authors. No item-level statistics are drawn. Recreated by Greyoak Research from results reported by Shafqat et al., Stroke (1999). Greyoak did not conduct this study.

Historical studyStroke1999

Reliable overall, uneven item by item

Shafqat et al. Role for telemedicine in acute stroke. Feasibility and reliability of remote administration of the NIH stroke scale. Stroke, 1999.

Twenty patients with ischemic stroke were scored on the NIH Stroke Scale by a neurologist at the bedside and by a second neurologist over a live audiovisual connection, with a nurse assisting. Total scores correlated closely, r = 0.97. Individual items did not behave alike: orientation, arm and leg motor function and neglect agreed excellently, while following commands and ataxia agreed poorly.

Why it matters for evaluation. A strong total score can hide weak components. Validity for video-based assessment needs to be shown finding by finding.

Limitation. A small study from 1999, with the audiovisual technology of the time and a nurse at the bedside.

Inputs

  • Text not used
  • Audio present, not isolated
  • Visual present, not isolated

A remote neurologist examined the patient over a live audiovisual link with a bedside nurse assisting.

Read the original paper doi:10.1161/01.STR.30.10.2141

How this bears on our work

Read together, these studies suggest that video and audio can carry clinical information that text does not, that physicians reading the same material often disagree, that a recording can only contain what its setup captured, and that a system with richer input can still miss subtle cues. None of them evaluates Greyoak’s cases or methods.

They are part of why our evaluations vary what a system can see and hear while keeping the case fixed, report where physicians disagree rather than a single answer, and score whether a system perceived a finding separately from what it recommended.

About our evaluations Physician benchmarks

Reported values last reviewed on October 13, 2026. Preprints are labeled as such and may since have been revised.