Background Doctors' ability to communicate with patients varies. Patient questionnaires are often used to assess doctors' communication skills.

Objective To investigate whether the Four Habits Patient Questionnaire (4HPQ) can be used to assess the different skill levels of doctors.

Design A cross-sectional study of 497 hospital encounters with 71 doctors. Encounters were videotaped and patients completed three post-visit questionnaires.

Setting A 500-bed general teaching hospital in Norway.

Main outcome The proportion of video-observed between-doctor variance that could be predicted by 4HPQ.

Results There were strong correlations between all patient-reported outcomes (range 0.71–0.80 at the doctor level, p<0.01). 4HPQ correlated significantly with video-observed behaviour at the doctor level (Pearson's r=0.42, p<0.01) and the encounter level (Pearson's r=0.27, p<0.01). The proportion of between-doctor variance not detectable by 4HPQ was 88%. The reason for this discordance was large within-doctor between-encounter variance observed in the videos, and small between-patient variance in patient reports. The maximum positive predictive value for the identification of poorly performing doctors (92%) was achieved with a cut-off score for 4HPQ of 82% (ie, patient assessments were concordant with expert observers for these doctors).

Conclusion Using a patient-reported questionnaire of doctors' communication skills, favourable assessments of doctors by patients were mostly discordant with the views of expert observers. Only very poor performance identified by patients was in agreement with the views of expert observers. The results suggest that patient reports alone may not be sufficient to identify all doctors whose communication skills need improvement training.