There is growing evidence that non-technical skills (NTS) are related to surgical outcomes and patient safety. The aim of this study was to further evaluate a behaviour rating system (NOTSS: Non-Technical Skills for Surgeons) which can be used for workplace assessment of the cognitive and social skills which are essential components of NTS. A novice group composed of consultant surgeons (n = 44) from five Scottish hospitals attended one of six experimental sessions and were trained to use the NOTSS system. They then used NOTSS to rate surgeons' behaviors in six simulated scenarios filmed in the operating room. The behaviours demonstrated in each scenario were compared to expert ratings to determine accuracy. The mode rating from the novice group (who received a short training session in behaviour assessment) was the same as the expert group in 50% of ratings. Where there was disagreement, novice raters tended to provide lower ratings than the experts. Novice raters require significant training in this emerging area of competence in order to accurately rate non-technical skills.