User-generated quality standards for youth mental health in primary care: a participatory research design using mixed methods
=============================================================================================================================

* Tanya Graham
* Diana Rose
* Joanna Murray
* Mark Ashworth
* André Tylee

## Abstract

**Objectives** To develop user-generated quality standards for young people with mental health problems in primary care using a participatory research model.

**Methods** 50 young people aged 16–25 from community settings and primary care participated in focus groups and interviews about their views and experiences of seeking help for mental health problems in primary care, cofacilitated by young service users and repeated to ensure respondent validation. A second group of young people also aged 16–25 who had sought help for any mental health problem from primary care or secondary care within the last 5 years were trained as focus groups cofacilitators (n=12) developed the quality standards from the qualitative data and participated in four nominal groups (n=28).

**Results** 46 quality standards were developed and ranked by young service users. Agreement was defined as 100% of scores within a two-point region. Group consensus existed for 16 quality standards representing the following aspects of primary care: better advertising and information (three); improved competence through mental health training and skill mix within the practice (two); alternatives to medication (three); improved referral protocol (three); and specific questions and reassurances (five). Alternatives to medication and specific questions and reassurances are aspects of quality which have not been previously reported.

**Conclusions** We have demonstrated the feasibility of using participatory research methods in order to develop user-generated quality standards. The development of patient-generated quality standards may offer a more formal method of incorporating the views of service users into quality improvement initiatives. This method can be adapted for generating quality standards applicable to other patient groups.

*   Quality improvement methodologies
*   Mental health
*   Primary care
*   Qualitative research
*   Health services research

## Introduction

For well over a decade, the National Institute for Health and Care Excellence (NICE) has developed a series of quality standards for the National Health Service to adopt in the four countries in the UK. These are based upon the best available evidence for three dimensions of quality: clinical effectiveness, patient safety and patient experience. NICE has encouraged the involvement of service users and carers in their quality standard development groups as well as including clinicians, managers and public health specialists. Patient and carer groups are also involved through an intensive consultation process. NICE quality standards relating specifically to young people cover depression in children and young people and focus on assessment and monitoring.1 Elsewhere, quality standards have been generated for primary care management of mental health problems in adults and children but have not involved young people in their development.2 ,3

Common mental health problems are relatively prevalent in young people aged 16–25.4 However, young people consult their general practitioners (GPs) infrequently5 even when their mental health problems are severe.6 When they do consult, they are unlikely to have their mental health problems identified.7 Young people report concerns about confidentiality, lack of respect for their health concerns and shortness of consultations in primary care8 and often associate their GPs only with physical health problems.9 GPs also report difficulties when managing this group of patients citing resource issues and training needs.8 Some GPs are uncomfortable making a diagnosis of depression in young people10 and are reluctant to discuss psychological problems with young people even when they perceive them to be present.11 Quality standards which address these problem areas may enable practitioners to improve the way services are delivered.

Features of quality care for youth-friendly health services developed by the Royal College of General Practitioners, the Royal College of Paediatrics and Child Health and the Department of Health include training, ease of access and ensuring confidentiality.12–14 But, the extent to which these quality standards are the responsibility of primary care is ill-defined. In addition, they lack focus on mental health problems or the involvement of young people other than inviting consumer organisations to comment on draft reports. Developing quality standards for primary care mental health that reflect the needs of young people is important because they represent a hard to reach group, whose use of primary care may be influenced by early experiences.15–17

Incorporating the patient experience into quality assessment of healthcare18 is a central tenet of government policy but it is difficult to assess how well this happens in practice. Patients and health professionals differ on what they consider to be important features of high quality care.19 Patients may not speak openly in development groups nor have their concerns acted upon.20 Young people have additional factors affecting their experience of healthcare. They may lack the confidence to challenge an adult in a position of authority such as a health professional who comes wrapped with all the traditional cultural and historical trappings of ‘expertise’. These issues are even more pronounced for different groups in society. The needs of young people with mental health problems tend to be assessed and defined for them by others—specifically adult health professionals—and as a result they tend to have little influence in the design and delivery of services.21

This study adopted a participatory approach to quality standard development to minimise the impact of different stakeholder groups working together. We involved young service users as focus group cofacilitators, in data analysis, project meetings and as an expert panel consisting entirely of young service users to enhance data collection and gain more indepth information.22 This is a novel approach to quality standard development as patients are usually represented in the minority and not the majority. This study aimed to produce young user-generated quality standards for primary care mental healthcare.

## Methods

### Study design

Decisions about what to research and how data are collected and analysed are not usually made by patients but by funders, clinicians and researchers. Treating research as a shared exercise between researcher and patient shifts the balance of power so that it becomes more equal.23 In a participatory research project this is done through reflexivity, disclosure and involving patients as coresearchers. This model of participatory research is often referred to as ‘cooperative enquiry’ between coresearchers with different perspectives resulting in new interpretations.24 In this study, young service users (coresearchers) worked on research tasks alongside the research team which included GP academics (AT, MA, HL), primary care researchers (JM) and service user researchers (TG, DR). Shared experiences of help seeking were disclosed among the research team, coresearchers and participants in an effort to reduce the differences in status. This encouraged the coresearchers to conduct their research tasks and incorporate their experiences into data collection and analysis.

This study was conducted in five stages (outlined in figure 1) adopting focus groups, interviews, participatory research groups and nominal group technique. Focus groups and nominal groups were adopted to allow young service users to work in partnership with other coresearchers. The guidelines adopted during focus and nominal group work which stress the importance of respect for one another, as well as confidentiality among the group, served to build trust among everyone in the group. Young service users were able to ask questions, write cofacilitator notes, contribute to debriefing meetings after the focus groups and comment on the themes developed from the analysis. Contributing to data generation and analysis enabled young service users to be meaningfully involved in the development of the quality standards. The nominal groups also complemented the participatory research design as they enabled participants to interact with one another. Sharing reasons for allocating a particular ranking encouraged young service users to share their views in a safe environment with people who had similar experiences. Being a member of the nominal group empowered young services users to make meaningful decisions about the results of the study.

![Figure 1](http://qualitysafety.bmj.com/https://qualitysafety.bmj.com/content/qhc/23/10/857/F1.medium.gif)

[Figure 1](http://qualitysafety.bmj.com/content/23/10/857/F1)

Figure 1 
Flow chart of the four phases in the development of the quality standards.

### Study participants and setting

Young people aged 16–25 from community settings and primary care participated in focus groups and interviews. Data collection took place at sixth form colleges (educational institutions where students aged 16–19 typically study for advanced level qualifications), a university, a drug and alcohol drop-in centre, a hostel and a research institute. A second group of young people (referred to as young service users) also aged 16–25 who had sought help for any mental health problem from primary care or secondary care within the last 5 years were trained as focus groups cofacilitators, developed the quality standards and participated in nominal groups. Young service users were recruited from primary care, child and adolescent mental health services, service user groups and student counselling groups. The study was conducted across four south London Primary Care Trusts, the local unit for the management of primary care and community services (Southwark; Lambeth; Lewisham and Croydon), which were superseded by Clinical Commissioning Groups on 1 April 2013.

### Phase I—from young service user to coresearcher: recruitment and training

Young service users were recruited to take part in the coresearcher training. They were aged 16–25 and had sought help for a mental health problem from a GP or mental health professional within the last 5 years. Letters were sent to all 16–25 year olds who had a diagnosis of depression or anxiety from two GP practices. Posters and fliers were also placed at three child and adolescent mental health services, a student counselling service, a homeless shelter and a supported housing project inviting 16–25 year olds to contact TG by text message, email or phone. The snowballing technique was also used to recruit coresearchers. Coresearchers were trained by TG, DR and AT for their role as focus group cofacilitators and paid for their time (box 2).

### Phase II—developing the quality standards: focus groups with young people in the community

Three pilot focus groups and three interviews with students and young people from a local homeless shelter and a local Muslim women's group were conducted to develop the topic guide, confirm that feedback to participants was feasible, and inform sampling and recruitment strategies. As most of the evidence on young people's views about the role of primary care and seeking help for mental health problems is based on students, we decided to widen the scope and include the views of more vulnerable young people such as those who were unemployed, at risk of drug abuse (type of drug not specified) and homeless.25 ,26 Sampling was also based upon ethnicity27 ,28 and gender5 ,6 which are known to affect help seeking for mental health problems. A majority of the sample were either in education or unemployed, so we also included employed young people to ensure a wide range of views and make comparisons. Some young people worked as administrators, one was an advisor for a mental health support charity and another was a researcher for an international health charity. During the recruitment stage, TG met with each group of participants to discuss the topic area and ensure they were comfortable discussing their views and experiences of seeking help for a mental health problem. TG discussed her own experiences of counselling during University and spoke about how it was important to conduct more research in this area. Each focus group was facilitated by TG (an experienced female qualitative researcher), cofacilitated by a coresearcher, and lasted between 45 and 60 min (see table 1 for demographic details). In the hostel and drop-in clinic group, the key worker was present during the focus groups. All but one focus group spoke openly about their views and experience of seeking help for mental health problems. The Asian female group felt it was a sensitive topic and so individual interviews were conducted with those focus group participants who agreed to take part. Preliminary interpretations of the data were fed back to the participants during a second focus group to ensure respondent validation and develop the coding frame (see box 1).31 ,34 All but two participants from the first set of focus groups participated in the second set of focus groups. Feeding back results through second focus groups enabled detailed further discussion of the topic resulting in theoretical saturation.35 Box 1 
### Development of the coding frame through thematic analysis of focus group and interview data

All focus groups were recorded, transcribed verbatim, edited by TG and supplemented by field notes. An initial coding framework was developed from the topic guide, key themes from the literature and the pilot work.29 ,30 The framework was validated by participants’ agreement with the preliminary interpretations.31 ,32 The coding frame was extended by reapplying it to all the transcripts through an iterative process of data abstraction. Software (Max QDA) was used to index and retrieve data. Data contained in each code were continually scrutinised to ensure consistency. Transcripts were coded independently by TG, two coresearchers, a GP academic (HL) and other researchers (JM and DR). Disagreements and alternative interpretations of the data were discussed and the coding frame was modified.33

View this table:
[Table 1](http://qualitysafety.bmj.com/content/23/10/857/T1)

Table 1 
Focus group demographics

### Phase III—developing the quality standards: participatory research groups with coresearchers

From the coding framework developed in the previous phase, codes considered most relevant to primary care were identified by two coresearchers, three GPs on the steering group (including HL) and TG. Participatory research groups facilitated by TG, AT, DR and JM were held with coresearchers to develop quality standards. The coresearchers read through all of the data represented by the codes and summarised the data into key words and phrases. These were then grouped together by similarity and refined into one or two sentences by the coresearchers and facilitators (see example in online supplementary figure S2). A pilot nominal group of 11 young service users (who were not involved in generating the quality standards from the focus group data) were asked to read through each standard and make amendments to avoid duplication and ensure clarity. We also wanted to test the feasibility of using nominal group technique with young service users.36 The pilot nominal group individually ranked each statement in order of importance on a 5-point scale (1—very important, 2—important, 3—no opinion, 4—not important, 5—not at all important). Importance was defined as ‘the extent to which the standard is necessary when suffering from emotional distress’. We chose nominal group technique rather than the Delphi process so that consensus could be reached as a result of group interaction and not individuals working in isolation. Encouraging group interaction was a vital component of the participatory methodology employed for involving young service users in the research process. It was also important to allow the young service users to ask questions of the facilitator as most had not been involved in such a process before.

### Phase IV—Developing the quality standards: reaching consensus among young service users

The quality standards were presented by TG to a second expert panel of 12 young service users. As with the pilot nominal group, the group individually ranked each statement in order of importance on a 5-point scale (1—very important, 2—important, 3—no opinion, 4—not important, 5—not at all important). Importance was defined as ‘the extent to which the standard is necessary when suffering from emotional distress’. Individual rankings from the first round were tabulated, anonymised and presented back to the group. Members extensively discussed reasons for their rankings. DR, AT and JM observed the group and kept a record of what was said to ensure completeness of data. The group was then asked to rerank each statement. The results were presented back to the youth expert panel who all agreed that the final set of quality standards accurately represented their preferences. Box 2 
### Ethical considerations

There were a number of ethical issues related to involving young service users as researchers that we needed to consider.37 Asking young people to cofacilitate focus groups and interpret data represented for the most part a new role in which they had little or no experience. We provided clear information about what the roles entailed and how they would be supported through their involvement (appropriate cofacilitator training, feeding back data and research results, and organising data analysis meetings). We carefully judged how much new information they could take in, and incorporated methods of group work that would keep them interested including ice breaker and group cohesion exercises. The impact of taking part in research tasks on the emotional well-being of the young service users was closely monitored by TG and the research team. TG held debriefing sessions after each focus group to discuss what the participants had talked about and how it affected the coresearchers.

## Results

### Phases I and II: recruiting young services users and collecting qualitative data

Twelve young service users took part in the focus group cofacilitator training sessions. Seven focus groups were conducted with 50 young people both with and without mental health problems each cofacilitated by young service users. Three interviews with Asian women were conducted to supplement the focus group data. Thematic analysis of the focus group and interview data analysis resulted in 17 codes and 49 subcodes representing a range of topics including confidentiality, counselling, features of an ideal service, reasons for and for not seeking help, other places for help, self-help, gender differences, medication, stigma, and experiencing stress in comparison with having a mental health problem (see online supplementary table S2).

### Phase III: participatory research groups

From the coding framework developed in phase II, nine codes and three subcodes considered most relevant to primary care were identified (see online supplementary table S2). Two participatory research groups were held with five coresearchers. The participatory research groups yielded a total of 49 quality standards from the focus group data (detailed in online supplementary table S3). The quality standards represent the following four areas of primary care: quality standards 1–6 refer to issues of practice organisation and skill mix; quality standards 7–18 represent broader treatment options and better referral protocols; quality standards 19–36 refer specifically to the consultation process, and include putting young people at ease, confidentiality and suggestions for improving interaction with young people; and quality standards 37–49 illustrate features of an ideal service such as improving accessibility and environmental preferences of a primary care practice. Of the 49 quality standards reviewed by the pilot nominal group, 27 were included, amendments were made to 19 and three were removed due to duplication. The amendments resulted in quality standards that were more succinct, better aligned to the terms used by young service users and with an appropriate emphasis as dictated by the young service users. The nominal group technique enabled young service users to work together with one another and the research team sharing experiences and discussing rankings within the group illustrating that it was a feasible method to use with this group. The pilot nominal group process resulted in 46 quality standards to be presented to the final nominal group in phase IV.

### Phase IV: Final nominal groups

The nominal group scores of the 12 young service users converged between rounds 1 and 2 in relation to each quality standard illustrating the importance of group interaction and reranking.36 Agreement on which standards should constitute the final quality standards was established by including only those standards where all the individual rankings were either 1 or 2 (very important or important). These rankings were calculated from the round 2 scores to incorporate the impact of group interaction. Sixteen standards met these criteria for agreement (figure 3). The standards apply to both practice and patient levels of service representing the following aspects of primary care: better advertising and information about local services and confidentiality policies; mental health training provision; access to different treatment options; improved referral protocol; and specific questions and reassurances during a consultation (see table 2).⇓

View this table:
[Table 2](http://qualitysafety.bmj.com/content/23/10/857/T2)

Table 2 
The final quality standards*

![Figure 2](http://qualitysafety.bmj.com/https://qualitysafety.bmj.com/content/qhc/23/10/857/F2.medium.gif)

[Figure 2](http://qualitysafety.bmj.com/content/23/10/857/F2)

Figure 2 
Example of how codes were developed into quality standards for ranking by the Youth Expert Panel.

![Figure 3](http://qualitysafety.bmj.com/https://qualitysafety.bmj.com/content/qhc/23/10/857/F3.medium.gif)

[Figure 3](http://qualitysafety.bmj.com/content/23/10/857/F3)

Figure 3 
Number of standards grouped by level of agreement. Agreement was based upon all 12 individuals ranking the standard as either ‘very important’ or ‘important’.

## Discussion

Sixteen quality standards for primary care mental health were developed using nominal group technique with 28 young service users. To the best of our knowledge, no other quality standards have been generated using participatory research methods with young service users as the sole stakeholder group. This novel approach to quality standard development led to different aspects of quality care for youth mental health in primary care being generated that have not been previously reported. The inclusive sampling strategy enabled the engagement of young people from a wide range of socio-economic groups. Young people were also recruited through GP practices and a range of other services so that those who did not access help from primary care were also included. These hard to reach patients are often not included in other studies.

### Focus on treatment options and communication during consultation

The quality standards are similar to primary care mental health quality standards in the areas of training,2 ,3 ensuring confidentiality and referral protocol.3 NICE quality standards also state that practitioners should be competent in delivering appropriate interventions,39 that ‘systems are in place to confirm a diagnosis’ and that information about treatment options is provided.1 Keeping skills up to date, advertising services and managing confidentiality also feature in quality standards for general youth healthcare developed by the Department of Health,12 the Royal College of General Practitioners13 and the Royal College of Paediatrics and Child Health.14

The user-generated quality standards differ from existing quality standards for young people in the areas of treatment options (quality standards 5 and 7) and communication during consultation (quality standards 11, 12, 14 and 15). Quality standards 5 and 7 call for greater consideration to be given to talking therapies as a form of treatment compared with medication. Despite the relatively new Improved Access to Psychological Therapies (IAPT) programme, access to talking therapies for young people remains restricted for two reasons. First, IAPT for under 18 year olds is limited to existing Child and Adolescent Mental Health services and is not routinely available within primary care. Second, NICE quality standard for depression in people aged 18 and over states that access to low-intensity psychological interventions should be limited to people with subthreshold, mild or moderate depression.39 This limits access to high-intensity psychological therapies and other talking therapies for young people (aged 18–24) with symptoms that do not meet the thresholds defined by NICE guidance.

Both sets of quality standards for primary care mental health refer to the importance of good communication3 and an ‘awareness of stigmatising feelings’ during consultation.2 These are quite general including ‘GPs should communicate effectively with both parents and children about the child's emotional and behavioural health’.3 The user-generated quality standards provide more specific questions and reassurances that encompass good quality care for young people with mental health problems, for example, ‘Primary care practitioners should ask questions about a young person's relationships and support network.’

### Ensuring quality assessment is meaningful to patients

Patient reported outcome measures (PROMs) and patient reported experience measures are increasingly being used to assess quality of primary care. However, it has been noted that PROMs used in the IAPT outcomes framework reflect professionals’ concerns and lack those of users who prefer to focus on recovery rather than severity of mental health problems.40 Other frequently used measures such as the Global Assessment of Functioning and General Health Questionnaire have also been criticised for not adequately capturing the experiences of mental health service users.41 In addition, there is little evidence to suggest that surveys about young people's experience of mental health services have an impact upon service development.21 This may be because practitioners find it difficult to translate survey results into a change in their practice or the way a service is delivered. User-generated quality standards may be a more effective way of incorporating patient experience into quality assessment and service development.

Involving patients as interviewers and in questionnaire development is a core feature of ‘user research’ in the mental health field42 ,43 leading to different questions being asked that may be more closely aligned to the needs and experiences of people receiving services.44 For this study, in addition to different questions being asked, new quality standards were developed that focus on aspects of care not previously reported, notably, treatment options and communication during consultation.

### Limitations

Although the use of nominal group technique ensures the user-generated quality standards have face validity, there are a number of other criteria which have not been assessed. Campbell *et al* argue that quality standards should be ‘tested for acceptability, feasibility, reliability, sensitivity to change and (content) validity’.38 Acceptability has been assessed by nominal group agreement of the results and focus groups and interviews with practitioners (to be published). The latter provided an opportunity to examine the issues that may arise during implementation. The remaining criteria would need to be assessed through piloting the user-generated quality standards for determining their validity when applied to service evaluation in routine practice.

### Adopting participatory research methodology to develop quality standards

There were a number of challenges that we negotiated throughout this project associated with participatory research methodology. The recruitment process was lengthy and involved meeting with each of the 28 young service users individually to discuss the project. Ensuring that the young service users remained engaged and were fully supported required significant researcher time. It was important to not overload young service users while respecting their roles as coresearchers. This process was managed by seeking verbal feedback on the experience of being a coresearcher throughout the study.

Managing expectations about what was possible to achieve was another important ethical issue. We reminded the young service users that there were other barriers that may affect the impact of the quality standards such as the GPs’ knowledge of mental health problems and resources to provide other treatment options. Involving young service users in the research process required the research team members to examine their own roles and negotiate shared decision-making with young service users. This was not always easy to achieve and required the authors to be self-aware and critical of the obstacles affecting their ability to share power and control over the research with the young service users.45

### Incorporating user-generated quality standards into practice

The next phase would be to incorporate the piloted quality standards with existing toolkits to aid practices in improving the quality of care provided to young people with mental health problems.46 This could be achieved by a working group of young patients and practitioners from a number of practices followed up by an evaluation. Such a programme of work could be embedded within existing networks such as the Royal College of General Practitioners’ Adolescent Health Group with learning events across practices through quality improvement collaboratives. A collaborative approach between patients and practitioners is pivotal as practitioners are generally supportive of those quality standards that they feel are useful and that they have some influence over.47 In this study, GP academics were involved in data analysis and facilitated the participatory research and nominal group meetings.

## Conclusions

The results illustrate that exploring young service users’ definitions of quality in primary care mental health leads to new interpretations as well as similarities with previously published standards. This builds upon our knowledge of what encompasses an optimal primary care service and can inform patient-centred practice. It is vitally important to fully involve young service users in the development of quality improvement initiatives that assess their experiences and needs. This study ultimately raises the question whether young service user views can have an impact on policy and commissioning decisions and practice and, if so, how that can occur. We have illustrated that through the use of a participatory research design, young service users’ views can be meaningfully incorporated into quality indicator development and can thereafter form part of the evidence base on which policy is based.

## Acknowledgments

We would like to thank the young people who contributed to this project. Their experiences and insights into the topic gave us a new perspective on how to improve quality in primary care mental health. We would like to pay tribute to Professor Helen Lester for her support during the project and drafting this paper.

## Footnotes

*   Contributors TG designed the data collection tools with DR, recruited the young service users and focus groups and interview participants, collected the data with the young service users as focus group cofacilitators, developed the quality standards, analysed the data with DR, AT, JM and the young service users and drafted and revised the paper. DR and AT developed the project protocol, supervised data collection, analysis and the quality indicator development and revised the paper. JM supervised quality indicator development and analysed the data. MA supervised data collection and analysis and drafted the paper. All authors had full access to and can take responsibility for the data and analyses. TG is the guarantor.

*   Funding This 4-year study was funded by a grant (£250 000) from the NHS London Executive (London Regional Office Research Programme) (grant number RCD02054). The study sponsors were King's College London. AT and DR are partly funded by the NIHR Mental Health Biomedical Research Centre at the Institute of Psychiatry, King's College London and the South London and Maudsley Foundation Trust.

*   Competing interests None.

*   Patient consent Obtained.

*   Ethics approval This study was approved by the Institute of Psychiatry Ethics Committee, King's College London reference number 139/01.

*   Provenance and peer review Not commissioned; externally peer reviewed.

*   Data sharing statement Relevant anonymised patient level data are available on reasonable request from the authors.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: [http://creativecommons.org/licenses/by-nc/4.0/](http://creativecommons.org/licenses/by-nc/4.0/)

## References

1.  National Institute for Health and Care Excellence. Depression in children and young people: quality standard (QS) 48. 2013. [http://guidance.nice.org.uk/QS48](http://guidance.nice.org.uk/QS48) (accessed 12 Mar 2014).
    
    

2.  Shield T, Campbell S, Rogers A, et al. Quality indicators for primary care mental health services. Qual Saf Health Care 2003;12:100–6.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjEyLzIvMTAwIjtzOjQ6ImF0b20iO3M6MTk6Ii9xaGMvMjMvMTAvODU3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

3.  Sayal K, Amarasinghe M, Robotham S, et al. Quality standards for child and adolescent mental health in primary care. BMC Fam Pract 2012;13:51.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1186/1471-2296-13-51&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=22672193&link_type=MED&atom=%2Fqhc%2F23%2F10%2F857.atom) 

4.  McManus S, Meltzer H, Brugha T, et al. Adult Psychiatric Morbidity in England 2007: Results of a Household Survey. UK: The Health and Social Care Information Centre, 2007. [http://www.hscic.gov.uk/catalogue/PUB02931](http://www.hscic.gov.uk/catalogue/PUB02931) (accessed 14 Apr 2014).
    
    

5.  Oliver MI, Pearson N, Coe N, et al. Help-seeking behaviour in men and women with common mental health problems: cross-sectional study. Br J Psychiatry 2005;186:297–301.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6ImJqcHJjcHN5Y2giO3M6NToicmVzaWQiO3M6OToiMTg2LzQvMjk3IjtzOjQ6ImF0b20iO3M6MTk6Ii9xaGMvMjMvMTAvODU3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

6.  Biddle L, Gunnell D, Sharp D, et al. Factors influencing help seeking in mentally distressed young adults: a cross-sectional survey. Br J Gen Pract 2004;54:248–53.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNTQvNTAxLzI0OCI7czo0OiJhdG9tIjtzOjE5OiIvcWhjLzIzLzEwLzg1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

7.  Haller DM, Sanci LA, Sawyer SM, et al. The identification of young people's emotional distress: a study in primary care. Br J Gen Pract 2009;59:e61–70.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNTkvNTYwL2U2MSI7czo0OiJhdG9tIjtzOjE5OiIvcWhjLzIzLzEwLzg1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

8.  Jacobson L, Richardson G, Parry-Langdon N, et al. How do teenagers and primary healthcare providers view each other? An overview of key themes. Br J Gen Pract 2001;51:811–16.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNTEvNDcxLzgxMSI7czo0OiJhdG9tIjtzOjE5OiIvcWhjLzIzLzEwLzg1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

9.  Biddle L, Donovan JL, Gunnell D, et al. Young adults’ perceptions of GPs as a help source for mental distress: a qualitative study. Br J Gen Pract 2006;56:924–31.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNTYvNTMzLzkyNCI7czo0OiJhdG9tIjtzOjE5OiIvcWhjLzIzLzEwLzg1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

10. Iliffe S, Williams G, Fernandez V, et al. General practitioners’ understanding of depression in young people: qualitative study. Prim Health Care Res Dev 2008;9:269–79.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1017/S1463423608000868&link_type=DOI) 

11. Martinez R, Reynolds S, Howe A. Factors that influence the detection of psychological problems in adolescents attending general practices. Br J Gen Pract 2006;56:594–9.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiYmpncCI7czo1OiJyZXNpZCI7czoxMDoiNTYvNTI5LzU5NCI7czo0OiJhdG9tIjtzOjE5OiIvcWhjLzIzLzEwLzg1Ny5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

12. Department of Health. ‘You're Welcome’: quality criteria for young people friendly health services. 2011. [http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH\_126813](http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_126813) (accessed 14 Mar 2014).
    
    

13. McPherson A. Adolescents in primary care. BMJ 2005;330:465–7.
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMzAvNzQ4OS80NjUiO3M6NDoiYXRvbSI7czoxOToiL3FoYy8yMy8xMC84NTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

14. Royal College of Paediatrics and Child Health. Bridging the gaps—healthcare for adolescents. Royal College of Paediatrics and Child Health, 2003. [http://www.rcpch.ac.uk/system/files/protected/page/bridging\_the\_gaps.pdf](http://www.rcpch.ac.uk/system/files/protected/page/bridging_the_gaps.pdf) (accessed 16 Mar 2014).
    
    

15. Rogers A, Hassell K, Nicolaas G. Demanding Patients? Analysing the use of primary care. Buckingham: Philadelphia Open University Press, 1999:30–50.
    
    

16. Dixon-Woods M, Cavers D, Agarwal S, et al. Conducting a critical interpretive synthesis of the literature on access to healthcare by vulnerable groups. BMC Med Res Methodol 2006;6:35.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1186/1471-2288-6-35&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=16872487&link_type=MED&atom=%2Fqhc%2F23%2F10%2F857.atom) 

17. Biddle L, Donovan J, Sharp D, et al. Explaining non-help-seeking amongst young adults with mental distress: a dynamic interpretive model of illness behaviour. Sociol Health Illn 2007;29:983–1002.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/j.1467-9566.2007.01030.x&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=18092979&link_type=MED&atom=%2Fqhc%2F23%2F10%2F857.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000251740900002&link_type=ISI) 

18. Darzi A. High Quality Care for All: NHS next stage review final report. London: Department of Health, 2008. [https://http://www.gov.uk/government/uploads/system/uploads/attachment\_data/file/228836/7432.pdf](https://http://www.gov.uk/government/uploads/system/uploads/attachment_data/file/228836/7432.pdf) (accessed 30 Apr 2014).
    
    

19. Campbell SM, Shield T, Rogers A, et al. How do stakeholder groups vary in a Delphi technique about primary mental health care and what factors influence their ratings? Qual Saf Health Care 2004;13:428–34.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjg6IjEzLzYvNDI4IjtzOjQ6ImF0b20iO3M6MTk6Ii9xaGMvMjMvMTAvODU3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

20. van Wersch A, Eccles M. Involvement of consumers in the development of evidence based clinical guidelines: practical experiences from the North of England evidence based guideline development programme. Qual Health Care 2001;10:10–16.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoicWhjIjtzOjU6InJlc2lkIjtzOjc6IjEwLzEvMTAiO3M6NDoiYXRvbSI7czoxOToiL3FoYy8yMy8xMC84NTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

21. Worrall-Davies A, Marino-Francis F. Eliciting Children's and Young People's Views of Child and Adolescent Mental Health Services: A Systematic Review of Best Practice. Child Adolesc Ment Health 2008;13:9–15.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1111/j.1475-3588.2007.00448.x&link_type=DOI) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000207853500002&link_type=ISI) 

22. Staley K. Exploring Impact: Public involvement in NHS, public health and social care research. Eastliegh: INVOLVE, 2009. [http://www.conres.co.uk/pdfs/Involve\_Exploring\_Impactfinal28.10.09.pdf](http://www.conres.co.uk/pdfs/Involve_Exploring_Impactfinal28.10.09.pdf) (accessed 14 Mar 2014).
    
    

23. 1.  Israel BA, 
    2.  Eng E, 
    3.  Schulz AJ, 
    4.  et al
    
    Israel BA, Eng E, Schulz AJ, et al. Introduction to methods in community-based participatory research for health. In: Israel BA, Eng E, Schulz AJ, et al. eds. Methods in community-based participatory research for health. San Fransisco, CA: Jossey-Bass, 2005:3–26.
    
    

24. Reason P, Bradbury HE. Handbook of action research: participative inquiry and practice. London: Sage, 2008:179–88.
    
    

25. Craig TK, Hodson S. Homeless youth in London: I. Childhood antecedents and psychiatric disorder. Psychol Med 1998;28:1379–88.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1017/S0033291798007351&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=9854279&link_type=MED&atom=%2Fqhc%2F23%2F10%2F857.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000077362600013&link_type=ISI) 

26. Edidin J, Ganim Z, Hunter S, et al. The mental and physical health of homeless youth: a literature review. Child Psychiatry Hum Dev 2012;43:354–75.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1007/s10578-011-0270-1&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=22120422&link_type=MED&atom=%2Fqhc%2F23%2F10%2F857.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000303380400003&link_type=ISI) 

27. Lamb J, Bower P, Rogers A, et al. Access to mental health in primary care: a qualitative meta-synthesis of evidence from the experience of people from ‘hard to reach’ groups. Health (London) 2012;16:76–104.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1177/1363459311403945&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=21486918&link_type=MED&atom=%2Fqhc%2F23%2F10%2F857.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000298603500005&link_type=ISI) 

28. Morgan C, Mallett R, Hutchinson G, et al. Pathways to care and ethnicity. 2: Source of referral and help-seeking: report from the ÆSOP study. Br J Psychiatry 2005;186:290–6.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTA6ImJqcHJjcHN5Y2giO3M6NToicmVzaWQiO3M6OToiMTg2LzQvMjkwIjtzOjQ6ImF0b20iO3M6MTk6Ii9xaGMvMjMvMTAvODU3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

29. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006;3:77–101.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1191/1478088706qp063oa&link_type=DOI) 

30. Mays N, Pope C. Quality in qualitative health research. Qualitative research in health care. Blackwell Publishing Ltd, 2007:82–101.
    
    

31. Lincoln YS, Guba EG. Naturalistic Inquiry. London: SAGE, 1985:250–88.
    
    

32. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups. Int J Qual Health Care 2007;19:349–57.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NjoiaW50cWhjIjtzOjU6InJlc2lkIjtzOjg6IjE5LzYvMzQ5IjtzOjQ6ImF0b20iO3M6MTk6Ii9xaGMvMjMvMTAvODU3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

33. Barbour RS. Checklists for improving rigour in qualitative research: a case of the tail wagging the dog? BMJ 2001;322(7294):1115–17.
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEzOiIzMjIvNzI5NC8xMTE1IjtzOjQ6ImF0b20iO3M6MTk6Ii9xaGMvMjMvMTAvODU3LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

34. Kidd PS, Parshall MB. Getting the focus and the group: enhancing analytical Rigor in focus group research. Qual Health Res 2000;10:293–308.
    
    [Abstract/FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3BxaHIiO3M6NToicmVzaWQiO3M6ODoiMTAvMy8yOTMiO3M6NDoiYXRvbSI7czoxOToiL3FoYy8yMy8xMC84NTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

35. 1.  Silverman D
    
    Green J, Thorogood N. Qualitative methods for health research. Silverman D. ed. London, Thousand Oaks, New Delhi: SAGE, 2007:103.
    
    

36. Murphy MK, Black NA, Lamoing DL, et al. Consensus development methods, and their use in clinical guideline development. Health Technol Assess 1998;2:i-iv, 1–88.
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=9561893&link_type=MED&atom=%2Fqhc%2F23%2F10%2F857.atom) 

37. Kirby P. A Guide to actively involving young people in research: for researchers, research commissioners, and managers. INVOLVE, 2004. [http://www.invo.org.uk/posttypepublication/a-guide-to-actively-involving-young-people-in-research/](http://www.invo.org.uk/posttypepublication/a-guide-to-actively-involving-young-people-in-research/) (accessed 30 Mar 2014).
    
    

38. Campbell SM, Braspenning J, Hutchinson A, et al. Research methods used in developing and applying quality indicators in primary care. BMJ 2003;326:816–19.
    
    [FREE Full Text](http://qualitysafety.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjYvNzM5My84MTYiO3M6NDoiYXRvbSI7czoxOToiL3FoYy8yMy8xMC84NTcuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

39. National Institute for Health and Care Excellence. Depression in adults quality standard (QS8). 2011. [http://publications.nice.org.uk/depression-in-adults-quality-standard-qs8](http://publications.nice.org.uk/depression-in-adults-quality-standard-qs8) (accessed 20 Mar 2014).
    
    

40. Fitzpatrick R, Gibbons E, Mackintosh A. An overview of patient-reported outcome measures for people with anxiety and depression. Oxford: Patient-reported Outcome Measurement Group, Department of Public Health, University of Oxford, 2009. [http://phi.uhce.ox.ac.uk/home.php](http://phi.uhce.ox.ac.uk/home.php) (accessed 20 Mar 2014).
    
    

41. Crawford MJ, Robotham D, Thana L, et al. Selecting outcome measures in mental health: the views of service users. J Ment Health 2011;20:336–46.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.3109/09638237.2011.577114&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=21770782&link_type=MED&atom=%2Fqhc%2F23%2F10%2F857.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000292914100004&link_type=ISI) 

42. Schneider B. Participatory action research, mental health service user research, and the hearing (our) voices projects. Int J Qual Methods 2012;11:152–65.
    
    

43. Rose D, Evans J, Sweeney A, et al. A model for developing outcome measures from the perspectives of mental health service users. Int Rev Psychiatry 2011;23:41–6.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.3109/09540261.2010.545990&link_type=DOI) 
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=21338297&link_type=MED&atom=%2Fqhc%2F23%2F10%2F857.atom) 
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000287524600007&link_type=ISI) 

44. Rose D, Wykes TIL, Farrier D, et al. What do clients think of cognitive remediation therapy?: a consumer-led investigation of satisfaction and side effects. Am J Psychiatr Rehabil 2008;11:181–204.
    
    [CrossRef](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10.1080/15487760801963694&link_type=DOI) 

45. Mason R, Boutilier M. The challenge of Genuine power sharing in participatory research: the gap between theory and practice. Can J Commun Ment Health 1996;15:145–52.
    
    [PubMed](http://qualitysafety.bmj.com/lookup/external-ref?access_num=10166891&link_type=MED&atom=%2Fqhc%2F23%2F10%2F857.atom) 

46. Freer M. A toolkit for GPs: the mental health consultation (with a young person). The Charile Waller Memorial Trust, 2012. [http://www.cwmt.org.uk/wp-content/uploads/2012/04/GPToolkit2012.pdf](http://www.cwmt.org.uk/wp-content/uploads/2012/04/GPToolkit2012.pdf) (accessed 20 Mar 2014).
    
    

47. Valenstein M, Mitchinson A, Ronis DL, et al. Quality indicators and monitoring of mental health services: what do frontline providers think? Am. J Psychiatry 2004;161: 146–53.
    
    [Web of Science](http://qualitysafety.bmj.com/lookup/external-ref?access_num=000221275700023&link_type=ISI)