Phase I

A random sample of comments (n=400) was triple coded by three researchers (AR-S, KH, RW), (two female/one male), each with more than 10 years’ experience with qualitative research. Data were deductively coded against an abridged version of an existing thematic framework designed for the PROM pilot study.11 The framework allows comments to be coded as positive or negative experiences of specific areas of care (ie, timely/delayed diagnosis; good/inadequate post-treatment care) and whether specific forms of information to prepare patients were lacking (ie, lack of information on treatment side-effects; psychological impact of cancer and treatment; self-management strategies). Consistency of inter-rater coding (Cohen's Kappa) ranged from substantial (0.64) to excellent (0.87). Inconsistencies between the existing framework and data were discussed between researchers, with disagreements resolved by a fourth researcher (JC).

Phase II

Algorithms were then trained and tested to identify comments within the whole dataset that specifically related to positive and negative experiences of care quality. The quality of the algorithm results depend on three factors.37 First, the quality of the data provided may make it more ‘difficult’ or ‘easy’ it is to identify patterns. Second, each algorithm is governed by different sequential sets of rules for identifying semantic or grammatical relationships within the text, and particular algorithms may suit some datasets better than others may. Third, the larger the training sets used the more accurate are algorithms likely to be at identifying similar comments within the wider dataset, but trade-offs with time and human coding are necessary to ensure the method is resource-efficient.37

To optimise the number of coded comments available to train algorithms, a second random sample of comments (n=400) (random sample 2) was coded to provide a combined term document matrix (TDM) of 800 comments, resulting in a two-dimensional grid of rows (samples) and columns (terms) representing the frequency that a term appears in the samples. The following ‘R’ settings in the TDM {tm} package were used in creating the TDM: removeNumbers=TRUE, stemWords=TRUE, weighting=tm::weightBin. No further processing was applied to this document.38 This TDM was then used as a template for mining the remaining comments in the database. A 50% sample was drawn randomly from the 800 to train algorithms, the remainder used to test them. A 10-fold cross validation (a technique in which an original sample is randomly split into 10 subsamples, with training conducted in 9 datasets, testing on one, and the process repeated 10 times) was then used to assess algorithm performance.38 Algorithm performance is measured as sensitivity (true positives/(true positives+false negatives)), precision (true positives/(true positives+false positives)) and by the f-score ((2×sensitivity×precision)/(sensitivity+precision)).39 Sensitivity describes the ability to identify all relevant comments of a given category; precision defines the ability to exclude non-relevant comments. The f-score describes overall performance, representing the harmonic mean of precision and sensitivity.39 Comparative analysis using t tests was then conducted between categories of individuals’ comments and their single index EQ5D score (summarising five domains: mobility; self-care; usual activities; pain/discomfort; anxiety/depression), to identify associations between them.

Phase III

All retrieved comments were then read and double coded by two researchers (AR-S, RW) to determine their relevance for each category of care experience, with a third researcher (JC) supporting interpretive analysis. Individual respondents often provided comments that were coded into more than one category, and some reported both positive and negative experiences, which were coded accordingly. Any disagreements between coders were resolved through discussion. Comments coded as irrelevant often described a patient's treatment journey but without conveying sentiments regarding their quality (eg, whether experiences were positive or negative).

Patterns and relationships were explored between reported care quality in individuals’ comments and their HRQoL, as measured by their summary EQ5D scores. Comments coded in each category were examined for references made to other themes and any reported relationships between them. These were explicit within many comments, but latent in others. Emerging patterns were then tested against all comments in relevant categories and a tentative model generated to illustrate where data from a number of participants indicated such patterns existed (figure 2).40 The concepts of validity and trustworthiness within qualitative research are understood in terms of the credibility and dependability of the analytical process.41 ,42 The emerging model was tested against all individual negative and positive comments to challenge its capacity to represent participants’ experiences. Selected quotes are used to illuminate particular types of experience.

• Download figure
• Open in new tab
• Download powerpoint

Figure 2

Model of factors influencing quality of patient experience. GP, general practitioner.

Development of text-mining approach

In the first analytical phase, the coding framework previously developed within the pilot PROM study11 was found to comprehensively code care experience comments in the CRC dataset. Of the 800 manually coded comments, 248 were coded as reporting positive care experiences (Excellent/good treatment/caring staff) and 85 as reporting negative experiences (poor staff attitude/lack of care). Extrapolation to the remaining comments in the dataset (n=4834) suggested that algorithms should be expected to find approximately 1436 positive comments (29.7% of all comments) and 491 negative comments (10.1%) in total (table 2).

In phase II initial testing indicated that some comment categories were easier to detect than others, and that algorithms trained to identify both positive and negative experience coded comments outperformed separate algorithms for each. Table 3 indicates the precision, sensitivity and f-scores calculated for each of the seven algorithms used. Highest overall sensitivity was found for the SVM (support vector machines) algorithm (78%), with precision considerably higher (83.5%), and an overall f-score of 80%. Based on the training and testing we would have expected approximately 1360 comments to be retrieved from the remaining comments (n=4834) through this one algorithm.

As each algorithm identifies comments by using different sets of rules to recognise semantic and grammatical patterns, they are each likely to retrieve varying numbers of comments regarding any specific theme.34 Only 313 comments were identified by all seven algorithms. However, 2645 records had at least one algorithm indicating they contained a care experience comment. Therefore, to optimise the number of retrieved comments from the yet unclassified dataset (n=4834), we retrieved those that had been identified separately by the four best performing algorithms (SVM; RF (Random Forests); TREE (Decision Trees); GLMNET (Generalised linear models network)). This retrieval strategy identified 2076 comments, of which 1688 (81.3%) were found to be relevant, and 388 (18.7%) irrelevant. Relevant comments comprised 1045 coded as positive, fewer comments (n=391, 8.4%) than expected from manual coding extrapolations, and 643 coded as negative, more comments (n=152, 3.2%) than expected (table 2). The content of positive comments was usually much less specific than for negative comments, and algorithms could only effectively identify two positive comment subcategories, with 109 (6%) comments describing timely diagnosis and 289 (17%) describing post-treatment care in positive terms. Although positive comments did exist within other subcategories (eg, coordination of care), they were too few to adequately train algorithms to identify them in the dataset. Algorithms were more successful in identifying subcategories for negative comments. The highest proportions of negative experiences related to inadequate post-treatment care (n=191, 11%) and to lack of information provided by staff concerning treatment side-effects (n=160, 9%) and self-management strategies (n=135, 8%).

Validation of results

A generalised approach to compare the performance of algorithms is the 10-fold cross validation, which divides the data sample into 10 blocks or folds and builds 10 evaluations on each fold in order to estimate the error variance. The feature to perform this validation in the RTools was used to validate the results obtained with the original 50/50 approach adopted. Results of the performance variations between the two methods described in table 3 shows that most algorithms remained unchanged except for the Maxentropy, which produced dramatically different results (0.014 10-fold compared with 0.67 in the original approach), a 25% reduction in the score for GLMNET and 20% improvement for Logitboost, which became the best performing algorithm. The use of four best performing algorithms to retrieve comments from the unclassified dataset, which performed well in the 10-fold cross validation, improved the retrieval strategy and controlled the limitations of the 50/50 approach.

Comparing coded experiences of care with reported summary EQ5D scores

Mean summary EQ5D scores were explored to determine whether individuals’ self-reported HRQoL could be correlated with their reported experiences of care quality. The mean summary EQ5D score for all survey respondents who left comments (‘All comments’) on any subject was 0.78, but was higher (0.85) for those respondents who provided ‘positive comments’ and lower (0.74) for those providing ‘negative comments’ (table 2). T tests demonstrated these to be significant associations (p≤0.001), indicating that those reporting positive experiences of care had correspondingly higher HRQoL than the mean for all free-text respondents, and those reporting negative experiences. Associations were also found between mean EQ5D scores for ‘All comments’ and the comment subcategories: timely diagnosis (p≤0.001); positive post-treatment care (p≤0.001); inadequate post-treatment care (p=0.002).

Experiences of care quality and their reported impact on quality of life

Given the quantitative evidence that lower HRQoL scores were associated with negative comments of care quality, in phase III researchers analysed comments to better understand this relationship: how did participants with lower or higher reported HRQoL describe their care? Any explanatory theory must include reference to mechanisms by which relationships between variables are generated.43 A tentative model was developed to illustrate relationships between themes emerging from the data (figure 2), in which evidence from comments suggested the link between HRQoL and different experiences of care quality. The top half of the figure represents those aspects of care associated by respondents with positive experiences of care, and which reportedly minimised or addressed challenges faced along the treatment pathway. The lower part of the figure identifies issues related to experiences of care that had a negative impact on respondents’ HRQoL.

Timeliness of diagnosis

Text-mining identified 109 (6.4%) comments describing their path to diagnosis as timely, compared with 36 comments (2.1%) describing it as delayed. Comments indicated that participants recognised a timely diagnosis had contributed to successful treatment outcomes, thus positively impacting on HRQoL:The early diagnosis of cancer and treatment has been essential to my excellent recovery. It was discovered after giving blood. I have returned to work a year ago and I have had no time off at all since despite going back early. (Male, age 45–55)

Other participants reported having taken advantage of the opportunity presented by the mailed national bowel screening, allowing them to become involved actively in diagnosis. Participants who experienced diagnostic delays reported general practitioners (GPs) as attributing symptoms to conditions other than CRC, or did not consider bowel cancer due to the patient's relative youth. Comments suggest experiences of delayed or misdiagnosis creates a perception of longer or more complex treatment.I receive help from my current GP but if my previous GP had not ignored all the clear signs of my cancer that I presented with over a period of one year I would not have had to undergo all the treatment etc. that followed the eventual diagnosis. (Male age 55–64)

Absence of coordinated/integrated patient care

Algorithms identified 78 (4.6%) comments relating to lack of coordinated care. Comments described delays that had a detrimental impact on available treatment options, discontinuity between GP practices and the hospital at the end of treatment, ‘confusion’ regarding which health professionals had responsibility to provide certain aspects of on-going care, absent care plans and limited access to community-based clinicians. Thus, the emotional impact of cancer and its treatment were not resolved, and sometimes represented a source of increasing stress over time. However, a lack of coordination and communication between health professionals and respondents could lead to delays that had a detrimental impact on available treatment options:I was told by my surgeon that I would have a follow-up appointment 3/12 after operation and she would then refer me to an oncologist to see if I needed any [adjuvant] treatment. At that point the surgeon was sick and it was cancelled. 5/12 after the op after much agitation by my GP I was seen by another rectal cancer surgeon who assured me that I had been referred to the oncologist. Four days later I was seen by the oncologist who said he had had no referral and it was too late to think about treatment. (Female, age 55–64)

Algorithms failed to identify positive comments of coordinated care, suggesting few existed. Nevertheless, examples were found among positive comments that described good communication between staff from different departments across the treatment journey, often facilitated by a specialist nurse.The role of the colorectal nurse in providing on-going contact and reassurance is absolutely vital in helping patients understand their condition, what is happening to them and to bolster their morale before during and after surgery. (Male, age 44–55)

Lack of patient preparation for cancer and its treatment

Algorithms identified comments in three separate subcategories relating to inadequate patient preparation. Some participants reported insufficient information concerning treatment side-effects (n=160 comments, 9.4%), the possible psychological impact of cancer and treatment (43 comments, 2.5%) and potential self-management strategies (135 comments, 7.9%) to ameliorate these. Comments reporting inadequate patient preparation often described how this could exacerbate physical and emotional problems. For example, some participants reported being insufficiently informed about available cancer therapies, with some individuals subsequently uneasy about their treatment decisions.The one area which has given me major problems has been the severity of the discomfort I have experienced since the stoma reversal. More information in that area would have been very helpful. My quality of life has suffered more since the stoma reversal than at any other time since the initial surgery. Had I realised how severe the reaction would be I might have elected to retain the stoma. (Male, age 54–65)

Comments indicated that without preparation, the negative impact of emotional problems experienced following cancer treatment could be more severe as they were less equipped to deal with them. From among the positive comments, examples could be found where individuals reported having been prepared by services for potential problems, and were thus better able to cope with these challenges.

Restricted opportunities for emotional support

Algorithms identified 78 comments (4.6%) that described a lack of emotional support, including sign-posting to cancer support groups, talking therapies and counselling. The need for emotional support transcended the patient journey from diagnosis, through treatment and post-treatment. When support was not available emotional and psychological problems could worsen. Ultimately, as indicated in figure 2, if support needs were not addressed as they arose during treatment a greater need for care post-treatment might arise.I did and still do feel ‘abandoned’ following surgery and treatment for colon cancer. I appreciate that the oncology and surgical departments are very busy but I would have liked some form of counselling following discharge. The anxiety doesn't go away, it just gets worse. (Female, age 65–74)

Although the algorithms were unable to identify positive descriptions of patient preparation, among positive comments examples were found of participants who had been provided with emotional support as part of their care and reported its importance to their HRQoL.

Quality of care post-treatment

Algorithms identified subcategories of comments relating to both good (289 comments, 17.1%) and inadequate post-treatment care (191 comments, 11.3%). Retrieved comments related primarily to accessibility of health professionals after finishing treatment, but also regular monitoring and follow-up consultations to detect recurrence. Both elements appeared to have a direct impact on HRQoL. Participants often contrasted ‘poor at best’ or ‘non-existent’ care following treatment with very good ‘hot house’ care during treatment.The effects of my treatment still affect me now but there is nowhere to go to access any support. This is so important as no one explained the emotional impact of living beyond bowel cancer after treatment or what side effects to expect of which there are many causing undue stress. A lot more support is needed for post cancer treatment. (Female, age 44–55)

When participants felt supported and monitored beyond the treatment phase they reported satisfaction and confidence in their care (eg, ‘My needs are met’ and ‘reassurance that problems will be dealt with’). Thus, psychological problems such as fears of cancer recurring were partly allayed and participants were able to ‘move on’/‘plan for the future’. However, such challenges were not addressed for those participants who felt abandoned by services due to a perceived lack of coordination between primary and secondary care. The lack of post-treatment support reported by some participants meant an added burden for carers.Would have liked more help and support after my surgery. Have been left to cope on my own. Would not have been able to live without my wife who took on everything. Not visited by district nurse on return from hospital. Lost records. Not called back for further check-up. Wife had to keep ringing. (Male, age 64–75)