Bean-counters, Lamp-posts and Rheumatologists

bean counterI was talking to some medical students recently about the use of outcome measures in rheumatology. I happened to mention that ‘big data’ was the next ‘big thing’ in medicine. Cue blank faces. Followed by quizzical looks and a sly smile. I suddenly realised to my chagrin that not everyone has read Topol’s ‘Creative Destruction of Medicine’ and that my students are beginning to doubt my sanity. I rapidly tried to regain some semblance of dignity by explaining that outcome measures are all the rage in rheumatology right now.

Readers of my blog may have not be surprised to hear that at times I can get a little obsessive about the science of measurement in medicine. So as a form of necessary self-therapy and reflection I am going to explore the darker side of measurement science using a couple of well-known quotations.

einsteinblackboard03The first quote, usually attributed to Albert Einstein (but possibly more accurately credited to a Professor of Sociology, William Bruce Cameron), is as follows:

“Not everything that can be counted counts, and not everything that counts can be counted.”

So not everything we measure or count actually matters. We need to be very selective before we decide that some measurement is essential or routine. There seems to be an assumption that the more we measure the more scientific our care will be and therefore the outcomes will improve. In the case of rheumatology there is now an assumption that if we have a patient with rheumatoid arthritis and we do not complete a ‘disease activity score’ and a disability assessment in every patient we are in some way negligent as doctors. Now I would be the first to argue that early arthritis trials have demonstrated the need to adopt a ‘treat to target’ approach using objective and quantifiable measures to gauge response – but to extrapolate this to patients with 30 years of arthritis may not be either scientific or kind to the person in front of us, particularly when they mention that they have had a recent bereavement. In times like these, ‘feeling the pain’ will often trump feeling the joints. Some measurements may be eminently ‘scientific’ but completely worthless, particularly in managing the individual.

The second part of this elegant chiasmus claims that not everything that counts can be counted. I hope that every health care worker is acutely aware of this fact, but it is at the core of an empathic approach to medicine. How the patient feels cannot easily be reduced to a score on a questionnaire. How well does the doctor understand the narrative from the patient’s perspective? Can we really quantify how well or badly the doctor or nurse empathises with their patient?  There is a feeling that recent changes in the organisation of medical work have disrupted the continuity of care and the integrity of the ‘team’ looking after the patient. Medical ‘clerk-ins’ are reduced to ticking boxes in lengthy proformas and medical handovers are reduced to a rapid fire exchange of facts and figures. Increasingly, doctors complain of being tied to the computer screen completing page after page of data, and having no time to have a proper talk with the patient.  maninfogIn the midst of this swirling fog of data, have we lost sight of the patient? When we teach our students the importance of rigorous data collection and outcome measurement, will we remember to refocus their attention back on the importance of empathy and the art of medicine?

The second quotation has been attributed to AE Houseman or Andrew Lang:

People Use Statistics as a Drunk Uses a Lamppost — For Support Rather Than Illumination

 

This is justifiably a popular quotation and any medical researcher worth his or her salt will doubtless be familiar with the dark arts of manipulating statistics to make the data seem to support your hypothesis. Reviewers of publications have become wise to this, and statistical scrutiny is a little more rigorous than it used to be. Even better, clinical trials registration requirements and ethics reviewers will insist on you defining the primary and secondary outcome measures before the study begins. This only goes to prove that nowadays you need a higher level of statistical jiggery-pokery’ to baffle your readers and ‘stay ahead of the game’.

 

dogLampost

Managers Use Statistics as a Dog Uses a Lamppost — For ‘Marking their territory’ Rather Than Illumination

 

The third quotation could be attributed to a certain PV Gardiner, the writer of this blog. He has clearly borrowed heavily from the afore-named Andrew Lang. He has also used polite and politically correct language within the quotes which readers may wish to replace with more descriptive and direct terminology.

I’ll tell you a story to illustrate the point. A few years ago we were told that we were obliged to carry out an ‘audit’ on our use of biologic drugs in the region. Nothing unusual with that, you may say – they are very expensive drugs, after all. However, we did detect quite a hostile and accusatory sub-text to the discussions. After a little ‘digging’ we discovered that somebody  in management ‘on-high’ had ‘discovered’ that we were using biologic drugs twice as much as the average in the UK. We thought we were using the same NICE guidelines to treat similar patients, so this seemed odd. Cue some anxious head-scratching and navel-gazing by the chastened rheumatologists. Strangely, our own figures hadn’t seemed to suggest any problem at all. Eventually, we discovered that someone in management had been using a single figure (the prevalence of RA in our region) that had changed everything. This figure suggested that RA was only half as common as in the rest of the UK. We were most intrigued. Could it be something in the pure Irish spring water? Could it be our fondness for Guinness or Bushmills whiskey? How had they arrived at such a figure, when there had never been a study of the epidemiology of RA in our region? We have never had a diagnostic database to record the prevalence of RA. In fact, they had obtained the figure from a reference found in none other than the official National Audit Office report into RA in England. That article had briefly referred to comparative data in other nations, and the quoted statistic actually came from a paragraph in a local newspaper where a journalist had mentioned in passing the estimated number of people with RA in our region. Well, it is just as well that we discovered the source of these scurrilous rumours. Otherwise the aforementioned honourable rheumatologists might well have wound up propping up the local lamp-posts (or was that the other way round?). We live to fight another day…

 

12 thoughts on “Bean-counters, Lamp-posts and Rheumatologists

    1. Perhaps we should reintroduce the teaching of classical rhetoric! People like William Osler had such a great way of summarising great insights in memorable sayings.

  1. Great blog Philip. You have managed to articulate something that has been niggling away at me for years – the idea that ‘feeling the pain’ may be better than ‘feeling the joints’. There is so much information that we can, as physicians, decode from our patient stories that cannot be captured using DAS 28 scores, HAQ scores etc. Listening, really listening is one of the best outcome measures we have. Not that you shouldn’t tick a few boxes as well of course.

    1. Thanks for your comments, Ronan. I do find it a tension sometimes, so I definitely treasure the occasional quiet clinic when there is more time to talk & teach.

  2. Phillip this was a great perspective on a very important issue. I started iRheum specifically for the purpose of helping Rheumatologists improve patient outcomes by using quantifiable metrics to treat to a goal. However I have never believed that there is a perfect metric that fits all auto-immune patients.

    There should always be a certain amount of art associated with the diagnosis because of the complexity of diagnosing something that normally cannot be seen. As a non-Rheumatologist myself, I alway appreciated Dr. Jim Louie’s presentations that always led with a painting of Monet emphasizing the Rheumatologist is truly an artist.

    With today’s expensive biologics, the payers are looking for justification in documented form. Every Rheumatologist I’ve ever worked with can diagnose RA with 99.99% accuracy within the first 5 minutes of meeting the patient without the expensive tests. But insurance companies no longer pay for diagnosis by Gestalt.

    So how can Rheumatologists maintain control of their profession while at the same time appease the bean counters?

    Working with the National Rheumatic Databank and the RAIN network I have realized the importance of using outcome measures to treat to a goal but not setting a specific severity level tied to reimbursement. It should be about apple to apple comparison for each individual patient, not an insurance company approving (or denying) because the patient was a DAS28 3V ESR of 3.0. But having some type of measurement, using any outcome measure consistently, tracked over time consistently is a very powerful tool to educate the patient and fine tune biologic therapy.

    Rheumatologists do not have a set treatment algorithm (like cardiology) and therefore the patients do not have a standard outcome measure to review and track to improve their health. Every diabetic patient knows they need to track and measure their blood sugar levels and A1C, but there is nothing like that in RA. But that’s okay because each patient must be treated individually.

    The best advice I’ve ever heard was “It doesn’t matter which outcome measure you use, just use the same one consistently on every visit.” I still believe that is the case and I hope that tracking the same metric over time for each individual patient will ultimately improve patient care by allowing the Rheumatologist to treat to a goal, not to a checkbox on an insurance form.

    Thank you again for the blog post!

    Scott Pribyl
    http://www.iRheum.com

    1. Scott,

      Thanks for your comment. I have admired the work of the RAIN investigator network and the National Rheumatology Databank for some time, as it has led to a lot of useful research about treatment strategies. We would love to set up a similar databank in our area. Your iRheum app for calculating DAS28 in the clinic is excellent & I hope I haven’t given the impression that I don’t support the use of these scores in the clinic. I was teaching a group of young students how to do a DAS28 and HAQ in a special study module last week: they learnt a lot from comparing these outcome measures with what they gleaned from a semi-structured interview.
      I know that most rheumatologists are aware of the limitations of outcome measures, but the same is not always true of our funders.
      In the UK, the rules of NICE (www.nice.org.uk) dictate that no patient with RA can be treated with biologics unless their DAS28 ESR is greater than 5.1 on two occasions one month apart. On the one hand this permits access to people with mild RA + FMS, on the other biologic treatment is denied to some very needy patients with lots of swollen joints but a relatively low ESR. I know that NICE are trying to be fair, but when the cut-off in the rest of the EU is a DAS28 ESR of 3.2 it doesn’t always work out that way for our patients. I do think we are obliged to speak out and raise awareness of these issues.

      Philip

  3. As a bean counter with a 30 year odd HLAB27 associated spondylo arthropathy on anti TNF’s (I just tell people it’s a RHA variant, makes it easier all around) I really enjoyed reading this post which makes lots of great points. Another quote, I might add is “what get’s measured, get’s managed” and good management should lead to improvements all round.
    For example, with very rare exceptions, any mental health support I got to manage the psychological impacts of a chronic disease was initiated and paid for by me. Yet, my mental health is key to my physical health and ability to manage.
    A King’s fund report shows a saving of £1 sterling in every £8 health spend when integrated health care treatments are provided. According to US research “Co morbidity between medical and mental conditions is the rule rather than the exception… adding an $560 additional cost per month to treating physical illness”
    So if the healthcare team were assessing the psychological impact of rheumatology disease on patients and organising appropriate interventions there would be better clinical outcomes and reduced costs. And yes, that does mean more listening.

    But the real issue is the different bean counters, funders, clinicians, patients need to collaborate together constructively and agree what it is important to measure both within services from different stakeholder perspectives and between different services. To the surprise of my own Rheumatologist, in some recent research he did, fatigue not pain was the real issue for patients, so just measuring pain consistently was not good enough. I’d certainly trade using more painkillers for extra energy.

    There also needs to be agreement on what these measures mean. For example I’ve often participated in research studies which ask about difficulties with activities of daily living. I used to answer, very little difficulty, because I actually have very little difficulty- but only because of the adaptations I’ve made years ago. For example moving to a bungalow.

    My experience indicates collaborative teamwork between all the different professional groups could improve. Not just with the bean counters & admin staff (pls, pls stop scheduling 10 patients for the one appointment time!) but even between doctors of different specialities and other health care professionals. For example, called to do physio after surgery, luckily checked in with surgeon and turned out it, was contra indicated due to type of surgery for another 6 weeks. Measuring teamwork (loads of tools available to do this) would help reduce this type of problem for patients.

    No one measure captures everything and as you say some things can’t be measured, but healthcare would benefit from a more rigorous approach to measure and agree what’s important to different stakeholders. In a progressive healthcare organisation, bean counters and their HR counterparts work constructively with all stakeholders to deliver a management information system that measures the important agreed measurements on a timely basis while recognising that not everything that’s important can be measured scientifically.

    1. Carol,
      Many thanks for sharing your valuable insights. I think that your comment should be elevated to a ‘guest post’ if you don’t mind – it certainly deserves to be read by a wider audience. One of the huge frustrations of working in the NHS is how hard it is to get funding for interventions that do not appear to have any immediate or measurable effect. Your examples of fatigue and psychological issues are just the issues I have been thinking about recently, and I hope to have another relevant article coming up very soon that touches on this. It seems that almost everyone in the hospital can send their patients to see a psychologist except rheumatologists (and orthopaedic specialists, to be fair). And getting integrated team work with physiotherapy and occupational therapy is also difficult. I think that if we can measure actual mobility before and after these interventions we can make a stronger case for funding. The stark reality of outpatient rheumatology and orthopaedic clinics is that the time pressure often squeezes out deeper discussions about psychology and participation: we need an integrated team to be involved to help draw these issues out and personalise the care that we give.
      My next post will also mention some of the discrepancy between disability questionnaires and what they actually mean: you seem to have been reading my thoughts. I am also getting around to posting about some specific research studies in this area, one which deals with Ankylosing Spondylitis. I’d better get down to it, then.
      I don’t know much about tools to measure teamwork – perhaps you could expand a little on that one? If the ‘team’ is not actually a team there is a problem. The occupational therapists and physiotherapists we work with are under separate management. They generally do a great job but it is not the same as an integrated team. I know rheumatologists in Ireland who were allocated budgets to set up an integrated team which I believe to be the best way forward. Irwin Lim’s ‘Connected Care’ model in Australia is also one I admire. By the way, there is some great research going on in this area at the moment by physiotherapists and occupational therapists in University College Dublin and the University of Ulster. Again, more of that to come.
      Thanks again for your comment.

      1. Hi Phil,
        Thanks for the lovely feedback, yes, would be very happy for the comment to be a guest post.
        Yes, clinical practise is improving down South are improving, and I’ve been at and enjoyed some of the lectures, but waiting lists and the pressure doc’s are working under are getting worse. You mentioned measuring mobility to justify additional interventions, but the health economists tend to use QUALY’s ( quality adjusted life years) for health resource allocation so it may be useful to use this as well.
        Regarding measuring team working, it can be a very scientific exercise, based on an overall organisation and human resource strategy to encourage certain types of behaviours. See the gold standard SHL (a leading psychometric supplier) framework here.http://www.assessmentanalytics.com/wp-content/uploads/2013/08/White-Paper-SHL-Universal-Competency-Framework.pdf
        So as part of this you measure people’s likely ways of behaving in a team and start encouraging these behaviours through selection, development and performance management.

        Or you can do a quicker more simplistic exercise with the team- whether virtual or real
        • By agreeing a team goal
        • Agreeing roles and accountability
        • Agree how the team are going to provide feedback to each other (survey monkey is a great free online tool)
        • What the team are going to feedback on
        • How the team processes will operate- meetings,emails, attendance, etc
        • Agree baseline assessment for the team as it currently operates
        There are a lot of proprietary tools which help with this, but this link here is a short free tool- though I’ve never used this it.http://www.ee.uidaho.edu/ee/power/jlaw/COURSES/CAPSTONE/F05/handouts/MeasureTeamPerformance.pdf

        From following NHS commentary, our own problems down here with UHI and the new under 6 GP’s contract and the comments on your own blog, trust seems to be in short supply in health care systems. As you know no matter how scientific/rigorous the codes, tools, systems etc if the person using them does not have personal integrity and inspire trust the tools etc are useless. So I see trust at all levels as vital in delivering better health care.
        Which is why I like Dr Jacky’s Jones work on measuring interagency trust. See http://www.ncbi.nlm.nih.gov/pubmed/21339194
        This free tool could easily be adapted by virtual teams drawn from different departments. It uses a 5 digit likert scale to get people to assess trust examples include such items as
        Partners can be relied on to do what they say they will do for the partnership
        Partners’ time and energy are wasted due to mistrust

        Have signed up for Dr Lim’s blog.Thanks.

        Carol

        1. Carol,
          Thanks again for all the suggestions which are very pertinent and helpful. I will take time to read over these and perhaps I should get back to you by email.
          Philip