Automated epistemic objectification in healthcare

The provision of healthcare is an intrinsically cooperative enterprise in which patients’ active engagement is crucial. It is paramount to recognize the value of patients' epistemic offerings in medical encounters and that their testimony is central to providing healthcare professionals epistemic access to their health state. These are fundamental preconditions to finding the most suitable course of medical action tailored to patients' needs in their singularity. 

However, the ever-growing body of literature tackling epistemic injustice in medicine and healthcare shows that identity prejudices often diminish patients’ credibility, thus hampering their possibility to play their role as epistemic subjects in medical encounters. Often, this leads to misrecognition of patients' epistemic status by, among others, hastily dismissing their testimony as irrelevant, thus leading to epistemic and practical harm. These situations are already highly problematic in healthcare encounters among human patients and physicians.

The introduction of artificial intelligence-based systems, such as machine learning (ML), in medical practice shows that medical care is no longer an exclusively human domain. So, what happens when an ML system, as an allegedly objective epistemic authority and powerful knowledge-generating entity, enters the picture by considerably influencing central medical procedures? As we argue in a recently published paper, we must be wary of ML systems' roles in crucial medical practices, such as providing treatment recommendations and diagnoses. More precisely, we advance the claim that ML systems can epistemically objectify patients in subtle but potentially extremely harmful ways that must be treated in their own right. In fact, these harms are not to be subsumed under other ethical concerns currently receiving extensive attention in the AI Ethics debate. 

Our paper discusses the hypothetical case of an ML system generating a treatment recommendation that goes against a patient's values. In similar situations, the patient is confronted with an unsuitable course of action. We argue that the patient risks being epistemically objectified if the system cannot pick up on their values and produce a new recommendation aligned with their expectations. This is the case because, crucially, when the patient needs to actively provide a piece of information relevant to further course of action, they are prevented from effectively doing so due to the system's setup. The bottom line is that the patient's status in this knowledge-producing endeavor may be degraded to merely being an object of medical action. 

Against these considerations, one could argue that a human physician could simply neutralize the recommendation produced by ML if it goes against the patient's values. However, we see this move as too simplistic for two main reasons. First, physicians may be epistemically dependent on an ML system and de facto unable to override it. Even though this is a surely undesirable scenario, it is recognizable in systems currently deployed. Second, disregarding ML systems' recommendations does not guarantee that physicians can readily find a further, more appropriate course of action. Rather, they might benefit from an ML system supporting their decision-making.

Overall, our general plea in the paper is thus to show the need for a flexible ML epistemology that can incorporate and adapt to newly acquired and ethically relevant information that patients can actively provide. This is key to avoid objectifying patients in morally salient medical interactions. 

Dr. Giorgia Pozzi is a postdoctoral researcher at Delft University of Technology (The Netherlands), working on the intersection between the ethics and epistemology of artificial intelligence (AI) in medicine and healthcare. She has a particular interest in tackling forms of epistemic injustice emerging due to the integration of AI systems in medical practice.

The Patient Experience Library

In this post, Havi Carel interviews Miles Sibley about the Patient Experience Library.

Havi Carel (HC): What is the Patient Experience Library? 

Miles Sibley: The Patient Experience Library acts as the national evidence base for patient experience and engagement. Operating as an online research database it contains studies and reports from sources including government bodies, patient voice organisations, health charities, academic institutions and policy think tanks. The repository runs to tens of thousands of documents, and more are being added all the time.  So the library also works on knowledge translation – helping policymakers, clinicians and patient advocates to understand what is in the research, and how to use it.  

HC: Why did you set it up? 

MS: We set up the library after realising that an NHS aiming to be both patient-centred and evidence-based actually had no coherent evidence base for patient experience.  Health services draw on multiple medical research databases to inform policy, professional training, clinical guidelines and more.  But there was no equivalent for patient experience.  So we built the library to plug the gap.   

HC: What are the barriers to accessing data on patient experience?

MS: Firstly, the data is a mess.  It comes from the Friends and Family Test, complaints, national patient surveys, the local Healthwatch network… All this and more is published across multiple websites and can be very hard to keep track of. 

Secondly, there is no organisational memory.  Government funded “patient voice” initiatives began with the Community Health Councils in the 1970s. Those were succeeded by Patient and Public Involvement Forums, then by Local Involvement Networks, before the current iteration, Healthwatch.  All gathered large volumes of evidence on patient experience, but it was never properly catalogued and preserved. So as each initiative shut down, its entire body of knowledge went with it.

Thirdly, there is no research prioritisation at the national or strategic level.  While medical research is (at least to some extent) geared towards reducing duplication and opening up new areas of knowledge, patient experience research is, essentially, a free-for-all.  There is extensive waste and duplication, along with large gaps – particularly when it comes to marginalised groups such as minority communities, or people with rare diseases.

One of the posters
by the Patient Experience Library

HC: Why do you think it's important to overcome these?

MS: For reasons of equity and justice.  

For example, the NHS promises equal access to its services for all citizens.  But the reality is that the poorest people in the poorest communities always have the poorest health, and usually have the worst access to services.  Statistics on mortality and morbidity can help us to understand some of the consequences.  But to really understand health inequity, we need to hear from the people most affected by it.

Another example:  inquiries into avoidable harm (eg from pelvic mesh, or deaths in maternity services) always tell of patient concerns being deflected and denied.  Medical records are respected and trusted, while patient feedback is dismissed as "anecdotal evidence".  This deep-seated epistemic injustice lies at the heart of harm.

One more: medicine has a fantastic learning infrastructure.  Clinicians are given world class learning support though professional training, clinical guidelines, practice protocols, diagnostic aids, conferences and journals.  But people charged with hearing "patient voice" (PALS teams, complaints handlers etc) have no professional qualifications, no analytics, and no professional forums.  Their learning support is deemed unimportant because lived experience is also deemed unimportant.

HC: What do you hope the patient experience library will achieve?

MS: We see a double standard in evidence-based practice: medical evidence is cherished and respected, while patient experience evidence is considered an optional extra.

So we want to end the double standard. Our steps towards that include:

• Building the evidence base. The library is an open access repository of evidence on patient experience: the only one of its kind in the world. Medical research databases are taken for granted – we want to show that a patient experience research database is equally feasible.
• Building analytics.  Patient experience evidence doesn't have to be incomprehensible.  Our surveys tracker, for example, puts key datasets for English NHS Trusts all in one place and cross-references key themes to aid understanding.
• Mapping the evidence.  Patient experience research should not be a free-for-all, with some communities permanently excluded.  Our evidence mapping shows that it is possible to lay the foundations for better prioritisation and higher value research.

Ultimately, we are on a mission to get patient experience evidence taken as seriously as medical evidence.  We are showing how that can be done.  Our work is both a challenge and a helping hand to an NHS that says it wants to be patient-centred.

Miles Sibley is a co-founder and Director of the Patient Experience Library. As an advocate for healthcare services that are both person-centred and evidence-based, he has served on the Board of HQIP, and is currently a Non-Executive Director with Care Opinion. He is the author of Inadmissible Evidence and has written numerous articles on patient and public involvement for the BMJ and others.