Table Of ContentBeyond the EULA: Improving consent for data
mining
LukeHuttonandTristanHenderson
7
1
0
2
n
a
J
7 Abstract Companiesandacademicresearchersmaycollect,process,anddistribute
2
large quantities of personal data without the explicit knowledge or consent of the
individuals to whom the data pertains. Existing forms of consent often fail to be
]
Y appropriatelyreadable and ethical oversightof data mining may not be sufficient.
C Thisraises the questionof whetherexistingconsentinstrumentsare sufficient, lo-
gistically feasible, or even necessary, for data mining. In this chapter, we review
.
s
the datacollectionandmininglandscape,includingcommercialand academicac-
c
[ tivities,andtherelevantdataprotectionconcerns,todeterminethetypesofconsent
instrumentsused.Usingthreecasestudies,weusethenewparadigmofhuman-data
1
interactiontoexaminewhethertheseexistingapproachesareappropriate.We then
v
9 introduceanapproachtoconsentthathasbeenempiricallydemonstratedtoimprove
9 onthestateoftheartanddelivermeaningfulconsent.Finally,weproposesomebest
9 practicesfordatacollectorstoensuretheirdataminingactivitiesdonotviolatethe
7
expectationsofthepeopletowhomthedatarelate.
0
.
1
0
7 1 Introduction
1
:
v
Theability ofcompaniesto collect,process,anddistributelargequantitiesofper-
i
X sonaldata,andtofurtheranalyse,mineandgeneratenewdatabasedoninferences
r fromthesedata,isoftendonewithouttheexplicitknowledgeorconsentofthein-
a
dividuals to whom the data pertains. Consent instruments such as privacy notices
or End User License Agreements(EULAs) are widely deployed,often presenting
LukeHutton
CentreforResearchinComputing,TheOpenUniversity,MiltonKeynes,MK76AA,UKe-mail:
[email protected]
TristanHenderson
School of Computer Science, University of St Andrews, St Andrews KY16 9SX, UK e-mail:
[email protected]
1
2 LukeHuttonandTristanHenderson
individualswiththousandsofwordsoflegaljargonthattheymaynotreadnorcom-
prehend,beforesolicitingagreementinordertomakeuseofaservice.Indeed,even
if an individual does have a reasonable understandingof the terms to which they
have agreed,such terms are oftencarefullydesignedto extendas muchflexibility
to the data collector aspossible to obtain evenmoredata, distributethem to more
stakeholders,andmakeinferencesbylinkingdatafrommultiplesources,despiteno
obviousagreementtothesenewpractices.
The lack of transparencybehind data collection and mining practicesthreatens
theagencyandprivacyofdatasubjects,withnopracticalwaytocontroltheseinvisi-
bledataflows,norcorrectmisinformationorinaccurateandinappropriateinferences
derivedfromlinkeddata.Existingdataprotectionregimesareofteninsufficientas
they are predicated on the assumption that an individual is able to detect when a
dataprotectionviolationhasoccurredinordertodemandrecourse,whichisrarely
thecasewhendataareopaquelyminedatscale.
Thesechallengesarenotuniquetocommercialactivities,however.Academicre-
searchersoftenmakeuseofdatasetscontainingpersonalinformation,suchasthose
collectedfromsocialnetworksitesordevicessuchasmobilephonesorfitnesstrack-
ers. Most researchersare boundby an obligationto seek ethicalapprovalfroman
institutionalreviewboard(IRB)beforeconductingtheirresearch.Theethicalpro-
tocols used, however, are inherited from post-war concerns regarding biomedical
experiments,andmaynotbeappropriateforInternet-mediatedresearch,wheremil-
lionsofdatapointscanbecollectedwithoutanypersonalinterventions.Thisraises
thequestionofwhetherexistingconsentinstrumentsaresufficient,logisticallyfea-
sible,orevennecessary,forresearchofthisnature.
Inthischapterwefirstreviewthedatacollectionandmininglandscape,includ-
ing commercial and academic activities, and the relevant data protection laws, to
determine the types of consent instruments used. Employing the newly-proposed
paradigmof Human-DataInteraction,we examinethree case studiesto determine
whetherthesemechanismsaresufficienttoupholdtheexpectationsofindividuals,to
providethemwithsufficientagency,legibilityandnegotiability,andwhetherprivacy
normsareviolatedbysecondaryusesofdatawhicharenotexplicitlysanctionedby
individuals. We then discuss various new dynamic and contextual approaches to
consent,whichhavebeenempiricallydemonstratedto improveonthe state ofthe
art and deliver meaningful consent. Finally, we propose some best practices that
data collectors can adopt to ensure their data mining activities do not violate the
expectationsofthepeopletowhomthedatarelate.
2 Background
Data mining is the statistical analysis of large-scale datasets to extract additional
patternsandtrends[16]. Thishasallowed commercial,state, andacademic actors
to answer questions which have not previously been possible, due to insufficient
data, analytical techniques, or computationalpower. Data mining is often charac-
BeyondtheEULA:Improvingconsentfordatamining 3
terised by the use of aggregate data to identify traits and trends which allow the
identificationandcharacterisationofclustersofpeopleratherthanindividuals,as-
sociations between events, and forecasting of future events. As such, it has been
used in a number of real-world scenarios such as optimising the layout of retail
stores, attempts to identify disease trends, and mass surveillance. Many classical
data mining and knowledge discovery applications involve businesses or market-
ing [12], such as clustering consumersinto groupsand attempting to predicttheir
behaviour.Thismayallowabusinesstounderstandtheircustomersandtargetpro-
motionsappropriately.Suchprofilingcan,however,beusedtocharacteriseindivid-
uals for the purposeof denyingservice when extendingcredit, leasing a property,
oracquiringinsurance.Insuchcases,thecollectionandprocessingofsensitivedata
canbeinvasive,withsignificantimplicationsfortheindividual,particularlywhere
decisionsaremadeonthebasisofinferencesthatmaynotbeaccurate,andtowhich
the individualis given no right of reply. This has become more importantof late,
asmorerecentdataminingapplicationsinvolvetheanalysisofpersonaldata,much
ofwhichiscollectedbyindividualsandcontributedtomarketersinwhathasbeen
termed “self-surveillance”[25]. Such personaldata have been demonstratedto be
highlyvaluable[48],andhaveevenbeendescribedasthenew“oil”intermsofthe
value of their resource [55]. Value aside, such data introduce new challenges for
consentastheycanoftenbecombinedtocreatenewinferencesandprofileswhere
previouslydatawouldhavebeenabsent[17].
Table1 SomerelevantEUlegislationthatmayapplytodataminingactivities
Legislation Somerelevantsections
DataProtectionDirective95/46/EC Dataprocessing(Art1),fairprocessing(Art6(1)),pur-
pose limitation(Art 6(2)), proportionality (Art 6(3)),
consent(Art7(1)),sensitivedata(Art8)
E-Privacy Directive 2002/58/EC as cookies(Art5),trafficdata(Art6),locationdata(Art
amendedby2009/136/EC 9),unsolicitedcommunication(Art13)
General Data Protection Regulation Consent(Art7),righttobeforgotten(Art17),rightto
2016/679 explanation(Art22),privacybydesign(Art25)
Data mining activities are legitimised through a combinationof legal and self-
regulatorybehaviours.In the EuropeanUnion,the Data ProtectionDirective [10],
and the forthcoming General Data Protection Regulation (GDPR) that will suc-
ceed it in 2018 [11] govern how data mining can be conducted legitimately. The
e-PrivacyDirectivealsofurtherregulatessomespecificaspectsofdataminingsuch
as cookies (Table 1). In the United States, a self-regulatoryapproach is generally
preferred,withtheFederalTradeCommissionofferingguidanceregardingprivacy
protections[42],consistingofsixcoreprinciples,butlackingthecoverageorlegal
backingoftheEU’sapproach.
UndertheGDPR,theprocessingofpersonaldataforanypurpose,includingdata
mining,is subjectto explicitopt-inconsentfromanindividual,priortowhichthe
datacontrollermustexplicitlystatewhatdataarecollected,thepurposeofprocess-
4 LukeHuttonandTristanHenderson
ingthem,andtheidentityofanyotherrecipientsofthedata.Althoughtherearea
numberof exceptions,consentmust generallybe soughtforindividualprocessing
activities, and cannot be broadly acquired a priori for undefined future uses, and
there are particular issues with data mining, transparency and accountability [7].
Solove [47] acknowledges these regulatory challenges, arguing that paternalistic
approachesarenotappropriate,asthesedenypeoplethefreedomtoconsenttopar-
ticularbeneficialusesoftheirdata.Thetimingofconsentrequestsandthefocusof
theserequestsneedtobemanagedcarefully;suchthinkinghasalsobecomeappar-
ent in the GDPR.1 The call for dynamic consent is consistent with Nissenbaum’s
modelofcontextualintegrity[39],whichpositsthatallinformationexchangesare
subject to context-specific norms, which governs to whom and for what purpose
informationsharingcanbeconsideredappropriate.Whenthecontextisdisrupted,
perhapsbychangingwithwhomdataareshared,orforwhatpurpose,privacyvio-
lationscanoccurwhenthisisnotconsistentwiththenormsoftheexistingcontext.
Therefore, consent can help to uphold contextual integrity by ensuring that if the
contextisperturbed,consentisrenegotiated,ratherthanassumed.
Reasoning about how personal data are used has resulted in a new paradigm,
human-datainteraction,whichplaceshumansatthe centreofdataflowsand pro-
videsaframeworkforstudyingpersonaldatacollectionanduseaccordingtothree
themes[35]:
• Legibility:Often,dataownersarenotawarethatdataminingiseventakingplace.
Even if they are, they may not know what is being collected or analysed, the
purposeoftheanalysis,ortheinsightsderivedfromit.
• Agency: The opaque nature of data mining often denies data owners agency.
Withoutanyengagementinthepractice,peoplehavenoabilitytoprovidemean-
ingfulconsent,iftheyareaskedtogiveconsentatall,norcorrectflaweddataor
reviewinferencesmadebasedontheirdata.
• Negotiability:The context in which data are collected and processed can often
change,whetherthroughanevolvinglegislativelandscape,databeingtradedbe-
tween organisations, or through companies unilaterally changing their privacy
policies or practices. Analysis can be based on the linking of datasets derived
fromdifferentstakeholders,allowinginsightsthatnosingleprovidercouldmake.
Thisisroutinelythecaseinprofilingactivitiessuchascreditscoring.Evenwhere
individualsattempttoobfuscatetheirdatatosubvertthispractice,itisoftenpos-
sible to re-identify them from such linked data [38]. Data owners should have
theabilitytoreviewhowtheirdataareusedascircumstanceschangeinorderto
upholdcontextualintegrity.
Earlydataprotectionregulationinthe1980saddressedtheincreaseinelectronic
datastorageandstrengthenedprotectionsagainstunsoliciteddirectmarketing[49].
Mail order companieswere able to developlarge databasesof customerdetails to
enable direct marketing, or the trading of such information between companies.
1 e.g., Article 7(3) which allows consent to be withdrawn, and Article 17 on the “right to be
forgotten”whichallowsinferencesanddatatobeerased.
BeyondtheEULA:Improvingconsentfordatamining 5
Whenacquiringconsentfortheprocessingofsuchinformationbecamemandatory,
suchasunderthe1984DataProtectionActintheUK,thisgenerallytooktheform
ofacheckboxonpaperforms,whereapotentialcustomercouldindicatetheirwill-
ingness for secondary processing of their data. As technology has evolved away
frommail-informsbeingtheprimarymeansofacquiringpersonalinformation,and
the scope and intent of data protectionmovesfrom regulatingdirect marketingto
a vast rangeof data-processingactivities, there hasbeen little regulatoryattention
paid to how consent is acquired. As such, consent is often acquired by asking a
usertotickacheckboxtoopt-inoroutofsecondaryuseoftheirdata.Thispractice
is well-entrenched,where people are routinely asked to agree to an End-User Li-
cenceAgreement(EULA)beforeaccessingsoftware,andmultipletermsofservice
andprivacypoliciesbeforeaccessingonlineservices,generallyconsistingofalong
legalagreementandan“IAgree”button.
Asignificantbodyofresearchconcludesthatsuchapproachestoacquiringcon-
sent are flawed. Luger et al. find that the terms and conditionsprovidedby major
energy companies are not sufficiently readable, excluding many from being able
to make informed decisions about whether they agree to such terms [29]. Indeed,
Obar and Oeldorf-Hirsch find that the vast majority of people do not even read
such documents[40], with all participantsin a user study accepting terms includ-
ing handingovertheirfirst-bornchildto use a socialnetworksite. McDonaldand
Cranormeasure the economiccost of readinglengthypolicies[31], notingthe in-
equityofexpectingpeopletospendanaverageoftenminutesoftheirtimereading
andcomprehendingacomplexdocumentinordertouseaservice.Freidmanetal.
cautionthatsimplyincludingmoreinformationandmorefrequentconsentinterven-
tionscanbecounter-productive,byfrustratingpeopleandleadingthemtomaking
morecomplacentconsentdecisions[13].
Academic data mining is subject to a different regulatory regime, with fewer
constraintsoverthesecondaryuseofdatafromadataprotectionperspective.This
isbalancedbyanethicalreviewregime,rootedinpost-warconcernoveralackof
ethicalrigourinbiomedicalresearch.IntheUS,ethicalreviewforhumansubjects
researchviaaninstitutionalreviewboard(IRB)isnecessarytoreceivefederalfund-
ing,andthesituationissimilarinmanyothercountries.Oneofthecentraltenetsof
ethicalhumanresearchistoacquireinformedconsentbeforeastudybegins[5].As
such,institutionshavedevelopedlargelystandardisedconsentinstruments[1]which
researcherscanusetomeettheserequirements.Whileintraditionallab-basedstud-
ies, these consent procedurescan be accompanied by an explanation of the study
fromaresearcher,ortheopportunityforaparticipanttoaskanyquestions,thisaf-
fordanceis generallynotavailablein onlinecontexts,effectivelyregressingto the
flawedEULAsdiscussedearlier.
Some of these weaknesseshave been examinedin the literature. Hamnes et al.
findthatconsentdocumentsinrheumatologicalstudiesarenotsufficientlyreadable
for the majorityof the population[15], a findingwhich is supportedby Vucˇemilo
and Borovecˇki who also find that medical consent forms often exclude important
information [53]. Donovan-Kicken et al. examine the sources of confusion when
reviewing such documents [8], which include insufficient discussion of risk and
6 LukeHuttonandTristanHenderson
lengthyor overlycomplexlanguage.Munteanuet al. examinethe ethics approval
process in a number of HCI research case studies, finding that participants often
agreedtoconsentinstrumentstheyhavenotreadorunderstood,andtherigidityof
suchprocessescanoftenbeatoddswithsuchstudieswherea“situationalinterpre-
tation”ofanagreedprotocolisneeded[36].Therealsolacksagreementamongre-
searchersabouthowtoconductsuchresearchinanethicalmanner,withVitaketal.
finding particular variability regarding whether data should be collected at large
scalewithoutconsent,orifacquiringconsentinsuchcasesisevenpossible[52].
Existing means of acquiring consent are inherited from a time when the scope
ofdatacollectionandprocessingwasperhapsconstrainedandcouldbewellunder-
stood.Now,evenwhenthetermsofdatacollectionandprocessingareunderstood
as written, whether registering for an online service, or participating in academic
research, it is not clear that the form of gaining the consent was meaningful, or
sufficient. Someone may provide consent to secondary use of their data, without
knowingwhatdatathisconstitutes,whowillbeabletoacquireit,forwhatpurpose,
orwhen.Thisisalreadyaconcernwhenconsideringtheredistributionandprocess-
ingofself-disclosedpersonallyidentifiableinformation,butbecomesincreasingly
complexwhenextendedto historicallocationdata,shoppingbehaviours,orsocial
network data, much of which are not directly provided by the individual, and are
nebulous in scale and content. Moreoverconcerns may change over time (the so-
called“privacyparadox”[3]thathasbeendemonstratedempirically[2,4]),which
mayrequirechangestopreviously-grantedconsent.
Returningtoourthreethemesoflegibility,agency,andnegotiability,wecansee
that:
• ExistingEULAsandconsentformsmaynotmeetabasicstandardoflegibility,
alienatingsignificantareasof the populationfromunderstandingwhattheyare
being asked to agree to. Furthermore, the specific secondary uses of their data
areoftennotexplained.
• EULAsand consentformsare often onlyused to secure permission once,then
oftenneveragain,denyingpeopleagencytorevoketheirconsentwhenamaterial
changeinhowtheirdataareusedarises.
• Individuals have no power to meaningfully negotiate how their data are used,
nortointelligentlyadoptprivacy-preservingbehaviours,astheygenerallydonot
knowwhichdataattributedtothemispotentiallyrisky.
3 Casestudies
Inthissectionweexamineanumberofreal-worldcasestudiestoidentifyinstances
where insufficient consent mechanisms were employed, failing to provide people
withlegibility,agency,andnegotiability.
BeyondtheEULA:Improvingconsentfordatamining 7
3.1 Taste,Ties, and Time
In2006,researchersatHarvardUniversitycollectedadatasetofFacebookprofiles
fromacohortofundergraduatestudents,named“Tastes,TiesandTime”(T3)[28].
At the time, Facebook considered individual universities to comprise networks
wheremembersoftheinstitutioncouldaccessthefullcontentofeachother’spro-
files, despitenothavinganexplicitfriendshipwitheachotherontheservice.This
designwasexploitedbyhavingresearchassistantsatthesameinstitutionmanually
extracttheprofilesofeachmemberofthecohort.
Subsequently,ananonymisedversionofthedatasetwasmadepubliclyavailable,
withstudentnamesandidentifiersremoved,andtermsandconditionsfordownload-
ingthedatasetmadeitclearthatdeanonymisationwasnotpermitted.Unfortunately,
thisprovedinsufficient,withaggregatestatisticssuchasthesizeofthecohortmak-
ing it possible to infer the college the dataset was derived from, and as some de-
mographic attributes were only represented by a single student, it was likely that
individualscouldbeidentified[56].
Individualswerenotawarethatthedatacollectiontookplace,anddidnotconsent
toitscollection,processing,norsubsequentrelease.Assuch,thiscasefallsshortin
ourthemesforacceptabledata-handlingpractices:
• Legibility:Individualswerenotawaretheirdatawascollectedorsubsequently
released.Withatangibleriskofindividualsbeingidentifiedwithouttheirknowl-
edge, the individual is not in a position to explore any legal remedies to hold
Facebook or the researchers responsible for any resulting harms. In addition,
even if consent were sought, it can be difficult for individualsto conceptualise
exactly which of their data would be included, considering the large numbers
of photos, location traces, status updates, and biographical information a typi-
caluser mightaccrueoveryears,withoutanaccessible meansofvisualisingor
selectivelydisclosingthesedata.
• Agency:Withoutnotification,individualusershadnowaytoopt-outofthedata
collection, nor preventthe release of their data. As a side-effect of Facebook’s
university-onlynetworkstructureatthetime,theonlywayforsomebodytoavoid
their data being used in such a mannerwas to leave these institution networks,
losingmuchoftheutilityoftheserviceintheprocess.ThisparallelsFacebook’s
approachto releasingotherproducts,suchas the introductionof NewsFeed in
2006.Bybroadcastingprofileupdatestoone’snetwork,theeffectivevisibilityof
theirdatawassubstantiallyincreased,withnowaytoopt-outofthefeaturewith-
outleavingtheserviceentirely.Thisillusorylossofcontrolwaswidelycriticised
atthetime[20].
• Negotiability:Inthisrespect,theuser’srelationshipwithFacebookitselfissig-
nificant.InadditiontoIRBapproval,thestudywasconductedwithFacebook’s
permission[28],butFacebook’sprivacypolicyatthetimedidnotallowforFace-
book to share their data with the researchers.2 Therefore, the existing context
2 Facebook Privacy Policy, February 2006: http://web.archive.org/web/
20060406105119/http://www.facebook.com/policy.php
8 LukeHuttonandTristanHenderson
forsharinginformationonFacebookwasdisruptedbythisstudy.Thisincludes
the normative expectation that data are shared with Facebook for the purpose
of sharing informationwith one’s social network,and notmyriad third parties.
Inaddition,nocontrolswereextendedtothepeopleinvolvedtopreventitfrom
happening,ortomakeapositivedecisiontopermitthisnewinformation-sharing
context.
3.2 Facebook emotionalcontagionexperiment
In2012,researchersatFacebookandCornellUniversityconductedalarge-scaleex-
perimenton689,003Facebookusers.Thestudymanipulatedthepresentationofsto-
riesin Facebook’sNews Feedproduct,whichaggregatesrecentcontentpublished
byauser’ssocialnetwork,todeterminewhetherbiasingtheemotionalcontentofthe
newsfeedaffectedtheemotionsthatpeopleexpressedintheirowndisclosures[27].
WhiletheT3studyhighlightedprivacyrisksofnonconsensualdatasharing,the
emotionalcontagion experimentraises differentpersonal risks from inappropriate
data miningactivities. For example,for a personsufferingfromdepression,being
subjectedtoanewsfeedofpredominantlydepressive-indicativecontentcouldhave
catastrophic consequences, particularly considering the hypothesis of the experi-
mentthatdepressivebehaviourwouldincreaseunderthesecircumstances.Consid-
ering the scale at which the experiment was conducted, there was no mechanism
for excludingsuch vulnerable people, nor measuring the impact on individualsto
mitigatesuchharms.Furthermore,asthestudywasnotage-restricted,childrenmay
haveunwittinglybeensubjectedtothestudy[19].Rucubernotesthattheharmsto
any one individualin such experimentscan be masked by the scale of the experi-
ment[43].
Beyondtheresearchcontext,thiscasehighlightsthebroaderimplicationsofthe
visibility of media, whether socially-derivedor from mainstreammedia, being al-
gorithmicallycontrolled.NapoliarguesthatthisexperimenthighlightsFacebook’s
abilitytoshapepublicdiscoursebyalteringthenewsfeed’salgorithmtointroduce
politicalbias[37],withoutanygovernancetoensurethatsuchnewmediaareacting
inthepublicinterest.ThemajorityofFacebookusersdonotknowthatsuchfiltering
happensatall,andthe selectivepresentationofcontentfromone’ssocialnetwork
cancausesocialrepercussionswheretheperceptionisthatindividualsarewithhold-
ingpostsfromsomeone,ratherthananalgorithmicinterventionbyFacebook[9].
This case shows one of the greater risks of opaquedata mining. Where people
areunawaresuchactivitiesaretakingplace,theyloseallpowertoactautonomously
to minimise the risk to themselves, evenputtingaside the responsibilityof the re-
searchers in this instance. We now consider how this case meets our three core
themes:
• Legibility:Individualswereunawarethattheywereparticipantsintheresearch.
Theywouldhavenoknowledgeorunderstandingofthealgorithmswhichchoose
whichcontentispresentedonthenewsfeed,andhowtheywerealteredforthis
BeyondtheEULA:Improvingconsentfordatamining 9
experiment,northatthenewsfeedisanythingotherthanachronologicalcollec-
tionofcontentprovidedbytheirsocialnetwork.Withoutthisinsight,thecauseof
aperceptiblechangeintheemotionalbiasinthenewsfeedcannotbereasoned.
Even if one is aware that the news feed is algorithmically controlled, without
knowingwhichdataarecollectedorusedinordertodeterminetherelevanceof
individualstories,itisdifficulttoreasonwhycertainstoriesaredisplayed.
• Agency: As in the T3 case, without awareness of the experiment being con-
ducted, individuals were unable to provide consent, nor opt-out of the study.
Withoutanunderstandingofthealgorithmswhichdrivethenewsfeed,norhow
theywereadjustedforthepurposesofthisexperiment,individualsareunableto
takeactions,suchaschoosingwhichinformationtodiscloseorhidefromFace-
book in an effortto controlthe inferencesFacebook makes, nor to correctany
inaccurate inferences. At the most innocuous level, this might be where Face-
bookhasfalselyinferredahobbyorinterest,andshowsmorecontentrelatingto
that. Of greaterconcernis when Facebook,or the researchersin thisstudy,are
unabletodetectwhenshowingmoredepressive-indicativecontentcouldpresent
arisk.
• Negotiability:Inconductingthisstudy,Facebookunilaterallychangedtherela-
tionshipitsusershavewiththeservice,exploitingthosewhoareunabletocontrol
howtheirinformationisused[45].Atthetimeofthestudy,theTermsofService
towhichusersagreewhenjoiningFacebookdidnotindicatethatdatacouldbe
used for research purposes[19], a clause which was added after the data were
collected. As a commercial operator collaborating with academic researchers,
thenatureofthestudywasambiguous,withFacebookhavinganinternalproduct
improvementmotivation,andCornellresearchersaimingtocontributetogener-
alisable knowledge.Cornell’sIRBdeemedthattheydidnotneedto reviewthe
studybecauseFacebookprovidedthedata,3,buttheethicalimpactontheunwit-
tingparticipantsisnotdependentonwhocollectedthedata.AsFacebookhasno
legalrequirementtoconductanethicalreviewoftheirownresearch,andwithout
oversightfrom the academic collaborators, these issues did not surface earlier.
Facebook has since adopted an internal ethics review process [24], however it
makeslittlereferencetomitigatingtheimpactonparticipants,andmostlyaims
to maximise benefit to Facebook. Ultimately, these actions by researchers and
institutions with which individuals have no prior relationship serve to disrupt
the existing contextualnorms concerningpeople’s relationship with Facebook,
withoutextendinganyabilitytorenegotiatethisrelationship.
3.3 NHSsharingdatawith Google
InFebruary2016,theGooglesubsidiaryDeepMindannouncedacollaborationwith
theNationalHealthService’sRoyalFreeLondonTrusttobuildamobileapplication
3Cornellstatement:https://perma.cc/JQ2L-TEXQ
10 LukeHuttonandTristanHenderson
titledStreamstosupportthedetectionofacutekidneyinjury(AKI)usingmachine
learningtechniques.The informationsharingagreementpermittingthiscollabora-
tion gives DeepMind ongoing and historical access to all identifiable patient data
collectedbythetrust’sthreehospitals[22].
While the project is targeted at supporting those at risk of AKI, data relating
to all patients are shared with DeepMind, whether they are at risk or not. There
is no attempt to gain the consent of those patients, or to provide an obvious opt-
out mechanism. The trust’s privacy policy only allows data to be shared without
explicitconsentfor“yourdirectcare”.4 ConsideringthatStreamsisonlyrelevant
to those beingtested forkidneydisease, itfollowsthatfor mostpeople,their data
arecollectedandprocessedwithoutanydirectcarebenefit[21],inviolationofthis
policy.Giventhediagnosticpurposeoftheapp,suchanapplicationcouldconstitute
amedicaldevice,howevernoregulatoryapprovalwassoughtbyDeepMindorthe
trust[21].
Permittingprivatecompaniestoconductdataminingwithinthemedicaldomain
disruptsexistingnorms,byoccupyingaspacethatliesbetweendirectpatientcare
andacademicresearch.Existingethicalapprovalanddata-sharingregulatorymech-
anisms have not been employed,or are unsuitable for properly evaluating the po-
tential impacts of such work. By not limiting the scope of the data collection nor
acquiringinformedconsent,thereisnoopportunityforindividualstoprotecttheir
data. In addition, without greater awareness of the collaboration, broader public
debate about the acceptability of the practice is avoided, which is of importance
consideringthesensitivityofthedatainvolved.Furthermore,thisfairlylimitedcol-
laborationcannormaliseabroadersharingofdatainthefuture,aneventualitywhich
ismorelikelygivenanongoingstrategicpartnershipforgedbetweenDeepMindand
thetrust[21].Wenowconsiderthiscasefromtheperspectiveofourthreethemes:
• Legibility:Neitherpatientswhocoulddirectlybenefitfromimproveddetection
of AKI, nor all other hospital patients, were aware that their data were being
shared with DeepMind. Indeed, this practice in many data mining activities –
identifyingpatternstoproduceinsightfrommyriaddata–risksviolatingafun-
damentalprincipleofdataprotectionregulation;thatofproportionality[11].
• Agency:TheNHScollectsdatafromitsfunctionsinanumberofdatabases,such
astheSecondaryUsesServicewhichprovidesa historicalrecordoftreatments
intheUK,andcanbemadeavailabletoresearchers.Awarenessofthisdatabase
anditsresearchpurposeismostlyconstrainedtoleafletsandposterssituatedin
GPpractices.Ifpatientswishtoopt-outoftheirdatabeingusedtheymustinsist
onit,andarelikelytoberemindedofthepublichealthbenefitanddiscouraged
fromopting-out[6].WithoutbeingabletoassumeawarenessoftheSUS,norin-
dividualconsentbeingacquired,itisdifficultforindividualstoactwithagency.
Evenassuming knowledgeof thiscollaboration,it wouldrequireparticularun-
derstanding of the functions of the NHS to know that opting-out of the SUS
wouldlimithistoricaltreatmentdata madeavailableto DeepMind.Evenwhere
someoneiswillingtosharetheirdatatosupporttheirdirectcare,theymaywish
4RoyalFreeLondonTrustPrivacyStatement:https://perma.cc/33YE-LYPF
