CrowdTone: Crowd-powered tone feedback and improvement system for emails

0.65 MB·

by  Rajan Vaish

CrowdTone: Crowd-powered tone feedback and
improvement system for emails
RajanVaish Andre´sMonroy-Herna´ndez
StanfordUniversity MicrosoftResearch
Stanford,CA,USA Redmond,WA,USA
[email protected] [email protected]
ABSTRACT give people access to pools of professional writers on-
In this paper, we present CrowdTone, a system designed to demand. However, these services can be costly and time-
help people set the appropriate tone in their email commu- consuming (e.g. finding the right people, setting a fair con-
7
nication. CrowdTone utilizes the context and content of an tract,spendingtimeinbackandforthcommunication,etc.)
1
0 emailmessagetoidentifyandsettheappropriatetonethrough
Previoussystemsresearchhaslookedintomoreefficientwrit-
2 aconsensus-buildingprocessexecutedbycrowdworkers.We
ing solutions that rely on crowds. Work in this space has
evaluated CrowdTone with 22 participants, who provided a
n proven that, with proper scaffolding, crowds can do expert-
totalof29emailsthattheyhadreceivedinthepast, andran
a qualityworksuchasshorteningtext,correctinggrammar,and
J themthroughCrowdTone.Participantsandprofessionalwrit-
finding and formatting citations [4, 16, 18, 24]. Despite all
ersassessedthequalityofimprovementsfindingasubstantial
7 thiswork,nosystemsresearchhasfocusedontone,andmore
increaseinthepercentageofemailsdeemed“appropriate”or
] “veryappropriate”—from25%tomorethan90%byrecipi- specificallyemails’tone.
C ents,andfrom45%to90%byprofessionalwriters.Addition- In order to address the need for tone improvement in email
H ally,therecipients’feedbackindicatedthatmorethan90%of communication, we created CrowdTone, a crowd-powered
theCrowdToneprocessedemailsshowedimprovement. tone-improvement system. CrowdTone first receives an
.
s email’smaincontent,alongwithbasicinformationaboutthe
c
[ sender, the receiver, and open-ended context elicited by the
AuthorKeywords user. Then, CrowdTone outputs an improved version of the
1 Crowdsourcing,AmazonMechanicalTurk,Tone email, with a tone appropriate to the context and receiver.
v
improvement,Emailwriting CrowdTone uses crowd workers from Amazon Mechanical
3
Turk to take emails through a step-by-step tone-scaffolding
9
process that identifies the original tone, improves it through
7
1 INTRODUCTION consensusworkflow,andoutputsthebestresult.
0 Settingtherighttone—attitudeexpressedthroughwords—
WeevaluatedCrowdTonewith22participantsrecruitedfrom
. inwrittencommunicationsoftendetermineswhetherareader
1 our own organization. We asked participants to provide
willinterpretthemessageasthewriterintended[13].
0 emails they had received that they had perceived as prob-
7 Despite changes in communication practices, email is still lematicwithregardstotone. Forexample,someparticipants
1 one of the most popular forms of communication in profes- shared emails they received from students being rude or in-
v: sional settings. Every day, over 200 billion emails are writ- appropriate. Participants were asked to remove the name of
i ten worldwide [29]. Despite its popularity, email is prone thesenderandotheridentifiableinformationtomaintaintheir
X
tomisunderstandingsjustlikeothertypesofwrittencommu- anonymity. Based on the participants’ feedback, more than
r nication [33, 15, 35, 20, 11, 5]. For example, researchers 90%oftheemailswereimprovedintone,whilethepercent-
a
havefoundthatemailsendersoftenhavetheerroneousbelief ageofemailmessagesdeemed“appropriate”or“veryappro-
thattheirrecipientswillidentifytheintendedemotionintheir priate” rose from about 25% to more than 90%. When sur-
messages [15, 20, 11], whiletherealityisthatclosetohalf veyed,75%oftheseparticipants“agreed”or“totallyagreed”
(44%)ofemailrecipientsfailtoidentifytheintendedtoneof when asked whether the CrowdTone emails were of high
anemailmessage. Furthermore, thesameresearchersfound quality — matching their writing expectations. Besides, the
that when the tone of an email is unclear, recipients tend to tone-scaffolding process was reported as easy and effective
interprettheemailbasedontheirstereotypesandexistingas- bythecrowd-workers.
sumptionsofthewriter.
The core contribution of this work is the design, imple-
Thereareafewtoolsthatautomaticallydetecttone[32, 25] mentation, and evaluation of a crowd-powered process that
and sentiment [12, 1] in writing, but in order to actually set self-identifies — without user instructions — and improves
theappropriatetone,peoplehireprofessionalcopywritersand the email tone with basic context and information provided.
proofreadersfortheirimportantcommunications. Though outside the scope of this paper, we believe that our
approachandfindingscanbeappliedtodifferentdomainsand
Only few people and organizations can afford their own ex-
mediaoranyformofwrittencommunications.
pert writers, so platforms like Upwork.com and Wordy.com
1
RELATEDWORK five North American writers from Upwork, an online free-
The need to communicate a message over email in the in- lancing platform. We selected only writers with a rating of
tended way is non-trivial [15, 20, 11, 5]. In fact, several at least 4.5 / 5 that had completed more than 100 hours of
online services, such as Wordzen [34] and Crystal [14], are writing-relatedworkonUpwork. Wegavethesewritersfive
tryingtoaddressthissameneed. Wordzenfocusesongram- emailseach,andaskedtoimprovethetoneandtoformulate
mar improvements, while Crystal makes useful suggestions and document a step-by-step process that begins with tone
based on the recipient’s personality, however none of these identificationandendswithtoneimprovement.
systemsfocusesonimprovingthetoneperse.
Only two of the five writers could formulate a process. The
The importance of tone [13], along with the need to under- otherthreefoundthistaskextremelydifficulttoexecute. One
standsentiment,hasattractedtheattentionofcompanieslike ofthesethreedescribedtheproblemthisway:
Google [12], IBM [32, 1] and Microsoft [2]. These com-
“Thereverseprocessisacomplicatedone. Therewritingof
panies have public APIs (Application Programming Inter-
theemailswasnotasdifficultasbreakingeachbeatdownas
faces) for analyzing sentiment. Although those APIs and
yourequested. Kindofnewterritoryforanyone”
othermachine-basedanalysistools[30]areextremelyuseful
forparsinglargedatasets,theyarenotgenerallydesignedfor By translating our learning from this exercise and other re-
end-users, and are not specifically designed to help improve sources [28], we explored a few pilot approaches to design-
emailtone. ingaworkflowthatcanproducehighqualityoutputusingba-
siccontext. Ourapproachutilizedthestep-by-stepworkflow
Crowdsourcing is another domain being explored for writ-
thatUpworkprofessionalsformulatedbasedontheirexperi-
ing assistance. Crowd-powered projects like Soylent [4],
ence and expertise. The rest of this section will provide a
Legion [22] and Chorus [23] have encouraged researchers
detaileddescriptionoftheprocess—frominputthroughthe
to harness the crowd for complex tasks — and have even
tonescaffoldingphasetooutput.
been implemented on different interfaces, including smart-
watches[27]. ProjectslikeTurkomatic[21],Ensemble[16],
Input
Crowdforge[18],andothers[31]havesucceededinconvert-
CrowdTonesupportsGUIandRESTbasedinput,andaccepts
ing macro tasks into micro chunks that makes it easier for
a set of mandatory and optional information to process the
crowdworkerstoaccomplishexpert-leveltaskssuchaswrit-
email. Primarily intended for a sender of an email, Crowd-
ingarticlesandstories.
Toneacceptsthefollowinginformation:
To accomplish those efforts, it is important for crowd-
1. MandatoryInformation: providesminimalcontext
powered systems to focus on task sequencing [6], scaffold-
ing,andcrowdcoordination. Crowdcoordinationtechniques (a) Sender: relationship(e.g. intern,student)
suchasiterativeandparallelcontributions [10,9,8,7]pro-
(b) Recipient: relationship(e.g. adviser,professor)
ducevaluableresultsthathavebeenusedtobuildprojectslike
(c) Emailsubject
CrowdCrit [24]andStoria [17]. CrowdCritusesscaffolding
and crowd coordination to provide expert-level critiques to (d) Emailcontent
designers. (e) Open-ended context short-description to provide in-
formationon,orthestorybehindtheemail
IndevelopingCrowdTone,weusedexistingtechniquesfrom
the literature, and developed our own sequencing and scaf- 2. OptionalInformation: providesmaximumcontext
foldingapproachestoidentifyandimproveemailtone.
(a) Gender: senderandrecipient
THECROWDTONESYSTEM (b) Nativelanguage: senderandrecipient
As shown in Figure 1, CrowdTone inputs email information (c) Hierarchy relationship (e.g. professionally senior,
and context, and produces an improved email via a tone- samelevel,orjunior),ifapplicable
scaffoldingandconsensusworkflowprocess.
(d) Relationshiptype: friendsandfamily,acquaintances,
strangers(coldemails)
Input: Email subject and Crowd coordination If the sender decides to input mandatory information only,
cinofnotremnat, tSioenn, dCeo rn atenxdt Receiver Tpornoece sscsa ffolding Conse nsus Output: Impro ved email tthoepyroavreidperoovpitdioinngalmininfoimrmaaltcioonn,tetxhte.yHwoiwllebveerp,riofvtihdeiyngdemcaidxe-
imumcontextthatCrowdTonecansupport.
Figure1. CrowdTonesystemoverview: Inputsemailinformationand
context,tonescaffoldingandconsensushelpscoordinatecrowdworkers
fromMTurktoproduceimprovedemail,improvedemailisproducedas Crowdcoordination—phase1:Thetonescaffoldingpro-
anoutput
cess,fromtoneidentificationtoimprovement
Identifyingandimprovingthetoneofawrittentextisacriti-
calandexpert/professional-leveltask. Inthissection,wede-
Designingtheworkflowprocess scribeourdesignprocessandthesystemthatproducesthree
The design of the process behind CrowdTone was informed improvedversionsoftheoriginalemail,bythreecrowdwork-
byaformativestudywithprofessionalwriters. Werecruited ers.CrowdTonedoesnotaccepttone-relatedinstructionasan
2
input. Theprocessfocusesinsteadonenablingcrowdwork- (c) Step 3: Use the list and suggestions from Step 2 as
erstoidentifyandimprovethetoneofanemail. self-instruction to revise and improve the email di-
rectly.
Identify current tone (d) Step4: IteratetheoutputfromStep3tofinetunethe
email’stone. Herethecrowdworkersareencouraged
Right No Identiftyo nthee right tomakefurtherimprovementsthroughdirectediting.
tone?
Identify things to
Yes improve Overall, each email undergoes and is reviewed by three
crowd-workersforconfidentconsensus. Emailsdeemedini-
Identify things to Fix the tone, edit
improve directly tially to have correct tone goes through two crowd-worker
Improve the tone, Polish the tone once 2nd phase of consensus Polish the tone once steps,whileone’swithincorrecttonegothroughfourcrowd-
edit directly more, continue editing more, continue editing workersteps. Eventually,producingthreeimprovedversions
Figure2. Toneidentificationtoimprovementprocess: Crowdworkers
peroriginalemail.
gothroughthisprocesstoeliminatetheneedfortone-relatedinstruc-
tionsfromrequester This process helps improve the tone without requiring tone-
relatedinstructionsfromthesender.
As shown in the Figure 2, crowd workers begin the pro-
cess by reviewing the original email and context from the Crowdcoordination—phase2:Consensus,choosingthe
perspective of its recipient. Acting from the recipient’s bestamongtheimprovedemailsfromlaststage
viewpoint, the crowd workers attempt to identify the cur- After getting three input-phase responses for each email,
rent tone using primary and secondary tone options. The CrowdTone coordinates the crowd to get consensus on and
two primary options are formal or informal. The ten then output the most improved email. Figure 3 shows an
secondary options are — appreciative/thankful, confident, overviewoftheconsensusworkflow.
courteous/respectful/polite,emotional/persuasive,enthusias-
tic/cheerful,light/humorous/friendliness,regretful/sorrowful, PhaseA:Selectingtwoemailversionsforconsensus
serious,cold/unfriendly,andenraged. After three input-phase responses are generated for each
email,thenextstepistoselecttwoofthethreerevisedemails
These options were developed after reviewing multiple arti-
forthefinalconsensusround.
cles on tone [28, 26, 36] and running multiple pilot studies.
Whilenotexhaustive,theyhelpthecrowdworkersmakede- If “yes” or “no” were selected twice by two crowd-workers
cisionsmorequickly. Asoneworkerdescribesit: (66.66%),thetwo“yes”ortwo“no”emailsaresentforward
forfurtherconsensusevaluation.
“Combiningtheprimaryandsecondarytonesincurrentand
correct makes it easier to quickly bring the ideas together, If an email received three “yes” or three “no” responses
enjoyedit” (100%),i.e. ifanemailwasunanimouslyfoundtobeofap-
propriateorinappropriatetonebythreecrowdworkers—the
After identifying the current primary and secondary tone,
two emails to be forwarded for further consensus evaluation
each crowd worker has to decide whether that tone is right
are chosen based on the similarity of attributes such as pri-
or not. Based on their yes-or-no decision, the workflow di-
maryandsecondarytone.
vergesinthefollowingmanner:
PhaseB:Selectingoneemailversionforfurtheriteration
1. If the crowd worker chooses “Yes”, he or she takes these
Inthesecondpartoftheconsensusphase,fromthetwoemail
actions:
versions (a and b) forwarded from Phase A, three different
(a) Step 1: Identify scope of improvement and at least crowd workers choose the version that — in their view —
oneinstanceoftextthatcouldstillbeimproved. is the best in terms of tone, and do another iteration to im-
provethetoneofthatemail. Astherearetwoemailversions,
(b) Step 2: Use the suggestions from Step 1 as self-
andthreedifferentcrowd-workerstomakethedecision—the
instruction and make the improvements. Here the
final email selected will either enjoy a majority of 66.66%,
crowdworkerdirectlyeditsandimprovestheemail.
or100%. Additionaliterationforimprovementalsobringsa
2. If the crowd worker chooses “No”, he or she takes these newperspectivefromanothercrowdmember.
actions:
(a) Step 1: Identify the right tone by choosing the ideal Output
Figure 4 shows a “before” and “after” example of an email
primaryandsecondarytoneoptionsfortheemailand
the appropriate intensity option — very, quite close, that went through the CrowdTone process. By the time
somewhat. Forexample, onecanidentifythetoneto eachemailhascompletedthetone-scaffoldingandconsensus
beveryformalandappreciative. phases, six crowd workers have helped produce this output.
However, depending on whether the tone of email sent was
(b) Step 2: Identify and list what needs to be improved
correct,theoutputwouldvary:
toachievetheidealtone. Herethecrowdworkersare
encouraged to be specific and list as many instances 1. If the original email tone was deemed to be correct, the
aspossible. outputprovides:
3
Max assignment = 3 per HIT
Max assignment = 3 per HIT
Launch a survey with
Response 1 top 2 options (A and
If 100% (on the basis Response 1
B) with matching Survey to
of agreement -yes/ primary tone (cluster) choose Final
no) between A response
Response 2 If > 66% (on the and B, and Response 2 > 66% (A
basis of agreement - Launch a survey with draw or B)
yes/no) top 2 options (A and consensus
B) that caused Response 3
Response 3
agreement – yes/no
Final iteration
Consensus phase, 3 responses to improve
From tone scaffolding phase,
by 3 workers for 2 options – over A or B –
3 responses by 3 workers for
A and B the winner
1 original email
Figure3.Workflowoftheconsensusapproachthathelpsselectthebestimprovedemail,andfacilitatesadditionalroundofiterationtoimprovethebest
onefurther
Figure4. Beforeandafterexample: OntheleftweseeinputgiventoCrowdTonewithadditionalcontextaboutsenderandrecipient’srelationship,
hierarchyandthenatureofemail.Ontherightweseetheimprovedemailthatwaslabeledasveryappropriatetonebyboth,recipientsandprofessionals
(a) Originaltoneandintensity expressedaneedforhelpfulfeedbacktoimprovetheiremail
tone. Ofthe92participants, 84%assessedthemselvestobe
(b) Improvedemailwithadditionalnotes
fluentandexpertinEnglishand71%ofparticipantswerefre-
2. If the original email tone was deemed to be incorrect, the quentemailusers,sendingmorethanfiveemailsperday. On
outputprovides: further investigation, we found that a substantial number of
respondents (74%) felt the need for help in instances when
(a) Originaltoneandintensity theywereprofessionallyemailingpeoplemoreseniortothem
(b) Righttoneandintensityfortheemail thattheydidnotknow. Theexpressedneedforhelplessened
somewhat(59%)withregardtoprofessionallyseniorpeople
(c) Improved email with additional notes about changes
theydidknow. Overall,mostpeople(75%)reportedneeding
andsuggestions
themosthelpwhentheyweresending“cold”emails,thatis,
CrowdTonesupportsgettinginputfromaRESTrequest,with emailstostrangers,especiallywhentheyneededtoaskafa-
eachemailprocessinggeneratingataskid,buttherequester vor or make a request. Figures 5 and 6 give an overview of
orsendercanalsogetaJSONformatoutput. thesesurveyresponsesand,takentogether,reflecttheimpor-
tanceofandneedforappropriatetoneinemails.
CROWDTONEEVALUATION
ToassesswhetherCrowdToneimprovesemailtone,wecon- Step2:Collectemailsandinitialinformationfromthepar-
ductedthefour-phaseevaluationdescribedbelow.
ticipatingrecipients
After understanding the use cases when tone-fixing is most
Step 1: Understanding email usage, types and tone re- helpful,wegatheredemailsreceivedbysomeofouremploy-
latedusecases eesalongwithadditionalinformationoneachemailviaasur-
To begin our evaluation, we ran a formative study within vey. We focused on the email recipients rather than senders
ourorganizationtobetterunderstandpeople’sneedswhenit becauseitistherecipients’assessmentofwhetheramessage
comesto“tonefixing”inemail. Wegathered92responsesto isappropriateorinappropriateintonethatmattersmost[15,
anonlinesurveyadministeredtostudents. Ofthese, 94%of 20,11,5]. Weaskedemployeesinourorganizationtoshare
respondentsthoughttonetobeimportantinemail,while60% anemailtheyhadreceivedthattheyconsideredtobedeficient
4
Seeking request or favor 75% Very Appropriate 4%
Declining a request or favor 50% Appropriate 22%
Scheduling a meeCng 27%
Neutral 39%
Applying for a job posiCon 54%
Not Appropriate 26%
Apologizing for a mistake 60%
Not at all Appropriate 9%
Reprimanding an employee 28%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
Figure 7. Appropriateness of the tone of original emails, assessed by
Others 13%
participatingrecipients
0% 10% 20% 30% 40% 50% 60% 70% 80%
Figure5.Emailtypesthatmostneedhelpwithtone
Upon asking about the senders, we learned that 45% were
strangerstotherecipients(coldemails),whiletheyknew52%
ofthem.And41%wereprofessionallyjuniortotherecipient,
Professionally senior (familiar) 59% while 38% were professionally same level, and 17% were
professionally senior. Based on the recipient’s perception,
Professionally same level (familiar) 23%
31% of the senders appeared to be native English speakers
Professionally junior (familiar) 15% and 43% seemed to be male, while 32% appeared female.
Overall,wereceivedabalancedsetofemails.
Professionally senior (cold email) 74%
Onascaleof1to5,with1being“notatallappropriate”and
Professionally same level (cold email) 58%
5being“veryappropriate”,wefoundthat7%werereported
Professionally junior (cold email) 32% bytherecipientstohavea“notatallappropriate”tone;24%
to have a “not appropriate” tone; while 41% were felt to be
Family or friends 8%
neutral. Theremaining 24% and3% emailswere rated“ap-
Others 10% propriate” and “very appropriate” by the recipients of these
emails. WecanseethedetailsinFigure7.
0% 10% 20% 30% 40% 50% 60% 70% 80%
Figure6. Categoryofrecipientsforwhichsendersneedthemosthelp Foreachsampleemail,theparticipatingrecipientswerepaid
withtone
$10 after they gave feedback on the CrowdTone-improved
version.
intone. Inresponse,wereceived29emailssentto22recip- Step 3: Run HITs (Human-Intelligent Tasks) on Mechani-
ients by people they knew or did not know. Before sending
calTurktogetoutputfromCrowdTone
these emails to us, the participants first made them anony-
CrowdToneispoweredbycrowdworkersfromAmazonMe-
mousbyremovingallidentifyinginformation.
chanicalTurk. Inthethirdstepofourevaluation,eachemail
Basedonapreliminarystudyconductedearlierwith92em- receivedthefollowingtwoCrowdTonecontexttreatments:
ployees, we learned about the type of emails where people
1. Minimalcontext: mandatoryopen-endedcontext.
need most help with tone fixing/improvement — therefore,
weaskedourrecipientstosendusemailsthat:
2. Maximum context: additional context with information
• askedforafavorormadearequest; such as: gender, native language, hierarchy relationship
andtypeofrelationshiptype.
• constituted a job application or a query regarding job op- Foreachofthesecontexttreatments,eachemailwentthrough
portunities;or two iteration related treatment — where, half of the time,
email went through two iterations of improvement; while
• wereapologeticorregretful. other half of the time, email went through three iterations
ofimprovement. Thethirditerationwasexecutedduringthe
Among these 22 recipients, 72% were male, while 28% fe-
consensus phase and caused negligible effect. Overall, for
male.35%werenativeEnglishspeakers,while86%assessed
each treatment, the email went through two phases — tone
themselvestobefluentorexpertinEnglishlanguage. Also,
scaffolding and the consensus process — and received revi-
55% participants were full time employees, while the rest
sioninputfromsixcrowdworkers.
wereinternsortemporaryworkers. Finally,67%participants
were active email users and sent more than five emails per To filter out non-English speakers from our crowd workers,
day. we chose only individuals located in the US. We also only
5
chose individuals with a 95% approval rating. Upon inquir-
Original Email - Recipient CrowdTone Email - Recipient
ingabouttheselectedworkers,wefoundthat58%weremale,
99%werenativeEnglishspeakers,and100%assessedthem- 70%
selves to be expert and fluent in English. Of these workers,
60%
31% had a high school degree, 51% had an undergraduate
collegedegree,and17%hadagraduatedegree. 50%
40% 4% 39%
Step 4: Evaluate CrowdTone output through people who
30% 52%
sharedemails(recipients) 4%
ToassesstheabilityofCrowdTonetoimproveemailtone,we
20% 39%
reached out to our participating recipients again, and asked
26%
them to assess the CrowdTone-generated emails in terms of 10% 22%
thetoneappropriatenessandimprovementinqualityoftone. 9% 4%
0%
Inthissecondroundofinteraction,wereceivedresponsesfor Not at all Not Appropriate Neutral Appropriate Very
24 of the original 29 emails. Five participants in the initial Appropriate Appropriate
group were not available to provide feedback on the newer Figure8.AppropriatenessoftoneforemailsgeneratedfromCrowdTone
versionsoftheemailstheysubmitted. comparedtooriginalemails—asratedbytherecipients
Step 5: Evaluate CrowdTone output through profession-
alsforfurthervalidation Original Email - Professional CrowdTone Email - Professional
Tomakeupfortheparticipantslostinthesecondroundofin- 100%
teractionwiththerecipients,andtogetasecondperspective, 90%
werecruitedthreeprofessionalwritersfromUpwork. These
80%
individuals had ratings of 4.5 or higher, were from North
70%
America,hadcompletedmorethan100hoursofworkonthe 41%
60%
Upwork platform, and assessed themselves to be experts in
English. 50%
10%
40%
The professionals were asked to fill in a similar survey as
30%
participating recipients, where they were shown an original 45% 48%
email with its context, followed by newer versions. Pro- 20% 38%
fessionals were asked to rate the tone appropriateness and 10% 14%
whether the newer versions were improved, and by how 0% 3%
Not at all Not Appropriate Neutral Appropriate Very
much.Wederivedconsensusfromtheirresponsesthathelped Appropriate Appropriate
us make decisions about the quality of work produced by
Figure9.AppropriatenessoftoneforemailsgeneratedfromCrowdTone
CrowdTone.
comparedtooriginalemails—asratedbytheprofessionals
RESULTSANDDISCUSSION
Inthissection,wediscusstheresultsofourCrowdToneeval-
Overall, we found CrowdTone produced emails that were
uation. Asthegoalforthisstudywastoevaluatethequality
bothappropriateintoneandimprovementsovertheoriginals.
ofCrowdToneoutput—foreachsampleemail,wereceived
feedbackfromrecipientsandprofessionalsviasurvey. These
individuals compared this version to the original. After re- CrowdToneproduceshigh-qualityemailswithorwithout
viewingthisfeedback,wecametothefollowingconclusions. additionalcontext
Besides extra work to provide additional context, a lot of
CrowdTonegeneratesemailswithappropriatetone timesusersmaynothavemorecontextualinformationofthe
According to the recipients’ feedback, the original emails recipient — such as: hierarchy relationship, gender and na-
weremostlyinappropriate. Only26%werejudgedtobe“ap- tive language. In this light, we wanted our system to be
propriate” or “very appropriate”. As Figure 8 shows, after able to produce high-quality results, with or without addi-
theCrowdToneprocess,thepercentageof“appropriate”and tional context — and hence to test whether additional con-
“veryappropriate”emailsroseto91%,withthemajoritycon- text substantially improves the quality. That is, whether the
sideredtobe“veryappropriate”. NoemailsfromCrowdTone CrowdTone results are substantially affected by the amount
wereassessedtobe“notatallappropriate.” ofcontextprovided.
According to the professionals’ feedback the percentage of AsFigure10shows,theevaluationrecipientsdesignated29%
“appropriate” and “very appropriate” emails rose from 45% ofemailswithminimalcontext(mandatory)inputtobe“ap-
fortheoriginalsto90%fortheCrowdToneemails.Noemails propriate”and46%oftheseemailstobe“veryappropriate”,
fromCrowdTonewerejudgedbytheprofessionalstobe“in- totaling 75%. At the same time, the recipients judged 50%
appropriate”(seeFigure9). of the CrowdTone emails with maximum context (including
6
Recipient - Min Context Recipient - Max Context Recipient Professional
90% 140%
80%
120%
70%
100%
60% 25% 55%
50%
50% 80%
40% 60%
45%
30%
21% 40%
46%
20% 63%
29% 20%
10% 4% 17% 29%
8%
0% 0% 4% 4%
Not at all Not Appropriate Neutral Appropriate Very Not at all No Same Some Substan=al
Appropriate Appropriate Improvement improvement improvement improvement
Figure10. AppropriatenessoftoneforemailsgeneratedfromCrowd- Figure12.CrowdToneimprovementovertheoriginalemail—asrated
Toneusingminimalandmaximumcontext—asratedbytherecipients bytherecipientsandprofessionals
Overall, we learned that adding additional context does not
Professional - Min Context Professional - Max Context
createasubstantialdifferenceinoutput—CrowdTone’stone
120%
scaffoldingandconsensusworkflowisrobustwithregardto
contextualinformation.
100%
80% 45% CrowdTone substantially improves the quality of tone
overtheoriginalemails
60%
As part of the same survey, participating recipients were
40% 28% askedtoquantifytheimprovementintheneweremail’stone
on a Likert scale from “not at all improved” to “significant
55%
20% improvement”.
24% 31%
3%
10% As Figure 12 shows, the recipients observed significant im-
0% 3%
Not at all Not Appropriate Neutral Appropriate Very provement in 63% and some improvement in 29% of the
Appropriate Appropriate emails, totaling 92%. Recipients observed no improvement
Figure11. AppropriatenessoftoneforemailsgeneratedfromCrowd- for4%ofemailsthathadalreadybeendeemed“appropriate”.
Toneusingminimalandmaximumcontext—asratedbytheprofes-
sionals The averaged responses of the three professionals indicated
that 55% of the emails showed significant improvement and
that45%showedsomeimprovement,totalingto100%.
additional information like hierarchy and relationship) to be
CrowdTone generates a professional-quality tone that
“appropriate”and25%tobe“veryappropriate”emails,also
matchesthewritingexpectationsofrecipient
totaling75%. Fromthis,welearnedthatthetotalpercentage
To understand whether CrowdTone produces high-quality
of CrowdTone emails judged to be “appropriate” and “very
emails,thatis,thosethatmeetthewritingexpectationsofre-
appropriate”isthesameforminimalandmaximuminputs.
cipientsorare“professional”inquality,weaskedthesurvey
As Figure 11 illustrates, the professional writers found 55% respondents whether the newer emails were of expert qual-
ofCrowdToneemailswithminimalcontexttobe“appropri- ity or would match the quality of an email they had written.
ate”and31%tobe“veryappropriate”,totaling86%. Incon- They responded on a Likert scale that ranged from “totally
trast,theyfound45%ofemailswithmaximumcontextinput disagree”to“totallyagree”.
tobe“appropriate”and28%tobe“veryappropriate”, total-
As Figure 13 illustrates, the recipients responded that they
ing72%.
“totally agreed” regarding 21% of the emails and “agreed”
After conducting a blind ranking of emails produced using regardingthehighqualityof54%oftheCrowdToneemails,
minimal and maximum context input, we found that partici- totaling 75%. The professionals indicated that they “totally
patingrecipientsandprofessionalsrankedoutputfrommaxi- agreed” regarding high quality for 38% of the emails and
mumcontextemailshigher—54%and61%ofthetime,re- “agreed” for 45%, totaling 83%. Together, the profession-
spectively. Thisisnotsubstantiallydifferentfromtheemails als and recipients found that the quality of the newer emails
withminimalcontextinputs. matchedthatofprofessionalwork.
7
The crowd is essential to CrowdTone. To help us get bet-
Recipient Professional
ter results and to optimize the crowd workers’ experience,
120% we incorporated tone scaffolding and consensus workflow
into CrowdTone. After recruiting crowd workers that satis-
100% fiedminimalrequirements—a95%approvalratingandgeo-
graphicallocationintheUnitedStates—wegothigh-quality
80% 45% resultsthatpleasedtheemailrecipientsandprofessionalwrit-
ers. Tohelpusunderstandthecrowd’sexperience,weasked
60% theworkerstogiveusfeedbackaspartoftheHIT.
Through a preliminary qualitative study, we found that the
40% 38%
crowddeemedtheCrowdToneprocesstobeextremelyeffec-
54% tive. Ithelpedthemidentifyandimproveemailtonethrough
20%
17% multiple iterations. Most crowd workers also acknowledged
21%
0% 4% 13% 8% their enjoyment of the HITs and asked to do more, as the
quotesbelowdemonstrate.
Totally disagree Disagree Neutral Agree Totally agree
Figure13. Agreementtohighqualitymatchingthatofprofessionals— “The HIT was easy enough. I understood what was being
asratedbytherecipientsandprofessionals askedofme,andIwasabletodothetaskquickly.”
“HITwasveryeasy. Idon’tseeanywayyoucouldhelpme
makeatonerelateddecisionfasterwiththisspecificHIT.”
No Yes
“The HIT was far more easy and straightforward than I ex-
120% pected. Pleasepostmore. ”
100% CONCLUSIONSANDFUTUREWORK
This paper presents CrowdTone, a crowd-powered system
80% thatreceivesemailcontentandcontext,andoutputsthesame
59% emailwithimprovedtone. CrowdToneutilizescrowdwork-
60% ersfromAmazonMechanicalTurk,whogothroughspecifi-
92%
callydesignedtonescaffoldingandconsensusphases—from
40% self-identifyingtonerelatedrequirementstofixingit—with-
outuserinstructions. Toevaluatethequalityoftheenhanced
20% 41% emailsproducedbyCrowdTone,wecollected29emailsfrom
22participants,andhadthemreviewedbysaidparticipating
8%
0% recipientsandthreeprofessionalwritersfromUpwork.Over-
Original Email CrowdTone Email
all,CrowdTone’scorecontributionsare:
Figure14.Chancesofrespondingback—asratedbytherecipients
• Robustnesswithregardtothepresenceorabsenceofcon-
textprovided.
• Substantial improvements in the quality of tone over the
CrowdTone increases the chances of hearing back from
originalemail. Basedontheparticipatingrecipients’eval-
therecipients uation,60%ormoreoftheemailsprocessedexhibitedsub-
One aspect that often determines the “success” of an email stantialimprovement.
iswhetheritelicitsaresponse. Althoughrecipientscanhave
• Generation of professional-quality tone that in the evalu-
multiple reasons for replying to an email, we focused here
ation matched the writing expectations of the recipients.
on tone as the primary motivator. Specifically, we investi-
Whensurveyed,75%oftheseparticipants“agreed”or“to-
gatedwhetherrecipientsrespondedtotheoriginalemailsand
tally agreed” when asked whether the CrowdTone emails
whether they would respond to the newer CrowdTone ver-
wereofhighquality.
sions.
AsFigure14shows,wefoundthatrecipientsdidnotrespond • Improved chances of getting a response back from email
to 41% of the original emails. With the emails generated recipients. Theparticipatingrecipientsindicatedthatthey
byCrowdTone,therecipientsjudgedthattheywouldnotre- wouldrespondtomorethan90%oftheemails,asubstan-
spond to just 8% of the emails. In other words they would tialimprovementfromtheoriginal59%.
respondto92%oftheCrowdToneemails. • Tone-scaffolding process was reported as easy and effec-
tivebycrowdworkers.
The crowd behind the CrowdTone found the tone-
ThroughCrowdTone,weextendexistingcrowd-poweredsys-
scaffolding process easy and effective in accomplishing tems research for efficient writing solutions, and present a
theirtask novel process that self-identifies and improves email tone.
8
Thatsaid,thesystemalsoshowedseverallimitationsthatare WordProcessorwithaCrowdInside.InProceedingsof
seedsforfuturework: the23NdAnnualACMSymposiumonUserInterface
SoftwareandTechnology(UIST’10).ACM,NewYork,
• Privacy. Participantsraisedconcernsabouttheprivacyim-
NY,USA,313–322.DOI:
plications of sharing private communication with crowd-
http://dx.doi.org/10.1145/1866029.1866078
workers without all the parties being aware of it. Future
work might explore ways of addressing this through ob- 5. KristinByron.2008.Carryingtooheavyaload? The
fuscationandexplicitnormsetting,andothermechanisms communicationandmiscommunicationofemotionby
exploredinsimilarpriorwork[19] email.AcademyofManagementReview33,2(2008),
309.
• Delay. Having to wait for the process of tone setting to
gothroughintroducesdelaysthatmightnotbesuitablefor 6. CarrieJCai,ShamsiTIqbal,andJaimeTeevan.2016.
someusers,andsomescenarios. Optimizingforreal-time ChainReactions: TheImpactofOrderonMicrotask
responsesthroughcrowd-retainersasitwasdoneinsimilar Chains.InProceedingsofCHI.
research[3]issomethingworthexploring.
7. StevenDow,JulieFortuna,DanSchwartz,Beth
• Deepintegration.Atthemoment,CrowdToneisnotdeeply Altringer,DanielSchwartz,andScottKlemmer.2011.
PrototypingDynamics: SharingMultipleDesigns
embeddedintheregularworkflowofsendingemails. Cre-
ImprovesExploration,GroupRapport,andResults.In
ating a plugin could be one way of addressing this, how-
ProceedingsoftheSIGCHIConferenceonHuman
ever,pluginsarenotuniversallyused,forexample,mobile
FactorsinComputingSystems(CHI’11).ACM,New
email clients rarely allow for plugins. However, deeper
York,NY,USA,2807–2816.DOI:
integration can provide access to conversation history —
makingCrowdTonemoreintelligentinitsabilitytodeter- http://dx.doi.org/10.1145/1978942.1979359
minedeepercontextandmeaning. Moreworkisneededto 8. StevenDow,AnandKulkarni,ScottKlemmer,and
understand the best mechanisms for integration into peo- Bjo¨rnHartmann.2012.ShepherdingtheCrowdYields
ple’sworkstyles. BetterWork.InProceedingsoftheACM2012
ConferenceonComputerSupportedCooperativeWork
• Restricted applications. Though CrowdTone can be uti-
(CSCW’12).ACM,NewYork,NY,USA,1013–1022.
lized to improve the tone of any type of written commu-
DOI:http://dx.doi.org/10.1145/2145204.2145355
nication, the current paper focuses on its application for
professionalemails. Infuture,weplantoapplythecoreof 9. StevenP.Dow,AlanaGlassco,JonathanKass,Melissa
our existing approach and expand it to different use cases Schwarz,DanielL.Schwartz,andScottR.Klemmer.
beyondemails: toothertypesofwrittencommunications, 2010.ParallelPrototypingLeadstoBetterDesign
domainsormedia. Results,MoreDivergence,andIncreasedSelf-efficacy.
ACMTrans.Comput.-Hum.Interact.17,4,Article18
Weseealotofpotentialforintroducingthird-partyfeedback
(Dec.2010),24pages.DOI:
andfixingintoourprofessionalcommunication. Largecom-
http://dx.doi.org/10.1145/1879831.1879836
panies often have marketing groups in charge of corporate
communication,butweenvisionthislevelofattentiontode- 10. StevenP.Dow,KateHeddleston,andScottR.Klemmer.
tail could be affordable for individuals. More generally we 2009.TheEfficacyofPrototypingUnderTime
see this work as yet another example of a potential for hy- Constraints.InProceedingsoftheSeventhACM
bridintelligencewherecrowdsandAIcancometogetherto ConferenceonCreativityandCognition(C&C’09).
optimizeprocessesandleveragethebestofbothworlds. ACM,NewYork,NY,USA,165–174.DOI:
http://dx.doi.org/10.1145/1640233.1640260
REFERENCES
11. NicholasEpleyandJustinKruger.2005.Whenwhat
1. IBMAlchemy.2015.AlchemySentimentAnalysisAPI.
youtypeisntwhattheyread: Theperseveranceof
(2015).http:
stereotypesandexpectanciesovere-mail.Journalof
//www.alchemyapi.com/api/sentiment-analysis.
ExperimentalSocialPsychology41,4(2005),414–422.
2. MicrosoftAzure.2015.LexiconBasedSentiment
12. Google.2015.SentimentAnalysisModel.(2015).
Analysis.(2015).https://azure.microsoft.com.
https://cloud.google.com/prediction/docs/
3. MichaelS.Bernstein,JoelBrandt,RobertC.Miller,and sentiment_analysis.
DavidR.Karger.2011.CrowdsinTwoSeconds:
13. NickGreer,JaimeTeevan,andShamsiTIqbal.2016.
EnablingRealtimeCrowd-poweredInterfaces.In
Anintroductiontotechnologicalsupportforwriting.
Proceedingsofthe24thAnnualACMSymposiumon
TechnicalReport.TechnicalReport.MicrosoftResearch
UserInterfaceSoftwareandTechnology(UIST’11).
TechReportMSR-TR-2016-001.
ACM,NewYork,NY,USA,33–42.DOI:
http://dx.doi.org/10.1145/2047196.2047201 14. CrystalProjectInc.2015.Crystal.(2015).
https://www.crystalknows.com/.
4. MichaelS.Bernstein,GregLittle,RobertC.Miller,
Bjo¨rnHartmann,MarkS.Ackerman,DavidR.Karger, 15. EricJaffe.2014.WhyIt’sSoHardToDetectEmotionIn
DavidCrowell,andKatrinaPanovich.2010.Soylent: A EmailsAndTexts.(2014).
9
http://www.fastcodesign.com/3036748/evidence/ 24. KurtLuther,Jari-LeeTolentino,WeiWu,AmyPavel,
why-its-so-hard-to-detect-emotion-in-emails. BrianP.Bailey,ManeeshAgrawala,Bjo¨rnHartmann,
andStevenP.Dow.2015.Structuring,Aggregating,and
16. JoyKim,JustinCheng,andMichaelS.Bernstein.2014.
EvaluatingCrowdsourcedDesignCritique.In
Ensemble: ExploringComplementaryStrengthsof
Proceedingsofthe18thACMConferenceonComputer
LeadersandCrowdsinCreativeCollaboration.In
SupportedCooperativeWork&SocialComputing
Proceedingsofthe17thACMConferenceonComputer
(CSCW’15).ACM,NewYork,NY,USA,473–485.
SupportedCooperativeWork&SocialComputing DOI:http://dx.doi.org/10.1145/2675133.2675283
(CSCW’14).ACM,NewYork,NY,USA,745–755.
DOI:http://dx.doi.org/10.1145/2531602.2531638 25. Lymbix.2015.ToneCheck.(2015).
http://tonecheck.com/.
17. JoyKimandAndresMonroy-Hernandez.2016.Storia:
26. MailChimp.2015.Voice&Tone.(2015).
SummarizingSocialMediaContentBasedonNarrative
http://voiceandtone.com/.
TheoryUsingCrowdsourcing.InProceedingsofthe
19thACMConferenceonComputer-Supported 27. MichaelNebeling,AlexandraTo,AnhongGuo,
CooperativeWork&SocialComputing(CSCW’16). AdrianAdeFreitas,JaimeTeevan,StevenPDow,and
ACM,NewYork,NY,USA,1018–1027.DOI: JeffreyPBigham.2016.WearWrite: Crowd-Assisted
http://dx.doi.org/10.1145/2818048.2820072 WritingfromSmartwatches.InProceedingsofCHI.
18. AniketKittur,BorisSmus,SusheelKhamkar,and 28. LisaPearlandMarkSteyvers.2013.Cmon YouShould
RobertE.Kraut.2011.CrowdForge: Crowdsourcing ReadThis: AutomaticIdentificationofTonefrom
ComplexWork.InProceedingsofthe24thAnnualACM LanguageText.InternationalJournalofComputational
SymposiumonUserInterfaceSoftwareandTechnology Linguistics(IJCL)4,1(2013),12–30.
(UIST’11).ACM,NewYork,NY,USA,43–52.DOI:
29. CraigSmith.2016.BytheNumbers: 60Incredible
http://dx.doi.org/10.1145/2047196.2047202
EmailStatistics.(2016).http://expandedramblings.
19. NicolasKokkalis,ThomasKo¨hn,CarlPfeiffer,Dima com/index.php/email-statistics/.
Chornyi,MichaelS.Bernstein,andScottR.Klemmer. 30. YlaR.TausczikandJamesW.Pennebaker.2010.The
2013.EmailValet: ManagingEmailOverloadThrough PsychologicalMeaningofWords: LIWCand
Private,AccountableCrowdsourcing.InProceedingsof ComputerizedTextAnalysisMethods.Journalof
the2013ConferenceonComputerSupported LanguageandSocialPsychology29,1(2010),24–54.
CooperativeWork(CSCW’13).ACM,NewYork,NY, DOI:http://dx.doi.org/10.1177/0261927X09351676
USA,1291–1300.DOI:
http://dx.doi.org/10.1145/2441776.2441922
31. JaimeTeevan,ShamsiIqbal,andCurtisvonVeh.2016.
20. JustinKruger,NicholasEpley,JasonParker,and SupportingCollaborativeWritingwithMicrotasks.In
Zhi-WenNg.2005.Egocentrismovere-mail: canwe ProceedingsofCHI.
communicateaswellaswethink? Journalof
32. IBMWatson.2015.ToneAnalyzer.(2015).
personalityandsocialpsychology89,6(2005),925.
http://www.ibm.com/smarterplanet/us/en/
21. AnandKulkarni,MatthewCan,andBjo¨rnHartmann. ibmwatson/developercloud/tone-analyzer.html.
2012.CollaborativelyCrowdsourcingWorkflowswith
33. LeaWinerman.2006.E-mailsandegos: Aninabilityto
Turkomatic.InProceedingsoftheACM2012
stepoutsideofone’sownheadmaybebehinde-mail
ConferenceonComputerSupportedCooperativeWork
miscommunication,accordingtorecentresearch.
(CSCW’12).ACM,NewYork,NY,USA,1003–1012.
AmericanPsychologicalAssociation37,2(2006),16.
DOI:http://dx.doi.org/10.1145/2145204.2145354
34. IncWordzen.2015.Wordzen.(2015).
22. WalterS.Lasecki,KyleI.Murray,SamuelWhite, http://www.wordzen.com/.
RobertC.Miller,andJeffreyP.Bigham.2011.
35. BusinessCoachingWorldwide.2007.DidYouKnow:
Real-timeCrowdControlofExistingInterfaces.In
Proceedingsofthe24thAnnualACMSymposiumon Thatemailrecipientscorrectlyinterprettheintended
UserInterfaceSoftwareandTechnology(UIST’11). toneofanemailmessageonly50%ofthetime? (2007).
ACM,NewYork,NY,USA,23–32.DOI: http://www.wabccoaches.com/bcw/2007_v3_i1/
didyouknow.html.
http://dx.doi.org/10.1145/2047196.2047200
36. WriteExpress.2015.Howtousetoneinyourwriting.
23. WalterS.Lasecki,RachelWesley,JeffreyNichols,
(2015).http://www.writeexpress.com/tone.html.
AnandKulkarni,JamesF.Allen,andJeffreyP.Bigham.
2013.Chorus: ACrowd-poweredConversational
Assistant.InProceedingsofthe26thAnnualACM
SymposiumonUserInterfaceSoftwareandTechnology
(UIST’13).ACM,NewYork,NY,USA,151–162.
DOI:http://dx.doi.org/10.1145/2501988.2502057
10