Table Of ContentSearch D-Lib: Go!
HOME | ABOUT D-LIB | CURRENT ISSUE | ARCHIVE | INDEXES | CALENDAR | AUTHOR GUIDELINES | SUBSCRIBE | CONTACT D-LIB
D-Lib Magazine
May/June 2017
Volume 23, Number 5/6
Table of Contents
Transforming Libraries and Archives through Crowdsourcing
Victoria Van Hyning, University of Oxford, Zooniverse
victoria [at] zooniverse.org
Samantha Blickhan, The Adler Planetarium, Zooniverse
samantha [at] zooniverse.org
Laura Trouille, The Adler Planetarium, Zooniverse
trouille [at] zooniverse.org
Chris Lintott, University of Oxford, Zooniverse
chris [at] zooniverse.org
https://doi.org/10.1045/may2017-vanhyning
Abstract
This article will showcase the aims and research goals of the project entitled "Transforming Libraries and Archives through
Crowdsourcing", recipient of a 2016 Institute for Museum and Library Services grant. This grant will be used to fund the creation of
four bespoke text and audio transcription projects which will be hosted on the Zooniverse, the world-leading research
crowdsourcing platform. These transcription projects, while supporting the research of four separate institutions, will also function
as a means to expand and enhance the Zooniverse platform to better support galleries, libraries, archives and museums (GLAM
institutions) in unlocking their data and engaging the public through crowdsourcing.
Keywords: Crowdsourcing, Citizen Humanities, GLAM, Transcription, IMLS
1 Overview1
As libraries, museums, and other cultural repositories digitize their collections and place them online, the challenges of
transforming these materials into useful and searchable sources of information are becoming increasingly apparent. While OCR and
handwriting recognition technology have opened up some print and manuscript corpora, and image and voice recognition software
are improving daily, there are still many tasks that require human intervention. For these, volunteer crowdsourcing is a viable and
vibrant solution.
The Zooniverse is the world-leading research crowdsourcing platform, hosting over 50 active projects and over 100 projects total
since its inception in 2007. The projects cover diverse subject areas from astronomy to zoology, engage over 1.5 million registered
2
volunteers, and have produced data used in more than a hundred peer-reviewed articles. The Zooniverse also hosts the Project
Builder, a free platform through which anyone can build their own project. The Zooniverse grew from a single project developed at
the University of Oxford in 2007, and is now developed and managed by a team based in Oxford and at the Adler Planetarium in
Chicago and the University of Minnesota (see Zooniverse Team for a more complete list).
In late 2016, the Institute for Museum and Library Services awarded a National Leadership Grant titled "Transforming Libraries and
Archives through Crowdsourcing (LG-71-16-0028-16)" to the Adler Planetarium and its collaborators to support the work of the
Zooniverse. Through this grant-funded effort, the Zooniverse will further expand and enhance its platform to better support
galleries, libraries, archives, and museums (GLAM institutions) in unlocking their data and engaging the public through
crowdsourcing.
1.1 What Can Crowdsourcing Offer GLAMs?
In 2010, author and professor Clay Shirky delivered a rousing TED talk in which he used the phrase "cognitive surplus" to describe the
one trillion hours of leisure time humans collectively accumulate each year (a great deal of which is spent watching television),
which could be harnessed to advance human knowledge through civic engagement. He concluded that: "free cultures get what they
celebrate. [...If we] celebrate and support and reward the people trying to use cognitive surplus to create civic value [...] we'll be
able to change society".[1] One way that GLAMs can harness this cognitive surplus is through web-based crowdsourcing. What Shirky
was describing was a type of "social machine", which Tim Berners-Lee defined as "new form[s] of social processes" emergent from
the Web, and involving both human and machine components.[2]
Academic crowdsourcing invites members of the public to work with specialists to conduct research: for example, to transcribe
documents or add metadata to a collection of images, video or audio clips. This data is used in real science, social science, or
humanities investigations and should, ideally, lead to publication. Crowdsourcing within GLAMs may not always be oriented around a
specific research question or publication, but around making collections more accessible for future research and usability. GLAM
crowdsourcing can be the seedbed of future scholarly research.
GLAMs have been engaging volunteers with their collections for well over a century, usually by inviting select individuals into an
institution and training them to do work that cannot be done by staff due to time or money constraints. On-site volunteers often
build up valuable knowledge and skills and contribute a great deal to their chosen institutions, but training and supervising them
also poses challenges. There is a limit to how many volunteers can be trained, supported on site, and indeed attracted and retained
in the first place. Online volunteering, enabled by crowdsourcing platforms such as Zooniverse.org, offer an alternative or
complementary form of engagement that has many benefits. Online projects can reach a wider range of individuals, including those
who are less able-bodied or geographically remote from the institution in which they want to volunteer and/or unable to travel.
Such projects require less training and time commitment from volunteers and typically attract a larger number of participants than
on-site programs. They also enable GLAMs to open up rare collections to the public without concern for their material safety and
3
security.
While crowdsourcing projects have proliferated in the last decade, few offer easy to use, open source, and free platforms on which
GLAM academics and amateur users can rely. The Zooniverse has the infrastructure, community, and technical expertise to
intervene at this critical stage.
1.2 How Does The Zooniverse Work?
All bespoke Zooniverse projects, including those built on the free Project Builder, have a few core components. Each image, audio
or video file (data point) in each project is independently assessed by multiple individuals, whose responses are then aggregated
using a variety of algorithms to determine what is in a given image. The amount of required responses for a task to be considered
"complete" varies, depending on the project. With relatively quick tasks, such as animal identification in Snapshot Serengeti,
upwards of 70 people will see each image. In tasks that require more time, such as transcription projects like Shakespeare's World
and AnnoTate, at least three people transcribe each line on each page. If enough people transcribe the same line and our algorithms
deem the line to be completed to a good enough standard, these are greyed out, while outstanding lines are available to future site
visitors. This approach was designed along the same principles that underpin all other Zooniverse projects, in which it is assumed
that volunteers should work independently on tasks, in order that no one individual should have undue influence over others in the
crowd. In the current IMLS project, however, we will test whether allowing volunteers to transcribe and work collaboratively
ultimately creates better data and/or better user experiences. We will be able to compare datasets from AnnoTate and
Shakespeare's World with text transcription datasets from the two new bespoke text transcription projects and, hopefully, with
datasets generated at other institutions that have online crowdsourcing projects. Zooniverse is in a unique position in being able to
gather these two very different kinds of data and compare them in order to determine the best outcomes. These findings will
ultimately drive our design of free tools on the Project Builder.
In addition to participating in the classification task, users have the opportunity to communicate with other volunteers through an
active, object-oriented discussion forum, called "Talk", associated with each project. Here volunteers can ask questions, interact
with researchers and fellow volunteers, create their own "collections", and use hashtags to group together posts or images of
interest. An example of the latter is #female from the Science Gossip project, which indicates female authors, illustrators and
printers contributing to the main scientific journals in the nineteenth century (visit the Science Gossip Talk board to view the
discussion around this tag). These interactions provide a rich set of experiences that allow users to personally experience the
community in which they are participating, beyond simply providing classifications. Additionally, the collections allow volunteers to
create their own research focal points within existing projects. During the process of transcribing, users can save images that
contain content that is pertinent to their research interests by adding them to a public collection. They can then use the Talk forum
to publicize their search, allowing other users to add images to that collection as well. In this way, the volunteer base can be
mobilized to help other volunteers with minimal effort required.
2 IMLS Funded Effort: Approach and Focus
Through the IMLS grant, the Zooniverse will engage in a research and development program to identify and implement
crowdsourcing best practices in the arenas of text and audio transcription for the purposes of unlocking big data currently trapped
in GLAM sources that cannot be machine read. Though to date the majority of Zooniverse projects have been based in STEM fields
rather than in the humanities, several text transcription projects have already been hosted on the site. For example, the first
Zooniverse humanities project was Ancient Lives, which invited volunteers to transcribe ancient papyri one letter at a time using a
clickable keyboard on their screen: volunteers did not have to be fluent in ancient Greek, they only needed to character match.
Over 250,000 volunteers participated in the project, and made more than 1.5 million transcriptions between 2011 and 2014.[3]
Furthermore, the computational pipeline used to convert individual identified letters into consensus-based transcriptions will
benefit future classification projects attempting consensus letter or line sequence identifications.[4]
By 2018 we will build four bespoke projects, two projects for text transcription and two projects for audio transcription, identified
through open calls, in order to test, iterate, and research the efficacy of new and existing approaches (including within current
Zooniverse and other projects) in these arenas. We will also develop the foundation for a GLAM-friendly data pipeline to export data
from a Zooniverse project into GLAM collections. These functionalities are among those most frequently requested by GLAM
institutions. We will work closely with four different GLAM institutions to build these bespoke crowdsourcing projects and
functionalities. The text transcription open call closed in February 2017, with thirty-one submissions. The audio transcription open
call will occur in fall 2017 (see Call for Projects).
From the lessons learned in building these bespoke projects, we will explore adding new tools and functionality to the Project
Builder, which is freely available to any institution or user who wishes to lead a project. It is a flexible, powerful, and easy-to-use
resource for building crowdsourcing projects, with a wide range of potential applications for GLAM collections, including text
transcription. A basic text transcription tool is currently available, but will be refined through this grant effort. The Zooniverse has
previously used this model of building bespoke projects in order to learn which tools are most useful, before implementing these
tools in the Project Builder. We recognize that volunteers' time is precious, and are therefore unwilling to waste it with tools that
are not proven to extract data in an efficient, high quality, and useful form. We will also draw on lessons learned from previous
experiences supporting transcription projects through Zooniverse and other platforms. For example, Operation War Diary which
launched in 2014 to commemorate the outbreak of the First World War, is a partnership between the National Archives (UK), the
Imperial War Museum, and the Zooniverse, which invites users to tag and transcribe dates, times, places, and names found in British
WWI field diaries. Historian Richard Grayson has used the data to penetrate more deeply than ever before into records of soldiers'
daily lives on the front.[5] All of the Operation War Diary metadata will eventually be integrated into the National Archive
catalogues. The process of integrating new metadata into an existing catalogue can be complicated, raising an important question
for any GLAM specialist seeking to harness crowdsourcing at their institution. For instance, it is essential to ensure, before starting a
project, that the current content management system (CMS) supports the storage of additional metadata, such as large amounts of
free-text. If not, it then becomes necessary to use an external resource to make available the results from the crowdsourcing
project. Zooniverse can and will do more to facilitate GLAMs and research groups to use and store their data.
Over the course of the IMLS project, we will also address the following research questions:
Q1: How can crowdsourcing be deployed in the arenas of text and audio transcription and metadata extraction
for the purposes of unlocking big data currently trapped in GLAM sources that cannot be machine read? What
methods produce the best data and make for the best user experience?
Q2: Does the current Zooniverse methodology of multiple independent transcribers and aggregation render
better results than allowing volunteers to see previous transcriptions by others or indeed collaborate to create
a single transcription? How does each methodology impact the quality of data, as well as depth of analysis and
participation?
Q3: How can we extend our crowdsourcing expertise to more GLAM professionals and learn from them, in turn,
how to adjust the Zooniverse platform to best meet their research and curatorial needs?
2.1 Addressing Q1 (Crowdsourcing for GLAM)
Only a platform like the Zooniverse can systematically address a question such as Q1: the community that has developed within the
platform is made up of volunteers who move across projects, allowing us to trace the impact of differences between projects on the
same volunteers. Zooniverse also has the infrastructure to implement A/B split experiments within a single project. This allows us
to develop projects incorporating different practices which are specifically aimed at understanding different methodologies.
Through the bespoke text and audio transcription projects, we will expand on the lessons learned through current Zooniverse text
transcription projects, including Ancient Lives, AnnoTate, Old Weather, Measuring the ANZACs, Shakespeare's World, Science Gossip,
Decoding the Civil War, Orchid Observers and Operation War Diary, as well as from external text transcription projects including
Transcribe Bentham, FromthePage, and Scripto.
In the bespoke projects created through the IMLS grant, the features optimizing volunteer engagement and retention will include:
Volunteer choice: volunteers choose which document to transcribe and can transcribe as little as a single line or as much as
an entire document. We have found through AnnoTate and Shakespeare's World that allowing users to transcribe smaller
fragments of text (without being required to complete an entire page) mitigates against forced or uncertain readings. We
hypothesize and plan to fully test whether allowing microtasking helps to retain volunteers, giving them the chance to build
up their skills and not make forced readings.
Keeping the task simple: in Shakespeare's World and AnnoTate, volunteers drop points at the start and end of individual
lines of text (not grammatical sentences) and transcribe the text contained between these two points. They do not use XML
4
markup itself, which has proven to be a major repellent to participants in other text transcription crowdsourcing projects.
Instead, volunteers highlight words within the transcribed line and choose among different features (e.g., insertion,
deletion, expansion, etc.). We propose to use these tagged words in each line to create simple TEI markup on the back-end,
for output into commonly used CMSs such as Drupal and Omeka.
Narrowing the content focus to support sense-making: In Shakespeare's World, the first release (or "chapter") consists of
recipes and letters, with more genres to follow. This type of structured approach will be applied to the bespoke projects, as
this supports creation of narratives within diverse collections, which in turn enables subject experts to more easily foster,
and volunteers to contribute to, discussions in Talk.
Features optimizing best practice in regard to data production and management will include:
Reliable, Scalable, Open Source Code Infrastructure: The foundation for the Zooniverse platform that includes the Project
Builder is an application written in Ruby on Rails which supports a powerful Application Programming Interface (API). The
API serves subjects — images, video or audio — for classification by volunteers via a workflow defined by the project, and
receives and records these classifications into a database. The frontend Javascript web software presents user interfaces to
volunteers and supports the Project Builder. All Zooniverse code is open source and available through Github.
Data Ingestion into Zooniverse: In the current Project Builder, research teams can upload batches of 500 to 1000 subjects
(images, videos, or audio clips) at a time by simply dragging and dropping the files. For larger collections and for bespoke
projects, typically the research team provides a hard drive and the Zooniverse team uploads the subjects to the API.
Through the projects proposed here, we will create a system to better support direct ingestion of large subject sets through
a user-friendly web interface, adding functionality to the foundation we already have in place within the Project Builder.
Useful Output for Curation: The Smithsonian Transcription Center is regularly cited as being successful in regard to their
output being easily ingestible by CMSs.[6] Current Zooniverse transcription projects are not set up with this functionality.
Currently, through our Project Builder for image annotation/marking projects, research teams can download the raw
classification results (i.e. all classifications by all volunteers) as well as automatically-generated aggregated results that
include confidence measures on consensus. Through this IMLS-funded effort, we will work with Meghan Ferriter of the
Smithsonian Transcription Center, who is on our board of advisors, to design data outputs for full text transcription and full
audio transcription that are suitable for ingestion into different GLAM CMSs. A key aspect of this effort is to continue
exploring best practices and approaches for transcription aggregation and confidence metrics, building on our efforts with
AnnoTate, Shakespeare's World, etc.
2.2 Addressing Research Q2 (Independent vs. Collaborative Transcription)
Through the two bespoke text transcription projects, we will investigate the impact on transcription quality and volunteer
experience when volunteers transcribe in isolation versus with knowledge of how others have transcribed the same document.
In terms of measuring impact on transcription quality, we will compare the rate of accuracy for individuals who transcribe in
isolation on projects such as AnnoTate and Shakespeare's World versus individuals who see previous transcriptions. We will also
compare the rate of accuracy in aggregated results for lines transcribed only by those working in isolation versus for lines in which
all but the first transcriber sees previous transcriptions. In order to measure impact on volunteer experience, we will analyze the
user behavior statistics we gather, e.g., number of transcriptions completed in a given session, length of session, number of sessions
overall, sentiment analysis of discussion forum comments, etc.
There are numerous open questions in this experiment: Does knowledge of other individuals' or collective transcriptions lead
individuals down the wrong path? Is transcription more or less accurate if people work in isolation or with an awareness of other
people's work? Does making transcriptions visible increase retention as a result of highlighting that an individual's effort is part of a
broader community effort or have the opposite effect? What environment best promotes skills acquisition, i.e. improved
paleography?
2.3 Addressing Research Q3 (Feedback/Training)
We will provide numerous opportunities for input and feedback from and training for the GLAM community, specifically by working
closely with our advisory board and four GLAM project partners throughout. In 2018 we will host feedback sessions at GLAM
conferences and summer schools targeting GLAM institutions with collections for which text transcription, audio transcription, or
image annotation/marking are of interest (we will include image annotation/marking because those tools are already included via
the Project Builder). This will allow for input from a broader set of institutions on our decisions and approach for building new
functionality into the Project Builder. In 2018—2019 we will host training workshops for GLAM professionals in using the Project
Builder to build their own crowdsourcing projects, incorporate the results into their databases and research, and sustain and
nurture their online volunteer communities.
3 Future Steps: Community Engagement, Output & How to Get Involved
The IMLS-Funded Project "Transforming Libraries and Archives through Crowdsourcing" is still in its beginning stages. Currently, we
are in the process of selecting the first two bespoke crowdsourcing text transcription projects to be built and incorporated into the
Zooniverse platform. The detail of our research questions will evolve alongside these new transcription projects, and during the
research and development process we will use conference presentations and feedback sessions to gather input which can then guide
the overall project design. The open call for the two bespoke audio transcription projects will occur in the fall of 2017. At this
point, the bespoke text transcriptions will be in beta review, allowing us to take advantage of lessons learned through that first
round of new projects. We believe that this self-reflexive method will simultaneously benefit our ongoing project while offering new
tools and ideas to the larger GLAM and academic community.
We anticipate this proposed effort will produce two peer-reviewed publications. One article will focus on the methodology for
creating, processing, and evaluating the data produced by the new projects. The second will focus on the results of our research
exploring the impact of individual versus collaborative text transcription. We also note that all Zooniverse code is freely available
under a liberal open source license which serves as an additional or parallel form of publication.
GLAM organizations keen to develop their own crowdsourcing projects should explore the available documentation on how to build a
project and best practices for the design, launch and long term phases of a project. While building a project is easy and requires
relatively little technical support from Zooniverse or your institution, make sure you have the time to work with your resulting data,
and time to support your online volunteer commmunity. Advertising the project's existence should be a long-term task, to avoid a
plateau or potential drop-off of user participation. For example, Shakespeare's World received a bump in the number of daily
classifications after an article was published in The New Yorker in January of 2017, over a year after the project's launch date.[7]
However, it does not suffice to merely advertise the existence of a project; researchers need to engage with their users on a regular
5
basis. Zooniverse's Talk platform, social media such as blogging, Twitter, Instagram, and indeed in-person or on-site events all
provide important channels for engaging current or potential volunteers with your collections. We believe that GLAM organizations,
with their long history of volunteer engagement, have many of the skills to work effectively with online volunteers, and will benefit
in new ways through cooperation with the crowd.
In conclusion, while this project is specifically focused on text and audio transcription, it is our hope that the results, including the
new Project Builder tools and GLAM data pipeline, will ultimately be used across a variety of disciplines and domains. We hope to
facilitate future partnerships between GLAM institutions and volunteer communities around the world, thus extending the aims and
outcomes of the National Digital Platform funded through this generous IMLS grant into an international digital platform that will
benefit many individuals and institutions.
Notes
1
Part of this article appeared previously as a blog post for CILIP, The Library and Information Association. Material is
reproduced by express permission of CILIP.
2
For a partial list of publications, please visit https://www.zooniverse.org/about/publications.
3Further discussion of the use of crowdsourcing in GLAM contexts can be found in Melissa Terras, "Crowdsourcing in the
Digital Humanities", in A New Companion to Digital Humanities, eds. Susan Schreibman, Ray Siemens, and John
Unsworth (John Wiley & Sons, 2016), 420-438, particularly in the section entitled "The Growth of Crowdsourcing in
Cultural and Heritage Applications" (pp. 423-28). See also Crowdsourcing Our Cultural Heritage, ed. Mia Ridge
(Ashgate, 2014).
4
Causer and Terras, "Many Hands Make Light Work", p. 81: "It would be fair to say that for volunteers, the XML mark-up
complicates participation, and it has undoubtedly dissuaded many from participating more fully, or at all." For opinions
from the volunteers about the process, the authors additionally refer the reader to Causer and Valerie Wallace,
"Building a Volunteer Community: Results and Findings from Transcribe Bentham", Digital Humanities Quarterly 6.2
(2012).
5
Or, as Zephyr Frank, et al. put it: "Paid advertising can generate large numbers of clicks on a website. It cannot,
however, produce good metadata or newly uploaded material that is relevant to the scholarly questions posed by
academic researchers." "Crowdsourcing for Humanities Research" (2016) Project White Paper.
References
[1] Clay Shirky, "How Cognitive Surplus Will Change the World", June 2010.
[2] Tim Berners-Lee with Mark Fischetti, Weaving the Web: The Original Design and Ultimate Destiny of the World Wide
Web by its Inventor (San Francisco: Harper, 1999).
[3] "P.Oxy 5156, Plutarch Moralia 660C, 661B-C (Quaestiones Convivales IV PR., 1.2)", in The Oxyrhynchus Papyri, R.-L.
Chang et al., eds, vol. 78 (London, Egypt Exploration Society, 2012), 97-98.
[4] Alex C. Williams et al., "A Computational Pipeline for Crowdsourced Transcriptions of Ancient Greek Papyrus
Fragments", in IEEE International Conference on Big Data, October 2014.
https://doi.org/10.1109/BigData.2014.7004460
[5] Richard Grayson, "A Life in the Trenches? The Use of Operation War Diary and Crowdsourcing Methods to Provide an
Understanding of the British Army's Day-to-Day Life on the Western Front", British Journal for Military History, 2.2
(2016), 160-85.
[6] Katie Mika, "Transcription Tools: a survey by Katie Mika, NDSR Resident", Harvard University, Ernst Mayr Library Blog.
[7] Roberta Kwok, "Crowdsourcing For Shakespeare", The New Yorker, 16 Jan. 2017.
About the Authors
Victoria Van Hyning is a Junior Research Fellow at Pembroke College, and a British Academy Postdoctoral Fellow. Her current
project, 'Court to Convent: Early Modern English Catholic Women's Autobiography', will reveal how Catholic women articulated
selfhood in the period when it was illegal to practice Catholicism, 1535 to 1829. She is also the Humanities PI of Zooniverse.org, the
world leading academic crowdsourcing organization. Her projects include Science Gossip, Shakespeare's World and AnnoTate.
Samantha Blickhan is the IMLS Postdoctoral Fellow in the Department of Citizen Science at the Adler Planetarium, working on
transcription projects for the Zooniverse. She received her Ph.D. in Musicology from Royal Holloway, University of London, with a
thesis on the palaeography of British song notation in the 12th and 13th centuries. Her research interests include music and
perception, and their relationships with writing systems, technology and pedagogy.
Laura Trouille is co-Investigator for Zooniverse and Director of Citizen Science at the Adler Planetarium where she leads the
Zooniverse web development and Teen Programs teams. While earning her Ph.D. in astronomy in 2010 studying galaxy evolution, she
also earned the Center for the Integration of Research, Teaching and Learning's Delta certificate for STEM education research. As a
CIERA Postdoctoral Fellow at Northwestern University's CIERA Center for Astrophysics, she continued her research on active galaxies
as well as co-led the Computational Thinking in STEM project, bringing computational thinking and modeling curricular materials to
high school science and math teachers.
Chris Lintott is a professor of astrophysics at the University of Oxford, where he is also a research fellow at New College. He is the
principle investigator for Galaxy Zoo and the Zooniverse, and his own research focuses on novel modes of crowdsourcing for
anomaly detection.
Copyright ® 2017 Victoria Van Hyning, Samantha Blickhan, Laura Trouille and Chris Lintott
