Search Frictions and the Design of Online Marketplaces

59 Pages·2015·3.52 MB·English

Search Frictions and the Design of Online Marketplaces
Andrey Fradkin
∗
MIT and Airbnb, Inc.
[email protected]
http://andreyfradkin.com/assets/SearchFrictions.pdf
September 30, 2015
Abstract
Online marketplaces increasingly act as intermediaries in labor, housing, dating,
and other markets where traders match with each other. These marketplaces use novel
data generated by users’ activities on the website to design algorithms and products
that influence the search and matching process. I use internal data from Airbnb, a
prominent online marketplace for housing rentals, to study the efficiency of this mar-
ket and the effects of ranking algorithms. I first show that potential guests engage in
limited search, are frequently rejected by hosts, and match at lower rates as a result. I
then estimate a micro-founded model of search and matching and use it to show that
if frictions were removed, there would be 102% more matches in the marketplace. I
propose and evaluate several ranking algorithms and show that a personalized algo-
rithm would increase the matching rate by up to 10% over the status quo. However,
due to equilibrium effects, the A/B search experiments favored by internet companies
can overstate the true treatment effect of an algorithm by over 100% in some cases.
∗Note, this paper is currently being split into two separate parts. I thank my advisors Jon Levin, Liran
Einav, CarolineHoxbyandLuigiPistaferriforguidanceandinspiration. IamgratefultoAirbnb(especially
theanalyticsanddatainfrastructureteams)forprovidingmewiththedataandforhelpingmeexecuteevery
aspectofthisproject. IthankJohnHorton, TimBresnahan, AliYurukolgu, RameshJohari, AlRoth, Doug
Bernheim, Przemek Jeziorski, Lanier Benkard, Frederic Panier, Dan Knoepfle, Itay Saporta-Eksten, Scott
Baker, Pete Troyan, Adam Vogel, Dan Preston and numerous seminar participants for comments. Financial
support was provided by the SIEPR Graduate Fellowship, the B.F. Haley and E.S. Shaw Fellowship for
Economics, the NET Institute, and Airbnb. The paper represents the views of the author and not Airbnb,
Inc.
1
1 Introduction
Online marketplaces have transformed how consumers search for jobs, apartments, spouses,
and consumer products. These marketplaces design products such as search ranking algo-
rithms, reputation systems, and user interfaces. However, little is known about the potential
of these policies to make marketplaces more efficient and the proper way to measure the
effects of these policies. I use detailed data on searches and transactions from Airbnb, a
prominent online marketplace for housing and accommodations, to model the efficiency and
the effects of marketplace policy. I propose three mechanisms that cause inefficiencies in
these markets: that consumers cannot consider all options available, that consumers don’t
know which sellers are willing to transact (due to seller screening, “stale vacancies,” or con-
gestion), and that some transactions may occur inefficiently early. I build an empirical model
that allows me to measure the impact of these frictions and use it to study platform changes
aimed at improving matching efficiency.
I find that frictions still play an important role on the internet and that marketplace
policy can reduce but not eliminate the negative consequences of those frictions. Searchers
onAirbnbtypicallyviewonlyasubsetofpotentialmatchesinthemarketandmorethan40%
of listings remain vacant for some dates. Furthermore, sellers reject proposals to transact
by searchers 49% of the time, causing searchers to leave the market although there are
potentiallygoodmatchesremaining.1 Frictionsinthemarketleadtolesstransactionvolume,
consumer surplus and revenue. If searchers had information about all the options in the
market and knew which sellers were willing to transact with them, there would be 102%
more matches and revenue per searcher would be $117 higher.
I propose and evaluate several new ranking algorithms aimed at improving market effi-
ciency. The preferred algorithm increases the match rate by 10% in counterfactual simula-
tions. However, because there is test-control interference and because the partial equilibrium
effect of a policy is not equivalent to the full equilibrium effect, the treatment effects of an
A/B test are generally not consistent for the true effect of a policy change. I also find that
ranking algorithms affect the relative contribution of searchers and sellers to the total num-
ber of matches in the market. This is especially important because, although the search
technology has changed in matching markets, most models of matching do not allow for the
shape of the aggregate matching function to change over time. Although I focus on rank-
ing algorithm design in this paper, my modeling framework can be used for other difficult
problems such as optimal marketing strategies, price recommendations and platform fees.2
1Horton [14] documents a similar negative effect of rejections on subsequent search effort in the context
of ODesk, an online labor market platform.
2Forexample,onedifficultissueforAirbnbhasbeentoascertainthemarginalvalueofahostversusguest
2
The setting of this paper is Airbnb (“Airbed and Breakfast”), a growing online market-
place for housing rentals that has served over 9 million guests between 2008 and late 2013.
Airbnb provides a platform for individuals to temporarily rent out an unused room or prop-
erty to travelers.3 Search on Airbnb begins when potential guests submit their trip details,
which consist of a city, dates of stay, and number of guests, into a search engine. The search
engine returns a set of listings in that city. If a listed option is appealing, the searcher sends
an inquiry to the host of that listing.4 An inquiry typically asks about the availability of the
listing and sometimes asks about other listing details. The host’s response is indicative of a
complex decision to let a stranger stay at her place. This decision combines issues of trust,
availability, hassle, and the possibility of better offers. If the host accepts, then the searcher
can choose to book the place.
Airbnb is an excellent environment for studying search and matching frictions for several
reasons. First, as in the labor and dating markets, both sides of the market have heteroge-
neous preferences towards the other side. Second, because hosts can only accommodate one
tripforasetofdates, thereisapotentialforlargecongestionfrictions. Burdettetal.[5]show
that markets where sellers have limited capacity (such as Airbnb, the dating market, and the
housing market) are characterized by different matching functions than settings where sellers
have a large capacity such as the college admissions market. Lastly, the Airbnb dataset con-
tains search, communication, and match behavior, whereas many datasets used for studying
search and matching markets lack search and communication data. An especially important
piece of information that I observe is whether the buyer or seller rejected the proposed trans-
action. Analogous data in the labor market would contain resume submissions, interview
invitations, and interview outcomes at a searcher-vacancy level.
I estimate models of consideration set formation, directed search, and rejection and com-
bine them into a simulation of market outcomes. The consideration set model determines
how many pages are seen by each searcher, which filters are applied on each page and what
listings are ultimately seen. This model is estimated using unique data from actual browsing
and filtering behavior of searchers on Airbnb. The directed search model determines which
listings from the consideration set are contacted by searchers. It is estimated using data
on inquiries sent from guests to hosts. In that model, the searcher’s utility for a particular
in a given market. This answer is impossible to answer well in a framework without cross-user externalities
and other network effects.
3SimilarmarketplacestoAirbnbexistinindustriessuchascontractlabor(ODesk,FreelancerandTaskrab-
bit),dating(OkCupid,Match.comandeHarmony),craftgoods(EtsyandEbay)andpersonaltransportation
(Uber and Lyft).
4Airbnb has been expanding its “Instant Book” feature, which allows searchers to book a listing without
communication.
3
listing is a function of listing characteristics such as location, size, and price and the match
quality between guest and host. For example, searchers for trips with more guests prefer
bigger properties and searchers who use a price filter are more price sensitive.
Upon receiving an inquiry, hosts choose whether to reject or accept a potential guest.
Rejectionbyhostshappensin49%ofallinquiries. Therearethreedistinctcausesofrejection
in search and matching markets: screening, congestion, and “stale” vacancies. Screening
occurswhenahostrejectsasearcherbecauseofthesearcher’spersonalortripcharacteristics.
For example, some hosts may not be comfortable sharing a space with an individual who
has no reviews. To capture the host’s decision process, I estimate a logistic regression of
hosts’ screening decisions as a function of guest and host characteristics. Congestion occurs
when a searcher sends an inquiry to a listing that is about to transact with another searcher.
Rejections due to congestion arise endogenously in my model because transactions take time
to clear. Lastly, rejections due to stale vacancies occur when a listing which is not actually
available for a given week is nonetheless visible in search and contacted.5 In my dataset,
over 20% of communications are screened, over 21% are sent to stale vacancies and less than
6% are affected by congestion.6 This result is interesting because much of the theoretical
literature on directed search such as Kircher [17], Albrecht et al. [2] and Burdett et al. [5]
has focused on congestion as the main cause of rejection in search markets.7
I combine the above models into a simulation of how the market clears over time. In the
simulation, searchers enter Airbnb looking for a place to stay in a given week and market,
and conduct search according to the directed search model. Inquiries sent by the searchers
are either accepted or rejected by hosts according to the reasons above. If a guest chooses to
book a property, the transaction takes time to clear. Additional searchers enter the market
sequentially and send inquiries. The actions of searchers and hosts then generate the aggre-
gate matching and rejection probabilities. The simulation comes within several percentage
points of matching the empirical booking rates, contact rates, and rejection frequencies even
though these moments were not used to determine the model parameters. This model is
similar to the model of online dating in Hitsch et al. [13], which includes directed search,
5Stale vacancies are similar to congestion from the perspective of the searcher but are generated by a
different seller behavior. Therefore, policies that reduce rejections due to congestion do not necessarily
reduce rejections due to stale vacancies and vice versa.
6There is no good data, to my knowledge, on these rejection reasons for other markets. Analogous
rejection reasons in the labor market are as follows. Screening rejections occur when a job applicant is not
qualifiedforthejob. Congestionrejectionsoccurwhenasuitableapplicantappliestoavacancythatisabout
to be filled by another applicant. Stale vacancies occur when a vacancy is listed although the employer is
not hiring.
7However, see Chade et al. [7] for an example where college applicants do not know which college will
accept them.
4
heterogeneous preferences, and screening. However, unlike in the setup in that paper, my
setupallowsmetoquantifyfrictions8 andtostudytheeffectofthemarketplacesearchengine
on outcomes.
I then use the model to evaluate the effects of counterfactual search ranking algorithms
on Airbnb. The baseline model I estimate uses a proxy for the ranking algorithm that
was running on Airbnb during the time of the estimation sample. In the counterfactual
policy experiments, I instead use one of three alternative ranking algorithms: one that
shows listings that are more relevant to searchers on average, one that shows listings tailored
to each searcher’s preferences, and one that shows listings which maximize each searcher’s
transaction probabilities. I show that all policies improve outcomes and that the later two
algorithms improve matching probabilities by 10%. I also simulate the results of the A/B
tests favored by internet platforms to determine which policies to pursue. I show that
the treatment effects estimated from A/B tests are typically different from the market level
treatment effects. This occurs for two reasons, first the treatment interferes with the control,
and second the partial equilibrium treatment effect is not equal to the the full equilibrium
treatment effect. These forces combine to overstate the true effects of a policy by over 100%
in some cases. To my knowledge, this is the first paper to quantitatively model the effect of
ranking algorithms in a search and matching market.9
The policy space for online marketplaces encompasses search ranking, site layout, new
matching mechanisms and explicit rules that affect the behavior of agents. Most market-
places rely on experimentation to learn about the effect of a particular policy. However,
experimentation must be used intelligently and in combination with other methods to learn
about the effects of policies in search and matching markets. Simulations like the one in the
paper can be used to quantify the potential bias of experiments, to find better experimental
designs, and to look for symptoms of test-control interference in the data. Second, because
the parameter space of possible marketplace policies is too large to explore with experimen-
tation alone, simulations can be used to choose among policies to be tested in the future.
Lastly, the effect of policies depends on time-varying market conditions such as the number
of agents in the market and the ratio of searchers to sellers. An experiment, on the other
hand, is typically run in a specific time period and set of markets. Understanding how mar-
ket conditions affect outcomes can inform experimental design and can help the marketplace
designer adapt policies to different situations.
8In Hitsch et al. [13], 68.3% of men in the sample never match with a partner. However, the paper does
not examine the extent to which those rejections are inefficient.
9GhoseandYang[11]andSantosandKoulayev[26]studytheeffectofrankingalgorithmsingenerichotel
search engines where seller preferences are unimportant. Horton [14], Casadesus-Masanell and Halaburda
[6] and Hagiu and Jullien [12] propose theoretical models where platforms have incentives to alter searchers’
5
The approach taken in this paper differs from the aggregate matching function approach
used in many papers concerning search markets (Petrongolo and Pissarides [23]). Aggregate
matching functions stipulate that the number of potential matches given a set of searchers
and vacancies is determined by a matching technology which is not, in general, based on
micro-foundations. The lack of micro-foundations makes aggregate functions unsuitable for
studying policies, technologies, and changes in market structure which affect the matching
process. For example, Lagos [20] derives a micro-founded aggregate matching function for
the taxicab market and shows that its shape will be sensitive to policy. Similarly, policies in
my model change the relative contribution of searchers and sellers in the matching function.
Specifically, a better ranking algorithm increases the weight on hosts and decreases the
weight on searchers in the matching function. This occurs because each searcher gets better
search results and is more likely to match holding the quality of hosts constant.
The paper is organized as follows. Section 2 gives more detail about Airbnb. Section 3
describes the model of directed search and section 4 describes the determinants of rejection.
Lastly, section 5 describes the model and shows the empirical results.
2 Data and Setting
2.1 Airbnb Background
Airbnb describes itself as a trusted community marketplace for people to list, discover, and
book unique accommodations around the world online or from a mobile phone.10 It is a
−
fast-growing startup that was founded in 2008 and that has more than doubled the number
of guests accommodated in every year of its operation. In 2012, Airbnb accommodated over
3 million guests and listed over 180 thousand new listings. It has listings in more than 34
thousand cities worldwide and is available in more than 30 languages.
Airbnb has created a market for a previously rare transaction: the rental of an apartment
or part of an apartment in a city for a short term stay by a stranger.11 These transactions
were not occurring previously because there were large costs to securely exchanging money,
communicating with strangers and evaluating a stranger’s trustworthiness. Airbnb was one
of the first platforms to provide a set of tools and services which enabled guests and hosts to
arrange stays in a relatively risk-free manner. These tools are important because hosts on
behavior through ranking algorithms.
10https://www.Airbnb.com/home/press
11Couchsurfing, a large travel based social network started in 2003, facilitates similar stays but without
monetary exchange. Craigslist has listed sublets and short-term vacation rentals since the late 1990’s.
Vacation rentals by owners in tourist towns have also existed for a long time.
6
Airbnbaretypicallynon-professionals,over80%ofwhomlistjustasingleroomorproperty.12
In a typical Airbnb transaction, guests and hosts have the ability to inconvenience each
other by being disruptive or by lying about the characteristics of a room or trip. Commu-
nication occurs on the platform both to screen counter-parties and to confirm the details of
the trip such as the key exchange and check-in times. Importantly, hosts are allowed to deny
potential guests for any reason. Guests and hosts have the opportunity to review each other
after the trip. In subsection 3.2, I show that the stock of reviews on the site provides impor-
tant information and affects both host and guest behavior.13 Users of Airbnb can disclose
information about themselves through photographs and textual descriptions. Airbnb also
provides free professional photography services to hosts and verifies users’ identity using on-
line social networks (Facebook, LinkedIn, and Twitter) and passports or driver’s licenses.14
In the empirical section I show that this information is important in the decisions of agents
on Airbnb.15
The payments mechanism on Airbnb is a big innovation compared to previous platforms
for accommodations such as Craigslist, where payments were handled offline. The market-
place holds payments in escrow until the stay has happened in order to prevent fraud. The
escrow service allows Airbnb to resolve disputes between guests and hosts and to enforce
the payment of security deposits and cancellation penalties. Airbnb also offers a one mil-
lion dollar insurance guarantee against major damage done to properties by guests. Airbnb
generates revenue by taking a percentage fee of every transaction that takes place on the
platform. Hosts typically have a 3% fee while guests have a variable fee that ranges between
6% and 12% depending on the details of the transaction.
Another important service that Airbnb provides is the management of pricing and prop-
erty calendars. For each date, each listing has a binary availability that is displayed on the
site. If a booking occurs, a host’s calendar is automatically blocked off for the dates of the
trip. Alternatively, the host can manually update the calendar to be unavailable for a set
of dates. In either case, the property can no longer be booked and will not show up in
search for the blocked off dates. Hosts might not update their calendars even if they are
unavailable because they are uncertain about future availability, want to see potential guests
or face updating costs. As a result, potential guests send inquiries to seemingly available
listings which are actually off the market.
12Infact,therearenowthirdpartyserviceslikeGuestyandAirenvythatmanageanAirbnbhost’slisting.
12There is a literature that documents the importance of reviews in a variety of online platforms. For
example,Pallais[22]showsthatreviewsarevaluableandunder-providedonOdesk,alabormarketplatform.
13For more details see: http://blog.airbnb.com/introducing-airbnb-verified-id/
14Lewis [21] shows that information voluntarily disclosed by sellers on Ebay Motors affects market prices.
7
2.2 The Search and Matching Process
Below is a list of steps each searcher undergoes before a match occurs. Even though the
examples below are from Airbnb, each of these steps occurs in other search and matching
marketplaces.
1. Using the Search Engine (Figure 1) - Searchers enter the travel dates, number of guests
and location into a search engine and receive a list of results. The search can then be refined
using filters and maps.
2. Investigation (Figure 2 and Figure 3) - The searcher clicks on a listing in search. The
subsequent page displays additional photos, amenities, reviews, responsiveness, house rules
and information about the host.
3. Communication (Figure 4) - The guest sends messages to hosts inquiring about room
details and availability. A host can respond by saying that the room is unavailable, that the
room is available or by asking a follow up question. The host might also not respond at all.
A guest may send multiple inquiries both initially and after receiving responses.
4. Booking - If a transaction is agreed upon, the guest can click the “Book It” button.
If the host accepts, the money is charged and taken in escrow by the platform. Some
transactions occur without a messaging step if the host allows the “Instant Book” feature.
5. Stay - After a trip is booked, there is further communication needed to exchange keys
and coordinate the details of the trip. Either side can cancel with a pre-specified penalty
rate.
2.3 Data Description
My dataset contains the full set of inquiries and transactions that occurred in major US
markets on Airbnb. Appropriate subsamples of the data are used throughout the paper. In
this section, I narrow my data set to comprise all contacts to US markets which occurred
for trips starting in 2012, while in the model sections I focus on one market and a narrower
set of dates. For each communication I observe the time a message was sent, who sent the
message, who received the message, and the content of the message. Each set of messages
between a guest and a host is aggregated into a “thread” that contains information about
trip dates, the number of guests, and the room of inquiry.
Igroupeachsetofcommunicationsinmydatasetbysearchspell(alternativelyreferredto
as a trip attempt). A search spell is defined as a set of inquiries and transactions pertaining
to a stay by a specific guest in a specific market and in a specific week of the start of the trip.
For each searcher, I observe the number of reviews, the number of pictures, demographics
and other characteristics. For each property and date I observe the listed price, number of
8
reviews, review score, location, number of pictures, and other characteristics.
There are some limitations to the dataset. I do not always observe the correct price in
cases when a host changes prices for a specific date over time. The reason for this is that the
data on date specific pricing is from monthly snapshots rather than from daily snapshots.
This is not a major problem because most hosts do not set date specific prices and instead
use a calendar wide listed price. Further, I do not observe the entire path of availability
in the market. For example, a property might initially be available for a given week, then
become unavailable and then become available again. In that case, I treat that property as
always being available. Lastly, some of the observed characteristics such as age and gender
are either missing or entered inaccurately by users.
2.4 Identifying Host Rejections
In this section I describe how to classify whether a host’s response to a guest indicates that
the host is interested in the transaction. For example, the response “Sorry, the place is
unavailable.” should be classified as a rejection whereas the response “Yes it is available. Go
ahead and book it.” should be classified as a non-rejection.
I use a combination of approaches to determine whether a response is a rejection. If an
inquiry led to a booking or was labeled as accepted by the host then it is classified as “not
rejected”. If a response was labeled by the host as a rejection or if there was no response
within a week then the response is classified as a rejection.
If the response does not fit into any of the above categories, a text classifier is applied to
the first (or in some cases second) response by a host to a guest. I use a common technique
in natural language processing called a regularized logistic regression to classify text (see
Appendix A for details). I combine the text classification with cases when the response is
discernible through other methods.16
In total, 49% of all inquiries were rejected. Of all responses classified as rejections, 37%
were non-responses, 30% were classified by the host, and 32% were classified the regularized
logistic regression. As a consequence, communication on Airbnb frequently fails to result
in a transaction. For US Markets in 2012, just 15% of inquiries and 48% of search spells
transact.
16Foreign language responses are removed from the analysis.
9
3 Search Behavior
3.1 Consideration Set Formation
Searchers in matching markets are often faced with hundreds, if not thousands, of choices.
People cannot possibly consider all of these options while making a decision. The purpose of
a model of consideration set formation is to specify which of the many options the searcher
considers before making a choice. These models are especially important in the case of
online marketplaces because search engines and recommendation algorithms affect outcomes
through their effect on consideration set formation. However, consideration sets are not
solely determined by the features of the marketplace. Searchers vary in the amount of pages
the view and the filters they apply while searching. In this section, I describe a way to use
data on search activity to model consideration set formation.
Potential guests on Airbnb search for a place to stay by entering a location, a set of
dates and the number of guests into a search engine. For each query, the website returns
a maximum of 21 available listings according to a ranking algorithm. Searchers can then
continuetothenextpageofresultsforagivenqueryortheycanmodifythesearchparameters
by using filters or a map (seen on the left side of Figure 1). These types of refinements are
frequently used on sites such as Airbnb, where the stock of listings is heterogeneous. Figure 5
displays filtering frequencies per search and searcher. Over 80% of searches and searchers
use at least one filter or map action. The most commonly used browsing actions (in order
of frequency) were room type filter, map, maximum price filter, neighborhood filter and the
next page of results. I do not model filters that are used less than 5% of the time such as
those pertaining to bedrooms, beds, and bathrooms.
Listings are ordered for each query according to their algorithmically determined scores,
with highest scores first. During the sample period, each score was a function of listing (but
not searcher) characteristics and the distance of the listing to the query. In a typical session,
searchers do not see all listings because there are hundreds or thousands of available listings
in a city.17 Although it is theoretically possible to obtain the exact ranking algorithm from
the code-base, it is infeasible in practice. The reason is that the search results are often the
result of a complex set of nonlinear transformations of underlying variables. Furthermore,
the variables used in the algorithm might not be logged on a high frequency basis.
It is, however, possible to estimate a proxy for the true ranking algorithm by using data
on what the actual search results. The dataset for estimating the algorithm is composed of
the set of listings that could have been displayed in search for a random sample non-filtered
17Searches in October of 2013 for stays in New York, Seattle, San Diego, Austin, Los Angeles, Chicago,
Miami and many other cities resulted in more than 1000 search results.
10