Data Science: Questions and Answers

388 Pages·2016·5.06 MB·English

by  George A Duckett

Table of Contents
About this book
Machine Learning (95 questions)
Bigdata (30 questions)
Data Mining (28 questions)
Classification (28 questions)
Neuralnetwork (23 questions)
Statistics (19 questions)
Python (19 questions)
Clustering (15 questions)
R (14 questions)
Text Mining (14 questions)
NLP (13 questions)
Dataset (12 questions)
Efficiency (11 questions)
Algorithms (11 questions)
Hadoop (11 questions)
SVM (11 questions)
Tools (9 questions)
Recommendation (9 questions)
Visualization (9 questions)
Databases (8 questions)
Feature Selection (8 questions)
NoSQL (7 questions)
Predictive Modeling (7 questions)
Definitions (6 questions)
Education (6 questions)
Search (5 questions)
Similarity (5 questions)
Social Network Analysis (5 questions)
Time Series (5 questions)
Scalability (4 questions)
Beginner (4 questions)
Data Cleaning (3 questions)
Aws (3 questions)
Graphs (3 questions)
Cross Validation (3 questions)
Apache Spark (3 questions)
Categorical Data (2 questions)
Hierarchical Data Format (2 questions)
Xgboost (2 questions)
Sequence (1 question)
Copyright
About this book
This book has been divided into categories where each question belongs to one or more
categories. The categories are listed based on how many questions they have; the question
appears in the most popular category. Everything is linked internally, so when browsing a
category you can easily flip through the questions contained within it. Where possible
links within questions and answers link to appropriate places within in the book. If a link
doesn’t link to within the book, then it gets a special icon, like this .
Machine Learning
Skip to questions,
Wiki by user dawny33
Overview
From The Discipline of Machine Learning by Tom Mitchell:
The field of Machine Learning seeks to answer the question “How can we build
computer systems that automatically improve with experience, and what are the
fundamental laws that govern all learning processes?” This question covers a broad
range of learning tasks, such as how to design autonomous mobile robots that learn to
navigate from their own experience, how to data mine historical medical records to
learn which future patients will respond best to which treatments, and how to build
search engines that automatically customize to their user’s interests. To be more
precise, we say that a machine learns with respect to a particular task T, performance
metric P, and type of experience E, if the system reliably improves its performance P
at task T, following experience E. Depending on how we specify T, P, and E, the
learning task might also be called by names such as data mining, autonomous
discovery, database updating, programming by example, etc.
The following journals are dedicated to research in Machine Learning:
Journal of Machine Learning Research (Open Access)
Machine Learning
International Journal of Machine Learning and Cybernetics
International Journal of Machine Learning and Computing (Open Access)
International Journal of Machine Learning and Computing (Open Access)
Questions
Q: Use liblinear on big data for semantic analysis
Tags: machine-learning (Next Q), bigdata (Next Q)
I use Libsvm to train data and predict classification on semantic analysis problem. But
it has a performance issue on large-scale data, because semantic analysis concerns n-
dimension problem.
Last year, Liblinear was release, and it can solve performance bottleneck. But it cost too
much memory. Is MapReduce the only way to solve semantic analysis problem on big
data? Or are there any other methods that can improve memory bottleneck on Liblinear?
Tags: machine-learning (Next Q), bigdata (Next Q)
User: puffin-gdi
Answer by sean-owen
Note that there is an early version of LIBLINEAR ported to Apache Spark . See mailing
list comments for some early details, and the project site .
Answer by marc-claesen
You can check out vowpal wabbit . It is quite popular for large-scale learning and
includes parallel provisions.
From their website:
VW is the essence of speed in machine learning, able to learn from terafeature
datasets with ease. Via parallel learning, it can exceed the throughput of any single
machine network interface when doing linear learning, a first amongst learning
algorithms.
Tags: machine-learning (Next Q), bigdata (Next Q)
Q: Why Is Overfitting Bad in Machine Learning?
Tags: machine-learning (Prev Q) (Next Q), predictive-modeling (Next Q)
Logic often states that by overfitting a model, its capacity to generalize is limited, though
this might only mean that overfitting stops a model from improving after a certain
complexity. Does overfitting cause models to become worse regardless of the complexity
of data, and if so, why is this the case?
Related: Followup to the question above, “When is a Model Underfitted? ”
Tags: machine-learning (Prev Q) (Next Q), predictive-modeling (Next Q)
User: blunders
Answer by alex-i
Overfitting is empirically bad. Suppose you have a data set which you split in two, test
and training. An overfitted model is one that performs much worse on the test dataset than
on training dataset. It is often observed that models like that also in general perform worse
on additional (new) test datasets than models which are not overfitted.
One way to understand that intuitively is that a model may use some relevant parts of the
data (signal) and some irrelevant parts (noise). An overfitted model uses more of the
noise, which increases its performance in the case of known noise (training data) and
decreases its performance in the case of novel noise (test data). The difference in
performance between training and test data indicates how much noise the model picks up;
and picking up noise directly translates into worse performance on test data (including
future data).
Summary: overfitting is bad by definition, this has not much to do with either complexity
or ability to generalize, but rather has to do with mistaking noise for signal.
P.S. On the “ability to generalize” part of the question, it is very possible to have a model
which has inherently limited ability to generalize due to the structure of the model (for
example linear SVM, …) but is still prone to overfitting. In a sense overfitting is just one
way that generalization may fail.
Answer by rubens
Overfitting, in a nutshell, means take into account too much information from your data
and/or prior knowledge, and use it in a model. To make it more straightforward, consider
the following example: you’re hired by some scientists to provide them with a model to
predict the growth of some kind of plants. The scientists have given you information
collected from their work with such plants throughout a whole year, and they shall
continuously give you information on the future development of their plantation.
So, you run through the data received, and build up a model out of it. Now suppose that, in
your model, you considered just as many characteristics as possible to always find out the
exact behavior of the plants you saw in the initial dataset. Now, as the production
continues, you’ll always take into account those characteristics, and will produce very
fine-grained results. However, if the plantation eventually suffer from some seasonal
change, the results you will receive may fit your model in such a way that your predictions
will begin to fail (either saying that the growth will slow down, while it shall actually
speed up, or the opposite).
Apart from being unable to detect such small variations, and to usually classify your
entries incorrectly, the fine-grain on the model, i.e., the great amount of variables, may
cause the processing to be too costly. Now, imagine that your data is already complex.
Overfitting your model to the data not only will make the classification/evaluation very
complex, but will most probably make you error the prediction over the slightest variation
you may have on the input.
Edit: This might as well be of some use, perhaps adding dynamicity to the above
explanation :D
Answer by kim
That’s because something called bias-variance dilema . The overfitted model means that
we will have more complex decision boundary if we give more variance on model. The
thing is, not only too simple models but also complex models are likely to have dis-
classified result on unseen data. Consequently, over-fitted model is not good as under-
fitted model. That’s why overfitting is bad and we need to fit the model somewhere in the
middle.
Tags: machine-learning (Prev Q) (Next Q), predictive-modeling (Next Q)
Q: Is there any APIs for crawling abstract of paper?
Tags: machine-learning (Prev Q) (Next Q), data-mining (Next Q)
If I have a very long list of paper names, how could I get abstract of these papers from
internet or any database?
The paper names are like “Assessment of Utility in Web Mining for the Domain of Public
Health”.
Does any one know any API that can give me a solution? I tried to crawl google scholar,
however, google blocked my crawler.
Tags: machine-learning (Prev Q) (Next Q), data-mining (Next Q)
User: alex-gao
Answer by alex-i
Look it up on:
Google Scholar link
Citeseer link
If you get a single exact title match then you have probably found the right article, and can
fill in the rest of the info from there. Both give you download links and bibtex-style
output. What you would likely want to do though to get perfect metadata is download and
parse the pdf (if any) and look for DOI-style identifier.
Please be nice and rate-limit your requests if you do this.
Answer by cwharland
arXiv has an API and bulk download but if you want something for paid journals it will
be hard to come by without paying an indexer like pubmed or elsevier or the like.
Tags: machine-learning (Prev Q) (Next Q), data-mining (Next Q)
Q: Machine learning techniques for estimating users’ age based on
Facebook sites they like
Tags: machine-learning (Prev Q) (Next Q), python (Next Q)
I have a database from my Facebook application and I am trying to use machine learning
to estimate users’ age based on what Facebook sites they like.
There are three crucial characteristics of my database:
the age distribution in my training set (12k of users in sum) is skewed towards
younger users (i.e. I have 1157 users aged 27, and 23 users aged 65);
many sites have no more than 5 likers (I filtered out the FB sites with less than 5
likers).
there’s many more features than samples.
So, my questions are: what strategy would you suggest to prepare the data for further
analysis? Should I perform some sort of dimensionality reduction? Which ML method
would be most appropriate to use in this case?
I mainly use Python, so Python-specific hints would be greatly appreciated.
Tags: machine-learning (Prev Q) (Next Q), python (Next Q)
User: wojciech-walczak
Answer by cwharland
One thing to start off with would be k-NN. The idea here is that you have a user/item
matrix and for some of the users you have a reported age. The age for a person in the user
item matrix might be well determined by something like the mean or median age of some
nearest neighbors in the item space.
So you have each user expressed as a vector in item space, find the k nearest neighbors
and assign the vector in question some summary stat of the nearest neighbor ages. You can
choose k on a distance cutoff or more realistically by iteratively assigning ages to a train
hold out and choosing the k that minimizes the error in that assignment.
If the dimensionality is a problem you can easily perform reduction in this setup by single
value decomposition choosing the m vectors that capture the most variance across the
group.
In all cases since each feature is binary it seems that cosine similarity would be your go to
distance metric.
I need to think a bit more about other approaches (regression, rf, etc…) given the narrow
focus of your feature space (all variants of the same action, liking) I think the user/item
approach might be the best.
One note of caution, if the ages you have for train are self reported you might need to
correct some of them. People on facebook tend to report ages in the decade they were
born. Plot a histogram of the birth dates (derived from ages) and see if you have spikes at
decades like 70s, 80s, 90s.
Answer by therriault
I recently did a similar project in Python (predicting opinions using FB like data), and had
good results with the following basic process:
1. Read in the training set (n = N) by iterating over comma-delimited like records line-
by-line and use a counter to identify the most popular pages
2. For each of the K most popular pages (I used about 5000, but you can play around
with different values), use pandas.DataFrame.isin to test whether each individual in
the training set likes each page, then make a N x K dataframe of the results (I’ll call it
xdata_train)
3. Create a series (I’ll call it ydata_train) containing all of the outcome variables (in my
case opinions, in yours age) with the same index as xdata_train
4. Set up a random forest classifier through scikit-learn to predict ydata_train based on
xdata_train
5. Use scikit-learn’s cross-validation testing to tweak parameters and refine accuracy
(tweaking number of popular pages, number of trees, min leaf size, etc.)
6. Output random forest classifier and list of most popular pages with pickle (or keep in
memory if you are doing everything at once)
7. Load in the rest of your data, load the list of popular pages (if necessary), and repeat
step 2 to produce xdata_new
8. Load the random forest classifier (if necessary) and use it to predict values for the
xdata_new data
9. Output the predicted scores to a new CSV or other output format of your choosing
In your case, you’d need to swap out the classifier for a regressor (so see here:
http://scikit-
learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html ) but
otherwise the same process should work without much trouble.
Also, you should be aware of the most amazing feature of random forests in Python:
instant parallelization! Those of us who started out doing this in R and then moved over
are always amazed, especially when you get to work on a machine with a few dozen cores
(see here: http://blog.yhathq.com/posts/comparing-random-forests-in-python-and-
r.html ).
Finally, note that this would be a perfect application for network analysis if you have the
data on friends as well as the individuals themselves. If you can analyze the ages of a
user’s friends, the age of the user will almost certainly be within a year or two of the
median among his or her friends, particularly if the users are young enough to have built
their friend networks while still in school (since most will be classmates). That prediction
would likely trump any you would get from modeling–this is a textbook example of a
problem where the right data > the right model every time.
Good luck!
Answer by damienfrancois
Another suggestion is to test the logistic regression . As an added bonus, the weights
(coefficients) of the model will give you an idea of which sites are age-distriminant.
Sklearn offers the sklearn.linear_model.LogisticRegression package that is designed to
handle sparse data as well.
As mentionned in the comments, in the present case, with more input variables than
samples, you need to regularize the model (with
sklearn.linear_model.LogisticRegression use the penalty='l1' argument).
Tags: machine-learning (Prev Q) (Next Q), python (Next Q)
Q: Is Data Science just a trend or is a long term concept?
Tags: machine-learning (Prev Q) (Next Q), bigdata (Prev Q) (Next Q), statistics (Next Q),
databases (Next Q), education (Next Q)
I see a lot of courses in Data Science emerging in the last 2 years. Even big universities
like Stanford and Columbia offers MS specifically in Data Science. But as long as I see, it
looks like data science is just a mix of computer science and statistics techniques. So I
always think about this. If it is just a trend and if in 10 years from now, someone will still
mention Data Science as an entire field or just a subject/topic inside CS or stats. What do
you think?
Tags: machine-learning (Prev Q) (Next Q), bigdata (Prev Q) (Next Q), statistics (Next Q),
databases (Next Q), education (Next Q)
User: filipe-ferminiano
Answer by marco13
The one thing that you can say for sure is: Nobody can say this for sure. And it might
indeed be opinion-based to some extent. The introduction of terms like “Big Data” that
some people consider as “hypes” or “buzzwords” don’t make it easier to flesh out an
appropriate answer here. But I’ll try.
In general, interdisciplinary fields often seem to have the problem of not being taken
serious by either of the fields they are spanning. However, the more research is invested
into a particular field, the greater is the urge to split this field into several sub-topics. And
these sub-topics sonner of later have to be re-combined in new ways, in order to prevent
an overspecialization, and to increase and broaden the applicability of techniques that are
developed by the (over?)specialized experts in the different fields.
And I consider “Data Science” as such an approach to combine the expertise and findings
from different fields. You described it as
…a mix of computer science and statistics techniques
And indeed, several questions here aim at the differentiation between data science and