Data Mining and Knowledge Discovery via Logic-Based Methods: Theory, Algorithms, and Applications

371 Pages·2010·6.591 MB·English

by  Evangelos Triantaphyllou

DATA MINING AND KNOWLEDGE DISCOVERY
VIA LOGIC-BASED METHODS
Springer Optimization and Its Applications
VOLUME 43
ManagingEditor
PanosM.Pardalos(UniversityofFlorida)
Editor–CombinatorialOptimization
Ding-ZhuDu(UniversityofTexasatDallas)
AdvisoryBoard
J.Birge(UniversityofChicago)
C.A.Floudas(PrincetonUniversity)
F.Giannessi(UniversityofPisa)
H.D.Sherali(VirginiaPolytechnicandStateUniversity)
T.Terlaky(McMasterUniversity)
Y.Ye(StanfordUniversity)
AimsandScope
Optimizationhasbeenexpandinginalldirectionsatanastonishingratedur-
ing the last few decades. New algorithmic and theoretical techniques have
beendeveloped,thediffusionintootherdisciplineshasproceededatarapid
pace, and our knowledge of all aspects of the field has grown even more
profound.Atthe sametime, oneofthemoststriking trendsin optimization
is the constantly increasing emphasis on the interdisciplinary nature of the
field.Optimizationhasbeenabasictoolinallareasofappliedmathematics,
engineering,medicine,economicsandothersciences.
The series Springer Optimization and Its Applications publishes under-
graduate and graduate textbooks, monographs and state-of-the-art exposi-
tory works that focus on algorithms for solving optimization problems and
alsostudyapplicationsinvolvingsuchproblems.Someofthetopicscovered
includenonlinearoptimization(convexandnonconvex),networkflowprob-
lems, stochastic optimization, optimal control, discrete optimization, mul-
tiobjective programming, description of software packages, approximation
techniquesandheuristicapproaches.
Forothertitlespublishedinthisseries,goto
http://www.springer.com/series/7393
DATA MINING AND KNOWLEDGE DISCOVERY
VIA LOGIC-BASED METHODS
Theory, Algorithms,and Applications
By
EVANGELOSTRIANTAPHYLLOU
LouisianaStateUniversity
BatonRouge,Louisiana,USA
123
EvangelosTriantaphyllou
LouisianaStateUniversity
DepartmentofComputerScience
298CoatesHall
BatonRouge,LA70803
USA
[email protected]
ISSN1931-6828
ISBN978-1-4419-1629-7 e-ISBN978-1-4419-1630-3
DOI10.1007/978-1-4419-1630-3
SpringerNewYorkDordrechtHeidelbergLondon
LibraryofCongressControlNumber:2010928843
MathematicsSubjectClassification(2010):62-07,68T05,90-02
(cid:2)c SpringerScience+BusinessMedia,LLC2010
Allrightsreserved.Thisworkmaynotbetranslatedorcopiedinwholeorinpartwithoutthewritten
permission of the publisher (Springer Science+Business Media, LLC, 233 Spring Street, New York,
NY 10013, USA),except for brief excerpts in connection with reviews orscholarly analysis. Usein
connectionwithanyformofinformationstorageandretrieval,electronicadaptation,computersoftware,
orbysimilarordissimilarmethodologynowknownorhereafterdevelopedisforbidden.
Theuseinthispublicationoftradenames,trademarks,servicemarks,andsimilarterms,eveniftheyare
notidentifiedassuch,isnottobetakenasanexpressionofopinionastowhetherornottheyaresubjectto
proprietaryrights.
Printedonacid-freepaper
SpringerispartofSpringerScience+BusinessMedia(www.springer.com)
This book is dedicated to a number of individuals and groups of people for
different reasons. It is dedicated to my mother Helen and the only sibling I have,
mybrotherAndreas.ItisdedicatedtomylatefatherJohn(Ioannis)andlategrand-
fatherEvangelosPsaltopoulos.
The unconditional support and inspiration of my wife, Juri, will always be
recognizedand,fromthebottomofmyheart,thisbookisdedicatedtoher.Itwould
have never been prepared without Juri’s continuous encouragement, patience, and
unique inspiration. It is also dedicated to Ragus and Ollopa (“Ikasinilab”) for their
unconditionalloveandsupport.Ollopawashelpingwiththisprojectallthewayuntil
totheverylastdaysofhiswonderfullife.Hewillalwaysliveinourmemories.Itis
also dedicated to my beloved family from Takarazuka. This book is also dedicated
toourverynewinspirationofourlives.
Asisthecasewithallmypreviousbooksandalsowithanyfutureones,thisbook
isdedicatedtoallthose(andtheyaremany)whoweretryingveryhardtoconvince
me, among other things, that I would never be able to graduate from elementary
schoolorpasstheentranceexamsforhighschool.
Foreword
Theimportanceofhavingefficientandeffectivemethodsfordataminingandknow-
ledgediscovery(DM&KD),towhichthepresentbookisdevoted,growseveryday
andnumeroussuchmethodshavebeendevelopedinrecentdecades.Thereexistsa
great variety of different settings for the main problem studied by data mining and
knowledge discovery, and it seems that a very popular one is formulated in terms
of binary attributes. In this setting, states of nature of the application area under
considerationaredescribedbyBooleanvectorsdefinedonsomeattributes.Thatis,
bydatapointsdefinedintheBooleanspaceoftheattributes.Itispostulatedthatthere
existsapartitionofthisspaceintotwoclasses,whichshouldbeinferredaspatterns
ontheattributeswhenonlyseveraldatapointsareknown,theso-calledpositiveand
negativetrainingexamples.
ThemainprobleminDM&KDisdefinedasfindingrulesforrecognizing(clas-
sifying)newdatapointsofunknownclass,i.e.,decidingwhichofthemarepositive
and which are negative. In other words, to infer the binary value of one more
attribute, called the goal or class attribute. To solve this problem, some methods
have been suggested which construct a Boolean function separating the two given
setsofpositiveandnegativetrainingdatapoints.Thisfunctioncanthenbeusedasa
decisionfunction,oraclassifier,fordividingtheBooleanspaceintotwoclasses,and
so uniquely deciding for every data point the class to which it belongs. This func-
tioncanbeconsideredastheknowledgeextractedfromthetwosetsoftrainingdata
points.
It was suggested in some early works to use as classifiers threshold functions
defined on the set of attributes. Unfortunately, only a small part of Boolean func-
tionscanberepresentedinsuchaform.Thisiswhythenormalform,disjunctiveor
conjunctive(DNForCNF),wasusedinsubsequentdevelopmentstorepresentarbi-
traryBooleandecisionfunctions.Itwasalsoassumedthatthesimplerthefunction
is (that is, the shorter its DNF or CNF representation is), the better classifier it is.
Thatassumptionwasoftenjustifiedwhensolvingdifferentreal-lifeproblems.This
booksuggestsanewdevelopmentofthisapproachbasedonmathematicallogicand,
especially,onusingBooleanfunctionsforrepresentingknowledgedefinedonmany
binaryattributes.
viii Foreword
Next, let us have a brief excursion into the history of this problem, by visit-
ingsomeoldandnewcontributions.Thefirstknownformalmethodsforexpressing
logicalreasoningareduetoAristotle(384BC–322BC)wholivedinancientGreece,
thenativelandoftheauthor.Itisknownashisfamoussyllogistics,thefirstdeduc-
tive system for producing new affirmations from some known ones. This can be
acknowledgedasbeingthefirstsystemoflogicalrecognition.Alongtimelater,in
the17thcentury,thenotionofbinarymathematicsbasedonatwo-valuesystemwas
proposedbyGottfriedLeibniz,aswellasacombinatorialapproachforsolvingsome
relatedproblems.Lateron,inthemiddleofthe19thcentury,GeorgeBoolewrotehis
seminalbooksThemathematicalanalysisoflogic:beinganessaytowardsacalculus
fordeductivereasoningandAnInvestigationoftheLawsofThoughtonWhichare
FoundedtheMathematicalTheoriesofLogicandProbabilities.Thesecontributions
servedasthefoundationsofmodernBooleanalgebraandspawnedmanybranches,
includingthetheoryofproofs,logicalinferenceandespeciallythetheoryofBoolean
functions.Theyarewidelyusedtodayincomputerscience,especiallyintheareaof
thedesignoflogiccircuitsandartificialintelligence(AI)ingeneral.
Thefirstreal-lifeapplicationsofthesetheoriestookplaceinthefirstthirtyyears
ofthe20thcentury.ThisiswhenShannon,NakashimaandShestakovindependently
proposedtoapplyBooleanalgebratothedescription,analysisandsynthesisofrelay
devices which were widely used at that time in communication, transportation and
industrialsystems.Theprogressinthisdirectionwasgreatlyacceleratedinthenext
fifty years due to the dawn of modern computers. This happened for two reasons.
First,inordertodesignmoresophisticatedcircuitsforthenewgenerationofcom-
puters,newefficientmethodswereneeded.Second,thecomputersthemselvescould
be used for the implementation of such methods, which would make it possible to
realizeverydifficultandlabor-consumingalgorithmsforthedesignandoptimization
ofmulticomponentlogiccircuits.Later,itbecameapparentthatmethodsdeveloped
for the previous purposes were also useful for an important problem in artificial
intelligence, namely, data mining and knowledge discovery, as well as for pattern
recognition.
Such methods are discussed in the present book, which also contains a wide
reviewofnumerouscomputationalresultsobtainedbytheauthorandotherresearches
in this area, together with descriptions of important application areas for their use.
These problems are combinatorially hard to solve, which means that their exact
(optimal) solutions are inevitably connected with the requirement to check many
differentintermediateconstructions,thenumberofwhichdependsexponentiallyon
the size of the input data. This is why good combinatorial methods are needed for
theirsolution.Fortunately,inmanycasesefficientalgorithmscouldbedevelopedfor
finding some approximate solutions, which are acceptable from the practical point
of view. This makes it possible to sufficiently reduce the number of intermediate
solutionsandhencetorestricttherunningtime.
A classical example of the above situation is the problem of minimizing a
Booleanfunctionindisjunctive(orconjunctive)normalform.Inthismonograph,this
task is pursued in the context of searching for a Boolean function which separates
two given subsets of the Boolean space of attributes (as represented by collections
Foreword ix
of positive and negative examples). At the same time, such a Boolean function is
desired to be as simple as possible. This means that incompletely defined Boolean
functionsareconsidered.Theauthor,ProfessorEvangelosTriantaphyllou,suggests
a set of efficient algorithms for inferring Boolean functions from training exam-
ples,includingafastheuristicgreedyalgorithm(calledOCAT),itscombinationwith
treesearchingtechniques(alsoknownasbranch-and-boundsearch),anincremental
learningalgorithm,andsoon.Thesemethodsareefficientandcanenableonetofind
good solutions in cases with many attributes and data points. Such cases are typi-
cal in many real-life situations where such problems arise. The special problem of
guidedlearningisalsoinvestigated.Thequestionnowiswhichnewtrainingexam-
ples (data points) to consider, one at a time, for training such that a small number
of new examples would lead to the inference of the appropriate Boolean function
quickly.
Special attention is also devoted to monotone Boolean functions. This is done
because such functions may provide adequate description in many practical situa-
tions.Theauthorstudiedexistingapproachesforthesearchofmonotonefunctions,
andsuggestsanewwayforinferringsuchfunctionsfromtrainingexamples.Akey
issueinthisparticularinvestigationistoconsiderthenumberofsuchfunctionsfora
givendimensionoftheinputdata(i.e.,thenumberofbinaryattributes).
Methods of DM&KD have numerous important applications in many different
domainsinreallife.Itisenoughtomentionsomeofthem,asdescribedinthisbook.
These are the problems of verifying software and hardware of electronic devices,
locating failuresinlogic circuits,processing of large amounts of data which repre-
sentnumeroustransactionsinsupermarketsinordertooptimizethearrangementof
goods,andsoon.OneadditionalfieldfortheapplicationofDM&KDcouldalsobe
mentioned,namely,thedesignoftwo-level(AND-OR)logiccircuitsimplementing
Boolean functions, defined on a small number of combinations of values of input
variables.
One of the most important problems today is that of breast cancer diagnosis.
Thisisacriticalproblembecausediagnosingbreastcancerearlymaysavethelives
ofmanywomen.Inthisbookitisshownhowtrainingdatasetscanbeformedfrom
descriptions of malignant and benign cases, how input data can be described and
analyzedinanobjectiveandconsistentmannerandhowthediagnosticproblemcan
beformulatedasanestedsystemoftwosmallerdiagnosticproblems.Alltheseare
doneinthecontextofBooleanfunctions.
The author correctly observes that the problem of DM&KD is far from being
fully investigated and more research within the framework of Boolean functions is
needed.Moreover,heofferssomepossibleextensionsforfutureresearchinthisarea.
Thisisdonesystematicallyattheendofeachchapter.
The descriptions of the various methods and algorithms are accompanied with
extensiveexperimentalresultsconfirmingtheirefficiency.Computationalresultsare
generatedasfollows.Firstasetoftestcasesisgeneratedregardingtheapproachto
betested.Nexttheproposedmethodsareappliedonthesetestproblemsandthetest
results are analyzed graphically and statistically. In this way, more insights on the
x Foreword
problem at hand can be gained and some areas for possible future research can be
identified.
The book is very well written in a way for anyone to understand with a mini-
mum background in mathematics and computer science concepts. However, this is
notdoneattheexpenseofthemathematicalrigorofthealgorithmicdevelopments.
Ibelievethatthisbookshouldberecommendedbothtostudentswhowishtolearn
about the foundations of logic-based approaches as they apply to data mining and
knowledge discovery along with their many applications, and also to researchers
whowishtodevelopnewmeansforsolvingmoreproblemseffectivelyinthisarea.
ProfessorArkadijZakrevskij
Minsk,Belarus
CorrespondingMemberoftheNationalAcademyof
SciencesofBelarus
Summerof2009