Knowledge Repo Documentation

21 Pages·2017·0.16 MB·English

Knowledge Repo Documentation
Release v0.7.6
Airbnb and contributors
Mar 17, 2018

Contents
1 Installation 1
2 Quickstart 3
2.1 SubmittingyourfirstKnowledgePost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
3 Deployment 5
3.1 CreatingKnowledgeRepositories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3.2 DeployingtheWebApplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
4 Workflows 9
4.1 Browsing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.2 Writing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
4.3 ProgrammaticInterface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
5 UndertheHood 13
5.1 TechnicalDetails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
6 Contributing 15
6.1 AddingsupportfornewKnowledgePostconversions . . . . . . . . . . . . . . . . . . . . . . . . . . 15
6.2 Addingextrastructureand/orverificationstotheknowledgepostconversionprocess . . . . . . . . . 15
6.3 Somethingelse? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
7 Feedback 17
i
ii
1
CHAPTER
Installation
Toinstalltheknowledgerepositorytooling(andallitsdependencies),simplyrun:
$ pip install --upgrade "knowledge-repo[all]"
Youcanalsoskipinstallingdependencieswhichareonlyrequiredinspecialcasesbyreplacingallwithoneormore
of the following (separated by commas): - ipynb : Installs the dependencies required for adding/converting Jupyter
notebookfiles-pdf : InstallsthedependenciesrequiredforuploadingPDFsusingthewebeditor-dev: Installsthe
dependenciesrequiredfordoingdevelopment,includingrunningthetests
Inadditiontotheknowledge_repoPythonlibrary,thisalsoinstallsaknowledge_reposcriptthatisusedtointeractwith
theKnowledgeRepositoryonthecommandline. Youcanverifythateverythingissetupappropriatelybyrunning:
$ knowledge_repo --version
ThisshouldshouldrunandshowthecurrentversionoftheKnowledgeRepo. Itmaythenthrowanexceptiondueto
a Knowledge Repository not being specified, but this is expected. If your shell environment cannot find the knowl-
edge_repoexecutable, youmayneedtocheckyour$PATH andensurethatyourPython’sexecutable“bin”folderis
onit.
IfyouareauserexpectingtointeractwithanexistingKnowledgeRepo,pleaserefertoQuickstart. Otherwise,ifyou
arelookingtocreateanewKnowledgeRepoinstallation,pleaserefertoDeployment.
1
KnowledgeRepoDocumentation,Releasev0.7.6
2 Chapter1. Installation
2
CHAPTER
Quickstart
ThissectionistargettedtowarduserswhoaresettingtheircomputersuptoworkwithanexistingKnowledgeRepo
installation, and guides a user through their first knowledge post submission. If you are looking to create a new
KnowledgeReporepositoryorserver,pleasereferinsteadtoDeployment.
2.1 Submitting your first Knowledge Post
ThereareseveralwaystocreateaKnowledgePost,ofwhichwhatfollowsisjustoneway. Formoreworkflows,refer
to: Writing.
Step1: Installtheknowledge_repotooling
Ifyouhavenotalreadydoneso,followtheinstallationinstructions: Installation.
Step2: Clonetheknowledgerepositorylocally[Gitrepositoriesonly]
Iftherepositorywithwhichyouaregoingtobecontributingisagitrepository,cloneitlocallyontoyourmachine.
Note: ifyouhavedirectaccesstoadatabaseknowledgerepository,thisstepisnotrequired.
$ git clone <git url> <local path>
Note: Ifyouarejusttestingtheworkflow,youcancreateatestgitrepositoryusing:
$ knowledge_repo --repo ./test_repo init
Note: Ifyouwillbeprimarilyusingasingleknowledgerepository,itispossibletoavoidpassingittotheknowl-
edge_repocommandeverytimebysettingtheKNOWLEDGE_REPOenvironmentvariable. Forexample:
export KNOWLEDGE_REPO=<repository uri/path>
3
KnowledgeRepoDocumentation,Releasev0.7.6
Forthistobepersistentaccrosssessions,addittoyourshellinitializationscript. IfKNOWLEDGE_REPOisset,and
pointstotheknowledgerepositorywithwhichyouwouldliketointeract,youcandropthe--repooptionsinthe
following.
Step3: CreateaKnowledgePosttemplate
KnowledgePosttemplatesaresamplefilesintheiroriginalformatwhichyoucanusetoavoidhavingtoremember
howmetadataisstoredand/oraddedtotheunderlyingpostdocuments.
ForaJupyternotebooktemplate:
$ knowledge_repo --repo <path/uri_of_repo> create ipynb example_post.ipynb
ForanRMarkdowntemplate:
$ knowledge_repo --repo <path/uri_of_repo> create Rmd example_post.Rmd
Othertemplatesmaybeavailable. Youcanseeallavailabletemplateshere.
Step4: Editthetemplateasnormal
Createwhatevercodecells/plots/LaTeX/etcthatyounormallywouldinyourwork.
Step5: Addyourposttotheknowledgerepository
$ knowledge_repo --repo <path/uri_of_repo> add example_post.ipynb -p project/
example_ipynb
˓→
$ knowledge_repo --repo <path/uri_of_repo> add example_post.Rmd -p project/example_
rmd
˓→
Notethatthe-poptionspecifiesthepathintherepositorytowhichthepostshouldbeadded.
Step6: Previewtheaddedpost
SometimesformattingmaydifferintheKnowledgeWebApplicationcomparedtothatshowninyournativeenvi-
ronment. Checkingthattherenderingiswhatyouexpectisagoodideabeforesubmittingitforpeerreview.
$ knowledge_repo --repo <path/uri_of_repo> preview <post_path_in_repository>
Step7: Submitpostforreview
Ifeverythinglooksgoodwhenpreviewed,thefinalstepofpostsubmissionissubmittingyourlocalpostsupstream
forreview,andultimately,publishing.
$ knowledge_repo --repo <path/uri_of_repo> preview <post_path_in_repository>
4 Chapter2. Quickstart
3
CHAPTER
Deployment
DeployingtheKnowledgeRepoinanorganizationisdoneintwosteps.
1. A knowledge repository (or repositories) must be created. These repositories are where knowledge posts are
pooledandmadeaccessibletousers.
2. DeploytheKnowledgeRepowebapplicationontopofthisrepository(orrepositories),whichthenactsasthe
primarygatewayforuserstoaccessthestoredknowledgeposts.
3.1 Creating Knowledge Repositories
The Knowledge Repo project supports multiple repository backends, all offering the same programmatic API (and
being subclasses of KnowledgeRepository). At this stage, two backends have been fully implemented: - GitKnowl-
edgeRepository: Arepositorybackedbyalocalgitrepository(optionallysyncedwithremotegitrepository). -Db-
KnowledgeRepository: A repository backed by a database backend (most databases supported by SQLAlchemy can
beused).
AllbackendsalsoallowconfigurationusingaPythonconfigurationfileat‘/.knowledge_repo_config’withintherepos-
itory. Atemplateforcreatingthisfileisavailablehere,ifonedoesnotalreadyexistortherepositoryconfigurationhas
grownoutofsyncwithupstreamchanges.
Warning: The.knowledge_repo_configfileisexecutedatruntimeonserverandonclientmachines. Assuch,a
malicioususerwithappropriatepermissionscouldsubmitarbitrarycodethatwillbeexecutedatruntimeonusers’
machines. UsingtheKnowledgeRepoinnon-trustedenvironmentsisnotrecommendeduntilthisisremedied.
3.1.1 Git Knowledge Repositories
Thefollowingcommandwillcreateanewrepositoryat<repo_path>:
$ knowledge_repo --repo <repo_path> init
5
KnowledgeRepoDocumentation,Releasev0.7.6
The result is a git repository at <repo_path> with a .knowledge_repo_config copied from the defaults found in the
repository source code. If a git repository was alreaedy found at <repo_path> it will upgrade it to be a knowledge
datarepository. ThisisusefulifyouarestartingarepositoryonaremoteservicelikeGitHub, asthisallowsyouto
clonetheremoterepositoryaspernormal;runthisscript;andthenpushtheinitializationbackintotheremoteservice
using git push. Otherwise, if you plan to host this repository on a remote service like GitHub, you must push this
repositorytothatremoteservice. ThiscanbedonebycreatingtheremoterepositorythroughGitHub(orwhichever
serviceyouplantouse)andthenrunning:
$ git remote add origin <url_of_remote_repository>
$ git push -u origin master
Users can then clone this repository, and point their local knowledge_repo script at it using --repo
<path_of_cloned_repository>.
3.1.2 Database Knowledge Repositories
Databaseknowledgerepositoriesaretheonlybackendsthatcurrentlyallowend-to-endpublishingofknowledgeposts
viathewebappinterface. Theyarecreatedondemand,wherepossible. Simplypointtheknowledge_reposcriptatit
using something akin to --repo mysql://username:password@hostname/database:table_name.
Ifthetabledoesnotexistitwillbecreatediftheactiveuserhastheappropriatepermissions.
Note: DatabaseKnowledge repositoriesalsosupporthavinga .knowledge_repo_configconfiguration, butoneisnot
automaticallyadded.
3.1.3 Repository Configuration
Asnotedearlier,allknowledgerepositorybackendssupportconfigurationviaaPythonfilethatisimportedfromthe
repository. Thisconfigurationcanoverridethedefaultsinthedefaultrepositoryconfigurationfoundhere.
Thisconfigurationfilewillallowyouto:
• Addpostprocessorstopostcontributionsfromtherepo. (seethepostprocessorsarrayoffunctions)
• Addrulesforwhichsubdirectoriespostscanbeaddedto. (seethepath_parse()function)
• Checkandmanagetheformatofauthornamesatcontributiontime
– Add logic to username_parse() to check post author names and raise exceptions when they don’t
match
– Addlogictousername_to_name()tomanagehowuser/authornamesaredisplayed,ex.“sally_smarts”
–>“SallySmarts”
– Add logic to username_to_email() to manage how user/author names are matched to emails, ex.
“sally_smarts”–>“[email protected]”
Pleaserefertothedefaultconfigurationfileitselfforfurtherdocumentation.
Note: Image handling is a good example of where post-processor configuration can be very useful. Knowledge
repositories’defaultbehavioristoaddthemarkdown’scontentsasistoyourknowledgepostgitrepository,including
images.IfyoudonothavegitLFSsetup,itmaybeinyourinteresttohavetheseimageshostedonsometypeofcloud
storage,sothatcloningthegitrepositorylocallyislesscumbersome.
Weprovideanexamplepostprocessorthataddssupportforpushingimagestocloudstorage. Touseit,simplyimport
orpasteitintoyour.knowledge_repo_configfile,andadditbynametoyourpostprocessorsconfigurationkey.
6 Chapter3. Deployment