ARTiS, an Asymmetric Real-Time Scheduler for Linux - HAL - INRIA

36 Pages·2016·0.73 MB·English

ARTiS, an Asymmetric Real-Time Scheduler for Linux
on Multi-Processor Architectures
Éric Piel, Philippe Marquet, Julien Soula, Christophe Osuna, Jean-Luc
Dekeyser
To cite this version:
Éric Piel, Philippe Marquet, Julien Soula, Christophe Osuna, Jean-Luc Dekeyser. ARTiS, an Asym-
metric Real-Time Scheduler for Linux on Multi-Processor Architectures. [Research Report] RR-5781,
INRIA. 2005, pp.32. ￿inria-00070240￿
HAL Id: inria-00070240
https://hal.inria.fr/inria-00070240
Submitted on 19 May 2006
HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est
archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents
entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non,
lished or not. The documents may come from émanant des établissements d’enseignement et de
teaching and research institutions in France or recherche français ou étrangers, des laboratoires
abroad, or from public or private research centers. publics ou privés.
INSTITUTNATIONALDERECHERCHEENINFORMATIQUEETENAUTOMATIQUE
ARTiS, an Asymmetric Real-Time Scheduler for
Linux on Multi-Processor Architectures
Éric PIEL —Philippe MARQUET —Julien SOULA — Christophe OSUNA —Jean-Luc
DEKEYSER
N° 5781
November2005
ThèmeCOM
(cid:13)
G
N
E apport
+
R
F
1-- de recherche(cid:13)
8 (cid:13)
7
5
R--
R
A/
RI
N
N I
R
S
9 I
9
3
6
9-
4
2
0
N
S
S
I

ARTiS, an Asymmetric Real-Time Scheduler for Linux on
Multi-Processor Architectures
(cid:201)ric PIEL ,Philippe MARQUET ,Julien SOULA ,Christophe OSUNA ,
Jean-Luc DEKEYSER
ThŁmeCOM(cid:151)SystŁmescommunicants
ProjetDaRT
Rapportderecherche n(cid:176)5781(cid:151)November2005(cid:151)32pages
Abstract: The ARTiS system is a real-timeextension of the GNU/Linux scheduler dedi-
cated to SMP (Symmetric Multi-Processors) systems. It allows to mix High Performance
Computing andreal-time. ARTiSexploitsthe SMParchitecturetoguaranteethepreemp-
tionofaprocessorwhenthesystemhastoscheduleareal-timetask. Theimplementation
isavailableasamodi(cid:2)cationoftheLinuxkernel,especiallyfocusing(butnotrestrictedto)
IA-64architecture.
ThebasicideaofARTiSistoassignaselectedsetofprocessorstoreal-timeoperations.
Amigrationmechanismofnon-preemptibletasksinsuresalatencylevelonthesereal-time
processors. Furthermore, speci(cid:2)c load-balancing strategies permit ARTiS to bene(cid:2)t from
the full power of the SMP systems: the real-time reservation, while guaranteed, is not
exclusiveanddoesnotimplyawasteofresources.
This document describes the theoretical approach of ARTiS as well as the details of
the Linux implementation. Several kind of measurements are also presented in order to
validatetheresults.
Key-words: real-time, multi-processor architecture, Linux, scheduling, load-balancing,
taskmigration
ThisworkispartiallysupportedbytheITEAproject01010,HYADES
UnitéderechercheINRIAFuturs
ParcClubOrsayUniversité,ZACdesVignes,
4,rueJacquesMonod,91893ORSAYCedex(France)
Téléphone:[phone]—Télécopie:[phone]??
ARTiS, un ordonnanceur temps-rØel asymØtrique pour Linux
sur architectures multi-processeurs
RØsumØ : Le systŁme ARTiS est une extension temps-rØel de GNU/Linux dØdiØe aux
architecturesSMP(multi-processeurssymØtriques). Ilpermetdemixercalcul(cid:224)hauteper-
formanceettemps-rØel.ARTiSexploitelacaractØristiqueSMPdel’architecturepourgaran-
tir la possible prØemption d’un processeur quand le systŁme doit ordonnancer une t(cid:226)che
temps-rØel. L’implØmentation est disponible sous la forme d’une modi(cid:2)cation du noyau
Linux,visantenparticulier(sansŒtreunerestriction)l’architectureIA-64.
Leprinciped’ARTiSestd’identi(cid:2)erunensembledeprocesseursdØdiØsauxopØrations
temps-rØel.UnmØcanismedemigrationautomatiquedesactivitØsnonprØemptiblesassure
une garantie de latence sur ces processeurs temps-rØel. De plus, une stratØgie spØci(cid:2)que
d’Øquilibrage de charge permet (cid:224) ARTiS d’exploiter la pleine puissance d’une machine
SMP:lesrØservationstemps-rØel,bienquegaranties,nesontpasexclusivesetn’entra(cid:238)nent
pasdesous-utilisationsdesressources.
Nous prØsentons ici l’approche thØorique d’ARTiS ainsi que les dØtails de
l’implØmentationdansLinux. DiffØrentstypesdemesuressontØgalementsprØsentØsa(cid:2)n
devaliderlesrØsultats.
Mots-clØs : temps-rØel, architecture multi-processeurs, Linux, ordonnancement, Øquili-
bragedecharge,migrationdet(cid:226)che
ARTiS,anAsymmetricReal-TimeSchedulerforLinuxonMulti-ProcessorArchitectures 3
1 Introduction
Computersystemsneedtoprovidecontinuouslymorecomputationalpowertofollowthe
newapplicationsdemand. Hardwareparallelismisausualsolutiontobringmoreperfor-
mance. While it was historically restrained to super-computers, parallelism is becoming
used in every kind of hardwaredue to requirements to reduce energy consumption and
temperatureoftheprocessors[2].
Concurrently, as the computer is more frequently used in systems which require in-
teractivity with the external world (as opposed to computational purpose), the need for
real-timepropertiesincreases. Severalapplicationdomainsrequirehardreal-timesupport
of the operating system: The application contains tasks that expect to communicate with
dedicatedhardwareinatimeconstrainedprotocol,forexampletoinsurereal-timeacqui-
sition. Those same real-time applications require large amount of computational power:
Forexampleinthespectrumradiosurveillanceapplicationsusedtoanalyzethewaveform
signatures, the communications andthe coveragehaveanincreasingneedofpowerwith
theapparitionoftheUMTS(greaterbandwidthandmorecomplexalgorithms).
Historically,thenotionsofHighPerformanceComputingandofReal-Timehaveoften
beenconsideredantinomic, the latteronebeingmostly onlyassociatedtotoonly embed-
ded devices. Nowadays, this assumption cannot be hold true and there are applications
whichcanbene(cid:2)tfrombothpropertiesatthesametime. Toourknowledge,therearestill
nowellde(cid:2)nedsystemthatcanprovidebothbene(cid:2)tsatthesametime. Inthisarticle,we
willdescribeasoftwaresolutionbasedonmulti-processorcomputerwhichstrivestomake
thosebothpropertiescohabit.
1.1 Multi-Processing andReal-timeApproaches
The usage of SMP (Symmetric Multi-Processors) to face computational power need is a
wellknownandeffectivesolution. Ithasalreadybeenexperimentedinthereal-timecon-
text[1]. TotakeadvantageofanSMParchitecture,anoperatingsystemneedstotakeinto
accountthesharedmemoryfacility,themigrationandload-balancingbetweenprocessors,
andthecommunicationpatternsbetweentasks. Thecomplexityofsuchanoperatingsys-
temmakesitlookmorelikeageneralpurposeoperatingsystem(GPOS)thanadedicated
real-timeoperatingsystem(RTOS).AnRTOSonSMPmachinesmustimplementallthese
mechanismsandconsiderhowtheyinterferewiththehardreal-timeconstraints. Thismay
explainwhyRTOS’sarealmostmono-processordedicated.
IntheirreviewofcurrentRTOS’s,StankovicandRajkumar[20]describeafulltaxonomy
of OS’s. The OS’s developed from scratch are an endanger specie mainly because of the
complexity to implement all the features now required by developers. The management
ofan SMPmachine is partofthis dif(cid:2)culties. In our situation, the necessaryengineering
worktoprovidebothafullfeatureRTOSsystemandmanageanSMParchitecturewould
betoocostlyeitherintimeormoney.
RR n(cid:176)5781
4 (cid:201)ricPIEL,PhilippeMARQUET,JulienSOULA,ChristopheOSUNA,Jean-LucDEKEYSER
Anotherapproachistohaveare-usableOSfromwhichthedevelopercanselectamong
a set of components those that will be useful for the targeted hardware. RTEMS [16, 21]
isanexampletothis, itisanOpen-SourcededicatedRTOSthatsupportsmulti-processor
systems.Still,SMPsupportislimited,astasksareboundtoaCPUduringthedesignphase.
ResearchkernelsareOS’swhichweredesignedinordertopresentoneorseveralnew
paradigmstohandleagivenproblem. Althoughitmightbeagoodapproacheitherwhen
thecurrentsolutions areverypoororthe newparadigmwouldbemucheasiertounder-
stand or to use, it is not always ef(cid:2)cient to force users to entirely re-consider the system
organisation (for instance by providing a complete new API set or by introducing new
concepts).
The last approach we will describe is to add real-time extension to a GPOS. This has
the advantageof providing to the users all the facilities of the later one, including better
development softwares. The following subsection will detail the different alternatives of
thisapproachbyusingLinuxasoriginalGPOS.
1.2 Real-timeWith Linux
The Linux kernel is able to ef(cid:2)ciently manage SMP platforms, but it has never been de-
signed asanRTOS.Technically, only soft real-timetasksaresupported, via theFIFO and
round-robinschedulingpolicies. McKenney[12]hasdescribedindetailthebroadnumber
ofsolutions that(cid:3)ourished along the lastfewyears. In addition to thededicatedRTOS’s
whichemulatetheLinuxABI,approachesvaryfromtheLinuxkernelnestedandseparated
fromasmallRTOStoreal-timesupportdirectlywithinthekernel.
A well known solution that adds real-time capabilities to the Linux kernel is the so-
called co-kernel approach. These Linux extensions consist in a small real-time kernel that
providesthe real-timeservicesand which runs the standardLinux kernel asanested OS
byconsideringitthelowestprioritytask.TheinterruptsarereroutedtotheLinuxkernelby
thereal-timekernel;thisvirtualizationoftheLinuxkernelinterruptsallowstheco-kernel
topreemptthe Linuxkernelwhen needed. RTLinux [7, 23]andRTAI [5]aretwo famous
systems based on this principle. I-Pipe [10]is anew project which allows such co-kernel
system to be easily built-up. The main drawbacks are the necessity of developing real-
timeprogramsdealingwithtwodifferentOSinstances(withdifferentAPI)andthelimited
supportofSMParchitectures.
AsomewhatoppositesolutionistoattempttoimprovetheLinuxkernellatenciesbyim-
provingthekernelitself. TheembeddedLinuxvendorMontaVistahasintroducedarather
simple and systematic patch of the Linux kernel [13] to ensure some preemption points
in the kernel, and doing so, to reduce the latencies. This patch, called (cid:147)kernel preemp-
tion(cid:148)andmaintainedbyRobertLove,wasadoptedbythe mainstreamLinuxkernel[14],
mainly because it also implies a reduction of the latency targeted by multimedia appli-
cations. Ingo Molnar continues to work in this direction by developing a patch called
(cid:147)preempt-rt(cid:148) which focuses on hard real-time latencies. The objective is to allow every
partofthekerneltobepreempted,includingcriticalsectionsandinterrupthandlers. The
INRIA
ARTiS,anAsymmetricReal-TimeSchedulerforLinuxonMulti-ProcessorArchitectures 5
drawbackisthedegradationofperformanceforsomesystemcallsaswellasthehightech-
nicaldif(cid:2)cultytowriteandverifythosemodi(cid:2)cations.
The lastsolution thatwewillpresentreliesonthe shieldedprocessorsorAsymmetric
Multi-Processingprinciple(AMP).Onsuchasystem,whichisbasedonamulti-processor
machine,theprocessorsarespecializedtoreal-timeornot.ConcurrentComputerCorpora-
tionRedHawkLinuxvariant[3,4]andSGIREACTIRIXvariant[19]followthisprinciple.
It has the advantage of being designed from the ground with both the support of multi-
processor(whichcanbringHPC)andthe respectofreal-timeproperties. However, since
onlyRTtasksareallowedtorunonshieldedCPUs,ifthosetasksarenotconsumingallthe
available power then there is free CPU time which is lost. The ARTiS scheduler extends
thisapproachbyalsoallowingnormaltaskstobeexecutedonthoseprocessorsaslongas
theyarenotendangeringthereal-timeproperties.
In this article, we start by de(cid:2)ning the principles of ARTiS as well as its formal be-
haviour. ThenfollowsadescriptionofourARTiSimplementationintheLinuxkerneland
thedeploymentofthisimplementation. Finally, theforthandlastsection presentsexper-
imental validation of the (cid:2)nal implementation, focusing on three different aspects of the
system, the interrupt latencies, the execution time variation and the load-balancing cor-
rectness.
2 ARTiS: Asymmetric Real-Time Scheduler
ARTiS is a real-time Linux extension that targets SMPs. Furthermore, the programming
model ARTiS promotes a user-space programming of the real-time tasks: programmers
usetheusualPOSIXand/orLinuxAPItode(cid:2)netheirapplications. ARTiSreal-timetasks
arereal-timeinthesensethattheyareidenti(cid:2)edwithahighpriorityandarenotperturbed
byanynonreal-timeactivities.Forthesetasks,wearetargetingamaximumresponsetime
below 300(cid:0) s. This limit was obtained after a study by the industrial partners concerning
theirrequirements.
TheARTiSsolutionkeepstheinterestsofbothGPOS’sandRTOS’sbyestablishingfrom
the SMP platform an Asymmetric Real-Time Scheduler in Linux. ARTiS keeps the full
Linux facilities for each process as well as the SMP Linux properties but also improves
the real-time behavior. The core of the ARTiS solution is based on a strong distinction
betweenreal-timeandnon-real-timeprocessorsandalsoonmigratingtaskswhichattempt
todisablethepreemptiononareal-timeprocessor. Anexampleoftypicalarchitectureofa
systembasedonARTiSispresentedin(cid:2)gure1.
2.1 PartitionoftheProcessors andProcesses
Processorsarepartitionedintotwosets,anNRTCPUset(Non-Real-Time)andanRTCPU
set(Real-Time). Eachonehasaparticularschedulingpolicy. Thepurposeistoinsurethe
bestinterruptlatencyforparticularprocessesrunningintheRTCPUset.
RR n(cid:176)5781
6 (cid:201)ricPIEL,PhilippeMARQUET,JulienSOULA,ChristopheOSUNA,Jean-LucDEKEYSER
TwoclassesofRTprocessesarede(cid:2)ned. These arestandardRTLinuxprocesses, they
justdifferintheirmapping:
(cid:149) Each RT CPU has one or several RT Linux tasks bound to it, called RT0 (a real-
timetaskofhighestpriority). EachofthesetaskshastheguaranteethatitsRTCPU
will stay entirely available to it. Only these user tasks are allowed to become non-
preemptibleontheircorrespondingRTCPU.Thispropertyinsuresalatencyaslow
as possible for all RT0 tasks. The RT0 tasks are the hard real-time tasks of ARTiS.
ExecutionofmorethanoneRT0taskononeRTCPUispossiblebutinthiscaseitis
uptothedevelopertoverifythefeasibilityofsuchascheduling.
(cid:149) Each RT CPU can run other RT Linux tasks but only in a preemptible state. De-
pending on their priority, these tasks are called RT1, RT2... or RT99. To general-
ize, we call them RT1+. They can use CPU resources ef(cid:2)ciently if RT0 tasks do
not consume all the CPU time. To keep a low latency for the RT0 tasks, the RT1+
tasks are automatically migrated to an NRT CPU by the ARTiS scheduler when
they are about to become non-preemptible (when they call preempt_disable()
or local_irq_disable()). The RT1+ tasks arethe soft real-timetasks of ARTiS.
Theyhaveno(cid:2)rmguarantees,buttheirrequirementsaretakenintoaccountbyabest
effortpolicy. Theyarealsothemainsupportoftheintensiveprocessingpartsofthe
targetedapplications.
(cid:149) The other, non-real-time, tasks are named (cid:147)Linux tasks(cid:148) in the ARTiS terminology.
They arenot relatedto any real-timerequirements. They cancoexist with real-time
tasks and are eligible for selection by the scheduler as long as the real-time tasks
donotrequiretheCPU.AsfortheRT1+,theLinuxtaskswillautomaticallymigrate
awayfromanRTCPUiftheytrytoenterintoanon-preemptiblecodesectiononsuch
aCPU.
(cid:149) The NRT CPUs mainly run Linux tasks. They also run RT1+ tasks which are in a
non-preemptiblestate. Toinsuretheload-balancingofthesystem,allthesetaskscan
migratetoanRTCPUbutonlyinapreemptiblestate.WhenanRT1+taskrunsonan
NRTCPU,itkeepsitshighpriorityabovetheLinuxtasks.
ARTiSthensupportsthreedifferentlevelsofreal-timeprocessing:RT0,RT1+andLinux.
RT0tasksareimplementedinordertominimizethejitterduetonon-preemptibleexecution
onthesameCPU.Notethatthesetasksarestilluser-spaceLinuxtasks.RT1+tasksaresoft
real-timetasksbuttheyareabletotakeadvantageoftheSMParchitecture,particularlyfor
intensivecomputing. Eventually,Linux taskscanrunwithout intrusion on theRTCPUs.
Then they can use the full resources of the SMP machines. This architecture is adapted
to large applications made of several components requiring different levels of real-time
guaranteesandofCPUpower.
INRIA
ARTiS,anAsymmetricReal-TimeSchedulerforLinuxonMulti-ProcessorArchitectures 7
2.2 MigrationMechanism
A particular migration mechanism has been de(cid:2)ned. It aims at insuring the low latency
of the RT0 tasks. Allthe RT1+ andLinux tasks running on an RT CPU areautomatically
migratedtowardanNRTCPUwhenthey trytodisablethepreemption. Oneofthemain
changes which is required from the original Linux load-balancing mechanism is the re-
movalofinter-CPUlocks. Suchlocksareextremelydangerousforthereal-timeproperties
ifanRTCPUhavetowaitafteranNRTCPU.Toeffectivelymigratethetasks,anNRTCPU
andanRTCPUhavetocommunicateviaqueues. BasedontheworkbyValois[22],ARTiS
implementsalock-freeFIFOwithonereaderandonewritertoavoidanyactivewaitofthe
scheduler.
2.3 Load-Balancing Policy
An ef(cid:2)cient load-balancing policy allows the full power of the SMP machine to be ex-
ploited.Usuallyaload-balancingmechanismaimstomovetherunningtasksacrossCPUs
inordertoinsurethatnoCPUisidlewhiletasksarewaitingtobescheduled. Ourcaseis
morecomplicatedbecauseofthespeci(cid:2)citiesoftheARTiStasks. TheRT0taskswillnever
migrate,byde(cid:2)nition. TheRT1+tasksshouldmigratebacktoRTCPUsquickerthanLinux
tasks: the RT CPUs offerlatency warrantiesthat the NRT CPUsdo not. To minimize the
latencyonRTCPUsandtoprovidethebestperformancesfortheglobalsystem,particular
asymmetricload-balancingalgorithmshavebeende(cid:2)ned[18].
2.4 InterprocessCommunicationMechanisms
ARTiS includes asymmetric communication mechanisms. On SMP machines, tasks ex-
changedatabyread/writemechanismsonthesharedmemory. Toinsurecoherence,criti-
calsectionsareneeded.Thosecriticalsectionsareprotectedfromsimultaneousconcurrent
accessbylock/unlock mechanisms. Thiscommunication schemeisnotsuitedtoourpar-
ticular case: an exchange of data between an RT0 task and a RT1+ task will involve the
migration of the RT1+ taskbeforethis later takes on the lock, to avoid entering in a non-
preemptiblestateonanRTCPU.Therefore,anasymmetriccommunicationpatternshould
uselockfreeFIFOinaone-reader/one-writercontext.
3 Implementation
TheARTiSmodeliscurrentlyimplementedasamodi(cid:2)cationofthe2.6Linuxkernel. The
implementationhasbeensuccessfullytestedonIA-64andx86architectures. Theplatform
dependant part is very small, in the order of few code lines. Thereforea port to another
architecturealreadysupportedbyLinuxshouldbefairlyeasy.Thisimplementationworks
onSMPhardwareandonmulti-threadedprocessors(cid:150)allowingcomputerswithonlyone,
RR n(cid:176)5781