Proceedings of the Linux Symposium

12 Pages·2008·0.16 MB·English

Reprintedfromthe
Proceedings of the
Linux Symposium
July 23rd–26th, 2008
Ottawa, Ontario
Canada
Conference Organizers
Andrew J. Hutton, Steamballoon, Inc., Linux Symposium,
Thin Lines Mountaineering
C. Craig Ross, Linux Symposium
Review Committee
Andrew J. Hutton, Steamballoon, Inc., Linux Symposium,
Thin Lines Mountaineering
Dirk Hohndel, Intel
Gerrit Huizenga, IBM
Dave Jones, Red Hat, Inc.
Matthew Wilson, rPath
C. Craig Ross, Linux Symposium
Proceedings Formatting Team
John W. Lockhart, Red Hat, Inc.
Gurhan Ozen, Red Hat, Inc.
Eugene Teo, Red Hat, Inc.
Kyle McMartin, Red Hat, Inc.
Jake Edge, LWN.net
Robyn Bergeron
Dave Boutcher, IBM
Mats Wichmann, Intel
Authorsretaincopyrighttoallsubmittedpapers,buthavegrantedunlimitedredistributionrights
toallasaconditionofsubmission.
‘Real Time’ vs. ‘Real Fast’: How to Choose?
PaulEMcKenney
IBMLinuxTechnologyCenter
[email protected]
Abstract 2.1 FuelInjection
Although the oft-used aphorism “real-time is not real- This rather fanciful fuel-injection scenario evaluates
fast”makesanicesoundbite,itdoesnotprovidemuch real-timeLinuxforcontrollingfuelinjectionforamid-
guidance to developers. This paper will provide the sizedindustrialenginewithamaximumrotationrateof
backgroundneededtomakeaconsidereddesignchoice 1500 RPM. This is slower than an automotive engine;
between“realtime”(gettingstartedasquicklyaspossi- whenall else isequal, larger mechanicalartifactsmove
ble) and “real fast” (getting done quickly once started). more slowly than do smaller ones. We will be ignoring
In many ways, “real fast” and “real time” are Aesop’s complicating factors such ascomputing how much fuel
tortoise and hare, respectively. But in the real world of istobeinjected.
real time, sometimes the race goes to the tortoise and
Ifwearerequiredtoinjectthefuelwithinonedegreeof
sometimesitgoestothehare,dependingontherequire-
topdeadcenter(thepointinthecombustioncyclewhere
mentsaswellasthedetailsoftheworkloadandtheen-
the piston is at the very top of the cylinder), what jitter
closingsoftwareenvironment.
canbetoleratedintheinjectiontiming? 1500RPMis25
RPS, which in turn is 9000 degrees per second. There-
1 Introduction
fore, a tolerance of one degree turns into a tolerance of
onenine-thousandthofasecond,orabout111microsec-
LinuxTM hasmademuchprogressinthereal-timearena
onds.
over the past ten years, particularly with the advent of
the -rt patchset [10], a significant fraction of which has Such an engine would likely have a rotational position
nowreachedmainline. Thisnaturallyleadstotheques- sensorthatmightgenerateaninterrupttoadevicedriver,
tionofwhichworkloadsgainimprovedperformanceby whichmightinturnawakenareal-timecontrolprocess.
running on real-time Linux. To help answer this ques- Thisprocesscouldthencalculatethetimeuntiltopdead
tion, we take a close look at the real-time vs. real-fast center for each cylinder, and then execute a sequence
distinctioninordertoproduceusefulcriteriaforchoos- of nanosleep() system calls to control the timing.
ingbetweenareal-timeandnon-real-timeLinux. The code to actuate the fuel injector might be a short
sequenceofmemorymappedI/O(MMIO)operations.
Section 2 looks at a pair of example applications in or-
der to make a clear distinction between real-time and Thisisaclassicreal-timescenario. Weneedtodosome-
real-fast,Section3examinessomefactorsgoverningthe thingbeforeadeadline,andfasterismostdefinitelynot
choice between real-time and real-fast, and Section 4 better. Injectingfueltooearlyisjustasbadasinjecting
givesanoverviewoftheunderlyingcausesofreal-time it too late. This situation calls for some benchmarking
Linux’s additional overhead. Section 5 lays out some and validation of the nanosleep() system call, for
simplecriteriatohelpchoosebetweenrealfastandreal example,withthecodeshowninFigure1. Oneachpass
time,andfinally,Section6presentsconcludingremarks. throughtheloop,lines2-5recordthestarttime,lines6-9
executethenanosleep()systemcallwiththespeci-
2 ExampleApplications fiedsleepduration,lines10-13recordtheendtime,and
lines14-16computethejitterinmicrosecondsandprint
This section considers a pair of diverse workloads, an it out. This jitter is negative if the nanosleep() call
embeddedfuel-injectionapplicationandaLinuxkernel did not sleep long enough, and positive if it slept too
build. long.
• 57 •
58 • ‘RealTime’vs.‘RealFast’: HowtoChoose?
1 for (i = 0; i < iter; i++) {
2 if (clock_gettime(CLOCK_MONOTONIC, &timestart) != 0) {
3 perror("clock_gettime 1");
4 exit(-1);
5 }
6 if (nanosleep(&timewait, NULL) != 0) {
7 perror("nanosleep");
8 exit(-1);
9 }
10 if (clock_gettime(CLOCK_MONOTONIC, &timeend) != 0) {
11 perror("clock_gettime 2");
12 exit(-1);
13 }
14 delta = (double)(timeend.tv_sec - timestart.tv_sec) * 1000000 +
15 (double)(timeend.tv_nsec - timestart.tv_nsec) / 1000.;
16 printf("iter %d delta %g\n", iter, delta - duration);
17 }
Figure1: LooptoValidatenanosleep()
It is important to use clock_gettime() with memory, as shown in Figure 3. The mlockall()
the CLOCK_MONOTONIC argument. The more- system call will lock all of the process’s current mem-
intuitive CLOCK_REALTIME argument to clock_ orydown(MCL_CURRENT),andallfuturemappingsas
gettime() means “real” as in real-world wall-clock well(MCL_FUTURE).
time, not as in real-time. System administrators and
Hardware irq handlers will preempt this code. How-
NTPcanadjustreal-worldwall-clocktime. Ifyouincor-
ever, the -rt Linux kernel has threaded irq handlers,
rectlyusegettimeofday()orCLOCK_REALTIME
which appear in the ps listing with names resem-
and the systems administrator sets the time back one
bling IRQ-16. You can check their priority using
minute, your program might fail to actuate the fuel in-
the sched_getscheduler() system call, or by
jectorsforafullminute,whichwillcausetheengineto
looking at the second-to-last field in /proc/<PID>
stop. Youhavebeenwarned!
/stat,where<PID>isreplacedbytheactualprocess
Before executing this validation code, it is first IDoftheirqthreadofinterest. Itispossibletorunyour
necessary to set a real-time scheduling priority, as real-timeapplicationatahigherprioritythanthatofthe
shown in Figure 2. Line 2-5 invokes sched_get_ threadedirqhandlers,butbewarnedthataninfiniteloop
priority_max() to obtain the highest possible insuchanapplicationcanlockoutyourirqs,whichcan
real-time (SCHED_FIFO) priority (or print an error) causeyoursystemtohang.
and lines 6-9 set the current process’s priority. You
If you are running on a multi-core system, another way
must have appropriate privileges to switch to a real-
to get rid of hardware-irq latencies is to direct them
time priority; either super-user or CAP_SYS_NICE.
to a specific CPU (also known as “hardware thread”).
Thereisalsoasched_get_priority_min()that
You can do this using /proc/irq/<IRQ>/smp_
gives the lowest priority for a given scheduler pol-
affinity, where<IRQ> isreplacedbytheirqnum-
icy, so that sched_get_priority_min(SCHED_
ber. You can then affinity your real-time program to
FIFO) returns the lowest real-time priority, allowing
someotherCPU,therebyinsulatingyourprogramfrom
applications to allocate multiple priority levels in an
interrupt latency. It may be necessary to pin various
implementation-independentmanner,ifdesired.
kernel daemons to subsets of the CPUs as well, and
However, real-time priority is not sufficient to obtain the schedutils taskset command may be used for
real-timebehavior,becausetheprogrammightstilltake this purpose (though care is required, as some of the
page faults. The fix is to lock all of the pages into per-CPU kernel daemons really do need to run on the
2008LinuxSymposium,VolumeTwo • 59
1 sp.sched_priority = sched_get_priority_max(SCHED_FIFO);
2 if (sp.sched_priority == -1) {
3 perror("sched_get_priority_max");
4 exit(-1);
5 }
6 if (sched_setscheduler(0, SCHED_FIFO, &sp) != 0) {
7 perror("sched_setscheduler");
8 exit(-1);
9 }
Figure2: SettingReal-TimePriority
1 if (mlockall(MCL_CURRENT | MCL_FUTURE) != 0) {
2 perror("mlockall");
3 exit(-1);
4 }
Figure3: PreventingPageFaults
1 tar -xjf linux-2.6.24.tar.bz2
corresponding CPU). This has the downside of pro-
2 cd linux-2.6.24
hibiting your real-time program from using all of the
3 make allyesconfig > /dev/null
CPUs,therebylimitingitsperformance. Thistechnique
4 time make -j8 > Make.out 2>&1
isnonethelessusefulinsomecases.
5 cd ..
6 rm -rf linux-2.6.24
Oncewehaveshutdownthesesourcesofnon-real-time
behavior, we can run the program on both a real-time
andanon-real-timeLinuxsystem. Inbothcases,werun Figure4: KernelBuildScript
on a four-CPU 2.2GHz x86 system running with low-
latencyfirmware.
2.2 KernelBuild
Even after taking all of these precautions, the non-real-
Since the canonical kernel-hacking workload is a ker-
time Linux fails miserably, missing the mark by up to
nel build, this section runs a kernel build on both a
3milliseconds. Non-real-timeLinuxsystemsarethere-
real-time and a non-real-time Linux. The script used
forecompletelyinappropriateforthisfuel-injectionap-
for this purpose is shown in Figure 4, featuring an 8-
plication.
way parallel build of the 2.6.24 Linux kernel given an
allyesconfig kernel configuration. The results (in
As one might hope, real-time Linux does much better.
decimalseconds)areshownonTable1,andasyoucan
Nanosleep always gets within 20 microseconds of the
see, real-time Linux is not helping this workload. The
requestedvalue,and99.999%ofthetimewithin13mi-
non-real-timeLinuxnotonlycompletedthebuildonav-
croseconds in a run of 10,000,000 trials. Please note
eragemorethan15%fasterthandidthereal-timeLinux,
that the results in this paper are from a lightly tuned
but did so using less than half of the kernel-mode CPU
system. Morecarefulconfiguration(forexample,using
time. Although there is much work in progress to nar-
dedicatedCPUs)mightwellproducebetterresults.
rowthisgap,someofwhichwilllikelybecompletebe-
fore this paper is published, there is no getting around
Ifreal-timeLinuxcansoeasilymeetsuchanaggressive
thefactthatthisisalargegap.
real-timeresponsegoal,itshoulddoextremelywellfor
moretypicalworkloads,right? Thisquestionistakenup Clearly, there are jobs for which real-time Linux is not
inthenextsection. therighttool!
60 • ‘RealTime’vs.‘RealFast’: HowtoChoose?
RealFast(s) RealTime(s)
1e+06
1350.4 1524.6
100000
RawData 1332.7 1574.2
s)
real 1314.5 1569.8 d
n 10000
o
Average 1332.6 1556.2 sec real fast
o
Std. Dev. 14.6 22.4 Micr 1000
3027.2 2940.9 cy (
en 100
RawData 3013.1 2982.2 Lat
real time
user 2996.1 2971.2 10
Average 3012.2 2964.7
1
Std. Dev. 12.7 17.5 1 10 100 1000 10000 100000
Duration of Task (Microseconds)
314.7 644.3
RawData 317.3 660.9 Figure 5: Real Time vs. Real Fast Against Work-Unit
sys 317.9 665.9 DurationforUser-ModeComputation
Average 316.6 657.0
Std. Dev. 1.4 9.2
1e+06
Table1: KernelBuildTimings
100000
s)
d
n 10000
o
2.3 Discussion sec real fast
o
cr 1000
Mi
y (
c
A key difference between these two applications is the en 100
at
L
duration of the computation. Fuel injection takes place real time
10
inmicroseconds,whilekernelbuildstakemanyseconds
orminutes. Inthefuel-injectionscenario, wearethere- 1
1 10 100 1000 10000 100000
forewillingtosacrificeconsiderableperformanceinor- Duration of Task (Microseconds)
der to meet microsecond-scale deadlines. In contrast,
Figure 6: Real Time vs. Real Fast Against Work-Unit
evenonaveryfastandheavilytunedmachine,handfuls
DurationforKernelBuild
ofmillisecondsaresimplyirrelevantonthekernel-build
timescale.
Thenextsectionwilllookmorecloselyattheseissues. small, a real-time system will out-perform a non-real-
time system by orders of magnitude. However, when
the duration of the unit of work exceeds a few tens of
3 FactorsGoverningRealTimeandRealFast
milliseconds,thereisnodiscernabledifferencebetween
thetwo.
In the previous section, we saw that the duration of the
work is a critical factor. Although there are a few ex- Furthermore, Figure 5 favors the real-time system be-
ceptions,real-timeresponseisusuallyonlyusefulwhen causeitassumesthatthereal-timesystemprocessesthe
performing very short units of work in response to a unit of work at the same rate as does the non-real-time
given real-time event. If the work unit is going to take system. However,inthekernel-buildscenariodiscussed
three weeks to complete, then starting the work a few in Section 2.2, the non-real-time Linux built the kernel
milliseconds late is unlikely to matter much. This re- 16.78% faster than did the real-time Linux. If we fac-
lationship is displayed in Figure 5 for work-unit dura- tor in this real-time slowdown, the non-real-time ker-
tions varying from one microsecond on the far left to nel offers slightly better overall latency than does the
100millisecondonthefarright,wheresmallerlatencies real-time kernel for units of work requiring more than
are better. The y-axis shows the total delay, including about ten milliseconds of processing, as shown in Fig-
theschedulinglatencyandthetimerequiredtoperform ure6. Thisbreakevenwouldvarydependingonthetype
the unit of work. If the unit of work to be done is quite of work. For example, floating-point processing speed
2008LinuxSymposium,VolumeTwo • 61
1e+06 ure8showstheresultingaverage,99.999percentilede-
lay, and maximum delay. The average jitter changes
100000
very little as we add tasks, which indicates that we are
ds) gettinggoodscalabilityfromareal-fastviewpoint. The
n 10000
o
ec real fast 99.999 percentile and maximum delays tell a different
s
o
Micr 1000 story, asbothincreasebymorethanafactorofthreeas
cy ( wegofromasingletaskto12paralleltasks.
en 100
at
L
real time Thisisakeypoint: obtainingthebestpossiblereal-time
10
response usually requires that the real-time system be
1 run at low utilization. This is in direct conflict with the
1 10 100 1000 10000 100000
Duration of Task (Microseconds) desire to conserve energy and reduce system footprint.
In some cases, it is possible to get around this conflict
Figure 7: Real Time vs. Real Fast Against Work-Unit
byputtingbothreal-timeandnon-realtimeworkloadon
DurationforHeavyI/O
thesamesystem,butsomecareisstillrequired. Toillus-
tratethis,runfourparalleldownloadsofakernelsource
treeontothesystem, thenunpackoneofthemanddoa
60
kernelbuild. Whenthenanosleeptestprogramruns
s) 50 atmaximumpriorityconcurrentlywiththiskernel-build
d
n
o workload,weseethe99.999%jitterat59microseconds
c
crose 40 max withtheworstcaseat146microseconds,whichisworse
mi
ep() jitter ( 2300 99.999% tmhaunlti-thmeillpiaseraclolenldrjuitntesr—sfbruotmsttihllemnounc-hreable-tttiemrethkaenrnethl.e
e
sl
o Advancingtechnologycanbeexpectedtoimprovereal-
n
na 10 average
time Linux’s ability to maintain real-time latencies in
0 face of increasing CPU utilization, and careful choice
0 2 4 6 8 10 12
of drivers and hardware might further improve the sit-
Number of Parallel Tasks
uation. Also, more-aggressive tuning might well pro-
Figure8: NanosleepJitterWithIncreasedLoad duce better results. For example, this workload does
not control the periodicity of the nanosleep() test
programs, so that all 12 instances might well try to run
wouldbelargelyindependentofthetypeofkernel(and simultaneously on a system that has but four CPUs. In
hence represented accurately by Figure 5), while heavy real-world systems, mechanical constraints often limit
I/O workloads would likely be profoundly affected by the number of events that can occur simultaneously, in
thekerneltype,asshowninFigure7,whichusesthe2- particular,enginesareconfiguredsothatitisimpossible
to-1increaseinkernel-buildsystemtimeasanestimate for all cylinders to fire simultaneously. That said, sites
of the slowdown. In this case, the crossover occurs at requiringthebestpossibleutilizationwilloftenneedto
aboutonemillisecond. sacrificesomereal-timeresponse.
Similarly, if you need to use virtualization to run mul-
In addition, a concern with worst-case behavior should
tiple operating-system instances on a single server, you
steer one towards real time, while a concern with
mostlikelyneedrealfastasopposedtorealtime. Again,
throughput or efficiency should steer one towards real
technology is advancing quite quickly in this area, es-
fast. Inshort,usereal-timesystemswhentheworktobe
pecially in the embedded space, so we may soon see
done is both time-critical and of short duration. There
production-quality virtualization environments that can
areexceptionstothisrule,buttheyarerare.
simultaneously support both real-time and real-fast op-
CPU utilization is another critical factor. To show this, erating systems. This is especially likely to work well
we run a number of the nanosleep() test programs if either: (1) CPUs and memory can be dedicated to
in parallel, with each program running 100,000 calls to a given operating instance or (2) the hypervisor (e.g.,
nanosleep in a loop (code shown in Figure 1). Fig- Linux with KVM) gives real-time response, but the
62 • ‘RealTime’vs.‘RealFast’: HowtoChoose?
guest operating systems need not do so. Longer term, queue. This locality reduces cache thrashing, yield-
itisquitepossiblethatboththehypervisorandtheguest ing extremely good performance and scalability, aside
OSeswillofferreal-timeresponse. from infrequent load-balancing operations. However,
real-timesystemsoftenimposetheconstraintthattheN
4 SourcesofReal-TimeOverhead highest-priority runnable tasks be running at any given
point in time, where N is the number of online CPUs.
The nanosleep() test program used the This constraint cannot be met without global schedul-
mlockall() system call to pin down memory ing, which re-introduces cache thrashing and lock con-
in order to avoid page-fault latencies. This is great for tention, degrading performance, especially on work-
this test program’s latency, but has the side-effect of loads with large numbers of runnable real-time tasks.
removing a chunk of memory from the VM system’s In the future, real-time Linux is likely to partition large
control, which limits the system’s ability to optimize SMP systems, so that this expensive global scheduling
memory usage. This can degrade throughput for some constraint will apply only within each partition rather
workloads. thanacrosstheentiresystem.
Real-time Linux’s more-aggressive preemption in- Real-time Linux requires high-resolution timers with
creases the overhead of locking and interrupts [2]. The tens-of-microseconds accuracy and precision, resulting
reasonfortheincreasedlockingoverheadisthatthecor- in higher-overhead timer management [3, 6]. However,
responding critical sections may be preempted. Sup- these high-resolution timers are implemented on a per-
pose that a given lock’s critical section is preempted, CPU basis, so that it is unlikely that this overhead will
and that each CPU subsequently attempts to acquire be visible at the system level for most workloads. In
the lock. Non-real-time spinlocks would deadlock at addition, real-time Linux distinguishes between real-
this point. The CPUs would each spin until they ac- time “timers” and non-real-time “timeouts,” and only
quired the lock, but the lock could not be released un- the real-time timers use new and more-expensive high-
til the lock holder got a chance to run. Therefore, resolution-timer infrastructure. Timeouts, for exam-
spinlock-acquisition primitives must block if they can- ple, TCP/IP retransmission timeouts, continue to use
notimmediatelyacquirethelock,resultinginincreased the original high-efficiency timer-wheel implementa-
overhead. The need to avoid priority inversion further tion, further reducing the likelihood of problematic
increases locking overhead. This overhead results in timeroverheads.
particularly severe performance degradation for some
Real-time Linux uses preemptible RCU, which has
disk-I/Obenchmarks,however,real-timeadaptivespin-
slightly higher read-side overhead than does Classic
locksmayprovidesubstantialimprovements[4]. Inad-
RCU [8]. However, the read-side difference is unlikely
dition, the performance of the user-level pthread_
to be visible at the system level for most workloads. In
mutex_lock() primitives may be helped by private
contrast,preemptibleRCU’supdate-side“grace-period”
futexes[5].
latency is significantly higher than that of RCU clas-
Threaded interrupts permit long-running interrupt han- sic [7]. If this becomes a problem, it should be possi-
dlers to be preempted by high-priority real-time pro- ble to expedite RCU grace period, albeit incurring ad-
cesses, greatly improving these processes’ real-time la- ditional overhead. It may then be possible to retire the
tency. However, this adds a pair of context switches ClassicRCUimplementation[9],butgiventhatClassic
to each interrupt even in absence of preemption, one RCU’s read-side overhead is exactly zero, careful anal-
to awaken the handler thread and another when it goes ysis will be required before such retirement can be ap-
back to sleep, and furthermore increases interrupt la- propriate.
tency. Devices with very short interrupt handlers can
In summary, the major contributors to the higher over-
specify IRQF_NODELAY in the flags field of their
head of real-time Linux include increased overhead of
struct irqaction to retain the old hardirq behav-
locking, threaded interrupts, real-time task scheduling,
ior, but this is not acceptable for handlers that run for
andincreasedRCUgrace-periodlatency. Thenextsec-
morethanasmallhandfulofmicroseconds.
tion gives some simple rules that help choose between
Linux’s O(1) scheduler is extremely efficient on SMP therealfastnon-real-timeLinuxkernelandthereal-time
systems, as a given CPU need only look at its own Linuxkernel.
2008LinuxSymposium,VolumeTwo • 63
5 HowtoChoose
Throughput Y
Only Goal?
Thechoiceofrealtimevs. realfastiseasedbyconsid-
N
eringthefollowingprinciples:
Peak Loads Y
Degrade
Response Time?
1. Consider whether the goal is to get a lot of work
done (real fast throughput), or to get a little bit of N
workdoneinapredictableanddeterministictime-
All Memory Y
frame(real-timelatency). Consumed?
2. Consider whether the hardware and software can N
accommodatetheheaviestpossiblepeakloadwith-
Virtualization Y
outmissingdeadlines(realtime),orwhetherocca-
Required?
sionalpeakloadswilldegraderesponsetimes(real
fast). It is common real-time practice to reserve N
some fraction of resources, for example, to limit
Basic Work Item Y
CPUutilizationto50%. >100ms? Real Fast
3. Consider memory utilization. If your workload
N
oversubscribesmemory,sothatpagefaultswilloc-
Real Time
cur,youcannotexpectreal-timeresponse.
4. If you use virtualization, you are unlikely to get Figure9: RealTimevs.RealFastDecisionFlow
real-timeresponse—thoughthismaybechanging.
5. Consider the workload. A process that executes
A rough rule-of-thumb decision flow is shown in Fig-
normal instructions in user mode will incur a
ure9. Ifyouonlycareaboutthroughput—theamountof
smaller real-time average-overhead penalty than
workcompletedperunittime—thenyouwantrealfast.
will a process that makes heavy use of kernel ser-
If cost, efficiency, or environmental concerns force you
vices.
torunathighCPUutilizationsothatpeakloadsdegrade
6. Focusonwork-itemcompletiontimeinsteadofon responsetimes,thenyouagainwantrealfast—andasa
start time. The longer the work item’s execution roughruleofthumb,themoreaggressiveyourreal-time
time,thelesshelpfulreal-timeLinuxwillbe. workload,theloweryourCPUutilizationmustbe. One
exceptiontothisoccursinsomescientificbarrier-based
computations,wherereal-timeLinuxcanreduceOSjit-
The need to focus on deterministic work-item comple- ter,allowingthebarriercomputationstocompletemore
tioncannotbestressedenough. Commonpracticeinthe quickly—andinthiscase,becausefloatingpointrunsat
real-timearenaistofocusonwhenthework-itemstarts, full speed on real-time Linux, this is one of those rare
in other words, on scheduling latency. This is under- cases where you get both real fast and real time simul-
standable, given the historic separation of the real-time taneously. Ifyourworkloadwillfillallofmemory,then
community into RTOS and real-time application devel- themlockall()systemcallbecomesinfeasible,forc-
opers,bothworkingonproprietaryproducts. Itishoped ingyoutoeitherpurchasemorememoryorallowthere-
that the advent of open-source real-time operating sys- sultingpagefaultsforceyoutogowithrealfast. Given
temswillmakeiteasierfordeveloperstotakethemore the current state of the art, if you need virtualization,
global viewpoint, focusing on the time required for the you are most likely in real-fast territory—though this
applicationtobothstartandfinishitswork. Pleasenote may soon be changing, especially for carefully config-
that it is important to focus on the proper level of de- ured systems. Finally, if each basic item of work takes
tail, for example, event-driven systems should analyze hundredsofmilliseconds,anyscheduling-latencybene-
deadlinesonaper-eventbasis. fitfromreal-timeLinuxislikelytobelostinthenoise.
64 • ‘RealTime’vs.‘RealFast’: HowtoChoose?
If you reach the real-time bubble in Figure 9, you may 3. ReducethepreemptableRCUgrace-periodlatency
need some benchmarking to see which of real time or penalty.
realfastworksbestforyourworkload. Nobenchmark-
4. Wherefeasible,adjustimplementationsothatper-
ingisneededtoseethataworkloadrequiring(say)100
formance penalties are incurred only when there
microseconds of processing with a 250-microsecond
areactuallyreal-timetasksinthesystem.
deadlinewillrequirereal-timeLinux,andthereappears
to be no shortage of applications of this type. In fact,
it appears that real-time processing is becoming more
It will also likely be possible to further optimize some
mainstream. This is due to the fact that the availability
ofthereal-timeimplementations. Inanycase,real-time
of real-time Linux has made it easier to integrate real-
Linux promises to remain an exciting and challenging
time systems into enterprise workloads [1], which are
areaforsometimetocome.
starting to require increasing amounts of real-time be-
havior. Wheretraditionalreal-timesystemswerestand-
alone systems, modern workloads increasingly require Acknowledgements
thatthereal-timesystemsbewiredintothelargerenter-
prise.
No article mentioning the -rt patchset would be com-
plete without a note of thanks to Ingo Molnar, Thomas
Gleixner, Sven Deitrich, K.R. Foley, Gene Heskett,
6 ConcludingRemarks
Bill Huey, Esben Neilsen, Nick Piggin, Steven Rost-
edt,MichalSchmidt,DanielWalker,andKarstenWiese.
Ifyourememberonlyonethingfromthispaper,letitbe
We all owe Ankita Garg, Vara Prasad, Ananth Mav-
this: “usetherighttoolforthejob!!”
inakayanahalli, Chirag Jog, and especially Darren Hart
and Paul Giangarra a debt of gratitude for their help in
Ongoingworktoreducetheoverheadofreal-timeLinux
makingthispaperhuman-readable. IamgratefultoPaul
willhopefullyreducetheperformancepenaltyimposed
Clarkefortheuseofhismachines,andtheABATteam
by the real-time kernel, which will in turn make real-
for providing easy access to these machines. Finally, I
time Linux the right tool for a greater variety of work-
owe thanks to Daniel Frye and Kathy Bennett for their
loads.
supportofthiseffort.
Mightreal-timeLinux’sperformancepenaltyeventually
be reduced to the point that real-time Linux is used for
LegalStatement
allworkloads? Thisoutcomemightseemquiteimpossi-
ble. Ontheotherhand,anynumberofimpossiblethings
This work represents the views of the authors and does not
havecometopassinmylifetime,includingspaceflight,
necessarilyrepresenttheviewofIBM.
computers beating humans at chess, my grandparents
usingcomputersonadailybasis,andasingleoperating-
LinuxisacopyrightofLinusTorvalds.
system-kernel source base scaling from cell phones to
supercomputers. Ihavesincelearnedtobeexceedingly Other company, product, and service names may be trade-
carefulaboutlabelingthingsimpossible. marksorservicemarksofothers.
Impossible or not, here are some challenging but rea-
sonable intermediate steps for the Linux kernel, some References
ofwhicharealreadyinprogress:
[1] BERRY, R. F., MCKENNEY, P. E., AND PARR,
F. N. Responsivesystems: Anintroduction. IBM
1. Reducethereal-timeperformancepenaltyformul- SystemsJournal47,2(April2008),197–206.
tiplecommunicationsstreams.
[2] CORBET, J. ApproachestorealtimeLinux.
2. Reduce the real-time performance penalty for Available:
mass-storageI/O.(Thisbecomesmoreurgentwith http://lwn.net/Articles/106010/
theadventofsolid-statedisks.) [ViewedMarch25,2008],October2004.