Table Of ContentDeriving Safety Cases from Machine-Generated Proofs
NurlidaBasirandBerndFischer EwenDenney
ECS,UniversityofSouthampton SGT,NASAAmesResearchCenter
SO171BJ,U.K MountainView,CA94035, U.S.A
(nb206r,b.fischer)@ecs.soton.ac.uk [email protected]
Abstract
Proofs provide detailed justification for the validity of claims and are widely used in formal
softwaredevelopmentmethods.However,theyareoftencomplexanddifficulttounderstand,because
theyusemachine-orientedformalisms;theymayalsobebasedonassumptionsthatarenotjustified.
Thiscausesconcernsaboutthetrustworthinessofusingformalproofsasargumentsinsafety-critical
applications. Here,wepresentanapproachtodevelopsafetycasesthatcorrespondtoformalproofs
foundbyautomatedtheoremproversandrevealtheunderlyingargumentationstructureandtop-level
assumptions.Weconcentrateonnaturaldeductionproofsandshowhowtoconstructthesafetycases
bycoveringtheprooftreewithcorrespondingsafetycasefragments.
1 Introduction
Demonstrating the safety of large and complex software-intensive systems requires marshalling large
amounts ofdiverse information, e.g., models, code ormathematical equations andformulas. Obviously
tools supported by automated analyses are needed to tackle this problem. For the highest assurance
levels,thesetoolsneedtoproduceatraceablesafetyargumentthatshowsinparticularwherethecodeas
wellastheargument itselfdepend onanyexternal assumptions butmanytechniques commonlyapplied
toensuresoftwaresafetydonotproduceenoughusableevidence(i.e.,justificationforthevalidityoftheir
claims)andcanthusnotprovideanyfurtherinsightsorarguments. Incontrast,informalsoftwaresafety
certification[3],formalproofsareavailableasevidence. However,theseproofsaretypicallyconstructed
by automated theorem provers (ATPs)based on machine-oriented calculi such as resolution [10]. They
are thus often too complex and too difficult to understand, because they spell out too many low-level
details. Moreover, the proofs may be based on assumptions that are not valid, or may contain steps
that are not justified. Consequently, concerns remain about using these proofs as arguments rather than
just evidence in safety-critical applications. In this paper we address these concerns by systematically
constructingsafetycasesthatcorrespondtoformalproofsfoundbyATPsandexplicitlyhighlighttheuse
ofassumptions.
The approach presented here reveals and presents the proof’s underlying argumentation structure
and top-level assumptions. We work with natural deduction (ND) proofs, which are closer to human
reasoningthanresolutionproofs. WeexplainhowtoconstructthesafetycasesbycoveringtheNDproof
tree with corresponding safety case fragments. The argument is built in the same top-down way as the
proof: itstartswiththeoriginaltheoremtobeprovedasthetopgoalandfollowsthedeductivereasoning
into subgoals, using the applied inference rules as strategies to derive the goals. However, we abstract
awaythe obvious steps toreduce the size of theconstructed safety cases. Thesafety cases thus provide
a“structured reading guide” for the proofs that allows users to understand the claims without having to
understand all the technical details of the formal proof machinery. This paper is a continuation of our
previous work toconstruct safety cases from information collected during the formal verification ofthe
code[2],buthereweconcentrate ontheproofsratherthantheverificationprocess.
1
DerivingSafetyCasesfromMachine-Generated Proofs Basir,FischerandDenney
2 Formal Software Safety Certification
Formal software safety certification uses formal techniques based on program logics to show that the
program does not violate certain conditions during its execution [3]. Asafety property isan exact char-
acterization oftheseconditions, basedontheoperational semanticsoftheprogramming language. Each
safetyproperty thusdescribes aclassofhazards. Thesafetyproperty isenforced byasafetypolicy, i.e.,
a set of verification rules that take initial set of safety requirements that formally represent the specific
hazardsidentifiedbyasafetyengineer[8],andderiveanumberofproofobligations. Showingthesafety
of a program is thus reduced to formally showing the validity of these proof obligations: a program is
considered safewrt. agivensafetyproperty ifproofs forthecorresponding safetyproof obligations can
be found. Formally, this amounts toshowing D∪A|=P⇒C for each obligation i.e., the formalization
of the underlying domain theory D and a set of formal certification assumptions A entail a conjecture,
whichconsists ofasetofpremisesPthathavetoimplythesafetyconditionC.
The different parts of these proof obligations have different levels of trustworthiness, and a safety
case should reflect this. The hypotheses and the safety condition are inferred from the program by
a trusted software component implementing the safety policy, and their construction can already be
explained in a safety case [2]. In contrast, both the domain theory and the assumptions are manually
constructed artifacts that require particular care. Inparticular, the safety case needs to highlight the use
ofassumptions. Thesehavebeenformulated inisolation bythesafety engineer andmaynotnecessarily
be justified, and are possibly inconsistent with the domain theory. Moreover, fragments of the domain
theory and the assumptions may be used in different contexts, so the safety case must reflect which of
them are available at each context. By elucidating the reasoning behind the certification process and
drawingattentiontopotentialcertificationproblems,thereislessofaneedtotrustthecertificationtools,
andinparticular, themanually constructed artifacts.
3 Converting Natural Deduction Proofs into Safety Cases
Naturaldeduction [6]systemsconsistofacollection ofproofrulesthatmanipulatelogicalformulasand
transform premises into conclusions. A conjecture is proven from a set of assumptions if a repeated
application of the rules can establish it as conclusion. Here, wefocus on some of the basic rules; a full
exposition oftheNDcalculus canbefoundintheliterature [6].
ConversionProcess. NDproofsaresimplytreesthatstartwiththeconjecture tobeprovenasroot,and
havegivenaxiomsorassumedhypothesesateachleaf. Eachnon-leafnodeisrecursivelyjustifiedbythe
proofs that start with its children as new conjectures. The edges between a node and all of its children
correspondtotheinferenceruleappliedinthisproofstep. Theprooftreestructureisthusarepresentation
of the underlying argumentation structure. Wecan use this interpretation topresent the proofs assafety
cases [7], which are structured arguments as well and represent the linkage between evidence (i.e., the
deductive reasoning oftheproofs from the assumptions tothederived conclusions) and claims (i.e.,the
original theorem to be proved). The general idea of the conversion from ND proofs to safety cases is
thus fairly straightforward. We consider the conclusion as a goal to be met; the premise(s) become(s)
the new subgoal(s). For each inference rule, we define a safety case template that represents the same
argumentation. The underlying similarity of proofs and safety cases has already been indicated in [7]
butasfarasweknow,thisideahasneverbeenfullyexploredorevenbeenappliedtomachine-generated
proofs(seeFigure1forsomeexamplerulesandtemplates). Here,weusetheGoalStructuringNotation
[7]astechnique toexplicitly represent thelogical flowoftheproof’sargumentation structure.
Implications. The implication elimination follows the general pattern sketched above but in the in-
troduction rule we again temporarily assume A as hypothesis together with the list of other available
2
DerivingSafetyCasesfromMachine-Generated Proofs Basir,FischerandDenney
H |= (A(cid:0)B) H |= B
Use implication to
Suppose premise J1: {A} can be used reduce goal to new
is true ((cid:0)i) apsro hvyep Bo thesis to s ubgoal ((cid:0)e)
H U {A} |= B H |= (A(cid:0)B) H |= A
=>-Rules
Safety Case Templates for =>-Rules
H |= " x A(x) H |= A (tx)
Show for an Show for all
eldeomamrebanitirtn ao (rf"y t ih) e Jfar1reb:s ithtrx a oisrby aj e nc t elemdoemntasi n(" e) Jre1p: laxc ceadn b bye t x
" """ -Rules H |= A (tx) H |= " x A(x)
Safety Case Templates fo"r """ -Rules
Figure1: SafetyCaseTemplatesforNaturalDeductionRules
hypoth eses, rather than deriving a proof for it. We then proceed to derive B, and discharge the hypoth-
esis by the introduction of the implication. The hypothesis A can be used at given in the prove of B,
but the conclusion A⇒B nolonger depends on the hypothesis A after Bhas been proved. In thesafety
casefragment,weuseajustificationtorecordtheuseofthehypothesisA,andthustomakesurethatthe
introduced hypotheses aretrackedproperly.
Universal quantifiers. The ND proof rules for quantifiers focus on the replacement of the bound
variables with objects and vice versa. Forexample, in the elimination rule for universal quantifiers, we
can conclude the validity of the formula for any chosen domain element t . In the introduction rule,
x
however,weneedtoshowitforanarbitary butfreshobjectt (thatis,adomainelementwhichdoesnot
x
appearelsewhereinH,A,orthedomaintheoryandassumptions). IfwecanderiveaproofofA,wherex
isreplacedbytheobjectt ,wecanthendischarge thisassumption byintroduction ofthequantifier. The
x
safetycasefragments recordthisreplacement asjustification. Thehypotheses available forthesubgoals
inthe∀-rulesarethesameasthoseintheoriginalgoals.
4 Hypothesis Handling
AnautomatedprovertypicallytreatsthedomaintheoryDandthecertificationassumptionsAaspremises
and tries to derive V(D∪A)∧P⇒C from an empty set of hypotheses. As the proof tree grows, these
premiseswillbeturnedintohypotheses, usingthe⇒-introduction rule(seeFigure1). Inallotherrules,
the hypotheses are simply inherited from the goal to the subgoals. However, not all hypotheses will
actually be used in the proof, and the safety case should highlight those that are actually used. This is
particularly important for the certification assumptions. We can achieve this by modifying the template
for the ⇒- introduction (see Figure 2a). We can distinguish between the hypotheses that are actually
used in the proof of the conclusion (denoted by A ,...,A ) and those that are vacuously discharged by
1 k
the ⇒-introduction (denoted byA +1,..,A ). Wecan thus usetwodifferent justifications tomarkthis
k n
distinction. Note that this is only a simplification of the presentation and does not change the structure
of the underlying proof, nor the validity of the original goal. It is thus different from using a relevant
implication [1]underwhichA⇒Bisonlyvalidifthehypothesis Aisactually used.
In order to minimize the number of hypotheses tracked by the safety case, we need to analyze the
proof tree from the leaves up, and propagate the hypotheses towards the root. By revealing only these
usedhypothesesasassumptions,thevalidityoftheiruseinderivingtheproofcanbecheckedmoreeasily.
3
DerivingSafetyCasesfromMachine-Generated Proofs Basir,FischerandDenney
C1:has_unit(float Hypothesis has_unit(float_7_0e_ Conclusion has_unit(float_7_0e_
H |= A1…(cid:217) Ak (cid:217) Ak+1(cid:217) …AN (cid:1) B _a7n_g0_eve_l )m isin auns _1, minus_1,ang_vel) is valid minus_1,ang_vel) is valid
externa l
hypoth esis
S{uAp1,p..o,As(ek(cid:1)} pai)rr eem trisuees uJJps12roe:: vd{{eAA t ko1B+, . 1p.,,…rAokv,}Ae uN Bs} e ndo tt o eAsrthagybuplmiosehthnmet esoinsvte or f AstrogMpu_umhsecyapnd_t e ortvu eler Jicihsso1y vvpn: aaocTllltiiuhhddsee ii fso itnsh e
Hypothesis is built by using The conclusion to be
ang_vel_7_0e_minus_1_ rule proved is an external
H U {A1,..,Ak} |= B from the valid axiom hypothesis
(Axiom)
ang_vel_7_0
e_minus_1_
a) b)
Figure2: Hypothesis Handling
Inourwork,wealsohighlighttheuseoftheexternalcertificationassumptionsthathavebeenformulated
inisolationbythesafetyengineer. Forexample,inFigure2b,thehypothesishas unit(float 7 0e minus 1,
ang vel),meaningthataparticular floatingpointvariablerepresents anangularvelocity, hasbeenspeci-
fiedasexternalassumption. Thisistrackedproperlyinthesafetycase,anditsroleinderivingtheproofs
canbechecked easily.
5 Proof Abstraction
We have applied our approach to proofs found by the Muscadet [9] theorem prover during the formal
certificationoftheframesafetyofacomponentofanattitudecontrolsystemasanexample. Muscadetis
basedonND,buttoimproveperformance,itimplementsavarietyofderivedrulesinadditiontothebasic
rulesofthecalculus. Thisincludesrulesfordedicatedequalityhandling, aswellasrulesthatthesystem
builds from thedefinitions and lemmas, andthat correspond theapplication ofthe givendefinitions and
lemmas. While these rules make the proofs shorter, their large number makes the proofs also in turn
more difficult to understand. This partially negates the original goal of using a ND prover. We thus
plan to optimize the resulting proofs by removing some of the book-keeping rules (e.g., return proof)
that are not central to the overall argumentation structure. Similarly, we plan to collapse sequences of
identical book-keeping rules into a single node. In general, however, we try to restructure the resulting
proof presentation to help in emphasizing the essential proof steps. In particular, we plan to group
sub-proofs that apply only axioms and lemmas from selected parts of the domain theory (e.g., ground
arithmeticorpartialorderreasoning)andrepresentthemasasinglestrategyapplication. Figure3shows
anexampleofthis. Here,thefirstabstractionstepcollapsesthesequencesrootedinG13andG14,noting
the lemmaswhich hadbeen used asstrategies as justifications, but keeping the branching that istypical
forthetransitivity. Asecondstepthenabstracts thisawayaswell.
6 Conclusions
We have described an approach whereby a safety case is used as a “structured reading” guide for the
safety proofs. Here, assurance is not implied by the trust in the ATPs but follows from the constructed
argument of the underlying proofs. However, the straightforward conversion of ND proofs into safety
cases turn out to be far from satisfactory as the proofs typically contain too many details. In practice, a
superabundance of such details isoverwhelming and unlikely tobe ofinterest anyway socareful use of
abstraction isneeded[5].
4
DerivingSafetyCasesfromMachine-Generated Proofs Basir,FischerandDenney
G12: leq (3, 5) Abstraction G12: leq (3, 5) Abstraction G12: leq (3, 5)
S 10: transitivity_leq S10r:e Paasrotniainl ogr der trJru1ale:n Bsiativsietyd_ olenq orSde1r0 r:e Paasrotinailn g tlJer1aq:n_ Bsgiatt1ivs,ie tgydt_ _ole5nq_ ,4
and gt_4_3
rules
G13: leq (4, 5) G14: leq (3, 4) G13: leq (4, 5) G14: leq (3, 4)
S1: S2:
S 11: leq_gt1 S 13: leq_gt1 Jo2n: l eBqa_sgetd1 , S11o:r dPearr tial S12o:r dPearr tial oJn3 :l eBqa_sgetd1 , (gatx_i5o_m4) (gatx_i4o_m3)
G15: gt (5, 4) G16: gt (4, 3) gt_5_4 rules reasoning reasoning gt_4_3 rules
S12: gt_5_4 S14: gt_4_3
S1: S2:
(aSxi1o:m ) (aSxi2o:m ) (gatx_i5o_m4) (gatx_i4o_m3)
gt_5_4 gt_4_3
Figure3: Abstraction ofProofSafetyC ase
The work we have described here is still in progress. So far, we have automatically derived safety
cases for the proofs found by the Muscadet prover [9]. This work complements our previous work [2]
whereweusedthehigh-levelstructureofannotationinferencetoexplicatethetop-levelstructureofsuch
softwaresafetycases. Weconsider thesafetycaseasafirststeptowardsafully-fledged softwarecertifi-
cate management system [4], which provides arange of certification services such as maintaining links
between different system artifacts and different varieties of informal and formal certificates, checking
their validity, or providing access to explicit audit trails. Wealso believe that our research willresult in
a comprehensive safety case (i.e., for the program being certified the safety logic, and the certification
system)thatwillclearlycommunicatethesafetyclaims,keysafetyrequirements, andevidence required
totrustthesoftwaresafety.
Acknowledgements. This material is based upon work supported by NASA under awards NCC2-1426 and
NNA07BB97C.ThefirstauthorisfundedbytheMalaysianGovernment,IPTAAcademicTrainingScheme.
References
[1] A.R.AndersonandN.Belnap. Entailment:thelogicofrelevanceandnecessity. PrincetonUniversityPress,
1975.
[2] N. Basir, E. Denney, and B. Fischer. Constructing a Safety Case for Automatically Generated Code from
FormalProgramVerificationInformation. InSAFECOMP’08,pages249–262,2008.
[3] E.DenneyandB.Fischer.CorrectnessofSource-LevelSafetyPolicies.InProc.FM2003:FormalMethods,
2003.
[4] E. Denneyand B. Fischer. Software Certification andSoftwareCertificate ManagementSystems(position
paper). Proceedingsofthe ASEWorkshoponSoftwareCertificateManagementSystems(SoftCeMent’05),
pages1–5,2005.
[5] E. Denney, J. Power, and K. Tourlas. Hiproofs: A Hierarchical Notion of Proof Tree. In Proceedings of
the21stAnnualConferenceonMathematicalFoundationsofProgrammingSemantics(MFPSXXI),volume
155,pages341–359,2006.
[6] M. Huth and M. Ryan. Logic in Computer Science Modelling and ReasoningaboutSystems, volume 2nd
Edition. CambridgeUniversityPress,2004.
[7] T. P. Kelly. ArguingSafety- A Systematic Approachto ManagingSafety Cases. PhD thesis, Universityof
York,1998.
[8] N.G.Leveson. Safeware:SystemSafetyandComputers. Addison-Wesley,1995.
[9] D.Pastre. MUSCADET2.3: AKnowledge-BasedTheoremProverBasedonNaturalDeduction. InIJCAR,
pages685–689,2001.
[10] J.A.Robinson. AMachine-OrientedLogicBasedontheResolutionPrinciple. ACM,1965.
5
