SDAMS: SPOrt Data Archiving and Management System

4 Pages·0.14 MB·English

by  L. Nicastro

SDAMS: SPOrt Data Archiving and Management System
∗ ∗
Luciano Nicastro and Giorgio Calderone
∗IFCAI-CNR,ViaU.LaMalfa153,90146Palermo,Italy
2
0
Abstract. SDAMSistheensembleofdatabase+softwarepackagesaimedtothearchiving,quick-lookanalysis,
0
2 off-line analysis, network accessibility and plotting of the SPOrt produced data. Many of the aspects related
to data archiving, analysis and distribution are common to almost all the astronomical experiments. SDAMS
n
ambition is to face and solve problems like accessibility and portability of the data on any hardware/software
a
platformin a way as simpler as possible, thougheffective.The system is conceivedin a way to be used either
J
by the scientific community interested in background radiation studies or by a wider public with low or null
4
knowledgeofthesubject.Theuserauthenticationsystemallowsustoapplydifferentlevelsofaccess,analysis
1
anddataretrieving.SDAMSwillbeaccessiblethroughanyWebbrowserthoughthemostefficientwaytouseit

1 isbywritingsimpleprograms.Graphicsandimagesusefulforoutreachpurposeswillbeproducedandputonthe
v Webonaregularbasis.
5
0
2
INTRODUCTION
1
0
2 SPOrt ([1]) on ISS is an experiment which will produce a limited amount of data with respect to other space
0 experiments. The foreseen total (i.e. housekeepings and scientific data) bit rate is ∼2.5 kbit s−1. This means that
h/ in 3 years of operationtime it will collect ∼30 Gbytesof data. In addition SPOrt has notpointingcapabilitiesand
p it is expectedto work in a stable and semi-automatic way. As a consequencethe outputdata stream will be simply
- identifiedbytimeandorbitalparametersoftheSpaceStation.Thedatamanagementsystem(i.e.quicklook,archive
o
andscientificanalysis)canbebuildtobebothsimpleandhighlyautomatic.
r
t TheSDAMSsystemhastwofundamentalparts:databaseserverandapplicationserver.Theformerperformsthe
s
a datastoringandallowsasimple(lowlevel)fastaccesstothem.TothisaimwechoseMySQLTM,averyeasytouse
: datamanagementsystemcapabletodealwitharelativelylargeamountofdata.OntheWebthereareseveralexamples
v
of intensive usage of MySQL with application tools and there are extensions to use it within several programming
i
X
languages(e.g.PHP).IthasalsotheadvantagetobeanOpenSourcepackage,thatmeansonecanmodifythesource
r codetomatchanyparticularneedandnolicenseisrequiredtouseit.Thisiscrucialtoallowastandard(system)tobe
a
developed.
The SDAMS applicationserveris a multithreadedserverwhich uses a simple C-written communicationprotocol
to interactwith itsclients. A task-orientedmodulelibraryallowsusto easily add furthercapabilitiesto thoseso far
foreseen. The fundamental task of this component is to accept client connections (through socket) and invoke the
appropriatemodulestoperformtherequiredtasks.
NewsaboutSDAMSwillbepostedonhttp://sport.tesre.bo.cnr.it.
HOWTO USE IT
ThereareseveralwaystouseSDAMS,thesimplestbeingthroughaWebinterface.Suchkindofinterfaceallowsusers
tosetupasimplequeryinordertorequestdataretrievalandvisualization.Thisisdone,forexample,selectingatime
orskycoordinatesrange.ItwillalsobepossibletoviewtheskyportionseenbySPOrt(pointingatthezenithofthe
ISS)inanygiventimeinterval.InadditiontoSun,MoonandsomePlanets,also"interesting"strongradiosourceswill
behighlightedintheskymaps.Access tothedata willbegrantedonlytoregisteredusers. However,thisistheless
flexiblewaytouseSDAMSasitonlyoffersalimitedsetofchoices;itwillbelikelyusedtoperformdata"quick-look"
or by users not directly involved in the SPOrt project or by the general public. The communication protocol is so
easy that it is possible to connectto SDAMS througha telnetsession and typingin the commands.The outputwill
be ASCII formatted. The most flexible way to access SDAMS is instead through a dedicated, user written (in any
language)program.Theonlyrequirementsare
• capabilitytoopenTCP/IPsockets,
• capabilitytoreadASCIIandbinaryfiles.
InthiscasetheoutputcanbeformattedasaFITSbinary(orASCII)table.Forthese"high-level"users,areference
manual will be available reporting characteristics of the protocol and a full description of commands and message
codes.OnceagainwenotehowSDAMSdoesnotputanyrestrictiontothehardware/softwareplatformoftheclient.
HOWITWORKS
Figures1and2showhowSDAMSworks:foreachclientsuccessfullyconnectedtothesocketanewthreadiscreated;
itisasortofsub-processrunningontheserver.Itwaitsforqueriesfromtheclientandexecutesthemindependently
of what the other clients are doing. To each query, written by using high level commands, corresponds a spawned
process which passes a reformatted low-levelquery to the database server. The output data are processed either by
C or IDLTM routines(dependingon the request)and then sent to the client in the requested format.This processis
repeated until a "disconnect" request is received. The "language" used by the client to put its query is very simple,
beinganASCIIstringcontainingthenameoftheproceduretouseanditsparameters.Forexample,toselectthedata
fortheStokesparameterQinthetimeinterval(MJD)52810–52813,inthe32GHzfrequencychannel,averagedover
30s,andperformapolynomialfit(thirdorder)withautomaticgraphicaloutput,thecommandis:
sel, par=’q’, dt=[52810.0,52813.0], fr=32, av=30, fit=’poly 3’
BydefaultthegraphicsgoesintoaPostScriptfile,butotherformatscanbechosen.
Itisworthtonotehow,inspitetheapplicationservertalkstothedatabaseserverusingtheSQL(StructuredQuery
Language) syntax, it is not necessary to write the queries using this low-level format (but can be done starting the
querywithSQL).Adedicatedmodule,theSQLwrapper,takescareoftranslatingtheinputcommandstring(aftera
validitycheck).Whentheclientqueriesrequirecomplexoperations(e.g.correlationanalysisorsky-mapproduction),
theapplicationserverfirstextractsthedataintoalocaldirectorythentheappropriate(IDL)routine(oneormore)is
invokedfortheanalysisandfinallytheoutputissenttotheclient(e.g.inFITSformat).TheclientsdonotneedanIDL
licenseastheapplicationsoftwareonlyrunsontheserver.Figure3showsanexampleofskycoveragemapextracted
usingthecommand"map, /coverage".Itassumesonlyabout20daysareelapsedsincethestartofthemission.
Addinganewfeaturetothesystemisextremelyeasy:itisenoughtoaddtheIDLroutine(accessingthedatausingthe
SDAMSstandards)tothelibraryandaddingitsnametothelistofallowedcommands.
ACKNOWLEDGMENTS
SDAMSispartoftheSPOrtprojectanditissupportedbyAgenziaSpazialeItaliana(ASI).
REFERENCES
1. Carretti,E., “The SPOrtproject”, in Astrophysical Polarized Backgrounds, edited by e. a. S. Cecchini, AIP Conference
Proceedings0,AmericanInstituteofPhysics,NewYork,2001,pp.xx–yy.
FIGURE 1. This diagram shows the main components of SDAMS (dashed inset) and the way it can interact with external
systems(clients).TheASDC(ASIScienceDataCenter)networkgivesaccesstotherawdatadownloadedfromtheInternational
SpaceStation.HereweassumethedatabaseresidesatIFCAIInstitute(Palermo)
NO New YES
Connection ?
Wait for a new Wait for a new
command command
SQL request SQL request
Run IDL Run IDL
analysis routine analysis routine
Output filter Output filter
Send data Send data
to client to client
Client 1 Client n
FIGURE2. Theschemeaboveshowstheparallelrunning ofthethreads(dashedinsets).Eachthreadisasub-process(onthe
server)performingtheclientqueriesanddataoutput
FIGURE3. Inthisimage(Galacticcoordinates)theSPOrtskycoverageafterabout20daysofoperativelifeisreported.Grey
scaleisproportionaltotheintegrationtime