1 /**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
18 Revision 1.14 2007/08/06 12:25:47 acolla
19 Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
20 read from the run logbook.
21 TestShuttle setup updated.
22 TRD data point configuration updated.
24 Revision 1.13 2007/05/30 06:35:21 jgrosseo
25 Adding functionality to the Shuttle/TestShuttle:
26 o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
27 o) Function to retrieve list of IDs for a given source (GetFileIDs)
28 These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
29 Example code has been added to the TestProcessor in TestShuttle
31 Revision 1.12 2007/04/27 07:06:48 jgrosseo
32 GetFileSources returns empty list in case of no files, but successful query
33 No mails sent in testmode
35 Revision 1.11 2007/04/04 10:33:36 jgrosseo
36 1) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
37 In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
39 2) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
41 3) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
43 4) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
45 5) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
46 If you always need DCS data (like before), you do not need to implement it.
48 6) The run type has been added to the monitoring page
50 Revision 1.10 2007/02/28 10:41:01 acolla
51 Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
52 AliPreprocessor::GetRunType() function.
53 Added some ldap definition files.
55 Revision 1.8 2007/02/13 11:22:25 acolla
56 Shuttle getters and setters of main/local OCDB/Reference storages, temp and log
57 folders moved to AliShuttleInterface
59 Revision 1.6 2006/11/06 14:22:47 jgrosseo
60 major update (Alberto)
61 o) reading of run parameters from the logbook
62 o) online offline naming conversion
63 o) standalone DCSclient package
65 Revision 1.5 2006/10/02 12:58:52 jgrosseo
66 Small interface change in StoreReferenceData
68 Revision 1.4 2006/08/08 14:19:07 jgrosseo
69 Update to shuttle classes (Alberto)
71 - Possibility to set the full object's path in the Preprocessor's and
72 Shuttle's Store functions
73 - Possibility to extend the object's run validity in the same classes
74 ("startValidity" and "validityInfinite" parameters)
75 - Implementation of the StoreReferenceData function to store reference
76 data in a dedicated CDB storage.
78 Revision 1.3 2006/07/11 12:44:32 jgrosseo
79 adding parameters for extended validity range of data produced by preprocessor
81 Revision 1.2 2006/06/06 14:20:05 jgrosseo
82 o) updated test preprocessor (alberto)
83 o) added comments to example macro
84 o) test shuttle implements new interface
86 Revision 1.2 2006/03/07 07:52:34 hristov
87 New version (B.Yordanov)
89 Revision 1.3 2005/11/17 17:47:34 byordano
90 TList changed to TObjArray
92 Revision 1.2 2005/11/17 14:43:22 byordano
95 Revision 1.1.1.1 2005/10/28 07:33:58 hristov
96 Initial import as subdirectory in AliRoot
98 Revision 1.1.1.1 2005/09/12 22:11:40 byordano
101 Revision 1.2 2005/08/29 21:15:47 byordano
107 // test implementation of the AliShuttleInterface, to be used for local tests of preprocessors
109 // reads files from the local disk
110 // stores to local CDB
111 // logs to the screen
114 #include "AliTestShuttle.h"
117 #include "AliCDBManager.h"
118 #include "AliCDBStorage.h"
119 #include "AliCDBMetaData.h"
120 #include "AliCDBPath.h"
121 #include "AliCDBId.h"
122 #include "AliPreprocessor.h"
126 #include <TObjString.h>
128 #include <TTimeStamp.h>
130 ClassImp(AliTestShuttle)
132 //______________________________________________________________________________________________
133 AliTestShuttle::AliTestShuttle(Int_t run, UInt_t startTime, UInt_t endTime) :
135 fStartTime(startTime),
137 fTimeCreated(startTime),
144 fTriggerConfiguration(""),
145 fTriggerDetectorMask(""),
150 fInputFiles = new TMap;
151 fRunParameters = new TMap;
152 fPreprocessors = new TObjArray;
154 fInputFiles->SetOwner(1);
155 fRunParameters->SetOwner(1);
156 fPreprocessors->SetOwner(1);
159 //______________________________________________________________________________________________
160 AliTestShuttle::~AliTestShuttle()
167 delete fRunParameters;
170 delete fPreprocessors;
177 //______________________________________________________________________________________________
178 Bool_t AliTestShuttle::Store(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
179 Int_t validityStart, Bool_t validityInfinite)
181 // Stores the CDB object
182 // This function should be called at the end of the preprocessor cycle
184 // This implementation just stores it on the local disk, the full AliShuttle
185 // puts it to the Grid FileCatalog
187 Int_t startRun = fRun - validityStart;
189 AliError("First valid run happens to be less than 0! Setting it to 0...");
194 if(validityInfinite) {
195 endRun = AliCDBRunRange::Infinity();
200 AliCDBId id(path, startRun, endRun);
202 return AliCDBManager::Instance()->GetStorage(fgkMainCDB)->Put(object, id, metaData);
205 //______________________________________________________________________________________________
206 Bool_t AliTestShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
208 // Stores the object as reference data
209 // This function should be called at the end of the preprocessor cycle
211 // This implementation just stores it on the local disk, the full AliShuttle
212 // puts it to the Grid FileCatalog
214 AliCDBId id(path, fRun, fRun);
216 return AliCDBManager::Instance()->GetStorage(fgkMainRefStorage)->Put(object, id, metaData);
219 //______________________________________________________________________________________________
220 Bool_t AliTestShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
223 // Stores reference file directly (without opening it).
225 // This implementation just stores it on the local disk, the full AliShuttle
226 // puts it to the Grid FileCatalog
228 AliCDBManager* man = AliCDBManager::Instance();
229 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
231 TString localBaseFolder = sto->GetBaseFolder();
233 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
235 return CopyFileLocally(targetDir, localFile, gridFileName);
238 //______________________________________________________________________________________________
239 Bool_t AliTestShuttle::StoreRunMetadataFile(const char* localFile, const char* gridFileName)
242 // Stores Run metadata file to the Grid, in the run folder
244 // Only GRP can call this function.
246 AliCDBManager* man = AliCDBManager::Instance();
247 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
249 TString localBaseFolder = sto->GetBaseFolder();
251 // Build Run level folder
252 // folder = /alice/data/year/lhcPeriod/runNb/Raw
254 TTimeStamp startTime(fStartTime);
256 TString year = Form("%d",startTime.GetDate());
259 TString lhcPeriod = GetRunParameter("LHCperiod");
261 if (lhcPeriod.Length() == 0)
263 Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
267 // TODO: currently SHUTTLE cannot write in /alice/data/ !!!!!
268 //TString targetDir = Form("%s/GRP/RunMetadata/alice/data/%s/%s/%d/Raw",
269 // localBaseFolder.Data(), year.Data(),
270 // lhcPeriod.Data(), fRun);
272 TString targetDir = Form("%s/GRP/RunMetadata/alice/simulation/%s/%s/%d/Raw",
273 localBaseFolder.Data(), year.Data(),
274 lhcPeriod.Data(), fRun);
276 return CopyFileLocally(targetDir, localFile, gridFileName);
279 //______________________________________________________________________________________________
280 Bool_t AliTestShuttle::CopyFileLocally(TString& targetDir, const char* localFile, const char* gridFileName)
283 // Stores file locally. Called by StoreReferenceFile and StoreRunMetadataFile
286 //try to open folder, if it does not exist
287 void* dir = gSystem->OpenDirectory(targetDir.Data());
289 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
290 Log("SHUTTLE", Form("StoreFileLocally - Can't open directory <%s>", targetDir.Data()));
295 gSystem->FreeDirectory(dir);
298 TString target = Form("%s/%s", targetDir.Data(), gridFileName);
300 Int_t result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
303 Log("SHUTTLE", Form("StoreFileLocally - %s does not exist", localFile));
307 result = gSystem->CopyFile(localFile, target);
311 Log("SHUTTLE", Form("StoreFileLocally - File %s stored locally to %s", localFile, target.Data()));
316 Log("SHUTTLE", Form("StoreFileLocally - Could not store file %s to %s!. Error code = %d",
317 localFile, target.Data(), result));
325 //______________________________________________________________________________________________
326 const char* AliTestShuttle::GetRefFilePrefix(const char* base, const char* detector)
329 // Get folder name of reference files
332 TString offDetStr(GetOfflineDetName(detector));
334 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
336 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
338 dir.Form("%s/%s", base, offDetStr.Data());
344 //______________________________________________________________________________________________
345 const char* AliTestShuttle::GetFile(Int_t system, const char* detector, const char* id, const char* source)
347 // This function retrieves a file from the given system (kDAQ, kDCS, kHLT) with the given file id
348 // and from the given source in the system.
349 // The function returnes the path to the local file.
351 // test implementation of GetFile
352 // takes files from the local disks, files are passen in a TMap in the constructor
355 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
356 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(key.Data()));
357 TMap* sourceList = 0;
359 sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
362 AliError(Form("Could not find any file in %s with id %s (%s)", fkSystemNames[system], id, key.Data()));
366 TObjString* fileName = 0;
367 TPair* fileNamePair = dynamic_cast<TPair*> (sourceList->FindObject(source));
369 fileName = dynamic_cast<TObjString*> (fileNamePair->Value());
372 AliError(Form("Could not find files from source %s in %s with id %s",
373 source, fkSystemNames[system], id));
377 return fileName->GetString().Data();
380 //______________________________________________________________________________________________
381 TList* AliTestShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
383 // Returns a list of sources in a given system that saved a file with the given id
385 // test implementation of GetFileSources
386 // takes files from the local disks, files are passen in a TMap in the constructor
390 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
392 key.Form("%s-%s", fkSystemNames[system], detector);
394 TList* list = new TList;
396 TIterator* iter = fInputFiles->MakeIterator();
398 while ((obj = iter->Next()))
400 TObjString* objStr = dynamic_cast<TObjString*> (obj);
403 Bool_t found = kFALSE;
406 found = (objStr->String().CompareTo(key) == 0);
409 found = objStr->String().BeginsWith(key);
413 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(objStr->String().Data()));
414 TMap* sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
416 TIterator* iter2 = sourceList->GetTable()->MakeIterator();
418 while ((obj2 = iter2->Next()))
420 TPair* pair = dynamic_cast<TPair*> (obj2);
423 if (!list->FindObject(pair->Key()))
424 list->Add(new TObjString(pair->Key()->GetName()));
433 if (list->GetEntries() == 0)
434 AliInfo(Form("Could not find any file in %s with id %s (%s)", fkSystemNames[system], id, key.Data()));
439 //______________________________________________________________________________________________
440 TList* AliTestShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
442 // Returns a list of ids in a given system that saved a file with the given source
444 // test implementation of GetFileSources
445 // takes files from the local disks, files are passen in a TMap in the constructor
449 key.Form("%s-%s", fkSystemNames[system], detector);
451 TList* list = new TList;
453 TIterator* iter = fInputFiles->MakeIterator();
455 while ((obj = iter->Next()))
457 TObjString* objStr = dynamic_cast<TObjString*> (obj);
460 if (objStr->String().BeginsWith(key))
462 Bool_t found = kFALSE;
464 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(objStr->String().Data()));
465 TMap* sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
467 TIterator* iter2 = sourceList->GetTable()->MakeIterator();
469 while ((obj2 = iter2->Next()))
471 TPair* pair = dynamic_cast<TPair*> (obj2);
474 if (strcmp(pair->Key()->GetName(), source) == 0)
483 TObjArray* tokens = objStr->String().Tokenize("-");
484 if (tokens->GetEntries() == 3)
486 TObjString* id = dynamic_cast<TObjString*> (tokens->At(2));
487 if (id && !list->FindObject(id->String()))
488 list->Add(new TObjString(id->String()));
498 if (list->GetEntries() == 0)
499 AliInfo(Form("Could not find any file in %s with source %s (%s)", fkSystemNames[system], source, key.Data()));
504 //______________________________________________________________________________________________
505 void AliTestShuttle::Log(const char* detector, const char* message, UInt_t level)
507 // test implementation of Log
508 // just prints to the screen
510 TString fullMessage = detector;
511 fullMessage.Append(Form(": %s",message));
512 AliLog::Message(level, fullMessage, MODULENAME(), ClassName(), FUNCTIONNAME(), __FILE__, __LINE__);
515 //______________________________________________________________________________________________
516 void AliTestShuttle::AddInputFile(Int_t system, const char* detector, const char* id, const char* source, const char* fileName)
519 // This function adds a file to the list of input files
520 // the list is stored in fInputFiles
521 // fInputFiles: TMap (key -> value)
522 // <system>-<detector>-<id> -> TMap (key -> value)
523 // <source> -> <filename>
527 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
528 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(key.Data()));
529 TMap* sourceList = 0;
531 sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
534 sourceList = new TMap;
535 fInputFiles->Add(new TObjString(key), sourceList);
538 sourceList->Add(new TObjString(source), new TObjString(fileName));
541 //______________________________________________________________________________________________
542 Bool_t AliTestShuttle::AddInputCDBEntry(AliCDBEntry* entry)
544 // This function adds an object in the OCDB to be later retrieved with GetFromOCDB
546 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
549 Log("SHUTTLE", "GetFromOCDB - Cannot activate main OCDB for query!");
553 return sto->Put(entry);
556 //______________________________________________________________________________________________
557 AliCDBEntry* AliTestShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
559 // returns obiect from OCDB valid for current run
561 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
564 Log("SHUTTLE", "GetFromOCDB - Cannot activate main OCDB for query!");
568 return (AliCDBEntry*) sto->Get(path, fRun);
571 //______________________________________________________________________________________________
572 void AliTestShuttle::Process()
574 // This function tests all preprocessors that are registered to it
575 // All preprocessors get the same dcs alias map and have access to the same list of files.
577 for (Int_t i=0; i<fPreprocessors->GetEntries(); ++i)
579 AliPreprocessor* preprocessor = dynamic_cast<AliPreprocessor*> (fPreprocessors->At(i));
582 if (preprocessor->ProcessRunType())
584 preprocessor->Initialize(fRun, fStartTime, fEndTime);
585 preprocessor->Process(fDcsAliasMap);
592 //______________________________________________________________________________________________
593 void AliTestShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
595 // registers a preprocessor
597 const char* detName = preprocessor->GetName();
598 if(strcmp("DET", detName) != 0) {
599 if(GetDetPos(detName) < 0)
600 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
603 fPreprocessors->Add(preprocessor);
606 //______________________________________________________________________________________________
607 void AliTestShuttle::AddInputRunParameter(const char* key, const char* value){
608 // set a run parameter (in reality it will be read from the DAQ logbook)
610 TObjString* keyObj = new TObjString(key);
611 if (fRunParameters->Contains(key)) {
612 AliWarning(Form("Parameter %s already existing and it will be replaced.", key));
613 delete fRunParameters->Remove(keyObj);
616 fRunParameters->Add(keyObj, new TObjString(value));
617 AliDebug(2, Form("Number of parameters: %d", fRunParameters->
621 //______________________________________________________________________________________________
622 const char* AliTestShuttle::GetRunType()
625 // get a run parameter
628 return fRunType.Data();
631 //______________________________________________________________________________________________
632 const char* AliTestShuttle::GetRunParameter(const char* key){
633 // get a run parameter
635 TObjString* value = dynamic_cast<TObjString*> (fRunParameters->GetValue(key));
637 AliError(Form("No such parameter: %s", key));
640 return value->GetName();
643 //______________________________________________________________________________________________
644 void AliTestShuttle::SetShuttleTempDir(const char* tmpDir)
646 // sets Shuttle temp directory
648 fgkShuttleTempDir = gSystem->ExpandPathName(tmpDir);
651 //______________________________________________________________________________________________
652 void AliTestShuttle::SetShuttleLogDir(const char* logDir)
654 // sets Shuttle log directory
656 fgkShuttleLogDir = gSystem->ExpandPathName(logDir);
659 //______________________________________________________________________________________________
660 const char* AliTestShuttle::GetTriggerConfiguration()
662 //returns trigger configuration
663 if (fTriggerConfiguration.Length()>0){
664 return fTriggerConfiguration;
668 //______________________________________________________________________________________________
669 const char* AliTestShuttle::GetCTPTimeParams()
671 //returns trigger configuration
672 if (fCTPtiming.Length()>0){
677 //______________________________________________________________________________________________
678 const char* AliTestShuttle::GetTriggerDetectorMask()
680 //returns trigger detector mask
681 if (fTriggerDetectorMask.Length()>0){
682 return fTriggerDetectorMask;
686 //______________________________________________________________________________________________
687 UInt_t AliTestShuttle::GetStartTimeDCSQuery()
689 // Return Start Time for the DCS query
691 // The call is delegated to AliShuttleInterface
693 return fTimeCreated-fDCSQueryOffset;
695 //______________________________________________________________________________________________
696 UInt_t AliTestShuttle::GetEndTimeDCSQuery()
698 // Return End Time for the DCS query
700 // The call is delegated to AliShuttleInterface
702 return fEndTime+fDCSQueryOffset;