1 /**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
18 Revision 1.14 2007/08/06 12:25:47 acolla
19 Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
20 read from the run logbook.
21 TestShuttle setup updated.
22 TRD data point configuration updated.
24 Revision 1.13 2007/05/30 06:35:21 jgrosseo
25 Adding functionality to the Shuttle/TestShuttle:
26 o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
27 o) Function to retrieve list of IDs for a given source (GetFileIDs)
28 These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
29 Example code has been added to the TestProcessor in TestShuttle
31 Revision 1.12 2007/04/27 07:06:48 jgrosseo
32 GetFileSources returns empty list in case of no files, but successful query
33 No mails sent in testmode
35 Revision 1.11 2007/04/04 10:33:36 jgrosseo
36 1) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
37 In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
39 2) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
41 3) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
43 4) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
45 5) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
46 If you always need DCS data (like before), you do not need to implement it.
48 6) The run type has been added to the monitoring page
50 Revision 1.10 2007/02/28 10:41:01 acolla
51 Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
52 AliPreprocessor::GetRunType() function.
53 Added some ldap definition files.
55 Revision 1.8 2007/02/13 11:22:25 acolla
56 Shuttle getters and setters of main/local OCDB/Reference storages, temp and log
57 folders moved to AliShuttleInterface
59 Revision 1.6 2006/11/06 14:22:47 jgrosseo
60 major update (Alberto)
61 o) reading of run parameters from the logbook
62 o) online offline naming conversion
63 o) standalone DCSclient package
65 Revision 1.5 2006/10/02 12:58:52 jgrosseo
66 Small interface change in StoreReferenceData
68 Revision 1.4 2006/08/08 14:19:07 jgrosseo
69 Update to shuttle classes (Alberto)
71 - Possibility to set the full object's path in the Preprocessor's and
72 Shuttle's Store functions
73 - Possibility to extend the object's run validity in the same classes
74 ("startValidity" and "validityInfinite" parameters)
75 - Implementation of the StoreReferenceData function to store reference
76 data in a dedicated CDB storage.
78 Revision 1.3 2006/07/11 12:44:32 jgrosseo
79 adding parameters for extended validity range of data produced by preprocessor
81 Revision 1.2 2006/06/06 14:20:05 jgrosseo
82 o) updated test preprocessor (alberto)
83 o) added comments to example macro
84 o) test shuttle implements new interface
86 Revision 1.2 2006/03/07 07:52:34 hristov
87 New version (B.Yordanov)
89 Revision 1.3 2005/11/17 17:47:34 byordano
90 TList changed to TObjArray
92 Revision 1.2 2005/11/17 14:43:22 byordano
95 Revision 1.1.1.1 2005/10/28 07:33:58 hristov
96 Initial import as subdirectory in AliRoot
98 Revision 1.1.1.1 2005/09/12 22:11:40 byordano
101 Revision 1.2 2005/08/29 21:15:47 byordano
107 // test implementation of the AliShuttleInterface, to be used for local tests of preprocessors
109 // reads files from the local disk
110 // stores to local CDB
111 // logs to the screen
114 #include "AliTestShuttle.h"
117 #include "AliCDBManager.h"
118 #include "AliCDBStorage.h"
119 #include "AliCDBMetaData.h"
120 #include "AliCDBPath.h"
121 #include "AliCDBId.h"
122 #include "AliPreprocessor.h"
126 #include <TObjString.h>
128 #include <TTimeStamp.h>
130 ClassImp(AliTestShuttle)
132 //______________________________________________________________________________________________
133 AliTestShuttle::AliTestShuttle(Int_t run, UInt_t startTime, UInt_t endTime) :
135 fStartTime(startTime),
137 fTimeCreated(startTime),
147 fInputFiles = new TMap;
148 fRunParameters = new TMap;
149 fPreprocessors = new TObjArray;
151 fInputFiles->SetOwner(1);
152 fRunParameters->SetOwner(1);
153 fPreprocessors->SetOwner(1);
156 //______________________________________________________________________________________________
157 AliTestShuttle::~AliTestShuttle()
164 delete fRunParameters;
167 delete fPreprocessors;
174 //______________________________________________________________________________________________
175 Bool_t AliTestShuttle::Store(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
176 Int_t validityStart, Bool_t validityInfinite)
178 // Stores the CDB object
179 // This function should be called at the end of the preprocessor cycle
181 // This implementation just stores it on the local disk, the full AliShuttle
182 // puts it to the Grid FileCatalog
184 Int_t startRun = fRun - validityStart;
186 AliError("First valid run happens to be less than 0! Setting it to 0...");
191 if(validityInfinite) {
192 endRun = AliCDBRunRange::Infinity();
197 AliCDBId id(path, startRun, endRun);
199 return AliCDBManager::Instance()->GetStorage(fgkMainCDB)->Put(object, id, metaData);
202 //______________________________________________________________________________________________
203 Bool_t AliTestShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
205 // Stores the object as reference data
206 // This function should be called at the end of the preprocessor cycle
208 // This implementation just stores it on the local disk, the full AliShuttle
209 // puts it to the Grid FileCatalog
211 AliCDBId id(path, fRun, fRun);
213 return AliCDBManager::Instance()->GetStorage(fgkMainRefStorage)->Put(object, id, metaData);
216 //______________________________________________________________________________________________
217 Bool_t AliTestShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
220 // Stores reference file directly (without opening it).
222 // This implementation just stores it on the local disk, the full AliShuttle
223 // puts it to the Grid FileCatalog
225 AliCDBManager* man = AliCDBManager::Instance();
226 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
228 TString localBaseFolder = sto->GetBaseFolder();
230 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
232 return CopyFileLocally(targetDir, localFile, gridFileName);
235 //______________________________________________________________________________________________
236 Bool_t AliTestShuttle::StoreRunMetadataFile(const char* localFile, const char* gridFileName)
239 // Stores Run metadata file to the Grid, in the run folder
241 // Only GRP can call this function.
243 AliCDBManager* man = AliCDBManager::Instance();
244 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
246 TString localBaseFolder = sto->GetBaseFolder();
248 // Build Run level folder
249 // folder = /alice/data/year/lhcPeriod/runNb/Raw
251 TTimeStamp startTime(fStartTime);
253 TString year = Form("%d",startTime.GetDate());
256 TString lhcPeriod = GetRunParameter("LHCperiod");
258 if (lhcPeriod.Length() == 0)
260 Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
264 // TODO: currently SHUTTLE cannot write in /alice/data/ !!!!!
265 //TString targetDir = Form("%s/GRP/RunMetadata/alice/data/%s/%s/%d/Raw",
266 // localBaseFolder.Data(), year.Data(),
267 // lhcPeriod.Data(), fRun);
269 TString targetDir = Form("%s/GRP/RunMetadata/alice/simulation/%s/%s/%d/Raw",
270 localBaseFolder.Data(), year.Data(),
271 lhcPeriod.Data(), fRun);
273 return CopyFileLocally(targetDir, localFile, gridFileName);
276 //______________________________________________________________________________________________
277 Bool_t AliTestShuttle::CopyFileLocally(TString& targetDir, const char* localFile, const char* gridFileName)
280 // Stores file locally. Called by StoreReferenceFile and StoreRunMetadataFile
283 //try to open folder, if it does not exist
284 void* dir = gSystem->OpenDirectory(targetDir.Data());
286 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
287 Log("SHUTTLE", Form("StoreFileLocally - Can't open directory <%s>", targetDir.Data()));
292 gSystem->FreeDirectory(dir);
295 TString target = Form("%s/%s", targetDir.Data(), gridFileName);
297 Int_t result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
300 Log("SHUTTLE", Form("StoreFileLocally - %s does not exist", localFile));
304 result = gSystem->CopyFile(localFile, target);
308 Log("SHUTTLE", Form("StoreFileLocally - File %s stored locally to %s", localFile, target.Data()));
313 Log("SHUTTLE", Form("StoreFileLocally - Could not store file %s to %s!. Error code = %d",
314 localFile, target.Data(), result));
322 //______________________________________________________________________________________________
323 const char* AliTestShuttle::GetRefFilePrefix(const char* base, const char* detector)
326 // Get folder name of reference files
329 TString offDetStr(GetOfflineDetName(detector));
331 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
333 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
335 dir.Form("%s/%s", base, offDetStr.Data());
341 //______________________________________________________________________________________________
342 const char* AliTestShuttle::GetFile(Int_t system, const char* detector, const char* id, const char* source)
344 // This function retrieves a file from the given system (kDAQ, kDCS, kHLT) with the given file id
345 // and from the given source in the system.
346 // The function returnes the path to the local file.
348 // test implementation of GetFile
349 // takes files from the local disks, files are passen in a TMap in the constructor
352 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
353 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(key.Data()));
354 TMap* sourceList = 0;
356 sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
359 AliError(Form("Could not find any file in %s with id %s (%s)", fkSystemNames[system], id, key.Data()));
363 TObjString* fileName = 0;
364 TPair* fileNamePair = dynamic_cast<TPair*> (sourceList->FindObject(source));
366 fileName = dynamic_cast<TObjString*> (fileNamePair->Value());
369 AliError(Form("Could not find files from source %s in %s with id %s",
370 source, fkSystemNames[system], id));
374 return fileName->GetString().Data();
377 //______________________________________________________________________________________________
378 TList* AliTestShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
380 // Returns a list of sources in a given system that saved a file with the given id
382 // test implementation of GetFileSources
383 // takes files from the local disks, files are passen in a TMap in the constructor
387 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
389 key.Form("%s-%s", fkSystemNames[system], detector);
391 TList* list = new TList;
393 TIterator* iter = fInputFiles->MakeIterator();
395 while ((obj = iter->Next()))
397 TObjString* objStr = dynamic_cast<TObjString*> (obj);
400 Bool_t found = kFALSE;
403 found = (objStr->String().CompareTo(key) == 0);
406 found = objStr->String().BeginsWith(key);
410 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(objStr->String().Data()));
411 TMap* sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
413 TIterator* iter2 = sourceList->GetTable()->MakeIterator();
415 while ((obj2 = iter2->Next()))
417 TPair* pair = dynamic_cast<TPair*> (obj2);
420 if (!list->FindObject(pair->Key()))
421 list->Add(new TObjString(pair->Key()->GetName()));
430 if (list->GetEntries() == 0)
431 AliInfo(Form("Could not find any file in %s with id %s (%s)", fkSystemNames[system], id, key.Data()));
436 //______________________________________________________________________________________________
437 TList* AliTestShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
439 // Returns a list of ids in a given system that saved a file with the given source
441 // test implementation of GetFileSources
442 // takes files from the local disks, files are passen in a TMap in the constructor
446 key.Form("%s-%s", fkSystemNames[system], detector);
448 TList* list = new TList;
450 TIterator* iter = fInputFiles->MakeIterator();
452 while ((obj = iter->Next()))
454 TObjString* objStr = dynamic_cast<TObjString*> (obj);
457 if (objStr->String().BeginsWith(key))
459 Bool_t found = kFALSE;
461 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(objStr->String().Data()));
462 TMap* sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
464 TIterator* iter2 = sourceList->GetTable()->MakeIterator();
466 while ((obj2 = iter2->Next()))
468 TPair* pair = dynamic_cast<TPair*> (obj2);
471 if (strcmp(pair->Key()->GetName(), source) == 0)
480 TObjArray* tokens = objStr->String().Tokenize("-");
481 if (tokens->GetEntries() == 3)
483 TObjString* id = dynamic_cast<TObjString*> (tokens->At(2));
484 if (id && !list->FindObject(id->String()))
485 list->Add(new TObjString(id->String()));
495 if (list->GetEntries() == 0)
496 AliInfo(Form("Could not find any file in %s with source %s (%s)", fkSystemNames[system], source, key.Data()));
501 //______________________________________________________________________________________________
502 void AliTestShuttle::Log(const char* detector, const char* message)
504 // test implementation of Log
505 // just prints to the screen
507 AliInfo(Form("%s: %s", detector, message));
510 //______________________________________________________________________________________________
511 void AliTestShuttle::AddInputFile(Int_t system, const char* detector, const char* id, const char* source, const char* fileName)
514 // This function adds a file to the list of input files
515 // the list is stored in fInputFiles
516 // fInputFiles: TMap (key -> value)
517 // <system>-<detector>-<id> -> TMap (key -> value)
518 // <source> -> <filename>
522 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
523 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(key.Data()));
524 TMap* sourceList = 0;
526 sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
529 sourceList = new TMap;
530 fInputFiles->Add(new TObjString(key), sourceList);
533 sourceList->Add(new TObjString(source), new TObjString(fileName));
536 //______________________________________________________________________________________________
537 Bool_t AliTestShuttle::AddInputCDBEntry(AliCDBEntry* entry)
539 // This function adds an object in the OCDB to be later retrieved with GetFromOCDB
541 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
544 Log("SHUTTLE", "GetFromOCDB - Cannot activate main OCDB for query!");
548 return sto->Put(entry);
551 //______________________________________________________________________________________________
552 AliCDBEntry* AliTestShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
554 // returns obiect from OCDB valid for current run
556 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
559 Log("SHUTTLE", "GetFromOCDB - Cannot activate main OCDB for query!");
563 return (AliCDBEntry*) sto->Get(path, fRun);
566 //______________________________________________________________________________________________
567 void AliTestShuttle::Process()
569 // This function tests all preprocessors that are registered to it
570 // All preprocessors get the same dcs alias map and have access to the same list of files.
572 for (Int_t i=0; i<fPreprocessors->GetEntries(); ++i)
574 AliPreprocessor* preprocessor = dynamic_cast<AliPreprocessor*> (fPreprocessors->At(i));
577 if (preprocessor->ProcessRunType())
579 preprocessor->Initialize(fRun, fStartTime, fEndTime);
580 preprocessor->Process(fDcsAliasMap);
586 //______________________________________________________________________________________________
587 void AliTestShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
589 // registers a preprocessor
591 const char* detName = preprocessor->GetName();
592 if(strcmp("DET", detName) != 0) {
593 if(GetDetPos(detName) < 0)
594 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
597 fPreprocessors->Add(preprocessor);
600 //______________________________________________________________________________________________
601 void AliTestShuttle::AddInputRunParameter(const char* key, const char* value){
602 // set a run parameter (in reality it will be read from the DAQ logbook)
604 TObjString* keyObj = new TObjString(key);
605 if (fRunParameters->Contains(key)) {
606 AliWarning(Form("Parameter %s already existing and it will be replaced.", key));
607 delete fRunParameters->Remove(keyObj);
610 fRunParameters->Add(keyObj, new TObjString(value));
611 AliDebug(2, Form("Number of parameters: %d", fRunParameters->
615 //______________________________________________________________________________________________
616 const char* AliTestShuttle::GetRunType()
619 // get a run parameter
622 return fRunType.Data();
625 //______________________________________________________________________________________________
626 const char* AliTestShuttle::GetRunParameter(const char* key){
627 // get a run parameter
629 TObjString* value = dynamic_cast<TObjString*> (fRunParameters->GetValue(key));
631 AliError(Form("No such parameter: %s", key));
634 return value->GetName();
637 //______________________________________________________________________________________________
638 void AliTestShuttle::SetShuttleTempDir(const char* tmpDir)
640 // sets Shuttle temp directory
642 fgkShuttleTempDir = gSystem->ExpandPathName(tmpDir);
645 //______________________________________________________________________________________________
646 void AliTestShuttle::SetShuttleLogDir(const char* logDir)
648 // sets Shuttle log directory
650 fgkShuttleLogDir = gSystem->ExpandPathName(logDir);
653 //______________________________________________________________________________________________
654 const char* AliTestShuttle::GetTriggerConfiguration()
656 //returns trigger configuration
657 if (fTriggerConfiguration.Length()>0){
658 return fTriggerConfiguration;
662 //______________________________________________________________________________________________
663 const UInt_t AliTestShuttle::GetStartTimeDCSQuery()
665 // Return Start Time for the DCS query
667 // The call is delegated to AliShuttleInterface
669 return fTimeCreated-fDCSQueryOffset;
671 //______________________________________________________________________________________________
672 const UInt_t AliTestShuttle::GetEndTimeDCSQuery()
674 // Return End Time for the DCS query
676 // The call is delegated to AliShuttleInterface
678 return fEndTime+fDCSQueryOffset;