several bugfixes
[u/mrichter/AliRoot.git] / SHUTTLE / TestShuttle / AliTestShuttle.cxx
CommitLineData
5c6b40ae 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
77dc0223 18Revision 1.14 2007/08/06 12:25:47 acolla
19Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
20read from the run logbook.
21TestShuttle setup updated.
22TRD data point configuration updated.
23
87968cd5 24Revision 1.13 2007/05/30 06:35:21 jgrosseo
25Adding functionality to the Shuttle/TestShuttle:
26o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
27o) Function to retrieve list of IDs for a given source (GetFileIDs)
28These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
29Example code has been added to the TestProcessor in TestShuttle
30
4a33bdd9 31Revision 1.12 2007/04/27 07:06:48 jgrosseo
32GetFileSources returns empty list in case of no files, but successful query
33No mails sent in testmode
34
86aa42c3 35Revision 1.11 2007/04/04 10:33:36 jgrosseo
361) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
37In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
38
392) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
40
413) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
42
434) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
44
455) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
46If you always need DCS data (like before), you do not need to implement it.
47
486) The run type has been added to the monitoring page
49
9827400b 50Revision 1.10 2007/02/28 10:41:01 acolla
51Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
52AliPreprocessor::GetRunType() function.
53Added some ldap definition files.
54
d461a8a7 55Revision 1.8 2007/02/13 11:22:25 acolla
56Shuttle getters and setters of main/local OCDB/Reference storages, temp and log
57folders moved to AliShuttleInterface
58
e19c88ea 59Revision 1.6 2006/11/06 14:22:47 jgrosseo
60major update (Alberto)
61o) reading of run parameters from the logbook
62o) online offline naming conversion
63o) standalone DCSclient package
64
eba76848 65Revision 1.5 2006/10/02 12:58:52 jgrosseo
66Small interface change in StoreReferenceData
67
8c77aee7 68Revision 1.4 2006/08/08 14:19:07 jgrosseo
69Update to shuttle classes (Alberto)
70
71- Possibility to set the full object's path in the Preprocessor's and
72Shuttle's Store functions
73- Possibility to extend the object's run validity in the same classes
74("startValidity" and "validityInfinite" parameters)
75- Implementation of the StoreReferenceData function to store reference
76data in a dedicated CDB storage.
77
84090f85 78Revision 1.3 2006/07/11 12:44:32 jgrosseo
79adding parameters for extended validity range of data produced by preprocessor
80
17111222 81Revision 1.2 2006/06/06 14:20:05 jgrosseo
82o) updated test preprocessor (alberto)
83o) added comments to example macro
84o) test shuttle implements new interface
85
5c6b40ae 86Revision 1.2 2006/03/07 07:52:34 hristov
87New version (B.Yordanov)
88
89Revision 1.3 2005/11/17 17:47:34 byordano
90TList changed to TObjArray
91
92Revision 1.2 2005/11/17 14:43:22 byordano
93import to local CVS
94
95Revision 1.1.1.1 2005/10/28 07:33:58 hristov
96Initial import as subdirectory in AliRoot
97
98Revision 1.1.1.1 2005/09/12 22:11:40 byordano
99SHUTTLE package
100
101Revision 1.2 2005/08/29 21:15:47 byordano
102some docs added
103
104*/
105
106//
107// test implementation of the AliShuttleInterface, to be used for local tests of preprocessors
108//
109// reads files from the local disk
110// stores to local CDB
111// logs to the screen
112//
113
114#include "AliTestShuttle.h"
115#include "AliLog.h"
116
117#include "AliCDBManager.h"
84090f85 118#include "AliCDBStorage.h"
5c6b40ae 119#include "AliCDBMetaData.h"
84090f85 120#include "AliCDBPath.h"
5c6b40ae 121#include "AliCDBId.h"
36137ac1 122#include "AliPreprocessor.h"
5c6b40ae 123
124#include <TMap.h>
125#include <TList.h>
5c6b40ae 126#include <TObjString.h>
e19c88ea 127#include <TSystem.h>
77dc0223 128#include <TTimeStamp.h>
5c6b40ae 129
130ClassImp(AliTestShuttle)
131
36137ac1 132//______________________________________________________________________________________________
17111222 133AliTestShuttle::AliTestShuttle(Int_t run, UInt_t startTime, UInt_t endTime) :
134 fRun(run),
135 fStartTime(startTime),
136 fEndTime(endTime),
36137ac1 137 fInputFiles(0),
eba76848 138 fRunParameters(0),
9827400b 139 fRunType(),
36137ac1 140 fPreprocessors(0),
141 fDcsAliasMap(0)
5c6b40ae 142{
143 // constructor
36137ac1 144
145 fInputFiles = new TMap;
eba76848 146 fRunParameters = new TMap;
36137ac1 147 fPreprocessors = new TObjArray;
441b0e9c 148
149 fInputFiles->SetOwner(1);
150 fRunParameters->SetOwner(1);
441b0e9c 151 fPreprocessors->SetOwner(1);
5c6b40ae 152}
153
36137ac1 154//______________________________________________________________________________________________
5c6b40ae 155AliTestShuttle::~AliTestShuttle()
156{
157 // destructor
36137ac1 158
159 delete fInputFiles;
160 fInputFiles = 0;
161
eba76848 162 delete fRunParameters;
163 fRunParameters = 0;
164
36137ac1 165 delete fPreprocessors;
166 fPreprocessors = 0;
167
168 delete fDcsAliasMap;
169 fDcsAliasMap = 0;
5c6b40ae 170}
171
36137ac1 172//______________________________________________________________________________________________
9827400b 173Bool_t AliTestShuttle::Store(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
84090f85 174 Int_t validityStart, Bool_t validityInfinite)
5c6b40ae 175{
176 // Stores the CDB object
177 // This function should be called at the end of the preprocessor cycle
178 //
179 // This implementation just stores it on the local disk, the full AliShuttle
180 // puts it to the Grid FileCatalog
181
84090f85 182 Int_t startRun = fRun - validityStart;
183 if(startRun < 0) {
184 AliError("First valid run happens to be less than 0! Setting it to 0...");
185 startRun=0;
186 }
17111222 187
84090f85 188 Int_t endRun = -1;
189 if(validityInfinite) {
190 endRun = AliCDBRunRange::Infinity();
191 } else {
192 endRun = fRun;
193 }
17111222 194
84090f85 195 AliCDBId id(path, startRun, endRun);
17111222 196
e19c88ea 197 return AliCDBManager::Instance()->GetStorage(fgkMainCDB)->Put(object, id, metaData);
84090f85 198}
17111222 199
84090f85 200//______________________________________________________________________________________________
9827400b 201Bool_t AliTestShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 202{
203 // Stores the object as reference data
204 // This function should be called at the end of the preprocessor cycle
205 //
206 // This implementation just stores it on the local disk, the full AliShuttle
207 // puts it to the Grid FileCatalog
208
8c77aee7 209 AliCDBId id(path, fRun, fRun);
84090f85 210
e19c88ea 211 return AliCDBManager::Instance()->GetStorage(fgkMainRefStorage)->Put(object, id, metaData);
5c6b40ae 212}
213
36137ac1 214//______________________________________________________________________________________________
9827400b 215Bool_t AliTestShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
216{
217 //
218 // Stores reference file directly (without opening it).
219 //
220 // This implementation just stores it on the local disk, the full AliShuttle
221 // puts it to the Grid FileCatalog
222
223 AliCDBManager* man = AliCDBManager::Instance();
77dc0223 224 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
9827400b 225
226 TString localBaseFolder = sto->GetBaseFolder();
227
77dc0223 228 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
9827400b 229
77dc0223 230 return CopyFileLocally(targetDir, localFile, gridFileName);
231}
232
233//______________________________________________________________________________________________
234Bool_t AliTestShuttle::StoreRunMetadataFile(const char* localFile, const char* gridFileName)
235{
236 //
237 // Stores Run metadata file to the Grid, in the run folder
238 //
239 // Only GRP can call this function.
9827400b 240
77dc0223 241 AliCDBManager* man = AliCDBManager::Instance();
242 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
243
244 TString localBaseFolder = sto->GetBaseFolder();
245
246 // Build Run level folder
247 // folder = /alice/data/year/lhcPeriod/runNb/Raw
248
249 TTimeStamp startTime(fStartTime);
250
251 TString year = Form("%d",startTime.GetDate());
252 year = year(0,4);
253
254 TString lhcPeriod = GetRunParameter("LHCperiod");
255
256 if (lhcPeriod.Length() == 0)
9827400b 257 {
77dc0223 258 Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
259 return 0;
260 }
261
262 // TODO: currently SHUTTLE cannot write in /alice/data/ !!!!!
263 //TString targetDir = Form("%s/GRP/RunMetadata/alice/data/%s/%s/%d/Raw",
264 // localBaseFolder.Data(), year.Data(),
265 // lhcPeriod.Data(), fRun);
266
267 TString targetDir = Form("%s/GRP/RunMetadata/alice/simulation/%s/%s/%d/Raw",
268 localBaseFolder.Data(), year.Data(),
269 lhcPeriod.Data(), fRun);
270
271 return CopyFileLocally(targetDir, localFile, gridFileName);
272}
273
274//______________________________________________________________________________________________
275Bool_t AliTestShuttle::CopyFileLocally(TString& targetDir, const char* localFile, const char* gridFileName)
276{
277 //
278 // Stores file locally. Called by StoreReferenceFile and StoreRunMetadataFile
279 //
280
281 //try to open folder, if it does not exist
282 void* dir = gSystem->OpenDirectory(targetDir.Data());
283 if (dir == NULL) {
284 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
285 Log("SHUTTLE", Form("StoreFileLocally - Can't open directory <%s>", targetDir.Data()));
9827400b 286 return kFALSE;
287 }
77dc0223 288
289 } else {
290 gSystem->FreeDirectory(dir);
9827400b 291 }
77dc0223 292
293 TString target = Form("%s/%s", targetDir.Data(), gridFileName);
294
295 Int_t result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
296 if (result)
297 {
298 Log("SHUTTLE", Form("StoreFileLocally - %s does not exist", localFile));
299 return kFALSE;
300 }
301
9827400b 302 result = gSystem->CopyFile(localFile, target);
303
304 if (result == 0)
305 {
77dc0223 306 Log("SHUTTLE", Form("StoreFileLocally - File %s stored locally to %s", localFile, target.Data()));
9827400b 307 return kTRUE;
308 }
309 else
310 {
77dc0223 311 Log("SHUTTLE", Form("StoreFileLocally - Could not store file %s to %s!. Error code = %d",
312 localFile, target.Data(), result));
9827400b 313 return kFALSE;
77dc0223 314 }
315
316
317
318}
319
320//______________________________________________________________________________________________
321const char* AliTestShuttle::GetRefFilePrefix(const char* base, const char* detector)
322{
323 //
324 // Get folder name of reference files
325 //
326
327 TString offDetStr(GetOfflineDetName(detector));
328 TString dir;
329 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
330 {
331 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
332 } else {
333 dir.Form("%s/%s", base, offDetStr.Data());
334 }
335
336 return dir.Data();
337
338
9827400b 339}
340
341//______________________________________________________________________________________________
5c6b40ae 342const char* AliTestShuttle::GetFile(Int_t system, const char* detector, const char* id, const char* source)
343{
344 // This function retrieves a file from the given system (kDAQ, kDCS, kHLT) with the given file id
345 // and from the given source in the system.
346 // The function returnes the path to the local file.
347 //
348 // test implementation of GetFile
349 // takes files from the local disks, files are passen in a TMap in the constructor
350
351 TString key;
352 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
353 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(key.Data()));
354 TMap* sourceList = 0;
355 if (sourceListPair)
356 sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
357 if (!sourceList)
358 {
359 AliError(Form("Could not find any file in %s with id %s (%s)", fkSystemNames[system], id, key.Data()));
360 return 0;
361 }
362
4a33bdd9 363 TObjString* fileName = 0;
5c6b40ae 364 TPair* fileNamePair = dynamic_cast<TPair*> (sourceList->FindObject(source));
4a33bdd9 365 if (fileNamePair)
366 fileName = dynamic_cast<TObjString*> (fileNamePair->Value());
5c6b40ae 367 if (!fileName)
368 {
36137ac1 369 AliError(Form("Could not find files from source %s in %s with id %s",
370 source, fkSystemNames[system], id));
5c6b40ae 371 return 0;
372 }
373
374 return fileName->GetString().Data();
375}
376
36137ac1 377//______________________________________________________________________________________________
5c6b40ae 378TList* AliTestShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
379{
380 // Returns a list of sources in a given system that saved a file with the given id
381 //
382 // test implementation of GetFileSources
383 // takes files from the local disks, files are passen in a TMap in the constructor
384
385 TString key;
4a33bdd9 386 if (id)
387 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
388 else
389 key.Form("%s-%s", fkSystemNames[system], detector);
390
391 TList* list = new TList;
392
393 TIterator* iter = fInputFiles->MakeIterator();
394 TObject* obj = 0;
395 while ((obj = iter->Next()))
5c6b40ae 396 {
4a33bdd9 397 TObjString* objStr = dynamic_cast<TObjString*> (obj);
398 if (objStr)
399 {
400 Bool_t found = kFALSE;
401 if (id)
402 {
403 found = (objStr->String().CompareTo(key) == 0);
404 }
405 else
406 found = objStr->String().BeginsWith(key);
407
408 if (found)
409 {
410 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(objStr->String().Data()));
411 TMap* sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
412
413 TIterator* iter2 = sourceList->GetTable()->MakeIterator();
414 TObject* obj2 = 0;
415 while ((obj2 = iter2->Next()))
416 {
417 TPair* pair = dynamic_cast<TPair*> (obj2);
418 if (pair)
419 {
420 if (!list->FindObject(pair->Key()))
421 list->Add(new TObjString(pair->Key()->GetName()));
422 }
423 }
424
425 delete iter2;
426 }
427 }
5c6b40ae 428 }
4a33bdd9 429
430 if (list->GetEntries() == 0)
431 AliInfo(Form("Could not find any file in %s with id %s (%s)", fkSystemNames[system], id, key.Data()));
432
433 return list;
434}
5c6b40ae 435
4a33bdd9 436//______________________________________________________________________________________________
437TList* AliTestShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
438{
439 // Returns a list of ids in a given system that saved a file with the given source
440 //
441 // test implementation of GetFileSources
442 // takes files from the local disks, files are passen in a TMap in the constructor
443
444
445 TString key;
446 key.Form("%s-%s", fkSystemNames[system], detector);
447
5c6b40ae 448 TList* list = new TList;
4a33bdd9 449
450 TIterator* iter = fInputFiles->MakeIterator();
451 TObject* obj = 0;
5c6b40ae 452 while ((obj = iter->Next()))
453 {
4a33bdd9 454 TObjString* objStr = dynamic_cast<TObjString*> (obj);
455 if (objStr)
456 {
457 if (objStr->String().BeginsWith(key))
458 {
459 Bool_t found = kFALSE;
460
461 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(objStr->String().Data()));
462 TMap* sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
463
464 TIterator* iter2 = sourceList->GetTable()->MakeIterator();
465 TObject* obj2 = 0;
466 while ((obj2 = iter2->Next()))
467 {
468 TPair* pair = dynamic_cast<TPair*> (obj2);
469 if (pair)
470 {
471 if (strcmp(pair->Key()->GetName(), source) == 0)
472 found = kTRUE;
473 }
474 }
475
476 delete iter2;
477
478 if (found)
479 {
480 TObjArray* tokens = objStr->String().Tokenize("-");
481 if (tokens->GetEntries() == 3)
482 {
483 TObjString* id = dynamic_cast<TObjString*> (tokens->At(2));
484 if (id && !list->FindObject(id->String()))
485 list->Add(new TObjString(id->String()));
486 }
487
488 delete tokens;
489
490 }
491 }
492 }
5c6b40ae 493 }
4a33bdd9 494
495 if (list->GetEntries() == 0)
496 AliInfo(Form("Could not find any file in %s with source %s (%s)", fkSystemNames[system], source, key.Data()));
497
5c6b40ae 498 return list;
499}
500
36137ac1 501//______________________________________________________________________________________________
5c6b40ae 502void AliTestShuttle::Log(const char* detector, const char* message)
503{
504 // test implementation of Log
505 // just prints to the screen
506
507 AliInfo(Form("%s: %s", detector, message));
508}
509
36137ac1 510//______________________________________________________________________________________________
511void AliTestShuttle::AddInputFile(Int_t system, const char* detector, const char* id, const char* source, const char* fileName)
512{
4a33bdd9 513 //
36137ac1 514 // This function adds a file to the list of input files
4a33bdd9 515 // the list is stored in fInputFiles
516 // fInputFiles: TMap (key -> value)
517 // <system>-<detector>-<id> -> TMap (key -> value)
518 // <source> -> <filename>
519 //
520
36137ac1 521 TString key;
522 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
523 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(key.Data()));
524 TMap* sourceList = 0;
525 if (sourceListPair)
526 sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
527 if (!sourceList)
528 {
529 sourceList = new TMap;
530 fInputFiles->Add(new TObjString(key), sourceList);
531 }
532
533 sourceList->Add(new TObjString(source), new TObjString(fileName));
534}
535
536//______________________________________________________________________________________________
d461a8a7 537Bool_t AliTestShuttle::AddInputCDBEntry(AliCDBEntry* entry)
538{
539 // This function adds an object in the OCDB to be later retrieved with GetFromOCDB
540
541 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
542 if (!sto)
543 {
544 Log("SHUTTLE", "GetFromOCDB - Cannot activate main OCDB for query!");
545 return 0;
546 }
547
548 return sto->Put(entry);
549}
550
551//______________________________________________________________________________________________
9827400b 552AliCDBEntry* AliTestShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
d461a8a7 553{
554// returns obiect from OCDB valid for current run
555
556 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
557 if (!sto)
558 {
559 Log("SHUTTLE", "GetFromOCDB - Cannot activate main OCDB for query!");
560 return 0;
561 }
562
563 return (AliCDBEntry*) sto->Get(path, fRun);
564}
565
566//______________________________________________________________________________________________
36137ac1 567void AliTestShuttle::Process()
568{
569 // This function tests all preprocessors that are registered to it
570 // All preprocessors get the same dcs alias map and have access to the same list of files.
571
572 for (Int_t i=0; i<fPreprocessors->GetEntries(); ++i)
573 {
574 AliPreprocessor* preprocessor = dynamic_cast<AliPreprocessor*> (fPreprocessors->At(i));
575 if (preprocessor)
576 {
17111222 577 preprocessor->Initialize(fRun, fStartTime, fEndTime);
36137ac1 578 preprocessor->Process(fDcsAliasMap);
579 }
580 }
581}
582
583//______________________________________________________________________________________________
584void AliTestShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
585{
586 // registers a preprocessor
587
eba76848 588 const char* detName = preprocessor->GetName();
589 if(strcmp("DET", detName) != 0) {
590 if(GetDetPos(detName) < 0)
591 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
441b0e9c 592 }
eba76848 593
594 fPreprocessors->Add(preprocessor);
595}
596
597//______________________________________________________________________________________________
598void AliTestShuttle::AddInputRunParameter(const char* key, const char* value){
599// set a run parameter (in reality it will be read from the DAQ logbook)
600
601 TObjString* keyObj = new TObjString(key);
602 if (fRunParameters->Contains(key)) {
603 AliWarning(Form("Parameter %s already existing and it will be replaced.", key));
604 delete fRunParameters->Remove(keyObj);
605
606 }
607 fRunParameters->Add(keyObj, new TObjString(value));
608 AliDebug(2, Form("Number of parameters: %d", fRunParameters->
609 GetEntries()));
610}
611
612//______________________________________________________________________________________________
9827400b 613const char* AliTestShuttle::GetRunType()
614{
615 //
616 // get a run parameter
617 //
441b0e9c 618
87968cd5 619 return fRunType.Data();
441b0e9c 620}
621
622//______________________________________________________________________________________________
eba76848 623const char* AliTestShuttle::GetRunParameter(const char* key){
624// get a run parameter
625
626 TObjString* value = dynamic_cast<TObjString*> (fRunParameters->GetValue(key));
627 if(!value) {
628 AliError(Form("No such parameter: %s", key));
629 return 0;
630 }
631 return value->GetName();
36137ac1 632}
e19c88ea 633
d461a8a7 634//______________________________________________________________________________________________
635void AliTestShuttle::SetShuttleTempDir(const char* tmpDir)
636{
637// sets Shuttle temp directory
638
639 fgkShuttleTempDir = gSystem->ExpandPathName(tmpDir);
640}
641
642//______________________________________________________________________________________________
643void AliTestShuttle::SetShuttleLogDir(const char* logDir)
644{
645// sets Shuttle log directory
646
647 fgkShuttleLogDir = gSystem->ExpandPathName(logDir);
648}
649