New function StoreRunMetadataFile added to preprocessor and Shuttle interface
[u/mrichter/AliRoot.git] / SHUTTLE / TestShuttle / AliTestShuttle.cxx
CommitLineData
5c6b40ae 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
87968cd5 18Revision 1.13 2007/05/30 06:35:21 jgrosseo
19Adding functionality to the Shuttle/TestShuttle:
20o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
21o) Function to retrieve list of IDs for a given source (GetFileIDs)
22These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
23Example code has been added to the TestProcessor in TestShuttle
24
4a33bdd9 25Revision 1.12 2007/04/27 07:06:48 jgrosseo
26GetFileSources returns empty list in case of no files, but successful query
27No mails sent in testmode
28
86aa42c3 29Revision 1.11 2007/04/04 10:33:36 jgrosseo
301) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
31In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
32
332) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
34
353) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
36
374) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
38
395) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
40If you always need DCS data (like before), you do not need to implement it.
41
426) The run type has been added to the monitoring page
43
9827400b 44Revision 1.10 2007/02/28 10:41:01 acolla
45Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
46AliPreprocessor::GetRunType() function.
47Added some ldap definition files.
48
d461a8a7 49Revision 1.8 2007/02/13 11:22:25 acolla
50Shuttle getters and setters of main/local OCDB/Reference storages, temp and log
51folders moved to AliShuttleInterface
52
e19c88ea 53Revision 1.6 2006/11/06 14:22:47 jgrosseo
54major update (Alberto)
55o) reading of run parameters from the logbook
56o) online offline naming conversion
57o) standalone DCSclient package
58
eba76848 59Revision 1.5 2006/10/02 12:58:52 jgrosseo
60Small interface change in StoreReferenceData
61
8c77aee7 62Revision 1.4 2006/08/08 14:19:07 jgrosseo
63Update to shuttle classes (Alberto)
64
65- Possibility to set the full object's path in the Preprocessor's and
66Shuttle's Store functions
67- Possibility to extend the object's run validity in the same classes
68("startValidity" and "validityInfinite" parameters)
69- Implementation of the StoreReferenceData function to store reference
70data in a dedicated CDB storage.
71
84090f85 72Revision 1.3 2006/07/11 12:44:32 jgrosseo
73adding parameters for extended validity range of data produced by preprocessor
74
17111222 75Revision 1.2 2006/06/06 14:20:05 jgrosseo
76o) updated test preprocessor (alberto)
77o) added comments to example macro
78o) test shuttle implements new interface
79
5c6b40ae 80Revision 1.2 2006/03/07 07:52:34 hristov
81New version (B.Yordanov)
82
83Revision 1.3 2005/11/17 17:47:34 byordano
84TList changed to TObjArray
85
86Revision 1.2 2005/11/17 14:43:22 byordano
87import to local CVS
88
89Revision 1.1.1.1 2005/10/28 07:33:58 hristov
90Initial import as subdirectory in AliRoot
91
92Revision 1.1.1.1 2005/09/12 22:11:40 byordano
93SHUTTLE package
94
95Revision 1.2 2005/08/29 21:15:47 byordano
96some docs added
97
98*/
99
100//
101// test implementation of the AliShuttleInterface, to be used for local tests of preprocessors
102//
103// reads files from the local disk
104// stores to local CDB
105// logs to the screen
106//
107
108#include "AliTestShuttle.h"
109#include "AliLog.h"
110
111#include "AliCDBManager.h"
84090f85 112#include "AliCDBStorage.h"
5c6b40ae 113#include "AliCDBMetaData.h"
84090f85 114#include "AliCDBPath.h"
5c6b40ae 115#include "AliCDBId.h"
36137ac1 116#include "AliPreprocessor.h"
5c6b40ae 117
118#include <TMap.h>
119#include <TList.h>
5c6b40ae 120#include <TObjString.h>
e19c88ea 121#include <TSystem.h>
5c6b40ae 122
123ClassImp(AliTestShuttle)
124
36137ac1 125//______________________________________________________________________________________________
17111222 126AliTestShuttle::AliTestShuttle(Int_t run, UInt_t startTime, UInt_t endTime) :
127 fRun(run),
128 fStartTime(startTime),
129 fEndTime(endTime),
36137ac1 130 fInputFiles(0),
eba76848 131 fRunParameters(0),
9827400b 132 fRunType(),
36137ac1 133 fPreprocessors(0),
134 fDcsAliasMap(0)
5c6b40ae 135{
136 // constructor
36137ac1 137
138 fInputFiles = new TMap;
eba76848 139 fRunParameters = new TMap;
36137ac1 140 fPreprocessors = new TObjArray;
441b0e9c 141
142 fInputFiles->SetOwner(1);
143 fRunParameters->SetOwner(1);
441b0e9c 144 fPreprocessors->SetOwner(1);
5c6b40ae 145}
146
36137ac1 147//______________________________________________________________________________________________
5c6b40ae 148AliTestShuttle::~AliTestShuttle()
149{
150 // destructor
36137ac1 151
152 delete fInputFiles;
153 fInputFiles = 0;
154
eba76848 155 delete fRunParameters;
156 fRunParameters = 0;
157
36137ac1 158 delete fPreprocessors;
159 fPreprocessors = 0;
160
161 delete fDcsAliasMap;
162 fDcsAliasMap = 0;
5c6b40ae 163}
164
36137ac1 165//______________________________________________________________________________________________
9827400b 166Bool_t AliTestShuttle::Store(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
84090f85 167 Int_t validityStart, Bool_t validityInfinite)
5c6b40ae 168{
169 // Stores the CDB object
170 // This function should be called at the end of the preprocessor cycle
171 //
172 // This implementation just stores it on the local disk, the full AliShuttle
173 // puts it to the Grid FileCatalog
174
84090f85 175 Int_t startRun = fRun - validityStart;
176 if(startRun < 0) {
177 AliError("First valid run happens to be less than 0! Setting it to 0...");
178 startRun=0;
179 }
17111222 180
84090f85 181 Int_t endRun = -1;
182 if(validityInfinite) {
183 endRun = AliCDBRunRange::Infinity();
184 } else {
185 endRun = fRun;
186 }
17111222 187
84090f85 188 AliCDBId id(path, startRun, endRun);
17111222 189
e19c88ea 190 return AliCDBManager::Instance()->GetStorage(fgkMainCDB)->Put(object, id, metaData);
84090f85 191}
17111222 192
84090f85 193//______________________________________________________________________________________________
9827400b 194Bool_t AliTestShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 195{
196 // Stores the object as reference data
197 // This function should be called at the end of the preprocessor cycle
198 //
199 // This implementation just stores it on the local disk, the full AliShuttle
200 // puts it to the Grid FileCatalog
201
8c77aee7 202 AliCDBId id(path, fRun, fRun);
84090f85 203
e19c88ea 204 return AliCDBManager::Instance()->GetStorage(fgkMainRefStorage)->Put(object, id, metaData);
5c6b40ae 205}
206
36137ac1 207//______________________________________________________________________________________________
9827400b 208Bool_t AliTestShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
209{
210 //
211 // Stores reference file directly (without opening it).
212 //
213 // This implementation just stores it on the local disk, the full AliShuttle
214 // puts it to the Grid FileCatalog
215
216 AliCDBManager* man = AliCDBManager::Instance();
217 AliCDBStorage* sto = man->GetStorage(fgkMainRefStorage);
218
219 TString localBaseFolder = sto->GetBaseFolder();
220
221 TString targetDir;
222 targetDir.Form("%s/%s", localBaseFolder.Data(), detector);
223
224 TString target;
225 target.Form("%s/%d_%s", targetDir.Data(), fRun, gridFileName);
226
227 Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
228 if (result)
229 {
230 result = gSystem->mkdir(targetDir, kTRUE);
231 if (result != 0)
232 {
233 Log("SHUTTLE", Form("StoreReferenceFile - Error creating base directory %s", targetDir.Data()));
234 return kFALSE;
235 }
236 }
237
238 result = gSystem->CopyFile(localFile, target);
239
240 if (result == 0)
241 {
242 Log("SHUTTLE", Form("StoreReferenceFile - Stored file %s locally to %s", localFile, target.Data()));
243 return kTRUE;
244 }
245 else
246 {
247 Log("SHUTTLE", Form("StoreReferenceFile - Storing file %s locally to %s failed with %d", localFile, target.Data(), result));
248 return kFALSE;
249 }
250}
251
252//______________________________________________________________________________________________
5c6b40ae 253const char* AliTestShuttle::GetFile(Int_t system, const char* detector, const char* id, const char* source)
254{
255 // This function retrieves a file from the given system (kDAQ, kDCS, kHLT) with the given file id
256 // and from the given source in the system.
257 // The function returnes the path to the local file.
258 //
259 // test implementation of GetFile
260 // takes files from the local disks, files are passen in a TMap in the constructor
261
262 TString key;
263 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
264 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(key.Data()));
265 TMap* sourceList = 0;
266 if (sourceListPair)
267 sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
268 if (!sourceList)
269 {
270 AliError(Form("Could not find any file in %s with id %s (%s)", fkSystemNames[system], id, key.Data()));
271 return 0;
272 }
273
4a33bdd9 274 TObjString* fileName = 0;
5c6b40ae 275 TPair* fileNamePair = dynamic_cast<TPair*> (sourceList->FindObject(source));
4a33bdd9 276 if (fileNamePair)
277 fileName = dynamic_cast<TObjString*> (fileNamePair->Value());
5c6b40ae 278 if (!fileName)
279 {
36137ac1 280 AliError(Form("Could not find files from source %s in %s with id %s",
281 source, fkSystemNames[system], id));
5c6b40ae 282 return 0;
283 }
284
285 return fileName->GetString().Data();
286}
287
36137ac1 288//______________________________________________________________________________________________
5c6b40ae 289TList* AliTestShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
290{
291 // Returns a list of sources in a given system that saved a file with the given id
292 //
293 // test implementation of GetFileSources
294 // takes files from the local disks, files are passen in a TMap in the constructor
295
296 TString key;
4a33bdd9 297 if (id)
298 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
299 else
300 key.Form("%s-%s", fkSystemNames[system], detector);
301
302 TList* list = new TList;
303
304 TIterator* iter = fInputFiles->MakeIterator();
305 TObject* obj = 0;
306 while ((obj = iter->Next()))
5c6b40ae 307 {
4a33bdd9 308 TObjString* objStr = dynamic_cast<TObjString*> (obj);
309 if (objStr)
310 {
311 Bool_t found = kFALSE;
312 if (id)
313 {
314 found = (objStr->String().CompareTo(key) == 0);
315 }
316 else
317 found = objStr->String().BeginsWith(key);
318
319 if (found)
320 {
321 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(objStr->String().Data()));
322 TMap* sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
323
324 TIterator* iter2 = sourceList->GetTable()->MakeIterator();
325 TObject* obj2 = 0;
326 while ((obj2 = iter2->Next()))
327 {
328 TPair* pair = dynamic_cast<TPair*> (obj2);
329 if (pair)
330 {
331 if (!list->FindObject(pair->Key()))
332 list->Add(new TObjString(pair->Key()->GetName()));
333 }
334 }
335
336 delete iter2;
337 }
338 }
5c6b40ae 339 }
4a33bdd9 340
341 if (list->GetEntries() == 0)
342 AliInfo(Form("Could not find any file in %s with id %s (%s)", fkSystemNames[system], id, key.Data()));
343
344 return list;
345}
5c6b40ae 346
4a33bdd9 347//______________________________________________________________________________________________
348TList* AliTestShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
349{
350 // Returns a list of ids in a given system that saved a file with the given source
351 //
352 // test implementation of GetFileSources
353 // takes files from the local disks, files are passen in a TMap in the constructor
354
355
356 TString key;
357 key.Form("%s-%s", fkSystemNames[system], detector);
358
5c6b40ae 359 TList* list = new TList;
4a33bdd9 360
361 TIterator* iter = fInputFiles->MakeIterator();
362 TObject* obj = 0;
5c6b40ae 363 while ((obj = iter->Next()))
364 {
4a33bdd9 365 TObjString* objStr = dynamic_cast<TObjString*> (obj);
366 if (objStr)
367 {
368 if (objStr->String().BeginsWith(key))
369 {
370 Bool_t found = kFALSE;
371
372 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(objStr->String().Data()));
373 TMap* sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
374
375 TIterator* iter2 = sourceList->GetTable()->MakeIterator();
376 TObject* obj2 = 0;
377 while ((obj2 = iter2->Next()))
378 {
379 TPair* pair = dynamic_cast<TPair*> (obj2);
380 if (pair)
381 {
382 if (strcmp(pair->Key()->GetName(), source) == 0)
383 found = kTRUE;
384 }
385 }
386
387 delete iter2;
388
389 if (found)
390 {
391 TObjArray* tokens = objStr->String().Tokenize("-");
392 if (tokens->GetEntries() == 3)
393 {
394 TObjString* id = dynamic_cast<TObjString*> (tokens->At(2));
395 if (id && !list->FindObject(id->String()))
396 list->Add(new TObjString(id->String()));
397 }
398
399 delete tokens;
400
401 }
402 }
403 }
5c6b40ae 404 }
4a33bdd9 405
406 if (list->GetEntries() == 0)
407 AliInfo(Form("Could not find any file in %s with source %s (%s)", fkSystemNames[system], source, key.Data()));
408
5c6b40ae 409 return list;
410}
411
36137ac1 412//______________________________________________________________________________________________
5c6b40ae 413void AliTestShuttle::Log(const char* detector, const char* message)
414{
415 // test implementation of Log
416 // just prints to the screen
417
418 AliInfo(Form("%s: %s", detector, message));
419}
420
36137ac1 421//______________________________________________________________________________________________
422void AliTestShuttle::AddInputFile(Int_t system, const char* detector, const char* id, const char* source, const char* fileName)
423{
4a33bdd9 424 //
36137ac1 425 // This function adds a file to the list of input files
4a33bdd9 426 // the list is stored in fInputFiles
427 // fInputFiles: TMap (key -> value)
428 // <system>-<detector>-<id> -> TMap (key -> value)
429 // <source> -> <filename>
430 //
431
36137ac1 432 TString key;
433 key.Form("%s-%s-%s", fkSystemNames[system], detector, id);
434 TPair* sourceListPair = dynamic_cast<TPair*> (fInputFiles->FindObject(key.Data()));
435 TMap* sourceList = 0;
436 if (sourceListPair)
437 sourceList = dynamic_cast<TMap*> (sourceListPair->Value());
438 if (!sourceList)
439 {
440 sourceList = new TMap;
441 fInputFiles->Add(new TObjString(key), sourceList);
442 }
443
444 sourceList->Add(new TObjString(source), new TObjString(fileName));
445}
446
447//______________________________________________________________________________________________
d461a8a7 448Bool_t AliTestShuttle::AddInputCDBEntry(AliCDBEntry* entry)
449{
450 // This function adds an object in the OCDB to be later retrieved with GetFromOCDB
451
452 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
453 if (!sto)
454 {
455 Log("SHUTTLE", "GetFromOCDB - Cannot activate main OCDB for query!");
456 return 0;
457 }
458
459 return sto->Put(entry);
460}
461
462//______________________________________________________________________________________________
9827400b 463AliCDBEntry* AliTestShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
d461a8a7 464{
465// returns obiect from OCDB valid for current run
466
467 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
468 if (!sto)
469 {
470 Log("SHUTTLE", "GetFromOCDB - Cannot activate main OCDB for query!");
471 return 0;
472 }
473
474 return (AliCDBEntry*) sto->Get(path, fRun);
475}
476
477//______________________________________________________________________________________________
36137ac1 478void AliTestShuttle::Process()
479{
480 // This function tests all preprocessors that are registered to it
481 // All preprocessors get the same dcs alias map and have access to the same list of files.
482
483 for (Int_t i=0; i<fPreprocessors->GetEntries(); ++i)
484 {
485 AliPreprocessor* preprocessor = dynamic_cast<AliPreprocessor*> (fPreprocessors->At(i));
486 if (preprocessor)
487 {
17111222 488 preprocessor->Initialize(fRun, fStartTime, fEndTime);
36137ac1 489 preprocessor->Process(fDcsAliasMap);
490 }
491 }
492}
493
494//______________________________________________________________________________________________
495void AliTestShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
496{
497 // registers a preprocessor
498
eba76848 499 const char* detName = preprocessor->GetName();
500 if(strcmp("DET", detName) != 0) {
501 if(GetDetPos(detName) < 0)
502 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
441b0e9c 503 }
eba76848 504
505 fPreprocessors->Add(preprocessor);
506}
507
508//______________________________________________________________________________________________
509void AliTestShuttle::AddInputRunParameter(const char* key, const char* value){
510// set a run parameter (in reality it will be read from the DAQ logbook)
511
512 TObjString* keyObj = new TObjString(key);
513 if (fRunParameters->Contains(key)) {
514 AliWarning(Form("Parameter %s already existing and it will be replaced.", key));
515 delete fRunParameters->Remove(keyObj);
516
517 }
518 fRunParameters->Add(keyObj, new TObjString(value));
519 AliDebug(2, Form("Number of parameters: %d", fRunParameters->
520 GetEntries()));
521}
522
523//______________________________________________________________________________________________
9827400b 524const char* AliTestShuttle::GetRunType()
525{
526 //
527 // get a run parameter
528 //
441b0e9c 529
87968cd5 530 return fRunType.Data();
441b0e9c 531}
532
533//______________________________________________________________________________________________
eba76848 534const char* AliTestShuttle::GetRunParameter(const char* key){
535// get a run parameter
536
537 TObjString* value = dynamic_cast<TObjString*> (fRunParameters->GetValue(key));
538 if(!value) {
539 AliError(Form("No such parameter: %s", key));
540 return 0;
541 }
542 return value->GetName();
36137ac1 543}
e19c88ea 544
d461a8a7 545//______________________________________________________________________________________________
546void AliTestShuttle::SetShuttleTempDir(const char* tmpDir)
547{
548// sets Shuttle temp directory
549
550 fgkShuttleTempDir = gSystem->ExpandPathName(tmpDir);
551}
552
553//______________________________________________________________________________________________
554void AliTestShuttle::SetShuttleLogDir(const char* logDir)
555{
556// sets Shuttle log directory
557
558 fgkShuttleLogDir = gSystem->ExpandPathName(logDir);
559}
560