]> git.uio.no Git - u/mrichter/AliRoot.git/blame - SHUTTLE/AliShuttle.cxx
git://git.uio.no / u / mrichter / AliRoot.git / blame
? search:
summary | shortlog | log | commit | commitdiff | tree
blob | blame (incremental) | history | HEAD
TH1 removed
[u/mrichter/AliRoot.git] / SHUTTLE / AliShuttle.cxx
CommitLineData
73abe331 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
2d9019b4 18Revision 1.43 2007/05/10 09:59:51 acolla
19Various bug fixes in StoreRefFilesToGrid; Cleaning of reference storage before processing detector (CleanReferenceStorage)
20
546242fb 21Revision 1.42 2007/05/03 08:01:39 jgrosseo
22typo in last commit :-(
23
8b739301 24Revision 1.41 2007/05/03 08:00:48 jgrosseo
25fixing log message when pp want to skip dcs value retrieval
26
651fdaab 27Revision 1.40 2007/04/27 07:06:48 jgrosseo
28GetFileSources returns empty list in case of no files, but successful query
29No mails sent in testmode
30
86aa42c3 31Revision 1.39 2007/04/17 12:43:57 acolla
32Correction in StoreOCDB; change of text in mail to detector expert
33
26758fce 34Revision 1.38 2007/04/12 08:26:18 jgrosseo
35updated comment
36
3c2a21c8 37Revision 1.37 2007/04/10 16:53:14 jgrosseo
38redirecting sub detector stdout, stderr to sub detector log file
39
3d8bc902 40Revision 1.35 2007/04/04 16:26:38 acolla
411. Re-organization of function calls in TestPreprocessor to make it more meaningful.
422. Added missing dependency in test preprocessors.
433. in AliShuttle.cxx: processing time and memory consumption info on a single line.
44
886d60e6 45Revision 1.34 2007/04/04 10:33:36 jgrosseo
461) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
47In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
48
492) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
50
513) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
52
534) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
54
555) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
56If you always need DCS data (like before), you do not need to implement it.
57
586) The run type has been added to the monitoring page
59
9827400b 60Revision 1.33 2007/04/03 13:56:01 acolla
61Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
62run type.
63
3301427a 64Revision 1.32 2007/02/28 10:41:56 acolla
65Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
66AliPreprocessor::GetRunType() function.
67Added some ldap definition files.
68
d386d623 69Revision 1.30 2007/02/13 11:23:21 acolla
70Moved getters and setters of Shuttle's main OCDB/Reference, local
71OCDB/Reference, temp and log folders to AliShuttleInterface
72
9d733021 73Revision 1.27 2007/01/30 17:52:42 jgrosseo
74adding monalisa monitoring
75
e7f62f16 76Revision 1.26 2007/01/23 19:20:03 acolla
77Removed old ldif files, added TOF, MCH ldif files. Added some options in
78AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
79SetShuttleLogDir
80
36c99a6a 81Revision 1.25 2007/01/15 19:13:52 acolla
82Moved some AliInfo to AliDebug in SendMail function
83
fc5a4708 84Revision 1.21 2006/12/07 08:51:26 jgrosseo
85update (alberto):
86table, db names in ldap configuration
87added GRP preprocessor
88DCS data can also be retrieved by data point
89
2c15234c 90Revision 1.20 2006/11/16 16:16:48 jgrosseo
91introducing strict run ordering flag
92removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
93
be48e3ea 94Revision 1.19 2006/11/06 14:23:04 jgrosseo
95major update (Alberto)
96o) reading of run parameters from the logbook
97o) online offline naming conversion
98o) standalone DCSclient package
99
eba76848 100Revision 1.18 2006/10/20 15:22:59 jgrosseo
101o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
102o) Merging Collect, CollectAll, CollectNew function
103o) Removing implementation of empty copy constructors (declaration still there!)
104
cb343cfd 105Revision 1.17 2006/10/05 16:20:55 jgrosseo
106adapting to new CDB classes
107
6ec0e06c 108Revision 1.16 2006/10/05 15:46:26 jgrosseo
109applying to the new interface
110
481441a2 111Revision 1.15 2006/10/02 16:38:39 jgrosseo
112update (alberto):
113fixed memory leaks
114storing of objects that failed to be stored to the grid before
115interfacing of shuttle status table in daq system
116
2bb7b766 117Revision 1.14 2006/08/29 09:16:05 jgrosseo
118small update
119
85a80aa9 120Revision 1.13 2006/08/15 10:50:00 jgrosseo
121effc++ corrections (alberto)
122
4f0ab988 123Revision 1.12 2006/08/08 14:19:29 jgrosseo
124Update to shuttle classes (Alberto)
125
126- Possibility to set the full object's path in the Preprocessor's and
127Shuttle's Store functions
128- Possibility to extend the object's run validity in the same classes
129("startValidity" and "validityInfinite" parameters)
130- Implementation of the StoreReferenceData function to store reference
131data in a dedicated CDB storage.
132
84090f85 133Revision 1.11 2006/07/21 07:37:20 jgrosseo
134last run is stored after each run
135
7bfb2090 136Revision 1.10 2006/07/20 09:54:40 jgrosseo
137introducing status management: The processing per subdetector is divided into several steps,
138after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
139can keep track of the number of failures and skips further processing after a certain threshold is
140exceeded. These thresholds can be configured in LDAP.
141
5164a766 142Revision 1.9 2006/07/19 10:09:55 jgrosseo
143new configuration, accesst to DAQ FES (Alberto)
144
57f50b3c 145Revision 1.8 2006/07/11 12:44:36 jgrosseo
146adding parameters for extended validity range of data produced by preprocessor
147
17111222 148Revision 1.7 2006/07/10 14:37:09 jgrosseo
149small fix + todo comment
150
e090413b 151Revision 1.6 2006/07/10 13:01:41 jgrosseo
152enhanced storing of last sucessfully processed run (alberto)
153
a7160fe9 154Revision 1.5 2006/07/04 14:59:57 jgrosseo
155revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
156
45a493ce 157Revision 1.4 2006/06/12 09:11:16 jgrosseo
158coding conventions (Alberto)
159
58bc3020 160Revision 1.3 2006/06/06 14:26:40 jgrosseo
161o) removed files that were moved to STEER
162o) shuttle updated to follow the new interface (Alberto)
163
b948db8d 164Revision 1.2 2006/03/07 07:52:34 hristov
165New version (B.Yordanov)
166
d477ad88 167Revision 1.6 2005/11/19 17:19:14 byordano
168RetrieveDATEEntries and RetrieveConditionsData added
169
170Revision 1.5 2005/11/19 11:09:27 byordano
171AliShuttle declaration added
172
173Revision 1.4 2005/11/17 17:47:34 byordano
174TList changed to TObjArray
175
176Revision 1.3 2005/11/17 14:43:23 byordano
177import to local CVS
178
179Revision 1.1.1.1 2005/10/28 07:33:58 hristov
180Initial import as subdirectory in AliRoot
181
73abe331 182Revision 1.2 2005/09/13 08:41:15 byordano
183default startTime endTime added
184
185Revision 1.4 2005/08/30 09:13:02 byordano
186some docs added
187
188Revision 1.3 2005/08/29 21:15:47 byordano
189some docs added
190
191*/
192
193//
194// This class is the main manager for AliShuttle.
195// It organizes the data retrieval from DCS and call the
b948db8d 196// interface methods of AliPreprocessor.
73abe331 197// For every detector in AliShuttleConfgi (see AliShuttleConfig),
198// data for its set of aliases is retrieved. If there is registered
b948db8d 199// AliPreprocessor for this detector then it will be used
200// accroding to the schema (see AliPreprocessor).
201// If there isn't registered AliPreprocessor than the retrieved
73abe331 202// data is stored automatically to the undelying AliCDBStorage.
203// For detSpec is used the alias name.
204//
205
206#include "AliShuttle.h"
207
208#include "AliCDBManager.h"
209#include "AliCDBStorage.h"
210#include "AliCDBId.h"
84090f85 211#include "AliCDBRunRange.h"
212#include "AliCDBPath.h"
5164a766 213#include "AliCDBEntry.h"
73abe331 214#include "AliShuttleConfig.h"
eba76848 215#include "DCSClient/AliDCSClient.h"
73abe331 216#include "AliLog.h"
b948db8d 217#include "AliPreprocessor.h"
5164a766 218#include "AliShuttleStatus.h"
2bb7b766 219#include "AliShuttleLogbookEntry.h"
73abe331 220
57f50b3c 221#include <TSystem.h>
58bc3020 222#include <TObject.h>
b948db8d 223#include <TString.h>
57f50b3c 224#include <TTimeStamp.h>
73abe331 225#include <TObjString.h>
57f50b3c 226#include <TSQLServer.h>
227#include <TSQLResult.h>
228#include <TSQLRow.h>
cb343cfd 229#include <TMutex.h>
9827400b 230#include <TSystemDirectory.h>
231#include <TSystemFile.h>
232#include <TFileMerger.h>
233#include <TGrid.h>
234#include <TGridResult.h>
73abe331 235
e7f62f16 236#include <TMonaLisaWriter.h>
237
5164a766 238#include <fstream>
239
cb343cfd 240#include <sys/types.h>
241#include <sys/wait.h>
242
73abe331 243ClassImp(AliShuttle)
244
b948db8d 245//______________________________________________________________________________________________
246AliShuttle::AliShuttle(const AliShuttleConfig* config,
247 UInt_t timeout, Int_t retries):
4f0ab988 248fConfig(config),
249fTimeout(timeout), fRetries(retries),
250fPreprocessorMap(),
2bb7b766 251fLogbookEntry(0),
eba76848 252fCurrentDetector(),
85a80aa9 253fStatusEntry(0),
cb343cfd 254fMonitoringMutex(0),
eba76848 255fLastActionTime(0),
e7f62f16 256fLastAction(),
9827400b 257fMonaLisa(0),
258fTestMode(kNone),
ffa29e93 259fReadTestMode(kFALSE),
260fOutputRedirected(kFALSE)
73abe331 261{
262 //
263 // config: AliShuttleConfig used
73abe331 264 // timeout: timeout used for AliDCSClient connection
265 // retries: the number of retries in case of connection error.
266 //
267
57f50b3c 268 if (!fConfig->IsValid()) AliFatal("********** !!!!! Invalid configuration !!!!! **********");
be48e3ea 269 for(int iSys=0;iSys<4;iSys++) {
57f50b3c 270 fServer[iSys]=0;
be48e3ea 271 if (iSys < 3)
2c15234c 272 fFXSlist[iSys].SetOwner(kTRUE);
57f50b3c 273 }
2bb7b766 274 fPreprocessorMap.SetOwner(kTRUE);
be48e3ea 275
276 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
277 fFirstUnprocessed[iDet] = kFALSE;
278
cb343cfd 279 fMonitoringMutex = new TMutex();
58bc3020 280}
281
b948db8d 282//______________________________________________________________________________________________
57f50b3c 283AliShuttle::~AliShuttle()
58bc3020 284{
9827400b 285 //
286 // destructor
287 //
58bc3020 288
b948db8d 289 fPreprocessorMap.DeleteAll();
be48e3ea 290 for(int iSys=0;iSys<4;iSys++)
57f50b3c 291 if(fServer[iSys]) {
292 fServer[iSys]->Close();
293 delete fServer[iSys];
eba76848 294 fServer[iSys] = 0;
57f50b3c 295 }
2bb7b766 296
297 if (fStatusEntry){
298 delete fStatusEntry;
299 fStatusEntry = 0;
300 }
cb343cfd 301
302 if (fMonitoringMutex)
303 {
304 delete fMonitoringMutex;
305 fMonitoringMutex = 0;
306 }
73abe331 307}
308
b948db8d 309//______________________________________________________________________________________________
57f50b3c 310void AliShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
58bc3020 311{
73abe331 312 //
b948db8d 313 // Registers new AliPreprocessor.
73abe331 314 // It uses GetName() for indentificator of the pre processor.
315 // The pre processor is registered it there isn't any other
316 // with the same identificator (GetName()).
317 //
318
eba76848 319 const char* detName = preprocessor->GetName();
320 if(GetDetPos(detName) < 0)
321 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
322
323 if (fPreprocessorMap.GetValue(detName)) {
324 AliWarning(Form("AliPreprocessor %s is already registered!", detName));
73abe331 325 return;
326 }
327
eba76848 328 fPreprocessorMap.Add(new TObjString(detName), preprocessor);
73abe331 329}
b948db8d 330//______________________________________________________________________________________________
3301427a 331Bool_t AliShuttle::Store(const AliCDBPath& path, TObject* object,
84090f85 332 AliCDBMetaData* metaData, Int_t validityStart, Bool_t validityInfinite)
73abe331 333{
9827400b 334 // Stores a CDB object in the storage for offline reconstruction. Objects that are not needed for
335 // offline reconstruction, but should be stored anyway (e.g. for debugging) should NOT be stored
336 // using this function. Use StoreReferenceData instead!
337 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
338 // finishes the data are transferred to the main storage (Grid).
b948db8d 339
3301427a 340 return StoreLocally(fgkLocalCDB, path, object, metaData, validityStart, validityInfinite);
84090f85 341}
342
343//______________________________________________________________________________________________
3301427a 344Bool_t AliShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 345{
9827400b 346 // Stores a CDB object in the storage for reference data. This objects will not be available during
347 // offline reconstrunction. Use this function for reference data only!
348 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
349 // finishes the data are transferred to the main storage (Grid).
85a80aa9 350
3301427a 351 return StoreLocally(fgkLocalRefStorage, path, object, metaData);
85a80aa9 352}
353
354//______________________________________________________________________________________________
3301427a 355Bool_t AliShuttle::StoreLocally(const TString& localUri,
85a80aa9 356 const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
357 Int_t validityStart, Bool_t validityInfinite)
358{
9827400b 359 // Store object temporarily in local storage. Parameters are passed by Store and StoreReferenceData functions.
360 // when the preprocessor finishes the data are transferred to the main storage (Grid).
361 // The parameters are:
362 // 1) Uri of the backup storage (Local)
363 // 2) the object's path.
364 // 3) the object to be stored
365 // 4) the metaData to be associated with the object
366 // 5) the validity start run number w.r.t. the current run,
367 // if the data is valid only for this run leave the default 0
368 // 6) specifies if the calibration data is valid for infinity (this means until updated),
369 // typical for calibration runs, the default is kFALSE
370 //
371 // returns 0 if fail, 1 otherwise
84090f85 372
9827400b 373 if (fTestMode & kErrorStorage)
374 {
375 Log(fCurrentDetector, "StoreLocally - In TESTMODE - Simulating error while storing locally");
376 return kFALSE;
377 }
378
3301427a 379 const char* cdbType = (localUri == fgkLocalCDB) ? "CDB" : "Reference";
2bb7b766 380
85a80aa9 381 Int_t firstRun = GetCurrentRun() - validityStart;
84090f85 382 if(firstRun < 0) {
9827400b 383 AliWarning("First valid run happens to be less than 0! Setting it to 0.");
84090f85 384 firstRun=0;
385 }
386
387 Int_t lastRun = -1;
388 if(validityInfinite) {
389 lastRun = AliCDBRunRange::Infinity();
390 } else {
391 lastRun = GetCurrentRun();
392 }
393
3301427a 394 // Version is set to current run, it will be used later to transfer data to Grid
395 AliCDBId id(path, firstRun, lastRun, GetCurrentRun(), -1);
2bb7b766 396
397 if(! dynamic_cast<TObjString*> (metaData->GetProperty("RunUsed(TObjString)"))){
398 TObjString runUsed = Form("%d", GetCurrentRun());
9e080f92 399 metaData->SetProperty("RunUsed(TObjString)", runUsed.Clone());
2bb7b766 400 }
84090f85 401
3301427a 402 Bool_t result = kFALSE;
84090f85 403
3301427a 404 if (!(AliCDBManager::Instance()->GetStorage(localUri))) {
405 Log("SHUTTLE", Form("StoreLocally - Cannot activate local %s storage", cdbType));
84090f85 406 } else {
3301427a 407 result = AliCDBManager::Instance()->GetStorage(localUri)
84090f85 408 ->Put(object, id, metaData);
409 }
410
411 if(!result) {
412
9827400b 413 Log(fCurrentDetector, Form("StoreLocally - Can't store object <%s>!", id.ToString().Data()));
3301427a 414 }
2bb7b766 415
3301427a 416 return result;
417}
84090f85 418
3301427a 419//______________________________________________________________________________________________
420Bool_t AliShuttle::StoreOCDB()
421{
9827400b 422 //
423 // Called when preprocessor ends successfully or when previous storage attempt failed (kStoreError status)
424 // Calls underlying StoreOCDB(const char*) function twice, for OCDB and Reference storage.
425 // Then calls StoreRefFilesToGrid to store reference files.
426 //
427
428 if (fTestMode & kErrorGrid)
429 {
430 Log("SHUTTLE", "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
431 Log(fCurrentDetector, "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
432 return kFALSE;
433 }
434
2d9019b4 435 Log("SHUTTLE","Storing OCDB data ...");
86aa42c3 436 Bool_t resultCDB = StoreOCDB(fgkMainCDB);
437
2d9019b4 438 Log("SHUTTLE","Storing reference data ...");
3301427a 439 Bool_t resultRef = StoreOCDB(fgkMainRefStorage);
9827400b 440
2d9019b4 441 Log("SHUTTLE","Storing reference files ...");
9827400b 442 Bool_t resultRefFiles = StoreRefFilesToGrid();
443
444 return resultCDB && resultRef && resultRefFiles;
3301427a 445}
446
447//______________________________________________________________________________________________
448Bool_t AliShuttle::StoreOCDB(const TString& gridURI)
449{
450 //
451 // Called by StoreOCDB(), performs actual storage to the main OCDB and reference storages (Grid)
452 //
453
454 TObjArray* gridIds=0;
455
456 Bool_t result = kTRUE;
457
458 const char* type = 0;
459 TString localURI;
460 if(gridURI == fgkMainCDB) {
461 type = "OCDB";
462 localURI = fgkLocalCDB;
463 } else if(gridURI == fgkMainRefStorage) {
464 type = "reference";
465 localURI = fgkLocalRefStorage;
466 } else {
467 AliError(Form("Invalid storage URI: %s", gridURI.Data()));
468 return kFALSE;
469 }
470
471 AliCDBManager* man = AliCDBManager::Instance();
472
473 AliCDBStorage *gridSto = man->GetStorage(gridURI);
474 if(!gridSto) {
475 Log("SHUTTLE",
476 Form("StoreOCDB - cannot activate main %s storage", type));
477 return kFALSE;
478 }
479
480 gridIds = gridSto->GetQueryCDBList();
481
482 // get objects previously stored in local CDB
483 AliCDBStorage *localSto = man->GetStorage(localURI);
484 if(!localSto) {
485 Log("SHUTTLE",
486 Form("StoreOCDB - cannot activate local %s storage", type));
487 return kFALSE;
488 }
489 AliCDBPath aPath(GetOfflineDetName(fCurrentDetector.Data()),"*","*");
490 // Local objects were stored with current run as Grid version!
491 TList* localEntries = localSto->GetAll(aPath.GetPath(), GetCurrentRun(), GetCurrentRun());
492 localEntries->SetOwner(1);
493
494 // loop on local stored objects
495 TIter localIter(localEntries);
496 AliCDBEntry *aLocEntry = 0;
497 while((aLocEntry = dynamic_cast<AliCDBEntry*> (localIter.Next()))){
498 aLocEntry->SetOwner(1);
499 AliCDBId aLocId = aLocEntry->GetId();
500 aLocEntry->SetVersion(-1);
501 aLocEntry->SetSubVersion(-1);
502
503 // If local object is valid up to infinity we store it only if it is
504 // the first unprocessed run!
505 if (aLocId.GetLastRun() == AliCDBRunRange::Infinity() &&
506 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
507 {
508 Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
509 "there are previous unprocessed runs!",
510 fCurrentDetector.Data(), aLocId.GetPath().Data()));
511 continue;
512 }
513
514 // loop on Grid valid Id's
515 Bool_t store = kTRUE;
516 TIter gridIter(gridIds);
517 AliCDBId* aGridId = 0;
518 while((aGridId = dynamic_cast<AliCDBId*> (gridIter.Next()))){
519 if(aGridId->GetPath() != aLocId.GetPath()) continue;
520 // skip all objects valid up to infinity
521 if(aGridId->GetLastRun() == AliCDBRunRange::Infinity()) continue;
522 // if we get here, it means there's already some more recent object stored on Grid!
523 store = kFALSE;
524 break;
525 }
526
527 // If we get here, the file can be stored!
528 Bool_t storeOk = gridSto->Put(aLocEntry);
529 if(!store || storeOk){
530
531 if (!store)
532 {
533 Log(fCurrentDetector.Data(),
534 Form("StoreOCDB - A more recent object already exists in %s storage: <%s>",
535 type, aGridId->ToString().Data()));
536 } else {
537 Log("SHUTTLE",
538 Form("StoreOCDB - Object <%s> successfully put into %s storage",
539 aLocId.ToString().Data(), type));
2d9019b4 540 Log(fCurrentDetector.Data(),
541 Form("StoreOCDB - Object <%s> successfully put into %s storage",
542 aLocId.ToString().Data(), type));
3301427a 543 }
84090f85 544
3301427a 545 // removing local filename...
546 TString filename;
547 localSto->IdToFilename(aLocId, filename);
548 AliInfo(Form("Removing local file %s", filename.Data()));
549 RemoveFile(filename.Data());
550 continue;
551 } else {
552 Log("SHUTTLE",
553 Form("StoreOCDB - Grid %s storage of object <%s> failed",
554 type, aLocId.ToString().Data()));
2d9019b4 555 Log(fCurrentDetector.Data(),
556 Form("StoreOCDB - Grid %s storage of object <%s> failed",
557 type, aLocId.ToString().Data()));
3301427a 558 result = kFALSE;
b948db8d 559 }
560 }
3301427a 561 localEntries->Clear();
2bb7b766 562
b948db8d 563 return result;
3301427a 564}
565
546242fb 566//______________________________________________________________________________________________
567Bool_t AliShuttle::CleanReferenceStorage(const char* detector)
568{
2d9019b4 569 // clears the directory used to store reference files of a given subdetector
546242fb 570
571 AliCDBManager* man = AliCDBManager::Instance();
572 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
2d9019b4 573 TString localBaseFolder = sto->GetBaseFolder();
574
575 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
576
577 Log("SHUTTLE", Form("Cleaning %s", targetDir.Data()));
578
579 TString begin;
580 begin.Form("%d_", GetCurrentRun());
581
582 TSystemDirectory* baseDir = new TSystemDirectory("/", targetDir);
583 if (!baseDir)
584 return kTRUE;
585
586 TList* dirList = baseDir->GetListOfFiles();
587 delete baseDir;
588
589 if (!dirList) return kTRUE;
590
591 if (dirList->GetEntries() < 3)
592 {
593 delete dirList;
594 return kTRUE;
595 }
596
597 Int_t nDirs = 0, nDel = 0;
598 TIter dirIter(dirList);
599 TSystemFile* entry = 0;
546242fb 600
2d9019b4 601 Bool_t success = kTRUE;
546242fb 602
2d9019b4 603 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
604 {
605 if (entry->IsDirectory())
606 continue;
607
608 TString fileName(entry->GetName());
609 if (!fileName.BeginsWith(begin))
610 continue;
611
612 nDirs++;
613
614 // delete file
615 Int_t result = gSystem->Unlink(fileName.Data());
616
617 if (result)
618 {
619 Log("SHUTTLE", Form("Could not delete file %s!", fileName.Data()));
620 success = kFALSE;
621 } else {
622 nDel++;
623 }
624 }
625
626 if(nDirs > 0)
627 Log("SHUTTLE", Form("CleanReferenceStorage - %d (over %d) reference files in folder %s were deleted.",
628 nDel, nDirs, targetDir.Data()));
629
630
631 delete dirList;
632 return success;
633
634
635
636
637
546242fb 638
639 Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
640 if (result == 0)
641 {
642 // delete directory
643 result = gSystem->Exec(Form("rm -r %s", targetDir.Data()));
644 if (result != 0)
645 {
646 Log("SHUTTLE", Form("StoreReferenceFile - Could not clear directory %s", targetDir.Data()));
647 return kFALSE;
648 }
649 }
650
651 result = gSystem->mkdir(targetDir, kTRUE);
652 if (result != 0)
653 {
654 Log("SHUTTLE", Form("StoreReferenceFile - Error creating base directory %s", targetDir.Data()));
655 return kFALSE;
656 }
657
658 return kTRUE;
659}
660
9827400b 661//______________________________________________________________________________________________
662Bool_t AliShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
663{
664 //
3c2a21c8 665 // Stores reference file directly (without opening it). This function stores the file locally.
9827400b 666 //
3c2a21c8 667 // The file is stored under the following location:
668 // <base folder of local reference storage>/<DET>/<RUN#>_<gridFileName>
669 // where <gridFileName> is the second parameter given to the function
670 //
9827400b 671
672 if (fTestMode & kErrorStorage)
673 {
674 Log(fCurrentDetector, "StoreReferenceFile - In TESTMODE - Simulating error while storing locally");
675 return kFALSE;
676 }
677
678 AliCDBManager* man = AliCDBManager::Instance();
679 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
680
681 TString localBaseFolder = sto->GetBaseFolder();
682
2d9019b4 683 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
9827400b 684
2d9019b4 685 //try to open folder, if does not exist
686 void* dir = gSystem->OpenDirectory(targetDir.Data());
687 if (dir == NULL) {
688 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
689 Log("SHUTTLE", Form("Can't open directory <%s>", targetDir.Data()));
690 return kFALSE;
691 }
692
693 } else {
694 gSystem->FreeDirectory(dir);
695 }
696
9827400b 697 TString target;
698 target.Form("%s/%d_%s", targetDir.Data(), GetCurrentRun(), gridFileName);
699
546242fb 700 Int_t result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
9827400b 701 if (result)
702 {
546242fb 703 Log("SHUTTLE", Form("StoreReferenceFile - %s does not exist", localFile));
704 return kFALSE;
9827400b 705 }
546242fb 706
9827400b 707 result = gSystem->CopyFile(localFile, target);
708
709 if (result == 0)
710 {
2d9019b4 711 Log("SHUTTLE", Form("StoreReferenceFile - File %s stored locally to %s", localFile, target.Data()));
9827400b 712 return kTRUE;
713 }
714 else
715 {
2d9019b4 716 Log("SHUTTLE", Form("StoreReferenceFile - Could not store file %s to %s!. Error code = %d",
546242fb 717 localFile, target.Data(), result));
9827400b 718 return kFALSE;
719 }
720}
721
722//______________________________________________________________________________________________
723Bool_t AliShuttle::StoreRefFilesToGrid()
724{
725 //
726 // Transfers the reference file to the Grid.
9827400b 727 //
86aa42c3 728 // The files are stored under the following location:
3c2a21c8 729 // <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
86aa42c3 730 //
9827400b 731
732 AliCDBManager* man = AliCDBManager::Instance();
733 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
734 if (!sto)
735 return kFALSE;
736 TString localBaseFolder = sto->GetBaseFolder();
737
2d9019b4 738 TString dir = GetRefFilePrefix(localBaseFolder.Data(), fCurrentDetector.Data());
739
9827400b 740 AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
741 if (!gridSto)
742 return kFALSE;
2d9019b4 743
9827400b 744 TString gridBaseFolder = gridSto->GetBaseFolder();
2d9019b4 745
746 TString alienDir = GetRefFilePrefix(gridBaseFolder.Data(), fCurrentDetector.Data());
9827400b 747
9827400b 748 TString begin;
749 begin.Form("%d_", GetCurrentRun());
750
751 TSystemDirectory* baseDir = new TSystemDirectory("/", dir);
3d8bc902 752 if (!baseDir)
753 return kTRUE;
754
2d9019b4 755 TList* dirList = baseDir->GetListOfFiles();
756 delete baseDir;
757
758 if (!dirList) return kTRUE;
759
760 if (dirList->GetEntries() < 3)
3d8bc902 761 {
2d9019b4 762 delete dirList;
9827400b 763 return kTRUE;
3d8bc902 764 }
2d9019b4 765
546242fb 766 if (!gGrid)
767 {
768 Log("SHUTTLE", "Connection to Grid failed: Cannot continue!");
2d9019b4 769 delete dirList;
546242fb 770 return kFALSE;
771 }
772
2d9019b4 773 Int_t nDirs = 0, nTransfer = 0;
774 TIter dirIter(dirList);
775 TSystemFile* entry = 0;
776
9827400b 777 Bool_t success = kTRUE;
3d8bc902 778 Bool_t first = kTRUE;
9827400b 779
2d9019b4 780 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
781 {
9827400b 782 if (entry->IsDirectory())
783 continue;
784
785 TString fileName(entry->GetName());
786 if (!fileName.BeginsWith(begin))
787 continue;
788
2d9019b4 789 nDirs++;
790
3d8bc902 791 if (first)
792 {
793 first = kFALSE;
794 // check that DET folder exists, otherwise create it
795 TGridResult* result = gGrid->Ls(alienDir.Data(), "a");
796
797 if (!result)
2d9019b4 798 {
799 delete dirList;
3d8bc902 800 return kFALSE;
2d9019b4 801 }
3d8bc902 802
546242fb 803 if (!result->GetFileName(1)) // TODO: It looks like element 0 is always 0!!
3d8bc902 804 {
805 if (!gGrid->Mkdir(alienDir.Data(),"",0))
806 {
807 Log("SHUTTLE", Form("StoreRefFilesToGrid - Cannot create directory %s",
808 alienDir.Data()));
2d9019b4 809 delete dirList;
3d8bc902 810 return kFALSE;
546242fb 811 } else {
812 Log("SHUTTLE",Form("Folder %s created", alienDir.Data()));
3d8bc902 813 }
814
546242fb 815 } else {
816 Log("SHUTTLE",Form("Folder %s found", alienDir.Data()));
3d8bc902 817 }
818 }
819
9827400b 820 TString fullLocalPath;
821 fullLocalPath.Form("%s/%s", dir.Data(), fileName.Data());
822
823 TString fullGridPath;
824 fullGridPath.Form("alien://%s/%s", alienDir.Data(), fileName.Data());
825
9827400b 826 TFileMerger fileMerger;
827 Bool_t result = fileMerger.Cp(fullLocalPath, fullGridPath);
828
829 if (result)
830 {
2d9019b4 831 Log("SHUTTLE", Form("StoreRefFilesToGrid - Copying local file %s to %s succeeded!", fullLocalPath.Data(), fullGridPath.Data()));
9827400b 832 RemoveFile(fullLocalPath);
2d9019b4 833 nTransfer++;
9827400b 834 }
835 else
836 {
2d9019b4 837 Log("SHUTTLE", Form("StoreRefFilesToGrid - Copying local file %s to %s FAILED!", fullLocalPath.Data(), fullGridPath.Data()));
9827400b 838 success = kFALSE;
839 }
840 }
2d9019b4 841
842 Log("SHUTTLE", Form("StoreRefFilesToGrid - %d (over %d) reference files in folder %s copied to Grid.", nTransfer, nDirs, dir.Data()));
843
844
845 delete dirList;
9827400b 846 return success;
847}
848
2d9019b4 849//______________________________________________________________________________________________
850const char* AliShuttle::GetRefFilePrefix(const char* base, const char* detector)
851{
852 //
853 // Get folder name of reference files
854 //
855
856 TString offDetStr(GetOfflineDetName(detector));
857 TString dir;
858 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
859 {
860 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
861 } else {
862 dir.Form("%s/%s", base, offDetStr.Data());
863 }
864
865 return dir.Data();
866
867
868}
3301427a 869//______________________________________________________________________________________________
870void AliShuttle::CleanLocalStorage(const TString& uri)
871{
9827400b 872 //
873 // Called in case the preprocessor is declared failed. Remove remaining objects from the local storages.
874 //
3301427a 875
876 const char* type = 0;
877 if(uri == fgkLocalCDB) {
878 type = "OCDB";
879 } else if(uri == fgkLocalRefStorage) {
546242fb 880 type = "Reference";
3301427a 881 } else {
882 AliError(Form("Invalid storage URI: %s", uri.Data()));
883 return;
884 }
885
886 AliCDBManager* man = AliCDBManager::Instance();
b948db8d 887
3301427a 888 // open local storage
889 AliCDBStorage *localSto = man->GetStorage(uri);
890 if(!localSto) {
891 Log("SHUTTLE",
892 Form("CleanLocalStorage - cannot activate local %s storage", type));
893 return;
894 }
895
896 TString filename(Form("%s/%s/*/Run*_v%d_s*.root",
546242fb 897 localSto->GetBaseFolder().Data(), GetOfflineDetName(fCurrentDetector.Data()), GetCurrentRun()));
3301427a 898
899 AliInfo(Form("filename = %s", filename.Data()));
900
901 AliInfo(Form("Removing remaining local files from run %d and detector %s ...",
902 GetCurrentRun(), fCurrentDetector.Data()));
903
904 RemoveFile(filename.Data());
905
906}
907
908//______________________________________________________________________________________________
909void AliShuttle::RemoveFile(const char* filename)
910{
9827400b 911 //
912 // removes local file
913 //
3301427a 914
915 TString command(Form("rm -f %s", filename));
916
917 Int_t result = gSystem->Exec(command.Data());
918 if(result != 0)
919 {
920 Log("SHUTTLE", Form("RemoveFile - %s: Cannot remove file %s!",
921 fCurrentDetector.Data(), filename));
922 }
73abe331 923}
924
b948db8d 925//______________________________________________________________________________________________
5164a766 926AliShuttleStatus* AliShuttle::ReadShuttleStatus()
927{
9827400b 928 //
929 // Reads the AliShuttleStatus from the CDB
930 //
5164a766 931
2bb7b766 932 if (fStatusEntry){
933 delete fStatusEntry;
934 fStatusEntry = 0;
935 }
5164a766 936
10a5a932 937 fStatusEntry = AliCDBManager::Instance()->GetStorage(GetLocalCDB())
2bb7b766 938 ->Get(Form("/SHUTTLE/STATUS/%s", fCurrentDetector.Data()), GetCurrentRun());
5164a766 939
2bb7b766 940 if (!fStatusEntry) return 0;
941 fStatusEntry->SetOwner(1);
5164a766 942
2bb7b766 943 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
944 if (!status) {
945 AliError("Invalid object stored to CDB!");
946 return 0;
947 }
5164a766 948
2bb7b766 949 return status;
5164a766 950}
951
952//______________________________________________________________________________________________
7bfb2090 953Bool_t AliShuttle::WriteShuttleStatus(AliShuttleStatus* status)
5164a766 954{
9827400b 955 //
956 // writes the status for one subdetector
957 //
2bb7b766 958
959 if (fStatusEntry){
960 delete fStatusEntry;
961 fStatusEntry = 0;
962 }
5164a766 963
2bb7b766 964 Int_t run = GetCurrentRun();
5164a766 965
2bb7b766 966 AliCDBId id(AliCDBPath("SHUTTLE", "STATUS", fCurrentDetector), run, run);
5164a766 967
2bb7b766 968 fStatusEntry = new AliCDBEntry(status, id, new AliCDBMetaData);
969 fStatusEntry->SetOwner(1);
5164a766 970
2bb7b766 971 UInt_t result = AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
7bfb2090 972
2bb7b766 973 if (!result) {
3301427a 974 Log("SHUTTLE", Form("WriteShuttleStatus - Failed for %s, run %d",
975 fCurrentDetector.Data(), run));
2bb7b766 976 return kFALSE;
977 }
e7f62f16 978
979 SendMLInfo();
7bfb2090 980
2bb7b766 981 return kTRUE;
5164a766 982}
983
984//______________________________________________________________________________________________
985void AliShuttle::UpdateShuttleStatus(AliShuttleStatus::Status newStatus, Bool_t increaseCount)
986{
9827400b 987 //
988 // changes the AliShuttleStatus for the given detector and run to the given status
989 //
5164a766 990
2bb7b766 991 if (!fStatusEntry){
992 AliError("UNEXPECTED: fStatusEntry empty");
993 return;
994 }
5164a766 995
2bb7b766 996 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
5164a766 997
2bb7b766 998 if (!status){
3301427a 999 Log("SHUTTLE", "UNEXPECTED: status could not be read from current CDB entry");
2bb7b766 1000 return;
1001 }
5164a766 1002
2c15234c 1003 TString actionStr = Form("UpdateShuttleStatus - %s: Changing state from %s to %s",
eba76848 1004 fCurrentDetector.Data(),
36c99a6a 1005 status->GetStatusName(),
eba76848 1006 status->GetStatusName(newStatus));
cb343cfd 1007 Log("SHUTTLE", actionStr);
1008 SetLastAction(actionStr);
5164a766 1009
2bb7b766 1010 status->SetStatus(newStatus);
1011 if (increaseCount) status->IncreaseCount();
5164a766 1012
2bb7b766 1013 AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
e7f62f16 1014
1015 SendMLInfo();
5164a766 1016}
e7f62f16 1017
1018//______________________________________________________________________________________________
1019void AliShuttle::SendMLInfo()
1020{
1021 //
1022 // sends ML information about the current status of the current detector being processed
1023 //
1024
1025 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1026
1027 if (!status){
3301427a 1028 Log("SHUTTLE", "SendMLInfo - UNEXPECTED: status could not be read from current CDB entry");
e7f62f16 1029 return;
1030 }
1031
1032 TMonaLisaText mlStatus(Form("%s_status", fCurrentDetector.Data()), status->GetStatusName());
1033 TMonaLisaValue mlRetryCount(Form("%s_count", fCurrentDetector.Data()), status->GetCount());
1034
1035 TList mlList;
1036 mlList.Add(&mlStatus);
1037 mlList.Add(&mlRetryCount);
1038
1039 fMonaLisa->SendParameters(&mlList);
1040}
1041
5164a766 1042//______________________________________________________________________________________________
1043Bool_t AliShuttle::ContinueProcessing()
1044{
9827400b 1045 // this function reads the AliShuttleStatus information from CDB and
1046 // checks if the processing should be continued
1047 // if yes it returns kTRUE and updates the AliShuttleStatus with nextStatus
2bb7b766 1048
57c1a579 1049 if (!fConfig->HostProcessDetector(fCurrentDetector)) return kFALSE;
1050
1051 AliPreprocessor* aPreprocessor =
1052 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1053 if (!aPreprocessor)
1054 {
1055 AliInfo(Form("%s: no preprocessor registered", fCurrentDetector.Data()));
1056 return kFALSE;
1057 }
1058
2bb7b766 1059 AliShuttleLogbookEntry::Status entryStatus =
eba76848 1060 fLogbookEntry->GetDetectorStatus(fCurrentDetector);
2bb7b766 1061
1062 if(entryStatus != AliShuttleLogbookEntry::kUnprocessed) {
9e080f92 1063 AliInfo(Form("ContinueProcessing - %s is %s",
2bb7b766 1064 fCurrentDetector.Data(),
1065 fLogbookEntry->GetDetectorStatusName(entryStatus)));
1066 return kFALSE;
1067 }
1068
1069 // if we get here, according to Shuttle logbook subdetector is in UNPROCESSED state
be48e3ea 1070
1071 // check if current run is first unprocessed run for current detector
1072 if (fConfig->StrictRunOrder(fCurrentDetector) &&
1073 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
1074 {
86aa42c3 1075 if (fTestMode == kNone)
1076 {
1077 Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering but this is not the first unprocessed run!"));
1078 return kFALSE;
1079 }
1080 else
1081 {
1082 Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - Although %s requires strict run ordering and this is not the first unprocessed run, the SHUTTLE continues"));
1083 }
be48e3ea 1084 }
1085
2bb7b766 1086 AliShuttleStatus* status = ReadShuttleStatus();
1087 if (!status) {
1088 // first time
1089 Log("SHUTTLE", Form("ContinueProcessing - %s: Processing first time",
1090 fCurrentDetector.Data()));
1091 status = new AliShuttleStatus(AliShuttleStatus::kStarted);
1092 return WriteShuttleStatus(status);
1093 }
1094
1095 // The following two cases shouldn't happen if Shuttle Logbook was correctly updated.
1096 // If it happens it may mean Logbook updating failed... let's do it now!
1097 if (status->GetStatus() == AliShuttleStatus::kDone ||
1098 status->GetStatus() == AliShuttleStatus::kFailed){
1099 Log("SHUTTLE", Form("ContinueProcessing - %s is already %s. Updating Shuttle Logbook",
1100 fCurrentDetector.Data(),
1101 status->GetStatusName(status->GetStatus())));
1102 UpdateShuttleLogbook(fCurrentDetector.Data(),
1103 status->GetStatusName(status->GetStatus()));
1104 return kFALSE;
1105 }
1106
3301427a 1107 if (status->GetStatus() == AliShuttleStatus::kStoreError) {
2bb7b766 1108 Log("SHUTTLE",
1109 Form("ContinueProcessing - %s: Grid storage of one or more objects failed. Trying again now",
1110 fCurrentDetector.Data()));
9827400b 1111 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1112 if (StoreOCDB()){
3301427a 1113 Log("SHUTTLE", Form("ContinueProcessing - %s: all objects successfully stored into main storage",
1114 fCurrentDetector.Data()));
2bb7b766 1115 UpdateShuttleStatus(AliShuttleStatus::kDone);
1116 UpdateShuttleLogbook(fCurrentDetector.Data(), "DONE");
1117 } else {
1118 Log("SHUTTLE",
1119 Form("ContinueProcessing - %s: Grid storage failed again",
1120 fCurrentDetector.Data()));
9827400b 1121 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
2bb7b766 1122 }
1123 return kFALSE;
1124 }
1125
1126 // if we get here, there is a restart
57c1a579 1127 Bool_t cont = kFALSE;
2bb7b766 1128
1129 // abort conditions
cb343cfd 1130 if (status->GetCount() >= fConfig->GetMaxRetries()) {
57c1a579 1131 Log("SHUTTLE", Form("ContinueProcessing - %s failed %d times in status %s - "
1132 "Updating Shuttle Logbook", fCurrentDetector.Data(),
2bb7b766 1133 status->GetCount(), status->GetStatusName()));
1134 UpdateShuttleLogbook(fCurrentDetector.Data(), "FAILED");
e7f62f16 1135 UpdateShuttleStatus(AliShuttleStatus::kFailed);
3301427a 1136
1137 // there may still be objects in local OCDB and reference storage
1138 // and FXS databases may be not updated: do it now!
9827400b 1139
1140 // TODO Currently disabled, we want to keep files in case of failure!
1141 // CleanLocalStorage(fgkLocalCDB);
1142 // CleanLocalStorage(fgkLocalRefStorage);
1143 // UpdateTableFailCase();
1144
1145 // Send mail to detector expert!
1146 AliInfo(Form("Sending mail to %s expert...", fCurrentDetector.Data()));
1147 if (!SendMail())
1148 Log("SHUTTLE", Form("ContinueProcessing - Could not send mail to %s expert",
1149 fCurrentDetector.Data()));
3301427a 1150
57c1a579 1151 } else {
1152 Log("SHUTTLE", Form("ContinueProcessing - %s: restarting. "
1153 "Aborted before with %s. Retry number %d.", fCurrentDetector.Data(),
1154 status->GetStatusName(), status->GetCount()));
9827400b 1155 Bool_t increaseCount = kTRUE;
1156 if (status->GetStatus() == AliShuttleStatus::kDCSError || status->GetStatus() == AliShuttleStatus::kDCSStarted)
1157 increaseCount = kFALSE;
1158 UpdateShuttleStatus(AliShuttleStatus::kStarted, increaseCount);
57c1a579 1159 cont = kTRUE;
2bb7b766 1160 }
1161
57c1a579 1162 return cont;
5164a766 1163}
1164
1165//______________________________________________________________________________________________
2bb7b766 1166Bool_t AliShuttle::Process(AliShuttleLogbookEntry* entry)
58bc3020 1167{
73abe331 1168 //
b948db8d 1169 // Makes data retrieval for all detectors in the configuration.
2bb7b766 1170 // entry: Shuttle logbook entry, contains run paramenters and status of detectors
1171 // (Unprocessed, Inactive, Failed or Done).
d477ad88 1172 // Returns kFALSE in case of error occured and kTRUE otherwise
73abe331 1173 //
1174
9827400b 1175 if (!entry) return kFALSE;
2bb7b766 1176
1177 fLogbookEntry = entry;
1178
9827400b 1179 AliInfo(Form("\n\n \t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^* \n",
1180 GetCurrentRun()));
2bb7b766 1181
e7f62f16 1182 // create ML instance that monitors this run
1183 fMonaLisa = new TMonaLisaWriter(Form("%d", GetCurrentRun()), "SHUTTLE", "aliendb1.cern.ch");
1184 // disable monitoring of other parameters that come e.g. from TFile
1185 gMonitoringWriter = 0;
2bb7b766 1186
e7f62f16 1187 // Send the information to ML
1188 TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
9827400b 1189 TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
e7f62f16 1190
1191 TList mlList;
1192 mlList.Add(&mlStatus);
9827400b 1193 mlList.Add(&mlRunType);
e7f62f16 1194
1195 fMonaLisa->SendParameters(&mlList);
3301427a 1196
9827400b 1197 if (fLogbookEntry->IsDone())
1198 {
1199 Log("SHUTTLE","Process - Shuttle is already DONE. Updating logbook");
1200 UpdateShuttleLogbook("shuttle_done");
1201 fLogbookEntry = 0;
1202 return kTRUE;
1203 }
1204
1205 // read test mode if flag is set
1206 if (fReadTestMode)
1207 {
3d8bc902 1208 fTestMode = kNone;
9827400b 1209 TString logEntry(entry->GetRunParameter("log"));
1210 //printf("log entry = %s\n", logEntry.Data());
1211 TString searchStr("Testmode: ");
1212 Int_t pos = logEntry.Index(searchStr.Data());
1213 //printf("%d\n", pos);
1214 if (pos >= 0)
1215 {
1216 TSubString subStr = logEntry(pos + searchStr.Length(), logEntry.Length());
1217 //printf("%s\n", subStr.String().Data());
1218 TString newStr(subStr.Data());
1219 TObjArray* token = newStr.Tokenize(' ');
1220 if (token)
1221 {
1222 //token->Print();
1223 TObjString* tmpStr = dynamic_cast<TObjString*> (token->First());
1224 if (tmpStr)
1225 {
1226 Int_t testMode = tmpStr->String().Atoi();
1227 if (testMode > 0)
1228 {
1229 Log("SHUTTLE", Form("Enabling test mode %d", testMode));
1230 SetTestMode((TestMode) testMode);
1231 }
1232 }
1233 delete token;
1234 }
1235 }
1236 }
1237
3d8bc902 1238 Log("SHUTTLE", Form("The test mode flag is %d", (Int_t) fTestMode));
1239
eba76848 1240 fLogbookEntry->Print("all");
57f50b3c 1241
1242 // Initialization
d477ad88 1243 Bool_t hasError = kFALSE;
5164a766 1244
2bb7b766 1245 AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
1246 if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
1247 AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
1248 if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
d477ad88 1249
57f50b3c 1250 // Loop on detectors in the configuration
b948db8d 1251 TIter iter(fConfig->GetDetectors());
2bb7b766 1252 TObjString* aDetector = 0;
b948db8d 1253
be48e3ea 1254 while ((aDetector = (TObjString*) iter.Next()))
1255 {
7bfb2090 1256 fCurrentDetector = aDetector->String();
5164a766 1257
9e080f92 1258 if (ContinueProcessing() == kFALSE) continue;
1259
2bb7b766 1260 AliInfo(Form("\n\n \t\t\t****** run %d - %s: START ******",
1261 GetCurrentRun(), aDetector->GetName()));
1262
9d733021 1263 for(Int_t iSys=0;iSys<3;iSys++) fFXSCalled[iSys]=kFALSE;
1264
e7f62f16 1265 Log(fCurrentDetector.Data(), "Starting processing");
85a80aa9 1266
be48e3ea 1267 Int_t pid = fork();
1268
1269 if (pid < 0)
1270 {
1271 Log("SHUTTLE", "ERROR: Forking failed");
1272 }
1273 else if (pid > 0)
1274 {
1275 // parent
1276 AliInfo(Form("In parent process of %d - %s: Starting monitoring",
1277 GetCurrentRun(), aDetector->GetName()));
1278
1279 Long_t begin = time(0);
1280
1281 int status; // to be used with waitpid, on purpose an int (not Int_t)!
1282 while (waitpid(pid, &status, WNOHANG) == 0)
1283 {
1284 Long_t expiredTime = time(0) - begin;
1285
1286 if (expiredTime > fConfig->GetPPTimeOut())
1287 {
9827400b 1288 TString tmp;
1289 tmp.Form("Process of %s time out. Run time: %d seconds. Killing...",
1290 fCurrentDetector.Data(), expiredTime);
1291 Log("SHUTTLE", tmp);
1292 Log(fCurrentDetector, tmp);
be48e3ea 1293
1294 kill(pid, 9);
1295
3301427a 1296 UpdateShuttleStatus(AliShuttleStatus::kPPTimeOut);
be48e3ea 1297 hasError = kTRUE;
1298
1299 gSystem->Sleep(1000);
1300 }
1301 else
1302 {
be48e3ea 1303 gSystem->Sleep(1000);
9827400b 1304
1305 TString checkStr;
1306 checkStr.Form("ps -o vsize --pid %d | tail -n 1", pid);
1307 FILE* pipe = gSystem->OpenPipe(checkStr, "r");
1308 if (!pipe)
1309 {
1310 Log("SHUTTLE", Form("Error: Could not open pipe to %s", checkStr.Data()));
1311 continue;
1312 }
1313
1314 char buffer[100];
1315 if (!fgets(buffer, 100, pipe))
1316 {
1317 Log("SHUTTLE", "Error: ps did not return anything");
1318 gSystem->ClosePipe(pipe);
1319 continue;
1320 }
1321 gSystem->ClosePipe(pipe);
1322
1323 //Log("SHUTTLE", Form("ps returned %s", buffer));
1324
1325 Int_t mem = 0;
1326 if ((sscanf(buffer, "%d\n", &mem) != 1) || !mem)
1327 {
1328 Log("SHUTTLE", "Error: Could not parse output of ps");
1329 continue;
1330 }
1331
1332 if (expiredTime % 60 == 0)
886d60e6 1333 Log("SHUTTLE", Form("%s: Checking process. Run time: %d seconds - Memory consumption: %d KB",
1334 fCurrentDetector.Data(), expiredTime, mem));
9827400b 1335
1336 if (mem > fConfig->GetPPMaxMem())
1337 {
1338 TString tmp;
1339 tmp.Form("Process exceeds maximum allowed memory (%d KB > %d KB). Killing...",
1340 mem, fConfig->GetPPMaxMem());
1341 Log("SHUTTLE", tmp);
1342 Log(fCurrentDetector, tmp);
1343
1344 kill(pid, 9);
1345
1346 UpdateShuttleStatus(AliShuttleStatus::kPPOutOfMemory);
1347 hasError = kTRUE;
1348
1349 gSystem->Sleep(1000);
1350 }
be48e3ea 1351 }
1352 }
1353
1354 AliInfo(Form("In parent process of %d - %s: Client has terminated.",
1355 GetCurrentRun(), aDetector->GetName()));
1356
1357 if (WIFEXITED(status))
1358 {
1359 Int_t returnCode = WEXITSTATUS(status);
1360
3301427a 1361 Log("SHUTTLE", Form("%s: the return code is %d", fCurrentDetector.Data(),
1362 returnCode));
be48e3ea 1363
9827400b 1364 if (returnCode == 0) hasError = kTRUE;
be48e3ea 1365 }
1366 }
1367 else if (pid == 0)
1368 {
1369 // client
1370 AliInfo(Form("In client process of %d - %s", GetCurrentRun(), aDetector->GetName()));
1371
ffa29e93 1372 AliInfo("Redirecting output...");
1373
546242fb 1374 if ((freopen(GetLogFileName(fCurrentDetector), "a", stdout)) == 0)
ffa29e93 1375 {
1376 Log("SHUTTLE", "Could not freopen stdout");
1377 }
1378 else
1379 {
1380 fOutputRedirected = kTRUE;
1381 if ((dup2(fileno(stdout), fileno(stderr))) < 0)
1382 Log("SHUTTLE", "Could not redirect stderr");
1383
1384 }
1385
9827400b 1386 Bool_t success = ProcessCurrentDetector();
1387 if (success) // Preprocessor finished successfully!
1388 {
3301427a 1389 // Update time_processed field in FXS DB
1390 if (UpdateTable() == kFALSE)
1391 Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!"));
1392
1393 // Transfer the data from local storage to main storage (Grid)
1394 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1395 if (StoreOCDB() == kFALSE)
1396 {
1397 AliInfo(Form("\n \t\t\t****** run %d - %s: STORAGE ERROR ****** \n\n",
1398 GetCurrentRun(), aDetector->GetName()));
1399 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
9827400b 1400 success = kFALSE;
3301427a 1401 } else {
1402 AliInfo(Form("\n \t\t\t****** run %d - %s: DONE ****** \n\n",
1403 GetCurrentRun(), aDetector->GetName()));
1404 UpdateShuttleStatus(AliShuttleStatus::kDone);
9827400b 1405 UpdateShuttleLogbook(fCurrentDetector, "DONE");
3301427a 1406 }
be48e3ea 1407 }
1408
4b95672b 1409 for (UInt_t iSys=0; iSys<3; iSys++)
1410 {
1411 if (fFXSCalled[iSys]) fFXSlist[iSys].Clear();
1412 }
1413
be48e3ea 1414 AliInfo(Form("Client process of %d - %s is exiting now with %d.",
9827400b 1415 GetCurrentRun(), aDetector->GetName(), success));
be48e3ea 1416
1417 // the client exits here
9827400b 1418 gSystem->Exit(success);
be48e3ea 1419
1420 AliError("We should never get here!!!");
1421 }
7bfb2090 1422 }
5164a766 1423
2bb7b766 1424 AliInfo(Form("\n\n \t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^* \n",
1425 GetCurrentRun()));
1426
1427 //check if shuttle is done for this run, if so update logbook
1428 TObjArray checkEntryArray;
1429 checkEntryArray.SetOwner(1);
9e080f92 1430 TString whereClause = Form("where run=%d", GetCurrentRun());
1431 if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) || checkEntryArray.GetEntries() == 0) {
1432 Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
1433 GetCurrentRun()));
1434 return hasError == kFALSE;
1435 }
b948db8d 1436
9e080f92 1437 AliShuttleLogbookEntry* checkEntry = dynamic_cast<AliShuttleLogbookEntry*>
1438 (checkEntryArray.At(0));
2bb7b766 1439
9e080f92 1440 if (checkEntry)
1441 {
1442 if (checkEntry->IsDone())
be48e3ea 1443 {
9e080f92 1444 Log("SHUTTLE","Process - Shuttle is DONE. Updating logbook");
1445 UpdateShuttleLogbook("shuttle_done");
1446 }
1447 else
1448 {
1449 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
be48e3ea 1450 {
9e080f92 1451 if (checkEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
be48e3ea 1452 {
9e080f92 1453 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
1454 checkEntry->GetRun(), GetDetName(iDet)));
1455 fFirstUnprocessed[iDet] = kFALSE;
be48e3ea 1456 }
1457 }
2bb7b766 1458 }
1459 }
1460
e7f62f16 1461 // remove ML instance
1462 delete fMonaLisa;
1463 fMonaLisa = 0;
1464
2bb7b766 1465 fLogbookEntry = 0;
85a80aa9 1466
a7160fe9 1467 return hasError == kFALSE;
73abe331 1468}
1469
b948db8d 1470//______________________________________________________________________________________________
9827400b 1471Bool_t AliShuttle::ProcessCurrentDetector()
73abe331 1472{
1473 //
2bb7b766 1474 // Makes data retrieval just for a specific detector (fCurrentDetector).
73abe331 1475 // Threre should be a configuration for this detector.
73abe331 1476
2bb7b766 1477 AliInfo(Form("Retrieving values for %s, run %d", fCurrentDetector.Data(), GetCurrentRun()));
73abe331 1478
2d9019b4 1479 if (!CleanReferenceStorage(fCurrentDetector.Data()))
546242fb 1480 return kFALSE;
1481
2c15234c 1482 TMap dcsMap;
1483 dcsMap.SetOwner(1);
73abe331 1484
85a80aa9 1485 Bool_t aDCSError = kFALSE;
3301427a 1486
1487 // call preprocessor
1488 AliPreprocessor* aPreprocessor =
1489 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1490
1491 aPreprocessor->Initialize(GetCurrentRun(), GetCurrentStartTime(), GetCurrentEndTime());
1492
1493 Bool_t processDCS = aPreprocessor->ProcessDCS();
d477ad88 1494
651fdaab 1495 if (!processDCS)
1496 {
1497 Log(fCurrentDetector, "The preprocessor requested to skip the retrieval of DCS values");
1498 }
8b739301 1499 else if (fTestMode & kSkipDCS)
2c15234c 1500 {
3d8bc902 1501 Log(fCurrentDetector, "In TESTMODE - Skipping DCS processing!");
9827400b 1502 }
1503 else if (fTestMode & kErrorDCS)
1504 {
3d8bc902 1505 Log(fCurrentDetector, "In TESTMODE - Simulating DCS error");
1506 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
9827400b 1507 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1508 return kFALSE;
2c15234c 1509 } else {
3301427a 1510
1511 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
1512
2c15234c 1513 TString host(fConfig->GetDCSHost(fCurrentDetector));
1514 Int_t port = fConfig->GetDCSPort(fCurrentDetector);
1515
1516 // Retrieval of Aliases
1517 TObjString* anAlias = 0;
546242fb 1518 Int_t iAlias = 0;
36c99a6a 1519 Int_t nTotAliases= ((TMap*)fConfig->GetDCSAliases(fCurrentDetector))->GetEntries();
2c15234c 1520 TIter iterAliases(fConfig->GetDCSAliases(fCurrentDetector));
1521 while ((anAlias = (TObjString*) iterAliases.Next()))
1522 {
1523 TObjArray *valueSet = new TObjArray();
1524 valueSet->SetOwner(1);
1525
546242fb 1526 iAlias++;
2c15234c 1527 aDCSError = (GetValueSet(host, port, anAlias->String(), valueSet, kAlias) == 0);
1528
1529 if(!aDCSError)
1530 {
2d9019b4 1531 if (((iAlias-1) % 500) == 0 || iAlias == nTotAliases)
1532 AliInfo(Form("Alias %s (%d of %d) - %d values collected",
1533 anAlias->GetName(), iAlias, nTotAliases, valueSet->GetEntriesFast()));
2c15234c 1534 dcsMap.Add(anAlias->Clone(), valueSet);
1535 } else {
1536 Log(fCurrentDetector,
1537 Form("ProcessCurrentDetector - Error while retrieving alias %s",
1538 anAlias->GetName()));
1539 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1540 dcsMap.DeleteAll();
9827400b 1541 return kFALSE;
2c15234c 1542 }
4f0ab988 1543 }
2c15234c 1544
1545 // Retrieval of Data Points
1546 TObjString* aDP = 0;
36c99a6a 1547 Int_t iDP = 0;
1548 Int_t nTotDPs= ((TMap*)fConfig->GetDCSDataPoints(fCurrentDetector))->GetEntries();
2c15234c 1549 TIter iterDP(fConfig->GetDCSDataPoints(fCurrentDetector));
1550 while ((aDP = (TObjString*) iterDP.Next()))
1551 {
1552 TObjArray *valueSet = new TObjArray();
1553 valueSet->SetOwner(1);
36c99a6a 1554 if (((iDP-1) % 500) == 0 || iDP == nTotDPs)
1555 AliInfo(Form("Querying DCS archive: DP %s (%d of %d)",
1556 aDP->GetName(), iDP++, nTotDPs));
2c15234c 1557 aDCSError = (GetValueSet(host, port, aDP->String(), valueSet, kDP) == 0);
1558
1559 if(!aDCSError)
1560 {
1561 dcsMap.Add(aDP->Clone(), valueSet);
1562 } else {
1563 Log(fCurrentDetector,
1564 Form("ProcessCurrentDetector - Error while retrieving data point %s",
1565 aDP->GetName()));
1566 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1567 dcsMap.DeleteAll();
9827400b 1568 return kFALSE;
2c15234c 1569 }
73abe331 1570 }
1571 }
b948db8d 1572
2bb7b766 1573 // DCS Archive DB processing successful. Call Preprocessor!
85a80aa9 1574 UpdateShuttleStatus(AliShuttleStatus::kPPStarted);
a7160fe9 1575
3301427a 1576 UInt_t returnValue = aPreprocessor->Process(&dcsMap);
b948db8d 1577
3301427a 1578 if (returnValue > 0) // Preprocessor error!
1579 {
9827400b 1580 Log(fCurrentDetector, Form("Preprocessor failed. Process returned %d.", returnValue));
cb343cfd 1581 UpdateShuttleStatus(AliShuttleStatus::kPPError);
9827400b 1582 dcsMap.DeleteAll();
1583 return kFALSE;
1584 }
1585
1586 // preprocessor ok!
1587 UpdateShuttleStatus(AliShuttleStatus::kPPDone);
1588 Log(fCurrentDetector, Form("ProcessCurrentDetector - %s preprocessor returned success",
1589 fCurrentDetector.Data()));
b948db8d 1590
2c15234c 1591 dcsMap.DeleteAll();
b948db8d 1592
9827400b 1593 return kTRUE;
2bb7b766 1594}
1595
1596//______________________________________________________________________________________________
1597Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
1598 TObjArray& entries)
1599{
9827400b 1600 // Query DAQ's Shuttle logbook and fills detector status object.
1601 // Call QueryRunParameters to query DAQ logbook for run parameters.
1602 //
2bb7b766 1603
fc5a4708 1604 entries.SetOwner(1);
1605
2bb7b766 1606 // check connection, in case connect
be48e3ea 1607 if(!Connect(3)) return kFALSE;
2bb7b766 1608
1609 TString sqlQuery;
441b0e9c 1610 sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
2bb7b766 1611
be48e3ea 1612 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1613 if (!aResult) {
1614 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
1615 return kFALSE;
1616 }
1617
fc5a4708 1618 AliDebug(2,Form("Query = %s", sqlQuery.Data()));
1619
2bb7b766 1620 if(aResult->GetRowCount() == 0) {
9827400b 1621 AliInfo("No entries in Shuttle Logbook match request");
1622 delete aResult;
1623 return kTRUE;
2bb7b766 1624 }
1625
1626 // TODO Check field count!
fc5a4708 1627 const UInt_t nCols = 22;
2bb7b766 1628 if (aResult->GetFieldCount() != (Int_t) nCols) {
1629 AliError("Invalid SQL result field number!");
1630 delete aResult;
1631 return kFALSE;
1632 }
1633
2bb7b766 1634 TSQLRow* aRow;
1635 while ((aRow = aResult->Next())) {
1636 TString runString(aRow->GetField(0), aRow->GetFieldLength(0));
1637 Int_t run = runString.Atoi();
1638
eba76848 1639 AliShuttleLogbookEntry *entry = QueryRunParameters(run);
1640 if (!entry)
1641 continue;
2bb7b766 1642
1643 // loop on detectors
eba76848 1644 for(UInt_t ii = 0; ii < nCols; ii++)
1645 entry->SetDetectorStatus(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1646
eba76848 1647 entries.AddLast(entry);
2bb7b766 1648 delete aRow;
1649 }
1650
2bb7b766 1651 delete aResult;
1652 return kTRUE;
1653}
1654
1655//______________________________________________________________________________________________
eba76848 1656AliShuttleLogbookEntry* AliShuttle::QueryRunParameters(Int_t run)
2bb7b766 1657{
eba76848 1658 //
1659 // Retrieve run parameters written in the DAQ logbook and sets them into AliShuttleLogbookEntry object
1660 //
2bb7b766 1661
1662 // check connection, in case connect
be48e3ea 1663 if (!Connect(3))
eba76848 1664 return 0;
2bb7b766 1665
1666 TString sqlQuery;
2c15234c 1667 sqlQuery.Form("select * from %s where run=%d", fConfig->GetDAQlbTable(), run);
2bb7b766 1668
be48e3ea 1669 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1670 if (!aResult) {
1671 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
eba76848 1672 return 0;
2bb7b766 1673 }
1674
eba76848 1675 if (aResult->GetRowCount() == 0) {
2bb7b766 1676 Log("SHUTTLE", Form("QueryRunParameters - No entry in DAQ Logbook for run %d. Skipping", run));
1677 delete aResult;
eba76848 1678 return 0;
2bb7b766 1679 }
1680
eba76848 1681 if (aResult->GetRowCount() > 1) {
2bb7b766 1682 AliError(Form("More than one entry in DAQ Logbook for run %d. Skipping", run));
1683 delete aResult;
eba76848 1684 return 0;
2bb7b766 1685 }
1686
eba76848 1687 TSQLRow* aRow = aResult->Next();
1688 if (!aRow)
1689 {
1690 AliError(Form("Could not retrieve row for run %d. Skipping", run));
1691 delete aResult;
1692 return 0;
1693 }
2bb7b766 1694
eba76848 1695 AliShuttleLogbookEntry* entry = new AliShuttleLogbookEntry(run);
2bb7b766 1696
eba76848 1697 for (Int_t ii = 0; ii < aResult->GetFieldCount(); ii++)
1698 entry->SetRunParameter(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1699
eba76848 1700 UInt_t startTime = entry->GetStartTime();
1701 UInt_t endTime = entry->GetEndTime();
1702
1703 if (!startTime || !endTime || startTime > endTime) {
1704 Log("SHUTTLE",
1705 Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d",
1706 run, startTime, endTime));
1707 delete entry;
2bb7b766 1708 delete aRow;
eba76848 1709 delete aResult;
1710 return 0;
2bb7b766 1711 }
1712
eba76848 1713 delete aRow;
2bb7b766 1714 delete aResult;
eba76848 1715
1716 return entry;
2bb7b766 1717}
1718
b948db8d 1719//______________________________________________________________________________________________
2c15234c 1720Bool_t AliShuttle::GetValueSet(const char* host, Int_t port, const char* entry,
1721 TObjArray* valueSet, DCSType type)
73abe331 1722{
9827400b 1723 // Retrieve all "entry" data points from the DCS server
1724 // host, port: TSocket connection parameters
1725 // entry: name of the alias or data point
1726 // valueSet: array of retrieved AliDCSValue's
1727 // type: kAlias or kDP
58bc3020 1728
73abe331 1729 AliDCSClient client(host, port, fTimeout, fRetries);
2c15234c 1730 if (!client.IsConnected())
1731 {
b948db8d 1732 return kFALSE;
73abe331 1733 }
1734
2c15234c 1735 Int_t result=0;
73abe331 1736
2c15234c 1737 if (type == kAlias)
1738 {
1739 result = client.GetAliasValues(entry,
1740 GetCurrentStartTime(), GetCurrentEndTime(), valueSet);
1741 } else
1742 if (type == kDP)
1743 {
1744 result = client.GetDPValues(entry,
1745 GetCurrentStartTime(), GetCurrentEndTime(), valueSet);
1746 }
1747
1748 if (result < 0)
1749 {
2bb7b766 1750 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get '%s'! Reason: %s",
2c15234c 1751 entry, AliDCSClient::GetErrorString(result)));
73abe331 1752
2c15234c 1753 if (result == AliDCSClient::fgkServerError)
1754 {
2bb7b766 1755 Log(fCurrentDetector.Data(), Form("GetValueSet - Server error: %s",
73abe331 1756 client.GetServerError().Data()));
1757 }
1758
1759 return kFALSE;
1760 }
1761
1762 return kTRUE;
1763}
b948db8d 1764
1765//______________________________________________________________________________________________
57f50b3c 1766const char* AliShuttle::GetFile(Int_t system, const char* detector,
1767 const char* id, const char* source)
b948db8d 1768{
9827400b 1769 // Get calibration file from file exchange servers
1770 // First queris the FXS database for the file name, using the run, detector, id and source info
1771 // then calls RetrieveFile(filename) for actual copy to local disk
1772 // run: current run being processed (given by Logbook entry fLogbookEntry)
1773 // detector: the Preprocessor name
1774 // id: provided as a parameter by the Preprocessor
1775 // source: provided by the Preprocessor through GetFileSources function
1776
1777 // check if test mode should simulate a FXS error
1778 if (fTestMode & kErrorFXSFiles)
1779 {
1780 Log(detector, Form("GetFile - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
1781 return 0;
1782 }
1783
57f50b3c 1784 // check connection, in case connect
9d733021 1785 if (!Connect(system))
eba76848 1786 {
9d733021 1787 Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
57f50b3c 1788 return 0;
1789 }
1790
1791 // Query preparation
9d733021 1792 TString sourceName(source);
d386d623 1793 Int_t nFields = 3;
1794 TString sqlQueryStart = Form("select filePath,size,fileChecksum from %s where",
1795 fConfig->GetFXSdbTable(system));
1796 TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
1797 GetCurrentRun(), detector, id);
1798
9d733021 1799 if (system == kDAQ)
1800 {
d386d623 1801 whereClause += Form(" and DAQsource=\"%s\"", source);
57f50b3c 1802 }
9d733021 1803 else if (system == kDCS)
eba76848 1804 {
9d733021 1805 sourceName="none";
57f50b3c 1806 }
9d733021 1807 else if (system == kHLT)
9e080f92 1808 {
d386d623 1809 whereClause += Form(" and DDLnumbers=\"%s\"", source);
9d733021 1810 nFields = 3;
9e080f92 1811 }
1812
9e080f92 1813 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
1814
1815 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
1816
1817 // Query execution
1818 TSQLResult* aResult = 0;
9d733021 1819 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
9e080f92 1820 if (!aResult) {
9d733021 1821 Log(detector, Form("GetFileName - Can't execute SQL query to %s database for: id = %s, source = %s",
1822 GetSystemName(system), id, sourceName.Data()));
9e080f92 1823 return 0;
1824 }
1825
1826 if(aResult->GetRowCount() == 0)
1827 {
1828 Log(detector,
9d733021 1829 Form("GetFileName - No entry in %s FXS db for: id = %s, source = %s",
1830 GetSystemName(system), id, sourceName.Data()));
9e080f92 1831 delete aResult;
1832 return 0;
1833 }
2bb7b766 1834
9e080f92 1835 if (aResult->GetRowCount() > 1) {
1836 Log(detector,
9d733021 1837 Form("GetFileName - More than one entry in %s FXS db for: id = %s, source = %s",
1838 GetSystemName(system), id, sourceName.Data()));
9e080f92 1839 delete aResult;
1840 return 0;
1841 }
1842
9d733021 1843 if (aResult->GetFieldCount() != nFields) {
9e080f92 1844 Log(detector,
9d733021 1845 Form("GetFileName - Wrong field count in %s FXS db for: id = %s, source = %s",
1846 GetSystemName(system), id, sourceName.Data()));
9e080f92 1847 delete aResult;
1848 return 0;
1849 }
1850
1851 TSQLRow* aRow = dynamic_cast<TSQLRow*> (aResult->Next());
1852
1853 if (!aRow){
9d733021 1854 Log(detector, Form("GetFileName - Empty set result in %s FXS db from query: id = %s, source = %s",
1855 GetSystemName(system), id, sourceName.Data()));
9e080f92 1856 delete aResult;
1857 return 0;
1858 }
1859
1860 TString filePath(aRow->GetField(0), aRow->GetFieldLength(0));
1861 TString fileSize(aRow->GetField(1), aRow->GetFieldLength(1));
d386d623 1862 TString fileChecksum(aRow->GetField(2), aRow->GetFieldLength(2));
9e080f92 1863
1864 delete aResult;
1865 delete aRow;
1866
d386d623 1867 AliDebug(2, Form("filePath = %s; size = %s, fileChecksum = %s",
1868 filePath.Data(), fileSize.Data(), fileChecksum.Data()));
9e080f92 1869
9e080f92 1870 // retrieved file is renamed to make it unique
9d733021 1871 TString localFileName = Form("%s_%s_%d_%s_%s.shuttle",
1872 GetSystemName(system), detector, GetCurrentRun(), id, sourceName.Data());
1873
9e080f92 1874
9d733021 1875 // file retrieval from FXS
4b95672b 1876 UInt_t nRetries = 0;
1877 UInt_t maxRetries = 3;
1878 Bool_t result = kFALSE;
1879
1880 // copy!! if successful TSystem::Exec returns 0
1881 while(nRetries++ < maxRetries) {
1882 AliDebug(2, Form("Trying to copy file. Retry # %d", nRetries));
1883 result = RetrieveFile(system, filePath.Data(), localFileName.Data());
1884 if(!result)
1885 {
1886 Log(detector, Form("GetFileName - Copy of file %s from %s FXS failed",
9d733021 1887 filePath.Data(), GetSystemName(system)));
4b95672b 1888 continue;
1889 } else {
1890 AliInfo(Form("File %s copied from %s FXS into %s/%s",
1891 filePath.Data(), GetSystemName(system),
1892 GetShuttleTempDir(), localFileName.Data()));
1893 }
9e080f92 1894
d386d623 1895 if (fileChecksum.Length()>0)
4b95672b 1896 {
1897 // compare md5sum of local file with the one stored in the FXS DB
1898 Int_t md5Comp = gSystem->Exec(Form("md5sum %s/%s |grep %s 2>&1 > /dev/null",
d386d623 1899 GetShuttleTempDir(), localFileName.Data(), fileChecksum.Data()));
9e080f92 1900
4b95672b 1901 if (md5Comp != 0)
1902 {
1903 Log(detector, Form("GetFileName - md5sum of file %s does not match with local copy!",
1904 filePath.Data()));
1905 result = kFALSE;
1906 continue;
1907 }
d386d623 1908 } else {
1909 Log(fCurrentDetector, Form("GetFile - md5sum of file %s not set in %s database, skipping comparison",
1910 filePath.Data(), GetSystemName(system)));
9d733021 1911 }
4b95672b 1912 if (result) break;
9e080f92 1913 }
1914
4b95672b 1915 if(!result) return 0;
1916
9d733021 1917 fFXSCalled[system]=kTRUE;
1918 TObjString *fileParams = new TObjString(Form("%s#!?!#%s", id, sourceName.Data()));
1919 fFXSlist[system].Add(fileParams);
9e080f92 1920
1921 static TString fullLocalFileName;
36c99a6a 1922 fullLocalFileName = TString::Format("%s/%s", GetShuttleTempDir(), localFileName.Data());
1923
9e080f92 1924 AliInfo(Form("fullLocalFileName = %s", fullLocalFileName.Data()));
1925
1926 return fullLocalFileName.Data();
2bb7b766 1927
1928}
1929
1930//______________________________________________________________________________________________
9d733021 1931Bool_t AliShuttle::RetrieveFile(UInt_t system, const char* fxsFileName, const char* localFileName)
9e080f92 1932{
9827400b 1933 //
1934 // Copies file from FXS to local Shuttle machine
1935 //
2bb7b766 1936
9e080f92 1937 // check temp directory: trying to cd to temp; if it does not exist, create it
9d733021 1938 AliDebug(2, Form("Copy file %s from %s FXS into %s/%s",
1939 GetSystemName(system), fxsFileName, GetShuttleTempDir(), localFileName));
9e080f92 1940
36c99a6a 1941 void* dir = gSystem->OpenDirectory(GetShuttleTempDir());
9e080f92 1942 if (dir == NULL) {
36c99a6a 1943 if (gSystem->mkdir(GetShuttleTempDir(), kTRUE)) {
1944 AliError(Form("Can't open directory <%s>", GetShuttleTempDir()));
9e080f92 1945 return kFALSE;
1946 }
1947
1948 } else {
1949 gSystem->FreeDirectory(dir);
1950 }
1951
9d733021 1952 TString baseFXSFolder;
1953 if (system == kDAQ)
1954 {
1955 baseFXSFolder = "FES/";
1956 }
1957 else if (system == kDCS)
1958 {
1959 baseFXSFolder = "";
1960 }
1961 else if (system == kHLT)
1962 {
1963 baseFXSFolder = "~/";
1964 }
1965
1966
1967 TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s/%s",
1968 fConfig->GetFXSPort(system),
1969 fConfig->GetFXSUser(system),
1970 fConfig->GetFXSHost(system),
1971 baseFXSFolder.Data(),
1972 fxsFileName,
36c99a6a 1973 GetShuttleTempDir(),
9e080f92 1974 localFileName);
1975
1976 AliDebug(2, Form("%s",command.Data()));
1977
4b95672b 1978 Bool_t result = (gSystem->Exec(command.Data()) == 0);
9e080f92 1979
4b95672b 1980 return result;
9e080f92 1981}
1982
1983//______________________________________________________________________________________________
9d733021 1984TList* AliShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
1985{
9827400b 1986 //
1987 // Get sources producing the condition file Id from file exchange servers
1988 //
1989
1990 // check if test mode should simulate a FXS error
1991 if (fTestMode & kErrorFXSSources)
1992 {
1993 Log(detector, Form("GetFileSources - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
1994 return 0;
1995 }
1996
9d733021 1997
1998 if (system == kDCS)
1999 {
2000 AliError("DCS system has only one source of data!");
2001 return NULL;
9d733021 2002 }
9e080f92 2003
2004 // check connection, in case connect
9d733021 2005 if (!Connect(system))
2006 {
2007 Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
2008 return NULL;
9e080f92 2009 }
2010
9d733021 2011 TString sourceName = 0;
2012 if (system == kDAQ)
2013 {
2014 sourceName = "DAQsource";
2015 } else if (system == kHLT)
2016 {
2017 sourceName = "DDLnumbers";
2018 }
2019
d386d623 2020 TString sqlQueryStart = Form("select %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
9e080f92 2021 TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
2022 GetCurrentRun(), detector, id);
2023 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2024
2025 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2026
2027 // Query execution
2028 TSQLResult* aResult;
9d733021 2029 aResult = fServer[system]->Query(sqlQuery);
9e080f92 2030 if (!aResult) {
9d733021 2031 Log(detector, Form("GetFileSources - Can't execute SQL query to %s database for id: %s",
2032 GetSystemName(system), id));
9e080f92 2033 return 0;
2034 }
2035
86aa42c3 2036 TList *list = new TList();
2037 list->SetOwner(1);
2038
9d733021 2039 if (aResult->GetRowCount() == 0)
2040 {
9e080f92 2041 Log(detector,
9d733021 2042 Form("GetFileSources - No entry in %s FXS table for id: %s", GetSystemName(system), id));
9e080f92 2043 delete aResult;
86aa42c3 2044 return list;
9e080f92 2045 }
2046
2047 TSQLRow* aRow;
9e080f92 2048
9d733021 2049 while ((aRow = aResult->Next()))
2050 {
9e080f92 2051
9d733021 2052 TString source(aRow->GetField(0), aRow->GetFieldLength(0));
2053 AliDebug(2, Form("%s = %s", sourceName.Data(), source.Data()));
2054 list->Add(new TObjString(source));
9e080f92 2055 delete aRow;
2056 }
9d733021 2057
9e080f92 2058 delete aResult;
2059
2060 return list;
2bb7b766 2061}
2062
2063//______________________________________________________________________________________________
9d733021 2064Bool_t AliShuttle::Connect(Int_t system)
2bb7b766 2065{
9827400b 2066 // Connect to MySQL Server of the system's FXS MySQL databases
2067 // DAQ Logbook, Shuttle Logbook and DAQ FXS db are on the same host
2068 //
57f50b3c 2069
9d733021 2070 // check connection: if already connected return
2071 if(fServer[system] && fServer[system]->IsConnected()) return kTRUE;
57f50b3c 2072
9d733021 2073 TString dbHost, dbUser, dbPass, dbName;
57f50b3c 2074
9d733021 2075 if (system < 3) // FXS db servers
2076 {
2077 dbHost = Form("mysql://%s:%d", fConfig->GetFXSdbHost(system), fConfig->GetFXSdbPort(system));
2078 dbUser = fConfig->GetFXSdbUser(system);
2079 dbPass = fConfig->GetFXSdbPass(system);
2080 dbName = fConfig->GetFXSdbName(system);
2081 } else { // Run & Shuttle logbook servers
2082 // TODO Will the Shuttle logbook server be the same as the Run logbook server ???
2083 dbHost = Form("mysql://%s:%d", fConfig->GetDAQlbHost(), fConfig->GetDAQlbPort());
2084 dbUser = fConfig->GetDAQlbUser();
2085 dbPass = fConfig->GetDAQlbPass();
2086 dbName = fConfig->GetDAQlbDB();
2087 }
57f50b3c 2088
9d733021 2089 fServer[system] = TSQLServer::Connect(dbHost.Data(), dbUser.Data(), dbPass.Data());
2090 if (!fServer[system] || !fServer[system]->IsConnected()) {
2091 if(system < 3)
2092 {
2093 AliError(Form("Can't establish connection to FXS database for %s",
2094 AliShuttleInterface::GetSystemName(system)));
2095 } else {
2096 AliError("Can't establish connection to Run logbook.");
57f50b3c 2097 }
9d733021 2098 if(fServer[system]) delete fServer[system];
2099 return kFALSE;
2bb7b766 2100 }
57f50b3c 2101
9d733021 2102 // Get tables
2103 TSQLResult* aResult=0;
2104 switch(system){
2105 case kDAQ:
2106 aResult = fServer[kDAQ]->GetTables(dbName.Data());
2107 break;
2108 case kDCS:
2109 aResult = fServer[kDCS]->GetTables(dbName.Data());
2110 break;
2111 case kHLT:
2112 aResult = fServer[kHLT]->GetTables(dbName.Data());
2113 break;
2114 default:
2115 aResult = fServer[3]->GetTables(dbName.Data());
2116 break;
2117 }
2118
2119 delete aResult;
2bb7b766 2120 return kTRUE;
2121}
57f50b3c 2122
9e080f92 2123//______________________________________________________________________________________________
9d733021 2124Bool_t AliShuttle::UpdateTable()
9e080f92 2125{
9827400b 2126 //
2127 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2128 //
9e080f92 2129
9d733021 2130 Bool_t result = kTRUE;
9e080f92 2131
9d733021 2132 for (UInt_t system=0; system<3; system++)
2133 {
2134 if(!fFXSCalled[system]) continue;
9e080f92 2135
9d733021 2136 // check connection, in case connect
2137 if (!Connect(system))
2138 {
2139 Log(fCurrentDetector, Form("UpdateTable - Couldn't connect to %s FXS database", GetSystemName(system)));
2140 result = kFALSE;
2141 continue;
9e080f92 2142 }
9e080f92 2143
9d733021 2144 TTimeStamp now; // now
2145
2146 // Loop on FXS list entries
2147 TIter iter(&fFXSlist[system]);
2148 TObjString *aFXSentry=0;
2149 while ((aFXSentry = dynamic_cast<TObjString*> (iter.Next())))
2150 {
2151 TString aFXSentrystr = aFXSentry->String();
2152 TObjArray *aFXSarray = aFXSentrystr.Tokenize("#!?!#");
2153 if (!aFXSarray || aFXSarray->GetEntries() != 2 )
2154 {
2155 Log(fCurrentDetector, Form("UpdateTable - error updating %s FXS entry. Check string: <%s>",
2156 GetSystemName(system), aFXSentrystr.Data()));
2157 if(aFXSarray) delete aFXSarray;
2158 result = kFALSE;
2159 continue;
2160 }
2161 const char* fileId = ((TObjString*) aFXSarray->At(0))->GetName();
2162 const char* source = ((TObjString*) aFXSarray->At(1))->GetName();
2163
2164 TString whereClause;
2165 if (system == kDAQ)
2166 {
2167 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DAQsource=\"%s\";",
2168 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2169 }
2170 else if (system == kDCS)
2171 {
2172 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\";",
2173 GetCurrentRun(), fCurrentDetector.Data(), fileId);
2174 }
2175 else if (system == kHLT)
2176 {
2177 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DDLnumbers=\"%s\";",
2178 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2179 }
2180
2181 delete aFXSarray;
9e080f92 2182
9d733021 2183 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2184 now.GetSec(), whereClause.Data());
9e080f92 2185
9d733021 2186 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
9e080f92 2187
9d733021 2188 // Query execution
2189 TSQLResult* aResult;
2190 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2191 if (!aResult)
2192 {
2193 Log(fCurrentDetector, Form("UpdateTable - %s db: can't execute SQL query <%s>",
2194 GetSystemName(system), sqlQuery.Data()));
2195 result = kFALSE;
2196 continue;
2197 }
2198 delete aResult;
9e080f92 2199 }
9e080f92 2200 }
2201
9d733021 2202 return result;
9e080f92 2203}
57f50b3c 2204
3301427a 2205//______________________________________________________________________________________________
2206Bool_t AliShuttle::UpdateTableFailCase()
2207{
9827400b 2208 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2209 // this is called in case the preprocessor is declared failed for the current run, because
2210 // the fields are updated only in case of success
3301427a 2211
2212 Bool_t result = kTRUE;
2213
2214 for (UInt_t system=0; system<3; system++)
2215 {
2216 // check connection, in case connect
2217 if (!Connect(system))
2218 {
2219 Log(fCurrentDetector, Form("UpdateTableFailCase - Couldn't connect to %s FXS database",
2220 GetSystemName(system)));
2221 result = kFALSE;
2222 continue;
2223 }
2224
2225 TTimeStamp now; // now
2226
2227 // Loop on FXS list entries
2228
2229 TString whereClause = Form("where run=%d and detector=\"%s\";",
2230 GetCurrentRun(), fCurrentDetector.Data());
2231
2232
2233 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2234 now.GetSec(), whereClause.Data());
2235
2236 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2237
2238 // Query execution
2239 TSQLResult* aResult;
2240 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2241 if (!aResult)
2242 {
2243 Log(fCurrentDetector, Form("UpdateTableFailCase - %s db: can't execute SQL query <%s>",
2244 GetSystemName(system), sqlQuery.Data()));
2245 result = kFALSE;
2246 continue;
2247 }
2248 delete aResult;
2249 }
2250
2251 return result;
2252}
2253
2bb7b766 2254//______________________________________________________________________________________________
2255Bool_t AliShuttle::UpdateShuttleLogbook(const char* detector, const char* status)
2256{
e7f62f16 2257 //
2258 // Update Shuttle logbook filling detector or shuttle_done column
2259 // ex. of usage: UpdateShuttleLogbook("PHOS", "DONE") or UpdateShuttleLogbook("shuttle_done")
2260 //
57f50b3c 2261
2bb7b766 2262 // check connection, in case connect
be48e3ea 2263 if(!Connect(3)){
2bb7b766 2264 Log("SHUTTLE", "UpdateShuttleLogbook - Couldn't connect to DAQ Logbook.");
2265 return kFALSE;
57f50b3c 2266 }
2267
2bb7b766 2268 TString detName(detector);
2269 TString setClause;
e7f62f16 2270 if(detName == "shuttle_done")
2271 {
2bb7b766 2272 setClause = "set shuttle_done=1";
e7f62f16 2273
2274 // Send the information to ML
2275 TMonaLisaText mlStatus("SHUTTLE_status", "Done");
2276
2277 TList mlList;
2278 mlList.Add(&mlStatus);
2279
2280 fMonaLisa->SendParameters(&mlList);
2bb7b766 2281 } else {
2bb7b766 2282 TString statusStr(status);
2283 if(statusStr.Contains("done", TString::kIgnoreCase) ||
2284 statusStr.Contains("failed", TString::kIgnoreCase)){
eba76848 2285 setClause = Form("set %s=\"%s\"", detector, status);
2bb7b766 2286 } else {
2287 Log("SHUTTLE",
2288 Form("UpdateShuttleLogbook - Invalid status <%s> for detector %s",
2289 status, detector));
2290 return kFALSE;
2291 }
2292 }
57f50b3c 2293
2bb7b766 2294 TString whereClause = Form("where run=%d", GetCurrentRun());
2295
441b0e9c 2296 TString sqlQuery = Form("update %s %s %s",
2297 fConfig->GetShuttlelbTable(), setClause.Data(), whereClause.Data());
57f50b3c 2298
2bb7b766 2299 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2300
2301 // Query execution
2302 TSQLResult* aResult;
be48e3ea 2303 aResult = dynamic_cast<TSQLResult*> (fServer[3]->Query(sqlQuery));
2bb7b766 2304 if (!aResult) {
2305 Log("SHUTTLE", Form("UpdateShuttleLogbook - Can't execute query <%s>", sqlQuery.Data()));
2306 return kFALSE;
57f50b3c 2307 }
2bb7b766 2308 delete aResult;
57f50b3c 2309
2310 return kTRUE;
2311}
2312
2313//______________________________________________________________________________________________
2bb7b766 2314Int_t AliShuttle::GetCurrentRun() const
2315{
9827400b 2316 //
2317 // Get current run from logbook entry
2318 //
57f50b3c 2319
2bb7b766 2320 return fLogbookEntry ? fLogbookEntry->GetRun() : -1;
57f50b3c 2321}
2322
2323//______________________________________________________________________________________________
2bb7b766 2324UInt_t AliShuttle::GetCurrentStartTime() const
2325{
9827400b 2326 //
2327 // get current start time
2328 //
57f50b3c 2329
2bb7b766 2330 return fLogbookEntry ? fLogbookEntry->GetStartTime() : 0;
57f50b3c 2331}
2332
2333//______________________________________________________________________________________________
2bb7b766 2334UInt_t AliShuttle::GetCurrentEndTime() const
2335{
9827400b 2336 //
2337 // get current end time from logbook entry
2338 //
57f50b3c 2339
2bb7b766 2340 return fLogbookEntry ? fLogbookEntry->GetEndTime() : 0;
57f50b3c 2341}
2342
b948db8d 2343//______________________________________________________________________________________________
2344void AliShuttle::Log(const char* detector, const char* message)
2345{
9827400b 2346 //
2347 // Fill log string with a message
2348 //
b948db8d 2349
36c99a6a 2350 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
84090f85 2351 if (dir == NULL) {
36c99a6a 2352 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE)) {
2353 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
84090f85 2354 return;
2355 }
b948db8d 2356
84090f85 2357 } else {
2358 gSystem->FreeDirectory(dir);
2359 }
b948db8d 2360
cb343cfd 2361 TString toLog = Form("%s (%d): %s - ", TTimeStamp(time(0)).AsString("s"), getpid(), detector);
e7f62f16 2362 if (GetCurrentRun() >= 0)
2363 toLog += Form("run %d - ", GetCurrentRun());
2bb7b766 2364 toLog += Form("%s", message);
2365
84090f85 2366 AliInfo(toLog.Data());
ffa29e93 2367
2368 // if we redirect the log output already to the file, leave here
2369 if (fOutputRedirected && strcmp(detector, "SHUTTLE") != 0)
2370 return;
b948db8d 2371
ffa29e93 2372 TString fileName = GetLogFileName(detector);
e7f62f16 2373
84090f85 2374 gSystem->ExpandPathName(fileName);
2375
2376 ofstream logFile;
2377 logFile.open(fileName, ofstream::out | ofstream::app);
2378
2379 if (!logFile.is_open()) {
2380 AliError(Form("Could not open file %s", fileName.Data()));
2381 return;
2382 }
7bfb2090 2383
84090f85 2384 logFile << toLog.Data() << "\n";
b948db8d 2385
84090f85 2386 logFile.close();
b948db8d 2387}
2bb7b766 2388
ffa29e93 2389//______________________________________________________________________________________________
2390TString AliShuttle::GetLogFileName(const char* detector) const
2391{
2392 //
2393 // returns the name of the log file for a given sub detector
2394 //
2395
2396 TString fileName;
2397
2398 if (GetCurrentRun() >= 0)
2399 fileName.Form("%s/%s_%d.log", GetShuttleLogDir(), detector, GetCurrentRun());
2400 else
2401 fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
2402
2403 return fileName;
2404}
2405
2bb7b766 2406//______________________________________________________________________________________________
2407Bool_t AliShuttle::Collect(Int_t run)
2408{
9827400b 2409 //
2410 // Collects conditions data for all UNPROCESSED run written to DAQ LogBook in case of run = -1 (default)
2411 // If a dedicated run is given this run is processed
2412 //
2413 // In operational mode, this is the Shuttle function triggered by the EOR signal.
2414 //
2bb7b766 2415
eba76848 2416 if (run == -1)
2417 Log("SHUTTLE","Collect - Shuttle called. Collecting conditions data for unprocessed runs");
2418 else
2419 Log("SHUTTLE", Form("Collect - Shuttle called. Collecting conditions data for run %d", run));
cb343cfd 2420
2421 SetLastAction("Starting");
2bb7b766 2422
2423 TString whereClause("where shuttle_done=0");
eba76848 2424 if (run != -1)
2425 whereClause += Form(" and run=%d", run);
2bb7b766 2426
2427 TObjArray shuttleLogbookEntries;
be48e3ea 2428 if (!QueryShuttleLogbook(whereClause, shuttleLogbookEntries))
2429 {
cb343cfd 2430 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2bb7b766 2431 return kFALSE;
2432 }
2433
9e080f92 2434 if (shuttleLogbookEntries.GetEntries() == 0)
2435 {
2436 if (run == -1)
2437 Log("SHUTTLE","Collect - Found no UNPROCESSED runs in Shuttle logbook");
2438 else
2439 Log("SHUTTLE", Form("Collect - Run %d is already DONE "
2440 "or it does not exist in Shuttle logbook", run));
2441 return kTRUE;
2442 }
2443
be48e3ea 2444 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2445 fFirstUnprocessed[iDet] = kTRUE;
2446
fc5a4708 2447 if (run != -1)
be48e3ea 2448 {
2449 // query Shuttle logbook for earlier runs, check if some detectors are unprocessed,
2450 // flag them into fFirstUnprocessed array
2451 TString whereClause(Form("where shuttle_done=0 and run < %d", run));
2452 TObjArray tmpLogbookEntries;
2453 if (!QueryShuttleLogbook(whereClause, tmpLogbookEntries))
2454 {
2455 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2456 return kFALSE;
2457 }
2458
2459 TIter iter(&tmpLogbookEntries);
2460 AliShuttleLogbookEntry* anEntry = 0;
2461 while ((anEntry = dynamic_cast<AliShuttleLogbookEntry*> (iter.Next())))
2462 {
2463 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2464 {
2465 if (anEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
2466 {
2467 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
2468 anEntry->GetRun(), GetDetName(iDet)));
2469 fFirstUnprocessed[iDet] = kFALSE;
2470 }
2471 }
2472
2473 }
2474
2475 }
2476
2477 if (!RetrieveConditionsData(shuttleLogbookEntries))
2478 {
cb343cfd 2479 Log("SHUTTLE", "Collect - Process of at least one run failed");
2bb7b766 2480 return kFALSE;
2481 }
2482
36c99a6a 2483 Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
eba76848 2484 return kTRUE;
2bb7b766 2485}
2486
2bb7b766 2487//______________________________________________________________________________________________
2488Bool_t AliShuttle::RetrieveConditionsData(const TObjArray& dateEntries)
2489{
9827400b 2490 //
2491 // Retrieve conditions data for all runs that aren't processed yet
2492 //
2bb7b766 2493
2494 Bool_t hasError = kFALSE;
2495
2496 TIter iter(&dateEntries);
2497 AliShuttleLogbookEntry* anEntry;
2498
2499 while ((anEntry = (AliShuttleLogbookEntry*) iter.Next())){
2500 if (!Process(anEntry)){
2501 hasError = kTRUE;
2502 }
4b95672b 2503
2504 // clean SHUTTLE temp directory
3301427a 2505 TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
2506 RemoveFile(filename.Data());
2bb7b766 2507 }
2508
2509 return hasError == kFALSE;
2510}
cb343cfd 2511
2512//______________________________________________________________________________________________
2513ULong_t AliShuttle::GetTimeOfLastAction() const
2514{
9827400b 2515 //
2516 // Gets time of last action
2517 //
2518
cb343cfd 2519 ULong_t tmp;
36c99a6a 2520
cb343cfd 2521 fMonitoringMutex->Lock();
be48e3ea 2522
cb343cfd 2523 tmp = fLastActionTime;
36c99a6a 2524
cb343cfd 2525 fMonitoringMutex->UnLock();
36c99a6a 2526
cb343cfd 2527 return tmp;
2528}
2529
2530//______________________________________________________________________________________________
2531const TString AliShuttle::GetLastAction() const
2532{
9827400b 2533 //
cb343cfd 2534 // returns a string description of the last action
9827400b 2535 //
cb343cfd 2536
2537 TString tmp;
36c99a6a 2538
cb343cfd 2539 fMonitoringMutex->Lock();
2540
2541 tmp = fLastAction;
2542
2543 fMonitoringMutex->UnLock();
2544
36c99a6a 2545 return tmp;
cb343cfd 2546}
2547
2548//______________________________________________________________________________________________
2549void AliShuttle::SetLastAction(const char* action)
2550{
9827400b 2551 //
cb343cfd 2552 // updates the monitoring variables
9827400b 2553 //
36c99a6a 2554
cb343cfd 2555 fMonitoringMutex->Lock();
36c99a6a 2556
cb343cfd 2557 fLastAction = action;
2558 fLastActionTime = time(0);
2559
2560 fMonitoringMutex->UnLock();
2561}
eba76848 2562
2563//______________________________________________________________________________________________
2564const char* AliShuttle::GetRunParameter(const char* param)
2565{
9827400b 2566 //
2567 // returns run parameter read from DAQ logbook
2568 //
eba76848 2569
2570 if(!fLogbookEntry) {
2571 AliError("No logbook entry!");
2572 return 0;
2573 }
2574
2575 return fLogbookEntry->GetRunParameter(param);
2576}
57c1a579 2577
d386d623 2578//______________________________________________________________________________________________
9827400b 2579AliCDBEntry* AliShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
d386d623 2580{
9827400b 2581 //
2582 // returns object from OCDB valid for current run
2583 //
d386d623 2584
9827400b 2585 if (fTestMode & kErrorOCDB)
2586 {
2587 Log(detector, "GetFromOCDB - In TESTMODE - Simulating error with OCDB");
2588 return 0;
2589 }
2590
d386d623 2591 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
2592 if (!sto)
2593 {
9827400b 2594 Log(detector, "GetFromOCDB - Cannot activate main OCDB for query!");
d386d623 2595 return 0;
2596 }
2597
2598 return dynamic_cast<AliCDBEntry*> (sto->Get(path, GetCurrentRun()));
2599}
2600
57c1a579 2601//______________________________________________________________________________________________
2602Bool_t AliShuttle::SendMail()
2603{
9827400b 2604 //
2605 // sends a mail to the subdetector expert in case of preprocessor error
2606 //
2607
2608 if (fTestMode != kNone)
2609 return kTRUE;
57c1a579 2610
36c99a6a 2611 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
57c1a579 2612 if (dir == NULL)
2613 {
36c99a6a 2614 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
57c1a579 2615 {
36c99a6a 2616 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
57c1a579 2617 return kFALSE;
2618 }
2619
2620 } else {
2621 gSystem->FreeDirectory(dir);
2622 }
2623
2624 TString bodyFileName;
36c99a6a 2625 bodyFileName.Form("%s/mail.body", GetShuttleLogDir());
57c1a579 2626 gSystem->ExpandPathName(bodyFileName);
2627
2628 ofstream mailBody;
2629 mailBody.open(bodyFileName, ofstream::out);
2630
2631 if (!mailBody.is_open())
2632 {
2633 AliError(Form("Could not open mail body file %s", bodyFileName.Data()));
2634 return kFALSE;
2635 }
2636
2637 TString to="";
2638 TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
2639 TObjString *anExpert=0;
2640 while ((anExpert = (TObjString*) iterExperts.Next()))
2641 {
2642 to += Form("%s,", anExpert->GetName());
2643 }
2644 to.Remove(to.Length()-1);
909732f7 2645 AliDebug(2, Form("to: %s",to.Data()));
57c1a579 2646
86aa42c3 2647 if (to.IsNull()) {
36c99a6a 2648 AliInfo("List of detector responsibles not yet set!");
2649 return kFALSE;
2650 }
2651
57c1a579 2652 TString cc="alberto.colla@cern.ch";
2653
546242fb 2654 TString subject = Form("%s Shuttle preprocessor FAILED in run %d !",
57c1a579 2655 fCurrentDetector.Data(), GetCurrentRun());
909732f7 2656 AliDebug(2, Form("subject: %s", subject.Data()));
57c1a579 2657
2658 TString body = Form("Dear %s expert(s), \n\n", fCurrentDetector.Data());
2659 body += Form("SHUTTLE just detected that your preprocessor "
546242fb 2660 "failed processing run %d!!\n\n", GetCurrentRun());
2661 body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n", fCurrentDetector.Data());
2662 body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
2663 body += Form("Find the %s log for the current run on \n\n"
2664 "\thttp://pcalishuttle01.cern.ch:8880/logs/%s_%d.log \n\n",
2665 fCurrentDetector.Data(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 2666 body += Form("The last 10 lines of %s log file are following:\n\n");
2667
909732f7 2668 AliDebug(2, Form("Body begin: %s", body.Data()));
57c1a579 2669
2670 mailBody << body.Data();
2671 mailBody.close();
2672 mailBody.open(bodyFileName, ofstream::out | ofstream::app);
2673
9d733021 2674 TString logFileName = Form("%s/%s_%d.log", GetShuttleLogDir(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 2675 TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
2676 if (gSystem->Exec(tailCommand.Data()))
2677 {
2678 mailBody << Form("%s log file not found ...\n\n", fCurrentDetector.Data());
2679 }
2680
2681 TString endBody = Form("------------------------------------------------------\n\n");
36c99a6a 2682 endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
2683 endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
546242fb 2684 endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
57c1a579 2685
909732f7 2686 AliDebug(2, Form("Body end: %s", endBody.Data()));
57c1a579 2687
2688 mailBody << endBody.Data();
2689
2690 mailBody.close();
2691
2692 // send mail!
2693 TString mailCommand = Form("mail -s \"%s\" -c %s %s < %s",
2694 subject.Data(),
2695 cc.Data(),
2696 to.Data(),
2697 bodyFileName.Data());
909732f7 2698 AliDebug(2, Form("mail command: %s", mailCommand.Data()));
57c1a579 2699
2700 Bool_t result = gSystem->Exec(mailCommand.Data());
2701
2702 return result == 0;
2703}
d386d623 2704
441b0e9c 2705//______________________________________________________________________________________________
9827400b 2706const char* AliShuttle::GetRunType()
441b0e9c 2707{
9827400b 2708 //
2709 // returns run type read from "run type" logbook
2710 //
441b0e9c 2711
2712 if(!fLogbookEntry) {
2713 AliError("No logbook entry!");
2714 return 0;
2715 }
2716
9827400b 2717 return fLogbookEntry->GetRunType();
441b0e9c 2718}
2719
d386d623 2720//______________________________________________________________________________________________
2721void AliShuttle::SetShuttleTempDir(const char* tmpDir)
2722{
9827400b 2723 //
2724 // sets Shuttle temp directory
2725 //
d386d623 2726
2727 fgkShuttleTempDir = gSystem->ExpandPathName(tmpDir);
2728}
2729
2730//______________________________________________________________________________________________
2731void AliShuttle::SetShuttleLogDir(const char* logDir)
2732{
9827400b 2733 //
2734 // sets Shuttle log directory
2735 //
d386d623 2736
2737 fgkShuttleLogDir = gSystem->ExpandPathName(logDir);
2738}