]> git.uio.no Git - u/mrichter/AliRoot.git/blame - SHUTTLE/AliShuttle.cxx
git://git.uio.no / u / mrichter / AliRoot.git / blame
? search:
summary | shortlog | log | commit | commitdiff | tree
blob | blame (incremental) | history | HEAD
preprocessor is run in a temp directory that is removed when process is finished.
[u/mrichter/AliRoot.git] / SHUTTLE / AliShuttle.cxx
CommitLineData
73abe331 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
5bac2bde 18Revision 1.49 2007/06/29 10:45:06 acolla
19Number of columns in MySql Shuttle logbook increased by one (HLT added)
20
db99d43e 21Revision 1.48 2007/06/21 13:06:19 acolla
22GetFileSources returns dummy list with 1 source if system=DCS (better than
23returning error as it was)
24
6297b37d 25Revision 1.47 2007/06/19 17:28:56 acolla
26HLT updated; missing map bug removed.
27
dc25836b 28Revision 1.46 2007/06/09 13:01:09 jgrosseo
29Switching to retrieval of several DCS DPs at a time (multiDPrequest)
30
a038aa70 31Revision 1.45 2007/05/30 06:35:20 jgrosseo
32Adding functionality to the Shuttle/TestShuttle:
33o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
34o) Function to retrieve list of IDs for a given source (GetFileIDs)
35These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
36Example code has been added to the TestProcessor in TestShuttle
37
4a33bdd9 38Revision 1.44 2007/05/11 16:09:32 acolla
39Reference files for ITS, MUON and PHOS are now stored in OfflineDetName/OnlineDetName/run_...
40example: ITS/SPD/100_filename.root
41
2d9019b4 42Revision 1.43 2007/05/10 09:59:51 acolla
43Various bug fixes in StoreRefFilesToGrid; Cleaning of reference storage before processing detector (CleanReferenceStorage)
44
546242fb 45Revision 1.42 2007/05/03 08:01:39 jgrosseo
46typo in last commit :-(
47
8b739301 48Revision 1.41 2007/05/03 08:00:48 jgrosseo
49fixing log message when pp want to skip dcs value retrieval
50
651fdaab 51Revision 1.40 2007/04/27 07:06:48 jgrosseo
52GetFileSources returns empty list in case of no files, but successful query
53No mails sent in testmode
54
86aa42c3 55Revision 1.39 2007/04/17 12:43:57 acolla
56Correction in StoreOCDB; change of text in mail to detector expert
57
26758fce 58Revision 1.38 2007/04/12 08:26:18 jgrosseo
59updated comment
60
3c2a21c8 61Revision 1.37 2007/04/10 16:53:14 jgrosseo
62redirecting sub detector stdout, stderr to sub detector log file
63
3d8bc902 64Revision 1.35 2007/04/04 16:26:38 acolla
651. Re-organization of function calls in TestPreprocessor to make it more meaningful.
662. Added missing dependency in test preprocessors.
673. in AliShuttle.cxx: processing time and memory consumption info on a single line.
68
886d60e6 69Revision 1.34 2007/04/04 10:33:36 jgrosseo
701) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
71In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
72
732) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
74
753) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
76
774) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
78
795) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
80If you always need DCS data (like before), you do not need to implement it.
81
826) The run type has been added to the monitoring page
83
9827400b 84Revision 1.33 2007/04/03 13:56:01 acolla
85Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
86run type.
87
3301427a 88Revision 1.32 2007/02/28 10:41:56 acolla
89Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
90AliPreprocessor::GetRunType() function.
91Added some ldap definition files.
92
d386d623 93Revision 1.30 2007/02/13 11:23:21 acolla
94Moved getters and setters of Shuttle's main OCDB/Reference, local
95OCDB/Reference, temp and log folders to AliShuttleInterface
96
9d733021 97Revision 1.27 2007/01/30 17:52:42 jgrosseo
98adding monalisa monitoring
99
e7f62f16 100Revision 1.26 2007/01/23 19:20:03 acolla
101Removed old ldif files, added TOF, MCH ldif files. Added some options in
102AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
103SetShuttleLogDir
104
36c99a6a 105Revision 1.25 2007/01/15 19:13:52 acolla
106Moved some AliInfo to AliDebug in SendMail function
107
fc5a4708 108Revision 1.21 2006/12/07 08:51:26 jgrosseo
109update (alberto):
110table, db names in ldap configuration
111added GRP preprocessor
112DCS data can also be retrieved by data point
113
2c15234c 114Revision 1.20 2006/11/16 16:16:48 jgrosseo
115introducing strict run ordering flag
116removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
117
be48e3ea 118Revision 1.19 2006/11/06 14:23:04 jgrosseo
119major update (Alberto)
120o) reading of run parameters from the logbook
121o) online offline naming conversion
122o) standalone DCSclient package
123
eba76848 124Revision 1.18 2006/10/20 15:22:59 jgrosseo
125o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
126o) Merging Collect, CollectAll, CollectNew function
127o) Removing implementation of empty copy constructors (declaration still there!)
128
cb343cfd 129Revision 1.17 2006/10/05 16:20:55 jgrosseo
130adapting to new CDB classes
131
6ec0e06c 132Revision 1.16 2006/10/05 15:46:26 jgrosseo
133applying to the new interface
134
481441a2 135Revision 1.15 2006/10/02 16:38:39 jgrosseo
136update (alberto):
137fixed memory leaks
138storing of objects that failed to be stored to the grid before
139interfacing of shuttle status table in daq system
140
2bb7b766 141Revision 1.14 2006/08/29 09:16:05 jgrosseo
142small update
143
85a80aa9 144Revision 1.13 2006/08/15 10:50:00 jgrosseo
145effc++ corrections (alberto)
146
4f0ab988 147Revision 1.12 2006/08/08 14:19:29 jgrosseo
148Update to shuttle classes (Alberto)
149
150- Possibility to set the full object's path in the Preprocessor's and
151Shuttle's Store functions
152- Possibility to extend the object's run validity in the same classes
153("startValidity" and "validityInfinite" parameters)
154- Implementation of the StoreReferenceData function to store reference
155data in a dedicated CDB storage.
156
84090f85 157Revision 1.11 2006/07/21 07:37:20 jgrosseo
158last run is stored after each run
159
7bfb2090 160Revision 1.10 2006/07/20 09:54:40 jgrosseo
161introducing status management: The processing per subdetector is divided into several steps,
162after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
163can keep track of the number of failures and skips further processing after a certain threshold is
164exceeded. These thresholds can be configured in LDAP.
165
5164a766 166Revision 1.9 2006/07/19 10:09:55 jgrosseo
167new configuration, accesst to DAQ FES (Alberto)
168
57f50b3c 169Revision 1.8 2006/07/11 12:44:36 jgrosseo
170adding parameters for extended validity range of data produced by preprocessor
171
17111222 172Revision 1.7 2006/07/10 14:37:09 jgrosseo
173small fix + todo comment
174
e090413b 175Revision 1.6 2006/07/10 13:01:41 jgrosseo
176enhanced storing of last sucessfully processed run (alberto)
177
a7160fe9 178Revision 1.5 2006/07/04 14:59:57 jgrosseo
179revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
180
45a493ce 181Revision 1.4 2006/06/12 09:11:16 jgrosseo
182coding conventions (Alberto)
183
58bc3020 184Revision 1.3 2006/06/06 14:26:40 jgrosseo
185o) removed files that were moved to STEER
186o) shuttle updated to follow the new interface (Alberto)
187
b948db8d 188Revision 1.2 2006/03/07 07:52:34 hristov
189New version (B.Yordanov)
190
d477ad88 191Revision 1.6 2005/11/19 17:19:14 byordano
192RetrieveDATEEntries and RetrieveConditionsData added
193
194Revision 1.5 2005/11/19 11:09:27 byordano
195AliShuttle declaration added
196
197Revision 1.4 2005/11/17 17:47:34 byordano
198TList changed to TObjArray
199
200Revision 1.3 2005/11/17 14:43:23 byordano
201import to local CVS
202
203Revision 1.1.1.1 2005/10/28 07:33:58 hristov
204Initial import as subdirectory in AliRoot
205
73abe331 206Revision 1.2 2005/09/13 08:41:15 byordano
207default startTime endTime added
208
209Revision 1.4 2005/08/30 09:13:02 byordano
210some docs added
211
212Revision 1.3 2005/08/29 21:15:47 byordano
213some docs added
214
215*/
216
217//
218// This class is the main manager for AliShuttle.
219// It organizes the data retrieval from DCS and call the
b948db8d 220// interface methods of AliPreprocessor.
73abe331 221// For every detector in AliShuttleConfgi (see AliShuttleConfig),
222// data for its set of aliases is retrieved. If there is registered
b948db8d 223// AliPreprocessor for this detector then it will be used
224// accroding to the schema (see AliPreprocessor).
225// If there isn't registered AliPreprocessor than the retrieved
73abe331 226// data is stored automatically to the undelying AliCDBStorage.
227// For detSpec is used the alias name.
228//
229
230#include "AliShuttle.h"
231
232#include "AliCDBManager.h"
233#include "AliCDBStorage.h"
234#include "AliCDBId.h"
84090f85 235#include "AliCDBRunRange.h"
236#include "AliCDBPath.h"
5164a766 237#include "AliCDBEntry.h"
73abe331 238#include "AliShuttleConfig.h"
eba76848 239#include "DCSClient/AliDCSClient.h"
73abe331 240#include "AliLog.h"
b948db8d 241#include "AliPreprocessor.h"
5164a766 242#include "AliShuttleStatus.h"
2bb7b766 243#include "AliShuttleLogbookEntry.h"
73abe331 244
57f50b3c 245#include <TSystem.h>
58bc3020 246#include <TObject.h>
b948db8d 247#include <TString.h>
57f50b3c 248#include <TTimeStamp.h>
73abe331 249#include <TObjString.h>
57f50b3c 250#include <TSQLServer.h>
251#include <TSQLResult.h>
252#include <TSQLRow.h>
cb343cfd 253#include <TMutex.h>
9827400b 254#include <TSystemDirectory.h>
255#include <TSystemFile.h>
256#include <TFileMerger.h>
257#include <TGrid.h>
258#include <TGridResult.h>
73abe331 259
e7f62f16 260#include <TMonaLisaWriter.h>
261
5164a766 262#include <fstream>
263
cb343cfd 264#include <sys/types.h>
265#include <sys/wait.h>
266
73abe331 267ClassImp(AliShuttle)
268
b948db8d 269//______________________________________________________________________________________________
270AliShuttle::AliShuttle(const AliShuttleConfig* config,
271 UInt_t timeout, Int_t retries):
4f0ab988 272fConfig(config),
273fTimeout(timeout), fRetries(retries),
274fPreprocessorMap(),
2bb7b766 275fLogbookEntry(0),
eba76848 276fCurrentDetector(),
85a80aa9 277fStatusEntry(0),
cb343cfd 278fMonitoringMutex(0),
eba76848 279fLastActionTime(0),
e7f62f16 280fLastAction(),
9827400b 281fMonaLisa(0),
282fTestMode(kNone),
ffa29e93 283fReadTestMode(kFALSE),
284fOutputRedirected(kFALSE)
73abe331 285{
286 //
287 // config: AliShuttleConfig used
73abe331 288 // timeout: timeout used for AliDCSClient connection
289 // retries: the number of retries in case of connection error.
290 //
291
57f50b3c 292 if (!fConfig->IsValid()) AliFatal("********** !!!!! Invalid configuration !!!!! **********");
be48e3ea 293 for(int iSys=0;iSys<4;iSys++) {
57f50b3c 294 fServer[iSys]=0;
be48e3ea 295 if (iSys < 3)
2c15234c 296 fFXSlist[iSys].SetOwner(kTRUE);
57f50b3c 297 }
2bb7b766 298 fPreprocessorMap.SetOwner(kTRUE);
be48e3ea 299
300 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
301 fFirstUnprocessed[iDet] = kFALSE;
302
cb343cfd 303 fMonitoringMutex = new TMutex();
58bc3020 304}
305
b948db8d 306//______________________________________________________________________________________________
57f50b3c 307AliShuttle::~AliShuttle()
58bc3020 308{
9827400b 309 //
310 // destructor
311 //
58bc3020 312
b948db8d 313 fPreprocessorMap.DeleteAll();
be48e3ea 314 for(int iSys=0;iSys<4;iSys++)
57f50b3c 315 if(fServer[iSys]) {
316 fServer[iSys]->Close();
317 delete fServer[iSys];
eba76848 318 fServer[iSys] = 0;
57f50b3c 319 }
2bb7b766 320
321 if (fStatusEntry){
322 delete fStatusEntry;
323 fStatusEntry = 0;
324 }
cb343cfd 325
326 if (fMonitoringMutex)
327 {
328 delete fMonitoringMutex;
329 fMonitoringMutex = 0;
330 }
73abe331 331}
332
b948db8d 333//______________________________________________________________________________________________
57f50b3c 334void AliShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
58bc3020 335{
73abe331 336 //
b948db8d 337 // Registers new AliPreprocessor.
73abe331 338 // It uses GetName() for indentificator of the pre processor.
339 // The pre processor is registered it there isn't any other
340 // with the same identificator (GetName()).
341 //
342
eba76848 343 const char* detName = preprocessor->GetName();
344 if(GetDetPos(detName) < 0)
345 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
346
347 if (fPreprocessorMap.GetValue(detName)) {
348 AliWarning(Form("AliPreprocessor %s is already registered!", detName));
73abe331 349 return;
350 }
351
eba76848 352 fPreprocessorMap.Add(new TObjString(detName), preprocessor);
73abe331 353}
b948db8d 354//______________________________________________________________________________________________
3301427a 355Bool_t AliShuttle::Store(const AliCDBPath& path, TObject* object,
84090f85 356 AliCDBMetaData* metaData, Int_t validityStart, Bool_t validityInfinite)
73abe331 357{
9827400b 358 // Stores a CDB object in the storage for offline reconstruction. Objects that are not needed for
359 // offline reconstruction, but should be stored anyway (e.g. for debugging) should NOT be stored
360 // using this function. Use StoreReferenceData instead!
361 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
362 // finishes the data are transferred to the main storage (Grid).
b948db8d 363
3301427a 364 return StoreLocally(fgkLocalCDB, path, object, metaData, validityStart, validityInfinite);
84090f85 365}
366
367//______________________________________________________________________________________________
3301427a 368Bool_t AliShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 369{
9827400b 370 // Stores a CDB object in the storage for reference data. This objects will not be available during
371 // offline reconstrunction. Use this function for reference data only!
372 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
373 // finishes the data are transferred to the main storage (Grid).
85a80aa9 374
3301427a 375 return StoreLocally(fgkLocalRefStorage, path, object, metaData);
85a80aa9 376}
377
378//______________________________________________________________________________________________
3301427a 379Bool_t AliShuttle::StoreLocally(const TString& localUri,
85a80aa9 380 const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
381 Int_t validityStart, Bool_t validityInfinite)
382{
9827400b 383 // Store object temporarily in local storage. Parameters are passed by Store and StoreReferenceData functions.
384 // when the preprocessor finishes the data are transferred to the main storage (Grid).
385 // The parameters are:
386 // 1) Uri of the backup storage (Local)
387 // 2) the object's path.
388 // 3) the object to be stored
389 // 4) the metaData to be associated with the object
390 // 5) the validity start run number w.r.t. the current run,
391 // if the data is valid only for this run leave the default 0
392 // 6) specifies if the calibration data is valid for infinity (this means until updated),
393 // typical for calibration runs, the default is kFALSE
394 //
395 // returns 0 if fail, 1 otherwise
84090f85 396
9827400b 397 if (fTestMode & kErrorStorage)
398 {
399 Log(fCurrentDetector, "StoreLocally - In TESTMODE - Simulating error while storing locally");
400 return kFALSE;
401 }
402
3301427a 403 const char* cdbType = (localUri == fgkLocalCDB) ? "CDB" : "Reference";
2bb7b766 404
85a80aa9 405 Int_t firstRun = GetCurrentRun() - validityStart;
84090f85 406 if(firstRun < 0) {
9827400b 407 AliWarning("First valid run happens to be less than 0! Setting it to 0.");
84090f85 408 firstRun=0;
409 }
410
411 Int_t lastRun = -1;
412 if(validityInfinite) {
413 lastRun = AliCDBRunRange::Infinity();
414 } else {
415 lastRun = GetCurrentRun();
416 }
417
3301427a 418 // Version is set to current run, it will be used later to transfer data to Grid
419 AliCDBId id(path, firstRun, lastRun, GetCurrentRun(), -1);
2bb7b766 420
421 if(! dynamic_cast<TObjString*> (metaData->GetProperty("RunUsed(TObjString)"))){
422 TObjString runUsed = Form("%d", GetCurrentRun());
9e080f92 423 metaData->SetProperty("RunUsed(TObjString)", runUsed.Clone());
2bb7b766 424 }
84090f85 425
3301427a 426 Bool_t result = kFALSE;
84090f85 427
3301427a 428 if (!(AliCDBManager::Instance()->GetStorage(localUri))) {
429 Log("SHUTTLE", Form("StoreLocally - Cannot activate local %s storage", cdbType));
84090f85 430 } else {
3301427a 431 result = AliCDBManager::Instance()->GetStorage(localUri)
84090f85 432 ->Put(object, id, metaData);
433 }
434
435 if(!result) {
436
9827400b 437 Log(fCurrentDetector, Form("StoreLocally - Can't store object <%s>!", id.ToString().Data()));
3301427a 438 }
2bb7b766 439
3301427a 440 return result;
441}
84090f85 442
3301427a 443//______________________________________________________________________________________________
444Bool_t AliShuttle::StoreOCDB()
445{
9827400b 446 //
447 // Called when preprocessor ends successfully or when previous storage attempt failed (kStoreError status)
448 // Calls underlying StoreOCDB(const char*) function twice, for OCDB and Reference storage.
449 // Then calls StoreRefFilesToGrid to store reference files.
450 //
451
452 if (fTestMode & kErrorGrid)
453 {
454 Log("SHUTTLE", "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
455 Log(fCurrentDetector, "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
456 return kFALSE;
457 }
458
2d9019b4 459 Log("SHUTTLE","Storing OCDB data ...");
86aa42c3 460 Bool_t resultCDB = StoreOCDB(fgkMainCDB);
461
2d9019b4 462 Log("SHUTTLE","Storing reference data ...");
3301427a 463 Bool_t resultRef = StoreOCDB(fgkMainRefStorage);
9827400b 464
2d9019b4 465 Log("SHUTTLE","Storing reference files ...");
9827400b 466 Bool_t resultRefFiles = StoreRefFilesToGrid();
467
468 return resultCDB && resultRef && resultRefFiles;
3301427a 469}
470
471//______________________________________________________________________________________________
472Bool_t AliShuttle::StoreOCDB(const TString& gridURI)
473{
474 //
475 // Called by StoreOCDB(), performs actual storage to the main OCDB and reference storages (Grid)
476 //
477
478 TObjArray* gridIds=0;
479
480 Bool_t result = kTRUE;
481
482 const char* type = 0;
483 TString localURI;
484 if(gridURI == fgkMainCDB) {
485 type = "OCDB";
486 localURI = fgkLocalCDB;
487 } else if(gridURI == fgkMainRefStorage) {
488 type = "reference";
489 localURI = fgkLocalRefStorage;
490 } else {
491 AliError(Form("Invalid storage URI: %s", gridURI.Data()));
492 return kFALSE;
493 }
494
495 AliCDBManager* man = AliCDBManager::Instance();
496
497 AliCDBStorage *gridSto = man->GetStorage(gridURI);
498 if(!gridSto) {
499 Log("SHUTTLE",
500 Form("StoreOCDB - cannot activate main %s storage", type));
501 return kFALSE;
502 }
503
504 gridIds = gridSto->GetQueryCDBList();
505
506 // get objects previously stored in local CDB
507 AliCDBStorage *localSto = man->GetStorage(localURI);
508 if(!localSto) {
509 Log("SHUTTLE",
510 Form("StoreOCDB - cannot activate local %s storage", type));
511 return kFALSE;
512 }
513 AliCDBPath aPath(GetOfflineDetName(fCurrentDetector.Data()),"*","*");
514 // Local objects were stored with current run as Grid version!
515 TList* localEntries = localSto->GetAll(aPath.GetPath(), GetCurrentRun(), GetCurrentRun());
516 localEntries->SetOwner(1);
517
518 // loop on local stored objects
519 TIter localIter(localEntries);
520 AliCDBEntry *aLocEntry = 0;
521 while((aLocEntry = dynamic_cast<AliCDBEntry*> (localIter.Next()))){
522 aLocEntry->SetOwner(1);
523 AliCDBId aLocId = aLocEntry->GetId();
524 aLocEntry->SetVersion(-1);
525 aLocEntry->SetSubVersion(-1);
526
527 // If local object is valid up to infinity we store it only if it is
528 // the first unprocessed run!
529 if (aLocId.GetLastRun() == AliCDBRunRange::Infinity() &&
530 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
531 {
532 Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
533 "there are previous unprocessed runs!",
534 fCurrentDetector.Data(), aLocId.GetPath().Data()));
535 continue;
536 }
537
538 // loop on Grid valid Id's
539 Bool_t store = kTRUE;
540 TIter gridIter(gridIds);
541 AliCDBId* aGridId = 0;
542 while((aGridId = dynamic_cast<AliCDBId*> (gridIter.Next()))){
543 if(aGridId->GetPath() != aLocId.GetPath()) continue;
544 // skip all objects valid up to infinity
545 if(aGridId->GetLastRun() == AliCDBRunRange::Infinity()) continue;
546 // if we get here, it means there's already some more recent object stored on Grid!
547 store = kFALSE;
548 break;
549 }
550
551 // If we get here, the file can be stored!
552 Bool_t storeOk = gridSto->Put(aLocEntry);
553 if(!store || storeOk){
554
555 if (!store)
556 {
557 Log(fCurrentDetector.Data(),
558 Form("StoreOCDB - A more recent object already exists in %s storage: <%s>",
559 type, aGridId->ToString().Data()));
560 } else {
561 Log("SHUTTLE",
562 Form("StoreOCDB - Object <%s> successfully put into %s storage",
563 aLocId.ToString().Data(), type));
2d9019b4 564 Log(fCurrentDetector.Data(),
565 Form("StoreOCDB - Object <%s> successfully put into %s storage",
566 aLocId.ToString().Data(), type));
3301427a 567 }
84090f85 568
3301427a 569 // removing local filename...
570 TString filename;
571 localSto->IdToFilename(aLocId, filename);
572 AliInfo(Form("Removing local file %s", filename.Data()));
573 RemoveFile(filename.Data());
574 continue;
575 } else {
576 Log("SHUTTLE",
577 Form("StoreOCDB - Grid %s storage of object <%s> failed",
578 type, aLocId.ToString().Data()));
2d9019b4 579 Log(fCurrentDetector.Data(),
580 Form("StoreOCDB - Grid %s storage of object <%s> failed",
581 type, aLocId.ToString().Data()));
3301427a 582 result = kFALSE;
b948db8d 583 }
584 }
3301427a 585 localEntries->Clear();
2bb7b766 586
b948db8d 587 return result;
3301427a 588}
589
546242fb 590//______________________________________________________________________________________________
591Bool_t AliShuttle::CleanReferenceStorage(const char* detector)
592{
2d9019b4 593 // clears the directory used to store reference files of a given subdetector
546242fb 594
595 AliCDBManager* man = AliCDBManager::Instance();
596 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
2d9019b4 597 TString localBaseFolder = sto->GetBaseFolder();
598
599 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
600
601 Log("SHUTTLE", Form("Cleaning %s", targetDir.Data()));
602
603 TString begin;
604 begin.Form("%d_", GetCurrentRun());
605
606 TSystemDirectory* baseDir = new TSystemDirectory("/", targetDir);
607 if (!baseDir)
608 return kTRUE;
609
610 TList* dirList = baseDir->GetListOfFiles();
611 delete baseDir;
612
613 if (!dirList) return kTRUE;
614
615 if (dirList->GetEntries() < 3)
616 {
617 delete dirList;
618 return kTRUE;
619 }
620
621 Int_t nDirs = 0, nDel = 0;
622 TIter dirIter(dirList);
623 TSystemFile* entry = 0;
546242fb 624
2d9019b4 625 Bool_t success = kTRUE;
546242fb 626
2d9019b4 627 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
628 {
629 if (entry->IsDirectory())
630 continue;
631
632 TString fileName(entry->GetName());
633 if (!fileName.BeginsWith(begin))
634 continue;
635
636 nDirs++;
637
638 // delete file
639 Int_t result = gSystem->Unlink(fileName.Data());
640
641 if (result)
642 {
643 Log("SHUTTLE", Form("Could not delete file %s!", fileName.Data()));
644 success = kFALSE;
645 } else {
646 nDel++;
647 }
648 }
649
650 if(nDirs > 0)
651 Log("SHUTTLE", Form("CleanReferenceStorage - %d (over %d) reference files in folder %s were deleted.",
652 nDel, nDirs, targetDir.Data()));
653
654
655 delete dirList;
656 return success;
657
658
659
660
661
546242fb 662
663 Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
664 if (result == 0)
665 {
666 // delete directory
667 result = gSystem->Exec(Form("rm -r %s", targetDir.Data()));
668 if (result != 0)
669 {
670 Log("SHUTTLE", Form("StoreReferenceFile - Could not clear directory %s", targetDir.Data()));
671 return kFALSE;
672 }
673 }
674
675 result = gSystem->mkdir(targetDir, kTRUE);
676 if (result != 0)
677 {
678 Log("SHUTTLE", Form("StoreReferenceFile - Error creating base directory %s", targetDir.Data()));
679 return kFALSE;
680 }
681
682 return kTRUE;
683}
684
9827400b 685//______________________________________________________________________________________________
686Bool_t AliShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
687{
688 //
3c2a21c8 689 // Stores reference file directly (without opening it). This function stores the file locally.
9827400b 690 //
3c2a21c8 691 // The file is stored under the following location:
692 // <base folder of local reference storage>/<DET>/<RUN#>_<gridFileName>
693 // where <gridFileName> is the second parameter given to the function
694 //
9827400b 695
696 if (fTestMode & kErrorStorage)
697 {
698 Log(fCurrentDetector, "StoreReferenceFile - In TESTMODE - Simulating error while storing locally");
699 return kFALSE;
700 }
701
702 AliCDBManager* man = AliCDBManager::Instance();
703 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
704
705 TString localBaseFolder = sto->GetBaseFolder();
706
2d9019b4 707 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
9827400b 708
2d9019b4 709 //try to open folder, if does not exist
710 void* dir = gSystem->OpenDirectory(targetDir.Data());
711 if (dir == NULL) {
712 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
713 Log("SHUTTLE", Form("Can't open directory <%s>", targetDir.Data()));
714 return kFALSE;
715 }
716
717 } else {
718 gSystem->FreeDirectory(dir);
719 }
720
9827400b 721 TString target;
722 target.Form("%s/%d_%s", targetDir.Data(), GetCurrentRun(), gridFileName);
723
546242fb 724 Int_t result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
9827400b 725 if (result)
726 {
546242fb 727 Log("SHUTTLE", Form("StoreReferenceFile - %s does not exist", localFile));
728 return kFALSE;
9827400b 729 }
546242fb 730
9827400b 731 result = gSystem->CopyFile(localFile, target);
732
733 if (result == 0)
734 {
2d9019b4 735 Log("SHUTTLE", Form("StoreReferenceFile - File %s stored locally to %s", localFile, target.Data()));
9827400b 736 return kTRUE;
737 }
738 else
739 {
2d9019b4 740 Log("SHUTTLE", Form("StoreReferenceFile - Could not store file %s to %s!. Error code = %d",
546242fb 741 localFile, target.Data(), result));
9827400b 742 return kFALSE;
743 }
744}
745
746//______________________________________________________________________________________________
747Bool_t AliShuttle::StoreRefFilesToGrid()
748{
749 //
750 // Transfers the reference file to the Grid.
9827400b 751 //
86aa42c3 752 // The files are stored under the following location:
3c2a21c8 753 // <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
86aa42c3 754 //
9827400b 755
756 AliCDBManager* man = AliCDBManager::Instance();
757 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
758 if (!sto)
759 return kFALSE;
760 TString localBaseFolder = sto->GetBaseFolder();
761
2d9019b4 762 TString dir = GetRefFilePrefix(localBaseFolder.Data(), fCurrentDetector.Data());
763
9827400b 764 AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
765 if (!gridSto)
766 return kFALSE;
2d9019b4 767
9827400b 768 TString gridBaseFolder = gridSto->GetBaseFolder();
2d9019b4 769
770 TString alienDir = GetRefFilePrefix(gridBaseFolder.Data(), fCurrentDetector.Data());
9827400b 771
9827400b 772 TString begin;
773 begin.Form("%d_", GetCurrentRun());
774
775 TSystemDirectory* baseDir = new TSystemDirectory("/", dir);
3d8bc902 776 if (!baseDir)
777 return kTRUE;
778
2d9019b4 779 TList* dirList = baseDir->GetListOfFiles();
780 delete baseDir;
781
782 if (!dirList) return kTRUE;
783
784 if (dirList->GetEntries() < 3)
3d8bc902 785 {
2d9019b4 786 delete dirList;
9827400b 787 return kTRUE;
3d8bc902 788 }
2d9019b4 789
546242fb 790 if (!gGrid)
791 {
792 Log("SHUTTLE", "Connection to Grid failed: Cannot continue!");
2d9019b4 793 delete dirList;
546242fb 794 return kFALSE;
795 }
796
2d9019b4 797 Int_t nDirs = 0, nTransfer = 0;
798 TIter dirIter(dirList);
799 TSystemFile* entry = 0;
800
9827400b 801 Bool_t success = kTRUE;
3d8bc902 802 Bool_t first = kTRUE;
9827400b 803
2d9019b4 804 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
805 {
9827400b 806 if (entry->IsDirectory())
807 continue;
808
809 TString fileName(entry->GetName());
810 if (!fileName.BeginsWith(begin))
811 continue;
812
2d9019b4 813 nDirs++;
814
3d8bc902 815 if (first)
816 {
817 first = kFALSE;
818 // check that DET folder exists, otherwise create it
819 TGridResult* result = gGrid->Ls(alienDir.Data(), "a");
820
821 if (!result)
2d9019b4 822 {
823 delete dirList;
3d8bc902 824 return kFALSE;
2d9019b4 825 }
3d8bc902 826
546242fb 827 if (!result->GetFileName(1)) // TODO: It looks like element 0 is always 0!!
3d8bc902 828 {
829 if (!gGrid->Mkdir(alienDir.Data(),"",0))
830 {
831 Log("SHUTTLE", Form("StoreRefFilesToGrid - Cannot create directory %s",
832 alienDir.Data()));
2d9019b4 833 delete dirList;
3d8bc902 834 return kFALSE;
546242fb 835 } else {
836 Log("SHUTTLE",Form("Folder %s created", alienDir.Data()));
3d8bc902 837 }
838
546242fb 839 } else {
840 Log("SHUTTLE",Form("Folder %s found", alienDir.Data()));
3d8bc902 841 }
842 }
843
9827400b 844 TString fullLocalPath;
845 fullLocalPath.Form("%s/%s", dir.Data(), fileName.Data());
846
847 TString fullGridPath;
848 fullGridPath.Form("alien://%s/%s", alienDir.Data(), fileName.Data());
849
9827400b 850 TFileMerger fileMerger;
851 Bool_t result = fileMerger.Cp(fullLocalPath, fullGridPath);
852
853 if (result)
854 {
2d9019b4 855 Log("SHUTTLE", Form("StoreRefFilesToGrid - Copying local file %s to %s succeeded!", fullLocalPath.Data(), fullGridPath.Data()));
9827400b 856 RemoveFile(fullLocalPath);
2d9019b4 857 nTransfer++;
9827400b 858 }
859 else
860 {
2d9019b4 861 Log("SHUTTLE", Form("StoreRefFilesToGrid - Copying local file %s to %s FAILED!", fullLocalPath.Data(), fullGridPath.Data()));
9827400b 862 success = kFALSE;
863 }
864 }
2d9019b4 865
866 Log("SHUTTLE", Form("StoreRefFilesToGrid - %d (over %d) reference files in folder %s copied to Grid.", nTransfer, nDirs, dir.Data()));
867
868
869 delete dirList;
9827400b 870 return success;
871}
872
2d9019b4 873//______________________________________________________________________________________________
874const char* AliShuttle::GetRefFilePrefix(const char* base, const char* detector)
875{
876 //
877 // Get folder name of reference files
878 //
879
880 TString offDetStr(GetOfflineDetName(detector));
881 TString dir;
882 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
883 {
884 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
885 } else {
886 dir.Form("%s/%s", base, offDetStr.Data());
887 }
888
889 return dir.Data();
890
891
892}
3301427a 893//______________________________________________________________________________________________
894void AliShuttle::CleanLocalStorage(const TString& uri)
895{
9827400b 896 //
897 // Called in case the preprocessor is declared failed. Remove remaining objects from the local storages.
898 //
3301427a 899
900 const char* type = 0;
901 if(uri == fgkLocalCDB) {
902 type = "OCDB";
903 } else if(uri == fgkLocalRefStorage) {
546242fb 904 type = "Reference";
3301427a 905 } else {
906 AliError(Form("Invalid storage URI: %s", uri.Data()));
907 return;
908 }
909
910 AliCDBManager* man = AliCDBManager::Instance();
b948db8d 911
3301427a 912 // open local storage
913 AliCDBStorage *localSto = man->GetStorage(uri);
914 if(!localSto) {
915 Log("SHUTTLE",
916 Form("CleanLocalStorage - cannot activate local %s storage", type));
917 return;
918 }
919
920 TString filename(Form("%s/%s/*/Run*_v%d_s*.root",
546242fb 921 localSto->GetBaseFolder().Data(), GetOfflineDetName(fCurrentDetector.Data()), GetCurrentRun()));
3301427a 922
923 AliInfo(Form("filename = %s", filename.Data()));
924
925 AliInfo(Form("Removing remaining local files from run %d and detector %s ...",
926 GetCurrentRun(), fCurrentDetector.Data()));
927
928 RemoveFile(filename.Data());
929
930}
931
932//______________________________________________________________________________________________
933void AliShuttle::RemoveFile(const char* filename)
934{
9827400b 935 //
936 // removes local file
937 //
3301427a 938
939 TString command(Form("rm -f %s", filename));
940
941 Int_t result = gSystem->Exec(command.Data());
942 if(result != 0)
943 {
944 Log("SHUTTLE", Form("RemoveFile - %s: Cannot remove file %s!",
945 fCurrentDetector.Data(), filename));
946 }
73abe331 947}
948
b948db8d 949//______________________________________________________________________________________________
5164a766 950AliShuttleStatus* AliShuttle::ReadShuttleStatus()
951{
9827400b 952 //
953 // Reads the AliShuttleStatus from the CDB
954 //
5164a766 955
2bb7b766 956 if (fStatusEntry){
957 delete fStatusEntry;
958 fStatusEntry = 0;
959 }
5164a766 960
10a5a932 961 fStatusEntry = AliCDBManager::Instance()->GetStorage(GetLocalCDB())
2bb7b766 962 ->Get(Form("/SHUTTLE/STATUS/%s", fCurrentDetector.Data()), GetCurrentRun());
5164a766 963
2bb7b766 964 if (!fStatusEntry) return 0;
965 fStatusEntry->SetOwner(1);
5164a766 966
2bb7b766 967 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
968 if (!status) {
969 AliError("Invalid object stored to CDB!");
970 return 0;
971 }
5164a766 972
2bb7b766 973 return status;
5164a766 974}
975
976//______________________________________________________________________________________________
7bfb2090 977Bool_t AliShuttle::WriteShuttleStatus(AliShuttleStatus* status)
5164a766 978{
9827400b 979 //
980 // writes the status for one subdetector
981 //
2bb7b766 982
983 if (fStatusEntry){
984 delete fStatusEntry;
985 fStatusEntry = 0;
986 }
5164a766 987
2bb7b766 988 Int_t run = GetCurrentRun();
5164a766 989
2bb7b766 990 AliCDBId id(AliCDBPath("SHUTTLE", "STATUS", fCurrentDetector), run, run);
5164a766 991
2bb7b766 992 fStatusEntry = new AliCDBEntry(status, id, new AliCDBMetaData);
993 fStatusEntry->SetOwner(1);
5164a766 994
2bb7b766 995 UInt_t result = AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
7bfb2090 996
2bb7b766 997 if (!result) {
3301427a 998 Log("SHUTTLE", Form("WriteShuttleStatus - Failed for %s, run %d",
999 fCurrentDetector.Data(), run));
2bb7b766 1000 return kFALSE;
1001 }
e7f62f16 1002
1003 SendMLInfo();
7bfb2090 1004
2bb7b766 1005 return kTRUE;
5164a766 1006}
1007
1008//______________________________________________________________________________________________
1009void AliShuttle::UpdateShuttleStatus(AliShuttleStatus::Status newStatus, Bool_t increaseCount)
1010{
9827400b 1011 //
1012 // changes the AliShuttleStatus for the given detector and run to the given status
1013 //
5164a766 1014
2bb7b766 1015 if (!fStatusEntry){
1016 AliError("UNEXPECTED: fStatusEntry empty");
1017 return;
1018 }
5164a766 1019
2bb7b766 1020 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
5164a766 1021
2bb7b766 1022 if (!status){
3301427a 1023 Log("SHUTTLE", "UNEXPECTED: status could not be read from current CDB entry");
2bb7b766 1024 return;
1025 }
5164a766 1026
2c15234c 1027 TString actionStr = Form("UpdateShuttleStatus - %s: Changing state from %s to %s",
eba76848 1028 fCurrentDetector.Data(),
36c99a6a 1029 status->GetStatusName(),
eba76848 1030 status->GetStatusName(newStatus));
cb343cfd 1031 Log("SHUTTLE", actionStr);
1032 SetLastAction(actionStr);
5164a766 1033
2bb7b766 1034 status->SetStatus(newStatus);
1035 if (increaseCount) status->IncreaseCount();
5164a766 1036
2bb7b766 1037 AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
e7f62f16 1038
1039 SendMLInfo();
5164a766 1040}
e7f62f16 1041
1042//______________________________________________________________________________________________
1043void AliShuttle::SendMLInfo()
1044{
1045 //
1046 // sends ML information about the current status of the current detector being processed
1047 //
1048
1049 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1050
1051 if (!status){
3301427a 1052 Log("SHUTTLE", "SendMLInfo - UNEXPECTED: status could not be read from current CDB entry");
e7f62f16 1053 return;
1054 }
1055
1056 TMonaLisaText mlStatus(Form("%s_status", fCurrentDetector.Data()), status->GetStatusName());
1057 TMonaLisaValue mlRetryCount(Form("%s_count", fCurrentDetector.Data()), status->GetCount());
1058
1059 TList mlList;
1060 mlList.Add(&mlStatus);
1061 mlList.Add(&mlRetryCount);
1062
1063 fMonaLisa->SendParameters(&mlList);
1064}
1065
5164a766 1066//______________________________________________________________________________________________
1067Bool_t AliShuttle::ContinueProcessing()
1068{
9827400b 1069 // this function reads the AliShuttleStatus information from CDB and
1070 // checks if the processing should be continued
1071 // if yes it returns kTRUE and updates the AliShuttleStatus with nextStatus
2bb7b766 1072
57c1a579 1073 if (!fConfig->HostProcessDetector(fCurrentDetector)) return kFALSE;
1074
1075 AliPreprocessor* aPreprocessor =
1076 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1077 if (!aPreprocessor)
1078 {
1079 AliInfo(Form("%s: no preprocessor registered", fCurrentDetector.Data()));
1080 return kFALSE;
1081 }
1082
2bb7b766 1083 AliShuttleLogbookEntry::Status entryStatus =
eba76848 1084 fLogbookEntry->GetDetectorStatus(fCurrentDetector);
2bb7b766 1085
1086 if(entryStatus != AliShuttleLogbookEntry::kUnprocessed) {
9e080f92 1087 AliInfo(Form("ContinueProcessing - %s is %s",
2bb7b766 1088 fCurrentDetector.Data(),
1089 fLogbookEntry->GetDetectorStatusName(entryStatus)));
1090 return kFALSE;
1091 }
1092
1093 // if we get here, according to Shuttle logbook subdetector is in UNPROCESSED state
be48e3ea 1094
1095 // check if current run is first unprocessed run for current detector
1096 if (fConfig->StrictRunOrder(fCurrentDetector) &&
1097 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
1098 {
86aa42c3 1099 if (fTestMode == kNone)
1100 {
1101 Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering but this is not the first unprocessed run!"));
1102 return kFALSE;
1103 }
1104 else
1105 {
1106 Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - Although %s requires strict run ordering and this is not the first unprocessed run, the SHUTTLE continues"));
1107 }
be48e3ea 1108 }
1109
2bb7b766 1110 AliShuttleStatus* status = ReadShuttleStatus();
1111 if (!status) {
1112 // first time
1113 Log("SHUTTLE", Form("ContinueProcessing - %s: Processing first time",
1114 fCurrentDetector.Data()));
1115 status = new AliShuttleStatus(AliShuttleStatus::kStarted);
1116 return WriteShuttleStatus(status);
1117 }
1118
1119 // The following two cases shouldn't happen if Shuttle Logbook was correctly updated.
1120 // If it happens it may mean Logbook updating failed... let's do it now!
1121 if (status->GetStatus() == AliShuttleStatus::kDone ||
1122 status->GetStatus() == AliShuttleStatus::kFailed){
1123 Log("SHUTTLE", Form("ContinueProcessing - %s is already %s. Updating Shuttle Logbook",
1124 fCurrentDetector.Data(),
1125 status->GetStatusName(status->GetStatus())));
1126 UpdateShuttleLogbook(fCurrentDetector.Data(),
1127 status->GetStatusName(status->GetStatus()));
1128 return kFALSE;
1129 }
1130
3301427a 1131 if (status->GetStatus() == AliShuttleStatus::kStoreError) {
2bb7b766 1132 Log("SHUTTLE",
1133 Form("ContinueProcessing - %s: Grid storage of one or more objects failed. Trying again now",
1134 fCurrentDetector.Data()));
9827400b 1135 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1136 if (StoreOCDB()){
3301427a 1137 Log("SHUTTLE", Form("ContinueProcessing - %s: all objects successfully stored into main storage",
1138 fCurrentDetector.Data()));
2bb7b766 1139 UpdateShuttleStatus(AliShuttleStatus::kDone);
1140 UpdateShuttleLogbook(fCurrentDetector.Data(), "DONE");
1141 } else {
1142 Log("SHUTTLE",
1143 Form("ContinueProcessing - %s: Grid storage failed again",
1144 fCurrentDetector.Data()));
9827400b 1145 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
2bb7b766 1146 }
1147 return kFALSE;
1148 }
1149
1150 // if we get here, there is a restart
57c1a579 1151 Bool_t cont = kFALSE;
2bb7b766 1152
1153 // abort conditions
cb343cfd 1154 if (status->GetCount() >= fConfig->GetMaxRetries()) {
57c1a579 1155 Log("SHUTTLE", Form("ContinueProcessing - %s failed %d times in status %s - "
1156 "Updating Shuttle Logbook", fCurrentDetector.Data(),
2bb7b766 1157 status->GetCount(), status->GetStatusName()));
1158 UpdateShuttleLogbook(fCurrentDetector.Data(), "FAILED");
e7f62f16 1159 UpdateShuttleStatus(AliShuttleStatus::kFailed);
3301427a 1160
1161 // there may still be objects in local OCDB and reference storage
1162 // and FXS databases may be not updated: do it now!
9827400b 1163
1164 // TODO Currently disabled, we want to keep files in case of failure!
1165 // CleanLocalStorage(fgkLocalCDB);
1166 // CleanLocalStorage(fgkLocalRefStorage);
1167 // UpdateTableFailCase();
1168
1169 // Send mail to detector expert!
1170 AliInfo(Form("Sending mail to %s expert...", fCurrentDetector.Data()));
1171 if (!SendMail())
1172 Log("SHUTTLE", Form("ContinueProcessing - Could not send mail to %s expert",
1173 fCurrentDetector.Data()));
3301427a 1174
57c1a579 1175 } else {
1176 Log("SHUTTLE", Form("ContinueProcessing - %s: restarting. "
1177 "Aborted before with %s. Retry number %d.", fCurrentDetector.Data(),
1178 status->GetStatusName(), status->GetCount()));
9827400b 1179 Bool_t increaseCount = kTRUE;
1180 if (status->GetStatus() == AliShuttleStatus::kDCSError || status->GetStatus() == AliShuttleStatus::kDCSStarted)
1181 increaseCount = kFALSE;
1182 UpdateShuttleStatus(AliShuttleStatus::kStarted, increaseCount);
57c1a579 1183 cont = kTRUE;
2bb7b766 1184 }
1185
57c1a579 1186 return cont;
5164a766 1187}
1188
1189//______________________________________________________________________________________________
2bb7b766 1190Bool_t AliShuttle::Process(AliShuttleLogbookEntry* entry)
58bc3020 1191{
73abe331 1192 //
b948db8d 1193 // Makes data retrieval for all detectors in the configuration.
2bb7b766 1194 // entry: Shuttle logbook entry, contains run paramenters and status of detectors
1195 // (Unprocessed, Inactive, Failed or Done).
d477ad88 1196 // Returns kFALSE in case of error occured and kTRUE otherwise
73abe331 1197 //
1198
9827400b 1199 if (!entry) return kFALSE;
2bb7b766 1200
1201 fLogbookEntry = entry;
1202
9827400b 1203 AliInfo(Form("\n\n \t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^* \n",
1204 GetCurrentRun()));
2bb7b766 1205
e7f62f16 1206 // create ML instance that monitors this run
1207 fMonaLisa = new TMonaLisaWriter(Form("%d", GetCurrentRun()), "SHUTTLE", "aliendb1.cern.ch");
1208 // disable monitoring of other parameters that come e.g. from TFile
1209 gMonitoringWriter = 0;
2bb7b766 1210
e7f62f16 1211 // Send the information to ML
1212 TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
9827400b 1213 TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
e7f62f16 1214
1215 TList mlList;
1216 mlList.Add(&mlStatus);
9827400b 1217 mlList.Add(&mlRunType);
e7f62f16 1218
1219 fMonaLisa->SendParameters(&mlList);
3301427a 1220
9827400b 1221 if (fLogbookEntry->IsDone())
1222 {
1223 Log("SHUTTLE","Process - Shuttle is already DONE. Updating logbook");
1224 UpdateShuttleLogbook("shuttle_done");
1225 fLogbookEntry = 0;
1226 return kTRUE;
1227 }
1228
1229 // read test mode if flag is set
1230 if (fReadTestMode)
1231 {
3d8bc902 1232 fTestMode = kNone;
9827400b 1233 TString logEntry(entry->GetRunParameter("log"));
1234 //printf("log entry = %s\n", logEntry.Data());
1235 TString searchStr("Testmode: ");
1236 Int_t pos = logEntry.Index(searchStr.Data());
1237 //printf("%d\n", pos);
1238 if (pos >= 0)
1239 {
1240 TSubString subStr = logEntry(pos + searchStr.Length(), logEntry.Length());
1241 //printf("%s\n", subStr.String().Data());
1242 TString newStr(subStr.Data());
1243 TObjArray* token = newStr.Tokenize(' ');
1244 if (token)
1245 {
1246 //token->Print();
1247 TObjString* tmpStr = dynamic_cast<TObjString*> (token->First());
1248 if (tmpStr)
1249 {
1250 Int_t testMode = tmpStr->String().Atoi();
1251 if (testMode > 0)
1252 {
1253 Log("SHUTTLE", Form("Enabling test mode %d", testMode));
1254 SetTestMode((TestMode) testMode);
1255 }
1256 }
1257 delete token;
1258 }
1259 }
1260 }
1261
3d8bc902 1262 Log("SHUTTLE", Form("The test mode flag is %d", (Int_t) fTestMode));
1263
eba76848 1264 fLogbookEntry->Print("all");
57f50b3c 1265
1266 // Initialization
d477ad88 1267 Bool_t hasError = kFALSE;
5164a766 1268
2bb7b766 1269 AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
1270 if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
1271 AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
1272 if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
d477ad88 1273
57f50b3c 1274 // Loop on detectors in the configuration
b948db8d 1275 TIter iter(fConfig->GetDetectors());
2bb7b766 1276 TObjString* aDetector = 0;
b948db8d 1277
be48e3ea 1278 while ((aDetector = (TObjString*) iter.Next()))
1279 {
7bfb2090 1280 fCurrentDetector = aDetector->String();
5164a766 1281
9e080f92 1282 if (ContinueProcessing() == kFALSE) continue;
1283
2bb7b766 1284 AliInfo(Form("\n\n \t\t\t****** run %d - %s: START ******",
1285 GetCurrentRun(), aDetector->GetName()));
1286
9d733021 1287 for(Int_t iSys=0;iSys<3;iSys++) fFXSCalled[iSys]=kFALSE;
1288
e7f62f16 1289 Log(fCurrentDetector.Data(), "Starting processing");
85a80aa9 1290
be48e3ea 1291 Int_t pid = fork();
1292
1293 if (pid < 0)
1294 {
1295 Log("SHUTTLE", "ERROR: Forking failed");
1296 }
1297 else if (pid > 0)
1298 {
1299 // parent
1300 AliInfo(Form("In parent process of %d - %s: Starting monitoring",
1301 GetCurrentRun(), aDetector->GetName()));
1302
1303 Long_t begin = time(0);
1304
1305 int status; // to be used with waitpid, on purpose an int (not Int_t)!
1306 while (waitpid(pid, &status, WNOHANG) == 0)
1307 {
1308 Long_t expiredTime = time(0) - begin;
1309
1310 if (expiredTime > fConfig->GetPPTimeOut())
1311 {
9827400b 1312 TString tmp;
1313 tmp.Form("Process of %s time out. Run time: %d seconds. Killing...",
1314 fCurrentDetector.Data(), expiredTime);
1315 Log("SHUTTLE", tmp);
1316 Log(fCurrentDetector, tmp);
be48e3ea 1317
1318 kill(pid, 9);
1319
3301427a 1320 UpdateShuttleStatus(AliShuttleStatus::kPPTimeOut);
be48e3ea 1321 hasError = kTRUE;
1322
1323 gSystem->Sleep(1000);
1324 }
1325 else
1326 {
be48e3ea 1327 gSystem->Sleep(1000);
9827400b 1328
1329 TString checkStr;
1330 checkStr.Form("ps -o vsize --pid %d | tail -n 1", pid);
1331 FILE* pipe = gSystem->OpenPipe(checkStr, "r");
1332 if (!pipe)
1333 {
1334 Log("SHUTTLE", Form("Error: Could not open pipe to %s", checkStr.Data()));
1335 continue;
1336 }
1337
1338 char buffer[100];
1339 if (!fgets(buffer, 100, pipe))
1340 {
1341 Log("SHUTTLE", "Error: ps did not return anything");
1342 gSystem->ClosePipe(pipe);
1343 continue;
1344 }
1345 gSystem->ClosePipe(pipe);
1346
1347 //Log("SHUTTLE", Form("ps returned %s", buffer));
1348
1349 Int_t mem = 0;
1350 if ((sscanf(buffer, "%d\n", &mem) != 1) || !mem)
1351 {
1352 Log("SHUTTLE", "Error: Could not parse output of ps");
1353 continue;
1354 }
1355
1356 if (expiredTime % 60 == 0)
886d60e6 1357 Log("SHUTTLE", Form("%s: Checking process. Run time: %d seconds - Memory consumption: %d KB",
1358 fCurrentDetector.Data(), expiredTime, mem));
9827400b 1359
1360 if (mem > fConfig->GetPPMaxMem())
1361 {
1362 TString tmp;
1363 tmp.Form("Process exceeds maximum allowed memory (%d KB > %d KB). Killing...",
1364 mem, fConfig->GetPPMaxMem());
1365 Log("SHUTTLE", tmp);
1366 Log(fCurrentDetector, tmp);
1367
1368 kill(pid, 9);
1369
1370 UpdateShuttleStatus(AliShuttleStatus::kPPOutOfMemory);
1371 hasError = kTRUE;
1372
1373 gSystem->Sleep(1000);
1374 }
be48e3ea 1375 }
1376 }
1377
1378 AliInfo(Form("In parent process of %d - %s: Client has terminated.",
1379 GetCurrentRun(), aDetector->GetName()));
1380
1381 if (WIFEXITED(status))
1382 {
1383 Int_t returnCode = WEXITSTATUS(status);
1384
3301427a 1385 Log("SHUTTLE", Form("%s: the return code is %d", fCurrentDetector.Data(),
1386 returnCode));
be48e3ea 1387
9827400b 1388 if (returnCode == 0) hasError = kTRUE;
be48e3ea 1389 }
1390 }
1391 else if (pid == 0)
1392 {
1393 // client
1394 AliInfo(Form("In client process of %d - %s", GetCurrentRun(), aDetector->GetName()));
1395
ffa29e93 1396 AliInfo("Redirecting output...");
1397
546242fb 1398 if ((freopen(GetLogFileName(fCurrentDetector), "a", stdout)) == 0)
ffa29e93 1399 {
1400 Log("SHUTTLE", "Could not freopen stdout");
1401 }
1402 else
1403 {
1404 fOutputRedirected = kTRUE;
1405 if ((dup2(fileno(stdout), fileno(stderr))) < 0)
1406 Log("SHUTTLE", "Could not redirect stderr");
1407
1408 }
1409
5bac2bde 1410 TString wd = gSystem->WorkingDirectory();
1411 TString tmpDir = Form("%s/%s_process",GetShuttleTempDir(),fCurrentDetector.Data());
1412
1413 gSystem->mkdir(tmpDir.Data());
1414 gSystem->ChangeDirectory(tmpDir.Data());
1415
9827400b 1416 Bool_t success = ProcessCurrentDetector();
5bac2bde 1417
1418 gSystem->ChangeDirectory(wd.Data());
1419
1420 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
1421
9827400b 1422 if (success) // Preprocessor finished successfully!
1423 {
3301427a 1424 // Update time_processed field in FXS DB
1425 if (UpdateTable() == kFALSE)
5bac2bde 1426 Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!",
1427 fCurrentDetector.Data()));
3301427a 1428
1429 // Transfer the data from local storage to main storage (Grid)
1430 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1431 if (StoreOCDB() == kFALSE)
1432 {
1433 AliInfo(Form("\n \t\t\t****** run %d - %s: STORAGE ERROR ****** \n\n",
1434 GetCurrentRun(), aDetector->GetName()));
1435 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
9827400b 1436 success = kFALSE;
3301427a 1437 } else {
1438 AliInfo(Form("\n \t\t\t****** run %d - %s: DONE ****** \n\n",
1439 GetCurrentRun(), aDetector->GetName()));
1440 UpdateShuttleStatus(AliShuttleStatus::kDone);
9827400b 1441 UpdateShuttleLogbook(fCurrentDetector, "DONE");
3301427a 1442 }
be48e3ea 1443 }
1444
4b95672b 1445 for (UInt_t iSys=0; iSys<3; iSys++)
1446 {
1447 if (fFXSCalled[iSys]) fFXSlist[iSys].Clear();
1448 }
1449
be48e3ea 1450 AliInfo(Form("Client process of %d - %s is exiting now with %d.",
9827400b 1451 GetCurrentRun(), aDetector->GetName(), success));
be48e3ea 1452
1453 // the client exits here
9827400b 1454 gSystem->Exit(success);
be48e3ea 1455
1456 AliError("We should never get here!!!");
1457 }
7bfb2090 1458 }
5164a766 1459
2bb7b766 1460 AliInfo(Form("\n\n \t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^* \n",
1461 GetCurrentRun()));
1462
1463 //check if shuttle is done for this run, if so update logbook
1464 TObjArray checkEntryArray;
1465 checkEntryArray.SetOwner(1);
9e080f92 1466 TString whereClause = Form("where run=%d", GetCurrentRun());
1467 if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) || checkEntryArray.GetEntries() == 0) {
1468 Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
1469 GetCurrentRun()));
1470 return hasError == kFALSE;
1471 }
b948db8d 1472
9e080f92 1473 AliShuttleLogbookEntry* checkEntry = dynamic_cast<AliShuttleLogbookEntry*>
1474 (checkEntryArray.At(0));
2bb7b766 1475
9e080f92 1476 if (checkEntry)
1477 {
1478 if (checkEntry->IsDone())
be48e3ea 1479 {
9e080f92 1480 Log("SHUTTLE","Process - Shuttle is DONE. Updating logbook");
1481 UpdateShuttleLogbook("shuttle_done");
1482 }
1483 else
1484 {
1485 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
be48e3ea 1486 {
9e080f92 1487 if (checkEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
be48e3ea 1488 {
9e080f92 1489 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
1490 checkEntry->GetRun(), GetDetName(iDet)));
1491 fFirstUnprocessed[iDet] = kFALSE;
be48e3ea 1492 }
1493 }
2bb7b766 1494 }
1495 }
1496
e7f62f16 1497 // remove ML instance
1498 delete fMonaLisa;
1499 fMonaLisa = 0;
1500
2bb7b766 1501 fLogbookEntry = 0;
85a80aa9 1502
a7160fe9 1503 return hasError == kFALSE;
73abe331 1504}
1505
b948db8d 1506//______________________________________________________________________________________________
9827400b 1507Bool_t AliShuttle::ProcessCurrentDetector()
73abe331 1508{
1509 //
2bb7b766 1510 // Makes data retrieval just for a specific detector (fCurrentDetector).
73abe331 1511 // Threre should be a configuration for this detector.
73abe331 1512
2bb7b766 1513 AliInfo(Form("Retrieving values for %s, run %d", fCurrentDetector.Data(), GetCurrentRun()));
73abe331 1514
2d9019b4 1515 if (!CleanReferenceStorage(fCurrentDetector.Data()))
546242fb 1516 return kFALSE;
1517
a038aa70 1518 TMap* dcsMap = 0;
3301427a 1519
1520 // call preprocessor
1521 AliPreprocessor* aPreprocessor =
1522 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1523
1524 aPreprocessor->Initialize(GetCurrentRun(), GetCurrentStartTime(), GetCurrentEndTime());
1525
1526 Bool_t processDCS = aPreprocessor->ProcessDCS();
d477ad88 1527
651fdaab 1528 if (!processDCS)
1529 {
1530 Log(fCurrentDetector, "The preprocessor requested to skip the retrieval of DCS values");
1531 }
8b739301 1532 else if (fTestMode & kSkipDCS)
2c15234c 1533 {
3d8bc902 1534 Log(fCurrentDetector, "In TESTMODE - Skipping DCS processing!");
9827400b 1535 }
1536 else if (fTestMode & kErrorDCS)
1537 {
3d8bc902 1538 Log(fCurrentDetector, "In TESTMODE - Simulating DCS error");
1539 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
9827400b 1540 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1541 return kFALSE;
2c15234c 1542 } else {
3301427a 1543
1544 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
1545
2c15234c 1546 TString host(fConfig->GetDCSHost(fCurrentDetector));
1547 Int_t port = fConfig->GetDCSPort(fCurrentDetector);
1548
a038aa70 1549 if (fConfig->GetDCSAliases(fCurrentDetector)->GetEntries() > 0)
2c15234c 1550 {
a038aa70 1551 dcsMap = GetValueSet(host, port, fConfig->GetDCSAliases(fCurrentDetector), kAlias);
1552 if (!dcsMap)
2c15234c 1553 {
a038aa70 1554 Log(fCurrentDetector, "ProcessCurrentDetector - Error while retrieving DCS aliases");
2c15234c 1555 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
9827400b 1556 return kFALSE;
2c15234c 1557 }
4f0ab988 1558 }
a038aa70 1559
1560 if (fConfig->GetDCSDataPoints(fCurrentDetector)->GetEntries() > 0)
2c15234c 1561 {
a038aa70 1562 TMap* dcsMap2 = GetValueSet(host, port, fConfig->GetDCSDataPoints(fCurrentDetector), kDP);
1563 if (!dcsMap2)
2c15234c 1564 {
a038aa70 1565 Log(fCurrentDetector, "ProcessCurrentDetector - Error while retrieving DCS data points");
2c15234c 1566 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
a038aa70 1567 if (dcsMap)
1568 delete dcsMap;
9827400b 1569 return kFALSE;
2c15234c 1570 }
a038aa70 1571
1572 if (!dcsMap)
1573 {
1574 dcsMap = dcsMap2;
1575 }
1576 else // merge
1577 {
1578 TIter iter(dcsMap2);
1579 TObjString* key = 0;
1580 while ((key = (TObjString*) iter.Next()))
1581 dcsMap->Add(key, dcsMap2->GetValue(key->String()));
1582
1583 dcsMap2->SetOwner(kFALSE);
1584 delete dcsMap2;
1585 }
73abe331 1586 }
a038aa70 1587
73abe331 1588 }
b948db8d 1589
dc25836b 1590 // still no map?
1591 if (!dcsMap)
1592 dcsMap = new TMap;
1593
2bb7b766 1594 // DCS Archive DB processing successful. Call Preprocessor!
85a80aa9 1595 UpdateShuttleStatus(AliShuttleStatus::kPPStarted);
a7160fe9 1596
a038aa70 1597 UInt_t returnValue = aPreprocessor->Process(dcsMap);
b948db8d 1598
3301427a 1599 if (returnValue > 0) // Preprocessor error!
1600 {
9827400b 1601 Log(fCurrentDetector, Form("Preprocessor failed. Process returned %d.", returnValue));
cb343cfd 1602 UpdateShuttleStatus(AliShuttleStatus::kPPError);
a038aa70 1603 dcsMap->DeleteAll();
1604 delete dcsMap;
9827400b 1605 return kFALSE;
1606 }
1607
1608 // preprocessor ok!
1609 UpdateShuttleStatus(AliShuttleStatus::kPPDone);
1610 Log(fCurrentDetector, Form("ProcessCurrentDetector - %s preprocessor returned success",
1611 fCurrentDetector.Data()));
b948db8d 1612
a038aa70 1613 dcsMap->DeleteAll();
1614 delete dcsMap;
b948db8d 1615
9827400b 1616 return kTRUE;
2bb7b766 1617}
1618
1619//______________________________________________________________________________________________
1620Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
1621 TObjArray& entries)
1622{
9827400b 1623 // Query DAQ's Shuttle logbook and fills detector status object.
1624 // Call QueryRunParameters to query DAQ logbook for run parameters.
1625 //
2bb7b766 1626
fc5a4708 1627 entries.SetOwner(1);
1628
2bb7b766 1629 // check connection, in case connect
be48e3ea 1630 if(!Connect(3)) return kFALSE;
2bb7b766 1631
1632 TString sqlQuery;
441b0e9c 1633 sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
2bb7b766 1634
be48e3ea 1635 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1636 if (!aResult) {
1637 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
1638 return kFALSE;
1639 }
1640
fc5a4708 1641 AliDebug(2,Form("Query = %s", sqlQuery.Data()));
1642
2bb7b766 1643 if(aResult->GetRowCount() == 0) {
9827400b 1644 AliInfo("No entries in Shuttle Logbook match request");
1645 delete aResult;
1646 return kTRUE;
2bb7b766 1647 }
1648
1649 // TODO Check field count!
db99d43e 1650 const UInt_t nCols = 23;
2bb7b766 1651 if (aResult->GetFieldCount() != (Int_t) nCols) {
1652 AliError("Invalid SQL result field number!");
1653 delete aResult;
1654 return kFALSE;
1655 }
1656
2bb7b766 1657 TSQLRow* aRow;
1658 while ((aRow = aResult->Next())) {
1659 TString runString(aRow->GetField(0), aRow->GetFieldLength(0));
1660 Int_t run = runString.Atoi();
1661
eba76848 1662 AliShuttleLogbookEntry *entry = QueryRunParameters(run);
1663 if (!entry)
1664 continue;
2bb7b766 1665
1666 // loop on detectors
eba76848 1667 for(UInt_t ii = 0; ii < nCols; ii++)
1668 entry->SetDetectorStatus(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1669
eba76848 1670 entries.AddLast(entry);
2bb7b766 1671 delete aRow;
1672 }
1673
2bb7b766 1674 delete aResult;
1675 return kTRUE;
1676}
1677
1678//______________________________________________________________________________________________
eba76848 1679AliShuttleLogbookEntry* AliShuttle::QueryRunParameters(Int_t run)
2bb7b766 1680{
eba76848 1681 //
1682 // Retrieve run parameters written in the DAQ logbook and sets them into AliShuttleLogbookEntry object
1683 //
2bb7b766 1684
1685 // check connection, in case connect
be48e3ea 1686 if (!Connect(3))
eba76848 1687 return 0;
2bb7b766 1688
1689 TString sqlQuery;
2c15234c 1690 sqlQuery.Form("select * from %s where run=%d", fConfig->GetDAQlbTable(), run);
2bb7b766 1691
be48e3ea 1692 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1693 if (!aResult) {
1694 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
eba76848 1695 return 0;
2bb7b766 1696 }
1697
eba76848 1698 if (aResult->GetRowCount() == 0) {
2bb7b766 1699 Log("SHUTTLE", Form("QueryRunParameters - No entry in DAQ Logbook for run %d. Skipping", run));
1700 delete aResult;
eba76848 1701 return 0;
2bb7b766 1702 }
1703
eba76848 1704 if (aResult->GetRowCount() > 1) {
2bb7b766 1705 AliError(Form("More than one entry in DAQ Logbook for run %d. Skipping", run));
1706 delete aResult;
eba76848 1707 return 0;
2bb7b766 1708 }
1709
eba76848 1710 TSQLRow* aRow = aResult->Next();
1711 if (!aRow)
1712 {
1713 AliError(Form("Could not retrieve row for run %d. Skipping", run));
1714 delete aResult;
1715 return 0;
1716 }
2bb7b766 1717
eba76848 1718 AliShuttleLogbookEntry* entry = new AliShuttleLogbookEntry(run);
2bb7b766 1719
eba76848 1720 for (Int_t ii = 0; ii < aResult->GetFieldCount(); ii++)
1721 entry->SetRunParameter(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1722
eba76848 1723 UInt_t startTime = entry->GetStartTime();
1724 UInt_t endTime = entry->GetEndTime();
1725
1726 if (!startTime || !endTime || startTime > endTime) {
1727 Log("SHUTTLE",
1728 Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d",
1729 run, startTime, endTime));
1730 delete entry;
2bb7b766 1731 delete aRow;
eba76848 1732 delete aResult;
1733 return 0;
2bb7b766 1734 }
1735
eba76848 1736 delete aRow;
2bb7b766 1737 delete aResult;
eba76848 1738
1739 return entry;
2bb7b766 1740}
1741
b948db8d 1742//______________________________________________________________________________________________
2c15234c 1743Bool_t AliShuttle::GetValueSet(const char* host, Int_t port, const char* entry,
1744 TObjArray* valueSet, DCSType type)
73abe331 1745{
9827400b 1746 // Retrieve all "entry" data points from the DCS server
1747 // host, port: TSocket connection parameters
1748 // entry: name of the alias or data point
1749 // valueSet: array of retrieved AliDCSValue's
1750 // type: kAlias or kDP
58bc3020 1751
73abe331 1752 AliDCSClient client(host, port, fTimeout, fRetries);
2c15234c 1753 if (!client.IsConnected())
1754 {
b948db8d 1755 return kFALSE;
73abe331 1756 }
1757
2c15234c 1758 Int_t result=0;
73abe331 1759
2c15234c 1760 if (type == kAlias)
1761 {
1762 result = client.GetAliasValues(entry,
1763 GetCurrentStartTime(), GetCurrentEndTime(), valueSet);
1764 } else
1765 if (type == kDP)
1766 {
1767 result = client.GetDPValues(entry,
1768 GetCurrentStartTime(), GetCurrentEndTime(), valueSet);
1769 }
1770
1771 if (result < 0)
1772 {
2bb7b766 1773 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get '%s'! Reason: %s",
2c15234c 1774 entry, AliDCSClient::GetErrorString(result)));
73abe331 1775
2c15234c 1776 if (result == AliDCSClient::fgkServerError)
1777 {
2bb7b766 1778 Log(fCurrentDetector.Data(), Form("GetValueSet - Server error: %s",
73abe331 1779 client.GetServerError().Data()));
1780 }
1781
1782 return kFALSE;
1783 }
1784
1785 return kTRUE;
1786}
b948db8d 1787
a038aa70 1788//______________________________________________________________________________________________
1789TMap* AliShuttle::GetValueSet(const char* host, Int_t port, const TSeqCollection* entries,
1790 DCSType type)
1791{
1792 // Retrieve all "entry" data points from the DCS server
1793 // host, port: TSocket connection parameters
1794 // entries: list of name of the alias or data point
1795 // type: kAlias or kDP
1796 // returns TMap of values, 0 when failure
1797
1798 const Int_t kSplit = 100; // maximum number of DPs at a time
1799
1800 Int_t totalEntries = entries->GetEntries();
1801
1802 TMap* result = 0;
1803
1804 for (Int_t index=0; index < totalEntries; index += kSplit)
1805 {
1806 Int_t endIndex = index + kSplit;
1807
1808 AliDCSClient client(host, port, fTimeout, fRetries);
1809 if (!client.IsConnected())
1810 return 0;
1811
1812 TMap* partialResult = 0;
1813
1814 if (type == kAlias)
1815 {
1816 partialResult = client.GetAliasValues(entries, GetCurrentStartTime(),
1817 GetCurrentEndTime(), index, endIndex);
1818 }
1819 else if (type == kDP)
1820 {
1821 partialResult = client.GetDPValues(entries, GetCurrentStartTime(),
1822 GetCurrentEndTime(), index, endIndex);
1823 }
1824
1825 if (partialResult == 0)
1826 {
1827 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get entries (%d...%d)! Reason: %s",
1828 index, endIndex, client.GetServerError().Data()));
1829
1830 if (result)
1831 delete result;
1832
1833 return 0;
1834 }
1835
1836 AliInfo(Form("Retrieved entries %d..%d (total %d); E.g. %s has %d values collected",
1837 index, endIndex, totalEntries, entries->At(index)->GetName(), ((TObjArray*)
1838 partialResult->GetValue(entries->At(index)->GetName()))->GetEntriesFast()));
1839
1840 if (!result)
1841 {
1842 result = partialResult;
1843 }
1844 else
1845 {
1846 TIter iter(partialResult);
1847 TObjString* key = 0;
1848 while ((key = (TObjString*) iter.Next()))
1849 result->Add(key, partialResult->GetValue(key->String()));
1850
1851 partialResult->SetOwner(kFALSE);
1852 delete partialResult;
1853 }
1854
1855 }
1856
1857 return result;
1858}
b948db8d 1859//______________________________________________________________________________________________
57f50b3c 1860const char* AliShuttle::GetFile(Int_t system, const char* detector,
1861 const char* id, const char* source)
b948db8d 1862{
9827400b 1863 // Get calibration file from file exchange servers
1864 // First queris the FXS database for the file name, using the run, detector, id and source info
1865 // then calls RetrieveFile(filename) for actual copy to local disk
1866 // run: current run being processed (given by Logbook entry fLogbookEntry)
1867 // detector: the Preprocessor name
1868 // id: provided as a parameter by the Preprocessor
1869 // source: provided by the Preprocessor through GetFileSources function
1870
1871 // check if test mode should simulate a FXS error
1872 if (fTestMode & kErrorFXSFiles)
1873 {
1874 Log(detector, Form("GetFile - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
1875 return 0;
1876 }
1877
57f50b3c 1878 // check connection, in case connect
9d733021 1879 if (!Connect(system))
eba76848 1880 {
9d733021 1881 Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
57f50b3c 1882 return 0;
1883 }
1884
1885 // Query preparation
9d733021 1886 TString sourceName(source);
d386d623 1887 Int_t nFields = 3;
1888 TString sqlQueryStart = Form("select filePath,size,fileChecksum from %s where",
1889 fConfig->GetFXSdbTable(system));
1890 TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
1891 GetCurrentRun(), detector, id);
1892
9d733021 1893 if (system == kDAQ)
1894 {
d386d623 1895 whereClause += Form(" and DAQsource=\"%s\"", source);
57f50b3c 1896 }
9d733021 1897 else if (system == kDCS)
eba76848 1898 {
9d733021 1899 sourceName="none";
57f50b3c 1900 }
9d733021 1901 else if (system == kHLT)
9e080f92 1902 {
d386d623 1903 whereClause += Form(" and DDLnumbers=\"%s\"", source);
9d733021 1904 nFields = 3;
9e080f92 1905 }
1906
9e080f92 1907 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
1908
1909 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
1910
1911 // Query execution
1912 TSQLResult* aResult = 0;
9d733021 1913 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
9e080f92 1914 if (!aResult) {
9d733021 1915 Log(detector, Form("GetFileName - Can't execute SQL query to %s database for: id = %s, source = %s",
1916 GetSystemName(system), id, sourceName.Data()));
9e080f92 1917 return 0;
1918 }
1919
1920 if(aResult->GetRowCount() == 0)
1921 {
1922 Log(detector,
9d733021 1923 Form("GetFileName - No entry in %s FXS db for: id = %s, source = %s",
1924 GetSystemName(system), id, sourceName.Data()));
9e080f92 1925 delete aResult;
1926 return 0;
1927 }
2bb7b766 1928
9e080f92 1929 if (aResult->GetRowCount() > 1) {
1930 Log(detector,
9d733021 1931 Form("GetFileName - More than one entry in %s FXS db for: id = %s, source = %s",
1932 GetSystemName(system), id, sourceName.Data()));
9e080f92 1933 delete aResult;
1934 return 0;
1935 }
1936
9d733021 1937 if (aResult->GetFieldCount() != nFields) {
9e080f92 1938 Log(detector,
9d733021 1939 Form("GetFileName - Wrong field count in %s FXS db for: id = %s, source = %s",
1940 GetSystemName(system), id, sourceName.Data()));
9e080f92 1941 delete aResult;
1942 return 0;
1943 }
1944
1945 TSQLRow* aRow = dynamic_cast<TSQLRow*> (aResult->Next());
1946
1947 if (!aRow){
9d733021 1948 Log(detector, Form("GetFileName - Empty set result in %s FXS db from query: id = %s, source = %s",
1949 GetSystemName(system), id, sourceName.Data()));
9e080f92 1950 delete aResult;
1951 return 0;
1952 }
1953
1954 TString filePath(aRow->GetField(0), aRow->GetFieldLength(0));
1955 TString fileSize(aRow->GetField(1), aRow->GetFieldLength(1));
d386d623 1956 TString fileChecksum(aRow->GetField(2), aRow->GetFieldLength(2));
9e080f92 1957
1958 delete aResult;
1959 delete aRow;
1960
d386d623 1961 AliDebug(2, Form("filePath = %s; size = %s, fileChecksum = %s",
1962 filePath.Data(), fileSize.Data(), fileChecksum.Data()));
9e080f92 1963
9e080f92 1964 // retrieved file is renamed to make it unique
9d733021 1965 TString localFileName = Form("%s_%s_%d_%s_%s.shuttle",
1966 GetSystemName(system), detector, GetCurrentRun(), id, sourceName.Data());
1967
9e080f92 1968
9d733021 1969 // file retrieval from FXS
4b95672b 1970 UInt_t nRetries = 0;
1971 UInt_t maxRetries = 3;
1972 Bool_t result = kFALSE;
1973
1974 // copy!! if successful TSystem::Exec returns 0
1975 while(nRetries++ < maxRetries) {
1976 AliDebug(2, Form("Trying to copy file. Retry # %d", nRetries));
1977 result = RetrieveFile(system, filePath.Data(), localFileName.Data());
1978 if(!result)
1979 {
1980 Log(detector, Form("GetFileName - Copy of file %s from %s FXS failed",
9d733021 1981 filePath.Data(), GetSystemName(system)));
4b95672b 1982 continue;
1983 } else {
1984 AliInfo(Form("File %s copied from %s FXS into %s/%s",
1985 filePath.Data(), GetSystemName(system),
1986 GetShuttleTempDir(), localFileName.Data()));
1987 }
9e080f92 1988
d386d623 1989 if (fileChecksum.Length()>0)
4b95672b 1990 {
1991 // compare md5sum of local file with the one stored in the FXS DB
1992 Int_t md5Comp = gSystem->Exec(Form("md5sum %s/%s |grep %s 2>&1 > /dev/null",
d386d623 1993 GetShuttleTempDir(), localFileName.Data(), fileChecksum.Data()));
9e080f92 1994
4b95672b 1995 if (md5Comp != 0)
1996 {
1997 Log(detector, Form("GetFileName - md5sum of file %s does not match with local copy!",
1998 filePath.Data()));
1999 result = kFALSE;
2000 continue;
2001 }
d386d623 2002 } else {
2003 Log(fCurrentDetector, Form("GetFile - md5sum of file %s not set in %s database, skipping comparison",
2004 filePath.Data(), GetSystemName(system)));
9d733021 2005 }
4b95672b 2006 if (result) break;
9e080f92 2007 }
2008
4b95672b 2009 if(!result) return 0;
2010
9d733021 2011 fFXSCalled[system]=kTRUE;
2012 TObjString *fileParams = new TObjString(Form("%s#!?!#%s", id, sourceName.Data()));
2013 fFXSlist[system].Add(fileParams);
9e080f92 2014
2015 static TString fullLocalFileName;
36c99a6a 2016 fullLocalFileName = TString::Format("%s/%s", GetShuttleTempDir(), localFileName.Data());
2017
9e080f92 2018 AliInfo(Form("fullLocalFileName = %s", fullLocalFileName.Data()));
2019
2020 return fullLocalFileName.Data();
2bb7b766 2021
2022}
2023
2024//______________________________________________________________________________________________
9d733021 2025Bool_t AliShuttle::RetrieveFile(UInt_t system, const char* fxsFileName, const char* localFileName)
9e080f92 2026{
9827400b 2027 //
2028 // Copies file from FXS to local Shuttle machine
2029 //
2bb7b766 2030
9e080f92 2031 // check temp directory: trying to cd to temp; if it does not exist, create it
9d733021 2032 AliDebug(2, Form("Copy file %s from %s FXS into %s/%s",
2033 GetSystemName(system), fxsFileName, GetShuttleTempDir(), localFileName));
9e080f92 2034
36c99a6a 2035 void* dir = gSystem->OpenDirectory(GetShuttleTempDir());
9e080f92 2036 if (dir == NULL) {
36c99a6a 2037 if (gSystem->mkdir(GetShuttleTempDir(), kTRUE)) {
2038 AliError(Form("Can't open directory <%s>", GetShuttleTempDir()));
9e080f92 2039 return kFALSE;
2040 }
2041
2042 } else {
2043 gSystem->FreeDirectory(dir);
2044 }
2045
9d733021 2046 TString baseFXSFolder;
2047 if (system == kDAQ)
2048 {
2049 baseFXSFolder = "FES/";
2050 }
2051 else if (system == kDCS)
2052 {
2053 baseFXSFolder = "";
2054 }
2055 else if (system == kHLT)
2056 {
dc25836b 2057 baseFXSFolder = "/opt/FXS";
9d733021 2058 }
2059
2060
2061 TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s/%s",
2062 fConfig->GetFXSPort(system),
2063 fConfig->GetFXSUser(system),
2064 fConfig->GetFXSHost(system),
2065 baseFXSFolder.Data(),
2066 fxsFileName,
36c99a6a 2067 GetShuttleTempDir(),
9e080f92 2068 localFileName);
2069
2070 AliDebug(2, Form("%s",command.Data()));
2071
4b95672b 2072 Bool_t result = (gSystem->Exec(command.Data()) == 0);
9e080f92 2073
4b95672b 2074 return result;
9e080f92 2075}
2076
2077//______________________________________________________________________________________________
9d733021 2078TList* AliShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
2079{
9827400b 2080 //
2081 // Get sources producing the condition file Id from file exchange servers
4a33bdd9 2082 // if id is NULL all sources are returned (distinct)
9827400b 2083 //
2084
2085 // check if test mode should simulate a FXS error
2086 if (fTestMode & kErrorFXSSources)
2087 {
2088 Log(detector, Form("GetFileSources - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2089 return 0;
2090 }
2091
9d733021 2092
2093 if (system == kDCS)
2094 {
6297b37d 2095 AliWarning("DCS system has only one source of data!");
2096 TList *list = new TList();
2097 list->SetOwner(1);
2098 list->Add(new TObjString(" "));
2099 return list;
9d733021 2100 }
9e080f92 2101
2102 // check connection, in case connect
9d733021 2103 if (!Connect(system))
2104 {
4a33bdd9 2105 Log(detector, Form("GetFileSources - Couldn't connect to %s FXS database", GetSystemName(system)));
9d733021 2106 return NULL;
9e080f92 2107 }
2108
9d733021 2109 TString sourceName = 0;
2110 if (system == kDAQ)
2111 {
2112 sourceName = "DAQsource";
2113 } else if (system == kHLT)
2114 {
2115 sourceName = "DDLnumbers";
2116 }
2117
4a33bdd9 2118 TString sqlQueryStart = Form("select distinct %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
2119 TString whereClause = Form("run=%d and detector=\"%s\"",
2120 GetCurrentRun(), detector);
2121 if (id)
2122 whereClause += Form(" and fileId=\"%s\"", id);
9e080f92 2123 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2124
2125 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2126
2127 // Query execution
2128 TSQLResult* aResult;
9d733021 2129 aResult = fServer[system]->Query(sqlQuery);
9e080f92 2130 if (!aResult) {
9d733021 2131 Log(detector, Form("GetFileSources - Can't execute SQL query to %s database for id: %s",
2132 GetSystemName(system), id));
9e080f92 2133 return 0;
2134 }
2135
86aa42c3 2136 TList *list = new TList();
2137 list->SetOwner(1);
2138
9d733021 2139 if (aResult->GetRowCount() == 0)
2140 {
9e080f92 2141 Log(detector,
9d733021 2142 Form("GetFileSources - No entry in %s FXS table for id: %s", GetSystemName(system), id));
9e080f92 2143 delete aResult;
86aa42c3 2144 return list;
9e080f92 2145 }
2146
2147 TSQLRow* aRow;
9e080f92 2148
9d733021 2149 while ((aRow = aResult->Next()))
2150 {
9e080f92 2151
9d733021 2152 TString source(aRow->GetField(0), aRow->GetFieldLength(0));
2153 AliDebug(2, Form("%s = %s", sourceName.Data(), source.Data()));
2154 list->Add(new TObjString(source));
9e080f92 2155 delete aRow;
2156 }
9d733021 2157
9e080f92 2158 delete aResult;
2159
2160 return list;
2bb7b766 2161}
2162
4a33bdd9 2163//______________________________________________________________________________________________
2164TList* AliShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
2165{
2166 //
2167 // Get all ids of condition files produced by a given source from file exchange servers
2168 //
2169
2170 // check if test mode should simulate a FXS error
2171 if (fTestMode & kErrorFXSSources)
2172 {
2173 Log(detector, Form("GetFileIDs - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2174 return 0;
2175 }
2176
2177 // check connection, in case connect
2178 if (!Connect(system))
2179 {
2180 Log(detector, Form("GetFileIDs - Couldn't connect to %s FXS database", GetSystemName(system)));
2181 return NULL;
2182 }
2183
2184 TString sourceName = 0;
2185 if (system == kDAQ)
2186 {
2187 sourceName = "DAQsource";
2188 } else if (system == kHLT)
2189 {
2190 sourceName = "DDLnumbers";
2191 }
2192
2193 TString sqlQueryStart = Form("select fileId from %s where", fConfig->GetFXSdbTable(system));
2194 TString whereClause = Form("run=%d and detector=\"%s\"",
2195 GetCurrentRun(), detector);
2196 if (sourceName.Length() > 0 && source)
2197 whereClause += Form(" and %s=\"%s\"", sourceName.Data(), source);
2198 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2199
2200 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2201
2202 // Query execution
2203 TSQLResult* aResult;
2204 aResult = fServer[system]->Query(sqlQuery);
2205 if (!aResult) {
2206 Log(detector, Form("GetFileIDs - Can't execute SQL query to %s database for source: %s",
2207 GetSystemName(system), source));
2208 return 0;
2209 }
2210
2211 TList *list = new TList();
2212 list->SetOwner(1);
2213
2214 if (aResult->GetRowCount() == 0)
2215 {
2216 Log(detector,
2217 Form("GetFileIDs - No entry in %s FXS table for source: %s", GetSystemName(system), source));
2218 delete aResult;
2219 return list;
2220 }
2221
2222 TSQLRow* aRow;
2223
2224 while ((aRow = aResult->Next()))
2225 {
2226
2227 TString id(aRow->GetField(0), aRow->GetFieldLength(0));
2228 AliDebug(2, Form("fileId = %s", id.Data()));
2229 list->Add(new TObjString(id));
2230 delete aRow;
2231 }
2232
2233 delete aResult;
2234
2235 return list;
2236}
2237
2bb7b766 2238//______________________________________________________________________________________________
9d733021 2239Bool_t AliShuttle::Connect(Int_t system)
2bb7b766 2240{
9827400b 2241 // Connect to MySQL Server of the system's FXS MySQL databases
2242 // DAQ Logbook, Shuttle Logbook and DAQ FXS db are on the same host
2243 //
57f50b3c 2244
9d733021 2245 // check connection: if already connected return
2246 if(fServer[system] && fServer[system]->IsConnected()) return kTRUE;
57f50b3c 2247
9d733021 2248 TString dbHost, dbUser, dbPass, dbName;
57f50b3c 2249
9d733021 2250 if (system < 3) // FXS db servers
2251 {
2252 dbHost = Form("mysql://%s:%d", fConfig->GetFXSdbHost(system), fConfig->GetFXSdbPort(system));
2253 dbUser = fConfig->GetFXSdbUser(system);
2254 dbPass = fConfig->GetFXSdbPass(system);
2255 dbName = fConfig->GetFXSdbName(system);
2256 } else { // Run & Shuttle logbook servers
2257 // TODO Will the Shuttle logbook server be the same as the Run logbook server ???
2258 dbHost = Form("mysql://%s:%d", fConfig->GetDAQlbHost(), fConfig->GetDAQlbPort());
2259 dbUser = fConfig->GetDAQlbUser();
2260 dbPass = fConfig->GetDAQlbPass();
2261 dbName = fConfig->GetDAQlbDB();
2262 }
57f50b3c 2263
9d733021 2264 fServer[system] = TSQLServer::Connect(dbHost.Data(), dbUser.Data(), dbPass.Data());
2265 if (!fServer[system] || !fServer[system]->IsConnected()) {
2266 if(system < 3)
2267 {
2268 AliError(Form("Can't establish connection to FXS database for %s",
2269 AliShuttleInterface::GetSystemName(system)));
2270 } else {
2271 AliError("Can't establish connection to Run logbook.");
57f50b3c 2272 }
9d733021 2273 if(fServer[system]) delete fServer[system];
2274 return kFALSE;
2bb7b766 2275 }
57f50b3c 2276
9d733021 2277 // Get tables
2278 TSQLResult* aResult=0;
2279 switch(system){
2280 case kDAQ:
2281 aResult = fServer[kDAQ]->GetTables(dbName.Data());
2282 break;
2283 case kDCS:
2284 aResult = fServer[kDCS]->GetTables(dbName.Data());
2285 break;
2286 case kHLT:
2287 aResult = fServer[kHLT]->GetTables(dbName.Data());
2288 break;
2289 default:
2290 aResult = fServer[3]->GetTables(dbName.Data());
2291 break;
2292 }
2293
2294 delete aResult;
2bb7b766 2295 return kTRUE;
2296}
57f50b3c 2297
9e080f92 2298//______________________________________________________________________________________________
9d733021 2299Bool_t AliShuttle::UpdateTable()
9e080f92 2300{
9827400b 2301 //
2302 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2303 //
9e080f92 2304
9d733021 2305 Bool_t result = kTRUE;
9e080f92 2306
9d733021 2307 for (UInt_t system=0; system<3; system++)
2308 {
2309 if(!fFXSCalled[system]) continue;
9e080f92 2310
9d733021 2311 // check connection, in case connect
2312 if (!Connect(system))
2313 {
2314 Log(fCurrentDetector, Form("UpdateTable - Couldn't connect to %s FXS database", GetSystemName(system)));
2315 result = kFALSE;
2316 continue;
9e080f92 2317 }
9e080f92 2318
9d733021 2319 TTimeStamp now; // now
2320
2321 // Loop on FXS list entries
2322 TIter iter(&fFXSlist[system]);
2323 TObjString *aFXSentry=0;
2324 while ((aFXSentry = dynamic_cast<TObjString*> (iter.Next())))
2325 {
2326 TString aFXSentrystr = aFXSentry->String();
2327 TObjArray *aFXSarray = aFXSentrystr.Tokenize("#!?!#");
2328 if (!aFXSarray || aFXSarray->GetEntries() != 2 )
2329 {
2330 Log(fCurrentDetector, Form("UpdateTable - error updating %s FXS entry. Check string: <%s>",
2331 GetSystemName(system), aFXSentrystr.Data()));
2332 if(aFXSarray) delete aFXSarray;
2333 result = kFALSE;
2334 continue;
2335 }
2336 const char* fileId = ((TObjString*) aFXSarray->At(0))->GetName();
2337 const char* source = ((TObjString*) aFXSarray->At(1))->GetName();
2338
2339 TString whereClause;
2340 if (system == kDAQ)
2341 {
2342 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DAQsource=\"%s\";",
2343 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2344 }
2345 else if (system == kDCS)
2346 {
2347 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\";",
2348 GetCurrentRun(), fCurrentDetector.Data(), fileId);
2349 }
2350 else if (system == kHLT)
2351 {
2352 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DDLnumbers=\"%s\";",
2353 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2354 }
2355
2356 delete aFXSarray;
9e080f92 2357
9d733021 2358 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2359 now.GetSec(), whereClause.Data());
9e080f92 2360
9d733021 2361 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
9e080f92 2362
9d733021 2363 // Query execution
2364 TSQLResult* aResult;
2365 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2366 if (!aResult)
2367 {
2368 Log(fCurrentDetector, Form("UpdateTable - %s db: can't execute SQL query <%s>",
2369 GetSystemName(system), sqlQuery.Data()));
2370 result = kFALSE;
2371 continue;
2372 }
2373 delete aResult;
9e080f92 2374 }
9e080f92 2375 }
2376
9d733021 2377 return result;
9e080f92 2378}
57f50b3c 2379
3301427a 2380//______________________________________________________________________________________________
2381Bool_t AliShuttle::UpdateTableFailCase()
2382{
9827400b 2383 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2384 // this is called in case the preprocessor is declared failed for the current run, because
2385 // the fields are updated only in case of success
3301427a 2386
2387 Bool_t result = kTRUE;
2388
2389 for (UInt_t system=0; system<3; system++)
2390 {
2391 // check connection, in case connect
2392 if (!Connect(system))
2393 {
2394 Log(fCurrentDetector, Form("UpdateTableFailCase - Couldn't connect to %s FXS database",
2395 GetSystemName(system)));
2396 result = kFALSE;
2397 continue;
2398 }
2399
2400 TTimeStamp now; // now
2401
2402 // Loop on FXS list entries
2403
2404 TString whereClause = Form("where run=%d and detector=\"%s\";",
2405 GetCurrentRun(), fCurrentDetector.Data());
2406
2407
2408 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2409 now.GetSec(), whereClause.Data());
2410
2411 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2412
2413 // Query execution
2414 TSQLResult* aResult;
2415 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2416 if (!aResult)
2417 {
2418 Log(fCurrentDetector, Form("UpdateTableFailCase - %s db: can't execute SQL query <%s>",
2419 GetSystemName(system), sqlQuery.Data()));
2420 result = kFALSE;
2421 continue;
2422 }
2423 delete aResult;
2424 }
2425
2426 return result;
2427}
2428
2bb7b766 2429//______________________________________________________________________________________________
2430Bool_t AliShuttle::UpdateShuttleLogbook(const char* detector, const char* status)
2431{
e7f62f16 2432 //
2433 // Update Shuttle logbook filling detector or shuttle_done column
2434 // ex. of usage: UpdateShuttleLogbook("PHOS", "DONE") or UpdateShuttleLogbook("shuttle_done")
2435 //
57f50b3c 2436
2bb7b766 2437 // check connection, in case connect
be48e3ea 2438 if(!Connect(3)){
2bb7b766 2439 Log("SHUTTLE", "UpdateShuttleLogbook - Couldn't connect to DAQ Logbook.");
2440 return kFALSE;
57f50b3c 2441 }
2442
2bb7b766 2443 TString detName(detector);
2444 TString setClause;
e7f62f16 2445 if(detName == "shuttle_done")
2446 {
2bb7b766 2447 setClause = "set shuttle_done=1";
e7f62f16 2448
2449 // Send the information to ML
2450 TMonaLisaText mlStatus("SHUTTLE_status", "Done");
2451
2452 TList mlList;
2453 mlList.Add(&mlStatus);
2454
2455 fMonaLisa->SendParameters(&mlList);
2bb7b766 2456 } else {
2bb7b766 2457 TString statusStr(status);
2458 if(statusStr.Contains("done", TString::kIgnoreCase) ||
2459 statusStr.Contains("failed", TString::kIgnoreCase)){
eba76848 2460 setClause = Form("set %s=\"%s\"", detector, status);
2bb7b766 2461 } else {
2462 Log("SHUTTLE",
2463 Form("UpdateShuttleLogbook - Invalid status <%s> for detector %s",
2464 status, detector));
2465 return kFALSE;
2466 }
2467 }
57f50b3c 2468
2bb7b766 2469 TString whereClause = Form("where run=%d", GetCurrentRun());
2470
441b0e9c 2471 TString sqlQuery = Form("update %s %s %s",
2472 fConfig->GetShuttlelbTable(), setClause.Data(), whereClause.Data());
57f50b3c 2473
2bb7b766 2474 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2475
2476 // Query execution
2477 TSQLResult* aResult;
be48e3ea 2478 aResult = dynamic_cast<TSQLResult*> (fServer[3]->Query(sqlQuery));
2bb7b766 2479 if (!aResult) {
2480 Log("SHUTTLE", Form("UpdateShuttleLogbook - Can't execute query <%s>", sqlQuery.Data()));
2481 return kFALSE;
57f50b3c 2482 }
2bb7b766 2483 delete aResult;
57f50b3c 2484
2485 return kTRUE;
2486}
2487
2488//______________________________________________________________________________________________
2bb7b766 2489Int_t AliShuttle::GetCurrentRun() const
2490{
9827400b 2491 //
2492 // Get current run from logbook entry
2493 //
57f50b3c 2494
2bb7b766 2495 return fLogbookEntry ? fLogbookEntry->GetRun() : -1;
57f50b3c 2496}
2497
2498//______________________________________________________________________________________________
2bb7b766 2499UInt_t AliShuttle::GetCurrentStartTime() const
2500{
9827400b 2501 //
2502 // get current start time
2503 //
57f50b3c 2504
2bb7b766 2505 return fLogbookEntry ? fLogbookEntry->GetStartTime() : 0;
57f50b3c 2506}
2507
2508//______________________________________________________________________________________________
2bb7b766 2509UInt_t AliShuttle::GetCurrentEndTime() const
2510{
9827400b 2511 //
2512 // get current end time from logbook entry
2513 //
57f50b3c 2514
2bb7b766 2515 return fLogbookEntry ? fLogbookEntry->GetEndTime() : 0;
57f50b3c 2516}
2517
b948db8d 2518//______________________________________________________________________________________________
2519void AliShuttle::Log(const char* detector, const char* message)
2520{
9827400b 2521 //
2522 // Fill log string with a message
2523 //
b948db8d 2524
36c99a6a 2525 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
84090f85 2526 if (dir == NULL) {
36c99a6a 2527 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE)) {
2528 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
84090f85 2529 return;
2530 }
b948db8d 2531
84090f85 2532 } else {
2533 gSystem->FreeDirectory(dir);
2534 }
b948db8d 2535
cb343cfd 2536 TString toLog = Form("%s (%d): %s - ", TTimeStamp(time(0)).AsString("s"), getpid(), detector);
e7f62f16 2537 if (GetCurrentRun() >= 0)
2538 toLog += Form("run %d - ", GetCurrentRun());
2bb7b766 2539 toLog += Form("%s", message);
2540
84090f85 2541 AliInfo(toLog.Data());
ffa29e93 2542
2543 // if we redirect the log output already to the file, leave here
2544 if (fOutputRedirected && strcmp(detector, "SHUTTLE") != 0)
2545 return;
b948db8d 2546
ffa29e93 2547 TString fileName = GetLogFileName(detector);
e7f62f16 2548
84090f85 2549 gSystem->ExpandPathName(fileName);
2550
2551 ofstream logFile;
2552 logFile.open(fileName, ofstream::out | ofstream::app);
2553
2554 if (!logFile.is_open()) {
2555 AliError(Form("Could not open file %s", fileName.Data()));
2556 return;
2557 }
7bfb2090 2558
84090f85 2559 logFile << toLog.Data() << "\n";
b948db8d 2560
84090f85 2561 logFile.close();
b948db8d 2562}
2bb7b766 2563
ffa29e93 2564//______________________________________________________________________________________________
2565TString AliShuttle::GetLogFileName(const char* detector) const
2566{
2567 //
2568 // returns the name of the log file for a given sub detector
2569 //
2570
2571 TString fileName;
2572
2573 if (GetCurrentRun() >= 0)
2574 fileName.Form("%s/%s_%d.log", GetShuttleLogDir(), detector, GetCurrentRun());
2575 else
2576 fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
2577
2578 return fileName;
2579}
2580
2bb7b766 2581//______________________________________________________________________________________________
2582Bool_t AliShuttle::Collect(Int_t run)
2583{
9827400b 2584 //
2585 // Collects conditions data for all UNPROCESSED run written to DAQ LogBook in case of run = -1 (default)
2586 // If a dedicated run is given this run is processed
2587 //
2588 // In operational mode, this is the Shuttle function triggered by the EOR signal.
2589 //
2bb7b766 2590
eba76848 2591 if (run == -1)
2592 Log("SHUTTLE","Collect - Shuttle called. Collecting conditions data for unprocessed runs");
2593 else
2594 Log("SHUTTLE", Form("Collect - Shuttle called. Collecting conditions data for run %d", run));
cb343cfd 2595
2596 SetLastAction("Starting");
2bb7b766 2597
2598 TString whereClause("where shuttle_done=0");
eba76848 2599 if (run != -1)
2600 whereClause += Form(" and run=%d", run);
2bb7b766 2601
2602 TObjArray shuttleLogbookEntries;
be48e3ea 2603 if (!QueryShuttleLogbook(whereClause, shuttleLogbookEntries))
2604 {
cb343cfd 2605 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2bb7b766 2606 return kFALSE;
2607 }
2608
9e080f92 2609 if (shuttleLogbookEntries.GetEntries() == 0)
2610 {
2611 if (run == -1)
2612 Log("SHUTTLE","Collect - Found no UNPROCESSED runs in Shuttle logbook");
2613 else
2614 Log("SHUTTLE", Form("Collect - Run %d is already DONE "
2615 "or it does not exist in Shuttle logbook", run));
2616 return kTRUE;
2617 }
2618
be48e3ea 2619 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2620 fFirstUnprocessed[iDet] = kTRUE;
2621
fc5a4708 2622 if (run != -1)
be48e3ea 2623 {
2624 // query Shuttle logbook for earlier runs, check if some detectors are unprocessed,
2625 // flag them into fFirstUnprocessed array
2626 TString whereClause(Form("where shuttle_done=0 and run < %d", run));
2627 TObjArray tmpLogbookEntries;
2628 if (!QueryShuttleLogbook(whereClause, tmpLogbookEntries))
2629 {
2630 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2631 return kFALSE;
2632 }
2633
2634 TIter iter(&tmpLogbookEntries);
2635 AliShuttleLogbookEntry* anEntry = 0;
2636 while ((anEntry = dynamic_cast<AliShuttleLogbookEntry*> (iter.Next())))
2637 {
2638 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2639 {
2640 if (anEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
2641 {
2642 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
2643 anEntry->GetRun(), GetDetName(iDet)));
2644 fFirstUnprocessed[iDet] = kFALSE;
2645 }
2646 }
2647
2648 }
2649
2650 }
2651
2652 if (!RetrieveConditionsData(shuttleLogbookEntries))
2653 {
cb343cfd 2654 Log("SHUTTLE", "Collect - Process of at least one run failed");
2bb7b766 2655 return kFALSE;
2656 }
2657
36c99a6a 2658 Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
eba76848 2659 return kTRUE;
2bb7b766 2660}
2661
2bb7b766 2662//______________________________________________________________________________________________
2663Bool_t AliShuttle::RetrieveConditionsData(const TObjArray& dateEntries)
2664{
9827400b 2665 //
2666 // Retrieve conditions data for all runs that aren't processed yet
2667 //
2bb7b766 2668
2669 Bool_t hasError = kFALSE;
2670
2671 TIter iter(&dateEntries);
2672 AliShuttleLogbookEntry* anEntry;
2673
2674 while ((anEntry = (AliShuttleLogbookEntry*) iter.Next())){
2675 if (!Process(anEntry)){
2676 hasError = kTRUE;
2677 }
4b95672b 2678
2679 // clean SHUTTLE temp directory
3301427a 2680 TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
2681 RemoveFile(filename.Data());
2bb7b766 2682 }
2683
2684 return hasError == kFALSE;
2685}
cb343cfd 2686
2687//______________________________________________________________________________________________
2688ULong_t AliShuttle::GetTimeOfLastAction() const
2689{
9827400b 2690 //
2691 // Gets time of last action
2692 //
2693
cb343cfd 2694 ULong_t tmp;
36c99a6a 2695
cb343cfd 2696 fMonitoringMutex->Lock();
be48e3ea 2697
cb343cfd 2698 tmp = fLastActionTime;
36c99a6a 2699
cb343cfd 2700 fMonitoringMutex->UnLock();
36c99a6a 2701
cb343cfd 2702 return tmp;
2703}
2704
2705//______________________________________________________________________________________________
2706const TString AliShuttle::GetLastAction() const
2707{
9827400b 2708 //
cb343cfd 2709 // returns a string description of the last action
9827400b 2710 //
cb343cfd 2711
2712 TString tmp;
36c99a6a 2713
cb343cfd 2714 fMonitoringMutex->Lock();
2715
2716 tmp = fLastAction;
2717
2718 fMonitoringMutex->UnLock();
2719
36c99a6a 2720 return tmp;
cb343cfd 2721}
2722
2723//______________________________________________________________________________________________
2724void AliShuttle::SetLastAction(const char* action)
2725{
9827400b 2726 //
cb343cfd 2727 // updates the monitoring variables
9827400b 2728 //
36c99a6a 2729
cb343cfd 2730 fMonitoringMutex->Lock();
36c99a6a 2731
cb343cfd 2732 fLastAction = action;
2733 fLastActionTime = time(0);
2734
2735 fMonitoringMutex->UnLock();
2736}
eba76848 2737
2738//______________________________________________________________________________________________
2739const char* AliShuttle::GetRunParameter(const char* param)
2740{
9827400b 2741 //
2742 // returns run parameter read from DAQ logbook
2743 //
eba76848 2744
2745 if(!fLogbookEntry) {
2746 AliError("No logbook entry!");
2747 return 0;
2748 }
2749
2750 return fLogbookEntry->GetRunParameter(param);
2751}
57c1a579 2752
d386d623 2753//______________________________________________________________________________________________
9827400b 2754AliCDBEntry* AliShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
d386d623 2755{
9827400b 2756 //
2757 // returns object from OCDB valid for current run
2758 //
d386d623 2759
9827400b 2760 if (fTestMode & kErrorOCDB)
2761 {
2762 Log(detector, "GetFromOCDB - In TESTMODE - Simulating error with OCDB");
2763 return 0;
2764 }
2765
d386d623 2766 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
2767 if (!sto)
2768 {
9827400b 2769 Log(detector, "GetFromOCDB - Cannot activate main OCDB for query!");
d386d623 2770 return 0;
2771 }
2772
2773 return dynamic_cast<AliCDBEntry*> (sto->Get(path, GetCurrentRun()));
2774}
2775
57c1a579 2776//______________________________________________________________________________________________
2777Bool_t AliShuttle::SendMail()
2778{
9827400b 2779 //
2780 // sends a mail to the subdetector expert in case of preprocessor error
2781 //
2782
2783 if (fTestMode != kNone)
2784 return kTRUE;
57c1a579 2785
36c99a6a 2786 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
57c1a579 2787 if (dir == NULL)
2788 {
36c99a6a 2789 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
57c1a579 2790 {
36c99a6a 2791 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
57c1a579 2792 return kFALSE;
2793 }
2794
2795 } else {
2796 gSystem->FreeDirectory(dir);
2797 }
2798
2799 TString bodyFileName;
36c99a6a 2800 bodyFileName.Form("%s/mail.body", GetShuttleLogDir());
57c1a579 2801 gSystem->ExpandPathName(bodyFileName);
2802
2803 ofstream mailBody;
2804 mailBody.open(bodyFileName, ofstream::out);
2805
2806 if (!mailBody.is_open())
2807 {
2808 AliError(Form("Could not open mail body file %s", bodyFileName.Data()));
2809 return kFALSE;
2810 }
2811
2812 TString to="";
2813 TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
2814 TObjString *anExpert=0;
2815 while ((anExpert = (TObjString*) iterExperts.Next()))
2816 {
2817 to += Form("%s,", anExpert->GetName());
2818 }
2819 to.Remove(to.Length()-1);
909732f7 2820 AliDebug(2, Form("to: %s",to.Data()));
57c1a579 2821
86aa42c3 2822 if (to.IsNull()) {
36c99a6a 2823 AliInfo("List of detector responsibles not yet set!");
2824 return kFALSE;
2825 }
2826
57c1a579 2827 TString cc="alberto.colla@cern.ch";
2828
546242fb 2829 TString subject = Form("%s Shuttle preprocessor FAILED in run %d !",
57c1a579 2830 fCurrentDetector.Data(), GetCurrentRun());
909732f7 2831 AliDebug(2, Form("subject: %s", subject.Data()));
57c1a579 2832
2833 TString body = Form("Dear %s expert(s), \n\n", fCurrentDetector.Data());
2834 body += Form("SHUTTLE just detected that your preprocessor "
546242fb 2835 "failed processing run %d!!\n\n", GetCurrentRun());
2836 body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n", fCurrentDetector.Data());
2837 body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
2838 body += Form("Find the %s log for the current run on \n\n"
2839 "\thttp://pcalishuttle01.cern.ch:8880/logs/%s_%d.log \n\n",
2840 fCurrentDetector.Data(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 2841 body += Form("The last 10 lines of %s log file are following:\n\n");
2842
909732f7 2843 AliDebug(2, Form("Body begin: %s", body.Data()));
57c1a579 2844
2845 mailBody << body.Data();
2846 mailBody.close();
2847 mailBody.open(bodyFileName, ofstream::out | ofstream::app);
2848
9d733021 2849 TString logFileName = Form("%s/%s_%d.log", GetShuttleLogDir(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 2850 TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
2851 if (gSystem->Exec(tailCommand.Data()))
2852 {
2853 mailBody << Form("%s log file not found ...\n\n", fCurrentDetector.Data());
2854 }
2855
2856 TString endBody = Form("------------------------------------------------------\n\n");
36c99a6a 2857 endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
2858 endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
546242fb 2859 endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
57c1a579 2860
909732f7 2861 AliDebug(2, Form("Body end: %s", endBody.Data()));
57c1a579 2862
2863 mailBody << endBody.Data();
2864
2865 mailBody.close();
2866
2867 // send mail!
2868 TString mailCommand = Form("mail -s \"%s\" -c %s %s < %s",
2869 subject.Data(),
2870 cc.Data(),
2871 to.Data(),
2872 bodyFileName.Data());
909732f7 2873 AliDebug(2, Form("mail command: %s", mailCommand.Data()));
57c1a579 2874
2875 Bool_t result = gSystem->Exec(mailCommand.Data());
2876
2877 return result == 0;
2878}
d386d623 2879
441b0e9c 2880//______________________________________________________________________________________________
9827400b 2881const char* AliShuttle::GetRunType()
441b0e9c 2882{
9827400b 2883 //
2884 // returns run type read from "run type" logbook
2885 //
441b0e9c 2886
2887 if(!fLogbookEntry) {
2888 AliError("No logbook entry!");
2889 return 0;
2890 }
2891
9827400b 2892 return fLogbookEntry->GetRunType();
441b0e9c 2893}
2894
d386d623 2895//______________________________________________________________________________________________
2896void AliShuttle::SetShuttleTempDir(const char* tmpDir)
2897{
9827400b 2898 //
2899 // sets Shuttle temp directory
2900 //
d386d623 2901
2902 fgkShuttleTempDir = gSystem->ExpandPathName(tmpDir);
2903}
2904
2905//______________________________________________________________________________________________
2906void AliShuttle::SetShuttleLogDir(const char* logDir)
2907{
9827400b 2908 //
2909 // sets Shuttle log directory
2910 //
d386d623 2911
2912 fgkShuttleLogDir = gSystem->ExpandPathName(logDir);
2913}