]> git.uio.no Git - u/mrichter/AliRoot.git/blame - SHUTTLE/AliShuttle.cxx
git://git.uio.no / u / mrichter / AliRoot.git / blame
? search:
summary | shortlog | log | commit | commitdiff | tree
blob | blame (incremental) | history | HEAD
Fixing segfault in example components. Bug #29877.
[u/mrichter/AliRoot.git] / SHUTTLE / AliShuttle.cxx
CommitLineData
73abe331 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
b41b252a 18Revision 1.55 2007/08/06 12:26:40 acolla
19Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
20read from the run logbook.
21
4859271b 22Revision 1.54 2007/07/12 09:51:25 jgrosseo
23removed duplicated log message in GetFile
24
4f0749a8 25Revision 1.53 2007/07/12 09:26:28 jgrosseo
26updating hlt fxs base path
27
42fde080 28Revision 1.52 2007/07/12 08:06:45 jgrosseo
29adding log messages in getfile... functions
30adding not implemented copy constructor in alishuttleconfigholder
31
1bcd28db 32Revision 1.51 2007/07/03 17:24:52 acolla
33root moved to v5-16-00. TFileMerger->Cp moved to TFile::Cp.
34
a986b218 35Revision 1.50 2007/07/02 17:19:32 acolla
36preprocessor is run in a temp directory that is removed when process is finished.
37
5bac2bde 38Revision 1.49 2007/06/29 10:45:06 acolla
39Number of columns in MySql Shuttle logbook increased by one (HLT added)
40
db99d43e 41Revision 1.48 2007/06/21 13:06:19 acolla
42GetFileSources returns dummy list with 1 source if system=DCS (better than
43returning error as it was)
44
6297b37d 45Revision 1.47 2007/06/19 17:28:56 acolla
46HLT updated; missing map bug removed.
47
dc25836b 48Revision 1.46 2007/06/09 13:01:09 jgrosseo
49Switching to retrieval of several DCS DPs at a time (multiDPrequest)
50
a038aa70 51Revision 1.45 2007/05/30 06:35:20 jgrosseo
52Adding functionality to the Shuttle/TestShuttle:
53o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
54o) Function to retrieve list of IDs for a given source (GetFileIDs)
55These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
56Example code has been added to the TestProcessor in TestShuttle
57
4a33bdd9 58Revision 1.44 2007/05/11 16:09:32 acolla
59Reference files for ITS, MUON and PHOS are now stored in OfflineDetName/OnlineDetName/run_...
60example: ITS/SPD/100_filename.root
61
2d9019b4 62Revision 1.43 2007/05/10 09:59:51 acolla
63Various bug fixes in StoreRefFilesToGrid; Cleaning of reference storage before processing detector (CleanReferenceStorage)
64
546242fb 65Revision 1.42 2007/05/03 08:01:39 jgrosseo
66typo in last commit :-(
67
8b739301 68Revision 1.41 2007/05/03 08:00:48 jgrosseo
69fixing log message when pp want to skip dcs value retrieval
70
651fdaab 71Revision 1.40 2007/04/27 07:06:48 jgrosseo
72GetFileSources returns empty list in case of no files, but successful query
73No mails sent in testmode
74
86aa42c3 75Revision 1.39 2007/04/17 12:43:57 acolla
76Correction in StoreOCDB; change of text in mail to detector expert
77
26758fce 78Revision 1.38 2007/04/12 08:26:18 jgrosseo
79updated comment
80
3c2a21c8 81Revision 1.37 2007/04/10 16:53:14 jgrosseo
82redirecting sub detector stdout, stderr to sub detector log file
83
3d8bc902 84Revision 1.35 2007/04/04 16:26:38 acolla
851. Re-organization of function calls in TestPreprocessor to make it more meaningful.
862. Added missing dependency in test preprocessors.
873. in AliShuttle.cxx: processing time and memory consumption info on a single line.
88
886d60e6 89Revision 1.34 2007/04/04 10:33:36 jgrosseo
901) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
91In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
92
932) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
94
953) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
96
974) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
98
995) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
100If you always need DCS data (like before), you do not need to implement it.
101
1026) The run type has been added to the monitoring page
103
9827400b 104Revision 1.33 2007/04/03 13:56:01 acolla
105Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
106run type.
107
3301427a 108Revision 1.32 2007/02/28 10:41:56 acolla
109Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
110AliPreprocessor::GetRunType() function.
111Added some ldap definition files.
112
d386d623 113Revision 1.30 2007/02/13 11:23:21 acolla
114Moved getters and setters of Shuttle's main OCDB/Reference, local
115OCDB/Reference, temp and log folders to AliShuttleInterface
116
9d733021 117Revision 1.27 2007/01/30 17:52:42 jgrosseo
118adding monalisa monitoring
119
e7f62f16 120Revision 1.26 2007/01/23 19:20:03 acolla
121Removed old ldif files, added TOF, MCH ldif files. Added some options in
122AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
123SetShuttleLogDir
124
36c99a6a 125Revision 1.25 2007/01/15 19:13:52 acolla
126Moved some AliInfo to AliDebug in SendMail function
127
fc5a4708 128Revision 1.21 2006/12/07 08:51:26 jgrosseo
129update (alberto):
130table, db names in ldap configuration
131added GRP preprocessor
132DCS data can also be retrieved by data point
133
2c15234c 134Revision 1.20 2006/11/16 16:16:48 jgrosseo
135introducing strict run ordering flag
136removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
137
be48e3ea 138Revision 1.19 2006/11/06 14:23:04 jgrosseo
139major update (Alberto)
140o) reading of run parameters from the logbook
141o) online offline naming conversion
142o) standalone DCSclient package
143
eba76848 144Revision 1.18 2006/10/20 15:22:59 jgrosseo
145o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
146o) Merging Collect, CollectAll, CollectNew function
147o) Removing implementation of empty copy constructors (declaration still there!)
148
cb343cfd 149Revision 1.17 2006/10/05 16:20:55 jgrosseo
150adapting to new CDB classes
151
6ec0e06c 152Revision 1.16 2006/10/05 15:46:26 jgrosseo
153applying to the new interface
154
481441a2 155Revision 1.15 2006/10/02 16:38:39 jgrosseo
156update (alberto):
157fixed memory leaks
158storing of objects that failed to be stored to the grid before
159interfacing of shuttle status table in daq system
160
2bb7b766 161Revision 1.14 2006/08/29 09:16:05 jgrosseo
162small update
163
85a80aa9 164Revision 1.13 2006/08/15 10:50:00 jgrosseo
165effc++ corrections (alberto)
166
4f0ab988 167Revision 1.12 2006/08/08 14:19:29 jgrosseo
168Update to shuttle classes (Alberto)
169
170- Possibility to set the full object's path in the Preprocessor's and
171Shuttle's Store functions
172- Possibility to extend the object's run validity in the same classes
173("startValidity" and "validityInfinite" parameters)
174- Implementation of the StoreReferenceData function to store reference
175data in a dedicated CDB storage.
176
84090f85 177Revision 1.11 2006/07/21 07:37:20 jgrosseo
178last run is stored after each run
179
7bfb2090 180Revision 1.10 2006/07/20 09:54:40 jgrosseo
181introducing status management: The processing per subdetector is divided into several steps,
182after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
183can keep track of the number of failures and skips further processing after a certain threshold is
184exceeded. These thresholds can be configured in LDAP.
185
5164a766 186Revision 1.9 2006/07/19 10:09:55 jgrosseo
187new configuration, accesst to DAQ FES (Alberto)
188
57f50b3c 189Revision 1.8 2006/07/11 12:44:36 jgrosseo
190adding parameters for extended validity range of data produced by preprocessor
191
17111222 192Revision 1.7 2006/07/10 14:37:09 jgrosseo
193small fix + todo comment
194
e090413b 195Revision 1.6 2006/07/10 13:01:41 jgrosseo
196enhanced storing of last sucessfully processed run (alberto)
197
a7160fe9 198Revision 1.5 2006/07/04 14:59:57 jgrosseo
199revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
200
45a493ce 201Revision 1.4 2006/06/12 09:11:16 jgrosseo
202coding conventions (Alberto)
203
58bc3020 204Revision 1.3 2006/06/06 14:26:40 jgrosseo
205o) removed files that were moved to STEER
206o) shuttle updated to follow the new interface (Alberto)
207
b948db8d 208Revision 1.2 2006/03/07 07:52:34 hristov
209New version (B.Yordanov)
210
d477ad88 211Revision 1.6 2005/11/19 17:19:14 byordano
212RetrieveDATEEntries and RetrieveConditionsData added
213
214Revision 1.5 2005/11/19 11:09:27 byordano
215AliShuttle declaration added
216
217Revision 1.4 2005/11/17 17:47:34 byordano
218TList changed to TObjArray
219
220Revision 1.3 2005/11/17 14:43:23 byordano
221import to local CVS
222
223Revision 1.1.1.1 2005/10/28 07:33:58 hristov
224Initial import as subdirectory in AliRoot
225
73abe331 226Revision 1.2 2005/09/13 08:41:15 byordano
227default startTime endTime added
228
229Revision 1.4 2005/08/30 09:13:02 byordano
230some docs added
231
232Revision 1.3 2005/08/29 21:15:47 byordano
233some docs added
234
235*/
236
237//
238// This class is the main manager for AliShuttle.
239// It organizes the data retrieval from DCS and call the
b948db8d 240// interface methods of AliPreprocessor.
73abe331 241// For every detector in AliShuttleConfgi (see AliShuttleConfig),
242// data for its set of aliases is retrieved. If there is registered
b948db8d 243// AliPreprocessor for this detector then it will be used
244// accroding to the schema (see AliPreprocessor).
245// If there isn't registered AliPreprocessor than the retrieved
73abe331 246// data is stored automatically to the undelying AliCDBStorage.
247// For detSpec is used the alias name.
248//
249
250#include "AliShuttle.h"
251
252#include "AliCDBManager.h"
253#include "AliCDBStorage.h"
254#include "AliCDBId.h"
84090f85 255#include "AliCDBRunRange.h"
256#include "AliCDBPath.h"
5164a766 257#include "AliCDBEntry.h"
73abe331 258#include "AliShuttleConfig.h"
eba76848 259#include "DCSClient/AliDCSClient.h"
73abe331 260#include "AliLog.h"
b948db8d 261#include "AliPreprocessor.h"
5164a766 262#include "AliShuttleStatus.h"
2bb7b766 263#include "AliShuttleLogbookEntry.h"
73abe331 264
57f50b3c 265#include <TSystem.h>
58bc3020 266#include <TObject.h>
b948db8d 267#include <TString.h>
57f50b3c 268#include <TTimeStamp.h>
73abe331 269#include <TObjString.h>
57f50b3c 270#include <TSQLServer.h>
271#include <TSQLResult.h>
272#include <TSQLRow.h>
cb343cfd 273#include <TMutex.h>
9827400b 274#include <TSystemDirectory.h>
275#include <TSystemFile.h>
a986b218 276#include <TFile.h>
9827400b 277#include <TFileMerger.h>
278#include <TGrid.h>
279#include <TGridResult.h>
73abe331 280
e7f62f16 281#include <TMonaLisaWriter.h>
282
5164a766 283#include <fstream>
284
cb343cfd 285#include <sys/types.h>
286#include <sys/wait.h>
287
73abe331 288ClassImp(AliShuttle)
289
b948db8d 290//______________________________________________________________________________________________
291AliShuttle::AliShuttle(const AliShuttleConfig* config,
292 UInt_t timeout, Int_t retries):
4f0ab988 293fConfig(config),
294fTimeout(timeout), fRetries(retries),
295fPreprocessorMap(),
2bb7b766 296fLogbookEntry(0),
eba76848 297fCurrentDetector(),
85a80aa9 298fStatusEntry(0),
cb343cfd 299fMonitoringMutex(0),
eba76848 300fLastActionTime(0),
e7f62f16 301fLastAction(),
9827400b 302fMonaLisa(0),
303fTestMode(kNone),
ffa29e93 304fReadTestMode(kFALSE),
305fOutputRedirected(kFALSE)
73abe331 306{
307 //
308 // config: AliShuttleConfig used
73abe331 309 // timeout: timeout used for AliDCSClient connection
310 // retries: the number of retries in case of connection error.
311 //
312
57f50b3c 313 if (!fConfig->IsValid()) AliFatal("********** !!!!! Invalid configuration !!!!! **********");
be48e3ea 314 for(int iSys=0;iSys<4;iSys++) {
57f50b3c 315 fServer[iSys]=0;
be48e3ea 316 if (iSys < 3)
2c15234c 317 fFXSlist[iSys].SetOwner(kTRUE);
57f50b3c 318 }
2bb7b766 319 fPreprocessorMap.SetOwner(kTRUE);
be48e3ea 320
321 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
322 fFirstUnprocessed[iDet] = kFALSE;
323
cb343cfd 324 fMonitoringMutex = new TMutex();
58bc3020 325}
326
b948db8d 327//______________________________________________________________________________________________
57f50b3c 328AliShuttle::~AliShuttle()
58bc3020 329{
9827400b 330 //
331 // destructor
332 //
58bc3020 333
b948db8d 334 fPreprocessorMap.DeleteAll();
be48e3ea 335 for(int iSys=0;iSys<4;iSys++)
57f50b3c 336 if(fServer[iSys]) {
337 fServer[iSys]->Close();
338 delete fServer[iSys];
eba76848 339 fServer[iSys] = 0;
57f50b3c 340 }
2bb7b766 341
342 if (fStatusEntry){
343 delete fStatusEntry;
344 fStatusEntry = 0;
345 }
cb343cfd 346
347 if (fMonitoringMutex)
348 {
349 delete fMonitoringMutex;
350 fMonitoringMutex = 0;
351 }
73abe331 352}
353
b948db8d 354//______________________________________________________________________________________________
57f50b3c 355void AliShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
58bc3020 356{
73abe331 357 //
b948db8d 358 // Registers new AliPreprocessor.
73abe331 359 // It uses GetName() for indentificator of the pre processor.
360 // The pre processor is registered it there isn't any other
361 // with the same identificator (GetName()).
362 //
363
eba76848 364 const char* detName = preprocessor->GetName();
365 if(GetDetPos(detName) < 0)
366 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
367
368 if (fPreprocessorMap.GetValue(detName)) {
369 AliWarning(Form("AliPreprocessor %s is already registered!", detName));
73abe331 370 return;
371 }
372
eba76848 373 fPreprocessorMap.Add(new TObjString(detName), preprocessor);
73abe331 374}
b948db8d 375//______________________________________________________________________________________________
3301427a 376Bool_t AliShuttle::Store(const AliCDBPath& path, TObject* object,
84090f85 377 AliCDBMetaData* metaData, Int_t validityStart, Bool_t validityInfinite)
73abe331 378{
9827400b 379 // Stores a CDB object in the storage for offline reconstruction. Objects that are not needed for
380 // offline reconstruction, but should be stored anyway (e.g. for debugging) should NOT be stored
381 // using this function. Use StoreReferenceData instead!
382 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
383 // finishes the data are transferred to the main storage (Grid).
b948db8d 384
3301427a 385 return StoreLocally(fgkLocalCDB, path, object, metaData, validityStart, validityInfinite);
84090f85 386}
387
388//______________________________________________________________________________________________
3301427a 389Bool_t AliShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 390{
9827400b 391 // Stores a CDB object in the storage for reference data. This objects will not be available during
392 // offline reconstrunction. Use this function for reference data only!
393 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
394 // finishes the data are transferred to the main storage (Grid).
85a80aa9 395
3301427a 396 return StoreLocally(fgkLocalRefStorage, path, object, metaData);
85a80aa9 397}
398
399//______________________________________________________________________________________________
3301427a 400Bool_t AliShuttle::StoreLocally(const TString& localUri,
85a80aa9 401 const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
402 Int_t validityStart, Bool_t validityInfinite)
403{
9827400b 404 // Store object temporarily in local storage. Parameters are passed by Store and StoreReferenceData functions.
405 // when the preprocessor finishes the data are transferred to the main storage (Grid).
406 // The parameters are:
407 // 1) Uri of the backup storage (Local)
408 // 2) the object's path.
409 // 3) the object to be stored
410 // 4) the metaData to be associated with the object
411 // 5) the validity start run number w.r.t. the current run,
412 // if the data is valid only for this run leave the default 0
413 // 6) specifies if the calibration data is valid for infinity (this means until updated),
414 // typical for calibration runs, the default is kFALSE
415 //
416 // returns 0 if fail, 1 otherwise
84090f85 417
9827400b 418 if (fTestMode & kErrorStorage)
419 {
420 Log(fCurrentDetector, "StoreLocally - In TESTMODE - Simulating error while storing locally");
421 return kFALSE;
422 }
423
3301427a 424 const char* cdbType = (localUri == fgkLocalCDB) ? "CDB" : "Reference";
2bb7b766 425
85a80aa9 426 Int_t firstRun = GetCurrentRun() - validityStart;
84090f85 427 if(firstRun < 0) {
9827400b 428 AliWarning("First valid run happens to be less than 0! Setting it to 0.");
84090f85 429 firstRun=0;
430 }
431
432 Int_t lastRun = -1;
433 if(validityInfinite) {
434 lastRun = AliCDBRunRange::Infinity();
435 } else {
436 lastRun = GetCurrentRun();
437 }
438
3301427a 439 // Version is set to current run, it will be used later to transfer data to Grid
440 AliCDBId id(path, firstRun, lastRun, GetCurrentRun(), -1);
2bb7b766 441
442 if(! dynamic_cast<TObjString*> (metaData->GetProperty("RunUsed(TObjString)"))){
443 TObjString runUsed = Form("%d", GetCurrentRun());
9e080f92 444 metaData->SetProperty("RunUsed(TObjString)", runUsed.Clone());
2bb7b766 445 }
84090f85 446
3301427a 447 Bool_t result = kFALSE;
84090f85 448
3301427a 449 if (!(AliCDBManager::Instance()->GetStorage(localUri))) {
450 Log("SHUTTLE", Form("StoreLocally - Cannot activate local %s storage", cdbType));
84090f85 451 } else {
3301427a 452 result = AliCDBManager::Instance()->GetStorage(localUri)
84090f85 453 ->Put(object, id, metaData);
454 }
455
456 if(!result) {
457
9827400b 458 Log(fCurrentDetector, Form("StoreLocally - Can't store object <%s>!", id.ToString().Data()));
3301427a 459 }
2bb7b766 460
3301427a 461 return result;
462}
84090f85 463
3301427a 464//______________________________________________________________________________________________
465Bool_t AliShuttle::StoreOCDB()
466{
9827400b 467 //
468 // Called when preprocessor ends successfully or when previous storage attempt failed (kStoreError status)
469 // Calls underlying StoreOCDB(const char*) function twice, for OCDB and Reference storage.
470 // Then calls StoreRefFilesToGrid to store reference files.
471 //
472
473 if (fTestMode & kErrorGrid)
474 {
475 Log("SHUTTLE", "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
476 Log(fCurrentDetector, "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
477 return kFALSE;
478 }
479
2d9019b4 480 Log("SHUTTLE","Storing OCDB data ...");
86aa42c3 481 Bool_t resultCDB = StoreOCDB(fgkMainCDB);
482
2d9019b4 483 Log("SHUTTLE","Storing reference data ...");
3301427a 484 Bool_t resultRef = StoreOCDB(fgkMainRefStorage);
9827400b 485
2d9019b4 486 Log("SHUTTLE","Storing reference files ...");
9827400b 487 Bool_t resultRefFiles = StoreRefFilesToGrid();
488
489 return resultCDB && resultRef && resultRefFiles;
3301427a 490}
491
492//______________________________________________________________________________________________
493Bool_t AliShuttle::StoreOCDB(const TString& gridURI)
494{
495 //
496 // Called by StoreOCDB(), performs actual storage to the main OCDB and reference storages (Grid)
497 //
498
499 TObjArray* gridIds=0;
500
501 Bool_t result = kTRUE;
502
503 const char* type = 0;
504 TString localURI;
505 if(gridURI == fgkMainCDB) {
506 type = "OCDB";
507 localURI = fgkLocalCDB;
508 } else if(gridURI == fgkMainRefStorage) {
509 type = "reference";
510 localURI = fgkLocalRefStorage;
511 } else {
512 AliError(Form("Invalid storage URI: %s", gridURI.Data()));
513 return kFALSE;
514 }
515
516 AliCDBManager* man = AliCDBManager::Instance();
517
518 AliCDBStorage *gridSto = man->GetStorage(gridURI);
519 if(!gridSto) {
520 Log("SHUTTLE",
521 Form("StoreOCDB - cannot activate main %s storage", type));
522 return kFALSE;
523 }
524
525 gridIds = gridSto->GetQueryCDBList();
526
527 // get objects previously stored in local CDB
528 AliCDBStorage *localSto = man->GetStorage(localURI);
529 if(!localSto) {
530 Log("SHUTTLE",
531 Form("StoreOCDB - cannot activate local %s storage", type));
532 return kFALSE;
533 }
534 AliCDBPath aPath(GetOfflineDetName(fCurrentDetector.Data()),"*","*");
535 // Local objects were stored with current run as Grid version!
536 TList* localEntries = localSto->GetAll(aPath.GetPath(), GetCurrentRun(), GetCurrentRun());
537 localEntries->SetOwner(1);
538
539 // loop on local stored objects
540 TIter localIter(localEntries);
541 AliCDBEntry *aLocEntry = 0;
542 while((aLocEntry = dynamic_cast<AliCDBEntry*> (localIter.Next()))){
543 aLocEntry->SetOwner(1);
544 AliCDBId aLocId = aLocEntry->GetId();
545 aLocEntry->SetVersion(-1);
546 aLocEntry->SetSubVersion(-1);
547
548 // If local object is valid up to infinity we store it only if it is
549 // the first unprocessed run!
550 if (aLocId.GetLastRun() == AliCDBRunRange::Infinity() &&
551 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
552 {
553 Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
554 "there are previous unprocessed runs!",
555 fCurrentDetector.Data(), aLocId.GetPath().Data()));
556 continue;
557 }
558
559 // loop on Grid valid Id's
560 Bool_t store = kTRUE;
561 TIter gridIter(gridIds);
562 AliCDBId* aGridId = 0;
563 while((aGridId = dynamic_cast<AliCDBId*> (gridIter.Next()))){
564 if(aGridId->GetPath() != aLocId.GetPath()) continue;
565 // skip all objects valid up to infinity
566 if(aGridId->GetLastRun() == AliCDBRunRange::Infinity()) continue;
567 // if we get here, it means there's already some more recent object stored on Grid!
568 store = kFALSE;
569 break;
570 }
571
572 // If we get here, the file can be stored!
573 Bool_t storeOk = gridSto->Put(aLocEntry);
574 if(!store || storeOk){
575
576 if (!store)
577 {
578 Log(fCurrentDetector.Data(),
579 Form("StoreOCDB - A more recent object already exists in %s storage: <%s>",
580 type, aGridId->ToString().Data()));
581 } else {
582 Log("SHUTTLE",
583 Form("StoreOCDB - Object <%s> successfully put into %s storage",
584 aLocId.ToString().Data(), type));
2d9019b4 585 Log(fCurrentDetector.Data(),
586 Form("StoreOCDB - Object <%s> successfully put into %s storage",
587 aLocId.ToString().Data(), type));
3301427a 588 }
84090f85 589
3301427a 590 // removing local filename...
591 TString filename;
592 localSto->IdToFilename(aLocId, filename);
593 AliInfo(Form("Removing local file %s", filename.Data()));
594 RemoveFile(filename.Data());
595 continue;
596 } else {
597 Log("SHUTTLE",
598 Form("StoreOCDB - Grid %s storage of object <%s> failed",
599 type, aLocId.ToString().Data()));
2d9019b4 600 Log(fCurrentDetector.Data(),
601 Form("StoreOCDB - Grid %s storage of object <%s> failed",
602 type, aLocId.ToString().Data()));
3301427a 603 result = kFALSE;
b948db8d 604 }
605 }
3301427a 606 localEntries->Clear();
2bb7b766 607
b948db8d 608 return result;
3301427a 609}
610
546242fb 611//______________________________________________________________________________________________
612Bool_t AliShuttle::CleanReferenceStorage(const char* detector)
613{
2d9019b4 614 // clears the directory used to store reference files of a given subdetector
546242fb 615
616 AliCDBManager* man = AliCDBManager::Instance();
617 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
2d9019b4 618 TString localBaseFolder = sto->GetBaseFolder();
619
620 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
621
622 Log("SHUTTLE", Form("Cleaning %s", targetDir.Data()));
623
624 TString begin;
625 begin.Form("%d_", GetCurrentRun());
626
627 TSystemDirectory* baseDir = new TSystemDirectory("/", targetDir);
628 if (!baseDir)
629 return kTRUE;
630
631 TList* dirList = baseDir->GetListOfFiles();
632 delete baseDir;
633
634 if (!dirList) return kTRUE;
635
636 if (dirList->GetEntries() < 3)
637 {
638 delete dirList;
639 return kTRUE;
640 }
641
642 Int_t nDirs = 0, nDel = 0;
643 TIter dirIter(dirList);
644 TSystemFile* entry = 0;
546242fb 645
2d9019b4 646 Bool_t success = kTRUE;
546242fb 647
2d9019b4 648 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
649 {
650 if (entry->IsDirectory())
651 continue;
652
653 TString fileName(entry->GetName());
654 if (!fileName.BeginsWith(begin))
655 continue;
656
657 nDirs++;
658
659 // delete file
660 Int_t result = gSystem->Unlink(fileName.Data());
661
662 if (result)
663 {
664 Log("SHUTTLE", Form("Could not delete file %s!", fileName.Data()));
665 success = kFALSE;
666 } else {
667 nDel++;
668 }
669 }
670
671 if(nDirs > 0)
672 Log("SHUTTLE", Form("CleanReferenceStorage - %d (over %d) reference files in folder %s were deleted.",
673 nDel, nDirs, targetDir.Data()));
674
675
676 delete dirList;
677 return success;
678
679
680
681
682
546242fb 683
684 Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
685 if (result == 0)
686 {
687 // delete directory
688 result = gSystem->Exec(Form("rm -r %s", targetDir.Data()));
689 if (result != 0)
690 {
691 Log("SHUTTLE", Form("StoreReferenceFile - Could not clear directory %s", targetDir.Data()));
692 return kFALSE;
693 }
694 }
695
696 result = gSystem->mkdir(targetDir, kTRUE);
697 if (result != 0)
698 {
699 Log("SHUTTLE", Form("StoreReferenceFile - Error creating base directory %s", targetDir.Data()));
700 return kFALSE;
701 }
702
703 return kTRUE;
704}
705
9827400b 706//______________________________________________________________________________________________
707Bool_t AliShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
708{
709 //
3c2a21c8 710 // Stores reference file directly (without opening it). This function stores the file locally.
9827400b 711 //
3c2a21c8 712 // The file is stored under the following location:
713 // <base folder of local reference storage>/<DET>/<RUN#>_<gridFileName>
714 // where <gridFileName> is the second parameter given to the function
715 //
9827400b 716
717 if (fTestMode & kErrorStorage)
718 {
719 Log(fCurrentDetector, "StoreReferenceFile - In TESTMODE - Simulating error while storing locally");
720 return kFALSE;
721 }
722
723 AliCDBManager* man = AliCDBManager::Instance();
724 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
725
726 TString localBaseFolder = sto->GetBaseFolder();
727
2d9019b4 728 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
9827400b 729
2d9019b4 730 //try to open folder, if does not exist
731 void* dir = gSystem->OpenDirectory(targetDir.Data());
732 if (dir == NULL) {
733 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
734 Log("SHUTTLE", Form("Can't open directory <%s>", targetDir.Data()));
735 return kFALSE;
736 }
737
738 } else {
739 gSystem->FreeDirectory(dir);
740 }
741
9827400b 742 TString target;
743 target.Form("%s/%d_%s", targetDir.Data(), GetCurrentRun(), gridFileName);
744
546242fb 745 Int_t result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
9827400b 746 if (result)
747 {
546242fb 748 Log("SHUTTLE", Form("StoreReferenceFile - %s does not exist", localFile));
749 return kFALSE;
9827400b 750 }
546242fb 751
9827400b 752 result = gSystem->CopyFile(localFile, target);
753
754 if (result == 0)
755 {
2d9019b4 756 Log("SHUTTLE", Form("StoreReferenceFile - File %s stored locally to %s", localFile, target.Data()));
9827400b 757 return kTRUE;
758 }
759 else
760 {
2d9019b4 761 Log("SHUTTLE", Form("StoreReferenceFile - Could not store file %s to %s!. Error code = %d",
546242fb 762 localFile, target.Data(), result));
9827400b 763 return kFALSE;
764 }
765}
766
767//______________________________________________________________________________________________
768Bool_t AliShuttle::StoreRefFilesToGrid()
769{
770 //
771 // Transfers the reference file to the Grid.
9827400b 772 //
86aa42c3 773 // The files are stored under the following location:
3c2a21c8 774 // <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
86aa42c3 775 //
9827400b 776
777 AliCDBManager* man = AliCDBManager::Instance();
778 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
779 if (!sto)
780 return kFALSE;
781 TString localBaseFolder = sto->GetBaseFolder();
782
2d9019b4 783 TString dir = GetRefFilePrefix(localBaseFolder.Data(), fCurrentDetector.Data());
784
9827400b 785 AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
786 if (!gridSto)
787 return kFALSE;
2d9019b4 788
9827400b 789 TString gridBaseFolder = gridSto->GetBaseFolder();
2d9019b4 790
791 TString alienDir = GetRefFilePrefix(gridBaseFolder.Data(), fCurrentDetector.Data());
9827400b 792
9827400b 793 TString begin;
794 begin.Form("%d_", GetCurrentRun());
795
796 TSystemDirectory* baseDir = new TSystemDirectory("/", dir);
3d8bc902 797 if (!baseDir)
798 return kTRUE;
799
2d9019b4 800 TList* dirList = baseDir->GetListOfFiles();
801 delete baseDir;
802
803 if (!dirList) return kTRUE;
804
805 if (dirList->GetEntries() < 3)
3d8bc902 806 {
2d9019b4 807 delete dirList;
9827400b 808 return kTRUE;
3d8bc902 809 }
2d9019b4 810
546242fb 811 if (!gGrid)
812 {
813 Log("SHUTTLE", "Connection to Grid failed: Cannot continue!");
2d9019b4 814 delete dirList;
546242fb 815 return kFALSE;
816 }
817
2d9019b4 818 Int_t nDirs = 0, nTransfer = 0;
819 TIter dirIter(dirList);
820 TSystemFile* entry = 0;
821
9827400b 822 Bool_t success = kTRUE;
3d8bc902 823 Bool_t first = kTRUE;
9827400b 824
2d9019b4 825 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
826 {
9827400b 827 if (entry->IsDirectory())
828 continue;
829
830 TString fileName(entry->GetName());
831 if (!fileName.BeginsWith(begin))
832 continue;
833
2d9019b4 834 nDirs++;
835
3d8bc902 836 if (first)
837 {
838 first = kFALSE;
839 // check that DET folder exists, otherwise create it
840 TGridResult* result = gGrid->Ls(alienDir.Data(), "a");
841
842 if (!result)
2d9019b4 843 {
844 delete dirList;
3d8bc902 845 return kFALSE;
2d9019b4 846 }
3d8bc902 847
546242fb 848 if (!result->GetFileName(1)) // TODO: It looks like element 0 is always 0!!
3d8bc902 849 {
850 if (!gGrid->Mkdir(alienDir.Data(),"",0))
851 {
852 Log("SHUTTLE", Form("StoreRefFilesToGrid - Cannot create directory %s",
853 alienDir.Data()));
2d9019b4 854 delete dirList;
3d8bc902 855 return kFALSE;
546242fb 856 } else {
857 Log("SHUTTLE",Form("Folder %s created", alienDir.Data()));
3d8bc902 858 }
859
546242fb 860 } else {
861 Log("SHUTTLE",Form("Folder %s found", alienDir.Data()));
3d8bc902 862 }
863 }
864
9827400b 865 TString fullLocalPath;
866 fullLocalPath.Form("%s/%s", dir.Data(), fileName.Data());
867
868 TString fullGridPath;
869 fullGridPath.Form("alien://%s/%s", alienDir.Data(), fileName.Data());
870
9827400b 871 TFileMerger fileMerger;
a986b218 872 Bool_t result = TFile::Cp(fullLocalPath, fullGridPath);
9827400b 873
874 if (result)
875 {
2d9019b4 876 Log("SHUTTLE", Form("StoreRefFilesToGrid - Copying local file %s to %s succeeded!", fullLocalPath.Data(), fullGridPath.Data()));
9827400b 877 RemoveFile(fullLocalPath);
2d9019b4 878 nTransfer++;
9827400b 879 }
880 else
881 {
2d9019b4 882 Log("SHUTTLE", Form("StoreRefFilesToGrid - Copying local file %s to %s FAILED!", fullLocalPath.Data(), fullGridPath.Data()));
9827400b 883 success = kFALSE;
884 }
885 }
2d9019b4 886
887 Log("SHUTTLE", Form("StoreRefFilesToGrid - %d (over %d) reference files in folder %s copied to Grid.", nTransfer, nDirs, dir.Data()));
888
889
890 delete dirList;
9827400b 891 return success;
892}
893
2d9019b4 894//______________________________________________________________________________________________
895const char* AliShuttle::GetRefFilePrefix(const char* base, const char* detector)
896{
897 //
898 // Get folder name of reference files
899 //
900
901 TString offDetStr(GetOfflineDetName(detector));
902 TString dir;
903 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
904 {
905 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
906 } else {
907 dir.Form("%s/%s", base, offDetStr.Data());
908 }
909
910 return dir.Data();
911
912
913}
3301427a 914//______________________________________________________________________________________________
915void AliShuttle::CleanLocalStorage(const TString& uri)
916{
9827400b 917 //
918 // Called in case the preprocessor is declared failed. Remove remaining objects from the local storages.
919 //
3301427a 920
921 const char* type = 0;
922 if(uri == fgkLocalCDB) {
923 type = "OCDB";
924 } else if(uri == fgkLocalRefStorage) {
546242fb 925 type = "Reference";
3301427a 926 } else {
927 AliError(Form("Invalid storage URI: %s", uri.Data()));
928 return;
929 }
930
931 AliCDBManager* man = AliCDBManager::Instance();
b948db8d 932
3301427a 933 // open local storage
934 AliCDBStorage *localSto = man->GetStorage(uri);
935 if(!localSto) {
936 Log("SHUTTLE",
937 Form("CleanLocalStorage - cannot activate local %s storage", type));
938 return;
939 }
940
941 TString filename(Form("%s/%s/*/Run*_v%d_s*.root",
546242fb 942 localSto->GetBaseFolder().Data(), GetOfflineDetName(fCurrentDetector.Data()), GetCurrentRun()));
3301427a 943
944 AliInfo(Form("filename = %s", filename.Data()));
945
946 AliInfo(Form("Removing remaining local files from run %d and detector %s ...",
947 GetCurrentRun(), fCurrentDetector.Data()));
948
949 RemoveFile(filename.Data());
950
951}
952
953//______________________________________________________________________________________________
954void AliShuttle::RemoveFile(const char* filename)
955{
9827400b 956 //
957 // removes local file
958 //
3301427a 959
960 TString command(Form("rm -f %s", filename));
961
962 Int_t result = gSystem->Exec(command.Data());
963 if(result != 0)
964 {
965 Log("SHUTTLE", Form("RemoveFile - %s: Cannot remove file %s!",
966 fCurrentDetector.Data(), filename));
967 }
73abe331 968}
969
b948db8d 970//______________________________________________________________________________________________
5164a766 971AliShuttleStatus* AliShuttle::ReadShuttleStatus()
972{
9827400b 973 //
974 // Reads the AliShuttleStatus from the CDB
975 //
5164a766 976
2bb7b766 977 if (fStatusEntry){
978 delete fStatusEntry;
979 fStatusEntry = 0;
980 }
5164a766 981
10a5a932 982 fStatusEntry = AliCDBManager::Instance()->GetStorage(GetLocalCDB())
2bb7b766 983 ->Get(Form("/SHUTTLE/STATUS/%s", fCurrentDetector.Data()), GetCurrentRun());
5164a766 984
2bb7b766 985 if (!fStatusEntry) return 0;
986 fStatusEntry->SetOwner(1);
5164a766 987
2bb7b766 988 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
989 if (!status) {
990 AliError("Invalid object stored to CDB!");
991 return 0;
992 }
5164a766 993
2bb7b766 994 return status;
5164a766 995}
996
997//______________________________________________________________________________________________
7bfb2090 998Bool_t AliShuttle::WriteShuttleStatus(AliShuttleStatus* status)
5164a766 999{
9827400b 1000 //
1001 // writes the status for one subdetector
1002 //
2bb7b766 1003
1004 if (fStatusEntry){
1005 delete fStatusEntry;
1006 fStatusEntry = 0;
1007 }
5164a766 1008
2bb7b766 1009 Int_t run = GetCurrentRun();
5164a766 1010
2bb7b766 1011 AliCDBId id(AliCDBPath("SHUTTLE", "STATUS", fCurrentDetector), run, run);
5164a766 1012
2bb7b766 1013 fStatusEntry = new AliCDBEntry(status, id, new AliCDBMetaData);
1014 fStatusEntry->SetOwner(1);
5164a766 1015
2bb7b766 1016 UInt_t result = AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
7bfb2090 1017
2bb7b766 1018 if (!result) {
3301427a 1019 Log("SHUTTLE", Form("WriteShuttleStatus - Failed for %s, run %d",
1020 fCurrentDetector.Data(), run));
2bb7b766 1021 return kFALSE;
1022 }
e7f62f16 1023
1024 SendMLInfo();
7bfb2090 1025
2bb7b766 1026 return kTRUE;
5164a766 1027}
1028
1029//______________________________________________________________________________________________
1030void AliShuttle::UpdateShuttleStatus(AliShuttleStatus::Status newStatus, Bool_t increaseCount)
1031{
9827400b 1032 //
1033 // changes the AliShuttleStatus for the given detector and run to the given status
1034 //
5164a766 1035
2bb7b766 1036 if (!fStatusEntry){
1037 AliError("UNEXPECTED: fStatusEntry empty");
1038 return;
1039 }
5164a766 1040
2bb7b766 1041 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
5164a766 1042
2bb7b766 1043 if (!status){
3301427a 1044 Log("SHUTTLE", "UNEXPECTED: status could not be read from current CDB entry");
2bb7b766 1045 return;
1046 }
5164a766 1047
2c15234c 1048 TString actionStr = Form("UpdateShuttleStatus - %s: Changing state from %s to %s",
eba76848 1049 fCurrentDetector.Data(),
36c99a6a 1050 status->GetStatusName(),
eba76848 1051 status->GetStatusName(newStatus));
cb343cfd 1052 Log("SHUTTLE", actionStr);
1053 SetLastAction(actionStr);
5164a766 1054
2bb7b766 1055 status->SetStatus(newStatus);
1056 if (increaseCount) status->IncreaseCount();
5164a766 1057
2bb7b766 1058 AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
e7f62f16 1059
1060 SendMLInfo();
5164a766 1061}
e7f62f16 1062
1063//______________________________________________________________________________________________
1064void AliShuttle::SendMLInfo()
1065{
1066 //
1067 // sends ML information about the current status of the current detector being processed
1068 //
1069
1070 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1071
1072 if (!status){
3301427a 1073 Log("SHUTTLE", "SendMLInfo - UNEXPECTED: status could not be read from current CDB entry");
e7f62f16 1074 return;
1075 }
1076
1077 TMonaLisaText mlStatus(Form("%s_status", fCurrentDetector.Data()), status->GetStatusName());
1078 TMonaLisaValue mlRetryCount(Form("%s_count", fCurrentDetector.Data()), status->GetCount());
1079
1080 TList mlList;
1081 mlList.Add(&mlStatus);
1082 mlList.Add(&mlRetryCount);
1083
1084 fMonaLisa->SendParameters(&mlList);
1085}
1086
5164a766 1087//______________________________________________________________________________________________
1088Bool_t AliShuttle::ContinueProcessing()
1089{
9827400b 1090 // this function reads the AliShuttleStatus information from CDB and
1091 // checks if the processing should be continued
1092 // if yes it returns kTRUE and updates the AliShuttleStatus with nextStatus
2bb7b766 1093
57c1a579 1094 if (!fConfig->HostProcessDetector(fCurrentDetector)) return kFALSE;
1095
1096 AliPreprocessor* aPreprocessor =
1097 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1098 if (!aPreprocessor)
1099 {
1100 AliInfo(Form("%s: no preprocessor registered", fCurrentDetector.Data()));
1101 return kFALSE;
1102 }
1103
2bb7b766 1104 AliShuttleLogbookEntry::Status entryStatus =
eba76848 1105 fLogbookEntry->GetDetectorStatus(fCurrentDetector);
2bb7b766 1106
1107 if(entryStatus != AliShuttleLogbookEntry::kUnprocessed) {
9e080f92 1108 AliInfo(Form("ContinueProcessing - %s is %s",
2bb7b766 1109 fCurrentDetector.Data(),
1110 fLogbookEntry->GetDetectorStatusName(entryStatus)));
1111 return kFALSE;
1112 }
1113
1114 // if we get here, according to Shuttle logbook subdetector is in UNPROCESSED state
be48e3ea 1115
1116 // check if current run is first unprocessed run for current detector
1117 if (fConfig->StrictRunOrder(fCurrentDetector) &&
1118 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
1119 {
86aa42c3 1120 if (fTestMode == kNone)
1121 {
1122 Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering but this is not the first unprocessed run!"));
1123 return kFALSE;
1124 }
1125 else
1126 {
1127 Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - Although %s requires strict run ordering and this is not the first unprocessed run, the SHUTTLE continues"));
1128 }
be48e3ea 1129 }
1130
2bb7b766 1131 AliShuttleStatus* status = ReadShuttleStatus();
1132 if (!status) {
1133 // first time
1134 Log("SHUTTLE", Form("ContinueProcessing - %s: Processing first time",
1135 fCurrentDetector.Data()));
1136 status = new AliShuttleStatus(AliShuttleStatus::kStarted);
1137 return WriteShuttleStatus(status);
1138 }
1139
1140 // The following two cases shouldn't happen if Shuttle Logbook was correctly updated.
1141 // If it happens it may mean Logbook updating failed... let's do it now!
1142 if (status->GetStatus() == AliShuttleStatus::kDone ||
1143 status->GetStatus() == AliShuttleStatus::kFailed){
1144 Log("SHUTTLE", Form("ContinueProcessing - %s is already %s. Updating Shuttle Logbook",
1145 fCurrentDetector.Data(),
1146 status->GetStatusName(status->GetStatus())));
1147 UpdateShuttleLogbook(fCurrentDetector.Data(),
1148 status->GetStatusName(status->GetStatus()));
1149 return kFALSE;
1150 }
1151
3301427a 1152 if (status->GetStatus() == AliShuttleStatus::kStoreError) {
2bb7b766 1153 Log("SHUTTLE",
1154 Form("ContinueProcessing - %s: Grid storage of one or more objects failed. Trying again now",
1155 fCurrentDetector.Data()));
9827400b 1156 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1157 if (StoreOCDB()){
3301427a 1158 Log("SHUTTLE", Form("ContinueProcessing - %s: all objects successfully stored into main storage",
1159 fCurrentDetector.Data()));
2bb7b766 1160 UpdateShuttleStatus(AliShuttleStatus::kDone);
1161 UpdateShuttleLogbook(fCurrentDetector.Data(), "DONE");
1162 } else {
1163 Log("SHUTTLE",
1164 Form("ContinueProcessing - %s: Grid storage failed again",
1165 fCurrentDetector.Data()));
9827400b 1166 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
2bb7b766 1167 }
1168 return kFALSE;
1169 }
1170
1171 // if we get here, there is a restart
57c1a579 1172 Bool_t cont = kFALSE;
2bb7b766 1173
1174 // abort conditions
cb343cfd 1175 if (status->GetCount() >= fConfig->GetMaxRetries()) {
57c1a579 1176 Log("SHUTTLE", Form("ContinueProcessing - %s failed %d times in status %s - "
1177 "Updating Shuttle Logbook", fCurrentDetector.Data(),
2bb7b766 1178 status->GetCount(), status->GetStatusName()));
1179 UpdateShuttleLogbook(fCurrentDetector.Data(), "FAILED");
e7f62f16 1180 UpdateShuttleStatus(AliShuttleStatus::kFailed);
3301427a 1181
1182 // there may still be objects in local OCDB and reference storage
1183 // and FXS databases may be not updated: do it now!
9827400b 1184
1185 // TODO Currently disabled, we want to keep files in case of failure!
1186 // CleanLocalStorage(fgkLocalCDB);
1187 // CleanLocalStorage(fgkLocalRefStorage);
1188 // UpdateTableFailCase();
1189
1190 // Send mail to detector expert!
1191 AliInfo(Form("Sending mail to %s expert...", fCurrentDetector.Data()));
1192 if (!SendMail())
1193 Log("SHUTTLE", Form("ContinueProcessing - Could not send mail to %s expert",
1194 fCurrentDetector.Data()));
3301427a 1195
57c1a579 1196 } else {
1197 Log("SHUTTLE", Form("ContinueProcessing - %s: restarting. "
1198 "Aborted before with %s. Retry number %d.", fCurrentDetector.Data(),
1199 status->GetStatusName(), status->GetCount()));
9827400b 1200 Bool_t increaseCount = kTRUE;
1201 if (status->GetStatus() == AliShuttleStatus::kDCSError || status->GetStatus() == AliShuttleStatus::kDCSStarted)
1202 increaseCount = kFALSE;
1203 UpdateShuttleStatus(AliShuttleStatus::kStarted, increaseCount);
57c1a579 1204 cont = kTRUE;
2bb7b766 1205 }
1206
57c1a579 1207 return cont;
5164a766 1208}
1209
1210//______________________________________________________________________________________________
2bb7b766 1211Bool_t AliShuttle::Process(AliShuttleLogbookEntry* entry)
58bc3020 1212{
73abe331 1213 //
b948db8d 1214 // Makes data retrieval for all detectors in the configuration.
2bb7b766 1215 // entry: Shuttle logbook entry, contains run paramenters and status of detectors
1216 // (Unprocessed, Inactive, Failed or Done).
d477ad88 1217 // Returns kFALSE in case of error occured and kTRUE otherwise
73abe331 1218 //
1219
9827400b 1220 if (!entry) return kFALSE;
2bb7b766 1221
1222 fLogbookEntry = entry;
1223
9827400b 1224 AliInfo(Form("\n\n \t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^* \n",
1225 GetCurrentRun()));
2bb7b766 1226
e7f62f16 1227 // create ML instance that monitors this run
1228 fMonaLisa = new TMonaLisaWriter(Form("%d", GetCurrentRun()), "SHUTTLE", "aliendb1.cern.ch");
1229 // disable monitoring of other parameters that come e.g. from TFile
1230 gMonitoringWriter = 0;
2bb7b766 1231
e7f62f16 1232 // Send the information to ML
1233 TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
9827400b 1234 TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
e7f62f16 1235
1236 TList mlList;
1237 mlList.Add(&mlStatus);
9827400b 1238 mlList.Add(&mlRunType);
e7f62f16 1239
1240 fMonaLisa->SendParameters(&mlList);
3301427a 1241
9827400b 1242 if (fLogbookEntry->IsDone())
1243 {
1244 Log("SHUTTLE","Process - Shuttle is already DONE. Updating logbook");
1245 UpdateShuttleLogbook("shuttle_done");
1246 fLogbookEntry = 0;
1247 return kTRUE;
1248 }
1249
1250 // read test mode if flag is set
1251 if (fReadTestMode)
1252 {
3d8bc902 1253 fTestMode = kNone;
9827400b 1254 TString logEntry(entry->GetRunParameter("log"));
1255 //printf("log entry = %s\n", logEntry.Data());
1256 TString searchStr("Testmode: ");
1257 Int_t pos = logEntry.Index(searchStr.Data());
1258 //printf("%d\n", pos);
1259 if (pos >= 0)
1260 {
1261 TSubString subStr = logEntry(pos + searchStr.Length(), logEntry.Length());
1262 //printf("%s\n", subStr.String().Data());
1263 TString newStr(subStr.Data());
1264 TObjArray* token = newStr.Tokenize(' ');
1265 if (token)
1266 {
1267 //token->Print();
1268 TObjString* tmpStr = dynamic_cast<TObjString*> (token->First());
1269 if (tmpStr)
1270 {
1271 Int_t testMode = tmpStr->String().Atoi();
1272 if (testMode > 0)
1273 {
1274 Log("SHUTTLE", Form("Enabling test mode %d", testMode));
1275 SetTestMode((TestMode) testMode);
1276 }
1277 }
1278 delete token;
1279 }
1280 }
1281 }
1282
3d8bc902 1283 Log("SHUTTLE", Form("The test mode flag is %d", (Int_t) fTestMode));
1284
eba76848 1285 fLogbookEntry->Print("all");
57f50b3c 1286
1287 // Initialization
d477ad88 1288 Bool_t hasError = kFALSE;
5164a766 1289
2bb7b766 1290 AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
1291 if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
1292 AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
1293 if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
d477ad88 1294
57f50b3c 1295 // Loop on detectors in the configuration
b948db8d 1296 TIter iter(fConfig->GetDetectors());
2bb7b766 1297 TObjString* aDetector = 0;
b948db8d 1298
be48e3ea 1299 while ((aDetector = (TObjString*) iter.Next()))
1300 {
7bfb2090 1301 fCurrentDetector = aDetector->String();
5164a766 1302
9e080f92 1303 if (ContinueProcessing() == kFALSE) continue;
1304
2bb7b766 1305 AliInfo(Form("\n\n \t\t\t****** run %d - %s: START ******",
1306 GetCurrentRun(), aDetector->GetName()));
1307
9d733021 1308 for(Int_t iSys=0;iSys<3;iSys++) fFXSCalled[iSys]=kFALSE;
1309
e7f62f16 1310 Log(fCurrentDetector.Data(), "Starting processing");
85a80aa9 1311
be48e3ea 1312 Int_t pid = fork();
1313
1314 if (pid < 0)
1315 {
1316 Log("SHUTTLE", "ERROR: Forking failed");
1317 }
1318 else if (pid > 0)
1319 {
1320 // parent
1321 AliInfo(Form("In parent process of %d - %s: Starting monitoring",
1322 GetCurrentRun(), aDetector->GetName()));
1323
1324 Long_t begin = time(0);
1325
1326 int status; // to be used with waitpid, on purpose an int (not Int_t)!
1327 while (waitpid(pid, &status, WNOHANG) == 0)
1328 {
1329 Long_t expiredTime = time(0) - begin;
1330
1331 if (expiredTime > fConfig->GetPPTimeOut())
1332 {
9827400b 1333 TString tmp;
1334 tmp.Form("Process of %s time out. Run time: %d seconds. Killing...",
1335 fCurrentDetector.Data(), expiredTime);
1336 Log("SHUTTLE", tmp);
1337 Log(fCurrentDetector, tmp);
be48e3ea 1338
1339 kill(pid, 9);
1340
3301427a 1341 UpdateShuttleStatus(AliShuttleStatus::kPPTimeOut);
be48e3ea 1342 hasError = kTRUE;
1343
1344 gSystem->Sleep(1000);
1345 }
1346 else
1347 {
be48e3ea 1348 gSystem->Sleep(1000);
9827400b 1349
1350 TString checkStr;
1351 checkStr.Form("ps -o vsize --pid %d | tail -n 1", pid);
1352 FILE* pipe = gSystem->OpenPipe(checkStr, "r");
1353 if (!pipe)
1354 {
1355 Log("SHUTTLE", Form("Error: Could not open pipe to %s", checkStr.Data()));
1356 continue;
1357 }
1358
1359 char buffer[100];
1360 if (!fgets(buffer, 100, pipe))
1361 {
1362 Log("SHUTTLE", "Error: ps did not return anything");
1363 gSystem->ClosePipe(pipe);
1364 continue;
1365 }
1366 gSystem->ClosePipe(pipe);
1367
1368 //Log("SHUTTLE", Form("ps returned %s", buffer));
1369
1370 Int_t mem = 0;
1371 if ((sscanf(buffer, "%d\n", &mem) != 1) || !mem)
1372 {
1373 Log("SHUTTLE", "Error: Could not parse output of ps");
1374 continue;
1375 }
1376
1377 if (expiredTime % 60 == 0)
886d60e6 1378 Log("SHUTTLE", Form("%s: Checking process. Run time: %d seconds - Memory consumption: %d KB",
1379 fCurrentDetector.Data(), expiredTime, mem));
9827400b 1380
1381 if (mem > fConfig->GetPPMaxMem())
1382 {
1383 TString tmp;
1384 tmp.Form("Process exceeds maximum allowed memory (%d KB > %d KB). Killing...",
1385 mem, fConfig->GetPPMaxMem());
1386 Log("SHUTTLE", tmp);
1387 Log(fCurrentDetector, tmp);
1388
1389 kill(pid, 9);
1390
1391 UpdateShuttleStatus(AliShuttleStatus::kPPOutOfMemory);
1392 hasError = kTRUE;
1393
1394 gSystem->Sleep(1000);
1395 }
be48e3ea 1396 }
1397 }
1398
1399 AliInfo(Form("In parent process of %d - %s: Client has terminated.",
1400 GetCurrentRun(), aDetector->GetName()));
1401
1402 if (WIFEXITED(status))
1403 {
1404 Int_t returnCode = WEXITSTATUS(status);
1405
3301427a 1406 Log("SHUTTLE", Form("%s: the return code is %d", fCurrentDetector.Data(),
1407 returnCode));
be48e3ea 1408
9827400b 1409 if (returnCode == 0) hasError = kTRUE;
be48e3ea 1410 }
1411 }
1412 else if (pid == 0)
1413 {
1414 // client
1415 AliInfo(Form("In client process of %d - %s", GetCurrentRun(), aDetector->GetName()));
1416
ffa29e93 1417 AliInfo("Redirecting output...");
1418
546242fb 1419 if ((freopen(GetLogFileName(fCurrentDetector), "a", stdout)) == 0)
ffa29e93 1420 {
1421 Log("SHUTTLE", "Could not freopen stdout");
1422 }
1423 else
1424 {
1425 fOutputRedirected = kTRUE;
1426 if ((dup2(fileno(stdout), fileno(stderr))) < 0)
1427 Log("SHUTTLE", "Could not redirect stderr");
1428
1429 }
1430
5bac2bde 1431 TString wd = gSystem->WorkingDirectory();
1432 TString tmpDir = Form("%s/%s_process",GetShuttleTempDir(),fCurrentDetector.Data());
1433
1434 gSystem->mkdir(tmpDir.Data());
1435 gSystem->ChangeDirectory(tmpDir.Data());
1436
9827400b 1437 Bool_t success = ProcessCurrentDetector();
5bac2bde 1438
1439 gSystem->ChangeDirectory(wd.Data());
1440
1441 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
1442
9827400b 1443 if (success) // Preprocessor finished successfully!
1444 {
3301427a 1445 // Update time_processed field in FXS DB
1446 if (UpdateTable() == kFALSE)
5bac2bde 1447 Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!",
1448 fCurrentDetector.Data()));
3301427a 1449
1450 // Transfer the data from local storage to main storage (Grid)
1451 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1452 if (StoreOCDB() == kFALSE)
1453 {
1454 AliInfo(Form("\n \t\t\t****** run %d - %s: STORAGE ERROR ****** \n\n",
1455 GetCurrentRun(), aDetector->GetName()));
1456 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
9827400b 1457 success = kFALSE;
3301427a 1458 } else {
1459 AliInfo(Form("\n \t\t\t****** run %d - %s: DONE ****** \n\n",
1460 GetCurrentRun(), aDetector->GetName()));
1461 UpdateShuttleStatus(AliShuttleStatus::kDone);
9827400b 1462 UpdateShuttleLogbook(fCurrentDetector, "DONE");
3301427a 1463 }
be48e3ea 1464 }
1465
4b95672b 1466 for (UInt_t iSys=0; iSys<3; iSys++)
1467 {
1468 if (fFXSCalled[iSys]) fFXSlist[iSys].Clear();
1469 }
1470
be48e3ea 1471 AliInfo(Form("Client process of %d - %s is exiting now with %d.",
9827400b 1472 GetCurrentRun(), aDetector->GetName(), success));
be48e3ea 1473
1474 // the client exits here
9827400b 1475 gSystem->Exit(success);
be48e3ea 1476
1477 AliError("We should never get here!!!");
1478 }
7bfb2090 1479 }
5164a766 1480
2bb7b766 1481 AliInfo(Form("\n\n \t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^* \n",
1482 GetCurrentRun()));
1483
1484 //check if shuttle is done for this run, if so update logbook
1485 TObjArray checkEntryArray;
1486 checkEntryArray.SetOwner(1);
9e080f92 1487 TString whereClause = Form("where run=%d", GetCurrentRun());
1488 if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) || checkEntryArray.GetEntries() == 0) {
1489 Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
1490 GetCurrentRun()));
1491 return hasError == kFALSE;
1492 }
b948db8d 1493
9e080f92 1494 AliShuttleLogbookEntry* checkEntry = dynamic_cast<AliShuttleLogbookEntry*>
1495 (checkEntryArray.At(0));
2bb7b766 1496
9e080f92 1497 if (checkEntry)
1498 {
1499 if (checkEntry->IsDone())
be48e3ea 1500 {
9e080f92 1501 Log("SHUTTLE","Process - Shuttle is DONE. Updating logbook");
1502 UpdateShuttleLogbook("shuttle_done");
1503 }
1504 else
1505 {
1506 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
be48e3ea 1507 {
9e080f92 1508 if (checkEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
be48e3ea 1509 {
9e080f92 1510 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
1511 checkEntry->GetRun(), GetDetName(iDet)));
1512 fFirstUnprocessed[iDet] = kFALSE;
be48e3ea 1513 }
1514 }
2bb7b766 1515 }
1516 }
1517
e7f62f16 1518 // remove ML instance
1519 delete fMonaLisa;
1520 fMonaLisa = 0;
1521
2bb7b766 1522 fLogbookEntry = 0;
85a80aa9 1523
a7160fe9 1524 return hasError == kFALSE;
73abe331 1525}
1526
b948db8d 1527//______________________________________________________________________________________________
9827400b 1528Bool_t AliShuttle::ProcessCurrentDetector()
73abe331 1529{
1530 //
2bb7b766 1531 // Makes data retrieval just for a specific detector (fCurrentDetector).
73abe331 1532 // Threre should be a configuration for this detector.
73abe331 1533
2bb7b766 1534 AliInfo(Form("Retrieving values for %s, run %d", fCurrentDetector.Data(), GetCurrentRun()));
73abe331 1535
2d9019b4 1536 if (!CleanReferenceStorage(fCurrentDetector.Data()))
546242fb 1537 return kFALSE;
1538
a038aa70 1539 TMap* dcsMap = 0;
3301427a 1540
1541 // call preprocessor
1542 AliPreprocessor* aPreprocessor =
1543 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1544
1545 aPreprocessor->Initialize(GetCurrentRun(), GetCurrentStartTime(), GetCurrentEndTime());
1546
1547 Bool_t processDCS = aPreprocessor->ProcessDCS();
d477ad88 1548
651fdaab 1549 if (!processDCS)
1550 {
1551 Log(fCurrentDetector, "The preprocessor requested to skip the retrieval of DCS values");
1552 }
8b739301 1553 else if (fTestMode & kSkipDCS)
2c15234c 1554 {
3d8bc902 1555 Log(fCurrentDetector, "In TESTMODE - Skipping DCS processing!");
9827400b 1556 }
1557 else if (fTestMode & kErrorDCS)
1558 {
3d8bc902 1559 Log(fCurrentDetector, "In TESTMODE - Simulating DCS error");
1560 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
9827400b 1561 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1562 return kFALSE;
2c15234c 1563 } else {
3301427a 1564
1565 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
1566
2c15234c 1567 TString host(fConfig->GetDCSHost(fCurrentDetector));
1568 Int_t port = fConfig->GetDCSPort(fCurrentDetector);
1569
a038aa70 1570 if (fConfig->GetDCSAliases(fCurrentDetector)->GetEntries() > 0)
2c15234c 1571 {
a038aa70 1572 dcsMap = GetValueSet(host, port, fConfig->GetDCSAliases(fCurrentDetector), kAlias);
1573 if (!dcsMap)
2c15234c 1574 {
a038aa70 1575 Log(fCurrentDetector, "ProcessCurrentDetector - Error while retrieving DCS aliases");
2c15234c 1576 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
9827400b 1577 return kFALSE;
2c15234c 1578 }
4f0ab988 1579 }
a038aa70 1580
1581 if (fConfig->GetDCSDataPoints(fCurrentDetector)->GetEntries() > 0)
2c15234c 1582 {
a038aa70 1583 TMap* dcsMap2 = GetValueSet(host, port, fConfig->GetDCSDataPoints(fCurrentDetector), kDP);
1584 if (!dcsMap2)
2c15234c 1585 {
a038aa70 1586 Log(fCurrentDetector, "ProcessCurrentDetector - Error while retrieving DCS data points");
2c15234c 1587 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
a038aa70 1588 if (dcsMap)
1589 delete dcsMap;
9827400b 1590 return kFALSE;
2c15234c 1591 }
a038aa70 1592
1593 if (!dcsMap)
1594 {
1595 dcsMap = dcsMap2;
1596 }
1597 else // merge
1598 {
1599 TIter iter(dcsMap2);
1600 TObjString* key = 0;
1601 while ((key = (TObjString*) iter.Next()))
1602 dcsMap->Add(key, dcsMap2->GetValue(key->String()));
1603
1604 dcsMap2->SetOwner(kFALSE);
1605 delete dcsMap2;
1606 }
73abe331 1607 }
a038aa70 1608
73abe331 1609 }
b948db8d 1610
dc25836b 1611 // still no map?
1612 if (!dcsMap)
1613 dcsMap = new TMap;
1614
2bb7b766 1615 // DCS Archive DB processing successful. Call Preprocessor!
85a80aa9 1616 UpdateShuttleStatus(AliShuttleStatus::kPPStarted);
a7160fe9 1617
a038aa70 1618 UInt_t returnValue = aPreprocessor->Process(dcsMap);
b948db8d 1619
3301427a 1620 if (returnValue > 0) // Preprocessor error!
1621 {
9827400b 1622 Log(fCurrentDetector, Form("Preprocessor failed. Process returned %d.", returnValue));
cb343cfd 1623 UpdateShuttleStatus(AliShuttleStatus::kPPError);
a038aa70 1624 dcsMap->DeleteAll();
1625 delete dcsMap;
9827400b 1626 return kFALSE;
1627 }
1628
1629 // preprocessor ok!
1630 UpdateShuttleStatus(AliShuttleStatus::kPPDone);
1631 Log(fCurrentDetector, Form("ProcessCurrentDetector - %s preprocessor returned success",
1632 fCurrentDetector.Data()));
b948db8d 1633
a038aa70 1634 dcsMap->DeleteAll();
1635 delete dcsMap;
b948db8d 1636
9827400b 1637 return kTRUE;
2bb7b766 1638}
1639
1640//______________________________________________________________________________________________
1641Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
1642 TObjArray& entries)
1643{
9827400b 1644 // Query DAQ's Shuttle logbook and fills detector status object.
1645 // Call QueryRunParameters to query DAQ logbook for run parameters.
1646 //
2bb7b766 1647
fc5a4708 1648 entries.SetOwner(1);
1649
2bb7b766 1650 // check connection, in case connect
be48e3ea 1651 if(!Connect(3)) return kFALSE;
2bb7b766 1652
1653 TString sqlQuery;
441b0e9c 1654 sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
2bb7b766 1655
be48e3ea 1656 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1657 if (!aResult) {
1658 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
1659 return kFALSE;
1660 }
1661
fc5a4708 1662 AliDebug(2,Form("Query = %s", sqlQuery.Data()));
1663
2bb7b766 1664 if(aResult->GetRowCount() == 0) {
9827400b 1665 AliInfo("No entries in Shuttle Logbook match request");
1666 delete aResult;
1667 return kTRUE;
2bb7b766 1668 }
1669
1670 // TODO Check field count!
db99d43e 1671 const UInt_t nCols = 23;
2bb7b766 1672 if (aResult->GetFieldCount() != (Int_t) nCols) {
1673 AliError("Invalid SQL result field number!");
1674 delete aResult;
1675 return kFALSE;
1676 }
1677
2bb7b766 1678 TSQLRow* aRow;
1679 while ((aRow = aResult->Next())) {
1680 TString runString(aRow->GetField(0), aRow->GetFieldLength(0));
1681 Int_t run = runString.Atoi();
1682
eba76848 1683 AliShuttleLogbookEntry *entry = QueryRunParameters(run);
1684 if (!entry)
1685 continue;
2bb7b766 1686
1687 // loop on detectors
eba76848 1688 for(UInt_t ii = 0; ii < nCols; ii++)
1689 entry->SetDetectorStatus(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1690
eba76848 1691 entries.AddLast(entry);
2bb7b766 1692 delete aRow;
1693 }
1694
2bb7b766 1695 delete aResult;
1696 return kTRUE;
1697}
1698
1699//______________________________________________________________________________________________
eba76848 1700AliShuttleLogbookEntry* AliShuttle::QueryRunParameters(Int_t run)
2bb7b766 1701{
eba76848 1702 //
1703 // Retrieve run parameters written in the DAQ logbook and sets them into AliShuttleLogbookEntry object
1704 //
2bb7b766 1705
1706 // check connection, in case connect
be48e3ea 1707 if (!Connect(3))
eba76848 1708 return 0;
2bb7b766 1709
1710 TString sqlQuery;
2c15234c 1711 sqlQuery.Form("select * from %s where run=%d", fConfig->GetDAQlbTable(), run);
2bb7b766 1712
be48e3ea 1713 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1714 if (!aResult) {
1715 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
eba76848 1716 return 0;
2bb7b766 1717 }
1718
eba76848 1719 if (aResult->GetRowCount() == 0) {
2bb7b766 1720 Log("SHUTTLE", Form("QueryRunParameters - No entry in DAQ Logbook for run %d. Skipping", run));
1721 delete aResult;
eba76848 1722 return 0;
2bb7b766 1723 }
1724
eba76848 1725 if (aResult->GetRowCount() > 1) {
2bb7b766 1726 AliError(Form("More than one entry in DAQ Logbook for run %d. Skipping", run));
1727 delete aResult;
eba76848 1728 return 0;
2bb7b766 1729 }
1730
eba76848 1731 TSQLRow* aRow = aResult->Next();
1732 if (!aRow)
1733 {
1734 AliError(Form("Could not retrieve row for run %d. Skipping", run));
1735 delete aResult;
1736 return 0;
1737 }
2bb7b766 1738
eba76848 1739 AliShuttleLogbookEntry* entry = new AliShuttleLogbookEntry(run);
2bb7b766 1740
eba76848 1741 for (Int_t ii = 0; ii < aResult->GetFieldCount(); ii++)
1742 entry->SetRunParameter(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1743
eba76848 1744 UInt_t startTime = entry->GetStartTime();
1745 UInt_t endTime = entry->GetEndTime();
1746
1747 if (!startTime || !endTime || startTime > endTime) {
1748 Log("SHUTTLE",
1749 Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d",
1750 run, startTime, endTime));
1751 delete entry;
2bb7b766 1752 delete aRow;
eba76848 1753 delete aResult;
1754 return 0;
2bb7b766 1755 }
1756
eba76848 1757 delete aRow;
2bb7b766 1758 delete aResult;
eba76848 1759
1760 return entry;
2bb7b766 1761}
1762
b948db8d 1763//______________________________________________________________________________________________
2c15234c 1764Bool_t AliShuttle::GetValueSet(const char* host, Int_t port, const char* entry,
1765 TObjArray* valueSet, DCSType type)
73abe331 1766{
9827400b 1767 // Retrieve all "entry" data points from the DCS server
1768 // host, port: TSocket connection parameters
1769 // entry: name of the alias or data point
1770 // valueSet: array of retrieved AliDCSValue's
1771 // type: kAlias or kDP
58bc3020 1772
73abe331 1773 AliDCSClient client(host, port, fTimeout, fRetries);
2c15234c 1774 if (!client.IsConnected())
1775 {
b948db8d 1776 return kFALSE;
73abe331 1777 }
1778
2c15234c 1779 Int_t result=0;
73abe331 1780
2c15234c 1781 if (type == kAlias)
1782 {
1783 result = client.GetAliasValues(entry,
1784 GetCurrentStartTime(), GetCurrentEndTime(), valueSet);
1785 } else
1786 if (type == kDP)
1787 {
1788 result = client.GetDPValues(entry,
1789 GetCurrentStartTime(), GetCurrentEndTime(), valueSet);
1790 }
1791
1792 if (result < 0)
1793 {
2bb7b766 1794 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get '%s'! Reason: %s",
2c15234c 1795 entry, AliDCSClient::GetErrorString(result)));
73abe331 1796
2c15234c 1797 if (result == AliDCSClient::fgkServerError)
1798 {
2bb7b766 1799 Log(fCurrentDetector.Data(), Form("GetValueSet - Server error: %s",
73abe331 1800 client.GetServerError().Data()));
1801 }
1802
1803 return kFALSE;
1804 }
1805
1806 return kTRUE;
1807}
b948db8d 1808
a038aa70 1809//______________________________________________________________________________________________
1810TMap* AliShuttle::GetValueSet(const char* host, Int_t port, const TSeqCollection* entries,
1811 DCSType type)
1812{
1813 // Retrieve all "entry" data points from the DCS server
1814 // host, port: TSocket connection parameters
1815 // entries: list of name of the alias or data point
1816 // type: kAlias or kDP
1817 // returns TMap of values, 0 when failure
1818
b41b252a 1819 AliDCSClient client(host, port, fTimeout, fRetries);
1820
a038aa70 1821 TMap* result = 0;
b41b252a 1822 if (type == kAlias)
a038aa70 1823 {
b41b252a 1824 result = client.GetAliasValues(entries, GetCurrentStartTime(),
1825 GetCurrentEndTime());
1826 }
1827 else if (type == kDP)
1828 {
1829 result = client.GetDPValues(entries, GetCurrentStartTime(),
1830 GetCurrentEndTime());
1831 }
a038aa70 1832
b41b252a 1833 if (result == 0)
1834 {
1835 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get entries! Reason: %s",
1836 client.GetServerError().Data()));
a038aa70 1837
b41b252a 1838 return 0;
a038aa70 1839 }
b41b252a 1840
a038aa70 1841 return result;
1842}
b41b252a 1843
b948db8d 1844//______________________________________________________________________________________________
57f50b3c 1845const char* AliShuttle::GetFile(Int_t system, const char* detector,
1846 const char* id, const char* source)
b948db8d 1847{
9827400b 1848 // Get calibration file from file exchange servers
1849 // First queris the FXS database for the file name, using the run, detector, id and source info
1850 // then calls RetrieveFile(filename) for actual copy to local disk
1851 // run: current run being processed (given by Logbook entry fLogbookEntry)
1852 // detector: the Preprocessor name
1853 // id: provided as a parameter by the Preprocessor
1854 // source: provided by the Preprocessor through GetFileSources function
1855
1856 // check if test mode should simulate a FXS error
1857 if (fTestMode & kErrorFXSFiles)
1858 {
1859 Log(detector, Form("GetFile - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
1860 return 0;
1861 }
1862
57f50b3c 1863 // check connection, in case connect
9d733021 1864 if (!Connect(system))
eba76848 1865 {
9d733021 1866 Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
57f50b3c 1867 return 0;
1868 }
1869
1870 // Query preparation
9d733021 1871 TString sourceName(source);
d386d623 1872 Int_t nFields = 3;
1873 TString sqlQueryStart = Form("select filePath,size,fileChecksum from %s where",
1874 fConfig->GetFXSdbTable(system));
1875 TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
1876 GetCurrentRun(), detector, id);
1877
9d733021 1878 if (system == kDAQ)
1879 {
d386d623 1880 whereClause += Form(" and DAQsource=\"%s\"", source);
57f50b3c 1881 }
9d733021 1882 else if (system == kDCS)
eba76848 1883 {
9d733021 1884 sourceName="none";
57f50b3c 1885 }
9d733021 1886 else if (system == kHLT)
9e080f92 1887 {
d386d623 1888 whereClause += Form(" and DDLnumbers=\"%s\"", source);
9d733021 1889 nFields = 3;
9e080f92 1890 }
1891
9e080f92 1892 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
1893
1894 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
1895
1896 // Query execution
1897 TSQLResult* aResult = 0;
9d733021 1898 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
9e080f92 1899 if (!aResult) {
9d733021 1900 Log(detector, Form("GetFileName - Can't execute SQL query to %s database for: id = %s, source = %s",
1901 GetSystemName(system), id, sourceName.Data()));
9e080f92 1902 return 0;
1903 }
1904
1905 if(aResult->GetRowCount() == 0)
1906 {
1907 Log(detector,
9d733021 1908 Form("GetFileName - No entry in %s FXS db for: id = %s, source = %s",
1909 GetSystemName(system), id, sourceName.Data()));
9e080f92 1910 delete aResult;
1911 return 0;
1912 }
2bb7b766 1913
9e080f92 1914 if (aResult->GetRowCount() > 1) {
1915 Log(detector,
9d733021 1916 Form("GetFileName - More than one entry in %s FXS db for: id = %s, source = %s",
1917 GetSystemName(system), id, sourceName.Data()));
9e080f92 1918 delete aResult;
1919 return 0;
1920 }
1921
9d733021 1922 if (aResult->GetFieldCount() != nFields) {
9e080f92 1923 Log(detector,
9d733021 1924 Form("GetFileName - Wrong field count in %s FXS db for: id = %s, source = %s",
1925 GetSystemName(system), id, sourceName.Data()));
9e080f92 1926 delete aResult;
1927 return 0;
1928 }
1929
1930 TSQLRow* aRow = dynamic_cast<TSQLRow*> (aResult->Next());
1931
1932 if (!aRow){
9d733021 1933 Log(detector, Form("GetFileName - Empty set result in %s FXS db from query: id = %s, source = %s",
1934 GetSystemName(system), id, sourceName.Data()));
9e080f92 1935 delete aResult;
1936 return 0;
1937 }
1938
1939 TString filePath(aRow->GetField(0), aRow->GetFieldLength(0));
1940 TString fileSize(aRow->GetField(1), aRow->GetFieldLength(1));
d386d623 1941 TString fileChecksum(aRow->GetField(2), aRow->GetFieldLength(2));
9e080f92 1942
1943 delete aResult;
1944 delete aRow;
1945
d386d623 1946 AliDebug(2, Form("filePath = %s; size = %s, fileChecksum = %s",
1947 filePath.Data(), fileSize.Data(), fileChecksum.Data()));
9e080f92 1948
9e080f92 1949 // retrieved file is renamed to make it unique
9d733021 1950 TString localFileName = Form("%s_%s_%d_%s_%s.shuttle",
1951 GetSystemName(system), detector, GetCurrentRun(), id, sourceName.Data());
1952
9e080f92 1953
9d733021 1954 // file retrieval from FXS
4b95672b 1955 UInt_t nRetries = 0;
1956 UInt_t maxRetries = 3;
1957 Bool_t result = kFALSE;
1958
1959 // copy!! if successful TSystem::Exec returns 0
1960 while(nRetries++ < maxRetries) {
1961 AliDebug(2, Form("Trying to copy file. Retry # %d", nRetries));
1962 result = RetrieveFile(system, filePath.Data(), localFileName.Data());
1963 if(!result)
1964 {
1965 Log(detector, Form("GetFileName - Copy of file %s from %s FXS failed",
9d733021 1966 filePath.Data(), GetSystemName(system)));
4b95672b 1967 continue;
4f0749a8 1968 }
9e080f92 1969
d386d623 1970 if (fileChecksum.Length()>0)
4b95672b 1971 {
1972 // compare md5sum of local file with the one stored in the FXS DB
1973 Int_t md5Comp = gSystem->Exec(Form("md5sum %s/%s |grep %s 2>&1 > /dev/null",
d386d623 1974 GetShuttleTempDir(), localFileName.Data(), fileChecksum.Data()));
9e080f92 1975
4b95672b 1976 if (md5Comp != 0)
1977 {
1978 Log(detector, Form("GetFileName - md5sum of file %s does not match with local copy!",
1979 filePath.Data()));
1980 result = kFALSE;
1981 continue;
1982 }
d386d623 1983 } else {
1984 Log(fCurrentDetector, Form("GetFile - md5sum of file %s not set in %s database, skipping comparison",
1985 filePath.Data(), GetSystemName(system)));
9d733021 1986 }
4b95672b 1987 if (result) break;
9e080f92 1988 }
1989
4b95672b 1990 if(!result) return 0;
1991
9d733021 1992 fFXSCalled[system]=kTRUE;
1993 TObjString *fileParams = new TObjString(Form("%s#!?!#%s", id, sourceName.Data()));
1994 fFXSlist[system].Add(fileParams);
9e080f92 1995
1996 static TString fullLocalFileName;
1bcd28db 1997 fullLocalFileName.Form("%s/%s", GetShuttleTempDir(), localFileName.Data());
36c99a6a 1998
1bcd28db 1999 Log(fCurrentDetector, Form("GetFile - Retrieved file with id %s and source %s from %s to %s", id, source, GetSystemName(system), fullLocalFileName.Data()));
9e080f92 2000
2001 return fullLocalFileName.Data();
2bb7b766 2002}
2003
2004//______________________________________________________________________________________________
9d733021 2005Bool_t AliShuttle::RetrieveFile(UInt_t system, const char* fxsFileName, const char* localFileName)
9e080f92 2006{
9827400b 2007 //
2008 // Copies file from FXS to local Shuttle machine
2009 //
2bb7b766 2010
9e080f92 2011 // check temp directory: trying to cd to temp; if it does not exist, create it
9d733021 2012 AliDebug(2, Form("Copy file %s from %s FXS into %s/%s",
2013 GetSystemName(system), fxsFileName, GetShuttleTempDir(), localFileName));
9e080f92 2014
36c99a6a 2015 void* dir = gSystem->OpenDirectory(GetShuttleTempDir());
9e080f92 2016 if (dir == NULL) {
36c99a6a 2017 if (gSystem->mkdir(GetShuttleTempDir(), kTRUE)) {
2018 AliError(Form("Can't open directory <%s>", GetShuttleTempDir()));
9e080f92 2019 return kFALSE;
2020 }
2021
2022 } else {
2023 gSystem->FreeDirectory(dir);
2024 }
2025
9d733021 2026 TString baseFXSFolder;
2027 if (system == kDAQ)
2028 {
2029 baseFXSFolder = "FES/";
2030 }
2031 else if (system == kDCS)
2032 {
2033 baseFXSFolder = "";
2034 }
2035 else if (system == kHLT)
2036 {
42fde080 2037 baseFXSFolder = "/opt/FXS/";
9d733021 2038 }
2039
2040
2041 TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s/%s",
2042 fConfig->GetFXSPort(system),
2043 fConfig->GetFXSUser(system),
2044 fConfig->GetFXSHost(system),
2045 baseFXSFolder.Data(),
2046 fxsFileName,
36c99a6a 2047 GetShuttleTempDir(),
9e080f92 2048 localFileName);
2049
2050 AliDebug(2, Form("%s",command.Data()));
2051
4b95672b 2052 Bool_t result = (gSystem->Exec(command.Data()) == 0);
9e080f92 2053
4b95672b 2054 return result;
9e080f92 2055}
2056
2057//______________________________________________________________________________________________
9d733021 2058TList* AliShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
2059{
9827400b 2060 //
2061 // Get sources producing the condition file Id from file exchange servers
4a33bdd9 2062 // if id is NULL all sources are returned (distinct)
9827400b 2063 //
1bcd28db 2064
2065 Log(detector, Form("GetFileSources - Retrieving sources with id %s from %s", id, GetSystemName(system)));
9827400b 2066
2067 // check if test mode should simulate a FXS error
2068 if (fTestMode & kErrorFXSSources)
2069 {
2070 Log(detector, Form("GetFileSources - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2071 return 0;
2072 }
2073
9d733021 2074 if (system == kDCS)
2075 {
6297b37d 2076 AliWarning("DCS system has only one source of data!");
2077 TList *list = new TList();
2078 list->SetOwner(1);
2079 list->Add(new TObjString(" "));
2080 return list;
9d733021 2081 }
9e080f92 2082
2083 // check connection, in case connect
9d733021 2084 if (!Connect(system))
2085 {
4a33bdd9 2086 Log(detector, Form("GetFileSources - Couldn't connect to %s FXS database", GetSystemName(system)));
9d733021 2087 return NULL;
9e080f92 2088 }
2089
9d733021 2090 TString sourceName = 0;
2091 if (system == kDAQ)
2092 {
2093 sourceName = "DAQsource";
2094 } else if (system == kHLT)
2095 {
2096 sourceName = "DDLnumbers";
2097 }
2098
4a33bdd9 2099 TString sqlQueryStart = Form("select distinct %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
2100 TString whereClause = Form("run=%d and detector=\"%s\"",
2101 GetCurrentRun(), detector);
2102 if (id)
2103 whereClause += Form(" and fileId=\"%s\"", id);
9e080f92 2104 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2105
2106 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2107
2108 // Query execution
2109 TSQLResult* aResult;
9d733021 2110 aResult = fServer[system]->Query(sqlQuery);
9e080f92 2111 if (!aResult) {
9d733021 2112 Log(detector, Form("GetFileSources - Can't execute SQL query to %s database for id: %s",
2113 GetSystemName(system), id));
9e080f92 2114 return 0;
2115 }
2116
86aa42c3 2117 TList *list = new TList();
2118 list->SetOwner(1);
2119
9d733021 2120 if (aResult->GetRowCount() == 0)
2121 {
9e080f92 2122 Log(detector,
9d733021 2123 Form("GetFileSources - No entry in %s FXS table for id: %s", GetSystemName(system), id));
9e080f92 2124 delete aResult;
86aa42c3 2125 return list;
9e080f92 2126 }
2127
1bcd28db 2128 Log(detector, Form("GetFileSources - Found %d sources", aResult->GetRowCount()));
9e080f92 2129
1bcd28db 2130 TSQLRow* aRow;
9d733021 2131 while ((aRow = aResult->Next()))
2132 {
9e080f92 2133
9d733021 2134 TString source(aRow->GetField(0), aRow->GetFieldLength(0));
2135 AliDebug(2, Form("%s = %s", sourceName.Data(), source.Data()));
2136 list->Add(new TObjString(source));
9e080f92 2137 delete aRow;
2138 }
9d733021 2139
9e080f92 2140 delete aResult;
2141
2142 return list;
2bb7b766 2143}
2144
4a33bdd9 2145//______________________________________________________________________________________________
2146TList* AliShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
2147{
2148 //
2149 // Get all ids of condition files produced by a given source from file exchange servers
2150 //
2151
1bcd28db 2152 Log(detector, Form("GetFileIDs - Retrieving ids with source %s with %s", source, GetSystemName(system)));
2153
4a33bdd9 2154 // check if test mode should simulate a FXS error
2155 if (fTestMode & kErrorFXSSources)
2156 {
2157 Log(detector, Form("GetFileIDs - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2158 return 0;
2159 }
2160
2161 // check connection, in case connect
2162 if (!Connect(system))
2163 {
2164 Log(detector, Form("GetFileIDs - Couldn't connect to %s FXS database", GetSystemName(system)));
2165 return NULL;
2166 }
2167
2168 TString sourceName = 0;
2169 if (system == kDAQ)
2170 {
2171 sourceName = "DAQsource";
2172 } else if (system == kHLT)
2173 {
2174 sourceName = "DDLnumbers";
2175 }
2176
2177 TString sqlQueryStart = Form("select fileId from %s where", fConfig->GetFXSdbTable(system));
2178 TString whereClause = Form("run=%d and detector=\"%s\"",
2179 GetCurrentRun(), detector);
2180 if (sourceName.Length() > 0 && source)
2181 whereClause += Form(" and %s=\"%s\"", sourceName.Data(), source);
2182 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2183
2184 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2185
2186 // Query execution
2187 TSQLResult* aResult;
2188 aResult = fServer[system]->Query(sqlQuery);
2189 if (!aResult) {
2190 Log(detector, Form("GetFileIDs - Can't execute SQL query to %s database for source: %s",
2191 GetSystemName(system), source));
2192 return 0;
2193 }
2194
2195 TList *list = new TList();
2196 list->SetOwner(1);
2197
2198 if (aResult->GetRowCount() == 0)
2199 {
2200 Log(detector,
2201 Form("GetFileIDs - No entry in %s FXS table for source: %s", GetSystemName(system), source));
2202 delete aResult;
2203 return list;
2204 }
2205
1bcd28db 2206 Log(detector, Form("GetFileIDs - Found %d ids", aResult->GetRowCount()));
2207
4a33bdd9 2208 TSQLRow* aRow;
2209
2210 while ((aRow = aResult->Next()))
2211 {
2212
2213 TString id(aRow->GetField(0), aRow->GetFieldLength(0));
2214 AliDebug(2, Form("fileId = %s", id.Data()));
2215 list->Add(new TObjString(id));
2216 delete aRow;
2217 }
2218
2219 delete aResult;
2220
2221 return list;
2222}
2223
2bb7b766 2224//______________________________________________________________________________________________
9d733021 2225Bool_t AliShuttle::Connect(Int_t system)
2bb7b766 2226{
9827400b 2227 // Connect to MySQL Server of the system's FXS MySQL databases
2228 // DAQ Logbook, Shuttle Logbook and DAQ FXS db are on the same host
2229 //
57f50b3c 2230
9d733021 2231 // check connection: if already connected return
2232 if(fServer[system] && fServer[system]->IsConnected()) return kTRUE;
57f50b3c 2233
9d733021 2234 TString dbHost, dbUser, dbPass, dbName;
57f50b3c 2235
9d733021 2236 if (system < 3) // FXS db servers
2237 {
2238 dbHost = Form("mysql://%s:%d", fConfig->GetFXSdbHost(system), fConfig->GetFXSdbPort(system));
2239 dbUser = fConfig->GetFXSdbUser(system);
2240 dbPass = fConfig->GetFXSdbPass(system);
2241 dbName = fConfig->GetFXSdbName(system);
2242 } else { // Run & Shuttle logbook servers
2243 // TODO Will the Shuttle logbook server be the same as the Run logbook server ???
2244 dbHost = Form("mysql://%s:%d", fConfig->GetDAQlbHost(), fConfig->GetDAQlbPort());
2245 dbUser = fConfig->GetDAQlbUser();
2246 dbPass = fConfig->GetDAQlbPass();
2247 dbName = fConfig->GetDAQlbDB();
2248 }
57f50b3c 2249
9d733021 2250 fServer[system] = TSQLServer::Connect(dbHost.Data(), dbUser.Data(), dbPass.Data());
2251 if (!fServer[system] || !fServer[system]->IsConnected()) {
2252 if(system < 3)
2253 {
2254 AliError(Form("Can't establish connection to FXS database for %s",
2255 AliShuttleInterface::GetSystemName(system)));
2256 } else {
2257 AliError("Can't establish connection to Run logbook.");
57f50b3c 2258 }
9d733021 2259 if(fServer[system]) delete fServer[system];
2260 return kFALSE;
2bb7b766 2261 }
57f50b3c 2262
9d733021 2263 // Get tables
2264 TSQLResult* aResult=0;
2265 switch(system){
2266 case kDAQ:
2267 aResult = fServer[kDAQ]->GetTables(dbName.Data());
2268 break;
2269 case kDCS:
2270 aResult = fServer[kDCS]->GetTables(dbName.Data());
2271 break;
2272 case kHLT:
2273 aResult = fServer[kHLT]->GetTables(dbName.Data());
2274 break;
2275 default:
2276 aResult = fServer[3]->GetTables(dbName.Data());
2277 break;
2278 }
2279
2280 delete aResult;
2bb7b766 2281 return kTRUE;
2282}
57f50b3c 2283
9e080f92 2284//______________________________________________________________________________________________
9d733021 2285Bool_t AliShuttle::UpdateTable()
9e080f92 2286{
9827400b 2287 //
2288 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2289 //
9e080f92 2290
9d733021 2291 Bool_t result = kTRUE;
9e080f92 2292
9d733021 2293 for (UInt_t system=0; system<3; system++)
2294 {
2295 if(!fFXSCalled[system]) continue;
9e080f92 2296
9d733021 2297 // check connection, in case connect
2298 if (!Connect(system))
2299 {
2300 Log(fCurrentDetector, Form("UpdateTable - Couldn't connect to %s FXS database", GetSystemName(system)));
2301 result = kFALSE;
2302 continue;
9e080f92 2303 }
9e080f92 2304
9d733021 2305 TTimeStamp now; // now
2306
2307 // Loop on FXS list entries
2308 TIter iter(&fFXSlist[system]);
2309 TObjString *aFXSentry=0;
2310 while ((aFXSentry = dynamic_cast<TObjString*> (iter.Next())))
2311 {
2312 TString aFXSentrystr = aFXSentry->String();
2313 TObjArray *aFXSarray = aFXSentrystr.Tokenize("#!?!#");
2314 if (!aFXSarray || aFXSarray->GetEntries() != 2 )
2315 {
2316 Log(fCurrentDetector, Form("UpdateTable - error updating %s FXS entry. Check string: <%s>",
2317 GetSystemName(system), aFXSentrystr.Data()));
2318 if(aFXSarray) delete aFXSarray;
2319 result = kFALSE;
2320 continue;
2321 }
2322 const char* fileId = ((TObjString*) aFXSarray->At(0))->GetName();
2323 const char* source = ((TObjString*) aFXSarray->At(1))->GetName();
2324
2325 TString whereClause;
2326 if (system == kDAQ)
2327 {
2328 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DAQsource=\"%s\";",
2329 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2330 }
2331 else if (system == kDCS)
2332 {
2333 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\";",
2334 GetCurrentRun(), fCurrentDetector.Data(), fileId);
2335 }
2336 else if (system == kHLT)
2337 {
2338 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DDLnumbers=\"%s\";",
2339 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2340 }
2341
2342 delete aFXSarray;
9e080f92 2343
9d733021 2344 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2345 now.GetSec(), whereClause.Data());
9e080f92 2346
9d733021 2347 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
9e080f92 2348
9d733021 2349 // Query execution
2350 TSQLResult* aResult;
2351 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2352 if (!aResult)
2353 {
2354 Log(fCurrentDetector, Form("UpdateTable - %s db: can't execute SQL query <%s>",
2355 GetSystemName(system), sqlQuery.Data()));
2356 result = kFALSE;
2357 continue;
2358 }
2359 delete aResult;
9e080f92 2360 }
9e080f92 2361 }
2362
9d733021 2363 return result;
9e080f92 2364}
57f50b3c 2365
3301427a 2366//______________________________________________________________________________________________
2367Bool_t AliShuttle::UpdateTableFailCase()
2368{
9827400b 2369 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2370 // this is called in case the preprocessor is declared failed for the current run, because
2371 // the fields are updated only in case of success
3301427a 2372
2373 Bool_t result = kTRUE;
2374
2375 for (UInt_t system=0; system<3; system++)
2376 {
2377 // check connection, in case connect
2378 if (!Connect(system))
2379 {
2380 Log(fCurrentDetector, Form("UpdateTableFailCase - Couldn't connect to %s FXS database",
2381 GetSystemName(system)));
2382 result = kFALSE;
2383 continue;
2384 }
2385
2386 TTimeStamp now; // now
2387
2388 // Loop on FXS list entries
2389
2390 TString whereClause = Form("where run=%d and detector=\"%s\";",
2391 GetCurrentRun(), fCurrentDetector.Data());
2392
2393
2394 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2395 now.GetSec(), whereClause.Data());
2396
2397 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2398
2399 // Query execution
2400 TSQLResult* aResult;
2401 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2402 if (!aResult)
2403 {
2404 Log(fCurrentDetector, Form("UpdateTableFailCase - %s db: can't execute SQL query <%s>",
2405 GetSystemName(system), sqlQuery.Data()));
2406 result = kFALSE;
2407 continue;
2408 }
2409 delete aResult;
2410 }
2411
2412 return result;
2413}
2414
2bb7b766 2415//______________________________________________________________________________________________
2416Bool_t AliShuttle::UpdateShuttleLogbook(const char* detector, const char* status)
2417{
e7f62f16 2418 //
2419 // Update Shuttle logbook filling detector or shuttle_done column
2420 // ex. of usage: UpdateShuttleLogbook("PHOS", "DONE") or UpdateShuttleLogbook("shuttle_done")
2421 //
57f50b3c 2422
2bb7b766 2423 // check connection, in case connect
be48e3ea 2424 if(!Connect(3)){
2bb7b766 2425 Log("SHUTTLE", "UpdateShuttleLogbook - Couldn't connect to DAQ Logbook.");
2426 return kFALSE;
57f50b3c 2427 }
2428
2bb7b766 2429 TString detName(detector);
2430 TString setClause;
e7f62f16 2431 if(detName == "shuttle_done")
2432 {
2bb7b766 2433 setClause = "set shuttle_done=1";
e7f62f16 2434
2435 // Send the information to ML
2436 TMonaLisaText mlStatus("SHUTTLE_status", "Done");
2437
2438 TList mlList;
2439 mlList.Add(&mlStatus);
2440
2441 fMonaLisa->SendParameters(&mlList);
2bb7b766 2442 } else {
2bb7b766 2443 TString statusStr(status);
2444 if(statusStr.Contains("done", TString::kIgnoreCase) ||
2445 statusStr.Contains("failed", TString::kIgnoreCase)){
eba76848 2446 setClause = Form("set %s=\"%s\"", detector, status);
2bb7b766 2447 } else {
2448 Log("SHUTTLE",
2449 Form("UpdateShuttleLogbook - Invalid status <%s> for detector %s",
2450 status, detector));
2451 return kFALSE;
2452 }
2453 }
57f50b3c 2454
2bb7b766 2455 TString whereClause = Form("where run=%d", GetCurrentRun());
2456
441b0e9c 2457 TString sqlQuery = Form("update %s %s %s",
2458 fConfig->GetShuttlelbTable(), setClause.Data(), whereClause.Data());
57f50b3c 2459
2bb7b766 2460 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2461
2462 // Query execution
2463 TSQLResult* aResult;
be48e3ea 2464 aResult = dynamic_cast<TSQLResult*> (fServer[3]->Query(sqlQuery));
2bb7b766 2465 if (!aResult) {
2466 Log("SHUTTLE", Form("UpdateShuttleLogbook - Can't execute query <%s>", sqlQuery.Data()));
2467 return kFALSE;
57f50b3c 2468 }
2bb7b766 2469 delete aResult;
57f50b3c 2470
2471 return kTRUE;
2472}
2473
2474//______________________________________________________________________________________________
2bb7b766 2475Int_t AliShuttle::GetCurrentRun() const
2476{
9827400b 2477 //
2478 // Get current run from logbook entry
2479 //
57f50b3c 2480
2bb7b766 2481 return fLogbookEntry ? fLogbookEntry->GetRun() : -1;
57f50b3c 2482}
2483
2484//______________________________________________________________________________________________
2bb7b766 2485UInt_t AliShuttle::GetCurrentStartTime() const
2486{
9827400b 2487 //
2488 // get current start time
2489 //
57f50b3c 2490
2bb7b766 2491 return fLogbookEntry ? fLogbookEntry->GetStartTime() : 0;
57f50b3c 2492}
2493
2494//______________________________________________________________________________________________
2bb7b766 2495UInt_t AliShuttle::GetCurrentEndTime() const
2496{
9827400b 2497 //
2498 // get current end time from logbook entry
2499 //
57f50b3c 2500
2bb7b766 2501 return fLogbookEntry ? fLogbookEntry->GetEndTime() : 0;
57f50b3c 2502}
2503
b948db8d 2504//______________________________________________________________________________________________
2505void AliShuttle::Log(const char* detector, const char* message)
2506{
9827400b 2507 //
2508 // Fill log string with a message
2509 //
b948db8d 2510
36c99a6a 2511 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
84090f85 2512 if (dir == NULL) {
36c99a6a 2513 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE)) {
2514 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
84090f85 2515 return;
2516 }
b948db8d 2517
84090f85 2518 } else {
2519 gSystem->FreeDirectory(dir);
2520 }
b948db8d 2521
cb343cfd 2522 TString toLog = Form("%s (%d): %s - ", TTimeStamp(time(0)).AsString("s"), getpid(), detector);
e7f62f16 2523 if (GetCurrentRun() >= 0)
2524 toLog += Form("run %d - ", GetCurrentRun());
2bb7b766 2525 toLog += Form("%s", message);
2526
84090f85 2527 AliInfo(toLog.Data());
ffa29e93 2528
2529 // if we redirect the log output already to the file, leave here
2530 if (fOutputRedirected && strcmp(detector, "SHUTTLE") != 0)
2531 return;
b948db8d 2532
ffa29e93 2533 TString fileName = GetLogFileName(detector);
e7f62f16 2534
84090f85 2535 gSystem->ExpandPathName(fileName);
2536
2537 ofstream logFile;
2538 logFile.open(fileName, ofstream::out | ofstream::app);
2539
2540 if (!logFile.is_open()) {
2541 AliError(Form("Could not open file %s", fileName.Data()));
2542 return;
2543 }
7bfb2090 2544
84090f85 2545 logFile << toLog.Data() << "\n";
b948db8d 2546
84090f85 2547 logFile.close();
b948db8d 2548}
2bb7b766 2549
ffa29e93 2550//______________________________________________________________________________________________
2551TString AliShuttle::GetLogFileName(const char* detector) const
2552{
2553 //
2554 // returns the name of the log file for a given sub detector
2555 //
2556
2557 TString fileName;
2558
2559 if (GetCurrentRun() >= 0)
2560 fileName.Form("%s/%s_%d.log", GetShuttleLogDir(), detector, GetCurrentRun());
2561 else
2562 fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
2563
2564 return fileName;
2565}
2566
2bb7b766 2567//______________________________________________________________________________________________
2568Bool_t AliShuttle::Collect(Int_t run)
2569{
9827400b 2570 //
2571 // Collects conditions data for all UNPROCESSED run written to DAQ LogBook in case of run = -1 (default)
2572 // If a dedicated run is given this run is processed
2573 //
2574 // In operational mode, this is the Shuttle function triggered by the EOR signal.
2575 //
2bb7b766 2576
eba76848 2577 if (run == -1)
2578 Log("SHUTTLE","Collect - Shuttle called. Collecting conditions data for unprocessed runs");
2579 else
2580 Log("SHUTTLE", Form("Collect - Shuttle called. Collecting conditions data for run %d", run));
cb343cfd 2581
2582 SetLastAction("Starting");
2bb7b766 2583
2584 TString whereClause("where shuttle_done=0");
eba76848 2585 if (run != -1)
2586 whereClause += Form(" and run=%d", run);
2bb7b766 2587
2588 TObjArray shuttleLogbookEntries;
be48e3ea 2589 if (!QueryShuttleLogbook(whereClause, shuttleLogbookEntries))
2590 {
cb343cfd 2591 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2bb7b766 2592 return kFALSE;
2593 }
2594
9e080f92 2595 if (shuttleLogbookEntries.GetEntries() == 0)
2596 {
2597 if (run == -1)
2598 Log("SHUTTLE","Collect - Found no UNPROCESSED runs in Shuttle logbook");
2599 else
2600 Log("SHUTTLE", Form("Collect - Run %d is already DONE "
2601 "or it does not exist in Shuttle logbook", run));
2602 return kTRUE;
2603 }
2604
be48e3ea 2605 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2606 fFirstUnprocessed[iDet] = kTRUE;
2607
fc5a4708 2608 if (run != -1)
be48e3ea 2609 {
2610 // query Shuttle logbook for earlier runs, check if some detectors are unprocessed,
2611 // flag them into fFirstUnprocessed array
2612 TString whereClause(Form("where shuttle_done=0 and run < %d", run));
2613 TObjArray tmpLogbookEntries;
2614 if (!QueryShuttleLogbook(whereClause, tmpLogbookEntries))
2615 {
2616 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2617 return kFALSE;
2618 }
2619
2620 TIter iter(&tmpLogbookEntries);
2621 AliShuttleLogbookEntry* anEntry = 0;
2622 while ((anEntry = dynamic_cast<AliShuttleLogbookEntry*> (iter.Next())))
2623 {
2624 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2625 {
2626 if (anEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
2627 {
2628 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
2629 anEntry->GetRun(), GetDetName(iDet)));
2630 fFirstUnprocessed[iDet] = kFALSE;
2631 }
2632 }
2633
2634 }
2635
2636 }
2637
2638 if (!RetrieveConditionsData(shuttleLogbookEntries))
2639 {
cb343cfd 2640 Log("SHUTTLE", "Collect - Process of at least one run failed");
2bb7b766 2641 return kFALSE;
2642 }
2643
36c99a6a 2644 Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
eba76848 2645 return kTRUE;
2bb7b766 2646}
2647
2bb7b766 2648//______________________________________________________________________________________________
2649Bool_t AliShuttle::RetrieveConditionsData(const TObjArray& dateEntries)
2650{
9827400b 2651 //
2652 // Retrieve conditions data for all runs that aren't processed yet
2653 //
2bb7b766 2654
2655 Bool_t hasError = kFALSE;
2656
2657 TIter iter(&dateEntries);
2658 AliShuttleLogbookEntry* anEntry;
2659
2660 while ((anEntry = (AliShuttleLogbookEntry*) iter.Next())){
2661 if (!Process(anEntry)){
2662 hasError = kTRUE;
2663 }
4b95672b 2664
2665 // clean SHUTTLE temp directory
3301427a 2666 TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
2667 RemoveFile(filename.Data());
2bb7b766 2668 }
2669
2670 return hasError == kFALSE;
2671}
cb343cfd 2672
2673//______________________________________________________________________________________________
2674ULong_t AliShuttle::GetTimeOfLastAction() const
2675{
9827400b 2676 //
2677 // Gets time of last action
2678 //
2679
cb343cfd 2680 ULong_t tmp;
36c99a6a 2681
cb343cfd 2682 fMonitoringMutex->Lock();
be48e3ea 2683
cb343cfd 2684 tmp = fLastActionTime;
36c99a6a 2685
cb343cfd 2686 fMonitoringMutex->UnLock();
36c99a6a 2687
cb343cfd 2688 return tmp;
2689}
2690
2691//______________________________________________________________________________________________
2692const TString AliShuttle::GetLastAction() const
2693{
9827400b 2694 //
cb343cfd 2695 // returns a string description of the last action
9827400b 2696 //
cb343cfd 2697
2698 TString tmp;
36c99a6a 2699
cb343cfd 2700 fMonitoringMutex->Lock();
2701
2702 tmp = fLastAction;
2703
2704 fMonitoringMutex->UnLock();
2705
36c99a6a 2706 return tmp;
cb343cfd 2707}
2708
2709//______________________________________________________________________________________________
2710void AliShuttle::SetLastAction(const char* action)
2711{
9827400b 2712 //
cb343cfd 2713 // updates the monitoring variables
9827400b 2714 //
36c99a6a 2715
cb343cfd 2716 fMonitoringMutex->Lock();
36c99a6a 2717
cb343cfd 2718 fLastAction = action;
2719 fLastActionTime = time(0);
2720
2721 fMonitoringMutex->UnLock();
2722}
eba76848 2723
2724//______________________________________________________________________________________________
2725const char* AliShuttle::GetRunParameter(const char* param)
2726{
9827400b 2727 //
2728 // returns run parameter read from DAQ logbook
2729 //
eba76848 2730
2731 if(!fLogbookEntry) {
2732 AliError("No logbook entry!");
2733 return 0;
2734 }
2735
2736 return fLogbookEntry->GetRunParameter(param);
2737}
57c1a579 2738
d386d623 2739//______________________________________________________________________________________________
9827400b 2740AliCDBEntry* AliShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
d386d623 2741{
9827400b 2742 //
2743 // returns object from OCDB valid for current run
2744 //
d386d623 2745
9827400b 2746 if (fTestMode & kErrorOCDB)
2747 {
2748 Log(detector, "GetFromOCDB - In TESTMODE - Simulating error with OCDB");
2749 return 0;
2750 }
2751
d386d623 2752 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
2753 if (!sto)
2754 {
9827400b 2755 Log(detector, "GetFromOCDB - Cannot activate main OCDB for query!");
d386d623 2756 return 0;
2757 }
2758
2759 return dynamic_cast<AliCDBEntry*> (sto->Get(path, GetCurrentRun()));
2760}
2761
57c1a579 2762//______________________________________________________________________________________________
2763Bool_t AliShuttle::SendMail()
2764{
9827400b 2765 //
2766 // sends a mail to the subdetector expert in case of preprocessor error
2767 //
2768
2769 if (fTestMode != kNone)
2770 return kTRUE;
57c1a579 2771
36c99a6a 2772 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
57c1a579 2773 if (dir == NULL)
2774 {
36c99a6a 2775 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
57c1a579 2776 {
36c99a6a 2777 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
57c1a579 2778 return kFALSE;
2779 }
2780
2781 } else {
2782 gSystem->FreeDirectory(dir);
2783 }
2784
2785 TString bodyFileName;
36c99a6a 2786 bodyFileName.Form("%s/mail.body", GetShuttleLogDir());
57c1a579 2787 gSystem->ExpandPathName(bodyFileName);
2788
2789 ofstream mailBody;
2790 mailBody.open(bodyFileName, ofstream::out);
2791
2792 if (!mailBody.is_open())
2793 {
2794 AliError(Form("Could not open mail body file %s", bodyFileName.Data()));
2795 return kFALSE;
2796 }
2797
2798 TString to="";
2799 TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
2800 TObjString *anExpert=0;
2801 while ((anExpert = (TObjString*) iterExperts.Next()))
2802 {
2803 to += Form("%s,", anExpert->GetName());
2804 }
2805 to.Remove(to.Length()-1);
909732f7 2806 AliDebug(2, Form("to: %s",to.Data()));
57c1a579 2807
86aa42c3 2808 if (to.IsNull()) {
36c99a6a 2809 AliInfo("List of detector responsibles not yet set!");
2810 return kFALSE;
2811 }
2812
57c1a579 2813 TString cc="alberto.colla@cern.ch";
2814
546242fb 2815 TString subject = Form("%s Shuttle preprocessor FAILED in run %d !",
57c1a579 2816 fCurrentDetector.Data(), GetCurrentRun());
909732f7 2817 AliDebug(2, Form("subject: %s", subject.Data()));
57c1a579 2818
2819 TString body = Form("Dear %s expert(s), \n\n", fCurrentDetector.Data());
2820 body += Form("SHUTTLE just detected that your preprocessor "
546242fb 2821 "failed processing run %d!!\n\n", GetCurrentRun());
2822 body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n", fCurrentDetector.Data());
2823 body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
2824 body += Form("Find the %s log for the current run on \n\n"
2825 "\thttp://pcalishuttle01.cern.ch:8880/logs/%s_%d.log \n\n",
2826 fCurrentDetector.Data(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 2827 body += Form("The last 10 lines of %s log file are following:\n\n");
2828
909732f7 2829 AliDebug(2, Form("Body begin: %s", body.Data()));
57c1a579 2830
2831 mailBody << body.Data();
2832 mailBody.close();
2833 mailBody.open(bodyFileName, ofstream::out | ofstream::app);
2834
9d733021 2835 TString logFileName = Form("%s/%s_%d.log", GetShuttleLogDir(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 2836 TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
2837 if (gSystem->Exec(tailCommand.Data()))
2838 {
2839 mailBody << Form("%s log file not found ...\n\n", fCurrentDetector.Data());
2840 }
2841
2842 TString endBody = Form("------------------------------------------------------\n\n");
36c99a6a 2843 endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
2844 endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
546242fb 2845 endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
57c1a579 2846
909732f7 2847 AliDebug(2, Form("Body end: %s", endBody.Data()));
57c1a579 2848
2849 mailBody << endBody.Data();
2850
2851 mailBody.close();
2852
2853 // send mail!
2854 TString mailCommand = Form("mail -s \"%s\" -c %s %s < %s",
2855 subject.Data(),
2856 cc.Data(),
2857 to.Data(),
2858 bodyFileName.Data());
909732f7 2859 AliDebug(2, Form("mail command: %s", mailCommand.Data()));
57c1a579 2860
2861 Bool_t result = gSystem->Exec(mailCommand.Data());
2862
2863 return result == 0;
2864}
d386d623 2865
441b0e9c 2866//______________________________________________________________________________________________
9827400b 2867const char* AliShuttle::GetRunType()
441b0e9c 2868{
9827400b 2869 //
2870 // returns run type read from "run type" logbook
2871 //
441b0e9c 2872
2873 if(!fLogbookEntry) {
2874 AliError("No logbook entry!");
2875 return 0;
2876 }
2877
9827400b 2878 return fLogbookEntry->GetRunType();
441b0e9c 2879}
2880
4859271b 2881//______________________________________________________________________________________________
2882Bool_t AliShuttle::GetHLTStatus()
2883{
2884 // Return HLT status (ON=1 OFF=0)
2885 // Converts the HLT status from the status string read in the run logbook (not just a bool)
2886
2887 if(!fLogbookEntry) {
2888 AliError("No logbook entry!");
2889 return 0;
2890 }
2891
2892 // TODO implement when HLTStatus is inserted in run logbook
2893 //TString hltStatus = fLogbookEntry->GetRunParameter("HLTStatus");
2894 //if(hltStatus == "OFF") {return kFALSE};
2895
2896 return kTRUE;
2897}
2898
d386d623 2899//______________________________________________________________________________________________
2900void AliShuttle::SetShuttleTempDir(const char* tmpDir)
2901{
9827400b 2902 //
2903 // sets Shuttle temp directory
2904 //
d386d623 2905
2906 fgkShuttleTempDir = gSystem->ExpandPathName(tmpDir);
2907}
2908
2909//______________________________________________________________________________________________
2910void AliShuttle::SetShuttleLogDir(const char* logDir)
2911{
9827400b 2912 //
2913 // sets Shuttle log directory
2914 //
d386d623 2915
2916 fgkShuttleLogDir = gSystem->ExpandPathName(logDir);
2917}