New function StoreRunMetadataFile added to preprocessor and Shuttle interface
[u/mrichter/AliRoot.git] / SHUTTLE / AliShuttle.cxx
CommitLineData
73abe331 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
1790d4b7 18Revision 1.58 2007/09/28 15:27:40 acolla
19
20AliDCSClient "multiSplit" option added in the DCS configuration
21in AliDCSMessage: variable MAX_BODY_SIZE set to 500000
22
542b6cc8 23Revision 1.57 2007/09/27 16:53:13 acolla
24Detectors can have more than one AMANDA server. SHUTTLE queries the servers sequentially,
25merges the dcs aliases/DPs in one TMap and sends it to the preprocessor.
26
1d172743 27Revision 1.56 2007/09/14 16:46:14 jgrosseo
281) Connect and Close are called before and after each query, so one can
29keep the same AliDCSClient object.
302) The splitting of a query is moved to GetDPValues/GetAliasValues.
313) Splitting interval can be specified in constructor
32
b41b252a 33Revision 1.55 2007/08/06 12:26:40 acolla
34Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
35read from the run logbook.
36
4859271b 37Revision 1.54 2007/07/12 09:51:25 jgrosseo
38removed duplicated log message in GetFile
39
4f0749a8 40Revision 1.53 2007/07/12 09:26:28 jgrosseo
41updating hlt fxs base path
42
42fde080 43Revision 1.52 2007/07/12 08:06:45 jgrosseo
44adding log messages in getfile... functions
45adding not implemented copy constructor in alishuttleconfigholder
46
1bcd28db 47Revision 1.51 2007/07/03 17:24:52 acolla
48root moved to v5-16-00. TFileMerger->Cp moved to TFile::Cp.
49
a986b218 50Revision 1.50 2007/07/02 17:19:32 acolla
51preprocessor is run in a temp directory that is removed when process is finished.
52
5bac2bde 53Revision 1.49 2007/06/29 10:45:06 acolla
54Number of columns in MySql Shuttle logbook increased by one (HLT added)
55
db99d43e 56Revision 1.48 2007/06/21 13:06:19 acolla
57GetFileSources returns dummy list with 1 source if system=DCS (better than
58returning error as it was)
59
6297b37d 60Revision 1.47 2007/06/19 17:28:56 acolla
61HLT updated; missing map bug removed.
62
dc25836b 63Revision 1.46 2007/06/09 13:01:09 jgrosseo
64Switching to retrieval of several DCS DPs at a time (multiDPrequest)
65
a038aa70 66Revision 1.45 2007/05/30 06:35:20 jgrosseo
67Adding functionality to the Shuttle/TestShuttle:
68o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
69o) Function to retrieve list of IDs for a given source (GetFileIDs)
70These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
71Example code has been added to the TestProcessor in TestShuttle
72
4a33bdd9 73Revision 1.44 2007/05/11 16:09:32 acolla
74Reference files for ITS, MUON and PHOS are now stored in OfflineDetName/OnlineDetName/run_...
75example: ITS/SPD/100_filename.root
76
2d9019b4 77Revision 1.43 2007/05/10 09:59:51 acolla
78Various bug fixes in StoreRefFilesToGrid; Cleaning of reference storage before processing detector (CleanReferenceStorage)
79
546242fb 80Revision 1.42 2007/05/03 08:01:39 jgrosseo
81typo in last commit :-(
82
8b739301 83Revision 1.41 2007/05/03 08:00:48 jgrosseo
84fixing log message when pp want to skip dcs value retrieval
85
651fdaab 86Revision 1.40 2007/04/27 07:06:48 jgrosseo
87GetFileSources returns empty list in case of no files, but successful query
88No mails sent in testmode
89
86aa42c3 90Revision 1.39 2007/04/17 12:43:57 acolla
91Correction in StoreOCDB; change of text in mail to detector expert
92
26758fce 93Revision 1.38 2007/04/12 08:26:18 jgrosseo
94updated comment
95
3c2a21c8 96Revision 1.37 2007/04/10 16:53:14 jgrosseo
97redirecting sub detector stdout, stderr to sub detector log file
98
3d8bc902 99Revision 1.35 2007/04/04 16:26:38 acolla
1001. Re-organization of function calls in TestPreprocessor to make it more meaningful.
1012. Added missing dependency in test preprocessors.
1023. in AliShuttle.cxx: processing time and memory consumption info on a single line.
103
886d60e6 104Revision 1.34 2007/04/04 10:33:36 jgrosseo
1051) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
106In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
107
1082) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
109
1103) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
111
1124) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
113
1145) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
115If you always need DCS data (like before), you do not need to implement it.
116
1176) The run type has been added to the monitoring page
118
9827400b 119Revision 1.33 2007/04/03 13:56:01 acolla
120Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
121run type.
122
3301427a 123Revision 1.32 2007/02/28 10:41:56 acolla
124Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
125AliPreprocessor::GetRunType() function.
126Added some ldap definition files.
127
d386d623 128Revision 1.30 2007/02/13 11:23:21 acolla
129Moved getters and setters of Shuttle's main OCDB/Reference, local
130OCDB/Reference, temp and log folders to AliShuttleInterface
131
9d733021 132Revision 1.27 2007/01/30 17:52:42 jgrosseo
133adding monalisa monitoring
134
e7f62f16 135Revision 1.26 2007/01/23 19:20:03 acolla
136Removed old ldif files, added TOF, MCH ldif files. Added some options in
137AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
138SetShuttleLogDir
139
36c99a6a 140Revision 1.25 2007/01/15 19:13:52 acolla
141Moved some AliInfo to AliDebug in SendMail function
142
fc5a4708 143Revision 1.21 2006/12/07 08:51:26 jgrosseo
144update (alberto):
145table, db names in ldap configuration
146added GRP preprocessor
147DCS data can also be retrieved by data point
148
2c15234c 149Revision 1.20 2006/11/16 16:16:48 jgrosseo
150introducing strict run ordering flag
151removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
152
be48e3ea 153Revision 1.19 2006/11/06 14:23:04 jgrosseo
154major update (Alberto)
155o) reading of run parameters from the logbook
156o) online offline naming conversion
157o) standalone DCSclient package
158
eba76848 159Revision 1.18 2006/10/20 15:22:59 jgrosseo
160o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
161o) Merging Collect, CollectAll, CollectNew function
162o) Removing implementation of empty copy constructors (declaration still there!)
163
cb343cfd 164Revision 1.17 2006/10/05 16:20:55 jgrosseo
165adapting to new CDB classes
166
6ec0e06c 167Revision 1.16 2006/10/05 15:46:26 jgrosseo
168applying to the new interface
169
481441a2 170Revision 1.15 2006/10/02 16:38:39 jgrosseo
171update (alberto):
172fixed memory leaks
173storing of objects that failed to be stored to the grid before
174interfacing of shuttle status table in daq system
175
2bb7b766 176Revision 1.14 2006/08/29 09:16:05 jgrosseo
177small update
178
85a80aa9 179Revision 1.13 2006/08/15 10:50:00 jgrosseo
180effc++ corrections (alberto)
181
4f0ab988 182Revision 1.12 2006/08/08 14:19:29 jgrosseo
183Update to shuttle classes (Alberto)
184
185- Possibility to set the full object's path in the Preprocessor's and
186Shuttle's Store functions
187- Possibility to extend the object's run validity in the same classes
188("startValidity" and "validityInfinite" parameters)
189- Implementation of the StoreReferenceData function to store reference
190data in a dedicated CDB storage.
191
84090f85 192Revision 1.11 2006/07/21 07:37:20 jgrosseo
193last run is stored after each run
194
7bfb2090 195Revision 1.10 2006/07/20 09:54:40 jgrosseo
196introducing status management: The processing per subdetector is divided into several steps,
197after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
198can keep track of the number of failures and skips further processing after a certain threshold is
199exceeded. These thresholds can be configured in LDAP.
200
5164a766 201Revision 1.9 2006/07/19 10:09:55 jgrosseo
202new configuration, accesst to DAQ FES (Alberto)
203
57f50b3c 204Revision 1.8 2006/07/11 12:44:36 jgrosseo
205adding parameters for extended validity range of data produced by preprocessor
206
17111222 207Revision 1.7 2006/07/10 14:37:09 jgrosseo
208small fix + todo comment
209
e090413b 210Revision 1.6 2006/07/10 13:01:41 jgrosseo
211enhanced storing of last sucessfully processed run (alberto)
212
a7160fe9 213Revision 1.5 2006/07/04 14:59:57 jgrosseo
214revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
215
45a493ce 216Revision 1.4 2006/06/12 09:11:16 jgrosseo
217coding conventions (Alberto)
218
58bc3020 219Revision 1.3 2006/06/06 14:26:40 jgrosseo
220o) removed files that were moved to STEER
221o) shuttle updated to follow the new interface (Alberto)
222
b948db8d 223Revision 1.2 2006/03/07 07:52:34 hristov
224New version (B.Yordanov)
225
d477ad88 226Revision 1.6 2005/11/19 17:19:14 byordano
227RetrieveDATEEntries and RetrieveConditionsData added
228
229Revision 1.5 2005/11/19 11:09:27 byordano
230AliShuttle declaration added
231
232Revision 1.4 2005/11/17 17:47:34 byordano
233TList changed to TObjArray
234
235Revision 1.3 2005/11/17 14:43:23 byordano
236import to local CVS
237
238Revision 1.1.1.1 2005/10/28 07:33:58 hristov
239Initial import as subdirectory in AliRoot
240
73abe331 241Revision 1.2 2005/09/13 08:41:15 byordano
242default startTime endTime added
243
244Revision 1.4 2005/08/30 09:13:02 byordano
245some docs added
246
247Revision 1.3 2005/08/29 21:15:47 byordano
248some docs added
249
250*/
251
252//
253// This class is the main manager for AliShuttle.
254// It organizes the data retrieval from DCS and call the
b948db8d 255// interface methods of AliPreprocessor.
73abe331 256// For every detector in AliShuttleConfgi (see AliShuttleConfig),
257// data for its set of aliases is retrieved. If there is registered
b948db8d 258// AliPreprocessor for this detector then it will be used
259// accroding to the schema (see AliPreprocessor).
260// If there isn't registered AliPreprocessor than the retrieved
73abe331 261// data is stored automatically to the undelying AliCDBStorage.
262// For detSpec is used the alias name.
263//
264
265#include "AliShuttle.h"
266
267#include "AliCDBManager.h"
268#include "AliCDBStorage.h"
269#include "AliCDBId.h"
84090f85 270#include "AliCDBRunRange.h"
271#include "AliCDBPath.h"
5164a766 272#include "AliCDBEntry.h"
73abe331 273#include "AliShuttleConfig.h"
eba76848 274#include "DCSClient/AliDCSClient.h"
73abe331 275#include "AliLog.h"
b948db8d 276#include "AliPreprocessor.h"
5164a766 277#include "AliShuttleStatus.h"
2bb7b766 278#include "AliShuttleLogbookEntry.h"
73abe331 279
57f50b3c 280#include <TSystem.h>
58bc3020 281#include <TObject.h>
b948db8d 282#include <TString.h>
57f50b3c 283#include <TTimeStamp.h>
73abe331 284#include <TObjString.h>
57f50b3c 285#include <TSQLServer.h>
286#include <TSQLResult.h>
287#include <TSQLRow.h>
cb343cfd 288#include <TMutex.h>
9827400b 289#include <TSystemDirectory.h>
290#include <TSystemFile.h>
a986b218 291#include <TFile.h>
9827400b 292#include <TGrid.h>
293#include <TGridResult.h>
73abe331 294
e7f62f16 295#include <TMonaLisaWriter.h>
296
5164a766 297#include <fstream>
298
cb343cfd 299#include <sys/types.h>
300#include <sys/wait.h>
301
73abe331 302ClassImp(AliShuttle)
303
b948db8d 304//______________________________________________________________________________________________
305AliShuttle::AliShuttle(const AliShuttleConfig* config,
306 UInt_t timeout, Int_t retries):
4f0ab988 307fConfig(config),
308fTimeout(timeout), fRetries(retries),
309fPreprocessorMap(),
2bb7b766 310fLogbookEntry(0),
eba76848 311fCurrentDetector(),
85a80aa9 312fStatusEntry(0),
cb343cfd 313fMonitoringMutex(0),
eba76848 314fLastActionTime(0),
e7f62f16 315fLastAction(),
9827400b 316fMonaLisa(0),
317fTestMode(kNone),
ffa29e93 318fReadTestMode(kFALSE),
319fOutputRedirected(kFALSE)
73abe331 320{
321 //
322 // config: AliShuttleConfig used
73abe331 323 // timeout: timeout used for AliDCSClient connection
324 // retries: the number of retries in case of connection error.
325 //
326
57f50b3c 327 if (!fConfig->IsValid()) AliFatal("********** !!!!! Invalid configuration !!!!! **********");
be48e3ea 328 for(int iSys=0;iSys<4;iSys++) {
57f50b3c 329 fServer[iSys]=0;
be48e3ea 330 if (iSys < 3)
2c15234c 331 fFXSlist[iSys].SetOwner(kTRUE);
57f50b3c 332 }
2bb7b766 333 fPreprocessorMap.SetOwner(kTRUE);
be48e3ea 334
335 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
336 fFirstUnprocessed[iDet] = kFALSE;
337
cb343cfd 338 fMonitoringMutex = new TMutex();
58bc3020 339}
340
b948db8d 341//______________________________________________________________________________________________
57f50b3c 342AliShuttle::~AliShuttle()
58bc3020 343{
9827400b 344 //
345 // destructor
346 //
58bc3020 347
b948db8d 348 fPreprocessorMap.DeleteAll();
be48e3ea 349 for(int iSys=0;iSys<4;iSys++)
57f50b3c 350 if(fServer[iSys]) {
351 fServer[iSys]->Close();
352 delete fServer[iSys];
eba76848 353 fServer[iSys] = 0;
57f50b3c 354 }
2bb7b766 355
356 if (fStatusEntry){
357 delete fStatusEntry;
358 fStatusEntry = 0;
359 }
cb343cfd 360
361 if (fMonitoringMutex)
362 {
363 delete fMonitoringMutex;
364 fMonitoringMutex = 0;
365 }
73abe331 366}
367
b948db8d 368//______________________________________________________________________________________________
57f50b3c 369void AliShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
58bc3020 370{
73abe331 371 //
b948db8d 372 // Registers new AliPreprocessor.
73abe331 373 // It uses GetName() for indentificator of the pre processor.
374 // The pre processor is registered it there isn't any other
375 // with the same identificator (GetName()).
376 //
377
eba76848 378 const char* detName = preprocessor->GetName();
379 if(GetDetPos(detName) < 0)
380 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
381
382 if (fPreprocessorMap.GetValue(detName)) {
383 AliWarning(Form("AliPreprocessor %s is already registered!", detName));
73abe331 384 return;
385 }
386
eba76848 387 fPreprocessorMap.Add(new TObjString(detName), preprocessor);
73abe331 388}
b948db8d 389//______________________________________________________________________________________________
3301427a 390Bool_t AliShuttle::Store(const AliCDBPath& path, TObject* object,
84090f85 391 AliCDBMetaData* metaData, Int_t validityStart, Bool_t validityInfinite)
73abe331 392{
9827400b 393 // Stores a CDB object in the storage for offline reconstruction. Objects that are not needed for
394 // offline reconstruction, but should be stored anyway (e.g. for debugging) should NOT be stored
395 // using this function. Use StoreReferenceData instead!
396 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
397 // finishes the data are transferred to the main storage (Grid).
b948db8d 398
3301427a 399 return StoreLocally(fgkLocalCDB, path, object, metaData, validityStart, validityInfinite);
84090f85 400}
401
402//______________________________________________________________________________________________
3301427a 403Bool_t AliShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 404{
9827400b 405 // Stores a CDB object in the storage for reference data. This objects will not be available during
406 // offline reconstrunction. Use this function for reference data only!
407 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
408 // finishes the data are transferred to the main storage (Grid).
85a80aa9 409
3301427a 410 return StoreLocally(fgkLocalRefStorage, path, object, metaData);
85a80aa9 411}
412
413//______________________________________________________________________________________________
3301427a 414Bool_t AliShuttle::StoreLocally(const TString& localUri,
85a80aa9 415 const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
416 Int_t validityStart, Bool_t validityInfinite)
417{
9827400b 418 // Store object temporarily in local storage. Parameters are passed by Store and StoreReferenceData functions.
419 // when the preprocessor finishes the data are transferred to the main storage (Grid).
420 // The parameters are:
421 // 1) Uri of the backup storage (Local)
422 // 2) the object's path.
423 // 3) the object to be stored
424 // 4) the metaData to be associated with the object
425 // 5) the validity start run number w.r.t. the current run,
426 // if the data is valid only for this run leave the default 0
427 // 6) specifies if the calibration data is valid for infinity (this means until updated),
428 // typical for calibration runs, the default is kFALSE
429 //
430 // returns 0 if fail, 1 otherwise
84090f85 431
9827400b 432 if (fTestMode & kErrorStorage)
433 {
434 Log(fCurrentDetector, "StoreLocally - In TESTMODE - Simulating error while storing locally");
435 return kFALSE;
436 }
437
3301427a 438 const char* cdbType = (localUri == fgkLocalCDB) ? "CDB" : "Reference";
2bb7b766 439
85a80aa9 440 Int_t firstRun = GetCurrentRun() - validityStart;
84090f85 441 if(firstRun < 0) {
9827400b 442 AliWarning("First valid run happens to be less than 0! Setting it to 0.");
84090f85 443 firstRun=0;
444 }
445
446 Int_t lastRun = -1;
447 if(validityInfinite) {
448 lastRun = AliCDBRunRange::Infinity();
449 } else {
450 lastRun = GetCurrentRun();
451 }
452
3301427a 453 // Version is set to current run, it will be used later to transfer data to Grid
454 AliCDBId id(path, firstRun, lastRun, GetCurrentRun(), -1);
2bb7b766 455
456 if(! dynamic_cast<TObjString*> (metaData->GetProperty("RunUsed(TObjString)"))){
457 TObjString runUsed = Form("%d", GetCurrentRun());
9e080f92 458 metaData->SetProperty("RunUsed(TObjString)", runUsed.Clone());
2bb7b766 459 }
84090f85 460
3301427a 461 Bool_t result = kFALSE;
84090f85 462
3301427a 463 if (!(AliCDBManager::Instance()->GetStorage(localUri))) {
464 Log("SHUTTLE", Form("StoreLocally - Cannot activate local %s storage", cdbType));
84090f85 465 } else {
3301427a 466 result = AliCDBManager::Instance()->GetStorage(localUri)
84090f85 467 ->Put(object, id, metaData);
468 }
469
470 if(!result) {
471
9827400b 472 Log(fCurrentDetector, Form("StoreLocally - Can't store object <%s>!", id.ToString().Data()));
3301427a 473 }
2bb7b766 474
3301427a 475 return result;
476}
84090f85 477
3301427a 478//______________________________________________________________________________________________
479Bool_t AliShuttle::StoreOCDB()
480{
9827400b 481 //
482 // Called when preprocessor ends successfully or when previous storage attempt failed (kStoreError status)
483 // Calls underlying StoreOCDB(const char*) function twice, for OCDB and Reference storage.
484 // Then calls StoreRefFilesToGrid to store reference files.
485 //
486
487 if (fTestMode & kErrorGrid)
488 {
489 Log("SHUTTLE", "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
490 Log(fCurrentDetector, "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
491 return kFALSE;
492 }
493
2d9019b4 494 Log("SHUTTLE","Storing OCDB data ...");
86aa42c3 495 Bool_t resultCDB = StoreOCDB(fgkMainCDB);
496
2d9019b4 497 Log("SHUTTLE","Storing reference data ...");
3301427a 498 Bool_t resultRef = StoreOCDB(fgkMainRefStorage);
9827400b 499
2d9019b4 500 Log("SHUTTLE","Storing reference files ...");
9827400b 501 Bool_t resultRefFiles = StoreRefFilesToGrid();
502
503 return resultCDB && resultRef && resultRefFiles;
3301427a 504}
505
506//______________________________________________________________________________________________
507Bool_t AliShuttle::StoreOCDB(const TString& gridURI)
508{
509 //
510 // Called by StoreOCDB(), performs actual storage to the main OCDB and reference storages (Grid)
511 //
512
513 TObjArray* gridIds=0;
514
515 Bool_t result = kTRUE;
516
517 const char* type = 0;
518 TString localURI;
519 if(gridURI == fgkMainCDB) {
520 type = "OCDB";
521 localURI = fgkLocalCDB;
522 } else if(gridURI == fgkMainRefStorage) {
523 type = "reference";
524 localURI = fgkLocalRefStorage;
525 } else {
526 AliError(Form("Invalid storage URI: %s", gridURI.Data()));
527 return kFALSE;
528 }
529
530 AliCDBManager* man = AliCDBManager::Instance();
531
532 AliCDBStorage *gridSto = man->GetStorage(gridURI);
533 if(!gridSto) {
534 Log("SHUTTLE",
535 Form("StoreOCDB - cannot activate main %s storage", type));
536 return kFALSE;
537 }
538
539 gridIds = gridSto->GetQueryCDBList();
540
541 // get objects previously stored in local CDB
542 AliCDBStorage *localSto = man->GetStorage(localURI);
543 if(!localSto) {
544 Log("SHUTTLE",
545 Form("StoreOCDB - cannot activate local %s storage", type));
546 return kFALSE;
547 }
548 AliCDBPath aPath(GetOfflineDetName(fCurrentDetector.Data()),"*","*");
549 // Local objects were stored with current run as Grid version!
550 TList* localEntries = localSto->GetAll(aPath.GetPath(), GetCurrentRun(), GetCurrentRun());
551 localEntries->SetOwner(1);
552
553 // loop on local stored objects
554 TIter localIter(localEntries);
555 AliCDBEntry *aLocEntry = 0;
556 while((aLocEntry = dynamic_cast<AliCDBEntry*> (localIter.Next()))){
557 aLocEntry->SetOwner(1);
558 AliCDBId aLocId = aLocEntry->GetId();
559 aLocEntry->SetVersion(-1);
560 aLocEntry->SetSubVersion(-1);
561
562 // If local object is valid up to infinity we store it only if it is
563 // the first unprocessed run!
564 if (aLocId.GetLastRun() == AliCDBRunRange::Infinity() &&
565 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
566 {
567 Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
568 "there are previous unprocessed runs!",
569 fCurrentDetector.Data(), aLocId.GetPath().Data()));
570 continue;
571 }
572
573 // loop on Grid valid Id's
574 Bool_t store = kTRUE;
575 TIter gridIter(gridIds);
576 AliCDBId* aGridId = 0;
577 while((aGridId = dynamic_cast<AliCDBId*> (gridIter.Next()))){
578 if(aGridId->GetPath() != aLocId.GetPath()) continue;
579 // skip all objects valid up to infinity
580 if(aGridId->GetLastRun() == AliCDBRunRange::Infinity()) continue;
581 // if we get here, it means there's already some more recent object stored on Grid!
582 store = kFALSE;
583 break;
584 }
585
586 // If we get here, the file can be stored!
587 Bool_t storeOk = gridSto->Put(aLocEntry);
588 if(!store || storeOk){
589
590 if (!store)
591 {
592 Log(fCurrentDetector.Data(),
593 Form("StoreOCDB - A more recent object already exists in %s storage: <%s>",
594 type, aGridId->ToString().Data()));
595 } else {
596 Log("SHUTTLE",
597 Form("StoreOCDB - Object <%s> successfully put into %s storage",
598 aLocId.ToString().Data(), type));
2d9019b4 599 Log(fCurrentDetector.Data(),
600 Form("StoreOCDB - Object <%s> successfully put into %s storage",
601 aLocId.ToString().Data(), type));
3301427a 602 }
84090f85 603
3301427a 604 // removing local filename...
605 TString filename;
606 localSto->IdToFilename(aLocId, filename);
607 AliInfo(Form("Removing local file %s", filename.Data()));
608 RemoveFile(filename.Data());
609 continue;
610 } else {
611 Log("SHUTTLE",
612 Form("StoreOCDB - Grid %s storage of object <%s> failed",
613 type, aLocId.ToString().Data()));
2d9019b4 614 Log(fCurrentDetector.Data(),
615 Form("StoreOCDB - Grid %s storage of object <%s> failed",
616 type, aLocId.ToString().Data()));
3301427a 617 result = kFALSE;
b948db8d 618 }
619 }
3301427a 620 localEntries->Clear();
2bb7b766 621
b948db8d 622 return result;
3301427a 623}
624
546242fb 625//______________________________________________________________________________________________
626Bool_t AliShuttle::CleanReferenceStorage(const char* detector)
627{
2d9019b4 628 // clears the directory used to store reference files of a given subdetector
546242fb 629
630 AliCDBManager* man = AliCDBManager::Instance();
631 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
2d9019b4 632 TString localBaseFolder = sto->GetBaseFolder();
633
634 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
635
636 Log("SHUTTLE", Form("Cleaning %s", targetDir.Data()));
637
638 TString begin;
639 begin.Form("%d_", GetCurrentRun());
640
641 TSystemDirectory* baseDir = new TSystemDirectory("/", targetDir);
642 if (!baseDir)
643 return kTRUE;
644
645 TList* dirList = baseDir->GetListOfFiles();
646 delete baseDir;
647
648 if (!dirList) return kTRUE;
649
650 if (dirList->GetEntries() < 3)
651 {
652 delete dirList;
653 return kTRUE;
654 }
655
656 Int_t nDirs = 0, nDel = 0;
657 TIter dirIter(dirList);
658 TSystemFile* entry = 0;
546242fb 659
2d9019b4 660 Bool_t success = kTRUE;
546242fb 661
2d9019b4 662 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
663 {
664 if (entry->IsDirectory())
665 continue;
666
667 TString fileName(entry->GetName());
668 if (!fileName.BeginsWith(begin))
669 continue;
670
671 nDirs++;
672
673 // delete file
674 Int_t result = gSystem->Unlink(fileName.Data());
675
676 if (result)
677 {
678 Log("SHUTTLE", Form("Could not delete file %s!", fileName.Data()));
679 success = kFALSE;
680 } else {
681 nDel++;
682 }
683 }
684
685 if(nDirs > 0)
686 Log("SHUTTLE", Form("CleanReferenceStorage - %d (over %d) reference files in folder %s were deleted.",
687 nDel, nDirs, targetDir.Data()));
688
689
690 delete dirList;
691 return success;
692
693
694
695
696
546242fb 697
698 Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
699 if (result == 0)
700 {
701 // delete directory
702 result = gSystem->Exec(Form("rm -r %s", targetDir.Data()));
703 if (result != 0)
704 {
705 Log("SHUTTLE", Form("StoreReferenceFile - Could not clear directory %s", targetDir.Data()));
706 return kFALSE;
707 }
708 }
709
710 result = gSystem->mkdir(targetDir, kTRUE);
711 if (result != 0)
712 {
713 Log("SHUTTLE", Form("StoreReferenceFile - Error creating base directory %s", targetDir.Data()));
714 return kFALSE;
715 }
716
717 return kTRUE;
718}
719
9827400b 720//______________________________________________________________________________________________
721Bool_t AliShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
722{
723 //
3c2a21c8 724 // Stores reference file directly (without opening it). This function stores the file locally.
9827400b 725 //
3c2a21c8 726 // The file is stored under the following location:
727 // <base folder of local reference storage>/<DET>/<RUN#>_<gridFileName>
728 // where <gridFileName> is the second parameter given to the function
729 //
9827400b 730
731 if (fTestMode & kErrorStorage)
732 {
733 Log(fCurrentDetector, "StoreReferenceFile - In TESTMODE - Simulating error while storing locally");
734 return kFALSE;
735 }
736
737 AliCDBManager* man = AliCDBManager::Instance();
738 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
739
740 TString localBaseFolder = sto->GetBaseFolder();
741
2d9019b4 742 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
9827400b 743
2d9019b4 744 //try to open folder, if does not exist
745 void* dir = gSystem->OpenDirectory(targetDir.Data());
746 if (dir == NULL) {
747 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
748 Log("SHUTTLE", Form("Can't open directory <%s>", targetDir.Data()));
749 return kFALSE;
750 }
751
752 } else {
753 gSystem->FreeDirectory(dir);
754 }
755
9827400b 756 TString target;
757 target.Form("%s/%d_%s", targetDir.Data(), GetCurrentRun(), gridFileName);
758
546242fb 759 Int_t result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
9827400b 760 if (result)
761 {
546242fb 762 Log("SHUTTLE", Form("StoreReferenceFile - %s does not exist", localFile));
763 return kFALSE;
9827400b 764 }
546242fb 765
9827400b 766 result = gSystem->CopyFile(localFile, target);
767
768 if (result == 0)
769 {
2d9019b4 770 Log("SHUTTLE", Form("StoreReferenceFile - File %s stored locally to %s", localFile, target.Data()));
9827400b 771 return kTRUE;
772 }
773 else
774 {
2d9019b4 775 Log("SHUTTLE", Form("StoreReferenceFile - Could not store file %s to %s!. Error code = %d",
546242fb 776 localFile, target.Data(), result));
9827400b 777 return kFALSE;
778 }
779}
780
781//______________________________________________________________________________________________
782Bool_t AliShuttle::StoreRefFilesToGrid()
783{
784 //
785 // Transfers the reference file to the Grid.
9827400b 786 //
86aa42c3 787 // The files are stored under the following location:
3c2a21c8 788 // <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
86aa42c3 789 //
9827400b 790
791 AliCDBManager* man = AliCDBManager::Instance();
792 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
793 if (!sto)
794 return kFALSE;
795 TString localBaseFolder = sto->GetBaseFolder();
796
2d9019b4 797 TString dir = GetRefFilePrefix(localBaseFolder.Data(), fCurrentDetector.Data());
798
9827400b 799 AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
800 if (!gridSto)
801 return kFALSE;
2d9019b4 802
9827400b 803 TString gridBaseFolder = gridSto->GetBaseFolder();
2d9019b4 804
805 TString alienDir = GetRefFilePrefix(gridBaseFolder.Data(), fCurrentDetector.Data());
9827400b 806
9827400b 807 TString begin;
808 begin.Form("%d_", GetCurrentRun());
809
810 TSystemDirectory* baseDir = new TSystemDirectory("/", dir);
3d8bc902 811 if (!baseDir)
812 return kTRUE;
813
2d9019b4 814 TList* dirList = baseDir->GetListOfFiles();
815 delete baseDir;
816
817 if (!dirList) return kTRUE;
818
819 if (dirList->GetEntries() < 3)
3d8bc902 820 {
2d9019b4 821 delete dirList;
9827400b 822 return kTRUE;
3d8bc902 823 }
2d9019b4 824
546242fb 825 if (!gGrid)
826 {
827 Log("SHUTTLE", "Connection to Grid failed: Cannot continue!");
2d9019b4 828 delete dirList;
546242fb 829 return kFALSE;
830 }
831
2d9019b4 832 Int_t nDirs = 0, nTransfer = 0;
833 TIter dirIter(dirList);
834 TSystemFile* entry = 0;
835
9827400b 836 Bool_t success = kTRUE;
3d8bc902 837 Bool_t first = kTRUE;
9827400b 838
2d9019b4 839 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
840 {
9827400b 841 if (entry->IsDirectory())
842 continue;
843
844 TString fileName(entry->GetName());
845 if (!fileName.BeginsWith(begin))
846 continue;
847
2d9019b4 848 nDirs++;
849
3d8bc902 850 if (first)
851 {
852 first = kFALSE;
853 // check that DET folder exists, otherwise create it
854 TGridResult* result = gGrid->Ls(alienDir.Data(), "a");
855
856 if (!result)
2d9019b4 857 {
858 delete dirList;
3d8bc902 859 return kFALSE;
2d9019b4 860 }
3d8bc902 861
546242fb 862 if (!result->GetFileName(1)) // TODO: It looks like element 0 is always 0!!
3d8bc902 863 {
864 if (!gGrid->Mkdir(alienDir.Data(),"",0))
865 {
866 Log("SHUTTLE", Form("StoreRefFilesToGrid - Cannot create directory %s",
867 alienDir.Data()));
2d9019b4 868 delete dirList;
3d8bc902 869 return kFALSE;
546242fb 870 } else {
871 Log("SHUTTLE",Form("Folder %s created", alienDir.Data()));
3d8bc902 872 }
873
546242fb 874 } else {
875 Log("SHUTTLE",Form("Folder %s found", alienDir.Data()));
3d8bc902 876 }
877 }
878
9827400b 879 TString fullLocalPath;
880 fullLocalPath.Form("%s/%s", dir.Data(), fileName.Data());
881
882 TString fullGridPath;
883 fullGridPath.Form("alien://%s/%s", alienDir.Data(), fileName.Data());
884
a986b218 885 Bool_t result = TFile::Cp(fullLocalPath, fullGridPath);
9827400b 886
887 if (result)
888 {
2d9019b4 889 Log("SHUTTLE", Form("StoreRefFilesToGrid - Copying local file %s to %s succeeded!", fullLocalPath.Data(), fullGridPath.Data()));
9827400b 890 RemoveFile(fullLocalPath);
2d9019b4 891 nTransfer++;
9827400b 892 }
893 else
894 {
2d9019b4 895 Log("SHUTTLE", Form("StoreRefFilesToGrid - Copying local file %s to %s FAILED!", fullLocalPath.Data(), fullGridPath.Data()));
9827400b 896 success = kFALSE;
897 }
898 }
2d9019b4 899
900 Log("SHUTTLE", Form("StoreRefFilesToGrid - %d (over %d) reference files in folder %s copied to Grid.", nTransfer, nDirs, dir.Data()));
901
902
903 delete dirList;
9827400b 904 return success;
905}
906
2d9019b4 907//______________________________________________________________________________________________
908const char* AliShuttle::GetRefFilePrefix(const char* base, const char* detector)
909{
910 //
911 // Get folder name of reference files
912 //
913
914 TString offDetStr(GetOfflineDetName(detector));
915 TString dir;
916 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
917 {
918 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
919 } else {
920 dir.Form("%s/%s", base, offDetStr.Data());
921 }
922
923 return dir.Data();
924
925
926}
3301427a 927//______________________________________________________________________________________________
928void AliShuttle::CleanLocalStorage(const TString& uri)
929{
9827400b 930 //
931 // Called in case the preprocessor is declared failed. Remove remaining objects from the local storages.
932 //
3301427a 933
934 const char* type = 0;
935 if(uri == fgkLocalCDB) {
936 type = "OCDB";
937 } else if(uri == fgkLocalRefStorage) {
546242fb 938 type = "Reference";
3301427a 939 } else {
940 AliError(Form("Invalid storage URI: %s", uri.Data()));
941 return;
942 }
943
944 AliCDBManager* man = AliCDBManager::Instance();
b948db8d 945
3301427a 946 // open local storage
947 AliCDBStorage *localSto = man->GetStorage(uri);
948 if(!localSto) {
949 Log("SHUTTLE",
950 Form("CleanLocalStorage - cannot activate local %s storage", type));
951 return;
952 }
953
954 TString filename(Form("%s/%s/*/Run*_v%d_s*.root",
546242fb 955 localSto->GetBaseFolder().Data(), GetOfflineDetName(fCurrentDetector.Data()), GetCurrentRun()));
3301427a 956
957 AliInfo(Form("filename = %s", filename.Data()));
958
959 AliInfo(Form("Removing remaining local files from run %d and detector %s ...",
960 GetCurrentRun(), fCurrentDetector.Data()));
961
962 RemoveFile(filename.Data());
963
964}
965
966//______________________________________________________________________________________________
967void AliShuttle::RemoveFile(const char* filename)
968{
9827400b 969 //
970 // removes local file
971 //
3301427a 972
973 TString command(Form("rm -f %s", filename));
974
975 Int_t result = gSystem->Exec(command.Data());
976 if(result != 0)
977 {
978 Log("SHUTTLE", Form("RemoveFile - %s: Cannot remove file %s!",
979 fCurrentDetector.Data(), filename));
980 }
73abe331 981}
982
b948db8d 983//______________________________________________________________________________________________
5164a766 984AliShuttleStatus* AliShuttle::ReadShuttleStatus()
985{
9827400b 986 //
987 // Reads the AliShuttleStatus from the CDB
988 //
5164a766 989
2bb7b766 990 if (fStatusEntry){
991 delete fStatusEntry;
992 fStatusEntry = 0;
993 }
5164a766 994
10a5a932 995 fStatusEntry = AliCDBManager::Instance()->GetStorage(GetLocalCDB())
2bb7b766 996 ->Get(Form("/SHUTTLE/STATUS/%s", fCurrentDetector.Data()), GetCurrentRun());
5164a766 997
2bb7b766 998 if (!fStatusEntry) return 0;
999 fStatusEntry->SetOwner(1);
5164a766 1000
2bb7b766 1001 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1002 if (!status) {
1003 AliError("Invalid object stored to CDB!");
1004 return 0;
1005 }
5164a766 1006
2bb7b766 1007 return status;
5164a766 1008}
1009
1010//______________________________________________________________________________________________
7bfb2090 1011Bool_t AliShuttle::WriteShuttleStatus(AliShuttleStatus* status)
5164a766 1012{
9827400b 1013 //
1014 // writes the status for one subdetector
1015 //
2bb7b766 1016
1017 if (fStatusEntry){
1018 delete fStatusEntry;
1019 fStatusEntry = 0;
1020 }
5164a766 1021
2bb7b766 1022 Int_t run = GetCurrentRun();
5164a766 1023
2bb7b766 1024 AliCDBId id(AliCDBPath("SHUTTLE", "STATUS", fCurrentDetector), run, run);
5164a766 1025
2bb7b766 1026 fStatusEntry = new AliCDBEntry(status, id, new AliCDBMetaData);
1027 fStatusEntry->SetOwner(1);
5164a766 1028
2bb7b766 1029 UInt_t result = AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
7bfb2090 1030
2bb7b766 1031 if (!result) {
3301427a 1032 Log("SHUTTLE", Form("WriteShuttleStatus - Failed for %s, run %d",
1033 fCurrentDetector.Data(), run));
2bb7b766 1034 return kFALSE;
1035 }
e7f62f16 1036
1037 SendMLInfo();
7bfb2090 1038
2bb7b766 1039 return kTRUE;
5164a766 1040}
1041
1042//______________________________________________________________________________________________
1043void AliShuttle::UpdateShuttleStatus(AliShuttleStatus::Status newStatus, Bool_t increaseCount)
1044{
9827400b 1045 //
1046 // changes the AliShuttleStatus for the given detector and run to the given status
1047 //
5164a766 1048
2bb7b766 1049 if (!fStatusEntry){
1050 AliError("UNEXPECTED: fStatusEntry empty");
1051 return;
1052 }
5164a766 1053
2bb7b766 1054 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
5164a766 1055
2bb7b766 1056 if (!status){
3301427a 1057 Log("SHUTTLE", "UNEXPECTED: status could not be read from current CDB entry");
2bb7b766 1058 return;
1059 }
5164a766 1060
2c15234c 1061 TString actionStr = Form("UpdateShuttleStatus - %s: Changing state from %s to %s",
eba76848 1062 fCurrentDetector.Data(),
36c99a6a 1063 status->GetStatusName(),
eba76848 1064 status->GetStatusName(newStatus));
cb343cfd 1065 Log("SHUTTLE", actionStr);
1066 SetLastAction(actionStr);
5164a766 1067
2bb7b766 1068 status->SetStatus(newStatus);
1069 if (increaseCount) status->IncreaseCount();
5164a766 1070
2bb7b766 1071 AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
e7f62f16 1072
1073 SendMLInfo();
5164a766 1074}
e7f62f16 1075
1076//______________________________________________________________________________________________
1077void AliShuttle::SendMLInfo()
1078{
1079 //
1080 // sends ML information about the current status of the current detector being processed
1081 //
1082
1083 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1084
1085 if (!status){
3301427a 1086 Log("SHUTTLE", "SendMLInfo - UNEXPECTED: status could not be read from current CDB entry");
e7f62f16 1087 return;
1088 }
1089
1090 TMonaLisaText mlStatus(Form("%s_status", fCurrentDetector.Data()), status->GetStatusName());
1091 TMonaLisaValue mlRetryCount(Form("%s_count", fCurrentDetector.Data()), status->GetCount());
1092
1093 TList mlList;
1094 mlList.Add(&mlStatus);
1095 mlList.Add(&mlRetryCount);
1096
1097 fMonaLisa->SendParameters(&mlList);
1098}
1099
5164a766 1100//______________________________________________________________________________________________
1101Bool_t AliShuttle::ContinueProcessing()
1102{
9827400b 1103 // this function reads the AliShuttleStatus information from CDB and
1104 // checks if the processing should be continued
1105 // if yes it returns kTRUE and updates the AliShuttleStatus with nextStatus
2bb7b766 1106
57c1a579 1107 if (!fConfig->HostProcessDetector(fCurrentDetector)) return kFALSE;
1108
1109 AliPreprocessor* aPreprocessor =
1110 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1111 if (!aPreprocessor)
1112 {
1113 AliInfo(Form("%s: no preprocessor registered", fCurrentDetector.Data()));
1114 return kFALSE;
1115 }
1116
2bb7b766 1117 AliShuttleLogbookEntry::Status entryStatus =
eba76848 1118 fLogbookEntry->GetDetectorStatus(fCurrentDetector);
2bb7b766 1119
1120 if(entryStatus != AliShuttleLogbookEntry::kUnprocessed) {
9e080f92 1121 AliInfo(Form("ContinueProcessing - %s is %s",
2bb7b766 1122 fCurrentDetector.Data(),
1123 fLogbookEntry->GetDetectorStatusName(entryStatus)));
1124 return kFALSE;
1125 }
1126
1127 // if we get here, according to Shuttle logbook subdetector is in UNPROCESSED state
be48e3ea 1128
1129 // check if current run is first unprocessed run for current detector
1130 if (fConfig->StrictRunOrder(fCurrentDetector) &&
1131 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
1132 {
86aa42c3 1133 if (fTestMode == kNone)
1134 {
1135 Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering but this is not the first unprocessed run!"));
1136 return kFALSE;
1137 }
1138 else
1139 {
1140 Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - Although %s requires strict run ordering and this is not the first unprocessed run, the SHUTTLE continues"));
1141 }
be48e3ea 1142 }
1143
2bb7b766 1144 AliShuttleStatus* status = ReadShuttleStatus();
1145 if (!status) {
1146 // first time
1147 Log("SHUTTLE", Form("ContinueProcessing - %s: Processing first time",
1148 fCurrentDetector.Data()));
1149 status = new AliShuttleStatus(AliShuttleStatus::kStarted);
1150 return WriteShuttleStatus(status);
1151 }
1152
1153 // The following two cases shouldn't happen if Shuttle Logbook was correctly updated.
1154 // If it happens it may mean Logbook updating failed... let's do it now!
1155 if (status->GetStatus() == AliShuttleStatus::kDone ||
1156 status->GetStatus() == AliShuttleStatus::kFailed){
1157 Log("SHUTTLE", Form("ContinueProcessing - %s is already %s. Updating Shuttle Logbook",
1158 fCurrentDetector.Data(),
1159 status->GetStatusName(status->GetStatus())));
1160 UpdateShuttleLogbook(fCurrentDetector.Data(),
1161 status->GetStatusName(status->GetStatus()));
1162 return kFALSE;
1163 }
1164
3301427a 1165 if (status->GetStatus() == AliShuttleStatus::kStoreError) {
2bb7b766 1166 Log("SHUTTLE",
1167 Form("ContinueProcessing - %s: Grid storage of one or more objects failed. Trying again now",
1168 fCurrentDetector.Data()));
9827400b 1169 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1170 if (StoreOCDB()){
3301427a 1171 Log("SHUTTLE", Form("ContinueProcessing - %s: all objects successfully stored into main storage",
1172 fCurrentDetector.Data()));
2bb7b766 1173 UpdateShuttleStatus(AliShuttleStatus::kDone);
1174 UpdateShuttleLogbook(fCurrentDetector.Data(), "DONE");
1175 } else {
1176 Log("SHUTTLE",
1177 Form("ContinueProcessing - %s: Grid storage failed again",
1178 fCurrentDetector.Data()));
9827400b 1179 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
2bb7b766 1180 }
1181 return kFALSE;
1182 }
1183
1184 // if we get here, there is a restart
57c1a579 1185 Bool_t cont = kFALSE;
2bb7b766 1186
1187 // abort conditions
cb343cfd 1188 if (status->GetCount() >= fConfig->GetMaxRetries()) {
57c1a579 1189 Log("SHUTTLE", Form("ContinueProcessing - %s failed %d times in status %s - "
1190 "Updating Shuttle Logbook", fCurrentDetector.Data(),
2bb7b766 1191 status->GetCount(), status->GetStatusName()));
1192 UpdateShuttleLogbook(fCurrentDetector.Data(), "FAILED");
e7f62f16 1193 UpdateShuttleStatus(AliShuttleStatus::kFailed);
3301427a 1194
1195 // there may still be objects in local OCDB and reference storage
1196 // and FXS databases may be not updated: do it now!
9827400b 1197
1198 // TODO Currently disabled, we want to keep files in case of failure!
1199 // CleanLocalStorage(fgkLocalCDB);
1200 // CleanLocalStorage(fgkLocalRefStorage);
1201 // UpdateTableFailCase();
1202
1203 // Send mail to detector expert!
1204 AliInfo(Form("Sending mail to %s expert...", fCurrentDetector.Data()));
1205 if (!SendMail())
1206 Log("SHUTTLE", Form("ContinueProcessing - Could not send mail to %s expert",
1207 fCurrentDetector.Data()));
3301427a 1208
57c1a579 1209 } else {
1210 Log("SHUTTLE", Form("ContinueProcessing - %s: restarting. "
1211 "Aborted before with %s. Retry number %d.", fCurrentDetector.Data(),
1212 status->GetStatusName(), status->GetCount()));
9827400b 1213 Bool_t increaseCount = kTRUE;
1214 if (status->GetStatus() == AliShuttleStatus::kDCSError || status->GetStatus() == AliShuttleStatus::kDCSStarted)
1215 increaseCount = kFALSE;
1216 UpdateShuttleStatus(AliShuttleStatus::kStarted, increaseCount);
57c1a579 1217 cont = kTRUE;
2bb7b766 1218 }
1219
57c1a579 1220 return cont;
5164a766 1221}
1222
1223//______________________________________________________________________________________________
2bb7b766 1224Bool_t AliShuttle::Process(AliShuttleLogbookEntry* entry)
58bc3020 1225{
73abe331 1226 //
b948db8d 1227 // Makes data retrieval for all detectors in the configuration.
2bb7b766 1228 // entry: Shuttle logbook entry, contains run paramenters and status of detectors
1229 // (Unprocessed, Inactive, Failed or Done).
d477ad88 1230 // Returns kFALSE in case of error occured and kTRUE otherwise
73abe331 1231 //
1232
9827400b 1233 if (!entry) return kFALSE;
2bb7b766 1234
1235 fLogbookEntry = entry;
1236
9827400b 1237 AliInfo(Form("\n\n \t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^* \n",
1238 GetCurrentRun()));
2bb7b766 1239
e7f62f16 1240 // create ML instance that monitors this run
1241 fMonaLisa = new TMonaLisaWriter(Form("%d", GetCurrentRun()), "SHUTTLE", "aliendb1.cern.ch");
1242 // disable monitoring of other parameters that come e.g. from TFile
1243 gMonitoringWriter = 0;
2bb7b766 1244
e7f62f16 1245 // Send the information to ML
1246 TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
9827400b 1247 TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
e7f62f16 1248
1249 TList mlList;
1250 mlList.Add(&mlStatus);
9827400b 1251 mlList.Add(&mlRunType);
e7f62f16 1252
1253 fMonaLisa->SendParameters(&mlList);
3301427a 1254
9827400b 1255 if (fLogbookEntry->IsDone())
1256 {
1257 Log("SHUTTLE","Process - Shuttle is already DONE. Updating logbook");
1258 UpdateShuttleLogbook("shuttle_done");
1259 fLogbookEntry = 0;
1260 return kTRUE;
1261 }
1262
1263 // read test mode if flag is set
1264 if (fReadTestMode)
1265 {
3d8bc902 1266 fTestMode = kNone;
9827400b 1267 TString logEntry(entry->GetRunParameter("log"));
1268 //printf("log entry = %s\n", logEntry.Data());
1269 TString searchStr("Testmode: ");
1270 Int_t pos = logEntry.Index(searchStr.Data());
1271 //printf("%d\n", pos);
1272 if (pos >= 0)
1273 {
1274 TSubString subStr = logEntry(pos + searchStr.Length(), logEntry.Length());
1275 //printf("%s\n", subStr.String().Data());
1276 TString newStr(subStr.Data());
1277 TObjArray* token = newStr.Tokenize(' ');
1278 if (token)
1279 {
1280 //token->Print();
1281 TObjString* tmpStr = dynamic_cast<TObjString*> (token->First());
1282 if (tmpStr)
1283 {
1284 Int_t testMode = tmpStr->String().Atoi();
1285 if (testMode > 0)
1286 {
1287 Log("SHUTTLE", Form("Enabling test mode %d", testMode));
1288 SetTestMode((TestMode) testMode);
1289 }
1290 }
1291 delete token;
1292 }
1293 }
1294 }
1295
3d8bc902 1296 Log("SHUTTLE", Form("The test mode flag is %d", (Int_t) fTestMode));
1297
eba76848 1298 fLogbookEntry->Print("all");
57f50b3c 1299
1300 // Initialization
d477ad88 1301 Bool_t hasError = kFALSE;
5164a766 1302
2bb7b766 1303 AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
1304 if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
1305 AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
1306 if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
d477ad88 1307
57f50b3c 1308 // Loop on detectors in the configuration
b948db8d 1309 TIter iter(fConfig->GetDetectors());
2bb7b766 1310 TObjString* aDetector = 0;
b948db8d 1311
be48e3ea 1312 while ((aDetector = (TObjString*) iter.Next()))
1313 {
7bfb2090 1314 fCurrentDetector = aDetector->String();
5164a766 1315
9e080f92 1316 if (ContinueProcessing() == kFALSE) continue;
1317
2bb7b766 1318 AliInfo(Form("\n\n \t\t\t****** run %d - %s: START ******",
1319 GetCurrentRun(), aDetector->GetName()));
1320
9d733021 1321 for(Int_t iSys=0;iSys<3;iSys++) fFXSCalled[iSys]=kFALSE;
1322
e7f62f16 1323 Log(fCurrentDetector.Data(), "Starting processing");
85a80aa9 1324
be48e3ea 1325 Int_t pid = fork();
1326
1327 if (pid < 0)
1328 {
1329 Log("SHUTTLE", "ERROR: Forking failed");
1330 }
1331 else if (pid > 0)
1332 {
1333 // parent
1334 AliInfo(Form("In parent process of %d - %s: Starting monitoring",
1335 GetCurrentRun(), aDetector->GetName()));
1336
1337 Long_t begin = time(0);
1338
1339 int status; // to be used with waitpid, on purpose an int (not Int_t)!
1340 while (waitpid(pid, &status, WNOHANG) == 0)
1341 {
1342 Long_t expiredTime = time(0) - begin;
1343
1344 if (expiredTime > fConfig->GetPPTimeOut())
1345 {
9827400b 1346 TString tmp;
1347 tmp.Form("Process of %s time out. Run time: %d seconds. Killing...",
1348 fCurrentDetector.Data(), expiredTime);
1349 Log("SHUTTLE", tmp);
1350 Log(fCurrentDetector, tmp);
be48e3ea 1351
1352 kill(pid, 9);
1353
3301427a 1354 UpdateShuttleStatus(AliShuttleStatus::kPPTimeOut);
be48e3ea 1355 hasError = kTRUE;
1356
1357 gSystem->Sleep(1000);
1358 }
1359 else
1360 {
be48e3ea 1361 gSystem->Sleep(1000);
9827400b 1362
1363 TString checkStr;
1364 checkStr.Form("ps -o vsize --pid %d | tail -n 1", pid);
1365 FILE* pipe = gSystem->OpenPipe(checkStr, "r");
1366 if (!pipe)
1367 {
1368 Log("SHUTTLE", Form("Error: Could not open pipe to %s", checkStr.Data()));
1369 continue;
1370 }
1371
1372 char buffer[100];
1373 if (!fgets(buffer, 100, pipe))
1374 {
1375 Log("SHUTTLE", "Error: ps did not return anything");
1376 gSystem->ClosePipe(pipe);
1377 continue;
1378 }
1379 gSystem->ClosePipe(pipe);
1380
1381 //Log("SHUTTLE", Form("ps returned %s", buffer));
1382
1383 Int_t mem = 0;
1384 if ((sscanf(buffer, "%d\n", &mem) != 1) || !mem)
1385 {
1386 Log("SHUTTLE", "Error: Could not parse output of ps");
1387 continue;
1388 }
1389
1390 if (expiredTime % 60 == 0)
886d60e6 1391 Log("SHUTTLE", Form("%s: Checking process. Run time: %d seconds - Memory consumption: %d KB",
1392 fCurrentDetector.Data(), expiredTime, mem));
9827400b 1393
1394 if (mem > fConfig->GetPPMaxMem())
1395 {
1396 TString tmp;
1397 tmp.Form("Process exceeds maximum allowed memory (%d KB > %d KB). Killing...",
1398 mem, fConfig->GetPPMaxMem());
1399 Log("SHUTTLE", tmp);
1400 Log(fCurrentDetector, tmp);
1401
1402 kill(pid, 9);
1403
1404 UpdateShuttleStatus(AliShuttleStatus::kPPOutOfMemory);
1405 hasError = kTRUE;
1406
1407 gSystem->Sleep(1000);
1408 }
be48e3ea 1409 }
1410 }
1411
1412 AliInfo(Form("In parent process of %d - %s: Client has terminated.",
1413 GetCurrentRun(), aDetector->GetName()));
1414
1415 if (WIFEXITED(status))
1416 {
1417 Int_t returnCode = WEXITSTATUS(status);
1418
3301427a 1419 Log("SHUTTLE", Form("%s: the return code is %d", fCurrentDetector.Data(),
1420 returnCode));
be48e3ea 1421
9827400b 1422 if (returnCode == 0) hasError = kTRUE;
be48e3ea 1423 }
1424 }
1425 else if (pid == 0)
1426 {
1427 // client
1428 AliInfo(Form("In client process of %d - %s", GetCurrentRun(), aDetector->GetName()));
1429
ffa29e93 1430 AliInfo("Redirecting output...");
1431
546242fb 1432 if ((freopen(GetLogFileName(fCurrentDetector), "a", stdout)) == 0)
ffa29e93 1433 {
1434 Log("SHUTTLE", "Could not freopen stdout");
1435 }
1436 else
1437 {
1438 fOutputRedirected = kTRUE;
1439 if ((dup2(fileno(stdout), fileno(stderr))) < 0)
1440 Log("SHUTTLE", "Could not redirect stderr");
1441
1442 }
1443
5bac2bde 1444 TString wd = gSystem->WorkingDirectory();
1445 TString tmpDir = Form("%s/%s_process",GetShuttleTempDir(),fCurrentDetector.Data());
1446
1447 gSystem->mkdir(tmpDir.Data());
1448 gSystem->ChangeDirectory(tmpDir.Data());
1449
9827400b 1450 Bool_t success = ProcessCurrentDetector();
5bac2bde 1451
1452 gSystem->ChangeDirectory(wd.Data());
1453
1454 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
1455
9827400b 1456 if (success) // Preprocessor finished successfully!
1457 {
3301427a 1458 // Update time_processed field in FXS DB
1459 if (UpdateTable() == kFALSE)
5bac2bde 1460 Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!",
1461 fCurrentDetector.Data()));
3301427a 1462
1463 // Transfer the data from local storage to main storage (Grid)
1464 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1465 if (StoreOCDB() == kFALSE)
1466 {
1467 AliInfo(Form("\n \t\t\t****** run %d - %s: STORAGE ERROR ****** \n\n",
1468 GetCurrentRun(), aDetector->GetName()));
1469 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
9827400b 1470 success = kFALSE;
3301427a 1471 } else {
1472 AliInfo(Form("\n \t\t\t****** run %d - %s: DONE ****** \n\n",
1473 GetCurrentRun(), aDetector->GetName()));
1474 UpdateShuttleStatus(AliShuttleStatus::kDone);
9827400b 1475 UpdateShuttleLogbook(fCurrentDetector, "DONE");
3301427a 1476 }
be48e3ea 1477 }
1478
4b95672b 1479 for (UInt_t iSys=0; iSys<3; iSys++)
1480 {
1481 if (fFXSCalled[iSys]) fFXSlist[iSys].Clear();
1482 }
1483
be48e3ea 1484 AliInfo(Form("Client process of %d - %s is exiting now with %d.",
9827400b 1485 GetCurrentRun(), aDetector->GetName(), success));
be48e3ea 1486
1487 // the client exits here
9827400b 1488 gSystem->Exit(success);
be48e3ea 1489
1490 AliError("We should never get here!!!");
1491 }
7bfb2090 1492 }
5164a766 1493
2bb7b766 1494 AliInfo(Form("\n\n \t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^* \n",
1495 GetCurrentRun()));
1496
1497 //check if shuttle is done for this run, if so update logbook
1498 TObjArray checkEntryArray;
1499 checkEntryArray.SetOwner(1);
9e080f92 1500 TString whereClause = Form("where run=%d", GetCurrentRun());
1501 if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) || checkEntryArray.GetEntries() == 0) {
1502 Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
1503 GetCurrentRun()));
1504 return hasError == kFALSE;
1505 }
b948db8d 1506
9e080f92 1507 AliShuttleLogbookEntry* checkEntry = dynamic_cast<AliShuttleLogbookEntry*>
1508 (checkEntryArray.At(0));
2bb7b766 1509
9e080f92 1510 if (checkEntry)
1511 {
1512 if (checkEntry->IsDone())
be48e3ea 1513 {
9e080f92 1514 Log("SHUTTLE","Process - Shuttle is DONE. Updating logbook");
1515 UpdateShuttleLogbook("shuttle_done");
1516 }
1517 else
1518 {
1519 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
be48e3ea 1520 {
9e080f92 1521 if (checkEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
be48e3ea 1522 {
9e080f92 1523 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
1524 checkEntry->GetRun(), GetDetName(iDet)));
1525 fFirstUnprocessed[iDet] = kFALSE;
be48e3ea 1526 }
1527 }
2bb7b766 1528 }
1529 }
1530
e7f62f16 1531 // remove ML instance
1532 delete fMonaLisa;
1533 fMonaLisa = 0;
1534
2bb7b766 1535 fLogbookEntry = 0;
85a80aa9 1536
a7160fe9 1537 return hasError == kFALSE;
73abe331 1538}
1539
b948db8d 1540//______________________________________________________________________________________________
9827400b 1541Bool_t AliShuttle::ProcessCurrentDetector()
73abe331 1542{
1543 //
2bb7b766 1544 // Makes data retrieval just for a specific detector (fCurrentDetector).
73abe331 1545 // Threre should be a configuration for this detector.
73abe331 1546
1d172743 1547 Log("SHUTTLE", Form("ProcessCurrentDetector - Retrieving values for %s, run %d",
1548 fCurrentDetector.Data(), GetCurrentRun()));
73abe331 1549
2d9019b4 1550 if (!CleanReferenceStorage(fCurrentDetector.Data()))
546242fb 1551 return kFALSE;
1552
1d172743 1553 TMap* dcsMap = new TMap();
3301427a 1554
1555 // call preprocessor
1556 AliPreprocessor* aPreprocessor =
1557 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1558
1559 aPreprocessor->Initialize(GetCurrentRun(), GetCurrentStartTime(), GetCurrentEndTime());
1560
1561 Bool_t processDCS = aPreprocessor->ProcessDCS();
d477ad88 1562
651fdaab 1563 if (!processDCS)
1564 {
1d172743 1565 Log(fCurrentDetector, "ProcessCurrentDetector -"
1566 " The preprocessor requested to skip the retrieval of DCS values");
651fdaab 1567 }
8b739301 1568 else if (fTestMode & kSkipDCS)
2c15234c 1569 {
1d172743 1570 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Skipping DCS processing");
9827400b 1571 }
1572 else if (fTestMode & kErrorDCS)
1573 {
1d172743 1574 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Simulating DCS error");
3d8bc902 1575 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
9827400b 1576 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1d172743 1577 delete dcsMap;
9827400b 1578 return kFALSE;
2c15234c 1579 } else {
3301427a 1580
1581 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
1582
1d172743 1583 // Query DCS archive
1584 Int_t nServers = fConfig->GetNServers(fCurrentDetector);
a038aa70 1585
1d172743 1586 for (int iServ=0; iServ<nServers; iServ++)
2c15234c 1587 {
1d172743 1588
1589 TString host(fConfig->GetDCSHost(fCurrentDetector, iServ));
1590 Int_t port = fConfig->GetDCSPort(fCurrentDetector, iServ);
542b6cc8 1591 Int_t multiSplit = fConfig->GetMultiSplit(fCurrentDetector, iServ);
1592
1790d4b7 1593 Log(fCurrentDetector, Form("ProcessCurrentDetector -"
1594 " Querying DCS Amanda server %s:%d (%d of %d)",
1595 host.Data(), port, iServ+1, nServers));
1d172743 1596
1597 TMap* aliasMap = 0;
1598 TMap* dpMap = 0;
1599
1600 if (fConfig->GetDCSAliases(fCurrentDetector, iServ)->GetEntries() > 0)
2c15234c 1601 {
1d172743 1602 aliasMap = GetValueSet(host, port,
542b6cc8 1603 fConfig->GetDCSAliases(fCurrentDetector, iServ),
1604 kAlias, multiSplit);
1d172743 1605 if (!aliasMap)
1606 {
1607 Log(fCurrentDetector,
1608 Form("ProcessCurrentDetector -"
1609 " Error retrieving DCS aliases from server %s",
1610 host.Data()));
1611 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
a038aa70 1612 delete dcsMap;
1d172743 1613 return kFALSE;
1614 }
2c15234c 1615 }
a038aa70 1616
1d172743 1617 if (fConfig->GetDCSDataPoints(fCurrentDetector, iServ)->GetEntries() > 0)
a038aa70 1618 {
1d172743 1619 dpMap = GetValueSet(host, port,
542b6cc8 1620 fConfig->GetDCSDataPoints(fCurrentDetector, iServ),
1621 kDP, multiSplit);
1d172743 1622 if (!dpMap)
1623 {
1624 Log(fCurrentDetector,
1625 Form("ProcessCurrentDetector -"
1626 " Error retrieving DCS data points from server %s",
1627 host.Data()));
1628 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1629 if (aliasMap) delete aliasMap;
1630 delete dcsMap;
1631 return kFALSE;
1632 }
a038aa70 1633 }
1d172743 1634
1635 // merge aliasMap and dpMap into dcsMap
1636 if(aliasMap) {
1637 TIter iter(aliasMap);
a038aa70 1638 TObjString* key = 0;
1639 while ((key = (TObjString*) iter.Next()))
1d172743 1640 dcsMap->Add(key, aliasMap->GetValue(key->String()));
1641
1642 aliasMap->SetOwner(kFALSE);
1643 delete aliasMap;
1644 }
1645
1646 if(dpMap) {
1647 TIter iter(dpMap);
1648 TObjString* key = 0;
1649 while ((key = (TObjString*) iter.Next()))
1650 dcsMap->Add(key, dpMap->GetValue(key->String()));
1651
1652 dpMap->SetOwner(kFALSE);
1653 delete dpMap;
a038aa70 1654 }
73abe331 1655 }
1656 }
dc25836b 1657
2bb7b766 1658 // DCS Archive DB processing successful. Call Preprocessor!
85a80aa9 1659 UpdateShuttleStatus(AliShuttleStatus::kPPStarted);
a7160fe9 1660
a038aa70 1661 UInt_t returnValue = aPreprocessor->Process(dcsMap);
b948db8d 1662
3301427a 1663 if (returnValue > 0) // Preprocessor error!
1664 {
9827400b 1665 Log(fCurrentDetector, Form("Preprocessor failed. Process returned %d.", returnValue));
cb343cfd 1666 UpdateShuttleStatus(AliShuttleStatus::kPPError);
a038aa70 1667 dcsMap->DeleteAll();
1668 delete dcsMap;
9827400b 1669 return kFALSE;
1670 }
1671
1672 // preprocessor ok!
1673 UpdateShuttleStatus(AliShuttleStatus::kPPDone);
1674 Log(fCurrentDetector, Form("ProcessCurrentDetector - %s preprocessor returned success",
1675 fCurrentDetector.Data()));
b948db8d 1676
a038aa70 1677 dcsMap->DeleteAll();
1678 delete dcsMap;
b948db8d 1679
9827400b 1680 return kTRUE;
2bb7b766 1681}
1682
1683//______________________________________________________________________________________________
1684Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
1685 TObjArray& entries)
1686{
9827400b 1687 // Query DAQ's Shuttle logbook and fills detector status object.
1688 // Call QueryRunParameters to query DAQ logbook for run parameters.
1689 //
2bb7b766 1690
fc5a4708 1691 entries.SetOwner(1);
1692
2bb7b766 1693 // check connection, in case connect
be48e3ea 1694 if(!Connect(3)) return kFALSE;
2bb7b766 1695
1696 TString sqlQuery;
441b0e9c 1697 sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
2bb7b766 1698
be48e3ea 1699 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1700 if (!aResult) {
1701 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
1702 return kFALSE;
1703 }
1704
fc5a4708 1705 AliDebug(2,Form("Query = %s", sqlQuery.Data()));
1706
2bb7b766 1707 if(aResult->GetRowCount() == 0) {
9827400b 1708 AliInfo("No entries in Shuttle Logbook match request");
1709 delete aResult;
1710 return kTRUE;
2bb7b766 1711 }
1712
1713 // TODO Check field count!
db99d43e 1714 const UInt_t nCols = 23;
2bb7b766 1715 if (aResult->GetFieldCount() != (Int_t) nCols) {
1716 AliError("Invalid SQL result field number!");
1717 delete aResult;
1718 return kFALSE;
1719 }
1720
2bb7b766 1721 TSQLRow* aRow;
1722 while ((aRow = aResult->Next())) {
1723 TString runString(aRow->GetField(0), aRow->GetFieldLength(0));
1724 Int_t run = runString.Atoi();
1725
eba76848 1726 AliShuttleLogbookEntry *entry = QueryRunParameters(run);
1727 if (!entry)
1728 continue;
2bb7b766 1729
1730 // loop on detectors
eba76848 1731 for(UInt_t ii = 0; ii < nCols; ii++)
1732 entry->SetDetectorStatus(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1733
eba76848 1734 entries.AddLast(entry);
2bb7b766 1735 delete aRow;
1736 }
1737
2bb7b766 1738 delete aResult;
1739 return kTRUE;
1740}
1741
1742//______________________________________________________________________________________________
eba76848 1743AliShuttleLogbookEntry* AliShuttle::QueryRunParameters(Int_t run)
2bb7b766 1744{
eba76848 1745 //
1746 // Retrieve run parameters written in the DAQ logbook and sets them into AliShuttleLogbookEntry object
1747 //
2bb7b766 1748
1749 // check connection, in case connect
be48e3ea 1750 if (!Connect(3))
eba76848 1751 return 0;
2bb7b766 1752
1753 TString sqlQuery;
2c15234c 1754 sqlQuery.Form("select * from %s where run=%d", fConfig->GetDAQlbTable(), run);
2bb7b766 1755
be48e3ea 1756 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1757 if (!aResult) {
1758 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
eba76848 1759 return 0;
2bb7b766 1760 }
1761
eba76848 1762 if (aResult->GetRowCount() == 0) {
2bb7b766 1763 Log("SHUTTLE", Form("QueryRunParameters - No entry in DAQ Logbook for run %d. Skipping", run));
1764 delete aResult;
eba76848 1765 return 0;
2bb7b766 1766 }
1767
eba76848 1768 if (aResult->GetRowCount() > 1) {
2bb7b766 1769 AliError(Form("More than one entry in DAQ Logbook for run %d. Skipping", run));
1770 delete aResult;
eba76848 1771 return 0;
2bb7b766 1772 }
1773
eba76848 1774 TSQLRow* aRow = aResult->Next();
1775 if (!aRow)
1776 {
1777 AliError(Form("Could not retrieve row for run %d. Skipping", run));
1778 delete aResult;
1779 return 0;
1780 }
2bb7b766 1781
eba76848 1782 AliShuttleLogbookEntry* entry = new AliShuttleLogbookEntry(run);
2bb7b766 1783
eba76848 1784 for (Int_t ii = 0; ii < aResult->GetFieldCount(); ii++)
1785 entry->SetRunParameter(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1786
eba76848 1787 UInt_t startTime = entry->GetStartTime();
1788 UInt_t endTime = entry->GetEndTime();
1789
1790 if (!startTime || !endTime || startTime > endTime) {
1791 Log("SHUTTLE",
1792 Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d",
1793 run, startTime, endTime));
1794 delete entry;
2bb7b766 1795 delete aRow;
eba76848 1796 delete aResult;
1797 return 0;
2bb7b766 1798 }
1799
eba76848 1800 delete aRow;
2bb7b766 1801 delete aResult;
eba76848 1802
1803 return entry;
2bb7b766 1804}
1805
a038aa70 1806//______________________________________________________________________________________________
1807TMap* AliShuttle::GetValueSet(const char* host, Int_t port, const TSeqCollection* entries,
542b6cc8 1808 DCSType type, Int_t multiSplit)
a038aa70 1809{
1810 // Retrieve all "entry" data points from the DCS server
1811 // host, port: TSocket connection parameters
1812 // entries: list of name of the alias or data point
1813 // type: kAlias or kDP
1814 // returns TMap of values, 0 when failure
542b6cc8 1815
1816 AliDCSClient client(host, port, fTimeout, fRetries, multiSplit);
b41b252a 1817
a038aa70 1818 TMap* result = 0;
b41b252a 1819 if (type == kAlias)
a038aa70 1820 {
b41b252a 1821 result = client.GetAliasValues(entries, GetCurrentStartTime(),
1822 GetCurrentEndTime());
1823 }
1824 else if (type == kDP)
1825 {
1826 result = client.GetDPValues(entries, GetCurrentStartTime(),
1827 GetCurrentEndTime());
1828 }
a038aa70 1829
b41b252a 1830 if (result == 0)
1831 {
1832 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get entries! Reason: %s",
1790d4b7 1833 client.GetErrorString(client.GetResultErrorCode())));
1834 if (client.GetResultErrorCode() == AliDCSClient::fgkServerError)
1835 Log(fCurrentDetector.Data(), Form("GetValueSet - Server error code: %s",
1836 client.GetServerError().Data()));
a038aa70 1837
b41b252a 1838 return 0;
a038aa70 1839 }
b41b252a 1840
a038aa70 1841 return result;
1842}
b41b252a 1843
b948db8d 1844//______________________________________________________________________________________________
57f50b3c 1845const char* AliShuttle::GetFile(Int_t system, const char* detector,
1846 const char* id, const char* source)
b948db8d 1847{
9827400b 1848 // Get calibration file from file exchange servers
1849 // First queris the FXS database for the file name, using the run, detector, id and source info
1850 // then calls RetrieveFile(filename) for actual copy to local disk
1851 // run: current run being processed (given by Logbook entry fLogbookEntry)
1852 // detector: the Preprocessor name
1853 // id: provided as a parameter by the Preprocessor
1854 // source: provided by the Preprocessor through GetFileSources function
1855
1856 // check if test mode should simulate a FXS error
1857 if (fTestMode & kErrorFXSFiles)
1858 {
1859 Log(detector, Form("GetFile - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
1860 return 0;
1861 }
1862
57f50b3c 1863 // check connection, in case connect
9d733021 1864 if (!Connect(system))
eba76848 1865 {
9d733021 1866 Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
57f50b3c 1867 return 0;
1868 }
1869
1870 // Query preparation
9d733021 1871 TString sourceName(source);
d386d623 1872 Int_t nFields = 3;
1873 TString sqlQueryStart = Form("select filePath,size,fileChecksum from %s where",
1874 fConfig->GetFXSdbTable(system));
1875 TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
1876 GetCurrentRun(), detector, id);
1877
9d733021 1878 if (system == kDAQ)
1879 {
d386d623 1880 whereClause += Form(" and DAQsource=\"%s\"", source);
57f50b3c 1881 }
9d733021 1882 else if (system == kDCS)
eba76848 1883 {
9d733021 1884 sourceName="none";
57f50b3c 1885 }
9d733021 1886 else if (system == kHLT)
9e080f92 1887 {
d386d623 1888 whereClause += Form(" and DDLnumbers=\"%s\"", source);
9d733021 1889 nFields = 3;
9e080f92 1890 }
1891
9e080f92 1892 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
1893
1894 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
1895
1896 // Query execution
1897 TSQLResult* aResult = 0;
9d733021 1898 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
9e080f92 1899 if (!aResult) {
9d733021 1900 Log(detector, Form("GetFileName - Can't execute SQL query to %s database for: id = %s, source = %s",
1901 GetSystemName(system), id, sourceName.Data()));
9e080f92 1902 return 0;
1903 }
1904
1905 if(aResult->GetRowCount() == 0)
1906 {
1907 Log(detector,
9d733021 1908 Form("GetFileName - No entry in %s FXS db for: id = %s, source = %s",
1909 GetSystemName(system), id, sourceName.Data()));
9e080f92 1910 delete aResult;
1911 return 0;
1912 }
2bb7b766 1913
9e080f92 1914 if (aResult->GetRowCount() > 1) {
1915 Log(detector,
9d733021 1916 Form("GetFileName - More than one entry in %s FXS db for: id = %s, source = %s",
1917 GetSystemName(system), id, sourceName.Data()));
9e080f92 1918 delete aResult;
1919 return 0;
1920 }
1921
9d733021 1922 if (aResult->GetFieldCount() != nFields) {
9e080f92 1923 Log(detector,
9d733021 1924 Form("GetFileName - Wrong field count in %s FXS db for: id = %s, source = %s",
1925 GetSystemName(system), id, sourceName.Data()));
9e080f92 1926 delete aResult;
1927 return 0;
1928 }
1929
1930 TSQLRow* aRow = dynamic_cast<TSQLRow*> (aResult->Next());
1931
1932 if (!aRow){
9d733021 1933 Log(detector, Form("GetFileName - Empty set result in %s FXS db from query: id = %s, source = %s",
1934 GetSystemName(system), id, sourceName.Data()));
9e080f92 1935 delete aResult;
1936 return 0;
1937 }
1938
1939 TString filePath(aRow->GetField(0), aRow->GetFieldLength(0));
1940 TString fileSize(aRow->GetField(1), aRow->GetFieldLength(1));
d386d623 1941 TString fileChecksum(aRow->GetField(2), aRow->GetFieldLength(2));
9e080f92 1942
1943 delete aResult;
1944 delete aRow;
1945
d386d623 1946 AliDebug(2, Form("filePath = %s; size = %s, fileChecksum = %s",
1947 filePath.Data(), fileSize.Data(), fileChecksum.Data()));
9e080f92 1948
9e080f92 1949 // retrieved file is renamed to make it unique
9d733021 1950 TString localFileName = Form("%s_%s_%d_%s_%s.shuttle",
1951 GetSystemName(system), detector, GetCurrentRun(), id, sourceName.Data());
1952
9e080f92 1953
9d733021 1954 // file retrieval from FXS
4b95672b 1955 UInt_t nRetries = 0;
1956 UInt_t maxRetries = 3;
1957 Bool_t result = kFALSE;
1958
1959 // copy!! if successful TSystem::Exec returns 0
1960 while(nRetries++ < maxRetries) {
1961 AliDebug(2, Form("Trying to copy file. Retry # %d", nRetries));
1962 result = RetrieveFile(system, filePath.Data(), localFileName.Data());
1963 if(!result)
1964 {
1965 Log(detector, Form("GetFileName - Copy of file %s from %s FXS failed",
9d733021 1966 filePath.Data(), GetSystemName(system)));
4b95672b 1967 continue;
4f0749a8 1968 }
9e080f92 1969
d386d623 1970 if (fileChecksum.Length()>0)
4b95672b 1971 {
1972 // compare md5sum of local file with the one stored in the FXS DB
1973 Int_t md5Comp = gSystem->Exec(Form("md5sum %s/%s |grep %s 2>&1 > /dev/null",
d386d623 1974 GetShuttleTempDir(), localFileName.Data(), fileChecksum.Data()));
9e080f92 1975
4b95672b 1976 if (md5Comp != 0)
1977 {
1978 Log(detector, Form("GetFileName - md5sum of file %s does not match with local copy!",
1979 filePath.Data()));
1980 result = kFALSE;
1981 continue;
1982 }
d386d623 1983 } else {
1984 Log(fCurrentDetector, Form("GetFile - md5sum of file %s not set in %s database, skipping comparison",
1985 filePath.Data(), GetSystemName(system)));
9d733021 1986 }
4b95672b 1987 if (result) break;
9e080f92 1988 }
1989
4b95672b 1990 if(!result) return 0;
1991
9d733021 1992 fFXSCalled[system]=kTRUE;
1993 TObjString *fileParams = new TObjString(Form("%s#!?!#%s", id, sourceName.Data()));
1994 fFXSlist[system].Add(fileParams);
9e080f92 1995
1996 static TString fullLocalFileName;
1bcd28db 1997 fullLocalFileName.Form("%s/%s", GetShuttleTempDir(), localFileName.Data());
36c99a6a 1998
1bcd28db 1999 Log(fCurrentDetector, Form("GetFile - Retrieved file with id %s and source %s from %s to %s", id, source, GetSystemName(system), fullLocalFileName.Data()));
9e080f92 2000
2001 return fullLocalFileName.Data();
2bb7b766 2002}
2003
2004//______________________________________________________________________________________________
9d733021 2005Bool_t AliShuttle::RetrieveFile(UInt_t system, const char* fxsFileName, const char* localFileName)
9e080f92 2006{
9827400b 2007 //
2008 // Copies file from FXS to local Shuttle machine
2009 //
2bb7b766 2010
9e080f92 2011 // check temp directory: trying to cd to temp; if it does not exist, create it
9d733021 2012 AliDebug(2, Form("Copy file %s from %s FXS into %s/%s",
2013 GetSystemName(system), fxsFileName, GetShuttleTempDir(), localFileName));
9e080f92 2014
36c99a6a 2015 void* dir = gSystem->OpenDirectory(GetShuttleTempDir());
9e080f92 2016 if (dir == NULL) {
36c99a6a 2017 if (gSystem->mkdir(GetShuttleTempDir(), kTRUE)) {
2018 AliError(Form("Can't open directory <%s>", GetShuttleTempDir()));
9e080f92 2019 return kFALSE;
2020 }
2021
2022 } else {
2023 gSystem->FreeDirectory(dir);
2024 }
2025
9d733021 2026 TString baseFXSFolder;
2027 if (system == kDAQ)
2028 {
2029 baseFXSFolder = "FES/";
2030 }
2031 else if (system == kDCS)
2032 {
2033 baseFXSFolder = "";
2034 }
2035 else if (system == kHLT)
2036 {
42fde080 2037 baseFXSFolder = "/opt/FXS/";
9d733021 2038 }
2039
2040
2041 TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s/%s",
2042 fConfig->GetFXSPort(system),
2043 fConfig->GetFXSUser(system),
2044 fConfig->GetFXSHost(system),
2045 baseFXSFolder.Data(),
2046 fxsFileName,
36c99a6a 2047 GetShuttleTempDir(),
9e080f92 2048 localFileName);
2049
2050 AliDebug(2, Form("%s",command.Data()));
2051
4b95672b 2052 Bool_t result = (gSystem->Exec(command.Data()) == 0);
9e080f92 2053
4b95672b 2054 return result;
9e080f92 2055}
2056
2057//______________________________________________________________________________________________
9d733021 2058TList* AliShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
2059{
9827400b 2060 //
2061 // Get sources producing the condition file Id from file exchange servers
4a33bdd9 2062 // if id is NULL all sources are returned (distinct)
9827400b 2063 //
1bcd28db 2064
2065 Log(detector, Form("GetFileSources - Retrieving sources with id %s from %s", id, GetSystemName(system)));
9827400b 2066
2067 // check if test mode should simulate a FXS error
2068 if (fTestMode & kErrorFXSSources)
2069 {
2070 Log(detector, Form("GetFileSources - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2071 return 0;
2072 }
2073
9d733021 2074 if (system == kDCS)
2075 {
6297b37d 2076 AliWarning("DCS system has only one source of data!");
2077 TList *list = new TList();
2078 list->SetOwner(1);
2079 list->Add(new TObjString(" "));
2080 return list;
9d733021 2081 }
9e080f92 2082
2083 // check connection, in case connect
9d733021 2084 if (!Connect(system))
2085 {
4a33bdd9 2086 Log(detector, Form("GetFileSources - Couldn't connect to %s FXS database", GetSystemName(system)));
9d733021 2087 return NULL;
9e080f92 2088 }
2089
9d733021 2090 TString sourceName = 0;
2091 if (system == kDAQ)
2092 {
2093 sourceName = "DAQsource";
2094 } else if (system == kHLT)
2095 {
2096 sourceName = "DDLnumbers";
2097 }
2098
4a33bdd9 2099 TString sqlQueryStart = Form("select distinct %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
2100 TString whereClause = Form("run=%d and detector=\"%s\"",
2101 GetCurrentRun(), detector);
2102 if (id)
2103 whereClause += Form(" and fileId=\"%s\"", id);
9e080f92 2104 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2105
2106 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2107
2108 // Query execution
2109 TSQLResult* aResult;
9d733021 2110 aResult = fServer[system]->Query(sqlQuery);
9e080f92 2111 if (!aResult) {
9d733021 2112 Log(detector, Form("GetFileSources - Can't execute SQL query to %s database for id: %s",
2113 GetSystemName(system), id));
9e080f92 2114 return 0;
2115 }
2116
86aa42c3 2117 TList *list = new TList();
2118 list->SetOwner(1);
2119
9d733021 2120 if (aResult->GetRowCount() == 0)
2121 {
9e080f92 2122 Log(detector,
9d733021 2123 Form("GetFileSources - No entry in %s FXS table for id: %s", GetSystemName(system), id));
9e080f92 2124 delete aResult;
86aa42c3 2125 return list;
9e080f92 2126 }
2127
1bcd28db 2128 Log(detector, Form("GetFileSources - Found %d sources", aResult->GetRowCount()));
9e080f92 2129
1bcd28db 2130 TSQLRow* aRow;
9d733021 2131 while ((aRow = aResult->Next()))
2132 {
9e080f92 2133
9d733021 2134 TString source(aRow->GetField(0), aRow->GetFieldLength(0));
2135 AliDebug(2, Form("%s = %s", sourceName.Data(), source.Data()));
2136 list->Add(new TObjString(source));
9e080f92 2137 delete aRow;
2138 }
9d733021 2139
9e080f92 2140 delete aResult;
2141
2142 return list;
2bb7b766 2143}
2144
4a33bdd9 2145//______________________________________________________________________________________________
2146TList* AliShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
2147{
2148 //
2149 // Get all ids of condition files produced by a given source from file exchange servers
2150 //
2151
1bcd28db 2152 Log(detector, Form("GetFileIDs - Retrieving ids with source %s with %s", source, GetSystemName(system)));
2153
4a33bdd9 2154 // check if test mode should simulate a FXS error
2155 if (fTestMode & kErrorFXSSources)
2156 {
2157 Log(detector, Form("GetFileIDs - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2158 return 0;
2159 }
2160
2161 // check connection, in case connect
2162 if (!Connect(system))
2163 {
2164 Log(detector, Form("GetFileIDs - Couldn't connect to %s FXS database", GetSystemName(system)));
2165 return NULL;
2166 }
2167
2168 TString sourceName = 0;
2169 if (system == kDAQ)
2170 {
2171 sourceName = "DAQsource";
2172 } else if (system == kHLT)
2173 {
2174 sourceName = "DDLnumbers";
2175 }
2176
2177 TString sqlQueryStart = Form("select fileId from %s where", fConfig->GetFXSdbTable(system));
2178 TString whereClause = Form("run=%d and detector=\"%s\"",
2179 GetCurrentRun(), detector);
2180 if (sourceName.Length() > 0 && source)
2181 whereClause += Form(" and %s=\"%s\"", sourceName.Data(), source);
2182 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2183
2184 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2185
2186 // Query execution
2187 TSQLResult* aResult;
2188 aResult = fServer[system]->Query(sqlQuery);
2189 if (!aResult) {
2190 Log(detector, Form("GetFileIDs - Can't execute SQL query to %s database for source: %s",
2191 GetSystemName(system), source));
2192 return 0;
2193 }
2194
2195 TList *list = new TList();
2196 list->SetOwner(1);
2197
2198 if (aResult->GetRowCount() == 0)
2199 {
2200 Log(detector,
2201 Form("GetFileIDs - No entry in %s FXS table for source: %s", GetSystemName(system), source));
2202 delete aResult;
2203 return list;
2204 }
2205
1bcd28db 2206 Log(detector, Form("GetFileIDs - Found %d ids", aResult->GetRowCount()));
2207
4a33bdd9 2208 TSQLRow* aRow;
2209
2210 while ((aRow = aResult->Next()))
2211 {
2212
2213 TString id(aRow->GetField(0), aRow->GetFieldLength(0));
2214 AliDebug(2, Form("fileId = %s", id.Data()));
2215 list->Add(new TObjString(id));
2216 delete aRow;
2217 }
2218
2219 delete aResult;
2220
2221 return list;
2222}
2223
2bb7b766 2224//______________________________________________________________________________________________
9d733021 2225Bool_t AliShuttle::Connect(Int_t system)
2bb7b766 2226{
9827400b 2227 // Connect to MySQL Server of the system's FXS MySQL databases
2228 // DAQ Logbook, Shuttle Logbook and DAQ FXS db are on the same host
2229 //
57f50b3c 2230
9d733021 2231 // check connection: if already connected return
2232 if(fServer[system] && fServer[system]->IsConnected()) return kTRUE;
57f50b3c 2233
9d733021 2234 TString dbHost, dbUser, dbPass, dbName;
57f50b3c 2235
9d733021 2236 if (system < 3) // FXS db servers
2237 {
2238 dbHost = Form("mysql://%s:%d", fConfig->GetFXSdbHost(system), fConfig->GetFXSdbPort(system));
2239 dbUser = fConfig->GetFXSdbUser(system);
2240 dbPass = fConfig->GetFXSdbPass(system);
2241 dbName = fConfig->GetFXSdbName(system);
2242 } else { // Run & Shuttle logbook servers
2243 // TODO Will the Shuttle logbook server be the same as the Run logbook server ???
2244 dbHost = Form("mysql://%s:%d", fConfig->GetDAQlbHost(), fConfig->GetDAQlbPort());
2245 dbUser = fConfig->GetDAQlbUser();
2246 dbPass = fConfig->GetDAQlbPass();
2247 dbName = fConfig->GetDAQlbDB();
2248 }
57f50b3c 2249
9d733021 2250 fServer[system] = TSQLServer::Connect(dbHost.Data(), dbUser.Data(), dbPass.Data());
2251 if (!fServer[system] || !fServer[system]->IsConnected()) {
2252 if(system < 3)
2253 {
2254 AliError(Form("Can't establish connection to FXS database for %s",
2255 AliShuttleInterface::GetSystemName(system)));
2256 } else {
2257 AliError("Can't establish connection to Run logbook.");
57f50b3c 2258 }
9d733021 2259 if(fServer[system]) delete fServer[system];
2260 return kFALSE;
2bb7b766 2261 }
57f50b3c 2262
9d733021 2263 // Get tables
2264 TSQLResult* aResult=0;
2265 switch(system){
2266 case kDAQ:
2267 aResult = fServer[kDAQ]->GetTables(dbName.Data());
2268 break;
2269 case kDCS:
2270 aResult = fServer[kDCS]->GetTables(dbName.Data());
2271 break;
2272 case kHLT:
2273 aResult = fServer[kHLT]->GetTables(dbName.Data());
2274 break;
2275 default:
2276 aResult = fServer[3]->GetTables(dbName.Data());
2277 break;
2278 }
2279
2280 delete aResult;
2bb7b766 2281 return kTRUE;
2282}
57f50b3c 2283
9e080f92 2284//______________________________________________________________________________________________
9d733021 2285Bool_t AliShuttle::UpdateTable()
9e080f92 2286{
9827400b 2287 //
2288 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2289 //
9e080f92 2290
9d733021 2291 Bool_t result = kTRUE;
9e080f92 2292
9d733021 2293 for (UInt_t system=0; system<3; system++)
2294 {
2295 if(!fFXSCalled[system]) continue;
9e080f92 2296
9d733021 2297 // check connection, in case connect
2298 if (!Connect(system))
2299 {
2300 Log(fCurrentDetector, Form("UpdateTable - Couldn't connect to %s FXS database", GetSystemName(system)));
2301 result = kFALSE;
2302 continue;
9e080f92 2303 }
9e080f92 2304
9d733021 2305 TTimeStamp now; // now
2306
2307 // Loop on FXS list entries
2308 TIter iter(&fFXSlist[system]);
2309 TObjString *aFXSentry=0;
2310 while ((aFXSentry = dynamic_cast<TObjString*> (iter.Next())))
2311 {
2312 TString aFXSentrystr = aFXSentry->String();
2313 TObjArray *aFXSarray = aFXSentrystr.Tokenize("#!?!#");
2314 if (!aFXSarray || aFXSarray->GetEntries() != 2 )
2315 {
2316 Log(fCurrentDetector, Form("UpdateTable - error updating %s FXS entry. Check string: <%s>",
2317 GetSystemName(system), aFXSentrystr.Data()));
2318 if(aFXSarray) delete aFXSarray;
2319 result = kFALSE;
2320 continue;
2321 }
2322 const char* fileId = ((TObjString*) aFXSarray->At(0))->GetName();
2323 const char* source = ((TObjString*) aFXSarray->At(1))->GetName();
2324
2325 TString whereClause;
2326 if (system == kDAQ)
2327 {
2328 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DAQsource=\"%s\";",
2329 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2330 }
2331 else if (system == kDCS)
2332 {
2333 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\";",
2334 GetCurrentRun(), fCurrentDetector.Data(), fileId);
2335 }
2336 else if (system == kHLT)
2337 {
2338 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DDLnumbers=\"%s\";",
2339 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2340 }
2341
2342 delete aFXSarray;
9e080f92 2343
9d733021 2344 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2345 now.GetSec(), whereClause.Data());
9e080f92 2346
9d733021 2347 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
9e080f92 2348
9d733021 2349 // Query execution
2350 TSQLResult* aResult;
2351 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2352 if (!aResult)
2353 {
2354 Log(fCurrentDetector, Form("UpdateTable - %s db: can't execute SQL query <%s>",
2355 GetSystemName(system), sqlQuery.Data()));
2356 result = kFALSE;
2357 continue;
2358 }
2359 delete aResult;
9e080f92 2360 }
9e080f92 2361 }
2362
9d733021 2363 return result;
9e080f92 2364}
57f50b3c 2365
3301427a 2366//______________________________________________________________________________________________
2367Bool_t AliShuttle::UpdateTableFailCase()
2368{
9827400b 2369 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2370 // this is called in case the preprocessor is declared failed for the current run, because
2371 // the fields are updated only in case of success
3301427a 2372
2373 Bool_t result = kTRUE;
2374
2375 for (UInt_t system=0; system<3; system++)
2376 {
2377 // check connection, in case connect
2378 if (!Connect(system))
2379 {
2380 Log(fCurrentDetector, Form("UpdateTableFailCase - Couldn't connect to %s FXS database",
2381 GetSystemName(system)));
2382 result = kFALSE;
2383 continue;
2384 }
2385
2386 TTimeStamp now; // now
2387
2388 // Loop on FXS list entries
2389
2390 TString whereClause = Form("where run=%d and detector=\"%s\";",
2391 GetCurrentRun(), fCurrentDetector.Data());
2392
2393
2394 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2395 now.GetSec(), whereClause.Data());
2396
2397 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2398
2399 // Query execution
2400 TSQLResult* aResult;
2401 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2402 if (!aResult)
2403 {
2404 Log(fCurrentDetector, Form("UpdateTableFailCase - %s db: can't execute SQL query <%s>",
2405 GetSystemName(system), sqlQuery.Data()));
2406 result = kFALSE;
2407 continue;
2408 }
2409 delete aResult;
2410 }
2411
2412 return result;
2413}
2414
2bb7b766 2415//______________________________________________________________________________________________
2416Bool_t AliShuttle::UpdateShuttleLogbook(const char* detector, const char* status)
2417{
e7f62f16 2418 //
2419 // Update Shuttle logbook filling detector or shuttle_done column
2420 // ex. of usage: UpdateShuttleLogbook("PHOS", "DONE") or UpdateShuttleLogbook("shuttle_done")
2421 //
57f50b3c 2422
2bb7b766 2423 // check connection, in case connect
be48e3ea 2424 if(!Connect(3)){
2bb7b766 2425 Log("SHUTTLE", "UpdateShuttleLogbook - Couldn't connect to DAQ Logbook.");
2426 return kFALSE;
57f50b3c 2427 }
2428
2bb7b766 2429 TString detName(detector);
2430 TString setClause;
e7f62f16 2431 if(detName == "shuttle_done")
2432 {
2bb7b766 2433 setClause = "set shuttle_done=1";
e7f62f16 2434
2435 // Send the information to ML
2436 TMonaLisaText mlStatus("SHUTTLE_status", "Done");
2437
2438 TList mlList;
2439 mlList.Add(&mlStatus);
2440
2441 fMonaLisa->SendParameters(&mlList);
2bb7b766 2442 } else {
2bb7b766 2443 TString statusStr(status);
2444 if(statusStr.Contains("done", TString::kIgnoreCase) ||
2445 statusStr.Contains("failed", TString::kIgnoreCase)){
eba76848 2446 setClause = Form("set %s=\"%s\"", detector, status);
2bb7b766 2447 } else {
2448 Log("SHUTTLE",
2449 Form("UpdateShuttleLogbook - Invalid status <%s> for detector %s",
2450 status, detector));
2451 return kFALSE;
2452 }
2453 }
57f50b3c 2454
2bb7b766 2455 TString whereClause = Form("where run=%d", GetCurrentRun());
2456
441b0e9c 2457 TString sqlQuery = Form("update %s %s %s",
2458 fConfig->GetShuttlelbTable(), setClause.Data(), whereClause.Data());
57f50b3c 2459
2bb7b766 2460 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2461
2462 // Query execution
2463 TSQLResult* aResult;
be48e3ea 2464 aResult = dynamic_cast<TSQLResult*> (fServer[3]->Query(sqlQuery));
2bb7b766 2465 if (!aResult) {
2466 Log("SHUTTLE", Form("UpdateShuttleLogbook - Can't execute query <%s>", sqlQuery.Data()));
2467 return kFALSE;
57f50b3c 2468 }
2bb7b766 2469 delete aResult;
57f50b3c 2470
2471 return kTRUE;
2472}
2473
2474//______________________________________________________________________________________________
2bb7b766 2475Int_t AliShuttle::GetCurrentRun() const
2476{
9827400b 2477 //
2478 // Get current run from logbook entry
2479 //
57f50b3c 2480
2bb7b766 2481 return fLogbookEntry ? fLogbookEntry->GetRun() : -1;
57f50b3c 2482}
2483
2484//______________________________________________________________________________________________
2bb7b766 2485UInt_t AliShuttle::GetCurrentStartTime() const
2486{
9827400b 2487 //
2488 // get current start time
2489 //
57f50b3c 2490
2bb7b766 2491 return fLogbookEntry ? fLogbookEntry->GetStartTime() : 0;
57f50b3c 2492}
2493
2494//______________________________________________________________________________________________
2bb7b766 2495UInt_t AliShuttle::GetCurrentEndTime() const
2496{
9827400b 2497 //
2498 // get current end time from logbook entry
2499 //
57f50b3c 2500
2bb7b766 2501 return fLogbookEntry ? fLogbookEntry->GetEndTime() : 0;
57f50b3c 2502}
2503
b948db8d 2504//______________________________________________________________________________________________
2505void AliShuttle::Log(const char* detector, const char* message)
2506{
9827400b 2507 //
2508 // Fill log string with a message
2509 //
b948db8d 2510
36c99a6a 2511 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
84090f85 2512 if (dir == NULL) {
36c99a6a 2513 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE)) {
2514 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
84090f85 2515 return;
2516 }
b948db8d 2517
84090f85 2518 } else {
2519 gSystem->FreeDirectory(dir);
2520 }
b948db8d 2521
cb343cfd 2522 TString toLog = Form("%s (%d): %s - ", TTimeStamp(time(0)).AsString("s"), getpid(), detector);
e7f62f16 2523 if (GetCurrentRun() >= 0)
2524 toLog += Form("run %d - ", GetCurrentRun());
2bb7b766 2525 toLog += Form("%s", message);
2526
84090f85 2527 AliInfo(toLog.Data());
ffa29e93 2528
2529 // if we redirect the log output already to the file, leave here
2530 if (fOutputRedirected && strcmp(detector, "SHUTTLE") != 0)
2531 return;
b948db8d 2532
ffa29e93 2533 TString fileName = GetLogFileName(detector);
e7f62f16 2534
84090f85 2535 gSystem->ExpandPathName(fileName);
2536
2537 ofstream logFile;
2538 logFile.open(fileName, ofstream::out | ofstream::app);
2539
2540 if (!logFile.is_open()) {
2541 AliError(Form("Could not open file %s", fileName.Data()));
2542 return;
2543 }
7bfb2090 2544
84090f85 2545 logFile << toLog.Data() << "\n";
b948db8d 2546
84090f85 2547 logFile.close();
b948db8d 2548}
2bb7b766 2549
ffa29e93 2550//______________________________________________________________________________________________
2551TString AliShuttle::GetLogFileName(const char* detector) const
2552{
2553 //
2554 // returns the name of the log file for a given sub detector
2555 //
2556
2557 TString fileName;
2558
2559 if (GetCurrentRun() >= 0)
2560 fileName.Form("%s/%s_%d.log", GetShuttleLogDir(), detector, GetCurrentRun());
2561 else
2562 fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
2563
2564 return fileName;
2565}
2566
2bb7b766 2567//______________________________________________________________________________________________
2568Bool_t AliShuttle::Collect(Int_t run)
2569{
9827400b 2570 //
2571 // Collects conditions data for all UNPROCESSED run written to DAQ LogBook in case of run = -1 (default)
2572 // If a dedicated run is given this run is processed
2573 //
2574 // In operational mode, this is the Shuttle function triggered by the EOR signal.
2575 //
2bb7b766 2576
eba76848 2577 if (run == -1)
2578 Log("SHUTTLE","Collect - Shuttle called. Collecting conditions data for unprocessed runs");
2579 else
2580 Log("SHUTTLE", Form("Collect - Shuttle called. Collecting conditions data for run %d", run));
cb343cfd 2581
2582 SetLastAction("Starting");
2bb7b766 2583
2584 TString whereClause("where shuttle_done=0");
eba76848 2585 if (run != -1)
2586 whereClause += Form(" and run=%d", run);
2bb7b766 2587
2588 TObjArray shuttleLogbookEntries;
be48e3ea 2589 if (!QueryShuttleLogbook(whereClause, shuttleLogbookEntries))
2590 {
cb343cfd 2591 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2bb7b766 2592 return kFALSE;
2593 }
2594
9e080f92 2595 if (shuttleLogbookEntries.GetEntries() == 0)
2596 {
2597 if (run == -1)
2598 Log("SHUTTLE","Collect - Found no UNPROCESSED runs in Shuttle logbook");
2599 else
2600 Log("SHUTTLE", Form("Collect - Run %d is already DONE "
2601 "or it does not exist in Shuttle logbook", run));
2602 return kTRUE;
2603 }
2604
be48e3ea 2605 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2606 fFirstUnprocessed[iDet] = kTRUE;
2607
fc5a4708 2608 if (run != -1)
be48e3ea 2609 {
2610 // query Shuttle logbook for earlier runs, check if some detectors are unprocessed,
2611 // flag them into fFirstUnprocessed array
2612 TString whereClause(Form("where shuttle_done=0 and run < %d", run));
2613 TObjArray tmpLogbookEntries;
2614 if (!QueryShuttleLogbook(whereClause, tmpLogbookEntries))
2615 {
2616 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2617 return kFALSE;
2618 }
2619
2620 TIter iter(&tmpLogbookEntries);
2621 AliShuttleLogbookEntry* anEntry = 0;
2622 while ((anEntry = dynamic_cast<AliShuttleLogbookEntry*> (iter.Next())))
2623 {
2624 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2625 {
2626 if (anEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
2627 {
2628 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
2629 anEntry->GetRun(), GetDetName(iDet)));
2630 fFirstUnprocessed[iDet] = kFALSE;
2631 }
2632 }
2633
2634 }
2635
2636 }
2637
2638 if (!RetrieveConditionsData(shuttleLogbookEntries))
2639 {
cb343cfd 2640 Log("SHUTTLE", "Collect - Process of at least one run failed");
2bb7b766 2641 return kFALSE;
2642 }
2643
36c99a6a 2644 Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
eba76848 2645 return kTRUE;
2bb7b766 2646}
2647
2bb7b766 2648//______________________________________________________________________________________________
2649Bool_t AliShuttle::RetrieveConditionsData(const TObjArray& dateEntries)
2650{
9827400b 2651 //
2652 // Retrieve conditions data for all runs that aren't processed yet
2653 //
2bb7b766 2654
2655 Bool_t hasError = kFALSE;
2656
2657 TIter iter(&dateEntries);
2658 AliShuttleLogbookEntry* anEntry;
2659
2660 while ((anEntry = (AliShuttleLogbookEntry*) iter.Next())){
2661 if (!Process(anEntry)){
2662 hasError = kTRUE;
2663 }
4b95672b 2664
2665 // clean SHUTTLE temp directory
3301427a 2666 TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
2667 RemoveFile(filename.Data());
2bb7b766 2668 }
2669
2670 return hasError == kFALSE;
2671}
cb343cfd 2672
2673//______________________________________________________________________________________________
2674ULong_t AliShuttle::GetTimeOfLastAction() const
2675{
9827400b 2676 //
2677 // Gets time of last action
2678 //
2679
cb343cfd 2680 ULong_t tmp;
36c99a6a 2681
cb343cfd 2682 fMonitoringMutex->Lock();
be48e3ea 2683
cb343cfd 2684 tmp = fLastActionTime;
36c99a6a 2685
cb343cfd 2686 fMonitoringMutex->UnLock();
36c99a6a 2687
cb343cfd 2688 return tmp;
2689}
2690
2691//______________________________________________________________________________________________
2692const TString AliShuttle::GetLastAction() const
2693{
9827400b 2694 //
cb343cfd 2695 // returns a string description of the last action
9827400b 2696 //
cb343cfd 2697
2698 TString tmp;
36c99a6a 2699
cb343cfd 2700 fMonitoringMutex->Lock();
2701
2702 tmp = fLastAction;
2703
2704 fMonitoringMutex->UnLock();
2705
36c99a6a 2706 return tmp;
cb343cfd 2707}
2708
2709//______________________________________________________________________________________________
2710void AliShuttle::SetLastAction(const char* action)
2711{
9827400b 2712 //
cb343cfd 2713 // updates the monitoring variables
9827400b 2714 //
36c99a6a 2715
cb343cfd 2716 fMonitoringMutex->Lock();
36c99a6a 2717
cb343cfd 2718 fLastAction = action;
2719 fLastActionTime = time(0);
2720
2721 fMonitoringMutex->UnLock();
2722}
eba76848 2723
2724//______________________________________________________________________________________________
2725const char* AliShuttle::GetRunParameter(const char* param)
2726{
9827400b 2727 //
2728 // returns run parameter read from DAQ logbook
2729 //
eba76848 2730
2731 if(!fLogbookEntry) {
2732 AliError("No logbook entry!");
2733 return 0;
2734 }
2735
2736 return fLogbookEntry->GetRunParameter(param);
2737}
57c1a579 2738
d386d623 2739//______________________________________________________________________________________________
9827400b 2740AliCDBEntry* AliShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
d386d623 2741{
9827400b 2742 //
2743 // returns object from OCDB valid for current run
2744 //
d386d623 2745
9827400b 2746 if (fTestMode & kErrorOCDB)
2747 {
2748 Log(detector, "GetFromOCDB - In TESTMODE - Simulating error with OCDB");
2749 return 0;
2750 }
2751
d386d623 2752 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
2753 if (!sto)
2754 {
9827400b 2755 Log(detector, "GetFromOCDB - Cannot activate main OCDB for query!");
d386d623 2756 return 0;
2757 }
2758
2759 return dynamic_cast<AliCDBEntry*> (sto->Get(path, GetCurrentRun()));
2760}
2761
57c1a579 2762//______________________________________________________________________________________________
2763Bool_t AliShuttle::SendMail()
2764{
9827400b 2765 //
2766 // sends a mail to the subdetector expert in case of preprocessor error
2767 //
2768
2769 if (fTestMode != kNone)
2770 return kTRUE;
57c1a579 2771
36c99a6a 2772 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
57c1a579 2773 if (dir == NULL)
2774 {
36c99a6a 2775 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
57c1a579 2776 {
36c99a6a 2777 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
57c1a579 2778 return kFALSE;
2779 }
2780
2781 } else {
2782 gSystem->FreeDirectory(dir);
2783 }
2784
2785 TString bodyFileName;
36c99a6a 2786 bodyFileName.Form("%s/mail.body", GetShuttleLogDir());
57c1a579 2787 gSystem->ExpandPathName(bodyFileName);
2788
2789 ofstream mailBody;
2790 mailBody.open(bodyFileName, ofstream::out);
2791
2792 if (!mailBody.is_open())
2793 {
2794 AliError(Form("Could not open mail body file %s", bodyFileName.Data()));
2795 return kFALSE;
2796 }
2797
2798 TString to="";
2799 TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
2800 TObjString *anExpert=0;
2801 while ((anExpert = (TObjString*) iterExperts.Next()))
2802 {
2803 to += Form("%s,", anExpert->GetName());
2804 }
2805 to.Remove(to.Length()-1);
909732f7 2806 AliDebug(2, Form("to: %s",to.Data()));
57c1a579 2807
86aa42c3 2808 if (to.IsNull()) {
36c99a6a 2809 AliInfo("List of detector responsibles not yet set!");
2810 return kFALSE;
2811 }
2812
57c1a579 2813 TString cc="alberto.colla@cern.ch";
2814
546242fb 2815 TString subject = Form("%s Shuttle preprocessor FAILED in run %d !",
57c1a579 2816 fCurrentDetector.Data(), GetCurrentRun());
909732f7 2817 AliDebug(2, Form("subject: %s", subject.Data()));
57c1a579 2818
2819 TString body = Form("Dear %s expert(s), \n\n", fCurrentDetector.Data());
2820 body += Form("SHUTTLE just detected that your preprocessor "
546242fb 2821 "failed processing run %d!!\n\n", GetCurrentRun());
2822 body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n", fCurrentDetector.Data());
2823 body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
2824 body += Form("Find the %s log for the current run on \n\n"
2825 "\thttp://pcalishuttle01.cern.ch:8880/logs/%s_%d.log \n\n",
2826 fCurrentDetector.Data(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 2827 body += Form("The last 10 lines of %s log file are following:\n\n");
2828
909732f7 2829 AliDebug(2, Form("Body begin: %s", body.Data()));
57c1a579 2830
2831 mailBody << body.Data();
2832 mailBody.close();
2833 mailBody.open(bodyFileName, ofstream::out | ofstream::app);
2834
9d733021 2835 TString logFileName = Form("%s/%s_%d.log", GetShuttleLogDir(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 2836 TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
2837 if (gSystem->Exec(tailCommand.Data()))
2838 {
2839 mailBody << Form("%s log file not found ...\n\n", fCurrentDetector.Data());
2840 }
2841
2842 TString endBody = Form("------------------------------------------------------\n\n");
36c99a6a 2843 endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
2844 endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
546242fb 2845 endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
57c1a579 2846
909732f7 2847 AliDebug(2, Form("Body end: %s", endBody.Data()));
57c1a579 2848
2849 mailBody << endBody.Data();
2850
2851 mailBody.close();
2852
2853 // send mail!
2854 TString mailCommand = Form("mail -s \"%s\" -c %s %s < %s",
2855 subject.Data(),
2856 cc.Data(),
2857 to.Data(),
2858 bodyFileName.Data());
909732f7 2859 AliDebug(2, Form("mail command: %s", mailCommand.Data()));
57c1a579 2860
2861 Bool_t result = gSystem->Exec(mailCommand.Data());
2862
2863 return result == 0;
2864}
d386d623 2865
441b0e9c 2866//______________________________________________________________________________________________
9827400b 2867const char* AliShuttle::GetRunType()
441b0e9c 2868{
9827400b 2869 //
2870 // returns run type read from "run type" logbook
2871 //
441b0e9c 2872
2873 if(!fLogbookEntry) {
2874 AliError("No logbook entry!");
2875 return 0;
2876 }
2877
9827400b 2878 return fLogbookEntry->GetRunType();
441b0e9c 2879}
2880
4859271b 2881//______________________________________________________________________________________________
2882Bool_t AliShuttle::GetHLTStatus()
2883{
2884 // Return HLT status (ON=1 OFF=0)
2885 // Converts the HLT status from the status string read in the run logbook (not just a bool)
2886
2887 if(!fLogbookEntry) {
2888 AliError("No logbook entry!");
2889 return 0;
2890 }
2891
2892 // TODO implement when HLTStatus is inserted in run logbook
2893 //TString hltStatus = fLogbookEntry->GetRunParameter("HLTStatus");
2894 //if(hltStatus == "OFF") {return kFALSE};
2895
2896 return kTRUE;
2897}
2898
d386d623 2899//______________________________________________________________________________________________
2900void AliShuttle::SetShuttleTempDir(const char* tmpDir)
2901{
9827400b 2902 //
2903 // sets Shuttle temp directory
2904 //
d386d623 2905
2906 fgkShuttleTempDir = gSystem->ExpandPathName(tmpDir);
2907}
2908
2909//______________________________________________________________________________________________
2910void AliShuttle::SetShuttleLogDir(const char* logDir)
2911{
9827400b 2912 //
2913 // sets Shuttle log directory
2914 //
d386d623 2915
2916 fgkShuttleLogDir = gSystem->ExpandPathName(logDir);
2917}