Monalisa started in Collect() function. Alive message to monitor is sent at each...
[u/mrichter/AliRoot.git] / SHUTTLE / AliShuttle.cxx
CommitLineData
73abe331 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
ee6f7523 18Revision 1.69 2007/12/12 10:06:29 acolla
19in AliShuttle.cxx: SHUTTLE logbook is updated in case of invalid run times:
20
21time_start==0 && time_end==0
22
23logbook is NOT updated if time_start != 0 && time_end == 0, because it may mean that the run is still ongoing.
24
1abfbb60 25Revision 1.68 2007/12/11 10:15:17 acolla
26Added marking SHUTTLE=DONE for invalid runs
27(invalid start time or end time) and runs with totalEvents < 1
28
b0e53b15 29Revision 1.67 2007/12/07 19:14:36 acolla
30in AliShuttleTrigger:
31
32Added automatic collection of new runs on a regular time basis (settable from the configuration)
33
34in AliShuttleConfig: new members
35
36- triggerWait: time to wait for DIM trigger (s) before starting automatic collection of new runs
37- mode: run mode (test, prod) -> used to build log folder (logs or logs_PROD)
38
39in AliShuttle:
40
41- logs now stored in logs/#RUN/DET_#RUN.log
42
7d4cf768 43Revision 1.66 2007/12/05 10:45:19 jgrosseo
44changed order of arguments to TMonaLisaWriter
45
468df1ce 46Revision 1.65 2007/11/26 16:58:37 acolla
47Monalisa configuration added: host and table name
48
b832ec02 49Revision 1.64 2007/11/13 16:15:47 acolla
50DCS map is stored in a file in the temp folder where the detector is processed.
51If the preprocessor fails, the temp folder is not removed. This will help the debugging of the problem.
52
b1d18693 53Revision 1.63 2007/11/02 10:53:16 acolla
54Protection added to AliShuttle::CopyFileLocally
55
7d43a416 56Revision 1.62 2007/10/31 18:23:13 acolla
57Furter developement on the Shuttle:
58
59- Shuttle now connects to the Grid as alidaq. The OCDB and Reference folders
60are now built from /alice/data, e.g.:
61/alice/data/2007/LHC07a/OCDB
62
63the year and LHC period are taken from the Shuttle.
64Raw metadata files are stored by GRP to:
65/alice/data/2007/LHC07a/<runNb>/Raw/RunMetadata.root
66
67- Shuttle sends a mail to DCS experts each time DP retrieval fails.
68
675f64cd 69Revision 1.61 2007/10/30 20:33:51 acolla
70Improved managing of temporary folders, which weren't correctly handled.
71Resolved bug introduced in StoreReferenceFile, which caused SPD preprocessor fail.
72
d524ade6 73Revision 1.60 2007/10/29 18:06:16 acolla
74
75New function StoreRunMetadataFile added to preprocessor and Shuttle interface
76This function can be used by GRP only. It stores raw data tags merged file to the
77raw data folder (e.g. /alice/data/2008/LHC08a/000099999/Raw).
78
79KNOWN ISSUES:
80
811. Shuttle cannot write to /alice/data/ because it belongs to alidaq. Tag file is stored in /alice/simulation/... for the time being.
822. Due to a bug in TAlien::Mkdir, the creation of a folder in recursive mode (-p option) does not work. The problem
83has been corrected in the root package on the Shuttle machine.
84
c88ad5db 85Revision 1.59 2007/10/05 12:40:55 acolla
86
87Result error code added to AliDCSClient data members (it was "lost" with the new implementation of TMap* GetAliasValues and GetDPValues).
88
1790d4b7 89Revision 1.58 2007/09/28 15:27:40 acolla
90
91AliDCSClient "multiSplit" option added in the DCS configuration
92in AliDCSMessage: variable MAX_BODY_SIZE set to 500000
93
542b6cc8 94Revision 1.57 2007/09/27 16:53:13 acolla
95Detectors can have more than one AMANDA server. SHUTTLE queries the servers sequentially,
96merges the dcs aliases/DPs in one TMap and sends it to the preprocessor.
97
1d172743 98Revision 1.56 2007/09/14 16:46:14 jgrosseo
991) Connect and Close are called before and after each query, so one can
100keep the same AliDCSClient object.
1012) The splitting of a query is moved to GetDPValues/GetAliasValues.
1023) Splitting interval can be specified in constructor
103
b41b252a 104Revision 1.55 2007/08/06 12:26:40 acolla
105Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
106read from the run logbook.
107
4859271b 108Revision 1.54 2007/07/12 09:51:25 jgrosseo
109removed duplicated log message in GetFile
110
4f0749a8 111Revision 1.53 2007/07/12 09:26:28 jgrosseo
112updating hlt fxs base path
113
42fde080 114Revision 1.52 2007/07/12 08:06:45 jgrosseo
115adding log messages in getfile... functions
116adding not implemented copy constructor in alishuttleconfigholder
117
1bcd28db 118Revision 1.51 2007/07/03 17:24:52 acolla
119root moved to v5-16-00. TFileMerger->Cp moved to TFile::Cp.
120
a986b218 121Revision 1.50 2007/07/02 17:19:32 acolla
122preprocessor is run in a temp directory that is removed when process is finished.
123
5bac2bde 124Revision 1.49 2007/06/29 10:45:06 acolla
125Number of columns in MySql Shuttle logbook increased by one (HLT added)
126
db99d43e 127Revision 1.48 2007/06/21 13:06:19 acolla
128GetFileSources returns dummy list with 1 source if system=DCS (better than
129returning error as it was)
130
6297b37d 131Revision 1.47 2007/06/19 17:28:56 acolla
132HLT updated; missing map bug removed.
133
dc25836b 134Revision 1.46 2007/06/09 13:01:09 jgrosseo
135Switching to retrieval of several DCS DPs at a time (multiDPrequest)
136
a038aa70 137Revision 1.45 2007/05/30 06:35:20 jgrosseo
138Adding functionality to the Shuttle/TestShuttle:
139o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
140o) Function to retrieve list of IDs for a given source (GetFileIDs)
141These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
142Example code has been added to the TestProcessor in TestShuttle
143
4a33bdd9 144Revision 1.44 2007/05/11 16:09:32 acolla
145Reference files for ITS, MUON and PHOS are now stored in OfflineDetName/OnlineDetName/run_...
146example: ITS/SPD/100_filename.root
147
2d9019b4 148Revision 1.43 2007/05/10 09:59:51 acolla
149Various bug fixes in StoreRefFilesToGrid; Cleaning of reference storage before processing detector (CleanReferenceStorage)
150
546242fb 151Revision 1.42 2007/05/03 08:01:39 jgrosseo
152typo in last commit :-(
153
8b739301 154Revision 1.41 2007/05/03 08:00:48 jgrosseo
155fixing log message when pp want to skip dcs value retrieval
156
651fdaab 157Revision 1.40 2007/04/27 07:06:48 jgrosseo
158GetFileSources returns empty list in case of no files, but successful query
159No mails sent in testmode
160
86aa42c3 161Revision 1.39 2007/04/17 12:43:57 acolla
162Correction in StoreOCDB; change of text in mail to detector expert
163
26758fce 164Revision 1.38 2007/04/12 08:26:18 jgrosseo
165updated comment
166
3c2a21c8 167Revision 1.37 2007/04/10 16:53:14 jgrosseo
168redirecting sub detector stdout, stderr to sub detector log file
169
3d8bc902 170Revision 1.35 2007/04/04 16:26:38 acolla
1711. Re-organization of function calls in TestPreprocessor to make it more meaningful.
1722. Added missing dependency in test preprocessors.
1733. in AliShuttle.cxx: processing time and memory consumption info on a single line.
174
886d60e6 175Revision 1.34 2007/04/04 10:33:36 jgrosseo
1761) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
177In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
178
1792) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
180
1813) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
182
1834) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
184
1855) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
186If you always need DCS data (like before), you do not need to implement it.
187
1886) The run type has been added to the monitoring page
189
9827400b 190Revision 1.33 2007/04/03 13:56:01 acolla
191Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
192run type.
193
3301427a 194Revision 1.32 2007/02/28 10:41:56 acolla
195Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
196AliPreprocessor::GetRunType() function.
197Added some ldap definition files.
198
d386d623 199Revision 1.30 2007/02/13 11:23:21 acolla
200Moved getters and setters of Shuttle's main OCDB/Reference, local
201OCDB/Reference, temp and log folders to AliShuttleInterface
202
9d733021 203Revision 1.27 2007/01/30 17:52:42 jgrosseo
204adding monalisa monitoring
205
e7f62f16 206Revision 1.26 2007/01/23 19:20:03 acolla
207Removed old ldif files, added TOF, MCH ldif files. Added some options in
208AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
209SetShuttleLogDir
210
36c99a6a 211Revision 1.25 2007/01/15 19:13:52 acolla
212Moved some AliInfo to AliDebug in SendMail function
213
fc5a4708 214Revision 1.21 2006/12/07 08:51:26 jgrosseo
215update (alberto):
216table, db names in ldap configuration
217added GRP preprocessor
218DCS data can also be retrieved by data point
219
2c15234c 220Revision 1.20 2006/11/16 16:16:48 jgrosseo
221introducing strict run ordering flag
222removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
223
be48e3ea 224Revision 1.19 2006/11/06 14:23:04 jgrosseo
225major update (Alberto)
226o) reading of run parameters from the logbook
227o) online offline naming conversion
228o) standalone DCSclient package
229
eba76848 230Revision 1.18 2006/10/20 15:22:59 jgrosseo
231o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
232o) Merging Collect, CollectAll, CollectNew function
233o) Removing implementation of empty copy constructors (declaration still there!)
234
cb343cfd 235Revision 1.17 2006/10/05 16:20:55 jgrosseo
236adapting to new CDB classes
237
6ec0e06c 238Revision 1.16 2006/10/05 15:46:26 jgrosseo
239applying to the new interface
240
481441a2 241Revision 1.15 2006/10/02 16:38:39 jgrosseo
242update (alberto):
243fixed memory leaks
244storing of objects that failed to be stored to the grid before
245interfacing of shuttle status table in daq system
246
2bb7b766 247Revision 1.14 2006/08/29 09:16:05 jgrosseo
248small update
249
85a80aa9 250Revision 1.13 2006/08/15 10:50:00 jgrosseo
251effc++ corrections (alberto)
252
4f0ab988 253Revision 1.12 2006/08/08 14:19:29 jgrosseo
254Update to shuttle classes (Alberto)
255
256- Possibility to set the full object's path in the Preprocessor's and
257Shuttle's Store functions
258- Possibility to extend the object's run validity in the same classes
259("startValidity" and "validityInfinite" parameters)
260- Implementation of the StoreReferenceData function to store reference
261data in a dedicated CDB storage.
262
84090f85 263Revision 1.11 2006/07/21 07:37:20 jgrosseo
264last run is stored after each run
265
7bfb2090 266Revision 1.10 2006/07/20 09:54:40 jgrosseo
267introducing status management: The processing per subdetector is divided into several steps,
268after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
269can keep track of the number of failures and skips further processing after a certain threshold is
270exceeded. These thresholds can be configured in LDAP.
271
5164a766 272Revision 1.9 2006/07/19 10:09:55 jgrosseo
273new configuration, accesst to DAQ FES (Alberto)
274
57f50b3c 275Revision 1.8 2006/07/11 12:44:36 jgrosseo
276adding parameters for extended validity range of data produced by preprocessor
277
17111222 278Revision 1.7 2006/07/10 14:37:09 jgrosseo
279small fix + todo comment
280
e090413b 281Revision 1.6 2006/07/10 13:01:41 jgrosseo
282enhanced storing of last sucessfully processed run (alberto)
283
a7160fe9 284Revision 1.5 2006/07/04 14:59:57 jgrosseo
285revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
286
45a493ce 287Revision 1.4 2006/06/12 09:11:16 jgrosseo
288coding conventions (Alberto)
289
58bc3020 290Revision 1.3 2006/06/06 14:26:40 jgrosseo
291o) removed files that were moved to STEER
292o) shuttle updated to follow the new interface (Alberto)
293
b948db8d 294Revision 1.2 2006/03/07 07:52:34 hristov
295New version (B.Yordanov)
296
d477ad88 297Revision 1.6 2005/11/19 17:19:14 byordano
298RetrieveDATEEntries and RetrieveConditionsData added
299
300Revision 1.5 2005/11/19 11:09:27 byordano
301AliShuttle declaration added
302
303Revision 1.4 2005/11/17 17:47:34 byordano
304TList changed to TObjArray
305
306Revision 1.3 2005/11/17 14:43:23 byordano
307import to local CVS
308
309Revision 1.1.1.1 2005/10/28 07:33:58 hristov
310Initial import as subdirectory in AliRoot
311
73abe331 312Revision 1.2 2005/09/13 08:41:15 byordano
313default startTime endTime added
314
315Revision 1.4 2005/08/30 09:13:02 byordano
316some docs added
317
318Revision 1.3 2005/08/29 21:15:47 byordano
319some docs added
320
321*/
322
323//
324// This class is the main manager for AliShuttle.
325// It organizes the data retrieval from DCS and call the
b948db8d 326// interface methods of AliPreprocessor.
73abe331 327// For every detector in AliShuttleConfgi (see AliShuttleConfig),
328// data for its set of aliases is retrieved. If there is registered
b948db8d 329// AliPreprocessor for this detector then it will be used
330// accroding to the schema (see AliPreprocessor).
331// If there isn't registered AliPreprocessor than the retrieved
73abe331 332// data is stored automatically to the undelying AliCDBStorage.
333// For detSpec is used the alias name.
334//
335
336#include "AliShuttle.h"
337
338#include "AliCDBManager.h"
339#include "AliCDBStorage.h"
340#include "AliCDBId.h"
84090f85 341#include "AliCDBRunRange.h"
342#include "AliCDBPath.h"
5164a766 343#include "AliCDBEntry.h"
73abe331 344#include "AliShuttleConfig.h"
eba76848 345#include "DCSClient/AliDCSClient.h"
73abe331 346#include "AliLog.h"
b948db8d 347#include "AliPreprocessor.h"
5164a766 348#include "AliShuttleStatus.h"
2bb7b766 349#include "AliShuttleLogbookEntry.h"
73abe331 350
57f50b3c 351#include <TSystem.h>
58bc3020 352#include <TObject.h>
b948db8d 353#include <TString.h>
57f50b3c 354#include <TTimeStamp.h>
73abe331 355#include <TObjString.h>
57f50b3c 356#include <TSQLServer.h>
357#include <TSQLResult.h>
358#include <TSQLRow.h>
cb343cfd 359#include <TMutex.h>
9827400b 360#include <TSystemDirectory.h>
361#include <TSystemFile.h>
a986b218 362#include <TFile.h>
9827400b 363#include <TGrid.h>
364#include <TGridResult.h>
73abe331 365
e7f62f16 366#include <TMonaLisaWriter.h>
367
5164a766 368#include <fstream>
369
cb343cfd 370#include <sys/types.h>
371#include <sys/wait.h>
372
73abe331 373ClassImp(AliShuttle)
374
b948db8d 375//______________________________________________________________________________________________
376AliShuttle::AliShuttle(const AliShuttleConfig* config,
377 UInt_t timeout, Int_t retries):
4f0ab988 378fConfig(config),
379fTimeout(timeout), fRetries(retries),
380fPreprocessorMap(),
2bb7b766 381fLogbookEntry(0),
eba76848 382fCurrentDetector(),
85a80aa9 383fStatusEntry(0),
cb343cfd 384fMonitoringMutex(0),
eba76848 385fLastActionTime(0),
e7f62f16 386fLastAction(),
9827400b 387fMonaLisa(0),
388fTestMode(kNone),
ffa29e93 389fReadTestMode(kFALSE),
390fOutputRedirected(kFALSE)
73abe331 391{
392 //
393 // config: AliShuttleConfig used
73abe331 394 // timeout: timeout used for AliDCSClient connection
395 // retries: the number of retries in case of connection error.
396 //
397
57f50b3c 398 if (!fConfig->IsValid()) AliFatal("********** !!!!! Invalid configuration !!!!! **********");
be48e3ea 399 for(int iSys=0;iSys<4;iSys++) {
57f50b3c 400 fServer[iSys]=0;
be48e3ea 401 if (iSys < 3)
2c15234c 402 fFXSlist[iSys].SetOwner(kTRUE);
57f50b3c 403 }
2bb7b766 404 fPreprocessorMap.SetOwner(kTRUE);
be48e3ea 405
406 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
407 fFirstUnprocessed[iDet] = kFALSE;
408
cb343cfd 409 fMonitoringMutex = new TMutex();
58bc3020 410}
411
b948db8d 412//______________________________________________________________________________________________
57f50b3c 413AliShuttle::~AliShuttle()
58bc3020 414{
9827400b 415 //
416 // destructor
417 //
58bc3020 418
b948db8d 419 fPreprocessorMap.DeleteAll();
be48e3ea 420 for(int iSys=0;iSys<4;iSys++)
57f50b3c 421 if(fServer[iSys]) {
422 fServer[iSys]->Close();
423 delete fServer[iSys];
eba76848 424 fServer[iSys] = 0;
57f50b3c 425 }
2bb7b766 426
427 if (fStatusEntry){
428 delete fStatusEntry;
429 fStatusEntry = 0;
430 }
cb343cfd 431
432 if (fMonitoringMutex)
433 {
434 delete fMonitoringMutex;
435 fMonitoringMutex = 0;
436 }
73abe331 437}
438
b948db8d 439//______________________________________________________________________________________________
57f50b3c 440void AliShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
58bc3020 441{
73abe331 442 //
b948db8d 443 // Registers new AliPreprocessor.
73abe331 444 // It uses GetName() for indentificator of the pre processor.
445 // The pre processor is registered it there isn't any other
446 // with the same identificator (GetName()).
447 //
448
eba76848 449 const char* detName = preprocessor->GetName();
450 if(GetDetPos(detName) < 0)
451 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
452
453 if (fPreprocessorMap.GetValue(detName)) {
454 AliWarning(Form("AliPreprocessor %s is already registered!", detName));
73abe331 455 return;
456 }
457
eba76848 458 fPreprocessorMap.Add(new TObjString(detName), preprocessor);
73abe331 459}
b948db8d 460//______________________________________________________________________________________________
3301427a 461Bool_t AliShuttle::Store(const AliCDBPath& path, TObject* object,
84090f85 462 AliCDBMetaData* metaData, Int_t validityStart, Bool_t validityInfinite)
73abe331 463{
9827400b 464 // Stores a CDB object in the storage for offline reconstruction. Objects that are not needed for
465 // offline reconstruction, but should be stored anyway (e.g. for debugging) should NOT be stored
466 // using this function. Use StoreReferenceData instead!
467 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
468 // finishes the data are transferred to the main storage (Grid).
b948db8d 469
3301427a 470 return StoreLocally(fgkLocalCDB, path, object, metaData, validityStart, validityInfinite);
84090f85 471}
472
473//______________________________________________________________________________________________
3301427a 474Bool_t AliShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 475{
9827400b 476 // Stores a CDB object in the storage for reference data. This objects will not be available during
477 // offline reconstrunction. Use this function for reference data only!
478 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
479 // finishes the data are transferred to the main storage (Grid).
85a80aa9 480
3301427a 481 return StoreLocally(fgkLocalRefStorage, path, object, metaData);
85a80aa9 482}
483
484//______________________________________________________________________________________________
3301427a 485Bool_t AliShuttle::StoreLocally(const TString& localUri,
85a80aa9 486 const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
487 Int_t validityStart, Bool_t validityInfinite)
488{
9827400b 489 // Store object temporarily in local storage. Parameters are passed by Store and StoreReferenceData functions.
490 // when the preprocessor finishes the data are transferred to the main storage (Grid).
491 // The parameters are:
492 // 1) Uri of the backup storage (Local)
493 // 2) the object's path.
494 // 3) the object to be stored
495 // 4) the metaData to be associated with the object
496 // 5) the validity start run number w.r.t. the current run,
497 // if the data is valid only for this run leave the default 0
498 // 6) specifies if the calibration data is valid for infinity (this means until updated),
499 // typical for calibration runs, the default is kFALSE
500 //
501 // returns 0 if fail, 1 otherwise
84090f85 502
9827400b 503 if (fTestMode & kErrorStorage)
504 {
505 Log(fCurrentDetector, "StoreLocally - In TESTMODE - Simulating error while storing locally");
506 return kFALSE;
507 }
508
3301427a 509 const char* cdbType = (localUri == fgkLocalCDB) ? "CDB" : "Reference";
2bb7b766 510
85a80aa9 511 Int_t firstRun = GetCurrentRun() - validityStart;
84090f85 512 if(firstRun < 0) {
9827400b 513 AliWarning("First valid run happens to be less than 0! Setting it to 0.");
84090f85 514 firstRun=0;
515 }
516
517 Int_t lastRun = -1;
518 if(validityInfinite) {
519 lastRun = AliCDBRunRange::Infinity();
520 } else {
521 lastRun = GetCurrentRun();
522 }
523
3301427a 524 // Version is set to current run, it will be used later to transfer data to Grid
525 AliCDBId id(path, firstRun, lastRun, GetCurrentRun(), -1);
2bb7b766 526
527 if(! dynamic_cast<TObjString*> (metaData->GetProperty("RunUsed(TObjString)"))){
528 TObjString runUsed = Form("%d", GetCurrentRun());
9e080f92 529 metaData->SetProperty("RunUsed(TObjString)", runUsed.Clone());
2bb7b766 530 }
84090f85 531
3301427a 532 Bool_t result = kFALSE;
84090f85 533
3301427a 534 if (!(AliCDBManager::Instance()->GetStorage(localUri))) {
535 Log("SHUTTLE", Form("StoreLocally - Cannot activate local %s storage", cdbType));
84090f85 536 } else {
3301427a 537 result = AliCDBManager::Instance()->GetStorage(localUri)
84090f85 538 ->Put(object, id, metaData);
539 }
540
541 if(!result) {
542
9827400b 543 Log(fCurrentDetector, Form("StoreLocally - Can't store object <%s>!", id.ToString().Data()));
3301427a 544 }
2bb7b766 545
3301427a 546 return result;
547}
84090f85 548
3301427a 549//______________________________________________________________________________________________
550Bool_t AliShuttle::StoreOCDB()
551{
9827400b 552 //
553 // Called when preprocessor ends successfully or when previous storage attempt failed (kStoreError status)
554 // Calls underlying StoreOCDB(const char*) function twice, for OCDB and Reference storage.
555 // Then calls StoreRefFilesToGrid to store reference files.
556 //
557
558 if (fTestMode & kErrorGrid)
559 {
560 Log("SHUTTLE", "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
561 Log(fCurrentDetector, "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
562 return kFALSE;
563 }
564
c88ad5db 565 Log("SHUTTLE","StoreOCDB - Storing OCDB data ...");
86aa42c3 566 Bool_t resultCDB = StoreOCDB(fgkMainCDB);
567
c88ad5db 568 Log("SHUTTLE","StoreOCDB - Storing reference data ...");
3301427a 569 Bool_t resultRef = StoreOCDB(fgkMainRefStorage);
9827400b 570
c88ad5db 571 Log("SHUTTLE","StoreOCDB - Storing reference files ...");
572 Bool_t resultRefFiles = CopyFilesToGrid("reference");
573
574 Bool_t resultMetadata = kTRUE;
575 if(fCurrentDetector == "GRP")
576 {
577 Log("StoreOCDB - SHUTTLE","Storing Run Metadata file ...");
578 resultMetadata = CopyFilesToGrid("metadata");
579 }
9827400b 580
c88ad5db 581 return resultCDB && resultRef && resultRefFiles && resultMetadata;
3301427a 582}
583
584//______________________________________________________________________________________________
585Bool_t AliShuttle::StoreOCDB(const TString& gridURI)
586{
587 //
588 // Called by StoreOCDB(), performs actual storage to the main OCDB and reference storages (Grid)
589 //
590
591 TObjArray* gridIds=0;
592
593 Bool_t result = kTRUE;
594
595 const char* type = 0;
596 TString localURI;
597 if(gridURI == fgkMainCDB) {
598 type = "OCDB";
599 localURI = fgkLocalCDB;
600 } else if(gridURI == fgkMainRefStorage) {
601 type = "reference";
602 localURI = fgkLocalRefStorage;
603 } else {
604 AliError(Form("Invalid storage URI: %s", gridURI.Data()));
605 return kFALSE;
606 }
607
608 AliCDBManager* man = AliCDBManager::Instance();
609
610 AliCDBStorage *gridSto = man->GetStorage(gridURI);
611 if(!gridSto) {
612 Log("SHUTTLE",
613 Form("StoreOCDB - cannot activate main %s storage", type));
614 return kFALSE;
615 }
616
617 gridIds = gridSto->GetQueryCDBList();
618
619 // get objects previously stored in local CDB
620 AliCDBStorage *localSto = man->GetStorage(localURI);
621 if(!localSto) {
622 Log("SHUTTLE",
623 Form("StoreOCDB - cannot activate local %s storage", type));
624 return kFALSE;
625 }
626 AliCDBPath aPath(GetOfflineDetName(fCurrentDetector.Data()),"*","*");
627 // Local objects were stored with current run as Grid version!
628 TList* localEntries = localSto->GetAll(aPath.GetPath(), GetCurrentRun(), GetCurrentRun());
629 localEntries->SetOwner(1);
630
631 // loop on local stored objects
632 TIter localIter(localEntries);
633 AliCDBEntry *aLocEntry = 0;
634 while((aLocEntry = dynamic_cast<AliCDBEntry*> (localIter.Next()))){
635 aLocEntry->SetOwner(1);
636 AliCDBId aLocId = aLocEntry->GetId();
637 aLocEntry->SetVersion(-1);
638 aLocEntry->SetSubVersion(-1);
639
640 // If local object is valid up to infinity we store it only if it is
641 // the first unprocessed run!
642 if (aLocId.GetLastRun() == AliCDBRunRange::Infinity() &&
643 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
644 {
645 Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
646 "there are previous unprocessed runs!",
647 fCurrentDetector.Data(), aLocId.GetPath().Data()));
648 continue;
649 }
650
651 // loop on Grid valid Id's
652 Bool_t store = kTRUE;
653 TIter gridIter(gridIds);
654 AliCDBId* aGridId = 0;
655 while((aGridId = dynamic_cast<AliCDBId*> (gridIter.Next()))){
656 if(aGridId->GetPath() != aLocId.GetPath()) continue;
657 // skip all objects valid up to infinity
658 if(aGridId->GetLastRun() == AliCDBRunRange::Infinity()) continue;
659 // if we get here, it means there's already some more recent object stored on Grid!
660 store = kFALSE;
661 break;
662 }
663
664 // If we get here, the file can be stored!
665 Bool_t storeOk = gridSto->Put(aLocEntry);
666 if(!store || storeOk){
667
668 if (!store)
669 {
670 Log(fCurrentDetector.Data(),
671 Form("StoreOCDB - A more recent object already exists in %s storage: <%s>",
672 type, aGridId->ToString().Data()));
673 } else {
674 Log("SHUTTLE",
675 Form("StoreOCDB - Object <%s> successfully put into %s storage",
676 aLocId.ToString().Data(), type));
2d9019b4 677 Log(fCurrentDetector.Data(),
678 Form("StoreOCDB - Object <%s> successfully put into %s storage",
679 aLocId.ToString().Data(), type));
3301427a 680 }
84090f85 681
3301427a 682 // removing local filename...
683 TString filename;
684 localSto->IdToFilename(aLocId, filename);
c88ad5db 685 Log("SHUTTLE", Form("StoreOCDB - Removing local file %s", filename.Data()));
3301427a 686 RemoveFile(filename.Data());
687 continue;
688 } else {
689 Log("SHUTTLE",
690 Form("StoreOCDB - Grid %s storage of object <%s> failed",
691 type, aLocId.ToString().Data()));
2d9019b4 692 Log(fCurrentDetector.Data(),
693 Form("StoreOCDB - Grid %s storage of object <%s> failed",
694 type, aLocId.ToString().Data()));
3301427a 695 result = kFALSE;
b948db8d 696 }
697 }
3301427a 698 localEntries->Clear();
2bb7b766 699
b948db8d 700 return result;
3301427a 701}
702
546242fb 703//______________________________________________________________________________________________
704Bool_t AliShuttle::CleanReferenceStorage(const char* detector)
705{
2d9019b4 706 // clears the directory used to store reference files of a given subdetector
546242fb 707
708 AliCDBManager* man = AliCDBManager::Instance();
709 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
2d9019b4 710 TString localBaseFolder = sto->GetBaseFolder();
711
712 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
713
d524ade6 714 Log("SHUTTLE", Form("CleanReferenceStorage - Cleaning %s", targetDir.Data()));
2d9019b4 715
716 TString begin;
717 begin.Form("%d_", GetCurrentRun());
718
719 TSystemDirectory* baseDir = new TSystemDirectory("/", targetDir);
720 if (!baseDir)
721 return kTRUE;
722
723 TList* dirList = baseDir->GetListOfFiles();
724 delete baseDir;
725
726 if (!dirList) return kTRUE;
727
728 if (dirList->GetEntries() < 3)
729 {
730 delete dirList;
731 return kTRUE;
732 }
733
734 Int_t nDirs = 0, nDel = 0;
735 TIter dirIter(dirList);
736 TSystemFile* entry = 0;
546242fb 737
2d9019b4 738 Bool_t success = kTRUE;
546242fb 739
2d9019b4 740 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
741 {
742 if (entry->IsDirectory())
743 continue;
744
745 TString fileName(entry->GetName());
746 if (!fileName.BeginsWith(begin))
747 continue;
748
749 nDirs++;
750
751 // delete file
752 Int_t result = gSystem->Unlink(fileName.Data());
753
754 if (result)
755 {
d524ade6 756 Log("SHUTTLE", Form("CleanReferenceStorage - Could not delete file %s!", fileName.Data()));
2d9019b4 757 success = kFALSE;
758 } else {
759 nDel++;
760 }
761 }
762
763 if(nDirs > 0)
764 Log("SHUTTLE", Form("CleanReferenceStorage - %d (over %d) reference files in folder %s were deleted.",
765 nDel, nDirs, targetDir.Data()));
766
767
768 delete dirList;
769 return success;
770
771
772
773
774
546242fb 775
776 Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
777 if (result == 0)
778 {
779 // delete directory
d524ade6 780 result = gSystem->Exec(Form("rm -rf %s", targetDir.Data()));
546242fb 781 if (result != 0)
782 {
d524ade6 783 Log("SHUTTLE", Form("CleanReferenceStorage - Could not clean directory %s", targetDir.Data()));
546242fb 784 return kFALSE;
785 }
786 }
787
788 result = gSystem->mkdir(targetDir, kTRUE);
789 if (result != 0)
790 {
c88ad5db 791 Log("SHUTTLE", Form("CleanReferenceStorage - Error creating base directory %s", targetDir.Data()));
546242fb 792 return kFALSE;
793 }
794
795 return kTRUE;
796}
797
9827400b 798//______________________________________________________________________________________________
799Bool_t AliShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
800{
801 //
3c2a21c8 802 // Stores reference file directly (without opening it). This function stores the file locally.
9827400b 803 //
3c2a21c8 804 // The file is stored under the following location:
805 // <base folder of local reference storage>/<DET>/<RUN#>_<gridFileName>
806 // where <gridFileName> is the second parameter given to the function
807 //
9827400b 808
809 if (fTestMode & kErrorStorage)
810 {
811 Log(fCurrentDetector, "StoreReferenceFile - In TESTMODE - Simulating error while storing locally");
812 return kFALSE;
813 }
814
815 AliCDBManager* man = AliCDBManager::Instance();
816 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
817
818 TString localBaseFolder = sto->GetBaseFolder();
819
d524ade6 820 TString target = GetRefFilePrefix(localBaseFolder.Data(), detector);
821 target.Append(Form("/%d_%s", GetCurrentRun(), gridFileName));
9827400b 822
d524ade6 823 return CopyFileLocally(localFile, target);
c88ad5db 824}
825
826//______________________________________________________________________________________________
827Bool_t AliShuttle::StoreRunMetadataFile(const char* localFile, const char* gridFileName)
828{
829 //
830 // Stores Run metadata file to the Grid, in the run folder
831 //
832 // Only GRP can call this function.
833
834 if (fTestMode & kErrorStorage)
835 {
836 Log(fCurrentDetector, "StoreRunMetaDataFile - In TESTMODE - Simulating error while storing locally");
837 return kFALSE;
838 }
839
840 AliCDBManager* man = AliCDBManager::Instance();
841 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
842
843 TString localBaseFolder = sto->GetBaseFolder();
844
845 // Build Run level folder
846 // folder = /alice/data/year/lhcPeriod/runNb/Raw
847
c88ad5db 848
675f64cd 849 TString lhcPeriod = GetLHCPeriod();
c88ad5db 850 if (lhcPeriod.Length() == 0)
851 {
852 Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
853 return 0;
854 }
855
675f64cd 856 TString target = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw/%s",
857 localBaseFolder.Data(), GetCurrentYear(),
d524ade6 858 lhcPeriod.Data(), GetCurrentRun(), gridFileName);
c88ad5db 859
d524ade6 860 return CopyFileLocally(localFile, target);
c88ad5db 861}
862
863//______________________________________________________________________________________________
d524ade6 864Bool_t AliShuttle::CopyFileLocally(const char* localFile, const TString& target)
c88ad5db 865{
866 //
867 // Stores file locally. Called by StoreReferenceFile and StoreRunMetadataFile
d524ade6 868 // Files are temporarily stored in the local reference storage. When the preprocessor
869 // finishes, the Shuttle calls CopyFilesToGrid to transfer the files to AliEn
870 // (in reference or run level folders)
c88ad5db 871 //
872
d524ade6 873 TString targetDir(target(0, target.Last('/')));
874
875 //try to open base dir folder, if it does not exist
2d9019b4 876 void* dir = gSystem->OpenDirectory(targetDir.Data());
877 if (dir == NULL) {
878 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
c88ad5db 879 Log("SHUTTLE", Form("StoreFileLocally - Can't open directory <%s>", targetDir.Data()));
2d9019b4 880 return kFALSE;
881 }
882
883 } else {
884 gSystem->FreeDirectory(dir);
885 }
9827400b 886
7d43a416 887 Int_t result = 0;
888
889 result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
9827400b 890 if (result)
891 {
c88ad5db 892 Log("SHUTTLE", Form("StoreFileLocally - %s does not exist", localFile));
546242fb 893 return kFALSE;
9827400b 894 }
546242fb 895
7d43a416 896 result = gSystem->GetPathInfo(target, 0, (Long64_t*) 0, 0, 0);
897 if (!result)
898 {
899 Log("SHUTTLE", Form("StoreFileLocally - target file %s already exist, removing...", target.Data()));
900 if (gSystem->Unlink(target.Data()))
901 {
902 Log("SHUTTLE", Form("StoreFileLocally - Could not remove existing target file %s!", target.Data()));
903 return kFALSE;
904 }
905 }
906
9827400b 907 result = gSystem->CopyFile(localFile, target);
908
909 if (result == 0)
910 {
c88ad5db 911 Log("SHUTTLE", Form("StoreFileLocally - File %s stored locally to %s", localFile, target.Data()));
9827400b 912 return kTRUE;
913 }
914 else
915 {
d524ade6 916 Log("SHUTTLE", Form("StoreFileLocally - Could not store file %s to %s! Error code = %d",
546242fb 917 localFile, target.Data(), result));
9827400b 918 return kFALSE;
919 }
c88ad5db 920
921
922
9827400b 923}
924
925//______________________________________________________________________________________________
c88ad5db 926Bool_t AliShuttle::CopyFilesToGrid(const char* type)
9827400b 927{
928 //
c88ad5db 929 // Transfers local files to the Grid. Local files can be reference files
930 // or run metadata file (from GRP only).
9827400b 931 //
c88ad5db 932 // According to the type (ref, metadata) the files are stored under the following location:
933 // ref --> <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
934 // metadata --> <run data folder>/<MetadataFileName>
86aa42c3 935 //
c88ad5db 936
9827400b 937 AliCDBManager* man = AliCDBManager::Instance();
938 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
939 if (!sto)
940 return kFALSE;
941 TString localBaseFolder = sto->GetBaseFolder();
9827400b 942
c88ad5db 943 TString dir;
944 TString alienDir;
9827400b 945 TString begin;
9827400b 946
c88ad5db 947 if (strcmp(type, "reference") == 0)
948 {
949 dir = GetRefFilePrefix(localBaseFolder.Data(), fCurrentDetector.Data());
950 AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
951 if (!gridSto)
952 return kFALSE;
953 TString gridBaseFolder = gridSto->GetBaseFolder();
954 alienDir = GetRefFilePrefix(gridBaseFolder.Data(), fCurrentDetector.Data());
955 begin = Form("%d_", GetCurrentRun());
956 }
957 else if (strcmp(type, "metadata") == 0)
958 {
c88ad5db 959
675f64cd 960 TString lhcPeriod = GetLHCPeriod();
c88ad5db 961
962 if (lhcPeriod.Length() == 0)
963 {
964 Log("SHUTTLE","CopyFilesToGrid - LHCPeriod not found in logbook!");
965 return 0;
966 }
967
675f64cd 968 dir = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw",
969 localBaseFolder.Data(), GetCurrentYear(),
c88ad5db 970 lhcPeriod.Data(), GetCurrentRun());
675f64cd 971 alienDir = dir(dir.Index("/alice/data/"), dir.Length());
972
c88ad5db 973 begin = "";
974 }
975 else
976 {
977 Log("SHUTTLE", "CopyFilesToGrid - Unexpected: type label must be reference or metadata!");
978 return kFALSE;
979 }
980
9827400b 981 TSystemDirectory* baseDir = new TSystemDirectory("/", dir);
3d8bc902 982 if (!baseDir)
983 return kTRUE;
984
2d9019b4 985 TList* dirList = baseDir->GetListOfFiles();
986 delete baseDir;
987
988 if (!dirList) return kTRUE;
989
990 if (dirList->GetEntries() < 3)
3d8bc902 991 {
2d9019b4 992 delete dirList;
9827400b 993 return kTRUE;
3d8bc902 994 }
2d9019b4 995
546242fb 996 if (!gGrid)
997 {
c88ad5db 998 Log("SHUTTLE", "CopyFilesToGrid - Connection to Grid failed: Cannot continue!");
2d9019b4 999 delete dirList;
546242fb 1000 return kFALSE;
1001 }
1002
2d9019b4 1003 Int_t nDirs = 0, nTransfer = 0;
1004 TIter dirIter(dirList);
1005 TSystemFile* entry = 0;
1006
9827400b 1007 Bool_t success = kTRUE;
3d8bc902 1008 Bool_t first = kTRUE;
9827400b 1009
2d9019b4 1010 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
1011 {
9827400b 1012 if (entry->IsDirectory())
1013 continue;
1014
1015 TString fileName(entry->GetName());
1016 if (!fileName.BeginsWith(begin))
1017 continue;
1018
2d9019b4 1019 nDirs++;
1020
3d8bc902 1021 if (first)
1022 {
1023 first = kFALSE;
c88ad5db 1024 // check that folder exists, otherwise create it
3d8bc902 1025 TGridResult* result = gGrid->Ls(alienDir.Data(), "a");
1026
1027 if (!result)
2d9019b4 1028 {
1029 delete dirList;
3d8bc902 1030 return kFALSE;
2d9019b4 1031 }
3d8bc902 1032
546242fb 1033 if (!result->GetFileName(1)) // TODO: It looks like element 0 is always 0!!
3d8bc902 1034 {
675f64cd 1035 // TODO It does not work currently! Bug in TAliEn::Mkdir
1036 // TODO Manually fixed in local root v5-16-00
c88ad5db 1037 if (!gGrid->Mkdir(alienDir.Data(),"-p",0))
3d8bc902 1038 {
c88ad5db 1039 Log("SHUTTLE", Form("CopyFilesToGrid - Cannot create directory %s",
3d8bc902 1040 alienDir.Data()));
2d9019b4 1041 delete dirList;
3d8bc902 1042 return kFALSE;
546242fb 1043 } else {
c88ad5db 1044 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s created", alienDir.Data()));
3d8bc902 1045 }
1046
546242fb 1047 } else {
c88ad5db 1048 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s found", alienDir.Data()));
3d8bc902 1049 }
1050 }
1051
9827400b 1052 TString fullLocalPath;
1053 fullLocalPath.Form("%s/%s", dir.Data(), fileName.Data());
1054
1055 TString fullGridPath;
1056 fullGridPath.Form("alien://%s/%s", alienDir.Data(), fileName.Data());
1057
a986b218 1058 Bool_t result = TFile::Cp(fullLocalPath, fullGridPath);
9827400b 1059
1060 if (result)
1061 {
c88ad5db 1062 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s succeeded!",
1063 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1064 RemoveFile(fullLocalPath);
2d9019b4 1065 nTransfer++;
9827400b 1066 }
1067 else
1068 {
c88ad5db 1069 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s FAILED!",
1070 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1071 success = kFALSE;
1072 }
1073 }
2d9019b4 1074
c88ad5db 1075 Log("SHUTTLE", Form("CopyFilesToGrid - %d (over %d) files in folder %s copied to Grid.",
1076 nTransfer, nDirs, dir.Data()));
2d9019b4 1077
1078
1079 delete dirList;
9827400b 1080 return success;
1081}
1082
2d9019b4 1083//______________________________________________________________________________________________
1084const char* AliShuttle::GetRefFilePrefix(const char* base, const char* detector)
1085{
1086 //
1087 // Get folder name of reference files
1088 //
1089
1090 TString offDetStr(GetOfflineDetName(detector));
1091 TString dir;
1092 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
1093 {
1094 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
1095 } else {
1096 dir.Form("%s/%s", base, offDetStr.Data());
1097 }
1098
1099 return dir.Data();
1100
1101
1102}
c88ad5db 1103
3301427a 1104//______________________________________________________________________________________________
1105void AliShuttle::CleanLocalStorage(const TString& uri)
1106{
9827400b 1107 //
1108 // Called in case the preprocessor is declared failed. Remove remaining objects from the local storages.
1109 //
3301427a 1110
1111 const char* type = 0;
1112 if(uri == fgkLocalCDB) {
1113 type = "OCDB";
1114 } else if(uri == fgkLocalRefStorage) {
546242fb 1115 type = "Reference";
3301427a 1116 } else {
1117 AliError(Form("Invalid storage URI: %s", uri.Data()));
1118 return;
1119 }
1120
1121 AliCDBManager* man = AliCDBManager::Instance();
b948db8d 1122
3301427a 1123 // open local storage
1124 AliCDBStorage *localSto = man->GetStorage(uri);
1125 if(!localSto) {
1126 Log("SHUTTLE",
1127 Form("CleanLocalStorage - cannot activate local %s storage", type));
1128 return;
1129 }
1130
1131 TString filename(Form("%s/%s/*/Run*_v%d_s*.root",
546242fb 1132 localSto->GetBaseFolder().Data(), GetOfflineDetName(fCurrentDetector.Data()), GetCurrentRun()));
3301427a 1133
c88ad5db 1134 AliDebug(2, Form("filename = %s", filename.Data()));
3301427a 1135
c88ad5db 1136 Log("SHUTTLE", Form("Removing remaining local files for run %d and detector %s ...",
3301427a 1137 GetCurrentRun(), fCurrentDetector.Data()));
1138
1139 RemoveFile(filename.Data());
1140
1141}
1142
1143//______________________________________________________________________________________________
1144void AliShuttle::RemoveFile(const char* filename)
1145{
9827400b 1146 //
1147 // removes local file
1148 //
3301427a 1149
1150 TString command(Form("rm -f %s", filename));
1151
1152 Int_t result = gSystem->Exec(command.Data());
1153 if(result != 0)
1154 {
1155 Log("SHUTTLE", Form("RemoveFile - %s: Cannot remove file %s!",
1156 fCurrentDetector.Data(), filename));
1157 }
73abe331 1158}
1159
b948db8d 1160//______________________________________________________________________________________________
5164a766 1161AliShuttleStatus* AliShuttle::ReadShuttleStatus()
1162{
9827400b 1163 //
1164 // Reads the AliShuttleStatus from the CDB
1165 //
5164a766 1166
2bb7b766 1167 if (fStatusEntry){
1168 delete fStatusEntry;
1169 fStatusEntry = 0;
1170 }
5164a766 1171
10a5a932 1172 fStatusEntry = AliCDBManager::Instance()->GetStorage(GetLocalCDB())
2bb7b766 1173 ->Get(Form("/SHUTTLE/STATUS/%s", fCurrentDetector.Data()), GetCurrentRun());
5164a766 1174
2bb7b766 1175 if (!fStatusEntry) return 0;
1176 fStatusEntry->SetOwner(1);
5164a766 1177
2bb7b766 1178 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1179 if (!status) {
1180 AliError("Invalid object stored to CDB!");
1181 return 0;
1182 }
5164a766 1183
2bb7b766 1184 return status;
5164a766 1185}
1186
1187//______________________________________________________________________________________________
7bfb2090 1188Bool_t AliShuttle::WriteShuttleStatus(AliShuttleStatus* status)
5164a766 1189{
9827400b 1190 //
1191 // writes the status for one subdetector
1192 //
2bb7b766 1193
1194 if (fStatusEntry){
1195 delete fStatusEntry;
1196 fStatusEntry = 0;
1197 }
5164a766 1198
2bb7b766 1199 Int_t run = GetCurrentRun();
5164a766 1200
2bb7b766 1201 AliCDBId id(AliCDBPath("SHUTTLE", "STATUS", fCurrentDetector), run, run);
5164a766 1202
2bb7b766 1203 fStatusEntry = new AliCDBEntry(status, id, new AliCDBMetaData);
1204 fStatusEntry->SetOwner(1);
5164a766 1205
2bb7b766 1206 UInt_t result = AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
7bfb2090 1207
2bb7b766 1208 if (!result) {
3301427a 1209 Log("SHUTTLE", Form("WriteShuttleStatus - Failed for %s, run %d",
1210 fCurrentDetector.Data(), run));
2bb7b766 1211 return kFALSE;
1212 }
e7f62f16 1213
1214 SendMLInfo();
7bfb2090 1215
2bb7b766 1216 return kTRUE;
5164a766 1217}
1218
1219//______________________________________________________________________________________________
1220void AliShuttle::UpdateShuttleStatus(AliShuttleStatus::Status newStatus, Bool_t increaseCount)
1221{
9827400b 1222 //
1223 // changes the AliShuttleStatus for the given detector and run to the given status
1224 //
5164a766 1225
2bb7b766 1226 if (!fStatusEntry){
1227 AliError("UNEXPECTED: fStatusEntry empty");
1228 return;
1229 }
5164a766 1230
2bb7b766 1231 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
5164a766 1232
2bb7b766 1233 if (!status){
c88ad5db 1234 Log("SHUTTLE", "UpdateShuttleStatus - UNEXPECTED: status could not be read from current CDB entry");
2bb7b766 1235 return;
1236 }
5164a766 1237
2c15234c 1238 TString actionStr = Form("UpdateShuttleStatus - %s: Changing state from %s to %s",
eba76848 1239 fCurrentDetector.Data(),
36c99a6a 1240 status->GetStatusName(),
eba76848 1241 status->GetStatusName(newStatus));
cb343cfd 1242 Log("SHUTTLE", actionStr);
1243 SetLastAction(actionStr);
5164a766 1244
2bb7b766 1245 status->SetStatus(newStatus);
1246 if (increaseCount) status->IncreaseCount();
5164a766 1247
2bb7b766 1248 AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
e7f62f16 1249
1250 SendMLInfo();
5164a766 1251}
e7f62f16 1252
1253//______________________________________________________________________________________________
1254void AliShuttle::SendMLInfo()
1255{
1256 //
1257 // sends ML information about the current status of the current detector being processed
1258 //
1259
1260 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1261
1262 if (!status){
3301427a 1263 Log("SHUTTLE", "SendMLInfo - UNEXPECTED: status could not be read from current CDB entry");
e7f62f16 1264 return;
1265 }
1266
1267 TMonaLisaText mlStatus(Form("%s_status", fCurrentDetector.Data()), status->GetStatusName());
1268 TMonaLisaValue mlRetryCount(Form("%s_count", fCurrentDetector.Data()), status->GetCount());
1269
1270 TList mlList;
1271 mlList.Add(&mlStatus);
1272 mlList.Add(&mlRetryCount);
1273
ee6f7523 1274 TString mlID;
1275 mlID.Form("%d", GetCurrentRun());
1276 fMonaLisa->SendParameters(&mlList, mlID);
e7f62f16 1277}
1278
5164a766 1279//______________________________________________________________________________________________
1280Bool_t AliShuttle::ContinueProcessing()
1281{
9827400b 1282 // this function reads the AliShuttleStatus information from CDB and
1283 // checks if the processing should be continued
1284 // if yes it returns kTRUE and updates the AliShuttleStatus with nextStatus
2bb7b766 1285
57c1a579 1286 if (!fConfig->HostProcessDetector(fCurrentDetector)) return kFALSE;
1287
1288 AliPreprocessor* aPreprocessor =
1289 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1290 if (!aPreprocessor)
1291 {
c88ad5db 1292 Log("SHUTTLE", Form("ContinueProcessing - %s: no preprocessor registered", fCurrentDetector.Data()));
57c1a579 1293 return kFALSE;
1294 }
1295
2bb7b766 1296 AliShuttleLogbookEntry::Status entryStatus =
eba76848 1297 fLogbookEntry->GetDetectorStatus(fCurrentDetector);
2bb7b766 1298
1299 if(entryStatus != AliShuttleLogbookEntry::kUnprocessed) {
c88ad5db 1300 Log("SHUTTLE", Form("ContinueProcessing - %s is %s",
2bb7b766 1301 fCurrentDetector.Data(),
1302 fLogbookEntry->GetDetectorStatusName(entryStatus)));
1303 return kFALSE;
1304 }
1305
1306 // if we get here, according to Shuttle logbook subdetector is in UNPROCESSED state
be48e3ea 1307
1308 // check if current run is first unprocessed run for current detector
1309 if (fConfig->StrictRunOrder(fCurrentDetector) &&
1310 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
1311 {
86aa42c3 1312 if (fTestMode == kNone)
1313 {
c88ad5db 1314 Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering"
1315 " but this is not the first unprocessed run!"));
86aa42c3 1316 return kFALSE;
1317 }
1318 else
1319 {
c88ad5db 1320 Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - "
1321 "Although %s requires strict run ordering "
1322 "and this is not the first unprocessed run, "
1323 "the SHUTTLE continues"));
86aa42c3 1324 }
be48e3ea 1325 }
1326
2bb7b766 1327 AliShuttleStatus* status = ReadShuttleStatus();
1328 if (!status) {
1329 // first time
1330 Log("SHUTTLE", Form("ContinueProcessing - %s: Processing first time",
1331 fCurrentDetector.Data()));
1332 status = new AliShuttleStatus(AliShuttleStatus::kStarted);
1333 return WriteShuttleStatus(status);
1334 }
1335
1336 // The following two cases shouldn't happen if Shuttle Logbook was correctly updated.
1337 // If it happens it may mean Logbook updating failed... let's do it now!
1338 if (status->GetStatus() == AliShuttleStatus::kDone ||
1339 status->GetStatus() == AliShuttleStatus::kFailed){
1340 Log("SHUTTLE", Form("ContinueProcessing - %s is already %s. Updating Shuttle Logbook",
1341 fCurrentDetector.Data(),
1342 status->GetStatusName(status->GetStatus())));
1343 UpdateShuttleLogbook(fCurrentDetector.Data(),
1344 status->GetStatusName(status->GetStatus()));
1345 return kFALSE;
1346 }
1347
3301427a 1348 if (status->GetStatus() == AliShuttleStatus::kStoreError) {
2bb7b766 1349 Log("SHUTTLE",
c88ad5db 1350 Form("ContinueProcessing - %s: Grid storage of one or more "
1351 "objects failed. Trying again now",
2bb7b766 1352 fCurrentDetector.Data()));
9827400b 1353 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1354 if (StoreOCDB()){
c88ad5db 1355 Log("SHUTTLE", Form("ContinueProcessing - %s: all objects "
1356 "successfully stored into main storage",
3301427a 1357 fCurrentDetector.Data()));
2bb7b766 1358 } else {
1359 Log("SHUTTLE",
1360 Form("ContinueProcessing - %s: Grid storage failed again",
1361 fCurrentDetector.Data()));
9827400b 1362 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
2bb7b766 1363 }
1364 return kFALSE;
1365 }
1366
1367 // if we get here, there is a restart
57c1a579 1368 Bool_t cont = kFALSE;
2bb7b766 1369
1370 // abort conditions
cb343cfd 1371 if (status->GetCount() >= fConfig->GetMaxRetries()) {
57c1a579 1372 Log("SHUTTLE", Form("ContinueProcessing - %s failed %d times in status %s - "
1373 "Updating Shuttle Logbook", fCurrentDetector.Data(),
2bb7b766 1374 status->GetCount(), status->GetStatusName()));
1375 UpdateShuttleLogbook(fCurrentDetector.Data(), "FAILED");
e7f62f16 1376 UpdateShuttleStatus(AliShuttleStatus::kFailed);
3301427a 1377
1378 // there may still be objects in local OCDB and reference storage
1379 // and FXS databases may be not updated: do it now!
9827400b 1380
1381 // TODO Currently disabled, we want to keep files in case of failure!
1382 // CleanLocalStorage(fgkLocalCDB);
1383 // CleanLocalStorage(fgkLocalRefStorage);
1384 // UpdateTableFailCase();
1385
1386 // Send mail to detector expert!
c88ad5db 1387 Log("SHUTTLE", Form("ContinueProcessing - Sending mail to %s expert...",
1388 fCurrentDetector.Data()));
9827400b 1389 if (!SendMail())
1390 Log("SHUTTLE", Form("ContinueProcessing - Could not send mail to %s expert",
1391 fCurrentDetector.Data()));
3301427a 1392
57c1a579 1393 } else {
1394 Log("SHUTTLE", Form("ContinueProcessing - %s: restarting. "
1395 "Aborted before with %s. Retry number %d.", fCurrentDetector.Data(),
1396 status->GetStatusName(), status->GetCount()));
9827400b 1397 Bool_t increaseCount = kTRUE;
c88ad5db 1398 if (status->GetStatus() == AliShuttleStatus::kDCSError ||
1399 status->GetStatus() == AliShuttleStatus::kDCSStarted)
1400 increaseCount = kFALSE;
675f64cd 1401
9827400b 1402 UpdateShuttleStatus(AliShuttleStatus::kStarted, increaseCount);
57c1a579 1403 cont = kTRUE;
2bb7b766 1404 }
1405
57c1a579 1406 return cont;
5164a766 1407}
1408
1409//______________________________________________________________________________________________
2bb7b766 1410Bool_t AliShuttle::Process(AliShuttleLogbookEntry* entry)
58bc3020 1411{
73abe331 1412 //
b948db8d 1413 // Makes data retrieval for all detectors in the configuration.
2bb7b766 1414 // entry: Shuttle logbook entry, contains run paramenters and status of detectors
1415 // (Unprocessed, Inactive, Failed or Done).
d477ad88 1416 // Returns kFALSE in case of error occured and kTRUE otherwise
73abe331 1417 //
1418
9827400b 1419 if (!entry) return kFALSE;
2bb7b766 1420
1421 fLogbookEntry = entry;
1422
c88ad5db 1423 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^*",
9827400b 1424 GetCurrentRun()));
2bb7b766 1425
e7f62f16 1426 // Send the information to ML
1427 TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
9827400b 1428 TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
e7f62f16 1429
1430 TList mlList;
1431 mlList.Add(&mlStatus);
9827400b 1432 mlList.Add(&mlRunType);
e7f62f16 1433
ee6f7523 1434 TString mlID;
1435 mlID.Form("%d", GetCurrentRun());
1436 fMonaLisa->SendParameters(&mlList, mlID);
3301427a 1437
9827400b 1438 if (fLogbookEntry->IsDone())
1439 {
1440 Log("SHUTTLE","Process - Shuttle is already DONE. Updating logbook");
1441 UpdateShuttleLogbook("shuttle_done");
1442 fLogbookEntry = 0;
1443 return kTRUE;
1444 }
1445
1446 // read test mode if flag is set
1447 if (fReadTestMode)
1448 {
3d8bc902 1449 fTestMode = kNone;
9827400b 1450 TString logEntry(entry->GetRunParameter("log"));
1451 //printf("log entry = %s\n", logEntry.Data());
1452 TString searchStr("Testmode: ");
1453 Int_t pos = logEntry.Index(searchStr.Data());
1454 //printf("%d\n", pos);
1455 if (pos >= 0)
1456 {
1457 TSubString subStr = logEntry(pos + searchStr.Length(), logEntry.Length());
1458 //printf("%s\n", subStr.String().Data());
1459 TString newStr(subStr.Data());
1460 TObjArray* token = newStr.Tokenize(' ');
1461 if (token)
1462 {
1463 //token->Print();
1464 TObjString* tmpStr = dynamic_cast<TObjString*> (token->First());
1465 if (tmpStr)
1466 {
1467 Int_t testMode = tmpStr->String().Atoi();
1468 if (testMode > 0)
1469 {
c88ad5db 1470 Log("SHUTTLE", Form("Process - Enabling test mode %d", testMode));
9827400b 1471 SetTestMode((TestMode) testMode);
1472 }
1473 }
1474 delete token;
1475 }
1476 }
1477 }
c88ad5db 1478
eba76848 1479 fLogbookEntry->Print("all");
57f50b3c 1480
1481 // Initialization
d477ad88 1482 Bool_t hasError = kFALSE;
5164a766 1483
675f64cd 1484 // Set the CDB and Reference folders according to the year and LHC period
1485 TString lhcPeriod(GetLHCPeriod());
1486 if (lhcPeriod.Length() == 0)
1487 {
7d4cf768 1488 Log("SHUTTLE","Process - LHCPeriod not found in logbook!");
1489 return 0;
675f64cd 1490 }
1491
1492 if (fgkMainCDB.Length() == 0)
1493 fgkMainCDB = Form("alien://folder=/alice/data/%d/%s/OCDB?user=alidaq?cacheFold=/tmp/OCDBCache",
1494 GetCurrentYear(), lhcPeriod.Data());
1495
1496 if (fgkMainRefStorage.Length() == 0)
1497 fgkMainRefStorage = Form("alien://folder=/alice/data/%d/%s/Reference?user=alidaq?cacheFold=/tmp/OCDBCache",
1498 GetCurrentYear(), lhcPeriod.Data());
1499
2bb7b766 1500 AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
1501 if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
1502 AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
1503 if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
d477ad88 1504
57f50b3c 1505 // Loop on detectors in the configuration
b948db8d 1506 TIter iter(fConfig->GetDetectors());
2bb7b766 1507 TObjString* aDetector = 0;
b948db8d 1508
be48e3ea 1509 while ((aDetector = (TObjString*) iter.Next()))
1510 {
7bfb2090 1511 fCurrentDetector = aDetector->String();
5164a766 1512
9e080f92 1513 if (ContinueProcessing() == kFALSE) continue;
1514
c88ad5db 1515 Log("SHUTTLE", Form("\t\t\t****** run %d - %s: START ******",
2bb7b766 1516 GetCurrentRun(), aDetector->GetName()));
1517
9d733021 1518 for(Int_t iSys=0;iSys<3;iSys++) fFXSCalled[iSys]=kFALSE;
1519
c88ad5db 1520 Log(fCurrentDetector.Data(), "Process - Starting processing");
85a80aa9 1521
be48e3ea 1522 Int_t pid = fork();
1523
1524 if (pid < 0)
1525 {
c88ad5db 1526 Log("SHUTTLE", "Process - ERROR: Forking failed");
be48e3ea 1527 }
1528 else if (pid > 0)
1529 {
1530 // parent
c88ad5db 1531 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Starting monitoring",
be48e3ea 1532 GetCurrentRun(), aDetector->GetName()));
1533
1534 Long_t begin = time(0);
1535
1536 int status; // to be used with waitpid, on purpose an int (not Int_t)!
1537 while (waitpid(pid, &status, WNOHANG) == 0)
1538 {
1539 Long_t expiredTime = time(0) - begin;
1540
1541 if (expiredTime > fConfig->GetPPTimeOut())
1542 {
9827400b 1543 TString tmp;
c88ad5db 1544 tmp.Form("Process - Process of %s time out. "
1545 "Run time: %d seconds. Killing...",
1546 fCurrentDetector.Data(), expiredTime);
9827400b 1547 Log("SHUTTLE", tmp);
1548 Log(fCurrentDetector, tmp);
be48e3ea 1549
1550 kill(pid, 9);
1551
3301427a 1552 UpdateShuttleStatus(AliShuttleStatus::kPPTimeOut);
be48e3ea 1553 hasError = kTRUE;
1554
1555 gSystem->Sleep(1000);
1556 }
1557 else
1558 {
be48e3ea 1559 gSystem->Sleep(1000);
9827400b 1560
1561 TString checkStr;
1562 checkStr.Form("ps -o vsize --pid %d | tail -n 1", pid);
1563 FILE* pipe = gSystem->OpenPipe(checkStr, "r");
1564 if (!pipe)
1565 {
c88ad5db 1566 Log("SHUTTLE", Form("Process - Error: "
1567 "Could not open pipe to %s", checkStr.Data()));
9827400b 1568 continue;
1569 }
1570
1571 char buffer[100];
1572 if (!fgets(buffer, 100, pipe))
1573 {
c88ad5db 1574 Log("SHUTTLE", "Process - Error: ps did not return anything");
9827400b 1575 gSystem->ClosePipe(pipe);
1576 continue;
1577 }
1578 gSystem->ClosePipe(pipe);
1579
1580 //Log("SHUTTLE", Form("ps returned %s", buffer));
1581
1582 Int_t mem = 0;
1583 if ((sscanf(buffer, "%d\n", &mem) != 1) || !mem)
1584 {
c88ad5db 1585 Log("SHUTTLE", "Process - Error: Could not parse output of ps");
9827400b 1586 continue;
1587 }
1588
1589 if (expiredTime % 60 == 0)
ee6f7523 1590 {
c88ad5db 1591 Log("SHUTTLE", Form("Process - %s: Checking process. "
1592 "Run time: %d seconds - Memory consumption: %d KB",
1593 fCurrentDetector.Data(), expiredTime, mem));
ee6f7523 1594 SendAlive();
1595 }
9827400b 1596
1597 if (mem > fConfig->GetPPMaxMem())
1598 {
1599 TString tmp;
c88ad5db 1600 tmp.Form("Process - Process exceeds maximum allowed memory "
1601 "(%d KB > %d KB). Killing...",
9827400b 1602 mem, fConfig->GetPPMaxMem());
1603 Log("SHUTTLE", tmp);
1604 Log(fCurrentDetector, tmp);
1605
1606 kill(pid, 9);
1607
1608 UpdateShuttleStatus(AliShuttleStatus::kPPOutOfMemory);
1609 hasError = kTRUE;
1610
1611 gSystem->Sleep(1000);
1612 }
be48e3ea 1613 }
1614 }
1615
c88ad5db 1616 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Client has terminated.",
be48e3ea 1617 GetCurrentRun(), aDetector->GetName()));
1618
1619 if (WIFEXITED(status))
1620 {
1621 Int_t returnCode = WEXITSTATUS(status);
1622
c88ad5db 1623 Log("SHUTTLE", Form("Process - %s: the return code is %d", fCurrentDetector.Data(),
3301427a 1624 returnCode));
be48e3ea 1625
9827400b 1626 if (returnCode == 0) hasError = kTRUE;
be48e3ea 1627 }
1628 }
1629 else if (pid == 0)
1630 {
1631 // client
c88ad5db 1632 Log("SHUTTLE", Form("Process - In client process of %d - %s", GetCurrentRun(),
1633 aDetector->GetName()));
be48e3ea 1634
c88ad5db 1635 Log("SHUTTLE", Form("Process - Redirecting output to %s log",fCurrentDetector.Data()));
ffa29e93 1636
546242fb 1637 if ((freopen(GetLogFileName(fCurrentDetector), "a", stdout)) == 0)
ffa29e93 1638 {
c88ad5db 1639 Log("SHUTTLE", "Process - Could not freopen stdout");
ffa29e93 1640 }
1641 else
1642 {
1643 fOutputRedirected = kTRUE;
1644 if ((dup2(fileno(stdout), fileno(stderr))) < 0)
c88ad5db 1645 Log("SHUTTLE", "Process - Could not redirect stderr");
ffa29e93 1646
1647 }
1648
5bac2bde 1649 TString wd = gSystem->WorkingDirectory();
675f64cd 1650 TString tmpDir = Form("%s/%s_%d_process", GetShuttleTempDir(),
1651 fCurrentDetector.Data(), GetCurrentRun());
5bac2bde 1652
d524ade6 1653 Int_t result = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
1654 if (!result) // temp dir already exists!
1655 {
1656 Log(fCurrentDetector.Data(),
1657 Form("Process - %s dir already exists! Removing...", tmpDir.Data()));
1658 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
675f64cd 1659 }
1660
1661 if (gSystem->mkdir(tmpDir.Data(), 1))
1662 {
1663 Log(fCurrentDetector.Data(), "Process - could not make temp directory!!");
1664 gSystem->Exit(1);
d524ade6 1665 }
1666
1667 if (!gSystem->ChangeDirectory(tmpDir.Data()))
1668 {
1669 Log(fCurrentDetector.Data(), "Process - could not change directory!!");
1670 gSystem->Exit(1);
1671 }
5bac2bde 1672
9827400b 1673 Bool_t success = ProcessCurrentDetector();
5bac2bde 1674
1675 gSystem->ChangeDirectory(wd.Data());
b1d18693 1676
9827400b 1677 if (success) // Preprocessor finished successfully!
1678 {
b1d18693 1679 // remove temporary folder
1680 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
1681
3301427a 1682 // Update time_processed field in FXS DB
1683 if (UpdateTable() == kFALSE)
5bac2bde 1684 Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!",
1685 fCurrentDetector.Data()));
3301427a 1686
1687 // Transfer the data from local storage to main storage (Grid)
1688 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1689 if (StoreOCDB() == kFALSE)
1690 {
c88ad5db 1691 Log("SHUTTLE",
1692 Form("\t\t\t****** run %d - %s: STORAGE ERROR ******",
3301427a 1693 GetCurrentRun(), aDetector->GetName()));
1694 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
9827400b 1695 success = kFALSE;
3301427a 1696 } else {
c88ad5db 1697 Log("SHUTTLE",
1698 Form("\t\t\t****** run %d - %s: DONE ******",
3301427a 1699 GetCurrentRun(), aDetector->GetName()));
1700 UpdateShuttleStatus(AliShuttleStatus::kDone);
9827400b 1701 UpdateShuttleLogbook(fCurrentDetector, "DONE");
3301427a 1702 }
c88ad5db 1703 } else
1704 {
1705 Log("SHUTTLE",
1706 Form("\t\t\t****** run %d - %s: PP ERROR ******",
1707 GetCurrentRun(), aDetector->GetName()));
be48e3ea 1708 }
1709
4b95672b 1710 for (UInt_t iSys=0; iSys<3; iSys++)
1711 {
1712 if (fFXSCalled[iSys]) fFXSlist[iSys].Clear();
1713 }
1714
c88ad5db 1715 Log("SHUTTLE", Form("Process - Client process of %d - %s is exiting now with %d.",
9827400b 1716 GetCurrentRun(), aDetector->GetName(), success));
be48e3ea 1717
1718 // the client exits here
9827400b 1719 gSystem->Exit(success);
be48e3ea 1720
1721 AliError("We should never get here!!!");
1722 }
7bfb2090 1723 }
5164a766 1724
c88ad5db 1725 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^*",
2bb7b766 1726 GetCurrentRun()));
1727
1728 //check if shuttle is done for this run, if so update logbook
1729 TObjArray checkEntryArray;
1730 checkEntryArray.SetOwner(1);
9e080f92 1731 TString whereClause = Form("where run=%d", GetCurrentRun());
b0e53b15 1732 if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) ||
1733 checkEntryArray.GetEntries() == 0) {
9e080f92 1734 Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
1735 GetCurrentRun()));
1736 return hasError == kFALSE;
1737 }
b948db8d 1738
9e080f92 1739 AliShuttleLogbookEntry* checkEntry = dynamic_cast<AliShuttleLogbookEntry*>
1740 (checkEntryArray.At(0));
2bb7b766 1741
9e080f92 1742 if (checkEntry)
1743 {
1744 if (checkEntry->IsDone())
be48e3ea 1745 {
9e080f92 1746 Log("SHUTTLE","Process - Shuttle is DONE. Updating logbook");
1747 UpdateShuttleLogbook("shuttle_done");
1748 }
1749 else
1750 {
1751 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
be48e3ea 1752 {
9e080f92 1753 if (checkEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
be48e3ea 1754 {
9e080f92 1755 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
1756 checkEntry->GetRun(), GetDetName(iDet)));
1757 fFirstUnprocessed[iDet] = kFALSE;
be48e3ea 1758 }
1759 }
2bb7b766 1760 }
1761 }
1762
1763 fLogbookEntry = 0;
85a80aa9 1764
a7160fe9 1765 return hasError == kFALSE;
73abe331 1766}
1767
b948db8d 1768//______________________________________________________________________________________________
9827400b 1769Bool_t AliShuttle::ProcessCurrentDetector()
73abe331 1770{
1771 //
2bb7b766 1772 // Makes data retrieval just for a specific detector (fCurrentDetector).
73abe331 1773 // Threre should be a configuration for this detector.
73abe331 1774
1d172743 1775 Log("SHUTTLE", Form("ProcessCurrentDetector - Retrieving values for %s, run %d",
1776 fCurrentDetector.Data(), GetCurrentRun()));
73abe331 1777
d524ade6 1778 TString wd = gSystem->WorkingDirectory();
1779
2d9019b4 1780 if (!CleanReferenceStorage(fCurrentDetector.Data()))
546242fb 1781 return kFALSE;
d524ade6 1782
1783 gSystem->ChangeDirectory(wd.Data());
1784
1d172743 1785 TMap* dcsMap = new TMap();
3301427a 1786
1787 // call preprocessor
1788 AliPreprocessor* aPreprocessor =
1789 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1790
1791 aPreprocessor->Initialize(GetCurrentRun(), GetCurrentStartTime(), GetCurrentEndTime());
1792
1793 Bool_t processDCS = aPreprocessor->ProcessDCS();
d477ad88 1794
651fdaab 1795 if (!processDCS)
1796 {
1d172743 1797 Log(fCurrentDetector, "ProcessCurrentDetector -"
1798 " The preprocessor requested to skip the retrieval of DCS values");
651fdaab 1799 }
8b739301 1800 else if (fTestMode & kSkipDCS)
2c15234c 1801 {
1d172743 1802 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Skipping DCS processing");
9827400b 1803 }
1804 else if (fTestMode & kErrorDCS)
1805 {
1d172743 1806 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Simulating DCS error");
3d8bc902 1807 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
9827400b 1808 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1d172743 1809 delete dcsMap;
9827400b 1810 return kFALSE;
2c15234c 1811 } else {
3301427a 1812
1813 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
1814
1d172743 1815 // Query DCS archive
1816 Int_t nServers = fConfig->GetNServers(fCurrentDetector);
a038aa70 1817
1d172743 1818 for (int iServ=0; iServ<nServers; iServ++)
2c15234c 1819 {
1d172743 1820
1821 TString host(fConfig->GetDCSHost(fCurrentDetector, iServ));
1822 Int_t port = fConfig->GetDCSPort(fCurrentDetector, iServ);
542b6cc8 1823 Int_t multiSplit = fConfig->GetMultiSplit(fCurrentDetector, iServ);
1824
1790d4b7 1825 Log(fCurrentDetector, Form("ProcessCurrentDetector -"
1826 " Querying DCS Amanda server %s:%d (%d of %d)",
1827 host.Data(), port, iServ+1, nServers));
1d172743 1828
1829 TMap* aliasMap = 0;
1830 TMap* dpMap = 0;
1831
1832 if (fConfig->GetDCSAliases(fCurrentDetector, iServ)->GetEntries() > 0)
2c15234c 1833 {
1d172743 1834 aliasMap = GetValueSet(host, port,
542b6cc8 1835 fConfig->GetDCSAliases(fCurrentDetector, iServ),
1836 kAlias, multiSplit);
1d172743 1837 if (!aliasMap)
1838 {
1839 Log(fCurrentDetector,
1840 Form("ProcessCurrentDetector -"
675f64cd 1841 " Error retrieving DCS aliases from server %s."
1842 " Sending mail to DCS experts!", host.Data()));
1d172743 1843 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
675f64cd 1844
1845 if (!SendMailToDCS())
1846 Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
1847
a038aa70 1848 delete dcsMap;
1d172743 1849 return kFALSE;
1850 }
2c15234c 1851 }
a038aa70 1852
1d172743 1853 if (fConfig->GetDCSDataPoints(fCurrentDetector, iServ)->GetEntries() > 0)
a038aa70 1854 {
1d172743 1855 dpMap = GetValueSet(host, port,
542b6cc8 1856 fConfig->GetDCSDataPoints(fCurrentDetector, iServ),
1857 kDP, multiSplit);
1d172743 1858 if (!dpMap)
1859 {
1860 Log(fCurrentDetector,
1861 Form("ProcessCurrentDetector -"
675f64cd 1862 " Error retrieving DCS data points from server %s."
1863 " Sending mail to DCS experts!", host.Data()));
1d172743 1864 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
675f64cd 1865
1866 if (!SendMailToDCS())
1867 Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
1868
1d172743 1869 if (aliasMap) delete aliasMap;
1870 delete dcsMap;
1871 return kFALSE;
1872 }
a038aa70 1873 }
1d172743 1874
1875 // merge aliasMap and dpMap into dcsMap
1876 if(aliasMap) {
1877 TIter iter(aliasMap);
a038aa70 1878 TObjString* key = 0;
1879 while ((key = (TObjString*) iter.Next()))
1d172743 1880 dcsMap->Add(key, aliasMap->GetValue(key->String()));
1881
1882 aliasMap->SetOwner(kFALSE);
1883 delete aliasMap;
1884 }
1885
1886 if(dpMap) {
1887 TIter iter(dpMap);
1888 TObjString* key = 0;
1889 while ((key = (TObjString*) iter.Next()))
1890 dcsMap->Add(key, dpMap->GetValue(key->String()));
1891
1892 dpMap->SetOwner(kFALSE);
1893 delete dpMap;
a038aa70 1894 }
73abe331 1895 }
1896 }
dc25836b 1897
b1d18693 1898 // save map into file, to help debugging in case of preprocessor error
1899 TFile* f = TFile::Open("DCSMap.root","recreate");
1900 f->cd();
1901 dcsMap->Write("DCSMap", TObject::kSingleKey);
1902 f->Close();
1903 delete f;
1904
2bb7b766 1905 // DCS Archive DB processing successful. Call Preprocessor!
85a80aa9 1906 UpdateShuttleStatus(AliShuttleStatus::kPPStarted);
a7160fe9 1907
a038aa70 1908 UInt_t returnValue = aPreprocessor->Process(dcsMap);
b948db8d 1909
3301427a 1910 if (returnValue > 0) // Preprocessor error!
1911 {
c88ad5db 1912 Log(fCurrentDetector, Form("ProcessCurrentDetector - "
1913 "Preprocessor failed. Process returned %d.", returnValue));
cb343cfd 1914 UpdateShuttleStatus(AliShuttleStatus::kPPError);
a038aa70 1915 dcsMap->DeleteAll();
1916 delete dcsMap;
9827400b 1917 return kFALSE;
1918 }
1919
1920 // preprocessor ok!
1921 UpdateShuttleStatus(AliShuttleStatus::kPPDone);
1922 Log(fCurrentDetector, Form("ProcessCurrentDetector - %s preprocessor returned success",
1923 fCurrentDetector.Data()));
b948db8d 1924
a038aa70 1925 dcsMap->DeleteAll();
1926 delete dcsMap;
b948db8d 1927
9827400b 1928 return kTRUE;
2bb7b766 1929}
1930
1931//______________________________________________________________________________________________
1932Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
1933 TObjArray& entries)
1934{
9827400b 1935 // Query DAQ's Shuttle logbook and fills detector status object.
1936 // Call QueryRunParameters to query DAQ logbook for run parameters.
1937 //
2bb7b766 1938
fc5a4708 1939 entries.SetOwner(1);
1940
2bb7b766 1941 // check connection, in case connect
be48e3ea 1942 if(!Connect(3)) return kFALSE;
2bb7b766 1943
1944 TString sqlQuery;
441b0e9c 1945 sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
2bb7b766 1946
be48e3ea 1947 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1948 if (!aResult) {
1949 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
1950 return kFALSE;
1951 }
1952
fc5a4708 1953 AliDebug(2,Form("Query = %s", sqlQuery.Data()));
1954
2bb7b766 1955 if(aResult->GetRowCount() == 0) {
c88ad5db 1956 Log("SHUTTLE", "No entries in Shuttle Logbook match request");
9827400b 1957 delete aResult;
1958 return kTRUE;
2bb7b766 1959 }
1960
1961 // TODO Check field count!
db99d43e 1962 const UInt_t nCols = 23;
2bb7b766 1963 if (aResult->GetFieldCount() != (Int_t) nCols) {
c88ad5db 1964 Log("SHUTTLE", "Invalid SQL result field number!");
2bb7b766 1965 delete aResult;
1966 return kFALSE;
1967 }
1968
2bb7b766 1969 TSQLRow* aRow;
1970 while ((aRow = aResult->Next())) {
1971 TString runString(aRow->GetField(0), aRow->GetFieldLength(0));
1972 Int_t run = runString.Atoi();
1973
eba76848 1974 AliShuttleLogbookEntry *entry = QueryRunParameters(run);
1975 if (!entry)
1976 continue;
2bb7b766 1977
1978 // loop on detectors
eba76848 1979 for(UInt_t ii = 0; ii < nCols; ii++)
1980 entry->SetDetectorStatus(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1981
eba76848 1982 entries.AddLast(entry);
2bb7b766 1983 delete aRow;
1984 }
1985
2bb7b766 1986 delete aResult;
1987 return kTRUE;
1988}
1989
1990//______________________________________________________________________________________________
eba76848 1991AliShuttleLogbookEntry* AliShuttle::QueryRunParameters(Int_t run)
2bb7b766 1992{
eba76848 1993 //
1994 // Retrieve run parameters written in the DAQ logbook and sets them into AliShuttleLogbookEntry object
1995 //
2bb7b766 1996
1997 // check connection, in case connect
be48e3ea 1998 if (!Connect(3))
eba76848 1999 return 0;
2bb7b766 2000
2001 TString sqlQuery;
2c15234c 2002 sqlQuery.Form("select * from %s where run=%d", fConfig->GetDAQlbTable(), run);
2bb7b766 2003
be48e3ea 2004 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 2005 if (!aResult) {
c88ad5db 2006 Log("SHUTTLE", Form("Can't execute query <%s>!", sqlQuery.Data()));
eba76848 2007 return 0;
2bb7b766 2008 }
2009
eba76848 2010 if (aResult->GetRowCount() == 0) {
2bb7b766 2011 Log("SHUTTLE", Form("QueryRunParameters - No entry in DAQ Logbook for run %d. Skipping", run));
2012 delete aResult;
eba76848 2013 return 0;
2bb7b766 2014 }
2015
eba76848 2016 if (aResult->GetRowCount() > 1) {
c88ad5db 2017 Log("SHUTTLE", Form("QueryRunParameters - UNEXPECTED: "
2018 "more than one entry in DAQ Logbook for run %d!", run));
2bb7b766 2019 delete aResult;
eba76848 2020 return 0;
2bb7b766 2021 }
2022
eba76848 2023 TSQLRow* aRow = aResult->Next();
2024 if (!aRow)
2025 {
c88ad5db 2026 Log("SHUTTLE", Form("QueryRunParameters - Could not retrieve row for run %d. Skipping", run));
eba76848 2027 delete aResult;
2028 return 0;
2029 }
2bb7b766 2030
eba76848 2031 AliShuttleLogbookEntry* entry = new AliShuttleLogbookEntry(run);
2bb7b766 2032
eba76848 2033 for (Int_t ii = 0; ii < aResult->GetFieldCount(); ii++)
2034 entry->SetRunParameter(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 2035
eba76848 2036 UInt_t startTime = entry->GetStartTime();
2037 UInt_t endTime = entry->GetEndTime();
2038
1abfbb60 2039// if (!startTime || !endTime || startTime > endTime)
2040// {
2041// Log("SHUTTLE",
2042// Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d. Skipping!",
2043// run, startTime, endTime));
2044//
2045// Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2046// fLogbookEntry = entry;
2047// if (!UpdateShuttleLogbook("shuttle_done"))
2048// {
2049// AliError(Form("Could not update logbook for run %d !", run));
2050// }
2051// fLogbookEntry = 0;
2052//
2053// delete entry;
2054// delete aRow;
2055// delete aResult;
2056// return 0;
2057// }
2058
2059 if (!startTime)
b0e53b15 2060 {
2061 Log("SHUTTLE",
1abfbb60 2062 Form("QueryRunParameters - Invalid parameters for Run %d: "
2063 "startTime = %d, endTime = %d. Skipping!",
2064 run, startTime, endTime));
b0e53b15 2065
2066 Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2067 fLogbookEntry = entry;
ee6f7523 2068 if (!UpdateShuttleLogbook("shuttle_ignored"))
b0e53b15 2069 {
2070 AliError(Form("Could not update logbook for run %d !", run));
2071 }
2072 fLogbookEntry = 0;
2073
2074 delete entry;
2075 delete aRow;
2076 delete aResult;
2077 return 0;
2078 }
2079
1abfbb60 2080 if (startTime && !endTime)
2081 {
2082 // TODO Here we don't mark SHUTTLE done, because this may mean
2083 //the run is still ongoing!!
2084 Log("SHUTTLE",
2085 Form("QueryRunParameters - Invalid parameters for Run %d: "
2086 "startTime = %d, endTime = %d. Skipping (Shuttle won't be marked as DONE)!",
2087 run, startTime, endTime));
2088
2089 //Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2090 //fLogbookEntry = entry;
2091 //if (!UpdateShuttleLogbook("shuttle_done"))
2092 //{
2093 // AliError(Form("Could not update logbook for run %d !", run));
2094 //}
2095 //fLogbookEntry = 0;
2096
2097 delete entry;
2098 delete aRow;
2099 delete aResult;
2100 return 0;
2101 }
2102
2103 if (startTime && endTime && (startTime > endTime))
2104 {
2105 Log("SHUTTLE",
2106 Form("QueryRunParameters - Invalid parameters for Run %d: "
2107 "startTime = %d, endTime = %d. Skipping!",
2108 run, startTime, endTime));
2109
2110 Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2111 fLogbookEntry = entry;
ee6f7523 2112 if (!UpdateShuttleLogbook("shuttle_ignored"))
1abfbb60 2113 {
2114 AliError(Form("Could not update logbook for run %d !", run));
2115 }
2116 fLogbookEntry = 0;
2117
2118 delete entry;
2119 delete aRow;
2120 delete aResult;
2121 return 0;
2122 }
2123
b0e53b15 2124 TString totEventsStr = entry->GetRunParameter("totalEvents");
2125 Int_t totEvents = totEventsStr.Atoi();
2126 if (totEvents < 1)
2127 {
eba76848 2128 Log("SHUTTLE",
b0e53b15 2129 Form("QueryRunParameters - Run %d has 0 events - Skipping!", run));
2130
2131 Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2132 fLogbookEntry = entry;
2133 if (!UpdateShuttleLogbook("shuttle_done"))
2134 {
2135 AliError(Form("Could not update logbook for run %d !", run));
2136 }
2137 fLogbookEntry = 0;
2138
eba76848 2139 delete entry;
2bb7b766 2140 delete aRow;
eba76848 2141 delete aResult;
2142 return 0;
2bb7b766 2143 }
2144
eba76848 2145 delete aRow;
2bb7b766 2146 delete aResult;
eba76848 2147
2148 return entry;
2bb7b766 2149}
2150
a038aa70 2151//______________________________________________________________________________________________
2152TMap* AliShuttle::GetValueSet(const char* host, Int_t port, const TSeqCollection* entries,
542b6cc8 2153 DCSType type, Int_t multiSplit)
a038aa70 2154{
2155 // Retrieve all "entry" data points from the DCS server
2156 // host, port: TSocket connection parameters
2157 // entries: list of name of the alias or data point
2158 // type: kAlias or kDP
2159 // returns TMap of values, 0 when failure
542b6cc8 2160
2161 AliDCSClient client(host, port, fTimeout, fRetries, multiSplit);
b41b252a 2162
a038aa70 2163 TMap* result = 0;
b41b252a 2164 if (type == kAlias)
a038aa70 2165 {
b41b252a 2166 result = client.GetAliasValues(entries, GetCurrentStartTime(),
2167 GetCurrentEndTime());
2168 }
2169 else if (type == kDP)
2170 {
2171 result = client.GetDPValues(entries, GetCurrentStartTime(),
2172 GetCurrentEndTime());
2173 }
a038aa70 2174
b41b252a 2175 if (result == 0)
2176 {
2177 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get entries! Reason: %s",
1790d4b7 2178 client.GetErrorString(client.GetResultErrorCode())));
2179 if (client.GetResultErrorCode() == AliDCSClient::fgkServerError)
2180 Log(fCurrentDetector.Data(), Form("GetValueSet - Server error code: %s",
2181 client.GetServerError().Data()));
a038aa70 2182
b41b252a 2183 return 0;
a038aa70 2184 }
b41b252a 2185
a038aa70 2186 return result;
2187}
b41b252a 2188
b948db8d 2189//______________________________________________________________________________________________
57f50b3c 2190const char* AliShuttle::GetFile(Int_t system, const char* detector,
2191 const char* id, const char* source)
b948db8d 2192{
9827400b 2193 // Get calibration file from file exchange servers
2194 // First queris the FXS database for the file name, using the run, detector, id and source info
2195 // then calls RetrieveFile(filename) for actual copy to local disk
2196 // run: current run being processed (given by Logbook entry fLogbookEntry)
2197 // detector: the Preprocessor name
2198 // id: provided as a parameter by the Preprocessor
2199 // source: provided by the Preprocessor through GetFileSources function
2200
2201 // check if test mode should simulate a FXS error
2202 if (fTestMode & kErrorFXSFiles)
2203 {
2204 Log(detector, Form("GetFile - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2205 return 0;
2206 }
2207
57f50b3c 2208 // check connection, in case connect
9d733021 2209 if (!Connect(system))
eba76848 2210 {
9d733021 2211 Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
57f50b3c 2212 return 0;
2213 }
2214
2215 // Query preparation
9d733021 2216 TString sourceName(source);
d386d623 2217 Int_t nFields = 3;
2218 TString sqlQueryStart = Form("select filePath,size,fileChecksum from %s where",
2219 fConfig->GetFXSdbTable(system));
2220 TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
2221 GetCurrentRun(), detector, id);
2222
9d733021 2223 if (system == kDAQ)
2224 {
d386d623 2225 whereClause += Form(" and DAQsource=\"%s\"", source);
57f50b3c 2226 }
9d733021 2227 else if (system == kDCS)
eba76848 2228 {
9d733021 2229 sourceName="none";
57f50b3c 2230 }
9d733021 2231 else if (system == kHLT)
9e080f92 2232 {
d386d623 2233 whereClause += Form(" and DDLnumbers=\"%s\"", source);
9d733021 2234 nFields = 3;
9e080f92 2235 }
2236
9e080f92 2237 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2238
2239 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2240
2241 // Query execution
2242 TSQLResult* aResult = 0;
9d733021 2243 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
9e080f92 2244 if (!aResult) {
9d733021 2245 Log(detector, Form("GetFileName - Can't execute SQL query to %s database for: id = %s, source = %s",
2246 GetSystemName(system), id, sourceName.Data()));
9e080f92 2247 return 0;
2248 }
2249
2250 if(aResult->GetRowCount() == 0)
2251 {
2252 Log(detector,
9d733021 2253 Form("GetFileName - No entry in %s FXS db for: id = %s, source = %s",
2254 GetSystemName(system), id, sourceName.Data()));
9e080f92 2255 delete aResult;
2256 return 0;
2257 }
2bb7b766 2258
9e080f92 2259 if (aResult->GetRowCount() > 1) {
2260 Log(detector,
9d733021 2261 Form("GetFileName - More than one entry in %s FXS db for: id = %s, source = %s",
2262 GetSystemName(system), id, sourceName.Data()));
9e080f92 2263 delete aResult;
2264 return 0;
2265 }
2266
9d733021 2267 if (aResult->GetFieldCount() != nFields) {
9e080f92 2268 Log(detector,
9d733021 2269 Form("GetFileName - Wrong field count in %s FXS db for: id = %s, source = %s",
2270 GetSystemName(system), id, sourceName.Data()));
9e080f92 2271 delete aResult;
2272 return 0;
2273 }
2274
2275 TSQLRow* aRow = dynamic_cast<TSQLRow*> (aResult->Next());
2276
2277 if (!aRow){
9d733021 2278 Log(detector, Form("GetFileName - Empty set result in %s FXS db from query: id = %s, source = %s",
2279 GetSystemName(system), id, sourceName.Data()));
9e080f92 2280 delete aResult;
2281 return 0;
2282 }
2283
2284 TString filePath(aRow->GetField(0), aRow->GetFieldLength(0));
2285 TString fileSize(aRow->GetField(1), aRow->GetFieldLength(1));
d386d623 2286 TString fileChecksum(aRow->GetField(2), aRow->GetFieldLength(2));
9e080f92 2287
2288 delete aResult;
2289 delete aRow;
2290
d386d623 2291 AliDebug(2, Form("filePath = %s; size = %s, fileChecksum = %s",
2292 filePath.Data(), fileSize.Data(), fileChecksum.Data()));
9e080f92 2293
9e080f92 2294 // retrieved file is renamed to make it unique
675f64cd 2295 TString localFileName = Form("%s/%s_%d_process/%s_%s_%d_%s_%s.shuttle",
2296 GetShuttleTempDir(), detector, GetCurrentRun(),
d524ade6 2297 GetSystemName(system), detector, GetCurrentRun(),
2298 id, sourceName.Data());
9d733021 2299
9e080f92 2300
9d733021 2301 // file retrieval from FXS
4b95672b 2302 UInt_t nRetries = 0;
2303 UInt_t maxRetries = 3;
2304 Bool_t result = kFALSE;
2305
2306 // copy!! if successful TSystem::Exec returns 0
2307 while(nRetries++ < maxRetries) {
2308 AliDebug(2, Form("Trying to copy file. Retry # %d", nRetries));
2309 result = RetrieveFile(system, filePath.Data(), localFileName.Data());
2310 if(!result)
2311 {
2312 Log(detector, Form("GetFileName - Copy of file %s from %s FXS failed",
9d733021 2313 filePath.Data(), GetSystemName(system)));
4b95672b 2314 continue;
4f0749a8 2315 }
9e080f92 2316
d386d623 2317 if (fileChecksum.Length()>0)
4b95672b 2318 {
2319 // compare md5sum of local file with the one stored in the FXS DB
d524ade6 2320 Int_t md5Comp = gSystem->Exec(Form("md5sum %s |grep %s 2>&1 > /dev/null",
2321 localFileName.Data(), fileChecksum.Data()));
9e080f92 2322
4b95672b 2323 if (md5Comp != 0)
2324 {
2325 Log(detector, Form("GetFileName - md5sum of file %s does not match with local copy!",
2326 filePath.Data()));
2327 result = kFALSE;
2328 continue;
2329 }
d386d623 2330 } else {
2331 Log(fCurrentDetector, Form("GetFile - md5sum of file %s not set in %s database, skipping comparison",
2332 filePath.Data(), GetSystemName(system)));
9d733021 2333 }
4b95672b 2334 if (result) break;
9e080f92 2335 }
2336
4b95672b 2337 if(!result) return 0;
2338
9d733021 2339 fFXSCalled[system]=kTRUE;
2340 TObjString *fileParams = new TObjString(Form("%s#!?!#%s", id, sourceName.Data()));
2341 fFXSlist[system].Add(fileParams);
9e080f92 2342
675f64cd 2343 static TString staticLocalFileName;
2344 staticLocalFileName.Form("%s", localFileName.Data());
2345
c88ad5db 2346 Log(fCurrentDetector, Form("GetFile - Retrieved file with id %s and "
2347 "source %s from %s to %s", id, source,
d524ade6 2348 GetSystemName(system), localFileName.Data()));
675f64cd 2349
d524ade6 2350 return staticLocalFileName.Data();
2bb7b766 2351}
2352
2353//______________________________________________________________________________________________
9d733021 2354Bool_t AliShuttle::RetrieveFile(UInt_t system, const char* fxsFileName, const char* localFileName)
9e080f92 2355{
9827400b 2356 //
2357 // Copies file from FXS to local Shuttle machine
2358 //
2bb7b766 2359
9e080f92 2360 // check temp directory: trying to cd to temp; if it does not exist, create it
d524ade6 2361 AliDebug(2, Form("Copy file %s from %s FXS into %s",
2362 GetSystemName(system), fxsFileName, localFileName));
2363
2364 TString tmpDir(localFileName);
2365
2366 tmpDir = tmpDir(0,tmpDir.Last('/'));
9e080f92 2367
d524ade6 2368 Int_t noDir = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
2369 if (noDir) // temp dir does not exists!
2370 {
2371 if (gSystem->mkdir(tmpDir.Data(), 1))
2372 {
2373 Log(fCurrentDetector.Data(), "RetrieveFile - could not make temp directory!!");
9e080f92 2374 return kFALSE;
2375 }
9e080f92 2376 }
2377
9d733021 2378 TString baseFXSFolder;
2379 if (system == kDAQ)
2380 {
2381 baseFXSFolder = "FES/";
2382 }
2383 else if (system == kDCS)
2384 {
2385 baseFXSFolder = "";
2386 }
2387 else if (system == kHLT)
2388 {
42fde080 2389 baseFXSFolder = "/opt/FXS/";
9d733021 2390 }
2391
2392
d524ade6 2393 TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s",
9d733021 2394 fConfig->GetFXSPort(system),
2395 fConfig->GetFXSUser(system),
2396 fConfig->GetFXSHost(system),
2397 baseFXSFolder.Data(),
2398 fxsFileName,
9e080f92 2399 localFileName);
2400
2401 AliDebug(2, Form("%s",command.Data()));
2402
4b95672b 2403 Bool_t result = (gSystem->Exec(command.Data()) == 0);
9e080f92 2404
4b95672b 2405 return result;
9e080f92 2406}
2407
2408//______________________________________________________________________________________________
9d733021 2409TList* AliShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
2410{
9827400b 2411 //
2412 // Get sources producing the condition file Id from file exchange servers
4a33bdd9 2413 // if id is NULL all sources are returned (distinct)
9827400b 2414 //
1bcd28db 2415
2416 Log(detector, Form("GetFileSources - Retrieving sources with id %s from %s", id, GetSystemName(system)));
9827400b 2417
2418 // check if test mode should simulate a FXS error
2419 if (fTestMode & kErrorFXSSources)
2420 {
2421 Log(detector, Form("GetFileSources - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2422 return 0;
2423 }
2424
9d733021 2425 if (system == kDCS)
2426 {
c88ad5db 2427 Log(detector, "GetFileSources - WARNING: DCS system has only one source of data!");
6297b37d 2428 TList *list = new TList();
2429 list->SetOwner(1);
2430 list->Add(new TObjString(" "));
2431 return list;
9d733021 2432 }
9e080f92 2433
2434 // check connection, in case connect
9d733021 2435 if (!Connect(system))
2436 {
4a33bdd9 2437 Log(detector, Form("GetFileSources - Couldn't connect to %s FXS database", GetSystemName(system)));
9d733021 2438 return NULL;
9e080f92 2439 }
2440
9d733021 2441 TString sourceName = 0;
2442 if (system == kDAQ)
2443 {
2444 sourceName = "DAQsource";
2445 } else if (system == kHLT)
2446 {
2447 sourceName = "DDLnumbers";
2448 }
2449
4a33bdd9 2450 TString sqlQueryStart = Form("select distinct %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
2451 TString whereClause = Form("run=%d and detector=\"%s\"",
2452 GetCurrentRun(), detector);
2453 if (id)
2454 whereClause += Form(" and fileId=\"%s\"", id);
9e080f92 2455 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2456
2457 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2458
2459 // Query execution
2460 TSQLResult* aResult;
9d733021 2461 aResult = fServer[system]->Query(sqlQuery);
9e080f92 2462 if (!aResult) {
9d733021 2463 Log(detector, Form("GetFileSources - Can't execute SQL query to %s database for id: %s",
2464 GetSystemName(system), id));
9e080f92 2465 return 0;
2466 }
2467
86aa42c3 2468 TList *list = new TList();
2469 list->SetOwner(1);
2470
9d733021 2471 if (aResult->GetRowCount() == 0)
2472 {
9e080f92 2473 Log(detector,
9d733021 2474 Form("GetFileSources - No entry in %s FXS table for id: %s", GetSystemName(system), id));
9e080f92 2475 delete aResult;
86aa42c3 2476 return list;
9e080f92 2477 }
2478
1bcd28db 2479 Log(detector, Form("GetFileSources - Found %d sources", aResult->GetRowCount()));
9e080f92 2480
1bcd28db 2481 TSQLRow* aRow;
9d733021 2482 while ((aRow = aResult->Next()))
2483 {
9e080f92 2484
9d733021 2485 TString source(aRow->GetField(0), aRow->GetFieldLength(0));
2486 AliDebug(2, Form("%s = %s", sourceName.Data(), source.Data()));
2487 list->Add(new TObjString(source));
9e080f92 2488 delete aRow;
2489 }
9d733021 2490
9e080f92 2491 delete aResult;
2492
2493 return list;
2bb7b766 2494}
2495
4a33bdd9 2496//______________________________________________________________________________________________
2497TList* AliShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
2498{
2499 //
2500 // Get all ids of condition files produced by a given source from file exchange servers
2501 //
2502
1bcd28db 2503 Log(detector, Form("GetFileIDs - Retrieving ids with source %s with %s", source, GetSystemName(system)));
2504
4a33bdd9 2505 // check if test mode should simulate a FXS error
2506 if (fTestMode & kErrorFXSSources)
2507 {
2508 Log(detector, Form("GetFileIDs - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2509 return 0;
2510 }
2511
2512 // check connection, in case connect
2513 if (!Connect(system))
2514 {
2515 Log(detector, Form("GetFileIDs - Couldn't connect to %s FXS database", GetSystemName(system)));
2516 return NULL;
2517 }
2518
2519 TString sourceName = 0;
2520 if (system == kDAQ)
2521 {
2522 sourceName = "DAQsource";
2523 } else if (system == kHLT)
2524 {
2525 sourceName = "DDLnumbers";
2526 }
2527
2528 TString sqlQueryStart = Form("select fileId from %s where", fConfig->GetFXSdbTable(system));
2529 TString whereClause = Form("run=%d and detector=\"%s\"",
2530 GetCurrentRun(), detector);
2531 if (sourceName.Length() > 0 && source)
2532 whereClause += Form(" and %s=\"%s\"", sourceName.Data(), source);
2533 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2534
2535 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2536
2537 // Query execution
2538 TSQLResult* aResult;
2539 aResult = fServer[system]->Query(sqlQuery);
2540 if (!aResult) {
2541 Log(detector, Form("GetFileIDs - Can't execute SQL query to %s database for source: %s",
2542 GetSystemName(system), source));
2543 return 0;
2544 }
2545
2546 TList *list = new TList();
2547 list->SetOwner(1);
2548
2549 if (aResult->GetRowCount() == 0)
2550 {
2551 Log(detector,
2552 Form("GetFileIDs - No entry in %s FXS table for source: %s", GetSystemName(system), source));
2553 delete aResult;
2554 return list;
2555 }
2556
1bcd28db 2557 Log(detector, Form("GetFileIDs - Found %d ids", aResult->GetRowCount()));
2558
4a33bdd9 2559 TSQLRow* aRow;
2560
2561 while ((aRow = aResult->Next()))
2562 {
2563
2564 TString id(aRow->GetField(0), aRow->GetFieldLength(0));
2565 AliDebug(2, Form("fileId = %s", id.Data()));
2566 list->Add(new TObjString(id));
2567 delete aRow;
2568 }
2569
2570 delete aResult;
2571
2572 return list;
2573}
2574
2bb7b766 2575//______________________________________________________________________________________________
9d733021 2576Bool_t AliShuttle::Connect(Int_t system)
2bb7b766 2577{
9827400b 2578 // Connect to MySQL Server of the system's FXS MySQL databases
2579 // DAQ Logbook, Shuttle Logbook and DAQ FXS db are on the same host
2580 //
57f50b3c 2581
9d733021 2582 // check connection: if already connected return
2583 if(fServer[system] && fServer[system]->IsConnected()) return kTRUE;
57f50b3c 2584
9d733021 2585 TString dbHost, dbUser, dbPass, dbName;
57f50b3c 2586
9d733021 2587 if (system < 3) // FXS db servers
2588 {
2589 dbHost = Form("mysql://%s:%d", fConfig->GetFXSdbHost(system), fConfig->GetFXSdbPort(system));
2590 dbUser = fConfig->GetFXSdbUser(system);
2591 dbPass = fConfig->GetFXSdbPass(system);
2592 dbName = fConfig->GetFXSdbName(system);
2593 } else { // Run & Shuttle logbook servers
2594 // TODO Will the Shuttle logbook server be the same as the Run logbook server ???
2595 dbHost = Form("mysql://%s:%d", fConfig->GetDAQlbHost(), fConfig->GetDAQlbPort());
2596 dbUser = fConfig->GetDAQlbUser();
2597 dbPass = fConfig->GetDAQlbPass();
2598 dbName = fConfig->GetDAQlbDB();
2599 }
57f50b3c 2600
9d733021 2601 fServer[system] = TSQLServer::Connect(dbHost.Data(), dbUser.Data(), dbPass.Data());
2602 if (!fServer[system] || !fServer[system]->IsConnected()) {
2603 if(system < 3)
2604 {
2605 AliError(Form("Can't establish connection to FXS database for %s",
2606 AliShuttleInterface::GetSystemName(system)));
2607 } else {
2608 AliError("Can't establish connection to Run logbook.");
57f50b3c 2609 }
9d733021 2610 if(fServer[system]) delete fServer[system];
2611 return kFALSE;
2bb7b766 2612 }
57f50b3c 2613
9d733021 2614 // Get tables
2615 TSQLResult* aResult=0;
2616 switch(system){
2617 case kDAQ:
2618 aResult = fServer[kDAQ]->GetTables(dbName.Data());
2619 break;
2620 case kDCS:
2621 aResult = fServer[kDCS]->GetTables(dbName.Data());
2622 break;
2623 case kHLT:
2624 aResult = fServer[kHLT]->GetTables(dbName.Data());
2625 break;
2626 default:
2627 aResult = fServer[3]->GetTables(dbName.Data());
2628 break;
2629 }
2630
2631 delete aResult;
2bb7b766 2632 return kTRUE;
2633}
57f50b3c 2634
9e080f92 2635//______________________________________________________________________________________________
9d733021 2636Bool_t AliShuttle::UpdateTable()
9e080f92 2637{
9827400b 2638 //
2639 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2640 //
9e080f92 2641
9d733021 2642 Bool_t result = kTRUE;
9e080f92 2643
9d733021 2644 for (UInt_t system=0; system<3; system++)
2645 {
2646 if(!fFXSCalled[system]) continue;
9e080f92 2647
9d733021 2648 // check connection, in case connect
2649 if (!Connect(system))
2650 {
2651 Log(fCurrentDetector, Form("UpdateTable - Couldn't connect to %s FXS database", GetSystemName(system)));
2652 result = kFALSE;
2653 continue;
9e080f92 2654 }
9e080f92 2655
9d733021 2656 TTimeStamp now; // now
2657
2658 // Loop on FXS list entries
2659 TIter iter(&fFXSlist[system]);
2660 TObjString *aFXSentry=0;
2661 while ((aFXSentry = dynamic_cast<TObjString*> (iter.Next())))
2662 {
2663 TString aFXSentrystr = aFXSentry->String();
2664 TObjArray *aFXSarray = aFXSentrystr.Tokenize("#!?!#");
2665 if (!aFXSarray || aFXSarray->GetEntries() != 2 )
2666 {
2667 Log(fCurrentDetector, Form("UpdateTable - error updating %s FXS entry. Check string: <%s>",
2668 GetSystemName(system), aFXSentrystr.Data()));
2669 if(aFXSarray) delete aFXSarray;
2670 result = kFALSE;
2671 continue;
2672 }
2673 const char* fileId = ((TObjString*) aFXSarray->At(0))->GetName();
2674 const char* source = ((TObjString*) aFXSarray->At(1))->GetName();
2675
2676 TString whereClause;
2677 if (system == kDAQ)
2678 {
2679 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DAQsource=\"%s\";",
2680 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2681 }
2682 else if (system == kDCS)
2683 {
2684 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\";",
2685 GetCurrentRun(), fCurrentDetector.Data(), fileId);
2686 }
2687 else if (system == kHLT)
2688 {
2689 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DDLnumbers=\"%s\";",
2690 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2691 }
2692
2693 delete aFXSarray;
9e080f92 2694
9d733021 2695 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2696 now.GetSec(), whereClause.Data());
9e080f92 2697
9d733021 2698 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
9e080f92 2699
9d733021 2700 // Query execution
2701 TSQLResult* aResult;
2702 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2703 if (!aResult)
2704 {
2705 Log(fCurrentDetector, Form("UpdateTable - %s db: can't execute SQL query <%s>",
2706 GetSystemName(system), sqlQuery.Data()));
2707 result = kFALSE;
2708 continue;
2709 }
2710 delete aResult;
9e080f92 2711 }
9e080f92 2712 }
2713
9d733021 2714 return result;
9e080f92 2715}
57f50b3c 2716
3301427a 2717//______________________________________________________________________________________________
2718Bool_t AliShuttle::UpdateTableFailCase()
2719{
9827400b 2720 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2721 // this is called in case the preprocessor is declared failed for the current run, because
2722 // the fields are updated only in case of success
3301427a 2723
2724 Bool_t result = kTRUE;
2725
2726 for (UInt_t system=0; system<3; system++)
2727 {
2728 // check connection, in case connect
2729 if (!Connect(system))
2730 {
2731 Log(fCurrentDetector, Form("UpdateTableFailCase - Couldn't connect to %s FXS database",
2732 GetSystemName(system)));
2733 result = kFALSE;
2734 continue;
2735 }
2736
2737 TTimeStamp now; // now
2738
2739 // Loop on FXS list entries
2740
2741 TString whereClause = Form("where run=%d and detector=\"%s\";",
2742 GetCurrentRun(), fCurrentDetector.Data());
2743
2744
2745 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2746 now.GetSec(), whereClause.Data());
2747
2748 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2749
2750 // Query execution
2751 TSQLResult* aResult;
2752 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2753 if (!aResult)
2754 {
2755 Log(fCurrentDetector, Form("UpdateTableFailCase - %s db: can't execute SQL query <%s>",
2756 GetSystemName(system), sqlQuery.Data()));
2757 result = kFALSE;
2758 continue;
2759 }
2760 delete aResult;
2761 }
2762
2763 return result;
2764}
2765
2bb7b766 2766//______________________________________________________________________________________________
2767Bool_t AliShuttle::UpdateShuttleLogbook(const char* detector, const char* status)
2768{
e7f62f16 2769 //
2770 // Update Shuttle logbook filling detector or shuttle_done column
2771 // ex. of usage: UpdateShuttleLogbook("PHOS", "DONE") or UpdateShuttleLogbook("shuttle_done")
2772 //
57f50b3c 2773
2bb7b766 2774 // check connection, in case connect
be48e3ea 2775 if(!Connect(3)){
2bb7b766 2776 Log("SHUTTLE", "UpdateShuttleLogbook - Couldn't connect to DAQ Logbook.");
2777 return kFALSE;
57f50b3c 2778 }
2779
2bb7b766 2780 TString detName(detector);
2781 TString setClause;
ee6f7523 2782 if (detName == "shuttle_done" || detName == "shuttle_ignored")
e7f62f16 2783 {
2bb7b766 2784 setClause = "set shuttle_done=1";
e7f62f16 2785
ee6f7523 2786 if (detName == "shuttle_done")
b0e53b15 2787 {
2788 // Send the information to ML
2789 TMonaLisaText mlStatus("SHUTTLE_status", "Done");
e7f62f16 2790
b0e53b15 2791 TList mlList;
2792 mlList.Add(&mlStatus);
2793
ee6f7523 2794 TString mlID;
2795 mlID.Form("%d", GetCurrentRun());
2796 fMonaLisa->SendParameters(&mlList, mlID);
b0e53b15 2797 }
2bb7b766 2798 } else {
2bb7b766 2799 TString statusStr(status);
2800 if(statusStr.Contains("done", TString::kIgnoreCase) ||
2801 statusStr.Contains("failed", TString::kIgnoreCase)){
eba76848 2802 setClause = Form("set %s=\"%s\"", detector, status);
2bb7b766 2803 } else {
2804 Log("SHUTTLE",
2805 Form("UpdateShuttleLogbook - Invalid status <%s> for detector %s",
2806 status, detector));
2807 return kFALSE;
2808 }
2809 }
57f50b3c 2810
2bb7b766 2811 TString whereClause = Form("where run=%d", GetCurrentRun());
2812
441b0e9c 2813 TString sqlQuery = Form("update %s %s %s",
2814 fConfig->GetShuttlelbTable(), setClause.Data(), whereClause.Data());
57f50b3c 2815
2bb7b766 2816 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2817
2818 // Query execution
2819 TSQLResult* aResult;
be48e3ea 2820 aResult = dynamic_cast<TSQLResult*> (fServer[3]->Query(sqlQuery));
2bb7b766 2821 if (!aResult) {
2822 Log("SHUTTLE", Form("UpdateShuttleLogbook - Can't execute query <%s>", sqlQuery.Data()));
2823 return kFALSE;
57f50b3c 2824 }
2bb7b766 2825 delete aResult;
57f50b3c 2826
2827 return kTRUE;
2828}
2829
2830//______________________________________________________________________________________________
2bb7b766 2831Int_t AliShuttle::GetCurrentRun() const
2832{
9827400b 2833 //
2834 // Get current run from logbook entry
2835 //
57f50b3c 2836
2bb7b766 2837 return fLogbookEntry ? fLogbookEntry->GetRun() : -1;
57f50b3c 2838}
2839
2840//______________________________________________________________________________________________
2bb7b766 2841UInt_t AliShuttle::GetCurrentStartTime() const
2842{
9827400b 2843 //
2844 // get current start time
2845 //
57f50b3c 2846
2bb7b766 2847 return fLogbookEntry ? fLogbookEntry->GetStartTime() : 0;
57f50b3c 2848}
2849
2850//______________________________________________________________________________________________
2bb7b766 2851UInt_t AliShuttle::GetCurrentEndTime() const
2852{
9827400b 2853 //
2854 // get current end time from logbook entry
2855 //
57f50b3c 2856
2bb7b766 2857 return fLogbookEntry ? fLogbookEntry->GetEndTime() : 0;
57f50b3c 2858}
2859
675f64cd 2860//______________________________________________________________________________________________
2861UInt_t AliShuttle::GetCurrentYear() const
2862{
2863 //
2864 // Get current year from logbook entry
2865 //
2866
2867 if (!fLogbookEntry) return 0;
2868
2869 TTimeStamp startTime(GetCurrentStartTime());
2870 TString year = Form("%d",startTime.GetDate());
2871 year = year(0,4);
2872
2873 return year.Atoi();
2874}
2875
2876//______________________________________________________________________________________________
2877const char* AliShuttle::GetLHCPeriod() const
2878{
2879 //
2880 // Get current LHC period from logbook entry
2881 //
2882
2883 if (!fLogbookEntry) return 0;
2884
2885 return fLogbookEntry->GetRunParameter("LHCperiod");
2886}
2887
b948db8d 2888//______________________________________________________________________________________________
2889void AliShuttle::Log(const char* detector, const char* message)
2890{
9827400b 2891 //
2892 // Fill log string with a message
2893 //
b948db8d 2894
7d4cf768 2895 TString logRunDir = GetShuttleLogDir();
2896 if (GetCurrentRun() >=0)
2897 logRunDir += Form("/%d", GetCurrentRun());
2898
2899 void* dir = gSystem->OpenDirectory(logRunDir.Data());
84090f85 2900 if (dir == NULL) {
7d4cf768 2901 if (gSystem->mkdir(logRunDir.Data(), kTRUE)) {
36c99a6a 2902 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
84090f85 2903 return;
2904 }
b948db8d 2905
84090f85 2906 } else {
2907 gSystem->FreeDirectory(dir);
2908 }
b948db8d 2909
cb343cfd 2910 TString toLog = Form("%s (%d): %s - ", TTimeStamp(time(0)).AsString("s"), getpid(), detector);
e7f62f16 2911 if (GetCurrentRun() >= 0)
2912 toLog += Form("run %d - ", GetCurrentRun());
2bb7b766 2913 toLog += Form("%s", message);
2914
84090f85 2915 AliInfo(toLog.Data());
ffa29e93 2916
2917 // if we redirect the log output already to the file, leave here
2918 if (fOutputRedirected && strcmp(detector, "SHUTTLE") != 0)
2919 return;
b948db8d 2920
ffa29e93 2921 TString fileName = GetLogFileName(detector);
e7f62f16 2922
84090f85 2923 gSystem->ExpandPathName(fileName);
2924
2925 ofstream logFile;
2926 logFile.open(fileName, ofstream::out | ofstream::app);
2927
2928 if (!logFile.is_open()) {
2929 AliError(Form("Could not open file %s", fileName.Data()));
2930 return;
2931 }
7bfb2090 2932
84090f85 2933 logFile << toLog.Data() << "\n";
b948db8d 2934
84090f85 2935 logFile.close();
b948db8d 2936}
2bb7b766 2937
ffa29e93 2938//______________________________________________________________________________________________
2939TString AliShuttle::GetLogFileName(const char* detector) const
2940{
2941 //
2942 // returns the name of the log file for a given sub detector
2943 //
2944
2945 TString fileName;
2946
2947 if (GetCurrentRun() >= 0)
7d4cf768 2948 {
2949 fileName.Form("%s/%d/%s_%d.log", GetShuttleLogDir(), GetCurrentRun(),
2950 detector, GetCurrentRun());
2951 } else {
ffa29e93 2952 fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
7d4cf768 2953 }
ffa29e93 2954
2955 return fileName;
2956}
2957
ee6f7523 2958//______________________________________________________________________________________________
2959void AliShuttle::SendAlive()
2960{
2961 // sends alive message to ML
2962
2963 TMonaLisaText mlStatus("SHUTTLE_status", "Alive");
2964
2965 TList mlList;
2966 mlList.Add(&mlStatus);
2967
2968 fMonaLisa->SendParameters(&mlList, "__PROCESSINGINFO__");
2969}
2970
2bb7b766 2971//______________________________________________________________________________________________
2972Bool_t AliShuttle::Collect(Int_t run)
2973{
9827400b 2974 //
2975 // Collects conditions data for all UNPROCESSED run written to DAQ LogBook in case of run = -1 (default)
2976 // If a dedicated run is given this run is processed
2977 //
2978 // In operational mode, this is the Shuttle function triggered by the EOR signal.
2979 //
2bb7b766 2980
eba76848 2981 if (run == -1)
2982 Log("SHUTTLE","Collect - Shuttle called. Collecting conditions data for unprocessed runs");
2983 else
2984 Log("SHUTTLE", Form("Collect - Shuttle called. Collecting conditions data for run %d", run));
cb343cfd 2985
2986 SetLastAction("Starting");
2bb7b766 2987
ee6f7523 2988 // create ML instance
2989 if (!fMonaLisa)
2990 fMonaLisa = new TMonaLisaWriter(fConfig->GetMonitorHost(), fConfig->GetMonitorTable());
2991
2992
2993 SendAlive();
2994
2bb7b766 2995 TString whereClause("where shuttle_done=0");
eba76848 2996 if (run != -1)
2997 whereClause += Form(" and run=%d", run);
2bb7b766 2998
2999 TObjArray shuttleLogbookEntries;
be48e3ea 3000 if (!QueryShuttleLogbook(whereClause, shuttleLogbookEntries))
3001 {
cb343cfd 3002 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2bb7b766 3003 return kFALSE;
3004 }
3005
9e080f92 3006 if (shuttleLogbookEntries.GetEntries() == 0)
3007 {
3008 if (run == -1)
3009 Log("SHUTTLE","Collect - Found no UNPROCESSED runs in Shuttle logbook");
3010 else
3011 Log("SHUTTLE", Form("Collect - Run %d is already DONE "
3012 "or it does not exist in Shuttle logbook", run));
3013 return kTRUE;
3014 }
3015
be48e3ea 3016 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
3017 fFirstUnprocessed[iDet] = kTRUE;
3018
fc5a4708 3019 if (run != -1)
be48e3ea 3020 {
3021 // query Shuttle logbook for earlier runs, check if some detectors are unprocessed,
3022 // flag them into fFirstUnprocessed array
3023 TString whereClause(Form("where shuttle_done=0 and run < %d", run));
3024 TObjArray tmpLogbookEntries;
3025 if (!QueryShuttleLogbook(whereClause, tmpLogbookEntries))
3026 {
3027 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
3028 return kFALSE;
3029 }
3030
3031 TIter iter(&tmpLogbookEntries);
3032 AliShuttleLogbookEntry* anEntry = 0;
3033 while ((anEntry = dynamic_cast<AliShuttleLogbookEntry*> (iter.Next())))
3034 {
3035 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
3036 {
3037 if (anEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
3038 {
3039 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
3040 anEntry->GetRun(), GetDetName(iDet)));
3041 fFirstUnprocessed[iDet] = kFALSE;
3042 }
3043 }
3044
3045 }
3046
3047 }
3048
3049 if (!RetrieveConditionsData(shuttleLogbookEntries))
3050 {
cb343cfd 3051 Log("SHUTTLE", "Collect - Process of at least one run failed");
2bb7b766 3052 return kFALSE;
3053 }
3054
36c99a6a 3055 Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
eba76848 3056 return kTRUE;
2bb7b766 3057}
3058
2bb7b766 3059//______________________________________________________________________________________________
3060Bool_t AliShuttle::RetrieveConditionsData(const TObjArray& dateEntries)
3061{
9827400b 3062 //
3063 // Retrieve conditions data for all runs that aren't processed yet
3064 //
2bb7b766 3065
3066 Bool_t hasError = kFALSE;
3067
3068 TIter iter(&dateEntries);
3069 AliShuttleLogbookEntry* anEntry;
3070
3071 while ((anEntry = (AliShuttleLogbookEntry*) iter.Next())){
3072 if (!Process(anEntry)){
3073 hasError = kTRUE;
3074 }
4b95672b 3075
3076 // clean SHUTTLE temp directory
d524ade6 3077 //TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
3078 //RemoveFile(filename.Data());
2bb7b766 3079 }
3080
3081 return hasError == kFALSE;
3082}
cb343cfd 3083
3084//______________________________________________________________________________________________
3085ULong_t AliShuttle::GetTimeOfLastAction() const
3086{
9827400b 3087 //
3088 // Gets time of last action
3089 //
3090
cb343cfd 3091 ULong_t tmp;
36c99a6a 3092
cb343cfd 3093 fMonitoringMutex->Lock();
be48e3ea 3094
cb343cfd 3095 tmp = fLastActionTime;
36c99a6a 3096
cb343cfd 3097 fMonitoringMutex->UnLock();
36c99a6a 3098
cb343cfd 3099 return tmp;
3100}
3101
3102//______________________________________________________________________________________________
3103const TString AliShuttle::GetLastAction() const
3104{
9827400b 3105 //
cb343cfd 3106 // returns a string description of the last action
9827400b 3107 //
cb343cfd 3108
3109 TString tmp;
36c99a6a 3110
cb343cfd 3111 fMonitoringMutex->Lock();
3112
3113 tmp = fLastAction;
3114
3115 fMonitoringMutex->UnLock();
3116
36c99a6a 3117 return tmp;
cb343cfd 3118}
3119
3120//______________________________________________________________________________________________
3121void AliShuttle::SetLastAction(const char* action)
3122{
9827400b 3123 //
cb343cfd 3124 // updates the monitoring variables
9827400b 3125 //
36c99a6a 3126
cb343cfd 3127 fMonitoringMutex->Lock();
36c99a6a 3128
cb343cfd 3129 fLastAction = action;
3130 fLastActionTime = time(0);
3131
3132 fMonitoringMutex->UnLock();
3133}
eba76848 3134
3135//______________________________________________________________________________________________
3136const char* AliShuttle::GetRunParameter(const char* param)
3137{
9827400b 3138 //
3139 // returns run parameter read from DAQ logbook
3140 //
eba76848 3141
3142 if(!fLogbookEntry) {
3143 AliError("No logbook entry!");
3144 return 0;
3145 }
3146
3147 return fLogbookEntry->GetRunParameter(param);
3148}
57c1a579 3149
d386d623 3150//______________________________________________________________________________________________
9827400b 3151AliCDBEntry* AliShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
d386d623 3152{
9827400b 3153 //
3154 // returns object from OCDB valid for current run
3155 //
d386d623 3156
9827400b 3157 if (fTestMode & kErrorOCDB)
3158 {
3159 Log(detector, "GetFromOCDB - In TESTMODE - Simulating error with OCDB");
3160 return 0;
3161 }
3162
d386d623 3163 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
3164 if (!sto)
3165 {
9827400b 3166 Log(detector, "GetFromOCDB - Cannot activate main OCDB for query!");
d386d623 3167 return 0;