removing all DPs
[u/mrichter/AliRoot.git] / SHUTTLE / AliShuttle.cxx
CommitLineData
73abe331 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
4508ab8b 18Revision 1.73 2007/12/14 19:31:36 acolla
19Sending email to DCS experts is temporarily commented
20
b5f1c82e 21Revision 1.72 2007/12/13 15:44:28 acolla
22Run type added in mail sent to detector expert (eases understanding)
23
6a1146c4 24Revision 1.71 2007/12/12 14:56:14 jgrosseo
25sending shuttle_ignore to ML also in case of 0 events
26
97e3c167 27Revision 1.70 2007/12/12 13:45:35 acolla
28Monalisa started in Collect() function. Alive message to monitor is sent at each Collect and every minute during preprocessor processing.
29
ee6f7523 30Revision 1.69 2007/12/12 10:06:29 acolla
31in AliShuttle.cxx: SHUTTLE logbook is updated in case of invalid run times:
32
33time_start==0 && time_end==0
34
35logbook is NOT updated if time_start != 0 && time_end == 0, because it may mean that the run is still ongoing.
36
1abfbb60 37Revision 1.68 2007/12/11 10:15:17 acolla
38Added marking SHUTTLE=DONE for invalid runs
39(invalid start time or end time) and runs with totalEvents < 1
40
b0e53b15 41Revision 1.67 2007/12/07 19:14:36 acolla
42in AliShuttleTrigger:
43
44Added automatic collection of new runs on a regular time basis (settable from the configuration)
45
46in AliShuttleConfig: new members
47
48- triggerWait: time to wait for DIM trigger (s) before starting automatic collection of new runs
49- mode: run mode (test, prod) -> used to build log folder (logs or logs_PROD)
50
51in AliShuttle:
52
53- logs now stored in logs/#RUN/DET_#RUN.log
54
7d4cf768 55Revision 1.66 2007/12/05 10:45:19 jgrosseo
56changed order of arguments to TMonaLisaWriter
57
468df1ce 58Revision 1.65 2007/11/26 16:58:37 acolla
59Monalisa configuration added: host and table name
60
b832ec02 61Revision 1.64 2007/11/13 16:15:47 acolla
62DCS map is stored in a file in the temp folder where the detector is processed.
63If the preprocessor fails, the temp folder is not removed. This will help the debugging of the problem.
64
b1d18693 65Revision 1.63 2007/11/02 10:53:16 acolla
66Protection added to AliShuttle::CopyFileLocally
67
7d43a416 68Revision 1.62 2007/10/31 18:23:13 acolla
69Furter developement on the Shuttle:
70
71- Shuttle now connects to the Grid as alidaq. The OCDB and Reference folders
72are now built from /alice/data, e.g.:
73/alice/data/2007/LHC07a/OCDB
74
75the year and LHC period are taken from the Shuttle.
76Raw metadata files are stored by GRP to:
77/alice/data/2007/LHC07a/<runNb>/Raw/RunMetadata.root
78
79- Shuttle sends a mail to DCS experts each time DP retrieval fails.
80
675f64cd 81Revision 1.61 2007/10/30 20:33:51 acolla
82Improved managing of temporary folders, which weren't correctly handled.
83Resolved bug introduced in StoreReferenceFile, which caused SPD preprocessor fail.
84
d524ade6 85Revision 1.60 2007/10/29 18:06:16 acolla
86
87New function StoreRunMetadataFile added to preprocessor and Shuttle interface
88This function can be used by GRP only. It stores raw data tags merged file to the
89raw data folder (e.g. /alice/data/2008/LHC08a/000099999/Raw).
90
91KNOWN ISSUES:
92
931. Shuttle cannot write to /alice/data/ because it belongs to alidaq. Tag file is stored in /alice/simulation/... for the time being.
942. Due to a bug in TAlien::Mkdir, the creation of a folder in recursive mode (-p option) does not work. The problem
95has been corrected in the root package on the Shuttle machine.
96
c88ad5db 97Revision 1.59 2007/10/05 12:40:55 acolla
98
99Result error code added to AliDCSClient data members (it was "lost" with the new implementation of TMap* GetAliasValues and GetDPValues).
100
1790d4b7 101Revision 1.58 2007/09/28 15:27:40 acolla
102
103AliDCSClient "multiSplit" option added in the DCS configuration
104in AliDCSMessage: variable MAX_BODY_SIZE set to 500000
105
542b6cc8 106Revision 1.57 2007/09/27 16:53:13 acolla
107Detectors can have more than one AMANDA server. SHUTTLE queries the servers sequentially,
108merges the dcs aliases/DPs in one TMap and sends it to the preprocessor.
109
1d172743 110Revision 1.56 2007/09/14 16:46:14 jgrosseo
1111) Connect and Close are called before and after each query, so one can
112keep the same AliDCSClient object.
1132) The splitting of a query is moved to GetDPValues/GetAliasValues.
1143) Splitting interval can be specified in constructor
115
b41b252a 116Revision 1.55 2007/08/06 12:26:40 acolla
117Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
118read from the run logbook.
119
4859271b 120Revision 1.54 2007/07/12 09:51:25 jgrosseo
121removed duplicated log message in GetFile
122
4f0749a8 123Revision 1.53 2007/07/12 09:26:28 jgrosseo
124updating hlt fxs base path
125
42fde080 126Revision 1.52 2007/07/12 08:06:45 jgrosseo
127adding log messages in getfile... functions
128adding not implemented copy constructor in alishuttleconfigholder
129
1bcd28db 130Revision 1.51 2007/07/03 17:24:52 acolla
131root moved to v5-16-00. TFileMerger->Cp moved to TFile::Cp.
132
a986b218 133Revision 1.50 2007/07/02 17:19:32 acolla
134preprocessor is run in a temp directory that is removed when process is finished.
135
5bac2bde 136Revision 1.49 2007/06/29 10:45:06 acolla
137Number of columns in MySql Shuttle logbook increased by one (HLT added)
138
db99d43e 139Revision 1.48 2007/06/21 13:06:19 acolla
140GetFileSources returns dummy list with 1 source if system=DCS (better than
141returning error as it was)
142
6297b37d 143Revision 1.47 2007/06/19 17:28:56 acolla
144HLT updated; missing map bug removed.
145
dc25836b 146Revision 1.46 2007/06/09 13:01:09 jgrosseo
147Switching to retrieval of several DCS DPs at a time (multiDPrequest)
148
a038aa70 149Revision 1.45 2007/05/30 06:35:20 jgrosseo
150Adding functionality to the Shuttle/TestShuttle:
151o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
152o) Function to retrieve list of IDs for a given source (GetFileIDs)
153These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
154Example code has been added to the TestProcessor in TestShuttle
155
4a33bdd9 156Revision 1.44 2007/05/11 16:09:32 acolla
157Reference files for ITS, MUON and PHOS are now stored in OfflineDetName/OnlineDetName/run_...
158example: ITS/SPD/100_filename.root
159
2d9019b4 160Revision 1.43 2007/05/10 09:59:51 acolla
161Various bug fixes in StoreRefFilesToGrid; Cleaning of reference storage before processing detector (CleanReferenceStorage)
162
546242fb 163Revision 1.42 2007/05/03 08:01:39 jgrosseo
164typo in last commit :-(
165
8b739301 166Revision 1.41 2007/05/03 08:00:48 jgrosseo
167fixing log message when pp want to skip dcs value retrieval
168
651fdaab 169Revision 1.40 2007/04/27 07:06:48 jgrosseo
170GetFileSources returns empty list in case of no files, but successful query
171No mails sent in testmode
172
86aa42c3 173Revision 1.39 2007/04/17 12:43:57 acolla
174Correction in StoreOCDB; change of text in mail to detector expert
175
26758fce 176Revision 1.38 2007/04/12 08:26:18 jgrosseo
177updated comment
178
3c2a21c8 179Revision 1.37 2007/04/10 16:53:14 jgrosseo
180redirecting sub detector stdout, stderr to sub detector log file
181
3d8bc902 182Revision 1.35 2007/04/04 16:26:38 acolla
1831. Re-organization of function calls in TestPreprocessor to make it more meaningful.
1842. Added missing dependency in test preprocessors.
1853. in AliShuttle.cxx: processing time and memory consumption info on a single line.
186
886d60e6 187Revision 1.34 2007/04/04 10:33:36 jgrosseo
1881) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
189In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
190
1912) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
192
1933) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
194
1954) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
196
1975) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
198If you always need DCS data (like before), you do not need to implement it.
199
2006) The run type has been added to the monitoring page
201
9827400b 202Revision 1.33 2007/04/03 13:56:01 acolla
203Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
204run type.
205
3301427a 206Revision 1.32 2007/02/28 10:41:56 acolla
207Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
208AliPreprocessor::GetRunType() function.
209Added some ldap definition files.
210
d386d623 211Revision 1.30 2007/02/13 11:23:21 acolla
212Moved getters and setters of Shuttle's main OCDB/Reference, local
213OCDB/Reference, temp and log folders to AliShuttleInterface
214
9d733021 215Revision 1.27 2007/01/30 17:52:42 jgrosseo
216adding monalisa monitoring
217
e7f62f16 218Revision 1.26 2007/01/23 19:20:03 acolla
219Removed old ldif files, added TOF, MCH ldif files. Added some options in
220AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
221SetShuttleLogDir
222
36c99a6a 223Revision 1.25 2007/01/15 19:13:52 acolla
224Moved some AliInfo to AliDebug in SendMail function
225
fc5a4708 226Revision 1.21 2006/12/07 08:51:26 jgrosseo
227update (alberto):
228table, db names in ldap configuration
229added GRP preprocessor
230DCS data can also be retrieved by data point
231
2c15234c 232Revision 1.20 2006/11/16 16:16:48 jgrosseo
233introducing strict run ordering flag
234removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
235
be48e3ea 236Revision 1.19 2006/11/06 14:23:04 jgrosseo
237major update (Alberto)
238o) reading of run parameters from the logbook
239o) online offline naming conversion
240o) standalone DCSclient package
241
eba76848 242Revision 1.18 2006/10/20 15:22:59 jgrosseo
243o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
244o) Merging Collect, CollectAll, CollectNew function
245o) Removing implementation of empty copy constructors (declaration still there!)
246
cb343cfd 247Revision 1.17 2006/10/05 16:20:55 jgrosseo
248adapting to new CDB classes
249
6ec0e06c 250Revision 1.16 2006/10/05 15:46:26 jgrosseo
251applying to the new interface
252
481441a2 253Revision 1.15 2006/10/02 16:38:39 jgrosseo
254update (alberto):
255fixed memory leaks
256storing of objects that failed to be stored to the grid before
257interfacing of shuttle status table in daq system
258
2bb7b766 259Revision 1.14 2006/08/29 09:16:05 jgrosseo
260small update
261
85a80aa9 262Revision 1.13 2006/08/15 10:50:00 jgrosseo
263effc++ corrections (alberto)
264
4f0ab988 265Revision 1.12 2006/08/08 14:19:29 jgrosseo
266Update to shuttle classes (Alberto)
267
268- Possibility to set the full object's path in the Preprocessor's and
269Shuttle's Store functions
270- Possibility to extend the object's run validity in the same classes
271("startValidity" and "validityInfinite" parameters)
272- Implementation of the StoreReferenceData function to store reference
273data in a dedicated CDB storage.
274
84090f85 275Revision 1.11 2006/07/21 07:37:20 jgrosseo
276last run is stored after each run
277
7bfb2090 278Revision 1.10 2006/07/20 09:54:40 jgrosseo
279introducing status management: The processing per subdetector is divided into several steps,
280after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
281can keep track of the number of failures and skips further processing after a certain threshold is
282exceeded. These thresholds can be configured in LDAP.
283
5164a766 284Revision 1.9 2006/07/19 10:09:55 jgrosseo
285new configuration, accesst to DAQ FES (Alberto)
286
57f50b3c 287Revision 1.8 2006/07/11 12:44:36 jgrosseo
288adding parameters for extended validity range of data produced by preprocessor
289
17111222 290Revision 1.7 2006/07/10 14:37:09 jgrosseo
291small fix + todo comment
292
e090413b 293Revision 1.6 2006/07/10 13:01:41 jgrosseo
294enhanced storing of last sucessfully processed run (alberto)
295
a7160fe9 296Revision 1.5 2006/07/04 14:59:57 jgrosseo
297revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
298
45a493ce 299Revision 1.4 2006/06/12 09:11:16 jgrosseo
300coding conventions (Alberto)
301
58bc3020 302Revision 1.3 2006/06/06 14:26:40 jgrosseo
303o) removed files that were moved to STEER
304o) shuttle updated to follow the new interface (Alberto)
305
b948db8d 306Revision 1.2 2006/03/07 07:52:34 hristov
307New version (B.Yordanov)
308
d477ad88 309Revision 1.6 2005/11/19 17:19:14 byordano
310RetrieveDATEEntries and RetrieveConditionsData added
311
312Revision 1.5 2005/11/19 11:09:27 byordano
313AliShuttle declaration added
314
315Revision 1.4 2005/11/17 17:47:34 byordano
316TList changed to TObjArray
317
318Revision 1.3 2005/11/17 14:43:23 byordano
319import to local CVS
320
321Revision 1.1.1.1 2005/10/28 07:33:58 hristov
322Initial import as subdirectory in AliRoot
323
73abe331 324Revision 1.2 2005/09/13 08:41:15 byordano
325default startTime endTime added
326
327Revision 1.4 2005/08/30 09:13:02 byordano
328some docs added
329
330Revision 1.3 2005/08/29 21:15:47 byordano
331some docs added
332
333*/
334
335//
336// This class is the main manager for AliShuttle.
337// It organizes the data retrieval from DCS and call the
b948db8d 338// interface methods of AliPreprocessor.
73abe331 339// For every detector in AliShuttleConfgi (see AliShuttleConfig),
340// data for its set of aliases is retrieved. If there is registered
b948db8d 341// AliPreprocessor for this detector then it will be used
342// accroding to the schema (see AliPreprocessor).
343// If there isn't registered AliPreprocessor than the retrieved
73abe331 344// data is stored automatically to the undelying AliCDBStorage.
345// For detSpec is used the alias name.
346//
347
348#include "AliShuttle.h"
349
350#include "AliCDBManager.h"
351#include "AliCDBStorage.h"
352#include "AliCDBId.h"
84090f85 353#include "AliCDBRunRange.h"
354#include "AliCDBPath.h"
5164a766 355#include "AliCDBEntry.h"
73abe331 356#include "AliShuttleConfig.h"
eba76848 357#include "DCSClient/AliDCSClient.h"
73abe331 358#include "AliLog.h"
b948db8d 359#include "AliPreprocessor.h"
5164a766 360#include "AliShuttleStatus.h"
2bb7b766 361#include "AliShuttleLogbookEntry.h"
73abe331 362
57f50b3c 363#include <TSystem.h>
58bc3020 364#include <TObject.h>
b948db8d 365#include <TString.h>
57f50b3c 366#include <TTimeStamp.h>
73abe331 367#include <TObjString.h>
57f50b3c 368#include <TSQLServer.h>
369#include <TSQLResult.h>
370#include <TSQLRow.h>
cb343cfd 371#include <TMutex.h>
9827400b 372#include <TSystemDirectory.h>
373#include <TSystemFile.h>
a986b218 374#include <TFile.h>
9827400b 375#include <TGrid.h>
376#include <TGridResult.h>
73abe331 377
e7f62f16 378#include <TMonaLisaWriter.h>
379
5164a766 380#include <fstream>
381
cb343cfd 382#include <sys/types.h>
383#include <sys/wait.h>
384
73abe331 385ClassImp(AliShuttle)
386
b948db8d 387//______________________________________________________________________________________________
388AliShuttle::AliShuttle(const AliShuttleConfig* config,
389 UInt_t timeout, Int_t retries):
4f0ab988 390fConfig(config),
391fTimeout(timeout), fRetries(retries),
392fPreprocessorMap(),
2bb7b766 393fLogbookEntry(0),
eba76848 394fCurrentDetector(),
85a80aa9 395fStatusEntry(0),
cb343cfd 396fMonitoringMutex(0),
eba76848 397fLastActionTime(0),
e7f62f16 398fLastAction(),
9827400b 399fMonaLisa(0),
400fTestMode(kNone),
ffa29e93 401fReadTestMode(kFALSE),
402fOutputRedirected(kFALSE)
73abe331 403{
404 //
405 // config: AliShuttleConfig used
73abe331 406 // timeout: timeout used for AliDCSClient connection
407 // retries: the number of retries in case of connection error.
408 //
409
57f50b3c 410 if (!fConfig->IsValid()) AliFatal("********** !!!!! Invalid configuration !!!!! **********");
be48e3ea 411 for(int iSys=0;iSys<4;iSys++) {
57f50b3c 412 fServer[iSys]=0;
be48e3ea 413 if (iSys < 3)
2c15234c 414 fFXSlist[iSys].SetOwner(kTRUE);
57f50b3c 415 }
2bb7b766 416 fPreprocessorMap.SetOwner(kTRUE);
be48e3ea 417
418 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
419 fFirstUnprocessed[iDet] = kFALSE;
420
cb343cfd 421 fMonitoringMutex = new TMutex();
58bc3020 422}
423
b948db8d 424//______________________________________________________________________________________________
57f50b3c 425AliShuttle::~AliShuttle()
58bc3020 426{
9827400b 427 //
428 // destructor
429 //
58bc3020 430
b948db8d 431 fPreprocessorMap.DeleteAll();
be48e3ea 432 for(int iSys=0;iSys<4;iSys++)
57f50b3c 433 if(fServer[iSys]) {
434 fServer[iSys]->Close();
435 delete fServer[iSys];
eba76848 436 fServer[iSys] = 0;
57f50b3c 437 }
2bb7b766 438
439 if (fStatusEntry){
440 delete fStatusEntry;
441 fStatusEntry = 0;
442 }
cb343cfd 443
444 if (fMonitoringMutex)
445 {
446 delete fMonitoringMutex;
447 fMonitoringMutex = 0;
448 }
73abe331 449}
450
b948db8d 451//______________________________________________________________________________________________
57f50b3c 452void AliShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
58bc3020 453{
73abe331 454 //
b948db8d 455 // Registers new AliPreprocessor.
73abe331 456 // It uses GetName() for indentificator of the pre processor.
457 // The pre processor is registered it there isn't any other
458 // with the same identificator (GetName()).
459 //
460
eba76848 461 const char* detName = preprocessor->GetName();
462 if(GetDetPos(detName) < 0)
463 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
464
465 if (fPreprocessorMap.GetValue(detName)) {
466 AliWarning(Form("AliPreprocessor %s is already registered!", detName));
73abe331 467 return;
468 }
469
eba76848 470 fPreprocessorMap.Add(new TObjString(detName), preprocessor);
73abe331 471}
b948db8d 472//______________________________________________________________________________________________
3301427a 473Bool_t AliShuttle::Store(const AliCDBPath& path, TObject* object,
84090f85 474 AliCDBMetaData* metaData, Int_t validityStart, Bool_t validityInfinite)
73abe331 475{
9827400b 476 // Stores a CDB object in the storage for offline reconstruction. Objects that are not needed for
477 // offline reconstruction, but should be stored anyway (e.g. for debugging) should NOT be stored
478 // using this function. Use StoreReferenceData instead!
479 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
480 // finishes the data are transferred to the main storage (Grid).
b948db8d 481
3301427a 482 return StoreLocally(fgkLocalCDB, path, object, metaData, validityStart, validityInfinite);
84090f85 483}
484
485//______________________________________________________________________________________________
3301427a 486Bool_t AliShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 487{
9827400b 488 // Stores a CDB object in the storage for reference data. This objects will not be available during
489 // offline reconstrunction. Use this function for reference data only!
490 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
491 // finishes the data are transferred to the main storage (Grid).
85a80aa9 492
3301427a 493 return StoreLocally(fgkLocalRefStorage, path, object, metaData);
85a80aa9 494}
495
496//______________________________________________________________________________________________
3301427a 497Bool_t AliShuttle::StoreLocally(const TString& localUri,
85a80aa9 498 const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
499 Int_t validityStart, Bool_t validityInfinite)
500{
9827400b 501 // Store object temporarily in local storage. Parameters are passed by Store and StoreReferenceData functions.
502 // when the preprocessor finishes the data are transferred to the main storage (Grid).
503 // The parameters are:
504 // 1) Uri of the backup storage (Local)
505 // 2) the object's path.
506 // 3) the object to be stored
507 // 4) the metaData to be associated with the object
508 // 5) the validity start run number w.r.t. the current run,
509 // if the data is valid only for this run leave the default 0
510 // 6) specifies if the calibration data is valid for infinity (this means until updated),
511 // typical for calibration runs, the default is kFALSE
512 //
513 // returns 0 if fail, 1 otherwise
84090f85 514
9827400b 515 if (fTestMode & kErrorStorage)
516 {
517 Log(fCurrentDetector, "StoreLocally - In TESTMODE - Simulating error while storing locally");
518 return kFALSE;
519 }
520
3301427a 521 const char* cdbType = (localUri == fgkLocalCDB) ? "CDB" : "Reference";
2bb7b766 522
85a80aa9 523 Int_t firstRun = GetCurrentRun() - validityStart;
84090f85 524 if(firstRun < 0) {
9827400b 525 AliWarning("First valid run happens to be less than 0! Setting it to 0.");
84090f85 526 firstRun=0;
527 }
528
529 Int_t lastRun = -1;
530 if(validityInfinite) {
531 lastRun = AliCDBRunRange::Infinity();
532 } else {
533 lastRun = GetCurrentRun();
534 }
535
3301427a 536 // Version is set to current run, it will be used later to transfer data to Grid
537 AliCDBId id(path, firstRun, lastRun, GetCurrentRun(), -1);
2bb7b766 538
539 if(! dynamic_cast<TObjString*> (metaData->GetProperty("RunUsed(TObjString)"))){
540 TObjString runUsed = Form("%d", GetCurrentRun());
9e080f92 541 metaData->SetProperty("RunUsed(TObjString)", runUsed.Clone());
2bb7b766 542 }
84090f85 543
3301427a 544 Bool_t result = kFALSE;
84090f85 545
3301427a 546 if (!(AliCDBManager::Instance()->GetStorage(localUri))) {
547 Log("SHUTTLE", Form("StoreLocally - Cannot activate local %s storage", cdbType));
84090f85 548 } else {
3301427a 549 result = AliCDBManager::Instance()->GetStorage(localUri)
84090f85 550 ->Put(object, id, metaData);
551 }
552
553 if(!result) {
554
9827400b 555 Log(fCurrentDetector, Form("StoreLocally - Can't store object <%s>!", id.ToString().Data()));
3301427a 556 }
2bb7b766 557
3301427a 558 return result;
559}
84090f85 560
3301427a 561//______________________________________________________________________________________________
562Bool_t AliShuttle::StoreOCDB()
563{
9827400b 564 //
565 // Called when preprocessor ends successfully or when previous storage attempt failed (kStoreError status)
566 // Calls underlying StoreOCDB(const char*) function twice, for OCDB and Reference storage.
567 // Then calls StoreRefFilesToGrid to store reference files.
568 //
569
570 if (fTestMode & kErrorGrid)
571 {
572 Log("SHUTTLE", "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
573 Log(fCurrentDetector, "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
574 return kFALSE;
575 }
576
c88ad5db 577 Log("SHUTTLE","StoreOCDB - Storing OCDB data ...");
86aa42c3 578 Bool_t resultCDB = StoreOCDB(fgkMainCDB);
579
c88ad5db 580 Log("SHUTTLE","StoreOCDB - Storing reference data ...");
3301427a 581 Bool_t resultRef = StoreOCDB(fgkMainRefStorage);
9827400b 582
c88ad5db 583 Log("SHUTTLE","StoreOCDB - Storing reference files ...");
584 Bool_t resultRefFiles = CopyFilesToGrid("reference");
585
586 Bool_t resultMetadata = kTRUE;
587 if(fCurrentDetector == "GRP")
588 {
589 Log("StoreOCDB - SHUTTLE","Storing Run Metadata file ...");
590 resultMetadata = CopyFilesToGrid("metadata");
591 }
9827400b 592
c88ad5db 593 return resultCDB && resultRef && resultRefFiles && resultMetadata;
3301427a 594}
595
596//______________________________________________________________________________________________
597Bool_t AliShuttle::StoreOCDB(const TString& gridURI)
598{
599 //
600 // Called by StoreOCDB(), performs actual storage to the main OCDB and reference storages (Grid)
601 //
602
603 TObjArray* gridIds=0;
604
605 Bool_t result = kTRUE;
606
607 const char* type = 0;
608 TString localURI;
609 if(gridURI == fgkMainCDB) {
610 type = "OCDB";
611 localURI = fgkLocalCDB;
612 } else if(gridURI == fgkMainRefStorage) {
613 type = "reference";
614 localURI = fgkLocalRefStorage;
615 } else {
616 AliError(Form("Invalid storage URI: %s", gridURI.Data()));
617 return kFALSE;
618 }
619
620 AliCDBManager* man = AliCDBManager::Instance();
621
622 AliCDBStorage *gridSto = man->GetStorage(gridURI);
623 if(!gridSto) {
624 Log("SHUTTLE",
625 Form("StoreOCDB - cannot activate main %s storage", type));
626 return kFALSE;
627 }
628
629 gridIds = gridSto->GetQueryCDBList();
630
631 // get objects previously stored in local CDB
632 AliCDBStorage *localSto = man->GetStorage(localURI);
633 if(!localSto) {
634 Log("SHUTTLE",
635 Form("StoreOCDB - cannot activate local %s storage", type));
636 return kFALSE;
637 }
638 AliCDBPath aPath(GetOfflineDetName(fCurrentDetector.Data()),"*","*");
639 // Local objects were stored with current run as Grid version!
640 TList* localEntries = localSto->GetAll(aPath.GetPath(), GetCurrentRun(), GetCurrentRun());
641 localEntries->SetOwner(1);
642
643 // loop on local stored objects
644 TIter localIter(localEntries);
645 AliCDBEntry *aLocEntry = 0;
646 while((aLocEntry = dynamic_cast<AliCDBEntry*> (localIter.Next()))){
647 aLocEntry->SetOwner(1);
648 AliCDBId aLocId = aLocEntry->GetId();
649 aLocEntry->SetVersion(-1);
650 aLocEntry->SetSubVersion(-1);
651
652 // If local object is valid up to infinity we store it only if it is
653 // the first unprocessed run!
654 if (aLocId.GetLastRun() == AliCDBRunRange::Infinity() &&
655 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
656 {
657 Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
658 "there are previous unprocessed runs!",
659 fCurrentDetector.Data(), aLocId.GetPath().Data()));
4508ab8b 660 result = kFALSE;
3301427a 661 continue;
662 }
663
664 // loop on Grid valid Id's
665 Bool_t store = kTRUE;
666 TIter gridIter(gridIds);
667 AliCDBId* aGridId = 0;
668 while((aGridId = dynamic_cast<AliCDBId*> (gridIter.Next()))){
669 if(aGridId->GetPath() != aLocId.GetPath()) continue;
670 // skip all objects valid up to infinity
671 if(aGridId->GetLastRun() == AliCDBRunRange::Infinity()) continue;
672 // if we get here, it means there's already some more recent object stored on Grid!
673 store = kFALSE;
674 break;
675 }
676
677 // If we get here, the file can be stored!
678 Bool_t storeOk = gridSto->Put(aLocEntry);
679 if(!store || storeOk){
680
681 if (!store)
682 {
683 Log(fCurrentDetector.Data(),
684 Form("StoreOCDB - A more recent object already exists in %s storage: <%s>",
685 type, aGridId->ToString().Data()));
686 } else {
687 Log("SHUTTLE",
688 Form("StoreOCDB - Object <%s> successfully put into %s storage",
689 aLocId.ToString().Data(), type));
2d9019b4 690 Log(fCurrentDetector.Data(),
691 Form("StoreOCDB - Object <%s> successfully put into %s storage",
692 aLocId.ToString().Data(), type));
3301427a 693 }
84090f85 694
3301427a 695 // removing local filename...
696 TString filename;
697 localSto->IdToFilename(aLocId, filename);
c88ad5db 698 Log("SHUTTLE", Form("StoreOCDB - Removing local file %s", filename.Data()));
3301427a 699 RemoveFile(filename.Data());
700 continue;
701 } else {
702 Log("SHUTTLE",
703 Form("StoreOCDB - Grid %s storage of object <%s> failed",
704 type, aLocId.ToString().Data()));
2d9019b4 705 Log(fCurrentDetector.Data(),
706 Form("StoreOCDB - Grid %s storage of object <%s> failed",
707 type, aLocId.ToString().Data()));
3301427a 708 result = kFALSE;
b948db8d 709 }
710 }
3301427a 711 localEntries->Clear();
2bb7b766 712
b948db8d 713 return result;
3301427a 714}
715
716//______________________________________________________________________________________________
546242fb 717Bool_t AliShuttle::CleanReferenceStorage(const char* detector)
718{
2d9019b4 719 // clears the directory used to store reference files of a given subdetector
546242fb 720
721 AliCDBManager* man = AliCDBManager::Instance();
722 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
2d9019b4 723 TString localBaseFolder = sto->GetBaseFolder();
724
725 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
726
d524ade6 727 Log("SHUTTLE", Form("CleanReferenceStorage - Cleaning %s", targetDir.Data()));
2d9019b4 728
729 TString begin;
730 begin.Form("%d_", GetCurrentRun());
731
732 TSystemDirectory* baseDir = new TSystemDirectory("/", targetDir);
733 if (!baseDir)
734 return kTRUE;
735
736 TList* dirList = baseDir->GetListOfFiles();
737 delete baseDir;
738
739 if (!dirList) return kTRUE;
740
741 if (dirList->GetEntries() < 3)
742 {
743 delete dirList;
744 return kTRUE;
745 }
746
747 Int_t nDirs = 0, nDel = 0;
748 TIter dirIter(dirList);
749 TSystemFile* entry = 0;
546242fb 750
2d9019b4 751 Bool_t success = kTRUE;
546242fb 752
2d9019b4 753 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
754 {
755 if (entry->IsDirectory())
756 continue;
757
758 TString fileName(entry->GetName());
759 if (!fileName.BeginsWith(begin))
760 continue;
761
762 nDirs++;
763
764 // delete file
765 Int_t result = gSystem->Unlink(fileName.Data());
766
767 if (result)
768 {
d524ade6 769 Log("SHUTTLE", Form("CleanReferenceStorage - Could not delete file %s!", fileName.Data()));
2d9019b4 770 success = kFALSE;
771 } else {
772 nDel++;
773 }
774 }
775
776 if(nDirs > 0)
777 Log("SHUTTLE", Form("CleanReferenceStorage - %d (over %d) reference files in folder %s were deleted.",
778 nDel, nDirs, targetDir.Data()));
779
780
781 delete dirList;
782 return success;
783
784
785
786
787
546242fb 788
789 Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
790 if (result == 0)
791 {
792 // delete directory
d524ade6 793 result = gSystem->Exec(Form("rm -rf %s", targetDir.Data()));
546242fb 794 if (result != 0)
795 {
d524ade6 796 Log("SHUTTLE", Form("CleanReferenceStorage - Could not clean directory %s", targetDir.Data()));
546242fb 797 return kFALSE;
798 }
799 }
800
801 result = gSystem->mkdir(targetDir, kTRUE);
802 if (result != 0)
803 {
c88ad5db 804 Log("SHUTTLE", Form("CleanReferenceStorage - Error creating base directory %s", targetDir.Data()));
546242fb 805 return kFALSE;
806 }
807
808 return kTRUE;
809}
810
811//______________________________________________________________________________________________
9827400b 812Bool_t AliShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
813{
814 //
3c2a21c8 815 // Stores reference file directly (without opening it). This function stores the file locally.
9827400b 816 //
3c2a21c8 817 // The file is stored under the following location:
818 // <base folder of local reference storage>/<DET>/<RUN#>_<gridFileName>
819 // where <gridFileName> is the second parameter given to the function
820 //
9827400b 821
822 if (fTestMode & kErrorStorage)
823 {
824 Log(fCurrentDetector, "StoreReferenceFile - In TESTMODE - Simulating error while storing locally");
825 return kFALSE;
826 }
827
828 AliCDBManager* man = AliCDBManager::Instance();
829 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
830
831 TString localBaseFolder = sto->GetBaseFolder();
832
d524ade6 833 TString target = GetRefFilePrefix(localBaseFolder.Data(), detector);
834 target.Append(Form("/%d_%s", GetCurrentRun(), gridFileName));
9827400b 835
d524ade6 836 return CopyFileLocally(localFile, target);
c88ad5db 837}
838
839//______________________________________________________________________________________________
840Bool_t AliShuttle::StoreRunMetadataFile(const char* localFile, const char* gridFileName)
841{
842 //
843 // Stores Run metadata file to the Grid, in the run folder
844 //
845 // Only GRP can call this function.
846
847 if (fTestMode & kErrorStorage)
848 {
849 Log(fCurrentDetector, "StoreRunMetaDataFile - In TESTMODE - Simulating error while storing locally");
850 return kFALSE;
851 }
852
853 AliCDBManager* man = AliCDBManager::Instance();
854 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
855
856 TString localBaseFolder = sto->GetBaseFolder();
857
858 // Build Run level folder
859 // folder = /alice/data/year/lhcPeriod/runNb/Raw
860
c88ad5db 861
675f64cd 862 TString lhcPeriod = GetLHCPeriod();
c88ad5db 863 if (lhcPeriod.Length() == 0)
864 {
865 Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
866 return 0;
867 }
868
675f64cd 869 TString target = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw/%s",
870 localBaseFolder.Data(), GetCurrentYear(),
d524ade6 871 lhcPeriod.Data(), GetCurrentRun(), gridFileName);
c88ad5db 872
d524ade6 873 return CopyFileLocally(localFile, target);
c88ad5db 874}
875
876//______________________________________________________________________________________________
d524ade6 877Bool_t AliShuttle::CopyFileLocally(const char* localFile, const TString& target)
c88ad5db 878{
879 //
880 // Stores file locally. Called by StoreReferenceFile and StoreRunMetadataFile
d524ade6 881 // Files are temporarily stored in the local reference storage. When the preprocessor
882 // finishes, the Shuttle calls CopyFilesToGrid to transfer the files to AliEn
883 // (in reference or run level folders)
c88ad5db 884 //
885
d524ade6 886 TString targetDir(target(0, target.Last('/')));
887
888 //try to open base dir folder, if it does not exist
2d9019b4 889 void* dir = gSystem->OpenDirectory(targetDir.Data());
890 if (dir == NULL) {
891 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
c88ad5db 892 Log("SHUTTLE", Form("StoreFileLocally - Can't open directory <%s>", targetDir.Data()));
2d9019b4 893 return kFALSE;
894 }
895
896 } else {
897 gSystem->FreeDirectory(dir);
898 }
9827400b 899
7d43a416 900 Int_t result = 0;
901
902 result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
9827400b 903 if (result)
904 {
c88ad5db 905 Log("SHUTTLE", Form("StoreFileLocally - %s does not exist", localFile));
546242fb 906 return kFALSE;
9827400b 907 }
546242fb 908
7d43a416 909 result = gSystem->GetPathInfo(target, 0, (Long64_t*) 0, 0, 0);
910 if (!result)
911 {
912 Log("SHUTTLE", Form("StoreFileLocally - target file %s already exist, removing...", target.Data()));
913 if (gSystem->Unlink(target.Data()))
914 {
915 Log("SHUTTLE", Form("StoreFileLocally - Could not remove existing target file %s!", target.Data()));
916 return kFALSE;
917 }
918 }
919
9827400b 920 result = gSystem->CopyFile(localFile, target);
921
922 if (result == 0)
923 {
c88ad5db 924 Log("SHUTTLE", Form("StoreFileLocally - File %s stored locally to %s", localFile, target.Data()));
9827400b 925 return kTRUE;
926 }
927 else
928 {
d524ade6 929 Log("SHUTTLE", Form("StoreFileLocally - Could not store file %s to %s! Error code = %d",
546242fb 930 localFile, target.Data(), result));
9827400b 931 return kFALSE;
932 }
c88ad5db 933
934
935
9827400b 936}
937
938//______________________________________________________________________________________________
c88ad5db 939Bool_t AliShuttle::CopyFilesToGrid(const char* type)
9827400b 940{
941 //
c88ad5db 942 // Transfers local files to the Grid. Local files can be reference files
943 // or run metadata file (from GRP only).
9827400b 944 //
c88ad5db 945 // According to the type (ref, metadata) the files are stored under the following location:
946 // ref --> <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
947 // metadata --> <run data folder>/<MetadataFileName>
86aa42c3 948 //
c88ad5db 949
9827400b 950 AliCDBManager* man = AliCDBManager::Instance();
951 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
952 if (!sto)
953 return kFALSE;
954 TString localBaseFolder = sto->GetBaseFolder();
9827400b 955
c88ad5db 956 TString dir;
957 TString alienDir;
9827400b 958 TString begin;
9827400b 959
c88ad5db 960 if (strcmp(type, "reference") == 0)
961 {
962 dir = GetRefFilePrefix(localBaseFolder.Data(), fCurrentDetector.Data());
963 AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
964 if (!gridSto)
965 return kFALSE;
966 TString gridBaseFolder = gridSto->GetBaseFolder();
967 alienDir = GetRefFilePrefix(gridBaseFolder.Data(), fCurrentDetector.Data());
968 begin = Form("%d_", GetCurrentRun());
969 }
970 else if (strcmp(type, "metadata") == 0)
971 {
c88ad5db 972
675f64cd 973 TString lhcPeriod = GetLHCPeriod();
c88ad5db 974
975 if (lhcPeriod.Length() == 0)
976 {
977 Log("SHUTTLE","CopyFilesToGrid - LHCPeriod not found in logbook!");
978 return 0;
979 }
980
675f64cd 981 dir = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw",
982 localBaseFolder.Data(), GetCurrentYear(),
c88ad5db 983 lhcPeriod.Data(), GetCurrentRun());
675f64cd 984 alienDir = dir(dir.Index("/alice/data/"), dir.Length());
985
c88ad5db 986 begin = "";
987 }
988 else
989 {
990 Log("SHUTTLE", "CopyFilesToGrid - Unexpected: type label must be reference or metadata!");
991 return kFALSE;
992 }
993
9827400b 994 TSystemDirectory* baseDir = new TSystemDirectory("/", dir);
3d8bc902 995 if (!baseDir)
996 return kTRUE;
997
2d9019b4 998 TList* dirList = baseDir->GetListOfFiles();
999 delete baseDir;
1000
1001 if (!dirList) return kTRUE;
1002
1003 if (dirList->GetEntries() < 3)
3d8bc902 1004 {
2d9019b4 1005 delete dirList;
9827400b 1006 return kTRUE;
3d8bc902 1007 }
2d9019b4 1008
546242fb 1009 if (!gGrid)
1010 {
c88ad5db 1011 Log("SHUTTLE", "CopyFilesToGrid - Connection to Grid failed: Cannot continue!");
2d9019b4 1012 delete dirList;
546242fb 1013 return kFALSE;
1014 }
1015
2d9019b4 1016 Int_t nDirs = 0, nTransfer = 0;
1017 TIter dirIter(dirList);
1018 TSystemFile* entry = 0;
1019
9827400b 1020 Bool_t success = kTRUE;
3d8bc902 1021 Bool_t first = kTRUE;
9827400b 1022
2d9019b4 1023 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
1024 {
9827400b 1025 if (entry->IsDirectory())
1026 continue;
1027
1028 TString fileName(entry->GetName());
1029 if (!fileName.BeginsWith(begin))
1030 continue;
1031
2d9019b4 1032 nDirs++;
1033
3d8bc902 1034 if (first)
1035 {
1036 first = kFALSE;
c88ad5db 1037 // check that folder exists, otherwise create it
3d8bc902 1038 TGridResult* result = gGrid->Ls(alienDir.Data(), "a");
1039
1040 if (!result)
2d9019b4 1041 {
1042 delete dirList;
3d8bc902 1043 return kFALSE;
2d9019b4 1044 }
3d8bc902 1045
546242fb 1046 if (!result->GetFileName(1)) // TODO: It looks like element 0 is always 0!!
3d8bc902 1047 {
675f64cd 1048 // TODO It does not work currently! Bug in TAliEn::Mkdir
1049 // TODO Manually fixed in local root v5-16-00
c88ad5db 1050 if (!gGrid->Mkdir(alienDir.Data(),"-p",0))
3d8bc902 1051 {
c88ad5db 1052 Log("SHUTTLE", Form("CopyFilesToGrid - Cannot create directory %s",
3d8bc902 1053 alienDir.Data()));
2d9019b4 1054 delete dirList;
3d8bc902 1055 return kFALSE;
546242fb 1056 } else {
c88ad5db 1057 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s created", alienDir.Data()));
3d8bc902 1058 }
1059
546242fb 1060 } else {
c88ad5db 1061 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s found", alienDir.Data()));
3d8bc902 1062 }
1063 }
1064
9827400b 1065 TString fullLocalPath;
1066 fullLocalPath.Form("%s/%s", dir.Data(), fileName.Data());
1067
1068 TString fullGridPath;
1069 fullGridPath.Form("alien://%s/%s", alienDir.Data(), fileName.Data());
1070
a986b218 1071 Bool_t result = TFile::Cp(fullLocalPath, fullGridPath);
9827400b 1072
1073 if (result)
1074 {
c88ad5db 1075 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s succeeded!",
1076 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1077 RemoveFile(fullLocalPath);
2d9019b4 1078 nTransfer++;
9827400b 1079 }
1080 else
1081 {
c88ad5db 1082 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s FAILED!",
1083 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1084 success = kFALSE;
1085 }
1086 }
2d9019b4 1087
c88ad5db 1088 Log("SHUTTLE", Form("CopyFilesToGrid - %d (over %d) files in folder %s copied to Grid.",
1089 nTransfer, nDirs, dir.Data()));
2d9019b4 1090
1091
1092 delete dirList;
9827400b 1093 return success;
1094}
1095
1096//______________________________________________________________________________________________
2d9019b4 1097const char* AliShuttle::GetRefFilePrefix(const char* base, const char* detector)
1098{
1099 //
1100 // Get folder name of reference files
1101 //
1102
1103 TString offDetStr(GetOfflineDetName(detector));
1104 TString dir;
1105 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
1106 {
1107 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
1108 } else {
1109 dir.Form("%s/%s", base, offDetStr.Data());
1110 }
1111
1112 return dir.Data();
1113
1114
1115}
c88ad5db 1116
2d9019b4 1117//______________________________________________________________________________________________
3301427a 1118void AliShuttle::CleanLocalStorage(const TString& uri)
1119{
9827400b 1120 //
1121 // Called in case the preprocessor is declared failed. Remove remaining objects from the local storages.
1122 //
3301427a 1123
1124 const char* type = 0;
1125 if(uri == fgkLocalCDB) {
1126 type = "OCDB";
1127 } else if(uri == fgkLocalRefStorage) {
546242fb 1128 type = "Reference";
3301427a 1129 } else {
1130 AliError(Form("Invalid storage URI: %s", uri.Data()));
1131 return;
1132 }
1133
1134 AliCDBManager* man = AliCDBManager::Instance();
b948db8d 1135
3301427a 1136 // open local storage
1137 AliCDBStorage *localSto = man->GetStorage(uri);
1138 if(!localSto) {
1139 Log("SHUTTLE",
1140 Form("CleanLocalStorage - cannot activate local %s storage", type));
1141 return;
1142 }
1143
1144 TString filename(Form("%s/%s/*/Run*_v%d_s*.root",
546242fb 1145 localSto->GetBaseFolder().Data(), GetOfflineDetName(fCurrentDetector.Data()), GetCurrentRun()));
3301427a 1146
c88ad5db 1147 AliDebug(2, Form("filename = %s", filename.Data()));
3301427a 1148
c88ad5db 1149 Log("SHUTTLE", Form("Removing remaining local files for run %d and detector %s ...",
3301427a 1150 GetCurrentRun(), fCurrentDetector.Data()));
1151
1152 RemoveFile(filename.Data());
1153
1154}
1155
1156//______________________________________________________________________________________________
1157void AliShuttle::RemoveFile(const char* filename)
1158{
9827400b 1159 //
1160 // removes local file
1161 //
3301427a 1162
1163 TString command(Form("rm -f %s", filename));
1164
1165 Int_t result = gSystem->Exec(command.Data());
1166 if(result != 0)
1167 {
1168 Log("SHUTTLE", Form("RemoveFile - %s: Cannot remove file %s!",
1169 fCurrentDetector.Data(), filename));
1170 }
73abe331 1171}
1172
b948db8d 1173//______________________________________________________________________________________________
5164a766 1174AliShuttleStatus* AliShuttle::ReadShuttleStatus()
1175{
9827400b 1176 //
1177 // Reads the AliShuttleStatus from the CDB
1178 //
5164a766 1179
2bb7b766 1180 if (fStatusEntry){
1181 delete fStatusEntry;
1182 fStatusEntry = 0;
1183 }
5164a766 1184
10a5a932 1185 fStatusEntry = AliCDBManager::Instance()->GetStorage(GetLocalCDB())
2bb7b766 1186 ->Get(Form("/SHUTTLE/STATUS/%s", fCurrentDetector.Data()), GetCurrentRun());
5164a766 1187
2bb7b766 1188 if (!fStatusEntry) return 0;
1189 fStatusEntry->SetOwner(1);
5164a766 1190
2bb7b766 1191 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1192 if (!status) {
1193 AliError("Invalid object stored to CDB!");
1194 return 0;
1195 }
5164a766 1196
2bb7b766 1197 return status;
5164a766 1198}
1199
1200//______________________________________________________________________________________________
7bfb2090 1201Bool_t AliShuttle::WriteShuttleStatus(AliShuttleStatus* status)
5164a766 1202{
9827400b 1203 //
1204 // writes the status for one subdetector
1205 //
2bb7b766 1206
1207 if (fStatusEntry){
1208 delete fStatusEntry;
1209 fStatusEntry = 0;
1210 }
5164a766 1211
2bb7b766 1212 Int_t run = GetCurrentRun();
5164a766 1213
2bb7b766 1214 AliCDBId id(AliCDBPath("SHUTTLE", "STATUS", fCurrentDetector), run, run);
5164a766 1215
2bb7b766 1216 fStatusEntry = new AliCDBEntry(status, id, new AliCDBMetaData);
1217 fStatusEntry->SetOwner(1);
5164a766 1218
2bb7b766 1219 UInt_t result = AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
7bfb2090 1220
2bb7b766 1221 if (!result) {
3301427a 1222 Log("SHUTTLE", Form("WriteShuttleStatus - Failed for %s, run %d",
1223 fCurrentDetector.Data(), run));
2bb7b766 1224 return kFALSE;
1225 }
e7f62f16 1226
1227 SendMLInfo();
7bfb2090 1228
2bb7b766 1229 return kTRUE;
5164a766 1230}
1231
1232//______________________________________________________________________________________________
1233void AliShuttle::UpdateShuttleStatus(AliShuttleStatus::Status newStatus, Bool_t increaseCount)
1234{
9827400b 1235 //
1236 // changes the AliShuttleStatus for the given detector and run to the given status
1237 //
5164a766 1238
2bb7b766 1239 if (!fStatusEntry){
1240 AliError("UNEXPECTED: fStatusEntry empty");
1241 return;
1242 }
5164a766 1243
2bb7b766 1244 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
5164a766 1245
2bb7b766 1246 if (!status){
c88ad5db 1247 Log("SHUTTLE", "UpdateShuttleStatus - UNEXPECTED: status could not be read from current CDB entry");
2bb7b766 1248 return;
1249 }
5164a766 1250
2c15234c 1251 TString actionStr = Form("UpdateShuttleStatus - %s: Changing state from %s to %s",
eba76848 1252 fCurrentDetector.Data(),
36c99a6a 1253 status->GetStatusName(),
eba76848 1254 status->GetStatusName(newStatus));
cb343cfd 1255 Log("SHUTTLE", actionStr);
1256 SetLastAction(actionStr);
5164a766 1257
2bb7b766 1258 status->SetStatus(newStatus);
1259 if (increaseCount) status->IncreaseCount();
5164a766 1260
2bb7b766 1261 AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
e7f62f16 1262
1263 SendMLInfo();
5164a766 1264}
e7f62f16 1265
1266//______________________________________________________________________________________________
1267void AliShuttle::SendMLInfo()
1268{
1269 //
1270 // sends ML information about the current status of the current detector being processed
1271 //
1272
1273 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1274
1275 if (!status){
3301427a 1276 Log("SHUTTLE", "SendMLInfo - UNEXPECTED: status could not be read from current CDB entry");
e7f62f16 1277 return;
1278 }
1279
1280 TMonaLisaText mlStatus(Form("%s_status", fCurrentDetector.Data()), status->GetStatusName());
1281 TMonaLisaValue mlRetryCount(Form("%s_count", fCurrentDetector.Data()), status->GetCount());
1282
1283 TList mlList;
1284 mlList.Add(&mlStatus);
1285 mlList.Add(&mlRetryCount);
1286
ee6f7523 1287 TString mlID;
1288 mlID.Form("%d", GetCurrentRun());
1289 fMonaLisa->SendParameters(&mlList, mlID);
e7f62f16 1290}
1291
5164a766 1292//______________________________________________________________________________________________
1293Bool_t AliShuttle::ContinueProcessing()
1294{
9827400b 1295 // this function reads the AliShuttleStatus information from CDB and
1296 // checks if the processing should be continued
1297 // if yes it returns kTRUE and updates the AliShuttleStatus with nextStatus
2bb7b766 1298
57c1a579 1299 if (!fConfig->HostProcessDetector(fCurrentDetector)) return kFALSE;
1300
1301 AliPreprocessor* aPreprocessor =
1302 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1303 if (!aPreprocessor)
1304 {
c88ad5db 1305 Log("SHUTTLE", Form("ContinueProcessing - %s: no preprocessor registered", fCurrentDetector.Data()));
57c1a579 1306 return kFALSE;
1307 }
1308
2bb7b766 1309 AliShuttleLogbookEntry::Status entryStatus =
eba76848 1310 fLogbookEntry->GetDetectorStatus(fCurrentDetector);
2bb7b766 1311
1312 if(entryStatus != AliShuttleLogbookEntry::kUnprocessed) {
c88ad5db 1313 Log("SHUTTLE", Form("ContinueProcessing - %s is %s",
2bb7b766 1314 fCurrentDetector.Data(),
1315 fLogbookEntry->GetDetectorStatusName(entryStatus)));
1316 return kFALSE;
1317 }
1318
1319 // if we get here, according to Shuttle logbook subdetector is in UNPROCESSED state
be48e3ea 1320
1321 // check if current run is first unprocessed run for current detector
1322 if (fConfig->StrictRunOrder(fCurrentDetector) &&
1323 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
1324 {
86aa42c3 1325 if (fTestMode == kNone)
1326 {
c88ad5db 1327 Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering"
1328 " but this is not the first unprocessed run!"));
86aa42c3 1329 return kFALSE;
1330 }
1331 else
1332 {
c88ad5db 1333 Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - "
1334 "Although %s requires strict run ordering "
1335 "and this is not the first unprocessed run, "
1336 "the SHUTTLE continues"));
86aa42c3 1337 }
be48e3ea 1338 }
1339
2bb7b766 1340 AliShuttleStatus* status = ReadShuttleStatus();
1341 if (!status) {
1342 // first time
1343 Log("SHUTTLE", Form("ContinueProcessing - %s: Processing first time",
1344 fCurrentDetector.Data()));
1345 status = new AliShuttleStatus(AliShuttleStatus::kStarted);
1346 return WriteShuttleStatus(status);
1347 }
1348
1349 // The following two cases shouldn't happen if Shuttle Logbook was correctly updated.
1350 // If it happens it may mean Logbook updating failed... let's do it now!
1351 if (status->GetStatus() == AliShuttleStatus::kDone ||
1352 status->GetStatus() == AliShuttleStatus::kFailed){
1353 Log("SHUTTLE", Form("ContinueProcessing - %s is already %s. Updating Shuttle Logbook",
1354 fCurrentDetector.Data(),
1355 status->GetStatusName(status->GetStatus())));
1356 UpdateShuttleLogbook(fCurrentDetector.Data(),
1357 status->GetStatusName(status->GetStatus()));
1358 return kFALSE;
1359 }
1360
3301427a 1361 if (status->GetStatus() == AliShuttleStatus::kStoreError) {
2bb7b766 1362 Log("SHUTTLE",
c88ad5db 1363 Form("ContinueProcessing - %s: Grid storage of one or more "
1364 "objects failed. Trying again now",
2bb7b766 1365 fCurrentDetector.Data()));
9827400b 1366 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1367 if (StoreOCDB()){
c88ad5db 1368 Log("SHUTTLE", Form("ContinueProcessing - %s: all objects "
1369 "successfully stored into main storage",
3301427a 1370 fCurrentDetector.Data()));
2bb7b766 1371 } else {
1372 Log("SHUTTLE",
1373 Form("ContinueProcessing - %s: Grid storage failed again",
1374 fCurrentDetector.Data()));
9827400b 1375 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
2bb7b766 1376 }
1377 return kFALSE;
1378 }
1379
1380 // if we get here, there is a restart
57c1a579 1381 Bool_t cont = kFALSE;
2bb7b766 1382
1383 // abort conditions
cb343cfd 1384 if (status->GetCount() >= fConfig->GetMaxRetries()) {
57c1a579 1385 Log("SHUTTLE", Form("ContinueProcessing - %s failed %d times in status %s - "
1386 "Updating Shuttle Logbook", fCurrentDetector.Data(),
2bb7b766 1387 status->GetCount(), status->GetStatusName()));
1388 UpdateShuttleLogbook(fCurrentDetector.Data(), "FAILED");
e7f62f16 1389 UpdateShuttleStatus(AliShuttleStatus::kFailed);
3301427a 1390
1391 // there may still be objects in local OCDB and reference storage
1392 // and FXS databases may be not updated: do it now!
9827400b 1393
1394 // TODO Currently disabled, we want to keep files in case of failure!
1395 // CleanLocalStorage(fgkLocalCDB);
1396 // CleanLocalStorage(fgkLocalRefStorage);
1397 // UpdateTableFailCase();
1398
1399 // Send mail to detector expert!
c88ad5db 1400 Log("SHUTTLE", Form("ContinueProcessing - Sending mail to %s expert...",
1401 fCurrentDetector.Data()));
9827400b 1402 if (!SendMail())
1403 Log("SHUTTLE", Form("ContinueProcessing - Could not send mail to %s expert",
1404 fCurrentDetector.Data()));
3301427a 1405
57c1a579 1406 } else {
1407 Log("SHUTTLE", Form("ContinueProcessing - %s: restarting. "
1408 "Aborted before with %s. Retry number %d.", fCurrentDetector.Data(),
1409 status->GetStatusName(), status->GetCount()));
9827400b 1410 Bool_t increaseCount = kTRUE;
c88ad5db 1411 if (status->GetStatus() == AliShuttleStatus::kDCSError ||
1412 status->GetStatus() == AliShuttleStatus::kDCSStarted)
1413 increaseCount = kFALSE;
675f64cd 1414
9827400b 1415 UpdateShuttleStatus(AliShuttleStatus::kStarted, increaseCount);
57c1a579 1416 cont = kTRUE;
2bb7b766 1417 }
1418
57c1a579 1419 return cont;
5164a766 1420}
1421
1422//______________________________________________________________________________________________
2bb7b766 1423Bool_t AliShuttle::Process(AliShuttleLogbookEntry* entry)
58bc3020 1424{
73abe331 1425 //
b948db8d 1426 // Makes data retrieval for all detectors in the configuration.
2bb7b766 1427 // entry: Shuttle logbook entry, contains run paramenters and status of detectors
1428 // (Unprocessed, Inactive, Failed or Done).
d477ad88 1429 // Returns kFALSE in case of error occured and kTRUE otherwise
73abe331 1430 //
1431
9827400b 1432 if (!entry) return kFALSE;
2bb7b766 1433
1434 fLogbookEntry = entry;
1435
c88ad5db 1436 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^*",
9827400b 1437 GetCurrentRun()));
2bb7b766 1438
e7f62f16 1439 // Send the information to ML
1440 TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
9827400b 1441 TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
e7f62f16 1442
1443 TList mlList;
1444 mlList.Add(&mlStatus);
9827400b 1445 mlList.Add(&mlRunType);
e7f62f16 1446
ee6f7523 1447 TString mlID;
1448 mlID.Form("%d", GetCurrentRun());
1449 fMonaLisa->SendParameters(&mlList, mlID);
3301427a 1450
9827400b 1451 if (fLogbookEntry->IsDone())
1452 {
1453 Log("SHUTTLE","Process - Shuttle is already DONE. Updating logbook");
1454 UpdateShuttleLogbook("shuttle_done");
1455 fLogbookEntry = 0;
1456 return kTRUE;
1457 }
1458
1459 // read test mode if flag is set
1460 if (fReadTestMode)
1461 {
3d8bc902 1462 fTestMode = kNone;
9827400b 1463 TString logEntry(entry->GetRunParameter("log"));
1464 //printf("log entry = %s\n", logEntry.Data());
1465 TString searchStr("Testmode: ");
1466 Int_t pos = logEntry.Index(searchStr.Data());
1467 //printf("%d\n", pos);
1468 if (pos >= 0)
1469 {
1470 TSubString subStr = logEntry(pos + searchStr.Length(), logEntry.Length());
1471 //printf("%s\n", subStr.String().Data());
1472 TString newStr(subStr.Data());
1473 TObjArray* token = newStr.Tokenize(' ');
1474 if (token)
1475 {
1476 //token->Print();
1477 TObjString* tmpStr = dynamic_cast<TObjString*> (token->First());
1478 if (tmpStr)
1479 {
1480 Int_t testMode = tmpStr->String().Atoi();
1481 if (testMode > 0)
1482 {
c88ad5db 1483 Log("SHUTTLE", Form("Process - Enabling test mode %d", testMode));
9827400b 1484 SetTestMode((TestMode) testMode);
1485 }
1486 }
1487 delete token;
1488 }
1489 }
1490 }
c88ad5db 1491
eba76848 1492 fLogbookEntry->Print("all");
57f50b3c 1493
1494 // Initialization
d477ad88 1495 Bool_t hasError = kFALSE;
5164a766 1496
675f64cd 1497 // Set the CDB and Reference folders according to the year and LHC period
1498 TString lhcPeriod(GetLHCPeriod());
1499 if (lhcPeriod.Length() == 0)
1500 {
7d4cf768 1501 Log("SHUTTLE","Process - LHCPeriod not found in logbook!");
1502 return 0;
675f64cd 1503 }
1504
1505 if (fgkMainCDB.Length() == 0)
1506 fgkMainCDB = Form("alien://folder=/alice/data/%d/%s/OCDB?user=alidaq?cacheFold=/tmp/OCDBCache",
1507 GetCurrentYear(), lhcPeriod.Data());
1508
1509 if (fgkMainRefStorage.Length() == 0)
1510 fgkMainRefStorage = Form("alien://folder=/alice/data/%d/%s/Reference?user=alidaq?cacheFold=/tmp/OCDBCache",
1511 GetCurrentYear(), lhcPeriod.Data());
1512
57f50b3c 1513 // Loop on detectors in the configuration
b948db8d 1514 TIter iter(fConfig->GetDetectors());
2bb7b766 1515 TObjString* aDetector = 0;
b948db8d 1516
4508ab8b 1517 Bool_t first = kTRUE;
1518
be48e3ea 1519 while ((aDetector = (TObjString*) iter.Next()))
1520 {
7bfb2090 1521 fCurrentDetector = aDetector->String();
5164a766 1522
9e080f92 1523 if (ContinueProcessing() == kFALSE) continue;
4508ab8b 1524
1525 if (first)
1526 {
1527 // only read QueryCDB when needed and only once
1528 AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
1529 if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
1530 AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
1531 if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
1532 first = kFALSE;
1533 }
9e080f92 1534
c88ad5db 1535 Log("SHUTTLE", Form("\t\t\t****** run %d - %s: START ******",
2bb7b766 1536 GetCurrentRun(), aDetector->GetName()));
1537
9d733021 1538 for(Int_t iSys=0;iSys<3;iSys++) fFXSCalled[iSys]=kFALSE;
1539
c88ad5db 1540 Log(fCurrentDetector.Data(), "Process - Starting processing");
85a80aa9 1541
be48e3ea 1542 Int_t pid = fork();
1543
1544 if (pid < 0)
1545 {
c88ad5db 1546 Log("SHUTTLE", "Process - ERROR: Forking failed");
be48e3ea 1547 }
1548 else if (pid > 0)
1549 {
1550 // parent
c88ad5db 1551 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Starting monitoring",
be48e3ea 1552 GetCurrentRun(), aDetector->GetName()));
1553
1554 Long_t begin = time(0);
1555
1556 int status; // to be used with waitpid, on purpose an int (not Int_t)!
1557 while (waitpid(pid, &status, WNOHANG) == 0)
1558 {
1559 Long_t expiredTime = time(0) - begin;
1560
1561 if (expiredTime > fConfig->GetPPTimeOut())
1562 {
9827400b 1563 TString tmp;
c88ad5db 1564 tmp.Form("Process - Process of %s time out. "
1565 "Run time: %d seconds. Killing...",
1566 fCurrentDetector.Data(), expiredTime);
9827400b 1567 Log("SHUTTLE", tmp);
1568 Log(fCurrentDetector, tmp);
be48e3ea 1569
1570 kill(pid, 9);
1571
3301427a 1572 UpdateShuttleStatus(AliShuttleStatus::kPPTimeOut);
be48e3ea 1573 hasError = kTRUE;
1574
1575 gSystem->Sleep(1000);
1576 }
1577 else
1578 {
be48e3ea 1579 gSystem->Sleep(1000);
9827400b 1580
1581 TString checkStr;
1582 checkStr.Form("ps -o vsize --pid %d | tail -n 1", pid);
1583 FILE* pipe = gSystem->OpenPipe(checkStr, "r");
1584 if (!pipe)
1585 {
c88ad5db 1586 Log("SHUTTLE", Form("Process - Error: "
1587 "Could not open pipe to %s", checkStr.Data()));
9827400b 1588 continue;
1589 }
1590
1591 char buffer[100];
1592 if (!fgets(buffer, 100, pipe))
1593 {
c88ad5db 1594 Log("SHUTTLE", "Process - Error: ps did not return anything");
9827400b 1595 gSystem->ClosePipe(pipe);
1596 continue;
1597 }
1598 gSystem->ClosePipe(pipe);
1599
1600 //Log("SHUTTLE", Form("ps returned %s", buffer));
1601
1602 Int_t mem = 0;
1603 if ((sscanf(buffer, "%d\n", &mem) != 1) || !mem)
1604 {
c88ad5db 1605 Log("SHUTTLE", "Process - Error: Could not parse output of ps");
9827400b 1606 continue;
1607 }
1608
1609 if (expiredTime % 60 == 0)
ee6f7523 1610 {
c88ad5db 1611 Log("SHUTTLE", Form("Process - %s: Checking process. "
1612 "Run time: %d seconds - Memory consumption: %d KB",
1613 fCurrentDetector.Data(), expiredTime, mem));
ee6f7523 1614 SendAlive();
1615 }
9827400b 1616
1617 if (mem > fConfig->GetPPMaxMem())
1618 {
1619 TString tmp;
c88ad5db 1620 tmp.Form("Process - Process exceeds maximum allowed memory "
1621 "(%d KB > %d KB). Killing...",
9827400b 1622 mem, fConfig->GetPPMaxMem());
1623 Log("SHUTTLE", tmp);
1624 Log(fCurrentDetector, tmp);
1625
1626 kill(pid, 9);
1627
1628 UpdateShuttleStatus(AliShuttleStatus::kPPOutOfMemory);
1629 hasError = kTRUE;
1630
1631 gSystem->Sleep(1000);
1632 }
be48e3ea 1633 }
1634 }
1635
c88ad5db 1636 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Client has terminated.",
be48e3ea 1637 GetCurrentRun(), aDetector->GetName()));
1638
1639 if (WIFEXITED(status))
1640 {
1641 Int_t returnCode = WEXITSTATUS(status);
1642
c88ad5db 1643 Log("SHUTTLE", Form("Process - %s: the return code is %d", fCurrentDetector.Data(),
3301427a 1644 returnCode));
be48e3ea 1645
9827400b 1646 if (returnCode == 0) hasError = kTRUE;
be48e3ea 1647 }
1648 }
1649 else if (pid == 0)
1650 {
1651 // client
c88ad5db 1652 Log("SHUTTLE", Form("Process - In client process of %d - %s", GetCurrentRun(),
1653 aDetector->GetName()));
be48e3ea 1654
c88ad5db 1655 Log("SHUTTLE", Form("Process - Redirecting output to %s log",fCurrentDetector.Data()));
ffa29e93 1656
546242fb 1657 if ((freopen(GetLogFileName(fCurrentDetector), "a", stdout)) == 0)
ffa29e93 1658 {
c88ad5db 1659 Log("SHUTTLE", "Process - Could not freopen stdout");
ffa29e93 1660 }
1661 else
1662 {
1663 fOutputRedirected = kTRUE;
1664 if ((dup2(fileno(stdout), fileno(stderr))) < 0)
c88ad5db 1665 Log("SHUTTLE", "Process - Could not redirect stderr");
ffa29e93 1666
1667 }
1668
5bac2bde 1669 TString wd = gSystem->WorkingDirectory();
675f64cd 1670 TString tmpDir = Form("%s/%s_%d_process", GetShuttleTempDir(),
1671 fCurrentDetector.Data(), GetCurrentRun());
5bac2bde 1672
d524ade6 1673 Int_t result = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
1674 if (!result) // temp dir already exists!
1675 {
1676 Log(fCurrentDetector.Data(),
1677 Form("Process - %s dir already exists! Removing...", tmpDir.Data()));
1678 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
675f64cd 1679 }
1680
1681 if (gSystem->mkdir(tmpDir.Data(), 1))
1682 {
1683 Log(fCurrentDetector.Data(), "Process - could not make temp directory!!");
1684 gSystem->Exit(1);
d524ade6 1685 }
1686
1687 if (!gSystem->ChangeDirectory(tmpDir.Data()))
1688 {
1689 Log(fCurrentDetector.Data(), "Process - could not change directory!!");
1690 gSystem->Exit(1);
1691 }
5bac2bde 1692
9827400b 1693 Bool_t success = ProcessCurrentDetector();
5bac2bde 1694
1695 gSystem->ChangeDirectory(wd.Data());
b1d18693 1696
9827400b 1697 if (success) // Preprocessor finished successfully!
1698 {
b1d18693 1699 // remove temporary folder
1700 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
1701
3301427a 1702 // Update time_processed field in FXS DB
1703 if (UpdateTable() == kFALSE)
5bac2bde 1704 Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!",
1705 fCurrentDetector.Data()));
3301427a 1706
1707 // Transfer the data from local storage to main storage (Grid)
1708 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1709 if (StoreOCDB() == kFALSE)
1710 {
c88ad5db 1711 Log("SHUTTLE",
1712 Form("\t\t\t****** run %d - %s: STORAGE ERROR ******",
3301427a 1713 GetCurrentRun(), aDetector->GetName()));
1714 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
9827400b 1715 success = kFALSE;
3301427a 1716 } else {
c88ad5db 1717 Log("SHUTTLE",
1718 Form("\t\t\t****** run %d - %s: DONE ******",
3301427a 1719 GetCurrentRun(), aDetector->GetName()));
1720 UpdateShuttleStatus(AliShuttleStatus::kDone);
9827400b 1721 UpdateShuttleLogbook(fCurrentDetector, "DONE");
3301427a 1722 }
c88ad5db 1723 } else
1724 {
1725 Log("SHUTTLE",
1726 Form("\t\t\t****** run %d - %s: PP ERROR ******",
1727 GetCurrentRun(), aDetector->GetName()));
be48e3ea 1728 }
1729
4b95672b 1730 for (UInt_t iSys=0; iSys<3; iSys++)
1731 {
1732 if (fFXSCalled[iSys]) fFXSlist[iSys].Clear();
1733 }
1734
c88ad5db 1735 Log("SHUTTLE", Form("Process - Client process of %d - %s is exiting now with %d.",
9827400b 1736 GetCurrentRun(), aDetector->GetName(), success));
be48e3ea 1737
1738 // the client exits here
9827400b 1739 gSystem->Exit(success);
be48e3ea 1740
1741 AliError("We should never get here!!!");
1742 }
7bfb2090 1743 }
5164a766 1744
c88ad5db 1745 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^*",
2bb7b766 1746 GetCurrentRun()));
1747
1748 //check if shuttle is done for this run, if so update logbook
1749 TObjArray checkEntryArray;
1750 checkEntryArray.SetOwner(1);
9e080f92 1751 TString whereClause = Form("where run=%d", GetCurrentRun());
b0e53b15 1752 if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) ||
1753 checkEntryArray.GetEntries() == 0) {
9e080f92 1754 Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
1755 GetCurrentRun()));
1756 return hasError == kFALSE;
1757 }
b948db8d 1758
9e080f92 1759 AliShuttleLogbookEntry* checkEntry = dynamic_cast<AliShuttleLogbookEntry*>
1760 (checkEntryArray.At(0));
2bb7b766 1761
9e080f92 1762 if (checkEntry)
1763 {
1764 if (checkEntry->IsDone())
be48e3ea 1765 {
9e080f92 1766 Log("SHUTTLE","Process - Shuttle is DONE. Updating logbook");
1767 UpdateShuttleLogbook("shuttle_done");
1768 }
1769 else
1770 {
1771 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
be48e3ea 1772 {
9e080f92 1773 if (checkEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
be48e3ea 1774 {
9e080f92 1775 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
1776 checkEntry->GetRun(), GetDetName(iDet)));
1777 fFirstUnprocessed[iDet] = kFALSE;
be48e3ea 1778 }
1779 }
2bb7b766 1780 }
1781 }
1782
1783 fLogbookEntry = 0;
85a80aa9 1784
a7160fe9 1785 return hasError == kFALSE;
73abe331 1786}
1787
b948db8d 1788//______________________________________________________________________________________________
9827400b 1789Bool_t AliShuttle::ProcessCurrentDetector()
73abe331 1790{
1791 //
2bb7b766 1792 // Makes data retrieval just for a specific detector (fCurrentDetector).
73abe331 1793 // Threre should be a configuration for this detector.
73abe331 1794
1d172743 1795 Log("SHUTTLE", Form("ProcessCurrentDetector - Retrieving values for %s, run %d",
1796 fCurrentDetector.Data(), GetCurrentRun()));
73abe331 1797
d524ade6 1798 TString wd = gSystem->WorkingDirectory();
1799
2d9019b4 1800 if (!CleanReferenceStorage(fCurrentDetector.Data()))
546242fb 1801 return kFALSE;
d524ade6 1802
1803 gSystem->ChangeDirectory(wd.Data());
1804
1d172743 1805 TMap* dcsMap = new TMap();
3301427a 1806
1807 // call preprocessor
1808 AliPreprocessor* aPreprocessor =
1809 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1810
1811 aPreprocessor->Initialize(GetCurrentRun(), GetCurrentStartTime(), GetCurrentEndTime());
1812
1813 Bool_t processDCS = aPreprocessor->ProcessDCS();
d477ad88 1814
651fdaab 1815 if (!processDCS)
1816 {
1d172743 1817 Log(fCurrentDetector, "ProcessCurrentDetector -"
1818 " The preprocessor requested to skip the retrieval of DCS values");
651fdaab 1819 }
8b739301 1820 else if (fTestMode & kSkipDCS)
2c15234c 1821 {
1d172743 1822 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Skipping DCS processing");
9827400b 1823 }
1824 else if (fTestMode & kErrorDCS)
1825 {
1d172743 1826 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Simulating DCS error");
3d8bc902 1827 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
9827400b 1828 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1d172743 1829 delete dcsMap;
9827400b 1830 return kFALSE;
2c15234c 1831 } else {
3301427a 1832
1833 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
1834
1d172743 1835 // Query DCS archive
1836 Int_t nServers = fConfig->GetNServers(fCurrentDetector);
a038aa70 1837
1d172743 1838 for (int iServ=0; iServ<nServers; iServ++)
2c15234c 1839 {
1d172743 1840
1841 TString host(fConfig->GetDCSHost(fCurrentDetector, iServ));
1842 Int_t port = fConfig->GetDCSPort(fCurrentDetector, iServ);
542b6cc8 1843 Int_t multiSplit = fConfig->GetMultiSplit(fCurrentDetector, iServ);
1844
1790d4b7 1845 Log(fCurrentDetector, Form("ProcessCurrentDetector -"
1846 " Querying DCS Amanda server %s:%d (%d of %d)",
1847 host.Data(), port, iServ+1, nServers));
1d172743 1848
1849 TMap* aliasMap = 0;
1850 TMap* dpMap = 0;
1851
1852 if (fConfig->GetDCSAliases(fCurrentDetector, iServ)->GetEntries() > 0)
2c15234c 1853 {
1d172743 1854 aliasMap = GetValueSet(host, port,
542b6cc8 1855 fConfig->GetDCSAliases(fCurrentDetector, iServ),
1856 kAlias, multiSplit);
1d172743 1857 if (!aliasMap)
1858 {
1859 Log(fCurrentDetector,
1860 Form("ProcessCurrentDetector -"
675f64cd 1861 " Error retrieving DCS aliases from server %s."
1862 " Sending mail to DCS experts!", host.Data()));
1d172743 1863 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
675f64cd 1864
b5f1c82e 1865 //if (!SendMailToDCS())
1866 // Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
675f64cd 1867
a038aa70 1868 delete dcsMap;
1d172743 1869 return kFALSE;
1870 }
2c15234c 1871 }
a038aa70 1872
1d172743 1873 if (fConfig->GetDCSDataPoints(fCurrentDetector, iServ)->GetEntries() > 0)
a038aa70 1874 {
1d172743 1875 dpMap = GetValueSet(host, port,
542b6cc8 1876 fConfig->GetDCSDataPoints(fCurrentDetector, iServ),
1877 kDP, multiSplit);
1d172743 1878 if (!dpMap)
1879 {
1880 Log(fCurrentDetector,
1881 Form("ProcessCurrentDetector -"
675f64cd 1882 " Error retrieving DCS data points from server %s."
1883 " Sending mail to DCS experts!", host.Data()));
1d172743 1884 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
675f64cd 1885
b5f1c82e 1886 //if (!SendMailToDCS())
1887 // Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
675f64cd 1888
1d172743 1889 if (aliasMap) delete aliasMap;
1890 delete dcsMap;
1891 return kFALSE;
1892 }
a038aa70 1893 }
1d172743 1894
1895 // merge aliasMap and dpMap into dcsMap
1896 if(aliasMap) {
1897 TIter iter(aliasMap);
a038aa70 1898 TObjString* key = 0;
1899 while ((key = (TObjString*) iter.Next()))
1d172743 1900 dcsMap->Add(key, aliasMap->GetValue(key->String()));
1901
1902 aliasMap->SetOwner(kFALSE);
1903 delete aliasMap;
1904 }
1905
1906 if(dpMap) {
1907 TIter iter(dpMap);
1908 TObjString* key = 0;
1909 while ((key = (TObjString*) iter.Next()))
1910 dcsMap->Add(key, dpMap->GetValue(key->String()));
1911
1912 dpMap->SetOwner(kFALSE);
1913 delete dpMap;
a038aa70 1914 }
73abe331 1915 }
1916 }
dc25836b 1917
b1d18693 1918 // save map into file, to help debugging in case of preprocessor error
4508ab8b 1919 /*TFile* f = TFile::Open("DCSMap.root","recreate");
b1d18693 1920 f->cd();
1921 dcsMap->Write("DCSMap", TObject::kSingleKey);
1922 f->Close();
4508ab8b 1923 delete f;*/
b1d18693 1924
2bb7b766 1925 // DCS Archive DB processing successful. Call Preprocessor!
85a80aa9 1926 UpdateShuttleStatus(AliShuttleStatus::kPPStarted);
a7160fe9 1927
a038aa70 1928 UInt_t returnValue = aPreprocessor->Process(dcsMap);
b948db8d 1929
3301427a 1930 if (returnValue > 0) // Preprocessor error!
1931 {
c88ad5db 1932 Log(fCurrentDetector, Form("ProcessCurrentDetector - "
1933 "Preprocessor failed. Process returned %d.", returnValue));
cb343cfd 1934 UpdateShuttleStatus(AliShuttleStatus::kPPError);
a038aa70 1935 dcsMap->DeleteAll();
1936 delete dcsMap;
9827400b 1937 return kFALSE;
1938 }
1939
1940 // preprocessor ok!
1941 UpdateShuttleStatus(AliShuttleStatus::kPPDone);
1942 Log(fCurrentDetector, Form("ProcessCurrentDetector - %s preprocessor returned success",
1943 fCurrentDetector.Data()));
b948db8d 1944
a038aa70 1945 dcsMap->DeleteAll();
1946 delete dcsMap;
b948db8d 1947
9827400b 1948 return kTRUE;
2bb7b766 1949}
1950
1951//______________________________________________________________________________________________
1952Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
1953 TObjArray& entries)
1954{
9827400b 1955 // Query DAQ's Shuttle logbook and fills detector status object.
1956 // Call QueryRunParameters to query DAQ logbook for run parameters.
1957 //
2bb7b766 1958
fc5a4708 1959 entries.SetOwner(1);
1960
2bb7b766 1961 // check connection, in case connect
be48e3ea 1962 if(!Connect(3)) return kFALSE;
2bb7b766 1963
1964 TString sqlQuery;
441b0e9c 1965 sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
2bb7b766 1966
be48e3ea 1967 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1968 if (!aResult) {
1969 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
1970 return kFALSE;
1971 }
1972
fc5a4708 1973 AliDebug(2,Form("Query = %s", sqlQuery.Data()));
1974
2bb7b766 1975 if(aResult->GetRowCount() == 0) {
c88ad5db 1976 Log("SHUTTLE", "No entries in Shuttle Logbook match request");
9827400b 1977 delete aResult;
1978 return kTRUE;
2bb7b766 1979 }
1980
1981 // TODO Check field count!
db99d43e 1982 const UInt_t nCols = 23;
2bb7b766 1983 if (aResult->GetFieldCount() != (Int_t) nCols) {
c88ad5db 1984 Log("SHUTTLE", "Invalid SQL result field number!");
2bb7b766 1985 delete aResult;
1986 return kFALSE;
1987 }
1988
2bb7b766 1989 TSQLRow* aRow;
1990 while ((aRow = aResult->Next())) {
1991 TString runString(aRow->GetField(0), aRow->GetFieldLength(0));
1992 Int_t run = runString.Atoi();
1993
eba76848 1994 AliShuttleLogbookEntry *entry = QueryRunParameters(run);
1995 if (!entry)
1996 continue;
2bb7b766 1997
1998 // loop on detectors
eba76848 1999 for(UInt_t ii = 0; ii < nCols; ii++)
2000 entry->SetDetectorStatus(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 2001
eba76848 2002 entries.AddLast(entry);
2bb7b766 2003 delete aRow;
2004 }
2005
2bb7b766 2006 delete aResult;
2007 return kTRUE;
2008}
2009
2010//______________________________________________________________________________________________
eba76848 2011AliShuttleLogbookEntry* AliShuttle::QueryRunParameters(Int_t run)
2bb7b766 2012{
eba76848 2013 //
2014 // Retrieve run parameters written in the DAQ logbook and sets them into AliShuttleLogbookEntry object
2015 //
2bb7b766 2016
2017 // check connection, in case connect
be48e3ea 2018 if (!Connect(3))
eba76848 2019 return 0;
2bb7b766 2020
2021 TString sqlQuery;
2c15234c 2022 sqlQuery.Form("select * from %s where run=%d", fConfig->GetDAQlbTable(), run);
2bb7b766 2023
be48e3ea 2024 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 2025 if (!aResult) {
c88ad5db 2026 Log("SHUTTLE", Form("Can't execute query <%s>!", sqlQuery.Data()));
eba76848 2027 return 0;
2bb7b766 2028 }
2029
eba76848 2030 if (aResult->GetRowCount() == 0) {
2bb7b766 2031 Log("SHUTTLE", Form("QueryRunParameters - No entry in DAQ Logbook for run %d. Skipping", run));
2032 delete aResult;
eba76848 2033 return 0;
2bb7b766 2034 }
2035
eba76848 2036 if (aResult->GetRowCount() > 1) {
c88ad5db 2037 Log("SHUTTLE", Form("QueryRunParameters - UNEXPECTED: "
2038 "more than one entry in DAQ Logbook for run %d!", run));
2bb7b766 2039 delete aResult;
eba76848 2040 return 0;
2bb7b766 2041 }
2042
eba76848 2043 TSQLRow* aRow = aResult->Next();
2044 if (!aRow)
2045 {
c88ad5db 2046 Log("SHUTTLE", Form("QueryRunParameters - Could not retrieve row for run %d. Skipping", run));
eba76848 2047 delete aResult;
2048 return 0;
2049 }
2bb7b766 2050
eba76848 2051 AliShuttleLogbookEntry* entry = new AliShuttleLogbookEntry(run);
2bb7b766 2052
eba76848 2053 for (Int_t ii = 0; ii < aResult->GetFieldCount(); ii++)
2054 entry->SetRunParameter(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 2055
eba76848 2056 UInt_t startTime = entry->GetStartTime();
2057 UInt_t endTime = entry->GetEndTime();
2058
1abfbb60 2059// if (!startTime || !endTime || startTime > endTime)
2060// {
2061// Log("SHUTTLE",
2062// Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d. Skipping!",
2063// run, startTime, endTime));
2064//
2065// Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2066// fLogbookEntry = entry;
2067// if (!UpdateShuttleLogbook("shuttle_done"))
2068// {
2069// AliError(Form("Could not update logbook for run %d !", run));
2070// }
2071// fLogbookEntry = 0;
2072//
2073// delete entry;
2074// delete aRow;
2075// delete aResult;
2076// return 0;
2077// }
2078
2079 if (!startTime)
b0e53b15 2080 {
2081 Log("SHUTTLE",
1abfbb60 2082 Form("QueryRunParameters - Invalid parameters for Run %d: "
2083 "startTime = %d, endTime = %d. Skipping!",
2084 run, startTime, endTime));
b0e53b15 2085
2086 Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2087 fLogbookEntry = entry;
ee6f7523 2088 if (!UpdateShuttleLogbook("shuttle_ignored"))
b0e53b15 2089 {
2090 AliError(Form("Could not update logbook for run %d !", run));
2091 }
2092 fLogbookEntry = 0;
2093
2094 delete entry;
2095 delete aRow;
2096 delete aResult;
2097 return 0;
2098 }
2099
1abfbb60 2100 if (startTime && !endTime)
2101 {
2102 // TODO Here we don't mark SHUTTLE done, because this may mean
2103 //the run is still ongoing!!
2104 Log("SHUTTLE",
2105 Form("QueryRunParameters - Invalid parameters for Run %d: "
2106 "startTime = %d, endTime = %d. Skipping (Shuttle won't be marked as DONE)!",
2107 run, startTime, endTime));
2108
2109 //Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2110 //fLogbookEntry = entry;
2111 //if (!UpdateShuttleLogbook("shuttle_done"))
2112 //{
2113 // AliError(Form("Could not update logbook for run %d !", run));
2114 //}
2115 //fLogbookEntry = 0;
2116
2117 delete entry;
2118 delete aRow;
2119 delete aResult;
2120 return 0;
2121 }
2122
2123 if (startTime && endTime && (startTime > endTime))
2124 {
2125 Log("SHUTTLE",
2126 Form("QueryRunParameters - Invalid parameters for Run %d: "
2127 "startTime = %d, endTime = %d. Skipping!",
2128 run, startTime, endTime));
2129
2130 Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2131 fLogbookEntry = entry;
ee6f7523 2132 if (!UpdateShuttleLogbook("shuttle_ignored"))
1abfbb60 2133 {
2134 AliError(Form("Could not update logbook for run %d !", run));
2135 }
2136 fLogbookEntry = 0;
2137
2138 delete entry;
2139 delete aRow;
2140 delete aResult;
2141 return 0;
2142 }
2143
b0e53b15 2144 TString totEventsStr = entry->GetRunParameter("totalEvents");
2145 Int_t totEvents = totEventsStr.Atoi();
2146 if (totEvents < 1)
2147 {
eba76848 2148 Log("SHUTTLE",
b0e53b15 2149 Form("QueryRunParameters - Run %d has 0 events - Skipping!", run));
2150
2151 Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2152 fLogbookEntry = entry;
97e3c167 2153 if (!UpdateShuttleLogbook("shuttle_ignored"))
b0e53b15 2154 {
2155 AliError(Form("Could not update logbook for run %d !", run));
2156 }
2157 fLogbookEntry = 0;
2158
eba76848 2159 delete entry;
2bb7b766 2160 delete aRow;
eba76848 2161 delete aResult;
2162 return 0;
2bb7b766 2163 }
2164
eba76848 2165 delete aRow;
2bb7b766 2166 delete aResult;
eba76848 2167
2168 return entry;
2bb7b766 2169}
2170
2171//______________________________________________________________________________________________
a038aa70 2172TMap* AliShuttle::GetValueSet(const char* host, Int_t port, const TSeqCollection* entries,
542b6cc8 2173 DCSType type, Int_t multiSplit)
a038aa70 2174{
2175 // Retrieve all "entry" data points from the DCS server
2176 // host, port: TSocket connection parameters
2177 // entries: list of name of the alias or data point
2178 // type: kAlias or kDP
2179 // returns TMap of values, 0 when failure
542b6cc8 2180
2181 AliDCSClient client(host, port, fTimeout, fRetries, multiSplit);
b41b252a 2182
a038aa70 2183 TMap* result = 0;
b41b252a 2184 if (type == kAlias)
a038aa70 2185 {
b41b252a 2186 result = client.GetAliasValues(entries, GetCurrentStartTime(),
2187 GetCurrentEndTime());
2188 }
2189 else if (type == kDP)
2190 {
2191 result = client.GetDPValues(entries, GetCurrentStartTime(),
2192 GetCurrentEndTime());
2193 }
a038aa70 2194
b41b252a 2195 if (result == 0)
2196 {
2197 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get entries! Reason: %s",
1790d4b7 2198 client.GetErrorString(client.GetResultErrorCode())));
2199 if (client.GetResultErrorCode() == AliDCSClient::fgkServerError)
2200 Log(fCurrentDetector.Data(), Form("GetValueSet - Server error code: %s",
2201 client.GetServerError().Data()));
a038aa70 2202
b41b252a 2203 return 0;
a038aa70 2204 }
b41b252a 2205
a038aa70 2206 return result;
2207}
b41b252a 2208
a038aa70 2209//______________________________________________________________________________________________
57f50b3c 2210const char* AliShuttle::GetFile(Int_t system, const char* detector,
2211 const char* id, const char* source)
b948db8d 2212{
9827400b 2213 // Get calibration file from file exchange servers
2214 // First queris the FXS database for the file name, using the run, detector, id and source info
2215 // then calls RetrieveFile(filename) for actual copy to local disk
2216 // run: current run being processed (given by Logbook entry fLogbookEntry)
2217 // detector: the Preprocessor name
2218 // id: provided as a parameter by the Preprocessor
2219 // source: provided by the Preprocessor through GetFileSources function
2220
2221 // check if test mode should simulate a FXS error
2222 if (fTestMode & kErrorFXSFiles)
2223 {
2224 Log(detector, Form("GetFile - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2225 return 0;
2226 }
2227
57f50b3c 2228 // check connection, in case connect
9d733021 2229 if (!Connect(system))
eba76848 2230 {
9d733021 2231 Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
57f50b3c 2232 return 0;
2233 }
2234
2235 // Query preparation
9d733021 2236 TString sourceName(source);
d386d623 2237 Int_t nFields = 3;
2238 TString sqlQueryStart = Form("select filePath,size,fileChecksum from %s where",
2239 fConfig->GetFXSdbTable(system));
2240 TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
2241 GetCurrentRun(), detector, id);
2242
9d733021 2243 if (system == kDAQ)
2244 {
d386d623 2245 whereClause += Form(" and DAQsource=\"%s\"", source);
57f50b3c 2246 }
9d733021 2247 else if (system == kDCS)
eba76848 2248 {
9d733021 2249 sourceName="none";
57f50b3c 2250 }
9d733021 2251 else if (system == kHLT)
9e080f92 2252 {
d386d623 2253 whereClause += Form(" and DDLnumbers=\"%s\"", source);
9d733021 2254 nFields = 3;
9e080f92 2255 }
2256
9e080f92 2257 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2258
2259 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2260
2261 // Query execution
2262 TSQLResult* aResult = 0;
9d733021 2263 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
9e080f92 2264 if (!aResult) {
9d733021 2265 Log(detector, Form("GetFileName - Can't execute SQL query to %s database for: id = %s, source = %s",
2266 GetSystemName(system), id, sourceName.Data()));
9e080f92 2267 return 0;
2268 }
2269
2270 if(aResult->GetRowCount() == 0)
2271 {
2272 Log(detector,
9d733021 2273 Form("GetFileName - No entry in %s FXS db for: id = %s, source = %s",
2274 GetSystemName(system), id, sourceName.Data()));
9e080f92 2275 delete aResult;
2276 return 0;
2277 }
2bb7b766 2278
9e080f92 2279 if (aResult->GetRowCount() > 1) {
2280 Log(detector,
9d733021 2281 Form("GetFileName - More than one entry in %s FXS db for: id = %s, source = %s",
2282 GetSystemName(system), id, sourceName.Data()));
9e080f92 2283 delete aResult;
2284 return 0;
2285 }
2286
9d733021 2287 if (aResult->GetFieldCount() != nFields) {
9e080f92 2288 Log(detector,
9d733021 2289 Form("GetFileName - Wrong field count in %s FXS db for: id = %s, source = %s",
2290 GetSystemName(system), id, sourceName.Data()));
9e080f92 2291 delete aResult;
2292 return 0;
2293 }
2294
2295 TSQLRow* aRow = dynamic_cast<TSQLRow*> (aResult->Next());
2296
2297 if (!aRow){
9d733021 2298 Log(detector, Form("GetFileName - Empty set result in %s FXS db from query: id = %s, source = %s",
2299 GetSystemName(system), id, sourceName.Data()));
9e080f92 2300 delete aResult;
2301 return 0;
2302 }
2303
2304 TString filePath(aRow->GetField(0), aRow->GetFieldLength(0));
2305 TString fileSize(aRow->GetField(1), aRow->GetFieldLength(1));
d386d623 2306 TString fileChecksum(aRow->GetField(2), aRow->GetFieldLength(2));
9e080f92 2307
2308 delete aResult;
2309 delete aRow;
2310
d386d623 2311 AliDebug(2, Form("filePath = %s; size = %s, fileChecksum = %s",
2312 filePath.Data(), fileSize.Data(), fileChecksum.Data()));
9e080f92 2313
9e080f92 2314 // retrieved file is renamed to make it unique
675f64cd 2315 TString localFileName = Form("%s/%s_%d_process/%s_%s_%d_%s_%s.shuttle",
2316 GetShuttleTempDir(), detector, GetCurrentRun(),
d524ade6 2317 GetSystemName(system), detector, GetCurrentRun(),
2318 id, sourceName.Data());
9d733021 2319
9e080f92 2320
9d733021 2321 // file retrieval from FXS
4b95672b 2322 UInt_t nRetries = 0;
2323 UInt_t maxRetries = 3;
2324 Bool_t result = kFALSE;
2325
2326 // copy!! if successful TSystem::Exec returns 0
2327 while(nRetries++ < maxRetries) {
2328 AliDebug(2, Form("Trying to copy file. Retry # %d", nRetries));
2329 result = RetrieveFile(system, filePath.Data(), localFileName.Data());
2330 if(!result)
2331 {
2332 Log(detector, Form("GetFileName - Copy of file %s from %s FXS failed",
9d733021 2333 filePath.Data(), GetSystemName(system)));
4b95672b 2334 continue;
4f0749a8 2335 }
9e080f92 2336
d386d623 2337 if (fileChecksum.Length()>0)
4b95672b 2338 {
2339 // compare md5sum of local file with the one stored in the FXS DB
d524ade6 2340 Int_t md5Comp = gSystem->Exec(Form("md5sum %s |grep %s 2>&1 > /dev/null",
2341 localFileName.Data(), fileChecksum.Data()));
9e080f92 2342
4b95672b 2343 if (md5Comp != 0)
2344 {
2345 Log(detector, Form("GetFileName - md5sum of file %s does not match with local copy!",
2346 filePath.Data()));
2347 result = kFALSE;
2348 continue;
2349 }
d386d623 2350 } else {
2351 Log(fCurrentDetector, Form("GetFile - md5sum of file %s not set in %s database, skipping comparison",
2352 filePath.Data(), GetSystemName(system)));
9d733021 2353 }
4b95672b 2354 if (result) break;
9e080f92 2355 }
2356
4b95672b 2357 if(!result) return 0;
2358
9d733021 2359 fFXSCalled[system]=kTRUE;
2360 TObjString *fileParams = new TObjString(Form("%s#!?!#%s", id, sourceName.Data()));
2361 fFXSlist[system].Add(fileParams);
9e080f92 2362
675f64cd 2363 static TString staticLocalFileName;
2364 staticLocalFileName.Form("%s", localFileName.Data());
2365
c88ad5db 2366 Log(fCurrentDetector, Form("GetFile - Retrieved file with id %s and "
2367 "source %s from %s to %s", id, source,
d524ade6 2368 GetSystemName(system), localFileName.Data()));
675f64cd 2369
d524ade6 2370 return staticLocalFileName.Data();
2bb7b766 2371}
2372
2373//______________________________________________________________________________________________
9d733021 2374Bool_t AliShuttle::RetrieveFile(UInt_t system, const char* fxsFileName, const char* localFileName)
9e080f92 2375{
9827400b 2376 //
2377 // Copies file from FXS to local Shuttle machine
2378 //
2bb7b766 2379
9e080f92 2380 // check temp directory: trying to cd to temp; if it does not exist, create it
d524ade6 2381 AliDebug(2, Form("Copy file %s from %s FXS into %s",
2382 GetSystemName(system), fxsFileName, localFileName));
2383
2384 TString tmpDir(localFileName);
2385
2386 tmpDir = tmpDir(0,tmpDir.Last('/'));
9e080f92 2387
d524ade6 2388 Int_t noDir = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
2389 if (noDir) // temp dir does not exists!
2390 {
2391 if (gSystem->mkdir(tmpDir.Data(), 1))
2392 {
2393 Log(fCurrentDetector.Data(), "RetrieveFile - could not make temp directory!!");
9e080f92 2394 return kFALSE;
2395 }
9e080f92 2396 }
2397
9d733021 2398 TString baseFXSFolder;
2399 if (system == kDAQ)
2400 {
2401 baseFXSFolder = "FES/";
2402 }
2403 else if (system == kDCS)
2404 {
2405 baseFXSFolder = "";
2406 }
2407 else if (system == kHLT)
2408 {
42fde080 2409 baseFXSFolder = "/opt/FXS/";
9d733021 2410 }
2411
2412
d524ade6 2413 TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s",
9d733021 2414 fConfig->GetFXSPort(system),
2415 fConfig->GetFXSUser(system),
2416 fConfig->GetFXSHost(system),
2417 baseFXSFolder.Data(),
2418 fxsFileName,
9e080f92 2419 localFileName);
2420
2421 AliDebug(2, Form("%s",command.Data()));
2422
4b95672b 2423 Bool_t result = (gSystem->Exec(command.Data()) == 0);
9e080f92 2424
4b95672b 2425 return result;
9e080f92 2426}
2427
2428//______________________________________________________________________________________________
9d733021 2429TList* AliShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
2430{
9827400b 2431 //
2432 // Get sources producing the condition file Id from file exchange servers
4a33bdd9 2433 // if id is NULL all sources are returned (distinct)
9827400b 2434 //
1bcd28db 2435
2436 Log(detector, Form("GetFileSources - Retrieving sources with id %s from %s", id, GetSystemName(system)));
9827400b 2437
2438 // check if test mode should simulate a FXS error
2439 if (fTestMode & kErrorFXSSources)
2440 {
2441 Log(detector, Form("GetFileSources - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2442 return 0;
2443 }
2444
9d733021 2445 if (system == kDCS)
2446 {
c88ad5db 2447 Log(detector, "GetFileSources - WARNING: DCS system has only one source of data!");
6297b37d 2448 TList *list = new TList();
2449 list->SetOwner(1);
2450 list->Add(new TObjString(" "));
2451 return list;
9d733021 2452 }
9e080f92 2453
2454 // check connection, in case connect
9d733021 2455 if (!Connect(system))
2456 {
4a33bdd9 2457 Log(detector, Form("GetFileSources - Couldn't connect to %s FXS database", GetSystemName(system)));
9d733021 2458 return NULL;
9e080f92 2459 }
2460
9d733021 2461 TString sourceName = 0;
2462 if (system == kDAQ)
2463 {
2464 sourceName = "DAQsource";
2465 } else if (system == kHLT)
2466 {
2467 sourceName = "DDLnumbers";
2468 }
2469
4a33bdd9 2470 TString sqlQueryStart = Form("select distinct %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
2471 TString whereClause = Form("run=%d and detector=\"%s\"",
2472 GetCurrentRun(), detector);
2473 if (id)
2474 whereClause += Form(" and fileId=\"%s\"", id);
9e080f92 2475 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2476
2477 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2478
2479 // Query execution
2480 TSQLResult* aResult;
9d733021 2481 aResult = fServer[system]->Query(sqlQuery);
9e080f92 2482 if (!aResult) {
9d733021 2483 Log(detector, Form("GetFileSources - Can't execute SQL query to %s database for id: %s",
2484 GetSystemName(system), id));
9e080f92 2485 return 0;
2486 }
2487
86aa42c3 2488 TList *list = new TList();
2489 list->SetOwner(1);
2490
9d733021 2491 if (aResult->GetRowCount() == 0)
2492 {
9e080f92 2493 Log(detector,
9d733021 2494 Form("GetFileSources - No entry in %s FXS table for id: %s", GetSystemName(system), id));
9e080f92 2495 delete aResult;
86aa42c3 2496 return list;
9e080f92 2497 }
2498
1bcd28db 2499 Log(detector, Form("GetFileSources - Found %d sources", aResult->GetRowCount()));
9e080f92 2500
1bcd28db 2501 TSQLRow* aRow;
9d733021 2502 while ((aRow = aResult->Next()))
2503 {
9e080f92 2504
9d733021 2505 TString source(aRow->GetField(0), aRow->GetFieldLength(0));
2506 AliDebug(2, Form("%s = %s", sourceName.Data(), source.Data()));
2507 list->Add(new TObjString(source));
9e080f92 2508 delete aRow;
2509 }
9d733021 2510
9e080f92 2511 delete aResult;
2512
2513 return list;
2bb7b766 2514}
2515
2516//______________________________________________________________________________________________
4a33bdd9 2517TList* AliShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
2518{
2519 //
2520 // Get all ids of condition files produced by a given source from file exchange servers
2521 //
2522
1bcd28db 2523 Log(detector, Form("GetFileIDs - Retrieving ids with source %s with %s", source, GetSystemName(system)));
2524
4a33bdd9 2525 // check if test mode should simulate a FXS error
2526 if (fTestMode & kErrorFXSSources)
2527 {
2528 Log(detector, Form("GetFileIDs - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2529 return 0;
2530 }
2531
2532 // check connection, in case connect
2533 if (!Connect(system))
2534 {
2535 Log(detector, Form("GetFileIDs - Couldn't connect to %s FXS database", GetSystemName(system)));
2536 return NULL;
2537 }
2538
2539 TString sourceName = 0;
2540 if (system == kDAQ)
2541 {
2542 sourceName = "DAQsource";
2543 } else if (system == kHLT)
2544 {
2545 sourceName = "DDLnumbers";
2546 }
2547
2548 TString sqlQueryStart = Form("select fileId from %s where", fConfig->GetFXSdbTable(system));
2549 TString whereClause = Form("run=%d and detector=\"%s\"",
2550 GetCurrentRun(), detector);
2551 if (sourceName.Length() > 0 && source)
2552 whereClause += Form(" and %s=\"%s\"", sourceName.Data(), source);
2553 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2554
2555 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2556
2557 // Query execution
2558 TSQLResult* aResult;
2559 aResult = fServer[system]->Query(sqlQuery);
2560 if (!aResult) {
2561 Log(detector, Form("GetFileIDs - Can't execute SQL query to %s database for source: %s",
2562 GetSystemName(system), source));
2563 return 0;
2564 }
2565
2566 TList *list = new TList();
2567 list->SetOwner(1);
2568
2569 if (aResult->GetRowCount() == 0)
2570 {
2571 Log(detector,
2572 Form("GetFileIDs - No entry in %s FXS table for source: %s", GetSystemName(system), source));
2573 delete aResult;
2574 return list;
2575 }
2576
1bcd28db 2577 Log(detector, Form("GetFileIDs - Found %d ids", aResult->GetRowCount()));
2578
4a33bdd9 2579 TSQLRow* aRow;
2580
2581 while ((aRow = aResult->Next()))
2582 {
2583
2584 TString id(aRow->GetField(0), aRow->GetFieldLength(0));
2585 AliDebug(2, Form("fileId = %s", id.Data()));
2586 list->Add(new TObjString(id));
2587 delete aRow;
2588 }
2589
2590 delete aResult;
2591
2592 return list;
2593}
2594
2595//______________________________________________________________________________________________
9d733021 2596Bool_t AliShuttle::Connect(Int_t system)
2bb7b766 2597{
9827400b 2598 // Connect to MySQL Server of the system's FXS MySQL databases
2599 // DAQ Logbook, Shuttle Logbook and DAQ FXS db are on the same host
2600 //
57f50b3c 2601
9d733021 2602 // check connection: if already connected return
2603 if(fServer[system] && fServer[system]->IsConnected()) return kTRUE;
57f50b3c 2604
9d733021 2605 TString dbHost, dbUser, dbPass, dbName;
57f50b3c 2606
9d733021 2607 if (system < 3) // FXS db servers
2608 {
2609 dbHost = Form("mysql://%s:%d", fConfig->GetFXSdbHost(system), fConfig->GetFXSdbPort(system));
2610 dbUser = fConfig->GetFXSdbUser(system);
2611 dbPass = fConfig->GetFXSdbPass(system);
2612 dbName = fConfig->GetFXSdbName(system);
2613 } else { // Run & Shuttle logbook servers
2614 // TODO Will the Shuttle logbook server be the same as the Run logbook server ???
2615 dbHost = Form("mysql://%s:%d", fConfig->GetDAQlbHost(), fConfig->GetDAQlbPort());
2616 dbUser = fConfig->GetDAQlbUser();
2617 dbPass = fConfig->GetDAQlbPass();
2618 dbName = fConfig->GetDAQlbDB();
2619 }
57f50b3c 2620
9d733021 2621 fServer[system] = TSQLServer::Connect(dbHost.Data(), dbUser.Data(), dbPass.Data());
2622 if (!fServer[system] || !fServer[system]->IsConnected()) {
2623 if(system < 3)
2624 {
2625 AliError(Form("Can't establish connection to FXS database for %s",
2626 AliShuttleInterface::GetSystemName(system)));
2627 } else {
2628 AliError("Can't establish connection to Run logbook.");
57f50b3c 2629 }
9d733021 2630 if(fServer[system]) delete fServer[system];
2631 return kFALSE;
2bb7b766 2632 }
57f50b3c 2633
9d733021 2634 // Get tables
2635 TSQLResult* aResult=0;
2636 switch(system){
2637 case kDAQ:
2638 aResult = fServer[kDAQ]->GetTables(dbName.Data());
2639 break;
2640 case kDCS:
2641 aResult = fServer[kDCS]->GetTables(dbName.Data());
2642 break;
2643 case kHLT:
2644 aResult = fServer[kHLT]->GetTables(dbName.Data());
2645 break;
2646 default:
2647 aResult = fServer[3]->GetTables(dbName.Data());
2648 break;
2649 }
2650
2651 delete aResult;
2bb7b766 2652 return kTRUE;
2653}
57f50b3c 2654
9e080f92 2655//______________________________________________________________________________________________
9d733021 2656Bool_t AliShuttle::UpdateTable()
9e080f92 2657{
9827400b 2658 //
2659 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2660 //
9e080f92 2661
9d733021 2662 Bool_t result = kTRUE;
9e080f92 2663
9d733021 2664 for (UInt_t system=0; system<3; system++)
2665 {
2666 if(!fFXSCalled[system]) continue;
9e080f92 2667
9d733021 2668 // check connection, in case connect
2669 if (!Connect(system))
2670 {
2671 Log(fCurrentDetector, Form("UpdateTable - Couldn't connect to %s FXS database", GetSystemName(system)));
2672 result = kFALSE;
2673 continue;
9e080f92 2674 }
9e080f92 2675
9d733021 2676 TTimeStamp now; // now
2677
2678 // Loop on FXS list entries
2679 TIter iter(&fFXSlist[system]);
2680 TObjString *aFXSentry=0;
2681 while ((aFXSentry = dynamic_cast<TObjString*> (iter.Next())))
2682 {
2683 TString aFXSentrystr = aFXSentry->String();
2684 TObjArray *aFXSarray = aFXSentrystr.Tokenize("#!?!#");
2685 if (!aFXSarray || aFXSarray->GetEntries() != 2 )
2686 {
2687 Log(fCurrentDetector, Form("UpdateTable - error updating %s FXS entry. Check string: <%s>",
2688 GetSystemName(system), aFXSentrystr.Data()));
2689 if(aFXSarray) delete aFXSarray;
2690 result = kFALSE;
2691 continue;
2692 }
2693 const char* fileId = ((TObjString*) aFXSarray->At(0))->GetName();
2694 const char* source = ((TObjString*) aFXSarray->At(1))->GetName();
2695
2696 TString whereClause;
2697 if (system == kDAQ)
2698 {
2699 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DAQsource=\"%s\";",
2700 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2701 }
2702 else if (system == kDCS)
2703 {
2704 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\";",
2705 GetCurrentRun(), fCurrentDetector.Data(), fileId);
2706 }
2707 else if (system == kHLT)
2708 {
2709 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DDLnumbers=\"%s\";",
2710 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2711 }
2712
2713 delete aFXSarray;
9e080f92 2714
9d733021 2715 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2716 now.GetSec(), whereClause.Data());
9e080f92 2717
9d733021 2718 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
9e080f92 2719
9d733021 2720 // Query execution
2721 TSQLResult* aResult;
2722 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2723 if (!aResult)
2724 {
2725 Log(fCurrentDetector, Form("UpdateTable - %s db: can't execute SQL query <%s>",
2726 GetSystemName(system), sqlQuery.Data()));
2727 result = kFALSE;
2728 continue;
2729 }
2730 delete aResult;
9e080f92 2731 }
9e080f92 2732 }
2733
9d733021 2734 return result;
9e080f92 2735}
57f50b3c 2736
2bb7b766 2737//______________________________________________________________________________________________
3301427a 2738Bool_t AliShuttle::UpdateTableFailCase()
2739{
9827400b 2740 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2741 // this is called in case the preprocessor is declared failed for the current run, because
2742 // the fields are updated only in case of success
3301427a 2743
2744 Bool_t result = kTRUE;
2745
2746 for (UInt_t system=0; system<3; system++)
2747 {
2748 // check connection, in case connect
2749 if (!Connect(system))
2750 {
2751 Log(fCurrentDetector, Form("UpdateTableFailCase - Couldn't connect to %s FXS database",
2752 GetSystemName(system)));
2753 result = kFALSE;
2754 continue;
2755 }
2756
2757 TTimeStamp now; // now
2758
2759 // Loop on FXS list entries
2760
2761 TString whereClause = Form("where run=%d and detector=\"%s\";",
2762 GetCurrentRun(), fCurrentDetector.Data());
2763
2764
2765 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2766 now.GetSec(), whereClause.Data());
2767
2768 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2769
2770 // Query execution
2771 TSQLResult* aResult;
2772 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2773 if (!aResult)
2774 {
2775 Log(fCurrentDetector, Form("UpdateTableFailCase - %s db: can't execute SQL query <%s>",
2776 GetSystemName(system), sqlQuery.Data()));
2777 result = kFALSE;
2778 continue;
2779 }
2780 delete aResult;
2781 }
2782
2783 return result;
2784}
2785
2786//______________________________________________________________________________________________
2bb7b766 2787Bool_t AliShuttle::UpdateShuttleLogbook(const char* detector, const char* status)
2788{
e7f62f16 2789 //
2790 // Update Shuttle logbook filling detector or shuttle_done column
2791 // ex. of usage: UpdateShuttleLogbook("PHOS", "DONE") or UpdateShuttleLogbook("shuttle_done")
2792 //
57f50b3c 2793
2bb7b766 2794 // check connection, in case connect
be48e3ea 2795 if(!Connect(3)){
2bb7b766 2796 Log("SHUTTLE", "UpdateShuttleLogbook - Couldn't connect to DAQ Logbook.");
2797 return kFALSE;
57f50b3c 2798 }
2799
2bb7b766 2800 TString detName(detector);
2801 TString setClause;
ee6f7523 2802 if (detName == "shuttle_done" || detName == "shuttle_ignored")
e7f62f16 2803 {
2bb7b766 2804 setClause = "set shuttle_done=1";
e7f62f16 2805
ee6f7523 2806 if (detName == "shuttle_done")
b0e53b15 2807 {
2808 // Send the information to ML
2809 TMonaLisaText mlStatus("SHUTTLE_status", "Done");
e7f62f16 2810
b0e53b15 2811 TList mlList;
2812 mlList.Add(&mlStatus);
2813
ee6f7523 2814 TString mlID;
2815 mlID.Form("%d", GetCurrentRun());
2816 fMonaLisa->SendParameters(&mlList, mlID);
b0e53b15 2817 }
2bb7b766 2818 } else {
2bb7b766 2819 TString statusStr(status);
2820 if(statusStr.Contains("done", TString::kIgnoreCase) ||
2821 statusStr.Contains("failed", TString::kIgnoreCase)){
eba76848 2822 setClause = Form("set %s=\"%s\"", detector, status);
2bb7b766 2823 } else {
2824 Log("SHUTTLE",
2825 Form("UpdateShuttleLogbook - Invalid status <%s> for detector %s",
2826 status, detector));
2827 return kFALSE;
2828 }
2829 }
57f50b3c 2830
2bb7b766 2831 TString whereClause = Form("where run=%d", GetCurrentRun());
2832
441b0e9c 2833 TString sqlQuery = Form("update %s %s %s",
2834 fConfig->GetShuttlelbTable(), setClause.Data(), whereClause.Data());
57f50b3c 2835
2bb7b766 2836 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2837
2838 // Query execution
2839 TSQLResult* aResult;
be48e3ea 2840 aResult = dynamic_cast<TSQLResult*> (fServer[3]->Query(sqlQuery));
2bb7b766 2841 if (!aResult) {
2842 Log("SHUTTLE", Form("UpdateShuttleLogbook - Can't execute query <%s>", sqlQuery.Data()));
2843 return kFALSE;
57f50b3c 2844 }
2bb7b766 2845 delete aResult;
57f50b3c 2846
2847 return kTRUE;
2848}
2849
2850//______________________________________________________________________________________________
2bb7b766 2851Int_t AliShuttle::GetCurrentRun() const
2852{
9827400b 2853 //
2854 // Get current run from logbook entry
2855 //
57f50b3c 2856
2bb7b766 2857 return fLogbookEntry ? fLogbookEntry->GetRun() : -1;
57f50b3c 2858}
2859
2860//______________________________________________________________________________________________
2bb7b766 2861UInt_t AliShuttle::GetCurrentStartTime() const
2862{
9827400b 2863 //
2864 // get current start time
2865 //
57f50b3c 2866
2bb7b766 2867 return fLogbookEntry ? fLogbookEntry->GetStartTime() : 0;
57f50b3c 2868}
2869
2870//______________________________________________________________________________________________
2bb7b766 2871UInt_t AliShuttle::GetCurrentEndTime() const
2872{
9827400b 2873 //
2874 // get current end time from logbook entry
2875 //
57f50b3c 2876
2bb7b766 2877 return fLogbookEntry ? fLogbookEntry->GetEndTime() : 0;
57f50b3c 2878}
2879
2880//______________________________________________________________________________________________
675f64cd 2881UInt_t AliShuttle::GetCurrentYear() const
2882{
2883 //
2884 // Get current year from logbook entry
2885 //
2886
2887 if (!fLogbookEntry) return 0;
2888
2889 TTimeStamp startTime(GetCurrentStartTime());
2890 TString year = Form("%d",startTime.GetDate());
2891 year = year(0,4);
2892
2893 return year.Atoi();
2894}
2895
2896//______________________________________________________________________________________________
2897const char* AliShuttle::GetLHCPeriod() const
2898{
2899 //
2900 // Get current LHC period from logbook entry
2901 //
2902
2903 if (!fLogbookEntry) return 0;
2904
2905 return fLogbookEntry->GetRunParameter("LHCperiod");
2906}
2907
2908//______________________________________________________________________________________________
b948db8d 2909void AliShuttle::Log(const char* detector, const char* message)
2910{
9827400b 2911 //
2912 // Fill log string with a message
2913 //
b948db8d 2914
7d4cf768 2915 TString logRunDir = GetShuttleLogDir();
2916 if (GetCurrentRun() >=0)
2917 logRunDir += Form("/%d", GetCurrentRun());
2918
2919 void* dir = gSystem->OpenDirectory(logRunDir.Data());
84090f85 2920 if (dir == NULL) {
7d4cf768 2921 if (gSystem->mkdir(logRunDir.Data(), kTRUE)) {
36c99a6a 2922 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
84090f85 2923 return;
2924 }
b948db8d 2925
84090f85 2926 } else {
2927 gSystem->FreeDirectory(dir);
2928 }
b948db8d 2929
cb343cfd 2930 TString toLog = Form("%s (%d): %s - ", TTimeStamp(time(0)).AsString("s"), getpid(), detector);
e7f62f16 2931 if (GetCurrentRun() >= 0)
2932 toLog += Form("run %d - ", GetCurrentRun());
2bb7b766 2933 toLog += Form("%s", message);
2934
84090f85 2935 AliInfo(toLog.Data());
ffa29e93 2936
2937 // if we redirect the log output already to the file, leave here
2938 if (fOutputRedirected && strcmp(detector, "SHUTTLE") != 0)
2939 return;
b948db8d 2940
ffa29e93 2941 TString fileName = GetLogFileName(detector);
e7f62f16 2942
84090f85 2943 gSystem->ExpandPathName(fileName);
2944
2945 ofstream logFile;
2946 logFile.open(fileName, ofstream::out | ofstream::app);
2947
2948 if (!logFile.is_open()) {
2949 AliError(Form("Could not open file %s", fileName.Data()));
2950 return;
2951 }
7bfb2090 2952
84090f85 2953 logFile << toLog.Data() << "\n";
b948db8d 2954
84090f85 2955 logFile.close();
b948db8d 2956}
2bb7b766 2957
2bb7b766 2958//______________________________________________________________________________________________
ffa29e93 2959TString AliShuttle::GetLogFileName(const char* detector) const
2960{
2961 //
2962 // returns the name of the log file for a given sub detector
2963 //
2964
2965 TString fileName;
2966
2967 if (GetCurrentRun() >= 0)
7d4cf768 2968 {
2969 fileName.Form("%s/%d/%s_%d.log", GetShuttleLogDir(), GetCurrentRun(),
2970 detector, GetCurrentRun());
2971 } else {
ffa29e93 2972 fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
7d4cf768 2973 }
ffa29e93 2974
2975 return fileName;
2976}
2977
2978//______________________________________________________________________________________________
ee6f7523 2979void AliShuttle::SendAlive()
2980{
2981 // sends alive message to ML
2982
2983 TMonaLisaText mlStatus("SHUTTLE_status", "Alive");
2984
2985 TList mlList;
2986 mlList.Add(&mlStatus);
2987
2988 fMonaLisa->SendParameters(&mlList, "__PROCESSINGINFO__");
2989}
2990
2991//______________________________________________________________________________________________
2bb7b766 2992Bool_t AliShuttle::Collect(Int_t run)
2993{
9827400b 2994 //
2995 // Collects conditions data for all UNPROCESSED run written to DAQ LogBook in case of run = -1 (default)
2996 // If a dedicated run is given this run is processed
2997 //
2998 // In operational mode, this is the Shuttle function triggered by the EOR signal.
2999 //
2bb7b766 3000
eba76848 3001 if (run == -1)
3002 Log("SHUTTLE","Collect - Shuttle called. Collecting conditions data for unprocessed runs");
3003 else
3004 Log("SHUTTLE", Form("Collect - Shuttle called. Collecting conditions data for run %d", run));
cb343cfd 3005
3006 SetLastAction("Starting");
2bb7b766 3007
ee6f7523 3008 // create ML instance
3009 if (!fMonaLisa)
3010 fMonaLisa = new TMonaLisaWriter(fConfig->GetMonitorHost(), fConfig->GetMonitorTable());
3011
3012
3013 SendAlive();
3014
2bb7b766 3015 TString whereClause("where shuttle_done=0");
eba76848 3016 if (run != -1)
3017 whereClause += Form(" and run=%d", run);
2bb7b766 3018
3019 TObjArray shuttleLogbookEntries;
be48e3ea 3020 if (!QueryShuttleLogbook(whereClause, shuttleLogbookEntries))
3021 {
cb343cfd 3022 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2bb7b766 3023 return kFALSE;
3024 }
3025
9e080f92 3026 if (shuttleLogbookEntries.GetEntries() == 0)
3027 {
3028 if (run == -1)
3029 Log("SHUTTLE","Collect - Found no UNPROCESSED runs in Shuttle logbook");
3030 else
3031 Log("SHUTTLE", Form("Collect - Run %d is already DONE "
3032 "or it does not exist in Shuttle logbook", run));
3033 return kTRUE;
3034 }
3035
be48e3ea 3036 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
3037 fFirstUnprocessed[iDet] = kTRUE;
3038
fc5a4708 3039 if (run != -1)
be48e3ea 3040 {
3041 // query Shuttle logbook for earlier runs, check if some detectors are unprocessed,
3042 // flag them into fFirstUnprocessed array
3043 TString whereClause(Form("where shuttle_done=0 and run < %d", run));
3044 TObjArray tmpLogbookEntries;
3045 if (!QueryShuttleLogbook(whereClause, tmpLogbookEntries))
3046 {
3047 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
3048 return kFALSE;
3049 }
3050
3051 TIter iter(&tmpLogbookEntries);
3052 AliShuttleLogbookEntry* anEntry = 0;
3053 while ((anEntry = dynamic_cast<AliShuttleLogbookEntry*> (iter.Next())))
3054 {
3055 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
3056 {
3057 if (anEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
3058 {
3059 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
3060 anEntry->GetRun(), GetDetName(iDet)));
3061 fFirstUnprocessed[iDet] = kFALSE;
3062 }
3063 }
3064
3065 }
3066
3067 }
3068
3069 if (!RetrieveConditionsData(shuttleLogbookEntries))
3070 {
cb343cfd 3071 Log("SHUTTLE", "Collect - Process of at least one run failed");
2bb7b766 3072 return kFALSE;
3073 }
3074
36c99a6a 3075 Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
eba76848 3076 return kTRUE;
2bb7b766 3077}
3078
2bb7b766 3079//______________________________________________________________________________________________
3080Bool_t AliShuttle::RetrieveConditionsData(const TObjArray& dateEntries)
3081{
9827400b 3082 //
3083 // Retrieve conditions data for all runs that aren't processed yet
3084 //
2bb7b766 3085
3086 Bool_t hasError = kFALSE;
3087
3088 TIter iter(&dateEntries);
3089 AliShuttleLogbookEntry* anEntry;
3090
3091 while ((anEntry = (AliShuttleLogbookEntry*) iter.Next())){
3092 if (!Process(anEntry)){
3093 hasError = kTRUE;
3094 }
4b95672b 3095
3096 // clean SHUTTLE temp directory
d524ade6 3097 //TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
3098 //RemoveFile(filename.Data());
2bb7b766 3099 }
3100
3101 return hasError == kFALSE;
3102}
cb343cfd 3103
3104//______________________________________________________________________________________________
3105ULong_t AliShuttle::GetTimeOfLastAction() const
3106{
9827400b 3107 //
3108 // Gets time of last action
3109 //
3110
cb343cfd 3111 ULong_t tmp;
36c99a6a 3112
cb343cfd 3113 fMonitoringMutex->Lock();
be48e3ea 3114
cb343cfd 3115 tmp = fLastActionTime;
36c99a6a 3116
cb343cfd 3117 fMonitoringMutex->UnLock();
36c99a6a 3118
cb343cfd 3119 return tmp;
3120}
3121
3122//______________________________________________________________________________________________
3123const TString AliShuttle::GetLastAction() const
3124{
9827400b 3125 //
cb343cfd 3126 // returns a string description of the last action
9827400b 3127 //
cb343cfd 3128
3129 TString tmp;
36c99a6a 3130
cb343cfd 3131 fMonitoringMutex->Lock();
3132
3133 tmp = fLastAction;
3134
3135 fMonitoringMutex->UnLock();
3136
36c99a6a 3137 return tmp;
cb343cfd 3138}
3139
3140//______________________________________________________________________________________________
3141void AliShuttle::SetLastAction(const char* action)
3142{
9827400b 3143 //
cb343cfd 3144 // updates the monitoring variables
9827400b 3145 //
36c99a6a 3146
cb343cfd 3147 fMonitoringMutex->Lock();
36c99a6a 3148
cb343cfd 3149 fLastAction = action;
3150 fLastActionTime = time(0);
3151
3152 fMonitoringMutex->UnLock();
3153}
eba76848 3154
3155//______________________________________________________________________________________________
3156const char* AliShuttle::GetRunParameter(const char* param)
3157{
9827400b 3158 //
3159 // returns run parameter read from DAQ logbook
3160 //
eba76848 3161