Fix for numerical precision
[u/mrichter/AliRoot.git] / SHUTTLE / AliShuttle.cxx
CommitLineData
73abe331 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
7d4cf768 18Revision 1.66 2007/12/05 10:45:19 jgrosseo
19changed order of arguments to TMonaLisaWriter
20
468df1ce 21Revision 1.65 2007/11/26 16:58:37 acolla
22Monalisa configuration added: host and table name
23
b832ec02 24Revision 1.64 2007/11/13 16:15:47 acolla
25DCS map is stored in a file in the temp folder where the detector is processed.
26If the preprocessor fails, the temp folder is not removed. This will help the debugging of the problem.
27
b1d18693 28Revision 1.63 2007/11/02 10:53:16 acolla
29Protection added to AliShuttle::CopyFileLocally
30
7d43a416 31Revision 1.62 2007/10/31 18:23:13 acolla
32Furter developement on the Shuttle:
33
34- Shuttle now connects to the Grid as alidaq. The OCDB and Reference folders
35are now built from /alice/data, e.g.:
36/alice/data/2007/LHC07a/OCDB
37
38the year and LHC period are taken from the Shuttle.
39Raw metadata files are stored by GRP to:
40/alice/data/2007/LHC07a/<runNb>/Raw/RunMetadata.root
41
42- Shuttle sends a mail to DCS experts each time DP retrieval fails.
43
675f64cd 44Revision 1.61 2007/10/30 20:33:51 acolla
45Improved managing of temporary folders, which weren't correctly handled.
46Resolved bug introduced in StoreReferenceFile, which caused SPD preprocessor fail.
47
d524ade6 48Revision 1.60 2007/10/29 18:06:16 acolla
49
50New function StoreRunMetadataFile added to preprocessor and Shuttle interface
51This function can be used by GRP only. It stores raw data tags merged file to the
52raw data folder (e.g. /alice/data/2008/LHC08a/000099999/Raw).
53
54KNOWN ISSUES:
55
561. Shuttle cannot write to /alice/data/ because it belongs to alidaq. Tag file is stored in /alice/simulation/... for the time being.
572. Due to a bug in TAlien::Mkdir, the creation of a folder in recursive mode (-p option) does not work. The problem
58has been corrected in the root package on the Shuttle machine.
59
c88ad5db 60Revision 1.59 2007/10/05 12:40:55 acolla
61
62Result error code added to AliDCSClient data members (it was "lost" with the new implementation of TMap* GetAliasValues and GetDPValues).
63
1790d4b7 64Revision 1.58 2007/09/28 15:27:40 acolla
65
66AliDCSClient "multiSplit" option added in the DCS configuration
67in AliDCSMessage: variable MAX_BODY_SIZE set to 500000
68
542b6cc8 69Revision 1.57 2007/09/27 16:53:13 acolla
70Detectors can have more than one AMANDA server. SHUTTLE queries the servers sequentially,
71merges the dcs aliases/DPs in one TMap and sends it to the preprocessor.
72
1d172743 73Revision 1.56 2007/09/14 16:46:14 jgrosseo
741) Connect and Close are called before and after each query, so one can
75keep the same AliDCSClient object.
762) The splitting of a query is moved to GetDPValues/GetAliasValues.
773) Splitting interval can be specified in constructor
78
b41b252a 79Revision 1.55 2007/08/06 12:26:40 acolla
80Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
81read from the run logbook.
82
4859271b 83Revision 1.54 2007/07/12 09:51:25 jgrosseo
84removed duplicated log message in GetFile
85
4f0749a8 86Revision 1.53 2007/07/12 09:26:28 jgrosseo
87updating hlt fxs base path
88
42fde080 89Revision 1.52 2007/07/12 08:06:45 jgrosseo
90adding log messages in getfile... functions
91adding not implemented copy constructor in alishuttleconfigholder
92
1bcd28db 93Revision 1.51 2007/07/03 17:24:52 acolla
94root moved to v5-16-00. TFileMerger->Cp moved to TFile::Cp.
95
a986b218 96Revision 1.50 2007/07/02 17:19:32 acolla
97preprocessor is run in a temp directory that is removed when process is finished.
98
5bac2bde 99Revision 1.49 2007/06/29 10:45:06 acolla
100Number of columns in MySql Shuttle logbook increased by one (HLT added)
101
db99d43e 102Revision 1.48 2007/06/21 13:06:19 acolla
103GetFileSources returns dummy list with 1 source if system=DCS (better than
104returning error as it was)
105
6297b37d 106Revision 1.47 2007/06/19 17:28:56 acolla
107HLT updated; missing map bug removed.
108
dc25836b 109Revision 1.46 2007/06/09 13:01:09 jgrosseo
110Switching to retrieval of several DCS DPs at a time (multiDPrequest)
111
a038aa70 112Revision 1.45 2007/05/30 06:35:20 jgrosseo
113Adding functionality to the Shuttle/TestShuttle:
114o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
115o) Function to retrieve list of IDs for a given source (GetFileIDs)
116These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
117Example code has been added to the TestProcessor in TestShuttle
118
4a33bdd9 119Revision 1.44 2007/05/11 16:09:32 acolla
120Reference files for ITS, MUON and PHOS are now stored in OfflineDetName/OnlineDetName/run_...
121example: ITS/SPD/100_filename.root
122
2d9019b4 123Revision 1.43 2007/05/10 09:59:51 acolla
124Various bug fixes in StoreRefFilesToGrid; Cleaning of reference storage before processing detector (CleanReferenceStorage)
125
546242fb 126Revision 1.42 2007/05/03 08:01:39 jgrosseo
127typo in last commit :-(
128
8b739301 129Revision 1.41 2007/05/03 08:00:48 jgrosseo
130fixing log message when pp want to skip dcs value retrieval
131
651fdaab 132Revision 1.40 2007/04/27 07:06:48 jgrosseo
133GetFileSources returns empty list in case of no files, but successful query
134No mails sent in testmode
135
86aa42c3 136Revision 1.39 2007/04/17 12:43:57 acolla
137Correction in StoreOCDB; change of text in mail to detector expert
138
26758fce 139Revision 1.38 2007/04/12 08:26:18 jgrosseo
140updated comment
141
3c2a21c8 142Revision 1.37 2007/04/10 16:53:14 jgrosseo
143redirecting sub detector stdout, stderr to sub detector log file
144
3d8bc902 145Revision 1.35 2007/04/04 16:26:38 acolla
1461. Re-organization of function calls in TestPreprocessor to make it more meaningful.
1472. Added missing dependency in test preprocessors.
1483. in AliShuttle.cxx: processing time and memory consumption info on a single line.
149
886d60e6 150Revision 1.34 2007/04/04 10:33:36 jgrosseo
1511) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
152In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
153
1542) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
155
1563) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
157
1584) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
159
1605) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
161If you always need DCS data (like before), you do not need to implement it.
162
1636) The run type has been added to the monitoring page
164
9827400b 165Revision 1.33 2007/04/03 13:56:01 acolla
166Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
167run type.
168
3301427a 169Revision 1.32 2007/02/28 10:41:56 acolla
170Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
171AliPreprocessor::GetRunType() function.
172Added some ldap definition files.
173
d386d623 174Revision 1.30 2007/02/13 11:23:21 acolla
175Moved getters and setters of Shuttle's main OCDB/Reference, local
176OCDB/Reference, temp and log folders to AliShuttleInterface
177
9d733021 178Revision 1.27 2007/01/30 17:52:42 jgrosseo
179adding monalisa monitoring
180
e7f62f16 181Revision 1.26 2007/01/23 19:20:03 acolla
182Removed old ldif files, added TOF, MCH ldif files. Added some options in
183AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
184SetShuttleLogDir
185
36c99a6a 186Revision 1.25 2007/01/15 19:13:52 acolla
187Moved some AliInfo to AliDebug in SendMail function
188
fc5a4708 189Revision 1.21 2006/12/07 08:51:26 jgrosseo
190update (alberto):
191table, db names in ldap configuration
192added GRP preprocessor
193DCS data can also be retrieved by data point
194
2c15234c 195Revision 1.20 2006/11/16 16:16:48 jgrosseo
196introducing strict run ordering flag
197removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
198
be48e3ea 199Revision 1.19 2006/11/06 14:23:04 jgrosseo
200major update (Alberto)
201o) reading of run parameters from the logbook
202o) online offline naming conversion
203o) standalone DCSclient package
204
eba76848 205Revision 1.18 2006/10/20 15:22:59 jgrosseo
206o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
207o) Merging Collect, CollectAll, CollectNew function
208o) Removing implementation of empty copy constructors (declaration still there!)
209
cb343cfd 210Revision 1.17 2006/10/05 16:20:55 jgrosseo
211adapting to new CDB classes
212
6ec0e06c 213Revision 1.16 2006/10/05 15:46:26 jgrosseo
214applying to the new interface
215
481441a2 216Revision 1.15 2006/10/02 16:38:39 jgrosseo
217update (alberto):
218fixed memory leaks
219storing of objects that failed to be stored to the grid before
220interfacing of shuttle status table in daq system
221
2bb7b766 222Revision 1.14 2006/08/29 09:16:05 jgrosseo
223small update
224
85a80aa9 225Revision 1.13 2006/08/15 10:50:00 jgrosseo
226effc++ corrections (alberto)
227
4f0ab988 228Revision 1.12 2006/08/08 14:19:29 jgrosseo
229Update to shuttle classes (Alberto)
230
231- Possibility to set the full object's path in the Preprocessor's and
232Shuttle's Store functions
233- Possibility to extend the object's run validity in the same classes
234("startValidity" and "validityInfinite" parameters)
235- Implementation of the StoreReferenceData function to store reference
236data in a dedicated CDB storage.
237
84090f85 238Revision 1.11 2006/07/21 07:37:20 jgrosseo
239last run is stored after each run
240
7bfb2090 241Revision 1.10 2006/07/20 09:54:40 jgrosseo
242introducing status management: The processing per subdetector is divided into several steps,
243after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
244can keep track of the number of failures and skips further processing after a certain threshold is
245exceeded. These thresholds can be configured in LDAP.
246
5164a766 247Revision 1.9 2006/07/19 10:09:55 jgrosseo
248new configuration, accesst to DAQ FES (Alberto)
249
57f50b3c 250Revision 1.8 2006/07/11 12:44:36 jgrosseo
251adding parameters for extended validity range of data produced by preprocessor
252
17111222 253Revision 1.7 2006/07/10 14:37:09 jgrosseo
254small fix + todo comment
255
e090413b 256Revision 1.6 2006/07/10 13:01:41 jgrosseo
257enhanced storing of last sucessfully processed run (alberto)
258
a7160fe9 259Revision 1.5 2006/07/04 14:59:57 jgrosseo
260revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
261
45a493ce 262Revision 1.4 2006/06/12 09:11:16 jgrosseo
263coding conventions (Alberto)
264
58bc3020 265Revision 1.3 2006/06/06 14:26:40 jgrosseo
266o) removed files that were moved to STEER
267o) shuttle updated to follow the new interface (Alberto)
268
b948db8d 269Revision 1.2 2006/03/07 07:52:34 hristov
270New version (B.Yordanov)
271
d477ad88 272Revision 1.6 2005/11/19 17:19:14 byordano
273RetrieveDATEEntries and RetrieveConditionsData added
274
275Revision 1.5 2005/11/19 11:09:27 byordano
276AliShuttle declaration added
277
278Revision 1.4 2005/11/17 17:47:34 byordano
279TList changed to TObjArray
280
281Revision 1.3 2005/11/17 14:43:23 byordano
282import to local CVS
283
284Revision 1.1.1.1 2005/10/28 07:33:58 hristov
285Initial import as subdirectory in AliRoot
286
73abe331 287Revision 1.2 2005/09/13 08:41:15 byordano
288default startTime endTime added
289
290Revision 1.4 2005/08/30 09:13:02 byordano
291some docs added
292
293Revision 1.3 2005/08/29 21:15:47 byordano
294some docs added
295
296*/
297
298//
299// This class is the main manager for AliShuttle.
300// It organizes the data retrieval from DCS and call the
b948db8d 301// interface methods of AliPreprocessor.
73abe331 302// For every detector in AliShuttleConfgi (see AliShuttleConfig),
303// data for its set of aliases is retrieved. If there is registered
b948db8d 304// AliPreprocessor for this detector then it will be used
305// accroding to the schema (see AliPreprocessor).
306// If there isn't registered AliPreprocessor than the retrieved
73abe331 307// data is stored automatically to the undelying AliCDBStorage.
308// For detSpec is used the alias name.
309//
310
311#include "AliShuttle.h"
312
313#include "AliCDBManager.h"
314#include "AliCDBStorage.h"
315#include "AliCDBId.h"
84090f85 316#include "AliCDBRunRange.h"
317#include "AliCDBPath.h"
5164a766 318#include "AliCDBEntry.h"
73abe331 319#include "AliShuttleConfig.h"
eba76848 320#include "DCSClient/AliDCSClient.h"
73abe331 321#include "AliLog.h"
b948db8d 322#include "AliPreprocessor.h"
5164a766 323#include "AliShuttleStatus.h"
2bb7b766 324#include "AliShuttleLogbookEntry.h"
73abe331 325
57f50b3c 326#include <TSystem.h>
58bc3020 327#include <TObject.h>
b948db8d 328#include <TString.h>
57f50b3c 329#include <TTimeStamp.h>
73abe331 330#include <TObjString.h>
57f50b3c 331#include <TSQLServer.h>
332#include <TSQLResult.h>
333#include <TSQLRow.h>
cb343cfd 334#include <TMutex.h>
9827400b 335#include <TSystemDirectory.h>
336#include <TSystemFile.h>
a986b218 337#include <TFile.h>
9827400b 338#include <TGrid.h>
339#include <TGridResult.h>
73abe331 340
e7f62f16 341#include <TMonaLisaWriter.h>
342
5164a766 343#include <fstream>
344
cb343cfd 345#include <sys/types.h>
346#include <sys/wait.h>
347
73abe331 348ClassImp(AliShuttle)
349
b948db8d 350//______________________________________________________________________________________________
351AliShuttle::AliShuttle(const AliShuttleConfig* config,
352 UInt_t timeout, Int_t retries):
4f0ab988 353fConfig(config),
354fTimeout(timeout), fRetries(retries),
355fPreprocessorMap(),
2bb7b766 356fLogbookEntry(0),
eba76848 357fCurrentDetector(),
85a80aa9 358fStatusEntry(0),
cb343cfd 359fMonitoringMutex(0),
eba76848 360fLastActionTime(0),
e7f62f16 361fLastAction(),
9827400b 362fMonaLisa(0),
363fTestMode(kNone),
ffa29e93 364fReadTestMode(kFALSE),
365fOutputRedirected(kFALSE)
73abe331 366{
367 //
368 // config: AliShuttleConfig used
73abe331 369 // timeout: timeout used for AliDCSClient connection
370 // retries: the number of retries in case of connection error.
371 //
372
57f50b3c 373 if (!fConfig->IsValid()) AliFatal("********** !!!!! Invalid configuration !!!!! **********");
be48e3ea 374 for(int iSys=0;iSys<4;iSys++) {
57f50b3c 375 fServer[iSys]=0;
be48e3ea 376 if (iSys < 3)
2c15234c 377 fFXSlist[iSys].SetOwner(kTRUE);
57f50b3c 378 }
2bb7b766 379 fPreprocessorMap.SetOwner(kTRUE);
be48e3ea 380
381 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
382 fFirstUnprocessed[iDet] = kFALSE;
383
cb343cfd 384 fMonitoringMutex = new TMutex();
58bc3020 385}
386
b948db8d 387//______________________________________________________________________________________________
57f50b3c 388AliShuttle::~AliShuttle()
58bc3020 389{
9827400b 390 //
391 // destructor
392 //
58bc3020 393
b948db8d 394 fPreprocessorMap.DeleteAll();
be48e3ea 395 for(int iSys=0;iSys<4;iSys++)
57f50b3c 396 if(fServer[iSys]) {
397 fServer[iSys]->Close();
398 delete fServer[iSys];
eba76848 399 fServer[iSys] = 0;
57f50b3c 400 }
2bb7b766 401
402 if (fStatusEntry){
403 delete fStatusEntry;
404 fStatusEntry = 0;
405 }
cb343cfd 406
407 if (fMonitoringMutex)
408 {
409 delete fMonitoringMutex;
410 fMonitoringMutex = 0;
411 }
73abe331 412}
413
b948db8d 414//______________________________________________________________________________________________
57f50b3c 415void AliShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
58bc3020 416{
73abe331 417 //
b948db8d 418 // Registers new AliPreprocessor.
73abe331 419 // It uses GetName() for indentificator of the pre processor.
420 // The pre processor is registered it there isn't any other
421 // with the same identificator (GetName()).
422 //
423
eba76848 424 const char* detName = preprocessor->GetName();
425 if(GetDetPos(detName) < 0)
426 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
427
428 if (fPreprocessorMap.GetValue(detName)) {
429 AliWarning(Form("AliPreprocessor %s is already registered!", detName));
73abe331 430 return;
431 }
432
eba76848 433 fPreprocessorMap.Add(new TObjString(detName), preprocessor);
73abe331 434}
b948db8d 435//______________________________________________________________________________________________
3301427a 436Bool_t AliShuttle::Store(const AliCDBPath& path, TObject* object,
84090f85 437 AliCDBMetaData* metaData, Int_t validityStart, Bool_t validityInfinite)
73abe331 438{
9827400b 439 // Stores a CDB object in the storage for offline reconstruction. Objects that are not needed for
440 // offline reconstruction, but should be stored anyway (e.g. for debugging) should NOT be stored
441 // using this function. Use StoreReferenceData instead!
442 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
443 // finishes the data are transferred to the main storage (Grid).
b948db8d 444
3301427a 445 return StoreLocally(fgkLocalCDB, path, object, metaData, validityStart, validityInfinite);
84090f85 446}
447
448//______________________________________________________________________________________________
3301427a 449Bool_t AliShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 450{
9827400b 451 // Stores a CDB object in the storage for reference data. This objects will not be available during
452 // offline reconstrunction. Use this function for reference data only!
453 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
454 // finishes the data are transferred to the main storage (Grid).
85a80aa9 455
3301427a 456 return StoreLocally(fgkLocalRefStorage, path, object, metaData);
85a80aa9 457}
458
459//______________________________________________________________________________________________
3301427a 460Bool_t AliShuttle::StoreLocally(const TString& localUri,
85a80aa9 461 const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
462 Int_t validityStart, Bool_t validityInfinite)
463{
9827400b 464 // Store object temporarily in local storage. Parameters are passed by Store and StoreReferenceData functions.
465 // when the preprocessor finishes the data are transferred to the main storage (Grid).
466 // The parameters are:
467 // 1) Uri of the backup storage (Local)
468 // 2) the object's path.
469 // 3) the object to be stored
470 // 4) the metaData to be associated with the object
471 // 5) the validity start run number w.r.t. the current run,
472 // if the data is valid only for this run leave the default 0
473 // 6) specifies if the calibration data is valid for infinity (this means until updated),
474 // typical for calibration runs, the default is kFALSE
475 //
476 // returns 0 if fail, 1 otherwise
84090f85 477
9827400b 478 if (fTestMode & kErrorStorage)
479 {
480 Log(fCurrentDetector, "StoreLocally - In TESTMODE - Simulating error while storing locally");
481 return kFALSE;
482 }
483
3301427a 484 const char* cdbType = (localUri == fgkLocalCDB) ? "CDB" : "Reference";
2bb7b766 485
85a80aa9 486 Int_t firstRun = GetCurrentRun() - validityStart;
84090f85 487 if(firstRun < 0) {
9827400b 488 AliWarning("First valid run happens to be less than 0! Setting it to 0.");
84090f85 489 firstRun=0;
490 }
491
492 Int_t lastRun = -1;
493 if(validityInfinite) {
494 lastRun = AliCDBRunRange::Infinity();
495 } else {
496 lastRun = GetCurrentRun();
497 }
498
3301427a 499 // Version is set to current run, it will be used later to transfer data to Grid
500 AliCDBId id(path, firstRun, lastRun, GetCurrentRun(), -1);
2bb7b766 501
502 if(! dynamic_cast<TObjString*> (metaData->GetProperty("RunUsed(TObjString)"))){
503 TObjString runUsed = Form("%d", GetCurrentRun());
9e080f92 504 metaData->SetProperty("RunUsed(TObjString)", runUsed.Clone());
2bb7b766 505 }
84090f85 506
3301427a 507 Bool_t result = kFALSE;
84090f85 508
3301427a 509 if (!(AliCDBManager::Instance()->GetStorage(localUri))) {
510 Log("SHUTTLE", Form("StoreLocally - Cannot activate local %s storage", cdbType));
84090f85 511 } else {
3301427a 512 result = AliCDBManager::Instance()->GetStorage(localUri)
84090f85 513 ->Put(object, id, metaData);
514 }
515
516 if(!result) {
517
9827400b 518 Log(fCurrentDetector, Form("StoreLocally - Can't store object <%s>!", id.ToString().Data()));
3301427a 519 }
2bb7b766 520
3301427a 521 return result;
522}
84090f85 523
3301427a 524//______________________________________________________________________________________________
525Bool_t AliShuttle::StoreOCDB()
526{
9827400b 527 //
528 // Called when preprocessor ends successfully or when previous storage attempt failed (kStoreError status)
529 // Calls underlying StoreOCDB(const char*) function twice, for OCDB and Reference storage.
530 // Then calls StoreRefFilesToGrid to store reference files.
531 //
532
533 if (fTestMode & kErrorGrid)
534 {
535 Log("SHUTTLE", "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
536 Log(fCurrentDetector, "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
537 return kFALSE;
538 }
539
c88ad5db 540 Log("SHUTTLE","StoreOCDB - Storing OCDB data ...");
86aa42c3 541 Bool_t resultCDB = StoreOCDB(fgkMainCDB);
542
c88ad5db 543 Log("SHUTTLE","StoreOCDB - Storing reference data ...");
3301427a 544 Bool_t resultRef = StoreOCDB(fgkMainRefStorage);
9827400b 545
c88ad5db 546 Log("SHUTTLE","StoreOCDB - Storing reference files ...");
547 Bool_t resultRefFiles = CopyFilesToGrid("reference");
548
549 Bool_t resultMetadata = kTRUE;
550 if(fCurrentDetector == "GRP")
551 {
552 Log("StoreOCDB - SHUTTLE","Storing Run Metadata file ...");
553 resultMetadata = CopyFilesToGrid("metadata");
554 }
9827400b 555
c88ad5db 556 return resultCDB && resultRef && resultRefFiles && resultMetadata;
3301427a 557}
558
559//______________________________________________________________________________________________
560Bool_t AliShuttle::StoreOCDB(const TString& gridURI)
561{
562 //
563 // Called by StoreOCDB(), performs actual storage to the main OCDB and reference storages (Grid)
564 //
565
566 TObjArray* gridIds=0;
567
568 Bool_t result = kTRUE;
569
570 const char* type = 0;
571 TString localURI;
572 if(gridURI == fgkMainCDB) {
573 type = "OCDB";
574 localURI = fgkLocalCDB;
575 } else if(gridURI == fgkMainRefStorage) {
576 type = "reference";
577 localURI = fgkLocalRefStorage;
578 } else {
579 AliError(Form("Invalid storage URI: %s", gridURI.Data()));
580 return kFALSE;
581 }
582
583 AliCDBManager* man = AliCDBManager::Instance();
584
585 AliCDBStorage *gridSto = man->GetStorage(gridURI);
586 if(!gridSto) {
587 Log("SHUTTLE",
588 Form("StoreOCDB - cannot activate main %s storage", type));
589 return kFALSE;
590 }
591
592 gridIds = gridSto->GetQueryCDBList();
593
594 // get objects previously stored in local CDB
595 AliCDBStorage *localSto = man->GetStorage(localURI);
596 if(!localSto) {
597 Log("SHUTTLE",
598 Form("StoreOCDB - cannot activate local %s storage", type));
599 return kFALSE;
600 }
601 AliCDBPath aPath(GetOfflineDetName(fCurrentDetector.Data()),"*","*");
602 // Local objects were stored with current run as Grid version!
603 TList* localEntries = localSto->GetAll(aPath.GetPath(), GetCurrentRun(), GetCurrentRun());
604 localEntries->SetOwner(1);
605
606 // loop on local stored objects
607 TIter localIter(localEntries);
608 AliCDBEntry *aLocEntry = 0;
609 while((aLocEntry = dynamic_cast<AliCDBEntry*> (localIter.Next()))){
610 aLocEntry->SetOwner(1);
611 AliCDBId aLocId = aLocEntry->GetId();
612 aLocEntry->SetVersion(-1);
613 aLocEntry->SetSubVersion(-1);
614
615 // If local object is valid up to infinity we store it only if it is
616 // the first unprocessed run!
617 if (aLocId.GetLastRun() == AliCDBRunRange::Infinity() &&
618 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
619 {
620 Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
621 "there are previous unprocessed runs!",
622 fCurrentDetector.Data(), aLocId.GetPath().Data()));
623 continue;
624 }
625
626 // loop on Grid valid Id's
627 Bool_t store = kTRUE;
628 TIter gridIter(gridIds);
629 AliCDBId* aGridId = 0;
630 while((aGridId = dynamic_cast<AliCDBId*> (gridIter.Next()))){
631 if(aGridId->GetPath() != aLocId.GetPath()) continue;
632 // skip all objects valid up to infinity
633 if(aGridId->GetLastRun() == AliCDBRunRange::Infinity()) continue;
634 // if we get here, it means there's already some more recent object stored on Grid!
635 store = kFALSE;
636 break;
637 }
638
639 // If we get here, the file can be stored!
640 Bool_t storeOk = gridSto->Put(aLocEntry);
641 if(!store || storeOk){
642
643 if (!store)
644 {
645 Log(fCurrentDetector.Data(),
646 Form("StoreOCDB - A more recent object already exists in %s storage: <%s>",
647 type, aGridId->ToString().Data()));
648 } else {
649 Log("SHUTTLE",
650 Form("StoreOCDB - Object <%s> successfully put into %s storage",
651 aLocId.ToString().Data(), type));
2d9019b4 652 Log(fCurrentDetector.Data(),
653 Form("StoreOCDB - Object <%s> successfully put into %s storage",
654 aLocId.ToString().Data(), type));
3301427a 655 }
84090f85 656
3301427a 657 // removing local filename...
658 TString filename;
659 localSto->IdToFilename(aLocId, filename);
c88ad5db 660 Log("SHUTTLE", Form("StoreOCDB - Removing local file %s", filename.Data()));
3301427a 661 RemoveFile(filename.Data());
662 continue;
663 } else {
664 Log("SHUTTLE",
665 Form("StoreOCDB - Grid %s storage of object <%s> failed",
666 type, aLocId.ToString().Data()));
2d9019b4 667 Log(fCurrentDetector.Data(),
668 Form("StoreOCDB - Grid %s storage of object <%s> failed",
669 type, aLocId.ToString().Data()));
3301427a 670 result = kFALSE;
b948db8d 671 }
672 }
3301427a 673 localEntries->Clear();
2bb7b766 674
b948db8d 675 return result;
3301427a 676}
677
546242fb 678//______________________________________________________________________________________________
679Bool_t AliShuttle::CleanReferenceStorage(const char* detector)
680{
2d9019b4 681 // clears the directory used to store reference files of a given subdetector
546242fb 682
683 AliCDBManager* man = AliCDBManager::Instance();
684 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
2d9019b4 685 TString localBaseFolder = sto->GetBaseFolder();
686
687 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
688
d524ade6 689 Log("SHUTTLE", Form("CleanReferenceStorage - Cleaning %s", targetDir.Data()));
2d9019b4 690
691 TString begin;
692 begin.Form("%d_", GetCurrentRun());
693
694 TSystemDirectory* baseDir = new TSystemDirectory("/", targetDir);
695 if (!baseDir)
696 return kTRUE;
697
698 TList* dirList = baseDir->GetListOfFiles();
699 delete baseDir;
700
701 if (!dirList) return kTRUE;
702
703 if (dirList->GetEntries() < 3)
704 {
705 delete dirList;
706 return kTRUE;
707 }
708
709 Int_t nDirs = 0, nDel = 0;
710 TIter dirIter(dirList);
711 TSystemFile* entry = 0;
546242fb 712
2d9019b4 713 Bool_t success = kTRUE;
546242fb 714
2d9019b4 715 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
716 {
717 if (entry->IsDirectory())
718 continue;
719
720 TString fileName(entry->GetName());
721 if (!fileName.BeginsWith(begin))
722 continue;
723
724 nDirs++;
725
726 // delete file
727 Int_t result = gSystem->Unlink(fileName.Data());
728
729 if (result)
730 {
d524ade6 731 Log("SHUTTLE", Form("CleanReferenceStorage - Could not delete file %s!", fileName.Data()));
2d9019b4 732 success = kFALSE;
733 } else {
734 nDel++;
735 }
736 }
737
738 if(nDirs > 0)
739 Log("SHUTTLE", Form("CleanReferenceStorage - %d (over %d) reference files in folder %s were deleted.",
740 nDel, nDirs, targetDir.Data()));
741
742
743 delete dirList;
744 return success;
745
746
747
748
749
546242fb 750
751 Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
752 if (result == 0)
753 {
754 // delete directory
d524ade6 755 result = gSystem->Exec(Form("rm -rf %s", targetDir.Data()));
546242fb 756 if (result != 0)
757 {
d524ade6 758 Log("SHUTTLE", Form("CleanReferenceStorage - Could not clean directory %s", targetDir.Data()));
546242fb 759 return kFALSE;
760 }
761 }
762
763 result = gSystem->mkdir(targetDir, kTRUE);
764 if (result != 0)
765 {
c88ad5db 766 Log("SHUTTLE", Form("CleanReferenceStorage - Error creating base directory %s", targetDir.Data()));
546242fb 767 return kFALSE;
768 }
769
770 return kTRUE;
771}
772
9827400b 773//______________________________________________________________________________________________
774Bool_t AliShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
775{
776 //
3c2a21c8 777 // Stores reference file directly (without opening it). This function stores the file locally.
9827400b 778 //
3c2a21c8 779 // The file is stored under the following location:
780 // <base folder of local reference storage>/<DET>/<RUN#>_<gridFileName>
781 // where <gridFileName> is the second parameter given to the function
782 //
9827400b 783
784 if (fTestMode & kErrorStorage)
785 {
786 Log(fCurrentDetector, "StoreReferenceFile - In TESTMODE - Simulating error while storing locally");
787 return kFALSE;
788 }
789
790 AliCDBManager* man = AliCDBManager::Instance();
791 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
792
793 TString localBaseFolder = sto->GetBaseFolder();
794
d524ade6 795 TString target = GetRefFilePrefix(localBaseFolder.Data(), detector);
796 target.Append(Form("/%d_%s", GetCurrentRun(), gridFileName));
9827400b 797
d524ade6 798 return CopyFileLocally(localFile, target);
c88ad5db 799}
800
801//______________________________________________________________________________________________
802Bool_t AliShuttle::StoreRunMetadataFile(const char* localFile, const char* gridFileName)
803{
804 //
805 // Stores Run metadata file to the Grid, in the run folder
806 //
807 // Only GRP can call this function.
808
809 if (fTestMode & kErrorStorage)
810 {
811 Log(fCurrentDetector, "StoreRunMetaDataFile - In TESTMODE - Simulating error while storing locally");
812 return kFALSE;
813 }
814
815 AliCDBManager* man = AliCDBManager::Instance();
816 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
817
818 TString localBaseFolder = sto->GetBaseFolder();
819
820 // Build Run level folder
821 // folder = /alice/data/year/lhcPeriod/runNb/Raw
822
c88ad5db 823
675f64cd 824 TString lhcPeriod = GetLHCPeriod();
c88ad5db 825 if (lhcPeriod.Length() == 0)
826 {
827 Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
828 return 0;
829 }
830
675f64cd 831 TString target = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw/%s",
832 localBaseFolder.Data(), GetCurrentYear(),
d524ade6 833 lhcPeriod.Data(), GetCurrentRun(), gridFileName);
c88ad5db 834
d524ade6 835 return CopyFileLocally(localFile, target);
c88ad5db 836}
837
838//______________________________________________________________________________________________
d524ade6 839Bool_t AliShuttle::CopyFileLocally(const char* localFile, const TString& target)
c88ad5db 840{
841 //
842 // Stores file locally. Called by StoreReferenceFile and StoreRunMetadataFile
d524ade6 843 // Files are temporarily stored in the local reference storage. When the preprocessor
844 // finishes, the Shuttle calls CopyFilesToGrid to transfer the files to AliEn
845 // (in reference or run level folders)
c88ad5db 846 //
847
d524ade6 848 TString targetDir(target(0, target.Last('/')));
849
850 //try to open base dir folder, if it does not exist
2d9019b4 851 void* dir = gSystem->OpenDirectory(targetDir.Data());
852 if (dir == NULL) {
853 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
c88ad5db 854 Log("SHUTTLE", Form("StoreFileLocally - Can't open directory <%s>", targetDir.Data()));
2d9019b4 855 return kFALSE;
856 }
857
858 } else {
859 gSystem->FreeDirectory(dir);
860 }
9827400b 861
7d43a416 862 Int_t result = 0;
863
864 result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
9827400b 865 if (result)
866 {
c88ad5db 867 Log("SHUTTLE", Form("StoreFileLocally - %s does not exist", localFile));
546242fb 868 return kFALSE;
9827400b 869 }
546242fb 870
7d43a416 871 result = gSystem->GetPathInfo(target, 0, (Long64_t*) 0, 0, 0);
872 if (!result)
873 {
874 Log("SHUTTLE", Form("StoreFileLocally - target file %s already exist, removing...", target.Data()));
875 if (gSystem->Unlink(target.Data()))
876 {
877 Log("SHUTTLE", Form("StoreFileLocally - Could not remove existing target file %s!", target.Data()));
878 return kFALSE;
879 }
880 }
881
9827400b 882 result = gSystem->CopyFile(localFile, target);
883
884 if (result == 0)
885 {
c88ad5db 886 Log("SHUTTLE", Form("StoreFileLocally - File %s stored locally to %s", localFile, target.Data()));
9827400b 887 return kTRUE;
888 }
889 else
890 {
d524ade6 891 Log("SHUTTLE", Form("StoreFileLocally - Could not store file %s to %s! Error code = %d",
546242fb 892 localFile, target.Data(), result));
9827400b 893 return kFALSE;
894 }
c88ad5db 895
896
897
9827400b 898}
899
900//______________________________________________________________________________________________
c88ad5db 901Bool_t AliShuttle::CopyFilesToGrid(const char* type)
9827400b 902{
903 //
c88ad5db 904 // Transfers local files to the Grid. Local files can be reference files
905 // or run metadata file (from GRP only).
9827400b 906 //
c88ad5db 907 // According to the type (ref, metadata) the files are stored under the following location:
908 // ref --> <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
909 // metadata --> <run data folder>/<MetadataFileName>
86aa42c3 910 //
c88ad5db 911
9827400b 912 AliCDBManager* man = AliCDBManager::Instance();
913 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
914 if (!sto)
915 return kFALSE;
916 TString localBaseFolder = sto->GetBaseFolder();
9827400b 917
c88ad5db 918 TString dir;
919 TString alienDir;
9827400b 920 TString begin;
9827400b 921
c88ad5db 922 if (strcmp(type, "reference") == 0)
923 {
924 dir = GetRefFilePrefix(localBaseFolder.Data(), fCurrentDetector.Data());
925 AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
926 if (!gridSto)
927 return kFALSE;
928 TString gridBaseFolder = gridSto->GetBaseFolder();
929 alienDir = GetRefFilePrefix(gridBaseFolder.Data(), fCurrentDetector.Data());
930 begin = Form("%d_", GetCurrentRun());
931 }
932 else if (strcmp(type, "metadata") == 0)
933 {
c88ad5db 934
675f64cd 935 TString lhcPeriod = GetLHCPeriod();
c88ad5db 936
937 if (lhcPeriod.Length() == 0)
938 {
939 Log("SHUTTLE","CopyFilesToGrid - LHCPeriod not found in logbook!");
940 return 0;
941 }
942
675f64cd 943 dir = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw",
944 localBaseFolder.Data(), GetCurrentYear(),
c88ad5db 945 lhcPeriod.Data(), GetCurrentRun());
675f64cd 946 alienDir = dir(dir.Index("/alice/data/"), dir.Length());
947
c88ad5db 948 begin = "";
949 }
950 else
951 {
952 Log("SHUTTLE", "CopyFilesToGrid - Unexpected: type label must be reference or metadata!");
953 return kFALSE;
954 }
955
9827400b 956 TSystemDirectory* baseDir = new TSystemDirectory("/", dir);
3d8bc902 957 if (!baseDir)
958 return kTRUE;
959
2d9019b4 960 TList* dirList = baseDir->GetListOfFiles();
961 delete baseDir;
962
963 if (!dirList) return kTRUE;
964
965 if (dirList->GetEntries() < 3)
3d8bc902 966 {
2d9019b4 967 delete dirList;
9827400b 968 return kTRUE;
3d8bc902 969 }
2d9019b4 970
546242fb 971 if (!gGrid)
972 {
c88ad5db 973 Log("SHUTTLE", "CopyFilesToGrid - Connection to Grid failed: Cannot continue!");
2d9019b4 974 delete dirList;
546242fb 975 return kFALSE;
976 }
977
2d9019b4 978 Int_t nDirs = 0, nTransfer = 0;
979 TIter dirIter(dirList);
980 TSystemFile* entry = 0;
981
9827400b 982 Bool_t success = kTRUE;
3d8bc902 983 Bool_t first = kTRUE;
9827400b 984
2d9019b4 985 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
986 {
9827400b 987 if (entry->IsDirectory())
988 continue;
989
990 TString fileName(entry->GetName());
991 if (!fileName.BeginsWith(begin))
992 continue;
993
2d9019b4 994 nDirs++;
995
3d8bc902 996 if (first)
997 {
998 first = kFALSE;
c88ad5db 999 // check that folder exists, otherwise create it
3d8bc902 1000 TGridResult* result = gGrid->Ls(alienDir.Data(), "a");
1001
1002 if (!result)
2d9019b4 1003 {
1004 delete dirList;
3d8bc902 1005 return kFALSE;
2d9019b4 1006 }
3d8bc902 1007
546242fb 1008 if (!result->GetFileName(1)) // TODO: It looks like element 0 is always 0!!
3d8bc902 1009 {
675f64cd 1010 // TODO It does not work currently! Bug in TAliEn::Mkdir
1011 // TODO Manually fixed in local root v5-16-00
c88ad5db 1012 if (!gGrid->Mkdir(alienDir.Data(),"-p",0))
3d8bc902 1013 {
c88ad5db 1014 Log("SHUTTLE", Form("CopyFilesToGrid - Cannot create directory %s",
3d8bc902 1015 alienDir.Data()));
2d9019b4 1016 delete dirList;
3d8bc902 1017 return kFALSE;
546242fb 1018 } else {
c88ad5db 1019 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s created", alienDir.Data()));
3d8bc902 1020 }
1021
546242fb 1022 } else {
c88ad5db 1023 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s found", alienDir.Data()));
3d8bc902 1024 }
1025 }
1026
9827400b 1027 TString fullLocalPath;
1028 fullLocalPath.Form("%s/%s", dir.Data(), fileName.Data());
1029
1030 TString fullGridPath;
1031 fullGridPath.Form("alien://%s/%s", alienDir.Data(), fileName.Data());
1032
a986b218 1033 Bool_t result = TFile::Cp(fullLocalPath, fullGridPath);
9827400b 1034
1035 if (result)
1036 {
c88ad5db 1037 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s succeeded!",
1038 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1039 RemoveFile(fullLocalPath);
2d9019b4 1040 nTransfer++;
9827400b 1041 }
1042 else
1043 {
c88ad5db 1044 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s FAILED!",
1045 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1046 success = kFALSE;
1047 }
1048 }
2d9019b4 1049
c88ad5db 1050 Log("SHUTTLE", Form("CopyFilesToGrid - %d (over %d) files in folder %s copied to Grid.",
1051 nTransfer, nDirs, dir.Data()));
2d9019b4 1052
1053
1054 delete dirList;
9827400b 1055 return success;
1056}
1057
2d9019b4 1058//______________________________________________________________________________________________
1059const char* AliShuttle::GetRefFilePrefix(const char* base, const char* detector)
1060{
1061 //
1062 // Get folder name of reference files
1063 //
1064
1065 TString offDetStr(GetOfflineDetName(detector));
1066 TString dir;
1067 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
1068 {
1069 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
1070 } else {
1071 dir.Form("%s/%s", base, offDetStr.Data());
1072 }
1073
1074 return dir.Data();
1075
1076
1077}
c88ad5db 1078
3301427a 1079//______________________________________________________________________________________________
1080void AliShuttle::CleanLocalStorage(const TString& uri)
1081{
9827400b 1082 //
1083 // Called in case the preprocessor is declared failed. Remove remaining objects from the local storages.
1084 //
3301427a 1085
1086 const char* type = 0;
1087 if(uri == fgkLocalCDB) {
1088 type = "OCDB";
1089 } else if(uri == fgkLocalRefStorage) {
546242fb 1090 type = "Reference";
3301427a 1091 } else {
1092 AliError(Form("Invalid storage URI: %s", uri.Data()));
1093 return;
1094 }
1095
1096 AliCDBManager* man = AliCDBManager::Instance();
b948db8d 1097
3301427a 1098 // open local storage
1099 AliCDBStorage *localSto = man->GetStorage(uri);
1100 if(!localSto) {
1101 Log("SHUTTLE",
1102 Form("CleanLocalStorage - cannot activate local %s storage", type));
1103 return;
1104 }
1105
1106 TString filename(Form("%s/%s/*/Run*_v%d_s*.root",
546242fb 1107 localSto->GetBaseFolder().Data(), GetOfflineDetName(fCurrentDetector.Data()), GetCurrentRun()));
3301427a 1108
c88ad5db 1109 AliDebug(2, Form("filename = %s", filename.Data()));
3301427a 1110
c88ad5db 1111 Log("SHUTTLE", Form("Removing remaining local files for run %d and detector %s ...",
3301427a 1112 GetCurrentRun(), fCurrentDetector.Data()));
1113
1114 RemoveFile(filename.Data());
1115
1116}
1117
1118//______________________________________________________________________________________________
1119void AliShuttle::RemoveFile(const char* filename)
1120{
9827400b 1121 //
1122 // removes local file
1123 //
3301427a 1124
1125 TString command(Form("rm -f %s", filename));
1126
1127 Int_t result = gSystem->Exec(command.Data());
1128 if(result != 0)
1129 {
1130 Log("SHUTTLE", Form("RemoveFile - %s: Cannot remove file %s!",
1131 fCurrentDetector.Data(), filename));
1132 }
73abe331 1133}
1134
b948db8d 1135//______________________________________________________________________________________________
5164a766 1136AliShuttleStatus* AliShuttle::ReadShuttleStatus()
1137{
9827400b 1138 //
1139 // Reads the AliShuttleStatus from the CDB
1140 //
5164a766 1141
2bb7b766 1142 if (fStatusEntry){
1143 delete fStatusEntry;
1144 fStatusEntry = 0;
1145 }
5164a766 1146
10a5a932 1147 fStatusEntry = AliCDBManager::Instance()->GetStorage(GetLocalCDB())
2bb7b766 1148 ->Get(Form("/SHUTTLE/STATUS/%s", fCurrentDetector.Data()), GetCurrentRun());
5164a766 1149
2bb7b766 1150 if (!fStatusEntry) return 0;
1151 fStatusEntry->SetOwner(1);
5164a766 1152
2bb7b766 1153 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1154 if (!status) {
1155 AliError("Invalid object stored to CDB!");
1156 return 0;
1157 }
5164a766 1158
2bb7b766 1159 return status;
5164a766 1160}
1161
1162//______________________________________________________________________________________________
7bfb2090 1163Bool_t AliShuttle::WriteShuttleStatus(AliShuttleStatus* status)
5164a766 1164{
9827400b 1165 //
1166 // writes the status for one subdetector
1167 //
2bb7b766 1168
1169 if (fStatusEntry){
1170 delete fStatusEntry;
1171 fStatusEntry = 0;
1172 }
5164a766 1173
2bb7b766 1174 Int_t run = GetCurrentRun();
5164a766 1175
2bb7b766 1176 AliCDBId id(AliCDBPath("SHUTTLE", "STATUS", fCurrentDetector), run, run);
5164a766 1177
2bb7b766 1178 fStatusEntry = new AliCDBEntry(status, id, new AliCDBMetaData);
1179 fStatusEntry->SetOwner(1);
5164a766 1180
2bb7b766 1181 UInt_t result = AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
7bfb2090 1182
2bb7b766 1183 if (!result) {
3301427a 1184 Log("SHUTTLE", Form("WriteShuttleStatus - Failed for %s, run %d",
1185 fCurrentDetector.Data(), run));
2bb7b766 1186 return kFALSE;
1187 }
e7f62f16 1188
1189 SendMLInfo();
7bfb2090 1190
2bb7b766 1191 return kTRUE;
5164a766 1192}
1193
1194//______________________________________________________________________________________________
1195void AliShuttle::UpdateShuttleStatus(AliShuttleStatus::Status newStatus, Bool_t increaseCount)
1196{
9827400b 1197 //
1198 // changes the AliShuttleStatus for the given detector and run to the given status
1199 //
5164a766 1200
2bb7b766 1201 if (!fStatusEntry){
1202 AliError("UNEXPECTED: fStatusEntry empty");
1203 return;
1204 }
5164a766 1205
2bb7b766 1206 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
5164a766 1207
2bb7b766 1208 if (!status){
c88ad5db 1209 Log("SHUTTLE", "UpdateShuttleStatus - UNEXPECTED: status could not be read from current CDB entry");
2bb7b766 1210 return;
1211 }
5164a766 1212
2c15234c 1213 TString actionStr = Form("UpdateShuttleStatus - %s: Changing state from %s to %s",
eba76848 1214 fCurrentDetector.Data(),
36c99a6a 1215 status->GetStatusName(),
eba76848 1216 status->GetStatusName(newStatus));
cb343cfd 1217 Log("SHUTTLE", actionStr);
1218 SetLastAction(actionStr);
5164a766 1219
2bb7b766 1220 status->SetStatus(newStatus);
1221 if (increaseCount) status->IncreaseCount();
5164a766 1222
2bb7b766 1223 AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
e7f62f16 1224
1225 SendMLInfo();
5164a766 1226}
e7f62f16 1227
1228//______________________________________________________________________________________________
1229void AliShuttle::SendMLInfo()
1230{
1231 //
1232 // sends ML information about the current status of the current detector being processed
1233 //
1234
1235 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1236
1237 if (!status){
3301427a 1238 Log("SHUTTLE", "SendMLInfo - UNEXPECTED: status could not be read from current CDB entry");
e7f62f16 1239 return;
1240 }
1241
1242 TMonaLisaText mlStatus(Form("%s_status", fCurrentDetector.Data()), status->GetStatusName());
1243 TMonaLisaValue mlRetryCount(Form("%s_count", fCurrentDetector.Data()), status->GetCount());
1244
1245 TList mlList;
1246 mlList.Add(&mlStatus);
1247 mlList.Add(&mlRetryCount);
1248
1249 fMonaLisa->SendParameters(&mlList);
1250}
1251
5164a766 1252//______________________________________________________________________________________________
1253Bool_t AliShuttle::ContinueProcessing()
1254{
9827400b 1255 // this function reads the AliShuttleStatus information from CDB and
1256 // checks if the processing should be continued
1257 // if yes it returns kTRUE and updates the AliShuttleStatus with nextStatus
2bb7b766 1258
57c1a579 1259 if (!fConfig->HostProcessDetector(fCurrentDetector)) return kFALSE;
1260
1261 AliPreprocessor* aPreprocessor =
1262 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1263 if (!aPreprocessor)
1264 {
c88ad5db 1265 Log("SHUTTLE", Form("ContinueProcessing - %s: no preprocessor registered", fCurrentDetector.Data()));
57c1a579 1266 return kFALSE;
1267 }
1268
2bb7b766 1269 AliShuttleLogbookEntry::Status entryStatus =
eba76848 1270 fLogbookEntry->GetDetectorStatus(fCurrentDetector);
2bb7b766 1271
1272 if(entryStatus != AliShuttleLogbookEntry::kUnprocessed) {
c88ad5db 1273 Log("SHUTTLE", Form("ContinueProcessing - %s is %s",
2bb7b766 1274 fCurrentDetector.Data(),
1275 fLogbookEntry->GetDetectorStatusName(entryStatus)));
1276 return kFALSE;
1277 }
1278
1279 // if we get here, according to Shuttle logbook subdetector is in UNPROCESSED state
be48e3ea 1280
1281 // check if current run is first unprocessed run for current detector
1282 if (fConfig->StrictRunOrder(fCurrentDetector) &&
1283 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
1284 {
86aa42c3 1285 if (fTestMode == kNone)
1286 {
c88ad5db 1287 Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering"
1288 " but this is not the first unprocessed run!"));
86aa42c3 1289 return kFALSE;
1290 }
1291 else
1292 {
c88ad5db 1293 Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - "
1294 "Although %s requires strict run ordering "
1295 "and this is not the first unprocessed run, "
1296 "the SHUTTLE continues"));
86aa42c3 1297 }
be48e3ea 1298 }
1299
2bb7b766 1300 AliShuttleStatus* status = ReadShuttleStatus();
1301 if (!status) {
1302 // first time
1303 Log("SHUTTLE", Form("ContinueProcessing - %s: Processing first time",
1304 fCurrentDetector.Data()));
1305 status = new AliShuttleStatus(AliShuttleStatus::kStarted);
1306 return WriteShuttleStatus(status);
1307 }
1308
1309 // The following two cases shouldn't happen if Shuttle Logbook was correctly updated.
1310 // If it happens it may mean Logbook updating failed... let's do it now!
1311 if (status->GetStatus() == AliShuttleStatus::kDone ||
1312 status->GetStatus() == AliShuttleStatus::kFailed){
1313 Log("SHUTTLE", Form("ContinueProcessing - %s is already %s. Updating Shuttle Logbook",
1314 fCurrentDetector.Data(),
1315 status->GetStatusName(status->GetStatus())));
1316 UpdateShuttleLogbook(fCurrentDetector.Data(),
1317 status->GetStatusName(status->GetStatus()));
1318 return kFALSE;
1319 }
1320
3301427a 1321 if (status->GetStatus() == AliShuttleStatus::kStoreError) {
2bb7b766 1322 Log("SHUTTLE",
c88ad5db 1323 Form("ContinueProcessing - %s: Grid storage of one or more "
1324 "objects failed. Trying again now",
2bb7b766 1325 fCurrentDetector.Data()));
9827400b 1326 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1327 if (StoreOCDB()){
c88ad5db 1328 Log("SHUTTLE", Form("ContinueProcessing - %s: all objects "
1329 "successfully stored into main storage",
3301427a 1330 fCurrentDetector.Data()));
2bb7b766 1331 UpdateShuttleStatus(AliShuttleStatus::kDone);
1332 UpdateShuttleLogbook(fCurrentDetector.Data(), "DONE");
1333 } else {
1334 Log("SHUTTLE",
1335 Form("ContinueProcessing - %s: Grid storage failed again",
1336 fCurrentDetector.Data()));
9827400b 1337 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
2bb7b766 1338 }
1339 return kFALSE;
1340 }
1341
1342 // if we get here, there is a restart
57c1a579 1343 Bool_t cont = kFALSE;
2bb7b766 1344
1345 // abort conditions
cb343cfd 1346 if (status->GetCount() >= fConfig->GetMaxRetries()) {
57c1a579 1347 Log("SHUTTLE", Form("ContinueProcessing - %s failed %d times in status %s - "
1348 "Updating Shuttle Logbook", fCurrentDetector.Data(),
2bb7b766 1349 status->GetCount(), status->GetStatusName()));
1350 UpdateShuttleLogbook(fCurrentDetector.Data(), "FAILED");
e7f62f16 1351 UpdateShuttleStatus(AliShuttleStatus::kFailed);
3301427a 1352
1353 // there may still be objects in local OCDB and reference storage
1354 // and FXS databases may be not updated: do it now!
9827400b 1355
1356 // TODO Currently disabled, we want to keep files in case of failure!
1357 // CleanLocalStorage(fgkLocalCDB);
1358 // CleanLocalStorage(fgkLocalRefStorage);
1359 // UpdateTableFailCase();
1360
1361 // Send mail to detector expert!
c88ad5db 1362 Log("SHUTTLE", Form("ContinueProcessing - Sending mail to %s expert...",
1363 fCurrentDetector.Data()));
9827400b 1364 if (!SendMail())
1365 Log("SHUTTLE", Form("ContinueProcessing - Could not send mail to %s expert",
1366 fCurrentDetector.Data()));
3301427a 1367
57c1a579 1368 } else {
1369 Log("SHUTTLE", Form("ContinueProcessing - %s: restarting. "
1370 "Aborted before with %s. Retry number %d.", fCurrentDetector.Data(),
1371 status->GetStatusName(), status->GetCount()));
9827400b 1372 Bool_t increaseCount = kTRUE;
c88ad5db 1373 if (status->GetStatus() == AliShuttleStatus::kDCSError ||
1374 status->GetStatus() == AliShuttleStatus::kDCSStarted)
1375 increaseCount = kFALSE;
675f64cd 1376
9827400b 1377 UpdateShuttleStatus(AliShuttleStatus::kStarted, increaseCount);
57c1a579 1378 cont = kTRUE;
2bb7b766 1379 }
1380
57c1a579 1381 return cont;
5164a766 1382}
1383
1384//______________________________________________________________________________________________
2bb7b766 1385Bool_t AliShuttle::Process(AliShuttleLogbookEntry* entry)
58bc3020 1386{
73abe331 1387 //
b948db8d 1388 // Makes data retrieval for all detectors in the configuration.
2bb7b766 1389 // entry: Shuttle logbook entry, contains run paramenters and status of detectors
1390 // (Unprocessed, Inactive, Failed or Done).
d477ad88 1391 // Returns kFALSE in case of error occured and kTRUE otherwise
73abe331 1392 //
1393
9827400b 1394 if (!entry) return kFALSE;
2bb7b766 1395
1396 fLogbookEntry = entry;
1397
c88ad5db 1398 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^*",
9827400b 1399 GetCurrentRun()));
2bb7b766 1400
e7f62f16 1401 // create ML instance that monitors this run
468df1ce 1402 fMonaLisa = new TMonaLisaWriter(fConfig->GetMonitorHost(), fConfig->GetMonitorTable(), Form("%d", GetCurrentRun()));
2bb7b766 1403
e7f62f16 1404 // Send the information to ML
1405 TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
9827400b 1406 TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
e7f62f16 1407
1408 TList mlList;
1409 mlList.Add(&mlStatus);
9827400b 1410 mlList.Add(&mlRunType);
e7f62f16 1411
1412 fMonaLisa->SendParameters(&mlList);
3301427a 1413
9827400b 1414 if (fLogbookEntry->IsDone())
1415 {
1416 Log("SHUTTLE","Process - Shuttle is already DONE. Updating logbook");
1417 UpdateShuttleLogbook("shuttle_done");
1418 fLogbookEntry = 0;
1419 return kTRUE;
1420 }
1421
1422 // read test mode if flag is set
1423 if (fReadTestMode)
1424 {
3d8bc902 1425 fTestMode = kNone;
9827400b 1426 TString logEntry(entry->GetRunParameter("log"));
1427 //printf("log entry = %s\n", logEntry.Data());
1428 TString searchStr("Testmode: ");
1429 Int_t pos = logEntry.Index(searchStr.Data());
1430 //printf("%d\n", pos);
1431 if (pos >= 0)
1432 {
1433 TSubString subStr = logEntry(pos + searchStr.Length(), logEntry.Length());
1434 //printf("%s\n", subStr.String().Data());
1435 TString newStr(subStr.Data());
1436 TObjArray* token = newStr.Tokenize(' ');
1437 if (token)
1438 {
1439 //token->Print();
1440 TObjString* tmpStr = dynamic_cast<TObjString*> (token->First());
1441 if (tmpStr)
1442 {
1443 Int_t testMode = tmpStr->String().Atoi();
1444 if (testMode > 0)
1445 {
c88ad5db 1446 Log("SHUTTLE", Form("Process - Enabling test mode %d", testMode));
9827400b 1447 SetTestMode((TestMode) testMode);
1448 }
1449 }
1450 delete token;
1451 }
1452 }
1453 }
c88ad5db 1454
eba76848 1455 fLogbookEntry->Print("all");
57f50b3c 1456
1457 // Initialization
d477ad88 1458 Bool_t hasError = kFALSE;
5164a766 1459
675f64cd 1460 // Set the CDB and Reference folders according to the year and LHC period
1461 TString lhcPeriod(GetLHCPeriod());
1462 if (lhcPeriod.Length() == 0)
1463 {
7d4cf768 1464 Log("SHUTTLE","Process - LHCPeriod not found in logbook!");
1465 return 0;
675f64cd 1466 }
1467
1468 if (fgkMainCDB.Length() == 0)
1469 fgkMainCDB = Form("alien://folder=/alice/data/%d/%s/OCDB?user=alidaq?cacheFold=/tmp/OCDBCache",
1470 GetCurrentYear(), lhcPeriod.Data());
1471
1472 if (fgkMainRefStorage.Length() == 0)
1473 fgkMainRefStorage = Form("alien://folder=/alice/data/%d/%s/Reference?user=alidaq?cacheFold=/tmp/OCDBCache",
1474 GetCurrentYear(), lhcPeriod.Data());
1475
2bb7b766 1476 AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
1477 if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
1478 AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
1479 if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
d477ad88 1480
57f50b3c 1481 // Loop on detectors in the configuration
b948db8d 1482 TIter iter(fConfig->GetDetectors());
2bb7b766 1483 TObjString* aDetector = 0;
b948db8d 1484
be48e3ea 1485 while ((aDetector = (TObjString*) iter.Next()))
1486 {
7bfb2090 1487 fCurrentDetector = aDetector->String();
5164a766 1488
9e080f92 1489 if (ContinueProcessing() == kFALSE) continue;
1490
c88ad5db 1491 Log("SHUTTLE", Form("\t\t\t****** run %d - %s: START ******",
2bb7b766 1492 GetCurrentRun(), aDetector->GetName()));
1493
9d733021 1494 for(Int_t iSys=0;iSys<3;iSys++) fFXSCalled[iSys]=kFALSE;
1495
c88ad5db 1496 Log(fCurrentDetector.Data(), "Process - Starting processing");
85a80aa9 1497
be48e3ea 1498 Int_t pid = fork();
1499
1500 if (pid < 0)
1501 {
c88ad5db 1502 Log("SHUTTLE", "Process - ERROR: Forking failed");
be48e3ea 1503 }
1504 else if (pid > 0)
1505 {
1506 // parent
c88ad5db 1507 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Starting monitoring",
be48e3ea 1508 GetCurrentRun(), aDetector->GetName()));
1509
1510 Long_t begin = time(0);
1511
1512 int status; // to be used with waitpid, on purpose an int (not Int_t)!
1513 while (waitpid(pid, &status, WNOHANG) == 0)
1514 {
1515 Long_t expiredTime = time(0) - begin;
1516
1517 if (expiredTime > fConfig->GetPPTimeOut())
1518 {
9827400b 1519 TString tmp;
c88ad5db 1520 tmp.Form("Process - Process of %s time out. "
1521 "Run time: %d seconds. Killing...",
1522 fCurrentDetector.Data(), expiredTime);
9827400b 1523 Log("SHUTTLE", tmp);
1524 Log(fCurrentDetector, tmp);
be48e3ea 1525
1526 kill(pid, 9);
1527
3301427a 1528 UpdateShuttleStatus(AliShuttleStatus::kPPTimeOut);
be48e3ea 1529 hasError = kTRUE;
1530
1531 gSystem->Sleep(1000);
1532 }
1533 else
1534 {
be48e3ea 1535 gSystem->Sleep(1000);
9827400b 1536
1537 TString checkStr;
1538 checkStr.Form("ps -o vsize --pid %d | tail -n 1", pid);
1539 FILE* pipe = gSystem->OpenPipe(checkStr, "r");
1540 if (!pipe)
1541 {
c88ad5db 1542 Log("SHUTTLE", Form("Process - Error: "
1543 "Could not open pipe to %s", checkStr.Data()));
9827400b 1544 continue;
1545 }
1546
1547 char buffer[100];
1548 if (!fgets(buffer, 100, pipe))
1549 {
c88ad5db 1550 Log("SHUTTLE", "Process - Error: ps did not return anything");
9827400b 1551 gSystem->ClosePipe(pipe);
1552 continue;
1553 }
1554 gSystem->ClosePipe(pipe);
1555
1556 //Log("SHUTTLE", Form("ps returned %s", buffer));
1557
1558 Int_t mem = 0;
1559 if ((sscanf(buffer, "%d\n", &mem) != 1) || !mem)
1560 {
c88ad5db 1561 Log("SHUTTLE", "Process - Error: Could not parse output of ps");
9827400b 1562 continue;
1563 }
1564
1565 if (expiredTime % 60 == 0)
c88ad5db 1566 Log("SHUTTLE", Form("Process - %s: Checking process. "
1567 "Run time: %d seconds - Memory consumption: %d KB",
1568 fCurrentDetector.Data(), expiredTime, mem));
9827400b 1569
1570 if (mem > fConfig->GetPPMaxMem())
1571 {
1572 TString tmp;
c88ad5db 1573 tmp.Form("Process - Process exceeds maximum allowed memory "
1574 "(%d KB > %d KB). Killing...",
9827400b 1575 mem, fConfig->GetPPMaxMem());
1576 Log("SHUTTLE", tmp);
1577 Log(fCurrentDetector, tmp);
1578
1579 kill(pid, 9);
1580
1581 UpdateShuttleStatus(AliShuttleStatus::kPPOutOfMemory);
1582 hasError = kTRUE;
1583
1584 gSystem->Sleep(1000);
1585 }
be48e3ea 1586 }
1587 }
1588
c88ad5db 1589 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Client has terminated.",
be48e3ea 1590 GetCurrentRun(), aDetector->GetName()));
1591
1592 if (WIFEXITED(status))
1593 {
1594 Int_t returnCode = WEXITSTATUS(status);
1595
c88ad5db 1596 Log("SHUTTLE", Form("Process - %s: the return code is %d", fCurrentDetector.Data(),
3301427a 1597 returnCode));
be48e3ea 1598
9827400b 1599 if (returnCode == 0) hasError = kTRUE;
be48e3ea 1600 }
1601 }
1602 else if (pid == 0)
1603 {
1604 // client
c88ad5db 1605 Log("SHUTTLE", Form("Process - In client process of %d - %s", GetCurrentRun(),
1606 aDetector->GetName()));
be48e3ea 1607
c88ad5db 1608 Log("SHUTTLE", Form("Process - Redirecting output to %s log",fCurrentDetector.Data()));
ffa29e93 1609
546242fb 1610 if ((freopen(GetLogFileName(fCurrentDetector), "a", stdout)) == 0)
ffa29e93 1611 {
c88ad5db 1612 Log("SHUTTLE", "Process - Could not freopen stdout");
ffa29e93 1613 }
1614 else
1615 {
1616 fOutputRedirected = kTRUE;
1617 if ((dup2(fileno(stdout), fileno(stderr))) < 0)
c88ad5db 1618 Log("SHUTTLE", "Process - Could not redirect stderr");
ffa29e93 1619
1620 }
1621
5bac2bde 1622 TString wd = gSystem->WorkingDirectory();
675f64cd 1623 TString tmpDir = Form("%s/%s_%d_process", GetShuttleTempDir(),
1624 fCurrentDetector.Data(), GetCurrentRun());
5bac2bde 1625
d524ade6 1626 Int_t result = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
1627 if (!result) // temp dir already exists!
1628 {
1629 Log(fCurrentDetector.Data(),
1630 Form("Process - %s dir already exists! Removing...", tmpDir.Data()));
1631 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
675f64cd 1632 }
1633
1634 if (gSystem->mkdir(tmpDir.Data(), 1))
1635 {
1636 Log(fCurrentDetector.Data(), "Process - could not make temp directory!!");
1637 gSystem->Exit(1);
d524ade6 1638 }
1639
1640 if (!gSystem->ChangeDirectory(tmpDir.Data()))
1641 {
1642 Log(fCurrentDetector.Data(), "Process - could not change directory!!");
1643 gSystem->Exit(1);
1644 }
5bac2bde 1645
9827400b 1646 Bool_t success = ProcessCurrentDetector();
5bac2bde 1647
1648 gSystem->ChangeDirectory(wd.Data());
b1d18693 1649
9827400b 1650 if (success) // Preprocessor finished successfully!
1651 {
b1d18693 1652 // remove temporary folder
1653 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
1654
3301427a 1655 // Update time_processed field in FXS DB
1656 if (UpdateTable() == kFALSE)
5bac2bde 1657 Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!",
1658 fCurrentDetector.Data()));
3301427a 1659
1660 // Transfer the data from local storage to main storage (Grid)
1661 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1662 if (StoreOCDB() == kFALSE)
1663 {
c88ad5db 1664 Log("SHUTTLE",
1665 Form("\t\t\t****** run %d - %s: STORAGE ERROR ******",
3301427a 1666 GetCurrentRun(), aDetector->GetName()));
1667 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
9827400b 1668 success = kFALSE;
3301427a 1669 } else {
c88ad5db 1670 Log("SHUTTLE",
1671 Form("\t\t\t****** run %d - %s: DONE ******",
3301427a 1672 GetCurrentRun(), aDetector->GetName()));
1673 UpdateShuttleStatus(AliShuttleStatus::kDone);
9827400b 1674 UpdateShuttleLogbook(fCurrentDetector, "DONE");
3301427a 1675 }
c88ad5db 1676 } else
1677 {
1678 Log("SHUTTLE",
1679 Form("\t\t\t****** run %d - %s: PP ERROR ******",
1680 GetCurrentRun(), aDetector->GetName()));
be48e3ea 1681 }
1682
4b95672b 1683 for (UInt_t iSys=0; iSys<3; iSys++)
1684 {
1685 if (fFXSCalled[iSys]) fFXSlist[iSys].Clear();
1686 }
1687
c88ad5db 1688 Log("SHUTTLE", Form("Process - Client process of %d - %s is exiting now with %d.",
9827400b 1689 GetCurrentRun(), aDetector->GetName(), success));
be48e3ea 1690
1691 // the client exits here
9827400b 1692 gSystem->Exit(success);
be48e3ea 1693
1694 AliError("We should never get here!!!");
1695 }
7bfb2090 1696 }
5164a766 1697
c88ad5db 1698 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^*",
2bb7b766 1699 GetCurrentRun()));
1700
1701 //check if shuttle is done for this run, if so update logbook
1702 TObjArray checkEntryArray;
1703 checkEntryArray.SetOwner(1);
9e080f92 1704 TString whereClause = Form("where run=%d", GetCurrentRun());
1705 if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) || checkEntryArray.GetEntries() == 0) {
1706 Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
1707 GetCurrentRun()));
1708 return hasError == kFALSE;
1709 }
b948db8d 1710
9e080f92 1711 AliShuttleLogbookEntry* checkEntry = dynamic_cast<AliShuttleLogbookEntry*>
1712 (checkEntryArray.At(0));
2bb7b766 1713
9e080f92 1714 if (checkEntry)
1715 {
1716 if (checkEntry->IsDone())
be48e3ea 1717 {
9e080f92 1718 Log("SHUTTLE","Process - Shuttle is DONE. Updating logbook");
1719 UpdateShuttleLogbook("shuttle_done");
1720 }
1721 else
1722 {
1723 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
be48e3ea 1724 {
9e080f92 1725 if (checkEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
be48e3ea 1726 {
9e080f92 1727 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
1728 checkEntry->GetRun(), GetDetName(iDet)));
1729 fFirstUnprocessed[iDet] = kFALSE;
be48e3ea 1730 }
1731 }
2bb7b766 1732 }
1733 }
1734
e7f62f16 1735 // remove ML instance
1736 delete fMonaLisa;
1737 fMonaLisa = 0;
1738
2bb7b766 1739 fLogbookEntry = 0;
85a80aa9 1740
a7160fe9 1741 return hasError == kFALSE;
73abe331 1742}
1743
b948db8d 1744//______________________________________________________________________________________________
9827400b 1745Bool_t AliShuttle::ProcessCurrentDetector()
73abe331 1746{
1747 //
2bb7b766 1748 // Makes data retrieval just for a specific detector (fCurrentDetector).
73abe331 1749 // Threre should be a configuration for this detector.
73abe331 1750
1d172743 1751 Log("SHUTTLE", Form("ProcessCurrentDetector - Retrieving values for %s, run %d",
1752 fCurrentDetector.Data(), GetCurrentRun()));
73abe331 1753
d524ade6 1754 TString wd = gSystem->WorkingDirectory();
1755
2d9019b4 1756 if (!CleanReferenceStorage(fCurrentDetector.Data()))
546242fb 1757 return kFALSE;
d524ade6 1758
1759 gSystem->ChangeDirectory(wd.Data());
1760
1d172743 1761 TMap* dcsMap = new TMap();
3301427a 1762
1763 // call preprocessor
1764 AliPreprocessor* aPreprocessor =
1765 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1766
1767 aPreprocessor->Initialize(GetCurrentRun(), GetCurrentStartTime(), GetCurrentEndTime());
1768
1769 Bool_t processDCS = aPreprocessor->ProcessDCS();
d477ad88 1770
651fdaab 1771 if (!processDCS)
1772 {
1d172743 1773 Log(fCurrentDetector, "ProcessCurrentDetector -"
1774 " The preprocessor requested to skip the retrieval of DCS values");
651fdaab 1775 }
8b739301 1776 else if (fTestMode & kSkipDCS)
2c15234c 1777 {
1d172743 1778 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Skipping DCS processing");
9827400b 1779 }
1780 else if (fTestMode & kErrorDCS)
1781 {
1d172743 1782 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Simulating DCS error");
3d8bc902 1783 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
9827400b 1784 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1d172743 1785 delete dcsMap;
9827400b 1786 return kFALSE;
2c15234c 1787 } else {
3301427a 1788
1789 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
1790
1d172743 1791 // Query DCS archive
1792 Int_t nServers = fConfig->GetNServers(fCurrentDetector);
a038aa70 1793
1d172743 1794 for (int iServ=0; iServ<nServers; iServ++)
2c15234c 1795 {
1d172743 1796
1797 TString host(fConfig->GetDCSHost(fCurrentDetector, iServ));
1798 Int_t port = fConfig->GetDCSPort(fCurrentDetector, iServ);
542b6cc8 1799 Int_t multiSplit = fConfig->GetMultiSplit(fCurrentDetector, iServ);
1800
1790d4b7 1801 Log(fCurrentDetector, Form("ProcessCurrentDetector -"
1802 " Querying DCS Amanda server %s:%d (%d of %d)",
1803 host.Data(), port, iServ+1, nServers));
1d172743 1804
1805 TMap* aliasMap = 0;
1806 TMap* dpMap = 0;
1807
1808 if (fConfig->GetDCSAliases(fCurrentDetector, iServ)->GetEntries() > 0)
2c15234c 1809 {
1d172743 1810 aliasMap = GetValueSet(host, port,
542b6cc8 1811 fConfig->GetDCSAliases(fCurrentDetector, iServ),
1812 kAlias, multiSplit);
1d172743 1813 if (!aliasMap)
1814 {
1815 Log(fCurrentDetector,
1816 Form("ProcessCurrentDetector -"
675f64cd 1817 " Error retrieving DCS aliases from server %s."
1818 " Sending mail to DCS experts!", host.Data()));
1d172743 1819 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
675f64cd 1820
1821 if (!SendMailToDCS())
1822 Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
1823
a038aa70 1824 delete dcsMap;
1d172743 1825 return kFALSE;
1826 }
2c15234c 1827 }
a038aa70 1828
1d172743 1829 if (fConfig->GetDCSDataPoints(fCurrentDetector, iServ)->GetEntries() > 0)
a038aa70 1830 {
1d172743 1831 dpMap = GetValueSet(host, port,
542b6cc8 1832 fConfig->GetDCSDataPoints(fCurrentDetector, iServ),
1833 kDP, multiSplit);
1d172743 1834 if (!dpMap)
1835 {
1836 Log(fCurrentDetector,
1837 Form("ProcessCurrentDetector -"
675f64cd 1838 " Error retrieving DCS data points from server %s."
1839 " Sending mail to DCS experts!", host.Data()));
1d172743 1840 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
675f64cd 1841
1842 if (!SendMailToDCS())
1843 Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
1844
1d172743 1845 if (aliasMap) delete aliasMap;
1846 delete dcsMap;
1847 return kFALSE;
1848 }
a038aa70 1849 }
1d172743 1850
1851 // merge aliasMap and dpMap into dcsMap
1852 if(aliasMap) {
1853 TIter iter(aliasMap);
a038aa70 1854 TObjString* key = 0;
1855 while ((key = (TObjString*) iter.Next()))
1d172743 1856 dcsMap->Add(key, aliasMap->GetValue(key->String()));
1857
1858 aliasMap->SetOwner(kFALSE);
1859 delete aliasMap;
1860 }
1861
1862 if(dpMap) {
1863 TIter iter(dpMap);
1864 TObjString* key = 0;
1865 while ((key = (TObjString*) iter.Next()))
1866 dcsMap->Add(key, dpMap->GetValue(key->String()));
1867
1868 dpMap->SetOwner(kFALSE);
1869 delete dpMap;
a038aa70 1870 }
73abe331 1871 }
1872 }
dc25836b 1873
b1d18693 1874 // save map into file, to help debugging in case of preprocessor error
1875 TFile* f = TFile::Open("DCSMap.root","recreate");
1876 f->cd();
1877 dcsMap->Write("DCSMap", TObject::kSingleKey);
1878 f->Close();
1879 delete f;
1880
2bb7b766 1881 // DCS Archive DB processing successful. Call Preprocessor!
85a80aa9 1882 UpdateShuttleStatus(AliShuttleStatus::kPPStarted);
a7160fe9 1883
a038aa70 1884 UInt_t returnValue = aPreprocessor->Process(dcsMap);
b948db8d 1885
3301427a 1886 if (returnValue > 0) // Preprocessor error!
1887 {
c88ad5db 1888 Log(fCurrentDetector, Form("ProcessCurrentDetector - "
1889 "Preprocessor failed. Process returned %d.", returnValue));
cb343cfd 1890 UpdateShuttleStatus(AliShuttleStatus::kPPError);
a038aa70 1891 dcsMap->DeleteAll();
1892 delete dcsMap;
9827400b 1893 return kFALSE;
1894 }
1895
1896 // preprocessor ok!
1897 UpdateShuttleStatus(AliShuttleStatus::kPPDone);
1898 Log(fCurrentDetector, Form("ProcessCurrentDetector - %s preprocessor returned success",
1899 fCurrentDetector.Data()));
b948db8d 1900
a038aa70 1901 dcsMap->DeleteAll();
1902 delete dcsMap;
b948db8d 1903
9827400b 1904 return kTRUE;
2bb7b766 1905}
1906
1907//______________________________________________________________________________________________
1908Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
1909 TObjArray& entries)
1910{
9827400b 1911 // Query DAQ's Shuttle logbook and fills detector status object.
1912 // Call QueryRunParameters to query DAQ logbook for run parameters.
1913 //
2bb7b766 1914
fc5a4708 1915 entries.SetOwner(1);
1916
2bb7b766 1917 // check connection, in case connect
be48e3ea 1918 if(!Connect(3)) return kFALSE;
2bb7b766 1919
1920 TString sqlQuery;
441b0e9c 1921 sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
2bb7b766 1922
be48e3ea 1923 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1924 if (!aResult) {
1925 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
1926 return kFALSE;
1927 }
1928
fc5a4708 1929 AliDebug(2,Form("Query = %s", sqlQuery.Data()));
1930
2bb7b766 1931 if(aResult->GetRowCount() == 0) {
c88ad5db 1932 Log("SHUTTLE", "No entries in Shuttle Logbook match request");
9827400b 1933 delete aResult;
1934 return kTRUE;
2bb7b766 1935 }
1936
1937 // TODO Check field count!
db99d43e 1938 const UInt_t nCols = 23;
2bb7b766 1939 if (aResult->GetFieldCount() != (Int_t) nCols) {
c88ad5db 1940 Log("SHUTTLE", "Invalid SQL result field number!");
2bb7b766 1941 delete aResult;
1942 return kFALSE;
1943 }
1944
2bb7b766 1945 TSQLRow* aRow;
1946 while ((aRow = aResult->Next())) {
1947 TString runString(aRow->GetField(0), aRow->GetFieldLength(0));
1948 Int_t run = runString.Atoi();
1949
eba76848 1950 AliShuttleLogbookEntry *entry = QueryRunParameters(run);
1951 if (!entry)
1952 continue;
2bb7b766 1953
1954 // loop on detectors
eba76848 1955 for(UInt_t ii = 0; ii < nCols; ii++)
1956 entry->SetDetectorStatus(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1957
eba76848 1958 entries.AddLast(entry);
2bb7b766 1959 delete aRow;
1960 }
1961
2bb7b766 1962 delete aResult;
1963 return kTRUE;
1964}
1965
1966//______________________________________________________________________________________________
eba76848 1967AliShuttleLogbookEntry* AliShuttle::QueryRunParameters(Int_t run)
2bb7b766 1968{
eba76848 1969 //
1970 // Retrieve run parameters written in the DAQ logbook and sets them into AliShuttleLogbookEntry object
1971 //
2bb7b766 1972
1973 // check connection, in case connect
be48e3ea 1974 if (!Connect(3))
eba76848 1975 return 0;
2bb7b766 1976
1977 TString sqlQuery;
2c15234c 1978 sqlQuery.Form("select * from %s where run=%d", fConfig->GetDAQlbTable(), run);
2bb7b766 1979
be48e3ea 1980 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1981 if (!aResult) {
c88ad5db 1982 Log("SHUTTLE", Form("Can't execute query <%s>!", sqlQuery.Data()));
eba76848 1983 return 0;
2bb7b766 1984 }
1985
eba76848 1986 if (aResult->GetRowCount() == 0) {
2bb7b766 1987 Log("SHUTTLE", Form("QueryRunParameters - No entry in DAQ Logbook for run %d. Skipping", run));
1988 delete aResult;
eba76848 1989 return 0;
2bb7b766 1990 }
1991
eba76848 1992 if (aResult->GetRowCount() > 1) {
c88ad5db 1993 Log("SHUTTLE", Form("QueryRunParameters - UNEXPECTED: "
1994 "more than one entry in DAQ Logbook for run %d!", run));
2bb7b766 1995 delete aResult;
eba76848 1996 return 0;
2bb7b766 1997 }
1998
eba76848 1999 TSQLRow* aRow = aResult->Next();
2000 if (!aRow)
2001 {
c88ad5db 2002 Log("SHUTTLE", Form("QueryRunParameters - Could not retrieve row for run %d. Skipping", run));
eba76848 2003 delete aResult;
2004 return 0;
2005 }
2bb7b766 2006
eba76848 2007 AliShuttleLogbookEntry* entry = new AliShuttleLogbookEntry(run);
2bb7b766 2008
eba76848 2009 for (Int_t ii = 0; ii < aResult->GetFieldCount(); ii++)
2010 entry->SetRunParameter(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 2011
eba76848 2012 UInt_t startTime = entry->GetStartTime();
2013 UInt_t endTime = entry->GetEndTime();
2014
2015 if (!startTime || !endTime || startTime > endTime) {
2016 Log("SHUTTLE",
2017 Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d",
2018 run, startTime, endTime));
2019 delete entry;
2bb7b766 2020 delete aRow;
eba76848 2021 delete aResult;
2022 return 0;
2bb7b766 2023 }
2024
eba76848 2025 delete aRow;
2bb7b766 2026 delete aResult;
eba76848 2027
2028 return entry;
2bb7b766 2029}
2030
a038aa70 2031//______________________________________________________________________________________________
2032TMap* AliShuttle::GetValueSet(const char* host, Int_t port, const TSeqCollection* entries,
542b6cc8 2033 DCSType type, Int_t multiSplit)
a038aa70 2034{
2035 // Retrieve all "entry" data points from the DCS server
2036 // host, port: TSocket connection parameters
2037 // entries: list of name of the alias or data point
2038 // type: kAlias or kDP
2039 // returns TMap of values, 0 when failure
542b6cc8 2040
2041 AliDCSClient client(host, port, fTimeout, fRetries, multiSplit);
b41b252a 2042
a038aa70 2043 TMap* result = 0;
b41b252a 2044 if (type == kAlias)
a038aa70 2045 {
b41b252a 2046 result = client.GetAliasValues(entries, GetCurrentStartTime(),
2047 GetCurrentEndTime());
2048 }
2049 else if (type == kDP)
2050 {
2051 result = client.GetDPValues(entries, GetCurrentStartTime(),
2052 GetCurrentEndTime());
2053 }
a038aa70 2054
b41b252a 2055 if (result == 0)
2056 {
2057 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get entries! Reason: %s",
1790d4b7 2058 client.GetErrorString(client.GetResultErrorCode())));
2059 if (client.GetResultErrorCode() == AliDCSClient::fgkServerError)
2060 Log(fCurrentDetector.Data(), Form("GetValueSet - Server error code: %s",
2061 client.GetServerError().Data()));
a038aa70 2062
b41b252a 2063 return 0;
a038aa70 2064 }
b41b252a 2065
a038aa70 2066 return result;
2067}
b41b252a 2068
b948db8d 2069//______________________________________________________________________________________________
57f50b3c 2070const char* AliShuttle::GetFile(Int_t system, const char* detector,
2071 const char* id, const char* source)
b948db8d 2072{
9827400b 2073 // Get calibration file from file exchange servers
2074 // First queris the FXS database for the file name, using the run, detector, id and source info
2075 // then calls RetrieveFile(filename) for actual copy to local disk
2076 // run: current run being processed (given by Logbook entry fLogbookEntry)
2077 // detector: the Preprocessor name
2078 // id: provided as a parameter by the Preprocessor
2079 // source: provided by the Preprocessor through GetFileSources function
2080
2081 // check if test mode should simulate a FXS error
2082 if (fTestMode & kErrorFXSFiles)
2083 {
2084 Log(detector, Form("GetFile - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2085 return 0;
2086 }
2087
57f50b3c 2088 // check connection, in case connect
9d733021 2089 if (!Connect(system))
eba76848 2090 {
9d733021 2091 Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
57f50b3c 2092 return 0;
2093 }
2094
2095 // Query preparation
9d733021 2096 TString sourceName(source);
d386d623 2097 Int_t nFields = 3;
2098 TString sqlQueryStart = Form("select filePath,size,fileChecksum from %s where",
2099 fConfig->GetFXSdbTable(system));
2100 TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
2101 GetCurrentRun(), detector, id);
2102
9d733021 2103 if (system == kDAQ)
2104 {
d386d623 2105 whereClause += Form(" and DAQsource=\"%s\"", source);
57f50b3c 2106 }
9d733021 2107 else if (system == kDCS)
eba76848 2108 {
9d733021 2109 sourceName="none";
57f50b3c 2110 }
9d733021 2111 else if (system == kHLT)
9e080f92 2112 {
d386d623 2113 whereClause += Form(" and DDLnumbers=\"%s\"", source);
9d733021 2114 nFields = 3;
9e080f92 2115 }
2116
9e080f92 2117 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2118
2119 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2120
2121 // Query execution
2122 TSQLResult* aResult = 0;
9d733021 2123 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
9e080f92 2124 if (!aResult) {
9d733021 2125 Log(detector, Form("GetFileName - Can't execute SQL query to %s database for: id = %s, source = %s",
2126 GetSystemName(system), id, sourceName.Data()));
9e080f92 2127 return 0;
2128 }
2129
2130 if(aResult->GetRowCount() == 0)
2131 {
2132 Log(detector,
9d733021 2133 Form("GetFileName - No entry in %s FXS db for: id = %s, source = %s",
2134 GetSystemName(system), id, sourceName.Data()));
9e080f92 2135 delete aResult;
2136 return 0;
2137 }
2bb7b766 2138
9e080f92 2139 if (aResult->GetRowCount() > 1) {
2140 Log(detector,
9d733021 2141 Form("GetFileName - More than one entry in %s FXS db for: id = %s, source = %s",
2142 GetSystemName(system), id, sourceName.Data()));
9e080f92 2143 delete aResult;
2144 return 0;
2145 }
2146
9d733021 2147 if (aResult->GetFieldCount() != nFields) {
9e080f92 2148 Log(detector,
9d733021 2149 Form("GetFileName - Wrong field count in %s FXS db for: id = %s, source = %s",
2150 GetSystemName(system), id, sourceName.Data()));
9e080f92 2151 delete aResult;
2152 return 0;
2153 }
2154
2155 TSQLRow* aRow = dynamic_cast<TSQLRow*> (aResult->Next());
2156
2157 if (!aRow){
9d733021 2158 Log(detector, Form("GetFileName - Empty set result in %s FXS db from query: id = %s, source = %s",
2159 GetSystemName(system), id, sourceName.Data()));
9e080f92 2160 delete aResult;
2161 return 0;
2162 }
2163
2164 TString filePath(aRow->GetField(0), aRow->GetFieldLength(0));
2165 TString fileSize(aRow->GetField(1), aRow->GetFieldLength(1));
d386d623 2166 TString fileChecksum(aRow->GetField(2), aRow->GetFieldLength(2));
9e080f92 2167
2168 delete aResult;
2169 delete aRow;
2170
d386d623 2171 AliDebug(2, Form("filePath = %s; size = %s, fileChecksum = %s",
2172 filePath.Data(), fileSize.Data(), fileChecksum.Data()));
9e080f92 2173
9e080f92 2174 // retrieved file is renamed to make it unique
675f64cd 2175 TString localFileName = Form("%s/%s_%d_process/%s_%s_%d_%s_%s.shuttle",
2176 GetShuttleTempDir(), detector, GetCurrentRun(),
d524ade6 2177 GetSystemName(system), detector, GetCurrentRun(),
2178 id, sourceName.Data());
9d733021 2179
9e080f92 2180
9d733021 2181 // file retrieval from FXS
4b95672b 2182 UInt_t nRetries = 0;
2183 UInt_t maxRetries = 3;
2184 Bool_t result = kFALSE;
2185
2186 // copy!! if successful TSystem::Exec returns 0
2187 while(nRetries++ < maxRetries) {
2188 AliDebug(2, Form("Trying to copy file. Retry # %d", nRetries));
2189 result = RetrieveFile(system, filePath.Data(), localFileName.Data());
2190 if(!result)
2191 {
2192 Log(detector, Form("GetFileName - Copy of file %s from %s FXS failed",
9d733021 2193 filePath.Data(), GetSystemName(system)));
4b95672b 2194 continue;
4f0749a8 2195 }
9e080f92 2196
d386d623 2197 if (fileChecksum.Length()>0)
4b95672b 2198 {
2199 // compare md5sum of local file with the one stored in the FXS DB
d524ade6 2200 Int_t md5Comp = gSystem->Exec(Form("md5sum %s |grep %s 2>&1 > /dev/null",
2201 localFileName.Data(), fileChecksum.Data()));
9e080f92 2202
4b95672b 2203 if (md5Comp != 0)
2204 {
2205 Log(detector, Form("GetFileName - md5sum of file %s does not match with local copy!",
2206 filePath.Data()));
2207 result = kFALSE;
2208 continue;
2209 }
d386d623 2210 } else {
2211 Log(fCurrentDetector, Form("GetFile - md5sum of file %s not set in %s database, skipping comparison",
2212 filePath.Data(), GetSystemName(system)));
9d733021 2213 }
4b95672b 2214 if (result) break;
9e080f92 2215 }
2216
4b95672b 2217 if(!result) return 0;
2218
9d733021 2219 fFXSCalled[system]=kTRUE;
2220 TObjString *fileParams = new TObjString(Form("%s#!?!#%s", id, sourceName.Data()));
2221 fFXSlist[system].Add(fileParams);
9e080f92 2222
675f64cd 2223 static TString staticLocalFileName;
2224 staticLocalFileName.Form("%s", localFileName.Data());
2225
c88ad5db 2226 Log(fCurrentDetector, Form("GetFile - Retrieved file with id %s and "
2227 "source %s from %s to %s", id, source,
d524ade6 2228 GetSystemName(system), localFileName.Data()));
675f64cd 2229
d524ade6 2230 return staticLocalFileName.Data();
2bb7b766 2231}
2232
2233//______________________________________________________________________________________________
9d733021 2234Bool_t AliShuttle::RetrieveFile(UInt_t system, const char* fxsFileName, const char* localFileName)
9e080f92 2235{
9827400b 2236 //
2237 // Copies file from FXS to local Shuttle machine
2238 //
2bb7b766 2239
9e080f92 2240 // check temp directory: trying to cd to temp; if it does not exist, create it
d524ade6 2241 AliDebug(2, Form("Copy file %s from %s FXS into %s",
2242 GetSystemName(system), fxsFileName, localFileName));
2243
2244 TString tmpDir(localFileName);
2245
2246 tmpDir = tmpDir(0,tmpDir.Last('/'));
9e080f92 2247
d524ade6 2248 Int_t noDir = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
2249 if (noDir) // temp dir does not exists!
2250 {
2251 if (gSystem->mkdir(tmpDir.Data(), 1))
2252 {
2253 Log(fCurrentDetector.Data(), "RetrieveFile - could not make temp directory!!");
9e080f92 2254 return kFALSE;
2255 }
9e080f92 2256 }
2257
9d733021 2258 TString baseFXSFolder;
2259 if (system == kDAQ)
2260 {
2261 baseFXSFolder = "FES/";
2262 }
2263 else if (system == kDCS)
2264 {
2265 baseFXSFolder = "";
2266 }
2267 else if (system == kHLT)
2268 {
42fde080 2269 baseFXSFolder = "/opt/FXS/";
9d733021 2270 }
2271
2272
d524ade6 2273 TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s",
9d733021 2274 fConfig->GetFXSPort(system),
2275 fConfig->GetFXSUser(system),
2276 fConfig->GetFXSHost(system),
2277 baseFXSFolder.Data(),
2278 fxsFileName,
9e080f92 2279 localFileName);
2280
2281 AliDebug(2, Form("%s",command.Data()));
2282
4b95672b 2283 Bool_t result = (gSystem->Exec(command.Data()) == 0);
9e080f92 2284
4b95672b 2285 return result;
9e080f92 2286}
2287
2288//______________________________________________________________________________________________
9d733021 2289TList* AliShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
2290{
9827400b 2291 //
2292 // Get sources producing the condition file Id from file exchange servers
4a33bdd9 2293 // if id is NULL all sources are returned (distinct)
9827400b 2294 //
1bcd28db 2295
2296 Log(detector, Form("GetFileSources - Retrieving sources with id %s from %s", id, GetSystemName(system)));
9827400b 2297
2298 // check if test mode should simulate a FXS error
2299 if (fTestMode & kErrorFXSSources)
2300 {
2301 Log(detector, Form("GetFileSources - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2302 return 0;
2303 }
2304
9d733021 2305 if (system == kDCS)
2306 {
c88ad5db 2307 Log(detector, "GetFileSources - WARNING: DCS system has only one source of data!");
6297b37d 2308 TList *list = new TList();
2309 list->SetOwner(1);
2310 list->Add(new TObjString(" "));
2311 return list;
9d733021 2312 }
9e080f92 2313
2314 // check connection, in case connect
9d733021 2315 if (!Connect(system))
2316 {
4a33bdd9 2317 Log(detector, Form("GetFileSources - Couldn't connect to %s FXS database", GetSystemName(system)));
9d733021 2318 return NULL;
9e080f92 2319 }
2320
9d733021 2321 TString sourceName = 0;
2322 if (system == kDAQ)
2323 {
2324 sourceName = "DAQsource";
2325 } else if (system == kHLT)
2326 {
2327 sourceName = "DDLnumbers";
2328 }
2329
4a33bdd9 2330 TString sqlQueryStart = Form("select distinct %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
2331 TString whereClause = Form("run=%d and detector=\"%s\"",
2332 GetCurrentRun(), detector);
2333 if (id)
2334 whereClause += Form(" and fileId=\"%s\"", id);
9e080f92 2335 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2336
2337 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2338
2339 // Query execution
2340 TSQLResult* aResult;
9d733021 2341 aResult = fServer[system]->Query(sqlQuery);
9e080f92 2342 if (!aResult) {
9d733021 2343 Log(detector, Form("GetFileSources - Can't execute SQL query to %s database for id: %s",
2344 GetSystemName(system), id));
9e080f92 2345 return 0;
2346 }
2347
86aa42c3 2348 TList *list = new TList();
2349 list->SetOwner(1);
2350
9d733021 2351 if (aResult->GetRowCount() == 0)
2352 {
9e080f92 2353 Log(detector,
9d733021 2354 Form("GetFileSources - No entry in %s FXS table for id: %s", GetSystemName(system), id));
9e080f92 2355 delete aResult;
86aa42c3 2356 return list;
9e080f92 2357 }
2358
1bcd28db 2359 Log(detector, Form("GetFileSources - Found %d sources", aResult->GetRowCount()));
9e080f92 2360
1bcd28db 2361 TSQLRow* aRow;
9d733021 2362 while ((aRow = aResult->Next()))
2363 {
9e080f92 2364
9d733021 2365 TString source(aRow->GetField(0), aRow->GetFieldLength(0));
2366 AliDebug(2, Form("%s = %s", sourceName.Data(), source.Data()));
2367 list->Add(new TObjString(source));
9e080f92 2368 delete aRow;
2369 }
9d733021 2370
9e080f92 2371 delete aResult;
2372
2373 return list;
2bb7b766 2374}
2375
4a33bdd9 2376//______________________________________________________________________________________________
2377TList* AliShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
2378{
2379 //
2380 // Get all ids of condition files produced by a given source from file exchange servers
2381 //
2382
1bcd28db 2383 Log(detector, Form("GetFileIDs - Retrieving ids with source %s with %s", source, GetSystemName(system)));
2384
4a33bdd9 2385 // check if test mode should simulate a FXS error
2386 if (fTestMode & kErrorFXSSources)
2387 {
2388 Log(detector, Form("GetFileIDs - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2389 return 0;
2390 }
2391
2392 // check connection, in case connect
2393 if (!Connect(system))
2394 {
2395 Log(detector, Form("GetFileIDs - Couldn't connect to %s FXS database", GetSystemName(system)));
2396 return NULL;
2397 }
2398
2399 TString sourceName = 0;
2400 if (system == kDAQ)
2401 {
2402 sourceName = "DAQsource";
2403 } else if (system == kHLT)
2404 {
2405 sourceName = "DDLnumbers";
2406 }
2407
2408 TString sqlQueryStart = Form("select fileId from %s where", fConfig->GetFXSdbTable(system));
2409 TString whereClause = Form("run=%d and detector=\"%s\"",
2410 GetCurrentRun(), detector);
2411 if (sourceName.Length() > 0 && source)
2412 whereClause += Form(" and %s=\"%s\"", sourceName.Data(), source);
2413 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2414
2415 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2416
2417 // Query execution
2418 TSQLResult* aResult;
2419 aResult = fServer[system]->Query(sqlQuery);
2420 if (!aResult) {
2421 Log(detector, Form("GetFileIDs - Can't execute SQL query to %s database for source: %s",
2422 GetSystemName(system), source));
2423 return 0;
2424 }
2425
2426 TList *list = new TList();
2427 list->SetOwner(1);
2428
2429 if (aResult->GetRowCount() == 0)
2430 {
2431 Log(detector,
2432 Form("GetFileIDs - No entry in %s FXS table for source: %s", GetSystemName(system), source));
2433 delete aResult;
2434 return list;
2435 }
2436
1bcd28db 2437 Log(detector, Form("GetFileIDs - Found %d ids", aResult->GetRowCount()));
2438
4a33bdd9 2439 TSQLRow* aRow;
2440
2441 while ((aRow = aResult->Next()))
2442 {
2443
2444 TString id(aRow->GetField(0), aRow->GetFieldLength(0));
2445 AliDebug(2, Form("fileId = %s", id.Data()));
2446 list->Add(new TObjString(id));
2447 delete aRow;
2448 }
2449
2450 delete aResult;
2451
2452 return list;
2453}
2454
2bb7b766 2455//______________________________________________________________________________________________
9d733021 2456Bool_t AliShuttle::Connect(Int_t system)
2bb7b766 2457{
9827400b 2458 // Connect to MySQL Server of the system's FXS MySQL databases
2459 // DAQ Logbook, Shuttle Logbook and DAQ FXS db are on the same host
2460 //
57f50b3c 2461
9d733021 2462 // check connection: if already connected return
2463 if(fServer[system] && fServer[system]->IsConnected()) return kTRUE;
57f50b3c 2464
9d733021 2465 TString dbHost, dbUser, dbPass, dbName;
57f50b3c 2466
9d733021 2467 if (system < 3) // FXS db servers
2468 {
2469 dbHost = Form("mysql://%s:%d", fConfig->GetFXSdbHost(system), fConfig->GetFXSdbPort(system));
2470 dbUser = fConfig->GetFXSdbUser(system);
2471 dbPass = fConfig->GetFXSdbPass(system);
2472 dbName = fConfig->GetFXSdbName(system);
2473 } else { // Run & Shuttle logbook servers
2474 // TODO Will the Shuttle logbook server be the same as the Run logbook server ???
2475 dbHost = Form("mysql://%s:%d", fConfig->GetDAQlbHost(), fConfig->GetDAQlbPort());
2476 dbUser = fConfig->GetDAQlbUser();
2477 dbPass = fConfig->GetDAQlbPass();
2478 dbName = fConfig->GetDAQlbDB();
2479 }
57f50b3c 2480
9d733021 2481 fServer[system] = TSQLServer::Connect(dbHost.Data(), dbUser.Data(), dbPass.Data());
2482 if (!fServer[system] || !fServer[system]->IsConnected()) {
2483 if(system < 3)
2484 {
2485 AliError(Form("Can't establish connection to FXS database for %s",
2486 AliShuttleInterface::GetSystemName(system)));
2487 } else {
2488 AliError("Can't establish connection to Run logbook.");
57f50b3c 2489 }
9d733021 2490 if(fServer[system]) delete fServer[system];
2491 return kFALSE;
2bb7b766 2492 }
57f50b3c 2493
9d733021 2494 // Get tables
2495 TSQLResult* aResult=0;
2496 switch(system){
2497 case kDAQ:
2498 aResult = fServer[kDAQ]->GetTables(dbName.Data());
2499 break;
2500 case kDCS:
2501 aResult = fServer[kDCS]->GetTables(dbName.Data());
2502 break;
2503 case kHLT:
2504 aResult = fServer[kHLT]->GetTables(dbName.Data());
2505 break;
2506 default:
2507 aResult = fServer[3]->GetTables(dbName.Data());
2508 break;
2509 }
2510
2511 delete aResult;
2bb7b766 2512 return kTRUE;
2513}
57f50b3c 2514
9e080f92 2515//______________________________________________________________________________________________
9d733021 2516Bool_t AliShuttle::UpdateTable()
9e080f92 2517{
9827400b 2518 //
2519 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2520 //
9e080f92 2521
9d733021 2522 Bool_t result = kTRUE;
9e080f92 2523
9d733021 2524 for (UInt_t system=0; system<3; system++)
2525 {
2526 if(!fFXSCalled[system]) continue;
9e080f92 2527
9d733021 2528 // check connection, in case connect
2529 if (!Connect(system))
2530 {
2531 Log(fCurrentDetector, Form("UpdateTable - Couldn't connect to %s FXS database", GetSystemName(system)));
2532 result = kFALSE;
2533 continue;
9e080f92 2534 }
9e080f92 2535
9d733021 2536 TTimeStamp now; // now
2537
2538 // Loop on FXS list entries
2539 TIter iter(&fFXSlist[system]);
2540 TObjString *aFXSentry=0;
2541 while ((aFXSentry = dynamic_cast<TObjString*> (iter.Next())))
2542 {
2543 TString aFXSentrystr = aFXSentry->String();
2544 TObjArray *aFXSarray = aFXSentrystr.Tokenize("#!?!#");
2545 if (!aFXSarray || aFXSarray->GetEntries() != 2 )
2546 {
2547 Log(fCurrentDetector, Form("UpdateTable - error updating %s FXS entry. Check string: <%s>",
2548 GetSystemName(system), aFXSentrystr.Data()));
2549 if(aFXSarray) delete aFXSarray;
2550 result = kFALSE;
2551 continue;
2552 }
2553 const char* fileId = ((TObjString*) aFXSarray->At(0))->GetName();
2554 const char* source = ((TObjString*) aFXSarray->At(1))->GetName();
2555
2556 TString whereClause;
2557 if (system == kDAQ)
2558 {
2559 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DAQsource=\"%s\";",
2560 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2561 }
2562 else if (system == kDCS)
2563 {
2564 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\";",
2565 GetCurrentRun(), fCurrentDetector.Data(), fileId);
2566 }
2567 else if (system == kHLT)
2568 {
2569 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DDLnumbers=\"%s\";",
2570 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2571 }
2572
2573 delete aFXSarray;
9e080f92 2574
9d733021 2575 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2576 now.GetSec(), whereClause.Data());
9e080f92 2577
9d733021 2578 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
9e080f92 2579
9d733021 2580 // Query execution
2581 TSQLResult* aResult;
2582 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2583 if (!aResult)
2584 {
2585 Log(fCurrentDetector, Form("UpdateTable - %s db: can't execute SQL query <%s>",
2586 GetSystemName(system), sqlQuery.Data()));
2587 result = kFALSE;
2588 continue;
2589 }
2590 delete aResult;
9e080f92 2591 }
9e080f92 2592 }
2593
9d733021 2594 return result;
9e080f92 2595}
57f50b3c 2596
3301427a 2597//______________________________________________________________________________________________
2598Bool_t AliShuttle::UpdateTableFailCase()
2599{
9827400b 2600 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2601 // this is called in case the preprocessor is declared failed for the current run, because
2602 // the fields are updated only in case of success
3301427a 2603
2604 Bool_t result = kTRUE;
2605
2606 for (UInt_t system=0; system<3; system++)
2607 {
2608 // check connection, in case connect
2609 if (!Connect(system))
2610 {
2611 Log(fCurrentDetector, Form("UpdateTableFailCase - Couldn't connect to %s FXS database",
2612 GetSystemName(system)));
2613 result = kFALSE;
2614 continue;
2615 }
2616
2617 TTimeStamp now; // now
2618
2619 // Loop on FXS list entries
2620
2621 TString whereClause = Form("where run=%d and detector=\"%s\";",
2622 GetCurrentRun(), fCurrentDetector.Data());
2623
2624
2625 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2626 now.GetSec(), whereClause.Data());
2627
2628 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2629
2630 // Query execution
2631 TSQLResult* aResult;
2632 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2633 if (!aResult)
2634 {
2635 Log(fCurrentDetector, Form("UpdateTableFailCase - %s db: can't execute SQL query <%s>",
2636 GetSystemName(system), sqlQuery.Data()));
2637 result = kFALSE;
2638 continue;
2639 }
2640 delete aResult;
2641 }
2642
2643 return result;
2644}
2645
2bb7b766 2646//______________________________________________________________________________________________
2647Bool_t AliShuttle::UpdateShuttleLogbook(const char* detector, const char* status)
2648{
e7f62f16 2649 //
2650 // Update Shuttle logbook filling detector or shuttle_done column
2651 // ex. of usage: UpdateShuttleLogbook("PHOS", "DONE") or UpdateShuttleLogbook("shuttle_done")
2652 //
57f50b3c 2653
2bb7b766 2654 // check connection, in case connect
be48e3ea 2655 if(!Connect(3)){
2bb7b766 2656 Log("SHUTTLE", "UpdateShuttleLogbook - Couldn't connect to DAQ Logbook.");
2657 return kFALSE;
57f50b3c 2658 }
2659
2bb7b766 2660 TString detName(detector);
2661 TString setClause;
e7f62f16 2662 if(detName == "shuttle_done")
2663 {
2bb7b766 2664 setClause = "set shuttle_done=1";
e7f62f16 2665
2666 // Send the information to ML
2667 TMonaLisaText mlStatus("SHUTTLE_status", "Done");
2668
2669 TList mlList;
2670 mlList.Add(&mlStatus);
2671
2672 fMonaLisa->SendParameters(&mlList);
2bb7b766 2673 } else {
2bb7b766 2674 TString statusStr(status);
2675 if(statusStr.Contains("done", TString::kIgnoreCase) ||
2676 statusStr.Contains("failed", TString::kIgnoreCase)){
eba76848 2677 setClause = Form("set %s=\"%s\"", detector, status);
2bb7b766 2678 } else {
2679 Log("SHUTTLE",
2680 Form("UpdateShuttleLogbook - Invalid status <%s> for detector %s",
2681 status, detector));
2682 return kFALSE;
2683 }
2684 }
57f50b3c 2685
2bb7b766 2686 TString whereClause = Form("where run=%d", GetCurrentRun());
2687
441b0e9c 2688 TString sqlQuery = Form("update %s %s %s",
2689 fConfig->GetShuttlelbTable(), setClause.Data(), whereClause.Data());
57f50b3c 2690
2bb7b766 2691 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2692
2693 // Query execution
2694 TSQLResult* aResult;
be48e3ea 2695 aResult = dynamic_cast<TSQLResult*> (fServer[3]->Query(sqlQuery));
2bb7b766 2696 if (!aResult) {
2697 Log("SHUTTLE", Form("UpdateShuttleLogbook - Can't execute query <%s>", sqlQuery.Data()));
2698 return kFALSE;
57f50b3c 2699 }
2bb7b766 2700 delete aResult;
57f50b3c 2701
2702 return kTRUE;
2703}
2704
2705//______________________________________________________________________________________________
2bb7b766 2706Int_t AliShuttle::GetCurrentRun() const
2707{
9827400b 2708 //
2709 // Get current run from logbook entry
2710 //
57f50b3c 2711
2bb7b766 2712 return fLogbookEntry ? fLogbookEntry->GetRun() : -1;
57f50b3c 2713}
2714
2715//______________________________________________________________________________________________
2bb7b766 2716UInt_t AliShuttle::GetCurrentStartTime() const
2717{
9827400b 2718 //
2719 // get current start time
2720 //
57f50b3c 2721
2bb7b766 2722 return fLogbookEntry ? fLogbookEntry->GetStartTime() : 0;
57f50b3c 2723}
2724
2725//______________________________________________________________________________________________
2bb7b766 2726UInt_t AliShuttle::GetCurrentEndTime() const
2727{
9827400b 2728 //
2729 // get current end time from logbook entry
2730 //
57f50b3c 2731
2bb7b766 2732 return fLogbookEntry ? fLogbookEntry->GetEndTime() : 0;
57f50b3c 2733}
2734
675f64cd 2735//______________________________________________________________________________________________
2736UInt_t AliShuttle::GetCurrentYear() const
2737{
2738 //
2739 // Get current year from logbook entry
2740 //
2741
2742 if (!fLogbookEntry) return 0;
2743
2744 TTimeStamp startTime(GetCurrentStartTime());
2745 TString year = Form("%d",startTime.GetDate());
2746 year = year(0,4);
2747
2748 return year.Atoi();
2749}
2750
2751//______________________________________________________________________________________________
2752const char* AliShuttle::GetLHCPeriod() const
2753{
2754 //
2755 // Get current LHC period from logbook entry
2756 //
2757
2758 if (!fLogbookEntry) return 0;
2759
2760 return fLogbookEntry->GetRunParameter("LHCperiod");
2761}
2762
b948db8d 2763//______________________________________________________________________________________________
2764void AliShuttle::Log(const char* detector, const char* message)
2765{
9827400b 2766 //
2767 // Fill log string with a message
2768 //
b948db8d 2769
7d4cf768 2770 TString logRunDir = GetShuttleLogDir();
2771 if (GetCurrentRun() >=0)
2772 logRunDir += Form("/%d", GetCurrentRun());
2773
2774 void* dir = gSystem->OpenDirectory(logRunDir.Data());
84090f85 2775 if (dir == NULL) {
7d4cf768 2776 if (gSystem->mkdir(logRunDir.Data(), kTRUE)) {
36c99a6a 2777 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
84090f85 2778 return;
2779 }
b948db8d 2780
84090f85 2781 } else {
2782 gSystem->FreeDirectory(dir);
2783 }
b948db8d 2784
cb343cfd 2785 TString toLog = Form("%s (%d): %s - ", TTimeStamp(time(0)).AsString("s"), getpid(), detector);
e7f62f16 2786 if (GetCurrentRun() >= 0)
2787 toLog += Form("run %d - ", GetCurrentRun());
2bb7b766 2788 toLog += Form("%s", message);
2789
84090f85 2790 AliInfo(toLog.Data());
ffa29e93 2791
2792 // if we redirect the log output already to the file, leave here
2793 if (fOutputRedirected && strcmp(detector, "SHUTTLE") != 0)
2794 return;
b948db8d 2795
ffa29e93 2796 TString fileName = GetLogFileName(detector);
e7f62f16 2797
84090f85 2798 gSystem->ExpandPathName(fileName);
2799
2800 ofstream logFile;
2801 logFile.open(fileName, ofstream::out | ofstream::app);
2802
2803 if (!logFile.is_open()) {
2804 AliError(Form("Could not open file %s", fileName.Data()));
2805 return;
2806 }
7bfb2090 2807
84090f85 2808 logFile << toLog.Data() << "\n";
b948db8d 2809
84090f85 2810 logFile.close();
b948db8d 2811}
2bb7b766 2812
ffa29e93 2813//______________________________________________________________________________________________
2814TString AliShuttle::GetLogFileName(const char* detector) const
2815{
2816 //
2817 // returns the name of the log file for a given sub detector
2818 //
2819
2820 TString fileName;
2821
2822 if (GetCurrentRun() >= 0)
7d4cf768 2823 {
2824 fileName.Form("%s/%d/%s_%d.log", GetShuttleLogDir(), GetCurrentRun(),
2825 detector, GetCurrentRun());
2826 } else {
ffa29e93 2827 fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
7d4cf768 2828 }
ffa29e93 2829
2830 return fileName;
2831}
2832
2bb7b766 2833//______________________________________________________________________________________________
2834Bool_t AliShuttle::Collect(Int_t run)
2835{
9827400b 2836 //
2837 // Collects conditions data for all UNPROCESSED run written to DAQ LogBook in case of run = -1 (default)
2838 // If a dedicated run is given this run is processed
2839 //
2840 // In operational mode, this is the Shuttle function triggered by the EOR signal.
2841 //
2bb7b766 2842
eba76848 2843 if (run == -1)
2844 Log("SHUTTLE","Collect - Shuttle called. Collecting conditions data for unprocessed runs");
2845 else
2846 Log("SHUTTLE", Form("Collect - Shuttle called. Collecting conditions data for run %d", run));
cb343cfd 2847
2848 SetLastAction("Starting");
2bb7b766 2849
2850 TString whereClause("where shuttle_done=0");
eba76848 2851 if (run != -1)
2852 whereClause += Form(" and run=%d", run);
2bb7b766 2853
2854 TObjArray shuttleLogbookEntries;
be48e3ea 2855 if (!QueryShuttleLogbook(whereClause, shuttleLogbookEntries))
2856 {
cb343cfd 2857 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2bb7b766 2858 return kFALSE;
2859 }
2860
9e080f92 2861 if (shuttleLogbookEntries.GetEntries() == 0)
2862 {
2863 if (run == -1)
2864 Log("SHUTTLE","Collect - Found no UNPROCESSED runs in Shuttle logbook");
2865 else
2866 Log("SHUTTLE", Form("Collect - Run %d is already DONE "
2867 "or it does not exist in Shuttle logbook", run));
2868 return kTRUE;
2869 }
2870
be48e3ea 2871 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2872 fFirstUnprocessed[iDet] = kTRUE;
2873
fc5a4708 2874 if (run != -1)
be48e3ea 2875 {
2876 // query Shuttle logbook for earlier runs, check if some detectors are unprocessed,
2877 // flag them into fFirstUnprocessed array
2878 TString whereClause(Form("where shuttle_done=0 and run < %d", run));
2879 TObjArray tmpLogbookEntries;
2880 if (!QueryShuttleLogbook(whereClause, tmpLogbookEntries))
2881 {
2882 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2883 return kFALSE;
2884 }
2885
2886 TIter iter(&tmpLogbookEntries);
2887 AliShuttleLogbookEntry* anEntry = 0;
2888 while ((anEntry = dynamic_cast<AliShuttleLogbookEntry*> (iter.Next())))
2889 {
2890 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2891 {
2892 if (anEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
2893 {
2894 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
2895 anEntry->GetRun(), GetDetName(iDet)));
2896 fFirstUnprocessed[iDet] = kFALSE;
2897 }
2898 }
2899
2900 }
2901
2902 }
2903
2904 if (!RetrieveConditionsData(shuttleLogbookEntries))
2905 {
cb343cfd 2906 Log("SHUTTLE", "Collect - Process of at least one run failed");
2bb7b766 2907 return kFALSE;
2908 }
2909
36c99a6a 2910 Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
eba76848 2911 return kTRUE;
2bb7b766 2912}
2913
2bb7b766 2914//______________________________________________________________________________________________
2915Bool_t AliShuttle::RetrieveConditionsData(const TObjArray& dateEntries)
2916{
9827400b 2917 //
2918 // Retrieve conditions data for all runs that aren't processed yet
2919 //
2bb7b766 2920
2921 Bool_t hasError = kFALSE;
2922
2923 TIter iter(&dateEntries);
2924 AliShuttleLogbookEntry* anEntry;
2925
2926 while ((anEntry = (AliShuttleLogbookEntry*) iter.Next())){
2927 if (!Process(anEntry)){
2928 hasError = kTRUE;
2929 }
4b95672b 2930
2931 // clean SHUTTLE temp directory
d524ade6 2932 //TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
2933 //RemoveFile(filename.Data());
2bb7b766 2934 }
2935
2936 return hasError == kFALSE;
2937}
cb343cfd 2938
2939//______________________________________________________________________________________________
2940ULong_t AliShuttle::GetTimeOfLastAction() const
2941{
9827400b 2942 //
2943 // Gets time of last action
2944 //
2945
cb343cfd 2946 ULong_t tmp;
36c99a6a 2947
cb343cfd 2948 fMonitoringMutex->Lock();
be48e3ea 2949
cb343cfd 2950 tmp = fLastActionTime;
36c99a6a 2951
cb343cfd 2952 fMonitoringMutex->UnLock();
36c99a6a 2953
cb343cfd 2954 return tmp;
2955}
2956
2957//______________________________________________________________________________________________
2958const TString AliShuttle::GetLastAction() const
2959{
9827400b 2960 //
cb343cfd 2961 // returns a string description of the last action
9827400b 2962 //
cb343cfd 2963
2964 TString tmp;
36c99a6a 2965
cb343cfd 2966 fMonitoringMutex->Lock();
2967
2968 tmp = fLastAction;
2969
2970 fMonitoringMutex->UnLock();
2971
36c99a6a 2972 return tmp;
cb343cfd 2973}
2974
2975//______________________________________________________________________________________________
2976void AliShuttle::SetLastAction(const char* action)
2977{
9827400b 2978 //
cb343cfd 2979 // updates the monitoring variables
9827400b 2980 //
36c99a6a 2981
cb343cfd 2982 fMonitoringMutex->Lock();
36c99a6a 2983
cb343cfd 2984 fLastAction = action;
2985 fLastActionTime = time(0);
2986
2987 fMonitoringMutex->UnLock();
2988}
eba76848 2989
2990//______________________________________________________________________________________________
2991const char* AliShuttle::GetRunParameter(const char* param)
2992{
9827400b 2993 //
2994 // returns run parameter read from DAQ logbook
2995 //
eba76848 2996
2997 if(!fLogbookEntry) {
2998 AliError("No logbook entry!");
2999 return 0;
3000 }
3001
3002 return fLogbookEntry->GetRunParameter(param);
3003}
57c1a579 3004
d386d623 3005//______________________________________________________________________________________________
9827400b 3006AliCDBEntry* AliShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
d386d623 3007{
9827400b 3008 //
3009 // returns object from OCDB valid for current run
3010 //
d386d623 3011
9827400b 3012 if (fTestMode & kErrorOCDB)
3013 {
3014 Log(detector, "GetFromOCDB - In TESTMODE - Simulating error with OCDB");
3015 return 0;
3016 }
3017
d386d623 3018 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
3019 if (!sto)
3020 {
9827400b 3021 Log(detector, "GetFromOCDB - Cannot activate main OCDB for query!");
d386d623 3022 return 0;
3023 }
3024
3025 return dynamic_cast<AliCDBEntry*> (sto->Get(path, GetCurrentRun()));
3026}
3027
57c1a579 3028//______________________________________________________________________________________________
3029Bool_t AliShuttle::SendMail()
3030{
9827400b 3031 //
3032 // sends a mail to the subdetector expert in case of preprocessor error
3033 //
3034
3035 if (fTestMode != kNone)
3036 return kTRUE;
57c1a579 3037
36c99a6a 3038 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
57c1a579 3039 if (dir == NULL)
3040 {
36c99a6a 3041 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
57c1a579 3042 {
675f64cd 3043 Log("SHUTTLE", Form("SendMail - Can't open directory <%s>", GetShuttleLogDir()));
57c1a579 3044 return kFALSE;
3045 }
3046
3047 } else {
3048 gSystem->FreeDirectory(dir);
3049 }
3050
3051 TString bodyFileName;
36c99a6a 3052 bodyFileName.Form("%s/mail.body", GetShuttleLogDir());
57c1a579 3053 gSystem->ExpandPathName(bodyFileName);
3054
3055 ofstream mailBody;
3056 mailBody.open(bodyFileName, ofstream::out);
3057
3058 if (!mailBody.is_open())
3059 {
675f64cd 3060 Log("SHUTTLE", Form("Could not open mail body file %s", bodyFileName.Data()));
57c1a579 3061 return kFALSE;
3062 }
3063
3064 TString to="";
3065 TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
3066 TObjString *anExpert=0;
3067 while ((anExpert = (TObjString*) iterExperts.Next()))
3068 {
3069 to += Form("%s,", anExpert->GetName());
3070 }
3071 to.Remove(to.Length()-1);
909732f7 3072 AliDebug(2, Form("to: %s",to.Data()));
57c1a579 3073
86aa42c3 3074 if (to.IsNull()) {
c88ad5db 3075 Log("SHUTTLE", "List of detector responsibles not yet set!");
36c99a6a 3076 return kFALSE;
3077 }
3078
57c1a579 3079 TString cc="alberto.colla@cern.ch";
3080
546242fb 3081 TString subject = Form("%s Shuttle preprocessor FAILED in run %d !",
57c1a579 3082 fCurrentDetector.Data(), GetCurrentRun());
909732f7 3083 AliDebug(2, Form("subject: %s", subject.Data()));
57c1a579 3084
3085 TString body = Form("Dear %s expert(s), \n\n", fCurrentDetector.Data());
3086 body += Form("SHUTTLE just detected that your preprocessor "
546242fb 3087 "failed processing run %d!!\n\n", GetCurrentRun());
7d4cf768 3088 body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n",
3089 fCurrentDetector.Data());
546242fb 3090 body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
7d4cf768 3091
3092 TString logFolder = "logs";
3093 if (fConfig->GetRunMode() == AliShuttleConfig::kProd)
3094 logFolder += "_PROD";
3095
3096
546242fb 3097 body += Form("Find the %s log for the current run on \n\n"
7d4cf768 3098 "\thttp://pcalishuttle01.cern.ch:8880/%s/%d/%s_%d.log \n\n",
3099 fCurrentDetector.Data(), logFolder.Data(), GetCurrentRun(),
3100 fCurrentDetector.Data(), GetCurrentRun());
3101 body += Form("The last 10 lines of %s log file are following:\n\n", fCurrentDetector.Data());
57c1a579 3102
909732f7 3103 AliDebug(2, Form("Body begin: %s", body.Data()));
57c1a579 3104
3105 mailBody << body.Data();
3106 mailBody.close();
3107 mailBody.open(bodyFileName, ofstream::out | ofstream::app);
3108
7d4cf768 3109 TString logFileName = Form("%s/%d/%s_%d.log", GetShuttleLogDir(),
3110 GetCurrentRun(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 3111 TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
3112 if (gSystem->Exec(tailCommand.Data()))
3113 {
3114 mailBody << Form("%s log file not found ...\n\n", fCurrentDetector.Data());
3115 }
3116
3117 TString endBody = Form("------------------------------------------------------\n\n");
675f64cd 3118 endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
3119 endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
3120 endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
3121
3122 AliDebug(2, Form("Body end: %s", endBody.Data()));
3123
3124 mailBody << endBody.Data();
3125
3126 mailBody.close();
3127
3128 // send mail!
3129 TString mailCommand = Form("mail -s \"%s\" -c %s %s < %s",
3130 subject.Data(),
3131 cc.Data(),
3132 to.Data(),
3133 bodyFileName.Data());
3134 AliDebug(2, Form("mail command: %s", mailCommand.Data()));
3135
3136 Bool_t result = gSystem->Exec(mailCommand.Data());
3137
3138 return result == 0;
3139}
3140
3141//______________________________________________________________________________________________
3142Bool_t AliShuttle::SendMailToDCS()
3143{
3144 //
3145 // sends a mail to the DCS experts in case of DCS error
3146 //
3147
3148 if (fTestMode != kNone)
3149 return kTRUE;
3150
3151 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
3152 if (dir == NULL)
3153 {
3154 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
3155 {
3156 Log("SHUTTLE", Form("SendMailToDCS - Can't open directory <%s>", GetShuttleLogDir()));
3157 return kFALSE;
3158 }
3159
3160 } else {