]> git.uio.no Git - u/mrichter/AliRoot.git/blame - SHUTTLE/AliShuttle.cxx
git://git.uio.no / u / mrichter / AliRoot.git / blame
? search:
summary | shortlog | log | commit | commitdiff | tree
blob | blame (incremental) | history | HEAD
sending shuttle_ignore to ML also in case of 0 events
[u/mrichter/AliRoot.git] / SHUTTLE / AliShuttle.cxx
CommitLineData
73abe331 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
97e3c167 18Revision 1.70 2007/12/12 13:45:35 acolla
19Monalisa started in Collect() function. Alive message to monitor is sent at each Collect and every minute during preprocessor processing.
20
ee6f7523 21Revision 1.69 2007/12/12 10:06:29 acolla
22in AliShuttle.cxx: SHUTTLE logbook is updated in case of invalid run times:
23
24time_start==0 && time_end==0
25
26logbook is NOT updated if time_start != 0 && time_end == 0, because it may mean that the run is still ongoing.
27
1abfbb60 28Revision 1.68 2007/12/11 10:15:17 acolla
29Added marking SHUTTLE=DONE for invalid runs
30(invalid start time or end time) and runs with totalEvents < 1
31
b0e53b15 32Revision 1.67 2007/12/07 19:14:36 acolla
33in AliShuttleTrigger:
34
35Added automatic collection of new runs on a regular time basis (settable from the configuration)
36
37in AliShuttleConfig: new members
38
39- triggerWait: time to wait for DIM trigger (s) before starting automatic collection of new runs
40- mode: run mode (test, prod) -> used to build log folder (logs or logs_PROD)
41
42in AliShuttle:
43
44- logs now stored in logs/#RUN/DET_#RUN.log
45
7d4cf768 46Revision 1.66 2007/12/05 10:45:19 jgrosseo
47changed order of arguments to TMonaLisaWriter
48
468df1ce 49Revision 1.65 2007/11/26 16:58:37 acolla
50Monalisa configuration added: host and table name
51
b832ec02 52Revision 1.64 2007/11/13 16:15:47 acolla
53DCS map is stored in a file in the temp folder where the detector is processed.
54If the preprocessor fails, the temp folder is not removed. This will help the debugging of the problem.
55
b1d18693 56Revision 1.63 2007/11/02 10:53:16 acolla
57Protection added to AliShuttle::CopyFileLocally
58
7d43a416 59Revision 1.62 2007/10/31 18:23:13 acolla
60Furter developement on the Shuttle:
61
62- Shuttle now connects to the Grid as alidaq. The OCDB and Reference folders
63are now built from /alice/data, e.g.:
64/alice/data/2007/LHC07a/OCDB
65
66the year and LHC period are taken from the Shuttle.
67Raw metadata files are stored by GRP to:
68/alice/data/2007/LHC07a/<runNb>/Raw/RunMetadata.root
69
70- Shuttle sends a mail to DCS experts each time DP retrieval fails.
71
675f64cd 72Revision 1.61 2007/10/30 20:33:51 acolla
73Improved managing of temporary folders, which weren't correctly handled.
74Resolved bug introduced in StoreReferenceFile, which caused SPD preprocessor fail.
75
d524ade6 76Revision 1.60 2007/10/29 18:06:16 acolla
77
78New function StoreRunMetadataFile added to preprocessor and Shuttle interface
79This function can be used by GRP only. It stores raw data tags merged file to the
80raw data folder (e.g. /alice/data/2008/LHC08a/000099999/Raw).
81
82KNOWN ISSUES:
83
841. Shuttle cannot write to /alice/data/ because it belongs to alidaq. Tag file is stored in /alice/simulation/... for the time being.
852. Due to a bug in TAlien::Mkdir, the creation of a folder in recursive mode (-p option) does not work. The problem
86has been corrected in the root package on the Shuttle machine.
87
c88ad5db 88Revision 1.59 2007/10/05 12:40:55 acolla
89
90Result error code added to AliDCSClient data members (it was "lost" with the new implementation of TMap* GetAliasValues and GetDPValues).
91
1790d4b7 92Revision 1.58 2007/09/28 15:27:40 acolla
93
94AliDCSClient "multiSplit" option added in the DCS configuration
95in AliDCSMessage: variable MAX_BODY_SIZE set to 500000
96
542b6cc8 97Revision 1.57 2007/09/27 16:53:13 acolla
98Detectors can have more than one AMANDA server. SHUTTLE queries the servers sequentially,
99merges the dcs aliases/DPs in one TMap and sends it to the preprocessor.
100
1d172743 101Revision 1.56 2007/09/14 16:46:14 jgrosseo
1021) Connect and Close are called before and after each query, so one can
103keep the same AliDCSClient object.
1042) The splitting of a query is moved to GetDPValues/GetAliasValues.
1053) Splitting interval can be specified in constructor
106
b41b252a 107Revision 1.55 2007/08/06 12:26:40 acolla
108Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
109read from the run logbook.
110
4859271b 111Revision 1.54 2007/07/12 09:51:25 jgrosseo
112removed duplicated log message in GetFile
113
4f0749a8 114Revision 1.53 2007/07/12 09:26:28 jgrosseo
115updating hlt fxs base path
116
42fde080 117Revision 1.52 2007/07/12 08:06:45 jgrosseo
118adding log messages in getfile... functions
119adding not implemented copy constructor in alishuttleconfigholder
120
1bcd28db 121Revision 1.51 2007/07/03 17:24:52 acolla
122root moved to v5-16-00. TFileMerger->Cp moved to TFile::Cp.
123
a986b218 124Revision 1.50 2007/07/02 17:19:32 acolla
125preprocessor is run in a temp directory that is removed when process is finished.
126
5bac2bde 127Revision 1.49 2007/06/29 10:45:06 acolla
128Number of columns in MySql Shuttle logbook increased by one (HLT added)
129
db99d43e 130Revision 1.48 2007/06/21 13:06:19 acolla
131GetFileSources returns dummy list with 1 source if system=DCS (better than
132returning error as it was)
133
6297b37d 134Revision 1.47 2007/06/19 17:28:56 acolla
135HLT updated; missing map bug removed.
136
dc25836b 137Revision 1.46 2007/06/09 13:01:09 jgrosseo
138Switching to retrieval of several DCS DPs at a time (multiDPrequest)
139
a038aa70 140Revision 1.45 2007/05/30 06:35:20 jgrosseo
141Adding functionality to the Shuttle/TestShuttle:
142o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
143o) Function to retrieve list of IDs for a given source (GetFileIDs)
144These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
145Example code has been added to the TestProcessor in TestShuttle
146
4a33bdd9 147Revision 1.44 2007/05/11 16:09:32 acolla
148Reference files for ITS, MUON and PHOS are now stored in OfflineDetName/OnlineDetName/run_...
149example: ITS/SPD/100_filename.root
150
2d9019b4 151Revision 1.43 2007/05/10 09:59:51 acolla
152Various bug fixes in StoreRefFilesToGrid; Cleaning of reference storage before processing detector (CleanReferenceStorage)
153
546242fb 154Revision 1.42 2007/05/03 08:01:39 jgrosseo
155typo in last commit :-(
156
8b739301 157Revision 1.41 2007/05/03 08:00:48 jgrosseo
158fixing log message when pp want to skip dcs value retrieval
159
651fdaab 160Revision 1.40 2007/04/27 07:06:48 jgrosseo
161GetFileSources returns empty list in case of no files, but successful query
162No mails sent in testmode
163
86aa42c3 164Revision 1.39 2007/04/17 12:43:57 acolla
165Correction in StoreOCDB; change of text in mail to detector expert
166
26758fce 167Revision 1.38 2007/04/12 08:26:18 jgrosseo
168updated comment
169
3c2a21c8 170Revision 1.37 2007/04/10 16:53:14 jgrosseo
171redirecting sub detector stdout, stderr to sub detector log file
172
3d8bc902 173Revision 1.35 2007/04/04 16:26:38 acolla
1741. Re-organization of function calls in TestPreprocessor to make it more meaningful.
1752. Added missing dependency in test preprocessors.
1763. in AliShuttle.cxx: processing time and memory consumption info on a single line.
177
886d60e6 178Revision 1.34 2007/04/04 10:33:36 jgrosseo
1791) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
180In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
181
1822) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
183
1843) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
185
1864) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
187
1885) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
189If you always need DCS data (like before), you do not need to implement it.
190
1916) The run type has been added to the monitoring page
192
9827400b 193Revision 1.33 2007/04/03 13:56:01 acolla
194Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
195run type.
196
3301427a 197Revision 1.32 2007/02/28 10:41:56 acolla
198Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
199AliPreprocessor::GetRunType() function.
200Added some ldap definition files.
201
d386d623 202Revision 1.30 2007/02/13 11:23:21 acolla
203Moved getters and setters of Shuttle's main OCDB/Reference, local
204OCDB/Reference, temp and log folders to AliShuttleInterface
205
9d733021 206Revision 1.27 2007/01/30 17:52:42 jgrosseo
207adding monalisa monitoring
208
e7f62f16 209Revision 1.26 2007/01/23 19:20:03 acolla
210Removed old ldif files, added TOF, MCH ldif files. Added some options in
211AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
212SetShuttleLogDir
213
36c99a6a 214Revision 1.25 2007/01/15 19:13:52 acolla
215Moved some AliInfo to AliDebug in SendMail function
216
fc5a4708 217Revision 1.21 2006/12/07 08:51:26 jgrosseo
218update (alberto):
219table, db names in ldap configuration
220added GRP preprocessor
221DCS data can also be retrieved by data point
222
2c15234c 223Revision 1.20 2006/11/16 16:16:48 jgrosseo
224introducing strict run ordering flag
225removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
226
be48e3ea 227Revision 1.19 2006/11/06 14:23:04 jgrosseo
228major update (Alberto)
229o) reading of run parameters from the logbook
230o) online offline naming conversion
231o) standalone DCSclient package
232
eba76848 233Revision 1.18 2006/10/20 15:22:59 jgrosseo
234o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
235o) Merging Collect, CollectAll, CollectNew function
236o) Removing implementation of empty copy constructors (declaration still there!)
237
cb343cfd 238Revision 1.17 2006/10/05 16:20:55 jgrosseo
239adapting to new CDB classes
240
6ec0e06c 241Revision 1.16 2006/10/05 15:46:26 jgrosseo
242applying to the new interface
243
481441a2 244Revision 1.15 2006/10/02 16:38:39 jgrosseo
245update (alberto):
246fixed memory leaks
247storing of objects that failed to be stored to the grid before
248interfacing of shuttle status table in daq system
249
2bb7b766 250Revision 1.14 2006/08/29 09:16:05 jgrosseo
251small update
252
85a80aa9 253Revision 1.13 2006/08/15 10:50:00 jgrosseo
254effc++ corrections (alberto)
255
4f0ab988 256Revision 1.12 2006/08/08 14:19:29 jgrosseo
257Update to shuttle classes (Alberto)
258
259- Possibility to set the full object's path in the Preprocessor's and
260Shuttle's Store functions
261- Possibility to extend the object's run validity in the same classes
262("startValidity" and "validityInfinite" parameters)
263- Implementation of the StoreReferenceData function to store reference
264data in a dedicated CDB storage.
265
84090f85 266Revision 1.11 2006/07/21 07:37:20 jgrosseo
267last run is stored after each run
268
7bfb2090 269Revision 1.10 2006/07/20 09:54:40 jgrosseo
270introducing status management: The processing per subdetector is divided into several steps,
271after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
272can keep track of the number of failures and skips further processing after a certain threshold is
273exceeded. These thresholds can be configured in LDAP.
274
5164a766 275Revision 1.9 2006/07/19 10:09:55 jgrosseo
276new configuration, accesst to DAQ FES (Alberto)
277
57f50b3c 278Revision 1.8 2006/07/11 12:44:36 jgrosseo
279adding parameters for extended validity range of data produced by preprocessor
280
17111222 281Revision 1.7 2006/07/10 14:37:09 jgrosseo
282small fix + todo comment
283
e090413b 284Revision 1.6 2006/07/10 13:01:41 jgrosseo
285enhanced storing of last sucessfully processed run (alberto)
286
a7160fe9 287Revision 1.5 2006/07/04 14:59:57 jgrosseo
288revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
289
45a493ce 290Revision 1.4 2006/06/12 09:11:16 jgrosseo
291coding conventions (Alberto)
292
58bc3020 293Revision 1.3 2006/06/06 14:26:40 jgrosseo
294o) removed files that were moved to STEER
295o) shuttle updated to follow the new interface (Alberto)
296
b948db8d 297Revision 1.2 2006/03/07 07:52:34 hristov
298New version (B.Yordanov)
299
d477ad88 300Revision 1.6 2005/11/19 17:19:14 byordano
301RetrieveDATEEntries and RetrieveConditionsData added
302
303Revision 1.5 2005/11/19 11:09:27 byordano
304AliShuttle declaration added
305
306Revision 1.4 2005/11/17 17:47:34 byordano
307TList changed to TObjArray
308
309Revision 1.3 2005/11/17 14:43:23 byordano
310import to local CVS
311
312Revision 1.1.1.1 2005/10/28 07:33:58 hristov
313Initial import as subdirectory in AliRoot
314
73abe331 315Revision 1.2 2005/09/13 08:41:15 byordano
316default startTime endTime added
317
318Revision 1.4 2005/08/30 09:13:02 byordano
319some docs added
320
321Revision 1.3 2005/08/29 21:15:47 byordano
322some docs added
323
324*/
325
326//
327// This class is the main manager for AliShuttle.
328// It organizes the data retrieval from DCS and call the
b948db8d 329// interface methods of AliPreprocessor.
73abe331 330// For every detector in AliShuttleConfgi (see AliShuttleConfig),
331// data for its set of aliases is retrieved. If there is registered
b948db8d 332// AliPreprocessor for this detector then it will be used
333// accroding to the schema (see AliPreprocessor).
334// If there isn't registered AliPreprocessor than the retrieved
73abe331 335// data is stored automatically to the undelying AliCDBStorage.
336// For detSpec is used the alias name.
337//
338
339#include "AliShuttle.h"
340
341#include "AliCDBManager.h"
342#include "AliCDBStorage.h"
343#include "AliCDBId.h"
84090f85 344#include "AliCDBRunRange.h"
345#include "AliCDBPath.h"
5164a766 346#include "AliCDBEntry.h"
73abe331 347#include "AliShuttleConfig.h"
eba76848 348#include "DCSClient/AliDCSClient.h"
73abe331 349#include "AliLog.h"
b948db8d 350#include "AliPreprocessor.h"
5164a766 351#include "AliShuttleStatus.h"
2bb7b766 352#include "AliShuttleLogbookEntry.h"
73abe331 353
57f50b3c 354#include <TSystem.h>
58bc3020 355#include <TObject.h>
b948db8d 356#include <TString.h>
57f50b3c 357#include <TTimeStamp.h>
73abe331 358#include <TObjString.h>
57f50b3c 359#include <TSQLServer.h>
360#include <TSQLResult.h>
361#include <TSQLRow.h>
cb343cfd 362#include <TMutex.h>
9827400b 363#include <TSystemDirectory.h>
364#include <TSystemFile.h>
a986b218 365#include <TFile.h>
9827400b 366#include <TGrid.h>
367#include <TGridResult.h>
73abe331 368
e7f62f16 369#include <TMonaLisaWriter.h>
370
5164a766 371#include <fstream>
372
cb343cfd 373#include <sys/types.h>
374#include <sys/wait.h>
375
73abe331 376ClassImp(AliShuttle)
377
b948db8d 378//______________________________________________________________________________________________
379AliShuttle::AliShuttle(const AliShuttleConfig* config,
380 UInt_t timeout, Int_t retries):
4f0ab988 381fConfig(config),
382fTimeout(timeout), fRetries(retries),
383fPreprocessorMap(),
2bb7b766 384fLogbookEntry(0),
eba76848 385fCurrentDetector(),
85a80aa9 386fStatusEntry(0),
cb343cfd 387fMonitoringMutex(0),
eba76848 388fLastActionTime(0),
e7f62f16 389fLastAction(),
9827400b 390fMonaLisa(0),
391fTestMode(kNone),
ffa29e93 392fReadTestMode(kFALSE),
393fOutputRedirected(kFALSE)
73abe331 394{
395 //
396 // config: AliShuttleConfig used
73abe331 397 // timeout: timeout used for AliDCSClient connection
398 // retries: the number of retries in case of connection error.
399 //
400
57f50b3c 401 if (!fConfig->IsValid()) AliFatal("********** !!!!! Invalid configuration !!!!! **********");
be48e3ea 402 for(int iSys=0;iSys<4;iSys++) {
57f50b3c 403 fServer[iSys]=0;
be48e3ea 404 if (iSys < 3)
2c15234c 405 fFXSlist[iSys].SetOwner(kTRUE);
57f50b3c 406 }
2bb7b766 407 fPreprocessorMap.SetOwner(kTRUE);
be48e3ea 408
409 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
410 fFirstUnprocessed[iDet] = kFALSE;
411
cb343cfd 412 fMonitoringMutex = new TMutex();
58bc3020 413}
414
b948db8d 415//______________________________________________________________________________________________
57f50b3c 416AliShuttle::~AliShuttle()
58bc3020 417{
9827400b 418 //
419 // destructor
420 //
58bc3020 421
b948db8d 422 fPreprocessorMap.DeleteAll();
be48e3ea 423 for(int iSys=0;iSys<4;iSys++)
57f50b3c 424 if(fServer[iSys]) {
425 fServer[iSys]->Close();
426 delete fServer[iSys];
eba76848 427 fServer[iSys] = 0;
57f50b3c 428 }
2bb7b766 429
430 if (fStatusEntry){
431 delete fStatusEntry;
432 fStatusEntry = 0;
433 }
cb343cfd 434
435 if (fMonitoringMutex)
436 {
437 delete fMonitoringMutex;
438 fMonitoringMutex = 0;
439 }
73abe331 440}
441
b948db8d 442//______________________________________________________________________________________________
57f50b3c 443void AliShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
58bc3020 444{
73abe331 445 //
b948db8d 446 // Registers new AliPreprocessor.
73abe331 447 // It uses GetName() for indentificator of the pre processor.
448 // The pre processor is registered it there isn't any other
449 // with the same identificator (GetName()).
450 //
451
eba76848 452 const char* detName = preprocessor->GetName();
453 if(GetDetPos(detName) < 0)
454 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
455
456 if (fPreprocessorMap.GetValue(detName)) {
457 AliWarning(Form("AliPreprocessor %s is already registered!", detName));
73abe331 458 return;
459 }
460
eba76848 461 fPreprocessorMap.Add(new TObjString(detName), preprocessor);
73abe331 462}
b948db8d 463//______________________________________________________________________________________________
3301427a 464Bool_t AliShuttle::Store(const AliCDBPath& path, TObject* object,
84090f85 465 AliCDBMetaData* metaData, Int_t validityStart, Bool_t validityInfinite)
73abe331 466{
9827400b 467 // Stores a CDB object in the storage for offline reconstruction. Objects that are not needed for
468 // offline reconstruction, but should be stored anyway (e.g. for debugging) should NOT be stored
469 // using this function. Use StoreReferenceData instead!
470 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
471 // finishes the data are transferred to the main storage (Grid).
b948db8d 472
3301427a 473 return StoreLocally(fgkLocalCDB, path, object, metaData, validityStart, validityInfinite);
84090f85 474}
475
476//______________________________________________________________________________________________
3301427a 477Bool_t AliShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 478{
9827400b 479 // Stores a CDB object in the storage for reference data. This objects will not be available during
480 // offline reconstrunction. Use this function for reference data only!
481 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
482 // finishes the data are transferred to the main storage (Grid).
85a80aa9 483
3301427a 484 return StoreLocally(fgkLocalRefStorage, path, object, metaData);
85a80aa9 485}
486
487//______________________________________________________________________________________________
3301427a 488Bool_t AliShuttle::StoreLocally(const TString& localUri,
85a80aa9 489 const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
490 Int_t validityStart, Bool_t validityInfinite)
491{
9827400b 492 // Store object temporarily in local storage. Parameters are passed by Store and StoreReferenceData functions.
493 // when the preprocessor finishes the data are transferred to the main storage (Grid).
494 // The parameters are:
495 // 1) Uri of the backup storage (Local)
496 // 2) the object's path.
497 // 3) the object to be stored
498 // 4) the metaData to be associated with the object
499 // 5) the validity start run number w.r.t. the current run,
500 // if the data is valid only for this run leave the default 0
501 // 6) specifies if the calibration data is valid for infinity (this means until updated),
502 // typical for calibration runs, the default is kFALSE
503 //
504 // returns 0 if fail, 1 otherwise
84090f85 505
9827400b 506 if (fTestMode & kErrorStorage)
507 {
508 Log(fCurrentDetector, "StoreLocally - In TESTMODE - Simulating error while storing locally");
509 return kFALSE;
510 }
511
3301427a 512 const char* cdbType = (localUri == fgkLocalCDB) ? "CDB" : "Reference";
2bb7b766 513
85a80aa9 514 Int_t firstRun = GetCurrentRun() - validityStart;
84090f85 515 if(firstRun < 0) {
9827400b 516 AliWarning("First valid run happens to be less than 0! Setting it to 0.");
84090f85 517 firstRun=0;
518 }
519
520 Int_t lastRun = -1;
521 if(validityInfinite) {
522 lastRun = AliCDBRunRange::Infinity();
523 } else {
524 lastRun = GetCurrentRun();
525 }
526
3301427a 527 // Version is set to current run, it will be used later to transfer data to Grid
528 AliCDBId id(path, firstRun, lastRun, GetCurrentRun(), -1);
2bb7b766 529
530 if(! dynamic_cast<TObjString*> (metaData->GetProperty("RunUsed(TObjString)"))){
531 TObjString runUsed = Form("%d", GetCurrentRun());
9e080f92 532 metaData->SetProperty("RunUsed(TObjString)", runUsed.Clone());
2bb7b766 533 }
84090f85 534
3301427a 535 Bool_t result = kFALSE;
84090f85 536
3301427a 537 if (!(AliCDBManager::Instance()->GetStorage(localUri))) {
538 Log("SHUTTLE", Form("StoreLocally - Cannot activate local %s storage", cdbType));
84090f85 539 } else {
3301427a 540 result = AliCDBManager::Instance()->GetStorage(localUri)
84090f85 541 ->Put(object, id, metaData);
542 }
543
544 if(!result) {
545
9827400b 546 Log(fCurrentDetector, Form("StoreLocally - Can't store object <%s>!", id.ToString().Data()));
3301427a 547 }
2bb7b766 548
3301427a 549 return result;
550}
84090f85 551
3301427a 552//______________________________________________________________________________________________
553Bool_t AliShuttle::StoreOCDB()
554{
9827400b 555 //
556 // Called when preprocessor ends successfully or when previous storage attempt failed (kStoreError status)
557 // Calls underlying StoreOCDB(const char*) function twice, for OCDB and Reference storage.
558 // Then calls StoreRefFilesToGrid to store reference files.
559 //
560
561 if (fTestMode & kErrorGrid)
562 {
563 Log("SHUTTLE", "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
564 Log(fCurrentDetector, "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
565 return kFALSE;
566 }
567
c88ad5db 568 Log("SHUTTLE","StoreOCDB - Storing OCDB data ...");
86aa42c3 569 Bool_t resultCDB = StoreOCDB(fgkMainCDB);
570
c88ad5db 571 Log("SHUTTLE","StoreOCDB - Storing reference data ...");
3301427a 572 Bool_t resultRef = StoreOCDB(fgkMainRefStorage);
9827400b 573
c88ad5db 574 Log("SHUTTLE","StoreOCDB - Storing reference files ...");
575 Bool_t resultRefFiles = CopyFilesToGrid("reference");
576
577 Bool_t resultMetadata = kTRUE;
578 if(fCurrentDetector == "GRP")
579 {
580 Log("StoreOCDB - SHUTTLE","Storing Run Metadata file ...");
581 resultMetadata = CopyFilesToGrid("metadata");
582 }
9827400b 583
c88ad5db 584 return resultCDB && resultRef && resultRefFiles && resultMetadata;
3301427a 585}
586
587//______________________________________________________________________________________________
588Bool_t AliShuttle::StoreOCDB(const TString& gridURI)
589{
590 //
591 // Called by StoreOCDB(), performs actual storage to the main OCDB and reference storages (Grid)
592 //
593
594 TObjArray* gridIds=0;
595
596 Bool_t result = kTRUE;
597
598 const char* type = 0;
599 TString localURI;
600 if(gridURI == fgkMainCDB) {
601 type = "OCDB";
602 localURI = fgkLocalCDB;
603 } else if(gridURI == fgkMainRefStorage) {
604 type = "reference";
605 localURI = fgkLocalRefStorage;
606 } else {
607 AliError(Form("Invalid storage URI: %s", gridURI.Data()));
608 return kFALSE;
609 }
610
611 AliCDBManager* man = AliCDBManager::Instance();
612
613 AliCDBStorage *gridSto = man->GetStorage(gridURI);
614 if(!gridSto) {
615 Log("SHUTTLE",
616 Form("StoreOCDB - cannot activate main %s storage", type));
617 return kFALSE;
618 }
619
620 gridIds = gridSto->GetQueryCDBList();
621
622 // get objects previously stored in local CDB
623 AliCDBStorage *localSto = man->GetStorage(localURI);
624 if(!localSto) {
625 Log("SHUTTLE",
626 Form("StoreOCDB - cannot activate local %s storage", type));
627 return kFALSE;
628 }
629 AliCDBPath aPath(GetOfflineDetName(fCurrentDetector.Data()),"*","*");
630 // Local objects were stored with current run as Grid version!
631 TList* localEntries = localSto->GetAll(aPath.GetPath(), GetCurrentRun(), GetCurrentRun());
632 localEntries->SetOwner(1);
633
634 // loop on local stored objects
635 TIter localIter(localEntries);
636 AliCDBEntry *aLocEntry = 0;
637 while((aLocEntry = dynamic_cast<AliCDBEntry*> (localIter.Next()))){
638 aLocEntry->SetOwner(1);
639 AliCDBId aLocId = aLocEntry->GetId();
640 aLocEntry->SetVersion(-1);
641 aLocEntry->SetSubVersion(-1);
642
643 // If local object is valid up to infinity we store it only if it is
644 // the first unprocessed run!
645 if (aLocId.GetLastRun() == AliCDBRunRange::Infinity() &&
646 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
647 {
648 Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
649 "there are previous unprocessed runs!",
650 fCurrentDetector.Data(), aLocId.GetPath().Data()));
651 continue;
652 }
653
654 // loop on Grid valid Id's
655 Bool_t store = kTRUE;
656 TIter gridIter(gridIds);
657 AliCDBId* aGridId = 0;
658 while((aGridId = dynamic_cast<AliCDBId*> (gridIter.Next()))){
659 if(aGridId->GetPath() != aLocId.GetPath()) continue;
660 // skip all objects valid up to infinity
661 if(aGridId->GetLastRun() == AliCDBRunRange::Infinity()) continue;
662 // if we get here, it means there's already some more recent object stored on Grid!
663 store = kFALSE;
664 break;
665 }
666
667 // If we get here, the file can be stored!
668 Bool_t storeOk = gridSto->Put(aLocEntry);
669 if(!store || storeOk){
670
671 if (!store)
672 {
673 Log(fCurrentDetector.Data(),
674 Form("StoreOCDB - A more recent object already exists in %s storage: <%s>",
675 type, aGridId->ToString().Data()));
676 } else {
677 Log("SHUTTLE",
678 Form("StoreOCDB - Object <%s> successfully put into %s storage",
679 aLocId.ToString().Data(), type));
2d9019b4 680 Log(fCurrentDetector.Data(),
681 Form("StoreOCDB - Object <%s> successfully put into %s storage",
682 aLocId.ToString().Data(), type));
3301427a 683 }
84090f85 684
3301427a 685 // removing local filename...
686 TString filename;
687 localSto->IdToFilename(aLocId, filename);
c88ad5db 688 Log("SHUTTLE", Form("StoreOCDB - Removing local file %s", filename.Data()));
3301427a 689 RemoveFile(filename.Data());
690 continue;
691 } else {
692 Log("SHUTTLE",
693 Form("StoreOCDB - Grid %s storage of object <%s> failed",
694 type, aLocId.ToString().Data()));
2d9019b4 695 Log(fCurrentDetector.Data(),
696 Form("StoreOCDB - Grid %s storage of object <%s> failed",
697 type, aLocId.ToString().Data()));
3301427a 698 result = kFALSE;
b948db8d 699 }
700 }
3301427a 701 localEntries->Clear();
2bb7b766 702
b948db8d 703 return result;
3301427a 704}
705
546242fb 706//______________________________________________________________________________________________
707Bool_t AliShuttle::CleanReferenceStorage(const char* detector)
708{
2d9019b4 709 // clears the directory used to store reference files of a given subdetector
546242fb 710
711 AliCDBManager* man = AliCDBManager::Instance();
712 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
2d9019b4 713 TString localBaseFolder = sto->GetBaseFolder();
714
715 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
716
d524ade6 717 Log("SHUTTLE", Form("CleanReferenceStorage - Cleaning %s", targetDir.Data()));
2d9019b4 718
719 TString begin;
720 begin.Form("%d_", GetCurrentRun());
721
722 TSystemDirectory* baseDir = new TSystemDirectory("/", targetDir);
723 if (!baseDir)
724 return kTRUE;
725
726 TList* dirList = baseDir->GetListOfFiles();
727 delete baseDir;
728
729 if (!dirList) return kTRUE;
730
731 if (dirList->GetEntries() < 3)
732 {
733 delete dirList;
734 return kTRUE;
735 }
736
737 Int_t nDirs = 0, nDel = 0;
738 TIter dirIter(dirList);
739 TSystemFile* entry = 0;
546242fb 740
2d9019b4 741 Bool_t success = kTRUE;
546242fb 742
2d9019b4 743 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
744 {
745 if (entry->IsDirectory())
746 continue;
747
748 TString fileName(entry->GetName());
749 if (!fileName.BeginsWith(begin))
750 continue;
751
752 nDirs++;
753
754 // delete file
755 Int_t result = gSystem->Unlink(fileName.Data());
756
757 if (result)
758 {
d524ade6 759 Log("SHUTTLE", Form("CleanReferenceStorage - Could not delete file %s!", fileName.Data()));
2d9019b4 760 success = kFALSE;
761 } else {
762 nDel++;
763 }
764 }
765
766 if(nDirs > 0)
767 Log("SHUTTLE", Form("CleanReferenceStorage - %d (over %d) reference files in folder %s were deleted.",
768 nDel, nDirs, targetDir.Data()));
769
770
771 delete dirList;
772 return success;
773
774
775
776
777
546242fb 778
779 Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
780 if (result == 0)
781 {
782 // delete directory
d524ade6 783 result = gSystem->Exec(Form("rm -rf %s", targetDir.Data()));
546242fb 784 if (result != 0)
785 {
d524ade6 786 Log("SHUTTLE", Form("CleanReferenceStorage - Could not clean directory %s", targetDir.Data()));
546242fb 787 return kFALSE;
788 }
789 }
790
791 result = gSystem->mkdir(targetDir, kTRUE);
792 if (result != 0)
793 {
c88ad5db 794 Log("SHUTTLE", Form("CleanReferenceStorage - Error creating base directory %s", targetDir.Data()));
546242fb 795 return kFALSE;
796 }
797
798 return kTRUE;
799}
800
9827400b 801//______________________________________________________________________________________________
802Bool_t AliShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
803{
804 //
3c2a21c8 805 // Stores reference file directly (without opening it). This function stores the file locally.
9827400b 806 //
3c2a21c8 807 // The file is stored under the following location:
808 // <base folder of local reference storage>/<DET>/<RUN#>_<gridFileName>
809 // where <gridFileName> is the second parameter given to the function
810 //
9827400b 811
812 if (fTestMode & kErrorStorage)
813 {
814 Log(fCurrentDetector, "StoreReferenceFile - In TESTMODE - Simulating error while storing locally");
815 return kFALSE;
816 }
817
818 AliCDBManager* man = AliCDBManager::Instance();
819 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
820
821 TString localBaseFolder = sto->GetBaseFolder();
822
d524ade6 823 TString target = GetRefFilePrefix(localBaseFolder.Data(), detector);
824 target.Append(Form("/%d_%s", GetCurrentRun(), gridFileName));
9827400b 825
d524ade6 826 return CopyFileLocally(localFile, target);
c88ad5db 827}
828
829//______________________________________________________________________________________________
830Bool_t AliShuttle::StoreRunMetadataFile(const char* localFile, const char* gridFileName)
831{
832 //
833 // Stores Run metadata file to the Grid, in the run folder
834 //
835 // Only GRP can call this function.
836
837 if (fTestMode & kErrorStorage)
838 {
839 Log(fCurrentDetector, "StoreRunMetaDataFile - In TESTMODE - Simulating error while storing locally");
840 return kFALSE;
841 }
842
843 AliCDBManager* man = AliCDBManager::Instance();
844 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
845
846 TString localBaseFolder = sto->GetBaseFolder();
847
848 // Build Run level folder
849 // folder = /alice/data/year/lhcPeriod/runNb/Raw
850
c88ad5db 851
675f64cd 852 TString lhcPeriod = GetLHCPeriod();
c88ad5db 853 if (lhcPeriod.Length() == 0)
854 {
855 Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
856 return 0;
857 }
858
675f64cd 859 TString target = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw/%s",
860 localBaseFolder.Data(), GetCurrentYear(),
d524ade6 861 lhcPeriod.Data(), GetCurrentRun(), gridFileName);
c88ad5db 862
d524ade6 863 return CopyFileLocally(localFile, target);
c88ad5db 864}
865
866//______________________________________________________________________________________________
d524ade6 867Bool_t AliShuttle::CopyFileLocally(const char* localFile, const TString& target)
c88ad5db 868{
869 //
870 // Stores file locally. Called by StoreReferenceFile and StoreRunMetadataFile
d524ade6 871 // Files are temporarily stored in the local reference storage. When the preprocessor
872 // finishes, the Shuttle calls CopyFilesToGrid to transfer the files to AliEn
873 // (in reference or run level folders)
c88ad5db 874 //
875
d524ade6 876 TString targetDir(target(0, target.Last('/')));
877
878 //try to open base dir folder, if it does not exist
2d9019b4 879 void* dir = gSystem->OpenDirectory(targetDir.Data());
880 if (dir == NULL) {
881 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
c88ad5db 882 Log("SHUTTLE", Form("StoreFileLocally - Can't open directory <%s>", targetDir.Data()));
2d9019b4 883 return kFALSE;
884 }
885
886 } else {
887 gSystem->FreeDirectory(dir);
888 }
9827400b 889
7d43a416 890 Int_t result = 0;
891
892 result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
9827400b 893 if (result)
894 {
c88ad5db 895 Log("SHUTTLE", Form("StoreFileLocally - %s does not exist", localFile));
546242fb 896 return kFALSE;
9827400b 897 }
546242fb 898
7d43a416 899 result = gSystem->GetPathInfo(target, 0, (Long64_t*) 0, 0, 0);
900 if (!result)
901 {
902 Log("SHUTTLE", Form("StoreFileLocally - target file %s already exist, removing...", target.Data()));
903 if (gSystem->Unlink(target.Data()))
904 {
905 Log("SHUTTLE", Form("StoreFileLocally - Could not remove existing target file %s!", target.Data()));
906 return kFALSE;
907 }
908 }
909
9827400b 910 result = gSystem->CopyFile(localFile, target);
911
912 if (result == 0)
913 {
c88ad5db 914 Log("SHUTTLE", Form("StoreFileLocally - File %s stored locally to %s", localFile, target.Data()));
9827400b 915 return kTRUE;
916 }
917 else
918 {
d524ade6 919 Log("SHUTTLE", Form("StoreFileLocally - Could not store file %s to %s! Error code = %d",
546242fb 920 localFile, target.Data(), result));
9827400b 921 return kFALSE;
922 }
c88ad5db 923
924
925
9827400b 926}
927
928//______________________________________________________________________________________________
c88ad5db 929Bool_t AliShuttle::CopyFilesToGrid(const char* type)
9827400b 930{
931 //
c88ad5db 932 // Transfers local files to the Grid. Local files can be reference files
933 // or run metadata file (from GRP only).
9827400b 934 //
c88ad5db 935 // According to the type (ref, metadata) the files are stored under the following location:
936 // ref --> <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
937 // metadata --> <run data folder>/<MetadataFileName>
86aa42c3 938 //
c88ad5db 939
9827400b 940 AliCDBManager* man = AliCDBManager::Instance();
941 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
942 if (!sto)
943 return kFALSE;
944 TString localBaseFolder = sto->GetBaseFolder();
9827400b 945
c88ad5db 946 TString dir;
947 TString alienDir;
9827400b 948 TString begin;
9827400b 949
c88ad5db 950 if (strcmp(type, "reference") == 0)
951 {
952 dir = GetRefFilePrefix(localBaseFolder.Data(), fCurrentDetector.Data());
953 AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
954 if (!gridSto)
955 return kFALSE;
956 TString gridBaseFolder = gridSto->GetBaseFolder();
957 alienDir = GetRefFilePrefix(gridBaseFolder.Data(), fCurrentDetector.Data());
958 begin = Form("%d_", GetCurrentRun());
959 }
960 else if (strcmp(type, "metadata") == 0)
961 {
c88ad5db 962
675f64cd 963 TString lhcPeriod = GetLHCPeriod();
c88ad5db 964
965 if (lhcPeriod.Length() == 0)
966 {
967 Log("SHUTTLE","CopyFilesToGrid - LHCPeriod not found in logbook!");
968 return 0;
969 }
970
675f64cd 971 dir = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw",
972 localBaseFolder.Data(), GetCurrentYear(),
c88ad5db 973 lhcPeriod.Data(), GetCurrentRun());
675f64cd 974 alienDir = dir(dir.Index("/alice/data/"), dir.Length());
975
c88ad5db 976 begin = "";
977 }
978 else
979 {
980 Log("SHUTTLE", "CopyFilesToGrid - Unexpected: type label must be reference or metadata!");
981 return kFALSE;
982 }
983
9827400b 984 TSystemDirectory* baseDir = new TSystemDirectory("/", dir);
3d8bc902 985 if (!baseDir)
986 return kTRUE;
987
2d9019b4 988 TList* dirList = baseDir->GetListOfFiles();
989 delete baseDir;
990
991 if (!dirList) return kTRUE;
992
993 if (dirList->GetEntries() < 3)
3d8bc902 994 {
2d9019b4 995 delete dirList;
9827400b 996 return kTRUE;
3d8bc902 997 }
2d9019b4 998
546242fb 999 if (!gGrid)
1000 {
c88ad5db 1001 Log("SHUTTLE", "CopyFilesToGrid - Connection to Grid failed: Cannot continue!");
2d9019b4 1002 delete dirList;
546242fb 1003 return kFALSE;
1004 }
1005
2d9019b4 1006 Int_t nDirs = 0, nTransfer = 0;
1007 TIter dirIter(dirList);
1008 TSystemFile* entry = 0;
1009
9827400b 1010 Bool_t success = kTRUE;
3d8bc902 1011 Bool_t first = kTRUE;
9827400b 1012
2d9019b4 1013 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
1014 {
9827400b 1015 if (entry->IsDirectory())
1016 continue;
1017
1018 TString fileName(entry->GetName());
1019 if (!fileName.BeginsWith(begin))
1020 continue;
1021
2d9019b4 1022 nDirs++;
1023
3d8bc902 1024 if (first)
1025 {
1026 first = kFALSE;
c88ad5db 1027 // check that folder exists, otherwise create it
3d8bc902 1028 TGridResult* result = gGrid->Ls(alienDir.Data(), "a");
1029
1030 if (!result)
2d9019b4 1031 {
1032 delete dirList;
3d8bc902 1033 return kFALSE;
2d9019b4 1034 }
3d8bc902 1035
546242fb 1036 if (!result->GetFileName(1)) // TODO: It looks like element 0 is always 0!!
3d8bc902 1037 {
675f64cd 1038 // TODO It does not work currently! Bug in TAliEn::Mkdir
1039 // TODO Manually fixed in local root v5-16-00
c88ad5db 1040 if (!gGrid->Mkdir(alienDir.Data(),"-p",0))
3d8bc902 1041 {
c88ad5db 1042 Log("SHUTTLE", Form("CopyFilesToGrid - Cannot create directory %s",
3d8bc902 1043 alienDir.Data()));
2d9019b4 1044 delete dirList;
3d8bc902 1045 return kFALSE;
546242fb 1046 } else {
c88ad5db 1047 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s created", alienDir.Data()));
3d8bc902 1048 }
1049
546242fb 1050 } else {
c88ad5db 1051 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s found", alienDir.Data()));
3d8bc902 1052 }
1053 }
1054
9827400b 1055 TString fullLocalPath;
1056 fullLocalPath.Form("%s/%s", dir.Data(), fileName.Data());
1057
1058 TString fullGridPath;
1059 fullGridPath.Form("alien://%s/%s", alienDir.Data(), fileName.Data());
1060
a986b218 1061 Bool_t result = TFile::Cp(fullLocalPath, fullGridPath);
9827400b 1062
1063 if (result)
1064 {
c88ad5db 1065 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s succeeded!",
1066 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1067 RemoveFile(fullLocalPath);
2d9019b4 1068 nTransfer++;
9827400b 1069 }
1070 else
1071 {
c88ad5db 1072 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s FAILED!",
1073 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1074 success = kFALSE;
1075 }
1076 }
2d9019b4 1077
c88ad5db 1078 Log("SHUTTLE", Form("CopyFilesToGrid - %d (over %d) files in folder %s copied to Grid.",
1079 nTransfer, nDirs, dir.Data()));
2d9019b4 1080
1081
1082 delete dirList;
9827400b 1083 return success;
1084}
1085
2d9019b4 1086//______________________________________________________________________________________________
1087const char* AliShuttle::GetRefFilePrefix(const char* base, const char* detector)
1088{
1089 //
1090 // Get folder name of reference files
1091 //
1092
1093 TString offDetStr(GetOfflineDetName(detector));
1094 TString dir;
1095 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
1096 {
1097 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
1098 } else {
1099 dir.Form("%s/%s", base, offDetStr.Data());
1100 }
1101
1102 return dir.Data();
1103
1104
1105}
c88ad5db 1106
3301427a 1107//______________________________________________________________________________________________
1108void AliShuttle::CleanLocalStorage(const TString& uri)
1109{
9827400b 1110 //
1111 // Called in case the preprocessor is declared failed. Remove remaining objects from the local storages.
1112 //
3301427a 1113
1114 const char* type = 0;
1115 if(uri == fgkLocalCDB) {
1116 type = "OCDB";
1117 } else if(uri == fgkLocalRefStorage) {
546242fb 1118 type = "Reference";
3301427a 1119 } else {
1120 AliError(Form("Invalid storage URI: %s", uri.Data()));
1121 return;
1122 }
1123
1124 AliCDBManager* man = AliCDBManager::Instance();
b948db8d 1125
3301427a 1126 // open local storage
1127 AliCDBStorage *localSto = man->GetStorage(uri);
1128 if(!localSto) {
1129 Log("SHUTTLE",
1130 Form("CleanLocalStorage - cannot activate local %s storage", type));
1131 return;
1132 }
1133
1134 TString filename(Form("%s/%s/*/Run*_v%d_s*.root",
546242fb 1135 localSto->GetBaseFolder().Data(), GetOfflineDetName(fCurrentDetector.Data()), GetCurrentRun()));
3301427a 1136
c88ad5db 1137 AliDebug(2, Form("filename = %s", filename.Data()));
3301427a 1138
c88ad5db 1139 Log("SHUTTLE", Form("Removing remaining local files for run %d and detector %s ...",
3301427a 1140 GetCurrentRun(), fCurrentDetector.Data()));
1141
1142 RemoveFile(filename.Data());
1143
1144}
1145
1146//______________________________________________________________________________________________
1147void AliShuttle::RemoveFile(const char* filename)
1148{
9827400b 1149 //
1150 // removes local file
1151 //
3301427a 1152
1153 TString command(Form("rm -f %s", filename));
1154
1155 Int_t result = gSystem->Exec(command.Data());
1156 if(result != 0)
1157 {
1158 Log("SHUTTLE", Form("RemoveFile - %s: Cannot remove file %s!",
1159 fCurrentDetector.Data(), filename));
1160 }
73abe331 1161}
1162
b948db8d 1163//______________________________________________________________________________________________
5164a766 1164AliShuttleStatus* AliShuttle::ReadShuttleStatus()
1165{
9827400b 1166 //
1167 // Reads the AliShuttleStatus from the CDB
1168 //
5164a766 1169
2bb7b766 1170 if (fStatusEntry){
1171 delete fStatusEntry;
1172 fStatusEntry = 0;
1173 }
5164a766 1174
10a5a932 1175 fStatusEntry = AliCDBManager::Instance()->GetStorage(GetLocalCDB())
2bb7b766 1176 ->Get(Form("/SHUTTLE/STATUS/%s", fCurrentDetector.Data()), GetCurrentRun());
5164a766 1177
2bb7b766 1178 if (!fStatusEntry) return 0;
1179 fStatusEntry->SetOwner(1);
5164a766 1180
2bb7b766 1181 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1182 if (!status) {
1183 AliError("Invalid object stored to CDB!");
1184 return 0;
1185 }
5164a766 1186
2bb7b766 1187 return status;
5164a766 1188}
1189
1190//______________________________________________________________________________________________
7bfb2090 1191Bool_t AliShuttle::WriteShuttleStatus(AliShuttleStatus* status)
5164a766 1192{
9827400b 1193 //
1194 // writes the status for one subdetector
1195 //
2bb7b766 1196
1197 if (fStatusEntry){
1198 delete fStatusEntry;
1199 fStatusEntry = 0;
1200 }
5164a766 1201
2bb7b766 1202 Int_t run = GetCurrentRun();
5164a766 1203
2bb7b766 1204 AliCDBId id(AliCDBPath("SHUTTLE", "STATUS", fCurrentDetector), run, run);
5164a766 1205
2bb7b766 1206 fStatusEntry = new AliCDBEntry(status, id, new AliCDBMetaData);
1207 fStatusEntry->SetOwner(1);
5164a766 1208
2bb7b766 1209 UInt_t result = AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
7bfb2090 1210
2bb7b766 1211 if (!result) {
3301427a 1212 Log("SHUTTLE", Form("WriteShuttleStatus - Failed for %s, run %d",
1213 fCurrentDetector.Data(), run));
2bb7b766 1214 return kFALSE;
1215 }
e7f62f16 1216
1217 SendMLInfo();
7bfb2090 1218
2bb7b766 1219 return kTRUE;
5164a766 1220}
1221
1222//______________________________________________________________________________________________
1223void AliShuttle::UpdateShuttleStatus(AliShuttleStatus::Status newStatus, Bool_t increaseCount)
1224{
9827400b 1225 //
1226 // changes the AliShuttleStatus for the given detector and run to the given status
1227 //
5164a766 1228
2bb7b766 1229 if (!fStatusEntry){
1230 AliError("UNEXPECTED: fStatusEntry empty");
1231 return;
1232 }
5164a766 1233
2bb7b766 1234 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
5164a766 1235
2bb7b766 1236 if (!status){
c88ad5db 1237 Log("SHUTTLE", "UpdateShuttleStatus - UNEXPECTED: status could not be read from current CDB entry");
2bb7b766 1238 return;
1239 }
5164a766 1240
2c15234c 1241 TString actionStr = Form("UpdateShuttleStatus - %s: Changing state from %s to %s",
eba76848 1242 fCurrentDetector.Data(),
36c99a6a 1243 status->GetStatusName(),
eba76848 1244 status->GetStatusName(newStatus));
cb343cfd 1245 Log("SHUTTLE", actionStr);
1246 SetLastAction(actionStr);
5164a766 1247
2bb7b766 1248 status->SetStatus(newStatus);
1249 if (increaseCount) status->IncreaseCount();
5164a766 1250
2bb7b766 1251 AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
e7f62f16 1252
1253 SendMLInfo();
5164a766 1254}
e7f62f16 1255
1256//______________________________________________________________________________________________
1257void AliShuttle::SendMLInfo()
1258{
1259 //
1260 // sends ML information about the current status of the current detector being processed
1261 //
1262
1263 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1264
1265 if (!status){
3301427a 1266 Log("SHUTTLE", "SendMLInfo - UNEXPECTED: status could not be read from current CDB entry");
e7f62f16 1267 return;
1268 }
1269
1270 TMonaLisaText mlStatus(Form("%s_status", fCurrentDetector.Data()), status->GetStatusName());
1271 TMonaLisaValue mlRetryCount(Form("%s_count", fCurrentDetector.Data()), status->GetCount());
1272
1273 TList mlList;
1274 mlList.Add(&mlStatus);
1275 mlList.Add(&mlRetryCount);
1276
ee6f7523 1277 TString mlID;
1278 mlID.Form("%d", GetCurrentRun());
1279 fMonaLisa->SendParameters(&mlList, mlID);
e7f62f16 1280}
1281
5164a766 1282//______________________________________________________________________________________________
1283Bool_t AliShuttle::ContinueProcessing()
1284{
9827400b 1285 // this function reads the AliShuttleStatus information from CDB and
1286 // checks if the processing should be continued
1287 // if yes it returns kTRUE and updates the AliShuttleStatus with nextStatus
2bb7b766 1288
57c1a579 1289 if (!fConfig->HostProcessDetector(fCurrentDetector)) return kFALSE;
1290
1291 AliPreprocessor* aPreprocessor =
1292 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1293 if (!aPreprocessor)
1294 {
c88ad5db 1295 Log("SHUTTLE", Form("ContinueProcessing - %s: no preprocessor registered", fCurrentDetector.Data()));
57c1a579 1296 return kFALSE;
1297 }
1298
2bb7b766 1299 AliShuttleLogbookEntry::Status entryStatus =
eba76848 1300 fLogbookEntry->GetDetectorStatus(fCurrentDetector);
2bb7b766 1301
1302 if(entryStatus != AliShuttleLogbookEntry::kUnprocessed) {
c88ad5db 1303 Log("SHUTTLE", Form("ContinueProcessing - %s is %s",
2bb7b766 1304 fCurrentDetector.Data(),
1305 fLogbookEntry->GetDetectorStatusName(entryStatus)));
1306 return kFALSE;
1307 }
1308
1309 // if we get here, according to Shuttle logbook subdetector is in UNPROCESSED state
be48e3ea 1310
1311 // check if current run is first unprocessed run for current detector
1312 if (fConfig->StrictRunOrder(fCurrentDetector) &&
1313 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
1314 {
86aa42c3 1315 if (fTestMode == kNone)
1316 {
c88ad5db 1317 Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering"
1318 " but this is not the first unprocessed run!"));
86aa42c3 1319 return kFALSE;
1320 }
1321 else
1322 {
c88ad5db 1323 Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - "
1324 "Although %s requires strict run ordering "
1325 "and this is not the first unprocessed run, "
1326 "the SHUTTLE continues"));
86aa42c3 1327 }
be48e3ea 1328 }
1329
2bb7b766 1330 AliShuttleStatus* status = ReadShuttleStatus();
1331 if (!status) {
1332 // first time
1333 Log("SHUTTLE", Form("ContinueProcessing - %s: Processing first time",
1334 fCurrentDetector.Data()));
1335 status = new AliShuttleStatus(AliShuttleStatus::kStarted);
1336 return WriteShuttleStatus(status);
1337 }
1338
1339 // The following two cases shouldn't happen if Shuttle Logbook was correctly updated.
1340 // If it happens it may mean Logbook updating failed... let's do it now!
1341 if (status->GetStatus() == AliShuttleStatus::kDone ||
1342 status->GetStatus() == AliShuttleStatus::kFailed){
1343 Log("SHUTTLE", Form("ContinueProcessing - %s is already %s. Updating Shuttle Logbook",
1344 fCurrentDetector.Data(),
1345 status->GetStatusName(status->GetStatus())));
1346 UpdateShuttleLogbook(fCurrentDetector.Data(),
1347 status->GetStatusName(status->GetStatus()));
1348 return kFALSE;
1349 }
1350
3301427a 1351 if (status->GetStatus() == AliShuttleStatus::kStoreError) {
2bb7b766 1352 Log("SHUTTLE",
c88ad5db 1353 Form("ContinueProcessing - %s: Grid storage of one or more "
1354 "objects failed. Trying again now",
2bb7b766 1355 fCurrentDetector.Data()));
9827400b 1356 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1357 if (StoreOCDB()){
c88ad5db 1358 Log("SHUTTLE", Form("ContinueProcessing - %s: all objects "
1359 "successfully stored into main storage",
3301427a 1360 fCurrentDetector.Data()));
2bb7b766 1361 } else {
1362 Log("SHUTTLE",
1363 Form("ContinueProcessing - %s: Grid storage failed again",
1364 fCurrentDetector.Data()));
9827400b 1365 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
2bb7b766 1366 }
1367 return kFALSE;
1368 }
1369
1370 // if we get here, there is a restart
57c1a579 1371 Bool_t cont = kFALSE;
2bb7b766 1372
1373 // abort conditions
cb343cfd 1374 if (status->GetCount() >= fConfig->GetMaxRetries()) {
57c1a579 1375 Log("SHUTTLE", Form("ContinueProcessing - %s failed %d times in status %s - "
1376 "Updating Shuttle Logbook", fCurrentDetector.Data(),
2bb7b766 1377 status->GetCount(), status->GetStatusName()));
1378 UpdateShuttleLogbook(fCurrentDetector.Data(), "FAILED");
e7f62f16 1379 UpdateShuttleStatus(AliShuttleStatus::kFailed);
3301427a 1380
1381 // there may still be objects in local OCDB and reference storage
1382 // and FXS databases may be not updated: do it now!
9827400b 1383
1384 // TODO Currently disabled, we want to keep files in case of failure!
1385 // CleanLocalStorage(fgkLocalCDB);
1386 // CleanLocalStorage(fgkLocalRefStorage);
1387 // UpdateTableFailCase();
1388
1389 // Send mail to detector expert!
c88ad5db 1390 Log("SHUTTLE", Form("ContinueProcessing - Sending mail to %s expert...",
1391 fCurrentDetector.Data()));
9827400b 1392 if (!SendMail())
1393 Log("SHUTTLE", Form("ContinueProcessing - Could not send mail to %s expert",
1394 fCurrentDetector.Data()));
3301427a 1395
57c1a579 1396 } else {
1397 Log("SHUTTLE", Form("ContinueProcessing - %s: restarting. "
1398 "Aborted before with %s. Retry number %d.", fCurrentDetector.Data(),
1399 status->GetStatusName(), status->GetCount()));
9827400b 1400 Bool_t increaseCount = kTRUE;
c88ad5db 1401 if (status->GetStatus() == AliShuttleStatus::kDCSError ||
1402 status->GetStatus() == AliShuttleStatus::kDCSStarted)
1403 increaseCount = kFALSE;
675f64cd 1404
9827400b 1405 UpdateShuttleStatus(AliShuttleStatus::kStarted, increaseCount);
57c1a579 1406 cont = kTRUE;
2bb7b766 1407 }
1408
57c1a579 1409 return cont;
5164a766 1410}
1411
1412//______________________________________________________________________________________________
2bb7b766 1413Bool_t AliShuttle::Process(AliShuttleLogbookEntry* entry)
58bc3020 1414{
73abe331 1415 //
b948db8d 1416 // Makes data retrieval for all detectors in the configuration.
2bb7b766 1417 // entry: Shuttle logbook entry, contains run paramenters and status of detectors
1418 // (Unprocessed, Inactive, Failed or Done).
d477ad88 1419 // Returns kFALSE in case of error occured and kTRUE otherwise
73abe331 1420 //
1421
9827400b 1422 if (!entry) return kFALSE;
2bb7b766 1423
1424 fLogbookEntry = entry;
1425
c88ad5db 1426 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^*",
9827400b 1427 GetCurrentRun()));
2bb7b766 1428
e7f62f16 1429 // Send the information to ML
1430 TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
9827400b 1431 TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
e7f62f16 1432
1433 TList mlList;
1434 mlList.Add(&mlStatus);
9827400b 1435 mlList.Add(&mlRunType);
e7f62f16 1436
ee6f7523 1437 TString mlID;
1438 mlID.Form("%d", GetCurrentRun());
1439 fMonaLisa->SendParameters(&mlList, mlID);
3301427a 1440
9827400b 1441 if (fLogbookEntry->IsDone())
1442 {
1443 Log("SHUTTLE","Process - Shuttle is already DONE. Updating logbook");
1444 UpdateShuttleLogbook("shuttle_done");
1445 fLogbookEntry = 0;
1446 return kTRUE;
1447 }
1448
1449 // read test mode if flag is set
1450 if (fReadTestMode)
1451 {
3d8bc902 1452 fTestMode = kNone;
9827400b 1453 TString logEntry(entry->GetRunParameter("log"));
1454 //printf("log entry = %s\n", logEntry.Data());
1455 TString searchStr("Testmode: ");
1456 Int_t pos = logEntry.Index(searchStr.Data());
1457 //printf("%d\n", pos);
1458 if (pos >= 0)
1459 {
1460 TSubString subStr = logEntry(pos + searchStr.Length(), logEntry.Length());
1461 //printf("%s\n", subStr.String().Data());
1462 TString newStr(subStr.Data());
1463 TObjArray* token = newStr.Tokenize(' ');
1464 if (token)
1465 {
1466 //token->Print();
1467 TObjString* tmpStr = dynamic_cast<TObjString*> (token->First());
1468 if (tmpStr)
1469 {
1470 Int_t testMode = tmpStr->String().Atoi();
1471 if (testMode > 0)
1472 {
c88ad5db 1473 Log("SHUTTLE", Form("Process - Enabling test mode %d", testMode));
9827400b 1474 SetTestMode((TestMode) testMode);
1475 }
1476 }
1477 delete token;
1478 }
1479 }
1480 }
c88ad5db 1481
eba76848 1482 fLogbookEntry->Print("all");
57f50b3c 1483
1484 // Initialization
d477ad88 1485 Bool_t hasError = kFALSE;
5164a766 1486
675f64cd 1487 // Set the CDB and Reference folders according to the year and LHC period
1488 TString lhcPeriod(GetLHCPeriod());
1489 if (lhcPeriod.Length() == 0)
1490 {
7d4cf768 1491 Log("SHUTTLE","Process - LHCPeriod not found in logbook!");
1492 return 0;
675f64cd 1493 }
1494
1495 if (fgkMainCDB.Length() == 0)
1496 fgkMainCDB = Form("alien://folder=/alice/data/%d/%s/OCDB?user=alidaq?cacheFold=/tmp/OCDBCache",
1497 GetCurrentYear(), lhcPeriod.Data());
1498
1499 if (fgkMainRefStorage.Length() == 0)
1500 fgkMainRefStorage = Form("alien://folder=/alice/data/%d/%s/Reference?user=alidaq?cacheFold=/tmp/OCDBCache",
1501 GetCurrentYear(), lhcPeriod.Data());
1502
2bb7b766 1503 AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
1504 if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
1505 AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
1506 if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
d477ad88 1507
57f50b3c 1508 // Loop on detectors in the configuration
b948db8d 1509 TIter iter(fConfig->GetDetectors());
2bb7b766 1510 TObjString* aDetector = 0;
b948db8d 1511
be48e3ea 1512 while ((aDetector = (TObjString*) iter.Next()))
1513 {
7bfb2090 1514 fCurrentDetector = aDetector->String();
5164a766 1515
9e080f92 1516 if (ContinueProcessing() == kFALSE) continue;
1517
c88ad5db 1518 Log("SHUTTLE", Form("\t\t\t****** run %d - %s: START ******",
2bb7b766 1519 GetCurrentRun(), aDetector->GetName()));
1520
9d733021 1521 for(Int_t iSys=0;iSys<3;iSys++) fFXSCalled[iSys]=kFALSE;
1522
c88ad5db 1523 Log(fCurrentDetector.Data(), "Process - Starting processing");
85a80aa9 1524
be48e3ea 1525 Int_t pid = fork();
1526
1527 if (pid < 0)
1528 {
c88ad5db 1529 Log("SHUTTLE", "Process - ERROR: Forking failed");
be48e3ea 1530 }
1531 else if (pid > 0)
1532 {
1533 // parent
c88ad5db 1534 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Starting monitoring",
be48e3ea 1535 GetCurrentRun(), aDetector->GetName()));
1536
1537 Long_t begin = time(0);
1538
1539 int status; // to be used with waitpid, on purpose an int (not Int_t)!
1540 while (waitpid(pid, &status, WNOHANG) == 0)
1541 {
1542 Long_t expiredTime = time(0) - begin;
1543
1544 if (expiredTime > fConfig->GetPPTimeOut())
1545 {
9827400b 1546 TString tmp;
c88ad5db 1547 tmp.Form("Process - Process of %s time out. "
1548 "Run time: %d seconds. Killing...",
1549 fCurrentDetector.Data(), expiredTime);
9827400b 1550 Log("SHUTTLE", tmp);
1551 Log(fCurrentDetector, tmp);
be48e3ea 1552
1553 kill(pid, 9);
1554
3301427a 1555 UpdateShuttleStatus(AliShuttleStatus::kPPTimeOut);
be48e3ea 1556 hasError = kTRUE;
1557
1558 gSystem->Sleep(1000);
1559 }
1560 else
1561 {
be48e3ea 1562 gSystem->Sleep(1000);
9827400b 1563
1564 TString checkStr;
1565 checkStr.Form("ps -o vsize --pid %d | tail -n 1", pid);
1566 FILE* pipe = gSystem->OpenPipe(checkStr, "r");
1567 if (!pipe)
1568 {
c88ad5db 1569 Log("SHUTTLE", Form("Process - Error: "
1570 "Could not open pipe to %s", checkStr.Data()));
9827400b 1571 continue;
1572 }
1573
1574 char buffer[100];
1575 if (!fgets(buffer, 100, pipe))
1576 {
c88ad5db 1577 Log("SHUTTLE", "Process - Error: ps did not return anything");
9827400b 1578 gSystem->ClosePipe(pipe);
1579 continue;
1580 }
1581 gSystem->ClosePipe(pipe);
1582
1583 //Log("SHUTTLE", Form("ps returned %s", buffer));
1584
1585 Int_t mem = 0;
1586 if ((sscanf(buffer, "%d\n", &mem) != 1) || !mem)
1587 {
c88ad5db 1588 Log("SHUTTLE", "Process - Error: Could not parse output of ps");
9827400b 1589 continue;
1590 }
1591
1592 if (expiredTime % 60 == 0)
ee6f7523 1593 {
c88ad5db 1594 Log("SHUTTLE", Form("Process - %s: Checking process. "
1595 "Run time: %d seconds - Memory consumption: %d KB",
1596 fCurrentDetector.Data(), expiredTime, mem));
ee6f7523 1597 SendAlive();
1598 }
9827400b 1599
1600 if (mem > fConfig->GetPPMaxMem())
1601 {
1602 TString tmp;
c88ad5db 1603 tmp.Form("Process - Process exceeds maximum allowed memory "
1604 "(%d KB > %d KB). Killing...",
9827400b 1605 mem, fConfig->GetPPMaxMem());
1606 Log("SHUTTLE", tmp);
1607 Log(fCurrentDetector, tmp);
1608
1609 kill(pid, 9);
1610
1611 UpdateShuttleStatus(AliShuttleStatus::kPPOutOfMemory);
1612 hasError = kTRUE;
1613
1614 gSystem->Sleep(1000);
1615 }
be48e3ea 1616 }
1617 }
1618
c88ad5db 1619 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Client has terminated.",
be48e3ea 1620 GetCurrentRun(), aDetector->GetName()));
1621
1622 if (WIFEXITED(status))
1623 {
1624 Int_t returnCode = WEXITSTATUS(status);
1625
c88ad5db 1626 Log("SHUTTLE", Form("Process - %s: the return code is %d", fCurrentDetector.Data(),
3301427a 1627 returnCode));
be48e3ea 1628
9827400b 1629 if (returnCode == 0) hasError = kTRUE;
be48e3ea 1630 }
1631 }
1632 else if (pid == 0)
1633 {
1634 // client
c88ad5db 1635 Log("SHUTTLE", Form("Process - In client process of %d - %s", GetCurrentRun(),
1636 aDetector->GetName()));
be48e3ea 1637
c88ad5db 1638 Log("SHUTTLE", Form("Process - Redirecting output to %s log",fCurrentDetector.Data()));
ffa29e93 1639
546242fb 1640 if ((freopen(GetLogFileName(fCurrentDetector), "a", stdout)) == 0)
ffa29e93 1641 {
c88ad5db 1642 Log("SHUTTLE", "Process - Could not freopen stdout");
ffa29e93 1643 }
1644 else
1645 {
1646 fOutputRedirected = kTRUE;
1647 if ((dup2(fileno(stdout), fileno(stderr))) < 0)
c88ad5db 1648 Log("SHUTTLE", "Process - Could not redirect stderr");
ffa29e93 1649
1650 }
1651
5bac2bde 1652 TString wd = gSystem->WorkingDirectory();
675f64cd 1653 TString tmpDir = Form("%s/%s_%d_process", GetShuttleTempDir(),
1654 fCurrentDetector.Data(), GetCurrentRun());
5bac2bde 1655
d524ade6 1656 Int_t result = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
1657 if (!result) // temp dir already exists!
1658 {
1659 Log(fCurrentDetector.Data(),
1660 Form("Process - %s dir already exists! Removing...", tmpDir.Data()));
1661 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
675f64cd 1662 }
1663
1664 if (gSystem->mkdir(tmpDir.Data(), 1))
1665 {
1666 Log(fCurrentDetector.Data(), "Process - could not make temp directory!!");
1667 gSystem->Exit(1);
d524ade6 1668 }
1669
1670 if (!gSystem->ChangeDirectory(tmpDir.Data()))
1671 {
1672 Log(fCurrentDetector.Data(), "Process - could not change directory!!");
1673 gSystem->Exit(1);
1674 }
5bac2bde 1675
9827400b 1676 Bool_t success = ProcessCurrentDetector();
5bac2bde 1677
1678 gSystem->ChangeDirectory(wd.Data());
b1d18693 1679
9827400b 1680 if (success) // Preprocessor finished successfully!
1681 {
b1d18693 1682 // remove temporary folder
1683 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
1684
3301427a 1685 // Update time_processed field in FXS DB
1686 if (UpdateTable() == kFALSE)
5bac2bde 1687 Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!",
1688 fCurrentDetector.Data()));
3301427a 1689
1690 // Transfer the data from local storage to main storage (Grid)
1691 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1692 if (StoreOCDB() == kFALSE)
1693 {
c88ad5db 1694 Log("SHUTTLE",
1695 Form("\t\t\t****** run %d - %s: STORAGE ERROR ******",
3301427a 1696 GetCurrentRun(), aDetector->GetName()));
1697 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
9827400b 1698 success = kFALSE;
3301427a 1699 } else {
c88ad5db 1700 Log("SHUTTLE",
1701 Form("\t\t\t****** run %d - %s: DONE ******",
3301427a 1702 GetCurrentRun(), aDetector->GetName()));
1703 UpdateShuttleStatus(AliShuttleStatus::kDone);
9827400b 1704 UpdateShuttleLogbook(fCurrentDetector, "DONE");
3301427a 1705 }
c88ad5db 1706 } else
1707 {
1708 Log("SHUTTLE",
1709 Form("\t\t\t****** run %d - %s: PP ERROR ******",
1710 GetCurrentRun(), aDetector->GetName()));
be48e3ea 1711 }
1712
4b95672b 1713 for (UInt_t iSys=0; iSys<3; iSys++)
1714 {
1715 if (fFXSCalled[iSys]) fFXSlist[iSys].Clear();
1716 }
1717
c88ad5db 1718 Log("SHUTTLE", Form("Process - Client process of %d - %s is exiting now with %d.",
9827400b 1719 GetCurrentRun(), aDetector->GetName(), success));
be48e3ea 1720
1721 // the client exits here
9827400b 1722 gSystem->Exit(success);
be48e3ea 1723
1724 AliError("We should never get here!!!");
1725 }
7bfb2090 1726 }
5164a766 1727
c88ad5db 1728 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^*",
2bb7b766 1729 GetCurrentRun()));
1730
1731 //check if shuttle is done for this run, if so update logbook
1732 TObjArray checkEntryArray;
1733 checkEntryArray.SetOwner(1);
9e080f92 1734 TString whereClause = Form("where run=%d", GetCurrentRun());
b0e53b15 1735 if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) ||
1736 checkEntryArray.GetEntries() == 0) {
9e080f92 1737 Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
1738 GetCurrentRun()));
1739 return hasError == kFALSE;
1740 }
b948db8d 1741
9e080f92 1742 AliShuttleLogbookEntry* checkEntry = dynamic_cast<AliShuttleLogbookEntry*>
1743 (checkEntryArray.At(0));
2bb7b766 1744
9e080f92 1745 if (checkEntry)
1746 {
1747 if (checkEntry->IsDone())
be48e3ea 1748 {
9e080f92 1749 Log("SHUTTLE","Process - Shuttle is DONE. Updating logbook");
1750 UpdateShuttleLogbook("shuttle_done");
1751 }
1752 else
1753 {
1754 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
be48e3ea 1755 {
9e080f92 1756 if (checkEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
be48e3ea 1757 {
9e080f92 1758 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
1759 checkEntry->GetRun(), GetDetName(iDet)));
1760 fFirstUnprocessed[iDet] = kFALSE;
be48e3ea 1761 }
1762 }
2bb7b766 1763 }
1764 }
1765
1766 fLogbookEntry = 0;
85a80aa9 1767
a7160fe9 1768 return hasError == kFALSE;
73abe331 1769}
1770
b948db8d 1771//______________________________________________________________________________________________
9827400b 1772Bool_t AliShuttle::ProcessCurrentDetector()
73abe331 1773{
1774 //
2bb7b766 1775 // Makes data retrieval just for a specific detector (fCurrentDetector).
73abe331 1776 // Threre should be a configuration for this detector.
73abe331 1777
1d172743 1778 Log("SHUTTLE", Form("ProcessCurrentDetector - Retrieving values for %s, run %d",
1779 fCurrentDetector.Data(), GetCurrentRun()));
73abe331 1780
d524ade6 1781 TString wd = gSystem->WorkingDirectory();
1782
2d9019b4 1783 if (!CleanReferenceStorage(fCurrentDetector.Data()))
546242fb 1784 return kFALSE;
d524ade6 1785
1786 gSystem->ChangeDirectory(wd.Data());
1787
1d172743 1788 TMap* dcsMap = new TMap();
3301427a 1789
1790 // call preprocessor
1791 AliPreprocessor* aPreprocessor =
1792 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1793
1794 aPreprocessor->Initialize(GetCurrentRun(), GetCurrentStartTime(), GetCurrentEndTime());
1795
1796 Bool_t processDCS = aPreprocessor->ProcessDCS();
d477ad88 1797
651fdaab 1798 if (!processDCS)
1799 {
1d172743 1800 Log(fCurrentDetector, "ProcessCurrentDetector -"
1801 " The preprocessor requested to skip the retrieval of DCS values");
651fdaab 1802 }
8b739301 1803 else if (fTestMode & kSkipDCS)
2c15234c 1804 {
1d172743 1805 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Skipping DCS processing");
9827400b 1806 }
1807 else if (fTestMode & kErrorDCS)
1808 {
1d172743 1809 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Simulating DCS error");
3d8bc902 1810 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
9827400b 1811 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1d172743 1812 delete dcsMap;
9827400b 1813 return kFALSE;
2c15234c 1814 } else {
3301427a 1815
1816 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
1817
1d172743 1818 // Query DCS archive
1819 Int_t nServers = fConfig->GetNServers(fCurrentDetector);
a038aa70 1820
1d172743 1821 for (int iServ=0; iServ<nServers; iServ++)
2c15234c 1822 {
1d172743 1823
1824 TString host(fConfig->GetDCSHost(fCurrentDetector, iServ));
1825 Int_t port = fConfig->GetDCSPort(fCurrentDetector, iServ);
542b6cc8 1826 Int_t multiSplit = fConfig->GetMultiSplit(fCurrentDetector, iServ);
1827
1790d4b7 1828 Log(fCurrentDetector, Form("ProcessCurrentDetector -"
1829 " Querying DCS Amanda server %s:%d (%d of %d)",
1830 host.Data(), port, iServ+1, nServers));
1d172743 1831
1832 TMap* aliasMap = 0;
1833 TMap* dpMap = 0;
1834
1835 if (fConfig->GetDCSAliases(fCurrentDetector, iServ)->GetEntries() > 0)
2c15234c 1836 {
1d172743 1837 aliasMap = GetValueSet(host, port,
542b6cc8 1838 fConfig->GetDCSAliases(fCurrentDetector, iServ),
1839 kAlias, multiSplit);
1d172743 1840 if (!aliasMap)
1841 {
1842 Log(fCurrentDetector,
1843 Form("ProcessCurrentDetector -"
675f64cd 1844 " Error retrieving DCS aliases from server %s."
1845 " Sending mail to DCS experts!", host.Data()));
1d172743 1846 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
675f64cd 1847
1848 if (!SendMailToDCS())
1849 Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
1850
a038aa70 1851 delete dcsMap;
1d172743 1852 return kFALSE;
1853 }
2c15234c 1854 }
a038aa70 1855
1d172743 1856 if (fConfig->GetDCSDataPoints(fCurrentDetector, iServ)->GetEntries() > 0)
a038aa70 1857 {
1d172743 1858 dpMap = GetValueSet(host, port,
542b6cc8 1859 fConfig->GetDCSDataPoints(fCurrentDetector, iServ),
1860 kDP, multiSplit);
1d172743 1861 if (!dpMap)
1862 {
1863 Log(fCurrentDetector,
1864 Form("ProcessCurrentDetector -"
675f64cd 1865 " Error retrieving DCS data points from server %s."
1866 " Sending mail to DCS experts!", host.Data()));
1d172743 1867 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
675f64cd 1868
1869 if (!SendMailToDCS())
1870 Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
1871
1d172743 1872 if (aliasMap) delete aliasMap;
1873 delete dcsMap;
1874 return kFALSE;
1875 }
a038aa70 1876 }
1d172743 1877
1878 // merge aliasMap and dpMap into dcsMap
1879 if(aliasMap) {
1880 TIter iter(aliasMap);
a038aa70 1881 TObjString* key = 0;
1882 while ((key = (TObjString*) iter.Next()))
1d172743 1883 dcsMap->Add(key, aliasMap->GetValue(key->String()));
1884
1885 aliasMap->SetOwner(kFALSE);
1886 delete aliasMap;
1887 }
1888
1889 if(dpMap) {
1890 TIter iter(dpMap);
1891 TObjString* key = 0;
1892 while ((key = (TObjString*) iter.Next()))
1893 dcsMap->Add(key, dpMap->GetValue(key->String()));
1894
1895 dpMap->SetOwner(kFALSE);
1896 delete dpMap;
a038aa70 1897 }
73abe331 1898 }
1899 }
dc25836b 1900
b1d18693 1901 // save map into file, to help debugging in case of preprocessor error
1902 TFile* f = TFile::Open("DCSMap.root","recreate");
1903 f->cd();
1904 dcsMap->Write("DCSMap", TObject::kSingleKey);
1905 f->Close();
1906 delete f;
1907
2bb7b766 1908 // DCS Archive DB processing successful. Call Preprocessor!
85a80aa9 1909 UpdateShuttleStatus(AliShuttleStatus::kPPStarted);
a7160fe9 1910
a038aa70 1911 UInt_t returnValue = aPreprocessor->Process(dcsMap);
b948db8d 1912
3301427a 1913 if (returnValue > 0) // Preprocessor error!
1914 {
c88ad5db 1915 Log(fCurrentDetector, Form("ProcessCurrentDetector - "
1916 "Preprocessor failed. Process returned %d.", returnValue));
cb343cfd 1917 UpdateShuttleStatus(AliShuttleStatus::kPPError);
a038aa70 1918 dcsMap->DeleteAll();
1919 delete dcsMap;
9827400b 1920 return kFALSE;
1921 }
1922
1923 // preprocessor ok!
1924 UpdateShuttleStatus(AliShuttleStatus::kPPDone);
1925 Log(fCurrentDetector, Form("ProcessCurrentDetector - %s preprocessor returned success",
1926 fCurrentDetector.Data()));
b948db8d 1927
a038aa70 1928 dcsMap->DeleteAll();
1929 delete dcsMap;
b948db8d 1930
9827400b 1931 return kTRUE;
2bb7b766 1932}
1933
1934//______________________________________________________________________________________________
1935Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
1936 TObjArray& entries)
1937{
9827400b 1938 // Query DAQ's Shuttle logbook and fills detector status object.
1939 // Call QueryRunParameters to query DAQ logbook for run parameters.
1940 //
2bb7b766 1941
fc5a4708 1942 entries.SetOwner(1);
1943
2bb7b766 1944 // check connection, in case connect
be48e3ea 1945 if(!Connect(3)) return kFALSE;
2bb7b766 1946
1947 TString sqlQuery;
441b0e9c 1948 sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
2bb7b766 1949
be48e3ea 1950 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1951 if (!aResult) {
1952 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
1953 return kFALSE;
1954 }
1955
fc5a4708 1956 AliDebug(2,Form("Query = %s", sqlQuery.Data()));
1957
2bb7b766 1958 if(aResult->GetRowCount() == 0) {
c88ad5db 1959 Log("SHUTTLE", "No entries in Shuttle Logbook match request");
9827400b 1960 delete aResult;
1961 return kTRUE;
2bb7b766 1962 }
1963
1964 // TODO Check field count!
db99d43e 1965 const UInt_t nCols = 23;
2bb7b766 1966 if (aResult->GetFieldCount() != (Int_t) nCols) {
c88ad5db 1967 Log("SHUTTLE", "Invalid SQL result field number!");
2bb7b766 1968 delete aResult;
1969 return kFALSE;
1970 }
1971
2bb7b766 1972 TSQLRow* aRow;
1973 while ((aRow = aResult->Next())) {
1974 TString runString(aRow->GetField(0), aRow->GetFieldLength(0));
1975 Int_t run = runString.Atoi();
1976
eba76848 1977 AliShuttleLogbookEntry *entry = QueryRunParameters(run);
1978 if (!entry)
1979 continue;
2bb7b766 1980
1981 // loop on detectors
eba76848 1982 for(UInt_t ii = 0; ii < nCols; ii++)
1983 entry->SetDetectorStatus(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1984
eba76848 1985 entries.AddLast(entry);
2bb7b766 1986 delete aRow;
1987 }
1988
2bb7b766 1989 delete aResult;
1990 return kTRUE;
1991}
1992
1993//______________________________________________________________________________________________
eba76848 1994AliShuttleLogbookEntry* AliShuttle::QueryRunParameters(Int_t run)
2bb7b766 1995{
eba76848 1996 //
1997 // Retrieve run parameters written in the DAQ logbook and sets them into AliShuttleLogbookEntry object
1998 //
2bb7b766 1999
2000 // check connection, in case connect
be48e3ea 2001 if (!Connect(3))
eba76848 2002 return 0;
2bb7b766 2003
2004 TString sqlQuery;
2c15234c 2005 sqlQuery.Form("select * from %s where run=%d", fConfig->GetDAQlbTable(), run);
2bb7b766 2006
be48e3ea 2007 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 2008 if (!aResult) {
c88ad5db 2009 Log("SHUTTLE", Form("Can't execute query <%s>!", sqlQuery.Data()));
eba76848 2010 return 0;
2bb7b766 2011 }
2012
eba76848 2013 if (aResult->GetRowCount() == 0) {
2bb7b766 2014 Log("SHUTTLE", Form("QueryRunParameters - No entry in DAQ Logbook for run %d. Skipping", run));
2015 delete aResult;
eba76848 2016 return 0;
2bb7b766 2017 }
2018
eba76848 2019 if (aResult->GetRowCount() > 1) {
c88ad5db 2020 Log("SHUTTLE", Form("QueryRunParameters - UNEXPECTED: "
2021 "more than one entry in DAQ Logbook for run %d!", run));
2bb7b766 2022 delete aResult;
eba76848 2023 return 0;
2bb7b766 2024 }
2025
eba76848 2026 TSQLRow* aRow = aResult->Next();
2027 if (!aRow)
2028 {
c88ad5db 2029 Log("SHUTTLE", Form("QueryRunParameters - Could not retrieve row for run %d. Skipping", run));
eba76848 2030 delete aResult;
2031 return 0;
2032 }
2bb7b766 2033
eba76848 2034 AliShuttleLogbookEntry* entry = new AliShuttleLogbookEntry(run);
2bb7b766 2035
eba76848 2036 for (Int_t ii = 0; ii < aResult->GetFieldCount(); ii++)
2037 entry->SetRunParameter(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 2038
eba76848 2039 UInt_t startTime = entry->GetStartTime();
2040 UInt_t endTime = entry->GetEndTime();
2041
1abfbb60 2042// if (!startTime || !endTime || startTime > endTime)
2043// {
2044// Log("SHUTTLE",
2045// Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d. Skipping!",
2046// run, startTime, endTime));
2047//
2048// Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2049// fLogbookEntry = entry;
2050// if (!UpdateShuttleLogbook("shuttle_done"))
2051// {
2052// AliError(Form("Could not update logbook for run %d !", run));
2053// }
2054// fLogbookEntry = 0;
2055//
2056// delete entry;
2057// delete aRow;
2058// delete aResult;
2059// return 0;
2060// }
2061
2062 if (!startTime)
b0e53b15 2063 {
2064 Log("SHUTTLE",
1abfbb60 2065 Form("QueryRunParameters - Invalid parameters for Run %d: "
2066 "startTime = %d, endTime = %d. Skipping!",
2067 run, startTime, endTime));
b0e53b15 2068
2069 Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2070 fLogbookEntry = entry;
ee6f7523 2071 if (!UpdateShuttleLogbook("shuttle_ignored"))
b0e53b15 2072 {
2073 AliError(Form("Could not update logbook for run %d !", run));
2074 }
2075 fLogbookEntry = 0;
2076
2077 delete entry;
2078 delete aRow;
2079 delete aResult;
2080 return 0;
2081 }
2082
1abfbb60 2083 if (startTime && !endTime)
2084 {
2085 // TODO Here we don't mark SHUTTLE done, because this may mean
2086 //the run is still ongoing!!
2087 Log("SHUTTLE",
2088 Form("QueryRunParameters - Invalid parameters for Run %d: "
2089 "startTime = %d, endTime = %d. Skipping (Shuttle won't be marked as DONE)!",
2090 run, startTime, endTime));
2091
2092 //Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2093 //fLogbookEntry = entry;
2094 //if (!UpdateShuttleLogbook("shuttle_done"))
2095 //{
2096 // AliError(Form("Could not update logbook for run %d !", run));
2097 //}
2098 //fLogbookEntry = 0;
2099
2100 delete entry;
2101 delete aRow;
2102 delete aResult;
2103 return 0;
2104 }
2105
2106 if (startTime && endTime && (startTime > endTime))
2107 {
2108 Log("SHUTTLE",
2109 Form("QueryRunParameters - Invalid parameters for Run %d: "
2110 "startTime = %d, endTime = %d. Skipping!",
2111 run, startTime, endTime));
2112
2113 Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2114 fLogbookEntry = entry;
ee6f7523 2115 if (!UpdateShuttleLogbook("shuttle_ignored"))
1abfbb60 2116 {
2117 AliError(Form("Could not update logbook for run %d !", run));
2118 }
2119 fLogbookEntry = 0;
2120
2121 delete entry;
2122 delete aRow;
2123 delete aResult;
2124 return 0;
2125 }
2126
b0e53b15 2127 TString totEventsStr = entry->GetRunParameter("totalEvents");
2128 Int_t totEvents = totEventsStr.Atoi();
2129 if (totEvents < 1)
2130 {
eba76848 2131 Log("SHUTTLE",
b0e53b15 2132 Form("QueryRunParameters - Run %d has 0 events - Skipping!", run));
2133
2134 Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
2135 fLogbookEntry = entry;
97e3c167 2136 if (!UpdateShuttleLogbook("shuttle_ignored"))
b0e53b15 2137 {
2138 AliError(Form("Could not update logbook for run %d !", run));
2139 }
2140 fLogbookEntry = 0;
2141
eba76848 2142 delete entry;
2bb7b766 2143 delete aRow;
eba76848 2144 delete aResult;
2145 return 0;
2bb7b766 2146 }
2147
eba76848 2148 delete aRow;
2bb7b766 2149 delete aResult;
eba76848 2150
2151 return entry;
2bb7b766 2152}
2153
a038aa70 2154//______________________________________________________________________________________________
2155TMap* AliShuttle::GetValueSet(const char* host, Int_t port, const TSeqCollection* entries,
542b6cc8 2156 DCSType type, Int_t multiSplit)
a038aa70 2157{
2158 // Retrieve all "entry" data points from the DCS server
2159 // host, port: TSocket connection parameters
2160 // entries: list of name of the alias or data point
2161 // type: kAlias or kDP
2162 // returns TMap of values, 0 when failure
542b6cc8 2163
2164 AliDCSClient client(host, port, fTimeout, fRetries, multiSplit);
b41b252a 2165
a038aa70 2166 TMap* result = 0;
b41b252a 2167 if (type == kAlias)
a038aa70 2168 {
b41b252a 2169 result = client.GetAliasValues(entries, GetCurrentStartTime(),
2170 GetCurrentEndTime());
2171 }
2172 else if (type == kDP)
2173 {
2174 result = client.GetDPValues(entries, GetCurrentStartTime(),
2175 GetCurrentEndTime());
2176 }
a038aa70 2177
b41b252a 2178 if (result == 0)
2179 {
2180 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get entries! Reason: %s",
1790d4b7 2181 client.GetErrorString(client.GetResultErrorCode())));
2182 if (client.GetResultErrorCode() == AliDCSClient::fgkServerError)
2183 Log(fCurrentDetector.Data(), Form("GetValueSet - Server error code: %s",
2184 client.GetServerError().Data()));
a038aa70 2185
b41b252a 2186 return 0;
a038aa70 2187 }
b41b252a 2188
a038aa70 2189 return result;
2190}
b41b252a 2191
b948db8d 2192//______________________________________________________________________________________________
57f50b3c 2193const char* AliShuttle::GetFile(Int_t system, const char* detector,
2194 const char* id, const char* source)
b948db8d 2195{
9827400b 2196 // Get calibration file from file exchange servers
2197 // First queris the FXS database for the file name, using the run, detector, id and source info
2198 // then calls RetrieveFile(filename) for actual copy to local disk
2199 // run: current run being processed (given by Logbook entry fLogbookEntry)
2200 // detector: the Preprocessor name
2201 // id: provided as a parameter by the Preprocessor
2202 // source: provided by the Preprocessor through GetFileSources function
2203
2204 // check if test mode should simulate a FXS error
2205 if (fTestMode & kErrorFXSFiles)
2206 {
2207 Log(detector, Form("GetFile - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2208 return 0;
2209 }
2210
57f50b3c 2211 // check connection, in case connect
9d733021 2212 if (!Connect(system))
eba76848 2213 {
9d733021 2214 Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
57f50b3c 2215 return 0;
2216 }
2217
2218 // Query preparation
9d733021 2219 TString sourceName(source);
d386d623 2220 Int_t nFields = 3;
2221 TString sqlQueryStart = Form("select filePath,size,fileChecksum from %s where",
2222 fConfig->GetFXSdbTable(system));
2223 TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
2224 GetCurrentRun(), detector, id);
2225
9d733021 2226 if (system == kDAQ)
2227 {
d386d623 2228 whereClause += Form(" and DAQsource=\"%s\"", source);
57f50b3c 2229 }
9d733021 2230 else if (system == kDCS)
eba76848 2231 {
9d733021 2232 sourceName="none";
57f50b3c 2233 }
9d733021 2234 else if (system == kHLT)
9e080f92 2235 {
d386d623 2236 whereClause += Form(" and DDLnumbers=\"%s\"", source);
9d733021 2237 nFields = 3;
9e080f92 2238 }
2239
9e080f92 2240 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2241
2242 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2243
2244 // Query execution
2245 TSQLResult* aResult = 0;
9d733021 2246 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
9e080f92 2247 if (!aResult) {
9d733021 2248 Log(detector, Form("GetFileName - Can't execute SQL query to %s database for: id = %s, source = %s",
2249 GetSystemName(system), id, sourceName.Data()));
9e080f92 2250 return 0;
2251 }
2252
2253 if(aResult->GetRowCount() == 0)
2254 {
2255 Log(detector,
9d733021 2256 Form("GetFileName - No entry in %s FXS db for: id = %s, source = %s",
2257 GetSystemName(system), id, sourceName.Data()));
9e080f92 2258 delete aResult;
2259 return 0;
2260 }
2bb7b766 2261
9e080f92 2262 if (aResult->GetRowCount() > 1) {
2263 Log(detector,
9d733021 2264 Form("GetFileName - More than one entry in %s FXS db for: id = %s, source = %s",
2265 GetSystemName(system), id, sourceName.Data()));
9e080f92 2266 delete aResult;
2267 return 0;
2268 }
2269
9d733021 2270 if (aResult->GetFieldCount() != nFields) {
9e080f92 2271 Log(detector,
9d733021 2272 Form("GetFileName - Wrong field count in %s FXS db for: id = %s, source = %s",
2273 GetSystemName(system), id, sourceName.Data()));
9e080f92 2274 delete aResult;
2275 return 0;
2276 }
2277
2278 TSQLRow* aRow = dynamic_cast<TSQLRow*> (aResult->Next());
2279
2280 if (!aRow){
9d733021 2281 Log(detector, Form("GetFileName - Empty set result in %s FXS db from query: id = %s, source = %s",
2282 GetSystemName(system), id, sourceName.Data()));
9e080f92 2283 delete aResult;
2284 return 0;
2285 }
2286
2287 TString filePath(aRow->GetField(0), aRow->GetFieldLength(0));
2288 TString fileSize(aRow->GetField(1), aRow->GetFieldLength(1));
d386d623 2289 TString fileChecksum(aRow->GetField(2), aRow->GetFieldLength(2));
9e080f92 2290
2291 delete aResult;
2292 delete aRow;
2293
d386d623 2294 AliDebug(2, Form("filePath = %s; size = %s, fileChecksum = %s",
2295 filePath.Data(), fileSize.Data(), fileChecksum.Data()));
9e080f92 2296
9e080f92 2297 // retrieved file is renamed to make it unique
675f64cd 2298 TString localFileName = Form("%s/%s_%d_process/%s_%s_%d_%s_%s.shuttle",
2299 GetShuttleTempDir(), detector, GetCurrentRun(),
d524ade6 2300 GetSystemName(system), detector, GetCurrentRun(),
2301 id, sourceName.Data());
9d733021 2302
9e080f92 2303
9d733021 2304 // file retrieval from FXS
4b95672b 2305 UInt_t nRetries = 0;
2306 UInt_t maxRetries = 3;
2307 Bool_t result = kFALSE;
2308
2309 // copy!! if successful TSystem::Exec returns 0
2310 while(nRetries++ < maxRetries) {
2311 AliDebug(2, Form("Trying to copy file. Retry # %d", nRetries));
2312 result = RetrieveFile(system, filePath.Data(), localFileName.Data());
2313 if(!result)
2314 {
2315 Log(detector, Form("GetFileName - Copy of file %s from %s FXS failed",
9d733021 2316 filePath.Data(), GetSystemName(system)));
4b95672b 2317 continue;
4f0749a8 2318 }
9e080f92 2319
d386d623 2320 if (fileChecksum.Length()>0)
4b95672b 2321 {
2322 // compare md5sum of local file with the one stored in the FXS DB
d524ade6 2323 Int_t md5Comp = gSystem->Exec(Form("md5sum %s |grep %s 2>&1 > /dev/null",
2324 localFileName.Data(), fileChecksum.Data()));
9e080f92 2325
4b95672b 2326 if (md5Comp != 0)
2327 {
2328 Log(detector, Form("GetFileName - md5sum of file %s does not match with local copy!",
2329 filePath.Data()));
2330 result = kFALSE;
2331 continue;
2332 }
d386d623 2333 } else {
2334 Log(fCurrentDetector, Form("GetFile - md5sum of file %s not set in %s database, skipping comparison",
2335 filePath.Data(), GetSystemName(system)));
9d733021 2336 }
4b95672b 2337 if (result) break;
9e080f92 2338 }
2339
4b95672b 2340 if(!result) return 0;
2341
9d733021 2342 fFXSCalled[system]=kTRUE;
2343 TObjString *fileParams = new TObjString(Form("%s#!?!#%s", id, sourceName.Data()));
2344 fFXSlist[system].Add(fileParams);
9e080f92 2345
675f64cd 2346 static TString staticLocalFileName;
2347 staticLocalFileName.Form("%s", localFileName.Data());
2348
c88ad5db 2349 Log(fCurrentDetector, Form("GetFile - Retrieved file with id %s and "
2350 "source %s from %s to %s", id, source,
d524ade6 2351 GetSystemName(system), localFileName.Data()));
675f64cd 2352
d524ade6 2353 return staticLocalFileName.Data();
2bb7b766 2354}
2355
2356//______________________________________________________________________________________________
9d733021 2357Bool_t AliShuttle::RetrieveFile(UInt_t system, const char* fxsFileName, const char* localFileName)
9e080f92 2358{
9827400b 2359 //
2360 // Copies file from FXS to local Shuttle machine
2361 //
2bb7b766 2362
9e080f92 2363 // check temp directory: trying to cd to temp; if it does not exist, create it
d524ade6 2364 AliDebug(2, Form("Copy file %s from %s FXS into %s",
2365 GetSystemName(system), fxsFileName, localFileName));
2366
2367 TString tmpDir(localFileName);
2368
2369 tmpDir = tmpDir(0,tmpDir.Last('/'));
9e080f92 2370
d524ade6 2371 Int_t noDir = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
2372 if (noDir) // temp dir does not exists!
2373 {
2374 if (gSystem->mkdir(tmpDir.Data(), 1))
2375 {
2376 Log(fCurrentDetector.Data(), "RetrieveFile - could not make temp directory!!");
9e080f92 2377 return kFALSE;
2378 }
9e080f92 2379 }
2380
9d733021 2381 TString baseFXSFolder;
2382 if (system == kDAQ)
2383 {
2384 baseFXSFolder = "FES/";
2385 }
2386 else if (system == kDCS)
2387 {
2388 baseFXSFolder = "";
2389 }
2390 else if (system == kHLT)
2391 {
42fde080 2392 baseFXSFolder = "/opt/FXS/";
9d733021 2393 }
2394
2395
d524ade6 2396 TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s",
9d733021 2397 fConfig->GetFXSPort(system),
2398 fConfig->GetFXSUser(system),
2399 fConfig->GetFXSHost(system),
2400 baseFXSFolder.Data(),
2401 fxsFileName,
9e080f92 2402 localFileName);
2403
2404 AliDebug(2, Form("%s",command.Data()));
2405
4b95672b 2406 Bool_t result = (gSystem->Exec(command.Data()) == 0);
9e080f92 2407
4b95672b 2408 return result;
9e080f92 2409}
2410
2411//______________________________________________________________________________________________
9d733021 2412TList* AliShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
2413{
9827400b 2414 //
2415 // Get sources producing the condition file Id from file exchange servers
4a33bdd9 2416 // if id is NULL all sources are returned (distinct)
9827400b 2417 //
1bcd28db 2418
2419 Log(detector, Form("GetFileSources - Retrieving sources with id %s from %s", id, GetSystemName(system)));
9827400b 2420
2421 // check if test mode should simulate a FXS error
2422 if (fTestMode & kErrorFXSSources)
2423 {
2424 Log(detector, Form("GetFileSources - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2425 return 0;
2426 }
2427
9d733021 2428 if (system == kDCS)
2429 {
c88ad5db 2430 Log(detector, "GetFileSources - WARNING: DCS system has only one source of data!");
6297b37d 2431 TList *list = new TList();
2432 list->SetOwner(1);
2433 list->Add(new TObjString(" "));
2434 return list;
9d733021 2435 }
9e080f92 2436
2437 // check connection, in case connect
9d733021 2438 if (!Connect(system))
2439 {
4a33bdd9 2440 Log(detector, Form("GetFileSources - Couldn't connect to %s FXS database", GetSystemName(system)));
9d733021 2441 return NULL;
9e080f92 2442 }
2443
9d733021 2444 TString sourceName = 0;
2445 if (system == kDAQ)
2446 {
2447 sourceName = "DAQsource";
2448 } else if (system == kHLT)
2449 {
2450 sourceName = "DDLnumbers";
2451 }
2452
4a33bdd9 2453 TString sqlQueryStart = Form("select distinct %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
2454 TString whereClause = Form("run=%d and detector=\"%s\"",
2455 GetCurrentRun(), detector);
2456 if (id)
2457 whereClause += Form(" and fileId=\"%s\"", id);
9e080f92 2458 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2459
2460 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2461
2462 // Query execution
2463 TSQLResult* aResult;
9d733021 2464 aResult = fServer[system]->Query(sqlQuery);
9e080f92 2465 if (!aResult) {
9d733021 2466 Log(detector, Form("GetFileSources - Can't execute SQL query to %s database for id: %s",
2467 GetSystemName(system), id));
9e080f92 2468 return 0;
2469 }
2470
86aa42c3 2471 TList *list = new TList();
2472 list->SetOwner(1);
2473
9d733021 2474 if (aResult->GetRowCount() == 0)
2475 {
9e080f92 2476 Log(detector,
9d733021 2477 Form("GetFileSources - No entry in %s FXS table for id: %s", GetSystemName(system), id));
9e080f92 2478 delete aResult;
86aa42c3 2479 return list;
9e080f92 2480 }
2481
1bcd28db 2482 Log(detector, Form("GetFileSources - Found %d sources", aResult->GetRowCount()));
9e080f92 2483
1bcd28db 2484 TSQLRow* aRow;
9d733021 2485 while ((aRow = aResult->Next()))
2486 {
9e080f92 2487
9d733021 2488 TString source(aRow->GetField(0), aRow->GetFieldLength(0));
2489 AliDebug(2, Form("%s = %s", sourceName.Data(), source.Data()));
2490 list->Add(new TObjString(source));
9e080f92 2491 delete aRow;
2492 }
9d733021 2493
9e080f92 2494 delete aResult;
2495
2496 return list;
2bb7b766 2497}
2498
4a33bdd9 2499//______________________________________________________________________________________________
2500TList* AliShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
2501{
2502 //
2503 // Get all ids of condition files produced by a given source from file exchange servers
2504 //
2505
1bcd28db 2506 Log(detector, Form("GetFileIDs - Retrieving ids with source %s with %s", source, GetSystemName(system)));
2507
4a33bdd9 2508 // check if test mode should simulate a FXS error
2509 if (fTestMode & kErrorFXSSources)
2510 {
2511 Log(detector, Form("GetFileIDs - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2512 return 0;
2513 }
2514
2515 // check connection, in case connect
2516 if (!Connect(system))
2517 {
2518 Log(detector, Form("GetFileIDs - Couldn't connect to %s FXS database", GetSystemName(system)));
2519 return NULL;
2520 }
2521
2522 TString sourceName = 0;
2523 if (system == kDAQ)
2524 {
2525 sourceName = "DAQsource";
2526 } else if (system == kHLT)
2527 {
2528 sourceName = "DDLnumbers";
2529 }
2530
2531 TString sqlQueryStart = Form("select fileId from %s where", fConfig->GetFXSdbTable(system));
2532 TString whereClause = Form("run=%d and detector=\"%s\"",
2533 GetCurrentRun(), detector);
2534 if (sourceName.Length() > 0 && source)
2535 whereClause += Form(" and %s=\"%s\"", sourceName.Data(), source);
2536 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2537
2538 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2539
2540 // Query execution
2541 TSQLResult* aResult;
2542 aResult = fServer[system]->Query(sqlQuery);
2543 if (!aResult) {
2544 Log(detector, Form("GetFileIDs - Can't execute SQL query to %s database for source: %s",
2545 GetSystemName(system), source));
2546 return 0;
2547 }
2548
2549 TList *list = new TList();
2550 list->SetOwner(1);
2551
2552 if (aResult->GetRowCount() == 0)
2553 {
2554 Log(detector,
2555 Form("GetFileIDs - No entry in %s FXS table for source: %s", GetSystemName(system), source));
2556 delete aResult;
2557 return list;
2558 }
2559
1bcd28db 2560 Log(detector, Form("GetFileIDs - Found %d ids", aResult->GetRowCount()));
2561
4a33bdd9 2562 TSQLRow* aRow;
2563
2564 while ((aRow = aResult->Next()))
2565 {
2566
2567 TString id(aRow->GetField(0), aRow->GetFieldLength(0));
2568 AliDebug(2, Form("fileId = %s", id.Data()));
2569 list->Add(new TObjString(id));
2570 delete aRow;
2571 }
2572
2573 delete aResult;
2574
2575 return list;
2576}
2577
2bb7b766 2578//______________________________________________________________________________________________
9d733021 2579Bool_t AliShuttle::Connect(Int_t system)
2bb7b766 2580{
9827400b 2581 // Connect to MySQL Server of the system's FXS MySQL databases
2582 // DAQ Logbook, Shuttle Logbook and DAQ FXS db are on the same host
2583 //
57f50b3c 2584
9d733021 2585 // check connection: if already connected return
2586 if(fServer[system] && fServer[system]->IsConnected()) return kTRUE;
57f50b3c 2587
9d733021 2588 TString dbHost, dbUser, dbPass, dbName;
57f50b3c 2589
9d733021 2590 if (system < 3) // FXS db servers
2591 {
2592 dbHost = Form("mysql://%s:%d", fConfig->GetFXSdbHost(system), fConfig->GetFXSdbPort(system));
2593 dbUser = fConfig->GetFXSdbUser(system);
2594 dbPass = fConfig->GetFXSdbPass(system);
2595 dbName = fConfig->GetFXSdbName(system);
2596 } else { // Run & Shuttle logbook servers
2597 // TODO Will the Shuttle logbook server be the same as the Run logbook server ???
2598 dbHost = Form("mysql://%s:%d", fConfig->GetDAQlbHost(), fConfig->GetDAQlbPort());
2599 dbUser = fConfig->GetDAQlbUser();
2600 dbPass = fConfig->GetDAQlbPass();
2601 dbName = fConfig->GetDAQlbDB();
2602 }
57f50b3c 2603
9d733021 2604 fServer[system] = TSQLServer::Connect(dbHost.Data(), dbUser.Data(), dbPass.Data());
2605 if (!fServer[system] || !fServer[system]->IsConnected()) {
2606 if(system < 3)
2607 {
2608 AliError(Form("Can't establish connection to FXS database for %s",
2609 AliShuttleInterface::GetSystemName(system)));
2610 } else {
2611 AliError("Can't establish connection to Run logbook.");
57f50b3c 2612 }
9d733021 2613 if(fServer[system]) delete fServer[system];
2614 return kFALSE;
2bb7b766 2615 }
57f50b3c 2616
9d733021 2617 // Get tables
2618 TSQLResult* aResult=0;
2619 switch(system){
2620 case kDAQ:
2621 aResult = fServer[kDAQ]->GetTables(dbName.Data());
2622 break;
2623 case kDCS:
2624 aResult = fServer[kDCS]->GetTables(dbName.Data());
2625 break;
2626 case kHLT:
2627 aResult = fServer[kHLT]->GetTables(dbName.Data());
2628 break;
2629 default:
2630 aResult = fServer[3]->GetTables(dbName.Data());
2631 break;
2632 }
2633
2634 delete aResult;
2bb7b766 2635 return kTRUE;
2636}
57f50b3c 2637
9e080f92 2638//______________________________________________________________________________________________
9d733021 2639Bool_t AliShuttle::UpdateTable()
9e080f92 2640{
9827400b 2641 //
2642 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2643 //
9e080f92 2644
9d733021 2645 Bool_t result = kTRUE;
9e080f92 2646
9d733021 2647 for (UInt_t system=0; system<3; system++)
2648 {
2649 if(!fFXSCalled[system]) continue;
9e080f92 2650
9d733021 2651 // check connection, in case connect
2652 if (!Connect(system))
2653 {
2654 Log(fCurrentDetector, Form("UpdateTable - Couldn't connect to %s FXS database", GetSystemName(system)));
2655 result = kFALSE;
2656 continue;
9e080f92 2657 }
9e080f92 2658
9d733021 2659 TTimeStamp now; // now
2660
2661 // Loop on FXS list entries
2662 TIter iter(&fFXSlist[system]);
2663 TObjString *aFXSentry=0;
2664 while ((aFXSentry = dynamic_cast<TObjString*> (iter.Next())))
2665 {
2666 TString aFXSentrystr = aFXSentry->String();
2667 TObjArray *aFXSarray = aFXSentrystr.Tokenize("#!?!#");
2668 if (!aFXSarray || aFXSarray->GetEntries() != 2 )
2669 {
2670 Log(fCurrentDetector, Form("UpdateTable - error updating %s FXS entry. Check string: <%s>",
2671 GetSystemName(system), aFXSentrystr.Data()));
2672 if(aFXSarray) delete aFXSarray;
2673 result = kFALSE;
2674 continue;
2675 }
2676 const char* fileId = ((TObjString*) aFXSarray->At(0))->GetName();
2677 const char* source = ((TObjString*) aFXSarray->At(1))->GetName();
2678
2679 TString whereClause;
2680 if (system == kDAQ)
2681 {
2682 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DAQsource=\"%s\";",
2683 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2684 }
2685 else if (system == kDCS)
2686 {
2687 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\";",
2688 GetCurrentRun(), fCurrentDetector.Data(), fileId);
2689 }
2690 else if (system == kHLT)
2691 {
2692 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DDLnumbers=\"%s\";",
2693 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2694 }
2695
2696 delete aFXSarray;
9e080f92 2697
9d733021 2698 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2699 now.GetSec(), whereClause.Data());
9e080f92 2700
9d733021 2701 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
9e080f92 2702
9d733021 2703 // Query execution
2704 TSQLResult* aResult;
2705 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2706 if (!aResult)
2707 {
2708 Log(fCurrentDetector, Form("UpdateTable - %s db: can't execute SQL query <%s>",
2709 GetSystemName(system), sqlQuery.Data()));
2710 result = kFALSE;
2711 continue;
2712 }
2713 delete aResult;
9e080f92 2714 }
9e080f92 2715 }
2716
9d733021 2717 return result;
9e080f92 2718}
57f50b3c 2719
3301427a 2720//______________________________________________________________________________________________
2721Bool_t AliShuttle::UpdateTableFailCase()
2722{
9827400b 2723 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2724 // this is called in case the preprocessor is declared failed for the current run, because
2725 // the fields are updated only in case of success
3301427a 2726
2727 Bool_t result = kTRUE;
2728
2729 for (UInt_t system=0; system<3; system++)
2730 {
2731 // check connection, in case connect
2732 if (!Connect(system))
2733 {
2734 Log(fCurrentDetector, Form("UpdateTableFailCase - Couldn't connect to %s FXS database",
2735 GetSystemName(system)));
2736 result = kFALSE;
2737 continue;
2738 }
2739
2740 TTimeStamp now; // now
2741
2742 // Loop on FXS list entries
2743
2744 TString whereClause = Form("where run=%d and detector=\"%s\";",
2745 GetCurrentRun(), fCurrentDetector.Data());
2746
2747
2748 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2749 now.GetSec(), whereClause.Data());
2750
2751 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2752
2753 // Query execution
2754 TSQLResult* aResult;
2755 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2756 if (!aResult)
2757 {
2758 Log(fCurrentDetector, Form("UpdateTableFailCase - %s db: can't execute SQL query <%s>",
2759 GetSystemName(system), sqlQuery.Data()));
2760 result = kFALSE;
2761 continue;
2762 }
2763 delete aResult;
2764 }
2765
2766 return result;
2767}
2768
2bb7b766 2769//______________________________________________________________________________________________
2770Bool_t AliShuttle::UpdateShuttleLogbook(const char* detector, const char* status)
2771{
e7f62f16 2772 //
2773 // Update Shuttle logbook filling detector or shuttle_done column
2774 // ex. of usage: UpdateShuttleLogbook("PHOS", "DONE") or UpdateShuttleLogbook("shuttle_done")
2775 //
57f50b3c 2776
2bb7b766 2777 // check connection, in case connect
be48e3ea 2778 if(!Connect(3)){
2bb7b766 2779 Log("SHUTTLE", "UpdateShuttleLogbook - Couldn't connect to DAQ Logbook.");
2780 return kFALSE;
57f50b3c 2781 }
2782
2bb7b766 2783 TString detName(detector);
2784 TString setClause;
ee6f7523 2785 if (detName == "shuttle_done" || detName == "shuttle_ignored")
e7f62f16 2786 {
2bb7b766 2787 setClause = "set shuttle_done=1";
e7f62f16 2788
ee6f7523 2789 if (detName == "shuttle_done")
b0e53b15 2790 {
2791 // Send the information to ML
2792 TMonaLisaText mlStatus("SHUTTLE_status", "Done");
e7f62f16 2793
b0e53b15 2794 TList mlList;
2795 mlList.Add(&mlStatus);
2796
ee6f7523 2797 TString mlID;
2798 mlID.Form("%d", GetCurrentRun());
2799 fMonaLisa->SendParameters(&mlList, mlID);
b0e53b15 2800 }
2bb7b766 2801 } else {
2bb7b766 2802 TString statusStr(status);
2803 if(statusStr.Contains("done", TString::kIgnoreCase) ||
2804 statusStr.Contains("failed", TString::kIgnoreCase)){
eba76848 2805 setClause = Form("set %s=\"%s\"", detector, status);
2bb7b766 2806 } else {
2807 Log("SHUTTLE",
2808 Form("UpdateShuttleLogbook - Invalid status <%s> for detector %s",
2809 status, detector));
2810 return kFALSE;
2811 }
2812 }
57f50b3c 2813
2bb7b766 2814 TString whereClause = Form("where run=%d", GetCurrentRun());
2815
441b0e9c 2816 TString sqlQuery = Form("update %s %s %s",
2817 fConfig->GetShuttlelbTable(), setClause.Data(), whereClause.Data());
57f50b3c 2818
2bb7b766 2819 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2820
2821 // Query execution
2822 TSQLResult* aResult;
be48e3ea 2823 aResult = dynamic_cast<TSQLResult*> (fServer[3]->Query(sqlQuery));
2bb7b766 2824 if (!aResult) {
2825 Log("SHUTTLE", Form("UpdateShuttleLogbook - Can't execute query <%s>", sqlQuery.Data()));
2826 return kFALSE;
57f50b3c 2827 }
2bb7b766 2828 delete aResult;
57f50b3c 2829
2830 return kTRUE;
2831}
2832
2833//______________________________________________________________________________________________
2bb7b766 2834Int_t AliShuttle::GetCurrentRun() const
2835{
9827400b 2836 //
2837 // Get current run from logbook entry
2838 //
57f50b3c 2839
2bb7b766 2840 return fLogbookEntry ? fLogbookEntry->GetRun() : -1;
57f50b3c 2841}
2842
2843//______________________________________________________________________________________________
2bb7b766 2844UInt_t AliShuttle::GetCurrentStartTime() const
2845{
9827400b 2846 //
2847 // get current start time
2848 //
57f50b3c 2849
2bb7b766 2850 return fLogbookEntry ? fLogbookEntry->GetStartTime() : 0;
57f50b3c 2851}
2852
2853//______________________________________________________________________________________________
2bb7b766 2854UInt_t AliShuttle::GetCurrentEndTime() const
2855{
9827400b 2856 //
2857 // get current end time from logbook entry
2858 //
57f50b3c 2859
2bb7b766 2860 return fLogbookEntry ? fLogbookEntry->GetEndTime() : 0;
57f50b3c 2861}
2862
675f64cd 2863//______________________________________________________________________________________________
2864UInt_t AliShuttle::GetCurrentYear() const
2865{
2866 //
2867 // Get current year from logbook entry
2868 //
2869
2870 if (!fLogbookEntry) return 0;
2871
2872 TTimeStamp startTime(GetCurrentStartTime());
2873 TString year = Form("%d",startTime.GetDate());
2874 year = year(0,4);
2875
2876 return year.Atoi();
2877}
2878
2879//______________________________________________________________________________________________
2880const char* AliShuttle::GetLHCPeriod() const
2881{
2882 //
2883 // Get current LHC period from logbook entry
2884 //
2885
2886 if (!fLogbookEntry) return 0;
2887
2888 return fLogbookEntry->GetRunParameter("LHCperiod");
2889}
2890
b948db8d 2891//______________________________________________________________________________________________
2892void AliShuttle::Log(const char* detector, const char* message)
2893{
9827400b 2894 //
2895 // Fill log string with a message
2896 //
b948db8d 2897
7d4cf768 2898 TString logRunDir = GetShuttleLogDir();
2899 if (GetCurrentRun() >=0)
2900 logRunDir += Form("/%d", GetCurrentRun());
2901
2902 void* dir = gSystem->OpenDirectory(logRunDir.Data());
84090f85 2903 if (dir == NULL) {
7d4cf768 2904 if (gSystem->mkdir(logRunDir.Data(), kTRUE)) {
36c99a6a 2905 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
84090f85 2906 return;
2907 }
b948db8d 2908
84090f85 2909 } else {
2910 gSystem->FreeDirectory(dir);
2911 }
b948db8d 2912
cb343cfd 2913 TString toLog = Form("%s (%d): %s - ", TTimeStamp(time(0)).AsString("s"), getpid(), detector);
e7f62f16 2914 if (GetCurrentRun() >= 0)
2915 toLog += Form("run %d - ", GetCurrentRun());
2bb7b766 2916 toLog += Form("%s", message);
2917
84090f85 2918 AliInfo(toLog.Data());
ffa29e93 2919
2920 // if we redirect the log output already to the file, leave here
2921 if (fOutputRedirected && strcmp(detector, "SHUTTLE") != 0)
2922 return;
b948db8d 2923
ffa29e93 2924 TString fileName = GetLogFileName(detector);
e7f62f16 2925
84090f85 2926 gSystem->ExpandPathName(fileName);
2927
2928 ofstream logFile;
2929 logFile.open(fileName, ofstream::out | ofstream::app);
2930
2931 if (!logFile.is_open()) {
2932 AliError(Form("Could not open file %s", fileName.Data()));
2933 return;
2934 }
7bfb2090 2935
84090f85 2936 logFile << toLog.Data() << "\n";
b948db8d 2937
84090f85 2938 logFile.close();
b948db8d 2939}
2bb7b766 2940
ffa29e93 2941//______________________________________________________________________________________________
2942TString AliShuttle::GetLogFileName(const char* detector) const
2943{
2944 //
2945 // returns the name of the log file for a given sub detector
2946 //
2947
2948 TString fileName;
2949
2950 if (GetCurrentRun() >= 0)
7d4cf768 2951 {
2952 fileName.Form("%s/%d/%s_%d.log", GetShuttleLogDir(), GetCurrentRun(),
2953 detector, GetCurrentRun());
2954 } else {
ffa29e93 2955 fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
7d4cf768 2956 }
ffa29e93 2957
2958 return fileName;
2959}
2960
ee6f7523 2961//______________________________________________________________________________________________
2962void AliShuttle::SendAlive()
2963{
2964 // sends alive message to ML
2965
2966 TMonaLisaText mlStatus("SHUTTLE_status", "Alive");
2967
2968 TList mlList;
2969 mlList.Add(&mlStatus);
2970
2971 fMonaLisa->SendParameters(&mlList, "__PROCESSINGINFO__");
2972}
2973
2bb7b766 2974//______________________________________________________________________________________________
2975Bool_t AliShuttle::Collect(Int_t run)
2976{
9827400b 2977 //
2978 // Collects conditions data for all UNPROCESSED run written to DAQ LogBook in case of run = -1 (default)
2979 // If a dedicated run is given this run is processed
2980 //
2981 // In operational mode, this is the Shuttle function triggered by the EOR signal.
2982 //
2bb7b766 2983
eba76848 2984 if (run == -1)
2985 Log("SHUTTLE","Collect - Shuttle called. Collecting conditions data for unprocessed runs");
2986 else
2987 Log("SHUTTLE", Form("Collect - Shuttle called. Collecting conditions data for run %d", run));
cb343cfd 2988
2989 SetLastAction("Starting");
2bb7b766 2990
ee6f7523 2991 // create ML instance
2992 if (!fMonaLisa)
2993 fMonaLisa = new TMonaLisaWriter(fConfig->GetMonitorHost(), fConfig->GetMonitorTable());
2994
2995
2996 SendAlive();
2997
2bb7b766 2998 TString whereClause("where shuttle_done=0");
eba76848 2999 if (run != -1)
3000 whereClause += Form(" and run=%d", run);
2bb7b766 3001
3002 TObjArray shuttleLogbookEntries;
be48e3ea 3003 if (!QueryShuttleLogbook(whereClause, shuttleLogbookEntries))
3004 {
cb343cfd 3005 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2bb7b766 3006 return kFALSE;
3007 }
3008
9e080f92 3009 if (shuttleLogbookEntries.GetEntries() == 0)
3010 {
3011 if (run == -1)
3012 Log("SHUTTLE","Collect - Found no UNPROCESSED runs in Shuttle logbook");
3013 else
3014 Log("SHUTTLE", Form("Collect - Run %d is already DONE "
3015 "or it does not exist in Shuttle logbook", run));
3016 return kTRUE;
3017 }
3018
be48e3ea 3019 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
3020 fFirstUnprocessed[iDet] = kTRUE;
3021
fc5a4708 3022 if (run != -1)
be48e3ea 3023 {
3024 // query Shuttle logbook for earlier runs, check if some detectors are unprocessed,
3025 // flag them into fFirstUnprocessed array
3026 TString whereClause(Form("where shuttle_done=0 and run < %d", run));
3027 TObjArray tmpLogbookEntries;
3028 if (!QueryShuttleLogbook(whereClause, tmpLogbookEntries))
3029 {
3030 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
3031 return kFALSE;
3032 }
3033
3034 TIter iter(&tmpLogbookEntries);
3035 AliShuttleLogbookEntry* anEntry = 0;
3036 while ((anEntry = dynamic_cast<AliShuttleLogbookEntry*> (iter.Next())))
3037 {
3038 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
3039 {
3040 if (anEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
3041 {
3042 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
3043 anEntry->GetRun(), GetDetName(iDet)));
3044 fFirstUnprocessed[iDet] = kFALSE;
3045 }
3046 }
3047
3048 }
3049
3050 }
3051
3052 if (!RetrieveConditionsData(shuttleLogbookEntries))
3053 {
cb343cfd 3054 Log("SHUTTLE", "Collect - Process of at least one run failed");
2bb7b766 3055 return kFALSE;
3056 }
3057
36c99a6a 3058 Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
eba76848 3059 return kTRUE;
2bb7b766 3060}
3061
2bb7b766 3062//______________________________________________________________________________________________
3063Bool_t AliShuttle::RetrieveConditionsData(const TObjArray& dateEntries)
3064{
9827400b 3065 //
3066 // Retrieve conditions data for all runs that aren't processed yet
3067 //
2bb7b766 3068
3069 Bool_t hasError = kFALSE;
3070
3071 TIter iter(&dateEntries);
3072 AliShuttleLogbookEntry* anEntry;
3073
3074 while ((anEntry = (AliShuttleLogbookEntry*) iter.Next())){
3075 if (!Process(anEntry)){
3076 hasError = kTRUE;
3077 }
4b95672b 3078
3079 // clean SHUTTLE temp directory
d524ade6 3080 //TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
3081 //RemoveFile(filename.Data());
2bb7b766 3082 }
3083
3084 return hasError == kFALSE;
3085}
cb343cfd 3086
3087//______________________________________________________________________________________________
3088ULong_t AliShuttle::GetTimeOfLastAction() const
3089{
9827400b 3090 //
3091 // Gets time of last action
3092 //
3093
cb343cfd 3094 ULong_t tmp;
36c99a6a 3095
cb343cfd 3096 fMonitoringMutex->Lock();
be48e3ea 3097
cb343cfd 3098 tmp = fLastActionTime;
36c99a6a 3099
cb343cfd 3100 fMonitoringMutex->UnLock();
36c99a6a 3101
cb343cfd 3102 return tmp;
3103}
3104
3105//______________________________________________________________________________________________
3106const TString AliShuttle::GetLastAction() const
3107{
9827400b 3108 //
cb343cfd 3109 // returns a string description of the last action
9827400b 3110 //
cb343cfd 3111
3112 TString tmp;
36c99a6a 3113
cb343cfd 3114 fMonitoringMutex->Lock();
3115
3116 tmp = fLastAction;
3117
3118 fMonitoringMutex->UnLock();
3119
36c99a6a 3120 return tmp;
cb343cfd 3121}
3122
3123//______________________________________________________________________________________________
3124void AliShuttle::SetLastAction(const char* action)
3125{
9827400b 3126 //
cb343cfd 3127 // updates the monitoring variables
9827400b 3128 //
36c99a6a 3129
cb343cfd 3130 fMonitoringMutex->Lock();
36c99a6a 3131
cb343cfd 3132 fLastAction = action;
3133 fLastActionTime = time(0);
3134
3135 fMonitoringMutex->UnLock();
3136}
eba76848 3137
3138//______________________________________________________________________________________________
3139const char* AliShuttle::GetRunParameter(const char* param)
3140{
9827400b 3141 //
3142 // returns run parameter read from DAQ logbook
3143 //
eba76848 3144
3145 if(!fLogbookEntry) {
3146 AliError("No logbook entry!");
3147 return 0;
3148 }
3149
3150 return fLogbookEntry->GetRunParameter(param);
3151}
57c1a579 3152
d386d623 3153//______________________________________________________________________________________________
9827400b 3154AliCDBEntry* AliShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
d386d623 3155{
9827400b 3156 //
3157 // returns object from OCDB valid for current run
3158 //
d386d623 3159
9827400b 3160 if (fTestMode & kErrorOCDB)
3161 {
3162 Log(detector, "GetFromOCDB - In TESTMODE - Simulating error with OCDB");
3163 return 0;
3164 }
3165
d386d623 3166 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
3167 if (!sto)
3168 {
9827400b 3169 Log(detector, "GetFromOCDB - Cannot activate main OCDB for query!");
d386d623 3170 return 0;
3171 }
3172
3173 return dynamic_cast<AliCDBEntry*> (sto->Get(path, GetCurrentRun()));
3174}
3175
57c1a579 3176//______________________________________________________________________________________________
3177Bool_t AliShuttle::SendMail()
3178{
9827400b 3179 //
3180 // sends a mail to the subdetector expert in case of preprocessor error
3181 //
3182
3183 if (fTestMode != kNone)
3184 return kTRUE;
57c1a579 3185
36c99a6a 3186 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
57c1a579 3187 if (dir == NULL)
3188 {
36c99a6a 3189 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
57c1a579 3190 {
675f64cd 3191 Log("SHUTTLE", Form("SendMail - Can't open directory <%s>", GetShuttleLogDir()));
57c1a579 3192 return kFALSE;
3193 }
3194
3195 } else {
3196 gSystem->FreeDirectory(dir);
3197 }
3198
3199 TString bodyFileName;
36c99a6a 3200 bodyFileName.Form("%s/mail.body", GetShuttleLogDir());
57c1a579 3201 gSystem->ExpandPathName(bodyFileName);
3202
3203 ofstream mailBody;
3204 mailBody.open(bodyFileName, ofstream::out);
3205
3206 if (!mailBody.is_open())
3207 {
675f64cd 3208 Log("SHUTTLE", Form("Could not open mail body file %s", bodyFileName.Data()));
57c1a579 3209 return kFALSE;
3210 }
3211
3212 TString to="";
3213 TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
3214 TObjString *anExpert=0;
3215 while ((anExpert = (TObjString*) iterExperts.Next()))
3216 {
3217 to += Form("%s,", anExpert->GetName());
3218 }
3219 to.Remove(to.Length()-1);
909732f7 3220 AliDebug(2, Form("to: %s",to.Data()));
57c1a579 3221
86aa42c3 3222 if (to.IsNull()) {
c88ad5db 3223 Log("SHUTTLE", "List of detector responsibles not yet set!");
36c99a6a 3224 return kFALSE;
3225 }
3226
57c1a579 3227 TString cc="alberto.colla@cern.ch";
3228
546242fb 3229 TString subject = Form("%s Shuttle preprocessor FAILED in run %d !",
57c1a579 3230 fCurrentDetector.Data(), GetCurrentRun());
909732f7 3231 AliDebug(2, Form("subject: %s", subject.Data()));
57c1a579 3232
3233 TString body = Form("Dear %s expert(s), \n\n", fCurrentDetector.Data());
3234 body += Form("SHUTTLE just detected that your preprocessor "
546242fb 3235 "failed processing run %d!!\n\n", GetCurrentRun());
7d4cf768 3236 body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n",
3237 fCurrentDetector.Data());
b0e53b15 3238 if (fConfig->GetRunMode() == AliShuttleConfig::kTest)
3239 {
3240 body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
3241 } else {
3242 body += Form("\thttp://pcalimonitor.cern.ch/shuttle.jsp?instance=PROD?time=168 \n\n");
3243 }
3244
7d4cf768 3245
3246 TString logFolder = "logs";
3247 if (fConfig->GetRunMode() == AliShuttleConfig::kProd)
3248 logFolder += "_PROD";
3249
3250
546242fb 3251 body += Form("Find the %s log for the current run on \n\n"
7d4cf768 3252 "\thttp://pcalishuttle01.cern.ch:8880/%s/%d/%s_%d.log \n\n",
3253 fCurrentDetector.Data(), logFolder.Data(), GetCurrentRun(),
3254 fCurrentDetector.Data(), GetCurrentRun());
3255 body += Form("The last 10 lines of %s log file are following:\n\n", fCurrentDetector.Data());
57c1a579 3256
909732f7 3257 AliDebug(2, Form("Body begin: %s", body.Data()));
57c1a579 3258
3259 mailBody << body.Data();
3260 mailBody.close();
3261 mailBody.open(bodyFileName, ofstream::out | ofstream::app);
3262
7d4cf768 3263 TString logFileName = Form("%s/%d/%s_%d.log", GetShuttleLogDir(),
3264 GetCurrentRun(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 3265 TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
3266 if (gSystem->Exec(tailCommand.Data()))
3267 {
3268 mailBody << Form("%s log file not found ...\n\n", fCurrentDetector.Data());
3269 }
3270
3271 TString endBody = Form("------------------------------------------------------\n\n");
675f64cd 3272 endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
3273 endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
3274 endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
3275
3276 AliDebug(2, Form("Body end: %s", endBody.Data()));
3277
3278 mailBody << endBody.Data();
3279
3280 mailBody.close();
3281
3282 // send mail!
3283 TString mailCommand = Form("mail -s \"%s\" -c %s %s < %s",
3284 subject.Data(),
3285 cc.Data(),
3286 to.Data(),
3287 bodyFileName.Data());
3288 AliDebug(2, Form("mail command: %s", mailCommand.Data()));
3289
3290 Bool_t result = gSystem->Exec(mailCommand.Data());
3291
3292 return result == 0;
3293}
3294
3295//______________________________________________________________________________________________
3296Bool_t AliShuttle::SendMailToDCS()
3297{
3298 //
3299 // sends a mail to the DCS experts in case of DCS error
3300 //
3301
3302 if (fTestMode != kNone)
3303 return kTRUE;
3304
3305 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
3306 if (dir == NULL)
3307 {
3308 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
3309 {
3310 Log("SHUTTLE", Form("SendMailToDCS - Can't open directory <%s>", GetShuttleLogDir()));
3311 return kFALSE;
3312 }
3313
3314 } else {
3315 gSystem->FreeDirectory(dir);
3316 }
3317
3318 TString bodyFileName;
3319 bodyFileName.Form("%s/mail.body", GetShuttleLogDir());
3320 gSystem->ExpandPathName(bodyFileName);
3321
3322 ofstream mailBody;
3323 mailBody.open(bodyFileName, ofstream::out);
3324
3325 if (!mailBody.is_open())
3326 {
3327 Log("SHUTTLE", Form("SendMailToDCS - Could not open mail body file %s", bodyFileName.Data()));
3328 return kFALSE;
3329 }
3330
3331 TString to="Vladimir.Fekete@cern.ch, Svetozar.Kapusta@cern.ch";
3332 //TString to="alberto.colla@cern.ch";
3333 AliDebug(2, Form("to: %s",to.Data()));
3334
3335 if (to.IsNull()) {
3336 Log("SHUTTLE", "List of detector responsibles not yet set!");
3337 return kFALSE;
3338 }
3339
3340 TString cc="alberto.colla@cern.ch";
3341
3342 TString subject = Form("Retrieval of data points for %s FAILED in run %d !",
3343 fCurrentDetector.Data(), GetCurrentRun());
3344 AliDebug(2, Form("subject: %s", subject.Data()));
3345
3346 TString body = Form("Dear DCS experts, \n\n");
3347 body += Form("SHUTTLE couldn\'t retrieve the data points for detector %s "
3348 "in run %d!!\n\n", fCurrentDetector.Data(), GetCurrentRun());
7d4cf768 3349 body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n",
3350 fCurrentDetector.Data());
b0e53b15 3351 if (fConfig->GetRunMode() == AliShuttleConfig::kTest)
3352 {
3353 body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
3354 } else {
3355 body += Form("\thttp://pcalimonitor.cern.ch/shuttle.jsp?instance=PROD?time=168 \n\n");
3356 }
7d4cf768 3357
3358 TString logFolder = "logs";
3359 if (fConfig->GetRunMode() == AliShuttleConfig::kProd)
3360 logFolder += "_PROD";
3361
3362
675f64cd 3363 body += Form("Find the %s log for the current run on \n\n"
7d4cf768 3364 "\thttp://pcalishuttle01.cern.ch:8880/%s/%d/%s_%d.log \n\n",
3365 fCurrentDetector.Data(), logFolder.Data(), GetCurrentRun(),
3366 fCurrentDetector.Data(), GetCurrentRun());
3367 body += Form("The last 10 lines of %s log file are following:\n\n", fCurrentDetector.Data());
675f64cd 3368
3369 AliDebug(2, Form("Body begin: %s", body.Data()));
3370
3371 mailBody << body.Data();
3372 mailBody.close();
3373 mailBody.open(bodyFileName, ofstream::out | ofstream::app);
3374
7d4cf768 3375 TString logFileName = Form("%s/%d/%s_%d.log", GetShuttleLogDir(), GetCurrentRun(),
3376 fCurrentDetector.Data(), GetCurrentRun());
675f64cd 3377 TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
3378 if (gSystem->Exec(tailCommand.Data()))
3379 {
3380 mailBody << Form("%s log file not found ...\n\n", fCurrentDetector.Data());
3381 }
3382
3383 TString endBody = Form("------------------------------------------------------\n\n");
36c99a6a 3384 endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
3385 endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
546242fb 3386 endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
57c1a579 3387
909732f7 3388 AliDebug(2, Form("Body end: %s", endBody.Data()));
57c1a579 3389
3390 mailBody << endBody.Data();
3391
3392 mailBody.close();
3393
3394 // send mail!
3395 TString mailCommand = Form("mail -s \"%s\" -c %s %s < %s",
3396 subject.Data(),
3397 cc.Data(),
3398 to.Data(),
3399 bodyFileName.Data());
909732f7 3400 AliDebug(2, Form("mail command: %s", mailCommand.Data()));
57c1a579 3401
3402 Bool_t result = gSystem->Exec(mailCommand.Data());
3403
3404 return result == 0;
3405}
d386d623 3406
441b0e9c 3407//______________________________________________________________________________________________
9827400b 3408const char* AliShuttle::GetRunType()
441b0e9c 3409{
9827400b 3410 //
3411 // returns run type read from "run type" logbook
3412 //
441b0e9c 3413
3414 if(!fLogbookEntry) {
3415 AliError("No logbook entry!");
3416 return 0;
3417 }
3418
9827400b 3419 return fLogbookEntry->GetRunType();
441b0e9c 3420}
3421
4859271b 3422//______________________________________________________________________________________________
3423Bool_t AliShuttle::GetHLTStatus()
3424{
3425 // Return HLT status (ON=1 OFF=0)
3426 // Converts the HLT status from the status string read in the run logbook (not just a bool)
3427
3428 if(!fLogbookEntry) {
3429 AliError("No logbook entry!");
3430 return 0;
3431 }
3432
3433 // TODO implement when HLTStatus is inserted in run logbook
3434 //TString hltStatus = fLogbookEntry->GetRunParameter("HLTStatus");
3435 //if(hltStatus == "OFF") {return kFALSE};
3436
3437 return kTRUE;
3438}
3439
d386d623 3440//______________________________________________________________________________________________
3441void AliShuttle::SetShuttleTempDir(const char* tmpDir)
3442{
9827400b 3443 //
3444 // sets Shuttle temp directory
3445 //
d386d623 3446
3447 fgkShuttleTempDir = gSystem->ExpandPathName(tmpDir);
3448}
3449
3450//______________________________________________________________________________________________
3451void AliShuttle::SetShuttleLogDir(const char* logDir)
3452{
9827400b 3453 //
3454 // sets Shuttle log directory
3455 //
d386d623 3456
3457 fgkShuttleLogDir = gSystem->ExpandPathName(logDir);
3458}