]> git.uio.no Git - u/mrichter/AliRoot.git/blame - SHUTTLE/AliShuttle.cxx
git://git.uio.no / u / mrichter / AliRoot.git / blame
? search:
summary | shortlog | log | commit | commitdiff | tree
blob | blame (incremental) | history | HEAD
Furter developement on the Shuttle:
[u/mrichter/AliRoot.git] / SHUTTLE / AliShuttle.cxx
CommitLineData
73abe331 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
675f64cd 18Revision 1.61 2007/10/30 20:33:51 acolla
19Improved managing of temporary folders, which weren't correctly handled.
20Resolved bug introduced in StoreReferenceFile, which caused SPD preprocessor fail.
21
d524ade6 22Revision 1.60 2007/10/29 18:06:16 acolla
23
24New function StoreRunMetadataFile added to preprocessor and Shuttle interface
25This function can be used by GRP only. It stores raw data tags merged file to the
26raw data folder (e.g. /alice/data/2008/LHC08a/000099999/Raw).
27
28KNOWN ISSUES:
29
301. Shuttle cannot write to /alice/data/ because it belongs to alidaq. Tag file is stored in /alice/simulation/... for the time being.
312. Due to a bug in TAlien::Mkdir, the creation of a folder in recursive mode (-p option) does not work. The problem
32has been corrected in the root package on the Shuttle machine.
33
c88ad5db 34Revision 1.59 2007/10/05 12:40:55 acolla
35
36Result error code added to AliDCSClient data members (it was "lost" with the new implementation of TMap* GetAliasValues and GetDPValues).
37
1790d4b7 38Revision 1.58 2007/09/28 15:27:40 acolla
39
40AliDCSClient "multiSplit" option added in the DCS configuration
41in AliDCSMessage: variable MAX_BODY_SIZE set to 500000
42
542b6cc8 43Revision 1.57 2007/09/27 16:53:13 acolla
44Detectors can have more than one AMANDA server. SHUTTLE queries the servers sequentially,
45merges the dcs aliases/DPs in one TMap and sends it to the preprocessor.
46
1d172743 47Revision 1.56 2007/09/14 16:46:14 jgrosseo
481) Connect and Close are called before and after each query, so one can
49keep the same AliDCSClient object.
502) The splitting of a query is moved to GetDPValues/GetAliasValues.
513) Splitting interval can be specified in constructor
52
b41b252a 53Revision 1.55 2007/08/06 12:26:40 acolla
54Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
55read from the run logbook.
56
4859271b 57Revision 1.54 2007/07/12 09:51:25 jgrosseo
58removed duplicated log message in GetFile
59
4f0749a8 60Revision 1.53 2007/07/12 09:26:28 jgrosseo
61updating hlt fxs base path
62
42fde080 63Revision 1.52 2007/07/12 08:06:45 jgrosseo
64adding log messages in getfile... functions
65adding not implemented copy constructor in alishuttleconfigholder
66
1bcd28db 67Revision 1.51 2007/07/03 17:24:52 acolla
68root moved to v5-16-00. TFileMerger->Cp moved to TFile::Cp.
69
a986b218 70Revision 1.50 2007/07/02 17:19:32 acolla
71preprocessor is run in a temp directory that is removed when process is finished.
72
5bac2bde 73Revision 1.49 2007/06/29 10:45:06 acolla
74Number of columns in MySql Shuttle logbook increased by one (HLT added)
75
db99d43e 76Revision 1.48 2007/06/21 13:06:19 acolla
77GetFileSources returns dummy list with 1 source if system=DCS (better than
78returning error as it was)
79
6297b37d 80Revision 1.47 2007/06/19 17:28:56 acolla
81HLT updated; missing map bug removed.
82
dc25836b 83Revision 1.46 2007/06/09 13:01:09 jgrosseo
84Switching to retrieval of several DCS DPs at a time (multiDPrequest)
85
a038aa70 86Revision 1.45 2007/05/30 06:35:20 jgrosseo
87Adding functionality to the Shuttle/TestShuttle:
88o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
89o) Function to retrieve list of IDs for a given source (GetFileIDs)
90These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
91Example code has been added to the TestProcessor in TestShuttle
92
4a33bdd9 93Revision 1.44 2007/05/11 16:09:32 acolla
94Reference files for ITS, MUON and PHOS are now stored in OfflineDetName/OnlineDetName/run_...
95example: ITS/SPD/100_filename.root
96
2d9019b4 97Revision 1.43 2007/05/10 09:59:51 acolla
98Various bug fixes in StoreRefFilesToGrid; Cleaning of reference storage before processing detector (CleanReferenceStorage)
99
546242fb 100Revision 1.42 2007/05/03 08:01:39 jgrosseo
101typo in last commit :-(
102
8b739301 103Revision 1.41 2007/05/03 08:00:48 jgrosseo
104fixing log message when pp want to skip dcs value retrieval
105
651fdaab 106Revision 1.40 2007/04/27 07:06:48 jgrosseo
107GetFileSources returns empty list in case of no files, but successful query
108No mails sent in testmode
109
86aa42c3 110Revision 1.39 2007/04/17 12:43:57 acolla
111Correction in StoreOCDB; change of text in mail to detector expert
112
26758fce 113Revision 1.38 2007/04/12 08:26:18 jgrosseo
114updated comment
115
3c2a21c8 116Revision 1.37 2007/04/10 16:53:14 jgrosseo
117redirecting sub detector stdout, stderr to sub detector log file
118
3d8bc902 119Revision 1.35 2007/04/04 16:26:38 acolla
1201. Re-organization of function calls in TestPreprocessor to make it more meaningful.
1212. Added missing dependency in test preprocessors.
1223. in AliShuttle.cxx: processing time and memory consumption info on a single line.
123
886d60e6 124Revision 1.34 2007/04/04 10:33:36 jgrosseo
1251) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
126In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
127
1282) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
129
1303) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
131
1324) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
133
1345) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
135If you always need DCS data (like before), you do not need to implement it.
136
1376) The run type has been added to the monitoring page
138
9827400b 139Revision 1.33 2007/04/03 13:56:01 acolla
140Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
141run type.
142
3301427a 143Revision 1.32 2007/02/28 10:41:56 acolla
144Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
145AliPreprocessor::GetRunType() function.
146Added some ldap definition files.
147
d386d623 148Revision 1.30 2007/02/13 11:23:21 acolla
149Moved getters and setters of Shuttle's main OCDB/Reference, local
150OCDB/Reference, temp and log folders to AliShuttleInterface
151
9d733021 152Revision 1.27 2007/01/30 17:52:42 jgrosseo
153adding monalisa monitoring
154
e7f62f16 155Revision 1.26 2007/01/23 19:20:03 acolla
156Removed old ldif files, added TOF, MCH ldif files. Added some options in
157AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
158SetShuttleLogDir
159
36c99a6a 160Revision 1.25 2007/01/15 19:13:52 acolla
161Moved some AliInfo to AliDebug in SendMail function
162
fc5a4708 163Revision 1.21 2006/12/07 08:51:26 jgrosseo
164update (alberto):
165table, db names in ldap configuration
166added GRP preprocessor
167DCS data can also be retrieved by data point
168
2c15234c 169Revision 1.20 2006/11/16 16:16:48 jgrosseo
170introducing strict run ordering flag
171removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
172
be48e3ea 173Revision 1.19 2006/11/06 14:23:04 jgrosseo
174major update (Alberto)
175o) reading of run parameters from the logbook
176o) online offline naming conversion
177o) standalone DCSclient package
178
eba76848 179Revision 1.18 2006/10/20 15:22:59 jgrosseo
180o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
181o) Merging Collect, CollectAll, CollectNew function
182o) Removing implementation of empty copy constructors (declaration still there!)
183
cb343cfd 184Revision 1.17 2006/10/05 16:20:55 jgrosseo
185adapting to new CDB classes
186
6ec0e06c 187Revision 1.16 2006/10/05 15:46:26 jgrosseo
188applying to the new interface
189
481441a2 190Revision 1.15 2006/10/02 16:38:39 jgrosseo
191update (alberto):
192fixed memory leaks
193storing of objects that failed to be stored to the grid before
194interfacing of shuttle status table in daq system
195
2bb7b766 196Revision 1.14 2006/08/29 09:16:05 jgrosseo
197small update
198
85a80aa9 199Revision 1.13 2006/08/15 10:50:00 jgrosseo
200effc++ corrections (alberto)
201
4f0ab988 202Revision 1.12 2006/08/08 14:19:29 jgrosseo
203Update to shuttle classes (Alberto)
204
205- Possibility to set the full object's path in the Preprocessor's and
206Shuttle's Store functions
207- Possibility to extend the object's run validity in the same classes
208("startValidity" and "validityInfinite" parameters)
209- Implementation of the StoreReferenceData function to store reference
210data in a dedicated CDB storage.
211
84090f85 212Revision 1.11 2006/07/21 07:37:20 jgrosseo
213last run is stored after each run
214
7bfb2090 215Revision 1.10 2006/07/20 09:54:40 jgrosseo
216introducing status management: The processing per subdetector is divided into several steps,
217after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
218can keep track of the number of failures and skips further processing after a certain threshold is
219exceeded. These thresholds can be configured in LDAP.
220
5164a766 221Revision 1.9 2006/07/19 10:09:55 jgrosseo
222new configuration, accesst to DAQ FES (Alberto)
223
57f50b3c 224Revision 1.8 2006/07/11 12:44:36 jgrosseo
225adding parameters for extended validity range of data produced by preprocessor
226
17111222 227Revision 1.7 2006/07/10 14:37:09 jgrosseo
228small fix + todo comment
229
e090413b 230Revision 1.6 2006/07/10 13:01:41 jgrosseo
231enhanced storing of last sucessfully processed run (alberto)
232
a7160fe9 233Revision 1.5 2006/07/04 14:59:57 jgrosseo
234revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
235
45a493ce 236Revision 1.4 2006/06/12 09:11:16 jgrosseo
237coding conventions (Alberto)
238
58bc3020 239Revision 1.3 2006/06/06 14:26:40 jgrosseo
240o) removed files that were moved to STEER
241o) shuttle updated to follow the new interface (Alberto)
242
b948db8d 243Revision 1.2 2006/03/07 07:52:34 hristov
244New version (B.Yordanov)
245
d477ad88 246Revision 1.6 2005/11/19 17:19:14 byordano
247RetrieveDATEEntries and RetrieveConditionsData added
248
249Revision 1.5 2005/11/19 11:09:27 byordano
250AliShuttle declaration added
251
252Revision 1.4 2005/11/17 17:47:34 byordano
253TList changed to TObjArray
254
255Revision 1.3 2005/11/17 14:43:23 byordano
256import to local CVS
257
258Revision 1.1.1.1 2005/10/28 07:33:58 hristov
259Initial import as subdirectory in AliRoot
260
73abe331 261Revision 1.2 2005/09/13 08:41:15 byordano
262default startTime endTime added
263
264Revision 1.4 2005/08/30 09:13:02 byordano
265some docs added
266
267Revision 1.3 2005/08/29 21:15:47 byordano
268some docs added
269
270*/
271
272//
273// This class is the main manager for AliShuttle.
274// It organizes the data retrieval from DCS and call the
b948db8d 275// interface methods of AliPreprocessor.
73abe331 276// For every detector in AliShuttleConfgi (see AliShuttleConfig),
277// data for its set of aliases is retrieved. If there is registered
b948db8d 278// AliPreprocessor for this detector then it will be used
279// accroding to the schema (see AliPreprocessor).
280// If there isn't registered AliPreprocessor than the retrieved
73abe331 281// data is stored automatically to the undelying AliCDBStorage.
282// For detSpec is used the alias name.
283//
284
285#include "AliShuttle.h"
286
287#include "AliCDBManager.h"
288#include "AliCDBStorage.h"
289#include "AliCDBId.h"
84090f85 290#include "AliCDBRunRange.h"
291#include "AliCDBPath.h"
5164a766 292#include "AliCDBEntry.h"
73abe331 293#include "AliShuttleConfig.h"
eba76848 294#include "DCSClient/AliDCSClient.h"
73abe331 295#include "AliLog.h"
b948db8d 296#include "AliPreprocessor.h"
5164a766 297#include "AliShuttleStatus.h"
2bb7b766 298#include "AliShuttleLogbookEntry.h"
73abe331 299
57f50b3c 300#include <TSystem.h>
58bc3020 301#include <TObject.h>
b948db8d 302#include <TString.h>
57f50b3c 303#include <TTimeStamp.h>
73abe331 304#include <TObjString.h>
57f50b3c 305#include <TSQLServer.h>
306#include <TSQLResult.h>
307#include <TSQLRow.h>
cb343cfd 308#include <TMutex.h>
9827400b 309#include <TSystemDirectory.h>
310#include <TSystemFile.h>
a986b218 311#include <TFile.h>
9827400b 312#include <TGrid.h>
313#include <TGridResult.h>
73abe331 314
e7f62f16 315#include <TMonaLisaWriter.h>
316
5164a766 317#include <fstream>
318
cb343cfd 319#include <sys/types.h>
320#include <sys/wait.h>
321
73abe331 322ClassImp(AliShuttle)
323
b948db8d 324//______________________________________________________________________________________________
325AliShuttle::AliShuttle(const AliShuttleConfig* config,
326 UInt_t timeout, Int_t retries):
4f0ab988 327fConfig(config),
328fTimeout(timeout), fRetries(retries),
329fPreprocessorMap(),
2bb7b766 330fLogbookEntry(0),
eba76848 331fCurrentDetector(),
85a80aa9 332fStatusEntry(0),
cb343cfd 333fMonitoringMutex(0),
eba76848 334fLastActionTime(0),
e7f62f16 335fLastAction(),
9827400b 336fMonaLisa(0),
337fTestMode(kNone),
ffa29e93 338fReadTestMode(kFALSE),
339fOutputRedirected(kFALSE)
73abe331 340{
341 //
342 // config: AliShuttleConfig used
73abe331 343 // timeout: timeout used for AliDCSClient connection
344 // retries: the number of retries in case of connection error.
345 //
346
57f50b3c 347 if (!fConfig->IsValid()) AliFatal("********** !!!!! Invalid configuration !!!!! **********");
be48e3ea 348 for(int iSys=0;iSys<4;iSys++) {
57f50b3c 349 fServer[iSys]=0;
be48e3ea 350 if (iSys < 3)
2c15234c 351 fFXSlist[iSys].SetOwner(kTRUE);
57f50b3c 352 }
2bb7b766 353 fPreprocessorMap.SetOwner(kTRUE);
be48e3ea 354
355 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
356 fFirstUnprocessed[iDet] = kFALSE;
357
cb343cfd 358 fMonitoringMutex = new TMutex();
58bc3020 359}
360
b948db8d 361//______________________________________________________________________________________________
57f50b3c 362AliShuttle::~AliShuttle()
58bc3020 363{
9827400b 364 //
365 // destructor
366 //
58bc3020 367
b948db8d 368 fPreprocessorMap.DeleteAll();
be48e3ea 369 for(int iSys=0;iSys<4;iSys++)
57f50b3c 370 if(fServer[iSys]) {
371 fServer[iSys]->Close();
372 delete fServer[iSys];
eba76848 373 fServer[iSys] = 0;
57f50b3c 374 }
2bb7b766 375
376 if (fStatusEntry){
377 delete fStatusEntry;
378 fStatusEntry = 0;
379 }
cb343cfd 380
381 if (fMonitoringMutex)
382 {
383 delete fMonitoringMutex;
384 fMonitoringMutex = 0;
385 }
73abe331 386}
387
b948db8d 388//______________________________________________________________________________________________
57f50b3c 389void AliShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
58bc3020 390{
73abe331 391 //
b948db8d 392 // Registers new AliPreprocessor.
73abe331 393 // It uses GetName() for indentificator of the pre processor.
394 // The pre processor is registered it there isn't any other
395 // with the same identificator (GetName()).
396 //
397
eba76848 398 const char* detName = preprocessor->GetName();
399 if(GetDetPos(detName) < 0)
400 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
401
402 if (fPreprocessorMap.GetValue(detName)) {
403 AliWarning(Form("AliPreprocessor %s is already registered!", detName));
73abe331 404 return;
405 }
406
eba76848 407 fPreprocessorMap.Add(new TObjString(detName), preprocessor);
73abe331 408}
b948db8d 409//______________________________________________________________________________________________
3301427a 410Bool_t AliShuttle::Store(const AliCDBPath& path, TObject* object,
84090f85 411 AliCDBMetaData* metaData, Int_t validityStart, Bool_t validityInfinite)
73abe331 412{
9827400b 413 // Stores a CDB object in the storage for offline reconstruction. Objects that are not needed for
414 // offline reconstruction, but should be stored anyway (e.g. for debugging) should NOT be stored
415 // using this function. Use StoreReferenceData instead!
416 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
417 // finishes the data are transferred to the main storage (Grid).
b948db8d 418
3301427a 419 return StoreLocally(fgkLocalCDB, path, object, metaData, validityStart, validityInfinite);
84090f85 420}
421
422//______________________________________________________________________________________________
3301427a 423Bool_t AliShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 424{
9827400b 425 // Stores a CDB object in the storage for reference data. This objects will not be available during
426 // offline reconstrunction. Use this function for reference data only!
427 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
428 // finishes the data are transferred to the main storage (Grid).
85a80aa9 429
3301427a 430 return StoreLocally(fgkLocalRefStorage, path, object, metaData);
85a80aa9 431}
432
433//______________________________________________________________________________________________
3301427a 434Bool_t AliShuttle::StoreLocally(const TString& localUri,
85a80aa9 435 const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
436 Int_t validityStart, Bool_t validityInfinite)
437{
9827400b 438 // Store object temporarily in local storage. Parameters are passed by Store and StoreReferenceData functions.
439 // when the preprocessor finishes the data are transferred to the main storage (Grid).
440 // The parameters are:
441 // 1) Uri of the backup storage (Local)
442 // 2) the object's path.
443 // 3) the object to be stored
444 // 4) the metaData to be associated with the object
445 // 5) the validity start run number w.r.t. the current run,
446 // if the data is valid only for this run leave the default 0
447 // 6) specifies if the calibration data is valid for infinity (this means until updated),
448 // typical for calibration runs, the default is kFALSE
449 //
450 // returns 0 if fail, 1 otherwise
84090f85 451
9827400b 452 if (fTestMode & kErrorStorage)
453 {
454 Log(fCurrentDetector, "StoreLocally - In TESTMODE - Simulating error while storing locally");
455 return kFALSE;
456 }
457
3301427a 458 const char* cdbType = (localUri == fgkLocalCDB) ? "CDB" : "Reference";
2bb7b766 459
85a80aa9 460 Int_t firstRun = GetCurrentRun() - validityStart;
84090f85 461 if(firstRun < 0) {
9827400b 462 AliWarning("First valid run happens to be less than 0! Setting it to 0.");
84090f85 463 firstRun=0;
464 }
465
466 Int_t lastRun = -1;
467 if(validityInfinite) {
468 lastRun = AliCDBRunRange::Infinity();
469 } else {
470 lastRun = GetCurrentRun();
471 }
472
3301427a 473 // Version is set to current run, it will be used later to transfer data to Grid
474 AliCDBId id(path, firstRun, lastRun, GetCurrentRun(), -1);
2bb7b766 475
476 if(! dynamic_cast<TObjString*> (metaData->GetProperty("RunUsed(TObjString)"))){
477 TObjString runUsed = Form("%d", GetCurrentRun());
9e080f92 478 metaData->SetProperty("RunUsed(TObjString)", runUsed.Clone());
2bb7b766 479 }
84090f85 480
3301427a 481 Bool_t result = kFALSE;
84090f85 482
3301427a 483 if (!(AliCDBManager::Instance()->GetStorage(localUri))) {
484 Log("SHUTTLE", Form("StoreLocally - Cannot activate local %s storage", cdbType));
84090f85 485 } else {
3301427a 486 result = AliCDBManager::Instance()->GetStorage(localUri)
84090f85 487 ->Put(object, id, metaData);
488 }
489
490 if(!result) {
491
9827400b 492 Log(fCurrentDetector, Form("StoreLocally - Can't store object <%s>!", id.ToString().Data()));
3301427a 493 }
2bb7b766 494
3301427a 495 return result;
496}
84090f85 497
3301427a 498//______________________________________________________________________________________________
499Bool_t AliShuttle::StoreOCDB()
500{
9827400b 501 //
502 // Called when preprocessor ends successfully or when previous storage attempt failed (kStoreError status)
503 // Calls underlying StoreOCDB(const char*) function twice, for OCDB and Reference storage.
504 // Then calls StoreRefFilesToGrid to store reference files.
505 //
506
507 if (fTestMode & kErrorGrid)
508 {
509 Log("SHUTTLE", "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
510 Log(fCurrentDetector, "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
511 return kFALSE;
512 }
513
c88ad5db 514 Log("SHUTTLE","StoreOCDB - Storing OCDB data ...");
86aa42c3 515 Bool_t resultCDB = StoreOCDB(fgkMainCDB);
516
c88ad5db 517 Log("SHUTTLE","StoreOCDB - Storing reference data ...");
3301427a 518 Bool_t resultRef = StoreOCDB(fgkMainRefStorage);
9827400b 519
c88ad5db 520 Log("SHUTTLE","StoreOCDB - Storing reference files ...");
521 Bool_t resultRefFiles = CopyFilesToGrid("reference");
522
523 Bool_t resultMetadata = kTRUE;
524 if(fCurrentDetector == "GRP")
525 {
526 Log("StoreOCDB - SHUTTLE","Storing Run Metadata file ...");
527 resultMetadata = CopyFilesToGrid("metadata");
528 }
9827400b 529
c88ad5db 530 return resultCDB && resultRef && resultRefFiles && resultMetadata;
3301427a 531}
532
533//______________________________________________________________________________________________
534Bool_t AliShuttle::StoreOCDB(const TString& gridURI)
535{
536 //
537 // Called by StoreOCDB(), performs actual storage to the main OCDB and reference storages (Grid)
538 //
539
540 TObjArray* gridIds=0;
541
542 Bool_t result = kTRUE;
543
544 const char* type = 0;
545 TString localURI;
546 if(gridURI == fgkMainCDB) {
547 type = "OCDB";
548 localURI = fgkLocalCDB;
549 } else if(gridURI == fgkMainRefStorage) {
550 type = "reference";
551 localURI = fgkLocalRefStorage;
552 } else {
553 AliError(Form("Invalid storage URI: %s", gridURI.Data()));
554 return kFALSE;
555 }
556
557 AliCDBManager* man = AliCDBManager::Instance();
558
559 AliCDBStorage *gridSto = man->GetStorage(gridURI);
560 if(!gridSto) {
561 Log("SHUTTLE",
562 Form("StoreOCDB - cannot activate main %s storage", type));
563 return kFALSE;
564 }
565
566 gridIds = gridSto->GetQueryCDBList();
567
568 // get objects previously stored in local CDB
569 AliCDBStorage *localSto = man->GetStorage(localURI);
570 if(!localSto) {
571 Log("SHUTTLE",
572 Form("StoreOCDB - cannot activate local %s storage", type));
573 return kFALSE;
574 }
575 AliCDBPath aPath(GetOfflineDetName(fCurrentDetector.Data()),"*","*");
576 // Local objects were stored with current run as Grid version!
577 TList* localEntries = localSto->GetAll(aPath.GetPath(), GetCurrentRun(), GetCurrentRun());
578 localEntries->SetOwner(1);
579
580 // loop on local stored objects
581 TIter localIter(localEntries);
582 AliCDBEntry *aLocEntry = 0;
583 while((aLocEntry = dynamic_cast<AliCDBEntry*> (localIter.Next()))){
584 aLocEntry->SetOwner(1);
585 AliCDBId aLocId = aLocEntry->GetId();
586 aLocEntry->SetVersion(-1);
587 aLocEntry->SetSubVersion(-1);
588
589 // If local object is valid up to infinity we store it only if it is
590 // the first unprocessed run!
591 if (aLocId.GetLastRun() == AliCDBRunRange::Infinity() &&
592 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
593 {
594 Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
595 "there are previous unprocessed runs!",
596 fCurrentDetector.Data(), aLocId.GetPath().Data()));
597 continue;
598 }
599
600 // loop on Grid valid Id's
601 Bool_t store = kTRUE;
602 TIter gridIter(gridIds);
603 AliCDBId* aGridId = 0;
604 while((aGridId = dynamic_cast<AliCDBId*> (gridIter.Next()))){
605 if(aGridId->GetPath() != aLocId.GetPath()) continue;
606 // skip all objects valid up to infinity
607 if(aGridId->GetLastRun() == AliCDBRunRange::Infinity()) continue;
608 // if we get here, it means there's already some more recent object stored on Grid!
609 store = kFALSE;
610 break;
611 }
612
613 // If we get here, the file can be stored!
614 Bool_t storeOk = gridSto->Put(aLocEntry);
615 if(!store || storeOk){
616
617 if (!store)
618 {
619 Log(fCurrentDetector.Data(),
620 Form("StoreOCDB - A more recent object already exists in %s storage: <%s>",
621 type, aGridId->ToString().Data()));
622 } else {
623 Log("SHUTTLE",
624 Form("StoreOCDB - Object <%s> successfully put into %s storage",
625 aLocId.ToString().Data(), type));
2d9019b4 626 Log(fCurrentDetector.Data(),
627 Form("StoreOCDB - Object <%s> successfully put into %s storage",
628 aLocId.ToString().Data(), type));
3301427a 629 }
84090f85 630
3301427a 631 // removing local filename...
632 TString filename;
633 localSto->IdToFilename(aLocId, filename);
c88ad5db 634 Log("SHUTTLE", Form("StoreOCDB - Removing local file %s", filename.Data()));
3301427a 635 RemoveFile(filename.Data());
636 continue;
637 } else {
638 Log("SHUTTLE",
639 Form("StoreOCDB - Grid %s storage of object <%s> failed",
640 type, aLocId.ToString().Data()));
2d9019b4 641 Log(fCurrentDetector.Data(),
642 Form("StoreOCDB - Grid %s storage of object <%s> failed",
643 type, aLocId.ToString().Data()));
3301427a 644 result = kFALSE;
b948db8d 645 }
646 }
3301427a 647 localEntries->Clear();
2bb7b766 648
b948db8d 649 return result;
3301427a 650}
651
546242fb 652//______________________________________________________________________________________________
653Bool_t AliShuttle::CleanReferenceStorage(const char* detector)
654{
2d9019b4 655 // clears the directory used to store reference files of a given subdetector
546242fb 656
657 AliCDBManager* man = AliCDBManager::Instance();
658 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
2d9019b4 659 TString localBaseFolder = sto->GetBaseFolder();
660
661 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
662
d524ade6 663 Log("SHUTTLE", Form("CleanReferenceStorage - Cleaning %s", targetDir.Data()));
2d9019b4 664
665 TString begin;
666 begin.Form("%d_", GetCurrentRun());
667
668 TSystemDirectory* baseDir = new TSystemDirectory("/", targetDir);
669 if (!baseDir)
670 return kTRUE;
671
672 TList* dirList = baseDir->GetListOfFiles();
673 delete baseDir;
674
675 if (!dirList) return kTRUE;
676
677 if (dirList->GetEntries() < 3)
678 {
679 delete dirList;
680 return kTRUE;
681 }
682
683 Int_t nDirs = 0, nDel = 0;
684 TIter dirIter(dirList);
685 TSystemFile* entry = 0;
546242fb 686
2d9019b4 687 Bool_t success = kTRUE;
546242fb 688
2d9019b4 689 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
690 {
691 if (entry->IsDirectory())
692 continue;
693
694 TString fileName(entry->GetName());
695 if (!fileName.BeginsWith(begin))
696 continue;
697
698 nDirs++;
699
700 // delete file
701 Int_t result = gSystem->Unlink(fileName.Data());
702
703 if (result)
704 {
d524ade6 705 Log("SHUTTLE", Form("CleanReferenceStorage - Could not delete file %s!", fileName.Data()));
2d9019b4 706 success = kFALSE;
707 } else {
708 nDel++;
709 }
710 }
711
712 if(nDirs > 0)
713 Log("SHUTTLE", Form("CleanReferenceStorage - %d (over %d) reference files in folder %s were deleted.",
714 nDel, nDirs, targetDir.Data()));
715
716
717 delete dirList;
718 return success;
719
720
721
722
723
546242fb 724
725 Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
726 if (result == 0)
727 {
728 // delete directory
d524ade6 729 result = gSystem->Exec(Form("rm -rf %s", targetDir.Data()));
546242fb 730 if (result != 0)
731 {
d524ade6 732 Log("SHUTTLE", Form("CleanReferenceStorage - Could not clean directory %s", targetDir.Data()));
546242fb 733 return kFALSE;
734 }
735 }
736
737 result = gSystem->mkdir(targetDir, kTRUE);
738 if (result != 0)
739 {
c88ad5db 740 Log("SHUTTLE", Form("CleanReferenceStorage - Error creating base directory %s", targetDir.Data()));
546242fb 741 return kFALSE;
742 }
743
744 return kTRUE;
745}
746
9827400b 747//______________________________________________________________________________________________
748Bool_t AliShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
749{
750 //
3c2a21c8 751 // Stores reference file directly (without opening it). This function stores the file locally.
9827400b 752 //
3c2a21c8 753 // The file is stored under the following location:
754 // <base folder of local reference storage>/<DET>/<RUN#>_<gridFileName>
755 // where <gridFileName> is the second parameter given to the function
756 //
9827400b 757
758 if (fTestMode & kErrorStorage)
759 {
760 Log(fCurrentDetector, "StoreReferenceFile - In TESTMODE - Simulating error while storing locally");
761 return kFALSE;
762 }
763
764 AliCDBManager* man = AliCDBManager::Instance();
765 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
766
767 TString localBaseFolder = sto->GetBaseFolder();
768
d524ade6 769 TString target = GetRefFilePrefix(localBaseFolder.Data(), detector);
770 target.Append(Form("/%d_%s", GetCurrentRun(), gridFileName));
9827400b 771
d524ade6 772 return CopyFileLocally(localFile, target);
c88ad5db 773}
774
775//______________________________________________________________________________________________
776Bool_t AliShuttle::StoreRunMetadataFile(const char* localFile, const char* gridFileName)
777{
778 //
779 // Stores Run metadata file to the Grid, in the run folder
780 //
781 // Only GRP can call this function.
782
783 if (fTestMode & kErrorStorage)
784 {
785 Log(fCurrentDetector, "StoreRunMetaDataFile - In TESTMODE - Simulating error while storing locally");
786 return kFALSE;
787 }
788
789 AliCDBManager* man = AliCDBManager::Instance();
790 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
791
792 TString localBaseFolder = sto->GetBaseFolder();
793
794 // Build Run level folder
795 // folder = /alice/data/year/lhcPeriod/runNb/Raw
796
c88ad5db 797
675f64cd 798 TString lhcPeriod = GetLHCPeriod();
c88ad5db 799 if (lhcPeriod.Length() == 0)
800 {
801 Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
802 return 0;
803 }
804
675f64cd 805 TString target = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw/%s",
806 localBaseFolder.Data(), GetCurrentYear(),
d524ade6 807 lhcPeriod.Data(), GetCurrentRun(), gridFileName);
c88ad5db 808
d524ade6 809 return CopyFileLocally(localFile, target);
c88ad5db 810}
811
812//______________________________________________________________________________________________
d524ade6 813Bool_t AliShuttle::CopyFileLocally(const char* localFile, const TString& target)
c88ad5db 814{
815 //
816 // Stores file locally. Called by StoreReferenceFile and StoreRunMetadataFile
d524ade6 817 // Files are temporarily stored in the local reference storage. When the preprocessor
818 // finishes, the Shuttle calls CopyFilesToGrid to transfer the files to AliEn
819 // (in reference or run level folders)
c88ad5db 820 //
821
d524ade6 822 TString targetDir(target(0, target.Last('/')));
823
824 //try to open base dir folder, if it does not exist
2d9019b4 825 void* dir = gSystem->OpenDirectory(targetDir.Data());
826 if (dir == NULL) {
827 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
c88ad5db 828 Log("SHUTTLE", Form("StoreFileLocally - Can't open directory <%s>", targetDir.Data()));
2d9019b4 829 return kFALSE;
830 }
831
832 } else {
833 gSystem->FreeDirectory(dir);
834 }
9827400b 835
546242fb 836 Int_t result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
9827400b 837 if (result)
838 {
c88ad5db 839 Log("SHUTTLE", Form("StoreFileLocally - %s does not exist", localFile));
546242fb 840 return kFALSE;
9827400b 841 }
546242fb 842
9827400b 843 result = gSystem->CopyFile(localFile, target);
844
845 if (result == 0)
846 {
c88ad5db 847 Log("SHUTTLE", Form("StoreFileLocally - File %s stored locally to %s", localFile, target.Data()));
9827400b 848 return kTRUE;
849 }
850 else
851 {
d524ade6 852 Log("SHUTTLE", Form("StoreFileLocally - Could not store file %s to %s! Error code = %d",
546242fb 853 localFile, target.Data(), result));
9827400b 854 return kFALSE;
855 }
c88ad5db 856
857
858
9827400b 859}
860
861//______________________________________________________________________________________________
c88ad5db 862Bool_t AliShuttle::CopyFilesToGrid(const char* type)
9827400b 863{
864 //
c88ad5db 865 // Transfers local files to the Grid. Local files can be reference files
866 // or run metadata file (from GRP only).
9827400b 867 //
c88ad5db 868 // According to the type (ref, metadata) the files are stored under the following location:
869 // ref --> <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
870 // metadata --> <run data folder>/<MetadataFileName>
86aa42c3 871 //
c88ad5db 872
9827400b 873 AliCDBManager* man = AliCDBManager::Instance();
874 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
875 if (!sto)
876 return kFALSE;
877 TString localBaseFolder = sto->GetBaseFolder();
9827400b 878
c88ad5db 879 TString dir;
880 TString alienDir;
9827400b 881 TString begin;
9827400b 882
c88ad5db 883 if (strcmp(type, "reference") == 0)
884 {
885 dir = GetRefFilePrefix(localBaseFolder.Data(), fCurrentDetector.Data());
886 AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
887 if (!gridSto)
888 return kFALSE;
889 TString gridBaseFolder = gridSto->GetBaseFolder();
890 alienDir = GetRefFilePrefix(gridBaseFolder.Data(), fCurrentDetector.Data());
891 begin = Form("%d_", GetCurrentRun());
892 }
893 else if (strcmp(type, "metadata") == 0)
894 {
c88ad5db 895
675f64cd 896 TString lhcPeriod = GetLHCPeriod();
c88ad5db 897
898 if (lhcPeriod.Length() == 0)
899 {
900 Log("SHUTTLE","CopyFilesToGrid - LHCPeriod not found in logbook!");
901 return 0;
902 }
903
675f64cd 904 dir = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw",
905 localBaseFolder.Data(), GetCurrentYear(),
c88ad5db 906 lhcPeriod.Data(), GetCurrentRun());
675f64cd 907 alienDir = dir(dir.Index("/alice/data/"), dir.Length());
908
c88ad5db 909 begin = "";
910 }
911 else
912 {
913 Log("SHUTTLE", "CopyFilesToGrid - Unexpected: type label must be reference or metadata!");
914 return kFALSE;
915 }
916
9827400b 917 TSystemDirectory* baseDir = new TSystemDirectory("/", dir);
3d8bc902 918 if (!baseDir)
919 return kTRUE;
920
2d9019b4 921 TList* dirList = baseDir->GetListOfFiles();
922 delete baseDir;
923
924 if (!dirList) return kTRUE;
925
926 if (dirList->GetEntries() < 3)
3d8bc902 927 {
2d9019b4 928 delete dirList;
9827400b 929 return kTRUE;
3d8bc902 930 }
2d9019b4 931
546242fb 932 if (!gGrid)
933 {
c88ad5db 934 Log("SHUTTLE", "CopyFilesToGrid - Connection to Grid failed: Cannot continue!");
2d9019b4 935 delete dirList;
546242fb 936 return kFALSE;
937 }
938
2d9019b4 939 Int_t nDirs = 0, nTransfer = 0;
940 TIter dirIter(dirList);
941 TSystemFile* entry = 0;
942
9827400b 943 Bool_t success = kTRUE;
3d8bc902 944 Bool_t first = kTRUE;
9827400b 945
2d9019b4 946 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
947 {
9827400b 948 if (entry->IsDirectory())
949 continue;
950
951 TString fileName(entry->GetName());
952 if (!fileName.BeginsWith(begin))
953 continue;
954
2d9019b4 955 nDirs++;
956
3d8bc902 957 if (first)
958 {
959 first = kFALSE;
c88ad5db 960 // check that folder exists, otherwise create it
3d8bc902 961 TGridResult* result = gGrid->Ls(alienDir.Data(), "a");
962
963 if (!result)
2d9019b4 964 {
965 delete dirList;
3d8bc902 966 return kFALSE;
2d9019b4 967 }
3d8bc902 968
546242fb 969 if (!result->GetFileName(1)) // TODO: It looks like element 0 is always 0!!
3d8bc902 970 {
675f64cd 971 // TODO It does not work currently! Bug in TAliEn::Mkdir
972 // TODO Manually fixed in local root v5-16-00
c88ad5db 973 if (!gGrid->Mkdir(alienDir.Data(),"-p",0))
3d8bc902 974 {
c88ad5db 975 Log("SHUTTLE", Form("CopyFilesToGrid - Cannot create directory %s",
3d8bc902 976 alienDir.Data()));
2d9019b4 977 delete dirList;
3d8bc902 978 return kFALSE;
546242fb 979 } else {
c88ad5db 980 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s created", alienDir.Data()));
3d8bc902 981 }
982
546242fb 983 } else {
c88ad5db 984 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s found", alienDir.Data()));
3d8bc902 985 }
986 }
987
9827400b 988 TString fullLocalPath;
989 fullLocalPath.Form("%s/%s", dir.Data(), fileName.Data());
990
991 TString fullGridPath;
992 fullGridPath.Form("alien://%s/%s", alienDir.Data(), fileName.Data());
993
a986b218 994 Bool_t result = TFile::Cp(fullLocalPath, fullGridPath);
9827400b 995
996 if (result)
997 {
c88ad5db 998 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s succeeded!",
999 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1000 RemoveFile(fullLocalPath);
2d9019b4 1001 nTransfer++;
9827400b 1002 }
1003 else
1004 {
c88ad5db 1005 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s FAILED!",
1006 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1007 success = kFALSE;
1008 }
1009 }
2d9019b4 1010
c88ad5db 1011 Log("SHUTTLE", Form("CopyFilesToGrid - %d (over %d) files in folder %s copied to Grid.",
1012 nTransfer, nDirs, dir.Data()));
2d9019b4 1013
1014
1015 delete dirList;
9827400b 1016 return success;
1017}
1018
2d9019b4 1019//______________________________________________________________________________________________
1020const char* AliShuttle::GetRefFilePrefix(const char* base, const char* detector)
1021{
1022 //
1023 // Get folder name of reference files
1024 //
1025
1026 TString offDetStr(GetOfflineDetName(detector));
1027 TString dir;
1028 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
1029 {
1030 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
1031 } else {
1032 dir.Form("%s/%s", base, offDetStr.Data());
1033 }
1034
1035 return dir.Data();
1036
1037
1038}
c88ad5db 1039
3301427a 1040//______________________________________________________________________________________________
1041void AliShuttle::CleanLocalStorage(const TString& uri)
1042{
9827400b 1043 //
1044 // Called in case the preprocessor is declared failed. Remove remaining objects from the local storages.
1045 //
3301427a 1046
1047 const char* type = 0;
1048 if(uri == fgkLocalCDB) {
1049 type = "OCDB";
1050 } else if(uri == fgkLocalRefStorage) {
546242fb 1051 type = "Reference";
3301427a 1052 } else {
1053 AliError(Form("Invalid storage URI: %s", uri.Data()));
1054 return;
1055 }
1056
1057 AliCDBManager* man = AliCDBManager::Instance();
b948db8d 1058
3301427a 1059 // open local storage
1060 AliCDBStorage *localSto = man->GetStorage(uri);
1061 if(!localSto) {
1062 Log("SHUTTLE",
1063 Form("CleanLocalStorage - cannot activate local %s storage", type));
1064 return;
1065 }
1066
1067 TString filename(Form("%s/%s/*/Run*_v%d_s*.root",
546242fb 1068 localSto->GetBaseFolder().Data(), GetOfflineDetName(fCurrentDetector.Data()), GetCurrentRun()));
3301427a 1069
c88ad5db 1070 AliDebug(2, Form("filename = %s", filename.Data()));
3301427a 1071
c88ad5db 1072 Log("SHUTTLE", Form("Removing remaining local files for run %d and detector %s ...",
3301427a 1073 GetCurrentRun(), fCurrentDetector.Data()));
1074
1075 RemoveFile(filename.Data());
1076
1077}
1078
1079//______________________________________________________________________________________________
1080void AliShuttle::RemoveFile(const char* filename)
1081{
9827400b 1082 //
1083 // removes local file
1084 //
3301427a 1085
1086 TString command(Form("rm -f %s", filename));
1087
1088 Int_t result = gSystem->Exec(command.Data());
1089 if(result != 0)
1090 {
1091 Log("SHUTTLE", Form("RemoveFile - %s: Cannot remove file %s!",
1092 fCurrentDetector.Data(), filename));
1093 }
73abe331 1094}
1095
b948db8d 1096//______________________________________________________________________________________________
5164a766 1097AliShuttleStatus* AliShuttle::ReadShuttleStatus()
1098{
9827400b 1099 //
1100 // Reads the AliShuttleStatus from the CDB
1101 //
5164a766 1102
2bb7b766 1103 if (fStatusEntry){
1104 delete fStatusEntry;
1105 fStatusEntry = 0;
1106 }
5164a766 1107
10a5a932 1108 fStatusEntry = AliCDBManager::Instance()->GetStorage(GetLocalCDB())
2bb7b766 1109 ->Get(Form("/SHUTTLE/STATUS/%s", fCurrentDetector.Data()), GetCurrentRun());
5164a766 1110
2bb7b766 1111 if (!fStatusEntry) return 0;
1112 fStatusEntry->SetOwner(1);
5164a766 1113
2bb7b766 1114 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1115 if (!status) {
1116 AliError("Invalid object stored to CDB!");
1117 return 0;
1118 }
5164a766 1119
2bb7b766 1120 return status;
5164a766 1121}
1122
1123//______________________________________________________________________________________________
7bfb2090 1124Bool_t AliShuttle::WriteShuttleStatus(AliShuttleStatus* status)
5164a766 1125{
9827400b 1126 //
1127 // writes the status for one subdetector
1128 //
2bb7b766 1129
1130 if (fStatusEntry){
1131 delete fStatusEntry;
1132 fStatusEntry = 0;
1133 }
5164a766 1134
2bb7b766 1135 Int_t run = GetCurrentRun();
5164a766 1136
2bb7b766 1137 AliCDBId id(AliCDBPath("SHUTTLE", "STATUS", fCurrentDetector), run, run);
5164a766 1138
2bb7b766 1139 fStatusEntry = new AliCDBEntry(status, id, new AliCDBMetaData);
1140 fStatusEntry->SetOwner(1);
5164a766 1141
2bb7b766 1142 UInt_t result = AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
7bfb2090 1143
2bb7b766 1144 if (!result) {
3301427a 1145 Log("SHUTTLE", Form("WriteShuttleStatus - Failed for %s, run %d",
1146 fCurrentDetector.Data(), run));
2bb7b766 1147 return kFALSE;
1148 }
e7f62f16 1149
1150 SendMLInfo();
7bfb2090 1151
2bb7b766 1152 return kTRUE;
5164a766 1153}
1154
1155//______________________________________________________________________________________________
1156void AliShuttle::UpdateShuttleStatus(AliShuttleStatus::Status newStatus, Bool_t increaseCount)
1157{
9827400b 1158 //
1159 // changes the AliShuttleStatus for the given detector and run to the given status
1160 //
5164a766 1161
2bb7b766 1162 if (!fStatusEntry){
1163 AliError("UNEXPECTED: fStatusEntry empty");
1164 return;
1165 }
5164a766 1166
2bb7b766 1167 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
5164a766 1168
2bb7b766 1169 if (!status){
c88ad5db 1170 Log("SHUTTLE", "UpdateShuttleStatus - UNEXPECTED: status could not be read from current CDB entry");
2bb7b766 1171 return;
1172 }
5164a766 1173
2c15234c 1174 TString actionStr = Form("UpdateShuttleStatus - %s: Changing state from %s to %s",
eba76848 1175 fCurrentDetector.Data(),
36c99a6a 1176 status->GetStatusName(),
eba76848 1177 status->GetStatusName(newStatus));
cb343cfd 1178 Log("SHUTTLE", actionStr);
1179 SetLastAction(actionStr);
5164a766 1180
2bb7b766 1181 status->SetStatus(newStatus);
1182 if (increaseCount) status->IncreaseCount();
5164a766 1183
2bb7b766 1184 AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
e7f62f16 1185
1186 SendMLInfo();
5164a766 1187}
e7f62f16 1188
1189//______________________________________________________________________________________________
1190void AliShuttle::SendMLInfo()
1191{
1192 //
1193 // sends ML information about the current status of the current detector being processed
1194 //
1195
1196 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1197
1198 if (!status){
3301427a 1199 Log("SHUTTLE", "SendMLInfo - UNEXPECTED: status could not be read from current CDB entry");
e7f62f16 1200 return;
1201 }
1202
1203 TMonaLisaText mlStatus(Form("%s_status", fCurrentDetector.Data()), status->GetStatusName());
1204 TMonaLisaValue mlRetryCount(Form("%s_count", fCurrentDetector.Data()), status->GetCount());
1205
1206 TList mlList;
1207 mlList.Add(&mlStatus);
1208 mlList.Add(&mlRetryCount);
1209
1210 fMonaLisa->SendParameters(&mlList);
1211}
1212
5164a766 1213//______________________________________________________________________________________________
1214Bool_t AliShuttle::ContinueProcessing()
1215{
9827400b 1216 // this function reads the AliShuttleStatus information from CDB and
1217 // checks if the processing should be continued
1218 // if yes it returns kTRUE and updates the AliShuttleStatus with nextStatus
2bb7b766 1219
57c1a579 1220 if (!fConfig->HostProcessDetector(fCurrentDetector)) return kFALSE;
1221
1222 AliPreprocessor* aPreprocessor =
1223 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1224 if (!aPreprocessor)
1225 {
c88ad5db 1226 Log("SHUTTLE", Form("ContinueProcessing - %s: no preprocessor registered", fCurrentDetector.Data()));
57c1a579 1227 return kFALSE;
1228 }
1229
2bb7b766 1230 AliShuttleLogbookEntry::Status entryStatus =
eba76848 1231 fLogbookEntry->GetDetectorStatus(fCurrentDetector);
2bb7b766 1232
1233 if(entryStatus != AliShuttleLogbookEntry::kUnprocessed) {
c88ad5db 1234 Log("SHUTTLE", Form("ContinueProcessing - %s is %s",
2bb7b766 1235 fCurrentDetector.Data(),
1236 fLogbookEntry->GetDetectorStatusName(entryStatus)));
1237 return kFALSE;
1238 }
1239
1240 // if we get here, according to Shuttle logbook subdetector is in UNPROCESSED state
be48e3ea 1241
1242 // check if current run is first unprocessed run for current detector
1243 if (fConfig->StrictRunOrder(fCurrentDetector) &&
1244 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
1245 {
86aa42c3 1246 if (fTestMode == kNone)
1247 {
c88ad5db 1248 Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering"
1249 " but this is not the first unprocessed run!"));
86aa42c3 1250 return kFALSE;
1251 }
1252 else
1253 {
c88ad5db 1254 Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - "
1255 "Although %s requires strict run ordering "
1256 "and this is not the first unprocessed run, "
1257 "the SHUTTLE continues"));
86aa42c3 1258 }
be48e3ea 1259 }
1260
2bb7b766 1261 AliShuttleStatus* status = ReadShuttleStatus();
1262 if (!status) {
1263 // first time
1264 Log("SHUTTLE", Form("ContinueProcessing - %s: Processing first time",
1265 fCurrentDetector.Data()));
1266 status = new AliShuttleStatus(AliShuttleStatus::kStarted);
1267 return WriteShuttleStatus(status);
1268 }
1269
1270 // The following two cases shouldn't happen if Shuttle Logbook was correctly updated.
1271 // If it happens it may mean Logbook updating failed... let's do it now!
1272 if (status->GetStatus() == AliShuttleStatus::kDone ||
1273 status->GetStatus() == AliShuttleStatus::kFailed){
1274 Log("SHUTTLE", Form("ContinueProcessing - %s is already %s. Updating Shuttle Logbook",
1275 fCurrentDetector.Data(),
1276 status->GetStatusName(status->GetStatus())));
1277 UpdateShuttleLogbook(fCurrentDetector.Data(),
1278 status->GetStatusName(status->GetStatus()));
1279 return kFALSE;
1280 }
1281
3301427a 1282 if (status->GetStatus() == AliShuttleStatus::kStoreError) {
2bb7b766 1283 Log("SHUTTLE",
c88ad5db 1284 Form("ContinueProcessing - %s: Grid storage of one or more "
1285 "objects failed. Trying again now",
2bb7b766 1286 fCurrentDetector.Data()));
9827400b 1287 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1288 if (StoreOCDB()){
c88ad5db 1289 Log("SHUTTLE", Form("ContinueProcessing - %s: all objects "
1290 "successfully stored into main storage",
3301427a 1291 fCurrentDetector.Data()));
2bb7b766 1292 UpdateShuttleStatus(AliShuttleStatus::kDone);
1293 UpdateShuttleLogbook(fCurrentDetector.Data(), "DONE");
1294 } else {
1295 Log("SHUTTLE",
1296 Form("ContinueProcessing - %s: Grid storage failed again",
1297 fCurrentDetector.Data()));
9827400b 1298 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
2bb7b766 1299 }
1300 return kFALSE;
1301 }
1302
1303 // if we get here, there is a restart
57c1a579 1304 Bool_t cont = kFALSE;
2bb7b766 1305
1306 // abort conditions
cb343cfd 1307 if (status->GetCount() >= fConfig->GetMaxRetries()) {
57c1a579 1308 Log("SHUTTLE", Form("ContinueProcessing - %s failed %d times in status %s - "
1309 "Updating Shuttle Logbook", fCurrentDetector.Data(),
2bb7b766 1310 status->GetCount(), status->GetStatusName()));
1311 UpdateShuttleLogbook(fCurrentDetector.Data(), "FAILED");
e7f62f16 1312 UpdateShuttleStatus(AliShuttleStatus::kFailed);
3301427a 1313
1314 // there may still be objects in local OCDB and reference storage
1315 // and FXS databases may be not updated: do it now!
9827400b 1316
1317 // TODO Currently disabled, we want to keep files in case of failure!
1318 // CleanLocalStorage(fgkLocalCDB);
1319 // CleanLocalStorage(fgkLocalRefStorage);
1320 // UpdateTableFailCase();
1321
1322 // Send mail to detector expert!
c88ad5db 1323 Log("SHUTTLE", Form("ContinueProcessing - Sending mail to %s expert...",
1324 fCurrentDetector.Data()));
9827400b 1325 if (!SendMail())
1326 Log("SHUTTLE", Form("ContinueProcessing - Could not send mail to %s expert",
1327 fCurrentDetector.Data()));
3301427a 1328
57c1a579 1329 } else {
1330 Log("SHUTTLE", Form("ContinueProcessing - %s: restarting. "
1331 "Aborted before with %s. Retry number %d.", fCurrentDetector.Data(),
1332 status->GetStatusName(), status->GetCount()));
9827400b 1333 Bool_t increaseCount = kTRUE;
c88ad5db 1334 if (status->GetStatus() == AliShuttleStatus::kDCSError ||
1335 status->GetStatus() == AliShuttleStatus::kDCSStarted)
1336 increaseCount = kFALSE;
675f64cd 1337
9827400b 1338 UpdateShuttleStatus(AliShuttleStatus::kStarted, increaseCount);
57c1a579 1339 cont = kTRUE;
2bb7b766 1340 }
1341
57c1a579 1342 return cont;
5164a766 1343}
1344
1345//______________________________________________________________________________________________
2bb7b766 1346Bool_t AliShuttle::Process(AliShuttleLogbookEntry* entry)
58bc3020 1347{
73abe331 1348 //
b948db8d 1349 // Makes data retrieval for all detectors in the configuration.
2bb7b766 1350 // entry: Shuttle logbook entry, contains run paramenters and status of detectors
1351 // (Unprocessed, Inactive, Failed or Done).
d477ad88 1352 // Returns kFALSE in case of error occured and kTRUE otherwise
73abe331 1353 //
1354
9827400b 1355 if (!entry) return kFALSE;
2bb7b766 1356
1357 fLogbookEntry = entry;
1358
c88ad5db 1359 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^*",
9827400b 1360 GetCurrentRun()));
2bb7b766 1361
e7f62f16 1362 // create ML instance that monitors this run
1363 fMonaLisa = new TMonaLisaWriter(Form("%d", GetCurrentRun()), "SHUTTLE", "aliendb1.cern.ch");
1364 // disable monitoring of other parameters that come e.g. from TFile
1365 gMonitoringWriter = 0;
2bb7b766 1366
e7f62f16 1367 // Send the information to ML
1368 TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
9827400b 1369 TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
e7f62f16 1370
1371 TList mlList;
1372 mlList.Add(&mlStatus);
9827400b 1373 mlList.Add(&mlRunType);
e7f62f16 1374
1375 fMonaLisa->SendParameters(&mlList);
3301427a 1376
9827400b 1377 if (fLogbookEntry->IsDone())
1378 {
1379 Log("SHUTTLE","Process - Shuttle is already DONE. Updating logbook");
1380 UpdateShuttleLogbook("shuttle_done");
1381 fLogbookEntry = 0;
1382 return kTRUE;
1383 }
1384
1385 // read test mode if flag is set
1386 if (fReadTestMode)
1387 {
3d8bc902 1388 fTestMode = kNone;
9827400b 1389 TString logEntry(entry->GetRunParameter("log"));
1390 //printf("log entry = %s\n", logEntry.Data());
1391 TString searchStr("Testmode: ");
1392 Int_t pos = logEntry.Index(searchStr.Data());
1393 //printf("%d\n", pos);
1394 if (pos >= 0)
1395 {
1396 TSubString subStr = logEntry(pos + searchStr.Length(), logEntry.Length());
1397 //printf("%s\n", subStr.String().Data());
1398 TString newStr(subStr.Data());
1399 TObjArray* token = newStr.Tokenize(' ');
1400 if (token)
1401 {
1402 //token->Print();
1403 TObjString* tmpStr = dynamic_cast<TObjString*> (token->First());
1404 if (tmpStr)
1405 {
1406 Int_t testMode = tmpStr->String().Atoi();
1407 if (testMode > 0)
1408 {
c88ad5db 1409 Log("SHUTTLE", Form("Process - Enabling test mode %d", testMode));
9827400b 1410 SetTestMode((TestMode) testMode);
1411 }
1412 }
1413 delete token;
1414 }
1415 }
1416 }
c88ad5db 1417
eba76848 1418 fLogbookEntry->Print("all");
57f50b3c 1419
1420 // Initialization
d477ad88 1421 Bool_t hasError = kFALSE;
5164a766 1422
675f64cd 1423 // Set the CDB and Reference folders according to the year and LHC period
1424 TString lhcPeriod(GetLHCPeriod());
1425 if (lhcPeriod.Length() == 0)
1426 {
1427 Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
1428 return 0;
1429 }
1430
1431 if (fgkMainCDB.Length() == 0)
1432 fgkMainCDB = Form("alien://folder=/alice/data/%d/%s/OCDB?user=alidaq?cacheFold=/tmp/OCDBCache",
1433 GetCurrentYear(), lhcPeriod.Data());
1434
1435 if (fgkMainRefStorage.Length() == 0)
1436 fgkMainRefStorage = Form("alien://folder=/alice/data/%d/%s/Reference?user=alidaq?cacheFold=/tmp/OCDBCache",
1437 GetCurrentYear(), lhcPeriod.Data());
1438
2bb7b766 1439 AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
1440 if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
1441 AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
1442 if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
d477ad88 1443
57f50b3c 1444 // Loop on detectors in the configuration
b948db8d 1445 TIter iter(fConfig->GetDetectors());
2bb7b766 1446 TObjString* aDetector = 0;
b948db8d 1447
be48e3ea 1448 while ((aDetector = (TObjString*) iter.Next()))
1449 {
7bfb2090 1450 fCurrentDetector = aDetector->String();
5164a766 1451
9e080f92 1452 if (ContinueProcessing() == kFALSE) continue;
1453
c88ad5db 1454 Log("SHUTTLE", Form("\t\t\t****** run %d - %s: START ******",
2bb7b766 1455 GetCurrentRun(), aDetector->GetName()));
1456
9d733021 1457 for(Int_t iSys=0;iSys<3;iSys++) fFXSCalled[iSys]=kFALSE;
1458
c88ad5db 1459 Log(fCurrentDetector.Data(), "Process - Starting processing");
85a80aa9 1460
be48e3ea 1461 Int_t pid = fork();
1462
1463 if (pid < 0)
1464 {
c88ad5db 1465 Log("SHUTTLE", "Process - ERROR: Forking failed");
be48e3ea 1466 }
1467 else if (pid > 0)
1468 {
1469 // parent
c88ad5db 1470 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Starting monitoring",
be48e3ea 1471 GetCurrentRun(), aDetector->GetName()));
1472
1473 Long_t begin = time(0);
1474
1475 int status; // to be used with waitpid, on purpose an int (not Int_t)!
1476 while (waitpid(pid, &status, WNOHANG) == 0)
1477 {
1478 Long_t expiredTime = time(0) - begin;
1479
1480 if (expiredTime > fConfig->GetPPTimeOut())
1481 {
9827400b 1482 TString tmp;
c88ad5db 1483 tmp.Form("Process - Process of %s time out. "
1484 "Run time: %d seconds. Killing...",
1485 fCurrentDetector.Data(), expiredTime);
9827400b 1486 Log("SHUTTLE", tmp);
1487 Log(fCurrentDetector, tmp);
be48e3ea 1488
1489 kill(pid, 9);
1490
3301427a 1491 UpdateShuttleStatus(AliShuttleStatus::kPPTimeOut);
be48e3ea 1492 hasError = kTRUE;
1493
1494 gSystem->Sleep(1000);
1495 }
1496 else
1497 {
be48e3ea 1498 gSystem->Sleep(1000);
9827400b 1499
1500 TString checkStr;
1501 checkStr.Form("ps -o vsize --pid %d | tail -n 1", pid);
1502 FILE* pipe = gSystem->OpenPipe(checkStr, "r");
1503 if (!pipe)
1504 {
c88ad5db 1505 Log("SHUTTLE", Form("Process - Error: "
1506 "Could not open pipe to %s", checkStr.Data()));
9827400b 1507 continue;
1508 }
1509
1510 char buffer[100];
1511 if (!fgets(buffer, 100, pipe))
1512 {
c88ad5db 1513 Log("SHUTTLE", "Process - Error: ps did not return anything");
9827400b 1514 gSystem->ClosePipe(pipe);
1515 continue;
1516 }
1517 gSystem->ClosePipe(pipe);
1518
1519 //Log("SHUTTLE", Form("ps returned %s", buffer));
1520
1521 Int_t mem = 0;
1522 if ((sscanf(buffer, "%d\n", &mem) != 1) || !mem)
1523 {
c88ad5db 1524 Log("SHUTTLE", "Process - Error: Could not parse output of ps");
9827400b 1525 continue;
1526 }
1527
1528 if (expiredTime % 60 == 0)
c88ad5db 1529 Log("SHUTTLE", Form("Process - %s: Checking process. "
1530 "Run time: %d seconds - Memory consumption: %d KB",
1531 fCurrentDetector.Data(), expiredTime, mem));
9827400b 1532
1533 if (mem > fConfig->GetPPMaxMem())
1534 {
1535 TString tmp;
c88ad5db 1536 tmp.Form("Process - Process exceeds maximum allowed memory "
1537 "(%d KB > %d KB). Killing...",
9827400b 1538 mem, fConfig->GetPPMaxMem());
1539 Log("SHUTTLE", tmp);
1540 Log(fCurrentDetector, tmp);
1541
1542 kill(pid, 9);
1543
1544 UpdateShuttleStatus(AliShuttleStatus::kPPOutOfMemory);
1545 hasError = kTRUE;
1546
1547 gSystem->Sleep(1000);
1548 }
be48e3ea 1549 }
1550 }
1551
c88ad5db 1552 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Client has terminated.",
be48e3ea 1553 GetCurrentRun(), aDetector->GetName()));
1554
1555 if (WIFEXITED(status))
1556 {
1557 Int_t returnCode = WEXITSTATUS(status);
1558
c88ad5db 1559 Log("SHUTTLE", Form("Process - %s: the return code is %d", fCurrentDetector.Data(),
3301427a 1560 returnCode));
be48e3ea 1561
9827400b 1562 if (returnCode == 0) hasError = kTRUE;
be48e3ea 1563 }
1564 }
1565 else if (pid == 0)
1566 {
1567 // client
c88ad5db 1568 Log("SHUTTLE", Form("Process - In client process of %d - %s", GetCurrentRun(),
1569 aDetector->GetName()));
be48e3ea 1570
c88ad5db 1571 Log("SHUTTLE", Form("Process - Redirecting output to %s log",fCurrentDetector.Data()));
ffa29e93 1572
546242fb 1573 if ((freopen(GetLogFileName(fCurrentDetector), "a", stdout)) == 0)
ffa29e93 1574 {
c88ad5db 1575 Log("SHUTTLE", "Process - Could not freopen stdout");
ffa29e93 1576 }
1577 else
1578 {
1579 fOutputRedirected = kTRUE;
1580 if ((dup2(fileno(stdout), fileno(stderr))) < 0)
c88ad5db 1581 Log("SHUTTLE", "Process - Could not redirect stderr");
ffa29e93 1582
1583 }
1584
5bac2bde 1585 TString wd = gSystem->WorkingDirectory();
675f64cd 1586 TString tmpDir = Form("%s/%s_%d_process", GetShuttleTempDir(),
1587 fCurrentDetector.Data(), GetCurrentRun());
5bac2bde 1588
d524ade6 1589 Int_t result = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
1590 if (!result) // temp dir already exists!
1591 {
1592 Log(fCurrentDetector.Data(),
1593 Form("Process - %s dir already exists! Removing...", tmpDir.Data()));
1594 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
675f64cd 1595 }
1596
1597 if (gSystem->mkdir(tmpDir.Data(), 1))
1598 {
1599 Log(fCurrentDetector.Data(), "Process - could not make temp directory!!");
1600 gSystem->Exit(1);
d524ade6 1601 }
1602
1603 if (!gSystem->ChangeDirectory(tmpDir.Data()))
1604 {
1605 Log(fCurrentDetector.Data(), "Process - could not change directory!!");
1606 gSystem->Exit(1);
1607 }
5bac2bde 1608
9827400b 1609 Bool_t success = ProcessCurrentDetector();
5bac2bde 1610
1611 gSystem->ChangeDirectory(wd.Data());
1612
1613 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
1614
9827400b 1615 if (success) // Preprocessor finished successfully!
1616 {
3301427a 1617 // Update time_processed field in FXS DB
1618 if (UpdateTable() == kFALSE)
5bac2bde 1619 Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!",
1620 fCurrentDetector.Data()));
3301427a 1621
1622 // Transfer the data from local storage to main storage (Grid)
1623 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1624 if (StoreOCDB() == kFALSE)
1625 {
c88ad5db 1626 Log("SHUTTLE",
1627 Form("\t\t\t****** run %d - %s: STORAGE ERROR ******",
3301427a 1628 GetCurrentRun(), aDetector->GetName()));
1629 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
9827400b 1630 success = kFALSE;
3301427a 1631 } else {
c88ad5db 1632 Log("SHUTTLE",
1633 Form("\t\t\t****** run %d - %s: DONE ******",
3301427a 1634 GetCurrentRun(), aDetector->GetName()));
1635 UpdateShuttleStatus(AliShuttleStatus::kDone);
9827400b 1636 UpdateShuttleLogbook(fCurrentDetector, "DONE");
3301427a 1637 }
c88ad5db 1638 } else
1639 {
1640 Log("SHUTTLE",
1641 Form("\t\t\t****** run %d - %s: PP ERROR ******",
1642 GetCurrentRun(), aDetector->GetName()));
be48e3ea 1643 }
1644
4b95672b 1645 for (UInt_t iSys=0; iSys<3; iSys++)
1646 {
1647 if (fFXSCalled[iSys]) fFXSlist[iSys].Clear();
1648 }
1649
c88ad5db 1650 Log("SHUTTLE", Form("Process - Client process of %d - %s is exiting now with %d.",
9827400b 1651 GetCurrentRun(), aDetector->GetName(), success));
be48e3ea 1652
1653 // the client exits here
9827400b 1654 gSystem->Exit(success);
be48e3ea 1655
1656 AliError("We should never get here!!!");
1657 }
7bfb2090 1658 }
5164a766 1659
c88ad5db 1660 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^*",
2bb7b766 1661 GetCurrentRun()));
1662
1663 //check if shuttle is done for this run, if so update logbook
1664 TObjArray checkEntryArray;
1665 checkEntryArray.SetOwner(1);
9e080f92 1666 TString whereClause = Form("where run=%d", GetCurrentRun());
1667 if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) || checkEntryArray.GetEntries() == 0) {
1668 Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
1669 GetCurrentRun()));
1670 return hasError == kFALSE;
1671 }
b948db8d 1672
9e080f92 1673 AliShuttleLogbookEntry* checkEntry = dynamic_cast<AliShuttleLogbookEntry*>
1674 (checkEntryArray.At(0));
2bb7b766 1675
9e080f92 1676 if (checkEntry)
1677 {
1678 if (checkEntry->IsDone())
be48e3ea 1679 {
9e080f92 1680 Log("SHUTTLE","Process - Shuttle is DONE. Updating logbook");
1681 UpdateShuttleLogbook("shuttle_done");
1682 }
1683 else
1684 {
1685 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
be48e3ea 1686 {
9e080f92 1687 if (checkEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
be48e3ea 1688 {
9e080f92 1689 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
1690 checkEntry->GetRun(), GetDetName(iDet)));
1691 fFirstUnprocessed[iDet] = kFALSE;
be48e3ea 1692 }
1693 }
2bb7b766 1694 }
1695 }
1696
e7f62f16 1697 // remove ML instance
1698 delete fMonaLisa;
1699 fMonaLisa = 0;
1700
2bb7b766 1701 fLogbookEntry = 0;
85a80aa9 1702
a7160fe9 1703 return hasError == kFALSE;
73abe331 1704}
1705
b948db8d 1706//______________________________________________________________________________________________
9827400b 1707Bool_t AliShuttle::ProcessCurrentDetector()
73abe331 1708{
1709 //
2bb7b766 1710 // Makes data retrieval just for a specific detector (fCurrentDetector).
73abe331 1711 // Threre should be a configuration for this detector.
73abe331 1712
1d172743 1713 Log("SHUTTLE", Form("ProcessCurrentDetector - Retrieving values for %s, run %d",
1714 fCurrentDetector.Data(), GetCurrentRun()));
73abe331 1715
d524ade6 1716 TString wd = gSystem->WorkingDirectory();
1717
2d9019b4 1718 if (!CleanReferenceStorage(fCurrentDetector.Data()))
546242fb 1719 return kFALSE;
d524ade6 1720
1721 gSystem->ChangeDirectory(wd.Data());
1722
1d172743 1723 TMap* dcsMap = new TMap();
3301427a 1724
1725 // call preprocessor
1726 AliPreprocessor* aPreprocessor =
1727 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1728
1729 aPreprocessor->Initialize(GetCurrentRun(), GetCurrentStartTime(), GetCurrentEndTime());
1730
1731 Bool_t processDCS = aPreprocessor->ProcessDCS();
d477ad88 1732
651fdaab 1733 if (!processDCS)
1734 {
1d172743 1735 Log(fCurrentDetector, "ProcessCurrentDetector -"
1736 " The preprocessor requested to skip the retrieval of DCS values");
651fdaab 1737 }
8b739301 1738 else if (fTestMode & kSkipDCS)
2c15234c 1739 {
1d172743 1740 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Skipping DCS processing");
9827400b 1741 }
1742 else if (fTestMode & kErrorDCS)
1743 {
1d172743 1744 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Simulating DCS error");
3d8bc902 1745 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
9827400b 1746 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1d172743 1747 delete dcsMap;
9827400b 1748 return kFALSE;
2c15234c 1749 } else {
3301427a 1750
1751 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
1752
1d172743 1753 // Query DCS archive
1754 Int_t nServers = fConfig->GetNServers(fCurrentDetector);
a038aa70 1755
1d172743 1756 for (int iServ=0; iServ<nServers; iServ++)
2c15234c 1757 {
1d172743 1758
1759 TString host(fConfig->GetDCSHost(fCurrentDetector, iServ));
1760 Int_t port = fConfig->GetDCSPort(fCurrentDetector, iServ);
542b6cc8 1761 Int_t multiSplit = fConfig->GetMultiSplit(fCurrentDetector, iServ);
1762
1790d4b7 1763 Log(fCurrentDetector, Form("ProcessCurrentDetector -"
1764 " Querying DCS Amanda server %s:%d (%d of %d)",
1765 host.Data(), port, iServ+1, nServers));
1d172743 1766
1767 TMap* aliasMap = 0;
1768 TMap* dpMap = 0;
1769
1770 if (fConfig->GetDCSAliases(fCurrentDetector, iServ)->GetEntries() > 0)
2c15234c 1771 {
1d172743 1772 aliasMap = GetValueSet(host, port,
542b6cc8 1773 fConfig->GetDCSAliases(fCurrentDetector, iServ),
1774 kAlias, multiSplit);
1d172743 1775 if (!aliasMap)
1776 {
1777 Log(fCurrentDetector,
1778 Form("ProcessCurrentDetector -"
675f64cd 1779 " Error retrieving DCS aliases from server %s."
1780 " Sending mail to DCS experts!", host.Data()));
1d172743 1781 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
675f64cd 1782
1783 if (!SendMailToDCS())
1784 Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
1785
a038aa70 1786 delete dcsMap;
1d172743 1787 return kFALSE;
1788 }
2c15234c 1789 }
a038aa70 1790
1d172743 1791 if (fConfig->GetDCSDataPoints(fCurrentDetector, iServ)->GetEntries() > 0)
a038aa70 1792 {
1d172743 1793 dpMap = GetValueSet(host, port,
542b6cc8 1794 fConfig->GetDCSDataPoints(fCurrentDetector, iServ),
1795 kDP, multiSplit);
1d172743 1796 if (!dpMap)
1797 {
1798 Log(fCurrentDetector,
1799 Form("ProcessCurrentDetector -"
675f64cd 1800 " Error retrieving DCS data points from server %s."
1801 " Sending mail to DCS experts!", host.Data()));
1d172743 1802 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
675f64cd 1803
1804 if (!SendMailToDCS())
1805 Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
1806
1d172743 1807 if (aliasMap) delete aliasMap;
1808 delete dcsMap;
1809 return kFALSE;
1810 }
a038aa70 1811 }
1d172743 1812
1813 // merge aliasMap and dpMap into dcsMap
1814 if(aliasMap) {
1815 TIter iter(aliasMap);
a038aa70 1816 TObjString* key = 0;
1817 while ((key = (TObjString*) iter.Next()))
1d172743 1818 dcsMap->Add(key, aliasMap->GetValue(key->String()));
1819
1820 aliasMap->SetOwner(kFALSE);
1821 delete aliasMap;
1822 }
1823
1824 if(dpMap) {
1825 TIter iter(dpMap);
1826 TObjString* key = 0;
1827 while ((key = (TObjString*) iter.Next()))
1828 dcsMap->Add(key, dpMap->GetValue(key->String()));
1829
1830 dpMap->SetOwner(kFALSE);
1831 delete dpMap;
a038aa70 1832 }
73abe331 1833 }
1834 }
dc25836b 1835
2bb7b766 1836 // DCS Archive DB processing successful. Call Preprocessor!
85a80aa9 1837 UpdateShuttleStatus(AliShuttleStatus::kPPStarted);
a7160fe9 1838
a038aa70 1839 UInt_t returnValue = aPreprocessor->Process(dcsMap);
b948db8d 1840
3301427a 1841 if (returnValue > 0) // Preprocessor error!
1842 {
c88ad5db 1843 Log(fCurrentDetector, Form("ProcessCurrentDetector - "
1844 "Preprocessor failed. Process returned %d.", returnValue));
cb343cfd 1845 UpdateShuttleStatus(AliShuttleStatus::kPPError);
a038aa70 1846 dcsMap->DeleteAll();
1847 delete dcsMap;
9827400b 1848 return kFALSE;
1849 }
1850
1851 // preprocessor ok!
1852 UpdateShuttleStatus(AliShuttleStatus::kPPDone);
1853 Log(fCurrentDetector, Form("ProcessCurrentDetector - %s preprocessor returned success",
1854 fCurrentDetector.Data()));
b948db8d 1855
a038aa70 1856 dcsMap->DeleteAll();
1857 delete dcsMap;
b948db8d 1858
9827400b 1859 return kTRUE;
2bb7b766 1860}
1861
1862//______________________________________________________________________________________________
1863Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
1864 TObjArray& entries)
1865{
9827400b 1866 // Query DAQ's Shuttle logbook and fills detector status object.
1867 // Call QueryRunParameters to query DAQ logbook for run parameters.
1868 //
2bb7b766 1869
fc5a4708 1870 entries.SetOwner(1);
1871
2bb7b766 1872 // check connection, in case connect
be48e3ea 1873 if(!Connect(3)) return kFALSE;
2bb7b766 1874
1875 TString sqlQuery;
441b0e9c 1876 sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
2bb7b766 1877
be48e3ea 1878 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1879 if (!aResult) {
1880 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
1881 return kFALSE;
1882 }
1883
fc5a4708 1884 AliDebug(2,Form("Query = %s", sqlQuery.Data()));
1885
2bb7b766 1886 if(aResult->GetRowCount() == 0) {
c88ad5db 1887 Log("SHUTTLE", "No entries in Shuttle Logbook match request");
9827400b 1888 delete aResult;
1889 return kTRUE;
2bb7b766 1890 }
1891
1892 // TODO Check field count!
db99d43e 1893 const UInt_t nCols = 23;
2bb7b766 1894 if (aResult->GetFieldCount() != (Int_t) nCols) {
c88ad5db 1895 Log("SHUTTLE", "Invalid SQL result field number!");
2bb7b766 1896 delete aResult;
1897 return kFALSE;
1898 }
1899
2bb7b766 1900 TSQLRow* aRow;
1901 while ((aRow = aResult->Next())) {
1902 TString runString(aRow->GetField(0), aRow->GetFieldLength(0));
1903 Int_t run = runString.Atoi();
1904
eba76848 1905 AliShuttleLogbookEntry *entry = QueryRunParameters(run);
1906 if (!entry)
1907 continue;
2bb7b766 1908
1909 // loop on detectors
eba76848 1910 for(UInt_t ii = 0; ii < nCols; ii++)
1911 entry->SetDetectorStatus(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1912
eba76848 1913 entries.AddLast(entry);
2bb7b766 1914 delete aRow;
1915 }
1916
2bb7b766 1917 delete aResult;
1918 return kTRUE;
1919}
1920
1921//______________________________________________________________________________________________
eba76848 1922AliShuttleLogbookEntry* AliShuttle::QueryRunParameters(Int_t run)
2bb7b766 1923{
eba76848 1924 //
1925 // Retrieve run parameters written in the DAQ logbook and sets them into AliShuttleLogbookEntry object
1926 //
2bb7b766 1927
1928 // check connection, in case connect
be48e3ea 1929 if (!Connect(3))
eba76848 1930 return 0;
2bb7b766 1931
1932 TString sqlQuery;
2c15234c 1933 sqlQuery.Form("select * from %s where run=%d", fConfig->GetDAQlbTable(), run);
2bb7b766 1934
be48e3ea 1935 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1936 if (!aResult) {
c88ad5db 1937 Log("SHUTTLE", Form("Can't execute query <%s>!", sqlQuery.Data()));
eba76848 1938 return 0;
2bb7b766 1939 }
1940
eba76848 1941 if (aResult->GetRowCount() == 0) {
2bb7b766 1942 Log("SHUTTLE", Form("QueryRunParameters - No entry in DAQ Logbook for run %d. Skipping", run));
1943 delete aResult;
eba76848 1944 return 0;
2bb7b766 1945 }
1946
eba76848 1947 if (aResult->GetRowCount() > 1) {
c88ad5db 1948 Log("SHUTTLE", Form("QueryRunParameters - UNEXPECTED: "
1949 "more than one entry in DAQ Logbook for run %d!", run));
2bb7b766 1950 delete aResult;
eba76848 1951 return 0;
2bb7b766 1952 }
1953
eba76848 1954 TSQLRow* aRow = aResult->Next();
1955 if (!aRow)
1956 {
c88ad5db 1957 Log("SHUTTLE", Form("QueryRunParameters - Could not retrieve row for run %d. Skipping", run));
eba76848 1958 delete aResult;
1959 return 0;
1960 }
2bb7b766 1961
eba76848 1962 AliShuttleLogbookEntry* entry = new AliShuttleLogbookEntry(run);
2bb7b766 1963
eba76848 1964 for (Int_t ii = 0; ii < aResult->GetFieldCount(); ii++)
1965 entry->SetRunParameter(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1966
eba76848 1967 UInt_t startTime = entry->GetStartTime();
1968 UInt_t endTime = entry->GetEndTime();
1969
1970 if (!startTime || !endTime || startTime > endTime) {
1971 Log("SHUTTLE",
1972 Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d",
1973 run, startTime, endTime));
1974 delete entry;
2bb7b766 1975 delete aRow;
eba76848 1976 delete aResult;
1977 return 0;
2bb7b766 1978 }
1979
eba76848 1980 delete aRow;
2bb7b766 1981 delete aResult;
eba76848 1982
1983 return entry;
2bb7b766 1984}
1985
a038aa70 1986//______________________________________________________________________________________________
1987TMap* AliShuttle::GetValueSet(const char* host, Int_t port, const TSeqCollection* entries,
542b6cc8 1988 DCSType type, Int_t multiSplit)
a038aa70 1989{
1990 // Retrieve all "entry" data points from the DCS server
1991 // host, port: TSocket connection parameters
1992 // entries: list of name of the alias or data point
1993 // type: kAlias or kDP
1994 // returns TMap of values, 0 when failure
542b6cc8 1995
1996 AliDCSClient client(host, port, fTimeout, fRetries, multiSplit);
b41b252a 1997
a038aa70 1998 TMap* result = 0;
b41b252a 1999 if (type == kAlias)
a038aa70 2000 {
b41b252a 2001 result = client.GetAliasValues(entries, GetCurrentStartTime(),
2002 GetCurrentEndTime());
2003 }
2004 else if (type == kDP)
2005 {
2006 result = client.GetDPValues(entries, GetCurrentStartTime(),
2007 GetCurrentEndTime());
2008 }
a038aa70 2009
b41b252a 2010 if (result == 0)
2011 {
2012 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get entries! Reason: %s",
1790d4b7 2013 client.GetErrorString(client.GetResultErrorCode())));
2014 if (client.GetResultErrorCode() == AliDCSClient::fgkServerError)
2015 Log(fCurrentDetector.Data(), Form("GetValueSet - Server error code: %s",
2016 client.GetServerError().Data()));
a038aa70 2017
b41b252a 2018 return 0;
a038aa70 2019 }
b41b252a 2020
a038aa70 2021 return result;
2022}
b41b252a 2023
b948db8d 2024//______________________________________________________________________________________________
57f50b3c 2025const char* AliShuttle::GetFile(Int_t system, const char* detector,
2026 const char* id, const char* source)
b948db8d 2027{
9827400b 2028 // Get calibration file from file exchange servers
2029 // First queris the FXS database for the file name, using the run, detector, id and source info
2030 // then calls RetrieveFile(filename) for actual copy to local disk
2031 // run: current run being processed (given by Logbook entry fLogbookEntry)
2032 // detector: the Preprocessor name
2033 // id: provided as a parameter by the Preprocessor
2034 // source: provided by the Preprocessor through GetFileSources function
2035
2036 // check if test mode should simulate a FXS error
2037 if (fTestMode & kErrorFXSFiles)
2038 {
2039 Log(detector, Form("GetFile - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2040 return 0;
2041 }
2042
57f50b3c 2043 // check connection, in case connect
9d733021 2044 if (!Connect(system))
eba76848 2045 {
9d733021 2046 Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
57f50b3c 2047 return 0;
2048 }
2049
2050 // Query preparation
9d733021 2051 TString sourceName(source);
d386d623 2052 Int_t nFields = 3;
2053 TString sqlQueryStart = Form("select filePath,size,fileChecksum from %s where",
2054 fConfig->GetFXSdbTable(system));
2055 TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
2056 GetCurrentRun(), detector, id);
2057
9d733021 2058 if (system == kDAQ)
2059 {
d386d623 2060 whereClause += Form(" and DAQsource=\"%s\"", source);
57f50b3c 2061 }
9d733021 2062 else if (system == kDCS)
eba76848 2063 {
9d733021 2064 sourceName="none";
57f50b3c 2065 }
9d733021 2066 else if (system == kHLT)
9e080f92 2067 {
d386d623 2068 whereClause += Form(" and DDLnumbers=\"%s\"", source);
9d733021 2069 nFields = 3;
9e080f92 2070 }
2071
9e080f92 2072 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2073
2074 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2075
2076 // Query execution
2077 TSQLResult* aResult = 0;
9d733021 2078 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
9e080f92 2079 if (!aResult) {
9d733021 2080 Log(detector, Form("GetFileName - Can't execute SQL query to %s database for: id = %s, source = %s",
2081 GetSystemName(system), id, sourceName.Data()));
9e080f92 2082 return 0;
2083 }
2084
2085 if(aResult->GetRowCount() == 0)
2086 {
2087 Log(detector,
9d733021 2088 Form("GetFileName - No entry in %s FXS db for: id = %s, source = %s",
2089 GetSystemName(system), id, sourceName.Data()));
9e080f92 2090 delete aResult;
2091 return 0;
2092 }
2bb7b766 2093
9e080f92 2094 if (aResult->GetRowCount() > 1) {
2095 Log(detector,
9d733021 2096 Form("GetFileName - More than one entry in %s FXS db for: id = %s, source = %s",
2097 GetSystemName(system), id, sourceName.Data()));
9e080f92 2098 delete aResult;
2099 return 0;
2100 }
2101
9d733021 2102 if (aResult->GetFieldCount() != nFields) {
9e080f92 2103 Log(detector,
9d733021 2104 Form("GetFileName - Wrong field count in %s FXS db for: id = %s, source = %s",
2105 GetSystemName(system), id, sourceName.Data()));
9e080f92 2106 delete aResult;
2107 return 0;
2108 }
2109
2110 TSQLRow* aRow = dynamic_cast<TSQLRow*> (aResult->Next());
2111
2112 if (!aRow){
9d733021 2113 Log(detector, Form("GetFileName - Empty set result in %s FXS db from query: id = %s, source = %s",
2114 GetSystemName(system), id, sourceName.Data()));
9e080f92 2115 delete aResult;
2116 return 0;
2117 }
2118
2119 TString filePath(aRow->GetField(0), aRow->GetFieldLength(0));
2120 TString fileSize(aRow->GetField(1), aRow->GetFieldLength(1));
d386d623 2121 TString fileChecksum(aRow->GetField(2), aRow->GetFieldLength(2));
9e080f92 2122
2123 delete aResult;
2124 delete aRow;
2125
d386d623 2126 AliDebug(2, Form("filePath = %s; size = %s, fileChecksum = %s",
2127 filePath.Data(), fileSize.Data(), fileChecksum.Data()));
9e080f92 2128
9e080f92 2129 // retrieved file is renamed to make it unique
675f64cd 2130 TString localFileName = Form("%s/%s_%d_process/%s_%s_%d_%s_%s.shuttle",
2131 GetShuttleTempDir(), detector, GetCurrentRun(),
d524ade6 2132 GetSystemName(system), detector, GetCurrentRun(),
2133 id, sourceName.Data());
9d733021 2134
9e080f92 2135
9d733021 2136 // file retrieval from FXS
4b95672b 2137 UInt_t nRetries = 0;
2138 UInt_t maxRetries = 3;
2139 Bool_t result = kFALSE;
2140
2141 // copy!! if successful TSystem::Exec returns 0
2142 while(nRetries++ < maxRetries) {
2143 AliDebug(2, Form("Trying to copy file. Retry # %d", nRetries));
2144 result = RetrieveFile(system, filePath.Data(), localFileName.Data());
2145 if(!result)
2146 {
2147 Log(detector, Form("GetFileName - Copy of file %s from %s FXS failed",
9d733021 2148 filePath.Data(), GetSystemName(system)));
4b95672b 2149 continue;
4f0749a8 2150 }
9e080f92 2151
d386d623 2152 if (fileChecksum.Length()>0)
4b95672b 2153 {
2154 // compare md5sum of local file with the one stored in the FXS DB
d524ade6 2155 Int_t md5Comp = gSystem->Exec(Form("md5sum %s |grep %s 2>&1 > /dev/null",
2156 localFileName.Data(), fileChecksum.Data()));
9e080f92 2157
4b95672b 2158 if (md5Comp != 0)
2159 {
2160 Log(detector, Form("GetFileName - md5sum of file %s does not match with local copy!",
2161 filePath.Data()));
2162 result = kFALSE;
2163 continue;
2164 }
d386d623 2165 } else {
2166 Log(fCurrentDetector, Form("GetFile - md5sum of file %s not set in %s database, skipping comparison",
2167 filePath.Data(), GetSystemName(system)));
9d733021 2168 }
4b95672b 2169 if (result) break;
9e080f92 2170 }
2171
4b95672b 2172 if(!result) return 0;
2173
9d733021 2174 fFXSCalled[system]=kTRUE;
2175 TObjString *fileParams = new TObjString(Form("%s#!?!#%s", id, sourceName.Data()));
2176 fFXSlist[system].Add(fileParams);
9e080f92 2177
675f64cd 2178 static TString staticLocalFileName;
2179 staticLocalFileName.Form("%s", localFileName.Data());
2180
c88ad5db 2181 Log(fCurrentDetector, Form("GetFile - Retrieved file with id %s and "
2182 "source %s from %s to %s", id, source,
d524ade6 2183 GetSystemName(system), localFileName.Data()));
675f64cd 2184
d524ade6 2185 return staticLocalFileName.Data();
2bb7b766 2186}
2187
2188//______________________________________________________________________________________________
9d733021 2189Bool_t AliShuttle::RetrieveFile(UInt_t system, const char* fxsFileName, const char* localFileName)
9e080f92 2190{
9827400b 2191 //
2192 // Copies file from FXS to local Shuttle machine
2193 //
2bb7b766 2194
9e080f92 2195 // check temp directory: trying to cd to temp; if it does not exist, create it
d524ade6 2196 AliDebug(2, Form("Copy file %s from %s FXS into %s",
2197 GetSystemName(system), fxsFileName, localFileName));
2198
2199 TString tmpDir(localFileName);
2200
2201 tmpDir = tmpDir(0,tmpDir.Last('/'));
9e080f92 2202
d524ade6 2203 Int_t noDir = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
2204 if (noDir) // temp dir does not exists!
2205 {
2206 if (gSystem->mkdir(tmpDir.Data(), 1))
2207 {
2208 Log(fCurrentDetector.Data(), "RetrieveFile - could not make temp directory!!");
9e080f92 2209 return kFALSE;
2210 }
9e080f92 2211 }
2212
9d733021 2213 TString baseFXSFolder;
2214 if (system == kDAQ)
2215 {
2216 baseFXSFolder = "FES/";
2217 }
2218 else if (system == kDCS)
2219 {
2220 baseFXSFolder = "";
2221 }
2222 else if (system == kHLT)
2223 {
42fde080 2224 baseFXSFolder = "/opt/FXS/";
9d733021 2225 }
2226
2227
d524ade6 2228 TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s",
9d733021 2229 fConfig->GetFXSPort(system),
2230 fConfig->GetFXSUser(system),
2231 fConfig->GetFXSHost(system),
2232 baseFXSFolder.Data(),
2233 fxsFileName,
9e080f92 2234 localFileName);
2235
2236 AliDebug(2, Form("%s",command.Data()));
2237
4b95672b 2238 Bool_t result = (gSystem->Exec(command.Data()) == 0);
9e080f92 2239
4b95672b 2240 return result;
9e080f92 2241}
2242
2243//______________________________________________________________________________________________
9d733021 2244TList* AliShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
2245{
9827400b 2246 //
2247 // Get sources producing the condition file Id from file exchange servers
4a33bdd9 2248 // if id is NULL all sources are returned (distinct)
9827400b 2249 //
1bcd28db 2250
2251 Log(detector, Form("GetFileSources - Retrieving sources with id %s from %s", id, GetSystemName(system)));
9827400b 2252
2253 // check if test mode should simulate a FXS error
2254 if (fTestMode & kErrorFXSSources)
2255 {
2256 Log(detector, Form("GetFileSources - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2257 return 0;
2258 }
2259
9d733021 2260 if (system == kDCS)
2261 {
c88ad5db 2262 Log(detector, "GetFileSources - WARNING: DCS system has only one source of data!");
6297b37d 2263 TList *list = new TList();
2264 list->SetOwner(1);
2265 list->Add(new TObjString(" "));
2266 return list;
9d733021 2267 }
9e080f92 2268
2269 // check connection, in case connect
9d733021 2270 if (!Connect(system))
2271 {
4a33bdd9 2272 Log(detector, Form("GetFileSources - Couldn't connect to %s FXS database", GetSystemName(system)));
9d733021 2273 return NULL;
9e080f92 2274 }
2275
9d733021 2276 TString sourceName = 0;
2277 if (system == kDAQ)
2278 {
2279 sourceName = "DAQsource";
2280 } else if (system == kHLT)
2281 {
2282 sourceName = "DDLnumbers";
2283 }
2284
4a33bdd9 2285 TString sqlQueryStart = Form("select distinct %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
2286 TString whereClause = Form("run=%d and detector=\"%s\"",
2287 GetCurrentRun(), detector);
2288 if (id)
2289 whereClause += Form(" and fileId=\"%s\"", id);
9e080f92 2290 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2291
2292 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2293
2294 // Query execution
2295 TSQLResult* aResult;
9d733021 2296 aResult = fServer[system]->Query(sqlQuery);
9e080f92 2297 if (!aResult) {
9d733021 2298 Log(detector, Form("GetFileSources - Can't execute SQL query to %s database for id: %s",
2299 GetSystemName(system), id));
9e080f92 2300 return 0;
2301 }
2302
86aa42c3 2303 TList *list = new TList();
2304 list->SetOwner(1);
2305
9d733021 2306 if (aResult->GetRowCount() == 0)
2307 {
9e080f92 2308 Log(detector,
9d733021 2309 Form("GetFileSources - No entry in %s FXS table for id: %s", GetSystemName(system), id));
9e080f92 2310 delete aResult;
86aa42c3 2311 return list;
9e080f92 2312 }
2313
1bcd28db 2314 Log(detector, Form("GetFileSources - Found %d sources", aResult->GetRowCount()));
9e080f92 2315
1bcd28db 2316 TSQLRow* aRow;
9d733021 2317 while ((aRow = aResult->Next()))
2318 {
9e080f92 2319
9d733021 2320 TString source(aRow->GetField(0), aRow->GetFieldLength(0));
2321 AliDebug(2, Form("%s = %s", sourceName.Data(), source.Data()));
2322 list->Add(new TObjString(source));
9e080f92 2323 delete aRow;
2324 }
9d733021 2325
9e080f92 2326 delete aResult;
2327
2328 return list;
2bb7b766 2329}
2330
4a33bdd9 2331//______________________________________________________________________________________________
2332TList* AliShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
2333{
2334 //
2335 // Get all ids of condition files produced by a given source from file exchange servers
2336 //
2337
1bcd28db 2338 Log(detector, Form("GetFileIDs - Retrieving ids with source %s with %s", source, GetSystemName(system)));
2339
4a33bdd9 2340 // check if test mode should simulate a FXS error
2341 if (fTestMode & kErrorFXSSources)
2342 {
2343 Log(detector, Form("GetFileIDs - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2344 return 0;
2345 }
2346
2347 // check connection, in case connect
2348 if (!Connect(system))
2349 {
2350 Log(detector, Form("GetFileIDs - Couldn't connect to %s FXS database", GetSystemName(system)));
2351 return NULL;
2352 }
2353
2354 TString sourceName = 0;
2355 if (system == kDAQ)
2356 {
2357 sourceName = "DAQsource";
2358 } else if (system == kHLT)
2359 {
2360 sourceName = "DDLnumbers";
2361 }
2362
2363 TString sqlQueryStart = Form("select fileId from %s where", fConfig->GetFXSdbTable(system));
2364 TString whereClause = Form("run=%d and detector=\"%s\"",
2365 GetCurrentRun(), detector);
2366 if (sourceName.Length() > 0 && source)
2367 whereClause += Form(" and %s=\"%s\"", sourceName.Data(), source);
2368 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2369
2370 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2371
2372 // Query execution
2373 TSQLResult* aResult;
2374 aResult = fServer[system]->Query(sqlQuery);
2375 if (!aResult) {
2376 Log(detector, Form("GetFileIDs - Can't execute SQL query to %s database for source: %s",
2377 GetSystemName(system), source));
2378 return 0;
2379 }
2380
2381 TList *list = new TList();
2382 list->SetOwner(1);
2383
2384 if (aResult->GetRowCount() == 0)
2385 {
2386 Log(detector,
2387 Form("GetFileIDs - No entry in %s FXS table for source: %s", GetSystemName(system), source));
2388 delete aResult;
2389 return list;
2390 }
2391
1bcd28db 2392 Log(detector, Form("GetFileIDs - Found %d ids", aResult->GetRowCount()));
2393
4a33bdd9 2394 TSQLRow* aRow;
2395
2396 while ((aRow = aResult->Next()))
2397 {
2398
2399 TString id(aRow->GetField(0), aRow->GetFieldLength(0));
2400 AliDebug(2, Form("fileId = %s", id.Data()));
2401 list->Add(new TObjString(id));
2402 delete aRow;
2403 }
2404
2405 delete aResult;
2406
2407 return list;
2408}
2409
2bb7b766 2410//______________________________________________________________________________________________
9d733021 2411Bool_t AliShuttle::Connect(Int_t system)
2bb7b766 2412{
9827400b 2413 // Connect to MySQL Server of the system's FXS MySQL databases
2414 // DAQ Logbook, Shuttle Logbook and DAQ FXS db are on the same host
2415 //
57f50b3c 2416
9d733021 2417 // check connection: if already connected return
2418 if(fServer[system] && fServer[system]->IsConnected()) return kTRUE;
57f50b3c 2419
9d733021 2420 TString dbHost, dbUser, dbPass, dbName;
57f50b3c 2421
9d733021 2422 if (system < 3) // FXS db servers
2423 {
2424 dbHost = Form("mysql://%s:%d", fConfig->GetFXSdbHost(system), fConfig->GetFXSdbPort(system));
2425 dbUser = fConfig->GetFXSdbUser(system);
2426 dbPass = fConfig->GetFXSdbPass(system);
2427 dbName = fConfig->GetFXSdbName(system);
2428 } else { // Run & Shuttle logbook servers
2429 // TODO Will the Shuttle logbook server be the same as the Run logbook server ???
2430 dbHost = Form("mysql://%s:%d", fConfig->GetDAQlbHost(), fConfig->GetDAQlbPort());
2431 dbUser = fConfig->GetDAQlbUser();
2432 dbPass = fConfig->GetDAQlbPass();
2433 dbName = fConfig->GetDAQlbDB();
2434 }
57f50b3c 2435
9d733021 2436 fServer[system] = TSQLServer::Connect(dbHost.Data(), dbUser.Data(), dbPass.Data());
2437 if (!fServer[system] || !fServer[system]->IsConnected()) {
2438 if(system < 3)
2439 {
2440 AliError(Form("Can't establish connection to FXS database for %s",
2441 AliShuttleInterface::GetSystemName(system)));
2442 } else {
2443 AliError("Can't establish connection to Run logbook.");
57f50b3c 2444 }
9d733021 2445 if(fServer[system]) delete fServer[system];
2446 return kFALSE;
2bb7b766 2447 }
57f50b3c 2448
9d733021 2449 // Get tables
2450 TSQLResult* aResult=0;
2451 switch(system){
2452 case kDAQ:
2453 aResult = fServer[kDAQ]->GetTables(dbName.Data());
2454 break;
2455 case kDCS:
2456 aResult = fServer[kDCS]->GetTables(dbName.Data());
2457 break;
2458 case kHLT:
2459 aResult = fServer[kHLT]->GetTables(dbName.Data());
2460 break;
2461 default:
2462 aResult = fServer[3]->GetTables(dbName.Data());
2463 break;
2464 }
2465
2466 delete aResult;
2bb7b766 2467 return kTRUE;
2468}
57f50b3c 2469
9e080f92 2470//______________________________________________________________________________________________
9d733021 2471Bool_t AliShuttle::UpdateTable()
9e080f92 2472{
9827400b 2473 //
2474 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2475 //
9e080f92 2476
9d733021 2477 Bool_t result = kTRUE;
9e080f92 2478
9d733021 2479 for (UInt_t system=0; system<3; system++)
2480 {
2481 if(!fFXSCalled[system]) continue;
9e080f92 2482
9d733021 2483 // check connection, in case connect
2484 if (!Connect(system))
2485 {
2486 Log(fCurrentDetector, Form("UpdateTable - Couldn't connect to %s FXS database", GetSystemName(system)));
2487 result = kFALSE;
2488 continue;
9e080f92 2489 }
9e080f92 2490
9d733021 2491 TTimeStamp now; // now
2492
2493 // Loop on FXS list entries
2494 TIter iter(&fFXSlist[system]);
2495 TObjString *aFXSentry=0;
2496 while ((aFXSentry = dynamic_cast<TObjString*> (iter.Next())))
2497 {
2498 TString aFXSentrystr = aFXSentry->String();
2499 TObjArray *aFXSarray = aFXSentrystr.Tokenize("#!?!#");
2500 if (!aFXSarray || aFXSarray->GetEntries() != 2 )
2501 {
2502 Log(fCurrentDetector, Form("UpdateTable - error updating %s FXS entry. Check string: <%s>",
2503 GetSystemName(system), aFXSentrystr.Data()));
2504 if(aFXSarray) delete aFXSarray;
2505 result = kFALSE;
2506 continue;
2507 }
2508 const char* fileId = ((TObjString*) aFXSarray->At(0))->GetName();
2509 const char* source = ((TObjString*) aFXSarray->At(1))->GetName();
2510
2511 TString whereClause;
2512 if (system == kDAQ)
2513 {
2514 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DAQsource=\"%s\";",
2515 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2516 }
2517 else if (system == kDCS)
2518 {
2519 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\";",
2520 GetCurrentRun(), fCurrentDetector.Data(), fileId);
2521 }
2522 else if (system == kHLT)
2523 {
2524 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DDLnumbers=\"%s\";",
2525 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2526 }
2527
2528 delete aFXSarray;
9e080f92 2529
9d733021 2530 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2531 now.GetSec(), whereClause.Data());
9e080f92 2532
9d733021 2533 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
9e080f92 2534
9d733021 2535 // Query execution
2536 TSQLResult* aResult;
2537 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2538 if (!aResult)
2539 {
2540 Log(fCurrentDetector, Form("UpdateTable - %s db: can't execute SQL query <%s>",
2541 GetSystemName(system), sqlQuery.Data()));
2542 result = kFALSE;
2543 continue;
2544 }
2545 delete aResult;
9e080f92 2546 }
9e080f92 2547 }
2548
9d733021 2549 return result;
9e080f92 2550}
57f50b3c 2551
3301427a 2552//______________________________________________________________________________________________
2553Bool_t AliShuttle::UpdateTableFailCase()
2554{
9827400b 2555 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2556 // this is called in case the preprocessor is declared failed for the current run, because
2557 // the fields are updated only in case of success
3301427a 2558
2559 Bool_t result = kTRUE;
2560
2561 for (UInt_t system=0; system<3; system++)
2562 {
2563 // check connection, in case connect
2564 if (!Connect(system))
2565 {
2566 Log(fCurrentDetector, Form("UpdateTableFailCase - Couldn't connect to %s FXS database",
2567 GetSystemName(system)));
2568 result = kFALSE;
2569 continue;
2570 }
2571
2572 TTimeStamp now; // now
2573
2574 // Loop on FXS list entries
2575
2576 TString whereClause = Form("where run=%d and detector=\"%s\";",
2577 GetCurrentRun(), fCurrentDetector.Data());
2578
2579
2580 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2581 now.GetSec(), whereClause.Data());
2582
2583 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2584
2585 // Query execution
2586 TSQLResult* aResult;
2587 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2588 if (!aResult)
2589 {
2590 Log(fCurrentDetector, Form("UpdateTableFailCase - %s db: can't execute SQL query <%s>",
2591 GetSystemName(system), sqlQuery.Data()));
2592 result = kFALSE;
2593 continue;
2594 }
2595 delete aResult;
2596 }
2597
2598 return result;
2599}
2600
2bb7b766 2601//______________________________________________________________________________________________
2602Bool_t AliShuttle::UpdateShuttleLogbook(const char* detector, const char* status)
2603{
e7f62f16 2604 //
2605 // Update Shuttle logbook filling detector or shuttle_done column
2606 // ex. of usage: UpdateShuttleLogbook("PHOS", "DONE") or UpdateShuttleLogbook("shuttle_done")
2607 //
57f50b3c 2608
2bb7b766 2609 // check connection, in case connect
be48e3ea 2610 if(!Connect(3)){
2bb7b766 2611 Log("SHUTTLE", "UpdateShuttleLogbook - Couldn't connect to DAQ Logbook.");
2612 return kFALSE;
57f50b3c 2613 }
2614
2bb7b766 2615 TString detName(detector);
2616 TString setClause;
e7f62f16 2617 if(detName == "shuttle_done")
2618 {
2bb7b766 2619 setClause = "set shuttle_done=1";
e7f62f16 2620
2621 // Send the information to ML
2622 TMonaLisaText mlStatus("SHUTTLE_status", "Done");
2623
2624 TList mlList;
2625 mlList.Add(&mlStatus);
2626
2627 fMonaLisa->SendParameters(&mlList);
2bb7b766 2628 } else {
2bb7b766 2629 TString statusStr(status);
2630 if(statusStr.Contains("done", TString::kIgnoreCase) ||
2631 statusStr.Contains("failed", TString::kIgnoreCase)){
eba76848 2632 setClause = Form("set %s=\"%s\"", detector, status);
2bb7b766 2633 } else {
2634 Log("SHUTTLE",
2635 Form("UpdateShuttleLogbook - Invalid status <%s> for detector %s",
2636 status, detector));
2637 return kFALSE;
2638 }
2639 }
57f50b3c 2640
2bb7b766 2641 TString whereClause = Form("where run=%d", GetCurrentRun());
2642
441b0e9c 2643 TString sqlQuery = Form("update %s %s %s",
2644 fConfig->GetShuttlelbTable(), setClause.Data(), whereClause.Data());
57f50b3c 2645
2bb7b766 2646 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2647
2648 // Query execution
2649 TSQLResult* aResult;
be48e3ea 2650 aResult = dynamic_cast<TSQLResult*> (fServer[3]->Query(sqlQuery));
2bb7b766 2651 if (!aResult) {
2652 Log("SHUTTLE", Form("UpdateShuttleLogbook - Can't execute query <%s>", sqlQuery.Data()));
2653 return kFALSE;
57f50b3c 2654 }
2bb7b766 2655 delete aResult;
57f50b3c 2656
2657 return kTRUE;
2658}
2659
2660//______________________________________________________________________________________________
2bb7b766 2661Int_t AliShuttle::GetCurrentRun() const
2662{
9827400b 2663 //
2664 // Get current run from logbook entry
2665 //
57f50b3c 2666
2bb7b766 2667 return fLogbookEntry ? fLogbookEntry->GetRun() : -1;
57f50b3c 2668}
2669
2670//______________________________________________________________________________________________
2bb7b766 2671UInt_t AliShuttle::GetCurrentStartTime() const
2672{
9827400b 2673 //
2674 // get current start time
2675 //
57f50b3c 2676
2bb7b766 2677 return fLogbookEntry ? fLogbookEntry->GetStartTime() : 0;
57f50b3c 2678}
2679
2680//______________________________________________________________________________________________
2bb7b766 2681UInt_t AliShuttle::GetCurrentEndTime() const
2682{
9827400b 2683 //
2684 // get current end time from logbook entry
2685 //
57f50b3c 2686
2bb7b766 2687 return fLogbookEntry ? fLogbookEntry->GetEndTime() : 0;
57f50b3c 2688}
2689
675f64cd 2690//______________________________________________________________________________________________
2691UInt_t AliShuttle::GetCurrentYear() const
2692{
2693 //
2694 // Get current year from logbook entry
2695 //
2696
2697 if (!fLogbookEntry) return 0;
2698
2699 TTimeStamp startTime(GetCurrentStartTime());
2700 TString year = Form("%d",startTime.GetDate());
2701 year = year(0,4);
2702
2703 return year.Atoi();
2704}
2705
2706//______________________________________________________________________________________________
2707const char* AliShuttle::GetLHCPeriod() const
2708{
2709 //
2710 // Get current LHC period from logbook entry
2711 //
2712
2713 if (!fLogbookEntry) return 0;
2714
2715 return fLogbookEntry->GetRunParameter("LHCperiod");
2716}
2717
b948db8d 2718//______________________________________________________________________________________________
2719void AliShuttle::Log(const char* detector, const char* message)
2720{
9827400b 2721 //
2722 // Fill log string with a message
2723 //
b948db8d 2724
36c99a6a 2725 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
84090f85 2726 if (dir == NULL) {
36c99a6a 2727 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE)) {
2728 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
84090f85 2729 return;
2730 }
b948db8d 2731
84090f85 2732 } else {
2733 gSystem->FreeDirectory(dir);
2734 }
b948db8d 2735
cb343cfd 2736 TString toLog = Form("%s (%d): %s - ", TTimeStamp(time(0)).AsString("s"), getpid(), detector);
e7f62f16 2737 if (GetCurrentRun() >= 0)
2738 toLog += Form("run %d - ", GetCurrentRun());
2bb7b766 2739 toLog += Form("%s", message);
2740
84090f85 2741 AliInfo(toLog.Data());
ffa29e93 2742
2743 // if we redirect the log output already to the file, leave here
2744 if (fOutputRedirected && strcmp(detector, "SHUTTLE") != 0)
2745 return;
b948db8d 2746
ffa29e93 2747 TString fileName = GetLogFileName(detector);
e7f62f16 2748
84090f85 2749 gSystem->ExpandPathName(fileName);
2750
2751 ofstream logFile;
2752 logFile.open(fileName, ofstream::out | ofstream::app);
2753
2754 if (!logFile.is_open()) {
2755 AliError(Form("Could not open file %s", fileName.Data()));
2756 return;
2757 }
7bfb2090 2758
84090f85 2759 logFile << toLog.Data() << "\n";
b948db8d 2760
84090f85 2761 logFile.close();
b948db8d 2762}
2bb7b766 2763
ffa29e93 2764//______________________________________________________________________________________________
2765TString AliShuttle::GetLogFileName(const char* detector) const
2766{
2767 //
2768 // returns the name of the log file for a given sub detector
2769 //
2770
2771 TString fileName;
2772
2773 if (GetCurrentRun() >= 0)
2774 fileName.Form("%s/%s_%d.log", GetShuttleLogDir(), detector, GetCurrentRun());
2775 else
2776 fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
2777
2778 return fileName;
2779}
2780
2bb7b766 2781//______________________________________________________________________________________________
2782Bool_t AliShuttle::Collect(Int_t run)
2783{
9827400b 2784 //
2785 // Collects conditions data for all UNPROCESSED run written to DAQ LogBook in case of run = -1 (default)
2786 // If a dedicated run is given this run is processed
2787 //
2788 // In operational mode, this is the Shuttle function triggered by the EOR signal.
2789 //
2bb7b766 2790
eba76848 2791 if (run == -1)
2792 Log("SHUTTLE","Collect - Shuttle called. Collecting conditions data for unprocessed runs");
2793 else
2794 Log("SHUTTLE", Form("Collect - Shuttle called. Collecting conditions data for run %d", run));
cb343cfd 2795
2796 SetLastAction("Starting");
2bb7b766 2797
2798 TString whereClause("where shuttle_done=0");
eba76848 2799 if (run != -1)
2800 whereClause += Form(" and run=%d", run);
2bb7b766 2801
2802 TObjArray shuttleLogbookEntries;
be48e3ea 2803 if (!QueryShuttleLogbook(whereClause, shuttleLogbookEntries))
2804 {
cb343cfd 2805 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2bb7b766 2806 return kFALSE;
2807 }
2808
9e080f92 2809 if (shuttleLogbookEntries.GetEntries() == 0)
2810 {
2811 if (run == -1)
2812 Log("SHUTTLE","Collect - Found no UNPROCESSED runs in Shuttle logbook");
2813 else
2814 Log("SHUTTLE", Form("Collect - Run %d is already DONE "
2815 "or it does not exist in Shuttle logbook", run));
2816 return kTRUE;
2817 }
2818
be48e3ea 2819 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2820 fFirstUnprocessed[iDet] = kTRUE;
2821
fc5a4708 2822 if (run != -1)
be48e3ea 2823 {
2824 // query Shuttle logbook for earlier runs, check if some detectors are unprocessed,
2825 // flag them into fFirstUnprocessed array
2826 TString whereClause(Form("where shuttle_done=0 and run < %d", run));
2827 TObjArray tmpLogbookEntries;
2828 if (!QueryShuttleLogbook(whereClause, tmpLogbookEntries))
2829 {
2830 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2831 return kFALSE;
2832 }
2833
2834 TIter iter(&tmpLogbookEntries);
2835 AliShuttleLogbookEntry* anEntry = 0;
2836 while ((anEntry = dynamic_cast<AliShuttleLogbookEntry*> (iter.Next())))
2837 {
2838 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2839 {
2840 if (anEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
2841 {
2842 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
2843 anEntry->GetRun(), GetDetName(iDet)));
2844 fFirstUnprocessed[iDet] = kFALSE;
2845 }
2846 }
2847
2848 }
2849
2850 }
2851
2852 if (!RetrieveConditionsData(shuttleLogbookEntries))
2853 {
cb343cfd 2854 Log("SHUTTLE", "Collect - Process of at least one run failed");
2bb7b766 2855 return kFALSE;
2856 }
2857
36c99a6a 2858 Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
eba76848 2859 return kTRUE;
2bb7b766 2860}
2861
2bb7b766 2862//______________________________________________________________________________________________
2863Bool_t AliShuttle::RetrieveConditionsData(const TObjArray& dateEntries)
2864{
9827400b 2865 //
2866 // Retrieve conditions data for all runs that aren't processed yet
2867 //
2bb7b766 2868
2869 Bool_t hasError = kFALSE;
2870
2871 TIter iter(&dateEntries);
2872 AliShuttleLogbookEntry* anEntry;
2873
2874 while ((anEntry = (AliShuttleLogbookEntry*) iter.Next())){
2875 if (!Process(anEntry)){
2876 hasError = kTRUE;
2877 }
4b95672b 2878
2879 // clean SHUTTLE temp directory
d524ade6 2880 //TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
2881 //RemoveFile(filename.Data());
2bb7b766 2882 }
2883
2884 return hasError == kFALSE;
2885}
cb343cfd 2886
2887//______________________________________________________________________________________________
2888ULong_t AliShuttle::GetTimeOfLastAction() const
2889{
9827400b 2890 //
2891 // Gets time of last action
2892 //
2893
cb343cfd 2894 ULong_t tmp;
36c99a6a 2895
cb343cfd 2896 fMonitoringMutex->Lock();
be48e3ea 2897
cb343cfd 2898 tmp = fLastActionTime;
36c99a6a 2899
cb343cfd 2900 fMonitoringMutex->UnLock();
36c99a6a 2901
cb343cfd 2902 return tmp;
2903}
2904
2905//______________________________________________________________________________________________
2906const TString AliShuttle::GetLastAction() const
2907{
9827400b 2908 //
cb343cfd 2909 // returns a string description of the last action
9827400b 2910 //
cb343cfd 2911
2912 TString tmp;
36c99a6a 2913
cb343cfd 2914 fMonitoringMutex->Lock();
2915
2916 tmp = fLastAction;
2917
2918 fMonitoringMutex->UnLock();
2919
36c99a6a 2920 return tmp;
cb343cfd 2921}
2922
2923//______________________________________________________________________________________________
2924void AliShuttle::SetLastAction(const char* action)
2925{
9827400b 2926 //
cb343cfd 2927 // updates the monitoring variables
9827400b 2928 //
36c99a6a 2929
cb343cfd 2930 fMonitoringMutex->Lock();
36c99a6a 2931
cb343cfd 2932 fLastAction = action;
2933 fLastActionTime = time(0);
2934
2935 fMonitoringMutex->UnLock();
2936}
eba76848 2937
2938//______________________________________________________________________________________________
2939const char* AliShuttle::GetRunParameter(const char* param)
2940{
9827400b 2941 //
2942 // returns run parameter read from DAQ logbook
2943 //
eba76848 2944
2945 if(!fLogbookEntry) {
2946 AliError("No logbook entry!");
2947 return 0;
2948 }
2949
2950 return fLogbookEntry->GetRunParameter(param);
2951}
57c1a579 2952
d386d623 2953//______________________________________________________________________________________________
9827400b 2954AliCDBEntry* AliShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
d386d623 2955{
9827400b 2956 //
2957 // returns object from OCDB valid for current run
2958 //
d386d623 2959
9827400b 2960 if (fTestMode & kErrorOCDB)
2961 {
2962 Log(detector, "GetFromOCDB - In TESTMODE - Simulating error with OCDB");
2963 return 0;
2964 }
2965
d386d623 2966 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
2967 if (!sto)
2968 {
9827400b 2969 Log(detector, "GetFromOCDB - Cannot activate main OCDB for query!");
d386d623 2970 return 0;
2971 }
2972
2973 return dynamic_cast<AliCDBEntry*> (sto->Get(path, GetCurrentRun()));
2974}
2975
57c1a579 2976//______________________________________________________________________________________________
2977Bool_t AliShuttle::SendMail()
2978{
9827400b 2979 //
2980 // sends a mail to the subdetector expert in case of preprocessor error
2981 //
2982
2983 if (fTestMode != kNone)
2984 return kTRUE;
57c1a579 2985
36c99a6a 2986 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
57c1a579 2987 if (dir == NULL)
2988 {
36c99a6a 2989 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
57c1a579 2990 {
675f64cd 2991 Log("SHUTTLE", Form("SendMail - Can't open directory <%s>", GetShuttleLogDir()));
57c1a579 2992 return kFALSE;
2993 }
2994
2995 } else {
2996 gSystem->FreeDirectory(dir);
2997 }
2998
2999 TString bodyFileName;
36c99a6a 3000 bodyFileName.Form("%s/mail.body", GetShuttleLogDir());
57c1a579 3001 gSystem->ExpandPathName(bodyFileName);
3002
3003 ofstream mailBody;
3004 mailBody.open(bodyFileName, ofstream::out);
3005
3006 if (!mailBody.is_open())
3007 {
675f64cd 3008 Log("SHUTTLE", Form("Could not open mail body file %s", bodyFileName.Data()));
57c1a579 3009 return kFALSE;
3010 }
3011
3012 TString to="";
3013 TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
3014 TObjString *anExpert=0;
3015 while ((anExpert = (TObjString*) iterExperts.Next()))
3016 {
3017 to += Form("%s,", anExpert->GetName());
3018 }
3019 to.Remove(to.Length()-1);
909732f7 3020 AliDebug(2, Form("to: %s",to.Data()));
57c1a579 3021
86aa42c3 3022 if (to.IsNull()) {
c88ad5db 3023 Log("SHUTTLE", "List of detector responsibles not yet set!");
36c99a6a 3024 return kFALSE;
3025 }
3026
57c1a579 3027 TString cc="alberto.colla@cern.ch";
3028
546242fb 3029 TString subject = Form("%s Shuttle preprocessor FAILED in run %d !",
57c1a579 3030 fCurrentDetector.Data(), GetCurrentRun());
909732f7 3031 AliDebug(2, Form("subject: %s", subject.Data()));
57c1a579 3032
3033 TString body = Form("Dear %s expert(s), \n\n", fCurrentDetector.Data());
3034 body += Form("SHUTTLE just detected that your preprocessor "
546242fb 3035 "failed processing run %d!!\n\n", GetCurrentRun());
3036 body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n", fCurrentDetector.Data());
3037 body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
3038 body += Form("Find the %s log for the current run on \n\n"
3039 "\thttp://pcalishuttle01.cern.ch:8880/logs/%s_%d.log \n\n",
3040 fCurrentDetector.Data(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 3041 body += Form("The last 10 lines of %s log file are following:\n\n");
3042
909732f7 3043 AliDebug(2, Form("Body begin: %s", body.Data()));
57c1a579 3044
3045 mailBody << body.Data();
3046 mailBody.close();
3047 mailBody.open(bodyFileName, ofstream::out | ofstream::app);
3048
9d733021 3049 TString logFileName = Form("%s/%s_%d.log", GetShuttleLogDir(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 3050 TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
3051 if (gSystem->Exec(tailCommand.Data()))
3052 {
3053 mailBody << Form("%s log file not found ...\n\n", fCurrentDetector.Data());
3054 }
3055
3056 TString endBody = Form("------------------------------------------------------\n\n");
675f64cd 3057 endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
3058 endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
3059 endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
3060
3061 AliDebug(2, Form("Body end: %s", endBody.Data()));
3062
3063 mailBody << endBody.Data();
3064
3065 mailBody.close();
3066
3067 // send mail!
3068 TString mailCommand = Form("mail -s \"%s\" -c %s %s < %s",
3069 subject.Data(),
3070 cc.Data(),
3071 to.Data(),
3072 bodyFileName.Data());
3073 AliDebug(2, Form("mail command: %s", mailCommand.Data()));
3074
3075 Bool_t result = gSystem->Exec(mailCommand.Data());
3076
3077 return result == 0;
3078}
3079
3080//______________________________________________________________________________________________
3081Bool_t AliShuttle::SendMailToDCS()
3082{
3083 //
3084 // sends a mail to the DCS experts in case of DCS error
3085 //
3086
3087 if (fTestMode != kNone)
3088 return kTRUE;
3089
3090 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
3091 if (dir == NULL)
3092 {
3093 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
3094 {
3095 Log("SHUTTLE", Form("SendMailToDCS - Can't open directory <%s>", GetShuttleLogDir()));
3096 return kFALSE;
3097 }
3098
3099 } else {
3100 gSystem->FreeDirectory(dir);
3101 }
3102
3103 TString bodyFileName;
3104 bodyFileName.Form("%s/mail.body", GetShuttleLogDir());
3105 gSystem->ExpandPathName(bodyFileName);
3106
3107 ofstream mailBody;
3108 mailBody.open(bodyFileName, ofstream::out);
3109
3110 if (!mailBody.is_open())
3111 {
3112 Log("SHUTTLE", Form("SendMailToDCS - Could not open mail body file %s", bodyFileName.Data()));
3113 return kFALSE;
3114 }
3115
3116 TString to="Vladimir.Fekete@cern.ch, Svetozar.Kapusta@cern.ch";
3117 //TString to="alberto.colla@cern.ch";
3118 AliDebug(2, Form("to: %s",to.Data()));
3119
3120 if (to.IsNull()) {
3121 Log("SHUTTLE", "List of detector responsibles not yet set!");
3122 return kFALSE;
3123 }
3124
3125 TString cc="alberto.colla@cern.ch";
3126
3127 TString subject = Form("Retrieval of data points for %s FAILED in run %d !",
3128 fCurrentDetector.Data(), GetCurrentRun());
3129 AliDebug(2, Form("subject: %s", subject.Data()));
3130
3131 TString body = Form("Dear DCS experts, \n\n");
3132 body += Form("SHUTTLE couldn\'t retrieve the data points for detector %s "
3133 "in run %d!!\n\n", fCurrentDetector.Data(), GetCurrentRun());
3134 body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n", fCurrentDetector.Data());
3135 body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
3136 body += Form("Find the %s log for the current run on \n\n"
3137 "\thttp://pcalishuttle01.cern.ch:8880/logs/%s_%d.log \n\n",
3138 fCurrentDetector.Data(), fCurrentDetector.Data(), GetCurrentRun());
3139 body += Form("The last 10 lines of %s log file are following:\n\n");
3140
3141 AliDebug(2, Form("Body begin: %s", body.Data()));
3142
3143 mailBody << body.Data();
3144 mailBody.close();
3145 mailBody.open(bodyFileName, ofstream::out | ofstream::app);
3146
3147 TString logFileName = Form("%s/%s_%d.log", GetShuttleLogDir(), fCurrentDetector.Data(), GetCurrentRun());
3148 TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
3149 if (gSystem->Exec(tailCommand.Data()))
3150 {
3151 mailBody << Form("%s log file not found ...\n\n", fCurrentDetector.Data());
3152 }
3153
3154 TString endBody = Form("------------------------------------------------------\n\n");
36c99a6a 3155 endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
3156 endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
546242fb 3157 endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
57c1a579 3158
909732f7 3159 AliDebug(2, Form("Body end: %s", endBody.Data()));
57c1a579 3160
3161 mailBody << endBody.Data();
3162
3163 mailBody.close();
3164
3165 // send mail!
3166 TString mailCommand = Form("mail -s \"%s\" -c %s %s < %s",
3167 subject.Data(),
3168 cc.Data(),
3169 to.Data(),
3170 bodyFileName.Data());
909732f7 3171 AliDebug(2, Form("mail command: %s", mailCommand.Data()));
57c1a579 3172
3173 Bool_t result = gSystem->Exec(mailCommand.Data());
3174
3175 return result == 0;
3176}
d386d623 3177
441b0e9c 3178//______________________________________________________________________________________________
9827400b 3179const char* AliShuttle::GetRunType()
441b0e9c 3180{
9827400b 3181 //
3182 // returns run type read from "run type" logbook
3183 //
441b0e9c 3184
3185 if(!fLogbookEntry) {
3186 AliError("No logbook entry!");
3187 return 0;
3188 }
3189
9827400b 3190 return fLogbookEntry->GetRunType();
441b0e9c 3191}
3192
4859271b 3193//______________________________________________________________________________________________
3194Bool_t AliShuttle::GetHLTStatus()
3195{
3196 // Return HLT status (ON=1 OFF=0)
3197 // Converts the HLT status from the status string read in the run logbook (not just a bool)
3198
3199 if(!fLogbookEntry) {
3200 AliError("No logbook entry!");
3201 return 0;
3202 }
3203
3204 // TODO implement when HLTStatus is inserted in run logbook
3205 //TString hltStatus = fLogbookEntry->GetRunParameter("HLTStatus");
3206 //if(hltStatus == "OFF") {return kFALSE};
3207
3208 return kTRUE;
3209}
3210
d386d623 3211//______________________________________________________________________________________________
3212void AliShuttle::SetShuttleTempDir(const char* tmpDir)
3213{
9827400b 3214 //
3215 // sets Shuttle temp directory
3216 //
d386d623 3217
3218 fgkShuttleTempDir = gSystem->ExpandPathName(tmpDir);
3219}
3220
3221//______________________________________________________________________________________________
3222void AliShuttle::SetShuttleLogDir(const char* logDir)
3223{
9827400b 3224 //
3225 // sets Shuttle log directory
3226 //
d386d623 3227
3228 fgkShuttleLogDir = gSystem->ExpandPathName(logDir);
3229}