Fixed bug in raw data unpacking; Added pedestal to signal fit; Added logic to deal...
[u/mrichter/AliRoot.git] / SHUTTLE / AliShuttle.cxx
CommitLineData
73abe331 1/**************************************************************************
2 * Copyright(c) 1998-1999, ALICE Experiment at CERN, All rights reserved. *
3 * *
4 * Author: The ALICE Off-line Project. *
5 * Contributors are mentioned in the code where appropriate. *
6 * *
7 * Permission to use, copy, modify and distribute this software and its *
8 * documentation strictly for non-commercial purposes is hereby granted *
9 * without fee, provided that the above copyright notice appears in all *
10 * copies and that both the copyright notice and this permission notice *
11 * appear in the supporting documentation. The authors make no claims *
12 * about the suitability of this software for any purpose. It is *
13 * provided "as is" without express or implied warranty. *
14 **************************************************************************/
15
16/*
17$Log$
d524ade6 18Revision 1.60 2007/10/29 18:06:16 acolla
19
20New function StoreRunMetadataFile added to preprocessor and Shuttle interface
21This function can be used by GRP only. It stores raw data tags merged file to the
22raw data folder (e.g. /alice/data/2008/LHC08a/000099999/Raw).
23
24KNOWN ISSUES:
25
261. Shuttle cannot write to /alice/data/ because it belongs to alidaq. Tag file is stored in /alice/simulation/... for the time being.
272. Due to a bug in TAlien::Mkdir, the creation of a folder in recursive mode (-p option) does not work. The problem
28has been corrected in the root package on the Shuttle machine.
29
c88ad5db 30Revision 1.59 2007/10/05 12:40:55 acolla
31
32Result error code added to AliDCSClient data members (it was "lost" with the new implementation of TMap* GetAliasValues and GetDPValues).
33
1790d4b7 34Revision 1.58 2007/09/28 15:27:40 acolla
35
36AliDCSClient "multiSplit" option added in the DCS configuration
37in AliDCSMessage: variable MAX_BODY_SIZE set to 500000
38
542b6cc8 39Revision 1.57 2007/09/27 16:53:13 acolla
40Detectors can have more than one AMANDA server. SHUTTLE queries the servers sequentially,
41merges the dcs aliases/DPs in one TMap and sends it to the preprocessor.
42
1d172743 43Revision 1.56 2007/09/14 16:46:14 jgrosseo
441) Connect and Close are called before and after each query, so one can
45keep the same AliDCSClient object.
462) The splitting of a query is moved to GetDPValues/GetAliasValues.
473) Splitting interval can be specified in constructor
48
b41b252a 49Revision 1.55 2007/08/06 12:26:40 acolla
50Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
51read from the run logbook.
52
4859271b 53Revision 1.54 2007/07/12 09:51:25 jgrosseo
54removed duplicated log message in GetFile
55
4f0749a8 56Revision 1.53 2007/07/12 09:26:28 jgrosseo
57updating hlt fxs base path
58
42fde080 59Revision 1.52 2007/07/12 08:06:45 jgrosseo
60adding log messages in getfile... functions
61adding not implemented copy constructor in alishuttleconfigholder
62
1bcd28db 63Revision 1.51 2007/07/03 17:24:52 acolla
64root moved to v5-16-00. TFileMerger->Cp moved to TFile::Cp.
65
a986b218 66Revision 1.50 2007/07/02 17:19:32 acolla
67preprocessor is run in a temp directory that is removed when process is finished.
68
5bac2bde 69Revision 1.49 2007/06/29 10:45:06 acolla
70Number of columns in MySql Shuttle logbook increased by one (HLT added)
71
db99d43e 72Revision 1.48 2007/06/21 13:06:19 acolla
73GetFileSources returns dummy list with 1 source if system=DCS (better than
74returning error as it was)
75
6297b37d 76Revision 1.47 2007/06/19 17:28:56 acolla
77HLT updated; missing map bug removed.
78
dc25836b 79Revision 1.46 2007/06/09 13:01:09 jgrosseo
80Switching to retrieval of several DCS DPs at a time (multiDPrequest)
81
a038aa70 82Revision 1.45 2007/05/30 06:35:20 jgrosseo
83Adding functionality to the Shuttle/TestShuttle:
84o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
85o) Function to retrieve list of IDs for a given source (GetFileIDs)
86These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
87Example code has been added to the TestProcessor in TestShuttle
88
4a33bdd9 89Revision 1.44 2007/05/11 16:09:32 acolla
90Reference files for ITS, MUON and PHOS are now stored in OfflineDetName/OnlineDetName/run_...
91example: ITS/SPD/100_filename.root
92
2d9019b4 93Revision 1.43 2007/05/10 09:59:51 acolla
94Various bug fixes in StoreRefFilesToGrid; Cleaning of reference storage before processing detector (CleanReferenceStorage)
95
546242fb 96Revision 1.42 2007/05/03 08:01:39 jgrosseo
97typo in last commit :-(
98
8b739301 99Revision 1.41 2007/05/03 08:00:48 jgrosseo
100fixing log message when pp want to skip dcs value retrieval
101
651fdaab 102Revision 1.40 2007/04/27 07:06:48 jgrosseo
103GetFileSources returns empty list in case of no files, but successful query
104No mails sent in testmode
105
86aa42c3 106Revision 1.39 2007/04/17 12:43:57 acolla
107Correction in StoreOCDB; change of text in mail to detector expert
108
26758fce 109Revision 1.38 2007/04/12 08:26:18 jgrosseo
110updated comment
111
3c2a21c8 112Revision 1.37 2007/04/10 16:53:14 jgrosseo
113redirecting sub detector stdout, stderr to sub detector log file
114
3d8bc902 115Revision 1.35 2007/04/04 16:26:38 acolla
1161. Re-organization of function calls in TestPreprocessor to make it more meaningful.
1172. Added missing dependency in test preprocessors.
1183. in AliShuttle.cxx: processing time and memory consumption info on a single line.
119
886d60e6 120Revision 1.34 2007/04/04 10:33:36 jgrosseo
1211) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
122In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
123
1242) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
125
1263) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
127
1284) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
129
1305) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
131If you always need DCS data (like before), you do not need to implement it.
132
1336) The run type has been added to the monitoring page
134
9827400b 135Revision 1.33 2007/04/03 13:56:01 acolla
136Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
137run type.
138
3301427a 139Revision 1.32 2007/02/28 10:41:56 acolla
140Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
141AliPreprocessor::GetRunType() function.
142Added some ldap definition files.
143
d386d623 144Revision 1.30 2007/02/13 11:23:21 acolla
145Moved getters and setters of Shuttle's main OCDB/Reference, local
146OCDB/Reference, temp and log folders to AliShuttleInterface
147
9d733021 148Revision 1.27 2007/01/30 17:52:42 jgrosseo
149adding monalisa monitoring
150
e7f62f16 151Revision 1.26 2007/01/23 19:20:03 acolla
152Removed old ldif files, added TOF, MCH ldif files. Added some options in
153AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
154SetShuttleLogDir
155
36c99a6a 156Revision 1.25 2007/01/15 19:13:52 acolla
157Moved some AliInfo to AliDebug in SendMail function
158
fc5a4708 159Revision 1.21 2006/12/07 08:51:26 jgrosseo
160update (alberto):
161table, db names in ldap configuration
162added GRP preprocessor
163DCS data can also be retrieved by data point
164
2c15234c 165Revision 1.20 2006/11/16 16:16:48 jgrosseo
166introducing strict run ordering flag
167removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
168
be48e3ea 169Revision 1.19 2006/11/06 14:23:04 jgrosseo
170major update (Alberto)
171o) reading of run parameters from the logbook
172o) online offline naming conversion
173o) standalone DCSclient package
174
eba76848 175Revision 1.18 2006/10/20 15:22:59 jgrosseo
176o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
177o) Merging Collect, CollectAll, CollectNew function
178o) Removing implementation of empty copy constructors (declaration still there!)
179
cb343cfd 180Revision 1.17 2006/10/05 16:20:55 jgrosseo
181adapting to new CDB classes
182
6ec0e06c 183Revision 1.16 2006/10/05 15:46:26 jgrosseo
184applying to the new interface
185
481441a2 186Revision 1.15 2006/10/02 16:38:39 jgrosseo
187update (alberto):
188fixed memory leaks
189storing of objects that failed to be stored to the grid before
190interfacing of shuttle status table in daq system
191
2bb7b766 192Revision 1.14 2006/08/29 09:16:05 jgrosseo
193small update
194
85a80aa9 195Revision 1.13 2006/08/15 10:50:00 jgrosseo
196effc++ corrections (alberto)
197
4f0ab988 198Revision 1.12 2006/08/08 14:19:29 jgrosseo
199Update to shuttle classes (Alberto)
200
201- Possibility to set the full object's path in the Preprocessor's and
202Shuttle's Store functions
203- Possibility to extend the object's run validity in the same classes
204("startValidity" and "validityInfinite" parameters)
205- Implementation of the StoreReferenceData function to store reference
206data in a dedicated CDB storage.
207
84090f85 208Revision 1.11 2006/07/21 07:37:20 jgrosseo
209last run is stored after each run
210
7bfb2090 211Revision 1.10 2006/07/20 09:54:40 jgrosseo
212introducing status management: The processing per subdetector is divided into several steps,
213after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
214can keep track of the number of failures and skips further processing after a certain threshold is
215exceeded. These thresholds can be configured in LDAP.
216
5164a766 217Revision 1.9 2006/07/19 10:09:55 jgrosseo
218new configuration, accesst to DAQ FES (Alberto)
219
57f50b3c 220Revision 1.8 2006/07/11 12:44:36 jgrosseo
221adding parameters for extended validity range of data produced by preprocessor
222
17111222 223Revision 1.7 2006/07/10 14:37:09 jgrosseo
224small fix + todo comment
225
e090413b 226Revision 1.6 2006/07/10 13:01:41 jgrosseo
227enhanced storing of last sucessfully processed run (alberto)
228
a7160fe9 229Revision 1.5 2006/07/04 14:59:57 jgrosseo
230revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
231
45a493ce 232Revision 1.4 2006/06/12 09:11:16 jgrosseo
233coding conventions (Alberto)
234
58bc3020 235Revision 1.3 2006/06/06 14:26:40 jgrosseo
236o) removed files that were moved to STEER
237o) shuttle updated to follow the new interface (Alberto)
238
b948db8d 239Revision 1.2 2006/03/07 07:52:34 hristov
240New version (B.Yordanov)
241
d477ad88 242Revision 1.6 2005/11/19 17:19:14 byordano
243RetrieveDATEEntries and RetrieveConditionsData added
244
245Revision 1.5 2005/11/19 11:09:27 byordano
246AliShuttle declaration added
247
248Revision 1.4 2005/11/17 17:47:34 byordano
249TList changed to TObjArray
250
251Revision 1.3 2005/11/17 14:43:23 byordano
252import to local CVS
253
254Revision 1.1.1.1 2005/10/28 07:33:58 hristov
255Initial import as subdirectory in AliRoot
256
73abe331 257Revision 1.2 2005/09/13 08:41:15 byordano
258default startTime endTime added
259
260Revision 1.4 2005/08/30 09:13:02 byordano
261some docs added
262
263Revision 1.3 2005/08/29 21:15:47 byordano
264some docs added
265
266*/
267
268//
269// This class is the main manager for AliShuttle.
270// It organizes the data retrieval from DCS and call the
b948db8d 271// interface methods of AliPreprocessor.
73abe331 272// For every detector in AliShuttleConfgi (see AliShuttleConfig),
273// data for its set of aliases is retrieved. If there is registered
b948db8d 274// AliPreprocessor for this detector then it will be used
275// accroding to the schema (see AliPreprocessor).
276// If there isn't registered AliPreprocessor than the retrieved
73abe331 277// data is stored automatically to the undelying AliCDBStorage.
278// For detSpec is used the alias name.
279//
280
281#include "AliShuttle.h"
282
283#include "AliCDBManager.h"
284#include "AliCDBStorage.h"
285#include "AliCDBId.h"
84090f85 286#include "AliCDBRunRange.h"
287#include "AliCDBPath.h"
5164a766 288#include "AliCDBEntry.h"
73abe331 289#include "AliShuttleConfig.h"
eba76848 290#include "DCSClient/AliDCSClient.h"
73abe331 291#include "AliLog.h"
b948db8d 292#include "AliPreprocessor.h"
5164a766 293#include "AliShuttleStatus.h"
2bb7b766 294#include "AliShuttleLogbookEntry.h"
73abe331 295
57f50b3c 296#include <TSystem.h>
58bc3020 297#include <TObject.h>
b948db8d 298#include <TString.h>
57f50b3c 299#include <TTimeStamp.h>
73abe331 300#include <TObjString.h>
57f50b3c 301#include <TSQLServer.h>
302#include <TSQLResult.h>
303#include <TSQLRow.h>
cb343cfd 304#include <TMutex.h>
9827400b 305#include <TSystemDirectory.h>
306#include <TSystemFile.h>
a986b218 307#include <TFile.h>
9827400b 308#include <TGrid.h>
309#include <TGridResult.h>
73abe331 310
e7f62f16 311#include <TMonaLisaWriter.h>
312
5164a766 313#include <fstream>
314
cb343cfd 315#include <sys/types.h>
316#include <sys/wait.h>
317
73abe331 318ClassImp(AliShuttle)
319
b948db8d 320//______________________________________________________________________________________________
321AliShuttle::AliShuttle(const AliShuttleConfig* config,
322 UInt_t timeout, Int_t retries):
4f0ab988 323fConfig(config),
324fTimeout(timeout), fRetries(retries),
325fPreprocessorMap(),
2bb7b766 326fLogbookEntry(0),
eba76848 327fCurrentDetector(),
85a80aa9 328fStatusEntry(0),
cb343cfd 329fMonitoringMutex(0),
eba76848 330fLastActionTime(0),
e7f62f16 331fLastAction(),
9827400b 332fMonaLisa(0),
333fTestMode(kNone),
ffa29e93 334fReadTestMode(kFALSE),
335fOutputRedirected(kFALSE)
73abe331 336{
337 //
338 // config: AliShuttleConfig used
73abe331 339 // timeout: timeout used for AliDCSClient connection
340 // retries: the number of retries in case of connection error.
341 //
342
57f50b3c 343 if (!fConfig->IsValid()) AliFatal("********** !!!!! Invalid configuration !!!!! **********");
be48e3ea 344 for(int iSys=0;iSys<4;iSys++) {
57f50b3c 345 fServer[iSys]=0;
be48e3ea 346 if (iSys < 3)
2c15234c 347 fFXSlist[iSys].SetOwner(kTRUE);
57f50b3c 348 }
2bb7b766 349 fPreprocessorMap.SetOwner(kTRUE);
be48e3ea 350
351 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
352 fFirstUnprocessed[iDet] = kFALSE;
353
cb343cfd 354 fMonitoringMutex = new TMutex();
58bc3020 355}
356
b948db8d 357//______________________________________________________________________________________________
57f50b3c 358AliShuttle::~AliShuttle()
58bc3020 359{
9827400b 360 //
361 // destructor
362 //
58bc3020 363
b948db8d 364 fPreprocessorMap.DeleteAll();
be48e3ea 365 for(int iSys=0;iSys<4;iSys++)
57f50b3c 366 if(fServer[iSys]) {
367 fServer[iSys]->Close();
368 delete fServer[iSys];
eba76848 369 fServer[iSys] = 0;
57f50b3c 370 }
2bb7b766 371
372 if (fStatusEntry){
373 delete fStatusEntry;
374 fStatusEntry = 0;
375 }
cb343cfd 376
377 if (fMonitoringMutex)
378 {
379 delete fMonitoringMutex;
380 fMonitoringMutex = 0;
381 }
73abe331 382}
383
b948db8d 384//______________________________________________________________________________________________
57f50b3c 385void AliShuttle::RegisterPreprocessor(AliPreprocessor* preprocessor)
58bc3020 386{
73abe331 387 //
b948db8d 388 // Registers new AliPreprocessor.
73abe331 389 // It uses GetName() for indentificator of the pre processor.
390 // The pre processor is registered it there isn't any other
391 // with the same identificator (GetName()).
392 //
393
eba76848 394 const char* detName = preprocessor->GetName();
395 if(GetDetPos(detName) < 0)
396 AliFatal(Form("********** !!!!! Invalid detector name: %s !!!!! **********", detName));
397
398 if (fPreprocessorMap.GetValue(detName)) {
399 AliWarning(Form("AliPreprocessor %s is already registered!", detName));
73abe331 400 return;
401 }
402
eba76848 403 fPreprocessorMap.Add(new TObjString(detName), preprocessor);
73abe331 404}
b948db8d 405//______________________________________________________________________________________________
3301427a 406Bool_t AliShuttle::Store(const AliCDBPath& path, TObject* object,
84090f85 407 AliCDBMetaData* metaData, Int_t validityStart, Bool_t validityInfinite)
73abe331 408{
9827400b 409 // Stores a CDB object in the storage for offline reconstruction. Objects that are not needed for
410 // offline reconstruction, but should be stored anyway (e.g. for debugging) should NOT be stored
411 // using this function. Use StoreReferenceData instead!
412 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
413 // finishes the data are transferred to the main storage (Grid).
b948db8d 414
3301427a 415 return StoreLocally(fgkLocalCDB, path, object, metaData, validityStart, validityInfinite);
84090f85 416}
417
418//______________________________________________________________________________________________
3301427a 419Bool_t AliShuttle::StoreReferenceData(const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData)
84090f85 420{
9827400b 421 // Stores a CDB object in the storage for reference data. This objects will not be available during
422 // offline reconstrunction. Use this function for reference data only!
423 // It calls StoreLocally function which temporarily stores the data locally; when the preprocessor
424 // finishes the data are transferred to the main storage (Grid).
85a80aa9 425
3301427a 426 return StoreLocally(fgkLocalRefStorage, path, object, metaData);
85a80aa9 427}
428
429//______________________________________________________________________________________________
3301427a 430Bool_t AliShuttle::StoreLocally(const TString& localUri,
85a80aa9 431 const AliCDBPath& path, TObject* object, AliCDBMetaData* metaData,
432 Int_t validityStart, Bool_t validityInfinite)
433{
9827400b 434 // Store object temporarily in local storage. Parameters are passed by Store and StoreReferenceData functions.
435 // when the preprocessor finishes the data are transferred to the main storage (Grid).
436 // The parameters are:
437 // 1) Uri of the backup storage (Local)
438 // 2) the object's path.
439 // 3) the object to be stored
440 // 4) the metaData to be associated with the object
441 // 5) the validity start run number w.r.t. the current run,
442 // if the data is valid only for this run leave the default 0
443 // 6) specifies if the calibration data is valid for infinity (this means until updated),
444 // typical for calibration runs, the default is kFALSE
445 //
446 // returns 0 if fail, 1 otherwise
84090f85 447
9827400b 448 if (fTestMode & kErrorStorage)
449 {
450 Log(fCurrentDetector, "StoreLocally - In TESTMODE - Simulating error while storing locally");
451 return kFALSE;
452 }
453
3301427a 454 const char* cdbType = (localUri == fgkLocalCDB) ? "CDB" : "Reference";
2bb7b766 455
85a80aa9 456 Int_t firstRun = GetCurrentRun() - validityStart;
84090f85 457 if(firstRun < 0) {
9827400b 458 AliWarning("First valid run happens to be less than 0! Setting it to 0.");
84090f85 459 firstRun=0;
460 }
461
462 Int_t lastRun = -1;
463 if(validityInfinite) {
464 lastRun = AliCDBRunRange::Infinity();
465 } else {
466 lastRun = GetCurrentRun();
467 }
468
3301427a 469 // Version is set to current run, it will be used later to transfer data to Grid
470 AliCDBId id(path, firstRun, lastRun, GetCurrentRun(), -1);
2bb7b766 471
472 if(! dynamic_cast<TObjString*> (metaData->GetProperty("RunUsed(TObjString)"))){
473 TObjString runUsed = Form("%d", GetCurrentRun());
9e080f92 474 metaData->SetProperty("RunUsed(TObjString)", runUsed.Clone());
2bb7b766 475 }
84090f85 476
3301427a 477 Bool_t result = kFALSE;
84090f85 478
3301427a 479 if (!(AliCDBManager::Instance()->GetStorage(localUri))) {
480 Log("SHUTTLE", Form("StoreLocally - Cannot activate local %s storage", cdbType));
84090f85 481 } else {
3301427a 482 result = AliCDBManager::Instance()->GetStorage(localUri)
84090f85 483 ->Put(object, id, metaData);
484 }
485
486 if(!result) {
487
9827400b 488 Log(fCurrentDetector, Form("StoreLocally - Can't store object <%s>!", id.ToString().Data()));
3301427a 489 }
2bb7b766 490
3301427a 491 return result;
492}
84090f85 493
3301427a 494//______________________________________________________________________________________________
495Bool_t AliShuttle::StoreOCDB()
496{
9827400b 497 //
498 // Called when preprocessor ends successfully or when previous storage attempt failed (kStoreError status)
499 // Calls underlying StoreOCDB(const char*) function twice, for OCDB and Reference storage.
500 // Then calls StoreRefFilesToGrid to store reference files.
501 //
502
503 if (fTestMode & kErrorGrid)
504 {
505 Log("SHUTTLE", "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
506 Log(fCurrentDetector, "StoreOCDB - In TESTMODE - Simulating error while storing in the Grid");
507 return kFALSE;
508 }
509
c88ad5db 510 Log("SHUTTLE","StoreOCDB - Storing OCDB data ...");
86aa42c3 511 Bool_t resultCDB = StoreOCDB(fgkMainCDB);
512
c88ad5db 513 Log("SHUTTLE","StoreOCDB - Storing reference data ...");
3301427a 514 Bool_t resultRef = StoreOCDB(fgkMainRefStorage);
9827400b 515
c88ad5db 516 Log("SHUTTLE","StoreOCDB - Storing reference files ...");
517 Bool_t resultRefFiles = CopyFilesToGrid("reference");
518
519 Bool_t resultMetadata = kTRUE;
520 if(fCurrentDetector == "GRP")
521 {
522 Log("StoreOCDB - SHUTTLE","Storing Run Metadata file ...");
523 resultMetadata = CopyFilesToGrid("metadata");
524 }
9827400b 525
c88ad5db 526 return resultCDB && resultRef && resultRefFiles && resultMetadata;
3301427a 527}
528
529//______________________________________________________________________________________________
530Bool_t AliShuttle::StoreOCDB(const TString& gridURI)
531{
532 //
533 // Called by StoreOCDB(), performs actual storage to the main OCDB and reference storages (Grid)
534 //
535
536 TObjArray* gridIds=0;
537
538 Bool_t result = kTRUE;
539
540 const char* type = 0;
541 TString localURI;
542 if(gridURI == fgkMainCDB) {
543 type = "OCDB";
544 localURI = fgkLocalCDB;
545 } else if(gridURI == fgkMainRefStorage) {
546 type = "reference";
547 localURI = fgkLocalRefStorage;
548 } else {
549 AliError(Form("Invalid storage URI: %s", gridURI.Data()));
550 return kFALSE;
551 }
552
553 AliCDBManager* man = AliCDBManager::Instance();
554
555 AliCDBStorage *gridSto = man->GetStorage(gridURI);
556 if(!gridSto) {
557 Log("SHUTTLE",
558 Form("StoreOCDB - cannot activate main %s storage", type));
559 return kFALSE;
560 }
561
562 gridIds = gridSto->GetQueryCDBList();
563
564 // get objects previously stored in local CDB
565 AliCDBStorage *localSto = man->GetStorage(localURI);
566 if(!localSto) {
567 Log("SHUTTLE",
568 Form("StoreOCDB - cannot activate local %s storage", type));
569 return kFALSE;
570 }
571 AliCDBPath aPath(GetOfflineDetName(fCurrentDetector.Data()),"*","*");
572 // Local objects were stored with current run as Grid version!
573 TList* localEntries = localSto->GetAll(aPath.GetPath(), GetCurrentRun(), GetCurrentRun());
574 localEntries->SetOwner(1);
575
576 // loop on local stored objects
577 TIter localIter(localEntries);
578 AliCDBEntry *aLocEntry = 0;
579 while((aLocEntry = dynamic_cast<AliCDBEntry*> (localIter.Next()))){
580 aLocEntry->SetOwner(1);
581 AliCDBId aLocId = aLocEntry->GetId();
582 aLocEntry->SetVersion(-1);
583 aLocEntry->SetSubVersion(-1);
584
585 // If local object is valid up to infinity we store it only if it is
586 // the first unprocessed run!
587 if (aLocId.GetLastRun() == AliCDBRunRange::Infinity() &&
588 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
589 {
590 Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
591 "there are previous unprocessed runs!",
592 fCurrentDetector.Data(), aLocId.GetPath().Data()));
593 continue;
594 }
595
596 // loop on Grid valid Id's
597 Bool_t store = kTRUE;
598 TIter gridIter(gridIds);
599 AliCDBId* aGridId = 0;
600 while((aGridId = dynamic_cast<AliCDBId*> (gridIter.Next()))){
601 if(aGridId->GetPath() != aLocId.GetPath()) continue;
602 // skip all objects valid up to infinity
603 if(aGridId->GetLastRun() == AliCDBRunRange::Infinity()) continue;
604 // if we get here, it means there's already some more recent object stored on Grid!
605 store = kFALSE;
606 break;
607 }
608
609 // If we get here, the file can be stored!
610 Bool_t storeOk = gridSto->Put(aLocEntry);
611 if(!store || storeOk){
612
613 if (!store)
614 {
615 Log(fCurrentDetector.Data(),
616 Form("StoreOCDB - A more recent object already exists in %s storage: <%s>",
617 type, aGridId->ToString().Data()));
618 } else {
619 Log("SHUTTLE",
620 Form("StoreOCDB - Object <%s> successfully put into %s storage",
621 aLocId.ToString().Data(), type));
2d9019b4 622 Log(fCurrentDetector.Data(),
623 Form("StoreOCDB - Object <%s> successfully put into %s storage",
624 aLocId.ToString().Data(), type));
3301427a 625 }
84090f85 626
3301427a 627 // removing local filename...
628 TString filename;
629 localSto->IdToFilename(aLocId, filename);
c88ad5db 630 Log("SHUTTLE", Form("StoreOCDB - Removing local file %s", filename.Data()));
3301427a 631 RemoveFile(filename.Data());
632 continue;
633 } else {
634 Log("SHUTTLE",
635 Form("StoreOCDB - Grid %s storage of object <%s> failed",
636 type, aLocId.ToString().Data()));
2d9019b4 637 Log(fCurrentDetector.Data(),
638 Form("StoreOCDB - Grid %s storage of object <%s> failed",
639 type, aLocId.ToString().Data()));
3301427a 640 result = kFALSE;
b948db8d 641 }
642 }
3301427a 643 localEntries->Clear();
2bb7b766 644
b948db8d 645 return result;
3301427a 646}
647
546242fb 648//______________________________________________________________________________________________
649Bool_t AliShuttle::CleanReferenceStorage(const char* detector)
650{
2d9019b4 651 // clears the directory used to store reference files of a given subdetector
546242fb 652
653 AliCDBManager* man = AliCDBManager::Instance();
654 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
2d9019b4 655 TString localBaseFolder = sto->GetBaseFolder();
656
657 TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
658
d524ade6 659 Log("SHUTTLE", Form("CleanReferenceStorage - Cleaning %s", targetDir.Data()));
2d9019b4 660
661 TString begin;
662 begin.Form("%d_", GetCurrentRun());
663
664 TSystemDirectory* baseDir = new TSystemDirectory("/", targetDir);
665 if (!baseDir)
666 return kTRUE;
667
668 TList* dirList = baseDir->GetListOfFiles();
669 delete baseDir;
670
671 if (!dirList) return kTRUE;
672
673 if (dirList->GetEntries() < 3)
674 {
675 delete dirList;
676 return kTRUE;
677 }
678
679 Int_t nDirs = 0, nDel = 0;
680 TIter dirIter(dirList);
681 TSystemFile* entry = 0;
546242fb 682
2d9019b4 683 Bool_t success = kTRUE;
546242fb 684
2d9019b4 685 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
686 {
687 if (entry->IsDirectory())
688 continue;
689
690 TString fileName(entry->GetName());
691 if (!fileName.BeginsWith(begin))
692 continue;
693
694 nDirs++;
695
696 // delete file
697 Int_t result = gSystem->Unlink(fileName.Data());
698
699 if (result)
700 {
d524ade6 701 Log("SHUTTLE", Form("CleanReferenceStorage - Could not delete file %s!", fileName.Data()));
2d9019b4 702 success = kFALSE;
703 } else {
704 nDel++;
705 }
706 }
707
708 if(nDirs > 0)
709 Log("SHUTTLE", Form("CleanReferenceStorage - %d (over %d) reference files in folder %s were deleted.",
710 nDel, nDirs, targetDir.Data()));
711
712
713 delete dirList;
714 return success;
715
716
717
718
719
546242fb 720
721 Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
722 if (result == 0)
723 {
724 // delete directory
d524ade6 725 result = gSystem->Exec(Form("rm -rf %s", targetDir.Data()));
546242fb 726 if (result != 0)
727 {
d524ade6 728 Log("SHUTTLE", Form("CleanReferenceStorage - Could not clean directory %s", targetDir.Data()));
546242fb 729 return kFALSE;
730 }
731 }
732
733 result = gSystem->mkdir(targetDir, kTRUE);
734 if (result != 0)
735 {
c88ad5db 736 Log("SHUTTLE", Form("CleanReferenceStorage - Error creating base directory %s", targetDir.Data()));
546242fb 737 return kFALSE;
738 }
739
740 return kTRUE;
741}
742
9827400b 743//______________________________________________________________________________________________
744Bool_t AliShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
745{
746 //
3c2a21c8 747 // Stores reference file directly (without opening it). This function stores the file locally.
9827400b 748 //
3c2a21c8 749 // The file is stored under the following location:
750 // <base folder of local reference storage>/<DET>/<RUN#>_<gridFileName>
751 // where <gridFileName> is the second parameter given to the function
752 //
9827400b 753
754 if (fTestMode & kErrorStorage)
755 {
756 Log(fCurrentDetector, "StoreReferenceFile - In TESTMODE - Simulating error while storing locally");
757 return kFALSE;
758 }
759
760 AliCDBManager* man = AliCDBManager::Instance();
761 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
762
763 TString localBaseFolder = sto->GetBaseFolder();
764
d524ade6 765 TString target = GetRefFilePrefix(localBaseFolder.Data(), detector);
766 target.Append(Form("/%d_%s", GetCurrentRun(), gridFileName));
9827400b 767
d524ade6 768 return CopyFileLocally(localFile, target);
c88ad5db 769}
770
771//______________________________________________________________________________________________
772Bool_t AliShuttle::StoreRunMetadataFile(const char* localFile, const char* gridFileName)
773{
774 //
775 // Stores Run metadata file to the Grid, in the run folder
776 //
777 // Only GRP can call this function.
778
779 if (fTestMode & kErrorStorage)
780 {
781 Log(fCurrentDetector, "StoreRunMetaDataFile - In TESTMODE - Simulating error while storing locally");
782 return kFALSE;
783 }
784
785 AliCDBManager* man = AliCDBManager::Instance();
786 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
787
788 TString localBaseFolder = sto->GetBaseFolder();
789
790 // Build Run level folder
791 // folder = /alice/data/year/lhcPeriod/runNb/Raw
792
793 TTimeStamp startTime(GetCurrentStartTime());
794
795 TString year = Form("%d",startTime.GetDate());
796 year = year(0,4);
797
798 TString lhcPeriod = GetRunParameter("LHCperiod");
799
800 if (lhcPeriod.Length() == 0)
801 {
802 Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
803 return 0;
804 }
805
806 // TODO: currently SHUTTLE cannot write in /alice/data/ !!!!!
d524ade6 807 //TString target = Form("%s/GRP/RunMetadata/alice/data/%s/%s/%d/Raw/%s",
c88ad5db 808 // localBaseFolder.Data(), year.Data(),
d524ade6 809 // lhcPeriod.Data(), GetCurrentRun(), gridFileName);
c88ad5db 810
d524ade6 811 TString target = Form("%s/GRP/RunMetadata/alice/simulation/%s/%s/%d/Raw/%s",
c88ad5db 812 localBaseFolder.Data(), year.Data(),
d524ade6 813 lhcPeriod.Data(), GetCurrentRun(), gridFileName);
814
815
c88ad5db 816
d524ade6 817 return CopyFileLocally(localFile, target);
c88ad5db 818}
819
820//______________________________________________________________________________________________
d524ade6 821Bool_t AliShuttle::CopyFileLocally(const char* localFile, const TString& target)
c88ad5db 822{
823 //
824 // Stores file locally. Called by StoreReferenceFile and StoreRunMetadataFile
d524ade6 825 // Files are temporarily stored in the local reference storage. When the preprocessor
826 // finishes, the Shuttle calls CopyFilesToGrid to transfer the files to AliEn
827 // (in reference or run level folders)
c88ad5db 828 //
829
d524ade6 830 TString targetDir(target(0, target.Last('/')));
831
832 //try to open base dir folder, if it does not exist
2d9019b4 833 void* dir = gSystem->OpenDirectory(targetDir.Data());
834 if (dir == NULL) {
835 if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
c88ad5db 836 Log("SHUTTLE", Form("StoreFileLocally - Can't open directory <%s>", targetDir.Data()));
2d9019b4 837 return kFALSE;
838 }
839
840 } else {
841 gSystem->FreeDirectory(dir);
842 }
9827400b 843
546242fb 844 Int_t result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
9827400b 845 if (result)
846 {
c88ad5db 847 Log("SHUTTLE", Form("StoreFileLocally - %s does not exist", localFile));
546242fb 848 return kFALSE;
9827400b 849 }
546242fb 850
9827400b 851 result = gSystem->CopyFile(localFile, target);
852
853 if (result == 0)
854 {
c88ad5db 855 Log("SHUTTLE", Form("StoreFileLocally - File %s stored locally to %s", localFile, target.Data()));
9827400b 856 return kTRUE;
857 }
858 else
859 {
d524ade6 860 Log("SHUTTLE", Form("StoreFileLocally - Could not store file %s to %s! Error code = %d",
546242fb 861 localFile, target.Data(), result));
9827400b 862 return kFALSE;
863 }
c88ad5db 864
865
866
9827400b 867}
868
869//______________________________________________________________________________________________
c88ad5db 870Bool_t AliShuttle::CopyFilesToGrid(const char* type)
9827400b 871{
872 //
c88ad5db 873 // Transfers local files to the Grid. Local files can be reference files
874 // or run metadata file (from GRP only).
9827400b 875 //
c88ad5db 876 // According to the type (ref, metadata) the files are stored under the following location:
877 // ref --> <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
878 // metadata --> <run data folder>/<MetadataFileName>
86aa42c3 879 //
c88ad5db 880
9827400b 881 AliCDBManager* man = AliCDBManager::Instance();
882 AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
883 if (!sto)
884 return kFALSE;
885 TString localBaseFolder = sto->GetBaseFolder();
9827400b 886
c88ad5db 887 TString dir;
888 TString alienDir;
9827400b 889 TString begin;
9827400b 890
c88ad5db 891 if (strcmp(type, "reference") == 0)
892 {
893 dir = GetRefFilePrefix(localBaseFolder.Data(), fCurrentDetector.Data());
894 AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
895 if (!gridSto)
896 return kFALSE;
897 TString gridBaseFolder = gridSto->GetBaseFolder();
898 alienDir = GetRefFilePrefix(gridBaseFolder.Data(), fCurrentDetector.Data());
899 begin = Form("%d_", GetCurrentRun());
900 }
901 else if (strcmp(type, "metadata") == 0)
902 {
903 TTimeStamp startTime(GetCurrentStartTime());
904
905 TString year = Form("%d",startTime.GetDate());
906 year = year(0,4);
907
908 TString lhcPeriod = GetRunParameter("LHCperiod");
909
910 if (lhcPeriod.Length() == 0)
911 {
912 Log("SHUTTLE","CopyFilesToGrid - LHCPeriod not found in logbook!");
913 return 0;
914 }
915
916 // TODO: currently SHUTTLE cannot write in /alice/data/ !!!!!
917 //dir = Form("%s/GRP/RunMetadata/alice/data/%s/%s/%d/Raw",
918 // localBaseFolder.Data(), year.Data(),
919 // lhcPeriod.Data(), GetCurrentRun());
920 //alienDir = dir(dir.Index("/alice/data/"), dir.Length());
921
922 dir = Form("%s/GRP/RunMetadata/alice/simulation/%s/%s/%d/Raw",
923 localBaseFolder.Data(), year.Data(),
924 lhcPeriod.Data(), GetCurrentRun());
925 alienDir = dir(dir.Index("/alice/simulation/"), dir.Length());
926 begin = "";
927 }
928 else
929 {
930 Log("SHUTTLE", "CopyFilesToGrid - Unexpected: type label must be reference or metadata!");
931 return kFALSE;
932 }
933
9827400b 934 TSystemDirectory* baseDir = new TSystemDirectory("/", dir);
3d8bc902 935 if (!baseDir)
936 return kTRUE;
937
2d9019b4 938 TList* dirList = baseDir->GetListOfFiles();
939 delete baseDir;
940
941 if (!dirList) return kTRUE;
942
943 if (dirList->GetEntries() < 3)
3d8bc902 944 {
2d9019b4 945 delete dirList;
9827400b 946 return kTRUE;
3d8bc902 947 }
2d9019b4 948
546242fb 949 if (!gGrid)
950 {
c88ad5db 951 Log("SHUTTLE", "CopyFilesToGrid - Connection to Grid failed: Cannot continue!");
2d9019b4 952 delete dirList;
546242fb 953 return kFALSE;
954 }
955
2d9019b4 956 Int_t nDirs = 0, nTransfer = 0;
957 TIter dirIter(dirList);
958 TSystemFile* entry = 0;
959
9827400b 960 Bool_t success = kTRUE;
3d8bc902 961 Bool_t first = kTRUE;
9827400b 962
2d9019b4 963 while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
964 {
9827400b 965 if (entry->IsDirectory())
966 continue;
967
968 TString fileName(entry->GetName());
969 if (!fileName.BeginsWith(begin))
970 continue;
971
2d9019b4 972 nDirs++;
973
3d8bc902 974 if (first)
975 {
976 first = kFALSE;
c88ad5db 977 // check that folder exists, otherwise create it
3d8bc902 978 TGridResult* result = gGrid->Ls(alienDir.Data(), "a");
979
980 if (!result)
2d9019b4 981 {
982 delete dirList;
3d8bc902 983 return kFALSE;
2d9019b4 984 }
3d8bc902 985
546242fb 986 if (!result->GetFileName(1)) // TODO: It looks like element 0 is always 0!!
3d8bc902 987 {
c88ad5db 988 if (!gGrid->Mkdir(alienDir.Data(),"-p",0))
3d8bc902 989 {
c88ad5db 990 Log("SHUTTLE", Form("CopyFilesToGrid - Cannot create directory %s",
3d8bc902 991 alienDir.Data()));
2d9019b4 992 delete dirList;
3d8bc902 993 return kFALSE;
546242fb 994 } else {
c88ad5db 995 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s created", alienDir.Data()));
3d8bc902 996 }
997
546242fb 998 } else {
c88ad5db 999 Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s found", alienDir.Data()));
3d8bc902 1000 }
1001 }
1002
9827400b 1003 TString fullLocalPath;
1004 fullLocalPath.Form("%s/%s", dir.Data(), fileName.Data());
1005
1006 TString fullGridPath;
1007 fullGridPath.Form("alien://%s/%s", alienDir.Data(), fileName.Data());
1008
a986b218 1009 Bool_t result = TFile::Cp(fullLocalPath, fullGridPath);
9827400b 1010
1011 if (result)
1012 {
c88ad5db 1013 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s succeeded!",
1014 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1015 RemoveFile(fullLocalPath);
2d9019b4 1016 nTransfer++;
9827400b 1017 }
1018 else
1019 {
c88ad5db 1020 Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s FAILED!",
1021 fullLocalPath.Data(), fullGridPath.Data()));
9827400b 1022 success = kFALSE;
1023 }
1024 }
2d9019b4 1025
c88ad5db 1026 Log("SHUTTLE", Form("CopyFilesToGrid - %d (over %d) files in folder %s copied to Grid.",
1027 nTransfer, nDirs, dir.Data()));
2d9019b4 1028
1029
1030 delete dirList;
9827400b 1031 return success;
1032}
1033
2d9019b4 1034//______________________________________________________________________________________________
1035const char* AliShuttle::GetRefFilePrefix(const char* base, const char* detector)
1036{
1037 //
1038 // Get folder name of reference files
1039 //
1040
1041 TString offDetStr(GetOfflineDetName(detector));
1042 TString dir;
1043 if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
1044 {
1045 dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
1046 } else {
1047 dir.Form("%s/%s", base, offDetStr.Data());
1048 }
1049
1050 return dir.Data();
1051
1052
1053}
c88ad5db 1054
3301427a 1055//______________________________________________________________________________________________
1056void AliShuttle::CleanLocalStorage(const TString& uri)
1057{
9827400b 1058 //
1059 // Called in case the preprocessor is declared failed. Remove remaining objects from the local storages.
1060 //
3301427a 1061
1062 const char* type = 0;
1063 if(uri == fgkLocalCDB) {
1064 type = "OCDB";
1065 } else if(uri == fgkLocalRefStorage) {
546242fb 1066 type = "Reference";
3301427a 1067 } else {
1068 AliError(Form("Invalid storage URI: %s", uri.Data()));
1069 return;
1070 }
1071
1072 AliCDBManager* man = AliCDBManager::Instance();
b948db8d 1073
3301427a 1074 // open local storage
1075 AliCDBStorage *localSto = man->GetStorage(uri);
1076 if(!localSto) {
1077 Log("SHUTTLE",
1078 Form("CleanLocalStorage - cannot activate local %s storage", type));
1079 return;
1080 }
1081
1082 TString filename(Form("%s/%s/*/Run*_v%d_s*.root",
546242fb 1083 localSto->GetBaseFolder().Data(), GetOfflineDetName(fCurrentDetector.Data()), GetCurrentRun()));
3301427a 1084
c88ad5db 1085 AliDebug(2, Form("filename = %s", filename.Data()));
3301427a 1086
c88ad5db 1087 Log("SHUTTLE", Form("Removing remaining local files for run %d and detector %s ...",
3301427a 1088 GetCurrentRun(), fCurrentDetector.Data()));
1089
1090 RemoveFile(filename.Data());
1091
1092}
1093
1094//______________________________________________________________________________________________
1095void AliShuttle::RemoveFile(const char* filename)
1096{
9827400b 1097 //
1098 // removes local file
1099 //
3301427a 1100
1101 TString command(Form("rm -f %s", filename));
1102
1103 Int_t result = gSystem->Exec(command.Data());
1104 if(result != 0)
1105 {
1106 Log("SHUTTLE", Form("RemoveFile - %s: Cannot remove file %s!",
1107 fCurrentDetector.Data(), filename));
1108 }
73abe331 1109}
1110
b948db8d 1111//______________________________________________________________________________________________
5164a766 1112AliShuttleStatus* AliShuttle::ReadShuttleStatus()
1113{
9827400b 1114 //
1115 // Reads the AliShuttleStatus from the CDB
1116 //
5164a766 1117
2bb7b766 1118 if (fStatusEntry){
1119 delete fStatusEntry;
1120 fStatusEntry = 0;
1121 }
5164a766 1122
10a5a932 1123 fStatusEntry = AliCDBManager::Instance()->GetStorage(GetLocalCDB())
2bb7b766 1124 ->Get(Form("/SHUTTLE/STATUS/%s", fCurrentDetector.Data()), GetCurrentRun());
5164a766 1125
2bb7b766 1126 if (!fStatusEntry) return 0;
1127 fStatusEntry->SetOwner(1);
5164a766 1128
2bb7b766 1129 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1130 if (!status) {
1131 AliError("Invalid object stored to CDB!");
1132 return 0;
1133 }
5164a766 1134
2bb7b766 1135 return status;
5164a766 1136}
1137
1138//______________________________________________________________________________________________
7bfb2090 1139Bool_t AliShuttle::WriteShuttleStatus(AliShuttleStatus* status)
5164a766 1140{
9827400b 1141 //
1142 // writes the status for one subdetector
1143 //
2bb7b766 1144
1145 if (fStatusEntry){
1146 delete fStatusEntry;
1147 fStatusEntry = 0;
1148 }
5164a766 1149
2bb7b766 1150 Int_t run = GetCurrentRun();
5164a766 1151
2bb7b766 1152 AliCDBId id(AliCDBPath("SHUTTLE", "STATUS", fCurrentDetector), run, run);
5164a766 1153
2bb7b766 1154 fStatusEntry = new AliCDBEntry(status, id, new AliCDBMetaData);
1155 fStatusEntry->SetOwner(1);
5164a766 1156
2bb7b766 1157 UInt_t result = AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
7bfb2090 1158
2bb7b766 1159 if (!result) {
3301427a 1160 Log("SHUTTLE", Form("WriteShuttleStatus - Failed for %s, run %d",
1161 fCurrentDetector.Data(), run));
2bb7b766 1162 return kFALSE;
1163 }
e7f62f16 1164
1165 SendMLInfo();
7bfb2090 1166
2bb7b766 1167 return kTRUE;
5164a766 1168}
1169
1170//______________________________________________________________________________________________
1171void AliShuttle::UpdateShuttleStatus(AliShuttleStatus::Status newStatus, Bool_t increaseCount)
1172{
9827400b 1173 //
1174 // changes the AliShuttleStatus for the given detector and run to the given status
1175 //
5164a766 1176
2bb7b766 1177 if (!fStatusEntry){
1178 AliError("UNEXPECTED: fStatusEntry empty");
1179 return;
1180 }
5164a766 1181
2bb7b766 1182 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
5164a766 1183
2bb7b766 1184 if (!status){
c88ad5db 1185 Log("SHUTTLE", "UpdateShuttleStatus - UNEXPECTED: status could not be read from current CDB entry");
2bb7b766 1186 return;
1187 }
5164a766 1188
2c15234c 1189 TString actionStr = Form("UpdateShuttleStatus - %s: Changing state from %s to %s",
eba76848 1190 fCurrentDetector.Data(),
36c99a6a 1191 status->GetStatusName(),
eba76848 1192 status->GetStatusName(newStatus));
cb343cfd 1193 Log("SHUTTLE", actionStr);
1194 SetLastAction(actionStr);
5164a766 1195
2bb7b766 1196 status->SetStatus(newStatus);
1197 if (increaseCount) status->IncreaseCount();
5164a766 1198
2bb7b766 1199 AliCDBManager::Instance()->GetStorage(fgkLocalCDB)->Put(fStatusEntry);
e7f62f16 1200
1201 SendMLInfo();
5164a766 1202}
e7f62f16 1203
1204//______________________________________________________________________________________________
1205void AliShuttle::SendMLInfo()
1206{
1207 //
1208 // sends ML information about the current status of the current detector being processed
1209 //
1210
1211 AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
1212
1213 if (!status){
3301427a 1214 Log("SHUTTLE", "SendMLInfo - UNEXPECTED: status could not be read from current CDB entry");
e7f62f16 1215 return;
1216 }
1217
1218 TMonaLisaText mlStatus(Form("%s_status", fCurrentDetector.Data()), status->GetStatusName());
1219 TMonaLisaValue mlRetryCount(Form("%s_count", fCurrentDetector.Data()), status->GetCount());
1220
1221 TList mlList;
1222 mlList.Add(&mlStatus);
1223 mlList.Add(&mlRetryCount);
1224
1225 fMonaLisa->SendParameters(&mlList);
1226}
1227
5164a766 1228//______________________________________________________________________________________________
1229Bool_t AliShuttle::ContinueProcessing()
1230{
9827400b 1231 // this function reads the AliShuttleStatus information from CDB and
1232 // checks if the processing should be continued
1233 // if yes it returns kTRUE and updates the AliShuttleStatus with nextStatus
2bb7b766 1234
57c1a579 1235 if (!fConfig->HostProcessDetector(fCurrentDetector)) return kFALSE;
1236
1237 AliPreprocessor* aPreprocessor =
1238 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1239 if (!aPreprocessor)
1240 {
c88ad5db 1241 Log("SHUTTLE", Form("ContinueProcessing - %s: no preprocessor registered", fCurrentDetector.Data()));
57c1a579 1242 return kFALSE;
1243 }
1244
2bb7b766 1245 AliShuttleLogbookEntry::Status entryStatus =
eba76848 1246 fLogbookEntry->GetDetectorStatus(fCurrentDetector);
2bb7b766 1247
1248 if(entryStatus != AliShuttleLogbookEntry::kUnprocessed) {
c88ad5db 1249 Log("SHUTTLE", Form("ContinueProcessing - %s is %s",
2bb7b766 1250 fCurrentDetector.Data(),
1251 fLogbookEntry->GetDetectorStatusName(entryStatus)));
1252 return kFALSE;
1253 }
1254
1255 // if we get here, according to Shuttle logbook subdetector is in UNPROCESSED state
be48e3ea 1256
1257 // check if current run is first unprocessed run for current detector
1258 if (fConfig->StrictRunOrder(fCurrentDetector) &&
1259 !fFirstUnprocessed[GetDetPos(fCurrentDetector)])
1260 {
86aa42c3 1261 if (fTestMode == kNone)
1262 {
c88ad5db 1263 Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering"
1264 " but this is not the first unprocessed run!"));
86aa42c3 1265 return kFALSE;
1266 }
1267 else
1268 {
c88ad5db 1269 Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - "
1270 "Although %s requires strict run ordering "
1271 "and this is not the first unprocessed run, "
1272 "the SHUTTLE continues"));
86aa42c3 1273 }
be48e3ea 1274 }
1275
2bb7b766 1276 AliShuttleStatus* status = ReadShuttleStatus();
1277 if (!status) {
1278 // first time
1279 Log("SHUTTLE", Form("ContinueProcessing - %s: Processing first time",
1280 fCurrentDetector.Data()));
1281 status = new AliShuttleStatus(AliShuttleStatus::kStarted);
1282 return WriteShuttleStatus(status);
1283 }
1284
1285 // The following two cases shouldn't happen if Shuttle Logbook was correctly updated.
1286 // If it happens it may mean Logbook updating failed... let's do it now!
1287 if (status->GetStatus() == AliShuttleStatus::kDone ||
1288 status->GetStatus() == AliShuttleStatus::kFailed){
1289 Log("SHUTTLE", Form("ContinueProcessing - %s is already %s. Updating Shuttle Logbook",
1290 fCurrentDetector.Data(),
1291 status->GetStatusName(status->GetStatus())));
1292 UpdateShuttleLogbook(fCurrentDetector.Data(),
1293 status->GetStatusName(status->GetStatus()));
1294 return kFALSE;
1295 }
1296
3301427a 1297 if (status->GetStatus() == AliShuttleStatus::kStoreError) {
2bb7b766 1298 Log("SHUTTLE",
c88ad5db 1299 Form("ContinueProcessing - %s: Grid storage of one or more "
1300 "objects failed. Trying again now",
2bb7b766 1301 fCurrentDetector.Data()));
9827400b 1302 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1303 if (StoreOCDB()){
c88ad5db 1304 Log("SHUTTLE", Form("ContinueProcessing - %s: all objects "
1305 "successfully stored into main storage",
3301427a 1306 fCurrentDetector.Data()));
2bb7b766 1307 UpdateShuttleStatus(AliShuttleStatus::kDone);
1308 UpdateShuttleLogbook(fCurrentDetector.Data(), "DONE");
1309 } else {
1310 Log("SHUTTLE",
1311 Form("ContinueProcessing - %s: Grid storage failed again",
1312 fCurrentDetector.Data()));
9827400b 1313 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
2bb7b766 1314 }
1315 return kFALSE;
1316 }
1317
1318 // if we get here, there is a restart
57c1a579 1319 Bool_t cont = kFALSE;
2bb7b766 1320
1321 // abort conditions
cb343cfd 1322 if (status->GetCount() >= fConfig->GetMaxRetries()) {
57c1a579 1323 Log("SHUTTLE", Form("ContinueProcessing - %s failed %d times in status %s - "
1324 "Updating Shuttle Logbook", fCurrentDetector.Data(),
2bb7b766 1325 status->GetCount(), status->GetStatusName()));
1326 UpdateShuttleLogbook(fCurrentDetector.Data(), "FAILED");
e7f62f16 1327 UpdateShuttleStatus(AliShuttleStatus::kFailed);
3301427a 1328
1329 // there may still be objects in local OCDB and reference storage
1330 // and FXS databases may be not updated: do it now!
9827400b 1331
1332 // TODO Currently disabled, we want to keep files in case of failure!
1333 // CleanLocalStorage(fgkLocalCDB);
1334 // CleanLocalStorage(fgkLocalRefStorage);
1335 // UpdateTableFailCase();
1336
1337 // Send mail to detector expert!
c88ad5db 1338 Log("SHUTTLE", Form("ContinueProcessing - Sending mail to %s expert...",
1339 fCurrentDetector.Data()));
9827400b 1340 if (!SendMail())
1341 Log("SHUTTLE", Form("ContinueProcessing - Could not send mail to %s expert",
1342 fCurrentDetector.Data()));
3301427a 1343
57c1a579 1344 } else {
1345 Log("SHUTTLE", Form("ContinueProcessing - %s: restarting. "
1346 "Aborted before with %s. Retry number %d.", fCurrentDetector.Data(),
1347 status->GetStatusName(), status->GetCount()));
9827400b 1348 Bool_t increaseCount = kTRUE;
c88ad5db 1349 if (status->GetStatus() == AliShuttleStatus::kDCSError ||
1350 status->GetStatus() == AliShuttleStatus::kDCSStarted)
1351 increaseCount = kFALSE;
9827400b 1352 UpdateShuttleStatus(AliShuttleStatus::kStarted, increaseCount);
57c1a579 1353 cont = kTRUE;
2bb7b766 1354 }
1355
57c1a579 1356 return cont;
5164a766 1357}
1358
1359//______________________________________________________________________________________________
2bb7b766 1360Bool_t AliShuttle::Process(AliShuttleLogbookEntry* entry)
58bc3020 1361{
73abe331 1362 //
b948db8d 1363 // Makes data retrieval for all detectors in the configuration.
2bb7b766 1364 // entry: Shuttle logbook entry, contains run paramenters and status of detectors
1365 // (Unprocessed, Inactive, Failed or Done).
d477ad88 1366 // Returns kFALSE in case of error occured and kTRUE otherwise
73abe331 1367 //
1368
9827400b 1369 if (!entry) return kFALSE;
2bb7b766 1370
1371 fLogbookEntry = entry;
1372
c88ad5db 1373 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^*",
9827400b 1374 GetCurrentRun()));
2bb7b766 1375
e7f62f16 1376 // create ML instance that monitors this run
1377 fMonaLisa = new TMonaLisaWriter(Form("%d", GetCurrentRun()), "SHUTTLE", "aliendb1.cern.ch");
1378 // disable monitoring of other parameters that come e.g. from TFile
1379 gMonitoringWriter = 0;
2bb7b766 1380
e7f62f16 1381 // Send the information to ML
1382 TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
9827400b 1383 TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
e7f62f16 1384
1385 TList mlList;
1386 mlList.Add(&mlStatus);
9827400b 1387 mlList.Add(&mlRunType);
e7f62f16 1388
1389 fMonaLisa->SendParameters(&mlList);
3301427a 1390
9827400b 1391 if (fLogbookEntry->IsDone())
1392 {
1393 Log("SHUTTLE","Process - Shuttle is already DONE. Updating logbook");
1394 UpdateShuttleLogbook("shuttle_done");
1395 fLogbookEntry = 0;
1396 return kTRUE;
1397 }
1398
1399 // read test mode if flag is set
1400 if (fReadTestMode)
1401 {
3d8bc902 1402 fTestMode = kNone;
9827400b 1403 TString logEntry(entry->GetRunParameter("log"));
1404 //printf("log entry = %s\n", logEntry.Data());
1405 TString searchStr("Testmode: ");
1406 Int_t pos = logEntry.Index(searchStr.Data());
1407 //printf("%d\n", pos);
1408 if (pos >= 0)
1409 {
1410 TSubString subStr = logEntry(pos + searchStr.Length(), logEntry.Length());
1411 //printf("%s\n", subStr.String().Data());
1412 TString newStr(subStr.Data());
1413 TObjArray* token = newStr.Tokenize(' ');
1414 if (token)
1415 {
1416 //token->Print();
1417 TObjString* tmpStr = dynamic_cast<TObjString*> (token->First());
1418 if (tmpStr)
1419 {
1420 Int_t testMode = tmpStr->String().Atoi();
1421 if (testMode > 0)
1422 {
c88ad5db 1423 Log("SHUTTLE", Form("Process - Enabling test mode %d", testMode));
9827400b 1424 SetTestMode((TestMode) testMode);
1425 }
1426 }
1427 delete token;
1428 }
1429 }
1430 }
c88ad5db 1431
eba76848 1432 fLogbookEntry->Print("all");
57f50b3c 1433
1434 // Initialization
d477ad88 1435 Bool_t hasError = kFALSE;
5164a766 1436
2bb7b766 1437 AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
1438 if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
1439 AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
1440 if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
d477ad88 1441
57f50b3c 1442 // Loop on detectors in the configuration
b948db8d 1443 TIter iter(fConfig->GetDetectors());
2bb7b766 1444 TObjString* aDetector = 0;
b948db8d 1445
be48e3ea 1446 while ((aDetector = (TObjString*) iter.Next()))
1447 {
7bfb2090 1448 fCurrentDetector = aDetector->String();
5164a766 1449
9e080f92 1450 if (ContinueProcessing() == kFALSE) continue;
1451
c88ad5db 1452 Log("SHUTTLE", Form("\t\t\t****** run %d - %s: START ******",
2bb7b766 1453 GetCurrentRun(), aDetector->GetName()));
1454
9d733021 1455 for(Int_t iSys=0;iSys<3;iSys++) fFXSCalled[iSys]=kFALSE;
1456
c88ad5db 1457 Log(fCurrentDetector.Data(), "Process - Starting processing");
85a80aa9 1458
be48e3ea 1459 Int_t pid = fork();
1460
1461 if (pid < 0)
1462 {
c88ad5db 1463 Log("SHUTTLE", "Process - ERROR: Forking failed");
be48e3ea 1464 }
1465 else if (pid > 0)
1466 {
1467 // parent
c88ad5db 1468 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Starting monitoring",
be48e3ea 1469 GetCurrentRun(), aDetector->GetName()));
1470
1471 Long_t begin = time(0);
1472
1473 int status; // to be used with waitpid, on purpose an int (not Int_t)!
1474 while (waitpid(pid, &status, WNOHANG) == 0)
1475 {
1476 Long_t expiredTime = time(0) - begin;
1477
1478 if (expiredTime > fConfig->GetPPTimeOut())
1479 {
9827400b 1480 TString tmp;
c88ad5db 1481 tmp.Form("Process - Process of %s time out. "
1482 "Run time: %d seconds. Killing...",
1483 fCurrentDetector.Data(), expiredTime);
9827400b 1484 Log("SHUTTLE", tmp);
1485 Log(fCurrentDetector, tmp);
be48e3ea 1486
1487 kill(pid, 9);
1488
3301427a 1489 UpdateShuttleStatus(AliShuttleStatus::kPPTimeOut);
be48e3ea 1490 hasError = kTRUE;
1491
1492 gSystem->Sleep(1000);
1493 }
1494 else
1495 {
be48e3ea 1496 gSystem->Sleep(1000);
9827400b 1497
1498 TString checkStr;
1499 checkStr.Form("ps -o vsize --pid %d | tail -n 1", pid);
1500 FILE* pipe = gSystem->OpenPipe(checkStr, "r");
1501 if (!pipe)
1502 {
c88ad5db 1503 Log("SHUTTLE", Form("Process - Error: "
1504 "Could not open pipe to %s", checkStr.Data()));
9827400b 1505 continue;
1506 }
1507
1508 char buffer[100];
1509 if (!fgets(buffer, 100, pipe))
1510 {
c88ad5db 1511 Log("SHUTTLE", "Process - Error: ps did not return anything");
9827400b 1512 gSystem->ClosePipe(pipe);
1513 continue;
1514 }
1515 gSystem->ClosePipe(pipe);
1516
1517 //Log("SHUTTLE", Form("ps returned %s", buffer));
1518
1519 Int_t mem = 0;
1520 if ((sscanf(buffer, "%d\n", &mem) != 1) || !mem)
1521 {
c88ad5db 1522 Log("SHUTTLE", "Process - Error: Could not parse output of ps");
9827400b 1523 continue;
1524 }
1525
1526 if (expiredTime % 60 == 0)
c88ad5db 1527 Log("SHUTTLE", Form("Process - %s: Checking process. "
1528 "Run time: %d seconds - Memory consumption: %d KB",
1529 fCurrentDetector.Data(), expiredTime, mem));
9827400b 1530
1531 if (mem > fConfig->GetPPMaxMem())
1532 {
1533 TString tmp;
c88ad5db 1534 tmp.Form("Process - Process exceeds maximum allowed memory "
1535 "(%d KB > %d KB). Killing...",
9827400b 1536 mem, fConfig->GetPPMaxMem());
1537 Log("SHUTTLE", tmp);
1538 Log(fCurrentDetector, tmp);
1539
1540 kill(pid, 9);
1541
1542 UpdateShuttleStatus(AliShuttleStatus::kPPOutOfMemory);
1543 hasError = kTRUE;
1544
1545 gSystem->Sleep(1000);
1546 }
be48e3ea 1547 }
1548 }
1549
c88ad5db 1550 Log("SHUTTLE", Form("Process - In parent process of %d - %s: Client has terminated.",
be48e3ea 1551 GetCurrentRun(), aDetector->GetName()));
1552
1553 if (WIFEXITED(status))
1554 {
1555 Int_t returnCode = WEXITSTATUS(status);
1556
c88ad5db 1557 Log("SHUTTLE", Form("Process - %s: the return code is %d", fCurrentDetector.Data(),
3301427a 1558 returnCode));
be48e3ea 1559
9827400b 1560 if (returnCode == 0) hasError = kTRUE;
be48e3ea 1561 }
1562 }
1563 else if (pid == 0)
1564 {
1565 // client
c88ad5db 1566 Log("SHUTTLE", Form("Process - In client process of %d - %s", GetCurrentRun(),
1567 aDetector->GetName()));
be48e3ea 1568
c88ad5db 1569 Log("SHUTTLE", Form("Process - Redirecting output to %s log",fCurrentDetector.Data()));
ffa29e93 1570
546242fb 1571 if ((freopen(GetLogFileName(fCurrentDetector), "a", stdout)) == 0)
ffa29e93 1572 {
c88ad5db 1573 Log("SHUTTLE", "Process - Could not freopen stdout");
ffa29e93 1574 }
1575 else
1576 {
1577 fOutputRedirected = kTRUE;
1578 if ((dup2(fileno(stdout), fileno(stderr))) < 0)
c88ad5db 1579 Log("SHUTTLE", "Process - Could not redirect stderr");
ffa29e93 1580
1581 }
1582
5bac2bde 1583 TString wd = gSystem->WorkingDirectory();
d524ade6 1584 TString tmpDir = Form("%s/%s_process", GetShuttleTempDir(), fCurrentDetector.Data());
5bac2bde 1585
d524ade6 1586 Int_t result = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
1587 if (!result) // temp dir already exists!
1588 {
1589 Log(fCurrentDetector.Data(),
1590 Form("Process - %s dir already exists! Removing...", tmpDir.Data()));
1591 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
1592 } else {
1593 if (gSystem->mkdir(tmpDir.Data(), 1))
1594 {
1595 Log(fCurrentDetector.Data(), "Process - could not make temp directory!!");
1596 gSystem->Exit(1);
1597 }
1598 }
1599
1600 if (!gSystem->ChangeDirectory(tmpDir.Data()))
1601 {
1602 Log(fCurrentDetector.Data(), "Process - could not change directory!!");
1603 gSystem->Exit(1);
1604 }
5bac2bde 1605
9827400b 1606 Bool_t success = ProcessCurrentDetector();
5bac2bde 1607
1608 gSystem->ChangeDirectory(wd.Data());
1609
1610 gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
1611
9827400b 1612 if (success) // Preprocessor finished successfully!
1613 {
3301427a 1614 // Update time_processed field in FXS DB
1615 if (UpdateTable() == kFALSE)
5bac2bde 1616 Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!",
1617 fCurrentDetector.Data()));
3301427a 1618
1619 // Transfer the data from local storage to main storage (Grid)
1620 UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
1621 if (StoreOCDB() == kFALSE)
1622 {
c88ad5db 1623 Log("SHUTTLE",
1624 Form("\t\t\t****** run %d - %s: STORAGE ERROR ******",
3301427a 1625 GetCurrentRun(), aDetector->GetName()));
1626 UpdateShuttleStatus(AliShuttleStatus::kStoreError);
9827400b 1627 success = kFALSE;
3301427a 1628 } else {
c88ad5db 1629 Log("SHUTTLE",
1630 Form("\t\t\t****** run %d - %s: DONE ******",
3301427a 1631 GetCurrentRun(), aDetector->GetName()));
1632 UpdateShuttleStatus(AliShuttleStatus::kDone);
9827400b 1633 UpdateShuttleLogbook(fCurrentDetector, "DONE");
3301427a 1634 }
c88ad5db 1635 } else
1636 {
1637 Log("SHUTTLE",
1638 Form("\t\t\t****** run %d - %s: PP ERROR ******",
1639 GetCurrentRun(), aDetector->GetName()));
be48e3ea 1640 }
1641
4b95672b 1642 for (UInt_t iSys=0; iSys<3; iSys++)
1643 {
1644 if (fFXSCalled[iSys]) fFXSlist[iSys].Clear();
1645 }
1646
c88ad5db 1647 Log("SHUTTLE", Form("Process - Client process of %d - %s is exiting now with %d.",
9827400b 1648 GetCurrentRun(), aDetector->GetName(), success));
be48e3ea 1649
1650 // the client exits here
9827400b 1651 gSystem->Exit(success);
be48e3ea 1652
1653 AliError("We should never get here!!!");
1654 }
7bfb2090 1655 }
5164a766 1656
c88ad5db 1657 Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^*",
2bb7b766 1658 GetCurrentRun()));
1659
1660 //check if shuttle is done for this run, if so update logbook
1661 TObjArray checkEntryArray;
1662 checkEntryArray.SetOwner(1);
9e080f92 1663 TString whereClause = Form("where run=%d", GetCurrentRun());
1664 if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) || checkEntryArray.GetEntries() == 0) {
1665 Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
1666 GetCurrentRun()));
1667 return hasError == kFALSE;
1668 }
b948db8d 1669
9e080f92 1670 AliShuttleLogbookEntry* checkEntry = dynamic_cast<AliShuttleLogbookEntry*>
1671 (checkEntryArray.At(0));
2bb7b766 1672
9e080f92 1673 if (checkEntry)
1674 {
1675 if (checkEntry->IsDone())
be48e3ea 1676 {
9e080f92 1677 Log("SHUTTLE","Process - Shuttle is DONE. Updating logbook");
1678 UpdateShuttleLogbook("shuttle_done");
1679 }
1680 else
1681 {
1682 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
be48e3ea 1683 {
9e080f92 1684 if (checkEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
be48e3ea 1685 {
9e080f92 1686 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
1687 checkEntry->GetRun(), GetDetName(iDet)));
1688 fFirstUnprocessed[iDet] = kFALSE;
be48e3ea 1689 }
1690 }
2bb7b766 1691 }
1692 }
1693
e7f62f16 1694 // remove ML instance
1695 delete fMonaLisa;
1696 fMonaLisa = 0;
1697
2bb7b766 1698 fLogbookEntry = 0;
85a80aa9 1699
a7160fe9 1700 return hasError == kFALSE;
73abe331 1701}
1702
b948db8d 1703//______________________________________________________________________________________________
9827400b 1704Bool_t AliShuttle::ProcessCurrentDetector()
73abe331 1705{
1706 //
2bb7b766 1707 // Makes data retrieval just for a specific detector (fCurrentDetector).
73abe331 1708 // Threre should be a configuration for this detector.
73abe331 1709
1d172743 1710 Log("SHUTTLE", Form("ProcessCurrentDetector - Retrieving values for %s, run %d",
1711 fCurrentDetector.Data(), GetCurrentRun()));
73abe331 1712
d524ade6 1713 TString wd = gSystem->WorkingDirectory();
1714
2d9019b4 1715 if (!CleanReferenceStorage(fCurrentDetector.Data()))
546242fb 1716 return kFALSE;
d524ade6 1717
1718 gSystem->ChangeDirectory(wd.Data());
1719
1d172743 1720 TMap* dcsMap = new TMap();
3301427a 1721
1722 // call preprocessor
1723 AliPreprocessor* aPreprocessor =
1724 dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
1725
1726 aPreprocessor->Initialize(GetCurrentRun(), GetCurrentStartTime(), GetCurrentEndTime());
1727
1728 Bool_t processDCS = aPreprocessor->ProcessDCS();
d477ad88 1729
651fdaab 1730 if (!processDCS)
1731 {
1d172743 1732 Log(fCurrentDetector, "ProcessCurrentDetector -"
1733 " The preprocessor requested to skip the retrieval of DCS values");
651fdaab 1734 }
8b739301 1735 else if (fTestMode & kSkipDCS)
2c15234c 1736 {
1d172743 1737 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Skipping DCS processing");
9827400b 1738 }
1739 else if (fTestMode & kErrorDCS)
1740 {
1d172743 1741 Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Simulating DCS error");
3d8bc902 1742 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
9827400b 1743 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1d172743 1744 delete dcsMap;
9827400b 1745 return kFALSE;
2c15234c 1746 } else {
3301427a 1747
1748 UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
1749
1d172743 1750 // Query DCS archive
1751 Int_t nServers = fConfig->GetNServers(fCurrentDetector);
a038aa70 1752
1d172743 1753 for (int iServ=0; iServ<nServers; iServ++)
2c15234c 1754 {
1d172743 1755
1756 TString host(fConfig->GetDCSHost(fCurrentDetector, iServ));
1757 Int_t port = fConfig->GetDCSPort(fCurrentDetector, iServ);
542b6cc8 1758 Int_t multiSplit = fConfig->GetMultiSplit(fCurrentDetector, iServ);
1759
1790d4b7 1760 Log(fCurrentDetector, Form("ProcessCurrentDetector -"
1761 " Querying DCS Amanda server %s:%d (%d of %d)",
1762 host.Data(), port, iServ+1, nServers));
1d172743 1763
1764 TMap* aliasMap = 0;
1765 TMap* dpMap = 0;
1766
1767 if (fConfig->GetDCSAliases(fCurrentDetector, iServ)->GetEntries() > 0)
2c15234c 1768 {
1d172743 1769 aliasMap = GetValueSet(host, port,
542b6cc8 1770 fConfig->GetDCSAliases(fCurrentDetector, iServ),
1771 kAlias, multiSplit);
1d172743 1772 if (!aliasMap)
1773 {
1774 Log(fCurrentDetector,
1775 Form("ProcessCurrentDetector -"
1776 " Error retrieving DCS aliases from server %s",
1777 host.Data()));
1778 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
a038aa70 1779 delete dcsMap;
1d172743 1780 return kFALSE;
1781 }
2c15234c 1782 }
a038aa70 1783
1d172743 1784 if (fConfig->GetDCSDataPoints(fCurrentDetector, iServ)->GetEntries() > 0)
a038aa70 1785 {
1d172743 1786 dpMap = GetValueSet(host, port,
542b6cc8 1787 fConfig->GetDCSDataPoints(fCurrentDetector, iServ),
1788 kDP, multiSplit);
1d172743 1789 if (!dpMap)
1790 {
1791 Log(fCurrentDetector,
1792 Form("ProcessCurrentDetector -"
1793 " Error retrieving DCS data points from server %s",
1794 host.Data()));
1795 UpdateShuttleStatus(AliShuttleStatus::kDCSError);
1796 if (aliasMap) delete aliasMap;
1797 delete dcsMap;
1798 return kFALSE;
1799 }
a038aa70 1800 }
1d172743 1801
1802 // merge aliasMap and dpMap into dcsMap
1803 if(aliasMap) {
1804 TIter iter(aliasMap);
a038aa70 1805 TObjString* key = 0;
1806 while ((key = (TObjString*) iter.Next()))
1d172743 1807 dcsMap->Add(key, aliasMap->GetValue(key->String()));
1808
1809 aliasMap->SetOwner(kFALSE);
1810 delete aliasMap;
1811 }
1812
1813 if(dpMap) {
1814 TIter iter(dpMap);
1815 TObjString* key = 0;
1816 while ((key = (TObjString*) iter.Next()))
1817 dcsMap->Add(key, dpMap->GetValue(key->String()));
1818
1819 dpMap->SetOwner(kFALSE);
1820 delete dpMap;
a038aa70 1821 }
73abe331 1822 }
1823 }
dc25836b 1824
2bb7b766 1825 // DCS Archive DB processing successful. Call Preprocessor!
85a80aa9 1826 UpdateShuttleStatus(AliShuttleStatus::kPPStarted);
a7160fe9 1827
a038aa70 1828 UInt_t returnValue = aPreprocessor->Process(dcsMap);
b948db8d 1829
3301427a 1830 if (returnValue > 0) // Preprocessor error!
1831 {
c88ad5db 1832 Log(fCurrentDetector, Form("ProcessCurrentDetector - "
1833 "Preprocessor failed. Process returned %d.", returnValue));
cb343cfd 1834 UpdateShuttleStatus(AliShuttleStatus::kPPError);
a038aa70 1835 dcsMap->DeleteAll();
1836 delete dcsMap;
9827400b 1837 return kFALSE;
1838 }
1839
1840 // preprocessor ok!
1841 UpdateShuttleStatus(AliShuttleStatus::kPPDone);
1842 Log(fCurrentDetector, Form("ProcessCurrentDetector - %s preprocessor returned success",
1843 fCurrentDetector.Data()));
b948db8d 1844
a038aa70 1845 dcsMap->DeleteAll();
1846 delete dcsMap;
b948db8d 1847
9827400b 1848 return kTRUE;
2bb7b766 1849}
1850
1851//______________________________________________________________________________________________
1852Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
1853 TObjArray& entries)
1854{
9827400b 1855 // Query DAQ's Shuttle logbook and fills detector status object.
1856 // Call QueryRunParameters to query DAQ logbook for run parameters.
1857 //
2bb7b766 1858
fc5a4708 1859 entries.SetOwner(1);
1860
2bb7b766 1861 // check connection, in case connect
be48e3ea 1862 if(!Connect(3)) return kFALSE;
2bb7b766 1863
1864 TString sqlQuery;
441b0e9c 1865 sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
2bb7b766 1866
be48e3ea 1867 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1868 if (!aResult) {
1869 AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
1870 return kFALSE;
1871 }
1872
fc5a4708 1873 AliDebug(2,Form("Query = %s", sqlQuery.Data()));
1874
2bb7b766 1875 if(aResult->GetRowCount() == 0) {
c88ad5db 1876 Log("SHUTTLE", "No entries in Shuttle Logbook match request");
9827400b 1877 delete aResult;
1878 return kTRUE;
2bb7b766 1879 }
1880
1881 // TODO Check field count!
db99d43e 1882 const UInt_t nCols = 23;
2bb7b766 1883 if (aResult->GetFieldCount() != (Int_t) nCols) {
c88ad5db 1884 Log("SHUTTLE", "Invalid SQL result field number!");
2bb7b766 1885 delete aResult;
1886 return kFALSE;
1887 }
1888
2bb7b766 1889 TSQLRow* aRow;
1890 while ((aRow = aResult->Next())) {
1891 TString runString(aRow->GetField(0), aRow->GetFieldLength(0));
1892 Int_t run = runString.Atoi();
1893
eba76848 1894 AliShuttleLogbookEntry *entry = QueryRunParameters(run);
1895 if (!entry)
1896 continue;
2bb7b766 1897
1898 // loop on detectors
eba76848 1899 for(UInt_t ii = 0; ii < nCols; ii++)
1900 entry->SetDetectorStatus(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1901
eba76848 1902 entries.AddLast(entry);
2bb7b766 1903 delete aRow;
1904 }
1905
2bb7b766 1906 delete aResult;
1907 return kTRUE;
1908}
1909
1910//______________________________________________________________________________________________
eba76848 1911AliShuttleLogbookEntry* AliShuttle::QueryRunParameters(Int_t run)
2bb7b766 1912{
eba76848 1913 //
1914 // Retrieve run parameters written in the DAQ logbook and sets them into AliShuttleLogbookEntry object
1915 //
2bb7b766 1916
1917 // check connection, in case connect
be48e3ea 1918 if (!Connect(3))
eba76848 1919 return 0;
2bb7b766 1920
1921 TString sqlQuery;
2c15234c 1922 sqlQuery.Form("select * from %s where run=%d", fConfig->GetDAQlbTable(), run);
2bb7b766 1923
be48e3ea 1924 TSQLResult* aResult = fServer[3]->Query(sqlQuery);
2bb7b766 1925 if (!aResult) {
c88ad5db 1926 Log("SHUTTLE", Form("Can't execute query <%s>!", sqlQuery.Data()));
eba76848 1927 return 0;
2bb7b766 1928 }
1929
eba76848 1930 if (aResult->GetRowCount() == 0) {
2bb7b766 1931 Log("SHUTTLE", Form("QueryRunParameters - No entry in DAQ Logbook for run %d. Skipping", run));
1932 delete aResult;
eba76848 1933 return 0;
2bb7b766 1934 }
1935
eba76848 1936 if (aResult->GetRowCount() > 1) {
c88ad5db 1937 Log("SHUTTLE", Form("QueryRunParameters - UNEXPECTED: "
1938 "more than one entry in DAQ Logbook for run %d!", run));
2bb7b766 1939 delete aResult;
eba76848 1940 return 0;
2bb7b766 1941 }
1942
eba76848 1943 TSQLRow* aRow = aResult->Next();
1944 if (!aRow)
1945 {
c88ad5db 1946 Log("SHUTTLE", Form("QueryRunParameters - Could not retrieve row for run %d. Skipping", run));
eba76848 1947 delete aResult;
1948 return 0;
1949 }
2bb7b766 1950
eba76848 1951 AliShuttleLogbookEntry* entry = new AliShuttleLogbookEntry(run);
2bb7b766 1952
eba76848 1953 for (Int_t ii = 0; ii < aResult->GetFieldCount(); ii++)
1954 entry->SetRunParameter(aResult->GetFieldName(ii), aRow->GetField(ii));
2bb7b766 1955
eba76848 1956 UInt_t startTime = entry->GetStartTime();
1957 UInt_t endTime = entry->GetEndTime();
1958
1959 if (!startTime || !endTime || startTime > endTime) {
1960 Log("SHUTTLE",
1961 Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d",
1962 run, startTime, endTime));
1963 delete entry;
2bb7b766 1964 delete aRow;
eba76848 1965 delete aResult;
1966 return 0;
2bb7b766 1967 }
1968
eba76848 1969 delete aRow;
2bb7b766 1970 delete aResult;
eba76848 1971
1972 return entry;
2bb7b766 1973}
1974
a038aa70 1975//______________________________________________________________________________________________
1976TMap* AliShuttle::GetValueSet(const char* host, Int_t port, const TSeqCollection* entries,
542b6cc8 1977 DCSType type, Int_t multiSplit)
a038aa70 1978{
1979 // Retrieve all "entry" data points from the DCS server
1980 // host, port: TSocket connection parameters
1981 // entries: list of name of the alias or data point
1982 // type: kAlias or kDP
1983 // returns TMap of values, 0 when failure
542b6cc8 1984
1985 AliDCSClient client(host, port, fTimeout, fRetries, multiSplit);
b41b252a 1986
a038aa70 1987 TMap* result = 0;
b41b252a 1988 if (type == kAlias)
a038aa70 1989 {
b41b252a 1990 result = client.GetAliasValues(entries, GetCurrentStartTime(),
1991 GetCurrentEndTime());
1992 }
1993 else if (type == kDP)
1994 {
1995 result = client.GetDPValues(entries, GetCurrentStartTime(),
1996 GetCurrentEndTime());
1997 }
a038aa70 1998
b41b252a 1999 if (result == 0)
2000 {
2001 Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get entries! Reason: %s",
1790d4b7 2002 client.GetErrorString(client.GetResultErrorCode())));
2003 if (client.GetResultErrorCode() == AliDCSClient::fgkServerError)
2004 Log(fCurrentDetector.Data(), Form("GetValueSet - Server error code: %s",
2005 client.GetServerError().Data()));
a038aa70 2006
b41b252a 2007 return 0;
a038aa70 2008 }
b41b252a 2009
a038aa70 2010 return result;
2011}
b41b252a 2012
b948db8d 2013//______________________________________________________________________________________________
57f50b3c 2014const char* AliShuttle::GetFile(Int_t system, const char* detector,
2015 const char* id, const char* source)
b948db8d 2016{
9827400b 2017 // Get calibration file from file exchange servers
2018 // First queris the FXS database for the file name, using the run, detector, id and source info
2019 // then calls RetrieveFile(filename) for actual copy to local disk
2020 // run: current run being processed (given by Logbook entry fLogbookEntry)
2021 // detector: the Preprocessor name
2022 // id: provided as a parameter by the Preprocessor
2023 // source: provided by the Preprocessor through GetFileSources function
2024
2025 // check if test mode should simulate a FXS error
2026 if (fTestMode & kErrorFXSFiles)
2027 {
2028 Log(detector, Form("GetFile - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2029 return 0;
2030 }
2031
57f50b3c 2032 // check connection, in case connect
9d733021 2033 if (!Connect(system))
eba76848 2034 {
9d733021 2035 Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
57f50b3c 2036 return 0;
2037 }
2038
2039 // Query preparation
9d733021 2040 TString sourceName(source);
d386d623 2041 Int_t nFields = 3;
2042 TString sqlQueryStart = Form("select filePath,size,fileChecksum from %s where",
2043 fConfig->GetFXSdbTable(system));
2044 TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
2045 GetCurrentRun(), detector, id);
2046
9d733021 2047 if (system == kDAQ)
2048 {
d386d623 2049 whereClause += Form(" and DAQsource=\"%s\"", source);
57f50b3c 2050 }
9d733021 2051 else if (system == kDCS)
eba76848 2052 {
9d733021 2053 sourceName="none";
57f50b3c 2054 }
9d733021 2055 else if (system == kHLT)
9e080f92 2056 {
d386d623 2057 whereClause += Form(" and DDLnumbers=\"%s\"", source);
9d733021 2058 nFields = 3;
9e080f92 2059 }
2060
9e080f92 2061 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2062
2063 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2064
2065 // Query execution
2066 TSQLResult* aResult = 0;
9d733021 2067 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
9e080f92 2068 if (!aResult) {
9d733021 2069 Log(detector, Form("GetFileName - Can't execute SQL query to %s database for: id = %s, source = %s",
2070 GetSystemName(system), id, sourceName.Data()));
9e080f92 2071 return 0;
2072 }
2073
2074 if(aResult->GetRowCount() == 0)
2075 {
2076 Log(detector,
9d733021 2077 Form("GetFileName - No entry in %s FXS db for: id = %s, source = %s",
2078 GetSystemName(system), id, sourceName.Data()));
9e080f92 2079 delete aResult;
2080 return 0;
2081 }
2bb7b766 2082
9e080f92 2083 if (aResult->GetRowCount() > 1) {
2084 Log(detector,
9d733021 2085 Form("GetFileName - More than one entry in %s FXS db for: id = %s, source = %s",
2086 GetSystemName(system), id, sourceName.Data()));
9e080f92 2087 delete aResult;
2088 return 0;
2089 }
2090
9d733021 2091 if (aResult->GetFieldCount() != nFields) {
9e080f92 2092 Log(detector,
9d733021 2093 Form("GetFileName - Wrong field count in %s FXS db for: id = %s, source = %s",
2094 GetSystemName(system), id, sourceName.Data()));
9e080f92 2095 delete aResult;
2096 return 0;
2097 }
2098
2099 TSQLRow* aRow = dynamic_cast<TSQLRow*> (aResult->Next());
2100
2101 if (!aRow){
9d733021 2102 Log(detector, Form("GetFileName - Empty set result in %s FXS db from query: id = %s, source = %s",
2103 GetSystemName(system), id, sourceName.Data()));
9e080f92 2104 delete aResult;
2105 return 0;
2106 }
2107
2108 TString filePath(aRow->GetField(0), aRow->GetFieldLength(0));
2109 TString fileSize(aRow->GetField(1), aRow->GetFieldLength(1));
d386d623 2110 TString fileChecksum(aRow->GetField(2), aRow->GetFieldLength(2));
9e080f92 2111
2112 delete aResult;
2113 delete aRow;
2114
d386d623 2115 AliDebug(2, Form("filePath = %s; size = %s, fileChecksum = %s",
2116 filePath.Data(), fileSize.Data(), fileChecksum.Data()));
9e080f92 2117
9e080f92 2118 // retrieved file is renamed to make it unique
d524ade6 2119 TString localFileName = Form("%s/%s_process/%s_%s_%d_%s_%s.shuttle",
2120 GetShuttleTempDir(), detector,
2121 GetSystemName(system), detector, GetCurrentRun(),
2122 id, sourceName.Data());
9d733021 2123
9e080f92 2124
9d733021 2125 // file retrieval from FXS
4b95672b 2126 UInt_t nRetries = 0;
2127 UInt_t maxRetries = 3;
2128 Bool_t result = kFALSE;
2129
2130 // copy!! if successful TSystem::Exec returns 0
2131 while(nRetries++ < maxRetries) {
2132 AliDebug(2, Form("Trying to copy file. Retry # %d", nRetries));
2133 result = RetrieveFile(system, filePath.Data(), localFileName.Data());
2134 if(!result)
2135 {
2136 Log(detector, Form("GetFileName - Copy of file %s from %s FXS failed",
9d733021 2137 filePath.Data(), GetSystemName(system)));
4b95672b 2138 continue;
4f0749a8 2139 }
9e080f92 2140
d386d623 2141 if (fileChecksum.Length()>0)
4b95672b 2142 {
2143 // compare md5sum of local file with the one stored in the FXS DB
d524ade6 2144 Int_t md5Comp = gSystem->Exec(Form("md5sum %s |grep %s 2>&1 > /dev/null",
2145 localFileName.Data(), fileChecksum.Data()));
9e080f92 2146
4b95672b 2147 if (md5Comp != 0)
2148 {
2149 Log(detector, Form("GetFileName - md5sum of file %s does not match with local copy!",
2150 filePath.Data()));
2151 result = kFALSE;
2152 continue;
2153 }
d386d623 2154 } else {
2155 Log(fCurrentDetector, Form("GetFile - md5sum of file %s not set in %s database, skipping comparison",
2156 filePath.Data(), GetSystemName(system)));
9d733021 2157 }
4b95672b 2158 if (result) break;
9e080f92 2159 }
2160
4b95672b 2161 if(!result) return 0;
2162
9d733021 2163 fFXSCalled[system]=kTRUE;
2164 TObjString *fileParams = new TObjString(Form("%s#!?!#%s", id, sourceName.Data()));
2165 fFXSlist[system].Add(fileParams);
9e080f92 2166
d524ade6 2167 static TString staticLocalFileName = localFileName;
2168 //fullLocalFileName.Form("%s/%s_process/%s", GetShuttleTempDir(), detector, localFileName.Data());
36c99a6a 2169
c88ad5db 2170 Log(fCurrentDetector, Form("GetFile - Retrieved file with id %s and "
2171 "source %s from %s to %s", id, source,
d524ade6 2172 GetSystemName(system), localFileName.Data()));
9e080f92 2173
d524ade6 2174 return staticLocalFileName.Data();
2bb7b766 2175}
2176
2177//______________________________________________________________________________________________
9d733021 2178Bool_t AliShuttle::RetrieveFile(UInt_t system, const char* fxsFileName, const char* localFileName)
9e080f92 2179{
9827400b 2180 //
2181 // Copies file from FXS to local Shuttle machine
2182 //
2bb7b766 2183
9e080f92 2184 // check temp directory: trying to cd to temp; if it does not exist, create it
d524ade6 2185 AliDebug(2, Form("Copy file %s from %s FXS into %s",
2186 GetSystemName(system), fxsFileName, localFileName));
2187
2188 TString tmpDir(localFileName);
2189
2190 tmpDir = tmpDir(0,tmpDir.Last('/'));
9e080f92 2191
d524ade6 2192 Int_t noDir = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
2193 if (noDir) // temp dir does not exists!
2194 {
2195 if (gSystem->mkdir(tmpDir.Data(), 1))
2196 {
2197 Log(fCurrentDetector.Data(), "RetrieveFile - could not make temp directory!!");
9e080f92 2198 return kFALSE;
2199 }
9e080f92 2200 }
2201
9d733021 2202 TString baseFXSFolder;
2203 if (system == kDAQ)
2204 {
2205 baseFXSFolder = "FES/";
2206 }
2207 else if (system == kDCS)
2208 {
2209 baseFXSFolder = "";
2210 }
2211 else if (system == kHLT)
2212 {
42fde080 2213 baseFXSFolder = "/opt/FXS/";
9d733021 2214 }
2215
2216
d524ade6 2217 TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s",
9d733021 2218 fConfig->GetFXSPort(system),
2219 fConfig->GetFXSUser(system),
2220 fConfig->GetFXSHost(system),
2221 baseFXSFolder.Data(),
2222 fxsFileName,
9e080f92 2223 localFileName);
2224
2225 AliDebug(2, Form("%s",command.Data()));
2226
4b95672b 2227 Bool_t result = (gSystem->Exec(command.Data()) == 0);
9e080f92 2228
4b95672b 2229 return result;
9e080f92 2230}
2231
2232//______________________________________________________________________________________________
9d733021 2233TList* AliShuttle::GetFileSources(Int_t system, const char* detector, const char* id)
2234{
9827400b 2235 //
2236 // Get sources producing the condition file Id from file exchange servers
4a33bdd9 2237 // if id is NULL all sources are returned (distinct)
9827400b 2238 //
1bcd28db 2239
2240 Log(detector, Form("GetFileSources - Retrieving sources with id %s from %s", id, GetSystemName(system)));
9827400b 2241
2242 // check if test mode should simulate a FXS error
2243 if (fTestMode & kErrorFXSSources)
2244 {
2245 Log(detector, Form("GetFileSources - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2246 return 0;
2247 }
2248
9d733021 2249 if (system == kDCS)
2250 {
c88ad5db 2251 Log(detector, "GetFileSources - WARNING: DCS system has only one source of data!");
6297b37d 2252 TList *list = new TList();
2253 list->SetOwner(1);
2254 list->Add(new TObjString(" "));
2255 return list;
9d733021 2256 }
9e080f92 2257
2258 // check connection, in case connect
9d733021 2259 if (!Connect(system))
2260 {
4a33bdd9 2261 Log(detector, Form("GetFileSources - Couldn't connect to %s FXS database", GetSystemName(system)));
9d733021 2262 return NULL;
9e080f92 2263 }
2264
9d733021 2265 TString sourceName = 0;
2266 if (system == kDAQ)
2267 {
2268 sourceName = "DAQsource";
2269 } else if (system == kHLT)
2270 {
2271 sourceName = "DDLnumbers";
2272 }
2273
4a33bdd9 2274 TString sqlQueryStart = Form("select distinct %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
2275 TString whereClause = Form("run=%d and detector=\"%s\"",
2276 GetCurrentRun(), detector);
2277 if (id)
2278 whereClause += Form(" and fileId=\"%s\"", id);
9e080f92 2279 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2280
2281 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2282
2283 // Query execution
2284 TSQLResult* aResult;
9d733021 2285 aResult = fServer[system]->Query(sqlQuery);
9e080f92 2286 if (!aResult) {
9d733021 2287 Log(detector, Form("GetFileSources - Can't execute SQL query to %s database for id: %s",
2288 GetSystemName(system), id));
9e080f92 2289 return 0;
2290 }
2291
86aa42c3 2292 TList *list = new TList();
2293 list->SetOwner(1);
2294
9d733021 2295 if (aResult->GetRowCount() == 0)
2296 {
9e080f92 2297 Log(detector,
9d733021 2298 Form("GetFileSources - No entry in %s FXS table for id: %s", GetSystemName(system), id));
9e080f92 2299 delete aResult;
86aa42c3 2300 return list;
9e080f92 2301 }
2302
1bcd28db 2303 Log(detector, Form("GetFileSources - Found %d sources", aResult->GetRowCount()));
9e080f92 2304
1bcd28db 2305 TSQLRow* aRow;
9d733021 2306 while ((aRow = aResult->Next()))
2307 {
9e080f92 2308
9d733021 2309 TString source(aRow->GetField(0), aRow->GetFieldLength(0));
2310 AliDebug(2, Form("%s = %s", sourceName.Data(), source.Data()));
2311 list->Add(new TObjString(source));
9e080f92 2312 delete aRow;
2313 }
9d733021 2314
9e080f92 2315 delete aResult;
2316
2317 return list;
2bb7b766 2318}
2319
4a33bdd9 2320//______________________________________________________________________________________________
2321TList* AliShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
2322{
2323 //
2324 // Get all ids of condition files produced by a given source from file exchange servers
2325 //
2326
1bcd28db 2327 Log(detector, Form("GetFileIDs - Retrieving ids with source %s with %s", source, GetSystemName(system)));
2328
4a33bdd9 2329 // check if test mode should simulate a FXS error
2330 if (fTestMode & kErrorFXSSources)
2331 {
2332 Log(detector, Form("GetFileIDs - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
2333 return 0;
2334 }
2335
2336 // check connection, in case connect
2337 if (!Connect(system))
2338 {
2339 Log(detector, Form("GetFileIDs - Couldn't connect to %s FXS database", GetSystemName(system)));
2340 return NULL;
2341 }
2342
2343 TString sourceName = 0;
2344 if (system == kDAQ)
2345 {
2346 sourceName = "DAQsource";
2347 } else if (system == kHLT)
2348 {
2349 sourceName = "DDLnumbers";
2350 }
2351
2352 TString sqlQueryStart = Form("select fileId from %s where", fConfig->GetFXSdbTable(system));
2353 TString whereClause = Form("run=%d and detector=\"%s\"",
2354 GetCurrentRun(), detector);
2355 if (sourceName.Length() > 0 && source)
2356 whereClause += Form(" and %s=\"%s\"", sourceName.Data(), source);
2357 TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
2358
2359 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2360
2361 // Query execution
2362 TSQLResult* aResult;
2363 aResult = fServer[system]->Query(sqlQuery);
2364 if (!aResult) {
2365 Log(detector, Form("GetFileIDs - Can't execute SQL query to %s database for source: %s",
2366 GetSystemName(system), source));
2367 return 0;
2368 }
2369
2370 TList *list = new TList();
2371 list->SetOwner(1);
2372
2373 if (aResult->GetRowCount() == 0)
2374 {
2375 Log(detector,
2376 Form("GetFileIDs - No entry in %s FXS table for source: %s", GetSystemName(system), source));
2377 delete aResult;
2378 return list;
2379 }
2380
1bcd28db 2381 Log(detector, Form("GetFileIDs - Found %d ids", aResult->GetRowCount()));
2382
4a33bdd9 2383 TSQLRow* aRow;
2384
2385 while ((aRow = aResult->Next()))
2386 {
2387
2388 TString id(aRow->GetField(0), aRow->GetFieldLength(0));
2389 AliDebug(2, Form("fileId = %s", id.Data()));
2390 list->Add(new TObjString(id));
2391 delete aRow;
2392 }
2393
2394 delete aResult;
2395
2396 return list;
2397}
2398
2bb7b766 2399//______________________________________________________________________________________________
9d733021 2400Bool_t AliShuttle::Connect(Int_t system)
2bb7b766 2401{
9827400b 2402 // Connect to MySQL Server of the system's FXS MySQL databases
2403 // DAQ Logbook, Shuttle Logbook and DAQ FXS db are on the same host
2404 //
57f50b3c 2405
9d733021 2406 // check connection: if already connected return
2407 if(fServer[system] && fServer[system]->IsConnected()) return kTRUE;
57f50b3c 2408
9d733021 2409 TString dbHost, dbUser, dbPass, dbName;
57f50b3c 2410
9d733021 2411 if (system < 3) // FXS db servers
2412 {
2413 dbHost = Form("mysql://%s:%d", fConfig->GetFXSdbHost(system), fConfig->GetFXSdbPort(system));
2414 dbUser = fConfig->GetFXSdbUser(system);
2415 dbPass = fConfig->GetFXSdbPass(system);
2416 dbName = fConfig->GetFXSdbName(system);
2417 } else { // Run & Shuttle logbook servers
2418 // TODO Will the Shuttle logbook server be the same as the Run logbook server ???
2419 dbHost = Form("mysql://%s:%d", fConfig->GetDAQlbHost(), fConfig->GetDAQlbPort());
2420 dbUser = fConfig->GetDAQlbUser();
2421 dbPass = fConfig->GetDAQlbPass();
2422 dbName = fConfig->GetDAQlbDB();
2423 }
57f50b3c 2424
9d733021 2425 fServer[system] = TSQLServer::Connect(dbHost.Data(), dbUser.Data(), dbPass.Data());
2426 if (!fServer[system] || !fServer[system]->IsConnected()) {
2427 if(system < 3)
2428 {
2429 AliError(Form("Can't establish connection to FXS database for %s",
2430 AliShuttleInterface::GetSystemName(system)));
2431 } else {
2432 AliError("Can't establish connection to Run logbook.");
57f50b3c 2433 }
9d733021 2434 if(fServer[system]) delete fServer[system];
2435 return kFALSE;
2bb7b766 2436 }
57f50b3c 2437
9d733021 2438 // Get tables
2439 TSQLResult* aResult=0;
2440 switch(system){
2441 case kDAQ:
2442 aResult = fServer[kDAQ]->GetTables(dbName.Data());
2443 break;
2444 case kDCS:
2445 aResult = fServer[kDCS]->GetTables(dbName.Data());
2446 break;
2447 case kHLT:
2448 aResult = fServer[kHLT]->GetTables(dbName.Data());
2449 break;
2450 default:
2451 aResult = fServer[3]->GetTables(dbName.Data());
2452 break;
2453 }
2454
2455 delete aResult;
2bb7b766 2456 return kTRUE;
2457}
57f50b3c 2458
9e080f92 2459//______________________________________________________________________________________________
9d733021 2460Bool_t AliShuttle::UpdateTable()
9e080f92 2461{
9827400b 2462 //
2463 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2464 //
9e080f92 2465
9d733021 2466 Bool_t result = kTRUE;
9e080f92 2467
9d733021 2468 for (UInt_t system=0; system<3; system++)
2469 {
2470 if(!fFXSCalled[system]) continue;
9e080f92 2471
9d733021 2472 // check connection, in case connect
2473 if (!Connect(system))
2474 {
2475 Log(fCurrentDetector, Form("UpdateTable - Couldn't connect to %s FXS database", GetSystemName(system)));
2476 result = kFALSE;
2477 continue;
9e080f92 2478 }
9e080f92 2479
9d733021 2480 TTimeStamp now; // now
2481
2482 // Loop on FXS list entries
2483 TIter iter(&fFXSlist[system]);
2484 TObjString *aFXSentry=0;
2485 while ((aFXSentry = dynamic_cast<TObjString*> (iter.Next())))
2486 {
2487 TString aFXSentrystr = aFXSentry->String();
2488 TObjArray *aFXSarray = aFXSentrystr.Tokenize("#!?!#");
2489 if (!aFXSarray || aFXSarray->GetEntries() != 2 )
2490 {
2491 Log(fCurrentDetector, Form("UpdateTable - error updating %s FXS entry. Check string: <%s>",
2492 GetSystemName(system), aFXSentrystr.Data()));
2493 if(aFXSarray) delete aFXSarray;
2494 result = kFALSE;
2495 continue;
2496 }
2497 const char* fileId = ((TObjString*) aFXSarray->At(0))->GetName();
2498 const char* source = ((TObjString*) aFXSarray->At(1))->GetName();
2499
2500 TString whereClause;
2501 if (system == kDAQ)
2502 {
2503 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DAQsource=\"%s\";",
2504 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2505 }
2506 else if (system == kDCS)
2507 {
2508 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\";",
2509 GetCurrentRun(), fCurrentDetector.Data(), fileId);
2510 }
2511 else if (system == kHLT)
2512 {
2513 whereClause = Form("where run=%d and detector=\"%s\" and fileId=\"%s\" and DDLnumbers=\"%s\";",
2514 GetCurrentRun(), fCurrentDetector.Data(), fileId, source);
2515 }
2516
2517 delete aFXSarray;
9e080f92 2518
9d733021 2519 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2520 now.GetSec(), whereClause.Data());
9e080f92 2521
9d733021 2522 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
9e080f92 2523
9d733021 2524 // Query execution
2525 TSQLResult* aResult;
2526 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2527 if (!aResult)
2528 {
2529 Log(fCurrentDetector, Form("UpdateTable - %s db: can't execute SQL query <%s>",
2530 GetSystemName(system), sqlQuery.Data()));
2531 result = kFALSE;
2532 continue;
2533 }
2534 delete aResult;
9e080f92 2535 }
9e080f92 2536 }
2537
9d733021 2538 return result;
9e080f92 2539}
57f50b3c 2540
3301427a 2541//______________________________________________________________________________________________
2542Bool_t AliShuttle::UpdateTableFailCase()
2543{
9827400b 2544 // Update FXS table filling time_processed field in all rows corresponding to current run and detector
2545 // this is called in case the preprocessor is declared failed for the current run, because
2546 // the fields are updated only in case of success
3301427a 2547
2548 Bool_t result = kTRUE;
2549
2550 for (UInt_t system=0; system<3; system++)
2551 {
2552 // check connection, in case connect
2553 if (!Connect(system))
2554 {
2555 Log(fCurrentDetector, Form("UpdateTableFailCase - Couldn't connect to %s FXS database",
2556 GetSystemName(system)));
2557 result = kFALSE;
2558 continue;
2559 }
2560
2561 TTimeStamp now; // now
2562
2563 // Loop on FXS list entries
2564
2565 TString whereClause = Form("where run=%d and detector=\"%s\";",
2566 GetCurrentRun(), fCurrentDetector.Data());
2567
2568
2569 TString sqlQuery = Form("update %s set time_processed=%d %s", fConfig->GetFXSdbTable(system),
2570 now.GetSec(), whereClause.Data());
2571
2572 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2573
2574 // Query execution
2575 TSQLResult* aResult;
2576 aResult = dynamic_cast<TSQLResult*> (fServer[system]->Query(sqlQuery));
2577 if (!aResult)
2578 {
2579 Log(fCurrentDetector, Form("UpdateTableFailCase - %s db: can't execute SQL query <%s>",
2580 GetSystemName(system), sqlQuery.Data()));
2581 result = kFALSE;
2582 continue;
2583 }
2584 delete aResult;
2585 }
2586
2587 return result;
2588}
2589
2bb7b766 2590//______________________________________________________________________________________________
2591Bool_t AliShuttle::UpdateShuttleLogbook(const char* detector, const char* status)
2592{
e7f62f16 2593 //
2594 // Update Shuttle logbook filling detector or shuttle_done column
2595 // ex. of usage: UpdateShuttleLogbook("PHOS", "DONE") or UpdateShuttleLogbook("shuttle_done")
2596 //
57f50b3c 2597
2bb7b766 2598 // check connection, in case connect
be48e3ea 2599 if(!Connect(3)){
2bb7b766 2600 Log("SHUTTLE", "UpdateShuttleLogbook - Couldn't connect to DAQ Logbook.");
2601 return kFALSE;
57f50b3c 2602 }
2603
2bb7b766 2604 TString detName(detector);
2605 TString setClause;
e7f62f16 2606 if(detName == "shuttle_done")
2607 {
2bb7b766 2608 setClause = "set shuttle_done=1";
e7f62f16 2609
2610 // Send the information to ML
2611 TMonaLisaText mlStatus("SHUTTLE_status", "Done");
2612
2613 TList mlList;
2614 mlList.Add(&mlStatus);
2615
2616 fMonaLisa->SendParameters(&mlList);
2bb7b766 2617 } else {
2bb7b766 2618 TString statusStr(status);
2619 if(statusStr.Contains("done", TString::kIgnoreCase) ||
2620 statusStr.Contains("failed", TString::kIgnoreCase)){
eba76848 2621 setClause = Form("set %s=\"%s\"", detector, status);
2bb7b766 2622 } else {
2623 Log("SHUTTLE",
2624 Form("UpdateShuttleLogbook - Invalid status <%s> for detector %s",
2625 status, detector));
2626 return kFALSE;
2627 }
2628 }
57f50b3c 2629
2bb7b766 2630 TString whereClause = Form("where run=%d", GetCurrentRun());
2631
441b0e9c 2632 TString sqlQuery = Form("update %s %s %s",
2633 fConfig->GetShuttlelbTable(), setClause.Data(), whereClause.Data());
57f50b3c 2634
2bb7b766 2635 AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
2636
2637 // Query execution
2638 TSQLResult* aResult;
be48e3ea 2639 aResult = dynamic_cast<TSQLResult*> (fServer[3]->Query(sqlQuery));
2bb7b766 2640 if (!aResult) {
2641 Log("SHUTTLE", Form("UpdateShuttleLogbook - Can't execute query <%s>", sqlQuery.Data()));
2642 return kFALSE;
57f50b3c 2643 }
2bb7b766 2644 delete aResult;
57f50b3c 2645
2646 return kTRUE;
2647}
2648
2649//______________________________________________________________________________________________
2bb7b766 2650Int_t AliShuttle::GetCurrentRun() const
2651{
9827400b 2652 //
2653 // Get current run from logbook entry
2654 //
57f50b3c 2655
2bb7b766 2656 return fLogbookEntry ? fLogbookEntry->GetRun() : -1;
57f50b3c 2657}
2658
2659//______________________________________________________________________________________________
2bb7b766 2660UInt_t AliShuttle::GetCurrentStartTime() const
2661{
9827400b 2662 //
2663 // get current start time
2664 //
57f50b3c 2665
2bb7b766 2666 return fLogbookEntry ? fLogbookEntry->GetStartTime() : 0;
57f50b3c 2667}
2668
2669//______________________________________________________________________________________________
2bb7b766 2670UInt_t AliShuttle::GetCurrentEndTime() const
2671{
9827400b 2672 //
2673 // get current end time from logbook entry
2674 //
57f50b3c 2675
2bb7b766 2676 return fLogbookEntry ? fLogbookEntry->GetEndTime() : 0;
57f50b3c 2677}
2678
b948db8d 2679//______________________________________________________________________________________________
2680void AliShuttle::Log(const char* detector, const char* message)
2681{
9827400b 2682 //
2683 // Fill log string with a message
2684 //
b948db8d 2685
36c99a6a 2686 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
84090f85 2687 if (dir == NULL) {
36c99a6a 2688 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE)) {
2689 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
84090f85 2690 return;
2691 }
b948db8d 2692
84090f85 2693 } else {
2694 gSystem->FreeDirectory(dir);
2695 }
b948db8d 2696
cb343cfd 2697 TString toLog = Form("%s (%d): %s - ", TTimeStamp(time(0)).AsString("s"), getpid(), detector);
e7f62f16 2698 if (GetCurrentRun() >= 0)
2699 toLog += Form("run %d - ", GetCurrentRun());
2bb7b766 2700 toLog += Form("%s", message);
2701
84090f85 2702 AliInfo(toLog.Data());
ffa29e93 2703
2704 // if we redirect the log output already to the file, leave here
2705 if (fOutputRedirected && strcmp(detector, "SHUTTLE") != 0)
2706 return;
b948db8d 2707
ffa29e93 2708 TString fileName = GetLogFileName(detector);
e7f62f16 2709
84090f85 2710 gSystem->ExpandPathName(fileName);
2711
2712 ofstream logFile;
2713 logFile.open(fileName, ofstream::out | ofstream::app);
2714
2715 if (!logFile.is_open()) {
2716 AliError(Form("Could not open file %s", fileName.Data()));
2717 return;
2718 }
7bfb2090 2719
84090f85 2720 logFile << toLog.Data() << "\n";
b948db8d 2721
84090f85 2722 logFile.close();
b948db8d 2723}
2bb7b766 2724
ffa29e93 2725//______________________________________________________________________________________________
2726TString AliShuttle::GetLogFileName(const char* detector) const
2727{
2728 //
2729 // returns the name of the log file for a given sub detector
2730 //
2731
2732 TString fileName;
2733
2734 if (GetCurrentRun() >= 0)
2735 fileName.Form("%s/%s_%d.log", GetShuttleLogDir(), detector, GetCurrentRun());
2736 else
2737 fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
2738
2739 return fileName;
2740}
2741
2bb7b766 2742//______________________________________________________________________________________________
2743Bool_t AliShuttle::Collect(Int_t run)
2744{
9827400b 2745 //
2746 // Collects conditions data for all UNPROCESSED run written to DAQ LogBook in case of run = -1 (default)
2747 // If a dedicated run is given this run is processed
2748 //
2749 // In operational mode, this is the Shuttle function triggered by the EOR signal.
2750 //
2bb7b766 2751
eba76848 2752 if (run == -1)
2753 Log("SHUTTLE","Collect - Shuttle called. Collecting conditions data for unprocessed runs");
2754 else
2755 Log("SHUTTLE", Form("Collect - Shuttle called. Collecting conditions data for run %d", run));
cb343cfd 2756
2757 SetLastAction("Starting");
2bb7b766 2758
2759 TString whereClause("where shuttle_done=0");
eba76848 2760 if (run != -1)
2761 whereClause += Form(" and run=%d", run);
2bb7b766 2762
2763 TObjArray shuttleLogbookEntries;
be48e3ea 2764 if (!QueryShuttleLogbook(whereClause, shuttleLogbookEntries))
2765 {
cb343cfd 2766 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2bb7b766 2767 return kFALSE;
2768 }
2769
9e080f92 2770 if (shuttleLogbookEntries.GetEntries() == 0)
2771 {
2772 if (run == -1)
2773 Log("SHUTTLE","Collect - Found no UNPROCESSED runs in Shuttle logbook");
2774 else
2775 Log("SHUTTLE", Form("Collect - Run %d is already DONE "
2776 "or it does not exist in Shuttle logbook", run));
2777 return kTRUE;
2778 }
2779
be48e3ea 2780 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2781 fFirstUnprocessed[iDet] = kTRUE;
2782
fc5a4708 2783 if (run != -1)
be48e3ea 2784 {
2785 // query Shuttle logbook for earlier runs, check if some detectors are unprocessed,
2786 // flag them into fFirstUnprocessed array
2787 TString whereClause(Form("where shuttle_done=0 and run < %d", run));
2788 TObjArray tmpLogbookEntries;
2789 if (!QueryShuttleLogbook(whereClause, tmpLogbookEntries))
2790 {
2791 Log("SHUTTLE", "Collect - Can't retrieve entries from Shuttle logbook");
2792 return kFALSE;
2793 }
2794
2795 TIter iter(&tmpLogbookEntries);
2796 AliShuttleLogbookEntry* anEntry = 0;
2797 while ((anEntry = dynamic_cast<AliShuttleLogbookEntry*> (iter.Next())))
2798 {
2799 for (UInt_t iDet=0; iDet<NDetectors(); iDet++)
2800 {
2801 if (anEntry->GetDetectorStatus(iDet) == AliShuttleLogbookEntry::kUnprocessed)
2802 {
2803 AliDebug(2, Form("Run %d: setting %s as \"not first time unprocessed\"",
2804 anEntry->GetRun(), GetDetName(iDet)));
2805 fFirstUnprocessed[iDet] = kFALSE;
2806 }
2807 }
2808
2809 }
2810
2811 }
2812
2813 if (!RetrieveConditionsData(shuttleLogbookEntries))
2814 {
cb343cfd 2815 Log("SHUTTLE", "Collect - Process of at least one run failed");
2bb7b766 2816 return kFALSE;
2817 }
2818
36c99a6a 2819 Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
eba76848 2820 return kTRUE;
2bb7b766 2821}
2822
2bb7b766 2823//______________________________________________________________________________________________
2824Bool_t AliShuttle::RetrieveConditionsData(const TObjArray& dateEntries)
2825{
9827400b 2826 //
2827 // Retrieve conditions data for all runs that aren't processed yet
2828 //
2bb7b766 2829
2830 Bool_t hasError = kFALSE;
2831
2832 TIter iter(&dateEntries);
2833 AliShuttleLogbookEntry* anEntry;
2834
2835 while ((anEntry = (AliShuttleLogbookEntry*) iter.Next())){
2836 if (!Process(anEntry)){
2837 hasError = kTRUE;
2838 }
4b95672b 2839
2840 // clean SHUTTLE temp directory
d524ade6 2841 //TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
2842 //RemoveFile(filename.Data());
2bb7b766 2843 }
2844
2845 return hasError == kFALSE;
2846}
cb343cfd 2847
2848//______________________________________________________________________________________________
2849ULong_t AliShuttle::GetTimeOfLastAction() const
2850{
9827400b 2851 //
2852 // Gets time of last action
2853 //
2854
cb343cfd 2855 ULong_t tmp;
36c99a6a 2856
cb343cfd 2857 fMonitoringMutex->Lock();
be48e3ea 2858
cb343cfd 2859 tmp = fLastActionTime;
36c99a6a 2860
cb343cfd 2861 fMonitoringMutex->UnLock();
36c99a6a 2862
cb343cfd 2863 return tmp;
2864}
2865
2866//______________________________________________________________________________________________
2867const TString AliShuttle::GetLastAction() const
2868{
9827400b 2869 //
cb343cfd 2870 // returns a string description of the last action
9827400b 2871 //
cb343cfd 2872
2873 TString tmp;
36c99a6a 2874
cb343cfd 2875 fMonitoringMutex->Lock();
2876
2877 tmp = fLastAction;
2878
2879 fMonitoringMutex->UnLock();
2880
36c99a6a 2881 return tmp;
cb343cfd 2882}
2883
2884//______________________________________________________________________________________________
2885void AliShuttle::SetLastAction(const char* action)
2886{
9827400b 2887 //
cb343cfd 2888 // updates the monitoring variables
9827400b 2889 //
36c99a6a 2890
cb343cfd 2891 fMonitoringMutex->Lock();
36c99a6a 2892
cb343cfd 2893 fLastAction = action;
2894 fLastActionTime = time(0);
2895
2896 fMonitoringMutex->UnLock();
2897}
eba76848 2898
2899//______________________________________________________________________________________________
2900const char* AliShuttle::GetRunParameter(const char* param)
2901{
9827400b 2902 //
2903 // returns run parameter read from DAQ logbook
2904 //
eba76848 2905
2906 if(!fLogbookEntry) {
2907 AliError("No logbook entry!");
2908 return 0;
2909 }
2910
2911 return fLogbookEntry->GetRunParameter(param);
2912}
57c1a579 2913
d386d623 2914//______________________________________________________________________________________________
9827400b 2915AliCDBEntry* AliShuttle::GetFromOCDB(const char* detector, const AliCDBPath& path)
d386d623 2916{
9827400b 2917 //
2918 // returns object from OCDB valid for current run
2919 //
d386d623 2920
9827400b 2921 if (fTestMode & kErrorOCDB)
2922 {
2923 Log(detector, "GetFromOCDB - In TESTMODE - Simulating error with OCDB");
2924 return 0;
2925 }
2926
d386d623 2927 AliCDBStorage *sto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
2928 if (!sto)
2929 {
9827400b 2930 Log(detector, "GetFromOCDB - Cannot activate main OCDB for query!");
d386d623 2931 return 0;
2932 }
2933
2934 return dynamic_cast<AliCDBEntry*> (sto->Get(path, GetCurrentRun()));
2935}
2936
57c1a579 2937//______________________________________________________________________________________________
2938Bool_t AliShuttle::SendMail()
2939{
9827400b 2940 //
2941 // sends a mail to the subdetector expert in case of preprocessor error
2942 //
2943
2944 if (fTestMode != kNone)
2945 return kTRUE;
57c1a579 2946
36c99a6a 2947 void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
57c1a579 2948 if (dir == NULL)
2949 {
36c99a6a 2950 if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
57c1a579 2951 {
36c99a6a 2952 AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
57c1a579 2953 return kFALSE;
2954 }
2955
2956 } else {
2957 gSystem->FreeDirectory(dir);
2958 }
2959
2960 TString bodyFileName;
36c99a6a 2961 bodyFileName.Form("%s/mail.body", GetShuttleLogDir());
57c1a579 2962 gSystem->ExpandPathName(bodyFileName);
2963
2964 ofstream mailBody;
2965 mailBody.open(bodyFileName, ofstream::out);
2966
2967 if (!mailBody.is_open())
2968 {
2969 AliError(Form("Could not open mail body file %s", bodyFileName.Data()));
2970 return kFALSE;
2971 }
2972
2973 TString to="";
2974 TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
2975 TObjString *anExpert=0;
2976 while ((anExpert = (TObjString*) iterExperts.Next()))
2977 {
2978 to += Form("%s,", anExpert->GetName());
2979 }
2980 to.Remove(to.Length()-1);
909732f7 2981 AliDebug(2, Form("to: %s",to.Data()));
57c1a579 2982
86aa42c3 2983 if (to.IsNull()) {
c88ad5db 2984 Log("SHUTTLE", "List of detector responsibles not yet set!");
36c99a6a 2985 return kFALSE;
2986 }
2987
57c1a579 2988 TString cc="alberto.colla@cern.ch";
2989
546242fb 2990 TString subject = Form("%s Shuttle preprocessor FAILED in run %d !",
57c1a579 2991 fCurrentDetector.Data(), GetCurrentRun());
909732f7 2992 AliDebug(2, Form("subject: %s", subject.Data()));
57c1a579 2993
2994 TString body = Form("Dear %s expert(s), \n\n", fCurrentDetector.Data());
2995 body += Form("SHUTTLE just detected that your preprocessor "
546242fb 2996 "failed processing run %d!!\n\n", GetCurrentRun());
2997 body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n", fCurrentDetector.Data());
2998 body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
2999 body += Form("Find the %s log for the current run on \n\n"
3000 "\thttp://pcalishuttle01.cern.ch:8880/logs/%s_%d.log \n\n",
3001 fCurrentDetector.Data(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 3002 body += Form("The last 10 lines of %s log file are following:\n\n");
3003
909732f7 3004 AliDebug(2, Form("Body begin: %s", body.Data()));
57c1a579 3005
3006 mailBody << body.Data();
3007 mailBody.close();
3008 mailBody.open(bodyFileName, ofstream::out | ofstream::app);
3009
9d733021 3010 TString logFileName = Form("%s/%s_%d.log", GetShuttleLogDir(), fCurrentDetector.Data(), GetCurrentRun());
57c1a579 3011 TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
3012 if (gSystem->Exec(tailCommand.Data()))
3013 {
3014 mailBody << Form("%s log file not found ...\n\n", fCurrentDetector.Data());
3015 }
3016
3017 TString endBody = Form("------------------------------------------------------\n\n");
36c99a6a 3018 endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
3019 endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
546242fb 3020 endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
57c1a579 3021
909732f7 3022 AliDebug(2, Form("Body end: %s", endBody.Data()));
57c1a579 3023
3024 mailBody << endBody.Data();
3025
3026 mailBody.close();
3027
3028 // send mail!
3029 TString mailCommand = Form("mail -s \"%s\" -c %s %s < %s",
3030 subject.Data(),
3031 cc.Data(),
3032 to.Data(),
3033 bodyFileName.Data());
909732f7 3034 AliDebug(2, Form("mail command: %s", mailCommand.Data()));
57c1a579 3035
3036 Bool_t result = gSystem->Exec(mailCommand.Data());
3037
3038 return result == 0;
3039}
d386d623 3040
441b0e9c 3041//______________________________________________________________________________________________
9827400b 3042const char* AliShuttle::GetRunType()
441b0e9c 3043{
9827400b 3044 //
3045 // returns run type read from "run type" logbook
3046 //
441b0e9c 3047
3048 if(!fLogbookEntry) {
3049 AliError("No logbook entry!");
3050 return 0;
3051 }
3052
9827400b 3053 return fLogbookEntry->GetRunType();
441b0e9c 3054}
3055
4859271b 3056//______________________________________________________________________________________________
3057Bool_t AliShuttle::GetHLTStatus()
3058{
3059 // Return HLT status (ON=1 OFF=0)
3060 // Converts the HLT status from the status string read in the run logbook (not just a bool)
3061
3062 if(!fLogbookEntry) {
3063 AliError("No logbook entry!");
3064 return 0;
3065 }
3066
3067 // TODO implement when HLTStatus is inserted in run logbook
3068 //TString hltStatus = fLogbookEntry->GetRunParameter("HLTStatus");
3069 //if(hltStatus == "OFF") {return kFALSE};
3070
3071 return kTRUE;
3072}
3073
d386d623 3074//______________________________________________________________________________________________
3075void AliShuttle::SetShuttleTempDir(const char* tmpDir)
3076{
9827400b 3077 //
3078 // sets Shuttle temp directory
3079 //
d386d623 3080
3081 fgkShuttleTempDir = gSystem->ExpandPathName(tmpDir);
3082}
3083
3084//______________________________________________________________________________________________
3085void AliShuttle::SetShuttleLogDir(const char* logDir)
3086{
9827400b 3087 //
3088 // sets Shuttle log directory
3089 //
d386d623 3090
3091 fgkShuttleLogDir = gSystem->ExpandPathName(logDir);
3092}