* provided "as is" without express or implied warranty. *
**************************************************************************/
-/*
-$Log$
-Revision 1.65 2007/11/26 16:58:37 acolla
-Monalisa configuration added: host and table name
-
-Revision 1.64 2007/11/13 16:15:47 acolla
-DCS map is stored in a file in the temp folder where the detector is processed.
-If the preprocessor fails, the temp folder is not removed. This will help the debugging of the problem.
-
-Revision 1.63 2007/11/02 10:53:16 acolla
-Protection added to AliShuttle::CopyFileLocally
-
-Revision 1.62 2007/10/31 18:23:13 acolla
-Furter developement on the Shuttle:
-
-- Shuttle now connects to the Grid as alidaq. The OCDB and Reference folders
-are now built from /alice/data, e.g.:
-/alice/data/2007/LHC07a/OCDB
-
-the year and LHC period are taken from the Shuttle.
-Raw metadata files are stored by GRP to:
-/alice/data/2007/LHC07a/<runNb>/Raw/RunMetadata.root
-
-- Shuttle sends a mail to DCS experts each time DP retrieval fails.
-
-Revision 1.61 2007/10/30 20:33:51 acolla
-Improved managing of temporary folders, which weren't correctly handled.
-Resolved bug introduced in StoreReferenceFile, which caused SPD preprocessor fail.
-
-Revision 1.60 2007/10/29 18:06:16 acolla
-
-New function StoreRunMetadataFile added to preprocessor and Shuttle interface
-This function can be used by GRP only. It stores raw data tags merged file to the
-raw data folder (e.g. /alice/data/2008/LHC08a/000099999/Raw).
-
-KNOWN ISSUES:
-
-1. Shuttle cannot write to /alice/data/ because it belongs to alidaq. Tag file is stored in /alice/simulation/... for the time being.
-2. Due to a bug in TAlien::Mkdir, the creation of a folder in recursive mode (-p option) does not work. The problem
-has been corrected in the root package on the Shuttle machine.
-
-Revision 1.59 2007/10/05 12:40:55 acolla
-
-Result error code added to AliDCSClient data members (it was "lost" with the new implementation of TMap* GetAliasValues and GetDPValues).
-
-Revision 1.58 2007/09/28 15:27:40 acolla
-
-AliDCSClient "multiSplit" option added in the DCS configuration
-in AliDCSMessage: variable MAX_BODY_SIZE set to 500000
-
-Revision 1.57 2007/09/27 16:53:13 acolla
-Detectors can have more than one AMANDA server. SHUTTLE queries the servers sequentially,
-merges the dcs aliases/DPs in one TMap and sends it to the preprocessor.
-
-Revision 1.56 2007/09/14 16:46:14 jgrosseo
-1) Connect and Close are called before and after each query, so one can
-keep the same AliDCSClient object.
-2) The splitting of a query is moved to GetDPValues/GetAliasValues.
-3) Splitting interval can be specified in constructor
-
-Revision 1.55 2007/08/06 12:26:40 acolla
-Function Bool_t GetHLTStatus added to preprocessor. It returns the status of HLT
-read from the run logbook.
-
-Revision 1.54 2007/07/12 09:51:25 jgrosseo
-removed duplicated log message in GetFile
-
-Revision 1.53 2007/07/12 09:26:28 jgrosseo
-updating hlt fxs base path
-
-Revision 1.52 2007/07/12 08:06:45 jgrosseo
-adding log messages in getfile... functions
-adding not implemented copy constructor in alishuttleconfigholder
-
-Revision 1.51 2007/07/03 17:24:52 acolla
-root moved to v5-16-00. TFileMerger->Cp moved to TFile::Cp.
-
-Revision 1.50 2007/07/02 17:19:32 acolla
-preprocessor is run in a temp directory that is removed when process is finished.
-
-Revision 1.49 2007/06/29 10:45:06 acolla
-Number of columns in MySql Shuttle logbook increased by one (HLT added)
-
-Revision 1.48 2007/06/21 13:06:19 acolla
-GetFileSources returns dummy list with 1 source if system=DCS (better than
-returning error as it was)
-
-Revision 1.47 2007/06/19 17:28:56 acolla
-HLT updated; missing map bug removed.
-
-Revision 1.46 2007/06/09 13:01:09 jgrosseo
-Switching to retrieval of several DCS DPs at a time (multiDPrequest)
-
-Revision 1.45 2007/05/30 06:35:20 jgrosseo
-Adding functionality to the Shuttle/TestShuttle:
-o) Function to retrieve list of sources from a given system (GetFileSources with id=0)
-o) Function to retrieve list of IDs for a given source (GetFileIDs)
-These functions are needed for dealing with the tag files that are saved for the GRP preprocessor
-Example code has been added to the TestProcessor in TestShuttle
-
-Revision 1.44 2007/05/11 16:09:32 acolla
-Reference files for ITS, MUON and PHOS are now stored in OfflineDetName/OnlineDetName/run_...
-example: ITS/SPD/100_filename.root
-
-Revision 1.43 2007/05/10 09:59:51 acolla
-Various bug fixes in StoreRefFilesToGrid; Cleaning of reference storage before processing detector (CleanReferenceStorage)
-
-Revision 1.42 2007/05/03 08:01:39 jgrosseo
-typo in last commit :-(
-
-Revision 1.41 2007/05/03 08:00:48 jgrosseo
-fixing log message when pp want to skip dcs value retrieval
-
-Revision 1.40 2007/04/27 07:06:48 jgrosseo
-GetFileSources returns empty list in case of no files, but successful query
-No mails sent in testmode
-
-Revision 1.39 2007/04/17 12:43:57 acolla
-Correction in StoreOCDB; change of text in mail to detector expert
-
-Revision 1.38 2007/04/12 08:26:18 jgrosseo
-updated comment
-
-Revision 1.37 2007/04/10 16:53:14 jgrosseo
-redirecting sub detector stdout, stderr to sub detector log file
-
-Revision 1.35 2007/04/04 16:26:38 acolla
-1. Re-organization of function calls in TestPreprocessor to make it more meaningful.
-2. Added missing dependency in test preprocessors.
-3. in AliShuttle.cxx: processing time and memory consumption info on a single line.
-
-Revision 1.34 2007/04/04 10:33:36 jgrosseo
-1) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
-In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
-
-2) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
-
-3) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
-
-4) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
-
-5) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
-If you always need DCS data (like before), you do not need to implement it.
-
-6) The run type has been added to the monitoring page
-
-Revision 1.33 2007/04/03 13:56:01 acolla
-Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
-run type.
-
-Revision 1.32 2007/02/28 10:41:56 acolla
-Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
-AliPreprocessor::GetRunType() function.
-Added some ldap definition files.
-
-Revision 1.30 2007/02/13 11:23:21 acolla
-Moved getters and setters of Shuttle's main OCDB/Reference, local
-OCDB/Reference, temp and log folders to AliShuttleInterface
-
-Revision 1.27 2007/01/30 17:52:42 jgrosseo
-adding monalisa monitoring
-
-Revision 1.26 2007/01/23 19:20:03 acolla
-Removed old ldif files, added TOF, MCH ldif files. Added some options in
-AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
-SetShuttleLogDir
-
-Revision 1.25 2007/01/15 19:13:52 acolla
-Moved some AliInfo to AliDebug in SendMail function
-
-Revision 1.21 2006/12/07 08:51:26 jgrosseo
-update (alberto):
-table, db names in ldap configuration
-added GRP preprocessor
-DCS data can also be retrieved by data point
-
-Revision 1.20 2006/11/16 16:16:48 jgrosseo
-introducing strict run ordering flag
-removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
-
-Revision 1.19 2006/11/06 14:23:04 jgrosseo
-major update (Alberto)
-o) reading of run parameters from the logbook
-o) online offline naming conversion
-o) standalone DCSclient package
-
-Revision 1.18 2006/10/20 15:22:59 jgrosseo
-o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
-o) Merging Collect, CollectAll, CollectNew function
-o) Removing implementation of empty copy constructors (declaration still there!)
-
-Revision 1.17 2006/10/05 16:20:55 jgrosseo
-adapting to new CDB classes
-
-Revision 1.16 2006/10/05 15:46:26 jgrosseo
-applying to the new interface
-
-Revision 1.15 2006/10/02 16:38:39 jgrosseo
-update (alberto):
-fixed memory leaks
-storing of objects that failed to be stored to the grid before
-interfacing of shuttle status table in daq system
-
-Revision 1.14 2006/08/29 09:16:05 jgrosseo
-small update
-
-Revision 1.13 2006/08/15 10:50:00 jgrosseo
-effc++ corrections (alberto)
-
-Revision 1.12 2006/08/08 14:19:29 jgrosseo
-Update to shuttle classes (Alberto)
-
-- Possibility to set the full object's path in the Preprocessor's and
-Shuttle's Store functions
-- Possibility to extend the object's run validity in the same classes
-("startValidity" and "validityInfinite" parameters)
-- Implementation of the StoreReferenceData function to store reference
-data in a dedicated CDB storage.
-
-Revision 1.11 2006/07/21 07:37:20 jgrosseo
-last run is stored after each run
-
-Revision 1.10 2006/07/20 09:54:40 jgrosseo
-introducing status management: The processing per subdetector is divided into several steps,
-after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
-can keep track of the number of failures and skips further processing after a certain threshold is
-exceeded. These thresholds can be configured in LDAP.
-
-Revision 1.9 2006/07/19 10:09:55 jgrosseo
-new configuration, accesst to DAQ FES (Alberto)
-
-Revision 1.8 2006/07/11 12:44:36 jgrosseo
-adding parameters for extended validity range of data produced by preprocessor
-
-Revision 1.7 2006/07/10 14:37:09 jgrosseo
-small fix + todo comment
-
-Revision 1.6 2006/07/10 13:01:41 jgrosseo
-enhanced storing of last sucessfully processed run (alberto)
-
-Revision 1.5 2006/07/04 14:59:57 jgrosseo
-revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
-
-Revision 1.4 2006/06/12 09:11:16 jgrosseo
-coding conventions (Alberto)
-
-Revision 1.3 2006/06/06 14:26:40 jgrosseo
-o) removed files that were moved to STEER
-o) shuttle updated to follow the new interface (Alberto)
-
-Revision 1.2 2006/03/07 07:52:34 hristov
-New version (B.Yordanov)
-
-Revision 1.6 2005/11/19 17:19:14 byordano
-RetrieveDATEEntries and RetrieveConditionsData added
-
-Revision 1.5 2005/11/19 11:09:27 byordano
-AliShuttle declaration added
-
-Revision 1.4 2005/11/17 17:47:34 byordano
-TList changed to TObjArray
-
-Revision 1.3 2005/11/17 14:43:23 byordano
-import to local CVS
-
-Revision 1.1.1.1 2005/10/28 07:33:58 hristov
-Initial import as subdirectory in AliRoot
-
-Revision 1.2 2005/09/13 08:41:15 byordano
-default startTime endTime added
-
-Revision 1.4 2005/08/30 09:13:02 byordano
-some docs added
-
-Revision 1.3 2005/08/29 21:15:47 byordano
-some docs added
-
-*/
+/* $Id$ */
//
// This class is the main manager for AliShuttle.
#include <sys/types.h>
#include <sys/wait.h>
+#include <signal.h>
+
ClassImp(AliShuttle)
//______________________________________________________________________________________________
Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
"there are previous unprocessed runs!",
fCurrentDetector.Data(), aLocId.GetPath().Data()));
+ result = kFALSE;
continue;
}
TString localBaseFolder = sto->GetBaseFolder();
// Build Run level folder
- // folder = /alice/data/year/lhcPeriod/runNb/Raw
+ // folder = /alice/data/year/lhcPeriod/runNb/raw
TString lhcPeriod = GetLHCPeriod();
return 0;
}
- TString target = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw/%s",
+ // TODO partitions with one detector only write data into LHCperiod_DET
+ TString partition = GetRunParameter("detector");
+
+ if (partition.Length() > 0 && partition != "ALICE")
+ {
+ lhcPeriod.Append(Form("_%s", partition.Data()));
+ Log(fCurrentDetector, Form("Run data tags merged file will be written in %s",
+ lhcPeriod.Data()));
+ }
+
+ TString target = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/raw/%s",
localBaseFolder.Data(), GetCurrentYear(),
lhcPeriod.Data(), GetCurrentRun(), gridFileName);
void* dir = gSystem->OpenDirectory(targetDir.Data());
if (dir == NULL) {
if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
- Log("SHUTTLE", Form("StoreFileLocally - Can't open directory <%s>", targetDir.Data()));
+ Log("SHUTTLE", Form("CopyFileLocally - Can't open directory <%s>", targetDir.Data()));
return kFALSE;
}
result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
if (result)
{
- Log("SHUTTLE", Form("StoreFileLocally - %s does not exist", localFile));
+ Log("SHUTTLE", Form("CopyFileLocally - %s does not exist", localFile));
return kFALSE;
}
result = gSystem->GetPathInfo(target, 0, (Long64_t*) 0, 0, 0);
if (!result)
{
- Log("SHUTTLE", Form("StoreFileLocally - target file %s already exist, removing...", target.Data()));
+ Log("SHUTTLE", Form("CopyFileLocally - target file %s already exist, removing...", target.Data()));
if (gSystem->Unlink(target.Data()))
{
- Log("SHUTTLE", Form("StoreFileLocally - Could not remove existing target file %s!", target.Data()));
+ Log("SHUTTLE", Form("CopyFileLocally - Could not remove existing target file %s!", target.Data()));
return kFALSE;
}
}
if (result == 0)
{
- Log("SHUTTLE", Form("StoreFileLocally - File %s stored locally to %s", localFile, target.Data()));
+ Log("SHUTTLE", Form("CopyFileLocally - File %s stored locally to %s", localFile, target.Data()));
return kTRUE;
}
else
{
- Log("SHUTTLE", Form("StoreFileLocally - Could not store file %s to %s! Error code = %d",
+ Log("SHUTTLE", Form("CopyFileLocally - Could not store file %s to %s! Error code = %d",
localFile, target.Data(), result));
return kFALSE;
}
return 0;
}
- dir = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/Raw",
+ // TODO partitions with one detector only write data into LHCperiod_DET
+ TString partition = GetRunParameter("detector");
+
+ if (partition.Length() > 0 && partition != "ALICE")
+ {
+ lhcPeriod.Append(Form("_%s", partition.Data()));
+ }
+
+ dir = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/raw",
localBaseFolder.Data(), GetCurrentYear(),
lhcPeriod.Data(), GetCurrentRun());
alienDir = dir(dir.Index("/alice/data/"), dir.Length());
mlList.Add(&mlStatus);
mlList.Add(&mlRetryCount);
- fMonaLisa->SendParameters(&mlList);
+ TString mlID;
+ mlID.Form("%d", GetCurrentRun());
+ fMonaLisa->SendParameters(&mlList, mlID);
}
//______________________________________________________________________________________________
return kFALSE;
}
- if (status->GetStatus() == AliShuttleStatus::kStoreError) {
+ if (status->GetStatus() == AliShuttleStatus::kStoreStarted || status->GetStatus() == AliShuttleStatus::kStoreError) {
Log("SHUTTLE",
Form("ContinueProcessing - %s: Grid storage of one or more "
"objects failed. Trying again now",
"successfully stored into main storage",
fCurrentDetector.Data()));
UpdateShuttleStatus(AliShuttleStatus::kDone);
- UpdateShuttleLogbook(fCurrentDetector.Data(), "DONE");
+ UpdateShuttleLogbook(fCurrentDetector, "DONE");
} else {
Log("SHUTTLE",
Form("ContinueProcessing - %s: Grid storage failed again",
Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^*",
GetCurrentRun()));
- // create ML instance that monitors this run
- fMonaLisa = new TMonaLisaWriter(fConfig->GetMonitorHost(), fConfig->GetMonitorTable(), Form("%d", GetCurrentRun()));
-
// Send the information to ML
TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
mlList.Add(&mlStatus);
mlList.Add(&mlRunType);
- fMonaLisa->SendParameters(&mlList);
+ TString mlID;
+ mlID.Form("%d", GetCurrentRun());
+ fMonaLisa->SendParameters(&mlList, mlID);
if (fLogbookEntry->IsDone())
{
TString lhcPeriod(GetLHCPeriod());
if (lhcPeriod.Length() == 0)
{
- Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
- return 0;
+ Log("SHUTTLE","Process - LHCPeriod not found in logbook!");
+ return 0;
}
if (fgkMainCDB.Length() == 0)
fgkMainRefStorage = Form("alien://folder=/alice/data/%d/%s/Reference?user=alidaq?cacheFold=/tmp/OCDBCache",
GetCurrentYear(), lhcPeriod.Data());
- AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
- if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
- AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
- if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
-
// Loop on detectors in the configuration
TIter iter(fConfig->GetDetectors());
TObjString* aDetector = 0;
+ Bool_t first = kTRUE;
+
while ((aDetector = (TObjString*) iter.Next()))
{
fCurrentDetector = aDetector->String();
if (ContinueProcessing() == kFALSE) continue;
+
+ if (first)
+ {
+ // only read QueryCDB when needed and only once
+ AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
+ if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
+ AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
+ if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
+ first = kFALSE;
+ }
Log("SHUTTLE", Form("\t\t\t****** run %d - %s: START ******",
GetCurrentRun(), aDetector->GetName()));
}
if (expiredTime % 60 == 0)
+ {
Log("SHUTTLE", Form("Process - %s: Checking process. "
"Run time: %d seconds - Memory consumption: %d KB",
fCurrentDetector.Data(), expiredTime, mem));
+ SendAlive();
+ }
if (mem > fConfig->GetPPMaxMem())
{
if (success) // Preprocessor finished successfully!
{
- // remove temporary folder
- gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
+ // remove temporary folder or DCS map
+ if (!fConfig->KeepTempFolder())
+ {
+ gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
+ } else if (!fConfig->KeepDCSMap())
+ {
+ gSystem->Exec(Form("rm -f %s/DCSMap.root",tmpDir.Data()));
+ }
// Update time_processed field in FXS DB
if (UpdateTable() == kFALSE)
TObjArray checkEntryArray;
checkEntryArray.SetOwner(1);
TString whereClause = Form("where run=%d", GetCurrentRun());
- if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) || checkEntryArray.GetEntries() == 0) {
+ if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) ||
+ checkEntryArray.GetEntries() == 0) {
Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
GetCurrentRun()));
return hasError == kFALSE;
}
}
- // remove ML instance
- delete fMonaLisa;
- fMonaLisa = 0;
-
fLogbookEntry = 0;
return hasError == kFALSE;
UpdateShuttleStatus(AliShuttleStatus::kDCSError);
if (!SendMailToDCS())
- Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
+ Log("SHUTTLE", Form("ProcessCurrentDetector - "
+ "Could not send mail to DCS experts!"));
delete dcsMap;
return kFALSE;
UpdateShuttleStatus(AliShuttleStatus::kDCSError);
if (!SendMailToDCS())
- Log("SHUTTLE", Form("ProcessCurrentDetector - Could not send mail to DCS experts!"));
+ Log("SHUTTLE", Form("ProcessCurrentDetector - "
+ "Could not send mail to DCS experts!"));
if (aliasMap) delete aliasMap;
delete dcsMap;
return kTRUE;
}
+//______________________________________________________________________________________________
+void AliShuttle::CountOpenRuns()
+{
+ // Query DAQ's Shuttle logbook and sends the number of open runs to ML
+
+ // check connection, in case connect
+ if (!Connect(3))
+ return;
+
+ TString sqlQuery;
+ sqlQuery = Form("select count(*) from %s where shuttle_done=0", fConfig->GetShuttlelbTable());
+
+ TSQLResult* aResult = fServer[3]->Query(sqlQuery);
+ if (!aResult) {
+ AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
+ return;
+ }
+
+ AliDebug(2,Form("Query = %s", sqlQuery.Data()));
+
+ if (aResult->GetRowCount() == 0) {
+ AliError(Form("No result for query %s received", sqlQuery.Data()));
+ return;
+ }
+
+ if (aResult->GetFieldCount() != 1) {
+ AliError(Form("Invalid field count for query %s received", sqlQuery.Data()));
+ return;
+ }
+
+ TSQLRow* aRow = aResult->Next();
+ if (!aRow) {
+ AliError(Form("Could not receive result of query %s", sqlQuery.Data()));
+ return;
+ }
+
+ TString result(aRow->GetField(0), aRow->GetFieldLength(0));
+ Int_t count = result.Atoi();
+
+ Log("SHUTTLE", Form("%d unprocessed runs", count));
+
+ delete aRow;
+ delete aResult;
+
+ TMonaLisaValue mlStatus("SHUTTLE_openruns", count);
+
+ TList mlList;
+ mlList.Add(&mlStatus);
+
+ fMonaLisa->SendParameters(&mlList, "__PROCESSINGINFO__");
+}
+
//______________________________________________________________________________________________
Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
TObjArray& entries)
entries.SetOwner(1);
// check connection, in case connect
- if(!Connect(3)) return kFALSE;
+ if (!Connect(3)) return kFALSE;
TString sqlQuery;
sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
UInt_t startTime = entry->GetStartTime();
UInt_t endTime = entry->GetEndTime();
- if (!startTime || !endTime || startTime > endTime) {
+// if (!startTime || !endTime || startTime > endTime)
+// {
+// Log("SHUTTLE",
+// Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d. Skipping!",
+// run, startTime, endTime));
+//
+// Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
+// fLogbookEntry = entry;
+// if (!UpdateShuttleLogbook("shuttle_done"))
+// {
+// AliError(Form("Could not update logbook for run %d !", run));
+// }
+// fLogbookEntry = 0;
+//
+// delete entry;
+// delete aRow;
+// delete aResult;
+// return 0;
+// }
+
+ if (!startTime)
+ {
+ Log("SHUTTLE",
+ Form("QueryRunParameters - Invalid parameters for Run %d: "
+ "startTime = %d, endTime = %d. Skipping!",
+ run, startTime, endTime));
+
+ Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
+ fLogbookEntry = entry;
+ if (!UpdateShuttleLogbook("shuttle_ignored"))
+ {
+ AliError(Form("Could not update logbook for run %d !", run));
+ }
+ fLogbookEntry = 0;
+
+ delete entry;
+ delete aRow;
+ delete aResult;
+ return 0;
+ }
+
+ if (startTime && !endTime)
+ {
+ // TODO Here we don't mark SHUTTLE done, because this may mean
+ //the run is still ongoing!!
Log("SHUTTLE",
- Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d",
- run, startTime, endTime));
+ Form("QueryRunParameters - Invalid parameters for Run %d: "
+ "startTime = %d, endTime = %d. Skipping (Shuttle won't be marked as DONE)!",
+ run, startTime, endTime));
+
+ //Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
+ //fLogbookEntry = entry;
+ //if (!UpdateShuttleLogbook("shuttle_done"))
+ //{
+ // AliError(Form("Could not update logbook for run %d !", run));
+ //}
+ //fLogbookEntry = 0;
+
+ delete entry;
+ delete aRow;
+ delete aResult;
+ return 0;
+ }
+
+ if (startTime && endTime && (startTime > endTime))
+ {
+ Log("SHUTTLE",
+ Form("QueryRunParameters - Invalid parameters for Run %d: "
+ "startTime = %d, endTime = %d. Skipping!",
+ run, startTime, endTime));
+
+ Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
+ fLogbookEntry = entry;
+ if (!UpdateShuttleLogbook("shuttle_ignored"))
+ {
+ AliError(Form("Could not update logbook for run %d !", run));
+ }
+ fLogbookEntry = 0;
+
+ delete entry;
+ delete aRow;
+ delete aResult;
+ return 0;
+ }
+
+ TString totEventsStr = entry->GetRunParameter("totalEvents");
+ Int_t totEvents = totEventsStr.Atoi();
+ if (totEvents < 1)
+ {
+ Log("SHUTTLE",
+ Form("QueryRunParameters - Run %d has 0 events - Skipping!", run));
+
+ Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
+ fLogbookEntry = entry;
+ if (!UpdateShuttleLogbook("shuttle_ignored"))
+ {
+ AliError(Form("Could not update logbook for run %d !", run));
+ }
+ fLogbookEntry = 0;
+
delete entry;
delete aRow;
delete aResult;
continue;
}
+ if (fileSize.Length()>0)
+ {
+ // compare filesize of local file with the one stored in the FXS DB
+ TString command=("stat --format=%s");
+ Int_t sizeComp = gSystem->Exec(Form("%s %s |grep %s 2>&1 > /dev/null",
+ command.Data(), localFileName.Data(),fileSize.Data()));
+
+ if ( sizeComp != 0)
+ {
+ Log(detector, Form("GetFileName - size of file %s does not match with local copy!",
+ filePath.Data()));
+ result = kFALSE;
+ continue;
+ }
+
+ } else {
+ Log(fCurrentDetector, Form("GetFile - size of file %s not set in %s database, skipping comparison",
+ filePath.Data(), GetSystemName(system)));
+ }
+
if (fileChecksum.Length()>0)
{
// compare md5sum of local file with the one stored in the FXS DB
// if id is NULL all sources are returned (distinct)
//
- Log(detector, Form("GetFileSources - Retrieving sources with id %s from %s", id, GetSystemName(system)));
+ if (id)
+ {
+ Log(detector, Form("GetFileSources - Querying %s FXS for files with id %s produced by %s", GetSystemName(system), id, detector));
+ } else {
+ Log(detector, Form("GetFileSources - Querying %s FXS for files produced by %s", GetSystemName(system), detector));
+ }
// check if test mode should simulate a FXS error
if (fTestMode & kErrorFXSSources)
TString detName(detector);
TString setClause;
- if(detName == "shuttle_done")
+ if (detName == "shuttle_done" || detName == "shuttle_ignored")
{
setClause = "set shuttle_done=1";
- // Send the information to ML
- TMonaLisaText mlStatus("SHUTTLE_status", "Done");
-
- TList mlList;
- mlList.Add(&mlStatus);
+ if (detName == "shuttle_done")
+ {
+ // Send the information to ML
+ TMonaLisaText mlStatus("SHUTTLE_status", "Done");
- fMonaLisa->SendParameters(&mlList);
+ TList mlList;
+ mlList.Add(&mlStatus);
+
+ TString mlID;
+ mlID.Form("%d", GetCurrentRun());
+ fMonaLisa->SendParameters(&mlList, mlID);
+ }
} else {
TString statusStr(status);
if(statusStr.Contains("done", TString::kIgnoreCase) ||
// Fill log string with a message
//
- void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
+ TString logRunDir = GetShuttleLogDir();
+ if (GetCurrentRun() >=0)
+ logRunDir += Form("/%d", GetCurrentRun());
+
+ void* dir = gSystem->OpenDirectory(logRunDir.Data());
if (dir == NULL) {
- if (gSystem->mkdir(GetShuttleLogDir(), kTRUE)) {
+ if (gSystem->mkdir(logRunDir.Data(), kTRUE)) {
AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
return;
}
TString fileName;
if (GetCurrentRun() >= 0)
- fileName.Form("%s/%s_%d.log", GetShuttleLogDir(), detector, GetCurrentRun());
- else
+ {
+ fileName.Form("%s/%d/%s_%d.log", GetShuttleLogDir(), GetCurrentRun(),
+ detector, GetCurrentRun());
+ } else {
fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
+ }
return fileName;
}
+//______________________________________________________________________________________________
+void AliShuttle::SendAlive()
+{
+ // sends alive message to ML
+
+ TMonaLisaText mlStatus("SHUTTLE_status", "Alive");
+
+ TList mlList;
+ mlList.Add(&mlStatus);
+
+ fMonaLisa->SendParameters(&mlList, "__PROCESSINGINFO__");
+}
+
//______________________________________________________________________________________________
Bool_t AliShuttle::Collect(Int_t run)
{
SetLastAction("Starting");
+ // create ML instance
+ if (!fMonaLisa)
+ fMonaLisa = new TMonaLisaWriter(fConfig->GetMonitorHost(), fConfig->GetMonitorTable());
+
+ SendAlive();
+ CountOpenRuns();
+
TString whereClause("where shuttle_done=0");
if (run != -1)
whereClause += Form(" and run=%d", run);
if (!RetrieveConditionsData(shuttleLogbookEntries))
{
Log("SHUTTLE", "Collect - Process of at least one run failed");
+ CountOpenRuns();
return kFALSE;
}
Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
+ CountOpenRuns();
return kTRUE;
}
if (fTestMode != kNone)
return kTRUE;
+
+ if (!fConfig->SendMail()) return kTRUE;
+
+ TString to="";
+ TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
+ TObjString *anExpert=0;
+ while ((anExpert = (TObjString*) iterExperts.Next()))
+ {
+ to += Form("%s,", anExpert->GetName());
+ }
+ if (to.Length() > 0)
+ to.Remove(to.Length()-1);
+ AliDebug(2, Form("to: %s",to.Data()));
+
+ if (to.IsNull()) {
+ Log("SHUTTLE", "List of detector responsibles not set!");
+ return kFALSE;
+ }
void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
if (dir == NULL)
return kFALSE;
}
- TString to="";
- TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
- TObjString *anExpert=0;
- while ((anExpert = (TObjString*) iterExperts.Next()))
+ TString cc="";
+ TIter iterAdmins(fConfig->GetAdmins(AliShuttleConfig::kGlobal));
+ TObjString *anAdmin=0;
+ while ((anAdmin = (TObjString*) iterAdmins.Next()))
{
- to += Form("%s,", anExpert->GetName());
+ cc += Form("%s,", anAdmin->GetName());
}
- to.Remove(to.Length()-1);
- AliDebug(2, Form("to: %s",to.Data()));
+ if (cc.Length() > 0)
+ cc.Remove(cc.Length()-1);
+ AliDebug(2, Form("cc: %s",to.Data()));
- if (to.IsNull()) {
- Log("SHUTTLE", "List of detector responsibles not yet set!");
- return kFALSE;
- }
-
- TString cc="alberto.colla@cern.ch";
-
- TString subject = Form("%s Shuttle preprocessor FAILED in run %d !",
- fCurrentDetector.Data(), GetCurrentRun());
+ TString subject = Form("%s Shuttle preprocessor FAILED in run %d (run type = %s)!",
+ fCurrentDetector.Data(), GetCurrentRun(), GetRunType());
AliDebug(2, Form("subject: %s", subject.Data()));
TString body = Form("Dear %s expert(s), \n\n", fCurrentDetector.Data());
body += Form("SHUTTLE just detected that your preprocessor "
- "failed processing run %d!!\n\n", GetCurrentRun());
- body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n", fCurrentDetector.Data());
- body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
+ "failed processing run %d (run type = %s)!!\n\n",
+ GetCurrentRun(), GetRunType());
+ body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n",
+ fCurrentDetector.Data());
+ if (fConfig->GetRunMode() == AliShuttleConfig::kTest)
+ {
+ body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
+ } else {
+ body += Form("\thttp://pcalimonitor.cern.ch/shuttle.jsp?instance=PROD&time=168 \n\n");
+ }
+
+
+ TString logFolder = "logs";
+ if (fConfig->GetRunMode() == AliShuttleConfig::kProd)
+ logFolder += "_PROD";
+
+
body += Form("Find the %s log for the current run on \n\n"
- "\thttp://pcalishuttle01.cern.ch:8880/logs/%s_%d.log \n\n",
- fCurrentDetector.Data(), fCurrentDetector.Data(), GetCurrentRun());
- body += Form("The last 10 lines of %s log file are following:\n\n");
+ "\thttp://pcalishuttle01.cern.ch:8880/%s/%d/%s_%d.log \n\n",
+ fCurrentDetector.Data(), logFolder.Data(), GetCurrentRun(),
+ fCurrentDetector.Data(), GetCurrentRun());
+ body += Form("The last 10 lines of %s log file are following:\n\n", fCurrentDetector.Data());
AliDebug(2, Form("Body begin: %s", body.Data()));
mailBody.close();
mailBody.open(bodyFileName, ofstream::out | ofstream::app);
- TString logFileName = Form("%s/%s_%d.log", GetShuttleLogDir(), fCurrentDetector.Data(), GetCurrentRun());
+ TString logFileName = Form("%s/%d/%s_%d.log", GetShuttleLogDir(),
+ GetCurrentRun(), fCurrentDetector.Data(), GetCurrentRun());
TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
if (gSystem->Exec(tailCommand.Data()))
{
Bool_t AliShuttle::SendMailToDCS()
{
//
- // sends a mail to the DCS experts in case of DCS error
+ // sends a mail to the DCS Amanda experts in case of DCS data point retrieval error
//
if (fTestMode != kNone)
return kTRUE;
+ if (!fConfig->SendMail()) return kTRUE;
+
void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
if (dir == NULL)
{
return kFALSE;
}
- TString to="Vladimir.Fekete@cern.ch, Svetozar.Kapusta@cern.ch";
- //TString to="alberto.colla@cern.ch";
+ TString to="";
+ TIter iterExperts(fConfig->GetAdmins(AliShuttleConfig::kAmanda));
+ TObjString *anExpert=0;
+ while ((anExpert = (TObjString*) iterExperts.Next()))
+ {
+ to += Form("%s,", anExpert->GetName());
+ }
+ if (to.Length() > 0)
+ to.Remove(to.Length()-1);
AliDebug(2, Form("to: %s",to.Data()));
if (to.IsNull()) {
- Log("SHUTTLE", "List of detector responsibles not yet set!");
+ Log("SHUTTLE", "List of Amanda server administrators not set!");
return kFALSE;
}
- TString cc="alberto.colla@cern.ch";
+ TString cc="";
+ TIter iterAdmins(fConfig->GetAdmins(AliShuttleConfig::kGlobal));
+ TObjString *anAdmin=0;
+ while ((anAdmin = (TObjString*) iterAdmins.Next()))
+ {
+ cc += Form("%s,", anAdmin->GetName());
+ }
+ if (cc.Length() > 0)
+ cc.Remove(cc.Length()-1);
+ AliDebug(2, Form("cc: %s",to.Data()));
TString subject = Form("Retrieval of data points for %s FAILED in run %d !",
fCurrentDetector.Data(), GetCurrentRun());
TString body = Form("Dear DCS experts, \n\n");
body += Form("SHUTTLE couldn\'t retrieve the data points for detector %s "
"in run %d!!\n\n", fCurrentDetector.Data(), GetCurrentRun());
- body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n", fCurrentDetector.Data());
- body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
+ body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n",
+ fCurrentDetector.Data());
+ if (fConfig->GetRunMode() == AliShuttleConfig::kTest)
+ {
+ body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
+ } else {
+ body += Form("\thttp://pcalimonitor.cern.ch/shuttle.jsp?instance=PROD?time=168 \n\n");
+ }
+
+ TString logFolder = "logs";
+ if (fConfig->GetRunMode() == AliShuttleConfig::kProd)
+ logFolder += "_PROD";
+
+
body += Form("Find the %s log for the current run on \n\n"
- "\thttp://pcalishuttle01.cern.ch:8880/logs/%s_%d.log \n\n",
- fCurrentDetector.Data(), fCurrentDetector.Data(), GetCurrentRun());
- body += Form("The last 10 lines of %s log file are following:\n\n");
+ "\thttp://pcalishuttle01.cern.ch:8880/%s/%d/%s_%d.log \n\n",
+ fCurrentDetector.Data(), logFolder.Data(), GetCurrentRun(),
+ fCurrentDetector.Data(), GetCurrentRun());
+ body += Form("The last 10 lines of %s log file are following:\n\n", fCurrentDetector.Data());
AliDebug(2, Form("Body begin: %s", body.Data()));
mailBody.close();
mailBody.open(bodyFileName, ofstream::out | ofstream::app);
- TString logFileName = Form("%s/%s_%d.log", GetShuttleLogDir(), fCurrentDetector.Data(), GetCurrentRun());
+ TString logFileName = Form("%s/%d/%s_%d.log", GetShuttleLogDir(), GetCurrentRun(),
+ fCurrentDetector.Data(), GetCurrentRun());
TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
if (gSystem->Exec(tailCommand.Data()))
{
git://git.uio.no / u / mrichter / AliRoot.git / blobdiff