* provided "as is" without express or implied warranty. *
**************************************************************************/
-/*
-$Log$
-Revision 1.41 2007/05/03 08:00:48 jgrosseo
-fixing log message when pp want to skip dcs value retrieval
-
-Revision 1.40 2007/04/27 07:06:48 jgrosseo
-GetFileSources returns empty list in case of no files, but successful query
-No mails sent in testmode
-
-Revision 1.39 2007/04/17 12:43:57 acolla
-Correction in StoreOCDB; change of text in mail to detector expert
-
-Revision 1.38 2007/04/12 08:26:18 jgrosseo
-updated comment
-
-Revision 1.37 2007/04/10 16:53:14 jgrosseo
-redirecting sub detector stdout, stderr to sub detector log file
-
-Revision 1.35 2007/04/04 16:26:38 acolla
-1. Re-organization of function calls in TestPreprocessor to make it more meaningful.
-2. Added missing dependency in test preprocessors.
-3. in AliShuttle.cxx: processing time and memory consumption info on a single line.
-
-Revision 1.34 2007/04/04 10:33:36 jgrosseo
-1) Storing of files to the Grid is now done _after_ your preprocessors succeeded. This is transparent, which means that you can still use the same functions (Store, StoreReferenceData) to store files to the Grid. However, the Shuttle first stores them locally and transfers them after the preprocessor finished. The return code of these two functions has changed from UInt_t to Bool_t which gives you the success of the storing.
-In case of an error with the Grid, the Shuttle will retry the storing later, the preprocessor does not need to be run again.
-
-2) The meaning of the return code of the preprocessor has changed. 0 is now success and any other value means failure. This value is stored in the log and you can use it to keep details about the error condition.
-
-3) New function StoreReferenceFile to _directly_ store a file (without opening it) to the reference storage.
-
-4) The memory usage of the preprocessor is monitored. If it exceeds 2 GB it is terminated.
-
-5) New function AliPreprocessor::ProcessDCS(). If you do not need to have DCS data in all cases, you can skip the processing by implemting this function and returning kFALSE under certain conditions. E.g. if there is a certain run type.
-If you always need DCS data (like before), you do not need to implement it.
-
-6) The run type has been added to the monitoring page
-
-Revision 1.33 2007/04/03 13:56:01 acolla
-Grid Storage at the end of preprocessing. Added virtual method to disable DCS query according to the
-run type.
-
-Revision 1.32 2007/02/28 10:41:56 acolla
-Run type field added in SHUTTLE framework. Run type is read from "run type" logbook and retrieved by
-AliPreprocessor::GetRunType() function.
-Added some ldap definition files.
-
-Revision 1.30 2007/02/13 11:23:21 acolla
-Moved getters and setters of Shuttle's main OCDB/Reference, local
-OCDB/Reference, temp and log folders to AliShuttleInterface
-
-Revision 1.27 2007/01/30 17:52:42 jgrosseo
-adding monalisa monitoring
-
-Revision 1.26 2007/01/23 19:20:03 acolla
-Removed old ldif files, added TOF, MCH ldif files. Added some options in
-AliShuttleConfig::Print. Added in Ali Shuttle: SetShuttleTempDir and
-SetShuttleLogDir
-
-Revision 1.25 2007/01/15 19:13:52 acolla
-Moved some AliInfo to AliDebug in SendMail function
-
-Revision 1.21 2006/12/07 08:51:26 jgrosseo
-update (alberto):
-table, db names in ldap configuration
-added GRP preprocessor
-DCS data can also be retrieved by data point
-
-Revision 1.20 2006/11/16 16:16:48 jgrosseo
-introducing strict run ordering flag
-removed giving preprocessor name to preprocessor, they have to know their name themselves ;-)
-
-Revision 1.19 2006/11/06 14:23:04 jgrosseo
-major update (Alberto)
-o) reading of run parameters from the logbook
-o) online offline naming conversion
-o) standalone DCSclient package
-
-Revision 1.18 2006/10/20 15:22:59 jgrosseo
-o) Adding time out to the execution of the preprocessors: The Shuttle forks and the parent process monitors the child
-o) Merging Collect, CollectAll, CollectNew function
-o) Removing implementation of empty copy constructors (declaration still there!)
-
-Revision 1.17 2006/10/05 16:20:55 jgrosseo
-adapting to new CDB classes
-
-Revision 1.16 2006/10/05 15:46:26 jgrosseo
-applying to the new interface
-
-Revision 1.15 2006/10/02 16:38:39 jgrosseo
-update (alberto):
-fixed memory leaks
-storing of objects that failed to be stored to the grid before
-interfacing of shuttle status table in daq system
-
-Revision 1.14 2006/08/29 09:16:05 jgrosseo
-small update
-
-Revision 1.13 2006/08/15 10:50:00 jgrosseo
-effc++ corrections (alberto)
-
-Revision 1.12 2006/08/08 14:19:29 jgrosseo
-Update to shuttle classes (Alberto)
-
-- Possibility to set the full object's path in the Preprocessor's and
-Shuttle's Store functions
-- Possibility to extend the object's run validity in the same classes
-("startValidity" and "validityInfinite" parameters)
-- Implementation of the StoreReferenceData function to store reference
-data in a dedicated CDB storage.
-
-Revision 1.11 2006/07/21 07:37:20 jgrosseo
-last run is stored after each run
-
-Revision 1.10 2006/07/20 09:54:40 jgrosseo
-introducing status management: The processing per subdetector is divided into several steps,
-after each step the status is stored on disk. If the system crashes in any of the steps the Shuttle
-can keep track of the number of failures and skips further processing after a certain threshold is
-exceeded. These thresholds can be configured in LDAP.
-
-Revision 1.9 2006/07/19 10:09:55 jgrosseo
-new configuration, accesst to DAQ FES (Alberto)
-
-Revision 1.8 2006/07/11 12:44:36 jgrosseo
-adding parameters for extended validity range of data produced by preprocessor
-
-Revision 1.7 2006/07/10 14:37:09 jgrosseo
-small fix + todo comment
-
-Revision 1.6 2006/07/10 13:01:41 jgrosseo
-enhanced storing of last sucessfully processed run (alberto)
-
-Revision 1.5 2006/07/04 14:59:57 jgrosseo
-revision of AliDCSValue: Removed wrapper classes, reduced storage size per value by factor 2
-
-Revision 1.4 2006/06/12 09:11:16 jgrosseo
-coding conventions (Alberto)
-
-Revision 1.3 2006/06/06 14:26:40 jgrosseo
-o) removed files that were moved to STEER
-o) shuttle updated to follow the new interface (Alberto)
-
-Revision 1.2 2006/03/07 07:52:34 hristov
-New version (B.Yordanov)
-
-Revision 1.6 2005/11/19 17:19:14 byordano
-RetrieveDATEEntries and RetrieveConditionsData added
-
-Revision 1.5 2005/11/19 11:09:27 byordano
-AliShuttle declaration added
-
-Revision 1.4 2005/11/17 17:47:34 byordano
-TList changed to TObjArray
-
-Revision 1.3 2005/11/17 14:43:23 byordano
-import to local CVS
-
-Revision 1.1.1.1 2005/10/28 07:33:58 hristov
-Initial import as subdirectory in AliRoot
-
-Revision 1.2 2005/09/13 08:41:15 byordano
-default startTime endTime added
-
-Revision 1.4 2005/08/30 09:13:02 byordano
-some docs added
-
-Revision 1.3 2005/08/29 21:15:47 byordano
-some docs added
-
-*/
+/* $Id$ */
//
// This class is the main manager for AliShuttle.
#include <TMutex.h>
#include <TSystemDirectory.h>
#include <TSystemFile.h>
-#include <TFileMerger.h>
+#include <TFile.h>
#include <TGrid.h>
#include <TGridResult.h>
#include <sys/types.h>
#include <sys/wait.h>
+#include <signal.h>
+
ClassImp(AliShuttle)
//______________________________________________________________________________________________
return kFALSE;
}
- AliInfo("Storing OCDB data ...");
+ Log("SHUTTLE","StoreOCDB - Storing OCDB data ...");
Bool_t resultCDB = StoreOCDB(fgkMainCDB);
- AliInfo("Storing reference data ...");
+ Log("SHUTTLE","StoreOCDB - Storing reference data ...");
Bool_t resultRef = StoreOCDB(fgkMainRefStorage);
- AliInfo("Storing reference files ...");
- Bool_t resultRefFiles = StoreRefFilesToGrid();
+ Log("SHUTTLE","StoreOCDB - Storing reference files ...");
+ Bool_t resultRefFiles = CopyFilesToGrid("reference");
- return resultCDB && resultRef && resultRefFiles;
+ Bool_t resultMetadata = kTRUE;
+ if(fCurrentDetector == "GRP")
+ {
+ Log("StoreOCDB - SHUTTLE","Storing Run Metadata file ...");
+ resultMetadata = CopyFilesToGrid("metadata");
+ }
+
+ return resultCDB && resultRef && resultRefFiles && resultMetadata;
}
//______________________________________________________________________________________________
Log("SHUTTLE", Form("StoreOCDB - %s: object %s has validity infinite but "
"there are previous unprocessed runs!",
fCurrentDetector.Data(), aLocId.GetPath().Data()));
+ result = kFALSE;
continue;
}
Log("SHUTTLE",
Form("StoreOCDB - Object <%s> successfully put into %s storage",
aLocId.ToString().Data(), type));
+ Log(fCurrentDetector.Data(),
+ Form("StoreOCDB - Object <%s> successfully put into %s storage",
+ aLocId.ToString().Data(), type));
}
// removing local filename...
TString filename;
localSto->IdToFilename(aLocId, filename);
- AliInfo(Form("Removing local file %s", filename.Data()));
+ Log("SHUTTLE", Form("StoreOCDB - Removing local file %s", filename.Data()));
RemoveFile(filename.Data());
continue;
} else {
Log("SHUTTLE",
Form("StoreOCDB - Grid %s storage of object <%s> failed",
type, aLocId.ToString().Data()));
+ Log(fCurrentDetector.Data(),
+ Form("StoreOCDB - Grid %s storage of object <%s> failed",
+ type, aLocId.ToString().Data()));
result = kFALSE;
}
}
return result;
}
+//______________________________________________________________________________________________
+Bool_t AliShuttle::CleanReferenceStorage(const char* detector)
+{
+ // clears the directory used to store reference files of a given subdetector
+
+ AliCDBManager* man = AliCDBManager::Instance();
+ AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
+ TString localBaseFolder = sto->GetBaseFolder();
+
+ TString targetDir = GetRefFilePrefix(localBaseFolder.Data(), detector);
+
+ Log("SHUTTLE", Form("CleanReferenceStorage - Cleaning %s", targetDir.Data()));
+
+ TString begin;
+ begin.Form("%d_", GetCurrentRun());
+
+ TSystemDirectory* baseDir = new TSystemDirectory("/", targetDir);
+ if (!baseDir)
+ return kTRUE;
+
+ TList* dirList = baseDir->GetListOfFiles();
+ delete baseDir;
+
+ if (!dirList) return kTRUE;
+
+ if (dirList->GetEntries() < 3)
+ {
+ delete dirList;
+ return kTRUE;
+ }
+
+ Int_t nDirs = 0, nDel = 0;
+ TIter dirIter(dirList);
+ TSystemFile* entry = 0;
+
+ Bool_t success = kTRUE;
+
+ while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
+ {
+ if (entry->IsDirectory())
+ continue;
+
+ TString fileName(entry->GetName());
+ if (!fileName.BeginsWith(begin))
+ continue;
+
+ nDirs++;
+
+ // delete file
+ Int_t result = gSystem->Unlink(fileName.Data());
+
+ if (result)
+ {
+ Log("SHUTTLE", Form("CleanReferenceStorage - Could not delete file %s!", fileName.Data()));
+ success = kFALSE;
+ } else {
+ nDel++;
+ }
+ }
+
+ if(nDirs > 0)
+ Log("SHUTTLE", Form("CleanReferenceStorage - %d (over %d) reference files in folder %s were deleted.",
+ nDel, nDirs, targetDir.Data()));
+
+
+ delete dirList;
+ return success;
+
+
+
+
+
+
+ Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
+ if (result == 0)
+ {
+ // delete directory
+ result = gSystem->Exec(Form("rm -rf %s", targetDir.Data()));
+ if (result != 0)
+ {
+ Log("SHUTTLE", Form("CleanReferenceStorage - Could not clean directory %s", targetDir.Data()));
+ return kFALSE;
+ }
+ }
+
+ result = gSystem->mkdir(targetDir, kTRUE);
+ if (result != 0)
+ {
+ Log("SHUTTLE", Form("CleanReferenceStorage - Error creating base directory %s", targetDir.Data()));
+ return kFALSE;
+ }
+
+ return kTRUE;
+}
+
//______________________________________________________________________________________________
Bool_t AliShuttle::StoreReferenceFile(const char* detector, const char* localFile, const char* gridFileName)
{
TString localBaseFolder = sto->GetBaseFolder();
- TString targetDir;
- targetDir.Form("%s/%s", localBaseFolder.Data(), detector);
+ TString target = GetRefFilePrefix(localBaseFolder.Data(), detector);
+ target.Append(Form("/%d_%s", GetCurrentRun(), gridFileName));
+
+ return CopyFileLocally(localFile, target);
+}
+
+//______________________________________________________________________________________________
+Bool_t AliShuttle::StoreRunMetadataFile(const char* localFile, const char* gridFileName)
+{
+ //
+ // Stores Run metadata file to the Grid, in the run folder
+ //
+ // Only GRP can call this function.
+
+ if (fTestMode & kErrorStorage)
+ {
+ Log(fCurrentDetector, "StoreRunMetaDataFile - In TESTMODE - Simulating error while storing locally");
+ return kFALSE;
+ }
+
+ AliCDBManager* man = AliCDBManager::Instance();
+ AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
+
+ TString localBaseFolder = sto->GetBaseFolder();
+
+ // Build Run level folder
+ // folder = /alice/data/year/lhcPeriod/runNb/raw
+
+
+ TString lhcPeriod = GetLHCPeriod();
+ if (lhcPeriod.Length() == 0)
+ {
+ Log("SHUTTLE","StoreRunMetaDataFile - LHCPeriod not found in logbook!");
+ return 0;
+ }
+
+ // TODO partitions with one detector only write data into LHCperiod_DET
+ TString partition = GetRunParameter("detector");
+
+ if (partition.Length() > 0 && partition != "ALICE")
+ {
+ lhcPeriod.Append(Form("_%s", partition.Data()));
+ Log(fCurrentDetector, Form("Run data tags merged file will be written in %s",
+ lhcPeriod.Data()));
+ }
+
+ TString target = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/raw/%s",
+ localBaseFolder.Data(), GetCurrentYear(),
+ lhcPeriod.Data(), GetCurrentRun(), gridFileName);
+
+ return CopyFileLocally(localFile, target);
+}
+
+//______________________________________________________________________________________________
+Bool_t AliShuttle::CopyFileLocally(const char* localFile, const TString& target)
+{
+ //
+ // Stores file locally. Called by StoreReferenceFile and StoreRunMetadataFile
+ // Files are temporarily stored in the local reference storage. When the preprocessor
+ // finishes, the Shuttle calls CopyFilesToGrid to transfer the files to AliEn
+ // (in reference or run level folders)
+ //
- TString target;
- target.Form("%s/%d_%s", targetDir.Data(), GetCurrentRun(), gridFileName);
+ TString targetDir(target(0, target.Last('/')));
- Int_t result = gSystem->GetPathInfo(targetDir, 0, (Long64_t*) 0, 0, 0);
+ //try to open base dir folder, if it does not exist
+ void* dir = gSystem->OpenDirectory(targetDir.Data());
+ if (dir == NULL) {
+ if (gSystem->mkdir(targetDir.Data(), kTRUE)) {
+ Log("SHUTTLE", Form("CopyFileLocally - Can't open directory <%s>", targetDir.Data()));
+ return kFALSE;
+ }
+
+ } else {
+ gSystem->FreeDirectory(dir);
+ }
+
+ Int_t result = 0;
+
+ result = gSystem->GetPathInfo(localFile, 0, (Long64_t*) 0, 0, 0);
if (result)
{
- result = gSystem->mkdir(targetDir, kTRUE);
- if (result != 0)
+ Log("SHUTTLE", Form("CopyFileLocally - %s does not exist", localFile));
+ return kFALSE;
+ }
+
+ result = gSystem->GetPathInfo(target, 0, (Long64_t*) 0, 0, 0);
+ if (!result)
+ {
+ Log("SHUTTLE", Form("CopyFileLocally - target file %s already exist, removing...", target.Data()));
+ if (gSystem->Unlink(target.Data()))
{
- Log("SHUTTLE", Form("StoreReferenceFile - Error creating base directory %s", targetDir.Data()));
+ Log("SHUTTLE", Form("CopyFileLocally - Could not remove existing target file %s!", target.Data()));
return kFALSE;
}
- }
-
+ }
+
result = gSystem->CopyFile(localFile, target);
if (result == 0)
{
- Log("SHUTTLE", Form("StoreReferenceFile - Stored file %s locally to %s", localFile, target.Data()));
+ Log("SHUTTLE", Form("CopyFileLocally - File %s stored locally to %s", localFile, target.Data()));
return kTRUE;
}
else
{
- Log("SHUTTLE", Form("StoreReferenceFile - Storing file %s locally to %s failed", localFile, target.Data()));
+ Log("SHUTTLE", Form("CopyFileLocally - Could not store file %s to %s! Error code = %d",
+ localFile, target.Data(), result));
return kFALSE;
}
+
+
+
}
//______________________________________________________________________________________________
-Bool_t AliShuttle::StoreRefFilesToGrid()
+Bool_t AliShuttle::CopyFilesToGrid(const char* type)
{
//
- // Transfers the reference file to the Grid.
+ // Transfers local files to the Grid. Local files can be reference files
+ // or run metadata file (from GRP only).
//
- // The files are stored under the following location:
- // <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
+ // According to the type (ref, metadata) the files are stored under the following location:
+ // ref --> <base folder of reference storage>/<DET>/<RUN#>_<gridFileName>
+ // metadata --> <run data folder>/<MetadataFileName>
//
-
+
AliCDBManager* man = AliCDBManager::Instance();
AliCDBStorage* sto = man->GetStorage(fgkLocalRefStorage);
if (!sto)
return kFALSE;
TString localBaseFolder = sto->GetBaseFolder();
-
- TString dir;
- dir.Form("%s/%s", localBaseFolder.Data(), GetOfflineDetName(fCurrentDetector));
- AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
- if (!gridSto)
- return kFALSE;
- TString gridBaseFolder = gridSto->GetBaseFolder();
+ TString dir;
TString alienDir;
- alienDir.Form("%s%s", gridBaseFolder.Data(), GetOfflineDetName(fCurrentDetector));
+ TString begin;
- if (!gGrid)
- return kFALSE;
+ if (strcmp(type, "reference") == 0)
+ {
+ dir = GetRefFilePrefix(localBaseFolder.Data(), fCurrentDetector.Data());
+ AliCDBStorage* gridSto = man->GetStorage(fgkMainRefStorage);
+ if (!gridSto)
+ return kFALSE;
+ TString gridBaseFolder = gridSto->GetBaseFolder();
+ alienDir = GetRefFilePrefix(gridBaseFolder.Data(), fCurrentDetector.Data());
+ begin = Form("%d_", GetCurrentRun());
+ }
+ else if (strcmp(type, "metadata") == 0)
+ {
+
+ TString lhcPeriod = GetLHCPeriod();
- TString begin;
- begin.Form("%d_", GetCurrentRun());
+ if (lhcPeriod.Length() == 0)
+ {
+ Log("SHUTTLE","CopyFilesToGrid - LHCPeriod not found in logbook!");
+ return 0;
+ }
+
+ // TODO partitions with one detector only write data into LHCperiod_DET
+ TString partition = GetRunParameter("detector");
+ if (partition.Length() > 0 && partition != "ALICE")
+ {
+ lhcPeriod.Append(Form("_%s", partition.Data()));
+ }
+
+ dir = Form("%s/GRP/RunMetadata/alice/data/%d/%s/%09d/raw",
+ localBaseFolder.Data(), GetCurrentYear(),
+ lhcPeriod.Data(), GetCurrentRun());
+ alienDir = dir(dir.Index("/alice/data/"), dir.Length());
+
+ begin = "";
+ }
+ else
+ {
+ Log("SHUTTLE", "CopyFilesToGrid - Unexpected: type label must be reference or metadata!");
+ return kFALSE;
+ }
+
TSystemDirectory* baseDir = new TSystemDirectory("/", dir);
if (!baseDir)
return kTRUE;
- TList* dirList = baseDir->GetListOfFiles();
- if (!dirList)
+ TList* dirList = baseDir->GetListOfFiles();
+ delete baseDir;
+
+ if (!dirList) return kTRUE;
+
+ if (dirList->GetEntries() < 3)
{
- delete baseDir;
+ delete dirList;
return kTRUE;
}
-
- Int_t nDirs = dirList->GetEntries();
+
+ if (!gGrid)
+ {
+ Log("SHUTTLE", "CopyFilesToGrid - Connection to Grid failed: Cannot continue!");
+ delete dirList;
+ return kFALSE;
+ }
+ Int_t nDirs = 0, nTransfer = 0;
+ TIter dirIter(dirList);
+ TSystemFile* entry = 0;
+
Bool_t success = kTRUE;
Bool_t first = kTRUE;
- for (Int_t iDir=0; iDir<nDirs; ++iDir)
- {
- TSystemFile* entry = dynamic_cast<TSystemFile*> (dirList->At(iDir));
- if (!entry)
- continue;
-
+ while ((entry = dynamic_cast<TSystemFile*> (dirIter.Next())))
+ {
if (entry->IsDirectory())
continue;
if (!fileName.BeginsWith(begin))
continue;
+ nDirs++;
+
if (first)
{
first = kFALSE;
- // check that DET folder exists, otherwise create it
+ // check that folder exists, otherwise create it
TGridResult* result = gGrid->Ls(alienDir.Data(), "a");
if (!result)
+ {
+ delete dirList;
return kFALSE;
+ }
- if (!result->GetFileName(0))
+ if (!result->GetFileName(1)) // TODO: It looks like element 0 is always 0!!
{
- if (!gGrid->Mkdir(alienDir.Data(),"",0))
+ // TODO It does not work currently! Bug in TAliEn::Mkdir
+ // TODO Manually fixed in local root v5-16-00
+ if (!gGrid->Mkdir(alienDir.Data(),"-p",0))
{
- Log("SHUTTLE", Form("StoreRefFilesToGrid - Cannot create directory %s",
+ Log("SHUTTLE", Form("CopyFilesToGrid - Cannot create directory %s",
alienDir.Data()));
- delete baseDir;
+ delete dirList;
return kFALSE;
+ } else {
+ Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s created", alienDir.Data()));
}
+ } else {
+ Log("SHUTTLE",Form("CopyFilesToGrid - Folder %s found", alienDir.Data()));
}
}
TString fullGridPath;
fullGridPath.Form("alien://%s/%s", alienDir.Data(), fileName.Data());
- Log("SHUTTLE", Form("StoreRefFilesToGrid - Copying local file %s to %s", fullLocalPath.Data(), fullGridPath.Data()));
-
- TFileMerger fileMerger;
- Bool_t result = fileMerger.Cp(fullLocalPath, fullGridPath);
+ Bool_t result = TFile::Cp(fullLocalPath, fullGridPath);
if (result)
{
- Log("SHUTTLE", Form("StoreRefFilesToGrid - Copying local file %s to %s succeeded", fullLocalPath.Data(), fullGridPath.Data()));
+ Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s succeeded!",
+ fullLocalPath.Data(), fullGridPath.Data()));
RemoveFile(fullLocalPath);
+ nTransfer++;
}
else
{
- Log("SHUTTLE", Form("StoreRefFilesToGrid - Copying local file %s to %s failed", fullLocalPath.Data(), fullGridPath.Data()));
+ Log("SHUTTLE", Form("CopyFilesToGrid - Copying local file %s to %s FAILED!",
+ fullLocalPath.Data(), fullGridPath.Data()));
success = kFALSE;
}
}
+
+ Log("SHUTTLE", Form("CopyFilesToGrid - %d (over %d) files in folder %s copied to Grid.",
+ nTransfer, nDirs, dir.Data()));
+
+
+ delete dirList;
+ return success;
+}
+
+//______________________________________________________________________________________________
+const char* AliShuttle::GetRefFilePrefix(const char* base, const char* detector)
+{
+ //
+ // Get folder name of reference files
+ //
+
+ TString offDetStr(GetOfflineDetName(detector));
+ TString dir;
+ if (offDetStr == "ITS" || offDetStr == "MUON" || offDetStr == "PHOS")
+ {
+ dir.Form("%s/%s/%s", base, offDetStr.Data(), detector);
+ } else {
+ dir.Form("%s/%s", base, offDetStr.Data());
+ }
- delete baseDir;
+ return dir.Data();
- return success;
+
}
//______________________________________________________________________________________________
if(uri == fgkLocalCDB) {
type = "OCDB";
} else if(uri == fgkLocalRefStorage) {
- type = "reference";
+ type = "Reference";
} else {
AliError(Form("Invalid storage URI: %s", uri.Data()));
return;
}
TString filename(Form("%s/%s/*/Run*_v%d_s*.root",
- localSto->GetBaseFolder().Data(), fCurrentDetector.Data(), GetCurrentRun()));
+ localSto->GetBaseFolder().Data(), GetOfflineDetName(fCurrentDetector.Data()), GetCurrentRun()));
- AliInfo(Form("filename = %s", filename.Data()));
+ AliDebug(2, Form("filename = %s", filename.Data()));
- AliInfo(Form("Removing remaining local files from run %d and detector %s ...",
+ Log("SHUTTLE", Form("Removing remaining local files for run %d and detector %s ...",
GetCurrentRun(), fCurrentDetector.Data()));
RemoveFile(filename.Data());
AliShuttleStatus* status = dynamic_cast<AliShuttleStatus*> (fStatusEntry->GetObject());
if (!status){
- Log("SHUTTLE", "UNEXPECTED: status could not be read from current CDB entry");
+ Log("SHUTTLE", "UpdateShuttleStatus - UNEXPECTED: status could not be read from current CDB entry");
return;
}
mlList.Add(&mlStatus);
mlList.Add(&mlRetryCount);
- fMonaLisa->SendParameters(&mlList);
+ TString mlID;
+ mlID.Form("%d", GetCurrentRun());
+ fMonaLisa->SendParameters(&mlList, mlID);
}
//______________________________________________________________________________________________
dynamic_cast<AliPreprocessor*> (fPreprocessorMap.GetValue(fCurrentDetector));
if (!aPreprocessor)
{
- AliInfo(Form("%s: no preprocessor registered", fCurrentDetector.Data()));
+ Log("SHUTTLE", Form("ContinueProcessing - %s: no preprocessor registered", fCurrentDetector.Data()));
return kFALSE;
}
fLogbookEntry->GetDetectorStatus(fCurrentDetector);
if(entryStatus != AliShuttleLogbookEntry::kUnprocessed) {
- AliInfo(Form("ContinueProcessing - %s is %s",
+ Log("SHUTTLE", Form("ContinueProcessing - %s is %s",
fCurrentDetector.Data(),
fLogbookEntry->GetDetectorStatusName(entryStatus)));
return kFALSE;
{
if (fTestMode == kNone)
{
- Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering but this is not the first unprocessed run!"));
+ Log("SHUTTLE", Form("ContinueProcessing - %s requires strict run ordering"
+ " but this is not the first unprocessed run!"));
return kFALSE;
}
else
{
- Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - Although %s requires strict run ordering and this is not the first unprocessed run, the SHUTTLE continues"));
+ Log("SHUTTLE", Form("ContinueProcessing - In TESTMODE - "
+ "Although %s requires strict run ordering "
+ "and this is not the first unprocessed run, "
+ "the SHUTTLE continues"));
}
}
return kFALSE;
}
- if (status->GetStatus() == AliShuttleStatus::kStoreError) {
+ if (status->GetStatus() == AliShuttleStatus::kStoreStarted || status->GetStatus() == AliShuttleStatus::kStoreError) {
Log("SHUTTLE",
- Form("ContinueProcessing - %s: Grid storage of one or more objects failed. Trying again now",
+ Form("ContinueProcessing - %s: Grid storage of one or more "
+ "objects failed. Trying again now",
fCurrentDetector.Data()));
UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
if (StoreOCDB()){
- Log("SHUTTLE", Form("ContinueProcessing - %s: all objects successfully stored into main storage",
+ Log("SHUTTLE", Form("ContinueProcessing - %s: all objects "
+ "successfully stored into main storage",
fCurrentDetector.Data()));
UpdateShuttleStatus(AliShuttleStatus::kDone);
- UpdateShuttleLogbook(fCurrentDetector.Data(), "DONE");
+ UpdateShuttleLogbook(fCurrentDetector, "DONE");
} else {
Log("SHUTTLE",
Form("ContinueProcessing - %s: Grid storage failed again",
// UpdateTableFailCase();
// Send mail to detector expert!
- AliInfo(Form("Sending mail to %s expert...", fCurrentDetector.Data()));
+ Log("SHUTTLE", Form("ContinueProcessing - Sending mail to %s expert...",
+ fCurrentDetector.Data()));
if (!SendMail())
Log("SHUTTLE", Form("ContinueProcessing - Could not send mail to %s expert",
fCurrentDetector.Data()));
"Aborted before with %s. Retry number %d.", fCurrentDetector.Data(),
status->GetStatusName(), status->GetCount()));
Bool_t increaseCount = kTRUE;
- if (status->GetStatus() == AliShuttleStatus::kDCSError || status->GetStatus() == AliShuttleStatus::kDCSStarted)
- increaseCount = kFALSE;
+ if (status->GetStatus() == AliShuttleStatus::kDCSError ||
+ status->GetStatus() == AliShuttleStatus::kDCSStarted)
+ increaseCount = kFALSE;
+
UpdateShuttleStatus(AliShuttleStatus::kStarted, increaseCount);
cont = kTRUE;
}
fLogbookEntry = entry;
- AliInfo(Form("\n\n \t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^* \n",
+ Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: START ^*^*^*^*^*^*^*^*^*^*^*^*",
GetCurrentRun()));
- // create ML instance that monitors this run
- fMonaLisa = new TMonaLisaWriter(Form("%d", GetCurrentRun()), "SHUTTLE", "aliendb1.cern.ch");
- // disable monitoring of other parameters that come e.g. from TFile
- gMonitoringWriter = 0;
-
// Send the information to ML
TMonaLisaText mlStatus("SHUTTLE_status", "Processing");
TMonaLisaText mlRunType("SHUTTLE_runtype", Form("%s (%s)", entry->GetRunType(), entry->GetRunParameter("log")));
mlList.Add(&mlStatus);
mlList.Add(&mlRunType);
- fMonaLisa->SendParameters(&mlList);
+ TString mlID;
+ mlID.Form("%d", GetCurrentRun());
+ fMonaLisa->SendParameters(&mlList, mlID);
if (fLogbookEntry->IsDone())
{
Int_t testMode = tmpStr->String().Atoi();
if (testMode > 0)
{
- Log("SHUTTLE", Form("Enabling test mode %d", testMode));
+ Log("SHUTTLE", Form("Process - Enabling test mode %d", testMode));
SetTestMode((TestMode) testMode);
}
}
}
}
}
-
- Log("SHUTTLE", Form("The test mode flag is %d", (Int_t) fTestMode));
-
+
fLogbookEntry->Print("all");
// Initialization
Bool_t hasError = kFALSE;
- AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
- if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
- AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
- if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
-
+ // Set the CDB and Reference folders according to the year and LHC period
+ TString lhcPeriod(GetLHCPeriod());
+ if (lhcPeriod.Length() == 0)
+ {
+ Log("SHUTTLE","Process - LHCPeriod not found in logbook!");
+ return 0;
+ }
+
+ if (fgkMainCDB.Length() == 0)
+ fgkMainCDB = Form("alien://folder=/alice/data/%d/%s/OCDB?user=alidaq?cacheFold=/tmp/OCDBCache",
+ GetCurrentYear(), lhcPeriod.Data());
+
+ if (fgkMainRefStorage.Length() == 0)
+ fgkMainRefStorage = Form("alien://folder=/alice/data/%d/%s/Reference?user=alidaq?cacheFold=/tmp/OCDBCache",
+ GetCurrentYear(), lhcPeriod.Data());
+
// Loop on detectors in the configuration
TIter iter(fConfig->GetDetectors());
TObjString* aDetector = 0;
+ Bool_t first = kTRUE;
+
while ((aDetector = (TObjString*) iter.Next()))
{
fCurrentDetector = aDetector->String();
if (ContinueProcessing() == kFALSE) continue;
+
+ if (first)
+ {
+ // only read QueryCDB when needed and only once
+ AliCDBStorage *mainCDBSto = AliCDBManager::Instance()->GetStorage(fgkMainCDB);
+ if(mainCDBSto) mainCDBSto->QueryCDB(GetCurrentRun());
+ AliCDBStorage *mainRefSto = AliCDBManager::Instance()->GetStorage(fgkMainRefStorage);
+ if(mainRefSto) mainRefSto->QueryCDB(GetCurrentRun());
+ first = kFALSE;
+ }
- AliInfo(Form("\n\n \t\t\t****** run %d - %s: START ******",
+ Log("SHUTTLE", Form("\t\t\t****** run %d - %s: START ******",
GetCurrentRun(), aDetector->GetName()));
for(Int_t iSys=0;iSys<3;iSys++) fFXSCalled[iSys]=kFALSE;
- Log(fCurrentDetector.Data(), "Starting processing");
+ Log(fCurrentDetector.Data(), "Process - Starting processing");
Int_t pid = fork();
if (pid < 0)
{
- Log("SHUTTLE", "ERROR: Forking failed");
+ Log("SHUTTLE", "Process - ERROR: Forking failed");
}
else if (pid > 0)
{
// parent
- AliInfo(Form("In parent process of %d - %s: Starting monitoring",
+ Log("SHUTTLE", Form("Process - In parent process of %d - %s: Starting monitoring",
GetCurrentRun(), aDetector->GetName()));
Long_t begin = time(0);
if (expiredTime > fConfig->GetPPTimeOut())
{
TString tmp;
- tmp.Form("Process of %s time out. Run time: %d seconds. Killing...",
- fCurrentDetector.Data(), expiredTime);
+ tmp.Form("Process - Process of %s time out. "
+ "Run time: %d seconds. Killing...",
+ fCurrentDetector.Data(), expiredTime);
Log("SHUTTLE", tmp);
Log(fCurrentDetector, tmp);
FILE* pipe = gSystem->OpenPipe(checkStr, "r");
if (!pipe)
{
- Log("SHUTTLE", Form("Error: Could not open pipe to %s", checkStr.Data()));
+ Log("SHUTTLE", Form("Process - Error: "
+ "Could not open pipe to %s", checkStr.Data()));
continue;
}
char buffer[100];
if (!fgets(buffer, 100, pipe))
{
- Log("SHUTTLE", "Error: ps did not return anything");
+ Log("SHUTTLE", "Process - Error: ps did not return anything");
gSystem->ClosePipe(pipe);
continue;
}
Int_t mem = 0;
if ((sscanf(buffer, "%d\n", &mem) != 1) || !mem)
{
- Log("SHUTTLE", "Error: Could not parse output of ps");
+ Log("SHUTTLE", "Process - Error: Could not parse output of ps");
continue;
}
if (expiredTime % 60 == 0)
- Log("SHUTTLE", Form("%s: Checking process. Run time: %d seconds - Memory consumption: %d KB",
- fCurrentDetector.Data(), expiredTime, mem));
+ {
+ Log("SHUTTLE", Form("Process - %s: Checking process. "
+ "Run time: %d seconds - Memory consumption: %d KB",
+ fCurrentDetector.Data(), expiredTime, mem));
+ SendAlive();
+ }
if (mem > fConfig->GetPPMaxMem())
{
TString tmp;
- tmp.Form("Process exceeds maximum allowed memory (%d KB > %d KB). Killing...",
+ tmp.Form("Process - Process exceeds maximum allowed memory "
+ "(%d KB > %d KB). Killing...",
mem, fConfig->GetPPMaxMem());
Log("SHUTTLE", tmp);
Log(fCurrentDetector, tmp);
}
}
- AliInfo(Form("In parent process of %d - %s: Client has terminated.",
+ Log("SHUTTLE", Form("Process - In parent process of %d - %s: Client has terminated.",
GetCurrentRun(), aDetector->GetName()));
if (WIFEXITED(status))
{
Int_t returnCode = WEXITSTATUS(status);
- Log("SHUTTLE", Form("%s: the return code is %d", fCurrentDetector.Data(),
+ Log("SHUTTLE", Form("Process - %s: the return code is %d", fCurrentDetector.Data(),
returnCode));
if (returnCode == 0) hasError = kTRUE;
else if (pid == 0)
{
// client
- AliInfo(Form("In client process of %d - %s", GetCurrentRun(), aDetector->GetName()));
+ Log("SHUTTLE", Form("Process - In client process of %d - %s", GetCurrentRun(),
+ aDetector->GetName()));
- AliInfo("Redirecting output...");
+ Log("SHUTTLE", Form("Process - Redirecting output to %s log",fCurrentDetector.Data()));
- if ((freopen(GetLogFileName(fCurrentDetector), "w", stdout)) == 0)
+ if ((freopen(GetLogFileName(fCurrentDetector), "a", stdout)) == 0)
{
- Log("SHUTTLE", "Could not freopen stdout");
+ Log("SHUTTLE", "Process - Could not freopen stdout");
}
else
{
fOutputRedirected = kTRUE;
if ((dup2(fileno(stdout), fileno(stderr))) < 0)
- Log("SHUTTLE", "Could not redirect stderr");
+ Log("SHUTTLE", "Process - Could not redirect stderr");
}
- Bool_t success = ProcessCurrentDetector();
- if (success) // Preprocessor finished successfully!
- {
- // Update time_processed field in FXS DB
- if (UpdateTable() == kFALSE)
- Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!"));
-
- // Transfer the data from local storage to main storage (Grid)
- UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
+ TString wd = gSystem->WorkingDirectory();
+ TString tmpDir = Form("%s/%s_%d_process", GetShuttleTempDir(),
+ fCurrentDetector.Data(), GetCurrentRun());
+
+ Int_t result = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
+ if (!result) // temp dir already exists!
+ {
+ Log(fCurrentDetector.Data(),
+ Form("Process - %s dir already exists! Removing...", tmpDir.Data()));
+ gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
+ }
+
+ if (gSystem->mkdir(tmpDir.Data(), 1))
+ {
+ Log(fCurrentDetector.Data(), "Process - could not make temp directory!!");
+ gSystem->Exit(1);
+ }
+
+ if (!gSystem->ChangeDirectory(tmpDir.Data()))
+ {
+ Log(fCurrentDetector.Data(), "Process - could not change directory!!");
+ gSystem->Exit(1);
+ }
+
+ Bool_t success = ProcessCurrentDetector();
+
+ gSystem->ChangeDirectory(wd.Data());
+
+ if (success) // Preprocessor finished successfully!
+ {
+ // remove temporary folder or DCS map
+ if (!fConfig->KeepTempFolder())
+ {
+ gSystem->Exec(Form("rm -rf %s",tmpDir.Data()));
+ } else if (!fConfig->KeepDCSMap())
+ {
+ gSystem->Exec(Form("rm -f %s/DCSMap.root",tmpDir.Data()));
+ }
+
+ // Update time_processed field in FXS DB
+ if (UpdateTable() == kFALSE)
+ Log("SHUTTLE", Form("Process - %s: Could not update FXS databases!",
+ fCurrentDetector.Data()));
+
+ // Transfer the data from local storage to main storage (Grid)
+ UpdateShuttleStatus(AliShuttleStatus::kStoreStarted);
if (StoreOCDB() == kFALSE)
{
- AliInfo(Form("\n \t\t\t****** run %d - %s: STORAGE ERROR ****** \n\n",
+ Log("SHUTTLE",
+ Form("\t\t\t****** run %d - %s: STORAGE ERROR ******",
GetCurrentRun(), aDetector->GetName()));
UpdateShuttleStatus(AliShuttleStatus::kStoreError);
success = kFALSE;
} else {
- AliInfo(Form("\n \t\t\t****** run %d - %s: DONE ****** \n\n",
+ Log("SHUTTLE",
+ Form("\t\t\t****** run %d - %s: DONE ******",
GetCurrentRun(), aDetector->GetName()));
UpdateShuttleStatus(AliShuttleStatus::kDone);
UpdateShuttleLogbook(fCurrentDetector, "DONE");
}
+ } else
+ {
+ Log("SHUTTLE",
+ Form("\t\t\t****** run %d - %s: PP ERROR ******",
+ GetCurrentRun(), aDetector->GetName()));
}
for (UInt_t iSys=0; iSys<3; iSys++)
if (fFXSCalled[iSys]) fFXSlist[iSys].Clear();
}
- AliInfo(Form("Client process of %d - %s is exiting now with %d.",
+ Log("SHUTTLE", Form("Process - Client process of %d - %s is exiting now with %d.",
GetCurrentRun(), aDetector->GetName(), success));
// the client exits here
}
}
- AliInfo(Form("\n\n \t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^* \n",
+ Log("SHUTTLE", Form("\t\t\t^*^*^*^*^*^*^*^*^*^*^*^* run %d: FINISH ^*^*^*^*^*^*^*^*^*^*^*^*",
GetCurrentRun()));
//check if shuttle is done for this run, if so update logbook
TObjArray checkEntryArray;
checkEntryArray.SetOwner(1);
TString whereClause = Form("where run=%d", GetCurrentRun());
- if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) || checkEntryArray.GetEntries() == 0) {
+ if (!QueryShuttleLogbook(whereClause.Data(), checkEntryArray) ||
+ checkEntryArray.GetEntries() == 0) {
Log("SHUTTLE", Form("Process - Warning: Cannot check status of run %d on Shuttle logbook!",
GetCurrentRun()));
return hasError == kFALSE;
}
}
- // remove ML instance
- delete fMonaLisa;
- fMonaLisa = 0;
-
fLogbookEntry = 0;
return hasError == kFALSE;
// Makes data retrieval just for a specific detector (fCurrentDetector).
// Threre should be a configuration for this detector.
- AliInfo(Form("Retrieving values for %s, run %d", fCurrentDetector.Data(), GetCurrentRun()));
+ Log("SHUTTLE", Form("ProcessCurrentDetector - Retrieving values for %s, run %d",
+ fCurrentDetector.Data(), GetCurrentRun()));
- TMap dcsMap;
- dcsMap.SetOwner(1);
-
- Bool_t aDCSError = kFALSE;
+ TString wd = gSystem->WorkingDirectory();
+
+ if (!CleanReferenceStorage(fCurrentDetector.Data()))
+ return kFALSE;
+
+ gSystem->ChangeDirectory(wd.Data());
+
+ TMap* dcsMap = new TMap();
// call preprocessor
AliPreprocessor* aPreprocessor =
if (!processDCS)
{
- Log(fCurrentDetector, "The preprocessor requested to skip the retrieval of DCS values");
+ Log(fCurrentDetector, "ProcessCurrentDetector -"
+ " The preprocessor requested to skip the retrieval of DCS values");
}
else if (fTestMode & kSkipDCS)
{
- Log(fCurrentDetector, "In TESTMODE - Skipping DCS processing!");
+ Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Skipping DCS processing");
}
else if (fTestMode & kErrorDCS)
{
- Log(fCurrentDetector, "In TESTMODE - Simulating DCS error");
+ Log(fCurrentDetector, "ProcessCurrentDetector - In TESTMODE: Simulating DCS error");
UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
UpdateShuttleStatus(AliShuttleStatus::kDCSError);
+ delete dcsMap;
return kFALSE;
} else {
UpdateShuttleStatus(AliShuttleStatus::kDCSStarted);
- TString host(fConfig->GetDCSHost(fCurrentDetector));
- Int_t port = fConfig->GetDCSPort(fCurrentDetector);
-
- // Retrieval of Aliases
- TObjString* anAlias = 0;
- Int_t iAlias = 1;
- Int_t nTotAliases= ((TMap*)fConfig->GetDCSAliases(fCurrentDetector))->GetEntries();
- TIter iterAliases(fConfig->GetDCSAliases(fCurrentDetector));
- while ((anAlias = (TObjString*) iterAliases.Next()))
+ // Query DCS archive
+ Int_t nServers = fConfig->GetNServers(fCurrentDetector);
+
+ for (int iServ=0; iServ<nServers; iServ++)
{
- TObjArray *valueSet = new TObjArray();
- valueSet->SetOwner(1);
-
- if (((iAlias-1) % 500) == 0 || iAlias == nTotAliases)
- AliInfo(Form("Querying DCS archive: alias %s (%d of %d)",
- anAlias->GetName(), iAlias++, nTotAliases));
- aDCSError = (GetValueSet(host, port, anAlias->String(), valueSet, kAlias) == 0);
+
+ TString host(fConfig->GetDCSHost(fCurrentDetector, iServ));
+ Int_t port = fConfig->GetDCSPort(fCurrentDetector, iServ);
+ Int_t multiSplit = fConfig->GetMultiSplit(fCurrentDetector, iServ);
- if(!aDCSError)
+ Log(fCurrentDetector, Form("ProcessCurrentDetector -"
+ " Querying DCS Amanda server %s:%d (%d of %d)",
+ host.Data(), port, iServ+1, nServers));
+
+ TMap* aliasMap = 0;
+ TMap* dpMap = 0;
+
+ if (fConfig->GetDCSAliases(fCurrentDetector, iServ)->GetEntries() > 0)
{
- dcsMap.Add(anAlias->Clone(), valueSet);
- } else {
- Log(fCurrentDetector,
- Form("ProcessCurrentDetector - Error while retrieving alias %s",
- anAlias->GetName()));
- UpdateShuttleStatus(AliShuttleStatus::kDCSError);
- dcsMap.DeleteAll();
- return kFALSE;
- }
- }
+ aliasMap = GetValueSet(host, port,
+ fConfig->GetDCSAliases(fCurrentDetector, iServ),
+ kAlias, multiSplit);
+ if (!aliasMap)
+ {
+ Log(fCurrentDetector,
+ Form("ProcessCurrentDetector -"
+ " Error retrieving DCS aliases from server %s."
+ " Sending mail to DCS experts!", host.Data()));
+ UpdateShuttleStatus(AliShuttleStatus::kDCSError);
+
+ if (!SendMailToDCS())
+ Log("SHUTTLE", Form("ProcessCurrentDetector - "
+ "Could not send mail to DCS experts!"));
- // Retrieval of Data Points
- TObjString* aDP = 0;
- Int_t iDP = 0;
- Int_t nTotDPs= ((TMap*)fConfig->GetDCSDataPoints(fCurrentDetector))->GetEntries();
- TIter iterDP(fConfig->GetDCSDataPoints(fCurrentDetector));
- while ((aDP = (TObjString*) iterDP.Next()))
- {
- TObjArray *valueSet = new TObjArray();
- valueSet->SetOwner(1);
- if (((iDP-1) % 500) == 0 || iDP == nTotDPs)
- AliInfo(Form("Querying DCS archive: DP %s (%d of %d)",
- aDP->GetName(), iDP++, nTotDPs));
- aDCSError = (GetValueSet(host, port, aDP->String(), valueSet, kDP) == 0);
-
- if(!aDCSError)
+ delete dcsMap;
+ return kFALSE;
+ }
+ }
+
+ if (fConfig->GetDCSDataPoints(fCurrentDetector, iServ)->GetEntries() > 0)
{
- dcsMap.Add(aDP->Clone(), valueSet);
- } else {
- Log(fCurrentDetector,
- Form("ProcessCurrentDetector - Error while retrieving data point %s",
- aDP->GetName()));
- UpdateShuttleStatus(AliShuttleStatus::kDCSError);
- dcsMap.DeleteAll();
- return kFALSE;
+ dpMap = GetValueSet(host, port,
+ fConfig->GetDCSDataPoints(fCurrentDetector, iServ),
+ kDP, multiSplit);
+ if (!dpMap)
+ {
+ Log(fCurrentDetector,
+ Form("ProcessCurrentDetector -"
+ " Error retrieving DCS data points from server %s."
+ " Sending mail to DCS experts!", host.Data()));
+ UpdateShuttleStatus(AliShuttleStatus::kDCSError);
+
+ if (!SendMailToDCS())
+ Log("SHUTTLE", Form("ProcessCurrentDetector - "
+ "Could not send mail to DCS experts!"));
+
+ if (aliasMap) delete aliasMap;
+ delete dcsMap;
+ return kFALSE;
+ }
+ }
+
+ // merge aliasMap and dpMap into dcsMap
+ if(aliasMap) {
+ TIter iter(aliasMap);
+ TObjString* key = 0;
+ while ((key = (TObjString*) iter.Next()))
+ dcsMap->Add(key, aliasMap->GetValue(key->String()));
+
+ aliasMap->SetOwner(kFALSE);
+ delete aliasMap;
+ }
+
+ if(dpMap) {
+ TIter iter(dpMap);
+ TObjString* key = 0;
+ while ((key = (TObjString*) iter.Next()))
+ dcsMap->Add(key, dpMap->GetValue(key->String()));
+
+ dpMap->SetOwner(kFALSE);
+ delete dpMap;
}
}
}
-
+
+ // save map into file, to help debugging in case of preprocessor error
+ TFile* f = TFile::Open("DCSMap.root","recreate");
+ f->cd();
+ dcsMap->Write("DCSMap", TObject::kSingleKey);
+ f->Close();
+ delete f;
+
// DCS Archive DB processing successful. Call Preprocessor!
UpdateShuttleStatus(AliShuttleStatus::kPPStarted);
- UInt_t returnValue = aPreprocessor->Process(&dcsMap);
+ UInt_t returnValue = aPreprocessor->Process(dcsMap);
if (returnValue > 0) // Preprocessor error!
{
- Log(fCurrentDetector, Form("Preprocessor failed. Process returned %d.", returnValue));
+ Log(fCurrentDetector, Form("ProcessCurrentDetector - "
+ "Preprocessor failed. Process returned %d.", returnValue));
UpdateShuttleStatus(AliShuttleStatus::kPPError);
- dcsMap.DeleteAll();
+ dcsMap->DeleteAll();
+ delete dcsMap;
return kFALSE;
}
Log(fCurrentDetector, Form("ProcessCurrentDetector - %s preprocessor returned success",
fCurrentDetector.Data()));
- dcsMap.DeleteAll();
+ dcsMap->DeleteAll();
+ delete dcsMap;
return kTRUE;
}
+//______________________________________________________________________________________________
+void AliShuttle::CountOpenRuns()
+{
+ // Query DAQ's Shuttle logbook and sends the number of open runs to ML
+
+ // check connection, in case connect
+ if (!Connect(3))
+ return;
+
+ TString sqlQuery;
+ sqlQuery = Form("select count(*) from %s where shuttle_done=0", fConfig->GetShuttlelbTable());
+
+ TSQLResult* aResult = fServer[3]->Query(sqlQuery);
+ if (!aResult) {
+ AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
+ return;
+ }
+
+ AliDebug(2,Form("Query = %s", sqlQuery.Data()));
+
+ if (aResult->GetRowCount() == 0) {
+ AliError(Form("No result for query %s received", sqlQuery.Data()));
+ return;
+ }
+
+ if (aResult->GetFieldCount() != 1) {
+ AliError(Form("Invalid field count for query %s received", sqlQuery.Data()));
+ return;
+ }
+
+ TSQLRow* aRow = aResult->Next();
+ if (!aRow) {
+ AliError(Form("Could not receive result of query %s", sqlQuery.Data()));
+ return;
+ }
+
+ TString result(aRow->GetField(0), aRow->GetFieldLength(0));
+ Int_t count = result.Atoi();
+
+ Log("SHUTTLE", Form("%d unprocessed runs", count));
+
+ delete aRow;
+ delete aResult;
+
+ TMonaLisaValue mlStatus("SHUTTLE_openruns", count);
+
+ TList mlList;
+ mlList.Add(&mlStatus);
+
+ fMonaLisa->SendParameters(&mlList, "__PROCESSINGINFO__");
+}
+
//______________________________________________________________________________________________
Bool_t AliShuttle::QueryShuttleLogbook(const char* whereClause,
TObjArray& entries)
entries.SetOwner(1);
// check connection, in case connect
- if(!Connect(3)) return kFALSE;
+ if (!Connect(3)) return kFALSE;
TString sqlQuery;
sqlQuery = Form("select * from %s %s order by run", fConfig->GetShuttlelbTable(), whereClause);
AliDebug(2,Form("Query = %s", sqlQuery.Data()));
if(aResult->GetRowCount() == 0) {
- AliInfo("No entries in Shuttle Logbook match request");
+ Log("SHUTTLE", "No entries in Shuttle Logbook match request");
delete aResult;
return kTRUE;
}
// TODO Check field count!
- const UInt_t nCols = 22;
+ const UInt_t nCols = 23;
if (aResult->GetFieldCount() != (Int_t) nCols) {
- AliError("Invalid SQL result field number!");
+ Log("SHUTTLE", "Invalid SQL result field number!");
delete aResult;
return kFALSE;
}
TSQLResult* aResult = fServer[3]->Query(sqlQuery);
if (!aResult) {
- AliError(Form("Can't execute query <%s>!", sqlQuery.Data()));
+ Log("SHUTTLE", Form("Can't execute query <%s>!", sqlQuery.Data()));
return 0;
}
}
if (aResult->GetRowCount() > 1) {
- AliError(Form("More than one entry in DAQ Logbook for run %d. Skipping", run));
+ Log("SHUTTLE", Form("QueryRunParameters - UNEXPECTED: "
+ "more than one entry in DAQ Logbook for run %d!", run));
delete aResult;
return 0;
}
TSQLRow* aRow = aResult->Next();
if (!aRow)
{
- AliError(Form("Could not retrieve row for run %d. Skipping", run));
+ Log("SHUTTLE", Form("QueryRunParameters - Could not retrieve row for run %d. Skipping", run));
delete aResult;
return 0;
}
UInt_t startTime = entry->GetStartTime();
UInt_t endTime = entry->GetEndTime();
- if (!startTime || !endTime || startTime > endTime) {
+// if (!startTime || !endTime || startTime > endTime)
+// {
+// Log("SHUTTLE",
+// Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d. Skipping!",
+// run, startTime, endTime));
+//
+// Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
+// fLogbookEntry = entry;
+// if (!UpdateShuttleLogbook("shuttle_done"))
+// {
+// AliError(Form("Could not update logbook for run %d !", run));
+// }
+// fLogbookEntry = 0;
+//
+// delete entry;
+// delete aRow;
+// delete aResult;
+// return 0;
+// }
+
+ if (!startTime)
+ {
Log("SHUTTLE",
- Form("QueryRunParameters - Invalid parameters for Run %d: startTime = %d, endTime = %d",
- run, startTime, endTime));
+ Form("QueryRunParameters - Invalid parameters for Run %d: "
+ "startTime = %d, endTime = %d. Skipping!",
+ run, startTime, endTime));
+
+ Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
+ fLogbookEntry = entry;
+ if (!UpdateShuttleLogbook("shuttle_ignored"))
+ {
+ AliError(Form("Could not update logbook for run %d !", run));
+ }
+ fLogbookEntry = 0;
+
+ delete entry;
+ delete aRow;
+ delete aResult;
+ return 0;
+ }
+
+ if (startTime && !endTime)
+ {
+ // TODO Here we don't mark SHUTTLE done, because this may mean
+ //the run is still ongoing!!
+ Log("SHUTTLE",
+ Form("QueryRunParameters - Invalid parameters for Run %d: "
+ "startTime = %d, endTime = %d. Skipping (Shuttle won't be marked as DONE)!",
+ run, startTime, endTime));
+
+ //Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
+ //fLogbookEntry = entry;
+ //if (!UpdateShuttleLogbook("shuttle_done"))
+ //{
+ // AliError(Form("Could not update logbook for run %d !", run));
+ //}
+ //fLogbookEntry = 0;
+
+ delete entry;
+ delete aRow;
+ delete aResult;
+ return 0;
+ }
+
+ if (startTime && endTime && (startTime > endTime))
+ {
+ Log("SHUTTLE",
+ Form("QueryRunParameters - Invalid parameters for Run %d: "
+ "startTime = %d, endTime = %d. Skipping!",
+ run, startTime, endTime));
+
+ Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
+ fLogbookEntry = entry;
+ if (!UpdateShuttleLogbook("shuttle_ignored"))
+ {
+ AliError(Form("Could not update logbook for run %d !", run));
+ }
+ fLogbookEntry = 0;
+
+ delete entry;
+ delete aRow;
+ delete aResult;
+ return 0;
+ }
+
+ TString totEventsStr = entry->GetRunParameter("totalEvents");
+ Int_t totEvents = totEventsStr.Atoi();
+ if (totEvents < 1)
+ {
+ Log("SHUTTLE",
+ Form("QueryRunParameters - Run %d has 0 events - Skipping!", run));
+
+ Log("SHUTTLE", Form("Marking SHUTTLE done for run %d", run));
+ fLogbookEntry = entry;
+ if (!UpdateShuttleLogbook("shuttle_ignored"))
+ {
+ AliError(Form("Could not update logbook for run %d !", run));
+ }
+ fLogbookEntry = 0;
+
delete entry;
delete aRow;
delete aResult;
}
//______________________________________________________________________________________________
-Bool_t AliShuttle::GetValueSet(const char* host, Int_t port, const char* entry,
- TObjArray* valueSet, DCSType type)
+TMap* AliShuttle::GetValueSet(const char* host, Int_t port, const TSeqCollection* entries,
+ DCSType type, Int_t multiSplit)
{
// Retrieve all "entry" data points from the DCS server
// host, port: TSocket connection parameters
- // entry: name of the alias or data point
- // valueSet: array of retrieved AliDCSValue's
+ // entries: list of name of the alias or data point
// type: kAlias or kDP
+ // returns TMap of values, 0 when failure
+
+ AliDCSClient client(host, port, fTimeout, fRetries, multiSplit);
- AliDCSClient client(host, port, fTimeout, fRetries);
- if (!client.IsConnected())
- {
- return kFALSE;
- }
-
- Int_t result=0;
-
+ TMap* result = 0;
if (type == kAlias)
{
- result = client.GetAliasValues(entry,
- GetCurrentStartTime(), GetCurrentEndTime(), valueSet);
- } else
- if (type == kDP)
+ result = client.GetAliasValues(entries, GetCurrentStartTime(),
+ GetCurrentEndTime());
+ }
+ else if (type == kDP)
{
- result = client.GetDPValues(entry,
- GetCurrentStartTime(), GetCurrentEndTime(), valueSet);
+ result = client.GetDPValues(entries, GetCurrentStartTime(),
+ GetCurrentEndTime());
}
- if (result < 0)
+ if (result == 0)
{
- Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get '%s'! Reason: %s",
- entry, AliDCSClient::GetErrorString(result)));
-
- if (result == AliDCSClient::fgkServerError)
- {
- Log(fCurrentDetector.Data(), Form("GetValueSet - Server error: %s",
+ Log(fCurrentDetector.Data(), Form("GetValueSet - Can't get entries! Reason: %s",
+ client.GetErrorString(client.GetResultErrorCode())));
+ if (client.GetResultErrorCode() == AliDCSClient::fgkServerError)
+ Log(fCurrentDetector.Data(), Form("GetValueSet - Server error code: %s",
client.GetServerError().Data()));
- }
- return kFALSE;
+ return 0;
}
-
- return kTRUE;
+
+ return result;
}
//______________________________________________________________________________________________
filePath.Data(), fileSize.Data(), fileChecksum.Data()));
// retrieved file is renamed to make it unique
- TString localFileName = Form("%s_%s_%d_%s_%s.shuttle",
- GetSystemName(system), detector, GetCurrentRun(), id, sourceName.Data());
+ TString localFileName = Form("%s/%s_%d_process/%s_%s_%d_%s_%s.shuttle",
+ GetShuttleTempDir(), detector, GetCurrentRun(),
+ GetSystemName(system), detector, GetCurrentRun(),
+ id, sourceName.Data());
// file retrieval from FXS
Log(detector, Form("GetFileName - Copy of file %s from %s FXS failed",
filePath.Data(), GetSystemName(system)));
continue;
+ }
+
+ if (fileSize.Length()>0)
+ {
+ // compare filesize of local file with the one stored in the FXS DB
+ TString command=("stat --format=%s");
+ Int_t sizeComp = gSystem->Exec(Form("%s %s |grep %s 2>&1 > /dev/null",
+ command.Data(), localFileName.Data(),fileSize.Data()));
+
+ if ( sizeComp != 0)
+ {
+ Log(detector, Form("GetFileName - size of file %s does not match with local copy!",
+ filePath.Data()));
+ result = kFALSE;
+ continue;
+ }
+
} else {
- AliInfo(Form("File %s copied from %s FXS into %s/%s",
- filePath.Data(), GetSystemName(system),
- GetShuttleTempDir(), localFileName.Data()));
- }
+ Log(fCurrentDetector, Form("GetFile - size of file %s not set in %s database, skipping comparison",
+ filePath.Data(), GetSystemName(system)));
+ }
if (fileChecksum.Length()>0)
{
// compare md5sum of local file with the one stored in the FXS DB
- Int_t md5Comp = gSystem->Exec(Form("md5sum %s/%s |grep %s 2>&1 > /dev/null",
- GetShuttleTempDir(), localFileName.Data(), fileChecksum.Data()));
+ Int_t md5Comp = gSystem->Exec(Form("md5sum %s |grep %s 2>&1 > /dev/null",
+ localFileName.Data(), fileChecksum.Data()));
if (md5Comp != 0)
{
TObjString *fileParams = new TObjString(Form("%s#!?!#%s", id, sourceName.Data()));
fFXSlist[system].Add(fileParams);
- static TString fullLocalFileName;
- fullLocalFileName = TString::Format("%s/%s", GetShuttleTempDir(), localFileName.Data());
-
- AliInfo(Form("fullLocalFileName = %s", fullLocalFileName.Data()));
-
- return fullLocalFileName.Data();
-
+ static TString staticLocalFileName;
+ staticLocalFileName.Form("%s", localFileName.Data());
+
+ Log(fCurrentDetector, Form("GetFile - Retrieved file with id %s and "
+ "source %s from %s to %s", id, source,
+ GetSystemName(system), localFileName.Data()));
+
+ return staticLocalFileName.Data();
}
//______________________________________________________________________________________________
//
// check temp directory: trying to cd to temp; if it does not exist, create it
- AliDebug(2, Form("Copy file %s from %s FXS into %s/%s",
- GetSystemName(system), fxsFileName, GetShuttleTempDir(), localFileName));
+ AliDebug(2, Form("Copy file %s from %s FXS into %s",
+ GetSystemName(system), fxsFileName, localFileName));
+
+ TString tmpDir(localFileName);
+
+ tmpDir = tmpDir(0,tmpDir.Last('/'));
- void* dir = gSystem->OpenDirectory(GetShuttleTempDir());
- if (dir == NULL) {
- if (gSystem->mkdir(GetShuttleTempDir(), kTRUE)) {
- AliError(Form("Can't open directory <%s>", GetShuttleTempDir()));
+ Int_t noDir = gSystem->GetPathInfo(tmpDir.Data(), 0, (Long64_t*) 0, 0, 0);
+ if (noDir) // temp dir does not exists!
+ {
+ if (gSystem->mkdir(tmpDir.Data(), 1))
+ {
+ Log(fCurrentDetector.Data(), "RetrieveFile - could not make temp directory!!");
return kFALSE;
}
-
- } else {
- gSystem->FreeDirectory(dir);
}
TString baseFXSFolder;
}
else if (system == kHLT)
{
- baseFXSFolder = "~/";
+ baseFXSFolder = "/opt/FXS/";
}
- TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s/%s",
+ TString command = Form("scp -oPort=%d -2 %s@%s:%s%s %s",
fConfig->GetFXSPort(system),
fConfig->GetFXSUser(system),
fConfig->GetFXSHost(system),
baseFXSFolder.Data(),
fxsFileName,
- GetShuttleTempDir(),
localFileName);
AliDebug(2, Form("%s",command.Data()));
{
//
// Get sources producing the condition file Id from file exchange servers
+ // if id is NULL all sources are returned (distinct)
//
+
+ if (id)
+ {
+ Log(detector, Form("GetFileSources - Querying %s FXS for files with id %s produced by %s", GetSystemName(system), id, detector));
+ } else {
+ Log(detector, Form("GetFileSources - Querying %s FXS for files produced by %s", GetSystemName(system), detector));
+ }
// check if test mode should simulate a FXS error
if (fTestMode & kErrorFXSSources)
return 0;
}
-
if (system == kDCS)
{
- AliError("DCS system has only one source of data!");
- return NULL;
+ Log(detector, "GetFileSources - WARNING: DCS system has only one source of data!");
+ TList *list = new TList();
+ list->SetOwner(1);
+ list->Add(new TObjString(" "));
+ return list;
}
// check connection, in case connect
if (!Connect(system))
{
- Log(detector, Form("GetFile - Couldn't connect to %s FXS database", GetSystemName(system)));
+ Log(detector, Form("GetFileSources - Couldn't connect to %s FXS database", GetSystemName(system)));
return NULL;
}
sourceName = "DDLnumbers";
}
- TString sqlQueryStart = Form("select %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
- TString whereClause = Form("run=%d and detector=\"%s\" and fileId=\"%s\"",
- GetCurrentRun(), detector, id);
+ TString sqlQueryStart = Form("select distinct %s from %s where", sourceName.Data(), fConfig->GetFXSdbTable(system));
+ TString whereClause = Form("run=%d and detector=\"%s\"",
+ GetCurrentRun(), detector);
+ if (id)
+ whereClause += Form(" and fileId=\"%s\"", id);
TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
return list;
}
- TSQLRow* aRow;
+ Log(detector, Form("GetFileSources - Found %d sources", aResult->GetRowCount()));
+ TSQLRow* aRow;
while ((aRow = aResult->Next()))
{
return list;
}
+//______________________________________________________________________________________________
+TList* AliShuttle::GetFileIDs(Int_t system, const char* detector, const char* source)
+{
+ //
+ // Get all ids of condition files produced by a given source from file exchange servers
+ //
+
+ Log(detector, Form("GetFileIDs - Retrieving ids with source %s with %s", source, GetSystemName(system)));
+
+ // check if test mode should simulate a FXS error
+ if (fTestMode & kErrorFXSSources)
+ {
+ Log(detector, Form("GetFileIDs - In TESTMODE - Simulating error while connecting to %s FXS", GetSystemName(system)));
+ return 0;
+ }
+
+ // check connection, in case connect
+ if (!Connect(system))
+ {
+ Log(detector, Form("GetFileIDs - Couldn't connect to %s FXS database", GetSystemName(system)));
+ return NULL;
+ }
+
+ TString sourceName = 0;
+ if (system == kDAQ)
+ {
+ sourceName = "DAQsource";
+ } else if (system == kHLT)
+ {
+ sourceName = "DDLnumbers";
+ }
+
+ TString sqlQueryStart = Form("select fileId from %s where", fConfig->GetFXSdbTable(system));
+ TString whereClause = Form("run=%d and detector=\"%s\"",
+ GetCurrentRun(), detector);
+ if (sourceName.Length() > 0 && source)
+ whereClause += Form(" and %s=\"%s\"", sourceName.Data(), source);
+ TString sqlQuery = Form("%s %s", sqlQueryStart.Data(), whereClause.Data());
+
+ AliDebug(2, Form("SQL query: \n%s",sqlQuery.Data()));
+
+ // Query execution
+ TSQLResult* aResult;
+ aResult = fServer[system]->Query(sqlQuery);
+ if (!aResult) {
+ Log(detector, Form("GetFileIDs - Can't execute SQL query to %s database for source: %s",
+ GetSystemName(system), source));
+ return 0;
+ }
+
+ TList *list = new TList();
+ list->SetOwner(1);
+
+ if (aResult->GetRowCount() == 0)
+ {
+ Log(detector,
+ Form("GetFileIDs - No entry in %s FXS table for source: %s", GetSystemName(system), source));
+ delete aResult;
+ return list;
+ }
+
+ Log(detector, Form("GetFileIDs - Found %d ids", aResult->GetRowCount()));
+
+ TSQLRow* aRow;
+
+ while ((aRow = aResult->Next()))
+ {
+
+ TString id(aRow->GetField(0), aRow->GetFieldLength(0));
+ AliDebug(2, Form("fileId = %s", id.Data()));
+ list->Add(new TObjString(id));
+ delete aRow;
+ }
+
+ delete aResult;
+
+ return list;
+}
+
//______________________________________________________________________________________________
Bool_t AliShuttle::Connect(Int_t system)
{
TString detName(detector);
TString setClause;
- if(detName == "shuttle_done")
+ if (detName == "shuttle_done" || detName == "shuttle_ignored")
{
setClause = "set shuttle_done=1";
- // Send the information to ML
- TMonaLisaText mlStatus("SHUTTLE_status", "Done");
-
- TList mlList;
- mlList.Add(&mlStatus);
+ if (detName == "shuttle_done")
+ {
+ // Send the information to ML
+ TMonaLisaText mlStatus("SHUTTLE_status", "Done");
- fMonaLisa->SendParameters(&mlList);
+ TList mlList;
+ mlList.Add(&mlStatus);
+
+ TString mlID;
+ mlID.Form("%d", GetCurrentRun());
+ fMonaLisa->SendParameters(&mlList, mlID);
+ }
} else {
TString statusStr(status);
if(statusStr.Contains("done", TString::kIgnoreCase) ||
return fLogbookEntry ? fLogbookEntry->GetEndTime() : 0;
}
+//______________________________________________________________________________________________
+UInt_t AliShuttle::GetCurrentYear() const
+{
+ //
+ // Get current year from logbook entry
+ //
+
+ if (!fLogbookEntry) return 0;
+
+ TTimeStamp startTime(GetCurrentStartTime());
+ TString year = Form("%d",startTime.GetDate());
+ year = year(0,4);
+
+ return year.Atoi();
+}
+
+//______________________________________________________________________________________________
+const char* AliShuttle::GetLHCPeriod() const
+{
+ //
+ // Get current LHC period from logbook entry
+ //
+
+ if (!fLogbookEntry) return 0;
+
+ return fLogbookEntry->GetRunParameter("LHCperiod");
+}
+
//______________________________________________________________________________________________
void AliShuttle::Log(const char* detector, const char* message)
{
// Fill log string with a message
//
- void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
+ TString logRunDir = GetShuttleLogDir();
+ if (GetCurrentRun() >=0)
+ logRunDir += Form("/%d", GetCurrentRun());
+
+ void* dir = gSystem->OpenDirectory(logRunDir.Data());
if (dir == NULL) {
- if (gSystem->mkdir(GetShuttleLogDir(), kTRUE)) {
+ if (gSystem->mkdir(logRunDir.Data(), kTRUE)) {
AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
return;
}
TString fileName;
if (GetCurrentRun() >= 0)
- fileName.Form("%s/%s_%d.log", GetShuttleLogDir(), detector, GetCurrentRun());
- else
+ {
+ fileName.Form("%s/%d/%s_%d.log", GetShuttleLogDir(), GetCurrentRun(),
+ detector, GetCurrentRun());
+ } else {
fileName.Form("%s/%s.log", GetShuttleLogDir(), detector);
+ }
return fileName;
}
+//______________________________________________________________________________________________
+void AliShuttle::SendAlive()
+{
+ // sends alive message to ML
+
+ TMonaLisaText mlStatus("SHUTTLE_status", "Alive");
+
+ TList mlList;
+ mlList.Add(&mlStatus);
+
+ fMonaLisa->SendParameters(&mlList, "__PROCESSINGINFO__");
+}
+
//______________________________________________________________________________________________
Bool_t AliShuttle::Collect(Int_t run)
{
SetLastAction("Starting");
+ // create ML instance
+ if (!fMonaLisa)
+ fMonaLisa = new TMonaLisaWriter(fConfig->GetMonitorHost(), fConfig->GetMonitorTable());
+
+ SendAlive();
+ CountOpenRuns();
+
TString whereClause("where shuttle_done=0");
if (run != -1)
whereClause += Form(" and run=%d", run);
if (!RetrieveConditionsData(shuttleLogbookEntries))
{
Log("SHUTTLE", "Collect - Process of at least one run failed");
+ CountOpenRuns();
return kFALSE;
}
Log("SHUTTLE", "Collect - Requested run(s) successfully processed");
+ CountOpenRuns();
return kTRUE;
}
}
// clean SHUTTLE temp directory
- TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
- RemoveFile(filename.Data());
+ //TString filename = Form("%s/*.shuttle", GetShuttleTempDir());
+ //RemoveFile(filename.Data());
}
return hasError == kFALSE;
if (fTestMode != kNone)
return kTRUE;
+
+ if (!fConfig->SendMail()) return kTRUE;
+
+ TString to="";
+ TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
+ TObjString *anExpert=0;
+ while ((anExpert = (TObjString*) iterExperts.Next()))
+ {
+ to += Form("%s,", anExpert->GetName());
+ }
+ if (to.Length() > 0)
+ to.Remove(to.Length()-1);
+ AliDebug(2, Form("to: %s",to.Data()));
+
+ if (to.IsNull()) {
+ Log("SHUTTLE", "List of detector responsibles not set!");
+ return kFALSE;
+ }
void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
if (dir == NULL)
{
if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
{
- AliError(Form("Can't open directory <%s>", GetShuttleLogDir()));
+ Log("SHUTTLE", Form("SendMail - Can't open directory <%s>", GetShuttleLogDir()));
+ return kFALSE;
+ }
+
+ } else {
+ gSystem->FreeDirectory(dir);
+ }
+
+ TString bodyFileName;
+ bodyFileName.Form("%s/mail.body", GetShuttleLogDir());
+ gSystem->ExpandPathName(bodyFileName);
+
+ ofstream mailBody;
+ mailBody.open(bodyFileName, ofstream::out);
+
+ if (!mailBody.is_open())
+ {
+ Log("SHUTTLE", Form("Could not open mail body file %s", bodyFileName.Data()));
+ return kFALSE;
+ }
+
+ TString cc="";
+ TIter iterAdmins(fConfig->GetAdmins(AliShuttleConfig::kGlobal));
+ TObjString *anAdmin=0;
+ while ((anAdmin = (TObjString*) iterAdmins.Next()))
+ {
+ cc += Form("%s,", anAdmin->GetName());
+ }
+ if (cc.Length() > 0)
+ cc.Remove(cc.Length()-1);
+ AliDebug(2, Form("cc: %s",to.Data()));
+
+ TString subject = Form("%s Shuttle preprocessor FAILED in run %d (run type = %s)!",
+ fCurrentDetector.Data(), GetCurrentRun(), GetRunType());
+ AliDebug(2, Form("subject: %s", subject.Data()));
+
+ TString body = Form("Dear %s expert(s), \n\n", fCurrentDetector.Data());
+ body += Form("SHUTTLE just detected that your preprocessor "
+ "failed processing run %d (run type = %s)!!\n\n",
+ GetCurrentRun(), GetRunType());
+ body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n",
+ fCurrentDetector.Data());
+ if (fConfig->GetRunMode() == AliShuttleConfig::kTest)
+ {
+ body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
+ } else {
+ body += Form("\thttp://pcalimonitor.cern.ch/shuttle.jsp?instance=PROD&time=168 \n\n");
+ }
+
+
+ TString logFolder = "logs";
+ if (fConfig->GetRunMode() == AliShuttleConfig::kProd)
+ logFolder += "_PROD";
+
+
+ body += Form("Find the %s log for the current run on \n\n"
+ "\thttp://pcalishuttle01.cern.ch:8880/%s/%d/%s_%d.log \n\n",
+ fCurrentDetector.Data(), logFolder.Data(), GetCurrentRun(),
+ fCurrentDetector.Data(), GetCurrentRun());
+ body += Form("The last 10 lines of %s log file are following:\n\n", fCurrentDetector.Data());
+
+ AliDebug(2, Form("Body begin: %s", body.Data()));
+
+ mailBody << body.Data();
+ mailBody.close();
+ mailBody.open(bodyFileName, ofstream::out | ofstream::app);
+
+ TString logFileName = Form("%s/%d/%s_%d.log", GetShuttleLogDir(),
+ GetCurrentRun(), fCurrentDetector.Data(), GetCurrentRun());
+ TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
+ if (gSystem->Exec(tailCommand.Data()))
+ {
+ mailBody << Form("%s log file not found ...\n\n", fCurrentDetector.Data());
+ }
+
+ TString endBody = Form("------------------------------------------------------\n\n");
+ endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
+ endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
+ endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
+
+ AliDebug(2, Form("Body end: %s", endBody.Data()));
+
+ mailBody << endBody.Data();
+
+ mailBody.close();
+
+ // send mail!
+ TString mailCommand = Form("mail -s \"%s\" -c %s %s < %s",
+ subject.Data(),
+ cc.Data(),
+ to.Data(),
+ bodyFileName.Data());
+ AliDebug(2, Form("mail command: %s", mailCommand.Data()));
+
+ Bool_t result = gSystem->Exec(mailCommand.Data());
+
+ return result == 0;
+}
+
+//______________________________________________________________________________________________
+Bool_t AliShuttle::SendMailToDCS()
+{
+ //
+ // sends a mail to the DCS Amanda experts in case of DCS data point retrieval error
+ //
+
+ if (fTestMode != kNone)
+ return kTRUE;
+
+ if (!fConfig->SendMail()) return kTRUE;
+
+ void* dir = gSystem->OpenDirectory(GetShuttleLogDir());
+ if (dir == NULL)
+ {
+ if (gSystem->mkdir(GetShuttleLogDir(), kTRUE))
+ {
+ Log("SHUTTLE", Form("SendMailToDCS - Can't open directory <%s>", GetShuttleLogDir()));
return kFALSE;
}
if (!mailBody.is_open())
{
- AliError(Form("Could not open mail body file %s", bodyFileName.Data()));
+ Log("SHUTTLE", Form("SendMailToDCS - Could not open mail body file %s", bodyFileName.Data()));
return kFALSE;
}
TString to="";
- TIter iterExperts(fConfig->GetResponsibles(fCurrentDetector));
+ TIter iterExperts(fConfig->GetAdmins(AliShuttleConfig::kAmanda));
TObjString *anExpert=0;
while ((anExpert = (TObjString*) iterExperts.Next()))
{
to += Form("%s,", anExpert->GetName());
}
- to.Remove(to.Length()-1);
+ if (to.Length() > 0)
+ to.Remove(to.Length()-1);
AliDebug(2, Form("to: %s",to.Data()));
if (to.IsNull()) {
- AliInfo("List of detector responsibles not yet set!");
+ Log("SHUTTLE", "List of Amanda server administrators not set!");
return kFALSE;
}
- TString cc="alberto.colla@cern.ch";
+ TString cc="";
+ TIter iterAdmins(fConfig->GetAdmins(AliShuttleConfig::kGlobal));
+ TObjString *anAdmin=0;
+ while ((anAdmin = (TObjString*) iterAdmins.Next()))
+ {
+ cc += Form("%s,", anAdmin->GetName());
+ }
+ if (cc.Length() > 0)
+ cc.Remove(cc.Length()-1);
+ AliDebug(2, Form("cc: %s",to.Data()));
- TString subject = Form("%s Shuttle preprocessor error in run %d !",
+ TString subject = Form("Retrieval of data points for %s FAILED in run %d !",
fCurrentDetector.Data(), GetCurrentRun());
AliDebug(2, Form("subject: %s", subject.Data()));
- TString body = Form("Dear %s expert(s), \n\n", fCurrentDetector.Data());
- body += Form("SHUTTLE just detected that your preprocessor "
- "exited with ERROR state in run %d!!\n\n", GetCurrentRun());
- body += Form("Please check %s status on the web page asap!\n\n", fCurrentDetector.Data());
- body += Form("The last 10 lines of %s log file are following:\n\n");
+ TString body = Form("Dear DCS experts, \n\n");
+ body += Form("SHUTTLE couldn\'t retrieve the data points for detector %s "
+ "in run %d!!\n\n", fCurrentDetector.Data(), GetCurrentRun());
+ body += Form("Please check %s status on the SHUTTLE monitoring page: \n\n",
+ fCurrentDetector.Data());
+ if (fConfig->GetRunMode() == AliShuttleConfig::kTest)
+ {
+ body += Form("\thttp://pcalimonitor.cern.ch:8889/shuttle.jsp?time=168 \n\n");
+ } else {
+ body += Form("\thttp://pcalimonitor.cern.ch/shuttle.jsp?instance=PROD?time=168 \n\n");
+ }
+
+ TString logFolder = "logs";
+ if (fConfig->GetRunMode() == AliShuttleConfig::kProd)
+ logFolder += "_PROD";
+
+
+ body += Form("Find the %s log for the current run on \n\n"
+ "\thttp://pcalishuttle01.cern.ch:8880/%s/%d/%s_%d.log \n\n",
+ fCurrentDetector.Data(), logFolder.Data(), GetCurrentRun(),
+ fCurrentDetector.Data(), GetCurrentRun());
+ body += Form("The last 10 lines of %s log file are following:\n\n", fCurrentDetector.Data());
AliDebug(2, Form("Body begin: %s", body.Data()));
mailBody.close();
mailBody.open(bodyFileName, ofstream::out | ofstream::app);
- TString logFileName = Form("%s/%s_%d.log", GetShuttleLogDir(), fCurrentDetector.Data(), GetCurrentRun());
+ TString logFileName = Form("%s/%d/%s_%d.log", GetShuttleLogDir(), GetCurrentRun(),
+ fCurrentDetector.Data(), GetCurrentRun());
TString tailCommand = Form("tail -n 10 %s >> %s", logFileName.Data(), bodyFileName.Data());
if (gSystem->Exec(tailCommand.Data()))
{
TString endBody = Form("------------------------------------------------------\n\n");
endBody += Form("In case of problems please contact the SHUTTLE core team.\n\n");
endBody += "Please do not answer this message directly, it is automatically generated.\n\n";
- endBody += "Sincerely yours,\n\n \t\t\tthe SHUTTLE\n";
+ endBody += "Greetings,\n\n \t\t\tthe SHUTTLE\n";
AliDebug(2, Form("Body end: %s", endBody.Data()));
return fLogbookEntry->GetRunType();
}
+//______________________________________________________________________________________________
+Bool_t AliShuttle::GetHLTStatus()
+{
+ // Return HLT status (ON=1 OFF=0)
+ // Converts the HLT status from the status string read in the run logbook (not just a bool)
+
+ if(!fLogbookEntry) {
+ AliError("No logbook entry!");
+ return 0;
+ }
+
+ // TODO implement when HLTStatus is inserted in run logbook
+ //TString hltStatus = fLogbookEntry->GetRunParameter("HLTStatus");
+ //if(hltStatus == "OFF") {return kFALSE};
+
+ return kTRUE;
+}
+
//______________________________________________________________________________________________
void AliShuttle::SetShuttleTempDir(const char* tmpDir)
{
git://git.uio.no / u / mrichter / AliRoot.git / blobdiff