adding number of open runs to monitoring

[u/mrichter/AliRoot.git] / SHUTTLE / doc / doc.tex

\documentclass[a4paper]{report}

\usepackage{graphicx}

\begin{document}

\title{\Huge Shuttle\\\normalsize Program for automatic data retrieval from DCS}

\maketitle

\tableofcontents

\listoftables

\listoffigures

\sloppy

\chapter{Shuttle - Program for automatic data retrieval from DCS}

\section{Overview}
	\label{sec:Shuttle:Overview}
	
	In DCS data base is stored information for particular real world object (RWO)
	(sensors, valves etc.) as temperature, voltage, state etc.  PVSS II is a SCADA 
	(Supervisory Control and Data Acquisition) system used by DCS for control, monitoring 
	and data acquisition. The abstract representation of any RWO is organized in
	DataPoints (DP). Every DP has particular type which can be a composite (tree-like) structure
	or primitive type as boolean, byte, integer, float. DP data is updated with certain refresh
	rate (usally less than 1Hz) forming a time series. As the data points which represent
	particular RWO are changed in time, additional attribute called 'alias' is assigned to them.
	Every alias corresponds to the same RWO all the time.

	When the value of some DP is changed, the new value and the timestamp is stored into the 
	data base. These value/timestamp pairs form the value series for particular DP. As DCS
	data is collected independently on the experimental runs, part of the value series 
	corresponds to time intervals which don't belong to any run. 

	Every sub-detector has a set of 'aliases' which represent some parameters (temperature, 
	pressure, voltage etc.) relevant to its calibration.  The working schema of 
	\emph{Shuttle} program is: collecting the data from DCS using AliDCSClient interface, 
	running the appropriate sub-detector preprocessor (one per sub-detector, used for preliminary 
	processing of the collected data) and storing the output of the preprocessor into the 
	calibration data base.  

\section{General schema}
	
	\begin{figure}
        	\begin{center}
			\includegraphics[width=0.9\textwidth]{pics/ShuttleSchema}
		\end{center}
		\caption{\emph{Shuttle} program general schema.}
		\label{shuttle:schema}
	\end{figure}

	On figure \ref{shuttle:schema} is presented the \emph{Shuttle} program general schema:
	\footnote{
		All names which follow the pattern Ali$<$name$>$ are classes part of
		AliRoot framework.
	}
	
	\begin{description}
		\item[AliShuttle] The \emph{Shuttle} program manager, holds one AliShuttleConfig and
		one AliCDBStorage object instances. It is used to manage data retrieval from DCS 
		\mbox{(through AliDCSClient)} and preliminary data processing. 

		\item[AliShuttleConfig] Reads the configuration from LDAP server. 
		
		\item[AliCDBPreProcessor] The interface for every sub-detector preprocessor.

		\item[AliCDBStorage] Interface to the calibration data base. Currently supports 
		GRID and Local storage.

		\item[AliShuttleTrigger] Waits for notification from DAQ when particular 
		experimental run is finished and runs AliShuttle.

		\item[AliDCSClient] Makes connection to AMANDA and communicates over AliDCSProtocol.

		\item[AMANDA]\footnote {
			AMANDA is an abbreviation of Ali MANager for Data Access.
		}
		Communication layer for data access to the historical data base 
		provided by DCS. Implements AliDCSProtocol. Communicates with PVSS DM through
		the internal PVSS protocol.
		\item[PVSS DM] It's a data manager, part of PVSS SCADA system used to organize the
		interaction with the underlying data base (RDBMS, local files etc.).

	\end{description}

\section{AliShuttle and AliCDBPreProcessor}
	\label{sec:shuttle:preprocessor}	

	AliShuttle is the \emph{Shuttle} program manager, holds one AliShuttleConfig and
	one AliCDBStorage object instances. It is used to manage data retrieval from DCS
	\mbox{(through AliDCSClient)} and preliminary data processing. Only those sub-detectors 
	for which there is a valid configuration are processed by AliShuttle.
	Two methods are used for triggering data retrieval and preprocessing:
	\begin{description}
		\item \mbox{void Process(Int\_t run, UInt\_t startTime, UInt\_t endTime)}\\
		Retrieves data for particular run which lasted in time interval
		\mbox{startTime - endTime} for every sub-detector in the configuration.

		\item \mbox{void Process(Int\_t run, UInt\_t startTime, UInt\_t endTime,
			const char* detector)}\\
		Retrieves data for particular run which lasted in time interval
		\mbox{startTime - endTime} for the specified detector.
	\end{description}

	Every subdetector can register to AliShuttle a specific preprocessor which implements 
	\mbox{AliCDBPreProcessor} interface. The aim of this preprocessor is to process
	(fit, average etc.) the data retrieved from DCS and create and store a specific for 
	the sub-detector needs objects (histograms, functions etc.) which can be used during the
	calibration process.\\
	\mbox{AliCDBPreProcessor} is a subclass of TNamed and GetName() method is used by
	\mbox{AliShuttle} to identify the sub-detector preprocessor.

	It has one method which provides storage to the underlying AliCDBStorage object.
	\begin{description}
		\item \mbox{Bool\_t Store(const char* specType, TObject* object,
                	AliCDBMetaData* metaData)}\\
			Stores \emph{object} with \emph{metaData} and identifier \emph{specType}.\\
			Returns kTRUE in case of success and kFALSE otherwise.
	\end{description}

	Following callback methods are used.
	\begin{description}
		\item \mbox{void Initialize(Int\_t run, UInt\_t startTime, UInt\_t endTime)}\\
		This method is called at the beginning of data retrieval for the sub-detector. 
		Parameters \emph{run}, \mbox{\emph{startTime}} and \mbox{\emph{endTime}}
		specify current experimental run number and the time interval it lasted.
		\item \mbox{void Finalize()}\\
		This method is called at the end of data retrieval for the sub-detector.
		\item \mbox{void Process(const char* alias, TList\& valueSet, Bool\_t hasError)}\\ 
		This method is called for every \emph{alias} in the relative to the sub-detector 
		set specified in the configuration.\\
		\emph{alias} specifies the current alias being processed.\\
		\emph{valueSet} is a collection of AliDCSValue (see. AliDCSClient) representing
		the value series retrieved from DCS.\\
		\emph{hasError} is an error flag indicating if some error (comunication error, 
		data base error etc.) occurred during the data retrieval.

	\end{description}

	If there is a valid configuration for some sub-detector but there isn't corresponding
	registered preprocessor, AliShuttle stores \emph{valueSet} to the calibration data base
	by default. In this case two properties in meta data (AliCDBMetaData), contain the value
	series time interval ("StartTime" and "EndTime" properties contian StartTime and EndTime 
	respectively stored in AliSimpleValue object).

\section{AliShuttleConfig}

	AliShuttleConfig provides transparent API to the \emph{Shuttle} configuration. Currently,
	only LDAP is used to keep the configuration. Configuration consists entries (one per 
	sub-detector) which comprise AMANDA server host and port and set of aliases that the specific
	sub-detector is interested in.  
	
\subsection{Configuration schema in LDAP}

        \begin{figure}
                \begin{center}
                        \includegraphics[width=0.9\textwidth]{pics/ShuttleConfig}
                \end{center}
                \caption{\emph{Shuttle} configuration schema in LDAP.}
                \label{shuttle:config}
        \end{figure}

	On figure \ref{shuttle:config} is presented the LDAP configuration schema.
	Every leaf (TPC, ITS, PHOS etc.) represents the specific sub-detector configuration.
	These entries have objectclass \emph{shuttleConfig} which imposes the attributes
	\emph{ipHost}, \emph{ipServicePort} and \emph{alias} as compulsory. Only \emph{alias}
	attribute is MULTI-VALUE attribute. All sub-detectors' entries belong to one base entry.
	\footnote {
		Default base entry distinguished name is 'dn: dc=alice,dc=cern,dc=ch'.
	}
	
	LDAP access model is used to provide write access to particular sub-detector entry for
	the relevant supporting group and annonymous read access to the whole configuration.

	Here is an example of sub-detector entry in ldif format which can be written to LDAP server:

	\begin{quote}
		\#ITS config\\
		dn: dt=ITS,dc=alice,dc=cern,dc=ch\\
		objectClass: shuttleConfig\\
		dt: ITS\\
		ipHost: 192.168.39.23\\
		ipServicePort: 4242\\
		alias: HighVol01\\
		alias: HighVol02\\
	\end{quote}

\subsection{Class overview}
	AliShuttleConfig reads the configuration from LDAP server on the object creation.\\
	Following methods are used to access the configuration:
	\begin{description}
		\item \mbox{\emph{Bool\_t IsValid() const}}\\
		Returns true if the configuration is properly read from LDAP server and false
		in otherwise.

		\item \mbox{\emph{const TList* GetDetectors() const}}\\
		Returns collection of TObjString objects which represent detectors' name for
		which there is a valid configuration entry.

		\item \mbox{\emph{Bool\_t HasDetector(const char* detector) const}}\\
		Returns true if there is a valid entry configuration for the specified detector.		
		\item \mbox{\emph{const char* GetHost(const char* detector) const}}\\
		Returns AMANDA server host for the specified detector.

		\item \mbox{\emph{const char* GetHost(const char* detector) const}}\\
		Returns AMANDA server port number for the specified detector.

		\item \mbox{\emph{const TList* GetAliases(const char* detector) const}}\\
		Returns collection of TObjString which represents the aliases for the specified 
		detector.
		
	\end{description}

\chapter{AliDCSClient}

\section{Overview}
	
	AliDCSClient provides API for historical data access to DCS data base. It's completely
	based on ROOT framework and there isn't any external libraries. It communicates
	with AMANDA server using AliDCSProtocol. AliDCSProtocol is a simple protocol which comprises
	set of messages (AliDCSMessage). Protocol messages are transfered over TCP/IP.
	
	The main functionality of AliDCSClient is to provide value series (stored into DCS 
	data base) for paticular \emph{alias}/\emph{DataPoint}\footnote {
		It was described in \ref{sec:Shuttle:Overview} that every real object is represented
		by one \emph{DataPoint} at time. As this \emph{DataPoint} can be changed in some
		moment, additional attribute called \emph{alias} is assigned to it. \emph{Alias}
		provides constant identifier for the underlying real object. If \emph{DataPoint}
		is changed its \emph{alias} attribute will be set the same so it will provide a 
		constant association \mbox{\emph{alias}-real object}.
	} in given time interval.
	This value series is represented by collection (TList) of AliDCSValue.

\section{Basic types and AliDCSValue}

	Every \emph{alias} which is used in the calibration has particular type.
	On table \ref{client:types} are listed the whole set of types. There are five primitive
	types and the corresponding dynamic types (arrays of primitive types with arbitrary number
	of elements).

	These types are wrapped by AliSimpleValue. Every instance of this class represents a value
	which type is one of the described. AliSimpleValue is an ROOT class and it can be 
	serialized by the standard ROOT persistent object mechanism.

	\emph{Alias} value series is a collection of value/timestamp pairs. Every such pair is 
	realized by AliDCSValue. AliDCSValue has two fields: AliDCSVlaue.value (AliSimpleValue) 
	and AliDCSValue.timestamp (UInt\_t). The following simple rule could be used to understand
	the meaning of value series:
	
	\emph{Every alias has value (AliDCSValue.value) from the moment this value
	became valid (AliDCSValue.timestamp) to the moment next value becomes valid.}



	\begin{table}
		\begin{center}
			\begin{tabular}{|l|c|}
				\hline
				Type & size in bytes\\
				\hline
				Boolean & 1\\
				Byte & 1\\
				Integer & 4\\
				Unsigned Integer & 4\\
				Float & 4\\
				\hline
				Dynamic Boolean & n * 1\\
				Dynamic Byte & n * 1\\
				Dynamic Integer & n * 4\\
				Dynamic Unsigned Integer & n * 4\\
				Dynamic Float & n * 4 \\
				\hline
			\end{tabular}
		\end{center}
		\label{client:types}
		\caption[Basic types]{
			Basic \emph{alias} types.  For dynamic types 'n' means the 
			number of the elements.  Every dynamic type can have arbitrary 
			number of elements.
		}
	\end{table}
	
\section {AliDCSProtocol}

	AliDCSProtocol is a communication protocol which comprises set of messages (AliDCSMessage).
	Every message is composed of two parts: \emph{header} and \emph{body}. Message \emph{header}
	(table \ref{client:protocol:header}) has fixed size and describes message \emph{body} (size
	and type).

	There are four message types every one of each has different message \emph{body} structure.
	Below is a detailed description of every message type:
	
	\begin{description}
		\item[Request] Message sent by the client to make a data request for 
		particular \emph{alias}/\emph{DataPoint} and given time interval (table 
		\ref{client:protocol:body:request}).

		\item[Count] If precedent request was valid and data could be retrieved from data
		base, AMANDA server sends this message to indicated the total number of values
		corresponding to it (table \ref{client:protocol:body:count}).

		\item[ResultSet] Every such message contains part of the requested data. Server sends
		series of these messages until it returns the total amount indicated by precedent
		Count message (table \ref{client:protocol:body:resultset}).

		\item[Error] Server sends this message if some error occurred. 
		(table \ref{client:protocol:body:error}). 

	\end{description}

	On figure \ref{client:protocol:flowchart} is shown the protocol flowchart.

	\begin{table}
		\begin{center}
			\begin{tabular}{|c|l|}
				\hline
				Byte & Meaning\\
				\hline
				0 - 1 & Message signature ('A' and 'D').\\
				3 & Message version (ver: 1).\\
				4 & Message type. Describes message \emph{body} type.\\
				5 - 8 & Message \emph{body} size (unsigned interger).\\
				\hline
			\end{tabular}
		\caption[Message header structure]{Message \emph{header} structure.}
		\label{client:protocol:header}
		\end{center}
	\end{table}

	\begin{table}
		\begin{center}
			\begin{tabular}{|p{1cm}|p{9cm}|}
				\hline
				Byte & Meaning\\
				\hline
				0 & Request identifier (1 = \emph{alias}, 2 = \emph{DataPoint}).\\
				1 - 4 & Request StartTime - Beginning of the requested interval
					(unsigned integer).\\
				5 - 8 & Request EndTime - End of the requested interval
					(unsigned integer).\\
				8 - end & Request string - Zero terminated string representing
				         requested \emph{alias}/\emph{DataPoint}.\\
				\hline
			\end{tabular}
		\caption[Request message structure]{Request message \emph{body} structure.}
		\label{client:protocol:body:request}
		\end{center}
	\end{table}

	\begin{table}
		\begin{center}
			\begin{tabular}{|p{1cm}|p{9cm}|}
				\hline
				Byte & Meaning\\
				\hline
				0 - 3 & Total number of values (unsigned integer).\\
				\hline
			\end{tabular}
		\caption[Count message structure]{Count message \emph{body} structure.}
		\label{client:protocol:body:count}
		\end{center}
	\end{table}

	 \begin{table}
                \begin{center}
                        \begin{tabular}{|p{1cm}|p{9cm}|}
                                \hline
                                Byte & Meaning\\
                                \hline
				0 & Value type identifier (On of the basic types on table: 
					\ref{client:types}). 
					Specifies the type of values contained in the message.\\
                                1 - 4 & Number of values contained in the message (unsigned integer).\\
				5 - end & Sequence of value/timestamp pairs. Size depends on value type.
					Timestmap size is always 4 bytes (unsigned integer). \\
                                \hline
                        \end{tabular}
                \caption[ResultSet message structure]{ResultSet message \emph{body} structure.}
                \label{client:protocol:body:resultset}
                \end{center}
        \end{table}

	\begin{table}
                \begin{center}
                        \begin{tabular}{|p{1cm}|p{9cm}|}
                                \hline
                                Byte & Meaning\\
                                \hline
                                0 & Server error code.\\
				  & (UnknownAlisDPName = 1\\
				  &  InvalidTimeRange = 2\\
				  &  InvalidBufferSize = 3\\
				  &  InvalidRequest = 4\\
				  &  UnsupportedType = 5\\
				  &  UnknownError = 6)\\
				1 - end & Server error string - Zero terminated string describing the error.\\
                                \hline
                        \end{tabular}
                \caption[Error message structure]{Error message \emph{body} structure.}
                \label{client:protocol:body:error}
                \end{center}
        \end{table}

	\begin{figure}
                \begin{center}
                        \includegraphics[width=1.1\textwidth,height=0.5\textheight]{pics/Protocol}
                \end{center}
                \caption{AliDCSProtocol flowchart.}
                \label{client:protocol:flowchart}
        \end{figure}

\section{AliDCSClient}

	AliDCSClient provides the client API. The following methods compose the most relevant part of
	the interface:
	
	\begin{description}
	
		\item \emph{AliDCSClient(const char* host, Int\_t port, UInt\_t timeout, Int\_t retries)} - 
			Class constructor. \\
			\emph{host} - Amanda server host.\\
			\emph{prot} - Amanda server port.\\
			\emph{time} - Timeout (in milliseconds) which is used during the communication.
			\emph{retries} - Number of tries after which the client considers connection for
					invalid and returns the corresponding error.
		
		\item \emph{Bool\_t IsConnected()} - Returns kTrue in case there is a valid connection with 
			AMANDA server.
		
		\item \emph{void Close()} - Close established connection.

		\item \emph{Int\_t GetDPValues(const char* dpName, UInt\_t startTime, UInt\_t endTime,
                                TList\& result)} - Makes synchronous request to AMANDA server for particular
				\emph{DataPoint} and time interval.\\
				\emph{dpName} - Requested \emph{DataPoint}.\\
				\emph{startTime} - Beginning of time interval (in absolute time).\\
				\emph{endTime} - End of time interval (in absolute time).\\
				\emph{result} - In this collection (TList) is returned the result of request
					(Collection of AliDCSValue).\\
				In case of error returns negative (error code) value otherwise returns the number
				of retrieved values.	

		\item \emph{Int\_t GetAliasValues(const char* alias, UInt\_t startTime, UInt\_t endTime,
                                TList\& result)} - Makes synchronous request to AMANDA server for particular
                                \emph{alias} and time interval.\\
                                \emph{alias} - Requested \emph{alias}.\\
                                \emph{startTime} - Beginning of time interval (in absolute time).\\
                                \emph{endTime} - End of time interval (in absolute time).\\
                                \emph{result} - In this collection (TList) is returned the result of request
                                        (Collection of AliDCSValue).\\
                                In case of error returns negative (error code) value otherwise returns the number
                                of retrieved values.
	
		\item \emph{AliDCSMessage::ErrorCode GetServerErrorCode()} - In case of Error message returned
			by AMANDA server this method returns the corresponding server error code.			
		\item \emph{const TString\& GetServerError()} - In case of Error message returned
			by AMANDA server this method returns the corresponding server error description.			
	\end{description}

	The following list contains the error codes which could be returned by \mbox{\emph{GetDPValues}} and 
	\mbox{\emph{GetAliasValues}} methods in case of error occurred (negative value returned).
	\begin{description}
		\item[fgkBadState] - There was no valid connection to AMANDA server when the request was made.

		\item[fgkTimeout] - Specified number of retries was made and for every one the timeout was 
				reached before AMANDA server sends a response message or accept the connection.

		\item[fgkBadMessage] - Invalid message received.

		\item[fgkCommError] - TPC/IP connection error occurred.

		\item[fgkServerError] - AMANDA server sent Error message. The corresponding error information
			can be retrieved by \mbox{\emph{GetServerErrorCode()}} and \mbox{\emph{GetServerError()}}.
	\end{description}

\end{document}