online QA documentation - Michael

[u/mrichter/AliRoot.git] / EMCAL / doc / QA.tex

\section{Run by run QA, how to and code}\r
\subsection{Online - Francesco, Michael}\r
\r
\subsubsection{Creation and checking of online QA histograms (AMORE)}\r
The QA histograms for the EMCal are created and filled by the class:\\ \r
\texttt{\$ALICE\_ROOT/EMCAL/AliEMCALQADataMakerRec}\\\r
It is run both during the offline reconstruction and by the online data quality monitoring framework. Its main methods are:\\\r
\r
\texttt{InitRaws()}:\r
All the QA histograms are created here. Their titles should be self-explaining. Each of them has 2 important flags:\\\r
\texttt{expert} - if true, the histogram is shipped only to the AMORE expert agent (run in the EMCal station), otherwise it is also shipped to the AMORE shifter agent (at the moment we have 4 histograms monitored by the DQM shifter).\\\r
\texttt{image} - if true, the plots is saved to the loogbok (there should be 9 plots in the logbook).\\\r
Each of those histograms is replicated 4 times by the amore framework and filled according to the event species (Calib, Cosmic, LowMultiplicity, HighMultiplicity).\\\r
\r
\texttt{MakeRaws(AliRawReader* rawReader)}:\r
Here we loop over the input raw data stream and we fill the histograms.\\\r
\r
\texttt{MakeRawsSTU(AliRawReader* rawReader)}:\r
STU raw data decoding, provided EMCal trigger experts.\\\r
\r
\texttt{GetCalibRefFromOCDB()}:\r
Get the reference histograms from the OCDB. The ratio between histograms from current run and the reference run is calculated in the \texttt{MakeRaws} method.\\\r
\r
\r
The QA histograms are then analyzed by the class:\\\r
\texttt{\$ALICE\_ROOT/EMCAL/AliEMCALQAChecker}\\\r
It checks the data contained in the 4 non-expert histograms. They are at the moment: \r
\begin{enumerate}\r
\item Ratio distribution of towers' amplitude in the current run w.r.t. the reference run.\r
\item Number of hits of L1 Gamma patch.\r
\item Number of hits of L1 Jet patch.\r
\item Number of links between TRU and STU. \r
\end{enumerate}\r
The main method is:\r
\texttt{CheckRaws()}: For the first histogram, we check how many\r
channels have the ratio in the region $0.8-1.2$. If there are more than\r
the threshold (default = 90\%), everything is ok, otherwise a red box with a ``call\r
expert'' message is displayed. For L1 Jet/Gamma patch, we check if the\r
rate of a single patch is higher than the average rate of all other\r
patches times a certain threshold value (default = 0.5):\r
\begin{equation}\r
Rate_{patch}/Rate_{Total} > Threshold /(1 + Threshold)\r
\end{equation}\r
\r
 If so red box with a ``hot spot - call expert'' message is\r
 displayed. Finally, if there is a number (default = 1) of missing\r
 STU-TRU links, another red box with a warning message is shown.\r
If one wants to change the thresholds, this can be done by updating\r
them in the database (type \texttt{amoreConfigFileBrowser d emc} at\r
any machine at P2 and edit the file \texttt{QAThresholds.configfile} accordingly).\r
\r
\subsubsection{How to's for EMCal AMORE experts}\r
\r
\paragraph{How to run AMORE at point 2 (P2):}\r
\r
Some useful scripts for running AMORE at P2 are located in the \texttt{\textasciitilde/amore} directory in the aldaqacr51 machine. \texttt{StartAmore.sh} is used to run the agent and the GUI (in the expert mode). It simply launches the \texttt{configMonitor.sh} and the \texttt{checkLogs.sh} (used to check the logs of the agents). \texttt{configMonitor.sh} setup the agent and the GUI using some kdialog commands and runs them.\r
Basically the agent is run by the following line in the script:\\\r
\texttt{sshdqm \$agentName nohup \$scriptsPATH/startAmoreAgent.sh \$onlineMode \&}\\\r
The agent is actually run on the EMCal DQM node via the sshdqm command, so this line basically is used to connect to our dqm node and run the agent there.\\\r
The GUI is launched in our machine by the command:\\\r
\texttt{amore -c EMC -m EMCUIQA(TRU) -g configFile.txt}\\\r
The \texttt{configFile.txt} file contains just one line, i.e. \texttt{runType i}, where \texttt {i} runs from 1 to 4 (1=LowMultiplicity, 2=HighMultiplicity, 3=Cosmic, 4=Calib) and it is used to select the event specie you want to display in the GUI (this file is automatically created each time you run \texttt{configMonitor.sh}).\\\r
Some other useful scripts (which are launched by \texttt{configMonitor.sh} using the sshdqm) are located in our DQM node. One can login to it by using:\\\r
\texttt{ssdqm EMCQA}\\\r
They are located in the \texttt{scripts} directory.\\\r
\texttt{startAmoreAgent.sh} basically runs the agent, it simply issues the command:\\\r
\texttt{amoreAgent -a EMCQA -s @\$1: -u}\\\r
The only parameter here is \$1, which is the gdc to monitor, defined in \texttt{configMonitor.sh}. \r
\texttt{killAgent.sh} kills any running agent. Be aware that there cannot be 2 running agents  of the same kind at the same time (i.e. just one EMCQA and one EMCQAshifter at a time). It can happen that some time the agent does not start when you try to run it. Most of the time this is because the dqm shifter is already running the EMCQA agent instead of just the EMCQAshifter one (they can run it from the dqm station even if they shouldn't). To check if the dqm shifter is running it, simply do \texttt{ps aux | grep EMCQA} in the dmq node, and see if there is an EMCQA process belonging to the user daq. If so, kindly ask the dqm shifter to please kill it.\\\r
The agent is always running using the same parameters, which are\r
stored in a database. In order to change them, you should run the\r
\texttt{amoreConfigFileBrowser} command in the aldaqacr51 machine. A\r
window will appear, there you can browse a lot of configuration\r
files. Our files are \texttt{EMCAL\_config.txt} (it contains just the\r
librares to be loaded by the agent, and the event species to monitor),\r
\texttt{QAdescriptions\-EMC.configfile} (it contains the descriptions\r
of the histograms shipped to the dqm shifter), and\r
\texttt{QAThresholds.configfile} (it contains the thresholds for the\r
QA checker - see above). You can edit them by pressing the edit button.\r
\r
\paragraph{How to run AMORE in the test machine:}\r
\r
The EMCal AMORE test machine can be reached via \texttt{ssh\r
  emcal@pcaldbl601}. The standard setup there is identical to the\r
setup at P2. In the home directory there are some symbolic links which\r
need to be changed in order to change the setup and test new\r
features. \texttt{alirootLink} usually links to\r
\texttt{/opt/aliroot-<some-ver>}, which is the version currently at P2\r
(daq team updates the software in the \texttt{/opt} directory when\r
needed). The same holds for \texttt{rootLink}, \texttt{amoreLink} and\r
\texttt{amoreSiteLink} (you should not need to change\r
\texttt{rootLink} and \texttt{amoreLink}). For testing some changes to\r
the EMCAL QA classes, one has to compile an own version of aliroot\r
with the new version of those classes, and then make a symbolic link\r
to the path of this aliroot version. There is an aliroot trunk version\r
which was used to be updated from time to time for tests in \texttt{fblanco/alisoft}. In the directory \texttt{myamoreStuff} there are 2 version of the AMORE modules: \texttt{current\_deploy} and \texttt{trunk} (of course you should do svn up from time to time). Let's suppose you want to make some modification to our AMORE GUI (the expert one) and test them. Remove the \texttt{amoreSiteLink} (it usually points to \texttt{/amoreSite}). Then do:\\\r
\texttt{ln -s myamoreStuff/amoreSite amoreSiteLink}\\\r
(or any other directory you would like to use). Then:\\\r
\texttt{cd trunk[current\_deploy]/amoreEMC}\\\r
At this point you can modify either the \texttt{src/ui/EMCUIQA} or \texttt{src/ui/EMCUIQATRU} class. This class contains just some manipulations of canvas/histograms to be shown in the expert panel and should be easy to understand. If there are some doubts, just ask. The first one also contains the hack we use in order to use our own reference file at P2 in used to calculate the ratio to reference (next section will explain how to create a new reference file). \texttt{make install} will install the modified libraries into \texttt{amoreSiteLink}. You can use some of the scripts in the \texttt{fblanco} directory to run the agent and the GUI.\r
In order to run the expert agent with the expert GUI you should do:\\\r
\texttt{amoreAgent -a EMCQA -s <some-raw-data>\footnote{If you want to create a raw data file, you should first download a chunk of raw data from alien. Then do:\\ \texttt{deroot file.root file.raw}.\\ Keep in mind that the test machine is shared with other detectors, so avoid to store a lot of raw data there and clean up the space from time to time.} -g emcal\_amore.cfg}\\\r
After doing ssh to the test machine in another terminal, you do \\\r
\texttt{amore -d EMC -m EMCUIQA(TRU) -g configGUI.txt}\\\r
If you want to test the shifter agent, you simply do:\\\r
\texttt{amoreAgent -a EMCQAshifter -s <some-raw-data>  -g emcal\_amore.cfg}\\\r
and type \texttt{amoreGui} in another terminal window.\\\r
You can commit changes to the trunk (and not to the current\_deploy) with the usual \texttt{svn commit}. Once you feel that your changes are ready to be deployed at P2, send an email with a request to date-support.\r
\r
\r
\r
\paragraph{How to create a new reference file:}\r
\r
In order to create a new reference file, you have to use the \texttt{doReco.sh} script. The only thing you should do there is the path to the raw data in alien and the number of chunks to analyze (check it in the alien path).\r
Remember to do \texttt{alien-token-init} and \texttt{source /tmp/gclient\_env\_\$UID} before running the script.\r
After the script is executed you will have some directories (chunck10, chunk11...). Each of them contains an \texttt{EMCAL.QA.<RunNumber>.root} file. You can do :\\\r
\texttt{hadd EMCAL.QA.0.root chunck10/EMCAL.QA.<RunNumber>.root \newline chunck11/EMCAL.QA.<RunNumber>.root}\\\r
to merge them (the output files should always be called \texttt{EMCAL.QA.0.root}).\r
At this point the output file can be already copied to the aldaqacr51 machine in the emcal station, in which we can change the QARef file whenever we want.\r
In order to do that, you should copy it to your lxplus area, then from the \texttt{\textasciitilde/amore/QARef} directory in the aldaqacr51 you can do:\\\r
\texttt{scp your-afs-account@aldaqgw01:/afs/cern.ch/<path-to-file>/<file> .}\\\r
Do:\\\r
\texttt{ln -s new-file QA.Ref.root}\\\r
and add a note to the notes file in the directory.\r
Then you have to create an OCDB file. In order to do that, you must do\\ \r
\texttt{aliroot Save2OCDB.C}\\\r
Standard parameters of the macro are ``EMCAL'' (detector name), 0 (run number) and ``12'' (year). You may need to change only the last one. The macro will create a directory named \texttt{QARef/EMCAL/QA/Calib/}, and the file \texttt{Run0\_999999999\_v0\_s0.root} in it. This file has to be committed to the \texttt{QARef/EMCAL/QA/Calib/} directory of aliroot. You can check it with the \texttt{checkCDB.C} macro (it just displays a couple of histograms).\r
%\subsection{}\r
\r
\subsubsection{Some more informations}\r
Further details about AMORE can be found here:\\\r
\href{https://ph-dep-aid.web.cern.ch/ph-dep-aid/amore/}{https://ph-dep-aid.web.cern.ch/ph-dep-aid/amore/}\\\r
The Twiki page with the general EMCal informations for the DQM shifter is:\\\r
\href{https://twiki.cern.ch/twiki/bin/viewauth/ALICE/EVEEMC}{https://twiki.cern.ch/twiki/bin/viewauth/ALICE/EVEEMC}\r
The Twiki page (DQM blackboard) with temporary informations for the DQM shifter is:\\\r
\href{https://twiki.cern.ch/twiki/bin/viewauth/ALICE/DQMBlackboard}{https://twiki.cern.ch/twiki/bin/viewauth/ALICE/DQMBlackboard}\r
\r
\subsection{Offline - Marie}\r
Analysis code, what we control, how\r
\r
\subsection{Event display}\r
\r
\subsection{Logbook tips}