

\section{Data acquisition}
\label{rwmacleod:sec1}

\subsection{Overview}

The data acquisition goal for \hd\ is
to accept the level 1 trigger rate
without incurring any DAQ system deadtime.
The high rate  of 70-180\khz\ drives the design of the trigger, the front-end electronics,
and the DAQ system. Both the \fadc\ and the \tdc\ digitizing modules
must store data for several microsecond
so that the level 1 trigger can be processed. 
When the level 1 trigger is satisfied
these devices must be read-out without deadtime.
When the level 1 trigger is asserted, a time slice of each ring buffer will be
copied, compressed and stored.
Events will be buffered into groups of at least 64 on each electronics board
and then transferred
first across a backplane to be built into crate event fragments
and then to a computer farm to be built into complete events.
The farm will perform a quick analysis
to reduce the event rate by approximately a factor of ten before recording to
magnetic media.
This design allows \hd\ to start running with a modest tagged photon rate
and then to scale-up by an order of magnitude.

The DAQ system will be an important part of the whole \hd\ data analysis
system.
The rates and complexity of the DAQ system for \hd\ are challenging,
nevertheless we will provide a flexible
system that will allow different modes of read-out:
\begin{itemize}
\item production physics events,
\item full \fadc\ and \tdc\ time window events, and
\item several microseconds streams of raw \fadc\ and \tdc\ data for the whole
detector.
\end{itemize}
Additionally, since demand for access to a DAQ system
during accelerator down-time is very high,
we propose that the system should
allow independent trigger, control and read-out of sub-systems
so that effective use of this time can be made.
Each of the components of the data acquisition system will be discussed below.

\subsection{Data flow}


%      \item   Digitization & Trigger

In \hd\
there are approximately 16,000 \fadc\  channels.
If we assume a typical occupancy of 2\%,
a 250 MHz, 8 bit \fadc,
a time window of 100 nanoseconds,
and that the full time window is read-out,
then the total amount of \fadc\ data  would be:
 16000  channels * 
0.02 (\% occupancy) * 
25 bytes/channel $=$ 8Kbytes per event.

The 25 bytes/\fadc\ channel is used to get an energy and a time signal.
Previous work \cite{vavra}
%% (ref J. Va'Vra: High Resolution Drift Chambers NIM A244, pp 391-415)
\cite{cussans}
%% (ref D.G. Cussans: Optimization of the pulse arrival time determination in the ZEUS central tracking detector FADC system NIM A362, pp 277-282)
indicates that a time resolution better than the \fadc\ sampling interval can be achieved by fitting the \fadc\ waveform.  We are studying algorithms for determining energy from a \fadc\ waveform.  Figure ~\ref{fig:FEU84-3_250_8}.
\begin{figure}
  \centerline{\epsfig{file=ch8_FEU84-3_250_8.eps,height=2.0in,width=2.0in}}
  \caption{Resolution as a function of energy for simulated FEU84-3 PMT sampled at 8 bits, 250 MHz. LGD resolution is shown by dotted line. FADC samples are summed to determine energy.}
  \label{fig:FEU84-3_250_8}
\end{figure}
shows results obtained for a simulation in which a PMT of the type used in the forward calorimeter was modeled and digitized by an ideal \fadc.  The energy was determined by simply summing the \fadc\ samples.

Reduction of the \fadc\ data will be performed in real-time.
The result will be an energy, a time, and a channel identifier.
This would allow us to reduce the amount of data per hit
to 10 bytes per \fadc\ channel thereby lowering the total \fadc\ data 
to 3.2 Kbytes per event.
The computational resources required to achieve this reduced data rate are
currently being studied.  It may be possible to build this processing into the gate array, or to include digital signal processors on the \fadc\ boards.


In \hd\
there are approximately 8,000 \tdc\ channels.
The data volume for the \tdc s would be:
 8000  channels * 
0.02 (\% occupancy) * 
4 bytes/channel *
 $=$ 640 bytes per event. The total event size would be about 4 Kbytes; we assume 5 Kbytes per event for the DAQ design.

Once the event fragments have been selected from a stream in each channel and reduced,
they must be built into complete events and
pass a filter that will reject accidentals and low energy events.
Board level event fragments will be read out over a VME backplane and
crate level event fragments will then be assembled
and sent to the level 3 trigger farm.
One important detail of the DAQ design is that the boards would have
to buffer 64 (or more) events locally
before transferring this group of event fragments
to the crate computer. This is necessary because the tasks that the
crate computer is required to do
can't be done today at a 100\khz\ interrupt rate
with a commercial real-time operating system.
At 64 events per buffer,
we will have to transfer about 23.5 KB in less than 640 microseconds.
This transfer speed (36.7 MB/sec)
is achievable with current VME technology. Depending on the CPU power
available, the crate CPU may reorganize
the crate event fragments so all data from one event is contiguous.


Using current technology this data transfer rate could be handled using either
gigabit ethernet or ATM OC-12 network links.
Regardless of the network hardware, significant CPU resources will be required at the
crate level to transmit the data. A fast switch with a backplane bandwidth
greater than 672 MB/second and more than 40 ports will also be required. 
This switch technology is under development
by many networking companies\footnote{One product from Cisco 
is described at http://www.cisco.com/warp/public/146/1971.html}
and in a few years it should be common and less expensive.

% FNAL model of event filter.
The architecture of the CPU farm 
which performs event building, filtering and recording is
presented below.
%  (design is stolen from FNAL)
Event building and recording will
be done in parallel on 10-20 computers.
Each of these systems would send events to a sub-farm for filtering.
The sub-farm nodes would send a pass/fail answer for each event received
and the master node would record all events that passed the filter.
It is possible that further information could be added to the event or that
the events selected by the filter could even be reformatted before being sent to the 
recording computer(s).

\subsection{Level 3 trigger}
The goal of the level 3 trigger is to reduce the event rate given by
the level 1 trigger to an acceptable rate to tape.  In low intensity
running ($10^7$ tagged photons/s) the level 1 trigger rate is
expected to be 15\khz.  Since the DAQ system is being designed to
handle this rate to tape, the level 3 trigger farm will not have to cut any
events, although it may be used to reduce the event rate somewhat.  In
high intensity mode where the level 1 rate may be as high as 180
\khz, the level 3 trigger must be able to reduce the event rate by a
factor of ten.

Most of these unwanted events result from an untagged (mostly lower
energy) photon interacting in coincidence with a tagged photon.
Rejecting these events means that level 3 must be able to calculate,
with reasonable accuracy, the energy of the photon which produced the
event.  This involves accurately reconstructing tracks, matching them
with the calorimeters, and adding additional energy deposited by
neutral particles in the calorimeters.

Because of the accuracy requirements and the demands of linking
information from different detectors, we have decided to use a
processor farm architecture for level 3 instead of building a
dedicated hardware processor.  All events passing the
level 1 trigger will be read into the level 3 processor farm where
they will be reconstructed; events passing the cuts applied will then
be written to tape.  This approach allows for algorithmic flexibility and
improvements, and the ability to  adjust cost-effectively to higher rates,
but it does put pressure on the DAQ system.

An estimate of the processing power required can be made as
follows.  The Hall-B full event analysis, 
with momentum resolution of less than 1\%, requires
approximately 45 milliseconds on a 21 SPECint processor (Pentium III - 500).
We believe that
\hd\ 
will be similar and will require a similar amount of processing power.  To
perform the level 3 trigger analysis we think that 1/10 the number of instructions
will be adequate because unlike the 1\% precision required in a full
reconstruction
we are aiming for only 2-5\% precision.  
In fact Hall-B has such a pattern recognition program in development; it
is able to achieve 10\% momentum resolution using only 3\% as much 
processing power.
Therefore  we believe that approximately 5 milliseconds on a 21 SPECint
processor will be required for the level 3 analysis of each event.
This yields a figure of 0.105 SPECints/ev/s as a good estimate
of the processing power required.
This means an aggregate processing power of 18,900 SPECints is required for
the level 3 analysis.  Allowing for 50\% processor utilization (due to
I/O overhead etc.) this means that a farm composed of 200 computers each
capable of 200 SPECints could handle the I/O and perform the level 3 analysis
for the high intensity configuration.

 Table \ref{tab:l2trig} shows the rates, sizes, and processing requirements
 for the level 3 trigger.

\begin{table}
  \begin{center}
    \begin{tabular}{|l|c|c|}
\hline \hline
                       & Low Rate       & High Rate \\
\hline \hline
Event Size              & 5 KB          & 5 KB \\ \hline
Event Rate to Farm      & 10\khz        & 180\khz \\ \hline
Data Rate to Farm       & 50 Mbytes/s   & 900 Mbytes/s \\ \hline
Num Links to Farm       & 20            & 20 \\ \hline
Data Rate per Link      & 2.5 Mbytes/s  & 45 Mbytes/s \\ \hline
Link Technology         & 100 Megabit Ethernet  & Gigabit Ethernet \\ \hline
Events/s per Link       & 500           & 9000 \\ \hline
SPECints/ev/s for L2    & 0.105         & 0.105 \\ \hline
Num SPECints/link       & 52.5 SPECints & 945 SPECints \\ \hline
Num SPECints/link x 2   & 105 SPECints  & 1890 SPECints \\ \hline
Num 200 SPECint         & 1             & 10 \\
processors/link         &               & \\ \hline
Total Num 200           & 20            & 200 \\
SPECint processors      &               & \\ \hline \hline
\end{tabular}

\caption{Rates, sizes, and processing requirements for the level 3 trigger.}
\label{tab:l2trig}
\end{center}
\end{table}

\subsection{Monitoring and control}

Monitoring and control tasks include hardware
configuration and control (``slow controls''), bookkeeping, online event monitoring, 
alarm systems,and messaging systems.
These are less demanding tasks than data
acquisition in \hd, and should not present unusual challenges.
We plan to follow some examples from Hall B, but to also make use of lessons
learned there. In particular, we plan to integrate offline data analysis
tools with the online software at the outset to reduce the total cost of software
development. 

%
% This not only avoids a
% large duplication of effort, but it also makes it seamless to take software
% developed on one data set and apply it directly to future runs.

 The number of events that can be processed online is largely
determined by the number of processors available.  In \hd\ a monitoring
component can be added to the main event filtering code to
gather statistics about both the event that is being written to tape
and those being discarded by the level 3 farm.
The choice of the type and number of
processors can wait until soon before the system is installed, when prices
are lower and performance is better known.


The framework for slow controls should be uniform for all subsystems
in \hd, but the choice is not obvious. VME-based {\sc epics} works in Hall B,
but does not mesh well with the online requirements and has proven
to be manpower intensive.  In fact, a number of Hall B systems do not use
{\sc epics} or VME, but instead resort to {\sc camac} or other options.
Other software possibilities include message-based systems and hardware
alternatives include PC or mixed hardware systems.  We believe that an open,
message-based system that implements a uniform interface to diverse hardware
is probably best.  Commercial products are currently available that implement
this at moderate cost.

Bookkeeping tasks include all recordable activities of the experiment
other than raw and calibration data.
We expect this will be done using relational and/or object-oriented
databases.  Current commercial and public domain database technology
should be adequate.

The alarm and messaging framework allows sub-systems to communicate
their state to monitoring programs and operators.
This system needs to be integrated across the entire
online, DAQ, and database systems in a simple, uniform manner.
The scale and performance requirements of this system are modest, and
similar to other projects in progress or completed at Jefferson lab.

%\subsection{Cost estimates}

%\begin{table}
%  \begin{center}
%    \begin{tabular}{|l|r|r|r|}
%      \hline
%Item            &   Unit cost              &  Quantity &    Total \\
%\hline \hline
%SBC CPU         &    3 \ \ \ \ \ \ \        &    35     &  105\\ \hline
%Main Switch     &   20 \ \ \ \ \ \ \        &     2     &   40 \\ \hline
%Sub-Farm Switch &    4 \ \ \ \ \ \ \        &    10     &   40 \\ \hline
%DAQ Sun SMP     &   20 \ \ \ \ \ \ \        &    10     &  200 \\ \hline
%DAQ Field CPU   &    2 \ \ \ \ \ \ \        &   200     &  400 \\ \hline
%Misc PC/Sun     &    3 \ \ \ \ \ \ \        &    10     &   30 \\ \hline
%DAQ Raid Disk   &   40 \ \ \ \ \ \ \        &     4     &  160 \\ \hline
%CC Raid Disk    &   40 \ \ \ \ \ \ \        &     4     &  160 \\ \hline
%DAQ Tape Drive  &  100 \ \ \ \ \ \ \        &     6     &  600 \\ \hline
%CC computer     &   20 \ \ \ \ \ \ \        &     2     &   40 \\ \hline
%CC Tape Silo    &  300 \ \ \ \ \ \ \       &     2?     &  600 \\ \hline
%Analysis Farm   &   2  \ \ \ \ \ \ \        &   250     &  500 \\ \hline
%Tape Media/Year &   0.1K/TB \ \ \ \ \ \ \   &   15000   & 1500 \\ \hline
%Rack/Desks/Misc &   30 \ \ \ \ \ \ \       &      1     &   30 \\
%\hline \hline
%{\bf Total}       &                        &            & 4405 \\ \hline
%\end{tabular}
%\caption[Estimated cost of the \hd\ Data Acquisition System.]
%{Estimated cost of the \hd\ Data Acquisition System. Costs are listed
%in thousands of 1999 US dollars. The abbreviations ``SBC'' and ``CC'' refer to
%``single board computer'' and ``computer center'', respectively.}
%\label{tab:daqcost}
%\end{center}
%\end{table}

%A significant cost savings can be realized if we recycle tapes after some
%period of time.
