Re: No-vxworks DAQ and Linux interrupt latency (fwd)

[Elton Smith <elton@jlab.org>]

---------- Forwarded message ----------
Date: Thu, 27 Apr 2000 14:39:45 -0500 (CDT)
From: Don Holmgren <djholm@fnal.gov>
To: Elliott Wolin <wolin@jlab.org>
Cc: Don Petravick <petravick@fnal.gov>, Vardan Gurjyan <gurjyan@jlab.org>,
     elton@jlab.org
Subject: Re: No-vxworks DAQ and Linux interrupt latency


Hi Elliot -

It has been a long time, indeed - since the last Real-Time conference I
think.

What we were doing was speeding up the timer tick rate to 1 KHz from 100
Hz in order to lower the latency on scheduling a job which was ready to
run following reciept of data via ATM.  We were only interested in
dropping that latency to 1 msec from 10 msec.  For this particular
application interrupt latency isn't important to us.

The RT Linux port to PPC has been out for a few months.  Victor Yodaiken
started a new company called fmslabs, and details are available there
(www.fmslabs.com).  The last interrupt latency figures I heard were from
his class at RT-99 - they were something like 10 usec (a little sub-10, I
think) with a few usec jitter on Pentium II class machines.  I've not
heard of figures for PPC boards, but he'd very likely answer your question
(someone on the mailing list surely would).  I don't know what limitations
exist; certainly PCI contributes. 

I'm doing DAQ on CDMS (Cryogenic Dark Matter Search) with MVME-2307's
running Linux.  Not RT Linux, just plain Linux - I'm only reading
digitizers and history buffers, and 1 msec latency in response to the
trigger is fine.

The MiniBooNE experiment is also going to use Linux on '230x or '260x
boards.  I've helped them get Linux going on their systems, but so far
they've not asked me about reading out data across the VME bus (and I
suspect they haven't started yet).  Also, I don't know their requirements. 
I've talked mostly to Ben Sapp (bsapp@nua.lampf.lanl.gov) and Eric Church
(church@nuebar.lampf.lanl.gov). 

Getting a '230x or '260x board running Linux is fairly straightforward -
not documented, but not bad (I'm always happy to help).  I believe the RT
Linux layer can be installed simply as a kernel module, so perhaps kernel
patches aren't necessary.  Measuring interrupt latency shouldn't be too
bad - maybe a two or three week exercise from beginning to end (of course,
who has two or three weeks free nowadays?). 

I've never been totally thrilled by the RT Linux approach.  They have an
executive "above" the Linux kernel which handles all interrupts and
strives to never disable interrupts; those destined for the kernel are
passed on, and the kernel runs when the RT layer isn't busy.  The
facilities in the RT layer are a bit primitive - no memory protection
amongst RT tasks, and special message queues used to communicate with
normal Linux user space.  My biggest attraction to Linux over VxWorks on
these boards is memory protection, so I'm not happy with the first
property. Also, the biggest contributors to interrupt latency in the
normal kernel are slow interrupt handlers; the solution is just not to do
that sort of I/O.  There are adequate real time features in the scheduler,
as well as scheduler patches, to setup process priorities in similar
detail to that available with VxWorks. 

Of course it's easy for me to spout off all of this "philosophy" - but,
I've not done any detailed measurements.  

I'm also starting to play with using a 617 to move data to a commodity PC
in lieu of a embedded PPC board.  Trouble is, a 617 card pair is about the
same cost as a '2307 and I'm guessing the performance may be similar
(well, at least limited by my digitizers in both approaches).  However,
it's certainly easier to scale processor horsepower on an external
commidity box than on an embedded computer. 

Don Holmgren



On Thu, 27 Apr 2000, Elliott Wolin wrote:

> Dear Dons,
> 
> Haven't seen you folks in a while...perhaps we'll meet at the next Real-Time
> conference, if they feel rich enough here to send me.
> 
> Anyway, I've been thinking about no-vxworks DAQ systems. 
> 
> We've had success reading vme modules into a linux box via a SBS-Bit3 model 617
> pci-vme bridge.  Could I eliminate the ppc in the vme crate and use the 617
> instead? Problem is interrupt latency.  
> 
> Alternatively, I could use vme dual-port memories with built-in dma engines and
> sequencers to move the data out of the front ends into the dual-ports, then read
> out the dual-ports at my leisure.
> 
> I seem to recall you modified the linux kernel to speed up interrupts...what's
> the latest on this?  Is real-time linux anywhere near available?   What is it's
> interrupt latency?  Is it limited by pci?
> 
> Anyone doing daq this way you know of?
> 
> I'm mainly thinking about this for the proposed Hall D experiment (exotic meson
> production search...i.e. very high precision search for gluonic excitations
> using fancy partial wave analysis) at JLab...at least a few years off.  But also
> for upgrades to existing detectors...might be able to eliminate vxworks and have
> no cpu's locked away in radiation areas.
> 
> Any ideas, leads, names, etc. would be appreciated.
> 
> 
> 				Sincerely,
> 					Elliott