Jobs hanging - JEventProcessor::fini() not being called

["Tarbert, Claire" <ctarbert@indiana.edu>]

Hi,

I have a problem with jobs occasionally hanging without giving any
error messages.  I'm using JANA-0.4.9 and release-2009-02-24 of the 
Hall D software to process hddm files produced by bggen.

The job loops over all events in the hddm file but then hangs and 
doesn't finish - they keep running but draw very little CPU.  Here's an 
example of the end of the log file:

>  367.9k events processed  (367.9k events read)  0.0Hz  (avg.: 788.6Hz)
>  367.9k events processed  (367.9k events read)  0.0Hz  (avg.: 786.1Hz)
> Thread 0 hasn't responded in 30 seconds. (run:event=2:367888) Delisting ...
> Caught HUP signal for thread 0x42eefbb0 thread exiting...
> Launching new thread ...
>  369.4k events processed  (369.4k events read)  0.0Hz  (avg.:
> 785.1Hz)     0kHz  (avg.: 786.8Hz)
>  373.9k events processed  (372.3k events read)  3.5kHz  (avg.: 788.9Hz)
>  378.0k events processed  (378.0k events read)  2.8kHz  (avg.: 795.7Hz)
>  381.6k events processed  (381.6k events read)  2.3kHz  (avg.: 800.9Hz)
>  386.8k
>  391.0k events processed  (391.5k events read)  2.7kHz  (avg.: 814.1Hz)
>  393.5k events processed  (393.5k events read)  2.7kHz  (avg.: 819.9Hz)
> No more event sources
> Thread 0x42eefbb0 completed gracefully

There should be some more printed statements after this that are coded
in my processor fini() function.  So the job finishes processing all
the events but doesn't get to the JEventProcessor::fini() function.

Is there somewhere between "No more event sources" and 
JEventProcessor::fini() that it could get hung up?

This only happens occasionally - I have been able to analyse the same 
files with the same code and the processes finish cleanly.  The problem 
only seems to arise when I have mutliple jobs running at the same time 
on the cluster at IU.

Cheers
Claire

--
Claire Tarbert
260 Swain Hall West               Phone: +1 812 855 8933
Department of Physics             E-mail: ctarbert@indiana.edu
Indiana University
727 E Third St
Bloomington
IN 47405-7105