[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[GlueX] First Hall D online software! (fwd)
---------- Forwarded message ----------
Date: Wed, 09 Feb 2005 13:32:33 -0500
From: Elliott Wolin <wolin@jlab.org>
To: IU Hall D Listserver <halld-collab@dustbunny.physics.indiana.edu>
Cc: Vardan Gurjyan <gurjyan@jlab.org>, Ed Jastrzembski <jastrzem@jlab.org>,
carl Timmer <timmer@jlab.org>, Dave Abbott <abbottd@jlab.org>
Subject: [GlueX] First Hall D online software!
To all interested in Hall D online software,
I present the first software specifically designed and written with the Hall D
DAQ/online/controls systems in mind. cMsg is an interprocess communication package that
includes an asynchronous publish/subscribe component and a synchronous peer-to-peer
component. Utilities include a network-accessible queuing package and a generic message
logging system. cMsg includes gateways to many other ipc systems, including EPICS channel
access. It is highly customizable and extendable.
cMsg is a product of the JLab Data Acquisition Group, of which I am a member. It was
first conceived with DAQ group needs in mind, but was expanded so that I feel it is now a
firm base upon which all Hall D online software can be constructed. It is a standalone
package and freely available (or, it will be soon!), and a draft of the user's guide is
attached.
For those familiar with the CLAS online, cMsg is similar to the SmartSockets commercial
ipc package that much of the CLAS online is based on, but somewhat expanded.
A generic publish/subscribe package can be used in many places other than
DAQ/online/controls systems, as is shown by the widespread use of publish/subscribe in the
commercial world. In particular, I think cMsg could play an important role in the Hall D
offline as well as in the online.
cMsg is in a beta release stage, and all comments and suggestions for improvement are welcome.
Sincerely,
Elliott
================================================================================
Those raised in a morally relative or neutral environment will hold
no truths to be self-evident.
Elliott Wolin
12000 Jefferson Ave MS 6B
Newport News, VA 23606
757-269-7365
================================================================================
to do:
cMsgException error codes
cMsg User's Guide
-----------------
E.Wolin, C.Timmer
7-Feb-2005
1. Introduction
----------------
The cMsg package is designed to provide client programs with a uniform
interface to an underlying messaging system via an API powerful enough
to encompass asynchronous publish/subscribe and synchronous
peer-to-peer messaging. The advantage of using the cMsg API is that
client programs need not change if the underlying messaging system is
modified or replaced.
But cMsg provides much more than a simple API. The package includes a
number of built-in messaging systems, including a complete,
stand-alone, asynchronous publish/subscribe and synchronous
peer-to-peer messaging system, as well as a persistent network queuing
system. Although cMsg is highly customizable and extendable, most
users will simply use one of the built-in messaging systems. In
addition, a number of useful utilites and examples are provided.
If you are familiar with the publish/subscribe and peer-to-peer
paradigms and want to jump right in and learn how to use the cMsg
package, skip to the tutorial section or look at the example programs,
and refer back to the next few sections as needed.
The cMsg package can be used at a number of levels, i.e. as an:
a. Abstract API to an arbitrary, underlying message system
b. Framework for implementation of underlying messaging systems
c. Proxy system to connect to remote messaging systems
d. Full implementation of publish/subscribe and other messaging systems
The first three levels are primarily of interest to cMsg developers
(see the cMsg Deveolper's Guide). In the rest of this document we
describe how to use the built-in messaging systems.
Asynchronous publish/subscribe messaging
----------------------------------------
The publish/subscribe messaging paradigm is quite powerful, and is
commonly used in business and industry. Briefly, in a
publish/subscribe system producers publish messages to abstract
subjects, and consumers subscribe to subjects they are interested in.
A message published to a subject may be delivered to any number of
consumers (one-to-many messaging). A client can be both a producer
and consumer of messages.
Producers do not know nor care if any clients are subscribed to the
subjects they publish to, and consumers neither know nor care about
the producers who publish to the subjects they subscribe to.
Note that in the cMsg publish/subscribe messaging system an additional
message field, "type", plays a similar role as the subject field
described above.
Synchronous peer-to-peer messaging
----------------------------------
Publish/subscribe messaging is asynchronous in that neither producer
nor consumer know about each other's existence. The cMsg API also
includes a number of synchronous messaging mechanisms, in which there
is some level of direct communication between the producer and
consumer of a message. These range from simple status notifications,
which may provide only partial information concerning the status of
the producer/consumer transaction, to direct, end-to-end communication
between a single producer and a single consumer.
Implementation
--------------
Currently cMsg clients can be written in Java (1.5 or higher) on Unix
and Windows, and in C on Unix and VXWorks. We provide Javadocs and
Doxygen docs to document the full Java and C client API. Note for
developers: cMsg domains can be written in C or Java, but subdomains
only in Java.
2. Messaging Basics
-------------------
cMsg messages are containers that hold a number of user and system
settable fields. Clients typically use the subject, type, and text
fields, but there are additional user settable fields (see below).
The cMsg system takes care of setting many other fields, including
creator, sender, senderHost, senderTime, receiver, receiverHost,
receiverTime, etc. See the API docs for a complete list of system
fields.
After message creation producers must set the message subject, type,
and text fields (strings, with empty strings allowed), and can
optionally set the userInt (integer), and userTime (Unix time) fields.
After the fields are set the message can be published. A message can
be published many times, and user fields can be modified in between as
desired.
The subject and type fields determine how messages are delivered.
Consumers can subscribe to any number of subject/type combinations,
and must provide a callback for each (callbacks need not be unique).
The wildcard characters "*" and "?" are allowed in subscriptions,
where * matches any number of characters, and ? matches a single
character.
The start() method enables delivery of messages to the callbacks, and
the stop() method stops delivery. If delivery is not enabled messages
will be lost to the client. A single message may be delivered to many
callbacks in a single client. Note that if a client subscribes to a
subject and then publishes to the same subject, it will receive the
message it published.
Finally, messages can be sent or received using one of the synchronous
mechanisms provided in the cMsg API. See the tutorial for more
details.
3. Domains and Universal Domain Locators
-----------------------------------------
Fundamental to cMsg is the notion of a domain, or messaging space.
Clients can connect to one or more domains, but subscriptions and
messaging activity generally are specific to individual domains.
I.e. a publication or subscription in one domain normally has no
effect in any other domain. Note that although inter-domain
communication in general is not supported, the cMsg gateway utility
can bridge domains under certain circumstances. Also, user-written
domains may implement inter-domain communication.
Domains are specified via a Universal Domain Locator, or UDL. When
connecting to a domain, clients supply three pieces of information:
the client name, which often needs to be unique within the domain; a
short client description; and the UDL. In analogy with http and other
resource locators, the general form of a UDL is:
cMsg:domainName://domainInfo?dpar1=val1&dpar2=val2...
where the leading cMsg (optional) refers to the cMsg package,
domainName identifies the domain, and domainInfo is interpreted by the
domain. The optional parameters are domain-specific, and any number
of them may be specified.
4. Default Domain Implementations
----------------------------------
A number of domains are implemented by default in the cMsg package.
The most important is the cMsg domain, which implements a full
publish/subscribe and peer-to-peer messaging system in one of its
subdomains (see below). Note that calls to API functions not
supported by a given domain return a "Not Implemented" error.
Developers can easily implement additional Java and/or C domains.
File domain
-----------
The File domain implements simple logging of messages to a local file,
and the form of the udl is:
cMsg:File://fileName?textOnly=booleanVal
where fileName specifies the name of a file locally accessible to the
client, and the file is opened in append mode. By default the entire
message is logged to the file as an XML fragment. If textOnly=true is
specified only a timestamp and the text field is logged. Only the
send() and syncSend() messaging API functions are supported. Currently
the File domain is only implemented for Java clients. Client names
need not be unique in this domain.
CA domain
---------
The CA domain implements a simple interface to EPICS Channel Access,
similar to the EZCA package, and the form of the udl is:
cMsg:CA://channelName?addr_list=list
where addr_list specifies the UDP broadcast address list. Supported
messaging API functions are send(), which implements a CA put;
flush(); subscribeAndGet() which gets the channel value one time and
the timeout is in msec; subscribe(), which implements CA "monitor on";
and unsubscribe, which implements "monitor off". Callbacks are
executed in their own threads.
Currently access is only supported for dbl values, and the dbl resides
in the text field as a string. Also, the CA domain is only
implemented for Java clients. Client names need not be unique in this
domain.
cMsg domain
-----------
The cMsg domain involves use of a proxy server whereby client requests
are not handled in the client itself, but instead are forwarded to a
remote cMsg server that actually performs the work on behalf of the
client. Communication between client and server is handled
transparently so the user does not know that a server is involved.
Thus, for example, underlying messaging systems that are not available
on VXWorks can be accessed from VXWorks via a server running on a
remote Unix node. Such services are called cMsg subdomains, and the
next section describes a number of them. Developers can easily
implement more.
The general form of the cMsg domain UDL is:
cMsg:cMsg://host:port/subdomain/subdomainInfo?spar1=val1&spar2=val2...
where host and optional port identify a cMsg domain server, subdomain
specifies a particular subdomain implementation, and subdomainInfo is
interpreted by the subdomain, as are the subdomain parameters.
Callbacks in the cMsg domain are run in their own threads, so be
careful with static data. Also, messages arriving for different
callbacks are queued separately. Thus a "high priority" subject/type
combination with its own callback will be promptly run even if large
numbers of messages arrive for other callbacks. The queues can be
configured to parallelize message processing in multiple threads. See
the API docs for details.
5. Starting the cMsg Domain Server
-----------------------------------
You must run a cMsg server if you want to connect to the cMsg domain
and its subdomains. The server is not needed for the other domains.
The cMsg domain server requires Java 1.5 or later:
$ java org.jlab.coda.cMsg.cMsgDomain.cMsgNameServer -h
Usage: java cMsgNameServer [-p <listening port>] [-d(ebug) <level>]
-p port number from 1024 to 65535
-d
-debug level of output with acceptable values of:
info for full output
warn for severity of warning or greater
error for severity of error or greater
severe for severity of "cannot go on"
none for no debug output (default)
The default port is 3456. Be sure to specify the port number in the
UDL in your client code if you do not use the default port.
Multiple cMsg servers on the same node must use different ports.
6. Default Subdomain Implementations
-------------------------------------
A number of subdomains are implemented in the cMsg server, including
the cMsg subdomain, which implements a complete asynchronous
publish/subcribe and synchronous peer-to-peer system (see below).
Developers can easily implement additional subdomains in Java 1.5 or
higher.
LogFile subdomain
-----------------
The LogFile subdomain is similar to the File domain, except that the
cMsg server performs the logging. Thus multiple clients can log to
the same file, whereas in the File domain the file is unique to the
client. Only the send() and syncSend() messaging functions are
supported.
The general form of the LogFile subdomain UDL is:
cMsg:cMsg://host:port/LogFile/fileName
where fileName is the name of the logging file used by the server.
Messages are logged as XML fragments. Client names need not be unique
in this subdomain.
CA subdomain
------------
The CA subdomain is similar to the CA domain, but again the cMsg
server actually executes the CA libarary commands, not the client.
Only the send(), syncSend(), subscribe(), unsubscribe(), and
subscribeAndGet() messaging functions are supported.
The general form of the CA subdomain UDL is:
cMsg:cMsg://host:port/CA/channelName?addr_list=list
where addr_list specifies the UDP broadcast address list. Client
names need not be unique in this subdomain.
Database subdomain
------------------
In the Database subdomain the cMsg server connects to a database and
executes the SQL statement appearing in the text field of a message.
Currently SQL queries that return data are not supported
(e.g. select). Only the send() and syncSend() messaging functions are
supported.
The general form of the Database subdomain UDL is:
cMsg:cMsg://host:port/Database?driver=myDriver&url=myURL&account=myAccount&password=myPassword
where myDriver is the JDBC driver to use to connect to the database,
myURL is the JDBC URL of the database, and optional myAccount and
myPassword may be required by the database. Client names need not be
unique in this subdomain.
Queue subdomain
---------------
The Queue subdomain implements network-accessible persistent message
queues via a JDBC-accessible database. Clients can post messages to
the queue via send() or syncSend(), and retrieve messages from the
head of the queue via sendAndGet().
The general form of the queue subdomain UDL is:
cMsg:cMsg://host:port/Queue/queueName?driver=myDriver&url=myURL&account=myAccount&password=myPassword
where queueName identifies the queue, myDriver is the JDBC driver to
use to connect to the database, myURL is the JDBC URL of the database,
and optional myAccount and myPassword may be required by the database.
Note that tables are created in the database by the cMsg server to
implement the queue. Also, client names need not be unique in this subdomain.
FileQueue subdomain
-------------------
The FileQueue subdomain is identical to the queue subdomain except
that the message queue is stored in files rather than in a table in a
database. This results in lowered performance, but of course no
database is needed. If multiple cMsg servers will access the same
FileQueue then the file system on which the queue files reside must
support Java 1.5-compatible file locking.
Only the send(), syncSend(), and sendAndGet() messaging functions are
supported.
The general form of the FileQueue subdomain UDL is:
cMsg:cMsg://host:port/FileQueue/queueName?dir=myDir
where queueName identifies the queue, and optional myDir specifies the
directory in which to store the queue files (default is the current
working directory of the cMsg server).
Note that many files are created by the cMsg server to implement the
queue, so choose the queue directory carefully. Client names need not
be unique in this subdomain.
SmartSockets subdomain
----------------------
The SmartSockets subdomain provides a gateway to the SmartSockets
(commercial) publish/subscribe interprocess communication package.
Only send(), subscribe(), and unsubscribe() messaging functions are
supported.
The general form of the SmartSockets subdomain UDL is:
cMsg:cMsg://host:port/SmartSockets/projectName
where projectName identifies the SmartSockets project.
Note that since in SmartSockets the messaging system is implemented by
a "cloud" of interconnected servers, inter-domain messaging is
possible if domains connect to SmartSockets servers within the same
cloud. Client names must be unique in this subdomain.
TcpServer subdomain
-------------------
The TcpServer subdomain provides access to tcpsever processes running
on VXWorks or Unix processors (tcpserver is part of the CODA data
acquisition package at JLab). Only the sendAndGet() messaging
function is supported, and communication with the tcpserver is
stateless.
Commands placed in the request message text field are forwarded to the
tcpserver for execution, and the resulting output is returned in the
text field of the response message.
The general form of the tcpserver subdomain UDL is:
cMsg:cMsg://host:port/TcpServer/srvHost:srvPort
where srvHost and srvPort refer to the host name and port where the
tcpserver process is running. Client names need not be unique in this
subdomain.
cMsg subdomain
--------------
The cMsg subdomain implements a complete asynchronous
publish/subscribe and synchronous peer-to-peer interprocess
communication package. All cMsg messaging API functions are
supported. Note that inter-server communication is currently not
supported.
The general form of the cMsg subdomain UDL is:
cMsg:cMsg://host:port/cMsg/namespace
where namespace identifies a messaging namespace. If namespace is not
supplied a default namespace is used.
Aside...why does cMsg appear three times in the UDL? First optionally
to denote the cMsg package, next to specify the cMsg domain, and
finally to specify the cMsg subdomain. Client names must be unique in
the cMsg subdomain.
7. Utilities and Example Programs
----------------------------------
A number of general purpose utility applications are provided. These
are fairly simple programs, and may easily be customized. All require
Java 1.5 or higher.
cMsgLogger
----------
The cMsgLogger logs messages that match subject/type to the screen, a
file, and/or a database. Similar functionality exists in the File
domain, and in the LogFile and Queue subdomains:
$ java org.jlab.coda.cMsg.apps.cMsgLogger -h
Usage:
java cMsgLogger [-name name] [-descr description] [-udl domain] [-subject subject] [-type type]
[-screen] [-file filename]
[-url url] [-table table] [-driver driver] [-account account] [-pwd password]
[-debug] [-verbose]
where udl is required, subject and type default to * and *, and
display to the screen is default. url is a JDBC url, table is the
name of the table to use, and driver,account,password are used when
connecting to the database.
cMsgQueue
---------
The cMsgQueue utilitiy queues messages to either a database or
file-based queuing system, same as the Queue and FileQueue subdomains.
Some differences are that in the Queue and FileQueue subdomains all
messages are queued, whereas here only messages that match
subject/type, specified on the cMsgLogger command line, get queued.
Also, here only the creator and user-settable fields are queued,
whereas all fields are stored in the two subdomains. Finally, in the
subdomains the cMsg server does the work, whereas here the cMsgQueue
application does the work.
Clients retrieve from the queue by executing the sendAndGet() method,
where the message must be sent to getSubject/getType, also specified
on the command line. The response message returned by the
sendAndGet() method is taken off the head of the queue.
To run the cMsgQueue application:
$ java org.jlab.coda.cMsg.apps.cMsgQueue -h
Usage:
java cMsgQueue [-name name] [-descr description] [-udl domain] [-subject subject] [-type type]
[-queue queueName]
[-getSubject getSubject] [-getType getType]
[-dir queueDir] [-base fileBase]
[-url url] [-driver driver] [-account account] [-pwd password] [-table table]
where udl is required, and queueName, subject, and type default to
"default", "*" and "*". name defaults to "cMsgQueue:queueName", and
getSubject and getType default to name and "*".
For file queuing, queueDir specifies the directory to hold the queue
files, and fileBase specifies the base for all file names (default
is "cMsgQueue_queueName_").
For database queuing url and driver must be specified, and account and
password may be required by the database. table defaults to
"cMsgQueue_queueName".
One instance of cMsgQueue can queue to either a file or database, not
both.
cMsgGateway
-----------
The cMsgGateway implements simple inter-domain communication for
domains that support the send() and subscribe() messaging API
functions. The cMsgGateway connects to two domains and subscribes to
the same subject/type combination in each. Messages that match the
subscription criteria in one domain are cross-posted to the other.
Note that a number of message fields get reset when the gateway
forwards or cross-posts the message (senderTime, senderHost, etc.)
Unchanged are the creator, subject, type, text, userInt, and userTime
fields.
$ java org.jlab.coda.cMsg.apps.cMsgGateway -h
Usage:
java cMsgGateway [-subject subject] [-type type]
[-name1 name1] [-udl1 udl1] [-descr1 descr1]
[-name2 name2] [-udl2 udl2] [-descr2 descr2]
[-debug]
where udl1 and udl2 are required, and the remainder are optional.
Subject and type default to * and * (i.e. subscribe to all subjects
and types), and both name1 and name2 default to cMsgGateway.
Example programs
----------------
The cMsg distribution contains a number of Java example programs
in org/jlab/coda/cMsg/apps, and C examples in the base directory.
The utility programs described above, also in the apps directory,
should be useful as examples, as well
8. Client and server shutdown
------------------------------
API calls exist to implement selective shutdown of clients and
servers. The default client shutdown handler causes the client to
exit. Programmers can override this behavior by supplying a custom
handler that, e.g, causes the client to simply disconnect from the
cMsg system rather than exiting. See the API docs for details.
9. Java Tutorial
-----------------
An application can connect to many different domains, and publications
and subscriptions in different domains are independent. To create the
cMsg system object and connect to a domain:
import org.jlab.coda.cMsg.*;
import org.jlab.coda.cMsg.cMsgException;
cMsg cMsgSys; // the cMsg system object
try {
cMsgSys = new cMsg(myUDL,myName,myDescr);
cMsgSys.connect();
} catch (cMsgException e) {
e.printStackTrace();
}
Note that most cMsg calls throw cMsgException, so they must be in try
blocks, as illustrated above. Below I do not include the try blocks
for simplicity.
To create a message and fill a few fields:
cMsgMessage msg = new cMsgMessage();
msg.setSubject(mySubject);
msg.setType(myType);
msg.setUserInt(myUserInt);
msg.setUserTime(new java.util.Date());
msg.setText(myText);
To send a message and flush the outgoing message queue:
cMsgSys.send(msg);
cMsgSys.flush();
Many messages can be sent before flushing the outgoing queue. Note
that internally the system is free to flush the queue at will.
To synchronously send a message:
int status = cMsgSys.syncSend(msg);
if(status != cMsgConstants.ok) {
// something went wrong...
}
where the nature of the failure, if indicated, is domain specific.
To subscribe with callback and user object:
cMsgSys.subscribe(mySubject, myType, myCB, myUserObject);
where the callback class is:
import org.jlab.coda.cMsg.cMsgCallbackAdapter;
class myCB extends cMsgCallbackAdapter {
public void callback(cMsgMessage msg, Object userObject) {
System.out.println("Subject is: " + msg.getSubject());
System.out.println("Type is: " + msg.getType());
System.out.println("Text is: " + msg.getText());
}
}
and the user object may be anything and exists solely for the
programmer's convenience.
In the cMsg domain the received messages are queued and delivered
serially to the callback in the order received. Configuration options
allow for parallelizing callback processing into multiple threads,
discard of messages after a maximum number are waiting to be
processed, etc. See the API docs for details.
To enable message receipt and delivery to callbacks:
cMsgSys.start();
VERY IMPORTANT: no messages will be delivered at all unless the
start() method is called! To disable message receipt use the stop()
method.
The subscribeAndGet() method performs a synchronous one-shot
subscribe:
cMsgMessage m = cMsgSys.subscribeAndGet(mySubject,myType,myTimeout);
if(m==null) {
// no message arrived within timeout
}
where myTimeout is in integer milliseconds. subscribeAndGet()
temporarily subscribes to mySubject/myType (existing subscriptions and
callbacks are unaffected), waits for a matching message to arrive,
then returns the message. If none arrives within the timeout it
returns null.
To synchronously send a message and get a private response from the
receiver:
cMsgMessage response = cMsgSys.sendAndGet(msg,myTimeout);
if(response==null) {
// no message arrived within timeout
}
where myTimeout is in integer milliseconds. When the receiver gets
the message it has to first recognize that this is a synchronous
request, then create the response message via a factory (and NOT via
the usual cMsgMessage constructor). Receiver code might look like:
// ...just got a message via a callback
// ...send a response if it is a synchronous request message
if(msg.isGetRequest()) {
cMsgMessage response = msg.response(); // create special response
response.setSubject(mySubject);
response.setType(myType);
response.setText(myText);
cMsgSys.send(response);
cMsgSys.flush();
}
If two receivers synchronously respond to the message as above the
first one sent gets returned by the sendAndGet() method in the client.
The second response is treated as a normally published message.
To disconnect from the domain:
cMsgSys.disconnect();
After disconnect() all subscriptions and other connection information
are lost.
10. C Tutorial
---------------
_______________________________________________
GlueX mailing list
GlueX@dustbunny.physics.indiana.edu
http://dustbunny.physics.indiana.edu/mailman/listinfo/gluex