Active Monitor and Sausalito
---------------------
Tim Hockin <thockin@cobalt.com>
Jun, 2000

1. Overview

	Active Monitor can, in the Sausalito environment, become a simple
	harness for module-provided monitoring routines.  This utilization of
	the Sausalito event model allows the core of the work to be common,
	with the details tailored to each module's needs.

	The Active Monitor (am) process itself (also called swatch) is reduced 
	to an engine that reads module config information, processes tests, 
	and records status information.  When status information changes, a 
	SET event occurs and CCE dispatches handlers registered on that event.

2. Configuration

	The am itself has some configuration options, which are stored in
	a configuration file.  The config file is written by handlers on an
	ActiveMonitor object in the odb.  The default conf file is 
	/etc/swatch.conf.  

	Monitored entities can have one of three states:  GREEN, YELLOW, or
	RED.  These signify that status is good, trouble, and severe
	trouble, respectively.

	Each monitored entity has a config file, which is placed in the 
	directory specified by the config_root directive in the swatch.conf 
	file.  The default config_root is /usr/sausalito/swatch/conf.  These 
	files are generated by handlers on namespaces on the ActiveMonitor 
	object in the odb.  

	All config file directives have the following syntax:
		property = value
	The first word on the line becomes the property name.  All text
	after the '=' will become the property value, excluding leading and
	trailing whitespace/newlines.  Config directives are not ordinal,
	and may be specified on any line within the file.  In the case of a
	duplicate key, the second key will override the first.  Blank lines
	are ignored.  Lines on which the first non-whitespace character is a
	pound (or hash : '#') will be ignored as comments.

	Modules can provide all their default values in their schema (see 
	Schema.txt).  A module monitor item namespace must, at minimum, have 
	the following fields defined, for processing by swatch:

		fieldname          typedef use
		---------          ------- ---
		enabled            boolean flag for am process
		currentState       amstate current state of the service
		lastChange         uint    time_t of last currentState change
		lastRun            uint	time_t of last test run
		currentMessag      scalar  current message for state 
		nameTag            scalar  an identifier for UI
		URL           	   scalar  name of a UI page for details
	
	A module monitor item namespace may have any other fields needed by
	the specific module.  For exec style tests (see below) all namespace
	fields will be stored as environment variables.  Any namespace which 
	does not want to show up in the UI (see aggregates below) can set 
	the optional 'hideUI' boolean property to 'true'. Any namespace on 
	the ActiveMonitor object which does not provide these fields will 
	be ignored.  

3.0 Execution

	Each test is run, in single-file manner, until all test are
	completed.  There are several styles of tests, specified by the
	'type' configuration directive.

	Type 'exec'
	----------
	Exec style tests fork a child process and exec the value of the
	'exec' configuration directive, with no command line options.  The
	elegance of this type is that the test program need know nothing of
	the Active Monitor core.  All it is responsible for is testing
	system state, optionally attempting to fix errors, and returning a
	status code and message.

	The exit code of the test process determines the new state of the
	service, as stored in the currentState property in CCE.  Available 
	exit codes are defined below:

		0              No Info
		1              Green       (status good)
		2              Yellow      (status warn)
		3              Red         (status trouble)
		255            No Change	

	These codes are also defined for programs to read in the
	/usr/sausalito/swatch/statecodes file.

	When swatch sets the currentState property, the module may
	have handlers registered to take some action to rectify the state.  
	This is left to the module to implement.

	Type 'tcp'
	----------
	The tcp type is implemented for the average network (tcp) service.
	The config directives 'port' and 'retries' are examined.  A tcp 
	connection is made to the specified port.  If a connection can
	not be made, the 'restart' field is examined and called via
	system().  The process repeats 'retries' times.  

	When completed, the current state is examined, and either
	'green_msg', 'yellow_msg', or 'red_msg' are examined from the config
	file, and returned as the status message.

	Type 'udp'
	----------
	The udp type behaves very much like the tcp type, but makes a udp
	socket.  Some small datum is sent on the socket.  The socket is then
	polled for < 5 seconds for incoming data.  If data is available (by
	select()), a recvfrom() is performed.  If the recvfrom() succeeds 
	or no data is available at the end of the timeout, the connection is
	assumed to have succeeded.  If recvfrom() fails, the connection is
	assumed to have failed.

	Type 'aggregate'
	---------------
	An aggregate type test is actually a place holder for multiple other
	tests.  The 'members' directive declares the number of sub-items to
	process, and the 'green_msg', 'yellow_msg', and 'red_msg' directives
	define the aggregate messages.

	Each sub-item may be any of the other test types, with the exception
	of 'aggregate'.  No aggregate may have another aggregate as a
	sub-item.  Each sub-item is processed, with each directive name
	prefixed by the item number and a period ('.').  For example, the
	type of the first sub-item is '1.type'.

	Sub-items are the most common use of the 'hideUI' schema property.

5.0 Internationalization

	All status messages are assumed to be internationalized tags, and
	will be interpolated as such.  Tgas should be of the general form
	"[[domain.tagName]]" (see sausalito i18n specs).
