The Object Data Component

Licensed and Supported Software
(C) Copyright 2002, Hume Integration Software
All Rights Reserved

Introduction

The Object Data Component (ODC) provides for simplified, high-performance access to SQL table data with configurable persistence.  The simplified access to SQL table data uses the normal array variable syntax of the Tcl programming environment.  By using the ODC, your application is able to manipulate global array data with high-performance, and to have changes asynchronously posted to a persistent database by the queuing of DMH subscription messages.  By default, table data that is manipulated in this way is persistent.   The same simplified access to attribute data can also be used for manipulating  transient data.  The design of the ODC provides for hiding these less significant changes from a persistent database using one-time method calls to set selected attributes as transient.  The current object data is always available with high performance by directly accessing in-memory SQL tables.  The simplified array variable access is a high-level alternative to executing SQL statements.

Installation

The ODC is provided as a Tcl package with the package name of HumeODC.  The files associated with the package, are usually distributed and installed as part of the "Hume Component Libraries" installation choice.  The setup program creates the directory, humelib1.0, under the lib directory of your Tcl installation.  So the files in the package are typically found in the directory path /usr/local/lib/humelib1.0/*.

The files needed for the HumeODC package include
 
Filename Description
en.msg Definitions of English language messages for all English locales including en_US.  The ODC uses the msgcat package to support multi-lingual deployment.
od_load_data.tcl Tcl code to load the ODC tables from a persistent database.
od_save_subs_init.tcl Tcl code to open table subscriptions for saving ODC data.
od_schema_db_init.tcl Tcl code to create the ODC schema in a persistent database.
od_transient.tcl Tcl code that implements the od_*transient* commands.
odc.tcl The core logic of the ODC.
pkgIndex.tcl An index of available Tcl packages in the directory.  Needed by Tcl to find the HumeODC package when required.
tclIndex An index of available Tcl commands in the directory.  Needed by Tcl to find the od_* commands after the HumeODC package has been required.

Initialization

If you are using the Event Processing Component (EPC), initialization of the ODC component is done automatically when the EPC is initialized.  The usual startup sequence to use the ODC follows:
  1. Your application processes start and establish DMH communication to each other.  The datahub Tcl application is commonly used to provide the DMH server function.   If you are running your application as NT services, it is common just to start the DMH server process as a service, and then have the server start the other application processes.
  2. If you are using a persistent database, you establish a DMH mailbox for SQL commands directed to the persistent database.  The dmh_SQLsrv Tcl application is commonly used to provide this function.
  3. You usually use in-memory tables in the DMH server process for the ODC.  The next step is to initialize the DMH server process and its in-memory tables to serve your application needs.  During your application initialization, a  typical sequence of  commands to use the ODC looks like the following:
global env
if { ![info exists env(LANG)] } { set env(LANG) en_US }
package require HumeODC
namespace import -force HumeODC::*
od_init DB
In order for messages to be presented with English text, there should be a definition of the environment variable LANG.  So the first two statements above supply a default LANG setting if the user has not specified one.  The next statement, package require HumeODC, causes the ODC installation directory to be added to the interpreter's auto_path variable, so package commands are found and can be used as necessary.  The namespace import statement is done for convenience so that command names do not have to be prefixed with the namespace name, HumeODC.  After the namespace import statement is executed, the od_init command can be called as od_init instead of HumeODC::od_init.  Calling the od_init procedure creates the in-memory SQL tables for object data, and initializes the simplified array access logic.

Some applications that use the ODC will not use a persistent database, since the high-level array access to table data is also very useful just for in-memory SQL tables.  When a persistent database is in use, the name of the mailbox that is used to process SQL commands for the persistent database is passed as an argument to the od_init call.  When the persistent mailbox name is not blank, (1) the ODC logic create tables in the persistent database if they are not present, (2) the ODC loads the initial data into the in-memory tables from the persistent database, and (3) the ODC opens data saving table subscriptions.  If for some reason a persistent database is in use but this initialization is not desired, pass an empty string for the mailbox name argument to the od_init call, and then when the call returns set the value of HumeODC::MB_DB to the persistent database mailbox name.  The value of the persistent database mailbox name is needed after startup only for the od_*transient* commands.

Usage Concepts

The ODC manages two SQL tables; od_attributes, which is used to store data attributes of items, and od_assoc, which is used to store associations of items to other items.

You define your own types of items.  To make this discussion more concrete, lets suppose we have defined a type of item called a LotBox which is type of container.  Lets also define Wafer as an item type.  The new types are easiest to define using Tcl code:

global ODC
lappend ODC(:Type) LotBox Wafer
As the result of executing the above statements, two new rows are added to the od_assoc table, and variable traces are created so that we can use LotBox and Wafer as global arrays.  Now we can have instances of the new types.  Suppose that every LotBox has a unique serial number, and that every Wafer has a unique identifying scribe.  Suppose there are three Wafers, w1, w2, and w3, currently in LotBox lb100.  We can save the association of these Wafers to the LotBox with the following statements:
global LotBox
set LotBox(lb100:Wafer) {w1 w2 w3}
Later in our code, we can easily determine the Wafer items associated with the LotBox:
puts $LotBox(lb100:Wafer)
w1 w2 w3
If you display the contents of table od_assoc, you will see three separate rows representing the association of each of these Wafer items to the LotBox.

Data attributes are simply named values that we wish to use.  Suppose we tested the wafers and we now have some test results we wish to save.  Each result has a name, and a value.  So the results can be saved as attribute data using statements like:

global Wafer
set Wafer(w1.bin1) 237
set Wafer(w2.bin1) 242
set Wafer(w3.bin1) 244
set Wafer(w1.Yield) 89.3
set Wafer(w2.Yield) 91.2
set Wafer(w3.Yield) 92.4
Each of the above assignment statements causes a row to be added to the od_attribute table.  If we examine a particular item, such as Wafer w3, we can determine all of its attribute values by using a wildcard attribute name, *:
puts $Wafer(w3.*)
Yield 92.4 bin1 244
The result is a list of alternating name and value elements.

The wildcard attribute expression also works for writing, too.  We can add other name, value attributes to the list, and easily save them in one statement:

set Wafer(w3.*) {Yield 92.4 bin1 244 bin2 22 bin3 13}
Another feature is the ability to save attribute values for the item type, and have them be the default values for item instances.  Suppose that most of the Wafers in our example are 200mm diameter, but there are some whose sizes are different.  We can set the default for all wafers as follows, and verify that it shows up for the particular wafer w3:
set Wafer(diameterMM) 200
puts $Wafer(w3.*)
Yield 92.4 bin1 92.4 bin2 92.4 bin3 92.4 diameterMM 200
Wafer w99 is different.  We can specifically set a different attribute value for it:
puts $Wafer(w99.*)
diameterMM 200
set Wafer(w99.diameterMM) 300
puts $Wafer(w99.*)
diameterMM 300

User-Defined Object Types

In the above discussion, the phrase "type of item" was used.  We have used the name item_type in the ODC tables for this concept.

The system user defines a set of item_types, such as Lot, Event, etc.  The item_types known to the system are stored in the od_assoc table.  Tcl programming statements have simplified access to the table data using global arrays.  Simplified access is supported for reading or writing.

The list of known item_type values in a newly installed system, includes only ODC.   A new installation of the Event Processing Component (EPC)  includes the following:

These values are represented in the od_assoc table with item_type='ODC', item_id='*', child_type='Type', and the child_id values being the type names.  Reading the value of ODC(*:Type) or  ODC(:Type) returns the list of known item_type values.

You can add new types to the od_assoc table using the SQL statements or the Datahub GUI.  It is also simple to append them as list items to the current list:

global ODC
lappend ODC(:Type) NewType1 NewType2 ...
If you are using a persistent database, adding new types need only be done once.

The system user should configure new item_type values before attempting to use the global array access techniques.

Attributes

The od_attribute table is used for the storage of attribute data.  The data in the table may be manipulated directly using SQL, but it is also convenient to access the data using global Tcl array variables as described next.

There is simplified access to the attribute values of specific item_type instances using the following notation:

set item_type(item_id.attribute) value

global MID Lot
set MID(hp34au.Softbin) 24
set Lot(6809AUQ5.case_quantity) 100

 There is simplified access to the item_type level to create attribute values which are default values for any instances of the item_type.  This data is stored in the table with the value of the item_id field set to *.
set item_type(attribute) value   ;# the implied item_id is '*'

$EPC(StateLogInterval)  is the same as  $EPC(*.StateLogInterval)
$EPC(EventLogInterval)
set Lot(case_quantity) 500

There is "inheritance" from the type to the instance.  If a lookup for the value of  item_type(item_id.attribute) is not found, the value of item_type(attribute) is returned if it exists - powerful and simple.  The mechanism is convenient to configure defaults for the creation of new items, and for the compact representation of repeated items.  If you explicitly write to item_type(item_id.attribute), it is written out for the instance whether or not it is the same as the type level default.

There is wildcard access to all the attributes for an instance.  If you lookup $item_type(item_id.*) we specially trap the "*" and return a list of alternating name and value for all of the attribute values.  We do this by looking up the attributes defined for the item_type first, $item_type(*.attribute), and then overlaying the instance data.  There is direct writing of all the attributes of an instance, the exact reverse of the lookup.  With this mode of writing, if the attribute of an instance is the same as it is for the item_type, it is not written out for the instance.  Any attributes of the instance that are in the table but are not found in the wildcard write, are deleted.

set item_type(item_id.*) [list name1 value1 name2 value2 \
    name3 value3 ... nameNvalueN]

set MID(hp34au.*) {Softbin 24 location h-15 engineer {J. Bond}\
   IP_address 192.168.2.3 MID_class hp3000 ... }

You cannot use the Tcl upvar command to work with the arrays using a different name since the array name is a key to the table mapping.

Transient Attributes

By default, attributes that are represented in the od_attribute table are persistent.  In other words, their value changes are posted to the persistent database as they happen.  The same simplified access to attribute data can also be used for manipulating less significant, transient data such as counts and state information that change many times per second.  It is a desirable performance boost to hide these busy, and less significant data changes from the persistent database.  The design of the od_attribute table supports this idea by having the column is_transient which is used as a boolean flag and set to 1 to hide the updates of selected rows from the persistent database.

There are a set of procedures, documented below,  provided with the ODC to help manage transient data.  There are some subtle aspects to how the subscriptions work that make using these procedures desirable.  For example, if you simply update the is_transient field for selected attributes, the persistent database does not save that you want a non-zero is_transient value.  So when you startup again, the attributes are back to being completely persistent.  Using the od_transient procedure to declare specified attributes as being transient takes care of making sure the persistent database saves that the specified attributes are transient.

Object Associations

There is simplified access to the association data stored in the od_assoc table using the following notation:
set item_type(item_id:child_type) id1 id2 ... idN
Using a different separator in the array subscript, the colon, :,  tells the software that an association is implied.

If the item_id portion of the array subscript is blank, it defaults to '*'.  So ODC(*:Type) or ODC(:Type) can both be used to access the same set of table records.

Similarly, if the child_type array subscript term is left blank, it defaults to the value * with a wildcard meaning.  The wildcard behavior is not so useful for reading because child_id values of different child_type values are mixed, but it can be very powerful for deleting everything associated with an item by assigning an empty string.  For example:

% set Shooter(Buff:Friend) {Bill Barney Buddy}
Bill Barney Buddy
% set Shooter(Buff:pocket_item) {Comb Wallet gum keys notepad passport pen}
Comb Wallet gum keys notepad passport pen
% set Shooter(Buff:)
Barney Bill Buddy Comb Wallet gum keys notepad passport pen
% set Shooter(Buff:) {}
% set Shooter(Buff:)

Server API

Your logic needs to call od_init if you are not using a component such as the EPC that calls it for you.

You optionally make use of the od_*transient* calls to manage whether your attributes are transient.


od_init -  initialization of the ODC.

Tcl Command Syntax

od_init {mb_db {}} {NAMELEN 32}
Return Value
"Object Data Component (ODC) Initialized Successfully." is returned for the usual successful case.

The procedure will return a Tcl error if the system cannot be initialized.

Description
This is the entry procedure to initialize the ODC.  The MB_DB argument is the DMH mailbox of the persistent database interface.  If a persistent database is not being used, the MB_DB argument value should be an empty string.  See the discussion of application initialization.

The optional NAMELEN argument is passed through to the schema create logic where it sets the number of characters specified in the varchar( ) table fields that are used for names.
 


od_schema_write -  Write SQL statements representing the database table
schema to a file.

Tcl Command Syntax

od_schema_write {filename dc_schema.sql} {DBtype hub}
Return Value
None
Description
This procedure calls od_schema and writes the result to a file with each create statement being written as a new line of text.  The file is not used by the ODC, this procedure exists for your use.  Set the DBtype argument to any value other than hub to have the primary table keys declared as NOT NULL.

od_transient - setting of the is_transient od_attribute table field value
od_not_transient - clearing of the is_transient od_attribute table field value
od_is_transient - reading of the is_transient od_attribute table field value
od_transient_save - post current transient values to the persistent database

Tcl Command Syntax

od_transient [attribute_name]+
od_not_transient [attribute_name]+
od_is_transient attribute_name
od_transient_save [attribute_name]+
Return Value
od_transient, od_not_transient, od_transient_save: row_count
od_is_transient: boolean
Description
One of the key benefits of the ODC  is the simplified access to SQL table data using the normal array variable syntax of the Tcl programming environment.  The application is able to manipulate data with high-performance in the datahub process, and to have changes posted to the persistent database by the queuing of DMH subscription messages.  By default, attributes that are manipulated in this way, and represented in the od_attribute table are persistent.  In other words, their value changes are posted to the persistent database as they happen.  The same simplified access to attribute data can also be used for manipulating less significant, transient data such as counts and state information that change many times per second.  It is a desirable performance boost to hide these busy, and less significant data changes from the persistent database.  The design of the od_attribute table supports this idea by having the column is_transient which is used as a boolean flag and set to 1 to hide the updates of selected rows from the persistent database.

These procedures are used to help manage transient data.  There are some subtle aspects to how the subscriptions work that make using these procedures desirable.  For example, if you simply update the is_transient field for selected attributes, the persistent database does not save that you want a non-zero is_transient value.  So when you startup again, the attributes are back to being completely persistent.  Using the od_transient procedure to declare specified attributes as being transient takes care of making sure the persistent database saves that the specified attributes are transient.

The format of the attribute_name arguments is expected to be one of the following:

  1. Item_type(item_id.*)
  2. Item_type(*.attribute) or equivalently Item_type(attribute)
  3. Item_type(item_id.attribute)
Case 1 identifies all existing attributes of a specified item.  For this case, the od_is_transient call returns 1 if any of the existing attributes of the item are transient.  Case 2 affects the single record for item_id='*' which is used as a default value for new instances, or as a working value for "Singleton" types.  Case 3 affects the single record; a specific attribute of a specified item.  For the od_transient call and case 3, if the record does not already exist, a table insertion is made with an empty value and the is_transient flag set.  For this case and the od_not_transient call, no insertion is made if the row does not already exist since the default behavior will insert a new row as not being transient.

The row_count value returned by the od_transient, od_not_transient, and od_transient_save calls show the number of table rows matching the input specifications and subject to the implied action of updating the is_transient value or being posted to the persistent database.
 

Table Schema

Identical sets of tables are created in the Tcl/Tk process that initializes the ODC, and in the persistent database, when a persistent database is being used.  The tables created and used by the ODC are described below.  The Tcl/Tk process where the ODC is executing is almost always a Datahub process.

od_assoc Table - Configuration of 1::N associations

There is simplified access to the association data stored in this table as presented above.

The suggested behavior for a configuration user interface follows.  The list of known item_type values is consulted by the configuration GUI and the user is only allowed to configure instances, attributes, and associations of known types.  There is also provision from the GUI to add or delete known types.  The GUI is integrated with the DCC security features, and it requires a different privilege to add or delete types than to edit instances.  The item_types that are in use have to be configured in order to setup variable traces to support the simplified access.

You cannot use upvar to work with the arrays using a different name since the name is a key to the table mapping.

The allowed size of child_id records is substantially smaller than that allowed for attributes.
 
Column Name Key Type Description
item_type PCK varchar(32) A system or user defined type such as EPC, Lot or MID.  Should be a single alphanumeric token without imbedded whitespace or punctuation. 
item_id PCK varchar(32) A user defined name for the instance of a type.  May contain whitespace.  Should not contain the colon,  the period, or other punctuation.
child_type PCK varchar(32) Usually, a system or user defined type such as Lot or MID.  Generally, the field can also be used as the name of the associated child_id value.  Should be a single alphanumeric token without imbedded whitespace or punctuation.
child_id PCK varchar(150) Usually, an identifier for a child_type, but generally just an associated named value.  The length of this field is limited because some persistent databases limit the total length of a composite key.

od_attribute Table - Persistent Object Attribute Data

The od_attribute table is used for the storage of attribute data.  The data in the table may be manipulated directly using SQL, but it is also convenient to access the data using Tcl array variables as described above.

The EPC runtime system assumes that there is a Type MID representing machines or state machines, and that instances of this type have the following attributes represented in this table:

Column Name Key Type Description
item_type PCK varchar(32) A system or user defined type such as Lot or MID.  Should be a single alphanumeric token without imbedded whitespace or punctuation.  All types should be represented in the od_assoc table in order to access the table data as global Tcl arrays.
item_id PCK varchar(32) A user defined name for the instance of a type.  May contain whitespace.  Should not contain the colon,  the period, or other punctuation.
attribute PCK varchar(32) A system defined or user defined name for an attribute such as state or recipe_id.
value varchar(2000) The value of the named attribute for the instance of the type.
is_transient int A boolean flag to determine whether updates to the record should also be fed to the persistent database.  Similar to the DCC dc_data discard_flag.  If the value is 1, updates are not passed on to the persistent database.  Inserts and deletes are always passed on to the persistent database so that records are present at startup.  Initial values at startup are usually set by business rules.  This field is desirable so the attribute system can be used for very busy data changes without worrying about the overhead.

Document Version

Date of last revision: $Date: 2008/09/04 21:25:13 $

This document covers the Hume Integration Software developed Object Data Component which is available for the Tcl 8.3/Tk 8.3 environment on the Windows 2000/NT and UNIX/POSIX platforms.