The Integrator Tunnel output object creates a dynamic memory model. Tunnel refers to a DiveLine capability where DiveLine can tunnel out and access data from external data sources. Tunnel allows DiveLine to filter back-end data based on user selections, as well as Access Control Lists (ACLs), and provides access to near real-time data.
Tunnel is implemented by having a DiveLine client open up an Integrator Tunnel script (saved with file extension .tnl). The Tunnel output object in this special script describes what data should be returned through DiveLine, and what columns should be Dimensions, Summaries and Info Fields. The Tunnel script can access data contained in flat files, spreadsheets, Models, or SQL or Access databases, and use any other Integrator objects to process the data. At a minimum, it must have an input object and the Tunnel output object.
An Integrator Tunnel output object writes data to a file when run by Integrator, but is the source of the data when the script is accessed using other DI software. The data from a Tunnel object can be viewed by any DiveLine client.
Although all DiveLine clients support Tunnel, user configurations must be set in DI-Config to allow Tunnel file execution in the 6.4 DiveLine namespace. Also, in terms of data security, DiveLine ACLs for .tnl files are similar to .mdl files. Access is granted or denied on the Memory Model, not on the source data. If the source data is a Model, ACLs will already have been applied when the Memory Model was built.
For 7.x projects, if access control rules are defined such that the user can see and open a Tunnel script (has read permission), the script can run. There are no additional access control checks on files that Tunnel scripts open from a Project. Tunnel scripts can only be created by authorized users and these scripts are considered trusted. This matches non-Project behavior.
Using Tunnels
When opening a Tunnel file, DiveLine runs the given script, reads the data returned by the Tunnel output object, and constructs a dynamic Memory Model that then opens. A Tunnel file can be used in the same way as a .mdl file can be used, that is in DivePlans and in Markers. It can be used in a Report Palette cell link or a Report Window item link. These "links" employ the standard "diveref:" syntax:
diveref:window_type["tunnel_file.tnl"
{parm="$TEXT"}.dive_dimension,
_attribute_name="attribute_value"]
For more information on window dive reference strings, see the ProDiver Help.
For Model calculation references, the general syntax is:
Summary Type[ "tunnel_file.tnl"
{ parameter1="valuea",
parameter2="valueb"}.Summary field,
dimension1="valuec", dimension2="valued" ]
where Summary Type is "Total", "Minimum", "Maximum", etc. For more information on calculation references, see ProDiver Help.
Tunnel Attributes
| Attribute | Type | Description |
|---|---|---|
| output_type (required) |
String | Identifies the object as a Tunnel output object. The value of the string is "tunnel". |
| input (required) |
String | Defines the object from which the data flow is arriving. |
| filename | String | Defines the output file to use when the script is run by Integrator. If included in the .tnl file, the DiveLine client ignores it. This is used to develop and debug the Integrator script before creating the executable script file (.tnl). The output file is written with column headers and a comma delimiter. This attribute is optional, but required when testing scripts using integ.exe. |
| dimensions (required) |
Array of Strings | Defines a list of columns that will be Dimensions in the resulting Model. This is equivalent to the dimensions attribute in the Build Description. |
| summary | Array of Strings | Defines a list of columns that will be Summary columns in the resulting Model. This is equivalent to the summary attribute in the Build Description. |
| info | Array of Strings | Defines a list of strings that contain Info Columns and corresponding Dimensions in the resulting Model. This is equivalent to the info attribute in the Build Description. Each string should be of
the form:
"<info column name>:<dimension name>" |
| types | Array of Strings |
Contains an array of strings, one for each column in the output cBase that should be given a specific type. The column name is followed by a colon (:) and the requested type for that column. For example: Units:integer. Possible types are string, integer, fixed100, double, boolean, date, period, and datetime. These correspond to Spectre date types. See Spectre Data Types. The types array is an alternate way to specify column types instead of the string and dates attributes. Columns can appear in both places, provided that the types do not conflict. NOTE: The Visual Integrator interface includes a list for each row where you can set these column types. |
| string | Array of Strings | Defines a list of columns that should be interpreted as string values rather than numeric values. |
| sort | Array of Strings | Defines a list of strings that contain Dimension columns and alternative sort columns. This is equivalent to the sort attribute in the Build Description. Each string should be in the form:
"<dimension column name>:<alternative sort column name>" |
| model_title | String | Defines the name of the resulting Model as it will be displayed in the Console. If this attribute does not exist, Diver will use "Dynamic Model" as the Model title. |
| return_model | String | Defines the name of an existing Model that will be returned as the result of the script. If this attribute is set, the Tunnel object does not return any raw data, but will return an existing Model. Earlier tasks in the script can build a Model file on demand and the Tunnel object is used to return the resulting Model. If this attribute is defined, place the Tunnel OUTP object in a Task by itself as when this attribute is defined, all other flows in the task containing the Tunnel object are unused and will not be run; the input attribute is ignored. |
| dates | Array of Strings | Contains an array of strings, one for each numeric item in the input file that should be treated as a date. This attribute should be paired with the formats attribute. It is important to identify date columns so they are stored properly in the Model, facilitating Diver functions (e.g., sorting, date and time functions, Time Series, etc.). |
| formats | Array of Strings | Contains an array of strings, one for each date element listed in the dates attribute. The field name is followed by a colon (:) and the date format.
Example:
"Discharge Date:YYYY/MM/DD" NOTE: A tunnel can be used in the model-engine or cBase-engine. If explicit types are set for a cBase, formats can be set on any column. In the model-engine, formats are only allowed for date fields. |
| calendars | Array of Strings
|
Contains an array of strings that defines calendars for period columns. The field name for the period column is followed by a colon (:) and the calendar name. Valid calendars are the ones described in the Spectre topics (Date and Time Functions, Time-Series Functions, Creating a Custom Calendar): gregorian month, gregorian quarter, iso8601 week, gregorian+<N> month or a custom calendar file. Examples: "Order Month:gregorian month" and "Fiscal period:/calendars/fiscal.cbase." |
| nullable | Array of Strings | Contains an array of strings, one for each Summary field that contains nullable data. Normally, Builder will treat null values in input as zeros. If a field is set to be nullable, Builder will distinguish between zeros and nulls and keep a count of the non-null values in the input. This count will be used when calculating averages and standard deviations on the field in Diver. The null values will be preserved in calculations wherever possible. Note that setting a column to be nullable will slightly increase the size of the Model. |
| build_on_demand | Boolean | If "true", this attribute defines the return_model to be built on demand. When the tunnel is run, DiveLine will check to see if the return_model exists. If so, it returns it immediately. If it does
not exist, the rest of the Tunnel script, which presumably will create the return_model, is
executed. This allows the tunnel designer to cache the results of building a return_model. To allow the return_model name to be parameterized based on tunnel parameters, tunnel will run any initial tasks which have a Setparm output object to set necessary parameters before checking the existence of the return model. This allows the tunnel script writer the opportunity to transform the tunnel parameters into other values for the Integrator script. |
| model_vars | Array of Strings | Contains an array of sub-strings, one for each user-defined Model variable to be set in the Model. Each sub-string consists of the variable name, an equal (=) sign, and the variable value. Each should be surrounded by quotes and separated by commas. These model variables are available in ProDiver through the $MODELVAR(<variable name>) reporter variable used in Reports and Report Palettes. See the ProDiver Help for more information. |
| trace_before | Sub-object | Traces data flows entering the specified object. This is equivalent to adding the Trace process object immediately before the current output object. See Embedded Trace Object. |
Parameters
When DiveLine runs a Tunnel file, certain DiveLine values are passed in as parameters to the Tunnel Integrator script. These values are given by DiveLine itself. To use these parameters, they must be declared in a PARM object in the Integrator script. These parameters all start with an underscore (_) to distinguish them from normal parameters that can be passed in from the Report Palette or a DivePort Click Action.
The following parameters are passed in by DiveLine to Tunnel scripts:
- $_user: The user name of the DiveLine user running the Tunnel script. This parameter can be used to filter data that is processed in the Tunnel script so that only user data is returned.
- $_pid: The process id (pid) of the DiveLine process running the Tunnel script. This parameter can be used to create and identify unique temporary areas for the Tunnel script to use.
- $_diveline: The DiveLine dataroot directory name for the DiveLine process running the Tunnel script.
- $_diveline_data: The directory name for the data area (e.g., DI_Solution\dl-dataroot\data) for the DiveLine process running the Tunnel script. This directory is the server pathname for the directory that appears as the home directory ("/") in DiveLine clients such as ProDiver.
- $_hash: This contains all parameter values available for use by an Integrator Tunnel script. It can be used to determine if anything has changed, allowing the developer to bypass re-executing logic that would produce the same result.
Values for these and other parameters passed to a Tunnel script are captured in the DiveLine usage logs. Look for the string, "Parameters:" in the Details section. The use of Tunnel for creating Memory Models is appropriate for small data volumes from secondary dives, when up-to-the-minute data is required.
TIP: Tunnel parameter names are case sensitive.
See Filter Process Object for information on using the quickview attribute for passing filter values from QuickViews.