Spectre Build Temporary Files

For those curious about temporary files used by Spectre.

cBases in use

If a DiveLine client, such as ProDiver or DivePort, has a cBase open when a separate process tries to rebuild that cBase, Spectre creates temporary files. That is, if Spectre cannot delete the existing cBase, Spectre renames it with a file name syntax of <filename>.tmp.in-use-######## and locks the file so it cannot be opened by another process while it is being written. The next time Spectre is run for that Build file, Spectre deletes the in-use file. This applies to processing on Windows. You might occasionally see temporary files in the project, but only as many as are in use at the time of the build. The temporary files do not accumulate over time.

Building cBases

When the spectre build command creates a cBase, the build converts text data into a binary format. It cannot save the data in the full column-oriented cBase format until it is finished reading all of the data. In most cases, the build does not need to have all of the data in memory at once, so Spectre saves partial results to disk to free up memory. Those partial results are saved in chunk files. After all the data has been processed, the chunk files are loaded and deleted, one at a time, to write the cBase.

An exception to this memory usage occurs if the Build script contains certain build operations (for example, add, sort, or filter; but not remove or rename). In these cases, after reading all of the inputs, the chunk files are loaded back into memory and Spectre consumes memory for the entire cBase in order to process those operations.

Temporary chunk files

The temporary files are named di-chunk-cache- with a randomly generated suffix. A chunk file's maximum size is about 500 KB. Each chunk file contains part of the data generated for a single column. Spectre creates as many chunk files as necessary to produce the cBase. If there are two Spectre builds running simultaneously, each build uses separate chunk files and does not interfere with the other build's chunk files.

Temporary file directory

By default, chunk files are stored in a temporary directory. On Linux, the directory path is supplied by the first environment variable found in the list TMPDIR, TMP, TEMP, TEMPDIR. If none of these are found, the build uses the /tmp directory. On Windows, the directory path is specified by the first environment variable found in the list TMP, TEMP, USERPROFILE. If none of these are found, the build uses the Windows directory C:\WINDOWS. There are a couple of ways that you can override the value of the default temporary directory. You can use the --temp-directory parameter of the spectre build command to specify your temporary directory. Also, you can set a value for --temp-directory with the Visual Integrator node for the cBase output object .

Removing temporary files

The files accumulate during the 'Reading xxx...' phase of the build, and usually are removed one at a time during the 'Writing xxx...' phase of the build (or before that if build operations are in use). The files are also removed if a fatal error occurs before 'Reading' completes. They might not be removed if the Spectre process is canceled or if it crashes. If you notice chuck-cache files in the projects directories, and you know that no builds are running, you can remove the files manually.