Database Utilities

The database of user's jobs is stored in the array called 'database' and this saved to a file project_directory/CCP4_DATABASE/database.def in a standard def format file. It is strongly discouraged to access the database array directly from any procedure outside of this file. An empty database is defined in the file $CCP4I_TOP/etc/database.def. Each job in a project is given a job id which is an integer which is incremented, from 1, for every new job. The currently supported database parameters are NJOBS (the number of the highest job id in the database) and then parameters *,$n where n is the job id and the required values of * are given below. These parameters are written to the project database file, there are additional parameters may be saved in the database array but not be saved to file.

TASKNAME The name of the type of task

DATE The date that the job started (if still running) or finished saved as the machine time.

STATUS A one-word job status

LOGFILE The file name of the log file (NOT the full path name)

TITLE A text string of the job title provided by the user

INPUT_FILES A list of the input file names (not paths UNLESS the user has specified Full path rather than used a directory alias)

INPUT_FILES_DIR A list of the project/directory aliases for the input files

INPUT_FILES_STATUS Currently unused

OUTPUT_FILES A list of the output file names (not paths UNLESS the user has specified Full path rather than used a directory alias)

INPUT_FILES_DIR A list of the project/directory aliases for the output files

OUTPUT_FILES_STATUS Currently unused

Each project that the user creates will have it's own database.def file and the appropriate file is loaded whenever the user changes between projects. The database.def file is updated from memory after every significant change. The time taken to read a database.def file is significant for projects with many jobs - it may be worth considering an alternative format for database.def in future.

Core Database Functions

Core Database Functions

These functions handle direct interactions with the database information stored in the database.def and global database array.

Argument list: <project> <db> <statusVar> <userVar> <timeVar>

Returns the status in statusVar - "unlocked" or "locked"

project Alias for project

db Directory for corresponding database

statusVar Name of variable to return status in

userVar Name of variable to return user in

timeVar Name of variable to return time in

Argument list: <project> <db> <args>

Returns status from DbLoadFile: 1 for success, 0 for failure.

project Project name (i.e. alias) for project

db Corresponding database directory

-grablock

Grab the lock from another process

Argument list: <project> <db>

project Alias of project to load

db Corresponding database directory

Argument list: None

DbItemExists Check whether a data item is defined in the database

Argument list: <job_id> <type>

job_id Job id (Ignored if type = NJOBS)

type The type of data to be checked - see

Argument list: <job_id> <args>

There is no content checking in this procedure - the user is responsible for sensible input. There can be one or more pairs of input param_name/param_value arguments.

job_id Job id

param_name Name of database parameter - see

Argument list: <job_id> <type>

job_id Job id (Ignored if type = NJOBS)

type The type of data to be returned - see

Argument list: <type> <pattern>

By default the jobs are sorted on job number in descending order.

type Name of data item to select on

pattern Regular expression pattern to match against

Argument list: <project> <db> <job_id> <db_display> <db_display_format>

project The alias for the project to access

db The corresponding database directory

job_id Job id

db_display A list of the database parameters to be displayed

db_display_format A list of the field widths (as integers) to use for displaying the items in the db_display list.

Argument list: <job_id>

job_id id for the job record to be removed

Argument list: <project> <db>

If there is a current 'open' database then make sure it is up to date and then unset the database array

project Alias for project

db Corresponding database directory

Argument list: <project> <db> <args>

This procedure contains a messy bit of code to set the status of jobs in the output file to 'ON_HOLD' without changing the status in the loaded database. This is used for creating regression test directories and databases in db_autotest_handler

Return 1 if save was successful, 0 if not

project Alias of the project for which the database is to be saved

db Corresponding database directory

-hold

Set job status in output file to ON_HOLD

-range first last

if -hold also set, set job status of jobs in range first to last to ON_HOLD

Commands for Handling Database Locking

Commands for Handling Database Locking

These commands handle creating, interrogating and removing lock files associated with CCP4i project database.def files.

Argument list: <project> <db>

Lock file name is database.LOCK

project Alias of project

db Corresponding database directory

Argument list: <project> <db> <userVar> <timeVar> <pidVar> <>

Return 0 is lock file exists and 1 (i.e. OK to open database file) if it does not.

project Alias of project

db Corresponding database directory

userVar Return name of user who created lock file

timeVar Return time of creation of lock file

pidVar (Optional) Return id of process which owns the lock file

Argument list: <project> <db>

Returns 1 if lock was created, 0 otherwise

project Alias of project

db Corresponding database directory

Argument list: <project> <db> <args>

-force

Force deletion of the lock even if not owned by the current process

APIs to Access the Database

APIs to Access the Database

These are CCP4i-specific commands to interact with the database. They are built on top of the core database commands.

Argument list: <arrayname> <taskname> <status> <STARTING>

This procedure will automatically extract data from the array associated with a task and load it into the database. The data in the task array must have the appropriate parameter names:

TITLE A title for the job

INPUT_FILES A list of the *parameter* names in the task array for input files (NB not the file names themselves)

OUTPUT_FILES A list of the *parameter* names in the task array for input files (NB not the file names themselves)

The procedure returns the job id.

arrayname The name of the task array

taskname The name of the task

status Optional status - default is 'STARTING'

Argument list: <job_id>

This should only be used if job_id is the last job in the project. This procedure is usually called if the user has opted to abort running a job.

job_id Job id

Argument list: <job_id> <dir>

There are standard file names for log file of the form jobid_taskname

This does not return the file extension and only includes the full directory path if the dir parameter is set. This will generally be used when you want to create the file.

job_id Job id

dir Optional project/directory alias or the word 'PROJECT' which will use the current project directory.

Argument list: <job_id>

If the file does not exists then return null. This will generally be used when you want to read the file.

job_id Job id

Argument list: <job_id> <logfileVar> <formatVar>

If the file does not exist or can not be read then the format is returned as 'NOFILE'. The format will otherwise be 'TEXT' or 'HTML'. This procedure reads the first few lines of the file to determine its format.

job_id Job id

logfileVar Returned full path name for log file

formatVar Returned file format

Argument list: <job_id> <mode> <filelistVar>

job_id Job id

mode either 'INPUT' or 'OUTPUT'

filelistVar Returned list of files with full path names

Argument list: <alias> <dir>

alias The one-word alias for the project

dir The full path for the project directory

Argument list: <dir>

Returns 1 if dir is a project or default directory, 0 if not.

dir Full path of the directory to check

Database GUI

Database GUI

These are procedures to draw the the job list from the database in the ccp4i main window.

Argument list: <args>

This procedure defines the parameters and format to be listed and calls db_update_list which actually draws the list.

-nosave

Do not save current database parameters to file.

Argument list: <project> <db> <db_display> <db_display_format> <\>

project The alias for the project to access

db The corresponding database directory

db_display A list of the database parameters to be displayed

db_display_format A list of the field widths (as integers) to use for displaying the items in the db_display list.

no_jobs_message The message to display if there are currently no jobs in the database.

Argument list: <mode> <w>

save current selection in archive(CURRENT_SELECTION)

mode Usually 'select' for if user selected job or 'clear' if user clicked F2

w The widget identifier for the listbox which displays the job list

Argument list: <command>

All of the utitlity menu items invoke this procedure when picked. This procedure checks that the appropriate number of jobs for the utility have been selected.

command Keyword to identify the utility that the user has selected.

Argument list: None

This procedure defines the menu content and then call db_draw_utilities which does the drawing.

Argument list: <optionlist> <commandlist> <messagelist> <\>

All information for drawing the menus is input as lists with one list item per menu item. This approach makes it easy to change the menu but ccp4i currently does not do this (its probably a bad thing anyway). It ought to be possible to write a similar procedure which draws the utilities as pull down menus and the user can then have the option on how to lay out the main window.

optionlist A list of the text to appear in menu. If the menu item has a submenu then the item is a list.

commandlist A list of the command keywords to be used by db_command_handler to invoke the right utility procedure.

messagelist A list of the menssages associated with each menu item. Note that there is no distinct messages for each submenu item.

activelist A list of 0/1 to indicate if a menu item is active. Currently all intput has these set to 1.

helpfile Name of a help file.

helplist List of the targets in the help file for each menu.

Argument list: None

This procedure is called when the user selects a job. The id of the menu is saved as SystemVar VIEW_JOB_JOBID.

The procedure tries to give appropriate options dependent on format of log file (? is it html) but does not look for tables in log file as this can be slow for big log files.

Initialising, Loading & Saving the Database

DbGetCurrentProject Return the project alias and database directory for the current project.

Argument list: <projectVar> <dbVar>

projectVar Name of variable in which the project alias will be returned.

dbVar Name of variable in which the database directory will be returned.

Argument list: None

Called from taskbrowser or whatever is main program

Argument list: <args>

This is essentially a wrapper to DbOpenDatabase which also performs some CCP4i-specific admin tasks, and invokes graphical components if prompting is required from the user.

-init

Should be specified the first time the procedure is invoked. It forces drawing of the utility menu if CCP4i is running in taskbrowser mode.

Argument list: None

This is essentially a wrapper to DbSaveFile.

Argument list: None

This is essentially a wrapper to DbCloseDatabase.

Argument list: None

Check for corrupted entries in the database, and restore missing elements with blanks or default values.

In principle the user should be able to repair the blanks using the edit job data utility - unfortunately the utility only allows the user to update a subset of the information for each job, so this is not an ideal solution.

Argument list: <project>

The user has changed the project so update the current project directory and database.

Argument list: <db> <utility_name> <identifier>

db Name of database file

utility_name type of database - expect it to be 'database'

identifier Identifier to write to file header - usually 'CCP4_Project_Database' - this is equivalent to the taskname in the usual task def files. A more descriptive identifier is used here in attempt to explain the file to anyone who reads the file itself.

Argument list: None

This is a wrapper to DbLockStatus.

Handling Input from Running Jobs

Handling Input from Running Jobs

Jobs which are run external to the main graphical ccp4i process return status updates using socket connections.

The information returned, the job status, finish time, and output files, is entered into the database using these procedures.

Argument list: <job_id> <project> <args>

If the update information is that STATUS is FINISHED then this procedure will update the job time to the finish time and call the task review procedures. Any task can optionally define a review procedure which is called after the job has finished (only a handful of tasks do have review procedures). Finally redraw the job list.

job_id The job number

project The database project of the job - beware if user has changed project then cannot update database

parameter_1,data_1,parameter_2,data_2.. List of pairs of parameter name and value.

Argument list: <job_id> <project> <args>

job_id Job id

project The database project of the job - beware if user has changed project then cannot update database

file_1,dir_1,file_2,dir_2.. A list of pairs of parameters, the first is the file name and the second the project/directectory alias. The file name should be just the file name and not the full path name. The dir is a project or directory alias.

Argument list: <job_id> <start_time> <0>

If the job start time is defined then also check that any file with the appropriate name was created between the job start and finish time. If the file does not exist or has wrong creation time then remove it from the list of output files.

Nb The output 'files' can also include directories. In this case the filename should be an empty string.

job_id Job id

start_time Optional job start time in machine time.

Argument list: <filename> <args>

Return a list of job ids which have output files with the given name

This procedure is most useful in checking that a currently running job not create a file of the same name.

-running

Only search through currently running jobs

Argument list: None

This procedure is usually called when ccp4i is started

Checks jobs with status RUNNING REMOTE ON_HOLD

Argument list: <logfile> <statusVar> <timeVar> <output_filesVar>

ccp4i scripts put a short section of termination status on end of log file which is usually not needed but may be checked for jobs which have RUNNING status when ccp4i graphical process is started.

logfile Name of log file

statusVar Returned status for job

timeVar Returned finish time for job

output_filesVar Returned list of output files (this will be list of file/dir alias pairs)

Argument list: <filename>

When specified output file already exists and user has opted to overwrite it then check thru all jobs in database to see if the overwritten file is a recognised output file. If it is then delete it from the output_file list of the previous job.

filename File name - NOT the full path name

Database Utility GUIs

db_archive_window Draw the window with archive/delete options

Argument list: <job_id_list>

call db_cleanup_handler and db_archive_handler to do the business

NB code from this procedure has been hacked together in the db_removeoutput_handler (which is used to delete files when the "Kill&Remove" option is selected). The common code should probably be farmed out to separate procedures in the future.

job_id_list List of job ids

Argument list: <job_id> <job_id_list>

job_id Job id for job with changed default options or 'ALL' if overall default is changed

job_id_list List of ids for all jobs currently in archive/delete window

Argument list: <job_id_list>

job_id_list List of the ids for jobs in archive/delete window

Argument list: <job_id>

This procedure is called when the user selects to Kill a running job and remove all output. It prepares input for and then calls db_archive_handler, which does the removal.

The procedure uses code hacked from db_archive_window. The common code should probably be farmed out to separate procedures in the future.

job_id The id of the job being removed

Argument list: <job_id_list> <action> <>

This procedure will delete temporary files and directories, def files, com files (the input scripts for the programs) and remove the job from the database. If action is not set then apply the cleanup defined by archive(DELETE_OPTIONS) parameter

job_id_list List of the ids for jobs in archive/delete window

action Action to apply to the jobs in list (optional)

Argument list: <job_id_list>

Calls db_restore_handler to do the business

job_id_list List of the ids for jobs in archive/delete window

Argument list: <nselect> <text>

nselect Text defining number of jobs that must be selected

text Text describing utitlity that user picked

Argument list: <job_id>

job_id Job id

Argument list: <job_id>

Use a conventional task interface called taskinfo

job_id Id for job to edit. If it is 0 then then enter info for a 'REPOERTED' job

Argument list: <arrayname> <job_id>

This procedure used by db_edit_info

arrayname Array name for task array

job_id Job id

Argument list: <job_id>

job_id Job id

Argument list: <filename> <job_id>

An archive file is one that has been gzipped and move to the poject database directory. Return 0 (no archive file) or 1 (archive file exists)

filename File name (NOT full path name)

job_id Job id.

Argument list: <filename> <job_id> <action>

Either archive or restore filename dependent on action

return a status for the file:

"null" can't find copy of file

"project" file in project directory (ie file with name filename exists)

"archive" archived version of file exists

"both" both project and archive version of file exist

filename Full path file name for file

job_id Job id

action Action required : 'archive' 'restore' or '' (do nothing)

Argument list: <filename> <job_id> <args>

filename File name (can be full path name)

job_id Job id

-ext

Append appropriate extension for compressed file

Argument list: <job_id>

job_id Job id

Argument list: <job_id>

The com files are the scripts which are input ot the programs

job_id Job id

Argument list: <f1> <f2>

See the Tcl documentation on lsort.

f1,f2 The com file names

Argument list: <job_id>

Use the KillScript or KillRemoteScript prcedures in the local.tcl

job_id Job id

Argument list: <job_id>

db_view_session_log Display the session log file

Argument list: None

Autotest Utilities

DbAutotest Handle command line running of autotest

Argument list: <args>

db_autotest Interface to automatic regression testing - rerun jobs in project

Argument list: None

the real work is done by db_autotest_handler

Argument list: <args>

Create a new project directory which is a sub-directory of the current project directory. Create a new database directory and database.def. Run all tasks in the project putting output files into the new project subdirectory.

-nographics

Running in command line (nographics) mode

Argument list: <mode> <job_id> <last_job> <old_project> <project> <args>

While the auto tests are running this procedure calls itself every couple of seconds and test if the current job is finished and if so then starts the next job

mode Mode is either 'init' for first time procedure is called or 'watch' thereafter

job_id The job id of the currently running job

last_job The id of the final job that the procedure is required to run.

old_project The project alias of the master project that is being rerun

project The project alias for the working project

-exit

Exit from ccp4i after completely the tests in this project

-remote machine

Run jobs on remote machine

Argument list: <old_project> <project> <job_id> <run_mode>

Change the directory aliases to that of the working project

old_project The project alias of the master project that is being rerun

project The project alias for the working project

job_id The job id

run_mode Set to 'remote' or 'background' - required to set the def file header

Argument list: <project> <user> <time>

project Alias of project

user Name of user who currently owns the lock

time Time when the lock was created

Argument list: <project>

Socket commands e.g. from running scripts which could not be executed immediately due to the main process being unaccessible, may be stored in a buffer file. DbFlushCommandBuffer flushes the buffer file and executes the commands using the "ccp4ish -socket" mechanism.

project Project alias

Handling Delayed Execution of Buffered Commands

DbGetBufferFile Return the name of the database command buffer file

Argument list: <project>

Clone of GetBufferFile in execute.tcl

project Alias for project

Starting Jobs (src/runjob.tcl)

Starting Jobs

The procedures in runjob.tcl will initiate the running of a job after the user has clicked the 'Run' button on a task interface.

Argument list: <mode> <taskname> <arrayname>

This procedure checks that a script file script/taskname.tcl exists and will fail if it is not found unless running in interactive mode. In future it is feasible that scripts could be written in other languages and run_command would need to be intelligent about this to ensure the script was started appropriately. There is currently some unused code to handle the case of csh script.

mode can be: background: run in background as separate process on same machine, edit: as background but show user program scripts to edit, remote: Give user options to choose to run remote/batch, interactive: create the database entry but return control to the task interface to run the job interactively, #d_arg taskname The name of the task

arrayname Task interface array

Argument list: <mode> <format> <def_file> <script_file> <job_id>

The command to run start a ccp4i process is taken from the configure array - inappropriate setting of this value is a likely cause for jobs not running. The RunRemote procedure is invoked to run jobs remotely.

mode = background or remote

def_file Name of def file containing parameters for job

script_file Script file - usually only required for running Python

job_id The project database id for the job

Argument list: <arrayname> <input_files> <output_files>

A list of array elements for the names of input and output files are keep in the array elements INPUT_FILES and OUTPUT_FILES respectively.

arrayname Task interface array

input_files List of array elements containing the names of input files

output_files List of array elements containing the names of output files

Argument list: <arrayname> <file>

arrayname Task interface array

filename Name of one input file

Argument list: <arrayname> <file>

arrayname Task interface array

filename Name of one output file

Argument list: <arrayname>

arrayname Task interface array

Argument list: <arrayname> <text>

only used in few tasks so far

arrayname Task interface array

text text to insert in task title

Argument list: <script_file> <job_id> <args>

script_file Name of def file containing task parameters

job_id The project database id for the job

-machine machine

Set a default remote machine

Argument list: <w> <arrayname> <action> <def_file> <job_id> <args>

There is a fragment of code to run in 'SERVER' mode - this would be an implementation to allow running 'remotely' on the Windows platform.

w The id of the window

arrayname 'Remote' window array

action User selection: 'abort' or 'run'

def_file Name of def file containing task parameters

job_id The project database id for the job