2. Understanding FileTrain

This section will give you a more in-depth understanding on how FileTrain's individual parts work and how they all, when put together, will work as a powerful tool for your digital workflow.

FileTrain is built with stations. A station is a holder for the building components sources, filters and actions. These individual components are put together within the station to work together. The three core components (source, filter and action) are explained in more detail below. The components can be re-used in different stations to make a complex workflow more understandable and simplify the maintainance. Each individual component (source, filter or action) executes it's own task within the framework of a station. Without the station the individual components can not work.

The core components are as stated unaware of other components. This feature gives the user possibillities to build very advanced workflows with little effort but it requires that the user know how the individual components work.

A source is a component that will search for objects. The most common type of source is a folder on the local drive or on a mounted network drive. Other sources may for example be a folder on a remote FTP server, an email account on a remote email server or a database.

FileTrain has four different built-in sources (see below).

Other sources such as database lookups, synchronization sources or other special sources may be developed as plugins to FileTrain. For more information about an individual source type, see the configuration section.

A folder source will look for objects in a folder on a local or remote network drive. The folder source is able to look down in a sub folder structure to detect objects in a sub level of the root folder.

The FTP source will use the FTP protocol to look for objects on a remote FTP server.

The email source is scanning an email server for emails. Special email filters will apply to such a source.

The timer source is a special type of source which is not actually looking for any files, folders or emails. This source will work as a alarm clock. When the alarm goes off this source will notify the station and the station may perform special tasks which do not require any objects to execute action upon. This may for example be used to send a daily report at a given time.

Note that the special timer filter is required for useful operations when using this timer source.

A filters is a component responsible for filtering the objects registered in a source. Different kind of objects may therefore be handled differently.

There are five built-in filter types in FileTrain, some of which will work with any kind of source and some are useful only on some sources. More advanced filters may be developed as plug-ins.

For more detailed coverage of the different filters, see the configuration section.

The email filter will apply to emails from an email source. Note that individual files will not be handled separately. The email will be treated as one object and handled as one. Do not use this filter if you wish to execute actions on attachments in an email. The actions following this filter will only have the actual email in scope (email body, sender etc) when they execute. This may be useful if you like to save text files with the email content.

If you like to execute actions on attachments you should use the file filter instead.

The file filter is filtering regular files, located locally or remotely on FTP or email servers.

With this filter you may set restrictions such as file name, file size, certain internal properties on images, PDF files or MP3 files and more.

Similar to the file filter but this filter will only handle folders.

This filter is a special filter which is needed if the station is holding a timer source. This filter is not a regular filter in terms of restrictions. A file filter may for example have a restriction to only let files with file suffix 'jpg' go through the filter. The timer filter will not have any restrictions but will only be passed if the source is a timer source that fires.

The actions are the components in FileTrain which will invoke a special handling upon an object, e.g. moving a file. In Many different kinds of actions are shipped with FileTrain. Other actions such as advanced database handling may be developed as plugins.

For more detailed information about each individual action, see the configuration section.

The three main component types; source, filter and actions, are put together in a station. The sources and actions may, depending on their types, experience problems when they are working, e.g. a folder source might not find the root folder or a FTP action may not be able to connect to the FTP server. When such an error occur the user may setup special actions for these events. These special actions are called error handling actions.

All configuration parts in FileTrain are collected under the configuration node in the main window of FileTrain. Here you can see all the individual components as well as the stations. To edit a certain component you may double click on the item and the corresponding setup window is opened. In the setup window for stations there are lists with the individual components, these components are also clickable for quick editing.

When a station is created or edited the station setup dialog will appear, see image below.

In the station setup window all the sources, filters and actions are shown. There are also a special tab for error actions (the error list depends on the type of sources and actions in the station).

The name and description of a station is only for your personal use and will have no impact on the workflow.

Activate on application start should be selected if you would like this station to start automatically when FileTrain starts. There will be a initial built-in delay when FileTrain starts up before the station actually starts in case you want to stop one or more stations that are set to automatically start when FileTrain starts.

For more information about the internal state and cache, please read the section below about cache.

The source list shows all the sources that are present in this station. You may see a list of all existing sources by clicking the 'Show sources' button. You may drag and drop sources from this list to the source list in the station. You may also add a new source with the add button under the source list. If you haven't yet created any source you may create a new source from the add button.

The filter/action tree can be populated with filters and corresponding actions for that filter. To add a filter, drag a filter from the shown filter list (click the 'Show filters' to show this list) or by clicking the add button under the tree. To add actions to a specific filter either drag and drop actions from the action list (click the 'Show actions' to show this list) or by first selecting a filter and then click the add button under the tree.

When all the sources, filters and actions are in place the station is theoretically ready to run. The individual components in a station may however cause errors and these errors can be taken care of in different ways. The station setup dialog has a tab for error handling.

In the error list to the left all the possible errors are seen (with the error popup above the list the errors from individual components may be viewed separately). Selecting an error in the error list will present the error handling action list for that particular error. With the add button under the error handling list an existing action may be added or a new error handling may be created.

The error shown at the top is special error which may occur if actions fails over and over in a station. In the advanced tab you may setup a maximum number of failures that you accept in this station. When this number of failures occurs for a certain object this object will be handled in this special error action.

In the advanced tab you may change how objects are being handled in the station.

The max concurrent handling is default set to one (1). This is the number of objects that may be handled in parallell in this station. If you for example use a source that detects 100 files in one scan, each file is handled separately in the station and only one file is handled at a time. If you change the concurrent handling to for example 5, 5 files will be handled at the same time. Be careful when changing this number. If you set a high number of concurrent handling the CPU usage will increase and the handling may in fact get slower if this number is too high and the sources/actions uses a lot of network communication such as FTP server connection.

Suspension will occur when an action in this station fails its handling. The object that are being handled are being put to suspension for the specified suspension timeout. When this occurs the next object in the queue will be handled in the station. When the suspension time is over the object is being handled again but actions that have already been invoked on the object will not be executed again. An object will be suspended and the action will try to handle the object over and over again. If you like to have a maximum number of suspensions before the object will be handled different (for example you may want to move the object after 10 failures to another folder) you may specify the number in this advanced setup.

 

The bucket is a holder for keys and values and may be very useful if its capabilities are fully understood.

There are two different buckets. Each station has their own global bucket where the values are kept even if FileTrain is stopped. There is also a temporary bucket which is created for each object that is handled.

This bucket feature was introduced in FileTrain version 4.0.7 to enable different actions to share and use data between eachother. The object bucket feature is very powerful and may for example be useful when one action wants to use information from another action.

An object's bucket lifetime starts when an object is detected in a source and started to be handled by a station and ends when all the actions have been executed. This means that for each object that is handled in a station a new bucket is created. The bucket's data is shared among the actions which are invoked on the object. Values that are put into the bucket can be used by any action that is performed after the value is put into the bucket.

A common way to get values from this bucket is with the macro %BUCKET[...]% (see the macro section for more details). Filters may also filter on a bucket value and there is also a special action to set the values for a bucket.

Before an object is being handled by the actions the station itself puts some core information into the bucket. Information about these is found in the table below;

Common bucket values set by all stations

Bucket key	Value description
source.email	If the object being handled is an email the value contains the data about this email. More information for developers can be found in the plug-in API.
source.file	If the object being handled is a file the value contains a reference to this file. More information for developers can be found in the plug-in API.
source.exif.<file path>	If the object being handled contains exif information, this bucket value contains that data. Note that all data is loaded lazily, see the bucket key exifLoaded.
exifLoaded.<file path>	A boolean value to see if the exif data has been loaded or not.
source.imageData.<file path>	If the object being handled contains iamge information, this bucket value contains that data. Note that all data is loaded lazily, see the bucket key imageDataLoaded.
imageDataLoaded.<file path>	A boolean value to see if the image datahas been loaded or not.
source.iptc.<file path>	If the object being handled contains IPTC information, this bucket value contains the IPTC data. Note that all data is loaded lazily, see the bucket key iptcLoaded.
iptcLoaded.<file path>	A boolean value to see if the iptc has been loaded or not.
source.mp3.<file path>	If the object being handled contains MP3 tags, this bucket value contains that data. Note that all data is loaded lazily, see the bucket key mp3Loaded.
mp3Loaded.<file path>	A boolean value to see if the mp3 data has been loaded or not.
source.xmp.<file path>	If the object being handled contains XMP information, this bucket value contains that data. Note that all data is loaded lazily, see the bucket key xmpLoaded.
xmpLoaded.<file path>	A boolean value to see if the XMP has been loaded or not.
sourceFileName	If the currently handled object is a regular file the value will be the name of this file. Normally this is accessible through the macro %F%%E% but some actions may for example work on external files where %F% will refer to this external file name and thus getting the source file name is easily done using this bucket key.
sourceFilePath	If the currently handled object is a regular file the value will be the path to this file (excluding the file name).

As an addition to the object bucket a more global bucket was introduced in FileTrain 6.2. This station bucket is a more static bucket where the values will be saved with the station even if FileTrain is shut down. This station bucket may be used for example to have a running number or an internal log .

The station bucket may be seen in the setup of a station, see image below.

A common way to get values from this bucket is with the macro %STATION_BUCKET[...]% (see the macro section for more details). Filters may also filter on this bucket value and there is also a special action to set the values for the bucket.

 

Each station has a very advanced internal cache system. This cache feature is used to prevent FileTrain to handle objects several times. The cache system will use resources and should be avoided when possible. The cache system consist of two parts, a cache and an internal state.

The internal state is a memory where the station will remember which actions have been executed on each object. The cache is a memory where the station will remember if an object has been handled or not. If you have a station where objects in the source location will be moved or deleted from the source location when FileTrain handles the objects you do not need to use any cache.

In an environment where FileTrain is scanning a source location where the objects will not be moved or deleted you will need to use the cache to avoid that FileTrain handles the objects over and over again.

Each object in a source location is compared with the station's internal cache before it is handled. If the object name is the same but the modification time is changed, the object is handled as a new object in FileTrain. The object is also handled as new if the object size is changed.

Throughout all the setup of FileTrain you will find fields which are yellow instead of white. These fields are macro fields and may contain special macro text. The macro text is special dynamic text. The macro concept is explained in more detail in the macro section. The macro text enables you to setup FileTrain to very advanced workflows.