Data Filters

Data filters in the Log Processing.cfg file limit the log entries that are considered in the dataset construction process.

The filters defined in the Log Processing.cfg file include the following:
  • End Time
  • Hash Threshold
  • Start Time

The filtering defined by these parameters occurs after log entries leave the decoders and after transformations but before their evaluation by the Log Entry Condition. See Log Entry Condition. In general, changing any of these parameters results in changes to the composition of the dataset.

The recommended technique for using Sensor data sources to construct a dataset that covers a specific period of time is to use the Start Time and End Time parameters for the dataset.

Using the Start Time and End Time parameters is preferred to other techniques, such as moving log files to separate them by directory. By setting the start and end times for the dataset, the data workbench server automatically uses only those log entries that occurred within the given interval. Assuming that the End Time is in the past, the data workbench server typically updates the dataset using the same set of log entries, even if the dataset is updated by, for example, adding a new transformation.