scdbstrip

Clean up a database from event and waveform quality parameters.

Description

SeisComP’s scmaster is continuously writing to the database. This causes the database to grow and to occupy much space on the harddisc. scdbstrip taggles this problem and removes processed objects from the database older than a configurable time span. The time comparison considers the object time, not the time of their creation.

The parameters which scdbstrip removes are

• Event parameters including events, origins, magnitudes, amplitudes, arrivals, picks, focal mechanisms, moment tensors

• Waveform quality control (QC) parameters.

scdbstrip will remove all events with an origin time and QC parameters older or younger than specified. Default is ‘older’. It will also remove all associated objects such as picks, origins, arrivals, amplitudes and so on.

scdbstrip does not run as a daemon. To remove old objects continuously scdbstrip should be added to the list of cronjobs running every e.g. 30 minutes. The more often it runs the less objects it has to remove and the faster it will unlock the database again. The timing and the parameters to be removed is controlled by module configuration or command-line options.

Hint

• For removing specific parameters and not all in a time range, use scdispatch along with XML files created by scxmldump and scqueryqc for event parameters and waveform QC parameters, respectively.

• For removing data availability parameters use scardac.

Known Issues

When running scdbstrip for the first time on a large database it can happen that it aborts in case of MYSQL with the following error message:

[  3%] Delete origin references of old events...08:48:22 [error]
execute("delete Object from Object, OriginReference, old_events where
Object._oid=OriginReference._oid and
OriginReference._parent_oid=old_events._oid") = 1206 (The total number
of locks exceeds the lock table size)

Exception: ERROR: command 'delete Object from Object, OriginReference,
old_events where Object._oid=OriginReference._oid and
OriginReference._parent_oid=old_events._oid' failed

That means your MYSQL server cannot hold enough data required for deletion. There are two solutions to this:

1. Increase the memory pool used by MYSQL by changing the configuration. The minimum is 64 MBytes but modern system typically have a larger default:

   innodb_buffer_pool_size = 64M
   

   The size of the new buffer depends on the size of the database that should be cleaned up. Read also the section Database Server Configuration. It provides more options for optimizing your database server.

2. Run scdbstrip on smaller batches for the first time:

   $ scdbstrip -d mysql://sysop:sysop@localhost/seiscomp --days 1000
   $ scdbstrip -d mysql://sysop:sysop@localhost/seiscomp --days 900
   ...
   $ scdbstrip -d mysql://sysop:sysop@localhost/seiscomp --days 100
   

Hint

In the examples, database connection parameters correspond to default values. You may thus replace -d mysql://sysop:sysop@localhost/seiscomp by -d localhost or -d mysql://.

Examples

• Remove event and waveform quality parameters older than 30 days

  scdbstrip -d mysql://sysop:sysop@localhost/seiscomp --days 30
  

• Remove event and waveform quality parameters newer than 30 days

  scdbstrip -d mysql://sysop:sysop@localhost/seiscomp --days 30 -i
  

• Only remove waveform QC parameters older than 30 days but no others

  scdbstrip -d mysql://sysop:sysop@localhost/seiscomp --days 30 --qc-only
  

• Remove event and waveform quality parameters before 2000-01-01 12:00:00

  scdbstrip -d mysql://sysop:sysop@localhost/seiscomp --datetime 2000-01-01T12:00:00
  

• Remove event and waveform quality parameters after 2000-01-01 12:00:00

  scdbstrip -d mysql://sysop:sysop@localhost/seiscomp --datetime 2000-01-01T12:00:00 -i
  

• Remove event and waveform quality parameters between 2000-01-01 12:00:00 ~ 2000-01-01 14:00:00

  scdbstrip -d mysql://sysop:sysop@localhost/seiscomp --time-window 2000-01-01T12:00:00~2000-01-01T14:00:00
  

• Remove event and waveform quality parameters before 2000-01-01 12:00:00 and after 2000-01-01 14:00:00

  scdbstrip -d mysql://sysop:sysop@localhost/seiscomp --time-window 2000-01-01T12:00:00~2000-01-01T14:00:00 -i
  

Module Configuration

etc/defaults/global.cfg

etc/defaults/scdbstrip.cfg

etc/global.cfg

etc/scdbstrip.cfg

~/.seiscomp/global.cfg

~/.seiscomp/scdbstrip.cfg

scdbstrip inherits global options.

database.cleanup.invertMode

Default: false

Type: boolean

Invert the selection of the specified time period, that is delete all parameters after the specified time period, not before. When a date range is specified, then delete all parameters before and after the time range, not in between.

database.cleanup.eventParameters

Default: true

Type: boolean

Strip all event parameters including events, origins, magnitudes, amplitudes, arrivals, picks, focal mechanisms.

database.cleanup.qualityControl

Default: true

Type: boolean

Strip waveform quality control (QC) parameters.

Note

database.cleanup.keep.* Parameters controlling the time to keep objects in the database. The time comparison considers the object time, not the time of their creation.

database.cleanup.keep.days

Default: 30

Type: int

The number of days to preserve in the database. This value is added to the whole timespan. Hours and minutes are configured separately.

database.cleanup.keep.hours

Default: 0

Type: int

The number of hours to preserve in the database. This value is added to the whole timespan. Days and minutes are configured separately.

database.cleanup.keep.minutes

Default: 0

Type: int

The number of minutes to preserve in the database. This value is added to the whole timespan. Days and hours are configured separately.

Command-Line Options

scdbstrip [options]

Generic

-h, --help

Show help message.

-V, --version

Show version information.

--config-file arg

Use alternative configuration file. When this option is used the loading of all stages is disabled. Only the given configuration file is parsed and used. To use another name for the configuration create a symbolic link of the application or copy it. Example: scautopick -> scautopick2.

--plugins arg

Load given plugins.

--first-new

Overrides configuration parameter firstNew.

Verbosity

--verbosity arg

Verbosity level [0..4]. 0:quiet, 1:error, 2:warning, 3:info, 4:debug.

-v, --v

Increase verbosity level (may be repeated, eg. -vv).

-q, --quiet

Quiet mode: no logging output.

--component arg

Limit the logging to a certain component. This option can be given more than once.

-s, --syslog

Use syslog logging backend. The output usually goes to /var/lib/messages.

-l, --lockfile arg

Path to lock file.

--console arg

Send log output to stdout.

--debug

Execute in debug mode. Equivalent to --verbosity=4 --console=1 .

--log-file arg

Use alternative log file.

Database

--db-driver-list

List all supported database drivers.

-d, --database arg

The database connection string, format: service://user:pwd@host/database. "service" is the name of the database driver which can be queried with "--db-driver-list".

--config-module arg

The config module to use.

--inventory-db arg

Load the inventory from the given database or file, format: [service://]location .

--db-disable

Do not use the database at all

Mode

--check

Checks if unreachable objects exist.

--clean-unused

Remove all unreachable objects when in checkmode. Default: off.

Objects

-E, --ep-only

Strip only event parameters. Other parameters, like QC, are are ignored.

-Q, --qc-only

Strip only waveform quality control (QC) parameters. Other parameters, like event parameters, are are ignored. Overrides ‘eq-only’.

Timespan

--days arg

Overrides configuration parameter database.cleanup.keep.days.

--hours arg

Overrides configuration parameter database.cleanup.keep.hours.

--minutes arg

Overrides configuration parameter database.cleanup.keep.minutes.

--datetime arg

Replaces the days:hours:minutes timespan definition by an arbitrary absolute timestamp in UTC. Format: %Y-%m-%dT%H:%M:%S.

-t, --time-window arg

Delete objects in the specified time window. Replaces the days:hours:minutes timespan definition by an arbitrary absolute time range in UTC. Format: startTime~endTime that is %Y-%m-%dT%H:%M:%S~%Y-%m-%dT%H:%M:%S

-i, --invert

Overrides configuration parameter database.cleanup.invertMode.

--keep-events

IDs of events to keep in the database separated with comma.