Ideas

Treasure Data's primary idea portal. 

Submit your ideas & feature requests directly to our product requirements team! We look forward to hearing from you.

Support new last_path feature to sort timestamp of last modification for File Input Plugins

Currently, File Input Plugins including S3 and GCS and etc. supports only last_path for incremental load.

But, there are difficult cases to all file to sort files by filename correctly.

For example,

several fluentd server send logs to S3, and load them by DataConnector hourly or daily.

When some fluentd servers has a problem, and send logs in delay.

Then file name might include old timestamp, then dataconnector couldn't pick up it.

 

Expected Results

Support a feature to sort timestamp of last modification for incremental load instead of last_path.

  • Toru Takahashi
  • Jun 8 2017