Ideas

Treasure Data's primary idea portal. 

Submit your ideas & feature requests directly to our product requirements team! We look forward to hearing from you.

Support embulk filter plugins for DataConnector

I wish you to support these plugins.

embulk-filter-column (0.4.0)
embulk-filter-query_string (0.1.2)
embulk-filter-row (0.2.0)
embulk-filter-split_column (0.1.0)
embulk-input-s3 (0.2.8)
embulk-output-td (0.3.2)
embulk-parser-apache-custom-log (0.4.0)
embulk-parser-none (0.2.0)
embulk-parser-query_string (0.3.1)
embulk-parser-regex (0.2.1)

because our log data(apache combined log format) at S3 contains unnecessary data (such as access to static files).

  • Yusuke Okura
  • Jun 10 2016
  • Toru Takahashi commented
    June 10, 2016 23:36

    So, your request is to support the following plugins.

    embulk-filter-column (0.4.0)
    embulk-filter-query_string (0.1.2)
    embulk-filter-row (0.2.0)
    embulk-filter-split_column (0.1.0)
    embulk-parser-apache-custom-log (0.4.0)
    embulk-parser-none (0.2.0)
    embulk-parser-regex (0.2.1)

    If you can write these use-case, please let us know. It would be helpful for our product team to prioritize our development.

  • Toru Takahashi commented
    June 10, 2016 23:36

    Hi, Currently we supports the following in the list.
    We are working on publishing documentation about it.

    - embulk-input-s3 0.2.8
    -- We are using it for data connector for s3. https://docs.treasuredata.com/articles/data-connector-amazon-s3
    - embulk-parser-query_string 0.3.1

    Regarding to embulk-output-td, you can import all data into only TD, and it's used by internal embulk-output-td plugin, which is more faster than embulk-output-td.

    Additionally, we support the following filter, and also we support default filter and parser of Embulk.
    - 'embulk-filter-add_time', '0.2.0'
    - 'embulk-filter-expand_json', '0.1.1'

  • Admin
    Kazuki Ohta commented
    February 4, 2017 05:26

    remove_columns filter was shipped. > https://docs.treasuredata.com/articles/releasenote-20170201#collection-filtering-columns-for-data-connectors