Treasure Data's primary idea portal.
Submit your ideas & feature requests directly to our product requirements team! We look forward to hearing from you.
Currently DataConnector S3 accepts bucket and prefix for input data file.
Specify import files with pattern not prefix, like YYYYMMDD_filename.tsv, is more controllable
I confirmed it works with data connector.
Thank you for the support!
Currently we can only specify forward match pattern at config:in:path_prefix in load.yml. So we have to separate bucket or use some same forward pattern for multiple files for a load operation.
- path_prefix: bucket_name
-->we have to create and manage so many bucket
- path_prefix: bucket_name/fielprefix
--> imports bucket_name/fielprefix* files.
- path_prefix: bucket_name/*_postfix.tsv
--> allow backward match
- path_prefix: bucket_name/*_patern.*
--> middle pattern match
This flexibility in file specification is strongly related to this feedback http://treasuredata.uservoice.com/forums/313566-general/suggestions/9463095--dataconnector-override-yml-file-parameter-by-spe .
Dataconnector operation is straight forward and required formal way. That prevent customer from moving to dataconnector from td import. Because many customers are now control their workflow with some batch script way. td import has executing parameter, so it works easer with such batch scripts than yml config of dataconnector.
You won't be notified about changes to this idea.