Ideas

Treasure Data's primary idea portal. 

Submit your ideas & feature requests directly to our product requirements team! We look forward to hearing from you.

Ensure all data connectors allow for changing source columns without requiring re-config (remove manual columns option setting in config)

Source data format sometimes changes and customer do not want to change config.yml setting for that. They want to dataconnector support auto columns setting like td import:auto.

 

td import can import following data with same command and no config changes.

 

```

$ cat data1.tsv

cola colb colc
1111 2222 3333
1111 2222 3333
1111 2222 3333
1111 2222 3333
1111 2222 3333
1111 2222 3333
1111 2222 3333

td import:auto --auto-create yada_demo.tsv_test --format tsv --column-header --time-value `date +%s` data1.tsv

 

$ cat data2.tsv
cola colb colc cold
1111 2222 3333 4444
1111 2222 3333 4444
1111 2222 3333 4444
1111 2222 3333 4444
1111 2222 3333 4444
1111 2222 3333 4444
1111 2222 3333 4444
1111 2222 3333 4444
1111 2222 3333 4444

 

td import:auto --auto-create yada_demo.tsv_test --format tsv --column-header --time-value `date +%s` data2.tsv

```

  • Ryutaro Yada
  • Jun 29 2016
  • Under Review
  • Admin
    Rob Parrish commented
    July 31, 2016 22:43

    Many Data Connectors already enable this functionality, but we will be gradually working all data connectors enable for "changing columns" over time. If there is a particular connector you want this change for, please let us know so we can try to prioritize our updates accordingly.