Treasure Data's primary idea portal. 

Submit your ideas & feature requests directly to our product requirements team! We look forward to hearing from you.

Incremental Load in Workflows without creating new data connector



I was wondering if there are any current plans to simplify adding incremental load in workflows than the current method which involves creating a separate data connector per table?


Currently it’s easy to import new tables using the replace or append method, but when you want to do incremental load, a new connector needs to be created per table. Are there plans on reducing this extra step and just having the wf create the ‘last_record’ parameter automatically, without having to do it with a data connector?



  • Brandyn Abrams
  • May 4 2017
  • Under Review
Active Requests?
Product Component Workflow Core
  • Brandyn Abrams commented
    June 14, 2017 19:48

    Awesome! Are other companies running into similar issues?

    We load all of our data in TD from each of our dbs .. and doing a select * everyday definitely slows down the process.

    Incremental loads would definitely be a lot more efficient, but creating & managing connectors for each table in each db on top of managing workflows would definitely be a headache.. having this process automated without creating additional connectors in workflow would be great.. if possible..