Ideas

Treasure Data's primary idea portal. 

Submit your ideas & feature requests directly to our product requirements team! We look forward to hearing from you.

[fluentd] DNS lookup access reduction

This is feedback from a customer. Please consider solution if possible.

They are uploading data with td-agent 20,000,000,000records per day.
In such situation there are 3,000,000 requests / day to DNS server.

fluent-plugin-td seems to produce each td-client for each connection and try to resolve name.
They are planning to put cache server to relax the load.

But it is better to have some option to reduce DNS access in td-agent side.
There will be some trade offs for keep connection alive.

Following customer's pains are common for enterprise customers with hybrid-cloud(on-premise and cloud) systems.

  • DNS servers are in on-premise and difficult to reinforcement servers in security reasons.
  • DNS is very critical and used by all other services, so huge traffic has big impact for their all systems.
  • Caching locally workaround requires additional operations.

 

  • Ryutaro Yada
  • Jun 23 2016
  • Toru Takahashi commented
    September 2, 2016 07:05

    Filed too. https://github.com/treasure-data/fluent-plugin-td/issues/23