Ideas

Treasure Data's primary idea portal. 

Submit your ideas & feature requests directly to our product requirements team! We look forward to hearing from you.

td workflow (CLI) worked with ruby simple proxy and proxy with authentication(id,password)

Hi, I test Treasure-data as evaluation user and td workflow today.

My work environment has proxy with auth(id, password), so i test simple ruby proxy.

I saw url bellow, then wrote it.

http://magazine.rubyist.net/?0002-WEBrickProxy

please prepare with command bellow.

gem install webrick

then wrote down it bellow (please save it as "prox.rb")

require 'webrick'

require 'webrick/httpproxy'

require 'uri'

#http://magazine.rubyist.net/?0002-WEBrickProxy

# make proxy server object

#s = WEBrick::HTTPProxyServer.new({})

s = WEBrick::HTTPProxyServer.new(

# bind address (default:nil)

:BindAddress => '127.0.0.1',

# port number(default:80)

:Port => 8080,

#log(default]:nil)

:Logger => WEBrick::Log::new("log.txt", WEBrick::Log::DEBUG),

# notice with header if it come through proxy proxy(default:true)

:ProxyVia => false,

# parent proxy(default: nil)

:ProxyURI => URI.parse('http://userid:password@mycompanyproxy:port/')

)

# capture SIGINT

Signal.trap('INT') do

   # if captured , shutdown.

   s.shutdown

end

# invoke server

s.start

---end of prox.rb

Then run it with command line bellow.

# proxy environment variable set properly.

ruby prox.rb

On another window, i set environment variable like bellow.

export HTTP_PROXY="127.0.0.1:8080”

export HTTPS_PROXY="127.0.0.1:8080"

so, I test td workflow.

bellow are proxy server log:

# localhost - - [23/Feb/2017:11:27:09 JST] "GET http://toolbelt.treasuredata.com/latest_version.pkg HTTP/1.1" 200 6

# - -> http://toolbelt.treasuredata.com/latest_version.pkg

# localhost - - [23/Feb/2017:11:27:11 JST] "GET http://toolbelt.treasuredata.com/digdag/jdk/mac_x64 HTTP/1.1" 302 0

# - -> http://toolbelt.treasuredata.com/digdag/jdk/mac_x64

# localhost - - [23/Feb/2017:11:27:11 JST] "CONNECT d3hq3slxjftf62.cloudfront.net:443 HTTP/1.1" 200 0

# - -> d3hq3slxjftf62.cloudfront.net:443

# localhost - - [23/Feb/2017:11:27:11 JST] "GET http://toolbelt.treasuredata.com/digdag/jdk/mac_x64 HTTP/1.1" 302 0

# - -> http://toolbelt.treasuredata.com/digdag/jdk/mac_x64

# localhost - - [23/Feb/2017:11:27:11 JST] "CONNECT d3hq3slxjftf62.cloudfront.net:443 HTTP/1.1" 200 0

# - -> d3hq3slxjftf62.cloudfront.net:443

# localhost - - [23/Feb/2017:11:27:27 JST] "GET http://toolbelt.treasure-data.com/digdag?user=userid%40mycompany.co.jp HTTP/1.1" 302 0

# - -> http://toolbelt.treasure-data.com/digdag?user=userid%40mycompany.co.jp

# localhost - - [23/Feb/2017:11:27:27 JST] "CONNECT dl.digdag.io:443 HTTP/1.1" 200 0

# - -> dl.digdag.io:443

# localhost - - [23/Feb/2017:11:27:28 JST] "CONNECT dl.bintray.com:443 HTTP/1.1" 200 0

# - -> dl.bintray.com:443

# localhost - - [23/Feb/2017:11:27:29 JST] "CONNECT akamai.bintray.com:443 HTTP/1.1" 200 0

# - -> akamai.bintray.com:443

I tested shown bellow.

1) td workflow init test

    cd test

   td workflow run test

2) td workflow check

3) sample query workflow of "nasdaq_analysis"

curl -o nasdaq_analysis.zip -L https://gist.github.com/danielnorberg/f839e5f2fd0d1a27d63001f4fd19b947/raw/d2d6dd0e3d419ea5d18b1c1e7ded9ec106c775d4/nasdaq_analysis.zip

unzip nasdaq_analysis.zip

cd nasdaq_analysis

4) td workflow:reset

5) and rerun above.

worked fine!.

 

if td command complain with proxy, please unset HTTP_PROXY and unset HTTPS_PROXY, and then reexec.

 

 

  • katsumi imaizumi
  • Feb 23 2017
  • Under Review
Active Requests?
Product Component Workflow Core