Add source level schema detection as a new box #213
Conversation
|
Any update on this ? |
|
Sry. Let me check this. |
| import sys | ||
| import os | ||
| import digdag | ||
| sys.path.append('/home/td-user/.local/lib/python3.6/site-packages') |
There was a problem hiding this comment.
This seems strange. It is for local, not server side one.
There was a problem hiding this comment.
No need to have this line for the latest image digdag/digdag-python:3.7
| os.system(f"{sys.executable} -m pip install --user pyyaml") | ||
| os.system(f"{sys.executable} -m pip install --user json") | ||
| os.system(f"{sys.executable} -m pip install --user requests") | ||
| os.system(f"{sys.executable} -m pip install --user digdag") |
There was a problem hiding this comment.
Remove this pip command. Instead of it, please add
import digdag
There was a problem hiding this comment.
With digdag/digdag-python:3.7, you don't need to use --user option anymore.
The following packages are installed:
- pandas
- requests
- pytd (old version, so would be better to use
pip install
also should remove the following packages:
- json (Python standard module)
- time (Python standard module)
- digdag (not on PyPI)
You should write like the following:
os.system(f"{sys.executable} -m pip install -U pytd==0.8.0")
os.system(f"{sys.executable} -m pip install pyyaml")
| headers = {"content-type": "application/json; charset=utf-8","authorization":"TD1 "your apikey""} | ||
| payload = json.dumps(job).encode("utf-8") if isinstance(job, dict) else job | ||
| print(payload) | ||
| with requests.post("https://api.treasuredata.com/v3/bulk_loads/guess", payload, headers=headers) as res: |
There was a problem hiding this comment.
Endpoint shouldn't be static. Please allow users to configure other endpoints.
|
|
||
| def connector_guess(self, job): | ||
|
|
||
| headers = {"content-type": "application/json; charset=utf-8","authorization":"TD1 "your apikey""} |
There was a problem hiding this comment.
Use apikey variable instead of writing apikey directly.
|
I will improve the code by myself later. |
| #Pass apikey as a secret | ||
| +secrets2: | ||
| docker: | ||
| image: "digdag/digdag-python:3.7.3-stretch" |
There was a problem hiding this comment.
Should use the latest image:
| image: "digdag/digdag-python:3.7.3-stretch" | |
| image: "digdag/digdag-python:3.7" |
| import sys | ||
| import os | ||
| import digdag | ||
| sys.path.append('/home/td-user/.local/lib/python3.6/site-packages') |
There was a problem hiding this comment.
No need to have this line for the latest image digdag/digdag-python:3.7
| os.system(f"{sys.executable} -m pip install --user pyyaml") | ||
| os.system(f"{sys.executable} -m pip install --user json") | ||
| os.system(f"{sys.executable} -m pip install --user requests") | ||
| os.system(f"{sys.executable} -m pip install --user digdag") |
There was a problem hiding this comment.
With digdag/digdag-python:3.7, you don't need to use --user option anymore.
The following packages are installed:
- pandas
- requests
- pytd (old version, so would be better to use
pip install
also should remove the following packages:
- json (Python standard module)
- time (Python standard module)
- digdag (not on PyPI)
You should write like the following:
os.system(f"{sys.executable} -m pip install -U pytd==0.8.0")
os.system(f"{sys.executable} -m pip install pyyaml")
| out: | ||
| type: td_internal | ||
| apikey: apikey | ||
| endpoint: api.treasuredata.com |
There was a problem hiding this comment.
Need to parameterize.
| incremental: false | ||
| out: | ||
| type: td_internal | ||
| apikey: apikey |
There was a problem hiding this comment.
Need to parameterize.
No description provided.