You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository has been archived by the owner on Sep 4, 2020. It is now read-only.
I changed the time range in dbg-data-preparation.ipynb to 30/04/2020 - 31/07/2020, and the ticker in dbg-stock-clustering.ipynb from BMW to VOW3.
In dbg-custom-rnn.ipython, I changed the path in ECR Repository as follow.
# Define model artifact name and image
account = session.boto_session.client('sts').get_caller_identity()['Account']
region = session.boto_session.region_name
image = '{}.dkr.ecr.{}.amazonaws.com/{}:latest'.format(account, region, artifactname)
os.chdir("/home/ec2-user/SageMaker/container")
!sh build_and_push.sh $artifactname
I ran the Model training code cell and hit the SyntaxError below.
Could anyone explain for me this error?
Parameter image_name will be renamed to image_uri in SageMaker Python SDK v2.
's3_input' class will be renamed to 'TrainingInput' in SageMaker Python SDK v2.
2020-08-12 15:58:15 Starting - Starting the training job...
2020-08-12 15:58:17 Starting - Launching requested ML instances......
2020-08-12 15:59:38 Starting - Preparing the instances for training...
2020-08-12 16:00:16 Downloading - Downloading input data...
2020-08-12 16:00:22 Training - Downloading the training image...
2020-08-12 16:01:13 Uploading - Uploading generated training model.2020-08-12 16:01:08.786584: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcudart.so.10.1'; dlerror: libcudart.so.10.1: cannot open shared object file: No such file or directory
2020-08-12 16:01:08.786650: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
Starting the training.
Hyperparameters file : {"target_stock": "BMW", "lag": "10", "interval": "D", "batch_size": "4096", "covariate_columns": "StartPrice, MinPrice, MaxPrice", "percent_train": "85.0", "covariate_stocks": "CON, DAI, PAH3, VOW3", "dropout_ratio": "0.1", "num_epochs": "1000", "target_column": "EndPrice", "horizon": "5", "num_units": "256"}
Hyperparameters initialized
Loading data from : /opt/ml/input/data/training/resampled_stockdata.csv
Loading data from : /opt/ml/input/data/training/resampled_stockdata.csv
Training data loaded
100 Stock symbols found.
Records for 65 trading days found.
0-CON#0111-DAI#0112-PAH3#0113-VOW3
Exception during training: invalid syntax (core.py, line 314)
Traceback (most recent call last):
File "/opt/program/train", line 255, in train
traindata.to_csv(os.path.join(model_path, trainfile))
File "/usr/local/lib/python3.5/dist-packages/pandas/core/generic.py", line 3020, in to_csv
formatter.save()
File "/usr/local/lib/python3.5/dist-packages/pandas/io/formats/csvs.py", line 157, in save
compression=self.compression)
File "/usr/local/lib/python3.5/dist-packages/pandas/io/common.py", line 344, in _get_handle
from s3fs import S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/__init__.py", line 1, in <module>
from .core import S3FileSystem, S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/core.py", line 8, in <module>
from fsspec import AbstractFileSystem
File "/usr/local/lib/python3.5/dist-packages/fsspec/__init__.py", line 10, in <module>
from .mapping import FSMap, get_mapper
File "/usr/local/lib/python3.5/dist-packages/fsspec/mapping.py", line 2, in <module>
from .core import url_to_fs
File "/usr/local/lib/python3.5/dist-packages/fsspec/core.py", line 314
out[0] = (f"{out[0][1]}://", out[0][1], out[0][2])
^
SyntaxError: invalid syntax
2020-08-12 16:01:19 Failed - Training job failed
UnexpectedStatusException Traceback (most recent call last)
<timed exec> in <module>
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in fit(self, inputs, wait, logs, job_name, experiment_config)
497 self.jobs.append(self.latest_training_job)
498 if wait:
--> 499 self.latest_training_job.wait(logs=logs)
500
501 def _compilation_job_name(self):
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/estimator.py in wait(self, logs)
1114 # If logs are requested, call logs_for_jobs.
1115 if logs != "None":
-> 1116 self.sagemaker_session.logs_for_job(self.job_name, wait=True, log_type=logs)
1117 else:
1118 self.sagemaker_session.wait_for_job(self.job_name)
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in logs_for_job(self, job_name, wait, poll, log_type)
3075
3076 if wait:
-> 3077 self._check_job_status(job_name, description, "TrainingJobStatus")
3078 if dot:
3079 print()
~/anaconda3/envs/python3/lib/python3.6/site-packages/sagemaker/session.py in _check_job_status(self, job, desc, status_key_name)
2669 ),
2670 allowed_statuses=,
-> 2671 actual_status=status,
2672 )
2673
UnexpectedStatusException: Error for Training job dbg-custom-rnn-D-BMW-2020-08-12-15-58-15-812: Failed. Reason: AlgorithmError: Exception during training: invalid syntax (core.py, line 314)
Traceback (most recent call last):
File "/opt/program/train", line 255, in train
traindata.to_csv(os.path.join(model_path, trainfile))
File "/usr/local/lib/python3.5/dist-packages/pandas/core/generic.py", line 3020, in to_csv
formatter.save()
File "/usr/local/lib/python3.5/dist-packages/pandas/io/formats/csvs.py", line 157, in save
compression=self.compression)
File "/usr/local/lib/python3.5/dist-packages/pandas/io/common.py", line 344, in _get_handle
from s3fs import S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/__init__.py", line 1, in <module>
from .core import S3FileSystem, S3File
File "/usr/local/lib/python3.5/dist-packages/s3fs/core.py", line 8, in <module>
from fsspec import AbstractFileSystem
File "/usr/local/lib/python3.5/dist-packages/fsspec/__init__.py", line 10, in <module>
from .mapping import FSMap, get_mapper
File "/usr/local/lib/python3.5/dist-packages/fsspec/map
The text was updated successfully, but these errors were encountered:
sylviahangnguyen
changed the title
Issue while training model
SyntaxError in core.py while running Model training cell in dbg-custom-rnn.ipython
Aug 12, 2020
Sign up for freeto subscribe to this conversation on GitHub.
Already have an account?
Sign in.
I changed the time range in
dbg-data-preparation.ipynb
to 30/04/2020 - 31/07/2020, and the ticker indbg-stock-clustering.ipynb
from BMW to VOW3.In
dbg-custom-rnn.ipython
, I changed the path in ECR Repository as follow.I ran the Model training code cell and hit the SyntaxError below.
Could anyone explain for me this error?
The text was updated successfully, but these errors were encountered: