Homec4science
Diffusion solarPV (master) Browse

solarPV_identification/output_failed_script.txt
563cfdd05f9bmaster

Last Change
Raw File

output_failed_script.txt
View Options

WARNING: Error in initializing MVAPICH2 ptmalloc library.Continuing without InfiniBand registration cache support.
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
Traceback (most recent call last):
File "model_main.py", line 196, in <module>
set_horovod(divmod(MPI.COMM_WORLD.rank, nnodes)[1], intra_comm)
File "/home/sroquett/solarPV_identification/CNN/mpi_routines.py", line 66, in set_horovod
hvd.init(intra_comm)
File "/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/horovod/common/basics.py", line 50, in init
"Horovod has not been built with MPI support. Ensure MPI is installed and "
ValueError: Horovod has not been built with MPI support. Ensure MPI is installed and reinstall Horovod with HOROVOD_WITH_MPI=1 to debug the build error.
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
Traceback (most recent call last):
File "model_main.py", line 196, in <module>
set_horovod(divmod(MPI.COMM_WORLD.rank, nnodes)[1], intra_comm)
File "/home/sroquett/solarPV_identification/CNN/mpi_routines.py", line 66, in set_horovod
hvd.init(intra_comm)
File "/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/horovod/common/basics.py", line 50, in init
"Horovod has not been built with MPI support. Ensure MPI is installed and "
ValueError: Horovod has not been built with MPI support. Ensure MPI is installed and reinstall Horovod with HOROVOD_WITH_MPI=1 to debug the build error.
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
Traceback (most recent call last):
File "model_main.py", line 196, in <module>
set_horovod(divmod(MPI.COMM_WORLD.rank, nnodes)[1], intra_comm)
File "/home/sroquett/solarPV_identification/CNN/mpi_routines.py", line 66, in set_horovod
hvd.init(intra_comm)
File "/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/horovod/common/basics.py", line 50, in init
"Horovod has not been built with MPI support. Ensure MPI is installed and "
ValueError: Horovod has not been built with MPI support. Ensure MPI is installed and reinstall Horovod with HOROVOD_WITH_MPI=1 to debug the build error.
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
Traceback (most recent call last):
File "model_main.py", line 196, in <module>
set_horovod(divmod(MPI.COMM_WORLD.rank, nnodes)[1], intra_comm)
File "/home/sroquett/solarPV_identification/CNN/mpi_routines.py", line 66, in set_horovod
hvd.init(intra_comm)
File "/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/horovod/common/basics.py", line 50, in init
"Horovod has not been built with MPI support. Ensure MPI is installed and "
ValueError: Horovod has not been built with MPI support. Ensure MPI is installed and reinstall Horovod with HOROVOD_WITH_MPI=1 to debug the build error.
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
Traceback (most recent call last):
File "model_main.py", line 196, in <module>
set_horovod(divmod(MPI.COMM_WORLD.rank, nnodes)[1], intra_comm)
File "/home/sroquett/solarPV_identification/CNN/mpi_routines.py", line 66, in set_horovod
hvd.init(intra_comm)
File "/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/horovod/common/basics.py", line 50, in init
"Horovod has not been built with MPI support. Ensure MPI is installed and "
ValueError: Horovod has not been built with MPI support. Ensure MPI is installed and reinstall Horovod with HOROVOD_WITH_MPI=1 to debug the build error.
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:532: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
Traceback (most recent call last):
File "model_main.py", line 196, in <module>
set_horovod(divmod(MPI.COMM_WORLD.rank, nnodes)[1], intra_comm)
File "/home/sroquett/solarPV_identification/CNN/mpi_routines.py", line 66, in set_horovod
hvd.init(intra_comm)
File "/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/horovod/common/basics.py", line 50, in init
"Horovod has not been built with MPI support. Ensure MPI is installed and "
ValueError: Horovod has not been built with MPI support. Ensure MPI is installed and reinstall Horovod with HOROVOD_WITH_MPI=1 to debug the build error.
Traceback (most recent call last):
File "model_main.py", line 196, in <module>
set_horovod(divmod(MPI.COMM_WORLD.rank, nnodes)[1], intra_comm)
File "/home/sroquett/solarPV_identification/CNN/mpi_routines.py", line 66, in set_horovod
hvd.init(intra_comm)
File "/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/horovod/common/basics.py", line 50, in init
"Horovod has not been built with MPI support. Ensure MPI is installed and "
ValueError: Horovod has not been built with MPI support. Ensure MPI is installed and reinstall Horovod with HOROVOD_WITH_MPI=1 to debug the build error.
Traceback (most recent call last):
File "model_main.py", line 196, in <module>
set_horovod(divmod(MPI.COMM_WORLD.rank, nnodes)[1], intra_comm)
File "/home/sroquett/solarPV_identification/CNN/mpi_routines.py", line 66, in set_horovod
hvd.init(intra_comm)
File "/home/sroquett/tensorflow-1.9/lib/python3.6/site-packages/horovod/common/basics.py", line 50, in init
"Horovod has not been built with MPI support. Ensure MPI is installed and "
ValueError: Horovod has not been built with MPI support. Ensure MPI is installed and reinstall Horovod with HOROVOD_WITH_MPI=1 to debug the build error.
srun: error: r08-nodegpu09: tasks 4-5,7: Exited with exit code 1
srun: Terminating job step 8067649.0
srun: error: r08-nodegpu08: tasks 1-3: Exited with exit code 1
slurmstepd: error: *** STEP 8067649.0 ON r08-nodegpu08 CANCELLED AT 2020-03-09T15:46:08 ***
srun: error: r08-nodegpu09: task 6: Exited with exit code 1
srun: error: r08-nodegpu08: task 0: Exited with exit code 1
c4science · Help