Repository navigation
tf_vae.json empty after running vae_train.py #31
Description
Greetings,
I am trying to reproduce the experiment on a DGX station I currently have access to, and the fist two steps looks alright, but the result of the command:
$ python vae_train.py
...
step 298000 35.82913 3.7688284 32.0603
step 298500 34.947067 2.9355032 32.011562
step 299000 35.83263 3.8249977 32.007633
step 299500 36.45114 4.418231 32.03291
step 300000 35.098816 3.0974069 32.001408
step 300500 35.483387 3.4664068 32.01698
step 301000 35.43274 3.4285662 32.004173
is an empty array:
$ cat tf_vae/vae.json
[]
According to the documentation the model should be saved on that file, so any hint about where to look for the problem is appreciated.
Thanks,
Alfredo
PS: I am using the following Dockerfile to recreate the environment in the paper, in case in might be relevant:
FROM tensorflow/tensorflow:1.8.0-gpu-py3
# gym-doom requirements
RUN apt-get update && apt-get install -y --no-install-recommends \
cmake \
zlib1g-dev \
libjpeg-dev \
libboost-all-dev \
gcc \
libsdl2-dev \
wget \
unzip \
python3-tk \
&& \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
# make python3 the default
RUN update-alternatives --remove python /usr/bin/python2 && \
update-alternatives --install /usr/bin/python python /usr/bin/python3 10
# NOTE overriding numpy version to match the paper's
# NOTE numpy==1.13.3 gives an error importing vizdoom
RUN pip install --upgrade pip && \
pip install --no-cache-dir --user --upgrade \
gym==0.9.4 \
ppaquette-gym-doom==0.0.6 \
cma==2.2.0 \
mpi4py==2.0.0
ENTRYPOINT ["/bin/bash"]
Activity
leekwoon commentedon Sep 4, 2019
Hi,
I think the problem comes from the location of
with tf.variable_scope('conv_vae', reuse=self.reuse):in __init functionI addressed this problem by moving it to _builg_graph function
asolano commentedon Sep 19, 2019
Thanks for your suggestion, @leekwoon.
I no longer have access to the DGX station but I tried the change in a AWS instance. For now it looks good, the vae.json file was generated.
Do you have a fork or pull request to check out for other necessary changes before continuing the training? I suspect there may be more troubles ahead.
asolano commentedon Sep 27, 2019
FWIW, I did find that step 3 of the GPU jobs showed some problems with the patched code so after a bit of failed troubleshooting I just decided to go back to a commit around the time the paper was published (c0cb2de) and try again. Everything worked as expected without changing any code. 👍
hardmaru commentedon Mar 5, 2020
Thanks for the testing, @asolano. Maybe I should just roll back the code to that time...