Change Details

This page and subpages document the installation of different deep learning pipelines at the BIOP to make use of GPUs As the installation of these can be tedious and is usually not well documented, we have gathered a lot of documentation here to help ourselves as well as people who may not have the time to dedicate two days to find out the missing dependencies //everyone// knows about but them. We brake this into two different sections ## Installation through `virtualenv` Virtualenv installations are lightweight and easy to understand (1 folder, one virtual environment ). Don't need it anymore, simply delete the directory. The setup is slightly more difficult as it involves installing CUDA and CuDNN and working out kinks in the windows environment variables. But as is outlined below, to compile most of the bleeding edge things, you need to do this anyway... ## Installation through `miniconda` Miniconda simplifies the part about installing CUDA and CuDNN which allows for CARE or N2V to be installed very conveniently. Unfortunately StarDist and CellPose for example, need these things to be configured on a system-wide way. So overall the benefit of using conda is minimal. Enabling GPU Support for Windows 10 = Prerequisites = # Get latest [[https://www.geforce.com/drivers|NVIDIA drivers]] # Get [[https://developer.nvidia.com/cuda-10.0-download-archive|CUDA 10.0 Toolkit]] # Get [[https://developer.nvidia.com/rdp/cudnn-download|cuDNN for CUDA 10.0]] (Needs you to login) # Get [[https://www.python.org/downloads/|Python 3.7 x64]] # Get [[https://nodejs.org/en/|NodeJS ]] for managing Jupyter Lab Install everything with defaults except Python. Make sure to add Python to the PATH when offered to. Everything installs automatically **except for cuDNN**. For cuDNN, copy paste the contents of the zip file into `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0` WARNING: You currently cannot go higher than CUDA 10.0, so make sure you downloaded that version! == Stardist Prerequisites == IMPORTANT: For Stardist You have an additional prerequisite! For Stardist to compile with `pip`, you need to install [[ https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019 | Build Tools for Visual Studio 2019 ]] The minimum** individual components** that you need are the following: # `MSVC v142 - VS 2019 C++ x64/x86 build tools` # `Windows 10 SDK (10.0.18362.0)` After this, you may need to restart your PC. == CellPose prerequisites == IMPORTANT: CellPose requires the prerequisites from StarDist above and more You will need to install Intel's MKL # [[ https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library/choose-download/windows.html | Intel® Math Kernel Library (Intel® MKL) for Windows]] IMPORTANT: You need to create an environment variable called CONDA_DLL_SEARCH_MODIFICATION_ENABLE and set any value to it. Otherwise you will run into errors with MKL == Installation of `pip` and `virtualenv` == After installing Python, update `pip` and `virtualenv` Start `cmd` as an administrator and use ``` python -m pip install --upgrade pip pip install virtualenv ``` = Create a CARE virutalenv = NOTE: For this example, we want to create a `CARE-TF-GPU` folder on the `D:\` drive == 1. Create the virtual environement== ``` virtualenv -p python D:\CARE-TF-GPU\ ``` == 2. Activate the virtual environement == ``` d: cd CARE-TF-GPU Scripts\activate ``` == 3. Install the required dependencies == ``` pip install tensorflow-gpu==1.14 nodejs csbdeep scikit-image jupyterlab jupyter_tensorboard ipywidgets widgetsnbextension ``` This will install Jupyter Lab and the most recent CARE package available to `pip`. NOTE: We include `ipywidgets` because we use them on //our// notebooks. `jupyter_tensorboard` is not installed by `csbdeep` so you need to explicitely install it as well. Sometimes we make use of `scikit-image` when reloading data, which is why it is there. `nodejs` is needed for Jupyter Lab and to install the extensions in step 5. == 4. Check that the `jupyter notebook` is loading the right kernel with == ``` jupyter kernelspec list ``` And your output should look like this ``` Available kernels: python3 d:\care-tf-gpu\share\jupyter\kernels\python3 ``` == 5. Install the JupyterLab extensions == ``` jupyter labextension install @jupyter-widgets/jupyterlab-manager & jupyter labextension install jupyterlab_tensorboard ``` = 6. Test = Launch Jupyter Notebook with ``` jupyter lab ``` Make a new Notebook and run a new cell with ```lang=python import tensorflow as tf with tf.device('/gpu:0'): a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a') b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b') c = tf.matmul(a, b) with tf.Session() as sess: print (sess.run(c)) ``` If that works, then TensorFlow is running for you. = Starting a session = Navigate to the folder where you created the virtual environement and go to the command line. ``` Scripts/activate jupyter lab ``` This should open your default browser and launch Jupyter Lab = Magic BAT file for Installation = All the steps we performed here are enclosed in the BAT script below. 1. Copy the lines below. 2. Create a `install-care.bat` file and copy them inside. 3. Modify the `installPath` to where you want to install the virtual environement (**avoid spaces in the name!**). 4. Save and run the `install-care.bat`. No admin rights are needed, normally. ``` @echo off REM Install Path for the Virtual Environement set installPath=D:\CARE-TF-GPU echo %installPath% REM Make sure pip is installed and up to date python -m pip install --upgrade pip REM Install virtualenv if it's not there yet pip install virtualenv REM Finally, create the dsesired virtual environement virtualenv -p python %installPath% REM in order: activate the environement, install all dependencies and test the kernel location cmd /k "cd /D %installPath%\Scripts\ & activate & pip install tensorflow-gpu nodejs csbdeep jupyterlab jupyter_tensorboard ipywidgets widgetsnbextension scikit-image & jupyter labextension install @jupyter-widgets/jupyterlab-manager &jupyter labextension install jupyterlab_tensorboard & jupyter kernelspec list PAUSE ``` = Magic BAT file for running (add as shortcut in desktop, for instance) = ``` @echo off cmd /k "cd /d D:\CARE-TF-GPU & Scripts\activate & jupyter lab" ```

This page and subpages document the installation of different deep learning pipelines at the BIOP to make use of GPUs As the installation of these can be tedious and is usually not fully documented, we have gathered a lot of documentation here to help ourselves as well as people who may not have the time to dedicate two days to find out the missing dependencies //everyone// knows about but them.

This page and subpages document the installation of different deep learning pipelines at the BIOP to make use of GPUs As the installation of these can be tedious and is usually not well documented, we have gathered a lot of documentation here to help ourselves as well as people who may not have the time to dedicate two days to find out the missing dependencies //everyone// knows about but them. We brake this into two different sections ## Installation through `virtualenv` Virtualenv installations are lightweight and easy to understand (1 folder, one virtual environment ). Don't need it anymore, simply delete the directory. The setup is slightly more difficult as it involves installing CUDA and CuDNN and working out kinks in the windows environment variables. But as is outlined below, to compile most of the bleeding edge things, you need to do this anyway... ## Installation through `miniconda` Miniconda simplifies the part about installing CUDA and CuDNN which allows for CARE or N2V to be installed very conveniently. Unfortunately StarDist and CellPose for example, need these things to be configured on a system-wide way. So overall the benefit of using conda is minimal. Enabling GPU Support for Windows 10 = Prerequisites = # Get latest [[https://www.geforce.com/drivers|NVIDIA drivers]] # Get [[https://developer.nvidia.com/cuda-10.0-download-archive|CUDA 10.0 Toolkit]] # Get [[https://developer.nvidia.com/rdp/cudnn-download|cuDNN for CUDA 10.0]] (Needs you to login) # Get [[https://www.python.org/downloads/|Python 3.7 x64]] # Get [[https://nodejs.org/en/|NodeJS ]] for managing Jupyter Lab Install everything with defaults except Python. Make sure to add Python to the PATH when offered to. Everything installs automatically **except for cuDNN**. For cuDNN, copy paste the contents of the zip file into `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0` WARNING: You currently cannot go higher than CUDA 10.0, so make sure you downloaded that version! == Stardist Prerequisites == IMPORTANT: For Stardist You have an additional prerequisite! For Stardist to compile with `pip`, you need to install [[ https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019 | Build Tools for Visual Studio 2019 ]] The minimum** individual components** that you need are the following: # `MSVC v142 - VS 2019 C++ x64/x86 build tools` # `Windows 10 SDK (10.0.18362.0)` After this, you may need to restart your PC. == CellPose prerequisites == IMPORTANT: CellPose requires the prerequisites from StarDist above and more You will need to install Intel's MKL # [[ https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library/choose-download/windows.html | Intel® Math Kernel Library (Intel® MKL) for Windows]] IMPORTANT: You need to create an environment variable called CONDA_DLL_SEARCH_MODIFICATION_ENABLE and set any value to it. Otherwise you will run into errors with MKL == Installation of `pip` and `virtualenv` == After installing Python, update `pip` and `virtualenv` Start `cmd` as an administrator and use ``` python -m pip install --upgrade pip pip install virtualenv ``` = Create a CARE virutalenv = NOTE: For this example, we want to create a `CARE-TF-GPU` folder on the `D:\` drive == 1. Create the virtual environement== ``` virtualenv -p python D:\CARE-TF-GPU\ ``` == 2. Activate the virtual environement == ``` d: cd CARE-TF-GPU Scripts\activate ``` == 3. Install the required dependencies == ``` pip install tensorflow-gpu==1.14 nodejs csbdeep scikit-image jupyterlab jupyter_tensorboard ipywidgets widgetsnbextension ``` This will install Jupyter Lab and the most recent CARE package available to `pip`. NOTE: We include `ipywidgets` because we use them on //our// notebooks. `jupyter_tensorboard` is not installed by `csbdeep` so you need to explicitely install it as well. Sometimes we make use of `scikit-image` when reloading data, which is why it is there. `nodejs` is needed for Jupyter Lab and to install the extensions in step 5. == 4. Check that the `jupyter notebook` is loading the right kernel with == ``` jupyter kernelspec list ``` And your output should look like this ``` Available kernels: python3 d:\care-tf-gpu\share\jupyter\kernels\python3 ``` == 5. Install the JupyterLab extensions == ``` jupyter labextension install @jupyter-widgets/jupyterlab-manager & jupyter labextension install jupyterlab_tensorboard ``` = 6. Test = Launch Jupyter Notebook with ``` jupyter lab ``` Make a new Notebook and run a new cell with ```lang=python import tensorflow as tf with tf.device('/gpu:0'): a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name='a') b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name='b') c = tf.matmul(a, b) with tf.Session() as sess: print (sess.run(c)) ``` If that works, then TensorFlow is running for you. = Starting a session = Navigate to the folder where you created the virtual environement and go to the command line. ``` Scripts/activate jupyter lab ``` This should open your default browser and launch Jupyter Lab = Magic BAT file for Installation = All the steps we performed here are enclosed in the BAT script below. 1. Copy the lines below. 2. Create a `install-care.bat` file and copy them inside. 3. Modify the `installPath` to where you want to install the virtual environement (**avoid spaces in the name!**). 4. Save and run the `install-care.bat`. No admin rights are needed, normally. ``` @echo off REM Install Path for the Virtual Environement set installPath=D:\CARE-TF-GPU echo %installPath% REM Make sure pip is installed and up to date python -m pip install --upgrade pip REM Install virtualenv if it's not there yet pip install virtualenv REM Finally, create the dsesired virtual environement virtualenv -p python %installPath% REM in order: activate the environement, install all dependencies and test the kernel location cmd /k "cd /D %installPath%\Scripts\ & activate & pip install tensorflow-gpu nodejs csbdeep jupyterlab jupyter_tensorboard ipywidgets widgetsnbextension scikit-image & jupyter labextension install @jupyter-widgets/jupyterlab-manager &jupyter labextension install jupyterlab_tensorboard & jupyter kernelspec list PAUSE ``` = Magic BAT file for running (add as shortcut in desktop, for instance) = ``` @echo off cmd /k "cd /d D:\CARE-TF-GPU & Scripts\activate & jupyter lab" ```As the installation of these can be tedious and is usually not fully documented, we have gathered a lot of documentation here to help ourselves as well as people who may not have the time to dedicate two days to find out the missing dependencies //everyone// knows about but them.