This page aims to contain the setup in order to be able to run the deep learning tools that are currently being used at the BIOP using a Python VENV. = Why VENV? Why do you hate Conda, Oli? = I do not hate Conda, it is very useful and powerful, but when it comes to scientific packages and unreleased GPU-based libraries, I find the "black box, it's OK, it's on Conda-Forge" approach lacking. With venv, the PC itself must be configured properly, which brings an understanding of the underlying dependencies that are installed implicitely with Conda. I am also comfortable doing theis because we are at a stage where things like having multiple CUDA versions and multiple Python versions is not harmful nor particularly difficult to handle. = What this guide will show you = First and foremost, this guide is intended for Windows 10. Linux has enough documentation that I luckily do not need to dive into, as we currently do not have any Linux based workstation at the facility. **You will learn to** 1. Install and manage multiple versions of Python: 3.7, 3.8 and 3.9 2. Install the NVIDIA CUDA libraries for CUDA 10.0, 10.1, 10.2, 11.1, and 11.2 on your PC as well as CuDNN for all these versions 3. Install the Microsoft Build Tools for C++ (Ideally just the bare minimum) 4. Install NodeJS (For Jupyer Lab) = What can you install with this? = This approach has enabled us to install all with GPU support: # StarDist with `gputools` # CSBDeep # Noise2Void # DenoiSeg # CellPose with PyTorch GPU # CellProfiler with Cellpose = What can you not install with this = # YAPIC: There is a dependency that is not available on Windows that makes this fail. But we are running it on our University's GPU nodes = Installation of all dependencies = NOTE: This setup is intended for a workstation with a good GPU and Windows 10 Installed. It has been tested with the following cards: **RTX Titan, GTX 1080, GTX 1080 Ti, RTX 2080 Ti** == Software to Install == - [[ https://www.nvidia.com/Download/index.aspx | Download Latest NVIDIA Drivers ]] - We use the `Studio Drivers` - [[ https://visualstudio.microsoft.com/downloads/#build-tools-for-visual-studio-2019 |Download Build Tools for Visual Studio 2019 ]] - Install the `Build tools for C++`, namely - `MSVC v142 - VS 2019 C++ x64/x86 build tools` - `Windows 10 SDK (10.0.18362.0)` - [[ https://developer.nvidia.com/cuda-toolkit-archive | Download CUDA Toolkits ]] Get the Installations for 10.0, 10.1, 10.2 and 11.2 - Install just the `Developer` and `Runtime` parts {F14514741, width=300, layout=center} - [[ https://developer.nvidia.com/rdp/cudnn-archive | Download CuDNN ]] This [[https://www.tensorflow.org/install/source#gpu| CuDNN Compatibility chart]] lets you know what works and what does not work for TensorFlow, which is the most finnicky of the frameworks. PyTorch is much more lenient, it seems. After you install each CUDA Toolkit, install the appropriate CuDNN. This is how I installed them and why |CUDA Version|CuDNN Version|Why |---------------|------------------|----- |10.0 | 7.6.5 | TensorFlow 1.13 to 1.15 |10.1 | 7.6.5 | TensorFlow 2.1.0 to 2.3.0 |10.2 | 8.1.1 | [[https://pytorch.org/get-started/locally/| PyTorch 1.9 Stable]] for CellPose |11.1 | 8.1.1 | [[https://pytorch.org/get-started/locally/| PyTorch 1.9 Stable]] for CellPose (More recent) |11.2 | 8.1.1 | TensorFlow 2.5.0 to 2.6.0 (Trials) - [[https://www.python.org/downloads/release/python-379/| Install Python 3.7]] Install for all users, and make sure that you also install the Python version manager `py` when prompted. - [[https://www.python.org/downloads/release/python-3810/| Install Python 3.8]] - [[https://www.python.org/downloads/release/python-396/| Install Python 3.9]] - This one is necessary for CellProfiler, for example - [[https://nodejs.org/en/download/|NodeJS]] - Install all defaults. **This is for adding extensions to Jupyter Lab**. == Environment Variables == NOTE: Environment Variables are file and folder locations your operating system looks into when you call a program or library. It is like the index of a library, telling you where to look for different things. In Windows, we can create and edit Environment Variables graphically. # Start typing "Environment" in the Windows searchbar until it suggests "Edit the system environment variables". # Click on {nav Environment Variables...} and use the {nav New...} button all the way down (The one that corresponds to the "System Variables") to create the two following environment variables. |VARIABLE NAME | Path (Default) | |-------------------|-----------------| |INCLUDE| `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include`| |LIB| `C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\lib\x64`| These are needed for compiling PyOpenCL, which is a dependency of `gpuools` for StarDist, but is beneficial for any other tools that use the GPU and may need compiling. NOTE: You will need to click on {nav New...} for each Environment Variable you create. WARNING: If you are compiling something else that isn't StarDist that needs a different version of CUDA, you NEED to change these paths during the compilation. After it is compiled, it will run no matter what these paths are. They are only needed at compiling. Did I mention compiling is the only time these environment variables are needed? In the end it should look like this: {F15073749, width=300, layout=center} Note how there is also a CUDA 10.0 version installed. This is OK and will be made clear below if you install Noise2Void. = Choosing Your Python Version = Because you have multiple Python versions, you can check which are available with the `py --list` command. ``` C:\Users\demo>py --list Installed Pythons found by py Launcher for Windows -3.9-64 * -3.8-64 -3.7-64 ``` So if we need to create a `venv` for python 3.7, we can write `py -3.7 -m venv d:\my-new-venv-py37` When activating the environment using `d:\my-new-venv-py37\Scripts\activate` We will have Python 3.7 backing it. = Creating Virtual Environments = NOTE: Consider upgrading `pip` to the latest vesion before starting: `python -m pip install --upgrade pip` === Windows Command Prompt `cmd` === IMPORTANT: We are running all the commands below from the **Windows Command Prompt**. Access it using {key Win R}, type `cmd` and hit enter. === StarDist with TensorFlow 1.15=== We are going to create a `env-stardist-tf15` environment in the `D:\` drive using python 3.7 From your command prompt: ``` d: mkdir environments py -3.7 -m venv env-stardist-tf15 env-stardist-tf15\Scripts\activate ``` (WARNING) **Checkpoint**: Ensure the right python version is installed using `where python` and you should have a result like below ``` D:\environments\env-stardist-tf15\Scripts\python.exe C:\Users\oburri\AppData\Local\Programs\Python\Python37\python.exe C:\Users\oburri\AppData\Local\Microsoft\WindowsApps\python.exe ``` Finally, we can install all of StarDist using the `stardist.txt` file below {F20639458} `pip install -r stardist.txt` Install tensorboard for Jupyter Lab `jupyter labextension install jupyterlab_tensorboard` === Noise2Void === Noise2Void currently needs a lower version of CUDA to function. Installations We are going to create a `env-noise2void-tf15` environment in the `D:\` drive using Python 3.7 Start a command prompt and type ``` d: py -3.7 -m venv env-noise2void-tf15 env-noise2void-tf15\Scripts\activate ``` Finally, we can install all of Noise2Void using the `n2v.txt` file below {F20639763} `pip install -r n2v.txt` Install tensorboard for Jupyter Lab `jupyter labextension install jupyterlab_tensorboard` === CellPose === We are going to create a `env-cellpose-torch19` environment in the `D:\` drive using python 3.8 (This is useful if you want to use CellProfiler) Start a command prompt and type ``` d: py -3.8 -m venv env-cellpose-torch19 env-cellpose-torch19\Scripts\activate ``` We can install all of Cellpose using the `cellpose.txt` file below {F20639887} `pip install -r cellpose.txt` === Cellpose with GPU ==== As per the current documentation , we need to uninstall the `torch` module and pick the right one for our CUDA version. Here we installed CUDA 11.1 with CuDNN 8.1.1: ``` pip uninstall torch pip install torch==1.9.0+cu111 torchvision==0.10.0+cu111 torchaudio===0.9.0 -f https://download.pytorch.org/whl/torch_stable.html ``` The last line is obtained from the [[ https://pytorch.org/get-started/locally/ | PyTorch webiste directly]]: {F20639906} === DenoiSeg === d: py -3.7 -m venv env-denoiseg-tf15 env-denoiseg-tf15\Scripts\activate pip install tensorflow-gpu==1.15 keras=2.2.5 jupyterlab jupyter-tensorboard denoiseg jupyter labextension install jupyterlab_tensorboard ``` {F20639887} `pip install -r cellpose.txt` = 'venv' Information & Examples = IMPORTANT: Notice how we always use `Scripts\activate` **before **running any pip commands. The `activate` command ensures that we are running within the virtual environment, like `conda activate`. To confirm you are running in the virtualenv, its name should be in parentheses on the left of the command prompt. {F15073659, width=700, layout=center} IMPORTANT: You should **not **have your Notebooks/Code in the same folder as your `virtualenv`. A `virtualenv` is a folder which contains the libraries and executables to create an //environment// for you to run your code. It is **independent**. Keeping your scripts in the same folder would be like storing your Excel results in `C:\Program Files\Microscoft Office`. == Example: Activating `env-stardist-tf15` and running JupyterLab in your project folder == Suppose we have the following setup: {F15073616, width=400, layout=center} Note how the `My Project` folder is not even on the same disk as our `env-stardist-tf15` . To start JupyterLab in your folder, do the following from the command prompt ``` d:\env-stardist-tf2\Scripts\activate cd /d "E:\My Project" jupyter lab ``` = Creating a shortcut to activate your `virtualenv` = === Example with Noise2Void === You can create a `Run Noise2Void.bat` file that you can keep somethere in your project folder that does this automatically. The syntax is a little different than above. Copy paste the code below into the file you just created. Adjust paths as needed. ``` call d:\env-noise2void-tf15\scripts\activate cd /d "E:\My Project" call jupyter lab ``` === Example with StarDist=== You can create a `Run StarDist-TF2.bat` file that you can keep somethere in your project folder that does this automatically. Copy paste the code below into the file you just created. Adjust paths as needed. ``` call d:\environments\stardist-tf2\scripts\activate cd /d "E:\My Project" call jupyter lab ```