Matrix multiplication example performed with OpenMP, OpenACC, BLAS, cuBLAS, and CUDA
This example requires the following packages:
- CUDA Toolkit 10.1
- PGI CE Compiler 19.10
Optional:
- Eclipse IDE C/C++
- Docker CE + NVIDIA-Docker v2
- PGI Docker image
- Jupyter Notebook
The following installation instructions have been tested on Ubuntu 18.04 and CUDA 10.0+.
OpenACC profiling with NVIDIA driver 418.67 and above requires elevated permissions. See here.
You can achieve this one of two ways.
- Run command with sudo
sudo LD_LIBRARY_PATH=/usr/local/cuda/extra/CUPTI/lib64:$LD_LIBRARY_PATH ./computeWorks_mm- Following Administration instructions.
sudo systemctl isolate multi-user # Stop the window manager sudo su # Switch to root modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia-vgpu-vfio nvidia # Unload dependent modules cd /etc/modprobe.d/ touch nvidia.conf # Create file named nvidia.conf echo -e "options nvidia "NVreg_RestrictProfilingToAdminUsers=0"" > nvidia.conf reboot
CUDA -> more details
- Download CUDA Toolkit
- Install (assuming file is in ~/Downloads)
sudo dpkg -i ~/Downloads/cuda-repo-ubuntu1804-10-1-local-10.1.168-418.67_1.0-1_amd64.deb sudo apt-key add /var/cuda-repo-<version>/7fa2af80.pub sudo apt-get update sudo apt-get install cuda
- Add paths to ~/.bashrc
echo -e "\n# CUDA paths" >> ~/.bashrc echo -e "export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}" >> ~/.bashrc echo -e "export LD_LIBRARY_PATH=/usr/local/cuda/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" >> ~/.bashrc echo -e "export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}" >> ~/.bashrc
- The CUPTI directory is required for OpenACC profiling
PGI Community Edition (Bare Metal) -> more details
Skip this step if you prefer to run utilize the PGI compiler in a Docker container.
- Download PGI CE Compiler
- Install (assuming file is in ~/Downloads)
export PGI_SILENT=true export PGI_ACCEPT_EULA=accept export PGI_INSTALL_DIR=/opt/pgi export PGI_INSTALL_TYPE=single export PGI_INSTALL_NVIDIA=true export PGI_INSTALL_JAVA=true export PGI_INSTALL_MPI=false export PGI_MPI_GPU_SUPPORT=false mkdir -p ~/Downloads/tmp tar xpfz ~/Downloads/pgilinux-2019-1910-x86-64.tar.gz -C ~/Downloads/tmp sudo -E ~/Downloads/tmp/install rm -rf ~/Downloads/tmp
- Add paths to
~/.bashrc
echo -e "\n# PGI paths" >> ~/.bashrc echo -e "export PGI=/opt/pgi" >> ~/.bashrc echo -e "export PATH=/opt/pgi/linux86-64/19.10/bin:$PATH" >> ~/.bashrc echo -e "export MANPATH=$MANPATH:/opt/pgi/linux86-64/19.10/man" >> ~/.bashrc echo -e "export LM_LICENSE_FILE=$LM_LICENSE_FILE:/opt/pgi/license.dat" >> ~/.bashrc
- Download Eclipse IDE C/C++
- Install (assuming file is in ~/Downloads)
sudo tar xpfz ~/Downloads/eclipse-cpp-2019-03-R-linux-gtk-x86_64.tar.gz -C /opt
sudo ln -s /opt/eclipse/eclipse /usr/local/bin/eclipse- Install Nsight Eclipse Plugin -> more details
bash /usr/local/cuda/bin/nsight_ee_plugins_manage.sh install /opt/eclipse
Docker -> more details
- Remove older Docker versions
sudo apt remove docker docker-engine docker.io containerd runc -y
- Install Docker CE -> more details
sudo apt update sudo apt install apt-transport-https ca-certificates curl software-properties-common -y curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" sudo apt update sudo apt install docker-ce -y
- Enable Docker commands without sudo
sudo usermod -aG docker $USER- Log out and back in
- Confirm $USER is in the docker group
groups
mnicely adm cdrom sudo dip plugdev lpadmin sambashare docker
- Verify docker runs without sudo
docker container run hello-world
NVIDIA Docker v2 -> more details
- Remove nvidia-docker v1, if installed.
docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f sudo apt-get purge nvidia-docker
- Add nvidia-docker v2 repository
curl -sL https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - distribution=$(. /etc/os-release; echo $ID$VERSION_ID) curl -sL https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list sudo apt-get update
- Install nvidia-docker v2
sudo apt-get install nvidia-docker2 -y
- Reload Docker daemon
sudo pkill -SIGHUP dockerd
- (Optional) Modify Docker daemon to storage images in /home versus /var. Usually /home has more space.
sudo nano /etc/docker/daemon.json
{
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
},
"experimental": true,
"graph": "/home/<whoami>/.docker",
"storage-driver": "overlay2"
}sudo service docker restart
- Verify you can launch docker container with access to GPU
docker run --runtime=nvidia --rm nvcr.io/nvidia/cuda:latest nvidia-smi
This create a Docker image containing the PGI CE Compiler.
- Download git project
git clone https://github.com/mnicely/computeWorks_examples.git
cd computeWorks_examples/computeWorks_mm/pgi_build- Download PGI CE Compiler
- Move PGI tar to Dockerfile directory (assuming file is in ~/Downloads)
cp ~/Downloads/pgilinux-2019-194-x86-64.tar.gz .
- Build PGI Docker image
docker build -t cuda-10.1_ubuntu-18.04_pgi-19.10 -f Dockerfile.cuda-10.1_ubuntu-18.04_pgi-19.10 .- Install PIP package manager
sudo apt install python-pip3
- Install JupyterLab
sudo -H pip3 install jupyter
- This approach requires PGI Community Edition (Bare Metal)
- Download git project
git clone https://github.com/mnicely/computeWorks_examples.git
cd computeWorks_examples/computeWorks_mm- Build computeWorks_mm binary
make
- Run computeWorks_mm <matrixSize | default=1024>
./computeWorks_mm 128
- This approach requires PGI Community Edition (Docker Image)
- Download git project
git clone https://github.com/mnicely/computeWorks_examples.git
cd computeWorks_examples/computeWorks_mm- Build computeWorks_mm binary
docker run --runtime=nvidia --rm -v $(pwd):/workspace -w /workspace cuda-10.1_ubuntu-18.04_pgi-19.10:latest make- Run computeWorks_mm <matrixSize | default=1024>
docker run --runtime=nvidia --rm -v $(pwd):/workspace -w /workspace cuda-10.1_ubuntu-18.04_pgi-19.10:latest ./computeWorks_mm 128Eclipse, with Nsight Eclipse Plugins offers full-featured IDE that provides an all-in-one integrated environment to edit, build, debug and profile CUDA-C applications.
- Open Eclipse
eclipse &- Import Project
- File -> Import...
- Select -> Git -> Projects from Git -> [Next >]
- Select Repository Store -> Clone URI -> [Next >]
- Source Git Repository -> URL -> https://github.com/mnicely/computeWorks_examples -> [Next >]
- Branch Selection -> [Next >]
- Local Destination -> [Next >]
- Select a wizard to use for importing projects -> Import existing Eclipse project -> [Next >]
- Import Projects -> [Finish]
- This approach requires PGI Community Edition (Bare Metal)
- Build Project
- Right click computeWorks_mm -> Build Project or Press Ctrl + B
- Run Project
- Right click computeWorks_mm -> Run As -> Local C/C++ Application
- This approach requires PGI Community Edition (Docker Image)
- Point to PGI Docker container
- Right click computeWorks_mm -> Properties
- C/C++ Build -> Settings
- Settings -> Container Settings -> select Build inside Docker Image
- Image -> cuda-10.1_ubuntu-18.04_pgi-19.10:latest
- Run Project
- Right click computeWorks_mm -> Run As -> Local C/C++ Application
- This approach requires Administration instructions.
- Open computeWorks_mm.ipynb
cd computeWorks_examples/computeWorks_mm/jupyter
jupyter-notebook computeWorks_mm.ipynb