https://github.com/DualCoder/vgpu_unlock Skip to content Sign up Sign up * Why GitHub? Features - + Mobile - + Actions - + Codespaces - + Packages - + Security - + Code review - + Project management - + Integrations - + GitHub Sponsors - + Customer stories- * Team * Enterprise * Explore + Explore GitHub - Learn and contribute + Topics - + Collections - + Trending - + Learning Lab - + Open source guides - Connect with others + The ReadME Project - + Events - + Community forum - + GitHub Education - + GitHub Stars program - * Marketplace * Pricing Plans - + Compare plans - + Contact Sales - + Education - [ ] [search-key] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up Sign up {{ message }} DualCoder / vgpu_unlock * Notifications * Star 327 * Fork 18 Unlock vGPU functionality for consumer grade GPUs. MIT License 327 stars 18 forks Star Notifications * Code * Issues 3 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights master Switch branches/tags [ ] Branches Tags Nothing to show {{ refName }} default View all branches Nothing to show {{ refName }} default View all tags 1 branch 0 tags Go to file Code Clone HTTPS GitHub CLI [https://github.com/D] Use Git or checkout with SVN using the web URL. [gh repo clone DualCo] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Go back Launching Xcode If nothing happens, download Xcode and try again. Go back Launching Visual Studio If nothing happens, download the GitHub extension for Visual Studio and try again. Go back Latest commit @DualCoder DualCoder Fix support for TU102 and TU106. ... 0675b56 Feb 27, 2021 Fix support for TU102 and TU106. 0675b56 Git stats * 5 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .gitignore Initial commit Jan 28, 2021 LICENSE Initial commit Jan 28, 2021 README.md Add list of vGPUs and add PCI ID for Tesla T4. Feb 24, 2021 kern.ld Add modifications to kernel module. Feb 22, 2021 vgpu_unlock Fix support for TU102 and TU106. Feb 27, 2021 vgpu_unlock_hooks.c Fix support for TU102 and TU106. Feb 27, 2021 View code vgpu_unlock Important! Description Dependencies: Installation: How it works vGPU supported? Userspace script: vgpu_unlock Kernel module hooks: vgpu_unlock_hooks.c Kernel module linker script: kern.ld How it all comes together README.md vgpu_unlock Unlock vGPU functionality for consumer grade GPUs. Important! This tool is very untested, use at your own risk. Description This tool enables the use of Geforce and Quadro GPUs with the NVIDIA vGPU software. NVIDIA vGPU normally only supports a few Tesla GPUs but since some Geforce and Quadro GPUs share the same physical chip as the Tesla this is only a software limitation for those GPUs. This tool aims to remove this limitation. Dependencies: * This tool requires Python3, the latest version is recommended. * The python package "frida" is required. pip3 install frida. * The tool requires the NVIDIA GRID vGPU driver. * "dkms" is highly recommended as it simplifies the process of rebuilding the driver alot. Installation: In the following instructions need to be replaced with the path to this repository on the target system and need to be replaced with the version of the NVIDIA GRID vGPU driver. Install the NVIDIA GRID vGPU driver, make sure to install it as a dkms module. ./nvidia-installer Modify the line begining with ExecStart= in /lib/systemd/system/ nvidia-vgpud.service and /lib/systemd/system/nvidia-vgpu-mgr.service to use vgpu_unlock as executable and pass the original executable as the first argument. Ex: ExecStart=/vgpu_unlock /usr/bin/nvidia-vgpud Reload the systemd daemons: systemctl daemon-reload Modify the file /usr/src/nvidia-/nvidia/os-interface.c and add the following line after the lines begining with #include at the start of the file. #include "/vgpu_unlock_hooks.c" Modify the file /usr/src/nvidia-/nvidia/nvidia.Kbuild and add the following line. ldflags-y += -T /kern.ld Remove the nvidia kernel module using dkms: dkms remove -m nvidia -v --all Rebuild and reinstall the nvidia kernel module using dkms: dkms install -m nvidia -v Reboot. --------------------------------------------------------------------- NOTE This script will only work if there exists a vGPU compatible Tesla GPU that uses the same physical chip as the actual GPU being used. --------------------------------------------------------------------- How it works vGPU supported? In order to determine if a certain GPU supports the vGPU functionality the driver looks at the PCI device ID. This identifier together with the PCI vendor ID is unique for each type of PCI device. In order to enable vGPU support we need to tell the driver that the PCI device ID of the installed GPU is one of the device IDs used by a vGPU capable GPU. Userspace script: vgpu_unlock The userspace services nvidia-vgpud and nvidia-vgpu-mgr uses the ioctl syscall to communicate with the kernel module. Specifically they read the PCI device ID and determines if the installed GPU is vGPU capable. The python script vgpu_unlock intercepts all ioctl syscalls between the executable specified as the first argument and the kernel. The script then modifies the kernel responses to indicate a PCI device ID with vGPU support and a vGPU capable GPU. Kernel module hooks: vgpu_unlock_hooks.c In order to exchange data with the GPU the kernel module maps the physical address space of the PCI bus into its own virtual address space. This is done using the ioremap* kernel functions. The kernel module then reads and writes data into that mapped address space. This is done using the memcpy kernel function. By including the vgpu_unlock_hooks.c file into the os-interface.c file we can use C preprocessor macros to replace and intercept calls to the iormeap and memcpy functions. Doing this allows us to maintain a view of what is mapped where and what data that is being accessed. Kernel module linker script: kern.ld This is a modified version of the default linker script provided by gcc. The script is modified to place the .rodata section of nv-kernel.o into .data section instead of .rodata, making it writable. The script also provide the symbols vgpu_unlock_nv_kern_rodata_beg and vgpu_unlock_nv_kern_rodata_end to let us know where that section begins and ends. How it all comes together After boot the nvidia-vgpud service queries the kernel for all installed GPUs and checks for vGPU capability. This call is intercepted by the vgpu_unlock python script and the GPU is made vGPU capable. If a vGPU capable GPU is found then nvidia-vgpu creates an MDEV device and the /sys/class/mdev_bus directory is created by the system. vGPU devices can now be created by echoing UUIDs into the create files in the mdev bus representation. This will create additional structures representing the new vGPU device on the MDEV bus. These devices can then be assigned to VMs, and when the VM starts it will open the MDEV device. This causes nvidia-vgpu-mgr to start communicating with the kernel using ioctl. Again these calls are intercepted by the vgpu_unlock python script and when nvidia-vgpu-mgr asks if the GPU is vGPU capable the answer is changed to yes. After that check it attempts to initialize the vGPU device instance. Initialization of the vGPU device is handled by the kernel module and it performs its own check for vGPU capability, this one is a bit more complicated. The kernel module maps the physical PCI address range 0xf0000000-0xf1000000 into its virtual address space, it then performs some magical operations which we don't really know what they do. What we do know is that after these operations it accesses a 128 bit value at physical address 0xf0029624, which we call the magic value. The kernel module also accessses a 128 bit value at physical address 0xf0029634, which we call the key value. The kernel module then has a couple of lookup tables for the magic value, one for vGPU capable GPUs and one for the others. So the kernel module looks for the magic value in both of these lookup tables, and if it is found that table entry also contains a set of AES-128 encrypted data blocks and a HMAC-SHA256 signature. The signature is then validated by using the key value mentioned earlier to calculate the HMAC-SHA256 signature over the encrypted data blocks. If the signature is correct, then the blocks are decrypted using AES-128 and the same key. Inside of the decrypted data is once again the PCI device ID. So in order for the kernel module to accept the GPU as vGPU capable the magic value will have to be in the table of vGPU capable magic values, the key has to generate a valid HMAC-SHA256 signature and the AES-128 decrypted data blocks has to contain a vGPU capable PCI device ID. If any of these checks fail, then the error code 0x56 "Call not supported" is returned. In order to make these checks pass the hooks in vgpu_unlock_hooks.c will look for a ioremap call that maps the physical address range that contain the magic and key values, recalculate the addresses of those values into the virtual address space of the kernel module, monitor memcpy operations reading at those addresses, and if such an operation occurs, keep a copy of the value until both are known, locate the lookup tables in the .rodata section of nv-kernel.o, find the signature and data bocks, validate the signature, decrypt the blocks, edit the PCI device ID in the decrypted data, reencrypt the blocks, regenerate the signature and insert the magic, blocks and signature into the table of vGPU capable magic values. And that's what they do. About Unlock vGPU functionality for consumer grade GPUs. Resources Readme License MIT License Releases No releases published Packages 0 No packages published Languages * C 82.9% * Python 17.1% * (c) 2021 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.