Try to run
modprobe -r nvidia_uvm; modprobe -r nvidia_modeset or if it fails change the order of the two modules. If you then still cannot modprobe -r nvidia add the output of lsmod | grep nvidia and lsof | grep nvidia to your question.
