I don't know anything about using OBS studio, but I guess for NVENC it needs the nvidia proprietary drivers. I suggest to install these first and get them running.