I attempted virtualizing our old RIP and had some success. Our setup required a total of three serial dongles daisy chained together. Although it recognized the first in the chain, I wasn't able to configure it to recognize the other two. The first dongle's pinout was standard DB25 with a null modem built into the plastic casing. The other two were nonstandard with only 3 pins each. I suspect VMWare had an issue with that, especially considering the first dongle in the chain was technically a terminal. If your pinout is standard then there's a good chance it will work.
If you haven't already I would suggest plugging the dongle into another PC, reboot both, and RDP into your RIP with local port sharing enabled. If this works then you would almost certainly be able to virtualize your RIP and run it in either Hyper-V or VMware without any extra configuration.
If your RIP OS is older than Server 2003 you can install a virtual serial port driver on both the RIP and another system on your network with the dongle. Configure them to talk to each other over your network and your RIP *shouldn't* know the difference. Again, this is assuming there isn't a weird pinout preventing proper pass thru in the software layer. If there is a null modem or loopback option in the virtual driver you might need to toggle it.
If your RIP still can't authenticate the dongle there are other things to try. Using a port sniffer or dumping the serial data to a text file on your existing RIP could give you an idea as to what the software is expecting to hear back from the dongle. If it's communicating asynchronously in 7 bits you could probably just replace the dongle with a simple script. Though I doubt you'll be that lucky.