I tried a number of things and settled on just dumping pulseaudio.
Code:
killall pulseaudio
Note Don't run this command when Rhythmbox or other sound applications are running, they could crash.

I didn't want to uninstall pulseaudio though because of the reboot/configuration issues so I just run this each time I start my computer. You can automatically do this by adding the command to session startup programs

System > Preferences > Sessions > Startup Programs (tab) > Add

It's a bit of a hack, but it's working.