The town of Hveragerdi, 40 minutes west of Reykjavik is known for it's intense geothermal activity. I read online (thanks wikitravel) that about an hour's hike north of town there was a hot spring river you could bathe in.
The directions were a little misleading as you actually had to drive straight through town past the golf course to the Reykjadalur recreation area, a dirt road marked with a sign. There you drive till you reach the River Varma and the Hot River Café where you'll find a parking lot and several cars. While decently crowded, this is not a major tourist attraction.
The climb is about an hour and it's pretty challenging and almost entirely uphill. You are rewarded with amazing views, waterfalls, and geothermal pools and springs all around.
Towards the end of the hike, just when you're thinking that you took the wrong trail and ready to quit, the trail takes a turn and you're rewarded with a mountain top valley with a spring running through it that is indeed, hot. You'll know it's the right spot because there should be other bather's strewn on the banks or in the dammed pools.
It feels like being in a warm bath of running water and was greatly relaxing (especially because I was hungover after a weekend of Reykjavik's insane nightlife). Unfortunately we were unable to stay as long as I'd long due to hunger and having packed no food.
If you go, plan ahead like a group of Icelanders we saw and bring plenty of water, lunch, and some cans of Gull.