I didn't do much in the town of El Calafate other than trek the Perito Moreno. I stayed in a hostel slightly in the outskirts of what you can call downtown and took a tour bus arranged by the trekking company to see the glacier. My first glacier visit, it was definitely impressive.
But what was really cool was strapping on the crompons and walking or trekking on the glacier. At the end of my trek I drank some scotch and ate a couple of alfajores which are offered on the glacier itself - yes your tour company will set up a table with the drinks and goodies - and yes that's where I became an alfajores addict!
To get to El Calafate, I flew from Buenos Aires and spent just two nights with the main objective being to trek the glacier. What a surreal experience. At one point on the glacier, I could see nothing but the different valleys and peaks. It felt other worldly.