My personal favorite destination on Phuket Island for spending a few good hours having good food and a decent walkabout is Old Phuket Town. It's not unlike other Straits Settlements, like Penang and Malacca, and is not always on mainstream tourist travel itineraries for Phuket.
What you get is a quaint, vintage Chinatown, which you could cover on foot in 3-5 hours. There are more than enough great local restaurants and coffee joints to provide respite from the heat and humidity. The main draws are Raya Restaurant (for great Thai cuisine) and Kopitiam (a traditional coffee shop).