To sort of echo roycpa, "top 5" or "best 5" is kind of tough to answer. I have not eaten at many of the places that people would lump in the "top" category, like Vic and Anthony's, or some of the more upscale places.
For fine dining, my favorite downtown is still Hugo's Cellar. I like the vibe of the place and their blackened salmon and steaks are really good. Plus the guy with the salad cart is such a nice touch.
I ate at the Chart House once. It was very good, but I think you could do better at McCormick and Schmick, even though that's not downtown.
I've also eaten many times at the Flame and Redwood Grill at the Cal. Both are a notch below Hugo's, but both are a good bit less expensive.
For casual dining, I still like 777 at Main Street Station, although I do think that has gone downhill a little bit. I also really enjoy Magnolia's at the 4Q, and the Café Cortez.