Cape Coast Castle should be on every tourist's list.  Its origins date back to 1653, when it was built by the Swedish.  Later taken and improved by the British, in 1844 it became the main center for British administration in the Gold Coast.  Its dungeons were used to hold slaves destined for the Americas.  It is now a UNESCO World Heritage site.