If you attempt to move through the fortress under the island, before confronting Vyemm up on top, you will meet a "a wayward dracurian guardian" that blocks the way. If you attempt to access the tunnels from down below, you will confront a invisible wall. This simply means you have not confronted Vyemm on top of the island.