When he is around, he will be grouped with 2-3 ire minions or ire warlocks. He is also a necromancer, and so will have a pet.
You don't have to kill the whole encounter for the Restoring Ghoulbane Quest to update - if he's grey to you, you may consider feigning death or running off the other 2-3 skeletons of the encounter so other players get a chance to kill him too (the specter's chance to spawn is quite low).