Best you could do is just download your subscribers as a .csv, remove who you don't want to contact anymore, and reupload it as your main emailing list.
This could be annoying if the issue of wanted someone removed comes up a lot, but for just one person every once in a while, then...it's still annoying lol.