You can delete a tag even if there are contents referencing it.
If you delete the tag, the content is not updated.

In attached screenshot I deleted the EN CE MOMENT tag from the tag explorer window, and my content still reference it. (this is annoying since we use tags as facets on search services, we now have a ghost value in the drop down list)