Is there a golden rule for what happens to destroyed cards you don't own?

For example, I know Broadsword would send a destroyed card to the destroyed pile, and Killer Frost's ability would send a card in the lineup to the destroyed pile. However, if I were to say destroy a villain in the lineup with the Joker's ability, after time traveling it, the destroyed card would go back into the lineup at the end of my turn.

If a card or ability explicitely references a zone where you can destroy a card, does it always go to the destroyed pile? Is that the answer?

It's just something that has crossed me up from time to time, and was curious about any golden rules on the matter.

Thanks!