Yes that is correct. Any time a card goes back to the hand, deck, fusion deck, or graveyard it always goes back to the owner.
The owner of the card is the person whose deck originally contained the card. The Controller is the person that is currently controling it.
Spot's Knight