Back when I flew a lot solo for business (or semi-solo — there were two or three of us traveling, but we all booked separately so we were scattered) I did not mind swapping with someone so they could sit together. On one occasion I swapped a middle seat and got a window (my preferred).

I have never had someone just sit in my seat without asking (or if that was their intent, they were gracious about moving to their assigned seat), but someone who just grabbed my seat and then got huffy about it would not make me likely to swap.

I certainly would not give up my seat if I were traveling with Deb (which is the only way I fly now). She's my emotional support human, and I suspect I would feel anxious without her next to me on landing. I only get anxious during landings — I don't remember the flight when we crashed during landing, but some part of me does, it seems. Having Deb holding my hand and rubbing it helps.