This cat hates me

It's not his fault really. His story is sad. He had an owner that had some issues. He was thrown against the wall repeatedly and starved. Eventually, the cat's original owner's caretaker took him away and gave him to my boyfriend. The cat doesn't like very many women in general.

He's getting better with me. He'll play with me and sometimes allow me to pet him but he still won't let me hold him like my boyfriend does.