There are documented cases of people with brain injuries thinking that their family members are impostors. They recognize them, but because of their brain injury, something looks wrong about the family members, so they assume they are messing with them.

A fascinating book about brain injuries is "The Man Who Mistook His Wife for a Hat", by Oliver Sacks.