The 28-year-old sock designer was notably absent from his stepsister's 18th birthday celebrations, with a source revealing he is ''ignoring'' the family.
They said: ''Rob refused to do anything for Kylie's birthday. He is invited to everything and he refuses to come to anything.
''He's just ignoring everyone. His mom, his sisters. He just doesn't care.''
And the insider also noted that his sister Khloe Kardashian, with whom he lives, is finding it hard to get him out the house.
They added to RadarOnline.com: ''[Khloe can't] get Rob to leave the house. He just hides out there all the time and doesn't care if he leaves or even gets dressed.
''Rob is really making his sisters and even his mom mad. Kim still thinks he's really being selfish and she want him to do things with them, not shut them out completely ... They don't know what to do anymore.''
Meanwhile, the 31-year-old beauty previously revealed she is helping her brother through ''a hard time''.
She said: ''Rob lives with me now, and I wouldn't have it any other way. I love having him with me. I have a big ass house and I wanted it so he could feel like, comfortable.
''He's going through a hard time now. And [my home is] behind gates. With this crazy lifestyle you have to think of places where you can still have your freedom. Kourtney and Scott live in the same gated area. My nieces and nephew come over at least three times a week. I have everything, my brother's there and I love him.''