The 28-year-old star has admitted his parents Mel Gibson and Robyn Moore Gibson were determined to keep him out of the spotlight when he was growing up, but he was in his element when he finally started meeting his dad's friends and was over the moon when the 53-year-old actor let him smack him in the chops.
Speaking to The Times newspaper, he recalled: ''Our parents kept us out of that type of world. Because it changes people, I think.
''He [Robert Downey Jr.] let me hit him in the face a couple of times''.
Milo was completely oblivious to the fact his father was famous until he started primary school and, even then, he was cautious about what he told his friends.
He explained: ''Yeah. I don't really like to tell people, but a lot of my friends already know and they don't care. They're friends with me because they're friends with me.''
However, it's almost impossible for him to hide the fact he's Mel's son nowadays as many people think they are the spitting image of one another.
He said: ''Sometimes people go, 'You look like Mel Gibson!' And I go, 'F**k, I wish!'''
Despite Mel's high level of fame, he still tried to do ''normal'' things with Milo and even tried to persuade him not to go into acting.
He said: ''Oh yeah. He'd take me to soccer games -- and the bedtime stories I got! Just fabricated out of his head, all sorts of crazy stuff ...
''When I was growing up everything was a little different -- paparazzi were more prevalent. They were chasing you around and he was, like, 'Don't do it! Stay away.' He never said I couldn't do it, but it wasn't something that I ever planned on doing.''