The 61-year-old actor reveals his dulcet tones helped land him the Emmy-winning voiceover role while he was filming U.S. sitcom Cheers back in the 1990s.
"Sam Simon - who created The Simpsons - was one of the writers on Cheers," he explained on Britain's The Graham Norton Show. "As I usually do, I walk onto the set sometimes and I would usually sing The Good Life, kind of in my best Tony Bennett derivative.
"(Sam) called me one afternoon and said, 'Hey, do you still sing? Could you sing a Cole Porter song - Ev'ry Time We Say Goodbye? Of course I could! I love that song.
"He said, 'We've got this character, who's never actually said anything on the show. He's Krusty the Clown's sideshow, and we call him Sideshow Bob. We want him to speak, finally. We thought you should do it!' I said I could certainly do that, and then I read the script and it was actually really funny and wonderful," Kelsey recalled.
The actor, who is most famous for his role as Dr. Frasier Crane on Cheers and the hugely successful spin-off show Frasier, also shared the inspiration behind his take on the murderous clown on the long-running animated series.
"I had logged away a voice years ago when I worked for a man named Ellis Rabb - who had started a theatre company in New York City and had great success," he remembered.
"When I was working for him, I was painting his bathroom and hanging wallpaper - stuff like that! I wasn't really a successful actor at the time. Ellis would regale me with stories when he'd come home from work and he'd say things like, 'Oooooh Kelsey, that baby should have been mine!' I thought, 'This guy, I'm gonna use him someday!'"
Meanwhile, the actor also made a rather personal confession on the talk show, as he revealed he had a tattoo on an intimate part of his body.
Declining to show host Graham Norton the ink, which is of his wife's name, he said instead, "It didn't hurt as much as my parrot biting me. That was much worse."
Kelsey married fourth wife Kayte Walsh, a 35-year-old British flight attendant, in 2011, and the couple share two children, three-year-old Faith and two-year-old Kelsey Gabriel.