The 40-year-old singer-turned-fashion designer enjoyed several nights of passion with 'The X Factor' boss after joining the judging panel in 2007, according to a new unauthorised biography, 'Sweet Revenge: The Intimate Life of Simon Cowell'.
But having just split with Kris Smith, the father of her 21-month-old son Ethan, the revelation is said to have pushed an already upset Dannii to breaking point.
A source said: ''She's understandably distraught by all of this coming out just as she's struggling to cope with the breakdown of her relationship with Kris.
''It was all such a long time ago and to have to deal with it now in the wake of the last few weeks is just too much. She is incredibly raw and this is the last thing she wants to endure.''
In the book, Simon, 52, is quoted as saying: ''I had a crush on her. It was Dannii's hair, the sexy clothes and the t**s. I was like a schoolboy. She was foxy. She was a real man's girl. Very feminine.''
According to the Daily Mirror newspaper, Simon allowed the author Tom Bower hundreds of hours of access to his life, although the music mogul apparently claims to have had no control over what was included in the finished product.
He has so far kept quiet over the situation, only breaking his silence to tweet: ''Just seen the TV ad for the unauthorised book on me. Yikes!''