A spokesman for Grimshaw, who spent just one season as a judge on the popular show, told The Sun the Radio One presenter made the decision to leave and had not been sacked.
The spokesman said: "After careful consideration, Nick's decided to rule himself out as a judge on this year's X Factor.
"He loved working on last year's series alongside Rita, Cheryl and Simon but it was very much a once-in-a-lifetime experience. He remains a massive fan of the show and is looking forward to watching it this year."
The ITV talent programme is said to be undergoing a shake-up of its judging and presenting talent under the supervision of boss Simon Cowell after it suffered a ratings slump in 2015.
One Direction star Louis Tomlinson is said to be in the running to replace Grimshaw, although Cowell could bring back former judge Louis Walsh, according to the newspaper.