Cavendish was ninth in the Madison, alongside Chorley star Wiggins, at the 2008 Games in Beijing and 29th in the road race on the opening day of London 2012.
The 31-year-old from the Isle of Man has won 30 Tour de France stages and is desperate for an Olympic medal, preferably a gold one.
The six-discipline omnium in which Cavendish has been selected, and placed sixth in at March’s Track World Championships in London, is something of a lottery.
And the four-rider, four-kilometres team pursuit offers a surer prospect of a podium place.
But while 2012 Tour winner Wiggins, Ed Clancy, Steven Burke and Owain Doull have had the event as their sole focus for more than a year, Cavendish has not and will likely only ride one of the three rounds if a team-mate is injured or falls ill.
“That’s the reason I left the Tour early, because of the team pursuit,” Cavendish said.
“For the omnium, finishing the Tour would’ve been a benefit. It’s a little bit disappointing, but it’s how it works.
“Especially Brad, he has been super stressed. He wants to be the hero and all that.
“I’m kind of just doing the omnium stuff now. That’s what I was aiming for the whole time. The team pursuit’s a bonus to that anyway.”
Cavendish’s comments could reopen old wounds with Wiggins.
The pair drifted apart after the Beijing Games, when Cavendish suffered the ignominy of being the only member of the track team not to return with a medal.
Wiggins had been fatigued by his exertions in winning the team pursuit and individual pursuit title before the Madison.
Then it was suggested the peloton conspired against Britain on the opening day of London 2012 in the road race.
“Both times at Olympics – it’s easy to say, ‘He’s making excuses’ – I don’t feel like it was my fault,” Cavendish added.
“Extraordinary factors meant I didn’t get those medals when I had done everything right leading up to them.”
Cavendish and Wiggins’ friendship was rekindled and Wiggins put in a phenomenal turn en route to Cavendish winning the road world title in Copenhagen in September 2011.
He then also led Cavendish to victory in Paris while wearing the Tour winner’s yellow jersey in July 2012.
The duo won the non-Olympic Madison at March’s Track World Championships in London and afterwards they spoke glowingly of each other.
Britain won the last two Olympic titles in the team pursuit, but will have to be better than ever to win gold in Rio.
Cavendish has not ridden an international team pursuit for some time and his best of four minutes 08.345 seconds is from a World Cup event in Moscow in December 2005.