Posh led 1-0 after two minutes courtesy of Jack Taylor, but the Magpies hit back through goals from Connor Hall and Lewis Baines.
Posh boss Darren Ferguson said: “Shocks like this are what the FA Cup is about.
“These games are televised because of the chance of a shock and they got exactly what they came here for.
“I’ve got no complaints as were second-best all over the pitch and we should have lost by more in all truth.
“Sometimes in a cup match you can lose to a deflection or have complete domination but the other team score from their only two chances.
“But that wasn’t the case today and I have to dig a bit deeper to get to the bottom of a performance that was completely unacceptable for this football club.
“I need to look at it, analyse it and come up with some solutions. I’ve got my own ideas.
“But I really hope Chorley go on to get a great draw with one of the big clubs away from home because they fully deserved to win.”