They did it with 10 men too, Andrew Hughes having been shown a red card when conceding the penalty which led to the Blades' second goal late in the first half.
The contest had looked to be out of PNE's grasp as the visitors went in two goals to the good at the interval.
Although Ryan Lowe's men had chances in that first half, it was United who stamped their mark on proceedings.
Jayden Bogle fired them ahead in the 17th minute, a lead increased six minutes before the interval through Billy Sharp's penalty.
Hughes had fouled Rhian Brewster in the box as he moved towards goal, the Welsh defender shown a red card.
North End refused to back down and accept their fate. They were much better in the second half and had genuine belief they could get back into the game.
After knocking on the door a few times and surviving a few scares at the other end, Alan Browne halved the deficit with 19 minutes remaining.,
Then as the clock ticked into the 89th minute, Emil Riis got on the end of Ched Evans' cross to score at the far post, sparking wild celebrations.
Lowe had made two changes to the side which started last Saturday's 1-1 draw with Birmingham, Scott Sinclair and Tom Barkhuizen coming in for Evans and Greg Cunningham.
Sinclair partnered Riis up front, with Barkhuizen playing at left wing-back.
The game was wide open from the start with a couple of early sights of goal falling PNE's way.
In the fifth minute, Barkhuizen got to the byline to the left of goal and pulled a low ball back for Hughes, the defender sliding a shot straight against the keeper.
Then Daniel Johnson saw a shot blocked in the box after good approach work from Riis.
Play swung to the other end, Oliver Norwwod seeing a shot blocked well by Ben Whiteman after a corner had been rolled out to him on the edge of the box.
Both sides had great chances to score within moments on one another, both opportunities coming on the counter attack,
Riis led a PNE attack down the centre of the pitch, the Dane striding through the United half.
As he approached the box he laid the ball off to his right to play in Sinclair who saw a shot parried by keeper Wes Foderingham.
The visitors broke up field straight away, Iliman Ndiaye curling a shot across goal from the left side of the box, one which Daniel Iversen dived to push past the far post at full stretch.
United started to get on top, keeping possession well and forcing North End on the back foot.
Their opening goal came in the 17th minute after they won a corner on the right.
It was delivered long to the far post where John Egan's header was parried by Iversen. The ball ran loose to the edge of the box to Bogle who drove in a low shot which touched off a Preston boot on the way past Iversen.
North End's job got a great deal harder in the 38th minute as they went down to 10 men and fell further behind.
Brewster got behind the PNE defence to move into the box pursued by Hughes.
Hughes appeared to stumble against Brewster as he lost his footing, the contact taking the United striker to the floor.
Referee David Web pointed to the spot and then produced the red card, deeming Hughes as the last man and not made a genuine attempt for the ball.
Once Hughes had left for the dressing room, Sharp stepped up to fire past Iversen from the spot.
To cover the loss of Hughes, Lowe brought on Cunningham with Sinclair the man sacrificed.
A second change was made at half-time, Joe Rafferty coming on for Brad Potts
Moments into the second half Iversen saved well from Ndiaye, before North End went close in the 48th minute.
Barkhuizen got down the right channel and lifted a cross into the middle which Browne met with a header, Foderingham diving to tip if behind,.
Brewster slammed a shot across goal and just wide after being played clear in the box as the play swung from end to end.
Riis got room in the box and saw his low shot come back off Foderingham before North End reduced the arrears in the 71st minute.
Johnson was the creator, getting down the left side of the box ffrom Evans' pass and pulling the ball back for Browne to open up his body and fire home a low right-foot shot.
Filled with belief, North End pressed for an equaliser and that arrived in the 89th minute.
Browne's pass played in Evans down the right hand side of the box and the Welshman played an inch-perfect pass across the box which Riis slid in to meet at the far post and score.
PNE: Iversen, van den Berg, Bauer, Hughes, Whiteman, Potts (Rafferty 46), Browne, Johnson, Barkhuizen (Evans 65), Riis, Sinclair (Cunningham 40). Subs (not used): Ripley, Lindsay, McCann, Maguire.
Sheffield United: Foderingham, Basham, Egan. Robinson, Bogle, Norwood, Hourihane, Stevens (Norrington-Davies 82), Ndiaye (Berge 82), Brewster (McGoldrick 70) Sharp. Subs (not used): Eastwood, Burke, Slater, Gordon.
Referee: David Webb (County Durham)
Attendance: 12954 (2,510 away)
Thanks for reading. If you value what we do and are able to support us, a digital subscription is just £1 month for the first two months. Try us today by clicking here