Things didn't look good for North End very early on when Gibbs-White gave the hosts the lead with less than seven minutes on the clock.
North End weathered the storm, though, and equalised on 20 minutes through Daniel Johnson. With the game heading for a draw Sheffield looked like they had clinched the points when Berge fired past Daniel Iversen to put United 2-1 up.
However it is never over till it's over in the Championship and Riis came to North End`s rescue with almost ninety-five minutes showing as he fired a Jordan Storey cross past Olsen in the home goal.
Things can change quickly in the transfer market and Preston North End have come to life after Huddersfield win
PNE player ratings vs Luton Town as Ben Whiteman and Liam Lindsay steal the show in hard-fought win
Match Report: Luton Town 0-1 Preston North End - Brad Potts stunner gives North End first Championship win
Ryan Lowe says a better Preston North End team will face Luton Town this time around
Preston North End boss felt his side found a way after wonder goal sees them past Luton Town
The were scenes of unbridled joy on the away end as the 571 travelling fans danced with delight and North End had secured the point their performance fully deserved.
Frankie McAvoy named the same 18 that were on duty at Bristol on Saturday with the starting line up unchanged.
The North End manager could be forgiven for being slightly apprehensive about coming to Bramall Lane as the Blades had put six past Peterborough on Saturday and looked in prime form.
It didn't start well as Patrick Bauer under hit a ball meant for touch and as it was recycled, the cross from Freeman found Gibbs-White who made no mistake with his half volley.
However North End shook off the early setback and started to knock the ball around quite comfortably, so it was no surprise when the scores were level after 20 minutes.
A beautiful pass from Ryan Ledson found Johnson behind the United defence and the captain made no mistake almost walking the ball into the net. Sean Maguire then had a chance that flew across goal and another just before half time that came to nothing.
No changes for either side going into the second half as the hosts started at full speed. North End once again calmed things down and in fact had the better chances in the early part of the second half with the strike duo of Riis and Maguire both having chances before the hour mark.
I just sensed that North End were starting to tire a little with 20 minutes to go and a few minutes later Iversen made a magnificent one on one save from Fleck as the home side went for the winner.
They looked like they had done the job when substitute Berge fired past Iversen with six minutes to go. The home fans in the 25,463 crowd were roaring their team on but Frankie's boys just kept on plodding in the hope the last chance would come.
Sure enough with 94 and a half minutes on the clock a free kick from Ledson was eventually played out to Storey on the right who chipped a gorgeous ball to the back post for Riis to take aim and score.
The silencea round Bramall Lane was deafening save for the loyal band on the away end who celebrated like it was the winner.
A thoroughly deserved point for North End in South Yorkshire then against a team who look mighty fast and dangerous on the counter attack.
After the first goal North End managed the home side very well and it was probably weariness that allowed United to take the lead late on.
Nevertheless this is not the Preston team of a month ago, this is a team now playing with some belief and with a formation they are comfortable with and who can execute the plan well.
I thought Johnson was prominent again and Ledson was conducting the orchestra in midfield for the first hour the way that Paul Gallagher used to do. We still have a long, long way to go and who knows where we would be in the league table with a another good striker on the books.
We aren't there yet and there is plenty to do but this group are starting to get their act together and that in itself bodes well for the Autumn and Winter months to come.
Support us and become a subscriber today. Enjoy unlimited access to local news, the latest football stories and new puzzles every day. With a digital subscription, you can see fewer ads, enjoy faster load times, and get access to exclusive newsletters and content. Click here to subscribe.