Peter Ainsley, 48, of West Road, Lancaster, admitted robbing the pensioner in Gage Street, Lancaster shortly after he left the Nationwide Building Society in Market Street after withdrawing cash on January 24.
The victim, who uses a walking aid and suffers osteoporosis, was taken to the Royal Lancaster Infirmary, where he spent two weeks with a suspected fractured hip and pain caused by the fall.
In a harrowing victim impact statement his daughter told the court the robbery had had a dramatic impact on her father who has lost confidence, is afraid to go into the city on his own, and has had security cameras fitted in his home.
Judge Philip Parry told Ainsley: "You should hang your head in shame."
Paul Cummings, prosecuting, said the pensioner had taken a taxi into Lancaster and had been in a cafe, before going to a Nationwide cashpoint where he withdrew Â£260.
He went into the bank to query his statement and then came back out and put his cash in his pocket.
Moments later he was approached by Ainsley who tried to engage him in conversation and asked him for a cigarette.
Mr Cummings said: "It was about 20 minutes later when he went into St Francis' Passage when he was approached again.
" By then they were near the Phoenix pub and the conversation was asking if he was going in there for a pint.
" Things then turned for the worse."
Ainsley shoved his hand in the pensioner's pocket. The frail man tried to stop him and asked what he was doing.
Ainsley stole cash Â£360 and his bank card then pushed him out of the way causing him to fall onto his back and bang his head.
He fled, leaving the elderly man on the ground.
Judge Parry added: "He is small frail and infirm and he regularly takes a bus or taxi into Lancaster and moves around the city with the aid of a walking frame.
"He has friends and family he used to visit regularly and on January 24 he'd been in to the city to do just that - to have a coffee, to see his granddaughter who works in the centre, and to do some shopping.
"He tried to stop you and then in order to complete the robbery and take the cash from this vulnerable 80 year old man, you pushed him - you pushed him with such sufficient force he fell on his back and hit his head."
"His daughter says your act has changed his life forever. He was proud to have a degree of independence. "
Ainsley, who has 96 convictions for 246 offences - including two thefts from elderly men - was arrested with Â£180 on him.
He made no comment in interview.
Darren Halsted, defending, said the defendant's intention was to 'pickpocket'.