And, while more and more are doing it, fewer and fewer are getting caught, according to the RAC.
Now, as the dangerous practice has become a deadly epidemic, the motoring organisation says it is time for tougher punishments, because many offenders believe they can get away with it.
“It is alarming to see some drivers have clearly relaxed their attitudes to the risks associated with this behaviour,” said RAC road safety spokesman Pete Williams.
“But more worrying is the increase in the percentage of motorists who admit to using a handheld device when driving. The fact that drivers have little or no confidence that they will be caught is a likely contributor to the problem.”
Drivers caught using mobiles can face a fine of as little as £30, with three points on their licence.
Department of Transport figures show that a driver distracted by their phone was a contributory factor in 492 accidents in 2014, 21 of them fatal crashes and a further 84 classed as serious.
The most recent figures in Lancashire revealed there were 66 crashes caused by drivers either making a call or texting at the wheel.
One of the worst ended in the death of Preston cyclist Paul Fingleton, 47, at a roundabout in Broughton. Former racing driver Frank Wrathall, 27, who was on the phone to his girlfriend when his car collided with Mr Fingleton, was jailed for 21 months for causing death by careless driving.
The RAC study showed 31 per cent of drivers admitted to using their mobile while at the wheel. The figure has quadrupled over the past two years.
One in five motorists think it is acceptable to check social media on their phone when in stationary traffic, compared with 14 per cent in 2014. A staggering 14 per cent confessed they had taken videos or photographs with their phones while driving. In January this year the numbers fined for illegal use of mobile phones at the wheel had dropped from 123,000 in 2011 to just 30,000.
And the number of people who feel it is acceptable to take a handheld call has doubled from seven per cent to 14 per cent in two years.
“Too many still think that the chance of being caught is so slim that they can get away with it,” said AA president Edmund King. “The AA believes penalty charges should be raised and the funds ring-fenced for advertising and greater enforcement.
“By the end we would hope handheld mobile phone use had become as socially unacceptable as drink-driving.”
The key reason many drivers felt they would not be caught was the reduction in the number of traffic police due to government cutbacks.
“With fewer officers out on the roads, more of these offences are going undetected,” said Jayne Willetts from the Police Federation.
More than 1,000 traffic police posts were lost between 2010 and 2014.
“It’s no good having laws if there aren’t the people to make them stick,” said Shadow Roads Minister Daniel Zeichner.