Three in four adults (74%) believe there are not enough places to pee in their area, according to a survey by the Royal Society for Public Health (RSPH).
Meanwhile, one in five (20%) said a lack of facilities deters them from leaving home as often as they would like.
Among people with medical conditions which can require frequent visits to the toilet, such as diabetes and bowel conditions, more than two in five (43%) said this was the case.
The RSPH is calling for Government action to address the "neglect" shown towards facilities in the UK.
Chief executive Shirley Cramer said: "Our report highlights that the dwindling public toilet numbers in recent years is a threat to health, mobility, and equality that we cannot afford to ignore.
"As is so often the case in this country, it is a health burden that falls disproportionately on already disadvantaged groups.
"Standing in the way of this necessary and serious policy discussion is a stubbornly persistent 'toilet taboo', a decade of cuts to local authorities, and an increasingly ingrained notion that public toilets are merely a 'nice-to-have'."
The RSPH report recommends that the provision of public toilets should be made compulsory in planning law, with equal access for women and transgender people.
It also calls for the Government to reverse funding cuts to councils and encourage schemes to boost funding for public toilets, for example by taking one penny from the price of every train and bus tickets to finance them.
"Public toilets are no luxury: it's high time we begin to see them as basic and essential parts of the community - just like pavements and street lights - that enable people to benefit from and engage with their surroundings," Ms Cramer said.
"It is deeply concerning that, amidst a national obesity crisis, at a time when public health policy is to encourage outdoor exercise, our declining public toilet provision is in fact encouraging more people to stay indoors."
A Local Government Association spokesman said: "Councils are doing everything they can to keep public toilets open and this includes running community toilet schemes, which enable local businesses like pubs, restaurants and shops to make more clean, safe and accessible toilets available to the public.
"Faced with an £8 billion funding gap by 2025 and growing demand pressures on adult social care, children's services and homelessness support, councils have had to make tough choices about how to manage dwindling resources.
"It is vital that the Spending Review fully funds councils to provide local services for our communities."
Jon Richards, Unison's head of local government, said: "Billions of pounds slashed from council budgets mean people struggle to spend a penny even when they're desperate.
"It's also a real inconvenience for those who work outside such as care assistants, district nurses, refuse collectors and others who don't have offices."
A Ministry of Housing, Communities and Local Government spokeswoman said: "Public toilets are a valuable community amenity so we encourage councils to keep them open.
"Councils, not central government, are responsible for managing their own resources and providing the local services people need, including public toilets."
More than 2,000 UK adults took part in the survey.