Professor John Underhill, chief scientist at Heriot-Watt University in Edinburgh, suggested the UK is "55 million years too late" for the gas extraction technique to work.
He said reservoirs of shale gas had been damaged by seismic activity 55 million years ago, causing some of the deposits to escape.
Fracking, or hydraulic fracturing, involves drilling into the earth then injecting liquid into the rock at high pressure, forcing apart fractures and allowing gas to escape.
Opponents of the technique claim it will harm the environment, but supporters say it is a beneficial source of energy.
Mr Underhill said: "Both sides of the hydraulic fracturing debate assume that the geology is a 'slam dunk' and it will work if exploration drilling goes ahead.
"Public support for fracking is at an all-time low of 17%, based in the main on environmental concerns, but the science shows that our country's geology is simply unsuitable for shale oil and gas production. The implication that because fracking works in the US, it must also work here is wrong.
"For hydraulic fracturing to be successful, a number of geological criteria must be met. The source rock should have a high organic content, a good thickness, be sufficiently porous and have the right mineralogy. The organic matter must have been buried to a sufficient depth and heated to the degree that the source rock produces substantial amounts of gas or oil.
"However, in locations where fulfilment of some of the criteria have led to large potential deposits, uplift and the faulted structure of the basins are detrimental to its ultimate recovery.
"Yet, the only question that has been addressed to date is how large the shale resource is in the UK. The inherent complexity of the sedimentary basins has not been fully appreciated or articulated and, as a result, the opportunity has been overhyped."
He highlighted three potential fracking sites which he said have been subject to deformation - the Weald basin in southern England, the Bowland Shale in Lancashire and the West Lothian Oil Shale in Scotland.
Mr Underhill added: "Areas that were once buried sufficiently deeply with temperatures at which oil and gas maturation occurs, lifted to levels where they are no longer actively generating petroleum. They have also been highly deformed by folds and faults that cause the shales to be offset and broken up into compartments. This has created pathways that have allowed some of the oil and gas to escape.
"There is a need to factor this considerable and fundamental geological uncertainty into the economic equation. It would be extremely unwise to rely on shale gas to ride to the rescue of the UK's gas needs only to discover that we're 55 million years too late."
Friends of the Earth Scotland (FoES) said the report "adds weight to the already overwhelming case against fracking in Scotland, or indeed anywhere in these Isles".
Head of campaigns, Mary Church, added: "FoES has long highlighted the misleading use of US shale gas economics by industry players like Ineos to make a case for fracking in Scotland.
"However, even if Scotland's geology presented ideal conditions for shale gas extraction, the case against fracking is still very clear.
"Fracking presents a whole host of risks to people's health and our local environment, while in the context of irreversible climate change, going after a new source of fossil fuels is quite literally the last thing we should be doing.
"Support for fracking is at an all time low. Over 60,000 people have responded to a consultation on fracking in Scotland with the vast majority calling for an all out ban.
"We urge the Scottish Government to put an end to this discussion by banning fracking once and for all, and focus instead on a rapid and fair transition to a zero carbon economy."