One of the group, dressed as Dr Seuss character the Grinch, was photographed pulling the hat down from the 65ft structure outside Gateshead, with the help of three others dressed as Santa.
The hat had been placed on top of the famous Anthony Gormley artwork on Christmas Eve, prompting delight from onlookers.
Even Gateshead Council had welcomed the stunt, describing the public response as "staggering" after the pictures made national news.
After taking it down, the group, which has asked to remain anonymous, told the Press Association: "We never in a million years expected it to hit the headlines the way it did.
"Because it's such good publicity, we thought we'd give the public something else to laugh and smile about before the end of the festive season and came up with the Grinch and a few Santas to take it down."
This year's successful stunt was the group's seventh attempt at placing the hat on the sculpture, with previous efforts blighted by problems including bad weather and a broken-down car.
Their first attempt saw them scale the structure only to discover their hat was too small to fit.
The stunt required meticulous planning, with ten people in five vans required to get the hat up using ropes and fishing line.
One member said he spent £90 on fabric to create the hat, sewing it all together with the help of his girlfriend and his grandmother's sewing machine.
The group said it had planned to leave the hat in place until the weekend was over to allow more people to see it, but decided to remove it early as increased winds could have risked making it a hazard.
This Christmas was not the first time the 200-tonne structure has been dressed up since being installed 20 years ago.
A group of Newcastle United fans placed a Newcastle shirt with Alan Shearer's name on it in 1998 while supermarket Morrisons was forced to apologise after beaming a bread advert on to its wings in 2015.