The surge has left 206 people dead and 202 injured, while 220 remain missing, Cesar Uruena, a Red Cross official, told AFP.
The violent weather that hit the southwestern town of Mocoa on Friday night "totally destroyed" 25 homes, he added.
They were the latest victims of floods that have struck the Pacific side of South America over recent months, also killing scores of people in Peru and Ecuador.
In the southwestern Colombian town of Mocoa, the surge swept away houses, bridges, vehicles and trees, leaving piles of wrecked timber and brown mud, army images from the area showed.
The mudslides struck late Friday after days of torrential rain.
President Juan Manuel Santos visited Mocoa, the capital of Putumayo department, on Saturday to supervise rescue efforts in the heavily forested region.
He declared a state of "public calamity" in a Twitter message, declaring measures to speed up rescue and aid operations. He expressed his condolences to victims' families.
"The latest death toll is 154. It is a truly terrible figure," Santos told reporters.
The Red Cross aid group said 400 people were injured and 220 were missing.
The Red Cross had initially put the death toll at 16 but warned it would rise because hundreds of people were missing.
"The number is rising enormously and at considerable speed," Rescue official Cesar Urena told AFP.
The disaster is of "large proportions," he added.
Watch: Colombia landslide kills more than 250
Nation in mourning
Putumayo Governor Sorrel Aroca called the development "an unprecedented tragedy" for the area.
There are "hundreds of families we have not yet found and whole neighborhoods have disappeared," he told W Radio.
Carlos Ivan Marquez, director of the National Disaster Risk Management Unit, told AFP the mudslides were caused by the rise of the Mocoa River and tributaries.
The rivers flooded causing a "big avalanche," the army said in a statement.
Soldiers carry a victim, in Mocoa, Colombia. (AAP) Source: AAP
Some 130 millimetres of rain fell Friday night, Santos said. "That means 30 per cent of monthly rainfall fell last night, which precipitated a sudden rise of several rivers," he said.
He promised earlier on Twitter to "guarantee assistance to the victims of this tragedy, which has Colombians in mourning."
"Our prayers are with the victims and those affected," he added.
Rescue efforts
The authorities activated a crisis group including local officials, military personnel, police and rescuers to search for missing people and begin removing mountains of debris, Marquez said.
A thousand emergency personnel were helping the rescue effort.
Mocoa, a town of 40,000 people, was left without power or running water.
"There are lots of people in the streets, lots of people displaced and many houses have collapsed," retired Mocoa resident Hernando Rodriguez, 69, said by telephone.
"People do not know what to do... there were no preparations" made for such a disaster, he said.
"We are just scarcely realizing what has happened to us."
Several deadly landslides have struck Colombia in recent months.
A landslide in November killed nine people in the southwestern rural town of El Tambo, officials said at the time.
A landslide the month before killed 10 people in the north of the country.
Watch: Historic flooding in Peru
[videocard video="907582531789"]
