"I was simply scared Koloskov told the daily Sport-Express after being forced to resign as president of the Russian Football Union (RFU) after more than 25 years in the job.
The 64-year-old was forced out on Thursday by the Kremlin bosses who were unhappy at the way he had been running the country's most popular sport.
He pointed the finger at Russia's Sports Minister Vyacheslav Fetisov -- his biggest critic -- for putting extra pressure on him to quit.
"It is one thing when people criticise you but then the criticism calms down and you can work normally Koloskov said.
"But when the pressure is constant which has been happening to me lately with the help of Fetisov it becomes unbearable.
"Even Russian President (Vladimir Putin) made things clear when he said: 'Koloskov is a good man but a bad football leader'...
Koloskov has had many enemies since taking over as Russia's soccer tsar in 1979.
"You begin to fear for your life after all because it's simple -- you have some mad people out there who think Koloskov must be physically eliminated he said.
"I thought about it and decided it's better to leave now."
In 1998 he was forced to hire bodyguards after receiving serious threats from his bitter rival Nikolai Tolstykh during their mud-slinging campaign for the RFU presidency.
"I am being directly threatened by one of the candidates Koloskov said at the time. "This person told a close friend and colleague of mine: 'Tell Koloskov that if he doesn't leave the RFS willingly then things will end badly for him'."
The two men have since reconciled their differences.