A man carrying a #ZumaMustFall poster was reportedly pushed around at the Ses'khona People's Rights Movement gathering in the CBD.
Witnesses say the man, believed to be a PAC member, was part of a group of members of the party taking part in the march.
"Leadership stopped the scuffle before it got out of hand," one protester said.
"We are not here to be violent."
Police have managed to keep the group in Keisergracht Street despite the group planning to make their way through the streets to the DA's Mill Street office.
They only had permission for a gathering, not for a march, the City of Cape Town said on Thursday morning.
Mothers with babies on their backs, MK soldiers in uniform and elderly people wearing their slippers were among those who marched up and down a portion of Keisergracht singing struggle songs.
Ses'khona spokesperson Sithembele Majova said the group was trying to convince police to allow them to make their way to Mill Street.
"The [DA-led city] is infringing on our right to march," he said.
Racism is at the heart of the issues they want addressed, Majova continued.
"Khayelitsha is not like Constantia, where services are delivered every day.
"Youth development in the Western Cape is in the hands of the privileged. Unemployed people suffer in this province because the DA want to run it like an island."
A request to receive a memorandum on the group's anti-racism march was sent to the public protector's office, the DA, the speaker of Parliament and the SA Human Rights Commission.
One protester said he won't leave until the DA's leaders come to face them.
"Yes, they must send [Mmusi] Maimane. We want him here."
