Al Jazeera broadcast footage of an unmasked man and his armed compatriot telling the crowd that al-Qaeda's fight was with America rather than the Yemeni military.
"We carry bombs for God's enemies," the man said at the rally, which took place on Monday.
"Soldiers, you should know that we do not want to fight you. There is no problem between you and us. Our problem is with America and its allies. Beware taking the side of America."
Yemen's government has been battling al-Qaeda in the country at the same time as dealing with a Shia uprising in the north and rising secessionist sentiment in the south.
The al-Qaeda rally took place in the southern Abyan province in an area that was bombed in an air raid last week.The government said at the time that the raid had foiled a planned series of suicide bombings by attacking targets that included an al-Qaeda training centre.
But dozens of civilians, including children, are thought to have been killed in the bombing.
Abbas al-Assal, a local human rights activist, said at the time that 64 people were killed, including 23 children and 17 women.
Abdul-Ghani al-Iryani, a political analyst based in Yemen, said that the air raids meant civilian casualties were unavoidable, but that the government could take steps to lessen public anger.
"Unfortunately collateral damage cannot be avoided in operations like this [as] al-Qaeda live with their families in their bases and training camps, so there's no way of avoiding it," he told Al Jazeera.
Al-Iryani said the government should announce the names of the al-Qaeda operatives killed in the raid.
"There's nothing that the government can do in regards to dealing with al-Qaeda per se, but addressing people's grievances in other issues could increase the credibility of the state," he said.
The New York Times newspaper said on Saturday that the US gave military hardware, intelligence and other support to Yemeni forces to carry out the attack.
Supporters of Yemen's so-called Southern Movement, which calls for independence and accuses the government in Sanaa of sidelining the south, participated in the rally which marked the bombing, calling for an investigation into the attack.
An explosion disrupted the rally and killed three people. The cause of the blast was unclear, though it has been blamed on a cache of unexploded munitions.
Fighters in Yemen and Saudi Arabia earlier this year announced a loose alliance under the name of "al-Qaeda in the Arab Peninsula", taking Yemen as their base.