Police said armed men stormed a mosque during prayers on Friday, following an explosion at a checkpoint near the city's military headquarters.
Pakistan's Geo TV reported a witness as saying armed men threw grenades into the mosque.
Rao Iqbal, the Rawalpindi police chief, said that at least 26 people have been killed and more than 30 wounded.
Al Jazeera's Imran Khan, reporting from the scene, said that the military had secured the area and that at least one helicopter was being used to search for the attackers.
"There were apparently eight attackers," he reported. "Three of them died and the others are supposed to be in the vicinity."
Major General Athar Abbas, the Pakistani military spokesman, confirmed the attacks but said he had no details on casualties.
"There have been killings. Bodies are shifted to the hospital but we don't know exactly how many," he was quoted by the AFP news agency as saying.
"It was an attack at a mosque in an area officers frequent."
The attack appeared to be the latest in a series to hit Pakistan in recent months as its military presses ahead with an offensive against al-Qaeda and the Taliban in the country's northwest.