They are concerned both by the relatively lax checks that are made on the visitors before they arrive and by the ease with which they can outstay their visas without anyone noticing.
As many as 13,000 visa applicants may have entered the country from Pakistan in a seven month period since October last year without any checks on their supporting documentation.
The security services fear that because most do not mix with home grown terrorists, they are able to operate under the intelligence radar, acting as sleeper cells until ready to launch attacks in Britain.
Every year around 100,000 visitors arrive in Britain from Pakistan alone, which has been described by the Prime Minister as being part of a "crucible of terror" along with Afghanistan.
They are supposed to be checked by Home Office visa staff working in Abu Dhabi in the United Arab Emirates.
But according to an official watchdog, the Independent Monitor for Entry Clearance, many visa officers do not have "enough time to go through applications carefully".
The security services are also worried about arrivals from Somalia, Yemen and North Africa.
MI5 have got 2,000 domestic extremists under surveillance across the country but is becoming increasingly concerned about the threat from abroad.
Similar concerns are felt in the police and one senior counter-terrorism officer told the Daily Telegraph: "There is a lack of control and supervision at our borders in the broadest sense.
"The problem is not confined to Pakistan, terrorists could arrive from anywhere, and we simply have no idea how many extremists may be here."
Police have discovered that the leader of an alleged plot to blow up shopping centres in Manchester last Easter ran a visa advice service in Peshawar, Pakistan.
He is thought to have helped other alleged members of his terrorist cell to arrive from Pakistan under the cover of student visas.
At least one arrived to attend a course at a "bogus college" that had already had its accreditation withdrawn.
The discovery of the group based in Manchester and Liverpool earlier this year set off alarm bells among counter-terrorism officials who believed the threat was coming under control.