Military offensives against the militants in the northwest had forced 1.9 million people to flee their homes over last few months - the vast majority seeking refuge with relatives and in schools, and the rest packed into sweltering camps.
The government laid on buses and trucks to return IDPs on Monday, the first day of large-scale organised returns, but only a fraction of the families earmarked for voluntary return actually left the camps.
Azam Khan, a senior official in the government's emergency response unit said 192 families out of an estimated 2,680 left three camps on Monday.
"We expect an increase in coming days," Khan told reporters at Charsadda, where 22 out of a planned 247 families left for the northwest Swat district.
Dozens of displaced people blocked a road outside one Charsadda camp vowing not to return until they received ATM bank cards on which they can draw 25,000 rupees (300 dollars) of financial aid to rebuild their lives.
Crops were left to rot during the two-month offensive and the local economy has been shattered.
"Some people did not receive their ATM cash cards and they refused to go until they got this card," said Khan.
At Jazolai, where about 4,000 families have been sheltering in a camp near the town of Nowshera, 50-year-old Shireenzada said he was unsure whether peace had returned to his home town of Barikot in Swat.
The government says it has worked hard to restore electricity and running water in main towns since the fighting but analysts warn that much needs to be done to sustain the returnees.
"They will start living a normal life if the environment is secure and their fundamental needs are addressed. Secure environment means army, police and civil administration," said independent analyst Imtiaz Gul.
AFP reporters said they saw just over 200 people leaving Jalozai and the nearby camp of Charsadda but officials swept aside concerns.
"Twenty-four buses reached Swat. These are full of IDPs," Bashir Ahmad Bilour, a senior provincial cabinet minister told reporters.
"We hope the situation will improve in the coming days and that people will come back with the passage of time," he said.