That possibility - which current and former U.S. officials raised on CBS' "Face the Nation" Sunday- has cast a shadow over excitement at Bergdahl's return after five years in Taliban captivity.
Sen. John McCain, R-Ariz., said of the five detainees on CBS News "Face the Nation": "These are the hardest of the hardcore."
The five former detainees include founders of the movement, senior commanders and one who could be prosecuted for war crimes because he presided over the mass killing of Shiite Muslims in Afghanistan in 2000 and 2001.
CBS News National Security Correspondent David Martin called the prisoner exchange a "get-out-of-jail-free card" for the five who were held at the Guantanamo Bay prison for over a decade. They are expected to enjoy a relatively comfortable life in Qatar, the Gulf emirate that helped broker the deal between the U.S. and the Taliban and will take custody of the men for a year. The Taliban leadership said in a press release that the men will live with their families.
Some members of Congress have called into question how secure the men will be within Qatar, which a senior defense official said has given the U.S. "appropriate assurances" it will be able to secure the detainees.
That is very likely, despite the concern, said Thomas Sanderson, the co-director and a senior fellow at the Center for Strategic and International Studies' Transnational Threats Project.
Qatar is a wealthy nation with strong security capabilities and ties to many actors in the Middle East on all sides of the political spectrum. It also hosts a major U.S. air base.
"This is a place that can easily control who comes and goes in their country. I don't think that there's any chance that these guys are going to slip out of the country," Sanderson said.
"Qatar is extremely mindful that in order to be viewed as an extremely reliable go between for the future and to continue to be seen as a reliable partner to the U.S. they have to keep up their end of the bargain here."
For the first year, there will be major U.S. pressure on the Qatari government to ensure the Taliban leaders remain in the country - particularly from President Obama. Once that year is up, however, the pressure will switch, with the Afghan Taliban and its allies in Pakistan working to ensure the men are released from custody as planned, Sanderson said.
With Qatar looking to boost its role on the world stage as a mediating power, the country will have to "walk right down the middle" of its relationship with both the U.S. and the Taliban, which nearly opened a political office in Doha, the Qatari capital, in 2013.
Once the yearlong waiting period is up, experts expect the leaders will return to either Afghanistan or Pakistan, although CBS News National Security Analyst Juan Zarate raised the possibility they could remain in Qatar, which some political and diplomatic officials in the Taliban have chosen to do.
"It's altogether possible that these gentlemen decide it's pretty comfortable, they have enough freedom to operate, enough freedom to communicate with their Taliban comrades out of Qatar, and decide not to leave," Zarate said.
The men could also serve as emissaries of the Taliban in other parts of the world to raise money and build political capital for the group, which is when their release begins to be problematic, Zarate added.
And while integration back into the Taliban organization is likely, it's unclear how quickly or completely that will happen.
"Whether or not they'll assume immediately a commanding post will be very interesting," Zarate said.
"The Taliban has obviously evolved over time. It's a bit more fractured than in the past, and there are other leaders who have taken a prominent role in the Taliban's campaign in Afghanistan as well as some of the activity in Pakistan, so whether or not they'll be able to jump right back into some sort of commanding role that mimics what they were doing in the past I think is a different question."
At a minimum, they will certainly provide political and fundraising value to the movement, he said.
Sanderson also suggested it might take time for the five men to reintegrate back into the fold. They might be viewed with suspicion because of the time they spent in American captivity, or find themselves edged out by other leaders who have risen in the ranks in their absence.
"They also have to deal with their own politics. It's not that they might be viewed as legitimately having been compromised, but these guys could not have influence anymore," Sanderson said. The current leadership could be considering the possibility of peace negotiations with the U.S. and Afghan governments, and unwelcome of more hard-line forces.
A former Guantanamo detainee who was released to the Afghan government in 2007 and set free a year later, Abdul Qayyum Zakir, returned to the Taliban and became a senior commander by 2010. But he resigned his post earlier this year, reportedly over deepening tensions with some of the more moderate Taliban leaders who wanted to pursue peace talks with Kabul, the Wall Street Journal reported.
If the men do become involved in the Taliban leadership or other known terrorist groups, Zarate said the U.S. will monitor them as much as possible. The government "has an assortment of tools to use to include arrests or in the battlefield actually killing these individuals if they are engaged in lethal activities."
And to the extent that the release might facilitate further peace talks with the Afghan government in Kabul, Zarate said the deal will help build trust between the U.S. and the Taliban but not necessarily between the Taliban and the government in Kabul.
The Afghan government "was not involved in this deal," Zarate pointed out. "The Afghan government at the end of the day is going to have to reconcile or come to some political agreement with the Taliban if that's to come to pass, and in some ways this deal undercuts faith and confidence in the Afghan government itself," he said.
"This was the U.S. and the Taliban, not the U.S., the Afghan government and the Taliban engaged in this prisoner swap. That then becomes problematic long term for the Afghan government."