A Demos spokesman said: "Less-secret services could make Britain safer."
The study calls for greater communication with trusted community leaders and individuals.
The report - entitled the Power of Unreason - says groups use conspiracy theories to recruit and radicalise people to commit acts of violence.
An example of one such theory is that the bombings in New York and London, on 11 September 2001 and 7 July 2005 respectively, were "inside jobs" carried out by authorities in the US and UK.
Other theories highlighted were that "freemasons control the world economy through manipulation of paper currency", that the UK government is "consciously seeking to destroy Islam" and that a "conspiracy between the Japanese government, the US, and the Jews existed to gain world domination".
The study claims such theories are frequently adopted by extremist groups to demonise outsiders, discredit moderates and push them in a more extreme and sometimes violent direction.
The report's authors made a number of recommendations concerning the counter-terrorism work carried out by MI5, MI6 and GCHQ and the government.
The publication of all National Security Council annual reports, including outlining the risks to national security and the current terrorist threat, was among their suggestions aimed at improving transparency.
They also called for increased openness in terrorism trials through reporting court proceedings and transcripts and for the provision of more information about policing around counter-terrorism.
However, the study did acknowledge that there were limits to what the government could do to restore trust and urged society as a whole to do more to counter the conspiracy theorists.
One way in which this could be done is by helping young people to think more critically, it said.
It recommended lessons on conspiracy theories and online sources like blogs, Wikipedia and newspapers for secondary school students that focused on digital literacy and 'counter knowledge'.
"More needs to be done in schools to teach young people digital literacy, such as being taught to tell the difference between propaganda and honest and accurate reporting," argues the think tank.
Jamie Bartlett, an extremism expert at Demos, said: "The more open the government is, the harder it is for extremist groups to make stories out of silence.
"Clearly, there are occasions when more transparency is not possible for reasons of national security, the safety of certain individuals, or resource constraints. But the degree to which conspiracy theories make up part of the extremist mind-set and world view suggests it needs to be confronted."
He said such theories "destroy the trust that exists between the government and communities, which is the basis of effective counter-terrorism work".