The judge ordered everyone except the suspects out of the courtroom, as protesters piled in and lawyers complained of intolerable conditions.
The suspects are accused of belonging to a shadowy ultra-nationalist network.
Prosecutors say the group plotted a series of attacks aimed at provoking the military into carrying out a coup.
The trial may revive tensions between the Islamist-rooted ruling AKP party and the secular military, analysts say.
Among the 86 suspects appearing before the Silivri prison-court were retired army officers, politicians, academics and also journalists, who are alleged to be members of the Ergenekon group.
The 2,455-page indictment holds the group responsible for at least two violent attacks - a bombing of a secularist newspaper in 2006 and an attack on a court the same year in which a judge was killed.
The attacks on these key parts of the secular establishment were supposed to provoke the military into launching a coup in defence of secular interests, it is alleged.
The suspects deny the charges, saying they are politically-motivated.
As the trial opened, the presiding judge asked spectators and reporters to leave the tiny courtroom, amid protests by defence lawyers that they could not work in such conditions.
Outside the courtroom, scores of demonstrators with Turkish flags held a protest rally. Many of them chanted: "The traitors are in parliament, the patriots are in prison."
As the trial quickly began to descend into disarray, the presiding judge decided to adjourn proceedings for several hours.
The trial is unusual in a number of ways: the sheer size of it and the fact that the defendants include retired Turkish military officers, the BBC's Pam O'Toole says.
That is something which would have been unthinkable only a few years ago, given the power of the military, which has mounted three coups since 1960 and, in 1997, eased the country's first pro Islamist prime minister from power, our correspondent says.
Then there is the nature and scope of the charges, some of which would not seem out of place in a Hollywood thriller, she adds.
But many Turks regard the trial as the latest stage in an ongoing power struggle between Turkey's secular nationalist establishment and the governing AKP.
The alleged plotters were indicted this summer, shortly before a decision in a court case aimed at closing down the AKP for allegedly becoming a focus for anti-secular activities.
Government critics believe the timing was no coincidence.
Some of them maintain the Ergenekon network simply does not exist; many others believe this trial is the AKP's revenge for the closure case, which in the end did not succeed in the Constitutional Court.
Meanwhile government supporters portray it as a step towards accountability and democracy by taking much needed action against so called "deep state" nationalist groups, who have previously been virtually untouchable.
The military denies any links to such groups.
Turks will watch this case closely, but it could be months, or even years, before there is a result, our correspondent says.