On the morning of November 9, 2016, journalists and pundits across the country scrambled to find an explanation for the election upset they had just observed. Donald Trump, a moderately successful real estate investor with no political experience, had beaten Hillary Clinton, a former US Senator, Secretary of State, and First Lady. Pollsters had predicted an easy win for Clinton, with several giving Trump a less than one in ten chance of victory. Election post-mortems offered a broad range of answers, from blaming a recrudescence of racism voters, economic anxiety in Appalachia, to any of a dozen or so strategic failures of the Clinton campaign.

But that frantic hunt to explain the outcome missed a question with far greater significance for the future of American politics: how did so unfathomable a candidate as Donald Trump secure the nomination of the Republican Party in the first place? The real surprise for scholars of the election happened not in November 2016 but months earlier, in July 2016, when Donald Trump won the Republican nomination.

The next three articles in this series will explore that question in-depth, but the shortest possible version is this: by 2016, the Republican Party had ceased to function as a modern party. The tools of party control of the nomination process—invented more than a century and a half ago—had been eroded through new technologies. This loss of control reintroduced collective action problems among a ballooning number of candidates, which then allowed Donald Trump to have a serious shot at the nomination. The general election victory of Donald Trump masked the fact that the Republican Party has been hollowed out, making party control of the nomination process the weakest it has been since the 1850s.

But before Democratic readers take heart from the heat death of the GOP, consider that the same, inexorable processes are even now operating on the Democratic Party, increasing the likelihood that previously unthinkable candidates—like Bernie Sanders, a Democratic Socialist from deep blue Vermont—will win the nomination, hurting the party’s chances of victory in the general election. Inasmuch as retail politics matter on the margins, then one lesson we learned in 2016 is that, when facing the least popular candidate in modern presidential history, it is a mistake to nominate the second least popular candidate in modern presidential history. Or even the third, as we might discover in 2020.

The Collective Action Problem

That phrase, “collective action problem,” may be unfamiliar to some readers. In economics, the term describes a way that individual incentives can undermine collective goals.  Even when people agree about the best possible outcome for their group, their own incentives can lead them to take actions that undermine the longed for outcome.

If you happen to have watched the 2001 movie A Beautiful Mind starring Russell Crowe, you have seen an illustration of a collective action problem.In one scene, a group of mathematicians at a bar spot an alluring blonde and her less attractive friends. The optimal strategy for ensuring that everyone went home with someone that night would be for the mathematicians to spread their attention equally among the women, with only one of the mathematicians pursuing the blonde. However, the mathematicians each individually prefer the blonde, leading to a collective action problem; if all four pursue the blonde, they will knock each other out of contention for her affections, and, in the end, alienate both her and her ignored friends. Each man goes home alone, their individual incentives having undermined a collective goal.

To put this in terms of presidential elections, a political party is a big tent full of people who generally agree that any one of a group of candidates who ascribe to the party’s platform would make a good candidate for the President. They can, however, nominate only one of those candidates; and it is in the party’s best interest to select that single candidate as quickly, efficiently, and with as little infighting as possible.

However, that collective consensus is complicated by contrary incentives for each individual candidate for office. Each prospective candidate (along with the second, fifth, thirteenth, and twenty-fourth prospective candidates) will reason that while, yes, only one person can win, there is always a chance it might be them; and even if it isn’t, running could lead to career-boosting national exposure or even a lesser administration post. The individual interest is to run regardless of the effects on the collective goal.

And that can work as long as the field of candidates doesn’t grow overly large. If there are just a few candidates, it can be positive, drawing attention to the race, energizing the base, and still have a relatively swift resolution. But if too many individual candidates run, it hinders the agreed-upon collective goal of quickly nominating a candidate so as to maximize their chances of winning in the general election. The nomination contest drags on, the debate schedule grows, the debates themselves become more bitter, money that could have been spent on the general gets spent on the primaries, and the eventual nominee emerges from the process late and limping. This hurts a party’s chances of winning the general election.

Historically, parties have developed mechanisms to prevent these kinds of collective action problems from arising in the first place, but those tools of party control have been critically undermined over the past two decades. Collective action problems have already emerged in the Republican Party, culminating in the nomination of Donald Trump in 2016, and threaten to do the same in the Democratic Party in 2020.

The Little Magician Pulls the Modern Political Party Out of His Hat

But our story begins more than a century and a half before the least popular presidential candidate in polled history won a high-backed chair and an eagle-festooned rug in the Oval Office. It begins with another presidential hopeful, who, in this case, happened to be the second shortest. Martin Van Buren, our eighth President, may have been short of stature (5’ 6”),but he was long on wits; his political instincts earned him the nickname of “the Little Magician.” Van Buren entered national politics in the 1820s just as the first party system was falling apart. It was a time known as the “Era of Good Feelings,” a reference to the fact that the old Federalist Party had imploded and the Democratic-Republican Party dominated national politics.

Yet the Democratic-Republican party was bedevilled by collective action problems. Party leadership had few tools for disciplining party members and, as a result, four candidates from the same party entered the general election in 1824. This divided the party vote, leading to no single candidate receiving a majority of the electoral votes. Per the US Constitution, this threw the election into the House of Representatives; all four candidates were Democratic-Republicans, ensuring the party a win, but a lack of a clear public mandate would harm whoever came out on top. After significant horse-trading, the House awarded the Presidency to the politically well-connected John Quincy Adams, who finished second in the electoral college, instead of to political outsider Andrew Jackson, who had won a plurality in both the popular vote and the electoral college.

Adams won the nod by giving the fourth-place finisher, Henry Clay, a cabinet position in exchange for his backing in the House. Jackson called this a “corrupt bargain,” but it was only possible because Jackson was one of three southerners who had run, while Adams had the advantage of being the sole representative of a more unified bloc of Northeastern states. A collective action problem had allowed Adams to slip by the more popular Jackson.

Van Buren, a US Senator by this point, disliked President Adams’ policies and decided that Jackson had the best chance to defeat Adams in an 1828 rematch. However, those who had backed the third place finisher in 1824, William Crawford of Georgia, liked neither Adams nor Jackson and could have once again split the southern vote, giving the incumbent Adams an advantage in 1828. So Van Buren leaned on his personal relationship with Crawford and his supporters, as well as his control of the large New York delegation—which was then a crucial swing state—to convince Crawford’s supporters to back Jackson. Now that the collective action problem was resolved, Jackson beat the less nationally-popular Adams handily in 1828.

The 1824 and 1828 elections had exposed internal tensions within the Democratic-Republican party, which split into two new parties, the Democrats (of which Jackson and Van Buren were members) and what would become the Whig Party. Heading into the 1832 election, Andrew Jackson had a lock on his party’s nomination. However, the vice president slot remained open. Van Buren, who aspired to the office given that it was a natural stepping stone to the presidency, pushed for the party to adopt a new mechanism for preventing collective action problems among candidates: the nominating convention. Van Buren had expressed interest shifting to the convention model as early as 1827, but it was not until 1832 that the first presidential nominating conventions were held.

Here is how a nominating convention prevents collective action problems. It allows a political party to quickly select a single nominee months before the general election, thus preventing multiple candidates from dividing the party vote, cutting down on intra-party squabbling, and helping the party present a united face against the opposing party’s candidate. And that’s precisely how it worked at the first Democratic nominating convention in 1832; Van Buren called in the many political favors owed him, easily secured the vice-presidential nomination, and served alongside Jackson during his second term.

Four years later, in 1836, it was Van Buren’s turn to run for President. Like in 1832, the nominating convention prevented any collective action problems from cropping up for the Democratic side. But the Whig Party was not so fortunate; the nascent party was too disorganized for effective party discipline and had not yet adopted the nominating convention format. As a result, four Whigs entered the race, divided their parties’ support, and allowed the “little magician” to pull off his greatest trick yet, winning the Presidency in 1836.

Tonight We’re Gonna Party Like It’s 1944

Van Buren had unlocked the key to modern party organization. The modern party’s primary purpose is to prevent collective action problems and keep internal bickering from undermining their electoral hopes. A tight party structure with robust national, regional, and local branches that whip voters to get behind a single candidate gives the party the best chance of winning in the general election. And a nominating convention helps the party select the most broadly palatable candidate before the general election and do so in a manner that energizes party supporters while maintaining party control of the process. That strategy ushered in more than a century of relative party stability; despite periodic ideological realignments, the routine party turnover of Van Buren’s time was over.

Party leadership developed additional tools for maintaining control of the nomination process other than the nominating convention itself. Conventions continued to be the site of significant horse-trading between candidates. While there was always a whiff of the Adams-Clay “corrupt bargain” about it, at least these deals and disputes occurred within the convention and not on the floor of the House of Representatives. And this bargaining built party unity around the eventual nominee, who might offer a cabinet post or the vice presidency to challengers, thus keeping them and their supporters in line.

This system worked because, unlike today, party delegates were not bound to any particular candidate. There were no binding primaries in which rank-and-file voters expressed their candidate preference and thus bound their state’s delegates to vote for said candidate. As a result, the nominating convention gave substantial power to the party hierarchy. Those in leadership with the largest personal networks had the most say over who the nominee would be. This evolved into an era of deep party control of the nomination process, the very real phenomenon of a group of party loyalists packed in smoke-filled rooms deciding who the next presidential nominee would be.

This process also emphasized electability, as party leaders wanted the most palatable candidate for the general electorate, who was not always the candidate who most closely adhered to any given faction of the party base. The most famous example of this logic was the nomination of Harry Truman as the Democratic vice presidential nominee in 1944. At the time, President Franklin Roosevelt was in declining health, so party leadership expected that whoever was the next vice president would finish Roosevelt’s term. The problem was that Roosevelt’s previous vice president, Henry Wallace, was a Soviet sympathizer who supported desegregation, alienating crucial blocs within the Democratic coalition. So party leadership worked behind the scenes to push Roosevelt to stay non-committal on the VP nod and then denied Wallace a first ballot victory at the nominating convention. Wallace did receive a plurality of the first ballot, but, after much back-and-forth over several alternative candidates, the leadership settled on a compromise candidate, the relatively moderate Harry Truman. Truman would, of course, finish out Roosevelt’s term after the latter’s death in 1945 and then go on to win a famous upset victory in his own right in 1948.

Viewed from one perspective, strict party control of the nomination had led to an anti-democratic outcome, the rejection of Henry Wallace for his radical beliefs; however, given the Cold War context, Henry Wallace would almost certainly have lost badly in 1948. The pragmatic choice was thus between a failed ideologue and a successful, albeit somewhat milquetoast, President. It is an example of how parties have a tough mandate, to simultaneously maintain a balance between representing the will of their members—which naturally pushes parties towards ideological purity—and ensuring party success in the general election—which incentivizes moderation in the pursuit of independent and swing voters. Until the 1960s, the modern party system tended to privilege general electability over ideological purity. If the goal of modern party systems is to win elections, then that was a sound strategy.

There Ain’t No Party Like a Modern Party, but the Modern Party Can Stop

However, those tools of party control increasingly alienated voters in the mid-20th century. Many in both major parties were frustrated by the consensus politics of the post-World War Two era, in which the parties hewed towards the political center, pushing both the Right and the Left to the margins. Progressive Democrats liked Truman no more than conservative Republicans liked Dwight Eisenhower, who was, after all, recruited by both parties to run as their nominee. Thus, when the New Right and New Left burst through the political levees in the 1960s, there was immense pressure to reform the party system and nominate candidates that reflected the will of party members rather than the machinations of party elites.

The most significant of these reforms was the binding primary. The first primaries dated back to the nineteenth century, but they were non-binding, turning them into little more than “beauty contests,” as political observers called the New Hampshire primary in 1952. (Although Truman’s surprise defeat in that “beauty contest” may have convinced him not to run for re-election.) That changed as a result of the 1968 election. President Lyndon Johnson, despite being mired in the unpopular Vietnam War, was considering running for reelection. But the rising anti-war faction of the Democratic Party preferred Eugene McCarthy, a Senator from Minnesota who had taken a strong stance against the war. McCarthy’s campaign shocked the party establishment after he came uncomfortably close to beating Johnson in the New Hampshire primary, in large part because of energetic support from his young, college-age volunteers who reporters gave the derisive nickname of the “childrens’ crusade.” (Half a century later another politically radical senator with a large contingent of enthusiastic, young supporters would outperform expectations in New England.)

His uncomfortably narrow win in New Hampshire convinced Johnson not to run for re-election. That led to a particularly chaotic primary field, especially after the assassination of presumptive frontrunner Robert Kennedy. McCarthy went on to win a majority of primary votes, leaving Johnson’s final heir apparent, Vice President Hubert Humphrey, to limp into the 1968 Democratic National Convention in Chicago. Despite the lack of party enthusiasm for Humphrey, party leadership calculated that McCarthy’s strident anti-war position would hurt their chances in the general election. So the usual wheels began turning, the gears of party control started grinding, and the convention nominated Humphrey. That led to an eruption of protests outside the convention hall in an embarassing, nationally-televised spectacle that likely contributed to Humphrey’s loss in the general election to Richard Nixon. (It also led to the nomination of what was arguably the finest Presidential candidate to ever grace the political stage: Pigasus, the 145 lb pig nominated by Yippie protestors before being eaten by the Chicago Police Department.)

Democratic Party leadership realized that in order to keep the next generation of party members from leaving for more radical alternatives, they needed to reform the nomination process. The resultant McGovern-Fraser Commission proposed changes before the 1972 election, the most sweeping of which was to make primaries binding on most of each state’s delegates. It was the end of an era in which candidates could win the primary process but lose at the convention. And, in so doing, it took away one of the key tools of control over the nomination process. Party leadership would have a much harder time defying the will of rank-and-file membership in order to select more moderate candidates.

However, these reforms, although calming the turmoil within the Democratic Party, also led to a series of nominees that struggled to compete on the national stage because of their progressive policies. What played well with the further Left-leaning contingents of the Democratic base in the primaries did not play as well with the electorate in the general election. That list of ideologically-consistent but doomed nominees included the titular head of the McGovern-Fraser Commission, George McGovern, a progressive senator from South Dakota who lost badly in the general election to Richard Nixon in 1972.

Eventually, the Democratic Party recalibrated its nomination process to give back some control to party elites, creating the “superdelegate” system in 1982 that reserved a substantial minority of unbound delegate positions filled by the party leadership. The Republican Party never adopted the more top-down, nationalized delegate system of the Democrats, preferring to leave primary design decisions to its state parties, but, even though implemented on a state-by-state basis, binding primaries became the norm for Republicans as well by the 1980s. Voters on both sides of the aisle demanded a more democratic process.

At first blush, it might seem like the rise of the binding primary should have fatally undermined the Van Buren-esque modern party system. After all, party leaders could no longer use the nominating convention to control the nomination process. However, by the 1970s, both parties had other tools at hand that could replace the disciplinary function of the nominating convention.

The first tool was the debate stage. While there were sporadic primary debates between candidates on radio and television in the 1940s and 1950s, the practice did not become commonplace until 1968, on the cusp of the McGovern-Fraser reforms. Televised debates became vitally important in the new era of binding primaries given that candidates now had to fight for rank-and-file votes rather than just appealing to party elites. But the party leadership controlled access to the primary debate stage and could use the debate qualification rules to push out candidates who they perceived to be too marginal or radical. They could also limit the number of debates or push them to poor time slots in order to protect an incumbent or other favored candidate from scrutiny.

The second tool was the donor list. Donations are the lifeblood of political campaigns, but, in the pre-digital era, the easiest path to political contributions ran through party-controlled donor lists. Both Republicans and Democrats not only kept track of high dollar donors but also cultivated relationships with them. Those relationships made it possible for party leaders to lean on donors to give or not give to particular candidates. Besides, wealthy donors preferred general election winners—and the possible rent-seeking benefits of political patronage that would follow—so their interests aligned with party leadership more often than not when it came to selecting centrist candidates with the best shot at victory.

So just as they lost one major tool of control over the nomination process, the parties found two sufficiently powerful substitutes. Marginal or radical candidates would now find it hard to make it onto the primary debate stage. If they cleared that hurdle and perhaps even went on to win a few delegates in the early primaries, that was fine. But if, at some point in the primary contest, the party leadership decided it was time to unify behind a single, generally-palatable candidate, they could then turn off the donor tap. It might not look exactly like the modern party system that Martin Van Buren conjured, but it functioned quite similarly.

However, by the end of the twentieth century, these two vital tools of party control were themselves under threat from new technologies, in particular the rise of conservative mass media and online crowdfunded political campaign. The writing was on the wall for the modern party system, but nobody knew how to read the signs until it was much too late. 

Part Two of this series coming soon. Trump is a symptom, not a cause.