There is a habit among some libertarians of pining for a lost age of liberty. There was, they argue, some period in history in which we were so much more free than today and that period is held up as the model to which libertarians should aspire as we try to roll back the state today. Most often, they look to sometime in the late 1800s as that golden age.

For example, in a fund-raising letter that asked people to help find more libertarians, Jacob Hornberger recently argued:

My favorite period in U.S. history is the latter part of the 1800s. It wasn’t a pure, 100-percent libertarian society by any means, but it is the closest that mankind has ever come. Imagine: No income tax, IRS, drug laws, DEA, Social Security, Medicare, or Medicaid; no farm subsidies, licensure, welfare, public schooling, gun control, immigration controls, Federal Reserve, paper money, minimum wage, or price controls; no military-industrial complex, CIA, NSA, TSA, FBI, Homeland Security, foreign aid, foreign wars, or foreign interventions. That was the most dynamic, exciting, invention-filled, prosperous, charitable, and free society ever.

I want to point out two major problems with this kind of argument. First, it’s just empirically false that the late 1800s were “the closest mankind has ever come” to a libertarian society or that it was anything resembling “the most…free society ever.” So as a claim about history, it’s just wrong. Second, and related, as a strategy for attracting more libertarians, it’s downright awful. Both of those problems come from the same source: ignoring the fact that over half the US population was significantly burdened by government interventions that no longer exist today. Hornberger, and other libertarians who make this kind of argument, ignore the fact that people of color, women, non-Christians, sexual minorities, immigrants, and many others were far from “free,” and hardly thought that era was a model for an ideal society. By being unable to see beyond what was perhaps true for white, propertied, males, libertarians who indulge in what we might call “nostalgi-tarianism” do libertarianism a huge disservice.

If we want to talk about how free a society is, we need to consider at least two dimensions. There are the things that Hornberger lists above, all of which are important ways in which the state might or might not limit people’s freedom. And he’s surely correct that those government interventions were largely absent from the latter part of the 1800s. However, he ignores the second dimension of freedom, which is the number of people to whom a full set of rights were available. In some sense, the total freedom of society is the sum of the freedoms of the whole population. And if some of that population lacks freedoms in important ways, that fact weighs against the more expansive freedoms of other groups.

In the case of the late 1800s, we have precisely this problem. For the moment, I will concede the claim that white, propertied men were freer then than now. Even if that’s true, we have to consider the freedoms of non-whites, women, and the other groups mentioned earlier. The post-Civil War era was still one of immense state-generated structural racism against African-Americans, including Jim Crow laws, state-sanctioned lynchings, barriers to employment and admission to public colleges and universities, and limits on their voting rights. Though the limits on married women’s ability to own property and sign contracts for themselves were falling during this period, they were still in place in a number of states depending on exactly which date you want to point to. Married women could still be beaten or raped with impunity by their husbands. Most states denied women the vote and the possibility of serving on a jury. Women faced pregnancy and marriage bars on employment, both public and private, until as late as the 1930s and 40s. Women faced similar barriers to higher education as did people of color. In addition, homosexual behavior could easily get you thrown in jail or worse, and Jews faced many of the same barriers as did blacks and women.

In our own time, most of these limits on freedom are gone. Given that women and non-white men make up over half the population, ignoring these draconian limits on their freedoms in the late 1800s is to ignore an enormous role played by the state. It’s true that most of those interferences with freedom would not show up in a measure of the scale of government activity, such as the percentage of GDP devoted to government expenditures, but that doesn’t mean the scope of freedom they limited was small. As one example, the military draft (which was in place in the late 1800s too) is a huge limit on people’s freedom, yet the actual cost of the draft in terms of government expenditures is not huge. The same is true of the aforementioned limits on married women’s economic participation or Jim Crow laws.

If we also consider the myriad ways in which even white men are freer today than they were back then, the nostalgi-tarian argument weakens even more. As one example, all Americans today have the ability to read and watch material that was prohibited back then. The Comstock Laws, passed in 1873, prohibited the use of the US Postal Service, and later other “common carriers” like the railroads, to transport any of the following: “obscenity, contraceptives, abortifacients, sex toys, personal letters with any sexual content or information, or any information regarding the above items.” These were substantial limits on freedom, and ones that applied to all Americans, that are ignored in Hornberger’s argument. After all, it took until the mid-1960s to get rid of a Connecticut law that criminalized encouraging people to use contraceptives. Add to that the demise of sodomy laws and other limits on people’s sexual choices and behavior, and not only are gays and lesbians far freer than they were in the 1800s, so is every straight married couple who wants to engage in oral or anal sex.

When we add it up, it seems obvious to me that over the last 200+ years the gains in economic freedom for women and non-whites, along with the gains in non-economic freedoms for everyone, significantly outweigh the loss in economic freedom for propertied white men. And while I completely agree with Hornberger and others making this sort of argument that the world would be a better place today if women and people of color enjoyed the same economic freedoms that white men did in the late 1800s, that’s a different argument than claiming that the late 1800s was the most free society ever. It seems demonstrably true that, in total, there is more freedom in America today than back then, given the ways in which a full set of rights has been extended to the majority of Americans who lacked them in the late 1800s.

The nostalgi-tarian argument is also a major strategic mistake if one wants to expand the appeal of libertarian ideas to new groups. Most women and people of color read an argument like Hornberger’s and recoil at the claim that the period in question was one of significant and substantive freedom. For many of them, their immediate reaction will be “well if that’s your vision of a free society, it would seem to have no room for me.” To leave unacknowledged the way the state (as well as the culture) discriminated against more than half the of the population, leaving them substantially unfree, is to ignore the real history and concerns of people who might be attracted to a libertarianism that actually heard those concerns. Libertarians have a very good story to tell about how the state, abetted by various Bootleggers and Baptists, has consistently made the lives of women, people of color, and sexual minorities miserable for 200 years by restricting their economic and social freedoms. But that story will never get off the ground if we keep calling the periods of time in which those restrictions were still significantly binding “the most free society ever.”

Part of the blind spot here is that too many libertarians do not want to talk about the importance of equality before the law. When one views libertarianism as part of the long liberal tradition, one has to recognize that it is a fundamental libertarian principle that the law should apply equally to all of us as moral beings. Equality before the law is a core principle of classical liberalism and libertarianism. The earlier arguments aside, you can’t argue that a period of time when the law didn’t apply equally to a majority of the population was a zenith of libertarian principles because equality before the law is a core libertarian principle! No matter how many ways you list that the state had less power than today, it had enormous power over that majority through their inequality before the law, which is a direct violation of libertarian principles. You can list all the alphabet-soup state agencies that you want and call those all violations of libertarian principles, but you also cannot ignore inequalities before the law, which are also violations of libertarian principles, and ones, I would argue, that have a much more harmful effect on human freedom.

Too many libertarians today continue to have a large blind spot about the concerns of people of color, women, LGBTs. It’s writing as if the white, straight, Christian, property owning male’s experience were universal, when it most certainly was and is not. And until we get rid of that blind spot and recognize the different experiences of those groups in American history and how it relates to freedom, libertarians are never going to widen our appeal.

This point is also the important truth in progressive criticisms of “white privilege” and the like. One need not have to buy the whole progressive story to recognize that different groups of people have different experiences of the social world and that writing as if one group’s were universal is to invoke a meaningful notion of privilege. The classical liberals of the 19^th century understood better than many modern libertarians that the fight for liberty was a fight against unearned privileges. Those sorts of privileges are precisely what white men had in the economic realm in the world of the late 1800s in light of the state’s limits on the economic freedom of women and people of color. Being mindful of the concerns of historically marginalized groups is not some form of the dreaded “political correctness,” and criticizing those of us who do take those concerns seriously for indulging in some sort of “snowflake” behavior is just boring and silly right-wing virtue signaling. There’s a substantive and historically-informed conversation to be had about these issues if libertarians are willing to participate in an intellectually mature way.

Libertarians need to recognize the legitimate concerns of historically marginalized groups and be mindful of just who the “we” is when we start invoking a “we” in these sorts of historical arguments. The experience of freedom and oppression is very different for different groups of the “we” that go under the name “Americans.” If we want those folks to hear our arguments sympathetically, we have to include them, their stories, and their concerns into how we understand freedom and American history.

This does not mean that every time we talk about American history we have to don sackcloth and ashes about how horrible America has been to people of color and women, nor does it mean we cannot tell stories of the irreplaceable positive contributions of white men to American history. What it does mean is that we should be asking ourselves how women and people of color might hear what we’re saying and how we’re saying it. Doing so is valuable for its own sake but also helps us to recognize their legitimate concerns. For example, had Hornberger’s appeal included something like the following, it could have avoided these problems:

The late 1800s were a time when governments intervened less with people’s economic decisions, which enabled those who were able to exercise their economic freedoms to create a dynamic and growing economy. Of course the scope of government power that did exist back then was often large, especially in the way it limited the economic opportunities of women and people of color. Imagine how much more dynamic that economy would have been, and how much more wealth and freedom we would have had, if governments had not limited those economic freedoms to a privileged few. Too often today, governments limit the economic freedoms of all Americans, and those with the fewest resources and least power often suffer the most as a result. As libertarians, we want a society that is free, peaceful, and prosperous. That is why we want to eliminate the government programs that stand in the way of all Americans being able to participate fully in the US economy. We look forward to a future in which all Americans can enjoy lives that are free, peaceful, and prosperous.

One can avoid nostalgi-tarianism without turning American history into one long apology. It’s always better to look forward than backward.

In a world where much of the right seems headed toward some form of fascism and more of the left toward some form of socialism, libertarians have an unprecedented opportunity to increase our numbers. The demographic reality of 21^st century America is that libertarianism is not going to grow significantly without attracting more people of color and more women. If the ideas of liberty are going to spread, and the promise of more freedom is going to be achieved, indulging in nostalgia for some past age of non-existent freedom is not going do the trick. We can recognize the power of the greater economic freedom of the past while also recognizing that the opportunities it created were available to a privileged minority. We need to turn our gaze from the past to the future. We should frame our arguments for economic freedom as a continuation of the promise of expanded freedoms for all Americans, and focus on the gains such freedoms will bring for all Americans. Doing so is a much better strategy than an ahistorical and strategically counter-productive nostalgi-tarianism.