When reading Mark Zuckerberg’s recent op-ed in the Wall Street Journal, one passage made me take a second beat:

As a thought experiment, imagine only one person had a superintelligent lawyer. He would have an unfair advantage in court—even if his position was wrong on the merits. That would lead to a worse society. But now imagine everyone has a superintelligent lawyer. Justice would be carried out much more fairly and efficiently than it is today. When people are empowered, they naturally check and balance each other and the power of larger institutions.

While Zuck presents this as a thought experiment, it’s a yarn that could only come from the Bay Area. I’ve been in the rooms that have produced similar ideas from top engineers. They imagine a world without friction and impediments, without lawyers and law.

For those in tech who are able to do so much with code, it is tempting to assume that intelligence is the binding constraint elsewhere in the economy. But knowledge that matters in business isn’t just raw intelligence. It is often the ability to navigate institutional structures.

Just in the practice of law, empirical research has found that a lawyers’ advantage is explained by their ability to navigate procedures and institutional relationships, not by their abstract knowledge of substantive law. Abstract knowledge remains important, but it is often complementary rather than sufficient.

Zuckerberg wants us to imagine a world where everyone could get a superintelligent lawyer. But what good is that when a human lawyer has to file your motion in court? Human attorneys remain the legally recognized interfaces between humans and the state.

Many professions are like this. Therapists and physicians, architects, engineers, crane operators, accountants, electricians, plumbers, pilots, pharmacists, and real-estate brokers all need licenses. In total, around a fifth of all jobs are staffed by people who have to be formally certified to perform a job.

And courts haven’t been especially amenable to automating these licenses. The US District Court for the Eastern District of North Carolina, for example, upheld a North Carolina restriction on drone operators, claiming their maps and models amount to illegal “surveying,” which is regulated by an occupational license in the state. But occupational licenses are just one restriction among many that privileges humans.

Across industries, humans are often required to approve a decision, be on site, hear an appeal, bear liability, certify the system, or be a public representative. Our legal and regulatory system is suffused with interlocking requirements that create systemic human privilege by defining, for example:

• Who may act — Occupational licenses require government approval to perform the legally defined activity. This is the strongest form of exclusion because the machine cannot independently qualify to provide service.  
• Humans must be involved — Staffing requirements exist throughout the economy. In the US, for example, freight trains are required to have two crew members on board, while European countries have gone to one person.
• Who decides — Automation may recommend a course of action, but only a human may issue the legally operative decision. For example, a doctor must make the final diagnosis even though an AI system summarized all of the evidence.
• Who controls the machine — A human may do almost nothing in ordinary operation but the law nevertheless requires a person to occupy the role. Pilots often use automated systems to fly but commercial airplanes must be staffed with a pilot and copilot.
• Who bears responsibility — Corporate officers have to certify regulatory filings just as a broker-dealer must supervise trading algorithms.

In addition to what’s already on the books, legislators are passing new laws that further define these roles. Rhode Island requires “grocery stores with self-checkouts to have a minimum of one staffed checkout for every three self-checkouts operating.” This past year, seven different states have passed laws that require human involvement when making insurance coverage or reimbursement decisions.

In the years ahead, our political system will have to resolve countless thorny questions about the boundary between humans and machines. Can an AI bear legal responsibility, hold a license, certify a decision, or represent someone before the state? Or will those powers remain vested in humans, no matter how capable the technology becomes?

How our political and legal system answers these questions will determine how deeply AI transforms the economy.