How a lawsuit about pencils can protect rights in the AI era

A new federal appeals court ruling in a case about one of the simplest forms of technology — the pencil — could prove important for ensuring government accountability in the age of artificial intelligence.

The U.S. Court of Appeals for the Federal Circuit last month issued a decision in a trade-related case involving U.S. Customs and Border Protection (CBP). The CBP had ruled that an importer of pencils had violated trade rules, while the importer argued that CBP had violated its due process rights. The dispute might seem banal, but it has much broader implications for governmental use of machine-learning algorithms developed by private contractors.

Many government agencies need to rely on private contractors to provide the expertise and skills to support AI-decision systems. When private contractors claim trade secret protection over their algorithms, they create a kind of legal “black box” question: Does the trade secret protection claimed by contractors inevitably deny individuals or corporations their due process rights? The clear message from the Federal Circuit’s recent decision is that trade secret protection must give way to due process.

To understand the importance of this new decision, consider an earlier case involving algorithms and due process. In 2017, a federal trial court held that a teachers’ union had a viable due process claim in a case where the teachers were denied access to information about an algorithm that the Houston school district used for professional evaluations. Because the school district had relied on a private contractor that claimed trade secret protection over the algorithm at issue, the school district argued that it could not provide more information to the teachers. The court denied the school district’s motion for summary judgment, reasoning that “when a public agency adopts a policy of making high stakes … decisions based on secret algorithms incompatible with minimum due process, the proper remedy is to overturn the policy, while leaving the trade secrets intact.” 

Back to the case decided last month by the Federal Circuit. Royal Brush Manufacturing v. United States landed in federal court when an importer of pencils challenged CBP’s accusation that the importer had evaded trade rules by claiming its imported pencils were made in the Philippines rather than in China. In its challenge to CBP’s decision, the importer argued that its due process rights were violated because it was not given access to photos and data that CBP had used in reaching its decision. CBP argued, though, that it couldn’t provide the importer with this information because they related to a third party — the manufacturing company in the Philippines that the importer claimed to have made its pencils. According to CBP, the business data collected from the Philippines’ firm, along with the photos of its manufacturing facilities, indicated that the firm lacked the capacity to make all the pencils that the importer had claimed to have bought. But CBP wouldn’t turn over the business data or photos to the importer because they comprised confidential business information, and the agency had a statutory obligation to protect that confidentiality.

The Federal Circuit rejected the government’s argument, reasoning that because the Due Process Clause of the Constitution requires adversely affected parties to see the information that government relies upon, this constitutional requirement trumped any statutory prohibitions on governmental disclosure of trade secrets: “Because the Constitution authorizes, and indeed requires, the release of confidential business information in this case, the Trade Secrets Act does not stand in the way of such release.” The Federal Circuit held that CBP could have shared the confidential business information subject to a protective order that would have prohibited its further disclosure.

If the Federal Circuit’s ruling is to be followed elsewhere, the upshot could be significant for anyone seeking to challenge agencies’ use of artificial intelligence on due process grounds. Litigants challenging the government’s application of machine-learning algorithms might now be able to rely on the Federal Circuit decision to gain access to information about those algorithms, even when they are developed and deployed by a private contractor who claims trade secret protection, as in the Houston case from 2017. Sunshine, in other words, could more easily penetrate the black boxes of an algorithmic state.

One difference between the two cases remains: CBP actually possessed the confidential business information requested by the importer, while the Houston school district apparently did not actually possess the information demanded by the teachers. As a result, more litigation can be expected. And, at least for now, agencies may continue to find that the safest step, as I have argued elsewhere, is to include disclosure provisions in public contracts for AI-related services. These provisions could require data analytic firms to share information about their algorithms, at least in the context of administrative or judicial challenges.

In the absence of contractual provisions, though, the decision in the Royal Brush case still suggests that litigants may be able to pierce through any veil of trade secret protection surrounding AI use by government agencies. For this reason, a court decision in a case about pencils could turn out to hold the key to accountability in an era of increasingly digitized government.

Cary Coglianese is the Edward B. Shils Professor of Law and Political Science, and the director of the Penn Program on Regulation, at the University of Pennsylvania.