Key Points

• Generative AI models do not appear ready to draft regulations for government agency staff. The policies in AI-drafted rules are overly influenced by the number of comments supporting a particular position, and the reasoning is too cursory.
• Current generative AI models show more promise in summarizing comments, which will likely save agency staff significant time and reduce the likelihood that they overlook important comments.
• Technology will undoubtedly advance, and future versions of AI may improve at evaluating the merits of comments and responding to them.

Read the PDF.

Executive Summary

Generative AI has attracted great attention in the policymaking sphere, including for agency rulemaking. This report compares a final rule drafted by a generative AI model with the Department of Transportation’s actual rule. The AI model’s policy recommendations appear overly sensitive to the number of commenters supporting a position rather than the strength of their evidence and reasoning. Moreover, the rule draft is relatively cursory and lacks the explanatory depth expected in agency rules. The results are more promising with respect to summarizing comments, which may save agency staff significant time and increase their responsiveness to comments. Future generations of generative AI will undoubtedly improve, meriting ongoing study of these issues.