Researcher: Eddie Tan

Introduction

This project examines how large language models evaluate corporate disclosures against selected items of SEC Regulation S-K. It tests how the choice of model, prompt, and filing format affects the scores the system assigns.

Methodology

The researcher extracted Items 101, 103, and 105 from Form 10-K filings obtained through SEC EDGAR. Those items cover the description of the business, legal proceedings, and risk factors. The evaluation converted the three items into a checklist of 30 obligations and prompted the models to cite filing evidence and evaluate each obligation.

Results & Analysis

Legal proceedings showed the largest prompt-related score difference in the displayed comparison. For Apellis Pharmaceuticals, the Item 103 score increased from 0.82 with the Detailed prompt to 0.90 with the simpler prompt labeled Native in the table. Its overall score increased from 0.89 to 0.93, while Apple’s overall score was 0.99 with both prompts.

Table comparing Detailed and Native prompt scores for Apple and Apellis Pharmaceuticals across Items 101, 103, and 105
Model-assigned disclosure scores by company, prompt type, and Regulation S-K item.

Conclusion

The report found that model choice and prompt design affected how strictly the system scored some disclosures. The researcher identified comparison against expert-labeled examples as a way to assess when those scores agree with human reviewers.