Annotating documents helps us to note our thoughts and pass knowledge and insights to others. However, pen-and-paper annotations can be difficult to share or disseminate, and are generally constrained by a document’s available margin space. Thus, prior work has explored the value of linking digital information to physical documents in augmented reality (AR). The smartphone comprises a commonly accessible digital device, but its small screen size and touchscreen interface make it yet ill-suited for annotating. We propose a smartphone AR system that leverages a document’s layout structure to make AR annotation selection and viewing more practical on a small screen. The system enables markerless tracking, digital annotation, and the sharing of those annotations across duplicate physical documents. Twelve participants tested our system’s efficacy and usability for annotating physical documents in AR. We found that our proposed interaction model enables participants to annotate efficiently and easily, demonstrating the feasibility of using smartphone AR to annotate physical documents.