I mean sliding window attention is the most basic way of getting long context window. For the OCR case it seems like it should be even simpler, since you don't even need to have the "sliding" portion, unless I"m missing something you don't need to retain anything about the previous pages to OCR a new page so you could just pick a short context window and restart from scratch each time. [^1]

Were people really trying to do OCR with vanilla attention?

[^1] Although maybe I guess looking at their demo, tables that span multiple pages might be a use-case for having some look back.