I think I located my answer.
"If hole alignment is a problem, putting a slot in the joint material facing the setting tool and using a hole of the recommended diameter in the second joint material will give the riveted joint its maximum strength."
This indicates that's fine-ish. Not sure about slotting exactly, the hole could be off in either direction but it is probably more likely to be off perpendicular to the sheetmetal bend axis....
I'm thinking I'll give it a shot and see how a clearance hole works...