Hi and welcome to the forum.
Maybe contact the manufacturer of the unit and run it by them. You might also find a suggested minimum stack diameter in some sort of installation or operators guide for that particular unit? As you stated... going from 8" to 5" will give you a hole that is 39% of the area of what the unit was apparently designed for. I'm guessing yes on the adverse effect but can't say for certain. Nobody likes to poke a hole in their roof but it really isn't that big a deal. Might want to bite the bullet and hire a professional / insured installer and feel good about the install being as the unit builder had intended...?