Blaziken FB LV.X

If I have a Hard Charm attached to Talonflame and use its "Jet Shoot" attack, then next turn my opponent does a simple attack that does 20 damage, can I pick the order of when to apply the Hard Charm's -20 reduction and "Jet Shoot's" +40 recoil? If I use Hard Charm to reduce the net damage to zero (20-20), could I then negate the "Jet Shoot" recoil?

No, it doesn't work like that. The check for effects on the defending player's Active Pokemon, such as Hard Charm and Jet Shoot, both happen at the same step in the damage calculation process. So you don't stop damage calculations mid-step just because the calculation was less than or equal to zero. The net effect of Hard Charm & Jet Shoot result in +20 damage at that step in the calculation process.

Talonflame's "Jet Shoot" attack says, "during your opponent's next turn, any damage done to this Pokemon by attacks is increased by 40 (after applying Weakness and Resistance)." But what if on the next turn the opponent attacks with a Fighting Pokemon that normally only does 10 damage? Since Talonflame has -20 Resistance to Fighting, does the damage reduce to zero and then +40 for the "Jet Shoot"?

No, since the damage was reduced to zero there is no damage done for Jet Shoot to increase. The attack ends up doing zero damage, period.