You can never have empirical evidence for any nontrivial counterfactual. That’s perhaps one of the simplest refutations of empiricism as an epistemology (although there are others).