I follow your point that predictions are not enough for complete causal relationship understanding. But what starting position would be better: having black box predictions or starting without this information? Perhaps these predictions can serve as hints and could speed up the understanding.
What about developing predictions with simple models that can be introspected and understood. A decision tree or a regression equation? Trade-off some accuracy for understandability.
