scald for creme caramel to dissolve sugar and add some preheat or it won't bake (poach) properly.
Don't heat milk for brioche. It benefits from a long proof period.
Its optional for pain de mie, if you have fresh yeast don't heat the milk, for dry yeast its gonna take all day if the milk is cold.
As you see with pain de mie there are multi variables even with the same item. I scald it, let it cool to 80F....using dry instant yeast.
Creme brule, gotta scald or boil the cream to dissolve the sugar.
I get raw milk from the amish, I usually scald for yogurt, not for breads or anything that gets boiled.
If you look at various youtube videos for pastry cream some bakers mix all the ingredients together cold, then cook from cold, its a really bad idea, the odds of scorching are greatly increased by not bring the milk to a boil first.
Its one thing to mess up a pint of pastry cream but if you try that with 5 gallon batches in a pro kitchen its a almost a certainty you'd have to make the walk of shame to the dumpster.
So, unless you're rolling the dice with raw milk....the odds of a problem coming from milk is slim, its far more likely to be the eggs.
When I worked in high volume places such as hotels, big restaurants, wedding catering etc I always preferred pasteurized egg products, especially cartons of liquid yolks. Its safe and chef was happy I saved time by not fiddling with shell eggs.
so back to the beginning, it depends what you're making, the type of ingredients and which method you choose to make it.