It should be fine. There's so much dried fruit and nuts in the cake it shouldnt make a big difference.
If you are going to add brandy to the cake after it's baked, infuse the brandy with the zest. Heat the brandy. Remove from heat. Add the zest. Let sit for 10 - 15 mins. strain the zest out before adding to the cake.
Or you can make a lemon - orange simple syrup and add to the cake.