Heat the orange and lemon zest, ginger, cinnamon stick, cloves, vanilla vodka, cardamom pods, port, red wine and sugar over a low heat until hot, but not boiling. Strain the mixture through a sieve onto the softened gelatine in a large bowl and stir well.
Spoon the soft fruit into a large glass serving bowl and pour enough of the jelly mixture to fill it a third of the way up. Chill in the fridge overnight, or until they have completely set.
Mix half of the whipped cream with the custard and vanilla in a bowl until well combined and spoon this mixture onto the set jelly.
To serve, spoon the remaining whipped cream onto the trifle and scatter over the flaked almonds.