Preheat oven to 350 degrees. Place whole apples on a baking sheet and bake until brown and tender, about 25 minutes. Gently crush the allspice berries and break up the cinnamon stick. Place the allspice, cinnamon, cloves, nutmeg, and ginger in a cheesecloth bag. In a large sauce pan place 2 cups of the ginger ale and the spice bag. Bring to a boil, reduce heat and simmer for 10 minutes. Remove spice bag and add the remaining ginger ale, the sherry and the brandy heat until hot without boiling. Whisk in milk and pour into a punch bowl garnished with the baked apples.