Preheat oven and line the bottom of a deep 9 inch round cake pan with parchment paper and grease well.
Place the peach slices in the bottom of the prepared pan in a single layer.
In a medium sized bowl, combine butter, brown sugar, nutmeg, and salt. Spoon brown sugar mixture over the sliced peaches in the pan and set aside.
To make the cake, whisk flour, baking powder, baking soda, cinnamon, and salt together in a medium bowl. Set aside.
In a large bowl, whisk sugar and butter together until smooth and fluffy. Add in the eggs and whisk until well incorporated. Add the milk, sour cream and vanilla and whisk until well combined.
Add the dry ingredients and mix just until combined.
Pour cake batter into the 9-inch cake pan over the peach layer.
Place the pan into the preheated oven and bake for 45-50 minutes or until baked through.
Remove from oven and let the cake cool for about 20 minutes.
Place a plate over the cake pan then flip it over to remove the cake from the pan. If it doesn’t come loose immediately, gently tap the bottom of the cake pan until it is released.
Serve with a scoop of ice cream or whipped cream if desired.