Preheat oven to 350ºF. Line 16 cupcake molds with paper liners and lightly spray them with non-stick spray.
Mix the milk, melted butter, sugar, lemon juice, and vanilla together. Then, add the baking powder and baking soda. Add the lemon extract if using; stir. The mixture will begin to fizz. Quickly mix in the flour until a smooth batter forms.
Fill about 16 muffin/cupcake molds 2/3 full with the batter and bake for about 25-30 minutes. While the cupcakes are baking, use an electric hand mixer to mix the cream cheese, powdered sugar, and softened butter to make the icing.
Once baked, allow the cupcakes to cool and drizzle the icing over top.