Stir together water, pumpkin purée, maple syrup, cinnamon sticks, ginger, cloves and allspice in a small saucepan. Bring to a boil over medium-high heat.
When the mixture boils, submerge the tea bags, cover, turn off heat and allow to steep for at least 15 minutes.
Strain the pumpkin mixture in a fine-mesh sieve with a small bowl or measuring cup underneath. This should yield 2 cups of chai concentrate.
Heat milk in a small saucepan. When it starts to steam, turn off the heat and whisk vigorously. Alternatively, you can use a milk frother.
Pour the pumpkin chai concentrate into a latte mug. Then, gently pour the steamed milk on top. If desired, garnish with ground cinnamon.