Melt the remaining 4 tablespoons of butter in another microwave-safe mixing bowl. Once melted, whisk in ⅓ cup of brown sugar until well combined. Pour the butter-sugar mixture into the bottom of a 9"x13" baking dish. Then, evenly distribute the cubes of brioche bread into the dish on top of the butter.
4 tablespoons (½ stick) unsalted butter, ⅓ cup + ¼ cup packed brown sugar, 14-16 ounces (1 whole loaf) brioche
In a large mixing bowl, thoroughly whisk together the eggs, milk, ¼ cup of brown sugar, vanilla extract, spices, and salt. Pour the custard over the bread cubes, pressing down on any dry pieces to ensure they're all soaked well.
⅓ cup + ¼ cup packed brown sugar, 6 large eggs, 2 cups 2% milk, 2 teaspoons vanilla extract, 1 teaspoon ground cinnamon, ¼ teaspoon ground nutmeg, ⅛ teaspoon kosher salt
Crumble the pecan streusel mixture evenly over the top of the bread. Let the casserole sit for 10 minutes (or cover and refrigerate overnight) to allow the bread to fully soak up the custard.
Cover the casserole dish with aluminum foil and bake for 30 minutes. Then, remove the foil and bake uncovered for an additional 25 minutes. The casserole will puff up and be slightly jiggly when it's ready, then deflate as it cools. Allow the casserole to cool for approximately 10 minutes before serving.