Boil sweet potatoes until they are very soft; about 25 minutes. Drain sweet potatoes and transfer them to a mixing bowl. Add brown sugar, butter, milk, vanilla, cinnamon, ginger and nutmeg. Mash until thoroughly combined and smooth. Pour mixture into a baking dish. In a small bowl mix pecans, brown sugar, flour and butter. Crumble over the top of the mashed sweet potatoes. Cover with aluminum foil and bake at 375°F for 30 minutes.