Whisk together flour, white sugar, and salt in a large bowl; cut in butter with a pastry cutter until the mixture resembles coarse crumbs.

Stir rum extract, almond extract, and water together in a small bowl. Stir into flour mixture, a little bit at a time, until mixture holds together in a ball when you squeeze it.

Place dough on a floured work surface and sprinkle with flour; roll out into a sheet 1/4 inch thick. Cut into shapes with floured cookie cutters, and place 1 inch apart on ungreased baking sheets. Cover with a layer of plastic wrap, and refrigerate for 30 minutes.

Preheat the oven to 325°F (165°C).

Bake cookies in the preheated oven just until the edges are lightly browned, 15 to 18 minutes. Allow to cool on sheets for 2 minutes before removing to wire racks to finish cooling. Cool completely before decorating.

To make the frosting: Whisk confectioners' sugar, 2 tablespoons milk and 2 teaspoons milk together in a bowl until smooth. If desired, divide frosting into small bowls and tint each to desired shade with food coloring.