Bricks are made of clays, and there are many different types of clay minerals all including metal ions within their silicate lattice. Below is an example of a model of the atomic structure of a clay mineral called Kaolin. The ions, or salts, sit in a layer between silicate ions (the red and black pyramids) and are leached out by weathering forming the silvery layer you observe. You can call it dirt, but it's a natural process and in time the salts will be washed away. Clays minerals themselves are formed as a result of weathering of primary minerals.
In the UK, we get China Clay from Cornwall. Cornwall sits about a huge pluton of granite that formed hundreds of millions of years ago. It's magma that never made it to the surface. That granites slowly chemically weathers to form clays which we use to make bricks. It's all a cycle of nature, and understandable through physics and chemistry.
View attachment 186846