Why does light reflect off a silvered surface but scatter off a painted wall? Is the explanation geometrical (depending on the flatness of the mirror surface relative to the wall) or is it chemical (depending on a quirk of silver)?

Some of both. It’s a chemical quirk of silver that enables it to reflect almost all the light energy, and to reflect all visible wavelengths about equally so the reflected light is about the same color as the incident light. It’s the flatness of the surface that lets the reflection be specular and show an image, as opposed to diffusely scattering the light back. An unpolished metal surface won't give a mirror image. I’d guess rough or unpolished silver just looks grey-white.

The free electrons in the silver minimize the penetration of light into the metal and so it is really the smoothness of the silver surface, and not anything beneath the surface, that counts for the specular image.

Non-metal objects lacking free electrons, though less reflective than a metal, can still give a specular reflection from a smooth surface, e.g., the surface of a still pool of water or a shiny leather shoe.

Paint is not a homogeneous substance like silver, but consists of pigment particles suspended in a transparent liquid. Liquid might penetrate further into paint than it does into metal, so depending on the paint, you might get a bit of glossy reflection off the smooth top surface of the paint, and a more diffuse reflection off the particles in the liquid below the surface.

There are paints that give glossy finishes and paints that give matte finishes. Presumably the difference is the roughness of the surface and/or subsurface, as determined by the size and concentration of the particles in the liquid.