No work is done on the wall because a person’s muscular force cannot displace a solid wall. It is very tiring to push the wall because in order to make a displacement, you keep increasing the force and exhaust a good amount of energy in the process.
It has to do with the process of working muscle. Muscles are basically the collection of filaments, and these can either slow down or contract during muscular activity. Each filament cannot maintain a constant force. In order to apply a constant force of larger magnitude, the filaments are contracting or slowing down at regular intervals.
Hence, at the microscopic level, your muscles are doing work; although you are doing no work with respect to the wall. Thus, it is tiring to push hard against a solid wall.