How concerned should you be that the buyers’ surveyor has found woodworm?

Woodworm is actually a beetle not a worm. (You may not be too worried about the correct taxonomy so long as they are not munching through your joists and rafters).

There are a variety of wood boring beetles which lay their eggs in wood. Usually the sign is holes in the wood, which is how adult insects go back and forth to the grubs inside. Concerns about woodworm are usually overdone. Woodworm don’t spread like a plague. They need fairly moist conditions, and not too much heat. Humans are quite the opposite: they need fairly dry conditions and a fair amount of heat. So, normal family living in a properly heated and ventilated house should be death to woodworm.

If there is no sign of wood dust or pellets around the holes, it may well be that this is simply an old infestation which has been treated; or the insects may have died out naturally because conditions ceased to be congenial for them.