Perhaps at long straights this might be true, but the downforce is essentially only very harmful in the corners if you push the car to use the extra DF. You don't need to, you can deliberate take the corner a bit slower. You can argue "why the extra DF then when you are not using it", but remember that teams still need to do a qualifying.Extra downforce can cause the tyres to degrade faster. There is a relationship with the mechanical properties of the tyre, the rotational speed and time. Degradation is inevitable but a car that is putting more load through the tyres over a period of time will cause them to degrade faster, ie the mechanical properties will degrade.
If there was no qualy, teams would remove the downforce they don't need to use, to get rid of the drag that comes along.
Mind you that while extra DF itself shouldn't be harmful, running bad airflow onto the tyres to get more DF (or better L/D) will make the tyres go away faster.