Threads waiting for work will not degrade performance. If your work involves those threads waiting for I/O then that in itself will not degrade performance either (as long as they block and don't poll of course).
All live threads consume resources however so if you are short on threads or memory then don't create too many of them.
Pre-starting a large number of threads in a pool can cause a startup delay of course. Generally you should let the pool create threads as needed until it gets to the core pool size.
Well of course it will slow down things a bit, lists in kernel space get longer, more work needs to be done by the scheduler and becaus eof larger data structures L2 cache misses are likely to happen.
You should set a useful value which makes sence instead of a too large "fits ever" value.
lg Clemens