Hi,
I have been in touch with the authors of JT/FT8 regarding the correct quality settings for this mode.
WSJT-X requests a 48000 Hz 16-bit sample stream and creates the same for transmit. WSJT-X down samples to 12000 Hz 16-bit as a first step on receive so if you choose 12000 Hz linear encoded 16-bit then at least you will have not taken anything away from the sample stream quality. But even then the operating system will have to up sample to 48000 Hz before feeding to WSJT-X, the effects should be pretty minimal. You can try 24000 Hz linear encoded 16-bit but in my testing there was little difference apart from consuming more network bandwidth and introducing more likelihood of glitches and latency.
Therefore, you should use setting 5 (12KHz linear 16). You should not use a compressed setting. For example, setting 4 (12KHz linear 12), means you will have discarded the bottom 4 bits of audio resolution and that is both unrecoverable and most important because that is where the weakest signals lie. Discarding 4 bits is equivalent to discarding roughly the lowest 24 dB of audio resolution.
In my experience so far using setting 5, I have decoded down to -24dB on rx and been decoded down to -24dB on tx.
Hope this information is helpful.
73 Graham VK3GA
