I did a bit more research on window function settings. Unfortunately, I'm finding it very difficult to figure out what settings people generally use for JTest, so what I've read thus far is just general information on the different window settings.
Originally, I had used the flat-top window because it most closely matched the look of other jitter measurements I've seen (i.e. the response starts to arc up at the base of the signal being played, which is 12KHz for the 48KHZ test).
Unfortunately, flat-top isn't the only window setting that produces similar results depending on the gear you're measuring, so I'm really not sure what others tend to use. I'm assuming there is a good reason for using what they do...
But, for the revised tests, I thought it would make more sense to run with whatever window function setting gave the most detailed amount of information. Generally this was the Kaiser5 or 7 window function setting. Otherwise, it seemed that a lot of measurement information would get "buried" (see my original Metrum ARTA spectrum measurements). As such, I also decided to try it with JTest, and it made it easier to see what was going on closer to that 12KHz signal.
Here is something I read about Kaiser windows: "The Kaiser-Bessel window is useful for detecting two signals of almost the same frequency but with significantly different amplitudes." My tests do seem to better display fine details with Kaiser window settings. Got that quote from:
http://zone.ni.com/reference/en-XX/help/371361H-01/lvanlsconcepts/char_smoothing_windows/On the other hand, the SB1240 did not show that arc before and after 12KHz even with the flat-top window, so perhaps such windows are actually used for a particular reason.
As for why the NOS1704 exhibits that dip in noise before and after 12KHz where as the Metrum did not with the Kaiser window setting, I do not know. You can see from my first NOS1704 jitter measurements that the different window setting does not exhibit the same dip. (Keeping in mind I'm looking at the non-JK measurements, which already did funky things.)
In the future, I will probably just measure using a few different windows and post all of them. As far as I could tell, it only really impacted the response immediately surrounding the test tone being played (sidebands stayed the same but might get "buried" when the response arcs up).
Update: The following page shows they use Hanning for their jitter tests. I think that's the default setting in ARTA. It might not matter so much as long as the sidebands and other jitter artifacts are clearly visible. They also have a rough chart for quick approximations to turn dB into ps, but I'm not sure how correct it might be. These measurements of the NOS1704 would likely mean more than 500ps of jitter. The Metrum was able to get below 180ps according to this chart with the same setup.
http://www.lessloss.com/page.html?id=42
