A Spectrum analyzer (whether in the RF Domain or optical) is a tool that is dual of the oscilloscope. An oscilloscope displays a waveform in time domain. When this is represented as a function, a Fourier transform can be used on it to obtain its spectrum. A spectrum analyzer displays this content.
Spectrum analyzers are very similar to radio receivers. A radio receiver could be classified into many types: (Super)heterodyne, crystal video, etc. Similar to a heterodyne receiver, which features a bandpass filter, mixer and low pass filter, a spectrum analyzer must tune over a specific range. This range must be very narrow, which requires a high Quality factor bandpass filter to operate. This is where the YIG (Yttrium Iron Garnet) filter comes into play. YIG has a very high quality factor and resonates when exposed to a DC magnetic field. This is what determines the spectrum analyzers “resolution bandwidth”. Of course, a narrow RBW means a less noisy display and better resolution. The tradeoff for this is increased sweep time. The sweep time is inversely proportional to the RBW squared.
A sweep generator is used to repetitively scan over the frequency band. The oscillator sweeps and repetitively mixes/multiples with the input signal and is filtered with a low pass filter. The low pass filter determines the spectrum analyzer’s “video bandwidth”.
An important concept with regards to bandwidth is thermal noise. Thermal noise is the single greatest source of noise in systems under 100 GHz. Past 100 GHz and into optics, shot noise becomes more apparent. However, bandwidth is the greatest contributor to thermal noise, as noise power is given as kTB. Since k is a constant and T has a relatively negligible effect on thermal noise (the main thing is that T is nonzero. At absolute zero, you have no thermal noise. Anything above that, you have thermal noise. The difference between a pretty cold device and a scorching hot one is only maybe 10 dBm or so. Just ballparking), this means that bandwidth has a huge effect on noise. A higher RBW increases the spectrum’s noise floor and makes it harder for closely spaced frequency components to be seen, as more frequency components are passed through the envelope detector.
Video bandwidth, on the other hand, typically determines resolution between power levels and smooths the display. It is important to note that the VBW contribution happens after data has been collected and does not affect the measurement results, whereas the RBW dictates the minimum measurable bandwidth.
Phase noise is also present in a spectrum analyzer and can affect measurements near the center frequency and results from phase jitter. Since this is pretty much a phase modulation, sidebands are produced near the center frequency which can interfere with measurement. Jitter refers to deviation from periodicity of a signal.