Public Lab Research note


New wavelength calibration procedure preview for Spectral Workbench 2.0

by warren | September 30, 2015 20:03 30 Sep 20:03 | #12258 | #12258

What I want to do

There's long been talk about a more intuitive, more precise, all-around better wavelength calibration procedure for Spectral Workbench, and I've spent time over the past couple weeks putting that together into a working interface. I'm generally tracking progress in this Github issue

It's not complete, but you can see what it's like by cloning this spectrum to play with:

https://spectralworkbench.org/spectrums/show2/58236

How it works

As you can see in the above screenshot, when you click on "Calibrate" in the new 2.0 Spectral Workbench beta interface, you're presented with two sliders, and a reference spectrum, which is actually @cfastie's "snowsky" CFL calibration, one I've used a lot as it's super sharp and has very low noise: https://spectralworkbench.org/spectrums/4474

The steps are:

  1. Click "Calibrate" in the new 2.0 beta interface.
  2. It will attempt to auto-calibrate. If it looks good, just click "Save."
  3. If not, drag the sliders yourself. The "Snap" checkbox will make the sliders "stick" to nearby peaks.
  4. Once you have a good match (the red vertical lines should intersect the graph peaks very precisely), click "Save."

Note: It does not handle image reversal well. Sometimes reversed images can cause problems, so if it looks like the auto-calibration is opposite -- i.e. right to left instead of left to right -- you could try to "Set new cross-section" which will get fresh graph of data out of the image. We hope to solve reversal problems once and for all in an upcoming version.

"Fit" and next steps

You may notice the green indicator that says "FIT" to the right. This is an attempt to compare the expected ratio of three known peaks to the peaks found in the image. Unfortunately, it has some shortcomings, and we'll probably be switching to a root-mean-square error approach, where we compare a much larger # of points to those of a known spectrum, after height-adjusting for the baseline and maximum peak height. We'll see if that works better, but for now, please be aware that the "FIT" indicator is not that reliable.

Why I'm interested

This project has integrated several ideas:

  • auto-calibration, based on @sreyanth's work during last summer's Google Summer of Code
  • a better, more visual UI for calibration
  • a measure of "how good" calibration is

Hopefully in the future, we can combine these -- if we can collect data on how good auto-calibration works for many different spectra, we could say that if it's "better than 90% fit" or something, it just auto-calibrates it in one click. But if it doesn't manage to get a good fit, it prompts the user for help.

Have trouble? Input, suggestions? Leave a comment!


21 Comments

That's a wonderful interface. I calibrated many times because it was fun to do.

The interface choices you made work really well, but you might be interested to know that I would have taken a different approach:

  1. I would have allowed the user to stretch the new spectrum to match the calibrated reference, not the other way around.
  2. I would have used the spectral graph trace of the reference instead of the photo of the diffraction pattern.

But it certainly works well the way it is now.

The Snowy Sky spectrum is very clear, but not as accurately calibrated as a calibration reference should be. It was calibrated with a macro by using the 405 nm and the 650 nm lines, so those are very accurate, but most of the other lines are not. For example, the 436 and 546 nm lines used in your calibration procedure are both more than 5 nm off. This is the spectrum that I recalibrated with a third order polynomial using 14 peaks. The final calibrated result exists in Excel but could be transferred to Spectral Workbench. Of course that is just the data, not the photo of the diffraction pattern. Somebody with MatLab or something could probably apply the polynomial to the image.

SW 2.0 looks very awesome.

Reply to this comment...


Thanks, Chris, for the encouraging feedback. A lot of thought went into this, and although in the first version you stretched your spectrum, it was easier to do it this way because:

  • here we don't need to move both the graph and the image, so it's smoother (less CPU)
  • here the thing you stretch is at the top, and there's no inclination to put the example reference between the spectrum and its graph
  • the eye more easily distinguishes color order and balance in the image than in the graph, in my opinion
  • a graph line would've overlapped other graph lines, rather than been placed alongside, which I thought would be visually more confusing (i.e. what's "on top"? which line is which?)
  • this is also a good opportunity to show folks what a really nice image should look like, so they can aim for that

I'd like to use your polynomial calibration, but without having an actual spectrum image which has been corrected, we can't show the user such an example. Do you know of one? Eventually, we should write a calibration function that can do that too -- actually irregularly stretch the image and corresponding data with as many control points as needed -- kind of like image warping with GCPs. I've modularized the calibration code to accommodate this in the future, but it's a big project. Actually, the whole ToolPane system is something designed so that other folks can publish their own ToolPanes (like different calibration methods) as user-generated utilities. That'll be covered in the upcoming API reference in the wiki.

The RMSE 'fitness' function, however, can be used with numerical data, so if you can publish your polynomial calibration as a CSV, I'd love to use it.

For starters, I'd like to use a more precisely calibrated image reference. I need an image with the following dimensions:

  • overall image pixel width (1818px in the current image)
  • distance in pixels to (middle) 436.6nm blue line from left edge of image (586px in current)
  • distance in pixels between blue and (second) 546.5nm green line (538px in current)

These are used in the procedure here: https://github.com/publiclab/spectral-workbench/blob/master/app/assets/javascripts/spectralworkbench/SpectralWorkbench.UI.ToolPaneTypes.js#L310

How was the interface speed on your computer? I am hoping to optimize it to speed things up, but was it usable?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Here is a text file with the 14 peaks I used to calibrate the CFL spectrum (in nm):
14CFLpeaks.txt

Here is a csv file of the intensity at each wavelength of the calibrated (3rd order polynomial) CFL spectrum:

3rdOrderSnowySkyCFL.txt

Here is a jpg image of the calibrated diffraction pattern of the Snowy Sky CFL spectrum (the emission lines are where they should be according to this source):

3rdPolySnowy1818.jpg

I think the jpeg has the dimensions you requested.

The slider is definitely a little jerky, but it does not affect the function very much.

Reply to this comment...


I did a check on the files I attached to the last comment and they are not as precise as I thought they would be.

SSCFLalign.png
A graph of the data file of intensity x calibrated wavelength (pink) overlain on the corrected diffraction pattern. Both data sources are from the Snowy Sky CFL spectrum.

The reason for the imprecision is that the image of the diffraction pattern was corrected differently than the data series of emission peak intensities. The diffraction image was corrected manually in Photoshop by moving individual emission lines or groups of lines so they matched an overlay of the known emission lines of a CFL. So the lines in the image should be within a nm or two of where they should be. The data file is the result of applying a 3rd order polynomial correction (based on the same known emission line data) to the intensity series. This should also be accurate to within a nm or two. But the error for the two results is not necessarily in the same direction for any particular line, so there is a discrepancy of a nm or two for some lines.

This is probably good enough for your current application, but be aware that these data sets have their limitations.

Reply to this comment...


OK, so we have two different needs:

First, a list of known spectral peaks:

  • from some known trusted
  • maybe an external source, like NIST?
  • we only need wavelengths and intensities
  • we should use this long-term as a means of assessing how good a calibration spectrum is

I'm going to use the 14CFLpeaks.txt file for this.

Second, an example reference spectrum:

  • which we can upgrade later
  • which should be viewable alongside existing spectra
  • which has as good precision/accuracy as we are hoping users will achieve

For this, I agree that we don't need something to the same standard as the first case. I'm curious though --

corrected manually in Photoshop by moving individual emission lines or groups of lines

How did you do this? With the Warp tool, somehow? Or did you just marquee select lines and translate them left and right?

Also, in the jpg of "the calibrated diffraction pattern of the Snowy Sky CFL spectrum" are the peaks at the same positions, i.e. 586px from left, and then 538px from there, as in the image I used? Or should I re-measure that?

One more thought. Although this would only work for a monochrome image (although we could colorize it for clarity, in theory), we could generate an image directly from the graph data, so it's 1:1. We could even do so from some external source, like NIST or something. Would that be of interest?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Oh, and is the 14CFLpeaks.txt from the Wikipedia reference? If so, it says:

Spectrum with peaks labelled taken with an Ocean Optics HR2000 spectrometer [1] of ambient light provided by fluorescent lamps. Spectrum taken by me (apparently en:user:Deglr6328). The spectrometer appears to be about ~.6 to .8 nm off judging from the location of known peaks. Interpretation of spectral peaks has been done using the NIST database of spectra for mercury [2] and an article on fluorescent light phosphors [3]. This spectrum is not calibrated for intensity.

Which should still be OK as a reference. Although if all 14 are available at http://physics.nist.gov/PhysRefData/ASD/lines_form.html, we could just use those.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


OK, I've filtered out the highest NIST peaks, and exported them in plain text. But these are in a vacuum, and for example, the 436.6nm line in the Wikipedia reference shows as 435.8335nm. Is this because it's in a vacuum? Or is 435.8335nm the more accurate number? @stoft, any input on this?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Also, I'm not clear what Hg II means -- an ion? Would we only be seeing Hg I, or would we see both? I'm wondering about the Hg I line at 546.0750nm vs. the Hg II line at 542.5253nm, which seems MUCH brighter, but that's not the case in the CFL spectra we see, or in Wikipedia's Ocean Optics example. Does that mean we aren't seeing any Hg II lines and I could filter those out of the NIST results?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


I see that the 14 peaks you've chosen are sourced from the NIST data, not the Wikipedia/Ocean Optics data.

OK, one last issue -- for intensities on the RMSE, I'm wondering if I can just use NISTs relative intensities, stretched to 0-255, or if enough of the peaks are not mercury, and so I should just use the intensities from the snowy sky spectrum. Thoughts?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Correcting the diffraction pattern was done this way:

marquee select lines and translate them left and right?

This was done in the Photoshop file that produced the lead image on the original note http://publiclab.org/notes/cfastie/3-6-2013/polynomial in which the dashed lines are the 14 emission lines I used for the polynomial calibration. These 14 lines are from the right hand column in the Wikipedia source (https://commons.wikimedia.org/wiki/File:Fluorescent_lighting_spectrum_peaks_labelled.svg). There is no guarantee that these known lines are the same ones produced by the particular CFL bulb I used, but the 14 I chose seemed to be pretty unambiguous.

Most of these 14 lines are not from mercury, so knowing the relative intensity of the emissions from all three elements might be hard. The intensity probably varies among bulbs manufactured with different recipes for the phosphor coating. Using the intensities from Snowy Sky is probably the way to go. However, the intensity is also a function of the camera's sensitivity at different wavelengths and will therefore vary among cameras. The PowerShot S95 used for Snowy Sky likely differs from the webcams.

I have never been able to parse the NIST database results. I don't know what Hg II is. The Wikipedia source seems to match what I get from a CFL bulb, so I just rely on that.

I measured the pixels to scale the jpeg to your specifications but there is error involved in deciding exactly where each peak is. You should confirm that it is good enough for your purposes.

we could generate an image directly from the graph data, so it's 1:1

It would much easier to use Spectral Workbench to do the reverse. Submit the jpeg of the corrected diffraction pattern and get a trace that is a perfect match. Calibrate it with a macro using the two endpoint peaks and it becomes the new NIST.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Now there is a new set of references.

poly-Snowy-ADJ2ovc.jpg
A newly corrected image (background) of the diffraction pattern of Snowy Sky CFL with emission lines shifted more carefully to match the 14 emission lines from Wikipedia (dashed lines), and the intensity trace (pink) derived in Spectral Workbench from the new image of the diffraction pattern.

Definitely not traceable to NBS.

https://spectralworkbench.org/spectrums/58584

Reply to this comment...


Hg (II), Hg (I), and Hg (0) are oxidation states of mercury. Hg (II) and Hg (0) are by far the most common forms of Hg in the environment -- Hg (I) is quite rare. However, I'm not sure if it would be prominent in compound fluorescent lights. I was under the impression that CFLs used Hg (0), since it is fairly inert whereas Hg (II) is reactive and will form compounds, but I'm not 100% sure. The Hg atomic absorptions spectrometers monitor at 253.7 nm, but alas, that is outside the range of our spectrometer.

Reply to this comment...


Oh, and clarification for my comment above: oxidation states are simply different common ions. Hg (0) is neutral, Hg (II) is short two electrons, so has a +2 charge.

Reply to this comment...


Hmm, is there a reason Hg (0) is not listed in the NIST spectra? Perhaps Hg (0) does not have any intense peaks? My list is filtered for most intense peaks only. Are you sure about Hg (0)? Wikipedia (which could be wrong of course) states:

Mercury exists in two main oxidation states, I and II. (citing Wang et al, 2007)

There was also some good information here: https://en.wikipedia.org/wiki/Fluorescent_lamp

And finally the mercury-vapor lamp article includes a listing of emission lines, including the ones we use for linear calibration, listed as 435.8nm and 546.1nm.

It notably does not specify which oxidation state, or list a second green peak, which makes me think that perhaps the 2nd green peak is indeed another element fluorescing? The article also provides this image of a mercury vapor lamp with no coatings, so it should be just pure mercury:

pure-mercury

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


I am absolutely positive that the most common forms of Hg are Hg(0) and Hg(II) -- my whole PhD is on mercury, specifically on Hg isotopes. The only thing I can think of for why they might say that in Wikipedia is that oxidation states are officially described for atoms that are in molecules or forming molecules, and Hg(0) is monatomic and inert (mostly), so maybe it is not an official oxidation state? Regardless, everyone discusses Hg(0) as an oxidation state.

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Ah, it makes sense why Hg(0) won't have a fluorescence line on compound fluorescent bulbs though: the liquid Hg(0) in the tube needs to be ionized in order conduct electricity!

Reply to this comment...


Oh, cool -- can you link us to your dissertation, as a source? I'm hoping this note can be a sort of bibliography for the calibration code decisions.

OK, so does Hg (0) have any emission lines at all? If not, could that be why it's not listed in the NIST database?

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


OK - I'm adapting the calibration slider to use the photoshop-corrected version of the Snow Sky spectrum.

I read the blue 436 peak at 210px from left, and the green 546 peak at 743px from left. The new zooming tool is great for this!

Does this look right?

Screenshot_2015-10-08_at_7.15.11_PM.png

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


I'm going to revise: 211px and 742px, out of a total of 1390px.

Reply to this comment...


I'm happy to post my dissertation, but it has nothing to do with this (it explores natural Hg isotopic composition and fractionation in the marine environment -- nothing to do with CFLs and fluorescence -- I actually learned from one experiment that is not included in my dissertation that Hg from wastewater treatment plants that sterilize with UV Hg lamps have craaaaaaaazy isotopic composition).

For your reading pleasure, here is five years of my life... ;-) : http://deepblue.lib.umich.edu/handle/2027.42/39366/browse?value=Gehrke%2C+Gretchen+E.&type=author

Is this a question? Click here to post it to the Questions page.

Reply to this comment...


Cool, thanks! I was mainly looking for a citation for the most common forms of Hg, and/or a clear reference for which is found in CFL bulbs. But awesome!

Reply to this comment...


Login to comment.