Realtime HDR (High Dynamic Range) Video for EyeTap Wearable Computers, FPGA-Based Seeing Aids, and "Glass Eye"

Steve Mann, Raymond Lo, Jason Huang, Valmiki Rampersad, Ryan Janzen, Tao Ai, Kalin Ovtcharov

Download Video | Abstract as published in SIGGRAPH, article number 11 | Research paper
*This work was presented at Siggraph 2012 Emerging Technology.

Abstract

In this demonstration, we present a specialized version of HDR (High Dynamic Range) imaging (use of multiple differently exposed input images for each extended-range output image [2, 1]), adapted for use in electric arc welding, which also shows promise as a general-purpose seeing aid. Tungsten Insert Gas (TIG) welding, in particular, presents an extremely high dynamic range scene (higher than most other welding processes). Since TIG welding requires keen eyesight and exact hand-to-eye coordination (i.e. more skill and more visual acuity than most other welding processes), being able to see in such extreme dynamic range is beneficial to welders and welding inspectors.
Our “WeldCam HDRchitecture” (abbreviated “HDRchitecture”) system uses one or more cameras, and optional active illumination systems, that can be used by welding schools and professionals to inspect welding in real-time. We present HDRchitecture as either a fixed camera system (e.g. for use on a tripod), or as a stereo EyeTap cybernetic welding helmet that records and streams live video from a welding booth to students or observers, nearby or remote. By capturing over a dynamic range of more than a million to one, we can see details that cannot be seen by the human eye or any currently existing commercially available cameras.
We also present a highly parallelizable and computationally efficient HDR reconstruction and tonemapping algorithm for extreme dynamic range scene. In comparison to most of the existing HDR work [1], our system can run in real-time, and requires no user intervention such as parameters fine tuning. It can also render images with a high image quality up to 1920x1080 resolution. HDRchitecture uses GPUs and multicore CPUs for real-time HDR processing. Our algorithm runs at an interactive frame rate (30 fps) and also enables stereoscopic vision. Additionally, a hardware implementation, which uses Field Programmable Gate Arrays (FPGAs), will be presented. The initial hardware configuration comprises an Atlys circuitboard manufactured by Digilent Inc., which is small enough to fit inside a large shirt pocket. The circuit board includes two HDMI camera inputs, one being used for the left eye, and the other for the right eye, as well as HDMI outputs fed back to the left and right eyes, after processing of the video signals. The circuit board facilitates processing by way of a Xilinx Spartan 6, model LX45 FPGA.
One goal of the demonstration is to show the future development of high dynamic range eyeglasses as a seeing aid and how such technology can be used to enhance human vision in extreme dynamic range scene such as welding.

[1] S. Kang, M. Uyttendaele, S. Winder, and R. Szeliski. High dynamic range video. ACM Transactions on Graphics, 22(3):319–325, 2003.
[2] S. Mann. Compositing multiple pictures of the same scene. In Proceedings of the 46th Annual IS&T Conference, volume 2, 1993.

Additional References from Related Paper

[1] S. Mann, “Compositing multiple pictures of the same scene,” in Proceedings of the 46th Annual IS&T Conference, Cambridge, Massachusetts, May 9-14 1993, The Society of Imaging Science and Technology, pp. 50–52, ISBN: 0-89208-171-6.
[2] S. Mann, “Comparametric equations with practical applications in quantigraphic image processing,” IEEE Trans. Image Proc., vol. 9, no. 8, pp. 1389–1406, August 2000, ISSN 1057-7149.
[3] M.A. Robertson, S. Borman, and R.L. Stevenson, “Estimation-theoretic approach to dynamic range enhancement using multiple exposures,” Journal of Electronic Imaging, vol. 12, pp. 219, 2003.
[4] S.B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski, “High dynamic range video,” ACM Transactions on Graphics, vol. 22, no. 3, pp. 319–325, 2003.
[5] S. Mann and R.W. Picard, “Being ‘undigital’ with digital cameras: Extending dynamic range by combining differently exposed pictures,” in Proc. IS&T’s 48th annual conference, Washington, D.C., May 7–11 1995, pp. 422–428, Also appears, M.I.T. M.L. T.R. 323, 1994, http://wearcam.org/ist95.htm.
[6] Steve Mann, Intelligent Image Processing, John Wiley and Sons, November 2 2001, ISBN: 0-471-40637-6.
[7] C. Pal, R. Szeliski, M. Uyttendaele, and N. Jojic, “Probability models for high dynamic range imaging,” in Computer Vision and Pattern Recognition, 2004. CVPR 2004. Proceedings of the 2004 IEEE Computer Society Conference on. IEEE, 2004, vol. 2, pp. II–173.
[8] M. Granados, B. Ajdin, M. Wand, C. Theobalt, H.P. Seidel, and H.P.A. Lensch, “Optimal hdr reconstruction with linear digital cameras,” in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010, pp. 215–222.
[9] Steve Mann, Ryan Janzen, and Tom Hobson, “Multisensor broadband high dynamic range sensing,” in Proc. of the 2011 ACM International Conference on Tangible, Embedded and Embodied Interaction, TEI’11, 2011, pp. 21–24.
[10] Corey Manders, Chris Aimone, and Steve Mann, “Camera response function recovery from different illuminations of identical subject matter.,” in ICIP, 2004, pp. 2965–2968.
[11] M. A. Ali and S. Mann, “Comparametric image compositing: Computationally efficient high dynamic range imaging,” in To appear, Proc. Int. Conf. Acoust., Speech, and Signal Processing (ICASSP). March 2012, IEEE.