James Fung, Ph.D.

Google ATAP (Advanced Technology and Projects)
Software, Project Tango
Email: fungja(at)google(dot)com
Alumni: EyeTap Personal Imaging Lab, University of Toronto

Jump to: [ Projects ] [ Academics ] [ Misc ]

GPU Accelerated Computer Vision

OpenCL Programming Guide Released (Jul 2011)

GTC Computer Vision on the GPU
GPU Technology Conference talk, 2009
Applying the Latest GPU Features to Computer Vision
GPU Technology Conference 2010 talk



Free Source library of Graphics Processing Unit (GPU) Hardware Accelerated Computer Vision available from http://openvidia.sourceforge.net and (faster access mirror site) at http://www.openvidia.org.
More information about OpenVIDIA GPU Computer Vision is in the book GPU Gems 2 in the chapter entitled ``Computer Vision on the GPU''.

Personal Imaging

Live Glogger Image Live Glogger Image Live Glogger Image Live Glogger Image Live Glogger Image Live Glogger Image Live Glogger Image Live Glogger Image

Check out Glogger

Computer Graphics for Computer Vision Research
Regenerative Brainwave Music Project


Regenerative Brainwave Music : ElectroBrainFunk


Deconism Gallery : DECONcert: Regenerative Music in the Key of EEG
On March 22nd, we held a brainwave music concert, where a computer sensed audience reaction to regenerate and alter to music on the fly, reacting to their responses to the music.

Post-Ironic Renaissance opening Cyborglog (GLOG) [link to the postironic page]
Digifest 2003 Glog
Rome Cyborglog (GLOG) (lucky you, you can watch vacation glogs)

Video Orbits Head Tracker (VOHT): Real Time Head Tracking: Images taken using the tracker with an EyeTap. Set your MPEG player to play this video at 11fps (the speed at which it was run live). Note that the real--world crosshair marker in the scene is just to judge the accuracy vs. the virtual--world crosshairs - no such markers are required to run the Video Orbits Head Tracker (VOHT).

-=Online Gallery of images=-

-=Transmitted EyeTap Image Archive=-

A signed HPLA (humanistic property license agreement), signed by the head of Information Access at the University of Toronto: This is something you can do regarding the Smart Card fiasco at UofT where the mandatory picture you had to take to get a student card was taken and sold to a company in Flordia.
This is what my tcard looks like now. Drilling out the smartchip provides a handy hole for a keyring.

Live Wireless EyeTap Images

Algebraic Projective Geometry (Video Orbits) for frame rate interpolation.
Video Orbits Scene: Here's a scene taken with and eyetap and processed with Video Orbits to create sort of an environment map. Fun Fun. (Its in Amsterdam Schipol airport)(won't play under windoze media player (this mpeg is opensource encoded)).

ISMR2001: Yokohama Japan - I presented a paper at ISMR2001, and had some extra time to tour Tokyo - here's some cyberspace-memories.

Wearcomp images taken in Italy. I attended SSGRR 2000 conference in L'Aquila, Italy and also had some time to tour Rome with my wearable and take some images.

Deliberately Diminished Reality: i.e.: spam2content

Academic Activities




[ click to download BibTex entries ]

Books/Chapters

A. Munshi, B. Gaster, T. Mattson, J. Fung, D. Ginsburg, “OpenCL Programming Guide”, AddisonWesley/Pearson Education, 2011

Area editor for ``Computer Vision'' section of ``GPU Computing Gems, Emerald Edition'', Morgan Kaufmann, 2011

James Fung. "Computer Vision on the GPU", in GPU Gems 2,Addison Wesley, 2005, chapter 40, pages 649-665, edited by Matt Pharr [ PDF (early edit) ] [ HTML (early edit) ] [ link ]

Refereed Publications

Music & Technology
Steve Mann, James Fung, and Ariel Garten, "DECONcert: Making Waves with Water, EEG, and Music", R. Kronland-Martinet, S. Ystad, and K. Jensen (Eds.): CMMR 2007, LNCS 4969, pp. 487-505. Springer-Verlag Berlin Heidelberg 2008 [ HTML ]

Steve Mann, Ryan Janzen, Raymond Lo, James Fung, "Non-Electrophonic Cyborg Instruments...", Proceedings of the 2007 Association of Computing Machinery (ACM) Multimedia (ACM MM 2007) Conference, September 24-29, Augsburg, Germany.

Steve Mann, James Fung, and Ariel Garten, "DECONcert: Bathing in the light, sound, and waters of the musical brainbaths", Proceedings of the 2007 International Computer Music Conference (ICMC2007), vol. 2, p. 204-211, August 27-31, Copenhagen, Denmark,

Mobile Multimedia
Steve Mann, James Fung and Raymond Lo, "Cyborglogging with Camera Phones: Steps Toward Equiveillance", Proceedings of the ACM Multimedia 2006 , Santa Barbara, California, Oct. 23--27, 2006 [ PDF ][ Presentation PDF ]

Computer Vision/GPU Programming
James Fung, Steve Mann, "Using Graphics Devices in Reverse: GPU-based Image Processing and Computer Vision", Proceedings of the IEEE Int'l Conf. on Multimedia & Expo, Hannover, Germany, June 23-26, 2008 [PDF]

James Fung, Steve Mann, Chris Aimone, "OpenVIDIA: Parallel GPU Computer Vision", Proceedings of the ACM Multimedia 2005 , Singapore, Nov. 6-11, 2005, pages 849-852 [ Adoble PDF ] [ HTML ] [ Web ]

James Fung, Steve Mann, "Projective Demosaicing Using Multiple Overlapping Images", Proceedings of the 2004 International Symposium on Intelligent Multimedia, Video and Speech Processing (ISIMP 2004), Hong Kong, October 22--24, 2004, pages 190-193 [ PDF ]

Rosco Hill, James Fung, Steve Mann, "Reality Window Manager: A User Interface For Mediated Reality", Proceedings of the 2004 IEEE International Conference on Image Processing (ICIP2004), Singapore, October 24-27, 2004. [ Adoble PDF ] [ HTML ]

James Fung, Steve Mann, "Using Multiple Graphics Cards as a General Purpose Parallel Computer : Applications to Computer Vision", Proceedings of the 17th International Conference on Pattern Recognition (ICPR2004) , Cambridge, United Kingdom, August 23-26, 2004, volume 1, pages 805-808. [ Adobe PDF ]

James Fung, Steve Mann, "Computer Vision Signal Processing on Graphics Processing Units", Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2004), Montreal, Quebec, Canada, May 17--21, 2004.

James Fung, Felix Tang, Steve Mann, "Mediated Reality Using Computer Graphics Hardware for Computer Vision", Proceedings of the International Symposium on Wearable Computing 2002 (ISWC2002), Seattle, Washington, USA, Oct 7-10, 2002, pp. 83--89.

James Fung, Steve Mann, "Exploring Humanistic Intelligence Through Physiologically Mediated Reality", Proceedings of the International Symposium on Mixed and Augmented Reality 2002 (ISMAR2002), September 30-October 1, 2002, Darmstadt, Germany, Page(s): 275- 276

Felix Tang, Chris Aimone, James Fung, Andrej Marjan, Steve Mann, "Seeing Eye to Eye: a shared mediated reality using EyeTap devices and the VideoOrbits Gyroscopic Head Tracker", Proceedings of the International Symposium on Mixed and Augmented Reality (ISMAR2002),September 30-October 1, 2002, Darmstadt, Germany, page(s): 267- 268

Steve Mann, Corey Manders, James Fung, "Painting with Looks: Photographic images from video using quantimetric processing", ACM Multimedia 2002

S. Mann and J. Fung, "EyeTap devices for augmented, deliberately diminished, or otherwise altered visual perception of rigid planar patches of real world scenes", PRESENCE, 2002, vol. 11, no. 2, pages 158-175, MIT Press.

Steve Mann, James Fung, "VideoOrbits on Eye Tap Devices for Deliberately Diminished Reality or Altering the Visual Perception of Rigid Planar Patches of a Real World Scene," Proceedings of the Second International Symposium on Mixed Reality, pp 48-55, March 14-15, 2001

Steve Mann, James Fung, and Eric Moncrieff, "EyeTap Technology for Wireless Electronic News Gathering," Mobile Computing and Communications Review, Vol. 3, No. 4, pp 19-26, October 1999

Steve Mann, James Fung, Corey Manders Living as Cyborgs: Trapped in a subjugatory Computer-Mediated Reality that extends to all hours of our day-to-day lives To appear in the proceedings of an arts conference: CAST01: Living in Mixed Realities conference, September 21-22, 2001 Schloss Birlinghoven, Sankt Augustin (Bonn, Germany): An ironic article about the dangers of corporatization of wearable technology, and also about empowerment through self-demotion/self-beauracratization

Hacking my a1200 phone. Links

motorolafans.com
openezx
opensource.motorola.com
mkezx
Manual exposure
dump yuv422 from the camera

Hacking D70

 # set the camera to store image from RAM memory onto the memory card
 gphoto2 --set-config /main/settings/capturetarget=Memory\ card

 # delete image 1 in the directory given by -f 
 gphoto2 -f /store_00010001/DCIM/100NCD70 -d 1

 # delete all images in the directory given by -f 
 gphoto2 -f /store_00010001/DCIM/100NCD70 -D  

 # to transfer to PC, use similar function using -P and -p to delete

"When you stumble, it usually makes you move forward faster."

Proverb


"Your options are only ever limited by your imagination."

maali-ism


"What is wrong is that we have invented the technology to eliminate scarcity, but we are deliberately throwing it away to benefit those who profit from scarcity."

John Gilmore, Electronic Frontier Foundation

Link to full article

"... the way constitutional rights are lost... Not in the thunder of a tyrant's edict, but in the soft judicial whispers of deference."

4th District Court of Appeal Justice David Sills (California)
[link]

Favorite Books



Just a picture of a minimalist desktop setup I played around with, using aterms tinted "Slategrey", and FVWM2, setting handles/borderwidths to 1.
That was my desktop. So here's the top of my desk. Messy you say? Ha. Should've seen it before. This panorama was created with the graphics hardware doing all the computations.


its cold. brr. gerstein is falling over.
and what's with the bizarre bales of hay anyways


These panorama generated entirely by a graphics card, from about 300 images from a webcam.
Nowadays, anyone can make easy panoramas using the online tool Glogger.

My interests include using Graphics Hardware for parallel general purpose computing, including implementing Computer Vision on the GPU. This work won the ACM Multimedia 2005 Open Source Software Award and resulted in the book chapter "Computer Vision on the GPU" in the popular GPU Gems 2 series of graphics programming books. This work has achieved implementation of vision algorithms on the GPU, including projective image stitching, Chirplet detection, Radon Transforms and natural feature processing and matching. I have been an author on 12 peer reviewed papers in IEEE and ACM conferences in the areas of parallel GPU Computer Vision and Mediated Reality. Merging technology with artistic endeavours, I helped create a system for the first ``Deconcert'' where 100 participants' brainwaves analyzed in real-time to drive music and sound synthesis. This work was featured on the Discovery Channel and Canadian national radio (Canadian Broadcasting Corporation, CBC). I am a student researcher in the SSHRC (Social Sciences and Humanities Research Council of Canada) Anonequity Project examining the importance of anonymity and authentication in a networked society. Currently I am involved in helping build and architect the Glogger system for continuous live upload from camera phones to a Glogger/Sousveillance community. I have acted as co-organizer for DASIP 2011 special session on GPU processing, and guest editor for the EURASIP Journal on Image and Video Processing special issue on Many-Core and Multi-Core Real Time Image Processing. Additionally I have served on the technical committee for GPUCV and CVPR sessions on the GPU processing. I am co-author of the book "OpenCL Programming Guide" published by Addison-Wesley, 2011.

Short Bio, up to 2011

James Fung has been applying GPUs to accelerate general purpose parallel computing since their graphics-only days, with a focus on image processing and computer vision. He received his Ph.D. in Electrical and Computer Engineering from the University of Toronto. He is at NVIDIA in the Developer Technology group. His work has been featured on Canadian national radio (CBC) and recently he has served as a guest editor for the EURASIP Journal on Image and Video Processing and session organizer at DASIP (2011). He is co-author of the "OpenCL Programming Guide" published by Addison-Wesley, 2011.

Best Viewed by Lynx, GLynx, Netscape, IE, or any other browser - 'cause this is pretty accessible HTML!
This page generated with the power of vi