vjeMTv · Audio/Video & Interactivity · Web · VJ · Multimedia Tech·Activ·Art·ist ·

Real-Time IR Tracking + Body Projection Mapping

Technical research for a new theater piece with Les Arpenteurs De l’Invisible, using real-time infrared tracking and projection mapping on the actors on stage.

(Thanks to the input of Mark Coniglio (cie TroikaRanch) and the Troikatronix (Isadora) community and this particular tutorial, as well as Toby K and the vvvv users who set me on track. Antoine M* also pointed the conference by Philippe Chaurand (Anomes) et Nicolas Bertrand “Retour d’expérience mêler corps et vidéo sur scène” during the (french) Symposium Captation Mouvement, held June 2018. Very interesting !)

Test setup : ASUS ROG G702VS Laptop – i7-7700HQ (2.8GHz) – 16 Go RAM – NVIDIA GeForce GTX 1070 – Win10 64 bit
Production setup : desktop computer – Intel Core i7-9700K (3.6 GHz / 4.9 GHz)+ 480 GB SSD + MSI GeForce GTX1060 Gaming X 3G


  • Mainstream Infrared Cameras
    • Kinect / Kinect 2 is kinda out of question because of the usb latency and shortness of connection cables,
    • Sony camera with nightshot feature – comp. out (additional info display on screen if not recording with Sony TRV35E model and standby mode entering for Sony TR511E lmodel)
    • Panasonic BP120 B&W surveillance camera – SDI out (720×576 PAL, shutter speed 1/50 s – TV LENS F1.2 4mm) in use for now.

  • Industrial Cameras
    • – USB3 or GigE camera with CMOS sensor (see differences with CCD) (lower latency / higher refresh rate : 60-100fps for 500€ worth cameras, with C-Mount so lenses are not so expensive)
  • + IR-Pass Filter Lee 87 (also mentioned : Lee 299 ?) on the lens or on the sensor of the camera (not in use on  screenshot below) – The IR pass filter stops visible light and let’s IR pass so the image is only made from IR light.
    Recyclers Tip !! Make diy IR Pass Filter from film contained in old 3,5″ / 5″ floppy disks and undeveloped film rolls ! Definitely worth trying, and i’ll keep you posted on that.
  • Video Capture Card

    • Blackmagic Design Intensity Shuttle –  Thunderbolt (only compatable w/ relatively new Mac – see here – but tests with a MacBook Pro Retina 15″ early 2013, supposedly ok, reveal that signal inconstancy makes capture impossible) – didn’t get my hands on a USB 3.0 one, but since it would only enable 1 videofeed at a time, it won’t suit our needs in the end (we’ll have at least an additional -non IR- live feed from the stage)
    • – Pinnacle Dazzle – USB 2.0 – in use for now
    • – PCI Express Blackmagic Design DeckLink Duo 2 (prod setup : 2 in + 2 out or 4 in or 4 out 3g SDI signals)
  • Infrared Lighting
    • – 96 IR Leds 12V Night Vision spots for CCTV camera (tests conducted with 4 units, setup as footlights upfront the actors)
    • – G/B/R/R gel swandwich on trad. lights : (g on lens side) : Green 90 +Blue 715 (Lee) or 83 (Roscolux) +2x Red 19 (no more than 50% intensity for a 3h max session – fire hazard !) Low side lights – parcan / pc – with barn doors (to cut IR from the ground)
      TIP : add-on a  Heat Shield Lee 269 transparent technical filter (to make the lifetime of the gels last longer. to be inserted between the lens and the coloured filters, leaving a fair air gap between the shield and them. the bigger the better apparently.)
    • prepare 2 distinct lighting schemes : 1 for IR and 1 for the visible light.
test IR outdoors 10m distance BP120 BW camera
Left :IR capture example with CCTV PB120 cam and direct lighting of the subject with 2 IR 12V LED units for a direct difference or threshold tracking, where whitest pixels are used as mask.Right :“infrared curtain” setup with trad. lighting, as shown in Mark Coniglio’s infrared tracking tutorial, for an inverted difference tracking with (black) shadow figures cut out on homogenous infrared (white) light wall behind them.


  • IR Tracking :
    Different possible methods

    • – Frame difference (threshold tracking)
    • – Contour-based detection (to be exprimented)
    • – Background subtraction (difference tracking of frozen image of the empty staged bathed in IR substracted from real-time video feed)

Tests conducted with isadora (win/mac) until now (very promising !!).
Other softwares like vvvv (win), processing, eyecon (win), eyesweb (win), TSPS (win/ios), millumin (ios), are also on the study, as well as a combination of some.

IR TRacking + body projection mapping tests by eMTv with Les Arpenteurs de L'Invisible, Nov 2020.

  • Camera / Projector Calibration

    • the high-end of the game (“from the litterature” i’ve been able to gather up to now) seems to be the modular system Rulr, by Elliot Woods (Kimchi and Chips) and Kyle McDonald, under openFrameworks.
    • – it seems though possible to setup for a “classic” corners + grid mapping of the captured stage, with long focal (telephoto) lenses on the projector and camera : zoom telephoto lens for the projector, and cmount 135mm / f2.8 focal lens for the cam (less barrel distortion to correct + easy setup of the projector + camera at the control booth)

Other questions :

  • What paint or coating will cut IR light best – in order to make set accessories disappear from the IR scene?
    As far as textile goes, some hints and tips from the army of course.. but what about these plastic coatings, a little more affordable I guess ? even though they’re still industry-orientated… Or will a simple mat paint be efficient ? Also, i’d like to try out carpets to avoid floor reflection.
  • What materials or textures spontaneously absorb IR rays ?
    (A friend suggested to give it a try by pointing a TV remote to the material and check through the smartphone camera if the surface is lit up or not. Gotta try that.)
  • Is is necessary to add an IR-Pass filter on a CCTV or GigE camera ? Yes, definitely if the camera is a color one – in any case, check before choosing camera that the image sensor if sensitive enough in the IR wavelengths between 780 nm and 1 mm.
  • Is it useful to add up several layers of IR-Pass filter instead of one ? (Tests to be conducted..)

Leave a Reply

Your email address will not be published.

  • Français
  • English