How ThoughtCam works

The Biosensors

In the "biosensor box", there are the myoscan, EEG, skin conductance, temperature and respiration sensors. (Note: we don't have the EKG nor the Heart Rate sensors.)

Click here to see pictures of how to connect the biosensors to your body and to the ProComp+ encoder.

(In addition, see "ProComp+ Biosensors" under Documents.)

Reading Data

ProComp+ developed by Thought Technology Ltd. has 8 channels for biosensor inputs. Channel A and B can read 256 samples per second. Channel C to H can read 32 samples per second. Therefore, when using brainwave sensors, either channel A and/or B should be used.

xcapliveprocomp continuously reads data from ProComp+ in iterations of 24 packets. Each packets contain 3 words (2 bytes per word). The first word contains data for Channel A. The second word contains data for Channel B. The third word contains one of the 24 possible items from a round-robin table. Out of the 24 items, each channel from C to H is represented 3 times and the battery channel is represented once.

(For more information on the contents of the packets, see "ProComp+ Communication Protocols" under Documents.)

After 24 packets are read, the data are parsed according to the channel offsets.The parsed data are then translated into the sample values of the corresponding channels. These values are stored into a 2-dimensional array.

These stored values are then:
1. Output to stdout as an ASCII graph.
2. Logged into the corresponding channel file along with a timestamp
3. Used to determine whether a picture should be taken.

In our current version of xcapliveprocomp, data can be read from all 8 channels as well as the battery level of the ProComp+ encoder. These data are logged into 9 files automatically. However, we only uses 3 channels - channel f, g, h - to connect to skin conductance, temperature and respiration respectively. We use these three sensors to capture Sympathetic Nervous System response of our body in order to take a picture during "fight" or "flight" scenarios.


Click here to see an ASCII plot.

Sensor data stored in the 2-D array are used to generate 4 ASCII graphs to stdout on the fly.

The first column of the ASCII plot corresponds to muscle sensor data from channel E. The second column corresponds to skin conductance sensor data from channel F. The third column corresponds to temperature sensor data. The last column corresponds to respiration sensor data.

Each graph has a predefined max and min value which is used to adjust the width and stepsize of the plot. The ASCII graphs are useful for real-time analyses which do not demand exact data values but need a rough idea of what is happening.

Log Files

The filename of the log files are in the format logX.txt where X corresponds to the channels. Channel A to H correspond to log0.txt to log7.txt respectively. log8.txt corresponds to battery power measurements which was used as a control to see whether the sensor data are reliable.

Each row corresponds to data taken during one iteration of reading data from ProComp+. The first column in the log file correspond to the frame number at which the samples are taken. The second column is the timestamp of the data logged into the file in seconds. (Use "" in the xcaplive directory to decode the time.) The remaining columns are the data samples. These data are useful for detailed analyses after the experiments. We used gnuplot to plot these data.

Using Sympathetic Nervous System Responses to trigger picture taking

The filename of the frames taken are in the format vXXXXXXX.ABC.jpg where XXXXXXX is the frame number and ABC indicates which sensor triggered the camera to take a picture. If A = 1, SC triggered the camera. If B = 1, Temperature triggered the camera. If C = 1, Respiration triggered the camera.

Skin conductance measures the sweat gland activity and pore size of the skin. When the subject is nervous or in shock, sweat gland activity and pore size of the skin increases due to the SNS. Increased sweat increases the ability for the skin to conduct electricity current. When SC exceeds the threshold, a picture we be taken.

The thermometer attached to the subject's thumb measures the skin temperature of the subject. When the subject is hyperventilating (e.g. due to shock), skin temperature decreases due to vasoconstriction triggered by the SNS in an attempt to bring blood away from surface tissues and pump more blood to the heart. Therefore, in this case, temperature has to drop below the threshold to trigger a picture to be taken.

The respiration elastic belt measures the amplitude of the subject's diaphragm or thoracic expansion depending on where the subject fasten the belt. (In our experiments, we fasten the belt to our diaphragm.) When the subject is breathing heavily, the expansion of the diaphragm increases. For example, when the subject is in shock, the SNS triggers the body to increase ventilation. In the case of heavy deep breathes, the respiration amplitude increases dramatically and triggers the camera to take pictures.

Note: In our initial observations, the thermometer and the skin conductance sensors must be fasten to the subject's fingers by tape to avoid any slack. Otherwise, results would be highly effected by movement of the fingers.

In addition, we observed that, there is a tendency for the sensor measurements to slowly drift up and down. To compensate this natural tendency and only try to capture the SNS response, we use a combination of moving averages and constant thresholds to determine when to take a picture as opposed to only using fixed threshold values. (We have also experimented with moving averages and an adapting threshold, but the results are similar.)

The algorithm we used is as follow:

Average_i = Average of X_i, X_i+1, X_i+2

Then, compare Average_i-1 + Threshold constant with Average_i:

For SC, takes picture if Average_i > Average_i-1 + 50

For Temperature, takes picture if Average_i < Average_i-1 - 5

For Respiration, takes picture if Average_i > Average_i-1 +35 Note: These three sensors will trigger a picture to be taken individually as well as in conjuction. All of these calculations are done in the funtion userPanic().

Taking a picture using the original Xcaplive functionalities

Once it is determined that a picture should be taken, our version of xcapliveprocomp "fools" original xcaplive functions into thinking that the character "g" has been input from the keyboard. (In the original xcaplive, user manually type in "g" to capture a single frame and "G" to capture frames continuously. Press "G" again to stop grapping frames, "q" to stop xcaplive or xcapliveprocomp from running.)