background image
DECONcert: Making Waves with Water, EEG,
and Music
Steve Mann
1
, James Fung
1
, and Ariel Garten
2
1
University of Toronto
Dept. of Electrical and Computer Engineering
Toronto, Ontario, Canada
2
Neuroconsulting
Toronto, Ontario, Canada
Abstract. We describe events in which music, water and the brain form
an immersive environment for human-computer and human-computer-
human collective engagement. The theme of sound wave production, re-
generation and audition from water waves and brain waves is our central
exploration, beginning with our DECONcerts in which participants, im-
mersed in water and connected to EEG equipment, regeneratively create
or affect live music by varying their alpha wave output. We explored the
five states-of-matter (Classical Elements) of solid ("Earth"), liquid ("Wa-
ter"), gas ("Air"), plasma ("Fire"), and quintessence ("Idea"), in the
context of immersive media (e.g. when the surrounding state-of-matter
was liquid). Some of these immersive environments spanned multiple
countries, by way of networked connectivity. We also expanded from
philosophical to therapeutic contexts by including Parkinson's patients
in our immersed environments.
1
Introduction
This paper presents a series of performances, art exhibits, and concerts, that ex-
plored the relationships between water waves, sound waves, and brainwaves (see
figure 1). These events merged a custom built EEG (Electroencephelograph or
"brainwave") computational system with music generation, immersive aquatic
spaces, and groups of immersively engaged performers and participatory audi-
ences. Participatory performances explored collective consciousness by creating
both physically shared spaces (connecting various groups of participants across
distant geographical boundaries) and shared human-computational networks.
The resulting collective immersive experiences were created using the media
of a shared immersive audio environment, and an aquatic environment where,
in some events, groups of participants and performers were actually immersed,
in whole, or in part, in water. These media are explorations of waves: in one
medium, acoustic waves; in another, aqueous waves (various performances stud-
ied caustics and wavefronts, as well as water-induced sounds); and, of course,
brainwaves.
R. Kronland-Martinet, S. Ystad, and K. Jensen (Eds.): CMMR 2007, LNCS 4969, pp. 487505, 2008.
c Springer-Verlag Berlin Heidelberg 2008
background image
488
S. Mann, J. Fung, and A. Garten
(a)
(b)
(c)
Fig. 1. DECONcert events allowed participants to explore issues of DECONtamina-
tion, music, water, and brainwave. (a) Participants being prepared with electrode paste
for EEG readings. (b,c) A separate spotlight on each participant responds to their in-
dividual level of visual arousal, thus turning participants into performers who are on
stage in the bath.
This theme of waves not only occurred at the observational (output) side of the
performance, but also at the input: both the ambient and acoustic environments
were generated by directly measuring and interpreting the brainwaves of the par-
ticipants. This paper discusses various performance art events together with the
philosophical implications and artistic narratives developed in the various events.
2
Creating Immersive Experiences with Humanistic
Intelligence
Humanistic intelligence (HI) is defined [7] as a signal processing framework in
which the processing apparatus is inextricably intertwined with the natural
background image
DECONcert: Making Waves with Water, EEG, and Music
489
capabilities of our human body and mind. Within the processing framework
of HI, the computational apparatus and user, in being intertwined as such, are
considered as a signal processing block, interacting as one with the outside world.
Within this block, the human and computer work together in a tight feedback
loop, with each accepting information and outputs from, and providing informa-
tion and inputs to, each other. In contrast to AI, which seeks to recreate human
intelligence on the machine, HI seeks to utilize the abilities of both the human
and machine to their fullest.
HI forms the structural framework in our explorations. In each of our ex-
ploratory performances, exhibits, and concerts, water, music and brainwaves, and
states-of-matter become the media to express Humanistic Intelligence. Though
at first inspection the HI framework may appear to express an individual's re-
lationship to computational apparatus, our events demonstrate the collective
nature of HI, where groups of participants are all connected to the computa-
tional system and thus to each other: a collective consciousness.
We use music and water and brainwaves as mediums to create group immer-
sion to allow participants to experience and become a collective consciousness.
We explore collective consciousness through: (1) collective consciousness (i.e.
using multiple participant brainwave inputs to drive an artistic process), (2) re-
presenting these brainwave signals in a shared multimedia environment where
audiovisual experiences, such as sound and visuals, are collectively experienced;
and (3) using water as a physical agent to bring participants into a shared space
that is truly, and literally immersive (in the sense of a communal bathing expe-
rience). These media explore issues of privacy and personal space.
3
The Events
Starting in July 2001
1
we had a series of events addressing issues of contam-
ination and biological warfare. The authors created a number of events, per-
formances, and concerts that dealt with issues surrounding decontamination.
These events were named DECONference, DECONversation, DECONsortium,
DECONtrol, DECONcert, and the like, making reference to DECONtamina-
tion. For example, a series of DECONcerts were presented as DECONtamina-
tion concerts in which participants were washed down with water prior to being
connected to EEG (brainwave) instrumentation.
Our DECONcert series explored a regenerative feedback loop between brain-
waves and music, as the collective consciousness of a large audience either gen-
erated music or modified music generated by other performers.
In our Powerplant
2
DEConcert, individuals contributed directly by playing
their brain as an instrument in an improvising live band, directing and taking
direction from more traditional musical interactions.
1
Our first event took place prior to the anthrax scare that came shortly after the
September 11th 2001 terrorist attacks.
2
The "Powerplant" is a Canadian contemporary art gallery.
background image
490
S. Mann, J. Fung, and A. Garten
In many of our DECONcerts, groups of people from around the world were
connected, over the Internet, from various different communal baths or aquatic
spaces. For example, in one DECONcert, we had groups of six bathers, at a time,
in one rooftop tub, each outfitted with EEG electrodes, connected to bathers,
three-at-a-time, in another distant tub that was located on the sidewalk of a
busy downtown street. Situating the bath on a busy sidewalk established a jux-
taposition of public and private, while inviting passers-by to stop, "doff their
duds", put on the EEG electrodes, and join in. The different group baths were
connected audiovisually, as well as electroencephalically (using EEG sensors),
across the World Wide Web, also by way of web cameras, microphones, and
various physiological signals such as EEG and ECG (Electrocardiogram).
In another concert, we invited a number of Parkinson's patients to participate
remotely from their hospital beds using equipment we sent out on loan. What was
remarkable about this form of participation, was the fact that the DECONcerts
were inclusive for people of any physical ability. In this form of cyborg space,
a person of lesser physical ability is still a full participant, since the primary
experimental control modality is brainwaves. All that is needed to be a full
participant is a sufficiently engageable brain.
An important artistic narrative was the juxtaposition of this corporeal tran-
scendence, combined with the physicality of passers-by in their disrobed and
electrified bodies, situated in a bath on a busy street.
One set of mini-concerts within the DECONcert series was called "Telematic
Tubs Against Terror". This was a series of events in which groups of individ-
uals were immersed in tubs of water and connected by way of EEG to form a
collective consciousness. This emerged as participants projected a sub-collective
from each tub (each "wash node"). These events explored a collective and dis-
tributed consciousness as people's brainwaves were made public, in concert with
the collectively shared experience of water and music.
Water and music also formed the backbone of our concert at the International
Computer Music Conference (ICMC). The creation of a novel instrument called
the hydraulophone, whose sound production is created using water as a medium,
invited the examination of other mediums in which sound could be produced.
Thus we created a physics-based orgonology in which musical instruments are
classified based on the state-of-matter (solid, liquid, gas, plasma, or quintessence)
of the intial sound-producing mechanism. We already created instruments from
non-matter, i.e. quintessence (bio)informatics, e.g. brainwaves.
3.1
DECONcert: Collaborative Music in the Key of EEG
DECONcerts were a form of audience-participatory concert in which the partic-
ipants' brainwaves determined the music they were experiencing.
The first DECONcert was, to the authors' knowledge, the first exploration of
music generated by collective consciousness (i.e. more than one person generating
music with their brainwaves together).
Our first collectively created concert, DECONcert 1, attracted enough interest
to require three separate sessions in the same evening, each for a different group
background image
DECONcert: Making Waves with Water, EEG, and Music
491
(a)
(b)
Fig. 2. DECONcert Performance: This group of 48 participants simultaneously and
collectively adjusted the musical environment with their brainwaves while remotely
connected to groups in other countries
of participants. For each session, we connected 48 people by way of their EEG
signals, which were collectively used to affect the audiovisual environment.
Using six 8-person EEG machines, donated by manufacturer Thought Tech-
nologies Limited, we were able to obtain connections from 48 people at the same
time.
In order to have the greatest flexibility we wrote our own GNU/Linux device
drivers for these machines, and we developed and implemented our own signal
processing algorithms. We developed a system to utilize multiple EEG signals
to clean the signal and look for collective alpha synchronization (which occurs,
for instance, when people close their eyes). Figure 2 shows images taken of the
first DECONcert performance.
DECONcert utilized electroencephalogram (EEG) sensors which sensed elec-
trical activity produced in the brains of the participants. The signals from the
brainwaves of the 48 participants were used as input to dynamically alter a
computationally controlled soundscape. DECONcert allowed the participants to
form a feedback loop with the computational process of musical composition.
The soundscape being generated was in response to the participants: the collec-
tive response from the group of participants is sensed by the computer, which
then alters the music based upon this response. Again, the participants hear the
music, and again respond, and again the computer senses and further alters the
background image
492
S. Mann, J. Fung, and A. Garten
sound. In this way, collaborative biofeedback is being used in conjunction with
an intelligent signal processing system to continually re-generate the music on
the fly.
A total of 3 DECONcerts were held with different configurations exploring
different methods of audience interaction. In each of the 3 DECONcerts that
were held, up to 48 audience members sat in front of the stage in 6 groups.
On stage, jazz musicians improvised on some combination of electric keyboard,
electric clarinet, trumpet, saxophone, drums, and/ or base. As audience members
listened to the concert, each member's brainstate determined the modulation of
the output of the musician's synthesized instruments. Some acoustic qualities
that the audience was able to modulate included pitch, volume, FM oscillation,
chorus, and distortion. Figure 3 shows images from DECONcerts 2 and 3.
The participant's raw EEG signal, and frequency distribution, were plotted
and projected onto a screen, so the participant could determine if he or she
was in an alpha brain state or a beta state. When all of the participants in a
single group reached alpha frequency (as determined by an averaging process),
the acoustic quality controlled by that group was modulated accordingly.
In this way, the participants' brainwaves collectively and continuously affect
music that was being heard. This process was both fluid and regenerative, in that
participants' brain states influenced the musical output, which in turn was re-
ceived (heard) by the participant's brain, which then influenced the participant's
brainstate which influenced the music output.
3.2
Powerplant Performance
In 2007, we undertook a new iteration of the brainwave musical interface system.
Rather than creating a feedback interaction between performer and audience, we
allowed participants to "jam along" with a renowned live improvising band in
a concert setting. We assigned each participant a note or chord. By reaching
a certain threshold of alpha activity (20% of total brainwave output measured
from occipital lobe) participants were able to increase the volume of that tone.
The increase was cumulative (temporally integrated) whereby the longer the
participant remained over threshold, the higher the volume went. This cumula-
tive (integrated) response mimics the response of aquatic instruments like the
hydraulophone, which responds to absement or presement (time time-integral
of displacement or reciprocal displacement) rather than to displacement or to
velocity (many other polyphonic instruments like the piano respond to velocity
rather than displacement or absement). This kind of response simulates the effect
of a water reservoir that fills up or empties out over time, giving the instrument's
user-interface an aquatic feel.
Once participants learned, through usage, how to gain control over the system,
they could play and mute, warp and vibrate their note, in time with the band.
Not only did the participants play along, but they also gave musical cues to the
other players. This is a direct example of a humanistic intelligence signaling block,
where human and machine work together directly affecting and responding to one
another's output as it becomes reified in the external physical world. See Fig 4.
background image
DECONcert: Making Waves with Water, EEG, and Music
493
(a)
(b)
Fig. 3. Regenerative Jazz Performance (a) DECONcert 2: Audience brainwaves mod-
ulate the sounds of a trio of performers. (b) DECONcet 3: A Jazz ensemble is affected
by audience brainwaves, with acoustic instruments modulated via sound filter.
3.3
Telematic Tubs Against Terror
Telematic Tubs Against Terror, also explored the creation of a collective and
communal consconsious, this time using the mediums of water and brainwaves,
rather than music and brainwaves. Figure 5 shows images taken from these
events. Two tubs of water were set up in different locations, one on a main street,
and one indoors 1.5 miles away. Eight EEG leads and several ECG leads were
suspended over the tubs.Two screens abutted the tubs. Each location received
the EEG and ECG information of the sister tub and projected it on one screen,
background image
494
S. Mann, J. Fung, and A. Garten
(a)
(b)
Fig. 4. Brainwave Performance at the Powerplant Art Gallery in Toronto: (a) Quintist
Ariel Garten performing; (b) A young audience member performs in the concert, after
a brief 5 minute training session
as well as receiving live video feed from the sister location projected onto the
second screen. Up to 8 (and sometimes more) participants at a time entered the
tub together, and connected themselves to the EEG and ECG leads. In this way
the participants were sharing not only physical space, but mental space as well.
3.4
Differentiating Brainstates to Create Control Interfaces
In DECONcert 1, we hooked up 48 people's EEG signals, which were collectively
used to affect the audio environment. Each audience member had a single EEG
lead held against the back of his or her head with comfortable headband, at
the location of the occipital lobe. As well, a wire was clipped to each ear for
grounding. The collective signals from groups of eight participants were cleaned,
and collective alpha synchronization (which occurs, for instance, when people
close their eyes) was detected.
The alpha-wave intensity increases when a person approaches a calm medi-
tative state of concentration and it is inversely proportional to the amount of
background image
DECONcert: Making Waves with Water, EEG, and Music
495
(a)
(b)
Fig. 5. Telematic Tubs Performance. (a) and (b) show two different "wash nodes"
where participants' EEGs and ECGs were read and shared between sites. A video link
connected each wash node with the others as well, and brainwave data was displayed
and re-presented remotely.
visual stimulant the person receives [3]. Experiments have shown that there ex-
ists a correlation between the mental activity of a person and their respective
EEG spectrum [2]. Lusted and Knapp explored brainwave interfaces [6]. An early
musical brainwave implementation was conducted by Lucier [4,5], also employ-
ing alpha waves as a sonic device. Rosenboom [8] worked with alpha waves for
music production. The music of this work extends the group dynamic of brain
wave music to 48 simultaneous participants, and explores both light and water
as additional mediums for immersive experiences.
Humans are generally described as being in one of 5 brain states, Alpha, a
calm creative state that is described by brainwave activity of 8-12 Hz, Delta,
slow brainwave less thank 2 Hz, is associated with deep sleep. Theta, a state
achieved by those in deep meditation or earlier stages of sleep, are classed as 4-8
Hz. Most individuals spend most of their day in beta waves, classified as any
wave activity over 12 Hz. [1] For our purposes, we tracked whether participants
were in Alpha state (8-12 Hz) or another state.
background image
496
S. Mann, J. Fung, and A. Garten
Signal Processing
and Control
Flexcomp Server
Computer
TCP/IP
Communication
Fibre Optic
Flexcomp
Encoder
8 EEG Channels
Flexcomp
Encoder
8 EEG Channels
Flexcomp Server
Computer
Fig. 6. EEG Multimedia Control System. The system is expandable to accommodate a
number of EEG channels, which may be connected to multiple participants. Addition-
ally, the TCP/IP connection allows the possibility of remote and wireless data analysis
and storage.
System Configuration. The basic configuration of the system is shown in
Figure
6. To digitize brainwave activity for analysis, a Thought Technology
FlexComp A/D encoder and ISA DSP2 Data Acquisition Card (DAC) are used.
These devices can provide up to a 2KHz brainwave sampling rate, and measure
brainwave activity down to a maximum of 5% error and 1V accuracy.
A set of custom programs were written to utilize the hardware for music
generation. Additionally a Linux device driver was written to interface with the
ISA data acquisition card. A server program communicates with the DAC, placed
on the ISA bus of a Linux system, and optically connected to the FlexComp
encoder hardware thereby making raw EEG data available over TCP/IP. A
client system connects to the server via TCP/IP and receives the EEG data,
upon which it performs the filtering and processing of brainwave data. Both
programs can run on a single Linux PC using the loopback address (127.0.0.1),
if the PC is sufficiently fast. Similarly the TCP/IP interface can be exploited to
allow communication between remote locations, as in the implementation of the
Telematic Tubs exhibit, and the remote Parkinson's patients event.
Additionally control of standard AC room lighting was achieved using a DMX-
512 dimmer system is used. DMX-512 is a simple packet-based digital protocol
for controlling stage lighting and other devices using an RS-485 serial interface
at 250kbaud connecting to a LanBox LCX DMX-512 Controller over a TCP/IP
socket.which sends a single text command to change room lighting levels. This
change is transmitted via the DMX protocol to a set of DMX dimmers, which
change the light intensities in the room as required. Figure 7 shows the EEG
controlled lighting environments used in the exhibits.
background image
DECONcert: Making Waves with Water, EEG, and Music
497
(a)
(b)
Fig. 7. Brainwave Controlled Immersive Lighting and System Sculpture. (a) The
DECONcert theatre. Six stations of EEG Electrodes hang from the ceiling. Gelled
lighting systems shining from above the skylights of the space use light to create a
collective immersive environment. The custom designed circular discs form EEG nodes
for eight participants, reflecting the "neuron" like design of the system where multiple
branches of input flow to a signal processing machine. (b) In turning the audience into
a participant, DECONcerts invert the relationship between audience and performer,
bringing the two together. A brainwave controlled spotlight shines on the participants
with the intensity controlled by their concentration state.
3.5
Screening Out Unusable Signals
EEG signals are typically orders of magnitude weaker than muscular signals.
Consequently, if the participant is moving their head, or their muscles are not
sufficiently relaxed, the EEG signal signal strength is weak in comparison to
the muscular electrical activity, which we consider noise. In this situation, we
cannot rely upon the analyzed EEG bands to produce a usable signal. In order
to detect these cases, we calculated the power of the received signal, and rejected
the signal above a certain threshold, which could be calibrated as the system
was used.
background image
498
S. Mann, J. Fung, and A. Garten
3.6
Real-Time Control of Live Musical Input
Because the system is used for real-time control, latency between the onset of
a desired EEG trend, and response of the system was found to be of interest.
We implemented several approaches to control. The first approach was to use
a variable counter which incremented so long as the participant's alpha wave-
band strength was above a threshold. The counter decremented when the alpha
waveband strength was below a threshold. Sound effects were triggered when
the counter was above a certain threshold. The advantage of this method was
that only sustained period of high alpha activity triggered a sound response from
the system, and made it quite robust with respect to a "false positive" alpha
strength detections. Additionally, this method allowed us to verify the efficacy of
our system at detecting alpha activity. However, the requirement for sustained
periods of alpha activity meant that the sound effects would only occur at typi-
cally longer than 1015 seconds after the onset of the alpha activity state. This
latency made it difficult for a casual participant to perceive their effect on the
sound. Similarly the window sized used for the frequency analysis is related to
the latency in the system. Longer windows allow for more reliable detection of
sustained mental state. However, this increased overall latency of the produced
control signal. Additionally, high, but short lived alpha activity is not well de-
tected in this case. Short windows allow for faster system response, but were
affected by noise.
3.7
EEG Based Music Composition
Our approach to EEG based musical composition was that of creating a general
programming framework, whose variables were continuously controlled by the
EEG signals of the participants.
There was a simple sequenced bassline (randomly choosing from 4 note pro-
gressions), a simple sequenced drum track (the complexity of the track altered
by the persons alpha), which utilized a sequencer. We used the counter method
described above. However, instead of only a single event above a preset thresh-
old, different ranges of the current value was used to determine the complexity
of the tracks.
For the bassline, several sets of notes' on and off toggles were under control of
the EEG. Thus, for higher activity, more notes were turned on and this made the
sound of the baseline appear more busy and complex. For the sequenced drum
track, the EEG was used to toggle different rhythm tracks on and off. When more
instruments were triggered on, the rhythm appeared more complex. Again, these
were turned on and off with respect to current alpha counter range. Additionally
pad and background sounds were randomly triggered by the amount of alpha
activity of the participant.
To maintain a musical consistency the tones (notes) were chosen from a prede-
termined scale (aeolian mode) so as minimize the disonnance which would occur
if completely random notes were used. For these effects, however, the primary
contribution of EEG control was to affect the filter frequencies of the tones,
background image
DECONcert: Making Waves with Water, EEG, and Music
499
which dramatically affected their quality. The control was achieved via sending
MIDI control signals to synthesizers.
Typically observed minimum and maximum alpha waveband strengths were
mapped to the range [0, 127] used by MIDI. This method represented a continu-
ous form of mapping alpha strength to control variables as no thresholding was
used. Additionally, however, we found that the most effective sound effects were
those which changed dramatically over their MIDI controlled range. In some
instances, we restricted the MIDI controller value into a range which produced
dramatic changes instead of the full [0, 127].
We found that this approach allowed the system to emulate the pseudo-
randomness of sweeps and pads which tend to occur, for instance, in electronic
music, and create those events under the control of the alpha. Similarly, the kick
drum or delay effects on a drum track are additionally triggered on and of under
the control of alpha waves.
Overall, we found that these approaches allowed for different parts to fade in
and out. For instance, the kick sometimes provided a beat, and would then fell
out, giving way to a more open segments and so on. The participants were able
to learn and control the system well by listening to the music feedback, over the
30 minutes they had to use it. At the end of the performance, the participants
understood their control of the music well.
3.8
Affecting Live Performance with EEG Signals
DECONcerts 2 & 3 used participant brainwaves to alter the sound qualities
of instruments being played by live musicians. Both acoustic and electronic in-
struments were affected. Electronic instrument sound qualities were affected by
varying MIDI parameters using the system of DECONcert 1. An electronic key-
board, electronic wind instrument and electronic drum pad were used.
To achieve variation on acoustic instruments, a digital mixer was used. For
acoustic instruments such as an amplified bass
3
, two audio channels were fed
into the mixer. One channel was the unaltered audio channel, and the second
was the same input run through a filter. The filter was either a pedal filter,
or a digital filter applied internally: a feature of the mixer equipment. Brain
wave signals drove the system to, via MIDI, crossfade between the unfiltered
and filtered channels. In this way the brainwaves altered the sound quality of
the acoustic instruments used at the event.
This created a challenging playing environment for the musician. The sound
quality of the musician's instrument changed in ways that were not under the
conscious, direct control of the musician. As the sound quality changed, the mu-
sician needed to adapt their playing style to match. For instance, the decay of
the note would change. With a short decay, the musician could perform quick,
stacatto phrases, while longer decays, phrases incorporating sustained notes were
more appropriate. In this way, participants affected the overall qualities of the mu-
sic despite the fact that the musicians were playing the instruments themselves.
3
For the moment, we consider the analog nature of the amplification of the vibration
of the strings of an electric bass guitar as "acoustic".
background image
500
S. Mann, J. Fung, and A. Garten
4
Aquatic Context: Waves in Water and Mind
The connection between brainwaves, water waves, and sound waves (in both air
and in water) was made all the more apparent in a recent performance at the
International Computer Music Conference (ICMC 2007) in Copenhagen. The
theme of the 2007 conference was "Immersed Music".
In keeping with this theme, we developed various forms of "Immersive Me-
dia" for public performance in Copenhagen's Vandkulturhuset. The name "Vand
kultur huset" means "water culture house" in Danish.
4.1
The States-of-Matter Quintet
This Immsersed Music concert at ICMC 2007 consisted of a performance by the
States-of-Matter Quintet, involving musical instruments we created that pro-
duced sound from each of the five states-of-matter:
Solid ("Earth");
Liquid ("Water");
Gas ("Air");
Plasma ("Fire");
Quintessence ("Idea").
These correspond to the Greek Classical Elements, the fifth element being Idea
(non-matter).
See Fig 8.
4.2
Surrounding Medium
Vandkulturhuset (Fig. 9),
The existence of immersive media raises the question of media itself. Thus
we may ask:
in what medium is the sound initially produced;
what is the surrounding medium;
in what medium is the listener immersed?
For example, the fact that a listener may be immersed in air, or in water, suggests
also that the sound need not be produced in the same medium in which it is
experienced.
5
Philosophical Implications/ Discussion
5.1
Human/Computer Feedback Interaction (HI)
Both Telematic Tubs Against Terror and DECONcert create a Humanistic In-
telligence feed-back loop using the elements of music, water and brain. As dis-
cussed, Humanistic Intelligence is defined as intelligence that arises from the
background image
DECONcert: Making Waves with Water, EEG, and Music
501
human being in the feedback loop of a computational process in which the hu-
man and computer are inextricably intertwined, in otherwords, it is a regen-
erative feedback loop. In DECONcert, regenerative music is the expression of
HI in music. In regenerative music the computer, instead of taking only active
cues from the musician, reads physiological signals from the musician/performer.
The music which the regenerative algorithm then creates will be heard by the
musician/performer. It is hoped that the music will in turn generate an emo-
tional response on the part of the musician/performer, and that that emotional
response will be detectable by the computer, which can then alter the music in
some way in response. Continuing in this fashion, it is clear that there is a well
defined feedback loop occurring between the human and the computer.
5.2
Regenerative Music
Jazz, the musical genre of the DECONcert performances, is a natural non-
computation example of regenerative music. Jazz is a free flowing style of musi-
cal improvisation in which the performers intuitively read one another's states,
mood and musical intention based on both the performer's sonic output as well
as conscious and subconscious communication between players. The players in
a sense create an immersive, responsive environment. Immersed in the music
that surrounds them, in the musical and collective `zone', they respond to one
another's output. The response of the audience, also encourages or discourages
the musicians' particular output. DECONcert takes this intuitive process and
turns it inside out.
Regenerative Music looks not only at the audience-musician interaction, but
also the musician-instrument response. It brings in the problem of how a musician
can learn to respond to this new physiologically driven instrument, as well as
how the instrument can learn to infer the wishes of the musician through their
physiological signals, in addition to the normal playing of the instrument. In a
sense, the musician and instrument each play off of each other, and together, both
can be viewed as an "instrument". The choice of how to map from physiological
signals into instrument behavior would be an artistic one, under the control of
the musician.
5.3
Collective Unconsciousness
Creating regenerative music then becomes a distributed process, where no one
individual has conscious control over the sound. In a sense, all individuals enter a
collective state in which no single individual is aware of or has conscious control
over the outcome. Collectively and communally, the audience determines what
will be heard musically, using the interface of their brains. The audience has
no determination over what the final outcome of the music will be, nor do the
musicians. Thus, with brains physically connected to one another though EEG
leads, the audience enters both physically and metaphorically into a collective
unconsciousness.
In Telematic Tubs Against Terror, collective unconsciousness was explored not
through music as its medium of expression, but through water. Sitting together
background image
502
S. Mann, J. Fung, and A. Garten
Solid ("Earth")
Instruments such as the guitar (chordophone) or cym-
bal (idiophone), pictured at left, produce sound by
matter in its solid state. Most idiophones such as the
cymbal, or Franklin's glass harmonica, will operate im-
mersed in air or in water.
Liquid ("Water")
A new category of instruments called hydraulophones
produce sound by matter in its liquid state. These in-
struments work well immersed in air or in water.
Gas ("Air")
Instruments such as the flute, work only in air. To
get them to sound underwater requires a surround-
ing of air around the fipple mechanism and at least
some air in the resonant cavity. The organflute ("flor-
gan"), a newly invented musical instrument (invented,
designed, built, and played by by S. Mann), combines
the user-interface of the flute (played by blocking fin-
ger holes) with the sound of the pipe organ.
Plasma ("Fire")
The plasmaphones, another newly invented category
of musical instruments, produce sound by matter in
its fourth state, namely plasma.
Informatics ("Idea")
Plato and Aristotle postulated a fifth state-of-matter,
which they called "Idea" or "Quintessence" (from
"quint" which means "fifth"). This covers thoughts,
mathematics, algorithms, and the like. A direct brain-
machine interface was used in the performance by the
States-of-Matter Quintet to represent the fifth state-
of-matter.
Fig. 8. Five States of Matter in Immersive Music. These five states-of-matter cor-
respond to the five Classical Greek Elements. Earth, Water, Air, Fire, and Idea
("Quintessence" meaning "fifth" element). Immersed Music concerts explored the im-
mersion of these five elements in both water and air. Immersed Music was the theme of
ICMC 2007, which included concerts at the DGI-byen "Vandkulturhuset" swim center.
background image
DECONcert: Making Waves with Water, EEG, and Music
503
(a)
(b)
Fig. 9. "Vandkulturhuset" is Danish for "Water Culture House". This cultural venue
in Copenhagen was the site of our "Immersed Music" concert.
in a pool of water collecting and sharing brainwave information, participants
were brought into the same collective DECONsciousness, as DECONcert. In
also known as the DGI Swim Centre, consists of six pools in the main DGI-byen
aquatics area, which provide a variety of different kinds of bathing experiences at
a variety of different temperatures. One of these pools is a 100 metre (320 foot)
round pool called "the ocean". The ocean pool allows for "endless" (no need to
stop and change direction) swims. This round pool also has a hydraulic stage
in the center for concerts or banquets, in which the round pool forms a moat
around the musicians or diners. Our performance actually brought a variety of
musical instruments right into the pool itself, creating truly immersive music!
See Fig. 10.
5.4
Sousveillance
Surveillance pervades our society, ostensibly to mitigate danger. As a collective
of individuals whose information is being shared amongst one another, we are
engaging in a "sousveillance" [9] of sorts. Whereas traditional surveillance is a
top-down affair, where some hierarchically superior "Big Brother" is watching
the movements of the general populace, in Telematic Tubs Against Terror and
DECONcert, the distributed sharing of private information creates a sousveil-
lance, a situation in which the subjects themselves are recording and sharing
their own information with one another. In an interesting twist, this sousveil-
lance creates a feedback loop between the audience and the recording device,
particularly the complex feedback loop in Telematic Tubs Against Terror be-
tween the video cameras and 2 separate wash nodes. In being recorded, partic-
ularly while in the vulnerable situation of bathing, one's behavior changes. In
Telematic Tubs, one realizes one is being recorded, and ones actions are affected
as such. Those different reactions are broadcast to the sister tub, who in seeing
the actions of their fellow bathers at a separate location, act differently. These
background image
504
S. Mann, J. Fung, and A. Garten
Fig. 10. Instruments that make sound from solid matter (left), liquid matter (middle)
and gaseous matter (right). Here instruments making sound from all three states-of-
matter (solid, liquid, and gas) are immersed in a surrounding medium of liquid. Only
the hydraulophone is designed to work properly when immersed in liquid.
reactions of the participants in the second tub are then recorded on video pro-
jected on a screen by the first tub, to further effect the behavior of the partici-
pants there, who cannot directly view on a screen their own behavior, only that
of the second tub. Thus a self-conscious distributed self-surveillance feedback
loop is created.
6
Conclusion
DECONcert and Telematic Tubs Against Terror speak of the interconnected
relationships between water, music and the brain. Using an intelligent signal
processing system to control musical output we created a collective consciousness
by highlighting both the humanistic intelligence human-computational feedback
loop as well as a physical shared immersive environment. In creating what we
refer to as a collective DECONsciousness, issues of privacy, contamination, and
control become significant. Participants become performers who are on stage in
the bath, self powering a distributed immersive experience with the pooling of
their brainwaves.
Acknowledgements
The authors would like to acknowledge the contributions from Corey Manders,
Chris Aimone, and Ryan Janzen. We also thank Thought Technologies Limited,
background image
DECONcert: Making Waves with Water, EEG, and Music
505
www.thoughttechnology.com, for donation of equipment that made much of this
work possible.
References
1. Brankack, J., Stewart, S.F.M.: Current source density analysis of the hippocampal
theta rhythm: Associated sustained potentials and candidate synaptic generators.
Brain Research 615(2), 310327 (1993)
2. Cromwell, L., Weibell, F.J., Pfeiffer, E.A., Usselman, L.B.: Biomedical Instrumen-
tation and Measurements. Prentice Hall, Inc., Englewood Cliffs (1973)
3. Legewie, H., Simonova, O., Crutzfeldt, O.D.: EEG Changes During Performance
of Various Tasks Under Open- and Closed-eyed Conditions. Elsevier Publishing
Company, Amsterdam (1969)
4. Lucier, A.: Music for Solo Performer (1965), for enormously amplified brain waves
and percussion. Lovely Music, Ltd., (1982)
5. Lucier, A.: Music for alpha waves, assorted percussion, and automated coded relays,
on "imaginary landscapes". Elektra/Nonesuch 79235-2 (1989)
6. Lusted, H.S., Knapp, R.B.: Controlling computers with neural signals. Scientific
American Magazine, 8288 (October 1996)
7. Mann, S.: Humanistic intelligence/humanistic computing: `wearcomp' as a new
framework for intelligent signal processing. Proceedings of the IEEE 86(11), 2123
2151 (1998), http://wearcam.org/procieee.htm
8. Rosenboom, D.: Method of producing sounds or light flashes with alpha brain waves
for artistic purposes. Leonardo 5(1) (1972)
9. Steve Mann, M.F., Fung, J., Baccanico, G.: Panopdecon: deconstructing, decon-
taminating, and decontextualizing panopticism in the postcyborg era. Surveillance
& Society 1(3), 375398 (2003)

Document Outline