Mobile Robotics LaboratoryInstitute of Systems and Robotics

ISR – Coimbra

Bayesian Cognitive Models for3D Structure and Motion Multimodal Perception

Multimodal Perception Systems 01/01/2006 - 31/12/2009

Fundação para a Ciência e a Tecnologia; PhD Scholarship – SRFH/BD/24628/2005

Contacts:João Filipe Ferreira, Jorge Dias {jfilipe,jorge}@isr.uc.pt

• Goals– To research generic Bayesian models to deal with fusion, multimodality, conflicts, and ambiguities in perception and apply them in artificial cognitive systems.

– To answer questions such as:

• Where are the limits on optimal sensory integration behaviour?

• What are the temporal aspects of sensory integration?

• How do top-down influences such as learning, memory and attention affect sensory integration?

• How do we solve the “correspondence problem” for sensory integration? How to answer the combination versus integration debate?

• How to answer the switching versus weighing controversy?

• What are the limits of crossmodal plasticity?

• Motivations

– A moving observer is presented with a non-static 3D scene – how does this observer perceive:• his own motion (egomotion);• the 3D structure of all objects in the scene;• the 3D trajectory and velocity of moving objects (independent motion)?

• Challenges– Perceptual uncertainties:

– Perceptual ambiguities:

Biological Perception,Bayesian Model

Artificial Perception,Bayesian Model

Artificial Observer

SensorReadings

Artificial Perception

Human/BiologicalObserver

Perception Psychophysical Study Model Analysis

Artificial & Biological

Model Output Comparison

EgomotionIllusions, Conflicts & Ambiguities

Model Re-

evaluation

Model Re-

evaluation

Model Re-

evaluation

Model Re-

evaluation

Model

SynthesisM

odel

SynthesisM

odel

SynthesisM

odel

Synthesis

Sensation

3D Scene

& Moving Objectsw/ Static Objects

3D Scene

& Moving Objectsw/ Static Objects

• Expected Outputs

– Development of novel perceptual computational models:

1. based on vision, audition and vestibular sensing;

2. which mimic biological multimodal perceptual fusion processes;

3. which perform perceptual fusion within a Bayesian framework.

Ideal Observer Bayesian Framework

Perceptual Module

Prior Knowledge

Sensory Processing

Posterior

Gain/ Loss Function

Bayes’ RuleDecision Rule

Response

Keypress

Audio

Vinput Voutput

Egomotion

Haptics

Goal

EEG

fMRI

…

Video

Timeline (s)

Psychophysics

Physiology

Stimulus Onset

Stimulus Offset

KeypressTrigger

Perception Action/Response

ChoiceSensation

Triggers

SynchTriggers

Stimulus

Vestibular/InertialSystem

Z

Y

X

Vestibular/InertialSystem

Z

Y

X

Visual SystemVisual SystemBayesian Framework

for MultimodalPromotion → Integration

Moving Objects/Background

Segmentation

3D Occupancy+

3D MotionMap

Ancillary Informationfor

Promotion3D Occupancy

+3D Motion

Map

3D Sound-Source+

3D MotionMap

Egomotion estimate

Auditory SystemAuditory System

3D Scene

& Moving Objectsw/ Static Objects

3D Scene

& Moving Objectsw/ Static Objects

Artificial systems: sensor accuracy and precision discretisation (analogue-to-digital) noise not accounted by artificial perception models round-off effects and data representation limitations

Biological systems: physical constraints on sensors discretisation (analogue-to-spike train) neural noise (firing apparently not due to stimuli) structural constraints on neural representations and computations

??