Notesale: Turn your study into money

Document Preview

Extracts from the notes are below, to see the PDF you'll receive please use the links above

---

2012
1
...
It can have occasional reward or
punishment, i
...
reinforcement learning, or it can be fully-supervised where each example includes correct
output
...
e
...
Examples of unsupervised
learning include competitive learning,
The difference between supervised and unsupervised learning is that supervised leaning has a teacher and
feedback about right or wrong inputs
...

Describe the perceptron learning algorithm and how it can be used to perform pattern recognition (5
marks)
...

The perceptron have to be trained first by known examples of input data
...
One type of input data could have target value 1, while another can have
target value 0
...

Then change the connection weights according to the delta rule:

Wij  Wij + ɛ Xjn (tin –oin)
Then present the next input pattern
...
The delta rule changes
weights to reduce this error
...

After many presentations of the whole training set, the network can find the best linear discrimination of
between two different input patterns
...
However, data will usually have anomalies
...
This
is a problem, but if data is fit exactly by the training data set, there will be over-fitting
...
However, the
threshold value is also important as it is a determinant of the decision boundary between the two input data
sets
...
Threshold = output (O), and assume XO= -1, then WO can serve the
same purpose as the threshold value, i
...
WO=T
...
Each output is trained using
the delta rule independently of the others
...
g
...
Hidden processing units can give the perceptron more power, so a single layer
1

2012
perceptron become multi-layer
...
The internal
representation is not known, so there must be error-back propagation to change each layer of the network
using a generalised delta rule
...
In this case,
changing a connection weight from any active neuron anywhere in the network will change the output firing
rate slightly
...
The effect of changing a
connection weight depends on the values of the errors, the different connections and the activations
upstream
...

Kohonen’s feature map- Biologically speaking, the neighbourhood function occur by physical spread of
chemical neurotransmitters or messengers
...
Training sets may have distributions of data to actual
sets of data, the linear decision boundary generated by perceptrons would be too rigid
...

Error-back propagation- very unlikely to be used in real life
...
Unlike Hebbian learning which is based on one synapse and its presynaptic and
postsynaptic firing rates
...

Unsupervised learning- Hebb’s law
...
However, it does not take into the temporal factor
...
If the post-synaptic neuron fires before pre-synaptic
neuron, there tend to be LTD
...

Hopfield’s (1982) associative memory network uses unsupervised learning, but the symmetrically connected
recurrent networks and limited storage capacity makes it less biologically plausible
...
Error propagation in supervised learning is not biologically plausible because the connection
weights are changed using non-local information
...

2
...

Competitive learning is an example of unsupervised learning
...

Algorithm sequence:
1
...

3
...


Random initial connection weights in the network
Present input pattern
Winner is generated
Hebbian learning occurs where the connection weight between the output neuron and the input
neuron that fired together increases
...
Normalisation: reduce the total size of connection weights to each output by dividing each weight by
the sum of all the weights so that the sum of all weights equal to 1
...

6
...
Different
outputs find their own clusters in input data
...
The lateral connections between outputs can
vary with neurons’ separation, I
...
excitatory connections between nearby neurons and inhibitory
connections between neurons far apart
...
This means neurons near the
winning neurons also respond to similar patterns (Wilshaw and Von der Marlsburg 1976)
...
So one may present many input patterns,
for each one, change weights according to:

Wij  Wij + ɛF (i,k) (Xjn – Wij)
Where (i,k) is the distance between the neurons
...

This means nearby outputs learn to be active to similar stimuli, which produces a 2D map of complex
(multidimensional) data
...

Describe the spatial firing patterns of ‘place cells,’ ‘head-direction cells’ and ‘grid cells’ recorded in freely
moving rats (6 marks)
...
The firing range of the
place cells are called ‘place fields’
...
However, in narrow arm mazes, where the animal is running
back and forth on linear tracts, the place cells fire differently depending on the direction of movement of the
rat
...
If all cues are removed, the place cells might
rotate slowly because the mouse doesn’t known the definitive location anymore
...
The cell fires whenever
the rat’s head points in a specific direction relative to the environment, irrespective of its location or
whether it is moving or still
...

Grid cells: found in layer 2/3 of medial entorhinal cortex, discovered by Hafting et al
...
They are similar
to place cells in that they have localised regions in which they fire, but grid cells fire in a hexagonally
symmetric pattern, which tessellates the environment in which the rat is placed
...
Grid scale increases as the cell
move from dorsal to ventral medial entorhinal cortex
...

Sharp’s (1991) model of place cell firing combines the idea that place fields results from ‘local views’ with the
idea that the hippocampus contains Hebb-like synaptic mechanisms that store patterns of sensory inputs, in
3

2012
order to model the activity of hippocampal place cells
...
The input layer are neocortical cells that have sensory responses to the elements of the
patterns of stimuli available to the animal’s sensory receptors
...
They are topographically
organised in the sensory cortex (unlike place cells)
...
Initially, which output neuron fires depend on the direction
cues, but as competitive algorithm is applied after exploration, neurons will tend to develop responses more
tuned to location rather than direction
...

Zhang (1996) and McNaughton et al
...
Imagine an environment bound by the x- and y-axis with a rat running around
in it
...
Each cell’s location
reflects the location of its firing field in the environment (note that this is not how it is arranged in the brain)
...
So a map is formed where each cell in this network is strongly connected to
those next to it and weakly connected to those far away
...
The bump (z-axis plot) indicates the rat’s location and
moves as the rat moves
...
If the connections between all the cells solely reflect
the cells separation, so that nearby cells have stronger connections than far apart ones, the set of ‘bumps’ of
activation form a continuous attractor
...
The cells at the current location excite the cells near it, but inhibit the cells far away to
ensure that there is only one ‘bump’, i
...
one location represented in the brain
...
These cells are
tuned to respond when a barrier lies at a specific distance from the rat in a particular allocentric direction
...
BVCs receives some
cues from the environment about distances of the boundaries
...
Because BVCs don’t care about the actual orientation of the rat, place
cells don’t either
...
BVCs
subsequently found in the subiculum of the hippocampal formation (Lever et al
...
However, there are
some questions for this network
...
The BVCs model is good at capturing sensory
aspects, but it doesn’t take learning into account
...
Place field stability in a
novel environment depends on NMDA receptors in CA3 (Kentros et al
...
2002)
...
2002) and
place cells distinguish locations with experiences (Barry et al, 2006)
...

3
...

A simple artificial neurons have a threshold logic function, the network uses a simple integrate and fire
model
...
The ‘net input’ to the neuron is calculated as the ‘weighted sum’ of input
activations
...
This is like calculating a dot product of vectors
...
g
...

Leady integrate and fire neuron model does coincidence detection
...

Jeffress (1948) came up with a coincidence detection model for interaural time difference calculation
...
Maybe input from left ear is via short axons, but input from right ear is via an opposing axons, so
the spike has to travel further from right ear
...

Coincidence detection requires accurate phase-locking of spike generation to stimulus onset
...

Time of firing of neurons are especially useful in scale-invariant recognition
...
g
...
Doesn’t matter if the person
speaks softly or loudly, the identity of the speaker can still be recognised
E
...
scent, either weak or smell, still recognisable
Pattern of activity in input neurons is a vector, x
...
To recognise input
pattern of activity x as similar to stored pattern x*, i
...
x ≈ λx*
...

Standard neural networks have to normalise inputs to do scale-invariant recognition
o Normalisation could be achieved by feedback inhibition
o But this still poses a problem as standard neural networks are strength dependent rather
than pattern dependent
...
This weak
input is not fundamental in deciding what the outcome is
...

Linear pattern separation (perceptrons: weighted sum of firing rates and threshold) are not suitable
for scale invariant recognition
Tuning to respond at a given absolute value (e
...
head direction and place cells, radial based
functions) are not suitable for scale-invariant recognition either
5

2012


Tuning to respond to given relative values are suitable for scale invariant recognition
...
This model of computation
explains how one scheme of neuroarchitecture can be used for very different sensory modalities and
seemingly different computations
...
g
...

When there is no input current, the sub-threshold cell potential oscillation (solid line) never exceeds the
firing threshold (horizontal broken line)
...

The pattern is coded by the timing of spikes relative to a sub-threshold membrane potential oscillation
...
e
...
If there is a big input current, then the shift of cell potential oscillation upwards will
occur relatively early in the oscillation wave
...
So small
inputs must fire near the peak of the sub-threshold membrane potential oscillation for it to be able to bring
it to firing threshold
...

Olfactory bulb, hippocampus have this oscillation so this might be how information is coded in them
...
The input neurons are coded by the strength of a
smell they’ve detected and their inputs are one spike each, but the timing of that input is crucial instead
...
So neuron
one fires early in the oscillation wave because it has a large current and brings the cell’s oscillation above
firing threshold
...

So all the neurons fire once per cycle at slightly differently points in the cycle
...
So all the
input neurons’ spikes will arrive simultaneously at the output neuron, and that will make a big impact on the
output cell, especially if the cell does coincidence detection
...
If there is a different pattern, the conduction delays will not fit and the firing will not
summate
...
EEG theta oscillations in the hippocampus means the place
field is determined by phase-locked firing patterns rather than firing rates
...

Provides alternative to standard models of input values = activation (firing rates) and processing using
variable connection strengths (more plausible due to existence of LTP and LTD)
...

Adding scale invariant:
If the time advance (Ti) is proportional to the log of the input current, log(xi), then the network will do scaleinvariant recognition, if the input current strength is doubled: log(2xi)= log(2) + log(xi)
...
By doubling, the output neuron will respond earlier by log(2)
...

Consider the shape of the oscillatory curve, it is roughly sinusoidal so it could produce something like a log
coding of stimulus intensity
...
So the shape of this membrane oscillation curve over
time means small increases in inputs are actually magnified in giving a big time advance than big inputs
...

Evaluation of model
The oscillation is 10 Hz, so that is 100 msec between each peak, therefore time advance of different neurons
have to be 10, 20 msec
...
e
...
However in myelinated axons, transmission delays are much shorter than this, usually 1 or
2 msec
...
In the
hippocampus there are weird axons from dentate gyrus granule cells going to CA3 pyramidal cells which are
not myelinated, so transmission may be slow
...
However even
if it is possible to have different delays, there is no learning rule for setting delays or means of changing
them
...
So a developmental
setting might exist so that important odours are hardwired and automatically detected, but it is not
particularly flexible like the standard model is, where there is plenty of evidence that you can rapidly change
the strength of connections in synapses
...
What are the ‘credit assignment’ and ‘temporal credit assignment’ problems (4 marks)?
These problems occur in reinforced learning, a form of slightly supervised learning
...
The solution is to have an internal critic to assess
performance and guides actions
...
The temporal difference learning is a slow ‘trial and error’ based
method, but it is biologically plausible
...

7

2012
How can occasional reinforcement from the environment be used to enable learning of the behavioural
responses which will maximise reinforcement (10 marks)?
If reinforcement is intermittent (e
...
only when goal is reached), an internal critic is developed to associate a
value with each situation
...
e
...
So
what output neurons should be active for active input neurons? E
...
if you are in NE according to your place
cell, you want to have an output of SW if that’s where the goal is
...
The critic learns an ‘evaluation function’, which tells you whether each situation is
good or bad depending on whether movement was closer or further away from the goal
...
When blundering
around, the critic assigns a value of your position (input neurons), the value increases when it’s closer to the
goal and decreases when it’s further away
...
Then the change in value and the occasional reward from the environment could be
used to change the connection weights of the input neurons so that good movements are more likely and
bad movements are less likely
...
e
...
now v(x)=1
...
now v(x’) =1
...

Discuss how such a reinforcement learning system might be implemented in the brain (6 marks)
...

They came up with a rule to try and understand how a stimulus, S, leads to a reward
...
In the brain there will be
lots of different neurons representation reinforcement and lots of neurons representing stimulus
...


w → w + εSδ; δ = r - wS
If the reward is present, the connection should increase in value
...

If the reward is not present, the connection weight should decrease
...


w → (1-ε) w + εr
This can be thought of as a simple ‘delta rule’ model
...
If stimulus is present, S=1
...
This is viewed as error-driven learning
...
So connection weight won’t change
...

If r < wS, then δ = -n, so the connection weight decreases
...
From the animal’s perspective, there is
only one reward, therefore you only really want one delta to tell you how good your prediction is overall for
the reward
...
If S1 is present but the received reward is bigger than the
expected reward, i
...
r >V, then delta will be positive and increase the connection weight of that stimulus
...

Kamin (1969) did an experiment that proved the second delta rule is what the brain would rather use
...


Second-order conditioning

The test shows that S2  r
Stimulus two is indirectly associated with reward, so the Rescorla-Wagner rule doesn’t work as there is no
reward in phase 2
...
The
temporal credit assignment problem comes into play
...
e
...
The sum for all times that τ≥t of all the future rewards
...

So use delta rule to ensure that this happens, i
...
modify connection weights to make V(t) closer to r (t) +
V(t+1), i
...
use:

wi (t+1) = wi (t) + εsiδ(t);
δ(t) = [r (t) + V(t+1)] – V(t)
So that δ is the difference between V(t) and estimate of all future reward
...
Code time as sequential activity of set of input
neurons
...
This is not
necessarily how time is represented in the brain
...

So in a classical conditioning experiment, CS occurs at a certain time (tcs) and reward occurs at a certain
time, (tR)
...
But δ(t) exists as a bump when the unexpected reward occurs
...

In the second trial, V(t) is not zero, it is positive at the time that reward occurs from the previous learning in
trial 1
...
At

the time of the reward, δ(t) =0 because r(t) = 1, V(t+1)=0, and V(t) =1, this is based on the assumption that
the connection weight of tR has been fully increased
...
e
...
Before the reward occurs, your own estimate value increases, so the connection of tR-1 increases
...
And delta will be positive just before V(t) increases
...
And delta is positive as
soon as the bell goes, because V(t+1)=1, and it is zero everywhere else because V(t) =1
...

Modelling action choice
Critic: estimates the value of the current state
...

The temporal difference learning rule, signalled by dopamine, take actual reward and estimate of reward
and use that as a learning rule to change the connection weights of input neurons
...

If V is correct, it allows future reward to be taken into account in reinforcing actions: solving the temporal
credit assignment
Dopamine’s role
Dopamine is a neuromodulatory molecule that comes out of the axons from substantia nigra
...
Involved
in reward and addiction, and motor control
...
1997
If animal is given unexpected reward, with
no CS, there is firing of dopaminergic
neurons
...
This is similar to the delta signal
described above
...
During the
first trial, the reward was unexpected, so
r(t) =1, and delta is positive 1
...
When there is no reward, δ(t)= 1, because r(t) is zero, i
...
no reward, V(t+1) =0, as there was no reward after the CS, but V(t) =1, i
...
there
was prediction of reward
...


11

2012
Dual dopamine systems project same signal to motivational (ventral [bottom] striatum) & motor (dorsal
[top] striatum) areas, for state evaluation & policy improvement respectively? Policy is how you are going to
act, the learning of the ‘actor’
...
Policy is another way of saying what the actor is
going to do
...
Dorsal striatum is more to do with caudate and putamen, therefore projections to
motor systems
...

So we expect lots of firing at reward during first trial, then this firing shifts to CS presentation and no firing at
reward
...
So in fMRI studies, you can look for
brain regions with activation patterns like this
...
2004)
...
Describe how ‘associative’ and ‘auto-associative’ models of memory work (6 marks)?
Associative models of memory
Feed-forward memory matrix is an example
...

This network learns input-output associations
...
This applies to sequence learning
...
It is a fully connected recurrent network with symmetrical connection
weights, units are active (+1) or inactive (-1)
...
The neurons are either inactive or active and the weights are either on or off
...
When one pattern is stored, its inverse is also
stored as the same connection weights are changed
...
14 x no
...
There will be overlapping of memories, and quality of memory gets
degraded
...
e
...

Also learning and retrieval have to be separated
...
So the brain needs to differentiate
encoding and retrieval of cues, otherwise spurious memories form
...
When a new pattern is imposed, the detonator synapse is activated
...
During learning, pattern is imposed on the network with such strength that it overrides any
retrieval
...
e
...
However, when retrieval information from a library, it is non-content
addressable
...
So when retrieving from a library, there is a ‘second party’ in play, whereas
for human memory, the content is directly used for retrieval
...

There is a network that an only store a limited number of patterns, too many patterns means interference
occur, spurious memories form
...
In order of the hippocampus to do this, you need very plastic synapses at a very fast learning raterapid ‘one-shot’ associative learning
...
This is the price to pay for fast learning rate
...
The
slow system has no problem with interference, can look at events over time and slowly extract underlie
structures, look at commonalities between events and make them stronger
...
This idea is based on Marr (1971)
...
People argue that during sleep, the hippocampus replays the event and trains the neocortex
to remember and associate the different inputs (smell, sight etc
...

o Is this transfer or training? Do memories truly leave the hippocampus?
Timescale? One night or twenty years → graded retrograde amnesia?
Generalisation / abstraction of meaning from multiple events leads to semantic memory from
episodic memory?

Hippocampus is an index (this index represents cross-modal biding) or convergence zone for disparate
neocortical sites which store the memory content
...
This is anatomical evidence
why memory transfer might be true
...

The standard consolidation model (Squire and Alvarez 1995)



States that all declarative memories are treated in the same way by the medial temporal lobe, this
means semantic memory of facts and episodic memory of events are the same
...
It binds together neocortical memory traces
...


Memory consolidation occurs when the hippocampal system repeatedly reactivates representations in
neocortex, leading eventually to strong interconnections among cortical sites, which can support memory
independently of the hippocampal system
...
He had anterograde amnesia and graded
retrograde amnesia
...
But with time,
memory gets consolidated and becomes semanticised in the neocortex, so their retrieval becomes

13

2012
independent of the hippocampus
...

However, there have been reports of hippocampal damaged patients having flat retrograde amnesia,
meaning that there is equally severe amnesia for recently and remotely learned information, e
...
patient VC
(Cipolotti et al
...

Patient Jon (Varha-Khadem et al
...

He had impaired episodic memory but preserved semantic and recognition memory
...
Also, Jon shows impaired recall but preserved unimodal recognition
...
However, results should be taken with caution- developmental compensation
...
(2005) tried to assess memories for the amount of detail rather than using crude questions
...
M, they showed there was a lack of detail for remote memories
...

Multiple Trace Theory (Nadel and Moscovitch 1997)
Every time we remember something we lay a new memory trace in the hippocampus
...
Also, with time they lose detail and only retain gist of event, i
...

become more semantic, and they are no longer mediated by hippocampus but by neocortex instead
...
This is because the
spatial and temporal contextual information that make a memory episodic is dependent on the involvement
of the hippocampus (spatial context) and frontal cortex (temporal context)
...

A study conducted by Gilboa et al
...
The findings show that remote and recent memories correlated with equal activity in the
hippocampus
...
This shows that context-rich memories are associated with hippo regardless of age
...
2006) was a taxi driver who suffered from a specific hippocampal lesion
...
His results showed that when he goes
off a main road (e
...
A road), he gets lost
...
However, the more complex roads still required the hippocampus
...

Semantic dementia patients lose older memories (reversed gradient to HM)
...

6
...
A population vector is the firing rate weighted vector sum of preferred
firing directions of neurons
...


14

2012
Basically each neuron has a preferred firing direction, and all the neurons of different firing direction has
different firing rates
...
This is a good way of averaging across all the noisy neurons
to get a good overall estimate of which direction the neurons are encoding
...
g
...

Reaching movement could be coded by population vector
...
These different neurons with different preferred reaching directions are all fire at
different rates, so just take the firing rate weighted vector sum of preferred directions to get an estimate of
which reaching direction the neurons are encoding (Georgopoulos et al
...

When an animal decide which way to reach, your motor cortex comes up with a population vector that
determine direction of movement, then this activity goes down in the spine and makes your limb move in
that direction
...

Population vectors can also be used for place cells and head direction cell firing
...
However, all values are not represented equally, there is a
central tendency
...
But a place cell tuning is a little sharper than cosine, because they have specific place fields
...

Discuss how movements of a limb to a given position in space might be aided by processing in the spinal
cord (7 marks)
...

The idea of convergence force fields originated with Bizzi et al
...
He stimulated the spinal cord, either
electrically or using glutamate at different levels and different areas of the spinal cord
...
This was expected
...
They were
called convergence force fields (CFF)
...
In other words, if the leg was already at the equilibrium position
to start with, stimulation of lateral neuropil region would produce no movement
...

Micro-stimulation at different levels of spine produces CFFs with different equilibrium positions
...


15

2012
This wiring is advantageous to the organism
...
This was what Bizzi et al
...
So if an
animal wants their leg to go anywhere, you can move it there from any position just by giving the right of
relative activation to different CFF generators, which all have equilibrium positions around the edge of the
work space
...
This is the equilibrium
point control hypothesis
...
of CFF generators
...
While the equilibrium point control hypothesis may be relevant for frogs,
it might not be as applicable to primates
...
In particular, fine movements of fingers and thumbs don’t seem to be
focused in this way
...

How might a sequence of items or actions be stored and reproduced in the correct order (7 marks)?
There are two ways in which serial learning might be implemented in neural networks: associative chaining
or competitive question
...
The
network uses asymmetric recurrent connections and have feedback from output neurons to the input
neuron, leading to dynamic output
...
g
...

Jordan (1986) and Elman (1990) were interested in motor control and how you can produce stereotyped
sequence of output given a particular input by using error back propagation
...
He called them context units
...
This means the internal
representation of the previous action is part of the input of the next action
...
This type of network can do
fixed association of small number of elements in order, learning of stereotypical action sequences, e
...

central pattern generators (controlling muscle contractions in walking, swimming, and heartbeat etc
...
However the patterns must be uncorrelated and number of patterns in the
stored sequence must be much less than the number of units in the network
...
Also, it doesn’t correlate with error data
...
It is a special mechanism for the serial selection of actions
(Grossberg 1978; Houghton 1990)
...
E
...
drinking tea and eating cake
...
So perhaps at the start the ‘tea’ neuron is stronger, so it will inhibit the cake neuron via
lateral inhibition
...

This means the inhibiton on cake neuron decreases and cake neuron wins
...
Competitive queuing establish an
activation gradient over the units representing the actions to be sequenced
...
2002
...
g
...
Monkeys drew the shapes as sequences of movement segments, and
these segments were associated with distinct patterns of neuronal ensemble activity in the prefrontal
16

2012
cortex
...
Even before the primate starts
to move, the different representations of activity have an order, with the first action is the most active

---