### 1 Tutorial 4.1: Testing and Error Estimation

- Reading: Stephen Marsland: Chapter 2.2
- Look up estimation and confidence intervals in statistics if you do not remember what it is and how to do it (e.g. Johnson and Bhattacharyya).

#### 1.1 Problem 1: Splitting the data set

From previous exercises, you should have a function which loads the data set and returns a list of pairs, where each pair contains a class label and a feature vector. We need a function to split it into a training set and a test set.

Ideally we should split randomly, but randomness is tricky. We will learn it next week; for now we split it naïvly and deterministically. Still, we need to make sure that both classes are represented in both sets. There are many ways to do this, but let’s take the opportunity to practice list processing in Haskell.

##### 1.1.1 Step 1: Splitting the classes

Write a function which takes a list of class label/feature vector pairs, and returns a list of lists, where each list includes pairs with the same class label.

- 1.
- We start with a function
`getByClass`

to get all the objects of a given class. Add the following declaration to your`ANNData`

module.`getByClass :: Double −> [(Double,[Double])]`

The first argument is a class $c$, and the second is a data set as returned by

−> [(Double,[Double])]`getData`

in the previous tutorial. The return value is a list of those objects from the input which belong to class $c$. - 2.
- Use list comprehension to add a definition for
`getByClass`

. - 3.
- Secondly, we add a function to sort the entire data set by class. Add the following type
declaration to
`ANNData`

:`splitClasses :: [Double] −> [(Double,[Double])]`

The first argument is the list of class labels in use. The second is a data set as returned from

−> [[(Double,[Double])]]`getData`

. The output is a list of lists, where each constituent list is the return value of a call to`getByClass`

- 4.
- Use
`map`

and`getByClass`

to define`splitClasses`

.

Optional improvement The above solution is not optimised for speed.

- 1.
- How many times does the
`splitClasses`

have to read through the`dl`

list? - 2.
- How can process the input list in a single pass?

##### 1.1.2 Step 2: Training and Test sets

Having a list of lists of objects as returned from `splitClasses`

, we need to take a fraction
$p$ of the elements for
training and the remaining $1-p$
of elements fort testing. from each list for training and the remainder for testing. We are looking for
the function with the following type.

`mkTestTrainSets :: Double −> [[(Double,[Double])]] `

−> ([(Double,[Double])],[(Double,[Double])])

The first argument is the percentage $p$.
The second argument is the list of lists as produced by the function in Step 1.

- 1.
- The
`mkTestTrainSets`

function takes a list of lists as input. Let’s start with a helper function which takes just one lists and splits it in two:`mkTestTrainSets’ :: Double −> [(Double,[Double])]`

Add the declaration to

−> ([(Double,[Double])],[(Double,[Double])])`ANNData`

- 2.
- Write $p$ and
$v$ for the
inputs, and $\left({v}_{1},{v}_{2}\right)$
for the output. Implement
`mkTestTrainSets’`

such that ${v}_{1}$ contains a fraction $p$ of the elements from $v$ and ${v}_{2}$ contains the rest. You need the following steps:- a)
- find the length $l$ of $v$
- b)
- calculate the number of elements $p\cdot l$ for ${v}_{1}$
- c)
- calculate the number of elements $\left(1-p\right)l$ for ${v}_{2}$
- d)
- split the input list $v$ into ${v}_{1}$ and ${v}_{2}$

In order to multiply an integer ($l$) with a float ($p$) you need to convert the integer using the

`fromIntegral`

function. You find the list functions you need in the list on page 127 of Simon Thompson’s book. - 3.
- We can now complete
`mkTestTrainSets`

as follows.`mkTestTrainSets :: Double −> [[(Double,[Double])]]`

−> ([(Double,[Double])],[(Double,[Double])])

mkTestTrainSets _ [] = ([],[])

mkTestTrainSets f (d:dl) = prepend e l

where l = mkTestTrainSets f dl

e = mkTestTrainSets’ f d

prepend (x,y) (x’,y’) = (x++x’,y++y’) - 4.
- Discuss: What exactly does the local
`prepend`

function above do? - 5.
- Discuss: Could this function have been written differently?

##### 1.1.3 Step 3: Testing it

- 1.
- Load your
`ANNData`

module in GHCi. - 2.
- Run the following test:
`s1 <− getData "wdbc.data"`

let s2 = splitClasses [0.0,1.0] s1

mkTestTrainSets 0.2 s2 - 3.
- Discuss: Is the output as expected? What should you expect?

#### 1.2 Problem 2: Testing on a single item

Given a test set (as obtained in Problem 1) and a trained perceptron (as obtained in Tutorial 4), we are going to test the perceptron. Let’s do a simple test for now, where we do not distinguish between false positives and false negatives.

- 1.
- Add the following declaration to your
`Perceptron`

module:`testNeuron’ :: Neuron −> (Double,[Double])] −> Bool`

The first input is a (trained) neuron and the second is a label/feature vector pair. The output is True if the given feature vector is correctly classified by the neuron. - 2.
- Implement the
`testNeuron’`

function. You can use the following skeleton as a basis`testNeuron’ n (t,xs) =`

You need to calculate the output

where y =`y`

from the neuron and compare it to the target value`t`

. - 3.
- Discuss: How can you test this function? If possible, make a test.

#### 1.3 Problem 3: Testing on a data set

Now we have a function to test a single neuron. How do you test the neuron on every item in the test set?

- 1.
- Add the following type declaration to your
`Perceptron`

module:`testNeuron :: Neuron −> [(Double,[Double]) −> [Bool]`

The return value is a list of boolean values, where True indicates an error and False indicates a correct classification.

- 2.
- Add a definition for the
`testNeuron`

function, by applying`testNeuron’`

to every object in the input list. - 3.
- Discuss: How can you test this function? If possible, make a test.

#### 1.4 Problem 4: Putting it together

Having solved Problem 2 as well as Tutorial 4, we are able to both train and test a perceptron. Now we need to put it all into one executable program.

- 1.
- Create a Main module which loads the breast cancer data trains a perceptron on part of the
data, and tests it on the remaining data. You may use this code:
`module Main where`

−− Import your own modules:

import ANNData −− load CSV data

import Perceptron −− neuron

p = 0.1

main :: IO ()

main = do

dl <− getData "wdbc.data"

let (testS,trainS) = mkTestTrainSets p (splitClasses [0,1] dl)

let dlen = length $ snd $ head testS

print $ "Input␣length:␣" ++ show dlen

let untrainedNeuron = initNeuron dlen

let trainTarget = map fst trainS

let trainInput = map snd trainS

let trainedNeuron = train 1000 0.25 trainInput trainTarget untrainedNeuron

print "Untrained␣network,␣test␣data:"

testPrint untrainedNeuron testS

print "Untrained␣network,␣training␣data:"

testPrint untrainedNeuron trainS

print "Trained␣network,␣test␣data:"

testPrint trainedNeuron testS

print "Trained␣network,␣training␣data:"

testPrint trainedNeuron trainS

testPrint n s = do

let result = testNeuron n s

let right = length [ x | x <− result, x ]

let wrong = length [ x | x <− result, not x ]

print ( "Correct:␣" ++ show right )

print ( "Errors:␣␣" ++ show wrong ) - 2.
- Add comments in the
`Main`

module to explain what each line does. - 3.
- Discuss: What is the constant
`p`

used for? - 4.
- Compile and test the program.
- 5.
- Discuss the output of the program. How do you interpret the different numbers?
- 6.
- Discuss. Are the error counts reasonable or not?

It is quite possible that you get a lot of classification errors. Don’t worry if you do. It is likely because the single neuron is a bit too simple. Next week we will discuss how to build networks of multiple neurons.

##### 1.4.1 Problem 5: scaling of features

A common problem in machine learning is that features with very high magnitude may dominate over others, so that the classifier is not properly trained in all dimensions.

- 1.
- Look at the breast cancer data set. What is the range of the features in different columns?
- 2.
- We have prepared a scaled version of the data set. All the features have been brought into the [0,1] range by linear scaling. Download the file and put it together with the other files.
- 3.
- Replace
`wdbc.data`

with`wdbc.csv`

in your`Main`

module. - 4.
- Recompile and test the program.
- 5.
- Compare your error rates to those of your class mates.
- 6.
- Discuss: How does scaling affect the error rates?

If the error rates are still bad, don’t worry. We will extend the single neuron perceptron to a neural network in the next couple of tutorials.

#### 1.5 Problem 6: Statistical analysis

Given the number of errors $e$ and the number of tests $n$, we can calculate the error rate $e\u2215n$. We are interested in the probability of error, when the classifier is used on a random, unknown object. Answer the following:

- 1.
- Discuss: What is the relationship between the error rate and the probability of error?
- 2.
- Discuss: What is the probability distribution of the number of errors $e$?
- 3.
- Calculate a 95% confidence interval for the error probability.