---
title:  "Testing, Data types and Reading in data"
author: "Emma Rand"
output:
  html_document:
    toc: true
    depth: 3
    toc_float:
      collapsed: false
      smooth_scroll: false
    theme: lumen
    highlight: pygments
  word_document: default
bibliography: "../refs/references.bib"
---
![](../../pics/17C.png)


# Introduction

## Session overview

This week we cover how we can classify variables in terms of the type of values they can take and their role in analysis. We consider the  impact variable type has on the tests that we conduct. In RStudio, we learn to read in data files and to summarise and plot data. We also cover saving figures and laying out a report in word. 

![Artwork by Allison Horst](../../pics/horst/continuous_discrete.png)

## Learning Outcomes 

By actively following the lecture and practical and carrying out the independent study the successful student will be able to:

* to able to explain what response and explanatory variables are, distinguish between data types and describe how these impact choice of test (MLO 1 and 2)
* demonstrate the process of hypothesis testing with an example and evaluate potential inferences (MLO 1 and 2)
* read in data in to RStudio, create simple summaries and plots using manual pages where necessary (MLO 3)
* create neat reports in Word which include text and figures (MLO 4)



## Philosophy

Workshops are not a test. It is expected that you often don't know how to start, make a lot of mistakes and need help.
Do not be put off and don't let what you can not do interfere with what you can do. You will benefit from collaborating with others and/or discussing your results.

The lectures and the workshops are [closely integrated](../BIO00017C.html) and it is expected that you are familar with the lecture content before the workshop. You need not understand every detail as the workshop should build and consolidate your understanding. You may wish to refer to the slides as you work through the workshop schedule.

## Slides

Tests, Data types and Reading in data: [pdf](../lects/web02Testing, Data types and Reading in data.pdf) (recommended) / [pptx](../lects/web02Testing, Data types and Reading in data.pptx)

Your [responses](../lects/responses_lecture2.pdf) from TurningPoint. 

## Getting started

![W](../../pics/W.png) Start RStudio from the [Start menu](../../pics/startmenu.png).

![R](../../pics/R.png) In RStudio, [set your working directory](../../pics/menusetwd.png) to the folder you created last week for your 17C Data Analysis work.

![R](../../pics/R.png) Make a [new script](../../pics/newscript.png) file called practical2.R to carry out the rest of the work. 

# Exercises

## Import Data

The data in [pigeon.txt](../data/pigeon.txt) are 40 measurements of [interorbital](../../pics/interorbital.png) width (in mm) in a sample of domestic pigeons measured to the nearest 0.1mm

![W](../../pics/W.png) Save a copy (right click and Save) to your working directory

![R](../../pics/R.png) Read the data in to R by using the `read.table()` command:

`pigeon <- read.table("pigeon.txt")`

```{r, message=FALSE, warning=FALSE, include=FALSE}
# I have my files organised differently so I use a different 'path' to my file
pigeon <- read.table("../data/pigeon.txt")
```

You should find an object called pigeon can be seen in the Environment window. If you click on that object a [view](../../pics/pigeondf.png) of the data appears in the same window as the script.

![R](../../pics/R.png) Use `str()` to see what data structure is created
```{r, message=FALSE, warning=FALSE, include=FALSE}
#---CODING ANSWER---
str(pigeon)
```

![R](../../pics/R.png) Change the variable name using ` names() `
```{r, message=FALSE, warning=FALSE}
names(pigeon)[1] <- "interorbital"

#and look again with str
str(pigeon)
```

## Exploring single group data

Values can be extract by indexing - giving the row number (`r`) and column number (`c`) of the element you want: `pigeon[r, c]`

The column is referred to using either

1. the index `pigeon[,1]` (note `pigeon[1]` will also work because R assumes you mean the column if you give only one index for a dataframe) 

2. or the dollar notation `pigeon$interorbital`. 

![R](../../pics/R.png) Can you extract the 14th value in the column? 
```{r echo=FALSE,results="hide",warning=FALSE,message=FALSE}
#---CODING ANSWER---
pigeon$interorbital[14]
#or
pigeon[14, 1] 
```


![R](../../pics/R.png) We can use the 'base' plotting system (as opposed to the ggplot plotting system) to make a histogram with:
```{r}
hist(pigeon$interorbital)
```

![R](../../pics/R.png) Can you create this histogram  by reading the manual page for `hist()`

```{r echo=FALSE}
#---CODING ANSWER---
names(pigeon)[1] <- "interorbital"
hist(pigeon$interorbital,
     xlim = c(8, 14),
     main = NULL,
     xlab = "Width (mm)",
     ylab = "Number of pigeons",
     col = "grey")
    ```
   
Note: the _x_ axis goes from 8 to 14 which is given by c(8,14) and to lose text you can set the appropriate argument to NULL.

It's useful to learn to `ggplot` way to do a histogram as it is far more more useful when we do more complex figures. `ggplot2` is one of the `tidyverse` packages.

![R](../../pics/R.png) Do a `ggplot()` histogram
```{r, message=FALSE, warning=FALSE}
# library statement is only needed once per session
library(tidyverse)

ggplot(data = pigeon, aes(x = interorbital)) +
  geom_histogram()

```

![R](../../pics/R.png) We can improve it by changes the number of bins
```{r, message=FALSE, warning=FALSE}
ggplot(data = pigeon, aes(x = interorbital)) +
  geom_histogram(bins = 10, colour = "black")

```



R provides many functions to allow you to make calculations on your data.

![R](../../pics/R.png) Find out which functions will calculate the mean, standard deviation and the number of cases, i.e., the length of the vector. Google or make intelligent guesses and verfiy with the manual.
```{r echo=FALSE,results="hide",warning=FALSE,message=FALSE}
#---CODING ANSWER---
mean(pigeon$interorbital)
sd(pigeon$interorbital)
length(pigeon$interorbital) 
```

## Import Data 2

Now you will practice with 'paths' and 'relative paths'. These are a threshold concept in computing.

The data in [pigeon2.txt](../data/pigeon2.txt) are similar data of interorbital distances in two columns rather than just one.  They are from two different populations, A and B.

![W](../../pics/W.png) Make a [folder](../../pics/folderdata.png) in your working directory called 'data'

![W](../../pics/W.png) Save a copy of [pigeon2.txt](../data/pigeon2.txt) to the 'data' folder

![R](../../pics/R.png) To read the data in to R you need to use the  relative path to the file in the `read.table()` command:

`data2 <- read.table("data/pigeon2.txt", header = TRUE)`

The `data/` part is the 'relative path' to the file. It says where the file is relative to your working directory:  pigeon2.txt is inside a folder (directory) called 'data' which is in your working directory. `header = TRUE`  tells R that first row of the data file contains the column  names. 

```{r, message=FALSE, warning=FALSE, include=FALSE}
#I have my files organised differently so I use a different 'path' to my file
data2 <- read.table("../data/pigeon2.txt", header = T)
```

![R](../../pics/R.png) look at the structure of the dataframe data2 using `str()`
```{r echo=FALSE,results="hide",warning=FALSE,message=FALSE}
#---CODING ANSWER---
str(data2) 
```

You can see that R has used the first value in each column as the variable name.

![R](../../pics/R.png) Find the mean for each population. Remember, to refer to a single column of the data we need to use the dollar notation 
```{r echo=FALSE,results="hide",warning=FALSE,message=FALSE}
#---CODING ANSWER---
mean(data2$A)
mean(data2$B) 
```

## Tidy format
Instead of having a population in each column, we very often have, **and want**, data organised so the measurements are all in one column and a second column gives the group. This format is described as 'tidy' [@Wickham2014-nl]: it has a variable in each column and only one observation (case) per row.

We can put this data in such a format with the `gather()` function from the `tidyverse`:

![R](../../pics/R.png) To use gather you need to give a `key` which will become the name of the variable indicating which column (group) the value came from and a `value` which will become the name of the measurements (responses). Examine the structure 
```{r}
# library statement is only needed once per session
data3 <- gather(data = data2, key = population, value = distance)
str(data3)
```

How do we find the mean for each group now? If we use `mean(data3$distance)` we will get the mean of the whole column, not the means for each population. 
We can use 'selection' to apply a function only where a particular condition is met. For example:
```{r} 
mean(data3$distance[data3$population == "A"])
``` 

This selects the rows in column `$distance` where the value in column `$population` is equal to A. `==` means 'is it equal to?' rather than 'make it equal to' given by `=`

![R](../../pics/R.png) Find the mean for population B using data3
```{r echo=FALSE,results="hide",warning=FALSE,message=FALSE}
#---CODING ANSWER---
mean(data3$distance[data3$population == "B"])
```

Since data are so often organised in this format (one column for the measure with additional columns giving the groups), R has easier ways to apply a function to each group. We can apply the mean function to the `$distance` column separately for each population using a combination of two `tidyverse` functions: `group_by()` and `summarise()` and the 'pipe' (`%>%`)

```{r}
data3 %>% 
  group_by(population) %>% 
  summarise(mean(distance))
``` 

`%>% ` is called the pipe. It is a special operator that is realtively new in R and makes code easier to read and write. Usually, when you want to give data to a function you pass it as an argument inside the brackets. The pipe allows you to have the data on the left, then a pipe then the function with and additional arguments. This especially useful when you want to apply one function after another. The independent study covers the pipe further.

As you can imagine, this is much more convenient than using selection when you have many different groups.

![R](../../pics/R.png) Find the number of cases and variance of each group using `group_by()` and `summarise()`.
```{r echo=FALSE,results="hide",warning=FALSE,message=FALSE}
#---CODING ANSWER---
# variance
data3 %>% 
  group_by(population) %>% 
  summarise(var(distance))

# the number of cases (rows) is given by the length of the vector
data3 %>% 
  group_by(population) %>% 
  summarise(length(distance))
```

# Word reports with figures

Now we will create a word document with some text and figures. I find the best way to include figures in a word document is to use a table with the borders turned off. This can make it easier to control the layout. [This](../../pics/wordhist.png) is what you are aiming for.

![W](../../pics/W.png) open a word document

![W](../../pics/W.png) add some text

![W](../../pics/W.png) insert a 2 x 2 table

![W](../../pics/W.png) merge the cells on the second row and add a figure legend

![R](../../pics/R.png) Create a histogram for each group, trying to match those illustrated.

![R](../../pics/R.png) Copy the figure using [Export | Copy to the Clipboard](../../pics/savehist.png)

![W](../../pics/W.png) Paste in to a table cell in word

![W](../../pics/W.png) select the table and give it [No borders](../../pics/noborders.png)


```{r echo=FALSE,results="hide",warning=FALSE,message=FALSE,fig.width=4,fig.height=4}
#---CODING ANSWER---
# histograms for the word document
# Create a histogram for each population, trying to match those illustrated.
hist(data3$distance[data3$population == "A"],
     xlim = c(8, 14),
     ylim = c(0, 15),
     main = NULL,
     xlab = "Width (mm)",
     ylab = "Number of pigeons",
     col = "pink")
hist(data3$distance[data3$population == "B"],
     xlim = c(8, 14),
     ylim = c(0, 15),
     main = NULL,
     xlab = "Width (mm)",
     ylab = "Number of pigeons",
     col = "lightblue")
```




# Independent study
You need to carry out this work before the next practical.

## 1. Understanding the pipe: ` %>% `

The `magrittr` package [@magrittr] is part of the tidyverse and includes the pipe operater which can improve code readability by:

* structuring sequences of data operations left-to-right (as opposed to from the inside and out),
* minimizing the need for intermediates, 
* making it easy to add steps anywhere in the sequence of operations.

For example, to apply a log-squareroot transformation you might use:
```{r}
# generate some numbers
nums <- sample(1:100, size = 10, replace = FALSE)

# transformation either:
# a) nested functions
tnums <- log(sqrt(nums))
# b) intermediates
sqrtnums <- sqrt(nums)
tnums <- log(sqrtnums)
```

Nesting the functions means you have to read inside out and creating intermediates can be cluttered. 
The pipe allows you to avoid these by taking the output of one operation as the input of the next. The pipe has long been used by Unix operating systems (where the pipe operator is |). The R pipe operator is `%>%`, a short cut for which is ctrl-shift-M. 

```{r}
tnums <- nums %>% 
  sqrt() %>% 
  log()
```

This is short for 
```{r}
tnums <- nums %>% 
  sqrt(.) %>% 
  log(.)
```

Where `.` stands for the object being passed in. In most cases, you don't need to include it but some functions require you to (for example when arguments are optional or there is ambiguity over which argument is meant).

A additional benefit of using the pipe is that solving problems step-by-step is made easier.

Also see [Pipes in R Tutorial For Beginners](https://www.datacamp.com/community/tutorials/pipe-r-tutorial) - I think a better title for this article is "Pipes For Beginners to pipes but with intermediate R experience" because it goes further than you need to be able to understand the pipe 'well enough'. However, it still contains some useful examples.

## 2. Myoglobin concentration

The myoglobin concentration of skeletal muscle of three species of seal in grams per kilogram of muscle was determined and the data are given in [seal.txt](../data/seal.txt). Each row represents an individual seal. The first column gives the myoglobin concentration and the second column indicates species.
Can you write a single page word report which includes some summary statistics (for example the means and standard deviations) for each seal species and some figures? 
 
## 3. Understanding Paths

Do make sure you understand what a path is [What do we mean by paths?](https://youtu.be/oZblx36rqq0) 

## 4. Script example

This is an example of a well formatted analysis [script](../scripts/exampleprac2.R) appropriate for some of the work we have carried out in this practical. Read it, run it! 

## 5. DataCamp Assignments

If you have not completed the other DataCamp assignments, you should still be able to access and complete them but they'll be indicated as late.

# The Code files

These contain answers and code even though they do not appear on the webpage itself.

[Rmd file](02TestingDataTypesReadingInData.Rmd)
The Rmd file is the file I use to compile the practical. Rmd stands for R markdown allow R code and ordinary text to be inter weaved to produce well-formatted reports including webpages.

[Plain script file](../scripts/02TestingDataTypesReadingInData.R)
This is plain script (.R) version of the practical

# Script example
This is an example of a well formatted analysis script suitable for submitting with a research report

[Script example](../scripts/exampleprac2.R)

# Objectives from previous sessions

[Introduction to module and RStudio](01IntroductionToModuleAndRStudio.html)

* to explain why we need statistical tests and the logic of hypothesis testing (MLO 1)
* use the R command line as a calculator and to assign variables (MLO 3)
* create and use the basic data types in R (MLO 3)
* find their way around the RStudio windows (MLO 3)
* create, use and save a script file to run r commands (MLO 3)
* search and understand manual pages (MLO 3)


![](../../pics/17Cend.png)

# References
