# These are equivalent
x = 3
y <- 47 Variables in R
Risk models are built out of vectors, matrices and data frames, and most of the errors in them come from the wrong object type rather than the wrong formula. This chapter covers the R structures the rest of the book uses — assignment, the use of vectors, matrices, lists and data frames. The discussion is framed with reference to portfolio-style examples such as asset weights and prices.
7.1 Assignment
In R, we use the equal sign, =, to assign a value to a variable. The variable’s name is on the left, and the value to be stored is on the right.
R also supports the left arrow operator <- for assignment, which is often preferred by R programmers and appears in many R style guides. Both operators work identically for most purposes:
We use = throughout this notebook because it aligns with Python and Julia and is more familiar to readers coming from those languages.
x = 3
y = 4
x == y # Two equal signs test for equality[1] FALSENote that two equal signs, ==, are used to test for equality, not assignment.
7.2 R data structures
R provides several built-in data structures for organising and manipulating data.
7.2.1 Vectors
R comes with vectors. Note that R does not know if they are column vectors or row vectors, which becomes important in matrix algebra.
v = vector(length=4)
v
v[] = NA
v
v[2:3] = 2
v
v=seq(1,5)
v
v=seq(-1,2,by=0.5)
v
v=c(1,3,7,3,0.4)*3
v[1] FALSE FALSE FALSE FALSE
[1] NA NA NA NA
[1] NA 2 2 NA
[1] 1 2 3 4 5
[1] -1.0 -0.5 0.0 0.5 1.0 1.5 2.0
[1] 3.0 9.0 21.0 9.0 1.2One way to create vectors is c().
x=c(1,4,0.9,"ss")
x[1] "1" "4" "0.9" "ss"Here, we used both numbers and strings, and all became a string.
x=c(1,4,0.9)
x[1] 1.0 4.0 0.9While here, we only have numbers, and they stay numbers.
7.2.2 Matrices
R can create two-dimensional matrices and higher-dimensional arrays. We usually only work with matrices, but we will encounter three-dimensional arrays in the multivariate volatility models.
Matrices can have column names, which can be quite useful.
m=matrix(ncol=1,nrow=3)
m
m=matrix(ncol=2,nrow=3)
m
m=matrix(3,ncol=2,nrow=3)
m [,1]
[1,] NA
[2,] NA
[3,] NA
[,1] [,2]
[1,] NA NA
[2,] NA NA
[3,] NA NA
[,1] [,2]
[1,] 3 3
[2,] 3 3
[3,] 3 3We often build matrices with cbind and rbind, or add rows and columns with the same functions.
v=c(1,3,7,3,0.4)*3
m=cbind(v,v)
m
m=rbind(v,v)
m v v
[1,] 3.0 3.0
[2,] 9.0 9.0
[3,] 21.0 21.0
[4,] 9.0 9.0
[5,] 1.2 1.2
[,1] [,2] [,3] [,4] [,5]
v 3 9 21 9 1.2
v 3 9 21 9 1.2We can access individual elements of matrices and vectors.
m[1,2]
m[,2]
m[2,]
m[1,3:5]
v[2:3]v
9
v v
9 9
[1] 3.0 9.0 21.0 9.0 1.2
[1] 21.0 9.0 1.2
[1] 9 21We can name the columns with colnames().
m=cbind(rnorm(4),rnorm(4))
m
colnames(m)=c("Stock A","Stock B")
m [,1] [,2]
[1,] -0.2965856 -0.5465354
[2,] 1.9564863 -0.2292965
[3,] -1.3030637 -2.3270109
[4,] -0.5214048 -0.6089451
Stock A Stock B
[1,] -0.2965856 -0.5465354
[2,] 1.9564863 -0.2292965
[3,] -1.3030637 -2.3270109
[4,] -0.5214048 -0.60894517.2.3 Lists
We often need to keep track of many variables that belong together, and the R list object is useful. It allows us to group multiple variables in one list.
l=list()
l$a=2
l$b= "R is great."
l=list(l=c(2,3),b="Risk")
w=list()
w$q= "my list"
w$l = l
w$df=data.frame(X1 = c(1, 2), X2 = c("VaR", "ES"))
w$q
[1] "my list"
$l
$l$l
[1] 2 3
$l$b
[1] "Risk"
$df
X1 X2
1 1 VaR
2 2 ESWe can find out what is in a list:
names(w)[1] "q" "l" "df"We can access individual elements:
w$l$l
[1] 2 3
$b
[1] "Risk"We make extensive use of lists in these notes.
7.3 Data frames
A data frame is a two-dimensional structure in which each column contains values of one variable, and each row contains one set of values from each column. It is the most common way of storing data in R and the one we will use the most.
One of the main advantages of a data frame over a matrix is that each column can have a different data type. For example, you can have one column with numbers, one with text, one with dates and one with logicals, whereas a matrix limits you to only one data type. Data frames also carry richer metadata than matrices. Keep in mind that a data frame needs all its columns to be of the same length.
Matrices and data frames each have their place, and we usually need both. Some R functions, especially those belonging to old libraries, only accept data frames and not matrices, or vice versa, even where either would do.
7.3.1 Iteratively creating data frames
Inserting values one by one into a data frame, for example with df[3,4] = 42, is expensive because each resize requires memory reallocation. For larger operations, such as the iterative fill in Chapter 26, pre-allocate a matrix, fill it, then convert to a data frame.
7.3.2 Accessing the data from columns
We can access data from columns by number, like df[,3], but since all the columns have names, it is usually much better to access them by column name, like df$returns.
7.3.3 Creating a data frame from scratch
There are several different ways to create a data frame. One is loading from a file, which we will do later. Alternatively, we could create a data frame from a list of vectors. This can easily be done with the data.frame() function:
df = data.frame(col1 = 1:3,
col2 = c("A", "B", "C"),
col3 = c(TRUE, TRUE, FALSE),
col4 = c(1.0, 2.2, 3.3))
print(df) col1 col2 col3 col4
1 1 A TRUE 1.0
2 2 B TRUE 2.2
3 3 C FALSE 3.3You have to specify the name of each column and what goes inside it. Note that all vectors need to be the same length. We can now check the structure:
str(df) # display the structure of the data frame
dim(df) # dimension
colnames(df) # column names'data.frame': 3 obs. of 4 variables:
$ col1: int 1 2 3
$ col2: chr "A" "B" "C"
$ col3: logi TRUE TRUE FALSE
$ col4: num 1 2.2 3.3
[1] 3 4
[1] "col1" "col2" "col3" "col4"7.3.4 Transforming a different object into a data frame
Sometimes you need both formats. Use a matrix for linear algebra, then convert the result to a data frame when you want named columns or mixed types. You can switch from matrix to data frame using as.data.frame() (and from data frame to matrix with as.matrix(), but remember all columns need the same data type).
For example, consider the matrix:
myMatrix = matrix(1:10, nrow = 5, ncol = 2, byrow = TRUE)
class(myMatrix)
myMatrix[1] "matrix" "array"
[,1] [,2]
[1,] 1 2
[2,] 3 4
[3,] 5 6
[4,] 7 8
[5,] 9 10We can now transform it into a data frame:
df = as.data.frame(myMatrix)
class(df)
df
str(df)[1] "data.frame"
V1 V2
1 1 2
2 3 4
3 5 6
4 7 8
5 9 10
'data.frame': 5 obs. of 2 variables:
$ V1: int 1 3 5 7 9
$ V2: int 2 4 6 8 10We can change the column names:
colnames(df) = c("Odd", "Even")
df Odd Even
1 1 2
2 3 4
3 5 6
4 7 8
5 9 107.4 Alternatives to data frames
The R data frames suffer from having been proposed decades ago and, therefore, lack some useful features one might expect, and they can be slow. In response, there are two alternatives, each with its own pros and cons.
We discuss one case where one of those is needed in Section 10.2.2, which deals with compressed CSV files.
7.4.1 data.table
The data.table class is designed for performance and features. It is by far the fastest when using large datasets, but it also has useful features built into it that facilitate data work. Outside this book, data.table is often the better choice for large datasets.
7.4.2 Tidy
The other main alternative is the tidyverse, a collection of packages built for data wrangling.
7.4.3 Data frames, data.table or tidy data?
The choice turns on one question. Do you care most about simplicity, speed or data wrangling? Data frames have the advantage of being built into R. They are relatively simple, and for basic calculations that do not need a lot of performance, they are sufficient.
data.table has the best performance, so if one has large datasets or is performing complicated data science operations on data, it is generally the best choice.
The tidyverse has the richest and most coherent way of doing data wrangling, that is, performing complicated operations on data. For many users in data science, the tidyverse is the only thing they use in R. Consequently, the tidyverse is the best choice for applications that are mostly in data science unless one needs performance.
If you search for opinions on data frames vs. data.table vs. tidy you often find strong views in favour of one of these. While these perspectives can be informative, all three have their own pros and cons. Choose whichever tool best fits your workflow and performance requirements.
We use data frames in this book because we want to keep the number of packages to a minimum and because they are sufficient for our purposes.
7.5 R-specific concepts
This section covers concepts and behaviours that are specific to R and important to understand for effective programming.
7.5.1 Missing values NA
We use a special value NA to indicate a missing value, i.e. we do not know what the value is. This becomes useful in backtesting in Chapter 26.
a=NA
a[1] NANA is distinct from NaN, which indicates an undefined numerical result, such as 0/0, rather than a missing value. The predicate is.nan() distinguishes the two, while is.na() also returns TRUE for NaN.
b=0/0
is.na(a)
is.nan(a)
is.na(b)
is.nan(b)[1] TRUE
[1] FALSE
[1] TRUE
[1] TRUE7.5.2 NULL
R is an old language that has evolved erratically over time. This means it has some features that can lead to difficult bugs. One is a variable type called NULL, which means nothing. While it can be useful, the problem is that NULL is used inconsistently and can cause unexpected behaviour.
1+What # variable What is not definedError: object 'What' not foundThat error makes sense. Now consider a list.
l=list()
l$WhatNULLThe variable What does not exist in the list l, but we can access it by l$What.
l$What+3numeric(0)Arithmetic on it then fails silently.
When we need to delete columns from a data frame or an element from a list, we assign NULL to it.
df$DeleteMe = NULL7.5.3 Scope and global assignment <<-
Variables in every programming language have a scope, meaning which part of the code can see them. For example, if you define a variable directly in R, it can be seen everywhere in your code, but if you define it inside a function, it is only visible within the function. The former is a global variable, while the second is a local variable.
Most programmers avoid global variables for good reason. They can lead to difficulty in finding bugs, which is a particular problem in R because it has a rather unfortunate way of dealing with missing variables.
Sometimes, global variables can only be avoided by making the code more complex, so it is a tricky trade-off. We use <<- to assign outside the function, which for functions defined at the top level means the global environment.
GlobalVariable <<- 123.456The objections to global variables are sensible, but they are not absolute. If a global solves a problem cleanly and safely, use it.
7.6 R programming essentials
7.6.1 Special characters
R uses both single quotes and double quotes for strings, and you can use either. That is particularly useful if you have to include a quotation mark inside a string, like
s= 'This is a quote character" in the middle of a string\n'
cat(s)This is a quote character" in the middle of a stringThe special character \n means a new line, quite handy for printing.
7.6.2 Printing: cat() vs. print()
The two printing functions we use are cat() and print(). print() displays an object on the console using its class’s default representation, while cat() concatenates and writes text, either to the console or to a file. To format numbers into strings, we use sprintf(), and a new line within a string is \n.
x=10
y=1.234
w= "risk"
cat(x)
cat('\n',x,w,'\n')
cat(sprintf("Important number for %s is x = %d\n", w, x))
s=sprintf("The return is %.1f%%",100*y)
cat(s,"\n")10
10 risk
Important number for risk is x = 10
The return is 123.4%print(x)
cat(sprintf("This is the answer. x = %d, and y = %s.\n", x, y))[1] 10
This is the answer. x = 10, and y = 1.234.print() is for console display. To send output to a file instead, use cat() with its file argument, capture.output(), or sink().
tmp=tempfile()
cat(sprintf("x = %d, y = %.3f\n", x, y), file = tmp)
capture.output(print(x), file = tmp, append = TRUE)
readLines(tmp)[1] "x = 10, y = 1.234" "[1] 10"cat() writes the formatted line directly to tmp, and capture.output() appends what print(x) would have shown on the console. sink() works differently, redirecting all console output for as long as it is open, which suits capturing a long block of mixed output rather than a single call.
7.6.3 Some useful functions
R has many functions. Below is a list of some of the most widely used in this book.
head: return the first part of an objecttail: return the last part of an objectcbind: combine by columnrbind: combine by rowcat: concatenate and printprint: print valuespasteandpaste0: concatenate stringssprintf: format strings with placeholders
7.7 Packages/libraries
R comes with a lot of functionality, as standard, but its strength lies in all the packages available for it. For statistics, the ecosystem is much richer than that of any other language. Some of these packages come with R, but most have to be downloaded separately, either using the install.packages() command or a menu in RStudio.
We load the packages using the library() command. Some of them come with annoying start-up messages, which can be suppressed by the suppressPackageStartupMessages() command.
The best practice is to load all the packages used in the code file at the top.
7.8 Matrix algebra
When dealing with vectors and matrices, * is element-by-element multiplication, while %*% is matrix multiplication. This becomes important when dealing with portfolios. Note that R vectors only have one dimension. They are not row or column vectors.
weight = c(0.3,0.7)
prices=cbind(runif(5)*100,runif(5)*100)
cat("Weights:\n")
print(weight)
cat("\nPrices:\n")
print(prices)Weights:
[1] 0.3 0.7
Prices:
[,1] [,2]
[1,] 59.471277 52.591376
[2,] 55.338402 66.145793
[3,] 25.646952 42.645443
[4,] 61.041902 33.986240
[5,] 6.562169 7.776175weight * prices # element-by-element multiplication [,1] [,2]
[1,] 17.841383 36.813963
[2,] 38.736881 19.843738
[3,] 7.694085 29.851810
[4,] 42.729332 10.195872
[5,] 1.968651 5.443323Note that R recycles the length-2 weight vector in column-major order across the whole matrix, not restarting at each column boundary. With 5 rows, column 1 receives 0.3, 0.7, 0.3, 0.7, 0.3. Recycling continues where it left off, so column 2 starts at 0.7, giving 0.7, 0.3, 0.7, 0.3, 0.7. This is because R stores matrices column by column internally. When the recycled vector length does not divide evenly into the number of matrix elements, R issues a warning, but otherwise the recycling is silent.
This recycling result is an R-specific footgun rather than the column weighting a portfolio calculation usually wants. The Python and Julia versions of this example scale each column of prices by the corresponding weight entry. To get that column weighting in R, use sweep():
sweep(prices, 2, weight, "*") # scale column 1 by weight[1], column 2 by weight[2] [,1] [,2]
[1,] 17.841383 36.813963
[2,] 16.601521 46.302055
[3,] 7.694085 29.851810
[4,] 18.312571 23.790368
[5,] 1.968651 5.443323weight %*% prices # matrix multiplicationError: non-conformable argumentsweight %*% t(prices) # matrix multiplication [,1] [,2] [,3] [,4] [,5]
[1,] 54.65535 62.90358 37.5459 42.10294 7.411974prices %*% weight # matrix multiplication [,1]
[1,] 54.655346
[2,] 62.903576
[3,] 37.545896
[4,] 42.102938
[5,] 7.4119747.9 Source files — source('functions.r')
To include other R files in some R code, use source('file.r').
Some of the code we develop below can be reused later. For that reason, we collect all the useful functions into an R source file called functions.r, kept at common/functions.r. _quarto.yml sets execute-dir: project, so every chapter’s R code executes with the project root as its working directory, and any chapter loads the shared file with source("common/functions.r", chdir = TRUE). The chdir argument switches to the file’s own folder only for the duration of sourcing it, protecting the call if the execution directory setting ever changes.
That is the R material the rest of the book relies on. The Python and Julia chapters that follow cover the same ground in those languages, so skip them unless you work in one. The book continues at Chapter 10.