library(lubridate)
library(zoo)
library(xts)
library(reshape2)
source("common/functions.r", chdir = TRUE)16 Best practice
On a twenty-line script, how you organise the work hardly matters. On anything larger it decides how many errors you make and how long they take to find. The larger the project, the more time it pays to spend on managing the workflow.
16.1 Data and libraries
Only the hashing libraries are new here.
import pandas as pd
import numpy as npusing DataFrames, CSV, Dates, Statistics16.2 Use functions and Lego
For a script of a few lines, statements executed in sequence are enough. Once a project grows to the size of most work in financial risk forecasting, put each repeated or distinct operation into a function. That separates data processing, modelling and reporting, so each can be run and tested on its own, and so you can call one without holding the rest of it in your head.
Think of code as Lego bricks. Each function does one job, and related functions belong in their own source file.
16.3 Workflow
In the type of work we are doing here, six steps make up the workflow. For each step, if the code involved is more than a handful of lines, keep it in a separate file:
- Download data;
- Process (clean) data;
- Identify the models to be applied to the data;
- Program up the code to load the data and run the models;
- Create output;
- Make a report.
flowchart LR A(Download data) --> B(Process data) B --> C(Modelling approach) C --> D(Write programs) D --> E(Make output) E --> F(Make report)
16.3.1 Prefer end-to-end runs without intermediate files
Write the workflow so that one call takes the raw data through to the finished report, without saving anything in between. A function like RunProject(Data=G20, Model=LeverageVol, Report=PDFSlides) loads the libraries, runs the analysis and generates the output in one step. Two things justify breaking that rule, and only two.
The first is downloading. API rate limits, network outages and data quality checks make it impractical to re-download on every run, so the download step is normally kept separate (see below). The second is expensive processing. Where a transformation takes minutes, saving the cleaned data once and reading it back is worth the intermediate file. Anything else, keep in the single run.
16.3.2 Downloading data
The most common source of data is outside data vendors such as EODHD, WRDS, Bloomberg or DBnomics (see Chapter 30). These vendors usually distribute data as CSV or JSON files. In some cases, we can, or need to, download this data on the command line or in a browser.
Download data through APIs when you can. An API lets programs talk to each other, cutting manual error and making updates easier. See the EODHD implementation in Section 30.0.1.1. EODHD, Bloomberg and DBnomics have an API interface. WRDS is typically accessed through its web query interface, although it also offers a Python API.
It is usually best to download data separately from the rest of the workflow. There are a few reasons why:
- Many data providers limit the amount and speed of data downloads;
- Reading a saved local file is much quicker than downloading again, especially for large datasets;
- A vendor can be offline;
- Your network access can be unreliable;
- A saved copy lets you check data quality with plots and tables before analysis, catching errors and omissions early.
The main exception is if we are doing real-time processing.
If using an API, save the data in the native binary format of the programming language to be used, like RData for R. Alternatively, save it as a CSV or Parquet file, which is especially useful if the data needs to be read by several different applications. For guidance on file formats, see Chapter 10 and Section 10.5.
16.3.2.1 The download function
Use a separate function, such as GetRawData(), that downloads data if it comes from an API or reads it in if it is in the local file, for example ReadRawData.r.
16.3.2.2 Hash
When code downloads vendor data, compare the new data with the saved copy. A hash — a digital fingerprint of the dataset — makes that easy.
library(digest)
old_hash = digest(old_data)
new_hash = digest(new_data)
if(old_hash != new_hash) {
# Data has changed, proceed with analysis
}Uses the digest package.
import hashlib
old_hash = hashlib.sha256(pd.util.hash_pandas_object(old_data).values).hexdigest()
new_hash = hashlib.sha256(pd.util.hash_pandas_object(new_data).values).hexdigest()
if old_hash != new_hash:
# Data has changed, proceed with analysisUses the hashlib module.
using SHA
old_hash = bytes2hex(sha256(sprint(CSV.write, old_data)))
new_hash = bytes2hex(sha256(sprint(CSV.write, new_data)))
if old_hash != new_hash
# Data has changed, proceed with analysis
endUses the SHA package.
This allows you to detect when data has been updated without manually comparing the entire dataset. The hashes compare an old and new version within the same language and run, and are not intended to match across languages.
16.3.3 Process data
The data you download usually needs to be in a useful form for analysis. Chapter 11 discusses the steps involved in transforming the sample data used here into a useful form.
16.3.3.1 Do not transform the data with Excel
Never transform data with Excel. The reasons are:
- Excel can mangle data without warning (see Section 10.1 for details);
- It is non-transparent and difficult to repeat. If you need to repeat the transformation — because data has been updated, for example — it is often impossible to know what was actually transformed;
- Every time you update your data, you have to repeat the Excel manipulation;
- By contrast, data manipulation in code is transparent and repeatable. You know exactly how data is transformed, and you can repeat the analysis every time you update your data.
The one exception is quick, throwaway exploration — eyeballing a CSV, checking a handful of values or sketching a chart for your own use. The rule applies to any data transformation step that feeds into your analysis. Those steps must live in code so they are transparent, repeatable and version-controlled.
16.3.3.2 The processing function
Use a separate function, such as Data=ProcessRawData(), that calls GetRawData(), for example functions.r.
Rules have exceptions. If processing is slow or the raw files are huge, split the job in two — process the raw data once, then save a cleaned file for later analysis.
16.3.4 Identify the models
The most important step is to decide on which model to apply to the data. While discussing model choice is outside the scope of these notes, we refer you to the Financial Risk Forecasting book and its slides.
16.3.5 Program
The code that runs the models will likely be the longest and change most frequently:
- Keep code modular by separating concerns into functions
- Use descriptive filenames, not
code-1.rorcode-monday.r - Avoid very long files — split them into logical units
- Track experiments systematically
The best solution is version control (Section 16.6) rather than ad-hoc file naming.
16.3.5.1 The modelling function
Use a separate function, such as Results=RunModels(options), that calls Data=ProcessRawData(), for example RunModels.r.
16.3.6 Create output
Once you have done the analysis, you need to create output, like tables and figures. Keep a separate file, such as MakeOutput.r, with one or more functions to make output, like MakeImportantTable(). Use one function per table or figure. You will often need to make different types of output — saving images as png and pdf and also automatically making a Word file with them.
Do not use screen grabs for figures. Export them from code.
16.3.6.1 The output functions
Keep a separate function for each output artefact, such as MakeTable(Results) or MakeFigure(Results), for example MakeOutput.r.
16.3.7 Make a report
The reporting step turns results into the tables, figures and documents you share, in whatever format suits the audience, ideally generated automatically from code. See Chapter 17.
16.4 Structuring projects
For the simplest project, you need only one R file, which contains all the code to process the data, run the models and make the output. Even then, keep all the code in functions.
For anything beyond the simplest project, use multiple files for the code — one for data processing, one to run the models and one for the reports. A separate file with common functions that are loaded into different parts of the code is also useful.
For larger projects, keep the code in multiple directories with several input files.
We have created an example project that conducts a small set of analyses presented in these notes. It is kept on GitHub (see Section 16.6). You can find it at https://github.com/Jon-Danielsson/Financial-Risk-Forecasting-Example-Project.
The directory structure below shows the S&P 500 index in RawData, the code to do all the analyses in Code, the Quarto files to generate the HTML, PDF, Word and PowerPoint reports in Output-code and finally, an example of the output in Output. This layout, with raw data, code and output in separate directories, suits a structured project like this one. For a single throwaway document, it is simpler to keep the report file alongside its data, as in Chapter 17.
├── Code
│ ├── FitModels.r
│ ├── ProcessRawData.r
│ ├── RunModels.r
│ └── libraries.r
├── Output
│ ├── pdf-presentation.pdf
│ ├── pdf-report.pdf
│ ├── powerpoint-presentation.pptx
│ ├── sp500-returns-sd.pdf
│ ├── sp500-returns-sd.png
│ ├── sp500-returns-sd.svg
│ └── word-report.docx
├── Output-code
│ ├── pdf-presentation.qmd
│ ├── pdf-report.qmd
│ ├── powerpoint-presentation.qmd
│ ├── presentation.qmd
│ ├── report.qmd
│ └── word-report.qmd
└── RawData
└── sp500.csv16.5 Use structured data
Projects become hard to manage when data, parameters and outputs are scattered across unrelated variables.
For all but the simplest projects, use structured data types to keep track of what we are using. R uses list(), Python uses dict and Julia uses Dict.
data = ProcessRawData()
paste(sort(names(data), method = "radix"), collapse = ", ")[1] "Price, Return, Ticker, UnAdjustedPrice, sp500, sp500tr"data = ProcessRawData()
", ".join(sorted(data.keys()))data = ProcessRawData()
join(sort(collect(keys(data))), ", ")The object data contains all the data we need. If we get new data, we include it in ProcessRawData(), and it is then available everywhere we call that function.
Similarly, if we have many input parameters, it is sensible to group them:
Parameters = list()
Parameters$Model = "GARCH11"
Parameters$WE = 1000
Parameters$probability = 0.01
Parameters$value = 1000Parameters = {
"Model": "GARCH11",
"WE": 1000,
"probability": 0.01,
"value": 1000
}Parameters = Dict(
"Model" => "GARCH11",
"WE" => 1000,
"probability" => 0.01,
"value" => 1000
)When we run a model, structure the workflow as:
Data = ProcessRawData()
Fit = RunModels(Data = Data, Model = "GARCH11")
VaR = RunVaR(Fit = Fit)
Report = MakeVaRReport(VaR = VaR)Data = ProcessRawData()
Fit = RunModels(Data=Data, Model="GARCH11")
VaR = RunVaR(Fit=Fit)
Report = MakeVaRReport(VaR=VaR)Data = ProcessRawData()
Fit = RunModels(Data=Data, Model="GARCH11")
VaR = RunVaR(Fit=Fit)
Report = MakeVaRReport(VaR=VaR)So, the object Report contains all the relevant information in one place.
16.6 Managing code versions — git
Version control solves the problem of tracking changes without creating files like version-1.r and version-2.r. Instead of manually naming versions, the system records a snapshot of your tracked files every time you commit.
The most popular version control system is git. A minimal workflow:
- Initialise a repository in your project folder
- Commit changes with descriptive messages as you work
- Review history or revert if needed
You can run git locally, within RStudio (File -> New project -> create a git repository) or with tools like GitHub Desktop. For collaboration or backup, host your repository on GitHub. Contributors can propose changes via pull requests, which the project owner reviews before merging.
Most professional environments expect familiarity with git. See the git documentation for further reading.
16.7 Reproducible environments
One problem that can emerge for long-duration projects is what happens when the libraries and language versions you are using are updated. This happens quite frequently and usually without any adverse consequences. Sometimes, however, new versions of libraries are incompatible with other libraries or change what they compute, so results differ.
This becomes a real problem when you need to rerun code years later. Academic papers submitted to journals and long-lived professional projects both face this challenge.
There are several ways to deal with this. Containers and virtual machines are different tools for isolating a runtime environment. A virtual machine emulates entire hardware and runs its own operating system kernel. A container shares the host’s kernel and packages only the application and its dependencies, which makes it lighter and faster to start.
Docker is the container platform most projects use. On macOS and Windows, Docker Desktop runs Linux containers inside a lightweight virtual machine, though the containers themselves remain containers. Setting it up takes real effort, which is the main reason to reach for a language-level environment manager instead.
A simpler way is to use language-specific environment managers:
Use renv (reproducible environments). This package records the exact versions of the packages you use so the project library can be recreated later. It records, but does not manage, the R version itself.
renv::init() # Initialise in your project
renv::snapshot() # Record current package versions
renv::restore() # Recreate the environment laterUse uv or venv with requirements.txt. The uv tool is a fast, modern package manager that handles virtual environments and dependencies.
uv init # Initialise a new project
uv add pandas numpy # Add packages (recorded in pyproject.toml)
uv sync # Recreate the environment laterAlternatively, with traditional venv, activate the environment before using pip:
python -m venv .venv
source .venv/bin/activate
pip freeze > requirements.txt
pip install -r requirements.txtUse Julia’s built-in Pkg environment system. Each project has a Project.toml listing its dependencies and a Manifest.toml recording the exact package versions.
using Pkg
Pkg.activate(".") # Activate project environment
Pkg.add("DataFrames") # Add packages (recorded in Project.toml)
Pkg.instantiate() # Recreate the environment laterThis adds another layer of complexity, and for most simple projects it may not be worth the effort. However, for complicated projects that will be in use for a long time, reproducible environments are helpful.
Record version information with the results. Save the run date, machine name and package versions alongside the output. See Chapter 26 for a practical example of saving results with metadata.