\[ % MathJax has no bm package; redefine \bm in terms of \boldsymbol, which it supports natively \newcommand{\bm}[1]{\boldsymbol{#1}} % --- Operators ----------------------------------------------------------- % Expectation operator \DeclareMathOperator{\E}{E} % Variance operator \DeclareMathOperator{\Var}{Var} % Covariance operator \DeclareMathOperator{\Cov}{Cov} % Correlation operator \DeclareMathOperator{\Corr}{Corr} % Rank operator (Spearman rank correlation) \DeclareMathOperator{\Rank}{Rank} % Skewness operator \DeclareMathOperator{\Skewness}{Skewness} % Kurtosis operator \DeclareMathOperator{\Kurtosis}{Kurtosis} % Difference/differential operator (upright d per ISO 80000-2) \newcommand{\Diff}{\mathrm{d}} % --- Risk Measures ------------------------------------------------------- % Value at Risk \DeclareMathOperator{\VaR}{VaR} % Expected Shortfall \DeclareMathOperator{\ES}{ES} % Marginal Value at Risk \DeclareMathOperator{\MVaR}{MVaR} % Component Value at Risk \DeclareMathOperator{\CompVaR}{CVaR} % Incremental Value at Risk \DeclareMathOperator{\IVaR}{IVaR} % Component Expected Shortfall \DeclareMathOperator{\CompES}{CES} % Weighted sensitivity for risk factor in FRTB SBM bucket \DeclareMathOperator{\WS}{WS} % Hedge-benefit ratio for FRTB DRC bucket \DeclareMathOperator{\HBR}{HBR} % Stress scenario risk measure for FRTB NMRF capital add-on \DeclareMathOperator{\SES}{SES} % Jump-to-default exposure for obligor (FRTB DRC) \DeclareMathOperator{\JTD}{JTD} % Loss given default for obligor (FRTB DRC) \DeclareMathOperator{\LGD}{LGD} % Default risk weight for obligor (FRTB DRC) \DeclareMathOperator{\RW}{RW} % ES ratio function \newcommand{\ESratio}{\lambda} % Arbitrary risk measure function (coherence axioms) \newcommand{\RiskMeasure}{\varphi} % Asset position (coherence axioms) \newcommand{\Asset}{A} % First named asset in examples \newcommand{\AssetA}{A} % Second named asset in examples \newcommand{\AssetB}{B} % Third named asset in examples \newcommand{\AssetC}{C} % Constant (risk measure axioms) \newcommand{\Constant}{c} % Bucket-level aggregate sensitivity in FRTB SBM \newcommand{\BucketAgg}{S} % Cross-bucket correlation in FRTB SBM \newcommand{\CrossBucketCorr}{\gamma} % Notional amount for an instrument (FRTB DRC and RRAO) \newcommand{\Notional}{\text{Notional}} % --- Distributions ------------------------------------------------------- % Binomial distribution \DeclareMathOperator{\Binomial}{Binomial} % Uniform distribution \DeclareMathOperator{\Uniform}{Uniform} % Normal distribution \newcommand{\NormalDist}{\mathcal{N}} % Student-t CDF \newcommand{\StudentCDF}{t} % Student-t PDF \newcommand{\StudentPDF}{f} % --- Returns ------------------------------------------------------------- % Price \newcommand{\Price}{P} % Simple (arithmetic) return \newcommand{\SimpleReturns}{R} % Log (compound) return \newcommand{\CompoundReturns}{y} % Matrix of compound returns \newcommand{\ReturnMatrix}{\bm{y}} % Dividend payment \newcommand{\dividend}{d} % --- Volatility ---------------------------------------------------------- % Realised variance \DeclareMathOperator{\RealisedVar}{RV} % Realised volatility (square root of realised variance) \DeclareMathOperator{\RealisedVol}{RVol} % Volatility \newcommand{\Vol}{\sigma} % GARCH intercept parameter \newcommand{\GARCHconst}{\omega} % ARCH coefficient (weight on past squared returns) \newcommand{\ARCHcoeff}{\alpha} % GARCH coefficient (weight on past variance) \newcommand{\GARCHcoeff}{\beta} % EWMA decay factor (typically 0.94) \newcommand{\EWMAdecay}{\lambda} % Standardized residual / error term \newcommand{\StdNormal}{\epsilon} % Leverage parameter in apARCH \newcommand{\APARCHleverage}{\zeta} % Power parameter in apARCH \newcommand{\APARCHpower}{\delta} % Degrees of freedom (Student-t) \newcommand{\DOF}{\nu} % GJR-GARCH leverage parameter \newcommand{\GJRleverage}{\gamma} % GARCH-X external regressor coefficient \newcommand{\RegressorCoeff}{\xi} % Mean (first moment) \newcommand{\Mean}{\mu} % Lag order in volatility models \newcommand{\Lag}{L} % Dummy variable (indicator in regressions) \newcommand{\DummyVar}{D} % --- Portfolio ----------------------------------------------------------- % Portfolio weight (scalar) \newcommand{\weight}{w} % Portfolio weight vector \newcommand{\weights}{\bm{w}} % Covariance matrix \newcommand{\CovMatrix}{\bm{\Sigma}} % Correlation matrix \newcommand{\CorrMatrix}{\bm{C}} % Correlation coefficient \newcommand{\correlation}{\rho} % Number of assets in portfolio \newcommand{\NumberAssets}{K} % Portfolio value \newcommand{\PortfolioValue}{\vartheta} % Asset beta (CAPM) \newcommand{\AssetBeta}{\beta} % Ledoit-Wolf shrinkage intensity \newcommand{\ShrinkageIntensity}{\delta} % Sample covariance matrix \newcommand{\SampleCov}{\bm{S}} % --- Time Series --------------------------------------------------------- % Sample size \newcommand{\SampleSize}{T} % Generic count (number of lags, payments, etc.) \newcommand{\Count}{N} % Estimation window length \newcommand{\EstWindow}{W_E} % Testing window length \newcommand{\TestWindow}{W_T} % Stress window length \newcommand{\StressWindow}{W_S} % AR coefficient \newcommand{\ARcoeff}{\phi} % MA coefficient \newcommand{\MAcoeff}{\psi} % --- Probability --------------------------------------------------------- % Probability level (e.g. 0.01 for 1% VaR) \newcommand{\probability}{p} % Cumulative sorted scenario weight \newcommand{\CumScenarioWeight}{\Omega} % p-quantile of the P and L distribution \newcommand{\Quantile}{q} % Quantile function \newcommand{\QuantileFunction}{Q} % Probability density function \newcommand{\PDF}{f} % Cumulative distribution function \newcommand{\CDF}{F} % Standard normal CDF \newcommand{\NormalCDF}{\Phi} % Inverse standard normal (quantile function) \newcommand{\NormalQuantile}{\Phi^{-1}} % Standard normal density function \newcommand{\NormalPDF}{\phi} % Likelihood function \newcommand{\lik}{\mathcal{L}} % Log-likelihood \newcommand{\LogLikelihood}{\ell} % Parameter vector (MLE estimation) \newcommand{\ParamSet}{\theta} % Parameter space \newcommand{\ParamSpace}{\Theta} % Fisher information matrix \newcommand{\FisherInfo}{\mathcal{I}} % Statistical power (Type II error rate) \newcommand{\StatPower}{\beta} % Significance level (CI coverage complement) \newcommand{\SignifLevel}{\gamma} % --- Options ------------------------------------------------------------- % Call option label \newcommand{\CallOption}{\text{call}} % Put option label \newcommand{\PutOption}{\text{put}} % Strike price \newcommand{\Strike}{X} % Risk-free interest rate \newcommand{\RiskFree}{r_f} % Option maturity (time to expiration) \newcommand{\OptionMaturity}{\tau} % Option delta \newcommand{\OptionDelta}{\Delta} % Option gamma \newcommand{\OptionGamma}{\Gamma} % Option vega \newcommand{\Vega}{\mathcal{V}} % Holding period \newcommand{\HoldingPeriod}{H} % Simulation profit/loss (signed; negative is a loss) \newcommand{\ProfitLoss}{\Pi} % Futures price \newcommand{\Futures}{F} % Hedge ratio \newcommand{\HedgeRatio}{h} % Units of basic asset held \newcommand{\StockHolding}{x^b} % Units of options held \newcommand{\OptionHolding}{x^o} % --- Interest Rates ------------------------------------------------------ % Bond convexity \newcommand{\Convexity}{C} % Interest rate / yield \newcommand{\Yields}{r} % Key rate change, basis points (scalar tenor component); used as \KeyRateChange_k or \KeyRateChange_{t,k}. EXCEPTION to the no-macro-to-macro-indirection rule, decided 2026-08-09: composed from \Yields on purpose, so a future change to the rate glyph propagates here without a follow-up edit. \Yields is defined earlier in generated output (same interest_rates category, TOML order), so expansion order is safe; verify this still holds if either entry is reordered. \newcommand{\KeyRateChange}{\Delta \Yields^{\mathrm{bp}}} % Key rate changes, basis points (vector); used bare or as \KeyRateChanges_t. EXCEPTION to the no-macro-to-macro-indirection rule, decided 2026-08-09: composed from \Yields on purpose, so a future change to the rate glyph propagates here without a follow-up edit. \bm{\Yields} depends on \bm/\boldsymbol correctly expanding a macro argument rather than a bare letter --- confirm this renders correctly the first time either consumer compiles it. \newcommand{\KeyRateChanges}{\Delta \bm{\Yields}^{\mathrm{bp}}} % Dollar value of a basis point \newcommand{\DV}{\text{DV01}} % Mean reversion speed (interest rate models) \newcommand{\MeanReversion}{\varkappa} % Modified duration \newcommand{\ModDur}{D} % Macaulay duration \newcommand{\MacDur}{D_{\text{Mac}}} % Effective duration \newcommand{\EffDur}{D_{\text{eff}}} % DV01 vector \newcommand{\DVvec}{\bm{d}} % Long-run mean rate (Vasicek/CIR/Hull-White) \newcommand{\LongRunRate}{\theta} % Hull-White time-varying drift \newcommand{\HullWhiteDrift}{\varphi} % Cash flow (coupon/principal payment) \newcommand{\CashFlow}{c} % --- Extreme Value ------------------------------------------------------- % Tail index (EVT, Pareto-type tails) \newcommand{\TailIndex}{\iota} % Shape parameter (xi = 1/iota) \newcommand{\ShapeParam}{\varsigma} % Threshold value (EVT) \newcommand{\Threshold}{u} % Extremal index (Leadbetter 1983) \newcommand{\ExtremalIndex}{\theta} % Multivariate extremal coefficient, theta in [1,d]. theta=1 complete tail dependence; theta=d asymptotic independence. Glyph collides with ExtremalIndex and ParamSet; disambiguated at source by macro name. \newcommand{\ExtremalCoef}{\theta} % GEV distribution function \newcommand{\GEV}{\mathcal{H}} % GPD distribution function \newcommand{\GPD}{\mathcal{G}} % GPD scale parameter \newcommand{\GPDscale}{\beta} % GPD scale parameter at a threshold u, distinct from the generic GPD scale \newcommand{\GPDscaleAtThreshold}{\beta_u} % Number of threshold exceedances (EVT) \newcommand{\TailCount}{C} % Sample maximum (block maxima, EVT) \newcommand{\SampleMax}{M} % Kurtosis value (realized, e.g. sample kurtosis) \newcommand{\KurtosisVal}{\kappa} % Pareto scaling constant (EVT) \newcommand{\ParetoConst}{\mathcal{A}} % Generic constant (EVT asymptotics) \newcommand{\RemainderConst}{\mathcal{C}} % Little-o asymptotic notation \newcommand{\LittleO}{o} % Moment order (EVT) \newcommand{\MomentOrder}{\mathscr{m}} % Doubled sample size (EVT block maxima) \newcommand{\DoubleSample}{D} % GEV normalizing location constant \newcommand{\GEVloc}{a} % GEV normalizing scale constant \newcommand{\GEVscale}{b} % Negative return (loss, EVT context) \newcommand{\NegativeReturn}{L} % --- Copulas ------------------------------------------------------------- % Copula function \newcommand{\Copula}{\mathcal{C}} % Copula density \newcommand{\CopulaDensity}{c} % Generator function (Archimedean copulas) \newcommand{\generator}{\varphi} % Copula dependence parameter \newcommand{\CopulaParam}{\theta} % Copula marginal parameter set \newcommand{\MarginalParamSet}{\eta} % Lower tail dependence coefficient \newcommand{\LowerTailDep}{\lambda_L} % Upper tail dependence coefficient \newcommand{\UpperTailDep}{\lambda_U} % Marginal CDF (second variable) \newcommand{\MarginalCDF}{G} % Joint density function \newcommand{\JointDensity}{h} % Marginal density function \newcommand{\MarginalDensity}{g} % Joint distribution function \newcommand{\JointCDF}{H} % Gaussian copula correlation parameter \newcommand{\CopulaCorr}{\rho} % --- Multivariate -------------------------------------------------------- % Diagonal matrix of conditional volatilities \newcommand{\DiagVolD}{\bm{D}} % Idiosyncratic variance matrix (diagonal, factor models) \newcommand{\IdioVarMatrix}{\bm{\Psi}} % Covariance matrix entry (two indices) \newcommand{\CovElement}{\sigma} % Auxiliary matrix in DCC dynamics \newcommand{\DCCauxQ}{\bm{Q}} % BEKK constant matrix \newcommand{\BEKKconst}{\bm{\Omega}} % BEKK ARCH parameter matrix \newcommand{\BEKKarch}{\bm{A}} % BEKK GARCH parameter matrix \newcommand{\BEKKgarch}{\bm{B}} % DCC rescaling diagonal matrix \newcommand{\DCCrescaleZ}{\bm{Z}} % DCC auxiliary matrix element \newcommand{\DCCelement}{q} % Factor loadings matrix (loadings/eigenvectors of the covariance matrix) \newcommand{\FactorLoadings}{\bm{\Lambda}} % Factor subscript label \newcommand{\factor}{\text{factor}} % PCA eigenvalue \newcommand{\Eigenvalue}{\lambda} % Number of factors in factor model \newcommand{\NumberFactors}{m} % Factor return vector \newcommand{\FactorReturn}{\bm{g}} % PCA factor score (scalar; not an eigenvector) \newcommand{\PCAFactor}{F} % BEKK cross-covariance coefficient \newcommand{\BEKKcross}{\delta} % Factor loading element \newcommand{\FactorLoading}{\Lambda} % DCC correlation persistence parameter \newcommand{\DCCxi}{\xi} % DCC news coefficient (weight on recent shocks) \newcommand{\DCCzeta}{\zeta} % --- Simulation ---------------------------------------------------------- % Number of simulation paths \newcommand{\NumberSims}{B} % Cholesky factor of covariance matrix \newcommand{\Cholesky}{\bm{L}} % Uniform random number \newcommand{\UniformDraw}{\tilde{u}} % --- Backtesting --------------------------------------------------------- % Violation ratio (observed/expected violations) \DeclareMathOperator{\ViolRatio}{VR} % Stressed Value at Risk \DeclareMathOperator{\StressedVaR}{SVaR} % QLIKE loss function for variance forecasts \DeclareMathOperator{\QLIKE}{QLIKE} % Mean squared error (forecast loss function) \DeclareMathOperator{\MSE}{MSE} % Mean absolute error (forecast loss function) \DeclareMathOperator{\MAE}{MAE} % Markov transition probability \newcommand{\TransProb}{\pi} % Exception indicator: 1 if y_t < -VaR_t \newcommand{\ExceptionInd}{\eta} % Exception count \newcommand{\ExceptionCount}{\upsilon} % Probability Integral Transform \newcommand{\ProbIntegralTransform}{\hat{u}} % Quantile score (pinball loss function) \newcommand{\QuantileScore}{\mathcal{S}} % Kolmogorov-Smirnov test statistic \newcommand{\KSstat}{D} % Test statistic (generic base letter) \newcommand{\TestStat}{J} % --- General ------------------------------------------------------------- % Regression intercept \newcommand{\RegressionIntercept}{a} % Regression slope \newcommand{\RegressionSlope}{\beta} % Risk factor \newcommand{\RiskFactor}{x} % Forecast horizon \newcommand{\ForecastHorizon}{h} % Block length (bootstrap) \newcommand{\BlockLength}{\ell} % Capital tau variant \newcommand{\DeliveryTime}{\Upsilon} % Today's calendar time in years (option pricing input; distinct from trading-date index t) \newcommand{\CalendarTime}{t^*} % Probability measure \newcommand{\ProbMeasure}{\mathbb{P}} % Risk-neutral measure \newcommand{\RiskNeutral}{\mathbb{Q}} % Pricing function (Black-Scholes, bond pricing, etc.) \newcommand{\PricingFn}{V} % Indicator function \newcommand{\Indicator}{\bm{1}} % Vector of ones \newcommand{\OnesVector}{\bm{1}} % Risk factor sensitivity (first-order P&L) \newcommand{\Sensitivity}{\delta} % Second-order sensitivity (convexity/gamma) \newcommand{\ConvexitySens}{\gamma} % Output floor percentage (Basel) \newcommand{\OutputFloor}{\alpha} % --- Subscript Labels ---------------------------------------------------- % Subscript label: annualized \newcommand{\Annual}{a} % Subscript label: implied \newcommand{\Implied}{I} % Subscript label: portfolio \newcommand{\Portfolio}{\pi} % --- Text Abbreviations -------------------------------------------- % S&P 500 index \newcommand{\SP}{\text{S\&P-500}} % Student-t distribution (text) \newcommand{\St}{\text{Student-t}} % Geopolitical Risk index (Caldara and Iacoviello) \newcommand{\GPR}{\text{GPR}} \]

10  File formats

File formats look mundane. They are not. In financial risk work, the wrong format corrupts data, slows analysis and makes collaboration harder.

10.1 Excel

The most common file format is the Excel spreadsheet. However, we do not recommend storing data in Excel for quantitative work because Excel can silently modify data:

  • Converting entries to dates (e.g., gene names like “SEPT1” become September 1)
  • Losing significant digits in long numbers
  • Mishandling decimal separators (comma vs decimal point)

For reliable data storage, use CSV or binary formats instead.

We know someone whose PhD was delayed by a year because he used Excel for his data, did not keep a backup of the original data he had collected by hand, and then Excel mangled the data to the point that it became useless.

Excel still has a place. We often export results to Excel for further analysis or to share them with people who prefer it. Many vendors and collaborators distribute data in Excel files, and in those cases, it is usually best to have your software read the data directly from those files instead of converting them first.

10.2 CSV

The best format for data is generally CSV. It is a simple text file that can be edited in any text editor. Any software package, including Excel, can read it, so it is universally accessible. If there is a problem with the data, it is easy to inspect since CSV files are pure text files that can be directly opened in any text editor. Below is the beginning of one of the files we will work with.

ticker,date,Close,Adjusted_close
MCD,20030102,16.55,9.6942
MCD,20030103,16.12,9.4423

The first row contains the names of the columns, and then we have one row per day.

CSV has drawbacks. The files are uncompressed, so they can become large. Some software also truncates floating point numbers when saving them.

10.2.1 Things to look out for

Check the documentation. CSV readers and writers have defaults that change the data. For example, we need to check:

  1. Whether the file has a header row (the first row containing column titles) or not.
  2. Whether the separator is a comma (,), a semicolon (;), a tab ( or another symbol.
  3. Whether the decimal symbol is a comma (,) rather than a decimal point (.).

10.2.2 Compressed CSV files

Large CSV files become unwieldy fast. In one of our projects they take more than 3 TB. Compress them. The files shrink by more than 90%.

Base R reads compressed CSV files directly, and can write them through an explicit compressed connection. read.csv("data.csv.gz") needs no extra package, though it is slow on large files, and write.csv(x, gzfile("data.csv.gz")) writes a compressed file the same way. data.table and readr are faster and more convenient for both, as discussed in Section 7.4, for example:

data.table::fread("data.csv.gz")

readr::read_csv("data.csv.gz") works the same way.

10.3 Parquet

The best alternative file format is Parquet. It compresses the data, and preserves the full floating point precision. Parquet files are also easy to exchange with other software packages. The disadvantage is that because it comes in a common binary form, it is hard to inspect the files and edit them.

10.4 JSON

One of the most popular file formats is JSON. You can see an example on the EODHD website.

[
 {
 "ticker": "MCD",
 "date": "2024-04-08",
 "open": 267.24,
 "high": 268.69,
 "low": 265.87,
 "close": 267.56,
 "adjusted_close": 267.56,
 "volume": 3753200
 },
 {
 "ticker": "MCD",
 "date": "2024-04-09",
 "open": 266.22,
 "high": 269.67,
 "low": 265.5,
 "close": 269.44,
 "adjusted_close": 269.44,
 "volume": 3934900
 }
]

The square brackets [] indicate the beginning and end of the data, the curly brackets {} beginning and end of each record, and then each observation has a label. The labels make JSON records self-describing, which reduces errors when exchanging data between systems, and a lot of data vendors deliver data by default in the JSON format. There are better choices for simply storing data.

10.5 Binary formats

All three languages we have discussed here have their own binary data format: RData for R, pickle for Python and Serialization for Julia, saved with the .jls extension used throughout this book. JLD2 is a third-party alternative.

Saving data in such a binary format has many advantages. It allows us to save even complicated objects and read them in exactly the same way later, something usually not possible with other formats, and binary files load and save quickly. RData is compressed by default, while pickle and JLD2 need compression to be requested explicitly. The downside is that it may be difficult to share such files between different programming languages.

Only load binary files you created yourself or that come from a source you trust. Unpickling data from an untrusted source can execute arbitrary code, and the same caution applies to loading any language-native object of unknown origin.

In Chapter 11, you will see how we load CSV data.

10.6 DuckDB

DuckDB is an embedded SQL database that runs inside your R, Python or Julia process without a separate server. It handles analytical queries — aggregations, joins and window functions — over local data without the overhead of setting up a traditional database. It reads directly from Parquet, CSV and other formats, so you can query large files without first loading them into memory.

We generally keep all our data in DuckDB. We use DuckDB and Parquet files for the data we make for extremerisk.org since we use R, Python and Julia for that site.

As a rule of thumb, use CSV for small files you want to inspect, edit or share across any software. Use Parquet once files get large or you work across languages in one common format. Use a trusted language-native binary format, RData, pickle or Serialization, only for objects you created yourself, when speed matters more than portability. Use DuckDB when you need to query data too large to load into memory, in any of these formats, without setting up a separate database server.