{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Autoregressions\n",
    "\n",
    "This notebook introduces autoregression modeling using the `AutoReg` model. It also covers aspects of `ar_select_order` assists in selecting models that minimize an information criteria such as the AIC. \n",
    "An autoregressive model has dynamics given by \n",
    "\n",
    "$$ y_t = \\delta + \\phi_1 y_{t-1} + \\ldots + \\phi_p y_{t-p} + \\epsilon_t. $$\n",
    "\n",
    "`AutoReg` also permits models with:\n",
    "\n",
    "* Deterministic terms (`trend`)\n",
    "  * `n`: No deterministic term \n",
    "  * `c`: Constant (default)\n",
    "  * `ct`: Constant and time trend\n",
    "  * `t`: Time trend only\n",
    "* Seasonal dummies (`seasonal`)\n",
    "  * `True` includes $s-1$ dummies where $s$ is the period of the time series (e.g., 12 for monthly)\n",
    "* Custom deterministic terms (`deterministic`)\n",
    "  * Accepts a `DeterministicProcess`\n",
    "* Exogenous variables (`exog`)\n",
    "  * A `DataFrame` or `array` of exogenous variables to include in the model\n",
    "* Omission of selected lags (`lags`)\n",
    "  * If `lags` is an iterable of integers, then only these are included in the model.\n",
    "\n",
    "The complete specification is\n",
    "\n",
    "$$ y_t = \\delta_0 + \\delta_1 t + \\phi_1 y_{t-1} + \\ldots + \\phi_p y_{t-p} + \\sum_{i=1}^{s-1} \\gamma_i d_i + \\sum_{j=1}^{m} \\kappa_j x_{t,j} + \\epsilon_t. $$\n",
    "\n",
    "where:\n",
    "\n",
    "* $d_i$ is a seasonal dummy that is 1 if $mod(t, period) = i$. Period 0 is excluded if the model contains a constant (`c` is in `trend`).\n",
    "* $t$ is a time trend ($1,2,\\ldots$) that starts with 1 in the first observation.\n",
    "* $x_{t,j}$ are exogenous regressors.  **Note** these are time-aligned to the left-hand-side variable when defining a model.\n",
    "* $\\epsilon_t$ is assumed to be a white noise process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This first cell imports standard packages and sets plots to appear inline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import pandas_datareader as pdr\n",
    "import seaborn as sns\n",
    "from statsmodels.tsa.api import acf, graphics, pacf\n",
    "from statsmodels.tsa.ar_model import AutoReg, ar_select_order"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This cell sets the plotting style, registers pandas date converters for matplotlib, and sets the default figure size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.set_style(\"darkgrid\")\n",
    "pd.plotting.register_matplotlib_converters()\n",
    "# Default figure size\n",
    "sns.mpl.rc(\"figure\", figsize=(16, 6))\n",
    "sns.mpl.rc(\"font\", size=14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first set of examples uses the month-over-month growth rate in U.S. Housing starts that has not been seasonally adjusted. The seasonality is evident by the regular pattern of peaks and troughs. We set the frequency for the time series to \"MS\" (month-start) to avoid warnings when using `AutoReg`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "data = pdr.get_data_fred(\"HOUSTNSA\", \"1959-01-01\", \"2019-06-01\")\n",
    "housing = data.HOUSTNSA.pct_change().dropna()\n",
    "# Scale by 100 to get percentages\n",
    "housing = 100 * housing.asfreq(\"MS\")\n",
    "fig, ax = plt.subplots()\n",
    "ax = housing.plot(ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can start with an AR(3).  While this is not a good model for this data, it demonstrates the basic use of the API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mod = AutoReg(housing, 3, old_names=False)\n",
    "res = mod.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`AutoReg` supports the same covariance estimators as `OLS`.  Below, we use `cov_type=\"HC0\"`, which is White's covariance estimator. While the parameter estimates are the same, all of the quantities that depend on the standard error change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res = mod.fit(cov_type=\"HC0\")\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sel = ar_select_order(housing, 13, old_names=False)\n",
    "sel.ar_lags\n",
    "res = sel.model.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`plot_predict` visualizes forecasts.  Here we produce a large number of forecasts which show the string seasonality captured by the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = res.plot_predict(720, 840)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`plot_diagnositcs` indicates that the model captures the key features in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(16, 9))\n",
    "fig = res.plot_diagnostics(fig=fig, lags=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seasonal Dummies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`AutoReg` supports seasonal dummies which are an alternative way to model seasonality.  Including the dummies shortens the dynamics to only an AR(2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sel = ar_select_order(housing, 13, seasonal=True, old_names=False)\n",
    "sel.ar_lags\n",
    "res = sel.model.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The seasonal dummies are obvious in the forecasts which has a non-trivial seasonal component in all periods 10 years in to the future."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = res.plot_predict(720, 840)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(16, 9))\n",
    "fig = res.plot_diagnostics(lags=30, fig=fig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seasonal Dynamics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While `AutoReg` does not directly support Seasonal components since it uses OLS to estimate parameters, it is possible to capture seasonal dynamics using an over-parametrized Seasonal AR that does not impose the restrictions in the Seasonal AR. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "yoy_housing = data.HOUSTNSA.pct_change(12).resample(\"MS\").last().dropna()\n",
    "_, ax = plt.subplots()\n",
    "ax = yoy_housing.plot(ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start by selecting a model using the simple method that only chooses the maximum lag.  All lower lags are automatically included. The maximum lag to check is set to 13 since this allows the model to next a Seasonal AR that has both a short-run AR(1) component and a Seasonal AR(1) component, so that\n",
    "\n",
    "$$ (1-\\phi_s L^{12})(1-\\phi_1 L)y_t = \\epsilon_t $$\n",
    "which becomes\n",
    "$$ y_t = \\phi_1 y_{t-1} +\\phi_s Y_{t-12} - \\phi_1\\phi_s Y_{t-13} + \\epsilon_t $$\n",
    "\n",
    "when expanded. `AutoReg` does not enforce the structure, but can estimate the nesting model \n",
    "\n",
    "$$ y_t = \\phi_1 y_{t-1} +\\phi_{12} Y_{t-12} - \\phi_{13} Y_{t-13} + \\epsilon_t. $$\n",
    "\n",
    "We see that all 13 lags are selected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sel = ar_select_order(yoy_housing, 13, old_names=False)\n",
    "sel.ar_lags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems unlikely that all 13 lags are required.  We can set `glob=True` to search all $2^{13}$ models that include up to 13 lags.\n",
    "\n",
    "Here we see that the first three are selected, as is the 7th, and finally the 12th and 13th are selected.  This is superficially similar to the structure described above.\n",
    "\n",
    "After fitting the model, we take a look at the diagnostic plots that indicate that this specification appears to be adequate to capture the dynamics in the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sel = ar_select_order(yoy_housing, 13, glob=True, old_names=False)\n",
    "sel.ar_lags\n",
    "res = sel.model.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(16, 9))\n",
    "fig = res.plot_diagnostics(fig=fig, lags=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also include seasonal dummies.  These are all insignificant since the model is using year-over-year changes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sel = ar_select_order(yoy_housing, 13, glob=True, seasonal=True, old_names=False)\n",
    "sel.ar_lags\n",
    "res = sel.model.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "source": [
    "## Industrial Production\n",
    "\n",
    "We will use the industrial production index data to examine forecasting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "data = pdr.get_data_fred(\"INDPRO\", \"1959-01-01\", \"2019-06-01\")\n",
    "ind_prod = data.INDPRO.pct_change(12).dropna().asfreq(\"MS\")\n",
    "_, ax = plt.subplots(figsize=(16, 9))\n",
    "ind_prod.plot(ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will start by selecting a model using up to 12 lags.  An AR(13) minimizes the BIC criteria even though many coefficients are insignificant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sel = ar_select_order(ind_prod, 13, \"bic\", old_names=False)\n",
    "res = sel.model.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use a global search which allows longer lags to enter if needed without requiring the shorter lags. Here we see many lags dropped. The model indicates there may be some seasonality in the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "sel = ar_select_order(ind_prod, 13, \"bic\", glob=True, old_names=False)\n",
    "sel.ar_lags\n",
    "res_glob = sel.model.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`plot_predict` can be used to produce forecast plots along with confidence intervals. Here we produce forecasts starting at the last observation and continuing for 18 months."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "ind_prod.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fig = res_glob.plot_predict(start=714, end=732)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The forecasts from the full model and the restricted model are very similar. I also include an AR(5) which has very different dynamics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "res_ar5 = AutoReg(ind_prod, 5, old_names=False).fit()\n",
    "predictions = pd.DataFrame(\n",
    "    {\n",
    "        \"AR(5)\": res_ar5.predict(start=714, end=726),\n",
    "        \"AR(13)\": res.predict(start=714, end=726),\n",
    "        \"Restr. AR(13)\": res_glob.predict(start=714, end=726),\n",
    "    }\n",
    ")\n",
    "_, ax = plt.subplots()\n",
    "ax = predictions.plot(ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The diagnostics indicate the model captures most of the the dynamics in the data. The ACF shows a patters at the seasonal frequency and so a more complete seasonal model (`SARIMAX`) may be needed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": false,
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "fig = plt.figure(figsize=(16, 9))\n",
    "fig = res_glob.plot_diagnostics(fig=fig, lags=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Forecasting\n",
    "\n",
    "Forecasts are produced using the `predict` method from a results instance. The default produces static forecasts which are one-step forecasts. Producing multi-step forecasts requires using `dynamic=True`.  \n",
    "\n",
    "In this next cell, we produce 12-step-heard forecasts for the final 24 periods in the sample.  This requires a loop.\n",
    "\n",
    "**Note**: These are technically in-sample since the data we are forecasting was used to estimate parameters. Producing OOS forecasts requires two models.  The first must exclude the OOS period.  The second uses the `predict` method from the full-sample model with the parameters from the shorter sample model that excluded the OOS period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "start = ind_prod.index[-24]\n",
    "forecast_index = pd.date_range(start, freq=ind_prod.index.freq, periods=36)\n",
    "cols = [\"-\".join(str(val) for val in (idx.year, idx.month)) for idx in forecast_index]\n",
    "forecasts = pd.DataFrame(index=forecast_index, columns=cols)\n",
    "for i in range(1, 24):\n",
    "    fcast = res_glob.predict(\n",
    "        start=forecast_index[i], end=forecast_index[i + 12], dynamic=True\n",
    "    )\n",
    "    forecasts.loc[fcast.index, cols[i]] = fcast\n",
    "_, ax = plt.subplots(figsize=(16, 10))\n",
    "ind_prod.iloc[-24:].plot(ax=ax, color=\"black\", linestyle=\"--\")\n",
    "ax = forecasts.plot(ax=ax)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Comparing to SARIMAX\n",
    "\n",
    "`SARIMAX` is an implementation of a Seasonal Autoregressive Integrated Moving Average with eXogenous regressors model.  It supports:\n",
    "\n",
    "* Specification of seasonal and nonseasonal AR and MA components\n",
    "* Inclusion of Exogenous variables\n",
    "* Full maximum-likelihood estimation using the Kalman Filter\n",
    "\n",
    "This model is more feature rich than `AutoReg`. Unlike `SARIMAX`, `AutoReg` estimates parameters using OLS.  This is faster and the problem is globally convex, and so there are no issues with local minima. The closed-form estimator and its performance are the key advantages of `AutoReg` over `SARIMAX` when comparing AR(P) models.  `AutoReg` also support seasonal dummies, which can be used with `SARIMAX` if the user includes them as exogenous regressors. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from statsmodels.tsa.api import SARIMAX\n",
    "\n",
    "sarimax_mod = SARIMAX(ind_prod, order=((1, 5, 12, 13), 0, 0), trend=\"c\")\n",
    "sarimax_res = sarimax_mod.fit()\n",
    "print(sarimax_res.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sarimax_params = sarimax_res.params.iloc[:-1].copy()\n",
    "sarimax_params.index = res_glob.params.index\n",
    "params = pd.concat([res_glob.params, sarimax_params], axis=1, sort=False)\n",
    "params.columns = [\"AutoReg\", \"SARIMAX\"]\n",
    "params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Custom Deterministic Processes\n",
    "\n",
    "The `deterministic` parameter allows a custom `DeterministicProcess` to be used. This allows for more complex deterministic terms to be constructed, for example one that includes seasonal components with two periods, or, as the next example shows, one that uses a Fourier series rather than seasonal dummies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from statsmodels.tsa.deterministic import DeterministicProcess\n",
    "\n",
    "dp = DeterministicProcess(housing.index, constant=True, period=12, fourier=2)\n",
    "mod = AutoReg(housing, 2, trend=\"n\", seasonal=False, deterministic=dp)\n",
    "res = mod.fit()\n",
    "print(res.summary())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig = res.plot_predict(720, 840)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
