{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Time series in Pastas\n", "*R.A. Collenteur, University of Graz, 2020*\n", "\n", "Time series are at the heart of time series analysis, and therefore need to be considered carefully when dealing with time series models. In this notebook more background information is provided on important characteristics of time series and how these may influence your modeling results. In general, Pastas depends heavily on Pandas for dealing with time series, but adds capabilities to deal with irregular time series and missing data.\n", "\n", "All time series should be provided to Pastas as `pandas.Series` with a `pandas.DatetimeIndex`. Internally these time series are stored in a `pastas.TimeSeries` object. The goal of this object is to validate the user-provided time series and enable resampling (changing frequencies) of the independent time series. The TimeSeries object also has capabilities to deal with missing data in the user-provided time series. As much of these operations occur internally, this notebook is meant to explain users what is happening and how to check for this.\n", "\n", "
\n", "\n", "Note\n", " \n", "* The standard Pastas data type for a date is the `pandas.Timestamp`.\n", "* The standard Pastas data type for a sequence of dates is the `pandas.DatetimeIndex` with `pandas.Timestamp`.\n", "* The standard Pastas data type for a time series is a `pandas.Series` with a `pandas.DatetimeIndex`\n", " \n", "
" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "jupyter": { "source_hidden": true }, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "import numpy as np\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import pastas as ps\n", "\n", "ps.show_versions()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Different types of time series\n", "Time series data may generally be defined as a set of data values observed at certain times, ordered in a way that the time indices are increasing. Many time series analysis method assume that the time step between the observations is regular, the time series has evenly-spaced observations. These evenly spaced time series may have missing data, but it will still be possible to lay the values on a time-grid with constant time steps. \n", "\n", "This is generally also assumed to be the case for the independent time series in hydrological studies. For example, the precipitation records may have some missing data but the precipitation is reported as the total rainfall over one day. In the case of missing data, we may impute a zero (no rain) or the rainfall amount from a nearby measurement station.\n", "\n", "Groundwater level time series do generally not share these characteristics with other hydrological time series, and are measured at irregular time intervals. This is especially true for historic time series that were measured by hand. The result is that the measurements can not be laid on a regular time grid. The figure below graphically shows the difference between the three types of time series." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "jupyter": { "source_hidden": true }, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "regular = pd.Series(\n", " index=pd.date_range(\"2000-01-01\", \"2000-01-10\", freq=\"D\"), data=np.ones(10)\n", ")\n", "missing_data = regular.copy()\n", "missing_data.loc[[\"2000-01-03\", \"2000-01-08\"]] = np.nan\n", "\n", "index = [t + pd.Timedelta(np.random.rand() * 24, unit=\"H\") for t in missing_data.index]\n", "irregular = missing_data.copy()\n", "irregular.index = index\n", "\n", "fig, axes = plt.subplots(3, 1, figsize=(6, 5), sharex=True, sharey=True)\n", "\n", "regular.plot(ax=axes[0], linestyle=\" \", marker=\"o\", x_compat=True)\n", "missing_data.plot(ax=axes[1], linestyle=\" \", marker=\"o\", x_compat=True)\n", "irregular.plot(ax=axes[2], linestyle=\" \", marker=\"o\", x_compat=True)\n", "\n", "for i, name in enumerate(\n", " [\"(a) Regular time steps\", \"(b) Missing Data\", \"(c) Irregular time steps\"]\n", "):\n", " axes[i].grid()\n", " axes[i].set_title(name)\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Independent and dependent time series \n", "We can differentiate between two types of input time series for Pastas models: the dependent and independent time series. The dependent time series are those that we want to explain (e.g., the groundwater levels) and the independent time series are those that we use to explain the dependent time series (e.g., precipitation or evaporation). The requirements for these time series are different:\n", "\n", "- The dependent time series may be of any kind: regular, missing data or irregular.\n", "- The independent time series has to have regular time steps.\n", "\n", "In practice, this means that the time series provided to `pastas.Model` may be of any kind, and that the time series used by the stressmodels (e.g., `pastas.RerchargeModel`) need to have regular time steps. The regular time steps are required to simulate contributions to the groundwater level fluctuations. As there are virtually no restrictions on the dependent time series, the remainder of this notebook will discuss primarily the independent time series.\n", "\n", "## How does the TimeSeries object validate a time series?\n", "To ensure that a time series can be used for simulation a number of things are checked and changed:\n", "\n", "1. Make sure the values are floats. Values are change to dtype=float if not.\n", "2. Make sure the index is a `pandas.DatetimeIndex`. Index is changed if not.\n", "3. Make sure the timestamps in the index are increasing. Index is sorted if not.\n", "4. Make sure there are no nan-values at the start and end of a time series.\n", "5. Determine the frequency of the time series.\n", "6. Make sure there are no duplicate indices. Values are averaged if this is the case.\n", "7. Remove or fill up nan-values, depending on the settings.\n", "\n", "For each of these steps an INFO message will be returned by Pastas to inform the user if a change is made. The first four steps generally do not have a large impact and are there to prevent some basic issues. Preferably, no changes are reported. \n", "\n", "### Frequency of the input data\n", "Pastas tries to determine the frequency in step 5, and will **always** report the result. It is generally good practice to double-check if the reported frequency agrees with what you know about the time series. Pastas will also report if no frequency can be inferred. If no frequency is reported there is probably some wrong and the user should fix either fix the input time series or provide Pastas with more information.\n", "\n", "Below we consider a time series with precipitation data, measured every day. We will use `settings=\"prec` as a shortcut for the settings to fill nans and resample. We will come back to those settings later." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "rain = pd.read_csv(\n", " \"../examples/data/rain_nb1.csv\", parse_dates=[\"date\"], index_col=\"date\"\n", ").squeeze()\n", "ps.TimeSeries(rain, settings=\"prec\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Pastas correctly report the frequency and we can continue with this time series. Note that the input time series thus agrees with all the checks for the time series validation. Let's now introduce a nan-value and see what happens." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "rain[\"1989-01-01\"] = np.nan\n", "ps.TimeSeries(rain, settings=\"prec\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This also works fine. The frequency was inferred (stored as freq_original) and one nan-value was filled up with 0.0. Now we take the same time series, but drop the nan-value." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ps.TimeSeries(rain.dropna(), settings=\"prec\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above result is probably not what we want. Pastas could not infer the frequency and therefore resorts to the `timestep_weighted_resample` method. Documentation for this method is available in utils.py.\n", "\n", "If we know the original frequency of the time series, we can tell this to Pastas through the `freq_original` argument. As we can see below, the user-provided frequency is used." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "rain = pd.read_csv(\n", " \"../examples/data/rain_nb1.csv\", parse_dates=[\"date\"], index_col=\"date\"\n", ").squeeze()\n", "rain[\"1989-01-01\"] = np.nan\n", "ps.TimeSeries(rain.dropna(), settings=\"prec\", freq_original=\"D\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The above example shows how to obtain the same or different result with four different methods. Some of these methods requires good knowledge about the TimeSeries object and how it processes your time series. It is often preferred to provide Pastas with a better initial time series by resampling it yourself. This has the additional benefit that you are interacting more closely with the data. Most of the examples also follow this pattern.\n", "\n", "
\n", "\n", "Best practice\n", "\n", "Try and modify your original time series such that Pastas returns a message that it was able to infer the frequency from the time series itself: **INFO: Inferred frequency for time series rain: freq=D**\n", " \n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Time series settings\n", "In the examples above we used the `settings` keyword when creating the TimeSeries. This is a shortcut method to select a number of settings from a predefined set of options. These predefined options can accessed through `ps.rcParams[\"timeseries\"]`:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "pd.DataFrame.from_dict(ps.rcParams[\"timeseries\"])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Each column name is a valid option for the `settings` argument. The rows shows the settings that may be chosen for changing the original time series. Once a TimeSeries is created, we can access the existing settings as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ts = ps.TimeSeries(rain, settings=\"prec\")\n", "ts.settings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This settings dictionary now includes both settings used to resample (sample_up, sample_down), extend (fill_before, fill_after), normalize (norm), and fill nans in the time series, but also dynamic settings such as the start and end date (tmin, tmax), the frequency (freq) and the time offset. \n", "\n", "To update these settings you the `update_series` method is available. For example, if we want to resample the above time series to a 7-day frequency and sum up the values we can use:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ts.update_series(freq=\"7D\", sample_down=\"sum\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Because the original series are stored in the TimeSeries object as well, it is also possible to go back again. The changing made to the time series are always started from the original validated time series again. For more information on the possible settings see the API-docs for the [TimeSeries and update_series method](https://pastas.readthedocs.io/en/latest/api/timeseries.html) on the documentation website." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## An example with a Pastas Model\n", "By now you may be wondering why all these settings exist in the first place. The main reason (apart from validating the user-provided time series) is to change the time step of the simulation of the independent time series. It may also be used to extend the time series in time.\n", "\n", "Below we load some time series, visualize them and create a Pastas model with precipitation and evaporation to explain the groundwater level fluctuations. It is generally recommended to plot your time series for a quick visual check of the input data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "head = pd.read_csv(\n", " \"../examples/data/B32C0639001.csv\", parse_dates=[\"date\"], index_col=\"date\"\n", ").squeeze()\n", "rain = pd.read_csv(\n", " \"../examples/data/rain_nb1.csv\", parse_dates=[\"date\"], index_col=\"date\"\n", ").squeeze()\n", "evap = pd.read_csv(\n", " \"../examples/data/evap_nb1.csv\", parse_dates=[\"date\"], index_col=\"date\"\n", ").squeeze()\n", "\n", "fig, axes = plt.subplots(3, 1, figsize=(10, 6), sharex=True)\n", "head.plot(ax=axes[0], x_compat=True, linestyle=\" \", marker=\".\")\n", "evap.plot(ax=axes[1], x_compat=True)\n", "rain.plot(ax=axes[2], x_compat=True)\n", "axes[0].set_ylabel(\"Head [m]\")\n", "axes[1].set_ylabel(\"Evap [mm/d]\")\n", "axes[2].set_ylabel(\"Rain [mm/d]\")\n", "\n", "plt.xlim(\"1985\", \"2005\");" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml = ps.Model(head)\n", "\n", "rch = ps.rch.Linear()\n", "rm = ps.RechargeModel(rain, evap, recharge=rch, rfunc=ps.Gamma(), name=\"rch\")\n", "ml.add_stressmodel(rm)\n", "\n", "ml.solve(noise=True, tmin=\"1990\", report=\"basic\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## What is the model freq?\n", "The output below shows that the time series have frequencies of `freq=D`. The fit report also shows a frequency of `freq=D`. The frequency reported in the fit_report is the time step of the simulation for the independent time series, and is internally passed on to the stressmodels. The user-provided dependent time series are stored in the stressmodel object and can be accessed as follows. " ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml.stressmodels[\"rch\"].stress" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we want to change the resample method, for example we want to sum the precipitation and evaporation when sampling down (e.g., daily to weekly) we may do the following:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "for stress in ml.stressmodels[\"rch\"].stress:\n", " stress.update_series(sample_down=\"sum\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After changing the methods for sampling down, we now solve the model with a simulation time step of 14 days. The precipitation and evaporation are then summed up over 14 day intervals, before being translated to a groundwater fluctuation using a respons function." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml.settings" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml.solve(freq=\"14D\", tmin=\"1980\", report=\"basic\")\n", "ml.plots.results(figsize=(10, 6), tmin=\"1970\");" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml.stressmodels[\"rch\"].stress[1].update_series(tmin=\"1960\")\n", "ml.stressmodels[\"rch\"].stress[1].settings" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another method to obtain the settings of the time series used in a stressmodel is as follows:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "ml.get_stressmodel_settings(\"rch\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Warnings\n", "Because the TimeSeries object is a relatively complicated object that can potentially change model results extra care has to be taken in some cases. Below is a number of outstanding warnings and the related GitHub issues. \n", "\n", "
\n", "\n", "A note on dependent time series\n", "\n", "The dependent time series (stored as `ml.oseries`) are also stored in a TimeSeries object and therefore have the same capabilities. Usage of these methods on the dependent time series is however experimental and not recommended for real world use cases. See also [Issue #68](https://github.com/pastas/pastas/issues/68) and [Discussion #199](https://github.com/pastas/pastas/discussions/199)\n", " \n", "
\n", "\n", "\n", "
\n", "\n", "A note on monthly data\n", "\n", "Monthly data is strictly irregular data, and poses additional challenges when resampling to regular frequencies. Pastas does not differentiate between monthly data reported at months end (`freq=M`) or months beginning (`freq=MS`) and the default settings are selected for `freq=M`. There may also be issues with extending the time series. See also [Issue #239](https://github.com/pastas/pastas/issues/239)\n", " \n", "
" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.7" } }, "nbformat": 4, "nbformat_minor": 4 }