Section - 8 Evaluate Model Performance

Now we get to see the results of our hard work! There are some additional data preparation steps we need to take before we can visualize the results in aggregate; if you are just looking for the charts showing the results they are shown later on in the “Visualize Results” section below.

8.1 Summarizing models

Because we know what really happened for the target variable in the test data we used in the previous step, we can get a good idea of how good the model performed on a dataset it has never seen before. We do this to avoid overfitting, which is the idea that the model may work really well on the training data we provided, but not on the new data that we want to predictions on. If the performance on the test set is good, that is a good sign. If the data is split into several subsets and each subset has consistent results for the training and test datasets, that is an even better sign the model may perform as expected.

The first row of the data is for the BTC cryptocurrency for the split number 1. For this row of data (and all others), we made predictions for the test_data using a linear regression model and saved the results in the lm_test_predictions column. The models were trained on the train_data and had not yet seen the results from the test_data, so how accurate was the model in its predictions on this data?

8.1.1 MAE

Each individual prediction can be compared to the observation of what actually happened, and for each prediction we can calculate the error between the two. We can then take all of the values for the error of the prediction relative to the actual observations, and summarize the performance as a Mean Absolute Error (MAE) of those values, which gives us a single value to use as an indicator of the accuracy of the model. The higher the MAE score, the higher the error, meaning the model performs worse when the value is larger.

8.1.2 RMSE

A common metric to evaluate the performance of a model is the Root Mean Square Error, which is similar to the MAE but squares and then takes the square root of the values. An interesting implication of this, is that the RMSE will always be larger or equal to the MAE, where a large degree of error on a single observation would get penalized more by the RMSE. The higher the RMSE value, the worse the performance of the model, and can range from 0 to infinity, meaning there is no defined limit on the amount of error you could have (unlike the next metric).

8.1.3 R Squared

The \(R^2\), also known as the coefficient of determination, is a measure that describes the strength in the correlation between the predictions made and the actual results. A value of 1.0 would mean that the predictions made were exactly identical to the actual results. A perfect score is usually concerning because even a great model shouldn’t be exactly 100% accurate and usually indicates a mistake was made that gave away the results to the model and would not perform nearly as good when put into practice in the real world, but in the case of the \(R^2\) the higher the score (from 0 to 1) the better.

8.1.4 Get Metrics

We can return the RMSE and \(R^2\) metrics for the BTC cryptocurrency and the split number 1 by using the postResample() function from the caret package:

postResample(pred = cryptodata_nested$lm_test_predictions[[1]], 
             obs = cryptodata_nested$test_data[[1]]$target_price_24h)
##            RMSE        Rsquared             MAE 
## 722.37708700857   0.00001231333 674.63486227479

We can extract the first element to return the RMSE metric, and the second element for the R Squared (R^2) metric. We are using [[1]] to extract the first element of the lm_test_predictions and test_data and compare the predictions to the actual value of the target_price24h column.

This model used the earliest subset of the data available for the cryptocurrency. How does the same model used to predict this older subset of the data perform when applied to the most recent subset of the data from the holdout?

We can get the same summary of results comparing the lm_holdout_predictions to what actually happened to the target_price_24h column of the actual holdout_data:

postResample(pred = cryptodata_nested$lm_holdout_predictions[[1]], 
             obs = cryptodata_nested$holdout_data[[1]]$target_price_24h)
##      RMSE  Rsquared       MAE 
##        NA 0.2333323        NA

The result above may show a value of NA for the RMSE metric. We will explain and resolve the issue later on.

8.1.5 Comparing Metrics

Why not just pick one metric and stick to it? We certainly could, but these two metrics complement each other. For example, if we had a model that always predicts a 0% price change for the time period, the model may have a low error but it wouldn’t actually be very informative in the direction or magnitude of those movements and the predictions and actuals would not be very correlated with each other which would lead to a low \(R^2\) value. We are using both because it helps paint a more complete picture in this sense, and depending on the task you may want to use a different set of metrics to evaluate the performance. It is also worth mentioning that if your target variable you are predicting is either 0 or 1, this would be a classification problem where different metrics become more appropriate to use.

These are indicators that should be taken with a grain of salt individually, but comparing the results across many different models for the same cryptocurrency can help us determine which models work best for the problem, and then comparing those results across many cryptocurrencies can help us understand which cryptocurrencies we can predict with the most accuracy.

Before we can draw these comparisons however, we will need to “standardize” the values to create a fair comparison across all dataasets.

8.2 Data Prep - Adjust Prices

Because cryptocurrencies can vary dramatically in their prices with some trading in the tens of thousands of dollars and others trading for less than a cent, we need to make sure to standardize the RMSE columns to provide a fair comparison for the metric.

Therefore, before using the postResample() function, let’s convert both the predictions and the target to be the % change in price over the 24 hour period, rather than the change in price ($).

This step is particularly tedious, but it is important. As with the rest of this tutorial, try to understand what we are doing and why even if you find the code overwhelming. All we are doing in this “Adjust Prices” section is we are adjusting all of the prices to be represented as percentage change between observations, which will allow us to draw a fair comparison of the metrics across all cryptocurrencies, which would not be possible using the prices themselves. If you want to skip the tedious steps and want to see the performance of the models visualized, click here to skip ahead.

8.2.1 Add Last Price

In order to convert the first prediction made to be a percentage, we need to know the previous price, which would be the last observation from the train data. Therefore, let’s make a function to add the last_price_train column and append it to the predictions made so we can calculate the % change of the first element relative to the last observation in the train data, before later removing the value not associated with the predictions:

last_train_price <- function(train_data, predictions){
  c(tail(train_data$price_usd,1), predictions)
}

We will first perform all steps on the linear regression models to make the code a little more digestible, and we will then perform the same steps for the rest of the models.

8.2.1.1 Test

Overwrite the old predictions for the first 4 splits of the test data using the new function created above:

cryptodata_nested <- mutate(cryptodata_nested,
                            lm_test_predictions = ifelse(split < 5,
                                                         map2(train_data, lm_test_predictions, last_train_price),
                                                         NA))

The mutate() function is used to create the new column lm_test_predictions assigning the value only for the first 4 splits where the test data would actually exist (the 5th being the holdout set) using the ifelse() function.

8.2.1.2 Holdout

Do the same but for the holdout now. For the holdout we need to take the last price point of the 5th split:

cryptodata_nested_holdout <- mutate(filter(cryptodata_nested, split == 5),
                                    lm_holdout_predictions = map2(train_data, lm_holdout_predictions, last_train_price))

Now join the holdout data to all rows based on the cryptocurrency symbol alone:

cryptodata_nested <- left_join(cryptodata_nested, 
                               select(cryptodata_nested_holdout, symbol, lm_holdout_predictions),
                               by='symbol')
# Remove unwanted columns
cryptodata_nested <- select(cryptodata_nested, -lm_holdout_predictions.x, -split.y)
# Rename the columns kept
cryptodata_nested <- rename(cryptodata_nested, 
                            lm_holdout_predictions = 'lm_holdout_predictions.y',
                            split = 'split.x')
# Reset the correct grouping structure
cryptodata_nested <- group_by(cryptodata_nested, symbol, split)

8.2.2 Convert to Percentage Change

Now we have everything we need to accurately calculate the percentage change between observations including the first one. Let’s make a new function to calculate the percentage change:

standardize_perc_change <- function(predictions){
  results <- (diff(c(lag(predictions, 1), predictions)) / lag(predictions, 1))*100
  # Exclude the first element, next element will be % change of first prediction
  tail(head(results, length(predictions)), length(predictions)-1)
}

Overwrite the old predictions with the new predictions adjusted as a percentage now:

cryptodata_nested <- mutate(cryptodata_nested,
                            lm_test_predictions = ifelse(split < 5,
                                                         map(lm_test_predictions, standardize_perc_change),
                                                         NA),
                            # Holdout for all splits
                            lm_holdout_predictions = map(lm_holdout_predictions, standardize_perc_change))

8.2.3 Actuals

Now do the same thing to the actual prices. Let’s make a new column called actuals containing the real price values (rather than the predicted ones):

actuals_create <- function(train_data, test_data){
  c(tail(train_data$price_usd,1), as.numeric(unlist(select(test_data, price_usd))))
}

Use the new function to create the new column actuals:

cryptodata_nested <- mutate(cryptodata_nested,
                            actuals_test = ifelse(split < 5,
                                             map2(train_data, test_data, actuals_create),
                                             NA))

8.2.3.1 Holdout

Again, for the holdout we need the price from the training data of the 5th split to perform the first calculation:

cryptodata_nested_holdout <- mutate(filter(cryptodata_nested, split == 5),
                                    actuals_holdout = map2(train_data, holdout_data, actuals_create))

Join the holdout data to all rows based on the cryptocurrency symbol alone:

cryptodata_nested <- left_join(cryptodata_nested, 
                               select(cryptodata_nested_holdout, symbol, actuals_holdout),
                               by='symbol')
# Remove unwanted columns
cryptodata_nested <- select(cryptodata_nested, -split.y)
# Rename the columns kept
cryptodata_nested <- rename(cryptodata_nested, split = 'split.x')
# Reset the correct grouping structure
cryptodata_nested <- group_by(cryptodata_nested, symbol, split)

8.2.4 Actuals as % Change

Now we can convert the new actuals to express the price_usd as a % change relative to the previous value using adapting the function from earlier:

actuals_perc_change <- function(predictions){
  results <- (diff(c(lag(predictions, 1), predictions)) / lag(predictions, 1))*100
  # Exclude the first element, next element will be % change of first prediction
  tail(head(results, length(predictions)), length(predictions)-1)
}
cryptodata_nested <- mutate(cryptodata_nested,
                            actuals_test = ifelse(split < 5,
                                             map(actuals_test, actuals_perc_change),
                                             NA),
                            actuals_holdout = map(actuals_holdout, actuals_perc_change))

8.3 Review Summary Statistics

Now that we standardized the price to show the percentage change relative to the previous period instead of the price in dollars, we can actually compare the summary statistics across all cryptocurrencies and have it be a fair comparison.

Let’s get the same statistic as we did at the beginning of this section, but this time on the standardized values. This time to calculate the RMSE error metric let’s use the rmse() function from the hydroGOF package because it allows us to set the na.rm = T parameter, and otherwise one NA value would return NA for the overall RMSE:

hydroGOF::rmse(cryptodata_nested$lm_test_predictions[[1]], 
               cryptodata_nested$actuals_test[[1]], 
               na.rm=T)
## [1] 0.4112694

8.3.1 Calculate R^2

Now we can do the same for the R Squared metric using the same postResample() function that we used at the start of this section:

evaluate_preds_rsq <- function(predictions, actuals){

  postResample(pred = predictions, obs = actuals)[[2]]

}
cryptodata_nested <- mutate(cryptodata_nested,
                            lm_rsq_test = unlist(ifelse(split < 5,
                                                         map2(lm_test_predictions, actuals_test, evaluate_preds_rsq),
                                                         NA)),
                            lm_rsq_holdout = unlist(map2(lm_holdout_predictions, actuals_holdout, evaluate_preds_rsq)))

Look at the results:

select(cryptodata_nested, lm_rsq_test, lm_rsq_holdout)
## # A tibble: 490 x 4
## # Groups:   symbol, split [490]
##    symbol split lm_rsq_test lm_rsq_holdout
##    <chr>  <dbl>       <dbl>          <dbl>
##  1 BTC        1      0.608           0.760
##  2 ETH        1      0.682           0.684
##  3 EOS        1      0.0121          0.841
##  4 LTC        1      0.790           0.181
##  5 BSV        1      0.459           0.778
##  6 ADA        1      0.591           0.589
##  7 TRX        1      0.623           0.551
##  8 ZEC        1      0.528           0.836
##  9 HT         1      0.631           0.550
## 10 XMR        1      0.488           0.783
## # ... with 480 more rows

8.3.2 Calculate RMSE

Similarly let’s make a function to get the RMSE metric for all models:

evaluate_preds_rmse <- function(predictions, actuals){

  hydroGOF::rmse(predictions, actuals, na.rm=T)

}

Now we can use the map2() function to use it to get the RMSE metric for both the test data and the holdout:

cryptodata_nested <- mutate(cryptodata_nested,
                            lm_rmse_test = unlist(ifelse(split < 5,
                                                         map2(lm_test_predictions, actuals_test, evaluate_preds_rmse),
                                                         NA)),
                            lm_rmse_holdout = unlist(map2(lm_holdout_predictions, actuals_holdout, evaluate_preds_rmse)))

Look at the results. Wrapping them in print(n=500) overwrites the behavior to only give a preview of the data so we can view the full results (up to 500 observations).

print(select(cryptodata_nested, lm_rmse_test, lm_rmse_holdout, lm_rsq_test, lm_rsq_holdout), n=500)
## # A tibble: 490 x 6
## # Groups:   symbol, split [490]
##     symbol split lm_rmse_test lm_rmse_holdout   lm_rsq_test lm_rsq_holdout
##     <chr>  <dbl>        <dbl>           <dbl>         <dbl>          <dbl>
##   1 BTC        1        0.411          0.302   0.608              0.760   
##   2 ETH        1        0.471          0.675   0.682              0.684   
##   3 EOS        1        1.70           0.658   0.0121             0.841   
##   4 LTC        1        0.693          1.59    0.790              0.181   
##   5 BSV        1        0.587          0.539   0.459              0.778   
##   6 ADA        1        0.589          0.575   0.591              0.589   
##   7 TRX        1        0.552          2.07    0.623              0.551   
##   8 ZEC        1        1.01           0.703   0.528              0.836   
##   9 HT         1        0.658          0.553   0.631              0.550   
##  10 XMR        1        0.585          0.421   0.488              0.783   
##  11 KNC        1        0.770          0.685   0.627              0.759   
##  12 ZRX        1        1.88           0.826   0.000150           0.853   
##  13 BAT        1        0.817          0.695   0.720              0.567   
##  14 BNT        1        0.301          0.539   0.898              0.735   
##  15 MANA       1        2.34           0.359   0.0147             0.850   
##  16 DGB        1        0.860          0.644   0.466              0.660   
##  17 ENJ        1        2.76           0.538   0.000156           0.808   
##  18 XEM        1        2.71           0.746   0.470              0.704   
##  19 BTG        1        0.824          0.600   0.618              0.559   
##  20 KMD        1        2.89           1.24    0.265              0.353   
##  21 ARDR       1        1.03           1.21    0.596              0.671   
##  22 ELF        1        1.51           3.52    0.0884             0.684   
##  23 NEXO       1        1.35           0.787   0.000857           0.605   
##  24 CHZ        1        1.93           0.667   0.0294             0.591   
##  25 BRD        1        0.768          0.713   0.626              0.469   
##  26 CKB        1        1.15           1.70    0.405              0.00637 
##  27 DCR        1        0.756          0.728   0.445              0.398   
##  28 WAXP       1        1.02           1.04    0.133              0.674   
##  29 OAX        1        1.34           1.37    0.198              0.0675  
##  30 LEO        1        0.508        NaN       0.0192            NA       
##  31 ETP        1        2.71           1.45    0.0184             0.352   
##  32 AVA        1        2.37           1.25    0.209              0.0360  
##  33 JST        1        1.03           1.46    0.550              0.170   
##  34 SRN        1        1.18           1.14    0.221              0.324   
##  35 LSK        1        1.99           0.593   0.465              0.530   
##  36 COTI       1        1.07           1.25    0.746              0.914   
##  37 NU         1        0.460          1.25    0.848              0.241   
##  38 TLM        1       11.7            0.524   0.343              0.911   
##  39 BIZZ       1        1.19          17.2     0.0647             0.0438  
##  40 PERP       1        1.12           1.22    0.689              0.704   
##  41 NWC        1        0.870          0.724   0.820              0.680   
##  42 OGN        1        3.81           0.790   0.0283             0.838   
##  43 UNI        1        0.674          0.639   0.783              0.805   
##  44 SOLO       1        1.03           2.27    0.0287             0.544   
##  45 ICP        1        0.735          1.01    0.858              0.451   
##  46 ETC        1        2.75           0.775   0.548              0.312   
##  47 REEF       1        0.667          0.977   0.873              0.332   
##  48 XVG        1        0.887          0.489   0.772              0.741   
##  49 RSR        1        0.579          0.597   0.822              0.875   
##  50 RARI       1        6.84           1.23    0.0986             0.466   
##  51 VIB        1        1.54           1.09    0.00751            0.0547  
##  52 TV         1        2.75          11.4     0.123              0.00474 
##  53 CDT        1        1.15           1.81    0.0171             0.337   
##  54 OKB        1        2.03           0.0913 NA                  0.0355  
##  55 CROOLD     1        1.58           1.17    0.193              0.0557  
##  56 TV         2       91.6           11.4    NA                  0.00474 
##  57 ETP        2        2.81           1.45    0.260              0.352   
##  58 XVG        2        0.979          0.489   0.569              0.741   
##  59 LEO        2       46.9          NaN       0.00352           NA       
##  60 SOLO       2        1.41           2.27    0.0624             0.544   
##  61 REEF       2        0.584          0.977   0.691              0.332   
##  62 SRN        2        0.731          1.14    0.787              0.324   
##  63 ELF        2        1.32           3.52    0.848              0.684   
##  64 OGN        2        0.839          0.790   0.525              0.838   
##  65 BRD        2        0.325          0.713   0.917              0.469   
##  66 WAXP       2        0.450          1.04    0.731              0.674   
##  67 NU         2        0.746          1.25    0.582              0.241   
##  68 RARI       2        1.27           1.23    0.503              0.466   
##  69 BTC        2        0.592          0.302   0.403              0.760   
##  70 EOS        2        0.409          0.658   0.905              0.841   
##  71 ETH        2        0.381          0.675   0.945              0.684   
##  72 LTC        2        0.423          1.59    0.801              0.181   
##  73 ADA        2        0.376          0.575   0.865              0.589   
##  74 BSV        2        0.469          0.539   0.862              0.778   
##  75 ZEC        2        0.638          0.703   0.850              0.836   
##  76 HT         2        0.845          0.553   0.312              0.550   
##  77 TRX        2        0.418          2.07    0.861              0.551   
##  78 KNC        2        0.726          0.685   0.783              0.759   
##  79 XMR        2        0.706          0.421   0.480              0.783   
##  80 ZRX        2        0.847          0.826   0.800              0.853   
##  81 BAT        2        0.692          0.695   0.888              0.567   
##  82 BNT        2        0.367          0.539   0.915              0.735   
##  83 MANA       2        0.487          0.359   0.690              0.850   
##  84 DGB        2        0.619          0.644   0.921              0.660   
##  85 ENJ        2        0.733          0.538   0.476              0.808   
##  86 XEM        2        0.523          0.746   0.823              0.704   
##  87 BTG        2        1.02           0.600   0.566              0.559   
##  88 KMD        2        1.03           1.24    0.816              0.353   
##  89 ARDR       2       12.7            1.21    0.00361            0.671   
##  90 NEXO       2        0.636          0.787   0.738              0.605   
##  91 CHZ        2        0.527          0.667   0.769              0.591   
##  92 CKB        2        0.508          1.70    0.884              0.00637 
##  93 DCR        2        0.612          0.728   0.745              0.398   
##  94 AVA        2        0.883          1.25    0.806              0.0360  
##  95 JST        2        0.655          1.46    0.871              0.170   
##  96 COTI       2        0.862          1.25    0.798              0.914   
##  97 BIZZ       2        0.337         17.2     0.984              0.0438  
##  98 PERP       2        2.22           1.22    0.0478             0.704   
##  99 NWC        2        1.40           0.724   0.932              0.680   
## 100 UNI        2        0.491          0.639   0.929              0.805   
## 101 ICP        2        0.464          1.01    0.874              0.451   
## 102 ETC        2        0.695          0.775   0.188              0.312   
## 103 LSK        2        0.958          0.593   0.948              0.530   
## 104 RSR        2        2.24           0.597   0.396              0.875   
## 105 TLM        2        1.15           0.524   0.494              0.911   
## 106 OAX        2        1.66           1.37    0.179              0.0675  
## 107 VIB        2        0.817          1.09    0.385              0.0547  
## 108 CDT        2        2.52           1.81    0.324              0.337   
## 109 CROOLD     2        1.24           1.17    0.224              0.0557  
## 110 OKB        2        9.51           0.0913  0.0999             0.0355  
## 111 ARPA       1        1.36           1.04    0.0218             0.414   
## 112 IHF        1        0.836          3.52    0.508              0.00440 
## 113 MDX        1        1.16           1.24    0.176              0.0363  
## 114 XYM        1        1.07           1.03    0.0147             0.301   
## 115 NEO        1        1.11           1.09    0.110              0.316   
## 116 JULD       1        2.45           1.88    0.376              0.134   
## 117 COMP       1        5.55           1.03    0.145              0.747   
## 118 ZIL        1        3.03           1.33    0.512              0.133   
## 119 BOSON      1       18.4            6.45    0.811              0.619   
## 120 ORN        1        1.11           1.34    0.391              0.834   
## 121 MAID       1        0.926          0.831   0.578              0.0256  
## 122 VLX        1        0.921          0.954   0.567              0.366   
## 123 SXP        1        2.25           1.63    0.0527             0.154   
## 124 ALPHA      1        4.62           1.03    0.0750             0.803   
## 125 CLT        1        1.57           1.75    0.00435            0.452   
## 126 JUV        1        0.778          1.73    0.143              0.268   
## 127 CVCOIN     1        5.92           4.13    0.192              0.00513 
## 128 INJ        1        3.64           1.26    0.0000000635       0.0353  
## 129 CRPT       1        7.62           7.51    0.00157            0.225   
## 130 WRX        1        1.00           0.922   0.483              0.304   
## 131 ETN        1        4.75           1.93    0.124              0.628   
## 132 ONE        1        1.54           2.08    0.140              0.338   
## 133 IQN        1        3.93           1.31    0.569              0.387   
## 134 XCH        1        1.38           0.932   0.764              0.0518  
## 135 SENSO      1        1.40           6.89    0.476              0.0508  
## 136 EGLD       1        2.02           1.32    0.361              0.753   
## 137 KLV        1        2.01           1.68    0.167              0.000924
## 138 LOC        1        1.03           1.27    0.0304             0.00421 
## 139 FIL        1        2.47           1.11    0.0114             0.231   
## 140 LPT        1        2.26           0.885   0.00184            0.233   
## 141 EVX        1        3.74           5.62    0.364              0.0160  
## 142 1INCH      1        1.76           1.11    0.0336             0.700   
## 143 TON        1        0.817          0.600   0.240              0.397   
## 144 MITH       1        1.26           1.30    0.148              0.00858 
## 145 DOGE       1        1.89           0.449   0.0610             0.295   
## 146 ONG        1        0.440          2.87    0.610              0.281   
## 147 POND       1        2.42           1.43    0.758              0.242   
## 148 CTSI       1        1.20           1.17    0.0697             0.507   
## 149 OCEAN      1        3.64           1.84    0.0126             0.00284 
## 150 NEAR       1        1.57           1.91    0.0222             0.0200  
## 151 DASH       1        1.10           0.939   0.0354             0.403   
## 152 DODO       1        1.34           1.50    0.862              0.124   
## 153 WBTC       1       47.9            0.906   0.328              0.166   
## 154 ETP        3        2.10           1.45    0.770              0.352   
## 155 TV         3     2423.            11.4     0.0435             0.00474 
## 156 LEO        3        2.41         NaN       0.180             NA       
## 157 XVG        3        2.13           0.489   0.105              0.741   
## 158 SOLO       3        0.864          2.27    0.0136             0.544   
## 159 REEF       3        0.321          0.977   0.946              0.332   
## 160 SRN        3        2.04           1.14    0.0562             0.324   
## 161 OGN        3        2.24           0.790   0.0423             0.838   
## 162 BRD        3        2.44           0.713   0.768              0.469   
## 163 WAXP       3        0.777          1.04    0.210              0.674   
## 164 RARI       3        3.04           1.23    0.625              0.466   
## 165 EOS        3        2.37           0.658   0.0207             0.841   
## 166 ETH        3        0.557          0.675   0.825              0.684   
## 167 LTC        3        1.26           1.59    0.267              0.181   
## 168 ADA        3        0.995          0.575   0.742              0.589   
## 169 BSV        3        1.15           0.539   0.254              0.778   
## 170 TRX        3        1.66           2.07    0.0761             0.551   
## 171 ZEC        3        1.33           0.703   0.229              0.836   
## 172 XMR        3        0.491          0.421   0.940              0.783   
## 173 KNC        3        1.47           0.685   0.127              0.759   
## 174 ZRX        3        1.82           0.826   0.0235             0.853   
## 175 BAT        3        1.58           0.695   0.117              0.567   
## 176 BNT        3        0.880          0.539   0.494              0.735   
## 177 MANA       3        0.903          0.359   0.627              0.850   
## 178 ENJ        3        1.19           0.538   0.671              0.808   
## 179 XEM        3        1.44           0.746   0.396              0.704   
## 180 KMD        3        3.74           1.24    0.180              0.353   
## 181 ARDR       3        1.08           1.21    0.731              0.671   
## 182 NEXO       3        0.668          0.787   0.900              0.605   
## 183 CHZ        3        0.818          0.667   0.811              0.591   
## 184 CKB        3        1.61           1.70    0.272              0.00637 
## 185 DCR        3        0.314          0.728   0.923              0.398   
## 186 AVA        3        1.62           1.25    0.405              0.0360  
## 187 JST        3        1.65           1.46    0.135              0.170   
## 188 ETC        3        0.546          0.775   0.584              0.312   
## 189 NU         3        1.34           1.25    0.204              0.241   
## 190 BTC        3        0.342          0.302   0.852              0.760   
## 191 BTG        3        2.32           0.600   0.00688            0.559   
## 192 ICP        3        0.702          1.01    0.903              0.451   
## 193 UNI        3        0.852          0.639   0.931              0.805   
## 194 BIZZ       3        2.37          17.2     0.711              0.0438  
## 195 PERP       3        2.06           1.22    0.148              0.704   
## 196 NWC        3        1.20           0.724   0.439              0.680   
## 197 COTI       3        2.20           1.25    0.0270             0.914   
## 198 HT         3        0.831          0.553   0.706              0.550   
## 199 DGB        3        1.81           0.644   0.00637            0.660   
## 200 LSK        3        0.594          0.593   0.865              0.530   
## 201 TLM        3        2.32           0.524   0.390              0.911   
## 202 ELF        3        0.665          3.52    0.684              0.684   
## 203 OAX        3        1.80           1.37    0.0130             0.0675  
## 204 CDT        3        2.30           1.81    0.539              0.337   
## 205 VIB        3        1.40           1.09    0.354              0.0547  
## 206 RSR        3        1.48           0.597   0.422              0.875   
## 207 ONG        2        1.85           2.87    0.0635             0.281   
## 208 MDX        2        1.29           1.24    0.0144             0.0363  
## 209 CROOLD     3        1.40           1.17    0.365              0.0557  
## 210 LPT        2        1.27           0.885   0.0586             0.233   
## 211 ETN        2        1.43           1.93    0.767              0.628   
## 212 POND       2        0.808          1.43    0.825              0.242   
## 213 ORN        2        1.90           1.34    0.239              0.834   
## 214 VLX        2        0.656          0.954   0.511              0.366   
## 215 JUV        2        1.30           1.73    0.0326             0.268   
## 216 CRPT       2        1.34           7.51    0.316              0.225   
## 217 MAID       2        2.15           0.831   0.0760             0.0256  
## 218 KLV        2        0.832          1.68    0.647              0.000924
## 219 CTSI       2        3.78           1.17    0.0302             0.507   
## 220 TON        2        0.288          0.600   0.334              0.397   
## 221 ARPA       2        3.02           1.04    0.277              0.414   
## 222 IHF        2        0.973          3.52    0.111              0.00440 
## 223 XYM        2        3.96           1.03    0.123              0.301   
## 224 NEO        2        2.07           1.09    0.356              0.316   
## 225 COMP       2        2.37           1.03    0.0448             0.747   
## 226 ZIL        2        1.87           1.33    0.0574             0.133   
## 227 BOSON      2        5.20           6.45    0.992              0.619   
## 228 SXP        2        1.89           1.63    0.000641           0.154   
## 229 ALPHA      2        1.55           1.03    0.104              0.803   
## 230 CLT        2        2.56           1.75    0.148              0.452   
## 231 INJ        2        1.47           1.26    0.653              0.0353  
## 232 WRX        2        1.75           0.922   0.212              0.304   
## 233 ONE        2        2.10           2.08    0.237              0.338   
## 234 XCH        2        0.931          0.932   0.0608             0.0518  
## 235 SENSO      2        8.63           6.89    0.0491             0.0508  
## 236 LOC        2        0.525          1.27    0.299              0.00421 
## 237 FIL        2        0.916          1.11    0.284              0.231   
## 238 1INCH      2        1.47           1.11    0.298              0.700   
## 239 DOGE       2        1.52           0.449   0.0338             0.295   
## 240 NEAR       2        1.49           1.91    0.852              0.0200  
## 241 DASH       2        1.98           0.939   0.334              0.403   
## 242 MITH       2        1.63           1.30    0.202              0.00858 
## 243 CVCOIN     2        3.69           4.13    0.409              0.00513 
## 244 EVX        2        1.33           5.62    0.0296             0.0160  
## 245 JULD       2        5.49           1.88    0.170              0.134   
## 246 DODO       2        2.73           1.50    0.648              0.124   
## 247 OCEAN      2        3.11           1.84    0.0703             0.00284 
## 248 IQN        2        1.89           1.31    0.286              0.387   
## 249 EGLD       2        6.61           1.32    0.295              0.753   
## 250 OKB        3        5.78           0.0913  0.0460             0.0355  
## 251 WBTC       2      121.             0.906   0.00271            0.166   
## 252 ETP        4        4.54           1.45    0.368              0.352   
## 253 LEO        4        1.02         NaN       0.678             NA       
## 254 ONG        3        0.831          2.87    0.613              0.281   
## 255 LPT        3        1.96           0.885   0.580              0.233   
## 256 MDX        3        1.91           1.24    0.225              0.0363  
## 257 SOLO       4        1.45           2.27    0.0813             0.544   
## 258 CRPT       3        6.87           7.51    0.0221             0.225   
## 259 JULD       3        4.06           1.88    0.237              0.134   
## 260 MAID       3        1.30           0.831   0.827              0.0256  
## 261 KLV        3        2.31           1.68    0.353              0.000924
## 262 TON        3        0.804          0.600   0.231              0.397   
## 263 ORN        3        2.27           1.34    0.709              0.834   
## 264 MITH       3        1.06           1.30    0.589              0.00858 
## 265 ARPA       3        1.01           1.04    0.749              0.414   
## 266 IHF        3        1.20           3.52    0.000324           0.00440 
## 267 XYM        3        1.11           1.03    0.130              0.301   
## 268 NEO        3        0.787          1.09    0.629              0.316   
## 269 COMP       3        1.07           1.03    0.259              0.747   
## 270 ZIL        3        0.729          1.33    0.773              0.133   
## 271 BOSON      3        3.55           6.45    0.822              0.619   
## 272 VLX        3        2.82           0.954   0.437              0.366   
## 273 ALPHA      3        2.28           1.03    0.192              0.803   
## 274 CLT        3        0.507          1.75    0.918              0.452   
## 275 INJ        3        0.638          1.26    0.803              0.0353  
## 276 WRX        3        1.52           0.922   0.880              0.304   
## 277 ONE        3        1.77           2.08    0.608              0.338   
## 278 XCH        3        0.693          0.932   0.212              0.0518  
## 279 SENSO      3        3.91           6.89    0.0221             0.0508  
## 280 LOC        3        0.740          1.27    0.804              0.00421 
## 281 FIL        3        0.544          1.11    0.696              0.231   
## 282 EVX        3        3.13           5.62    0.00346            0.0160  
## 283 1INCH      3        0.844          1.11    0.640              0.700   
## 284 NEAR       3        1.23           1.91    0.650              0.0200  
## 285 DASH       3        1.47           0.939   0.732              0.403   
## 286 SXP        3        1.14           1.63    0.418              0.154   
## 287 CTSI       3        1.53           1.17    0.0436             0.507   
## 288 JUV        3        2.20           1.73    0.463              0.268   
## 289 DOGE       3        0.696          0.449   0.246              0.295   
## 290 OGN        4        1.70           0.790   0.358              0.838   
## 291 DODO       3        3.91           1.50    0.436              0.124   
## 292 CDT        4       16.6            1.81    0.101              0.337   
## 293 CVCOIN     3        7.85           4.13    0.591              0.00513 
## 294 NU         4        2.09           1.25    0.0290             0.241   
## 295 OAX        4        1.23           1.37    0.430              0.0675  
## 296 ETC        4        0.367          0.775   0.901              0.312   
## 297 EOS        4        0.412          0.658   0.949              0.841   
## 298 ETH        4        1.50           0.675   0.763              0.684   
## 299 LTC        4        0.636          1.59    0.773              0.181   
## 300 ADA        4        0.617          0.575   0.940              0.589   
## 301 BSV        4        0.760          0.539   0.158              0.778   
## 302 TRX        4        1.38           2.07    0.822              0.551   
## 303 ZEC        4        0.570          0.703   0.713              0.836   
## 304 XMR        4        0.482          0.421   0.837              0.783   
## 305 KNC        4        0.645          0.685   0.740              0.759   
## 306 BAT        4        0.668          0.695   0.871              0.567   
## 307 BNT        4        1.07           0.539   0.640              0.735   
## 308 MANA       4        1.02           0.359   0.133              0.850   
## 309 ENJ        4        0.975          0.538   0.498              0.808   
## 310 XEM        4        0.440          0.746   0.914              0.704   
## 311 KMD        4        0.790          1.24    0.896              0.353   
## 312 ARDR       4        0.602          1.21    0.952              0.671   
## 313 NEXO       4        0.922          0.787   0.296              0.605   
## 314 CHZ        4        1.09           0.667   0.0624             0.591   
## 315 CKB        4        0.893          1.70    0.660              0.00637 
## 316 DCR        4        1.97           0.728   0.521              0.398   
## 317 AVA        4        1.57           1.25    0.271              0.0360  
## 318 JST        4        1.52           1.46    0.911              0.170   
## 319 BTC        4        0.348          0.302   0.901              0.760   
## 320 ZRX        4        0.898          0.826   0.617              0.853   
## 321 BTG        4        1.38           0.600   0.225              0.559   
## 322 ICP        4        1.19           1.01    0.385              0.451   
## 323 UNI        4        0.909          0.639   0.926              0.805   
## 324 BIZZ       4        7.58          17.2     0.0487             0.0438  
## 325 PERP       4        2.71           1.22    0.506              0.704   
## 326 NWC        4        0.740          0.724   0.934              0.680   
## 327 COTI       4        0.882          1.25    0.261              0.914   
## 328 HT         4        0.896          0.553   0.834              0.550   
## 329 DGB        4        1.43           0.644   0.531              0.660   
## 330 TLM        4        1.56           0.524   0.183              0.911   
## 331 RARI       4        1.39           1.23    0.861              0.466   
## 332 POND       3        6.63           1.43    0.0515             0.242   
## 333 LSK        4        0.405          0.593   0.914              0.530   
## 334 WAXP       4        0.375          1.04    0.975              0.674   
## 335 BRD        4        0.377          0.713   0.857              0.469   
## 336 OCEAN      3        3.96           1.84    0.157              0.00284 
## 337 ELF        4        1.83           3.52    0.856              0.684   
## 338 ETN        3        4.51           1.93    0.00000218         0.628   
## 339 IQN        3        0.682          1.31    0.548              0.387   
## 340 REEF       4        0.920          0.977   0.716              0.332   
## 341 VIB        4        1.02           1.09    0.956              0.0547  
## 342 WBTC       3      148.             0.906   0.0228             0.166   
## 343 SRN        4        2.43           1.14    0.115              0.324   
## 344 RSR        4        0.930          0.597   0.556              0.875   
## 345 EGLD       3        1.62           1.32    0.0784             0.753   
## 346 XVG        4        0.688          0.489   0.696              0.741   
## 347 CROOLD     4        1.01           1.17    0.506              0.0557  
## 348 TV         4        4.98          11.4     0.0523             0.00474 
## 349 OKB        4       11.5            0.0913  0.0756             0.0355  
## 350 ONG        4        2.44           2.87    0.634              0.281   
## 351 MDX        4        1.36           1.24    0.360              0.0363  
## 352 LPT        4      NaN              0.885  NA                  0.233   
## 353 CRPT       4        6.29           7.51    0.0314             0.225   
## 354 JULD       4       16.4            1.88    0.101              0.134   
## 355 ORN        4        0.727          1.34    0.439              0.834   
## 356 EVX        4        2.39           5.62    0.308              0.0160  
## 357 KLV        4        3.57           1.68    0.155              0.000924
## 358 MITH       4        4.68           1.30    0.106              0.00858 
## 359 ARPA       4        1.17           1.04    0.312              0.414   
## 360 IHF        4        0.440          3.52    0.591              0.00440 
## 361 XYM        4        1.08           1.03    0.0280             0.301   
## 362 NEO        4        2.14           1.09    0.00948            0.316   
## 363 COMP       4        1.30           1.03    0.465              0.747   
## 364 ZIL        4        1.86           1.33    0.103              0.133   
## 365 BOSON      4       10.7            6.45    0.346              0.619   
## 366 MAID       4        0.863          0.831   0.350              0.0256  
## 367 VLX        4        3.06           0.954   0.894              0.366   
## 368 CLT        4       11.8            1.75    0.778              0.452   
## 369 INJ        4        1.45           1.26    0.378              0.0353  
## 370 WRX        4        0.577          0.922   0.790              0.304   
## 371 ONE        4        1.22           2.08    0.968              0.338   
## 372 XCH        4        1.23           0.932   0.674              0.0518  
## 373 SENSO      4        3.77           6.89    0.553              0.0508  
## 374 LOC        4        0.317          1.27    0.802              0.00421 
## 375 FIL        4        6.19           1.11    0.482              0.231   
## 376 NEAR       4        1.58           1.91    0.752              0.0200  
## 377 DASH       4        0.597          0.939   0.909              0.403   
## 378 ALPHA      4        0.774          1.03    0.817              0.803   
## 379 1INCH      4        2.04           1.11    0.347              0.700   
## 380 CTSI       4        3.21           1.17    0.0642             0.507   
## 381 SXP        4        0.758          1.63    0.742              0.154   
## 382 TON        4        0.437          0.600   0.259              0.397   
## 383 DOGE       4        0.671          0.449   0.405              0.295   
## 384 DODO       4        2.63           1.50    0.986              0.124   
## 385 POND       4        1.61           1.43    0.278              0.242   
## 386 OCEAN      4        0.818          1.84    0.536              0.00284 
## 387 IQN        4        0.694          1.31    0.242              0.387   
## 388 CVCOIN     4        2.19           4.13    0.0275             0.00513 
## 389 JUV        4        1.13           1.73    0.819              0.268   
## 390 WBTC       4        0.265          0.906   0.827              0.166   
## 391 ETN        4        0.474          1.93    0.986              0.628   
## 392 LEO        5       NA            NaN      NA                 NA       
## 393 EGLD       4        4.62           1.32    0.0214             0.753   
## 394 ETP        5       NA              1.45   NA                  0.352   
## 395 SOLO       5       NA              2.27   NA                  0.544   
## 396 OAX        5       NA              1.37   NA                  0.0675  
## 397 TV         5       NA             11.4    NA                  0.00474 
## 398 ETH        5       NA              0.675  NA                  0.684   
## 399 EOS        5       NA              0.658  NA                  0.841   
## 400 LTC        5       NA              1.59   NA                  0.181   
## 401 BSV        5       NA              0.539  NA                  0.778   
## 402 ZEC        5       NA              0.703  NA                  0.836   
## 403 TRX        5       NA              2.07   NA                  0.551   
## 404 KNC        5       NA              0.685  NA                  0.759   
## 405 XMR        5       NA              0.421  NA                  0.783   
## 406 BAT        5       NA              0.695  NA                  0.567   
## 407 BNT        5       NA              0.539  NA                  0.735   
## 408 MANA       5       NA              0.359  NA                  0.850   
## 409 ENJ        5       NA              0.538  NA                  0.808   
## 410 XEM        5       NA              0.746  NA                  0.704   
## 411 ARDR       5       NA              1.21   NA                  0.671   
## 412 KMD        5       NA              1.24   NA                  0.353   
## 413 NEXO       5       NA              0.787  NA                  0.605   
## 414 CHZ        5       NA              0.667  NA                  0.591   
## 415 CKB        5       NA              1.70   NA                  0.00637 
## 416 DCR        5       NA              0.728  NA                  0.398   
## 417 AVA        5       NA              1.25   NA                  0.0360  
## 418 JST        5       NA              1.46   NA                  0.170   
## 419 RARI       5       NA              1.23   NA                  0.466   
## 420 ETC        5       NA              0.775  NA                  0.312   
## 421 BTC        5       NA              0.302  NA                  0.760   
## 422 ADA        5       NA              0.575  NA                  0.589   
## 423 ZRX        5       NA              0.826  NA                  0.853   
## 424 BTG        5       NA              0.600  NA                  0.559   
## 425 ICP        5       NA              1.01   NA                  0.451   
## 426 TLM        5       NA              0.524  NA                  0.911   
## 427 UNI        5       NA              0.639  NA                  0.805   
## 428 BIZZ       5       NA             17.2    NA                  0.0438  
## 429 LSK        5       NA              0.593  NA                  0.530   
## 430 NWC        5       NA              0.724  NA                  0.680   
## 431 COTI       5       NA              1.25   NA                  0.914   
## 432 NU         5       NA              1.25   NA                  0.241   
## 433 DGB        5       NA              0.644  NA                  0.660   
## 434 WAXP       5       NA              1.04   NA                  0.674   
## 435 PERP       5       NA              1.22   NA                  0.704   
## 436 HT         5       NA              0.553  NA                  0.550   
## 437 BRD        5       NA              0.713  NA                  0.469   
## 438 OGN        5       NA              0.790  NA                  0.838   
## 439 ELF        5       NA              3.52   NA                  0.684   
## 440 VIB        5       NA              1.09   NA                  0.0547  
## 441 ONG        5       NA              2.87   NA                  0.281   
## 442 RSR        5       NA              0.597  NA                  0.875   
## 443 CDT        5       NA              1.81   NA                  0.337   
## 444 SRN        5       NA              1.14   NA                  0.324   
## 445 REEF       5       NA              0.977  NA                  0.332   
## 446 CROOLD     5       NA              1.17   NA                  0.0557  
## 447 XVG        5       NA              0.489  NA                  0.741   
## 448 OKB        5       NA              0.0913 NA                  0.0355  
## 449 OCEAN      5       NA              1.84   NA                  0.00284 
## 450 EVX        5       NA              5.62   NA                  0.0160  
## 451 MAID       5       NA              0.831  NA                  0.0256  
## 452 CTSI       5       NA              1.17   NA                  0.507   
## 453 ARPA       5       NA              1.04   NA                  0.414   
## 454 IHF        5       NA              3.52   NA                  0.00440 
## 455 XYM        5       NA              1.03   NA                  0.301   
## 456 NEO        5       NA              1.09   NA                  0.316   
## 457 COMP       5       NA              1.03   NA                  0.747   
## 458 ZIL        5       NA              1.33   NA                  0.133   
## 459 BOSON      5       NA              6.45   NA                  0.619   
## 460 WBTC       5       NA              0.906  NA                  0.166   
## 461 CLT        5       NA              1.75   NA                  0.452   
## 462 INJ        5       NA              1.26   NA                  0.0353  
## 463 WRX        5       NA              0.922  NA                  0.304   
## 464 ONE        5       NA              2.08   NA                  0.338   
## 465 XCH        5       NA              0.932  NA                  0.0518  
## 466 SENSO      5       NA              6.89   NA                  0.0508  
## 467 LOC        5       NA              1.27   NA                  0.00421 
## 468 FIL        5       NA              1.11   NA                  0.231   
## 469 LPT        5       NA              0.885  NA                  0.233   
## 470 POND       5       NA              1.43   NA                  0.242   
## 471 NEAR       5       NA              1.91   NA                  0.0200  
## 472 DASH       5       NA              0.939  NA                  0.403   
## 473 ALPHA      5       NA              1.03   NA                  0.803   
## 474 KLV        5       NA              1.68   NA                  0.000924
## 475 1INCH      5       NA              1.11   NA                  0.700   
## 476 TON        5       NA              0.600  NA                  0.397   
## 477 MDX        5       NA              1.24   NA                  0.0363  
## 478 SXP        5       NA              1.63   NA                  0.154   
## 479 DOGE       5       NA              0.449  NA                  0.295   
## 480 DODO       5       NA              1.50   NA                  0.124   
## 481 VLX        5       NA              0.954  NA                  0.366   
## 482 CVCOIN     5       NA              4.13   NA                  0.00513 
## 483 MITH       5       NA              1.30   NA                  0.00858 
## 484 JULD       5       NA              1.88   NA                  0.134   
## 485 JUV        5       NA              1.73   NA                  0.268   
## 486 CRPT       5       NA              7.51   NA                  0.225   
## 487 IQN        5       NA              1.31   NA                  0.387   
## 488 ORN        5       NA              1.34   NA                  0.834   
## 489 ETN        5       NA              1.93   NA                  0.628   
## 490 EGLD       5       NA              1.32   NA                  0.753

Out of 490 groups, 229 had an equal or lower RMSE score for the holdout than the test set.

8.4 Adjust Prices - All Models

Let’s repeat the same steps that we outlined above for all models.

8.4.1 Add Last Price

cryptodata_nested <- mutate(cryptodata_nested,
                            # XGBoost:
                            xgb_test_predictions = ifelse(split < 5,
                                                         map2(train_data, xgb_test_predictions, last_train_price),
                                                         NA),
                            # Neural Network:
                            nnet_test_predictions = ifelse(split < 5,
                                                         map2(train_data, nnet_test_predictions, last_train_price),
                                                         NA),
                            # Random Forest:
                            rf_test_predictions = ifelse(split < 5,
                                                         map2(train_data, rf_test_predictions, last_train_price),
                                                         NA),
                            # PCR:
                            pcr_test_predictions = ifelse(split < 5,
                                                         map2(train_data, pcr_test_predictions, last_train_price),
                                                         NA))
8.4.1.0.1 Holdout
cryptodata_nested_holdout <- mutate(filter(cryptodata_nested, split == 5),
                                    # XGBoost:
                                    xgb_holdout_predictions = map2(train_data, xgb_holdout_predictions, last_train_price),
                                    # Neural Network:
                                    nnet_holdout_predictions = map2(train_data, nnet_holdout_predictions, last_train_price),
                                    # Random Forest:
                                    rf_holdout_predictions = map2(train_data, rf_holdout_predictions, last_train_price),
                                    # PCR:
                                    pcr_holdout_predictions = map2(train_data, pcr_holdout_predictions, last_train_price))

Join the holdout data to all rows based on the cryptocurrency symbol alone:

cryptodata_nested <- left_join(cryptodata_nested, 
                               select(cryptodata_nested_holdout, symbol, 
                                      xgb_holdout_predictions, nnet_holdout_predictions, 
                                      rf_holdout_predictions, pcr_holdout_predictions),
                               by='symbol')
# Remove unwanted columns
cryptodata_nested <- select(cryptodata_nested, -xgb_holdout_predictions.x, 
                            -nnet_holdout_predictions.x,-rf_holdout_predictions.x, 
                            -pcr_holdout_predictions.x, -split.y)
# Rename the columns kept
cryptodata_nested <- rename(cryptodata_nested, 
                            xgb_holdout_predictions = 'xgb_holdout_predictions.y',
                            nnet_holdout_predictions = 'nnet_holdout_predictions.y',
                            rf_holdout_predictions = 'rf_holdout_predictions.y',
                            pcr_holdout_predictions = 'pcr_holdout_predictions.y',
                            split = 'split.x')
# Reset the correct grouping structure
cryptodata_nested <- group_by(cryptodata_nested, symbol, split)

8.4.2 Convert to % Change

Overwrite the old predictions with the new predictions adjusted as a percentage now:

cryptodata_nested <- mutate(cryptodata_nested,
                            # XGBoost:
                            xgb_test_predictions = ifelse(split < 5,
                                                         map(xgb_test_predictions, standardize_perc_change),
                                                         NA),
                            # holdout - all splits
                            xgb_holdout_predictions = map(xgb_holdout_predictions, standardize_perc_change),
                            # nnet:
                            nnet_test_predictions = ifelse(split < 5,
                                                         map(nnet_test_predictions, standardize_perc_change),
                                                         NA),
                            # holdout - all splits
                            nnet_holdout_predictions = map(nnet_holdout_predictions, standardize_perc_change),
                            # Random Forest:
                            rf_test_predictions = ifelse(split < 5,
                                                         map(rf_test_predictions, standardize_perc_change),
                                                         NA),
                            # holdout - all splits
                            rf_holdout_predictions = map(rf_holdout_predictions, standardize_perc_change),
                            # PCR:
                            pcr_test_predictions = ifelse(split < 5,
                                                         map(pcr_test_predictions, standardize_perc_change),
                                                         NA),
                            # holdout - all splits
                            pcr_holdout_predictions = map(pcr_holdout_predictions, standardize_perc_change))

8.4.3 Add Metrics

Add the RMSE and \(R^2\) metrics:

cryptodata_nested <- mutate(cryptodata_nested,
                            # XGBoost - RMSE - Test
                            xgb_rmse_test = unlist(ifelse(split < 5,
                                                         map2(xgb_test_predictions, actuals_test, evaluate_preds_rmse),
                                                         NA)),
                            # And holdout:
                            xgb_rmse_holdout = unlist(map2(xgb_holdout_predictions, actuals_holdout ,evaluate_preds_rmse)),
                            # XGBoost - R^2 - Test
                            xgb_rsq_test = unlist(ifelse(split < 5,
                                                         map2(xgb_test_predictions, actuals_test, evaluate_preds_rsq),
                                                         NA)),
                            # And holdout:
                            xgb_rsq_holdout = unlist(map2(xgb_holdout_predictions, actuals_holdout, evaluate_preds_rsq)),
                            # Neural Network - RMSE - Test
                            nnet_rmse_test = unlist(ifelse(split < 5,
                                                         map2(nnet_test_predictions, actuals_test, evaluate_preds_rmse),
                                                         NA)),
                            # And holdout:
                            nnet_rmse_holdout = unlist(map2(nnet_holdout_predictions, actuals_holdout, evaluate_preds_rmse)),
                            # Neural Network - R^2 - Test
                            nnet_rsq_test = unlist(ifelse(split < 5,
                                                         map2(nnet_test_predictions, actuals_test, evaluate_preds_rsq),
                                                         NA)),
                            # And holdout:
                            nnet_rsq_holdout = unlist(map2(nnet_holdout_predictions, actuals_holdout, evaluate_preds_rsq)),
                            # Random Forest - RMSE - Test
                            rf_rmse_test = unlist(ifelse(split < 5,
                                                         map2(rf_test_predictions, actuals_test, evaluate_preds_rmse),
                                                         NA)),
                            # And holdout:
                            rf_rmse_holdout = unlist(map2(rf_holdout_predictions, actuals_holdout, evaluate_preds_rmse)),
                            # Random Forest - R^2 - Test
                            rf_rsq_test = unlist(ifelse(split < 5,
                                                         map2(rf_test_predictions, actuals_test, evaluate_preds_rsq),
                                                         NA)),
                            # And holdout:
                            rf_rsq_holdout = unlist(map2(rf_holdout_predictions, actuals_holdout, evaluate_preds_rsq)),
                            # PCR - RMSE - Test
                            pcr_rmse_test = unlist(ifelse(split < 5,
                                                         map2(pcr_test_predictions, actuals_test, evaluate_preds_rmse),
                                                         NA)),
                            # And holdout:
                            pcr_rmse_holdout = unlist(map2(pcr_holdout_predictions, actuals_holdout, evaluate_preds_rmse)),
                            # PCR - R^2 - Test
                            pcr_rsq_test = unlist(ifelse(split < 5,
                                                         map2(pcr_test_predictions, actuals_test, evaluate_preds_rsq),
                                                         NA)),
                            # And holdout:
                            pcr_rsq_holdout = unlist(map2(pcr_holdout_predictions, actuals_holdout, evaluate_preds_rsq)))

Now we have RMSE and \(R^2\) values for every model created for every cryptocurrency and split of the data:

select(cryptodata_nested, lm_rmse_test, lm_rsq_test, lm_rmse_holdout, lm_rsq_holdout)
## # A tibble: 490 x 6
## # Groups:   symbol, split [490]
##    symbol split lm_rmse_test lm_rsq_test lm_rmse_holdout lm_rsq_holdout
##    <chr>  <dbl>        <dbl>       <dbl>           <dbl>          <dbl>
##  1 BTC        1        0.411      0.608            0.302          0.760
##  2 ETH        1        0.471      0.682            0.675          0.684
##  3 EOS        1        1.70       0.0121           0.658          0.841
##  4 LTC        1        0.693      0.790            1.59           0.181
##  5 BSV        1        0.587      0.459            0.539          0.778
##  6 ADA        1        0.589      0.591            0.575          0.589
##  7 TRX        1        0.552      0.623            2.07           0.551
##  8 ZEC        1        1.01       0.528            0.703          0.836
##  9 HT         1        0.658      0.631            0.553          0.550
## 10 XMR        1        0.585      0.488            0.421          0.783
## # ... with 480 more rows

Only the results for the linear regression model are shown. There are equivalent columns for the XGBoost, neural network, random forest and PCR models.

8.5 Evaluate Metrics Across Splits

Next, let’s evaluate the metrics across all splits and keeping moving along with the model validation plan as was originally outlined. Let’s create a new dataset called [cryptodata_metrics][splits]{style=“color: blue;”} that is not grouped by the split column and is instead only grouped by the symbol:

cryptodata_metrics <- group_by(select(ungroup(cryptodata_nested),-split),symbol)

8.5.1 Evaluate RMSE Test

Now for each model we can create a new column giving the average RMSE for the 4 cross-validation test splits:

rmse_test <- mutate(cryptodata_metrics,
                      lm = mean(lm_rmse_test, na.rm = T),
                      xgb = mean(xgb_rmse_test, na.rm = T),
                      nnet = mean(nnet_rmse_test, na.rm = T),
                      rf = mean(rf_rmse_test, na.rm = T),
                      pcr = mean(pcr_rmse_test, na.rm = T))

Now we can use the gather() function to summarize the columns as rows:

rmse_test <- unique(gather(select(rmse_test, lm:pcr), 'model', 'rmse', -symbol))
# Show results
rmse_test
## # A tibble: 490 x 3
## # Groups:   symbol [98]
##    symbol model  rmse
##    <chr>  <chr> <dbl>
##  1 BTC    lm    0.423
##  2 ETH    lm    0.729
##  3 EOS    lm    1.22 
##  4 LTC    lm    0.753
##  5 BSV    lm    0.742
##  6 ADA    lm    0.644
##  7 TRX    lm    1.00 
##  8 ZEC    lm    0.887
##  9 HT     lm    0.808
## 10 XMR    lm    0.566
## # ... with 480 more rows

Now tag the results as having been for the test set:

rmse_test$eval_set <- 'test'

8.5.2 Holdout

Now repeat the same process for the holdout set:

rmse_holdout <- mutate(cryptodata_metrics,
                      lm = mean(lm_rmse_holdout, na.rm = T),
                      xgb = mean(xgb_rmse_holdout, na.rm = T),
                      nnet = mean(nnet_rmse_holdout, na.rm = T),
                      rf = mean(rf_rmse_holdout, na.rm = T),
                      pcr = mean(pcr_rmse_holdout, na.rm = T))

Again, use the gather() function to summarize the columns as rows:

rmse_holdout <- unique(gather(select(rmse_holdout, lm:pcr), 'model', 'rmse', -symbol))
# Show results
rmse_holdout
## # A tibble: 490 x 3
## # Groups:   symbol [98]
##    symbol model  rmse
##    <chr>  <chr> <dbl>
##  1 BTC    lm    0.302
##  2 ETH    lm    0.675
##  3 EOS    lm    0.658
##  4 LTC    lm    1.59 
##  5 BSV    lm    0.539
##  6 ADA    lm    0.575
##  7 TRX    lm    2.07 
##  8 ZEC    lm    0.703
##  9 HT     lm    0.553
## 10 XMR    lm    0.421
## # ... with 480 more rows

Now tag the results as having been for the holdout set:

rmse_holdout$eval_set <- 'holdout'

8.5.3 Union Results

Now we can union() the results to stack the rows from the two datasets on top of each other:

rmse_scores <- union(rmse_test, rmse_holdout)

8.6 Evaluate R^2

Now let’s repeat the same steps we took for the RMSE metrics above for the \(R^2\) metric as well.

8.6.1 Test

For each model again we will create a new column giving the average \(R^2\) for the 4 cross-validation test splits:

rsq_test <- mutate(cryptodata_metrics,
                      lm = mean(lm_rsq_test, na.rm = T),
                      xgb = mean(xgb_rsq_test, na.rm = T),
                      nnet = mean(nnet_rsq_test, na.rm = T),
                      rf = mean(rf_rsq_test, na.rm = T),
                      pcr = mean(pcr_rsq_test, na.rm = T))

Now we can use the gather() function to summarize the columns as rows:

rsq_test <- unique(gather(select(rsq_test, lm:pcr), 'model', 'rsq', -symbol))
# Show results
rsq_test
## # A tibble: 490 x 3
## # Groups:   symbol [98]
##    symbol model   rsq
##    <chr>  <chr> <dbl>
##  1 BTC    lm    0.691
##  2 ETH    lm    0.804
##  3 EOS    lm    0.472
##  4 LTC    lm    0.658
##  5 BSV    lm    0.433
##  6 ADA    lm    0.785
##  7 TRX    lm    0.595
##  8 ZEC    lm    0.580
##  9 HT     lm    0.621
## 10 XMR    lm    0.686
## # ... with 480 more rows

Now tag the results as having been for the test set

rsq_test$eval_set <- 'test'

8.6.2 Holdout

Do the same and calculate the averages for the holdout sets:

rsq_holdout <- mutate(cryptodata_metrics,
                      lm = mean(lm_rsq_holdout, na.rm = T),
                      xgb = mean(xgb_rsq_holdout, na.rm = T),
                      nnet = mean(nnet_rsq_holdout, na.rm = T),
                      rf = mean(rf_rsq_holdout, na.rm = T),
                      pcr = mean(pcr_rsq_holdout, na.rm = T))

Now we can use the gather() function to summarize the columns as rows:

rsq_holdout <- unique(gather(select(rsq_holdout, lm:pcr), 'model', 'rsq', -symbol))
# Show results
rsq_holdout
## # A tibble: 490 x 3
## # Groups:   symbol [98]
##    symbol model   rsq
##    <chr>  <chr> <dbl>
##  1 BTC    lm    0.760
##  2 ETH    lm    0.684
##  3 EOS    lm    0.841
##  4 LTC    lm    0.181
##  5 BSV    lm    0.778
##  6 ADA    lm    0.589
##  7 TRX    lm    0.551
##  8 ZEC    lm    0.836
##  9 HT     lm    0.550
## 10 XMR    lm    0.783
## # ... with 480 more rows

Now tag the results as having been for the holdout set:

rsq_holdout$eval_set <- 'holdout'

8.6.3 Union Results

rsq_scores <- union(rsq_test, rsq_holdout)

8.7 Visualize Results

Now we can take the same tools we learned in the Visualization section from earlier and visualize the results of the models.

8.7.1 RMSE Visualization

8.7.2 Both

Now we have everything we need to use the two metrics to compare the results.

8.7.2.1 Join Datasets

First join the two objects rmse_scores and rsq_scores into the new object **plot_scores:

plot_scores <- merge(rmse_scores, rsq_scores)

8.7.2.2 Plot Results

Now we can plot the results on a chart:

ggplot(plot_scores, aes(x=rsq, y=rmse, color = model)) +
  geom_point() +
  ylim(c(0,10))

Running the same code wrapped in the ggplotly() function from the plotly package (as we have already done) we can make the chart interactive. Try hovering over the points on the chart with your mouse.

ggplotly(ggplot(plot_scores, aes(x=rsq, y=rmse, color = model, symbol = symbol)) +
           geom_point() +
           ylim(c(0,10)),
         tooltip = c("model", "symbol", "rmse", "rsq"))

The additional tooltip argument was passed to ggpltoly() to specify the label when hovering over the individual points.

8.7.3 Results by the Cryptocurrency

We can use the facet_wrap() function from ggplot2 to create an individual chart for each cryptocurrency:

ggplot(plot_scores, aes(x=rsq, y=rmse, color = model)) +
  geom_point() +
  geom_smooth() +
  ylim(c(0,10)) +
  facet_wrap(~symbol)

Every 12 hours once this document reaches this point, the results are saved to GitHub using the pins package (which we used to read in the data at the start), and a separate script running on a different server creates the complete dataset in our database over time. You won’t be able to run the code shown below (nor do you have a reason to):

# register board
board_register("github", repo = "predictcrypto/pins", token=pins_key)
# Add current date time
plot_scores$last_refreshed <- Sys.time()
# pin data
pin(plot_scores, board='github', name='crypto_tutorial_results_latest')

8.8 Interactive Dashboard

Use the interactive app below to explore the results over time by the individual cryptocurrency. Use the filters on the left sidebar to visualize the results you are interested in:

If you have trouble viewing the embedded dashboard you can open it here instead: https://predictcrypto.shinyapps.io/tutorial_latest_model_summary/

The default view shows the holdout results for all models. Another interesting comparison to make is between the holdout and the test set for fewer models (2 is ideal).

The app shown above also has a button to Show Code. If you were to show the code and copy and paste it into an RStudio session on your computer into a file with the .Rmd file extension and you then Knit the file, the same exact app should show up on your computer, no logins or setup outside of the packages required for the code to run; RStudio should automatically prompt you to install packages that are not currently installed on your computer.

8.9 Visualizations - Historical Metrics

We can pull the same data into this R session using the pin_get() function:

metrics_historical <- pin_get(name = "full_metrics")

The data is limited to metrics for runs from the past 30 days and includes new data every 12 hours. Using the tools we used in the data prep section, we can answer a couple more questions.

8.9.1 Best Models

Overall, which model has the best metrics for all runs from the last 30 days?

8.9.1.1 Summarize the data

# First create grouped data
best_models <- group_by(metrics_historical, model, eval_set)
# Now summarize the data
best_models <- summarize(best_models,
                         rmse = mean(rmse, na.rm=T),
                         rsq  = mean(rsq, na.rm=T))
# Show results
best_models
## # A tibble: 10 x 4
## # Groups:   model [5]
##    model eval_set  rmse    rsq
##    <chr> <chr>    <dbl>  <dbl>
##  1 lm    holdout  15.5  0.506 
##  2 lm    test      4.07 0.478 
##  3 nnet  holdout   4.31 0.149 
##  4 nnet  test      4.63 0.164 
##  5 pcr   holdout   2.72 0.252 
##  6 pcr   test      2.92 0.278 
##  7 rf    holdout   3.96 0.114 
##  8 rf    test      3.83 0.129 
##  9 xgb   holdout   5.01 0.0693
## 10 xgb   test      4.56 0.0885

8.9.1.2 Plot RMSE by Model

ggplot(best_models, aes(model, rmse, fill = eval_set)) + 
  geom_bar(stat = "identity", position = 'dodge') +
  ggtitle('RMSE by Model', 'Comparing Test and Holdout')

8.9.1.3 Plot \(R^2\) by Model

ggplot(best_models, aes(model, rsq, fill = eval_set)) + 
  geom_bar(stat = "identity", position = 'dodge') +
  ggtitle('R^2 by Model', 'Comparing Test and Holdout')

8.9.2 Most Predictable Cryptocurrency

Overall, which cryptocurrency has the best metrics for all runs from the last 30 days?

8.9.2.1 Summarize the data

# First create grouped data
predictable_cryptos <- group_by(metrics_historical, symbol, eval_set)
# Now summarize the data
predictable_cryptos <- summarize(predictable_cryptos,
                         rmse = mean(rmse, na.rm=T),
                         rsq  = mean(rsq, na.rm=T))
# Arrange from most predictable (according to R^2) to least 
predictable_cryptos <- arrange(predictable_cryptos, desc(rsq))
# Show results
predictable_cryptos
## # A tibble: 178 x 4
## # Groups:   symbol [89]
##    symbol eval_set  rmse   rsq
##    <chr>  <chr>    <dbl> <dbl>
##  1 NAV    test      3.30 0.434
##  2 POA    holdout   4.60 0.423
##  3 CUR    holdout   6.09 0.410
##  4 CND    test      1.84 0.374
##  5 CND    holdout   5.24 0.360
##  6 SEELE  holdout   8.88 0.355
##  7 ADXN   test      9.26 0.348
##  8 RCN    test      5.03 0.337
##  9 BTC    test      1.32 0.331
## 10 SUN    holdout   3.17 0.330
## # ... with 168 more rows

Show the top 15 most predictable cryptocurrencies (according to the \(R^2\)) using the formattable package (Ren and Russell 2016) to color code the cells:

formattable(head(predictable_cryptos ,15), 
            list(rmse = color_tile("#71CA97", "red"),
                 rsq = color_tile("firebrick1", "#71CA97")))
symbol eval_set rmse rsq
NAV test 3.299791 0.4338237
POA holdout 4.596582 0.4229192
CUR holdout 6.088416 0.4098434
CND test 1.835020 0.3737691
CND holdout 5.238670 0.3601346
SEELE holdout 8.876745 0.3548372
ADXN test 9.263022 0.3478368
RCN test 5.030197 0.3367737
BTC test 1.321379 0.3307333
SUN holdout 3.172350 0.3299285
AAB test 39.511003 0.3262746
ETH test 1.721290 0.3197170
LTC test 2.102832 0.3189001
LEO test 1.695632 0.3166553
RCN holdout 7.564985 0.3130917

8.9.3 Accuracy Over Time

8.9.3.1 Summarize the data

# First create grouped data
accuracy_over_time <- group_by(metrics_historical, date_utc)
# Now summarize the data
accuracy_over_time <- summarize(accuracy_over_time, 
                                rmse = mean(rmse, na.rm=T),
                                rsq  = mean(rsq, na.rm=T))
# Ungroup data
accuracy_over_time <- ungroup(accuracy_over_time)
# Convert date/time
accuracy_over_time$date_utc <- anytime(accuracy_over_time$date_utc)
# Show results
accuracy_over_time
## # A tibble: 30 x 3
##    date_utc             rmse   rsq
##    <dttm>              <dbl> <dbl>
##  1 2021-01-12 00:00:00  4.05 0.241
##  2 2021-01-13 00:00:00  4.29 0.236
##  3 2021-01-14 00:00:00  4.06 0.251
##  4 2021-01-16 00:00:00  4.25 0.214
##  5 2021-01-17 00:00:00  3.78 0.199
##  6 2021-01-18 00:00:00  3.88 0.212
##  7 2021-01-19 00:00:00  3.86 0.204
##  8 2021-01-20 00:00:00  4.65 0.207
##  9 2021-01-21 00:00:00  3.41 0.222
## 10 2021-01-22 00:00:00  4.05 0.231
## # ... with 20 more rows

8.9.3.2 Plot RMSE

Remember, for RMSE the lower the score, the more accurate the models were.

ggplot(accuracy_over_time, aes(x = date_utc, y = rmse, group = 1)) +
  # Plot RMSE over time
  geom_point(color = 'red', size = 2) +
  geom_line(color = 'red', size = 1)

8.9.3.3 Plot R^2

For the R^2 recall that we are looking at the correlation between the predictions made and what actually happened, so the higher the score the better, with a maximum score of 1 that would mean the predictions were 100% correlated with each other and therefore identical.

ggplot(accuracy_over_time, aes(x = date_utc, y = rsq, group = 1)) +
  # Plot R^2 over time
  geom_point(aes(x = date_utc, y = rsq), color = 'dark green', size = 2) +
  geom_line(aes(x = date_utc, y = rsq), color = 'dark green', size = 1)

Refer back to the interactive dashboard to take a more specific subset of results instead of the aggregate analysis shown above.

References

Ren, Kun, and Kenton Russell. 2016. Formattable: Create Formattable Data Structures. https://CRAN.R-project.org/package=formattable.