Recently Kaggle launched two new competitions with time-series data - Santander Product Recommendation and Outbrain Click Prediction . And as with so many time-series competitions a choice of cross-validation strategy will be decisive factor. It is right time to look at cross-validation for time-series data on an example of a past competition. In the first part of the post we looked at the general approach to time-series competition, on how to create features using time dependencies. And conclusion was that one can create a lot of features, so many features that it will be really easy to overfit, so a solid cross-validation strategy is extremely important. In this part I will use a recent Kaggle competition Grupo Bimbo Inventory Demand to look at possible approaches to cross-validation with time-series datasets.
Bimbo Group supplies over 1 million stores in Mexico with bakery and related products. In the competition Bimbo Group asked participants to predict how much products they should supply to each of their clients(stores). And they gave a lot of data - the train set contained 75 millions rows. We are given 7 weeks of sales of Bimbo to their clients. We are also given other data (such as returns of non-sold products). The full description of the competition and its datasets are on the competition web page. But I’d like to concentrate on cross-validation and omit less relevant details. So for the purpose of this blog I’ve made the following simplifications to the data:
- the columns were renamed to English equivalents
- the target is Sales in pesos
- features like sales in units and returns are ignored
- the number of categorical feature reduced
- I do not distinguish between a client and a store
- the weeks numbering starts from 1
So the head of the modified train set looks like
| Week | Depo | Product | Client | Sales |
|---|---|---|---|---|
| 1 | 1110 | 1453543 | 1242 | 45.84 |
| 1 | 1110 | 9503385 | 5380 | 14.82 |
| 1 | 1110 | 1159580 | 30571 | 68.75 |
| 1 | 1110 | 198780 | 42434 | 45.8 |
| 1 | 1110 | 9503385 | 37057 | 15 |
| 1 | 1110 | 18641 | 31719 | 30.36 |
| 1 | 1110 | 1457784 | 30571 | 43.75 |
| 1 | 1110 | 817243 | 1146 | 21.39 |
| 1 | 1110 | 181232 | 1309 | 13.52 |
It’s just first 8 rows of the train set and in this competition we were given the data set of 75 millions of rows. Which on the one hand looks like a lot but on the other hand - considering that there are almost 900,000 of clients and 2,600 products - it is not really much. For example, when it comes to tuples (Product, Client) the data are sparse and indeed there are more that millions of such tuples which are in the test set and not presented in the train set at all. And the size of the data and this sparsity are important when choosing cross-validation strategy.
But the most important factor when deciding on a cross-validation approach of any competition is how the test set is split to the public and private subsets.
For those who has never taken part in Kaggle competition - a few words about what private/test split means. The final standing of a Kaggle competition decided by how well participants models performed on the test set - for which, obviously, the competitors do not know the target variable. During a competition the preliminary ranking of the participants could be seen on the public leaderboard. This lederboard shows how submitted solutions scored on a portion of the test set. This portion is called public test set. But the final results of the competition will be decided by the score on the private portion of the full test set. And the difference between the preliminary standing on the public leaderboard and the final standing on the private leaderboard might be significant, very significant - participants could lose hundreds of places. For example, only three competitors who were in the top ten by the public LB made it to the final top ten of Rossmann Store Sales competition, which also was a time-series one.
The usual problem is that participants overfit to the public test set.
It helps to avoid overfitting when you know how the test set was split. On many competitions the organizers split the test set randomly - like 30% public portion, 70% private portion. But for time series competitions more natural way of splitting the test set is by time.
And the test set of Bimbo competition was split by time.
In this competition we were given seven weeks of data to train a model and we were asked to predict the sales for the following two weeks.
The competitions organizers clearly stated that test set split about 50%/50% private/public and the split is by time. It means that one of the test set weeks is used for the public leaderboard and the other for the private leaderboard. The organized did not say which week is private which is public. But one can spend just two submissions to easily figure that out. In case of Bimbo the first week of the test set was public and the second week was private.
We want our cross-validation score to tell us how our model will perform on the private leaderboard - and it means that cross-validation arrangements should reflect how the final evaluation of the competition is done. But with time-series data not only cross-validation but the entire approach to the competition depends on how the test set is split.
Why?
Because the final result depends only on how well the model performs on the second week of the test set, for the final standing it is not important how well you predicted the first week. It’s maybe easier to predict the fist week of the test set as it’s closer to the train set - but the final score is decided by the second week. And as I illustrated in the first part we will have less features for the second week of the test set as we just do not have Sales of the previous week, we can use only Sales with lag 2 or more to engineer feature. And it might mean that we should have two different models - one for the first week of the test set and the other for the second week. But when we know that only the second week matters we actually do not need to build model for the first week. But we also should understand that some participants of the competition will do that and they will look good at the public leaderboard.
So we have made a decision to build a model to predict only the second week of the test set. It means that for feature engineering we can only use Sales with lag two and more, we cannot use the immediately preceding week, because we do not have sales for the week immediately before the second week of the test set.
And we should make sure that our validation set is not contaminated with the data which the second week of test set cannot have. And such contamination could be subtle. For example, some common approaches to encoding categorical variables could leak information to validation sets - and the cv score would be overoptimistic.
Ideally we would like our cross-validation arrangements to emulate private/public test split as close as possible. In case of Bimbo it means that we should hold the last two weeks of the train set as the validation set and we will train model on the remaining data and when predicting the holdout set we will have honest, but slighly pessimistic, estimation of of public score and private score. The first week of the validation set is an analogue of the public portion of the test set, and the second week corresponds to the public part of the test set. It’s even possible to estimate the difference between performance of the model on the public and private leaderboards.
But we rarely can do ideal things. There is a question - how much pessimistic is that slighlty. And with this ideal approach it could be too much.
We have 7 weeks of history to train our models. If we hold the last two weeks as a validation set just 5 weeks of the data are left for training. Not that much. Moreover, as we are interested in predicting the second week of the test set we will use engineered features with lag 2 or more. But the first two week of the train set do not have such features so we have reduced our train set significantly. And when it comes to aggregation features - they are not reliable when computed on just 5 weeks or even less.
And in this particular competition it was even more significant as when we look at a tuple (Product, Client) we see that in the seven weeks of the train data a particular product could be delivered to the store only for two, three or often just for one of the seven weeks. So when we withhold two weeks of the train data we lose a lot of information.
As a result cv score could become not just pessimistic but unreliable because the model would be too weak and very different from the one trained on the whole train set.
But do we really need to hold two weeks for validation set? We don’t. As we are really interested in predicting the second week of the train set we can hold for validation just one week - it will mimic the second week of the test set. We keep week 7(the last week of the train set) as the validation set and when we compute features for this week we do not not use Sales of the previous week (week 6) because the second week of the test set doesn’t have this information.
So far we have made three important decisions:
- we build model to predict only the second week of the test set
- we hold the last week of the training set as the validation set which mimics the second week of the test set
- we engineer features using lag 2 and more
One more decision to be made. Which weeks of the train set will we use to train models?
First, look at the table below
| Week | Product | Client | mean_Sales | mean_Sales_Client | mean_Sales_Product | Sales_lag_2 |
|---|---|---|---|---|---|---|
| 3 | 357 | 43507 | NA | 22.41 | 26.52 | NA |
| 3 | 30415 | 43507 | NA | 22.41 | 25.73 | NA |
| 3 | 35144 | 43507 | 20.5 | 22.41 | 56.04 | 20.5 |
| 3 | 30532 | 43507 | 38.9 | 22.41 | 48.22 | 38.9 |
| 3 | 36748 | 43507 | NA | 22.41 | 30.31 | NA |
| 3 | 43067 | 43507 | NA | 22.41 | 19.13 | NA |
| 3 | 43069 | 43507 | 7.41 | 22.41 | 34.46 | 7.41 |
| 3 | 43285 | 43507 | NA | 22.41 | 70.25 | NA |
| 4 | 357 | 43507 | 29.16 | 24.7 | 27.04 | 29.16 |
| 4 | 30415 | 43507 | 7.5 | 24.7 | 25.96 | 7.5 |
| 4 | 35141 | 43507 | 41 | 24.7 | 48.96 | NA |
| 4 | 35144 | 43507 | 20.5 | 24.7 | 55.77 | 20.5 |
| 4 | 30532 | 43507 | 42.79 | 24.7 | 50.29 | 46.68 |
| 4 | 36748 | 43507 | NA | 24.7 | 30 | NA |
| 4 | 43067 | 43507 | NA | 24.7 | 18.75 | NA |
| 4 | 43069 | 43507 | 14.82 | 24.7 | 34.62 | 22.23 |
| 4 | 43285 | 43507 | 63.36 | 24.7 | 68.89 | 63.36 |
| 5 | 357 | 43507 | 29.16 | 27.57 | 27.28 | 29.16 |
| 5 | 30415 | 43507 | 11.25 | 27.57 | 25.67 | 15 |
| 5 | 35141 | 43507 | 41 | 27.57 | 49.05 | NA |
| 5 | 35144 | 43507 | 20.5 | 27.57 | 55.01 | 20.5 |
| 5 | 30532 | 43507 | 51.87 | 27.57 | 49.96 | 70.02 |
| 5 | 36748 | 43507 | 11.53 | 27.57 | 29.79 | 11.53 |
| 5 | 43067 | 43507 | 19.26 | 27.57 | 18.63 | 19.26 |
| 5 | 43069 | 43507 | 12.35 | 27.57 | 33.81 | 7.41 |
| 5 | 43285 | 43507 | 84.48 | 27.57 | 68.73 | 105.6 |
| 6 | 357 | 43507 | 29.16 | 26.33 | 27.12 | 29.16 |
| 6 | 35141 | 43507 | 30.75 | 26.33 | 49 | 20.5 |
| 6 | 35144 | 43507 | 20.5 | 26.33 | 54.2 | 20.5 |
| 6 | 30532 | 43507 | 46.68 | 26.33 | 49.85 | 31.12 |
| 6 | 43285 | 43507 | 73.92 | 26.33 | 68.63 | 52.8 |
the table illustrates some points about the choices we discussed above. I do not include weeks 1 and 2 as all the features computed with lag of two weeks are empty. And wee see that for week 3 a lot of features are empty as well. And all aggregation features for week 3 were computed just for one week - so they are not really aggregations. In this particular competition and usually with time-series data - central tendencies features like mean or median are the most powerful. But they need enough data-points to become reliable. It means that weeks 3,4 are just do not have robust features.
So we are left with weeks 5,6,7 for final models, and with week 5,6 for cross-validation models.
We can train model on weeks 5,6,7 , or 6,7 or just 7. We can do all those combinations and than use the predictions on the ensemble level.
Still we cannot reliably compare performance of the model trained on weeks 5,6,7 and the model on week 6,7 using just cross-validation.
But in this case a feedback of public leaderboard could be very helpful.
I would like to elaborate on that.
I already mentioned that public LB is based on the first week of the test set - and this week is easier to predict than the second as it is closer to train set and we have more information to train a model. But there is not much sense to look good at the public LB but fail miserably at the private LB.
What we could do instead - is to train a separate model for the first week of the test set - which does not use the last week of the train set at all. Yes, we throw away a whole week of the train set but we treat the first week of the test as the second. So score on the public leaderboard will a become much better estimate of the private score, and also now we can evaluate how well our CV procedure works and adjust it if we see problems.
Summing everything up we have the following cross-validation procedure for Bimbo competition:
- We keep the last week of train set(week 7) as validation set.
- We compute features for the validation set and for training set with lag 2 or more (the same condition as for private part of the test set)
- We train model for validation using week 6 (and features based on weeks 1-4). We make prediction and compute score for week 7. We use the same model to predict the first week of the test set(week 8).
- For the second week of the test set(week 9) we build a new model, this time using all the data of the train set but still all lagged features are computed with lag 2 or more. We train this model on the last week of the train set.
With this approach we have reliable cross-validation plus public LB score becomes a good proxy for private LB score.
Such approach is conservative, and CV score would be a little bit pessimistic - and there might be reason when you want to be more aggressive. But it is not the absolute cross-validation score of a model which matters it is that the CV score moves in the same direction as the test score - and with this approach and using LB set as additional validation one is much less likely to overfit.
It has been a long post but having participated in several competitions with time-series data I have learnt that it is much more effective to invest time, run some experiments to get a solid cross-validation strategy at the very beginning of any time-series competition. Feature engineering, hyper-parameters tuning, stacking will work only if you can rely on your cross-validation score.
P.S.
Cross-validation approach described in the post works well but mostly for one-level models. For any ensemble which uses something more complex than a weighted average cross-validation becomes much more complex. And cross-validation strategy for ensembles is a huge topic on itself.
No comments:
Post a Comment