Arsthbt

I have written an ensemble using Scikit Learn VotingClassifier.

I have set a seed in the cross validation section. However, it does not appear to 'hold'. Meaning, If I re-run the code block I get different results. (I can only assume each run of the code block is dividing the dataset into folds with different constituents instead of 'freezing' the random state.

Here is the code:

#Voting Ensemble of Classification
#Create Submodels
num_folds = 10
seed =7
kfold = KFold(n_splits=num_folds, random_state=seed)
estimators = []
model1 =LogisticRegression()
estimators.append(('LR',model1))
model2 = KNeighborsClassifier()
estimators.append(('KNN',model2))
model3 = GradientBoostingClassifier()
estimators.append(('GBM',model3))
#Create the ensemble
ensemble = VotingClassifier(estimators,voting='soft')
results = cross_val_score(ensemble, X_train, Y_train,cv=kfold)
print(results) 

The results printed are the results of the 10 CV fold training. If I run this code block several times I get the following results:

1:

[0.70588235 0.94117647 1. 0.82352941 0.94117647 0.88235294
 0.8125 0.875 0.8125 0.9375 ]

2:

[0.76470588 0.94117647 1. 0.82352941 0.94117647 0.88235294
 0.8125 0.875 0.8125 0.875 ]

3:

[0.76470588 0.94117647 1. 0.82352941 0.94117647 0.88235294
 0.8125 0.875 0.8125 0.875 ]

4:

[0.76470588 0.94117647 1. 0.82352941 1. 0.88235294
 0.8125 0.875 0.625 0.875 ]

So it appears my random_state=seed isn't holding.

What is incorrect?

Thanks in advance.

Random seed of models (LogisticRegression, GradientBoostingClassifier) needs to be fixed too, so that their random behavior becomes reproducible. Here is a working example that produces the same result over multiple runs:

import sklearn
from sklearn.model_selection import KFold, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import GradientBoostingClassifier, VotingClassifier
import numpy as np

#Voting Ensemble of Classification
#Create Submodels
num_folds = 10
seed =7

# Data
np.random.seed(seed)
feature_1 = np.random.normal(0, 2, 10000)
feature_2 = np.random.normal(5, 6, 10000)
X_train = np.vstack([feature_1, feature_2]).T
Y_train = np.random.randint(0, 2, 10000).T

kfold = KFold(n_splits=num_folds, random_state=seed)
estimators = []
model1 =LogisticRegression(random_state=seed)
estimators.append(('LR',model1))
model2 = KNeighborsClassifier()
estimators.append(('KNN',model2))
model3 = GradientBoostingClassifier(random_state=seed)
estimators.append(('GBM',model3))
#Create the ensemble
ensemble = VotingClassifier(estimators,voting='soft')
results = cross_val_score(ensemble, X_train, Y_train,cv=kfold)
print('sklearn version', sklearn.__version__)
print(results)

Output:

sklearn version 0.19.1
[0.502 0.496 0.483 0.513 0.515 0.508 0.517 0.499 0.515 0.504]