Nsryjdtyk

I trained a model on a set and tried to use it on all subsets.

Mathematically the total rmse and mae (mean average error) should be in between the single rsme's and mae's. But all the single rmse' and mae's are higher than the total one.

I did the following:

%pyspark

def preprocessing(features, attributes):



    features_2 = features[attributes]

    y = features['y'].values

    x = features_2.values 



    robustScaler = RobustScaler(quantile_range=(25.0,75.0))

    xScaled = robustScaler.fit_transform(x[:,1:x.shape[1]])



    xScaled[xScaled < -2.0] = -2.0 

    xScaled[xScaled > 2.0] = 2.0

    xCustomers = x[:,0]

    xCustomers_reshaped = xCustomers.reshape((x[:,0].size, 1)) 

    x_TS = xScaled 

    x_T0 = xScaled[:,:] 

    x_T0_all = np.hstack((np.ones((x_T0.shape[0], 1)), x_T0, x_T0**2, x_T0**3)) 

    xCustR = xCustomers.reshape((x[:,0].size, 1)) 

    x_TS_all = np.hstack((xCustR*np.ones((x_TS.shape[0], 1)), xCustR*x_TS, xCustR*(x_TS**2), xCustR*(x_TS**3))) 

    x_all = np.hstack((x_T0_all, x_TS_all))

    variable_names = features_2.columns.get_values()[1:].tolist() 

    return x_all, variable_names, y



def trainModel(features,attributes,optAlpha):

    x_all, variable_names, y = preprocessing(features, attributes)

    ridge = linear_model.Ridge(fit_intercept=False, copy_X=True, alpha=optAlpha, solver='auto')

    ridge.fit(x_all, y)

    return ridge



def useModel(features,ridge,attributes):

    x_all, variable_names, y = preprocessing(features, attributes)

    y_pred = ridge.predict(x_all)

    rmse = np.sqrt(mean_squared_error(y,y_pred))

    mae = mean_absolute_error(y, y_pred)    

    print "RMSE on test set: ", round(rmse,2)

    print "MAE on test set:  ", round(mae,2)

    return y_pred, y, rmse, mae



ridge = trainModel(df_features_train, attributes, optAlpha)

useModel(df_features_train,ridge,attributes)



RMSE on test set:  67.05

MAE on test set:   52.5

Now I tried to use the useModel-function including the preprocessing on all different orgIDs separately.

orgIDError = pd.DataFrame(,columns=['orgID','rmse','mae'])



for orgID in df_features['orgID'].unique():

    yPred, y, rmse, mae = useModel(df_features_train[df_features_train.orgID == orgID],ridge,attributes)

    df = pd.DataFrame([[orgID,rmse,mae]],columns=['orgID','rmse','mae'])

    orgIDError = orgIDError.append(df)

print(orgIDError)



   orgID       rmse          mae

0  615   194.848564   155.502885

0  577   101.156573    76.083797

0  957  1564.256952   814.316566

0  763   832.782755   501.865561

0  616  1337.456555   860.404253

0  968   526.207558   347.265139

0  954  1570.315284  1149.191017

0  874   241.254153   202.429037

0  554   402.013992   344.846957

0  950  1073.348186   673.874603

Any ideas what went wrong?

I found it my self.

The robustScaler in the preprocessing is working differently on different sets / subsets.

Therefore, the values in the subsets are prepared differently and therefore no longer fit the model.