Training RMSE higher than Validation RMSE in H2O
I am using the H2O-DeepLearning Model for a Regression Problem. What i observe is that Training RMSE is higher than Validation RMSE. I am using the model with default parameter which is two hidden layers with 200 Neurons each and no l1/l2 Regularization. Activation is Rectifier. No Dropout added.
I am wondering how can i tune the hyperparameters two get Training RMSE below Validation RMSE?
Any hints for parameter?
I am using separate Train, Validation and Test-Set. Training Data has 1958826 Samples, Validation and Test set have 599380 Samples each.
R-squared Value is around 0.65 - 0.7
Edit: While I am experiencing lower Validation RMSE than Training RMSE, it seems that the Values for Residual-Deviance in Training are lower than for Validation. So this seems fine.
Edit:
Training:
RMSE: 0.3592
Deviance: 0.0071
Validation:
RMSE: 0.3403
Deviance: 0.0082
I am doing quantile regression (if that is any help) and i have separate train/test data sets, while i splitted the test-set into validation and test with the h2o-split_frame-method.
validation deep-learning regression h2o
asked Nov 26 '18 at 8:46
dnks23dnks23
829
|
show 1 more comment
I am using the H2O-DeepLearning Model for a Regression Problem. What i observe is that Training RMSE is higher than Validation RMSE. I am using the model with default parameter which is two hidden layers with 200 Neurons each and no l1/l2 Regularization. Activation is Rectifier. No Dropout added.
I am wondering how can i tune the hyperparameters two get Training RMSE below Validation RMSE?
Any hints for parameter?
I am using separate Train, Validation and Test-Set. Training Data has 1958826 Samples, Validation and Test set have 599380 Samples each.
R-squared Value is around 0.65 - 0.7
Edit: While I am experiencing lower Validation RMSE than Training RMSE, it seems that the Values for Residual-Deviance in Training are lower than for Validation. So this seems fine.
Edit:
Training:
RMSE: 0.3592
Deviance: 0.0071
Validation:
RMSE: 0.3403
Deviance: 0.0082
I am doing quantile regression (if that is any help) and i have separate train/test data sets, while i splitted the test-set into validation and test with the h2o-split_frame-method.
validation deep-learning regression h2o
asked Nov 26 '18 at 8:46
dnks23dnks23
829
can you post what you get for rmse and residual deviance for your train and valid sets? how did you split up your data (was it a random split?). Do you think your validation dataset is representative of your train set?
– Lauren
Nov 27 '18 at 0:48
@Lauren i updated the post.
– dnks23
Nov 27 '18 at 8:29
in the future please don't double post: stats.stackexchange.com/questions/378660/…
– Lauren
Nov 27 '18 at 18:21
@Lauren okay thanks. but did the added information help you? Do you have an answer?
– dnks23
Nov 27 '18 at 18:49
sorry for the delayed response. It's hard to say why you are seeing these results, especially since the difference is small. I would take a look at the distribution of each of your features and if there is a noticeable difference between the feature distributions in the validation set compared to the train. Maybe there is something about the split that makes the valid set easier to predict. If you run multiple trials, with different seeds, do you ever see valid outperform train? here's some more info on tuning: github.com/h2oai/h2o-tutorials/tree/master/tutorials/…
– Lauren
Dec 3 '18 at 23:37
|
show 1 more comment
I am using the H2O-DeepLearning Model for a Regression Problem. What i observe is that Training RMSE is higher than Validation RMSE. I am using the model with default parameter which is two hidden layers with 200 Neurons each and no l1/l2 Regularization. Activation is Rectifier. No Dropout added.
I am wondering how can i tune the hyperparameters two get Training RMSE below Validation RMSE?
Any hints for parameter?
I am using separate Train, Validation and Test-Set. Training Data has 1958826 Samples, Validation and Test set have 599380 Samples each.
R-squared Value is around 0.65 - 0.7
Edit: While I am experiencing lower Validation RMSE than Training RMSE, it seems that the Values for Residual-Deviance in Training are lower than for Validation. So this seems fine.
Edit:
Training:
RMSE: 0.3592
Deviance: 0.0071
Validation:
RMSE: 0.3403
Deviance: 0.0082
I am doing quantile regression (if that is any help) and i have separate train/test data sets, while i splitted the test-set into validation and test with the h2o-split_frame-method.
validation deep-learning regression h2o
asked Nov 26 '18 at 8:46
dnks23dnks23
829
I am using the H2O-DeepLearning Model for a Regression Problem. What i observe is that Training RMSE is higher than Validation RMSE. I am using the model with default parameter which is two hidden layers with 200 Neurons each and no l1/l2 Regularization. Activation is Rectifier. No Dropout added.
I am wondering how can i tune the hyperparameters two get Training RMSE below Validation RMSE?
Any hints for parameter?
I am using separate Train, Validation and Test-Set. Training Data has 1958826 Samples, Validation and Test set have 599380 Samples each.
R-squared Value is around 0.65 - 0.7
Edit: While I am experiencing lower Validation RMSE than Training RMSE, it seems that the Values for Residual-Deviance in Training are lower than for Validation. So this seems fine.
Edit:
Training:
RMSE: 0.3592
Deviance: 0.0071
Validation:
RMSE: 0.3403
Deviance: 0.0082
I am doing quantile regression (if that is any help) and i have separate train/test data sets, while i splitted the test-set into validation and test with the h2o-split_frame-method.
validation deep-learning regression h2o
validation deep-learning regression h2o
asked Nov 26 '18 at 8:46
dnks23dnks23
829
asked Nov 26 '18 at 8:46
dnks23dnks23
829
edited Nov 27 '18 at 8:29
dnks23
asked Nov 26 '18 at 8:46
dnks23dnks23
829
asked Nov 26 '18 at 8:46
dnks23dnks23
829
asked Nov 26 '18 at 8:46
dnks23dnks23
829
829
can you post what you get for rmse and residual deviance for your train and valid sets? how did you split up your data (was it a random split?). Do you think your validation dataset is representative of your train set?
– Lauren
Nov 27 '18 at 0:48
@Lauren i updated the post.
– dnks23
Nov 27 '18 at 8:29
in the future please don't double post: stats.stackexchange.com/questions/378660/…
– Lauren
Nov 27 '18 at 18:21
@Lauren okay thanks. but did the added information help you? Do you have an answer?
– dnks23
Nov 27 '18 at 18:49
sorry for the delayed response. It's hard to say why you are seeing these results, especially since the difference is small. I would take a look at the distribution of each of your features and if there is a noticeable difference between the feature distributions in the validation set compared to the train. Maybe there is something about the split that makes the valid set easier to predict. If you run multiple trials, with different seeds, do you ever see valid outperform train? here's some more info on tuning: github.com/h2oai/h2o-tutorials/tree/master/tutorials/…
– Lauren
Dec 3 '18 at 23:37
|
show 1 more comment
can you post what you get for rmse and residual deviance for your train and valid sets? how did you split up your data (was it a random split?). Do you think your validation dataset is representative of your train set?
– Lauren
Nov 27 '18 at 0:48
@Lauren i updated the post.
– dnks23
Nov 27 '18 at 8:29
in the future please don't double post: stats.stackexchange.com/questions/378660/…
– Lauren
Nov 27 '18 at 18:21
@Lauren okay thanks. but did the added information help you? Do you have an answer?
– dnks23
Nov 27 '18 at 18:49
sorry for the delayed response. It's hard to say why you are seeing these results, especially since the difference is small. I would take a look at the distribution of each of your features and if there is a noticeable difference between the feature distributions in the validation set compared to the train. Maybe there is something about the split that makes the valid set easier to predict. If you run multiple trials, with different seeds, do you ever see valid outperform train? here's some more info on tuning: github.com/h2oai/h2o-tutorials/tree/master/tutorials/…
– Lauren
Dec 3 '18 at 23:37
can you post what you get for rmse and residual deviance for your train and valid sets? how did you split up your data (was it a random split?). Do you think your validation dataset is representative of your train set?
– Lauren
Nov 27 '18 at 0:48
can you post what you get for rmse and residual deviance for your train and valid sets? how did you split up your data (was it a random split?). Do you think your validation dataset is representative of your train set?
– Lauren
Nov 27 '18 at 0:48
@Lauren i updated the post.
– dnks23
Nov 27 '18 at 8:29
@Lauren i updated the post.
– dnks23
Nov 27 '18 at 8:29
in the future please don't double post: stats.stackexchange.com/questions/378660/…
– Lauren
Nov 27 '18 at 18:21
in the future please don't double post: stats.stackexchange.com/questions/378660/…
– Lauren
Nov 27 '18 at 18:21
@Lauren okay thanks. but did the added information help you? Do you have an answer?
– dnks23
Nov 27 '18 at 18:49
@Lauren okay thanks. but did the added information help you? Do you have an answer?
– dnks23
Nov 27 '18 at 18:49
sorry for the delayed response. It's hard to say why you are seeing these results, especially since the difference is small. I would take a look at the distribution of each of your features and if there is a noticeable difference between the feature distributions in the validation set compared to the train. Maybe there is something about the split that makes the valid set easier to predict. If you run multiple trials, with different seeds, do you ever see valid outperform train? here's some more info on tuning: github.com/h2oai/h2o-tutorials/tree/master/tutorials/…
– Lauren
Dec 3 '18 at 23:37
sorry for the delayed response. It's hard to say why you are seeing these results, especially since the difference is small. I would take a look at the distribution of each of your features and if there is a noticeable difference between the feature distributions in the validation set compared to the train. Maybe there is something about the split that makes the valid set easier to predict. If you run multiple trials, with different seeds, do you ever see valid outperform train? here's some more info on tuning: github.com/h2oai/h2o-tutorials/tree/master/tutorials/…
– Lauren
Dec 3 '18 at 23:37
|
show 1 more comment
1 Answer
1
active
oldest
votes
Here are a few recommendations of what you can try.
- Train for more epochs
- Decrease batch size
- Increase the number of neurons in the hidden layers.
It is possible that a low number of epochs is the cause for worse train performance in your regression problem.
answered Dec 6 '18 at 21:14
LaurenLauren
3,5311515
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53477412%2ftraining-rmse-higher-than-validation-rmse-in-h2o%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Here are a few recommendations of what you can try.
- Train for more epochs
- Decrease batch size
- Increase the number of neurons in the hidden layers.
It is possible that a low number of epochs is the cause for worse train performance in your regression problem.
answered Dec 6 '18 at 21:14
LaurenLauren
3,5311515
add a comment |
Here are a few recommendations of what you can try.
- Train for more epochs
- Decrease batch size
- Increase the number of neurons in the hidden layers.
It is possible that a low number of epochs is the cause for worse train performance in your regression problem.
answered Dec 6 '18 at 21:14
LaurenLauren
3,5311515
add a comment |
Here are a few recommendations of what you can try.
- Train for more epochs
- Decrease batch size
- Increase the number of neurons in the hidden layers.
It is possible that a low number of epochs is the cause for worse train performance in your regression problem.
answered Dec 6 '18 at 21:14
LaurenLauren
3,5311515
Here are a few recommendations of what you can try.
- Train for more epochs
- Decrease batch size
- Increase the number of neurons in the hidden layers.
It is possible that a low number of epochs is the cause for worse train performance in your regression problem.
answered Dec 6 '18 at 21:14
LaurenLauren
3,5311515
answered Dec 6 '18 at 21:14
LaurenLauren
3,5311515
answered Dec 6 '18 at 21:14
LaurenLauren
3,5311515
answered Dec 6 '18 at 21:14
LaurenLauren
3,5311515
3,5311515
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53477412%2ftraining-rmse-higher-than-validation-rmse-in-h2o%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
can you post what you get for rmse and residual deviance for your train and valid sets? how did you split up your data (was it a random split?). Do you think your validation dataset is representative of your train set?
– Lauren
Nov 27 '18 at 0:48
@Lauren i updated the post.
– dnks23
Nov 27 '18 at 8:29
in the future please don't double post: stats.stackexchange.com/questions/378660/…
– Lauren
Nov 27 '18 at 18:21
@Lauren okay thanks. but did the added information help you? Do you have an answer?
– dnks23
Nov 27 '18 at 18:49
sorry for the delayed response. It's hard to say why you are seeing these results, especially since the difference is small. I would take a look at the distribution of each of your features and if there is a noticeable difference between the feature distributions in the validation set compared to the train. Maybe there is something about the split that makes the valid set easier to predict. If you run multiple trials, with different seeds, do you ever see valid outperform train? here's some more info on tuning: github.com/h2oai/h2o-tutorials/tree/master/tutorials/…
– Lauren
Dec 3 '18 at 23:37