understanding the fundamentals of quantile() and quantiles
up vote
1
down vote
favorite
If there are, say 59 observations, I'm confused why quantile(data, probs = 0.05) would put 3 values under the 5th percentile, as 3/59 = ~0.051.
library(tidyverse)
a <- seq(c(1:59))
b <-rnorm(59)
df <- data.frame(a,b)
df_5thperc <- df %>% summarize(`05%` = quantile(b,
probs=0.05))
y <- mean(df_5thperc$`05%`)
ggplot() + geom_point(data = df, aes(x = a, y = b)) +
geom_hline(yintercept = y, color = "blue")
r statistics
asked Nov 18 at 2:36
doconnor
19211
add a comment |
up vote
1
down vote
favorite
If there are, say 59 observations, I'm confused why quantile(data, probs = 0.05) would put 3 values under the 5th percentile, as 3/59 = ~0.051.
library(tidyverse)
a <- seq(c(1:59))
b <-rnorm(59)
df <- data.frame(a,b)
df_5thperc <- df %>% summarize(`05%` = quantile(b,
probs=0.05))
y <- mean(df_5thperc$`05%`)
ggplot() + geom_point(data = df, aes(x = a, y = b)) +
geom_hline(yintercept = y, color = "blue")
r statistics
asked Nov 18 at 2:36
doconnor
19211
Quantiles AFAIK just means dividing up your observations into 5 evenly distributed buckets. Note that if you data set is not evenly divisible by 5, then the buckets won't be even.
– Tim Biegeleisen
Nov 18 at 2:40
2
read?quantilefor excruciating details about how the quantiles are defined/computed ...
– Ben Bolker
Nov 18 at 2:58
2
@TimBiegeleisen - wouldn't that be a quintile rather than a quantile?
– thelatemail
Nov 18 at 2:59
2
Not entirely clear what you're asking. If R put the quantile between the 2d and 3d observation, then P(x<Q)=0.0339, obviously too small. If Q is between the 4th and 5th observation then P(x<Q)=0.0678, obviously too big. Putting Q between the 3d and 4th observation and getting P(x<Q)=0.0508 is the best that can be achieved (the 0.05 quantile will never have P(x<Q)==0.05 unless N is divisible by 20).
– Ben Bolker
Nov 18 at 3:02
@thelatemail You have released the cat which bit my tongue!
– Tim Biegeleisen
Nov 18 at 3:03
add a comment |
up vote
1
down vote
favorite
up vote
1
down vote
favorite
If there are, say 59 observations, I'm confused why quantile(data, probs = 0.05) would put 3 values under the 5th percentile, as 3/59 = ~0.051.
library(tidyverse)
a <- seq(c(1:59))
b <-rnorm(59)
df <- data.frame(a,b)
df_5thperc <- df %>% summarize(`05%` = quantile(b,
probs=0.05))
y <- mean(df_5thperc$`05%`)
ggplot() + geom_point(data = df, aes(x = a, y = b)) +
geom_hline(yintercept = y, color = "blue")
r statistics
asked Nov 18 at 2:36
doconnor
19211
If there are, say 59 observations, I'm confused why quantile(data, probs = 0.05) would put 3 values under the 5th percentile, as 3/59 = ~0.051.
library(tidyverse)
a <- seq(c(1:59))
b <-rnorm(59)
df <- data.frame(a,b)
df_5thperc <- df %>% summarize(`05%` = quantile(b,
probs=0.05))
y <- mean(df_5thperc$`05%`)
ggplot() + geom_point(data = df, aes(x = a, y = b)) +
geom_hline(yintercept = y, color = "blue")
r statistics
r statistics
asked Nov 18 at 2:36
doconnor
19211
asked Nov 18 at 2:36
doconnor
19211
edited Nov 18 at 2:45
asked Nov 18 at 2:36
doconnor
19211
asked Nov 18 at 2:36
doconnor
19211
asked Nov 18 at 2:36
doconnor
19211
19211
Quantiles AFAIK just means dividing up your observations into 5 evenly distributed buckets. Note that if you data set is not evenly divisible by 5, then the buckets won't be even.
– Tim Biegeleisen
Nov 18 at 2:40
2
read?quantilefor excruciating details about how the quantiles are defined/computed ...
– Ben Bolker
Nov 18 at 2:58
2
@TimBiegeleisen - wouldn't that be a quintile rather than a quantile?
– thelatemail
Nov 18 at 2:59
2
Not entirely clear what you're asking. If R put the quantile between the 2d and 3d observation, then P(x<Q)=0.0339, obviously too small. If Q is between the 4th and 5th observation then P(x<Q)=0.0678, obviously too big. Putting Q between the 3d and 4th observation and getting P(x<Q)=0.0508 is the best that can be achieved (the 0.05 quantile will never have P(x<Q)==0.05 unless N is divisible by 20).
– Ben Bolker
Nov 18 at 3:02
@thelatemail You have released the cat which bit my tongue!
– Tim Biegeleisen
Nov 18 at 3:03
add a comment |
Quantiles AFAIK just means dividing up your observations into 5 evenly distributed buckets. Note that if you data set is not evenly divisible by 5, then the buckets won't be even.
– Tim Biegeleisen
Nov 18 at 2:40
2
read?quantilefor excruciating details about how the quantiles are defined/computed ...
– Ben Bolker
Nov 18 at 2:58
2
@TimBiegeleisen - wouldn't that be a quintile rather than a quantile?
– thelatemail
Nov 18 at 2:59
2
Not entirely clear what you're asking. If R put the quantile between the 2d and 3d observation, then P(x<Q)=0.0339, obviously too small. If Q is between the 4th and 5th observation then P(x<Q)=0.0678, obviously too big. Putting Q between the 3d and 4th observation and getting P(x<Q)=0.0508 is the best that can be achieved (the 0.05 quantile will never have P(x<Q)==0.05 unless N is divisible by 20).
– Ben Bolker
Nov 18 at 3:02
@thelatemail You have released the cat which bit my tongue!
– Tim Biegeleisen
Nov 18 at 3:03
Quantiles AFAIK just means dividing up your observations into 5 evenly distributed buckets. Note that if you data set is not evenly divisible by 5, then the buckets won't be even.
– Tim Biegeleisen
Nov 18 at 2:40
Quantiles AFAIK just means dividing up your observations into 5 evenly distributed buckets. Note that if you data set is not evenly divisible by 5, then the buckets won't be even.
– Tim Biegeleisen
Nov 18 at 2:40
2
2
read
?quantile for excruciating details about how the quantiles are defined/computed ...– Ben Bolker
Nov 18 at 2:58
read
?quantile for excruciating details about how the quantiles are defined/computed ...– Ben Bolker
Nov 18 at 2:58
2
2
@TimBiegeleisen - wouldn't that be a quintile rather than a quantile?
– thelatemail
Nov 18 at 2:59
@TimBiegeleisen - wouldn't that be a quintile rather than a quantile?
– thelatemail
Nov 18 at 2:59
2
2
Not entirely clear what you're asking. If R put the quantile between the 2d and 3d observation, then P(x<Q)=0.0339, obviously too small. If Q is between the 4th and 5th observation then P(x<Q)=0.0678, obviously too big. Putting Q between the 3d and 4th observation and getting P(x<Q)=0.0508 is the best that can be achieved (the 0.05 quantile will never have P(x<Q)==0.05 unless N is divisible by 20).
– Ben Bolker
Nov 18 at 3:02
Not entirely clear what you're asking. If R put the quantile between the 2d and 3d observation, then P(x<Q)=0.0339, obviously too small. If Q is between the 4th and 5th observation then P(x<Q)=0.0678, obviously too big. Putting Q between the 3d and 4th observation and getting P(x<Q)=0.0508 is the best that can be achieved (the 0.05 quantile will never have P(x<Q)==0.05 unless N is divisible by 20).
– Ben Bolker
Nov 18 at 3:02
@thelatemail You have released the cat which bit my tongue!
– Tim Biegeleisen
Nov 18 at 3:03
@thelatemail You have released the cat which bit my tongue!
– Tim Biegeleisen
Nov 18 at 3:03
add a comment |
1 Answer
1
active
oldest
votes
up vote
3
down vote
accepted
To expand on @BenBolker, you could consider the type parameter for the quantile() function. You are using a continuous distribution so types 4 through 9 are relevant. For example:
b[b < quantile(b, probs = c(.05), type = 9)]
Types 4 and 6 will give what you were probably expecting
[1] -1.893092 -3.263889
while 5, 7, 8, and 9 will give
[1] -1.893092 -1.538927 -3.263889
The help file gives much detail about why, but in the end it comes down to the fact that there is no agreed upon method to estimate sample quantiles (including the median).
answered Nov 18 at 3:27
Elin
5,06621536
thank you @Elin. So say there are 59 student test scores, or 59 annual snow fall totals, depending on the method, there's some gray area as to whether two or three measurements are within the, say, highest 5th percentile? I wonder, any sense if certain disciplines, like say sociology or hydrology, tend to stick to one vs another? I saw some types were default in certain software.
– doconnor
Nov 18 at 3:57
2
Yes, but I wouldn't say it is gray, either there are 2 or 3. The method is about how the value is calculated. The calculated values are different. As a result different specific observations fall below it. I would say that your observation about the software is right -- in a given discipline I'd probably use the method that corresponds to one of the popular software packages or otherwise explain why I chose a particular method. You should be conscious of why you are using a particular method and if you read the references at the bottom of the help file they may help you think about that.
– Elin
Nov 18 at 4:13
1
thank you. absolute coincidence that hydrology is mentioned there! In there I seeThe default method is type 7, as used by S and by R < 2.0.0.but under Author, there's a reference tothe version used in R >= 2.0.0, without a # given (I think the latest R version is ~3.4). Do you read it as the default still being type 7, or am I missing something? I'm wondering if it's the same type 7, just computed maybe more efficiently.
– doconnor
Nov 18 at 4:38
1
I think, but don't know, that the types were introduced in version 2.0.0. R is an open source version of S so since S uses type 7, R by default uses type 7 and probably matches the previous default which would have matched S. Also the R core team cares a lot about backward compatibility so it would have wanted existing code to yields the same result as it had before types were introduced.
– Elin
Nov 18 at 9:50
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
3
down vote
accepted
To expand on @BenBolker, you could consider the type parameter for the quantile() function. You are using a continuous distribution so types 4 through 9 are relevant. For example:
b[b < quantile(b, probs = c(.05), type = 9)]
Types 4 and 6 will give what you were probably expecting
[1] -1.893092 -3.263889
while 5, 7, 8, and 9 will give
[1] -1.893092 -1.538927 -3.263889
The help file gives much detail about why, but in the end it comes down to the fact that there is no agreed upon method to estimate sample quantiles (including the median).
answered Nov 18 at 3:27
Elin
5,06621536
thank you @Elin. So say there are 59 student test scores, or 59 annual snow fall totals, depending on the method, there's some gray area as to whether two or three measurements are within the, say, highest 5th percentile? I wonder, any sense if certain disciplines, like say sociology or hydrology, tend to stick to one vs another? I saw some types were default in certain software.
– doconnor
Nov 18 at 3:57
2
Yes, but I wouldn't say it is gray, either there are 2 or 3. The method is about how the value is calculated. The calculated values are different. As a result different specific observations fall below it. I would say that your observation about the software is right -- in a given discipline I'd probably use the method that corresponds to one of the popular software packages or otherwise explain why I chose a particular method. You should be conscious of why you are using a particular method and if you read the references at the bottom of the help file they may help you think about that.
– Elin
Nov 18 at 4:13
1
thank you. absolute coincidence that hydrology is mentioned there! In there I seeThe default method is type 7, as used by S and by R < 2.0.0.but under Author, there's a reference tothe version used in R >= 2.0.0, without a # given (I think the latest R version is ~3.4). Do you read it as the default still being type 7, or am I missing something? I'm wondering if it's the same type 7, just computed maybe more efficiently.
– doconnor
Nov 18 at 4:38
1
I think, but don't know, that the types were introduced in version 2.0.0. R is an open source version of S so since S uses type 7, R by default uses type 7 and probably matches the previous default which would have matched S. Also the R core team cares a lot about backward compatibility so it would have wanted existing code to yields the same result as it had before types were introduced.
– Elin
Nov 18 at 9:50
add a comment |
up vote
3
down vote
accepted
To expand on @BenBolker, you could consider the type parameter for the quantile() function. You are using a continuous distribution so types 4 through 9 are relevant. For example:
b[b < quantile(b, probs = c(.05), type = 9)]
Types 4 and 6 will give what you were probably expecting
[1] -1.893092 -3.263889
while 5, 7, 8, and 9 will give
[1] -1.893092 -1.538927 -3.263889
The help file gives much detail about why, but in the end it comes down to the fact that there is no agreed upon method to estimate sample quantiles (including the median).
answered Nov 18 at 3:27
Elin
5,06621536
thank you @Elin. So say there are 59 student test scores, or 59 annual snow fall totals, depending on the method, there's some gray area as to whether two or three measurements are within the, say, highest 5th percentile? I wonder, any sense if certain disciplines, like say sociology or hydrology, tend to stick to one vs another? I saw some types were default in certain software.
– doconnor
Nov 18 at 3:57
2
Yes, but I wouldn't say it is gray, either there are 2 or 3. The method is about how the value is calculated. The calculated values are different. As a result different specific observations fall below it. I would say that your observation about the software is right -- in a given discipline I'd probably use the method that corresponds to one of the popular software packages or otherwise explain why I chose a particular method. You should be conscious of why you are using a particular method and if you read the references at the bottom of the help file they may help you think about that.
– Elin
Nov 18 at 4:13
1
thank you. absolute coincidence that hydrology is mentioned there! In there I seeThe default method is type 7, as used by S and by R < 2.0.0.but under Author, there's a reference tothe version used in R >= 2.0.0, without a # given (I think the latest R version is ~3.4). Do you read it as the default still being type 7, or am I missing something? I'm wondering if it's the same type 7, just computed maybe more efficiently.
– doconnor
Nov 18 at 4:38
1
I think, but don't know, that the types were introduced in version 2.0.0. R is an open source version of S so since S uses type 7, R by default uses type 7 and probably matches the previous default which would have matched S. Also the R core team cares a lot about backward compatibility so it would have wanted existing code to yields the same result as it had before types were introduced.
– Elin
Nov 18 at 9:50
add a comment |
up vote
3
down vote
accepted
up vote
3
down vote
accepted
To expand on @BenBolker, you could consider the type parameter for the quantile() function. You are using a continuous distribution so types 4 through 9 are relevant. For example:
b[b < quantile(b, probs = c(.05), type = 9)]
Types 4 and 6 will give what you were probably expecting
[1] -1.893092 -3.263889
while 5, 7, 8, and 9 will give
[1] -1.893092 -1.538927 -3.263889
The help file gives much detail about why, but in the end it comes down to the fact that there is no agreed upon method to estimate sample quantiles (including the median).
answered Nov 18 at 3:27
Elin
5,06621536
To expand on @BenBolker, you could consider the type parameter for the quantile() function. You are using a continuous distribution so types 4 through 9 are relevant. For example:
b[b < quantile(b, probs = c(.05), type = 9)]
Types 4 and 6 will give what you were probably expecting
[1] -1.893092 -3.263889
while 5, 7, 8, and 9 will give
[1] -1.893092 -1.538927 -3.263889
The help file gives much detail about why, but in the end it comes down to the fact that there is no agreed upon method to estimate sample quantiles (including the median).
answered Nov 18 at 3:27
Elin
5,06621536
answered Nov 18 at 3:27
Elin
5,06621536
answered Nov 18 at 3:27
Elin
5,06621536
answered Nov 18 at 3:27
Elin
5,06621536
5,06621536
thank you @Elin. So say there are 59 student test scores, or 59 annual snow fall totals, depending on the method, there's some gray area as to whether two or three measurements are within the, say, highest 5th percentile? I wonder, any sense if certain disciplines, like say sociology or hydrology, tend to stick to one vs another? I saw some types were default in certain software.
– doconnor
Nov 18 at 3:57
2
Yes, but I wouldn't say it is gray, either there are 2 or 3. The method is about how the value is calculated. The calculated values are different. As a result different specific observations fall below it. I would say that your observation about the software is right -- in a given discipline I'd probably use the method that corresponds to one of the popular software packages or otherwise explain why I chose a particular method. You should be conscious of why you are using a particular method and if you read the references at the bottom of the help file they may help you think about that.
– Elin
Nov 18 at 4:13
1
thank you. absolute coincidence that hydrology is mentioned there! In there I seeThe default method is type 7, as used by S and by R < 2.0.0.but under Author, there's a reference tothe version used in R >= 2.0.0, without a # given (I think the latest R version is ~3.4). Do you read it as the default still being type 7, or am I missing something? I'm wondering if it's the same type 7, just computed maybe more efficiently.
– doconnor
Nov 18 at 4:38
1
I think, but don't know, that the types were introduced in version 2.0.0. R is an open source version of S so since S uses type 7, R by default uses type 7 and probably matches the previous default which would have matched S. Also the R core team cares a lot about backward compatibility so it would have wanted existing code to yields the same result as it had before types were introduced.
– Elin
Nov 18 at 9:50
add a comment |
thank you @Elin. So say there are 59 student test scores, or 59 annual snow fall totals, depending on the method, there's some gray area as to whether two or three measurements are within the, say, highest 5th percentile? I wonder, any sense if certain disciplines, like say sociology or hydrology, tend to stick to one vs another? I saw some types were default in certain software.
– doconnor
Nov 18 at 3:57
2
Yes, but I wouldn't say it is gray, either there are 2 or 3. The method is about how the value is calculated. The calculated values are different. As a result different specific observations fall below it. I would say that your observation about the software is right -- in a given discipline I'd probably use the method that corresponds to one of the popular software packages or otherwise explain why I chose a particular method. You should be conscious of why you are using a particular method and if you read the references at the bottom of the help file they may help you think about that.
– Elin
Nov 18 at 4:13
1
thank you. absolute coincidence that hydrology is mentioned there! In there I seeThe default method is type 7, as used by S and by R < 2.0.0.but under Author, there's a reference tothe version used in R >= 2.0.0, without a # given (I think the latest R version is ~3.4). Do you read it as the default still being type 7, or am I missing something? I'm wondering if it's the same type 7, just computed maybe more efficiently.
– doconnor
Nov 18 at 4:38
1
I think, but don't know, that the types were introduced in version 2.0.0. R is an open source version of S so since S uses type 7, R by default uses type 7 and probably matches the previous default which would have matched S. Also the R core team cares a lot about backward compatibility so it would have wanted existing code to yields the same result as it had before types were introduced.
– Elin
Nov 18 at 9:50
thank you @Elin. So say there are 59 student test scores, or 59 annual snow fall totals, depending on the method, there's some gray area as to whether two or three measurements are within the, say, highest 5th percentile? I wonder, any sense if certain disciplines, like say sociology or hydrology, tend to stick to one vs another? I saw some types were default in certain software.
– doconnor
Nov 18 at 3:57
thank you @Elin. So say there are 59 student test scores, or 59 annual snow fall totals, depending on the method, there's some gray area as to whether two or three measurements are within the, say, highest 5th percentile? I wonder, any sense if certain disciplines, like say sociology or hydrology, tend to stick to one vs another? I saw some types were default in certain software.
– doconnor
Nov 18 at 3:57
2
2
Yes, but I wouldn't say it is gray, either there are 2 or 3. The method is about how the value is calculated. The calculated values are different. As a result different specific observations fall below it. I would say that your observation about the software is right -- in a given discipline I'd probably use the method that corresponds to one of the popular software packages or otherwise explain why I chose a particular method. You should be conscious of why you are using a particular method and if you read the references at the bottom of the help file they may help you think about that.
– Elin
Nov 18 at 4:13
Yes, but I wouldn't say it is gray, either there are 2 or 3. The method is about how the value is calculated. The calculated values are different. As a result different specific observations fall below it. I would say that your observation about the software is right -- in a given discipline I'd probably use the method that corresponds to one of the popular software packages or otherwise explain why I chose a particular method. You should be conscious of why you are using a particular method and if you read the references at the bottom of the help file they may help you think about that.
– Elin
Nov 18 at 4:13
1
1
thank you. absolute coincidence that hydrology is mentioned there! In there I see
The default method is type 7, as used by S and by R < 2.0.0. but under Author, there's a reference to the version used in R >= 2.0.0, without a # given (I think the latest R version is ~3.4). Do you read it as the default still being type 7, or am I missing something? I'm wondering if it's the same type 7, just computed maybe more efficiently.– doconnor
Nov 18 at 4:38
thank you. absolute coincidence that hydrology is mentioned there! In there I see
The default method is type 7, as used by S and by R < 2.0.0. but under Author, there's a reference to the version used in R >= 2.0.0, without a # given (I think the latest R version is ~3.4). Do you read it as the default still being type 7, or am I missing something? I'm wondering if it's the same type 7, just computed maybe more efficiently.– doconnor
Nov 18 at 4:38
1
1
I think, but don't know, that the types were introduced in version 2.0.0. R is an open source version of S so since S uses type 7, R by default uses type 7 and probably matches the previous default which would have matched S. Also the R core team cares a lot about backward compatibility so it would have wanted existing code to yields the same result as it had before types were introduced.
– Elin
Nov 18 at 9:50
I think, but don't know, that the types were introduced in version 2.0.0. R is an open source version of S so since S uses type 7, R by default uses type 7 and probably matches the previous default which would have matched S. Also the R core team cares a lot about backward compatibility so it would have wanted existing code to yields the same result as it had before types were introduced.
– Elin
Nov 18 at 9:50
add a comment |
draft saved
draft discarded
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53357399%2funderstanding-the-fundamentals-of-quantile-and-quantiles%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Quantiles AFAIK just means dividing up your observations into 5 evenly distributed buckets. Note that if you data set is not evenly divisible by 5, then the buckets won't be even.
– Tim Biegeleisen
Nov 18 at 2:40
2
read
?quantilefor excruciating details about how the quantiles are defined/computed ...– Ben Bolker
Nov 18 at 2:58
2
@TimBiegeleisen - wouldn't that be a quintile rather than a quantile?
– thelatemail
Nov 18 at 2:59
2
Not entirely clear what you're asking. If R put the quantile between the 2d and 3d observation, then P(x<Q)=0.0339, obviously too small. If Q is between the 4th and 5th observation then P(x<Q)=0.0678, obviously too big. Putting Q between the 3d and 4th observation and getting P(x<Q)=0.0508 is the best that can be achieved (the 0.05 quantile will never have P(x<Q)==0.05 unless N is divisible by 20).
– Ben Bolker
Nov 18 at 3:02
@thelatemail You have released the cat which bit my tongue!
– Tim Biegeleisen
Nov 18 at 3:03