Bayesian Logistic Regression, conditional probability integration
$begingroup$
In Andrew NG's Lectures (CS229), the Bayesian Logistic Regression section contained a formula;
$$P(Y|X,S)=int_theta P(Y|X,theta)P(theta|S)dtheta$$
Here, $theta$ is treated as a random variable.
$S$ is the set of points ${[X^{(i)},Y^{(i)}]_{i=1}^{m}}$.
Using conditional probabilities, it does make intuitive sense although I would really appreciate a rigorous proof of the equation.
From what I got:
$$ P(Y|X,S)=int_theta P(Y,theta|X,S)dtheta $$
$$ = int_theta P(Y|theta,X,S)P(theta|X,S)dtheta$$
Does it assume any sort of independence?
Again, I get the intuition, but I can't seem to arrive at the final answer. A written out proof or required equations to prove the result would be appreciated.
probability machine-learning bayesian logistic-regression
asked Jul 7 '18 at 7:19
Utsav DuttaUtsav Dutta
61
$endgroup$
add a comment |
$begingroup$
In Andrew NG's Lectures (CS229), the Bayesian Logistic Regression section contained a formula;
$$P(Y|X,S)=int_theta P(Y|X,theta)P(theta|S)dtheta$$
Here, $theta$ is treated as a random variable.
$S$ is the set of points ${[X^{(i)},Y^{(i)}]_{i=1}^{m}}$.
Using conditional probabilities, it does make intuitive sense although I would really appreciate a rigorous proof of the equation.
From what I got:
$$ P(Y|X,S)=int_theta P(Y,theta|X,S)dtheta $$
$$ = int_theta P(Y|theta,X,S)P(theta|X,S)dtheta$$
Does it assume any sort of independence?
Again, I get the intuition, but I can't seem to arrive at the final answer. A written out proof or required equations to prove the result would be appreciated.
probability machine-learning bayesian logistic-regression
asked Jul 7 '18 at 7:19
Utsav DuttaUtsav Dutta
61
$endgroup$
$begingroup$
What are $X$ and $Y$? Discrete or continuous random variables (or vectors)? Please include as much context as possible directly in the question instead of pointing to some course.
$endgroup$
– paf
Jul 7 '18 at 7:21
1
$begingroup$
@paf In the context of this problem, X(i) and Y(i) are points in the sample space, S. X(i) is an n dimensional vector and Y(i) is the corresponding classification of that point. X and Y themselves are continuous random variables.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:26
$begingroup$
Treat X as a new point in the hyperspace and P(Y=y|X) as the probability of classifying the new point X as Y=y.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:27
$begingroup$
related: math.stackexchange.com/questions/1882178/…
$endgroup$
– Henry
Jul 7 '18 at 8:17
$begingroup$
What does $int_theta ldots dtheta$ means (where $theta$ is a r.v.)?
$endgroup$
– d.k.o.
Jul 7 '18 at 19:24
add a comment |
$begingroup$
In Andrew NG's Lectures (CS229), the Bayesian Logistic Regression section contained a formula;
$$P(Y|X,S)=int_theta P(Y|X,theta)P(theta|S)dtheta$$
Here, $theta$ is treated as a random variable.
$S$ is the set of points ${[X^{(i)},Y^{(i)}]_{i=1}^{m}}$.
Using conditional probabilities, it does make intuitive sense although I would really appreciate a rigorous proof of the equation.
From what I got:
$$ P(Y|X,S)=int_theta P(Y,theta|X,S)dtheta $$
$$ = int_theta P(Y|theta,X,S)P(theta|X,S)dtheta$$
Does it assume any sort of independence?
Again, I get the intuition, but I can't seem to arrive at the final answer. A written out proof or required equations to prove the result would be appreciated.
probability machine-learning bayesian logistic-regression
asked Jul 7 '18 at 7:19
Utsav DuttaUtsav Dutta
61
$endgroup$
In Andrew NG's Lectures (CS229), the Bayesian Logistic Regression section contained a formula;
$$P(Y|X,S)=int_theta P(Y|X,theta)P(theta|S)dtheta$$
Here, $theta$ is treated as a random variable.
$S$ is the set of points ${[X^{(i)},Y^{(i)}]_{i=1}^{m}}$.
Using conditional probabilities, it does make intuitive sense although I would really appreciate a rigorous proof of the equation.
From what I got:
$$ P(Y|X,S)=int_theta P(Y,theta|X,S)dtheta $$
$$ = int_theta P(Y|theta,X,S)P(theta|X,S)dtheta$$
Does it assume any sort of independence?
Again, I get the intuition, but I can't seem to arrive at the final answer. A written out proof or required equations to prove the result would be appreciated.
probability machine-learning bayesian logistic-regression
probability machine-learning bayesian logistic-regression
asked Jul 7 '18 at 7:19
Utsav DuttaUtsav Dutta
61
asked Jul 7 '18 at 7:19
Utsav DuttaUtsav Dutta
61
asked Jul 7 '18 at 7:19
Utsav DuttaUtsav Dutta
61
asked Jul 7 '18 at 7:19
Utsav DuttaUtsav Dutta
61
asked Jul 7 '18 at 7:19
Utsav DuttaUtsav Dutta
61
61
$begingroup$
What are $X$ and $Y$? Discrete or continuous random variables (or vectors)? Please include as much context as possible directly in the question instead of pointing to some course.
$endgroup$
– paf
Jul 7 '18 at 7:21
1
$begingroup$
@paf In the context of this problem, X(i) and Y(i) are points in the sample space, S. X(i) is an n dimensional vector and Y(i) is the corresponding classification of that point. X and Y themselves are continuous random variables.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:26
$begingroup$
Treat X as a new point in the hyperspace and P(Y=y|X) as the probability of classifying the new point X as Y=y.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:27
$begingroup$
related: math.stackexchange.com/questions/1882178/…
$endgroup$
– Henry
Jul 7 '18 at 8:17
$begingroup$
What does $int_theta ldots dtheta$ means (where $theta$ is a r.v.)?
$endgroup$
– d.k.o.
Jul 7 '18 at 19:24
add a comment |
$begingroup$
What are $X$ and $Y$? Discrete or continuous random variables (or vectors)? Please include as much context as possible directly in the question instead of pointing to some course.
$endgroup$
– paf
Jul 7 '18 at 7:21
1
$begingroup$
@paf In the context of this problem, X(i) and Y(i) are points in the sample space, S. X(i) is an n dimensional vector and Y(i) is the corresponding classification of that point. X and Y themselves are continuous random variables.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:26
$begingroup$
Treat X as a new point in the hyperspace and P(Y=y|X) as the probability of classifying the new point X as Y=y.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:27
$begingroup$
related: math.stackexchange.com/questions/1882178/…
$endgroup$
– Henry
Jul 7 '18 at 8:17
$begingroup$
What does $int_theta ldots dtheta$ means (where $theta$ is a r.v.)?
$endgroup$
– d.k.o.
Jul 7 '18 at 19:24
$begingroup$
What are $X$ and $Y$? Discrete or continuous random variables (or vectors)? Please include as much context as possible directly in the question instead of pointing to some course.
$endgroup$
– paf
Jul 7 '18 at 7:21
$begingroup$
What are $X$ and $Y$? Discrete or continuous random variables (or vectors)? Please include as much context as possible directly in the question instead of pointing to some course.
$endgroup$
– paf
Jul 7 '18 at 7:21
1
1
$begingroup$
@paf In the context of this problem, X(i) and Y(i) are points in the sample space, S. X(i) is an n dimensional vector and Y(i) is the corresponding classification of that point. X and Y themselves are continuous random variables.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:26
$begingroup$
@paf In the context of this problem, X(i) and Y(i) are points in the sample space, S. X(i) is an n dimensional vector and Y(i) is the corresponding classification of that point. X and Y themselves are continuous random variables.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:26
$begingroup$
Treat X as a new point in the hyperspace and P(Y=y|X) as the probability of classifying the new point X as Y=y.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:27
$begingroup$
Treat X as a new point in the hyperspace and P(Y=y|X) as the probability of classifying the new point X as Y=y.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:27
$begingroup$
related: math.stackexchange.com/questions/1882178/…
$endgroup$
– Henry
Jul 7 '18 at 8:17
$begingroup$
related: math.stackexchange.com/questions/1882178/…
$endgroup$
– Henry
Jul 7 '18 at 8:17
$begingroup$
What does $int_theta ldots dtheta$ means (where $theta$ is a r.v.)?
$endgroup$
– d.k.o.
Jul 7 '18 at 19:24
$begingroup$
What does $int_theta ldots dtheta$ means (where $theta$ is a r.v.)?
$endgroup$
– d.k.o.
Jul 7 '18 at 19:24
add a comment |
1 Answer
1
active
oldest
votes
$begingroup$
The assumption applied is that of conditional independence. If $Z_1$ and $Z_2$ are independent then $p(Z_1lvert Z_2) = P(Z_1)$ hence $Z_2$ can be removed from the set of variables on which one is conditioning. Similarly with conditional independence if $Z_1$ and $Z_3$ are conditionally independent given $Z_2$ then
$p(Z_1lvert Z_2,Z_3) = p(Z_1lvert Z_2)$
so ones $Z_2$ is "controlled for" $Z_1$ does not depend on $Z_3$.
So the author is assuming that
$$p(Ylvert theta , X, S) = p(Ylvert theta ,X)$$
the dependent variable $Y$ only depends on $S$ through $X$ and $theta$
$$p(theta lvert X,S) = p(theta lvert S)$$
so parameters $theta$ do not depend on $X$ once $S$ is given.
answered Jan 1 at 12:50
Jesper HybelJesper Hybel
1127
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2843523%2fbayesian-logistic-regression-conditional-probability-integration%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
The assumption applied is that of conditional independence. If $Z_1$ and $Z_2$ are independent then $p(Z_1lvert Z_2) = P(Z_1)$ hence $Z_2$ can be removed from the set of variables on which one is conditioning. Similarly with conditional independence if $Z_1$ and $Z_3$ are conditionally independent given $Z_2$ then
$p(Z_1lvert Z_2,Z_3) = p(Z_1lvert Z_2)$
so ones $Z_2$ is "controlled for" $Z_1$ does not depend on $Z_3$.
So the author is assuming that
$$p(Ylvert theta , X, S) = p(Ylvert theta ,X)$$
the dependent variable $Y$ only depends on $S$ through $X$ and $theta$
$$p(theta lvert X,S) = p(theta lvert S)$$
so parameters $theta$ do not depend on $X$ once $S$ is given.
answered Jan 1 at 12:50
Jesper HybelJesper Hybel
1127
$endgroup$
add a comment |
$begingroup$
The assumption applied is that of conditional independence. If $Z_1$ and $Z_2$ are independent then $p(Z_1lvert Z_2) = P(Z_1)$ hence $Z_2$ can be removed from the set of variables on which one is conditioning. Similarly with conditional independence if $Z_1$ and $Z_3$ are conditionally independent given $Z_2$ then
$p(Z_1lvert Z_2,Z_3) = p(Z_1lvert Z_2)$
so ones $Z_2$ is "controlled for" $Z_1$ does not depend on $Z_3$.
So the author is assuming that
$$p(Ylvert theta , X, S) = p(Ylvert theta ,X)$$
the dependent variable $Y$ only depends on $S$ through $X$ and $theta$
$$p(theta lvert X,S) = p(theta lvert S)$$
so parameters $theta$ do not depend on $X$ once $S$ is given.
answered Jan 1 at 12:50
Jesper HybelJesper Hybel
1127
$endgroup$
add a comment |
$begingroup$
The assumption applied is that of conditional independence. If $Z_1$ and $Z_2$ are independent then $p(Z_1lvert Z_2) = P(Z_1)$ hence $Z_2$ can be removed from the set of variables on which one is conditioning. Similarly with conditional independence if $Z_1$ and $Z_3$ are conditionally independent given $Z_2$ then
$p(Z_1lvert Z_2,Z_3) = p(Z_1lvert Z_2)$
so ones $Z_2$ is "controlled for" $Z_1$ does not depend on $Z_3$.
So the author is assuming that
$$p(Ylvert theta , X, S) = p(Ylvert theta ,X)$$
the dependent variable $Y$ only depends on $S$ through $X$ and $theta$
$$p(theta lvert X,S) = p(theta lvert S)$$
so parameters $theta$ do not depend on $X$ once $S$ is given.
answered Jan 1 at 12:50
Jesper HybelJesper Hybel
1127
$endgroup$
The assumption applied is that of conditional independence. If $Z_1$ and $Z_2$ are independent then $p(Z_1lvert Z_2) = P(Z_1)$ hence $Z_2$ can be removed from the set of variables on which one is conditioning. Similarly with conditional independence if $Z_1$ and $Z_3$ are conditionally independent given $Z_2$ then
$p(Z_1lvert Z_2,Z_3) = p(Z_1lvert Z_2)$
so ones $Z_2$ is "controlled for" $Z_1$ does not depend on $Z_3$.
So the author is assuming that
$$p(Ylvert theta , X, S) = p(Ylvert theta ,X)$$
the dependent variable $Y$ only depends on $S$ through $X$ and $theta$
$$p(theta lvert X,S) = p(theta lvert S)$$
so parameters $theta$ do not depend on $X$ once $S$ is given.
answered Jan 1 at 12:50
Jesper HybelJesper Hybel
1127
answered Jan 1 at 12:50
Jesper HybelJesper Hybel
1127
answered Jan 1 at 12:50
Jesper HybelJesper Hybel
1127
answered Jan 1 at 12:50
Jesper HybelJesper Hybel
1127
1127
add a comment |
add a comment |
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f2843523%2fbayesian-logistic-regression-conditional-probability-integration%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
$begingroup$
What are $X$ and $Y$? Discrete or continuous random variables (or vectors)? Please include as much context as possible directly in the question instead of pointing to some course.
$endgroup$
– paf
Jul 7 '18 at 7:21
1
$begingroup$
@paf In the context of this problem, X(i) and Y(i) are points in the sample space, S. X(i) is an n dimensional vector and Y(i) is the corresponding classification of that point. X and Y themselves are continuous random variables.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:26
$begingroup$
Treat X as a new point in the hyperspace and P(Y=y|X) as the probability of classifying the new point X as Y=y.
$endgroup$
– Utsav Dutta
Jul 7 '18 at 7:27
$begingroup$
related: math.stackexchange.com/questions/1882178/…
$endgroup$
– Henry
Jul 7 '18 at 8:17
$begingroup$
What does $int_theta ldots dtheta$ means (where $theta$ is a r.v.)?
$endgroup$
– d.k.o.
Jul 7 '18 at 19:24