Squeezing function defined over infinite range down to finite
$begingroup$
Abstract
What I'm generally trying to do is to squeeze the first derivative of the sigmoid function defined over infinite range into finite. I.e.
$$f(x)=frac{1}{1+e^{-x}}$$
This function is heavily used as an activation function for neural networks. It's integral has close relation to another relu activation function which is also heavily used with neural networks:
$$int{f(x)}{dx}=ln(1+e^{-x})-ln(e^{-x})$$
It's derivative is useful in modeling and is similar to normal distribution pdf function:
$$frac{d}{dx}f(x)=frac{e^{-x}}{(1+e^{-x})^{2}}=df(x)$$
It is also simple to evaluate for computer, it's derivative is expressed in terms of function itself. But $df(x)$ is defined over infinite range which makes it unsuitable for some modeling tasks where the range should be finite. Taking into account the fact that the most of the values close to zero are located outside [-10..10] range, this is shown by the following equation:
$$int_{-infty}^{infty}{df(x)}{dx}-int_{-10}^{10}{df(x)}{dx}=1-frac{e^{10}-e^{-10}}{(1+e^{10})(1+e^{-10})}=0.00009079573740486882...$$
That means that all of the values outside [-10..10] would introduce minor effect to the shape of function when it's range is squeezed from infinite to finite.
So how do we squeeze the range? It can be done through the use of $arctanh(x)$:
That goes as an argument to $df$ function. So when approaching +/-1 arctanh would asymptotically approach +/-$infty$ thus when used as an argument it will map the argument range from infinite to finite:
$$df(arctanh(x))=frac{sqrt{1-x^2}}{(1+frac{sqrt{1-x^2}}{x+1})^2(x+1)}$$
while the function doesn't even look similar to what was originally intended it is easy to get the shape close through a factor before arctanh(x):
$$df(arctanh(x)*6)$$
yields to:
thus getting the final equation equal to:
$$df(n*arctanh(x))=frac{(1-x^2)^{frac n2}}{(1+frac{(1-x^{2})^{frac n2}}{(x+1)^n})^2(x+1)^n}=F(x,n).$$
It's interesting to see how $4*F(x,pi)$ is similar to $frac{sin(xpi+frac{pi}{2})}{2}+frac 12$:
Which could essentially serve as yet another way to approximate sine.
Problem
While playing with $F(x,n)$ I've noticed that it's hard to find values of $n$ that suffice some condition, for instance I want to find $n$ such that it best approximates $frac{sin(xpi+frac{pi}{2})}{2}+frac 12$. Let's define an error function of an approximation:
$$err(x, n) = (4·df(n·arctanh(x))-frac 12·sin(xpi+frac{pi}{2})-frac 12)^2$$
or
$$err(x, n) = (frac{(1-x^2)^{frac n2}}{(1+frac{(1-x^{2})^{frac n2}}{(x+1)^n})^2(x+1)^n}-(frac{sin(xpi+frac{pi}{2})}{2}+frac 12))^2.$$
Then we would need to find $n$ such that:
$$frac{d}{dn}int_{-1}^{1}{err(x, n)}{dx}=0.$$
Please note that $n$ is real here
This is where things start getting messy, an integral over $F(x, n)$ doesn't seem to have an analytical (closed) form nor it is clear how to approach differentiation of such integral due to different integration and differentiation variables. I don't understand how to differentiate such anintegral, so thought it would be better to ask here?
Thank you in advance!
integration derivatives optimization
edited Jan 2 at 18:08
amWhy
1
asked Jan 2 at 17:45
Lu4Lu4
1285
$endgroup$
add a comment |
$begingroup$
Abstract
What I'm generally trying to do is to squeeze the first derivative of the sigmoid function defined over infinite range into finite. I.e.
$$f(x)=frac{1}{1+e^{-x}}$$
This function is heavily used as an activation function for neural networks. It's integral has close relation to another relu activation function which is also heavily used with neural networks:
$$int{f(x)}{dx}=ln(1+e^{-x})-ln(e^{-x})$$
It's derivative is useful in modeling and is similar to normal distribution pdf function:
$$frac{d}{dx}f(x)=frac{e^{-x}}{(1+e^{-x})^{2}}=df(x)$$
It is also simple to evaluate for computer, it's derivative is expressed in terms of function itself. But $df(x)$ is defined over infinite range which makes it unsuitable for some modeling tasks where the range should be finite. Taking into account the fact that the most of the values close to zero are located outside [-10..10] range, this is shown by the following equation:
$$int_{-infty}^{infty}{df(x)}{dx}-int_{-10}^{10}{df(x)}{dx}=1-frac{e^{10}-e^{-10}}{(1+e^{10})(1+e^{-10})}=0.00009079573740486882...$$
That means that all of the values outside [-10..10] would introduce minor effect to the shape of function when it's range is squeezed from infinite to finite.
So how do we squeeze the range? It can be done through the use of $arctanh(x)$:
That goes as an argument to $df$ function. So when approaching +/-1 arctanh would asymptotically approach +/-$infty$ thus when used as an argument it will map the argument range from infinite to finite:
$$df(arctanh(x))=frac{sqrt{1-x^2}}{(1+frac{sqrt{1-x^2}}{x+1})^2(x+1)}$$
while the function doesn't even look similar to what was originally intended it is easy to get the shape close through a factor before arctanh(x):
$$df(arctanh(x)*6)$$
yields to:
thus getting the final equation equal to:
$$df(n*arctanh(x))=frac{(1-x^2)^{frac n2}}{(1+frac{(1-x^{2})^{frac n2}}{(x+1)^n})^2(x+1)^n}=F(x,n).$$
It's interesting to see how $4*F(x,pi)$ is similar to $frac{sin(xpi+frac{pi}{2})}{2}+frac 12$:
Which could essentially serve as yet another way to approximate sine.
Problem
While playing with $F(x,n)$ I've noticed that it's hard to find values of $n$ that suffice some condition, for instance I want to find $n$ such that it best approximates $frac{sin(xpi+frac{pi}{2})}{2}+frac 12$. Let's define an error function of an approximation:
$$err(x, n) = (4·df(n·arctanh(x))-frac 12·sin(xpi+frac{pi}{2})-frac 12)^2$$
or
$$err(x, n) = (frac{(1-x^2)^{frac n2}}{(1+frac{(1-x^{2})^{frac n2}}{(x+1)^n})^2(x+1)^n}-(frac{sin(xpi+frac{pi}{2})}{2}+frac 12))^2.$$
Then we would need to find $n$ such that:
$$frac{d}{dn}int_{-1}^{1}{err(x, n)}{dx}=0.$$
Please note that $n$ is real here
This is where things start getting messy, an integral over $F(x, n)$ doesn't seem to have an analytical (closed) form nor it is clear how to approach differentiation of such integral due to different integration and differentiation variables. I don't understand how to differentiate such anintegral, so thought it would be better to ask here?
Thank you in advance!
integration derivatives optimization
edited Jan 2 at 18:08
amWhy
1
asked Jan 2 at 17:45
Lu4Lu4
1285
$endgroup$
1
$begingroup$
Differentiating with respect to $n$? Is $n$ a real number or natural number here?
$endgroup$
– LoveTooNap29
Jan 2 at 17:56
$begingroup$
$n$ is real number @LoveTooNap29
$endgroup$
– Lu4
Jan 2 at 17:57
$begingroup$
FWIW, derivatives under integral signs can be calculated using the Leibniz integral rule
$endgroup$
– David M.
Jan 3 at 1:41
add a comment |
$begingroup$
Abstract
What I'm generally trying to do is to squeeze the first derivative of the sigmoid function defined over infinite range into finite. I.e.
$$f(x)=frac{1}{1+e^{-x}}$$
This function is heavily used as an activation function for neural networks. It's integral has close relation to another relu activation function which is also heavily used with neural networks:
$$int{f(x)}{dx}=ln(1+e^{-x})-ln(e^{-x})$$
It's derivative is useful in modeling and is similar to normal distribution pdf function:
$$frac{d}{dx}f(x)=frac{e^{-x}}{(1+e^{-x})^{2}}=df(x)$$
It is also simple to evaluate for computer, it's derivative is expressed in terms of function itself. But $df(x)$ is defined over infinite range which makes it unsuitable for some modeling tasks where the range should be finite. Taking into account the fact that the most of the values close to zero are located outside [-10..10] range, this is shown by the following equation:
$$int_{-infty}^{infty}{df(x)}{dx}-int_{-10}^{10}{df(x)}{dx}=1-frac{e^{10}-e^{-10}}{(1+e^{10})(1+e^{-10})}=0.00009079573740486882...$$
That means that all of the values outside [-10..10] would introduce minor effect to the shape of function when it's range is squeezed from infinite to finite.
So how do we squeeze the range? It can be done through the use of $arctanh(x)$:
That goes as an argument to $df$ function. So when approaching +/-1 arctanh would asymptotically approach +/-$infty$ thus when used as an argument it will map the argument range from infinite to finite:
$$df(arctanh(x))=frac{sqrt{1-x^2}}{(1+frac{sqrt{1-x^2}}{x+1})^2(x+1)}$$
while the function doesn't even look similar to what was originally intended it is easy to get the shape close through a factor before arctanh(x):
$$df(arctanh(x)*6)$$
yields to:
thus getting the final equation equal to:
$$df(n*arctanh(x))=frac{(1-x^2)^{frac n2}}{(1+frac{(1-x^{2})^{frac n2}}{(x+1)^n})^2(x+1)^n}=F(x,n).$$
It's interesting to see how $4*F(x,pi)$ is similar to $frac{sin(xpi+frac{pi}{2})}{2}+frac 12$:
Which could essentially serve as yet another way to approximate sine.
Problem
While playing with $F(x,n)$ I've noticed that it's hard to find values of $n$ that suffice some condition, for instance I want to find $n$ such that it best approximates $frac{sin(xpi+frac{pi}{2})}{2}+frac 12$. Let's define an error function of an approximation:
$$err(x, n) = (4·df(n·arctanh(x))-frac 12·sin(xpi+frac{pi}{2})-frac 12)^2$$
or
$$err(x, n) = (frac{(1-x^2)^{frac n2}}{(1+frac{(1-x^{2})^{frac n2}}{(x+1)^n})^2(x+1)^n}-(frac{sin(xpi+frac{pi}{2})}{2}+frac 12))^2.$$
Then we would need to find $n$ such that:
$$frac{d}{dn}int_{-1}^{1}{err(x, n)}{dx}=0.$$
Please note that $n$ is real here
This is where things start getting messy, an integral over $F(x, n)$ doesn't seem to have an analytical (closed) form nor it is clear how to approach differentiation of such integral due to different integration and differentiation variables. I don't understand how to differentiate such anintegral, so thought it would be better to ask here?
Thank you in advance!
integration derivatives optimization
edited Jan 2 at 18:08
amWhy
1
asked Jan 2 at 17:45
Lu4Lu4
1285
$endgroup$
Abstract
What I'm generally trying to do is to squeeze the first derivative of the sigmoid function defined over infinite range into finite. I.e.
$$f(x)=frac{1}{1+e^{-x}}$$
This function is heavily used as an activation function for neural networks. It's integral has close relation to another relu activation function which is also heavily used with neural networks:
$$int{f(x)}{dx}=ln(1+e^{-x})-ln(e^{-x})$$
It's derivative is useful in modeling and is similar to normal distribution pdf function:
$$frac{d}{dx}f(x)=frac{e^{-x}}{(1+e^{-x})^{2}}=df(x)$$
It is also simple to evaluate for computer, it's derivative is expressed in terms of function itself. But $df(x)$ is defined over infinite range which makes it unsuitable for some modeling tasks where the range should be finite. Taking into account the fact that the most of the values close to zero are located outside [-10..10] range, this is shown by the following equation:
$$int_{-infty}^{infty}{df(x)}{dx}-int_{-10}^{10}{df(x)}{dx}=1-frac{e^{10}-e^{-10}}{(1+e^{10})(1+e^{-10})}=0.00009079573740486882...$$
That means that all of the values outside [-10..10] would introduce minor effect to the shape of function when it's range is squeezed from infinite to finite.
So how do we squeeze the range? It can be done through the use of $arctanh(x)$:
That goes as an argument to $df$ function. So when approaching +/-1 arctanh would asymptotically approach +/-$infty$ thus when used as an argument it will map the argument range from infinite to finite:
$$df(arctanh(x))=frac{sqrt{1-x^2}}{(1+frac{sqrt{1-x^2}}{x+1})^2(x+1)}$$
while the function doesn't even look similar to what was originally intended it is easy to get the shape close through a factor before arctanh(x):
$$df(arctanh(x)*6)$$
yields to:
thus getting the final equation equal to:
$$df(n*arctanh(x))=frac{(1-x^2)^{frac n2}}{(1+frac{(1-x^{2})^{frac n2}}{(x+1)^n})^2(x+1)^n}=F(x,n).$$
It's interesting to see how $4*F(x,pi)$ is similar to $frac{sin(xpi+frac{pi}{2})}{2}+frac 12$:
Which could essentially serve as yet another way to approximate sine.
Problem
While playing with $F(x,n)$ I've noticed that it's hard to find values of $n$ that suffice some condition, for instance I want to find $n$ such that it best approximates $frac{sin(xpi+frac{pi}{2})}{2}+frac 12$. Let's define an error function of an approximation:
$$err(x, n) = (4·df(n·arctanh(x))-frac 12·sin(xpi+frac{pi}{2})-frac 12)^2$$
or
$$err(x, n) = (frac{(1-x^2)^{frac n2}}{(1+frac{(1-x^{2})^{frac n2}}{(x+1)^n})^2(x+1)^n}-(frac{sin(xpi+frac{pi}{2})}{2}+frac 12))^2.$$
Then we would need to find $n$ such that:
$$frac{d}{dn}int_{-1}^{1}{err(x, n)}{dx}=0.$$
Please note that $n$ is real here
This is where things start getting messy, an integral over $F(x, n)$ doesn't seem to have an analytical (closed) form nor it is clear how to approach differentiation of such integral due to different integration and differentiation variables. I don't understand how to differentiate such anintegral, so thought it would be better to ask here?
Thank you in advance!
integration derivatives optimization
integration derivatives optimization
edited Jan 2 at 18:08
amWhy
1
asked Jan 2 at 17:45
Lu4Lu4
1285
edited Jan 2 at 18:08
amWhy
1
asked Jan 2 at 17:45
Lu4Lu4
1285
edited Jan 2 at 18:08
amWhy
1
edited Jan 2 at 18:08
amWhy
1
edited Jan 2 at 18:08
amWhy
1
1
asked Jan 2 at 17:45
Lu4Lu4
1285
asked Jan 2 at 17:45
Lu4Lu4
1285
asked Jan 2 at 17:45
Lu4Lu4
1285
1285
1
$begingroup$
Differentiating with respect to $n$? Is $n$ a real number or natural number here?
$endgroup$
– LoveTooNap29
Jan 2 at 17:56
$begingroup$
$n$ is real number @LoveTooNap29
$endgroup$
– Lu4
Jan 2 at 17:57
$begingroup$
FWIW, derivatives under integral signs can be calculated using the Leibniz integral rule
$endgroup$
– David M.
Jan 3 at 1:41
add a comment |
1
$begingroup$
Differentiating with respect to $n$? Is $n$ a real number or natural number here?
$endgroup$
– LoveTooNap29
Jan 2 at 17:56
$begingroup$
$n$ is real number @LoveTooNap29
$endgroup$
– Lu4
Jan 2 at 17:57
$begingroup$
FWIW, derivatives under integral signs can be calculated using the Leibniz integral rule
$endgroup$
– David M.
Jan 3 at 1:41
1
1
$begingroup$
Differentiating with respect to $n$? Is $n$ a real number or natural number here?
$endgroup$
– LoveTooNap29
Jan 2 at 17:56
$begingroup$
Differentiating with respect to $n$? Is $n$ a real number or natural number here?
$endgroup$
– LoveTooNap29
Jan 2 at 17:56
$begingroup$
$n$ is real number @LoveTooNap29
$endgroup$
– Lu4
Jan 2 at 17:57
$begingroup$
$n$ is real number @LoveTooNap29
$endgroup$
– Lu4
Jan 2 at 17:57
$begingroup$
FWIW, derivatives under integral signs can be calculated using the Leibniz integral rule
$endgroup$
– David M.
Jan 3 at 1:41
$begingroup$
FWIW, derivatives under integral signs can be calculated using the Leibniz integral rule
$endgroup$
– David M.
Jan 3 at 1:41
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "69"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3059745%2fsqueezing-function-defined-over-infinite-range-down-to-finite%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Mathematics Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmath.stackexchange.com%2fquestions%2f3059745%2fsqueezing-function-defined-over-infinite-range-down-to-finite%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
$begingroup$
Differentiating with respect to $n$? Is $n$ a real number or natural number here?
$endgroup$
– LoveTooNap29
Jan 2 at 17:56
$begingroup$
$n$ is real number @LoveTooNap29
$endgroup$
– Lu4
Jan 2 at 17:57
$begingroup$
FWIW, derivatives under integral signs can be calculated using the Leibniz integral rule
$endgroup$
– David M.
Jan 3 at 1:41