| Topic: |
Science > Physics |
| User: |
"Gregory L. Hansen" |
| Date: |
15 Jun 2004 03:47:52 PM |
| Object: |
Fitting Functions |
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and got
invtau=0.626. That looks a lot more like the value I expected, so that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a few
percent.
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world?
--
"The polhode rolls without slipping on the herpolhode lying in the
invariable plane." -- Goldstein, Classical Mechanics 2nd. ed., p207.
.
|
|
| User: "Franz Heymann" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 03:42:13 AM |
|
|
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in message
news:cann9o$t4h$1@hood.uits.indiana.edu...
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a
peice of
data and that's that. But depending on the starting conditions I
got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And
I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant,
and got
invtau=0.626. That looks a lot more like the value I expected, so
that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a
few
percent.
When fitting to data like that, how can I have reasonable confidence
that
my number is the one that corresponds to the real world?
Non-linear least square fitting is hell.
If you are using a "search for a minimum" programme, you are very
likely to end up in a local minimum. Shake the parameters a little
when you reach one and continue the search, to see if you can urge it
over the edges of the well. I am a little out of touch, but I think
there are programmes around which do this rattling automatically.
Franz
.
|
|
|
| User: "Richard Herring" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 06:15:35 AM |
|
|
In message <cap154$os9$1@sparta.btinternet.com>, Franz Heymann
<notfranz.heymann@btopenworld.com> writes
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in message
news:cann9o$t4h$1@hood.uits.indiana.edu...
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a
peice of
data and that's that. But depending on the starting conditions I
got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And
I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant,
and got
invtau=0.626. That looks a lot more like the value I expected, so
that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a
few
percent.
When fitting to data like that, how can I have reasonable confidence
that
my number is the one that corresponds to the real world?
Non-linear least square fitting is hell.
If you are using a "search for a minimum" programme, you are very
likely to end up in a local minimum. Shake the parameters a little
when you reach one and continue the search, to see if you can urge it
over the edges of the well. I am a little out of touch, but I think
there are programmes around which do this rattling automatically.
The buzz-phrase is "meta-heuristic optimisation algorithms". Simulated
annealing and tabu search are popular ones. SA might work well for this
problem - basically you make random "thermal" variations in the
parameters with a "temperature" that decreases over time.
--
Richard Herring
.
|
|
|
| User: "Franz Heymann" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 05:42:49 PM |
|
|
"Richard Herring" <junk@[127.0.0.1]> wrote in message
news:bbXe6FZXvC0AFwYw@baesystems.com...
In message <cap154$os9$1@sparta.btinternet.com>, Franz Heymann
<notfranz.heymann@btopenworld.com> writes
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in
message
news:cann9o$t4h$1@hood.uits.indiana.edu...
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a
peice of
data and that's that. But depending on the starting conditions I
got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good.
And
I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held
constant,
and got
invtau=0.626. That looks a lot more like the value I expected,
so
that's
what I'll keep. What the hell, it's as good as any other.
Lowest
chisquare, but the chisquares of the first three only differed by
a
few
percent.
When fitting to data like that, how can I have reasonable
confidence
that
my number is the one that corresponds to the real world?
Non-linear least square fitting is hell.
If you are using a "search for a minimum" programme, you are very
likely to end up in a local minimum. Shake the parameters a little
when you reach one and continue the search, to see if you can urge
it
over the edges of the well. I am a little out of touch, but I
think
there are programmes around which do this rattling automatically.
The buzz-phrase is "meta-heuristic optimisation algorithms".
Never heard of it {:-((
Simulated annealing and tabu search are popular ones.
That's the technique I had in mind when I said rattle the parameters a
little.
SA might work well for this
problem - basically you make random "thermal" variations in the
parameters with a "temperature" that decreases over time.
That's it.
Franz
.
|
|
|
|
| User: "Gregory L. Hansen" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 09:32:55 AM |
|
|
In article <bbXe6FZXvC0AFwYw@baesystems.com>,
Richard Herring <richard.herring@baesystems.com> wrote:
In message <cap154$os9$1@sparta.btinternet.com>, Franz Heymann
<notfranz.heymann@btopenworld.com> writes
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in message
news:cann9o$t4h$1@hood.uits.indiana.edu...
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a
peice of
data and that's that. But depending on the starting conditions I
got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And
I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant,
and got
invtau=0.626. That looks a lot more like the value I expected, so
that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a
few
percent.
When fitting to data like that, how can I have reasonable confidence
that
my number is the one that corresponds to the real world?
Non-linear least square fitting is hell.
If you are using a "search for a minimum" programme, you are very
likely to end up in a local minimum. Shake the parameters a little
when you reach one and continue the search, to see if you can urge it
over the edges of the well. I am a little out of touch, but I think
there are programmes around which do this rattling automatically.
The buzz-phrase is "meta-heuristic optimisation algorithms". Simulated
annealing and tabu search are popular ones. SA might work well for this
problem - basically you make random "thermal" variations in the
parameters with a "temperature" that decreases over time.
That actually sounds like a more sophisticated version of something I was
thinking about. That is, give a range of acceptable initial guesses, and
take random initial guesses from within that range for several trials.
Decreasing the "temperature" over time must be the assumption that after
you've hit a few local minima, the deepest one will be closer to the
global minimum and so you can narrow the search.
--
"Let us learn to dream, gentlemen, then perhaps we shall find the
truth... But let us beware of publishing our dreams before they have been
put to the proof by the waking understanding." -- Friedrich August Kekulé
.
|
|
|
|
|
|
| User: "Old Man" |
|
| Title: Re: Fitting Functions |
15 Jun 2004 06:33:16 PM |
|
|
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in message
news:cann9o$t4h$1@hood.uits.indiana.edu...
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and got
invtau=0.626. That looks a lot more like the value I expected, so that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a few
percent.
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world
Any set of initial values biases the result. Why select
a nonsensical set like (0, 0, 0, 0) ?
Old Man would first set and hold invtau to the expected
value whilst searching on y0, A, and x0. then, holding the
new values for y0, A, and x0 constant, search on invtau
for the best fit. Finally , with the starting values from
the last best fit, search on all for parameters for the best
fit.
[Old Man]
.
|
|
|
| User: "Gregory L. Hansen" |
|
| Title: Re: Fitting Functions |
15 Jun 2004 08:25:26 PM |
|
|
In article <VvCdnQ95qNPdGlLdRVn-gQ@prairiewave.com>,
Old Man <nomail@nomail.net> wrote:
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in message
news:cann9o$t4h$1@hood.uits.indiana.edu...
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and got
invtau=0.626. That looks a lot more like the value I expected, so that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a few
percent.
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world
Any set of initial values biases the result. Why select
a nonsensical set like (0, 0, 0, 0) ?
Old Man would first set and hold invtau to the expected
value whilst searching on y0, A, and x0. then, holding the
new values for y0, A, and x0 constant, search on invtau
for the best fit. Finally , with the starting values from
the last best fit, search on all for parameters for the best
fit.
[Old Man]
I like Old Man's suggestion; give the fitting function as much information
as you can. That's why I included a zero point with the new function
y0 + A*exp(-invtau*(x-x0))
In the form of the built-in function that would be
y0 + ( A*exp(invtau*x0) ) exp(-invtau*x)
and the multiplying factor, which also includes invtau, can get very big
when x0 is far from zero. I had x0=40 minutes, and x0=60 minutes.
Except my initial invtau wasn't especially well known, either. And there
are some quantities, like the thermal contact resistance, that are too
poorly known to allow a very good theoretical starting point.
Be suspicious of any result that looks like x=0.123 +- 5e-13.
--
"You're not as dumb as you look. Or sound. Or our best testing
indicates." -- Monty Burns to Homer Simpson
.
|
|
|
| User: "Franz Heymann" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 03:42:14 AM |
|
|
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in message
news:cao7i6$2j4$2@hood.uits.indiana.edu...
In article <VvCdnQ95qNPdGlLdRVn-gQ@prairiewave.com>,
Old Man <nomail@nomail.net> wrote:
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in
message
news:cann9o$t4h$1@hood.uits.indiana.edu...
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a
peice of
data and that's that. But depending on the starting conditions I
got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good.
And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held
constant, and got
invtau=0.626. That looks a lot more like the value I expected,
so that's
what I'll keep. What the hell, it's as good as any other.
Lowest
chisquare, but the chisquares of the first three only differed by
a few
percent.
When fitting to data like that, how can I have reasonable
confidence that
my number is the one that corresponds to the real world
Any set of initial values biases the result. Why select
a nonsensical set like (0, 0, 0, 0) ?
Old Man would first set and hold invtau to the expected
value whilst searching on y0, A, and x0. then, holding the
new values for y0, A, and x0 constant, search on invtau
for the best fit. Finally , with the starting values from
the last best fit, search on all for parameters for the best
fit.
[Old Man]
I like Old Man's suggestion; give the fitting function as much
information
as you can. That's why I included a zero point with the new
function
y0 + A*exp(-invtau*(x-x0))
In the form of the built-in function that would be
y0 + ( A*exp(invtau*x0) ) exp(-invtau*x)
and the multiplying factor, which also includes invtau, can get very
big
when x0 is far from zero. I had x0=40 minutes, and x0=60 minutes.
Except my initial invtau wasn't especially well known, either. And
there
are some quantities, like the thermal contact resistance, that are
too
poorly known to allow a very good theoretical starting point.
Be suspicious of any result that looks like x=0.123 +- 5e-13.
The eye is actually a good least-square fitter. Why don't you plot
the results, eyeball a curve through it and measure the actual
parameters of the curve. That will give you a good enough starting
point to make it reasonably certain that you will end up at the real
minimum when you do the formal fitting.
Franz
--
"You're not as dumb as you look. Or sound. Or our best testing
indicates." -- Monty Burns to Homer Simpson
.
|
|
|
| User: "Gregory L. Hansen" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 08:57:26 AM |
|
|
In article <cap155$os9$2@sparta.btinternet.com>,
Franz Heymann <franz.heymann@btopenworld.com> wrote:
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in message
news:cao7i6$2j4$2@hood.uits.indiana.edu...
The eye is actually a good least-square fitter. Why don't you plot
the results, eyeball a curve through it and measure the actual
parameters of the curve. That will give you a good enough starting
point to make it reasonably certain that you will end up at the real
minimum when you do the formal fitting.
I've done that more in the reverse way. I'd try a fit, then eyeball the
fitted function and sometimes the residual. But just the fact that it can
be a judgement call makes me wonder how good my judgement is.
--
"In any case, don't stress too much--cortisol inhibits muscular
hypertrophy. " -- Eric Dodd
.
|
|
|
|
|
|
|
| User: "Randy Poe" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 12:42:44 AM |
|
|
(Gregory L. Hansen) wrote in message news:<cann9o$t4h$1@hood.uits.indiana.edu>...
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and got
invtau=0.626.
I wonder what "depending on the starting conditions" means. Are these
all with the same data values? What is your range of data? What values
of A are you coming up with?
These can't all "look" equally good, and they can't all represent
minimum chi-square. I suspect what's happening is that the
numerical nature of the data is such that the chi-square value
is very insensitive to changes in that parameter. So you are
trying to use an iterative procedure to find the minimum of
a surface which has a wide relatively flat region.
That looks a lot more like the value I expected, so that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare,
Well then actually it's not as good as any other, since chisquare
is what you're minimizing. So this is the correct choice.
I think you're seeing poor conditioning. Also perhaps your
termination conditions are not stringent enough, so the
iterative procedure decides it's done when you might want it
to go on longer.
Have you plotted these fits and your data on a log-log
plot? If you do, I suspect you will no longer think they
look equally good. If that's the case, it also suggests
that you should do your fitting on log-log axes, in effect.
There's a numerical analysis newsgroup, and also a statistics
newsgroup. If you want pointers on how best to condition your
data for reliable fitting, those would be the places to go.
The numerical newsgroup is (I think) sci.math.num-analysis
and the stats newsgroup is sci.stat.math.
but the chisquares of the first three only differed by a few
percent.
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world?
.
|
|
|
|
| User: "Ken Muldrew" |
|
| Title: Re: Fitting Functions |
15 Jun 2004 05:52:59 PM |
|
|
(Gregory L. Hansen) wrote:
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world?
What do you mean by that last phrase? Why should a number correspond
to the real world? Do you have some theory behind your fitting
function or did you just pull it out of a hat because the shape of the
function corresponds to your data? I'm not sure if you're trying to
summarize some data in a numerical parameter or if you're trying to
discriminate between different functions (different theories).
Ken Muldrew
kmuldrezw@ucalgazry.ca
(remove all letters after y in the alphabet)
.
|
|
|
| User: "Gregory L. Hansen" |
|
| Title: Re: Fitting Functions |
15 Jun 2004 08:12:50 PM |
|
|
In article <40cf7c8a.11484858@news.ucalgary.ca>,
Ken Muldrew <kmuldrezw@ucalgazry.ca> wrote:
glhansen@steel.ucs.indiana.edu (Gregory L. Hansen) wrote:
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world?
What do you mean by that last phrase? Why should a number correspond
to the real world? Do you have some theory behind your fitting
function or did you just pull it out of a hat because the shape of the
function corresponds to your data? I'm not sure if you're trying to
summarize some data in a numerical parameter or if you're trying to
discriminate between different functions (different theories).
The data is from the temperature versus time of a radiometer target when
heat is applied to it. Everything is sufficiently linear that I expect an
exponential to high precision.
But my question was more general than that particular data set. If you
can get a very different fit depending on your initial guess, how can you
be sure that any particular guess gives the right answer? You can
compare the chi-squares to find the best fit, but like I said, my
chi-squares differed by a few percent for values of invtau that differed
by a factor of two, and then I tried something different and got a lower
chi-square with a different value for invtau. Maybe it was just luck
that I tried the right thing, and maybe I didn't try the right thing and
haven't yet found the lowest chi-square.
--
"There's nary an animal alive that can outrun a greased Scottsman!" --
Groundskeeper Willy
.
|
|
|
|
|
| User: "John T Lowry" |
|
| Title: Re: Fitting Functions |
15 Jun 2004 09:41:08 PM |
|
|
I suspect you need what is referred to as the "F function" for this
purpose.
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in message
news:cann9o$t4h$1@hood.uits.indiana.edu...
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a
peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I
fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and
got
invtau=0.626. That looks a lot more like the value I expected, so
that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a
few
percent.
When fitting to data like that, how can I have reasonable confidence
that
my number is the one that corresponds to the real world?
--
"The polhode rolls without slipping on the herpolhode lying in the
invariable plane." -- Goldstein, Classical Mechanics 2nd. ed., p207.
.
|
|
|
|
| User: "" |
|
| Title: Re: Fitting Functions |
15 Jun 2004 10:08:08 PM |
|
|
In article <cann9o$t4h$1@hood.uits.indiana.edu>, (Gregory L. Hansen) writes:
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and got
invtau=0.626. That looks a lot more like the value I expected, so that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a few
percent.
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world?
In general, you cannot. A sufficient amount of beer will improve the
confidence (ale is better than lager in this respect) and few shots of
Jack Daniels may work even better, but the effect is transient.
To the point, if your fitting procedure estimates only the fit
parameters, not their errors as well, then it is not good enough.
Mati Meron | "When you argue with a fool,
meron@cars.uchicago.edu | chances are he is doing just the same"
.
|
|
|
| User: "Gregory L. Hansen" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 09:26:04 AM |
|
|
In article <sQOzc.31$45.13163@news.uchicago.edu>,
<mmeron@cars3.uchicago.edu> wrote:
In article <cann9o$t4h$1@hood.uits.indiana.edu>,
glhansen@steel.ucs.indiana.edu (Gregory L. Hansen) writes:
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and got
invtau=0.626. That looks a lot more like the value I expected, so that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a few
percent.
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world?
In general, you cannot. A sufficient amount of beer will improve the
confidence (ale is better than lager in this respect) and few shots of
Jack Daniels may work even better, but the effect is transient.
To the point, if your fitting procedure estimates only the fit
parameters, not their errors as well, then it is not good enough.
I use Igor Pro, and that will give fits, uncertainties, chi-squares,
covariance matrices, confidence intervals... I don't even know what
confidence intervals are.
But to give an example from yesterday, these are four different fits to
the same set of data. The first three were fit to y0+A*exp(-invtau*x),
the fourth to y+A*exp(-invtau*(x-x0)) with x0 given and fixed.
invtau chi_square
-------------- ----------
0.245 +- 5e-13 9.92e-14
0.403 +- 0.154 9.50e-14
0.467 +- 0.163 9.40e-14
0.626 +- 0.194 9.32e-14
The first point can clearly be discarded because its uncertainty is
unreasonably small; it has to be an artifact of something going wrong in
the fit. The one with the lowest chi-square has the highest uncertainty.
I haven't tried to find an expected standard deviation in chi-squares, but
e.g. the last two differ by 0.85%, which off the cuff I'd have thought is
not a significant difference, while the returned values differ by 30%,
which is huge. And the data is noisy enough that it's not really obvious
from eyeballing it that one is better than the other.
I'm more inclined to believe the fourth one because of the extra
information I gave it; the heater switched state at x0 = 40 minutes.
The elbow of the exponential can be adjusted by changing either A or
invtau, and the latter fitting function expands to
y0 + A exp(invtau*x0) exp(-invtau*x)
and the former's multiplying factor, A'=A*exp(invtau*x0), makes me
nervous because it contains several physically meaningful parameters and
because that exponential turns small variations into big changes.
It's always better to clean up the raw data, but this doesn't really
contain science, it's just helping me to understand the equipment as I
play with it. I can't think of a good way to clean up the data that
doesn't involve more time and work than it's worth.
--
"A good plan executed right now is far better than a perfect plan
executed next week."
-Gen. George S. Patton
.
|
|
|
| User: "Franz Heymann" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 05:42:48 PM |
|
|
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in message
news:capl9s$he4$1@hood.uits.indiana.edu...
In article <sQOzc.31$45.13163@news.uchicago.edu>,
<mmeron@cars3.uchicago.edu> wrote:
In article <cann9o$t4h$1@hood.uits.indiana.edu>,
glhansen@steel.ucs.indiana.edu (Gregory L. Hansen) writes:
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a
peice of
data and that's that. But depending on the starting conditions I
got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good.
And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant,
and got
invtau=0.626. That looks a lot more like the value I expected, so
that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by
a few
percent.
When fitting to data like that, how can I have reasonable
confidence that
my number is the one that corresponds to the real world?
In general, you cannot. A sufficient amount of beer will improve
the
confidence (ale is better than lager in this respect) and few shots
of
Jack Daniels may work even better, but the effect is transient.
To the point, if your fitting procedure estimates only the fit
parameters, not their errors as well, then it is not good enough.
I use Igor Pro, and that will give fits, uncertainties, chi-squares,
covariance matrices, confidence intervals... I don't even know what
confidence intervals are.
But to give an example from yesterday, these are four different fits
to
the same set of data. The first three were fit to
y0+A*exp(-invtau*x),
the fourth to y+A*exp(-invtau*(x-x0)) with x0 given and fixed.
invtau chi_square
-------------- ----------
0.245 +- 5e-13 9.92e-14
0.403 +- 0.154 9.50e-14
0.467 +- 0.163 9.40e-14
0.626 +- 0.194 9.32e-14
Something is not right.
How come your chi_squares are so minute?
[snip]
Franz
.
|
|
|
| User: "Gregory L. Hansen" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 07:57:15 PM |
|
|
In article <caqid8$cn7$7@titan.btinternet.com>,
Franz Heymann <franz.heymann@btopenworld.com> wrote:
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in message
news:capl9s$he4$1@hood.uits.indiana.edu...
In article <sQOzc.31$45.13163@news.uchicago.edu>,
<mmeron@cars3.uchicago.edu> wrote:
In article <cann9o$t4h$1@hood.uits.indiana.edu>,
glhansen@steel.ucs.indiana.edu (Gregory L. Hansen) writes:
invtau chi_square
-------------- ----------
0.245 +- 5e-13 9.92e-14
0.403 +- 0.154 9.50e-14
0.467 +- 0.163 9.40e-14
0.626 +- 0.194 9.32e-14
Something is not right.
How come your chi_squares are so minute?
Every time I learn about chi-square, I forget it in an hour. All I really
remember is that smaller is supposed to be better.
--
"You're not as dumb as you look. Or sound. Or our best testing
indicates." -- Monty Burns to Homer Simpson
.
|
|
|
| User: "Franz Heymann" |
|
| Title: Re: Fitting Functions |
18 Jun 2004 12:08:26 AM |
|
|
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in message
news:caqq9b$22c$1@hood.uits.indiana.edu...
In article <caqid8$cn7$7@titan.btinternet.com>,
Franz Heymann <franz.heymann@btopenworld.com> wrote:
"Gregory L. Hansen" <glhansen@steel.ucs.indiana.edu> wrote in
message
news:capl9s$he4$1@hood.uits.indiana.edu...
In article <sQOzc.31$45.13163@news.uchicago.edu>,
<mmeron@cars3.uchicago.edu> wrote:
In article <cann9o$t4h$1@hood.uits.indiana.edu>,
glhansen@steel.ucs.indiana.edu (Gregory L. Hansen) writes:
invtau chi_square
-------------- ----------
0.245 +- 5e-13 9.92e-14
0.403 +- 0.154 9.50e-14
0.467 +- 0.163 9.40e-14
0.626 +- 0.194 9.32e-14
Something is not right.
How come your chi_squares are so minute?
Every time I learn about chi-square, I forget it in an hour. All I
really
remember is that smaller is supposed to be better.
No. Have another look at the definition of chi-square.
Franz
.
|
|
|
|
|
|
| User: "" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 04:15:58 PM |
|
|
In article <capl9s$he4$1@hood.uits.indiana.edu>, (Gregory L. Hansen) writes:
In article <sQOzc.31$45.13163@news.uchicago.edu>,
<mmeron@cars3.uchicago.edu> wrote:
In article <cann9o$t4h$1@hood.uits.indiana.edu>,
(Gregory L. Hansen) writes:
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and got
invtau=0.626. That looks a lot more like the value I expected, so that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a few
percent.
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world?
In general, you cannot. A sufficient amount of beer will improve the
confidence (ale is better than lager in this respect) and few shots of
Jack Daniels may work even better, but the effect is transient.
To the point, if your fitting procedure estimates only the fit
parameters, not their errors as well, then it is not good enough.
I use Igor Pro, and that will give fits, uncertainties, chi-squares,
covariance matrices, confidence intervals... I don't even know what
confidence intervals are.
Well, you better check. same as with instrumentation, it is
worthwhile knowing what the various knobs and buttons are doing.
But to give an example from yesterday, these are four different fits to
the same set of data. The first three were fit to y0+A*exp(-invtau*x),
the fourth to y+A*exp(-invtau*(x-x0)) with x0 given and fixed.
invtau chi_square
-------------- ----------
0.245 +- 5e-13 9.92e-14
0.403 +- 0.154 9.50e-14
0.467 +- 0.163 9.40e-14
0.626 +- 0.194 9.32e-14
The first point can clearly be discarded because its uncertainty is
unreasonably small; it has to be an artifact of something going wrong in
the fit.
Yes, my feelings exactly.
The one with the lowest chi-square has the highest uncertainty.
I haven't tried to find an expected standard deviation in chi-squares, but
e.g. the last two differ by 0.85%, which off the cuff I'd have thought is
not a significant difference, while the returned values differ by 30%,
which is huge.
Why do you think it is huge? Looking at the table above they appear
to be within each other's error bars. What you've here is a situation
where the fit quality is nearly independent of the value of this
specific parameter, over a broad range. That's why the error bars are
so big. The less the chi-square depends on a specific parameter, the
poorer is the determination of said paramater.
As to why the fit converges to different values, there are two
possibilities. One is that you've local minima. To investigate this
you would have to generate a sample the function values around such
point and look at it (yes, I know, it is 3D, but you can pick 2d cross
sections). The other possibility (which seems more likely to me in
this case) is that (due to the very slight dependence on nitau) the
convergence to the minimum is so slow that the routine simply decides
that that's good enough and calls it quits. Most minimization routines
use some smallness criterion where when the rate of change (the
gradient, to be strict) gets small enough they decide that "it is good
enough".
And the data is noisy enough that it's not really obvious
from eyeballing it that one is better than the other.
I'm more inclined to believe the fourth one because of the extra
information I gave it; the heater switched state at x0 = 40 minutes.
The elbow of the exponential can be adjusted by changing either A or
invtau, and the latter fitting function expands to
y0 + A exp(invtau*x0) exp(-invtau*x)
and the former's multiplying factor, A'=A*exp(invtau*x0), makes me
nervous because it contains several physically meaningful parameters and
because that exponential turns small variations into big changes.
It is worth than this since it folds few parameters into a single
parameter. For any specific value of nitau, a change in A can be
nullified by an appropriate change in x0. That's an illposed problem,
the answer is no longer unique. I suggest you drop this x0.
It's always better to clean up the raw data, but this doesn't really
contain science, it's just helping me to understand the equipment as I
play with it. I can't think of a good way to clean up the data that
doesn't involve more time and work than it's worth.
Well, you're in a better position to judge than I'm.
Mati Meron | "When you argue with a fool,
meron@cars.uchicago.edu | chances are he is doing just the same"
.
|
|
|
| User: "Gregory L. Hansen" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 05:01:29 PM |
|
|
In article <iM2Ac.13$25.4207@news.uchicago.edu>,
<mmeron@cars3.uchicago.edu> wrote:
In article <capl9s$he4$1@hood.uits.indiana.edu>,
glhansen@steel.ucs.indiana.edu (Gregory L. Hansen) writes:
In article <sQOzc.31$45.13163@news.uchicago.edu>,
<mmeron@cars3.uchicago.edu> wrote:
In article <cann9o$t4h$1@hood.uits.indiana.edu>,
glhansen@steel.ucs.indiana.edu (Gregory L. Hansen) writes:
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and got
invtau=0.626. That looks a lot more like the value I expected, so that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a few
percent.
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world?
In general, you cannot. A sufficient amount of beer will improve the
confidence (ale is better than lager in this respect) and few shots of
Jack Daniels may work even better, but the effect is transient.
To the point, if your fitting procedure estimates only the fit
parameters, not their errors as well, then it is not good enough.
I use Igor Pro, and that will give fits, uncertainties, chi-squares,
covariance matrices, confidence intervals... I don't even know what
confidence intervals are.
Well, you better check. same as with instrumentation, it is
worthwhile knowing what the various knobs and buttons are doing.
But to give an example from yesterday, these are four different fits to
the same set of data. The first three were fit to y0+A*exp(-invtau*x),
the fourth to y+A*exp(-invtau*(x-x0)) with x0 given and fixed.
invtau chi_square
-------------- ----------
0.245 +- 5e-13 9.92e-14
0.403 +- 0.154 9.50e-14
0.467 +- 0.163 9.40e-14
0.626 +- 0.194 9.32e-14
The first point can clearly be discarded because its uncertainty is
unreasonably small; it has to be an artifact of something going wrong in
the fit.
Yes, my feelings exactly.
The one with the lowest chi-square has the highest uncertainty.
I haven't tried to find an expected standard deviation in chi-squares, but
e.g. the last two differ by 0.85%, which off the cuff I'd have thought is
not a significant difference, while the returned values differ by 30%,
which is huge.
Why do you think it is huge? Looking at the table above they appear
to be within each other's error bars. What you've here is a situation
where the fit quality is nearly independent of the value of this
specific parameter, over a broad range. That's why the error bars are
so big. The less the chi-square depends on a specific parameter, the
poorer is the determination of said paramater.
As to why the fit converges to different values, there are two
possibilities. One is that you've local minima. To investigate this
you would have to generate a sample the function values around such
point and look at it (yes, I know, it is 3D, but you can pick 2d cross
sections). The other possibility (which seems more likely to me in
this case) is that (due to the very slight dependence on nitau) the
convergence to the minimum is so slow that the routine simply decides
that that's good enough and calls it quits. Most minimization routines
use some smallness criterion where when the rate of change (the
gradient, to be strict) gets small enough they decide that "it is good
enough".
There's ways to adjust all of that. But when there aren't boxes to check
or fill in, it's easy to forget everything that's there.
And the data is noisy enough that it's not really obvious
from eyeballing it that one is better than the other.
I'm more inclined to believe the fourth one because of the extra
information I gave it; the heater switched state at x0 = 40 minutes.
The elbow of the exponential can be adjusted by changing either A or
invtau, and the latter fitting function expands to
y0 + A exp(invtau*x0) exp(-invtau*x)
and the former's multiplying factor, A'=A*exp(invtau*x0), makes me
nervous because it contains several physically meaningful parameters and
because that exponential turns small variations into big changes.
It is worth than this since it folds few parameters into a single
parameter. For any specific value of nitau, a change in A can be
nullified by an appropriate change in x0. That's an illposed problem,
the answer is no longer unique. I suggest you drop this x0.
No, the x0 is held fixed. I know what it is. And it turned out in
subsequent data sets that by fitting with that function instead of the
built-in function, I get reasonable and similar numbers for the different
segments of heating and cooling, with little effort. And the built-in
function gave values of invtau that were all over the place, with error
bars that were all over the place.
As I'd considered it, that x0 was there in the built-in function all
along, but it was hidden in the multiplying constant as in the expression
I gave above,
A_{built-in} = A*exp(invtau*x0)
x0 is physically meaningful, it's when the heater switches state. A is
physically meaningful, it's the heater output. A_{built-in} is just a
meaningless number.
--
"Let us learn to dream, gentlemen, then perhaps we shall find the
truth... But let us beware of publishing our dreams before they have been
put to the proof by the waking understanding." -- Friedrich August Kekulé
.
|
|
|
| User: "" |
|
| Title: Re: Fitting Functions |
16 Jun 2004 05:28:31 PM |
|
|
In article <caqfvp$ueh$1@hood.uits.indiana.edu>, (Gregory L. Hansen) writes:
In article <iM2Ac.13$25.4207@news.uchicago.edu>,
<mmeron@cars3.uchicago.edu> wrote:
In article <capl9s$he4$1@hood.uits.indiana.edu>,
(Gregory L. Hansen) writes:
In article <sQOzc.31$45.13163@news.uchicago.edu>,
<mmeron@cars3.uchicago.edu> wrote:
In article <cann9o$t4h$1@hood.uits.indiana.edu>,
(Gregory L. Hansen) writes:
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and got
invtau=0.626. That looks a lot more like the value I expected, so that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a few
percent.
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world?
In general, you cannot. A sufficient amount of beer will improve the
confidence (ale is better than lager in this respect) and few shots of
Jack Daniels may work even better, but the effect is transient.
To the point, if your fitting procedure estimates only the fit
parameters, not their errors as well, then it is not good enough.
I use Igor Pro, and that will give fits, uncertainties, chi-squares,
covariance matrices, confidence intervals... I don't even know what
confidence intervals are.
Well, you better check. same as with instrumentation, it is
worthwhile knowing what the various knobs and buttons are doing.
But to give an example from yesterday, these are four different fits to
the same set of data. The first three were fit to y0+A*exp(-invtau*x),
the fourth to y+A*exp(-invtau*(x-x0)) with x0 given and fixed.
invtau chi_square
-------------- ----------
0.245 +- 5e-13 9.92e-14
0.403 +- 0.154 9.50e-14
0.467 +- 0.163 9.40e-14
0.626 +- 0.194 9.32e-14
The first point can clearly be discarded because its uncertainty is
unreasonably small; it has to be an artifact of something going wrong in
the fit.
Yes, my feelings exactly.
The one with the lowest chi-square has the highest uncertainty.
I haven't tried to find an expected standard deviation in chi-squares, but
e.g. the last two differ by 0.85%, which off the cuff I'd have thought is
not a significant difference, while the returned values differ by 30%,
which is huge.
Why do you think it is huge? Looking at the table above they appear
to be within each other's error bars. What you've here is a situation
where the fit quality is nearly independent of the value of this
specific parameter, over a broad range. That's why the error bars are
so big. The less the chi-square depends on a specific parameter, the
poorer is the determination of said paramater.
As to why the fit converges to different values, there are two
possibilities. One is that you've local minima. To investigate this
you would have to generate a sample the function values around such
point and look at it (yes, I know, it is 3D, but you can pick 2d cross
sections). The other possibility (which seems more likely to me in
this case) is that (due to the very slight dependence on nitau) the
convergence to the minimum is so slow that the routine simply decides
that that's good enough and calls it quits. Most minimization routines
use some smallness criterion where when the rate of change (the
gradient, to be strict) gets small enough they decide that "it is good
enough".
There's ways to adjust all of that. But when there aren't boxes to check
or fill in, it's easy to forget everything that's there.
That's why I prefer, to the extent possible, to use routines I write
myself. This way I know what's in there. Well, to be exact, I know
it for a while, but coming back to some of them, later, it often takes
a while to figure out what's going on.
And the data is noisy enough that it's not really obvious
from eyeballing it that one is better than the other.
I'm more inclined to believe the fourth one because of the extra
information I gave it; the heater switched state at x0 = 40 minutes.
The elbow of the exponential can be adjusted by changing either A or
invtau, and the latter fitting function expands to
y0 + A exp(invtau*x0) exp(-invtau*x)
and the former's multiplying factor, A'=A*exp(invtau*x0), makes me
nervous because it contains several physically meaningful parameters and
because that exponential turns small variations into big changes.
It is worth than this since it folds few parameters into a single
parameter. For any specific value of nitau, a change in A can be
nullified by an appropriate change in x0. That's an illposed problem,
the answer is no longer unique. I suggest you drop this x0.
No, the x0 is held fixed. I know what it is.
ah, OK. I was under the impression that it is a fit parameter.
Mati Meron | "When you argue with a fool,
meron@cars.uchicago.edu | chances are he is doing just the same"
.
|
|
|
|
|
|
|
|
| User: "Creighton Hogg" |
|
| Title: Re: Fitting Functions |
15 Jun 2004 04:06:33 PM |
|
|
On Tue, 15 Jun 2004, Gregory L. Hansen wrote:
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and got
invtau=0.626. That looks a lot more like the value I expected, so that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a few
percent.
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world?
Looking thru a book on numerical analysis, I think you have to turn to
statistical techniques to give the probability that the fit is the correct
one. I don't know much about how to do this, except that it's a part of
the branch of statistics called "hypothesis testing". Assumedly books on
applied statistics will go into all the gory, painful detail on how to do
it.
.
|
|
|
| User: "Gregory L. Hansen" |
|
| Title: Re: Fitting Functions |
15 Jun 2004 08:29:43 PM |
|
|
In article <Pine.LNX.4.44.0406151555360.11523-100000@azalea.hep.wisc.edu>,
Creighton Hogg <wchogg@hep.wisc.edu> wrote:
On Tue, 15 Jun 2004, Gregory L. Hansen wrote:
It seems so simple; let the computer fit y0+A*exp(-invtau*x) to a peice of
data and that's that. But depending on the starting conditions I got
invtau = 0.246, 0.405, 0.467, and they all looked pretty good. And I fit
to my own function, y0+A*exp(-invtau(x-x0)) with x0 held constant, and got
invtau=0.626. That looks a lot more like the value I expected, so that's
what I'll keep. What the hell, it's as good as any other. Lowest
chisquare, but the chisquares of the first three only differed by a few
percent.
When fitting to data like that, how can I have reasonable confidence that
my number is the one that corresponds to the real world?
Looking thru a book on numerical analysis, I think you have to turn to
statistical techniques to give the probability that the fit is the correct
one. I don't know much about how to do this, except that it's a part of
the branch of statistics called "hypothesis testing". Assumedly books on
applied statistics will go into all the gory, painful detail on how to do
it.
Hypothesis testing is something different. I suffer from the old problem
of local minima in parameter space. The typical fitting function will
adjust a set of parameters and calculate a chi-square for each set. And
it will go "downhill", selecting the parameters that minimize chi-square.
But it could also walk down a local valley, find the bottom, and call
that the best fit, even if there's a precipice just over the next ridge.
--
"What are the possibilities of small but movable machines? They may or
may not be useful, but they surely would be fun to make."
-- Richard P. Feynman, 1959
.
|
|
|
|
|
|
Related Articles |
|
|
