Comments on: Floating point numbers are a leaky abstraction http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/ The blog of John D. Cook Sat, 11 Feb 2012 01:10:06 -0500 http://wordpress.org/?v=2.8.4 hourly 1 By: Floating point error is the least of my worries — The Endeavour http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-111489 Floating point error is the least of my worries — The Endeavour Tue, 01 Nov 2011 13:23:28 +0000 http://www.johndcook.com/blog/?p=1923#comment-111489 [...] Floating point numbers are a leaky abstraction Avoiding overflow, underflow, and loss of precision Just an approximation [...] [...] Floating point numbers are a leaky abstraction Avoiding overflow, underflow, and loss of precision Just an approximation [...]

]]> By: What I’ve been reading this week: Low level links, Legos for adults and a free AI course by Peter Norvig. http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-98388 What I’ve been reading this week: Low level links, Legos for adults and a free AI course by Peter Norvig. Sun, 14 Aug 2011 01:24:13 +0000 http://www.johndcook.com/blog/?p=1923#comment-98388 [...] into some articles ruminating about floating point arithmetic.  On John D. Cook’s blog,  Numbers are a Leaky Abstraction and Anatomy of a Floating Point Number are two interesting posts that deal with the various [...] [...] into some articles ruminating about floating point arithmetic.  On John D. Cook’s blog,  Numbers are a Leaky Abstraction and Anatomy of a Floating Point Number are two interesting posts that deal with the various [...]

]]> By: John http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-92635 John Tue, 12 Jul 2011 13:00:13 +0000 http://www.johndcook.com/blog/?p=1923#comment-92635 Thanks! I've updated the link with the Oracle one. Thanks! I’ve updated the link with the Oracle one.

]]> By: Chas Emerick http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-92625 Chas Emerick Tue, 12 Jul 2011 11:59:55 +0000 http://www.johndcook.com/blog/?p=1923#comment-92625 Many links to sun.com are now useless…including yours to <a href="http://download.oracle.com/docs/cd/E19957-01/806-3568/ncg_goldberg.html" rel="nofollow">What Every Computer Scientist Should Know About Floating-Point Arithmetic</a> (corrected). Many links to sun.com are now useless…including yours to What Every Computer Scientist Should Know About Floating-Point Arithmetic (corrected).

]]> By: Al Chou http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-92138 Al Chou Sat, 09 Jul 2011 00:16:43 +0000 http://www.johndcook.com/blog/?p=1923#comment-92138 There's a whole discipline -- numerical computation -- about calculating with floating point numbers efficiently and maximizing both accuracy and precision. A lot of the hard work is done with pencil and paper analysis before any code is written. The techniques discussed above are just a few of the things done in numerical analysis. There’s a whole discipline — numerical computation — about calculating with floating point numbers efficiently and maximizing both accuracy and precision. A lot of the hard work is done with pencil and paper analysis before any code is written. The techniques discussed above are just a few of the things done in numerical analysis.

]]> By: Cfr http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-63956 Cfr Thu, 03 Feb 2011 20:36:26 +0000 http://www.johndcook.com/blog/?p=1923#comment-63956 Also, operations with floating point numbers are not pure functional on different machines/compilers. Also, operations with floating point numbers are not pure functional on different machines/compilers.

]]> By: Yaroslav Bulatov http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-48468 Yaroslav Bulatov Wed, 13 Oct 2010 20:43:32 +0000 http://www.johndcook.com/blog/?p=1923#comment-48468 When your errors are multiplicative, and you have choice in the order in which to perform operations, another “trick” is to arrange the computation so that the greatest norm operators are applied last. Here’s an example http://yaroslavvb.blogspot.com/2010/09/order-matters.html When your errors are multiplicative, and you have choice in the order in which to perform operations, another “trick” is to arrange the computation so that the greatest norm operators are applied last. Here’s an example http://yaroslavvb.blogspot.com/2010/09/order-matters.html

]]> By: There isn’t a googol of anything — The Endeavour http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-48448 There isn’t a googol of anything — The Endeavour Wed, 13 Oct 2010 17:39:20 +0000 http://www.johndcook.com/blog/?p=1923#comment-48448 [...] to calculate binomial probabilities Floating point numbers are a leaky abstraction ? [...] [...] to calculate binomial probabilities Floating point numbers are a leaky abstraction ? [...]

]]> By: Will Dwinnell http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-31724 Will Dwinnell Thu, 28 Jan 2010 11:21:02 +0000 http://www.johndcook.com/blog/?p=1923#comment-31724 Tom Ochs wrote a series of articles about exactly this sort of thing in "Computer Language" magazine some years ago. Tom Ochs wrote a series of articles about exactly this sort of thing in “Computer Language” magazine some years ago.

]]> By: Tomas http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-31256 Tomas Wed, 20 Jan 2010 16:01:26 +0000 http://www.johndcook.com/blog/?p=1923#comment-31256 And of course Kevin did talk about it first, my oversight. And of course Kevin did talk about it first, my oversight.

]]> By: John http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-31214 John Wed, 20 Jan 2010 02:35:06 +0000 http://www.johndcook.com/blog/?p=1923#comment-31214 Tomas: I agree that the "move to the logspace" trick is very important. I talk about it <a href="http://www.johndcook.com/blog/2008/04/24/how-to-calculate-binomial-probabilities/" rel="nofollow">here</a> in the context of computing binomial probabilities. Tomas: I agree that the “move to the logspace” trick is very important. I talk about it here in the context of computing binomial probabilities.

]]> By: Tomas http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-31213 Tomas Wed, 20 Jan 2010 02:27:27 +0000 http://www.johndcook.com/blog/?p=1923#comment-31213 I'm surprised to not see the "move to the logspace" basic trick... which is to compute exp(\sum \log x_i) in place of \prod(x_i). Works great for probabilities - they tend to be from some distribution and then central limit works for you... Also normalizing small (or large) numbers to sum up to, say, 1.0 is better accomplished by first subtracting the max of all the logarithms. I’m surprised to not see the “move to the logspace” basic trick… which is to compute exp(\sum \log x_i) in place of \prod(x_i). Works great for probabilities – they tend to be from some distribution and then central limit works for you…
Also normalizing small (or large) numbers to sum up to, say, 1.0 is better accomplished by first subtracting the max of all the logarithms.

]]> By: Don’t invert that matrix — The Endeavour http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-31014 Don’t invert that matrix — The Endeavour Tue, 19 Jan 2010 14:48:46 +0000 http://www.johndcook.com/blog/?p=1923#comment-31014 [...] Floating point numbers are a leaky abstraction ? X [...] [...] Floating point numbers are a leaky abstraction ? X [...]

]]> By: links for 2009-04-13 | Yostivanich.com http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-16202 links for 2009-04-13 | Yostivanich.com Mon, 20 Apr 2009 02:17:05 +0000 http://www.johndcook.com/blog/?p=1923#comment-16202 [...] Floating point numbers are a leaky abstraction — The Endeavour Good remainder of some of the very real problems relying upon an abstracted system for accuracy. (tags: computer computerscience technology programming mathematics precision accuracy) [...] [...] Floating point numbers are a leaky abstraction — The Endeavour Good remainder of some of the very real problems relying upon an abstracted system for accuracy. (tags: computer computerscience technology programming mathematics precision accuracy) [...]

]]> By: John http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-16192 John Sun, 19 Apr 2009 18:40:30 +0000 http://www.johndcook.com/blog/?p=1923#comment-16192 Jack: Every floating point number does represent a rational number, so I could see your point. But I'd still say that floating point numbers are trying to approximate real numbers. Arithmetic operations are designed to be approximately correct in the analytical sense of the real line, not exactly correct in the algebraic sense of a quotient field. I think fixed-point numbers are a better model for rational numbers. Jack: Every floating point number does represent a rational number, so I could see your point. But I’d still say that floating point numbers are trying to approximate real numbers. Arithmetic operations are designed to be approximately correct in the analytical sense of the real line, not exactly correct in the algebraic sense of a quotient field. I think fixed-point numbers are a better model for rational numbers.

]]> By: Jack http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-16191 Jack Sun, 19 Apr 2009 18:16:04 +0000 http://www.johndcook.com/blog/?p=1923#comment-16191 I disagree with your idea that floating-point is supposed to represent "real numbers". They represent rational numbers. If you want to approximate reals with rationals, that's your choice, but you should at least understand the difference, and why you can get into trouble thereby. I disagree with your idea that floating-point is supposed to represent “real numbers”. They represent rational numbers.

If you want to approximate reals with rationals, that’s your choice, but you should at least understand the difference, and why you can get into trouble thereby.

]]> By: Math Teachers at Play #5 « Let’s Play Math! http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-16075 Math Teachers at Play #5 « Let’s Play Math! Fri, 17 Apr 2009 11:16:58 +0000 http://www.johndcook.com/blog/?p=1923#comment-16075 [...] Cook presents Floating point numbers are a leaky abstraction. This is the first of two blog posts about how computer arithmetic works and what problems to look [...] [...] Cook presents Floating point numbers are a leaky abstraction. This is the first of two blog posts about how computer arithmetic works and what problems to look [...]

]]> By: John http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-16052 John Fri, 17 Apr 2009 02:48:26 +0000 http://www.johndcook.com/blog/?p=1923#comment-16052 David Eisner: I updated the links to point to an updated version of Goldberg's paper. Thanks for pointing out the error. David Eisner: I updated the links to point to an updated version of Goldberg’s paper. Thanks for pointing out the error.

]]> By: EastwoodDC http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-16001 EastwoodDC Thu, 16 Apr 2009 04:40:52 +0000 http://www.johndcook.com/blog/?p=1923#comment-16001 This also ties in with your post a while back on accurately calculation the variance. This also ties in with your post a while back on accurately calculation the variance.

]]> By: John Cowan http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-15889 John Cowan Tue, 14 Apr 2009 01:28:41 +0000 http://www.johndcook.com/blog/?p=1923#comment-15889 Not to mention the problems that arise because we think in decimal but compute in binary. Does 0.1 + 0.2 = 0.3? No, not in binary floats, it doesn't. Not to mention the problems that arise because we think in decimal but compute in binary. Does 0.1 + 0.2 = 0.3? No, not in binary floats, it doesn’t.

]]> By: David Eisner http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-15879 David Eisner Mon, 13 Apr 2009 20:29:10 +0000 http://www.johndcook.com/blog/?p=1923#comment-15879 Did anybody else notice the error in Figure D-1 of David Goldberg's paper (http://docs.sun.com/source/806-3568/ncg_goldberg.html#1374) ? The "0" should be at the left of the number line, not at the hash mark corresponding to 1/2. Did anybody else notice the error in Figure D-1 of David Goldberg’s paper (http://docs.sun.com/source/806-3568/ncg_goldberg.html#1374) ? The “0″ should be at the left of the number line, not at the hash mark corresponding to 1/2.

]]> By: Kevin Peterson http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-15843 Kevin Peterson Mon, 13 Apr 2009 06:43:06 +0000 http://www.johndcook.com/blog/?p=1923#comment-15843 Don't forget a Bayesian classifier -- it involves multiplying large numbers of small probabilities, then normalizing them. After getting strange results, and switching to BigDecimal, I learned that the usual solution is to do all calculations in log space. Don’t forget a Bayesian classifier — it involves multiplying large numbers of small probabilities, then normalizing them. After getting strange results, and switching to BigDecimal, I learned that the usual solution is to do all calculations in log space.

]]> By: John Moeller http://www.johndcook.com/blog/2009/04/06/numbers-are-a-leaky-abstraction/comment-page-1/#comment-15456 John Moeller Mon, 06 Apr 2009 17:51:11 +0000 http://www.johndcook.com/blog/?p=1923#comment-15456 Nice post. This is a huge problem in sums too, especially when the addends are different by many orders of magnitude. Nice post. This is a huge problem in sums too, especially when the addends are different by many orders of magnitude.

]]>