http://www.johndcook.com/blog/2008/04/04/learning-is-not-the-same-as-gaining-information/ The blog of John D. Cook Sat, 11 Feb 2012 22:42:11 -0500 http://wordpress.org/?v=2.8.4 hourly 1 http://www.johndcook.com/blog/2008/04/04/learning-is-not-the-same-as-gaining-information/comment-page-1/#comment-26354 Gareth Thu, 22 Oct 2009 09:46:39 +0000 http://www.johndcook.com/blog/2008/04/04/learning-is-not-the-same-as-gaining-information/#comment-26354 Wouldn't this problem be resolved if instead you calculated the differential relative (Kullback) entropy between the posterior and the prior? The posterior may be more dispersed than the prior but maybe the differential relative entropy (which is positive) reflects the situation after the observation better then the difference between the differential entropy of the posterior and the differential entropy of the prior (which as you observe is negative in this example). Wouldn’t this problem be resolved if instead you calculated the differential relative (Kullback) entropy between the posterior and the prior? The posterior may be more dispersed than the prior but maybe the differential relative entropy (which is positive) reflects the situation after the observation better then the difference between the differential entropy of the posterior and the differential entropy of the prior (which as you observe is negative in this example).

]]>