In this post, I said that an Archimedean spiral has the polar equation

*r* = *b* θ^{1/n}

and applied this here to rolls of carpet.

When *n* = 1, the length of the spiral for θ running from 0 to *T* is approximately

½ *bT*²

with the approximation becoming more accurate [1] as *T* increases.

In this post we want to look at the general case where *n* might not be 1. In that case the arc length is given by a hypergeometric function, and finding the asymptotic behavior for large *T* requires evaluating a hypergeometric function at a large argument.

Here’s an example with *n* = 3.

Now the arms are not equally spaced but instead grow closer together.

According to MathWorld, the length of the spiral with *n* > 1 and θ running from 0 to *T* is given by

As I discussed here, the power series defining a hypergeometric function _{2}*F*_{1} diverges for arguments outside the unit disk, but the function can be extended by analytic continuation using the identity

Most of the terms above will drop out in the limit as *z* = −*n*²*T*² gets large. The 1/*z* terms go to zero, and hypergeometric functions equal 1 when *z* = 0, and so the *F* terms on the right hand side above go to 1 as *T* increases.

In our application the hypergeometric parameters are *a* = −1/2, *b* = 1/2*n*, and *c* = 1 + 1/2*n*. The term

(−*z*)^{−a} = (−*z*)^{1/2}

matters, but the term

(−*z*)^{−b} = (−*z*)^{−1/2n}

goes to zero and can be ignored.

We find that

*L* ≈ *k T*^{1 + 1/n}

where the constant *k* is given by

*k* = *bn* Γ(1 + 1/2*n*) Γ(1/2 + 1/2*n*) / Γ(1/2*n*) Γ(3/2 + 1/2*n*)

When *n* = 1, this reduces to *L* ≈ ½ *bT*² as before.

[1] The *relative* approximation error decreases approximately quadratically in *T*. But the *absolute* error grows algorithmically.

If you know the dimensions of a carpet, what will the dimensions be when you roll it up into a cylinder?

If you know the dimensions of a rolled-up carpet, what will the dimensions be when you unroll it?

This post answers both questions.

The edge of a rolled-up carpet can be described as an Archimedian spiral. In polar coordinates, this spiral has the equation

*r* = *h*θ / 2π

where *h* is the thickness of the carpet. The previous post gave an exact formula for the length *L* of the spiral, given the maximum value of θ which we denoted *T*. It also gave a simple approximation that is quite accurate when *T* is large, namely

*L* = *hT*² / 4π

If *r*_{1} is the radius of the carpet as a rolled up cylinder, *r*_{1} = *hT* / 2π and so *T* = 2π *r*_{1} / *h*. So when we **unroll** the carpet

*L* = *hT*² / 4π = π*r*_{1}² / *h*.

Now suppose we know the length *L* and want to find the radius *r* when we **roll up** the carpet.

*T* = √(*hL*/π).

The discussion so far has assumed that the spiral starts from the origin, i.e. the carpet is rolled up so tightly that there’s no hole in the middle. This may be a reasonable assumption for a very flexible carpet. But if the carpet is stiff, the rolled up carpet will not be a solid cylinder but a cylinder with a cylindrical hole in the middle.

In the case of a hollow cylinder, there is an inner radius *r*_{0} and an outer radius *r*_{1}. This means θ runs from *T*_{0} = 2π *r*_{0}/*h* to *T*_{1} = 2π*r*_{1}/*h*.

To find the length of the spiral running from *T*_{0 } to *T*_{1} we find the length of a spiral that runs from 0 to *T*_{1} and subtract the length of a spiral from 0 to *T*_{0}

*L* = π*r*_{1}² / *h* − π*r*_{0}² / *h* = π(*r*_{1}² − *r*_{0}²)/*h*.

This approximation is even better because the approximation is least accurate for small *T*, and we’ve subtracted off that part.

Now let’s go the other way around and find the outer radius *r*_{1} when we know the length *L*. We also need to know the inner radius *r*_{0}. So suppose we are wrapping the carpet around a cardboard tube of radius *r*_{0}. Then

*r*_{1} = √(*r*_{0}² + *hL*/π).

*r* = *b* θ^{1/n}

This post will look at the case *n* = 1. I may look at more general values of *n* in a future post. (Update: See here.) The case *n* = 1 is the simplest case, and it’s the case I needed for the client project that motivated this post.

In this case the spacing between points where the spiral crosses an axis is constant. Call this constant *h*. Then

*h* = 2π*b*.

For example, when rolling up a carpet, *h* corresponds to the thickness of the carpet.

Suppose θ runs from 0 to 2π*m*, wrapping around the origin *m* times. We could approximate the spiral by *m* concentric circles of radius *h*, 2*h*, 3*h*, …, *mh*. To visualize this, we’re approximating the length of the red spiral on the left with that of the blue circles on the right.

We could approximate this further by saying we have *m*/2 circles whose average radius is π*mb*. This suggests the length of the spiral should be approximately

2π²*m*²*b*

How good is this approximation? What happens to the relative error as θ increases? Intuitively, each wrap around the origin is more like a circle as θ increases, so we’d expect the approximation to improve for large θ.

According to Mathworld, the exact length of the spiral is

π*bm* √(1 + (2π*m*)²) + *b* arcsinh(2π*m*) /2

When *m* is so large that we can ignore the 1 in √(1 + (2π*m*)²) then the first term is the same as the circle approximation, and all that’s left is the arcsinh term, which is on the order of log *m* because

arcsinh(*x*) = log(*x* + (1 + *x*²)^{1/2}).

So for large *m*, the arc length is on the order of *m*² while the error is on the order of log *m*. This means the relative error is *O*( log(*m*) / *m*² ). [1]

We’ve assumed *m* was an integer because that makes it easier to visual approximating the spiral by circles, but that assumption is not necessary. We could restate the problem in terms of the final value of θ. Say θ runs from 0 to *T*. Then we could solve

*T* = 2π*m*

for *m* and say that the approximate arc length is

½ *bT*²

and the exact length is

½ *b**T*(1 + *T*²)^{1/2} + ½ *b* arcsinh(*T*).

The relative approximation error is *O*( log(*T*) / *T*² ).

Here’s a plot of the error as a function of *T* assuming *b* = 1.

[1] The error in approximating √(1 + (2π*m*)²) with 2π*m* is on the order of 1/(4π*m*) and so is smaller than the logarithmic term.

The hypergeometric function *F*(*a*, *b*; *c*; *z*) is defined by a power series in *z* whose coefficients are functions of *a*, *b*, and *c*. However, this power series has radius of convergence 1. This means you can’t use the series to evaluate *F*(*a*, *b*; *c*; *z*) for *z < −*1.

It’s important to keep in mind the difference between a function and its power series representation. The former may exist where the latter does not. A simple example is the function *f*(*z*) = 1/(1 − *z*). The power series for this function has radius 1, but the function is defined everywhere except at *z* = 1.

Although the series defining *F*(*a*, *b*; *c*; *z*) is confined to the unit disk, the function itself is not. It can be extended analytically beyond the unit disk, usually with a branch cut along the real axis for *z* ≥ 1.

It’s good to know that our function *can* be evaluated for large negative *x*, but *how* do we evaluate it?

Hypergeometric functions satisfy a huge number of identities, the simplest of which are known as the linear transformation formulas even though they are not linear transformations of *z*. They involve *bilinear* transformations *z*, a.k.a. fractional linear transformations, a.k.a. Möbius transformations. [1]

One such transformation is the following, found in A&S 15.3.4 [2].

If *z* < 1, then 0 < *z*/(*z* − 1) < 1, which is inside the radius of convergence. However, as *z* goes off to −∞, *z*/(*z* − 1) approaches 1, and the convergence of the power series will be slow.

A more complicated, but more efficient, formula is A&S 15.3.7, a linear transformation formula relates *F* at *z* to two other hypergeometric functions evaluated at 1/*z*. Now when *z* is large, 1/*z* is small, and these series will converge quickly.

[1] It turns out these transformations *are* linear, but not as functions of a complex argument. They’re linear as transformations on a projective space. More on that here.

[2] A&S refers to the venerable Handbook of Mathematical Functions by Abramowitz and Stegun.

The post Hypergeometric function of a large negative argument first appeared on John D. Cook.]]>`more`

will show you more of your file, one screen at a time.
Now everyone uses `less`

, and `more`

is all but forgotten.

Daniel Halbert wrote `more`

in 1978. Mark Nudelman a similar program with more functionality in 1984 which he named `less`

. The name was a pun on the aphorism “less is more” [1]. Soon `less`

completely replaced `more`

.

I’m curious why I ever used `more`

, since `less`

had taken over before I touched Unix. One possibility is that someone who was accustomed to `more`

showed me that command. Another possibility is that I learned it from reading The Unix Programming Environment which came out in November 1983. It includes `more`

but not `less`

.

My laptop contains executables for `more`

and `less`

in `/usr/bin`

. The command

diff less more

returns nothing, indicating that the binaries are identical: `less`

literally is `more`

.

My desktop has distinct binaries for `less`

and `more`

. The `more`

binary is much smaller, and so I assume it is limited to the original functionality of `more`

, more or less.

- Less isn’t more. Just enough is more.
- More data, less accuracy
- More theoretical power, less real power

[1] I don’t know who coined the phrase “less is more,” but it is associated with architect Ludwig Mies van der Rohe (1886–1969) who often quoted it. He did not apply the principle to is own name, however. He was born Ludwig Mies and later appended van der Rohe.

The post More is less first appeared on John D. Cook.]]>So much of 20th century statistics was just a waste of time, computing precise answers to useless questions.

He’s right. I taught mathematical statistics at GSBS [1, 2] several times, and each time I taught it I became more aware of how pointless some of the material was.

I do believe mathematical statistics is useful, even some parts whose usefulness isn’t immediately obvious, but there were other parts of the curriculum I couldn’t justify spending time on [3].

I’ll say this in partial defense of computing precise answers to useless questions: it can be fun and good for your career.

Mathematics problems coming out of statistics can be more interesting, and even more useful, than the statistical applications that motivate them. Several times in my former career a statistical problem of dubious utility lead to an interesting mathematical puzzle.

Solving practical research problems in statistics is *hard*, and can be hard to publish. If research addresses a practical problem that a reviewer isn’t aware of, a journal may reject it. The solution to a problem in mathematical statistics, regardless of its utility, is easier to publish.

Outside of academia there is less demand for precise answers to useless questions. A consultant can be sure that a client finds a specific project useful because they’re willing to directly spend money on it. Maybe the client is mistaken, but they’re betting that they’re not.

Academics get grants for various lines of research, but this isn’t the same kind of feedback because the people who approve grants are not spending their own money. Imagine a grant reviewer saying “I think this research is so important, I’m not only going to approve this proposal, I’m going to write the applicant a $5,000 personal check.”

Consulting clients may be giving away someone else’s money too, but they have a closer connection to the source of the funds than a bureaucrat has when dispensing tax money.

[1] When I taught there, GSBS was The University of Texas Graduate School of Biomedical Sciences. I visited their website this morning, and apparently GSBS is now part of, or at least co-branded with, MD Anderson Cancer Center.

There was a connection between GSBS and MDACC at the time. Some of the GSBS instructors, like myself, were MDACC employees who volunteered to teach a class.

[2] Incidentally, there was a connection between GSBS and Allen Downey: one of my students was his former student, and he told me what a good teacher Allen was.

[3] I talk about utility a lot in this post, but I’m not a utilitarian. There are good reasons to learn things that are not obviously useful. But the appeal of statistics is precisely its utility, and so statistics that isn’t useful is particularly distasteful.

Pure math is beautiful (and occasionally useful) and applied math is useful (and occasionally beautiful). But there’s no reason to study fake applied math that is neither beautiful or useful.

The post Precise answers to useless questions first appeared on John D. Cook.]]>In five-card poker, the more pairs the better. Better here means less likely. One pair is better than no pair, and two pairs is better than one pair. But in six-card or seven-card poker, a hand with no pair is less likely than a hand with one pair.

For a five-card hand, the probabilities of 0, 1, or 2 pair are 0.5012, 0.4226, and 0.0475 respectively.

For a six-card hand, the same probabilities are 0.3431, 0.4855, and 0.1214.

For a seven-card hand, the probabilities are 0.2091, 0.4728, and 0.2216.

- Chances a card doesn’t move in a shuffle
- Matches between two decks of cards
- Serious applications of a party trick

[1] Y. L. Cheung. Why Poker Is Played with Five Cards. The Mathematical Gazette, Dec., 1989, Vol. 73, No. 466 (Dec., 1989), pp. 313–315

The post Pairs in poker first appeared on John D. Cook.]]>What I find more surprising is that a systematic search finds mean relationships that are far more accurate. The radius of Jupiter is within 5% of the geometric mean of the radii of the Earth and Sun. But all the mean relations below have an error less than 1%.

The radius of Mercury equals the geometric mean of the radii of the Moon and Mars, within 0.052%.

The radius of Mars equals the harmonic mean of the radii of the Moon and Jupiter, within 0.08%.

The radius of Uranus equals the arithmetic-geometric mean of the radii of Earth and Saturn, within 0.0018%.

See the links below for more on AM, GM, and AGM.

Now let’s look at masses.

The mass of Earth is the geometric mean of the masses of Mercury and Neptune, within 2.75%. This is the least accurate approximation in this post.

The mass of Pluto is the harmonic mean of the masses of the Moon and Mars, within 0.7%.

The mass of Uranus is the arithmetic-geometric mean of the masses of of the Moon and Saturn, within 0.54%.

In terms of radii,

The ratio on the left equals 9.95 and the ratio on the left equals 10.98.

The subscripts are the astronomical symbols for the Sun (☉, U+2609), Jupiter (♃, U+2643), and Earth (, U+1F728). I produced them in LaTeX using the `mathabx`

package and the commands `\Sun`

, `Jupiter`

, and `Earth`

.

The the `mathabx`

symbol for Jupiter is a little unusual. It looks italicized, but that’s not because the symbol is being used in math mode. Notice that the vertical bar in the symbol for Earth is vertical, i.e. not italicized.

The post Earth : Jupiter :: Jupiter : Sun first appeared on John D. Cook.]]>

I was listening to the latest episode of the Space Rocket History podcast. The show includes some audio from a documentary on Pioneer 11 that mentioned that a man would weigh 500 pounds on Jupiter.

My immediate thought was “Is that all?! Is this ‘man’ a 100 pound boy?”

The documentary was correct and my intuition was wrong. And the implied mass of the man in the documentary is 190 pounds.

Jupiter has more than 300 times more mass than the earth. Why is its surface gravity only 2.6 times that of the earth?

Although Jupiter is very massive, it is also very large. Gravitational attraction is proportional to mass, but inversely proportional to the square of distance.

A satellite in orbit 100,000 km from the center of Jupiter would feel 300 times as much gravity as one in orbit the same distance from the center of Earth. But the surface of Jupiter is further from its center of mass than the surface of Earth is from its center of mass.

The mass of Jupiter is 318 times that of Earth, and the its mean radius is 11 times that of Earth. So the ratio of gravity on the surface of Jupiter to gravity on the Earth’s surface is

318 / 11² = 2.63

Now suppose a planet had the same density as Earth but a radius of *r* Earth radii. Then its mass would be *r*³ times greater, but its surface gravity would only be *r* times greater since gravity follows an inverse square law. So if Jupiter were made of the same stuff as Earth, its surface gravity would be 11 times greater. But Jupiter is a gas giant, so its surface gravity is only 2.6 times greater.

The people who generate regulatory guidance documents are not legislators. Legislators delegate to agencies to make rules, and agencies delegate to other organizations to make guidelines. For example [1],

Even HHS, which has express cybersecurity rulemaking authority under the Health Insurance Portability and Accountability Act (HIPAA), has put a lot of the details of what it considers adequate cybersecurity into non-binding guidelines.

I’m not a lawyer, so nothing I can should be considered legal advice. However, the authors of [1] are lawyers.

The legal status of guidance documents is contested. According to [2], Executive Order 13892 said that agencies

may not treat noncompliance with a standard of conduct announced solely in a guidance document as itself a violation of applicable statutes or regulations.

Makes sense to me, but EO 13992 revoked EO 13892.

Again according to [3],

Under the common law, it used to be that government advisories, guidelines, and other non-binding statements were non-binding hearsay [in private litigation]. However, in 1975, the Fifth Circuit held that advisory materials … are an exception to the hearsay rule … It’s not clear if this is now the majority rule.

In short, it’s fuzzy.

[1] Jim Dempsey and John P. Carlin. Cybersecurity Law Fundamentals, Second Edition, page 245.

[2] Ibid., page 199.

[3] Ibid., page 200.

The post Are guidance documents laws? first appeared on John D. Cook.]]>

Laguerre’s method is very robust in the sense that it is likely to converge to a root, regardless of the starting point. However, it may be difficult to predict *which* root the method will end up at. To visualize this, we color points according to which root they converge to.

First, let’s look at the polynomial

(*x* − 2)(*x* − 4)(*x* − 24)

which clearly has roots at 2, 4, and 24. We’ll generate random starting points and color them blue, orange, or green depending on whether they converge to 2, 4, or 24. Here’s the result.

To make this easier to see, let’s split it into each color: blue, orange, and green.

Now let’s change our polynomial by moving the root at 4 to 4*i*.

(*x* − 2)(*x* − 4*i*)(*x* − 24)

Here’s the combined result.

And here is each color separately.

As we explained last time, the area taken up by the separate colors seems to exceed the total area. That is because the colors are so intermingled that many of the dots in the images cover some territory that belongs to another color, even though the dots are quite small.

The post More Laguerre images first appeared on John D. Cook.]]>The captcha was to listen to three audio clips at a time and say which one contains bird sounds. This is a really clever test, because humans can tell the difference between real bird sounds and synthesized bird-like sounds. And we’re generally good at recognizing bird sounds even against a background of competing sounds. But some of these were ambiguous, and I had real birds chirping outside my window while I was doing the captcha.

You have to do 20 of these tests, and apparently you have to get all 20 right. I didn’t. So I tried again. On the last test I accidentally clicked the start-over button rather than the submit button. I wasn’t willing to listen to another 20 triples of audio clips, so I switched over to the visual captcha tests.

These kinds of tests could be made less annoying and more secure by using a Bayesian approach.

Suppose someone solves 19 out of 20 puzzles correctly. You require 20 out of 20, so you have them start over. When you do, you’re throwing away information. You require 20 more puzzles, despite the fact that they only missed one. And if a bot had solved say 8 out of 20 puzzles, you’d let it pass if it passes the next 20.

If you wipe your memory after every round of 20 puzzles, and allow unlimited do-overs, then any sufficiently persistent entity will almost certainly pass eventually.

Bayesian statistics reflects common sense. After someone (or something) has correctly solved 19 out of 20 puzzles designed to be hard for machines to solve, your conviction that this entity is human is higher than if they/it had solved 8 out of 20 correctly. You don’t need as much additional evidence in the first case as in the latter to be sufficiently convinced.

Here’s how a Bayesian captcha could work. You start out with some distribution on your probability θ that an entity is human, say a uniform distribution. You present a sequence of puzzles, recalculating your posterior distribution after each puzzle, until the posterior probability that this entity is human crosses some upper threshold, say 0.95, or some lower threshold, say 0.50. If the upper threshold is crossed, you decide the entity is likely human. If the lower threshold is crossed, you decide the entity is likely not human.

If solving 20 out of 20 puzzles correctly crosses your threshold of human detection, then after solving 19 out 20 correctly your posterior probability of humanity is close to the upper threshold and would only require a few more puzzles. And if an entity solved 8 out of 20 puzzles correctly, that may cross your lower threshold. If not, maybe only a few more puzzles would be necessary to reject the entity as non-human.

When I worked at MD Anderson Cancer Center we applied this approach to adaptive clinical trials. A clinical trial might stop early because of particularly good results or particularly bad results. Clinical trials are expensive, both in terms of human costs and financial costs. Rejecting poor treatments quickly, and sending promising treatments on to the next stage quickly, is both humane and economical.

This morning I read a really good article, Fifty Things you can do with a Software Defined Radio. The article includes a rule of thumb for how long an antenna needs to be.

My rule of thumb was to divide 72 by the frequency in MHz, and take that as the length of each side of the dipole in meters [1]. That’d make the whole antenna a bit shorter than half of the wavelength.

Ideally an antenna should be as long as half a wavelength of the signal you want to receive. Light travels 3 × 10^{8} meters per second, so one wavelength of a 1 MHz signal is 300 m. A quarter wavelength, the length of one side of a dipole antenna, would be 75 m. Call it 72 m because 72 has lots of small factors, i.e. it’s usually mentally easier to divide things into 72 than 75. Rounding 75 down to 72 results in the antenna being a little shorter than ideal. But antennas are forgiving, especially for receiving.

**Update**: There’s more to replacing 75 with 72 than simplifying mental arithmetic. See Markus Meier’s comment below.

Just as the Rule of 72 for antennas rounds 75 down to 72, the Rule of 72 for interest rounds 69.3 up to 72, both for ease of mental calculation.

The approximation step comes from the approximation log(1 + *x*) ≈ *x* for small *x*, a first order Taylor approximation.

The last line would be 2 rather than 2.05 if we replaced 72 with 100 log(2) = 69.3. That’s where the factor of 69.3 mentioned above comes from.

[1] The post actually says centimeters, but the author meant to say meters.

The post Antenna length: Another rule of 72 first appeared on John D. Cook.]]>AlphaFold 2, FourCastNet and CorrDiff are exciting. AI-driven autonomous labs are going to be a big deal [1]. Science codes now use AI and machine learning to make scientific discoveries on the world’s most powerful computers [2].

It’s common practice for scientists to ask questions about the validity, reliability and accuracy of the mathematical and computational methods they use. And many have voiced concerns about the lack of explainability and potential pitfalls of AI models, in particular deep neural networks (DNNs) [3].

The impact of this uncertainty varies highly according to project. Science projects that are able to easily check AI-generated results against ground truth may not be that concerned. High-stakes projects like design of a multimillion dollar spacecraft with high project risks may ask about AI model accuracy with more urgency.

Understanding of the properties of DNNs is still in its infancy, with many as-yet unanswered questions. However, in the last few years some significant results have started to come forth.

A fruitful approach to analyzing DNNs is to see them as function approximators (which, of course, they are). One can study how accurately DNNs approximate a function representing some physical phenomenon in a domain (for example, fluid density or temperature).

The approximation error can be measured in various ways. A particularly strong measure is “sup-norm” or “max-norm” error, which requires that the DNN approximation be accurate at *every point* of the target function’s domain (“uniform approximation”). Some science problems may have a weaker requirement than this, such as low RMS or 2-norm error. However, it’s not unreasonable to ask about max-norm approximation behaviors of numerical methods [4,5].

An illuminating paper by Ben Adcock and Nick Dexter looks at this problem [6]. They show that standard DNN methods applied even to a simple 1-dimensional problem can result in “glitches”: the DNN as a whole matches the function well but at some points totally misapproximates the target function. For a picture that shows this, see [7].

Other mathematical papers have subsequently shed light on these behaviors. I’ll try to summarize the findings below, though the actual findings are very nuanced, and many details are left out here. The reader is advised to refer to the respective papers for exact details.

The findings address three questions: 1) how many DNN parameters are required to approximate a function well? 2) how much data is required to train to a desired accuracy? and 3) what algorithms are capable of training to the desired accuracy?

How large does the neural network need to be for accurate uniform approximation of functions? If tight max-norm approximation requires an excessively large number of weights, then use of DNNs is not computationally practical.

Some answers to this question have been found—in particular, a result ^{1} is given in [8, Theorem 4.3; cf. 9, 10]. This result shows that the number of neural network weights required to approximate an arbitrary function to high max-norm accuracy grows *exponentially* in the dimension of the input to the function.

This dependency on dimension is no small limitation, insofar as this is not the dimension of physical space (e.g., 3-D) but the dimension of the input vector (such as the number of gridcells), which for practical problems can be in the tens [11] or even millions or more.

Sadly, this rules out the practical use of DNN for some purposes. Nonetheless, for many practical applications of deep learning, the approximation behaviors are not nearly so pessimistic as this would indicate (cp. [12]). For example, results are more optimistic:

- if the target function has a strong smoothness property;
- if the function is not arbitrary but is a composition of simpler functions;
- if the training and test data are restricted to a (possibly unknown) lower dimensional manifold in the high dimensional space (this is certainly the case for common image and language modeling tasks);
- if the average case behavior for the desired problem domain is much better than the worst case behavior addressed in the theorem;
- The theorem assumes multilayer perceptron and ReLU activation; other DNN architectures may perform better (though the analysis is based on multidimensional Taylor’s theorem, which one might conjecture applies also to other architectures).
- For many practical applications, very high accuracy is not a requirement.
- For some applications, only low 2-norm error is sufficient, (not low max-norm).
- For the special case of physics informed neural networks (PINNs), stronger results hold.

Thus, not all hope is lost from the standpoint of theory. However, certain problems for which high accuracy is required may not be suitable for DNN approximation.

Assuming your space of neural network candidates is expressive enough to closely *represent* the target function—how much training data is required to actually *find* a good approximation?

A result ^{2} is given in [13, Theorem 2.2] showing that the number of training samples required to train to high max-norm accuracy grows, again, *exponentially* in the dimension of the input to the function.

The authors concede however that “if additional problem information about [the target functions] can be incorporated into the learning problem it may be possible to overcome the barriers shown in this work.” One suspects that some of the caveats given above might also be relevant here. Additionally, if one considers 2-norm error instead of max-norm error, the data requirement grows polynomially rather than exponentially, making the training data requirement much more tractable. Nonetheless, for some problems the amount of data required is so large that attempts to “pin down” the DNN to sufficient accuracy become intractable.

The amount of training data may be sufficient to *specify* a suitable neural network. But, will standard methods for *finding* the weights of such a DNN be effective for solving this difficult nonconvex optimization problem?

A recent paper [14] from Max Tegmark’s group empirically studies DNN training to high accuracy. They find that as the input dimension grows, training to very high accuracy with standard stochastic gradient descent methods becomes difficult or impossible.

They also find second order methods perform much better, though these are more computationally expensive and have some difficulty also when the dimension is higher. Notably, second order methods have been used effectively for DNN training for some science applications [15]. Also, various alternative training approaches have been tried to attempt to stabilize training; see, e.g., [16].

Application of AI methods to scientific discovery continues to deliver amazing results, in spite of lagging theory. Ilya Sutskever has commented, “Progress in AI is a game of faith. The more faith you have, the more progress you can make” [17].

Theory of deep learning methods is in its infancy. The current findings show some cases for which use of DNN methods may not be fruitful, Continued discoveries in deep learning theory can help better guide how to use the methods effectively and inform where new algorithmic advances are needed.

^{1} Suppose the function to be approximated takes d inputs and has the smoothness property that all n^{th} partial derivatives are continuous (i.e., is in C^{n}(Ω) for compact Ω). Also suppose a multilayer perceptron with ReLU activation functions must be able to approximate any such function to max-norm no worse than ε. Then the number of weights required is at least a fixed constant times (1/ε)^{d/(2n)}.

^{2} Let F be the space of all functions that can be approximated exactly by a broad class of ReLU neural networks. Suppose there is a training method that can recover all these functions up to max-norm accuracy bounded by ε. Then the number of training samples required is at least a fixed constant times (1/ε)^{d}.

[1] “Integrated Research Infrastructure Architecture Blueprint Activity (Final Report 2023),” https://www.osti.gov/biblio/1984466.

[2] Joubert, Wayne, Bronson Messer, Philip C. Roth, Antigoni Georgiadou, Justin Lietz, Markus Eisenbach, and Junqi Yin. “Learning to Scale the Summit: AI for Science on a Leadership Supercomputer.” In *2022 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)*, pp. 1246-1255. IEEE, 2022, https://www.osti.gov/biblio/2076211.

[3] “Reproducibility Workshop: The Reproducibility Crisis in ML‑based Science,” Princeton University, July 28, 2022, https://sites.google.com/princeton.edu/rep-workshop.

[4] Wahlbin, L. B. (1978). Maximum norm error estimates in the finite element method with isoparametric quadratic elements and numerical integration. RAIRO. Analyse numérique, 12(2), 173-202, https://www.esaim-m2an.org/articles/m2an/pdf/1978/02/m2an1978120201731.pdf

[5] Kashiwabara, T., & Kemmochi, T. (2018). Maximum norm error estimates for the finite element approximation of parabolic problems on smooth domains. https://arxiv.org/abs/1805.01336.

[6] Adcock, Ben, and Nick Dexter. “The gap between theory and practice in function approximation with deep neural networks.” *SIAM Journal on Mathematics of Data Science* 3, no. 2 (2021): 624-655, https://epubs.siam.org/doi/10.1137/20M131309X.

[7] “Figure 5 from The gap between theory and practice in function approximation with deep neural networks | Semantic Scholar,” https://www.semanticscholar.org/paper/The-gap-between-theory-and-practice-in-function-Adcock-Dexter/156bbfc996985f6c65a51bc2f9522da2a1de1f5f/figure/4

[8] Gühring, I., Raslan, M., & Kutyniok, G. (2022). Expressivity of Deep Neural Networks. In P. Grohs & G. Kutyniok (Eds.), Mathematical Aspects of Deep Learning (pp. 149-199). Cambridge: Cambridge University Press. doi:10.1017/9781009025096.004, https://arxiv.org/abs/2007.04759.

[9] D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Netw., 94:103–114, 2017, https://arxiv.org/abs/1610.01145

[10] I. Gühring, G. Kutyniok, and P. Petersen. Error bounds for approximations with deep relu neural networks in W^{s,p} norms. Anal. Appl. (Singap.), pages 1–57, 2019, https://arxiv.org/abs/1902.07896

[11] Matt R. Norman, “The MiniWeather Mini App,” https://github.com/mrnorman/miniWeather

[12] Lin, H.W., Tegmark, M. & Rolnick, D. Why Does Deep and Cheap Learning Work So Well?. J Stat Phys 168, 1223–1247 (2017). https://doi.org/10.1007/s10955-017-1836-5

[13] Berner, J., Grohs, P., & Voigtlaender, F. (2022). Training ReLU networks to high uniform accuracy is intractable. ICLR 2023, https://openreview.net/forum?id=nchvKfvNeX0.

[14] Michaud, E. J., Liu, Z., & Tegmark, M. (2023). Precision machine learning. Entropy, 25(1), 175, https://www.mdpi.com/1099-4300/25/1/175.

[15] Markidis, S. (2021). The old and the new: Can physics-informed deep-learning replace traditional linear solvers?. *Frontiers in big Data*, *4*, 669097, https://www.frontiersin.org/articles/10.3389/fdata.2021.669097/full

[16] Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2006). Greedy layer-wise training of deep networks. Advances in neural information processing systems, 19, https://papers.nips.cc/paper_files/paper/2006/hash/5da713a690c067105aeb2fae32403405-Abstract.html

[17] “Chat with OpenAI CEO and and Co-founder Sam Altman, and Chief Scientist Ilya Sutskever,” https://www.youtube.com/watch?v=mC-0XqTAeMQ&t=250s

The post Hallucinations of AI Science Models first appeared on John D. Cook.]]>10! = 7! × 5! × 3! × 1!

Are there more examples like this?

What would you call the pattern on the right? I don’t think there’s a standard name, but here’s why I think it should be called double super factorial or super double factorial.

The factorial of a positive number *n* is the product of the positive numbers up to and including *n*. The super factorial of *n* is the product of the *factorials* of the positive numbers up to and including *n*. So, for example, 7 super factorial would be

7! × 6! × 5! × 4! × 3! × 2! × 1!

The double factorial of a positive number *n* is the product of all the positive numbers up to *n* with the same parity of *n*. So, for example, the double factorial of 7 would be

7!! = 7 × 5 × 3 × 1.

The pattern at the top of the post is like super factorial, but it only includes odd terms, so it’s like a cross between super factorial and double factorial, hence double super factorial.

Denote the double super factorial of *n* as dsf(*n*), the product of the factorials of all numbers up to *n* with the same parity as *n*. That is,

dsf(*n*) = *n*! × (*n* − 2)! × (*n* − 4)! × … × 1

where the 1 at the end is 1! if *n* is odd and 0! if *n* is even. In this notation, the observation at the top of the post is

10! = dsf(7).

We can see by re-arranging terms that a double super factorial is also a super double factorial. For example, look at

dsf(7) = 7! × 5! × 3! × 1!

If we separate out the first term in each factorial we have

(7 × 5 × 3 × 1)(6! × 4! × 2!) = 7!! dsf(6)

We can keep going and show in general that

dsf(*n*) = *n*!! × (*n *− 1)!! × (*n *− 2)!! … × 1

We could call the right hand side super double factorial, sdf(*n*). Just as a super factorial is a product of factorials, a super double factorials is a product of double factorials. Therefore

dsf(*n*) = sdf(*n*).

Are there more solutions to

*n*! = dsf(*m*).

besides *n* = 10 and *m* = 7? Yes, here are some.

0! = dsf(0)

1! = dsf(1)

2! = dsf(2)

3! = dsf(3)

6! = dsf(5)

There are no solutions to

*n*! = dsf(*m*)

if *n* > 10. Here’s a sketch of a proof.

Bertrand’s postulate says that for *n* > 1 there is always a prime *p* between *n* and 2*n*. Now *p* divides (2*n*)! but *p* cannot divide dsf(*n*) because dsf(*n*) only has factors less than or equal to *n*.

If we can show that for some *N*, *n* > *N* implies (2*n*)! < dsf(*n*) then there are no solutions to

*n*! = dsf(*m*)

for *n* > 2*N* because there is a prime *p* between *N* and 2*N* that divides the left side but not the right. In fact *N* = 12. We can show empirically there are no solutions for *n* = 11 up to 24, and the proof shows there are no solutions for *n* > 24.

The most interesting thing about Laguerre’s method is that it nearly always converges to a root, no matter where you start. Newton’s method, on the other hand, converges quickly **if** you start sufficiently close to a root. If you don’t start close enough to a root, the method might cycle or go off to infinity.

The first time I taught numerical analysis, the textbook we used began the section on Newton’s method with the following nursery rhyme:

There was a little girl,

Who had a little curl,

Right in the middle of her forehead.

When she was good,

She was very, very good,

But when she was bad, she was horrid.

When Newton’s method is good, it is very, very good, but when it is bad it is horrid.

Laguerre’s method is not well understood. Experiments show that it nearly always converges, but there’s not much theory to explain what’s going on.

The method is robust in that it is very likely to converge to **some** root, but it may not converge to the root you expected unless you start sufficiently close. The rest of the post illustrates this.

Let’s look at the polynomial

*p*(*x*) = 3 + 22*x* + 20*x*² + 24*x*³.

This polynomial has roots at 0.15392, and at -0.3397 ± 0.83468*i*.

We’ll generate 2,000 random starting points for Laguerre’s method and color its location according to which root it converges to. Points converging to the real root are colored blue, points converging to the root with positive imaginary part are colored orange, and points converging to the root with negative imaginary part are colored green.

Here’s what we get:

This is hard to see, but we can tell that there aren’t many blue dots, about an equal number of orange and green dots, and the latter are thoroughly mixed together. This means the method is unlikely to converge to the real root, and about equally likely to converge to either of the complex roots.

Let’s look at just the starting points that converge to the real root, its basin of attraction. To get more resolution, we’ll generate 100,000 starting points and make the dots smaller.

The convergence region is pinched near the root; you can start fairly near the root along the real axis but converge to one of the complex roots. Notice also that there are scattered points far from the real root that converge to that point.

Next let’s look at the points that converge to the complex root in the upper half plane.

Note that the basin of attraction appears to take up over half the area. But the corresponding basin of attraction for the root in the lower half plane *also* appears to take up over half the area.

They can’t both take up over half the area. In fact. both take up about 48%. But the two regions are very intertwined. Due to the width of the dots used in plotting, each green dot covers a tiny bit of area that belongs to orange, and vice versa. That is, the fact that both appear to take over half the area shows how commingled they are.

I’m not a lawyer, so this isn’t legal advice. Even the HHS, who coin the term “Breach Safe Harbor” in their guidance portal, weasels out of saying they’re giving legal guidance by saying “The contents of this database lack the force and effect of law, except as authorized by law …”

You can’t just say that data were encrypted before they were breached. Weak encryption won’t cut it. You have to use acceptable algorithms and procedures.

How can you know whether you’ve encrypted data well enough to be covered Breach Safe Harbor? HHS cites four NIST publications for further guidance. (Not that I’m giving legal advice. I’m merely citing the HHS, who also is not giving legal advice.)

Here are the four publications.

- NIST SP 800-111. Guide to Storage Encryption Technologies for End User Devices
- NIST SP 800-52. Guidelines for the Selection, Configuration, and Use of Transport Layer Security (TLS) Implementations
- NIST SP 800-113. Guide to SSL VPNs
- NIST Special Publication 800-88, Revision 1: Guidelines for Media Sanitization

At one point Tennessee law said a breach of encrypted data was still a breach. According to Dempsey and Carlin [1]

In 2016, Tennessee repealed its encryption safe harbor, requiring notice of breach of even encrypted data, but then in 2017, after criticism, the state restored a safe harbor for “information that has been encrypted in accordance with the current version of the Federal Information Processing Standard (FIPS) 140-2 if the encryption key has not been acquired by an unauthorized person.”

This is interesting for a couple reasons. First, there is a precedent for requiring notification of encrypted data. Second, this underscores the point above that encryption in general is **not** sufficient to avoid having to give notice of a breach: **standard-compliant** encryption is sufficient.

If you would like technical or statistical advice on how to prevent or prepare for a data breach, or how to respond after a data breach after the fact, we can help.

[1] Jim Dempsey and John P. Carlin. Cybersecurity Law Fundamentals, Second Edition.

The post Breach Safe Harbor first appeared on John D. Cook.]]>The example claims that

TEXTCOLLBYfGiJUETHQ4hAcKSMd5zYpgqf1YRDhkmxHkhPWptrkoyz28wnI9V0aHeAuaKnak

and

TEXTCOLLBYfGiJUETHQ4hEcKSMd5zYpgqf1YRDhkmxHkhPWptrkoyz28wnI9V0aHeAuaKnak

have the same hash value.

This raises several questions.

Are these two strings really different, and if so, how do they differ? If you stare at the strings long enough you can see that they do indeed differ by one character. But how could you compare long strings like this in a more automated way?

How could you compute the MD5 hash values of the strings to verify that they are the same?

The following Python code addresses the questions above.

from hashlib import md5 from difflib import ndiff def showdiff(a, b): for i,s in enumerate(ndiff(a, b)): if s[0]==' ': continue elif s[0]=='-': print(u'Delete "{}" from position {}'.format(s[-1],i)) elif s[0]=='+': print(u'Add "{}" to position {}'.format(s[-1],i)) a = "TEXTCOLLBYfGiJUETHQ4hAcKSMd5zYpgqf1YRDhkmxHkhPWptrkoyz28wnI9V0aHeAuaKnak" b = "TEXTCOLLBYfGiJUETHQ4hEcKSMd5zYpgqf1YRDhkmxHkhPWptrkoyz28wnI9V0aHeAuaKnak" showdiff(a, b) ahash = md5(a.encode('utf-8')).hexdigest() bhash = md5(b.encode('utf-8')).hexdigest() assert(ahash == bhash)

The basis of the `showdiff`

function was from an answer to a question on Stack Overflow.

The output of the call to `showdiff`

is as follows.

Delete "A" from position 21 Add "E" to position 22

This means we can form string `b`

from `a`

by changing the ‘A’ in position 21 to an ‘E’. There was only one difference between the two strings in this example, but `showdiff`

could be useful for understanding more complex differences.

The assert statement passes because both strings hash to faad49866e9498fc1719f5289e7a0269.

Early in the book is the example of finding the distance from a point **q** to a line of the form **p** + *t***v**.

If you define **u** = **q** − **p** then a straightforward derivation shows that the distance *d* from **q** to the line is given by

But as the author explains, it is better to calculate *d* by

Why is that? The two expressions are algebraically equal, but the latter is better suited for numerical calculation.

The cardinal rule of numerical calculation is to avoid subtracting nearly equal floating point numbers. If two numbers agree to *b* bits, you may lose up to *b* bits of significance when computing their difference.

If **u** and **v** are vectors with large magnitude, but **q** is close to the line, then the first equation subtracts two large, nearly equal numbers under the square root.

The second equation involves subtraction too, but it’s less obvious. This is a common theme in numerical computing. Imagine this dialog.

[Student produces first equation.]

Mentor: Avoid subtracting nearly equal numbers.

[Student produces section equation.]

Student: OK, did it.

Mentor: That’s much better, though it could still have problems.

Where is there a subtraction in the second equation? We started with a subtraction in defining **u**. More subtly, the definition of cross product involves subtractions. But these subtractions involve smaller numbers than the first formula, because the first formula subtracts squared values. Eric Lengyel points this out in his book.

None of this may matter in practice, until it does matter, which is a common pattern in numerical computing. You implement something like the first formula, something that can be derived directly. You implicitly have in mind vectors whose magnitude is comparable to *d* and this guides your choice of unit tests, which all pass.

Some time goes by and a colleague tells you your code is failing. Impossible! You checked your derivation by hand and in Mathematica. Your unit tests all pass. Must be your colleague’s fault. But it’s not. Your code would be correct in infinite precision, but in an actual computer it fails on inputs that violate your implicit assumptions.

This can be frustrating, but it can also be fun. Implementing equations from a freshman textbook accurately, efficiently, and robustly is *not* a freshman-level exercise.

Lately I’ve been helping a colleague to add worker threads to his GUI-based Windows application.

Thread programming can be tricky. Here are a few things I’ve learned along the way.

**Performance**. This app does compute-intensive work. It is helpful to offload this very compute-heavy work to a worker thread. Doing this frees the main thread to service GUI requests better.

**Thread libraries**. Windows has multiple thread libraries, for example Microsoft Foundation Class library threads and C++ standard library threads. It is hazardous to use different thread libraries in the same app. In the extreme case, different thread libraries, such as GOMP vs. LOMP, used in resp. the GCC and LLVM compiler families, have different threading runtimes which keep track of threads in different ways. Mixing them in the same code can cause hazardous silent errors.

**Memory fences** are a thing. Different threads can run on different processor cores and hold variables in different respective L1 caches that are not flushed (this to maintain high performance). An update to a variable by one thread is not guaranteed to be visible to other threads without special measures. For example, one could safely transfer information using `::PostMessage`

coupled with a handler function on the receiver thread. Or one could send a signal using an MFC `CEvent`

on one thread and read its `Lock`

on the other. Also, a thread launch implicitly does a memory fence, so that, at least then, the new thread is guaranteed to correctly see the state of all memory locations.

**GUI access** should be done from the master thread only, not a worker thread. Doing so can result in deadlock. A worker thread can instead `::PostMessage`

to ask the master thread to do a GUI action.

**Thread launch.** By default `AfxBeginThread`

returns a thread handle which MFC takes care of deleting when no longer needed. If you want to manage the life cycle of the handle yourself, you can do something like:

```
myWorkerThreadHandle = AfxBeginThread(myFunc, myParams,
THREAD_PRIORITY_NORMAL, 0, CREATE_SUSPENDED);
myWorkerThreadHandle->m_bAutoDelete = false;
myWorkerThreadHandle->ResumeThread();
```

**Joint use of a shared library** like the DAO database library has hazards. One should beware of using the library to allocate something in one thread and deallocating in another, as this will likely allocate in a thread-local heap or stack instead of a shared thread-safe heap, this resulting in a crash.

**Initialization**. One should call `CoInitializeEx(NULL, COINIT_APARTMENTTHREADED)`

and `AfxDaoInit()`

(if using DAO) at thread initialization on both master and worker threads, and correspondingly `CoUninitialize()`

and `AfxDaoTerm()`

at completion of the thread.

**Monitoring of thread state** can be done with

`WaitForSingleObject(myWorkerThreadHandle->m_hThread, 0)`

to determine if the thread has completed or `WaitForSingleObject(myWorkerThreadHandle->m_hThread, INFINITE)`

for a blocking wait until completion.

**Race conditions** are always a risk but can be avoided by careful reasoning about execution. Someone once said up to 90% of code errors can be found by desk checking [1]. Race conditions are notoriously hard to debug, partly because they can occur nondeterministically. There are tools for trying to find race condition errors, though I’ve never tried them.

So far I find no rigorous specification of the MFC threading model online that touches on all these concerns. Hopefully this post is useful to someone else working through these issues.

[1] Dasso, Aristides., Funes, Ana. Verification, Validation and Testing in Software Engineering. United Kingdom: Idea Group Pub., 2007, p. 163.

The post Experiences with Thread Programming in Microsoft Windows first appeared on John D. Cook.]]>In 1667 James Gregory came up with a more efficient way to carry out Archimedes’ approach. Tom Edgar and David Richeson cite Gregory’s theorem in [1] as follows.

Let

I_{k}andC_{k}denote the areas of regulark-gons inscribed in and circumscribed around a given circle. ThenI_{2n}is the geometric mean ofI_{n}andC_{k}, andC_{2n}is the harmonic mean ofI_{2n}andC_{n}; that is,

We can start with *n* = 4. Obviously the square circumscribed around the unit circle has vertices at (±1, ±1) and area 4. A little thought finds that the inscribed square has vertices at (±√2/2, ±√2/2) and area 2.

The following Python script finds π to six decimal places.

n = 4 I, C = 2, 4 # inscribed area, circumscribed area delta = 2 while delta > 1e-6: I = (I*C)**0.5 C = 2*C*I / (C + I) delta = C - I n *= 2 print(n, I, C, delta)

The script stops when *n* = 8192, meaning that the difference between the areas of an inscribed and circumscribed 8192-gon is less than 10^{−6}. The final output of the script is.

8192 3.1415923 3.1415928 4.620295e-07

If we average the upper and lower bound we get one more decimal place of accuracy.

Gregory’s algorithm isn’t fast, though it’s much faster than carrying out Archimedes’ approach by computing each area from scratch.

Gregory gives an interesting recursion. It has an asymmetry that surprised me: you update *I*, then *C*. You don’t update them simultaneously as you do when you’re computing the arithmetic-geometric mean.

[1] Tom Edgar and David Richeson. A Visual Proof of Gregory’s Theorem. Mathematics Magazine, December 2019, Vol. 92, No. 5 (December 2019), pp. 384–386

The post Accelerating Archimedes first appeared on John D. Cook.]]>If the cable were pulled perfectly taut, we would have *s* = *x* and there would be no sag. But in general, *s* > *x* and the cable is somewhat lower in the middle than on the two ends. The sag is the difference between the height at the end points and the height in the middle.

The equation of a catenary is

*y* = *a* cosh(*t*/*a*)

and the length of the catenary between *t* = −*x* and *t* = *x* is

2*s* = 2*a* sinh(*x*/*a*).

You could solve this equation for *a* and the sag will be

*g* = *a* cosh(*x*/*a*) − *a.*

Solving for *a* given *s* is not an insurmountable problem, but there is a simple approximation for *g* that has an error of less than 1%. This approximation, given in [1], is

*g*² = (*s* − *x*)(*s* + *x*/2).

To visualize the error, I set *x* = 1 and let *a* vary to produce the plot below.

The relative error is always less than 1/117. It peaks somewhere around *x* = 1/3 and decreases from there, the error getting smaller as *a* increases (i.e. the cable is pulled tighter).

[1] J. S. Frame. An approximation for the dip of a catenary. Pi Mu Epsilon Journal, Vol. 1, No. 2 (April 1950), pp. 44–46

The post How much will a cable sag? A simple approximation first appeared on John D. Cook.]]>Is the pattern of letters in *Mississippi* literally unique or just uncommon? What is the shortest word with a unique letter pattern? The longest word?

We can answer these questions by looking at **normalized cryptograms**, a sort of word signature. These are formed by replacing the first letter in a word with ‘A’, the next unique letter with ‘B’, etc. The normalized cryptogram of *Mississippi* is ABCCBCCBDDB.

The set of English words is fuzzy, but for my purposes I will take “words” to mean the entries in the dictionary file `american-english`

on my Linux box, removing words that contain apostrophes or triple letters. I computed the cryptogram of each word, then looked for those that only appear once.

Relative to the list of words I used, yes, *Mississippi* is unique.

The shortest word with a unique cryptogram is *eerie*. [1]

The longest word with a unique cryptogram is *ambidextrously*. Every letter in this 14-letter word appears only once.

[1] Update: *eerie* is a five-letter example, but there are more. Jack Kennedy pointed out *amass*, *llama*, and *mamma* in the comments. I noticed *eerie* because its cryptogram comes first in alphabetical order.

64 million scientific papers have been published since 1996 [1].

Assuming you can actually *find* the information you want in the first place—how can you organize your findings to be able to recall and use them later?

It’s not a trifling question. Discoveries often come from uniting different obscure pieces of information in a new way, possibly from very disparate sources.

Many software tools are used today for notetaking and organizing information, including simple text files and folders, Evernote, GitHub, wikis, Miro, mymind, Synthical and Notion—to name a diverse few.

AI tools can help, though they can’t always recall correctly and get it right, and their ability to find connections between ideas is elementary. But they are getting better [2,3].

One perspective was presented by Jared O’Neal of Argonne National Laboratory, from the standpoint of laboratory notebooks used by teams of experimental scientists [4]. His experience was that as problems become more complex and larger, researchers must invent new tools and processes to cope with the complexity—thus “reinventing the lab notebook.”

While acknowledging the value of paper notebooks, he found electronic methods essential because of distributed teammates. In his view many streams of notes are probably necessary, using tools such as GitLab and Jupyter notebooks. Crucial is the actual discipline and methodology of notetaking, for example a hierarchical organization of notes (separating high-level overview and low-level details) that are carefully written to be understandable to others.

A totally different case is the research methodology of 19th century scientist Michael Faraday. He is not to be taken lightly, being called by some “the best experimentalist in the history of science” (and so, perhaps, even compared to today) [5].

A fascinating paper [6] documents Faraday’s development of “a highly structured set of retrieval strategies as dynamic aids during his scientific research.” He recorded a staggering 30,000 experiments over his lifetime. He used 12 different kinds of record-keeping media, including lab notebooks proper, idea books, loose slips, retrieval sheets and work sheets. Often he would combine ideas from different slips of paper to organize his discoveries. Notably, his process to some degree varied over his lifetime.

Certain motifs emerge from these examples: the value of well-organized notes as memory aids; the need to thoughtfully innovate one’s notetaking methods to find what works best; the freedom to use multiple media, not restricted to a single notetaking tool or format.

Do you have a favorite method for organizing your research? If so, please share in the comments below.

[1] How Many Journal Articles Have Been Published? https://publishingstate.com/how-many-journal-articles-have-been-published/2023/

[2] “Multimodal prompting with a 44-minute movie | Gemini 1.5 Pro Demo,” https://www.youtube.com/watch?v=wa0MT8OwHuk

[3] Geoffrey Hinton, “CBMM10 Panel: Research on Intelligence in the Age of AI,” https://www.youtube.com/watch?v=Gg-w_n9NJIE&t=4706s

[4] Jared O’Neal, “Lab Notebooks For Computational Mathematics, Sciences, Engineering: One Ex-experimentalist’s Perspective,” Dec. 14, 2022, https://www.exascaleproject.org/event/labnotebooks/

[5] “Michael Faraday,” https://dlab.epfl.ch/wikispeedia/wpcd/wp/m/Michael_Faraday.htm

[6] Tweney, R.D. and Ayala, C.D., 2015. Memory and the construction of scientific meaning: Michael Faraday’s use of notebooks and records. *Memory Studies*, *8*(4), pp.422-439. https://www.researchgate.net/profile/Ryan-Tweney/publication/279216243_Memory_and_the_construction_of_scientific_meaning_Michael_Faraday’s_use_of_notebooks_and_records/links/5783aac708ae3f355b4a1ca5/Memory-and-the-construction-of-scientific-meaning-Michael-Faradays-use-of-notebooks-and-records.pdf

Warren Weaver [1] introduced what he called the surprise index to quantify how surprising an event is. At first it might seem that the probability of an event is enough for this purpose: the lower the probability of an event, the more surprise when it occurs. But Weaver’s notion is more subtle than this.

Let *X* be a discrete random variable taking non-negative integer values such that

Then the surprise index of the *i*th event is defined as

Note that if *X* takes on values 0, 1, 2, … *N*−1 all with equal probability 1/*N*, then *S*_{i} = 1, independent of *N*. If *N* is very large, each outcome is rare but not surprising: because all events are equally rare, no specific event is surprising.

Now let *X* be the number of legs a human selected at random has. Then *p*_{2} ≈ 1, and so the numerator in the definition of *S*_{i} is approximately 1 and *S*_{2} is approximately 1, but *S*_{i} is large for any value of *i* ≠ 2.

The hard part of calculating the surprise index is computing the sum in the numerator. This is the same calculation that occurs in many contexts: Friedman’s index of coincidence, collision entropy in physics, Renyi entropy in information theory, etc.

Weaver comments that he tried calculating his surprise index for Poisson and binomial random variables and had to admit defeat. As he colorfully says in a footnote:

I have spent a few hours trying to discover that someone else had summed these series and spent substantially more trying to do it myself; I can only report failure, and a conviction that it is a dreadfully sticky mess.

A few years later, however, R. M. Redheffer [2] was able to solve the Poisson case. His derivation is extremely terse. Redheffer starts with the generating function for the Poisson

and then says

Let

x=e^{iθ}; thene^{−iθ}; multiply; integrate from 0 to 2π and simplify slightly to obtainThe integral on the right is recognized as the zero-order Bessel function …

Redheffer then “recognizes” an expression involving a Bessel function. Redheffer acknowledges in a footnote at a colleague M. V. Cerrillo was responsible for recognizing the Bessel function.

It is surprising that the problem Weaver describes as a “dreadfully sticky mess” has a simple solution. It is also surprising that a Bessel function would pop up in this context. Bessel functions arise frequently in solving differential equations but not that often in probability and statistics.

When Redheffer says “Let *x* = *e*^{iθ}; then *e*^{−iθ}; multiply; integrate from 0 to 2π” he means that we should evaluate both sides of the equation for the Poisson generating function equation at these two values of *x*, multiply the results, and average the both sides over the interval [0, 2π].

On the right hand side this means calculating

This reduces to

because

i.e. the integral evaluates to 1 when *m* = *n* but otherwise equals zero.

On the left hand side we have

Cerrillo’s contribution was to recognize the integral as the Bessel function *J*_{0} evaluated at -2*i*λ or equivalently the modified Bessel function *I*_{0} evaluated at -2λ. This follows directly from equations 9.1.18 and 9.6.16 in Abramowitz and Stegun.

Putting it all together we have

Using the asymptotic properties of *I*_{0} Redheffer notes that for large values of λ,

[1] Warren Weaver, “Probability, rarity, interest, and surprise,” The Scientific Monthly, Vol 67 (1948), p. 390.

[2] R. M. Redheffer. A Note on the Surprise Index. The Annals of Mathematical Statistics, Mar., 1951, Vol. 22, No. 1 pp. 128ndash;130.

The post A surprising result about surprise index first appeared on John D. Cook.]]>Brainerd [1] suggested the following estimator based on a Markov chain model of language. The estimated vocabulary is the number *N* satisfying the equation

The left side is a decreasing function of *N*, so you could solve the equation by finding a values of *N* that make the sum smaller and larger than *n*, then use a bisection algorithm.

We can see that the model is qualitatively reasonable. If every word is unique, i.e. *x* = *n*, then the solution is *N* = ∞. If you haven’t seen any repetition, you the author could keep writing new words indefinitely. As the amount of repetition increases, the estimate of *N* decreases.

Brainerd’s model is simple, but it tends to underestimate vocabulary. More complicated models might do a better job.

Problems analogous to estimating vocabulary size come up in other applications. For example, an ecologist might want to estimate the number of new species left to be found based on the number of species seen so far. In my work in data privacy I occasionally have to estimate diversity in a population based on diversity in a sample. Both of these examples are analogous to estimating potential new words based on the words you’ve seen.

[1] Brainerd, B. On the relation between types and tokes in literary text, J. Appl. Prob. 9, pp. 507-5

The post Estimating an author’s vocabulary first appeared on John D. Cook.]]>Simple substitution ciphers can be broken by frequency analysis: the most common letter probably corresponds to E, the next most common letter probably corresponds to T, etc. But that’s only for English prose. Maybe the message was composed in French. Or maybe it was composed in Japanese, then transliterated into the Latin alphabet so it could be transmitted via Morse code. You’d like to know what language the message was written in before you try identifying letters via their frequency.

William Friedman’s idea was to compute a statistic, what he dubbed the index of coincidence, to infer the probable language of the source. Since this statistic only depends on symbol frequencies, it gives the same value whether computed on clear text or text encrypted with simple substitution. It also gives the same value if the text has been run through a transposition cipher as well.

(Classical cryptanalysis techniques, such as computing the index of coincidence, are completely useless against modern cryptography. And yet ideas from classical cryptanalysis are still useful for other applications. Here’s an example that came up in a consulting project recently.)

As I mentioned at the top of the post, you’d try breaking the simplest encryption first. If the index of coincidence is lower than you’d expect for a natural language, you might suspect that the message has been encrypted using polyalphabetic substitution. That is, instead of using one substitution alphabet for every letter, maybe the message has been encrypted using a cycle of *n* different alphabets, such as the Vigenère cypher.

How would you break such a cipher? First, you’d like to know what *n* is. How would you do that? By trial and error. Try splitting the text into groups of letters according to their position mod *n*, then compute the index of coincidence again for each group. If the index statistics are much larger when *n* = 7, you’re probably looking a message encrypted with a key of length 7.

The source language would still leave its signature. If the message was encrypted by cycling through seven scrambled alphabets, each group of seven letters would most likely have the statistical distribution of the language used in the clear text.

Friedman’s index of coincidence, published in 1922, was one statistic that could be computed based on letter frequencies, one that worked well in practice, but you could try other statistics, and presumably people did. The index of coincidence is essentially Rényi entropy with parameter α = 2. You might try different values of α.

If the approach above doesn’t work, you might suspect that the text was not encrypted one letter at a time, even using multiple alphabets. Maybe pairs of letters were encrypted, as in the Playfair cipher. You could test this hypothesis by looking that the frequencies of pairs of letters in the encrypted text, calculating an index of coincidence (or some other statistic) based on pairs of letters.

Here again letter pair frequencies may suggest the original language. It might not distinguish Spanish from Portuguese, for example, but it would distinguish Japanese written in Roman letters from English.

The post Detecting the language of encrypted text first appeared on John D. Cook.]]>The more you can know about the solution to a differential equation before you attempt to solve it numerically the better. At a minimum, you’d like to know whether there even is a solution before you compute it. Unfortunately, a lot of theorems along these lines are local in nature: the theorem assures you that a solution exists in *some* interval, but doesn’t say how big that interval might be.

Here’s a nice theorem from [1] that tells you that a solution is going to blow up in finite time, and it even tells you what that time is.

The initial value problem

*y*′ = *g*(*y*)

with *y*(0) = *y*_{0} with *g*(*y*) > 0 blows up at *T* if and only if the integral

converges to *T*.

Note that it is not necessary to first find a solution then see whether the solution blows up.

Note also that an upper (or lower) bound on the integral gives you an upper (or lower) bound on *T*. So the theorem is still useful if the integral is hard to evaluate.

This theorem applies only to autonomous differential equations, i.e. the right hand side of the equation depends only on the solution *y* and not on the solution’s argument *t*. The differential equation alluded to at the top of the post is not autonomous, and so the theorem above does not apply. There are non-autonomous extensions of the theorem presented here (see, for example, [2]) but I do not know of a theorem that would cover the differential equation presented here.

[1] Duff Campbell and Jared Williams. Exloring finite-time blow-up. Pi Mu Epsilon Journal, Spring 2003, Vol. 11, No. 8 (Spring 2003), pp. 423–428

[2] Jacob Hines. Exploring finite-time blow-up of separable differential equations. Pi Mu Epsilon Journal, Vol. 14, No. 9 (Fall 2018), pp. 565–572

The post Blow up in finite time first appeared on John D. Cook.]]>The general pattern of widgets and subwidgets is that a widget is a set with some kind of structure, and a subwidget is a subset that has the same structure. This applies to vector spaces and subspaces, manifolds and submanifolds, lattices and sublattices, etc. Once you know the definition of a group, you can guess the definition of a subgroup.

But the definition of a normal subgroup is not something anyone would guess immediately after learning the definition of a group. The definition is not difficult, but its motivation isn’t obvious.

A subgroup *H* of a group *G* is a *normal* subgroup if for every *g* ∈ *G*,

*g*^{−1}*H**g* = *H*.

That is, if *h* is an element of *H*, *g ^{−1}hg* is also an element of

There’s an equivalent definition of normal subgroup that I only ran across recently in a paper by Francis Masat [1]. A subgroup *H* of a group *G* is normal if for every pair of elements *a* and *b* such that *ab* is in *H*, *ba* is also in *H*. With this definition it’s obvious that every subgroup of an Abelian group is normal because *ab* = *ba* for any *a* and *b*.

It’s an easy exercise to show that Masat’s definition is equivalent to the usual definition. Masat’s definition seems a little more motivated. It’s requiring some vestige of commutativity. It says that a subgroup *H* of a non-Abelian group *G* has some structure in common with subgroups of normal groups if this weak replacement for commutativity holds.

Category theory has a way of defining subobjects in general that basically formalizes the notion of widgets and subwidgets above. It also has a way of formalizing normal subobjects, but this is more recent and more complicated.

The nLab page on normal subobjects says “The notion was found relatively late.” The page was last edited in 2016 and says it is “to be finished later.” Given how exhaustively thorough nLab is on common and even not-so-common topics, this implies that the idea of normal subobjects is not mainstream.

I found a recent paper that discusses normal subobjects [2] and suffice it to say it’s complicated. This suggests that although analogs of subgroups are common across mathematics, the idea of a normal subgroup is more or less unique to group theory.

- The relation “normal subgroup of” is not transitive
- Normal and non-normal subgroups
- Analogy between prime numbers and simple groups

[1] Francis E. Masat. A Useful Characterization of a Normal Subgroup. Mathematics Magazine, May, 1979, Vol. 52, No. 3, pp. 171–173

[2] Dominique Bourn and Giuseppe Metere. A note on the categorical notions of normal subobject and of equivalence class. Theory and Applications of Categories, Vol 36, No. 3, 2021, pp. 65–101.

The post Normal subgroups are subtle first appeared on John D. Cook.]]>