A linear combination of sines and cosines

*a* sin(*x*) + *b* cos(*x*)

can be written as a sine with a phase shift

*A* sin(*x* + φ).

Going between {*a*, *b*} and {*A*, φ} is the calculation I’d like to save. For completeness I also include the case

*A* cos(*x* + ψ).

Define

*f*(*x*) = *a* sin(*x*) + *b* cos(*x*)

and

*g*(*x*) = *A* sin(*x* + φ).

Both functions satisfy the differential equation

*y*″ + *y* = 0

and so *f* = *g* if and only if *f*(0) = *g*(0) and *f′*(0) = *g′*(0).

Setting the values at 0 equal implies

*b *= *A* sin(φ)

and setting the derivatives at 0 equal implies

*a* = *A* cos(φ).

Taking the ratio of these two equations shows

*b*/*a* = tan(φ)

and adding the squares of both equations shows

*a*² + *b*² = *A*².

First we consider the case

*a* sin(*x*) + *b* cos(*x*) = *A* sin(*x* + φ).

If *a* and *b* are given,

A = √(*a*² + *b*²)

and

φ = tan^{−1}(*b* / *a*).

If *A* and φ are given,

*a* = *A* cos(φ)

and

*b* = *A* sin(φ)

from the previous section.

Now suppose we want

*a* sin(*x*) + *b* cos(*x*) = *A* cos(*x* + ψ)

If *a* and *b* are given, then

A = √(*a*² + *b*²)

as before and

ψ = − tan^{−1}(*a* / *b*).

If *A* and ψ are given then

*a* = − *A* sin(ψ)

and

*b* = *A* cos(ψ).

find . -name '*.py' -type f -print0 | grep -i "$1" find . -name '*.py' -type f -print0 | xargs -0 grep -il "$1"

This works well for searching file *contents* but behaves unexpectedly when searching for file *names*.

If I have a file named `frodo.py`

in the directory, the script will return

grep: (standard input): binary file matches

Binary file matches?! I wasn’t searching binary files. I was searching files with names consisting entirely of ASCII characters. Where is a binary file coming from?

If we cut off the pipe at the end of the first line of the script and run

find . -name '*.py' -type f -print0

we get something like

.elwing.py/.frodo.py/gandalf.py

with no apparent non-ASCII characters. But if we pipe the output through `xxd`

to see a hex dump, we see that there are invisible null characters after each file name.

One way to fix our script would be to add a `-a`

option to the call to `grep`

, telling to treat the input as ASCII. But this will return the same output as above. The output of `find`

is treated as one long (ASCII) string, which matches the regular expression.

Another possibility would be to add a `-o`

flag to direct `grep`

to return just the match. But this is less than ideal as well. If you were looking for file names containing a Q, for example, you’d get `Q`

as your output, which doesn’t tell you the full file name.

There may be better solutions [1], but my solution was to insert a call to `strings`

in the pipeline:

find . -name '*.py' -type f -print0 | strings | grep -i "$1"

This will extract the ASCII strings out of the input it receives, which has the effect of splitting the string of file names into individual names.

By default the `strings`

command defines an ASCII string to be a string of 4 or more consecutive ASCII characters. A file with anything before the `.py`

extension will necessarily have at least four characters, but the analogous script to search C source files would overlook a file named `x.c`

. You could fix this by using `strings -n 3`

to find sequences of three or more ASCII characters.

If you don’t have the `strings`

command installed, you could use `sed`

to replace the null characters with newlines.

find . -name '*.py' -type f -print0 | sed 's/\x0/\n/g' | grep -i "$1"

Note that the null character is denoted `\x0`

rather than simply `\0`

.

[1] See the comments for better solutions. I really appreciate your feedback. I’ve learned a lot over the years from reader comments.

The post Resolving a mysterious problem with find first appeared on John D. Cook.]]>*ax* + *by* = *N*.

I initially missed the constraint that *x* and *y* must be positive, in which result is well known (Bézout’s lemma) and there’s no requirement for *N* to be large. The positivity constraint makes things more interesting.

The problem is called the Postage Stamp Problem because it says that given any two stamps whose values are relatively prime, say a 5¢ stamp and a 21¢ stamp, you can make any sufficiently large amount of postage using just those two stamps.

A natural question is how large is “sufficiently large,” and the answer turns out to be all integers larger than

*ab* − *a* − *b*.

So in our example, you cannot make 79¢ postage out of 5¢ and 21¢ stamps, but you can make 80¢ or any higher amount.

If you’ve been reading this blog for a while, you may recognize this as a special case of the Chicken McNugget problem, which you can think of as the Postage Stamp problem with possibly more than two stamps.

There are two things in particular that were common knowledge at the time that I would be conspicuously ignorant of: interpolation tricks and geometry.

People from previous eras knew interpolation at a deeper level than citing the Lagrange interpolation theorem, out of necessity. They learned time-saving tricks have since been forgotten.

The biggest gap in my knowledge would be geometry. Mathematicians a century ago had a far deeper knowledge of geometry, particularly synthetic geometry, i.e. geometry in the style of Euclid rather than in the style of Descartes.

Sometimes older math books use notation or terminology that has since changed. I imagine I’d make a few gaffs, not immediately understanding a basic term or using a term that wasn’t coined until later.

If I had to teach a class, I’d choose something like real and complex analysis. Whittaker & Watson’s book on the subject was first published in 1902 and remains a common reference today. The only thing I find jarring about that book is that “show” is spelled “shew.” Makes me think of Ed Sullivan. But I think I’d have a harder time teaching a less advanced class.

Copernicus’ model was simpler, but it was less accurate. The increasingly complex models before Copernicus were refinements. They were not ad hoc, nor were they *unnecessarily* complex, if you must center your coordinate system on Earth.

It’s easy to draw the wrong conclusion from Copernicus, and from any number of other scientists who were able to greatly simplify a previous model. One could be led to believe that whenever something is too complicated, there must be a simpler approach. Sometimes there is, and sometimes there isn’t.

If there isn’t a simpler model, the time spent searching for one is wasted. If there *is* a simpler model, the time searching for one might still be wasted. Pursuing brute force progress might lead to a simpler model faster than pursuing a simpler model directly.

It all depends. Of course it’s wise to spend at least some time looking for a simple solution. But I think we’re fed too many stories in which the hero comes up with a simpler solution by stepping back from the problem.

Most progress comes from the kind of incremental grind that doesn’t make an inspiring story for children. And when there is a drastic simplification, that simplification usually comes *after* grinding on a problem, not *instead* of grinding on it.

3Blue1Brown touches on this in this video. The video follows two hypothetical problem solvers, Alice and Bob, who attack the same problem. Alice is the clever thinker and Bob is the calculating drudge. Alice’s solution of the original problem is certainly more elegant, and more likely to be taught in a classroom. But Bob’s approach generalizes in a way that Alice’s approach, as far as we know, does not.

One way to approach this problem would be to set up a system of equations for the coefficients of the sines and cosines. If you have *N* data points, you will get a system of *N* equations in *N* unknowns. The system will have a unique solution, though this is not obvious *a priori*.

Another approach would be to use the discrete Fourier transform (DFT). This is the approach that would commonly be used in practice. It’s even further from obvious *a priori* that this would work, but it does. (The DFT is so often computed using the FFT algorithm that the transform is often referred to by the algorithm name. If you’d like, mentally substitute FFT for DFT in the rest of the post.)

There are multiple ways to motivate the DFT, and the way suggested by the name is to derive the DFT as a discrete approximation to the (continuous) Fourier transform. Why should should a discrete approximation to an integral transform also solve an interpolation problem? This doesn’t sound inevitable, or even plausible, but it is the case.

Another way to motivate the DFT is as the least-squares solution to fitting a sum of sines and cosines to a set of points. Since this is phrased as an optimization problem rather than an interpolation problem, it is clear that it will have a solution. However, it is not clear that the error in the optimal fit will in fact be zero. Furthermore, the equation for the coefficients in the solution is the same as the equation for the DFT. You can find a derivation in [1].

Let’s take the vector [3, 1, 4, 1, 5, 9] and find trig functions that pass through these points. We can use the FFT as implemented in Python’s SciPy library to find a set of complex exponentials that pass through the points.

from scipy.fft import fft from numpy import exp, array, pi, round x = array([3, 1, 4, 1, 5, 9]) y = fft(x) N = len(x) z = [sum([exp(2j*pi*k*n/N)*y[k] for k in range(N)])/N for n in range(N)]

Aside from rounding errors on the order of 10^{−15} the vector `z`

equals the vector `x`

.

Turning the expression for `z`

above into a mathematical expression, we have

*f*(*z*) = *y*_{0} + *y*_{1} exp(*n*π*i*/3) + *y*_{2} exp(2*n*π*i*/3) + *y*_{3} exp(*n*π*i*) + *y*_{4} exp(4*n*π*i*/3) + *y*_{5} exp(5*n*π*i*/3)

where the *y*‘s come from the FFT above.

To find sines and cosines we need to use Euler’s formula

exp(*i*θ) = cos(θ) + *i* sin(θ)

Because started with real data `x`

, there will be symmetries in the FFT components `x`

that simplify the reduction of the complex function *f* to a real-valued function *g* using sines and cosines; some of the components will be conjugate and so the complex parts cancel out.

6* g*(*x*) = *y*_{0} + (*y*_{1} + *y*_{5}) cos(π*x*/3) + (*y*_{2} + *y*_{4}) cos(2π*x*/3) + *y*_{3} cos(π*x*)

+ *i* (*y*_{1} − *y*_{5}) sin(π*x*/3) + (*y*_{2} − *y*_{4}) cos(2π*x*/3)

and so

*g*(*x*) = 3.833 + 0.8333 cos(π*x*/3) − 1.833 cos(2π*x*/3) + 0.1666 cos(π*x*)

− 2.5981 sin(π*x*/3) − 2.0207 cos(2π*x*/3)

Here’s a plot that verifies that *g*(*x*) passes through the specified points.

[1] William L. Briggs and Van Emden Henson. The DFT: An Owner’s Manual for the Discrete Fourier Transform. SIAM 1995.

The post Trigonometric interpolation first appeared on John D. Cook.]]>Let *f*(*t*) be a function on [0, ∞) and *F*(*s*) be the Laplace transform of *f*(*t*).

Then the *n*th moment of *f*,

is equal to then *n*th derivative of *F*, evaluated at 0, with an alternating sign:

To see this, differentiate with respect to *s* inside the integral defining the Laplace transform. Each time you differentiate you pick up a factor of −*t*, so differentiating *n* times you pick up a term (−1)^{n} *t*^{n}, and evaluating at *s* = 0 makes the exponential term go away.

It’s fascinating that there’s such a thing as the World Jigsaw Puzzle Championship. The winning team of the two-person thousand-piece puzzle round can assemble a Ravensburger puzzle in less than an hour—that’s about 3 -1/2 seconds per piece.

It makes you wonder, how could you measure the hardness of a jigsaw puzzle? And what would a “hardest puzzle” look like?

You can find some puzzles that are claimed to be “hardest.” A first step to creating a hard puzzle is removing the front image entirely, eliminating visual cues. Some puzzles do this. This can be partly simulated with a normal puzzle by putting it together with the image side face down.

But you can do more: make every piece a square exactly the same size, and the “tab“ and “blank“ on each of the four sides of each square exactly the same size, shape and alignment. One can see that this gives you 2^{4} = 16 different unique kinds of puzzle piece, however, due to symmetries you actually have six different kinds of puzzle piece. Now, you are left with a bare minimum of shape cues to help you assemble the puzzle.

For the sake of the argument, one can ignore the “edge” pieces. One can see this from the fact that for a rectangular puzzle, as you scale up the size, the number of edge pieces proportional to the interior pieces of the puzzle becomes smaller and smaller. For example for a 36X28=1008 piece puzzle, there are (2*36+2*28-4)=124 edge pieces, about 12 percent of the whole. The ratio shrinks as you scale up puzzle size. And there is such a thing as a 42,000-piece jigsaw puzzle.

The solution complexity class for assembling such a puzzle is known to be NP-complete. This means that the best known algorithms require compute time that grows exponentially as the number of pieces is increased.

Of course there are simpler special cases. Consider a checkerboard configuration, in which “black square“ location pieces have four tabs and “red square” locations have four blanks, thus only two different kinds of piece. A competent puzzler could assemble this one piece per second, 17 minutes for a 1000-piece puzzle.

However, as far as we know, the number of “special cases” like this is vanishing small for large puzzles. An intuition for this can be taken from Kolmogorov complexity theory, using a counting argument to show that only very few long strings can be compressed to a short program.

Most cases are really hard, for example, constructed by making a totally random assignment for selecting the configuration of each piece-to-piece boundary. Solving this is is really hard because you may have to backtrack: you may have the puzzle largely complete, but due to ambiguities, you may need to backtrack because of a wrong decision made early.

A very weak upper bound in the assembly time can be derived by considering the brute force case. For a 1000-piece puzzle, there are 1000! (factorial) possible ways to (attempt to) assemble the puzzle. Suppose one places every piece and then fully backtracks for each trial—this gives 1000 x 1000! puzzle piece placement steps. Using Stirling’s approximation, this is approximately 10^{4468} placement steps.

A human, assuming generously a rate of one step per second, at 31 million seconds/year nonstop, would take on the order of 10^{4460} years. Assuming 10 billion simultaneous puzzlers—order 10^{4450} years.

Now assume the on-record world’s fastest computer, the ORNL Frontier system (though see here), roughly 10 exaflops (10^{19}) mixed precision, and assuming one puzzle step per operation (overly optimistic). Assume further that every atom in the known universe (est. 10^{82}) is a Frontier computer system. In this case—solving the puzzle takes order 10^{4359} years. That’s a thousand billion billion … (repeat “billion” 484 times here) years.

As I mentioned, this is a very weak upper bound. One could solve the problem using a SAT solver with sophisticated CDCL backtracking. Or one could use special methods derived from statistical mechanics that handle random SAT instances. However, to the best of our knowledge, the runtime still grows exponentially as a function of puzzle size. The best worst-case computational complexity for SAT solvers currently known is order O^{∗}(1.306995^{n}) (see here, here, and here). So, simply by constructing puzzles in the way we’ve described with more and more pieces, you will soon reach a point of having a puzzle that is unfathomably difficult to solve.

It’s conceivable that a yet-undiscovered method could be found for which the solution cost would in fact grow polynomially instead of exponentially. But people have been trying to answer this question one way or the other for over 50 years and it’s still unsolved, with an open million dollar prize for proof or counterexample. So we just don’t know.

What about quantum computing? It can solve some exponential complexity problems like factoring in polynomial time. But how about the SAT problem? It is simply not known (see here, here) whether a polynomial-time quantum algorithm exists for arbitrary NP-complete problems. None are known, and it might not be possible at all (though research continues).

So we’re faced with the curious possibility: a jigsaw puzzle could be constructed for which, you’re holding the box of pieces in your hand, you spill it on the floor, and there is no single or joint intelligence or computational mechanism, existing or even possible, in the entire universe that could ever reassemble it.

The post The impossible puzzle first appeared on John D. Cook.]]>The use of the word “moment” in mathematics is related to its use in physics, as in **moment arm** or **moment of inertia**. For a non-negative integer *n*, the *n*th moment of a function *f* is the integral of *x*^{n} *f*(*x*) over the function’s domain.

If two continuous functions *f* and *g* have all the same moments, are they the same function? The answer is yes for functions over a finite interval, but no for functions over an unbounded interval.

Now let’s consider starting with a set of moments rather than starting with a function. Given a set of moments *m*_{0}, *m*_{1}, *m*_{2}, … is there a function that has these moments? Typically no.

A better question is what conditions on the moments are necessary for there to exist a function with these moments. This question breaks into three questions

- The Hausdorff moment problem
- The Stieltjes moment problem
- The Hamburger moment problem

depending on whether the function domain is a finite interval, a half-bounded interval, or the real line. For each problem there are known conditions that are necessary and sufficient, but the conditions are different for each problem.

Interestingly, each of the three names Hausdorff, Stieltjes, and Hamburger are well known. Felix Hausdorff is best known for his work in topology: Hausdorff spaces, etc. Thomas Stieltjes is best known for the Riemann-Stieltjes integral, and for his work on continued fractions. Hans Ludwig Hamburger is not as well known, though his last name is certainly familiar.

A practical question in probability is how well a finite number of moments determine a probability distribution. They cannot uniquely determine the distribution, but the do establish bounds for how different the two distributions can be. See this post.

Now that most arithmetic is carried out in double precision, you can often get away with not thinking about such things. Except when you can’t. The vagaries of floating point computation still matter occasionally, even with double precision, though not as often as they did with single precision.

Although most computing has moved from **single precision** to **double precision**, there is increasing interest in going the opposite direction, from single precision to **half precision**. The main driver is neural networks. You don’t need a lot of precision in weights, and you’ve got a **lot** of numbers to store. So instead of taking 64 bits to store a double precision number, or 32 bits to store a single precision number. you might want to use a 16 bit or even 8 bit floating point number. That way you can fit more weights in memory at once.

However, when you move to lower precision numbers, you now have to think again about the things numerical analysts thought about a couple generations ago, such as different ways of rounding. You might think that floating point rounding could be modeled by random variables. If so, you’re in good company, because John von Neumann suggested this in 1947. But a few years later people began to realize that floating point rounding errors are *not* random. Or to be more precise, they began to realize that modeling rounding errors as random was inadequate; of course they knew that rounding errors weren’t literally random.

But what it rounding errors *were* random? This would lead to more error cancellation than we see in practice with floating point arithmetic. With **stochastic rounding**, the rounded values become unbiased estimators of the values they would like to represent but cannot represent exactly. Now the central limit theorem and all that come to your aid. More on applications of stochastic rounding here.

(To be pedantic a moment, stochastic rounding isn’t truly random, but uses pseudorandom numbers to implement a procedure which is well modeled by randomness. Random is as random does.)

I’ve been writing code for the Z3 SMT solver for several months now. Here are my findings.

Python is used here as the base language. Python/Z3 feels like a two-layer programming model—declarative code for Z3, imperative code for Python. In this it seems reminiscent of C++/CUDA programming for NVIDIA GPUs—in that case, mixed CPU and GPU imperative code. Either case is a clever combination of methodologies that is surprisingly fluent and versatile, albeit not a perfect blend of seamless conceptual cohesion.

Other comparisons:

- Both have two separate memory spaces (CUDA CPU/GPU memories for one; pure Python variables and Z3 variables for the other).
- Both can be tricky to debug. In earlier days, CUDA had no debugger, so one had to fall back to the trusty “printf” statement (for a while it didn’t even have that!). If the code crashed, you might get no output at all. To my knowledge, Z3 has no dedicated debugger. If the problem being solved comes back as satisfiable, you can print out the discovered model variables, but if satisfiability fails, you get very little information. Like some other novel platforms, something of a “black box.”
- In both cases, programmer productivity can be well-served by developing custom abstractions. I developed a Python class to manage multidimensional arrays of Z3 variables, this was a huge time saver.

There are differences too, of course.

- In Python, “=” is assignment, but in Z3, one only has “==”, logical or numeric equality, not assignment per se. Variables are set once and can’t be changed—sort of a “write-once variables” programming model—as is natural to logic programming.
- Code speed optimization is challenging. Code modifications for Z3 constraints/variables can have extreme and unpredictable runtime effects, so it’s hard to optimize. Z3 is solving an NP-complete problem after all, so runtimes can theoretically increase massively. Speedups can be massive also; one round of changes I made gave 2000X speedup on a test problem. Runtime of CUDA code can be unpredictable to a lesser degree, depending on the PTX and SASS code generation phases and the aggressive code optimizations of the CUDA compiler. However, it seems easier to “see through” CUDA code, down to the metal, to understand expected performance, at least for smaller code fragments. The Z3 solver can output statistics of the solve, but these are hard to actionably interpret for a non-expert.
- Z3 provides many, many algorithmic tuning parameters (“tactics”), though it’s hard to reason about which ones to pick. Autotuners like FastSMT might help. Also there have been some efforts to develop tools to visualize the solve process, this might be of help.

It would be great to see more modern tooling support and development of community best practices to help support Z3 code developers.

The post How hard is constraint programming? first appeared on John D. Cook.]]>

where sinc(*x*) = sin(π*x*)/π*x*. This is also called the sinc expansion, or the Whittaker cardinal after its discoverer E. T. Whittaker [1].

This is called the band-limited expansion of *f* because each term in the infinite sum is band-limited, i.e. only has Fourier spectrum within a finite band, because the Fourier transform of the sinc function is a step function supported between −1/2 and 1/2. [2]

The band-limited expansion has a lot of nice mathematical properties, leading Whittaker to call it “a function of royal blood in the family of entire functions, whose distinguished properties separate it from its bourgeois brethren.”

We can find a band-limited approximation for *f* by taking only a finite number of terms in the sum. An advantage of the band-limited approximation over a truncated Fourier series is that the former converges faster, making it useful in numerical algorithms [3]. Here’s an example of approximating the function exp(−*x*²) by taking *h *= 1 and using three terms, i.e. *k* running from −1 to 1.

You can improve the accuracy of the approximation by decreasing the size of *h* or by increasing *N*. This post explains how to pick the trade-off between *h* and *N* to minimize approximation error.

[1] E. T. Whittaker, On the functions which are represented by the expansions of the interpolation theory, Proc. Roy. Soc. Edinburgh, 35 (1915), pp. 181–194.

[2] You may get a different interval of support if you use a different convention for defining the Fourier transform. Unfortunately there are many conventions.

[3] Frank Stenger. Numerical Methods based on Whittaker Cardinal, or Sinc Functions. Source: SIAM Review, Apr., 1981, Vol. 23, No. 2 (Apr., 1981), pp. 165-224

The post Band-limited expansion first appeared on John D. Cook.]]>The most basic theory of delay differential equations is fairly simple. Suppose you have an equation like the following.

*u*′(*t*) + *u*(*t* − ω) = *f*(*t*).

To uniquely determine a solution, you’d need an initial condition. And we’d need more than the value of *u*(0). We’d need a function *g*(*t*) that give the value of *u* on the entire interval [0, ω].

So we initially have the value of *u* over [0, ω]. Next, over the interval [ω, 2ω] the value of *u*(*t* − ω) is known. We could replace that term in the DDE with *g*(*t*), And after we’ve solved our equation over [ω, 2ω], we can use the solution to solve the equation over [2ω, 3ω]. This process is called the **method of steps**.

Although you *can* solve DDEs using the method of steps, this might not be the best approach. It might be more computationally efficient, or theoretically convenient, to use another method to solve such equations, such as Laplace transforms. The method of transforms might convince you that a solution exists, but it might not, for example, be the best way to determine the limiting behavior of solutions.

The post Delay differential equations first appeared on John D. Cook.]]>

The use of Laplace transforms is presented is as follows:

- Transform your differential equation into an algebraic equation.
- Solve the algebraic equation.
- Invert the transform to obtain your solution.

This is correct, but step 3 is typically presented in a misleading way. For pedagogical reasons, students are only given problems for which the last step is easy. They’re given a table with functions on the left and transforms on the right, and you compute an inverse transform by recognizing the result of step 2 in the right column.

Because of the limitations listed above, Laplace transforms, *as presented in an introductory course*, can only solve problems that could just as easily be solved by other methods presented in the same course.

What good is it, *in an undergraduate classroom setting*, if you reduce a problem to inverting a Laplace transform but the inverse problem doesn’t have a simple solution?

Of course *in practice*, rather than in a classroom, it might be very useful to reduce a complicated problem to the problem of inverting a Laplace transform. The latter problem may not be trivial, but it’s a standard problem. You could ask someone to solve the inversion problem who does not understand where the transform of the solution came from.

The most well-known Laplace inversion theorem states that if *f* is a function and *F* is the Laplace transform of *f*, then you can recover *f* from *F* via the following integral.

It’s understandable that you wouldn’t want to present this to most differential equation students. It’s not even clear what the right hand side means, much less how you would calculate it. As for what it means, it says you can calculate the integral along any line parallel to the imaginary axis. In practice, the integral may be evaluated using contour integration, in particular using the so-called Bromwich contour.

It might be difficult to invert the Laplace transform, either numerically or analytically, but at least this is a separate problem from whatever led to this. Maybe the original problem was more difficult, such as a complicated delay differential equation.

There is a lesser-known theorem for inverting a Laplace transform, the Post-Widder theorem. It says

where *F*^{(n)} is the *n*th derivative of *F*. This may not be an improvement—it might be much worse than evaluating the integral above—but it’s an option. It doesn’t involve functions of a complex variable, so in that sense it is more elementary [1].

[1] The use of the word *elementary* in mathematics can be puzzling. Particularly in the context of number theory, *elementary* essentially means “without using complex variables.” An elementary proof may be far more difficult to follow than a proof using complex variables.

For example, it follows directly from the definition that the gamma function Γ(*s*) is the Mellin transform of the function *e*^{−x}.

I ran across an exercise that states an impressive-looking theorem about the Mellin transform, namely that

where *F*(*s*) denotes the Mellin transform of *f*(*x*).

I suppose the theorem looks impressive, at least in part, because it involves the Riemann zeta function. But although the zeta function holds deep mysteries—a conjecture about the location of its zeros may be the most famous open question in mathematics—not everything involving the zeta function is deep. The zeta function has, for example, come up here many times in the context of fairly simple probability problems.

I don’t know whether the theorem above is important. It may be. It resembles the sampling theorem, an important theorem in signal processing, and maybe it has analogous applications. But in any case it is not difficult to prove. A simple change of variables applied to the definition shows that

The proof using the equation above is simply this:

A sawtooth wave takes its name from the shape of its waveform: it looks like the edge of a saw. It also sounds a little jagged.

Sawtooth waves have come up several times here. For one thing, they have rich harmonics. Because the wave form is discontinuous, the Fourier coefficients decay to zero slowly. I wrote about that here. The post is about square waves and triangular waves, but sawtooth waves are very similar.

Here’s a post oscillators with a sawtooth forcing function.

I took sawtooth functions in a different direction in this post that started with an exercise from Knuth’s TAOCP. This led me down a rabbit hole on replicative functions and multiplication theorems in different contexts.

If I remember correctly the sound used for red alterts in Star Trek TOS started with a sawtooth wave. Early synthesizers had sawtooth generators because, as mentioned above, these waves are rich in overtones and can be manipulated to create interesting sounds such as the red alert sound.

The post Sawtooth waves first appeared on John D. Cook.]]>I’m sure I’ve written about this quote somewhere, but I can’t find where. The quote comes from *A Mathematician’s Miscellany* by J. E. Littlewood, citing Besicovitch.

I’ve more often seen the quote concluding with “Pioneering work is ugly.” Maybe that’s what Besicovitch actually said, but I suspect it came from someone misremembering/improving Littlewood’s citation. Since the words are in parentheses, perhaps Besicovitch didn’t say them at all but Littlewood added them as commentary.

One way of interpreting the quote is to say it takes more creativity to produce a rough draft than to edit it.

The quote came to mind when I was talking to a colleague about the value of ugly code, code that is either used once or that serves as a prototype for something more lasting.

This is nearly the opposite of the attitude I had as a software developer and as a software team manager. But it all depends on context. Software techniques need to scale down as well as scale up. It doesn’t make sense to apply the same formality to disposable code and to enterprise software.

Yes, supposedly disposable code can become permanent. And as soon as someone realizes that disposable code isn’t being disposed of it should be tidied up. But to write every little one-liner as if it is going to be preserved for posterity is absurd.

The post Pioneering work is ugly first appeared on John D. Cook.]]>A new Mersenne prime discovery was announced today: 2^{p} − 1 is prime for *p* = 136279841. The size of the new Mersenne prime is consistent with what was predicted.

For many years now, the largest known prime has been a Mersenne prime. That is because there is an special algorithm for testing whether a Mersenne number is prime, the Lucas-Lehmer test. Because of this algorithm, Mersenne numbers can be tested for primality far more efficiently than can arbitrary numbers of comparable size.

There are now 52 known Mersenne primes, but the number just announced may not be the 52nd Mersenne prime. It has been confirmed that the 2^{136279841} − 1 is prime, but it has not been confirmed that there are no Mersenne primes between the 51st Mersenne prime and the number just announced. There could be gaps.

If you were to write the latest Mersenne prime in hexadecimal, it would be a 1 followed by 34,069,960 F’s.

Where in the world did *that* come from?

I’ll sketch where the equation above came from, but first let’s find *W* and log_{2} *W*.

If you take the determinant and multiply by *W*^{10} you get a 10th degree polynomial. A plot shows this polynomial has a root between 1.4 and 1.5.

Applying the bisection method to that interval shows *W* = 1.4529 and so log_{2} *W* = 0.5389 which agrees with the value 0.539 that Shannon gives in his paper. This means that a telegraph can transmit a little more than one bit of information per unit time.

The result above means that a telegraph can transmit a little more than half a bit of information per unit time. But what is the unit of time implicit in Shannon’s example?

We think of Morse code as having two symbols—dot and dash—but in reality it has four symbols: dot, dash, intra-character space, and intra-word space. Shannon assumes that these four symbols take 2, 4, 3, and 6 units of time. Note that the time slice assigned to a dot or a dash includes enough silence at the end to distinguish dots and dashes.

Shannon does not assume that a telegraph is limited to send valid Morse code, only that it can transmit symbols of the four lengths discussed above, and that there are no consecutive spaces. He creates a finite automata model in which an intra-character space or an intra-word space can only transition to a dot or a dash.

Shannon defines channel capacity in [1] as

where *N*(*T*) is the number of allowed signals of duration *T*. The hard part is calculating, or estimating, *N*(*T*). In the case of a telegraph, we can image calculating *N*(*T*) recursively via

where the *t*‘s with subscripts are the times required to transmit each of the four possible symbols. The four terms on the right side consider the possibilities that the symbol transmitted immediately before time *t* is a dot, dash, intra-character space, or intra-word space.

The equation for *N*(*t*) is a finite difference equation, and so the asymptotic behavior of the solution is well known. Since we’re taking a limit as *T* goes to infinity, we only need to know the asymptotic behavior, so this works out well.

If you squint at the determinant at the top of the post, it looks sorta like an eigenvalue calculation, as hinted at by the −1 terms on the diagonal. And in fact that is what is going on. There’s an eigenvalue problem hidden in there that describes the asymptotic behavior of the difference equation.

Here is Theorem 1 from Shannon’s paper.

Let

b_{ij}^{(s)}be the duration of thes^{th}symbol which is allowable in stateiand leads to statej. The channel capacityCis equal to log_{2}WwhereWis the largest real root of the determinant equationwhere δ

_{ij}= 1 ifi=jand is zero otherwise.

This theorem accounts for the determinant at the top of the post. It’s a 2 by 2 determinant because Shannon’s model of a telegraph is a finite automaton with two states, depending on whether the preceding symbol was or was not a space.

[1] Claude Shannon. A Mathematical Theory of Communication. The Bell System Technical Journal, Vol. 27, pp. 379–423, July, October, 1948.

The post Channel capacity of a telegraph first appeared on John D. Cook.]]>Every odd square ends in 1 in base 8, and if you cut off the 1 you have a triangular number.

A number is an odd square if and only if it is the square of an odd number, so odd squares have the form (2*n* + 1)².

Both parts of the theorem above follow from the calculation

( (2*n* + 1)² − 1 ) / 8 = *n*(*n* + 1) / 2.

In fact, we can strengthen the theorem. Not only does writing the *n*th odd square in base 8 and chopping off the final digit give *some* triangular number, it gives the *n*th triangular number.

For this post, **RNG** will mean a physical, true random number generator.

A PRNG may be suitable for many uses—Monte Carlo simulation, numerical integration, game development, etc.—but not be suitable for cryptography. A PNRG which *is* suitable for cryptography is called a **CSPRNG** (cryptographically secure pseudorandom number generator).

A PRNG may have excellent statistical properties, and pass standard test suites for random number generators, and yet be insecure. The output of an insecure generator may have no *statistical* regularities, and yet have regularities that a sneaky cryptanalyst can exploit. For example, the popular Mersenne Twister is fine for simulations but its output can be predicted after a relatively short run. The prediction depends on a clever set of calculations that would be unnatural from a statistical perspective, which is why its statistical performance is better than its cryptographic performance.

CSPRNGs tend to be much slower than PRNGs, so you pay for security. And for a non-cryptographic application this cost isn’t worth it.

In general, statistical tests give **necessary but not sufficient** conditions for a PRNG to be a CSPRNG. If a PRNG fails statistical tests, it has some sort of regularity that potentially could be exploited by cryptanalysis. I had to tell a client once that although his PRNG passed the standard statistical tests, I was pretty sure I could break the cryptographic system he wanted to use it in. This news was not well received.

I suspect that a physical RNG with good statistical properties *will* have good cryptographic properties as well, contrary to the usual case. Cloudflare famously uses lava lamps to generate random bits for TLS keys. Cryptanalysts have exploited minor flaws in PRNGs, and so the lava lamps give Cloudflare one less thing to worry about. (I’m sure they still have plenty else to worry about.)

A physical RNG might fail statistical tests. For example, maybe the physical process is truly random but biased. Or maybe the process of turning physical phenomena into numbers introduces some bias. But it’s hard to imagine that an RNG could have a clean bill of statistical health and yet have a cryptographically exploitable weakness. It’s conceivable that a statistically impeccable physical RNG might have some unforeseen exploitable regularity, but this seems highly doubtful.

2*prR* = *abc.*

We could rewrite this as

2*rR* = *abc* / (*a* + *b* + *c*)

The right hand side is maximized when *a* = *b* = *c*. To prove this, maximize *abc* subject to the constraint *a + b* + *c* = *p* using Lagrange multipliers. This says

[*bc*, *ac*, *ab*] = λ[1, 1, 1]

and so *ab* = *bc* = *ac*, and from there we conclude *a* = *b* = *c*. This means among triangles with any given perimeter, the product of the inner and outer radii is maximized for an equilateral triangle.

The inner radius for an equilateral triangle is (√3 / 6)*a* and the outer radius is *a*/√3, so the maximum product is *a*²/6.

Let *a*, *b*, and *c* be the sides of a triangle.

Let *p* be perimeter of the triangle.

Let *r* be the radius of the largest circle that can be inscribed in the triangle, and let *R* be the radius of the circle through the vertices of the triangle.

Then all six numbers can be related in one equation:

2*prR* = *abc.*

The post Relating six properties of a triangle in one equation first appeared on John D. Cook.]]>

The idea of the Burrows-Wheeler transform is to permute text in before compressing it. The hope is that the permutation will make the repetition in the text easier for a compression algorithm to take advantage of. The permutation must be reversible, so the compressed file can be uncompressed later.

It’s not immediately obvious that the Burrows-Wheeler transform is reversible; that may be the topic of another post some day. This post will explain what the transform is and show that it rearranges text in a way that makes run-length encoding more efficient.

The Burrows-Wheeler transform (BWT) lists all the rotations of a string in a table, sorts the table, then takes the last column as the transform value. (Software implementing the BWT does not need to actually construct the table, but it gives the same result as if it did.)

Before tabulating the rotations, the algorithm adds a character for beginning of string and one for end of string. We’ll use ^ and $ respectively, based on the meaning of these symbols in regular expressions.

Here is an example applied to `wabisabi`

.

^wabisabi$ ^wabisabi$ $^wabisabi $^wabisabi i$^wabisab abi$^wabis bi$^wabisa abisabi$^w abi$^wabis bi$^wabisa sabi$^wabi bisabi$^wa isabi$^wab i$^wabisab bisabi$^wa isabi$^wab abisabi$^w sabi$^wabi wabisabi$^ wabisabi$^

The table on the left lists all rotations of `^wabisabi$`

, and the table on the right is the sorted form of the table on the left. The last column, `$iswaabbi^`

, is the BWT of `wabisabi`

. Note that the `a`

‘s and the `b`

‘s are grouped together in the transformed string.

When we sort the table we run into an annoying complication: what is the sort order of the markers for beginning of text and end of text? I used the Python code from the Wikipedia article on BWT for the examples below, so I followed the code’s sorting convention that the begin string symbol comes first, then the end string symbol, then ordinary text.

At the top of the post I mentioned the lyrics to Jingle Bells as an example of text with repetition. If we calculate the BTW of

jingle bells, jingle bells, jingle all the way.

we get

$.eee,,lessy w lllhbbnnntjjj ^lgggaeelliiill a

Now we have a lot of repeated letters, which means we could compress the string with run-length encoding. For example, we could write `j3`

rather than `jjj`

to indicate a string of three j’s.

$.e3,2les2y w 2l3hb2n3tj3 2^lg3ae2l2i3l2 2a

This isn’t much shorter, though in practice run length encoding would be implemented in binary and would not use an ASCII character `2`

to indicate that a character is repeated.

We could make a string of text even more compressible if we simply sorted it. But sorting is not reversible: any permutation of the text would lead to the same sorted text. This is what we’d get if we sorted the text in the example above

,,.aabbeeeeeeggghiiijjjlllllllllnnnsstwy

Run length encoding compresses this even better, though the result cannot be uncompressed.

72.a2b2e6g3h3j3l9n3s2twy

The way to think of BWT is that **it partially sorts text in a reversible way**. This is kinda paradoxical: sorting is not reversible. You either sort the characters, or you perform a reversible permutation. You can’t do both. But due to the repetition patterns in natural language text [1] you can in practice do both.

The BWT applied to arbitrary text won’t partially sort it. You can’t apply a reversible operation to random input and make it more ordered. But when applied to ordinary prose, BWT does tend to cluster letters together, especially when applied to longer inputs.

As an example of longer input, I applied the BWT to Lincoln’s Gettysburg Address (1454 characters). Here’s an excerpt from the middle of the output.

arrouiuuiga sttcctsss tttttttttwtttwt TtttttttttTsttttt tttwtttww ttttgwww tsgggtLddddddhhddffhmvw f tvtnttvafattttttttteb h hh hrn sflleelcllsrallllllaiap ueretppptg aiuaaaa laobhooeeo e n oioiieaoaaaaoooooieoeooi aooiaaaaaauuei uiiioioiieiir o

This shows that the output is indeed partially sorted.

[1] The BWT is applied to more than natural text. It is used, for example, in compressing genetic sequence data. The key property of the input is that some symbols tend to follow other symbols. This is the kind of repetition that the algorithm is intended to exploit.

The post Preprocessing text to make it more compressible first appeared on John D. Cook.]]>Frequency modulation (FM) came later. With FM radio, changes to the frequency of the carrier wave carry the signal.

I go into the mathematical details of AM radio here and of FM radio here.

Pinter [1] gives a clear explanation of why the inventor of FM radio, Edwin Howard Armstrong, correctly predicted that FM radio transmissions would be less affected by noise.

Armstrong reasoned that the effect of random noise is primarily to amplitude-modulate the carrier without consistently producing frequency derivations.

In other words, noise tends to be a an unwanted amplitude modulation, not a frequency modulation.

FM radio was able to achieve levels of noise reduction that people steeped in AM radio thought would be impossible. As J. R. Carson eloquently but incorrectly concluded

… as the essential nature of the problem is more clearly perceived, we are unavoidably forced to the conclusion that static, like the poor, will always be with us.

But as Pinter observes

The substantial reduction of noise in a FM receiver by use of a limiter was indeed a startling discovery, contrary to the behavior of AM systems, because experience with such systems had shown that the noise contribution to the modulation of the carrier could not be eliminated without partial elimination of the message.

[1] Philip F. Pinter. Modulation, Noise, and Spectral Analysis. McGraw-Hill 1965.

The post Why does FM sound better than AM? first appeared on John D. Cook.]]>The function 1/*z* is a Möbius transformation, and so it maps lines and circles to lines and circles. And in the case of a polar grid, lines go to lines and circles go to circles. The overall grid is unchanged, even though circles swap places with other circles, and lines swap places with other lines. To see this, note that

1/*re*^{iθ} = (1/*r*)*e*^{−iθ}

This means that the function 1/*z* maps a circle of radius *r* to a circle of radius 1/*r*, and it maps a line of slope *m* to a line of slope −1/*m*.

Things get more interesting when we shift the reciprocal function to *w* = 1/(*z* − *c*). Now lines will map to circles, unless the line passes through *c*. Circles will still map to circles, unless the circle passes through *c*, but we won’t have circles swapping places because the function 1/(*z* − *c*) is not its own inverse, unlike 1/*z*. Here’s a plot with *c* = 0.1.

Note that circles map to circles, but the center of a circle does not necessarily map to the center of the image circle. So the green circles in the *z*-plane map to green circles in the *w*-plane, but the circles in the *z*-plane are concentric while their images in the *w*-plane are not.

The blue lines in the *z*-plane become blue circles in the *w*-plane, though some of the circles are too large to fit in the frame of the plot.

As we discussed in the previous post, Möbius transformations are easier to discuss if you adjoint a point at infinity to the complex plane. The radial lines in the *z*-plane all meet at ∞, and so their images in the *w*-plane all meet at 1/(∞ – c) = 0.

One final thing to note is that however small ε > 0 is, the images of the grid lines are very different under 1/(*z* − ε) compared to 1/*z*. Individual *z*‘s that start out close together are mapped to *w*‘s that are close together, but the images of the system of lines and circles is qualitatively different for ε > 0 versus ε = 0.

*f*(*z*) = (*a**z* + *b*)/(*cz* + *d*)

with *ad* ≠ *bc* are called bilinear transformations or Möbius transformations. These functions have three degrees of freedom—there are four parameters, but multiplying all parameters by a constant defines the same function—and so you can uniquely determine such a function by picking three points and specifying where they go.

Here’s an explicit formula for the Möbius transformation that takes *z*_{1}, *z*_{2}, and *z*_{3} to *w*_{1}, *w*_{2}, and *w*_{3}.

To see that this is correct, or at least possible, note that if you set *z* = *z*_{i} and *w* = *w*_{i} for some *i* then two rows of the matrix are equal and so the determinant is zero.

You can pick three points in one complex plane, the *z*-plane, and three points in another complex plane, the *w*-plane, and find a Möbius transformation *w* = *f*(*z*) taking the *z*-plane to the *w*-plane sending the specified *z*‘s to the specified *w*‘s.

If you view the three points as vertices of a triangle, you’re specifying that one triangle gets mapped to another triangle. However, the sides of your triangle may or may not be straight lines.

Möbius transformations map circles and lines to circles and lines, but a circle might become a line or vice versa. So the straight lines of our original triangle may map to straight lines or they may become circular arcs. How can you tell whether the image of a side of a triangle will be straight or curved?

It’ll be easier if we add a point ∞ to the complex plane and think of lines as infinitely big circles, circles that pass through ∞.

The Möbius transformation (*a**z* + *b*)/(*cz* + *d*) takes ∞ to *a*/*c * and it takes −*d*/*c* to ∞.

The sides of a triangle are line segments. If we look at the entire line, not just the segment, then this line is mapped to a circle. If this line contains the point that gets mapped to ∞ then the image of the line is an infinite circle (i.e. a line). Otherwise the image of the line is a finite circle.

The line between *z*_{1} and *z*_{2} can be parameterized by

*z*_{1} + *t*(*z*_{2} − *z*_{1})

where *t* is real. So the image of this line will be a line if and only if

*z*_{1} + *t*(*z*_{2} − *z*_{1}) = −*d*/*c*

for some real *t*. So solve for *t* and see whether you get a real number.

Note that if the point that is mapped to ∞ lies inside the line segment, not just on the line, then the image of that side of the triangle is infinitely long.

To keep things as simple as possible without being trivial, we’ll use the Möbius transformation *f*(*z*) = 1/*z*. Clearly the origin is the point that is mapped to ∞. The side of a triangle is mapped to a straight line if and only if the side is part of a line through the origin.

First let’s look at the triangle with vertices at (1, 1), (1, 4), and (5, 1). None of the sides is on a line that extends to the origin, so all sides map to circular arcs.

Next let’s move the second point from (1, 4) to (4, 4). The line running between (1, 1) and (4, 4) goes through the origin, and so the segment along that line maps to a straight line.

Draw a golden ellipse and its inscribed and circumscribed circles. In other words draw the largest circle that can fit inside and the smallest circle outside that contains the ellipse.

Then the area of the ellipse equals the area of the annulus bounded by the two circles. That is, the area of the green region

equals the area of the orange region.

The proof is straight forward. Let *a* be the semimajor axis and *b* the semiminor axis, with *a* = φ*b*.

Then the area of the annulus is

π(*a*² − *b*²) = π*b*²(φ² − 1).

The area of the ellipse is

π*ab* = πφ*b*².

The result follows because the golden ratio satisfies

φ² − 1 = φ.

(This assumes *ABC* has unit area. Otherwise divide the area of each of the three triangles by the area of *ABC*. We will assume for the rest of this post that the triangle *ABC* has unit area.)

Areal coordinates take three numbers two describe a point in two dimensional space. Why would you do that? It’s often useful to use an overdetermined coordinate system. The benefit of adding one more coordinate is that you get a coordinate system matched to the geometry of the triangle. For example, the vertices of the triangle have coordinates (1, 0, 0), (0, 1, 0), and (0, 0, 1), regardless of the shape of the triangle.

Here is an example of a theorem [1] that is convenient to state in terms of areal coordinates but that would be more complicated in Cartesian coordinates.

First we need to define the **midpoint triangle**, also called the **medial triangle**. This is the triangle whose vertices are the midpoints of each side of *ABC*. In terms of areal coordinates, the vertices of this triangle are (0, ½, ½), (½, 0, ½), and (½, ½, 0).

Now let *P* be any point inside the midpoint triangle of *ABC*. Then there is a unique ellipse *E* inscribed in *ABC* and centered at *P*.

Let (α, β, γ) be the areal coordinates of *P*. Then the area of *E* is

Because *P* is inside the medial triangle, each of the areal coordinates are less than ½ and so the quantity under the square root is positive.

Finding the equation of the inscribed ellipse is a bit complicated, but that’s not necessary in order to find its area.

[1] Ross Honsberger. Mathematical Plumbs. 1979

The post Areal coordinates and ellipse area first appeared on John D. Cook.]]>The function *d* varies erratically as the following plot shows.

But if you take the running average of *d*

*f*(*n*) = (*d*(1) + *d*(2) + *d*(3) + … + *d*(*n*)) / *n*

then this function is remarkably smoother.

Not only that, the function *f*(*n*) is asymptotically equal to log(*n*).

Incidentally, directly computing *f*(*n*) for *n* = 1, 2, 3, …, *N* would be inefficient because most of the work in computing *f*(*m*) would be duplicated in computing *f*(*m* + 1). The inefficiency isn’t noticeable for small *N* but matters more as *N* increases. It would be more efficient to iteratively compute *f*(*n*) by

*f*(*n* + 1) = (*n* *f*(*n*) + *d*(*n* + 1)) / (*n* + 1).

Several people have asked for a reference for the results above. I didn’t know of one when I wrote the post, but thanks to reader input I now know that the result is due to Dirichlet. He proved that

*f*(*n*) = log(*n*) + 2γ − 1 + o(1).

Here γ is the Euler-Mascheroni constant.

You can find Dirichlet’s 1849 paper (in German) here. You can also find the result in Tom Apostol’s book Introduction to Analytic Number Theory.