Naive interpolation of rotation matrices does not produce a rotation matrix. That is, if R1 and R2 are rotation (orthogonal) matrices and 0 < t < 1, then
is not in general a rotation matrix.
You can represent rotations with unit quaternions rather than orthogonal matrices (see details here), so a reasonable approach might be to interpolate between the rotations represented by unit quaternions q1 and q2 using
but this has a similar problem: the quaternion above is not a unit quaternion.
One way to patch this up would be to normalize the expression above, dividing by its norm. That would indeed produce unit quaternions, and hence correspond to rotations. However, uniformly varying t from 0 to 1 does not produce a uniform rotation.
The solution, first developed by Ken Shoemake [1], is to use spherical linear interpolation or SLERP.
Let θ be the angle between q1 and q2. Then the spherical linear interpolation between q1 and q2 is given by
Now q(t) is a unit quaternion, and uniformly increasing t from 0 to 1 creates a uniform rotation.
[1] Ken Shoemake. “Animating Rotation with Quaternion Curves.” SIGGRAPH 1985.
I wonder if geometric algebra primitives (rotors? I’m not an expert) would allow this kind of rotation interpolation using a more natural/naive approach.
Following up from my previous comment, I should point out the excellent interactive article “Let’s remove quaternions from every 3D engine” (https://marctenbosch.com/quaternions/).