I’ve enjoyed following Stephen Boyd’s lectures on convex optimization. I stumbled across a draft version of his textbook a few years ago but didn’t realize at first that the author and the lecturer were the same person. I recommend the book, but I especially recommend the lectures. My favorite parts of the lectures are the off-hand remarks about which methods are well known and which are not, what different fields call the same methods, which concerns are merely academic minutia and which are practically important. This kind of information is almost impossible to pick up just from reading books and articles.
A surprisingly large set of problems can be formulated as minimizing a convex function subject to convex constraints, and algorithms for solving these problems are very well developed. It’s roughly true that an optimization problem is tractable if and only if it is convex. Of course that’s not exactly true. Many special non-convex problems can be solved easily, though if you write down an arbitrary non-convex optimization problem, it’s probably hard to solve. On the other hand, enormous convex optimization problems can often be solved quite efficiently.
I became interested in optimization a few years ago in order to solve some problems that came up at work. After going through Boyd’s lectures, I’d like to go back and do a better job on some of these problems. For example, the EffTox dose-finding software solves an optimization problem to determine prior parameters based on input from doctors. That optimization is slow and not very robust. If I can find the time to do it, I think I could improve that software.
When it comes to optimization, there are a few rules of thumb I try to follow. First, I try to formulate the problem as a tractable one, i.e., (quasi)convex with a convex feasible region, a linear program, or a discrete linear program (which is non-convex,but for which very effective algorithms exists). Second, I try to find the most parsimonious formulation. This is where experience and skills come in. Last, I never code the algorithm. However simple, existing codes are usually 2-3 times faster than a naive code. There are excellent open-source and commercial codes that do the job, i.e., lpsolve, the COIN-OR family, MOSEK, CPLEX, etc. A lot of them already have python/R/matlab interfaces.