Lessons from Mathematical Optimization – the basis for current AI
Machine Learning and many parts of current ‘AI’ systems are built on top of the field of Mathematical Optimization. There is a lot of speculation about how AI will change software development. Mathematical Optimization has existed since before computers. Gauss and Newton both proposed iterative methods for solving optimization problems. Those methods were done by hand, and later when computers arrived on the scene, they were programmed into computers. Computers have been used to solve optimization problems since the 1950s, if not earlier. There may be some interesting insights we can glean about the possible evolution of AI usage by looking at how computers impacted the field of mathematical optimization itself.
How Mathematical Optimization works
When you do mathematical optimization, you provide the objective and you specify constraints, then an algorithm is used to search for the best solution to your problem. The problem specification is called a program. The objective is a measurable function that you are trying to maximize or minimize. The constraints are functions that have to be satisfied in order for a solution to be valid. You also specify parameters that the algorithm can adjust to change the values of the objective function. A combination of parameters that satisfies the constraints is called a feasible solution, and the set of all possible parameters that satisfies the program constraints is known as the feasible region. At each step in the algorithm, a new set of parameters is used, the objective function is measured, and the constraints are checked. The algorithm keeps searching through the feasible region until it finds the ‘best’ solution, as measured by the objective function.
You can search through the feasible region by hand to find better solutions to the program. This is how problems were solved before computers. But interesting programs may have billions of possible solutions and computers are much faster than people for that kind of activity. So when computers came along, they took over the search portion of mathematical optimization. The search algorithm is programmed into the computer, and it does the work of iterating through possible solutions until the best one is found. So a person will specify the optimization program, and the computer will search for the solution. Keep that in mind for later, it will be important.
An Analogy Between Optimization and Software Development
Let’s try to think of software development as a mathematical optimization problem and see where that goes.
When you build software, you have two important roles. The first role we will call the Product Owner. The Product Owner determines what to build. The second role we will call the Developer. The Developer builds something that satisfies the Product Owner’s desires. If you’re working on your own project, you may fill both roles. On a bigger team, you may have multiple people filling those roles. But let’s keep it simple and just say there is one person acting as the Product Owner, and one person acting as the Developer.
The Product Owner sets the constraints for the software. The constraints might be detailed behavioral specifications, or they might be a suite of tests, or they might be vague statements, or they might be technology restrictions, and so on. The Product Owner might also set a more open-ended objective, like “Make the response time as low as you can.” The developer is then responsible for finding a program that satisfies the constraints, and maximizes or minimizes the objective. Commonly a developer will create a program that works (satisfies the constraints), and then iteratively modify it to improve performance as measured by the objective.
The only feasible programs are those that satisfy all the constraints. There are usually many, many possible implementations, or configurations of code, that can pass all the tests. Or, phrased differently, there are many feasible configurations of code. All of the feasible program implementations put together form the feasible region of solutions for the desired program. The Developer searches through this space to find the one that is ‘best’ according to some criteria.
In mathematical optimization terms, the Product Owner is defining the program, and the Developer is performing the search algorithm.
And now things look interesting. In mathematical optimization, a person defines the program, and the computer performs the search. So it would seem that the activity of finding the configuration of code that satisfies the Product Owner’s specifications would be a good candidate for a computer to perform.
There is still a long way to go
Using generative AI for software development has plenty of room for improvement. It is still in its infancy. Mathematical optimization was adopted for major industrial use in the 1950s and 1960s. Sixty or seventy years later, it is still an active field of research and is probably underutilized in business applications. Most people I talk to have never even heard of the field of Operations Research, which is responsible for the development of much of mathematical optimization.
One thing that needs improvement is feedback mechanisms. When a computer solves an optimization program, it has a source of feedback on the quality of its solutions. It can evaluate a solution using the objective function to get a number that can be compared against other solutions. There is an iterative feedback loop where it tries a solution, evaluates it, then tries a different solution, evaluates it, and so on. We will need similar structured feedback mechanisms for software programs and the AI producing them.
Along with that is the challenge of software complexity. One aspect of that complexity is evaluating how ‘good’ a solution is. Which aspects of performance matter? Do we care about code quality, and how do we measure it? What about technology choices? Is it better to use a 3rd party library in the solution or not? Another aspect of that complexity is the requirements (or constraints) specification. Even in software projects with thousands of tests describing the software’s behavior, there are invariably undocumented behaviors or requirements that may be tribal knowledge. If a computer is used to find a software implementation, it needs to be given all the necessary constraints.
We may also need more precise means of communicating with generative AI. Mathematical optimization programs are specified using the language of math. Math constructs are highly structured and have precise meanings. There are few, if any, ambiguities. English and other natural languages are full of ambiguities and imprecisions. Even many programming languages are very imprecise when it comes to defining structures. Most software program requirements are defined as automated tests. But those are just sampling techniques where you only specify part of the function domain and what the outputs should be, not rules in the same way that a mathematical function is. Imagine a test for a function that always needs to return a positive value. You specify positive test cases for 1, 2, 100, and MAX_INT, and negative test cases for 0, -1, -1000, and -1*MAX_INT, instead of saying x > 0 . The output x has an infinite number of possible values and you can’t list them all. If the computer is trying to find a feasible solution that satisfies the test cases, how does it infer the rule x > 0 from a handful of test cases? If you could simply write your test as x > 0 , that is a more precise rule that can be used check program feasibility as the computer iterates through many possible implementations. Perhaps programming language concepts like dependent types will facilitate better language precision.
How to prepare yourself
Let’s go back to the mathematical optimization analogy. The computer does the work of finding the solution to your optimization program. But it’s not a magic process where anybody anywhere can just get a mathematical optimization problem solved. First off, you have to know that mathematical optimization even exists and what it might be useful for to even pose a useful program to the computer to solve for you. Beyond that, you need some level of understanding of the structure of problems and how to model them so that the computer can solve them. You need to be able to interpret solutions. You need to understand the details of how the problems are solved so you can tweak your program to get the outcomes you want. And so on. Put another way, you need to be able to communicate intelligently with the computer about mathematical optimization in order to actually use it for mathematical optimization. That’s going to be true for generative AI usage as well.
With AI tools taking over the task of searching for a suitable code implementation to solve a particular need, the human component of software development shifts towards specification and modeling. What are we trying to accomplish with this software? What are the constraints? People still need to understand how to build things with software so that they can effectively communicate what needs to get done to the computer. It’s not that different than how senior engineers or architects or product owners work today. They lay out the problem and the high-level characteristics of the solution. They document any constraints that need to be met, such as specific technologies that need to be used, or minimum performance requirements, such as latency, throughput, or memory usage. They communicate that to other people who are responsible for finding a suitable code implementation. They give feedback on the work, and iterate until release.
So the takeaway here is: practice communication. Practice writing clear behavioral specifications. Practice writing tests that precisely describe the requirements of the software implementation. Learn programming languages that can express more program constraints at compile time. Practice domain modeling. Practice communicating software behaviors to your teammates. Learn everything you can about different algorithms, data structures, and architectures. And don’t worry too much about writing the best implementations. As generative AI becomes more useful for writing software, its going to find better implementations than you will. But if you can’t communicate effectively with it, you won’t be able to use it.
