Fabricated Complexity

There is a quote in computer science, “the solution to a hard problem, when solved, is simple.” I don’t know who to attribute it to, but I have repeatedly found myself arriving at very simple and elegant solutions to hard problems—problems in natural language, in AI, and in application development.

Anna Liu mentioned a talk by Willty Zwaenepoel on research and fabricated complexity.

He spoke of "Fabricated Complexity" - and basically about his observation that researchers often over complicate issues to make them seem 'interesting and novel' and to be accepted by the academic peer review process, while real practical/applicable ideas that lead to useful innovations often are actually based on 'simple ideas'.

He also came to the conclusion that 'design by contract/stable interfaces' are the key to successful (maintainable) innovation, despite he and his team spent many years of building some of the most 'sophisticated/complex' algorithms in distributed systems technologies, it was down to these simple software engineering concepts that would lead any innovation to wide adoption.

One paper that comes to mind is from the Spec# at Microsoft, Abstract Interpretation with Alien Expressions and Heap Structures. It took me a few readings to digest this paper. I gathered that “polyhedra domains” refers to the feasible region in the simplex algorithm, but much of the other content made more sense after going through related literature and course material.

What is more surprising is that I came up with a simpler, more elegant and more general solution than those mentioned in this paper. Perhaps, fabricated complexity leads to complex solutions. If simple, familiar ideas were used, would our mind be able to discover more associations with other related ideas? There may also be a greater tendency to preserve the beauty of a system when the components themselves are beautiful; when not, anything goes.

In economics, unions effectively drive up wages while limiting employment. Professional organizations maintain high levels of income of their highly-paid members by creating legal barriers of entry to new (usually younger) entrants. They lobby the government to impose stricter requirements for licensing, while, at the same time, urging grandfather clauses for existing practitioners. New interns must undergo many years of schooling, learn a new vocabulary, and work hours with low pay for a number of years. The same holds especially true for researchers of academia. 

I am not making a value judgment here; such practices might attract only the most dedicated people and also improve quality. We don’t want to reward those who only see the monetary incentive, but are not willing to put in effort. For the most part, the tendency for professionals to sharply divide the world between “knows” and “knows-not” is unconscious but convenient.

Just as professionals introduce a more complicated vocabulary, software companies often come up with complicated names in their products and libraries when simple ones would do. Since computer programming is so broadly applicable and valuable, an arcane vocabulary in this industry does not serve us well. While it’s not as basic a skill as reading and elementary mathematics, it ranks with business vocabulary, which is more accessible than the vocabulary of other professions.

Apple has always used friendly names in developer APIs. In contrast, Microsoft often unnecessarily complicated its API, especially COM, with cryptic names and hungarian notation, unintentionally driving users to the Java space and other platforms. Perhaps, this is partly because the architect of COM/OLE used to be a high-energy physicist from Oxford. His influence can be seen through the use of physical terms like source and sink in some COM interfaces. I once spoke to an OLE dev lead, who remarked that OLE development was incompatible with a short development time. It became clear to me that Microsoft APIs were unconsciously designed for other large software companies, most of which have since been vanquished by you know who.

The .NET Framework, especially with the help of the FX tool, enforces that the name of every type and method can be found a certain dictionary. Because it was designed for multiple languages, it had to serve the lowest common denominator; it had to be easy to use by the VB developer. This I think is one of the many reasons for its success.

One problem with functional languages comes from its heritage in academia. Hence, newcomers are often intimidated by the unfamiliar terminology, which actually refers to simple ideas. Functional programming languages look hard or too mathematical, when in fact they should actually be conceptually easier. The LINQ team has made many functional constructs dramatically more accessible by giving them names that sound more like conversational English. Operators like Map, filter, reduce, fold were changed to the names of their SQL counterparts such as select, where, sum, aggregate and so on. Terms like lambda expressions or closures might also become more accessible if described as anonymous functions or blocks. (Don’t try to figure out why closures are called “closures.” The origin of the name, which I have forgotten, is now meaningless today, but there use to be an “open” counterpart to the concept.)

Comments

[John Rusk]

"Fabricated Complexity" is good term (and by no means limited to academia, unfortunately).

[John Rusk]

Good as in "a good description of a bad phenomenon"

[NoiseEHC]

If you want to see a C# like functional language then check Nemerle. It rocks!
nemerle.org

[Drazen Dotlic]

Actually, the quote about simple solution is from the book "Positioning" (I'm positive you've read it as a future ISV).

More details and some of my observations about complexity of problems related to simplicity of solutions can be found in this post: http://drazen.dotlic.name/weblog/archive/2004/09/27/183.aspx

[Wesner Moise]

I believe there is another source (not just positioning) and that it is also tied to computer science.

Here's another one:
For every complex problem, there is a solution that is simple, neat, and wrong.

[mgsloan]

The problem with 'anonymous function' is that it doesn't imply that it may use variables from the parent scope. This is, indeed why closures are called closures - they capture the lexical environment.

I disagree, though - certainly many things are overcomplicated due to reasons you state, but this doesn't mean every hard problem has a seemingly simple solution.

[Jonathan Aquino]

Amen to good naming. Blocks over closures! Good semantics are invaluable.

[Lance FIsher]

In one sentence you've explained lambdas and closures to me. Thanks.

[Kevin Greer]

Good Article. This could help to explain why OO had better adoption than functional programming (FP), despite being more complex. The Smalltalk gang were creating a system for children and so they described it simply. The FP crowd on the other hand took a less accessable academic/mathematical approach to describing their work.

[Chris W.]

From Richard W. Hamming [http://en.wikipedia.org/wiki/Hamming], the creator of the Hamming code and author of 'You and Your Research' [http://www.cs.virginia.edu/~robins/YouAndYourResearch.html]:

"It is better to solve the right problem the wrong way than to solve the wrong problem the right way."

I would suggest that a simple solution to a hard problem is almost always found by reconceiving the formulation of the problem. Having done this, the first attempt at a solution may be wrong, but it helps one see the strength of the problem formulation---that one is on the right track to achieving a simple and correct solution.

(The downside is that one actually has to think hard.)

[Jim]

Naming of types and members is so incredibly important. Having worked with an English as a second language programmer, I saw first hand how she struggled to name methods and variables - the end result was that the exact meanings of these members was unclear to her and other people.

Likewise, a strong vocabulary is useful for disecting a problem, and arriving at a solution - (infact, isn't there some doubt whether humans could have reasoned thought without language...)

On the one hand powerful vocab is useful, but on the other it has to reach it's audience, otherwise it's no more helpful the impricise language.