Smart Software

Siri Technology

Considerable press puffery has followed Apple’s announcement of Siri dramatizing the significant nature of the natural language technology. Siri is touted as a $200 million DARPA-funded research in the same league as the Internet and GPS.

Siri is much advanced over existing popular voice control technology in its acceptance of free-form input, access to web services, and conversational feedback; however, it probably not too much of an advance. Microsoft and Google should catch up quickly. Microsoft previewed a future vision of its acquired Tellme voice control technology, and Google is likely to advance its Voice Actions to match Apple’s developments.

Voiced-based assistant technology is ideal for a mobile device because the phone is designed to host conversations, geolocation is available, internet data is available, and all the user’s information is already stored in the device. Because contextual information is abundant and the domain knowledge is constrained, speech recognition can be very effective.

My reaction to Siri was “finally”—finally a realization of state of art in technology into commercial products. There’s a lot of technology that I encountered in academia decades ago that have not yet made their way to consumers such as natural language analysis. In college, I wrote a Prolog-based natural language parser based on definite clause grammars that took parsed text and converted it into a semantic representation, on which we perform queries. As part of my entrepreneurial work, I rewrote the Link Grammar parser from CMU and obtained licenses to various natural language data sources like COMLEX, NOMLEX and WordNet. It felt odd that I might be the first to sell a consumer natural-language product aside from grammar checkers and translation software.

So, finally, but…  I am a bit skeptical about the technological advances, since I am simply one developer and, though I have leverage off the work of others, I did not need $200M. First, Siri uses Dragon’s speech recognition technology, which any developer can license as part of the Nuance Mobile Developer Program. Parsing natural language. Second, the DARPA halo is just for dramatics, but the kind of natural language analysis has been around for awhile. I credit Apple for pushing the quality of the communication beyond mere keyword and structure recognition and for putting existing art into its products. I suspect that most of those stated 300 SRI researchers consulted but did not actually work on CALO (let alone full-time). From the looks of the project involved, most of the technology appears in the backend (much of which may not even be relevant to Siri), very little in natural language analysis.

In How Siri Works, the author presents his own skepticism of Siri technology. Jeff is more impressed with application and integration of Siri into the OS than with the technology itself. Siri performs operations on a limited set of operations centered around built-in iPhone applications, plus it integrates with a number of web services such as Yelp, Wolfram|Alpha, OpenTable and Wikipedia. Despite the limitations, it still is an impressive achievement given the naturalness of the implementation.

Another CALO engineer confirms my thoughts of Siri as a compelling but not terribly advanced technology:

I worked at SRI on the CALO project, and built prototypes of the system that was spun off into SIRI. The system uses a simple semantic task model to map language to actions. There is no deep parsing - the model does simple keyword matching and slot filling, and it turns out that with some clever engineering, this is enough to make a very compelling system. It is great to see it launch as a built-in feature on the iPhone.

The NLP approach is based on work at Dejima, an NLP startup: “Iterative Statistical Language Model Generation for Use with an Agent-Oriented Natural Language Interface”

A lot of the work is grounded in Adam Cheyer's (CTO of SIRI) work on the Open Agent Architecture: A more recent publication from Adam and Didier Guzzoni on the Active architecture, which is probably the closest you'll come to a public explanation of how SIRI works: Active, a Platform for Building Intelligent Software

His comments on the natural language parsing left me disappointed, but it’s possible that Apple upgraded that natural language processing capabilities of Siri with its homegrown version after acquiring Siri. However, after reading the Dejima paper, it turns out that a traditional parser may have too rigid a grammar for the short, conversional, and often grammatically incorrect speech input.

NLIs often use text as their main input modality; speech is however, a natural and, in many cases, preferred modality for NLIs. Various speech recognition techniques can be used to provide a speech front end to an NLI. Grammar-based recognizers are rigid and unforgiving, and thus can overshadow the robustness and usability of a good NLI. Word-spotting recognizers are reliable only when the input consists of short utterances, and the number of words to be spotted at each  given time is small. Dictation engines are processor and memory intensive, and often speaker dependent. The dictation vocabulary is often considerably larger than required for domain-specific tasks. General statistical language models (SLMs), although robust enough to be used as a front end for a structured domain, requires a very large training corpus. This is time consuming and expensive since a large number of users needs to be sampled and all speech has to be transcribed.

The system proposed in the paper is a statistical language designed explicitly for an agent-oriented natural language speech-based interface. It does have its failings such as the weaknesses detailed in Siri in Practice where Siri has trouble with parenthetical or quoted expressions that have might have been more properly handled with a grammar-based recognizer, assuming that Apple has not changed Siri’s parser.

Another telling failure is Siri’s response to “What’s the best iPhone wallpaper?” in which Siri responds with a canned response to “What’s the best phone?” as if it simply did not process the word “wallpaper” and simply hooked on keywords “best” and “iPhone.”

The response vaguely resembles that of fairly unsophisticated chatterbots. Siri could be performing a keyword match either on the syntactic or the semantic level. It would be easy to test this hypothesis by asking Siri variations of the questions. I doubt the seriousness of these mistakes, because Apple might be using a different catch-all, keyword matching-system for unanticipated queries.

Siri and the Human Connection—The Eliza Effect

In accordance with combining technology with the liberal arts, I believe that Apple attempted in Siri to establish a more human connection to the iPhone user, first, by allowing it to understand free-form conversation and, second, by giving it a personality whereby the user may feel that she has a relationship with a humanoid rather than a machine. Literally, this has been the vision of the Knowledge Navigator with its animated virtual human and other social interface projects over the past couple decades.

 

The Wall Street Journal recently explored Apple’s motivation in Siri while asking the question whether smartphones are becoming smart alecks? The reporter noted that the original creators of Siri put “deep thought” into its personality, giving it “a light attitude.”

When Apple began integrating Siri into the iPhone, the team focused on keeping its personality friendly and humble—but also with an edge, according to a person who worked at Apple on the project. As Apple's engineers worked on the software, they were often thinking, "How would we want a person to respond?" this person said.

The Siri group, one of the largest software teams at Apple, fine-tuned Siri's responses in an attempt to forge an emotional tie with its customers. To that end, Siri regularly uses a customer's nickname in responses, as well as those of other important people and places in his or her life. "We thought of it almost as a person on the phone," this person said.

As for its effect, we see evidence in twitter and news articles repeating many of the sassy responses of Siri. My iPhone says the darndest things, writes one reporter. In actuality, the emotional bond between the iPhone and users that Apple attempted to forge through Siri has been described years ago as the “Eliza effect.” The Eliza effect is the bond that has been observed from users chatting with less sophisticated chatterbots.

 

 

 

In a series of articles on the history of Eliza, the first and most widely known chatterbot, Jimmy Maher recounts the unexpected connection with humans who worked with it.

Perhaps the first person to interact extensively with Eliza was Weizenbaum’s secretary: “My secretary, who had watched me work on the program for many months and therefore surely knew it to be merely a computer program, started conversing with it. After only a few interchanges with it, she asked me to leave the room.” Her reaction was not unusual; Eliza became something of a sensation at MIT and the other university campuses to which it spread, and Weizenbaum an unlikely minor celebrity. Mostly people just wanted to talk with Eliza, to experience this rare bit of approachable fun in a mid-1960s computing world that was all Business (IBM) or Quirky Esoterica (the DEC hackers).

Weizenbaum’s reaction to all of this has become almost as famous as the Eliza program itself. When he saw people like his secretary engaging in lengthy heart-to-hearts with Eliza, it… well, it freaked him the hell out. The phenomenon Weizenbaum was observing was later dubbed “the Eliza effect” by Sherry Turkle, which she defined as the tendency “to project our feelings onto objects and to treat things as though they were people.” In computer science and new media circles, the Eliza effect has become shorthand for a user’s tendency to assume based on its surface properties that a program is much more sophisticated, much more intelligent, than it really is.

All that aside, I also believe that, at least in his strong reaction to the Eliza effect itself, Weizenbaum was missing something pretty important. He believed that his parlor trick of a program had induced “powerful delusional thinking in quite normal people.” But that’s kind of an absurd notion, isn’t it? Could his own secretary, who, as he himself stated, had “watched [Weizenbaum] work on the program for many months,” really believe that in those months he had, working all by himself, created sentience? I’d submit that she was perfectly aware that Eliza was a parlor trick of one sort or another, but that she willingly surrendered to the fiction of a psychotherapy session. It’s no great insight to state that human beings are imminently capable of “believing” two contradictory things at once, nor that we willingly give ourselves over to fictional worlds we know to be false all the time. Doing so is in the very nature of stories, and we do it every time we read a novel, see a movie, play a videogame. Not coincidentally, the rise of the novel and of the movie were both greeted with expressions of concern that were not all that removed from those Weizenbaum expressed about Eliza.

Aside from deluding oneself that the computer is human, the user also, aware of the failings of the computer, attempts to maintain this delusion by steering their behavior as Sherry Turkle writes in The Second Self.

As one becomes experienced with the ways of Eliza, one can direct one’s remarks either to “help” the program make seemingly pertinent responses or to provoke nonsense. Some people embark on an all-out effort to “psych out” the program, to understand its structure in order to trick it and expose it as a “mere machine.” Many more do the opposite. I spoke with people who told me of feeling “let down” when they had cracked the code and lost the illusion of mystery. I often saw people trying to protect their relationships with Eliza by avoiding situations that would provoke the program into making a predictable response. They didn’t ask questions that they knew would “confuse” the program, that would make it “talk nonsense.” And they went out of their way to ask questions in a form that they believed would provoke a lifelike response. People wanted to maintain the illusion that Eliza was able to respond to them.

Just imagine the potential impact in consumer brand loyalty that a well-designed assistant like Siri could impart, should users willfully engage in the illusion of a human-like assistant and even actively maintaining this self-deception. In the least technologically savvy portions of the population, the user may not even understand the technological limitations and could really believe the device truly understands her.

Technology & Liberal Arts

Steve Jobs often attributes that the popularity of Apple products comes from merging technology with the humanities.

During the introduction of the first iPad, Jobs notes that "the reason that Apple is able to create products like iPad is because we always try to be at the intersection of technology and liberal arts, to be able to get the best of both, to make extremely advanced products in a technology point of view but also have them be intuitive, easy to use, fun to use so they really fit the users. The users don’t have to come to them, they come to the users. It’s the combination of these two things that has let us make the kind of creative products like the iPad. "

He repeated this theme during the unveiling of the second iPad as he proclaimed a new post-PC era. “It’s in Apple’s DNA that technology alone is not enough. It is a marriage of technology with the liberal arts and humanities. The competitors we see it as a new PC market. That’s not the right approach. Tablet is a computer that needs to be easier to use than a PC and should be more intuitive.”

As emerging competitors in the tablet space have attempted to one-up Apple with better “specs,” Apple simply ignores their moves, instead talking about products like GarageBand in the iPad2 launch to enable users to produce studio-quality music without effort. Each product launch features an exciting consumer software element like iMovie for iPhone 3GS, FaceTime, or Siri that reaffirms the human element in software. Apple products are beautiful, fun, personal, and magical. It just works.

I am reminded of a post “Apple is professional, the web is amateur,” praising Apple’s craftsmanship and show of love in its software; after reading the post, I played with the iLife suite and was completely stunned with the quality of websites that could be created with iWeb. (In contrast, Microsoft FrontPage was the surest way to produce an amateurish webpage as if none of the developers designed webpages. My encounter with Dreamweaver in 2000 was a “wake up” moment in the mediocrity of some of Microsoft applications.) As opposed to Android and Microsoft commercials that push vague, unauthentic, corporate branding messages, Apple’s products speak for themselves front-and-center in commercials with their impeccable beauty and magical qualities, whether it’s the richness of applications (especially those augmenting reality using phone-based sensors or tapping into web services like OpenTable), understanding of natural speech, or video-based calls.

This stance of melding humanities into technology harkens back to the Steve Job’s launch of the “Think Different” campaign, which shifted Apple’s marketing focus from technology to values, which is a passion.

The Apple brand has clearly suffered from neglect in this area in the last few years and we need to bring it back. The way to do that is not to talk about speeds and feeds; not to talk about MIPS and megahertz; not to talk about why we’re better than Windows…

Our customers want to know who is Apple and what is it that we stand for, where do we fit in this world. What we are about isn’t making boxes to get people to get their jobs done, although we do that well, we do that better than almost anybody in some cases. But Apple is about something more than that. Apple at the core, it’s core value, is that we believe that people with passion can change the world for the better.

In the recent CBS interview , Jobs is taped saying that Microsoft never had the humanities and liberal arts in its DNA, that they are a pure technology company. He also believes Google is turning the same way.

Siri and the AI Revolution

With natural user interfaces (like touchscreens and microphones) and sensory rich devices (like GPS and cameras) rapidly become mainstream due to the rise of smartphones and tablets alongside growing acceptance of speech recognition, augmented reality and now the conversational UI in the form of Siri, there’s been growing predictions of a coming AI revolution in the computer industry.

• Siri; The Artificial Intelligence Revolution has begun.
• Apple Siri: the next big revolution in how we interact with gadgets?
• Apple’s Siri Is as Revolutionary as the Mac

In an interview with 9to4mac, Norman Winarsky, cofounder of Siri, predicted before the iPhone launch that the Siri Assistant would be a world-changing event. The other cofounder Tom Gruber, former Siri CEO, articulated the underpinnings for this change in his Jan 2010 Web 3.0 conference keynote, “Big Think Small Screen: How semantic computing in the cloud will revolutionize the consumer experience on the phone.”

After the iPhone launch, there is still widespread skepticism about the practicality of Siri and conversation interfaces given the past failures of speech and handwriting recognition software, but I believe that Apple has placed an indelible germ in people’s minds that will continue to grow in coming years that will manifest itself into new products and expectations. I would not be surprised, if Siri, in future versions, morph into an animated person as in the Knowledge Navigator videos.

Siri is also born in the right age, where mobile devices that can wirelessly connect to servers, maintain a copy of user’s important information and contacts, and retrieve context like geolocation and where third-party services like Yelp for businesses, OpenTable for restaurant reservations, and Wolfram Alpha for general facts abound on the semantic and social Web. There is also ample memory and computing resources available within a mobile device; natural languages assets have accumulated over time; the web has also offered a rich amount of information for data mining and machine learning. In a mobile device designed to host phone conversations, speech may even be the most convenient form of input, since the keyboard is so small and the relevant application, if there is one, may be several clicks away.

With developments such as Watson, Siri, natural user interfaces, real-time Google translations, AI might finally be here.

I am somewhat worried, though, since my entire business strategy is based on AI technologies and the giant software companies are coming in.

In a presentation of my business plan in 2002 for a natural language software company, one feedback that I received was that AI is a dead end with allusions to massive failures in the 1980s with Japan’s Fifth Generation Project. I was amused, not worried. AI is a big field. I am familiar with those failed projects which involved Prolog and expert systems but have very little relationship to the work that I am doing of which I already produced proofs of concept. A fellow business school classmate who helped with my business plan could not comprehend that a computer could truly understand natural language, offering up arguments as to why it was impossible. How quickly, times change.

Conversational Interfaces: Siri & Shrdlu

Apple announced Siri as a voice-activated “personal assistant” available exclusively in the iPhone 4S.

Siri reminds of a few projects from the past, but it’s the first apparent demonstration of natural language understanding in a consumer product (excluding simplistic natural-language-based games like Zork).

First, it’s the realization of the conversational interface envisioned by Apple’s Knowledge Navigator video in 1987 and a couple other copycat vision videos produced by IBM and Microsoft. I could easily see the Siri user interface expanding to include a digital, three-dimensionally rendered person in the future.

Second, it embodies the inherently social interaction people have with their computers—a thesis put forth by the social interface work of two Stanford researchers (Cliff Nass and Byron Reeves) that lead to the Microsoft Bob user interface project and the Microsoft Agent technology.

Mostly, it reminds me of SHRDLU - a natural language program written way back in 1968 (around the same time as the Mother of All Demos which predicted mouse-based interfaces, windows, and video conferencing) that presented the first compelling demonstration of a conversational interface in which the computer fully understood and operated upon the direction of a user. Although ELIZA existed four years earlier, ELIZA was built on heuristics and dumb pattern matching without any real understanding or memory of its conversation with humans.

There are other projects over the years like mention expert systems. Recent products in popular technology include IBM’s Watson and CYC, which can analyze the Web and direct answers to natural language queries, and Wolfram Alpha (also used as a source in Siri), which acts a computation engine to deliver or derive answers to natural language query from an organized database of facts.

SHRDLU was developed as part of a PhD dissertation generations ago both in Moore’s Law and human terms, and the original source code in LISP is still available. Despite press excitement that Siri was based on a DARPA-funded CALO project, many aspects of SHRDLU share similarities with Siri. The SHRDLU computer program was able to perform actions, ask questions to clarify any ambiguous requests, recall the history the conversation, learn new terminology.

Below is a transcript of a conversation used in SHRDLU demo:

Person: PICK UP A BIG RED BLOCK.

Computer: OK. (does it)

Person: GRASP THE PYRAMID.

Computer: I DON'T UNDERSTAND WHICH PYRAMID YOU MEAN.

Person: FIND A BLOCK WHICH IS TALLER THAN THE ONE YOU ARE HOLDING AND PUT IT INTO THE BOX.

Computer: BY "IT", I ASSUME YOU MEAN THE BLOCK WHICH IS TALLER THAN THE ONE I AM HOLDING.

Computer: OK. (does it)

Person: WHAT DOES THE BOX CONTAIN?

Computer: THE BLUE PYRAMID AND THE BLUE BLOCK.

Person: WHAT IS THE PYRAMID SUPPORTED BY?

Computer: THE BOX.

Person: HOW MANY BLOCKS ARE NOT IN THE BOX?

Computer: FOUR OF THEM.

Person: IS AT LEAST ONE OF THEM NARROWER THAN THE ONE WHICH I TOLD YOU TO PICK UP?

Computer: YES, THE RED CUBE.

Person: IS IT SUPPORTED?

Computer: YES, BY THE TABLE.

Person: CAN THE TABLE PICK UP BLOCKS?

Computer: NO.

Person: CAN A PYRAMID BE SUPPORTED BY A BLOCK?

Computer: YES.

Person: CAN A PYRAMID SUPPORT A PYRAMID?

Computer: I DON'T KNOW.

Person: STACK UP TWO PYRAMIDS.

Computer: I CAN'T.

Person: WILL YOU PLEASE STACK UP BOTH OF THE RED BLOCKS AND EITHER A GREEN CUBE OR A PYRAMID?

Computer: OK. (does it)

Person: WHICH CUBE IS SITTING ON THE TABLE?

Computer: THE LARGE GREEN ONE WHICH SUPPORTS THE RED PYRAMID.

Person: IS THERE A LARGE BLOCK BEHIND A PYRAMID?

Computer: YES, THREE OF THEM: A LARGE RED ONE, A LARGE GREEN CUBE, AND THE BLUE ONE.

Person: PUT A SMALL ONE ONTO THE GREEN CUBE WHICH SUPPORTS A PYRAMID.

Computer: OK. (does it)

Person: PUT THE LITTLEST PYRAMID ON TOP OF IT.

Computer: OK. (does it)

Person: DOES THE SHORTEST THING THE TALLEST PYRAMID'S SUPPORT SUPPORTS SUPPORT ANYTHING GREEN?

Computer: YES, THE GREEN PYRAMID.

Person: WHAT COLOR IS IT?

Computer: BY "IT", I ASSUME YOU MEAN THE SHORTEST THING THE TALLEST PYRAMID'S SUPPORT SUPPORTS.

Computer: RED

Person: HOW MANY THINGS ARE ON TOP OF GREEN CUBES?

Computer: I'M NOT SURE WHAT YOU MEAN BY "ON TOP OF "IN THE PHRASE "ON TOP OF GREEN CUBES " .

DO YOU MEAN:

1 - DIRECTLY ON THE SURFACE

2 - ANWHERE ON TOP OF ?

Person: 2

Computer: THREE OF THEM.

Person: HAD YOU TOUCHED ANY PYRAMID BEFORE YOU PUT THE GREEN ONE ON THE LITTLE CUBE?

Computer: YES, THE GREEN ONE.

Person: WHEN DID YOU PICK IT UP?

Computer: WHILE I WAS STACKING UP THE RED CUBE, A LARGE RED BLOCK AND A LARGE GREEN CUBE.

Person: WHY?

Computer: TO GET RID OF IT.

Person: WHY DID YOU DO THAT?

Computer: TO CLEAN OFF THE RED CUBE.

Person: WHY DID YOU CLEAR OFF THAT CUBE?

Computer: TO PUT IT ON A LARGE GREEN CUBE.

Person: WHY DID YOU DO THAT?

Computer: TO STACK UP THE RED CUBE , A LARGE RED BLOCK AND A LARGE GREEN CUBE.

Person: WHY DID YOU DO THAT?

Computer: BECAUSE YOU ASKED ME TO.

Person: HOW DID YOU DO IT?

Computer: BY PUTTING A LARGE RED BLOCK ON THE TABLE ; THEN LETTING GO OF IT ; THEN PUTTING A LARGE GREEN CUBE ON IT ; THEN LETTING GO OF THAT CUBE ; THEN PUTTING THE RED CUBE ON THAT CUBE; THEN LETTING GO OF THAT CUBE.

Person: HOW MANY OBJECTS DID YOU TOUCH WHILE YOU WERE DOING IT?

Computer: FOUR OF THEM.

Person: WHAT DID THE RED CUBE SUPPORT BEFORE YOU STARTED TO CLEAN IT OFF?

Computer: THE GREEN PYRAMID.

Person: THERE WERE FIVE BLOCKS TO THE LEFT OF THE BOX THEN.

Computer: NO, ONLY FOUR OF THEM: THE RED CUBE, TWO LARGE GREEN CUBES AND A LARGE RED BLOCK.

Person: PUT THE BLUE PYRAMID ON THE BLOCK IN THE BOX.

Computer: OK. (does it)

Person: IS THERE ANYTHING WHICH IS BIGGER THAN EVERY PYRAMID BUT IS NOT AS WIDE AS THE THING THAT SUPPORTS IT?

Computer: YES, THE BLUE BLOCK.

Person: DOES A STEEPLE

Computer interrupting: SORRY, I DON'T KNOW THE WORD "STEEPLE".

Person: A "STEEPLE" IS A STACK WHICH CONTAINS TWO GREEN CUBES AND A PYRAMID.

Computer: I UNDERSTAND.

Person: ARE THERE ANY STEEPLES NOW?

Computer: NO.

Person: BUILD ONE.

Computer: OK. (does it)

Person: CALL THE BIGGEST BLOCK "SUPERBLOCK".

Computer: OK.

Person: HAVE YOU PICKED SUPERBLOCK UP SINCE WE BEGAN?

Computer: YES.

Person: WHY DID YOU DROP IT?

Computer: BY "IT", I ASSUME YOU MEAN THE SUPERBLOCK.

I CAN'T EXPLAIN A NON-EXISTENT EVENT.

Person: IS THERE ANYTHING TO THE RIGHT OF THE RED PYRAMID?

Computer: YES, FOUR THINGS: THE BOX , THE RED CUBE, THE BLUE PYRAMID AND THE BLOCK NAMED SUPERBLOCK

Person: THANK YOU.

Computer: YOU'RE WELCOME!

Getting Back to Blogging

It’s been a long while since I blogged. It isn’t good to blog when angry or exhausted, and the quality of my posts have been in decline. The blog had become a liability for me. I basically forced a bunch of posts in the past few years, but I plan to reintroduce myself and the company that I am developing. I have a lot of material for blog posts that I have accumulated over time.

Isn’t it hard to believe that I have not had an outside job for almost a decade, just working seven days a week building my own software company?

I don’t have much of life; I don’t go out, date, or party. I live in the Seattle area, but I haven’t been in the city of Seattle for the several past years.

What do I do?

• I develop software 7 days a week at home, most of the day.
• I listen to university lectures and take online courses to stay educated.
• I read academic papers and textbooks, and books on software and business.
• I stay current on technology…. I learned to write iOS and WP7 applications, learned about Cloud programming.
• I follow news from various sites (hacker news—a site for software entrepreneurs) to stay current
• I attend both software and business conferences.
• I play chess at Crossroads Mall once a month. I watch movies once a month.
• I am training to run in a race at the end of October. I also bike and lift weights to stay fit.
• I go on vacation for about two weeks a year to see my family.

What kind of software do I build?

Basically, desktop software with some kind of AI. My software consists of the following:

• Application/UI framework
• Layout engine.
• Symbolic AI engine.
• DSLs.

I have written and rewritten hundreds of thousands of lines of code. Six years ago, I compared the amount of code that I had rewritten to the open-source version of the .NET libraries, and my line counts were about the same.

I tell my parents that once I ship my first software, that the others will follow extremely quickly, because my designs allow for reuse rate in the high 90s. That I never intended to release just one product but to enable many to be release in quick succession.

I have designed my software, so that it would be easy to port to a different platform—Mac versus Windows versus Web and Tablet versus Phone versus Desktop.

To speed development, I also rely on the following:

• Open source (both code and databases)
• Third party component libraries
• Licensed data and icons
• Run-time and compile-time code generation.
• Declarative programming like DSLs and value-centric (quasi-functional) programming
• C#

iPhone Launch Predictions

I often try to predict new features before product launches and developer conferences such as last year’s PDC, the iPad2 launch, the Google I/O conference, Apple’s WWDC keynote, Amazon Fire announcement, and Microsoft’s Build conference last month. These predictions are based on rumors, leaks, and an overall assessment of a company’s product strategy. Then, I assess my performance against actual announcements. These previous predictions would have made good blog material.

Apple’s launch of the iPhone 5 or 4S (or both) is just 10 hours away. Here is my try for the iPhone 5 launch, but first some preliminaries.

Background

This year has been very secretive with Apple delaying announcement of new iOS features by two to three months and by shifting the calendar for the iPhone launch to give less time for the competition to match, respond, or neutralize Apple advantages as in year’s past. The new features that were announced were mostly competitive response (notifications, i-messaging) or new business avenues such as subscriptions, though some like iCloud represent a major strategic milestone for the company. The iOS announcement was underwhelming given expectations for truly new innovations. Apple, given its secretive nature, likely reserved the most important announcements for the actual launch so as not tip its hand too early.

Google has also become very secretive with new features in upcoming Android versions. The announcements at Google I/O centered around competitive response such as matching Apple’s content prowess with Google Movies and Google Music, expanding the hardware ecosystem (Google @ Home and Google open accessory) against AirPlay and AirPrint, and getting hardware manufacturers to issue timely software updates. Google will be formally announcing the next version of Android next week.

Meanwhile, Microsoft’s own plan to bring Windows Phone into parity has come into fruition (no pun intended) with the recently released Mango, version 7.5 of the operating system. Unfortunately, for Microsoft, Google and Apple have yet to release their cards and they may still announce game-changing advances.

I fully suspected a year ago the convergence of Windows Phone 8 and Windows 8. Convergence implied that the Windows 8 would be ported to ARM, that Windows would incorporate the Metro interface, and that a future version of Windows Phone would abandon the Window CE and run on top of the Windows 8 codebase. My reasoning was based on the branding of the Windows Phone, the restriction of Windows Phone to managed libraries (Silverlight and XNA) which may not have to be recompiled to run on desktop Window, and later the version numbering (7.5) of Mango (Mango is substantial enough that it should really have been a full version number upgrade—much more so than with previous versions of Window Mobile). I was taken aback by new editions of the Windows Mobile/Embedded in summer of 2010, but the announcement of Windows being ported to ARM and Sinofsky’s “System on a Chip” presentation at CES charting the rapid advances in mobile hardware in memory and processor speed towards eventual parity with desktop systems all but confirmed my view.

It makes sense strategy. Mobile operating systems were becoming increasingly competitive and posing a threat to Windows. Having two separate operating systems, one for mobile and one for desktop, is wasteful, splitting resources and duplicating efforts. The Windows CE experiment resulted in APIs that were incomplete and inconsistent with that of the full Windows; in .NET CE, for example, the garbage collector uses a different algorithm and only a tiny subset of the graphics capabilities of Windows are supported (gradients and transforms fail). With one OS, Windows Phone will always have the full functionality that Windows offers.

The recent unveiling of Microsoft Windows 8 tablet strategy potentially represents a change in Microsoft fortunes in mobile computing. Windows 8 offers a no-compromise hybrid solution featuring the Metro and classic Windows interfaces, which seem to offer a compelling value proposition that Google (and possibly Apple) won’t be able to defend against. However, it is important to note that both Android and iOS are forked from serious, proven desktop operating systems—Linux and Mac OS X, respectively—and both companies develop Office-compatible desktop applications. Apple has already begun converging the user interface with Mac OS X “Lion” and could be converging the underlying OS for a hybrid tablet/laptop solution as well; there are rumors of an ARM based version of a future Mac laptop, and iPhone applications already run in an x86-based iPhone simulator. Likewise, Google is seeking to converge Chrome OS with Android, though there is still the possibility of using a traditional Linux desktop or a 3D desktop interface from the BumpTop acquisition.

Predictions:

In making my predictions, I generally follow this path.

First, I take into account the product development cycle. Features are selected and specified early in the product cycle, then scheduled, developed and later tested before going out to beta and eventually being released. This means that new features are most likely designed based on the competitive market landscape at the time of the last product launch. Thus, it is dangerous for a product cycle to be too long in a fast-moving market. The new features in iOS3 addressed a number of features in Blackberry such as cut-and-paste and unified inboxes, but ignored multitasking support offered by Palm and Droids. iOS4 later added multitasking.

Second, I take into account the company’s DNA and strategy. Microsoft spends $9 billion in R&D, Google spends half as much as Microsoft (about $4 billion), and Apple spends half as much as Google (about $2 billion). Microsoft is spread thin over a variety of products, but not as much as before. Its product cycles are too long. Both Google and Apple have fewer products to spread the wealth through. Part of Google’s speed and efficiency extends to its reliance on open source. Google doesn’t sell products, just advertising, so it has less exposure to copyright infringement. Another part may be due to its web-based applications, which have fewer deployment issues. Apple is very focused. Its efficiencies stem from using open source for some products like Safari, vertical integration between software and hardware (to reduce testing costs), and ready abandonment of legacy technologies. Apple also migrates technologies back and forth between iOS and Mac OS such as a common subset of the operating system. I think we will see gesture-based home buttons for instance. Apple is extraordinarily efficient with its R&D dollars. It doesn’t waste dollars on low-value features and it’s always looking for new business models (advertisings, content stores, subscriptions, hardware).

Third, I look at acquisitions and other actions to decipher (“read between the lines”) the company’s strategy. For instance, branding WP7 with the “Windows” name only make sense in a convergence strategy. If an action doesn’t make sense, it’s probably because the company’s strategy has not been revealed, or the company is buying time after a major product delay.

Fourth, I also look at rumors and leaks.

Here are my predictions:

It is widely reported that the carriers are AT&T, Verizon and Sprint. T-Mobile is left out. The hardware is redesigned with a larger, 8MP camera, edge-to-edge display, thinner profile, tapered edges, world phone, and faster chips. NFC is likely; I don’t imagine it being difficult to add. A Blackberry-like LED notification light is also suggested by an iOS setting. LTE may still be in waiting, though there has been some evidence of its arrival with LTE equipment at Apple stores and it could very well have been the cause of the iPhone delay. iCloud will be another star in the announcement.

Voice Assistant. This is based on speech recognition technology licensed from Nuance with the additional capabilities from the SiRi acquisition in April 2010. This will include dictation and voice control. Some rumors indicate a dialogue and the ability to perform searches around local activities.

Home Button Gestures. The use of an oval capacitive home button is wildly reported. The oblong button is probably for multitouch gesture support (primarily sliding) because the existing home button is already overloaded for many functions. It would combat the perceived advantage of multi-button Android and Windows Phone devices. It would offer unlimited functionality through the home button while retain the simplicity of one button. The edge-to-edge display prevents the use of the bezel gestures. A gesture-based home button is an example of Apple reusing a feature from its other touch-based input devices (trackpads, mice).

FaceBook integration. When iOS 5 was announced with Twitter integration, it was widely assumed this was due to a feud between the two companies. FaceBook integration could have been so important that it was help up for the launch to build excitement. Twitter and FaceBook integration had been leaked a year earlier. The Twitter-only announcement might have allowed for developer testing, and the actual integration of both FaceBook and Twitter may be more comprehensive a la Windows Phone. (Reportedly, the FaceBook iPad app is being held up for the Apple announcement.)

Face Recognition. Face recognition could be used for authentication and photo-tagging. Apple acquired Polar Rose earlier for its face recognition technology, which has probably seen use already in Apple’s iPad app, Photo Booth.

Camera. The camera application in iOS 5 already introduces a number of new features. The camera is reported to play a major part of the announcement. There have also been leaks about support for panoramic photos. My own hope is that the Camera uses gyroscopes for image stabilization and image deblurring. I think gyroscopes have been used for the high dynamic range shots in iOS 4.1, since multiple snapshots have to be coordinated spatially, but theoretically gyroscopes could extend the effective megapixels and zoom level of a camera by utilizing multiple shots. I investigated creating a camera app that used the gyroscopes to create a better binocular application.

Mapping. Mapping advances were notably absent from the improvements to iOS5 in the developer conference last April except for the addition of printing to the Maps applications and some notable API support such as geocoding. There have earlier been rumors of new turn-by-turn technology and Apple’s statement after the geolocation privacy fracas mentioned the crowdsourcing of traffic data in the indefinite future. The statement was unusual as Apple typically prefers not to mention features too early. Apple acquired PlaceBase in summer of 2009 and has been hiring software engineers to bring its mapping technology “to the next level” since 2009; the results of these efforts have yet to be realized. On the other hand, Apple might not improve Maps as Apple extended a license agreement with Google in June.