Reading Recipes with Siri or Apple Watch

I’m finally learning to cook some decent food, so I need to be able to read recipes. For a while, I was reading them out of Simplenote on my iPhone. However, I encountered several frustrations with this approach (and this applies to any notes or text app really):

  • When you’re not editing, Simplenote shows the note such that each line is an item for VoiceOver; i.e. you flick right to read the next line. However, if you bump the screen or perform the wrong gesture accidentally, you can easily lose your position. If the screen locks or you have to switch apps, you lose your position completely, since VoiceOver doesn’t restore focus to the last focused item in apps.
  • When editing, you can review the note line by line using the rotor. The advantage here is that the cursor doesn’t get lost when you switch apps or the screen locks. However, lines can be smaller than is ideal due to the screen size, so one recipe instruction might get split across multiple lines. Also, moving the editing cursor with VoiceOver is notoriously buggy, often getting stuck, etc. Finally, again, if you bump the screen or perform the wrong gesture, you can lose your position (or worse, accidentally type text into the document).
  • The screen lock problem could be solved by disabling auto lock, but that obviously has an impact on battery.
  • Having to repeatedly take my phone out of my pocket to read the next instruction was impractical, especially given the risk of losing my spot in the recipe. Leaving it on a bench somewhere meant I had to keep walking back to wherever my phone was located, which was similarly tedious. This might seem simple enough, but when you’re moving around a lot, using your hands for other things, getting your hands dirty, etc., it just isn’t efficient.

I considered a couple of solutions:

Read more

My quest for a New Mobile Phone

Disclaimer: This is primarily based on my own personal experience. Also, this all happened last year, so some of the finer details are a bit vague now.

Early last year, my trusty 6 year old Nokia 6600 was finally starting to die and I decided it was past time to move on. (Amusingly, that phone even survived being accidentally dunked in a glass of wine.) My aim was to satisfy all of my portable technology needs with one device, including phone, email, web browser, audio player (standard 3.5 mm audio socket essential), synchronisable calendar, synchronisable contact manager, note taker, ebook reader and portable file storage. And so the quest began.

Making the (First) Choice


My ideal mobile platform was Android. Aside from satisfying all of my needs, it is an open, modern platform. Unfortunately, although I seriously entertained the idea, a great deal of research and playing with the Android emulator led me to realise that Android's accessibility was at an unacceptably poor state for me.

I considered the iPhone. I've written about the iPhone's out-of-the-box accessibility to blind people before and have played with an iPhone several times since. Although I was pretty impressed, especially by the fact that VoiceOver comes out-of-the-box, there were two major problems with the iPhone for me. First, I dislike the closed nature of the iPhone environment, commonly known as the "walled garden" or "do things Apple's way or not at all". Aside from the principle (you all know I'm a big advocate for openness), I would be unable to play ogg vorbis (my audio format of choice) in iPod, I would have to transfer music and files with iTunes (which I detest), I couldn't use it as a portable file storage device, and I would be limited to apps that Apple permitted (unless I wanted to jailbreak). Second, I wanted a device with a physical keyboard. In the end, I decided against the iPhone.

I briefly considered another Symbian Series 60 phone. However, based on past experience (both mine and others'), I didn't think I would be able to play audio and use the screen reader simultaneously, which immediately disqualified it for me, although I've since been informed that is no longer true on some newer phones. I also feel it is a dying platform. There are probably some other reasons i discounted it, but i can't remember them now. If nothing else, I wasn't entirely happy with it and wanted a change.

Finally, I settled on Windows Mobile, specifically the Sony Ericsson Xperia X1, with Mobile Speak. I guess Windows Mobile is a dying platform too, but at the time, I felt it was perhaps less so and provided more functionality for me. While the operating system itself isn't open, you can develop and install apps as you please without being restricted to a single app store. There are several ogg vorbis players for Windows Mobile. I also had the option of buying Mobile Geo for navigation if I wanted to later. I was warned by someone who played with an older phone that Mobile Speak on Windows Mobile was fairly unresponsive, but unable to test this myself, I hoped that it might be better with a less resource intensive voice such as Fonix and/or a newer phone or that I'd get used to it.

Frustration, Pain and Misery


A bit less than $800 later, I got my new phone and Mobile Speak in June. I expected and accepted that it would take me some time to get used to it. I loved finally being able to access the internet from my phone and play audio through proper stereo headphones. However, despite this, the next few months were just downright painful and frequently induced rage bordering on violence. I often had the urge to throw my new phone across the room several times a day.

Primarily, this was due to Mobile Speak. I found it to be hideously unresponsive, unstable, unreliable, inconsistent and otherwise buggy as all hell.
  • The unresponsiveness proved to be unacceptable for me, often taking around half a second to respond to input and events, even using Fonix. Aside from the general inefficiency this caused, this made reading text incredibly tedious, and despite the physical keyboard, typing was painful due to the slow response to backspace and cursor keys.
  • Mobile Speak crashed or froze far too often and there was no way to resurrect it without restarting the phone.
  • In Internet Explorer, working with form controls was extremely inconsistent and unreliable, especially multi-line editable text fields. Quick navigation (moving by heading, etc.) was very slow and failed to work altogether in many cases. On my phone, Google services (including Google Search, even the mobile version, of all things!) refused to render at all.
  • I encountered problems when reading email as well. Sometimes, Mobile Speak wouldn't render emails. Others, it wouldn't let me navigate to the headers of the email, which is essential if you want to download the rest of a message that hasn't been fully downloaded.
  • Reading text in Word Mobile was even slower than everywhere else, which made reading ebooks infeasible.
  • Braille display scrolling is either broken or unintuitive. On my 40 cell display, Mobile Speak only seemed to scroll half the display and I couldn't find a way to change this.
  • Definitely quite a few other bugs I can't remember all of the details about...
It's worth noting that I'm not saying that this is all entirely Mobile Speak's fault. I suspect Windows Mobile and other applications may play a part in this dodginess.

I had two other major gripes with Mobile Speak.
  • Despite years of experience with screen readers, I found the Mobile Speak commands, especially the touch interface, to be tedious and difficult to learn. The touch interface is inherently slow to use because you need to wait after certain taps to avoid them being construed as double or triple taps.
  • Mobile Speak's phone number licensing model was a major annoyance for me when I went overseas for a few days. Mobile Speak allows you to use it with a SIM card with a different number for 12 hours, but you have to reinsert the original SIM card after 12 hours if you want Mobile Speak to continue functioning as a licensed copy. Also, I seem to recall that this also applied if the phone was in airplane mode.


There were other things that irritated me about my phone and its applications.
  • I found Windows Mobile in general to be very sluggish. Even something as fundamental to a phone as dialling on the phone keypad or adjusting the volume was incredibly laggy, sometimes taking several seconds to respond to key presses.
  • Far too many apps, including Google Maps and both of the free ogg vorbis players I tried, had significant accessibility problems.
  • Windows Media doesn't have support for bookmarking, which made reading audio books infeasible. There was a paid app that provided this and other functionality i wanted, but i wasn't willing to pay for it in case I discovered it too had major accessibility problems.
  • Windows Mobile doesn't have enough levels on its volume control.
  • If the phone is switched to silent, all audio is silenced, including Mobile Speak.
  • Internet Explorer doesn't support tabbed browsing!
  • Windows Mobile only supports one Exchange Active Sync account, which meant I couldn't maintain separate personal and work calendars.
  • More...


The Snapping Point


In the end, after less than 6 months, I just couldn't take it any more. I tried to learn to live with it for at least a couple of years, as I'd already spent so much money on it, but it was truly unbearable. It's worth noting that a close friend of mine had a very similar experience with Windows Mobile and also gave up in equivalent disgust around the same time. It particularly angers me that I paid $315 for a piece of software as buggy as Mobile Speak. I started playing with Jen's iPhone a bit more, and finally, I gave in and got my own.

The iPhone: Peace at Last


For reasons I mentioned above, I felt like I was going to the dark side when I made the decision to switch to the iPhone. Among other things, it's a bit hypocritical of me, given my belief in and advocacy for openness. Nevertheless, I have not looked back once since I got it. It has truly changed my life.

The in-built accessibility of the iPhone is amazing. I strongly believe that accessibility should not incur an extra cost for the user and Apple have done just that. VoiceOver is very responsive. Usage is fairly intuitive. All of the in-built apps and the majority of third party apps are accessible. Once you get used to it and start to remember where things are on the screen, navigating with the touch screen becomes incredibly efficient. The support for braille displays is excellent; I can see this being very useful next time I need to give a presentation. The triple-click home feature means that I can even toggle VoiceOver on Jen's phone when needed, which has been really useful for us when she is driving and needs me to read directions.

I still hate iTunes, but thankfully, I rarely have to use it. I manage music and other audio on my phone using the iPod manager component for my audio player, foobar2000, which is excellent and even transcodes files in unsupported formats on the fly. The iPod app is great, supporting gapless playback for music and automatic bookmarks for audio books and podcasts.

Other highlights:
  • Very nice email and web browsing experience.
  • Push notifications for mail, Twitter, Facebook and instant messaging.
  • Multiple calendars.
  • Skype on my phone, which is far nicer than being tied to my computer when on a Skype call.
  • Voice control, which works well most of the time.
  • Smooth reading of ebooks using iBooks.


Resultant to all of this, I find I spend far less time in front of my computer outside of work hours. Also, when I'm away from home, even on holiday for a week, I often just take my phone. Previously, I had to take my notebook almost everywhere.

Like all things, the iPhone isn't perfect. I still dislike the walled garden, and I have to live with transcoded audio and can't use my iPhone as a USB storage device because of it. I am definitely slower at typing on the iPhone than I was on the numeric keypad on my old Nokia 6600, although perhaps surprisingly, I'm probably faster on the iPhone than I was on the Xperia X1. There are definitely bugs, some of which I encounter on a daily basis and are very annoying.

Even so, I love the iPhone. I'm willing to make some sacrifices, and I can live with bugs that are at least consistent and easy enough to work around. On the rare occasions that VoiceOver crashes, I can easily restart it with a triple click of the Home button. I wish I'd gone for the iPhone in the first place and not wasted so much money on the Windows Mobile solution, but ah well, live and learn.

Touch Screen Keyboard Difficulty for the Blind: A Potential Solution

There's been a lot of talk about VoiceOver on the iPhone 3G S... and so there should be. I suspect (and others seem to concur) that Apple's model for touch screen accessibility ought to be quite useable under most circumstances, but that typing on the touch screen keyboard could be rather challenging, perhaps the most challenging aspect of the interface.

Thinking about this, I had a brainwave: perhaps a Braille touch screen keyboard could be a nicer solution for blind users. A Braille keyboard doesn't require you to shift your fingers horizontally or vertically, which eliminates the challenge of finding characters. Instead, you only need to raise or lower your fingers in various combinations. I'm guessing that the multi-touch touch screen should be able to detect these combinations. I can't remember how big the touch screen on the iPhone is, but I suspect it is probably large enough to allow for the fingers to be placed in Braille typing formation. I can't type as fast on a braille keyboard as I can on a QWERTY keyboard, but I'm a hell of a lot faster on a Braille keyboard than on a phone keypad. In fact, I'm wondering whether this method of input would actually be faster (assuming a proficient Braille typist) than a sighted user using the normal touch screen keyboard.

Surely I'm not the first to wonder about this... or is this really a new idea?

New Apple iPhone Accessible to the Blind: My Thoughts

Apple have just announced that the soon-to-be-released iPhone 3G S will include the VoiceOver screen reader, among other accessibility enhancements. Although I'm still uncertain as to the efficiency of a touch screen interface for the blind, this is fantastic news. For the first time, an every day mobile phone/PDA will include a screen reader as part of the core product at no extra cost. Apple's inclusion of VoiceOver into Mac OS X was revolutionary news, and they've now done it again with the iPhone. I'm sometimes rather cynical towards Apple, but I am continually impressed by their commitment to incorporating out-of-the-box accessibility into their products.

It's worth noting that the iPhone is not the first touch screen phone to include accessibility for the blind. A suite of self-voicing applications called Eyes-Free is available for Android which enables blind users to use many functions of the phone. While this is great to see, it's disappointing that this is what I call isolating (or isolationalist) accessibility; i.e. blind users must use a different set of applications to everyone else to access the phone and are thus isolated from the experience of other users. This appears to be a (in my opinion disappointing) trend for some sections of Google, as demonstrated in the separate, so-called "ARIA enhanced" and "accessible" versions of some of their services, wherein they often present a different interface for blind users instead of integrating accessibility right into the existing interface. I think that isolating accessibility certainly has its place - it can sometimes make for a more friendly and easier learnt interface and is the only practical option in some extremely visual scenarios - but in general, I believe it is extremely limiting and inflexible. Admittedly, a touch screen interface is inherently visual and thus presents a new set of challenges. Not only does the user interface need to be spoken or brailled, but the method of input needs significant adaptation to be used by a blind person. This is probably why the Android Eyes-Free developers chose the path they did. However, Apple have taken a better, more generic approach similar to that of most modern operating systems, allowing blind users to use the same applications as everyone else. The new iPhone incorporates accessibility into the core of the operating system and VoiceOver modifies the input method as well as reading the user interface, which allows any application to be accessible with VoiceOver, including all of the in-built applications.

Experience will determine whether Apple's implementation is optimal for blind users. Regardless, they've truly raised the bar for mobile accessibility.

Disclaimer: I have not had any personal experience with either VoiceOver on the iPhone 3G S or Android Eyes-Free. These thoughts are solely based on the information I have gleaned from various internet sources.

iPhone Ocarina

Okay. So some of you will know that I'm a bit (?) cynical of the iPhone. This is partly because it seems like yet another flashy new piece of largely closed, proprietary technology with lots of over-exaggerated hype. I've always been cynical of a lot of Apple stuff in general because of the huge number of people that see Apple as gods who can do no wrong. In truth, I think my cynicism towards the iPhone is mostly due to the fact that it has a touch screen which I can't use. Yeah, yeah, I know - I'm a sore loser. :) Anyway, despite my cynicism, even *I* was wowed reading this article about an iPhone application called Ocarina. From the article:

Once you install and open this program, your iPhone's screen displays four colored circles of different sizes. These are the "holes" that you cover with your fingers, as you would the holes on a flute. Then you blow into the microphone hole at the bottom of the iPhone, and presto: the haunting, expressive, beautiful sound of a wind instrument comes from the iPhone speaker.
Different combinations of fingers on those four "holes" produce the different notes of the scale. (You can change the key in Preferences--no doubt a first on a cellphone.) Tilting the phone up or down controls the vibrato.

Now *that* is pretty damned cool. I haven't actually played with it myself, but I'm sure it could keep me amused for a while... assuming I could figure out where to put my fingers. :)