Alright, gather ‘round, aspiring communication chefs! We’re about to embark on a culinary adventure – one where we cook up something genuinely useful for the world of Augmentative and Alternative Communication (AAC). This post dives into how we built tools to automatically correct sentences – specifically tackling tricky issues like typos and words mashed together without spaces. My goal here is to explain this in a way that makes sense, even if you’re not a tech wizard! I’ve gone with a cooking analogy. Now those of you know me know I love an analogy but also know that most of the time they are awful. This maybe one of them and I may take this a bit too far in this post..

What we did: We explored two main paths to auto-correction, like choosing between baking a cake from scratch or using a fancy pre-made mix:

1. Fine-tuning a Language Model: We took a small but powerful AI model (called t5-small) and taught it to correct noisy sentences. Think of this as perfecting a secret family recipe.
2. Developing Algorithmic Code: We also wrote specific computer code designed to fix spelling errors and add spaces where words were run together. This is more like mastering a basic, reliable cooking technique.
3. (Slightly less exciting but..well spoiler alert: the fastest and “best” using an online Large language model)

This project was built some time ago, and while some data might be a bit outdated, the core ideas are still super relevant. You can find all the code at https://github.com/AceCentre/Correct-A-Sentence/tree/main/helper-scripts.

A quick note: If you’re looking for a current solution for an existing AAC system, Smartbox’s Grid3 “Fix Tool” is worth checking out. My work here was done before that tool existed, and it’s why you won’t see me rebuilding our model today!

Hold up. Before we continue if you do want to do this I recommend two things! 1. DONT RUN MY CODE but look at it by all means and be critical of it! 2. Let me know of improvements but 3. DON’T GET YOUR LLM TO MAKE THESE SCRIPTS FOR YOU. You’ll learn heaps more if you do it yourself. I say this with a recent history of geting LLM’s churning out code. Do the harder thing. Your brain will thank you.

The Core Ingredient: Why Automatic Correction Matters for AAC

AAC systems need to do a lot of things but ultimately for an end user they need to allow fast ways of getting your thoughts out from what you are trying to say - and understandable ones at at that. This isn’t just my hunch; it’s what we consistently hear from users, staff, and project feedback. The key term here is efficiency.

Sidenote: Efficiency in AAC isn’t just about the output. It could be speedier services, better quality support, or even device improvements like smarter input detection, but just to be clear for this post we are focusing on efficiency in output

The challenge often comes from “noisy” input – things like typing fast, or facing physical access challenges. Auto-correctionis our secret weapon for cleaning up this messy input and helping users get their message across more effectively.

Unlike prediction (where the system tries to guess what you’ll type next), auto-correction actively fixes mistakes afterthey’ve been made. Think about your phone – it’s constantly correcting your typos before you even see them. This happens seamlessly in the background, making it feel like your typing is perfect! (And you really notice it when it doesn’t work!) So, why isn’t this standard in AAC, where mistakes are equally common?

Approach 1: Building Algorithmic Correction Tools (Baking Our Own!)

When creating an AAC correction tool, it’s helpful to first understand the common “noisy” sentence types. These are the kinds of input issues we need to fix:

• Ihaveaapaininmmydneck (No spaces and typos – a double whammy!)
• I wanaat a cuupoof te pls (Double key presses – sticky keys, anyone?)
• u want a oakey of cjeese please (Hitting nearby keys – positional errors)
• can u brus my air (Missing letters/deletions – vanishing letters!)
• Can you help me? Can you help me? (Repeated phrases – when a stored message goes wild!)

These sentences are typically short. While word prediction can help create perfect sentences, it often requires significant visual scanning and mental effort, which isn’t always ideal for AAC users.

Sidenote: We actually don’t know this for sure. There are some papers that document these types of errors but in short we don’t have a great grip on what is actually written by AAC users.. more on that later..

Tackling Words Without Spaces and Typos (Writingwitjoutspacesandtypos)

This is one of the trickiest problems. How do you separate words when they’re all jammed together?

Well turns out if you know some python code there is a clever library called wordsegment. This tool uses information about how common single words (“unigrams”) and two-word phrases (“bigrams”) are to figure out where words should be. Here’s an example:

from wordsegment import load, segment  
load()  
segment('thisisatest')  

# Output: \['this', 'is', 'a', 'test'\]

It’s very effective! However, language is deeply personal. If wordsegment doesn’t know a user’s unique slang or abbreviations (like “biccie” for biscuit), it will struggle. For example, Iwouldlikeabicciewithmycuppatea might become ['i', 'would', 'like', 'a', 'bi', 'ccie', 'with', 'my', 'cuppa', 'tea'].

Truth is though, even with slight imperfections, the meaning is often still clear, especially if the user is communicating with a familiar partner. Sometimes, “good enough” is perfectly acceptable! In AAC, “co-construction” – where communication partners work together to understand – is super important. It happens all the time and you need to imagine any AAC system doesn’t work without it. We don’t always need to achieve perfect output.

Adding Our Secret Family Ingredients (Personalizing Algorithmic Tools with User Data!)

Imagine if we could incorporate a person’s actual vocabulary and common phrases into wordsegment’s data. This would significantly improve its accuracy for that specific user. We could take a user’s real-life language and use it to enhance the tool’s understanding, like using your grandma’s secret spice blend!

The wordsegment library allows for this kind of customization. When we did this work we even created specific “typo-heavy” two-word phrases based on real typing errors to make it smarter. This is a solid starting point, though capturing every personal nuance is challenging.

Another option is a “fuzzy” search approach. This method is light on memory and doesn’t require powerful computer graphics cards (GPUs), making it suitable for running directly on a device. While it might not be lightning-fast, it could be “quick enough” for many situations. However, it still faces challenges with acronyms, names, and highly unusual abbreviations.

Approach 2: Customizing Large Language Models (LLMs) (The Fancy Pre-Made Sauce)

What about the powerful Large Language Models (LLMs) like OpenAI’s GPT-3.5 Turbo? Let’s try wrapping our noisy sentences in an OpenAI GPT-3.5 Turbo API and see what happens:

The results are impressive! You can immediately see the benefit: users can focus on typing without being distracted by predictions. This could dramatically improve communication speed.

However, a key consideration with online LLMs is privacy. When communication data is sent to a cloud service, it raises questions about how that data is handled and to be fair some users - and providers of AAC are nervous of this. While some services offer opt-out options for data being used for model training, the notion of personal communication data residing in the cloud is something to address carefully.

It’s fair to say that whether “sending data to the cloud” is “unacceptable” is a nuanced point. Many people routinely use cloud services for sensitive personal information (like email, documents, or medical records), and some are increasingly comfortable sharing personal data with online AIs. While some AAC users will undoubtedly prefer to keep all their communication entirely offline, others may not find it unacceptable, especially if there are clear privacy controls in place.

The primary privacy considerations often boil down to:

• User Comfort and Perception: Users might be uncomfortable with the technology if they perceive the privacy risk to be greater than it actually is, or if they don’t trust the AAC providers to have robust privacy controls.
• Provider Controls: The risk of poor privacy or security policies by an individual AAC provider, which could lead to chat data being logged, stored, or mishandled in the cloud.
• Security Measures: The risk of security breaches providing unauthorized access to chat data flowing through APIs.

These risks can and should be mitigated through robust security measures and strong privacy controls, such as not logging or storing chats unnecessarily, and avoiding associating them with user-specific IDs in API calls. If an AAC provider implements these correctly, the benefits of advanced online LLM capabilities might outweigh the concerns for many users, even if it doesn’t allay everyone’s apprehension. But either way - it would be darned easier to explain to end users if data never left the device.

Therefore, for AAC, we need a solution that is:

• Efficient: Quick to cook!
• Privacy-focused: All in your kitchen, no external diners!
• AAC-text savvy: Understands our unique culinary style!

Training Our Own Language Model for Correction (The Holy Grail: Baking Our Own Language Model!)

Spoiler alert - This is what we want to make - note this video is all running on device

Building our own language model from scratch sounds daunting, but it’s more achievable than you might think! The fundamental ingredients are:

1. Data: A large collection of “noisy” sentences (like actual user input) alongside their perfectly “cleaned” versions. The more this data resembles real AAC communication, the better the model will perform.
2. Even More Data: Seriously, data is the most crucial ingredient.
3. Code: The “recipe” for teaching the model.
4. Computing Resources: Some time and a computer (preferably with a graphics card).

The biggest challenge is DATA. There’s very little publicly available “AAC user dialogue” data. What is “AAC-like text”? It’s a surprising gap in our industry’s research – we don’t have large datasets showing how people actually use AAC. Are stored phrases used all the time? What kind of unique language do users create? We’re often just guessing.

So, we have to “imagine” what AAC-like data looks like. Researchers like Keith Vertanen have made fantastic attempts with artificial datasets. We also believe “spoken” language datasets might be a good starting point, as AAC aims to enable “speaking” through a device. We looked at corpora like:

• BNC 2014 Spoken
• AAC Text
• Daily Dialog

The problem? These are mostly perfectly transcribed texts, lacking the glorious typos and grammatical quirks our correction tool needs to learn from. We needed truly “noisy” input.

Injecting Noise: Making Our Data Deliberately Messy

To make our pristine datasets messy, we artificially injected errors using tools like nlpaug. While this creates synthetic noise, we also sought out real typos from sources like the TOEFL Spell dataset (essays by English language learners). For grammar correction, we also looked at datasets like JFLEG Data and subsets of C4-200M which contain grammatically incorrect sentences with their corrected versions.

Our Grand Baking Plan: Data Preparation:

1. Take each spoken text corpus.
2. Inject typos into the sentences and strip out common grammar elements (like commas and apostrophes, which AAC users often omit).
3. Create three versions of each sentence for training:
   • With typos.
   • With typos and compressed (no spaces).
   • The correct, clean version (but still without spaces, to train for de-compression).
4. Also, incorporate a grammar baseline layer using the grammar datasets.

We then used a “text-to-text transformer” model (specifically, the t5-small model) to learn from this data. This type of model is designed to take one text input and transform it into another, making it perfect for correction tasks. We chose t5-small because it’s lightweight and efficient enough to run on a device.

You can find the script for preparing the data here (you’ll need to download the BNC2014 corpus separately). The training process took several hours on a decent GPU. The resulting model is available on Hugging Face.

The Taste Test: How Did Our Bake Perform?

Let’s look at how our different correction techniques performed on a set of 39 test sentences (which contained both compression and typos):

• Algorithmic Approach (Wordsegment + Spelling Engine): Took about 14 seconds to process the test sentences. This approach is surprisingly quick for what it does, and it has no memory issues. However, its major limitation is that it will fail if it encounters words not in its dictionary, or novel grammatical structures.
• Online LLM (Azure/OpenAI GPT Turbo 16K): Took an astonishing 13 seconds! This speed, even with an external network call, is remarkable.

Now, for our custom-trained models:

Here are some example results comparing the incorrect sentence to the correct version and outputs from various models:

Our custom-trained model (HappyWill) actually outperformed GPT in terms of average similarity! While its total time was longer because we ran it on a small cloud machine, it would be much faster on a local device – and, crucially, it keeps all data private.

However, it’s important to note the “inbuilt” algorithmic technique (the Wordsegment one). While its “accuracy” (meaning, how often it perfectly matched the exact expected output) was low for our specific test, its output was surprisingly readable, and it has no privacy or significant memory concerns.

You can try the inbuilt algorithmic correction yourself for a short while via our API:

curl -X 'POST' \
  'https://correctasentence.acecentre.net/correction/correct_sentence' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "text": "howareyou",
  "correct_typos": true,
  "correction_method": "inbuilt"
}'

Or access the API directly in a browser: https://correctasentence.acecentre.net (just remember to set "correction_method": "inbuilt").

So, the next crucial step in our culinary journey? We need to hunt down, or create, an even richer, noisier, and more authentic corpus of data. The quest for the perfect “ingredients” continues!

(Oh dear.. This cooking analogy really went too far… I do apologise)

Key Takeaways

From this project, we’ve identified a few critical points for developing communication tools:

• Understanding Language Models for Correction (How to Broadly Make a Language Model for Correction): We’ve shown how a custom, fine-tuned language model (like t5-small) can be developed to correct complex errors while keeping user data private and enabling on-device processing. It’s about tailoring the perfect sauce to your needs.
• Algorithmic Approaches Can Be Effective (Algorithmic Approaches May Be As Good In Some Circumstances): For specific problems, like adding spaces or basic spelling, simpler algorithmic approaches can be surprisingly readable and efficient, offering a viable alternative to complex AI models, especially when novel input isn’t a primary concern. This highlights the importance of matching the tool to the specific type of communication challenge – sometimes, a good old stew is just what you need!
• The Need to Understand User Corpora Better (We Need to Understand Better Users Own Corpora): Regardless of the approach, the effectiveness of any correction tool is heavily dependent on the training data. There’s a significant need for more authentic, “noisy” datasets reflecting how AAC users actually communicate. Truly understanding users’ unique vocabulary and communication patterns – their personal “flavor profiles” – is essential for building the most effective tools.

I hate having to do this. But I have.

I’ve made ANOTHER tool to load up your env vars in a terminal. Why? Because I had a mare getting direnv to work on my specific Windows machine. It was really due to the way our office locked down the way a users $PROFILE works - that is it puts everything on OneDrive and there was a ton of admin issues getting it all working correctly. So, stupidly, rather than fixing that properly I found it (more fun?) to build a cross platform simplifed direnv tool.

So its called envloader, FInd details here and how to use.

In short Let’s imagine your directory - or one above it - has a .env or a .envrc file. It might look like this.

MICROSOFT_TOKEN_TRANS="token-for-trans"
CONFIG_ENCRYPTION_KEY="RqliKRLnkQxTOzgx_molctcWXM_alqSaNYVxCTBjma0="
SIMPLE_VAR=hello123
JSON_VAR='{"name": "test", "value": 42}'

This is common syntax for dotenv and direnv, but it won’t work as-is across all terminals. You need a tool to convert it properly for your shell. direnv IS your tool of choice, but if you like me, you cant just get it working - or the numerous other tools out there like it and you spend too many hours fiddling to get them to work (because you have a Windows machine which has really tight admin rights on it) you could try this little tool I’ve made..

Install on Mac/Linux

brew tap willwade/envloade
brew install willwade/envloader/envloader

or Using Scoop

scoop bucket add envloader https://github.com/willwade/envloader
scoop install envloader

then in a terminal you can run envloader and press enter. It will find a .env or .envrc and convert the lines appropriately for your shell.

e.g.

envloader

Make your life a bit easier and write an alias in your env - details here

I haven’t gone to town building lots of neat integration tricks - and unlike direnv it DOESN’T autoload on entering/exiting a directory. But running ‘envload’ now does the trick just fine for me. Maybe for you too.

Following on from “Ask Advice, Not permission” See also “Monkey Management”.

Monkey Management ideas aren’t new (eg see this which quotes it as being from 1975). It’s basically describing the problem of how unsolved problems of employees are pushed upwards causing the “monkey” to jump on a manager’s back. This increases workload for a manager and makes focus difficult. This piece nicely discusses the issue in the view of Product management.

From Monkey Management:

Despite the complexity, monkey ownership continues for Product. It is your circus, but not your job to train monkeys.

Monkey management is effective when there is organizational complexity. It has two critical benefits. The first is time management and the second is creating high agency staff who are able to deal with problems autonomously.

If you understand whose back the monkey is on, you can understand the art of time management and delegation.

A comment from HN

The reason monkey keeps jumping around is because there are ZERO directly responsible owners of the delivery of the cross-functional outcome for the business.

In my company, we have PM, Sr. PM, Director of Product, VP of product - interfacing with Designer, Sr. Designer, Design manager, Sr. Design manager, director of design - interfacing with Eng 2, Sr Eng, Eng manager, Sr Eng manager, Eng director, Sr. Eng director, Eng VP.

Nobody can tell who owns the final decisions, decisions cannot be bubbled up, every management chain is only focused on their own goals. There is no decision-making structure at all. Inevitably projects get delayed or there are unaccounted issues. Then each management chain stack ranks their reports for not achieving goals - never once accepting that the empire structure never made any decisions when it was necessary.

The empire structure has to go. It is dysfunctional, doesn’t work, and only causes grief to everyone involved. Tasks are unnecessarily hard. It is easy to do. Just make your highest paid people directly responsible for outcomes. Give them the freedom to pull people from various org functions to get a project to success.

From Ask for Advice, Not Permission

The problem with permission is that you are implicitly asking someone else to take some responsibility for your decision. You aren’t inviting them to participate in its success — permission is hardly seen as a value adding behavior — but if it goes wrong you might end up involving them in the failure: “Hey, I asked that team and they said it was fine.

and

Advice, on the other hand, is easy. “Hey, I was thinking about doing X, what advice would you give me on that?” In this instance you are showing a lot of respect to the person you are asking but not saddling them with responsibility because the decision is still on you.

Its a great read. Whats interesting though is what happens when it goes wrong. From HN

“…and when someone asks for your permission (probably because you’re in the person’s management chain), one response could be: “you don’t need my permission, if you think it is a good idea after getting input, go for it. If it turns out to be a bad idea, share your learnings so we don’t repeat the same mistake.””

So sometimes.. just sometimes.. a couple of months works pays off. Or I guess we could frame this post around successful product development - keeping the end user the focus of your development and involving them at all stages of your development cycle is key. We are proud to shout loudly about this.

So two months ago we saw Judith. Technology has not been successful for her before. (You can read more why in my previous posts). So we rapidly developed a SwiftUI app that was unique in its operation - you dragged from one letter to the next. Trying it with Judith we still had a problem - positional errors and missing letters were high. So we then rapidly worked on a “correct-a-sentence system” and settled on using a Azure OpenAI API (GPT3.5) and then developed our own custom model alongside this (needing something that worked offline and privacy first was key (NB: The videos is this offline model - Not GPT). A few iterations of this and its correcting as good as GPT 3.5 83% of the time.

So two months later, what does this all mean for the end user? They can now write on technolgy to communicate for the very first time. Happy - well so far yes. But long term AAC use is complex and not just down to the technology. So lets be curious and proud at the same time. There is more to do

• Small tweaks in the UI
• Its correcting a bit too much at times - sometimes adding more grammar changes than I would like - and equally dangerous - losing words. To fix this we could further train our custom model.

and then more I’m sure the next time we review this cycle of the iterative development cycle.

What does this process look like overall?

We follow the double-diamond development cycle. It’s a pretty common approach yet following it is not always easy.

So lessons so far?

• Iterate Fast: Rapid prototyping and adjustments are key to finding viable solutions for complex, unique solutions. There is a much longer tail to development for solid, reliable products but getting somewhere fast helps to see whats important.
• User-Centric Design: Keeping the user at the heart of all stages of development ensures that the technology we develop truly meets their needs. Its not always easy but it can be done. Its important to do this as part of a team though. It can so easily be lost as to whether you are getting carried away or whether you are on track. We pride ourselves on at Ace Centre being a transdisciplinary team where never one person sees the whole picture.
• Continuous Learning: Every challenge presents a learning opportunity, pushing us to constantly improve. And heck - its good fun too seeing big gains.
• Passion and Persistence: A keen interest in making a difference and the drive to keep pushing forward are indispensable.