LEGO Braille Bricks and Audio Instructions: In-Depth Interviews

Most geek parents have fond memories of building LEGO sets or creating their own imaginative structures. For blind geek parents or blind geek kids, playing with LEGO bricks was a different experience or non-experience. In the company’s continued mission of engaging children through play, LEGO announced that in 2020 they will release a line of Braille Bricks to create a more inclusive experience. In this interview, we spoke with Matthew Shifrin (whose website legofortheblind.com offers text-based instructions), LEGO, and the AI Institute about their different roles in the new inclusive offerings.

Matthew Shifrin

Q: Can you explain the process of creating the text-based instructions? E.g. how long did take, what was the trail/error process like, and what was the biggest surprise in the process?
A: My friend Lilya originally came up with the idea. Though I played with LEGO as a child, I couldn’t build sets on my own, since I couldn’t read the graphical instructions. This was unfortunate since all my friends at the time were building spaceships and fortresses, while I was trying to guess how a jumble of seemingly random pieces might be assembled to form something recognizable.

Lilya created a system of text-based instructions, taking the graphical instructions and writing them out in Braille, she came up with names for every piece that was used in a set, and the instructions worked better than I could have ever imagined. After Lilya created the first set of text-based instructions, we started legofortheblind.com, a website where we posted text-based instructions for every single LEGO set we could get our hands on.

When the site launched, we got hundreds of e-mails from parents of blind children and blind kids asking for different sets to be adapted, and we had to turn them down. We were just two people—Lilya wrote the instructions, and I built the sets and checked the instructions for errors. That’s when I realized I had to turn this project up a notch. So I reached out to LEGO and asked them whether they’d be interested in creating their own text-based instructions, so blind kids could build LEGO sets on their own. Lilya died two years ago, and I promised myself that I would not let this project rest until LEGO took action. I’m so glad that they’ve taken it upon themselves to make their sets accessible to blind children.

Q: In the video on the LEGO audio site, you discussed the social aspects of building with LEGO, engaging with other children with a common learning tool. How did the process of working with LEGO help you reinforce your vision?
A: It’s an honor to work with The LEGO Group and see their commitment to accessibility. It’s an amazing feeling to finish a set because I know that I built it on my own. It’s important to give blind children aha moments so that they too can say, “Look, mom, look what I built!”

The word independence gets thrown around a lot when teaching blind children, and it’s so important that they have the experience of building sets on their own, since LEGO helps improve spatial awareness and spatial reasoning, areas that blind children struggle in. I learn so much from every set I build.

Q: As a learning tool, how do you feel the LEGO Braille Bricks themselves help bring kids together in new ways?
A: LEGO Braille Bricks offer a groundbreaking and inclusive opportunity to teach and develop different skills among blind, visually impaired, and sighted children using the same tool. This not only helps teach but allows visually impaired and sighted children to work alongside each other in the class and ensures no one feels excluded.

Q: You discussed the types of information—like the double-decker bus—blind children can get from building LEGO. Tell us a little bit about your personal experience in engaging differently with the world because you were able to build these structures/sets.
A: As a blind person, cultural landmarks are an abstract concept. I know that the Statue of Liberty is an iconic symbol of America, but I have no idea how it’s shaped. It’s a woman holding a torch and a scroll, that’s all I know.

If I build the Statue out of LEGO, then I become intimately familiar with its shaping and am better able to understand the world around me. So if someone says, “We can see the Statue Of Liberty from here,” I can say, “Oh, I know what it looks like. I built it.”

Q: How has the experience of working with children been for you? How did the kids react?
A: Kids I know who have been able to build the sets have been ecstatic. They love the feeling of being able to build something on their own and genuinely enjoy the process and learn so much along the way.

Q: How do you see the text-based LEGO instructions and the LEGO Braille Bricks impacting the blind community?
A: In the United States only 10% of children who are blind are learning to read Braille, compared with over 50% in the 1950s (National Federation of the Blind report). This can impact long-term independence and confidence for visually impaired kids around the work.

Q: Are there any other insights you’d like to give our readers, either as parents or LEGO fans?
A: We have seen an incredible amount of pride from both kids and parents. Not only has this given the kids an opportunity to do something independently that they otherwise could not, LEGO play has given them the means to experience the world and play in a way that hasn’t been possible before.

We’ve also heard from some younger kids that this is hard, but in the end, when they feel the success of being able to play with the car they just built, they can’t wait to keep building. The pride of creation that we see on the children’s faces after having built something on their own, or with modest help from friends/family, is invaluable. These are big steps and once they get the hang of it, there’s really no limitations to what they can build.

LEGO Team

Q: LEGO Braille Bricks and traditional LEGO bricks have different “peg” positions. Describe the process of finding a way to integrate the two types of brick together.
A: All LEGO elements, including LEGO Braille Bricks, are produced on the same system—the LEGO System in Play, meaning that while the LEGO Braille Bricks have different studs missing they are still fully compatible with ordinary LEGO bricks and elements.

Q: Describe your process for conceptualizing the Braille Bricks and bringing them to fruition.
A: It’s important to note that while the two projects are aimed at a similar audience, they are not interlinked and have been run as two separate projects, which have each been inspired by outside sources.

In regards to LEGO Braille Bricks, The LEGO Group has over the years been approached by many organizations with the idea to adapt LEGO bricks to Braille. In 2011, The LEGO Foundation was approached by the Danish Association of the blind, but it wasn’t until 2017 that The LEGO Foundation was encouraged by a Brazilian charity—The Dorina Nowill Foundation—to start-up a project to develop a prototype for testing.

Q: The research on the website is interesting. The idea that fewer blind children are learning Braille because technology enables them to adopt audio recordings. Can you give us some more insight into what you’ve learned about the impact that has on blind children as they grow into adulthood? How did you integrate that research into the process of creating the bricks?
A: Young blind children have the same dreams and aspirations for their future as sighted children. But as they grow older, they risk involuntary isolation as a consequence of exclusion from social and sporting activities as well as taking the same educational and career paths as sighted children. Today, the number of blind people receiving an education is drastically declining, leading to greater unemployment.

Research shows that learning Braille can widen opportunities for blind and visually impaired children, enabling them to experience intellectual freedom, independence, and equal access to study and work. Braille opens doors for blind people to further develop a wide breadth of skills—all helping to build the confidence needed to pursue their dreams and aspirations in life

According to the European Blind Union (EBU), Braille knowledge leads to better literacy—correct spelling, reading and writing—and greater understanding of text structure. Braille users often have a higher level of education and have better employment opportunities. So, despite technical advancement, it is still highly important for children to learn Braille and LEGO Braille Bricks provide a revolutionary platform to teach Braille in a very playful and therefore child appropriate manner.

Q: LEGObraillebricks.com notes the importance of “inclusive learning.” Describe the approach taken to bridge the gaps between blind and non-blind children?
A: The LEGO Foundation and LEGO Group have a shared mission to inspire and develop the builders of tomorrow and believe learning through play is the most powerful way for children to develop a breadth of skills—whether they be visually impaired or not.

Not only does this tool help visually impaired kids learn math and reading, but it allows visually impaired and sighted children to work side by side, it allows teachers to better support students with vision impairment, and it helps sighted kids learn more about Braille, all of which is happening in fun, playful, and engaging ways.

Q: As a parent, the idea of the Braille bricks is really exciting. With the letters on them, it gives non-blind children the opportunity to learn Braille as well. If you were in a classroom, how would you want a teacher to use these? What suggestions would you give?
A: LEGO Braille Bricks are a tool for a truly inclusive classroom, as the bricks are molded with studs corresponding to the Braille alphabet, as well as letters, so that children who are blind or have low vision can learn side by side with other students and teachers who are sighted.

What’s especially noteworthy is that, compared with a Braille machine, the LEGO bricks allow for tinkering, trying, getting it wrong, and trying again. This makes it a tool not just for teaching Braille, but for students who are blind or low vision to practice core subjects like math and language. The possibilities are endless with the power of play and using these in the classroom.

The LEGO Foundation is currently developing the pedagogical concept, which will be an essential part of the LEGO Braille Brick toolkits—this will provide exactly this type of recommendation and idea starters. Teachers that have been testing the bricks for some time have already implemented their own ways of using them and also getting students to come up with fun suggestions, i.e. for word spelling games, foreign languages, math subtraction/multiplication, etc.

AI Institute

Q: From my limited understanding of “Big Data,” the AI Institute really needed to find a way to take an unstructured data set (i.e. images not text) and translate it into audio. Talk to us a little bit about what that process was like, where you started, what the struggles were, and how you solved some of those problems?
A: We at OFAI were thrilled that LEGO approached us to work with them on this project. I have fond memories of playing with LEGO as a kid and find it rewarding to be part of an effort to bring the joys of LEGO to kids who are blind or visually impaired, thus formerly having been excluded from the experience of building LEGO models.

While we often think of employing AI-based methods to work with machine learning on large amounts of data, with the project “LEGO® Audio & Braille Instructions,” we started out with just a few XML-based representations and the textual sample descriptions. This initial sample came from Matthew Shifrin and his project LEGO for the Blind.

The key was to articulate how each brick connects to the rest of the structure in an intelligent way. With that in mind, we developed an algorithm to translate technically coded building steps into natural language prompts.

For example, “Build a bison. Find a black brick 2×4. Put it on the table.” We manually tested and edited all of the prompts generated by the system, with Matthew as well as children in the UK and Denmark.

Based on their feedback, reference data was created which in the future will allow for applying statistical machine learning methods to enhance and improve the generation process—striving for a deeper comprehension of the spatial and conceptual properties of the LEGO world.

Q: What was it like scaling the project? Was it more difficult than initially expected? Easier?
A: This project has been in the works for over two years and something LEGO hadn’t done before, so it did take some time to understand how we could scale. With the LEGO® Audio & Braille Instructions, we started out just with a few XML-based representations and the textual sample descriptions provided by Matthew Shifrin.

While trying to understand how these models were organized according to their building steps and orientation in 3D-space, we arrived at a solution that takes into account how each brick connects to the rest of the structure in an intelligent way. Based on that, we developed strategies that enable the algorithm to translate technically coded building steps into natural language prompts (such as “Build a bison. Find a black brick 2×4. Put it on the table.”) that, in the end, guides listeners (e.g. a visually impaired child) in building their LEGO models.

To ensure the end-user gets high-quality instructions, the building prompts generated by the system can be manually corrected and edited. As a bonus effect, reference data is created, which then allows for applying statistical machine learning methods to enhance and improve the generation process—striving for a deeper comprehension of the spatial and conceptual properties of the LEGO world.

Q: Going forward, how are you planning to integrate these instructions for all sets?
A: We have ambitions to add new sets early next year, once we hear back from users. The success of this pilot will help us determine how much of the portfolio we can support next year, so it’s too early to commit to a date and number. Offering a fun and high-quality play experience is our absolute key focus and will drive the pace for scale.

Q: From a roadmap perspective, can you give us any insight into what we can look forward to? What you’d like us to know?
A: Our long term ambition is for all LEGO sets to be fully accessible to users with visual impairments. However, providing a fun a high-quality play experience will always drive the pace in which we scale.