Lesson Do You See What I See?

Learning Objectives

After this lesson, students should be able to:

• Apply some steps of the design process to solving a problem.
• Describe the engineering design process and the software design process.
• Explain the importance of using a design process to develop a solution to a problem.
• Explain optical character representation (OCR) and techniques that can be used by OCR.
• Develop an algorithm for OCR.

Educational Standards

Each TeachEngineering lesson or activity is correlated to one or more K-12 science, technology, engineering or math (STEM) educational standards.

All 100,000+ K-12 STEM standards covered in TeachEngineering are collected, maintained and packaged by the Achievement Standards Network (ASN), a project of D2L (www.achievementstandards.org).

In the ASN, standards are hierarchically structured: first by source; e.g., by state; within source by type; e.g., science or mathematics; within type by subtype, then by grade, etc.

Common Core State Standards - Math

International Technology and Engineering Educators Association - Technology

State Standards

Worksheets and Attachments

Pre-Req Knowledge

Students must have an understanding of Java programming basics. Although their knowledge need not be advanced, students must be able to develop code that needs to be tested in order to be able to design test cases as part of the larger design process.

Introduction/Motivation

(Have available computers with Internet access, one per group, so students can conduct their own background research on optical character recognition.)

In this lesson, you will work as if you were an engineer challenged to develop a solution to a problem. You will use the software/system design process (see Figure 1) to solve a problem that is very complex. Your task is to learn about and then design a computer program to perform optical character recognition (OCR). The creation of OCR is VERY complex, so to begin, it will be greatly simplified. You will learn about OCR and then create an algorithm that can be used to "read" a number off a typical high school scoreboard.

In the associated activity Java Programming of OCR, you will implement your algorithm in a Java programming environment and test/redesign it with the goal of getting it to function correctly. To complete this task, you will follow the steps that all engineers use to solve problems—the engineering design process. All types of engineers perform these same steps: determine the problem requirements and constraints, come up with ideas and design the best solution, create/implement that solution, test and refine it and then see where the solution takes them over time (evolution). Software engineers do exactly this when they develop new software. They have a problem in mind for the software to solve while meeting certain requirements. Next, a design is created, implemented and tested. When (not if) the software application fails in some aspect, engineers return to earlier steps of the design process in order to improve the product.

The software/system design process has some similarities to the scientific method, with which you may be familiar. Both are problem solving processes that guide scientists and engineers in a structured way to contribute to the world of human knowledge, but they do so in different ways. The steps of the scientific method include: formulate a question, make a hypothesis, make predictions, test, and analyze. Scientists make testable explanations and predictions about the world to find out how nature works while engineers create solutions to problems, generating new things such as products, websites, environments and experiences. The design process that engineers use serves them well to develop solutions to complex problems. To learn the process, you must practice in controlled situations. This lesson and activity gives you a "playground" to explore.

To begin, let's start with researching optical character recognition (OCR) processes. Your goal is to learn about the challenges of getting a computer to read text, which is not an easy task. (Write the following questions on the classroom board.) Use these questions to help frame your research:

• What are some uses of optical character recognition (OCR)?
• What kinds of people (jobs) would use OCR or find it useful?
• What are some challenges in developing OCR software?
• How can computers learn to read human writing?

In your research, look at different OCR methods and take notes on what you learn. (Write the following topics on the board.) Some example OCR methods and techniques are:

• Image pre-processing
• Image de-skewing
• Focusing of the window on the relevant data
• Seven-segment specific testing

When you have gained some understanding of OCR and its methods through research, you'll focus on the more specific problem of reading a digit from a sports scoreboard.

(Continue by delivering the content in the Lesson Background section.)

Lesson Background and Concepts for Teachers

Implementing the Lesson

Divide the class into groups of two or three students each, and present the engineering challenge: To read ONE digit from a typical high school scoreboard, such as one you might find on a football field or in a gymnasium. These digits are generally set up in a seven-segment array (see Figure 2). To do this, you will perform the first two steps of the system/software design process: identify requirements of the problem and create an algorithm (your "design solution") on paper that will later be implemented and tested. Your algorithm can be in the form of a flow chart, pseudocode or something similar. An example might be: "Look at the seven major points on Figure 2 and decide if the points are lighted or not. After gathering the "lighted" or "not lighted" information, determine what digit you have." This is an example of pseudocode. The seven points in question are the seven segments that make up scoreboard numbers.

When each group has what it thinks is a viable algorithm, have groups share their ideas and discuss each algorithm's perceived strengths and weaknesses.

Then lead a class discussion, going through the following research questions to give students the opportunity to share what they learned about OCR:

• What are some uses of OCR? (Example answer: To turn printed text into digital text. To turn a photograph of a license plate into a known license plate identifier composed of digital letters and numbers.)
• What kinds of people (jobs) would use OCR or find it useful? (Example answers: People who work at the post office use OCR for automatic routing of mail. Scholars who want to make the writing in old manuscripts and out-of-print books searchable. )
• What are some challenges in developing OCR software? (Example answer: Letters have more than one way of being represented, such as capitals vs. lowercase, script vs. printed/block, handwritten vs. typed.)
• How can computers learn to read human writing? (Answer: Machine learning, testing and examples.)

Background & Concepts

The testing of software without an oracle—situations in which it is impossible to test all possible cases—is very difficult. The lesson is a hands-on design-based project in which students are faced with developing a software application in which testing is very difficult. They are challenged to design, implement, test and redesign in order to correctly address the problem. They are able to test in "real-world" situations by using photographs of scoreboards from the Internet or school sports venues. Expect students to find that their algorithm projects do not function correctly all the time. Attaining correctness is part of the challenge. However, another part of the activity is how students handle the design process and the vagueness of the project. Consider all this when assessing students.

Testing Software without an Oracle: Software is tested to make sure that the code does what the programmers and designers intend. Some programs can be easily tested with a set of known input data and a corresponding set of known or expected output data. This "test oracle" is used to see if the software passes or fails a series of tests. Unfortunately, it is far more common that software, once in use, will be subjected to a wide array of input data such that it is impossible or impractical to test every case in the software design process. In these cases, because the exact set of input data is unknown, oracles cannot be created to test the programs.

Several manufacturers are embarking on an exciting field with an enormous potential impact on society—the automation of the driving process by creating driverless cars. Much like optical character recognition, it is not possible to ever say that a driverless vehicle is "perfect," thus software engineers must test as much as they can because the existence of so many potential scenarios makes it impossible to test every one. Thus, the algorithm must be adaptive enough to handle situations that were not expected or tested.

Optical Character Recognition: OCR is an emerging computer science technology in which an image of text is "fed" to a computer and processed to convert the image into a text document. (Note: "Fed" is the general term for getting data into the computer where it can be used. Images might include those retrieved from the Internet or captured with a digital camera as photographs.) In image form, letters and numbers are only pixels of color. Once identified via OCR as specific letters and numbers, the text is able to be manipulated, searched, indexed or otherwise modified. OCR is a prime example of a case in which every possible data input cannot be tested. Computer scientists must create a robust algorithm to determine what letter is being displayed in an assemblage of pixels in an image. A variety of methods and techniques have been developed to help prepare and identify pertinent text for OCR processing.

Image Pre-Processing: To make images more readable, we eliminate unnecessary data. One way is to convert color images to black and white images to improve the contrast. This can be done so the image only uses black or white, no grays, resulting in a starkly contrasted image that is easier to work with.

Image De-Skewing: Often, images are captures at angles, instead of head-on. By identifying the corners of the information we are interested in, we can adjust the image to make the text appear on straight lines even if the photo was taken at an angle.

Focusing of the Window on the Relevant Data: Since scoreboards do not move and cameras can be mounted in static locations, we can manually identify sub-areas within larger scoreboard images where individual digits are expected to appear. By only looking at known, single digits, we can focus our algorithms.

Seven-Segment Specific Testing: Each number in a display is comprised of seven segments. To read letters, we would need more than seven data points. The benefit of narrowing our OCR to scoreboards is that if we have knowledge of these seven segments and their on/off statuses, we can determine which numbers are being displayed.

Machine Learning and Training: Machine learning is a technique of enabling computers to decide what aspects of a letter are important. For example, a letter is fed into a computer and the computer is told that the letter is an "A." More examples of the letter "A" are fed into the computer until it creates enough of its own rules so as to be able to identify what an "A" is, at which point a random letter could be correctly identified as an "A" by the computer based on criteria unknown to the programmer but understood by the computer.

Applications: Currently, OCR is used to digitize books and other writing created before the advent of computers. This enables the whole of human writing to be stored in digital format so that it can be protected, indexed and searched. The U.S. Postal Service uses OCR as a time- and cost-saving measure to automatically route mail without human intervention. Automatic number plate recognition uses OCR to read vehicle registration plates for use at toll roads, toll bridges, red light cameras, speed monitoring cameras, police surveillance, etc.

Associated Activities

• Java Programming of OCR - Using a Java programming environment, student groups create the algorithms for OCR that they developed in the lesson. They continue with the steps of the engineering design process as they implement, test and revise their algorithms. Ultimately, students create programs that can read numbers on a scoreboard.

Vocabulary/Definitions

engineering design process: A series of steps engineers use as they design solutions for problems and challenges. The steps include: defining a problem, brainstorming, researching and generating ideas, identifying criteria and specifying constraints, exploring possibilities, selecting an approach, developing a design proposal, making a model or prototype, testing and evaluating the design, refining the design, creating or making it, and communicating processes and results.

Java: An object-oriented computer programming language.

optical character recognition: The mechanical or electronic conversion of handwritten, typewritten or printed text into machine-encoded text.

pseudocode: A high-level description of how a computer program or other algorithm functions.

test oracle: In software testing, a set of known input data and a corresponding set of known or expected output data.

Assessment

Pre-Lesson Assessment

Questions: Obtain a sense of students' base knowledge of the engineering design process by asking them the following questions:

• Describe any experience you have had with the engineering design process or scientific method.
• What were the steps in the design process?
• How did the steps function together in a cyclical process?

Post-Introduction Assessment

Observations: As students are engaged in the lesson research and algorithm development, ask yourself the following (or similar) questions:

• Do students understand what the design process is and how to apply it?
• Are students able to explain what OCR is and how it works?
• Can students verbalize how different OCR techniques work?

Lesson Summary Assessment

Post-Quiz: Administer the Do You See What I See? Post-Quiz, composed of three essay questions. Review students' answers to gauge how much they know about the software design process, OCR and how the design process plays a role in developing a successful algorithm. Example answers are provided in the Do You See What I See? Post-Quiz Answer Key.

Additional Multimedia Support

The source of the seven-segment display number graphic, which may be helpful to show students as an overhead projector image or handout, is available at Wikimedia Commons at https://en.wikipedia.org/wiki/File:7_segment_display_labeled.svg.

See the Science Buddies' Comparing the Engineering Design Process and the Scientific Method for good explanations and flow charts of the steps of both processes at http://www.sciencebuddies.org/engineering-design-process/engineering-design-compare-scientific-method.shtml.

References

Copyright

Contributors

Supporting Program

Acknowledgements

The contents of this digital library curriculum were developed as a part of the RET in Engineering and Computer Science Site on Infusing Mobile Platform Applied Research into Teaching (IMPART) Program at the University of Nebraska-Omaha under National Science Foundation RET grant number CNS 1201136. However, these contents do not necessarily represent the policies of the National Science Foundation, and you should not assume endorsement by the federal government.