Robotic Vision

Robotic Vision introduces you to the field of computer vision and the mathematics and algorithms that underpin it.You’ll learn how to interpret images to determine the color, size, shape and position of objects in the scene.We’ll work with you to build an intelligent vision system that can recognise objects of different colours and shapes.

Combined with the Introduction to Robotics MOOC, this course is based on a 13 week undergraduate course Introduction to Robotics at the Queensland University of Technology.

If you have completed the Robot Project in Introduction to Robotics, you will be able to combine this with the vision system you will develop in this course (optional).

Course Outcomes

By the end of this course you should be able to:

• describe and explain the utility of vision as a sensor for robots and evaluate the challenges inherent in visual information.
• describe the underlying principles of common image processing techniques and the circumstances where they are applicable, the rationale for reducing image pixels to features and the principles image region segmentation and feature extraction.
• describe the mathematical and geometric principles underlying the formation of images.
• describe the principles of continuous spectra, trichromatic vision and the separation of chrominance and luminance information.
• demonstrate the software skills to import images from a variety of sources into MATLAB and perform a number of image processing and feature extraction algorithms using MATLAB.
• apply the mathematical and algorithmic and control principles of computer vision to implement a working vision system (applies to optional project).
• integrate the robot and vision into a functional system in which the desired object is recognised and the robot moves to it (applies to optional project).

Pathways through the course

You can follow three paths through the MOOC:

1. You can audit the course: Register and have access to all the content, lectures, quizzes and programming assignments which you can view at your leisure. You do not necessarily have to submit the assessments. You will not receive a certificate at the end of the course.
2. To receive a certificate of completion you should register for the course, study all the content and successfully complete at least four assessable quizzes and four programming assignments.
3. An additional option is to complete a robot building project which will be peer assessed.

Workload

You should spend about 4–8 hours per week on this course. Depending on your level of skill with MATLAB and programming in general, your studies might include:

• 2 hours viewing the lecture videos and completing the self-assessment quizzes;
• 30 minutes for each of six weekly assessable quizzes
• 2 hours for each of six weekly programming assignments;
• 1–2 hours building the robot or vision system (optional project) or doing further research and/or communicating on forums.

Time required for the last two items will depend on your level of skill with MATLAB and programming in general.

Certificate of Achievement 

Throughout the course you will be asked to complete quizzes and programming exercises. These will be automatically marked. If you pass 4 quizzes and 4 programming assignments you will receive a Certificate of Achievement. The programming assignments will consist of several MATLAB tasks and will be based on the lecture content for that week.

The robot project will be peer assessed but doesn’t count towards the Certificate of Completion.

Assumed knowledge

To complete the MOOC you should know some basic programming (either of MATLAB or of an object-oriented programming language) and some of the following areas of maths: matrices, vectors and spaces, eigenvalues and eigenvectors. Below are links to these topics on the Khan Academy. We believe this knowledge will be necessary to complete the course, and recommend that you view these before Week 1 begins, but you might prefer to watch them on an ‘as needed’ basis after you view our content.

• Vectors
• Matrices:
  • basic matrix operations including multiplication and inversion
  • zero and identity matrices
  • vector transformations with matrices
  • familiarity with the concepts of eigenvalues and eigenvectors.

Note: A free version of MATLAB will be available for the duration of the course. Also, Professor Corke’s textbook (not essential) will be available at a significant discount.

Requirements Hardware

1. Computer: You will need a computer that is capable of running MATLAB. For System Requirements, visit the MathWorks’ System Requirements page.

Software

1. MATLAB. This is a proprietary technical computing and visualisation package developed by The Mathworks Inc. It is widely used in the engineering industry, in scientific research and for undergraduate education. MATLAB is a core part of the course and is available for free to students studying this course. It is used to illustrate techniques in the lectures (via screencasts) and also for the assignment (you need to write programs and submit them for automatic grading). A free MATLAB licence and software for installation will be available when you enrol in the course.
2. Open source toolboxes for MATLAB. These will also be available from the course site.

Other

1. Textbook (optional). Access to the textbook written by Professor Peter Corke (2011), Robotics, Vision and Control: Fundamental Algorithms in MATLAB (Springer) is optional, but considered beneficial.

Progress through the Course 

The course content will be released weekly (Midnight Sundays UTC). Each week you will go through the video materials in two lectures. The videos are interspersed with non-assessed quiz questions and exercises to check your understanding. After the two lectures you will be presented with two assessable tasks: a quiz and a programming activity. (You will need to pass four of the six quizzes, and four of the six programming tasks to receive a Certificate of Completion.) Finally, an optional – but exciting – activity is to build your own robot vision system. Throughout the course you will also be encouraged to take part in discussion forums.

Week 1

This week we will look at how robots sense the world around them. We will explore the sense of vision, what this means for robots, and learn how to get an image into MATLAB where we can begin to process it. The skills you’ll learn and the tools we’ll use will be essential for the MATLAB assignments and the project.

• Lecture 1: Robot Vision
• Lecture 2: Getting images into a computer

Week 2

This week we will discuss how to process images with simple arithmetic and logical operations to perform operations such as thresholding, green screening and change detection. Then we look at operations that allow us to blur or find edges within images, find distinctive patterns or select only objects of particular shapes. 

• Lecture 3: Image processing
• Lecture 4: Spatial operators

Week 3

This week we’ll ask the question “How can a robot know what it’s looking at?” We will see how robots can extract simple meaning from an image, describing objects in terms of their size, shape and position. Then we look at color, its importance in image processing, and how we can describe and select objects based on their color.

• Lecture 5: Feature extraction
• Lecture 6: What is color?

Week 4

So far we’ve assumed the existence of an image. This week we will learn how images are formed by cameras. The first lecture presents a simple geometric approach to understanding this, while the second lecture uses homogeneous coordinates to express this process in terms of matrices and vectors which we can calculate easily using MATLAB.

• Lecture 7: Image formation
• Lecture 8: Image geometry

Week 5

This week we will how humans perceive the 3D nature of the world using binocular vision, how stereo displays and TVs work, and how we can mimic this in a computer to extract a 3D model of the world from two images. Then we followup on lecture 4 and discuss advanced image processing techniques for selecting shapes, rotating and scaling images, and finding dominant lines and points within an image.

• Lecture 9: 3D vision
• Lecture 10: Advanced image processing operations

Week 6

In this final week we discuss the way that an image changes as the camera moves. Then we invert this and compute how the camera should move so that the image changes in the way we want, a technique commonly known as visual servoing.

• Lecture 11: Vision and motion