CISC849 F2019

Course information

Title	CISC849 Robot Vision and Learning
Description	Survey of image-based 2-D and 3-D sensing algorithms for mobile robot navigation and interaction, including motion estimation, obstacle segmentation, terrain modeling, and object recognition, with a particular focus on deep learning techniques to dramatically improve performance.
When	Tuesdays and Thursdays, 2-3:15 pm
Where	Gore 317
Instructor	Christopher Rasmussen, 446 Smith Hall, cer@cis.udel.edu
Office hours	Mondays and Wednesdays, 2-3 pm
Grading	20% Oral paper presentation (individual or pairs, 30 minutes) 30% Two programming assignments (individual) 50% Final project (teams of 1-3) 10% = 2 page proposal, including planned methods, citations of relevant papers, data sources, and division of labor 10% = Joint 15-minute presentation on final results, with accompanying slides 30% = Actual results and estimated effort, factoring in difficulty of problem tackled
Academic policies	Programming projects are due by midnight of the deadline day (with a grace period of a few hours afterward...after sunrise is definitely late). A late homework is a 0 without a valid prior excuse. To give you a little flexibility, you have 6 "late days" to use on homeworks to extend the deadline by one day each without penalty. No more than three late days may be used per assignment. Late days will automatically be subtracted, but as a courtesy please notify the instructor in an e-mail of your intention to use late days before the deadline. See submission instructions below. Students can discuss problems with one another in general terms, but must work independently on programming assignments. This also applies to online and printed resources: you may consult them as references (as long as you cite them), but the words and source code you turn in must be yours alone. The University's policies on academic dishonesty are set forth in the student code of conduct here.
Homeworks	Assignment submissions should consist of a directory containing all code (your .cpp files, makefile, etc.), any output data generated (e.g., images, movies, etc.), and an explanation of your approach, what worked and didn't work, etc. contained in a separate text or HTML file. Do not submit executables or .o files, please! The directory you submit for each assignment should be packaged by tar'ing and gzip'ing it or just zip'ing it. The resulting file should be submitted through Canvas. You may develop your C/C++ code in any fashion that is convenient--that is, with any compiler and operating system that you want. However, we will be grading your homework on a Linux system with a makefile, and so you must avoid OS- and hardware-specific functions and provide a makefile for us that will work (like one of the templates above).

Possible Papers to Present (not a complete list)

• Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours, Pinto and Gupta, ICRA 2016
• Learning real manipulation tasks from virtual demonstrations using LSTM, Rahmatizadeh et al., AAAI 2018
• Learning to Fly by Crashing, Gandhi, Pinto, and Gupta, IROS 2017
• VoxNet: A 3D Convolutional Neural Network for real-time object recognition, Maturana and Scherer, IROS 2015
• PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation, Qi, Su, Mo, and Guibas, CVPR 2017
• Real-Time Navigation in 3D Environments Based on Depth Camera Data, D. Maier, A. Hornung, and M. Bennewitz, Humanoids 2012
• Robotic Grasping of Novel Objects using Vision, A. Saxena, J. Driemeyer, A. Ng, IJRR 2008
• DeepLoco: Dynamic Locomotion Skills Using Hierarchical Deep Reinforcement Learning, X. Peng, G. Berseth, K. Yin, and M. van de Panne, SIGGRAPH 2017
• High fidelity day/night stereo mapping with vegetation and negative obstacle detection for vision-in-the-loop walking, M. Bajracharya et al., IROS 2013

Instructions for Homeworks

Software	Point Cloud Library (PCL). Latest version on GitHub seems to be 1.8.1. Should be installable on Linux/Mac/Windows Tutorials Documentation Installation on Ubuntu 16.04 (including prerequisite libraries)
Note	TensorFlow is available for multiple operating systems, but the processing demands can be high, especially if your computer does not have a new-ish Nvidia GPU If you have significant problems installing/using PCL or TensorFlow on your own computer, I have a GPU workstation in my lab that can be used for selected training jobs. Contact me *after* you have determined that your own machine is insufficient

Schedule

Note: The blue squares in the "#" column below indicate Tuesdays.

#	Date	Topic	Links/Readings/videos	Assignments/slides
1	Aug. 27	Background		slides
2	Aug. 29	Finish background		slides
3	Sep. 3	Introduction to DARPA Urban Challenge (UC), Robotics Challenge (DRC) and algorithm components	"Crashes and Traffic Jams in Military Test of Robotic Vehicles" (NYTimes, Nov. 5, 2007) UC highlights (6:06), Stanford UC clips DRC highlights (5:47) NOVA "Rise of the Robots"	slides
4	Sep. 5	Introduction to ROS; PCL tutorial	ARGOS challenge overview (8:00)	slides Sample PCL programs (these require ROS) ETH ROS mini course (in particular: overview, RViz, TF), ETH case study "ANYmal at the ARGOS Challenge"
5	Sep. 10 Register/add deadline	Plane/obstacle/object segmentation (3-D)	RANSAC background	HW #1 plane_fit.cpp (No ROS required)
6	Sep. 12	Finish plane segmentation, clustering, ICP		slides
7	Sep. 17	Image classification background		Stanford CS231n Image classification slides (6-61),
8	Sep. 19	More classification background		Loss/optimization slides (8-10, 37-77, finish with web demo), Intro to NN (83-88) HW #1 due
9	Sep. 24	Introduction to convolutional neural networks	Stanford CS231n Convolutional Neural Networks slides (15-63, 71-78), CNN architectures (8-36), Detection and Segmentation slides (15-27)
10	Sep. 26	Finish detection & segmentation background, introduction to TensorFlow library	YOLOv2, Redmon and Farhadi (CVPR 2017)	Stanford CS231n Detection and Segmentation slides (44-49, 53-85), deep learning software slides (21-33, 40-59) TensorFlow code sample from Stanford lecture
11	Oct. 1	More on TensorFlow programming		HW #2
12	Oct. 3
	Oct. 8	NO CLASS Instructor away	"Learning Locomotion over Rough Terrain using Terrain Templates", M. Kalakrishnan, J. Buchli, P. Pastor, and S. Schaal, IROS 2009	slides DeepLoco highlights video (6:23)
13	Oct. 10	Perception for stepping		HW #2 due Friday, March 23
14	Oct. 15
15	Oct. 17
16	Oct. 22 Withdraw deadline	Imitation learning	A Machine Learning Approach to Visual Perception of Forest Trails for Mobile Robots, Giusti et al., IEEE Robotics and Automation Letters, 2016 End to End Learning for Self-Driving cars, Bojarski et al., 2016 Dave-2 driving a Lincoln, Quadcopter Navigation in the Forest..., DAgger for drones Hal Daume DAgger explanation (9:26-12:43)	Berkeley Levine course IL slides (DAgger defined on slide 14); CMU Fragkiadaki IL slides (1-29) Paper presentation choice due
17	Oct. 24	Reinforcement learning		UCL Silver course: RL slides (7-42), RL video (15:40-21:20, 1:02:13-1:04:40 contains movies missing from PDF); Stanford deep-Q lecture slides (23-62); original deep-Q paper
18	Oct. 29	Student paper presentation
19	Oct. 31	Student paper presentation		Project proposal due Sunday, April 15
20	Nov. 5	Student paper presentations
21	Nov. 7	Student paper presentations
22	Nov. 12	Student paper presentation
23	Nov. 14	Student paper presentations
24	Nov. 19	Student paper presentation
25	Nov. 21	Instructor paper presentation	Rasmussen, "Real-Time SLAM with Octree Evidence Grids for Exploration in Underwater Tunnels", N. Fairfield, G. Kantor, and D. Wettergreen, JFR 2007	DepthX slides
	Nov. 26	NO CLASS Instructor away
	Nov. 28	NO CLASS Instructor away
26	Dec. 3	Final project presentations part I
27	Dec. 5	Final project presentations part II