Programming short course and workshop on Deep Learning and Computer Vision for Autonomous Systems

Short description

This three day short course and workshop provides an in-depth presentation of programming tools and techniques for various computer vision and deep learning problems encountered in autonomous systems. Special attention will be paid to drones in infrastructure inspection, which is one of the main application areas of drone technologies. The same machine learning and computer vision problems do occur in other drone applications as well, e.g., for drone cinematography, land/marine surveillance, search&rescue, and 3D modeling. The short course consists of three parts (A,B,C), each having lectures and a programming workshop with hands-on lab exercises.

Part A will focus on Deep Learning. The lectures of this part provide a solid background on Deep Neural Networks (DNN) topics, notably convolutional NNs (CNNs) and deep learning for object detection. Various DNN programming tools will be presented, e.g., PyTorch, Keras, Tensorflow. The hands-on programming workshop will be on PyTorch basics and target detection with Pytorch.

Part B lectures will focus on computer vision algorithms, namely on 2D target tracking, 3D target localization techniques (giving the attendants the opportunity to master state of the art video trackers), parallel GPU, multi-core CPU architectures and GPU programming (CUDA). Two programming workshops will take place. The first one will be on CUDA programming, focusing on 2D convolution algorithms. The second one will be on how to use OpenCV (the most used library for computer vision) for target tracking.

As autonomous systems execute missions (e.g., AV inspection), Part C lectures will focus on Drone perception. Before mission execution, it is best simulated, using drone mission simulation tools. Such simulations will be presented using AirSim. Additionally a programming workshop on ROS and Gazebo simulations for drones will take place.

The lectures and programming tools will provide programming skills for the various computer vision and deep learning problems encountered in drone inspection, which is one of the main application areas of drone technologies. The same machine learning and computer vision problems do occur in other drone applications as well, e.g., for drone cinematography, land/marine surveillance, search&rescue.

Lectures and programming workshops will be in English. PDF files will be available at the end of the course.

Part A (8 hours), Deep learning sample topic list

1. Deep neural networks. Convolutional NNs.
2. Deep learning for target detection.
3. Image classification with CNNs.
4. Target detection with PyTorch.

Part B (8 hours), Computer vision sample topic list

1. 2D target tracking and 3D target localization.
2. Parallel GPU and multi-core CPU architectures. GPU programming.
3. CUDA programming.
4. OpenCV programming for object tracking.

Part C (8 hours), Drone perception sample topic list

1. Drone mission simulations.
2. Drone perception and estimation.
3. ROS UAL Abstraction Layer for drone applications.
4. Drone perception with ROS.
5. Drone mission simulations in Airsim.

WHEN?

The course will take place on 19-21 August 2020.

WHERE?

All lectures and workshops will be delivered remotely.

You can find additional information about the city of Thessaloniki and details on how to get to the city here.

HOW?

Each registrant will use her/his own computer for a) participating in the course and b) for running the programming exercises. A standard PC with a stable internet connection is required. The participants are also required to own a Google account for the workshops exercises. Finally, instructions on how to install AirSim can be found here: https://microsoft.github.io/AirSim/. All lectures and workshops will be delivered remotely using Zoom. The course link is the following: https://authgr.zoom.us/j/91483406779

PROGRAM

Date/time*	19/08/2020	20/08/2020	21/08/2020
Topic	Deep Learning	Computer Vision	Autonomous systems planning and control
8:00-8:30	Registration
	LECTURES	LECTURES	LECTURES
8:30-9:00	Introduction to autonomous systems
9:00-10:00	Deep neural networks – Convolutional NNs	2D target tracking	Drone mission simulations.
10:00-11:00	Deep learning for target detection	Parallel GPU and multi-core CPU architectures – GPU programming.	Drone perception and estimation.
11:00-11:30	Coffee break	Coffee break	Coffee break
	WORKSHOPS	WORKSHOPS	WORKSHOPS
11:30-13:30	Image classification with CNNs.	CUDA programming.	ROS UAL Abstraction Layer for drone applications.
13:30-14:30	Lunch break	Lunch break	Lunch break
14:30-16:30	PyTorch: Understand the core functionalities of an object detector. Training and deployment.	OpenCV programming for object tracking	Drone perception with ROS
16:30-18:30			Drone mission simulations in Airsim.
20:00	Welcome party		Goodbye party

* Eastern European Summer Time (EEST, UTC+3 hours)

** This programme is indicative and may be modified without prior notice by announcing (hopefully small) changes in lectures/lecturers.

REGISTRATION

———————————————————————————————————————————————————

Early registration (till 30/06/2020):

• Standard: 200 Euros

• Reduced registration for young professionals (up to 2 years after graduation): 100 Euros

• Unemployed or Undergraduate/MSc/PhD student*: 50 Euros

Later or on-site registration (after 30/06/2020):

• Standard: 210 Euros

• Reduced registration for young professionals (up to 2 years after graduation): 110 Euros

• Unemployed or Undergraduate/MSc/PhD student*: 60 Euros

———————————————————————————————————————————————————

 

If by clicking the <<Register Now!>> button the page opens in Greek language, please find on the upper right part of the page the option to change it in English. (it is next to the search field)

All lectures and workshops will be in English.

A certificate of attendance will be provided.

*** Due to the special COVID-19 circumstances, the 2020 edition of the «Programming short course and workshop on Deep Learning and Computer Vision for Autonomous Systems» will take place as web course on 19-21 August 2020. Remote participation will be available via teleconferencing. ***

Cancellation policy:

• 70% refund for cancellation up to 15/06/2020
• 50% refund for cancellation up to 15/07/2020
• 0% refund afterwards

Every effort will be undertaken to run the course as planned. However, due to the special COVID-19 circumstances, the organizer (AUTH) reserves right to cancel the event anytime by simple notice to the registrants (by email by announcing it in the course www page). In this case, each registrant will be reimbursed 100% for the registration fee. However, the organizer will be not held liable for any other loss incurred to the registrants (e.g., for air tickets, hotels or any other travel arrangements).

Presentations

Presentations and lab notes will be available to the attendees.

TOPICS

Part A (first day, 2 lectures, 2 programming exercises) 19/08/2020
Deep Learning

The lectures of Part A provide a solid background on the topics of Deep neural networks. Convolutional NNs and deep learning for object detection. Various DNN programming tools will be presented, e.g., PyTorch, Keras, Tensorflow.

The hands-on programming workshop will be on PyTorch basics and target detection with PyTorch.

1. Deep neural networks. Convolutional NNs:
Abstract: From multi-layer Perceptrons to deep architectures. Fully connected layers. Convolutional layers. Tensors and mathematical formulations. Pooling. Training convolutional NNs. Initialization. Data augmentation. Batch Normalization. Dropout. Deployment on embedded systems. Lightweight deep learning. DNN programming tools (e.g., PyTorch, Keras, Tensorflow).

2. Deep learning for target detection:
Abstract: Recently, Convolutional Neural Networks (CNNs) have been used for object/target (e.g., car, pedestrian, road sign) detection with great results. However, using such CNN models on embedded processors for real-time processing is prohibited by HW constraints. In that sense various architectures and settings will be examined in order to facilitate and accelerate the use of embedded CNN-based object detectors with limited computational capabilities. The following target detection topics will be presented: Object detection as search and classification task. Detection as classification and regression task. Modern architectures for target detection (e.g., RCNN, Faster-RCNN, YOLO, SSD). Lightweight architectures. Data augmentation. Deployment. Evaluation and benchmarking.

3. Image classification with CNNs:
Abstract: Learn how to use basic Keras modules and build an Image Classifier using Python and Keras. Different CNN architectures will be programmed and trained on a commonly used classification dataset.

4. Target detection with PyTorch:
Abstract: Given some parts of the code, build and train an object detector in PyTorch. Dataset preparation, data loaders, dealing with unequal number of boxes for each image, understanding the core functionality of an object detector. Training and deployment.

Part B (second day, 2 lectures, 2 programming exercises) 20/08/2020:
Computer Vision

Part B lectures will focus on computer vision algorithms, namely on 2D target tracking, 3D target localization techniques (giving the attendants the opportunity to see state of the art video trackers), parallel GPU and multi-core CPU architectures and GPU programming (CUDA). Two programming workshops will take place. The first one will be on CUDA programming, focusing on 2D convolution algorithms. The second one will be on how to use OpenCV (the most used library for computer vision) for target tracking.

1. 2D target tracking:
Abstract: Target tracking is a crucial component of many computer vision systems. Many approaches regarding face/object detection and tracking in videos have been proposed. In this lecture, video tracking methods using correlation filters or convolutional neural networks are presented, focusing on video trackers that are capable of achieving real time performance for long-term tracking on a UAV platform.

2. Parallel GPU and multi-core CPU architectures . GPU programming:
Abstract: GPU’s unique architectural features are emphasized through CPU-GPU comparison. GPU’s architecture in terms of ALUs and memory types is given in detail in order to introduce the GPU’s programming special characteristics. The audience becomes familiar with terms such as grid, block, thread, kernel, etc. and the general layout of a CUDA program is presented. Cuda keywords are explained by presenting simple CUDA programs. Finally, areas where GPU programming achieves outstanding performance are mentioned and 2D convolution algorithm implementations are demonstrated.

3. CUDA programming:
Abstract: 2D and 3D convolutions are very important tools both for computer vision (e.g., for target tracking and for deep learning (convolutional NNs). Learn how to implement a 2D convolution between an image and a mask with CUDA.

4. OpenCV programming for object tracking:
Abstract: The first part of this tutorial will have an introduction to the OpenCV library using Python. Students can learn how to perform basic image processing operations, such as reading and displaying an image, extracting ROIs, applying filters etc. In the second part of the tutorial, the students will learn how to perform visual object tracking in video sequences, with correlation filter based tracking algorithms and OpenCV.

Part C (third day, 2 lectures, 1 programming exercise) 21/08/2020:
Drone planning and control

As drones execute missions (e.g., AV shooting, inspection), Part C lectures will focus on drone perception. Before mission execution, it is best simulated, using drone mission simulation. Such simulations will be presented using AirSim. Additionally a programming workshop on ROS and Gazebo simulations for drones will take place.

1. Drone mission simulations:
Abstract: Machine learning algorithms need large amounts of quality data to be trained efficiently. Gathering and annotating that sheer amount of data is a time-consuming and error-prone task. Those problems limit scale and quality. Synthetic data generation has become increasingly popular due to fast generation and automatic annotation.  To this end, in this lecture AirSim software and its capabilities will be presented.

2. Drone perception and estimation:
Abstract: This lecture will revise perception and estimation techniques for aerial robot applications. It will start describing main robot perception problems and summarizing the foundations of statistical multisensor estimation. Later, it will provide insights into the main perception techniques used in aerial robotics focusing on 6 DoF localization, mapping and SLAM problems. Finally, it will present examples where estimation techniques are synergistically combined with ML and DL methods.

3. ROS UAL Abstraction Layer for drone applications:
Abstract: This lecture will explain how to control a drone using ROS and Gazebo simulations. It will start with a brief introduction to ROS (topics and services) and Gazebo (worlds, models and plugins). Then it will present the UAL, an abstraction layer that provides a simple and easy interface to simulate and control drones in ROS. Finally, students will perform an exercise to practice with the learned tools.

4. Drone perception with ROS:
Abstract: This workshop will provide a hands-on approach to the basic concepts for implementing a perception pipeline targeted at aerial robots. The described methods will be programmed or analysed in a remote virtual machine running Ubuntu 16 and ROS (attendants should have a computer with the Google Chrome browser installed and a stable internet connection). The workshop will focus on how to interconnect the different submodules of the pipeline with ROS, and how to evaluate the system using a pre-recorded dataset. Additionally, notes on how to fuse information from other sensors such as an IMU or an altimeter will be provided. The developed code will be written in C++, using mainly the ROS, OpenCV and Eigen libraries.

5. Drone mission simulations in Airsim:
Abstract: In this workshop students will learn how to create and configure UAV mission simulations using the Airsim API. Data acquisition during a simulation will also be presented as well as all necessary steps to program a simulated UAV to perform an orbit around a target object.

IF I HAVE A QUESTION?

Contact

LECTURERS & TUTORS

Educational record of Prof. I. Pitas

Prof I. Pitas was Visiting/Adjunct/Honorary Professor/Researcher and lectured at several Universities: University of Toronto (Canada), University of British Columbia (Canada), EPFL (Switzerland), Chinese Academy of Sciences (China),  University of Bristol (UK), Tampere University of Technology (Finland), Yonsei University (Korea), Erlangen-Nurnberg University (Germany), National University of Malaysia, Henan University (China). He delivered 90 invited/keynote lectures in prestigious international Conferences and top Universities worldwide. He ran 17 short courses and tutorials on Autonomous Systems, Computer Vision and Machine Learning, most of them in the past 3 years in many countries, e.g., USA, UK, Italy, Finland, Greece, Australia, N. Zealand, Korea, Taiwan, Sri Lanka, Bhutan.

PAST COURSE EDITIONS

2019

Participants: 53, Countries: UK, Germany, Sweden, Norway, Italy, Greece, Croatia, Slovakia.

Registrant comments:

(Anonymous) “… The lectures during the workshops were really good. …”,

(Anonymous) “… Course material was very appealing and perfectly adequate. …”

SPONSORS

Sponsored by:

Aerial-Core, https://aerial-core.eu/

If you want to be our sponsor send us an email here: koroniioanna@csd.auth.gr

Other Aerial core schools:

IEEE RAS Summer School on Multi-Robot Systems

SAMPLE COURSE MATERIAL. RELATED LITERATURE

Target Detection (pdf)

1) Multidrone Project (MULTIple DRONE platform for media production), funded by the EU (2017-19), within the scope of the H2020 framework,  https://multidrone.eu/

2) Semi-Supervised Subclass Support Vector Data Description for image  and video classification,     V. Mygdalis, A. Iosifidis, A. Tefas, I.  Pitas, Neurocomputing, vol. 291, pp. 237-241, 2018

3) Face detection Hindering, P. Chriskos, J. Munro, V. Mygdalis, I. Pitas, Proceedings of the IEEE Global Conference on Signal and Information Processing (GLOBALSIP), Quebec, Montreal, 2017

4) 2D visual tracking for sports UAV cinematography applications, O. Zachariadis, V. Mygdalis, I. Mademlis, I. Pitas, Proceedings of the IEEE Global  Conference on Signal and Information Processing (GLOBALSIP), Montreal, Canada, 2017

5) Neurons With Paraboloid Decision Boundaries for Improved Neural Network Classification Performance, N. Tsapanos, A. Tefas, N. Nikolaidis and I. Pitas, IEEE Transactions on Neural Networks and Learning Systems (TNNLS), vol. 30, issue 1, pp. 284-294, 2019

6) Convolutional Neural Networks for Visual Information Analysis with Limited Computing Resources, P. Nousi, E. Patsiouras, A. Tefas, I. Pitas, Proceedings of the IEEE International Conference on Image Processing (ICIP), Athens, Greece, 2018

7) Overview of drone cinematography for sports filming, I. Mademlis, V. Mygdalis, C. Raptopoulou, N.Nikolaidis, N. Heise, T. Koch, T. Wagner, A. Messina, F. Negro, S. Metta, I.Pitas, European  Conference on Visual Media Production (CVMP), London, UK, 2017

8) Challenges in Autonomous UAV cinematography: An overview, I. Mademlis, V. Mygdalis, N. Nikolaidis, I. Pitas, Proceedings of the IEEE International Conference on Multimedia and Expo (ICME), San Diego, USA, 2018

9) Learning Multi-graph regularization for SVM classification, V.Mygdalis, A.Tefas, I.Pitas, Proceedings of the IEEE International Conference on Image Processing (ICIP), Athens, Greece, 2018

10) UAV Cinematography Constraints Imposed by Visual Target Trackers, I. Karakostas, I. Mademlis, N. Nikolaidis, I. Pitas, Proceedings of the IEEE International Conference on Image Processing (ICIP), Athens, Greece, 2018

11) Efficient camera control using 2D visual information for unmanned  aerial vehicle-based cinematography, N. Passalis, A. Tefas, I. Pitas, Proceedings of the IEEE International Symposium on Circuits and Systems (ISCAS), Florence, Italy, 2018

12) The future of media production through multi-drones’ eyes, A. Messina, S. Metta, M. Montagnuolo, F. Negro, V. Mygdalis, I. Pitas, J.  Capitán, A. Torres, S. Boyle, D. Bull, F. Zhang, International  Broadcasting Convention (IBC), Amsterdam, Netherlands, 2018

13) Quality Preserving Face De-Identification Against Deep CNNs, P.  Chriskos, R. Zhelev, V. Mygdalis, I. Pitas, Proceedings of the IEEE International Workshop on Machine Learning for Signal Processing (MLSP), Aalborg, Denmark, 2018

14) Improving Face Pose Estimation using Long-Term Temporal Averaging  for Stochastic Optimization, N. Passalis, A. Tefas, Proceedings of the International Conference on Engineering Applications of Neural Networks, EANN 2017, Athens, Greece, 2017

15) Discriminatively Trained Autoencoders for Fast and Accurate Face Recognition, P. Nousi, A. Tefas, Proceedings of the International Conference on Engineering Applications of Neural Networks, EANN, Athens, Greece, 2017

16) Concept Detection and Face Pose Estimation Using Lightweight Convolutional Neural Networks for Steering Drone Video Shooting, N. Passalis, A. Tefas, Proceedings of the European Signal Processing Conference (EUSIPCO), Kos, Greece, 2017

17) Human Crowd Detection for Drone Flight Safety Using Convolutional Neural Networks,     M.Tzelepi, A.Tefas, Proceedings of the European Signal  Processing Conference (EUSIPCO), Kos, Greece, 2017

18) Lightweight Two-Stream Convolutional Face Detection, D. Triantafyllidou, P. Nousi, A. Tefas, Proceedings of the European Signal Processing Conference (EUSIPCO), Kos, Greece, August, 2017

19) Fast Deep Convolutional Face Detection in the Wild Exploiting Hard Sample Mining, D. Triantafyllidou, P. Nousi, A. Tefas, Big Data Research, Elsevier, vol. 11, pp. 65-76, 2018

20) Learning Bag-of-Features Pooling for Deep Convolutional Neural   Networks, N. Passalis, A. Tefas, Proceedings of the International Conference on Computer Vision (ICCV), Venice, Italy, 2017

21) Self-Supervised Auto-encoders for Clustering and Classification, P. Nousi, A. Tefas, Evolving Systems Journal, Springer, pp 1–14, 2018

22) Unsupervised Knowledge Transfer using Similarity Embeddings, N. Passalis, A. Tefas,     IEEE Transactions on Neural Networks and Learning Systems (TNNLS), vol. 30, issue 3, pp. 946-950, 2018

23) Recurrent Attention for Deep Neural Object Detection, G.  Symeonidis, A. Tefas, Hellenic Conference on Artificial Intelligence (SETN), Rio Patras, Greece, 2018

24) Neural Network Knowledge Transfer using Unsupervised Similarity  Matching, N. Passalis, A. Tefas, Proceedings of the International Conference on  Pattern Recognition (ICPR), Beijing, China, 2018

25) Deep reinforcement learning for frontal view person shooting using  drones, N. Passalis, A. Tefas, Proceedings of the IEEE Conference on Evolving and Adaptive Intelligent Systems (EAIS), Rhodes, Greece, 2018

26) A Multidrone Approach for Autonomous Cinematography Planning, A. Torres-Gonzalez, J. Capitan, R. Cunha, A. Ollero and I. Mademlis, Proceedings of the Iberian Robotics Conference (ROBOT), 2017

27) Decentralized safe conflict resolution for multiple robots in dense scenarios, E. Ferrera, J. Capitán, A.R. Castaño and P.J. Marrón, Robotics and Autonomous Systems, vol. 91, pp. 179-193, 2017

28) Cooperative perimeter surveil using Bluetooth framework under communication constraints,     J.M. Aguilar, P. R. Soria, B.C. Arrue and A. Ollero, Proceedings of the Iberian Robotics Conference (ROBOT), 2017

29) Applying Frontier Cells Based Exploration and Lazy Theta* Path Planning over Single Grid-Based World Representation for Autonomous Inspection of Large 3D Structures with an UAS,     M. Faria,  I. Maza and A. Viguria, Journal of Intelligent & Robotic Systems, accapted Springer

30) Discriminative Optimization: Theory and Applications to Computer Vision Problems, J. Vongkulbhisal, F. De la Torre, and J. P. Costeira, IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), vol. 41, issue 4, pp. 829 – 843, 2018

31) Integrated Visual Servoing Solution to Quadrotor Stabilization and Attitude Estimation Using a Pan and Tilt Camera, D. Cabecinhas, S. Brás, R. Cunha, C. Silvestre, P. Oliveira, IEEE Transactions on  Control Systems Technology, vol. 27, issue 1, pp. 14-29, 2017

32) UAL: An Abstraction Layer for Unmanned Aerial Vehicles,  F.  Real, A. Torres-González, P. Ramón-Soria, J. Capitán and A. Ollero, Proceedings of the International Symposium on Aerial Robotics (ISAR), Philadelphia, PA, USA, 2018

33) Inverse Composition Discriminative Optimization for Point Cloud  Registration, J. Vongkulbhisal, B. I. Ugalde, F. De la Torre, J. P.  Costeira, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, USA, 2018

34) P. Chriskos, O.Zoidi, A.Tefas and I.Pitas, De-identifying facial images using singular value decomposition and projections, Multimedia Tools and Applications, Springer, vol. 76, issue 3, pp. 3435-3468, 2017

35) Cooperative Unmanned Aerial Systems for Fire Detection, Monitoring and Extinguishing, L. Merino, J.R. Martinez-de Dios, A.  Ollero, In “Handbook of Unmanned Aerial Vehicles”, ISBN 978-90-481-9706-4, Springer, 2015

36) Shot Type Feasibility in Autonomous UAV Cinematography, I.  Karakostas, I. Mademlis, N. Nikolaidis, I. Pitas, Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 2019

37) High-Level Multiple-UAV Cinematography Tools for Covering Outdoor Events, I. Mademlis, V. Mygdalis, N. Nikolaidis, M. Montagnuolo, F.  Negro, A. Messina, I. Pitas, IEEE Transactions on Broadcasting, accepted for publication, 2019

38) Autonomous Unmanned Aerial Vehicles Filming in Dynamic Unstructured Outdoor Environments, I. Mademlis, N. Nikolaidis, A.  Tefas, I. Pitas, T. Wagner, A. Messina,

IEEE Signal Processing Magazine, vol. 36, issue 1, pp. 147-153, 2019

39) Deep Convolutional Feature Histograms for Visual Object Tracking, P. Nousi, A. Tefas, I. Pitas, Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Brighton, UK, 2019

40) Semantic Map Annotation Through UAV Video Analysis Using Deep Learning Models in ROS, E. Kakaletsis, M. Tzelepi, P.I. Kaplanoglou, C. Symeonidis, N. Nikolaidis, A. Tefas, I. Pitas, Proceedings of the International Conference on Multimedia Modeling (MMM), Thessaloniki, Greece, 2019

41) Exploiting multiplex data relationships in Support Vector Machines, V. Mygdalis, A. Tefas, I. Pitas, Pattern Recognition, Elsevier, vol. 85, pp. 70-77, 2019

42) Deep reinforcement learning for controlling frontal person close-up shooting, N. Passalis, A. Tefas, Neurocomputing, Elsevier, vol. 335, pp. 37-47, 2019

43) Graph Embedded Convolutional Neural Networks in Human Crowd Detection for Drone Flight Safety, M. Tzelepi, A. Tefas, IEEE Transactions on Emerging Topics in Computational Intelligence, accepted for publication, 2019

44) Training Lightweight Deep Convolutional Neural Networks Using Bag-of-Features Pooling, N. Passalis, A. Tefas, IEEE Transactions on Neural Networks and Learning Systems, accepted for publication, 2018

Useful Links

Prof. Ioannis Pitas: https://scholar.google.gr/citations?user=lWmGADwAAAAJ&hl=el

Multidrone project:  https://multidrone.eu/

Icarus Research Team: https://icarus.csd.auth.gr/

Laboratory of Artificial Intelligence and Information Analysis: http://www.aiia.csd.auth.gr/

Department of Informatics, Aristotle University of Thessaloniki (AUTH): http://www.csd.auth.gr/en/

Thessaloniki: https://wikitravel.org/en/Thessaloniki