Workshop for student hackathons focused on IoT dev

Overview

Scenario: The Mutt Matcher (IoT version)

According to the World Health Organization there are more than 200 million stray dogs worldwide. The American Society for the Prevention of Cruelty to Animals estimates over 3 million dogs enter their shelters annually - about 6 dogs per minute! Anything that can reduce the time and effort to take in strays can potentially help millions of dogs every year.

Different breeds have different needs, or react differently to people, so when a stray or lost dog is found, identifying the breed can be a great help.

A Raspberry Pi with a camera

Your team has been asked by a fictional animal shelter to build a Mutt Matcher - a device to help determine the breed of a dog when it has been found. This will be an IoT (Internet of Things) device based around a Raspberry Pi with a camera, and will take a photo of the dog, and then use an image classifier Machine learning (ML) model to determine the breed, before uploading the results to a web-based IoT application.

This device will help workers and volunteers to be able to quickly detect the breed and make decisions on the best way to approach and care for the dog.

An application dashboard showing the last detected breed as a German wire pointer, as well as a pie chart of detected breeds

The animal shelter has provided a set of images for a range of dog breeds to get you started. These can be used to train the ML model using a service called Custom Vision.

Pictures of dogs

Prerequisites

Each team member will need an Azure account. With Azure for Students, you can access $100 in free credit, and a large suite of free services!

Your team should be familiar with the following:

Hardware

To complete this workshop fully, ideally you will need a Raspberry Pi (model 3 or 4), and a camera. The camera can be a Raspberry Pi Camera module, or a USB web cam.

💁 If you don't have a Raspberry Pi, you can run this workshop using a PC or Mac to simulate an IoT device, with either a built in or external webcam.

Software

Each member of your team will also need the following software installed:

Resources

A series of resources will be provided to help your team determine the appropriate steps for completion. The resources provided should provide your team with enough information to achieve each goal.

These resources include:

  • Appropriate links to documentation to learn more about the services you are using and how to do common tasks
  • A pre-built application template for the cloud service part of your IoT application
  • Full source code for your IoT device

If you get stuck, you can always ask a mentor for additional help.

Exploring the application

Icons for Custom Vision, IoT Central and Raspberry Pi

The application your team will build will consist of 3 components:

  • An image classifier running in the cloud using Microsoft Custom Vision

  • An IoT application running in the cloud using Azure IoT Central

  • A Raspberry Pi based IoT device with a camera

The application flow described below

When a dog breed needs to be detected:

  1. A button on the IoT application is clicked

  2. The IoT application sends a command to the IoT device to detect the breed

  3. The IoT device captures an image using it's camera

  4. The image is sent to the image classifier ML model in the cloud to detect the breed

  5. The results of the classification are sent back to the IoT device

  6. The detected breed is sent from the IoT device to the IoT application

Goals

Your team will set up the Pi, ML model and IoT application, then connect everything to gether by deploying code to the IoT device.

💁 Each goal below defines what you need to achieve, and points you to relevant on-line resources that will show you how the cloud services or tools work. The aim here is not to provide you with detailed steps to complete the task, but allow you to explore the documentation and learn more about the services as you work out how to complete each goal.

  1. Set up your Raspberry Pi and camera: You will need to set up a clean install of Raspberry Pi OS on your Pi and ensure all the required software is installed.

    💻 If you are using a PC or Mac instead of a Pi, your team will need to set this up instead.

  2. Train your ML model: Your team will need to train the ML model in the cloud using Microsoft Custom Vision. You can train and test this model using the images that have been provided by the animal shelter.

  3. Set up your IoT application: Your team will set up an IoT application in the cloud using IoT Central, an IoT software-as-a-service (SaaS) platform. You will be provided with a pre-built application template to use.

  4. Deploy device code to your Pi: The code for the IoT device needs to be configured and deployed to the Raspberry Pi. You will then be able to test out your application.

    💻 If you are using a PC or Mac instead of a Pi, your team will need to run the device code locally.

💁 The first 3 goals can be worked on concurrently, with different team members working on different steps. Once these 3 are completed, the final step can be worked on by the team.

Validation

This workshop is designed to be a goal-oriented self-exploration of Azure and related technologies. Your team can validate some of the goals using the supplied validation scripts, and instructions are provided where relevant. Your team can then validate the final solution by using the IoT device to take a picture of one of the provided testing images and ensuring the correct result appears in the IoT application.

Where do we go from here?

This project is designed as a potential seed for ideas and future development during your hackathon. Other hack ideas for similar IoT devices that use image classification include:

  • Trash sorting into landfill, recycling, and compost.

  • Identification of disease in plant leaves.

  • Detecting skin cancer by classification of moles.

Improvements you could make to this device include:

  • Adding hardware such as a button to take a photograph, instead of relying on the IoT application.

  • Adding a screen or LCD display to the IoT device to show the breed.

  • Migrating the image classifier to the edge to allow the device to run without connectivity using Azure IoT Edge.

Learn more

You can learn more about using Custom Vision to train image classifiers and object detectors using the following resources:

You can learn more about Azure IoT Central using the following resources:

If you enjoy working with IoT, you can learn more using the following resource:

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

Owner
Microsoft
Open source projects and samples from Microsoft
Microsoft
Интеграция Home Assistant с ЛК "Интер РАО"

ЕЛК ЖКХ «Интер РАО» для Home Assistant Предоставление информации о текущем состоянии ваших аккаунтов в ЕЛК ЖКХ. Введение @ TODO @ Установка Посредство

Alexander Ryazanov 27 Nov 05, 2022
Code for the onshape macropad.

Onshape_Macropad Code for the onshape macropad. This is a macropad built using the Pimoroni Keybow and the KPrepublic Enclosure. pimoroni_keybow kprep

Justin Cole 1 Nov 23, 2021
[unmaintained] WiFi tools for linux

Note: This project is unmaintained. While I would love to keep up the development on this project, it is difficult for me for several reasons: I don't

Rocky Meza 288 Dec 13, 2022
Create a low powered, renewable generation forecast display with a Raspberry Pi Zero & Inky wHAT.

GB Renewable Forecast Display This Raspberry Pi powered eInk display aims to give you a quick way to time your home energy usage to help balance the g

Andy Brace 32 Jul 02, 2022
Aqara Camera G3 integration for Home Assistant

Aqara Camera G3 integration for Home Assistant ATTENTION: The component only works after enabled telnet. Only supportd stream. Not support still image

14 Dec 18, 2022
My 500 LED xmas tree

xmastree2020 This repository contains the code used for Matt's Christmas tree, as featured in "I wired my tree with 500 LED lights and calculated thei

Stand-up Maths 581 Jan 07, 2023
iot-dashboard: Fully integrated architecture platform with a dashboard for Logistics Monitoring, Internet of Things.

Fully integrated architecture platform with a dashboard for Logistics Monitoring, Internet of Things. Written in Python. Flask applicati

2 Jul 29, 2022
LifeSaver automatically, periodically saves USB flash drive data into the PC

LifeSaver automatically, periodically saves USB flash drive data into the PC. Theoriticaly it will work with any any connected drive ex - Hard Disk ,SSD ... But, can't handle Backing up multipatition

siddharth dhaka 4 Sep 26, 2021
Intel Realsense t265 into Unreal Engine

t265_UE Intel Realsense t265 into Unreal Engine. Windows only, and Livelink plugin is 4.26.2 only at the moment. Might recompile it for different vers

Bjarke Aagaard 30 Jan 02, 2023
GUI wrapper designed for convenient service work with TI CC1352/CC2538/CC2652 based Zigbee sticks or gateways. Packed into single executable file

ZigStar GW Multi tool is GUI wrapper firtsly designed for convenient service work with Zig Star LAN GW, but now supports any TI CC1352/CC2538/CC2652 b

133 Jan 01, 2023
Python module for the qwiic serial control motor driver

Qwiic_SCMD_Py Python module for the qwiic motor driver This python package is a port of the existing SparkFun Serial Controlled Motor Driver Arduino L

SparkFun Electronics 6 Dec 06, 2022
Custom component for MPC-HC for home-assistant

mpc_hc The current mpchc integration in homeassistant violates ADR0004, so it will be deleted from core. This is just the existing integration copied

3 Dec 15, 2022
Workshop for student hackathons focused on IoT dev

Scenario: The Mutt Matcher (IoT version) According to the World Health Organization there are more than 200 million stray dogs worldwide. The American

Microsoft 15 Aug 10, 2022
ArucoFollow - A script for Robot Operating System and it is a part of a project Robot

ArucoFollow ArucoFollow is a script for Robot Operating System and it is a part

5 Jan 25, 2022
A script for performing OTA update over BLE on ESP32

A script for performing OTA update over BLE on ESP32

Felix Biego 18 Dec 15, 2022
Get input from OLED Joystick, Runs command, Displays output on OLED Screen (Great for P4wnP1)

p4wnsolo-joyterm Gets text input from OLED Joystick Runs the command you typed Displays output on OLED Screen (Great for P4wnP1 - even better on Raspb

PawnSolo 7 Dec 19, 2022
Python code written to utilize the Korlan usb2can hardware to send and receive data over the can-bus on a 2008 Nissan 350z

nissan_ecu_hacking Python code written to utilize the Korlan usb2can hardware to send and receive data over the can-bus on a 2008 Nissan 350z My goal

Liam Goss 11 Sep 24, 2022
Alarm Control Panel component for Zigbee Keypads using action_transaction field

hass_transaction_alarm_panel Alarm Control Panel component for Zigbee Keypads using action_transaction field. Works together with zigbee2mqtt Supporte

Konstantin 4 Jun 09, 2022
3D-printable hexagonal mirror array capable of reflecting sunlight into arbitrary patterns

3D-printable hexagonal mirror array capable of reflecting sunlight into arbitrary patterns

Ben Bartlett 2.3k Dec 30, 2022
MicroPython driver for 74HC595 shift registers

MicroPython 74HC595 A MicroPython library for 74HC595 8-bit shift registers. There's both an SPI version and a bit-bang version, each with a slightly

Mike Causer 17 Nov 29, 2022