Experiment with Vision Sensing Using fitlet2 and Pixy2

Moderators: hassellbear, Andrey.Mazlin

Post Reply
hassellbear
Posts: 106
Joined: Mon May 28, 2012 12:25 pm

Experiment with Vision Sensing Using fitlet2 and Pixy2

Post by hassellbear »

Overview
Vision Sensing.jpg
Vision Sensing.jpg (8.84 KiB) Viewed 15177 times
Vision Sensing represents by far the best overall method of perceiving the surrounding environment. While other sensory technologies such as millimeter wave radar, lidar, acoustic, thermal, chemical , and so on are very useful, none of them provides the level, quality, and variety of information that vision does.

Previously, electro-optical sensory systems were quite expensive and were mostly found in military hardware such as the AGM-65 Maverick missile and similar systems deployed around the world. Now however, affordable and capable vision sensing devices are available to the rest of us. One such device is the Pixy2 camera which is capable of line following and object recognition. Available for around $60 US, Pixy2 is a great addition to small, capable platforms such as fitlet2.

Older Machine Vision Sensing Technology - Expensive and Limited to Military Applications
Maverick.jpg
Maverick.jpg (30.48 KiB) Viewed 15177 times

Pixy2 - Modern Machine Vision Sensing Technology - Affordable and Available to All
Pixy2.jpg
Pixy2.jpg (46.29 KiB) Viewed 15177 times


About Pixy2

Pixy2 is the latest incarnation of Pixy which was born of a partnership between the Carnegie Mellon Robotics Institute and Charmed Labs. The hardware and software utilized in Pixy2 are open source, so it is an ideal device for use by the makers among us.
Pixy2 Open Source.jpg
Pixy2 Open Source.jpg (8.58 KiB) Viewed 15174 times


Pixy2 has a couple of sensing modes that are quite useful for the maker - especially for robotics applications. These modes are:

1. Object Detection and Recognition

Pixy2 learns and detects objects based on colors using an algorithm known as Color Connected Components. Color based object recognition was utilized because of its efficiency, speed, and relatively "bullet-proof" performance. Pixy2 has the ability to simultaneously detect and track a large number of objects simultaneously. This is a capability that was once reserved for only very complex and expensive military systems.
Orange.png
Orange.png (126.91 KiB) Viewed 15177 times
2. Line Tracking

Pixy2 has the ability to track and follow lines using its custom line tracking algorithm. Unlike photodetector based solutions, Pixy2's vision based line tracking solution has the ability to anticipate by looking ahead and the ability to easily identify and navigate through intersections.
Line Tracking 1.png
Line Tracking 1.png (42.06 KiB) Viewed 15177 times


Hardware Used
Hardware.jpg
Hardware.jpg (26.33 KiB) Viewed 15177 times
The hardware used for this demonstration is:

1. fitlet2
2. Pixy2 camera system
3. Pixy2 Pan-Tilt kit



Software

The basic software package for configuring and experimenting with Pixy2 is PixyMon. It is available for Linux, Mac, and Windows. This app note focuses on the Linux version. The preferred distro for the Linux version of PixyMon is Linux Mint which is what I use.
LM Friendly.jpg
LM Friendly.jpg (6.85 KiB) Viewed 15174 times
Software Installation

Detailed instructions for installing PixyMon in Linux are found here:

https://docs.pixycam.com/wiki/doku.php? ... n_on_linux

In my case I installed PixyMon in /usr/local/src/pixy2 in order to adhere to what I perceive to be standard Linux practice - although there are many different opinions concerning what is standard practice in Linux.

Also, I wrote a short script PixyMon.sh which I made executable and placed in /home/[username]/bin for universal access.

PixyMon.sh
#!/bin/bash
# Script to Run PixyMon
cd /
./usr/local/src/pixy2/build/pixymon/PixyMon




Test Results

Pixy2 in conjunction with fitlet2 proved to be a fun and interesting system to explore. I was impressed with Pixy2's ability to accurately and consistently detect, identify, and track objects. The following images and videos provide more detail.

1. Object Detection, Recognition, and Tracking

Basic Shape/Color Recognition - Primary Colors
Primary.jpg
Primary.jpg (143.77 KiB) Viewed 15174 times
Basic Shape/Color Recognition - Blended Secondary Colors
Secondary.jpg
Secondary.jpg (95.87 KiB) Viewed 15174 times
[/size]

Basic Shape/Color Recognition - Multiple Objects
Multi.jpeg
Multi.jpeg (150.4 KiB) Viewed 15174 times
Basic Shape/Color Tracking - PixyMon - Ripe Tomato


Basic Shape/Color Tracking - Pixy2 Pan/Tilt Kit - Ripe Tomato



2. Line Tracking

Since I am using Pixy2 in a more or less static configuration, I was not able to make a detailed exploration of its line tracking performance. However, the following picture shows that Pixy2 does its best to search its field of view for lines to follow.
Line Tracking.png
Line Tracking.png (65.52 KiB) Viewed 15174 times


Potential Projects

1. Lunar Tracker for aiming "Moon Bounce" laser and microwave transmissions.
2. Laser tracker for mobile platform.
3. Automated fruit/vegetable harvesting system using color/shape recognition.
4. Vehicle "road ahead" hazard detection, recognition, and warning system.
5. Etc.



Actual Line Following Project (By Others)

Personally, I think a fitlet2 would make a fine field-hardened replacement for the laptop used in the following video.





Resources for Developers, Experimentors, and Makers

APIs, Libraries, and Software
https://pixycam.com/downloads-pixy2/

Wiki
https://docs.pixycam.com/wiki/doku.php?id=wiki:v2:start

Forums
https://forum.pixycam.com/




Where Do fitlet2 and Pixy2 Go From Here?
Where To.jpeg
Where To.jpeg (4.17 KiB) Viewed 15174 times
That remains to be seen, but I bet they could be the foundation for a Very Cool Robot!!

tamir
Site Admin
Posts: 419
Joined: Mon Dec 25, 2017 4:21 pm

Re: Experiment with Vision Sensing Using fitlet2 and Pixy2

Post by tamir »

Very interesting stuff right there.
Thank you for this enriching post.

Post Reply

Return to “Cool stuff with fitlet2”