Application Note Description
Amazon Sagemaker is a fully managed service that provides the ability to build, train, and deploy machine learning models that can be deployed on FLIR Firefly-DL cameras. Sagemaker removes the heavy lifting from each step of the machine learning process to make it easier to develop high quality models. This application note aims to provide the following:
- Provide links to Amazon Web Services (AWS) resources on how to setup an AWS account and request resources (e.g. EC2).
- Provide some information on Sagemaker and how to setup a Notebook instance.
- Provide some information on S3 storage and how to a create storage bucket.
- Provide a working code example using Jupyter notebook to train an image classification model.
Preparing for use
Before you use your camera, we recommend that you are aware of the following resources available from our website:
- Camera Reference for the camera—HTML document containing specifications, EMVA imaging, installation guide, and technical reference for the camera model. Replace <PART-NUMBER> with your model's part number:
- Getting Started Manual for the camera—provides information on installing components and software needed to run the camera.
- Technical Reference for the camera—provides information on the camera’s specifications, features and operations, as well as imaging and acquisition controls.
- Firmware updates—ensure you are using the most up-to-date firmware for the camera to take advantage of improvements and fixes.
- Tech Insights—Subscribe to our bi-monthly email updates containing information on new knowledge base articles, new firmware and software releases, and Product Change Notices (PCN).
- Amazon Sagemaker Python SDK— You can find an overview and API documentation for Sagemaker Python SDK here. The project homepage is in Github: https://github.com/aws/Sagemaker-python-sdk, where you can find the SDK source and installation instructions for the library
- Amazon Sagemaker API Reference Documentation— Amazon Sagemaker service provides APIs for creating and managing Amazon Sagemaker resources. More information can be found here.
AWS Account Setup
To create an account:
1. Navigate to the Amazon AWS website: https://aws.amazon.com
2. If you don't already have one, create an AWS account. You can find some instructions on the account setup process here.
3. Sign in to your account console using the Sign In to the Console button.
All new AWS accounts are allocated several cloud compute resources by default. These resources include images, instances, volumes, and snapshots. When you create your AWS account, AWS sets default quotas (also referred to as limits) on these resources on a per-region basis. You can find a full list of the available resources to you by navigating to user profile menu and selecting My Service Quotas. More information regarding AWS EC2 resources can be found here.
To request AWS services:
1. Under AWS services, search for Amazon Elastic Compute Cloud (Amazon (EC2).
This brings you to a list of EC2 cloud compute services available in your region. You can also find your account specific applied quota for each service.
2. Search for "Running Dedicated P2 Hosts", which gives you access to an accelerated compute instance (a dedicated GPU machine which can speed up your training job). You can also find more information about other types of instances here.
3. By default the applied quota for a dedicated P2 host service is zero. Submit a request to AWS support to increase your dedicated P2 host service. You need a minimum of one dedicated P2 host instance to complete the training job given in the example code. You can find more information on how to request a limit increase here.
Note: the resource limit increase may take several days before it is applied.
S3 Storage Bucket
As part of the notebook example you need to provide an S3 storage bucket address. This cloud storage is used to store both your training images, as well as the trained model artifacts.
To create an S3 storage bucket:
1. Under AWS Services, navigate to your S3 console.
2. Create a new bucket.
3. Enter a name for your bucket and from the drop-down select a region/location. We recommend selecting a region that is the same as your dedicated P2 host service. Keep the default configuration options and permissions settings.
Note: keep a note of the bucket name. You need to provide this later when you run the notebook example.
Amazon Sagemaker is a fully managed machine learning service. It provides an integrated Jupyter notebook instance for ease of access to your data sources for exploration and analysis. It also provides common machine learning algorithms that are optimized to run efficiently against extremely large data in a distributed environment.
To set up your Sagemaker notebook:
1. Download the example Jupyter notebook and Python script.
2. From the AWS Find Services drop-down, launch Amazon Sagemaker.
3. On the Notebook instances tab, click Create notebook instance.
4. Enter a name for your notebook instance.
5. Under Permissions and encryption create a new IAM role and select Any S3 bucket (or enter your Specific S3 bucket) to allow the notebook instance access to your S3 bucket. Keep the other default settings.
6. Click Create notebook instance to initialize and create a new instance. It might take several minutes for the instance to initialize.
7. Click open Jupyter to start your Jupyter notebook.
8. Upload the example code files from step 1 to your notebook instance.
You can now launch the Jupyter notebook example (example_classification_tensorflow_Sagemaker.ipnb) inside your newly created instance, and run through the python code for training a model.
Jupyter Notebook Example for Sagemaker
What is Jupyter Notebook?
Jupyter Notebook is an open source web application that you can use to create and share documents that contain live code, equations, visualizations, and text.
How does it work?
The notebook consists of a sequence of cells. The cells could either be a Markdown cell (Text description) or a code cell (Python code). You can click on each code cell to highlight it, then click run (or ctrl-enter) to run the python code in the cell. When the cell is running it displays "In [*]" in the left margin. Once it has finished the * changes to a number such as "In " where the number represents the order in which the cell was run relative to the other cells. The output for each cell is printed out below the cell.
If you restart the Notebook kernel (or stop the notebook instance) the output from each run code cell is still displayed, however, you need to re-run the code cell to restore the actual output values.