Boto3 - cloud management system for Python

Very often there is a need to manage cloud systems from the script program. Similar tasks arise when deploying and maintaining a large number of instances in the cloud. Therefore, managing hundreds or thousands of images from the console or web admin panel is a rather laborious and difficult task, if done manually. Both system administrators and DevOps strive to maximize the mechanization of these processes for the deployment and management of cloud systems. This reduces the risk of errors during system maintenance and allows you to quickly redeploy the necessary instances in the event of a failure. 

Boto3 is one of the common and favorite cloud management systems for Amazon and is available for programming in Python. This SDK allows you to flexibly, simply, and efficiently write complex scripts to manage resources and automate the use of cloud systems.

What is Cloud Management System?

Cloud management refers to software and technologies designed to operate and monitor applications, data, and services located in the cloud. Cloud management tools and methodologies help ensure optimal and fast operation of resources in the form of cloud computing services and the correct interaction with users and other services. Such management technologies allow working with private, hybrid, and public cloud-based systems.

For example, you can write effective scripts for monitoring APIs or saving backups of important information, scaling cloud resources, collecting statistics from supported software, and so on. There is a fairly wide range of problems associated with infrastructure management in cloud systems, and in particular on AWS. 

The correct and predictable functioning of all services and the possibility of a detailed analysis of the infrastructure makes it possible to avoid risks when accompanying projects at all levels, including DevOps. Automation of cloud management at the level of script programs significantly reduces the risks of human errors associated with the manual management of cloud systems.

AWS solves these problems through script automation. Using libraries for scripting languages that can be easily turned on and used in understandable scripts is very useful when executing control commands that require a lot of manual action. To avoid errors in manual cloud management, users transfer the control load to a simple script that takes care of the necessary operations.

Why use Python for cloud management?

Python is one of the core technologies for DevOps. Its flexibility, simplicity, and accessibility make this language ideal for managing cloud systems. DevOps can automate actions with instances and other cloud resources, visualize data and improve their workflow with cloud management frameworks.

Other languages besides Python are used in DevOps, for example, shell, ruby, perl and so on, but not a single language in DevOp can provide its simplicity, efficiency, and reliability. That's why managing cloud systems is conveniently done in Python.

As mentioned in the article Why Python is a crucial part of the DevOps toolchain: “Because DevOps is an approach built for agility and for handling change, engineers need to embrace polyglotism. But there’s one language that’s coming out as a crucial component of the DevOps toolchain — Python. In this year’s Skill Up survey, publisher Packt found that Python was the primary language used by those working in DevOps. Indeed, it was a language that dominated across job roles – from web development to security to data science – a fact which underscores Python’s flexibility and adaptability. But it’s in DevOps that we can see Python’s true strengths. If DevOps is a modern, novel phenomenon in the software world, it’s significant that Python is the tool that DevOps practitioners share as a common language.”

Boto3 as an example of cloud management on Python

Boto3 provides unique Amazon cloud management capabilities with Python. This library is both very simple and very extensive, as it works with all possible AWS cloud services. In addition, it takes very little time to master it, since very good documentation with examples has been written. The list of library functions has more than two hundred sections, which allows you access to every necessary option for managing the cloud system from the Python script.

Boto3 makes it possible to turn instances from a script on and off, monitor the necessary resources, manage database systems, manage network resources, manage IoT systems, configure artificial intelligence systems, work with file storages, Alexa, manage lambda services, etc. This is just a small list of the possibilities of Boto3, and the beauty and ease of use of each individual function contribute to the wide dissemination of this framework in the practical tasks of DevOps. Boto3 has the following main features to manage AWS cloud from Python:

• Resources
• Collections
• Low-level Clients
• Paginators
• Session
• Credentials
• Configuration
• Extensibility Guide

The interface of Boto3 is based on two basic concepts - resources and collections. There is also an understandable mechanism for sessions and pagination.

Resources represent an object-oriented interface to Amazon Web Services (AWS). They provide a higher-level abstraction than the raw, low-level calls made by service clients. A collection provides an iterable interface to a group of resources. The process of sending subsequent requests to continue where a previous request left off is called pagination because Some AWS operations return results that are incomplete and require subsequent requests.

Here is an approximate and very shortened list of systems that Python from Boto3 allows managing.  These are only the most interesting and most popular AWS services, (the full list includes two hundred services):

• AlexaForBusiness: a low-level client representing Alexa For Business.
• APIGateway: Amazon API Gateway for creating, publishing, maintaining, monitoring, and securing REST and WebSocket APIs.
• AppConfig: Amazon AppConfig to create, manage, and quickly deploy application configurations.
• AutoScaling: monitors your applications and automatically adjusts capacity to maintain steady, predictable performance at the lowest possible cost
• Backup: scripts for a fully managed backup service that makes it easy to centralize and automate the back up of data across AWS services in the cloud.
• Budgets: ability to set custom budgets that alert you when your costs or usage exceed, or are forecasted to exceed your budgeted amount.
• CostExplorer: enables you to view and analyze your costs and usage. You can explore your usage and costs using the main graph, the Cost Explorer cost, and usage reports.
• Cloud9: script control for the cloud-integrated development environment (IDE), which allows you to create, run and debug code using only a browserю
• CloudDirectory: multi-tenant directory-based store in AWS. These directories scale automatically to hundreds of millions of objects as needed for applications
• CloudFormation: interaction with AWS CloudFormation, which makes it possible to use programming languages or a simple text file to automatically safely simulate and allocate all the resources needed for applications across all regions and user accounts.
• CloudFront: it’s a developer-friendly global content delivery network (CDN) service that delivers fast, secure data, video, applications, and APIs to customers around the world with low latency and high speed.
• CloudSearch: Amazon CloudSearch is a managed service in the AWS Cloud that makes it simple and cost-effective to set up, manage, and scale a search solution for your website or application.
• CloudWatch: Amazon CloudWatch is a monitoring and surveillance service that provides data and powerful analytic information to monitor applications and respond to system-wide performance changes.
• CodeBuild, CodeCommit, CodeDeploy: it is a fully managed continuous integration service that compiles source code, performs testing, and generates software packages ready for deployment
• DynamoDB: it is a database of key-value pairs and documents that provides a delay of less than 10 milliseconds when working at any scale
• EBS: Amazon Elastic Block Store (EBS) is an easy-to-use, high-performance block storage service built for use with Amazon Elastic Compute Cloud (EC2).
• EC2: it is a web service that provides secure scalable computing resources in the cloud.
• EMR: It is an industry-leading cloud-optimized big data platform that enables you to quickly and economically process massive amounts of data at any scale.
• ElasticsearchService: It is a fully managed service that allows you to easily and safely deploy and use Elasticsearch to process search queries at any scale.
• EventBridge: a serverless event bus that allows you to easily integrate applications using data from your own applications, integrated SaaS applications.
• IoT: AWS IoT Offers Extensive Capabilities Spanning Many Components - From Peripherals to the Cloud
• Kinesis: with Amazon Kinesis, you can simply collect, process and analyze streaming data in real-time to get analytic results in a timely manner and respond quickly to new information.
• Lambda: AWS Lambda lets you run code without provisioning and managing servers.
• MachineLearning: the broadest and most diverse set of machine learning and AI services for your business.
• NetworkManager: Transit Gateway network manager enables you to see network activity in many locations from one single dashboard.
• Pricing:  AWS Price List Service for calculating cloud services pricing.
• RDS: Amazon Relational Database Service (Amazon RDS) makes it easy to configure, use, and scale relational databases in the cloud.
• Route53: Amazon Route 53 is a highly available, scalable, cloud-based domain name system (DNS) web service.
• S3: Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading performance, scalability, availability, and data security.
• SimpleDB: Amazon SimpleDB is a highly accessible NoSQL data warehouse that fully provides database administration.
• ServiceDiscovery: allowing services to discover and interact with each other for microservice architecture applications.

For instance, the code example to create a bucket on S3 looks very clear and effective. It creates a bucket with the given name on the specified region (us-east-1 by default):

import logging
import boto3
from botocore.exceptions import ClientError

def create_bucket(bucket_name, region=None):
    # Create bucket
    try:
        if region is None:
            s3_client = boto3.client('s3')
            s3_client.create_bucket(Bucket=bucket_name)
        else:
            s3_client = boto3.client('s3', region_name=region)
            location = {'LocationConstraint': region}
            s3_client.create_bucket(Bucket=bucket_name,
                                    CreateBucketConfiguration=location)
    except ClientError as e:
        logging.error(e)
        return False

    return True

Or here is another example that reloads a specific EC2 instance from a Python script. The code is as simple and straightforward as possible:

import boto3
from botocore.exceptions import ClientError

ec2 = boto3.client('ec2')

try:
    ec2.reboot_instances(InstanceIds=['INSTANCE_ID'], DryRun=True)
except ClientError as e:
    if 'DryRunOperation' not in str(e):
        print("You don't have permission to reboot instances.")
        raise
try:
    response = ec2.reboot_instances(InstanceIds=['INSTANCE_ID'], DryRun=False)
    print('Success', response)
except ClientError as e:
    print('Error', e)

On the other hand, it can and should be noted that Boto3 is tied to the AWS infrastructure and the transition of the DevOps process to another cloud system will require the processing of Python scripts. This is not an easy task and will require a significant amount of time and effort. Therefore, when developing a cloud resource management system on Boto3, you need to take into account its interaction with this particular cloud system.

You can see in more detail and clearly the strength and simplicity of Boto3 in this wonderful article about working with S3 buckets “Python, Boto3, and AWS S3: Demystified.” 

Conclusion

In conclusion, using Boto3 for managing and monitoring a variety of cloud services is a relatively clear and practical task. This technology is really worth using in real projects, and our company has a positive experience in its practical application. 

Some DevOps talk about additional time resources for writing scripts for managing cloud systems but believe that each small script in 4-5 lines of code in Python can save you days and weeks of time manually managing the clouds, or finding errors that led to incorrect actions by system administrators. Boto3 is simple and effective, as practice shows, and our system administrators and DevOps will help you configure this technology in your project.