Cloud computing for neural networks. How to run Jupyter notebook with Tensorflow in the cloud
by Svitla Team
In practice, quite often there will be a need to run Jupyter Notebook in the cloud, wanting to spend as little effort as possible on it and still get a reliable and efficient computing environment.
This task can be useful both to students for practical work and in real projects for solving real problems.
So, let's choose one of the cloud services to run Python in the Jupyter Notebook and find out what other features are available in this environment.
It’s very good if such a cloud service is available in a free account.
Cloud computing for AI and ML
There is no doubt that now in many cloud systems all kinds of solutions are available for machine learning and artificial intelligence.
These cloud solutions can be multi-purpose, for solving general problems, or specialized for solving narrow problems such as in the fields of natural language processing, computer vision, etc.
Such solutions are now available on the market from major cloud service providers. Therefore, the cloud provider described in this article is an example, and you can find a similar approach for running Jupyter Notebook for your cloud provider.
Why Tensorflow? Because it allows the following functionality:
- one of the most powerful computing environments
- system using dataflow
- ability to use GPU
- the availability of ready-made and proven mathematical methods for neural networks
- high-level access in Python
- availability of rapid prototyping system on Keras
Well, in general, this system is very common in the community, is used in huge projects and is free to use.
Why Jupyter notebook?
Jupyter Notebook is a very interesting IDE, In fact, it is also the de facto standard for developers of neural networks, machine learning, and artificial intelligence.
Jupyter Notebook has very interesting properties as a development environment:
- has the ability to run in the browser
- can be installed on the server and on the local computer
- each piece of code can be run separately
- can draw graphics and display photos
- support interactive data science and scientific computing
- extensible solutions by installing libraries
- various programming languages can be used (Python, R, Julia)
- it is free
Finding Cloud solution with Jupyter
As we have said, the cloud solution for launching Jupyter Notebook is already available from many leading vendors, including:
- Amazon's SageMaker Notebooks
- Google's Colaboratory
- Microsoft's Azure Notebook
- and Jupyo
Let's dwell on one of the popular cloud providers, Google Colab. This cloud provider gives you access to Jupyter Notebook for free and out of the box and has very remarkable properties.
Running Jupyter Notebook
In order to launch Jupyter Notebook on Google Colab, you need to go to https://colab.research.google.com
Fig. 1. Opening Google Colab
In this dialog box, you will get the opportunity to choose from ready-made examples in Google Colab or download the file from your computer.
It is also possible to use your Google Drive for the project file. It's a great combination of two cloud services, isn't it?
And besides all these great features, there is another great opportunity to use your repository in git.
By opening the page in Google Colab https://colab.research.google.com/notebooks/welcome.ipynb, you can find instructions on the main functions of the Jupyter Notebook in this cloud environment.
Fig. 2. Welcome to the Colaboratory tutorial in Jupyter Notebook
In addition, you can configure the runtime kernel in two more ways: hosted runtime and local runtime. This makes it possible to use another server (for example, on your computer) for the computing runtime.
Fig. 3. Connecting to hosted or local runtime
Running Tensorflow, Numpy, Scipy, and OpenCV
Let's open a new project in Jupyter Notebook via the “File” menu (“New Python 3 Notebook”).
Fig. 4. Opening a new Notebook in Google Colab
Now, let's enter two lines of code into the cell and click the circle with an arrow to launch Tensorflow. This Python code will print the version of the Tensorflow, in our case 1.14.0
Fig. 5. Running Tensorflow library in Jupyter Notebook
And in the same way, you can run numpy, scipy, and opencv.
Fig. 6. Running numpy, scipy, and opencv on Jupyter Notebook
And what if your favorite Python library is not installed in Google Colab? There is a way to fix this.
!pip install yourlibrary
Where yourlibrary is the name of your library. We show this using the seaborn library (statistical data visualization - seaborn) as an example.
Fig. 7. Installing a new Python library in Google Colab
Additional features of Google Colab
It is worth noting separately the following features in Google Colab.
These are such great features for a free account. For instance, for working with data Google Colab allows:
- Loading data: Google Drive, Sheets, and Google Cloud Storage
- Charts: visualizing data
- Using BigQuery
And, in addition to this, again, in a free account, using accelerated hardware is available:
- TensorFlow with GPUs
- TensorFlow with TPUs
To do this, turn on the GPU. In the menu Edit-> Notebook settings, select the option Hardware Accelerator -> GPU.
Fig. 8. Setting Google Colab to use GPU
And then run this code to check for the presence of the GPU:
Fig. 9. Testing GPU on Google Colab
In the same way, you can use TPU (Tensor Processing Unit) on Google Colab.
Let's summarize the possibilities of developing machine learning and artificial intelligence projects in cloud environments. We took only one cloud provider as an example. But you can take advantage of other solutions that provide similar functionality to run Jupyter Notebook in the cloud. What are the benefits of this:
- firstly, you can work with a Notebook from any computer
- store project files on Google Drive
- use the coolest libraries for Python
- use about 12 Gb of memory for runtime
- if necessary, deliver the necessary libraries
- switch to Tensorflow version 2
- build graphs and display photos
- and soon the implementation for the AR language will be released
- lead joint development using the GIT repository
- use GPU and TPU even in a free account
- switch to the required runtime on your computer
And this cloud solution can be both an additional development environment for your project, and one of the main effective tools.
Let's meet Svitla
We look forward to sharing our expertise, consulting you about your product idea, or helping you find the right solution for an existing project.
Your message is received. Svitla's sales manager of your region will contact you to discuss how we could be helpful.