The Basics of TensorFlow: Understanding Tensors, Graphs, Sessions, and Operations

When it comes to building AI systems, TensorFlow has become ubiquitous. It has grown to become the library of choice for most machine learning and deep learning projects.  Researchers, programmers, and data scientists even predict that the library will gain even more traction in the coming years. 

But truth be told, TensorFlow can get complex when you attempt to go beyond ready-made examples and try out things from scratch. Without using APIs, applying the key concepts and methods of TensorFlow to build models can be slightly advanced. Without a doubt, you must have a firm understanding of the basics of TensorFlow before you can begin to build models with fresh ideas. 

 And that’s why in this tutorial, we shall explain the very basics of TensorFlow. We shall go bare to explain when Tensors are and how to navigate the TensorFlow library to do the basic stuff. By the end of this tutorial, you will learn:

1. What is a tensor 
2. How tensors are represented 
3. The components of s TensorFlow program
4. Creating tensors in TensorFlow
5. Creating special tensors 
6. The shape and datatype of a tensor 
7. How to change the shape of a tensor 
8. How to change the datatype of a tensor
9. Running operations in TensorFlow with tf.Session()
10. Mathematical operations in TensorFlow

Sure, you’re excited to get on this journey. Let’s get started. 

We’ll begin by understanding what Tensors are. 

What is a Tensor?

We start with this because the name TensorFlow is derived from the word tensor. A tensor is simply a matrix of any given dimension which represents all forms of data. A tensor can be defined by its shape, which is the number of sides in the tensor. The shape of a tensor is called its dimensionality. 

Putting it differently, we can define a tensor as a multidimensional array. A tensor could be zero-dimensional (0-D) as we have in scalar values. It could be one-dimension (1-D) as in a line or vector, it could be two-dimensional (2-D) as in a matrix, it could be three-dimensional (3-D) as we have in the color in images, and so on. 

The TensorFlow library represents all data structures as tensors. In other words, whatever form of data you feed into the TensorFlow library is represented as a tensor. The library is called TensorFlow is what really happens in the flowing of tensors in a computational graph. A computational graph is a representation that indicates how the computation takes place. It shows how every operation (called the op node) is linked to each other. We will get to discuss graphs in more detail momentarily. 

How are Tensors Represented

Tensors are represented in TensorFlow as a collection of arrays in a given dimension. Say, we have a 4 by 2 matrix given below.

1 2 3 4 5 6 7 8 

In TensorFlow, it is written as 

[[1, 2],

    [3, 4],

    [5, 6],

    [7, 8]]

If it were are 3 by 3 matrix which can be visualized in a vector space like this

It is written in TensorFlow as 

[ [[1, 2],  

       [[3, 4],  

       [[5, 6],  

       [[7,8] ]

Soon enough, we will discuss how to create and store these tensors using TensorFlow. Note that tensors with zero dimension (a scaler) or dimensions greater than 3 are difficult to visualize since we as humans work in a 3-dimensional space. TensorFlow can, however, work with tensors of higher dimensions. 

The Components of a TensorFlow Program

A TensorFlow program can be split into 3 main components. 

1. Computational Graph: The computational graph can be seen as where all mathematical operations are performed. A computational graph comprises nodes and edges. The nodes in a graph indicate where the operations on tensors are carried out to return new tensors. 
2. Tensor: The data returned after each operation is called the tensors. As various operations progress, new tensors are outputted. The edges of a computational graph are the tensors. 
3. Session: The session runs the defined operations from the computational graph. Inside a session, you can either print a tensor or run operations on an operation to return an output tensor. 

Let’s go get into coding with TensorFlow. 

Creating Tensors in TensorFlow

As mentioned earlier, TensorFlow carries out all its computations using tensors. When defining a tensor in TensorFlow, 3 properties are fundamental. 

First, the name of the tensor (name). Second, the dimension (shape). And lastly, the data type (dtype). There are four types of tensors you can create with TensorFlow. 

1. tf.constant
2. tf.Variable or tf.get_variable
3. tf.placeholder
4. tf.SparseTensor

Before we go on to discuss how to create these tensors in TensorFlow, we will start by importing the library. You import TensorFlow with the following import statement. 

#import TensorFlow library
import tensorflow as tf

‘tf’ is called an alias. It is not compulsory to use an alias when working with python libraries, but it is recommended. It is particularly useful when the library’s name is a lengthy word. It can be replaced with shorter words, which makes your code cleaner. Note that you are not constrained to use ‘tf’ as ‘tensorflow’ alias but since it is widely used, we will stick to it throughout the course of this tutorial. 

If the import statement above gives a ‘ModuleNotFound’ error, it means that you do not have TensorFlow installed on your machine. You can refer to our previous tutorial where we discussed how to download and install TensorFlow on your machine.

Creating Tensors in TensorFlow with tf.constant

You can create a constant tensor of n-dimension using the tf.constant() method. As mentioned earlier, the method takes in 3 key parameters, the value of the tensor, the data type, and the name of the tensor.

tf.constant(
    value,
    dtype=None,
    shape=None,
    name='Const',
    verify_shape=False,
)
Docstring:
Creates a constant tensor.

The resulting tensor is populated with values of type `dtype`, as
specified by arguments `value` and (optionally) `shape` 

It is important to state the value of the tensor. The other parameters are given default values by TensorFlow. Let’s say you want to create a one-dimensional tensor populated with numbers 1 to 10. You can write the following code. 

#print a rank 1, one-dimensional tensor populated from number 1 to 10
tf.constant([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], dtype=tf.float32)

Output:

<tf.Tensor 'Const_2:0' shape=(10,) dtype=float32>

Note: The output printed is not the tensor but the tensor object, with its properties. This is because the TensorFlow object has not been run yet. We will get to how to run a tensor using the tf.Session() class later in this tutorial. 

Let’s take another example. Say this time, you want to create a 3 by 3 matrix populated with a constant value, 2. We can as well use the tf.constant() method. The value argument will be given as 2 while the shape argument will be given as (3, 3). The code to do this is written below. 

#print a rank 1, 3 by 3 tensor populated with ones
tf.constant(value=1, shape=(3, 3))

Output:

<tf.Tensor 'Const_3:0' shape=(3, 3) dtype=int32>

Another way to go about this is to explicitly pass the matrix as the value argument. See this code below. 

tf.constant([[1, 1, 1,],
            [1, 1, 1],
            [1, 1, 1]])

Output:

<tf.Tensor 'Const_4:0' shape=(3, 3) dtype=int32>

Creating Tensors with String and Boolean Data Types

We have been creating tensors of int and float data types. But you can also create tensors of string or Boolean dtype. 

#create TensorFlow with Boolean dtype
tf.constant([False, True, True, False])

Output:

<tf.Tensor 'Const_7:0' shape=(4,) dtype=bool>

As seen in the output, the tensor has a Boolean dtype. 

If we want to create a tensor with the string data type, it is as easy as writing

#create TensorFlow with string dtype
tf.constant(['Tensors'])

Output:

<tf.Tensor 'Const_8:0' shape=(1,) dtype=string>

Creating Special Tensors

Examples of special tensors are identity tensors or tensors filled with ones or zeros.  Since this kind of tensor is identified by a particular format, TensorFlow has methods that can be used to create them easily. We are only required to specify the shape of the tensor we want to create. 

To create a tensor of ones, we use the tf.ones() method. It takes a compulsory parameter of the shape of the tensor to be created. To create a 1 by 10 tensor, populated with one, see the code below

#create tensor populated with ones
tf.ones(10)

Output:

<tf.Tensor 'ones:0' shape=(10,) dtype=float32>

If we want to create tensors with more rows and columns greater than 1, we can pass the number of rows and columns as a list. The code below creates a 3 by 3 tensor populated with ones. 

#create a 3 by 3 tensor populated with ones
tf.ones([3, 3])

Output:

<tf.Tensor 'ones_1:0' shape=(3, 3) dtype=float32>

To create a tensor with all elements populated with zeros, we use the tf.zeros() method. Like the tf.ones() method, this method requires the shape of the tensor passed as an argument. 

Let’s see an example. To create a tensor with all elements set to zero, the code below does just that. 

#create a tensor populated with zeros
tf.zeros(10)

Output:

<tf.Tensor 'zeros_1:0' shape=(10,) dtype=float32>

Finally, let’s discuss identity tensor. For those who do not know, an identity tensor is a square tensor whose diagonal elements are ones while the other elements are zeros. An important property of an identity tensor is that when an identity tensor is multiplied with a tensor, it returns the tensor itself.

To create an identity tensor in TensorFlow, we use the tf.eye() method. Let’s create a 4 by 4 identity tensor. 

#create a 4 by 4 identity tensor
tf.eye(4)

Output:

<tf.Tensor 'eye/TensorDiag:0' shape=(4, 4) dtype=float32>

The Shape and Datatype of a Tensor

We mentioned that a tensor is defined by three properties: its value, shape, and datatype. When you create a tensor of some given value, the shape and datatype of the tensor are guessed by the TensorFlow algorithm. 

However, if you wish to print the shape of a tensor, you can do so with the shape attribute of the tensor. Let’s take an example

#create a 2 by 3 tensor
tensor_a = tf.constant([[1, 2, 3],
                       [4, 5, 6]])
#print the shape of the tensor
print(tensor_a.shape)

Output:

TensorShape([Dimension(2), Dimension(3)])

As seen in the output, the tensor has 2 rows and 3 columns.

If you wish to print the datatype for a tensor, on the other hand. You can do this with the dtype attribute of the tensor. Let’s see an example. 

#create a 2 by 3 tensor
tensor_a = tf.constant([[1, 2, 3],
                       [4, 5, 6]])
#print the datatype of the tensor
print(tensor_a.dtype)

Output:

<dtype: 'int32'>

As seen in the result, the datatype of the tensor is int32. But what if we wish to change the shape of data type of an already created tensor. How do we go about this?

Changing the Shape of a Tensor. 

There may be situations where you need to change the shape of a tensor. In Convolutional Neural Networks (CNN), you will be required to shape the shape of the input data (image array) to a different shape. To change the shape of a tensor, you use the tf.reshape() method. This method takes 2 important arguments: the tensor you want to reshape and the new shape you want for the tensor. See the example below. 

#create a 2 by 3 tensor
tensor_a = tf.constant([[1, 2, 3],
                       [4, 5, 6]])
#reshape the above tensor into a 3 by 2 tensor
tensor_a_reshaped = tf.reshape(tensor_a, [3, 2])
#print the shape of the reshaped tensor
print(tensor_a_reshaped.shape)

Output:

(3, 2)

Changing the Datatype of a Tensor

As seen from previous examples, upon creating a tensor, TensorFlow assigns a data type automatically. You can decide to change the datatype to another one using the tf.cast() method. The tf.cast() method takes two important arguments: the tensor to cast and the new dtype. Let’s see an example. 

#create a 2 by 3 tensor
tensor_a = tf.constant([[1, 2, 3],
                       [4, 5, 6]])
#print datatype for the tensor
print('The dtype of the initial tensor is ', tensor_a.dtype)

#change the datatype to float32
tensor_a_changed_dtype = tf.cast(tensor_a, tf.float32)
#print the datatype of the cast tensor
print('The dtype of the cast tensor is ', tensor_a_changed_dtype.dtype)

Output:

The dtype of the initial tensor is  <dtype: 'int32'>
The dtype of the cast tensor is  <dtype: 'float32'>

Running Operations in TensorFlow with tf.Session()

Before now, we have not printed the tensors created on the terminal. When we attempt to print the tensors, it returns its properties, i.e. its name, shape, and tensor. To print a tensor or the result of any tensor operation. You need to call the tf.Session() class. The class must first be instantiated. The tensor is printed on the terminal using the run() method of the instantiated class. In the last example, we defined a tensor_a. To print the tensor_a, we write the code below.

#create a tensor_a
tensor_a = tf.constant([[1, 2, 3],
                       [4, 5, 6]])

#call the Session class
with tf.Session() as sess:
    result =sess.run(tensor_a)
    #print the result
    print(result)

Output:

[[1 2 3] [4 5 6]]

Now let’s get back to other methods of creating Tensors

Creating Tensors Variables

We have looked at creating tensors with tf.constant(). While it is great to use that method if the tensor value would not change, it is uncommon. To create a more dynamic tensor whose values can be used as input to another operation, we create a variable tensor. 

You can create a using the tf.Variable() class. Using this method, you must specify some initial value of the tensor of any shape and datatype. 

To run operations with the values of the created variable, you must explicitly assign the value of the tensor to the variable itself. This is done by initializing the variable. Creating a variable only store the tensor value to that variable. You can see the initialization operation as calling a variable from a saved file to be ready for use. 

To initialize the variable for an operation, the most common method to use is the global_variables_initializer() method. Note that the initializer method must be run in a session. Let’s see an example where we implement this. 

#create a tensor variable
tensor_a = tf.Variable([1, 2, -4])

#initialize the variable
init_op = tf.global_variables_initializer()

#run a graph in session
with tf.Session() as sess:
    #run the variable initializer in a session.
    sess.run(init_op)
    #run the tensor operation
    result = sess.run(tensor_a)
    #print the result
    print(result)

You could also choose to create a variable by using the tf.get_variable() method. 

tf.get_variable(name = "", values, dtype, initializer)
argument
- `name = ""`: Name of the variable
- `values`: Dimension of the tensor
- `dtype`: Type of data. Optional
- `initializer`: How to initialize the tensor. Optional
If the initializer is specified, there is no need to include the `values` as the shape of `initializer` is used.  

This method is preferable since it doesn’t necessarily take an initial value and the initializer can be specified when calling the method. The tf.Variable() class always creates a new variable while the get_variable() method checks and gets existing variables with the same parameters. If there are no existing variables with the same parameters, it creates a new variable. 

Let’s say we want to create a 2 by 1 variable tensor using the tf.get_variable() method. When we call the method without defining the initializer, TensorFlow creates a tensor of random values with the defined shape. 

#creates a 2 by 1 variable tensor
var1 = tf.get_variable('var1', [2, 1])
#print the tensor
print(var1)

Note: The name argument is the name stored by TensorFlow while the variable name is the name stored by the Python compiler. To avoid confusion, you may use the same names in both places.

Output:

<tf.Variable 'var1:0' shape=(2, 1) dtype=float32_ref>

If we wish to specify the initial values of the tensor, we can do this with the initializer parameter. For instance, setting the initializer parameter to tf.zers_initializer fills the variable with zeros. You can also pass a constant tensor as the values of the initializer parameter. This will populate the variable tensor with the values of the constant tensor. Note that when the initializer parameter is defined, we do not need to pass the shape of the variable tensor as it takes the shape of the initializer tensor.

 Let’s see an example

#create 3 by 3 constant tensor
tensor_a = tf.constant([1, 2, 3, 4, 5, 6, 7, 8, 9])
tensor_a = tf.reshape(tensor_a, [3, 3])

#create a variable tensor whose first value is the constant tensor
var_tensor = tf.get_variable('var_tensor', initializer=tensor_a)
#check the shape of the variable tensor
var_tensor.shape

Output:

TensorShape([Dimension(3), Dimension(3)])

As seen, even without defining the shape of the variable tensor explicitly, it takes the shape of the constant tensor. 

Creating Tensors with Placeholders

The third method of creating a tensor is by using placeholders. This is done with the tf.placeholder() method. Placeholders allow you to feed the data outside of your TensorFlow graph. In other words, the data in a placeholder is not given when creating the TensorFlow graph, but rather, when running the graph in a session. 

Signature: tf.placeholder(dtype, shape=None, name=None)
arguments:
- `dtype`: Type of data
- `shape`: dimension of the placeholder. Optional. By default, shape of the data
- `name`: Name of the placeholder. Optional   
Returns: A `Tensor` that may be used as a handle for feeding a value, but not evaluated directly.

The placeholder() method takes the data type of the data as an important parameter. The name and shape are optional parameters. When the name is not defined, TensorFlow assigns the tensor with a default name. The shape of the tensor can be determined when it is later defined in the session. 

When using placeholders, we have mentioned that the data is fed when running the session. Upon calling the session, we pass the data using the feed_dict argument. This is a dictionary whose key is the variable name of the placeholder and values, the data to be received as numbers of arrays. 

Let’s take an example

#create a placeholder tensor_a
tensor_a = tf.placeholder(tf.float32)
#create a tensor b
tensor_b = 2 + tensor_a

#run a session
with tf.Session() as sess:
    #define tensor_a to be a 0-D tensor of 5
    #run the tensoor_b
    result = sess.run(tensor_b, feed_dict={tensor_a: 5})
    #print the result
    print(result)

Output:

7.0

Mathematical Operations in TensorFlow

TensorFlow can perform all the basic operations in mathematics such as addition, subtraction, multiplication, division, finding squares and square roots, exponents and so on. This is important in data manipulation. Let’s list out some of the common basic operators and the required argument

1. tf.add(a, b): Returns a + b element-wise.
2. tf.subtract(a, b): Returns a – b element-wise.
3. tf.multiply(a, b): Returns a * b element-wise.
4. tf.div(a, b): Divides a / b elementwise
5. tf.pow(a, b): Computes the power of one value to another.
6. tf.sqrt(a): Computes square root of ‘a’ element-wise.
7. tf.exp(a): Computes exponential of ‘a’ element-wise.  \\(b = ea\\).

Let’s take some examples

#create a tensor a
tensor_a = tf.constant([1, 2, 3])
#create a tensor b
tensor_b = tf.constant([-1, 4, 9])

#perform element-wise addition on tensor a and b
tensor_c = tf.add(tensor_a, tensor_b)
#create a session
with tf.Session() as sess:
    #run the session 
    result = sess.run(tensor_c)
    #print the result
    print(result)

Output:

[ 0 6 12]

It’s almost the same procedure for other operations. If we want to find the exponential of a tensor, for instance, the code below is an example of how to accomplish that. 

#create a tensor a
tensor_a = tf.constant([1, 2, 3], dtype=tf.float64)

#perform element-wise addition on tensor a and b
tensor_b = tf.exp(tensor_a)
#create a session
with tf.Session() as sess:
    #run the session 
    result = sess.run(tensor_b)
    #print the result
    print(result)

Note: Since exponents are in decimals, the datatype cannot be set to the default int32 or any int datatype.

Output:

[ 2.71828183 7.3890561 20.08553692]

Conclusion

To sum it all up, we started by defining what a tensor is and how they are represented in TensorFlow. 

We said that a TensorFlow program can be split into two parts, the computational graph, tensor, and session. We went on to explain the different types of ways to define a tensor with TensorFlow. 

We said that tf.constant() creates a tensor whose values remain the same. tf.variable() creates a tensor such that its values can be updated after an operation and  tf.placeholders() allows you to withdraw the values and even shape of a tensor until you’re ready to run a session. 

We discussed the shape and datatype of tensors and how to change them using the tf.reshape() method and tf.cast() method respectively. 

We finally tied it all together by creating a computational graph (initial tensors) and running a session (based on some operation) to produce output results.