Overview
Python tokens are the smallest individual units in the Python program. The Python interpreter parses Python program code into tokens before it is executed.
Python Tokens
A token is the smallest unit in the Python language’s syntax. It is also known as
a lexical unit. The Python interpreter recognizes these tokens during the lexical analysis phase before the code is executed.
The lexical analysis phase( tokenization) is the first phase of the compilation process.
Its primary goal is to convert the source code of a program into a sequence of tokens.
A lexical analyzer is responsible for performing this task. The lexical analysis phase involves breaking down the source code into these tokens by removing comments and whitespace and categorizing the remaining symbols into specific token types.
The Python tokens are as follows:
- Keywords
- Identifiers
- Literals
- Operators
- Delimiters
- Comments
- Indentation
| Python token | Description |
| Keywords | Keywords have special meanings and are reserved by the Python language. Keywords have a predefined meaning and purpose in the language syntax. We cannot use keywords as variables or identifiers because they already have a predefined meaning in the language syntax. |
| Identifiers | Names given to variables, functions, classes, etc. |
| Literals | Constants representing values. |
| Operators | Operators are symbols that perform some action or computation on the variables. |
| Delimiters | Delimiters are symbols that are used to organize Python statements. For example: Parentheses () Colon : |
| Comments | Lines beginning with # are treated as comments. |
| Indentation | Python uses indentation to indicate blocks of code. |
Example
Identify the Python tokens in the following Python code snippet:
You can use the tokenize module or the tokenize() function to see the tokens in a Python script.
Tokens are like the building blocks of a Python program. They are essential for writing, understanding, and debugging Python code.
—
Python Tutorials
Python Tutorial on this website can be found at:
https://www.testingdocs.com/python-tutorials/
More information on Python is available at the official website: