🐍 Python Tokens: Complete Guide with Examples

So, What Exactly Are Python Tokens?

Think of tokens as the building blocks of Python code — the smallest units that the Python interpreter understands. When you write a program, Python breaks it down into these tokens to figure out what you want it to do.

Here's a simple way to think about it: just like how English sentences are made up of words and punctuation, Python programs are made up of tokens. Each token has a specific meaning to the interpreter.

Python has five main types of tokens:

• Keywords: Reserved words with special meanings
• Identifiers: Names for variables, functions, and classes
• Literals: Fixed values like numbers and strings
• Operators: Symbols that perform operations
• Delimiters: Punctuation that structures your code

1. Keywords — Python's Reserved Words

Keywords are reserved words that have a special meaning in Python. You cannot use them as variable names, function names, or identifiers.

Think of keywords as Python's built-in vocabulary — they're the words that Python understands without any explanation. Here are some common keywords:

Keywords in Python (36 in total):

False      class      finally    is         return
None       continue   for        lambda     try
True       def        from       nonlocal   while
and        del        global     not        with
as         elif       if         or         yield
assert     else       import     pass
break      except     in         raise

Here's how you can list all the keywords yourself:

# Python script to list all keywords
import keyword
print(keyword.kwlist)

2. Identifiers — Naming Your Code

Identifiers are names you give to variables, functions, classes, and other objects. They help you refer to things in your code.

Here are the rules for naming identifiers in Python:

• Start with a letter or underscore: Must begin with a letter (a-z, A-Z) or underscore (_)
• Can contain numbers: After the first character, you can use numbers (0-9)
• Case-sensitive: myVar and myvar are different
• Can't use keywords: You can't use if, while, class, etc.
• No special characters: Only letters, numbers, and underscores are allowed

3. Literals — Fixed Values

Literals are fixed values that you write directly in your code. They represent constant values that don't change.

Python has several types of literals:

# Examples of different literals
name = "Python"           # String literal
age = 30                  # Integer literal
price = 19.99             # Float literal
is_popular = True         # Boolean literal
nothing = None            # None literal

4. Operators — Performing Operations

Operators are symbols that tell Python to perform specific operations on values and variables.

Python has several types of operators:

Common Operators in Python:

Arithmetic:  +   -   *   /   //   %   **
Comparison:  ==  !=  >   <   >=   <=
Logical:     and  or  not
Assignment:   =   +=  -=  *=  /=
Membership:   in  not in
Identity:     is  is not

# Examples of operators
a = 10
b = 3

print(a + b)   # 13 (addition)
print(a - b)   # 7 (subtraction)
print(a * b)   # 30 (multiplication)
print(a / b)   # 3.33 (division)
print(a % b)   # 1 (modulus - remainder)
print(a ** b)  # 1000 (power)

# Comparison
print(a > b)   # True
print(a == b)  # False

5. Delimiters — Structuring Your Code

Delimiters are punctuation marks that help organize your code. They define where blocks start and end, and they separate elements.

Common Delimiters in Python:

( )   Parentheses    - Used for function calls and grouping
[ ]   Square brackets - Used for lists and indexing
{ }   Curly braces   - Used for dictionaries and sets
;     Semicolon      - Separates statements (rarely used)
:     Colon          - Indicates start of a block
,     Comma          - Separates items
.     Period         - Access object attributes

# Examples of delimiters
my_list = [1, 2, 3]        # [ ] and ,
my_dict = {"name": "Python"} # { } and :
print(my_dict["name"])     # ( ) and [ ]

Try It Yourself!

Now it's your turn! Write and run Python code directly in your browser:

Quick Quiz - Test Your Knowledge

Let's see what you've learned about Python tokens:

Frequently Asked Questions

📚 Where to Go From Here

Now that you understand Python tokens, here are some related topics to explore: