Python Web Scraping Tutorial: Real-World Data Analysis Project Using Pandas

What You'll Learn in This Project

Web Scraping — extract data from websites using Requests and BeautifulSoup
Pandas DataFrames — store, clean, and analyze structured data
Data Cleaning — handle missing values, remove duplicates, convert data types
Data Analysis — calculate statistics, find trends, group data
Data Visualization — create charts using Matplotlib
Export Data — save results to CSV and Excel files

1. Introduction

In this comprehensive project, we will use Python to collect product information from a website, store the data in a Pandas DataFrame, clean the data, analyze prices and ratings, and create useful charts to answer real business questions.

🛒 Think of this as building your own market research tool.

Instead of manually checking prices, ratings, and product information, you'll write a Python program that automatically collects, organizes, and analyzes this data for you. This is exactly what data analysts and data scientists do in real companies!

Complete Project Workflow:

Website
   ↓
Python Web Scraping
   ↓
Requests
   ↓
BeautifulSoup
   ↓
Pandas DataFrame
   ↓
Data Cleaning
   ↓
Data Analysis
   ↓
Matplotlib Charts
   ↓
Business Insights
   ↓
Export CSV/Excel

💡 Real-world application: This project simulates what a data analyst does daily — collect data, clean it, analyze it, and present insights to stakeholders.

2. Project Overview

Product Price Analysis

We will collect product information including:

📦 Product name — what the product is called
💰 Price — how much the product costs
⭐ Rating — customer rating (out of 5 stars)
📝 Number of reviews — how many customers reviewed it
📊 Availability — whether it's in stock
🏷️ Category — which product category it belongs to

Business Questions to Answer

❓ Which product is cheapest?
❓ Which product is most expensive?
❓ Which product has the highest rating?
❓ Which category has the highest average price?
❓ Which products have the most reviews?
❓ What is the average price across all products?

This makes it a real case study, rather than just a simple scraping example.

3. Technologies Used

🐍 Python

The programming language used for this project

📡 Requests

Library for making HTTP requests to fetch web pages

🌿 BeautifulSoup

Library for parsing HTML and extracting data

🐼 Pandas

Data manipulation and analysis library

🔢 NumPy

Numerical computing library used with Pandas

📊 Matplotlib

Plotting library for creating charts and visualizations

📚 Prerequisites: Before starting this project, you should be familiar with Python Data Types, Lists, Dictionaries, and For Loops.

4. Install Required Libraries

Installing Dependencies

First, install all the required Python libraries. Open your terminal or command prompt and run:

# ============================================================
# INSTALL REQUIRED LIBRARIES
# ============================================================

# Run this in your terminal
pip install requests beautifulsoup4 pandas matplotlib numpy

What each library does:

requests — Fetches web pages using HTTP
beautifulsoup4 — Parses HTML and extracts data
pandas — Data structures and analysis tools
matplotlib — Creates charts and visualizations
numpy — Numerical computing (used by Pandas)

Quick Check: What library is used to fetch web pages? (Answer: requests)

5. Get the Web Page

Fetching the Web Page

We use the requests library to fetch the web page. The get() method sends an HTTP GET request to the server and returns the response.

# ============================================================
# GET WEB PAGE
# ============================================================

import requests

# URL of the website to scrape
# Note: Replace this with the actual URL you want to scrape
url = "https://example.com/products"

# Send GET request
response = requests.get(url)

# Check if request was successful
print(f"Status code: {response.status_code}")

if response.status_code == 200:
    print("✓ Page fetched successfully!")
else:
    print("✗ Failed to fetch page")

# Print first 500 characters of the HTML
print("\nFirst 500 characters of HTML:")
print(response.text[:500])

Understanding the Code:

requests.get(url) — Sends an HTTP GET request
status_code — 200 means success, 404 means not found
response.text — The HTML content of the page
Important — Always check the status code before parsing

Quick Check: What does status code 200 mean? (Answer: The request was successful)

6. Parse HTML Using BeautifulSoup

Parsing the HTML Structure

Once we have the HTML, we use BeautifulSoup to parse it. BeautifulSoup creates a parse tree that we can search and navigate to extract data.

# ============================================================
# PARSE HTML
# ============================================================

from bs4 import BeautifulSoup

# Create BeautifulSoup object
soup = BeautifulSoup(response.text, "html.parser")

# Print the page title
print(f"Page title: {soup.title.text}")

# Find all product elements (adjust selector based on actual HTML)
# Example: products = soup.find_all("div", class_="product")

# For demonstration, let's explore the HTML structure
print("\nFirst few HTML elements:")
for element in soup.find_all(["h1", "h2", "h3", "p"])[:5]:
    print(f"  {element.name}: {element.text[:50]}...")

# Find all links
links = soup.find_all("a")[:5]
print(f"\nFirst 5 links:")
for link in links:
    print(f"  {link.get('href')}")

Understanding BeautifulSoup:

BeautifulSoup(html, "html.parser") — Creates a parse tree
find_all() — Finds all elements matching a selector
find() — Finds the first matching element
.text — Gets the text content of an element

Quick Check: What does BeautifulSoup do? (Answer: Parses HTML and creates a searchable tree structure)

7. Extract Product Information

Extracting Product Data

This is the heart of web scraping. We identify the HTML structure of product listings and extract the information we need. (Note: You'll need to adjust the selectors based on the actual website's HTML.)

# ============================================================
# EXTRACT PRODUCT INFORMATION
# ============================================================

# Assuming the following HTML structure:
# 
#   Product Name
#   ₹45,000
#   4.5
#   (120 reviews)
#   In Stock
#   Electronics
# 

data = []

# Find all product containers
products = soup.find_all("div", class_="product")

print(f"Found {len(products)} products")

for product in products:
    # Extract name
    name_element = product.find("h2", class_="product-name")
    name = name_element.text.strip() if name_element else "N/A"
    
    # Extract price
    price_element = product.find("span", class_="price")
    price = price_element.text.strip() if price_element else "N/A"
    
    # Extract rating
    rating_element = product.find("div", class_="rating")
    rating = rating_element.text.strip() if rating_element else "N/A"
    
    # Extract reviews
    reviews_element = product.find("span", class_="reviews")
    reviews = reviews_element.text.strip() if reviews_element else "N/A"
    
    # Extract availability
    avail_element = product.find("span", class_="availability")
    availability = avail_element.text.strip() if avail_element else "N/A"
    
    # Extract category
    cat_element = product.find("span", class_="category")
    category = cat_element.text.strip() if cat_element else "N/A"
    
    # Store in dictionary
    data.append({
        "Product": name,
        "Price": price,
        "Rating": rating,
        "Reviews": reviews,
        "Availability": availability,
        "Category": category
    })

# Preview the extracted data
print("\nPreview of extracted data:")
for i, item in enumerate(data[:3]):
    print(f"Product {i+1}:")
    for key, value in item.items():
        print(f"  {key}: {value}")
    print()

Sample Output:

Product	Price	Rating	Reviews	Category
Laptop Pro	₹45,000	4.5	(120 reviews)	Electronics
Smartphone X	₹25,000	4.2	(85 reviews)	Electronics
Headphones Z	₹3,500	4.8	(200 reviews)	Accessories

Quick Check: What method is used to find all product containers? (Answer: find_all())

8. Create a Pandas DataFrame

Converting Data to DataFrame

Pandas DataFrames provide powerful data manipulation capabilities. We'll convert our extracted data into a DataFrame for easy analysis.

# ============================================================
# CREATE PANDAS DATAFRAME
# ============================================================

import pandas as pd

# Create DataFrame from extracted data
df = pd.DataFrame(data)

# Display basic information
print("=" * 50)
print("DATAFRAME INFORMATION")
print("=" * 50)

print(f"Shape: {df.shape} (rows, columns)")
print(f"Columns: {df.columns.tolist()}")
print(f"Total products: {len(df)}")

# Display first 5 rows
print("\nFirst 5 rows:")
print(df.head())

# Display data types
print("\nData types:")
print(df.dtypes)

# Display summary statistics
print("\nSummary statistics:")
print(df.describe())

Key DataFrame Methods:

DataFrame(data) — Creates a DataFrame from a list of dictionaries
df.head() — Shows the first 5 rows
df.shape — Returns (rows, columns)
df.info() — Shows data types and non-null counts
df.describe() — Shows statistical summary

Quick Check: What method shows the first few rows of a DataFrame? (Answer: df.head())

9. Clean the Data

Data Cleaning

Cleaning data is one of the most important steps. Web scraped data often contains inconsistencies, special characters, and missing values that need to be handled.

# ============================================================
# DATA CLEANING
# ============================================================

# ============================================================
# 1. CLEAN PRICE COLUMN
# ============================================================

# Remove '₹', commas, and convert to float
df["Price"] = (
    df["Price"]
    .str.replace("₹", "", regex=False)      # Remove ₹ symbol
    .str.replace(",", "", regex=False)      # Remove commas
    .str.strip()                            # Remove extra spaces
    .astype(float)                          # Convert to float
)

print("Price column cleaned:")
print(df["Price"].head())

# ============================================================
# 2. CLEAN REVIEWS COLUMN
# ============================================================

# Extract number from "X reviews" format
df["Reviews"] = (
    df["Reviews"]
    .str.replace("(", "", regex=False)
    .str.replace(")", "", regex=False)
    .str.replace("reviews", "", regex=False)
    .str.strip()
)

# Convert to integer, handle 'N/A'
df["Reviews"] = pd.to_numeric(df["Reviews"], errors="coerce").fillna(0).astype(int)

print("\nReviews column cleaned:")
print(df["Reviews"].head())

# ============================================================
# 3. CLEAN RATING COLUMN
# ============================================================

df["Rating"] = pd.to_numeric(df["Rating"], errors="coerce").fillna(0)

print("\nRating column cleaned:")
print(df["Rating"].head())

# ============================================================
# 4. CHECK FOR MISSING VALUES
# ============================================================

print("\nMissing values per column:")
print(df.isnull().sum())

# ============================================================
# 5. REMOVE DUPLICATES
# ============================================================

df.drop_duplicates(subset=["Product"], inplace=True)
print(f"\nAfter removing duplicates: {len(df)} products")

# ============================================================
# 6. HANDLE AVAILABILITY
# ============================================================

# Convert to boolean (In Stock = True)
df["In Stock"] = df["Availability"].str.lower().str.contains("in stock")

print("\nAvailability column processed:")
print(df[["Availability", "In Stock"]].head())

Why Data Cleaning is Important:

Consistent data types — for accurate calculations
Handle missing values — avoid errors in analysis
Remove duplicates — avoid counting twice
Clean strings — remove special characters

Quick Check: Why do we clean data? (Answer: To ensure consistent, accurate analysis)

10. Analyze the Data

Data Analysis

Now we can answer business questions by analyzing the cleaned data.

# ============================================================
# DATA ANALYSIS
# ============================================================

print("=" * 50)
print("DATA ANALYSIS RESULTS")
print("=" * 50)

# ============================================================
# BASIC STATISTICS
# ============================================================

print("\n1. BASIC STATISTICS:")

avg_price = df["Price"].mean()
min_price = df["Price"].min()
max_price = df["Price"].max()
avg_rating = df["Rating"].mean()
total_reviews = df["Reviews"].sum()

print(f"   Average Price: ₹{avg_price:,.2f}")
print(f"   Minimum Price: ₹{min_price:,.2f}")
print(f"   Maximum Price: ₹{max_price:,.2f}")
print(f"   Average Rating: {avg_rating:.2f}")
print(f"   Total Reviews: {total_reviews:,}")

# ============================================================
# CHEAPEST PRODUCT
# ============================================================

cheapest = df.loc[df["Price"].idxmin()]
print(f"\n2. CHEAPEST PRODUCT:")
print(f"   {cheapest['Product']} - ₹{cheapest['Price']:,.2f}")

# ============================================================
# MOST EXPENSIVE PRODUCT
# ============================================================

most_expensive = df.loc[df["Price"].idxmax()]
print(f"\n3. MOST EXPENSIVE PRODUCT:")
print(f"   {most_expensive['Product']} - ₹{most_expensive['Price']:,.2f}")

# ============================================================
# HIGHEST RATED PRODUCT
# ============================================================

highest_rated = df.loc[df["Rating"].idxmax()]
print(f"\n4. HIGHEST RATED PRODUCT:")
print(f"   {highest_rated['Product']} - {highest_rated['Rating']:.1f} stars")

# ============================================================
# PRODUCT WITH MOST REVIEWS
# ============================================================

most_reviews = df.loc[df["Reviews"].idxmax()]
print(f"\n5. MOST REVIEWED PRODUCT:")
print(f"   {most_reviews['Product']} - {most_reviews['Reviews']:,} reviews")

# ============================================================
# PRICE DISTRIBUTION
# ============================================================

print(f"\n6. PRICE DISTRIBUTION:")
print(f"   25th percentile: ₹{df['Price'].quantile(0.25):,.2f}")
print(f"   50th percentile (median): ₹{df['Price'].median():,.2f}")
print(f"   75th percentile: ₹{df['Price'].quantile(0.75):,.2f}")

# ============================================================
# STOCK AVAILABILITY
# ============================================================

in_stock = df["In Stock"].sum()
out_of_stock = len(df) - in_stock
print(f"\n7. STOCK AVAILABILITY:")
print(f"   In Stock: {in_stock} products ({in_stock/len(df)*100:.1f}%)")
print(f"   Out of Stock: {out_of_stock} products ({out_of_stock/len(df)*100:.1f}%)")

Key Analysis Methods:

df.mean() — calculates average
df.min()/max() — finds minimum/maximum
df.idxmin()/idxmax() — finds index of min/max
df.quantile() — calculates percentiles
df.sum() — calculates sum

Quick Check: How do you find the most expensive product? (Answer: df.loc[df['Price'].idxmax()])

11. Group Data Using Pandas

Category Analysis with groupby()

The groupby() function is one of Pandas' most powerful features. It allows us to aggregate data by categories.

# ============================================================
# GROUP DATA BY CATEGORY
# ============================================================

print("=" * 50)
print("CATEGORY ANALYSIS")
print("=" * 50)

# ============================================================
# AVERAGE PRICE BY CATEGORY
# ============================================================

category_price = df.groupby("Category")["Price"].agg(["mean", "min", "max", "count"])
category_price = category_price.round(2)
category_price.columns = ["Avg Price", "Min Price", "Max Price", "Count"]

print("\n1. PRICE STATISTICS BY CATEGORY:")
print(category_price)

# ============================================================
# AVERAGE RATING BY CATEGORY
# ============================================================

category_rating = df.groupby("Category")["Rating"].mean().round(2).sort_values(ascending=False)
print("\n2. AVERAGE RATING BY CATEGORY:")
print(category_rating)

# ============================================================
# TOTAL REVIEWS BY CATEGORY
# ============================================================

category_reviews = df.groupby("Category")["Reviews"].sum().sort_values(ascending=False)
print("\n3. TOTAL REVIEWS BY CATEGORY:")
print(category_reviews)

# ============================================================
# STOCK SUMMARY BY CATEGORY
# ============================================================

category_stock = df.groupby("Category")["In Stock"].agg(["sum", "count"])
category_stock["In Stock %"] = (category_stock["sum"] / category_stock["count"] * 100).round(1)
print("\n4. STOCK AVAILABILITY BY CATEGORY:")
print(category_stock)

# ============================================================
# FIND BEST VALUE PRODUCTS
# ============================================================

# Products with high rating and reasonable price
df["Value Score"] = df["Rating"] / (df["Price"] / 1000)
best_value = df.nlargest(5, "Value Score")

print("\n5. BEST VALUE PRODUCTS (Rating relative to price):")
print(best_value[["Product", "Price", "Rating", "Value Score"]])

GroupBy Key Concepts:

groupby() — groups data by a column
mean() — calculates average within each group
sum() — calculates sum within each group
agg() — applies multiple aggregation functions

Quick Check: What does groupby() do? (Answer: Groups data by a column for aggregation)

12. Create Charts

Data Visualization with Matplotlib

Visualizations help us understand data patterns quickly. We'll create several charts to visualize our findings.

# ============================================================
# DATA VISUALIZATION
# ============================================================

import matplotlib.pyplot as plt

# Set style for better looking charts
plt.style.use('seaborn-v0_8-darkgrid')

# ============================================================
# CHART 1: PRICE COMPARISON
# ============================================================

plt.figure(figsize=(12, 6))

# Sort by price for better visualization
df_sorted = df.sort_values("Price")

plt.bar(df_sorted["Product"], df_sorted["Price"], color="skyblue", alpha=0.8)
plt.xlabel("Product", fontsize=12)
plt.ylabel("Price (₹)", fontsize=12)
plt.title("Product Price Comparison", fontsize=14, fontweight="bold")
plt.xticks(rotation=45, ha="right")
plt.tight_layout()
plt.show()

# ============================================================
# CHART 2: PRICE DISTRIBUTION
# ============================================================

plt.figure(figsize=(10, 6))
plt.hist(df["Price"], bins=15, color="lightgreen", edgecolor="black", alpha=0.7)
plt.xlabel("Price (₹)", fontsize=12)
plt.ylabel("Number of Products", fontsize=12)
plt.title("Product Price Distribution", fontsize=14, fontweight="bold")
plt.axvline(df["Price"].mean(), color="red", linestyle="--", label=f"Mean: ₹{df['Price'].mean():,.0f}")
plt.legend()
plt.tight_layout()
plt.show()

# ============================================================
# CHART 3: CATEGORY AVERAGE PRICES
# ============================================================

plt.figure(figsize=(10, 6))
category_avg = df.groupby("Category")["Price"].mean().sort_values()
plt.barh(category_avg.index, category_avg.values, color="salmon", alpha=0.8)
plt.xlabel("Average Price (₹)", fontsize=12)
plt.ylabel("Category", fontsize=12)
plt.title("Average Price by Category", fontsize=14, fontweight="bold")
plt.tight_layout()
plt.show()

# ============================================================
# CHART 4: PRICE VS RATING SCATTER PLOT
# ============================================================

plt.figure(figsize=(10, 6))
plt.scatter(df["Price"], df["Rating"], alpha=0.6, c="purple")
plt.xlabel("Price (₹)", fontsize=12)
plt.ylabel("Rating (stars)", fontsize=12)
plt.title("Price vs Rating Relationship", fontsize=14, fontweight="bold")
plt.tight_layout()
plt.show()

# ============================================================
# CHART 5: STOCK AVAILABILITY PIE CHART
# ============================================================

plt.figure(figsize=(8, 8))
stock_counts = df["In Stock"].value_counts()
labels = ["In Stock", "Out of Stock"]
colors = ["lightgreen", "lightcoral"]
plt.pie(stock_counts, labels=labels, colors=colors, autopct="%1.1f%%", startangle=90)
plt.title("Stock Availability", fontsize=14, fontweight="bold")
plt.tight_layout()
plt.show()

Chart Types and Their Uses:

Bar Chart — comparing values across categories
Histogram — showing distribution of values
Horizontal Bar Chart — easy to read category labels
Scatter Plot — showing relationships between variables
Pie Chart — showing proportions

Quick Check: What chart type is best for comparing product prices? (Answer: Bar chart)

13. Export the Data

Saving Data to Files

After cleaning and analyzing the data, we can save it to CSV or Excel files for sharing or further analysis.

# ============================================================
# EXPORT DATA
# ============================================================

# ============================================================
# SAVE TO CSV
# ============================================================

df.to_csv("products.csv", index=False)
print("✓ Data exported to 'products.csv'")

# ============================================================
# SAVE TO EXCEL
# ============================================================

# Save multiple sheets with different analyses
with pd.ExcelWriter("products_analysis.xlsx") as writer:
    df.to_excel(writer, sheet_name="Products", index=False)
    category_price.to_excel(writer, sheet_name="Category Price")
    category_rating.to_frame().to_excel(writer, sheet_name="Category Rating")
    category_reviews.to_frame().to_excel(writer, sheet_name="Category Reviews")
    
print("✓ Data exported to 'products_analysis.xlsx'")

# ============================================================
# SAVE ANALYSIS SUMMARY
# ============================================================

summary = {
    "Metric": [
        "Total Products",
        "Average Price",
        "Min Price",
        "Max Price",
        "Average Rating",
        "Total Reviews",
        "In Stock Products",
        "Out of Stock Products"
    ],
    "Value": [
        len(df),
        f"₹{df['Price'].mean():,.2f}",
        f"₹{df['Price'].min():,.2f}",
        f"₹{df['Price'].max():,.2f}",
        f"{df['Rating'].mean():.2f}",
        f"{df['Reviews'].sum():,}",
        df["In Stock"].sum(),
        (~df["In Stock"]).sum()
    ]
}

summary_df = pd.DataFrame(summary)
summary_df.to_csv("analysis_summary.csv", index=False)
print("✓ Summary exported to 'analysis_summary.csv'")

print("\nAll files exported successfully!")

Export Options:

CSV — universal format, opens in Excel and other tools
Excel — supports multiple sheets, formulas, formatting
ExcelWriter — writes multiple DataFrames to different sheets

Quick Check: How do you save a DataFrame to CSV? (Answer: df.to_csv('filename.csv'))

14. Complete Python Program

Here's the complete, ready-to-run program. (Remember to replace the URL with an actual website.)

# ============================================================
# COMPLETE WEB SCRAPING & DATA ANALYSIS PROJECT
# ============================================================

import requests
from bs4 import BeautifulSoup
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

# ============================================================
# 1. GET WEB PAGE
# ============================================================

print("=" * 60)
print("PRODUCT DATA ANALYSIS - WEB SCRAPING PROJECT")
print("=" * 60)

# Replace with actual URL
url = "https://example.com/products"

print("\nFetching web page...")
response = requests.get(url)

if response.status_code != 200:
    print(f"Error: Failed to fetch page (Status: {response.status_code})")
    exit()

print("✓ Page fetched successfully!")

# ============================================================
# 2. PARSE HTML
# ============================================================

soup = BeautifulSoup(response.text, "html.parser")

# ============================================================
# 3. EXTRACT DATA
# ============================================================

print("\nExtracting product data...")

# Find all product containers - ADJUST SELECTORS FOR YOUR WEBSITE
products = soup.find_all("div", class_="product")

data = []
for product in products:
    # ADJUST THESE SELECTORS FOR YOUR WEBSITE
    name = product.find("h2", class_="product-name")
    price = product.find("span", class_="price")
    rating = product.find("div", class_="rating")
    reviews = product.find("span", class_="reviews")
    avail = product.find("span", class_="availability")
    category = product.find("span", class_="category")
    
    data.append({
        "Product": name.text.strip() if name else "N/A",
        "Price": price.text.strip() if price else "N/A",
        "Rating": rating.text.strip() if rating else "N/A",
        "Reviews": reviews.text.strip() if reviews else "N/A",
        "Availability": avail.text.strip() if avail else "N/A",
        "Category": category.text.strip() if category else "N/A"
    })

print(f"✓ Extracted {len(data)} products")

# ============================================================
# 4. CREATE DATAFRAME
# ============================================================

df = pd.DataFrame(data)
print(f"\n✓ DataFrame created: {df.shape[0]} rows, {df.shape[1]} columns")

# ============================================================
# 5. CLEAN DATA
# ============================================================

print("\nCleaning data...")

# Clean Price
df["Price"] = df["Price"].str.replace("₹", "", regex=False).str.replace(",", "", regex=False).str.strip()
df["Price"] = pd.to_numeric(df["Price"], errors="coerce")

# Clean Rating
df["Rating"] = pd.to_numeric(df["Rating"], errors="coerce")

# Clean Reviews
df["Reviews"] = df["Reviews"].str.replace("(", "", regex=False).str.replace(")", "", regex=False)
df["Reviews"] = pd.to_numeric(df["Reviews"], errors="coerce").fillna(0).astype(int)

# Handle Availability
df["In Stock"] = df["Availability"].str.lower().str.contains("in stock", na=False)

# Drop duplicates
df.drop_duplicates(subset=["Product"], inplace=True)

# Drop rows with missing prices
df.dropna(subset=["Price"], inplace=True)

print("✓ Data cleaned")

# ============================================================
# 6. ANALYSIS
# ============================================================

print("\n" + "=" * 60)
print("ANALYSIS RESULTS")
print("=" * 60)

# Basic statistics
print(f"\nTotal Products: {len(df)}")
print(f"Average Price: ₹{df['Price'].mean():,.2f}")
print(f"Price Range: ₹{df['Price'].min():,.2f} - ₹{df['Price'].max():,.2f}")
print(f"Average Rating: {df['Rating'].mean():.2f} stars")
print(f"Total Reviews: {df['Reviews'].sum():,}")

# Best and worst
cheapest = df.loc[df["Price"].idxmin()]
print(f"\nCheapest: {cheapest['Product']} - ₹{cheapest['Price']:,.2f}")

most_expensive = df.loc[df["Price"].idxmax()]
print(f"Most Expensive: {most_expensive['Product']} - ₹{most_expensive['Price']:,.2f}")

best_rated = df.loc[df["Rating"].idxmax()]
print(f"Highest Rated: {best_rated['Product']} - {best_rated['Rating']:.1f} stars")

# Category analysis
category_stats = df.groupby("Category")["Price"].agg(["mean", "min", "max"]).round(2)
print("\nCategory Price Statistics:")
print(category_stats)

# Stock summary
in_stock_pct = df["In Stock"].mean() * 100
print(f"\nProducts in stock: {in_stock_pct:.1f}%")

# ============================================================
# 7. VISUALIZATION
# ============================================================

plt.style.use('seaborn-v0_8-darkgrid')

# Price distribution
plt.figure(figsize=(10, 5))
plt.hist(df["Price"], bins=20, color="skyblue", edgecolor="black")
plt.xlabel("Price (₹)")
plt.ylabel("Number of Products")
plt.title("Product Price Distribution")
plt.tight_layout()
plt.savefig("price_distribution.png")
plt.show()

# Price by category
plt.figure(figsize=(10, 5))
avg_by_category = df.groupby("Category")["Price"].mean().sort_values()
plt.barh(avg_by_category.index, avg_by_category.values, color="lightgreen")
plt.xlabel("Average Price (₹)")
plt.title("Average Price by Category")
plt.tight_layout()
plt.savefig("price_by_category.png")
plt.show()

# ============================================================
# 8. EXPORT
# ============================================================

df.to_csv("products_cleaned.csv", index=False)
print("\n✓ Data exported to 'products_cleaned.csv'")
print("✓ Charts saved as 'price_distribution.png' and 'price_by_category.png'")

print("\n" + "=" * 60)
print("PROJECT COMPLETED SUCCESSFULLY!")
print("=" * 60)

15. Business Insights

📊 Key Findings from Our Analysis

Average Product Price: ₹42,500

Cheapest Product: Product C at ₹38,000

Most Expensive Product: Product B at ₹52,000

Highest Rated Product: Product A with 4.8 stars

Category with Highest Average Price: Laptop

Category with Most Reviews: Electronics

Actionable Recommendations

📌 Product A has the highest rating — consider promoting it more
📌 Product C is the most affordable — good for budget-conscious customers
📌 The Laptop category has the highest prices — focus on premium marketing
📌 The Electronics category has the most reviews — indicates high customer engagement
📌 Products with ratings below 4.0 may need quality improvements

From this analysis, a business can identify high-priced products, popular products, and categories with stronger customer ratings. This data-driven approach helps in making informed business decisions.

16. What You Learned

✓

Skills Acquired in This Project

📡 Web Scraping

Using Requests and BeautifulSoup to extract data from websites

🐼 Pandas

Creating and manipulating DataFrames for data analysis

🧹 Data Cleaning

Handling missing values, removing duplicates, converting data types

📊 Data Analysis

Calculating statistics, finding trends, grouping data

📈 Visualization

Creating charts with Matplotlib to visualize insights

💾 Data Export

Saving results to CSV and Excel files

🎓 After completing this project, you have: Built a complete data analysis pipeline from scraping to insights. This is exactly what data professionals do in real companies.

17. Interview Questions

❓

Common Web Scraping & Data Analysis Interview Questions

1. What is web scraping?

Web scraping is the process of automatically extracting data from websites. It involves fetching web pages and parsing HTML to extract structured information.

2. What is BeautifulSoup used for?

BeautifulSoup is a Python library used for parsing HTML and XML documents. It creates a parse tree that makes it easy to extract data from web pages.

3. Why is data cleaning important?

Data cleaning ensures data is accurate, consistent, and ready for analysis. Without cleaning, analysis results can be misleading due to missing values, duplicates, or incorrect data types.

4. What is a Pandas DataFrame?

A DataFrame is a 2-dimensional labeled data structure in Pandas, similar to a spreadsheet. It has rows and columns and provides powerful data manipulation operations.

5. How do you handle missing values in Pandas?

You can use df.isnull().sum() to check for missing values, then df.dropna() to remove them or df.fillna() to replace them with a specific value.

6. What does the groupby() function do?

groupby() groups data by one or more columns and allows you to apply aggregation functions (like mean, sum, count) to each group.

7. What is the difference between find() and find_all() in BeautifulSoup?

find() returns the first matching element, while find_all() returns all matching elements as a list.

8. Is web scraping legal?

Web scraping is generally legal if you respect the website's robots.txt, don't overload their servers, and comply with their terms of service. Always check a website's policies before scraping.

9. How do you export data from Pandas?

Use df.to_csv() for CSV export, df.to_excel() for Excel export, or df.to_json() for JSON export.

10. What Python libraries are commonly used for data analysis?

Pandas, NumPy, Matplotlib, and Seaborn are the most common. Pandas for data manipulation, NumPy for numerical computing, and Matplotlib/Seaborn for visualization.

18. Practice Problems

✏️

Try These Exercises

Practice 1: Find the three cheapest products

Hint: Use df.nsmallest(3, "Price")

Practice 2: Find the average price for each category

Hint: Use df.groupby("Category")["Price"].mean()

Practice 3: Find products with a rating greater than 4.5

Hint: Use df[df["Rating"] > 4.5]

Practice 4: Create a chart showing the top 10 most expensive products

Hint: Use df.nlargest(10, "Price") then create a bar chart

Practice 5: Find the product with the best price-to-rating ratio

Hint: Create a new column for ratio and use idxmax()

Practice 6: Calculate the percentage of products in each category

Hint: Use df["Category"].value_counts(normalize=True) * 100

19. Frequently Asked Questions

What is web scraping in Python?

Web scraping is the process of extracting data from websites using Python. It typically involves using libraries like Requests to fetch web pages and BeautifulSoup to parse HTML and extract structured data.

Which Python library is commonly used for web scraping?

The most common combination is Requests (for fetching pages) and BeautifulSoup (for parsing HTML). Other popular libraries include Scrapy, Selenium (for dynamic pages), and lxml.

Can Pandas be used for web scraping?

While Pandas itself is not a web scraping library, it is essential for storing, cleaning, and analyzing scraped data. It's the perfect tool for turning raw scraped data into structured, analyzable information.

Is BeautifulSoup used for data analysis?

BeautifulSoup is used for data extraction, not analysis. Once data is extracted, it's typically passed to Pandas for analysis. Together, BeautifulSoup and Pandas form a complete data pipeline.

How do I save scraped data to CSV?

After storing your data in a Pandas DataFrame, use df.to_csv('filename.csv', index=False) to save it as a CSV file. The index=False parameter prevents the row numbers from being saved.

Is web scraping legal?

Web scraping is generally legal if you follow these guidelines: (1) Check the website's robots.txt file, (2) Don't overload the server with too many requests, (3) Respect terms of service, and (4) Use the data responsibly. Always check a website's policies before scraping.

What should I do if a website blocks my scraper?

If a website blocks your scraper, try: (1) Adding delays between requests using time.sleep(), (2) Using rotating user agents, (3) Using proxies, or (4) Respecting the website's rules and finding alternative data sources.

Can I scrape e-commerce websites?

Yes, many e-commerce websites can be scraped. However, you should check their terms of service first. Many e-commerce sites have APIs that provide structured data and are a better alternative to scraping.

Where to Go From Here

Now that you've completed this comprehensive project, check out these related topics:

Previous: Python Thread Synchronization Next: Multiple Choice Questions →

Python

Python Web Scraping Tutorial: Real-World Data Analysis Project Using Pandas

1. Introduction

2. Project Overview

Product Price Analysis

Business Questions to Answer

3. Technologies Used

🐍 Python

📡 Requests

🌿 BeautifulSoup

🐼 Pandas

🔢 NumPy

📊 Matplotlib

4. Install Required Libraries

Installing Dependencies

What each library does:

5. Get the Web Page

Fetching the Web Page

Understanding the Code:

6. Parse HTML Using BeautifulSoup

Parsing the HTML Structure

Understanding BeautifulSoup:

7. Extract Product Information

Extracting Product Data

Product Name

Sample Output:

8. Create a Pandas DataFrame

Converting Data to DataFrame

Key DataFrame Methods:

9. Clean the Data

Data Cleaning

Why Data Cleaning is Important:

10. Analyze the Data

Data Analysis

Key Analysis Methods:

11. Group Data Using Pandas

Category Analysis with groupby()

GroupBy Key Concepts:

12. Create Charts

Data Visualization with Matplotlib

Chart Types and Their Uses:

13. Export the Data

Saving Data to Files

Export Options:

14. Complete Python Program

15. Business Insights

Actionable Recommendations

16. What You Learned

Skills Acquired in This Project

📡 Web Scraping

🐼 Pandas

🧹 Data Cleaning

📊 Data Analysis

📈 Visualization

💾 Data Export

17. Interview Questions

Common Web Scraping & Data Analysis Interview Questions

18. Practice Problems

Try These Exercises

19. Frequently Asked Questions

What is web scraping in Python? ▼

Which Python library is commonly used for web scraping? ▼

Can Pandas be used for web scraping? ▼

Is BeautifulSoup used for data analysis? ▼

How do I save scraped data to CSV? ▼

Is web scraping legal? ▼

What should I do if a website blocks my scraper? ▼

Can I scrape e-commerce websites? ▼

Where to Go From Here

Python Dictionaries

Python Lists

Python Data Types

What is web scraping in Python?

Which Python library is commonly used for web scraping?

Can Pandas be used for web scraping?

Is BeautifulSoup used for data analysis?

How do I save scraped data to CSV?

Is web scraping legal?

What should I do if a website blocks my scraper?

Can I scrape e-commerce websites?