Web scraping is an essential skill for developers who need to collect structured data from the web. This article walks you through creating a small, robust scraper in Python using requests and Beautiful Soup, explains how the pieces fit together, and highlights ethical and technical best practices.

\n \n\n

Prerequisites

\n \n\n \n

You should be comfortable with basic Python (functions, lists, dicts) and have Python 3.8+ installed. Familiarity with HTML (tags, attributes) and CSS selectors is helpful but not required.

\n \n\n

Installing dependencies

\n \n\n \n

Install the minimal packages you’ll need:

\n \n

pip install requests beautifulsoup4

\n\n \n

Optionally, install lxml for faster parsing:

\n \n

pip install lxml

\n\n

Your first scraper

\n \n\n \n

The following example fetches a page, parses it with Beautiful Soup, and extracts titles and links from article-like elements. It demonstrates headers, timeouts, basic error handling, and polite pauses.

\n \n

import requests\nfrom bs4 import BeautifulSoup\nimport time\nimport csv\n\nURL = 'https://example.com/articles'\nHEADERS = {'User-Agent': 'MyScraper/1.0 (+https://example.com/contact)'}\n\ndef fetch(url):\n    try:\n        resp = requests.get(url, headers=HEADERS, timeout=10)\n        resp.raise_for_status()\n        return resp.text\n    except requests.RequestException as e:\n        print('Request failed:', e)\n        return None\n\ndef parse(html):\n    soup = BeautifulSoup(html, 'lxml')\n    results = []\n    for card in soup.select('article.post'):\n        title_el = card.select_one('h2 a')\n        if not title_el:\n            continue\n        title = title_el.get_text(strip=True)\n        link = title_el.get('href')\n        summary = (card.select_one('p.summary') or card.select_one('p')).get_text(strip=True)\n        results.append({'title': title, 'link': link, 'summary': summary})\n    return results\n\nif __name__ == '__main__':\n    html = fetch(URL)\n    if not html:\n        raise SystemExit('Failed to fetch page')\n    items = parse(html)\n    with open('articles.csv', 'w', newline='', encoding='utf-8') as f:\n        writer = csv.DictWriter(f, fieldnames=['title', 'link', 'summary'])\n        writer.writeheader()\n        writer.writerows(items)\n    print(f'Saved {len(items)} items')

\n\n

How it works

\n \n\n

Fetching the page

\n \n\n \n

The requests.get call retrieves the HTML. Use a realistic User-Agent, a timeout, and handle exceptions (network errors, non-2xx status codes) using raise for status.

\n \n\n

Parsing with Beautiful Soup

\n \n\n \n

Beautiful Soup converts HTML into a parse tree. You can select elements using .select (CSS selectors) or helper methods like .find all, .find, and .select one. Using lxml as the parser improves speed and robustness.

\n \n\n

Extracting data

\n \n\n \n

After locating elements, use .get_text(strip=True) for text and .get(‘href’) for attributes. Normalize or resolve relative URLs with urllib.parse.urljoin when needed.

\n \n\n

Best practices and etiquette

\n \n\n

Respect robots.txt and site terms of service. Programmatic scraping can be legally sensitive.
Rate limit your requests (time.sleep between requests) and avoid high concurrency on small sites.
Cache responses when possible to reduce load.
Set a meaningful User-Agent and provide contact info if appropriate.
Use timeouts and retry/backoff strategies for transient errors.
Avoid scraping private or paywalled content without explicit permission.

\n \n\n\n\n\n\n\n

Handling pagination and sessions

\n \n\n \n

For paginated content, iterate through pages (next links or query params). Use requests.Session() to persist cookies and reduce overhead if the site uses session-based navigation.

\n \n\n

When pages are dynamic (JavaScript)

\n \n\n \n

If the content is rendered client-side, consider these options:

\n \n\n

Reverse-engineer the API calls made by the page (best when available).
Use a headless browser (Playwright, Selenium) or requests-html for JS execution.
Check network tab to find JSON endpoints — scraping JSON is more robust than parsing HTML.

\n \n\n\n\n

Error handling and robustness

\n \n\n

Validate selectors against multiple pages (sites change layout).
Log failures and sample HTML for debugging.
Add unit tests for parsing functions using saved HTML fixtures.

\n \n\n\n\n

Storing and using scraped data

\n \n\n \n

Save results to CSV, JSON, or a database depending on volume. Respect data privacy and usage policies when storing or sharing scraped data.

\n \n\n

Advanced topics (next steps)

\n \n\n

Parallel scraping with care: use async requests (aiohttp) or controlled thread pools and always obey site limits.
Proxies and IP rotation: for large-scale scraping only, and ensure you comply with legal and ethical rules.
Structured extraction: use tools like extruct or specialized libraries when extracting microdata, JSON-LD, or RDFa.

\n \n\n\n\n

Conclusion

\n \n\n \n

Beautiful Soup with requests is an efficient, maintainable starting point for many scraping tasks. Start small, respect the target site, and make incremental improvements: handle errors, add retries, and consider API-based or headless-browser approaches only when necessary.

\n \n\n \n

Happy scraping — and be kind to the servers you depend on!

\n ","protected",false,"date_gmt","2025-11-19T12:45:31.000Z","excerpt",{"_320":328,"_322":323},"

A practical, developer-focused guide to building a simple web scraper in Python with requests and Beautiful Soup. Covers setup, parsing techniques, basic error handling, and best practices.

\n","modified_gmt","2025-11-19T12:45:33.000Z","web-scraping-with-python-your-first-scraper-using-beautiful-soup","status","publish","sticky","tags",[337,338,339,340,341],77,79,59,78,76,{"_320":343},"Web Scraping with Python: Your First Scraper Using Beautiful Soup","type","post",{"_26":347,"_14":348,"_28":349,"_39":40,"_41":42,"_43":347,"_350":348,"_44":351,"_46":47,"_352":353,"_354":355,"_356":107,"_48":49,"_357":358,"_50":362},"Web Scraping with Python: First Scraper Using Beautiful Soup","Learn how to build your first web scraper in Python using requests and Beautiful Soup. Includes setup, parsing examples, best practices, and error handling.",{"_30":31,"_32":32,"_33":34,"_35":36,"_37":38},"og_description","https://wordpress.fulldev.pl/2025/11/19/web-scraping-with-python-your-first-scraper-using-beautiful-soup/","article_published_time","2025-11-19T13:45:31+00:00","article_modified_time","2025-11-19T13:45:33+00:00","author","twitter_misc",{"_359":107,"_360":361},"Written by","Est. reading time","4 minutes",{"_52":53,"_54":363},[364,392,401,405,411],{"_57":365,"_59":366,"_61":367,"_356":368,"_369":343,"_370":353,"_371":355,"_372":373,"_374":375,"_376":19,"_84":377,"_378":379,"_384":385,"_67":68,"_87":386},"Article","https://wordpress.fulldev.pl/2025/11/19/web-scraping-with-python-your-first-scraper-using-beautiful-soup/#article",{"_59":351},{"_16":107,"_59":86},"headline","datePublished","dateModified","mainEntityOfPage",{"_59":351},"wordCount",539,"commentCount",{"_59":86},"keywords",[380,381,382,383,185],"beautifulsoup","data extraction","Python","requests","articleSection",[185],[387],{"_57":388,"_16":389,"_91":390},"CommentAction","Comment",[391],"https://wordpress.fulldev.pl/2025/11/19/web-scraping-with-python-your-first-scraper-using-beautiful-soup/#respond",{"_57":393,"_59":351,"_60":351,"_16":347,"_61":394,"_370":353,"_371":355,"_14":348,"_64":395,"_67":68,"_87":397},"WebPage",{"_59":63},{"_59":396},"https://wordpress.fulldev.pl/2025/11/19/web-scraping-with-python-your-first-scraper-using-beautiful-soup/#breadcrumb",[398],{"_57":399,"_91":400},"ReadAction",[351],{"_57":70,"_59":396,"_71":402},[403,404],{"_57":74,"_75":76,"_16":77,"_78":79},{"_57":74,"_75":81,"_16":343},{"_57":83,"_59":63,"_60":79,"_16":47,"_14":15,"_84":406,"_87":407,"_67":68},{"_59":86},[408],{"_57":90,"_91":409,"_96":410},{"_57":93,"_94":95},{"_57":98,"_99":100,"_101":102},{"_57":412,"_59":86,"_16":107,"_108":413,"_118":414,"_120":415,"_60":416},[105,106],{"_57":110,"_67":68,"_59":111,"_60":112,"_113":112,"_114":115,"_116":115,"_117":107},{"_59":111},[122],"https://wordpress.fulldev.pl/author/fulldev/","categories_json",[419],{"_10":184,"_12":81,"_14":15,"_16":185,"_20":186},"tags_json",[422],{"_10":337,"_12":76,"_14":15,"_16":380,"_20":380},[],"prev",[],"total","groupedTags",[],"perPage","tag",{"_10":337,"_12":76,"_14":15,"_16":380,"_18":-5,"_20":380,"_22":432,"_24":433,"_458":459},"post_tag",{"_26":434,"_28":435,"_39":40,"_41":42,"_43":434,"_44":436,"_46":47,"_48":49,"_50":437},"beautifulsoup Archives - Fulldev.pl",{"_30":31,"_32":32,"_33":34,"_35":36,"_37":38},"https://wordpress.fulldev.pl/tag/beautifulsoup/",{"_52":53,"_54":438},[439,443,447,453],{"_57":58,"_59":436,"_60":436,"_16":434,"_61":440,"_64":441,"_67":68},{"_59":63},{"_59":442},"https://wordpress.fulldev.pl/tag/beautifulsoup/#breadcrumb",{"_57":70,"_59":442,"_71":444},[445,446],{"_57":74,"_75":76,"_16":77,"_78":79},{"_57":74,"_75":81,"_16":380},{"_57":83,"_59":63,"_60":79,"_16":47,"_14":15,"_84":448,"_87":449,"_67":68},{"_59":86},[450],{"_57":90,"_91":451,"_96":452},{"_57":93,"_94":95},{"_57":98,"_99":100,"_101":102},{"_57":454,"_59":86,"_16":107,"_108":455,"_118":456,"_120":457},[105,106],{"_57":110,"_67":68,"_59":111,"_60":112,"_113":112,"_114":115,"_116":115,"_117":107},{"_59":111},[122],"id_","691dca8f10895acfc3768e13","actionData","errors"]

Articles with tag beautifulsoup:

Web Scraping with Python: Your First Scraper Using Beautiful Soup