The starting point of media monitoring is turning coverage scattered across news sites into structured data. Built around the cross-region access capability of 9HTTP's Dynamic Residential Proxies, this article focuses on the concrete operations: defining collection fields, extracting titles and publish dates, identifying sources, and handling duplicate content caused by multi-channel syndication.
1. Three Things to Do Before Collecting
- Define your source list: List the news sites, regions, and languages you need to cover. The clearer the sources, the easier it is to write field rules and to scope deduplication later.
- Check collection permissions: Review the target sites' robots protocols and terms of service to confirm what content may be retrieved automatically and what frequency limits apply. Being publicly visible does not mean unlimited collection is allowed; obtaining authorization and following the rules is a prerequisite.
- Define fields and formats: Decide in advance which fields to store for each article and in what format (for example, JSON or CSV), to avoid rework midway.
2. Fields Available on a News Page
A news record typically contains the following fields, most of which can be extracted from the page's HTML structure, meta tags, or structured data:
| Field | Description | Common Locations |
| Title | The article's main headline; some pages include subheadlines | <h1> tag, <meta property="og:title">, structured data headline |
| Publish date | First publication time; some pages also show an update time | <meta property="article:published_time">, structured data datePublished |
| Source | Media name or domain; syndicated articles need the original outlet distinguished | Page header, URL domain, structured data publisher |
| Author | Byline of the writer or editor | Byline area, structured data author |
| Body | The main text of the report | <article> or the body container node |
| Tags / section | Category or topic keywords | Tag area, structured data keywords |
| Canonical URL | The canonical article URL | <link rel="canonical"> or the page URL |
Among these, title, publish date, source, and URL are the four most important fields for media monitoring: titles match topics, dates support timeline analysis, sources distinguish media channels, and URLs provide unique identifiers for deduplication.
3. Extracting the Title
The title usually sits in the page's single <h1> tag. A Python example:
import requests
from bs4 import BeautifulSoup
url = "https://example-news.com/article/12345"
headers = {"User-Agent": "Mozilla/5.0 (compatible; media-monitor/1.0)"}
response = requests.get(url, headers=headers, timeout=30)
soup = BeautifulSoup(response.text, "html.parser")
title = soup.select_one("h1").get_text(strip=True)
print(title)Two things to note:
- Fall back to meta tags: On some sites the
<h1>includes the site name or promotional copy, while<meta property="og:title">is usually cleaner. Fetch both sources and record their priority when storing. - Keep the original text: Do not modify the title during extraction. Cleaning (trimming whitespace, normalizing punctuation) belongs in a separate processing step, which makes the data traceable.
4. Extracting the Publish Date
Publish dates are more scattered than titles. Common locations, ordered by reliability:
- Structured data:
datePublishedin the page's JSON-LD, well formatted — use this first. - Meta tags:
article:published_time; most news sites follow the Open Graph convention. - Displayed time on the page: Requires parsing natural-language formats (for example, "3 hours ago"); least reliable, use only as a fallback.
Extraction and normalization example:
import json, re
from datetime import datetime, timezone
def extract_date(soup):
# Priority 1: JSON-LD
for script in soup.select('script[type="application/ld+json"]'):
try:
data = json.loads(script.string)
items = data if isinstance(data, list) else [data]
for item in items:
if isinstance(item, dict) and item.get("datePublished"):
return item["datePublished"]
except (json.JSONDecodeError, TypeError):
continue
# Priority 2: meta tag
meta = soup.select_one('meta[property="article:published_time"]')
return meta["content"] if meta else None
date_raw = extract_date(soup)
# Normalize to UTC in ISO 8601 format
if date_raw:
dt = datetime.fromisoformat(date_raw.replace("Z", "+00:00"))
date_utc = dt.astimezone(timezone.utc).isoformat()
print(date_utc)Two common mistakes:
- Time zones: The same report shows different local times on different regional sites. Convert to UTC before storing, or timeline analysis will produce out-of-order results.
- Update times: Some pages update
dateModifiedas events develop. Media monitoring generally treatsdatePublishedas authoritative; keepdateModifiedas an additional field to identify continuously updated stories.
5. Extracting the Source
The source field has two layers to distinguish:
- Media name: The outlet credited on the page (for example, a wire service or a newspaper), extractable from structured data
publisher.nameor the page header. - Domain: The domain of the site actually serving the page, extracted from the URL.
When the two differ, the article is usually syndicated: site A republished a report from wire service B. In media monitoring, store both fields — media names support coverage statistics, while domains and syndication relationships support propagation analysis. The "Source: XXX" line in the body and the original-article link field can also help identify syndication.
6. Deduplicating Multi-Channel Syndication
The same story often appears on dozens of sites, and deduplication is the most labor-intensive part of news collection. Three commonly used layers:
- URL normalization: Within one site, remove tracking parameters (
utm_*,spm, and similar) and deduplicate by the canonical URL (<link rel="canonical">). This layer is the cheapest and produces the fewest false positives. - Title matching: Syndicated headlines across sites are often identical or slightly modified. Match exactly after standardizing titles (trim whitespace, normalize full-width/half-width characters and casing); for substantially edited headlines, use edit distance or a similarity threshold (for example, 0.9 or above) to flag suspected duplicates, then spot-check manually.
- Content fingerprinting: Compute fingerprints over the extracted body (for example, MinHash or SimHash over paragraphs) and merge highly similar records. This layer catches "reworded headline, identical content" soft syndication; it is the most expensive to implement and works best as a supplement after the first two layers.
When merging duplicate records, keep these field conventions:
- Keep the earliest publish date and the original source as the article's first appearance.
- Do not delete the other channels' records. Convert them into "syndication records" with their own domains and dates, which supports measuring how widely the story spread.
7. Update Cadence After Storage
- Partition by date: News data is naturally organized by time. Partition by publish date or collection date to simplify incremental processing and backtracking.
- Collect incrementally: Use article listing pages or sitemaps as entry points and process only new links since the last run; skip re-fetching article bodies unless
dateModifiedchanged. - Match frequency to update cadence: News pages update anywhere from minute-level to daily. Set collection frequency to match each site's update rhythm, within the limits its terms of service allow.
8. How 9HTTP Supports Cross-Region News Collection
News sites typically show different homepage and section content to visitors from different regions. Cross-region media monitoring therefore needs network exits in the corresponding regions to see the news lists actually displayed there. 9HTTP's Dynamic Residential Proxies are billed by traffic, cover 195+ countries and regions, and support rotating and sticky sessions — suited to fetching public news pages region by region.
Whether you integrate your own scripts or existing collection tools, validate the target site's responses and field structure with a small batch of requests first, and scale up only after the parsing rules prove effective.
Frequently Asked Questions
How Does News Collection Relate to Sentiment Monitoring?
News collection is the data stage of sentiment monitoring: public coverage is first collected into structured records, then the monitoring system performs topic matching, sentiment analysis, and trend statistics. Collection quality (field completeness and deduplication accuracy) directly determines how reliable the monitoring results are. For use cases, see Multi-Region Sentiment Monitoring Data Collection.
Do Identical Titles Always Mean the Same Story?
No. Same-title content may be a series or a follow-up update. The title-matching layer should keep a "suspected duplicate" flag and merge only after checking publish dates, body fingerprints, and sources with manual spot checks.
What Should You Pay Attention to When Collecting Public News?
Follow the target sites' robots protocols and terms of service, control request frequency, and collect only the public fields your business requires. Content involving personal information is also subject to privacy regulations — being publicly visible does not mean it can be processed without restriction.
Why Do News Lists Look Different Across Regions?
News sites' homepages and sections are usually localized by the visitor's region. Cross-region monitoring needs to access from network exits in the target regions to see what is actually displayed there — which is why multi-region media monitoring pairs with region-targeted proxy exits.
Conclusion
The core of news collection is turning unstructured pages into analyzable field records: the four fields of title, publish date, source, and URL determine data usability, while the three deduplication layers (URL normalization, title matching, content fingerprinting) determine how clean the data is. Validate field rules on a small scale first, then expand the collection scope — and follow the target sites' rules — so your media monitoring rests on a stable data foundation.