Add a News tab: set releases and links to Wizards' articles
The app knew nothing about the game outside your own collection — no way to see that a set drops next week, or that a Secret Lair was announced. Two sources with very different reliability, kept apart so one failing can't take the page down. Releases come from Scryfall's documented /sets endpoint. Clicking a set fetches its cards live rather than reading reference.db, so a set released or spoiled since the last card-data download still works; ownership is joined locally by oracle_id, and the same add-to-library stepper works there as anywhere else. Articles have no supported source at all: Wizards publishes no RSS or Atom feed and no public API, and their page is a Nuxt app whose CSS classes are per-build hashes. So the app links out rather than reproducing anyone's writing, which keeps the scraped surface to titles and hrefs. parse_articles keys on nothing cosmetic — anchors pointing at /en/news/<category>/<slug>, taking the anchor's own text as the title. A restyle can't break that; only a change to their URL scheme could, and the fixture tests fail loudly when it does rather than the page quietly going empty. Verified against a live capture: 15 articles, correct titles and categories. Two things the fixtures caught. Where an article is linked several times per card, the first link is a "Learn More" button, so the longest text wins rather than the first. And a *successful* fetch yielding zero articles is treated as a failure, since that is the signature of their markup changing. get_set_cards returns a full CardRef rather than the handful of fields the grid needs, so the tab renders the same CardTile as the rest of the app and gets click-to-preview instead of a lookalike that does nothing when clicked. Both feeds cache to news-cache.json for six hours and fall back to whatever was last seen, so opening the tab offline shows stale content with its age rather than an error. Also promotes the HTTP agent out of ingest.rs into a shared http module, so timeouts and User-Agent can't drift between the ingester and anything else that reaches the network. Adds tauri-plugin-opener for opening articles externally.
This commit is contained in:
53
crates/mtgc-core/src/http.rs
Normal file
53
crates/mtgc-core/src/http.rs
Normal file
@@ -0,0 +1,53 @@
|
||||
//! Shared HTTP plumbing.
|
||||
//!
|
||||
//! One agent configuration for every outbound request, so timeouts and the
|
||||
//! User-Agent can't drift apart between the bulk ingester and anything else
|
||||
//! that reaches the network.
|
||||
|
||||
use std::io::{BufReader, Read};
|
||||
|
||||
use crate::error::Result;
|
||||
|
||||
/// Identifies this app to the services it calls. Scryfall's API guidelines ask
|
||||
/// for a descriptive User-Agent so they can tell clients apart.
|
||||
pub(crate) const USER_AGENT: &str = "mtgc/0.1 (personal library catalog)";
|
||||
|
||||
/// A default-configured agent.
|
||||
///
|
||||
/// Without explicit timeouts a stalled socket hangs the caller forever with no
|
||||
/// way out, and mobile networks stall like this routinely.
|
||||
///
|
||||
/// `timeout_read` is per read call, not for the whole transfer, so a
|
||||
/// multi-hundred-megabyte download is fine as long as it keeps making progress.
|
||||
pub(crate) fn agent() -> ureq::Agent {
|
||||
ureq::AgentBuilder::new()
|
||||
.timeout_connect(std::time::Duration::from_secs(30))
|
||||
.timeout_read(std::time::Duration::from_secs(60))
|
||||
.build()
|
||||
}
|
||||
|
||||
/// GET `url` as a buffered stream, for responses too large to hold in memory.
|
||||
pub(crate) fn get_reader(url: &str) -> Result<impl Read> {
|
||||
let resp = agent().get(url).set("User-Agent", USER_AGENT).call()?;
|
||||
Ok(BufReader::with_capacity(1 << 20, resp.into_reader()))
|
||||
}
|
||||
|
||||
/// GET `url` as text. For pages small enough to buffer — `ureq` caps this at
|
||||
/// 10MB, which is far above anything we ask for.
|
||||
pub(crate) fn get_string(url: &str) -> Result<String> {
|
||||
Ok(agent()
|
||||
.get(url)
|
||||
.set("User-Agent", USER_AGENT)
|
||||
.call()?
|
||||
.into_string()?)
|
||||
}
|
||||
|
||||
/// GET `url` and deserialize the JSON body.
|
||||
pub(crate) fn get_json<T: serde::de::DeserializeOwned>(url: &str) -> Result<T> {
|
||||
Ok(agent()
|
||||
.get(url)
|
||||
.set("User-Agent", USER_AGENT)
|
||||
.set("Accept", "application/json")
|
||||
.call()?
|
||||
.into_json()?)
|
||||
}
|
||||
@@ -23,7 +23,6 @@ use crate::error::{CoreError, Result};
|
||||
use crate::model::{ColorSet, Legalities};
|
||||
|
||||
const BULK_INDEX_URL: &str = "https://api.scryfall.com/bulk-data";
|
||||
const USER_AGENT: &str = "mtgc/0.1 (personal library catalog)";
|
||||
|
||||
const STAGE_INDEX: &str = "index";
|
||||
const STAGE_ORACLE: &str = "oracle";
|
||||
@@ -202,7 +201,7 @@ fn build_into(
|
||||
overall_percent: overall_percent(&stages, STAGE_ORACLE, 0.0),
|
||||
stages: stages.clone(),
|
||||
});
|
||||
let reader = http_reader(&bulk.oracle_uri)?;
|
||||
let reader = crate::http::get_reader(&bulk.oracle_uri)?;
|
||||
// Progress wraps the *compressed* stream, since `compressed_size` is what
|
||||
// the bulk index reports; decompression happens after the byte count.
|
||||
let reader = progress_reader(
|
||||
@@ -222,7 +221,7 @@ fn build_into(
|
||||
overall_percent: overall_percent(&stages, STAGE_PRINTINGS, 0.0),
|
||||
stages: stages.clone(),
|
||||
});
|
||||
let reader = http_reader(&bulk.default_uri)?;
|
||||
let reader = crate::http::get_reader(&bulk.default_uri)?;
|
||||
let reader = progress_reader(
|
||||
reader,
|
||||
bulk.default_size,
|
||||
@@ -344,27 +343,12 @@ struct BulkUris {
|
||||
default_size: u64,
|
||||
}
|
||||
|
||||
/// HTTP agent with timeouts. Without them a stalled connection hangs forever:
|
||||
/// cancellation is only polled per *parsed record*, so if no bytes arrive, no
|
||||
/// records parse and `CancelToken` is never consulted — the ingest wedges with
|
||||
/// no way out. Mobile networks stall like this routinely.
|
||||
///
|
||||
/// `timeout_read` is per read call, not for the whole transfer, so a
|
||||
/// multi-hundred-megabyte download is fine as long as it keeps making progress.
|
||||
fn agent() -> ureq::Agent {
|
||||
ureq::AgentBuilder::new()
|
||||
.timeout_connect(std::time::Duration::from_secs(30))
|
||||
.timeout_read(std::time::Duration::from_secs(60))
|
||||
.build()
|
||||
}
|
||||
|
||||
/// The agent's timeouts matter especially here: cancellation is only polled per
|
||||
/// *parsed record*, so if no bytes arrive, no records parse and `CancelToken`
|
||||
/// is never consulted — without a read timeout the ingest wedges with no way
|
||||
/// out. See [`crate::http::agent`].
|
||||
fn fetch_bulk_uris() -> Result<BulkUris> {
|
||||
let index: BulkIndex = agent()
|
||||
.get(BULK_INDEX_URL)
|
||||
.set("User-Agent", USER_AGENT)
|
||||
.set("Accept", "application/json")
|
||||
.call()?
|
||||
.into_json()?;
|
||||
let index: BulkIndex = crate::http::get_json(BULK_INDEX_URL)?;
|
||||
|
||||
let find = |kind: &str| {
|
||||
index
|
||||
@@ -383,11 +367,6 @@ fn fetch_bulk_uris() -> Result<BulkUris> {
|
||||
})
|
||||
}
|
||||
|
||||
fn http_reader(url: &str) -> Result<impl Read> {
|
||||
let resp = agent().get(url).set("User-Agent", USER_AGENT).call()?;
|
||||
Ok(BufReader::with_capacity(1 << 20, resp.into_reader()))
|
||||
}
|
||||
|
||||
// ---- Streaming JSONL parse ----------------------------------------------
|
||||
|
||||
/// Deserialize newline-delimited JSON record-by-record, invoking `f` per record
|
||||
|
||||
@@ -10,9 +10,11 @@ pub mod decklist;
|
||||
pub mod decks;
|
||||
pub mod error;
|
||||
pub mod filters;
|
||||
pub(crate) mod http;
|
||||
pub mod ingest;
|
||||
pub mod library;
|
||||
pub mod model;
|
||||
pub mod news;
|
||||
pub mod search;
|
||||
pub mod sync;
|
||||
pub mod synergy;
|
||||
|
||||
493
crates/mtgc-core/src/news.rs
Normal file
493
crates/mtgc-core/src/news.rs
Normal file
@@ -0,0 +1,493 @@
|
||||
//! Keeping up with the game outside your own collection: set releases, and
|
||||
//! links to Wizards' news articles.
|
||||
//!
|
||||
//! Two sources with very different reliability, kept deliberately separate so
|
||||
//! one failing can't take the other down.
|
||||
//!
|
||||
//! **Sets** come from Scryfall's documented `/sets` endpoint — stable JSON,
|
||||
//! same service the card data already comes from.
|
||||
//!
|
||||
//! **Articles** have no supported source at all. Wizards publishes no RSS or
|
||||
//! Atom feed and no public API, and their news page is a server-rendered Nuxt
|
||||
//! app whose CSS classes are per-build hashes. So we read the rendered HTML,
|
||||
//! but only for what's cheap and stable: an article's **title and link**. The
|
||||
//! app links out rather than reproducing anyone's writing or images.
|
||||
//!
|
||||
//! [`parse_articles`] therefore keys on nothing cosmetic. It looks for anchors
|
||||
//! pointing at `/en/news/<category>/<slug>` and takes the anchor's own text as
|
||||
//! the title, which is how that page is built. Category index pages are the
|
||||
//! two-segment form (`/en/news/announcements`) and fall out naturally. A
|
||||
//! restyle can't break this; only a change to their URL scheme, or to putting
|
||||
//! titles somewhere other than the link, would — and the tests below fail
|
||||
//! loudly when that happens instead of the page quietly going empty.
|
||||
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use crate::error::Result;
|
||||
|
||||
const SETS_URL: &str = "https://api.scryfall.com/sets";
|
||||
const NEWS_URL: &str = "https://magic.wizards.com/en/news";
|
||||
const NEWS_ORIGIN: &str = "https://magic.wizards.com";
|
||||
|
||||
/// Set categories that aren't interesting as "releases".
|
||||
///
|
||||
/// Scryfall models tokens, art series, promo packs and Alchemy drops as their
|
||||
/// own sets sharing a release date with the real one, so a single set week
|
||||
/// otherwise shows up as six near-identical rows.
|
||||
const NOISE_SET_TYPES: &[&str] = &[
|
||||
"token",
|
||||
"memorabilia",
|
||||
"promo",
|
||||
"alchemy",
|
||||
"minigame",
|
||||
"vanguard",
|
||||
"treasure_chest",
|
||||
];
|
||||
|
||||
/// A published or upcoming Magic set.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct SetInfo {
|
||||
pub code: String,
|
||||
pub name: String,
|
||||
pub set_type: String,
|
||||
/// ISO date (`YYYY-MM-DD`). Absent for a few unscheduled sets, which sort last.
|
||||
pub released_at: Option<String>,
|
||||
pub card_count: i64,
|
||||
pub icon_uri: String,
|
||||
/// Scryfall's page for the set, for "see everything in this set".
|
||||
pub scryfall_uri: String,
|
||||
}
|
||||
|
||||
/// A news article — title and link only, by design.
|
||||
#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
|
||||
pub struct Article {
|
||||
/// URL segment Wizards files it under, e.g. "announcements", "feature".
|
||||
pub category: String,
|
||||
pub title: String,
|
||||
/// Absolute URL, for opening in the system browser.
|
||||
pub url: String,
|
||||
}
|
||||
|
||||
/// Scryfall's `/sets` response.
|
||||
#[derive(Deserialize)]
|
||||
struct SetsResponse {
|
||||
data: Vec<RawSet>,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
struct RawSet {
|
||||
code: String,
|
||||
name: String,
|
||||
#[serde(default)]
|
||||
set_type: String,
|
||||
#[serde(default)]
|
||||
released_at: Option<String>,
|
||||
#[serde(default)]
|
||||
card_count: i64,
|
||||
#[serde(default)]
|
||||
icon_svg_uri: String,
|
||||
#[serde(default)]
|
||||
scryfall_uri: String,
|
||||
/// Digital-only sets can't be bought in paper, so they're not "releases"
|
||||
/// in the sense this page means.
|
||||
#[serde(default)]
|
||||
digital: bool,
|
||||
}
|
||||
|
||||
/// Parse Scryfall's `/sets` payload into release-worthy sets, newest first.
|
||||
///
|
||||
/// Filters out digital-only sets and the [`NOISE_SET_TYPES`]. Sets with no
|
||||
/// announced date sort last rather than being dropped — an unscheduled set is
|
||||
/// still news.
|
||||
pub fn parse_sets(json: &str) -> Result<Vec<SetInfo>> {
|
||||
let resp: SetsResponse = serde_json::from_str(json)?;
|
||||
let mut sets: Vec<SetInfo> = resp
|
||||
.data
|
||||
.into_iter()
|
||||
.filter(|s| !s.digital && !NOISE_SET_TYPES.contains(&s.set_type.as_str()))
|
||||
.map(|s| SetInfo {
|
||||
code: s.code,
|
||||
name: s.name,
|
||||
set_type: s.set_type,
|
||||
released_at: s.released_at.filter(|d| !d.is_empty()),
|
||||
card_count: s.card_count,
|
||||
icon_uri: s.icon_svg_uri,
|
||||
scryfall_uri: s.scryfall_uri,
|
||||
})
|
||||
.collect();
|
||||
// Newest first; undated last. Reverse-lexicographic works because the
|
||||
// dates are zero-padded ISO.
|
||||
sets.sort_by(|a, b| match (&a.released_at, &b.released_at) {
|
||||
(Some(x), Some(y)) => y.cmp(x),
|
||||
(Some(_), None) => std::cmp::Ordering::Less,
|
||||
(None, Some(_)) => std::cmp::Ordering::Greater,
|
||||
(None, None) => a.name.cmp(&b.name),
|
||||
});
|
||||
Ok(sets)
|
||||
}
|
||||
|
||||
/// Extract article titles and links from the rendered news page.
|
||||
///
|
||||
/// Never fails: unrecognised markup yields an empty list, leaving the caller to
|
||||
/// decide what to show. A hard error here would be indistinguishable from being
|
||||
/// offline, and neither should take down the rest of the page.
|
||||
pub fn parse_articles(html: &str) -> Vec<Article> {
|
||||
let mut out: Vec<Article> = Vec::new();
|
||||
let mut rest = html;
|
||||
|
||||
while let Some(start) = rest.find("<a ") {
|
||||
rest = &rest[start..];
|
||||
// Anchor open tag.
|
||||
let Some(open_end) = rest.find('>') else { break };
|
||||
let tag = &rest[..open_end];
|
||||
let body_start = open_end + 1;
|
||||
let Some(close) = rest[body_start..].find("</a") else {
|
||||
rest = &rest[body_start..];
|
||||
continue;
|
||||
};
|
||||
let body = &rest[body_start..body_start + close];
|
||||
rest = &rest[body_start + close..];
|
||||
|
||||
let Some(href) = attr(tag, "href") else { continue };
|
||||
let Some((category, _slug)) = article_path(&href) else {
|
||||
continue;
|
||||
};
|
||||
let title = text_of(body);
|
||||
if title.is_empty() {
|
||||
continue;
|
||||
}
|
||||
let url = if href.starts_with("http") {
|
||||
href.clone()
|
||||
} else {
|
||||
format!("{NEWS_ORIGIN}{href}")
|
||||
};
|
||||
// The same article is linked several times per card — the image, the
|
||||
// heading, and a "Learn More" button. Keep the longest text: the
|
||||
// generic call-to-action is always shorter than the real title, and
|
||||
// preferring the *first* picks the button on the hero article.
|
||||
match out.iter_mut().find(|a| a.url == url) {
|
||||
Some(existing) => {
|
||||
if title.chars().count() > existing.title.chars().count() {
|
||||
existing.title = title;
|
||||
}
|
||||
}
|
||||
None => out.push(Article { category, title, url }),
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// The `<category>` of an article path, or `None` if this isn't one.
|
||||
///
|
||||
/// Article paths have exactly two segments below `/en/news/`. The one-segment
|
||||
/// form is a category index ("/en/news/announcements"), which is not an article.
|
||||
fn article_path(href: &str) -> Option<(String, String)> {
|
||||
let path = href.strip_prefix(NEWS_ORIGIN).unwrap_or(href);
|
||||
let path = path.split(['?', '#']).next().unwrap_or(path);
|
||||
let rest = path.strip_prefix("/en/news/")?;
|
||||
let mut parts = rest.trim_end_matches('/').split('/');
|
||||
let category = parts.next().filter(|s| !s.is_empty())?;
|
||||
let slug = parts.next().filter(|s| !s.is_empty())?;
|
||||
if parts.next().is_some() {
|
||||
return None; // deeper than an article
|
||||
}
|
||||
Some((category.to_string(), slug.to_string()))
|
||||
}
|
||||
|
||||
/// Value of `name="..."` in an HTML open tag.
|
||||
fn attr(tag: &str, name: &str) -> Option<String> {
|
||||
let key = format!("{name}=\"");
|
||||
let start = tag.find(&key)? + key.len();
|
||||
let end = tag[start..].find('"')?;
|
||||
Some(decode_entities(&tag[start..start + end]))
|
||||
}
|
||||
|
||||
/// Visible text of a fragment: tags stripped, entities decoded, whitespace
|
||||
/// collapsed. Anchor bodies wrap their text in styling spans.
|
||||
fn text_of(fragment: &str) -> String {
|
||||
let mut plain = String::with_capacity(fragment.len());
|
||||
let mut in_tag = false;
|
||||
for ch in fragment.chars() {
|
||||
match ch {
|
||||
'<' => in_tag = true,
|
||||
'>' => in_tag = false,
|
||||
c if !in_tag => plain.push(c),
|
||||
_ => {}
|
||||
}
|
||||
}
|
||||
decode_entities(&plain).split_whitespace().collect::<Vec<_>>().join(" ")
|
||||
}
|
||||
|
||||
/// Decode the HTML entities this page actually uses.
|
||||
///
|
||||
/// Deliberately partial: titles are display text, so an unrecognised entity
|
||||
/// surviving verbatim is a cosmetic blemish, not a correctness problem — and
|
||||
/// far better than pulling in a dependency to be exhaustive.
|
||||
fn decode_entities(s: &str) -> String {
|
||||
if !s.contains('&') {
|
||||
return s.to_string();
|
||||
}
|
||||
let named = [
|
||||
("&", "&"),
|
||||
("<", "<"),
|
||||
(">", ">"),
|
||||
(""", "\""),
|
||||
("'", "'"),
|
||||
(" ", " "),
|
||||
("—", "—"),
|
||||
("–", "–"),
|
||||
("…", "…"),
|
||||
("•", "•"),
|
||||
("©", "©"),
|
||||
("−", "−"),
|
||||
// Product names carry these constantly ("Magic: The Gathering® | The
|
||||
// Hobbit™"). The live page happens to emit them as literal UTF-8, but
|
||||
// handling both forms costs nothing.
|
||||
("®", "®"),
|
||||
("™", "™"),
|
||||
// Smart quotes, which article titles are full of.
|
||||
("’", "\u{2019}"),
|
||||
("‘", "\u{2018}"),
|
||||
("”", "\u{201D}"),
|
||||
("“", "\u{201C}"),
|
||||
("é", "é"),
|
||||
("á", "á"),
|
||||
("â", "â"),
|
||||
("í", "í"),
|
||||
("ó", "ó"),
|
||||
("Ó", "Ó"),
|
||||
("ö", "ö"),
|
||||
];
|
||||
let mut out = s.to_string();
|
||||
for (from, to) in named {
|
||||
if out.contains(from) {
|
||||
out = out.replace(from, to);
|
||||
}
|
||||
}
|
||||
// Numeric forms, decimal and hex.
|
||||
while let Some(start) = out.find("&#") {
|
||||
let Some(len) = out[start..].find(';') else { break };
|
||||
let body = &out[start + 2..start + len];
|
||||
let parsed = body
|
||||
.strip_prefix(['x', 'X'])
|
||||
.and_then(|h| u32::from_str_radix(h, 16).ok())
|
||||
.or_else(|| body.parse::<u32>().ok())
|
||||
.and_then(char::from_u32);
|
||||
match parsed {
|
||||
Some(c) => out.replace_range(start..=start + len, &c.to_string()),
|
||||
// Unparseable: leave it alone, but don't spin on it forever.
|
||||
None => break,
|
||||
}
|
||||
}
|
||||
out
|
||||
}
|
||||
|
||||
/// Fetch the current set list from Scryfall.
|
||||
pub fn fetch_sets() -> Result<Vec<SetInfo>> {
|
||||
parse_sets(&crate::http::get_string(SETS_URL)?)
|
||||
}
|
||||
|
||||
/// Fetch the latest article links from the Wizards news page.
|
||||
pub fn fetch_articles() -> Result<Vec<Article>> {
|
||||
Ok(parse_articles(&crate::http::get_string(NEWS_URL)?))
|
||||
}
|
||||
|
||||
/// Every card in a set, as raw Scryfall JSON pages.
|
||||
///
|
||||
/// Returned unparsed: the caller pairs these with local ownership data, and the
|
||||
/// shape it needs is the shell's concern rather than the core's.
|
||||
pub fn fetch_set_cards_json(set_code: &str) -> Result<String> {
|
||||
let url = format!(
|
||||
"https://api.scryfall.com/cards/search?q=set%3A{}&unique=cards&order=set",
|
||||
urlencode(set_code)
|
||||
);
|
||||
crate::http::get_string(&url)
|
||||
}
|
||||
|
||||
/// Follow a Scryfall-supplied `next_page` URL.
|
||||
///
|
||||
/// Restricted to Scryfall so a compromised or malformed response can't redirect
|
||||
/// us at an arbitrary host.
|
||||
pub fn fetch_url(url: &str) -> Result<String> {
|
||||
if !url.starts_with("https://api.scryfall.com/") {
|
||||
return Err(crate::error::CoreError::Other(format!(
|
||||
"refusing to follow a page link outside Scryfall: {url}"
|
||||
)));
|
||||
}
|
||||
crate::http::get_string(url)
|
||||
}
|
||||
|
||||
/// Percent-encode a set code. Codes are alphanumeric in practice; this exists
|
||||
/// so a hand-typed or future code can't inject query syntax.
|
||||
fn urlencode(s: &str) -> String {
|
||||
s.chars()
|
||||
.flat_map(|c| {
|
||||
if c.is_ascii_alphanumeric() || c == '-' || c == '_' {
|
||||
vec![c]
|
||||
} else {
|
||||
format!("%{:02X}", c as u32).chars().collect()
|
||||
}
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
/// Shaped like the real page: hashed CSS classes, the title repeated in a
|
||||
/// heading and again as the anchor's text, category index links in the nav,
|
||||
/// and styling spans inside the anchor body.
|
||||
const NEWS_FIXTURE: &str = r##"
|
||||
<!DOCTYPE html><html><body>
|
||||
<nav class="css-a1B2c">
|
||||
<a href="/en/news/announcements">Announcements</a>
|
||||
<a href="/en/news/feature">Feature</a>
|
||||
<a href="/en/news/archive">Archive</a>
|
||||
<a href="/en/products/the-hobbit">Products</a>
|
||||
</nav>
|
||||
<div data-article-hub-card-v="1.8.1" class="css-P1ULX">
|
||||
<div class="css-KLPvC">Announcements</div>
|
||||
<h3 class="css-blB8m">Everything Announced for 2027</h3>
|
||||
<div class="css-mjrnX"><p>Dive into a new plane.</p></div>
|
||||
<a data-navigation-type="server-side" href="/en/news/announcements/everything-announced-2027" class="css-1+IR9">
|
||||
Everything Announced for 2027
|
||||
</a>
|
||||
</div>
|
||||
<div data-article-hub-card-v="1.8.1">
|
||||
<a href="/en/news/feature/the-hobbit-prerelease-guide">
|
||||
<span class="css-xY">Magic: The Gathering® | The Hobbit™</span> Prerelease Guide
|
||||
</a>
|
||||
<a href="/en/news/feature/the-hobbit-prerelease-guide" class="css-dup">
|
||||
<img src="/x.png" alt="">
|
||||
</a>
|
||||
</div>
|
||||
<a href="https://magic.wizards.com/en/news/feature/rules-update">Rules Update & Errata</a>
|
||||
<a href="/en/news/feature/deep/nested/path">Too deep to be an article</a>
|
||||
</body></html>
|
||||
"##;
|
||||
|
||||
#[test]
|
||||
fn extracts_articles_by_link_shape_not_by_styling() {
|
||||
let got = parse_articles(NEWS_FIXTURE);
|
||||
|
||||
let titles: Vec<&str> = got.iter().map(|a| a.title.as_str()).collect();
|
||||
assert_eq!(
|
||||
titles,
|
||||
vec![
|
||||
"Everything Announced for 2027",
|
||||
// Entities decoded, nested spans flattened, whitespace collapsed.
|
||||
"Magic: The Gathering® | The Hobbit™ Prerelease Guide",
|
||||
"Rules Update & Errata",
|
||||
]
|
||||
);
|
||||
assert_eq!(got[0].category, "announcements");
|
||||
assert_eq!(got[1].category, "feature");
|
||||
assert_eq!(
|
||||
got[0].url,
|
||||
"https://magic.wizards.com/en/news/announcements/everything-announced-2027",
|
||||
"relative hrefs become absolute so they can be opened"
|
||||
);
|
||||
assert_eq!(
|
||||
got[2].url, "https://magic.wizards.com/en/news/feature/rules-update",
|
||||
"already-absolute hrefs are left alone"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn ignores_category_pages_unrelated_links_and_repeats() {
|
||||
let got = parse_articles(NEWS_FIXTURE);
|
||||
let urls: Vec<&str> = got.iter().map(|a| a.url.as_str()).collect();
|
||||
|
||||
assert!(
|
||||
!urls.iter().any(|u| u.ends_with("/en/news/announcements")),
|
||||
"a category index is not an article"
|
||||
);
|
||||
assert!(!urls.iter().any(|u| u.contains("/products/")));
|
||||
assert!(!urls.iter().any(|u| u.contains("nested")), "deeper paths aren't articles");
|
||||
assert_eq!(
|
||||
urls.iter().filter(|u| u.contains("prerelease-guide")).count(),
|
||||
1,
|
||||
"the same article linked twice should appear once"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn a_generic_call_to_action_never_wins_over_the_real_title() {
|
||||
// Observed on the live page: the hero article is linked first by a
|
||||
// "Learn More" button, then again by its heading. Taking the first
|
||||
// link would file the article under "Learn More".
|
||||
let html = r#"
|
||||
<a href="/en/news/feature/big-story">Learn More</a>
|
||||
<h3>The Actual Headline of the Story</h3>
|
||||
<a href="/en/news/feature/big-story">The Actual Headline of the Story</a>
|
||||
"#;
|
||||
let got = parse_articles(html);
|
||||
assert_eq!(got.len(), 1);
|
||||
assert_eq!(got[0].title, "The Actual Headline of the Story");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unrecognisable_markup_yields_nothing_rather_than_failing() {
|
||||
// The day Wizards redesigns, this is what we get. It must not panic,
|
||||
// and must not be mistaken for "there is no news".
|
||||
assert!(parse_articles("<html><body><p>hello</p></body></html>").is_empty());
|
||||
assert!(parse_articles("").is_empty());
|
||||
assert!(parse_articles("<a href=\"/en/news/x/y\"></a>").is_empty(), "empty title");
|
||||
assert!(parse_articles("<a href=").is_empty(), "truncated tag");
|
||||
}
|
||||
|
||||
const SETS_FIXTURE: &str = r#"{
|
||||
"data": [
|
||||
{"code":"ecl","name":"Lorwyn Eclipsed","set_type":"expansion","released_at":"2026-01-23",
|
||||
"card_count":408,"icon_svg_uri":"https://svgs.scryfall.io/sets/ecl.svg",
|
||||
"scryfall_uri":"https://scryfall.com/sets/ecl","digital":false},
|
||||
{"code":"tecl","name":"Lorwyn Eclipsed Tokens","set_type":"token","released_at":"2026-01-23",
|
||||
"card_count":13,"icon_svg_uri":"","scryfall_uri":"","digital":false},
|
||||
{"code":"yecl","name":"Alchemy: Lorwyn Eclipsed","set_type":"alchemy","released_at":"2026-02-03",
|
||||
"card_count":30,"icon_svg_uri":"","scryfall_uri":"","digital":true},
|
||||
{"code":"fut","name":"Future Sight","set_type":"expansion","released_at":"2007-05-04",
|
||||
"card_count":180,"icon_svg_uri":"","scryfall_uri":"","digital":false},
|
||||
{"code":"unk","name":"Unannounced Thing","set_type":"expansion",
|
||||
"card_count":0,"icon_svg_uri":"","scryfall_uri":"","digital":false}
|
||||
]
|
||||
}"#;
|
||||
|
||||
#[test]
|
||||
fn sets_are_newest_first_with_undated_last() {
|
||||
let got = parse_sets(SETS_FIXTURE).unwrap();
|
||||
let codes: Vec<&str> = got.iter().map(|s| s.code.as_str()).collect();
|
||||
assert_eq!(
|
||||
codes,
|
||||
vec!["ecl", "fut", "unk"],
|
||||
"newest dated first, undated last"
|
||||
);
|
||||
assert_eq!(got[0].card_count, 408);
|
||||
assert_eq!(got[2].released_at, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn token_and_digital_sets_are_not_releases() {
|
||||
let got = parse_sets(SETS_FIXTURE).unwrap();
|
||||
let codes: Vec<&str> = got.iter().map(|s| s.code.as_str()).collect();
|
||||
assert!(
|
||||
!codes.contains(&"tecl"),
|
||||
"a token set shares its date with the real one and would just be noise"
|
||||
);
|
||||
assert!(!codes.contains(&"yecl"), "digital-only can't be bought in paper");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn malformed_set_json_is_an_error_not_a_panic() {
|
||||
assert!(parse_sets("not json").is_err());
|
||||
assert!(parse_sets(r#"{"data":[]}"#).unwrap().is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn set_codes_are_escaped_into_the_query() {
|
||||
assert_eq!(urlencode("ecl"), "ecl");
|
||||
assert_eq!(urlencode("a b"), "a%20b");
|
||||
assert_eq!(urlencode("x&q=1"), "x%26q%3D1", "can't break out of the query");
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user