Types of Checks

Note

Ozzie Gooen · August 2025 · Completeness: ★★☆ (developing)

Unpublished QURI draft, posted as-is — rough and unfinished; collaborator comments have been removed.

Ozzie Gooen’s take

A quick brainstorm for RoastMyPost.

Opus 4.8’s summary

An August 2025 document consisting almost entirely of a single table cataloging types of automated checks that could be run on written content, presumably by an LLM agent. Rows cover link-status, link-relevancy, credibility, spelling, grammar, markdown formatting, name, arithmetic and advanced-math, and editorial-consistency checks, each rated along Importance, Challenge, Subjectivity, and Frequency, with notes on implementation difficulties (e.g. bot-blocking sites, math needing surrounding context). The cell on credibility checks notes that a more advanced version would use audience-dependent credibility standards. There is essentially no prose beyond the table; it is a structured planning artifact rather than an essay.

		Importance	Challenge	Subjectivity	Frequency
Link Status Check	Does the link exist? (It could be hallucinated, a mistake, or a dead link)Extra: If it fails, it would be nice if we could have a simple AI agent who would try to search and find it.Challenge: Many websites block bots, so it can be surprisingly difficult to check the website.	Low	Low	Low	High
Link Relevancy Check	Does the link have the basic content it is implied to have? This can be quite tricky to validate. Many links aren’t to the direct source referenced, but instead a related website, after which the user is expected to find the source. Ideally we’d have a simple agent run a few steps to investigate.	Medium	Medium	Medium	Medium
Credibility Checks	Are sources of credibility cited in the piece actually as credible as implied? This will likely involve doing some digging. Also applies for cases where it’s claimed that a credible source said X, but they only technically said X. It probably would be good to have a long-lasting list of different sources and their general credibility ratings. A more advanced version would have audience-dependent credibility standards.	Medium	Medium	High	Medium
Spell Check	Not too hard to do a basic job. LLMs can be a bit more advanced. One challenge is choosing UK vs. US English.	Low	Low	Low	High
Grammar Check	Similar to spell check, but can be more subjective	Low	Low	Medium	High
Markdown Formatting Check	Is the item formatted correctly? This can be messy, as different websites format MD differently. I think this isn’t a major concern for content written by humans, but it seems like something to check when it’s by LLMs. I think LLMs often get MD wrong.	Low	Low	Low	Medium
Name check	Are all person/place/etc names in the doc correct? Are they correctly spelled out? This often will require some searching. Bonus points if you can return a relevant link in each case. Wikipedia is the gold standard, other pages can also work.	Low	Low	Low	Medium
Math Check: Arithmetic	Are all simple (i.e. not advanced math) equations in the doc correct? This can ideally be verified with a formal math equation.	Medium	Low	Low	Medium
Math Check: Advanced	Check if advanced math has issues.One major challenge with doing this is context -> Many descriptions of math might reference key previous parts. There might be awkward branching with several strands of thought. Ideally this could be formally checked with Python or similar, though this is often fairly slow.	Medium (Used on LessWrong a fair bit)	Medium	Low-Medium	Medium
Editorial consistency checks	Does the document follow consistent standards? Are there any clear issues, like points made multiple times?	Low	Medium. Difficult to do	Low-Medium	Low (Most blog posts are consistent, longer docs less so)