llms.txt and llms-full.txt support for Razor SSG and Razor Press
Benefits for Large Language Models
Large language models gain a lot of their knowledge from information on websites, but their small context windows aren't capable of handling most websites whose complex HTML Pages contain a mix of content, scripts, styles, navigation, ads and structural elements that can introduce noise and ambiguity. Parsing this content requires additional preprocessing to extract meaningful content while filtering out irrelevant elements, which can be inefficient and error-prone.
Instead LLMs thrive on clean, structured text. To address this, llmstxt.org
have proposed adding /llms.txt and llms-full.txt markdown files to websites to provide a more
concise, expert-level information that LLMs can consume in a single, accessible location that's
ideal for quickly adding context to LLMs, improves text comprehension, enhances model accuracy,
streamlines content ingestion for RAG workflows and other AI-powered applications.
Markdown was chosen for content as it's lightweight, clean, and structured specifically for readability. Markdown files contain pure text with minimal formatting, making them an ideal source for LLM training and retrieval. Because markdown is designed for capturing content, it ensures that LLMs receive high-quality, contextually relevant information without the distractions of complex page layouts or extraneous code.
Support for llms.txt and llms-full.txt
Fortunately razor-ssg and razor-press
have fully embraced Markdown for maintaining their content, making it trivial to support generating
/llms.txt and llms-full.txt files from your original static markdown content from the
new Llms.cshtml
and LlmsFull.cshtml
pages whose content is sourced from the Websites Markdown
/_pages folder rendered from the
links in its sidebar.json
and Blog posts in the /_posts folder in descending order.
Here are some examples of llms.txt and llms-full.txt files generated from the Razor SSG and Razor Press website Templates:
Create a new Razor SSG or Razor Pages Project
Razor SSG is our FREE Project Template for creating fast, statically generated Websites and Blogs with Markdown & C# Razor Pages, whilst Razor Press is more focused on maintaining documentation websites.
