reader-lm-0.5b

reader-lm-0.5b is a 494M-parameter small language model specialized for converting messy HTML documents into clean, structured Markdown. It addresses a critical bottleneck in modern data pipelines: transforming web content for LLM consumption without brittle rule-based systems or resource-intensive large models. It achieves professional-grade HTML processing with a compact footprint, handling documents up to 256K tokens.

The model employs a 'shallow-but-wide' decoder-only architecture specifically optimized for selective-copy operations rather than creative text generation. It features 24 transformer layers with 896 hidden dimensions, 14 query heads, and 2 key-value heads (GQA configuration). Training involved two distinct stages: first, training on shorter, simpler HTML (32K tokens) to learn basic conversion patterns; second, training on complex, real-world HTML (128K tokens) to handle challenging cases with inline CSS, scripts, and complex DOM structures. The model incorporates contrastive search during training and implements a repetition detection mechanism to prevent degeneration issues like token loops. A zigzag-ring-attention mechanism enables handling of extremely long sequences up to 256K tokens while maintaining stable performance.

The model achieves a ROUGE-L score of 0.56, indicating strong content preservation, and maintains a low token error rate of 0.34, showing minimal hallucination. In qualitative evaluations across 22 diverse HTML sources (news articles, blog posts, e-commerce pages in multiple languages), it shows particular strength in structure preservation and Markdown syntax usage. It excels at handling complex modern web pages where inline CSS and scripts can expand to hundreds of thousands of tokens — a scenario where traditional rule-based approaches (BeautifulSoup, Readability) often fail. However, it may require additional processing for highly dynamic or JavaScript-heavy pages where content is rendered client-side.

The model works best with raw HTML input and doesn't require special prefixes or instructions. For optimal performance, implement the provided repetition detection mechanism to prevent token loops in output generation. The model supports multiple languages and various HTML structures, but is specifically designed for content extraction and Markdown conversion — do not use it for text generation, summarization, or direct question answering. Available through AWS SageMaker for production deployment and via a Google Colab notebook for experimentation. While the model handles documents up to 256K tokens, processing very large inputs may require additional memory management. For production workloads, CUDA-capable GPU is recommended. For higher accuracy, use reader-lm-1.5b or ReaderLM-v2. For JSON extraction, use ReaderLM-v2 which supports multi-objective optimization for both Markdown and JSON output.