reader-lm-1.5b

reader-lm-1.5b is a 1.54B-parameter small language model that converts complex HTML documents into clean Markdown, outperforming models 50× its size including GPT-4 and Gemini-1.5-Pro on HTML-to-Markdown conversion. It handles documents up to 256K tokens natively, eliminating the need for expensive chunking operations. The model's 'shallow-but-wide' architecture is optimized for selective-copy operations, achieving high accuracy without the computational overhead of general-purpose LLMs.

The model employs a 'shallow-but-wide' decoder-only architecture: 28 transformer layers with 12 query heads and 2 key-value heads (GQA), 1536 hidden dimensions, and 8960 intermediate dimensions. This configuration challenges the conventional wisdom that deeper models are always better — for selective-copy tasks like HTML-to-Markdown, width (more parameters per layer) is more effective than depth (more layers). Training followed two stages: (1) short-and-simple HTML (32K tokens) for basic conversion patterns, (2) long-and-hard HTML (128K tokens) for real-world complexity. The zigzag-ring-attention mechanism enables 256K-token processing. Contrastive search and repetition detection prevent common small-LM pathologies like degeneration and token loops.

The model achieves ROUGE-L of 0.72 and Token Error Rate of 0.19, significantly outperforming GPT-4 (0.43 ROUGE-L, 0.50 TER) and Gemini-1.5-Pro (0.42 ROUGE-L, 0.48 TER) on HTML-to-Markdown conversion. Qualitative evaluations across four key dimensions — header extraction, main content extraction, rich structure preservation, and Markdown syntax usage — show consistent high accuracy across diverse document types: news articles, blog posts, landing pages, and forum posts. The model handles multiple languages including English, German, Japanese, and Chinese. This performance is achieved while processing documents up to 256K tokens, eliminating chunking overhead that larger models typically require.

Use this model for complex HTML document processing where accuracy and efficiency are paramount. It requires CUDA-capable GPU infrastructure for optimal performance but runs on more modest hardware than general-purpose LLMs. Available through AWS SageMaker and Azure Marketplace. The model is specifically optimized for HTML-to-Markdown conversion and is not suitable for general-purpose text generation or other NLP tasks. When processing extremely long documents (approaching 512K tokens), performance may degrade as this exceeds the model's training range (256K). Implement the provided repetition detection mechanisms and consider using contrastive search during inference to maintain output quality. For JSON extraction alongside Markdown, use ReaderLM-v2 which supports multi-objective optimization.