The text discusses the author's experience with setting up LLM tracing for hinbox using Braintrust and Langfuse. The author initially tried using Braintrust but found it to be rudimentary and lacking in capturing important details...
About:
Alex is an ML Engineer at ZenML with a PhD in History, experienced in various programming ecosystems and an author of books on Afghanistan.
Website:
Specializations:
Interests:
Outgoing Links:
Subscribe to RSS:
The text discusses the importance of instrumenting AI applications and demonstrates how to use Phoenix and litellm to capture inputs and outputs for LLM-powered systems. It also explores the use of BatchSpanProcessor and litellm c...
Alex Strick van Linschoten has developed hinbox, an entity extraction system to help historians and researchers build structured knowledge databases from primary source documents. The tool processes historical documents, academic ...
The text discusses the process of error analysis to find failure modes in LLM application improvement. It outlines a five-step process and emphasizes the importance of analyzing data, clustering failure modes, and iterating to ref...
The text is a summary of the first session of the Hamel/Shreya AI Evals course, focusing on a ‘three gulfs’ mental model for LLM application development and the importance of systematic evaluation and improvement processes. It dis...
The text provides initial impressions of Google Deepmind’s new iteration of Gemini Deep Research that uses their 2.5 Pro model. The author discusses the strengths and weaknesses of the tool, including its verbosity, handling of fo...
The author took a week off to work on a side project, experimenting with different tools and processes. They worked with local models, vision models, and non-English languages, and shared insights on prompting, instruction followi...
The author built a Model Context Protocol (MCP) server for Beeminder to connect AI assistants with personal goal tracking data. The post explains the implementation using Claude Desktop, the role of MCP, and the potential of AI-po...
The text discusses the challenges of document translation and the limitations of Large Language Models (LLMs) in this area. The author, Alex Strick van Linschoten, introduces 'tinbox', a tool he developed to address these challeng...
The text discusses the challenges of document translation and the limitations of Large Language Models (LLMs) in this area. The author, Alex Strick van Linschoten, introduces 'tinbox', a tool he developed to address these challeng...
The text is a reflection on the Hugging Face Agents course, focusing on code agents, evaluations and testing, general patterns, chat templates, and points of leverage for engineers. The author expresses curiosity and confusion abo...
Chapter 10 of Chip Huyen’s 'AI Engineering' explores modern AI system architecture patterns and user feedback mechanisms, covering the evolution from simple API integrations to complex agent-based systems, including practical impl...
Chapter 9 of Chip Huyen’s ‘AI Engineering’ book is a guide to ML inference optimization covering compute and memory bottlenecks, performance metrics, and practical implementation strategies. It emphasizes the critical business nec...
The text explores Chapter 8 of Chip Huyen’s ‘AI Engineering,’ examining the intricate landscape of dataset engineering through the lenses of curation, augmentation, and processing. It discusses data curation, quality criteria, cov...
The text explores the implementation of finetuning effectively, emphasizing it as a last-resort approach after simpler solutions like prompt engineering and RAG have been exhausted. It discusses the decision to finetune, reasons t...
The text explores the implementation of finetuning effectively, emphasizing it as a last-resort approach after simpler solutions like prompt engineering and RAG have been exhausted. It discusses the decision to finetune, reasons t...
The chapter discusses the unification of Retrieval-Augmented Generation (RAG) and Agents as sophisticated approaches to context construction. It covers RAG's purpose, retrieval architecture, cost considerations, optimisation techn...
The text provides a comprehensive guide to AI system evaluation, synthesizing Chapter 4 of Chip Huyen’s ‘AI Engineering.’ It details practical frameworks for assessing AI models, covering evaluation criteria, model selection strat...
Chapter 3 of 'AI Engineering' by Chip Huyen addresses the fundamental question of how to evaluate open-ended responses from foundation models and LLMs at a high level. It provides a comprehensive framework for understanding variou...
The text provides a detailed analysis of Chapter 1 from Chip Huyen’s ‘AI Engineering’ book, covering the transition from ML Engineering to AI Engineering, the three-layer AI stack, and modern development paradigms. It includes ins...
The text provides detailed notes on Chapters 10 and 11 of ‘Prompt Engineering for LLMs’ by Berryman and Ziegler, focusing on LLM application evaluation and future trends. It includes insights on comprehensive testing frameworks, m...
Chapter 6 of 'Prompt Engineering for LLMs' provides a detailed breakdown of prompt structure, document types, and optimization strategies for effective prompt engineering, offering practical tips on information positioning and con...
Chapter 6 of 'Prompt Engineering for LLMs' provides a detailed breakdown of prompt structure, document types, and optimization strategies for effective prompt engineering, offering practical tips on information positioning and con...
Chapter 5 of ‘Prompt Engineering for LLMs’ explores static content (fixed instructions and few-shot examples) versus dynamic content (runtime-assembled context like RAG) in prompts, offering tactical guidance on implementation cho...