Building software with AI

AI software is often built in layers. PyTorch, a Python package, provides tensors, NumPy-style arrays that can live on a CPU or a GPU, and builds deep neural networks on a tape-based automatic differentiation system. Hugging Face Transformers, which works with PyTorch 2.5 or later, provides model definitions for text, vision, audio, video, and multimodal models, for both inference and training. Its README says that a model definition supported there is compatible with many training frameworks and inference engines, including vLLM and SGLang.12

Serving a model to an application often means running an inference server. vLLM, originally developed in UC Berkeley's Sky Computing Lab, is a library for language model inference and serving that manages attention key and value memory with PagedAttention, batches incoming requests continuously, and includes an OpenAI-compatible API server. It runs on NVIDIA, AMD, and Intel GPUs and on x86, ARM, and PowerPC CPUs, among other hardware. SGLang is an open-source inference framework for language, vision-language, and diffusion models, and its package includes SGLang Diffusion, an image and video generation engine.34

Application code often reaches a model through an SDK or framework. Vercel's AI SDK is a provider-agnostic TypeScript toolkit with one API for providers such as OpenAI, Anthropic, and Google; by default it sends requests through the Vercel AI Gateway, and developers can instead connect to a provider directly. LangChain offers a standard interface for models, embeddings, and vector stores. DSPy takes a different approach: developers write compositional Python code, and DSPy's algorithms optimize the prompts and weights. Check which service an SDK calls by default before you send it data.567

Coding agents read, edit, and run code, so their permissions matter. Gemini CLI, Google's Apache-2.0 terminal agent, has built-in tools for file operations, shell commands, web fetching, and Google Search grounding. Its optional sandbox isolates shell commands and file changes from the host: a Docker or Podman sandbox mounts the project directory, and the default macOS profile confines writes to that directory while allowing broad reads and network access. OpenHands warns that running its agent server without a sandbox gives the agent full access to your filesystem. GitHub's Spec Kit instead gives coding agents structured processes and templates, such as spec-driven development.891011

Experiment tracking records what was run and what it produced. With the wandb library from Weights & Biases, a training script starts a run with wandb.init() and logs metrics with run.log(); the quickstart has you create a W&B account and API key, and results are viewed on wandb.ai. MLflow provides tracing, evaluation, and prompt management for agents and language model applications, and its quickstart runs an MLflow server on your own machine and logs OpenAI client calls to it. Logs and traces can contain your data, so check where a tracking tool stores them.1213