How LLMs Actually Work, Real-World Applications, and the Models Shaping Our Future

If you’ve been following tech news over the last few years, or if you read my recent posts on building RAG pipelines in Node.js and scaling distributed systems, you know we are living through a massive shift in software engineering.

A few years ago, AI was mostly something data scientists worked on in Python notebooks using statistical libraries. Today, as web developers, we write code that calls Large Language Model (LLM) APIs, orchestrates vector databases, and builds intelligent agents that can read documents, generate code, and answer user queries.

To most people, tools like ChatGPT, Claude, and Gemini feel like actual magic. You type a prompt into a text box, hit enter, and a sentence appears, word by word, giving you complex code snippets, explaining quantum physics, or summarizing a 50-page legal document.

As software developers, we know that magic is just technology we haven’t broken down into system components yet.

In this deep dive, we are going to strip away the hype, pull back the curtain, and explore how LLMs actually work under the hood, look at how they are being used in real-world applications, and break down the major models defining the industry today.

Part 1: How LLMs Work (Demystifying the Neural Mechanics)

At its absolute core, stripped of all marketing jargon, a Large Language Model is a probabilistic next-token predictor.

It does not “think” like a human brain, it does not possess consciousness, and it does not have a secret database of facts stored as text files on a server. Instead, it is a massive mathematical function (a neural network with billions of numerical weights) trained to answer one fundamental question:

“Given this exact sequence of preceding words, what is statistically the most likely next word to follow?”

Let’s break down the multi-step journey a sentence takes from the moment you hit “send” in your browser to the moment the model outputs a response.

+-----------------------------------------------------------------------+
|                       THE LLM EXECUTION PIPELINE                       |
+-----------------------------------------------------------------------+
|                                                                       |
|  [ Raw User Text Input ]                                              |
|            │                                                          |
|            ▼                                                          |
|   1. TOKENIZATION   ───> Breaks text into numerical token IDs        |
|            │                                                          |
|            ▼                                                          |
|   2. VECTOR EMBEDDINGS ─> Maps tokens to high-dimensional space      |
|            │                                                          |
|            ▼                                                          |
|   3. TRANSFORMER BLOCKS                                               |
|      ├── Multi-Head Self-Attention (Passes context between tokens)    |
|      └── Feed-Forward Network  (Processes patterns)                   |
|            │                                                          |
|            ▼                                                          |
|   4. PROBABILITY DISTRIBUTION (Softmax Layer over Vocabulary)          |
|            │                                                          |
|            ▼                                                          |
|   5. TOKEN SELECTION & DECODING (Appends next token & loops)          |
|                                                                       |
+-----------------------------------------------------------------------+

Step 1: Tokenization (Converting Human Text to Machine Numbers)

Computers cannot process raw English, Punjabi, or JavaScript strings directly. They only process numbers. The first step in any LLM pipeline is Tokenization.

A tokenizer breaks down input text into small fragments called tokens. A token isn’t always a complete word; it can be a full word, a sub-word, a prefix, a suffix, or even a single punctuation mark.

• The sentence: "I love building web applications."
• Might be split into tokens: ["I", " love", " build", "ing", " web", " applications", "."]

Each unique token in a model’s vocabulary is assigned a specific integer ID. For instance, in OpenAI’s cl100k_base tokenizer, the word "hello" might map to ID 15339, while "world" maps to 1917.

Raw Text: "Hello world"  ──> Tokenizer ──> Token IDs: [15339, 1917]

Why Sub-word Tokenization Matters:
If a model only used complete words, its vocabulary list would need to store millions of words in every human language, making the neural network unnecessarily massive. By breaking text into sub-words (like "build" + "ing"), the model can process brand new, unseen words just by combining familiar sub-tokens.

Step 2: Vector Embeddings (Mapping Meaning to Math)

Once the text is converted into a list of integer IDs, those integers are passed into an Embedding Layer.

An embedding maps each token ID to a dense vector—a long list of floating-point numbers (often 4,096 to 12,288 dimensions long). These numbers place the token in a high-dimensional mathematical space where geometric distance equals semantic meaning.

                        High-Dimensional Vector Space
                                (Simplified 2D)

                        ┌─────────────────────────┐
                        │                         │
                        │    O (King)             │
                        │    │                    │
                        │    │ Vector Difference  │
                        │    ▼                    │
                        │    O (Queen)            │
                        │                         │
                        │        O (Apple)        │
                        │        O (Banana)       │
                        │                         │
                        └─────────────────────────┘

In this vector space:

• The vector for "king" minus the vector for "man" plus the vector for "woman" results in a vector that sits mathematically very close to "queen".
• The words "apple" and "banana" sit close to each other because they both appear near words like "fruit", "sweet", and "eat".

These initial embeddings give the neural network its foundational understanding of dictionary definitions and conceptual relationships.

Step 3: The Transformer Architecture & Self-Attention

Before 2017, natural language processing models (like RNNs and LSTMs) processed text sequentially, one word at a time, from left to right. This was slow and had a major memory flaw: by the time the model reached word #50 in a paragraph, it had already forgotten key details from word #1.

In 2017, researchers at Google published the landmark paper “Attention Is All You Need”, introducing the Transformer Architecture.

Transformers revolutionized AI because they can process an entire block of text all at once, in parallel, using a mechanism called Self-Attention.

How Self-Attention Works: Resolving Context

Consider these two sentences:

1. “The bank of the river was covered in green grass.”
2. “I need to go to the bank to deposit my paycheck.”

The word "bank" is spelled identically in both sentences, but its meaning is completely different. A traditional dictionary look-up cannot tell the difference.

Self-Attention solves this. As text passes through the Transformer’s attention layers, every single token looks at every other token in the prompt and asks: “How much attention should I pay to the surrounding words to understand my own context?”

• In Sentence 1, the token "bank" pays high attention to "river" and "grass". Its mathematical vector is updated dynamically to mean geological riverbank.
• In Sentence 2, the token "bank" pays high attention to "deposit" and "paycheck". Its vector is updated to mean financial institution.

Sentence: "The animal didn't cross the street because it was too tired."

Question: What does "it" refer to?

Attention Weights for "it":
  - "The"    [ 0.01 ]
  - "animal" [ 0.85 ]  <── Highest Attention Score!
  - "cross"  [ 0.04 ]
  - "street" [ 0.10 ]

Through self-attention, the model builds a rich, context-aware understanding of every word in relation to the entire context window.

Step 4: Probability Distribution & Sampling (Selecting the Next Word)

After the text passes through dozens of Transformer layers, the model outputs a final raw array of numbers called Logits.

These logits are passed through a mathematical function called Softmax, which converts them into a clean probability distribution over the model’s entire vocabulary (usually around 100,000 possible tokens).

Suppose your prompt was: “The capital of France is…”

The model might calculate the following probabilities for the very next token:

┌─────────────┬─────────────┐
│ Token       │ Probability │
├─────────────┼─────────────┤
│ " Paris"    │    94.2%    │
│ " a"        │     3.1%    │
│ " the"      │     1.5%    │
│ " located"  │     0.8%    │
│ " London"   │     0.1%    │
└─────────────┴─────────────┘

Controlling Randomness: Temperature and Top-P

How does the model choose which word to output from this probability list? That’s where hyper-parameters come in:

• Temperature (0.0 to 2.0):
• At Temperature = 0.0, the model always picks the single highest-probability token (" Paris"). This makes the output deterministic, predictable, and ideal for coding or math.
• At Temperature = 0.8+, the model flattens the probability scores, giving lower-probability words a chance to be picked. This produces creative, novel, but potentially less factual writing.
• Top-P (Nucleus Sampling): Limits the selection pool to a percentage cumulative probability cutoff. For example, Top-P = 0.9 tells the model to only sample from the top words whose combined probabilities add up to 90%, cutting off unlikely noise words.

Once the next token is picked (e.g., " Paris"), it is appended to the original prompt, and the entire process repeats in a loop to predict the token after that. This step-by-step loop is why LLM responses stream onto your screen one fragment at a time!

Step 5: How LLMs Are Trained (Pre-training vs. RLHF)

Building a modern conversational LLM involves three distinct training phases:

┌────────────────────────────────────────────────────────────────────────┐
│                        LLM TRAINING PIPELINE                           │
├────────────────────────────────────────────────────────────────────────┤
│                                                                        │
│  1. PRE-TRAINING (Unsupervised Learning)                               │
│     - Feed petabytes of raw internet text (web pages, books, code).     │
│     - Learn language patterns, world knowledge, and syntax.            │
│     - Result: "Base Model" (Great at completing text, bad at chat).    │
│                                                                        │
│  2. SUPERVISED FINE-TUNING (SFT)                                       │
│     - Feed thousands of curated Q&A prompt templates written by humans. │
│     - Teaches the model to act like a helpful assistant.               │
│     - Result: "Instruct Model".                                        │
│                                                                        │
│  3. RLHF (Reinforcement Learning from Human Feedback)                  │
│     - Human reviewers rate multiple model responses from best to worst.│
│     - Train a reward model to align outputs with human preferences     │
│       (safety, truthfulness, helpfulness).                             │
│                                                                        │
└────────────────────────────────────────────────────────────────────────┘

1. Pre-training: The expensive foundation phase. The model reads hundreds of billions of web pages, books, Wikipedia entries, and public code repositories. It spends millions of GPU-hours guessing hidden words in sentences, learning language grammar and world knowledge.
2. Supervised Fine-Tuning (SFT): A raw base model doesn’t answer questions; if you prompt it with “What is the capital of France?”, it might just complete your text with “and what is the capital of Germany?”. SFT uses human-curated instruction datasets to train the model to respond in a helpful Q&A conversational format.
3. RLHF (Reinforcement Learning from Human Feedback): Humans rate various model answers to teach it safety guidelines, tone alignment, and how to avoid harmful or toxic output.

Part 2: Popular Real-World Applications of LLMs

While non-technical users view LLMs as simple chatbots, software developers and enterprise architects are building massive ecosystems around them. Here are the core application pillars transforming industries today:

                              LLM APPLICATION PILLARS

            ┌────────────────────────────┼────────────────────────────┐
            ▼                            ▼                            ▼
  ┌──────────────────┐         ┌──────────────────┐         ┌──────────────────┐
  │ Code Generation  │         │ RAG & Enterprise │         │ Autonomous AI    │
  │  & Assistance    │         │ Search Engines   │         │  Agentic Systems │
  └──────────────────┘         └──────────────────┘         └──────────────────┘

1. Automated Software Engineering & Code Generation

Tools like GitHub Copilot, Cursor IDE, and Claude Code have fundamentally changed how software is written.

• Inline Autocomplete: LLMs complete complex algorithms, regular expressions, and boilerplate boilerplate syntax in real-time as developers type.
• Codebase refactoring: Developers can highlight an entire legacy backend file and ask the LLM to rewrite it from callback-style code to async/await or convert JavaScript logic into TypeScript types.
• Bug Diagnosis: Pasting terminal stack traces into an LLM allows it to trace variable references, identify null pointer exceptions, and suggest exact fixes.

2. Enterprise Search & RAG (Retrieval-Augmented Generation)

As explored in our RAG architectural guide, enterprises use LLMs combined with Vector Databases (like MongoDB Atlas Vector Search, Pinecone, or Qdrant) to query internal corporate knowledge bases.

• Internal HR & Legal Bots: Employees ask questions like “What is our parental leave policy?”, and the system retrieves the specific internal document pages, feeds them to the LLM as context, and generates an exact, cited answer.
• Customer Support Automation: Support ticket systems read incoming user complaints, fetch user order histories, and generate personalized resolution drafts for support agents to review and send with a single click.

3. Agentic Workflows & Function Calling (Tool-Use)

Modern LLMs are no longer passive text generators; they can act as the reasoning engine of autonomous software agents.

• Through Function Calling (Tool Use), an LLM can decide when it needs external data and output a structured JSON schema requesting an API call.
• Example Agent Flow:

1. User: “Book a flight to New Delhi for next Tuesday and send the confirmation to my email.”
2. LLM evaluates prompt and calls search_flights(destination="DEL", date="2026-07-28").
3. App backend runs the flight API and sends results back to the LLM.
4. LLM calls book_ticket(flight_id="AI-102") and then calls send_email(...).

4. Data Extraction & Unstructured Data Processing

Before LLMs, extracting structured data from unstructured sources (like scanned invoices, medical records, or handwritten PDFs) required fragile custom Regex patterns and brittle OCR scripts.

• LLMs accept unstructured text, HTML, or images and reliably format the extracted details into strict, validated JSON schemas ready for insertion into SQL or MongoDB databases.

Part 3: Popular LLMs Today (The Model Landscape)

The LLM ecosystem has matured into a fiercely competitive market divided into two main camps: Proprietary Closed-Source Models (accessed via cloud APIs) and Open-Source / Open-Weights Models (which you can self-host on your own hardware).

                             THE LLM ECOSYSTEM
                                     │
         ┌───────────────────────────┴───────────────────────────┐
         ▼                                                       ▼
  PROPRIETARY MODELS                                     OPEN-WEIGHTS MODELS
  (Hosted APIs)                                          (Self-Hostable)
  ├── OpenAI (GPT-4o, o3-mini)                           ├── Meta (Llama 3.3)
  ├── Anthropic (Claude 3.5 Sonnet)                      ├── DeepSeek (V3, R1)
  └── Google (Gemini 1.5 Pro / Flash)                    └── Mistral AI (Mistral Large)

Proprietary Models (API-First)

1. OpenAI (GPT-4o, o1, o3-mini)

OpenAI remains the commercial benchmark in the industry.

• GPT-4o (“Omni”): A natively multimodal model that processes text, audio, and visual inputs simultaneously within a single neural network, delivering ultra-low latency responses.
• o1 & o3 Series (Reasoning Models): Unlike standard next-token predictors, OpenAI’s reasoning models use Chain-of-Thought (CoT) inference processing before outputting an answer. They generate internal “thinking steps” to solve complex mathematical proofs, competitive programming challenges, and intricate scientific research problems.

2. Anthropic (Claude 3.5 Sonnet, Claude 3 Opus)

Founded by former OpenAI researchers, Anthropic prioritizes safety, steerability, and coding performance.

• Claude 3.5 Sonnet: Widely considered by developers to be the single best model for software engineering, web development, and structural logic. Its ability to navigate complex codebases without losing context or introducing subtle syntax bugs has made it the default engine for tools like Cursor and Claude Code.
• Artifacts Feature: Popularized the user-interface pattern where generated code, SVG diagrams, and React components render live alongside the chat stream.

3. Google (Gemini 1.5 Pro & 1.5 Flash)

Google’s flagship AI models excel in context length and multimodal processing.

• Massive Context Window (2 Million+ Tokens): Gemini 1.5 Pro features a massive context window. Developers can upload entire code repositories, 1-hour videos, or full length books into a single prompt call without needing complex RAG chunking pipelines.
• Developer Ecosystem Integration: Seamlessly integrated into Google Cloud Platform (Vertex AI), Firebase, and Google Workspace ecosystems.

Open-Weights & Open-Source Models (Run Anywhere)

1. Meta (Llama 3.3 Series)

Meta’s release of the open-weights Llama family transformed the AI landscape by giving developers, researchers, and enterprises access to state-of-the-art models that can be hosted locally or deployed on private cloud infrastructure.

• Why it matters: Businesses with strict data privacy requirements (like healthcare providers, banks, or defense contractors) cannot send customer data over external APIs. With Llama 3.3 (70B), they can run enterprise-grade LLMs completely offline within their own private AWS VPCs.

2. DeepSeek (DeepSeek-V3 & DeepSeek-R1)

DeepSeek shocked the global AI community by releasing open-weights models that match top proprietary models at a fraction of the traditional training cost.

• Architectural Innovations: Uses Multi-Head Latent Attention (MLA) and Mixture-of-Experts (MoE) architecture, where only a fraction of the total parameters are activated for any single token query, drastically reducing GPU compute requirements.
• DeepSeek-R1: An open-weights reasoning model that uses reinforcement learning to generate detailed step-by-step thinking paths before delivering an answer, making advanced reasoning open and accessible to developers worldwide.

3. Mistral AI (Mistral Small / Medium / Large)

Based out of France, Mistral AI builds efficient, high-performance open-weights models optimized for low latency and easy deployment on edge devices and local developer rigs.

Part 4: Key Concepts & Hyper-parameters Summary

For web developers working with LLM APIs, here is a handy quick-reference table for common parameters and metrics:

Conclusion: The Horizon for Developers

Understanding how LLMs function changes how you approach software engineering. When you stop viewing an LLM as a magical black box and start viewing it as a probabilistic engine that converts text to numerical vectors and processes context through self-attention, you can design better prompts, build more resilient RAG architectures, and build reliable agentic applications.

As open-weights models like Llama and DeepSeek get more efficient and proprietary models like Claude and Gemini expand in reasoning and multimodal capabilities, the line between traditional web development and AI engineering is disappearing.

Are you building with LLM APIs, local open-source models, or custom RAG pipelines in your stack? Let me know in the comments below!

Divyanshu Rawat

25 year old AI & MERN Stack Developer based in Mohali, Punjab, India, with a strong interest in system design, Artificial Intelligence, Large Language Models (LLMs), and scalable software architecture. Passionate about building innovative, intelligent, and high-performance applications.