# GitHub resources for Data Science & AI

_36 thoughts, 31 connections_

### 10 GitHub repos every Data Scientist should bookmark in 2026
Tags: #data-science #ai-engineering #github
Priority: ★★★★★
Connected: enables "rasbt/LLMs-from-scratch", enables "karpathy/nn-zero-to-hero", enables "mlabonne/llm-course", enables "DataTalksClub/llm-zoomcamp", enables "microsoft/generative-ai-for-beginners", enables "Shubhamsaboo/awesome-llm-apps", enables "GokuMohandas/Made-With-ML", enables "vllm-project/vllm", enables "langchain-ai/langgraph", enables "dair-ai/Prompt-Engineering-Guide"

A practical learning map for Data Scientists whose role is expanding into LLMs, RAG, evaluation, serving, MLOps, and agents.

The point is not to become a full AI Engineer overnight. It is to understand more of the modern stack so you can move confidently from notebooks and models into production AI systems.

Learning path:

Foundations & model internals
→ RAG and application building
→ Production / MLOps / serving
→ Agents and context engineering

Use this map as a guided reading and project list. Each child node explains what the repository teaches, why it matters for a Data Scientist, what to focus on first, and a practical way to learn from it.

### rasbt/LLMs-from-scratch
Tags: #foundations #llm #pytorch
Connected: depends on "10 GitHub repos every Data Scientist should bookmark in 2026"

Focus: Foundations

What it teaches:
• Build a GPT-style large language model step by step in PyTorch.
• Tokenization, embeddings, attention, transformer blocks, pretraining, and fine-tuning.
• How the pieces of an LLM fit together instead of treating the model as a black box.

Why it matters for a Data Scientist:
Modern Data Science increasingly involves LLM-based systems. You do not need to train foundation models yourself, but understanding their internals makes debugging, evaluation, and model selection much easier.

What to focus on first:
1. Tokenization and embeddings
2. Self-attention
3. Transformer blocks
4. Training loop
5. Fine-tuning

Practical project:
Build the smallest working GPT-style model you can, train it on a tiny text corpus, then change one component and observe what happens.

Best use: Deep understanding, not quick API usage.

Source:
https://github.com/rasbt/LLMs-from-scratch

### karpathy/nn-zero-to-hero
Tags: #deep-learning #foundations #neural-nets
Connected: depends on "10 GitHub repos every Data Scientist should bookmark in 2026"

Focus: Deep Learning Core

What it teaches:
• Neural networks from first principles.
• Backpropagation, gradients, optimization, language modeling, and progressively deeper architectures.
• How frameworks automate the math you need to understand when models behave badly.

Why it matters for a Data Scientist:
LLMs sit on top of deep-learning fundamentals. This repo strengthens the mental models needed to diagnose training instability, overfitting, bad gradients, and unexpected model behavior.

What to focus on first:
1. Micrograd and backpropagation
2. Makemore
3. Embeddings and hidden layers
4. Training dynamics
5. Transformer intuition

Practical project:
Implement a tiny neural network without hiding behind high-level libraries, then reproduce the same model in PyTorch and compare the two.

Best use: Rebuilding intuition when deep learning has become too abstract.

Source:
https://github.com/karpathy/nn-zero-to-hero

### mlabonne/llm-course
Tags: #fine-tuning #llm #evaluation
Connected: depends on "10 GitHub repos every Data Scientist should bookmark in 2026"

Focus: Model Specialization

What it teaches:
• The practical LLM Engineer roadmap.
• Fine-tuning with LoRA / QLoRA.
• Quantization approaches such as GGUF / AWQ.
• Dataset preparation, evaluation, and model specialization.

Why it matters for a Data Scientist:
Once you move beyond prompting, you need to know when to customize a model, how to do it efficiently, and how to measure whether the specialized model is actually better.

What to focus on first:
1. When fine-tuning is justified
2. LoRA / QLoRA intuition
3. Dataset quality
4. Quantization trade-offs
5. Evaluation after specialization

Practical project:
Take a small open model and specialize it on a narrow domain dataset. Compare the base model and tuned model on a fixed evaluation set.

Best use: Moving from model user to model customizer.

Source:
https://github.com/mlabonne/llm-course

### DataTalksClub/llm-zoomcamp
Tags: #rag #production #retrieval
Connected: depends on "10 GitHub repos every Data Scientist should bookmark in 2026"

Focus: RAG & Production

What it teaches:
• Practical LLM application engineering.
• Vector databases and semantic search.
• Hybrid search and retrieval-augmented generation.
• Evaluation of RAG systems.
• Containerized deployment and production thinking.

Why it matters for a Data Scientist:
RAG is one of the most common ways Data Scientists turn company knowledge into useful AI applications. The hard part is not calling the model. It is retrieval quality, evaluation, and deployment.

What to focus on first:
1. Embeddings and vector search
2. Chunking and retrieval quality
3. Hybrid search
4. RAG evaluation
5. Deployment

Practical project:
Build a RAG assistant over a small set of technical documents. Create 20 test questions and measure whether retrieval and answers improve as you change chunking and search settings.

Best use: Learning how to move from LLM demo to usable application.

Source:
https://github.com/DataTalksClub/llm-zoomcamp

### microsoft/generative-ai-for-beginners
Tags: #genai #full-stack #agents
Connected: depends on "10 GitHub repos every Data Scientist should bookmark in 2026"

Focus: Full-Stack Generative AI

What it teaches:
• A structured path from basic generative AI concepts to practical application building.
• API integration, structured outputs, function calling, UX patterns, and lightweight agents.
• How model capabilities connect to actual product experiences.

Why it matters for a Data Scientist:
Many Data Scientists know modeling better than application design. This repo helps bridge the gap between model output and a usable AI feature inside a real workflow.

What to focus on first:
1. Model/API interaction
2. Structured outputs
3. Function calling
4. UX patterns for AI
5. Lightweight agent behavior

Practical project:
Build a small assistant that reads a user request, returns structured JSON, calls one tool, and explains the result in plain language.

Best use: Understanding the full application layer around a model.

Source:
https://github.com/microsoft/generative-ai-for-beginners

### Shubhamsaboo/awesome-llm-apps
Tags: #architectures #llm-apps #multimodal
Connected: depends on "10 GitHub repos every Data Scientist should bookmark in 2026"

Focus: Reference Architectures

What it teaches:
• Ready-to-run examples of modern LLM applications.
• RAG patterns, multimodal apps, voice workflows, local-model integrations, and agentic systems.
• How different components are assembled into complete applications.

Why it matters for a Data Scientist:
Sometimes the fastest way to learn is to inspect a working architecture. This repo gives you patterns you can compare, adapt, and critique instead of starting every AI application from a blank page.

What to focus on first:
1. Pick one architecture close to your work
2. Trace the data flow end to end
3. Identify where retrieval, tools, memory, and models sit
4. Understand the evaluation points
5. Rebuild a simplified version

Practical project:
Choose one reference app and recreate only its essential path with your own dataset. Document which components were necessary and which were optional.

Best use: Architecture inspiration and rapid experimentation.

Source:
https://github.com/Shubhamsaboo/awesome-llm-apps

### GokuMohandas/Made-With-ML
Tags: #mlops #production #systems
Connected: depends on "10 GitHub repos every Data Scientist should bookmark in 2026"

Focus: MLOps & Systems

What it teaches:
• End-to-end production machine learning engineering.
• Data pipelines, testing, CI/CD, serving, monitoring, and reproducibility.
• How to structure ML work so it survives beyond a notebook or one-off experiment.

Why it matters for a Data Scientist:
A model has little value if nobody can reliably use, update, monitor, or reproduce it. This repo helps close the gap between experimentation and a maintainable production system.

What to focus on first:
1. Project structure
2. Testing ML code and data
3. Pipelines and reproducibility
4. Deployment patterns
5. Monitoring and iteration

Practical project:
Take one existing notebook project and convert it into a small reproducible ML service with tests, a training pipeline, versioned artifacts, and basic monitoring.

Best use: Learning what happens after the notebook.

Source:
https://github.com/GokuMohandas/Made-With-ML

### vllm-project/vllm
Tags: #serving #inference #llm
Connected: depends on "10 GitHub repos every Data Scientist should bookmark in 2026"

Focus: High-Throughput Serving

What it teaches:
• How modern LLM inference engines serve many requests efficiently.
• PagedAttention, continuous batching, KV-cache management, and throughput optimization.
• The systems trade-offs behind latency, memory use, and concurrency.

Why it matters for a Data Scientist:
Once an LLM application reaches real users, inference cost and speed become part of the product. Understanding serving helps you make better model and architecture choices instead of treating deployment as somebody else's problem.

What to focus on first:
1. Why inference is expensive
2. KV cache intuition
3. Continuous batching
4. Memory vs throughput trade-offs
5. Serving benchmarks

Practical project:
Serve the same small open model with a basic setup and with vLLM. Compare latency and throughput under multiple concurrent requests.

Best use: Understanding the systems layer behind production LLMs.

Source:
https://github.com/vllm-project/vllm

### langchain-ai/langgraph
Tags: #agents #orchestration #langgraph
Connected: depends on "10 GitHub repos every Data Scientist should bookmark in 2026"

Focus: Agent Orchestration

What it teaches:
• How to build stateful, cyclic AI workflows instead of simple one-shot chains.
• Agent state, branching, tool use, persistence, checkpoints, and human-in-the-loop controls.
• How to design systems where an AI can take multiple steps while remaining observable and controllable.

Why it matters for a Data Scientist:
As AI work moves from prediction to action, Data Scientists increasingly need to understand workflow design, state, failure handling, and evaluation across multiple steps.

What to focus on first:
1. State and graph structure
2. Nodes and conditional edges
3. Tool calling
4. Persistence and checkpoints
5. Human approval points

Practical project:
Build a small research agent that searches, summarizes, checks its own result, and pauses for human approval before producing a final recommendation.

Best use: Learning how agent systems are orchestrated beyond a single prompt.

Source:
https://github.com/langchain-ai/langgraph

### dair-ai/Prompt-Engineering-Guide
Tags: #prompting #context #security
Connected: depends on "10 GitHub repos every Data Scientist should bookmark in 2026"

Focus: Context Engineering

What it teaches:
• Prompting patterns from simple zero-shot instructions to more structured reasoning and tool-oriented patterns.
• ReAct-style workflows, grounding, context design, and defensive prompting.
• How prompt quality, context quality, and system constraints interact.

Why it matters for a Data Scientist:
Prompting is not just writing clever instructions. In production AI systems, the bigger skill is designing the right context, examples, constraints, retrieved information, and evaluation around the model.

What to focus on first:
1. Clear task instructions
2. Few-shot examples
3. Grounding with external context
4. ReAct and tool use
5. Prompt injection and defensive design

Practical project:
Take one recurring analytical task and create three versions: a basic prompt, a grounded prompt with reference data, and a tool-using workflow. Compare accuracy and failure modes.

Best use: Moving from prompt writing to context engineering.

Source:
https://github.com/dair-ai/Prompt-Engineering-Guide

### 10 GitHub repos for local LLM inference
Tags: #local-llm #inference #data-science
Priority: ★★★★★
Connected: depends on "Local Models, Serving & Inference", enables "ggerganov/llama.cpp + llama-cpp-python", enables "vllm-project/vllm", enables "sgl-project/sglang", enables "turboderp/exllamav2", enables "ml-explore/mlx-lm", enables "NVIDIA/TensorRT-LLM", enables "dottxt-ai/outlines", enables "ollama/ollama", enables "exo-explore/exo", enables "flashinfer-ai/flashinfer"

A practical map for Data Scientists choosing tools to run LLMs locally or on their own hardware.

When evaluating local inference tools, the most important trade-offs are usually:
• latency: time-to-first-token and decode speed
• memory footprint: quantization and KV-cache efficiency
• Python integration: ease of use in notebooks, scripts, and batch jobs
• output control: structured, deterministic, schema-constrained generation
• hardware fit: CPU, NVIDIA GPU, Apple Silicon, or multi-device setups

Ecosystem map:
CPU & hybrid RAM → llama.cpp, llama-cpp-python, Ollama
NVIDIA GPU engines → vLLM, SGLang, ExLlamaV2, TensorRT-LLM
Apple Silicon / distributed → MLX-LM, Exo
Inference control / kernels → Outlines, FlashInfer

Use this map to choose by workload rather than hype. Each child node explains what the tool is, why it matters for Data Science inference, what to focus on first, and when it is the best fit.

### ggerganov/llama.cpp + llama-cpp-python
Tags: #llama.cpp #gguf #local-inference
Connected: depends on "10 GitHub repos for local LLM inference"

Focus: CPU / hybrid local inference

What it is:
llama.cpp is the foundational C/C++ engine for running GGUF-quantized LLMs across CPUs and consumer GPUs through Metal, CUDA, and ROCm. llama-cpp-python adds Python bindings for direct use from notebooks and scripts.

Why it matters for Data Scientists:
It gives unusually fine control over local inference without requiring a separate web service. You can tune GPU offloading, CPU threads, sampling, grammar-constrained decoding, and memory use while keeping inference in-process.

What to focus on first:
1. GGUF quantization
2. n_gpu_layers and CPU/GPU offloading
3. context size and memory trade-offs
4. sampling controls
5. grammar / structured decoding

Best fit:
When you want maximum portability and control across laptops, desktops, CPUs, Apple Silicon, or mixed hardware.

Practical project:
Run the same model at two quantization levels and compare memory use, latency, and output quality from a Jupyter notebook.

Sources:
https://github.com/ggerganov/llama.cpp
https://github.com/abetlen/llama-cpp-python

### vllm-project/vllm
Tags: #vllm #gpu #batch-inference
Connected: depends on "10 GitHub repos for local LLM inference"

Focus: high-throughput GPU inference

What it is:
A Python/C++ inference engine built around PagedAttention and continuous batching.

Why it matters for Data Scientists:
It is useful not only for serving APIs but also for offline batch inference. The native Python API makes it practical for scoring large datasets without HTTP overhead.

What to focus on first:
1. PagedAttention intuition
2. continuous batching
3. KV-cache efficiency
4. SamplingParams
5. offline batch inference

Best fit:
Large batch workloads on GPUs where throughput and memory efficiency matter.

Practical project:
Score a dataframe with thousands of prompts using the Python API and compare throughput with a naive Transformers loop.

Source:
https://github.com/vllm-project/vllm

### sgl-project/sglang
Tags: #sglang #radixattention #inference
Connected: depends on "10 GitHub repos for local LLM inference"

Focus: fast structured and repeated-prompt inference

What it is:
An inference and programming system designed for complex prompts, multi-turn workflows, structured generation, and efficient reuse of prompt prefixes.

Why it matters for Data Scientists:
RadixAttention can reuse KV states across repeated prefixes, which is valuable for synthetic-data generation, few-shot experiments, evaluation loops, and agent workflows.

What to focus on first:
1. RadixAttention
2. repeated-prefix caching
3. structured decoding
4. multi-turn workflows
5. throughput benchmarking

Best fit:
Workloads with many related prompts, repeated context, or agentic loops.

Practical project:
Run a few-shot classification pipeline across thousands of rows and compare performance with and without prefix reuse.

Source:
https://github.com/sgl-project/sglang

### turboderp/exllamav2
Tags: #exllamav2 #nvidia #quantization
Connected: depends on "10 GitHub repos for local LLM inference"
Source: https://github.com/turboderp-org/exllamav2 (verified 2026-09-10)

Focus: fast single-stream NVIDIA inference

What it is:
A specialized C++/CUDA inference engine tuned for NVIDIA GPUs and EXL2/GPTQ quantized models.

Why it matters for Data Scientists:
It is useful when you care about fast local token generation on consumer GPUs and need aggressive quantization to fit larger models into limited VRAM.

What to focus on first:
1. EXL2 quantization
2. VRAM vs quality trade-offs
3. single-stream decode speed
4. KV-cache quantization
5. long-context memory use

Best fit:
Single-user local inference on NVIDIA cards such as RTX-class GPUs.

Practical project:
Run one model at several EXL2 bitrates and compare VRAM usage, tokens/sec, and answer quality.

Source:
https://github.com/turboderp-org/exllamav2

### ml-explore/mlx-lm
Tags: #mlx-lm #apple-silicon #local-llm
Connected: depends on "10 GitHub repos for local LLM inference"

Focus: Apple Silicon local inference

What it is:
Apple-native LLM tooling built on MLX for M-series chips and unified memory.

Why it matters for Data Scientists:
It makes Macs unusually capable local inference machines because CPU and GPU share the same memory pool, reducing data movement overhead.

What to focus on first:
1. unified-memory intuition
2. 4-bit / 8-bit quantization
3. model loading and generation
4. memory limits by Mac configuration
5. batch vs interactive inference

Best fit:
Local experimentation on Apple Silicon where simplicity and large unified memory matter.

Practical project:
Run the same model in 4-bit and higher precision, then compare memory usage, latency, and quality.

Source:
https://github.com/ml-explore/mlx-lm

### NVIDIA/TensorRT-LLM
Tags: #tensorrt-llm #nvidia #inference
Connected: depends on "10 GitHub repos for local LLM inference"

Focus: maximum NVIDIA inference performance

What it is:
NVIDIA's dedicated toolkit for compiling and optimizing LLM inference on CUDA hardware.

Why it matters for Data Scientists:
It exposes the hardware-level performance layer behind production LLM systems, including fused kernels, low-precision execution, and in-flight batching.

What to focus on first:
1. FP8 / INT4 execution
2. kernel fusion
3. in-flight batching
4. latency vs throughput trade-offs
5. deployment complexity

Best fit:
NVIDIA-heavy environments where squeezing out maximum inference performance matters more than setup simplicity.

Practical project:
Benchmark the same model with a standard PyTorch/Transformers path and TensorRT-LLM, then compare latency and throughput.

Source:
https://github.com/NVIDIA/TensorRT-LLM

### dottxt-ai/outlines
Tags: #outlines #structured-output #json
Connected: depends on "10 GitHub repos for local LLM inference"

Focus: deterministic structured generation

What it is:
A structured text-generation library that constrains model outputs using regex, JSON schemas, and grammars.

Why it matters for Data Scientists:
Many analytical workflows need reliable machine-readable outputs. Outlines constrains generation so outputs can follow a schema instead of depending on retries and prompt wording alone.

What to focus on first:
1. JSON-schema constrained generation
2. regex constraints
3. grammar-based decoding
4. Pydantic integration
5. backend integration with local inference engines

Best fit:
Extraction, classification, synthetic-data generation, or any workflow where malformed output breaks downstream code.

Practical project:
Force a local model to return a validated Pydantic object for 1,000 rows and measure parse failures versus prompt-only JSON generation.

Source:
https://github.com/dottxt-ai/outlines

### ollama/ollama
Tags: #ollama #local-llm #prototyping
Connected: depends on "10 GitHub repos for local LLM inference"

Focus: easiest local model setup

What it is:
A local model runtime and CLI that simplifies pulling models, handling quantization, and using CPU/GPU acceleration without manual conversion or compilation.

Why it matters for Data Scientists:
It is one of the fastest ways to get from zero to a working local model for notebooks, scripts, prototyping, tool calling, and multimodal experiments.

What to focus on first:
1. model pulling and Modelfiles
2. Python SDK usage
3. structured outputs / tool calling
4. local API integration
5. model and memory selection

Best fit:
Quick experiments where setup speed matters more than low-level control.

Practical project:
Build a notebook that sends a dataframe row to a local model through Ollama and returns a structured classification plus explanation.

Source:
https://github.com/ollama/ollama

### exo-explore/exo
Tags: #exo #distributed #local-llm
Connected: depends on "10 GitHub repos for local LLM inference"

Focus: distributed local inference across devices

What it is:
A decentralized inference framework that can split models across multiple heterogeneous devices on a local network.

Why it matters for Data Scientists:
It explores a different way to overcome local memory limits: pool several consumer devices instead of buying a single larger machine.

What to focus on first:
1. model partitioning
2. device discovery
3. network bottlenecks
4. heterogeneous hardware trade-offs
5. fault tolerance / orchestration

Best fit:
Experiments with very large models that do not fit in one machine's memory.

Practical project:
Combine two local devices, run a model that is too large for one of them alone, and measure the latency cost of distribution.

Source:
https://github.com/exo-explore/exo

### flashinfer-ai/flashinfer
Tags: #flashinfer #kernels #gpu
Connected: depends on "10 GitHub repos for local LLM inference"

Focus: low-level GPU inference kernels

What it is:
A high-performance GPU kernel library for LLM serving and generation, including attention kernels used by modern inference backends.

Why it matters for Data Scientists:
You may not need to use it directly, but it helps explain where inference speed actually comes from when working below high-level serving frameworks.

What to focus on first:
1. paged attention
2. grouped-query attention
3. decode kernels
4. KV-cache interaction
5. integration with custom PyTorch loops

Best fit:
Custom inference systems, performance research, or engineers building below the framework layer.

Practical project:
Benchmark a custom attention-heavy generation loop with optimized kernels and compare decode latency.

Source:
https://github.com/flashinfer-ai/flashinfer

### Local Models, Serving & Inference
Tags: #guide
Connected: enables "10 GitHub repos for local LLM inference"

## The main idea

Local inference is a hardware and workload decision: CPU/GPU memory, quantization, throughput, latency, concurrency and output constraints.

### Practical example

**Practice:** benchmark same small model in a laptop runtime and a serving engine; report time to first token, tokens/s, RAM/VRAM and output quality.

### Further reading

**Existing resource:** retained '10 GitHub repos for local LLM inference' collection.

### Check your understanding

- What decision or problem does this solve?
- What simple baseline could you compare against?
- What assumption or failure mode matters most?

### Fine-tuning vs Prompting vs RAG
Tags: #guide

## The main idea

**Choose based on what must change.** Prompting for instructions/format; RAG when factual context lives outside weights and changes; fine-tuning for consistent learned behavior or domain specialization after baselines.

### How to use this

**Compare:** accuracy, freshness, cost, latency, privacy, maintenance.

### Common mistakes

**Pitfall:** fine-tuning is not a reliable mechanism for keeping fast-changing facts current.

### Check your understanding

- What decision or problem does this solve?
- What simple baseline could you compare against?
- What assumption or failure mode matters most?

### Agents, Tool Use & Evaluations
Tags: #guide

## The main idea

Agents combine model decisions with tools and stateful workflows. Start with a deterministic workflow unless branching autonomy adds real value.

### Practical example

**Practice:** research assistant retrieves a document, extracts structured facts, checks evidence and requests human approval before any external action.

### How to use this

**Evaluate:** task success, tool errors, unsafe actions, cost and reproducibility.

### Further reading

**Reference:** https://langchain-ai.github.io/langgraph/

### Check your understanding

- What decision or problem does this solve?
- What simple baseline could you compare against?
- What assumption or failure mode matters most?

### Data Scientist Universe
Tags: #data-science #learning-map #main-hub
Priority: ★★★★★
Connected: enables "01. Mathematics & Statistics"

# Data Scientist Universe

**A connected knowledge map for understanding, practicing, and applying Data Science, from fundamental statistics to modern AI systems.**

Whether you're starting your journey or working as an experienced Data Scientist, this mindspace helps you understand concepts, write code, choose methods, and solve real problems.

## 🎯 What will you find here?

Data Science is more than training machine learning models. It combines mathematics, programming, statistics, experimentation, engineering, and decision-making.

This universe connects nine disciplines:

1. **Mathematics & Statistics:** Probability, distributions, inference, linear algebra, and optimization.
2. **Python, SQL & Data Processing:** Programming, querying, transforming, and managing data.
3. **Data Analysis & Visualization:** Exploratory analysis, patterns, communication, and insights.
4. **Machine Learning:** Regression, classification, clustering, feature engineering, and evaluation.
5. **Deep Learning:** Neural networks, PyTorch, optimization, and model architectures.
6. **Generative AI & Agents:** Transformers, LLMs, RAG, fine-tuning, inference, and agentic systems.
7. **Experimentation & Causal ML:** A/B testing, causal inference, and estimating intervention effects.
8. **MLOps & Production:** Pipelines, deployment, monitoring, testing, and reproducibility.
9. **Real-World Applications:** End-to-end projects, case studies, technical decisions, and practical workflows.

## 🧒 New to Data Science?

Imagine you're trying to understand why some students perform better in exams than others.

You might:

- **Collect data:** Study hours, attendance, and exam scores.
- **Explore:** Find average scores and patterns.
- **Use statistics:** Determine whether observed differences could be due to chance.
- **Build models:** Predict scores using available information.
- **Evaluate:** Test how well those predictions work on unseen students.
- **Make decisions:** Identify where additional support might help, then test whether it actually does.

That's Data Science: moving from questions and data to evidence and better decisions.

## 🧭 How to navigate this universe

Each connected topic helps you progress through four stages:

- **Understand:** Simple intuition, explanations, examples, and mathematical foundations.
- **Implement:** Python or SQL code, worked examples, and technical references.
- **Evaluate:** Assumptions, limitations, alternatives, and common mistakes.
- **Apply:** Practical exercises, real problems, and decision frameworks.

You don't have to read everything in order. Start with the question or problem you're facing.

## 📚 Trusted learning references

Our lessons build on original documentation, academic material, and reproducible examples.

- [Introduction to Statistical Learning](https://www.statlearning.com/) — Statistical learning and machine learning foundations.
- [scikit-learn User Guide](https://scikit-learn.org/stable/user_guide.html) — Classical machine learning methods and implementation.
- [Python Documentation](https://docs.python.org/3/) — Programming language reference.
- [PyTorch Tutorials](https://pytorch.org/tutorials/) — Deep learning implementation.
- [NIST Engineering Statistics Handbook](https://www.itl.nist.gov/div898/handbook/) — Statistical methods and practical applications.

Individual lessons provide more specific sources.

---

**Our guiding principle:** Don't just learn what a method does. Understand why it works, when to use it, how to implement it, and how to recognize when it fails.

### 01. Mathematics & Statistics
Tags: #mathematics #statistics #domain-hub
Priority: ★★★★
Connected: depends on "Data Scientist Universe", enables "Descriptive Statistics & Data Exploration", enables "Probability & Distributions"

## Purpose

Understand the mathematics behind data, uncertainty, predictions, and learning algorithms. The goal is to understand assumptions and reasoning, not merely memorize formulas.

## 🧒 Explain it to a 15-year-old

Imagine coaching a basketball team and recording player scores.

- **Statistics:** What is a typical score?
- **Probability:** How likely is a 20-point game?
- **Linear algebra:** How can we organize and calculate with all player results?
- **Calculus:** How does a prediction change as inputs change?
- **Optimization:** How can a model adjust to make fewer mistakes?

## 🧭 Six learning areas

### 1. Descriptive Statistics & Data Exploration
Mean, median, mode, percentiles, variance, standard deviation, distributions, outliers, covariance, and correlation.

### 2. Probability & Distributions
Conditional probability, independence, Bayes' theorem, expected value, Normal, Binomial, and Poisson distributions.

### 3. Statistical Inference
Populations, samples, sampling bias, confidence intervals, bootstrapping, hypothesis testing, statistical power, and effect size.

### 4. Linear Algebra
Vectors, matrices, dot products, projections, eigenvectors, and singular value decomposition.

### 5. Calculus & Optimization
Derivatives, partial derivatives, gradients, loss functions, and gradient descent.

### 6. Statistical Modeling & Judgment
Likelihood, assumptions, bias-variance trade-offs, regularization, and prediction versus causal explanation.

## 🎯 Learning outcomes

- Summarize and explain variation in data.
- Reason about uncertainty and sampling error.
- Understand the mathematics behind common ML algorithms.
- Recognize assumptions and limitations.
- Defend conclusions with appropriate evidence.

## 📚 Sources

1. [OpenStax: Introductory Statistics](https://openstax.org/books/introductory-statistics-2e/pages/1-introduction)
2. [NIST: Engineering Statistics Handbook](https://www.itl.nist.gov/div898/handbook/)
3. [MIT: Matrix Methods in Data Analysis and ML](https://ocw.mit.edu/courses/18-065-matrix-methods-in-data-analysis-signal-processing-and-machine-learning-spring-2018/)

**Principle:** Understand what numbers mean and when they justify conclusions.

### Descriptive Statistics & Data Exploration
Tags: #statistics #EDA #topic-hub
Priority: ★★★★
Connected: depends on "01. Mathematics & Statistics", enables "Mean, Median, Mode & Weighted Averages", enables "Variance, Standard Deviation & IQR", enables "Distributions, Skewness & Outliers", enables "Covariance, Correlation & Relationships", enables "Data Quality & Exploratory Data Analysis"

# Descriptive Statistics & Data Exploration

**Before building a model, understand your data.**

Descriptive statistics summarizes observed data. Exploratory Data Analysis (EDA) uses summaries and visualizations to investigate patterns, unusual observations, possible errors, and questions worth exploring.

## 🧒 Explain it to a 15-year-old

Imagine recording ten students' journeys to school, in minutes:

`10, 12, 12, 13, 14, 14, 15, 16, 17, 57`

One student takes **57 minutes**, far longer than the rest.

- **Mean:** 18 minutes.
- **Median:** 14 minutes.
- **Range:** 57 − 10 = 47 minutes.
- **Unusual observation:** 57 minutes deserves investigation.

Perhaps that student lives in another town. EDA is more than calculating numbers: it means asking **why** the patterns appear.

## 🧠 Five connected learning areas

### 1. Central tendency: What is typical?

- **Mean:** Sum divided by count.
- **Median:** Middle sorted value.
- **Mode:** Most frequent value.

Extreme observations affect the mean more strongly than the median.

**Next lesson:** Mean, Median, Mode & Weighted Averages.

### 2. Dispersion: How much do values vary?

- **Range:** Maximum minus minimum.
- **Variance:** Average squared deviation (population and sample formulas differ).
- **Standard deviation:** Spread in original measurement units.
- **Interquartile range:** Spread of the middle 50%.

**Next lesson:** Variance, Standard Deviation & IQR.

### 3. Distributions: What shape does the data have?

- Symmetry versus skewness.
- One peak versus multiple peaks.
- Long tails and extreme values.
- Histograms, box plots and density plots.

**Next lesson:** Distributions, Skewness & Outliers.

### 4. Relationships: How do variables move together?

- **Covariance:** Direction of joint variation.
- **Correlation:** Standardized direction and strength of a linear relationship.
- **Scatter plot:** A visual check for patterns and unusual points.

**Warning:** Correlation does not establish causation.

**Next lesson:** Covariance, Correlation & Relationships.

### 5. Data quality: Can we trust the observations?

Check missing values, duplicates, invalid units, inconsistent categories, impossible measurements, and sample composition. An unusual value is not automatically an error.

**Next lesson:** Data Quality & Exploratory Analysis.

## 🐍 Python quick start

```python
import pandas as pd

minutes = pd.Series(
    [10, 12, 12, 13, 14, 14, 15, 16, 17, 57]
)

print(minutes.describe())
print("Median:", minutes.median())
minutes.plot.hist(bins=6)
```

The table summarizes observations; the histogram reveals their shape. **Interpretation** is the next step.

## ⚠️ Common mistakes

1. Reporting only the mean when extreme values distort it.
2. Removing unusual observations without checking their origin.
3. Ignoring subgroup differences hidden by overall summaries.
4. Treating correlation as proof of causation.
5. Trusting statistics calculated from incorrect or biased data.

## 🎯 After these lessons, you should be able to

- Choose meaningful summary measures.
- Describe variation and distribution shape.
- Investigate unusual values.
- Explore associations responsibly.
- Create and interpret basic Python summaries.

## 📚 Sources

1. [OpenStax: Introductory Statistics, Chapter 2](https://openstax.org/books/introductory-statistics/pages/2-introduction)
2. [NIST: Exploratory Data Analysis](https://www.itl.nist.gov/div898/handbook/eda/eda.htm)
3. [pandas: Descriptive Statistics](https://pandas.pydata.org/docs/user_guide/basics.html#descriptive-statistics)

**Key takeaway:** Descriptive statistics summarizes what we observed. EDA helps us investigate what those observations might mean.

### Mean, Median, Mode & Weighted Averages
Tags: #statistics #central-tendency #python
Connected: depends on "Descriptive Statistics & Data Exploration"
Source: https://openstax.org/books/introductory-statistics-2e/pages/2-5-measures-of-the-center-of-the-data (verified 2026-10-12)

## 🎯 What are we measuring?

A **measure of central tendency** summarizes what is typical in a dataset.

- **Mean:** Arithmetic average.
- **Median:** Middle sorted value.
- **Mode:** Most frequent value.
- **Weighted mean:** Average accounting for different contributions.

## 🧒 Explain it to a 15-year-old

Five people earn monthly salaries (hundreds of dollars):

`30, 35, 35, 40, 160`

- **Mean:** Share the total equally: (30 + 35 + 35 + 40 + 160) / 5 = **60**.
- **Median:** Sort and select the middle: `30, 35, [35], 40, 160` → **35**.
- **Mode:** The most common value is **35**.

Four out of five earn 40 or less, yet the mean is 60 because one high salary pulls it upward. **The mean isn't wrong; it answers a different question.**

## 🧠 Mathematical intuition

### Arithmetic mean

`x̄ = (Σ xᵢ) / n`

- `xᵢ`: each value; `n`: number of observations.
- The mean minimizes the **sum of squared deviations** from a chosen center. This connects to least-squares regression.

### Median

- Odd number of observations: select the middle.
- Even number: average the two middle values.

For `2, 4, 6, 10`, median = (4 + 6) / 2 = **5**.

A median minimizes the **sum of absolute deviations**. This connects to absolute-error loss functions.

### Mode

The most frequent observation. For `Red, Blue, Red, Green, Red`, the mode is **Red**. There may be multiple modes. The mode also works for categories.

## ⚖️ Weighted mean

`Weighted mean = (Σ wᵢxᵢ) / (Σ wᵢ)`

| Assessment | Score | Weight |
|---|---:|---:|
| Coursework | 80 | 30% |
| Final exam | 90 | 70% |

Result: (80 × 0.30) + (90 × 0.70) = **87**.

**Group-average trap:** Store A sells 10 items at an average of $20; Store B sells 90 at an average of $40.

- Incorrect unweighted average of store means: **$30**.
- Correct average across all items: (10 × 20 + 90 × 40) / 100 = **$38**.

## 💡 When should you use each?

| Situation | Good starting choice |
|---|---|
| Roughly symmetric numeric data | Mean |
| Strong skew or extreme values | Median |
| Most common category | Mode |
| Grades, group averages, unequal contributions | Weighted mean |

These are guidelines, not rigid rules. The **mean** remains correct when your question is total value divided by count, even for skewed data.

## 🐍 Python example

```python
import numpy as np
import pandas as pd

salaries = pd.Series([30, 35, 35, 40, 160])
print("Mean:", salaries.mean())
print("Median:", salaries.median())
print("Modes:", salaries.mode().tolist())

scores = np.array([80, 90])
weights = np.array([0.30, 0.70])
print("Weighted mean:",
      np.average(scores, weights=weights))
```

**Expected:** mean 60, median 35, modes [35], weighted mean 87.

**Try it:** Change 160 to 300. Which measures change?

## ⚠️ Common mistakes

1. Mistaking the mean for what most individuals experience.
2. Averaging group means without accounting for group sizes.
3. Ignoring systematic missing values.
4. Assuming there is always one unique mode.
5. Reporting the center without a measure of spread.
6. Averaging numeric codes for categories without meaningful numeric interpretation.

## 🔗 Further connections

- **Variance, Standard Deviation & IQR:** how widely values vary.
- **Distributions, Skewness & Outliers:** why mean and median differ.
- **Statistical Inference:** estimating population centers using samples.
- **Linear Regression:** squared error and the mean.
- **Model Evaluation:** MAE versus MSE.

## 📚 Sources

1. [OpenStax: Measures of the Center](https://openstax.org/books/introductory-statistics-2e/pages/2-5-measures-of-the-center-of-the-data)
2. [NIST: Measures of Location](https://itl.nist.gov/div898/handbook/eda/section3/eda351.htm)
3. [NumPy: Weighted Average](https://numpy.org/doc/stable/reference/generated/numpy.average.html)

**Takeaway:** Choose the average that answers your question, not simply the one most familiar to you.

### Variance, Standard Deviation & IQR
Tags: #statistics #dispersion #python
Connected: depends on "Descriptive Statistics & Data Exploration"
Source: https://openstax.org/books/introductory-statistics-2e/pages/2-7-measures-of-the-spread-of-the-data (verified 2026-10-12)

# Variance, Standard Deviation & IQR

## 🎯 What are we measuring?

An average describes the center, but not **how far observations spread around it**. This is variability or dispersion.

- **Variance:** Average squared distance from the mean.
- **Standard deviation (SD):** Square root of variance, in the original units.
- **Interquartile range (IQR):** Spread of the middle 50%.

## 🧒 Explain it to a 15-year-old

Two basketball players score points over five games:

| Game | Player A | Player B |
|---|---:|---:|
| 1 | 8 | 2 |
| 2 | 9 | 6 |
| 3 | 10 | 10 |
| 4 | 11 | 14 |
| 5 | 12 | 18 |
| **Average** | **10** | **10** |

Both average 10 points. Player A is more consistent, while Player B varies widely. Averages alone cannot show this difference.

## 🧠 How variance works

Using Player A: `8, 9, 10, 11, 12`.

### Step 1: Calculate the mean

(8 + 9 + 10 + 11 + 12) / 5 = **10**.

### Step 2: Square each deviation

| Score | Score − 10 | Squared deviation |
|---|---:|---:|
| 8 | -2 | 4 |
| 9 | -1 | 1 |
| 10 | 0 | 0 |
| 11 | 1 | 1 |
| 12 | 2 | 4 |

Squaring prevents positive and negative differences from canceling, and gives larger deviations more weight.

### Step 3: Average those squares

Population variance: `σ² = Σ(xᵢ − μ)² / N`

- `xᵢ`: each observation
- `μ`: population mean
- `N`: population count

For Player A, variance = (4 + 1 + 0 + 1 + 4) / 5 = **2 points²**.

## 📏 Standard deviation

`σ = √(σ²)`

SD transforms variance back into the original units.

- **Player A:** √2 ≈ **1.41 points**.
- **Player B:** variance 32; SD √32 ≈ **5.66 points**.

| Measure | A | B |
|---|---:|---:|
| Mean | 10 | 10 |
| Population variance | 2 | 32 |
| Population SD | 1.41 | 5.66 |

SD measures spread around the mean, **not literally mean absolute distance**.

## 🔢 Sample vs population variance

When the five games are a sample rather than the complete population:

`s² = Σ(xᵢ − x̄)² / (n − 1)`

Dividing by `n − 1` accounts for estimating the mean from that sample and produces an unbiased estimator of population variance under standard random-sampling assumptions.

For Player A:

- Population variance: 10/5 = **2**.
- Sample variance: 10/4 = **2.5**.
- Sample SD: √2.5 ≈ **1.58**.

**Subtlety:** Unbiased sample variance does not imply unbiased sample SD.

## 📦 Interquartile range

`IQR = Q3 − Q1`

- `Q1`: 25th percentile.
- `Q3`: 75th percentile.

For `8, 9, 10, 11, 12`, using linear-interpolation quartiles:

Q1 = 9, Q3 = 11, so **IQR = 2**.

Change the last score from 12 to 100. SD increases sharply, but IQR remains 2 with this quartile method.

**Why?** IQR focuses on the central half of observations and is less sensitive to extremes. Quartile conventions may differ on small datasets.

## ⚖️ When to use each

| Situation | Useful measure |
|---|---|
| Roughly symmetric data | SD |
| Skewed data or outliers | IQR |
| Squared-error mathematics | Variance |
| Consistency comparisons | Mean + SD |
| Income distributions | Median + IQR |

Always examine the distribution, not only summary numbers.

## 🐍 Python implementation

```python
import numpy as np
import pandas as pd

a = pd.Series([8, 9, 10, 11, 12])
b = pd.Series([2, 6, 10, 14, 18])

def describe_spread(x):
    return {
        "mean": x.mean(),
        "population_var": x.var(ddof=0),
        "sample_var": x.var(ddof=1),
        "population_sd": x.std(ddof=0),
        "sample_sd": x.std(ddof=1),
        "iqr": x.quantile(.75) - x.quantile(.25)
    }

print(describe_spread(a))
print(describe_spread(b))
```

**Try:** Replace Player A's last score with 100. Compare SD with IQR.

pandas defaults to `ddof=1` for SD and variance; NumPy defaults to `ddof=0`.

## ⚠️ Common mistakes

1. Confusing squared units of variance with the original units of SD.
2. Forgetting the difference between `n` and `n − 1`.
3. Treating SD as robust to extreme observations.
4. Assuming mean ± SD always encloses a fixed percentage of values.
5. Comparing SD across unrelated measurement scales.
6. Reporting only IQR and hiding tail behavior.

## 🔗 Connected learning

Mean, Median, Mode & Weighted Averages; Distributions, Skewness & Outliers; Confidence Intervals & Standard Error; Z-Scores & Standardization; Regression Loss Functions.

## 📚 Sources

1. [OpenStax: Measures of Spread](https://openstax.org/books/introductory-statistics-2e/pages/2-7-measures-of-the-spread-of-the-data)
2. [NIST: Exploratory Data Analysis](https://www.itl.nist.gov/div898/handbook/eda/eda.htm)
3. [pandas: Descriptive Statistics](https://pandas.pydata.org/docs/user_guide/basics.html#descriptive-statistics)

**Key takeaway:** The mean tells us where the center is. Variance, SD and IQR tell us how much observations differ.

### Distributions, Skewness & Outliers
Tags: #EDA #skewness #outliers
Connected: depends on "Descriptive Statistics & Data Exploration"
Source: https://www.itl.nist.gov/div898/handbook/eda/eda.htm (verified 2026-10-19)

# Distributions, Skewness & Outliers

**Averages tell us the center. Spread tells us variability. Distribution shape reveals what both may hide.**

## 🎯 The key idea

A distribution shows which values occur and how often. Investigate **symmetry, tails, peaks, and unusual observations**.

This lesson addresses empirical *shape*. Theoretical families such as Normal, Binomial, Poisson, and Exponential distributions deserve their own **Probability Distributions** topic.

## 🧒 Explain it to a 15-year-old

Ten deliveries took (minutes):

`10, 11, 12, 12, 13, 14, 15, 16, 17, 60`

- **Mean:** 18 minutes.
- **Median:** 13.5 minutes.
- Nine deliveries finished within 17 minutes.
- One took **60 minutes**.

The average is pulled upward by one long delay. Why: traffic, address error, or genuine longer journey? **Explore the pattern before making a decision.**

## 📊 Four shapes

### Symmetric
Values roughly balance on both sides of the center. With perfect symmetry and a finite mean, mean and median coincide.

### Right-skewed (positive)
Long tail extends toward **high values**. Typical examples: incomes or delays. Mean is often above median, but not always.

### Left-skewed (negative)
Long tail extends toward **low values**. Example: mostly high scores on an easy exam.

### Multimodal
Several peaks. Could indicate mixed populations, e.g. short walking commutes and longer driving commutes. Peaks alone do not prove subgroups.

## 🧠 Measure skewness

Population skewness:

`γ₁ = E[(X − μ)³] / σ³`

- `X` is a random variable.
- `μ` is its population mean.
- `σ` is its population standard deviation.
- `E` denotes expectation.

Cubing emphasizes large deviations while preserving positive/negative direction.

| Value | Meaning |
|---|---|
| Positive | Greater influence from high-value tail |
| Negative | Greater influence from low-value tail |
| Near zero | Low measured asymmetry; not proof of symmetry |

Small-sample skewness can be unstable, and estimators differ.

## 🔎 Detect potential outliers using IQR

`IQR = Q3 − Q1`

- Lower fence: `Q1 − 1.5 × IQR`
- Upper fence: `Q3 + 1.5 × IQR`

For the **same ten deliveries** (linear-interpolation quartiles):

- Q1 = 12; Q3 = 15.75; IQR = **3.75**.
- Lower fence = **6.375**.
- Upper fence = **21.375**.
- The 60-minute delivery is flagged.

**Flagged does not mean wrong.** Investigate before excluding data. Quartile conventions may vary.

## ⚖️ How should we respond?

| Finding | Possible response |
|---|---|
| Confirmed data entry error | Correct or exclude with documentation |
| Genuine extreme event | Retain and investigate |
| Strong skew | Report median/IQR alongside other summaries |
| Multiple peaks | Examine meaningful segments |
| Skew affects modeling assumptions | Consider justified transformations |

## 🐍 Python implementation

```python
import pandas as pd
import matplotlib.pyplot as plt

x = pd.Series(
    [10, 11, 12, 12, 13, 14, 15, 16, 17, 60]
)
print("Mean:", x.mean())
print("Median:", x.median())
print("Skewness:", x.skew())

q1, q3 = x.quantile([0.25, 0.75])
iqr = q3 - q1
lower, upper = q1 - 1.5*iqr, q3 + 1.5*iqr
print("Flagged:", x[(x < lower) | (x > upper)].tolist())

x.plot.hist(bins=8)
plt.xlabel("Delivery time (minutes)")
plt.show()
```

**Try it:** Change 60 to 20. Compare skewness, mean, median, and outlier flags.

## ⚠️ Common mistakes

1. Assuming all observations follow a Normal distribution.
2. Deleting every value beyond an IQR fence.
3. Believing positive skew always implies mean > median.
4. Treating multimodality as proof of separate groups.
5. Trusting one histogram bin width without checking others.
6. Transforming variables without assessing interpretation.

## 🔗 Connected concepts

- Mean, Median, Mode & Weighted Averages
- Variance, Standard Deviation & IQR
- Data Quality & Exploratory Analysis
- **Probability Distributions** (future separate topic)

## 📚 Sources

1. [NIST: Histograms](https://www.itl.nist.gov/div898/handbook/eda/section3/histogra.htm)
2. [NIST: Exploratory Data Analysis](https://www.itl.nist.gov/div898/handbook/eda/eda.htm)
3. [pandas: Skewness](https://pandas.pydata.org/docs/reference/api/pandas.Series.skew.html)
4. [pandas: Quantiles](https://pandas.pydata.org/docs/reference/api/pandas.Series.quantile.html)

**Takeaway:** Understand the shape before choosing how to summarize, transform, or model your data.

### Covariance, Correlation & Relationships
Tags: #statistics #correlation #python
Connected: depends on "Descriptive Statistics & Data Exploration"
Source: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html (verified 2026-10-19)

# Covariance, Correlation & Relationships

## 🎯 What are we measuring?

- **Covariance:** Do two variables tend to vary in the same or opposite directions?
- **Correlation:** How strong is their standardized association?
- **Scatter plot:** What shape does their relationship actually have?

Used in exploratory analysis, regression and feature investigation.

## 🧒 Explain it to a 15-year-old

Imagine five students:

| Student | Study hours | Score |
|---|---:|---:|
| A | 1 | 2 |
| B | 2 | 4 |
| C | 3 | 5 |
| D | 4 | 4 |
| E | 5 | 5 |

Generally, more hours go with higher scores, although not perfectly.

Think of two people walking: moving together suggests a positive relationship; moving in opposite directions suggests a negative relationship. **No straight-line trend** does not mean no relationship of any kind.

## 🧠 Covariance step by step

Covariance checks whether observations tend to be above or below their own variable's mean together.

### Step 1: Calculate averages

- Average study hours = **3**
- Average exam score = **4**

### Step 2: Multiply deviations

| Hours − 3 | Score − 4 | Product |
|---:|---:|---:|
| -2 | -2 | 4 |
| -1 | 0 | 0 |
| 0 | 1 | 0 |
| 1 | 0 | 0 |
| 2 | 1 | 2 |

Sum of products = **6**.

### Step 3: Sample covariance

`s_xy = Σ(xᵢ − x̄)(yᵢ − ȳ) / (n − 1)`

With five observations: `6 / 4 = 1.5`.

- **Positive:** often above/below their means together.
- **Negative:** often on opposite sides of their means.
- **Zero:** no covariance-detected *linear* association.

**Limitation:** Covariance depends on units and scale. A value of 100 is not inherently a stronger relationship than 2.

## 📏 Pearson correlation

Pearson's `r` standardizes covariance:

`r = s_xy / (s_x × s_y)`

For our five students:

- Covariance = **1.5**
- Sample SD of hours ≈ **1.581**
- Sample SD of scores ≈ **1.225**
- Correlation ≈ **0.775**

| r | Meaning |
|---|---|
| +1 | Perfect positive linear relation |
| Near +1 | Strong positive linear association |
| 0 | No linear correlation |
| Near -1 | Strong negative linear association |
| -1 | Perfect negative linear relation |

Correlation has **no units** and lies between **-1 and +1** when defined. Strength labels depend on context.

## 🔄 Pearson vs Spearman

### Pearson
Measures **linear** association on original numerical values. Sensitive to extreme observations.

### Spearman
Measures **monotonic** association using ranks. Monotonic means consistently increasing or decreasing, not necessarily following a straight line.

Example:

| X | Y |
|---:|---:|
| 1 | 1 |
| 2 | 4 |
| 3 | 9 |
| 4 | 16 |

Spearman = **1** because rankings match perfectly. Pearson is positive but below 1 because the pattern is curved.

| Situation | Start with |
|---|---|
| Straight-line pattern | Pearson |
| Increasing/decreasing curved pattern | Spearman |
| Ordinal rankings | Spearman |
| Non-monotonic pattern | Scatter plot and more suitable models |

## 🐍 Python implementation

```python
import pandas as pd
import matplotlib.pyplot as plt
from scipy.stats import pearsonr, spearmanr

df = pd.DataFrame({
    "hours": [1, 2, 3, 4, 5],
    "score": [2, 4, 5, 4, 5]
})

print("Covariance:",
      df["hours"].cov(df["score"]))
print("Pearson:",
      pearsonr(df["hours"], df["score"]).statistic)
print("Spearman:",
      spearmanr(df["hours"], df["score"]).statistic)

plt.scatter(df["hours"], df["score"])
plt.xlabel("Study hours")
plt.ylabel("Exam score")
plt.show()
```

**Expected:** covariance = 1.5; Pearson ≈ 0.775.

**Try:** Change one score to an extreme value and observe both correlations.

## ⚠️ Important limitations

1. **Correlation does not prove causation.** Hot weather may increase both ice-cream sales and swimming.
2. **Zero Pearson correlation does not mean independence.** Relationships can be curved.
3. **Outliers can distort Pearson correlation.**
4. **Confounders** can create or hide associations.
5. Relationships may differ dramatically between subgroups.
6. A correlation from just five observations has high sampling uncertainty.

## 🔗 Connected concepts

- Mean, Median, Mode & Weighted Averages
- Variance, Standard Deviation & IQR
- Scatter Plots & Exploratory Analysis
- Linear Regression
- Multicollinearity & Feature Selection
- Causal Inference & Confounding

## 📚 Sources

1. [SciPy: Pearson correlation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.pearsonr.html)
2. [SciPy: Spearman correlation](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.spearmanr.html)
3. [NIST: Engineering Statistics Handbook](https://www.itl.nist.gov/div898/handbook/)

**Takeaway:** Covariance describes directional co-variation; correlation standardizes association. Neither establishes causation.

### Data Quality & Exploratory Data Analysis
Tags: #EDA #data-quality #python
Connected: depends on "Descriptive Statistics & Data Exploration"
Source: https://www.itl.nist.gov/div898/handbook/eda/eda.htm (verified 2026-10-19)

# Data Quality & Exploratory Data Analysis

**Good data science begins by questioning data, not trusting it.** Data quality checks whether observations are fit for purpose. Exploratory Data Analysis (EDA) uses statistics and visuals to investigate patterns, assumptions and anomalies.

## 🧒 Explain it to a 15-year-old

Your school measures journeys to school:

| Student | Minutes | Transport |
|---|---:|---|
| A | 15 | Bus |
| B | 20 | Walk |
| C | 18 | Bus |
| C | 18 | Bus |
| D | Missing | Car |
| E | -10 | Walk |
| F | 180 | Bus |

What jumps out?

- **Duplicate:** C appears twice.
- **Missing:** D has no duration.
- **Impossible:** -10 minutes.
- **Unusual:** 180 minutes. Perhaps a major delay, not an error!

Don't calculate a confident average before investigating the records.

## 🧠 Six dimensions of data quality

| Dimension | Key question |
|---|---|
| Completeness | Are essential values missing? |
| Validity | Do values obey the defined rules? |
| Consistency | Do systems or columns disagree? |
| Uniqueness | Are there unintended duplicates? |
| Accuracy | Do the values reflect reality? |
| Timeliness | Is the information recent enough? |

A value of 180 minutes can be valid (positive duration) but not necessarily accurate. **Clean data is not automatically representative data.**

## 🔍 A practical EDA workflow

### 1. Understand the dataset

- What does **one row** represent (the grain)?
- What do the fields, units and codes mean?
- Who collected the data, how, and when?
- What decision will the analysis support?

### 2. Profile quality

Inspect data types, missingness, duplicates, invalid values, categories, and unexpected joins.

### 3. Explore each variable

- **Numeric:** mean, median, SD, IQR, histogram, box plot.
- **Categorical:** counts and bar chart.
- **Temporal:** trends, seasonality, gaps.

### 4. Explore relationships

Use scatter plots, grouped summaries and correlations. Bus trips may take longer because of journey distances, not necessarily the transport type.

### 5. Investigate anomalies

Distinguish actual events from measurement failures. Avoid arbitrary removal.

### 6. Document decisions

Record original values, cleaning rules, effects on results, uncertainties and open questions.

## 🐍 Python: an initial audit

```python
import pandas as pd
import numpy as np

df = pd.DataFrame({
    "student": ["A","B","C","C","D","E","F"],
    "minutes": [15,20,18,18,np.nan,-10,180],
    "transport": [
        "Bus","Walk","Bus","Bus","Car","Walk","Bus"
    ]
})

print(df.info())
print(df.describe(include="all"))
print("Missing:", df.isna().sum())
print("Exact duplicates:", df.duplicated().sum())
print("Negative durations:", (df.minutes < 0).sum())
print("Investigate:", df.loc[df.minutes > 120])

clean = df.drop_duplicates().copy()
clean.loc[clean.minutes < 0, "minutes"] = np.nan

print(clean.groupby("transport")["minutes"]
      .agg(["count", "mean", "median"]))
```

**Why this cleaning?** In our example, exact duplicates are redundant, and negative durations violate the definition of elapsed time. The 180-minute journey remains pending investigation. Real projects need domain-specific, documented rules.

## ⚠️ Frequent mistakes

1. Deleting all missing rows without investigating why values are absent.
2. Automatically excluding extremes that may represent real events.
3. Forgetting data grain and multiplying rows through joins.
4. Treating valid records as necessarily accurate.
5. Preparing model features with future data or test-set information (leakage).
6. Assuming an association proves causation.

## 🎯 A good EDA deliverable

Produce **findings, identified quality problems, decisions made, remaining uncertainty, and recommended next investigation**. A notebook full of charts alone is not enough.

## 🔗 Further lessons

- Mean, Median, Mode & Weighted Averages
- Variance, Standard Deviation & IQR
- Distributions, Skewness & Outliers
- Covariance, Correlation & Relationships
- Missing Data Mechanisms & Imputation
- Data Leakage & Validation

## 📚 Sources

1. [NIST: Exploratory Data Analysis](https://www.itl.nist.gov/div898/handbook/eda/eda.htm)
2. [pandas: Working with Missing Data](https://pandas.pydata.org/docs/user_guide/missing_data.html)
3. [pandas: Merge Validation](https://pandas.pydata.org/docs/user_guide/merging.html)

**Takeaway:** Investigate first, clean with purpose, and document what you learn.

### Probability & Distributions
Tags: #probability #distributions #topic-hub
Priority: ★★★★
Connected: depends on "01. Mathematics & Statistics", enables "Probability Fundamentals", enables "Conditional Probability & Bayes' Theorem"
Source: https://openstax.org/books/introductory-statistics-2e/pages/4-introduction (verified 2026-10-19)

# Probability & Distributions

**Probability is the mathematical language of uncertainty.** Probability distributions describe possible outcomes and their likelihoods. These concepts support inference, risk analysis, predictive modeling and machine learning.

## 🧒 Explain it to a 15-year-old

A basketball player makes 70% of free throws. What is the chance of scoring the next shot, all three next shots, or exactly seven out of ten?

- **Probability** measures the chance of an event.
- **A probability distribution** assigns probabilities to possible outcomes, such as scoring 0, 1, …, 10 shots.

If successive shots are independent and success probability stays 0.7:

`P(three successes) = 0.7³ = 0.343 = 34.3%`

Assumptions matter. Fatigue or pressure might change shot probabilities.

## 🧭 Six learning areas

### 1. Probability Fundamentals

**Question:** How likely is an event?

- Sample spaces and events
- Unions, intersections and complements
- Independent versus mutually exclusive events
- Addition and multiplication rules

**Application:** Probability of one or more equipment failures.

### 2. Conditional Probability & Bayes' Theorem

**Question:** How does new information change our beliefs?

- Conditional, joint and marginal probabilities
- Base rates and Bayes' theorem
- Independence versus conditional independence

**Application:** Probability of machine failure after an anomaly alert.

### 3. Random Variables & Expected Value

**Question:** How can uncertain outcomes be numerical?

- Discrete and continuous random variables
- Probability mass function (PMF)
- Probability density function (PDF)
- Cumulative distribution function (CDF)
- Expected value and variance

**Application:** Expected cost from operational failures.

### 4. Discrete Probability Distributions

**Question:** How do we model countable outcomes?

- **Bernoulli:** One success/failure trial.
- **Binomial:** Number of successes over fixed independent trials with the same probability.
- **Poisson:** Event counts in an interval under the assumptions of a constant-rate Poisson process.
- **Geometric:** Number of trials until the first success.

**Application:** Defects, incidents and arrivals.

### 5. Continuous Probability Distributions

**Question:** How do we model continuous measurements and waiting times?

- **Uniform:** Constant density on a bounded interval.
- **Normal:** Symmetric bell-shaped model.
- **Exponential:** Waiting times in a constant-rate Poisson process.
- **Lognormal:** Positive quantity whose logarithm is normal.
- **Gamma and Weibull:** Flexible positive-valued and lifetime models.

**Application:** Measurement errors, service time and equipment lifetime.

### 6. Choosing & Validating Distributions

**Question:** Which probabilistic model fits our problem and evidence?

- Decide whether outcomes are discrete or continuous.
- Check support, shape and assumptions.
- Estimate parameters.
- Compare observed and theoretical patterns with histograms and Q-Q plots.
- Check goodness of fit, tail behavior and overdispersion.

**Application:** Determine whether Poisson adequately models incident counts.

## 🔗 Learning path

**Probability rules → Conditional probability → Random variables → Probability distributions → Statistical inference**

Each major distribution family will receive its own detailed lesson with teenage-level intuition, equations, Python, assumptions and applications.

## 🎯 Learning outcomes

- Calculate probabilities of events and combinations.
- Interpret conditional probability and Bayesian updates.
- Distinguish discrete from continuous quantities.
- Choose and check candidate distribution models.
- Calculate expected outcomes in Python.
- Explain what assumptions make answers trustworthy.

## 📚 Sources

1. [OpenStax: Introductory Statistics](https://openstax.org/books/introductory-statistics-2e/pages/4-introduction)
2. [OpenStax: Discrete and Continuous Probability Distributions](https://openstax.org/books/principles-data-science/pages/3-5-discrete-and-continuous-probability-distributions)
3. [NIST: Engineering Statistics Handbook](https://www.itl.nist.gov/div898/handbook/)
4. [SciPy: Statistical Functions](https://docs.scipy.org/doc/scipy/reference/stats.html)

**Takeaway:** Probability describes uncertainty about events; distributions describe how probability is allocated across possible outcomes.

### Probability Fundamentals
Tags: #probability #foundations #python
Connected: depends on "Probability & Distributions"
Source: https://openstax.org/books/introductory-statistics-2e/pages/3-introduction (verified 2026-10-19)

# Probability Fundamentals

## 🎯 What is probability?

**Probability measures how likely an event is**, from 0 (impossible) to 1 (certain). It makes uncertainty measurable.

## 🧒 Explain it to a 15-year-old

Roll a fair six-sided die.

- Sample space: `{1,2,3,4,5,6}`
- Chance of a 6: `1/6 ≈ 16.7%`
- Chance of an even number `{2,4,6}`: `3/6 = 50%`

Count favorable outcomes over all outcomes **only when outcomes are equally likely**.

## 🧠 Core concepts

### Sample space and events

The **sample space** (Ω) is every possible outcome. An **event** is a subset: `A={2,4,6}` for an even die roll.

For equally likely outcomes:

`P(A) = favorable outcomes / all outcomes`

For unequal probabilities use a probability model or carefully estimated frequencies, not simple counting.

### 🔄 Complement: NOT A

`P(not A) = 1 − P(A)`

If a machine has 5% failure probability, nonfailure probability is **95%**.

Complements are especially helpful for finding the chance of *at least one* event.

### ➕ Addition: A OR B

One die roll:

- A = even `{2,4,6}`
- B = greater than 4 `{5,6}`
- A or B = `{2,4,5,6}`: **4/6 ≈ 66.7%**

`P(A ∪ B) = P(A) + P(B) − P(A ∩ B)`

Subtract the overlap: outcome 6 belongs to both. If overlap is impossible, the events are **mutually exclusive**.

### ✖️ Multiplication: A AND B

Two **independent** die rolls:

`P(two sixes) = (1/6) × (1/6) = 1/36 ≈ 2.78%`

For dependent events, the general rule is:

`P(A ∩ B) = P(A) × P(B | A)`

`P(B | A)` means probability of B **given** A has occurred.

### ⚖️ Independent vs mutually exclusive

| Independent | Mutually exclusive |
|---|---|
| Knowing A does not change P(B) | A and B cannot occur together |
| Example: separate fair die rolls | Example: rolling both a 2 and 6 on one die |

Two mutually exclusive events with positive probabilities cannot be independent.

## 💡 Real-world application

Three independent machines each have a 10% probability of failing during a shift.

Instead of adding three 10% risks, calculate the complement:

- All three working: `0.9 × 0.9 × 0.9 = 0.729`
- At least one failure: `1 − 0.729 = 0.271`

**Answer: 27.1%.**

**Assumption:** failures are independent. Shared power problems can violate that assumption.

## 🐍 Python: calculate and simulate

```python
import numpy as np

p_failure = 0.10
n_machines = 3
analytical = 1 - (1 - p_failure)**n_machines
print("Exact model probability:", analytical)

rng = np.random.default_rng(42)
sim = rng.random((100000, n_machines)) < p_failure
estimate = sim.any(axis=1).mean()
print("Simulation estimate:", estimate)
```

The analytical result is **0.271**. The simulation fluctuates around it.

**Try:** Set `n_machines = 10`. What changes and why?

## ⚠️ Common mistakes

1. Adding overlapping probabilities without subtracting the overlap.
2. Multiplying as though events are independent without checking.
3. Confusing independent with mutually exclusive.
4. Assuming all outcomes are equally likely.
5. Treating an event's probability as a guarantee.
6. Believing previous independent random outcomes force future compensation.

## 🔗 Further learning

- Conditional Probability & Bayes' Theorem
- Random Variables & Expected Value
- Bernoulli and Binomial Distributions
- Statistical Independence

## 📚 Sources

1. [OpenStax: Probability](https://openstax.org/books/introductory-statistics-2e/pages/3-introduction)
2. [OpenStax: Two Basic Rules](https://openstax.org/books/introductory-statistics-2e/pages/3-3-two-basic-rules-of-probability)
3. [NumPy: Random Generator](https://numpy.org/doc/stable/reference/random/generator.html)

**Key takeaway:** To calculate combined-event probabilities correctly, pay attention to complements, overlap and dependence.

### Conditional Probability & Bayes' Theorem
Tags: #bayes #probability #inference
Connected: depends on "Probability & Distributions"
Source: https://openstax.org/books/principles-data-science/pages/3-4-probability-theory (verified 2026-10-21)

# Conditional Probability & Bayes' Theorem

**Conditional probability measures an event's chance given evidence. Bayes' theorem updates beliefs using that evidence and the underlying base rate.**

## 🎯 Two different questions

- `P(Alarm | Fault)`: How likely is an alarm **if a fault exists**?
- `P(Fault | Alarm)`: How likely is a fault **if an alarm sounds**?

These are **not interchangeable**.

## 🧒 Explain it to a 15-year-old

Imagine 1,000 machines. Only 5% are faulty. An alarm detects 90% of faults but falsely alerts for 10% of healthy machines.

| | Faulty | Healthy |
|---|---:|---:|
| Alarm | 45 | 95 |
| No alarm | 5 | 855 |
| Total | 50 | 950 |

There are **140 alarms**, but only **45 real faults**.

`P(Fault | Alarm) = 45 / 140 ≈ 32.1%`

Although the detector catches 90% of faults, most alarms are false because faults are rare. This is the **base-rate effect**.

## 🧠 Conditional probability

`P(A | B) = P(A ∩ B) / P(B)`, where `P(B) > 0`.

- **A:** event we care about.
- **B:** known evidence.
- **A ∩ B:** both events happen.

Knowing B changes the relevant group. Here we look among the **140 alarms**, not all 1,000 machines.

## 🔄 Bayes' theorem

`P(A | B) = P(B | A) × P(A) / P(B)`

For a binary outcome:

`P(A | B) = [P(B | A)P(A)] / [P(B | A)P(A) + P(B | not A)P(not A)]`

| Term | Meaning |
|---|---|
| Prior `P(A)` | Chance before new evidence |
| Likelihood `P(B | A)` | Chance of evidence if A is true |
| Evidence `P(B)` | Overall chance of observing B |
| Posterior `P(A | B)` | Updated chance after evidence |

### Calculate the machine example

- Prior fault rate: `P(Fault) = 0.05`
- Detection rate: `P(Alarm | Fault) = 0.90`
- False-alarm rate: `P(Alarm | Healthy) = 0.10`

`P(Fault | Alarm) = (0.90 × 0.05) / [(0.90 × 0.05) + (0.10 × 0.95)]`

**Result: 0.3214 = 32.14%.**

## ⚖️ Why the base rate matters

Keep the *same* detector but imagine **50% of machines are faulty**:

`P(Fault | Alarm) = (0.9 × 0.5) / [(0.9 × 0.5) + (0.1 × 0.5)] = 0.90`

Now an alarm means a **90%** fault probability!

A detector's positive predictive value changes when the condition's prevalence changes.

## 💡 Data Science applications

- **Fraud:** Chance of fraud given suspicious behavior.
- **Maintenance:** Chance of a breakdown after a sensor alert.
- **Medical screening:** Chance of disease after a positive test.
- **Spam filtering:** Chance of spam given message content.
- **Bayesian models:** Update uncertainty about parameters using observed data.

## 🐍 Python implementation

```python
p_fault = 0.05
detection_rate = 0.90
false_alarm_rate = 0.10

p_alarm = (
    detection_rate * p_fault
    + false_alarm_rate * (1 - p_fault)
)
posterior = detection_rate * p_fault / p_alarm
print(f"P(Fault | Alarm): {posterior:.2%}")
# 32.14%
```

**Try:** Change the fault prevalence to 1%. Does the alarm become more or less trustworthy?

## ⚠️ Common mistakes

1. Confusing `P(A | B)` with `P(B | A)`.
2. Ignoring the prevalence or **base rate**.
3. Assuming high sensitivity means most positive alerts are true.
4. Confusing **sensitivity** with **precision**.
5. Assuming prevalence never changes between populations.
6. Combining evidence as if conditionally independent without checking.

## 🔗 Further reading paths

- Probability Fundamentals
- Random Variables & Expected Value
- Naive Bayes Classification
- Bayesian Statistical Inference
- Precision, Recall & Confusion Matrix

These are conceptual references, **not sibling graph links**.

## 📚 Sources

1. [OpenStax: Probability Theory](https://openstax.org/books/principles-data-science/pages/3-4-probability-theory)
2. [OpenStax: Rules of Probability](https://openstax.org/books/introductory-statistics-2e/pages/3-3-two-basic-rules-of-probability)
3. [Statistics LibreTexts: Base Rates](https://stats.libretexts.org/Bookshelves/Introductory_Statistics/Introductory_Statistics_%28Lane%29/05%3A_Probability/5.12%3A_Base_Rates)

**Takeaway:** The chance of evidence given a condition differs from the chance of the condition given evidence. Bayes' theorem connects them through the base rate.

## Connections
- **10 GitHub repos every Data Scientist should bookmark in 2026** → **rasbt/LLMs-from-scratch** — "foundation: understand how LLMs work"
- **10 GitHub repos every Data Scientist should bookmark in 2026** → **karpathy/nn-zero-to-hero** — "foundation: rebuild deep-learning intuition"
- **10 GitHub repos every Data Scientist should bookmark in 2026** → **mlabonne/llm-course** — "model specialization and fine-tuning"
- **10 GitHub repos every Data Scientist should bookmark in 2026** → **DataTalksClub/llm-zoomcamp** — "application layer: RAG and production"
- **10 GitHub repos every Data Scientist should bookmark in 2026** → **microsoft/generative-ai-for-beginners** — "application layer: full-stack generative AI"
- **10 GitHub repos every Data Scientist should bookmark in 2026** → **Shubhamsaboo/awesome-llm-apps** — "application layer: reference architectures"
- **10 GitHub repos every Data Scientist should bookmark in 2026** → **GokuMohandas/Made-With-ML** — "production layer: MLOps and systems"
- **10 GitHub repos every Data Scientist should bookmark in 2026** → **vllm-project/vllm** — "production layer: LLM serving and inference"
- **10 GitHub repos every Data Scientist should bookmark in 2026** → **langchain-ai/langgraph** — "agent layer: orchestration and stateful workflows"
- **10 GitHub repos every Data Scientist should bookmark in 2026** → **dair-ai/Prompt-Engineering-Guide** — "agent layer: prompting, grounding, and context engineering"
- **10 GitHub repos for local LLM inference** → **ggerganov/llama.cpp + llama-cpp-python** — "CPU and hybrid local inference foundation"
- **10 GitHub repos for local LLM inference** → **vllm-project/vllm** — "high-throughput GPU inference"
- **10 GitHub repos for local LLM inference** → **sgl-project/sglang** — "efficient repeated-context and structured inference"
- **10 GitHub repos for local LLM inference** → **turboderp/exllamav2** — "fast quantized inference on consumer NVIDIA GPUs"
- **10 GitHub repos for local LLM inference** → **ml-explore/mlx-lm** — "Apple Silicon and unified-memory inference"
- **10 GitHub repos for local LLM inference** → **NVIDIA/TensorRT-LLM** — "maximum performance on NVIDIA hardware"
- **10 GitHub repos for local LLM inference** → **dottxt-ai/outlines** — "structured and deterministic output control"
- **10 GitHub repos for local LLM inference** → **ollama/ollama** — "fastest zero-config local experimentation"
- **10 GitHub repos for local LLM inference** → **exo-explore/exo** — "distributed local inference across consumer devices"
- **10 GitHub repos for local LLM inference** → **flashinfer-ai/flashinfer** — "low-level attention and decode kernels"
- **Local Models, Serving & Inference** → **10 GitHub repos for local LLM inference** — "provides the existing ten-repository local-inference reference collection"
- **Data Scientist Universe** → **01. Mathematics & Statistics** — "organizes foundational mathematics and statistics"
- **01. Mathematics & Statistics** → **Descriptive Statistics & Data Exploration** — "introduces exploratory and descriptive statistical methods"
- **Descriptive Statistics & Data Exploration** → **Mean, Median, Mode & Weighted Averages** — "develops measures of central tendency with mathematical intuition and Python examples"
- **Descriptive Statistics & Data Exploration** → **Variance, Standard Deviation & IQR** — "teaches how to measure the spread of observations around their center"
- **Descriptive Statistics & Data Exploration** → **Distributions, Skewness & Outliers** — "explains shape, asymmetry, and unusual observations in exploratory data analysis"
- **Descriptive Statistics & Data Exploration** → **Covariance, Correlation & Relationships** — "teaches how paired variables co-vary and how to assess linear and monotonic associations"
- **Descriptive Statistics & Data Exploration** → **Data Quality & Exploratory Data Analysis** — "teaches practical data-quality auditing and exploratory analysis before modeling"
- **01. Mathematics & Statistics** → **Probability & Distributions** — "introduces event uncertainty and probability distribution models"
- **Probability & Distributions** → **Probability Fundamentals** — "develops basic event probability, complements, unions, intersections and independence"
- **Probability & Distributions** → **Conditional Probability & Bayes' Theorem** — "explains conditional event probabilities and updating beliefs using Bayes' theorem"

---
_Shared from [Mindlify](https://mindlify.co) — AI-powered thought networks_