> ## Documentation Index
> Fetch the complete documentation index at: https://aitutorial.dev/llms.txt
> Use this file to discover all available pages before exploring further.

# Videos and Volumetric Data

> Extending 2D vision to video and to 3D medical volumes like CT and MRI

export const QuizQuestion = ({question, options, answer, explanation}) => {
  const [selected, setSelected] = useState(null);
  const [revealed, setRevealed] = useState(false);
  const handleSelect = index => {
    if (revealed) return;
    setSelected(index);
    setRevealed(true);
  };
  const isCorrect = selected === answer;
  const getOptionClass = i => {
    const classes = ['quiz-option'];
    if (revealed) {
      classes.push('quiz-option-disabled');
      if (i === answer) classes.push('quiz-option-correct'); else if (i === selected && !isCorrect) classes.push('quiz-option-wrong');
    }
    return classes.join(' ');
  };
  return <div className="quiz-card">
      <p className="quiz-question">{question}</p>
      <div className="quiz-options">
        {options.map((option, i) => <button key={i} onClick={() => handleSelect(i)} className={getOptionClass(i)}>
            <span className="quiz-letter">{String.fromCharCode(65 + i)}</span>
            {option}
          </button>)}
      </div>
      {revealed && <div className={`quiz-feedback ${isCorrect ? 'quiz-feedback-correct' : 'quiz-feedback-wrong'}`}>
          <strong>{isCorrect ? 'Correct!' : 'Incorrect.'}</strong> {explanation}
        </div>}
    </div>;
};

export const Quiz = ({title = "Check Your Understanding", children}) => {
  return <div style={{
    marginTop: '24px'
  }}>
      <div className="quiz-title">{title}</div>
      {children}
    </div>;
};

export const CodeEditor = ({file = 'src/hello_world.ts', lines, title = 'Code Example', repo = 'ai-tutorial/typescript-examples', height = '650px', functionName, theme: userTheme}) => {
  const STORAGE_KEY = 'openai_api_key';
  const GEMINI_STORAGE_KEY = 'gemini_api_key';
  const ANTHROPIC_STORAGE_KEY = 'anthropic_api_key';
  const PROVIDER_STORAGE_KEY = 'llm_playground_provider';
  if (!functionName) {
    console.warn('CodeEditor: functionName parameter is required');
  }
  const hasCreatedEnvRef = useRef(false);
  const vmRef = useRef(null);
  const [isMaximized, setIsMaximized] = useState(false);
  const [isCollapsed, setIsCollapsed] = useState(false);
  const [isStuck, setIsStuck] = useState(false);
  const [iframeKey, setIframeKey] = useState(0);
  const [showApiKeyDialog, setShowApiKeyDialog] = useState(false);
  const [apiKey, setApiKey] = useState('');
  const [error, setError] = useState('');
  const [success, setSuccess] = useState(false);
  const [isSubmitting, setIsSubmitting] = useState(false);
  const [isValidating, setIsValidating] = useState(false);
  const [detectedTheme, setDetectedTheme] = useState('dark');
  useEffect(() => {
    if (typeof window === 'undefined') return;
    const checkTheme = () => {
      const isDark = document.documentElement.classList.contains('dark');
      setDetectedTheme(isDark ? 'dark' : 'light');
    };
    checkTheme();
    const observer = new MutationObserver(checkTheme);
    observer.observe(document.documentElement, {
      attributes: true,
      attributeFilter: ['class']
    });
    return () => observer.disconnect();
  }, []);
  const theme = userTheme || detectedTheme;
  const [selectedProvider, setSelectedProvider] = useState(() => {
    if (typeof window === 'undefined') return 'gemini';
    return localStorage.getItem(PROVIDER_STORAGE_KEY) || 'gemini';
  });
  const isApiKeyConfigured = () => {
    const openaiKey = localStorage.getItem(STORAGE_KEY);
    const geminiKey = localStorage.getItem(GEMINI_STORAGE_KEY);
    const anthropicKey = localStorage.getItem(ANTHROPIC_STORAGE_KEY);
    return openaiKey !== null && openaiKey.trim().length > 0 || geminiKey !== null && geminiKey.trim().length > 0 || anthropicKey !== null && anthropicKey.trim().length > 0;
  };
  const dispatchApiKeyChanged = () => {
    if (typeof window !== 'undefined' && window.dispatchEvent) {
      window.dispatchEvent(new CustomEvent('apiKeyChanged', {
        detail: {
          configured: isApiKeyConfigured()
        }
      }));
    }
  };
  const saveApiKey = apiKey => {
    if (apiKey && apiKey.trim()) {
      const trimmedKey = apiKey.trim();
      localStorage.setItem(STORAGE_KEY, trimmedKey);
      dispatchApiKeyChanged();
      return true;
    }
    return false;
  };
  const buildEnvContent = () => {
    const openaiKey = localStorage.getItem(STORAGE_KEY)?.trim();
    const geminiKey = localStorage.getItem(GEMINI_STORAGE_KEY)?.trim();
    const anthropicKey = localStorage.getItem(ANTHROPIC_STORAGE_KEY)?.trim();
    if (!openaiKey && !geminiKey && !anthropicKey) {
      return `OPENAI_MODEL=gpt-4.1-nano
OPENAI_API_KEY=sk-mock-key-1234567890abcdef
GEMINI_MODEL=gemini-2.5-flash-lite
GOOGLE_GENERATIVE_AI_API_KEY=
GOOGLE_API_KEY=
ANTHROPIC_API_KEY=
AI_PROVIDER=openai
# API key not found in browser storage
# To configure your API key:
# 1. For Gemini (free): Go to https://aistudio.google.com/apikey
# 2. For OpenAI: Go to https://platform.openai.com/api-keys
# 3. For Claude: Go to https://console.anthropic.com/settings/keys
# 4. Enter it in the configuration form above this editor
# 5. The .env file will be automatically updated with your key`;
    }
    const envLines = ['# Using the API key(s) you configured. This file will be created when the dialog is loaded.'];
    if (openaiKey) {
      envLines.push(`OPENAI_MODEL=gpt-4.1-nano`);
      envLines.push(`OPENAI_API_KEY=${openaiKey}`);
    }
    if (geminiKey) {
      envLines.push(`GEMINI_MODEL=gemini-2.5-flash-lite`);
      envLines.push(`# Vercel AI SDK uses GOOGLE_GENERATIVE_AI_API_KEY, LangChain uses GOOGLE_API_KEY`);
      envLines.push(`GOOGLE_GENERATIVE_AI_API_KEY=${geminiKey}`);
      envLines.push(`GOOGLE_API_KEY=${geminiKey}`);
    }
    if (anthropicKey) {
      envLines.push(`ANTHROPIC_API_KEY=${anthropicKey}`);
    }
    const provider = anthropicKey ? 'anthropic' : geminiKey ? 'gemini' : 'openai';
    envLines.push(`AI_PROVIDER=${provider}`);
    return envLines.join('\n');
  };
  const updateEnvFile = async vm => {
    if (!vm) return;
    try {
      await vm.applyFsDiff({
        create: {
          'env/.env': buildEnvContent(),
          'env/run.conf': `file=${file}`
        },
        destroy: []
      });
      hasCreatedEnvRef.current = true;
    } catch (error) {
      console.error('Failed to write env files:', error);
      hasCreatedEnvRef.current = false;
    }
  };
  useEffect(() => {
    if (!isApiKeyConfigured()) {
      setShowApiKeyDialog(true);
    }
    const handleApiKeyChanged = () => {
      if (isApiKeyConfigured()) {
        setShowApiKeyDialog(false);
      }
    };
    if (typeof window !== 'undefined') {
      window.addEventListener('apiKeyChanged', handleApiKeyChanged);
      return () => {
        window.removeEventListener('apiKeyChanged', handleApiKeyChanged);
      };
    }
  }, []);
  const validateApiKey = async (key, provider) => {
    try {
      const urls = {
        gemini: 'https://generativelanguage.googleapis.com/v1beta/models?key=' + encodeURIComponent(key.trim()),
        openai: 'https://api.openai.com/v1/models',
        anthropic: 'https://api.anthropic.com/v1/models'
      };
      const headerMap = {
        gemini: {
          'Content-Type': 'application/json'
        },
        openai: {
          'Authorization': `Bearer ${key.trim()}`,
          'Content-Type': 'application/json'
        },
        anthropic: {
          'x-api-key': key.trim(),
          'anthropic-version': '2023-06-01',
          'Content-Type': 'application/json'
        }
      };
      const url = urls[provider];
      const headers = headerMap[provider];
      const response = await fetch(url, {
        method: 'GET',
        headers
      });
      if (response.ok) {
        return {
          valid: true
        };
      } else if (response.status === 401 || response.status === 403) {
        return {
          valid: false,
          error: 'Invalid API key. Please check your key and try again.'
        };
      } else if (response.status === 429) {
        return {
          valid: false,
          error: 'Rate limit exceeded. Please try again later.'
        };
      } else {
        const errorData = await response.json().catch(() => ({}));
        return {
          valid: false,
          error: errorData.error?.message || `API request failed with status ${response.status}`
        };
      }
    } catch (err) {
      if (err.name === 'TypeError' && err.message.includes('fetch')) {
        return {
          valid: false,
          error: 'Network error. Please check your connection and try again.'
        };
      }
      return {
        valid: false,
        error: err.message || 'Failed to validate API key. Please try again.'
      };
    }
  };
  const handleSkipConfiguration = () => {
    const skipKey = 'sk-<configure-your-key>';
    saveApiKey(skipKey);
    setShowApiKeyDialog(false);
  };
  const handleApiKeySubmit = async e => {
    e.preventDefault();
    setError('');
    setSuccess(false);
    setIsSubmitting(true);
    const providerNames = {
      gemini: 'Gemini',
      openai: 'OpenAI',
      anthropic: 'Claude'
    };
    if (!apiKey || !apiKey.trim()) {
      setError(`Please enter your ${providerNames[selectedProvider]} API key`);
      setIsSubmitting(false);
      return;
    }
    const trimmedKey = apiKey.trim();
    if (selectedProvider === 'openai' && !trimmedKey.startsWith('sk-')) {
      setError('Invalid API key format. OpenAI API keys should start with "sk-"');
      setIsSubmitting(false);
      return;
    }
    if (selectedProvider === 'anthropic' && !trimmedKey.startsWith('sk-ant-')) {
      setError('Invalid API key format. Anthropic API keys should start with "sk-ant-"');
      setIsSubmitting(false);
      return;
    }
    setIsValidating(true);
    setError('');
    const validation = await validateApiKey(trimmedKey, selectedProvider);
    setIsValidating(false);
    if (!validation.valid) {
      setError(validation.error || 'Invalid API key. Please check your key and try again.');
      setIsSubmitting(false);
      return;
    }
    try {
      const storageKeys = {
        gemini: GEMINI_STORAGE_KEY,
        openai: STORAGE_KEY,
        anthropic: ANTHROPIC_STORAGE_KEY
      };
      localStorage.setItem(storageKeys[selectedProvider], trimmedKey);
      localStorage.setItem(PROVIDER_STORAGE_KEY, selectedProvider);
      dispatchApiKeyChanged();
      setSuccess(true);
      setApiKey('');
      setTimeout(() => {
        window.location.reload();
      }, 1000);
    } catch (err) {
      setError(err.message || 'Failed to save API key. Please try again.');
      setIsSubmitting(false);
    }
  };
  const baseFilePath = file || 'src/hello_world.ts';
  let filePath = baseFilePath;
  if (typeof lines === 'string' && lines.trim()) {
    const lineParts = lines.split('-');
    if (lineParts.length === 2) {
      filePath = `${filePath}:L${lineParts[0].trim()}-L${lineParts[1].trim()}`;
    } else {
      filePath = `${filePath}:L${lineParts[0].trim()}`;
    }
  } else if (typeof lines === 'object' && lines.start !== undefined) {
    filePath = lines.end !== undefined ? `${filePath}:L${lines.start}-L${lines.end}` : `${filePath}:L${lines.start}`;
  }
  const stackblitzUrl = `https://stackblitz.com/github/${repo}?file=${encodeURIComponent(filePath)}&embed=1&view=editor&theme=${theme}`;
  const loadSDK = () => {
    return new Promise((resolve, reject) => {
      if (window.StackBlitzSDK || window.stackblitzSDK) {
        resolve(window.StackBlitzSDK || window.stackblitzSDK);
        return;
      }
      if (document.querySelector('script[data-stackblitz-sdk]')) {
        const checkInterval = setInterval(() => {
          if (window.StackBlitzSDK || window.stackblitzSDK) {
            clearInterval(checkInterval);
            resolve(window.StackBlitzSDK || window.stackblitzSDK);
          }
        }, 100);
        setTimeout(() => {
          clearInterval(checkInterval);
          reject(new Error('SDK loading timeout'));
        }, 10000);
        return;
      }
      const script = document.createElement('script');
      script.src = 'https://unpkg.com/@stackblitz/sdk/bundles/sdk.umd.js';
      script.async = true;
      script.setAttribute('data-stackblitz-sdk', 'true');
      script.onload = () => {
        const sdk = window.StackBlitzSDK || window.stackblitzSDK;
        if (sdk) {
          resolve(sdk);
        } else {
          reject(new Error('SDK loaded but not available on window'));
        }
      };
      script.onerror = () => {
        reject(new Error('Failed to load StackBlitz SDK'));
      };
      document.head.appendChild(script);
    });
  };
  const LOAD_TIMEOUT_MS = 10000;
  const iframeElRef = useRef(null);
  const reloadCountRef = useRef(0);
  const handleRetry = () => {
    vmRef.current = null;
    hasCreatedEnvRef.current = false;
    reloadCountRef.current = 0;
    setIsStuck(false);
    setIframeKey(prev => prev + 1);
  };
  const iframeRef = iframe => {
    iframeElRef.current = iframe;
  };
  const connectToVM = async iframe => {
    const sdk = await loadSDK();
    return sdk.connect(iframe);
  };
  const handleIframeLoad = async () => {
    const iframe = iframeElRef.current;
    if (!iframe) return;
    if (reloadCountRef.current > 0) {
      try {
        const vm = await connectToVM(iframe);
        vmRef.current = vm;
        await updateEnvFile(vm);
      } catch (_) {}
      return;
    }
    try {
      if (vmRef.current) return;
      const vm = await Promise.race([connectToVM(iframe), new Promise((_, reject) => setTimeout(() => reject(new Error('connect timeout')), LOAD_TIMEOUT_MS))]);
      vmRef.current = vm;
      await updateEnvFile(vm);
    } catch (error) {
      console.error('Failed to connect to StackBlitz VM:', error);
      if (typeof window !== 'undefined' && window.gtag) {
        window.gtag('event', 'load_refresh_error', {
          event_category: 'stackblitz',
          event_label: file,
          error_message: error.message
        });
      }
      reloadCountRef.current = 1;
      setTimeout(() => {
        setIframeKey(prev => prev + 1);
      }, 2000);
    }
  };
  const isSafari = typeof navigator !== 'undefined' && (/^((?!chrome|android).)*safari/i).test(navigator.userAgent);
  if (isSafari) {
    return <div className="code-editor-dialog-container" style={{
      height: height
    }}>
        <div className="code-editor-dialog-box">
          <h2 className="code-editor-dialog-title">
            <svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#f59e0b" strokeWidth="2">
              <path d="M10.29 3.86L1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z"></path>
              <line x1="12" y1="9" x2="12" y2="13"></line>
              <line x1="12" y1="17" x2="12.01" y2="17"></line>
            </svg>
            Browser Not Supported
          </h2>
          <p className="code-editor-dialog-description">
            The interactive code editor is not supported on Safari. Please use <strong>Chrome</strong>, <strong>Edge</strong>, or <strong>Firefox</strong> to run the examples.
          </p>
        </div>
      </div>;
  }
  if (showApiKeyDialog) {
    return <div className="code-editor-dialog-container" style={{
      height: height
    }}>
        <div className="code-editor-dialog-box">
          <h2 className="code-editor-dialog-title">
            <svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#f59e0b" strokeWidth="2">
              <path d="M10.29 3.86L1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z"></path>
              <line x1="12" y1="9" x2="12" y2="13"></line>
              <line x1="12" y1="17" x2="12.01" y2="17"></line>
            </svg>
            Configure API Key
          </h2>

          <p className="code-editor-dialog-description">
            All interactive examples execute entirely within your browser environment, ensuring complete security and privacy.
            Your API key is stored locally in your browser's storage and is never transmitted to external servers.
          </p>

          <div className="llm-provider-tabs" style={{
      marginBottom: '16px'
    }}>
            <button type="button" onClick={() => {
      setSelectedProvider('gemini');
      setError('');
      setApiKey('');
    }} className={`llm-provider-tab ${selectedProvider === 'gemini' ? 'llm-provider-tab-active' : ''}`}>
              Gemini <span className="llm-provider-tab-badge">Free</span>
            </button>
            <button type="button" onClick={() => {
      setSelectedProvider('openai');
      setError('');
      setApiKey('');
    }} className={`llm-provider-tab ${selectedProvider === 'openai' ? 'llm-provider-tab-active' : ''}`}>
              OpenAI
            </button>
            <button type="button" onClick={() => {
      setSelectedProvider('anthropic');
      setError('');
      setApiKey('');
    }} className={`llm-provider-tab ${selectedProvider === 'anthropic' ? 'llm-provider-tab-active' : ''}`}>
              Claude
            </button>
          </div>

          {selectedProvider === 'gemini' && <div className="llm-gemini-recommendation" style={{
      marginBottom: '16px'
    }}>
              Gemini offers a generous free tier — great for learning! Get your free API key at{' '}
              <a href="https://aistudio.google.com/apikey" target="_blank" rel="noopener noreferrer" className="code-editor-link">
                aistudio.google.com/apikey
              </a>
            </div>}

          {selectedProvider === 'openai' && <div className="code-editor-info-box">
              <p className="code-editor-info-box-title">
                Don't have an API key?
              </p>
              <p className="code-editor-info-box-text">
                Get one at{' '}
                <a href="https://platform.openai.com/api-keys" target="_blank" rel="noopener noreferrer" className="code-editor-link">
                  platform.openai.com/api-keys
                </a>
              </p>
            </div>}

          {selectedProvider === 'anthropic' && <div className="code-editor-info-box">
              <p className="code-editor-info-box-title">
                Don't have an API key?
              </p>
              <p className="code-editor-info-box-text">
                Get one at{' '}
                <a href="https://console.anthropic.com/settings/keys" target="_blank" rel="noopener noreferrer" className="code-editor-link">
                  console.anthropic.com/settings/keys
                </a>
              </p>
            </div>}

          <form onSubmit={handleApiKeySubmit}>
            <div className="code-editor-form-group">
              <label htmlFor="api-key-input" className="code-editor-label">
                {selectedProvider === 'gemini' ? 'Gemini' : 'OpenAI'} API Key
              </label>
              <input id="api-key-input" type="password" value={apiKey} onChange={e => {
      setApiKey(e.target.value);
      setError('');
      setSuccess(false);
    }} placeholder={selectedProvider === 'openai' ? 'sk-...' : 'Gemini API Key'} disabled={isSubmitting} className={`code-editor-input ${error ? 'code-editor-input-error' : ''}`} />
            </div>

            {isValidating && <div className="code-editor-message code-editor-message-info">
                <span className="code-editor-message-icon">⏳</span>
                <span>Validating API key...</span>
              </div>}

            {error && !isValidating && <div className="code-editor-message code-editor-message-error">
                <span className="code-editor-message-icon">⚠️</span>
                <span>{error}</span>
              </div>}

            {success && <div className="code-editor-message code-editor-message-success">
                <span className="code-editor-message-icon">✓</span>
                <span>API key saved successfully! Loading editor...</span>
              </div>}

            <button type="submit" disabled={isSubmitting || isValidating || !apiKey.trim()} className="code-editor-button">
              {isValidating ? 'Validating...' : isSubmitting ? 'Saving...' : 'Save API Key'}
            </button>
          </form>

          <button type="button" onClick={handleSkipConfiguration} disabled={isSubmitting || isValidating} className="code-editor-button-secondary">
            Skip Configuration
          </button>

          <div className="code-editor-footer">
            <p className="code-editor-footer-text">
              Alternatively, you may checkout the source code from{' '}
              <a href="https://github.com/ai-tutorial/typescript-examples" target="_blank" rel="noopener noreferrer" className="code-editor-link code-editor-link-break">
                https://github.com/ai-tutorial/typescript-examples
              </a>
              {' '}and run the examples locally.
            </p>
          </div>
        </div>
      </div>;
  }
  const toggleMaximize = () => setIsMaximized(!isMaximized);
  const toggleCollapse = () => setIsCollapsed(!isCollapsed);
  return <div className={`code-editor-wrapper ${isMaximized ? 'maximized' : ''} ${isCollapsed ? 'collapsed' : ''}`} data-theme={theme}>
      <div className="code-editor-header">
        <div className="code-editor-title">
          <svg width="14" height="14" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
            <path d="M11 4H4a2 2 0 0 0-2 2v14a2 2 0 0 0 2 2h14a2 2 0 0 0 2-2v-7"></path>
            <path d="M18.5 2.5a2.121 2.121 0 0 1 3 3L12 15l-4 1 1-4 9.5-9.5z"></path>
          </svg>
          {title}
        </div>
        <div className="code-editor-controls">
          {!isMaximized && <button className="code-editor-collapse-button" onClick={toggleCollapse} title={isCollapsed ? "Expand" : "Collapse"} type="button">
              <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
                {isCollapsed ? <polyline points="6 9 12 15 18 9" /> : <polyline points="6 15 12 9 18 15" />}
              </svg>
            </button>}
          <button className="code-editor-maximize-button" onClick={toggleMaximize} title={isMaximized ? "Minimize" : "Maximize (Focus Mode)"} type="button">
            <svg xmlns="http://www.w3.org/2000/svg" width="24" height="24" viewBox="0 0 24 24" fill="none" stroke="currentColor" strokeWidth="2" strokeLinecap="round" strokeLinejoin="round">
              {isMaximized ? <><path d="M4 14h6v6" /><path d="M20 10h-6V4" /><path d="M14 10l7-7" /><path d="M3 21l7-7" /></> : <><path d="M15 3h6v6" /><path d="M9 21H3v-6" /><path d="M21 3l-7 7" /><path d="M3 21l7-7" /></>}
            </svg>
          </button>
        </div>
      </div>

      {!isCollapsed && <div style={{
    position: 'relative',
    height: isMaximized ? 'auto' : height,
    flex: isMaximized ? 1 : 'none'
  }}>
          <iframe key={iframeKey} ref={iframeRef} onLoad={handleIframeLoad} src={stackblitzUrl} className="code-editor-iframe" style={{
    height: '100%',
    flex: isMaximized ? 1 : 'none'
  }} title={title || 'Code Example'} allow="accelerometer; camera; encrypted-media; geolocation; gyroscope; hid; microphone; midi; payment; usb; xr-spatial-tracking" sandbox="allow-forms allow-modals allow-popups allow-presentation allow-same-origin allow-scripts" />

          {isStuck && <div className="code-editor-stuck-overlay">
              <div className="code-editor-stuck-box">
                <svg width="20" height="20" viewBox="0 0 24 24" fill="none" stroke="#f59e0b" strokeWidth="2">
                  <path d="M10.29 3.86L1.82 18a2 2 0 0 0 1.71 3h16.94a2 2 0 0 0 1.71-3L13.71 3.86a2 2 0 0 0-3.42 0z"></path>
                  <line x1="12" y1="9" x2="12" y2="13"></line>
                  <line x1="12" y1="17" x2="12.01" y2="17"></line>
                </svg>
                <p>StackBlitz is taking too long to load. This can happen when the repository was recently updated.</p>
                <button type="button" className="code-editor-button" onClick={handleRetry} style={{
    marginTop: '8px'
  }}>
                  Retry
                </button>
              </div>
            </div>}
        </div>}
    </div>;
};

Two dimensions are not enough for half the real world. Autonomous driving has to reason over time, radiology has to reason over a stack of anatomical slices, and content moderation has to understand 30-second clips, not 30 still frames. This page covers how the tools you already have generalize to video and to 3D volumetric data, with a focus on medical imaging.

## Video: Time as an Extra Axis

A video adds a **temporal** axis on top of the usual spatial ones:

```text theme={null}
Still image:     (C, H, W)
Video clip:      (T, C, H, W)         for a single clip
Video batch:     (N, T, C, H, W)      batched
```

From here, the architectural choices fall into a handful of patterns.

### Frame-by-Frame

The simplest option: run your 2D model on every frame independently, optionally smooth the outputs with a temporal filter or tracker. Works astonishingly well for many tasks (detection, classification, segmentation) because modern 2D models are strong and video frames are highly redundant. This is the default you should try first.

### 3D Convolutions

Replace `Conv2D(k, k)` with `Conv3D(t, k, k)`. The filter now slides across time as well as space and can learn motion features directly. Classic models: **C3D**, **I3D** (inflated 3D — take a 2D ImageNet-pretrained network and "inflate" every 2D filter into 3D). Expensive, but still a competitive baseline.

### Two-Stream

One network sees RGB frames, another sees **optical flow** (the pixel-level motion between consecutive frames). Their predictions get combined. Popular in early deep video work; still useful when motion is the primary signal (e.g., action recognition).

### Temporal Transformers

Tubelet an image into patches over both space and time, then run a transformer. Models: **ViViT**, **TimeSformer**, **MViT**, **VideoMAE**, **Video Swin**. These have largely taken over video classification and are the backbone behind modern video foundation models and text-to-video generators like **Sora**.

### Tasks You'll See

| Task                                | What it outputs                                                                            |
| ----------------------------------- | ------------------------------------------------------------------------------------------ |
| **Action recognition**              | A class per clip ("running", "cooking") — the "ImageNet" of video                          |
| **Temporal action detection**       | `(class, start_time, end_time)` triples across a long video                                |
| **Multi-object tracking (MOT)**     | Per-object trajectories across frames — detector + association algorithm (SORT, ByteTrack) |
| **Video object segmentation (VOS)** | A mask for each target across every frame — SAM 2 is the current default                   |
| **Text-to-video generation**        | A video sampled from a text prompt — Sora, Gen-3, Veo                                      |

## Volumetric Data: 3D From the Start

Unlike video, where most of the information is still 2D and time is a bonus, medical volumes are genuinely 3D: anatomy is continuous across slices, and features (tumors, vessels, bones) live in 3D space.

### CT (Computed Tomography)

A CT scanner rotates an X-ray source and detector around the patient and reconstructs a stack of cross-sectional slices. Output is typically:

```text theme={null}
(D, H, W) in Hounsfield Units (HU)       — single scalar per voxel
typical shape: ~(30–500, 512, 512)
```

**Hounsfield Units** are calibrated physical units: air is −1000, water is 0, bone is +1000 to +2000. Windowing (clipping to a target HU range and rescaling to `[0, 1]`) is the equivalent of "normalization" for CT — different windows emphasize different tissues (lung window, bone window, soft-tissue window). Your network will see completely different images depending on which window you pick.

### MRI (Magnetic Resonance Imaging)

MRI produces volumes too, but with a twist: the same anatomy is imaged with **multiple pulse sequences** (T1, T2, FLAIR, DWI), each highlighting different tissue properties. Treat each sequence as a channel:

```text theme={null}
(C, D, H, W)   with C = number of sequences (often 3 or 4)
```

Unlike CT, MRI intensities have **no fixed physical meaning** — the same tumor might be bright on one scan and dim on another from a different scanner. Per-volume z-score normalization is standard.

### DICOM, NIfTI, and the Loader Problem

Medical data formats are their own ecosystem:

* **DICOM (.dcm):** the clinical standard. Each slice is a separate file with extensive metadata (patient ID, acquisition parameters, orientation). Loading requires sorting slices by `SliceLocation` and respecting patient orientation.
* **NIfTI (.nii / .nii.gz):** the research standard. A single file containing the 3D array plus an affine matrix mapping voxel coordinates to physical (world) space.
* **Affine / orientation:** the affine matrix tells you which axis is superior-inferior, anterior-posterior, left-right in the patient's frame. Training on mixed orientations without canonicalizing produces bizarre bugs.

Libraries: **pydicom**, **nibabel**, **SimpleITK**, **MONAI** (the PyTorch-based library that packages medical-specific layers, losses, and augmentations).

### Architectures for 3D Medical Imaging

* **3D U-Net:** the direct extension of 2D U-Net with `Conv3D` — still the workhorse.
* **nnU-Net (2021):** a self-configuring 3D U-Net pipeline. It inspects your dataset and picks spacing, patching, and architecture automatically. Wins medical segmentation benchmarks year after year with almost no tuning.
* **Swin UNETR, UNETR:** transformer-based 3D encoders that trade more compute for better long-range context.
* **TotalSegmentator:** a nnU-Net-based model that segments 100+ anatomical structures on whole-body CT out of the box.

### Why You Can't Just Use 2D Models Slice-by-Slice

You can, and for some tasks (bone fracture detection, large-organ outlines) it works fine. But:

* **Small structures** (a 3-voxel nodule, a thin vessel) visible across only a few slices get missed or flicker between them.
* **3D context** matters: a structure's cross-section often looks the same as something else in a single slice and is only distinguishable via its 3D shape.
* **Memory:** 3D U-Nets on full `(300, 512, 512)` volumes don't fit on a GPU. You train on **patches** (e.g., `(96, 96, 96)`) and stitch predictions at inference — a standard technique with gotchas at patch boundaries.

## Worked Example: Load a CT Volume, Window It, Feed a Patch to a Network

<CodeEditor file="src/computer-vision/ct_patch_pipeline.ts" functionName="main" lines="1-100" title="CT volume loading, windowing, patching, inference (placeholder)" />

### Practical Implication

Medical imaging looks like regular computer vision but has a pile of domain conventions — physical units, multi-modality, orientation, patch-based training — that are invisible in the model architecture but determine whether the pipeline works at all. Use **MONAI** or **nnU-Net**; they encode a decade of hard-won conventions so you don't rediscover them one bug at a time.

#### ❌ Antipattern

Training a 2D ImageNet-pretrained ResNet on raw CT slice PNGs with default RGB normalization. You've thrown away HU calibration, 3D context, orientation, and slice spacing — the result is a model that "works" on the dev set and fails silently on scans from a different hospital.

#### ✅ Best Practice

Resample every volume to a fixed physical spacing, window to the tissue range that matters for your task, train a 3D U-Net (or nnU-Net) on random 3D patches with patient-level train/val splits, and evaluate on volumes from a held-out scanner or site.

<Quiz>
  <QuizQuestion question="You train a lung-nodule detector on CT scans from one hospital and deploy it to another. Sensitivity drops from 92% to 61%. What's the most likely cause and fix?" options={["The model is too small — retrain with more parameters", "Distribution shift from different scanners and acquisition protocols (spacing, kernel, dose). Canonical preprocessing (resampling to fixed spacing, HU windowing, z-score normalization) and multi-site training data close most of this gap", "The hospital is sending older images"]} answer={1} explanation="Cross-site CT performance drops are almost always scanner/protocol differences. Canonicalizing physical spacing, HU windowing, and including multiple sites in training are the standard mitigations." />

  <QuizQuestion question="For a general-purpose 'recognize the action in this 10-second clip' model, you don't have compute to train 3D CNNs from scratch. What's a pragmatic starting point?" options={["Run your strongest 2D image classifier on every frame and majority-vote — or fine-tune a pretrained video transformer like VideoMAE/ViViT, which ships with strong action-recognition weights", "Build a two-stream network with hand-designed optical flow", "Train a new model with 3D convolutions from scratch"]} answer={0} explanation="Pretrained video transformers (VideoMAE, ViViT, VideoSwin) fine-tuned on your data are the current default. Frame-wise majority-voting with a strong 2D model is a very competitive cheap baseline — sometimes it's all you need." />
</Quiz>