• Home
  • Advanced Features

Advanced Features

Deep dive into fast-brake's advanced capabilities, tokenizer internals, performance tuning, and extensibility

fast-brake provides advanced features for power users who need fine-grained control over ES feature detection, performance optimization, and custom integrations.

Tokenizer Deep Dive

The tokenizer is the heart of fast-brake’s accuracy. Understanding its internals helps you optimize performance and troubleshoot edge cases.

Token Types and Structure

interface Token {
  type: string;        // 'keyword', 'identifier', 'operator', 'string', etc.
  value: string;       // The actual token text
  start: number;       // Start position in source
  end: number;         // End position in source
  line: number;        // Line number (1-based)
  column: number;      // Column number (1-based)
}

State Management

The tokenizer tracks context for accurate detection:

enum TokenState {
  NORMAL = 0,          // Regular code
  STRING_SINGLE = 1,   // Inside 'single quotes'
  STRING_DOUBLE = 2,   // Inside "double quotes"
  TEMPLATE = 3,        // Inside `template literals`
  COMMENT_LINE = 4,    // Inside // line comments
  COMMENT_BLOCK = 5,   // Inside /* block comments */
  REGEX = 6           // Inside /regex/ patterns
}

Advanced Tokenization Examples

import { Tokenizer } from 'fast-brake/tokenizer';

const code = `
  const arrow = () => {};
  const str = "This => is not an arrow function";
  // This => is also not an arrow function
  const regex = /=>/; // Neither is this
`;

const tokenizer = new Tokenizer(code);
const allTokens = tokenizer.tokenize();
const codeTokens = tokenizer.getCodeTokens(); // Excludes strings/comments

console.log('All tokens:', allTokens.length);     // ~20 tokens
console.log('Code tokens:', codeTokens.length);   // ~12 tokens (no strings/comments)

// Find actual arrow functions (not in strings/comments)
const arrowTokens = codeTokens.filter(t => t.value === '=>');
console.log('Real arrow functions:', arrowTokens.length); // 1

Performance Tuning

Choosing the Right Mode

Quick Mode - When Speed Matters Most:

import { detect } from 'fast-brake';

// Development builds with hot reloading
const features = detect(code, { quick: true });

// Build tools where speed is critical
const isCompatible = check(code, { target: 'es5', quick: true });

// Batch processing thousands of files
const results = files.map(file => 
  getMinimumESVersion(file.content, { quick: true })
);

Full Mode - When Accuracy is Critical:

// Production builds
const features = detect(code, { quick: false }); // default

// CI/CD pipelines
fastBrake(code, { target: 'es2018' }); // always uses full mode

// Code analysis and reporting
const analysis = detect(codebase, { quick: false });

Batch Processing Optimization

import { getDetector } from 'fast-brake';

// Reuse detector instance for better performance
const detector = getDetector();

// Process multiple files efficiently
const results = files.map(file => ({
  path: file.path,
  features: detector.detect(file.content, { quick: true }),
  minVersion: detector.getMinimumVersion(file.content, { quick: true })
}));

// Memory-efficient streaming for large codebases
async function* processLargeCodebase(files) {
  const detector = getDetector();
  
  for (const file of files) {
    yield {
      path: file.path,
      result: detector.detect(file.content, { quick: true })
    };
  }
}

Memory Management

// Efficient processing with minimal memory allocation
function analyzeCodebase(files, options = {}) {
  const detector = getDetector(); // Singleton - reuses compiled patterns
  const results = [];
  
  for (const file of files) {
    // Process one file at a time to minimize memory usage
    const features = detector.detect(file.content, options);
    
    // Store only essential data
    results.push({
      file: file.path,
      featureCount: features.length,
      minVersion: features.reduce((max, f) => 
        getVersionIndex(f.version) > getVersionIndex(max) ? f.version : max, 'es5'
      )
    });
  }
  
  return results;
}

Custom Feature Detection

Extending Pattern Matching

import { QUICK_PATTERNS, FEATURE_VERSIONS } from 'fast-brake/constants';

// Add custom patterns (for internal use)
const customPatterns = {
  ...QUICK_PATTERNS,
  
  // Custom company-specific patterns
  company_api: /CompanyAPI\./,
  legacy_jquery: /\$\(/,
  custom_decorator: /@CustomDecorator/
};

const customVersions = {
  ...FEATURE_VERSIONS,
  company_api: 'es2020',
  legacy_jquery: 'es5',
  custom_decorator: 'es2015'
};

Building Custom Detectors

import { Detector } from 'fast-brake/detector';

class CustomDetector extends Detector {
  detect(code, options) {
    // Get standard features
    const standardFeatures = super.detect(code, options);
    
    // Add custom detection logic
    const customFeatures = this.detectCustomFeatures(code);
    
    return [...standardFeatures, ...customFeatures];
  }
  
  detectCustomFeatures(code) {
    const features = [];
    
    // Detect company-specific patterns
    if (/CompanyAPI\./.test(code)) {
      features.push({
        name: 'company_api',
        version: 'es2020',
        line: this.getLineNumber(code, code.indexOf('CompanyAPI.')),
        column: code.indexOf('CompanyAPI.') + 1
      });
    }
    
    // Detect legacy jQuery usage
    if (/\$\(/.test(code)) {
      features.push({
        name: 'legacy_jquery',
        version: 'es5',
        line: this.getLineNumber(code, code.indexOf('$(')),
        column: code.indexOf('$(') + 1
      });
    }
    
    return features;
  }
}

// Use custom detector
const customDetector = new CustomDetector();
const features = customDetector.detect(code, { target: 'es2018' });

Advanced Integration Patterns

Webpack Plugin

class FastBrakeWebpackPlugin {
  constructor(options = {}) {
    this.target = options.target || 'es5';
    this.mode = options.mode || 'full';
    this.failOnIncompatible = options.failOnIncompatible || false;
  }
  
  apply(compiler) {
    compiler.hooks.compilation.tap('FastBrakePlugin', (compilation) => {
      compilation.hooks.optimizeChunkAssets.tap('FastBrakePlugin', (chunks) => {
        const { detect, check } = require('fast-brake');
        
        chunks.forEach(chunk => {
          chunk.files.forEach(filename => {
            if (!filename.endsWith('.js')) return;
            
            const asset = compilation.assets[filename];
            const source = asset.source();
            
            // Check compatibility
            const isCompatible = check(source, { 
              target: this.target, 
              quick: this.mode === 'quick' 
            });
            
            if (!isCompatible) {
              const features = detect(source, { 
                target: this.target,
                quick: this.mode === 'quick'
              });
              
              const incompatible = features.filter(f => 
                this.getVersionIndex(f.version) > this.getVersionIndex(this.target)
              );
              
              const message = `${filename} contains incompatible features: ${
                incompatible.map(f => f.name).join(', ')
              }`;
              
              if (this.failOnIncompatible) {
                compilation.errors.push(new Error(message));
              } else {
                compilation.warnings.push(new Error(message));
              }
            }
          });
        });
      });
    });
  }
  
  getVersionIndex(version) {
    const versions = ['es5', 'es2015', 'es2016', 'es2017', 'es2018', 'es2019', 'es2020', 'es2021', 'es2022'];
    return versions.indexOf(version);
  }
}

// Usage
module.exports = {
  plugins: [
    new FastBrakeWebpackPlugin({
      target: 'es2018',
      mode: 'full',
      failOnIncompatible: true
    })
  ]
};

Rollup Plugin

import { detect, check } from 'fast-brake';

function fastBrakePlugin(options = {}) {
  const target = options.target || 'es5';
  const mode = options.mode || 'full';
  
  return {
    name: 'fast-brake',
    
    generateBundle(outputOptions, bundle) {
      Object.entries(bundle).forEach(([fileName, chunk]) => {
        if (chunk.type === 'chunk' && chunk.code) {
          const isCompatible = check(chunk.code, { 
            target, 
            quick: mode === 'quick' 
          });
          
          if (!isCompatible) {
            const features = detect(chunk.code, { target, quick: mode === 'quick' });
            const incompatible = features.filter(f => 
              getVersionIndex(f.version) > getVersionIndex(target)
            );
            
            this.warn(`${fileName} contains incompatible features: ${
              incompatible.map(f => f.name).join(', ')
            }`);
          }
        }
      });
    }
  };
}

// Usage
export default {
  plugins: [
    fastBrakePlugin({
      target: 'es2018',
      mode: 'full'
    })
  ]
};

ESLint Integration

// eslint-plugin-fast-brake
const { detect } = require('fast-brake');

module.exports = {
  rules: {
    'es-version-compatibility': {
      meta: {
        type: 'problem',
        docs: {
          description: 'Enforce ES version compatibility'
        },
        schema: [
          {
            type: 'object',
            properties: {
              target: { type: 'string' },
              mode: { type: 'string', enum: ['quick', 'full'] }
            },
            additionalProperties: false
          }
        ]
      },
      
      create(context) {
        const options = context.options[0] || {};
        const target = options.target || 'es5';
        const mode = options.mode || 'full';
        
        return {
          Program(node) {
            const sourceCode = context.getSourceCode();
            const text = sourceCode.getText();
            
            try {
              const features = detect(text, { 
                target, 
                quick: mode === 'quick' 
              });
              
              const incompatible = features.filter(f => 
                getVersionIndex(f.version) > getVersionIndex(target)
              );
              
              incompatible.forEach(feature => {
                context.report({
                  node,
                  message: `ES feature "${feature.name}" requires ${feature.version} but target is ${target}`,
                  loc: feature.line ? {
                    line: feature.line,
                    column: feature.column || 0
                  } : undefined
                });
              });
            } catch (error) {
              // Fallback to quick mode on tokenizer errors
              if (mode === 'full') {
                const quickFeatures = detect(text, { target, quick: true });
                // Process quick features...
              }
            }
          }
        };
      }
    }
  }
};

Error Handling and Debugging

Advanced Error Information

import { fastBrake } from 'fast-brake';

try {
  fastBrake(code, { target: 'es5' });
} catch (error) {
  if (error.feature) {
    console.log('Compatibility Error Details:');
    console.log(`  Feature: ${error.feature.name}`);
    console.log(`  Requires: ${error.feature.version}`);
    console.log(`  Target: ${error.target}`);
    console.log(`  Location: line ${error.feature.line}, column ${error.feature.column}`);
    
    if (error.feature.snippet) {
      console.log(`  Code: ${error.feature.snippet}`);
    }
    
    // Get suggestions for fixing
    const suggestions = getSuggestions(error.feature, error.target);
    console.log(`  Suggestions: ${suggestions.join(', ')}`);
  }
}

function getSuggestions(feature, target) {
  const suggestions = [];
  
  switch (feature.name) {
    case 'arrow_functions':
      suggestions.push('Use function expressions', 'Add Babel transform');
      break;
    case 'optional_chaining':
      suggestions.push('Use logical AND (&&)', 'Add optional-chaining polyfill');
      break;
    case 'nullish_coalescing':
      suggestions.push('Use logical OR (||)', 'Add nullish-coalescing polyfill');
      break;
    default:
      suggestions.push(`Transpile to ${target}`, 'Add appropriate polyfill');
  }
  
  return suggestions;
}

Debugging Tokenizer Issues

import { Tokenizer } from 'fast-brake/tokenizer';

function debugTokenization(code) {
  const tokenizer = new Tokenizer(code);
  const allTokens = tokenizer.tokenize();
  const codeTokens = tokenizer.getCodeTokens();
  
  console.log('=== Tokenization Debug ===');
  console.log(`Total tokens: ${allTokens.length}`);
  console.log(`Code tokens: ${codeTokens.length}`);
  console.log(`Filtered out: ${allTokens.length - codeTokens.length}`);
  
  // Show filtered tokens (strings, comments)
  const filteredTokens = allTokens.filter(t => 
    !codeTokens.some(ct => ct.start === t.start)
  );
  
  console.log('\nFiltered tokens (strings/comments):');
  filteredTokens.forEach(token => {
    console.log(`  ${token.type}: "${token.value}" at line ${token.line}`);
  });
  
  // Show potential ES features in filtered tokens
  console.log('\nPotential false positives:');
  filteredTokens.forEach(token => {
    if (token.value.includes('=>')) {
      console.log(`  Arrow function syntax in ${token.type}: "${token.value}"`);
    }
    if (token.value.includes('?.')) {
      console.log(`  Optional chaining in ${token.type}: "${token.value}"`);
    }
  });
}

// Usage
const problematicCode = `
  const arrow = () => {};
  const str = "This => looks like an arrow function";
  // This => comment also looks suspicious
  const regex = /=>/g;
`;

debugTokenization(problematicCode);

Performance Monitoring

Benchmarking Custom Code

import { detect, getMinimumESVersion } from 'fast-brake';

function benchmarkDetection(code, iterations = 1000) {
  console.log('=== Performance Benchmark ===');
  
  // Warm up
  for (let i = 0; i < 10; i++) {
    detect(code, { quick: true });
    detect(code, { quick: false });
  }
  
  // Benchmark quick mode
  const quickStart = performance.now();
  for (let i = 0; i < iterations; i++) {
    detect(code, { quick: true });
  }
  const quickTime = performance.now() - quickStart;
  
  // Benchmark full mode
  const fullStart = performance.now();
  for (let i = 0; i < iterations; i++) {
    detect(code, { quick: false });
  }
  const fullTime = performance.now() - fullStart;
  
  console.log(`Quick mode: ${(quickTime / iterations).toFixed(3)}ms per call`);
  console.log(`Full mode: ${(fullTime / iterations).toFixed(3)}ms per call`);
  console.log(`Speed difference: ${(fullTime / quickTime).toFixed(1)}x`);
  
  return {
    quickAvg: quickTime / iterations,
    fullAvg: fullTime / iterations,
    speedup: fullTime / quickTime
  };
}

// Usage
const testCode = `
  const arrow = () => {};
  class MyClass {
    #private = 'secret';
    async method() {
      return await fetch('/api');
    }
  }
`;

benchmarkDetection(testCode);

Memory Usage Monitoring

function monitorMemoryUsage(files) {
  const initialMemory = process.memoryUsage();
  console.log('Initial memory:', formatMemory(initialMemory));
  
  const detector = getDetector();
  const results = [];
  
  files.forEach((file, index) => {
    const features = detector.detect(file.content, { quick: false });
    results.push({ file: file.path, features });
    
    // Check memory every 100 files
    if (index % 100 === 0) {
      const currentMemory = process.memoryUsage();
      console.log(`After ${index + 1} files:`, formatMemory(currentMemory));
    }
  });
  
  const finalMemory = process.memoryUsage();
  console.log('Final memory:', formatMemory(finalMemory));
  
  return results;
}

function formatMemory(memUsage) {
  return {
    rss: `${Math.round(memUsage.rss / 1024 / 1024)}MB`,
    heapUsed: `${Math.round(memUsage.heapUsed / 1024 / 1024)}MB`,
    heapTotal: `${Math.round(memUsage.heapTotal / 1024 / 1024)}MB`
  };
}

Best Practices for Advanced Usage

1. Mode Selection Strategy

// Development: Quick mode for speed
const devConfig = {
  mode: 'quick',
  target: 'es2015'
};

// CI/CD: Full mode for accuracy
const ciConfig = {
  mode: 'full',
  target: 'es2018',
  throwOnFirst: true
};

// Production analysis: Full mode with detailed reporting
const prodConfig = {
  mode: 'full',
  target: 'es5',
  generateReport: true
};

2. Error Recovery Patterns

function robustDetection(code, options) {
  try {
    // Try full mode first
    return detect(code, { ...options, quick: false });
  } catch (error) {
    console.warn('Full mode failed, falling back to quick mode:', error.message);
    
    try {
      // Fallback to quick mode
      return detect(code, { ...options, quick: true });
    } catch (quickError) {
      console.error('Both modes failed:', quickError.message);
      return []; // Return empty array as last resort
    }
  }
}

3. Caching for Large Codebases

const detectionCache = new Map();

function cachedDetection(code, options) {
  const cacheKey = `${code.length}-${JSON.stringify(options)}`;
  
  if (detectionCache.has(cacheKey)) {
    return detectionCache.get(cacheKey);
  }
  
  const result = detect(code, options);
  detectionCache.set(cacheKey, result);
  
  // Limit cache size
  if (detectionCache.size > 1000) {
    const firstKey = detectionCache.keys().next().value;
    detectionCache.delete(firstKey);
  }
  
  return result;
}

4. Progressive Enhancement

function progressiveDetection(code, target) {
  // Start with quick mode for immediate feedback
  const quickResult = detect(code, { target, quick: true });
  
  // If quick mode finds issues, run full mode for accuracy
  if (quickResult.length > 0) {
    const fullResult = detect(code, { target, quick: false });
    return {
      mode: 'full',
      features: fullResult,
      quickFeatures: quickResult
    };
  }
  
  return {
    mode: 'quick',
    features: quickResult
  };
}

These features support complex use cases while keeping detection fast.