📊Data Scientist

Supervised learning trains on labeled examples to predict known targets, covering classification and regression. Unsupervised learning finds structure in unlabeled data through clustering, dimensionality reduction and anomaly detection. A third family, reinforcement learning, learns from reward signals. Spam detection is supervised while customer segmentation is typically unsupervised.

Overfitting is when a model memorizes training noise and generalizes poorly, shown by training metrics far better than validation. Prevent it with more representative data, cross-validation, simpler models, L1 and L2 regularization, dropout, early stopping and careful feature selection. Always compare train versus validation curves rather than training score alone.

Bias is error from overly simple assumptions (underfitting) while variance is error from sensitivity to training noise (overfitting). Total error decomposes into bias plus variance plus irreducible noise. Increasing capacity lowers bias but raises variance and vice versa, so tune complexity via validation curves, regularization and ensembling to find the sweet spot.

As feature dimensions grow, data becomes sparse: distances concentrate, neighborhoods empty out, and models need exponentially more samples to generalize. Mitigate with feature selection, PCA or embeddings, regularization and domain-driven reduction. Tree models and strong regularization cope better than kNN or unregularized linear models in high dimensions.

L1 (lasso) penalizes absolute weights and drives some exactly to zero, performing feature selection and yielding sparse models. L2 (ridge) penalizes squared weights, shrinking them smoothly without zeroing, which handles correlated features better. Elastic net combines both. Choose L1 for sparsity and L2 for stability, tuning strength via cross-validation.

Go beyond accuracy, especially with imbalance: use precision, recall, F1, ROC-AUC and PR-AUC from a confusion matrix, plus calibration and business-cost-weighted thresholds. Evaluate on a held-out set with stratified cross-validation, slice by cohort to catch hidden failures, and track inference latency and stability alongside offline scores.

Precision: the share of predicted positives that are truly positive, penalizing false alarms, which matters for spam or fraud flags. Recall: the share of actual positives the model finds, penalizing misses, which matters for disease screening, so tune thresholds and report F1 plus PR curves for imbalance.

Boosting builds shallow trees sequentially where each tree corrects prior residuals with shrinkage, subsampling and regularization on depth, learning rate and leaf weights. Tune with early stopping on validation, handle missing values natively, and calibrate outputs when probabilities matter. They dominate tabular data with strong accuracy and interpretability trade-offs.

K-means assigns points to K centroids by nearest distance, recomputes centroids as means, and iterates to reduce within-cluster variance. Choose K with elbow, silhouette or business sense, scale features first, and restart multiple times to avoid local minima. It assumes spherical clusters and struggles with varied densities.

Random forests bag many trees trained on bootstrap samples with random feature subsets, then vote or average to cut variance while keeping low bias. They resist overfitting better than single trees, handle mixed types and missingness, and give impurity or permutation importance. Tune tree count, depth and features per split with out-of-bag validation.

Transformers use self-attention to weigh all token relationships in parallel with positional signals, stacking encoder or decoder layers for classification, tagging or generation. Fine-tune pretrained models on labeled data with modest learning rates, chunk long docs to context limits, and evaluate on held-out slices plus robustness probes. They beat bag-of-words by modeling order and context.

Logistic regression models binary odds with a linear score passed through the sigmoid to output probabilities. It trains by maximum likelihood, yields interpretable coefficients as log odds, and needs scaled or encoded features plus regularization for stability. Threshold outputs by business cost and check calibration before use.

Combine domain judgment with filters like mutual information, wrappers like recursive elimination, and embedded methods like L1 or tree importance, validated by cross-validated lift. Remove leaky, duplicate and unstable features, check train-serve parity, and favor simpler sets with equal performance. Document why each kept feature belongs.

SVMs find the max-margin hyperplane separating classes, using kernels like RBF to lift non-linear data into separable spaces. Tune C for margin versus errors plus kernel width, scale features first, and prefer them for small high-dimensional sets like text. They predict well but scale poorly to massive data.

Version data, code and artifacts, validate with shadow or canary releases plus rollback gates, serve via batch or low-latency endpoints with autoscaling, and monitor features, predictions and business metrics for drift. Automate retraining with tests for schema and quality, and log lineage for audit. Document fallback rules for model failures.

Trees split data greedily by impurity gains like Gini or entropy, growing leaves for prediction with clear rules. Prune with max depth, min samples and cost-complexity tuning validated by cross-validation to stop memorizing noise. They interpret well but overfit alone, so ensembles often follow.

Search with random or Bayesian optimization over meaningful ranges using cross-validation, starting coarse then refining around winners with early stopping. Track trials with seeds and budgets, separate validation from final test, and favor robust flat regions over sharp peaks. Automate with tools like Optuna and log everything.

Word2Vec learns static vectors from co-occurrence so similar words sit close, cheap and useful for baselines but blind to polysemy. BERT-style models give contextual embeddings per sentence via transformers, capturing meaning shifts at higher compute cost. Fine-tune contextual models for accuracy and distill for serving.

Naive Bayes applies Bayes theorem with a strong conditional-independence assumption, counting feature likelihoods per class for fast text and spam baselines. It trains on little data, handles high dimensions well, and outputs calibrated-ish scores with smoothing for unseen words. It falters when features correlate strongly.

SHAP attributes predictions with Shapley values for consistent global and local importance, while LIME fits local surrogate models around individual predictions for intuitive what-if views. Validate explanations with ablations, check stability across seeds, and pair with partial dependence for direction. Never present them as causal proof.

DBSCAN grows clusters from core points with enough neighbors inside radius epsilon, marking sparse points as noise without needing K upfront. It finds arbitrary shapes and flags outliers naturally but struggles with varied densities and needs scaled distances plus tuned epsilon. Validate with silhouette on core points and domain review.

Leakage happens when training sees future or target-derived info like post-event fields, IDs or imputation from full data. Prevent with temporal splits, point-in-time feature joins, pipelines that fit transforms on train folds only, and audits of top features for impossibility. Red-team with date-shifted validation.

User-based methods find similar users and suggest what they liked, item-based methods recommend items similar to past likes via co-ratings, and matrix factorization learns latent factors for both. Handle cold starts with content or popularity fallbacks, evaluate with ranking metrics offline plus online lifts, and refresh for drift.

KNN classifies or regresses by voting among the K closest training points under a distance like Euclidean, making it simple and non-parametric. Scale features, pick K by cross-validation, and use indexes for speed since inference scans neighbors. It struggles in high dimensions and with imbalanced sparse data.

Track input distributions with PSI or KS tests, missing rates and prediction shifts plus accuracy on labeled samples and business KPIs. Alert on joint drift, slice by cohort, and trigger retrain or rollback playbooks with shadow validation. Log features and versions so incidents replay exactly.

Detect with VIFs, correlation heatmaps and unstable coefficients that flip signs, then drop or combine proxies, gather more data, or regularize with ridge or elastic net. Prefer PCA for pure prediction and domain-guided selection for inference. Report how choices change key estimates.

Use internal metrics like silhouette, Davies-Bouldin and stability across seeds plus human review of samples and downstream lift in A/B tests. Compare against baselines and business segments for actionability, not just scores. Visualize with UMAP-style projections cautiously since they distort distances.

Grade relevance per position with discounts for lower ranks: NDCG rewards highly relevant items early with graded gains, MAP averages precision over relevant recalls for binary judgments. Collect unbiased labels, handle position bias with randomization or IPS, and complement offline scores with interleaving plus online lifts.

Use batch for high-throughput non-urgent scoring like nightly churn with cheap compute and easy QA, and online for millisecond personalization or fraud with autoscaled endpoints and fallbacks. Decide from latency SLAs, cost per prediction and freshness needs, often mixing both with a feature store. Load-test p99 before launch.

A Type I error (false positive) rejects a true null hypothesis and is controlled by significance level alpha. A Type II error (false negative) misses a real effect, with power equal to one minus the Type II rate. Illustrate with medical tests or spam filters, where the business cost of each error type decides thresholds and sample sizes.

A/B testing randomly splits users into control and treatment to measure causal lift on a metric. Size the sample with power analysis using baseline rate, minimum detectable effect, significance alpha and power (usually 80 percent), then run for full weekly cycles to avoid novelty and day-of-week effects. Mention guardrail metrics, peeking corrections with sequential methods, and segment analysis.

The Central Limit Theorem says sample means of i.i.d. variables approach a normal distribution as sample size grows, regardless of the underlying shape. This justifies confidence intervals and many hypothesis tests on means. Mention the n-greater-than-30 rule of thumb, skew caveats, and its role in A/B test analysis.

A p-value is the probability of seeing results this extreme if the null hypothesis were true, with significance declared when it falls below alpha like 0.05. It is not the probability the hypothesis is true, so pair it with effect sizes, confidence intervals and pre-registered plans. Guard against peeking and multiple testing inflation.

Use the mean for symmetric numeric data, the median for skewed data with outliers like income, and the mode for categorical or discrete peaks like most common plan. Report spread alongside center and visualize distributions before choosing. In interviews, note how outliers pull the mean but not the median.

Frequentist: treats parameters as fixed and quantifies long-run error rates with p-values plus confidence intervals, without prior beliefs. Bayesian: treats parameters as distributions updated from priors to posteriors via Bayes theorem, giving direct probability statements and natural sequential updating at the cost of prior choice and computation.

Randomize units into control and treatment with blocking or stratification on key covariates, pre-register primary metrics plus guardrails, size with power analysis, and run full cycles to cover weekly effects. Isolate interference with geo or cluster designs when spillover exists. Document exclusions and analyze by intent to treat.

Correlation: two variables move together statistically, measured by coefficients, without proving one causes the other. Causation: a change in one variable produces change in another, established through randomization, natural experiments or strong causal designs that rule out confounders and reverse causality.

Testing many hypotheses inflates false positives, so control family-wise error with Bonferroni or Holm for strict gates, or false discovery rate with Benjamini-Hochberg for exploratory screens. Pre-register primary endpoints, limit peeks, and report both raw and adjusted values. Choose the method from the cost of a false claim.

A 95 percent confidence interval is a range procedure that covers the true parameter in 95 percent of repeated samples, not a 95 percent chance the truth lies inside one realized interval. Width reflects sample size and variance, so pair with point estimates and assumptions checks. Prefer intervals over p-values alone for decisions.

Variance: the average squared deviation from the mean, in squared units, useful for math like bias-variance decomposition. Standard deviation: the square root of variance in original units, useful for intuitive spread, confidence bands and outlier rules.

Use sequential methods like group-sequential boundaries, always-valid p-values or Bayesian stopping rules with pre-set decision criteria instead of daily significance checks. Pre-register looks, metrics and minimum runtime, and correct for early stopping in estimates. Report the stopping rule alongside results.

Simpsons paradox occurs when an aggregate trend reverses within subgroups, like a treatment looking better overall but worse in every hospital due to case-mix differences. Always slice by key segments, weight properly, and model confounders instead of trusting pooled rates. It teaches why segmentation precedes conclusions.

The normal bell curve is symmetric with mean, median and mode equal, governed by mean and standard deviation with known tail probabilities. It models many natural sums via the Central Limit Theorem and underpins z-tests plus intervals. Check normality with plots and tests, transforming or switching methods when skew is strong.

Compare treatment versus control changes before and after an intervention, assuming parallel trends without it. Validate with pre-trend plots, placebo time windows and robust standard errors, and control for composition shifts. It suits policy or geo rollouts where randomization is impossible.

Bootstrap resamples observed data with replacement thousands of times to approximate sampling distributions for means, medians or model metrics without formulas. Derive percentile or BCa intervals, estimate stability, and validate when theory is messy. It needs representative samples and fails for tiny or dependent data without block variants.

Decompose trend, seasonality and holidays, test stationarity with differencing for ARIMA or use additive Prophet-style regressors for business calendars and changepoints. Validate with rolling-origin backtests, horizon-specific errors and residual checks, not random splits. Include exogenous drivers like promos and monitor drift after launch.

Sampling bias skews results when some groups are over- or under-represented versus the target, like app-only surveys missing offline users. Prevent with clear population definitions, random or stratified sampling, weighting adjustments, and nonresponse analysis. Always compare sample demographics to ground truth.

Power analysis sizes samples from baseline rates, minimum detectable effect, alpha and desired power like 80 percent plus variance and allocation ratios. Smaller effects need much larger samples, so negotiate meaningful MDEs upfront and plan runtime in full weeks. Re-estimate variance from pilot data before launch.

Use rank-based tests like Mann-Whitney for two groups, Kruskal-Wallis for many, Wilcoxon signed-rank for pairs, and Spearman for monotonic links when normality fails. They resist outliers and skew but give less power and vaguer effect sizes, so report medians plus bootstrap intervals. Check assumptions with plots first.

Assume linearity, independent errors, homoscedasticity, normal residuals and low multicollinearity. Check residual plots, QQ plots, VIFs and Durbin-Watson style tests, then fix with transforms, robust errors or regularization. Report coefficient intervals and validate on held-out data, not just R squared.

Conditional probability P of A given B restricts outcomes to B cases, and Bayes theorem flips it using priors and likelihoods to update beliefs with evidence. Apply to spam filters and medical tests where base rates matter, and avoid ignoring priors. Illustrate with a worked test-accuracy example.

Interactions let one variable change another slope, like ads working better for new users, modeled with product terms and visualized with marginal plots. Center inputs to aid interpretation, test joint significance, and avoid overfitting with many terms. Report subgroup effects with intervals, not just p-values.

Survival analysis models time-to-event with censoring, using Kaplan-Meier curves for group survival, log-rank tests for differences, and Cox models for covariate hazards. Handle right censoring, check proportional-hazards assumptions, and report median survival plus hazard ratios. It suits churn, failure and trial outcomes.

First diagnose the mechanism (MCAR, MAR, MNAR) and the share of missingness. Options include dropping rows or columns when minimal, mean, median, mode or model-based imputation, adding missingness-indicator features, and algorithms that handle NaN natively. Never silently fill with zeros; validate that the strategy does not leak future information or bias key segments.

Profile missingness, duplicates, mixed types and outliers, then standardize formats, parse dates, fix encodings and validate ranges with logged rules. Keep raw snapshots, make cleaning idempotent and tested, and document every assumption for audit. Validate downstream metrics before and after cleaning.

Resample with stratified splits, class weights or SMOTE-style augmentation, choose PR-AUC and F1 over accuracy, and tune thresholds to business costs. Try anomaly detection framings for extreme rarity and validate on time-based splits to avoid leakage. Always report confusion matrices sliced by cohort.

ETL: extracts, transforms before loading into a warehouse, enforcing schemas early for governed curated marts. ELT: loads raw data into a lake or cloud warehouse first then transforms with scalable SQL, favoring flexibility, replay and schema-on-read for fast iteration.

Define versioned transformations in a feature store with point-in-time correct joins to prevent leakage, backfill reproducibly, and serve consistent offline and online values. Monitor freshness, distributions and missing rates with ownership per feature. Test pipelines with data contracts and unit fixtures.

Flag with z-scores, IQR fences, isolation forests or domain rules, then inspect whether outliers are errors, rare truths or leakage. Winsorize, transform or model robustly rather than blindly deleting, and sensitivity-test results with and without them. Log every removal decision for review.

Window functions compute ranks, running totals, moving averages and lag deltas with OVER plus PARTITION BY and ORDER BY without collapsing rows. Use ROW_NUMBER for deduping, LAG for period changes, and frame clauses for rolling windows. They replace expensive self-joins and keep grain intact for downstream models.

Publish events to Kafka topics with schemas and retention, process with Spark Structured Streaming using watermarks and checkpointing for late data, and sink idempotent upserts to lakes or stores. Monitor consumer lag, state size and exactly-once limits, and backfill by replaying topics. Separate real-time and batch paths with clear SLAs.

Filter early, project only needed columns, broadcast small tables, partition on join keys to avoid shuffles, and use bucketed or sorted layouts for repeated joins. Skew hot keys by salting, validate row-count fan-out, and prefer columnar formats with predicate pushdown. Profile join order with EXPLAIN plans.

CSV: plain-text rows with commas, human readable but bulky, slow to scan and weakly typed. Parquet: columnar binary format with compression, predicate pushdown and rich types, far faster and cheaper for analytics at the cost of needing Spark or warehouse readers instead of any text editor.

Enforce schema, range, uniqueness and freshness checks with Great Expectations style suites, quarantine bad batches with alerts, and track quality SLIs over time. Own rules per source, version thresholds, and block promotion on failures. Review violations jointly with upstream producers.

Type 1 overwrites history, Type 2 adds versioned rows with effective dates and current flags to preserve truth over time, and Type 3 keeps limited previous values. Choose Type 2 for audits and trend analysis, join facts on surrogate keys with date logic, and backfill carefully. Document retention for each dimension.

Normalization: rescales features to a bounded range like zero to one with min-max, preserving shape but sensitive to outliers, good for neural nets and distances with known bounds. Standardization: centers to zero mean with unit variance via z-scores, robust for linear models and PCA when scales differ and outliers exist.

Block candidates with keys like email or name plus location, score pairs with fuzzy rules or models, cluster matches, and survivor-merge with lineage. Validate precision and recall on labeled pairs, process incrementally with windowed state, and keep audit trails for merges. Tune thresholds to business cost of false merges.

Cut shuffles with partitioning, bucketing and broadcast joins, cache only reused frames with proper storage levels, and use columnar Parquet with pushdown plus adaptive query execution. Tune executor memory, cores and shuffle partitions from stage metrics, and skew-split hot keys. Profile the Spark UI before guessing.

Extract hour, weekday, month, holiday and elapsed-since-event parts plus cyclical sine-cosine encodings, tenure and recency gaps. Avoid leakage by computing only from past data at prediction time with point-in-time joins. Validate lift by ablation and watch for holiday drift yearly.

Star schema: central fact tables joined to denormalized dimension tables for fast simple analytics queries with some redundancy. Normalized schema: splits data into many linked tables to remove redundancy for integrity and writes, at the cost of complex JOINs and slower BI queries.

Clean HTML and boilerplate, normalize case and unicode, tokenize with domain-aware rules, remove or keep stopwords per task, then lemmatize and build vocabularies with min frequencies plus subwords for OOV. Split before fitting vectorizers to avoid leakage and benchmark against simple TF-IDF baselines. Track preprocessing versions with data.

Pin code, data versions, seeds and environments with containers plus lockfiles, orchestrate pipelines with versioned configs, and log params, metrics and artifacts in a registry. Review with deterministic checks and one-command rebuilds from raw to report. Treat notebooks as drafts and pipelines as truth.

Define grain and dedupe keys first, use distinct counts on user ids, attribute shared events with clear rules, and reconcile with funnel totals. Pre-aggregate in idempotent jobs with watermarks and test edge cases like refunds and retries. Document definitions so product and finance match.

Offline features are batch-computed point-in-time tables for training with replay, while online features are low-latency key-value snapshots for live inference that must match training logic. Keep one definition compiled to both paths, monitor skew between them, and backfill carefully. Flag staleness budgets per feature.

Check shape, types, missing rates, cardinality, duplicates and basic distributions plus correlations and target leakage hints, all in a notebook with reusable functions. Interview stakeholders for definitions, flag anomalies early, and write a one-page data brief before modeling. Timebox to avoid rabbit holes.