Classification with K-Nearest Neighbors

K-Nearest Neighbors (KNN) is one of the simplest and most intuitive machine learning algorithms for classification tasks.

It predicts the class of a new data point by looking at the majority label among its nearest neighbors in the training data.

1. Store the entire training dataset.
2. For a new data point:
   • Calculate the distance to all training samples (commonly Euclidean distance).
   • Select the k closest neighbors.
   • Assign the most common class among those neighbors.

Let’s use Scikit-learn to apply KNN classification to the classic Iris dataset.

Copy
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import classification_report, accuracy_score

# Load dataset
iris = load_iris()
X, y = iris.data, iris.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Create KNN model
knn = KNeighborsClassifier(n_neighbors=3)

# Train the model
knn.fit(X_train, y_train)

# Predict
y_pred = knn.predict(X_test)

# Evaluate
print(f"Accuracy: {accuracy_score(y_test, y_pred):.2f}")
print("Classification Report:", classification_report(y_test, y_pred))

• Small k → captures local patterns but may be sensitive to noise.
• Large k → produces smoother decision boundaries but may underfit.

A good approach is to try different k-values and choose the one that gives the best validation accuracy.

• KNN is non-parametric — it doesn’t assume an underlying data distribution and requires no explicit model training.
• Performs well on small to medium datasets, but can be computationally expensive for large ones.
• Always scale your features — KNN relies on distance, so unscaled data can distort results.