特征工程是将原始数据转换为更好表达问题本质的过程。特征工程决定了模型的上限，而算法只是逼近这个上限。

一、特征工程概述

特征工程包括特征构造、特征提取、特征选择三个方面。

二、特征构造

特征构造是根据领域知识创建新特征。

代码示例：特征构造

import numpy as np
import pandas as pd
from sklearn.preprocessing import PolynomialFeatures
import matplotlib.pyplot as plt
from sklearn.decomposition import PCA
from sklearn.feature_selection import SelectKBest, mutual_info_regression
from sklearn.ensemble import RandomForestRegressor
from sklearn.preprocessing import StandardScaler
import warnings
warnings.filterwarnings("ignore")

print("=" * 60)
print("第6篇：特征工程")
print("=" * 60)

# 模拟房价预测数据集
np.random.seed(42)
n_samples = 1000

data = {
    "area": np.random.uniform(50, 200, n_samples),
    "rooms": np.random.randint(1, 6, n_samples),
    "age": np.random.uniform(0, 30, n_samples),
    "subway": np.random.uniform(0.5, 10, n_samples),
    "schools": np.random.randint(0, 5, n_samples)
}

df = pd.DataFrame(data)

# 目标变量
df["price"] = (df["area"] * 3 + df["rooms"] * 5 +
               df["subway"] * (-3) + df["schools"] * 10 -
               df["age"] * 0.5 + np.random.randn(n_samples) * 10)

print("原始数据:")
print(df.head())

# 1. 多项式特征
print("\n" + "-" * 50)
print("1. 多项式特征构造")
print("-" * 50)

X = df[["area", "rooms"]].values
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)

print(f"原始特征数: {X.shape[1]}")
print(f"多项式特征数: {X_poly.shape[1]}")

# 2. 交互特征
print("\n" + "-" * 50)
print("2. 交互特征构造")
print("-" * 50)

df["area_per_room"] = df["area"] / (df["rooms"] + 1)
df["age_subway"] = df["age"] * df["subway"]
df["score"] = df["schools"] - df["subway"] * 0.5

print("新增交互特征:")
print(df[["area_per_room", "age_subway", "score"]].head())

# 3. 领域特征
print("\n" + "-" * 50)
print("3. 领域特定特征")
print("-" * 50)

df["is_new"] = (df["age"] < 3).astype(int)
df["near_subway"] = (df["subway"] < 1).astype(int)
df["good_school"] = (df["schools"] >= 3).astype(int)

print(df[["is_new", "near_subway", "good_school"]].head())

三、特征提取与降维

当特征数量非常多时，可以使用PCA进行降维。

代码示例：PCA降维

print("\n" + "=" * 50)
print("主成分分析（PCA）降维")
print("=" * 50)

feature_cols = ["area", "rooms", "age", "subway", "schools",
                "area_per_room", "age_subway", "score"]
X = df[feature_cols].values

# 标准化
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# PCA
pca = PCA()
X_pca = pca.fit_transform(X_scaled)

# 方差解释比例
print("各主成分的方差解释比例:")
for i, ratio in enumerate(pca.explained_variance_ratio_[:5]):
    print(f"  PC{i+1}: {ratio:.4f} ({ratio*100:.2f}%)")

print(f"\n前3个主成分累计解释方差: {sum(pca.explained_variance_ratio_[:3])*100:.2f}%")

# 可视化
fig, axes = plt.subplots(1, 2, figsize=(14, 5))

axes[0].bar(range(1, len(pca.explained_variance_ratio_) + 1),
            pca.explained_variance_ratio_, alpha=0.7)
axes[0].plot(range(1, len(pca.explained_variance_ratio_) + 1),
             np.cumsum(pca.explained_variance_ratio_), "ro-")
axes[0].set_xlabel("Principal Component")
axes[0].set_ylabel("Variance Ratio")
axes[0].set_title("PCA Variance Explained")
axes[0].legend(["Cumulative", "Individual"])

scatter = axes[1].scatter(X_pca[:, 0], X_pca[:, 1], c=df["price"], cmap="viridis", alpha=0.6)
axes[1].set_xlabel("PC1")
axes[1].set_ylabel("PC2")
axes[1].set_title("PCA Visualization")
plt.colorbar(scatter, ax=axes[1], label="Price")

plt.tight_layout()
plt.savefig("pca_analysis.png", dpi=150)
plt.show()

四、特征选择

特征选择是从众多特征中选择最相关的子集。

代码示例：特征选择

print("\n" + "=" * 50)
print("特征选择方法")
print("=" * 50)

y = df["price"].values

# 1. 相关性分析
print("\n1. 特征与目标变量的相关性:")
correlations = df.corr()["price"].drop("price").sort_values(key=abs, ascending=False)
print(correlations.round(4))

# 2. SelectKBest
selector_f = SelectKBest(k=5)
X_f = selector_f.fit_transform(X_scaled, y)
selected_features = [feature_cols[i] for i in selector_f.get_support(indices=True)]

print(f"\n2. SelectKBest选择的特征: {selected_features}")

# 3. 互信息分析
mi_scores = mutual_info_regression(X_scaled, y)
mi_df = pd.DataFrame({"feature": feature_cols, "mi_score": mi_scores})
mi_df = mi_df.sort_values("mi_score", ascending=False)

print(f"\n3. 互信息得分:")
for _, row in mi_df.iterrows():
    print(f"  {row['feature']}: {row['mi_score']:.4f}")

# 4. 随机森林特征重要性
print("\n4. 随机森林特征重要性:")
rf = RandomForestRegressor(n_estimators=100, random_state=42)
rf.fit(X_scaled, y)

for feat, imp in sorted(zip(feature_cols, rf.feature_importances_), key=lambda x: x[1], reverse=True):
    bar = "#" * int(imp * 50)
    print(f"  {feat:15s}: {imp:.4f} {bar}")

# 可视化
fig, ax = plt.subplots(figsize=(10, 6))
ax.barh(mi_df["feature"], mi_df["mi_score"], color="steelblue")
ax.set_xlabel("Mutual Information Score")
ax.set_title("Feature Importance (Mutual Information)")
plt.tight_layout()
plt.savefig("feature_selection.png", dpi=150)
plt.show()

五、总结

• 特征工程是机器学习成功的关键
• 特征构造：多项式特征、交互特征、领域知识特征
• 特征提取：PCA降维可以减少维度
• 特征选择：过滤法、嵌入法各有优缺点
• 好的特征应具备可区分性、简洁性