After data sanity checks and data analysis ,add new features and add the new cleaned dataset

[venkatesh21bit]

[venkatesh21bit]

Create Route-Based Features
This is the single most impactful thing you can do. Instead of treating origin_port and destination_port as separate entities, combine them to create a route feature. Then, you can calculate statistics for each route.

Action: Create a new column route = df['Origin_Port'] + '_' + df['Destination_Port'].

New Features to Create:

route_avg_delay: The historical average delay for that specific route.

route_reliability_score: The historical on-time delivery percentage for that route.

route_shipment_count: The total number of shipments on that route (a measure of traffic).

[venkatesh21bit]

Engineer Deeper Time-Based Features
Go beyond just the date. Extract cyclical patterns that affect logistics.

Action: From your Shipping_Date column, create new features.

New Features to Create:

shipping_month: (1-12)

shipping_quarter: (1-4)

is_month_end: (Boolean: True if the day is > 25, False otherwise)

is_quarter_end: (Boolean)

[venkatesh21bit]

Develop Interaction Features
Create features that capture the interaction between two different variables.

Action: Combine categorical features to create new, more specific ones.

New Features to Create:

carrier_on_mode: (e.g., "CarrierA_Air", "CarrierA_Sea"). This captures that some carriers are better at specific transportation modes.

route_and_carrier: This will show the performance of a specific carrier on a specific route.

[venkatesh21bit]

Scale Numerical Features
Use StandardScaler from scikit-learn on your numerical features (Costs, Weight_kg, etc.) to normalize their range, which helps many models perform better.
Wrap Everything in a Pipeline
Combine all your preprocessing steps (Target Encoding, Scaling) into a single sklearn.pipeline.Pipeline object.