spatio-temporal modelingforecast metro train delays in and around nyc and new jersey

methodology 

data collection 

• train trip data: granular performance data from over 667,000 nj transit and amtrak train trips during october to december 2019 (pre-pandemic period), covering busy travel periods like thanksgiving and christmas. we sourced this dataset from pranav badami’s nj transit and amtrak performance records.
• station data: latitude and longitude details for nj transit and amtrak stations were merged with the train trip dataset to create a geospatial data frame (trip.sf). this allowed us to analyze spatial patterns and integrate station-level information into the model.
• census data: demographic and transportation data from the nyc and nj census were included. we used this data to analyze commuting patterns, population density, and transportation usage. the spatial information was utilized for mapping and joining purposes, creating a geospatial object (sf) that includes geometry.
• transportation data: infrastructure statistics from the bureau of transportation statistics, including metrics like the percentage of bridges in poor condition and the proportion of residents commuting by public transit. these data points were extracted from the national transportation atlas database (ntad), incorporating federal and state-level transportation information.
• weather data: real-time weather conditions for both new york city and newark, nj, were analyzed. we used riem_measures to import weather data from the newark liberty international airport. the weather dataset (njweather.panel) includes hourly measures of temperature, precipitation, wind speed, and visibility, covering the october to december period.

data wrangling and preparation

1. train trips data: we selected data from october to december 2019, a period representing typical pre-pandemic train operations, including holiday traffic. this dataset required extensive cleaning due to inconsistencies in temporal data. we sorted the records by 60-minute intervals to standardize the time series.
             
2. train stations data: the station dataset was integrated into the main train trips dataset by joining latitude and longitude information. this allowed us to create a geospatial data frame (trip.sf), providing accurate mapping and spatial analysis of train movements.
             

3. census data integration: we imported census data for both new york and new jersey, focusing on variables related to transportation, such as the number of commuters and the total use of public transit. this data was used to assign origin-destination pairs to the train trips, enriching the dataset with demographic context. the census tracts were extracted as geospatial objects for mapping and analysis.
4. transportation data analysis: sourced from the ntad, this dataset includes information on bridge conditions and commuting trends. it contains 22 columns with variables like the percentage of poor-condition bridges and the proportion of workers commuting within the county. this data helped link infrastructure quality to delay patterns in our analysis.            
5. weather data processing: we divided the weather data into two panels:
• nyc weather panel: focused on weather conditions at new york city destination stations, accounting for factors like fog, wind gusts, and precipitation that may influence train delays.
• nj weather panel: used data from the newark liberty international airport, capturing hourly weather variables such as temperature, wind speed, and precipitation for the new jersey area. this panel dataset was crucial for identifying correlations between adverse weather conditions and train delays.

feature engineering

• station lags: delays at one station often ripple through to subsequent stops, making it challenging for trains to regain punctuality once delayed. to account for this, we introduced station lag features, which measure the influence of delays at preceding stations. we sorted the train data by stop sequence and calculated delay minutes for the three previous stations. this “3-lag range” captures the impact of upstream delays on subsequent stops, enhancing the model's ability to predict cascading disruptions.
  
• 
  

• time lags: train delays exhibit strong temporal dependencies, particularly during peak travel times. we engineered time lag features to capture serial autocorrelation in delay patterns. the dataset was grouped by 60-minute intervals, and we created lag variables that represent delay minutes from previous time periods (e.g., lag 1 hour, lag 2 hours, lag 3 hours). these time lags help the model detect recurring delay patterns over time.
  
• 
  
• panel: after creating station and time lags, we merged these features into a comprehensive final.panel dataset. this panel includes station lags, time lags, train trip data, census data, and weather information, forming a robust framework for our predictive model.
  
• 
  
• serial autocoorelation analysis: our analysis revealed a positive linear relationship between delay times and time lag features. visual inspection of the plots confirmed that delays tend to persist over time, indicating strong temporal autocorrelation. similarly, station lag features showed a clear linear correlation, particularly highlighting the significant influence of delays at previous stations on the next stop’s punctuality.
• 
  

• weather correlation: extreme weather conditions can severely impact train schedules, leading to widespread delays. we examined the relationship between various weather factors (e.g., temperature, wind speed, precipitation, visibility, gusts, and ice accretion) and delay times. the analysis indicated that high wind speeds and heavy precipitation at departure stations in new jersey are strongly correlated with longer delay minutes. incorporating weather data as features improved the model’s ability to predict delays under adverse conditions.
  

• correlation matrix: to identify and remove redundant features, we visualized the relationships between variables using a correlation matrix. strong correlations were observed between certain demographic and transportation features. for example, the number of resident workers who work from home was closely related to the number of resident workers commuting within the county. based on these findings, we excluded highly correlated variables like total_pop and number.of.resident.workers.who.work.at.home to reduce multicollinearity and enhance the model’s efficiency.
  
• 
  

predictive modeling: regression 

evaluation and model performance

• mae: the inclusion of station lags resulted in a 50% reduction in mae, highlighting their importance in delay prediction.
• residual analysis: errors were higher for specific stations (e.g., absecon on the atlantic city line) due to potential issues like outdated infrastructure.
• cross-validation: conducted using weekly partitions, confirming the model’s generalizability across different time periods.

conclusion

key takeaways:

• high predictive accuracy: the model effectively forecasts short delays, aiding passengers in making informed travel decisions.
• spatial error patterns: observed correlations point to specific infrastructure issues, providing actionable insights for targeted improvements.
• strong generalizability: cross-validation demonstrated consistent model performance across various time periods, supporting its application in real-time scenarios.

improvement

1. simplifying predictors: currently, the model uses over 20 features. applying methods like decision trees or lasso regression could streamline the feature set, reducing complexity while maintaining accuracy.
2. addressing potential overfitting: despite the model’s good fit, we are cautious of overfitting due to the extensive feature set. further regularization techniques may help ensure robust performance.
3. enhancing accuracy for specific lines and stations: the model's performance was weaker for certain lines (e.g., atlantic city line) and stations (e.g., absecon). incorporating additional spatial features or alternative data sources may help mitigate these issues.
4. handling extreme delays: the model struggled with outliers involving severe delays. exploring advanced techniques like gradient boosting or deep learning could improve predictions for these rare but impactful events.

reference