Pedestrian-Counting-System-Melbourne-preprocessed

Pedestrian Counting System published by the city of Melbourne, preprocessed data.

From original source:
-----
This dataset contains hourly pedestrian counts since 2009 from pedestrian sensor devices located across the city. The data is updated on a monthly basis and can be used to determine variations in pedestrian activity throughout the day.
The sensor_id column can be used to merge the data with the Pedestrian Counting System - Sensor Locations dataset which details the location, status and directional readings of sensors. Any changes to sensor locations are important to consider when analysing and interpreting pedestrian counts over time.
Importants notes about this dataset:

- Where no pedestrians have passed underneath a sensor during an hour, a count of zero will be shown for the sensor for that hour.

- Directional readings are not included, though we hope to make this available later in the year. Directional readings are provided in the Pedestrian Counting System - Past Hour (counts per minute) dataset.
The Pedestrian Counting System helps to understand how people use different city locations at different times of day to better inform decision-making and plan for the future. A representation of pedestrian volume which compares each location on any given day and time can be found in our Online Visualisation.
-----

We have acquired the data by scrapping the website 'https://www.pedestrian.melbourne.vic.gov.au/#date=1-6-2010&time=8', as the data seems to be incomplete
from https://data.melbourne.vic.gov.au/explore/dataset/pedestrian-counting-system-monthly-counts-per-hour/information/.

There are 54 columns:

id_series: The id of the time series.

date: The date of the time series in the format "%Y-%m-%d %H:%M:$S".

time_step: The time step on the time series.

value_X (X from 0 to 50): The values of the time series, which will be used for the forecasting task.

Preprocessing:

1 - Standardize the date to the format %Y-%m-%d %H:%M:%S

2 - Replaced values 'na' and 'undefined' to NaNs and casted value columns to float.

Even though the values are 'int', we use float to accomodate NaN values.

3 - Coalesced columns that we judged that were the same in a single column.

The columns that we judged that were the same were the tuple:

('Bourke St-Russel St (West)', 'Bourke St-Russell St (West)'),
('Flinders La - Swanston St (West) Temporary', 'Flinders La-Swanston St (West) Temporary'),
('Flinders Ln - Degraves St (Crossing)', 'Flinders Ln -Degraves St (Crossing)'),
('Flinders Ln - Degraves St (North)', 'Flinders Ln -Degraves St (North)'),
('Flinders Ln - Degraves St (South)', 'Flinders Ln -Degraves St (South)'),
('Flinders St - ACMI', 'Flinders St- ACMI'),
('Flinders St Station Underpass', 'Flinders Street Station Underpass'),
('Flinders St-Spark La', 'Flinders St-Spark Lane'),
('Lincoln-Swanston (W)', 'Lincoln-Swanston (West)'),
('Macaulay Rd - Bellair St', 'Macaulay Rd-Bellair St'),
('QV Market-Elizabeth (West)', 'QV Market-Elizabeth St (West)'),
('Queen St (West)', 'Queen Street (West)'),
('Spring St - Flinders St (West)', 'Spring St- Flinders St (West)'),
('State Library - New', 'State Library- New'),
('St Kilda Rd-Alexandra Gardens', 'St. Kilda-Alexandra Gardens')

4 - Replaced negative values with NaNs.

5 - Dropped columns with the last non NaN value before 2024.

6 - Dropped columns with the first non NaN value after 2018.

7 - Select the data between the years 2019 and 2023.

8 - Renamed value columns to 'value_X' where X is between 0 and 50.

9 - Created column 'id_series' with value 0, there is only one (multivariate) series, and column 'time_step' with increasing values of the time_step.

10 - Casted 'date' to str, 'time_step' to int, 'value_X' to float, and defined 'id_series' as 'category'.