====== Connecting Malaysia's Transport Data APIs to OpenTripPlanner and OneBusAway ======

===== Problem Statement =====

I recently moved to Kuala Lumpur, Malaysia and rely on public transportation as part of my daily commute.

==== Wayfinding ====

The Klang Valley has an extensive public transportation network. However, its planning is constrained physically and financially by the valley's existing road network sprawl, undulating topography and [[https://civil.utm.my/wp-content/uploads/2016/12/Land-Acquisition-for-Klang-Valley-Mass-Rapid-Transit-Line-1-Project.pdf|tedious land acquisition]], resulting in a large number of [[https://myrapid.com.my/bus-train/rapid-kl/rapid-kl-integrated-transit-map/|interchanges between rail lines]] and transit modes that are not immediately obvious. Often, the best route or easiest transfer is not always at a designated interchange or hub). Thus, **wayfinding** is a problem I face.

==== Realtime Information ====

The Klang Valley has [[https://www.planningmalaysia.org/index.php/pmj/article/download/Article%201-8/36|extensive suburban sprawl]], so the last-mile journey relies on the  [[https://myrapid.com.my/bus-train/rapid-kl/bus/|bus network]] (other than car ownership - park and ride). However, that sprawl, coupled with Klang Valley's notorious traffic congestion, makes it expensive and challenging (from my experience as a public bus captain, I would say, impossible) to maintain a consistent and short service interval on the bus network.

Short of public transportation reform, I want to make best use of the level of service that is already there.

For example, the loop service (service 650) that serves my taman (estate, literally "park") runs at a planned frequency of between 20 to 40 minutes, but due to congestion along Jalan Klang Lama (compounded by inclement weather), can have headways degraded to 40 to 60 minutes.

With such long headways, the wait for the bus becomes half of, or more, of the commute time; rather than giving up on public transportation, having **realtime information** (specifically, realtime bus arrival times) means I can make more efficient use of my time by running errands or having a meal while waiting for the bus.

===== Existing Solutions =====

=== Maps ===

  * [[https://myrapid.com.my/bus-train/rapid-kl/rapid-kl-integrated-transit-map/|RapidKL Integrated Transit Map]]

=== Directions (Wayfinding), Schedules and Arrival Times ===

  * [[https://www.google.com.my/maps|Google Maps]]
    * Good:
      * Wayfinding (directions)
    * Bad:
        * Live arrival times are not accurate
        * Scheduled departure times are not accurate or missing entirely
  * [[https://moovitapp.com/|Moovit]] ([[https://apps.apple.com/us/app/moovit-public-transit/id498477945|App Store]]/[[https://play.google.com/store/apps/details?id=com.tranzmate|Play Store]])
    * Good:
      * Live arrival times are accurate (±1-3 min)
      * Scheduled departure times
    * Bad:
      * Constant and annoying full-screen advertisements (or monthly subscription)
      * Live bus location is behind monthly subscription paywall
  * [[https://myrapid.com.my/pulse/|MyRapid PULSE]] ([[https://apps.apple.com/my/app/myrapid-pulse/id6736607730|App Store]]/[[https://play.google.com/store/apps/details?id=com.prasarana.pulse.v4|Play Store]])
    * Good:
       * Live arrival times are accurate (±1-3 min)
       * Scheduled departure times
    * Bad:
      * Wayfinding (directions)
===== ... And a Few New Problems =====

==== No Swiss Army Knife ====

There is no //Swiss Army knife// app that does everything (wayfinding + schedules + arrival times) well without having some kind of catch.

**A new problem arises that multiple apps/information/facilities above need to be used to find the best route.**

==== Closed Source + Ads ====

Closed Source is probably a weak criticism to make, as are ads, since the GTFS datasets are huge and processing them is computationally (and thus financially) expensive. I think these solutions should exist for the general public who are willing to tolerate advertisements or pay to have an ad-free experience, at least, until a better option arises. After all, developers need to eat and earn a living wage, too.

But it is a problem to me because:

  * Closed source means we cannot learn how the data is consumed, computed and presented, and philosophically I believe in freedom of knowledge rather than be fully dependent on black boxes built by corporations
  * If I dislike ads and am willing to bear the cost of running my own infrastructure, then I want an ad-free self-hosted experience

Furthermore, knowing how my solution works and bearing the cost of hosting it, I am incentivized to make it cost-effective rather than being incentivized to burden the user with ads or a subscription instead.

===== Building a Solution =====

Given the lack of a comprehensive solution, and fully at risk of [[https://xkcd.com/927/|creating yet another competing and imperfect solution]], I decided to explore building a solution.

Even if I do not succeed (or get distracted by other commitments), at least the documentation below will serve as a starting point for anyone who finds themselves experiencing the same problems I did.

:!: This is a **living document** and the documentation below will continue to evolve as I try/build/fix things :!:

===== Malaysia's Transport Data APIs =====

Building a solution starts strong, because the [[https://data.gov.my/|Government of Malaysia Open Data Portal]] provides two authoritative public transportation data APIs in standard [[https://gtfs.org/|GTFS]] formats.

  * [[https://developer.data.gov.my/realtime-api/gtfs-static|GTFS Static API]] - consumed to build OpenTripPlanner Graph
  * [[https://developer.data.gov.my/realtime-api/gtfs-realtime|GTFS Realtime API]] - consumed by OpenTripPlanner Realtime Updaters

===== OpenTripPlanner =====

> OpenTripPlanner (OTP) is an open source multi-modal trip planner, focusing on travel by scheduled public transportation in combination with bicycling, walking, and mobility services including bike share and ride hailing.

Instructions are written with [[https://docs.docker.com/compose/install/|Docker Compose]] in mind, but it's possible (with many more steps) to [[https://docs.opentripplanner.org/en/latest/Basic-Tutorial/|run OpenTripPlanner standalone]] with the configuration below.

**Demo: https://otp.ndoo.my/**

==== Preparation - Files ====

  - **Create a working directory for OpenTripPlanner**, for this example, we will use ''./opentripplanner'', you may wish to use different persistent storage e.g. for a Docker volume
    * ''mkdir opentripplanner && cd ./opentripplanner''
  - **Create configuration files**
    - **Build Configuration** to download regional OpenStreepMap data and GTFS data (GTFS are not the documented API links [[https://github.com/opentripplanner/OpenTripPlanner/issues/6351|due to URI extension validation in OpenTripPlanner]])\\ <file json build-config.json>{
  "osm": [
    {
      "source": "https://download.geofabrik.de/asia/malaysia-singapore-brunei-latest.osm.pbf"
    }
  ],
  "transitFeeds": [
    {
      "feedId": "ktmb",
      "source": "https://openapi-malaysia-transport.s3.ap-southeast-1.amazonaws.com/ktmb/gtfs_ktmb.zip",
      "type": "gtfs"
    },
    {
      "feedId": "mybas-johor",
      "source": "https://openapi-malaysia-transport.s3.ap-southeast-1.amazonaws.com/mybas-johor/gtfs_mybas.zip",
      "type": "gtfs"
    },
    {
      "feedId": "rapid-bus-kl",
      "source": "https://openapi-malaysia-transport.s3.ap-southeast-1.amazonaws.com/prasarana/gtfs_rapid_bus_kl.zip",
      "type": "gtfs"
    },
    {
      "feedId": "rapid-bus-kuantan",
      "source": "https://openapi-malaysia-transport.s3.ap-southeast-1.amazonaws.com/prasarana/gtfs_rapid_bus_kuantan.zip",
      "type": "gtfs"
    },
    {
      "feedId": "rapid-bus-mrtfeeder",
      "source": "https://openapi-malaysia-transport.s3.ap-southeast-1.amazonaws.com/prasarana/gtfs_rapid_bus_mrtfeeder.zip",
      "type": "gtfs"
    },
    {
      "feedId": "rapid-bus-penang",
      "source": "https://openapi-malaysia-transport.s3.ap-southeast-1.amazonaws.com/prasarana/gtfs_rapid_bus_penang.zip",
      "type": "gtfs"
    },
    {
      "feedId": "rapid-rail-kl",
      "source": "https://openapi-malaysia-transport.s3.ap-southeast-1.amazonaws.com/prasarana/gtfs_rapid_rail_kl.zip",
      "type": "gtfs"
    }
  ],
  "transitModelTimeZone": "Asia/Kuala_Lumpur"
}
</file>
    - Router Configuration for Malaysia GTFS Realtime vehicle position feeds\\ <file json router-config.json>{
  "updaters": [
    {
      "type": "vehicle-positions",
      "frequency": "PT30S",
      "url": "https://api.data.gov.my/gtfs-realtime/vehicle-position/mybas-johor",
      "feedId": "mybas-johor"
    },
    {
      "type": "vehicle-positions",
      "frequency": "PT30S",
      "url": "https://api.data.gov.my/gtfs-realtime/vehicle-position/ktmb",
      "feedId": "ktmb"
    },
    {
      "type": "vehicle-positions",
      "frequency": "PT30S",
      "url": "https://api.data.gov.my/gtfs-realtime/vehicle-position/prasarana?category=rapid-bus-kl",
      "feedId": "rapid-bus-kl"
    },
    {
      "type": "vehicle-positions",
      "frequency": "PT30S",
      "url": "https://api.data.gov.my/gtfs-realtime/vehicle-position/prasarana?category=rapid-bus-mrtfeeder",
      "feedId": "rapid-bus-mrtfeeder"
    },
    {
      "type": "vehicle-positions",
      "frequency": "PT30S",
      "url": "https://api.data.gov.my/gtfs-realtime/vehicle-position/prasarana?category=rapid-bus-kuantan",
      "feedId": "rapid-bus-kuantan"
    },
    {
      "type": "vehicle-positions",
      "frequency": "PT30S",
      "url": "https://api.data.gov.my/gtfs-realtime/vehicle-position/prasarana?category=rapid-bus-penang",
      "feedId": "rapid-bus-penang"
    }
  ]
}
</file>

==== Preparation - Docker Compose ====

<file yaml compose.yaml>services:
  opentripplanner:
    command: --load --serve
    environment:
      - JAVA_TOOL_OPTIONS=-Xmx8G
    expose:
      - 8080
    image: opentripplanner/opentripplanner:latest
    restart: unless-stopped
    volumes:
      - /path/to/opentripplanner:/var/opentripplanner
</file>

==== Build Graph ====

  * Docker Compose:\\ ''%%docker compose run --rm opentripplanner --build --save%%''
  * Docker:\\ ''%% docker run --rm -v ./opentripplanner:/var/opentripplanner docker.io/opentripplanner/opentripplanner:latest --build --save%%''

==== Start OpenTripPlanner Server ====

    * Docker Compose
      * ''%%docker compose up -d opentripplanner%%''
      * ''%%docker compose logs -f opentripplanner%%'' to view logs
    * Docker (foreground): ''%%docker run -it --rm -p 8080:8080 -v ./opentripplanner:/var/opentripplanner docker.io/opentripplanner/opentripplanner:latest --load --serve%%''

FIXME unfinished writing, a conclusion and some screenshots here would be nice

===== Connecting to OneBusAway =====

We will use OneBusAway as a mobile app frontend to OneTripPlanner.

FIXME unfinished writing, OneBusAway + OpenTripPlanner works on Android, but not iOS; there was [[https://github.com/OneBusAway/otpkit|a GSoC 2024 project to support OpenTripPlanner on OneBusAway iOS]] but the app implementation never happened.