NickLinRef is designed to accurately extract points or linestrings from the
Western Australia Road Network
geometry using road/slk_from/slk_to or just road/slk. Get results in either
GeoJSON or WKT formats.
Get results in Excel using the
=WEBSERVICE() formula,
in PowerBI using the
Web.Contents() function,
or best of all, in my custom PowerBI visual
NickMapBI to dynamically
create and visualise geometry based on live data. Below is a screenshot of NickMapBI:
- 1. Getting Started
- 2. Distinguishing Features
- 3. Usage
- 4. Running the Server Yourself
- 5. Related Projects
First see 4. Running the Server Yourself or 4.2. Compilation to get your copy of the server up and running.
Next See the 3. Usage section for some examples and detailed documentation.
NickLinRef is different from existing web services because it can accurately
truncate road centreline geometry at the requested slk_from and slk_to
endpoints. It can also correctly interpolate to find a Latitude Longitude point
at a requested slk. These features are not available in other existing APIs
such as those available from https://data.wa.gov.au.
- Superior performance under high traffic (for example if users are calling it
from a large excel sheet using
=WEBSERVICE()) - Low server resource consumption (CPU, RAM, storage etc).
- Support for GeoJSON or WKT output formats.
- Support for
offsetoperation which can assist in visualizing data for different lanes. - Bandwidth-efficient
/batch/mode to integrate with applications
(Such as custom PowerBI visuals)
When the web service is running locally (on your own machine) it can be accessed at the following address by default:
http://localhost:8080/...
Query parameters must be added to the end of the url to get the desired results;
a query begins with ? followed by a number of {name}={value} pairs separated
by &.
For example, the following request returns two points at road=H001 and
slk=2. Note that the response contains two points because there are two
carriageways at this location. The response is in the default 'geojson' format:
http://localhost:8080/?road=H001&slk=2&cwy=LS
{"type":"Feature","geometry":{"type":"MultiPoint", "coordinates":[
[115.89702617983814,-31.97176876372234],
[115.89692159880637,-31.97178473847775]
]}}In the next example, a multi-linestring is returned rather than points. The
search has been limited to the Left and Single carriageways, and an offset of
-10 metres from the centreline has been applied, and the format is wkt:
http://localhost:8080/?road=H001&slk_from=1&slk_to=1.1&cwy=LS&offset=-10&f=wkt
MULTILINESTRING ((115.88771097361135 -31.967604589743765,
115.88776331305647 -31.96753166223028,115.88782456479156 -31.967494045166685,
115.88808285746482 -31.967581573012584,115.88834117332095 -31.967675703676065),
(115.88735210575929 -31.967327078117492,115.88761740846113 -31.967472091243042),
(115.88761495220085 -31.96747075121283,115.88782449298621 -31.967576711138406))
In this final example, the latlon format is used to obtain a simple Latitude /
Longitude value. Note that f=latlon averages all matching point results to
ensure only a single coordinate is retuned:
http://localhost:8080/?road=H001&slk=2&f=latlon
-31.971776751100045,115.89697388932225
The full set of allowed parameters are summarised in the following tables:
| Name | Description | Allowed Values | Example | Case Sensitive | Required | Default |
|---|---|---|---|---|---|---|
road |
Road Number | Valid Road / PSP Number (See supported network types) | road=H001 |
✔️ | ✔️ | - |
slk_from |
SLK to start the segment. If omitted, or -Infinity, return from start of road. | Any Number or Infinity |
slk_from=1.55 |
✔️ | -Infinity | |
slk_to |
SLK to end the segment. If omitted, or +Infinity, return up to end of road. | Any Number or Infinity(> slk_from) |
slk_to=2.3 |
✔️ | +Infinity | |
cwy |
Filter for the carriageway. See cwy Parameter |
L R S LS RS LR LRS |
cwy=RS |
✔️ | LRS |
|
offset |
Metres to offset the resulting line from the road centre line. See offset Parameter |
Positive or Negative Number Note: Large values can cause blank output |
offset=-3.5 |
✔️ | 0 |
|
f |
Desired response format (See 4.3.3. f= Parameter) |
geojson wkt json |
f=geojson |
✔️ | geojson |
|
m |
EXPERIMENTAL Option to include M linear slk coordinates. |
true false |
m=true |
✔️ | false |
| Name | Description | Allowed Values | Example | Case Sensitive | Required | Default |
|---|---|---|---|---|---|---|
road |
Road Number | Valid Road / PSP Number (See supported network types) | road=H001 |
✔️ | ✔️ | - |
slk |
SLK of the point | Positive Number | slk=3 |
✔️ | ✔️ | - |
cwy |
Filter for the carriageway. See cwy Parameter |
L R S LS RS LR LRS |
cwy=RS |
✔️ | LRS |
|
offset |
Metres to offset the resulting point from the road centre line. See offset Parameter |
Positive or Negative Number Note: Large values can cause blank output |
offset=4 |
✔️ | 0 |
|
f |
Desired response format. (See 4.3.3. f= Parameter) |
geojson wkt json latlon latlondir |
f=geojson |
✔️ | geojson |
The following diagram illustrates the effect of the carriageway filter parameter
cwy= with some random examples.
Note that any combination of L, R and S can be
used. The filter is applied before any points or lines are sampled from the
centreline.
The following diagram illustrates the effect of the offset parameter offset=.
The offset is applied before any points or lines are sampled from the road centreline.
Note a rough approximation is used to generate offset points and linestrings. Large offset values may cause blank response values. Try to keep it less then +- 40 metres.
| Format | Specification | Notes |
|---|---|---|
f=geojson |
https://geojson.org/ | Responses are always wrapped in a Feature and will always be either "type":"MultiLineString" or "type":"MultiPoint" |
f=json |
Derived from geojson | Nested array like the "coordinates":... attribute in the the geojson MultiLineString or MultiPoint specifications. It is intended to reduce unnecessary json overhead. |
f=wkt |
https://www.ogc.org/standard/sfa/ | |
f=latlon |
{latitude},{longitude} |
Responses are always a single comma separated pair. If multiple points would have been returned (eg for left and right carriageway) then the average of these is returned. |
f=latlondir |
{latitude},{longitude},{direction} |
Responses are always a single comma separated triplet. If multiple points would have been returned (eg for left and right carriageway) then the average of these is returned. The direction is in degrees measured anti-clockwise-positive from east. It is the is the direction of increasing SLK. |
See also Coordinate Reference System (CRS)
Show mode works the same as described above, except that instead of returning
raw data, it displays an interactive map when viewed in a web browser. This is
useful to confirm that queries are working as intended. Simply add /show/ to
the url before the query parameters:
http://localhost:8080/show/?road=H001&slk_from=1&slk_to=2&cwy=LS&offset=-10&f=wkt
Query mode can easily be used from Excel with the =WEBSERVICE() formula, or
from Power BI using the =Web.Contents() function.
/batch/ mode is a bandwidth efficient alternative query method meant for
integration with apps and PowerBI custom visuals.
This mode expects a POST request to http://localhost:8080/batch/ and does
not use url query parameters. See details below.
Click to expand details of `/batch/` Mode
Batch mode allows only linestring requests. The request is in a packed binary
format, the response will be gzipped a geojson FeatureCollection object.
The body of the request must be binary data consisting of a series of frames with the format shown below. Any number of frames can be packed into a single request.
Frame format:
| Byte Length | Type | Value |
|---|---|---|
| 1 | Uint8 | Number of bytes in road string x |
x |
Utf8 String | road number |
| 4 | Float32 Little Endian | slk_from in kilometres |
| 4 | Float32 Little Endian | slk_to in kilometres |
| 4 | Float32 Little Endian | offset in metres |
| 1 | Uint8 | cwy (carriageways) (see table below) |
Value of cwy:
cwy |
Carriageway | Binary | Decimal |
|---|---|---|---|
R |
Right only | 0b0000_0001 |
1 |
S |
Single only | 0b0000_0010 |
2 |
RS |
Right & Single | 0b0000_0011 |
3 |
L |
Left only | 0b0000_0100 |
4 |
LR |
Left & Right | 0b0000_0101 |
5 |
LS |
Left & Single | 0b0000_0110 |
6 |
LRS |
All | any other value |
There is an example batch query implementation in __static_http/main.js
however a simplified version is shown below:
// =========== Helper functions: ===========
let CWY_LOOKUP = {
L: 0b0000_0100,
R: 0b0000_0001,
S: 0b0000_0010,
LR: 0b0000_0101,
LS: 0b0000_0110,
RS: 0b0000_0011,
LRS: 0b0000_0111
}
function binary_encode_request(road, slk_from, slk_to, offset, cwy) {
let text_encoder = new TextEncoder();
let road_bytes = text_encoder.encode(road);
let buffer = new ArrayBuffer(1 + road_bytes.length + 4 + 4 + 4 + 1);
let road_name_chunk = new Uint8Array(buffer, 0, 1 + road_bytes.length);
road_name_chunk[0] = road_bytes.length;
road_name_chunk.set(road_bytes, 1);
let data_view = new DataView(buffer, 1 + road_bytes.length);
data_view.setFloat32(0, slk_from, true) // LITTLE ENDIAN
data_view.setFloat32(4, slk_to, true) // LITTLE ENDIAN
data_view.setFloat32(8, offset, true) // LITTLE ENDIAN
data_view.setUint8(12, CWY_LOOKUP[cwy.toUpperCase()] ?? 0); // use 0 if lookup fails
return new Uint8Array(buffer);
}
// =========== Perform Batch Query ============
// Build batch query:
let request_body_parts = [
binary_encode_request("H001", 1.0, 1.1, 0, "LRS"),
binary_encode_request("H001", 3.0, 3.2, 0, "LS"),
binary_encode_request("H002", 4.1, 4.2, 20, "LS")
];
// Find total query length in bytes:
let request_body_byte_length = request_body_parts.reduce(
(total, item) => total + item.byteLength,
0 // initial value of total
);
// Pack all queries into a single byte array:
let request_body = new Uint8Array(request_body_byte_length);
request_body_parts.reduce((offset, byte_array) => {
request_body.set(byte_array, offset);
return offset + byte_array.byteLength;
},
0 // initial offset
)
// Send the request to the server
fetch("http://localhost:8080/batch/", {
method: "POST",
body: request_body
}
)
.then(response => response.json())
.then(json => {
let features = [];
for (multi_line_string_coordinates of json) {
if (multi_line_string_coordinates == null) continue;
features.push({
type: "Feature",
geometry: {
type: "MultiLineString",
coordinates: multi_line_string_coordinates
}
});
}
let result = {
type:"FeatureCollection",
features
}
// DONE! 'result' is now a standard GeoJSON feature collection.
// ready to be used in a map or for other purposes.
console.log(JSON.stringify(result));
});The output of the script above is shown below:
{
"type":"FeatureCollection",
"features":[
{"type":"Feature","geometry":{"type":"MultiLineString","coordinates":[[[115.88778395521496,-31.967656968008896],[115.88783629466008,-31.96758404049541],[115.88784873900369,-31.967580563399395],[115.88804803150612,-31.96766437971354],[115.88831038467791,-31.967760094769094]],[[115.88778395521496,-31.967656968008896],[115.8877412235351,-31.967713333565143],[115.887742230063,-31.967723398843077],[115.8879828817074,-31.96785406445076]],[[115.8879828817074,-31.96785406445076],[115.88808710308484,-31.967912260057517]],[[115.8873090202252,-31.96740590345527],[115.88757432292704,-31.96755091658082]],[[115.8875744144296,-31.96755091658082],[115.88778395521496,-31.967656876506396]]]}},
{"type":"Feature","geometry":{"type":"MultiLineString","coordinates":[[[115.90371854775746,-31.97874013724704],[115.90387696204294,-31.978944811166286],[115.90481214543112,-31.980341530482082]]]}},
{"type":"Feature","geometry":{"type":"MultiLineString","coordinates":[[[115.79676132053224,-32.08478062909291],[115.79696805522983,-32.08570245636619]]]}}
]
}A new batch request route has been added to allow requesting both line and point features.
This route accepts either a POSTor GET requests.
This version of the batch route does not use the complex binary protocol described above.
The format of the request is as follows;
{
"format":"wkt",
"items":[
{
"road":"H001",
"slk_from":10,
"slk_to":20,
"offset":10
},
{
"road":"H016",
"slk":10
},
{
"road":"H015",
"slk":10
}
]
}When url encoded the above query should look like;
format=wkt&items=%5B%7B%27road%27%3A+%27H001%27%2C+%27slk_from%27%3A+10%2C+%27slk_to%27%3A+20%2C+%27offset%27%3A+10%7D%2C+%7B%27road%27%3A+%27H016%27%2C+%27slk%27%3A+10%7D%2C+%7B%27road%27%3A+%27H015%27%2C+%27slk%27%3A+10%7D%5D
Formats supported are restricted to wkt, geojson or json
The result type is always a JSON list which is the same length as the "items" specified in the request.
Items that did not return a result may be either null or, due to a long outstanding issue, an invalid empty geometry like MULTIPOINT () or a GeoJSON object with 0 coordinates.
If format=wkt is specified, then each item will be either null or a "wkt string". The result should look like;
200
["MULTILINESTRING ((115.94759478043025 -32.02724359866268,...., 116.00968185354726 -32.0797516372457))","MULTIPOINT ((115.8018372737292 -31.890062043719624),(115.80208275968369 -31.88999214761082))","MULTIPOINT ((115.85393141776753 -32.048215776792965),(115.85365425352151 -32.04809166414051))"]The /point and /line routes support BOTH GET and POST requests.
Note: The first version of this server supported only
GETrequests against the root path/using url query parameters. To add supportPOSTrequests, I had to create two new routes/lineand/point.For backward compatibility
GET, requests against root/will continue to work.
POST requests made against /line or /route must use the same parameters as
documented above, but formatted as JSON in the body of the request. For example
GET localhost:8080/?road=H001&slk=10 becomes POST localhost:8080/point with
JSON body {"road":"H001", "slk":10}
SLK stands for "Straight Line Kilometre" and is sometimes called 'chainage' or 'kilometrage' in other contexts.
At Main Roads Western Australia SLK refers to an "adjusted" linear measure which has discontinuities called 'Points of Equation' (POE) (there are between 100 and 200 points of equation throughout the state road network) where there is an abrupt increase or decrease in SLK. This is done so that when asset locations are recorded by SLK, these records are not invalidated when a road realignment project modifies the length of a road.
This software has no special compensation to handle POE discontinuities. Please expect results at POEs to have gaps or overlaps.
The non-adjusted linear measure is called "True Distance".
This software is only capable of looking up Lat/Lon from SLK. True distance is not yet supported.
This tool is capable of querying all road network types included in this dataset https://portal-mainroads.opendata.arcgis.com/datasets/mainroads::road-network/about
| Network Type | Support |
|---|---|
| State Roads | ✔️ |
| Proposed State Roads | ✔️ |
| Local Government Roads | ✔️ |
| Main Roads Controlled Path | ✔️ |
| Miscellaneous Road | ✔️ |
| Crossover | ✔️ |
The coordinate system of the returned geometry depends on the coordinate system
downloaded from NLR_DATA_SOURCE_URL.
However, offset= feature will only work correctly with EPSG:4326 (which is
also called WGS84. See https://spatialreference.org/ref/epsg/wgs-84/) This is
because the &offset=... uses an approximation to convert from meters to
degrees assuming that there are about 111320 metres per degree.
On windows you can use a pre-compiled version of this application; download the latest version from releases and extract the zip file.
Run nicklinref.exe then visit
http://localhost:8080/?road=H001&slk_from=1.5&slk_to=3 to test if it is
working.
On Linux the best way to use this software is to clone this repository and build it yourself. See instructions below.
Install required packages:
sudo apt update
sudo apt-get install build-essential pkg-config gitNote: On some linux distros, the packages
libssl-devandbuild-essentialmay have different names.build-essentialcontains thelibcpackage which is needed by some rust packages to interact with other platform code.
Note: In previous version
libssl-devandpkg-configmay be needed. Newer version use rust-tls and do not depend on openssl.
Install Rust:
Follow the this guide https://www.rust-lang.org/tools/install Probably something like this:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
source "$HOME/.cargo/env"Clone this repository
git clone https://github.com/thehappycheese/nicklinref_rust
cd nicklinref_rustBuild and run:
cargo run --releaseInstall rust: https://www.rust-lang.org/tools/install. You may be prompted to install some microsoft visual C++ thing which is used for linking native executables.
Clone this repository
git clone https://github.com/thehappycheese/nicklinref_rust
cd nicklinref_rustBuild and run:
cargo run --releaseLinRef can be configured using either environment variable or command line
arguments. (Previous versions supported a config.json option, but support for
this is dropped because it was never used)
nicklinref supports a help command flag which will print out the most current command line documentation.
nicklinref.exe --helpUsage: nicklinref [OPTIONS]
Options:
--ip-address <NLR_ADDR>
The IP address to listen on [env: NLR_ADDR=] [default: 127.0.0.1]
--port <NLR_PORT>
The port to listen on [env: NLR_PORT=] [default: 8080]
--data-file <NLR_DATA_FILE>
File path to where the cache data file is/will be stored, including file name [env: NLR_DATA_FILE=] [default: ./data/data.json.lz4]
--static-http <NLR_STATIC_HTTP>
Folder path containing static http files for the /show/ route [env: NLR_STATIC_HTTP=] [default: ./__static_http]
--force-update-data
Cause the old data cache file to be deleted and re-downloaded [env: NLR_FORCE_UPDATE_DATA=]
--data-source-url <NLR_DATA_SOURCE_URL>
Url of the esri rest service hosting the road network data [env: NLR_DATA_SOURCE_URL=] [default: https://mrgis.ma...]
-h, --help
Print help
As an alternative to command line options, environment variables can be used instead.
Environment variables are overridden by any command line options.
| Property | Description |
|---|---|
NLR_ADDR |
A string containing an IPV4 or IPV6 address. Using 127.0.0.1 will limit traffic to your own machine for testing purposes. 0.0.0.0 will allow requests from anywhere on the local network. |
NLR_PORT |
A port number. |
NLR_DATA_FILE |
The filename of the data cached from NLR_DATA_SOURCE_URL. The directory must already exist. If the file does not already exist then it will be created and fresh data will be downloaded. |
NLR_DATA_SOURCE_URL |
This is the ArcGIS REST service where the road network is downloaded from. It is assumed that multiple requests are needed and the &resultOffset=... parameter is used to repeatedly fetch more data. Only certain fields are fetched outFields=ROAD,START_SLK,END_SLK,CWY and the output spatial reference is specified &outSR=4326. ESRI's own json format (&f=json) is expected because &f=geojson does not seem to work properly. Also note that currently the field names ROAD, START_SLK, END_SLK, CWY are hard-coded and must exist on the incoming data. |
NLR_STATIC_HTTP |
Used by the /show/ feature to display an interactive map. The directory specified by this config option should exist or I think the application may crash on startup. The directory can probably be empty though if it is not required. The __static_http folder in this repo contains the files required. |
To refresh your data, simply manually delete the file specified by the
NLR_DATA_FILE option and restart the application. Alternatively add the
--force-update-data flag to the command line when launching the server. Fresh
data will be downloaded.
Megalinref is an attempt to bring the functionality of this server directly to python. It is a rust-powered python library that will do all the same things as this server, but without the overhead of running a rest service on localhost.
NickMapBI is a custom PowerBI visual which calls into a running instance of NickLinRef.
This repo is a rust implementation of my previous project written in python: https://github.com/thehappycheese/linear_referencing_geocoding_server