Skip to content

Commit 9e9f9e7

Browse files
committed
Moved tile expiring and initial rendering to separate scripts, more efficient tile expiring
1 parent 920afa7 commit 9e9f9e7

File tree

7 files changed

+336
-282
lines changed

7 files changed

+336
-282
lines changed

CHANGELOG.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,13 @@
22

33
(all changes without author notice are by [@rurseekatze](https://github.com/rurseekatze)
44

5+
## 0.2 (16.02.2014)
6+
7+
* Moved tile expiring to separate script
8+
* Added some config parameters for a more efficient tile expiring
9+
* Moved initial rendering to separate script for security and performance reasons
10+
* Some minor bugfixes
11+
512
## 0.1 (22.01.2014)
613

714
* Moved current development stage from OpenRailwayMap to separate repository, see OpenRailwayMap repository for the past development

README.md

Lines changed: 17 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,9 @@ You can set various options to configure your tileserver:
184184

185185
* `maxPrerender` Highest zoomlevel in which tiles are prerendered in initial rendering run. Tiles in higher zoomlevels will be rendered just on request. Change this value to increase or decrease the load for your system. As higher the value, as more tiles have to be rendered. If your selected value is too low, tile requests will be slow, so you should find a value that balances system load and request times. _Default: `8`_
186186

187-
* `maxCached` Highest zoomlevel in which tiles are cached. Tiles in higher zoomlevels will be rendered just on request and removed from the filesystem cache instead of rerendering if they are expired. Change this value to increase or decrease the load for your system. As higher the value, as more tiles have to be rerendered. If your selected value is too low, tile requests will be slow, so you should find a value that balances system load and request times. _Default: `15`_
187+
* `maxCached` Highest zoomlevel in which tiles are cached. Tiles in higher zoomlevels will be rendered just on request and removed from the filesystem cache instead of rerendering if they are expired. Change this value to increase or decrease the load for your system. As higher the value is, as more tiles have to be rerendered. If your selected value is too low, tile requests will be slow, so you should find a value that balances system load and request times. _Default: `16`_
188+
189+
* `minExpiring` Lowest zoomlevel in which tiles are only marked as expired if they are affected by an edit. Tiles in lower zoomlevels will be marked as expired only by a manual expiring (such as in an update script). Change this value to increase or decrease the load for your system. As lower the value is, as more tiles have to be rerendered. If your selected value is too high, too many not-affected tiles will be marked as expired, so you should find a value that balances system load and request times. _Default: `10`_
188190

189191
* `maxsockets` Maximum number of concurring http connections. The optimal value depends on your environment (hardware, operating system, system settings, ...), so you should try some values to get the optimal performance. _Default: `100`_
190192

@@ -208,7 +210,7 @@ You can set various options to configure your tileserver:
208210

209211
Start the initial rendering:
210212

211-
$ curl "http://localhost:9000/init"
213+
$ node init-rendering.js
212214

213215
## Usage
214216

@@ -282,6 +284,14 @@ You can set various options to configure your tileserver:
282284
railmap.redraw();
283285
...
284286

287+
There is also the possibility to force the rerendering of a single tile. Just add `/dirty` at the end of a tile URL. Example:
288+
289+
http://tiles.YOURDOMAIN.org/STYLENAME/z/x/y.png/dirty
290+
291+
or
292+
293+
http://tiles.YOURDOMAIN.org/vector/z/x/y.png/dirty
294+
285295
## Update database and tiles
286296

287297
Use osm2pgsql to update your database. To rerender all expired tiles, you need a file that contains a list of expired tiles. Such a command could look like this:
@@ -294,15 +304,16 @@ You can set various options to configure your tileserver:
294304

295305
Run
296306

297-
$ curl "http://localhost:9000/loadlist"
307+
$ node expire-tiles.js path/to/expired_tiles
298308

299309
to load the list of expired tiles and to mark all these tiles as expired. They will be rerendered on their next request or deleted from cache if they are highzoom tiles.
300310

301-
By requesting
311+
`Note:` It is not efficient to go through this list for all zoom levels, so by default only tiles zoom>=maxPrerender and zoom<=maxCached are marked as expired. For other zoomlevels you can mark all affected tiles by executing
302312

303-
$ curl "http://localhost:9000/status"
313+
$ cd /path/to/your/vector/tiles
314+
$ find <zoom> -exec touch -t 197001010000 {} \;
304315

305-
you can get the current number of tiles in the queue.
316+
You can also execute these commands to expire all tiles after a change of your stylesheet.
306317

307318
## References
308319

config.json

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@
22
"tileSize" : 256,
33
"prefix" : "railmap",
44
"database" : "railmap",
5-
"vtiledir" : "../tiles",
6-
"tiledir" : "../bitmap-tiles",
7-
"expiredtilesdir" : "../../olm/import",
5+
"vtiledir" : "/home/www/sites/194.245.35.149/site/orm/tiles",
6+
"tiledir" : "/home/www/sites/194.245.35.149/site/orm/bitmap-tiles",
7+
"expiredtilesdir" : "/home/www/sites/194.245.35.149/site/olm/import",
88
"scriptdir" : "../js",
99
"styledir" : "../styles",
1010
"zoomOffset" : 0,
@@ -15,7 +15,8 @@
1515
"geomcolumn" : "way",
1616
"pxtolerance" : 1.8,
1717
"maxPrerender" : 8,
18-
"maxCached" : 15,
18+
"maxCached" : 16,
19+
"minExpiring" : 8,
1920
"maxsockets" : 100,
2021
"tileserverPort" : 9000
2122
}

expire-tiles.js

Lines changed: 209 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,209 @@
1+
/*
2+
node-tileserver Copyright (C) 2014 Alexander Matheisen
3+
This program comes with ABSOLUTELY NO WARRANTY.
4+
This is free software, and you are welcome to redistribute it under certain conditions.
5+
See https://github.com/rurseekatze/node-tileserver for details.
6+
*/
7+
8+
9+
// load configuraion file
10+
var configuration = require('./config.json');
11+
12+
13+
// include necessary modules
14+
var fs = require('graceful-fs');
15+
var cluster = require('cluster');
16+
var os = require('os');
17+
var assert = require('assert');
18+
var http = require("http");
19+
var url = require("url");
20+
var mkdirp = require('mkdirp');
21+
var byline = require('byline');
22+
var touch = require("touch");
23+
24+
25+
if (process.argv[2] && process.argv[2].length > 0 && fs.existsSync(process.argv[2]))
26+
{
27+
expireTileList(process.argv[2], function(err)
28+
{
29+
if (err)
30+
console.log('Some problems occurred.');
31+
else
32+
console.log('Finished.');
33+
});
34+
}
35+
else
36+
console.log('Given filename invalid or file cannot be found.');
37+
38+
39+
// load an osm2pgsql list of expired tiles and marks all tiles as expired
40+
function expireTileList(filename, callback)
41+
{
42+
console.log('Checking if list of expired tiles exists...');
43+
if (fs.existsSync(filename))
44+
{
45+
console.log('Reading list of expired tiles...');
46+
var stream = byline(fs.createReadStream(filename));
47+
48+
stream.on('data', function(line)
49+
{
50+
var tile = line.toString().split("/");
51+
52+
expireTile(tile[0], tile[1], tile[2]);
53+
54+
// mark child tiles
55+
var tileset = childTiles(tile[0], tile[1], tile[2]);
56+
for (var tilesetIndex=0; tilesetIndex<tileset.length; tilesetIndex++)
57+
expireTile(tileset[tilesetIndex][0], tileset[tilesetIndex][1], tileset[tilesetIndex][2]);
58+
59+
// mark parent tiles
60+
var parentTileset = parentTiles(tile[0], tile[1], tile[2]);
61+
for (var parentIndex=0; parentIndex<parentTileset.length; parentIndex++)
62+
expireTile(parentTileset[parentIndex][0], parentTileset[parentIndex][1], parentTileset[parentIndex][2]);
63+
});
64+
65+
stream.on('end', function(line)
66+
{
67+
console.log('Expired tiles successfully marked.');
68+
return process.nextTick(function()
69+
{
70+
callback(false);
71+
});
72+
});
73+
74+
stream.on('error', function(line)
75+
{
76+
console.log('Cannot read list of expired tiles. Aborting.');
77+
return process.nextTick(function()
78+
{
79+
callback(true);
80+
});
81+
});
82+
}
83+
else
84+
{
85+
console.log('Cannot find expired-tiles-file. Aborting.');
86+
return process.nextTick(function()
87+
{
88+
callback(true);
89+
});
90+
}
91+
}
92+
93+
94+
// removes all rendering styles of a bitmap tile from the cache
95+
function removeBitmapTile(zoom, x, y, selectedStyle)
96+
{
97+
var selectedStyle = selectedStyle || 0;
98+
99+
if (selectedStyle > configuration.styles.length)
100+
return;
101+
102+
var filepath = configuration.tiledir+'/'+configuration.styles[selectedStyle]+'/'+zoom+'/'+x+'/'+y+'.png';
103+
if (fs.existsSync(filepath))
104+
{
105+
fs.unlinkSync(filepath);
106+
removeBitmapTile(zoom, x, y, selectedStyle);
107+
}
108+
}
109+
110+
111+
// marks a tile and all it's subtiles and parent tiles as expired or deletes them if necessary
112+
function expireTile(zoom, x, y)
113+
{
114+
if (parseInt(zoom) <= configuration.maxCached)
115+
markTileExpired(zoom, x, y);
116+
// remove all tiles higher than maxcached
117+
else
118+
{
119+
var filepath = configuration.vtiledir+'/'+zoom+'/'+x+'/'+y+'.json';
120+
if (fs.existsSync(filepath))
121+
{
122+
fs.unlinkSync(filepath);
123+
removeBitmapTile(zoom, x, y);
124+
}
125+
}
126+
}
127+
128+
129+
// marks a tile as expired
130+
function markTileExpired(zoom, x, y)
131+
{
132+
//var filepath = configuration.vtiledir+'/'+zoom+'/'+x+'/'+y+'.json';
133+
var filepath = '/home/www/sites/194.245.35.149/site/orm/tiles/'+zoom+'/'+x+'/'+y+'.json';
134+
135+
if (fs.existsSync(filepath))
136+
touch.sync(filepath, {time: new Date(10)});
137+
}
138+
139+
140+
// returns a list of all subtiles of a certain tile
141+
function childTiles(zoom, x, y)
142+
{
143+
zoom = parseInt(zoom);
144+
x = parseInt(x);
145+
y = parseInt(y);
146+
147+
var tiles = new Array();
148+
var tilesize = 1;
149+
150+
for (var z=zoom+1; z<=configuration.maxCached; z++)
151+
{
152+
x = x*2;
153+
y = y*2;
154+
tilesize = tilesize*2;
155+
156+
for (var i=x; i<(x+tilesize); i++)
157+
{
158+
for (var j=y; j<(y+tilesize); j++)
159+
tiles.push(new Array(z, i, j));
160+
161+
j -= tilesize;
162+
}
163+
164+
i -= tilesize;
165+
}
166+
167+
return tiles;
168+
}
169+
170+
171+
// returns a list of all tiles in lower zoomlevels that contain a certain tile
172+
function parentTiles(zoom, x, y)
173+
{
174+
zoom = parseInt(zoom);
175+
x = parseInt(x);
176+
y = parseInt(y);
177+
178+
var tiles = new Array();
179+
180+
while (zoom >= configuration.minExpiring)
181+
{
182+
zoom--;
183+
x = parseInt(x/2);
184+
y = parseInt(y/2);
185+
tiles.push(new Array(zoom, x, y));
186+
}
187+
188+
return tiles;
189+
}
190+
191+
192+
// returns the subtiles of a certain tile
193+
function subTiles(zoom, x, y)
194+
{
195+
zoom = parseInt(zoom);
196+
x = parseInt(x);
197+
y = parseInt(y);
198+
199+
x = x*2;
200+
y = y*2;
201+
zoom++;
202+
203+
return new Array(
204+
new Array(zoom, x, y),
205+
new Array(zoom, x+1, y),
206+
new Array(zoom, x, y+1),
207+
new Array(zoom, x+1, y+1)
208+
);
209+
}

0 commit comments

Comments
 (0)