7.0 Indexing

Indexing

We store the filter database into json files using sleekDB. Searching for a specific IP address is still fast when you have maybe 1000 entries. But, it just doesn't make any sense to loop through each json file to find the IP we are looking for.

For this, I have implemented my own Indexing class. Indexing creates a maping of Host IP and filter's ID to which it belongs to. First we take the IP and then convert it into decimal, which generates a fixed length of string. I then split the fixed length string into multiple chunks and those chunks becomes our directory structure. Finally the mapping entry is stored in a clear text file.

Example: Ip address: 8.8.8.8

• gets converted to 134744072
• directory structure is indexes/134/744/072/
• file inside the above directory structure is 134744072.txt which has the ID of the filter this IP belongs to.

When we lookup, we convert the IP to decimal and then search in the indexes folder. Once found, we make a direct call to the database with that ID, saving us time to loop through each file and searching for the address in each file.

I also index the API calls and BIN file search we make for ip2location, so the next lookup is via Index and not via API or BIN file.

See the indexes folder when you play around with filters to see it in action.

Auto Indexing

Auto indexing is enabled by default. If in case you want to reindex, you can do so.

via Cli Tool

admin@phpterminal:firewall(config)# filters reindex force

Reindexed all host IP addresses

admin@phpterminal:firewall(config)# 

via PHP Class

$firewall->indexes->reindexFilters(true);

Delete Indexing

via Cli Tool

admin@phpterminal:firewall(config)# filters reindex force norebuild

Deleted all host IP indexes

admin@phpterminal:firewall(config)# 

via PHP Class

$firewall->indexes->reindexFilters(true, true);