BGPReader Tutorial

BGPReader is a command-line tool (bgpreader) that is installed as part of the libBGPStream package. See the BGPReader documentation for detailed usage information.

The most commonly used command-line options are:

bgpreader -w <start>[,<stop>] [-p <project>] [-t <type>] [-c <collector>] [-T <resource-type>] [-m]


Below we provide the following tutorials:


Replace bgpdump with bgpreader

BGPReader can output data in the same format as BGPdump (using the -m option). In this way, bgpreader can be used as a drop-in replacement for bgpdump in some cases.

Suppose you want to process all the BGP information associated with BGP records generated by RIS RRC04 in the 20 minute time interval: Sat, 10 Oct 2015 15:50:00 UTC to Sat, 10 Oct 2015 16:10:00 UTC.

In this example, when using bgpdump the user must download each dump file within the time interval desired (6 files). A hypothetical processing script, my_script.pl receives an unsorted flow of data, also, it also implements some filter in order to ignore the data that are outside the desired interval.

$ cd raw-data
$ wget http://data.ris.ripe.net/rrc04/2015.10/bview.20151010.1600.gz
$ wget http://data.ris.ripe.net/rrc04/2015.10/updates.20151010.1610.gz
$ wget http://data.ris.ripe.net/rrc04/2015.10/updates.20151010.1605.gz
$ wget http://data.ris.ripe.net/rrc04/2015.10/updates.20151010.1600.gz
$ wget http://data.ris.ripe.net/rrc04/2015.10/updates.20151010.1555.gz
$ wget http://data.ris.ripe.net/rrc04/2015.10/updates.20151010.1550.gz
$ cd ..
$ find ./raw-data/ -type f | xargs -n 1 | bgpdump -m  | perl my_script.pl


With bgpreader the same operation is accomplished with a single command. The processing script, my_script.pl receives a sorted flow of data that contains only the information associated with the desired time interval.

$ bgpreader -w 1444492200,1444493400 -c rrc04  -m | perl my_script.pl


1 second of Route Views Linx updates using the bgpdump format

The following command outputs (using the bgpdump -m format) the BGPStream elems contained in the BGPStream records that comply with the following filters:

  • are contained in Updates dumps generated by the Route Views LINX collector
  • their timestamp is exactly Sat, 10 Oct 2015 17:34:02 UTC
$ bgpreader -w 1444498442,1444498442 -c route-views.linx -m
BGP4MP|1444498442|A|195.66.224.175|13030|46.219.122.0/24|13030 5580 21011 31148 31148 31148|IGP|195.66.224.175|0|1|65123:276 65123:2000 65123:2002 65123:10016 13030:1 13030:7208 13030:50000 13030:51107|AG|31148 94.76.105.10|
BGP4MP|1444498442|W|195.66.224.138|2914|209.212.8.0/24
BGP4MP|1444498442|W|195.66.224.138|2914|205.151.210.0/23
...
BGP4MP|1444498442|A|195.66.236.175|13030|118.193.51.0/24|13030 2828 10026 17444 17444 133115|IGP|195.66.236.175|0|1|13030:2 13030:2828 13030:51903 13030:7215|NAG||
BGP4MP|1444498442|A|195.66.236.175|13030|177.154.84.0/22|13030 12989 28640 262401 262401 262401 262401 262401 262401 262401 262401 262949|IGP|195.66.236.175|0|1|13030:1 13030:3 13030:50000 13030:51502 13030:7209|NAG||
BGP4MP|1444498442|A|195.66.236.175|13030|177.154.80.0/22|13030 12989 28640 262401 262401 262401 262401 262401 262401 262401 262401 262949|IGP|195.66.236.175|0|1|13030:1 13030:3 13030:50000 13030:51502 13030:7209|NAG||


The above command outputs 114 announcements, and 11 withdrawals as observed by 14 unique peer ASns.


BGP Stream elems observed by RIS collectors updates in 2 minutes

The following command outputs the BGPStream elems contained in the BGPStream records that comply with the following filters:

  • are contained in Updates dumps generated by RIS collectors
  • their timestamp is in the interval Sat, 10 Oct 2015 17:34:00 UTC - Sat, 10 Oct 2015 17:36:00 UTC
$ bgpreader -w 1444498440,1444498560 -p ris -t updates
U|A|1444498440.000000|ris|rrc00|||7018|12.0.1.63|43.224.84.0/24|12.0.1.63|7018 1299 15412 9304 132167|132167|7018:5000 7018:38000||
U|A|1444498440.000000|ris|rrc00|||7018|12.0.1.63|43.224.85.0/24|12.0.1.63|7018 1299 15412 9304 132167|132167|7018:5000 7018:38000||
U|A|1444498440.000000|ris|rrc00|||8758|2001:8e0:0:ffff::9|2801:80:e90::/48|2001:8e0:0:ffff::9|8758 8758 6939 3549 18881 264583|264583|8758:200 8758:301||
...
U|W|1444498440.000000|ris|rrc00|||50300|2a00:1c10:10::8|2801:80:e90::/48||||||
U|A|1444498440.000000|ris|rrc00|||57821|2001:67c:26f4::1|2801:80:e90::/48|2001:67c:26f4::1|57821 6939 3549 18881 264583|264583|||
U|A|1444498440.000000|ris|rrc00|||1836|146.228.1.3|177.154.80.0/22|146.228.1.3|1836 12989 52840 262949 262949 262949 262949 262949 262949 262949 262949 262949 262949 262949 262949|262949|1836:120 1836:3100 1836:3110||


The above command outputs 181,118 announcements, 151 state messages, and 7,882 withdrawals as observed by 13 collectors (and 209 unique peer ASns).


BGP Stream elems with filters

The following command outputs the BGPStream elems contained in the BGPStream records that comply with the following filters:

  • are contained in Updates dumps generated by rrc06 or route-views.jinx
  • their timestamp is in the interval Wed, 01 Apr 2015 00:07:27 UTC - Wed, 01 Apr 2015 00:07:54 UTC
  • the peer ASn is either 25152 or 37105
  • if the message is an announcement, then
    • the announced prefix is equal or a more specific of 2620:110:9004::/40, 154.73.128.0/17, or 202.70.88.0/21
    • at least one community matches one of the following filters 2914:* (the ASn field is equal to 2914), or *:300 (the value field is equal to 300).
$ bgpreader  -t updates -c rrc06 -c route-views.jinx -w1427846847,1427846874 -j 25152 -j 37105 -k 2620:110:9004::/40 -k 154.73.128.0/17 -k 202.70.88.0/21 -y 2914:* -y *:300
U|A|1427846850.000000|ris|rrc06|||25152|202.249.2.185|202.70.88.0/21|202.249.2.185|25152 2914 15412 9304 23752|23752|2914:410 2914:1408 2914:2401 2914:3400||
U|A|1427846860.000000|ris|rrc06|||25152|202.249.2.185|202.70.88.0/21|202.249.2.185|25152 2914 15412 9304 23752|23752|2914:410 2914:1408 2914:2401 2914:3400||
U|A|1427846871.000000|ris|rrc06|||25152|2001:200:0:fe00::6249:0|2620:110:9004::/48|2001:200:0:fe00::6249:0|25152 2914 3356 13620|13620|2914:420 2914:1001 2914:2000 2914:3000||
U|A|1427846874.000000|routeviews|route-views.jinx|||37105|196.223.14.46|154.73.136.0/24|196.223.14.84|37105 37549|37549|37105:300||
U|A|1427846874.000000|routeviews|route-views.jinx|||37105|196.223.14.46|154.73.137.0/24|196.223.14.84|37105 37549|37549|37105:300||
U|A|1427846874.000000|routeviews|route-views.jinx|||37105|196.223.14.46|154.73.138.0/24|196.223.14.84|37105 37549|37549|37105:300||
U|A|1427846874.000000|routeviews|route-views.jinx|||37105|196.223.14.46|154.73.139.0/24|196.223.14.84|37105 37549|37549|37105:300||


RRC00 in real-time

The following command outputs the BGPStream elems contained in the BGPStream records that comply with the following filters:

  • are contained in RIBs and _ Updates dumps_ generated by the RIS RRC00 collector
  • their timestamp is greater or equal to Sat, 15 Oct 2015 17:12:00 UTC
$ bgpreader -w 1444929120 -c rrc00 
U|A|1444929120|ris|rrc00|1836|2a01:2a8::3|2c0f:fe90::/32|2a01:2a8::3|1836 174 6453 30844 37105 37105 37105 36943|36943|1836:110 1836:6000 1836:6031||
U|A|1444929120|ris|rrc00|1836|2a01:2a8::3|2c0f:fe90::/32|2a01:2a8::3|1836 6939 30844 37105 37105 37105 36943|36943|1836:3200 1836:3210||
U|A|1444929120|ris|rrc00|1836|146.228.1.3|168.128.104.0/21|146.228.1.3|1836 3356 2914 44568 44568|44568|1836:110 1836:6000 1836:6001 2914:410 2914:1203 2914:2201 2914:3200 3356:2 3356:22 3356:86 3356:502 3356:666 3356:2066||
U|A|1444929120|ris|rrc00|8758|212.25.27.44|62.112.24.0/21|212.25.27.44|8758 8220 47377|47377|8220:65080 8220:65401 8758:110 8758:300||
...


BGPStream with Local Cache

BGPStream version 2 added support for local caching. Users can provide a cache directory to BGPStream, and BGPStream will use the directory to save resources used in the query. In BGPReader, you can enable cache feature by specifying a cache directory using -o cache-dir=DIRECTORY option.

The following command put cached files to /tmp/cache-test/ for a query of updates:

$ bgpreader -w 1444498442,1444498442 -c route-views.linx -m -o cache-dir=/tmp/cache-test
BGP4MP|1444498442|A|195.66.224.175|13030|46.219.122.0/24|13030 5580 21011 31148 31148 31148|IGP|195.66.224.175|0|1|65123:276 65123:2000 65123:2002 65123:10016 13030:1 13030:7208 13030:50000 13030:51107|AG|31148 94.76.105.10|
BGP4MP|1444498442|W|195.66.224.138|2914|209.212.8.0/24
BGP4MP|1444498442|W|195.66.224.138|2914|205.151.210.0/23
...

This command will cache the following files at /tmp/cache-test:

$ ls -l /tmp/cache-test
total 6144
drwxr-xr-x   3 USER  GROUP       96 Oct 18 12:51 ./
drwxrwxrwt  33 root  GROUP     1056 Oct 18 12:51 ../
-rw-r--r--   1 USER  GROUP  2466076 Oct 18 12:51 routeviews.route-views.linx.updates.1444498200.900.cache

The next time an query using the same resource will directly use the locally cached files instead of re-downloading from remote locations.



Realtime Live Streams from Route Views and RIPE RIS

Both Route Views and RIPE NCC serve realtime live stream data to the public. Route Views provides a BMP live stream from many of its peers and RIPE RIS provides RIS Live. In BGPStream version 2, we added support for both live stream sources.

To access the live stream data, a user just needs to set project (-p) to routeviews-stream for the Route Views BMP stream or ris-live for the RIS Live stream.

The following command will start a real-time live stream from Route Views BMP stream.

$ bgpreader -p routeviews-stream
WARN: No time window specified, defaulting to live mode
U|A|1602281859.663705|routeviews-stream|is-ah-bmp1|saopaulo2|200.160.6.203|52873|187.16.220.216|114.5.9.0/24|187.16.220.216|6447 52873 12956 6453 4761|4761|||
U|A|1602281859.663713|routeviews-stream|is-ah-bmp1|saopaulo2|200.160.6.203|52873|187.16.220.216|64.68.236.0/22|187.16.220.216|6447 52873 12956 3356 13904 13904 13904 13904 13904 13904|13904|||
U|A|1602281860.068754|routeviews-stream|is-ah-bmp1|saopaulo2|200.160.6.203|52873|187.16.220.216|64.68.236.0/22|187.16.220.216|6447 52873 12956 3356 13904|13904|||
U|A|1602281860.077008|routeviews-stream|is-ah-bmp1|saopaulo2|200.160.6.203|52873|187.16.220.216|184.27.202.0/24|187.16.220.216|6447 52873 12956 3257|3257|||
U|A|1602281860.077026|routeviews-stream|is-ah-bmp1|saopaulo2|200.160.6.203|52873|187.16.220.216|104.77.0.0/24|187.16.220.216|6447 52873 12956 3257|3257|||
U|A|1602281860.077031|routeviews-stream|is-ah-bmp1|saopaulo2|200.160.6.203|52873|187.16.220.216|204.93.46.0/23|187.16.220.216|6447 52873 12956 3257|3257|||
U|A|1602281860.077036|routeviews-stream|is-ah-bmp1|saopaulo2|200.160.6.203|52873|187.16.220.216|23.10.241.0/24|187.16.220.216|6447 52873 12956 3257|3257|||
U|A|1602281860.077041|routeviews-stream|is-ah-bmp1|saopaulo2|200.160.6.203|52873|187.16.220.216|173.223.54.0/24|187.16.220.216|6447 52873 12956 3257|3257|||
U|A|1602281860.077045|routeviews-stream|is-ah-bmp1|saopaulo2|200.160.6.203|52873|187.16.220.216|104.77.131.0/24|187.16.220.216|6447 52873 12956 3257|3257|||
U|A|1602281860.077051|routeviews-stream|is-ah-bmp1|saopaulo2|200.160.6.203|52873|187.16.220.216|184.51.121.0/24|187.16.220.216|6447 52873 12956 3257|3257|||
...

The following command will start a real-time live stream from RIS Live.

$ bgpreader -p ris-live
WARN: No time window specified, defaulting to live mode
U|A|1569948264.660000|ris-live|rrc15|||28186|187.16.219.51|209.177.171.0/24|187.16.219.51|28186 3549 3356 1299 18465|18465|||
U|A|1569948264.660000|ris-live|rrc15|||28186|2001:12f8::219:51|2c0f:fe30::/32|2001:12f8::219:51|28186 30844 37006|37006|||
U|A|1569948264.660000|ris-live|rrc15|||42473|187.16.216.121|185.26.216.0/23|187.16.216.121|42473 16735 3257 28917 48858 39785 39785|39785|16735:3 16735:6101 47147:1502 47147:2002 47147:2104 47147:2300 47147:2405 47147:2604||
U|A|1569948264.660000|ris-live|rrc15|||42473|2001:12f8::121|2404:5780:3::/48|2001:12f8::121|42473 16735 6939 2914 20473 9558|9558|47147:1502 47147:2002 47147:2104 47147:2300 47147:2405 47147:2604||
U|A|1569948264.660000|ris-live|rrc15|||42473|187.16.216.121|185.26.216.0/23|187.16.216.121|42473 16735 1299 48858 39785 39785|39785|16735:3 16735:6101 47147:1502 47147:2002 47147:2104 47147:2300 47147:2405 47147:2604||
U|A|1569948264.660000|ris-live|rrc15|||42473|187.16.216.121|209.177.171.0/24|187.16.216.121|42473 16735 174 1299 18465|18465|16735:3 16735:6101 47147:1502 47147:2002 47147:2104 47147:2300 47147:2405 47147:2604||
U|A|1569948264.680000|ris-live|rrc07|||6667|194.68.123.136|177.136.12.0/24|194.68.123.136|6667 3491 7738 52871|52871|6667:3001 6667:4004 6667:5209||
U|A|1569948264.680000|ris-live|rrc07|||6667|2001:7f8:d:ff::136|2404:5780:3::/48|2001:7f8:d:ff::136|6667 174 20473 9558|9558|174:21001 174:22013 6667:3004 6667:4005 6667:5009 6667:6000||
U|A|1569948264.680000|ris-live|rrc07|||6667|194.68.123.136|185.137.56.0/22|194.68.123.136|6667 29119|29119|6667:3001 6667:4004 6667:5209||
...


Other data interfaces: how to use singlefile

The following command outputs the BGPStream elems contained in the BGPStream records that comply with the following filters:

  • are contained in http://archive.routeviews.org/bgpdata/2015.10/UPDATES/updates.20151016.1630.bz2
  • their timestamp is in the interval Fri, 16 Oct 2015 16:30:00 UTC - Fri, 16 Oct 2015 16:42:35 UTC

$ bgpreader -d singlefile -o upd-file=http://archive.routeviews.org/bgpdata/2015.10/UPDATES/updates.20151016.1630.bz2 -w 1445013000,1445013755
U|A|1445013000.000000|singlefile|singlefile|||13030|213.144.128.203|212.22.66.0/24|213.144.128.203|13030 12389 12389 12389 12389 12389 12389 41938 8359 50618 35189 201432|201432|13030:1 13030:3 13030:50000 13030:51202 13030:7179||
U|A|1445013000.000000|singlefile|singlefile|||3130|147.28.7.2|76.191.107.0/24|147.28.7.2|3130 11404 22059|22059|3130:380||
U|A|1445013000.000000|singlefile|singlefile|||3130|147.28.7.2|64.34.125.0/24|147.28.7.2|3130 2914 7922 11404 22059|22059|2914:420 2914:1008 2914:2000 2914:3000 3130:380||
U|A|1445013000.000000|singlefile|singlefile|||8492|85.114.0.217|185.37.192.0/23|85.114.0.217|8492 21011 34251 203957|203957|8492:1201 8492:1602||
U|A|1445013000.000000|singlefile|singlefile|||3549|208.51.134.246|185.37.192.0/23|208.51.134.246|3549 3257 21011 34251 203957|203957|3549:2203 3549:2623 3549:30840 3549:31250||
U|A|1445013000.000000|singlefile|singlefile|||8492|85.114.0.217|185.37.192.0/23|85.114.0.217|8492 21011 34251 203957|203957|8492:1306 8492:1601 50384:3901 50384:4001||
U|A|1445013000.000000|singlefile|singlefile|||8492|85.114.0.217|78.106.205.0/24|85.114.0.217|8492 20485 3356 6762 3216 8402|8402|8492:1102 8492:1601 15774:20607 20485:20607||
U|A|1445013000.000000|singlefile|singlefile|||8492|85.114.0.217|78.106.205.0/24|85.114.0.217|8492 20485 3356 6762 3216 8402|8402|8492:1307 8492:1601 15774:20607 20485:20607 25478:3000 25478:3005||
U|A|1445013000.000000|singlefile|singlefile|||3561|206.24.210.80|185.37.192.0/23|206.24.210.80|3561 3257 21011 34251 203957|203957|||
U|A|1445013000.000000|singlefile|singlefile|||1668|66.185.128.1|185.37.192.0/23|66.185.128.1|1668 3257 21011 34251 203957|203957|||174 202140|202140|||
...