BGPReader Tutorial

BGPReader is a command-line tool (bgpreader) that is installed as part of the BGPStream Core package. See the BGPReader documentation for detailed usage information.

The most commonly used command-line options are:

$ bgpreader -w <start>[,<stop>] [-p <project] [-t <type] [-c collector] [-m]


Below we provide the following tutorials:


Replace bgpdump with bgpreader

BGPReader can output data in the same format as BGPdump (using the -m option). In this way, bgpreader can be used as a drop-in replacement for bgpdump in some cases.

Suppose you want to process all the BGP information associated with BGP records generated by RIS RRC04 in the 20 minute time interval: Sat, 10 Oct 2015 15:50:00 UTC to Sat, 10 Oct 2015 16:10:00 UTC.

In this example, when using bgpdump the user must download each dump file within the time interval desired (6 files). A hypothetical processing script, my_script.pl receives an unsorted flow of data, also, it also implements some filter in order to ignore the data that are outside the desired interval.

$ cd raw-data
$ wget http://data.ris.ripe.net/rrc04/2015.10/bview.20151010.1600.gz
$ wget http://data.ris.ripe.net/rrc04/2015.10/updates.20151010.1610.gz
$ wget http://data.ris.ripe.net/rrc04/2015.10/updates.20151010.1605.gz
$ wget http://data.ris.ripe.net/rrc04/2015.10/updates.20151010.1600.gz
$ wget http://data.ris.ripe.net/rrc04/2015.10/updates.20151010.1555.gz
$ wget http://data.ris.ripe.net/rrc04/2015.10/updates.20151010.1550.gz
$ cd ..
$ find ./raw-data/ -type f | xargs -n 1 | bgpdump -m  | perl my_script.pl


With bgpreader the same operation is accomplished with a single command. The processing script, my_script.pl receives a sorted flow of data that contains only the information associated with the desired time interval.

$ bgpreader -w 1444492200,1444493400 -c rrc04  -m | perl my_script.pl


1 second of Route Views Linx updates using the bgpdump format

The following command outputs (using the bgpdump -m format) the BGPStream elems contained in the BGPStream records that comply with the following filters:

  • are contained in Updates dumps generated by the Route Views LINX collector
  • their timestamp is exactly Sat, 10 Oct 2015 17:34:02 UTC
$ bgpreader -w 1444498442,1444498442 -c route-views.linx -m
BGP4MP|1444498442|A|195.66.224.175|13030|46.219.122.0/24|13030 5580 21011 31148 31148 31148|IGP|195.66.224.175|0|1|65123:276 65123:2000 65123:2002 65123:10016 13030:1 13030:7208 13030:50000 13030:51107|AG|31148 94.76.105.10|
BGP4MP|1444498442|W|195.66.224.138|2914|209.212.8.0/24
BGP4MP|1444498442|W|195.66.224.138|2914|205.151.210.0/23
...
BGP4MP|1444498442|A|195.66.236.175|13030|118.193.51.0/24|13030 2828 10026 17444 17444 133115|IGP|195.66.236.175|0|1|13030:2 13030:2828 13030:51903 13030:7215|NAG||
BGP4MP|1444498442|A|195.66.236.175|13030|177.154.84.0/22|13030 12989 28640 262401 262401 262401 262401 262401 262401 262401 262401 262949|IGP|195.66.236.175|0|1|13030:1 13030:3 13030:50000 13030:51502 13030:7209|NAG||
BGP4MP|1444498442|A|195.66.236.175|13030|177.154.80.0/22|13030 12989 28640 262401 262401 262401 262401 262401 262401 262401 262401 262949|IGP|195.66.236.175|0|1|13030:1 13030:3 13030:50000 13030:51502 13030:7209|NAG||


The above command outputs 114 announcements, and 11 withdrawals as observed by 14 unique peer ASns.


BGP Stream elems observed by RIS collectors updates in 2 minutes

The following command outputs the BGPStream elems contained in the BGPStream records that comply with the following filters:

  • are contained in Updates dumps generated by RIS collectors
  • their timestamp is in the interval Sat, 10 Oct 2015 17:34:00 UTC - Sat, 10 Oct 2015 17:36:00 UTC
$ bgpreader -w 1444498440,1444498560 -p ris -t updates
U|A|1444498440|ris|rrc12|13237|80.81.192.74|212.93.166.0/24|80.81.192.74|13237 1299 174 39386 39386 39386 39386 39919|39919|1299:25000||
U|A|1444498440|ris|rrc12|13237|80.81.192.74|91.151.162.0/24|80.81.192.74|13237 1299 174 39386 39386 39386 39386 39919|39919|1299:25000||
U|A|1444498440|ris|rrc12|13237|80.81.192.74|212.93.177.0/24|80.81.192.74|13237 1299 174 39386 39386 39386 39386 39919|39919|1299:25000||
...
U|A|1444498560|ris|rrc12|13237|80.81.192.74|214.26.240.0/24|80.81.192.74|13237 1299 209 721 27065 1733 27067 5800|5800|1299:20000||
U|A|1444498560|ris|rrc12|25220|80.81.194.140|214.26.240.0/24|80.81.194.140|25220 3356 209 721 27065 1733 27067 5800|5800|3356:2 3356:22 3356:86 3356:501 3356:666 3356:2065||
U|A|1444498560|ris|rrc12|25220|80.81.194.140|214.13.75.0/24|80.81.194.140|25220 3356 209 721 27065 1733 27067 5800|5800|3356:2 3356:22 3356:86 3356:501 3356:666 3356:2065|| 


The above command outputs 181,118 announcements, 151 state messages, and 7,882 withdrawals as observed by 13 collectors (and 209 unique peer ASns).


BGP Stream elems with filters

The following command outputs the BGPStream elems contained in the BGPStream records that comply with the following filters:

  • are contained in Updates dumps generated by rrc06 or route-views.jinx
  • their timestamp is in the interval Wed, 01 Apr 2015 00:07:27 UTC - Wed, 01 Apr 2015 00:07:54 UTC
  • the peer ASn is either 25152 or 37105
  • if the message is an announcement, then
    • the announced prefix is equal or a more specific of 2620:110:9004::/40, 154.73.128.0/17, or 202.70.88.0/21
    • at least one community matches one of the following filters 2914:* (the ASn field is equal to 2914), or *:300 (the value field is equal to 300).
$ bgpreader  -t updates -c rrc06 -c route-views.jinx -w1427846847,1427846874 -j 25152 -j 37105 -k 2620:110:9004::/40 -k 154.73.128.0/17 -k 202.70.88.0/21 -y 2914:* -y *:300
U|A|1427846850|ris|rrc06|25152|202.249.2.185|202.70.88.0/21|202.249.2.185|25152 2914 15412 9304 23752|23752|2914:410 2914:1408 2914:2401 2914:3400||
U|A|1427846860|ris|rrc06|25152|202.249.2.185|202.70.88.0/21|202.249.2.185|25152 2914 15412 9304 23752|23752|2914:410 2914:1408 2914:2401 2914:3400||
U|A|1427846871|ris|rrc06|25152|2001:200:0:fe00::6249:0|2620:110:9004::/48|2001:200:0:fe00::6249:0|25152 2914 3356 13620|13620|2914:420 2914:1001 2914:2000 2914:3000||
U|A|1427846874|routeviews|route-views.jinx|37105|196.223.14.46|154.73.136.0/24|196.223.14.84|37105 37549|37549|37105:300||
U|A|1427846874|routeviews|route-views.jinx|37105|196.223.14.46|154.73.137.0/24|196.223.14.84|37105 37549|37549|37105:300||
U|A|1427846874|routeviews|route-views.jinx|37105|196.223.14.46|154.73.138.0/24|196.223.14.84|37105 37549|37549|37105:300||
U|A|1427846874|routeviews|route-views.jinx|37105|196.223.14.46|154.73.139.0/24|196.223.14.84|37105 37549|37549|37105:300||


RRC00 in real-time

The following command outputs the BGPStream elems contained in the BGPStream records that comply with the following filters:

  • are contained in RIBs and _ Updates dumps_ generated by the RIS RRC00 collector
  • their timestamp is greater or equal to Sat, 15 Oct 2015 17:12:00 UTC
$ bgpreader -w 1444929120 -c rrc00 
U|A|1444929120|ris|rrc00|1836|2a01:2a8::3|2c0f:fe90::/32|2a01:2a8::3|1836 174 6453 30844 37105 37105 37105 36943|36943|1836:110 1836:6000 1836:6031||
U|A|1444929120|ris|rrc00|1836|2a01:2a8::3|2c0f:fe90::/32|2a01:2a8::3|1836 6939 30844 37105 37105 37105 36943|36943|1836:3200 1836:3210||
U|A|1444929120|ris|rrc00|1836|146.228.1.3|168.128.104.0/21|146.228.1.3|1836 3356 2914 44568 44568|44568|1836:110 1836:6000 1836:6001 2914:410 2914:1203 2914:2201 2914:3200 3356:2 3356:22 3356:86 3356:502 3356:666 3356:2066||
U|A|1444929120|ris|rrc00|8758|212.25.27.44|62.112.24.0/21|212.25.27.44|8758 8220 47377|47377|8220:65080 8220:65401 8758:110 8758:300||
...


The above command operates continuously. As soon as data from RRC00 becomes available to BGPStream it is printed out.


Other data interfaces: how to use singlefile

The following command outputs the BGPStream elems contained in the BGPStream records that comply with the following filters:

  • are contained in http://archive.routeviews.org/bgpdata/2015.10/UPDATES/updates.20151016.1630.bz2
  • their timestamp is in the interval Fri, 16 Oct 2015 16:30:00 UTC - Fri, 16 Oct 2015 16:42:35 UTC

$ bgpreader -d singlefile -o upd-file,http://archive.routeviews.org/bgpdata/2015.10/UPDATES/updates.20151016.1630.bz2 -w 1445013000,1445013755 U|A|1445013000|singlefile_ds|singlefile_ds|13030|213.144.128.203|212.22.66.0/24|213.144.128.203|13030 12389 12389 12389 12389 12389 12389 41938 8359 50618 35189 201432|201432|13030:1 13030:3 13030:50000 13030:51202 13030:7179|| U|A|1445013000|singlefile_ds|singlefile_ds|3130|147.28.7.2|76.191.107.0/24|147.28.7.2|3130 11404 22059|22059|3130:380|| U|A|1445013000|singlefile_ds|singlefile_ds|3130|147.28.7.2|64.34.125.0/24|147.28.7.2|3130 2914 7922 11404 22059|22059|2914:420 2914:1008 2914:2000 2914:3000 3130:380|| U|A|1445013000|singlefile_ds|singlefile_ds|8492|85.114.0.217|185.37.192.0/23|85.114.0.217|8492 21011 34251 203957|203957|8492:1201 8492:1602|| U|A|1445013000|singlefile_ds|singlefile_ds|3549|208.51.134.246|185.37.192.0/23|208.51.134.246|3549 3257 21011 34251 203957|203957|3549:2203 3549:2623 3549:30840 3549:31250|| ... U|W|1445013755|singlefile_ds|singlefile_ds|2914|129.250.0.11|199.251.30.0/24|||||| U|W|1445013755|singlefile_ds|singlefile_ds|2914|129.250.0.11|137.10.0.0/16|||||| U|A|1445013755|singlefile_ds|singlefile_ds|1299|80.91.255.137|185.13.64.0/22|80.91.255.137|1299 3356 44141|44141||| U|A|1445013755|singlefile_ds|singlefile_ds|1299|80.91.255.137|91.236.153.0/24|80.91.255.137|1299 174 202140|202140|||


The above command outputs 114 announcements, and 11 withdrawals as observed by 14 unique peer ASns.