What's New in BGPStream V2

BGPStream V2 adds several new features, including native support for true real-time data access via BMP and performance improvements. See the Changes section below for a summary and the GitHub releases page for details.

See the Live BMP Stream section below for help accessing the new realtime BMP stream.

Release available now. The V2 API is now stable and has been tested by many beta testers (including the ARTEMIS project). Click the button below to get started with BGPStream V2.

Download BGPStream V2.2.0

Changes

New Features and Improvements in 2.2.0

  • Improved Kafka transport
    • Kakfa transport performance improvement
    • Kakfa now tries blocking for a while before returning an EOF
    • Prefers round-robin rebalance strategy
  • Imporved message parsing
    • Ensure BGPStream can move past invalid messages
    • Fix "incompatible pointer type" warning in pfx_set code
    • Prefers round-robin rebalance strategy
    • Support RFC 8050 MRT Additional Path extension
    • Fix parsing of missing OPEN/UPDATE messages

New Features and Improvements in 2.1.0

  • Iteration over prefix set

New Features and Improvements in 2.0.0

  • Native BMP support
    • libBGPStream now supports processing raw BMP data in the same way as MRT.
    • Currently the "singlefile" and the new "kafka" data interfaces provide access to BMP data. The "singlefile" interface can be used to process local dumps of raw BMP data whereas the "kafka" interface can be used to access BMP (or MRT) data from a Kafka cluster.
  • Realtime data stream via RIS Live
    • libBGPStream now supports obtaining realtime BGP data from RIPE's RIS Live BGP message stream .
    • RIS Live stream access point is retrieved from data broker and is directly accessible using "kafka" transport.
    • bgpreader now default to live-stream mode if no time window is specified, which include all archive data as they come in, as well as live stream data sources like RIS-Live and OpenBMP stream.
    • To start streaming using bgpreader, just specify that we want data from project "ris-live" without any time-window: bgpreader -p ris-live
    • For more detailed usage of the RIS Live stream via BGPStream, please checkout our tutorial on accessing live stream data source using bgpreader and using pybgpstream.
  • Realtime data stream via RouteViews BMP stream
    • libBGPStream now supports obtaining realtime BGP data from RouteViews' BMP message stream.
    • RouteViews stream access point is retrieved from data broker and is directly accessible using our new "http" transport.
    • To start streaming using bgpreader, just specify that we want data from project "routeviews-stream" without any time-window: bgpreader -p routeviews-stream
    • For more detailed usage of the RouteViews BMP stream via BGPStream, please checkout our tutorial on accessing live stream data source using bgpreader and using pybgpstream.
  • Realtime data stream via OpenBMP
    • libBGPStream now supports obtaining realtime BGP data from a Kafka cluster.
    • Access to private OpenBMP feeds is supported using the "kafka" data interface, whereas the "broker" data interface may be used to access the public BMP stream provided as part of the CAIDA BGPStream project.
    • See the Private BMP Stream tutorials below for more information.
  • Local caching of dump files (optional)
    • Data files processed by the broker can now be cached to a local directory which is checked before downloading a dump file.
    • Previously, when using BGPStream to repeatedly process the same data (e.g., when testing/debugging code), poor network connectivity could add overhead to processing time.
    • The caching implementation is thread safe and can support parallel instances of BGPStream (either as threads or separate processes).
    • The cache can be enabled by setting the cache-dir parameter of the "broker" data interface. E.g., by passing -o cache-dir=/path/to/cache to bgpreader, or by calling stream.set_data_interface_option("broker", "cache-dir", "/path/to/cache") from PyBGPStream.
    • Thanks to Mingwei Zhang for contributing this feature.
  • New high-level PyBGPStream API (prototype)
    • There is now a high-level "Pythonic" API for accessing BGP data using BGPStream.
    • Previously the only Python interface was _pybgpstream, a low-level, almost exact bindings to the libBGPStream C API.
    • See the API docs below for more information.
  • New filter interface
    • libBGPStream now supports a "BPF-like" syntax for specifying filters.
    • E.g., collector route-views.sg and type ribs and prefix exact 192.172.226.0/24 would extract only RIB records matching the given prefix from the route-views.sg collector.
    • For BGPReader, this feature may be accessed using the -f option.
    • For PyBGPStream, filters can be specified either using the filter parameter to the BGPStream constructor, or by calling the parse_filter_string method on an existing stream.
    • While the previous APIs for specifying filters are still available, this interface should be used as new filter types will only be available through this interface, and eventually the other filter methods will be deprecated.
    • See the documentation for more information.
    • Thanks to Shane Alcock for contributing this feature.
  • BGPStream Docker Image
    • We now provide official Docker image for BGPStream and PyBGPStream. Users can get access to both the development environment and the bundled bgpreader tool directly from the BGPStream docker image.
    • You can access the docker with the image name caida/bgpstream on DockerHub.
    • For more usage examples, please check out the tutorial page at BGPStream Docker tutorials .

Bug Fixes

  • Patricia Tree Fixes
    • We have made several fixes to the patricia trie implementation used by libBGPStream.
    • For most users this only affected prefix filtering. Previously all prefix filters were treated as "ANY" filters (rather than MORE/EXACT/LESS).

Performance Improvements

  • New MRT parser
    • Along with developing a BMP parser, we also developed a new MRT parser, libParseBGP to replace the fork of libbgpdump we used in V1.
    • libParseBGP should be much faster than libbgpdump (~30% faster in some tests we ran) and is designed to be easier to maintain and extend.
    • It is possible/likely that there are some MRT peculiarities that are not correctly handled by libParseBGP (yet). If you come across any problems, please contact us.
  • Completely re-designed resource management
    • To support addition of other data formats (e.g., BMP), and data transport mechanisms (e.g., Kafka), we completely re-designed the core resource management components of libBGPStream.
    • In addition to simplifying adding support for new formats, the new implementation appears to perform ~10% better than V1.
  • Improved Record API
    • We changed the get_next_record API to return a borrowed pointer to an internal record structure, rather than filling a structure passed by the user.
    • This minimizes memory allocations and copying within libBGPStream, improving performance.

PyBGPStream Improvements

  • BGPElem fields attribute is now cached so that subsequent field accesses do not needlessly rebuild the entire fields dictionary.
  • By popular demand, communities in PyBGPStream are now returned as a set of "asn:value" strings

Misc. Improvements

  • Added bgpstream_pfx_copy method
  • Added bgpstream_as_path_get_origin_val method to extract origin ASN as simple integer. (Contributed by Samir Al-Sheikh.)
  • No longer require time interval to be set. This simplifies use of the "singlefile" data interface.
  • Added documentation of the AS Path string format.
  • BGPReader data interface options are now specified as -o <param>=<value> rather than -o <param>,<value>.

Known Issues

  • RIS live stream disconnects when mixing batch and real-time resources (#116)
  • Kafka transport appears to block when consuming from non-existent topic (#127)

Public Live BMP Stream

As part of the BGPStream project, we have begun to provide a public BMP feed. Currently we are only providing data from a few Route Views and Cisco Research peers, but we expect additional peers to be added soon. (If you would like to contribute a feed, please contact us at bgpstream-info@caida.org.)

We are providing access to these feeds by way of a publicly-accessible, read-only, Kafka cluster (bmp.bgpstream.caida.org:9092) which contains raw BMP data encapsulated in a custom OpenBMP message header. (We plan to contribute the code we developed to generate these headers back to the upstream OpenBMP repository.)

Accessing the Stream

In this release, access to this live BMP feed is available via the broker data interface alongside the traditional MRT-based data from Route Views and RIPE RIS. This data has caida-bmp as the project name, so can be filtered using the standard project filters. See below for examples of how to use this interface.

From BGPReader

Accessing the feed from BGPReader is as simple as choosing the "caida-bmp" project without time-window:

bgpreader -p caida-bmp

From PyBGPStream

import pybgpstream
for elem in pybgpstream.BGPStream(project="caida-bmp"):
    print(elem)

Private OpenBMP Collector

If you have a router that you would like to monitor using OpenBMP and BGPStream, you can use the dockerized OpenBMP deployment from the OpenBMP project, and then simply configure the "kafka" data interface of BGPStream to stream data from your collector.

1. Setting up the OpenBMP collector

You will first need to deploy and configure an OpenBMP collector and Kafka instance. The easiest way to do this is using the docker container provided by the OpenBMP project. See the OpenBMP documentation for a detailed tutorial. (You do not need to run any consumers to use OpenBMP with BGPStream.)

2. Configure router to send BMP data to the collector

See the OpenBMP documentation for some sample router configurations.

3. Install BGPStream

See the BGPStream Install instructions for more information.

5. Configure BGPStream to stream data from your collector

Use the "kafka" data interface of BGPStream, and configure it to point to your OpenBMP Kafka instance.

For example, if you are using the BGPReader CLI: 

bgpreader -d kafka \
  -o brokers=<docker_host>:9092 \
  -o topic=openbmp.bmp_raw

Or, if you are using the PyBGPStream Python API here is a minimal working example:

import pybgpstream
stream = pybgpstream.BGPStream(data_interface="kafka")
stream.set_data_interface_option("kafka", "brokers", "<docker_host>:9092")
stream.set_data_interface_option("kafka", "topic", "openbmp.bmp_raw")
for elem in stream:
    print(elem)

Feedback

We appreciate your support of this project, and your willingness to help us test the next version. Please create an issue on either the libBGPStream or PyBGPStream GitHub repos detailing any problems you have, or alternatively, contact bgpstream-info@caida.org.