fleet-epm-caching.md

High-level overview of EPR endpoints

• Searching
  • https://epr.elastic.co/search?package=apm
  • Used for the “grid” pages in the integrations UI
• Package info
  • https://epr.elastic.co/package/apm/8.0.0/
  • Used for fetching package info objects for registry packages when they are not cached
• Package download or "archive"
  • https://epr.elastic.co/epr/apm/apm-8.0.0.zip/
  • Used for anything that involves a package’s data/configuration specified in manifest.yml files and elsewhere in the directory structure of the package
  • e.g. Package details screen /app/integrations/detail/system-1.10.0/overview
  • e.g. Package installation

Fleet’s approach to caching package data

• Fleet utilizes two in-memory caches for caching package data

  • An "archive entry cache" for caching each file included in a package's .zip archive keyed by a filepath: Code
  • A "package info cache" for caching packageInfo objects keyed by a ${name}-${version} string: Code

• Whenever Fleet calls getPackageFromSource (Code) we will attempt to pull the package archive and package info from their respective caches

  • If the cached archive does not exist, we will download the .zip archive from EPR, “unpack” it using this method: https://github.com/elastic/kibana/blob/main/x-pack/plugins/fleet/server/services/epm/archive/index.ts#L33-L61
  • If the cached package info object does not exist, the approach is similar. We will fetch the package info from EPR and store it in the package info cache for future requests.
  • If the package is installed, however, we will build the cache from the installed assets instead in getEsPackage: Caller and source

• Caching package archives

  • When Fleet calls getRegistryPackage, we'll attempt to pull the package archive file list from the cache. If it doesn't exist, the full package archive is fetched from EPR, parsed, and cached. Code

Problems with the current approach

• In-memory caching is inherently unreliable in high-availability Kibana environments with multiple Kibana instances
• Package data in the cache doesn’t expire, so if a user went through and visited every single package they’d probably crash Kibana by causing it to go out of memory
• There are inconsistencies between how Fleet does its “parsing” of package archives and how the same logic is implemented in the package registry service to serve endpoints like this.
  • Hopefully band-aid fix in elastic/kibana#126915 is enough to keep things stable here, but this is a point of brittleness for sure
• Fleet makes inefficient use of package info knowing this cache is primed and available, e.g.
• The archive parsing and caching code is generally difficult to reason about, unmaintained, and undocumented

Relevant issues on the backlog

• elastic/kibana#115032
• elastic/kibana#125794