BOSH Links: Why and How

bosh-links-why-and-how.md

BOSH Links: Why and How

Audience: anyone in the BOSH ecosystem, whether you work on something open-source or proprietary

"BOSH Links" is a feature which simplifies how data is shared between BOSH-deployed jobs that need to collaborate with one another (e.g. a web server and its backing database). Like many BOSH things, the whole "BOSH Links" thing can seem counter-intuitive at first, and it may not be clear why things are the way they are. This note hopes to show some of the powerful benefits of BOSH Links, and provide examples and explanations that make things more intuitive.

1. Why?
2. BASIC: A simple problem statement
3. BASIC: Putting the puzzle pieces together
4. ADVANCED: But what about...
5. ADVANCED: More features
6. ADVANCED: Links++
7. Action items for release authors

The "Why?" section tells you why Links were invented and why you would care, as a release author building releases for an operator to deploy, or as an operator juggling a bunch of BOSH deployments. The "BASIC" sections will familiarize you with the basic mechanics of Links, introducing some problems with manifests that need solving, showing how they can be solved with Links. The discussion here is fairly low-level and concrete, giving you most of the requisite know-how to work with links.

After reading the BASIC sections, take a break 😌 and tinker around with some releases and manifests yourself.

Sooner or later, you're probably going to have a bunch of "but what about..." questions about Links, common use cases which may not appear to play nicely with Links. Those will be addressed in the "ADVANCED: But what about..." section. The problems raised in the "BASIC" sections could be solved without Links, so it's important to see "ADVANCED: More features" to see how the same Links abstractions can be used to solve the real-world problems that can't easily be solved today in the BOSH ecosystem.

It's also really important to understand that the bigger problems that need to be solved when it comes to managing BOSH releases and manifests aren't solved by one or two features alone. Links need to be combined with other new features in the BOSH ecosystem to see the full benefits, which is discussed in the "ADVANCED: Links++" section.

Finally, if you're a release author, check out the final section to see how your work fits into bringing these benefits to the BOSH ecosystem.

1. Why?

As a standalone feature, links help simplify BOSH manifests, making them:

• easier to read; and
• simpler to build.

Bigger picture, these things would ideally lower the barrier to entry to using BOSH and consuming BOSH-releases, and enable lighterweight tooling to generate manifests, so that people's BOSH workflows aren't subject to "lock-in" (hard dependencies on tools like spiff, spruce, enaml, or Ops Manager). Furthermore, a real-world operator is managing multiple deployments (maybe Cloud Foundry, MySQL, Redis, RabbitMQ, etc.), not just one. Managing all the data that needs to be shared across them is very difficult and is a problem a lot of the aforementioned tools try to solve, all in their own different ways. BOSH Links are a crucial part of a BOSH-native solution to this problem.

Amplified by other "BOSH 2.0" features such as Global Networking, Cloud Config, Operations Files, and CredHub, Links allow manifests to be further simplified, removing most environment-specific details from manifests such as credentials, IPs, IaaS-specific cloud properties, and other common structural features (e.g. backing Cloud Foundry with an internal WebDAV blobstore vs. S3). This additionally makes manifests:

• useful to share; and
• reasonable to extend.

Ideally, these things further reduce friction within the ecosystem, and reduce error-prone duplication of effort, especially in cases where one wants to build a commercial distribution of some BOSH-deployable product on top of an open-source deployment configuration.

2. BASIC: A simple problem statement

Imagine a simple deployment with a web server and a database. The web server will need to know the address and port of the database, as well as username/password access credentials. Oftentimes, we don't want to have to:

1. specify the username and password twice in the manifest (once for the database to configure itself, and once for the web server to know how to talk to the database);
2. specify the username or password even once, something else could just generate some secure credentials, it just matters that web server and database have the same configuration;
3. specify the port explicitly, the database should default to binding to some port, and web server should have a way of discovering this port;
4. specify an IP for the database twice (again, once for the database VM itself, and once for the web server to know where the database is); and
5. specify the database IP even once, something else can pick an IP and just make sure both jobs know about this IP.

We'll ignore point 2 and 5 for now, but the rest can be addressed by BOSH Links alone.

Stepping back for a second, let's look at this from first principles and level-set the conversation. We want to run a database process and a web server process. Running these processes usually involes a start script and some configuration files. We package the source code and/or executable binaries together with templates for start scripts and configuration files inside our BOSH releases. A BOSH deployment manifest includes environment-specific values for properties and network addresses; these values are used to render the templates and those rendered templates turn into the actual start scripts and configuration files used at runtime.

Rather than having to specify this data multiple times in the manifest (once for consumers like web servers and once for providers like the database), Links allow a form of dependency injection, where the parameters for constructing/configuring the database only need to be specified (at most) once, and the web server can simply declare its dependency on the database configuration. BOSH Links then injects that data in the consuming job's (web server's) template rendering context. Another analogy is service discovery, in this case it's data discovery; the consumer job simply declares it wants to find "database" data, and ta-da, there it is in its template rendering context, an operator didn't need to manually feed it that data.

3. BASIC: Putting the puzzle pieces together

Let's look at how Links play into:

• release job specs;
• release job templates; and
• a deployment manifest

to solve the problem.

The web_server's job spec declares its dependency:

~/workspace/releases/app/jobs/web_server/spec

---
name: web_server

templates:
  web_server_ctl.sh.erb: bin/web_server_ctl
  config.json.erb: config.json

packages:
  - web_server_binary

# DEPENDENCIES DECLARED IN THIS SECTION
consumes:
- {name: database, type: database}

properties:
  greeting:
    description: Greeting to display to the user when they reach the landing page
    default: Hello, user!
  port:
    description: Port that the web server binds to
    default: 8080

and uses its "database" dependency in one of its templates:

~/workspace/releases/app/jobs/web_server/templates/config.json.erb

{
  "database_address": "<%= link("database").instances[0].address %>",
  "database_port": <%= link("database").p("port") %>,
  "database_username": "<%= link("database").p("username") %>",
  "database_password": "<%= link("database").p("password") %>",
  "greeting": "<%= p("greeting") %>",
  "port": <%= p("port") %>
}

There are a couple more pieces to the puzzle, namely the database job spec and the relevant part of the BOSH manifest, but let me bring up a few questions which we'll address after:

1. What does the whole {name: database, type: database} thing mean? Are these magic strings?
2. What is link("database"), and what is .instances[0] about, and what is that .address accessor?
3. Speaking of which, what is the .p(...) method about on the link("database") object?

Here's the database job's spec:

~/workspace/releases/app/jobs/database/spec

---
name: database

templates:
  database_ctl.sh.erb: bin/database_ctl
  config.json.erb: config.json

packages:
  - database_binary

# EXPORTED DATA DECLARED IN THIS SECTION
provides:
- {name: database, type: database, properties: [port, username, password]}

properties:
  port:
    description: Port that the database binds to
    default: 5432
  username:
    description: Username used to access the database
  password:
    description: Password used to access the database
  max_connections:
    description: Maximum number of database connections
    default: 500

And let's look at an example manifest:

~/workspace/deployments/galileo_environment/my_app.yml

---
name: my_app

releases:
- name: app
  version: latest

instance_groups:
- name: web_server
  jobs:
  - release: app
    name: web_server
    properties:
      greeting: "Namaste!"
  # <uninteresting>
  instances: 3
  azs: [z1, z2, z3]
  networks: [{name: default}]
  vm_type: default
  stemcell: default
  # </uninteresting>
  
- name: database
  networks: [{name: default, static_ips: [10.0.16.4]}]
  jobs:
  - release: app
    name: database
    properties:
      username: admin
      password: passw0rd
  # <uninteresting>
  instances: 1
  azs: [z1]
  vm_type: default
  stemcell: default
  # </uninteresting>

# <uninteresting>
update:
  canaries: 1
  canary_watch_time: 10000-600000
  update_watch_time: 10000-600000
  max_in_flight: 1
  serial: true

stemcells:
- alias: default
  os: ubuntu-trusty
  version: latest
# </uninteresting>

If we look at these examples in the context of the 5 items enumerated in the simple problem statement, note that the web_server instance group is not provided any configuration in the manifest regarding the address or credentials of the database. Those data are only specified once, and only on the database instance group (namely 10.0.16.4, admin, and passw0rd). Also note that the database port is not specified in the manifest at all!

It's unfortunate that I have to specifify the IP at all, since now this manifest may not work in your environment if you happen to have a different IP range. And it's unfortunate that the username and password are in this manifest, both because it creates busy-work for us to generate them and put them in the manifest, and it's really bad from a security perspective to have these plainly visible in the manifest. We'll address these points later, but let's first answer some of the questions raised earlier.

About those magic strings...

We see the string database in a ton of places. The web_server job spec consumes a link with name: database and type: database. One of the web_server's templates references link("database"). The database job itself has name: database (at the top level), and it also provides a link with name: database and type: database. Finally, the manifest has an instance group with name: database, and that instance group has a job with name: database. Let's adjust the example files so that only the strings that truly need to match do so, we'll give things contrived names to clarify their purpose (and we'll remove the <uninteresting> bits):

~/workspace/releases/app/jobs/web_server_process/spec

---
name: web_server_process

templates:
  web_server_process_ctl.sh.erb: bin/web_server_process_ctl
  config.json.erb: config.json

packages:
  - web_server_binary

# DEPENDENCIES DECLARED IN THIS SECTION
consumes:
- {name: local_database_data_object_reference, type: database_data_class}

properties:
  greeting:
    description: Greeting to display to the user when they reach the landing page
    default: Hello, user!
  port:
    description: Port that the web server binds to
    default: 8080

~/workspace/releases/app/jobs/web_server_process/templates/config.json.erb

{
  "database": {
    "address": "<%= link("local_database_data_object_reference").instances[0].address %>",
    "port": <%= link("local_database_data_object_reference").p("port") %>,
    "username": "<%= link("local_database_data_object_reference").p("username") %>",
    "password": "<%= link("local_database_data_object_reference").p("password") %>"
  },
  "greeting": "<%= p("greeting") %>",
  "port": <%= p("port") %>
}

~/workspace/releases/app/jobs/database_process/spec

---
name: database_process

templates:
  database_process_ctl.sh.erb: bin/database_process_ctl
  config.json.erb: config.json

packages:
  - database_binary

# EXPORTED DATA DECLARED IN THIS SECTION
provides:
- {name: relevant_later, type: database_data_class, properties: [port, username, password]}

properties:
  port:
    description: Port that the database binds to
    default: 5432
  username:
    description: Username used to access the database
  password:
    description: Password used to access the database
  max_connections:
    description: Maximum number of database connections
    default: 500

~/workspace/deployments/galileo_environment/my_app.yml

---
# <uninteresting></uninteresting> bits removed for clarity!

name: my_app

releases:
- name: app
  version: latest

instance_groups:
- name: web_server_instances
  jobs:
  - release: app
    name: web_server_process
    properties:
      greeting: "Namaste!"

- name: database_instance
  networks: [{name: default, static_ips: [10.0.16.4]}]
  jobs:
  - release: app
    name: database_process
    properties:
      username: admin
      password: passw0rd

So now we have:

• The deployment manifest has an instance group called database_instance, which represents the database instance that will be deployed to the cloud.
• That database instance runs one job, the database_process, which refers to the BOSH job database_process in the app release.
• The database_process job exports or provides some data, and the "type" of data it exports is called database_data_class. The name of the exported data is relevant_later because it's irrelevant for now.
• The web_server_process consumes data of type database_data_class. This is essentially where it declares the dependency it needs injected. I used the class suffix to hint at that. Indeed, the fact this is under type: database_data_class, the type part is sort of a hint to the fact that it acts like a class name that you're dependency-injecting. On the other hand, this type key is very unfortunately named, and may be better to think of as a "tag" or "label" that simply used for matching, there's no underlying "type system" that makes sure your data "compiles", there are no "standard library" types like "URL" or "RSAKeyPair" that you might imagine would be common link types, they're just arbitrary, meaningless strings.
• The web_server_process names the link it consumes local_database_data_object_reference, and then at template-render time, this reference is used to actually access the data its depending on, which we see in the config.json.erb as link("local_database_data_object_reference").

About that link("...") object, and its .instances[0], and the .address accessor

Notice that we only had to specify the IP on the database_instance (as 10.0.16.4), nowhere in the manifest under the web_server_instances did we include this IP. When a job consumes a link, it gets a bunch of data for free about the instances (in this case database_instance) that are running the job (in this case database_process) which provide the link it consumes. NOTE: the web server needs to know not only details specific to how the database process is configured (e.g. username and password), but also some infrastructure-y/network-y data about the VM(s) this process is running on, (e.g. address).

link("local_database_data_object_reference").instances is an array of objects representing instances/VMs on which the job-providing-the-database_data_class-link is running. We only have one such instance (see instances: 1 in the manifest), hence link("local_database_data_object_reference").instances[0]. This object has the data about that VM, including its address, accessed via the .address accessor. If our web server were designed to talk to some HA database that had multiple instances, each with their own address, you might see something like this instead:

{
  "database": {
    "addresses": <%= link("local_database_data_object_reference").instances.map(&:address) %>, 
    ...
  }
}

There's all sorts of other data you get for free about the instances, see the "Available instance object methods" here.

The .p(...) method on the link(...) object

There's a bunch of job-configuration-specific data which you don't get "for free". However, if a job providing a link excplicitly exports certain properties, then consuming jobs can access the values. Note that while the database_process job has a property called max_connections declared in its job spec, that isn't named amongst the properties provided in the relevant_later link. That's because it's an internal detail that something talking to this database doesn't really need to know about. It does export the port, username, and password properties.

From the database_process perspective, the port value comes from the default 5432, whereas the username and password are specified in the manifest where the database_process job itself is configured. From the web_server_process perspective, these properties are all accessed in the job templates, via the p method on the link("local_database_data_object_reference") object. This behaves similar to the top-level p method in the template rendering context. p("greeting") is used to extract the value of the greeting property defined directly in the web_server_process job spec, and ...p("port") is used to grab the port data off of the local_database_data_object_reference link.

Aside: It's worth highlighting that this can definitely seem a bit awkward at first. A job specifies that it provides a certain link, and then you have a job that can consume that link. When the job consumes the link(...) object, it gets access to some of the property configuration data of the providing job via link(...).p(...) which makes sense -- the job declares that it's providing a link, and thus I can access properties of that job. But I can also get data about the VMs (or more generally, instances) running that job at deploy time via the link(...).instances array.

The instance group doesn't declare explicitly that it's providing any link, but the instance group happens to run a job, and that job provides the link, and so I get access to necessary information about the instances as well. The abstraction may seem a little funny, but you definitely need things like the IP and port of a database you depend on -- the port is something the database job or process care about, the IP is something that has more to do with the VM or instance on which that database process is running. So link(...) provides both of these kinds of data.

Break Time

4. ADVANCED: But what about...

Backwards compatibility with existing job templates, job specs, and deployment manifests

Before links, the web_server_process job spec would have had a bigger properties section that might have looked like this:

properties:
  greeting:
    description: Greeting to display to the user when they reach the landing page
    default: Hello, user!
  port:
    description: Port that the web server binds to
    default: 8080
  database.address:
    description: Address of the database used to back this web server
  database.port:
    description: Port of the database used to back this web server
  database.username:
    description: Username used to access the backing database
  database.password:
    description: Password used to access the backing database

and the template might have looked like this:

~/workspace/releases/app/jobs/web_server_process/templates/config.json.erb

{
  "database": {
    "address": "<%= p("database.address") %>",
    "port": <%= p("database.port") %>,
    "username": "<%= p("database.username") %>",
    "password": "<%= p("database.password") %>"
  },
  "greeting": "<%= p("greeting") %>",
  "port": <%= p("port") %>
}

The unfortunate thing in this world is the manifest would have had to have a lot more duplicated junk, you'd typically see things like this:

~/workspace/deployments/galileo_environment/my_app_without_links.yml

---
name: my_app_without_links

releases:
- name: app
  version: latest

instance_groups:
- name: web_server_instances
  jobs:
  - release: app
    name: web_server_process

- name: database_instance
  networks: [{name: default, static_ips: [10.0.16.4]}]
  jobs:
  - release: app
    name: database_process

properties:
  # For web server
  greeting: "Namaste!"
  database:
    address: 10.0.16.4
    username: admin
    password: passw0rd
    port: 5432
  
  # For database
  username: admin
  password: passw0rd
  port: 5432

I have to specify the IP, username, and password twice. The database port defaults to 5432 but without links, the web_server_process job needs to be configured with that data explicitly. For sanity, to avoid coupling the database.port property provided to the web_server_process in the deployment manifest with the port property's default value in the database_process job spec, it's best to also explicitly set the database_process's port value in the manifest to match the value given to the web_server_process in the manifest to make sure they truly match, even if you happen to provide the same value that the database_process job spec happens to default to currently. So there's actually triplication going on there, not mere duplication.

Well, links are nice to avoid these issues, but how do I support both ways of configuring my jobs for compatibility reasons? In the short term, you might want to express logic like, "if the old-style properties are provided, honour those; if not, but something else in my deployment is providing the data via a link I can consume, then use that data". Check out how the NATS smoke-tests job determines the port of the nats server job here:

releases/nats-release/jobs/smoke-tests/templates/config.json.erb

<%
  ...
  
  nats_port = nil
  if_p("nats.port") { |port| nats_port = port }
  unless nats_port
    nats_port = link("nats").p("nats.port")
  end
%>

{
  ...
  "Port": <%= nats_port %>
}

The key is the usage of the if_p helper function.

The logic above "prefers" the "old-style" properties, and only if they're not present, looks for the link. For compatbility with older BOSH Directors, you want this logic because they won't know anything about the link method, and you don't want the Director to reach that line of code at template-render time. Operators are recommended to upgrade their Directors to a version that supports Links (which have been GA for several months now), at which point one can invert the logic to prefer links over "old-style" properties.

In the medium-term future, start to use the if_link helper and optional: true when specifying Links consumed by your release jobs, so that your ERB templates look more like this:

<%
  ...
  
  nats_port = nil
  if_link("nats") { |nats_link| nats_port = nats_link.p("nats.port") }
  unless nats_port
    nats_port = p("nats.port")
  end
%>

{
  ...
  "Port": <%= nats_port %>
}

In the long-term, one can drop support for "old-style" properties altogether. Worst case, the link values can be specified explicitly in the manifest (just as properties are today, but in a different location).

What happens if multiple things provide the same type of link?

In the basic example, the name of the link being provided wasn't relevant, hence the name relevant_later. BOSH knows to connect the data coming from the database_process to the web_server_process implicitly based on the type of the link, namely database_data_class. But if your manifest happened to have two jobs, in two instances groups (or the same for that matter) which both happened to provide links of the same database_data_class type, how would BOSH disambiguate? By default, it won't, and you'll get a nice error message when you try to bosh deploy, for example:

- Multiple instance groups provide links of type 'database_data_class'. Cannot decide which one to use for instance group 'web_server_instances'.
   my_app.database_instance.database_process.relevant_later
   my_app.another_database_instance.database_process.relevant_later

The types of the consumed and provided links are specified in the BOSH releases, but if you're an operator who only has control of the BOSH manifest, you need some way in the manifest to disambiguate.

Enter as and from. Think of this as being analogous to import renaming in code, e.g.

foo.py

import numpy as np

...

or a Golang analogy:

main.go

package main

import (
  vaultapi "github.com/hashicorp/vault/api"
  consulapi "github.com/hashicorp/consul/api"
)

Here's a contrived example where there are two copies of our web app, one in Hindi and one in French. Each one is backed by a different database (because the database stores the number of views, and we want to see which app is more popular so we want view counts stored in separate databases).

~/workspace/deployments/galileo_environment/my_app_a_b_testing.yml

---
name: my_app_a_b_testing

releases:
- name: app
  version: latest

instance_groups:
- name: hindi_web_server_instances
  jobs:
  - release: app
    name: web_server_process
    properties:
      greeting: "Namaste!"
    consumes:
      local_database_data_object_reference: {from: hindi_database}

- name: french_web_server_instances
  jobs:
  - release: app
    name: web_server_process
    properties:
      greeting: "Bonjour!"
    consumes:
      local_database_data_object_reference: {from: french_database}

- name: hindi_database_instance
  networks: [{name: default, static_ips: [10.0.16.4]}]
  jobs:
  - release: app
    name: database_process
    properties:
      username: admin
      password: surya44chand
    provides:
      relevant_later: {as: hindi_database}

- name: french_database_instance
  networks: [{name: default, static_ips: [10.0.16.5]}]
  jobs:
  - release: app
    name: database_process
    properties:
      username: admin
      password: soleil44lune
    provides:
      relevant_later: {as: french_database}

There's two distinct copies of the database_process job, one in the french_database_instance and one in the hindi_database_instance. They both provide a link of type database_data_class, so BOSH lets you disambiguate the two sets of link data by saying that, for the purpose of this manifest, they will each be known as something else, something unique. One bunch of database_data_class link data shall be known as hindi_database, and the other as french_database. There happen to be two jobs wanting to consume data of type database_data_class, but they want to consume different instances of that data. Locally, they want to reference those consumed links as local_database_data_object_reference, but the French web server is going to get that from french_database and the Hindi web server from the hindi_database.

What if the link types don't match?

You might not have your database and web server in the same release. You might consume the database from some release made by a different team or community. What are the chances they happened to provide a link with type database_data_class? There's no central authority ensuring the consistency of that magic string across release authors. as and from can help with this too. You might want to use a different release which provides a database, and its job provides a link with a much less contrived type than database_data_class, maybe its type is database (like it was before we made the basic example contrived) and its name is postgres (as opposed to relevant_later).

You could do this:

~/workspace/deployments/galileo_environment/my_app_with_other_database.yml

---
name: my_app_with_other_database

releases:
- name: app
  version: latest
- name: other_database # NOTE: we're going to use a database job from a different release
  version: latest

instance_groups:
- name: web_server_instances
  jobs:
  - release: app
    name: web_server_process
    properties:
      greeting: "Namaste!"
    consumes:
      local_database_data_object_reference: {from: explicit_db_link_name}

- name: database_instance
  networks: [{name: default, static_ips: [10.0.16.4]}]
  jobs:
  - release: other_database
    name: database
    properties:
      username: admin
      password: surya44chand
    provides:
      postgres: {as: explicit_db_link_name}

Gotcha: It's more than the link types which need to match, the properties provided by the link, and then consumed via link(...).p(...) need to match. In general, that's not going to be the case. The link ecosystem is very young, and there are no established conventions. Compare for instance the properties provided by the Cloud Foundry community's postgres-boshrelease here to Cloud Foundry's cf-mysql-release here. The web server can't truly swap out alternative implementations of the database_data_class link implementation. Maybe someday!

What about external data from something not managed by BOSH, like an RDS database?

BOSH has no way to know the address, port, or credentials of your RDS database, so you'll have to manually provide this link data. If your release has done the work to make itself backwards compatible with old manifests, you might be tempted to provide this data in the traditional properties sections at the top level of the manifest, or under a particular job listed within an instance group. A better thing to do would be to put all the data in the links whenever possible, so that you can eventually drop support for non-link data. That way, you only have one source of data to worry about and can eventually drop the if_p branching for backwards compatibility in your release templates. You can explicitly spell out what data a job consumes for one of its links directly in the manifest. It's just as desribed on the BOSH docs page on Manual Linking:

~/workspace/deployments/galileo_environment/my_app_with_rds.yml

---
name: my_app_with_rds

releases:
- name: app
  version: latest

instance_groups:
- name: web_server_instances
  jobs:
  - release: app
    name: web_server_process
    properties:
      greeting: "Namaste!"
    consumes:
      local_database_data_object_reference:
        instances:
        - address: teswfbquts.cabsfabuo7yr.us-east-1.rds.amazonaws.com
        properties:
          port: 3306
          username: some-username
          password: some-password
          name: my-app

5. ADVANCED: More features

Thank you webmaster! But our database_process is in another deployment!

What if you want your database to be managed by your DBA, and someone else manages the web server? You might have your database_process and your web_server_process in different deployments, managed by different "BOSH teams" leveraging the new BOSH RBAC features available via the BOSH+UAA integration. You can still link data across deployments:

~/workspace/deployments/galileo_environment/standalone_database.yml

---
name: standalone_database # <== NOTE THIS DEPLOYMENT NAME

releases:
- name: app
  version: latest

instance_groups:
- name: database_instance
  networks: [{name: default, static_ips: [10.0.16.4]}]
  jobs:
  - release: app
    name: database
    properties:
      username: admin
      password: surya44chand
    provides:
      relevant_later:
        as: explicit_db_link_name
        shared: true

~/workspace/deployments/galileo_environment/standalone_webserver.yml

---
name: standalone_webserver

releases:
- name: app
  version: latest

instance_groups:
- name: web_server_instances
  jobs:
  - release: app
    name: web_server_process
    properties:
      greeting: "Namaste!"
    consumes:
      local_database_data_object_reference:
        from: explicit_db_link_name
        deployment: standalone_database # <== DEPLOYMENT NAME REFERENCED HERE

A few things:

• To expose data to other deployments, the link being provided needs to have shared: true set
• "Implicit linking" (where things are matched on the type defined in the job spec) doesn't work across deployments. Therefore, you must have as set on the link being provided and from on the link being consumed, and the values of these things needs to match across deployment manifests, as you can see in the above two manifest examples. This is the magic string that glues the data together across deployments.
• The consuming link must also specify the name of the deployment it is consuming from, in case multiple deployments are providing a link with the same as value.

Self-linking for clustering

Clustered jobs like etcd and Consul often need to know the addresses of their peers. This can be accomplished when a job consumes a link which it itself provides. This can be a bit awkward, but it works. See how the nats job in the nats-release specifies it in the job spec and then leverages it in one of its templates.

6. ADVANCED: Links++

We started talking about all the things we didn't want to specify in the manifest. Most go away automatically with Links, but relying on some other new BOSH features, we can also get rid of having to choose that static IP, as well as the username/password for the database.

The manifest examples above have already removed all the network and resource pool configuration, so you know it's using Cloud Config and hence Global Networking. This means BOSH can actually pick static IPs for you. That is to say, we never really needed to set the static_ips in the above manifests in the first place!

Here, Links + Global Networking are working together. Since web_server_process depended on IPs of database_process, you'd (A) need to explicitly configure web_server_process in the manifest by providing it the IPs of database_process. And to make sure database_process actually has those IPs you told web_server_process, you'd (B) need to explicitly set static IPs for database_process. But since web_server_process's job templates can get the information via link(...).instances.map(&:address), then (A) goes away. And now you have reason to be explicit about what IPs database_process gets, and BOSH is able to pick the IPs for you, so (B) goes away. One of these features alone isn't enough to get the IPs out of the manifest, but together they are.

The username and password removal requires a bit more explanation, and there's a bit more work required in BOSH and CredHub to make things really seamless. There's actually a lot more junk in the manifest that isn't directly or indirectly related to links that would be nice to remove (e.g. Do I really need to specify azs or can it automatically stripe my job across all available ones? Could there be a default network specified in the Cloud Config so I don't need to keep specifying it in the instance groups?). A manifest might eventually look like this:

~/workspace/deployments/galileo_environment/my_app_future.yml

---
# DON'T NEED TO SPECIFY NAME, IT GETS SPECIFIED ON THE COMMAND LINE WHEN I 'bosh deploy my_app_future ...'

releases:
- name: app
  version: latest

instance_groups:
- name: web_server_instances
  jobs:
  - release: app
    name: web_server_process
    properties:
      greeting: "Namaste!"
  instances: 3
  vm:
    memory: 2GB
    cpu: 4
    ephemeral_disk: 20GB

- name: database_instance
  jobs:
  - release: app
    name: database_process
    properties:
      username: ((db_username)) # CREDHUB WILL GENERATE A USERNAME AND STORE IT INTERNALLY 
                                # WITH A NAMESPACED REFERENCE LIKE /my_bosh/my_app_future/db_username
      password: ((db_password)) # SIMILAR TO ABOVE
  instances: 1
  azs: [z1]
  vm:
    memory: 2GB
    cpu: 16
    ephemeral_disk: 20GB
  persistent_disk: 500GB
  
update:
  canaries: 1
  canary_watch_time: 10000-600000
  update_watch_time: 10000-600000
  max_in_flight: 1
  serial: true

stemcells:
- alias: whatever
  default: true
  os: ubuntu-trusty
  version: latest

7. Action items for release authors

There are nearly enough features in BOSH today to manage a foundation consisting of multiple deployments such as Cloud Foundry, MySQL, RabbitMQ, Redis, etc., with:

• no IPs specified in any manifest;
• no credentials specified in any manifest;
• all internal "uninteresting" credentials automatically generated by CredHub (not by the operator using certrap, openssl, etc.);
• no error-prone repetition of the same values in multiple places;

Furthermore, the existing BOSH features allow the same set of manifests used for one foundation to be used in a different foundation without the headache of making a ton of environment-specific adjustments. And going even further, existing BOSH features support a level of extensibility so that a vendor-specific distribution of Cloud Foundry, etc. can make heavy reuse of open source distributions such as cf-deployment. The next major milestones require release authors to build the little bits of glue that tie everything together. For 99% of releases out there, linkifying your individual release is cheap and easy, though it's most impactful when it's done across the board. If one release linkifies but most others don't, the operator doesn't see any big wins, e.g. instead of having to hand-maintain 100 IPs, they get to hand-maintain 99.

Be fruitful and linkify!

Hi Amit,

We were noticing that the example files currently contain an empty relevant_later link. Later, the username, password, and port properties are referenced while mentioning that the max_connections property is not exposed. It seems that currently none of the properties are exposed and require

provides:
  - name: relevant_later
    type: relevant_later
    properties:
    - username
    - password
    - port

to be available.

Good use of gifs. I like the idea of break time if this is being structured as a course. As a (static) docs page it might make more sense to divide it up into "tutorial: getting started" and "tutorial: advanced" and let the reader decide how to manage their time.

I don't think I saw anything incorrect, but it was a little hard to parse. There's a large amount of information density, so perhaps consider breaking up the multi-sentence paragraphs into smaller ones? I know it's subjective, but it would probably improve my experience.

Some minor feedback below:

Remove the main function from the golang analogy for importing - the import () section is sufficient to get the point across.

Minor copy changes e.g.:

Removing words which might offend the user like "obviously"

introducing some problems with manifests that obviously need solving

might better read:

introducing some problems with manifests that need solving

Redundant usage of words like "basic":

The basic problems raised in the "BASIC" sections could be solved without Links

might be better as:

The problems raised in the "BASIC" sections could be solved without Links

Spurious use of first-person tense:

but let me bring up a few questions which we'll address after

could be anything else that doesn't insert an individual into the narrative.

Thanks @JohannaSmith, I fixed that omission.

As far as readability, and whether it helps improve your understanding of links, do you have any feedback? If you were already familiar with links, do you have a colleague who is less familiar, or could you imagine yourself in those shoes?

Thanks @robdimsdale, will incorporate that feedback.

This doc is fantastic. I thought the following were really helpful:

• Explaining the difference between the name and type of a link. I've seen a decent amount of confusion about this when release teams implement links.
• Explaining some of the current shortcomings of links -- for example, the way that property names still need to be synchronized in different releases.

Thank you webmaster! But our database_process is in another deployment!

Brilliant.

I'm trying to convert the etcd-release and its manifests to links

When I deploy I get a "Multiple instance groups provide links" error:

  - Multiple instance groups provide links of type 'consul'. Cannot decide which one to use for instance group 'etcd'.
     etcd.consul.consul_agent.consul
     etcd.etcd.consul_agent.consul

The two instance groups include the consul_agent job:

instance_groups:
- name: consul
  jobs:
  - name: consul_agent
    release: consul
- name: etcd
  jobs:
  - name: consul_agent
    release: consul
  - name: etcd
    release: etcd

This topic is solved in https://gist.github.com/Amit-PivotalLabs/c39528248b8cdc4ba8e347f8aa68abb6#what-happens-if-multiple-things-provide-the-same-type-of-link I think; but perhaps mention the above error message in the text to help googling find the solution.

For the example above, I changed it to:

instance_groups:
- name: consul
  jobs:
  - name: consul_agent
    release: consul
    provides:
      consul: {as: consul_server}
    consumes:
      consul: {from: consul_server}
- name: etcd
  jobs:
  - name: consul_agent
    release: consul
    consumes:
      consul: {from: consul_server}
  - name: etcd
    release: etcd

I tried to consume a link from another deployment but the error message suggests I'm getting it wrong.

I am deploying eats (acceptance tests for etcd), where I try to reference the aliased consul_server link in the message above:

instance_groups:
- name: acceptance-tests
  jobs:
  - name: acceptance-tests
    release: etcd
    consumes:
      consul:
        from: consul_server
        deployment: etcd

But the error message suggests it is ignoring deployment: etcd:

  - Can't resolve link 'consul_server' in instance group 'acceptance-tests' on job 'acceptance-tests' in deployment 'eats'. Please make sure the link was provided and shared.

The error message shows it is defaulting to deployment: eats rather than deployment: etcd that I specified.

Ahh, shared: true. BRB.

Thanks @Amit-PivotalLabs for this document.

For anyone else who wants to see a diff of a release that was upgraded to links; and a bosh2/cloud-config manifest + readme for deploying it, then dingotiles/etcd2-boshrelease@v95.1...v95.2 might be useful to you.

(Note: at time of writing, the eats.yml manifest errand didn't work for me; not sure if that's because the repo is a hard fork and I have to update somethings)

PR to etcd-release cloudfoundry-attic/etcd-release#38

Perhaps include a mention of optional: true which allows manifests to use if_link("db")

consumes:
- name: db
  type: postgresql
  optional: true

Is there a guide to migrating a running pre-links deployment to a link deployment? e.g. merging explicit AZ jobs cell_z1,cell_z2 into multi-az instance groups cell?

Update: http://bosh.io/docs/migrated-from.html#migrate

instance_groups:
- name: etcd
  azs: [z1, z2]
  migrated_from:
  - {name: etcd_z1, az: z1}
  - {name: etcd_z2, az: z2}

Thanks @drnic, great feedback, updated accordingly. You want to use the migrated_from to take advantage of the BOSH-native notion of AZs, which is an orthogonal concern to Links. The doc you linked to captures the gist of it.

Hello @Amit-PivotalLabs thanks for the very helpful explanation.

We are actually facing a problem while trying to integrate links with one of our bosh releases and we think that our use case may not be fully covered by the current links functionalities.
I think it could be seen as a subcase of What happens if multiple things provide the same type of link?.

The specific problem is due to the fact that we do not know in advance, (referring to the examples in this document) the number of database_instance instance groups.
At the same time we need to access them somehow dynamically from our configuration templates and specifically from the same instance group, something like the following:

instance_groups:
- name: web_server_instances
  jobs:
  - release: app
    name: web_server_process
    properties:
      greeting: "Namaste!"
    consumes:
      hindi_database_data_object_reference: {from: hindi_database}
      french_database_data_object_reference: {from: french_database}

<%
database_links = Array.new
database_links << link("hindi_database_data_object_reference")
database_links << link("french_database_data_object_reference")
%>

database_addresses={
<% database_links.each_with_index do |database, i| %>
  "<%= link("database").instances[0].address %>"<%= ',' if i < (database_links.size - 1) %>
<% end %>
}

but this ofc doesn't work very elegantly, for example, if we decided to add an italian_database_data_object_reference: {from: italian_database}, we would be forced to modify all the template files accordingly.

Would be really great to have something of this kind:

<%
database_links = link_by_type("database_data_class")
%>

database_addresses={
<% database_links.each_with_index do |database, i| %>
  "<%= link("database").instances[0].address %>"<%= ',' if i < (database_links.size - 1) %>
<% end %>
}

I have the feeling that we are missing something or trying to bend links functionalities a bit too much. Is there a better approach in order to achieve the same result?

Thanks a lot
Claudio

Hey @claudio-benfatto, I migrated your question here, since it'll be hard to have a threaded conversation on this Gist's comments.

@Amit-PivotalLabs,

I believe you still do need to specify deployment name both in ~/workspace/deployments/galileo_environment/my_app_future.yml and on the command line with the -d flag, otherwise BOSH CLI will error.

I'm still fairly new to BOSH and had a difficult time with some parts of this. It would have been very useful to me if one of the first things explained is that links are a mechanism for making values available for template expansion and that the mechanism for users to add their own values to this context is by exporting properties defined by spec files. If they have other effects it would be good to explicitly describe those as well.
I still haven't been able to figure out the relationship between links and manifests. The examples go to some length to highlight the uninteresting parts of the manifest, but doesn't explain what is interesting about what remains. I think that what is interesting may be something that was not required by the manifest, but to my novice eye there is nothing that I would have expected to be in the manifest that is missing. What I did notice is that every property defined by the spec is still accounted for by either a default value or definition in the manifest, so links didn't seem to do anything to make writing the manifest easier. I think it would be good to add more explanatory text about what is supposed to be noticed about the manifest file and, perhaps, comments that include the code that would otherwise have had to be written could be added to the manifest to make the effect the links are having more clear.

How i can link dynamic IP of the neighbor job to manifest, not to template, like we do with static_ips:

instance_groups:
  - name: web
    properties:
      key: (( jobs.postgres.networks.default.static_ips.[0] )) 

Hey @Alexvianet you cannot get the value of a link to show up in a manifest. Without links, if job A needs to know the IP(s) of job B, you would have to:

• assign static IPs to the instance group that job B is in (in the manifest)
• ensure job A's spec defines a property that allows you to provide it B's IPs (in the BOSH release for job A)
• copy the value of the static IPs for job B into the properties section for job A in whatever instance group it's in (in the manifest)

You may be using tools like spruce or spiff that make the static IP assignment and copying not entirely manual. However BOSH can now natively do IP assignment and property "copying" for you via Global Networking and Links, respectively. So instead the workflow would be:

• ensure job B provides a link (in the release for job B)
• ensure job A consumes a link, and ensure the ERB templates pull the IP info out of the link (in the release for job A)
• if the type of the provided and consumed link match, and this is all within the same deployment manifest, then "implicit linking" will take over and you won't need to do anything else
• otherwise, make sure you're explicit about the links being provided by job B in its instance group and consumed by job A in its instance group (in the deployment manifest)

@Amit-PivotalLabs Firstly, a wonderful and insightful document on the usage of BOSH links.
On the topic of BOSH links, do deployment addon jobs support links as well? I have a deployment manifest with an addon job defined. The release spec of one of the component jobs of the addon release defines a provides section. In the same manifest, I have several instance groups and component jobs which need to consume the links defined in the addon job.
I end up with an error as below:

Task 7912 | 17:16:37 | Preparing deployment: Preparing deployment (00:00:02)
                     L Error: Failed to resolve links from deployment 'target-deployment'. See errors below:
  - Failed to resolve link 'my_link' with type 'my_link_class' from job 'job' in instance group 'instance_group1'. Multiple link providers found:
    - Link provider 'my_link' from job 'addon_job' in instance group 'instance_group1' in deployment 'target-deployment'
    - Link provider 'my_link' from job 'addon_job' in instance group 'instance_group2' in deployment 'target-deployment'

Hey @abh1kg thanks for your praise, I'm glad you found this useful!

Unfortunately Gist comments don't result in notifications so I'm only seeing this now, apologies. I think the core BOSH team would be able to better answer your question. I've created a GitHub issue with your question here on your behalf.

Cheers!

Thanks again @Amit-PivotalLabs. I'll follow the GitHub issue

Hi,
I am getting error while deploying harbor with Bosh release
Error: Link property admin_password_for_smoketest in template harbor is not defined in release spec