Skip to main content

Divvi Up Command Line Tutorial

Learn Divvi Up via a local instance of all Divvi Up components running via docker compose.

Following this tutorial will:

  • Install required software and tools
  • Run Divvi Up and a pair of aggregators locally in Docker
  • Shard some measurements into reports and upload them, simulating DAP clients
  • Collect the results of an aggregation

If any of these terms are unfamiliar please read the How it Works doc.

Requirements​

To get started, you'll need:

  • A Unix like shell on macOS, Linux or Windows
  • A divviup binary (see instructions below)
  • A compose.yaml (see instructions below)
  • Docker and Docker Compose.

If you are unable to meet these requirements or would prefer to watch this tutorial being performed as a video see the command line video demo

Install divviup​

divviup is a command line (CLI) tool for doing basic operations on the Divvi Up and Distributed Aggregation Protocol (DAP) API endpoints. See divviup --help for details on all of the commands.

The tool is available for download for macOS, Windows or Linux from the divviup-api releases page.

Download compose.yaml​

A compose.yaml will be used to download and run all required containers to run a Divvi Up environment.

Create a folder for this tutorial

mkdir divviup-tutorial
cd divviup-tutorial

And download the latest release of the compose.yaml found on the divviup-api releases page page.

curl -L https://github.com/divviup/divviup-api/releases/latest/download/compose.yaml > compose.yaml

Start Divvi Up​

First, get Divvi Up running locally. In your shell, navigate to the directory containing compose.yaml and run:

docker compose up --wait

This will:

  • deploy two Janus aggregators to act as leader and helper in your tasks
  • deploy the Divvi Up control plane to coordinate the aggregators
  • pair the aggregators with the control plane to make them available for use in task definitions
  • create an account named "demo"

Configure Auth​

Set an environment variable to direct divviup to use your local control plane and to use an empty API token:

export DIVVIUP_API_URL=http://localhost:8080
export DIVVIUP_TOKEN=""

Next, obtain the ID of the demo account:

divviup account list

You should get a single account back with the name "demo", like this:

[
  {
    "id": "8ea1fc65-a669-48dd-96ac-c920fcb0ae90",
    "name": "demo",
    "created_at": "2024-06-14T22:48:00.683659Z",
    "updated_at": "2024-06-14T22:48:00.683659Z",
    "admin": false
  }
]

Now, set an environment variable for the account ID:

# Substitute the account ID that was generated by your Divvi Up instance
export DIVVIUP_ACCOUNT_ID=8ea1fc65-a669-48dd-96ac-c920fcb0ae90

Configure Leader and Helper Aggregator​

List the aggregators and identify the ID of both a leader and a helper.

divviup aggregator list

The output will contain JSON objects like:

  {
    "id": "3650870b-56e6-4eac-8944-b7ca36569b33",
    "role": "Either",
    "name": "Divvi Up staging-dap-09-1",
    "dap_url": "https://staging-dap-09-1.api.example.com/",
    "api_url": "https://staging-dap-09-1.api.example.com/aggregator-api",
    "is_first_party": true,
    "vdafs": [
      "Prio3Count",
      "Prio3Sum",
      "Prio3Histogram",
      "Prio3SumVec"
    ],
    "query_types": [
      "TimeInterval",
      "FixedSize"
    ],
    "protocol": "DAP-09",
    "features": [
      "TokenHash",
      "TimeBucketedFixedSize",
      "UploadMetrics"
    ]
  },
  {
    "id": "96301951-c848-4a57-b4f5-32812e4db1be",
    "account_id": null,
    "created_at": "2024-03-21T22:47:15.467139Z",
    "updated_at": "2024-04-18T17:33:30.439465Z",
    "deleted_at": null,
    "role": "Either",
    "name": "Divvi Up staging-dap-09-2",
    "dap_url": "https://staging-dap-09-2.api.example.com/",
    "api_url": "https://staging-dap-09-2.api.example.com/aggregator-api",
    "is_first_party": false,
    "vdafs": [
      "Prio3Count",
      "Prio3Sum",
      "Prio3Histogram",
      "Prio3SumVec"
    ],
    "query_types": [
      "TimeInterval",
      "FixedSize"
    ],
    "protocol": "DAP-09",
    "features": [
      "TokenHash",
      "UploadMetrics",
      "TimeBucketedFixedSize"
    ]
  }

Set the two IDs into environment variables. NOTE: These IDs will vary based on your configuration.

# Substitute the leader and helper aggregator IDs that were generated by your Divvi Up instance
export LEADER_ID=3650870b-56e6-4eac-8944-b7ca36569b33
export HELPER_ID=96301951-c848-4a57-b4f5-32812e4db1be

Collector Credential​

Next, generate a collector-credential for the task. The collector credential will be used by the collector to export the aggregated telemetry.

divviup collector-credential generate --save

The output will be like:

{
  "id": "0a0f8ea8-b603-4416-b138-b7f217153bb7",
  "hpke_config": {
    "id": 144,
    "kem_id": "X25519HkdfSha256",
    "kdf_id": "HkdfSha256",
    "aead_id": "Aes128Gcm",
    "public_key": "V9IpdJxS91MHPiNTjwDk9DFS-5M_neVrPxlmvolmTTo"
  },
  "created_at": "2024-05-03T15:23:56.624726Z",
  "deleted_at": null,
  "updated_at": "2024-05-03T15:23:56.624727Z",
  "name": "collector-credential-144",
  "token_hash": "VItYJdAyWYIvooe8GzkGnVTvaMkvWc9G-eiwxudfWww",
  "token": "A6JDAYPiYDXmNyh-OpYXGw"
}

Saved new collector credential to /your/current/directory/collector-credential-144.json. Keep this file safe!
warning

This credential isn't sensitive because it's only useful in your local instance of Divvi Up. But collector credentials generated against the production service should be carefully managed. Consider using a password manager to protect them.

Make a note of the path where the credential was saved, and save the collector credential ID to an environment variable.

# Substitute the collector credential path and ID that were generated by your Divvi Up instance
export COLLECTOR_CREDENTIAL_PATH=/your/current/directory/collector-credential-144.json
export COLLECTOR_CREDENTIAL_ID=0a0f8ea8-b603-4416-b138-b7f217153bb7

Create a Task​

Create the the histogram task. In this case the task is a set of values from 0 to 10 for use in collecting a net-promoter score for a survey.

 divviup task create --name net-promoter-score \
    --leader-aggregator-id $LEADER_ID --helper-aggregator-id $HELPER_ID \
    --collector-credential-id $COLLECTOR_CREDENTIAL_ID \
    --vdaf histogram --categorical-buckets 0,1,2,3,4,5,6,7,8,9,10 \
    --min-batch-size 100 --max-batch-size 200 --time-precision 60sec

The output will contain a JSON object:

{
  "id": "Siwa4QTEnQXMfRPyhir8AzS4EBqfTebmEzKfvajDgYk",
  "account_id": "a9c571ba-5f3d-4814-8d8b-c5bb0f5030b7",
  "name": "net-promoter-score",
  "vdaf": {
    "type": "histogram",
    "buckets": ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "10"],
    "chunk_length": 4
  },
  "min_batch_size": 100,
  "max_batch_size": null,
  "created_at": "2024-05-03T15:27:56.229891Z",
  "updated_at": "2024-05-03T15:27:56.229891Z",
  "time_precision_seconds": 60,
  "report_count": 0,
  "aggregate_collection_count": 0,
  "expiration": "2025-05-03T15:27:55.88511Z",
  "leader_aggregator_id": "3650870b-56e6-4eac-8944-b7ca36569b33",
  "helper_aggregator_id": "96301951-c848-4a57-b4f5-32812e4db1be",
  "collector_credential_id": "0a0f8ea8-b603-4416-b138-b7f217153bb7",
  "report_counter_interval_collected": 0,
  "report_counter_decode_failure": 0,
  "report_counter_decrypt_failure": 0,
  "report_counter_expired": 0,
  "report_counter_outdated_key": 0,
  "report_counter_success": 0,
  "report_counter_too_early": 0,
  "report_counter_task_expired": 0
}

Save the ID of the task into an environment variable.

# Substitute the task ID that was generated by your Divvi Up instance
export TASK_ID=Siwa4QTEnQXMfRPyhir8AzS4EBqfTebmEzKfvajDgYk

Upload Measurement Reports​

Upload a random set of 150 measurements.

for i in {1..150}; do
  measurement=$(( $RANDOM % 10 ))
  divviup dap-client upload --task-id $TASK_ID --measurement $measurement;
done

Note that the random measurements will only be between 0 and 9 inclusive, while the buckets were created for measurements between 0 and 10 inclusive. This will leave the "10" bucket empty for demonstration purposes.

Wait a little while to let the aggregators run the aggregation jobs. Then, get collection results:

Collect the Aggregated Value​

divviup dap-client collect \
    --task-id $TASK_ID \
    --collector-credential-file $COLLECTOR_CREDENTIAL_PATH \
    --current-batch

You will get a result like:

Number of reports: 113
Interval start: 2024-06-05 21:31:00 UTC
Interval end: 2024-06-05 21:34:00 UTC
Interval length: 180s
Aggregation result: [14, 10, 10, 13, 13, 8, 16, 13, 9, 7, 0]
collection: Collection { partial_batch_selector: PartialBatchSelector { batch_identifier: BatchId(wPLBlC6iHWp_YBBAYP_ig5nal0FOz1QlLSaC42U7sm0) }, report_count: 113, interval: (2024-06-05T21:31:00Z, TimeDelta { secs: 180, nanos: 0 }), aggregate_result: [14, 10, 10, 13, 13, 8, 16, 13, 9, 7, 0] }

If you get an error like "The batch implied by the query is invalid", then the aggregators are still working on processing your uploaded reports. Given them a few more minutes.

If you get fewer reports in the collection than you uploaded, that's because you sent the collection request too soon and not all the reports were prepared yet. Those reports will be available in a later collection, after enough additional reports are uploaded to meet the minimum batch size.

Next Steps​

Congratulations! You've just used secure multi-party computation to do a privacy-preserving aggregation over secret shares of measurements. Now you can try experimenting with different tasks. Try creating different histograms with different numbers of buckets, or see if you can use Prio3SumVec to compute sums over bit vectors.