This article is essential because it explains what a DAC is, its architecture, how it works, the role of the all the DAC actors (Coordinator, Members and Observers).
In that blog post, we explain how to interact with DAC and how to do a “complete loop”:
Push data,
Retrieve certificate,
Retrieve data.
We are going to configure and run:
One DAC Coordinator,
Two DAC Members,
One DAC Observer.
What are we going to configure and run?
In this blog post, we are going to configure each actor of DAC:
One DAC Coordinator - It is responsible to receive payloads from client, propagate them to DAC Members and Observers. It also stores and provides the DAC certificates of payloads stored by Members.
Two DAC Members - They have a forever connection to the Coordinator and will get notified of new payloads from the Coordinator, after which they will download the data and send back a signature to the Coordinator to attest that they will make the data available upon request
One DAC Observer - Like the Members, Observer will also download payloads from Coordinator, but it will not sign it. It is also responsible for retrieving missing data from DAC Member. This actor is meant to be used to provide data requested by Smart Rollup node via the reveal data channel.
The schema below, from Nomadic Labs blog post, shows the previously explained interactions.
We will not dive into technical details about DAC implementation in this blog post.
Trilitech's schema of DAC Infrastructure
Create data directories and setup environment variables
Set up the environment variables to ease usage of next commands to configure and run actors:
# Directory where the config and data will be stored for the Coordinator.
export COORDINATOR_DIR=${HOME}/.tezos-ghostnet/dac-coordinator
export COORDINATOR_REVEAL_DATA_DIR=${HOME}/.tezos-ghostnet/dac-coordinator/reveals
# Same as previous directory, but for one Member.
export DAC_MEMBER_0_DIR=${HOME}/.tezos-ghostnet/dac-member-0
export DAC_MEMBER_0_REVEAL_DATA_DIR=${HOME}/.tezos-ghostnet/dac-member-0/reveals
# Second Member.
export DAC_MEMBER_1_DIR=${HOME}/.tezos-ghostnet/dac-member-1
export DAC_MEMBER_1_REVEAL_DATA_DIR=${HOME}/.tezos-ghostnet/dac-member-1/reveals
# Finally, the Observer which is like an archiver in this tutorial.
export OBSERVER_DIR=${HOME}/.tezos-ghostnet/dac-observer
export OCTEZ_CLIENT_DIR=${HOME}/.tezos-ghostnet/client
# Reveal_data_dir is the shared folder between DAC Observer and your Smart Rollup.
export REVEAL_DATA_DIR=${HOME}/.tezos-ghostnet/soru/wasm_2_0_0
export GHOSTNET_RPC=https://ghostnet.tezos.marigold.dev
Set up the directories listed in the environment variable above:
We can check that the accounts have been created via the following commands. Note that commands for retrieving information of tz4 accounts have an additional bls prefix after the global parameters.
octez-client -d ${OCTEZ_CLIENT_DIR} -E ${GHOSTNET_RPC} bls show address member0
Warning:
This is NOT the Tezos Mainnet.
Do NOT use your fundraiser keys on this network.
Hash: tz4SttKjMnG63m2XcguPC3zjmPEQpVfHzsn1
Public Key: BLpk1vs61UGcpfoSMtqFAx3JUVLQRi72Ra7cqkdhR68fd9ZM4UV6dqVF2Ew8LMEPCR2ncey5UXS5
octez-client -d ${OCTEZ_CLIENT_DIR} -E ${GHOSTNET_RPC} bls show address member1
Warning:
This is NOT the Tezos Mainnet.
Do NOT use your fundraiser keys on this network.
Hash: tz4HQKiMhopVXhrWyRMDKoMV5E5trggHW8s2
Public Key: BLpk1qrzm2vb1JaRd5tm5NtSuaHK3nJziMiqE56qC2Tug9GNDmnSeh5pXQRaPvrSNk94uPoz3Uy4
Save both the public key and the public key hash of the members, we will need them later
Penultimate step which is quite straightforward, we are going to configure each actor with octez-dac-node configure command.
1. Configure the Coordinator
octez-dac-node configure as coordinator with data availability committee members ${MEMBER0_PK} ${MEMBER1_PK} --data-dir ${COORDINATOR_DIR} --rpc-port 11110 --reveal-data-dir ${COORDINATOR_REVEAL_DATA_DIR}
If you plan to run a Smart Rollups, you need to make sure the order of the keys is the same. In our example we went for MEMBER0 then MEMBER1 , then in your rollup you must set something like:
let member0_pk = PublicKeyBls::from_base58_check( "BLpk1vs61UGcpfoSMtqFAx3JUVLQRi72Ra7cqkdhR68fd9ZM4UV6dqVF2Ew8LMEPCR2ncey5UXS5",
)
.map_err(|_err| Error::FromBase58CheckError)?;
let member1_pk = PublicKeyBls::from_base58_check(
"BLpk1qrzm2vb1JaRd5tm5NtSuaHK3nJziMiqE56qC2Tug9GNDmnSeh5pXQRaPvrSNk94uPoz3Uy4",
)
.map_err(|_err| Error::FromBase58CheckError)?;
Ok(vec![member0_pk, member1_pk])
2. Run the Coordinator
octez-dac-node -d ${OCTEZ_CLIENT_DIR} -E ${GHOSTNET_RPC} run --data-dir ${COORDINATOR_DIR}
You must see the logs of starting Coordinator:
Sep 08 14:34:19.074: Starting the DAC node
Sep 08 14:34:19.075: The DAC node is running in mode Coordinator
Sep 08 14:34:19.083: DAC Node RPC server is started
Sep 08 14:34:19.083: The DAC node is listening to 127.0.0.1:11110
Sep 08 14:34:19.083: Started tracking layer 1's node
Sep 08 14:34:19.300: Resolved plugin for protocol PtNairobiyss
Sep 08 14:34:19.301: The DAC node is ready
Sep 08 14:34:19.301: Started tracking layer 1's node
Sep 08 14:34:19.449: Head of layer 1's node updated to
Sep 08 14:34:19.449: BMRjxvG4N6o87XmL2iM7n3L1gcbSNq2Ytvw8vdMFdxcHnpWJpem at level 3813166
3. Configure the two Members
As easily as the Coordinator, the Members only needs to know their own related environment variables and the RPC address:port of the Coordinator:
octez-dac-node configure as committee member with coordinator 127.0.0.1:11110 and signer ${MEMBER0_PKH} --rpc-port 11111 --data-dir ${DAC_MEMBER_0_DIR} --reveal-data-dir ${DAC_MEMBER_0_REVEAL_DATA_DIR}
octez-dac-node configure as committee member with coordinator 127.0.0.1:11110 and signer ${MEMBER1_PKH} --rpc-port 11112 --data-dir ${DAC_MEMBER_1_DIR} --reveal-data-dir ${DAC_MEMBER_1_REVEAL_DATA_DIR}
4. Run the two Members
octez-dac-node -d ${OCTEZ_CLIENT_DIR} -E ${GHOSTNET_RPC} run --data-dir ${DAC_MEMBER_0_DIR}
And see the starting logs:
Sep 08 14:34:38.056: Starting the DAC node
Sep 08 14:34:38.057: The DAC node is running in mode Committee_member
Sep 08 14:34:38.064: DAC Node RPC server is started
Sep 08 14:34:38.064: The DAC node is listening to 127.0.0.1:11111
Sep 08 14:34:38.064: Started tracking layer 1's node
Sep 08 14:34:38.228: Resolved plugin for protocol PtNairobiyss
Sep 08 14:34:38.228: The DAC node is ready
Sep 08 14:34:38.228: Started tracking layer 1's node
Sep 08 14:34:38.228: Subscribed to root hashes stream
Sep 08 14:34:38.317: Head of layer 1's node updated to
Sep 08 14:34:38.317: BL5kexCuYpzXaX2Hj9EJfEwrgpoABFey3nvNEc1ShuKA3jwqB3j at level 3813168
Then next member:
octez-dac-node -d ${OCTEZ_CLIENT_DIR} -E ${GHOSTNET_RPC} run --data-dir ${DAC_MEMBER_1_DIR}
And also see its logs:
Sep 08 14:34:43.426: Starting the DAC node
Sep 08 14:34:43.427: The DAC node is running in mode Committee_member
Sep 08 14:34:43.433: DAC Node RPC server is started
Sep 08 14:34:43.433: The DAC node is listening to 127.0.0.1:11112
Sep 08 14:34:43.433: Started tracking layer 1's node
Sep 08 14:34:43.624: Resolved plugin for protocol PtNairobiyss
Sep 08 14:34:43.624: The DAC node is ready
Sep 08 14:34:43.624: Subscribed to root hashes stream
Sep 08 14:34:43.624: Started tracking layer 1's node
Sep 08 14:34:43.710: Head of layer 1's node updated to
Sep 08 14:34:43.710: BKnhUm7TaVvk2wxKniyC4hR7kLythpUAUfnPqmCVvHC417yqmtf at level 3813169
When your two members are started, they subscribe to the Coordinator streaming root_hash endpoint, and you must see the following line on the Coordinator logs:
Sep 08 14:34:38.232: Subscription of another dac node to the hash streamer handled successfully.
[...]
Sep 08 14:34:43.626: Subscription of another dac node to the hash streamer handled successfully.
We have twice this line since we have started two Members. This line will also be printed one starting the Observer.
5. Configure the Observer
This Observer will be run as an archiver. If you want to use it for your Smart Rollups needs, simply share the reveal_data_dir between Observer and Smart Rollups.
octez-dac-node configure as observer with coordinator 127.0.0.1:11110 and committee member rpc addresses 127.0.0.1:11111 127.0.0.1:11112 --data-dir ${OBSERVER_DIR} --reveal-data-dir ${REVEAL_DATA_DIR}
6. Run the observer
octez-dac-node -d ${OCTEZ_CLIENT_DIR} -E ${GHOSTNET_RPC} run --data-dir ${OBSERVER_DIR}
Its logs:
Sep 08 14:37:23.422: Starting the DAC node
Sep 08 14:37:23.422: The DAC node is running in mode Observer
Sep 08 14:37:23.426: DAC Node RPC server is started
Sep 08 14:37:23.426: The DAC node is listening to 127.0.0.1:10832
Sep 08 14:37:23.426: Started tracking layer 1's node
Sep 08 14:37:23.619: Resolved plugin for protocol PtNairobiyss
Sep 08 14:37:23.619: The DAC node is ready
Sep 08 14:37:23.619: Started tracking layer 1's node
Sep 08 14:37:23.619: Subscribed to root hashes stream
Sep 08 14:37:23.762: Head of layer 1's node updated to
Sep 08 14:37:23.762: BM1VVZNs3un8rfa685CwqUY3LL1FMttQTQ5fMeoJcs1oeQUknwg at level 3813189
And once again, on the Coordinator logs:
Sep 08 14:37:23.620: Subscription of another dac node to the hash streamer handled successfully.
🎉 You just successfully deployed your full DAC infrastructure!!!
This task was relatively straightforward to complete and it will enable a better TPS for your Smart Rollups!
Interact with each actor.
In this last step, we are going to fetch HTTP APIs of each actor to make sure everything is working fine.
Push data to Coordinator.
The first interesting thing to do to validate the entire infrastructure is working well is to push data to the Coordinator.
Once Coordinator has done its job, it will stream the root_hash to the Members, which are going to validate and sign the data and finally send back Signature to Coordinator.
Since you have read the previous blog post announcing Community DAC you are supposed to already know how to post data, retrieve corresponding certificate and finally fetch back data, but I will re-explain it here to add the logs of each actor!
Let’s push I run my own full DAC infrastructure! string!
Sep 08 14:48:41.729: New root hash pushed to the data streamer:
Sep 08 14:48:41.729: 0021c6a837f6fbe0b7a5e7ed0c60520f676dd99f834e6e2e94289dfc481[...]
Which means this root_hash (fully displayed in the return of curl but not on logs) had been pushed to your two Members and your Observer.
On member0 side:
Sep 08 14:48:41.729: Received new root hash via monitoring rpc
Sep 08 14:48:41.729: 0021c6a837f6fbe0b7a5e7ed0c60520f676dd99f834e6e2e94289dfc481[...]
Sep 08 14:48:41.731: Finished processing previously received root hash
Sep 08 14:48:41.731: 0021c6a837f6fbe0b7a5e7ed0c60520f676dd99f834e6e2e94289dfc481[...]
Sep 08 14:48:41.741: New signature from member pushed to the coordinator:
Sep 08 14:48:41.741: BLsigAPKwmV6sPgLFG4TP9sjPtJUcn2uSw7t4ADzHMS3F6xKRTqomi1mNCo[...]
And on member1 side:
Sep 08 14:48:41.729: Received new root hash via monitoring rpc
Sep 08 14:48:41.729: 0021c6a837f6fbe0b7a5e7ed0c60520f676dd99f834e6e2e94289dfc481[...]
Sep 08 14:48:41.731: Finished processing previously received root hash
Sep 08 14:48:41.731: 0021c6a837f6fbe0b7a5e7ed0c60520f676dd99f834e6e2e94289dfc481[...]
Sep 08 14:48:41.738: New signature from member pushed to the coordinator:
Sep 08 14:48:41.738: BLsigAFrkizwHW6x9HSDBcxmhUx4Zri8RCvAa4g6KJfDTjMoMevSSNQXY1u[...]
Which means our two Members had successfully donwloaded the data and signed it, and sent back their signature to Coordinator.
On the Observer side, a simple line saying Observer had well received the root_hash and saved corresponding data:
Sep 08 14:48:41.729: Received new root hash via monitoring rpc
Sep 08 14:48:41.729: 0021c6a837f6fbe0b7a5e7ed0c60520f676dd99f834e6e2e94289dfc481[...]
Sep 08 14:48:41.731: Finished processing previously received root hash
Sep 08 14:48:41.731: 0021c6a837f6fbe0b7a5e7ed0c60520f676dd99f834e6e2e94289dfc481[...]
You can also have a look on the data stored locally in the REVEAL_DATA_DIR folder
$ cd $REVEAL_DATA_DIR && ls
Date Modified Name
8 sep 14:48 0021c6a837f6fbe0b7a5e7ed0c60520f676dd99f834e6e2e94289dfc4813718fc8
Let’s now retrieve the Certificate!
Retrieve the certificate
We are going to fetch the JSON certificate, which is human readable:
The Observer also expose one last endpoint GET /missing_page/{root_hash} which is useful if you notice your Observer had missed some data.
For example, you stop it for a few minutes, send data to Coordinator, restart Observer, the Observer won't receive the root_hash pushed when it was off. Simply call GET /missing_page/{missing_root_hash} and Observer will retrieve the corresponding data by fetching the GET /preimage of the Coordinator.
You’ve successfully deployed your own complete DAC infrastructure! Now, you can operate a Smart Rollup alongside your Observer to achieve improved TPS!
As you may have noted, our initial setup was only local, which isn’t ideal for production use.
In the next section, we’ll guide you through deploying it on Kubernetes!
Production-ready DAC infra on Kubernetes
In this section, we will outline the steps to deploy a full DAC infrastructure on Kubernetes. This tutorial is cloud provider-agnostic.
Prerequesites: a good understanding of Kubernetes.
Kubernetes manifest files
You can access the YAML files referenced in this section from this public repository, feel free to reuse and adapt them according to your needs.
This setup demands the Octez binaries alongside a few other utilities, all encapsulated within a Docker image. The most straightforward way to obtain this is by using the official images available on DockerHub. These images come preloaded with the octez-dac-node binary.
To initialize the nodes, we will run multiple shell scripts. For ease, we have opted to include Bash as a binary within the Docker image. However, for a sleeker setup and enhanced security, you might want to execute these shell scripts in an InitContainer.
Env vars and feature toggle flags:
In Kubernetes, updating a ConfigMap not automatically trigger a restart of pods that have mounted that ConfigMap. Pods using ConfigMaps retain the old version until manually restarted. Pods will continue to use the previous version of the ConfigMap until they are restarted. We will use that behavior to conveniently manage some feature flags.
Example of toggles and env vars we set in the configmap:
For instance, if for some reason we want to clean the data-dir of the coordinator we can set WIPE_COORDINATOR_DIR: "YES" and restart the pod so that it gets cleaned up at startup.
Storing scripts in a configmap:
Bash scripts are stored in a configmap and mounted in the pods. Basically each script does the following:
If you’re operating the DAC on a Kubernetes instance or a private server managed by a cloud provider, setting up Authorization and Authentication is essential. Most often, this is done using the cloud provider’s IAM (Identity and Access Management). Such implementations ensure that only authorized users can access sensitive cloud resources and that they can execute only permitted actions.
For maintaining security specifically within Kubernetes, it’s imperative to follow its security best practices, which include:
Role-Based Access Control (RBAC)
Network segmentation and policies
Encrypt data stored in Kubernetes’ etcd key-value store
Implement policy engines
Limit access to Kubernetes API
Configure security context for pods and containers
Use namespaces to isolate workloads
etc
For those who use GitOps workflows, tools like SOPS and SealedSecrets come in handy. They offer effective ways to encrypt secrets - in this context, the private keys of wallets.
Exposing DAC endpoints:
The octez-dac-node binary does not support CORS headers. Therefore, we use a reverse proxy positioned between the DAC nodes and the client to append CORS headers to HTTP requests.
As illustrated in the provided ingress manifests, we use the Nginx ingress controller as a reverse proxy to supplement the CORS headers:
Additionally, some ingress configuration options need tweaking:
# increase max body size to be able to transfer large files to the DAC:
nginx.ingress.kubernetes.io/proxy-body-size: 2g
# suppress proxy timeouts:
nginx.ingress.kubernetes.io/proxy-read-timeout: "2147483647" # We want the timeout to never trigger. There's no way to turn it off per se, so we achieve this by setting an unrealistically large number.
nginx.ingress.kubernetes.io/proxy-send-timeout: "2147483647"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "2147483647"
In our case we also use Nginx ingress controller as a way to expose DAC endpoints over the internet on a public endpoint.
You can also reach your DAC from within Kubernetes on its corresponding service: <name-of-service>.<k8s-namespace>.svc:80 Like so:
Please note that we had to specify the --rpc-addr 0.0.0.0 --rpc-port in the octez-dac-nodecommand for each node to ensure it listens for connections from the desired IP address and port.
Monitoring:
As of now, the octez-dac-node does not produce metrics that we could use to monitor our nodes.
Nonetheless, we can configure a livenessProbe in our deployments, allowing Kubernetes to periodically verify if the container is alive:
DAC data not only resides on Persistent Volume Claims (PVCs) – a dependable storage solution within Kubernetes – but is also replicated across all DAC community Members and Observers. Despite this inherent redundancy, it remains prudent to have an external backup and recovery strategy for your PVCs. We opted for Velero, an open-source tool, that offers the ability to safely backup, recover, and migrate Kubernetes cluster resources and persistent volumes, and comes equipped with a disaster recovery feature.
Voilà! 🎉
In this long blost post, you successfully:
Run your own full DAC infrastructure locally
Deployed it!
In our next blog post, we will show how to use DAC inside your Smart Rollups!
If you want to know more about Marigold, please follow us on social media (Twitter, Reddit, Linkedin)!
.svg)
.svg)
(1).png){kind=small}
.png){kind=small}
(1).png){kind=small}
(1).png){kind=small}
(1).png){kind=small}
.png){kind=small}
