Applying DevOps in blockchain development!
IT Svit has ample experience with both sides of the blockchain software delivery process. We have developed various blockchain products for customers from finances, eCommerce, mining, shipping, transportation, real estate and energy market, social services, and environmental control.
We helped the startups fully leverage all the benefits of blockchain architecture:
- decentralization of operations for interactions between numerous peers
- immutability of records to provide an undisputable ledger of data
- speed of processing to enable instant operations
- security to ensure the anonymity of transactions
Our in-depth expertise with software development and operations applied to the development and production of blockchain-based applications helped us deliver numerous useful products like:
- a system of coupons for enabling cross-sales in a huge network of eCommerce shops
- a system of QR-codes to replace the excise stamps in retail
- a transparent energy market using smart contracts
- a registry of land trading and real estate ownership records
- a productivity audit app for a mining corporation
- a cargo manifest checker for the seaports association
- lots of cryptocurrency exchange apps for various customers
- and many, many more blockchain-based solutions
All of these products had to be developed and many of them had to be integrated into a complex existing infrastructure on time and without disrupting the operations. This means that every code commit has had to be tested extensively at test environments and staging servers before being pushed to production. To ensure sufficient scalability and cost-efficiency, these apps were deployed to AWS, Google Cloud or Microsoft Azure, each of which has specific services and features useful for blockchain development.
However, most importantly, these apps required a decent DevOps expertise to ensure the resources were always correctly allocated and available. IT Svit has provided the required DevOps engineers and best practices to enable managing the underlying cloud Infrastructure as Code, automation of testing new code batches through Continuous Integration and seamless pushing of new app features to production through the Continuous Delivery pipelines.
DevOps for blockchain apps — IT Svit style
One of IT Svit customers wanted to containerize a blockchain app. The Docker image is readily available, so the customers wanted us to configure the deployment using Kubernetes manifests. However, using Kubernetes for infrastructure configuration allowed us to enable multiple useful operations like backups and restoration, logging, monitoring and smart alerting, utilizing dashboards from our internal monitoring system with Prometheus + Grafana and many other functions (measuring the blockchain length, syncing the data real-time, etc.)
We had to solve the following challenges:
- Ensure problem-free app deployment to AWS infrastructure using Kubernetes manifests
- Provide convenient and informative system monitoring based on ChatOps best practices
- Enable automated system backups and restoration
The solution we came up with includes Sumologic platform with smart alerts sent to Slack, thus implementing the ChatOps principles. For example, if the blockchain API stopped accepting new blocks — we receive an alert immediately and begin fixing the issue at once. The workflow we have implemented involved auto-restoration from the last backup, auto-update of image registry after a new blockchain version is released, hourly backups of all blockchain data, etc.
This system works using the following approach: once an hour the cluster checks the blockchain app health within the container. If the app runs normally, a snapshot is saved to S3 storage. If the app is down, the container is rebooted and its persistent volume is deleted. As a part of the launch routine, the Kuberntes pode checks the empty persistent volume and the latest snapshot of data is uploaded from the AWS S3 bucket. This takes from 2 to 20 minutes based on the blockchain app complexity and the volume of data stored.
Leveraging our solution enables the customer to run multiple blockchain apps within a single Kubernetes cluster securely, with in-depth monitoring through a convenient dashboard, smart alerting to Slack, detailed logging with Sumologic and much, much more.
Results of applying DevOps approach to blockchain app development
DevOps approach to infrastructure management ensures the system runs efficiently at all times. To meet the security requirements, the Kubernetes cluster runs in the internal network, which can be accessed through VPN only. The public access is available to the load balancers only. There is a separate bastion host with SSH-auth for managing the VPN server, so the system is protected from external access. We have implemented the following features:
- Logs rotation, so once the log file reaches the maximum size, it is gzipped and stored in the Kubernetes pode. The content of each log is read by SuboLogic collector and sent to the central SumoLogic instance for processing.
- Backups rotation so every backup made is pushed to an S3 bucket, where the limited number of backups is stored. The latest backup replaces the oldest, so the system is kept up-to-date and does not grow in size exponentially.
- Free pode space evaluation helps us monitor the degree of pode volume filling, due to the unrestricted growth of the blockchain blocks. Thus said, we are alerted when the adjustments to the pode volume have to be done.
This is but one example of the DevOps workflows that can be configured for development and production of blockchain-based applications. IT Svit can provide any kind of architecture and cloud infrastructure required to help you adopt DevOps delivery process for your products using the blockchain technology. We are always ready to help!
