Header Calculations Bitcoin
I’ve wanted to write an article explaining the technical breakdown of how a cryptocurrency works for some time now. While there may not be anything new in this article it does attempt to tie many of the disparate sources of information together and explain, in the simplest form, how a coin like bitcoin and invariably the majority of coins out there, calculate their block headers and actually process things onto the blockchain.
An Introduction
Dramatic innovation in any industry is hard to come by and it is even more rare in industries that are insulated to that change, like the world of finance. We have been lucky enough to watch as something that has never been done before was conceived, developed, and since has sent a ripple through the industry: Bitcoin. Since its inception, Bitcoin has been a dramatic change in how people view finances and has evolved into an archetype mimicked by hundreds of other cryptocurrencies. Spinning off various clones, impersonators, and chains attempting to ‘fix’ issues with the Bitcoin network. Inevitably, however, at the core of most cryptocurrencies is still the engine that powers Bitcoin.
I figured this then would be the natural starting point to begin to dissect how most cryptocurrencies actually work. By ‘work’ I’m really aiming to demonstrate a technical deep dive into what is going on under the hood, specifically focusing on the block header calculations. For those in the industry we have all by now heard the tired explanations given by the majority of people speaking on the topic who, I would wager at a technical level don’t really know how things are working (ex. “it actually uses information from the previous block…” , “it’s distributed…”, “It uses encryption..” etc.). Hopefully along the way I can help illustrate how exactly the Bitcoin blockchain headers work by providing the ‘how?’ that is often left out.
As I mentioned, in this article I’ll be using Bitcoin as the example and in later articles I hope to write further technical write-ups on additional features of the Bitcoin network (Merkle Root Calculations, Transactions Calculations and their various inputs and outputs). I would also like to go through how other coins may be using different hashing algorithms, other types of ‘Proofs’ (like Auxiliary Proof of Work, Proof of Stake etc.), and possibly even get into the Ethereum world; so hopefully stay tuned.
Quick Brief: Mining and the Target
To explain how blocks are processed on the Bitcoin network it is impossible to not bring up mining. I will do my best to keep this part short as I want to spend the majority of this article on the actual technical moving pieces that make up the mining process.
For a brief and simple intro, the processing of transactions is the main goal of the network and it is a relatively easy process (I say easy in terms of complexity, not in terms of processing power):
· A transaction between two addresses is introduced to the network,
· Nodes have a look at both wallets,
· make sure they both have the required funds,
· if they do then add them to a ledger of transactions,
· bundle these transactions with multiple other transactions
Lastly, try to process that block of transactions on the blockchain. This final piece is where the complexity comes in. Once all the steps above are complete, the miners will solve a very difficult cryptographic hash (the ‘Proof of Work’) in a race to subvert the value the bitcoin protocol has set. That is to say, there will be a hexadecimal number (or target) that the bitcoin protocol creates based off of how many participants there are in the network. The goal of mining is to attempt to reach a hashed output that is smaller than that target. If the miner can be the first, with a calculated and broadcasted block header, below the target then they will have their block of transactions added to the chain, receive the reward (currently 12.5 bitcoins), and the praise of the network (…probably). In a bit more detail, a miner will bundle all the transactions they are looking to process, combine several bits of information about the block, hash the information and then compare to the target. If they have created a hash that is larger than the target, they will adjust their nonce and try again (I will go into this process in a bit more detail below).
Because the miners are only looking to get lower than the target means that there is theoretically more than one result that is considered “valid” and the race is for whoever reaches it first.
If I were to write out an equation for the process it would be:
If you didn’t get the above, don’t worry. Hopefully I will do a better job explaining in the next couple steps. First let’s look at the process of creating a block header. For those looking to follow along I’ll be using block 550204 on the Bitcoin network to walk through the calculation of this block header. The block header, and answer to the calculation we are going to compute should be:
000000000000000000d6b504c80ca9087b6a44ac908b2d634237344764d023a
The Block header
The block header is essentially the proof that all the information that comprise that particular block fulfills the requirements the Bitcoin protocol is seeking to verify. This ensures the block is valid and should be able to be attached to the chain. This process is very important and without the ability to link blocks together, the blockchain wouldn’t be able to validate true blocks from fake/imposter blocks and inescapably rendering the entire process useless. With Bitcoin, this is a nice and succinct process, as everything involved in completing the mining process is stored in the block (I will explain how this differs in a later article for coins like Litecoin).
In short, the calculation of the block header breaks down to be the hash and the combination of these 6 pieces of information:
Version
Previous Block
Merkle Root
Timestamp
Difficulty
Nonce
These 6 Pieces of information sync the chain together and once arranged and combined look something like this:
Let’s start going through all the pieces, and as we do it’s very important to keep in mind formatting matters and depending upon the block explorer you are using you may see the output in different formats. For all information we will have to be sure everything has been converted to HEXIDECIMAL (if it isn’t already) and then we will have to reverse the various output’s byte order. Okay, with that out of the way we can jump into the definitions and values of our example block’s information:
Version
The Version is which version of the Bitcoin protocol is being used. That’s really it. Not too much flash but it’s used to keep track of any changes that may have occurred on the network. Like I mentioned before, this data can be represented in a few different ways (Decimal or HEX). The block explorer I’m using on block 550204 outputs the version as ‘536870912’ which is not in the right format, so when we convert to HEX we get:
20000000
Previous Block
After the version we want to pull in the previous block’s block header, which is pretty easy as all the work to calculate the last block has already been completed. For block 550203 the header is:
000000000000000000025e631d4a6e455d3e62a9c8cdb071db7c4bbbc7cd15db
Merkle Root
Really the most important piece of information in a block, this hash is the link between the block and all the transactions that are within the block and that the miner is looking to process.
I won’t go into the exact process of bundling the transactions or how the Merkle Root is calculated but in its simplest terms, the root is comprised of the hash of all the hashed transactions within the block. This is an important process which we won’t dig too much further into but if you want to dissect what a Merkle Tree is and how it works, I suggest giving this a scan. In any case, the miner bundles the transactions and calculates a Merkle Root:
e9baa7dfae6f2fe4bcef43907527bde694c78d3210007db6e2364acec6aafedd
Timestamp
Interestingly, due to the time it takes a miner to calculate (read: guess) the nonce, the timestamp is the only field outside of the Nonce that will be changing per block calculation. Every couple of seconds the timestamp will change, and while this has nothing to do with validation of the header like the example we are going through, it is interesting to note because if you’re familiar with how hashes work, a small difference in a value that is being hashed changes the output completely.
At any rate, the timestamp on my explorer is illustrated as:
11/15/2018 3:58:08 PM
This time stamp will occasionally be represented in EST, or PST and we will need to be sure we are looking at the time from UTC (or Universal Time Code). Once it is in UTC we will need to convert to Unix time:
1542254288
Finally, it will need to be converted to HEX formatting leaving us with our final value as:
5BECEED0
Difficulty
The next step is to retrieve the difficulty, also not the most difficult (puns). As it can be done quite quickly from any block explorer. The difficulty is how difficult it is for a miner to find a solution to the Proof of Work statement and directly relates to how the target has been generated. Difficulty’s role in this calculation is a bit interesting as we won’t be using the actual difficulty, we will be using the “Bits” value. Bits is a packaged way of storing the target of the original block header, and it also fits into the calculation of the difficulty (it is better explained here and here).
The explorer we are using shows Bits as:
388443538
Once we can convert to HEX we get:
17272D92
Nonce
I like to equate the nonce to the dial on a safe to a safe cracker, it’s consistently changed to try and find the right number to solve the Proof of Work requirement and subvert the target. Like we mentioned above if the miner can find the hash that is below the target they win, if this was an equation, the nonce would be our variable (x).
Our block explorer says this block’s Nonce was found to be:
3080241068
Once we convert to HEX we end up with:
B798BFAC
Formatting, Hashing, and Finishing
OK, for those following along at home we now have our 6 pieces of information:
From here we need to reverse the byte order of all of our information. This byte order and the process of ordering from ‘big-little’ or ‘little-big’ (called endianness) could have an article all to itself. So for all our sanity’s sake, lets just stick with saying we are “reversing it’s byte order” (for those interested in a deeper dive, read more here). With that in mind we should be left with values that look like this:
Now we need to concatenate all this information into one long annoying string in the order listed above. This gives us:
00000020db15cdc7bb4b7cdb71b0cdc8a9623e5d456e4a1d635e02000000000000000000ddfeaac6ce4a36e2b67d0010328dc794e6bd27759043efbce42f6faedfa7bae99097ed5b922d2717acbf98b7
Which isn’t pretty, but we can finally start the Hashing process. We will need to use a SHA256 Hash. (NOTE: if you don’t happen to have a hash calculator lying around, I would suggest using this one as some other generators consider your input a string and will give you an incorrect answer).
If we apply the hash on the above string, we should end up with the below as an output:
44efc4426f98adb5a53ca4b4f1a0fbd5b2a18fa4db0f29302fc9074e2c002cb2
We will have to hash it again, which gives us:
3a024d7644732334d6b208c94aa4b68790ca804c506b0d000000000000000000
Lastly, we’ll reverse the byte order into big-endian format (Again for those interested in endianness):
0000000000000000000d6b504c80ca9087b6a44ac908b2d634237344764d023a
Which if we did everything right should equal the block header, which noted above was:
000000000000000000d6b504c80ca9087b6a44ac908b2d634237344764d023a
While doing something like this isn’t necessary it does give the participants of the network a bit of comfort that Bitcoin does in fact do some manually provable things, such as utilizing a SHA256 hashing algorithm and that the information in one block is in fact linking to the next block by using the previous block hashes in its block header calculation. These are really important pieces of information to not only the network, but also in reassuring those who are interested in getting involved in Bitcoin/Crypto community that the Bitcoin blockchain is “working”.
Hopefully this will help those who find themselves on the same path as I did when I first got to the crypto community, trying to see evidence that the coin is working without an entire code review. While I’m sure there are easier processes to convince those hesitant to work within the space this is almost always my go to reaction. With any luck there are others out there who inevitably need to mess with the inner-workings of something to see all the moving pieces like I do (sorry about that VCR, Mom).
