John Newbery's blog

The Economics of Public EV Charging in the UK

2023-02-08T00:00:00+00:00

Originally published on the Gridcog Blog.

Introduction

A few months ago, we published an article on The Economics of Public EV Charging in Australia. In that article, we attempted to find the price that owner-operators of EV chargers should charge users in order to break even on their investment over a 10 year period. The analysis was run for different size EV chargers located within a shopping centre in different cities across Australia, since energy and network costs vary widely across the country.

Public EV charging prices have become a hot topic in UK news in recent weeks, with articles comparing the driving costs of EVs and ICE vehicles, and recent stories about multiple EV charger operators introducing time-of-day dynamic pricing. It seemed like a good time to dive in and do some analysis on the financials of operating an EV charger in GB.

In this analysis, we selected a single site in London for our shopping centre (network costs vary much less by region in GB than in Australia), and ran the simulation for a 50kW, 175kW and 250kW charger.

Assumptions

Just like for Australia, we needed to start with a number of assumptions about the physical and economic characteristics of the EV charger and its usage.

Asset cost

We model a shopping centre adding a single EV charger of each type, where the cost of the charger is amortised over the 10 year duration of the project, with a fixed leasing cost that covers installation, operation and maintenance. We use similar infrastructure costs as for Australia, converted into GBP.

Energy supply costs

We’re not an energy price forecaster, so once again we turned to our friends at Cornwall Insight for our wholesale energy price curve. We’ve used the central case forecast for GB Day Ahead prices. To keep the model simple, we assume that the site owner takes the wholesale price for energy.

Network Costs

The shopping centre is located in London, and we’ve used the UK Power Networks London Low Voltage Site Specific tariff. We also included triad charges - although these are being phased out, until we know what they’ll be replaced by these are the best indication of what future transmission use-of-system costs might be.

Physical Constraints

As for the Australian analysis, we assume no physical constraints for the site connection. Although EV chargers are demand-heavy, we’re co-locating it within a site that already has a peak demand of around 900kW. That means that the EV charger’s demand does not have a significant impact on the site’s peak demand.

In other situations, physical constraints may require the site owner to reduce the peak demand impact of the EV chargers on the site connection, either by restricting charging events or by investing in co-located energy assets such as battery storage.

Charger Utilisation

We use the same assumptions about charger utilisation as for our previous post - 5 charging events of 30kWh per day in 2023, increasing by around 350% over the duration of the 10-year period. The chargers are in use during the opening hours of the shopping centre, from 7am to 8pm. That limits the maximum possible load factor to be around 54%.

Modelling

Our standard approach when modelling behind-the-meter projects is to begin with a baseline scenario that captures the physical and economic characteristics of the site under business-as-usual conditions. We configure our baseline assumptions, which consist of:

Physical aspects of the site, such as location, grid connection and constraints
Electricity demand for the site. In this project, we simply load in historical data, and then cast that forward to forecast future demand.
Economic characteristics, including markets that the site is exposed to, network and retail tariffs, and cash flows between multiple participants For this analysis, our business-as-usual scenario is very straightforward. We have a demand curve for the shopping centre, along with a network tariff and wholesale energy market prices that are both costs for the shopping centre owner.

We then build three alternative scenarios, each of which models a different sized EV charger. For each scenario, we configure the leasing cost of the asset (assuming zero capex since the asset is leased for a fixed annual cost), along with the expected utilisation model. The simulation uses an agent-based model to forecast the time of arrival of vehicles within the charging window. If a vehicle arrives while the charger is in use, it joins a queue and waits until the charger becomes available to start charging. If the queue length exceeds one hour, then the driver leaves without charging. This has implications for the low power charger in later years - the number of cars that want to use the charger exceeds the charger’s capacity for delivering energy at certain times, meaning potential customers are turned away.

The graphs below show the total energy delivered and load factor of each charger for each year.

For each half-hour time interval, the simulation software generates the total site demand, including both the pre-existing demand from the shopping centre and the additional demand from the EV charger. In the image below, the top graph is the energy demand in the business-as-usual scenario, and the bottom graph is the energy demand with a 50kW charger. The blue trace represents the total grid import, and the grey trace represents the additional demand from the EV chargers. This time period is pulled from 2023 in the project, when EV charger demand is at its lowest.

The total cost of supplying the demanded electricity (including the energy wholesale price and network charges) is calculated for both the baseline and 50kW charger scenarios. The difference between these is added to the annual lease for the EV charger to obtain the total cost for the charger. Dividing this cost by the energy delivered gives the breakeven price for the EV charger.

Results

If we graph the breakeven price of the EV chargers across the length of the project, the results are perhaps unsurprising:

The more powerful the charger, the higher the charging price needs to be for the asset owner to break even. That’s because the higher asset leasing costs are shared between the same number of customers, and the higher power demand also results in slightly higher network costs.
Over the 10 year project period, the breakeven price drops by over 60%. This is due to the forecast for wholesale energy dropping over time, and the increasing utilisation of the EV charger meaning the fixed overheads are shared over a larger customer base.

Visualised in table format, we can dig into the data in more detail:

Interestingly, the breakeven price for a 250kW charger is 60% higher than a 50kW charger in 2023 (52.9p/kWh against 33.2p/kWh), but only 43% higher in 2032 (19.9p/kWh against 13.9p/kWh). This is because the lower powered charger is not able to deliver as much energy, and has to turn away potential customers. The fixed costs are therefore shared over fewer charging events.

If we drill down into the cost stack, we can see where the difference in breakeven price originates. The higher powered chargers have a higher leasing cost, which dominates the cost stack in early years. They also result in marginally higher network costs. Over time, the fixed asset opex dominates less due to higher utilisation and the difference in network costs increases, but not enough to offset the reduced effect of the opex.

Finally, we might want to ask ourselves what the breakeven price is over the entire 10 year project:

According to our model, EV charger operators offering charging at those prices would be losing money today. However, we do know that many EV charging companies are operating at a loss today in order to gain market share. Presumably their hope is that as energy prices decrease and EV adoption picks up, they’ll start to turn a profit on those charging stations. It’s impossible to say how that’ll work out for them, except that they’ll need to have deep pockets in the meantime!

Conclusion

Owning and operating public EV charging infrastructure in a profitable way is a complex challenge and one that the market is only just beginning to tackle. Companies investing in EV charging today are having to take a view on a range of important assumptions, including charger utilisation, the physical behaviour of the hardware, site and network constraints and of course energy supply cost, both in terms of the commodity and network.

Being able to quickly model different deployment scenarios with different physical and commercial assumptions is therefore critical to building confidence in investment and pricing decisions.

UK Residential Solar PV Payback Analysis

2022-12-09T00:00:00+00:00

Originally published on the Gridcog Blog.

Introduction

Our previous post looked at the difference between what a residential solar owner in London might be paid under the various SEG export tariffs and what that energy would be worth on the balancing market. We found that the export tariffs on offer were much lower than both the very high wholesale prices and the import tariffs (for energy consumption rather than generation) that the suppliers are offering to customers.

However, installing a PV system could still be a good investment for households who self-consume the majority of generated energy and so reduce the amount of energy purchased from their supplier. With the current energy price guarantee at an eye watering 34p/kWh from October 1st, solar is becoming an increasingly attractive option for many households.

Assumptions

We modelled out the following assumptions in the Gridcognition software to see how long it’d take to pay off the investment in a new residential solar system:

The total annual consumption for our typical household is about 5MWh. 30-min resolution interval data provided by the Greater London Authority.
The property is located in SE London. We get solar irradiance data for a typical meteorological year from Solcast.
Sensitivity to solar system size assessed using 1kW, 2kW, 3kW, 4kW and 5kW, all oriented 180° south at a tilt of 35°.
Sensitivity to energy supply costs assessed using 16p, 20p, 24p, 28p and 34p per kWh. That more or less bookends the pricing from the lows of the Covid era to October’s price cap.
Model is run both with and without export under an 15p/kWh Smart Export Guarantee (SEG) tariff. That’s the best fixed rate available on the market today (Octopus Energy’s Fixed Outgoing tariff).
Solar capex costs assumed to be £1,300 per kW.

Results

The payback period is very sensitive to both the import and export tariffs, as you may expect. At one extreme, a 5kW array with a 16p/kWh import tariff and no export tariff would take 26.2 years to pay off, which is beyond the rated lifespan of many panels. At the other extreme, a 1kW array with 34p/kWh import and 15p/kWh export would be paid off in 4.5 years.

Since payback time will often favour lower capex at the expense of longer-term value, let’s also take a look at NPV after 10 years, based on a 5% discount rate:

Even without an export tariff, any size of solar array is a worthwhile investment if import tariffs stay at their elevated rate of 34p/kWh and excess generation can be sold at 15p/kWh. For systems without any export, only arrays sized at 1kW or 2kW will have a positive NPV after 10 years.

Interval data

Here’s a closer look at the energy flows for a 3kW system during the month of June.

In a typical week in summer, the PV system is exporting significant amounts of energy to the grid, and also reducing peak demand by just under one third.

Even in winter, the PV system is exporting some excess generation to the grid on sunny days, but isn’t making much of a dent on consumed energy or peak demand.

Across the year, a significant portion of the house’s energy demands are met by the PV system.

Analysis

Assuming the import rate remains at 34p/kWh (and that’s a big assumption), a 4.5 year payback period for a 1kW system is a great deal. If you can afford (or get financing for) a small solar array and you have a suitable roof, it’s probably a good investment.
At 34p/kWh, any size PV system gives a positive return after 10 years (assuming an export tariff of 15p/kWh).
There’s a huge amount of uncertainty in future energy prices. They’ll be impacted by both unpredictable global events and also government interventions in the energy market. Making investment decisions based on forecasted prices obviously carries risks. On the other hand, households may like the fact that generating their own energy reduces their exposure to volatile and unpredictable energy markets.

Are Increasing Wholesale Prices Impacting Solar Installations?

Another way to answer the question of whether PV systems are a good investment is to look at whether new systems are actually being deployed. Fortunately, the Department of Business, Energy and Industrial Strategy publishes monthly figures of new solar capacity installations, broken down by deployment size.

Here’s the monthly small-scale (<50kW) installations in the UK over the past 12 years:

It’s been quite a rollercoaster for the UK residential solar industry! The peaks in early 2012 and 2016 coincide with changes to rates and rules of the FiT scheme, and the huge outlier in March 2019 lines up with the end of the scheme and the rush to get installations accredited before the deadline of April 1st. Since that date, small-scale solar installations have been growing steadily without subsidies, with just a small blip down in early 2020 during the COVID lockdowns and a noticeable uptick since the beginning of 2022.

Let’s zoom in on the last three years and compare installations with Australia, which has a much larger and more mature residential solar industry. We took June 2019 as a baseline to avoid any distortions caused by the ending of the FiT scheme, since it’s likely that many installations were brought forward to meet the deadline and inventory and sales pipelines would be drained for a few months afterwards.

The Australian data was taken from Sunwiz, and includes all systems up to 100kW. The comparison therefore isn’t exactly like-for-like, but is directionally correct.

Australia starts from a much higher baseline of 178 MW of installation per month. However, there’s only moderate growth in rate of installation, with ~200-300MW for most of the period. Still, that’s around 3GW of new installed capacity per year!

The UK, on the other hand, starts from a much lower baseline of 8MW in June 2019, but rises to 56MW in September 2022. That’s an increase of over 650%.

Conclusion

As energy wholesale prices continue to be elevated and volatile, and retail tariff rates reflect those high prices, households’ economic case for residential solar installations will only grow stronger. Even without feed-in tariffs or SEG tariffs that accurately reflect the fair value of exported energy, it’s likely that more and more households will install small-scale solar generation to save money on their energy bills.

Solar Export Tariffs in GB

2022-09-05T00:00:00+00:00

Originally published on the Gridcog Blog.

Introduction

Our previous post compared the value that residential PV solar owners in Australia can get from the various Feed in Tariff (FiT) schemes in different states. We found that the relative economics of wholesale market prices and FiT schemes varied widely, from pretty generous FiT pricing in Melbourne and Adelaide, through to much less attractive pricing in Sydney and Brisbane.

Today, we’re going to do a similar exercise for the UK. Like Australia, the UK has two electricity markets, and for today’s analysis we’re going to be focusing just on Great Britain (Northern Ireland shares an electricity market with Ireland).

England, Scotland, and Wales share a single energy market with the same renewables regulations and incentives. We’ll start by looking at GB’s renewables incentive schemes before we dive into the modelling.

The Feed in Tariff and Smart Export Guarantee

The UK’s old Feed in Tariff scheme was launched in 2010 and offered fixed export tariffs for low carbon and renewable generators, segmented by technology type, capacity and energy efficiency. That scheme was closed to new participants in March 2019, although those already signed up to the FiT will continue to be eligible for the remainder of their 20 or 25 year term.

In January 2020, the government launched the Smart Export Guarantee, a new scheme for small-scale low-carbon generators. Unlike the FiT, the SEG doesn’t dictate the rate for exported energy. Instead, it simply compels large energy suppliers (those with more than 150,000 domestic customers) to offer an export tariff.

The SEG does not determine the terms of the tariff, except that the rate must be “above zero”. Small-scale generators are therefore motivated to shop around for the best export tariff. Interestingly, the generator can choose different suppliers as their SEG provider and their energy provider.

The SEG landscape

Ofgem publishes a list of all SEG licensees every year. For the third year of the scheme (2022/23), there were twelve mandatory licensees (suppliers with more than 150,000 domestic customers) and three voluntary licensees (smaller suppliers that chose to opt in to the scheme).

One of the mandatory licensees (Bulb) has been placed in special administration but is still accepting new customers, and two of the voluntary licensees (Cilleni Energy Supply and Smart Pay Energy) do not appear to have active websites. Residential solar owners therefore do have a bit of choice on where to sell their excess energy generation. Just like for picking an energy provider, there are price comparison websites to help generators find the best deal.

All of the active SEG licensees offer fixed rate export tariffs, with Octopus’s Outgoing Fixed tariff being the most generous at 7.5p/kWh and E’s SEG January2020v.1 tariff the stingiest at 1p/kWh. Most of the other fixed rate tariffs sit in the 3-6p/kWh range. Given that most electricity consumers will be on an import rate fixed by Ofgem’s energy cap of 52p/kWh from October 1st, all of these fixed rate tariffs are leaving the generator significantly out of pocket.

A supplier buying energy from a small-scale generator at 4p/kWh and selling at 52p/kWh is making a 13x markup (ignoring distribution and transmission losses). That’s a pretty juicy deal for the supplier, and a pretty bad deal for the solar household.

Octopus Energy is the only supplier to advertise a variable rate tariff, the Outgoing Agile tariff. That tariff has significantly outperformed the fixed rate tariffs in recent months, which is perhaps unsurprising given the elevated balancing market spot prices over that period. The Outgoing Agile tariff is only available to customers who use Octopus as their energy provider.

The model

Just as we did in Australia, we’ve modelled a typical house with a 5kW PV solar system, but this time we’ve transported it to south east London (and remembered to move the panels to our south-facing roof).

London’s high latitude and cloudy skies mean we’re not going to get as many kWh out of our PV system as we did in sunny Perth or Sydney, but our previous analysis has shown that even with lower irradiance, London may be a better investment for C&I solar due to the available network tariffs. Let’s see how the investment stacks up when the owner is on an SEG tariff.

We’ve assumed an SEG flat rate of 5p/kWh for 2020, rising to 7p/kWh for 2021-22. That’s the right ballpark, but if anything may be a little generous given the tariffs that are currently available.

Results

Wholesale energy prices were suppressed throughout 2020, so even with a low export rate of 5p/kWh, households feeding in to the grid would have been ahead.

However, since mid 2021 wholesale prices have increased dramatically – a combination of increased demand after lockdowns and gas price rises driven by supply chain issues and most recently Russia’s invasion of Ukraine. SEG rates have not kept pace with those prices, even as suppliers pass increased costs onto their customers.

If you thought Brisbanites and Sydneysiders were getting a bad deal, then Londoners have it even worse. In the first 8 months of 2022, our model household would get paid £345 below the ‘fair value’ wholesale price for their exported energy.

Here’s the interval data for a typical week in February for the data fiends. Even in the middle of winter, the solar system is exporting some energy to the grid on most days.

Conclusion

All of the fixed rate SEG tariffs significantly underpay the generator compared to the fair market value of the energy they’re exporting. Even Octopus Energy’s relatively generous Outgoing Fixed tariff offers far below the market price.

It’s therefore unlikely that you’ll get a fair price from your SEG supplier for exported energy, which creates a very strong incentive for households to self-consume all of their generated electricity, by shifting loads to the middle of the day or adding new loads like electric vehicle charging, electric heat pumps, and home batteries.

In our next post we’ll look at the other financial aspects of installing solar, including what the expected payback period is for different sized systems, and opportunities for load shifting. We’ll also look at recent installation numbers for small-scale solar systems in the UK.

It’s our hope that as the rate of rooftop solar adoption increases and high prices cause energy prosumers become more price-sensitive, we’ll see SEG licensees improve the export tariffs that they offer. Who knows, perhaps the SEG landscape may one day evolve into a truly competitive market where the SEG licensees are competing on the rate they offer generators!

Notes

The load data is for a typical UK household with an annual consumption of 5MWh, with a load shape from the Greater London Authority.
The PV system has a capacity of 5kW oriented south with a tilt of 35°.
All generation is self-consumed if possible. Excess generation is exported to the grid under a fixed rate SEG tariff.

Comparing Contingency Frequency Control Markets in Australia and the UK

2022-08-01T00:00:00+00:00

Originally published on the Gridcog Blog.

Introduction

In a recent blog post, we compared the financial case for installing solar (and optionally battery storage) to a supermarket in Australia versus the UK. That analysis only considered using the solar generation and battery storage to reduce spend on energy imported from the grid, and showed that although the solar irradiation in London was around two-thirds that of Sydney, the savings were much greater due to the nature of the network tariffs available in each location.

However, when considering the economics of deploying distributed energy resources, reducing spend on energy is only part of the picture. Grid edge assets are able to participate in markets and provide network support services. These new revenue streams can greatly affect the economics of the project and strengthen the case for deploying new assets. In this post we’ll look at one type of market, contingency frequency response, and compare the revenue opportunities in Australia and the UK.

What is Contingency Frequency Response?

The primary role of the System Operator is to maintain the security and reliability of the electricity grid. As well as operating a market for the sale and purchase of electrical power in specified time blocks, the operator must also procure reliability services in order to contain the system within its specified operating parameters (regulation services), and to quickly bring the system back into those operating parameters in the case of a fault (contingency services).

One operating parameter that must be maintained for the grid to remain stable is the frequency of the A/C being delivered. In both the UK and Australia, the frequency of the grid is 50Hz, with a narrow operating band around that specified frequency (49.8Hz – 50.2 Hz in the UK, 49.85Hz – 50.15 Hz in Australia). If the frequency strays outside that band, the system operator moves quickly to adjust generation or load in order to maintain the balance of supply and demand in the system. Although the System Operator is controlling the dispatch of generation and load, those resources are provided by market participants, and competitively bid for in specified time blocks.

In Australia, AEMO operates six FCAS (Frequency Control Ancillary Services) markets in the National Energy Market: fast raise and lower; slow raise and lower; and delayed raise and lower. In the UK, National Grid ESO operates two DC (Dynamic Containment) markets: low DC and high DC. These are relatively new services, with a soft launch for low DC in late 2020 and a full launch with low and high DC in late 2021.

How does FCAS compare to DC?

We’ve analysed what the FCAS and DC markets have been worth for the last few years. Going back to 2019 means we can include some of the more spectacular contingency events seen on the Australian NEM. For 2022, we’ve prorated the January-July figures, for an easier like-for-like comparison with previous years.

Our analysis shows that so far in 2022, the value that UK battery owners can capture from participating in the DC markets is over twice that of their Australian counterparts selling into the FCAS market. It remains to be seen whether those high prices in the DC markets will remain, or whether new entrants will drive down the price.

Asset owners will also want to try to stack frequency support with some other value streams like wholesale arbitrage. On that front the NEM in Australia is very different from the UK with $16,000/MWh top-to-bottom price swings and 5 minute settlement periods which are favourable to rapid operating characteristics of batteries.

Notes

The six contingency FCAS markets in the NEM have been rolled up into two (FCAS Raise and FCAS Lower)
The Australian NEM prices are taken from the NSW trading region
Dynamic Containment Low (DCL) is equivalent to FCAS Raise and DCH to FCAS Lower
All values presented in Australian dollars

LABITCONF 2021 - Future Protocol Changes

2021-11-19T00:00:00+00:00

I spoke at LaBitConf 2021 in San Salvador. I gave a talk on future Bitcoin protocol changes and appeared on a panel about open source Bitcoin protocol development.

Talk slides

The Brink Podcast

2021-11-10T00:00:00+00:00

Gloria and I started a podcast about Bitcoin protocol development. We plan to get very technical and explore areas of the protocol and code that you won't hear about on other podcasts! Join us to learn about how Bitcoin is built, and what protocol developers think about when they're working on the code.

CopperCast - Open Source Bitcoin Funding

2021-07-22T00:00:00+00:00

I appeared on CopperCast to discuss funding open source Bitcoin protocol development, and how Brink is supporting open source developers through its grant and fellowship programs.

The B Word - Open Source Money

2021-07-21T00:00:00+00:00

I gave a talk about Bitcoin as Open Source Money at the ARK Invest/Square Crypto B Word event.

slides

MIT Bitcoin Expo 2021 - Developer Panel

2021-04-04T00:00:00+00:00

I was on a remote panel for the 2021 MIT Bitcoin Expo with Pieter Wuille and Gloria Zhao, hosted by Brian Bishop.

Coinscrum - Funding Bitcoin Core Development

2021-03-04T00:00:00+00:00

I chatted with Nicholas Gregory about Taproot, Bitcoin Core funding and setting up Brink.

Bitcoin Magazine - Developing Bitcoin

2020-12-29T00:00:00+00:00

In this interview, I talked to Christian Keroles to discuss why I started Brink, and the process of establishing the organization.

Christian then turned the conversation to the bigger picture of Bitcoin development. We discussed the difference in developing Bitcoin in a bear market versus a bull market, what the Bitcoin development process looks like and how a distributed group of stakeholders can make sure that developers are working on the most important things in the protocol.

Lastly, we discussed Taproot and how it demonstrates that the Bitcoin community has improved its coordination. We also discussed the concept of ossification in the Bitcoin protocol and how developers think about Bitcoin's code.

Additional topics discussed included:

Launching Brink with Mike Schmitt
Bitcoin Optech
Becoming a 501(c)(3) nonprofit
How the donation landscape is changing for Bitcoin development
Comparing and contrasting development in a bull vs. bear market
Bitcoin and ossification
Reflecting on the taproot process

MIT DCI Security Roundtable - A History of Bitcoin Security

2020-12-04T00:00:00+00:00

I spoke at MIT DCI's 2020 Security Roundtable about the history of security in the Bitcoin Core project. The talk was not recorded, but you can see the talk slides here.

What Bitcoin Did - Building a Bitcoin Developer Community

2020-12-04T00:00:00+00:00

In this interview, I talked to Peter about leaving Chaincode Labs, funding for open source Bitcoin development and starting Brink.

On The Brink Podcast - Funding Bitcoin Development

2020-11-24T00:00:00+00:00

In this interview, we talked about:

My history in Bitcoin open source development and how I came to found Brink
Why I left Chaincode and struck out on his own
Brink’s mandate and foundational purpose
Lessons learned from the Bitcoin Foundation
The future of Bitcoin Optech
The state of funding for Bitcoin development
The accessibility of Bitcoin protocol development today
Whether the existence of financial incentives cannibalizes the intrinsic motivation to work on open source
Why I work on Bitcoin
Why ossification might be more remote than we expect
Whether Bitcoin’s developer funding model exposes it to corporate capture
The political implications of Bitcoin having a sole reference implementation
The importance of distinguishing the validation element of Bitcoin Core from the other components
Whether Bitcoin protocol development is meritocratic or technocratic.
Why the structurelessness of Bitcoin core dev raises the barriers to entry
Whether Bitcoin protocol development is adequately funded right now
Whether developer funding equates to influence in the Bitcoin protocol development
The dispersion of Bitcoin protocol development influence

The Stephan Livera Podcast - Announcing Brink

2020-11-24T00:00:00+00:00

In this interview, I talked to Stephan about establishing Brink to support open source Bitcoin protocol development. Topics covered:

Why I started Brink
How developer choose their priorities
Mentoring new developers
Bitcoin development culture

she256 Onboarding to Bitcoin - Contracts

2020-04-08T00:00:00+00:00

As part of the she256 "Onboarding to Bitcoin" webinar, I gave a presentation about contracts in Bitcoin.

slides

MIT Bitcoin Expo 2020 - Developer Panel

2020-03-07T00:00:00+00:00

I was on a panel at the 2020 MIT Bitcoin Expo with Pieter Wuille, Cory Fields and Amiti Uttarwar.

The Chaincode Podcast

2020-01-27T00:00:00+00:00

I started a podcast with Adam Jonas at Chaincode Labs. We interviewed influential Bitcoin protocol developers about deep technical topics in Bitcoin development. While I was at Chaincode, we produced 7 episodes:

I was very happy to see that after a long hiatus, the Chaincode podcast continued with Adam Jonas and Mark Erhardt hosting. They've had some great guests, including Carl Dong, Pieter Wuille and Clara Shikhelman.

LABITCONF 2019 - Taproot and Schnorr Signatures

2019-12-03T00:00:00+00:00

I gave a talk at LABITCONF 2019 in San Salvador on the upcoming Taproot and Schnorr Signatures soft fork. Unfortunately, the video of the talk is not available online, but you can see the talk slides here.

What Bitcoin Did - Building a Bitcoin Developer Community

2019-08-09T00:00:00+00:00

In this interview, I explained to Peter what working on the Bitcoin protocol is like, the risks of a reducing block subsidy for miners, and privacy and development on the Lightning network.

Peter also spoke to 3 of the residents on the Chaincode Summer Residency. They talked about their diverse backgrounds, how they ended up in New York, and what they'll be working on once the residency is over.

Chaincode Summer Residency 2019

2019-06-17T00:00:00+00:00

In 2019, we hosted the third Chaincode Residency, our most ambitious to date. I co-organized and co-hosted the program, and also gave a series of technical talks:

Security Models

Wallet Development

World Crypto Network - Onboarding Bitcoin Developers

2019-06-09T00:00:00+00:00

Jonas and I sat down with Max Hillebrand during the Amsterdam Breaking Bitcoin conference to discuss getting into Bitcoin open source development, and the next iteration of the Chaincode residency.

Magical Crypto Conference 2019 - Protocol Development Panel

2019-05-12T00:00:00+00:00

I was on a panel discussing Bitcoin Protocol Development with Eric Lombrozo, Matt Corallo and Luke Dashjr, hosted by Katherine Wu at the Magical Crypto Conference 2019.

We talked about:

How changes are made to the Bitcoin protocol
How the schnorr/taproot softfork will improve privacy and fungibility in Bitcoin
The Binance hack and the possibility of exchanges attempting deep reorgs
Maximalism and other blockchain projects

Scaling Bitcoin Operations

2019-02-15T00:00:00+00:00

I gave a talk to BitMEX engineers about "Scaling Bitcoin Operations". In the talk I gave a background on Bitcoin's fee market and Bitcoin Optech, spoke about various techniques available today for reducing transaction fees (payment batching, segwit and patient spending), and discussed upcoming techniques for reducing fees (Lightning Network, Schnorr signatures and Threshold/multisignatures).

Slides

Lightning Applications Residency

2018-10-22T00:00:00+00:00

I organized and hosted the Lightning Apps residency in New York in October 2018.

The opening presentation is viewable online, along with all of the technical talks and resident project demos.

Noded 0.29.0 - The Inflation Bug

2018-10-13T00:00:00+00:00

I appeared on the Noded podcast a third time to discuss the recent Bitcoin inflation bug, and the reaction to it.

Transcript

Bitcoin Edge Dev++ Tokyo - Digital Signatures and Script

2018-10-04T00:00:00+00:00

I gave two educational talks at the second iteration of Bitcoin Edge Dev++ at Keio university in Tokyo, immediately preceeding the Scaling Bitcoin Conference.

I covered the following topics:

Digital signatures - finite fields, elliptic curves, Schnorr signatures and ECDSA

Corrections:

The slide on Cyclic Groups said "The integers modulo p for any number p is a cyclic group". It should say "The integers modulo p for any prime p is a cyclic group". (If p is composite, ie p = ab, then the order of a in G is b, so G is not cyclic).
during the talk, I was asked “how does the verifier know K in the non-interactive schnorr identification protocol?” I said that K can be provided by the prover. That’s wrong. In fact, the verifier calculates K himself and then checks that e is the hash of K. The verification step is checking that e does in fact commit to K.

The slides above are corrected.

Bitcoin script

Crypto News Network - Perspective

2018-08-21T00:00:00+00:00

I chatted with Vortex on the Crypto News Network about Bitcoin Optech and other Bitcoin topics.

Chaincode Lightning Apps Residency

2018-08-20T00:00:00+00:00

Chaincode is proud to announce a Lightning applications residency program, taking place in New York, Oct 22–26 2018! We’ve already hosted two residency programs, in 2016 and 2018 . Those were focused on the Bitcoin protocol and contributing to Bitcoin Core. This time we’re going to try something a bit different.

The Lightning network is maturing every day and we’re beginning to see great applications like satoshis.place and yalls.org being built on top of it. There’s a huge amount of excitement growing around Lightning applications, but knowledge of how to build those applications is currently scarce and diffuse.

We want to help bootstrap the emerging Lightning applications developer ecosystem, so we’re hosting a week-long residency specifically for Lightning application developers (as opposed to Lightning protocol developers), organised by us, with mentoring and speaking by experts in the Lightning applications space.

We’ve got a stellar line-up of guest speakers:

Chris Steward is the founder of SuredBits, a real-time micropayment-powered data service, built on top of the Lightning network. He is also the maintainer of bitcoin-s and a former Chaincode resident.
Christian Decker is an engineer at Blockstream and is a maintainer of the c-lightning Lightning implementation. He is also co-author of the 2015 Duplex Micropayment Channels paper and the 2018 eltoo paper.
Elaine Ou is an engineer at Global Financial Access. She has implemented the LightningBuddy and Jellybean applications on top of Lightning.
Jack Mallers is the lead developer of zap, a desktop and iOS wallet for Lightning payments
Justin Camarena is an engineer at bitrefill, the first payment processor to accept mainnet Lightning payments. He was the lead engineer for integrating Lightning into bitrefill’s payment system.
lightningk0ala is the creator of the wildly popular satoshis.place Lightning app. He’s an expert on Lightning-powered games.

We’re looking for experienced applications developers, perhaps with a background in web apps or Ethereum apps, who want to learn about integrating Lightning payments into applications. No experience of the Bitcoin or Lightning protocols are required.

The residency will be project-based, and applicants will build a prototype application over the course of the week. Residents will have a chance to present their work on Friday afternoon.

All guest speaker talks and presentations will be video recorded and made freely available. We want to scale the benefits of this residency to engineers who aren’t able to join us in person.

We’re really excited about this new residency and are looking forward to some excellent talks and amazing new Lightning applications

Originally posted at http://lightningresidency.com

London Bitcoin Devs - Bitcoin Core 0.17

2018-08-16T00:00:00+00:00

On 16 August 2018, I gave a presentation to the London Bitcoin Devs meetup group covering:

The upcoming Bitcoin Core v0.17 release
Chaincode Labs
Bitcoin Optech
General Q & A

Slides: BitcoinCore17.pdf

Video: youtube

Transcript: http://diyhpl.us/wiki/transcripts/london-bitcoin-devs/jnewbery-bitcoin-core-v0.17/

Noded 0.19.0 - Launching Optech

2018-08-03T00:00:00+00:00

James O'Beirne and I sat down with Pierre and Michael to discuss Bitcoin Optech, and to answer a selection of listener questions.

Tales From The Crypt #36 - Optech, Taproot and Lightning

2018-08-01T00:00:00+00:00

I chatted with Marty again. We talked about Bitcoin Optech, as well diving into Schnorr signatures, Taproot, the Lightning Network, and getting cosmic over some heady Bitcoin theories.

Distributed 2018 - Optech Launch Announcement

2018-07-20T00:00:00+00:00

Steve Lee and I announced the launch of Optech at Distributed 2018 in a chat with Aaron Van Wirdum.

We talked about what led us to start Bitcoin Optech and what we hoped to achieve with the project.

Optech's launch was covered in Bitcoin Magazine and Coindesk.

MIT Bitcoin Expo 2018 - Scalability Panel

2018-03-17T00:00:00+00:00

I was on a panel at the 2017 MIT Bitcoin Expo with Cory Fields and Andrew Poelstra, hosted by Paul Wegzyn. The audio is very poor for the first fifteen minutes of the video, but improves after that.

We talked about scalability, protocol development and the difference between onchain transactions and layer 2 solutions.

Noded 0.6.0 - Contributing to Bitcoin Core

2017-12-23T00:00:00+00:00

I chatted with Pierre and Michael about about the Bitcoin test framework, the Core process, and Bitcoin hypotheticals.

Tales From The Crypt #5 - Bitcoin and the Bitcoin Core process

2017-12-06T00:00:00+00:00

I chatted with Marty about some of the technical details of Bitcoin, Bitcoin Core's process, keeping the long-term in mind, and what everyone can do to make Bitcoin better.

Recurse Center - Bitcoin for Hackers

2017-11-30T00:00:00+00:00

I gave a talk at the Recurse Center in New York titled "Bitcoin for Hackers" (so-named because the previous name for the Recurse Center was Hacker School). The talk was a very quick introduction to electronic coins, the double-spend problem, and how Bitcoin solves that problem with proof of work.

Slides and Jupyter notebook are available here.

Bitcoin Edge Dev++ Stanford - Blocks, Blockchain, P2P, Mempool and HD Wallets

2017-11-02T00:00:00+00:00

I gave a series of educational talks at the Bitcoin Edge Dev++ event at Stanford, immediately preceeding the Scaling Bitcoin Conference.

I covered the following topics:

Building a Stronger Bitcoin Developer Community

2017-10-17T00:00:00+00:00

In Summer 2016, I’d been interested in Bitcoin for a little over a year. I’d been slowly becoming more and more enchanted by this strange and wonderful technology, and I’d finally made the step of quitting my job of eight years with the intention of finding a way to make a living by working on Bitcoin. Matt Corallo’s announcement of the first Hacker Residency was very happy timing. I no longer had any work responsibilities and the circumstances were exactly right for me to fly to New York and dedicate four weeks to diving into the deep technical details of Bitcoin with some of the most prolific and established developers in the space. I applied, and to my astonishment I was accepted onto the residency a few days later.

The program itself was everything I hoped for. I finally had the chance to be around people who were as enthusiastic and passionate about this bizarre social experiment as I was, to talk for hours about forks and consensus and security models, and to make my first contributions to Bitcoin Core. It’s fair to say that the program had a profound impact on my life. At the end of the four weeks I was offered a full time job by Chaincode Labs. A few months later I moved permanently to New York and now have the privilege of working full time on Bitcoin.

Announcing the second Chaincode Hacker Residency

That’s why I’m so delighted to be coordinating the second iteration of Chaincode’s residency program with Matt. The first program set me on the course to be a contributor to Bitcoin Core, and having spoken to a lot of very smart and technical people with a deep interest in Bitcoin, I know that there’s a real hunger out there for a program of this type. I’m hoping that we find some of those people and help them share in something that has given me so much pleasure over the last 12 months.

We’ve decided to slightly change the format of the program this time. Instead of a single four-week session, we’ve split the program into two two-week sessions. We hope that this will make the residency accessible to more people — even those who are currently in full-time employment.

The first session, Security Concerns and Adversarial Thinking is aimed at technical professionals who already have experience working on Bitcoin, perhaps on wallets, exchanges or applications, and who want a deeper understanding of the security assumptions and threat models behind the system. By the end of the two weeks, participants will have a deeper understanding of the security trade-offs behind the decisions we make, and have the vocabulary and conceptual basis to contribute to discussions on Bitcoin’s direction and future.

The second session, Becoming a Bitcoin Core Contributor is aimed at software developers with an interest in Bitcoin Core, but who have not yet made contributions to the project. We hope that by the end of the two weeks, our residents will be comfortable contributing to the project autonomously — opening pull requests, reviewing, testing and so on.

Doing both sessions back-to-back is also a possibility for residents who can dedicate four weeks to working on and learning about Bitcoin.

Wednesday night meetups

I’m really hopeful that we can build a stronger developer community for Bitcoin, both here in New York and more widely. I know there are people who’d like to participate in the residency program but who can’t take two or four weeks off to focus exclusively on Bitcoin. To make sure we’re able to reach them, we’re starting a second initiative to help people who want to get involved in the Bitcoin developer community.

On Wednesday evenings leading up to the residency, we’ll be opening our doors to a small group of developers who want to participate and contribute to open-source Bitcoin projects. We’ll go through some of the material from the residency program and be on hand to help with pull requests and Bitcoin projects. I’m sure that there are plenty of people out there who’d love to get involved, but just need a few pointers from established contributors. This program is for them.

If either of these programs sound interesting to you, please check out the Hacker Residency website for full details and consider applying. If you have any questions, comments or just want to talk about the Bitcoin community, please reach out to me on twitter @jfnewbery.

originally posted at https://medium.com/@jfnewbery/

Offchain - Bitcoin Core 0.15

2017-09-19T00:00:00+00:00

I chatted with Jimmy Song about how I got into Bitcoin Core development, and the new Bitcoin Core 0.15 release.

What's New in Bitcoin Core V0.15 - Part 5

2017-09-18T00:00:00+00:00

This is the final article in this series about features and improvements in Bitcoin Core v0.15. Today I’ll talk about a small implementation change by Matt Corallo that gives a huge boost in performance for nodes that are in sync and keeping up with the blockchain.

Script Caching (PR 10192)

Block validation: now up to 50% faster!

This is another great performance improvement, this time in new block validation time when the node is up to date with to the chain tip. Under normal circumstances, blocks will now validate around 40–50% faster.

How can we achieve such a dramatic speed-up? It starts from the observation that once your node has been up and running for some time, every time it receives a block it will already have seen the majority of transactions within that block. That’s because the miner who created the block is collecting the transactions for the block from the same P2P network that your node is connected to, and so your node already will have seen the transactions as they were being propagated across that network.

In early versions of Bitcoin Core, almost every transaction would therefore be validated twice — first when it was received as an unconfirmed transaction and accepted into your mempool, and again when it was received in a block. Doing the same computational task twice is an obvious target for optimization, and the first big improvement here was the addition of a signature cache in v0.7. Signature validation involves elliptic curve math and is the most expensive part of script validation, so if we can save having to do a signature validation twice, that’s a big win. After v0.7, if Bitcoin Core received an unconfirmed transaction that validated and was accepted into the mempool, it would cache the result of the signature validations. Then, if it received a block containing that same transaction, instead of validating the signatures again, it would just look up the result in its cache.

This small change results in a huge performance improvement in block validation times.

The obvious question is: why not cache the validity of the entire transaction? Wouldn’t that be an even bigger performance win? Unfortunately, that’s not possible since transaction validity is context-specific. That’s to say that a transaction can be valid in one block, but invalid in a different block. Here’s one way that a consensus failure could occur if we cached the transaction validity:

The main chain tip is at height 1000
Transaction A is received with nLockTime set to block 1001. It is evaluated as valid since the next block will be height 1001.
A miner re-orgs block 1000 out of the main chain by building block 1000’ on top of block 999. Block 1000’ includes transaction A.

The result is a chain-split. Those nodes that first saw transaction A before the re-org consider block 1000’ to be valid since they already evaluated transaction A as valid. Those nodes that didn’t see transaction A before the re-org see block 1000’ as invalid since transaction A is invalid in any block before 1001.

So we can’t simply cache the transaction validity without introducing the risk of a consensus failure, but we can (and do) cache the signature validity. Matt observed that we can do something between the two: we can cache the validity of the scripts within a transaction, without caching the validity of the entire transaction. Crucially, within a Bitcoin transaction, validity of the scriptSigs is not contextual of any outside information. This is a really important point, and the recent CLTV (Check Locktime Verify) and CSV (Check Sequence Verify) BIPs were very carefully designed to preserve this property.

Matt’s change adds a script cache in addition to the signature cache, so from v0.15, if we receive an unconfirmed transaction we’ll cache both the results from the ECDSA signature validations, and the results from the input script evaluations.

Unlike Pieter’s per-output chainstate db change, this wasn’t a particularly large code change (+449 lines, -43 lines compared to +1109 lines, -1055 lines for Pieter’s change), but don’t underestimate the difficulty. People have attempted to do similar things to this before and introduced consensus bugs like the one described above. See PR 6077 and PR 5835 for example. The value of Matt’s contribution is that he has a deep understanding of the consensus code and more experience in writing and maintaining that code than almost anyone else. Very few people could have implemented something like this without introducing consensus bugs.

The result of this change is that new block validation is roughly 40–50% faster on average. A huge improvement!

Conclusion

Bitcoin Core v0.15 final was released this week and is available for download from bitcoincore.org. This release brings a few really nice new user features and some significant performance improvements, as well as laying the groundwork for further improvements and features in v0.16. There’s a whole slew of smaller changes that I haven’t included here. Check the release notes if you want a complete listing.

Thanks to Matt Corallo and Jimmy Song for feedback and input.

originally posted at https://bitcointechtalk.com/

What's New in Bitcoin Core V0.15 - Part 4

2017-09-17T00:00:00+00:00

In this series, I’ve already covered one performance improvement (per-output chainstate db) and a couple of user features (better fee estimation and bumpfee in the GUI). Today I’ll talk about another user feature, multi-wallet.

Multi-wallet Support (PR 8694)

Multi-wallet is a feature that’s long been requested by users, and has now been implemented in Bitcoin Core. Users are now able to load more than one wallet when starting the software. Each wallet is completely separated from all others, with its own keys, addresses, balance and transaction outputs.

There are many possible use-cases for multi-wallet, such as:

Having separate ‘personal’ and ‘business’ wallets, with accounting for those wallets completely separated.
One user having multiple wallets for different purposes, for example a ‘savings’ wallet and a ‘checking’ wallet.
One user having a normal transacting wallet, and an additional wallet for a 2nd layer application such as lightning, tumblebit or joinmarket.
Multiple users having a wallet on a shared node. Note that there is no authentication for individual wallets, so the users must trust each other. Future versions may introduce authentication for separate user access to individual wallets.

When using multi-wallet, each wallet is available through the bitcoin-cli command line tool and the JSON RPC interface. In v0.15 the GUI will only display and allow transactions to be created and signed on the first wallet. However, additional loaded wallets will continue to remain synced to the node and will continue to keep track of their balances and transaction which involve them. This usage pattern, called ‘wallet warming’, is useful if you want your home node to keep track of the wallet’s transactions and balances, but transact with that wallet on different software (eg a separate instance of Bitcoin Core). Future versions should allow all wallets to be controlled through the GUI.

I’m excited about multi-wallet in v0.15, and I think that changes in v0.16 and beyond will build even more functionality on top. As an example, v0.16 should allow wallets to be dynamically loaded and unloaded during runtime, so it will no longer be necessary to stop and restart the node in order to load a new wallet. Loading and unloading a wallet should be as simple as opening a new document in a word processor.

originally posted at https://bitcointechtalk.com/

What's New in Bitcoin Core V0.15 - Part 3

2017-09-16T00:00:00+00:00

Yesterday, I wrote about how fee estimation is better in v0.15, but what happens if you send a transaction without enough fee to be included in a block? Somehow you need to encourage miners to include your transaction, or it’ll be stuck in limbo forever (or at least until it drops out of nodes’ mempools). There are two ways to ‘unstick’ your transaction and get it into a block:

Child Pays For Parent (CPFP) — a new transaction which spends the outputs of the stuck transaction. This child transaction attaches enough fee to pay for itself and its parent transactions to be included in a block. Since v0.13, the Bitcoin Core mining code is intelligent enough to consider ‘packages’ of transactions when filling a block, and will look at the total fee-rate across the ancestors and descendant (this feature should rightly be called Descendants-Pay-For-Ancestors since the mining code considers chains of transactions up to 25 in length). That means that if you create a descendant transaction with a high fee, the miner will include both the ancestors and descendant transactions in the next block so that it can collect the large total fee for the transaction chain.
Replace By Fee (RBF) — a new transaction which essentially double-spends the stuck transaction. The signer of the original transaction creates a new replacement transaction that spends the same inputs but pays additional fee (usually by reducing the amount of bitcoin for the change output and leaving the extra bitcoin as fee for the miner). If the replacement transaction attaches enough fee, then miners will be incentivized to include it in a block.

Replace-by-fee in the GUI (PRs 9592 and 9697)

Since v0.12, Bitcoin Core has supported ‘opt-in RBF’. This is where a transaction signals that it is replaceable and other nodes will allow RBF transactions to replace the original if enough additional fee is attached. Since v0.14, Bitcoin Core has also included a bumpfee command which will create a replacement transaction for an unconfirmed transaction. v0.15 brings this functionality into the GUI. New transactions can be marked as ‘opt-in RPF’ and stuck transactions can be bumped.

This is great news for end-users who interact with Bitcoin Core through the GUI rather than through the Command Line. For the first time they’ll be able to mark transactions as replaceable and then create RBF transactions if they get stuck. The end result should be users not over-paying transaction fees. That’s good for the individual user and also for the fee market overall — if all users are able to pitch their fee rate more accurately, the block space will be allocated more efficiently to those who value it.

originally posted at https://bitcointechtalk.com/

What's New in Bitcoin Core V0.15 - Part 2

2017-09-15T00:00:00+00:00

This article continues the series on enhancements and new features in Bitcoin Core v0.15. Yesterday I talked about a low-level technical change that results in dramatic performance wins. Today, I’ll cover something much more visible to users — significantly improved fee estimation.

Better Fee Estimation (PR 10199)

I recently wrote about Bitcoin Core fee estimation in v0.14. It’s a huge subject and I was only able to scratch the surface. I recommend you take a look at that article first to familiarise yourself with the concepts and terminology used in Bitcoin Core’s fee estimation.

Bitcoin Core v0.15 takes the same framework and makes some substantial improvements that makes fee estimation a lot more accurate and reactive to changes in prevailing fee rates. Here’s a high-level summary of the changes:

The algorithm is smarter about making sure enough data has been gathered in order to return a valid estimate. This means that fee estimation is less sensitive to extreme outliers skewing the estimates. If not enough data has been gathered, then the algorithm will return a fallback default fee.
The fee rate buckets are smaller, which means that the fee estimate returned by the algorithm should be more precise.
Estimates are now tracked on 3 different time horizons, which means the estimates adjust more quickly to changes in conditions, and also allow targets of up to 1008 blocks (~1 week). Previously only targets of up to 25 blocks could be used. This change is really useful for users who want their transaction included at some point, but are prepared to wait for a few days to substantially reduce their fee.
Fee estimates can now be conservative or economical. Conservative estimates use longer time horizons to produce an estimate which is less susceptible to rapid downward changes in fee conditions. Economical estimates use shorter time horizons and will be more affected by short-term changes in fee conditions. Economical estimates may be considerably lower during periods of low transaction activity (for example over weekends), but may result in transactions being unconfirmed if prevailing fees increase rapidly. For transactions that are marked as replaceable, the wallet will use an economical estimate by default, since the fee can be ‘bumped’ if the fee conditions change rapidly (See PR 10589).
A new estimaterawfee RPC has been added so that advanced users can implement their own fee estimates using the raw data collected by Bitcoin Core.

There are a number of changes under the covers that allow these improvements to be exposed to the user:

The whole algorithm is repeated three times over different time horizons: Short history (for targets up to 12 blocks), Medium history (for targets up to 48 blocks) and Long history (for targets up to 1008 blocks). For the longer time horizons, Bitcoin Core uses ranges of confirmation targets rather than keeping track of every target-bucket pair.
The estimateSmartFee() calculation previously used a success rate of 95% at the target number of blocks. That’s changed to now requires a 60% success rate at half the target, an 85% rate at target and a 95% rate at double the target.
The estimateSmartFee() calculation has also been improved so that if it fails to hit the success threshold for a certain target-bucket, it will see if it can get back above the threshold by continuing to add datapoints from lower fee rate buckets. This makes the algorithm more robust against outliers skewing the results.
The fee buckets are now spaced at 5% intervals instead of 10%. This is made possible by the above change, since the algorithm is now less susceptible to small datasets in a target-bucket pair resulting in a bad estimate.
Transactions which leave the mempool due to eviction or other non-confirmed reasons are now taken into account by the fee estimation logic, leading to more accurate fee estimates.
The estimateSmartFee() interface now takes a conservative/economical argument. Conservative estimates require a 95% success threshold for double the target to be met at longer time horizons.

As explained in my previous article, the Bitcoin Core fee estimation algorithm simply records and reports meaningful statistics about past events in order to give the user a reasonable estimate of how much fee they need to attach. It doesn’t attempt to be forward-looking and by design does not try to be too smart.

The changes in v0.15 continue this philosophy. The v0.15 algorithm isn’t trying to be too clever, but simply extends the existing algorithm to improve performance under a range of circumstances.

For full details of the algorithmic changes, see Alex Morcos’s gist and the Pull Request.

Thanks to Matt Corallo, Alex Morcos and Jimmy Song for input and feedback.

originally posted at https://bitcointechtalk.com/

What's New in Bitcoin Core V0.15 - Part 1

2017-09-14T00:00:00+00:00

I’m very excited about the recent Bitcoin Core v0.15 release. This is the first major release that I’ve been involved in from start to finish, and there are some great new features and improvements. Over the next few days, I’ll present a series of articles highlighting five of the most interesting changes to look out for.

I’m going to dive quite deep into the technical details of these changes since I think they’re all interesting and instructive. If you’re not interested in those details and just want a high-level overview of what’s new, then the release notes are a better place to look.

We’ll start by getting under the hood and look at an implementation change. Users shouldn’t see any changes in behaviour from this change, except for their Bitcoin Core client performing much better.

Per-output chainstate db (PR 10195)

Per-output chainstate db is a fantastic win in Initial Block Download time and general performance, as well as being a huge improvement in code simplicity. It also removes a potential DOS vector where an attacker could exhaust a node’s memory with carefully constructed transactions. Users shouldn’t see any functional difference, but it gives such a performance improvement that it’s worth looking at in more detail.

First, a bit of blockchain theory and history. The blockchain is simply an ordered log of the transactions that have been accepted by the network up to certain time. Once the blocks have been validated, that blockchain data is no longer useful to the node. What we’re actually interested in is the UTXO set — the set of all transaction outputs that have not yet been spent. All full nodes must keep a copy of this set so they can verify that transactions are spending coins that actually exist and that the signatures for those transactions are valid. In Bitcoin Core, that UTXO set is stored in a database called the chainstate db. Ideally a node would store this entirely in memory for fast lookups, but the chainstate can be flushed to disk if the node doesn’t have enough RAM to store the entire set.

The original Bitcoin software stored the entire block tree data, transactions and spends in a database called blkindex. Transactions would never get removed from that database, even when all of the outputs of those transactions had been spent. That’s fine up to a certain point, but it’s not scalable. Transactions grow linearly over time, and as the database grows, the time it takes to seek all the outputs for a new block in the database also grows. The result would be that the database would continue expanding and block validation and propagation would slow down more and more over time. If we’d kept using this model, all full nodes would be required to keep the full blockchain (roughly 130GB at the time of writing), and block validation would be really slow.

The first major change to the model came in v0.8, which included ultraprune. Ultraprune split up blkindex into a blocks database (for storing the historical blockchain), and a chainstate database (for storing transactions and outputs). The chainstate database was structured per-transaction. That meant that the main entry in the database was a list, containing all the outputs of a given transaction. As soon as all the outputs from a transaction had been spent, the entire transaction could be pruned from the chainstate database. This split also made block pruning possible, which was implemented in v0.11. The result of those changes together meant that today it’s possible to run a full validating node with only 4–5GB of disk space.

That’s the historical context. So what’s changed in v0.15? The chainstate database has been changed to be structured per-output instead of per-transaction. In practice that means:

It’s much faster to read and write individual outputs from the database, since we don’t need to read and write all of the containing transaction’s unspent outputs, just the output that we’re interested in. That means that validating a block is faster.
Memory usage is much smoother and can’t be blown up when reading/writing large transactions from the database. That means that we can use the chainstate cache more efficiently and flush to disk less often.
The code is much simpler, and we can make future improvements to flushing the chainstate database to disk.

There’s one small drawback to this:

The chainstate database becomes slightly larger. The reason the chainstate was initially implemented per-transaction was that it saved disk space. Instead of storing the transaction id once for each output, it only needed to be stored once across all the outputs in a transaction. Now, we need to store the txid once per output. This is partially mitigated by compression in the database layer which can avoid writing the same txid multiple times. In practice, the chainstate database will be about 15% larger with this change.

So what’s the bottom line for users? There are some performance results in the Pull Request:

Initial Block Download (IBD) and reindex is 30–40% faster
IBD and reindex use 10–20% less memory
IBD flushes to disk far less frequently.

30–40% speed-up in IBD is an enormous win. It means that when you start up a fresh node, you’ll catch up to the chain tip much more quickly.

Thanks to Pieter Wuille, Matt Corallo and Jimmy Song for input and feedback.

originally posted at https://bitcointechtalk.com/

An Introduction to Bitcoin Core Fee Estimation

2017-09-12T00:00:00+00:00

Why we have fees

Space in the Bitcoin blockchain is a limited resource, and given a low enough price, demand for that space will far exceed the supply. If block space is free, people will find all kinds of uses for it — decentralised gambling, uploading an entire copy of the Bitcoin whitepaper, and timestamping individual documents are just a few examples we’ve seen in the past.

To make sure that the limited space in the blocks is allocated to those who value it most and who are prepared to pay the most for it, Bitcoin has a fee market. In effect, the mempool acts as a decentralized clearinghouse — users place bids for block space into the mempool (in the form of transactions with fee attached), and miners will take transactions and place them in the next block based on the fee attached. The larger your fee, the more likely it is that your transaction outbids the competing transactions and that the miner selects your transaction for inclusion in the next block.

Why fee estimation is a hard problem

How does a user know what an appropriate fee is? It turns out that’s a really difficult question to answer, for a few reasons:

Supply is unpredictable. Over long time horizons, supply is predictable. There’s approximately 2MB of space every 10 minutes (or to be more precise, there’s 4M block weight every 10 minutes). But because of the Poisson distribution of block discovery, that supply is lumpy and unpredictable over shorter time periods. One in a hundred blocks is discovered within 7 seconds of the previous block, and one in a hundred takes over 45 minutes to find. That means that there might be a ‘lucky’ run where several blocks are discovered within a few minutes, and all the high fee transactions are drained out of the mempool. On the other hand, there might be no block discovered for half an hour or more, in which case the mempool will slowly fill with higher fee paying transactions.
Demand is also unpredictable. We definitely see cyclicity in transaction flow — weekends are generally quieter than week days, so the mempool is emptier and fees are lower. However, just like the supply side, demand is unpredictable in the short run. For example, even on weekends, demand tends to spike during rapid changes in bitcoin price.
Different users have different requirements. Like any market, each participant has their own reasons for wanting to ‘buy’ block space. I might have a really urgent transaction that needs to be confirmed in the next half hour, or maybe I need to close an expiring smart contract in the next 6 hours, or perhaps I need to timestamp something and I’m happy as long as it gets confirmed some time in the next week. A one-size-fits-all model for fee estimation is not able to capture all of those different use-cases. Estimating the correct fee for a given circumstance is therefore difficult, but it’s a really important aspect of having a usable system. If your fee estimation is too high, then you’re wasting money every time you send a Bitcoin transaction. If it’s too low, then you won’t get your transaction confirmed as quickly as you’d like, and using the system becomes really frustrating.

We can base our fee estimation on various data points. Let’s have a look at them in turn. Note that these fee estimation techniques will be used on a full validating node. We need our own mempool and full blockchain to be able to make estimates based on observed events on the network.

Fee Estimation based on current mempool

A naive fee estimation algorithm would look at your mempool and set your transaction fee at a level that puts it in the highest-paying 2MB of transactions. You might expect that strategy to get your transaction confirmed in the next block. Unfortunately, it’s not quite as simple as that for a number of reasons:

Very recent transactions may not make it into the next block. Miners are already working on the next block when you submit your transaction. Depending on how they refresh the block as they work on it, they may not include your transaction.
Whatever time you send your transaction into the mempool, the expected time you’ll have to wait for the next block is 10 minutes. This is a fundamental (and often misunderstood) property of the Poisson distribution that block discovery follows. If you place your transaction into the mempool 8 minutes after the previous block, the expected time you’ll need to wait isn’t 2 minutes — it’s another 10 minutes. Therefore you’re not just competing against what’s in the mempool now, you’re competing against what’s going to appear in the mempool over the next (probably) 10 minutes.
Looking at a snapshot of the mempool doesn’t take into account the fact that there will be lucky runs of blocks and slow runs of blocks in the future. It might give you some information about how much fee you need for your transaction to be included in the next block or two, but can’t tell you anything about how much fee you need if you want your transaction included in the next 100 or 200 blocks.
There’s no such thing as the mempool. Each node will have different transactions in its mempool depending on factors such as how long it’s been running for, its connectivity to other nodes, its local mempool policies, and so on. What you see in your mempool might not be indicative of what miners are seeing in theirs. For those reasons, just a snapshot of the current mempool isn’t going to give us enough information to properly estimate the required fee.

Fee Estimation based on historic blocks

This appears to be a better way to estimate fees at first glance, since the blockchain is an immutable history of exactly which transactions were included. However, if we only looked at historic blocks, our fee estimation algorithm would be trivially gameable by miners. A miner could stuff his block with private, high fee rate transactions. Fee estimators that only looked at historic blocks would be fooled into increasing their estimates because it would appear that a large fee is required to be confirmed in a block.

This is a very cheap attack for miners, since the cost to them is just the opportunity cost of excluding real, fee-paying transactions. If our fee estimator requires that transactions be seen in the mempool before they’re included in a block, then the cost increases dramatically and the attack becomes infeasible. The miner would have to broadcast his high fee rate transactions and risk having to pay those fees to another miner.

Fee Estimation based on mempool history

Bitcoin Core therefore considers the historic data for transactions in the mempool and in recent blocks. If we have a large enough sample of past transactions, we can make a good model of what fee would have been required to be included within a certain target number of blocks.

How Bitcoin Core estimates fees (before v0.15)

Note 1: fee estimation in Bitcoin Core is rather complex. This article can only give an overview of the concepts behind the fee estimation algorithm and makes a few simplifications. For more details, see Alex Morcos’s high level description of the fee estimation algorithm or for all the gory details take a look at the code itself.

Note 2: This article describes the fee estimation algorithm from Bitcoin Core v0.14. There are a number of changes in v0.15 which I’ll cover in a future article.

Concepts: buckets and targets

We’ll start by defining a couple of concepts that are used by the algorithm: buckets and targets.

Bitcoin transactions fee rates fall within an almost continuous range from 1 satoshi per byte up to many hundreds of satoshis per byte, and even higher for some extreme outliers (See Jochen Hoenicke’s visualisation for an idea of what the current mempool looks like). Keeping track of every fee rate attached to every transaction would require a lot of computation and storage, so the first thing Bitcoin Core does is group those transaction fee rates into buckets, where each bucket corresponds to a range of fee rates. That’s a bit like how Jochen’s visualisation groups the transactions into different colored bands.

Bitcoin Core wants to be able to make estimates over a very large range of fee rates, so the lowest bucket starts at 1s/B and extends to 1.1 s/B (10% higher). The next bucket is from 1.1 s/B to 1.21 s/B (10% higher than 1.1). The next bucket is from 1.21 to 10% higher than that, and so on. These exponentially-spaced intervals are extended all the way up to 10,000 s/B. The last bucket is anything at 9,400s/B and above. At todays exchange rate ($4200), that’s approximately $88 for a median sized transaction (225 bytes).

The second concept is the number of blocks between a transaction entering the mempool and being accepted in a block. We’ll call that the target (since we’ll use this when calculating how much fee we need to attach in order to be included within a target number of blocks). In v0.14, Bitcoin Core keeps track of targets from 1 block up to 25 blocks.

Keeping track

To estimate how much fee a transaction requires in order to be confirmed within a certain number of blocks, Bitcoin Core stores a historic record of the transactions it’s seen in the mempool and blocks. It keeps:

A. the number of transactions that entered the mempool in each fee rate bucket; and
B. for each bucket-target pair, the number of transactions that successfully made it into the blockchain within the target number of blocks.

For any target-bucket pair, Bitcoin Core is then able to find the probability that a transaction with that fee rate would have been included within that target number of blocks by dividing B by A.

Responding to changes in prevailing fee rate

The Bitcoin network is a dynamic system and prevailing fee rates change over time (if this wasn’t true, we wouldn’t need fee estimation at all!). We want our fee estimator to be reactive to changes in prevailing fee rate and to give more weight to recent events than older events, since they’re more likely to be indicative of future fee rates.

For that reason, Bitcoin Core uses an exponentially weighted moving average, or put more simply, we pay more attention to recent blocks than older blocks. Every time Bitcoin Core receives a block it multiplies its counters for old blocks by a decay value of 0.998 (equating to a half-life of 346 blocks). That means that as a block recedes into the past, it counts for less and less in our fee estimation algorithm. A block from 2.4 days ago carries approximately half the weight of the most recent block. A block from 4.8 days ago carries about one quarter of the weight, a block from 7.2 days ago carries about one eighth the weight, and so on.

Note that this moving average is independent from the 24 block target range mentioned earlier. We’re interested in up to 24 blocks for transaction inclusion, but we’ll use historical data going back over many blocks.

estimateSmartFee(): a usable interface

To make a useful interface, we need to know what question the user is asking. Bitcoin users almost certainly aren’t asking:

If I attach a fee rate in bucket X to my transaction, what’s the exact probability of it confirming in Y blocks?

Instead, they need to know:

I want my transaction confirmed within Y blocks. How much fee should I attach to make that likely to happen?

Enter estimateSmartFee(), which is designed to do just that. estimateSmartFee() works as follows:

The user provides a target number of blocks that they want the transaction to confirm within.
For that target number, look at the largest bucket (anything above 9,400 s/B) and verify that for that fee, the probability of being confirmed in that number of transactions is at least 95%. (if this fails, then we’re really in trouble — it’s saying that even paying the astronomical fee of 9,400 s/B is probably not going to be enough. Remember, that’s about $88 for a normal-sized transaction at today’s prices).
Look at the next highest fee bucket (roughly 8,600–9,400 s/B) and verify that for that fee, the probability of being confirmed in that number of blocks is at least 95%
Keep working down the buckets until we find one where the probability of being confirmed is less than 95%
Take the previous bucket (the lowest value bucket that had a probability of at least 95% of being included) and return the median fee of all transactions in that bucket.

When calculating the probabilities for each target-bucket, the algorithm is smart enough to make sure that there’s a large enough sample size of transactions. That means that if there’s a small number high-fee transactions which weren’t confirmed for some reason, the fee estimation algorithm won’t get thrown off.

We use a high threshold of 95% because we want the fee estimator to be able to tell us that a transaction at a certain fee rate would almost certainly have been included within target blocks. The fee estimator does not have perfect knowledge of why transactions were included in a block (for example, there may have been out-of-band payments to the miner to get a certain class of transactions confirmed quicker than other transactions with the same or lower fee rate), so we set the bar high but not at 100% to give us a high confidence in our estimate.

estimateSmartFee() is used by the estimatesmartfee, sendtoaddress and sendmany RPCs, and when sending a transaction in the GUI.

Conclusion

Fee estimation is a difficult problem. A fee estimator is essentially trying to answer a question about the future based on an incomplete picture of the past. It’s not clear how well we can measure the success of different fee estimation algorithms, in part because different users have very different requirements of the system. Furthermore, the fee market in Bitcoin is constantly changing, and a fee estimation algorithm that is successful in today’s Bitcoin environment may not be successful in the event of large systemic changes (eg the emergence of another coin that mines using SHA-256, the roll out of segregated witness transactions, secular changes in overall demand and mempool congestion, and so on). Bitcoin Core provides a fee estimation algorithm that is appropriate for a large number of users in a wide variety of fee market circumstances. Other wallets may have very different approaches to fee estimation which could be more successful for certain users in certain circumstances.

Bitcoin Core’s fee estimation doesn’t try to be too smart. It simply records and reports meaningful statistics about past events, and uses that data to give the user a reasonable estimate of how much fee they need to attach in order to have their transaction included within N blocks. It doesn’t try to be forward-looking, and it’s easy to describe exactly why a given estimate has been produced based on past statistics. It can react to changes in the prevailing fee market and give reasonable estimates in a wide range of conditions.

A more sophisticated algorithm may attempt to be more forward-looking. However, the more complex the algorithm, the more difficult it becomes to describe its operation and results, and the more difficult it is to argue that the algorithm is safe against manipulation.

There are cases where the Bitcoin Core fee estimator does not perform well, notably when there are very rapid changes in the prevailing fee market. The fee estimations in Bitcoin Core v0.15 builds on the same framework as the v0.14 algorithm, but has many improvements to make it more robust in the case of changing circumstances and outlier events. I’ll describe what those changes are in a future blog post.

originally posted at https://bitcointechtalk.com/

Contributing to Bitcoin Core, a Personal Account

2017-08-11T00:00:00+00:00

In January of this year, I moved to New York to take a job contributing full time to open source Bitcoin projects. These are some of my experiences in those first few months.

First of all, I recognize that I’m incredibly fortunate. Not only do I have tremendous freedom to work on what I think is important or interesting, I get to sit alongside and work directly with brilliantly talented and motivated people. All the work I do is open source, and I’m contributing to a fascinating technology which will be hugely important over the coming years. There’s almost nothing more I could ask for, and I often need to remind myself that this is real, and that somehow this is my living.

But in more practical terms, what have I actually learned since I’ve been here?

Start small

I’m the first to admit that I’m not the best or most experienced programmer. I started my professional career as a software developer but very quickly moved into a support role, managing customer projects and support teams. I made that choice because that was what would let me meet interesting people and travel to exotic places. That seemed much more intriguing than working on code, and as a result I’ve spent the last few years working with customers and teams, rather than writing and debugging code.

Bitcoin is a very complex project, one which runs on multiple platforms in multiple environments, and which at its heart is a consensus engine — where each participant must walk forward in lockstep, or else the whole system will partition and fall apart. In fact, in bitcoin’s early days this came close to happening on more than one occasion. Needless to say, any changes to those parts of the code need to be made extremely carefully, and there are really only a few people in the world who understand the system well enough to be able to safely work in those areas. Other people have tried to make changes to consensus, and more often than not they’ve failed spectacularly.

So as a noob, what to do? Well, although I’m no C++ wizard, I can hold my own in Python, which is what the bitcoin functional test framework is written in. I was able to clean up some of that code, fix bugs in the tests, and uncover and fix bugs in the product code. Through working on the tests, I was slowly able to get exposure to other parts of the codebase — the RPC server and client, the wallet, the build system, and so on, which I’m now able to make modest contributions to.

Here’s my first merged code commit to Bitcoin Core: adding extra testing for the bitcoin-tx utility. A modest contribution, but we all need to start somewhere!

Hold yourself accountable

As my experience of the code base has increased, so too has the surface over which I can work. Every day, the list of things that I think I could or should work on, and the number of projects I think would be a worthwhile use of my time, increases rather than decreases. With such freedom, it’s very easy to spread oneself too thin, or to constantly pick up and then drop projects. I’ve done this myself and found it to be ultimately unsatisfying.

Furthermore, an open-source project isn’t like a normal job. There aren’t management lines holding you to account and enforcing deadlines from the top down. Any contribution you make is down to you, and if you start something and then drop it later, there’s really nothing anyone else can do about it. Everything you contribute is voluntary.

I’ve found it useful to pick a more challenging longer-term project and commit to seeing it through. In the first few months, for me that project was refactoring the functional test framework into using a TestNode class, which I very much hope will be finished in the next couple of weeks. It’s been a long and at times frustrating journey to get this far, but I think ultimately the test framework will be in a much better shape when I’m finished. My next goal is to push forward wallet-server separation starting here, and lay the groundwork for full hardware wallet integration in v0.16 or v0.17. Having a goal, sharing it with other people, and holding yourself accountable to it is a really useful exercise.

Sharpen the tools

Always, always, always be sharpening your tools and adding new skills and techniques. When I landed this gig in October last year, I knew that my basic skills weren’t up to scratch. So I read the pro git book cover to cover, and the vim bible, and a tmux primer, and a linux overview. Currently I’m working through Understanding Cryptography. Next up is a book on Elliptic Curves, and then a treatment of the automake system. I’ll also re-read some C++ text books to remind myself of everything I’ve forgotten.

(aside: if you haven’t read the Pro Git and Practical Vim books, go out and do that now. The time spent mastering git concepts like interactive rebase, rerere and filter-branch, and making your vim life more efficient will repay the invested time with interest)

I’ve also instigated a whitepaper Wednesday at work. Each week one of us will read and present an interesting whitepaper or technical topic to the group.

The pace of technical innovation in Bitcoin is breakneck, so to be a useful contributor and to be able to keep up with the latest developments we all need to be constantly refreshing our skills and learning new concepts.

Offer help and ask for help

I’m not the best C++ programmer, but thanks to my time with Pro Git, I’m pretty handy with branches and commits. I’ve been able to help take load off long-standing contributors by rebasing and tidying up PRs (eg here and here). If that frees up their time to work on more important stuff, then it’s a useful contribution. At the other end of the spectrum, there are first-time contributors who often trip up over basic concepts like squashing commits and rebasing PRs. Spending a bit of time helping those people is definitely a good use of time if it gets the PR unblocked, and even more so if it encourages the contributor to come back and contribute to the project again.

Conversely, I’ve been very happily surprised by the amount of help long-standing bitcoin contributors have been prepared to give me and the time they’ve spent teaching me things. I’m always blown away by how much time sipa is prepared to spend teaching and explaining (take a look at his Stack Exchange profile if you don’t believe me). Cory Fields has also been extremely generous with his time, explaining the build system to me even when I’ve had the most basic questions. Matt Corallo, Suhas Daftuar and Alex Morcos deserve special mention for getting me started on my Bitcoin developer journey in September last year and bringing me up to speed on the code and how to be a useful contributor.

Bitcoin has an unfair reputation for being an unwelcoming environment for new contributors. The one thing that all of the long-standing contributors share is that they care deeply about the project. If you genuinely want to help, and are respectful of their time, then most will be more than happy to share expertise with you.

Be a contributor, not a developer

One part of being a useful contributor to an open source project is writing good code, but there are many more: understanding what other people want from the project and working in the same direction as them rather than against them; enticing people to review and merge your PRs; making your communications in PRs and reviews clear, convincing and respectful; structuring your PRs to make the reviewers job easier; and cultivating good relationships with other contributors are just some. Work on those skills. It’s no good writing awesome code if it’s doing something that people don’t want in the project, or if you can’t get anyone to review it. This is especially important if you’re a newcomer and haven’t built up a reputation in the project — try to make it easy for people to review and ACK your PRs and you’re more likely to get a good response.

Interested?

My list isn’t complete and is no way prescriptive. Open source is a broad church, and everyone contributes in their own way.

If you want a more practical primer on how to get started on contributing to Bitcoin Core, Jimmy Song has written a superb introduction.

Please reach out to me if have any thoughts or suggestions of your own, or if you want help getting started. I’m always happy to help people get their first commit merged!

originally posted at https://bitcointechtalk.com/

BC2 2017 - Tokyo

2017-07-31T00:00:00+00:00

I gave a talk at Digital Garage's BC2 workshop on the topic of "Contributing to Bitcoin Core". The slides are here. The talk wasn't filmed, but it formed the basis for a later blog post.

What Did Bitcoin Core Contributors Ever Do for Us?

2017-07-21T00:00:00+00:00

When you fire core, I presume you're also going to ...
— John Newbery (@jfnewbery) July 21, 2017

This afternoon, I tweeted a response to the strange suggestion that we fire core. I’ve ignored the fact that Bitcoin Core isn’t a person or an organization (in the traditional sense), and that everyone is perfectly free to run their own re-implementation of Bitcoin, but instead focused on a few of the things you’d need to do if you were serious about rejecting the work of the Bitcoin Core contributors.

I’ve included a mix of enhancements that have been around for one or more releases, stuff that’s going in to 0.15 right now, and longer-term development or research work. People who don’t actively follow the bitcoin core github repository probably aren’t aware of the wide range of work that the contributors are doing. Hopefully this gives some insight into that world.

Full disclosure: I contribute to Bitcoin Core.

So here goes, what have Bitcoin Core developers ever done for us?

... switch back from Pieter's libsecp256k1 to OpenSSL, so your transaction signing/validation is 5x slower and less secure ...
— John Newbery (@jfnewbery) July 21, 2017

libsecp256k1 is a heavily optimized library for doing elliptic curve math over Bitcoin’s secp256k1 curve. Elliptic curve math is used for creating signatures to spend transactions, and for validating signatures in transactions that you receive from the network. In addition to being several times faster than OpenSSL, libsecp256k1 has much better protection against timing, derandomization and side-channel attacks. Bottom line: transaction signing/validation is faster and more secure.

libsecp256k1 was mostly written by Pieter Wuille, Andrew Poelstra, Peter Dettman and Greg Maxwell, and started being used by Bitcoin Core in v0.12. Pieter, Andrew, Peter and Greg continue to maintain and make improvements to libsecp256k1.

... remove Suhas's pruning code, so you need to store tens of GBs of dead data if you want to run a full node ...
— John Newbery (@jfnewbery) July 21, 2017

Pruning allows a full node to discard old blockchain data, while still fully validating all the consensus rules. It means that users can run a Bitcoin Core node on diskspace-constrained hardware and enjoy the exact same security of running a full node. The blockchain is now over 125GB, but a pruning node can be fully synced to the network with just 2–3GB of disk storage.

Automatic pruning functionality was added in Bitcoin Core V0.11. The code was mostly written by Suhas Daftuar, Alex Morcos, Adam Weiss and Brad Andrews. Pruning was further improved in V0.14 to allow pruning to be run manually by the user. Principal contributors were Brad Andrews and Russ Yanofsky.

... revert Luke's multiwallet so you can only access a single wallet on your node ...
— John Newbery (@jfnewbery) July 21, 2017

Multiwallet is a long-requested and wanted feature that has finally been merged in v0.15 🎉. This allows users to have completely segregated wallets, for use-cases such as separating business and personal accounts or having wallets for different purposes running concurrently. We don’t yet have separate authentication for different users, but future versions may allow multiple users to safely access different wallets on the same Bitcoin node.

Luke Dashjr contributed multiwallet in V0.15. The RPC interface was provided by Jonas Schnelli.

... undo Cory's net refactor so your transactions and blocks take longer to propagate across the network ...
— John Newbery (@jfnewbery) July 21, 2017

Having a fast, efficient networking layer is essential for quick block/transaction propagation and the overall health of the network. The faster that blocks are able to propagate through the network, the lower the stale block rate, and the more secure the network is.

Cory Fields has been doing continued work to refactor the networking code for several releases, with a major and significant improvement delivered in V0.14. V0.15 and future releases will continue to see improvements in cleaning up and isolating the network code from the server code.

... throw out Greg's signature aggregation research so your blocks will contain fewer transactions and take 2-4x longer to validate ...
— John Newbery (@jfnewbery) July 21, 2017

One of the most exciting benefits of segregated witness is that it gives us the ability to upgrade the scripting language. There are several exciting script upgrades under investigation, but the one that will probably get most attention in upcoming releases is Schnorr signature support. Schnorr signatures are an alternative signature scheme to the currently used ECDSA which allow multiple signatures to be added together or ‘aggregated’. This is a win for scalability (since if multiple signature are added together, the aggregated signatures takes up only as much space as one of input signatures), validation cost (since only one signature needs to be validated instead of many) and privacy (since an aggregated signature doesn’t reveal whether the input was a single signature or many signatures).

Greg Maxwell, Pieter Wuille and Andrew Poesltra have been investigating Schnorr signature aggregation, particularly working to make sure that the aggregation scheme is safe from the signature cancellation problem.

... get rid of Russ's process separation groundwork, so your signing code will be in the same memory space as your network code ...
— John Newbery (@jfnewbery) July 21, 2017

Currently, Bitcoin Core exists as a single process, with shared memory access between the network code, consensus code, wallet code and user interface code. Ideally we’d like to separate the wallet into a separate process, so if the networking function was hacked or compromised, your wallet function and private keys would be safer.

Russ Yanofsky has been doing the groundwork for process separation during V0.15. Look out for further progress in this area in V0.16 and V0.17.

... erase Alex's fee estimation improvements so you pay too much for your transactions...
— John Newbery (@jfnewbery) July 21, 2017

Every transaction submitted to the Bitcoin network attaches a user-chosen fee, which goes to the miner who confirms that transaction in a block. Set the fee too low and your transaction won’t get confirmed in a block. Set it too high and you’ve donated money to the miner unnecessarily. Between those two extremes is a continuum of where to set your fee — a high fee will probably get you confirmed in the next block, a slightly lower fee might see your transaction confirmed in the next 3 or 4 blocks, and even lower than that and your transaction could take a few hours to get confirmed. Choosing the correct fee for your transaction is a hard problem, requiring knowledge of the current state of the network and some smarts to predict what will happen depending on where you set the fee.

Alex Morcos has spent a lot of time thinking about and analyzing fee strategies and significantly improved the fee logic in V0.15.

... back out Matt's multithreading and scheduling refactors, so your node is mostly limited to running a single process ...
— John Newbery (@jfnewbery) July 21, 2017

Bitcoin Core runs multiple threads so different tasks can be run in parallel on a multi-core computer. However, a lot of the functions grab a ‘global lock’ before doing their work, and other threads need to wait for that global lock to be released before they can continue with their work. Result: Bitcoin Core is not as able to do as much work in parallel as we’d like. If we can reduce the places where that global lock is grabbed and held, then Bitcoin Core would be able to things like run wallet tasks, validate blocks and serve blocks and transactions to peers simultaneously .

This is very delicate work and requires a deep understanding of Bitcoin Core’s multi-threading model. Get it wrong and you could easily cause crashes or memory corruption. Matt Corallo has done a lot of the plumbing work for this in V0.15. We should see some major payoff for that work in V0.16.

... delete Jonas's and Nicolas's hardware wallet integration work, so your private keys will always be in your hot wallet...
— John Newbery (@jfnewbery) July 21, 2017

Several companies now offer hardware wallets, which allow you to keep your private keys and signing code on a dedicated security device. That’s a much, much more secure model than having your private keys on a network-connected computer, which could potentially get hacked. Sadly there’s no standardized interface for hardware wallets, so each vendor provides their own software wallet to use with their hardware wallet. It’d be great if Bitcoin Core could support hardware wallets so users could benefit from running fully hardware-separated signing code behind the most secure Bitcoin full node.

HWW support is probably at least a couple of releases away, but Jonas Schnelli and Nicolas Dorier have already been doing some early work to make sure that Bitcoin Core is ready for HWW support. Hopefully we’ll see some more progress on this in V0.16 or V0.17.

... and find someone who's prepared to do Wladimir's thankless and never-ending task of merging and preparing releases?
— John Newbery (@jfnewbery) July 21, 2017

Of course, none of these code changes would be of any use at all if we didn’t have repository maintainers to do the work of signing and merging all the commits, making sure translations are ready, preparing release notes, and doing all the other things that turn a bunch of code changes into a software product that normal people can run. Wladimir van der Laan is lead maintainer and has been tirelessly doing that work for many releases. Everyone in the Bitcoin community owes him a huge debt of gratitude.

I’ve tried to give shout-outs to the main contributors behind each of these features. Open-source software is a collaborative activity and there are far more who have contributed code, review, testing time, documentation and much more to these and other initiatives. If I’ve made any egregious omissions, please accept my apologies and message me on twitter so I can set the record straight!

originally posted at https://medium.com/@jfnewbery/

John Newbery's blog

The Economics of Public EV Charging in the UK

Introduction

Assumptions

Asset cost

Energy supply costs

Network Costs

Physical Constraints

Charger Utilisation

Modelling

Results

Conclusion

UK Residential Solar PV Payback Analysis

Introduction

Assumptions

Results

Interval data

Analysis

Are Increasing Wholesale Prices Impacting Solar Installations?

Conclusion

Solar Export Tariffs in GB

Introduction

The Feed in Tariff and Smart Export Guarantee

The SEG landscape

The model

Results

Conclusion

Notes

Comparing Contingency Frequency Control Markets in Australia and the UK

Introduction

What is Contingency Frequency Response?

How does FCAS compare to DC?

Notes

LABITCONF 2021 - Future Protocol Changes

The Brink Podcast

CopperCast - Open Source Bitcoin Funding

The B Word - Open Source Money

MIT Bitcoin Expo 2021 - Developer Panel

Coinscrum - Funding Bitcoin Core Development

Bitcoin Magazine - Developing Bitcoin

MIT DCI Security Roundtable - A History of Bitcoin Security

What Bitcoin Did - Building a Bitcoin Developer Community

On The Brink Podcast - Funding Bitcoin Development

The Stephan Livera Podcast - Announcing Brink

she256 Onboarding to Bitcoin - Contracts

MIT Bitcoin Expo 2020 - Developer Panel

The Chaincode Podcast

LABITCONF 2019 - Taproot and Schnorr Signatures

What Bitcoin Did - Building a Bitcoin Developer Community

Chaincode Summer Residency 2019

Security Models

Wallet Development

World Crypto Network - Onboarding Bitcoin Developers

Magical Crypto Conference 2019 - Protocol Development Panel

Scaling Bitcoin Operations

Lightning Applications Residency

Noded 0.29.0 - The Inflation Bug

Bitcoin Edge Dev++ Tokyo - Digital Signatures and Script

Digital signatures - finite fields, elliptic curves, Schnorr signatures and ECDSA

Bitcoin script

Crypto News Network - Perspective

Chaincode Lightning Apps Residency

London Bitcoin Devs - Bitcoin Core 0.17

Noded 0.19.0 - Launching Optech

Tales From The Crypt #36 - Optech, Taproot and Lightning

Distributed 2018 - Optech Launch Announcement

MIT Bitcoin Expo 2018 - Scalability Panel

Noded 0.6.0 - Contributing to Bitcoin Core

Tales From The Crypt #5 - Bitcoin and the Bitcoin Core process

Recurse Center - Bitcoin for Hackers

Bitcoin Edge Dev++ Stanford - Blocks, Blockchain, P2P, Mempool and HD Wallets

Blocks and the Blockchain

The Peer-to-peer network

The mempool

HD Wallets

Building a Stronger Bitcoin Developer Community

Announcing the second Chaincode Hacker Residency

Wednesday night meetups

Offchain - Bitcoin Core 0.15

What's New in Bitcoin Core V0.15 - Part 5