ChatGPT is exceptional at finding bugs without requiring any meaningful additional context beyond the source code. This makes it an excellent first place to start your testing journey.
One of the first things you learn as a smart contract coder is that contracts are extremely unforgiving when it comes to vulnerabilities. Contracts are immutable by default. At the same time, they’re capable of handling extraordinary amounts of money. This makes security (and smart contract testing) arguably the biggest concern for any team of web3 developers.
But auditing contracts—the typical last of line defense against bugs—has historically been time-consuming and expensive. Fortunately, there has been an explosion in powerful tools that make security and smart contract testing cheaper, simpler, and faster.
In this article, we will explore testing smart contracts using two of these tools: ChatGPT by OpenAI and Diligence Fuzzing by ConsenSys.
Testing an Ethereum Smart Contract with ChatGPT and Diligence Fuzzing
Let’s walk through a smart contract for an Ethereum project that features vulnerable ERC-20 code and see what the two tools can do for us.
Diligence Fuzzing is a tool from ConsenSys that implements fuzzing for web3. Fuzzing is a dynamic testing technique where random (or semi-random) inputs called “fuzz” are generated and injected into code. Fuzzing can help reveal bugs and vulnerabilities that weren’t caught by traditional testing methods.
And of course, ChatGPT is a large language model (LLM) AI. Unless you’ve been living under a rock, you’re probably already aware of ChatGPT and some of its capabilities. It’s one of the most exciting technologies to have come out in the last decade. Not only can it help you with testing (as we’ll see in this article), but also with smart contract development, writing tests, and more.
Let’s get started!
Step 1: Install Python, pip, npm, and Node.js
In order to install the tools required to run Diligence Fuzzing, you first need the following on your local machine: the latest versions of Python and pip and the latest versions of Node.js and npm.
Check that you have all four running by using these commands:
$ node -v
$ npm -v
$ python –-version
$ pip --versionStep 2: Install the Vulnerable Contract Repository
In addition to creating Diligence Fuzzing and the Scribble annotation language, ConsenSys has made public a set of repositories containing vulnerable contracts. To keep it simple, you’ll be using one of these.
You will use a repo that contains a vulnerable implementation of the ERC-20 standard (the standard for cryptocurrencies and fungible tokens).
Clone this repository using the following command:
$ git clone https://github.com/ConsenSys/scribble-exercise-1.git contracts-testThis repository has a folder called contracts. Inside this folder, you will find a file called vulnerableERC20.sol. As mentioned earlier, this file contains a faulty implementation of the ERC-20 standard.
This is the contract you’re going to test using ChatGPT and later, Diligence Fuzzing.
Step 3: Create an OpenAI Account
ChatGPT happens to be extremely proficient at a wide variety of tasks—including web3 development. (See my previous article on creating a smart contract with ChatGPT.)
You can ask ChatGPT to unpack a complex blockchain concept, write smart contracts that implement something you had in mind (for instance, a decentralized lottery), or even explain a piece of code to you and find potential vulnerabilities.
As you may have already guessed, it’s that last capability we’re interested in for this article.
In order to access ChatGPT, you need to create an OpenAI account. Once you’ve created an account, access ChatGPT’s UI.
Talking to ChatGPT is as simple as talking to a person on a messaging app. If this is your first time using the tool, I strongly recommend you test out some example prompts (for example, “Explain quantum computing in simple terms”) to get comfortable.
Step 4: Find Vulnerabilities in the Contract
The key to getting right answers from ChatGPT is being extremely clear and detailed about your request. Paste the following prompt into the UI:
Following is a smart contract I've written in Solidity. As a first step in testing its safety, I want you to examine the code, and let me know of all the security vulnerabilities that you find.
```
pragma solidity ^0.6.0;
contract VulnerableToken {
uint256 private _totalSupply;
mapping (address => uint256) private _balances;
mapping (address => mapping (address => uint256)) private _allowances;
constructor() public {
_totalSupply = 1000000;
_balances[msg.sender] = 1000000;
}
function totalSupply() external view returns (uint256) {
return _totalSupply;
}
function balanceOf(address _owner) external view returns (uint256) {
return _balances[_owner];
}
function allowance(address _owner, address _spender) external view returns (uint256) {
return _allowances[_owner][_spender];
}
function transfer(address _to, uint256 _value) external returns (bool) {
address from = msg.sender;
require(_value <= _balances[from]);
uint256 newBalanceFrom = _balances[from] - _value;
uint256 newBalanceTo = _balances[_to] + _value;
_balances[from] = newBalanceFrom;
_balances[_to] = newBalanceTo;
emit Transfer(msg.sender, _to, _value);
return true;
}
function approve(address _spender, uint256 _value) external returns (bool) {
address owner = msg.sender;
_allowances[owner][_spender] = _value;
emit Approval(owner, _spender, _value);
return true;
}
function transferFrom(address _from, address _to, uint256 _value) external returns (bool) {
uint256 allowed = _allowances[_from][msg.sender];
require(_value <= allowed);
require(_value <= _balances[_from]);
_balances[_from] -= _value;
_balances[_to] += _value;
_allowances[_from][msg.sender] -= _value;
emit Transfer(_from, _to, _value);
return true;
}
event Transfer(address indexed _from, address indexed _to, uint256 _value);
event Approval(address indexed _owner, address indexed _spender, uint256 _value);
}
```ChatGPT should give you a response that looks something like this:
Upon reviewing the code, I have identified several security vulnerabilities and
concerns in the provided smart contract:
1. Lack of Access Control: The contract does not implement any access control
mechanism, allowing anyone to call the `transfer`, `approve`, and `transferFrom`
functions. This means that anyone can transfer tokens on behalf of another
address and modify allowances.
2. Integer Underflow and Overflow: The code uses basic arithmetic operations
without any checks for potential underflow or overflow. For example, in the
`transfer` and `transferFrom` functions, the subtraction and addition of token
balances can lead to unintended behavior if the resulting balance exceeds the
maximum value that can be stored in a `uint256`. You should include checks to
prevent these issues.
3. Lack of Require Statements for Input Validation: The contract lacks require
statements to validate inputs in various functions. For instance, in the
`transfer` and `transferFrom` functions, there is no validation to ensure
that the recipient address (`_to`) is not the zero address. Additionally,
the `approve` function does not validate that the spender address (`_spender`)
is not the zero address.
4. Potential Reentrancy Attacks: The contract is susceptible to reentrancy
attacks, where a malicious contract could repeatedly call the `transferFrom`
function to drain the token balance of another address. It is recommended to
use the "Checks-Effects-Interactions" pattern or implement a reentrancy guard
to mitigate this vulnerability.
5. Lack of Events for Critical State Changes: While the contract emits
`Transfer` and `Approval` events, it is missing events for critical state
changes, such as when the total supply is modified or token balances are
updated. Emitting events for these events helps provide transparency and
enables easier tracking of contract activities.
6. Fixed Token Supply: The contract initializes with a fixed token supply
of 1,000,000 tokens, and there are no provisions for minting or burning
tokens. If you intend to have a fixed supply, this may not be a vulnerability.
However, if you plan to have a dynamic token supply, you'll need to
implement minting and burning functions while considering proper access
control.
To enhance the security of your smart contract, consider addressing these
vulnerabilities and implementing additional security measures based on your
specific requirements.
ChatGPT gives you quite a few tips! While they’re mostly generic, they can still help you uncover some key issues.
Beware, though—in a few cases, such as the tip on Access Control, it’s plain wrong. And believe it or not, ChatGPT actually ends up missing the most glaring error in our contract— something that has to do with transferring tokens to ourselves. This is an extremely nuanced error that ChatGPT would only be able to pick up given additional prompting. (Note: with enough prompting, I was able to get ChatGPT to find the error.)
The key lesson here is that ChatGPT can assist in debugging, but only as a starting point. At least for the time being, it’s best at identifying errors that are syntactical, obvious, or common. This might be most helpful for beginning or intermediate developers—or maybe for tired and overworked senior ones! For more nuanced cases, you’ll need to turn to more advanced tools.
Step 5: Sign Up for Diligence Fuzzing
The exercise above shows that, at least at the time of writing (June 2023), ChatGPT can do a lot—but is not a complete substitute for a well-versed web3 developer, especially one who can leverage state-of-the-art testing tools.
In this step, you’re going to learn to use one such tool: Diligence Fuzzing.
As mentioned above, fuzzing is a well-established testing technique that involves sending random, often malformed data to a particular program in an attempt to trigger unexpected behaviors or crashes. This method has proved exceptionally good at detecting security issues that traditional testers often miss.
Diligence Fuzzing implements traditional fuzzing in a web3 environment. The tool has been created to detect critical vulnerabilities and unthought-of issues using simple annotations in Scribble.
In order to use this tool, you need to first sign up for a free account with Diligence Fuzzing. The Free Tier subscription is more than enough for this tutorial, but if you find yourself doing a lot of critical testing, consider moving to a paid tier.
Once you’ve subscribed, you should see a dashboard that looks something like this.
Step 6: Create a Diligence Fuzzing API Key
In order to access Diligence Fuzzing’s capabilities, you need to create an API key here.
You can name your key anything you want. Keep this key secret and safe. You’ll need it later.
Step 7: Install Required Packages and Libraries
First, you’re going to install Truffle, Scribble, and the Fuzzing CLI on your local machine. Run the following commands:
$ cd contracts-test
$ npm install
$ npm install -g truffle eth-scribble ganache
$ pip3 install diligence-fuzzingStep 8: Create a Fuzzing Campaign
Navigate to the contracts-test folder you cloned earlier. Open this repository in your favorite code editor (for example, VS Code) and open the file called .fuzz.yml. You should see a line that says key which has an empty string associated with it. Replace the latter with your API key.
For Diligence Fuzzing to work correctly, you need to annotate our contract with rules you think should be followed. As mentioned earlier, the annotation language here is Scribble.
For this tutorial, you don’t need to know how to write Scribble yourself (though you should take a few moments to get familiar with it for future projects). The repository you downloaded comes (by default) with an annotated contract in the contracts/solution folder. You’ll be using this as a base to annotate the vulnerable contract.
Replace the contents of vulnerableERC20.sol with the following:
pragma solidity ^0.6.0;
contract VulnerableToken {
uint256 private _totalSupply;
mapping (address => uint256) private _balances;
mapping (address => mapping (address => uint256)) private _allowances;
constructor() public {
_totalSupply = 1000000;
_balances[msg.sender] = 1000000;
}
function totalSupply() external view returns (uint256) {
return _totalSupply;
}
function balanceOf(address _owner) external view returns (uint256) {
return _balances[_owner];
}
function allowance(address _owner, address _spender) external view returns (uint256) {
return _allowances[_owner][_spender];
}
/// #if_succeeds msg.sender != _to ==> _balances[_to] == old(_balances[_to]) + _value;
/// #if_succeeds msg.sender != _to ==> _balances[msg.sender] == old(_balances[msg.sender]) - _value;
/// #if_succeeds msg.sender == _to ==> _balances[msg.sender] == old(_balances[_to]);
/// #if_succeeds old(_balances[msg.sender]) >= _value;
function transfer(address _to, uint256 _value) external returns (bool) {
address from = msg.sender;
require(_value <= _balances[from]);
// _balances[from] -= _value;
// _balances[_to] += _value;
uint256 newBalanceFrom = _balances[from] - _value;
uint256 newBalanceTo = _balances[_to] + _value;
_balances[from] = newBalanceFrom;
_balances[_to] = newBalanceTo;
emit Transfer(msg.sender, _to, _value);
return true;
}
function approve(address _spender, uint256 _value) external returns (bool) {
address owner = msg.sender;
_allowances[owner][_spender] = _value;
emit Approval(owner, _spender, _value);
return true;
}
function transferFrom(address _from, address _to, uint256 _value) external returns (bool) {
uint256 allowed = _allowances[_from][msg.sender];
require(_value <= allowed);
require(_value <= _balances[_from]);
_balances[_from] -= _value;
_balances[_to] += _value;
_allowances[_from][msg.sender] -= _value;
emit Transfer(_from, _to, _value);
return true;
}
event Transfer(address indexed _from, address indexed _to, uint256 _value);
event Approval(address indexed _owner, address indexed _spender, uint256 _value);
}
The comments in lines 25-28 represent annotations written in Scribble. Even if you’ve never been exposed to this language before, the contents should be pretty self-explanatory.
You’re all set. Start fuzzing by running the following command:
make fuzz
This should prompt your terminal to run a series of commands.
Notice that the output also contains a URL that provides more information about your campaign.
This campaign takes approximately five minutes to run. But once it’s done, you get a detailed report about its findings, including vulnerabilities.
Notice that under the Property section, the report states that one user assertion has failed. Upon closer inspection, you see that it’s this one:
/// #if_succeeds msg.sender == _to ==> _balances[msg.sender] == old(_balances[_to]);You can look at the transaction details to find out more.
In essence, it’s telling you there’s a vulnerability when the sender and the receiver are the same!
Diligence Fuzzing does a great job at a specific task—it automatically creates a wide variety of inputs to generate unit tests and system tests that find bugs and vulnerabilities in your smart contract.
Create a ChatGPT and Diligence Plugins
You could take this even further by combining the power of the two testing tools using a ChatGPT plugin. This plugin could:
-
Execute the code to initialize a Diligence Fuzzing campaign against the contract
-
Parse the results and use them to do something interesting: directly rewrite the code, come up with suggestions for improvements, automatically rewrite and retest the code until there are no found defects, and more
You can use this starting point to create the plugin—or in the spirit of this article, even ask ChatGPT to write the plugin for/with you!
Conclusion
In this tutorial, you explored two very powerful tools for smart contract testing and security audits.
ChatGPT is exceptional at finding bugs without requiring any meaningful additional context beyond the source code. This makes it an excellent first place to start your testing journey.
However, as you’ve seen, ChatGPT tends to go off track and often produces wrong answers. As a substitute for a complete audit, it has a long way to go. This is where a tool like Diligence Fuzzing can shine. It may require a little more manual intervention, but it’s much more comprehensive in terms of coverage.
Have a really great day!
