IPFS Introduction

Introduction#

InterPlanetary File System (IPFS) is a network transmission protocol designed to implement distributed storage, sharing, and persistence of files. It is a content-addressable peer-to-peer hypermedia distribution protocol. Nodes in the IPFS network form a distributed file system.

It has the following features:

Support for resilient networks: In traditional Internet, files are stored on centralized servers. If your file server goes down, the file becomes inaccessible. But in IPFS, you can still retrieve files from other nodes.
More difficult content censorship: Because files on IPFS can come from many different sources, it is harder to block them.
Faster file retrieval speed: Since you can retrieve files from nearby nodes instead of distant ones, the access speed is greatly improved (similar to CDN).

Today's World Wide Web is structured around ownership and access rights, which means that in order to obtain a file from its owner, you must first obtain access rights. However, IPFS is based on the concept of ownership and participation, where people own each other's files and participate in providing files.

IPFS can only function better when people actively participate. If you use IPFS to share files on your computer, but then you shut it down, others will no longer be able to retrieve files from you. However, if copies of the files are stored on other computers running IPFS, these files can still be shared.

How It Works#

IPFS follows three basic principles:

Unique identification through content addressing

In general, the path of a file is based on its location, whether it is a local file /Users/..../code.js or an accessible file on the network http://www.google.com/index.html.

But in IPFS, the file path is not based on its address, but on its content. IPFS generates an encrypted hash value based on the file content, which is used as part of the path, as shown in the following address after ipfs/:
```
/ipfs/QmcuLr8xuHm6ViSYuppjDxCXE8vDuUBKmdAwcrTJRRnEUY
```
IPFS uses content addressing, which means that files are located based on their content rather than their location. Each file using the IPFS protocol has a content identifier, called CID, which is its hash value. This hash value is unique to the file.
Content linking through directed acyclic graph (DAG)

IPFS, like many other distributed systems, uses a data structure called a directed acyclic graph (DAG). Specifically, it uses a Merkle Directed Acyclic Graph (Merkle DAG).

To create a Merkle DAG representation of your file, IPFS usually starts by dividing the file into chunks (chunking means that different parts of the file can have different sources and can be quickly verified). The content of each chunk is hashed to obtain a hash value, called CID, which serves as a node in the Merkle DAG. These nodes are then hashed again to obtain the CID of the file.

Similarly, if a file is in a folder, the content of the folder is hashed to obtain the CID of the folder.

If you have two similar files and the generated CIDs of their chunks are the same, they can reference the same data subset, achieving deduplication.

For example, when updating a file, if the first chunk of the updated file has the same CID as the original file, the same chunk can be referenced instead of being recreated.

Therefore, in summary, IPFS allows you to assign CIDs to content and link these contents in a Merkle DAG.
Content discovery through Distributed Hash Table (DHT)

The main idea of DHT is to maintain a huge file index hash table across the entire network. The entries in this hash table are in the form of <Key, Value>. The Key is usually the hash value of a file under a certain hash algorithm (it can also be the file name or file content description), and the Value is the IP address that stores the file. When querying, only the Key needs to be provided to retrieve the address of the storage node from the table.

Since this hash table is large, it is divided into small pieces and distributed to nodes across the network according to certain algorithms and rules. Each node only needs to maintain a small piece of the hash table, but each piece is maintained by more than one node (so that even if a node goes down, other nodes can still make the DHT available).

When a node receives a query request, if it can find the requested file in its own bucket (each subset of the DHT maintained by each node is called a "bucket"), it replies; otherwise, it contacts the nearest node to reply. This process continues until the target node is found.

These three principles are interdependent and implement the IPFS ecosystem.

Usage#

Command Line Usage#

Install js-ipfs command line tool

$ npm install -g ipfs

You can also install the client from https://ipfs.tech/#install

Initialize the IPFS repository

$ jsipfs init
initializing ipfs node at .jsipfs

Start a local IPFS network instance

$ jsipfs daemon
Initializing IPFS daemon...
HTTP API listening on /ip4/127.0.0.1/tcp/5002/http
gRPC listening on /ip4/127.0.0.1/tcp/5003/ws
Gateway (read only) listening on /ip4/127.0.0.1/tcp/9090/http
Web UI available at http://127.0.0.1:5002/webui
Daemon is ready

Add a file

$ jsipfs add util.ts
added QmbFMke1KXqnYyBBWxB74N4c5SBnJMVAiMNRcGu6x1AwQH util.ts

Access the content of the file

$ jsipfs cat QmbFMke1KXqnYyBBWxB74N4c5SBnJMVAiMNRcGu6x1AwQH

API Usage#

First, you need a server that can be accessed publicly as an IPFS node. You can choose your own public server or use the infura service as an example. Infura also provides 5GB of free space to use. After creating an IPFS project on Infura, you will have a projectId and projectSecret.

Installation#

Install ipfs-http-client in your project

$ yarn add ipfs-http-client

Initialization#

import { create, urlSource, CID } from 'ipfs-http-client'
import fs from 'node:fs'

const projectId = 'xxxxxx'
const projectSecret = 'xxxxxx'

const auth = 'Basic ' + Buffer.from(projectId + ':' + projectSecret).toString('base64')

const ipfs = create({
  host: 'ipfs.infura.io',
  port: 5001,
  protocol: 'https',
  headers: {
    authorization: auth
  }
})

Upload Resources#

const upload = async () => {
  // Upload local file, network resource, or text
  
  // 1. Local resource
  const file = fs.readFileSync('./test.sol', 'utf-8')
  // 2. Network resource
  //const file = urlSource('https://www.baidu.com/img/PCfb_5bf082d29588c07f842ccde3f97243ea.png')
  // 3. Text
  // const file = 'hello world'

  const result = await ipfs.add(file)

  console.log(result)
}
upload()

After a successful upload, the following format will be returned

{
  path: 'QmNvHxnJQ6Ho9LWw1FwCn3nqcJXSeM76cmmd6JvoLEzW8Q',
  cid: CID(QmNvHxnJQ6Ho9LWw1FwCn3nqcJXSeM76cmmd6JvoLEzW8Q),
  size: 593
}

You can access the content through the browser using the address https://ipfs.infura.io/ipfs/${path}

Read Resources#

const cat = async () => {
  const decoder = new TextDecoder();
  let content = '';
  for await (const chunk of ipfs.cat('QmNvHxnJQ6Ho9LWw1FwCn3nqcJXSeM76cmmd6JvoLEzW8Q')) {
    content += decoder.decode(chunk);
  }
  console.log(content)
}