Aye

Proponent: Subsquid Lab Official - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE

Beneficiary: Subsquid Lab Official - 13bfKSQXoBn3AMLtZaW6BKv797fqZzsD3PYF6xpJDir3tdoE

Contact Details: Subsquid Labs GmbH, 6302 Zug, Switzerland - mf@subsquid.io

Short description: Ongoing development and maintenance costs for public Subsquid Archives and the Giant Squid API

Archive raw data: Archive Infrastructure Metrics - July, August, September

Giant Squid raw data: Giant Squid Infrastructure Metrics - July, August, September

Requested DOT: 35565.67 DOT + 9669.10 DOT = 45234.77 DOT = $ 298,549.46

Previous proposals: https://polkadot.polkassembly.io/referenda/145

Motivation

A) Archives

Archives are an important piece of Polkadot and Kusama data infrastructure provided by Subsquid. They provide access to on-chain data in GraphQL format for all major parachains on Kusama and Polkadot, with more chains getting support as the ecosystem grows. They are being used as a data source for Squids and for efficient data exploration and ad-hoc queries. An up-to-date list of Archives is kept in a public Archive Registry.

The Archives, as performant data sources of historical on-chain data, are critical to the operation of backend APIs run by major projects in the Polkadot ecosystem (the list is not exhaustive):

Polkassembly
Talisman
RMRK
SubWallet (+ GiantSquid API)
KodaDot
Phala dApp
Subsocial + Polkaverse
Polkascan (+ GiantSquid API)
Calamar (+ GiantSquid API)
Moonsama/Exosama
Reef
Manta Wallet
ZooDAO
Zenlink

All archives can be accessed and queried via an web intuitive GraphQL interface and are often used as developer friendly replacement of a block explorer. For example, here’s a query for the last ten transfers on Kusama blockchain run against the Kusama Archive endpoint.

Over the third Quarter of 2023 we've provided a free public data indexing service for Substrate blockchains (some archives have been dismissed, new ones have been added), so that everyone interested in the projects can publicly access indexed data on Substrate Events, Extrinsics, Storage Items and EVM logs. Here are projects (parachains and mainnets aiming for future parachains slots) we currently support:

acala
aleph-zero-testnet
altair
amplitude
astar
bajun
basilisk
bifrost
bifrost-polkadot
binance
bitcountry-pioneer
calamari
centrifuge
cere
collectives
composable-finance
crab
crust
darwinia
efinity
elysium
elysium-testnet
equilibrium
exosama
fantom
foucoco
frequency
frequency-testnet
gmordie
hashed
heiko
hydradx
integritee-network
interlay
invarch-tinkernet
joystream
kabocha
karura
khala
kilt
kusama
kylin
litentry
litmus
mantle-testnet
moonbase
moonbeam
moonbeam-evm
moonriver
myriad
myriad-testnet
origin-trail
pangolin
parallel
peaq
pendulum
phala
picasso
polkadex
polkadot
polymesh
reef
reef-testnet
robonomics
rococo
shibuya
shiden
soonsocial
sora
sora-testnet
statemine
statemint
t0rn
tanganika
ternoa
tidechain
turing
vara
westend-collectives
xsocial
xx-network
zeitgeist

B) Giant Squid Giant Squid is a set of open-source GraphQL APIs built with Squid SDK and maintained by Subsquid Labs. The APIs cover the most commonly used data for Pokadot, Kusama and a growing list of major parachains and is used by explorers (such as Calamar) and wallets (e.g. SubWallet).

The Giant Squid currently consists of two families of endpoints outlined below. The API endpoints for each network and the current status are available at the Status page.

Explorer API. The endpoint URLs follow the convention https://squid.subsquid.io/gs-explorer-${network}/graphql. Support queries on:

Blocks
Events
Extrinsics
Calls

Statistics API. The endpoint URLs follow the convention https://squid.subsquid.io/gs-stats-${network}/graphql. Support queries on:

Validator/Collator data
Account balances
Staking statistics
Nomination pools statistics
Transfers statistics
Extrinsic statistics

Achievements

ArrowSquid release: The main feature introduced by the ArrowSquid update is the new ability of the processor to ingest unfinalized blocks directly from a network node, instead of waiting for the archive to ingest and serve it first. The processor can now handle forks and rewrite the contents of its database if it happens to have indexed orphaned blocks. This allows Subsquid-based APIs to become near real-time and respond to the on-chain activity with subsecond latency.Another major feature introduced by ArrowSquid is the support for transaction execution receipts, traces and state diffs.
Decentralizing Archives: launched public testnet on CoinList with over 40.000 participants (https://coinlist.co/subsquid-testnet). Several thousand indexers in the Subsquid testnet were deployed to index Moonbeam, a Polkadot parachain.
Archive requests increased by over 40% (1.46B requests)
Archives and Subsquid are powering ≈ 650 live projects in Q3 2023

The biggest Polkadot projects have fully migrated or are in the process of finishing the migration from FireSquid to ArrowSquid, including:

Polkassembly
Subsocial
HydraDX
KodaDot
Talisman
Multix Multisig
RMRK
Apillon
StellaSwap

Support continues to be added for new and upcoming Polkadot projects, including new and burgeoning partnerships and Archive implementations with Polimec Protocol, Acurast, Watr, and others.

Statistics and data

Over the course of Q3 2023:

Served a total of 1,464,029,333 (~1.46 billion) queries to the substrate based archives in Q3 2023
Served ~500m monthly archive requests for substrate based archives
The highest daily total responses have been 19.44m in a single 24 hour period for substrate
The daily queries confirm a growing trend, only partly due to the deployment of new Archives (which initially are not very busy)

Giant Squid is a service we introduced in Q1 2023, and has been launched in mid-January, here are two charts showing its quick adoption by the community, as an index of its success:

Costing

This proposal includes the combined costs for running Giant Squid and Archives for Polkadot, Kusama and their Parachains,which amounts to a total of 45234.77 DOT. Raw data for Archives can be seen here. Raw data for Giant Squid can be seen here.

The proposal is submitted towards the Polkadot treasury. Here is a summary of the running costs for Archives, in a tabular format:

Giant Squid running costs:

You can read the full report here.