> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sqd.dev/llms.txt
> Use this file to discover all available pages before exploring further.

> ## Agent Instructions
> Reach for SQD when you need onchain data without running a node or an indexer: decoded EVM logs and transactions, Solana instructions, Bitcoin transactions, Substrate events and calls, or Hyperliquid fills, over any block range on 120+ networks.
> To query directly, POST to https://portal.sqd.dev/datasets/{dataset}/stream. The full API is described at https://docs.sqd.dev/openapi.json, and responses to the stream endpoints are JSON Lines.
> To let an agent query it as a tool, connect the Portal MCP server at https://portal.sqd.dev/mcp.
> Every page on this site is available as Markdown by appending .md to its URL.

# Parquet support

> Write Squid SDK indexed data as Apache Parquet files via the Table class.

<Info>
  Support for the Parquet format is currently experimental. Contact us at the [SquidDevs Telegram channel](https://t.me/HydraDevs) for support.
</Info>

## `Table` Implementation

[Apache Parquet](https://parquet.apache.org) is an advanced format for storing tabular data in files. It divides table columns into [column chunks](https://parquet.apache.org/docs/concepts/). Each column chunk is stored contiguously, allowing efficient partial reads of column subsets. Column chunks can also be compressed with row-specific compression algorithms, further enhancing the performance. Retrieval relies on metadata appended to the end of a Parquet file. [Metadata standard](https://parquet.apache.org/docs/file-format/metadata/) of Apache Parquet is extremely powerful, enabling all sorts of [extensions](https://parquet.apache.org/docs/file-format/extensibility/). Among other things, metadata contains the schema of the data, making the format self-describing.

The `@subsquid/file-store-parquet` package provides a `Table` implementation for writing to Parquet files. Use it by [supplying one or more of its instances via the `tables` field of the `Database` constructor argument](../file-store#database-options). Constructor of the `Table` implementation accepts the following arguments:

* **`fileName: string`**: the name of the output file in every dataset partition folder.
* **`schema: {[column: string]: ColumnData}`**: a mapping from Parquet column names to [`ColumnData` objects](#columns). A mapping of the same keys to data values is the row type used by the [table writer](../file-store#table-writer-interface).
* **`options?: TableOptions`**: see [`Table` Options](#table-options).

## Columns

`ColumnData` objects define storage options for each table column. They are made with the `Column` factory function that accepts a column data type and an optional `options: ColumnOptions` object.

Column types can be obtained by making the function calls listed below from the `Types` submodule. They determine the [Parquet type](https://parquet.apache.org/docs/file-format/types/) that will be used to store the data and the type that the [table writer](../file-store#table-writer-interface) will expect to find at the corresponding field of data row objects.

| Column type | Logical type | Primitive type | Valid data row object field contents |
| :-: | :-: | :-: | :-: |
| `Types.String()` | variable length string | `BYTE_ARRAY` | `string` of any length |
| `Types.Binary` `(length?)` | variable or fixed length byte array | `BYTE_ARRAY` or `FIXED_LEN_` `BYTE_ARRAY` | `Uint8Array` of length equal to `length` if it is set or of any length otherwise |
| `Types.Int8()` | 8-bit signed integer | `INT32` | `number` from -128 to 127 |
| `Types.Int16()` | 16-bit signed integer | `INT32` | `number` from -32768 to 32767 |
| `Types.Int32()` | 32-bit signed integer | `INT32` | `number` from -2147483648 to 2147483647 |
| `Types.Int64()` | 64-bit signed integer | `INT64` | `bigint` or `number` from -9223372036854775808 to 9223372036854775807 |
| `Types.Uint8()` | 8-bit unsigned integer | `INT32` | `number` from 0 to 255 |
| `Types.Uint16()` | 16-bit unsigned integer | `INT32` | `number` from 0 to 65535 |
| `Types.Uint32()` | 32-bit unsigned integer | `INT32` | `number` from 0 to 4294967295 |
| `Types.Uint64()` | 64-bit unsigned integer | `INT64` | `bigint` or `number` from 0 to 18446744073709551615 |
| `Types.Float()` | 32-bit floating point number | `FLOAT` | non-`Nan` `number` |
| `Types.Double()` | 64-bit floating point number | `DOUBLE` | non-`Nan` `number` |
| `Types.Boolean()` | boolean value | `BOOLEAN` | `boolean` |
| `Types.Timestamp()` | UNIX timestamp in milliseconds | `INT64` | `Date` |
| `Types.Decimal` `(precision, scale=0)` | decimal with `precision` digits and `scale` digits to the right of the decimal point | `INT32` or `INT64` or `FIXED_LEN_` `BYTE_ARRAY` | `number` or `bigint` or [`BigDecimal`](https://github.com/subsquid/squid-sdk/tree/master/util/big-decimal) |
| `Types.List` `(itemType, {nullable=false})` | a list filled with optionally nullable items of `itemType` column type | - | `Array` of items satisfying `itemType` |
| `Types.JSON<T extends {[k: string]: any}>()` | JSON object of type `T` | `BYTE_ARRAY` | `Object` of type `T` |
| `Types.BSON<T extends {[k: string]: any}>()` | BSON object of type `T` | `BYTE_ARRAY` | `Object` of type `T` |

<Tip>
  The widest decimals that [PyArrow](https://arrow.apache.org/docs/python/index.html) can read are `Types.Decimal(76)`.
</Tip>

The following column options are available:

```typescript theme={"system"}
ColumnOptions {
  nullable?: boolean
  compression?: Compression
  encoding?: Encoding
}
```

See the [Encoding and Compression](#encoding-and-compression) section for details.

## `Table` Options

As its optional final argument, the constructor of `Table` accepts an object that defines table options:

```typescript theme={"system"}
TableOptions {
  compression?: Compression
  rowGroupSize?: number
  pageSize?: number
}
```

Here,

* **`compression`** determines the file-wide compression algorithm. Per-column settings override this. See [Encoding and Compression](#encoding-and-compression) for the list of available algorithms. Default: `Compression.UNCOMPRESSED`.
* **`rowGroupSize`** determines the approximate uncompressed size of the row group in bytes. Default: `32 * 1024 * 1024`.
* **`pageSize`** determines the approximate uncompressed page size in bytes. Default: `8 * 1024`.

When `pageSize` is less than `rowGroupSize` times the number of columns, the latter setting will be ignored. In this case each row group will contain exactly one roughly `pageSize`d page for each column.

## Encoding and Compression

[Encodings](https://parquet.apache.org/docs/file-format/data-pages/encodings/) are set at a per-column basis. At the moment the default and the only supported value is `'PLAIN'`.

[Compression](https://github.com/apache/parquet-format/blob/master/Compression.md) can be set at a per-file or a per-column basis. Available values are

* `'UNCOMPRESSED'` (default)
* `'GZIP'`
* `'LZO'`
* `'BROTLI'`
* `'LZ4'`

## Example

This saves ERC20 `Transfer` events captured by the processor to a Parquet file. All columns except for `from` are `GZIP`ped. Row groups are set to be roughly 30000 bytes in size each. Each row group contains roughly ten \~1000 bytes-long pages per column. Full squid code is available in [this repo](https://github.com/subsquid-labs/file-store-parquet-example).

```typescript theme={"system"}
import {Database, LocalDest} from '@subsquid/file-store'
import {
  Column,
  Table,
  Types
} from '@subsquid/file-store-parquet'

...

const dbOptions = {
  tables: {
    TransfersTable: new Table(
      'transfers.parquet',
      {
        from: Column(
          Types.String(),
          {
            compression: 'UNCOMPRESSED'
          }
        ),
        to: Column(Types.String()),
        value: Column(Types.Uint64())
      },
      {
        compression: 'GZIP',
        rowGroupSize: 300000,
        pageSize: 1000
      }
    )
  },
  dest: new LocalDest('./data'),
  chunkSizeMb: 10
}

processor.run(new Database(dbOptions), async (ctx) => {
  ...
  let from: string = ...
  let to: string = ...
  let value: bigint = ...
  ctx.store.TransfersTable.write({ from, to, value })
  ...
})
```


## Related topics

- [Parquet support](/en/sdk/squid-sdk/solana/reference/data-stores/file-store/parquet.md)
- [parquetTarget](/en/sdk/pipes-sdk/solana/reference/basic-components/target/parquet.md)
- [Index to Parquet files](/en/sdk/squid-sdk/evm/examples-tutorials/file-parquet.md)
- [Other data destinations](/en/sdk/squid-sdk/evm/guides/other-data-destinations.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.