基于区块链的毕业设计Ethereum ETL for PostgreSQL – 用于PostgreSQL的以太坊ETL

本文提供基于区块链的毕业设计国外最新区块链项目源码下载,包括solidity,eth,fabric等blockchain区块链,基于区块链的毕业设计Ethereum ETL for PostgreSQL – 用于PostgreSQL的以太坊ETL 是一篇很好的国外资料

Ethereum ETL for PostgreSQL

The steps below will allow you to bootstrap a PostgreSQL database in GCP with full historical and real-time Ethereum data: blocks, transactions, logs, token_transfers, and traces.

The whole process will take between 24 and 72 hours.

Prerequisites:

  • Python 3.6+
  • gcloud
  • psql
  • Cloud SQL Proxy

1. Export Ethereum data from BigQuery to CSV files in GCS

  • Install gcloud and run gcloud auth login
  • Run
pip install -r requirements.txt export BUCKET=<your_gcs_bucket> bash ethereum_bigquery_to_gcs.sh $BUCKET

Optionally provide start and end dates: bash ethereum_bigquery_to_gcs.sh $BUCKET 2020-01-01 2020-01-31

Exporting to CSV files is going to take about 10 minutes.

2. Import data from CSV files to PostgreSQL database in Cloud SQL

  • Create a new Cloud SQL instance
export CLOUD_SQL_INSTANCE_ID=ethereum-0 export ROOT_PASSWORD=<your_password> gcloud sql instances create $CLOUD_SQL_INSTANCE_ID --database-version=POSTGRES_11 --root-password=$ROOT_PASSWORD      --storage-type=SSD --storage-size=100 --cpu=4 --memory=6      --database-flags=temp_file_limit=2147483647

Notice the storage size is set to 100 GB. It will scale up automatically to around 1.5 TB when we load in the data.

  • Add Cloud SQL service account to GCS bucket as objectViewer. Run gcloud sql instances describe $CLOUD_SQL_INSTANCE_ID, then copy serviceAccountEmailAddress from the output and add it to the bucket.

  • Create the database and the tables:

gcloud sql databases create ethereum --instance=$CLOUD_SQL_INSTANCE_ID  # Install Cloud SQL Proxy following the instructions here https://cloud.google.com/sql/docs/mysql/sql-proxy#install ./cloud_sql_proxy -instances=myProject:us-central1:${CLOUD_SQL_INSTANCE_ID}=tcp:5433  cat schema/*.sql | psql -U postgres -d ethereum -h 127.0.0.1  --port 5433 -a
  • Run import from GCS to Cloud SQL:
bash ethereum_gcs_to_cloud_sql.sh $BUCKET $CLOUD_SQL_INSTANCE_ID

Importing to Cloud SQL is going to take between 12 and 24 hours.

A few performance optimization tips for initial loading of the data:

  • Turn off fsync https://www.postgresql.org/docs/11/runtime-config-wal.html.
  • Use UNLOGGED tables.
  • Turn OFF auto backups and vacuum on Google Cloud SQL instance.

3. Apply indexes to the tables

  • Run:
cat indexes/*.sql | psql -U postgres -d ethereum -h 127.0.0.1  --port 5433 -a

Creating indexes is going to take between 12 and 24 hours. Depending on the queries you’re going to run you may need to create more indexes or partition the tables.

Cloud SQL instance will cost you between $200 and $500 per month depending on whether you use HDD or SSD and on the machine type.

4. Streaming

Use ethereumetl stream command to continually pull data from an Ethereum node and insert it to Postgres tables: https://github.com/blockchain-etl/ethereum-etl/tree/develop/docs/commands.md#stream.

Follow the instructions here to deploy it to Kubernetes: https://github.com/blockchain-etl/blockchain-etl-streaming.


4。流媒体

下面的步骤将允许您使用完整的历史和实时以太坊eth数据(块、事务、日志、令牌传输和跟踪)在GCP中引导PostgreSQL数据库。整个过程需要24到72小时。

先决条件:

可选地提供开始和结束日期:bash ethereumu bigqueryu to_gcs.sh公司$BUCKET 2020-01-01 2020-01-31导出到CSV文件大约需要10分钟。

  • Python 3.6+
  • gcloud
  • psql
  • 云SQL代理

1. Export Ethereum data from BigQuery to CSV files in GCS

  • 安装gcloud并运行gcloud auth login
  • 运行
pip install -r requirements.txt export BUCKET=<your_gcs_bucket> bash ethereum_bigquery_to_gcs.sh $BUCKET

请注意,存储大小设置为100 GB。当我们加载数据时,它将自动扩展到1.5 TB左右。

将云SQL服务帐户作为objectViewer添加到GCS bucket。运行gcloud sql instances description$CLOUDu sqlu INSTANCEu ID,然后从输出中复制serviceAccountEmailAddress并将其添加到bucket中。

2. Import data from CSV files to PostgreSQL database in Cloud SQL

  • 创建新的云SQL实例
export CLOUD_SQL_INSTANCE_ID=ethereum-0 export ROOT_PASSWORD=<your_password> gcloud sql instances create $CLOUD_SQL_INSTANCE_ID --database-version=POSTGRES_11 --root-password=$ROOT_PASSWORD      --storage-type=SSD --storage-size=100 --cpu=4 --memory=6      --database-flags=temp_file_limit=2147483647

创建数据库和表:

  • 将云SQL服务帐户作为objectViewer添加到GCS bucket。运行gcloud sql instances description$CLOUDu sqlu INSTANCEu ID,然后从输出中复制serviceAccountEmailAddress并将其添加到bucket中。要从云创建表,请关闭gcli>
  • 使用未标记的表。
gcloud sql databases create ethereum --instance=$CLOUD_SQL_INSTANCE_ID  # Install Cloud SQL Proxy following the instructions here https://cloud.google.com/sql/docs/mysql/sql-proxy#install ./cloud_sql_proxy -instances=myProject:us-central1:${CLOUD_SQL_INSTANCE_ID}=tcp:5433  cat schema/*.sql | psql -U postgres -d ethereum -h 127.0.0.1  --port 5433 -a
  • 关闭Google Cloud SQL实例的自动备份和真空。
bash ethereum_gcs_to_cloud_sql.sh $BUCKET $CLOUD_SQL_INSTANCE_ID

创建索引需要12到24小时。根据要运行的查询,可能需要创建更多索引或对表进行分区。

云SQL实例每月的成本在200到500美元之间,具体取决于您使用的是HDD还是SSD以及机器类型。

  • 跑步:
  • Use UNLOGGED tables.
  • Turn OFF auto backups and vacuum on Google Cloud SQL instance.

3. Apply indexes to the tables

  • Run:
cat indexes/*.sql | psql -U postgres -d ethereum -h 127.0.0.1  --port 5433 -a

使用ethereumetl stream命令不断从以太坊eth节点提取数据并将其插入Postgres表:https://github.com/blockchain etl/ethereum etl/tree/develope/docs/commands.md#stream。

按照此处的说明将其部署到Kubernetes:https://github.com/blockchain-etl/blockchain-etl-streaming。用于PostgreSQL的以太坊ethETL。将以太坊eth数据从BigQuery导出到GCS中的CSV文件。将CSV文件中的数据导入云SQL中的PostgreSQL数据库。对表应用索引

4。流媒体

  • Python 3.6+
  • gcloud
  • psql
  • 云SQL代理
  • 安装gcloud并运行gcloud auth login
  • 运行
  • 创建新的云SQL实例
  • 将云SQL服务帐户作为objectViewer添加到GCS bucket。运行gcloud sql instances description$CLOUDu sqlu INSTANCEu ID,然后从输出中复制serviceAccountEmailAddress并将其添加到bucket中。要从云创建表,请关闭gcli>
  • 使用未标记的表。
  • 关闭Google Cloud SQL实例的自动备份和真空。
  • 跑步:
  • 4. Streaming

    Use ethereumetl stream command to continually pull data from an Ethereum node and insert it to Postgres tables: https://github.com/blockchain-etl/ethereum-etl/tree/develop/docs/commands.md#stream.

    Follow the instructions here to deploy it to Kubernetes: https://github.com/blockchain-etl/blockchain-etl-streaming.

    部分转自网络,侵权联系删除区块链源码网

    www.interchains.cc

    https://www.interchains.cc/19574.html

    区块链毕设网(www.interchains.cc)全网最靠谱的原创区块链毕设代做网站 部分资料来自网络,侵权联系删除! 最全最大的区块链源码站 ! QQ3039046426
    区块链知识分享网, 以太坊dapp资源网, 区块链教程, fabric教程下载, 区块链书籍下载, 区块链资料下载, 区块链视频教程下载, 区块链基础教程, 区块链入门教程, 区块链资源 » 基于区块链的毕业设计Ethereum ETL for PostgreSQL – 用于PostgreSQL的以太坊ETL

    提供最优质的资源集合

    立即查看 了解详情