<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[The Web3 Engineering Masterclass]]></title><description><![CDATA[The Web3 Engineering Masterclass]]></description><link>https://subgraphdev.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sun, 08 Nov 2026 23:36:06 GMT</lastBuildDate><atom:link href="https://subgraphdev.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Firehose: A High-Speed Gateway to Blockchain Data]]></title><description><![CDATA[Discover how Firehose, a blockchain-agnostic data extraction engine, is reshaping the way we access and utilize blockchain data.]]></description><link>https://subgraphdev.hashnode.dev/firehose-a-high-speed-gateway-to-blockchain-data</link><guid isPermaLink="true">https://subgraphdev.hashnode.dev/firehose-a-high-speed-gateway-to-blockchain-data</guid><category><![CDATA[Blockchain]]></category><category><![CDATA[Web3]]></category><category><![CDATA[#thegraph]]></category><category><![CDATA[indexing]]></category><dc:creator><![CDATA[Ujjwal Thakur]]></dc:creator><pubDate>Sat, 16 Sep 2023 05:41:51 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1694842598432/66b1abaa-115a-4b76-8bdc-02c12cf0d56b.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In the modern world, data is the lifeblood, akin to the role of oil in the industrial revolution. The importance of data in today's world cannot be overstated, as it drives decision-making, innovation, and progress across various industries. It empowers businesses with insights, fuels research breakthroughs in fields like medicine, it enhances athlete performance, talent scouting, and fan engagement, ultimately contributing to more informed, innovative, and improved decision-making.</p>
<p>In our data-driven world, blockchain stands out as a game-changer. It's a decentralized marvel, a digital ledger that ensures secure, transparent, and unalterable data records. This technology has not only revolutionized how we store and manage data in the digital age but has also ushered in new and innovative use cases. Blockchain data inherits transparency, immutability, and security, ensuring that the data used for decision-making is reliable and tamper-proof, generating various use cases across various sectors. It can revolutionize healthcare by securely storing patient records and enabling seamless data sharing among healthcare providers. It can improve the supply chain by enabling real-time tracking of goods and reducing the risk of fraudulent activities, thus enhancing financial transparency.</p>
<h2 id="heading-blockchain-data-amp-its-accessibility-issues">Blockchain data &amp; its accessibility issues</h2>
<p>As previously discussed, blockchain technology holds the potential for a groundbreaking transformation in the way we store, manage, and conduct data transactions. Its decentralized and transparent characteristics provide an unparalleled level of security and trust within the digital realm. Nevertheless, despite its myriad advantages, gaining access to blockchain data presents a significant challenge, particularly for developers building data-intensive applications.</p>
<ol>
<li><p><strong>Lack of Standardization:</strong> Different blockchains, like Bitcoin and Ethereum, use their own ways to save and share data. This can be tough for developers because they have to learn how to talk to each other separately. It's a bit like having to learn many languages instead of just one that everyone can understand easily. So, until all blockchains agree on the same set of rules to store and share data, dealing with blockchain data will continue to be a bit of a tricky task.</p>
</li>
<li><p><strong>Slow Extraction Process:</strong> The process of extracting data from the blockchain can be quite sluggish. This is because it involves a form of communication with a blockchain node known as JSON-RPC polling, a method employed to request data from the blockchain node. The process entails sending a request to the node and subsequently awaiting a response. This can be a time-consuming procedure, primarily due to the necessity of initiating individual requests for each specific piece of data.</p>
</li>
<li><p><strong>Lack of Insights:</strong> While the blockchain's promise of secure and transparent data storage is undeniable, it's crucial to recognize that the raw data residing on the blockchain is not inherently useful for gaining actionable insights. This data, in its unprocessed form, lacks context and structure. To derive meaningful insights from blockchain data, a resource-intensive transformation process is essential. This procedure involves cleaning, structuring, storing, and making it easily available. The transformation journey from raw blockchain data to valuable insights is a challenging endeavor that demands significant resources and expertise.</p>
</li>
</ol>
<p>To address the challenges of a lack of standardization, a slow extraction process, and a deficiency in insights within the blockchain data space, <strong>The Graph</strong>, a decentralized indexing protocol, has forged a partnership with <strong>StreamingFast</strong>, a distinguished expert in scalable cross-chain architecture for streaming blockchain data. After StreamingFast's integration into The Graph ecosystem as one of its core developers, they introduced two groundbreaking innovations: <strong>Firehose</strong>, a blockchain-agnostic, high-performance data extraction engine, and <strong>Substreams</strong>, a powerful parallelized engine designed to process blockchain data efficiently.</p>
<h2 id="heading-what-is-firehose">What is Firehose?</h2>
<p>Firehose is a blockchain-agnostic, high-performance data extraction engine that provides real-time access to blockchain data. It addresses the issue of slow data extraction by offering a solution for extracting data from blockchains in real time, storing it in flat files, and making it readily available for various transformations and processes.</p>
<p>Firehose is constructed using a component-based design, dividing the entire system or software into distinct components. The Firehose components include the <strong>Firehose-enabled Blockchain Node, Reader, Merger, Relayer, and Firehose gRPC</strong>. Each of these components is developed, tested, and maintained independently. These components communicate with one another to ensure the proper functioning of the system.</p>
<h2 id="heading-why-firehose">Why Firehose?</h2>
<p>Firehose brings forth a range of substantial advantages when it comes to handling blockchain data. Among all the advantages, these emerge as particularly prominent and valuable:</p>
<ol>
<li><p><strong>High Speed:</strong> Firehose is designed to significantly increase the speed and performance of blockchain data extraction. It offers a streamlined approach to data processing, reducing the risk of downtime that can be caused by linear data extraction methods. This leads to faster and more efficient blockchain data access and utilization.</p>
</li>
<li><p><strong>Chain Agnostic:</strong> Firehose is chain agnostic, which means that it can be used to stream data from any blockchain. This is a huge advantage because it allows developers to build applications that can access data from multiple blockchains. As new blockchains are created, Firehose can be used to stream data from those blockchains as well.</p>
</li>
<li><p><strong>Efficient and Fast Indexing:</strong> Enabling any blockchain to be 'firehose-enabled' provides immediate support for Substreams, high-speed parallel processing, and integration into The Graph. This, in turn, facilitates rapid and efficient indexing support by The Graph indexers.</p>
</li>
<li><p>C<strong>ommunity Support</strong>: Enabling Firehose provides support from a community of developers experienced in building substreams and subgraphs. This can result in faster adoption of your chain among Dapp builders.</p>
</li>
</ol>
<p><strong>Firehose</strong> offers a promising solution to the longstanding challenges of accessing and leveraging blockchain data. With its ability to provide real-time, secure, and efficient data extraction from any blockchain, Firehose opens doors to new possibilities for developers, researchers, and businesses. As we harness the power of blockchain data through these cutting-edge tools, we stand on the cusp of transformative breakthroughs across multiple industries, all driven by the lifeblood of our digital age: data. The journey towards a more accessible, insightful, and data-driven future continues, and with innovations like Firehose, it promises to be an exciting and fruitful one. We can learn more about Firehose from <a target="_blank" href="https://firehose.streamingfast.io/">here</a>.</p>
]]></content:encoded></item><item><title><![CDATA[Substreams: Unlocking Lightning-Fast Indexing of Blockchain Data]]></title><description><![CDATA[Explore how Substreams is changing the landscape blockchain data indexing technology.]]></description><link>https://subgraphdev.hashnode.dev/substreams-unlocking-lightning-fast-indexing-of-blockchain-data</link><guid isPermaLink="true">https://subgraphdev.hashnode.dev/substreams-unlocking-lightning-fast-indexing-of-blockchain-data</guid><category><![CDATA[Blockchain]]></category><category><![CDATA[subgraph]]></category><category><![CDATA[indexing]]></category><category><![CDATA[The Graph]]></category><category><![CDATA[substreams]]></category><dc:creator><![CDATA[Ujjwal Thakur]]></dc:creator><pubDate>Wed, 21 Jun 2023 05:19:49 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1687154464389/b01b8845-0287-4d20-87b0-b2fe0374b63b.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Building data-heavy applications is always hard for developers, it is even more challenging when developers have to deal with blockchain data. The process of accessing or extracting blockchain data is not only <strong>slow</strong> but it is also <strong>complex</strong> due to the built-in functionality of blockchain and smart contracts. Smart contracts are designed to store data, transfer assets, and perform other operations within a decentralized network or blockchain. Performing basic read operations in a smart contract is relatively easy. For example, In the case of any NFTs project like Bored Ape Yacht Club, we can perform basic read operations on smart contracts like getting the owner of a certain Ape, getting the content URI of an Ape based on their ID, or the total supply, as these read operations are programmed directly into the smart contract, if we wanted to query for apes that are owned by a certain address, and filter by one of its characteristics, we would not be able to get that information by interacting directly with the contract itself. We can see that the more complex operations, such as filtering and aggregating data, are much more difficult to achieve. This is because, the storage capacity of the blockchain is limited, which can be a challenge when working with large datasets. Since each node in the network must store a copy of the entire blockchain, storing large amounts of data can become impractical, especially when considering the cost of storage and processing power.</p>
<p>In the past, developers faced challenges in accessing and utilizing blockchain data due to limitations in functionality. To bridge this gap, they resorted to constructing centralized indexing servers as a workaround. This approach required developers to operate their full nodes to access the data stored on the blockchain. The indexing servers would retrieve data from the blockchain, store it in a separate database, and provide an API for developers to interact with. However, this solution had significant drawbacks.</p>
<p>Firstly, the process of maintaining and operating these centralized indexing servers was resource-intensive and time-consuming. Running a full node and continuously synchronizing it with the blockchain required substantial computational power and storage capacity. Additionally, as new blocks were added to the blockchain, the indexing servers had to update their database, which further strained their resources.</p>
<p>Secondly, the reliance on centralized indexing servers was not cost-efficient. Setting up and maintaining these servers incurred expenses, such as hardware, infrastructure, and maintenance costs. These expenses could be burdensome for developers, particularly for projects with limited resources or smaller teams.</p>
<p>Moreover, building and managing centralized indexing servers introduced complexity and potential points of failure. The task involved intricate programming, database management, and API development, which increased the likelihood of errors. Developers had to carefully design and maintain the servers to ensure accurate and reliable data retrieval. This reliance on external servers also made the system vulnerable to various risks. Users had to trust the indexing server teams to operate the servers correctly and securely. However, there were uncertainties, such as the possibility of the projects going out of business, modifying data for strategic reasons, being acquired by other entities, or making inadvertent mistakes. These factors eroded the trust and reliability of the system, undermining the desired advantages of blockchain technology.</p>
<p>Consequently, this situation resembled the centralized nature of traditional web applications (Web 2.0) where users relied on intermediaries for data access and manipulation. It was evident that a more decentralized and efficient solution was necessary to overcome the limitations and challenges of performing complex operations on blockchain data.</p>
<p>Therefore, developers began exploring alternative approaches that minimize reliance on resource-intensive and centralized indexing servers. They sought ways to directly access and process blockchain data without sacrificing security, reliability, and decentralization. This involved the development of new technologies, protocols, and tools that provide efficient and user-friendly methods for interacting with blockchain data. By leveraging these advancements, developers aim to empower users to perform complex operations on blockchain data while maintaining the core principles of decentralization and trustlessness.</p>
<p>A solution to this problem is to use decentralized indexing and querying services, this is where <strong>The Graph</strong> comes to rescue us. The Graph is a decentralized protocol for indexing and querying data from blockchain networks.The Graph provides a marketplace of <strong>subgraphs</strong>, which are open APIs that index and expose specific data from the blockchain. Developers build their subgraphs, which can be used to index and query data from specific smart contracts. The Graph enables developers to leverage the power of the blockchain network without having to run a full node or rely on centralized indexing servers.</p>
<h2 id="heading-how-do-the-subgraphs-work">How do the subgraphs work?</h2>
<p>In simple words, a subgraph is a piece of code written by a developer that defines how to ingest, index and serve data. A subgraph extract data from the blockchain, process it and stores it so that it can be easily queried via <strong>GraphQL</strong>. Developers use subgraphs to feed data to the front-end interface of their application. Developers define the criteria for the index by writing a GraphQL schema, and mapping functions are created to map the relevant data to the schema. The subgraph is then deployed and made available for other developers to use, making it easier for them to find and use specific data on the blockchain.</p>
<p>At the core level, a subgraph can be considered an ETL process in the context of blockchain data. ETL stands for <strong>extraction(E), transformation(T), and loading(L)</strong> of blockchain data. Let's delve into each of them in detail:</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1686968337342/8eceb368-3cfa-4b79-a35a-4d86c6743003.png" alt class="image--center mx-auto" /></p>
<ol>
<li><strong>Extraction:</strong> The extraction process involves retrieving data from the blockchain. In the case of Ethereum, the developer typically interacts with the Ethereum client or a service provider's API to extract relevant data. This can include fetching blocks, transactions, events, and other information from the blockchain. In the case of the subgraphs, the extraction process is done by the Graph node JSON-RPC polling in which The Graph node connects to the target blockchain network and listens for events emitted by smart contracts or other blockchain entities. It utilizes specialized libraries or APIs provided by the blockchain platform to monitor the network for new transactions, blocks, or specific events.</li>
</ol>
<details><summary>JSON-RPC polling refers to a method of communication between a client and a server using the JSON-RPC protocol. In this approach, the client regularly sends requests (polls) to the server to retrieve updated information or perform specific actions.</summary><div data-type="detailsContent"></div></details>

<ol>
<li><p><strong>Transformation:</strong> After extracting the raw data, it needs to be transformed into a structured format that is suitable for analysis and querying. The transformation step involves cleaning, normalizing, and organizing the data. This process may include tasks such as data type conversions, merging related data, filtering irrelevant information, and applying business logic to create derived data points. The goal is to convert the extracted data into a consistent and coherent format.</p>
</li>
<li><p><strong>Loading:</strong> Once the data has been transformed, it needs to be loaded into a database for efficient storage and retrieval. The transformed data is loaded into a dedicated database, often optimized for fast and efficient querying. This step ensures that the data extracted and transformed by the subgraph is readily available for developers to query and utilize in their applications. By preloading the data into a database, the subgraph minimizes the need for real-time interactions with the blockchain, leading to faster response times and improved performance.</p>
</li>
</ol>
<p>After the database is populated with the extracted blockchain data, the Graph node exposes a query API that developers can interact with. This API provides a standardized interface to retrieve specific subsets of data from the subgraph based on developers' queries, allowing them to access the blockchain data easily and efficiently. By following these steps, a subgraph developer can extract data from the blockchain, transform it into a usable format, load it into a database, and enable users to query and interact with the data efficiently. This process is crucial for building decentralized applications on top of blockchain protocols, as it provides a structured and accessible layer for blockchain data.</p>
<h2 id="heading-limitations-of-subgraphs-syncing-composability-and-real-time-data-lag">Limitations of Subgraphs: Syncing, Composability and Real-Time Data Lag</h2>
<p>A pretty awesome solution, but like any technology, subgraphs also have their limitations. One significant challenge with subgraphs is the time it takes for them to sync and update with the blockchain. As blockchains continuously grow and new data is added, subgraphs need to stay synchronized to provide accurate and up-to-date information. However, the syncing process can be time-consuming, particularly for large blockchains with extensive transaction histories. This delay between blockchain updates and subgraph synchronization means that there can be a lag in accessing the most recent data, which can be a hindrance for developers and applications that require real-time or near-real-time information. It's an aspect that developers and users need to consider while leveraging subgraphs for their data extraction and querying needs.</p>
<p>Another notable issue with subgraphs is their isolated nature at the indexing level, which can impact the overall efficiency and flexibility of data extraction. Each subgraph operates independently and focuses on indexing and serving data for specific smart contracts or dApps. While this isolation allows for modularity and specialization, it also means that data from different subgraphs are not easily combinable or cross-referenced within the indexing process. This lack of composability can result in limitations when developers want to access and analyze data that spans multiple subgraphs or requires aggregating information from various sources. It may require additional efforts to manually reconcile data from different subgraphs, potentially leading to increased complexity and reduced efficiency in data extraction and analysis. The isolated nature of subgraphs can hinder the ability to leverage synergies between different datasets. By treating each subgraph as a separate entity, it becomes challenging to create connections and derive insights from the interplay of data across multiple blockchain applications or smart contracts. This limitation may restrict the full potential of data analysis and exploration, hindering the development of complex, data-centric applications.</p>
<h2 id="heading-enhancing-subgraph-performance-through-innovations">Enhancing Subgraph Performance through Innovations</h2>
<p>We saw that the current tech state of the subgraph has some scope for improvement, one is to tackle large syncing time which occurs due to the linear extraction process used by The Graph Node to extract smart contract data and the other issue is isolation. So it is clear from this, if we want to improve the performance of subgraphs we need to look into these two things. It appears that to improve the overall performance of the subgraphs we need a solution that can improve the <strong>extraction(E)</strong> layer of the subgraphs or a technology that can improve the extraction speed of The Graph node.</p>
<p>In 2021, <a target="_blank" href="https://www.streamingfast.io/">StreamingFast</a>, a company specializing in scalable, cross-chain architecture for streaming blockchain data, joined <strong>The Graph</strong> as a core protocol developer. Their primary goal was to enhance the extraction process of subgraphs. As part of the Graph Core Developer team, <strong>StreamingFast</strong> introduced <strong>Firehose</strong>, a high-performance data ingestion method from blockchains, integrating it into The Graph. Initially, complex subgraphs could take weeks to sync, causing delays for developers working with The Graph. To address this issue, StreamingFast developed a prototype called <strong>Sparkle</strong>, significantly reducing the sync time from weeks to hours for a particular subgraph. Building on this success, StreamingFast expanded <strong>Sparkle's</strong> capabilities, resulting in the creation of <strong>substreams</strong> that can scale across all subgraphs on different chains.</p>
<h2 id="heading-what-are-substreams">What are substreams?</h2>
<p>Substream is a new powerful <strong>parallelized engine</strong> to process blockchain data developed for <strong>The Graph Network</strong> by <strong>StreamingFast</strong>. The Substreams engine is responsible for running developer-defined data transformations to process blockchain data.</p>
<p>Earlier we defined a subgraph as an ETL process in the context of blockchain data. ETL stands for Extract, Transform, and Load. In the context of blockchain data, ETL refers to the process of extracting data from the blockchain, transforming it into a format that can be used by applications, and loading it into a data warehouse or database. In the new subgraph architecture, the extraction of blockchain data will be carried out by a graph node equipped with Firehose capabilities. Subsequently, the transformation of this data will be performed utilizing the substreams functionality.</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1687075864214/6c8a61c3-daea-4874-b62d-cc3d1fa0bd26.png" alt class="image--center mx-auto" /></p>
<p><strong>Substream</strong> is a rethinking of the <strong>transformation</strong> layer, where instead of writing subgraph handlers in AssemblyScript, you will write substream module in Rust. Substreams Modules are small pieces of Rust code running in a <a target="_blank" href="https://webassembly.org/">WebAssembly (WASM)</a> virtual machine. Rust's performance and safety features, combined with the extensive library ecosystem, empower Substreams to handle complex computations and transformations efficiently. This powerful combination ensures Substreams' adaptability and effectiveness in processing blockchain data on-the-fly.</p>
<h3 id="heading-benefits-of-using-substreams">Benefits of using substreams:</h3>
<ol>
<li><p>Speed: Substreams prioritize speed through their parallelized architecture and streaming-first design, ensuring high throughput for indexing blockchain data efficiently.</p>
</li>
<li><p>Composability: Substreams offer composability, enabling easy combinations of modules to create complex indexing pipelines. This flexibility ensures scalability, allowing developers to tailor indexing solutions according to specific requirements.</p>
</li>
<li><p>Reusability: Substreams emphasize reusability, allowing developers to employ them for multiple indexing tasks. This cost-effective approach saves time and effort by avoiding the need to recreate indexing logic for different tasks.</p>
</li>
<li><p>Custom sinks: Substreams support custom sinks, facilitating seamless integration with developers' preferred data storage or analytics solutions. This flexibility empowers developers to create customized indexing solutions aligned with their unique needs.</p>
</li>
<li><p>Scalability: Substreams are designed with scalability in mind, effortlessly handling large volumes of data. Their architecture enables seamless expansion to accommodate growing demands.</p>
</li>
<li><p>Flexibility: Substreams offer exceptional flexibility, capable of indexing a wide range of blockchain data. This versatility allows for diverse applications and use cases, accommodating different types of data in the blockchain ecosystem.</p>
</li>
</ol>
<p>In conclusion, the introduction of substreams, firehose, and custom sinks within The Graph ecosystem has revolutionized the extraction, transformation, and loading (ETL) layer of subgraph development. These advancements have addressed critical limitations and challenges faced by developers working with blockchain data. Substreams offer speed, composability, reusability, customizability, scalability, and flexibility, making them a powerful solution for indexing blockchain data efficiently. The integration of <strong>Firehose</strong> has significantly reduced syncing times, enabling near-real-time access to up-to-date information. Additionally, the use of custom sinks allows seamless integration with preferred data storage or analytics solutions. By leveraging these technologies, developers can build data-heavy applications on top of blockchain networks while maintaining decentralization, trustlessness, and performance. The Graph has successfully provided a decentralized and efficient solution that empowers developers to perform complex operations on blockchain data, opening up new possibilities for the development of innovative and impactful decentralized applications. You can explore and learn more about substreams <a target="_blank" href="https://substreams.streamingfast.io/">here</a> .</p>
<hr />
<p>If you enjoyed reading my blog, I would greatly appreciate it if you could take a moment to show your support by liking and commenting on it. Your feedback and engagement are incredibly valuable to me and will help me to continue improving and creating content that is informative and engaging. I'll be making a bit of content for The Graph in the coming days. Follow me on <a target="_blank" href="https://twitter.com/subgraphdev"><strong>Twitter</strong></a> and Hashnode to be alerted about my next article in this series. You can find me on all platforms <strong>.</strong><a class="user-mention" href="https://hashnode.com/@subgraphdev">@subgraphdev</a>.</p>
<hr />
]]></content:encoded></item><item><title><![CDATA[Building Subgraphs: A Guide to Understanding and Creating Them]]></title><description><![CDATA[Discover the world of subgraph development and learn how to create powerful subgraphs in this comprehensive guide.]]></description><link>https://subgraphdev.hashnode.dev/building-subgraphs-a-guide-to-understanding-and-creating-them</link><guid isPermaLink="true">https://subgraphdev.hashnode.dev/building-subgraphs-a-guide-to-understanding-and-creating-them</guid><category><![CDATA[Blockchain]]></category><category><![CDATA[Web3]]></category><category><![CDATA[Data Science]]></category><category><![CDATA[#thegraph]]></category><dc:creator><![CDATA[Ujjwal Thakur]]></dc:creator><pubDate>Fri, 19 May 2023 04:10:29 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1687581324862/a02673a8-b50b-4ed3-bbf6-3ccb6989e4d8.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>In the first two articles of this series, we explored the significance of data in the blockchain ecosystem and the transformative role of The Graph in facilitating data access for decentralized applications (DApps). The second article provided an overview of The Graph's decentralized network, highlighting the various participants and their collaborative efforts in maintaining network operations. In this third article, our focus will be on understanding subgraphs, setting up the required tools on our local machine, and building a basic subgraph for an ERC20 token. So let's get started.</p>
<h3 id="heading-what-is-a-subgraph">What is a subgraph?</h3>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1683521055084/6f15278f-1e1a-472f-b6e1-87b16471a208.png" alt class="image--center mx-auto" /></p>
<p><strong>Subgraph</strong> - the mysterious buzzword that's been popping up everywhere in each of our articles. But is a subgraph like a mini-map for blockchain data, or more like a GPS for developers who are lost in the blockchain wilderness? Is it a group of superheroes who band together to fight blockchain villains, who organize and analyze blockchain data? Is it like little black boxes that you can store all your blockchain data in, or more like tiny submarines that can dive deep into the data ocean and retrieve specific pieces of information? What it is? Your head might be floated with similar kinds of questions. Let's try to answer these questions.</p>
<p>Imagine you have a massive library with thousands of books, and you want to create an index to help readers find specific books more easily. To do this, you might create an index page that lists all the books in the library, grouped by author or subject, along with their location on the shelves. Each book in the library would have its entry on the index page, with all its relevant details like author name, title, genre, and publication date. In the same way, a subgraph is like an index page for a specific set of data on the blockchain. A subgraph is an open <strong>API</strong> that defines how to ingest, index and serve data. A subgraph extract data from the blockchain, process it and stores it so that it can be easily queried via <strong>GraphQL</strong>. Developers use subgraphs to feed data to the front-end interface of their application. Developers define the criteria for the index by writing a GraphQL schema, and mapping functions are created to extract and map the relevant data to the schema. The subgraph is then deployed and made available for other developers to use, making it easier for them to find and use specific data on the blockchain.</p>
<h3 id="heading-how-do-the-subgraphs-work">How do the subgraphs work?</h3>
<p>In simple words, a subgraph is a piece of code written by a developer that defines how to ingest, index and serve data. A subgraph extract data from the blockchain, process it and stores it so that it can be easily queried via <strong>GraphQL</strong>. Developers use subgraphs to feed data to the front-end interface of their application. Developers define the criteria for the index by writing a GraphQL schema, and mapping functions are created to map the relevant data to the schema. The subgraph is then deployed and made available for other developers to use, making it easier for them to find and use specific data on the blockchain.</p>
<p>At the core level, a subgraph can be considered an ETL process in the context of blockchain data. ETL stands for <strong>extraction(E), transformation(T), and loading(L)</strong> of blockchain data. Let's delve into each of them in detail:</p>
<p><img src="https://cdn.hashnode.com/res/hashnode/image/upload/v1686968337342/8eceb368-3cfa-4b79-a35a-4d86c6743003.png" alt class="image--center mx-auto" /></p>
<ol>
<li><p><strong>Extraction:</strong> The extraction process involves retrieving data from the blockchain. In the case of Ethereum, the developer typically interacts with the Ethereum client or a service provider's API to extract relevant data. This can include fetching blocks, transactions, events, and other information from the blockchain. In the case of the subgraphs, the extraction process is done by the Graph node JSON-RPC polling in which The Graph node connects to the target blockchain network and listens for events emitted by smart contracts or other blockchain entities. It utilizes specialized libraries or APIs provided by the blockchain platform to monitor the network for new transactions, blocks, or specific events.</p>
</li>
<li><p><strong>Transformation:</strong> After extracting the raw data, it needs to be transformed into a structured format that is suitable for analysis and querying. The transformation step involves cleaning, normalizing, and organizing the data. This process may include tasks such as data type conversions, merging related data, filtering irrelevant information, and applying business logic to create derived data points. The goal is to convert the extracted data into a consistent and coherent format.</p>
</li>
<li><p><strong>Loading:</strong> Once the data has been transformed, it needs to be loaded into a database for efficient storage and retrieval. The transformed data is loaded into a dedicated database, often optimized for fast and efficient querying. This step ensures that the data extracted and transformed by the subgraph is readily available for developers to query and utilize in their applications. By preloading the data into a database, the subgraph minimizes the need for real-time interactions with the blockchain, leading to faster response times and improved performance.</p>
</li>
</ol>
<p>After the database is populated with the extracted blockchain data, the Graph node exposes a query API that developers can interact with. This API provides a standardized interface to retrieve specific subsets of data from the subgraph based on developers' queries, allowing them to access the blockchain data easily and efficiently. By following these steps, a subgraph developer can extract data from the blockchain, transform it into a usable format, load it into a database, and enable users to query and interact with the data efficiently. This process is crucial for building decentralized applications on top of blockchain protocols, as it provides a structured and accessible layer for blockchain data.</p>
<h3 id="heading-technical-prerequisites-for-getting-started-with-subgraph-development">Technical Prerequisites for Getting Started with Subgraph Development</h3>
<p>Before diving into subgraph development, it's essential to have a solid understanding of the technical prerequisites. These prerequisites will equip you with the necessary knowledge and tools to effectively build subgraphs and harness the power of The Graph protocol.</p>
<ol>
<li><p>GraphQL Fundamentals: To work with subgraphs effectively, it's essential to understand GraphQL, the query language for APIs. Familiarize yourself with GraphQL concepts like schemas, types, queries, and mutations. The official <a target="_blank" href="https://graphql.org/learn/">GraphQL</a> website provides comprehensive documentation and tutorials. If</p>
<p> you prefer video tutorials, <a target="_blank" href="https://youtu.be/yqWzCV0kU_c">this</a> beginner-friendly video tutorial offers a great introduction to GraphQL.</p>
</li>
<li><p>AssemblyScript: AssemblyScript is a programming language that aims to provide a subset of the JavaScript/TypeScript syntax while allowing for direct compilation to WebAssembly. It can be seen as a variant of TypeScript that focuses on generating efficient WebAssembly code. AssemblyScript shares similarities with TypeScript, making it familiar to developers already experienced with JavaScript or TypeScript. AssemblyScript is stricter than normal TypeScript, yet provides a familiar syntax. During the subgraph development, we use AssemblyScript is used to write mapping functions. The mappings take data from a particular source and transform it into entities that are defined within our schema.</p>
</li>
<li><p>Smart Contract: Subgraphs often interact with data stored in smart contracts on the blockchain. So having a good understanding of smart contracts is always good. As a subgraph developer, we do not need to understand the whole smart contract, we need to understand just enough to be able to capture data for interaction. Key areas of focus for subgraph developers when dealing with smart contracts include:</p>
</li>
</ol>
<ul>
<li><p><strong>Finding the core contract:</strong> Decentralized applications (DApps) typically consist of multiple smart contracts that communicate with one another. In most scenarios, there is a central contract that serves as the core of the application, responsible for updating user state and managing key functionalities. Understanding and interacting with this core contract is crucial for subgraph developers when working with DApps. By focusing on the core contract, subgraph developers can effectively capture and process the relevant data and events necessary for their subgraph's functionality and integration with the DApp.</p>
</li>
<li><p><strong>State Variables</strong>: Identify the state variables that hold critical information within the contract. These variables may represent balances, ownership details, or other important data points. Consider capturing and representing them accurately in your subgraph.</p>
</li>
<li><p><strong>Contract Functions</strong>: Understand the various functions defined within the contract. Pay attention to the input parameters, return values, and the purpose of each function. This will help you identify the relevant data and events to capture in your subgraph.</p>
</li>
<li><p><strong>Event Emission</strong>: Look for events emitted by the contract. Events are crucial for capturing important state changes and updates within the contract. Analyze the event structures, including their parameters and data types.</p>
</li>
<li><p><strong>Contract Upgradability</strong>: If the contract has provisions for upgradability or proxy patterns, be aware of how it affects data access and storage.</p>
<p>  By focusing on these aspects while reading a smart contract, you can gain a comprehensive understanding of its structure, functionality, and data flow. This knowledge will guide you in accurately capturing and representing the relevant data within your subgraph.</p>
</li>
</ul>
<p>By meeting these technical prerequisites and utilizing the recommended sources, you'll be well-prepared to embark on your subgraph development journey, harnessing The Graph's capabilities to index and query data from decentralized applications.</p>
<h3 id="heading-setting-up-your-environment-for-subgraph-development">Setting up Your Environment for Subgraph Development</h3>
<p>Before we dive into building our subgraph, it's important to ensure we have the necessary tools installed on our local machine. In the next article, we will guide you through the process of developing your subgraph. But for now, let's focus on installing the required tools so that we can get started. By having the necessary tools installed, we can work efficiently and effectively. This ensures that we can build a quality subgraph without any unnecessary delays or issues. So, let's take a step back and ensure that we have everything set up correctly before we start building our subgraph.</p>
<p>Before proceeding with the installation of any other tools required for subgraph development, it is essential to check if Node.js is already installed on your local machine. If you do not already have Node.js installed on your local machine, you can download and install the latest version from the <a target="_blank" href="https://nodejs.org/en/download">official website</a>. Once installed, you can check if Node.js is properly installed on your machine by running the following command in your terminal:</p>
<pre><code class="lang-bash">node -v
</code></pre>
<p>This command will display the version of Node.js installed on your machine. If a version number is displayed, then Node.js is properly installed and configured on your machine, and you can proceed with the installation of other tools required for subgraph development.</p>
<p>If Node.js is not installed or if an error message is displayed, you will need to download and install Node.js before proceeding with subgraph development. By ensuring that Node.js is installed correctly, you can ensure that you have a stable foundation for subgraph development.</p>
<p>Once you have installed Node.js and npm, you can proceed to install the Graph CLI tool. This tool enables developers to create and manage subgraphs easily. You can install the Graph CLI by running the following command in your terminal:</p>
<p>To install <strong>The Graph CLI</strong> with npm:</p>
<pre><code class="lang-bash">npm install -g @graphprotocol/graph-cli
</code></pre>
<p>To install <strong>The Graph CLI</strong> with yarn:</p>
<pre><code class="lang-bash">yarn global add @graphprotocol/graph-cli
</code></pre>
<p>To check if The Graph CLI is installed on your local machine, you can run the following command in your terminal.</p>
<pre><code class="lang-bash">graph --version
</code></pre>
<p>By checking if The Graph CLI is installed on your local machine, you can ensure that you have access to the necessary tools for subgraph development. The Graph CLI is an essential tool for subgraph development as it allows you to create, deploy and manage subgraphs on The Graph Network.</p>
<p>we have successfully set up our local machines and installed all the necessary tools required to kickstart our subgraph development process. With everything in place, we are now ready to delve into the practical aspects of subgraph development. In our next article, we will embark on the exciting task of building a subgraph for an ERC-721 token. This hands-on experience will allow us to apply the knowledge we've gained and witness firsthand the power and versatility of subgraphs in capturing and processing data from smart contracts. Stay tuned for an immersive and insightful exploration into the world of subgraph development!</p>
<hr />
<p>If you enjoyed reading my blog, I would greatly appreciate it if you could take a moment to show your support by liking and commenting on it. Your feedback and engagement are incredibly valuable to me and will help me to continue improving and creating content that is informative and engaging. I'll be making a bit of content for The Graph in the coming days. Follow me on Twitter and Hashnode to be alerted about my next article in this series. In the next article, we build our first subgraph. You can find me on all platforms @subgraphdev</p>
<hr />
]]></content:encoded></item><item><title><![CDATA[Unleashing the Power of Blockchain Data with The Graph]]></title><description><![CDATA[Data is the lifeline of the modern world. With the incredible amount of information being generated every day, it is more important than ever to use data…]]></description><link>https://subgraphdev.hashnode.dev/unleashing-the-power-of-blockchain-data-with-the-graph</link><guid isPermaLink="true">https://subgraphdev.hashnode.dev/unleashing-the-power-of-blockchain-data-with-the-graph</guid><category><![CDATA[Web3]]></category><category><![CDATA[Blockchain]]></category><category><![CDATA[indexing]]></category><category><![CDATA[Querying]]></category><category><![CDATA[The Graph]]></category><dc:creator><![CDATA[Ujjwal Thakur]]></dc:creator><pubDate>Thu, 04 May 2023 03:37:09 GMT</pubDate><enclosure url="https://cdn.hashnode.com/res/hashnode/image/upload/v1687581728819/c46100a1-7cee-491b-a753-5943f6120c86.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<h2 id="heading-importance-of-data-in-the-modern-world">Importance of Data in the Modern World</h2>
<p>Data is the lifeline of the modern world. With the incredible amount of information being generated every day, it is more important than ever to use data to make informed decisions. As the famous quote by Clive Humby says, "Data is the new oil." Just as oil fueled the industrial revolution, data is fueling the digital revolution. Businesses are using data to gain insights into customer behavior, optimize their operations, and develop new products and services. Researchers are using data to make breakthroughs in the fields of medicine and environmental science. Governments are using data to improve public services and make informed policy decisions. Data is being used in sports to analyze athlete performance, track player statistics, and reduce the risk of injury. Teams are also using data to scout and recruit new talent, engage with fans, and enhance the viewing experience through advanced statistics and immersive technologies. Overall, data is improving our world by enabling us to make more informed decisions, develop new solutions to complex problems, and improve the lives of people around the world.</p>
<p>In the modern era, the availability and accessibility of information have had a profound impact on society and individuals' experiences. With the rise of the internet and other digital technologies, people now have access to an unprecedented amount of information at their fingertips, which has transformed the way we live, work, and interact with one another. The ability to store, find, manage, and share information has become essential for many aspects of modern life, from education and research to business and social networking. As such, the structure of society and the experiences of individuals are increasingly shaped by how we create, consume, and interact with information. The internet has become a ubiquitous source of information for people around the world. With millions of websites and online resources covering a vast range of topics, the internet has made it easier than ever before to access information on virtually any subject. The internet has transformed the way we create, manage and share information.</p>
<h3 id="heading-blockchain-and-the-next-gen-of-internet">Blockchain and the Next Gen. of Internet</h3>
<p>The internet of today has a client-server architecture. Client-server architecture is a way of organizing computer programs or applications where one program (the client) makes requests for information or services from another program (the server). This has been a fundamental part of the Internet and computing infrastructure for decades and has enabled many useful and valuable applications and services. One of the main problems with client-server architecture is that it is inherently centralized, meaning that it relies on a central server or authority to manage and control access to the system. This centralization creates a single point of failure and vulnerability, which can be exploited by hackers, governments, or other bad actors to disrupt or control the system. This centralization can also result in power imbalances and inequalities, as those who control the servers or the data can exercise disproportionate control and influence over the system and its users. While client-server architecture has enabled many valuable applications and services, it also has certain limitations and drawbacks that can have negative impacts on society, particularly in terms of centralization, privacy, and security. As a result, there is a growing need to explore and develop more decentralized and distributed architectures, such as peer-to-peer networks, that can provide greater resilience, privacy, and security. <strong>Blockchain</strong> technology offers a potential solution to these problems, as it is a decentralized, distributed system that eliminates the need for a central authority to manage and control access to the system.</p>
<p><strong><em>Blockchain</em></strong> is a decentralized, distributed ledger technology that allows for secure, transparent, and tamper-proof record-keeping of transactions and data. In a blockchain system, transactions or data are recorded in blocks, which are linked together in a chronological chain. Each block contains a cryptographic hash of the previous block, creating an immutable and secure record of all the transactions or data in the chain. One of the key benefits of blockchain technology is that it eliminates the need for a centralized server. Instead, transactions or data are verified and validated by a distributed network of nodes. This decentralized architecture eliminates the single point of failure and vulnerability that comes with centralized servers, making the system more resilient and secure and, it enables greater privacy and control for users, as they can maintain ownership and control over their data and identities without relying on centralized intermediaries.</p>
<p>Blockchain technology has seen a surge in popularity over the past few years, leading to a large number of applications being built on this technology. Applications built on top of blockchains are called <strong>Dapps or Decentralized Applications</strong>. Whenever users interact with these Dapps, every interaction with the Dapp is recorded on the blockchain. The large amounts of data generated by Dapps can be very useful for a variety of purposes, from improving transparency and accountability to enabling new forms of innovation and analysis. While this increase in data may seem overwhelming at first, it can be very useful in several ways. First, it provides a transparent and auditable record of all transactions and interactions on the Dapp, which can improve trust and accountability. Second, this data can be used for analytics and insights, enabling developers and stakeholders to understand how the Dapp is being used and to identify areas for improvement. Third, the data generated by Dapps can be used to create new applications and services, leveraging the power of blockchain technology to enable innovative new use cases.</p>
<h3 id="heading-blockchain-data-andamp-its-accessibility-issues">Blockchain data &amp; its accessibility issues</h3>
<p>Blockchain data has enormous potential for innovation and value creation, but it has several challenges around its accessibility. Every Dapps interact with a <strong>smart contract</strong>, Smart contracts are self-executing computer programs that are deployed on a blockchain network. They are designed to enforce the terms of a contract by automatically executing predefined rules and conditions encoded in the contract's code. Smart contracts can be used to store data, transfer assets, and perform other operations within a decentralized network. Performing basic read operations in a smart contract is relatively easy. For example, In the case of Bored Ape Yacht Club, we can perform basic read operations on smart contracts like getting the owner of a certain Ape, getting the content URI of an Ape based on their ID, or the total supply, as these read operations are programmed directly into the smart contract, if we wanted to query for apes that are owned by a certain address, and filter by one of its characteristics, we would not be able to get that information by interacting directly with the contract itself. We can see that the more complex operations, such as filtering and aggregating data, are much more difficult to achieve. This is because, the storage capacity of the blockchain is limited, which can be a challenge when working with large datasets. Since each node in the network must store a copy of the entire blockchain, storing large amounts of data can become impractical, especially when considering the cost of storage and processing power.</p>
<p>Earlier developers used to get around this gap in functionality by building centralized indexing servers. Developers have to run their full node to the data stored on the blockchain. These servers pull data from Ethereum, store it in a database, and expose it over an API. This is a very resource-intensive and time-consuming task. It is also not cost-efficient and can be a complex and error-prone task. This is brittle as users need to trust these teams to continue to operate these servers correctly. The projects could go out of business, modify the data for strategic reasons, get acquired, or simply make mistakes. Suddenly we’re not far from where we started with web2. How can developers overcome the limitations of performing complex operations on data stored on the blockchain, without relying on resource-intensive and centralized indexing servers?</p>
<h3 id="heading-solving-the-problem-of-blockchain-data-accessibility-with-the-graph">Solving the problem of blockchain data accessibility with The Graph</h3>
<p>A solution to this problem is to use decentralized indexing and querying services, this is where <strong>The Graph</strong> comes to rescue us. The Graph is a decentralized protocol for indexing and querying data from blockchain networks, including Ethereum, IPFS, Arweave and more. It enables developers to build decentralized applications that can access and query blockchain data without relying on centralized intermediaries or resource-intensive processes. The Graph provides a marketplace of <strong>subgraphs</strong>, which are open APIs that index and expose specific data from the blockchain. Developers can build their subgraphs, which can be used to index and query data for specific use cases. The Graph enables developers to leverage the power of the blockchain network without having to run a full node or rely on centralized indexing servers.</p>
<p>When a developer wants to query data from a blockchain network, they can use The Graph's Query API to request the data they need. The Query API sends the request to the appropriate subgraph, which has already indexed and stored the data from the blockchain network. The subgraph then processes the query and returns the requested data to the developer. The Graph uses a decentralized network of indexers to index and stores the data from the blockchain network. Indexers are nodes that run The Graph's software and provide indexing services to the network. They compete to provide the best indexing service, which ensures that the data is accurate and up-to-date. Developers can create their subgraphs to index and query specific data from the blockchain network. The Graph provides a set of tools and libraries to make it easier for developers to create subgraphs. The Graph also provides a marketplace where developers can discover and use existing subgraphs that have been created by other developers. The marketplace allows developers to find subgraphs that are relevant to their use case, which can save time and effort in building their decentralized applications.</p>
<p>We can see how The Graph is an essential part of the Web3 tech stack as it provides a decentralized indexing protocol for querying and indexing data from blockchain networks. By enabling developers to efficiently search, retrieve and analyze data from various blockchain networks, The Graph makes it easier to build decentralized applications with rich user interfaces and data-driven functionalities. Without The Graph, developers would need to manually parse through large amounts of raw blockchain data, which can be time-consuming and resource-intensive. The Graph's decentralized indexing protocol simplifies this process, making it easier for developers to access and use blockchain data in their applications.</p>
<hr />
<p>If you enjoyed reading my blog, I would greatly appreciate it if you could take a moment to show your support by liking and commenting on it. Your feedback and engagement are incredibly valuable to me and will help me to continue improving and creating content that is informative and engaging. I'll be making a bit of content for The Graph in the coming days. Follow me on <a target="_blank" href="https://twitter.com/subgraphdev">Twitter</a> and Hashnode to be alerted about my next article in this series. In the next article, we will explore The Graph Network and its participant's role. You can find me on all platforms <strong>.@subgraphdev</strong>.</p>
<hr />
]]></content:encoded></item></channel></rss>