TheDinarian
News • Business • Investing & Finance
Lightweight and Flexible Data Access for Algorand
March 24, 2023
post photo preview

Algorand has released a new tool for blockchain data access: Conduit. Conduit is a modular plugin-based tool and a powerful upgrade to the one-size-fits-all Indexer. Conduit allows dapps to get exactly the data they need in an affordable deployment.

Useful, but bulky: the Indexer

The Indexer is a ready-to-go open-source tool that pulls the data from the blockchain, stores it in a database, and offers an API to serve that data. The existence of the Indexer has been a significant boon for the Algorand ecosystem, allowing anybody to easily read the Algorand blockchain.

However, the Indexer has historically had one major drawback: it is expensive to run. There are two main reasons for this:

  1. Running an Indexer requires also running an archival node that stores every block since the beginning of the blockchain.
  2. The Indexer collects the entire blockchain history (every transaction since block zero) in a Postgres database.

These facts make the Indexer a multi-Terabyte deployment. A typical Indexer requires a number of expensive resources, and these multiply for production deployments needing redundancy, load-balancing, and covering multiple regions.

The scale of the Indexer also makes it slow to initialize, and only capable of serving the specific queries for which it is indexed. As the Algorand blockchain has grown, it has become impractical for smaller projects to maintain their own Indexers.

Consequently, the ecosystem mostly relies on a few API/data providers. These providers run Indexers and charge dapps for their API calls. This is more economical and practical than each group running their own Indexer, but it presents other inflexibilities.

Dapps should have an accessible option to own their own data access infrastructure. This is what Conduit was built for.

Conduit, the Basics

Conduit is a new solution with several major advantages:

  1. Conduit does not require running an archival algod node.
  2. Conduit lets users filter incoming blockchain data, allowing them to collect strictly the data they need for their applications.
  3. Conduit offers a data pruning feature that allows users to automatically delete old transactions when pruning is enabled.
  4. With Conduit, users can build custom data exporters that use the data destination of their choice.
  5. Conduit is designed as an extensible plugin architecture. Any community-contributed plugin can be integrated by anyone.

Conduit allows users to configure their own data pipelines for filtering, aggregation, and storage of transactions and accounts on any Algorand network.

A Conduit pipeline is composed of an importer, optional processor(s), and exporter plugins. Along with the Conduit release, the following noteworthy plugins are made available.

  • Algod importer — fetches blocks from an algod REST API.
  • Filter processor — filters data based on transaction fields.
  • Postgres exporter — writes the data to a Postgres database.
  • File writer exporter — writes the data to a file.

Configuring a Conduit pipeline requires defining which plugins to use, and if necessary, configuring the plugins. For example, the filter processor requires a definition of what to filter.

This is best demonstrated with an example. See a basic walkthrough here.

Conduit’s Filter Processor

The filter processor is a key new feature introduced with Conduit. It allows users to filter the transaction data based on any transaction field — transaction type, app ID, asset ID, sender, receiver, amount, etc. These filters can also be combined.

Since many transactions are submitted as grouped transactions, the filter processor allows users to choose whether or not to include the entire transaction group when the filter conditions are met.

The filter processor will always include inner transactions for transactions that match the specified filter conditions.

Full details on the filter processor are here.

A New Node Configuration for Conduit: Follow Mode

Conduit is used to track data from the blockchain and make it available to the off-chain world. Every time a new block is created on-chain, Conduit is informed about every change to every piece of state since the prior block, such as new accounts created, app states updated, boxes deleted, etc.

Some dapps use an object called the ApplyData to track some kinds of state changes, however this approach is technically limited. Not all changes are reflected in this object, and ApplyData are only cached for 4 rounds on non-archival nodes, meaning that delayed handling of ApplyData updates for more than 15 or so seconds will result in an unrecoverable state error.

The old Indexer architecture solved these challenges by requiring access to an archival algod node. Indexer used a “local ledger” to track the state changes from round to round, and thus avoided the incomplete ApplyData object. The drawback of this design is the need for an expensive archival node.

Conduit instead requires access to a node in a new lightweight “follow mode” configuration which replaces the need for the archival configuration. Conduit can pause and unpause this node’s round updates as required. The pause functionality ensures that the Conduit process will not miss out on any blockchain state updates. Conduit also makes use of a new “state delta” endpoint introduced in the node to eliminate the requirement for a large local ledger.

A node with follow mode enabled cannot participate in consensus, as votes based on paused state information would be rejected. Similarly, submitting transactions to such a node is not possible, as acceptance based on paused, outdated state information might be judged invalid by the rest of the blockchain.

Conduit as an Extensible Tool

Focusing on open-source principles and decentralization, Conduit’s design encourages custom-built solutions, setting it apart from the Indexer. In our initial release, we encourage new plugin submissions via PRs to the Conduit repository. We aim for the plugin framework to inspire community involvement, allowing everyone to benefit from shared efforts. Currently, we’re engaging the community to identify optimal management for externally-supported plugins long-term (join the conversation on Discord #conduit channel!)

We have already seen the development of a Kafka plugin by a community member (Iridium#4127 on Discord), who has this to say about Conduit:

“… it [Conduit] allows [you] to choose your … targeted product (e.g. Kafka) to quickly build a plugin and let the data flow. Mainly it’s just importing the correct library — configure your connection and use the library to send messages to your system. Receiving is already handled by Conduit.”

Comparing Deployments: Legacy Indexer vs. Conduit Architecture

Indexer, legacy architecture

  • Requires an archival algod node, which requires at least 1.1 TB of storage.
  • Requires a Postgres database with full historical data, or 1.5 TB of storage.

Source for the above: howbigisalgorand.com

Conduit architecture

  • Requires a node with “follow mode” enabled, which requires 40 GB of storage (like other non-archival nodes).
  • Conduit can use a Postgres database, or a different data store. The user can store full historical data, or a subset. This is at most 1.5 TB if storing the full history, and could be as little as a few GB.

The costs of these deployments will vary depending on whether users are self-hosted or using cloud providers (and vary greatly by provider). However, the storage costs will be strictly less for a Conduit-backed deployment.

Note that storage will likely be the major cost factor, and bandwidth and compute requirements are similar across both architectures.

Continued Indexer Support

We are continuing to support the existing releases of Indexer which run its old architecture (using the archival node) at this time. If users would like to continue using the Indexer but also want to save costs by removing the need for an archival node, they have the option to run an Indexer backed by Conduit. The Indexer interface remains the same. See our migration guide here.

Conduit Builds Better Apps

Conduit was designed to be flexible and extensible, intended to allow developers to build whatever data solution fits their needs. As such, Conduit has countless applications.

Want to run Conduit to support your dapp reporting needs?

Want to extend the Indexer API?

Want to power an event-driven system based on on-chain events?

Want to scale your API Provider service by using CockroachDB?

Want to dump everything to S3 and just query that?

The limitations imposed by the Indexer’s rigidity no longer apply. While Conduit doesn’t provide everything for free, it offers users the flexibility to build what they need.

Link

community logo
Join the TheDinarian Community
To read more articles like this, sign up and join my community today
0
What else you may like…
Videos
Podcasts
Posts
Articles
Brad Garlinghouse In Washington 🚀

It’s time for a fair and open level playing field.

Under Gary Gensler it was quite the opposite.

  • Brad Garlinghouse
    July 9, 2025
00:01:56
More Of The Same...l

🚨 JUST IN: Patriot Tom Fitton, who has been fighting DOJ and FBI to release documents for years, has practically thrown in the towel.

👉 "The justice department and the FBI are irredeemably compromised and corrupted.
The leadership needs to understand that and act accordingly." ~Tom Fitton

00:01:30
Christine Lagarde just gave Ripple & Circle A Shoutout!
00:00:44
👉 Coinbase just launched an AI agent for Crypto Trading

Custom AI assistants that print money in your sleep? 🔜

The future of Crypto x AI is about to go crazy.

👉 Here’s what you need to know:

💠 'Based Agent' enables creation of custom AI agents
💠 Users set up personalized agents in < 3 minutes
💠 Equipped w/ crypto wallet and on-chain functions
💠 Capable of completing trades, swaps, and staking
💠 Integrates with Coinbase’s SDK, OpenAI, & Replit

👉 What this means for the future of Crypto:

1. Open Access: Democratized access to advanced trading
2. Automated Txns: Complex trades + streamlined on-chain activity
3. AI Dominance: Est ~80% of crypto 👉txns done by AI agents by 2025

🚨 I personally wouldn't bet against Brian Armstrong and Jesse Pollak.

👉 Coinbase just launched an AI agent for Crypto Trading

same for: https://coinmarketcap.com/community/articles/686e68f5d405956445e039ff/

🚨 Ripple Picks BNY Mellon to Back RLUSD Stablecoin Amid Major Surge 🚨

Ripple has selected BNY Mellon, one of the world’s largest and most trusted financial institutions, to serve as the primary custodian for its RLUSD stablecoin. This decision comes as RLUSD experiences a surge in demand, highlighting growing institutional interest in Ripple’s stablecoin offering.

🔹 Institutional Partnership

🔹 BNY Mellon will safeguard the reserves backing RLUSD, ensuring transparency, security, and regulatory compliance for the stablecoin.

🔹 This partnership is designed to build trust with both institutional and retail users by leveraging BNY Mellon’s expertise in asset custody.

🔹 RLUSD’s Rapid Growth

🔹 RLUSD has seen a significant increase in adoption, reflecting confidence in Ripple’s approach to stablecoins and its commitment to compliance and transparency.

🔹 The collaboration with BNY ...

From Wall Street to Web3: Building Tomorrow’s Digital Asset Markets

COMMITTEE ON BANKING, HOUSING, AND URBAN AFFAIRS will meet in OPEN SESSION, HYBRID FORMAT to conduct a hearing entitled, “From Wall Street to Web3: Building Tomorrow’s Digital Asset Markets.” The witnesses will be: The Honorable Summer Mersinger, CEO, Blockchain Association; Mr. Jonathan Levin, CEO, Chainalysis; Mr. Dan Robinson, General Partner, Paradigm; Mr. Brad Garlinghouse, CEO, Ripple; The Honorable Timothy Massad, Research Fellow and Director of Digital Assets Policy Project of the Mossavar-Rahmani Center for Business and Government, Kennedy School of Government at Harvard University, former CFTC Chairman; and Mr. Richard Painter, S. Walter Richey Professor of Corporate Law, University of Minnesota Law School, former Associate Counsel to the President and chief White House ethics lawyer.

https://www.banking.senate.gov/hearings/from-wall-street-to-web3-building-tomorrows-digital-asset-markets

‼️XRP ETF INFOGRAPHIC REVEALS AMERICAN EXPRESS UTILIZES XRP‼️

“A well-known company that uses XRP is American Express, which leverages RippleNet to enable realtime cross-border payments for corporate clients.

Through its partnership with Ripple, American Express uses XRP indirectlyvia Ripple's infrastructure to facilitate faster and more transparent transactions between the U.S. and international markets, helping businesses move money efficiently and reduce settlement times from days to seconds.”✅

OP: Smqkedqg

post photo preview
post photo preview
Musk Turns On Starlink to Save Iranians from Regime’s Internet Crackdown

Elon Musk, the world’s richest man and a visionary behind SpaceX, has flipped the switch on Starlink, delivering internet to Iranians amid a brutal regime crackdown.

This move comes on the heels of Israeli strikes targeting Iran’s nuclear facilities, as the Islamic Republic cuts off online access.

The former Department of Government Efficiency chief activated Starlink satellite internet service for Iranians on Saturday following the Islamic Republic's decision to impose nationwide internet restrictions.

As the Jerusalem Post reports, that the Islamic Republic’s Communications Ministry announced the move, stating, "In view of the special conditions of the country, temporary restrictions have been imposed on the country’s internet."

This action followed a series of Israeli attacks on Iranian targets.

Starlink, a SpaceX-developed satellite constellation, provides high-speed internet to regions with limited connectivity, such as remote areas or conflict zones.

Elizabeth MacDonald, a Fox News contributor, highlighted its impact, noting, "Elon Musk turning on Starlink for Iran in 2022 was a game changer. Starlink connects directly to SpaceX satellites, bypassing Iran’s ground infrastructure. That means even during government-imposed shutdowns or censorship, users can still get online, and reportedly more than 100,000 inside Iran are doing that."

During the 2022 "Woman, Life, Freedom" protests, Starlink enabled Iranians to communicate and share footage globally despite network blackouts," she added.

MacDonald also mentioned ongoing tests of "direct-to-cell" capabilities, which could allow smartphone connections without a dish, potentially expanding access and supporting free expression and protest coordination.

Musk confirmed the activation, noting on Saturday, "The beams are on."

This follows the regime’s internet shutdowns, which were triggered by Israeli military actions.

Adding to the tension, Israeli Prime Minister Benjamin Netanyahu addressed the Iranian people on Friday, urging resistance against the regime.

"Israel's fight is not against the Iranian people. Our fight is against the murderous Islamic regime that oppresses and impoverishes you,” he said.

Meanwhile, Reza Pahlavi, the exiled son of Iran’s last monarch, called on military and security forces to abandon the regime, accusing Supreme Leader Ayatollah Ali Khamenei in a Persian-language social media post of forcing Iranians into an unwanted war.

Starlink has been a beacon in other crises. Beyond Iran, Musk has leveraged Starlink to assist people during natural disasters and conflicts.

In the wake of hurricanes and earthquakes, Starlink has provided critical internet access to affected communities, enabling emergency communications and coordination.

Similarly, during the Ukraine-Russia conflict, Musk activated Starlink to support Ukrainian forces and civilians, ensuring they could maintain contact and access vital information under dire circumstances.

The genius entrepreneur, is throwing a lifeline to the oppressed in Iran, and the libs can’t stand it.

Conservative talk show host Mark Levin praised Musk’s action, reposting a message stating that Starlink would "reconnect the Iranian people with the internet and put the final nail in the coffin of the Iranian regime."

"God bless you, Elon. The Starlink beams are on in Iran!" Levin wrote.

Musk, who recently stepped down from leading the DOGE in the Trump administration, has apologized to President Trump for past criticisms, including his stance on the One Big Beautiful Bill.

Source

🙏 Donations Accepted 🙏

If you find value in my content, consider showing your support via:

💳 PayPal: 
1) Simply scan the QR code below 📲
2) or visit https://www.paypal.me/thedinarian

🔗 Crypto – Support via Coinbase Wallet to: [email protected]

Or Buy me a coffee: https://buymeacoffee.com/thedinarian

Your generosity keeps this mission alive, for all! Namasté 🙏 Crypto Michael ⚡  The Dinarian

Read full Article
post photo preview
GENIUS Act lets State banks conduct some business nationwide. Regulators object

The Senate passed the GENIUS Act for stablecoins last week, but significant work remains before it becomes law. The House has a different bill, the STABLE Act, with notable differences that must be reconciled. State banking regulators have raised strong objections to a provision in the GENIUS Act that would allow state banks to operate nationwide without authorization from host states or a federal regulator.

The controversial clause permits a state bank with a regulated stablecoin subsidiary to provide money transmitter and custodial services in any other state. While host states can impose consumer protection laws, they cannot require the usual authorization and oversight typically needed for out-of-state banking operations.

The Conference of State Bank Supervisors welcomed some changes in the GENIUS Act but remains adamantly opposed to this particular provision. In a statement, CSBS said:

“Critical changes must be made during House consideration of the legislation to prevent unintended consequences and further mitigate financial stability risks. CSBS remains concerned with the dramatic and unsupported expansion of the authority of uninsured banks to conduct money transmission or custody activities nationwide without the approval or oversight of host state supervisors (Sec. 16(d)).”

The National Conference of State Legislatures expressed similar concerns in early June, stating:

“We urge you to oppose Section 16(d) and support state authority to regulate financial services in a manner that reflects local conditions, priorities and risk tolerances. Preserving the dual banking system and respecting state autonomy is essential to the safety, soundness and diversity of our nation’s financial sector.”

Evolution of nationwide authorization

Section 16 addresses several issues beyond stablecoins, including preventing a recurrence of the SEC’s SAB 121, which forced crypto assets held in custody onto balance sheets. However, the nationwide authorization subsection was added after the legislation cleared the Senate Banking Committee, with two significant modifications since then.

Originally, the provision applied only to special bank charters like Wyoming’s Special Purpose Depository Institutions or Connecticut’s Innovation Banks. Examples include crypto-focused Custodia Bank and crypto exchange Kraken in Wyoming, plus traditional finance player Fnality US in Connecticut. Recently the scope was expanded to cover most state chartered banks with stablecoin subsidiaries, possibly due to concerns about competitive advantages.

Simultaneously, the clause was substantially tightened. The initial version allowed state chartered banks to provide money transmission and custody services nationwide for any type of asset, which would include cryptocurrencies. Now these activities can only be conducted by the stablecoin subsidiary, and while Section 16(d) doesn’t explicitly limit services to stablecoins, the GENIUS Act currently restricts issuers to stablecoin related activities.

However, the House STABLE Act takes a more permissive approach, allowing regulators to decide which non-stablecoin activities are permitted. If the House version prevails in reconciliation, it could result in a significant expansion of allowed nationwide banking activities beyond stablecoins.

Is it that bad?

As originally drafted, the clause seemed overly permissive.

The amended clause makes sense for stablecoin issuers. They want to have a single regulator and be able to provide the stablecoin services throughout the United States. But it also leans into the perception outside of crypto that this is just another form of regulatory arbitrage.

The controversy over Section 16(d) reflects concerns about creating a regulatory gap that allows banks to operate interstate without the oversight typically required from either federal or state authorities. As the two Congressional chambers work toward reconciliation, lawmakers must decide whether stablecoin legislation should include provisions that effectively reduce traditional banking oversight requirements.

Source

🙏 Donations Accepted 🙏

If you find value in my content, consider showing your support via:

💳 PayPal: 
1) Simply scan the QR code below 📲
2) or visit https://www.paypal.me/thedinarian

🔗 Crypto – Support via Coinbase Wallet to: [email protected]

Or Buy me a coffee: https://buymeacoffee.com/thedinarian

Your generosity keeps this mission alive, for all! Namasté 🙏 Crypto Michael ⚡  The Dinarian

Read full Article
post photo preview
Dubai regulator VARA classifies RWA issuance as licensed activity
Virtual Asset Regulatory Authority (VARA) leads global regulatory framework - makes RWA issuance licensed activity in Dubai.

Real-world assets (RWAs) issuance is now licensed activity in Dubai.

~ Actual law.
~ Not a legal gray zone.
~ Not a whitepaper fantasy.

RWA issuance and listing on secondary markets is defined under binding crypto regulation.

It’s execution by Dubai.

Irina Heaver explained:

“RWA issuance is no longer theoretical. It’s now a regulatory reality.”

VARA defined:

- RWAs are classified as Asset-Referenced Virtual Assets (ARVAs)

- Secondary market trading is permitted under VARA license

- Issuers need capital, audits, and legal disclosures

- Regulated broker-dealers and exchanges can now onboard and trade them

This closes the gap that killed STOs in 2018.

No more tokenization without venues.
No more assets without liquidity.

UAE is doing what Switzerland, Singapore, and Europe still haven’t:

Creating enforceable frameworks for RWA tokenization that actually work.

Matthew White, CEO of VARA, said it perfectly:

“Tokenization will redefine global finance in 2025.”

He’s not exaggerating.

$500B+ market predicted next year.

And the UAE just gave it legal rails.

~Real estate.
~Private credit.
~Shariah-compliant products.

Everything is in play.

This is how you turn hype into infrastructure.

What Dubai is doing now is 3 years ahead of everyone else.

Founders, investors, ecosystem builders:

You want to build real-world assets onchain.

Don’t waste another year waiting for clarity.

Come to Dubai.

It’s already here.

 

Source

🙏 Donations Accepted 🙏

If you find value in my content, consider showing your support via:

💳 PayPal: 
1) Simply scan the QR code below 📲
2) or visit https://www.paypal.me/thedinarian

🔗 Crypto – Support via Coinbase Wallet to: [email protected]

Or Buy me a coffee: https://buymeacoffee.com/thedinarian

Your generosity keeps this mission alive, for all! Namasté 🙏 Crypto Michael ⚡  The Dinarian

 

Read full Article
See More
Available on mobile and TV devices
google store google store app store app store
google store google store app tv store app tv store amazon store amazon store roku store roku store
Powered by Locals