GRTiQ Podcast: 26 Jim Cousins and Gary Morris

00:01
The following podcast is for informational purposes only the contents of this podcast do not constitute tax legal or investment advice, take responsibility for your own decisions, consult with the proper professionals and do your own research.

00:22
You know, this is a relatively new paradigm that we’re working in, right? There’s no you know, I often talk when I’m talking to others about this, I talk about the fact that there’s no HR function to sort of glue everything together, right? It’s, it’s on all of us to help make that sort of thing happen, you know.

01:08
Welcome to the GRTiQ Podcast. Today I’m speaking with Jim Cousins and Gary Morris. Jim and Gary are both members of The Graph Council, and are well respected voices not only in the Indexer community, but The Graph ecosystem at large. Both Jim and Gary have been featured on the podcast before and are familiar voices within the community. In addition to being members of The Graph Council, Jim operates his own Indexer operation called WaveFive. Gary also operates his own Indexer operation 0xFury, while also working on the Staking Facilities team. I wanted to have both Jim and Gary back onto the podcast to discuss some timely topics related to things like the state of the Indexer community at The Graph. How the Indexer community is working alongside StreamingFast and Figment and a host of other important topics, including their opinions on the recent community discussions surrounding keeping stake decentralized.

So Gary, Jim, I want to start with a discussion about these recent core developer announcements. One was StreamingFast, and one with Figment. What can you tell listeners about the importance of these new core development partnerships, and how you’re experiencing those or seeing those from the Indexer perspective?

02:27
So I think, you know, at the highest level, the core dev announcements: StreamingFast and Figment grants is, you know, is signals some serious commitment from The Graph Foundation around making the vision real. So, you know, we’ve got hosted right now, we are migrating to main net eventually, sunsetting hosting. But along the way, there’s a number of technical challenges that need to be met, right, and we want to meet them quickly. And that’s no mean feat. Hiring in this space is, you know, it’s probably one of the hardest spaces to hire the best people in, because generally speaking, the people who are already in this space, they’re doing well, and they’re aligned in the way that they want to be aligned to already. So getting people to move on, you know, as to other opportunities is quite difficult. So by incentivizing entire teams or portions of teams to come and contribute to The Graph node into the indexing software, it’s a big deal, right? There’s a, there’s a big difference between an having an existing team with a bunch of IP and experience in this data space that we’re all in, and maybe hiring a few engineers to get them, you know, and getting them up to speed. So this is a huge commitment from the foundation over the next five years. And very specifically to what you know, you may have heard Yaniv use the term ‘Blitzscale’. That’s what these teams are going to be working on. Right, they’re going to be taking their technology and applying it to The Graph node quickly. The piece that still to, you know, to really be communicated by all of us really, at all levels within the foundation, and within the core dev. teams is exactly what the technical part of that vision looks like. And that’s what we are working on right now, internally. So the core dev. teams are, you know, they’re all pulling for feedback from all the different communities, for example, StreamingFast, they have a huge amount of indexing performance technology, some really quite profound improvements in the indexing performance space. And they’re essentially putting their what we call RFCs, or Requests For Comments out onto the forum. So this is like a pre proposal. This is usually the term we use in the forums. They’re putting them out onto the forums, showing us what it is that they have today and where they want to go and how they want to integrate these new tools and intellectual property into The Graph software stack. And as we go through that process, and we actually start to solidify a roadmap, then the actual vision that these core dev. teams are bringing will start to become clearer. So I guess that’s my summary of what this brings to. I mean, it’s not just the Indexer community, right? It’s the it’s all communities, all stakeholders.

05:21
And like Jim said, it’s not just about, you know, getting more hands on deck, these are teams that are, you know, they already have the experience that we’re looking for. When one of these acquisitions was announced, someone said to me, why didn’t they just use maybe a fraction of that money to salary a bunch of dev’s, but you need to buy a package deal like that can’t just clump together a bunch of dev’s, and hope that they’re also aligned with your vision, regardless of their skills. So the foundation is basically acquiring the whole package deal in one sweep as it were. And that’s something that we really need to get, hit the ground running and get moving quickly. And that’s the other thing, if, if the foundation didn’t look to acquire this talent, you know, this could have presented extra competition, or they could have even been acquired by, you know, potential competitors of what we’re trying to build here. So it’s not just the onboarding of those that have the skills, talent experience and the alignment with us. But it’s also basically, you know, from a business development perspective, it’s cutting away the potential for competition to basically take our milkshake as it were.

06:33

And Jim, this was something that really hit me in Indexer office hours that you did with Figment. And I posted it on Twitter, and a lot of people responded. And it was this interesting perspective that, you know, Figment is a very large enterprise, it’s doing a lot of cool things on like, 35, different Proof of Stake networks. And so the team, their capabilities, it runs very deep. And yet, they kind of seems pushed aside their own interests, and partnered with The Graph, because of this vision for Web 3, was that’s something you picked up on, too?

07:06
Yeah, we were lucky to have Andy, Andy Cronk, the Chief Product officer from Figment on that office hour session, and Andy was very pragmatic about, you know, his viewpoint on the grant and Figments, existing server, you know, their existing services, profit making services, and how all of these services that are Figment independent services, you know, with the existing revenue streams, you know, they’re essentially sacrificing those revenue streams in order to bring them over to The Graph, which is obviously it’s not just a matter of flicking a switch, there’s a lot of work to be done. So it really goes to show that at the Foundation, obviously, we think a lot about competition and things like that. But that’s exactly the exact same thing that you know, these potential competitors are also thinking about, you know, Andy told a very good narrative around, you know, how they assessed where they could have gone in terms of being a competitor, just continuing, you know, what would their they’ve currently got going in terms of their data hub product, or joining a much bigger ecosystem, right. And I think this, you know, we sometimes use this term community M&A. What it really means to me is not really about a financial M&A or equity, M&A, but it’s about a alignment, right? And, and community M&A, right? So, and it may be even end customer M&A. So you’re actually taking, you’re potentially delivering the same service, but you’re now part of a much bigger ecosystem. And you’re sharing your technology. And you’re also seeing the benefit of technology shared by other core dev. teams. So yeah, I very much saw that communicated very well by Andy in the office hours. And it was a topic of, of discussion when the due diligence was being done as well, obviously.

08:50
Well, I want to talk a little bit about this commitment to decentralization, because it seems like if you go to the most common denominator between the shared vision and what people are trying to accomplish in the space, it’s this process towards decentralization. And I want to take a minute and it’s a little bit educational Indexer 101. But for listeners that don’t appreciate or understand how Indexers at The Graph, enable decentralization. Gary, would you take a minute and just kind of describe in a way that a non-technical person can understand how it is that The Graph is enabling decentralization?

09:25
Yeah, so basically, for a number of years now, I believe The Graph has been running their hosted service. And we’re basically forging a path towards moving all of the traffic and end users as it were projects onto the main net decentralized service. And this is run, obviously, as most listeners probably know, by all the indexes at The Graph, and there’s many upsides of decentralization whether you talk about, you know, redundancy, lowered risks in first Certain attacks.

10:01
You know, sort of trying to paint the two pictures, right. So the hosted service centralized, the main net, where Indexers deliver content, decentralized to a degree, right. But the hosted service is essentially a giant Indexer. So if we just think of it as a single Indexer, right now on its own delivering queries, it has the Indexer stack, it has something called The Graph Node, which is responsible for doing the indexing, that Graph Node is connected to it, let’s just, let’s just assume it’s just Ethereum for now. So it’s connected to an Ethereum node set up as an archive nodes. So that means it’s capturing all possible data from the blockchain. And it’s using that data in order to index subgraphs, right. And so The Graphs are instructions around how data should be stored in a retrievable fashion, right, so that we can have our consumers querying us to get data out of our databases. The difference between the hosted service and main net is that you essentially have that whole stack that I just described, repeated over and over, you know, 150, about 150 times right now. And every one of those deployments is usually on some different type of architecture. So you know, it could be on servers in Tokyo, could be on servers in somewhere on the American continent, could be anywhere in the world, right. So immediately, what we’re starting to do is we’re starting to distribute or decentralized the geography of the delivery network for this data, then you also have Indexers running their own different sort of configurations. So you’ll have some indexes that are running one version of Ethereum nodes. You know, one very popular one for us that most people use right now is called Open Ethereum. Some Indexers might be running Aragon, Aragon is what we generally refer to as a next generation Ethereum. client, rivet brings a lot of benefits, and a lot of people are starting to use that both in test net and main net, you could be deploying in a cloud, you can be deploying to Amazon, Azure, Microsoft Azure, Google Cloud, you can be deploying to a dedicated server that you rent from a company like Hetzner, right? There’s a infrastructure provider, they own dented data centers all over the world. Or like WaveFive, you could be self-hosting. So you could own your servers, Own your dedicated internet links, own all of your Ethereum archive nodes, own all of the infrastructure for your network on all of your power infrastructure. So we have all these different Indexers with different configurations all over the world, instead of one single central entity that’s just running one version. And that’s sort of the decentralization ethos, right? We’re pushing ourselves out the decentralization on the decentralization spectrum. And what that means for end users is that their data is more reliable, right. So if, if the central service was to go down, that would impact probably, you know, a lot, or maybe all consumers on the central service, the hosted service. If one of 1 of 150 Indexers goes down, there’s far less likely to be some kind of service interruption because another Indexer is likely to pick up the query business that the Indexer that’s gone down, maybe they’ve lost their internet connection, maybe they’ve had a disk failure. Some other Indexer will pick up that traffic. So there’s a lot of resiliency improvements, you also have the fact that your data source isn’t one particular method, or one particular version of an Ethereum node, right, you have all these different Indexers with all their different node configurations out there, all coming together to sort of agree on consensus around the integrity of the data. And that’s where we get into that sort of more high level narrative around, you know, owning your own data or being confident of the data that’s being served to you. You know, one of the reasons that a lot of us are sort of committed to this decentralization progress is sort of progressive decentralization is that it’s pushing the authority around data from a central entity out to many entities. So everybody’s watching everyone else to make sure that the integrity of the data remains despite having 150 Indexers all delivering that data that covers some of it. But you know, it’s such a huge topic.

14:24
Gary is a follow up to that then how would you describe, given the mechanism that Jim just outlined, how then with this distributed decentralized system, our queries made to one Indexer, not the other if they’re all backing each other up? You know, and your Indexers running? And Jim’s is and you’re, you’re serving these queries? How come you get the query fee and Jim doesn’t in that scenario?

14:47
Well, it’s funny because that’s determined by the gateways which are actually currently, you know, a slight aspect of centralization in themselves. Indexers are though we’re basically just one lm In the big picture of The Graph, as far as the decentralization goes, each Indexer also has their infrastructure on a spectrum. If you can imagine, some might self host, like Jim says he does, I also do this, some might run their Indexes in the cloud. Sometimes they might have multiple indexes in the same infrastructure, maybe some run their own independent Ethereum endpoints, some might share endpoints with others, or even rent them out to others, which you know, is not decentralized as it could be. But things like this can actually help smaller operations get off the ground. So you know, this is fine to a degree, as long as it’s not the majority of the network doing it, in my opinion, I’m of the opinion that you should aim to whatever level presents the strongest feasible level of decentralization in your own infrastructure. And the some of that is pretty much always going to be preferable to the monolithic, hosted service. We’ve already seen, you know, downtown happening on the hosted service, it’s unavoidable. And the more indexes we have, the lower this risk potential becomes, if we ignore other variables, such as, you know, state distribution, and how that might relate to query load. But as to your answer, yes, query load at the moment is determined by the gateways, which I believe the team planned on open sourcing someday.

16:23
Just to add a little bit about the gateways. So in terms of your original question around, how does the query go to Gary? Or how does a query go to Jim, the gateways are actually scoring Indexers on a number of, I guess, what you might call Key Performance Indicators, or they refer to The Graph is the utility score of an Indexer. And how this works. We don’t have the surf source code for this. But we have it’s been explained a number of times now, and it’s documented in a few places on the repo, The Graph Repo, the certain things are taken into consideration, such as your latency to the gateways, or whether your data freshness. So you know, have you managed to sync a specific subgraph all the way to the chain head? So the go to the very latest block on the network? What’s your social reputation within the system? Are you a new Indexer? a number of different sort of Key Performance Indicators to figure out whether or not you are suitable for providing that query data. And the gateway will select multiple indexes based on that, in order to serve the data that the consumers are demanding?

17:29
I want to ask just a general follow up question. You’re both on The Graph Council and you’re working with Indexers of different sizes, what’s the state of indexing then at The Graph, if you had to give just kind of a summary of where we’re at since testnet, to present, and I’ll give each of you a chance to just kind of give a summary observation of where we’re at with indexing at The Graph.

17:50
I think that can vary widely, because we have quite a lot of different Indexers. And that goes all the way from indexes that don’t really partake in, you know, serving queries and have 100% cuts are not interested in delegations all the way to large and small operations are like that are very, very involved in the community. So I think it’s, it’s very broad by In my opinion, I’ve seen a lot more contributions coming out of the woodwork as we go along. A lot of people starting to take their infrastructure more seriously, and their operation more seriously, you know, basically turning their Indexer into a sovereign business as it were. But I think that there’s always room for improvement. And yeah, with the introduction of new subgraphs into the network, and how we’re seeing signal moving around quite a lot, you know, a lot more curation up in actively into the network. There’s been a, you know, an exponential increase in the effort required from indexes. And yeah, there’s a lot more going on. And another part that Indexers have had to look out for is what’s known as I believe the SLA, there’s a there’s an Oracle that is run by the team that actually determines which subgraphs are eligible for rewards. And as some of the subgraphs that have been launched and on supported networks.

19:18
Right. So I think the answer is Indexers are having to work harder, and indexes are having to really think hard about their risk. So in terms of what indexing looks like now, compared to maybe the last time we talked, Indexers are sort of having to face the reality that they need to manage their risk. And they’re also having to face the reality that indexing as it stands today is not a job that you can automate away and, you know, carry on with your life. There’s a lot of time to be spent, as making sure that you make the right decision so that you’re making, you know, you’re meeting your Delegators and your own expectations in terms of revenue and yield. There are many pitfalls around risk for Indexers now that are coming to light. I’ll give you one example because there are many, but let’s just let’s just have a look at one example. A lot of Indexers will find new subgraphs or maybe existing subgraphs that have quite a low signal on them less than 100 GRT signaled on them. But if you do the math’s behind, you know, the stake on the subgraph, you’ll find out that actually, it’s very profitable to allocate to that subgraph. So what you’ll see is some Indexers putting large amounts of GRT on the subgraphs that are less than 100 GRT. And they’re thinking that they’re accumulating a lot of GRT over the period of the allocation, their Delegators think the same. Once the subgraph hits that 100 GRT level, there’s a function on Main net called the subgraph oracle, the subgraph oracle’s actually checking subgraph to make sure that they can be indexed correctly, and that there’s no special features, because there’s a lot of options to add special features to subgraph special features that are not yet supported on Main net. So I’ll give you my one of my own example. So there was a specific subgraph, I won’t name it, that just ticked over that 100 GRT signaled event. So that triggered the Oracle to go and have a look at the subgraph. And it found a feature on that subgraph, which is not supported on the network today, by not supported, what it basically means is, we’re not sure that we can, you know, give you 100% confidence that will return the right you know, the correct data every time or the same data every time for this feature. So it immediately sends a transaction to shut off rewards for that subgraph. Now I was allocated to that subgraph for 19 days, right, so you can imagine the amount of GRT that had been building up against that allocation. And because the Oracle had closed that, you know, had closed off the feature of staking record rewards for that subgraph I had to sacrifice all those rewards, there was no way for me to get them. So these are that’s just one example of many risks that Indexers are now facing. It’s a sort of reality check, right. And a lot of us were already warning back before migration started that, you know, this path of, you know, birthing the network is not going to be easy, there’s going to be a lot of work to be done, there’s going to be a lot of ambiguity along the way. So that’s really what the Indexer community is working through right now. We’re working through the ambiguity of all these different types of events that can happen. We’re working through the ambiguity of curation signal not really being a reflection of whether a subgraph is worth indexing. You know, we’re starting new initiatives around a lot of these problems, so that we can address them going on word. So there are no longer like this issue of should I do A or should I do B for an Indexer? There’s a lot going on in the Indexer Community.

22:50
Yeah, there’s been a lot more for Indexers to do and pay attention to, especially with the introduction of curation, from what I’ve seen, because with the introduction of curation to the masses, there’s been a lot more subgraphs entering the men. And since the majority of revenues for indexes in their Delegators comes from awards, indexes need to pay attention to signal versus allocation totals per subgraph. But you also need to consider, you know, the return against allocation fees as in gas costs. And even then, there’s a lot more to consider, you know, such as the SLA which Jim just spoken. Add to that, that some subgraphs might fail. And some people think that the arbitration rules aren’t entirely clear over what to do in instances like this. There’s just a lot more to do a lot more to consider. But thankfully, the way we’ve seen this play out, from my observation, at least is a higher engagement of many Indexers in the community. And, you know, more activity on the network, not just from indexes, but from other stakeholders. There’s been, you know, a great Curator community that was born from basically some teething issues when that came around. As far as indexes go, though, and our community, I think that the way things are going these are the first steps where people who are really paying attention will start to rise above you know, a little bit of wheat from the chaff is going to play out from this I think.

24:23
I want to talk more about curation and what your observations have been with the launch of curation. Before I do that, I want to ask a couple follow up questions about the state of Indexers at The Graph. The first question is about the total number of Indexers. Do either of you expect or did you anticipate more Indexers coming to The Graph? Or is the at the time of the recording were 158 total Indexers, I believe, is that kind of the magic number that you plan to see persists?

24:51
I don’t think there is a magic number. Something that we’ve you know, thought about and we’ve debated a number of times now in the early days was you know, this Sort of requirement to have 100,000 GRT in order to start an Indexer. And there’s a good reason why that number is so high, right? This is not like your typical everyday garden variety, Proof of Stake network where you, you go to the repo, read how to deploy the node, open some ports, and away you go, you’re validating transactions, and you’re taking delegations, etc. It’s far more complex than that. And that’s why there’s such a capital requirement to get you started as an Indexer. Now, I think it’s important in the early days to have that sort of close knit community, smaller community of Indexers that can, you know, really work together, but there comes a Point, right, where you know, I’m in, I’m in a relations position at The Graph. So part of my job is to know the Indexers. And to talk to the Indexer, who is the human behind the name, right, there’s going to come a time when I can’t do that anymore. And it’s something we’re thinking about quite a bit in the foundation now is how do we grow the Indexer base, both through initiatives and through technology. But as we’re in this sort of, you know, I keep calling it the birthing phase, because I really want to try and delineate this phase we’re in now from like a growth phase, right, we’re in the birthing phase where it’s very chaotic. That’s the time when you want to have the real, the folks who have got the real skin in the game and are not going to disappear. Because things get a little bit harder, right? Once we can get over that, and we have a good bit of the ambiguity behind us, I think that’s the time when you want to start really pushing to grow the Indexer community via multiple means, you know, maybe you standardize the software, and it’s very easy to deploy an Indexer now. So maybe you could consider dropping the, you know, the initial stake requirement, maybe, you know, Ethereum nodes are much easier to deploy, you don’t need eight terabytes like you do today with an open Ethereum node, you maybe need 1.5 with an Aragon node, that’s maybe another reason to consider, you know, making it less of a capital intensive requirement to join the network. But the reality also is that at this Point where we are sort of, we’re skewed towards the staking rewards, right? We don’t have huge amounts of query business yet, right? We’re building to get there. In that situation, it’s very difficult for someone with a small stake, even 100,000 GRT, to be very, you know, to make a big impact on the network and still make a return. You know, to give you an example, the small Indexers that I talked to that have maybe just started on the network, they’re looking at allocating to only one subgraph, right, because if they allocate to any more than one subgraph every month, they immediately lose all of their profit via Ethereum transaction fees, right. So really, now is not the time, in my opinion, to be concerned about the number of Indexers. If we had a serious sort of drop in those numbers, yes, I would be concerned that the growth phase I think, or Indexers will correlate with the growth phase in the fees. So as we start to steer our strategies as Indexers from optimizing for indexing the rewards to optimizing for indexing rewards and query fees, that’s the time when we can start to think about bringing a lot more Indexers into the space. And there’s loads of cool stuff going on right now chats around, you know, the things that the new core dev teams are bringing in how their new technologies make it a lot easier for new people to start an indexing business.

28:24
Yeah, it’s not just about Indexer quantity, but more so quality, there’s plenty of indexes who don’t participate in the community, they sit there cuts, the 100%, etc. I wouldn’t really call these a net positive or the network. But this is something you see on basically any network, there’s often, you know, lazy nodes that were that are just there to take a share of the revenue by a Proof of Stake or however, but due to the dynamics of The Graph, and the fact it’s basically a competitive marketplace between indexes, it means that those who put in the effort can get ahead, I’d like to see this dynamic play out quicker than is, as I’ve already seen some of those with lower stakes, but you know, fairly high interest and engagement leave the network because they feel that there’s not enough advantage in being an Indexer when they have such a low stake amount. So they turned to delegation. But I think we’ve got a great core of really engaged indexes. So I’m not too worried about the timelines over this because I see it as more of a natural and unavoidable outcome anyway.

30:45
What about this topic of decentralizing stake are either of you concerned about The Graphs ability to keep stake decentralized?

30:53
I actually made a post, I think it was in February on this very topic. And there’s another one that’s just popped up by Oliver from the foundation. So the topics kind of come into our head again. And you know, I think anyone listening should go and check that out, and maybe even check out the prior discussions, because we’re going through the motions at the moment of determining how we should best decentralize stake how much we should decentralize that stake, and there’s a lot of different and widely varying opinions of how important this is and how they should be done. Personally, I think it’s pretty critical to address this early on, you know, in the infancy of a network, because if you let centralization take root, I think it’s, it’s like a weed, it’s going to be so hard to get rid of. So yeah, discussions are ongoing. I’d like to see more engagement from more people because this this affects everyone no matter the position in the network. Any stakeholder has an interest in this because it’s something so important that will actually keep the network strong in the long run.

31:59
Yeah, I would agree. And I would back up those comments with, you know, my own opinion on stake decentralization. I mean, Gary back in, you know, the very early days of the forums, that’s his post, there sort of outlines his feelings and mine as well. This is something that you often get called a Doomsayer, for mentioning, you know, on any network, but you’ll often find is conveniently swept under the rug, that there’s you know, you’ll often see the same players in the top 10 most delegated validators on the network, my biggest concern is that we don’t openly address this type of issue prior to the real big participants coming along, which by which I’m basically mean exchanges, right? Because when exchanges come along, and they suck up all of that stake that maybe someone else is willing to sell, they put themselves in a position where they make it so easy for their users to just pseudo stake on the network with no bonding or thawing. That is very difficult to then turn the knobs, I guess, on stake decentralization after the fact. So for me, these conversations that are going on are extremely important. And Gary talked there about wanting to see more, more people involved in the conversation. And that’s really, to me is paramount, because what I’m most interested in is ideas, right? I’m not really interested in punitive ideas that sort of take from the rich, that type of thing. I’m really not interested in that. But I’m interested to hear about ideas that can capture more, more of the GRT. That’s outside of the network. Right now. Andy from Figment made a good Point that, you know, if you look at the numbers, about 30%, or less, just less than 30% of GRT is in the protocol right now. So what can we be doing now in terms of promotions or parameter changes on the network or some new Dynamics for delegation ratio is what can we be doing in that space? In order to sort of entice that GRT into the network? And also have it disperse among more Indexers? So to me, it’s all about the conversation and seeing as many people expressing their opinion in that conversation is possible. To anyone looking out for that Gary mentioned that a post by Oliver from the foundation around stake centralization, get in there and get involved.

34:22
I’d like to add to that, though, that one very important aspect that you touched on there is how can we lead future delegations in a manner that increases the decentralization of stake and allocation, right. But I do think it’s also important to consider how we got here in the first place. So you know, there’s various variables as we know, set that determine how much capacity an Indexer can have and things like this. And, you know, I believe there was a lot of consideration and you know, analytics done on what the perfect numbers were. But at the end of the day, the way we’re seeing things play out, things are not really headed in the direction we would like, in the network. I think I think most people would agree with that maybe, you know, certain top 10s wouldn’t, as you said, but I think it’s important to also address whether the balance and the variables we have right now are optimal. I don’t believe that we got lucky. And we hit the nail on the head the first time, especially now that we’ve seen it play out in a live manner rather than theoretically. So I would I would say that, yeah, I agree that it’s not about punitive, but I think there should be, or there could be, let’s say, if it makes sense, some changes that fix retroactively mistakes made with the introduction of certain variables, or, you know, however, those were determined.

35:54
Yeah, I mean, I fully agree with you. Right. So when I say punitive, what I really mean is something that directly goes into the pockets of law of Indexers based on their stake, right, something along those lines. Now, if we’re looking at something like I’ve seen a few different things mentioned around in different ways or mechanisms for the for the delegation ratio, the type of thing I’m interested in is things that could in theory, not just, you know, obviously, I’m most focused on that 70%, right, because it’s the biggest amount, you know, that outside the protocol. But, you know, I would like to see ideas, mechanisms that start the process of just naturally dispersing it more. So that will in reality is that will mean that some will move from indexes with very large delegations into indexes with smaller delegations, you know, I’m not saying that it’s as cut and dry as you just don’t take from pockets, right. I just want to see the network be the thing that actually pulls the incentives out to make that stake move, rather than something that feels like you’re dipping into a pocket. Right? That’s, that’s really the message I was trying to bring there.

36:56
Yeah. Well, I know what you meant. I just wanted to clarify that for listeners, because something we’ve seen is, you know, even the ideas or the you know, the suggestions that we address, these balances can often have a narrative formed around them. We’re in certain people are ramming these ideas and changes as potentially punitive. They will, as you say, affected, you know, maybe some indexes, maybe a handful, who knows, it depends what changes we’re talking about. But I think there’s a large difference between, as you said, dipping into the pockets, and, you know, addressing a balance, which may alter the amount of allocation someone would have, in a way that benefits the majority, right?

37:41
Yeah, yeah, we were on the same page on this. There’s no doubt about it. And it’s just a good conversation. And again, it comes back to this I thing I’d like to see around participation in these conversations. So if you look at sort of the list of people who’ve commented on that, so far, they’re all like, either big validators, or the thought leaders within the community. We need more participation. We need more ideas we need small Indexers training in we need middle sized Indexers chime in, we need to hear the whole gamut. Right? I understand that, you know, some of the big validators feel a little bit under the under the microscope, they maybe feel threatened. So I understand some of the responses. And but I also agree, there is some hyperbole in that discussion. But that’s just, I guess that’s just par for the course. Right? We just need to be able to keep talking about it and not, not get too tied up. Yeah, exactly.

38:27
I think we need to, you know, extend to stay on topic and think of a solution if one’s needed, which, you know, I feel there is.

38:35
I appreciate you to doing such great job deconstructing not only that topic, but proposing some ways that the right ideas might rise to the top to address it in a long term fashion. Something else that recently cropped up that, you know, to borrow from what Gary said there kind of had its own narrative. Was someone going around and closing allocations on Indexers. And, you know, I don’t really understand how they did that, or necessarily why they did that. But I think a lot of people noticed it, and there was some discussion of it on Twitter. What do you all make of what happened?

39:05
Well, basically, as an Indexer, when you set an allocation, you’re eligible for awards from the allocation for 28 Epochs, not to be confused with 28 days, it’s not precisely the same. And once you’re above that threshold, you no longer earn rewards from can’t recall now if it’s the 29th, or the 30th, I think if it gets when it’s above 29. So the 30th Epoch, could be wrong though. A Delegator can close an allocation on an Indexers behalf because, you know, basically that the Indexer is acting in a negligent manner. And maybe the Delegator wants to get their funds back. So yeah, somebody has been going around in recent weeks and finding allocations that have gone above this threshold. For whatever reason, these indexes have been neglecting their allocations. Just forcibly closing them. And when they forcibly close that the rewards are burnt. So, you know, there’s been some, there’s been a lot of money burned by this, I was looking at some Indexers, I won’t name them. But you know, there’s some quite large non names, and they had, you know, allocations of millions of GRT. For a full allocation lifetime, you know, 28 Epochs there was, you know, it depends on what subgraph and the signal ratio and stuff was, there could have been, I don’t know, 50,000, or 100,000, just burnt, simply because the Indexer wasn’t paying enough attention to close their allocations. There’s no benefit, I guess, for the Delegator in this circumstance. But I think I would imagine that the reason someone is doing this is basically, to punish Indexers that aren’t taking the role seriously, that’s what I would imagine because the there’s no monetary gain for the person closing on the indexes behalf. So I think this was some kind of, I don’t want to say, Robin Hood move. But you know, you could take it that way. Because even those rewards are already allocated and then burnt. So it’s not necessarily taking from them and giving away to others. I guess one effect it has is lessening the total supply of GRT. So every of our stakeholder benefits.

41:28
Yeah. It’s like a deflationary. It’s a punishment for that Indexer. But it’s a deflationary win for other Indexers. And I actually believe that those funds, Gary are not actually burnt. They’re just never minted. Right, right. Look at the transactions, right, where your transactions come from when you close an allocation. It’s actually minting the GRT. At that Point, as far as I understand, something that’s not clear to me, though, that is, is this the way it was designed, because you know, we’re coming in we’ve got up against, we’re talking about ambiguity and risk, right. Another sort of example of a sort of risk decision that Indexers are having to make is around the complexity of a subgraph, you may find sometimes as an Indexer, that you’ve allocated to a new subgraph, and then you go and do a little bit of math to work out how fast it’s sinking, and you come to the conclusion that it’s actually going to take more than 28 days to sink. So in theory, you could still be syncing that subgraph still continue to be an active Indexer. But you haven’t closed your allocation because you can’t you haven’t synced to all the ways the chain head, and some vigilante comes along, delegates, one GRT to you and closes your allocation for you and sacrifices all the rewards. To me, I don’t feel like maybe that’s quite what this mechanism was designed for. So that’s not actually something I’ve asked Brandon Ramirez at Edge & Node. But I’d like to maybe understand a bit more about what was the original thinking behind that functionality is it working as intended, certainly isn’t working as intended for good Indexers that maybe made a bad a bad decision on a complex subgraph. But I’ll be honest, I do appreciate checks and balances that sort of keep Indexers on their toes and make sure that they can’t just run a passive operation forever, and expect to benefit from it.

43:12
Yeah, I guess this goes back to one of the other questions about the state of indexing. So this is yet another thing I guess you would say that Indexers now will have to be aware of, because before as Jim said, If you allocate to a subgraph, and then you realize it’s, you know, super heavy going to take quite a while to synchronize so that you can even close the allocation. Before now, you ever needed to consider whether to bail out early by, you know, basically closing your allocation forcibly yourself and forfeit in any rewards for spar, or risk, keep an open and close letter with the less good POI. Once you’re finally in sync. I think now that it’s clear that somebody or some people are paying attention to Indexer activity in this manner, that second option is now off the table completely. It’s just that it’s a whole new dynamic just a spanner in the works for indexes. But I don’t think it’s a bad one. Because as we touched on in a recent office hours, I feel like this is just all part of the you know, the the puzzle that is the strategic decisions made by an Indexer. So there’s so many considerations to make. And this is this is just another one of them. And as Jim said this, at least avoid rewarding those who run a passive operation or want to run past the max allocation length and still take rewards for that. Because at the end of the day, you’re meant to be synchronous data so that you can serve queries. If you’re not synced, you wouldn’t be serving the freshest data. So should you be rewarded for that, especially if you go beyond the allocation link? That’s debatable, in my opinion.

44:54
So I want to piggyback on this discussion with this idea related to arbitration and disputes and to be 100% honest, you know, this isn’t something I fully understand or grasp. But in addition to someone going around and kind of closing these allocations, can you help us understand how arbitrations and disputes might relate to these types of issues?

45:18
So the subject of Delegators closing an Indexer is allocations with a 0x0 POI, although that’s part of our previous discussion, the arbitration and the dispute pieces, specifically around Indexers submitted POIs right, so maybe we start off with what is a POI? POI stands for Proof of Indexing. So if you imagine that an Indexer has indexed a bunch of data, and they don’t get it going to get rewarded for it, then you need a mechanism that will allow the Indexer to prove that they’ve actually indexed the data. And that’s done by taking a number of essentially measurements, mathematical measurements of the data, and then hashing it with the Indexers address. Right, that’s a very rough explanation of what the POI is. These POIs are submitted on-chain when an Indexer settles their allocations. And there’s this role within the network called Fishermen and Fishermen are essentially surfing POI eyes, they’re looking at the pies that have been recorded on the chain, comparing them and seeing if maybe an Indexer is has delivered a bad POI. If they think the Indexer to divert or a pod POI, they can then raise a dispute. A dispute costs them 10,000 GRT to raise, and essentially starts a process of arbitration, where an arbiter who is a person, of which there are three on the network right now, part of a multisig wallet. The arbiter then assesses these disputed PI’s asks for a bunch of information from the Indexer in question, to see if there was a malicious, POI generated or if it was just simply an issue with the software or the node, they’ll go into real detail around, you know, the logs on the Indexers The Graph stack, basically, to try and figure that out. Now, if the Indexer is found to have done something malicious, then they’re going to get slashed. At the moment, the slashing for the index of rewards, I believe is 2.5% of self-stake. So as you can see, there’s this play off between the 10,000 GRT, the disputer, or the Fishermen has to submit for a dispute and the amount of GRT they might be able to make if they are really confident that the you know, the POI they found is malicious.

47:49
So what is it that Delegators need to understand about arbitration to disputes? How does it apply to what they’re doing or their role in the ecosystem.

48:00
So the arbitration process is specifically about Indexers and their behavior on the network. So from a delegated perspective, you know, if I was just, I’m a Delegator, and I’m reviewing my Indexers every month or whatever, what I would be doing is I’d be going and looking on the forums, there’s a specific forum section for arbitration. And you will see these posts that start with GDR and then a number. And I believe that stands for The Graph Dispute Resolution. So every time a dispute is raised on-chain, one of the arbitrators, there’s three arbitrators on the network, one of the arbitrators will post one of these GDR and request data from an Indexer. About, you know, when the POI was submitted that that has been dis, you know, disputed as malicious. What was your node doing? What was your graph node doing? What were you doing on, you know, on your indexing stack? They’ll try and work out if, if what happened was malicious, or if it was just some kind of bug within the software. So as a Delegator, you can actually go in and follow these conversations. And, you know, very recently, we’ve had some sort of some high energy GDR, shall we say, where, you know, some very strong, you know, some strong signals have been documented that an Indexer is potentially acting maliciously. So as a Delegator, you can go into these discussions, you don’t necessarily need to be able to follow all the detail of the logs. But if you read the posts, that other contributors, maybe the disputer themselves is putting on that forum posts, you’ll be able to read about, you know, potentially the behavior of that Indexer. So I’d be looking out to see if my Indexers appear within the arbitration section on the forum. And if they do, I’d be looking at how they react, how they respond, and what the eventual outcome of the dispute is. A dispute can go three ways. It can either go in into a draw, which means that the distributor gets there. There are 10,000 GRT back, which is what they have to use, they have to deposit in order to create a dispute. And the Indexer doesn’t lose any stake, right? They’re not slashed, you can then have an arbiter vote in the Indexers favor, which would sort of be sending a signal that the disputer was wasting everybody’s time, they would lose their 10,000 GRT that they submitted, the Indexer would carry on and is assumed to have done nothing wrong. And then you can have it go the other way, which is the nuclear option, whereby it’s deemed that whatever the Indexer did was, was malicious, the disputer wins, they keep their 10,000 GRT. And then they also receive 2.5% of the Indexer self-stake, which is the slash basically, as a reward, right. That’s the reward for performing the Fishermen role on the network.

50:50
As far as the dispute process goes, what I’d like to see happen soon, is some precedents being set as far as outcomes go, because so far, we’ve just seen draws, and that may be the valid outcome. That’s not up to me to decide. I won’t comment on any particular cases. But the way it should work in my understanding is that for instance, if a POI is valid, the Fishermen’s deposit should then be forfeit. This is the way I believe it was intended to be. That’s yet to happen. You know, maybe that just wasn’t the the outcome of the cases so far. But if an action was malicious, then of course, the Indexer should be slashed. If an issue was the result of a software book, a draw should be declared, which we’ve seen happen so far, and in a few cases already. And the reason I bring up these outcomes and the precedence’s that need to be set is because so someone ask in Discord about a certain Indexer. You know, what you can tell me about them, blah, blah, blah, and someone linked them to a dispute involving the Indexer. And to me before an outcome is determined whether that Indexer did anything wrong, the fact they have a dispute against them is completely meaningless, they might be found guilty, and they might be found innocent. But until some precedence is set where Fishermen are expected to do their job thoroughly, I think there’s, you’re just introducing the chance for a lot of noise and a lot of work for the arbitration team. And not just for them, but also for Indexers, because as soon as the dispute is presented, the Indexer then has to, you know, provide a response to the arbitration guides to prove their innocence, it’s in their hands, the onus is on them to prove they did nothing wrong. And until you start saying, look, submit a dispute that you believe is valid, or your deposit is going to be used as a deposit and you might lose it. I think this is just going to start to frustrate people more and more, because they’re, I feel people, whether they’re Fishermen or Indexers they’re not actually seeing any closure for the cases that have happened so far. So that’s something I hope will change soon. Just seeing some actual judgments being passed. Maybe it just hasn’t happened so far. And everything has been a software bug related. I’m not sure that’s the case. But I just hope we see something happening soon that actually prevents mass disputing because I’ve actually commented on this earlier and in the community that I’ve observed on-chain. Sometimes people will talk about allocations that they feel should be disputed based on maybe not even anything to do with, you know, a good or bad POI, but because of the way they understand the arbitration charter. And so some people seem to have set up some bots so that as soon as those allocations are closed, they then their bot, then immediately, literally, in the next block, you know about a 10 second gap, sends a dispute against that Indexer. To me, this is not being diligent, unless you’re absolutely certain that you’re on the right side of the rules. And in that case, if you’ve automated it this way, and you’re so certain of you being in the right, then as a fisherman, you’ve then forfeit your deposit. In my opinion, this is how it should be.

54:14
Jim and Gary, I want to talk a little bit about curation. Obviously, this is something new to the community and a lot of people have interest in it. What are some early observations, either things we’ve learned or things that we should anticipate as it relates to curation?

54:27
I think one of the most interesting things that we’ve seen is that again, we always think about the network in terms of steady state is going to be like this, it will be like this in the future, right. But I think the bonding curve dynamics have shown us that maybe some of these mechanisms are not in their current form entirely conducive to what we originally thought they would, you know, answer in terms of objectives.

54:53
I think when curation went live, a lot of people were not entirely prepared to participate, let’s say a lot of people got caught out by various things happening in the network. And I think, from that there was at least a positive outcome from what I’ve seen a Curator, specific community formed around the issues people had during the first, you know, desert a week of curation going live. And they seem to come to the realization as a group and start helping each other with this, that they need to be very diligent, assessing subgraphs, which, you know, of course, is the point of a Curators role. But things they weren’t expecting, like having to determine the source, you know, the owner who published the subgraph, you know, is this legitimately owned by the team, you know, might still be good and legitimate data for a certain project? But is it owned by that project? Will it be used by that project? Will it see query traffic? So I think when curation went live, a lot of people soon came to all these understandings, and you know how to act really quick. And it kind of goes back to what I said about Fishermen, I think that people are realizing the magnitude of different roles in the network. And that goes for indexes to since the introduction of all these new subgraphs, and all this new stuff going on in the network, whether it’s forcible closing, dated allocations, or anything, a lot of people are realizing they really need to pull their socks up. And think about what’s required by this role to actually do a good job if you want to be successful and have a positive impact in the network.

56:41

I think just to add to that, there’s a lot of positive stuff going on the curation side. And we absolutely do we need the best people to be curating, right, that’s the job at the end of the day is curating the quality of subgraphs. And whether the subgraphs, you know, will command the query fee that the signal says they should, right now. That’s not really the case. Right. So, signal is supposed to lead Indexers, Indexers should in theory be able to allocate just on signal weight, again, in a steady state network allocate just based on signal weight. Because the signal is supposed to be a signal from the Curators that there’s going to be some proportional amount of query fees on that subgraph. But as we are in this early stage, that’s just not. That’s just not the state, right. And it’s not going to be the state either, I don’t think until a couple of things happen. One, that the level, you know, the volume, or the amount of liquidity in the signal side increases significantly to reduce the volatility, and two, the query traffic starts to really pick up. And to be honest, I don’t think it takes more than one of the real big dapps to finally turn the spigot on query data on them on Main net to make that happen, right. A lot of people will suddenly start paying attention when we see that huge demand for query fee coming in and you know, potentially, rewards starting to increase to something competitive around, you know, against the staking rewards.

But also, you know, there’s other positive things happening on the sidelines. You know, from the very moment that curation started, I started getting DMS from subgraph developers. And I started to see this new business model appearing that I hadn’t even considered. If you imagine that a subgraph developer or a subgraph development shop that specializes just in doing subgraph maintenance, aligns themselves on a new subgraph. So they take maybe one of the big dapps or mega dap, and you know, they deploy their subgraph for them. And they signal they’re the first to signal against it. So they’ve aligned themselves immediately, in terms of the signal they’ve put on the subgraph and the returns, they’ll get on that signal from the query fees. What I’m seeing is these individuals or these teams, then going out to the big dapps, and explaining what they’ve done, and offering to maintain their subgraphs for them purely based on the alignment they’ve got out of the curation market. So it’s no additional expense to those teams apart from paying for the queries themselves. They just worked with these dev teams, in order to have the subgraph maintained and upgraded when it needs to be upgraded via PRs and whatnot. That’s, you know, I think is inevitable, right, that’s what’s going to happen. But there are, you know, one of the things we are seeing is that, you know, yes, the bonding curve side of things is problematic. And there’s a lot of talk about fixing that on the forums. But there’s also small features that need to come in, like, you know, a really good example is we don’t today have the ability to transfer ownership of a subgraph from one party to another. So this business model that I just described around, you know, subgraph development and maintenance shops. That business model cannot work until that improvement goes in. And it’s in the works now, right? So we will have potentially business models that are based purely off, you know, aligning, Dev. shops aligning themselves along, you know, along the lines of the curation signal. So there’s a real mix of stuff going on. But over the top of all of it, what I have noticed is that immediately we’ve seen thought leaders and leadership appear from people within the Curator community. And that’s a really positive thing to me. There’s a number of names that come to light immediately, when I think about people who are taking the lead, they’re very responsible and diligent about their position, they’re communicating with Indexers, they’re talking about the health of subgraph, they’re trying to do the right thing. So again, I think a lot of this comes down to communication, and fixing these problems that we were seeing is all about getting involved on the forums and getting your ideas out there to see if anyone else you know, thinks those ideas might work.

1:00:57
I really appreciate both of your time today and providing some updates on the state of a lot of different things happening with The Graph ecosystem. We talked about the new core development partnerships, the state of curation, and the state of indexing. So I think the last question I want to ask is in relationship to what we should look for next important milestones, as we continue in the evolution of The Graph. What are you guys looking for? What would you encourage Delegators to look for?

1:01:23
I think, you know, just from my sort of position, where I sort of have a position, working on Relations at the Foundation, and on the council, something that I’m thinking about, and I know a lot of people the foundation are thinking about is transparency in the development process. So Martine and the foundation is championing something around this at the moment, there is this, in my opinion, there’s this need for us, you know, we’ve done very well, I think, I think the founders, and some of the leaders, Yaniv Tal, Tegan Kline, Eva Beylin, have done a very good job of sort of setting the culture and the ideologies maybe around what The Graph is trying to achieve. But often, after somebody bought into that, you know, they see that they see the vision, their next question is usually, you know, how are we going to get there? Right, we’ve got these core dev. teams, now, they have some amazing technologies that they’re bringing to the floor. So how do we get from this sort of, you know, disjointed, maybe this disjointed group of technologists into a homogenous delivery machine, right, that’s going to deliver a bunch of, you know, improvements to the software. And I think that some of the initiatives that are going on in the foundation will help address some of that, such that, you know, the core dev. teams have all come to consensus on, you know, what the roadmap might look like for the next, I don’t know, six to 12 months.

So I think, you know, whether you’re a Delegator or any other participants stakeholder within the network, that’s the type of thing I would be looking out for, you know, we know where we want to go, we know how StreamingFast going to improve the performance of indexing, which brings all kinds of second tier effects to the network. We know that Figment are experts on many other networks. And they’re going to be bringing accelerated multi-chain indexing to the network. How do we get there? What does it really look like? And I think that’s something that everybody should be looking out for in the near term. I imagine a lot of that will be communicated via the forums. If you look out for there’s a there’s an RFC in the forums, from Alexander from StreamingFasts, a really excellent piece of, of writing around StreamingFast Firehose technology, which is the key really to increasing the speed of indexing. That’s the start of the process. Look out for more of that there’s more coming from Figment. And then I would also be looking out for increased transparency around what the core devs are doing, and what the, you know, the big picture looks like from a technical Point of view.

1:04:02
To be honest, I think you’ve summed that up quite well, because I was going to say that I think what’s most interesting is going to be exactly what you just talked about the introduction of these two new big core dev teams and how they fit into the big picture. Because for now, what we’ve had is the original The Graph team, now known as Edge & Node, you know, kind of the center of development and the thought leaders we kind of like look to for direction of, you know, the path to the vision as it were, and I’m very interested in how the new teams are actually going to fit in to this not only you know, as far as development and their technical skills go but also their, their thoughts on the vision and how to get there fits in with what we already have. And you know, the community that’s already been formed.

1:04:56
Yeah, that’s a great Point. You know, the idea of The Graph culture is building slowly. I think all of us feel it. And, you know, it’s all well and good that, you know, we have these new teams on board, but how do we get all of those teams sort of, you know, getting to know each other of we’re learning how to work with each other on a day to day basis. This is a brand, you know, this is a relatively new paradigm that we’re working in, right? There’s no, you know, I often talk when I’m talking to others about this, I talk about the fact that there’s no HR function to sort of glue everything together, right? It’s on all of us to help make that sort of thing happen, you know, reaching out to the new teams, talking to them, getting to know who they are as individuals, that sort of stuff. So a big part of this, I think, is also, you know, developing The Graph culture. And I think you’ll see that, you know, happening more and more over time, be all of the different communication channels as we go ahead.