The Division can be played any number of ways. You can go solo through the streets of Mahattan, or you can team up and form a squad to take on the different factions that await. Maybe you like to take chances with other players who might turn on you and steal your loot, making the Dark Zone your preferred play area. Whatever the case may be, The Division’s online infrastructure makes it all seamless, thanks to a complex, server-oriented system.
“The key thing for The Division was to have a seamless experience,” says says Anders Holmquist, technical director at Massive. “We want you to get into the game and play with your friends – no matter where you are are and no matter where they are – and be able to go from PvE to PvP without any gates or issues. That was one of the big challenges. There’s this entire big server environment behind all of this, like full-blown MMO architecture.”
Holmquist explains that while there are additional challenges to using this kind of infrastructure, it has given the team some flexibility they wouldn’t have otherwise. “Since most of the gameplay is actually running on the server, most things we can fix server-side, as you’ve seen through the server hot fixes,” he explains. “Most of the exploits have been fixed server-side. The problem is when the fix we’re doing server-side means that the client is showing something different than what it should.”
To listen to Division community developer Hamish Bode’s full interview with Anders Holmquist, click here. You can also find a transcript below.
What’s been your road getting to [technical director at Massive]?
AH: Actually, I started doing telecom and mobile multimedia, once upon a time. Then I got into game development, starting as a gameplay programmer. Then I ended up on The Division. I was part of the original concept team, so I’ve been there since the first design document. Somehow I ended up as both the technical director for The Division and the Snowdrop engine.
Obviously, this is a complex topic, but how would you describe the online infrastructure for The Division?
AH: The key thing for The Division was to have a seamless experience. We want you to get into the game and play with your friends – no matter where you are are and no matter where they are – and be able to go from PvE to PvP without any gates or issues. That was one of the big challenges. There’s this entire big server environment behind all of this, like full-blown MMO architecture.
By that, you mean our dedicated servers that are placed everywhere?
Before launch, as we were going through the beta phases, people were asking “where am I connecting to?” In New Zealand and Australia, we’ve been connecting to servers on the West Coast of the US and random Asian servers for so long.
AH: We do have servers in Sydney, so your friends are good. We basically have servers all over the world – in Hong Kong, both coasts of the US, and several server centers in Europe. There should be dedicated servers near you that give good pings and no lag.
A lot of people talk about how they go into the Dark Zone and there are obviously people who speak different languages, which from a gameplay perspective makes sense when you’re in New York. But a lot of people are saying they get matchmade on different language servers, but language is not one of the things we check for.
AH: Well, that’s been one of the things we’ve been talking about – what we should be matchmaking on. So, we matchmake on a lot of things, like where your friends are, people you’ve played with before, your ping to that location … we’re not matchmaking on language or region right now, or country I should say. You might end up with people from China if you’re from Europe, and so on. That can also happen, for example, if all of the European servers all full. We’ll spill you over to other servers.
Do you think that’s something that will ever change?
AH: Depends on what the community feels and what’s appreciated.
I think that’s one of those things that would be interesting to look at. Sorry, I sidelined you from how you were explaining our infrastructure.
AH: The topic of infrastructure is very big, so that’s tricky. But the way we make it so you can play with anybody is that we have core data centers that hold your characters, your account, and everything like that. And then we have smaller data centers where the actual dedicated servers are. When you want to connect to a server, that server’s going to ask for your character data. It’s going to arrive from one of the core data centers. Most of them are in Montreal, actually. That loads up your character, and then you’re ready to go on that server.
But you’re not communicating with the server while you play the game?
AH: No. That’s only when you connect. When you go into a server, you’re already being matchmade with people on the same level, with the same gear score, and friends – everything we think that will make your experience better. We’re doing that already when you connect to the server.
When you say connected to the server, is it your first entrance into the game or when you cross into the Dark Zone?
AH: No, your first entrance into the game. We’re already matchmaking you to try to have people you like nearby. But then, we do it once more when you enter either a shared space, like the hub or safe house, or if you go to the Dark Zone. As you approach the Dark Zone, we actually matchmake you again to see which Dark Zone bracket you should go into.
That’s actually one of the questions that’s been thrown at me. When does that matchmaking start happening? Is it proximity?
AH: Long before [you go through the gates]. The gates are more for gameplay purposes. A lot of people have been thinking we do a server transfer or something like that, which is not true. You’re actually matchmade long before that, because we want you to be able to look into the Dark Zone and hear what’s going on in the Dark Zone.
People have been saying, “I hear gunshots inside the Dark Zone” and some people have said “no, that’s just fabricated gunfire,” but when you cross into that Dark Zone, you’re actually in that instance where that gunfire is happening.
AH: Yeah. You can even see those players – the players that are inside your Dark Zone – you can see them from the outside because if you’re that close, you’ve already been assigned to that Dark Zone.
What are the benefits of using the system we have – this network of dedicated servers? Why do we do that?
AH: We do it for a couple of reasons, one being that it’s easier to give a good experience. We know what kind of hardware they’re running on. We know what kind of Internet connections they have. If you’re running a peer to peer connection, which is common for FPS and other shooters, you’re completely dependent on the people you’re playing with and how good their computers are or how good their Internet connection is.
Since we have dedicated servers, we can guarantee performance. We can guarantee the network flow. Another part of that is we can guarantee gameplay. Basically, the server is the one with the true game state. It sets the rules for what’s going on. It does all the combat math. It generates loot. It handles all of the things we want to have controlled. If you’re running a peer to peer game, you don’t have that control. That other computer [acting] as the host can do whatever it wants.
That makes sense. I know some people have been throwing around terms like “authoritative.” Can you explain what we have and what that means? Because there’s been some confusion around that.
AH: The basic principle of The Division is that the server is authoritative. Basically, whatever the server says goes – what happens in combat, what loot is generated, and so on. We had two exceptions to this. One is your local player’s movement and basically as you’re moving around, the server won’t correct you. The server will validate to check that you’re not going into areas you shouldn’t or you’re trying to go through walls. Things like that. But it’s not going to correct your client’s movement because if it does that then you get a very jarring experience.
The same is also true for your aiming and when your bullet hits. The server won’t correct you. It will try to validate that you made a valid shot if everything is proper, but it’s not going to correct your aim.
In this situation, the server has the power?
But there are some things in the client that have been problematic.
AH: The biggest problem we had [is that] there were features introduced that didn’t have the proper server validation. That was a big problem we had, especially during beta where we found quite a lot of features and said, “This is missing the proper checks on the server.” We fixed them – almost all of them are fixed. There are a couple of things that are harder to completely block from the server. In those cases, we added cheat detection. One of those things, for example, was the fire rate.
So we should keep fire rate client-side because it’s a better experience and then detect it on our side?
AH: That’s what we’re doing now. We have quite a lot of data on people cheating with fire rate right now. I think there will be some bans coming in the future.
Right now, it’s Friday the 29th, and you might have seen an article go out this morning about a big wave of bans that are incoming. That’s basically formulated from this new cheat detection.
AH: The cheat detection has been there the entire time.
Exactly, but we’ve upgraded that cheat detection?
AH: What we’ve been doing is adding cheat detection for certain systems, but mostly, we’ve been tweaking the numbers because we want to try avoid having false positives. It’s been running all along. Even people that we’re cheating before, we know about it, but we wanted to tweak it to make sure we knew who the true cheaters were.
We don’t want to ban people who haven’t been cheating.
AH: That would be bad, yes.
One of the things that people are a little confused about, but we added the “\report” system on PC. People are asking if we’re getting their reports, and we are definitely getting those.
AH: What I can say that we’re also doing is that when you report someone, that allows us to turn on special tracking for them, so we have even more detailed data for them. It’s definitely very useful.
You used the word “MMO” – is this setup we’re using common? Do we have something unique?
AH: The setup we have is quite common with MMOs. Where people are getting a bit confused or where it gets a bit complicated is the fact that we’re both an MMO and a shooter. People are looking at things like more traditional FPS games and shooters, where there’s a 32-player multiplayer game, and they’re comparing that to our network model. They’re not really doing the same thing. We’re running servers with thousands of players and several tens of thousands of AI doing very complicated behaviors. It’s not really the same environment. We’re more of a hybrid.
What parts of those environments do you think we can’t compare? You talked about rubber-banding and those sorts of things before. Are those the kinds of things you’re talking about?
AH: Definitely. Many of our players are playing the game like a single-player game or a co-op game. For some multiplayer games, they put the very hard caps on. For example, if you have more than 200 ping, you’re not going to be able to play. [They’re] going to kick you. That’s not something we could do, because if you’re playing alone, it’s completely ridiculous that we kick you because you have a high ping. It has to be a network model that works both as a single-player or co-op game on the MMO servers. And at the same time, [it needs] to handle multiplayer, competitive PvP. That’s why we have a very interesting hybrid solution that is different from what others are doing.
You mentioned creating that smoother PvP experience. What are the biggest things we’ve done to make that happen?
AH: The biggest thing is having the way the network is set up, where you control your movement, but it’s validated – that means you’re never going to get these corrections to your character. We’re also fairly tolerant of high ping and packet loss. We can blend that very well. Even if your opponent is lagging, we hide it fairly well. The downside of that is: “Oh, I ran behind the door, and somebody shot me through it.” Because on their client, you weren’t behind the door yet when you were shot. And how much we allow that delay – that’s something we’re tweaking right now. What kind of lag tolerance do we want to allow? How much do we want to allow a player to be lagging behind everyone else and still be able to do damage? That’s something we’re tweaking and looking at right now.
Can we go into a little more detail on that? You talked about how some people could be appearing to be shot behind doors.
AH: What happens is on your client, somebody’s shooting at you, and you run behind the door. But on their client, they’re lagging. Maybe they have bad Internet or something or they have a lot of packet loss, so they’re lagging a bit. On their screen, you still haven’t ran past the door when they’re shooting you. On their screen, the bullets are hitting you. Those bullets have to go to the server and do a judgment call, basically. How much will I allow my state to be rolled back to allow this damage? Maybe we’re a bit too generous with that right now, so we’re looking at tightening that a bit. But that’s why on your client, it will appear that you already arrived behind the door when the bullets hit.
What would you say has been the biggest challenge?
AH: I think the biggest challenge has been that we not only have the player to consider. On one server, it can easily be 15,000 active AIs at the same time. That puts a lot of constraints on how and what we simulate on the server side. For example, that’s one of the reasons for fire rate being only validated on the server. That’s been the biggest challenge, having that many players and that many AIs doing what’s single-player level of quality of AI and action in a multiplayer game – in an MMO, basically.
If those things didn’t happen on the server, then everyone would have a different experience.
What’s the benefit to doing it like this?
AH: Since most of the gameplay is actually running on the server, most things we can fix server side, as you’ve seen through the server hot fixes. Most of the exploits have been fixed server side. The problem is when the fix we’re doing server side means that the client is showing something different than what it should. Maybe the UI isn’t displaying the right number. Sometimes we make the call that, “OK, we want to wait until a client patch to fix this, so it’s obvious what’s going on and it’s showing the right thing.”
There are some things that need to be in the client patch, right? What defines something that can’t be server side?
AH: What defines something that can’t be server side is usually it’s either something input related or it’s something related to how your client is moving or physics. What we can do server side is probably do a blunt fix for it, like “OK, if this happens, we’re just going to kick you or kill your character.” But we can’t do a good fix for it. Often, we need to fix something on the client physics to make a fix that’s more proper, that doesn’t have the risk of hitting innocent people.
We don’t want to break anything by fixing something.
Can you give any specific examples of what you mean [by inputs]?
AH: I think the biggest problem we have is the mobile cover, for example. It’s causing a lot of problems because people use the physics to glitch through walls and things like that. The problem there is that we can detect when you’ve glitched through the walls, but it can happen even if you’re not intending to do it. The risk is that we hit people that are innocent that aren’t actively trying to cheat.
In all fairness, there are a lot of people who aren’t innocent, which is a problem we have. Is there anything we’re doing client side to combat those things?
AH: Yes. There are a lot of talents being fixed. There’s a lot of skills being made safer, and also when we know they’re safer client side, we know we can have more harsh penalties for it server side because we know it’s not going to happen by accident.
What would you say are some of the main things you’ve learned from this? Is there anything you didn’t expect to happen that did?
AH: There are a couple of things. I think we underestimated the amount of process to catch security issues. We need to have more strict processes for that. We also were surprised by how quickly people were cheating. This is probably our highest priority. This is what everyone is talking about. This is what everyone cares about.
We’ve said for a long time that Massive has a rich history in developing PC games, but we’re largely PC gamers as well. But we’re largely experiencing all of the issues everyone else is experiencing.
AH: As a developer, it’s very frustrating, especially when you know we can fix them.
We haven’t talked as much about trusted client and trusted server as much as I wanted to. When you’re talking about trust, what does this mean?
AH: It’s basically about who gets to decide what happens, and what you want is that the server decides as much as possible. What we have is that the server basically decides everything except for a very few small things – your client’s movement, your aiming, and some small features related to shooting that the client doesn’t decide, but has some influence over.
But you mentioned something earlier about authoritative and validating?
AH: Authoritative means the server just does it. For example, damage and health – the server has complete control over them. It will just say this is what happened. Validating is: if you’re moving, the server isn’t moving you, but it’s going to check how far you moved during this period. Was it too fast? Are you speed hacking? “OK, I think you are. I’m going to flag you and report you. You’re going to get banned.” That’s the difference, whereas for most things, you can’t even try to break the server system, because the server will just overrule you.
What’s your biggest focus as we move through the next big updates we have coming?
AH: Besides cheating and exploits, just stabilizing and trying to nail down all of the bugs. Make sure that the new Incursions work great, and that all the new content is working as it should. Those are the most important things for us right now. Just to give as good an experience as possible, basically.
Is there anything else you’d want to say out there to people listening?
AH: We’re very aware of the problems that are there, and we’re aware of them. We’re fixing them. We know who’s cheating. They’re going to get banned.