FLIGHT #1 SPEC & GOALS
In case you missed this in our last development blog, here’s a quick catch-up on what the first flight entailed and what our key goals were:
The first public flight was designed to help stress test all the core matchmaking updates the team has been working on over the past few months. This major foundational overhaul and modernization work is really at the core of the biggest changes to MCC – many of the issues and areas of feedback fans have been most vocal about stem from the game’s original matchmaking systems.
One important note is that this flight was intentionally setup to run with peer-to-peer networking. You may recall that we’ve already announced that following the official retail update, all MCC MP matchmaking games will run via dedicated server. However, for this flight, the dedicated servers weren’t quite fully ready for prime time and we still needed to validate and test peer-to-peer networking (which is what will be used for Custom Games, among other scenarios).
Additionally, this first flight also included several other changes such as the slightly updated main menu UI, the removal of the “vote” button, and the inclusion of “continuous matchmaking.” HDR video settings were also enabled just prior to the flight officially kicking off.
To help validate updates to our network transport layer and improvements to the matchmaking and MP experience for players, the team looked at two key data points:
- Matchmaking success rate
In both of these metrics, the team set baseline goals that would represent improvements over the current retail MCC offering and validate that these new updates are working as intended. To facilitate this data, the first flight was setup with a targeted content and experiential offering:
- Halo 2 Anniversary Multiplayer on four maps
- Team Slayer to 25 kills / 5 min game timer
The flight build was setup this way to really promote throughput above all else – the act of searching and joining games and continuously rolling from game to game was the most important aspect of the flight. Approximately 2,150 players were invited to participate in the first flight and were chosen based on a few different criteria (one of which being those who were most active in MCC MP in the past six months). We decided to leave the build accessible full time once it was available, but to help focus efforts and the population we landed on three official playtime windows:
- Friday, April 27: 7PM to 10PM PST
- Saturday, April 28: 10AM to 1PM PST
- Sunday, April 29: 4PM to 7PM PST
Outside of the specific goals for the game itself, this first flight also carried some unique goals of its own:
- Establish the proper processes and procedures to release game builds (and rapidly iterate fixes/updates as needed)
- Ensure our MCC Message of the Day (rebuilt on new/updated internal services) worked as intended
- Validate our Xbox Insider Hub dependencies and toolsets
- Activate our MCC Insider Program participant pool and run the complete communications cycle (initial confirmation, build availability outreach, survey follow-up)
FLIGHT #1 RESULTS
Thanks to you all, we did it! The team has deemed the first flight a resounding success. Not only did we hit our key goals for time-to-match and matchmaking success, we also feel pretty good about the results of the other misc. goals noted above.
Let’s break it down:
- More than 14,000 total matchmaking attempts over the course of the three official playtest sessions
- Matchmaking success rates significantly exceeded Flight 1 goals
- Time-to-match results were significantly faster than our Flight 1 goals. The vast majority of players were into matches in less than 38s. (that’s pretty quick!)
From a purely technical level, metrics and the story the data told exceed expectations. We’re still waiting on a final apples-to-apples comparison between some of this flight data and the current retail MCC equivalent (it’s complex to align the various telemetry pieces) but it’s clear that the matchmaking and networking updates have already offered some big improvements. Our data folks also need to take into account things like population size and geographic locale, among other factors.
Like me, you may be wondering, “why would there ever be a case of an unsuccessful match?” It could really boil down to all sorts of things from legitimate errors (which we’re actively hunting for), to NAT incompatibilities, to just the nature of the internet itself (particularly in the case of a few of our participants outside of North America who surely had some challenges connecting to a peer-to-peer lobby across the world).
GAME BUILD RELEASE PROCESS
Overall the process of compiling and releasing builds went smoothly. You may not even realize this but from the initial time the first flight became available for download in the marketplace several updates were released before the weekend.
First, thanks to feedback from players who jumped in right away on Wednesday, the team was able to identify and resolve a ‘back fill’ issue that was sometimes resulting in an error with the 8th member of a lobby wasn’t correctly filled. This issue was quickly addressed with an internal build, tested, and then deployed quickly and flawlessly on Thursday.
The team also released another build update just prior to the Friday official playtest that enabled the new/updated HDR video settings in the main menu. That wasn’t so much driven by response to an issue as it was just a feature that was now ready and they were eager to get it out into the wild.
Lastly, we also received feedback and validated that somehow an audio issue was introduced that resulted in the main menu music being muted. This too was quickly fixed, tested, and released late Friday night following the first session, ensuring things were in good order for the weekend.
To get a sense of the types of feedback we received from the first flight, please welcome Dr. Jessie Thomas – a member of the Xbox Research Team who not only administers the MCC Insider Program for us but is also at the helm of the Halo Community Feedback Program. Take it away, Jessie!
INSIDER SURVEY RESULTS
Hiya MCC Insiders!
Just to reiterate, a huge THANK YOU to all who participated and filled out your surveys. Your feedback is extremely valuable and a crucial part of driving improvements to MCC. In an effort to maintain high transparency with our Insiders, I’d like to share a portion of the survey results from Flight 1.
Upon sign up to the MCC Insider Program, folks were asked to indicate how much they agree with a series of statements regarding the technical quality of the retail version of MCC as a baseline. We then asked this same set of questions after Flight 1 to allow for comparison.
As shown below, Flight 1 received statistically higher ratings across all five categories demonstrating the improved technical quality of the Flight 1 build. However, the degree of improvement across the categories varied. While we saw big improvements in ratings for connectivity and build stability, there was a smaller improvement in the rating for performance issues like lag and framerate issues.
In general, Flight 1 respondents felt very positive about the MCC Insider Program so far. Metrics regarding build functionality, expectations, developer communication, and visual quality received the highest scores with over 50% strongly agreeing it was a positive experience. As we expected, there was a more neutral response to the amount of content in Flight 1 due to the limited scope.
We also received lots of great qualitative feedback from our MCC Insiders on what they liked and disliked about their Flight 1 experience. Here are the Top 3 Likes and Dislikes as well as some quotes from our Insiders explaining their perspective.
- Quick Matchmaking
“Matchmaking speeds were so fast that many times, I was in a match less than 30 seconds after I’d selected to play H2A Anniversary.”
“I thought the matchmaking was a great improvement and I was pleasantly surprised at how quick I was able to find a match and start playing. Keep up the great work, and I can’t wait for the rest of the updates!”
2. Continuous Play
“I really liked how quick I was able to get into each match, and how I was able to stay within the matchmaking playlist between each of the matches.”
3. New UI
“Menu change was amazing. Found games pretty fast. Much more user friendly.”
1. Lag / Framerate Issues
“The lag during each match. Tested on multiple networks/connection types; still experienced severe lag (particularly on Warlock and Shrine)”
2. Intense matches due to initial population and skill-based matchmaking not enacted
“Appeared to be a complete lack of any skill-based matchmaking leading to tons of lopsided matches with usually one or two people holding that team down”
3. Desire for other games/modes
“Halo 2 Anniversary was the only option, it got a little frustrating playing the same 4 maps but other than that it was a very nice experience overall”
That’s all for now! Be sure to keep a lookout on your emails for invitations to future flight sessions, announcements, and surveys from the MCC Insider Program. You guys are awesome!
KEY TAKEAWAYS & LEARNINGS
While it may not seem like much, successfully executing this first flight was a big milestone for the team. Not only was it the first public testing of significant networking and system work at the foundation of MCC, it was also the first of what will be an ongoing public flight process that we wish to continually streamline and improve upon.
Communication – We are working on improved processes and means with which to communicate with flight participants. As we try to maintain confidentiality, it does create challenges where we can’t just use a Waypoint forum to post known issues and important news. We’re exploring options to do better at two-way communication next time. However, participants should also get in the habit of checking the Xbox Insider Hub app regularly, as that is a key source of up-to-date information.
Latency – One thing we heard from some players is that the game was “laggy” for them. Since we were using peer-to-peer networking with a small, geographically diverse population, this is generally to be expected and in many cases somewhat unavoidable. The team will continue to keep a close eye on this as we move into Flight #2.
Performance – Some players reported performance drops in certain areas on some of the Halo 2 Anniversary Maps. The team is aware and will continue to monitor.
Sweaty – When we picked some of the most active MCC players to participate – and there was no skill-based matchmaking in play – we expected things to get a bit sweaty. And boy did they. If you’re like me, you likely got accustomed to losing badly as you found the flight #1 player population to be full of lethal killers that never missed a shot.
Early Access – We pushed the flight build out on Wednesday, a full two days before the first official playtest, and it was great to see so many people immediately jumping in. In fact, that initial response exceeded our expectations, helped us uncover a few early issues, and is something we’ll look to do going forward.
Content & Engagement – The first official play session on Friday had the largest number of players with a slight drop across each of the next two days. We’re going to look at ways to leverage more/different content and/or further focus activities to help players stay interested over the span of the entire flight.
We also heard your feedback that you’d really like it if there was a greater variety of content. For this flight, we focused the offerings to really drill into our specific goals while also juggling a desire to keep the download size manageable and keep the initial experience aligned with what we felt was one of MCC’s more broadly liked and consistent offerings. Going forward, we’ll definitely be mindful of these tradeoffs and when possible, include more content to keep things a bit more interesting.
GGs – For me personally, I’d just like to add that this first flight was a really cool experience. In many ways, it really reminded me of the old, early days of playing Halo 2 or Halo 3 right before launch. We had a small, tight group of players who were super passionate and really wanted to play and participate. People were using voice chat and it was pretty common to face the same opponents and teammates multiple matches over which made the whole thing feel unusually quaint. Total strangers would send me “GG” messages after thoroughly kicking my ass and overall the whole thing just had a special, tight knit vibe to it that’s harder to come by nowadays with massive online populations. It was such a communal experience, and a weird deja vu experience for me, that I literally didn’t mind only winning 2 out of my first ~25 games.
SPECIAL THANKS – OVER ACHIEVERS
While we’re supremely grateful and thankful for everyone who put time in to help out with our first MCC Insider Flight, we do want to give a few super duper extra special shout-outs to some players we feel really overachieved in this area!
Officially, it’s not a contest – and of course it’s hard to single out just one member of our beloved community – but if we have to, then it simply has to be “Fishmvn.” They managed to play the most matches (209), rack up the most wins (156), accumulate the most kills (2070), and get the most assists (498) across our three official MCC Insider flight play sessions.
Fishmvn, we salute you!
Here’s a look at the top 5 players with the most matches played:
- Fishmvn – 209
- NitsudustiN – 165
- FRiCTiON just – 154
- Halo Pro – 134
- Vengeful Vadam – 132
NEXT FLIGHT – COMING SOON!
The team is continuing to make progress on the overall MCC update (now working on “Milestone 6” for those keeping score) and they’re starting to prepare for Flight #2. We’re still locking on the finer details and specifics but here’s a quick snapshot of some of the current thinking:
- High level goal: Test dedicated server deployment and ongoing stability work
- Participants: Overall flight population will be increased (specifics TBD but looking at ~2X the first pool). Yes, participants from the first flight will continue to roll into subsequent flight rings.
- Metrics: Dedicated server performance, time-to-match, match success, # of errors
- Content: Based on your feedback, we intend to mix things up next flight and offer different/more content. We’ll have locked details to share soon.
Remember, anyone and everyone can join the MCC Insider Program to help shape the future of MCC. Whether you get into this next flight or not, we’ll have plenty more opportunities ahead.
You can read more about the MCC Insider Program and sign up HERE. Also, remember that one way you can improve your odds of getting selected for a flight is to PLAY MCC!
Thank you again to everyone who joined us for this first flight and who has offered their support as the team works through this ambitious update to MCC. We’ll have more info to share soon!