Friday, July 14, 2017

The Concurrent User Drop And The Release Cycle

To say that the number of concurrent users in EVE Online dropped over the past few years is a pretty accurate statement. When CCP Seagull became the executive producer of EVE Online in July 2014, the number of concurrent users on Tranquility was approximately 26,000. Looking at the numbers on EVE-Offline, the concurrent user numbers had dropped down to 22,000 over the past month.

Given the level of "EVE is dying!" angst seen amongst some members of the player base, I expected to see a much greater decline than the approximately 15% drop in concurrent users over the past three years. One of the hallmarks of CCP Seagull's years in charge of development is the desire to break away from the cult of the subscription and not require EVE players to maintain as many accounts as previously. In some respects, I believe the decline is a feature in her overall plans, not a bug.

I believe that one of CCP Seagull's major decisions unintentionally contributed to the decline of concurrent users, however. The decision to move from a two-expansions a year to a five week release cycle I believe cost EVE Online a lot of activity. The graphic above is one example. In June 2014, CCP launched Kronos, the last of the old style EVE expansions. Would Tranquility have experienced greater player activity if CCP had launched a Kronos-level expansion in June 2017? I think so.

During the Age of Expansions, EVE typically experienced month-over-month growth during three months: December, January, and June.

Growth during winter is expected, but why the increasing activity in June? Because CCP traditionally launched expansions in late May or June. The expansion would draw people into the game, thus mitigating the traditional summer fall off. Even Incarna followed the pattern, with the steep drop-off in player activity occurring in September, not June or July.

Perhaps surprising to many players, the largest July drop in activity occurred in 2009, not 2011. I see two causes for the drop. First, the critically acclaimed Apocrypha expansion launched in March, not in June of 2009. The second, and perhaps more important reason, was the anti-bot/anti-RMT operation known as Unholy Rage that launched on June 22, 2009.

Returning to the present day, I believe CCP could boost numbers, or at least slow the decline, by targeting the June and November content releases more like the old style expansions than just another incremental drop. Hopefully, the upcoming winter expansion is a move in that direction. Now, if we just see a summer expansion in 2018, perhaps we will see a reduction in the cries that the game is dying.

Thursday, July 6, 2017

EVE's Zombie Farms: Full And Responsible Disclosure

One of the big challenges for the developers of MMORPGs is making sure that new major features in their games integrate seamlessly with older systems. The task becomes even more critical when adding free-to-play elements to an existing mature game. Unfortunately for EVE Online's developer CCP,  the game system involved is the skill injector system that allows players to trade skill points amongst each other.

While skill point trading has several weaknesses, the issue today causing elements of the player base to pull out their pitchforks is a bug introduced in the Ascension expansion on 15 November 2016. Ascension introduced clone states, CCP's term for the system that introduced elements of free-to-play to EVE. Alpha clones, the new F2P option, differed from the subscription Omega clone option by only offering the use and ability to train a subset of skills as well as only possessing the 24 hour skill queue phased out in the Phoebe expansion. In the Ascension patch notes, CCP specifically stated the behavior of an account that lapses from Omega to Alpha state.
"Accounts lapsing from Omega to Alpha will have their training queue paused and will have previously trained skills outside the Alpha set disabled until subscription is renewed"
On 3 March, a Reddit user posted that CCP's code did not work as intended.
Since the introduction of Alpha clones it's been possible to train only certain skills as an Alpha (the way it should be IMO).

However, what I have not seen talked about is that you can fill your skill queue as an Omega, let your subscription expire, and now as an Alpha the character will continue to train those Omega skills for free at the Omega 2x speed. The skill queue only stopping if you log into that account.

It's currently possible to plex an account, fill the skill queue, and train skills for free until you decide to log in and extract. Before repeating the process again, getting multiple months worth of skill injectors for a single plex and the extractor cost.

Is this something that people consider acceptable or another thing for CCP to fix?
According to Wikipedia, an exploit, "is the use of a bug or glitches, game system, rates, hit boxes, or speed, etc. by a player to their advantage in a manner not intended by the game's designers." Given that CCP explicitly stated in the patch notes the expected behavior when an account lapses from Omega to Alpha status, what the Reddit user described in March is an exploit.

On 15 June, CCP acknowledged the exploit in an article published in the news channel:
"After thorough investigation and discussion with the Council of Stellar Management, the decision has been made to declare 'Ghost Training' an exploit.

"'Ghost Training' is defined as the use of alpha account status to accrue skillpoints at a more rapid rate than they are gained through normal alpha account gameplay, and/or train omega skills on an alpha account.

This notification serves to inform pilots that as of the date and time stamp in its header, the use of Ghost Training is considered an exploit. Any users found to have been knowingly abusing Ghost Training will be subject to reprimand on a case by case basis as per the EVE Online EULA.

We’d like to thank all the pilots who have reported this to us both publicly and via the support and bug reporting systems, and confirm that a fix is expected to be deployed to address this issue next week.

A reimbursement is also incoming for those pilots who may have lost skill training time as part of a related deployment last week, as highlighted in this news item.
Lead GM Lelouch clarified the statement on Reddit:
"You're right, the wording is a bit vague/misleading. To clarify: We are going after past abusers.

"While all exploit abuse is bad, we consider any abuse of a publicly declared exploit to be even worse. That's the distinction we were trying to make in the notification. It doesn't mean that we're going to let abuse predating the announcement slide. I apologize for the confusion.

"I want to take the opportunity to urge anyone who intentionally abused this issue to come forward by submitting a support ticket.

"I also want to make it clear that our objective is to go after those who intentionally abused this exploit.

"You have nothing to worry about if you just happened to benefit from this exploit on accident because your account lapsed for a few days. There is a pretty clear distinction between this and a skillpoint farm that's been set up to benefit from this issue."
Players who believe an exploit isn't an exploit if CCP does not make a public statement are incorrect. In the case of ghost training, utilizing the exploit runs afoul of both the EULA and the Terms of Service. Paragraph 6A6 (Conduct) of the EULA states:
6. You may not engage in any conduct that results in an Account containing items, objects, currency, character attributes, rank, or status that are inappropriate for the level or rank of the character contained in the Account, including without limitation arranging, making or accepting transfers of items to a character without adequate consideration, thereby augmenting or aggregating items in an Account and increasing its value for an Account sale.
In addition, the Terms of Service makes the following point:
23. You may not exploit any bug in EVE Online to gain an unfair advantage over other players. You may not communicate the existence of any exploitable bug to others directly or through a public forum. Bugs should be reported through the bug reporting tool on our website.
Oh. Revealing the existence of an exploit on a public forum like Reddit is a violation of the Terms of Service. How serious does CCP regard disclosing exploits? CCP's Suspension and Ban Policy states:
Severe offences may result in an immediate ban without warning; however, warnings may be given for first time offenses, followed by account suspensions of varying degree and ultimately a permanent ban if a player:
  • c. Is aware of an exploitable bug and fails to report it to Game Masters and/or distributes the information to other players.
Those who only revealed the exploit on Reddit can expect a warning, if that, compared to the immediate ban those who utilize the exploit after CCP's exploit announcement. Still, CCP frowns on the public discussion of exploits, especially exploits that not only impact the game economy but defraud the company out of real world money.

CCP's policies spell out a practice called bug secrecy. The idea is that hackers cannot exploit a bug, otherwise known as a vulnerability, if they do not know about the bug. Of course, one can see the problem with such thinking in the case of EVE Online. If a vulnerability exists, players will find the weakness and mercilessly exploit the bug.

At the opposite end of the security spectrum is a theory called full disclosure. According to Wikipedia:
"Full disclosure is the practice of publishing analysis of software vulnerabilities as early as possible, making the data accessible to everyone without restriction. The primary purpose of widely disseminating information about vulnerabilities is so that potential victims are as knowledgeable as those who attack them."
According to security technologist Bruce Schneier, bug secrecy relies on two false assumptions. The first is hackers cannot discover vulnerabilities on their own. The second is that software companies will spend time and money fixing secret vulnerabilities. Schneier argues that full disclosure is the only reason software companies patch their software:
"To understand why the second assumption isn't true, you need to understand the underlying economics. To a software company, vulnerabilities are largely an externality. That is, they affect you -- the user -- much more than they affect it. A smart vendor treats vulnerabilities less as a software problem, and more as a PR problem. So if we, the user community, want software vendors to patch vulnerabilities, we need to make the PR problem more acute.

"Full disclosure does this. Before full disclosure was the norm, researchers would discover vulnerabilities in software and send details to the software companies -- who would ignore them, trusting in the security of secrecy. Some would go so far as to threaten the researchers with legal action if they disclosed the vulnerabilities.

"Later on, researchers announced that particular vulnerabilities existed, but did not publish details. Software companies would then call the vulnerabilities "theoretical" and deny that they actually existed. Of course, they would still ignore the problems, and occasionally threaten the researcher with legal action. Then, of course, some hacker would create an exploit using the vulnerability -- and the company would release a really quick patch, apologize profusely, and then go on to explain that the whole thing was entirely the fault of the evil, vile hackers.

"It wasn't until researchers published complete details of the vulnerabilities that the software companies started fixing them."
A third school of thought on how to deal with software vulnerabilities exists called responsible disclosure. Again, according to Wikipedia:
"Responsible disclosure is a computer security term describing a vulnerability disclosure model. It is like full disclosure, with the addition that all stakeholders agree to allow a period of time for the vulnerability to be patched before publishing the details. Developers of hardware and software often require time and resources to repair their mistakes. Hackers and computer security scientists have the opinion that it is their social responsibility to make the public aware of vulnerabilities with a high impact. Hiding these problems could cause a feeling of false security. To avoid this, the involved parties join forces and agree on a period of time for repairing the vulnerability and preventing any future damage. Depending on the potential impact of the vulnerability, the expected time needed for an emergency fix or workaround to be developed and applied and other factors, this period may vary between a few days and several months. It is easier to patch software by using the Internet as a distribution channel."
In 2011, CCP's head of information security, CCP Sreegs (aka Darius JOHNSON), published a dev blog that spelled out that responsible disclosure was the model he wanted to follow in encouraging players to report security issues. The dev blog introduced the PLEX for Snitches program in which players can contact the security team at with vulnerabilities and possibly receive a PLEX reward, depending on the severity of the exploit found.

Perhaps the most famous EVE exploit covered under the responsible disclosure policy occurred in 2012. A group of five players from Goonswarm manipulated the new faction warfare system to earn a tremendous amount of loyalty points before informing the security team about the exploit. After receiving information about the exploit, CCP made several changes to prevent anyone else from taking advantage of the vulnerability.

The group of players, although they did report the exploit to CCP, didn't quite follow the guidelines. CCP Sreegs explained the rationale behind the resolution of the situation:
"The people who sought to benefit from this exploit will receive no gain from this system. Because this was essentially a system where you could print LP, even if ISK was provided as an input, it is classified as an exploit.

"Because the players made efforts to inform us about the issue their accounts will remain in good standing. We have temporarily seized all LP points and store items from them. Once we're done determining how much each person has benefitted we will remove the LP gained value in LP and items and return the ISK invested in the purchase of items to them. This essentially will set each of them back to the original point at which they began this activity. The person who reported the issue will receive the usual PLEX for Snitches reward.

"I wrote a blog on 'Responsible Disclosures' a year or so ago. In that blog I mention that telling us about something after you've used the heck out of it isn't what we consider to be responsible. We do our best to be lenient in cases such as this but we want this to serve as a notice to the community that the proper time to alert us to the issue was before actually using the system. I can understand a desire to test the limits but we don't believe two weeks of testing a bug or exploit should net a tremendous benefit in lieu of reporting it in the first place, and that is another reason why the LP activity will be reversed back to zero."
I mention the aftermath of the factional warfare exploit for one main reason. The five individuals in question avoided bans since they reported the exploit, but CCP confiscated their profits. Following the same logic, all of the individuals involved in using the Ghost Training exploit in the operation of their skill point farms should not expect to keep their profits. For those who never reported the exploit, or reported the exploit but continued to profit from the bug, I expect harsher treatment. I believe the words in CCP Sreegs' 2011 dev blog still hold true:
"Computers aren't very good at logging intent and believe it or not there are documented cases where people who are out to do bad things have lied about their intentions. If we're witnessing an exploit being taken advantage of in our logs then, from our perspective, an exploit is being taken advantage of and the consequences for such actions are not light."