There’s pros and cons for having a tightly-controlled marketplace which requires app certification.   The main advantage is that you have a better ratio of higher-quality, non-malware apps in the marketplace.   The main disadvantage is that the certification process experience is entirely dependent on the quality of tester you get.

I’ve been trying to release Location Sender, the free app I built as an example for the PMVP framework, for 8 days now.   It has failed certification 3 times so far, with half the issues being legitimate, and the rest trivial or downright wrong.   I love getting certification failed notices when the issues are legitimate, because that keeps normal users from experiencing bugs.   Of course, the best solution is to do more robust testing in the first place so the bugs aren’t there and I pass certification the first time, but we all know that’s less likely to happen.

But I hate it when the tester was obviously being lazy or obtuse and fails my app submission over issues that are not applicable or were fixed in a previous round.   That’s another 2+ days wasted.

My first submission passed certification, but I had submitted it under the name “Here I Am!” – but I didn’t do my due diligence and search to see if that name was already being used, which it was.   So I pulled the app after it passed because I didn’t want to be a jerk to the other app’s creator.   So I changed the name to Location Sender, and resubmitted it.   Much to my surprise, it failed certification that it had passed 2 days ago under a different name.   That was silly, but I was ok with it because they were legitimate issues that the first tester didn’t catch – namely 1.1) I didn’t provide a way for the user to opt-in/out of location services and 1.2) I didn’t include a privacy policy.

So I modified the app to resolve both those issues, and made a common developer mistake: added a new feature in a bugfix release.   I also added in a way for the user to opt-in/out of app analytics (I use Preemptive’s RIS), but I didn’t do enough testing on it, which brought me to my 2nd certification failure.   The issues this time: 2.1) The application crashes when analytics are disabled and has to be reinstalled and 2.2) the app does not provide a way for the user to prevent it from running under the lock screen.

Issue 2.1 was totally legitimate and my fault for not testing it fully, and quite frankly I’m ashamed that I didn’t catch it before testing.   So I removed analytics entirely until I can debug the issue.   However issue 2.2 is puzzling because my app DOESN’T run under the lock screen!   I looked in my code, and I have references to a utility for the lock screen setting, however, it was never enabled.   So i’m guessing that MS analyzes the code for references to that setting, and the tester never bothered to check if it was being used – only that the app had that reference and that I didn’t provide a way to disable a functionality that could never be enabled anyway.   Sigh.   So it’s half the tester’s fault for not bothering to check and it’s half my fault for leaving useless code in.   So I fixed these issues and resubmitted, for the 4th time.

Imagine my shock when the app failed certification for a third time!   This time the issues were non-existent, non-applicable, and illegitimate – I was furious that they are making me waste another 4 days to get this app out there.   They had 2 issues:
3.1) That the application violates the prohibited app policy because they somehow thought that I required the user to buy something else to enable full functionality.   I have no idea where the tester got this idea, and apparently they didn’t either, as they didn’t give me any feedback as to why they thought my app violated their policy.   An app that is free, the source code is freely available, has no ads, has no references to websites which require money, etc.   The tester got it completely wrong here and now they’re just pissing me off.

Also, the other issue 3.2) was the same as 1.1, where the tester claims I didn’t provide a way for the user to opt-in/out of location services!   Which is total BS, because I fixed that issue 2 submissions ago!   I added in a dialog that pops up on first run and a settings page, I don’t know how I can make it any clearer.   Obviously the tester was being lazy and completely missed the settings page with the “Disable Location Services Access” button.

So I resubmitted the same exact XAP file and left some scathing comments in the tester notes.   We’ll see if the app passes this time or if I end up getting banned from the marketplace for hurting their tester’s feelings.