(Note to readers: Hi! This is Mike Julian, Corey’s business partner/wrangler/handler. I know you’re used to hearing Corey’s antics and opinions here, so you’re in for a special treat: me!)
In August, AWS announced that people like you and me could take the AWS Certified Cloud Practitioner Exam from the comfort of our own homes with a lot more flexibility in scheduling. Immediately, I thought it’d make for a fun story if I took it, so that’s exactly what I did.
My original plan was to quietly take the exam, and if I failed, man, that would make for a hilarious story. If I passed, I passed, and that would be the end of it. I expected to pass, given what I do, so figured the chance for a great story was minimal.
But boy was I wrong. What follows is the tale of that time I took the Cloud Practitioner exam from home, the shitshow I experienced, and how AWS and Pearson Vue are more on top of the ball than you might imagine.
Scheduling
Scheduling the exam was pretty straightforward and painless. AWS’s Certification site links into Pearson Vue’s site, and the Cloud Practitioner exam has a new option to take it remotely. Easy peasy.
A short while into scheduling, I ran into the first bug on Amazon’s side. Amazon’s Certification site (well, CertMetrics.com, a vendor to Amazon) yells at you to make sure your name matches your ID. But I couldn’t change it:
Pearson’s site also yells at you for the same reason and unhelpfully directs you back to certmetrics.com/amazon/.
Apparently, my Amazon account is old enough that it was pulled over from another system, so I had to reach out to AWS support to get it changed. That took a couple of days.
After you’ve scheduled the exam, you click on a link to make sure your system is compatible with the testing software. When I first did this, the system test was a page on Amazon’s site and it told me everything was fine. (Coincidentally, this page no longer seems to exist—this wouldn’t be the first time we’ve felt gaslighted by Amazon.)
Exam Day
Exam day! Oh boy! How exciting!
To start the exam, Pearson has you download an app called Pearson OnVUE. It has its own system test, which mine conveniently failed; it couldn’t detect the microphone or camera.
As luck would have it, there’s nowhere in the app to request assistance. The troubleshooting link in the app gives you some vague 101-level recommendations then finishes by telling you to contact your IT administrator.
Yeah, that’s helpful. 🙄
To contact Pearson, I had to go to their website and find their phone number. There didn’t seem to be a number specific to the OnVUE app. But they did answer the phones quickly at least.
Sadly, things did not improve.
After going through incredibly basic (and painful!) troubleshooting–and the rep seemingly unaware that the OnVUE even existed–Pearson’s help desk rescheduled my exam for a few weeks out while they worked on my issue. They gave me a confirmation number and said I’d hear from someone by email.
As an aside, I asked Pearson’s support rep where I could check the status of my issue online. No matter how I explained what I was asking for, the rep kept telling me someone would contact me via email. I took this to mean Pearson doesn’t have an online portal for support requests, as the rep seemingly didn’t even understand the premise of my question.
In any case, I received a followup email from Pearson the next day:
Okay, cool. So they’ve rescheduled my exam…but there’s no mention of the problems I had with the app?! And since this note is coming from a no-reply email address, I can’t reply and correct them (for the love of god people, stop doing this to your customers—it’s asinine!).
Welp.
Making Waves
A side effect of having a Professional Twitter Shitposter as a business partner and running various publications in the industry is that a lot of people start paying attention to you.
Some of those people are in positions to care about problems like these.
My tweet thread made it over to the folks at Amazon, which got the ball rolling. Two days after the testing fiasco, I received a call from an SVP at Pearson Vue. We had a lovely chat about what happened with my exam and what was going on internally with it. They asked if I would be amenable to discussing my issue more in-depth with one of their staffers, which I was happy to do.
That decision led to a 45-minute call with a VP of Product over the AWS online exam and OnVUE app. We had a great discussion about user experience, privacy, security, the difficulties in online testing, and the tradeoffs of balancing concerns around cheating and the customer experience.
I learned far more about the intricacies of online exams than I ever expected. Despite the flak Pearson gets for their testing platform, they really do put quite a lot of thought into ensuring the testing experience works well and that they deliver the perfect balance of test integrity and user privacy.
That same day, Pearson refunded the exam fee for my troubles ($100), which was a very nice gesture. I also heard from someone inside the AWS Training & Certification group who checked in to see how things were going and asked if I needed any additional assistance.
As for my problem with the OnVUE app? It all stemmed from a bug when running on OSX Mojave 10.14.6, which had released on August 26, 2019—only a week before my exam. Just generally bad timing all around, it seems. Pearson released a fix for the bug on September 10, about a week after this whole experiment began (and two weeks after the bug was introduced)
Pearson has a lot of work to do ahead of them, though
Not everything is rosy, though. I still had some pretty serious concerns about Pearson’s messaging around the exam requirements:
For an exam scheduled to last over an hour-and-a-half, these are pretty strict and, frankly, a little ridiculous.
Pearson assured me that these requirements are to set up expectations that you, the test taker, need a fairly sterile environment and if not, you could have your exam invalidated—but not that you immediately will have it invalidated.
If they used softer language here, I think it would go a long way to assuage public concerns, result in a better user experience overall, and make Pearson (and by extension, AWS) look less like the bad guy.
The System Test
Here’s an interesting bit I experienced: Despite the requirements being explicit about having no phones nearby, you do need your phone to do the environment checks prior to the exam. And my proctor specifically asked me to grab my phone, which I go into more below.
To be on the safe side, I went through the system test a few days before my exam—this time using only Pearson’s app to test. And, oh boy, do I have Opinions™.
The Pearson system test is actually the same app you use for the exam. After inputting your unique testing code and phone number, it does the camera/audio test then asks you to grab your phone for environment tests. You take a photo of your ID (front and back), then four photos of the room you’ll be testing in: one facing the desk, one behind the desk, and one each to the left and right of the desk. These are submitted to Pearson for review. Once that’s done, the system test is complete.
The day of the exam, frustratingly, you need to do this all over again. Why couldn’t the results simply be saved? 🙁
Exam Day
After taking pictures of my environment all over again, I was put into a queue where I waited for my proctors to review the information. They took their time here, so despite having all of that done by my 9:15 a.m. testing time, I ended up starting the exam about 10 minutes late.
The OnVUE app has a built-in text chat and voice call functionality, which is neat and helpful. The proctor called me via the app to do some additional environmental checks—checks which seemed excessively onerous to me.
The proctor wanted to see quite a bit of my desk, asking me to move my camera around. Since my setup is cable-tied together, this didn’t work too well. I ended up having to use the front camera on my phone in front of the webcam to act as a mirror. It was as awesome as it sounds.
I typically keep my laptop closed and stashed under my two external monitors. The proctor had me turn off both of them and use only my laptop and also wanted to see the unplugged cables. I tried to explain that just wasn’t going to happen since my webcam is fixed on top of my external monitor. Thankfully, he accepted the two HDMI cables on my desk as proof enough.
My proctor even had me remove my Fitbit from my wrist. I’m not sure what their threat model is here. But I have to assume it includes “any technology at all, anywhere, ever.”
At one point during the exam, a different proctor pinged me via chat and asked me to remove my earbuds. I had forgotten they were in from when I was speaking with the first proctor during the environment check, who never told me to remove them when the test was complete.
After finishing the exam and attempting to close the exam app, my computer locked up. Fully unresponsive. Because it hijacks some OS stuff, I couldn’t even get the Force Quit window to pop up. I ultimately had to do a hard shutdown of my laptop by holding down the power button—an experience I haven’t had in years.
Despite all this, it’s still a pretty great idea
Overall, I have mixed feelings about the whole operation. The bug and the testing experience were incredibly frustrating and I didn’t even need the certification in the first place. I set out to do this on the slim chance it might make for a good story.
Imagine how much worse this could have been for someone who actually truly needed this and didn’t have the public presence we have to get Amazon’s and Pearson’s attention so quickly.
That said, Amazon and Pearson were super responsive and involved with getting this whole thing resolved, so major props to them.
Online testing is a hard problem and they’re putting more effort into it than first meets the eye. I learned that Pearson VUE (and other testing companies like them) work quite hard on establishing a testing environment that balances the need for user privacy with test integrity, and many of the companies these testing companies are providing the environment for also have their own requirements, which can lead to some onerous rules. They’re all based on experience and their expertise in the space, and everyone is indeed trying to maintain a consistent and fair experience for all test-takers involved, so it’s kinda hard to fault them too much.
As for AWS, expanding the certification program to allow for online options is going to allow many, many more people to take the exams who otherwise might not be able to (e.g., those who live in rural areas and those who work jobs where they can’t take time off for a testing center). This is a very welcome change and I’m looking forward to seeing what comes next.
