Mon 22 Sep 2008
A lot has been made of the issues with phone service on iProvo and the blame laid at the feet of World Wide Packets (now owned by Ciena). After getting a techincal overview of what’s going on with the devices, it appears that the blame is well-placed. As promised, earlier, here’s the explanation as to why the WWP portals are a big bucket of fail and how UTOPIA managed to dodge most of those issues.
The central problem is that the portals used in iProvo were built around MGCP, a protocol that competed with (and lost to) SIP. MGCP is designed so that the switch handles all of the calling features, including things like generating dial tone. The portal, then, is a “dumb” device without a lot of processing power. Once it became clear that SIP had beaten MGCP in the marketplace, WWP issued a patch to add SIP support to the devices. This introduced many of the problems that users are now familiar with.
The primary problem is that the portals were never built with enough processing power to handle all of the client-side things that SIP is supposed to do (including, of all things, dial-tone generation). This frequently led to processing delays and software locks. Also consider that this patch is not a proper implementation of SIP, but rather an emulation stack designed to translate SIP commands into MGCP so that the portal could understand it. It goes without saying that this emulation layer adds even more processing overhead to the mix with even more problems. With all of this taken into account, it’s pretty obvious why the phone issues on iProvo ended up being so bad for so many customers.
That begs the obvious question: why did iProvo stick with the WWP devices despite the obvious problems with them? It’s simple: money. They bought 10,000 of the things right off the bat and instead of writing off the portals as a loss or only using them for non-phone customers, they just kept hooking them up. Nuvont and Veracity ended up deploying their own IADs and ditched using the voice features of the portal to avoid those issues. UTOPIA had detected the MGCP issues early on and forbade providers from using the built-in voice features of the portals, instead requiring them to use their own IAD. It was a smart move that kept voice working stably on UTOPIA.
Because the problems with the WWP portals are caused in large part by the hardware, no software patch can right them. The portals will have to be replaced or, as a much cheaper solution, an IAD will need to be given to each phone user instead of using the voice features of the portal. In either event, it serves as a warning to not bet too heavily on a single technology. Whatever bulk discount iProvo got on the WWP devices was almost certainly eaten up in customer churn.
September 23rd, 2008 at 7:46 am
So, you’re placing the blame of the voice problems on iProvo? That’s surprising. It was never mandatory that a service provider use the phone adapters built into the WWP CPE. The way I see it, is that it would be the responsibility of the service provider to test and verify proper interoperability with a given phone IAD before using it. How is this different than testing whatever external IAD the service providers use now? There have been problems with several brands of external IADs as well that have since been abandoned. Granted, it’s unfortunate that the WWP CPE isn’t the best IAD out there, but I never expect it to be, nor would I expect that from any fiber CPE.
September 23rd, 2008 at 8:20 am
I don’t think that I placed all of the blame on iProvo. I think the management made a bad call in their technology bets and didn’t compensate for it by taking the same policy stance that UTOPIA did. It’s nothing on the techs; it’s not like there was a whole lot they could really do about it, you know? Given that MSTAR was relying so heavily on the integrated IAD, I can see how they would constantly be calling the NOC and driving the techs up the wall.
No, it wasn’t mandatory to use it, but given that using it didn’t incur additional costs to the service provider, there was a strong incentive to do so. You can certainly tell the difference between MSTAR and Veracity in how they handled those problems. MSTAR tried to muddle their way through it whereas Veracity recognized the problem and took steps to correct it. Of course, the financial situations between the two companies was also very different.
September 23rd, 2008 at 10:19 am
As far as Brandon’s comment goes he failed to mention that iProvo did not allow service providers to put any sort of external IAD on thier battery backup. There is actually law that requires that, I will have to look it up. From a service provider point of view it was either buy IAD’s and battery back-ups for every customer (which was pretty ridiculous having 2 battery back-ups on a customer’s wall when 1 was more than capable of handleing the WWP device and the IAD.)Which they usually didn’t have money to do, so customer’s would not have any phone service in the event of a power outage. Or use the WWP device and just tell customers some features like Caller ID Call Waiting would not be available. The service providers couldn’t get the best of both worlds because iProvo said they could not touch thier battery back-up’s. Utopia on the other hand used the same battery back-up’s and had no problem with service providers using the battery back-up. Which I can tell you from experience has no issues what so ever. I was one of the people who tested the power consumption of the Sipura’s and PAP2T’s which Veracity and MSTAR used. And it was so little it was hard to even monitor.
So I guess what I am really saying is it came back to iProvo employees having some sort of complex about “THEIR” equiptment and “THEIR” network being protected from service providers. Which was the attitude across the board until Broadweave came in and actually started working as a team with providers, or at least making some sort of initiative to do so.
September 23rd, 2008 at 10:56 am
I used the WWP for phone service for a couple of years. Then I had Nuvont install an IAD. The phone service was better, but it wasn’t plugged into the battery backup.
I was waiting for the Broadweave deal to close to request that I get an IAD that was compatible with the battery backup. When it was revealed that Nuvont would continue to be my provider, I sent a support email to Nuvont requesting that my IAD be plugged into my battery backup. Two days later a service tech stopped by and replaced the device with a new one that is plugged into my battery backup.
A pleasant experience, all around. Thanks Nuvont.
September 23rd, 2008 at 11:02 am
That is true, Veracity/Nuvont has been doing that for years on the iProvo UPS’s. They just do it, I am not sure if they ever got permission from iProvo to do it either. Well it doesn’t matter anymore, I am sure broadweave doesn’t care. Here is my question, why did Nuvont come out? I thought Broadweave was the service provider?
So they kick Mstar off thier network then buy data transport from them and allow Nuvont to stay on? That is interesting? Jesse, any comments on that?
September 23rd, 2008 at 11:28 am
Matt: Most of this is hearsay, so take it that way.
The word on the street is that MSTAR was ready to fight the sale until Broadweave pulled out the Club of Millon-Dollar Debt +5 and more-or-less bullied MSTAR into backing off. They traded the debt load for the customer list. Since Broadweave didn’t have a path to the Internet and likely no content licenses for video (especially when the deal with Veracity collapsed), buying from MSTAR made sense for them. MSTAR needed the money, Broadweave needed the services.
Since there was no club to wield against Nuvont, they could continue operating under their ad-hoc prior arrangement though I’m surprised Broadweave didn’t try and use the lack of a contract against them. It might have something to do with the business partnership announced between Nuvont and Broadweave on some kind of new VoIP provider? And there’s no way that Broadweave would threaten Veracity when they use them for their voice trunks.
This is pretty much my understanding of what’s gone down and why there is a bifurcated network. Another wrinkle is that MSTAR signed up a bunch of VERY low-margin MDUs that comprised a significant chunk of their subscriber base. Those customers had a wholesale ARPU of less than $12 per month and typically only subscribe to a single service. I imagine that’s why Broadweave doesn’t want to talk about the financial numbers.
September 24th, 2008 at 7:40 am
Jesse, Matt,
Did you read the part about the Sipura IAD performing well for istilllikeprovo? It could have been done this way for Veracity and Mstar from the start. I have heard that Provo engineers actually warned them about using the WWP on-board SIP IAD, but management on both sides insisted on a hurried, untested SIP, deployment based upon using the WWP IAD and several different voice switches that were never certified to be compatible by either WWP or the switch manufacturers. They ignored the iProvo engineer’s recommendations. Of course this was expedient due to the sudden demise of Homenet which should never have been a provider on the network anyway. On the other hand there were always IADs available that had certified compatibility with the switch of the day. So bottom line, iProvo management and the providers chose to save maybe $100K to hang external IADs and totally blew the phone service.
The idea that the power supply battery backup is legally mandated is absurd. Vonage and a few hundred other Internet based phone services with millions of customers, have no battery backup. What good is the battery backup in an electrical outage when your digital phone that requires 110 VAC power won’t work anyhow?
WWP actually wrote something like 20 firmware updates for their portal trying to support the providers voice switch moving target. The providers made no effort to get compatible and were often a year behind in allowing deployment of WWP’s firmware updates for their customer portals. Imagine blaming a manufacturer for compatibility issues when you are not even using the latest, greatest software. Absurd!
By the way, the problems on the portals SIP platform was not about resources, but always about SIP communication. The WWP portals had problems talking to the provider chosen voice switches.
It’s almost like iProvo management and service provider management went out of their way to provide crappy phone service. It did not have to happen.
The really sad part is that no one learned the lessons that needed to be learned here. Broadweave still thinks a portal based IAD is the way to go and will needlessly replace all the WWP portal with the next disaster. IADs do not belong on the CPE. IADs have a whole different life cycle and need to be provider driven.
October 2nd, 2008 at 8:52 am
The contract for all service providers on the iProvo network placed the responsibility clearly on the service provider to insure that their phone switch/solution was fully tested and compatible with the WWP portal.
Any service provider could elect to install external IAD as they had to do in UTOPIA. UTOPIA’s original plan called for using the built in IAD and they were unable to make this work, or were focused only on making it work with AT&T their original savior super provider. At that point they decided to just stop trying and move to the use of external IAD’s.
The use of external IAD’s of course adds additional cost. You are paying for the IAD built into the portal (both UTOPIA and iProvo portals had a built in IAD they were paying for) and then you don’t use the IAD you paid for when you bought the portal and add an external IAD. This also eliminates a port on the portal from being used for data or video. In UTOPIA’s case often requiring the addition of a switch to provide video to 2-3 TV’s that most homes have. Another additional cost.
One thing that is correct is that there were some at iProvo (mostly engineers) that were not bending over backwards to support service providers using anything that belonged to iProvo and made some decisions that made life and operation by service providers “difficult”. This was made clear by the consultants report or ask anyone that worked for a service provider.
But all service providers on the iProvo network clearly knew they were contractually responsible for making their phone solution work with the WWP Portal. WWP was contractually bound to assist them.
…and yes, WWP provided many patches and fixes that service providers were very, very slow at testing and applying.
In hindsight, it appears the movement is to no longer attempt to build the IAD into the portal. The savings on the portal cover some of the cost of the IAD. I think UTOPIA may now be using portals without the IAD built in?
But of course since they are now focused on business and NOT residential installs it might not really matter?
October 2nd, 2008 at 9:03 am
Jesse, you point to Provo not doing anything about the portal problem because they had purchased 10,000 portals up front and didn’t want to spend the money to replace them.
I would suggest that replacing them would not be the smart thing to do. The problem can be fixed by using an external IAD as UTOPIA did to fix the problem with the IAD in their portal. Adding an IAD might add $50 while replacing a portal might cost $500.
UTOPIA had a similar (but worse) issue with their purchase of 10,000 Amino 120 set top boxes. Everyone had known that these boxes cause video to freeze on a regular basis on about 100+ channels. Yet UTOPIA has done nothing to replace these boxes. Replacing a set top box costs much less than replacing the portal, and there is no easy work around like just adding an IAD to fix the problem.
It’s possible UTOPIA’s deal with CSI Digital for headend services could fix the problem.
October 2nd, 2008 at 10:31 am
It’s not just iProvo that didn’t want to spend money to replace them; it’s that providers largely didn’t want to spend money for their own IAD either. Veracity was service-driven enough to do this on their own. MSTAR, the largest provider, decided to cheap out and not do it, then blamed iProvo for all of the phone problems. Their expectation seemed to be that iProvo was responsible for making sure the IAD worked.
Believe me, I’d be very glad if UTOPIA dumped the Amino 120. I haven’t heard a single positive thing about the device from any source. In the end, however, I think UTOPIA is realizing that they aren’t responsible for the STBs, the providers are. All UTOPIA (or any wholesale network) should do is provide a solid data link between the customer premises, the service provider’s POP and resources such as a common video head-end.
October 2nd, 2008 at 11:39 am
Perhaps the “NEW” UTOPIA sees it should not be responsible for set top boxes, but the reality is UTOPIA is the one that bought the boxes, owns the boxes and rents them to service providers.
…and refused to pay the money to fix their problem.
I do agree the set top box should have been the responsibility of the service provider from the beginning.
The service provider would NEVER have bought 10,000 boxes at once, they would never have bought all HDTV boxes. As soon as they found the problem they would have started buying some other box. UTOPIA had (has) thousands in the warehouse and just keeps issuing them.
And Mstar continues to take the calls from the unhappy customers for video problems that are UTOPIA’s responsibility.