Friday, May 05, 2017

The Price of Everything

#PowerSwitch The relationship between the retailer and the customer can be beset by calculation on both sides. The retailer is trying to extract enough data about the customer to calculate the next best action, while the customer is trying to extract the best deal.

There is nothing new about customers comparing products and prices between neighbouring shops, and merchants selling similar goods can often be found in close proximity in order to attract more customers. (This is especially true for specialist and occasional purchases: in large cities, whole streets or districts may be associated with specific types of shop. London has Denmark Street for musical instruments, Hatton Garden for jewellery, Saville Row for made-to-measure suits, and so on.)

But nowadays the villain, apparently, is eCommerce. As a significant share of the retail business migrates from the high street to the Internet, many retailers are concerned about so-called showrooming. It may seem unfair that a customer can spend loads of time in the high street, wasting the time of the shop assistants and shop-soiling the goods, before purchasing the same goods online at a better price. To add insult to injury, some people not only practice showrooming, but then blog about how guilty it makes them feel.

The assumption here is that the Internet can generally undercut the High Street, and there are several reasons why this assumption is plausible.
  • Internet businesses compete on price rather than service, so the prices must be good.
  • An internet store can provide economies of scale - serving the whole country or region from a single warehouse, instead of needing an outlet in each town.
  • An internet store can offer a much larger range of goods without increasing the cost of inventory - the so-called Long Tail phenomenon
  • An internet store typically has lower overheads - cheaper premises and fewer staff
  • An internet business may be run as a start-up, with less "dead wood". So it is more agile and less bureaucratic. 
However, there are some counterbalancing concerns.
  • The economic and logistical costs of delivery and return can be significant, especially for low-ticket items. With clothing in particular, customers may order the same item in three different sizes, and then return the ones that don't fit.
  • Investors previously poured money into internet businesses, and the early strategic focus was on growth rather than profit. As internet business become more mature, investors will be looking to see some decent returns on their investment, and margins will be pushed up.
  • And then there is differential pricing ...
One of the key differences between traditional stores and online stores is in pricing. Although high street retailers often drop prices to clear stock - for example, supermarkets have elaborate relabelling systems to mark-down groceries before their sell-by date - they do not yet have sophisticated mechanisms for dynamic pricing. Whereas an online retailer can change the prices as often as it wishes, and therefore charge you whatever it thinks you will pay. According to Jerry Useem,
"The price of the headphones Google recommends may depend on how budget-conscious your web history shows you to be."
I heard Ariel Ezrachi talking about this phenomenon at the PowerSwitch conference in Cambridge a few weeks ago. (I have not yet read his new book.)
"There is an assumption is that the internet is a blessing when it comes to competition. Endless choice. Ability to reduce costs to close to zero. etc ... What you see online has very little to do with the ideas we have of market power, market dynamics, etc. everything is artificial. It looks like a regular market, with apples or fish. But because it’s all monitored, it’s not like that at all. What you see online is not a reflection of the market. You see “the Truman Show” — a reality designed just for you, a controlled ecosystem." (via Laura James's liveblog)

In his play Lady Windermere's Fan, Wilde offered the following contrast between the cynic and the sentimentalist.
Lord Darlington: What cynics you fellows are!
Cecil Graham: What is a cynic?
Lord Darlington: A man who knows the price of everything and the value of nothing.
Cecil Graham: And a sentimentalist, my dear Darlington, is a man who sees an absurd value in everything, and doesn’t know the market price of any single thing.

According to one of the participants at the PowerSwitch conference, some eCommerce sites quote higher prices for Apple users, based on the idea that they are less price-sensitive and can afford to pay more. In other words, the cynical Internet regards Apple users as sentimentalists.

If there is an alternative to this calculative thinking, it comes down to reestablishing trust. Perhaps then retailers and consumers alike can avoid an artificial choice between cynicism and sentimentalism.




Emma Brockes, I found something I like in a store. Is it wrong to buy it online for less? (Guardian, 3 May 2017)

Ariel Ezrachi and Maurice Stucke, Virtual Competition: The Promise and Perils of the Algorithm-Driven Economy (Harvard University Press, 2016) - more links via publisher's page

Laura James, Power Switch - Conference Report (31 March 2017)

Joshua Kopstein, Is Amazon Price-Gouging You? (Vocativ, 4 May 2017) via @charlesarthur

Jerry Useem, How Online Shopping Makes Suckers of Us All (Atlantic, May 2017)

Price-bots can collude against consumers (Economist, 6 May 2017)

The Dilemma of Showrooming, (Daniels Fund Ethics Initiative, University of New Mexico)


Related posts: Predictive Showrooming (December 2012), Showrooming and Multi-Sided Markets (December 2012), Showrooming in the Knowledge Economy (December 2012).

Monday, May 01, 2017

Lawrence Wilkes

My friend and former colleague Lawrence Wilkes died on Friday, after a short illness. Lawrence and I joined James Martin Associates (JMA) on the same day in 1986, so we had known each other for half a lifetime.

JMA was a small consultancy advising organizations on the use of the Information Engineering Methodology and assisting Texas Instruments in developing and supporting the IEF toolset. The Information Engineering part of the company was acquired by Texas Instruments Software in 1991. In 1997, it was sold to Sterling Software and many of us left the company. David Sprott and Lawrence set up the CBD Forum (later CBDI Forum), as a think tank for component-based development, evolving into component-based development and integration, and then evolving into service-oriented architecture (SOA).

As David has written in his fulsome tribute, he and Lawrence spent several years explaining SOA to the large technology companies, including IBM, Intel, Microsoft and Sun. (I can add that they had an article on SOA in the very first issue of the Microsoft Architecture Journal, and I co-wrote something with him for the second issue.)

For my part, I collaborated frequently with them, and became a regular contributor to the monthly CBDI Journal. When the CBDI Forum merged with the US-based consulting firm Everware, I joined Everware-CBDI as a full-time consultant for a few years, working with Lawrence and others to develop a substantial knowledgebase for service architecture and engineering. Although many of us contributed content, it was Lawrence who provided the overall structure and turned our contributions into a coherent whole.

Lawrence was a tireless innovator and perceptive industry analyst, generous with his energy and insight to colleagues and friends. It was a shock when I learned of his illness and forced retirement, and a further shock to learn of his quick demise. I will miss him.


Links

Lawrence Wilkes Blog, Slideshare

David Sprott and Lawrence Wilkes, Understanding Service-Oriented Architecture (Microsoft Architecture Journal 1, January 2004)

Lawrence Wilkes and Richard Veryard, Service-Oriented Architecture: Considerations for Agile Systems (Microsoft Architecture Journal 2, April 2004)

David Sprott, Remembering Lawrence Wilkes – SOA Pioneer (30 April 2017)

Tuesday, April 25, 2017

Uber's Self-Defeat Device

Uber's version of "rational self-interest" has led to further accusations of covert activity and unfair competitive behaviour. Rival ride company Lyft is suing Uber in the Californian courts, claiming that Uber used a secret software program known as "Hell" to invade the privacy of the Lyft drivers, in violation of the California Invasion of Privacy Act and Federal Wiretap Act.

This covert activity, if proven, would go way beyond normal competitive intelligence, such as that provided by firms like Slice Intelligence, which harvests and interprets receipts from consumer email. (Slice Intelligence has confirmed to the New York Times that it sells anonymized data from ride receipts from both Uber and Lyft, but declined to say who purchased this data.)

It has also transpired that Apple caught Uber cheating on the iPhone app, including fingerprinting and continuing to identify phones after the app was deleted, in contravention to App Store privacy guidelines. Uber CEO Travis Kalanick got a personal reprimand from Apple CEO Tim Cook, but the iPhone app remains on the App Store, and Uber continues to use fingerprinting worldwide.

Uber continues to be massively loss-making, and the mathematics remain unfavourable. So the critical question for the service economy is whether firms like Uber can ever become viable without turning themselves into defacto monopolies, either by political lobbying or by covert action.




Megan Rose Dickey, Uber gets sued over alleged ‘Hell’ program to track Lyft drivers (TechCrunch, 24 April 2017)

Mike Isaac, Uber's CEO plays with fire (New York Times, 23 April 2017)

Andrew Liptak, Uber tried to fool Apple and got caught (The Verge, 23 April 2017)

Andrew Orlowski, Uber cloaked its spying and all it got from Apple was a slap on the wrist (The Register, 24 Apr 2017)

Olivia Solon and Julia Carrie Wong, Hell of a ride: even a PR powerhouse couldn't get Uber on track (Guardian, 14 April 2017)


Related Posts

Uber Mathematics (Nov 2016) Uber Mathematics 2 (Dec 2016) Uber Mathematics 3 (Dec 2016)
Uber's Defeat Device and Denial of Service (March 2017)

Saturday, April 08, 2017

Another Update on Deconfliction

As the situation in Syria goes from worse to worser, the word "deconfliction" has reappeared in the press. On Friday, following a chemical attack on the Syrian population apparently by the Syrian government, the USA bombed a Syrian government airbase.

 "Russian forces were notified in advance of the strike using the established deconfliction line. US military planners took precautions to minimize risk to Russian or Syrian personnel located at the airfield," said a Pentagon spokesperson.

A few hours later, the Russian Foreign Ministry announced it was suspending the deconfliction agreement, accusing the Americans of "a gross, obvious and unwarranted violation of international law".

The normal purpose of deconfliction is to avoid so-called "friendly fire". But in the case of the deconfliction line in Syria, a more practical objective would be to avoid minor incidents that might escalate into major war. (Anne McElvoy quotes a senior former British commander in Iraq talking about the jeopardy of the next crucial months in Syria: "powers tripping over each other – or America hitting the Russians by accident".) We might fondly imagine that the Pentagon and the Russian Foreign Ministry still share this objective, and will continue to share a limited amount of tactical information for that purpose, despite public disavowals of coordination. Deconfliction as minimum viable coordination.

Much less serious, and therefore more entertaining, is the "friendly fire" that has meanwhile broken out within the White House. Gun metaphors abound (cross-hairs, opened fire). Successful businessmen understand the need to establish clear division of responsibilities and loose coupling between different executives - otherwise everyone needs to consider everything, and nothing gets done. But this is not a simple matter - excessive division of responsibilities results in organizational silos. Large organizations need just enough coordination - in other words, deconfliction. It is not yet clear whether President Trump understands this, or whether he thinks he can follow President Roosevelt's approach to "creative tension".



Bethan McKernan, Syria air strikes: US 'warned Russia ahead of airbase missile bombardment' (Independent, 7 April 2017 11:42)

May Bulman, US air strikes in Syria: Russia suspends agreement preventing direct conflict with American forces (Independent, 7 April 2017 15:39)

Matt Gertz, Breitbart takes on Jared Kushner: Steve Bannon is shielded as Trump’s son-in-law is in the crosshairs (Salon, 6 April 2017)

Matt Gertz, To Defend Bannon, Breitbart Has Opened Fire On The President's Son-In-Law (Media Matters, 6 April 2017)

Anne McElvoy, Washington is confused by Trump’s act. What became of America First? (Guardian, 9 April 2017)

Reuters, Kushner and Bannon agree to 'bury the hatchet' after White House peace talks (Guardian, 9 April 2017)


Related Posts

What is Deconfliction? (March 2008)
Update on Deconfliction (November 2015)
The Art of the New Deal - Trump and Intelligence (February 2017)

Thursday, March 30, 2017

Right to Repair

One of the interesting dynamics of the service economy lies in the dialectic opposition between open and proprietary. I have mentioned some useful conceptual models in previous posts: Amin and Cohendet have proposed a model that classifies capabilities/services according to the dimensions of knowledge intensity and trust; meanwhile, Max Boisot's iSpace model traces the dynamics of knowledge from proprietary to open.

In my post on the New Economics of Manufacturing (Nov 2015), I described some of the economic forces behind the shift away from manufacturing products (including spare parts) and towards services.

Instead of trying to sell you overpriced tyres, the car manufacturer must make sure that only its accredited partners have the software to balance the wheels properly. In other words, not just architecting the product or even the process, but architecting the whole ecosystem.

But consumers (and regulators) are fighting back. Car owners in the USA have already won the right to repair, and now the farmers of Nebraska are now fighting a similar battle against the tractor manufacturers. True openness would force the manufacturers to publish the repair manuals as well as the interfaces, and allow independent repair shops and knowledgeable consumers to repair their own equipment without relying upon some dodgy download or counterfeit component.

This matches the Boisot model of stuff flowing from the proprietary world into the open world. I'm sure there will be more examples of this to come ...




Jason Koebler, Five States Are Considering Bills to Legalize the 'Right to Repair' Electronics (Motherboard 23 Jan 2017)

Jason Koebler, Why American Farmers Are Hacking Their Tractors With Ukrainian Firmware (Motherboard, 21 March 2017)

Gabe Nelson, Automakers agree to 'right to repair' deal (Automotive News, 25 January 2014)

Olivia Solon, A right to repair: why Nebraska farmers are taking on John Deere and Apple (Guardian, 6 March 2017)


Related posts

Knowledge and Culture (April 2006)
Tesco outsources core eCommerce (March 2009)
Ecosystem SOA (October 2009)
The New Economics of Manufacturing (November 2015)






Thursday, March 16, 2017

From Dodgy Data to Dodgy Policy - Mrs May's Immigration Targets

The TotalData™ value chain is about the flow from raw data to business decisions (including evidence-based policy decisions).

In this post, I want to talk about an interesting example of a flawed data-driven policy. The UK Prime Minister, Theresa May, is determined to reduce the number of international students visiting the UK. This conflicts with the advice she is getting from nearly everyone, including her own ministers.

As @Skapinker explains in the Financial Times, there are a number of mis-steps in this case.
  • Distorted data collection. Mrs May's policy is supported by raw data indicating the number of students that return to their country of origin. These are estimated measurements, based on daytime and evening surveys taken at UK airports. Therefore students travelling on late-night flights to such countries as China, Nigeria, Hong Kong, Saudi Arabia and Singapore are systematically excluded from the data.
  • Disputed data definition. Most British people do not regard international students as immigrants. But as May stubbornly repeated to a parliamentary committee in December 2016, she insists on using an international definition of migration, which includes any students that stay for more than 12 months.
  • Conflating measurement with target. Mrs May told the committee that "the target figures are calculated from the overall migration figures, and students are in the overall migration figures because it is an international definition of migration". But as Yvette Cooper pointed out "The figures are different from the target. ... You choose what to target."
  • Refusal to correct baseline. Sometimes the easiest way to achieve a goal is to move the goalposts. Some people are quick to use this tactic, while others instinctively resist change. Mrs May is in the latter camp, and appears to regard any adjustment of the baseline as backsliding and morally suspect.
If you work with enterprise data, you may recognize these anti-patterns.




David Runciman, Do your homework (London Review of Books Vol. 39 No. 6, 16 March 2017)

Michael Skapinker, Theresa May’s clampdown on international students is a mystery (Financial Times, 15 March 2017)

International students and the net migration target: Should students be taken out? (Migration Observatory, 25 Jun 2015)

Oral evidence: The Prime Minister (House of Commons HC 833, 20 December 2016) 


TotalData™ is a trademark of Reply Ltd. All rights reserved

Thursday, March 09, 2017

Inspector Sands to Platform Nine and Three Quarters

Last week was not a good one for the platform business. Uber continues to receive bad publicity on multiple fronts, as noted in my post on Uber's Defeat Device and Denial of Service (March 2017). And on Tuesday, a fat-fingered system admin at AWS managed to take out a significant chunk of the largest platform on the planet, seriously degrading online retail in the Northern Virginia (US-EAST-1) Region. According to one estimate, performance at over half of the top internet retailers was hit by 20 percent or more, and some websites were completely down.

What have we learned from this? Yahoo Finance tells us not to worry.
"The good news: Amazon has addressed the issue, and is working to ensure nothing similar happens again. ... Let’s just hope ... that Amazon doesn’t experience any further issues in the near future."

Other commentators are not so optimistic. For Computer Weekly, this incident
"highlights the risk of running critical systems in the public cloud. Even the most sophisticated cloud IT infrastructure is not infallible."

So perhaps one lesson is not to trust platforms. Or at least not to practice wilful blindness when your chosen platform or cloud provider represents a single point of failure.

One of the myths of cloud, according to Aidan Finn,
"is that you get disaster recovery by default from your cloud vendor (such as Microsoft and Amazon). Everything in the cloud is a utility, and every utility has a price. If you want it, you need to pay for it and deploy it, and this includes a scenario in which a data center burns down and you need to recover. If you didn’t design in and deploy a disaster recovery solution, you’re as cooked as the servers in the smoky data center."

Interestingly, Amazon itself was relatively unaffected by Tuesday's problem. This may have been because they split their deployment across multiple geographical zones. However, as Brian Guy points out, there are significant costs involved in multi-region deployment, as well as data protection issues. He also notes that this question is not (yet) addressed by Amazon's architectural guidelines for AWS users, known as the Well-Architected Framework.

Amazon recently added another pillar to the Well-Architected Framework, namely operational excellence. This includes such practices as performing operations with code: in other words, automating operations as much as possible. Did someone say Fat Finger?




Abel Avram, The AWS Well-Architected Framework Adds Operational Excellence (InfoQ, 25 Nov 2016)

Julie Bort, The massive AWS outage hurt 54 of the top 100 internet retailers — but not Amazon (Business Insider, 1 March 2017)

Aidan Finn, How to Avoid an AWS-Style Outage in Azure (Petri, 6 March 2017)

Brian Guy, Analysis: Rethinking cloud architecture after the outage of Amazon Web Services (GeekWire, 5 March 2017)

Daniel Howley, Why you should still trust Amazon Web Services even though it took down the internet (Yahoo Finance, 6 March 2017)

Chris Mellor, Tuesday's AWS S3-izure exposes Amazon-sized internet bottleneck (The Register, 1 March 2017)

Shaun Nichols, Amazon S3-izure cause: Half the web vanished because an AWS bod fat-fingered a command (The Register, 2 March 2017)

Cliff Saran, AWS outage shows vulnerability of cloud disaster recovery (Computer Weekly, 6 March 2017)

Sunday, March 05, 2017

Uber's Defeat Device and Denial of Service

Perhaps you already know about Distributed Denial of Service (DDOS). In this post, I'm going to talk about something quite different, which we might call Centralized Denial of Service.

This week we learned that Uber had developed a defeat device called Greyball - a fake Uber app whose purpose was to frustrate investigations by regulators and law enforcement, especially designed for those cities where regulators were suspicious of the Uber model.

In 2014, Erich England, a code enforcement inspector in Portland, Oregon, tried to hail an Uber car downtown in a sting operation against the company. However, Uber recognized that Mr England was a regulator, and cancelled his booking. 

It turns out that Uber had developed algorithms to be suspicious of such people. According to the New York Times, grounds for suspicion included trips to and from law enforcement offices, or credit cards associated with selected public agencies. (Presumably there were a number of false positives generated by excessive suspicion or √úberverdacht.)

But as Adrienne Lafrance points out, if a digital service provider can deny service to regulators (or people it suspects to be regulators), it can also deny service on other grounds. She talks to Ethan Zuckerman, the director of the Center for Civic Media at MIT, who observes that
"Greyballing police may primarily raise the concern that Uber is obstructing justice, but Greyballing for other reasons—a bias against Muslims, for instance—would be illegal and discriminatory, and it would be very difficult to make the case it was going on."
One might also imagine Uber trying to discriminate against people with extreme political opinions, and defending this in terms of the safety of their drivers. Or discriminating against people with special needs, such as wheelchair users.

Typically, people who are subject to discrimination have less choice of service providers, and a degraded service overall. But if there is a defacto monopoly, which is of course where Uber wishes to end up in as many cities as possible, then its denial of service is centralized and more extreme. Once you have been banned by Uber, and once Uber has driven all the other forms of public transport out of existence, you have no choice but to walk.




Mike Isaac, How Uber Deceives the Authorities Worldwide (New York Times, 3 March 2017)

Adrienne LaFrance, Uber’s Secret Program Raises Questions About Discrimination (The Atlantic, 3 March 2017)

Saturday, February 04, 2017

Personalized emails (not)

Here's a sample from my email inbox, which arrived yesterday.

Dear Richard
I know how important your organization's big data strategy is. That's why I want to personally invite you to attend our webinar. 

How does he know? Is he basing his knowledge on big data or extremely small data? I'm curious to know which.

And what is his idea of a personal invitation? Does he think that personalization is achieved by having his email software insert my first name into the first line? Gosh, how very customer-centric!

But at least the email arrived at a civilized time. Unlike the one that arrived as I was getting into bed the other night, from an eCRM system whose idea of personalization didn't extend to checking what time zone I was in. I guess one must be grateful for these small mercies.

Sunday, January 01, 2017

The Unexpected Happens

When Complex Event Processing (CEP) emerged around ten years ago, one of the early applications was real-time risk management. In the financial sector, there was growing recognition for the need for real-time visibility - continuous calibration of positions – in order to keep pace with the emerging importance of algorithmic trading. This is now relatively well-established in banking and trading sectors; Chemitiganti argues that the insurance industry now faces similar requirements.

In 2008, Chris Martins, then Marketing Director for CEP firm Apama, suggested considering CEP as a prospective "dog whisperer" that can help manage the risk of the technology "dog" biting its master.

But "dog bites master" works in both directions. In the case of Eliot Spitzer, the dog that bit its master was the anti money-laundering software that he had used against others.

And in the case of algorithmic trading, it seems we can no longer be sure who is master - whether black swan events are the inevitable and emergent result of excessive complexity, or whether hostile agents are engaged in a black swan breeding programme.  One of the first CEP insiders to raise this concern was John Bates, first as CTO at Apama and subsequently with Software AG. (He now works for a subsidiary of SAP.)

from Dark Pools by Scott Patterson

And in 2015, Bates wrote that "high-speed trading algorithms are an alluring target for cyber thieves".

So if technology is capable of both generating unexpected events and amplifying hostile attacks, are we being naive to imagine we use the same technology to protect ourselves?

Perhaps, but I believe there are some productive lines of development, as I've discussed previously on this blog and elsewhere.


1. Organizational intelligence - not relying either on human intelligence alone or on artificial intelligence alone, but looking for establishing sociotechnical systems that allow people and algorithms to collaborate effectively.

2. Algorithmic biodiversity - maintaining multiple algorithms, developed by different teams using different datasets, in order to detect additional weak signals and generate "second opinions".





John Bates, Algorithmic Terrorism (Apama, 4 August 2010). To Catch an Algo Thief (Huffington Post, 26 Feb 2015)

John Borland, The Technology That Toppled Eliot Spitzer (MIT Technology Review, 19 March 2008) via Adam Shostack, Algorithms for the War on the Unexpected (19 March 2008)

Vamsi Chemitiganti, Why the Insurance Industry Needs to Learn from Banking’s Risk Management Nightmares.. (10 September 2016)

Theo Hildyard, Pillar #6 of Market Surveillance 2.0: Known and unknown threats (Trading Mesh, 2 April 2015)

Neil Johnson et al, Financial black swans driven by ultrafast machine ecology (arXiv:1202.1448 [physics.soc-ph], 7 Feb 2012)

Chris Martins, CEP and Real-Time Risk – “The Dog Whisperer” (Apama, 21 March 2008)

Scott Patterson, Dark Pools - The Rise of A. I. Trading Machines and the Looming Threat to Wall Street (Random House, 2013). See review by David Leinweber, Are Algorithmic Monsters Threatening The Global Financial System? (Forbes, 11 July 2012)

Richard Veryard, Building Organizational Intelligence (LeanPub, 2012)

Related Posts

The Shelf-Life of Algorithms (October 2016)