Privacy risks from iOS photo metadata

There’s a ton of very personal information associated with a photo that you take with your smartphone. By default, the phone captures all of the camera settings (aperture, shutter speed, focal length). But it also captures location and timestamp. The timestamp and location from a photo, or series of photos, can be used by a domestic violence perpetrator to infer places a victim frequents, and their patterns of travel. When this information is posted via a photo sharing service or social media account, it can be an unexpected (and even unknown, silent) privacy breach. 

A Twitter conversation the other night prompted this post. A very senior graphics engineer was surprised to see how much of her personal information and travel patterns was exposed to a stalker ex-partner via photo sharing. The location history revealed by someone’s photo stream is at least as rich (and endangering) as the direct location history determined from GPS. That’s a dangerous privacy breach. If it caught a senior engineer by surprise, imagine how many non-technical smartphone customers are at risk!

All of this photo reference information is commonly referred to as metadata, but that’s an imprecise technical buzzword. Properly written messaging and photo sharing apps will educate the customer about what’s being captured, shared, and posted. “It’s not just the photo, but we’re going to tell the world where you were and when you were there. And once we post it, that information will be available forever, and indexed by all of your favorite search engines.” Many apps won’t be quite that honest. And many customers won’t pay attention. If they do pay attention, they might not remember, years later, that they had given permission, when domestic violence becomes a possibility or reality.

Apple can help this by making an iOS app’s photo sharing permissions more granular.

At the moment, there are three levels of permission for access to the camera and the camera roll. They are defined in Cocoa Keys. They are: NSPhotoLibraryAddUsageDescription (write-only access to the photo library);  NSCameraUsageDescription (direct capture of the camera image); and NSPhotoLibraryUsageDescription (full read-write access to the photo library’s images and metadata).

An additional level of granularity, call it NSPhotoLibraryImagesUsageDescription, would help. This proposed new setting would allow an app to read the images in the photo library. It would not allow photo editing, metadata editing, or metadata viewing. If a customer grants NSPhotoLibraryImagesUsageDescription access to an app, that app cannot (deliberately or inadvertently) share the customer’s position history via photos. The privacy fence would be enforced by the operating system. And that’s exactly what we want an operating system to do.

I’ve filed this as rdar://33421676 with Apple. Dupe freely!

I have no idea what the analogous answer for Android is. Drop me a note if you know, and I’ll update this post.

Updating SceneKit WWDC 2013 slides for Xcode 7

With recent changes to the AppKit headers, you need to make a couple of changes to the WWDC 2013 SceneKit Slides code to get it to build. There are some cool examples in that year’s talk/sample code that didn’t make it into 2014’s.

In the ASCPresentationViewController, switch from a method declaration for the -view superclass override to a property in the header, and specify @dynamic for that property in the implementation.

@property (strong) SCNView *view;

//- (SCNView *)view;

 

@dynamic view;

//- (SCNView *)view {

//    return (SCNView *)[super view];

//}

I also updated the .xcodeproj to current standards, and fixed a couple of int/NSInteger/NSUinteger mismatches.

I’ve submitted it to Apple as rdar://23829155. In the meantime, here are the diffs:

diff --git a/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCPresentationViewController.h b/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCPresentationViewController.h
index 7d66316..bb0e54f 100644
--- a/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCPresentationViewController.h
+++ b/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCPresentationViewController.h
@@ -55,7 +55,9 @@
@property (weak) id <ASCPresentationDelegate> delegate;

// View controller
-- (SCNView *)view;
+// Hal Mueller change: make this a property, @dynamic, to compile under Xcode 7/10.11 SDK
+@property (strong) SCNView *view;
+//- (SCNView *)view;
- (id)initWithContentsOfFile:(NSString *)path;

// Presentation outline
diff --git a/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCPresentationViewController.m b/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCPresentationViewController.m
index 46d9e00..1c914b6 100644
--- a/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCPresentationViewController.m
+++ b/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCPresentationViewController.m
@@ -91,9 +91,10 @@ typedef NS_ENUM(NSUInteger, ASCLightName) {

#pragma mark - View controller

-- (SCNView *)view {
- return (SCNView *)[super view];
-}
+@dynamic view;
+//- (SCNView *)view {
+// return (SCNView *)[super view];
+//}

- (id)initWithContentsOfFile:(NSString *)path {
if ((self = [super initWithNibName:nil bundle:nil])) {
@@ -660,12 +661,12 @@ typedef NS_ENUM(NSUInteger, ASCLightName) {

#pragma mark - Misc

-CGFloat _lightSaturationAtSlideIndex(int index) {
+CGFloat _lightSaturationAtSlideIndex(NSInteger index) {
if (index >= 4) return 0.1; // colored
return 0; // black and white
}

-CGFloat _lightHueAtSlideIndex(int index) {
+CGFloat _lightHueAtSlideIndex(NSInteger index) {
if (index == 4) return 0; // red
if (index == 5) return 200/360.0; // blue
return 0; // black and white
diff --git a/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCSlideTextManager.m b/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCSlideTextManager.m
index ce17c6f..cdc12a4 100644
--- a/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCSlideTextManager.m
+++ b/SceneKit_Slides_WWDC2013/Scene Kit Session WWDC 2013/Sources/ASCSlideTextManager.m
@@ -71,7 +71,7 @@ static CGFloat const TEXT_FLATNESS = 0.4;
return self;
}

-- (NSColor *)colorForTextType:(ASCTextType)type level:(int)level {
+- (NSColor *)colorForTextType:(ASCTextType)type level:(NSUInteger)level {
switch (type) {
case ASCTextTypeSubtitle:
return [NSColor colorWithDeviceRed:160/255.0 green:182/255.0 blue:203/255.0 alpha:1];

Options for Full Text Search in Core Data

Last weekend Chris Olds and I were discussing text search engines, and in particular how to take advantage of them to speed up searches of free-form text in Core Data. Here’s a summary of what we found. I haven’t tested or implemented any of these ideas. This is simply a summary of what’s out there.

I’m not including techniques that deal with fast searches of short text fields: normalizing your query strings and searchable text, using case-insensitive searches, etc. That’s all well documented by Apple and in the usual Core Data reference books.

I did run across one very cool article outlining a profiling method I hadn’t ever seen before. The Art & Logic Blog goes one step further in the typical use of com.apple.CoreData.SQLDebug. Take advantage of the fact that you have SQLite installed on your Mac! You can paste the SQL query being logged by your iOS app into SQLite on your Mac, and use the EXPLAIN QUERY command there to understand the search plan.

Full Text Search

Full text search (FTS) is about finding search terms within large bodies of text. This is different from matching someone’s last name to the lastName attribute in a Core Data entity. Imagine instead that your Core Data database contains notes, or newspaper articles, or patent descriptions, or travel resort reviews, and you want to search within the text of those articles. The brute force method is to scan all of the text of each article, searching for matches to the search term. That takes a very long time, and doesn’t always give you the results you want.

Ideally, your FTS within Core Data will respond as quickly as Google or Bing does when you enter a search term. The results will be ranked by relevance, The search will handle word stemming correctly: if I enter a search for “lodge”, I probably want to see results containing “lodges” or “lodging”, too. Core Data does not handle any of these need.

Roll Your Own

Michael Heyeck wrote an 8 part series of blog articles describing how to build your own FTS capability directly within Core Data, using only Core Data tools and constructs. It’s a very comprehensive series, and it’s a shame it isn’t more widely known. He doesn’t just teach you how to do FTS in Core Data. He also shows you how to read and understand the SQL queries that are generated on your behalf, and how to modify your NSPredicates and data model design to make the queries fast.

The series includes source code for a Notes application with FTS, under BSD license.

Search Kit

When you type something into the Spotlight search bar on your Mac, you’re using FTS. Mac OS X has already built an FTS index of the files on your system, and queries that index. Search Kit is the Foundation framework that Apple uses to deliver those search results, and it’s available to you too. The catch? It’s Mac only, and not integrated into Core Data.

When we were chatting, I mentioned to Chris that Search Kit would make a terrific NSHipster topic. The next day, that’s what happened! The NSHipster article also summarizes the technical issues in Full Text Search nicely.

Indragie Karunaratne has a project on Github that uses Search Kit to back Core Data searches. I’ve only read over the source, and haven’t tried it, but it looks solid. His approach is to build a Search Kit index that returns NSManagedObjectIDs of Core Data objects matching a particular full text search.

Commercial Library

Locayta makes their FTS mobile search engine available to iOS developers: free for non-commercial use, $1000 per commercial app. It’s not integrated with Core Data. An approach similar to the one Indragie Karunaratne took with Search Kit integration would probably work, though.

Hackery

The backing store most commonly used with Core Data, SQLite, includes FTS support. It’s just not exposed in any Core Data API (at least, not as of iOS 6.1).

Wolfert de Kraker describes a technique for using the SQLite FTS4 engine simultaneously with Core Data. It involves creating a Virtual Table within the same SQLite database that Core Data uses. Then he uses FMDB to create a search method which uses the FTS4 search to respond to UISearchDisplayController delegate calls. NSManagedObjectIDs are returned as the raw SQLite search results, and then Core Data retrieves these objects.

This 2010 Stack Overflow answer describes a similar approach. A different answer a few months later makes a sideways variation: instead of storing NSManagedObjectIDs in the shadow SQLite table, store SQLite row IDs as Core Data attributes.

These solutions included a custom copy of SQLite in their projects. Although they are iOS projects, I see no reason you couldn’t use the same approach on OS X.

I found two other blog posts describing other implementations of this approach, one from Regular Rate & Rhythm and one from Long Weekend Mobile, both from 2010.

I have to say that it makes me very nervous to think of mucking around in Core Data’s SQLite file. Call me superstitious.

Open Source FTS

We looked at two long-established open source FTS engines, Xapian and Lucene.

Lucene is a Java-based search engine, part of the Apache project. A port to Gnustep, Lucene Kit, was begun in 2005 and seems to have languished for a while. The most current version I found was https://github.com/zbowling/LuceneKit, which was active as recently as 2012.

Xapian is a C++ search engine, and the one that Chris uses in his production code. It is presently licensed under GPL, which would make for some complications if you were to include it  in an iOS project. There was some mention on the Xapian forum of writing an Objective-C binding. The conclusion was that it should be straightforward, but that no one has done it yet.

 

Core Data lockup with parent/child contexts on iOS 5.1

I ran into an interesting Core Data lockup problem recently. It only arises on iOS 5.1.

I have a master-detail setup, and the master tableview has 3 different sorting schemes. I wrote a method for each of these schemes to create an NSFetchedResultsController, initialize it with proper sort descriptors, give it a cache name, and perform the initial fetch.

When the tableview appears, I fire off the appropriate method to get the NSFetchedResultsController I’ll use immediately. But then I wanted to warm up the caches for the other two modes, in a background GCD queue. Here’s what it looked like:

– (void)warmUpCachesExcepting:(NSInteger)dataviewMode

{

    dispatch_async( dispatch_get_global_queue( DISPATCH_QUEUE_PRIORITY_DEFAULT, 0), ^{

        NSManagedObjectContext *childContext = [[NSManagedObjectContext alloc]

                                    initWithConcurrencyType:NSPrivateQueueConcurrencyType];

        childContext.parentContext = self.staticdataMOC;

        if (dataviewMode != tailNumberDataviewMode) {

            @autoreleasepool {

                [[self class] tailNumberFetchedResultsControllerForContext:childContext

                                                                  delegate:nil];

            };

        }

        if (dataviewMode != ownerDataviewMode) {

            @autoreleasepool {

                [[self class] registrantFetchedResultsControllerForContext:childContext

                                                                  delegate:nil];

            };

        }

        if (dataviewMode != manufacturerDataviewMode) {

            @autoreleasepool {

                [[self class] makeAndModelFetchedResultsControllerForContext:childContext

                                                                    delegate:nil];

            };

        }

        [childContext release];

    });

}

Create a new managed object context, set its parent to my original MOC, go to work on the child. This code worked fine on iOS 6. When I switch sorting orders, the change is snappy. 

When I tested this code on an original iPad running iOS 5.1, the app would hang. I did some digging on Stack Overflow and other places, and saw comments suggesting that parent/child contexts had some problems on 5.1

As it turns out, I don’t need to pass results back to the parent MOC. I just wanted a separate context for my background thread. I changed the setup to create the new MOC and then point its persistent store coordinator back to the original MOC’s PSC.

                       NSManagedObjectContext *workingContext = [[NSManagedObjectContext alloc]

                                                      initWithConcurrencyType:NSPrivateQueueConcurrencyType];

                       workingContext.persistentStoreCoordinator = self.staticdataMOC.persistentStoreCoordinator;


Works like a charm now. I didn’t dig deep enough to see exactly where and why I was hanging.

iOS 6 maps are better for applications

I’m very happy to see the new look brought to iOS maps via Apple’s switch to an in-house solution. Much noise has been made about the errors and the lost detail. I think the lost detail is an improvement.

The other night, a friend and I did a side-by-side comparison of my app HistoryPointer on iOS 6 and iOS 5 (bonus points if you can figure out where we were sitting). HistoryPointer displays a bunch of points of interest on a map (further details aren’t especially relevant). On the top is the HistoryPointer running on iOS 6 (with maps from Apple); on the bottom it’s running under iOS 5 (with maps from Google).

IMG 1009

Photo

To my eye, the iOS 6 version is much easier to read when you’re looking for the overlaid points of interest.There are fewer labels on the Apple map than on the Google map. The color scheme is less intrusive. There are some problems with label placement on the Apple map: State Route 99, for example, near the right-hand edge, is missing its street name (Aurora Avenue) on the Apple version even though other less important streets are labeled.

I see lots of potential in this move. There’s a control panel to adjust label size on map views. There’s obviously some dynamic label generation and pruning going on. I like the prospect of enhancements to MapKit to allow programmatic control of many of these parameters. Imagine what you could do with API to do these things:

  • adjust the size of (or omit!) certain kinds of labels.
  • control the orientation of the map to maximize the use of the screen space. North doesn’t always have to be up.
  • apply a custom color scheme, so that your overlaid data is easier to read.
  • omit certain kinds of features. Not everyone wants driving directions. There are many applications where the streets, highways, and manmade features are simply clutter.

Think about the possibilities. How would you like your app’s MKMapView to be different? File those radars! I have several of my own in mind.

Drobo FS and Lion: update

I just had a phone call from a Drobo senior engineer. He was very frank and direct. It was the sort of conversation two developers have when nobody from management is in the room.

Without going into detail, I have to say that I was impressed. They have of course been testing this setup thoroughly, since the very first Lion developer previews. The Drobo engineer outlined for me the testing procedures they’re using right now, to try to replicate the failures some of us are seeing. They haven’t been able to replicate it. If you can’t make something bleed, it’s hard to kill it.

If you’ve ever shipped software, you’ve faced this situation. A customer experiences some bug, maybe even an intermittent one, that you can’t reproduce yourself. It is maddeningly frustrating for both the developer and the customer.

We were on the phone for 45 minutes. He had very specific logfiles that he wanted from my system. He laid out for me the plan they have for killing this problem, and the multiple approaches seem very sound to me.

They do indeed need the performance tests that first-level support has been asking us to run.

Based on what I learned today I’m going to hang in there for a while longer.

I have created a new Time Machine share on the FS, and I’m running a backup to it now from one of my Lion machines. It’s working fine. I’m going to give it a couple more hours, then kill it, and apply the procedure that Sébastien used. The only change I’ll make is to mount the shares manually using SMB, instead of having to play Beat The Clock.

One other update is that my Snow Leopard machine, which was (immediately after the new firmware) seeing absurdly low throughput, is now functioning fine. I didn’t touch anything. I just let it work.

Drobo FS problems under Mac OS X Lion

I’ve been very disappointed with my Drobo FS during the switch to Lion. I had been using the FS for Time Machine for 3 Macs, and it had been a stable operation for 6 months.

The Lion problems seem to stem from the tighter AFP requirements in Lion. Drobo was clearly not ready. The latest firmware update has made matters worse.

When I upgraded my first machine to Lion a week ago, Time Machine could not connect to the FS. I filed a support ticket with Drobo, and received this astounding response:

We are currently in the process of a firmware update to fix apple’s more stringent AFP requirements. This is just not an issue with drobo. Although we are working on a solutionto fix the issue I do recommend making a complaint to apple if enough customers complain about what they did they may roll it back.

Read that last sentence again. Drobo wanted their customers to ask Apple to change the low-level data communication specs on a major OS upgrade, after the upgrade was released. I was dumbfounded.

Anyone who is in the software business has missed a deadline. I was inclined to cut Drobo some slack for not being ready. But that request for me to lobby Apple on their behalf really left me wondering. Were they simply not able to write a driver that worked with the new spec? Were they planning to drop Mac support? Or were they just behind schedule?

There was one other funny thing happening. When I booted the Lion machine, the Accounts/Login screen appeared, and I could move my cursor around with the mouse. But mouse clicks were ignored. It was impossible to log in! If I shut down the Drobo FS, I was able to log in to the Lion machine. I suspect some background process was hung, waiting for the Drobo. This happened every single time I booted the Lion machine.

On Monday, 4 days after Lion was released, Drobo posted a new version (1.2.0) of the firmware for the FS. Their release notes claimed that it fixed the Time Machine incompatibility. I installed it last night, and things got much worse.

With the new version of the Drobo FS firmware, neither of my machines that are still running Snow Leopard can reliably connect to the Time Machine backup shares. One machine, a Mac Mini that sees very little filesystem activity, takes 20 to 30 minutes to connect to the Drobo. The 5 to 8 MB of data transfer takes about 30 minutes. The other machine, a MacBook Pro, has not yet managed to connect to the FS for a backup. I am seeing frequent Finder freezes (every few minutes) on both Snow Leopard and Lion, and occasional crashes that require a hard reboot.

If you have machines that will remain on Snow Leopard for a while, I suggest that you not install the latest Drobo FS firmware, because it will break Time Machine and make your Finder creep. If you’re in an all-Lion environment, then it doesn’t matter which version of the firmware you use, since neither of them works.

One Twitter follower suggested the Promise DS 4600 as a Drobo replacement. I’m going to give Drobo a few more days before I give up on them completely. I have Backblaze installed on all the machines too, so it’s not a crisis for me yet.

The creation date of the latest FS firmware is July 19, two days before Lion was released, and 6 days before the firmware was released. I have a hunch that things are very busy at Drobo right now.

UPDATE Friday, July 29

I just called Drobo Support. The technician acknowledged that the 1.2.0 firmware does not solve Lion compatibility. He gave me instructions for downloading and running the test suite at http://www.aja.com/ajashare/AJA_System_Test_v601.zip to measure network performance. I haven’t done that yet.

One commenter suggested downgrading the firmware to the previous version. The Drobo support person told me that that would require the datapack to be reset, resulting in total data loss.

My personal advice is still that you skip the 1.2.0 firmware update and wait for their next attempt.