Endless memory mode not deleting uploaded photos

[kleptophobiac]

I have an X2 Connect card and an EOS XSi. The camera is set up to take pictures every 10 seconds in the lowest resolution possible (each JPG ends up weighing in at 1.3MB or so).

I'm trying to take timelapse photographs of an extended construction project (six sessions of ~36 hours each). The X2 is set in endless memory mode with a 30% threshold. My "online" mode is configured to send the photos to an FTP server on my local network. All of this works just fine, and the photos are rolling in to the FTP in near real-time.

Midnight rolled over and I noticed that there was no new folder showing pictures for the new date. So I busted out a ladder and climbed up into the rafters and found that the card was full (!!) and my camera hadn't been taking pictures during much of the process I was trying to document.

As far as I know I'm running the latest firmware on the card. It was registered just this week and I haven't seen any buzz about a new firmware since.

Any help would be much appreciated. Thanks!

-Sasha

[MikeV]

Endless memory mode requires that the card be power-cycled on a regular basis. The actual deletion process occurs over several power on-power off cycles, as a regular consumer would be doing when taking pictures.

Endless Memory does NOT function if the card is powered on all the time.

See this thread regarding someone else's issue doing similar time-lapse photos, with posts from Eye-Fi about how Endless Memory functions.

viewtopic.php?f=2&t=2986

[kleptophobiac]

Hm, I see. Given that the time between exposures is only 10 seconds and that I am running for ~36 hour intervals, I don't think I'll be able to pull off a 3x45 second power cycle several times during the course of a session. Are there any other workarounds (hacks/cheap tricks welcome)? If not, I think I'm going to use a netbook at the end of a long USB cable instead. The EyeFi is cool, but it's going to be for my simpler/more casual shooting.

I got the card specifically for timelapse applications, and I feel like the hardware is probably capable of doing exactly what I want. Overall the product feels like it's this close to being exactly what I was hoping for. It's an amazing feat of hardware engineering, but I feel like a lot of the more prosumer/geek-friendly software features are missing. In particular, direct FTP and true endless memory modes would be exceedingly useful and don't seem unattainable. I read the posts explaining away why they aren't implemented, but I must admit to feeling unimpressed after reading them. The reasoning sounds like it's coming from a stubbornly principled defeatist. Maybe you can't do SSL and passive FTP. Maybe on-the-fly deletion doesn't work with all cameras. But that's no reason to give up on them, just make sure that the limitations are made explicit. Some functionality is better than no functionality.

While I suspect EyeFi will never do this, offering an SDK to allow people to make their own firmware for the cards would open them up to a lot of really interesting uses beyond casual photography. It would also be an easy way to shut up people like me asking for new features: "Oh yeah, we don't have the time to code that. Here's the URL for the SDK and the community forum. Have fun!"

Incidentally, does the Eye-Fi card directly interface to the onboard flash and act as an intermediary between the camera and the memory, or does it multiplex the SD interface the camera and the onboard microcomputer? The packaging is so tight that I often feel an uncontrollable curiosity pushing me to xray the card or tear it apart just to check it out. Luckily it usually happens late at night when I'm too tired to break my new toy.

[MikeV]

I'm not going to comment further, but I'm sure that one of the Eye-Fi employees will be more than happy to chat with you regarding the actual limitations of what they can do with their hardware, and why they do what they do with it.

Many have asked for some kind of SDK, and I have no doubt that if they were to open the card itself to third-party development, there would be an explosion in the number of specialized uses for the card... but it's their R&D that has gone into the hardware, and I'm sure they want to maximize their investment in everything before turning the reins over to the rest of the world to do with their hardware what they will.

[mazingaZ]

Indeed I prefer so far the way it worked before the last update. The it was sufficient one power cycle to delete all the uploaded pictures.
Please consider the option to allow to delete images without to turn the camera off and on at all!!!
I'm pretty sure we are not the only 2 people that need this feature!!!

Endless memory mode requires that the card be power-cycled on a regular basis. The actual deletion process occurs over several power on-power off cycles, as a regular consumer would be doing when taking pictures.

Endless Memory does NOT function if the card is powered on all the time.

See this thread regarding someone else's issue doing similar time-lapse photos, with posts from Eye-Fi about how Endless Memory functions.

viewtopic.php?f=2&t=2986

[berend]

Given that the time between exposures is only 10 seconds and that I am running for ~36 hour intervals, I don't think I'll be able to pull off a 3x45 second power cycle several times during the course of a session.

Sasha,

I am not sure exactly what camera setup you are using for your time-lapse work, but I don't think there's anything about the parameters you've specified above that is "unworkable" with how endless memory works. Does the camera that you're using allow for itself to power-cycle in automated fashion or do you have it permanently powered-on or have it on an external power-cycler? Have you looked into using CHDK, by the way?

I got the card specifically for timelapse applications, and I feel like the hardware is probably capable of doing exactly what I want. Overall the product feels like it's this close to being exactly what I was hoping for. It's an amazing feat of hardware engineering, but I feel like a lot of the more prosumer/geek-friendly software features are missing. In particular, direct FTP and true endless memory modes would be exceedingly useful and don't seem unattainable.

This close does not make a workable feature, unfortunately. While you may think the reasons for the implementation are arbitrary, believe me, we have worked with cameras from all brands and the differing types of block, file-system metadata and media metadata caching behaviors that they all implement in the name of "efficiency" run a very wide spectrum. Given that the failure mode when endless memory's "behind the scenes" manipulation of the file system would clash with the camera's operation in a looser implementation is complete data loss due to file system inconsistency, entertaining ideas like "just productize it and let me select unsafe operation, maybe it is compatible with my camera" is not a realistic production direction. There are absolutely no widely-used cameras on the market that do zero caching of file system data and would therefore be completely fine with the file system changing behind their backs arbitrarily.

I read the posts explaining away why they aren't implemented, but I must admit to feeling unimpressed after reading them. The reasoning sounds like it's coming from a stubbornly principled defeatist.

It is not from a stubbornly principled defeatist. It is a well-informed, realistic and keeping the entire picture of running a consumer-electronics start-up company in mind perspective, instead of one that brushes aside "the devil's in the details" aspects and attempts to do too much with half-features, leading to an inevitable lack of focus and execution. Heck, if we were stubbornly principled defeatists, we would have never stood in front of a board room full of investors and made a case for designing our own chips, when what they funded a few years ago earlier was an off-the-shelf hardware integration and software development company.

Maybe you can't do SSL and passive FTP. Maybe on-the-fly deletion doesn't work with all cameras. But that's no reason to give up on them, just make sure that the limitations are made explicit.

The limitations that you accept are different than the limitations that Jane accepts which are different than the limitations that John accepts. If you go down this path of half-features in a widely-used consumer product, pretty soon you have a nightmare of a science project that cannot meet anyone's expectations satisfactorily.

Some functionality is better than no functionality.

Exactly. Think about that some more please.

While I suspect EyeFi will never do this, offering an SDK to allow people to make their own firmware for the cards would open them up to a lot of really interesting uses beyond casual photography.

Along that vein, have you looked in to using one of the open-source Eye-Fi Helper substitutes (one Python based and at least one C++-based) to eliminate your need for using the standard software for the time-lapse usage without needing to resort to FTP?

Incidentally, does the Eye-Fi card directly interface to the onboard flash and act as an intermediary between the camera and the memory, or does it multiplex the SD interface the camera and the onboard microcomputer? The packaging is so tight that I often feel an uncontrollable curiosity pushing me to xray the card or tear it apart just to check it out. Luckily it usually happens late at night when I'm too tired to break my new toy.

Eye-Fi has its own in-house-designed custom ASIC that is the camera SD slave interface, the NAND flash controller and the ARM-based SoC platform on which the firmware runs. Here's a short blog that we had put out a while ago when the X2 version first launched (with a photo of the internals to save you the trouble of X-raying):

http://www.eye.fi/blog/inside-scoop-on-the-pro-x2

Berend

[berend]

Indeed I prefer so far the way it worked before the last update. The it was sufficient one power cycle to delete all the uploaded pictures.

I guarantee to you that it was not sufficient to power cycle once to delete all the uploaded pictures. It has always taken a number of power cycles to have the photos be deleted and the space be freed. The behavior change that you are seeing is simply related to at which stage in the multiple power-cycles and in batches of how many photos the items disappeared from the camera's display. That they disappeared did not mean that the space was freed and the end to end behavior has not changed, other than striking a balance between starting any pending uploads more readily instead of getting stuck in doing too much endless memory processing on every power-on.

Please consider the option to allow to delete images without to turn the camera off and on at all!!!

It has been considered many times before from the inception of the endless memory feature and I can tell you unequivocally that it will never happen. Digital cameras are not designed to work with media storage cards that spontaneously change their contents.

Berend

[mazingaZ]

Hi Berend,
after the update you sent me the card is working even worse. Now the upload fails eaach time after several hours and the images that are deleted are a small set.
What I can't understand is how is possible you can delete few images but not a higher number for each power cycle.
When a new firmware upgrade will be released? In the meantime please can you send me the old one. At least with that the upload on the laptop never failed.

Thanks,

Davide

[phedders]

It has been considered many times before from the inception of the endless memory feature and I can tell you unequivocally that it will never happen. Digital cameras are not designed to work with media storage cards that spontaneously change their contents.

Berend

However if Sonycanikopansanyo released a camera/firmware that _could_ cope - if signalled correctly... then the EF and the camera could work together... no?

Seriously EFI must be talking with the manufacturers about working more magic

[mazingaZ]

Hi Berend,
after the official update the uploading never failed.
The camera has been powered down 8 time and neither 1 photo has been deleted!!!
Also during the shooting I've found the camera with "no memory card" error displayed.

I really think there is something very wrong here.
Please can you do something to fix these problems?

Davide

[Atys]

Hi,

I just can't believe that the cams caching the fat, maybe they shadowing it whitch is far enough not to work parallel on the fs.
As i noticed on the pics its seems that the ar6001 emulates the sd for the cam, so ecos could fake a lot bigger card and handle the fs/upload
internally to never get it full, assuming there is a wifi connection and fast enough.
Or simply just emulate a much bigger volume and keep the card in busy state while the photo is uploaded. Would be a nice feature!:)

[Denis_L]

I have another question - does the deletion of photos happen immediately on power on, or it takes some time after power on to delete? In which exact moment of time the deletion takes a place?

And one more - is it planned to implement realtime deletion of photos in some far future or it's not supposed at all?

Thanks,
Denis

[MikeV]

Just look back to Berend's post in this same thread, and this post as well to get all kinds of details on how Endless Memory works, and why realtime file deletion is not possible.

[Denis_L]

This answers my second question, but regarding the first one - I got even more questions instead of answers now. I'm a developer, implementing handling for EyeFi card behavior in our camera, and I need extremely exact information on how EyeFi card processes files, in order to handle it correctly.

What I understand now is that it needs 4 (?) power cycles to initiate files deletion, and file uploads are prioritized over files deletion. So, am I right, that file deletion could happen not immediately after power on, but it could take some time if there are some files already queued for upload? If so, then can we figure this time out or not?

And moreover, if this will take some time, then won't it be the same as real-time deletion, since the camera will be already working and managing its files, when the deletion will happen? If no, then what is the difference?

Can anybody please answer these questions or point me, where I can get the answers?

Thanks, Denis

[dcalcutt]

How disappointing. 4 cyles to delete the older images.
I'm taking this piece of crap card back.

The card should be deleting the pictures autoamtically. Never mind the cycling the the card.

Please, us professional uses need a card that can autoamtically upload the pictures to a FTP site and autoamtically delete the files once the uplaod is finished. Plain and simple.

How designed it this way? What purpose is there in doing it any other way?

[Denis_L]

Is it still possible to get answers to my questions (viewtopic.php?f=2&t=3117#p14852) from EyeFi support or not? I'd be very glad, if so.

[Denis_L]

Just still want EyeFi to answer these simple questions (viewtopic.php?f=2&t=3117#p14852). What is the problem to shed some light on how Endless memory mode works in detail?

[berend]

And moreover, if this will take some time, then won't it be the same as real-time deletion, since the camera will be already working and managing its files, when the deletion will happen? If no, then what is the difference?

No, it's not the same thing as real-time deletion. A deletion from the user's point of view is a set of many file system operations that happen to take place all at once to remove the file from the directory listing and free the space taken by the file. The novelty of our approach is that staging those operations in a specific order and requiring that power-cycles take place before proceeding further to certain steps provides for certain assumptions about what part of the operation has been unequivocally committed to the file system and not subject to a collision with the camera's file system cache. For a more detailed explanation, you'll have to wait for the USPTO to make our patent application public.

Best regards,

Berend

[Denis_L]

Sounds good, but what we see is that our file system database crashes and camera freezes, when EyeFi performs this deletion in Endless mode. Looks like we anyway need to handle this on camera side to avoid problems. Just one more small question - Is the information on how to properly handle it in camera is also unavailable until your patents become public?

[Denis_L]

And btw, can you say, how soon this this information will be available? Maybe some rough estimate?

[berend]

Sounds good, but what we see is that our file system database crashes and camera freezes, when EyeFi performs this deletion in Endless mode.

Please PM me and let me know what this camera is and what exactly you mean with "our file system database" - if you're implying that you have an index of items on the card that's aside from the file system representation itself and don't expect anything that's in that index to be removed unless your camera executes the deletion, then there could be issues with consistency there. I would suggest that anytime you encounter such an inconsistency, you rebuild the index, but it's hard to offer suggestions with vague information. Like I said, PM me and let me know what this camera is.

Looks like we anyway need to handle this on camera side to avoid problems. Just one more small question - Is the information on how to properly handle it in camera is also unavailable until your patents become public?

The novelty of the approach is that to a very large extent (hard to say or imply 100% certainty), with a reasonable well-implemented FAT file system, Eye-Fi's endless memory technology should not require the host device to know or do anything to interoperate. Some natural error handling (for missing files) may be required if the camera chooses to have a secondary index of the items on the card (beyond the implicit index that is the file system itself), but that's nothing more than proper error handling.

Good night,

Berend