Corrupted EXT4 fs on embedded Linux (A20 Olimex Lime 2) with clean poweoff

[Julien]

Hi there,

I am having a problem with the A20 Olimex Lime 2, I have posted a topic on StackExchange: https://unix.stackexchange.com/questions/748926/corrupted-ext4-filesystem-on-embedded-linux-a20-olimex-lime-2-with-clean-power

No solution was found, maybe you can help me with this?

[LubOlimex]

What is the board doing when you issue the shutdown? Maybe some file was open for writing or editing in the background? Maybe try issuing "sync" before the shutdown?

Did you test those cards with F3 or H2testw? This is mandatory if you run in problems. Counterfeit SD cards are an issue and it is always worth checking with F3/H2testw directly after purchase.

It is significant info that the thing occurs after half a day (and not like random from 10 minutes to 1 week).

System file corruption is pretty common problem with embedded computers. It is rare for it to occur due to software shutdown tho. Especially when your setup has a battery. Still maybe check these:

https://www.embedded.com/protecting-your-embedded-software-against-memory-corruption/
https://www.embeddedts.com/assets/preventing-filesystem-corruption-in-embedded-linux

[Julien]

Thank you for your reply.

I tried adding "sync" before shutdown and it made no difference.

I have tried with several sd cards and I run a h2test on them but there is no error shown.

I think the problem is with Docker, because the problem only occurs with that program. For example, if I disable Docker, there are no more problems.
Maybe Docker can't run on an embedded system? But in this case, why does it work on eMMC? Maybe the problem is a combination of the SD card problem and Docker?

[LubOlimex]

Mmmm, I've heard before that Docker performs a lot of I/O operations and might quickly wear out SD cards. Also that some docker containers might fill the SD card without you even noticing. Maybe browse the Docker forums or search online for similar cases, search for "Docker on SD card".

Check here:

https://www.portainer.io/blog/sd-card-longevity-and-docker/kubernetes

https://www.reddit.com/r/raspberry_pi/comments/100s8gg/sd_card_reliability_issues_when_running_docker/

I believe most people recommend having the Docker on SSD or HDD, which is alright since A20-OlinuXino-LIME2 has native SATA controller and also you can always attach USB hard disk.

[LubOlimex]

Also - what size are the SD cards? Are they smaller than the eMMC?

[Julien]

Thank you, you make interesting reading.

I use 32GB sd cards, but they are partitioned for a total of 8GB of storage.
The docker files are stored on a 4GB partition.

The eMMC use the same partitioning.


boot : Contains the kernels and it's assets // (64MB)
root : The main filesystem                          // (~3,9GB)
data : To store data (ex : logs)                   // (4GB)        <-- Docker files