1. Attachments are working again! Check out this thread for more details and to report any other bugs.

OCD File Deletion

Discussion in 'Fred's House of Pancakes' started by TonyPSchaefer, Nov 9, 2009.

  1. TonyPSchaefer

    TonyPSchaefer Your Friendly Moderator
    Staff Member

    Joined:
    May 11, 2004
    14,816
    2,498
    66
    Location:
    Far-North Chicagoland
    Vehicle:
    2017 Prius Prime
    Model:
    Prime Advanced
    I have two separate programs that claim to "shred" files. One lets you tell it the number of times to overwrite the disk space. The other lets you choose one of four different overwrite options; one of which is called "Guttman" and claims to overwrite 35 times.

    Anyway, I decided to give them a try and I "shredded" a folder. Then, to see how they did, I ran a File Undelete app. The undelete app found the files overwritten. Most reported as having been overwritten but were still listed as files. There were also files from a long time ago.

    So now my OCD is kicking in.
    Now I want to know how to remove a file, shred it, overwrite it, and keep it from being listed as a file that was ever on my drive. I assume the undelete app is pulling file listings from the FAT. This makes me think that the FAT is bloated with lists of files that no longer exist; I would like to remove those.

    Is there a way to do that?
     
  2. TheForce

    TheForce Stop War! Lets Rave! Make Love!

    Joined:
    May 30, 2005
    3,461
    537
    0
    Location:
    Wheelersburg, Ohio
    Vehicle:
    2006 Prius
    Model:
    N/A
    I dont know about that kind of software but if you really want overkill for your privacy you might want to check out TrueCrypt.

    It will encrypt your whole hard drive and you can have a LOOOOOONG password. Use a whole sentence with punctuations and everything.

    With true crypt you can have an encrypted OS inside an encrypted OS. This way if you are forced to give out your password you can give them the fake password. They will boot into the fake OS and be non the wiser because as long as you dont tell them you have another password they cant tell you have another real OS. If they try to check all they see is encrypted data.

    Also you can have a custom login prompt when the PC finishes booting the BIOS. You can set it to "Operating system not found". Thats all that's displayed. If you ever seen that prompt before you know your SOL. If you start typing at this prompt nothing happens. You dont see anything change. So all you do here is type in your password, hit enter, and your PC boots the OS. Its pretty cool.

    This works great for laptops.

    Did I just send your OCD into overdrive? :madgrin::flame:
     
  3. jayman

    jayman Senior Member

    Joined:
    Oct 21, 2004
    13,439
    640
    0
    Location:
    Winnipeg Manitoba
    Vehicle:
    2004 Prius
    Both FAT, and NTFS, maintain a directory of files on the drive. You must have an old operating system, such as Windows 98, to still be using FAT. Windows XP, Vista, 7 use NTFS

    With NTFS, the MFT or Master File Table is the file directory. The correct term for this information is "metadata" or, literally, "data about data."

    The MFT stores the file name, creation date, permissions, etc. Typically, when you "delete" a file in Windows, you only replace the first letter of the file name with an agreed-upon character, such as "?"

    This lets the OS know that the physical and logical space on the hard drive is available for another file. Otherwise there would be allocation errors and other problems cropping up

    NTFS also tracks file changes with the USN Journal (Update Sequence Number). The USN Journal record in the file uses a 64 bit space, so it has a fair amount of room to record changes

    You may want to reference this Microsoft article on the USN Journal

    Keeping an Eye onYour NTFS Drives:the Windows 2000 Change Journal Explained-- MSJ, September 1999

    and this one

    Change Journals (Windows)

    There is also the VSS or Volume Shadow Service. At the volume level, it's used to provide versioning for backups, and to allow for incremental rollbacks. See this article

    Volume Shadow Copy Service (Windows)

    As you can tell, after going through all of that, the same features that allow NTFS to provide versioning, self-healing, and for the MFT to have dynamic properties, make it troublesome to really "delete" a file.

    There are programs that are HIPAA, FACTA, FISMA, and, depending on application, NIST-FIPS and NSA compliant. The programs you mentioned do not appear to be compliant to these standards. The compliant file security programs can cost $100 and up.

    After 3 overwrites, any more is academic. You reach a point of diminishing returns very quickly. The DOD 5220.22-M disk sanitizing standard calls for 3 overwrites in a specific sequence.

    However, if truly sensitive data must be deleted, then one must use a dedicated hardware disk eraser. These cost thousands, and I won't get into how they actually work. I have a security clearance and intend to keep it

    Regarding bloated FAT's, that is unlikely to become a problem unless the hard drive is >80% full. NTFS MFT is dynamic, it shouldn't be a problem at all.

    To improve performance, its useful to use a proper hard drive sanitizer to wipe the drive, then reformat and reinstall the OS.

    But if you have truly sensitive data, you need either expensive software tools to have a reasonable means of assuring data confidentiality; or, you must physically destroy the hard drive. Eg: degauss it

    Some OS's are better than others at dealing with security. For routine internet browsing, I prefer Ubuntu. My Ubuntu machine is formatted EXT4
     
  4. fuzzy1

    fuzzy1 Senior Member

    Joined:
    Feb 26, 2009
    17,323
    10,169
    90
    Location:
    Western Washington
    Vehicle:
    Other Hybrid
    Model:
    N/A
    Probably off topic, but I don't have a better place to post this.

    An earlier thread, for which I cannot recall the proper keywords to locate with Search, mentioned someone in serious trouble due to improper files on his machine.

    Yesterday a local news outlet had a story about some people being remotely victimized when their machines were compromised. See Framed for child porn - by a PC virus. For those unwilling to click an unknown link, go to Seattle's KOMO TV station, and look under Technology news.

    The story is a bit simplistic. I suspect that most of the victims are not intentionally targeted for framing, but merely had a compromisable machine that someone else could use as a remote and untraceable server. But it does mention some legal resources with experience assisting the victims.
     
  5. hyo silver

    hyo silver Awaaaaay

    Joined:
    Mar 2, 2005
    15,232
    1,563
    0
    Location:
    off into the sunset
    Vehicle:
    2004 Prius
    Model:
    N/A
    There are a number of tools that will do that very nicely. Like...a hammer, a large screwdriver, or even a handy rock. :p
     
    1 person likes this.
  6. sandsw

    sandsw Member

    Joined:
    Feb 24, 2007
    99
    37
    0
    Location:
    Sydney Australia
    Vehicle:
    2010 Prius
    Model:
    V
    This actually depends on how determined folks are to read the drive. There are some really interesting techniques out there to resurrect data off physically damaged drives. Way out of my price range...

    Performing a backup of all the files on the disk, followed by a low-level format (or three) followed by a re-installation from backup would be one (tiresome) way to do it.

    Cheers
    Warwick
     
  7. jayman

    jayman Senior Member

    Joined:
    Oct 21, 2004
    13,439
    640
    0
    Location:
    Winnipeg Manitoba
    Vehicle:
    2004 Prius
    A Windows format does nothing, to very little, to sanitize a hard drive. It's a common misconception that formatting destroys data.

    You must use a specific overwrite sequence, preferably with a bootable CD that starts up the machine in some form of Linux, such as GenToo

    There are machines that look like giant industrial paper shredders. Instead of paper, you insert the hard drive, there is loud grinding sounds like a worn out coffee grinder, and metal confetti gushes out
     
  8. qbee42

    qbee42 My other car is a boat

    Joined:
    Mar 2, 2006
    18,058
    3,074
    7
    Location:
    Northern Michigan
    Vehicle:
    2006 Prius
    How have you found the stability of EXT4? I'm still running my Ubuntu machines on EXT3. I was a little leery of the new delayed writing scheme, but it seems to be working out well, as well as the other improvements, although I can't say that I often need more than 32,000 files in one directory. :D

    Tom
     
  9. Rae Vynn

    Rae Vynn Artist In Residence

    Joined:
    May 21, 2007
    6,038
    707
    0
    Location:
    Tumwater, WA USA
    Vehicle:
    2007 Prius
    Model:
    Two
    A rather strong magnet, passed firmly across the hard drive, will do the trick :thumb:
     
  10. eagle33199

    eagle33199 Platinum Member

    Joined:
    Mar 2, 2006
    5,122
    268
    0
    Location:
    Minnesota
    Vehicle:
    2015 Prius v wagon
    Model:
    Two
    Yes, i prefer to go the route of a large electromagnet, 30 seconds in the microwave, a trip through the wood chipper and finally a nice bonfire... I'd like to see someone try to resurrect data off of that!
     
  11. TonyPSchaefer

    TonyPSchaefer Your Friendly Moderator
    Staff Member

    Joined:
    May 11, 2004
    14,816
    2,498
    66
    Location:
    Far-North Chicagoland
    Vehicle:
    2017 Prius Prime
    Model:
    Prime Advanced
    Thanks for the replies, guys. Especially the long ones with very good info. I thoroughly destroy hard drives before I take them for electronics recycling. I was thinking more about a drive I intended on continuing to use.

    I said "FAT" because way back when I earned my CompSci degree, FAT was what was used. Since I'm only on the sidelines of technology these days, I tend to show my age more than savvy.

    The whole thing got started, as I said, when I wanted to see how well the application removed files. One thing led to another and eventually I was wondering why it was even still listed as a file even when I wanted it completely gone. I figured that I was certainly missing something. Let's say that the DoD performs a file erasure using three passes in a specific sequence. Wouldn't file "Government Building Blueprints and Secret Access Points.PPT" still be listed in the NTFS? Perhaps it's badly damaged and overwritten a few times but the fact that it once existed (and probably exists somewhere else) is still easily verified.

    I know I'm overthinking this. That's why I mentioned OCD in the title.

    So my suggestion would be that a software application designed to "shred" a file and/or folder should likewise remove it from the NTFS listing. Or, barring that, rename the file to nonsensical characters.

    Oh and Jay,
    I like the idea of TrueCrypt in theory. Sounds pretty cool. Also sounds like more than I'm really willing to implement at this point.

    Fuzzy, I was just in a meeting and one of the team members was surfing the web (4 hour meeting and very boring). He told us about the virus. Needless to say, we started wondering what would happen if our work computer got infested and we didn't know about it until IT and Legal and HR come to escort us out of the building to hand us over to the FBI.
     
  12. hyo silver

    hyo silver Awaaaaay

    Joined:
    Mar 2, 2005
    15,232
    1,563
    0
    Location:
    off into the sunset
    Vehicle:
    2004 Prius
    Model:
    N/A
    Ohhhh...well, why didn't you say so? :D

    Speaking of ecycling, should anything be done to the chips? Like, before somebody else reads your bank information?

    Those poor people in the news story lost everything, yet did nothing to deserve it. I don't remember either Aldous Huxley or George Orwell warning us about this sort of thing. Was there a Spider Robinson doomsday scenario I missed?
     
  13. fuzzy1

    fuzzy1 Senior Member

    Joined:
    Feb 26, 2009
    17,323
    10,169
    90
    Location:
    Western Washington
    Vehicle:
    Other Hybrid
    Model:
    N/A
    Based on that other thread that I cannot locate, I had to wonder whether this has already happened, to the acquaintance of the originator of that thread.

    My vague memory thought that the thread was started by one of our northern moderators.
     
  14. jayman

    jayman Senior Member

    Joined:
    Oct 21, 2004
    13,439
    640
    0
    Location:
    Winnipeg Manitoba
    Vehicle:
    2004 Prius
    Tom

    I used ext4 since 9.04 (Jaunty Jackelope) came out. I think my motherboard specs are similar to yours. This machine is dedicated to Ubuntu and web browsing only, no dual boot at all

    Of course, you have to manually partition the hard drive if you want to use ext4

    I noted a 3 second faster bootup using ext4 vs ext3. No big deal. I have had no issues using ext4. Those worried about the delayed write would be wise to have a UPS, which I do

    Doing the basic DoD wipe, the actual file is gone. The file entry in the MFT is still there, but it will look like

    ?overnment Building Blueprints and Secret Access Points.PPT

    So, the evidence will be there that the file once existed on the hard drive. The actual file will probably be unrecoverable

    There are programs that will actually wipe out the entries in the MFT and VSS. These programs tend to cause issues with startup, Windows XP and Vista will insist on running Chkdsk to check the disk for "consistency"

    Which is a way of saying that during boot, Windows became aware that the MFT and VSS were tampered with

    Regarding browsing from work, I have to caution to never, ever EVER do that! Many corporations monitor internet usage with programs like SonicWall, and will have a detailed audit trail as a result. This sort of monitoring is required for HIPAA, FACTA, and FISMA compliance

    Just don't do it, period. I've served as an expert witness during security investigations and court trials, the wealth of information available is frightening

    If you suspect a problem with your computer at work, immediately notify IT Support. To not do so really shoots you in the foot