The
USN Journal keeps track of changes made to files and directories on an
NTFS volume. This page contains notes and source code for reading the journal with
CCL's foreign interface. An important requirement to note is that this only works if you run
Clozure CL with administrative privileges. So select the "Run as administrator" option when starting Windows Terminal. Otherwise you'll get an
ERROR_ACCESS_DENIED when trying to access the change journal.
First, a primer on how pointers in Lisp work. In C, a pointer includes a type indicating what it points to. So it's common to cast a pointer from one type to another. But in Lisp, a pointer is just an encapsulated foreign address. In CCL, the encapsulation is called a MACPTR. (See Referencing and Using Foreign Memory Addresses for details.) In a way, this simplifies things, because you don't have to bother with casting. But in another way, it can make things more complicated, because it might be easier to forget what kind of data the pointer is pointing to (or where the end of it is). Incrementing a pointer is about as easy as incrementing an integer. The code below uses %incf-ptr and %inc-ptr, which are the destructive and non-destructive forms, respectively.
So we have some preliminary helper definitions. One of these is a function for converting a Windows FILETIME into human-readable form. There's also a macro, with-heap-ivector, for allocating some foreign memory and later disposing of it. This is used to make a buffer for reading journal entries. The buffer is an (unsigned-byte 64) vector because when reading the journal, the first eight bytes are the next USN to start reading from. So it's expedient to just grab the first element of the vector. This works because USN records have 64-bit alignment.
(defmacro bytes (foreign-type)
"Size in bytes of FOREIGN-TYPE (like sizeof in C)."
`(ccl::foreign-size ,foreign-type :bytes))
(define-symbol-macro +usn-size+ (bytes #>USN))
(defmacro with-heap-ivector ((vector macptr size element-type) &body body)
`(multiple-value-bind (,vector ,macptr) (make-heap-ivector ,size ,element-type)
(unwind-protect (progn ,@body) (dispose-heap-ivector ,vector))))
(defun file-time-string (file-time)
(rlet ((system-time #>SYSTEMTIME))
(when (zerop (#_FileTimeToSystemTime file-time system-time))
(return-from file-time-string (values nil (#_GetLastError))))
(format nil "~4,'0D-~2,'0D-~2,'0D ~2,'0D:~2,'0D:~2,'0D.~3,'0D"
(pref system-time #>SYSTEMTIME.wYear)
(pref system-time #>SYSTEMTIME.wMonth)
(pref system-time #>SYSTEMTIME.wDay)
(pref system-time #>SYSTEMTIME.wHour)
(pref system-time #>SYSTEMTIME.wMinute)
(pref system-time #>SYSTEMTIME.wSecond)
(pref system-time #>SYSTEMTIME.wMilliseconds))))
(defun usn-file-name (usn-record)
(get-encoded-string :utf-16le
(pref usn-record #>USN_RECORD.FileName)
(pref usn-record #>USN_RECORD.FileNameLength)))
(defun %open-usn-journal (&optional (drive-letter "C"))
(let ((volume-name (format nil "\\\\.\\~A:" drive-letter)))
(with-cstrs ((volume-name-ptr volume-name))
(#_CreateFileA volume-name-ptr
#$GENERIC_READ
(logior #$FILE_SHARE_READ #$FILE_SHARE_WRITE)
+null-ptr+ #$OPEN_EXISTING 0 +null-ptr+))))
(defun open-usn-journal (&optional (drive-letter "C"))
(let ((volume-handle (%open-usn-journal drive-letter)))
(if (eql volume-handle #$INVALID_HANDLE_VALUE)
(error "Couldn't open ~A." drive-letter)
volume-handle)))
(defun close-usn-journal (volume-handle)
(when (eql volume-handle #$INVALID_HANDLE_VALUE)
(error "The volume handle isn't valid."))
(#_CloseHandle volume-handle))
(defun %query-usn-journal (volume journal-data bytes-ptr)
(#_DeviceIoControl volume #$FSCTL_QUERY_USN_JOURNAL
+null-ptr+ 0 journal-data (bytes #>USN_JOURNAL_DATA)
bytes-ptr +null-ptr+))
(defun query-usn-journal-id (volume)
(rlet ((journal-data #>USN_JOURNAL_DATA) (bytes-ptr :unsigned-long))
(when (plusp (%query-usn-journal volume journal-data bytes-ptr))
(pref journal-data #>USN_JOURNAL_DATA.UsnJournalID))))
(defun %device-read-usn-journal (volume read-data buffer buffer-ptr)
(let ((buffer-size (* (length buffer) 8)))
(rletz ((bytes-ptr :unsigned-long))
(if (plusp (#_DeviceIoControl
volume #$FSCTL_READ_USN_JOURNAL
read-data (bytes #>READ_USN_JOURNAL_DATA)
buffer-ptr buffer-size bytes-ptr +null-ptr+))
(%get-unsigned-long bytes-ptr)
(let ((device-error-code (#_GetLastError)))
(error "DeviceIoControl (FSCTL_READ_USN_JOURNAL) ~
failed with error code: ~D." device-error-code))))))
(defun read-usn-journal (volume &optional journal-id (start-usn 0))
(unless journal-id (setq journal-id (query-usn-journal-id volume)))
(rletz ((read-data #>READ_USN_JOURNAL_DATA))
(setf (pref read-data #>READ_USN_JOURNAL_DATA.StartUsn) start-usn)
(setf (pref read-data #>READ_USN_JOURNAL_DATA.ReasonMask) #xFFFFFFFF)
(setf (pref read-data #>READ_USN_JOURNAL_DATA.UsnJournalID) journal-id)
(with-heap-ivector (buffer buffer-ptr (* +usn-size+ 1024) '(unsigned-byte 64))
(let ((total-bytes (%device-read-usn-journal volume read-data buffer buffer-ptr))
(usn-record (%inc-ptr buffer-ptr +usn-size+)) (next-usn (aref buffer 0)))
(decf total-bytes +usn-size+)
(values (loop while (plusp total-bytes)
collect (let* ((usn-record-name (usn-file-name usn-record))
(usn-record-length (pref usn-record #>USN_RECORD.RecordLength))
(usn-record-reason (pref usn-record #>USN_RECORD.Reason))
(usn-record-time (pref usn-record #>USN_RECORD.TimeStamp))
(usn-record-time-string (file-time-string usn-record-time)))
(%incf-ptr usn-record usn-record-length)
(decf total-bytes usn-record-length)
(list usn-record-name
usn-record-reason
usn-record-time-string)))
journal-id
next-usn)))))
(defun select-usn-journal (wildcard volume)
(flet ((interesting-file-p (file-record)
(pathname-match-p (car file-record)
wildcard)))
(loop with file-records = nil and current-usn = 0
with journal-id = (query-usn-journal-id volume) and next-usn
do (multiple-value-setq (file-records journal-id next-usn)
(read-usn-journal volume journal-id current-usn))
nconc (delete-if-not #'interesting-file-p file-records)
while (> next-usn current-usn)
do (setq current-usn next-usn))))
(defun print-file-records (file-records &optional (stream t))
(dolist (file-record file-records (list-length file-records))
(destructuring-bind (file-name reason file-time) file-record
(format stream "~A [~8,'0X] ~A~%" file-time reason file-name)))) To test this, we'll download a file and look for the resulting entries. Open the journal:
(defvar *volume* (open-usn-journal)) Then save the CLiki home page as a single file, and read the journal from the REPL:
(print-file-records (select-usn-journal "*.mhtml" *volume*))
Then the output will be something like:
2022-03-15 01:51:52.729 [00000100] CLiki_ the common lisp wiki.mhtml
2022-03-15 01:51:52.738 [80000300] CLiki_ the common lisp wiki.mhtml
2022-03-15 01:51:53.252 [00000100] CLiki_ the common lisp wiki.mhtml
2022-03-15 01:51:53.310 [00000102] CLiki_ the common lisp wiki.mhtml
2022-03-15 01:51:53.311 [80000102] CLiki_ the common lisp wiki.mhtml
2022-03-15 01:51:53.903 [00080000] CLiki_ the common lisp wiki.mhtml
2022-03-15 01:51:53.915 [80080000] CLiki_ the common lisp wiki.mhtml
So this covers a period of 1.186 seconds. The reason codes are a little hard to follow, partly because multiple "reasons" can be packed together into one 32-bit field. There's some creating, writing, and closing of the file going on here. Also note that the FileName is just that: it doesn't include the full pathname. For that, we would need to use the FileReferenceNumber.
Apache 2,
Programming Tips