Updated Dreamwidth backup script

[garote]

For quite a while I've been looking for some nice way to get a complete backup of my Dreamwidth content onto my local machine. And I gotta wonder... Is this not a very popular thing? There are a lot of users on here, posting a lot of cool and unique content. Wouldn't they want to have a copy, just in case something goes terribly wrong?

I found a Python script that does a backup, and was patched to work with Dreamwidth, but the backup took the form of a huge pile of XML files. Thousands of them. I wanted something more flexible, so I forked the script and added an optional flag that writes everything (entries, comments, userpic info) to a single SQLite database.

https://github.com/GBirkel/ljdump

Folks on MacOS can just grab the contents of the repo and run the script. All the supporting modules should already be present in the OS. Windows people will need to install some version of Python.

For what it's worth, here's the old discussion forum for the first version of the script, released way back around 2009.

Update, 2024-03-25:

The script now also downloads and stores tag and mood information.

Update, 2024-03-26:

After synchronizing, the script now generates browseable HTML files of the journal, including entries for individual pages with comment threads, and linked history pages showing 20 entries at a time.

Moods, music, tags, and custom icons are shown for the entries where applicable.

Currently the script uses the stylesheet for my personal journal (this one), but you can drop in the styles for yours and it should accept them. The structure of the HTML is rendered as close as possible to what Dreamwidth makes.

Update, 2024-03-28:

The script can also attempt to store local copies of the images embedded in journal entries. It organizes them by month in an images folder next to all the HTML. This feature is enabled with a "--cache_images" argument.

Every time you run it, it will attempt to cache 200 more images, going from oldest to newest. It will skip over images it's already tried and failed to fetch, until 24 hours have gone by, then it will try those images once again.

The image links in your entries are left unchanged in the database. They're swapped for local links only in the generated HTML pages.

Update, 2024-04-02:

The script is now ported to Python 3, and tested on both Windows and MacOS. I've added new setup instructions for both that are a little easier to follow.

Update, 2024-04-30:

Added an option to stop the script from trying to cache images that failed to cache once already.

2024-06-26: Version 1.7.6

Attempt to fix music field parsing for some entries.
Fix for crash on missing security properties for some entries.
Image fetch timeout reduced from 5 seconds to 4 seconds.

2024-08-14: Version 1.7.7

Slightly improves unicode handling in tags and the music field.

2024-09-07: Version 1.7.8

Changes "stop at fifty" command line flag to a "max n" argument, with a default of 400, and applies it to comments as well as entries. This may help people who have thousands of comments complete their initial download. I recommend using the default at least once, then using a value of 1500 afterward until you're caught up.

2024-09-18: Version 1.7.9

Table of contents for the table of contents!
First version of an "uncached images" report to help people find broken image links in their journal.

2025-09-25: Version 1.8

Added some more unicode guardrails for things like icon keywords and the journal name.

Comments

[juan_gandhi]

Thank you!

[garote]

Would you be willing to test the latest version?

[dmm]

thanks!

[silveradept]

Seem to be running into an error where certain entries end up getting an SQLite Error, where "Error binding paramter 12: type 'Binary' is not supported." I have no idea what the issue is, but it seems to happen most commonly in entries where there are characters with diacritical marks on them in the music field. It's not all of them, specifically, but it does track pretty well. Maybe it's having Unicode parsing issues?

[garote]

Dangit, I thought I ironed all those out. The music field you say! I'll poke at it and see what I find.

<3

[quietmoment]

Thank you for the script! FYI I ran into a little error on windows

line 823, in get_or_create_cached_image_record
    date_or_none = date_first_seen.strftime('%s')
ValueError: Invalid format string

Got around it by adding import time with the other imports and changing line 823 (now 824) to date_or_none = time.mktime(date_first_seen.timetuple()) (fix stolen from here, dunno if it's a good fix tho)

EDIT: I also ended up making some more changes to download images hosted on Dreamwidth, also in their original resolution - patch file below in case its handy.
Edit again: fix running ljdumptohtml.py alone, allow images to have attributes between <img and src="

Patch file

--- C:/dw/ljdump-1.7.4/ChangeLog	Sat May 11 00:00:00 2024
+++ C:/dw/ljdump-1.7.4-patched/ChangeLog	Sat May 11 00:00:00 2024
--- C:/dw/ljdump-1.7.4/ljdump.config.sample	Sat May 11 00:00:00 2024
+++ C:/dw/ljdump-1.7.4-patched/ljdump.config.sample	Sat May 11 00:00:00 2024
@@ -3,4 +3,5 @@
     <server>https://livejournal.com</server>
     <username>myaccount</username>
     <password>mypassword</password>
+    <ljuniq>ljuniq cookie if you want to download Dreamwidth hosted images</ljuniq>
 </ljdump>
--- C:/dw/ljdump-1.7.4/ljdump.py	Sat May 11 00:00:00 2024
+++ C:/dw/ljdump-1.7.4-patched/ljdump.py	Sat May 11 00:00:00 2024
@@ -70,7 +70,7 @@
     return e[0].firstChild.nodeValue
 
 
-def ljdump(journal_server, username, password, journal_short_name, verbose=True, stop_at_fifty=False, make_pages=False, cache_images=False, retry_images=True):
+def ljdump(journal_server, username, password, ljuniq, journal_short_name, verbose=True, stop_at_fifty=False, make_pages=False, cache_images=False, retry_images=True):
 
     m = re.search("(.*)/interface/xmlrpc", journal_server)
     if m:
@@ -417,6 +417,7 @@
         ljdumptohtml(
             username=username,
             journal_short_name=journal_short_name,
+            ljuniq=ljuniq,
             verbose=verbose,
             cache_images=cache_images,
             retry_images=retry_images
@@ -444,6 +445,11 @@
             password = password_els[0].childNodes[0].data
         else:
             password = getpass("Password: ")
+        ljuniq_els = config.documentElement.getElementsByTagName("ljuniq")
+        if len(ljuniq_els) > 0:
+            ljuniq = ljuniq_els[0].childNodes[0].data
+        else:
+            ljuniq = getpass("ljuniq cookie (for Dreamwidth hosted image downloads, leave blank otherwise): ")
         journals = [e.childNodes[0].data for e in config.documentElement.getElementsByTagName("journal")]
         if not journals:
             journals = [username]
@@ -457,6 +463,7 @@
         print
         username = input("Username: ")
         password = getpass("Password: ")
+        ljuniq = getpass("ljuniq cookie (for Dreamwidth hosted image downloads, leave blank otherwise): ")
         print
         print("You may back up either your own journal, or a community.")
         print("If you are a community maintainer, you can back up both entries and comments.")
@@ -474,6 +481,7 @@
             journal_server=journal_server,
             username=username,
             password=password,
+            ljuniq=ljuniq,
             journal_short_name=journal,
             verbose=args.verbose,
             stop_at_fifty=args.fifty,
--- C:/dw/ljdump-1.7.4/ljdumpsqlite.py	Sat May 11 00:00:00 2024
+++ C:/dw/ljdump-1.7.4-patched/ljdumpsqlite.py	Sat May 11 00:00:00 2024
@@ -30,6 +30,8 @@
 from sqlite3 import Error
 from xml.sax import saxutils
 from builtins import str
+import time
+import re
 
 
 # Subclass of tzinfo swiped mostly from dateutil
@@ -803,6 +805,10 @@
         SELECT id, url, filename, date_first_seen, date_last_attempted, cached FROM cached_images
         WHERE url = :url""", {'url': image_url})
     row = cur.fetchone()
+    pattern = re.compile('^https://(\w+).dreamwidth.org/file/\d+x\d+/(.+)')
+    if pattern.match(image_url):
+        result = pattern.search(image_url)
+        get_or_create_cached_image_record(cur, verbose, 'https://' + result.group(1) + '.dreamwidth.org/file/' + result.group(2), date_first_seen)
     if row:
         if verbose:
             print('Found image cache record for: %s' % (image_url))
@@ -820,7 +826,7 @@
             print('Creating image cache record for: %s' % (image_url))
         date_or_none = None
         if date_first_seen:
-            date_or_none = date_first_seen.strftime('%s')
+            date_or_none = time.mktime(date_first_seen.timetuple())
         data = {
             "id": None,
             "url": image_url,
--- C:/dw/ljdump-1.7.4/ljdumptohtml.py	Sat May 11 00:00:00 2024
+++ C:/dw/ljdump-1.7.4-patched/ljdumptohtml.py	Mon May 13 23:37:54 2024
@@ -385,12 +385,18 @@
 
 def resolve_cached_image_references(content, image_urls_to_filenames):
     # Find any image URLs
-    urls_found = re.findall(r'img[^\"\'()<>]*\ssrc\s?=\s?[\'\"](https?:/+[^\s\"\'()<>]+)[\'\"]', content, flags=re.IGNORECASE)
+    urls_found = re.findall(r'<img[^<>]*\ssrc\s?=\s?[\'\"](https?:/+[^\s\"\'()<>]+)[\'\"]', content, flags=re.IGNORECASE)
     # Find the set of URLs that have been resolved to local files
     resolved_urls = []
     for image_url in urls_found:
         if image_url in image_urls_to_filenames:
             resolved_urls.append(image_url)
+        pattern = re.compile('^https://(\w+).dreamwidth.org/file/\d+x\d+/(.+)')
+        if pattern.match(image_url):
+            result = pattern.search(image_url)
+            full_url = 'https://' + result.group(1) + '.dreamwidth.org/file/' + result.group(2)
+            if full_url in image_urls_to_filenames:
+                resolved_urls.append(full_url)
     # Swap them in
     for image_url in resolved_urls:
         filename = image_urls_to_filenames[image_url]
@@ -630,10 +636,15 @@
     return html_as_string
 
 
-def download_entry_image(img_url, journal_short_name, subfolder, url_id):
+def download_entry_image(img_url, journal_short_name, subfolder, url_id, entry_url, ljuniq):
     try:
-        image_req = urllib.request.urlopen(img_url, timeout = 5)
+        pattern = re.compile('^https://(\w+).dreamwidth.org/file/')
+        headers = {}
+        if pattern.match(img_url):
+            headers = {'Referer': entry_url, 'Cookie': "ljuniq="+ljuniq}
+        image_req = urllib.request.urlopen(urllib.request.Request(img_url, headers = headers), timeout = 5)
         if image_req.headers.get_content_maintype() != 'image':
+            print('Content type was not expected, image skipped: ', img_url, image_req.headers.get_content_maintype())
             return (1, None)
         extension = MimeExtensions.get(image_req.info()["Content-Type"], "")
 
@@ -681,7 +692,7 @@
         return (1, None)
 
 
-def ljdumptohtml(username, journal_short_name, verbose=True, cache_images=True, retry_images=True):
+def ljdumptohtml(username, journal_short_name, ljuniq, verbose=True, cache_images=True, retry_images=True):
     if verbose:
         print("Starting conversion for: %s" % journal_short_name)
 
@@ -741,8 +752,17 @@
                 e_id = entry['itemid']
                 entry_date = datetime.utcfromtimestamp(entry['eventtime_unix'])
                 entry_body = entry['event']
-                urls_found = re.findall(r'img[^\"\'()<>]*\ssrc\s?=\s?[\'\"](https?:/+[^\s\"\'()<>]+)[\'\"]', entry_body, flags=re.IGNORECASE)
+                initial_urls_found = re.findall(r'<img[^<>]*\ssrc\s?=\s?[\'\"](https?:/+[^\s\"\'()<>]+)[\'\"]', entry_body, flags=re.IGNORECASE)
                 subfolder = entry_date.strftime("%Y-%m")
+                urls_found = []
+                for image_url in initial_urls_found:
+                    urls_found.append(image_url)
+                    pattern = re.compile('^https://(\w+).dreamwidth.org/file/\d+x\d+/(.+)')
+                    if pattern.match(image_url):
+                        result = pattern.search(image_url)
+                        full_url = 'https://' + result.group(1) + '.dreamwidth.org/file/' + result.group(2)
+                        urls_found.append(full_url)
+
                 for image_url in urls_found:
                     cached_image = get_or_create_cached_image_record(cur, verbose, image_url, entry_date)
                     try_cache = True
@@ -758,7 +778,7 @@
                         image_id = cached_image['id']
                         cache_result = 0
                         img_filename = None
-                        (cache_result, img_filename) = download_entry_image(image_url, journal_short_name, subfolder, image_id)
+                        (cache_result, img_filename) = download_entry_image(image_url, journal_short_name, subfolder, image_id, entry['url'], ljuniq)
                         if (cache_result == 0) and (img_filename is not None):
                             report_image_as_cached(cur, verbose, image_id, img_filename, entry_date)
                             image_resolve_max -= 1
@@ -955,6 +975,11 @@
         journals = [e.childNodes[0].data for e in config.documentElement.getElementsByTagName("journal")]
         if not journals:
             journals = [username]
+        ljuniq_els = config.documentElement.getElementsByTagName("ljuniq")
+        if len(ljuniq_els) > 0:
+            ljuniq = ljuniq_els[0].childNodes[0].data
+        else:
+            ljuniq = getpass("ljuniq cookie (for Dreamwidth hosted image downloads, leave blank otherwise): ")
     else:
         print("ljdumptohtml - livejournal (or Dreamwidth, etc) archive to html utility")
         print
@@ -976,6 +1001,7 @@
         ljdumptohtml(
             username=username,
             journal_short_name=journal,
+            ljuniq=ljuniq,
             verbose=args.verbose,
             cache_images=args.cache_images,
             retry_images=args.retry_images
--- C:/dw/ljdump-1.7.4/README.md	Sat May 11 00:00:00 2024
+++ C:/dw/ljdump-1.7.4-patched/README.md	Sat May 11 00:00:00 2024
--- C:/dw/ljdump-1.7.4/stylesheet.css	Sat May 11 00:00:00 2024
+++ C:/dw/ljdump-1.7.4-patched/stylesheet.css	Sat May 11 00:00:00 2024

Re: <3

[garote]

I've made a new release incorporating a slightly rewritten version of your changes... But I'm not sure how the ljuniq thing works. Where do we get that value?

https://github.com/GBirkel/ljdump/releases/tag/v1.7.5

KeyError: 'protected'

[antzpantz]

Hi there! Thank you for reviving ljdump. I saw that there have been updates over the past number of months so I'm very appreciative of your efforts.

I've just started attempting to backup my old LJ but have come up with this error which I'm not sure what has caused it. Maybe it's because I have Private posts? It's been a long time since I've used LJ so I've probably forgotten what privacy categories exist. Here's the error, it appears after adding 'moods':

Adding new mood with name: jealous
Adding new mood with name: nervous
Traceback (most recent call last):
File "C:\Users\Anthony\Downloads\ljdump-1.7.5\ljdump.py", line 488, in
ljdump(
File "C:\Users\Anthony\Downloads\ljdump-1.7.5\ljdump.py", line 348, in ljdump
'security_protected': t['security']['protected'],
KeyError: 'protected'

Re: KeyError: 'protected'

[garote]

Interesting! Well if there's no key there, then there's no 'protected' status to report, true or false...
So it may not be about the entry being protected, it may be about the data coming from the server just being weirdly incomplete. Maybe some old entry before a schema change, or some weird side-effect of an import.
Thanks for the info! I'll see if I can cook up a patch for this later today.

[justbrandie]

Hi! Thank you so much for creating this. I am very much not a programmer, but I ran the script several times and have seen that the log shows all of my entries are archived. However, I am getting an error, and I don't see any HTML files, just a .db file that I don't know how to open.

This is the last bit of output:

Traceback (most recent call last):
File "./ljdump.py", line 501, in
ljdump(
File "./ljdump.py", line 430, in ljdump
ljdumptohtml(
File "/Users/brandie 1/Downloads/ljdump-1.7.6/ljdumptohtml.py", line 765, in ljdumptohtml
cached_image = get_or_create_cached_image_record(cur, verbose, url_to_cache, entry_date)
File "/Users/brandie 1/Downloads/ljdump-1.7.6/ljdumpsqlite.py", line 832, in get_or_create_cached_image_record
cur.execute("""
sqlite3.OperationalError: near "RETURNING": syntax error

[garote]

Huh, that's an interesting one. An SQL language error. If I'm reading this right, the version of SQLite you're using doesn't support the "RETURNING" clause, which was introduced in version 3.35 (2021-03-12). Tell me some details about the computer you're running this on?

[falkner]

Hi! Thank you for your work on this script. I've run into an error while trying to back-up my journal:

Traceback (most recent call last):
File "-\ljdump.py", line 501, in
ljdump(
File "-\ljdump.py", line 177, in ljdump
insert_or_update_event(cur, verbose, ev)
File "-\ljdumpsqlite.py", line 417, in insert_or_update_event
cur.execute("""
sqlite3.InterfaceError: Error binding parameter :props_taglist - probably unsupported type.

I have tried re-running the script multiple times but it always errors out and stops at the same entry with this error.

[garote]

Is this dreamwidth or livejournal?
How old is the entry?
Is there more than one tag assigned to that entry?
Anything unique about those tags?

[falkner]

Apologies for coming back with another report... It seems that the script is timing out while trying to fetch comments? (I didn't even realize it would back-up those.)

Fetching journal comments for: falkner
*** Error fetching comment meta, possibly not community maintainer?
*** HTTP Error 504: Gateway Time-out
Fetching current users map from database
Traceback (most recent call last):
File "-\ljdump.py", line 501, in
ljdump(
File "-\ljdump.py", line 245, in ljdump
usermap = get_users_map(cur, verbose)
File "-\ljdumpsqlite.py", line 780, in get_users_map
cur.execute("SELECT id, name FROM users_map")
sqlite3.ProgrammingError: Cannot operate on a closed cursor.

[garote]

Hmm... "Gateway timeout" makes me think it's some kind of throttling happening server-side. Did this happen during a particularly long run of the script?

Thank you + Q

[(Anonymous)]

First thanks; pulled old LJ on Mac OS and am so thankful to have it.
Second: the script skipped about ~7% of entries. Is there a way to re-run that would get it to re-try these non-pulled entries, rather than start from last pull/looking for new entries?

Re: Thank you + Q

[garote]

Unfortunately, it's out of my (the script's) control which entries LJ decides are available in the history API, so if any are missing there isn't much I can do. (Note that it's normal for there to be gaps in the ID numbers that the script appears to be fetching and that's not an indication of a skipped entry.) Do you have an error log or an example entry you can point me at?

[annieeats]

Thank you for doing this and keeping it up!

[garote]

You're quite welcome! Let me know if you hit any snags.

Also: Intrigued: What is "cat tipping"?

[miimiicry]

Thank you so much for this ! I tried many things but couldn't make them work. Your script is very easy to understand and I got to make a back-up of all my entries !

[garote]

You're very welcome!

Images unable to cache?

[naivette]

Hello,

Thank you for creating such an amazing tool! I wanted to backup my dreamwidth journal locally, but I couldn't figure out the image cache setting.

It works when I double click ljdump.py, but the images aren't cached. When I open terminal and type in this code, ./ljdump.py --cache_images the terminal just opens and closes a window, but nothing happens. I'm not really sure what I'm doing wrong...

Re: Images unable to cache?

[garote]

Hello! It sounds like you're using the terminal on Windows... Unfortunately I haven't used that environment in years, so you may need to grab someone who can sit down at the keyboard with you and investigate more directly.

[lb_lee]

Thank you so much for making this script! I have an error that came up that nobody else has mentioned in the thread, and I'm stumped, because while I know what this error MEANS, not so in this context:

Traceback (most recent call last):
File "./ljdump.py", line 508, in
ljdump(
File "./ljdump.py", line 91, in ljdump
ljsession = getljsession(journal_server, username, password)
File "./ljdump.py", line 55, in getljsession
r = urllib.request.urlopen(journal_server+"/interface/flat", data=data)
File "/usr/lib/python3.10/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.10/urllib/request.py", line 525, in open
response = meth(req, response)
File "/usr/lib/python3.10/urllib/request.py", line 634, in http_response
response = self.parent.error(
File "/usr/lib/python3.10/urllib/request.py", line 563, in error
return self._call_chain(*args)
File "/usr/lib/python3.10/urllib/request.py", line 496, in _call_chain
result = func(*args)
File "/usr/lib/python3.10/urllib/request.py", line 643, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

I am using Linux Mint and new to Linux, so am totally willing to bet this is something totally obvious that I just don't know.

[garote]

A possibility... For Dreamwidth, I put "https://dreamwidth.org" in the server part of the configuration file... Is that what you're using?

https://github.com/GBirkel/ljdump/blob/master/ljdump.config.sample

[falkner]

I am back once again...

This error seems to occur if there's unicode characters in the display name (in my case it was "♪"):
Traceback (most recent call last): File -\ljdump.py", line 508, in <module> ljdump( File "-\ljdump.py", line 387, in ljdump insert_or_update_user_info(cur, verbose, File "-\ljdumpsqlite.py", line 304, in insert_or_update_user_info cur.execute(""" sqlite3.InterfaceError: Error binding parameter :fullname - probably unsupported type.

[Running it on Windows this time around.] I got around it by temporarily removing the character in the name, but I'm reporting anyway in case anybody else comes across this issue.

[garote]

Thanks for the feedback! I've made a new release that should fix this error.

[wembley]

First: Thank you so much for doing this, the tool is amazing.

Second: Probably a dumb question, but: so I messed around a little with the html of the index/table of contents and saved the style sheet from my current dreamwidth journal in the folder -- https://wembley.dreamwidth.org/res/4147489/stylesheet?1745745106 -- but I can't get the darker blue left-hand sidebar to appear with the list of tags and such. I'm not very tech-savvy, I just know a teeny bit of HTML and I don't really understand CSS. Do you know what code I should add in to put the sidebar back?

Third, about the actual tool again: A really awesome person backed up my Livejournal with a different method and it worked really well, but I wanted to test out your version of LJDump just for fun and see how it worked on my old LJ, since it worked really well on my DW (though I had to do the max-1500 thing or it would get rate-limited). When I tried it on my Livejournal (wemblee.livejournal.com), this happened:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1348, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1286, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1332, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1281, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1041, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 979, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/http/client.py", line 1458, in connect
self.sock = self._context.wrap_socket(self.sock,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 517, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1075, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/ssl.py", line 1346, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1002)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Applications/Downloader apps/Livejournal downloaders/ljdump-1.7.9/ljdump.py", line 508, in
ljdump(
File "/Applications/Downloader apps/Livejournal downloaders/ljdump-1.7.9/ljdump.py", line 91, in ljdump
ljsession = getljsession(journal_server, username, password)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Applications/Downloader apps/Livejournal downloaders/ljdump-1.7.9/ljdump.py", line 55, in getljsession
r = urllib.request.urlopen(journal_server+"/interface/flat", data=data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 216, in urlopen
return opener.open(url, data, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 519, in open
response = self._open(req, data)
^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 496, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1391, in https_open
return self.do_open(http.client.HTTPSConnection, req,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/urllib/request.py", line 1351, in do_open
raise URLError(err)
urllib.error.URLError: urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1002)

Btw, this part:

urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1002)

had the "<" ">" brackets around it but when I kept those in this comment, DW didn't like it and said it was an unclosed HTML bracket, so just FYI.

Thank you so much for updating this program, it really is awesome of you. <3

[garote]

Huh! Well, the error seems to be saying there is a security misconfiguration in the site where your journal is hosted. But I have my doubts about that error.

When you set up the "ljdump.config" file, try just putting https://livejournal.com into the "server" section, and put "wembley" just in the "username" section. Give that a try.

[harukami]

Hello!! Thank you so much for doing this, it's great peace of mind to be able to back things up.

However, maybe you can help me with a problem! Although it worked fine on several personal journals, on one of them, it only output 3 of 12 journal entries, and would not build an index for it (it output the journal.db file, a userpics folder, and an entries folder, and only put 3 entries in the folder).

When it runs for the first time, all posts get listed in the terminal output as it runs. It can't be because some entries were locked, because one of the three was a locked entry. Some of the ones that were missed, but not all, had lj-cut text/readmores in them.

Ultimately I'm just trying to figure out: what sometimes gets posts missed, so I can try to fix that?

Cheers, and thanks again!

[ambyr]

Thank you, this was tremendously useful!

[garote]

You’re welcome!

[doranwen]

I tried running it on my old LJ (now imported to DW) and got that fine, but when I tried to download a community I was a member of, it complained that I wasn't a maintainer when it got to fetching comment metadata. (And it wouldn't allow me to download a friend's LJ - which only had public posts.) Is there any way to download communities we are just members of? Or the visible-to-us posts in a friend's LJ? Some of my old friends are not on LJ anymore and I've lost contact with them, but their LJs are still up. I'd like to be able to save those to read once LJ dies, as it seems like it might soonish. There's a paid tool that lets one save anything visible to oneself, but it requires changing theme and I've got an old theme that I won't be able to get back if I switch away, so I can't run it on my main LJ account to save the communities and LJs that I have access to there. :/

[garote]

Unfortunately this program probably won't be able to help you, since it's reliant on the API and whatever a user is allowed to access by it. A tool based on the brute-force style method of web scraping (i.e. pretending to be a human operating a web browser) would work for your purposes, but I'm not sure what to recommend.

Any volunteers?