Fixed some issues yesterday and tried uploading to test results and it failed. Just realized that the refactored file is now about 80 Mb in size. Can you increase the limit or is that a hard limit?
Here’s the issue: when you first open a tree, the entire tree is downloaded into your browser. This makes navigation and search really fast, but it means that large trees require a long time to open and require a lot of memory. In order to keep people from having timeouts and running out of memory, I impose a limit of 75M and 75,000 people.
I have noticed it can take a while to come up when I first open one with a lot in it. I hadn’t looked to see what’s actually going on, that does explain it.
So by folding all the extracted facts and stuff Ancestry does not normally export into my Gedcom it a little more than quadruples the size. If I manually entered all those facts into Rootsfinder, and web clipper helps accelerate that process, I very well could eventually end up in the same boat at some point though…
Dallan,
Would you consider the option of allowing people to upload larger Gedcom files as well as have more people in a tree with some kind of notice or disclaimer perhaps that larger sizes can lead to performance issues? For me it is an issue as a full Gedcom upload easily exceeds 80M or 90M now and once you had some of the improved media handling and other enhancements implemented I was hoping to move over and host my main tree on Rootsfinder. It also doesn’t seem fair to existing users who are not aware of the limits and may someday hit one of them.
On a separate topic if anyone else has followed this thread and is interested the tools I wrote for extracting everything from a tree on Ancestry.com and refactoring the Gedcom to fold all the additional information back in can be found on Github at GitHub - cdhorn/ancestry-tools: Tools for working with Ancestry.com Gedcom files and all associated media items
Thanks,
Chris
What I need to do is figure out a way to not download everything up-front, but instead to download incrementally when trees are very large. Then RootsFinder could support larger GEDCOMs. This will become even more important for mobile.
Can you give me an idea of the number of people you’re talking about? Is it over 100,000?
Yeah, the size is not due to the number of people in my tree, I’m not anywhere near those numbers. It’s the supporting data that is the issue for me as that grows at a much faster rate. If you recall I’m extracting and loading every possible detail from Ancestry.com that I can.
So a vanilla Ancestry.com Gedcom that you would normally load might have a source citation like this, if you are even lucky enough to even have a PAGE tag for it as many apparently don’t:
2 SOUR @S-156356944@
3 PAGE Year: 1910; Census Place: Manhattan Ward 11, New York, New York; Roll: T624_1010; Page: 9A; Enumeration District: 0226; FHL microfilm: 1375023
3 _APID 1,7884::18611538
That expands to this once I refactor the Gedcom with all the information in my tree that they choose not to include in their Gedcom export process:
2 SOUR @S-156356944@
3 PAGE Year: 1910; Census Place: Manhattan Ward 11, New York, New York; Roll: T624_1010; Page: 9A; Enumeration District: 0226; FHL microfilm: 1375023
3 DATA
4 TEXT Able to Read: Yes
5 CONT Able to Write: Yes
5 CONT Age in 1910: 27
5 CONT Birth Year: abt 1883
5 CONT Birthplace: Russia
5 CONT Employer, Employee or Other: Wage Earner
5 CONT Farm or House: House
5 CONT Father’s Birthplace: Russia
5 CONT Gender: Male
5 CONT Home Owned or Rented: Rent
5 CONT Home in 1910: Manhattan Ward 11, New York, New York
5 CONT Household Members: Isidore Einwomer; 27; Mamie Hirsctshorn; 25; Rebecca Hirsctshorn; 1
5 CONT Immigration Year: 1903[1893]
5 CONT Industry: Pants
5 CONT Marital Status: Married
5 CONT Mother’s Birthplace: Russia
5 CONT Name: Isidore Einwomer[Isidore Hirsctshorn] [Isidore Herschhorn]
5 CONT Native Tongue: English
5 CONT Naturalization Status: Alien
5 CONT Number of weeks out of work: No
5 CONT Occupation: Operator
5 CONT Out of Work: N
5 CONT Race: White
5 CONT Relation to Head of House: Head
5 CONT Spouse’s Name: Mamie Hirsctshorn
5 CONT Street: Goerck St
5 CONT Years Married: 2
3 NOTE Join Ancestry
3 OBJE @M101884@
3 OBJE @M121488@
So I don’t loose any relevant facts anymore and I include the image of the original census page and a screenshot of the source citation page for good measure.
So while I have 13,036 individuals in my tree last time I synced it a few weeks ago I also have 1031 source records supporting 100,500 source citations almost all of which have additional facts like the above; 47,591 media items mainly supporting some 200,529 references from those source citations; and 148,109 note records although some 144,936 of those are URL links back to the associated page for an item in my tree on Ancestry.com for quick reference.
How about if I temporarily disable the GEDCOM import limitations for a few days so you can try importing your GEDCOM, and you tell me what your experience is like opening the GEDCOM the first time? And also send me the URL and give me permission to try opening it as well?
How many megabytes is it?
I never did get back to you on this, so much going on, sorry about that.
I need to pull a fresh Gedcom and resync with Ancestry as I have not done so in a while. The last one I generated was 84 MB or so.
Maybe you could provide a user option to override the Gedcom size limit with a warning that overall performance could be degraded or something to that effect?
Separate question, you don’t have an unpublished but publicly accessible API for interfacing with Rootsfinder do you?
And another question, is compression configured on the web server? Gzip gets that 84 MB Gedcom down to 7 MB.
I see it appears to be, never mind that question.
Hi,
Now that I have the GEDCOM media program finally launched and i’m getting my head above water, I’d like to follow up on this.
First, if you’re interested, I’d like to do an experiment on the GEDCOM size. I can raise the size limit from 75Mb to 100Mb tomorrow afternoon. Would you be willing to try uploading your GEDCOM and see if you can successfully open your tree after it uploads? If you’re successful, even if the tree opens slowly (which it almost certainly will), I’ll leave it at the higher limit.
Second, I’d like to look into adding the Ancestry media into the GEDCOM media uploader, like we were talking about so long ago. Would you be willing to test it with your Ancestry GEDCOM once I get it added?
Finally, I do have an unpublished but publicly accessible API for interfacing with RootsFinder. It’s a pretty straightforward REST API. I’d be happy to talk more about it if you’re interested. Why do you ask?
Hey Dallan, sorry for not getting back sooner, many other things going on with the summer here. 
Regarding the Gedcom, I just generated an updated one, it’s currently around 90 MB ballpark. 14261 individual records, 52286 media records, 162318 note records, 109884 source citations. Let me know if you raise the limit and I’ll upload it.
As for the other two, could you drop me an email offline?
Thanks,
Chris