Forum:DB item population

Forum index > DB item population

By the way, would you guys be OK with us possibly adding the contents of the Darth Hater database to the wiki automatically or semi-automatically? If we go forward with it, we'll make sure that it is done in whatever formar you guys choose for things like items etc. I think having even basic versions of these articles on the wiki would give it more of an advantage over the other SWTOR wikis out there. Ausir 11:22, 22 November 2011 (UTC)

Personally, I wouldn't care for this. The database has many errors at the moment, things that seem to be an issue across all SWTOR databases. Items are labeled incorrectly, item source lists are incomplete (though this is something we should consider only generalizing here), variable items are not recognized, variable icons for items are not even considered, item stats are incorrect in many instances and/or out of date, etc. These are just some things I've noticed by going through the DB casually.

I think it would end up taking the same length of time to go through and correct these errors as it would to add the items individually by hand. It might even take longer. In the mean time, we'd have a load of error-filled articles we could only correct as we encounter things in the game. I guess what it boils down to is what is more important to us as a project, quantity or quality? I prefer quality myself, believing that if you strive for this goal others will take note, help out, and quantity will soon follow. That's just my opinion, though.

This is just my opinion, of course. If we do populate this project in this manner, I'll work with what we've got. Still, if there were some way of addressing the errors, of course, so that we could have both quality and quantity from the start, I'd be all for that. ;D -- Heaven's Agent 14:36, 22 November 2011 (UTC)

Hm, I'll think of what would be the best way to do this, then. How about we automatically upload all the icons for now, though? Ausir 15:42, 22 November 2011 (UTC)

The icons would work. We'd still have to go through and categorize them individually, but we'd have to do that anyway. Would we be able to specify a licensing template and define its parameters, and then have it be added to each icon's file page automatically? -- Heaven's Agent 15:45, 22 November 2011 (UTC)

Sure, I'm sure that's possible. I'll look into it. Ausir 15:46, 22 November 2011 (UTC)

As for importing the articles themselves, I think that while it has some downsides, there are also major advantages. While some checking of their contents will be required, I think verifying and correcting the (mostly numerical) data that is already presented in the proper wiki format (which I think can be arranged) requires less work than actually creating the articles in the proper format from scratch. And each such unverified article could have a template at the top saying that the data is automatically imported and needs verification (with an appropriate category as well), which could be removed after it's actually verified. And for some new users, verifying such data in articles that have already been created will surely be less intimidating than actually creating them from scratch. Not saying it's something we necessarily need to do, just something to consider. Ausir 16:25, 22 November 2011 (UTC)

I'm for importing articles and going through and checking them. Having a category where they're all dumped to so we know which ones to check will definitely help. The more information we have in here when the game goes live, the better. Gives people a springboard to adjust or add additional information from. -- Alianin T • C 16:36, 22 November 2011 (UTC)

I agree, just as long as whatever scheme is used doesn't leave us in danger of having content overwritten, I think this is a great idea. — Eyes File:User-Eyes-Sig.png 16:38, 22 November 2011 (UTC)

Sounds good then; I may have reservations about the import, but I can work them out in the long run. We'll have to focus on getting the ItemInfobox Template in a more finalized form; I was going to simply expand upon it as we encountered the need for new parameters in the game. I'll start actively searching for those parameters, so that it'll be ready.

Is the DB limited to what is collected, or does it pull all data about an item? I ask because the current DB articles lack a lot of information contained in the game's item tool tips. If a DB article shows all the information that is collected, I'll only have a few parameters to add to get the template ready. If everything is collected but simply hidden from the output (which I kind of hope is the case), I and anyone else in the beta are really going to have to get to work. :D -- Heaven's Agent 16:48, 22 November 2011 (UTC)

Actually, now that I think about it I'd rather we not import the icons from the DB. DarthHaterDB uses .gif-format icons. Images with this format do not re-size well at all, distorting horribly when scaled larger or smaller. From a practical standpoint a wiki needs to be able to re-size its images as appropriate; we need either .jpeg or .png-format icons. -- Heaven's Agent 17:00, 22 November 2011 (UTC)

Alright, I'll look into the technical side of things in terms of what is possible with a database dump and I'll let you know what can be done. In the meantime, keep up the good work on manually updating the wiki. Even if we import things, I'll make sure it will not override anything you add. Let me know if you have any questions or concerns. I'm also looking into integrating the database with the wiki more, e.g. database entries having links ot the wiki. Ausir 17:17, 22 November 2011 (UTC)

I've talk to our tech people, and they think it's probably be best to do it like we did it at Diablo Wiki - import everything into a Data: namespace, where the data will be included in infobox templates (like this one - you will, of course, be able to design the actual templates yourselves), and then create stub pages for all the items in the main namespace that will transclude the infoboxes from the data namespace. This will allow for data to be automatically updated in the future without overwriting the improved pages.

Whether you'll want them to be automatically updated later on or whether you'll prefer to check them and update them yourself will be up to you. You could also copy the data pages instead of transcluding them on the ones you've checked and improved manually, which will keep them from being automatically updated, while keeping the data pages transcluded on the ones that have not yet been checked manually, and can therefore be updated automatically along with updates to the database. If this sounds too complicated or if you have any questions or suggestions, please let me know. Ausir 00:30, 23 November 2011 (UTC)

So we're talking about automatically populating infoboxes for each item, and then essentially using those infoboxes as templates. That could be very useful.

The transclusion method seems to be the most beneficial of the proposed uses of the information; if we were to copy the information from the Data files, and put it into articles separately so that they couldn't be updated automatically, there would be little point in transferring the data in the first place. We'd still be creating each and every article by hand. Any word on what information the DB actually collects? We'll need to know so that we can prepare the infobox to handle all relevant data. -- Heaven's Agent 04:55, 23 November 2011 (UTC)

Well, generally the DB has the data that is displayed on the DB website. I'm not sure if there's any more than that. For now, let's make sure that our templates cover as much of the content that is displayed in the DB as possible. User:Ausir (talk) 16:59, 28 November 2011 (UTC)

Update: looks like we might be able to start with importing a data dump on items here next week. Stay tuned! Ausir (talk) 19:32, 2 December 2011 (UTC)

After going through quite a few items from the database, I'm a bit torn on whether or not to import. There's quite a bit of missing or incorrect information and I'm not sure how the updates to it work. For the articles I've been creating I've been referencing information in-game (when possible) and a variety of SW:TOR information sites and going with the most common information (e.g. mission rewards). I know we're going to have to comb through each article dumped here, but I'm wondering what's going to be the most efficient option. Anyway, just something to consider. -- Alianin T • C 19:43, 2 December 2011 (UTC)

I know what you mean. We're looking at the possibility of a lot of inaccurate articles being dumped on us all at once. It will be an extremely long-term project to go through them manually and update them one by one.

I'm not sure we need to transfer all the items here, anyway. If one of our users wants a general listing of items, we should be directing them to the DB. That's what the DB is for, and it can do the job so much better than we can. The only items we really need here are mission rewards, unique drops, crating items, and purchasable gear. That may be a lot of items, but these are all things that can be added to the project individually as we move forward: mission rewards will be added as our collection of mission articles grows, we can catch purchasable gear as we document vendor NPCs, unique drops will be added alongside the mobs that drop them, and crafting items can be added as the crew skills articles grow and are polished. We'll probably want to include moddable items as well; I'd like to establish a visual styles directory at some point.

The idea of the data being updated automatically is tempting, but we have no control over the data gathering process. We can't change bugs in the process or alter its function as our needs demand. We have no way to determine the accuracy of the information other than by checking it manually each time, which largely defeats the purpose of this type of data import. As I said before, I'm not too fond of the idea, but I'll run with whatever everyone else decides; in either case, we'll make things work. -- Heaven's Agent 20:01, 2 December 2011 (UTC)

Well, even if the data itself has to be updated manually, doing that will likely still mean less work than checking all the data and then creating the article manually, given that all the formatting, templates, categories etc. will be automatically added as well. And all such articles will be clearly marked with a template that says they need to be checked. Maybe we should import a portion of the data for now and see how well it works out?

And I disagree about directing users to the DB instead of covering items here - while both sites should complement each other, there's no reason for the wiki not to cover this part of the game as well, especially if we can cover also details that the database won't. Ausir (talk) 20:09, 2 December 2011 (UTC)

I chose my words poorly. We definitely want to document items, as they are a vital part of the game. We will be able to document information the DB lacks, and we'll be presenting the information in a much more personable manner. I was just trying to emphasize the we shouldn't be doing things like trying to document every mob that might drop a random item, or trying to set up large, complicated item listing in an attempt to create some sort of comparative display. There are simply some things we should leave to the DB. -- Heaven's Agent 20:21, 2 December 2011 (UTC)

Anyway, I think even automatically setting up pages for each item with empty infoboxes (or partial data) would be more useful than not setting them up at all, as it means that less time and skill is required to update them than to create them from scratch, especially for new, inexperienced users. For new users, creating a new page can be more of a daunting task than updating the stats on an existing one. And the earlier the pages actually are on the wiki, the earlier they'll be indexed by Google and will be more likely to come up in searches, thus directing new users here. Ausir (talk) 20:25, 2 December 2011 (UTC)

Actually, the fact that they would come up on Google is one of the very reasons I don't want to bring this information over. The communities going to recognize the information as a data dump pretty easily, something which does not tend to go over well when folks are looking for a wiki. Additionally, we may have more Google searches leading here, but those links will often lead to incomplete articles and, at times, incorrect information. We might actually get a negative reputation among the community from such usage. -- Heaven's Agent 21:06, 2 December 2011 (UTC)

Ausir, you're right about it being more daunting for new users to create rather than modify articles. Can we import some items since those are the most plentiful and least likely to contain errors and then go from there? -- Alianin T • C 21:30, 2 December 2011 (UTC)

Heaven's Agent: Well, they might be incomplete at first, but still, from my experience with wikis, users who come to a wiki from Google and find incomplete data, often simply start correcting/filling out the data they find and join the community, which they might not have done if they hadn't come to the wiki from Google in the first place. And I don't think this would give us a negative reputation - wikis being incomplete until they're built up by the community is an inherent feature of wikis, after all. And as I said, many new users might find it easier to modify such article than to actually start a new one from scratch.

Alianin: I was thinking of doing a dump with just items for now, actually. But I think the best course of action would be to start with just a fraction of the data at first to identify any potential problems. Ausir (talk) 21:32, 2 December 2011 (UTC)

Just FYI, in order to make the import easier, it's going to use a (slot) (type) category scheme, which means that while the current armor categories (e.g. Category:Legs-slot light armor) will stay the same, the ones for weapons will be renamed - e.g. Category:Main hand-slot lightsaber. This way they can be generated using the same naming scheme. I know that using singular instead of plural is not the usual convention for catgory names, but this time it would be hard to generate it automatically otherwise. Fortunately, we only have a few weapon articles for now, so there won't be much need for changes in existing articles. Ausir (talk) 18:01, 15 December 2011 (UTC)

I've got no problem with the new categorization. It may go against conventions, but there's a reason for doing so.

Another concern about importing item information recently occurred to me. If an item in game has two or more possible icons, due to differing appearances based on character faction, the DB only includes one of those icons. An item's icon depicts the item's appearance once equipped, so without both icons we are neglecting to document a big part of an item for half of our users. When I brought this up I was told the documentation of multiple icons was not a priority.

On our end I was thinking that we could just add additional icons by hand. However, I'm finding in a lot of instances that these alternate icons are simply not used on the database. It appears most of the information was gathered from the perspective of a single faction, the Sith Empire, and as a result the icons of the opposite faction are rarely used at all. As a result, we can't even hunt down these icons ourselves. -- Heaven's Agent 19:37, 15 December 2011 (UTC)

We have done an initial import, although there have been some issues that will be fixed with another import, which should be done on Monday. It will not overwrite any updates done to the pages manually, though. Ausir (talk) 02:52, 18 December 2011 (UTC)

I've noticed that as I've been working on the Synthweaving schematics. It actually does a good job of filling in gaps; It was good to have a tooltip for items I hadn't fully defined yet, even if it was incomplete and, in some cases, had one or two variables off. I think the latter is more an issue with the DB Information Gathering, though. There are many things the DB needs corrected if it is going to be viewed as a reliable information source.

Is there anywhere specific you'd like us to mention errors that we notice with the importing process? Would this forum thread be fine? -- Heaven's Agent 03:45, 18 December 2011 (UTC)

Yes, this thread would be fine. Things like e.g. capitalization in the categories are going to be fixed in the next import. Ausir (talk) 09:44, 18 December 2011 (UTC)

New import

A new import is now in progress. See e.g. Forcebound Adept's Cowl. The armor and weapon categories now use proper capitalization, and GIF images are embedded for the icons via an URL for now. Since we weren't able to upload PNG icons instead automatically, I figured it's best to embed the GIFs for now, and then replace them with properly uploaded PNGs, instead of leaving them empty or uploading all the GIFs here and then having to delete them when replaced by PNGs. Ausir (talk) 17:30, 20 December 2011 (UTC)

Good idea about the gifs. Any news on your end about item with multiple icons? It's bad enough that we don't have access to some icons, but even worse that they won't even be recognized on imported items. -- Heaven's Agent 20:07, 20 December 2011 (UTC)

For items with multiple icons they will need to be added manually, as for now we're just using the ones used by the DB. Ausir (talk) 21:28, 20 December 2011 (UTC)

Is there any way to look through the icons? Right now I'm having difficulty locating the alternate icons, due to the fact that they aren't used in the DB at all. -- Heaven's Agent 22:16, 20 December 2011 (UTC)

No idea, I'm afraid. Ausir (talk) 22:22, 20 December 2011 (UTC)

Errors

There is no "belt" slot in this game, nor a "leg" slot or a "gloves" slot. They should be "waist", "legs", and "hands", respectively. We need to use the proper terminology from this game, not holdovers from WoW or other fantasy MMORPGs. I don't think there's a "helm" slot either, that is simply "head", but I'm going to have to log in to check that and I'm stuck in a queue. I love launch day. :D -- Heaven's Agent 20:16, 20 December 2011 (UTC)

Just verified. It is "head" slot. -- Heaven's Agent 20:47, 20 December 2011 (UTC)

These are the terms used in the db, so any changes will need to be done either manually or via a bot. Ausir (talk) 21:28, 20 December 2011 (UTC)

Update: These terms will be fixed in the next db import, but it will have to wait until after the holidays. Ausir (talk) 21:59, 20 December 2011 (UTC)

I'm adding the missing categories even if for now they're redundant - at this point it is better to have the redundant categories added than to have red category links on so many articles. Ausir (talk) 18:39, 21 December 2011 (UTC)

As you might have noticed, we're in the process of fixing the incorrect categories using a bot. I've just noticed that e.g. Gunrunner's Pistol is in main hand/offhand categories. Does this mean that the "primary ranged" and "secondary ranged" categories are wrong, and should be replaced with "main hand" and "offhand" like with melee weapons? Ausir (talk) 02:25, 11 January 2012 (UTC)

Ranged weapons use main hand/offhand like melee weapons. It's just labeled "Main Hand (Ranged)" and "Offhand (Ranged)". -- Alianin T • C 12:47, 11 January 2012 (UTC)

All category names should be fixed now. Ausir (talk) 20:19, 12 January 2012 (UTC)

I just noticed a discrepancy. Shield and power generators and focii are labeled in game as "Offhand (General)" but I see shield-slot categories. This slot doesn't exist, so do we want to lump them all into a "Offhand-slot general" category or subcategorize them? -- Alianin T • C 21:34, 12 January 2012 (UTC)

How about "Offhand-slot armor"? Ausir (talk) 15:13, 25 January 2012 (UTC)

Most slot names should now be fixed. Please report any more errors you find. Ausir (talk) 04:58, 21 January 2012 (UTC)

Instructions for Contributors?

What should we be doing when checking items? For Compact Medpac, I checked the data, it was correct. I removed the link to the database and inserted manual data, including Category:Medpacs. For Battle Fortitude Stim, I checked the data, it was correct. I left the link to the database and made a small change to the text and included the Category:Stimpacks. Which is better: manually re-entering the data or leaving the link to the database in? Ota (talk) 18:00, 24 December 2011 (UTC)

You'll want to manually re-enter the data and format the page as a regular wiki article, including a brief introduction, source and usage sections. That said, you're welcome to update only the infobox if you don't want to mess with anything else. If this is the case, we simply ask that you add the {{stub}} tag to the article so that someone can be alerted to its status and finish it in the future.

You can use existing articles as examples of information that should be included. Additionally, try to include the appropriate item icon. If you cannot find one currently included in the project, feel free to let me know and I'll get it added. -- Heaven's Agent 18:07, 24 December 2011 (UTC)