Talk:XTools/2021

This page is a feedback forum for XTools. For reporting bugs, it's preferred that you use Phabricator.

If the issue is urgent and you're unable to use Phabricator, feel free to ping one of the active maintainers.

For research purposes, I'd like to get an XTools report for c.2k accounts (all bots) from en-wiki inception to 31 Dec 2020 (or via monthly counts I guess I can remove >2020年01月01日 edits). Most accounts are low activity and no longer active but data set includes some big interwiki bots and the AV bots that have high edit counts. I've given up trying to do this via the API (I'm not a programmer) but nor do I want to 'just' run a curl-type script against this tool in a way that is not respectful of server load and disruption for other users. I'm open to an alternative suggestion if this is not good, plus will be happy to give acknowledgement for assistance in any publication using the data set (if desired). Any ideas as how best to proceed?(happy to take further conversation to my user page here or on en-wiki (same username)). Mwra (talk) 12:44, 7 February 2021 (UTC) Reply

The biggest barrier I think will be getting data for very prolific accounts like w:User:ClueBot NG. We cannot feasibly run a full report for an account with millions of edits. This would take perhaps an hour or more to finish, just for that single account. XTools currently only shows limited data for accounts with over 450,000 edits.

What exactly are you trying to do? Depending on what data you need, there may be more efficient ways to attain it than through XTools (say a Quarry query or use of the various RESTBase APIs).

> nor do I want to 'just' run a curl-type script against this tool in a way that is not respectful of server load and disruption for other users

We appreciate that :) Indeed any high-volume access to the user-facing tool could cause service disruption. The XTools API on the other hand is designed for this very purpose. However we ask that you still ensure requests are made synchronously (i.e., you get the response back before making a new request). — MusikAnimal ^talk 22:07, 8 February 2021 (UTC) Reply

Thanks for the follow-up. As to your question...

I'm updating the dataset I made (manually!) of all bots ever used in en-wiki up to mid-2017 as part of my doctoral research^†. I actually over stated the number of accounts which is closer to 1,140 but it's still a fair number and, as you note, includes some big hitters like Cluebot NV. AV-bots tend to bet the most coverage but I'm analysing the bot space more widely, I'm looking both at the nature of bot work and how that meshes with wider perspectives such as user requests for bot effort.

XTools and previous similar report types have been useful for drilling down into actual bot work, as for many bots it's not unambiguously clear from BRFAs or edits exactly what they are doing. That's not presuming anything bad, but it is hard to get a clear picture of the overall work.

In order of need/want of data, it breaks down this way (all for a fixed 2020年12月31日 25:59:59 cut-off):

• en-wiki edit count to cut-off
• monthly breakdowns for the latter
• nice to have: break down by namespace

The additional XTools info is icing on the cake but could be pulled into a interesting useful hypertextual dataset. Thus it would be nice to have the bigger bots in there (even if the larger ones need a different approach).

On a check with the 1,068 items in the 2017 set:

over 1k edits 774 user accounts

of which over 10k edits 457

of which over 100k edits 171

of which over 450k edits 49 (XTools cut-off for full report)

of which over 1M edits 19

I hear you on the synchronous call aspect as I don't want want to make unreasonable draw on resource and he data-acquisition work isn't time-sensitive given the cut-off.

I'm aware than some bots edit in more than one wiki, though that it out of scope for my current work.

^†. The thesis should be up on my uni server soon (I'm just waiting some correction sign-offs) but I'm happy to discuss the bot work in the meantime if such niche stuff is on interest.

Does that help? Mwra (talk) 23:12, 8 February 2021 (UTC) Reply

It sounds like the User month counts API might give you all you need. The results are the edit counts per year/month, grouped by namespace. So long as you're okay with waiting, this should work fine for most of the bot accounts (but please again remember to make requests synchronously). If you are uncomfortable using the API, you can use XTools' Edit Counter but ask only for the month counts, which will improve performance quite a bit: https://xtools.wmflabs.org/ec-monthcounts

For bots with over 450,000 edits, you'll have to run the query yourself. Going off of the query used by XTools, I came up with this Quarry result for ClueBot NG. It took 750 seconds to complete. You can fork that query and change the username accordingly for the 49 accounts with over 450K edits.

Hope this helps! — MusikAnimal ^talk 16:20, 17 February 2021 (UTC) Reply

A big thank you - lots of useful pointers. Monthly granularity is fine for a first pass. Going deeper into the namespaces, etc. can wait, though the Quarry approach looks good^†. I genuinely appreciate the points you've given. As a non-coder, some things are hard to guess at outset as it all just looks like ... code.

Your proviso about not hogging resource is duly noted. (I agree, but don't mind being reminded of the need to think of other users of a finite resource).

As I want to run c.50 large queries (i.e. bots with > 450k edits) is this fair use of Quarry, or should I be setting up something on toolforge. For c.50 items I'm perfectly happy, from my end to may a Quarry query for each. Doing that sequentially, of itself, means I'm not pounding the server. But, I am adding 50 stored queries to Quarry. Again, the intent is to be considerate of other users.

^†. My first (successful!) experiment based on your query: https://quarry.wmflabs.org/query/52576 Mwra (talk) 12:16, 18 February 2021 (UTC) Reply

Unlike in XTools, Quarry has a proper job queue, so you shouldn't need to worry about over-using it. I think it's still possible for queries to timeout, though; but the threshold as I recall is pretty high. On XTools anything over 600 seconds gets automatically killed. — MusikAnimal ^talk 17:46, 18 February 2021 (UTC) Reply

Yes. A few notes in case it helps later visitors seeking help. I ran 49 accounts via Quarry using the query above, changing the bot user account name before re-submitting. Only one timed out, and I had to re-run one as I forgot to download the last run's data before changing the query code. Run times varied from c.2 minutes to c.15 minutes but with little relation to result data size, so I guess it's down to server load. I was able to do complete the task in c.6 hours, leaving a browser window open and waiting for completion. I did check documentation and if the query appears to lack a target wiki that's because (per Quarry's documentation) queries default to the 'en' wikipedia if no other is explicitly coded. Mwra (talk) 10:16, 19 February 2021 (UTC) Reply

@MusikAnimal If I use XTools for user "Uncle G's 'bot" (https://xtools.wmflabs.org/ec/en.wikipedia.org/Uncle%20G%27s%20%27bot) the edit count is

But using the Quarry query discussed above, that user's overall edit count is 5,450. Comparing the two, it seems the user contributions

Meanwhile, The Global Account Information page for the same user (

Comparing XTools (5,461) and Quarry/User-contributions (5,450) it seems the former has an over-count of 11 (10 in the first month (2005/8) of use and 1 in the second). Is this likely just less accurate record-keeping in the early days of Wikipedia?

But the Wikipedia API (https://en.wikipedia.org/w/api.php?action=query&list=users&ususers=Uncle_G%27s_%27bot&usprop=blockinfo%7Cgroups%7Cimplicitgroups%7Ceditcount%7Cregistration%7Crights) gives the edit total as 7,202.

The user's last edit was in January 2013 so this isn't a case of recent edits being missed.

Which is the most accurate source for edit counts? In general references to edit counts are deletions included, or is this an edge case relating to those accounts with deletion rights.

Can the Quarry query be modified to also list deletions, as that would ensure the 49 users in my set with >450,000 edits would align better with XTools info? Mwra (talk) 16:12, 19 February 2021 (UTC) Reply

CentralAuth and the Wikimedia API are basically a running tally of all edits made by that user. The logic as to what counts as an edit has changed over the years; For instance, page moves in MediaWiki used to not be counted as edits, but they do now. Thus I consider this "system edit count" to be an approximation. XTools and Quarry are more accurate, or at least consistent in the way they count edits, given they query the live database in real time.

The difference you see between XTools and Quarry here is because they are two different queries. The query I gave you is for the month counts, which should match what you see when using https://xtools.wmflabs.org/ec-monthcounts. The Edit Counter totals use this much more simple query: quarry:query/52714. Why they are slightly different might have to do with the use of action_revision specialized view instead of just actor. I'll try to investigate further when I have time, but either way I would be surprised if there were ever any dramatic differences.

As for deletions, the "system" edit count includes those, while the month counts query I gave you does not (but it can). Here's an example for Uncle G's 'bot (but as a separate query from the live revisions): quarry:query/52715 — MusikAnimal ^talk 06:01, 23 February 2021 (UTC) Reply

I created this page called w:en:Halime Hatun (fictional character), however on XTools it says one of my friends, IronManCap, who contributed to the article a lot created it, and then on my account information thingy page (I really don't know what you call it), it also says I created it. I don't know if this is an issue or there's an issue behind it. See his page and then see mine and then see that page's info and then see that page's history. Not much of an issue, just really confusing. D1982291 (talk) 17:32, 23 February 2021 (UTC) Reply

I feel like I'm being ignored. D1982291 (talk) 12:02, 25 February 2021 (UTC) Reply

Not being ignored! I saw your first message, just didn't have a good answer for you. Thanks for creating phab:T275764. This deserves a bit more investigation. — MusikAnimal ^talk 22:38, 25 February 2021 (UTC) Reply

What do you mean by investigation, @MusikAnimal? How this happened? D1982291 (talk) 18:19, 1 March 2021 (UTC) Reply

I mean I think it's a bug in the software, but it's not immediately apparent what the issue is, so we'll need to investigate :) Right now we're in the middle of a major update to XTools which is time-sensitive, so I may not be able to tend to this for another week or so. — MusikAnimal ^talk 20:07, 1 March 2021 (UTC) Reply

The new black background makes things harder to read, especially the parts that are written in dark blue. Can the white background be restored, please? SarahSV ^(talk) 19:14, 22 March 2021 (UTC) Reply

We just pushed an update that brightens the color of the links. Does it look better now?

You must have your browser or operating system set to prefer the dark mode. You can turn that off to restore the old look. But our intention again is to make a dark mode that is as legible as the light mode, so if you have any other feedback, please share :) — MusikAnimal ^talk 20:36, 22 March 2021 (UTC) Reply

I want to turn the dark mode off but keep my dark mode on in my browser. RXerself (talk) 10:11, 25 March 2021 (UTC) Reply

Would be great to be able to turn off the dark mode (of this app, without having to turn off dark mode for the browser/windows). No offense but this dark mode implementation is near-unreadable for me due to low-contrasting colors (and I have no visual impairments). Levivich (talk) 15:10, 25 May 2021 (UTC) Reply

Now that my edit count is over 500K, I can no longer access my details. I saw a thread below in which @MusikAnimal: had indicated that they were planning to bump the maximum count higher. Is such an update still in the works? Lepricavark (talk) 21:39, 31 May 2021 (UTC) Reply

I have bumped it to 550K for now. I can't promise it will stay that way. Further down the road, we may require users be logged in to query for users with a high edit count, and put it through a job queue so as not to disrupt the performance of other users (similar to how Quarry works). That's a lot of work, though, and I don't see it happening anytime soon. Until then, we have to have some sane limit or else querying for accounts such as your own will disrupt every other user of XTools. Frankly though, you should be proud of this. Very few editors ever become as prolific as you have! :) — MusikAnimal ^talk 20:05, 8 June 2021 (UTC) Reply

Thank you, I really appreciate it! I saw in another thread that someone else was able to get their information via Quarry, but I was not able to figure out how that site works. Lepricavark (talk) 01:53, 9 June 2021 (UTC) Reply

My edit count has climbed over 600K, and I have not been able to access my details for a while. Would it be possible to bump up the max count again? Lepricavark (talk) 18:12, 30 April 2022 (UTC) Reply

Right now, probably not... We haven't gotten a large number of query timeouts, but we have seen an increase in the "XTools is currently overloaded" errors, which super bad because that means the service is down for everyone. This issue only increases when we permit longer-running queries. There are a few things we can try to mitigate this which I will look into soon. I'll update you here when I've gotten a chance to look more into it. Apologies for the inconvenience! — MusikAnimal ^talk 23:25, 2 May 2022 (UTC) Reply

That's totally fine. I certainly don't expect you to jeopardize the entire service just for me. I will appreciate anything you are able to do, but there's no hurry whatsoever. Lepricavark (talk) 04:27, 3 May 2022 (UTC) Reply

Is the 'live edit count' displayed by the ec tool working properly? If I look at the contribution history of user Caustic3 for example, a page search for the string 'Reverted' yields 42 occurrences (out of 112 total edits). However, the ec tool shows "100% live / 0% deleted". It's not like the "deleted" figure is pegged at zero; running it against my username shows 417 deleted edits. Why the discrepancy? Mathglot (talk) 23:24, 20 June 2021 (UTC) Reply

Deleted edits refers to edits to pages that have been deleted. These edits are visible to admins at Special:DeletedContributions. Reverted edits are considered "live" in the sense they haven't been deleted (they're visible at Special:Contributions). It's of course possible and likely that the content introduced with reverted edits are no longer visible on the page, though.

More info at XTools/Edit Counter#Deleted edits. I've added a note about reverted edits. — MusikAnimal ^talk 04:00, 22 June 2021 (UTC) Reply

Hi! My "Mounth counts" chart has one problem: 2011-08 is so exceptional by the huge total of edits that it messes all the other months of all the other years. They look like nothing. How can I hide 2011-08 so my chart can work normally? MisterSanderson (talk) 07:49, 20 July 2021 (UTC) Reply

There is phab:T177658 which is about introducing a logarithmic scale to the charts. That would help here, too. In the meantime, since you primarily edited the mainspace during the month of 2011-08, you can hide the mainspace from the view and the bars for the other namespaces will be more visible. To do this, use the ×ばつ icon when hovering over the rows in the namespace table. — MusikAnimal ^talk 15:42, 20 July 2021 (UTC) Reply

Thanks for the tip! It is a really good one.

It would be much simpler to implement, and have much more satisfactory results, if each year count/month count had an "X" too to be hidden, and the graph recalculated on-time, like on the Namespace Table.

The logarithmic scale probably would be more difficult to implement, as it is already tooking some years to be replied to on Phabricator.

What do you think? MisterSanderson (talk) 22:25, 20 July 2021 (UTC) Reply

Logarithmic scale shouldn't be too difficult. We have a similar option in the Pageviews Analysis tool, which uses the same charting library. I just didn't see a demand for it other than the sole request back in 2017, so I never gave it much thought. You asking for it gives it a bit more priority :)

Adding ×ばつ links for each month would actually be considerably harder. That I don't think I'd be able to implement anytime soon, I'm afraid. — MusikAnimal ^talk 22:47, 20 July 2021 (UTC) Reply

Using xtools pages created, I've created a tool which compute gender statistics about the list of articles created by a user. It's pure Javascript.

See here https://observablehq.com/@pac02/user-level-gender-statistics-for-wikipedia

It could be useful to add this information on https://xtools.wmflabs.org/pages in the future. PAC2 (talk) 20:00, 25 September 2021 (UTC) Reply

What could that be useful for, and how would you calculate it? 2003:CB:F703:9600:7596:FFD6:C37E:FB62 (talk) 09:50, 2 October 2021 (UTC) Reply

All computations are here https://observablehq.com/@pac02/user-level-gender-statistics-for-wikipedia. The idea is :

• get the list of articles created
• get the value P21 on Wikidata using Wikidata API
• compute gender statistics PAC2 (talk) 05:15, 7 October 2021 (UTC) Reply

Hello, since the XTools limit is at 550,000, I was wondering if anyone could help me with making a Quarry function. Would it be possible to see my Top edited pages (like top 100 or 1000) in the mainspace on a Quarry? If so, I just need some advice on what to code it as. If this cannot be done, I understand. This is about all I need from XTools for my personal info. Red Director (talk) 17:56, 2 October 2021 (UTC) Reply

See quarry:query/59028. Just fork that and hit "Submit Query" to get the latest results.

That said, since the new replicas went live, timeouts have been rare, so I've gone ahead and bumped the edit count maximum to 600,000K. You're the only one who ever asks about this, which is why I haven't done it any sooner :) — MusikAnimal ^talk 17:20, 4 October 2021 (UTC) Reply

Perfect! Thanks for your help. I shouldn't need the limited bumped anymore if Quarry works. I know my edit count is in the small range of users that are affected. Thanks for all you do. Red Director (talk) 17:45, 4 October 2021 (UTC) Reply

Sorry to be so needy. Is there any way you can add a start date and end date filter to the one you made for me? I tried playing around with the code based on other user's codes and I cannot get it to work. Tried multiple times. It may not even be possible and that is fine. Red Director (talk) 19:47, 4 October 2021 (UTC) Reply

I've updated quarry:query/59028 with a date range clause, and some instructional text in the comments. Let me know if that helps. — MusikAnimal ^talk 20:02, 4 October 2021 (UTC) Reply

Yes! Thanks Red Director (talk) 20:25, 4 October 2021 (UTC) Reply

Hello again. Query 59077 is the reference point for this request. I need some assistance adding a sort field. Is it possible for the query to show the last time I modified an article using a timestamp? What I have found allows a sort by the first time I have edited a page by timestamp but I cannot figure how to make it do the last time. I know this is very picky on my end so I understand if it cannot be done. I have experimented and failed using various fields. Red Director (talk) 23:49, 31 January 2022 (UTC) Reply

XTools should exclude AWB-tagged edits in the time card, or at least provide the option to hide those edits, especially since high-volume editing with the use of AWB tends to drown out all the normal edits performed by an editor throughout the day. Chlod (talk) 13:36, 14 November 2021 (UTC) Reply

This seems more fitting for the AutoEdits tool. Going by tags alone should be fast enough, but preferably, as with AutoEdits, we'd also use regular expressions on edit summaries to account for edits made before the tag was introduced. I've filed a task at phab:T295703. — MusikAnimal ^talk 16:21, 15 November 2021 (UTC) Reply

Is there anyway to have those of us who have done some flood editing taken out of bot lists?

https://xtools.wmflabs.org/articleinfo/meta.wikimedia.org/Spam%20blacklist

Noting that I have this issue on numbers of wikis. — billinghurst sDrewth 10:04, 23 November 2021 (UTC) Reply