Automated UFO Trials

[Bomb Bloke]

Note that when designing a test map, windows are useful (because shots go through them some of the time). Fences, less so.

We can custom build tiles, too, which simplifies things somewhat.

 

BB, when I run your fire-logging scenario, both the alien and the soldier are at the "bottom" of the map. So any misses to the "left" disappear off the map.

At the bottom of the map? They shouldn't be; I placed the alien in the dead center.

 

I haven't got your email. Which box did you send it to?

[MikeTheRed]

I haven't got your email. Which box did you send it to?

 

The same on your SC Profile. Let me know if you want it sent somewhere else. SC won't let me attach things here or in PMs.

[Bomb Bloke]

No luck as yet. Hasn't hit my junk filters or trash can, let alone my inbox.

 

I have an address on this server, account name "bombbloke".

[MikeTheRed]

No luck as yet. Hasn't hit my junk filters or trash can, let alone my inbox.

 

I have an address on this server, account name "bombbloke".

 

I just sent it again, both to hotmail and your s/n at strategycore.co.uk

[Bomb Bloke]

Rightyo, my StrategyCore box remains empty, but I've now picked up a mail in my HotMail box. Which refuses to let me download the file as it is a "virus".

 

Now that memory reasserts itself, I remember that to send ZIP files through HotMail, you must not only rename them to something like MP3, you must change the header bytes as well. Annoying to say the least.

 

I have yet another address at tasmail.com, username "bomb_bloke".

[MikeTheRed]

Ok, I sent it there. You should be able to see from the cc list on the hotmail one, whether I got your SC address right.

[Bomb Bloke]

Oh. I get it. You're shooting the wrong floater. I forgot about him.

 

My fault really, as he's the first thing you see, though I did point out the map co-ordinates of the one you're supposed to aim at. The extra is there to prevent an annoying bug in the game. Shots against him won't be recorded, as UnitRef[81] doesn't flag when you shoot a "friendly" unit.

 

I advise just using the script file. It'll make things that much easier, and I strongly doubt a human can outpace it.

 

You got the StrategyCore address right, but the box is still empty. Odd. I got the file in my TasMail account, however.

[MikeTheRed]

Ah, doi on me. Oh well, at least it was something easy to solve.

 

Right, the results look reasonable now. And - mirabile vide - the script works great! You are el33t dude I myself was not able to get AHK to work with the DOS or WinCE version.

 

At first, the script stopped after running once. Then I saw my fire logger DOS box had a different name than yours (bb_fire.bat, started from a desktop shortcut). So, anybody else trying to run this: if the script stops after one round, just look at the title bar of BB's fire logger and edit all the C:\WINDOWS\system32\cmd.exe references at the end of the AHK script to match exactly what the logger's title bar says... that's all it needs.

 

Right, I'm getting 4 shots a minute, too. I'll leave it running when not using my PC, and collect data for various FFAs.

 

A couple of little questions:

1) It's using Snap shots, right?

2) There appear to be some missing wall tiles sort of "behind" the alien - but that's just some freaky shading/shadow thing, right?

 

P.S. The email I sent to you at SC was returned with error "Site strategycore.co.uk (216.118.97.121) said data: 550 This message contains a malicious file or content - (Oversized.Zip)". I guess it kicks back emails with zip attachments.

[Knan]

Sounds rather like the mailserver said: "the zip file was too big, I don't want it, go away".

[MikeTheRed]

Sounds rather like the mailserver said: "the zip file was too big, I don't want it, go away".

It can't handle 20k? Maybe we should take up a collection fund for more server space

[Knan]

All right then. Bad error message. Baaaad. Now behave. *thwack*

[Bomb Bloke]

A couple of little questions:

1) It's using Snap shots, right?

2) There appear to be some missing wall tiles sort of "behind" the alien - but that's just some freaky shading/shadow thing, right?

 

Yup, snap shots. The "missing walls" are covered by fog of war.

 

I've recieved RAR and MP3 files of up to nearly a megabyte in the SC account before. On the other hand, the server has recently been "upgraded", so anything could be the problem.

[MikeTheRed]

Cool... I left it running overnight. It stopped after six hours, at the point where the logger was trying to load TACTICAL. Hmm.

 

Anyway, I'm about to leave for Thanksgiving with the parents. I'll be back tomorrow afternoon. I've changed the settings to FFA 50 by editing OBDATA to 100 FA, and the soldier to 50 FA. The source savegame is in \GAME_98, so that's where soldiers edits should be done, right? Also I changed the Laser Rifle to strength of 255, to reduce the chance that 0 damage is rolled.

 

If I'm reading your notes right, in the future, I should be able to move the soldier, save the game, then copy it to GAME_98. This will let me test different distances, etc.

 

Does everything above sound like there should be no problem?

 

Thanks again for your logger and the AHK script!

[Bomb Bloke]

Game_98 is the correct one to edit, but remember that the logger is hard coded to calculate results based on the soldier's original location. Likewise, the logger allows you to edit his accuracy.

 

I'd need to make a new version to allow moving of the soldier. On the other hand, you could move the alien about: Direct hits will still be recorded as striking his original location, but the angle will still come out as (0,0).

 

Sometimes the script will mess up. I can make it very stable indeed, and in this case I can eliminate rubbish data in the results, but if the system misses a single key stroke just once, then the script won't do it again. Best way around it is to disable scheduled tasks on your system. Virus scans, for example, use enough CPU power to potentially mess up the scripts timing.

[MikeTheRed]

Ok, I'm back - and the script's still running, ~32 hours later! This would've

 

FWIW, my virus scanner did run while I was just away. But I have a dual core CPU so it may not have impacted it much. From a system perspective, your logger isn't using the PC much. The WinWait documentation says it waits indefinitely unless otherwise specified (and of course you didn't otherwise specify).

 

All in all, your logger seems pretty stable to me... if you can think of something obvious to improve, go for it. Otherwise, why not wait for me to give feedback on abends (if there are any more) and, if they point to some problem area, you can work on that.

 

I thought it would be better to move the alien, than the soldier. The reason being, to get the most precision on the spread of the bullets (i.e., always have the soldier firing as far from the back wall as possible). That shouldn't be a problem, eh? I'll just need to save a game with the moved alien, then copy it to GAME98. Wait - the soldier doesn't have a psi amp. I'll have to use the wiki to find out how to hack a move. No biggie.

 

I've noticed something odd... whenever I kill the alien (now that the laser is strength 255), my soldier's morale drops from 255 to 29. Odd... but undoubtedly irrelevant to firing testing.

 

Right, I'm so used to editing Firing Accuracy myself that I forgot you had that in there, as of that writing. (Then I got reminded right after posting that message.) Thanks for that.

 

Right now, I need to do something else, so I'm trying another FFA, and still haven't had any chance to look at any results. But did want to whip off this note.

[Bomb Bloke]

You can either use my map editer to move the alien, or use the psi amp which is below the soldier on the ground. If you mind control him, remember to wait until the effect wears off before you continue testing.

 

Also remember that you'll have to redo the AHK script to aim at the aliens new location.

 

It occurs to me that the alien isn't needed at all. The soldier could just aim at the tile on the other side of the map.

 

When you kill the alien, morale goes up. Because I set it too high, it loops around to 0 and starts counting up from there. Set it back down to the usual 100 to remove that glitch.

 

Remember to remove any results which say the bullet ended up in tile (0,0,0). They are all misfires. Raising the bullet strength will, as you mentioned, give you less of these, but do remember to remove those you do get.

[MikeTheRed]

It occurs to me that the alien isn't needed at all. The soldier could just aim at the tile on the other side of the map.

Maybe it is needed, maybe it isn't... more grist for the testing mill

 

I forgot about 255 wrapping around. Damn, I get so single-minded sometimes. It makes perfect sense.

 

Good selection of weapons down there. Just in case it matters. I'll try using the psi amp and re-working AHK.

 

Back to data collection... and other things IRL... Thanks again for the cool tools!

[MikeTheRed]

BB, your logger worked fine last night (9+ hours). I pulled in the data and played with it some. FWIW I'm getting ~235 samples/hour; this was with removing 1 second at the point where it waits for the user to possibly enter 1 to stop.

 

There are too many numbers to make some nice tables at this first point, because 1) I think I need more data, and 2) have a few questions about what I'm seeing / whether it's working right...

 

First, an overview of results. There have been 3 tests: FFA 0, 50, and 90. N 1421, 7367, and 2200. FFA 0 was Laser damage 60, the other two damage 255. All have a soldier at the edge of the map shooting at an alien in the middle of the map (25 tiles away); both are at level 1 (floating 1 tile above ground) and BB's logger records where on the map shots hit (or if the alien is hit).

 

The interesting part is that the angles of misses definitely increase, with decreasing FFA. Angles of Width (neg. means left): FFA 0: -22.2 to +23.6; FFA 50: -18.1 to +16.8; FFA 90: -9.3 to +9.3. Angle of Height (neg. means up): FFA 0: -2.3 to +3.4; FFA 50: -2.3 to +2.1; FFA 90: -1.2 to +1.2. FFA 0 is lopsided because some are being lost off the top of the map.

 

Averages for Angle of Width were all near 0, as might be expected. Angle of Height is probably a 0 average also but FFA 0 is a little messed up by the lost shots.

 

Clearly, we're seeing an "oval" pattern, much wider than taller. An exact picture is a bit hard to come by, though, given the limitations on Angle of Height (there are only 4 elevations that can be recorded, unless it hits the ground). And it's clear that the min and max angles of missed shots decrease as Accuracy increases.

 

Unfortunately, trying to see the pattern of how missed shots were distributed is difficult. As I said, the averages are near zero... but that could result from flat-line, bell curve, or even more funky distributions. I'll have to think about that... I wonder if I could try to reconstruct in 3D, how many times each tile was hit?? Hmm.

 

The percent hits on the alien was as follows: FFA 0: 2.2%,; FFA 50: 49.0%; FFA 90%: 86.7%. At first glance, this seems consistent with my simple model for firing accuracy.

 

Note that there were a number of hits on the far wall, directly "behind" the alien. Thus, you can't trust angles of 0, 0 to say there was a hit on the alien. Instead, you have to use the coordinates (relative or absolute) of the alien. BB, can you confirm for me - are you using unitref[81] as a guide for outputting the alien's coordinates? (How exactly does the unitref[81] test fit into your output?)

 

Also note that 29% of the FFA 0 results were "lost", aka Abs. Coords. 0,0,0. However, NEITHER of the other tests had a single lost shot.

 

I can understand that none angled off of the map for FFAs 50 and 90... but even at Damage 255, 1 in 511 shots should be Damage 0. For the alien, Damage 0 hits would still set Unitref[81]. Ok. But for tiles, they should not register. 3760 of the FFA 50 shots missed, which should have resulted in ~7 lost shots for FFA 50. But I didn't see a single one. The FFA 90 test had only 293 misses, so it could easily have not lost a single shot.

 

Because none of the FFA 50 misses were lost, I wonder if I am interpreting "lost" correctly relative to your output. And/or is it working right?

 

I see that all the back wall tiles have the same Angle of Height. I can see this for the same level as the shooter, but not for the other levels... tiles farther off to the side have a longer hypotenuse and different angle. I can figure them myself. If a tile is considered 24 pixels high, how many is it wide, I wonder?

 

Thanks again! - Mike

[Bomb Bloke]

Going by my theory the target should be hit more often then any other tile. Being the central point it's open to more possible lines of fire. This would not be true if the alien was only a single pixil in size however, but as it stands slight misses to the left or right will still clip the alien. The effect becomes more pronounced as accuracy increases, of course.

 

BB, can you confirm for me - are you using unitref[81] as a guide for outputting the alien's coordinates? (How exactly does the unitref[81] test fit into your output?)

Indeed, UnitRef[81] is used. If it is higher then 0, the logger outputs hard coded co-ords which match where I placed the alien (meaning it'll say the same thing even if you move the alien). Otherwise, it checks the map to see which tile was hit. If no tile was hit, the result is (0,0,0).

 

Remember, UnitRef[81] doesn't flag if the alien is mind controlled - The logger will "break" if that happens. Hence why you got results of (0,0,0) when you shot the extra alien on the map.

 

I can understand that none angled off of the map for FFAs 50 and 90... but even at Damage 255, 1 in 511 shots should be Damage 0.

No, your results are correct - Weapon fire does between 25% - 75% damage to terrain, never 0% (unless rounding very small values, not a problem when using shot power of 255). All shots would have registered so long as they didn't fly off the map.

 

If a tile is considered 24 pixels high, how many is it wide, I wonder?

16 pixils/units. Hence a tile is 16x16x24.

[zaimoni]

Since the LOFTEMPS data is 16x16x12, may we assume that each LOFTEMPS voxel is two graphical voxels stacked on top of each other?

====

Just got a chance to try out a new monitor against X-COM CE. At 1440x900 with weird aspect ratio, I'm getting enough pixellation to see details in real-time (shot speed 1, 1GHz Pentium III). [Have to tell the monitor to slide the screen around, but then again the only after-Thanskgiving bargain about it was the price.]

 

FFA 51 Laser Rifle Aimed Shot against Floater, range 35 due north, same level, XCOM unit standing.

** With the extended length of the laser shot compared to the other two, it was possible to visually observe an exact hit vs. a calculated miss immediately: the first jag in the laser beam's rendering gives it away. If we tied video recording into AutoHotKey (FRAPS?), we could use the video to (manually) measure the angles to high precision. The "target" centroid voxel is at the same height and E-W offset as the laser firing voxel. I haven't tested the N-S offset yet.

 

More generally:

* The game draws an "impact halo" labeling the voxel where it thinks the hit occurred. That halo never occurs in free space for unarmored XCOM units. In particular, it is possible for a plasma shot to miss an unarmored XCOM unit by passing one voxel above the shoulder next to the head. I'm guessing now that the LOFTEMPS data for the floater says that hitting the Floater's cloak doesn't count (all of the pass-through graphics shots have been through the cloak).

[Bomb Bloke]

Since the LOFTEMPS data is 16x16x12, may we assume that each LOFTEMPS voxel is two graphical voxels stacked on top of each other?

That's the current assumption, given that all evidence counts towards a tile being 24 units high.

 

It is possible to produce multiple places a bullet can land which follow the exact same vertexes on a 2D monitor. Luckily, for this to happen there have to be some oddly shaped objects in the area (basically upside down pyramids), so I don't think this will be a problem when shooting units.

[MikeTheRed]

An update: I am collecting data overnight and then while away at work, so that's ~17 hours each day. But I'm not analyzing it until I have another big bunch, since it takes a little time to massage the CSV, whenever it's sucked into the database. By the weekend, I should have lots of FFA 0, 25, 50, 75, and 90, so we can see the "cone angle" trend.

 

BB, it only crashed that one, first time I tried it, so it seems real stable. I had a few windows up in the background that first time, but I don't leave anything else running now, so maybe one of them got nasty.

 

Some simple tests for the distribution have occurred to me. For example, each side (left and right, up and down) should be a mirror of the other - if it's symmetrical. Also, one can compare the "wings" (misses to the left and right of the target)... if it has a radius of, say, 7 (where directly behind the alien is 0), I could pool/average the data from both sides for 1) tiles 5 & 6 and 2) tiles 2 & 3 (assuming they look comparable), and see if there's a difference. This would be a test for central tendency. I wouldn't want to use tile 7, since it's probably partial, or 1, since there may be some "accidental hits" that the alien soaked up, that were bound for tile 1.

 

BB, if you ever revise the logger, here are a few ideas. But none are critical... it's a fine applet.

 

1) If you could put a "Begin" and "End" line in the TXT log, that would save me from doing it. With some relevant things like date and time, FA for weapon and soldier (or just FFA)... maybe absolute locations of soldier and target, though that's less necessary. Something like:

 

*** BEGIN 11/26/06 11:50 pm FFA 25 Dist 25 ***

 

Don't put commas in it

 

Right now I'm adding notes like that one manually, because I suck it all into my mdb at once, after multiple sessions. (It's easier for it to be one doc, so I don't have to import several.) Thus I need to see when I changed settings, and it's otherwise interesting to see things like the speed.

 

It would be output when the logger started, and when it ended. Also see #4.

 

2) If you could collect a few things on the alien, I could also provide averages on e.g. damage (which would be a confirmation of the Damage equation) but more importantly, Fatal Wounds. I don't think anybody's figured out the relationship of FWs to Health... as long as I'm doing a zillion shots, it may become obvious after collecting data. It's probably a relationship between penetrating damage and FWs, like it was for armor. But if you think any other variable might matter, throw it in the log too.

 

One thing, though, is that the log (and resulting db) gets pretty big with all these individual lines. (MS Access is such a bloated pig.) So try to keep anything repeated each line, to a minimum. E.g., instead of outputting all 6 FW areas, there's probably only 1 area hit. So you could put the number of FWs, then maybe an abbreviation for where (L=left arm, R=Rt arm, l=L. leg, r=Rt. leg, T=Torso, H=head). (And test that only 1 area got a FW.)

 

This may even provide the interesting result that, e.g., with a very small cone (high FFA) at distance 1, maybe only the torso is hit, indicating that they actually track hits, instead of just randomly assigning FWs. I doubt they do, but if the data is already being output, it'd be easy to test.

 

I'll hack the armor to 0, and worry about the weapon strength. BTW thanks for pointing out the Damage to tiles... I even wrote the summary at the top of the wiki Damage page, then overlooked it. All's well that ends well.

 

You can also stick damage to floor tiles in there, if you want. I'll worry about the fact that e.g. 255 could make it spin a bunch. (I'll tone down weapon strength, actually.)

 

3) Are you doing a test that: either exactly 1 tile is hit OR the alien is hit (UR[81]=1) OR the shot was lost? This will just help make sure that neither I (nor any other user) accidentally puts a "used" game in slot 98, and then all the results are mucked up. I imagine your logger might only need to test this when it first starts (see next point).

 

4) Speaking of which: I believe your logger is doing the whole nine yards each time it runs, of copying G98 to G10, swapping modified map, etc. If you want to bother, you could instead make the "copy in" and "clean up" be options on the logger's main menu. Maybe it will speed it up some... although probably not tremendously. So it's your choice. FWIW you're welcome to use two slots (9 & 10) so you can have the pristine savegame in one, and the results in the other.

 

IIRC you made the logger before figuring out AHK. Now that we see it works fine (for more than one person, laugh) and is real stable, it might be worth it to optimize the process a little like this. But it's no biggie either way.

 

5) FWIW, it's probably too much hassle to get your logger to move the alien. A.k.a., I can probably do it myself quicker than the time it'd take you to program it. But if it's real easy, go for it.

 

Speaking in general, it seems to me like it's best to move the alien, since that leaves the most spread for misses (with soldier at the other edge of map). It also makes results directly comparable for seeing if there's a simple cone (not related to the distance to the target, like you're thinking). Otherwise, it's kind of hard to compare results, given how e.g. there are only 5 vertical levels/angles, etc. ... if you move the soldier, then you're getting much different results (in terms of the tiles hit by misses), simply because you moved him.

 

Ultimately I'll do at least a little testing of variations, like moving the soldier, maybe testing diagonals, etc. (If I don't get too bored.) But most of it will probably have the soldier stay where he is.

 

***

 

All the above is optional. The logger is perfectly fine as it is... and it's one helluva tool, at that. But if you want to tackle any of the above, that'd be good too.

[Bomb Bloke]

Regarding the collection of alien statistics, that can be achieved by merging my original logger with the new aiming logger.

 

Ignore the bit about XcomUtil, that's just making sure you have a split EXE (which you already do). Once extracted, read through the file "bb_log.txt" to get a feel for how it's supposed to work, then run the fire logger and perform one test shot. Exit out, then run "bb_reset.bat" to prepare the old logger for operation.

 

Right click "bb_fire.bat" and select "Edit" off the context menu. Change the file to match this (just copy and paste the whole thing):

 

set CLASSPATH=%CLASSPATH%;.;

if not exist %1ufoexe\black.exe goto loop

%1ufoexe\black

%1sound\sndstart

 

cd bb_tact

 

:loop

firelog

if errorlevel 1 goto cont

 

goto done

 

:cont

 

cd..

 

if not exist %1ufoexe\black.exe goto wintact

%1ufoexe\black

%1ufo2exe\tactical "1" %1

cd bb_tact

java Log

goto loop

 

:wintact

start /w xcloader tactical.exe "1"

cd bb_tact

java Log

goto loop

 

:done

 

pause

Now when you use the firing logger, you'll also trigger the original. Hey presto, health and fatal wounds all recorded... Along with just about everything else I've ever managed to think up.

 

Yeah, I know, the data's all scattered amongst a ton of files. But I'm sure you can work something out.

 

I'm fairly certain that fatal wounds are directly linked to where the unit got hit.

 

It is the case that the logger checks to see if the soldier has hit the alien, and if so, a result is outputted right away. If the shot missed the alien, the map saved to slot 10 is compared to the map saved in slot 98. If you "mess up" the original map, all saves to slot 10 will be "messed up" in the same way. Because the logger looks for differences between the maps, phantom shots won't appear in the results. However, if you knock a wall out or something, it'll be impossible to tell if a shot went to that location (as you can't shoot a wall that isn't there). Likewise, if you copy a save to slot 98 in which the soldier has already fired a shot succesfully at the alien, the maps would never get compared at all.

 

If the soldier did not hit the alien, and the logger finds no difference between the original map file and the saved map file, then the final shot location is never determined and (0,0,0) ends up as the output.

 

The logger never moves the game from slot 98 to slot 10. Instead, it tells UFO to load the game directly from slot 98. Technically, you can make the game load from just about any folder, but I decided to stick with convention (ie. GAME_xx). The logger does move backup files around so your original slot 10 save game isn't lost when you save over it, but this is only done when you first run and when you later exit the program. No "house keeping" is performed while testing is actually progress, just a comparason of the before and after save files.

 

Moving the soldier really is not recommended. The logger depends on him being where he currently is. I may edit this later, to allow diagonal shots and the like, but for now you're stuck with his current position. The relative shot locations and angles of fire are all calculated according to it.

[MikeTheRed]

Sounds good, BB. First I'll play with the data I've generated so far, then try including your original logger. Also, it's good to hear you're already doing some of those efficiency things I mentioned... I didn't know you could load a savegmae from most any directory.

[MikeTheRed]

I got busy last weekend and didn't get a chance to pull the data until now...

 

Dammit, there's something screwy. In my first look (above), 28.6% (407/1421) shots went off the map at FFA 0. This time I added a lot more data - but the new FFA 0 data only had 1.4% (51/3554) shots off the map.

 

I've double-checked everything I did and have no idea what went wrong. It also makes all the other numbers suspect... if this is so screwed up, who knows what's going on with them, too.

 

Be that as it may, I have been adding to the FFAs I sampled previously, plus added some more (FFA 10, FFA 104).

 

At least I've now semi-automated the data suck-in. That's the good news. The bad news is I'll have to re-do some of this to determine where the problem might be and/or which data to kick out.

 

This stuff is slow going, since it runs overnight, and I usually only have time to import and look at the data once a week (on the weekend). Oh well.