Server Issues (Fly South Bug FIXED!)

[=VG= ciro (Inactive Duty)]

So is this related to the known mission bug and that crashes PR?

Does PR crash on a certain map? mission?

I placed a script that checks if Mumble and PR are running and restarts them if they are not. If thats not working thats starange i'll take a look tomorrow evening.

Could be that the script only checks every hour i think, may have to change that.

How often would you like the server to restart? I know that a lot of Arma 3 servers restart at least 4 times in 24hr is that sensible for PR too?

 

 

__________

UPDATE:

1 hour ago, =VG= SemlerPDX said:

Alon has provided a file and this issue should be gone now.  The fix is in place and the server has gone for the last 8 hours or so without a crash.

Please actively try to crash the server by flying south of the map border, and report if it actually crashes the server.  We think it will not.

If the server does crash, please note the time (in ZULU/PRT/GMT) and the map (and if known, the next map that was going to be loading).  The crash dump system is still running, so I can cross reference any crashes with dump logs and provide them to Alon if needed.  As the issue involves memory allocation, it may be difficult to reproduce even if it is not fixed.  But we're assuming it is.

 

Quote

[R-Dev] AlonTavor wrote ...

Thanks, I have found the exact cause.

Someone forgot to check bounds on the heightmap array in the AI module. Going east or west just reads the next row of the array. Going north or south overflows/underflows the array and reads junk data. If memory layout is such that before/after the array the page is unalloacted, accessing that memory segfaults.

Interestingly, this seems to happen only with south? I guess the initialization order could guarantee that the next pages are allocated.
Since memory allocation methods depend on the OS, on some systems it can happen more often.

I'll try to fix it for 1.6

 

Thanks again to everyone for your patience and understanding throughout this whole thing!!  Cheers!

 

[=VG= 0100011000101]

37 minutes ago, =VG= ciro said:

So is this related to the known mission bug and that crashes PR?

Does PR crash on a certain map? mission?

all maps.

Currently the server ceahed every 2nd or 3rd game, so he automatically restart...

At the moment it is really bad. 60-70% of the crahes are due to the south-bug but we also have more crashes when entering a new map, so when it's done loading and you could actually join. then 5min is nothing at all and then you are back in the server selection ... but there are also more chrashes in the middle of the game, without someone flew or was too far in the south.

that the server does not come back after a crash has not occurred since a few weeks ... at least not when I was online.

 

Unfortunately, you can not say why the server crashed, unless you are just on the map-view.

From now on I will write down the maps that crash when loading but the rest will be hard to find unless you look at all demo files.

 

[=VG= 0100011000101]

but does anyone know why the server just crashed at south? I mean something has to be different than in the north, west or east ...

how meticulous was the search for the bug? and what about the other servers? only coop or even deployment?
I did not find anything about this bug at realitymod.

it may make sense to carry information about this bug together so that we can narrow it down.

[=VG= SemlerPDX]

21 hours ago, =VG= ciro said:

I placed a script that checks if Mumble and PR are running and restarts them if they are not.

Not for Mumble, but TCAdmin already does this for PR... it's why the server restarts so quickly when it crashes (99% of the time)

20 hours ago, 0100011000101 said:

how meticulous was the search for the bug?

Quite.

The server was even freshly installed and given a week without any serious VG modifications to any configs aside from PR Admin assignments and the maps to make it COOP.  When the server demonstrated the "fly south bug" crash more than once (on more than one map), we were fairly certain the bug resides in the game.  Melon has personally poured through pages of code chasing any lead as far as it would go, to no success.

Really can't wait until this one gets wiped out - worst bug ever.

 

20 hours ago, 0100011000101 said:

it may make sense to carry information about this bug together so that we can narrow it down.

It does seem to be specific to COOP mode. 

[Xenalite]

Can we resign people from CAS if we know that they are notorious for flying south and crashing? It's probably a worse disruption to the server than people TKing in main.

[=VG= Acro1]

6 hours ago, =VG= SemlerPDX said:

The server was even freshly installed and given a week without any serious VG modifications to any configs aside from PR Admin assignments and the maps to make it COOP.  When the server demonstrated the "fly south bug" crash more than once (on more than one map), we were fairly certain the bug resides in the game.  Melon has personally poured through pages of code chasing any lead as far as it would go, to no success.

Is there a way to publicize the respective files so we can have a look at them?

 

2 hours ago, Xenalite said:

Can we resign people from CAS if we know that they are notorious for flying south and crashing? It's probably a worse disruption to the server than people TKing in main.

While this could fix an acute situation, I do think resigning people 'before the crime' is  a situation we best try to avoid. How many incidents before someone is 'notorious' ? It will take several days before every admin knows that player X is a southflyer, and several days again if we decide to give them a second chance at flying. Half of the admins will resign them on sight, half won't. This will cause a lot of confusion and create an unfair feeling towards many pilots. Even the best of our aviators hit that southern edge once in a while, and when this goes unpunished it widens the unfairness gap. The people who are notorious south-crashers are usually also disruptive in other ways (stealing, bad communicative behaviour,...) and are already being dealt with. Unfortunately, the only fair and effective solution to this problem, IMO, is fixing the bug.

 

Many new pilots join every day, with decent flying skills, that end up flying out of bounds because they didn't know of this issue. We must continue our efforts to make those players aware so it doesn't happen again. Most of them have good intentions, so we must give them that chance.

 

I fully understand your idea though and in theory it looks great! Unfortunately though, these are still human players, and humans have their flaws.

[=VG= 0100011000101]

This morning at work, I've tried something ...
even if that is a bit of a dickmove, I tried to crash empty (!) servers. Unfortunately, I did not find an empty deployment server that had air-assets, otherwise I would have tried it as well. When I create a local game, whether deployment or coop, the server stays alive. but in coop I could crash blackhound and jtg when I flew in the south of the map but doubles PRTA server does not crash! I flew there on operation archer 2 times in the south and only died, the server remained stable.
is this map dependent or does prta have a solution to the problem?

[=VG= Melon Muncher]

I never tested deployment. Back when I was really trying to find the cause I found most of the time It's not map specific it's just random

Here's a bit of the logging I did when testing. Clean server means I reset it fully. the only consistent thing was that the first map was less likely to crash after a hard reset.

Khamisyah first map, clean server- No crash
Khamisiyah after crash on khamisiyah - Crash
Muttrah after Khamisiyah (clean) - Crash
Muttrah first map, clean server - No crash
Muttrah after crash on muttrah - crash
Jabal after muttrah (clean) - no crash
Marlin after jabal after muttrah (clean) - no crash
Barracuda after Marlin after jabal after muttrah (clean) - crash
Jabal clean server - no crash
Jabal std after jabal alt (clean) - no crash
Barracuda after jabal std after jabal alt (clean) - no crash
Silent eagle after Barracuda after jabal std after jabal alt (clean) - no crash
Silent Eagle, clean server - no crash, then crash 2 times
Vadso after silent - Crash

 

I thought about making a combat area that insta kills when you fly too close to the south border but it's not possible to do serverside.

[=VG= Acro1]

30 minutes ago, 0100011000101 said:

This morning at work, I've tried something ...
even if that is a bit of a dickmove, I tried to crash empty (!) servers. Unfortunately, I did not find an empty deployment server that had air-assets, otherwise I would have tried it as well. When I create a local game, whether deployment or coop, the server stays alive. but in coop I could crash blackhound and jtg when I flew in the south of the map but doubles PRTA server does not crash! I flew there on operation archer 2 times in the south and only died, the server remained stable.
is this map dependent or does prta have a solution to the problem?

Funnily enough, I tried doing the same thing on our own server since it was empty this morning. Driving and flying south in Khamisiyah STD caused no crash with either helo, jet or humvee. I tried local games too and had no crash either.

[BinkleDinkle]

• Only happens on south-side of map.
• Only happens with air assets. Tried all other asset types. ( UAV didn't crash server )
• Only happens on VG server. Many of us have tried it on every other COOP and multiple Deployment servers of which none have crashed.
• Happens on all maps.
• Will also crash if Bots fly south.

 

When I was making some layers i went through ~ two weeks of demos looking for crashes looking for ways to limit the chances

• None looked to been done maliciously.
• Wide range of people have done it, with only few people causing it repeatedly.
• CAS Jets leading cause of crashing.
• Helicopter caused crashing nearly always caused by people struggling to land at bases along southern part of maps. Khamisiyah a prime example.

https://docs.google.com/spreadsheets/d/1TmvNL0Lwp2la-sb49QJOGPWXqePe2UTQUl5QMQDGLAI/edit?usp=sharing ( Not complete list as some of the demos didn't record last few seconds before crash )

[=VG= 0100011000101]

14 minutes ago, =VG= BinkleDinkle said:

Only happens on VG server.

thats not true.

1 hour ago, 0100011000101 said:

but in coop I could crash blackhound and jtg when I flew in the south of the map

but really cool that you do such work.
I also have the feeling that it is not map-dependent but that it can theoretically happen anywhere, but some maps are more vulnerable than others. can it have something to do with "air"? if other vehicles do not crash, so that they are bound to the ground that is not present when flying. maybe you should "drive" slowly with a jet from the map and not fly ^ ^
the other thought was: the server crashed with me every time i was dead ... so if you were punished because you flew into the red area. is it possible that it has something to do with it?

[Xenalite]

4 hours ago, =VG= Melon Muncher said:

I thought about making a combat area that insta kills when you fly too close to the south border but it's not possible to do serverside.

I did that in old BF2 with Python modding. It killed a player within a certain box area. Don't think the same is possible in PR.

Also, killing the player doesn't do it - it's the vehicle that crashes the server. I once realized I won't make it and will fly sotuhbound, so I quickly suicided, and the F16 on Bijar still crashed it.

[=VG= 0100011000101]

I have seen that the time until you die when you are in the red marked area, apparently can adjust ... at least I had moments where I died after 9 sec and on other servers/maps only after 15 or 30 sec ...
is not the best solution but if you keep this time very short then some players die more because they can not make it out of this zone but maybe you can not fly far enough to crash the server.

another thought I had concerns the map size. but probably would not be realized.
Often Coop uses only a part of the map. So in deployment or at inf and std, there are often extremely different sizes of the same map. Is it possible to make the maps bigger in the south and to provide them with a larger red area? So so that you can not use more map, but more map is there?

[=VG= SemlerPDX]

10 hours ago, Xenalite said:

Can we resign people from CAS if we know that they are notorious for flying south and crashing? It's probably a worse disruption to the server than people TKing in main.

People taking advantage of ANY known bug, exploit, or instability should get the same treatment as everyone:  Warned for their actions, Kicked and/or Banned for subsequent/frequent abuse or ignorance if it is consistent or intentional!

*Taking into account what Acro1 wrote in reply to your question.  We gotta be damn sure, so we don't punish people unnecessarily.

[=VG= SemlerPDX]

8 hours ago, =VG= BinkleDinkle said:

Only happens on VG server.

First, I want to say, I am not trying to call you out personally, BinkleDinkle.  No offense is intended at all, I'm sure we all respect you as a valuable VG member and PR Admin.  You are not the only person that has made this false statement so confidently in the past, not the only one this reply is directed at.

But I would like misinformation such as this to stop right here and now. This is simply not true. 
Enough is enough, and no one on the Head Administration team at VG is a liar.  When we say that we have experienced this bug on other PR COOP Server(s), it is a factual statement coming straight from an experience we had when we were actively hunting this bug and trying to get to the bottom of it all.  When you say such things, you are essentially calling myself and/or Melon a liar.  We have zero motivation to lie about this.

Your peer and fellow admin (Binary) has just found for himself that this is untrue.  When a statement like that is made, it is now essentially calling (Binary) a liar, too.  Again, enough is enough.  No longer does anyone need to blame VG, or Melon, or myself, as liars when we state with certainty that this is NOT only possible on the VG PR Server, when one of your own peers has experienced it, too.  Not sure we ever found this on Deployment, so AFAIK, this is a COOP bug, it is painfully inconsistent, and maddeningly difficult to identify triggering factors.

_

We cannot let our opinions about each other enter into these sorts of matters, we can't be treating each other this way and disbelieving each other, and acting as if there is a division between us who Administrate this community and everyone else, between those who regularly play on the server, and those of us who spend more time managing the servers and/or playing other games online, as if we are allowing this bug to perpetuate - we all have the same goal to run clean, stable, fun gaming servers at VG.  Just because we aren't there daily doesn't mean we don't feel your pain.  For many of us at VG, PR was/is our first love.  If you feel people like myself and Melon have "given up", I'd like to remind you that weeks were spent on this with NO leads that paid out, and no serious progress whatsoever.  If anything was gained, Melon found and/or ruled out a many number of reasons THAT DO NOT contribute to this issue.   But that is truly the extent of it all.  When you try something over and over for weeks straight with no results, no new ideas, and no new leads, you have to put it on the back burner or you'll go mad.

 

11 hours ago, Acro1 said:

Is there a way to publicize the respective files so we can have a look at them?

To anyone with programming knowledge and experience, and a drive to eradicate this bug once and for all, we'd encourage you to apply to join the PR Dev team so that you would have access to files that VG is not in a position to share with the public, or even our PR Admins.  Anyone wishing to hunt this bug needs to be armed with more than just our config files, they need full SA Forums & R-Dev Forums access, and ability to access compiled python files, something that is only available to PR Contributors and PR Developers.

Thank you all for understanding, and for your help!

[=VG= 0100011000101]

I do not want to get involved in this debate, but the error is provable on JTG. practically every time I fly out of the map the server crashes. ssg and prta, on the other hand, seem to be "uncrashable". at SSG I've tried it 10 times and not even the server crashed. with JTG on the other hand always. could it be that this error has something to do with the bot / kit configuration?

the nice thing about JTG is that there is mapvoting on and if you're alone on the server you win the voting right away. So if an inf map is loaded just vote for kashan and 5 sec later it loads the map, then you fly to the south, the server crashed,  loads new and you make a mapvote on kashan ^^

 

JTG:

Maps: Kashan std Crashable.

• Bots only on enemy side
• Kit-restriction
• you can be SL
• no teamswitching
• map voting on

________________________________________________

VG:

Maps: Nearly all Crashable.

• Bots only on enemy side
• Kit-restriction
• you can be SL
• no teamswitching
• map voting off

________________________________________________

OWLS:

Maps: ??? Uncrashable.

• Bots only on enemy side
• no kit-restriction
• you can be SL
• teamswitching is on
• map voting off

________________________________________________

SSG:

Maps: Shahahahaha Uncrashable.

• Bots on both sides
• kit-restriction
• you can not be SL (in a bot squad)
• teamswitching is on
• map voting off

________________________________________________

Blackhound:

Maps: Kashan std Uncrashable (now...pretty shure that i crashed Blackhound before)

• Bots only on enemy side
• no kit-restriction
• you can be SL
• teamswitching is on
• map voting on

________________________________________________

PRTA:

Not empty now.

 

the enumerations above are for me and will be expanded over time. but why should I deny them to others ...
Feel free to add other points, maps and ideas to these lists.

5 hours ago, =VG= SemlerPDX said:

To anyone with programming knowledge and experience, and a drive to eradicate this bug once and for all, we'd encourage you to apply to join the PR Dev team so that you would have access to files that VG is not in a position to share with the public, or even our PR Admins. Thank you for understanding, and for your help!

I just want to narrow down/find this error, but I did not want to be dev. Especially since I have hardly phyton-experiences and I have not otherwise very much concerned with the structure of pr.
but often you do not need any great programming skills to find errors. you just have to understand what happens where ...

Of course it is understandable that such server / config files are not shared with everyone. but if there is a way to give one or the other such an insight, that would certainly be helpful; p

[Xenalite]

Can you crash the server if you are the only one on it? If the bug was related the number of players online, then I think this myth of VG being the only COOP server that crashes is because we always have the most people online.

[=VG= ciro (Inactive Duty)]

10 hours ago, =VG= BinkleDinkle said:

• Only happens on VG server. Many of us have tried it on every other COOP and multiple Deployment servers of which none have crashed.

This  is good for me to know, may I have the details of the other servers that don't crash that have a mission that VG run that does crash?

I'm re-installing PR now to fully understand whats going on...

In July (sem) was the installation fresh or did we copy files from an old instance?

I would like to re-install as a new instance and test that with 'stock' missions and compare crash logs....

I link this from PR forums, if you could PM me these files.....

https://www.realitymod.com/forum/showthread.php?t=147961

In order for us to be able to debug certain crashes, we need you to use a program called ProcDump. With that you can generate crash dumps that help us to figure out what is going wrong.

Download: https://docs.microsoft.com/en-us/sys...loads/procdump

1. Start PR
2. Open command line as administrator
3. Navigate to the location you put procdump (Example: "cd C:\Users\User\Downloads\Procdump")
4. Run procdump through command line like this:
   Code:

   
   procdump.exe -ma -b -e 1 -g prbf2.exe

Alternative:
Create a shortcut to procdump.exe and add the arguments to the target path

 

Code:

 -ma -b -e 1 -g prbf2.exe
Example:
C:\Users\User\Downloads\Procdump\procdump.exe -ma -b -e 1 -g ts3client_win64.exe

Run the shortcut as administrator after launching PR.

If you are crashing at PR launch please see here:

https://www.realitymod.com/forum/showthread.php?t=134952

or

https://www.realitymod.com/forum/forumdisplay.php?f=360

***do not post error logs here***

Only PM me the error log if after successful connection then crash.

 

 

 

 

Quote

 

 

 

[=VG= 0100011000101]

3 minutes ago, =VG= ciro said:

Only happens on VG server. Many of us have tried it on every other COOP and multiple Deployment servers of which none have crashed.

thats really not the case. you can go to JTG now, run kashan and try it... I tried it several times and every time the server crashed.

and of course I was alone on the server.

[=VG= ciro (Inactive Duty)]

https://gaming.stackexchange.com/questions/358024/project-reality-bf2-flying-south-out-of-bounds-crashes-the-coop-server-whats

 

Nice one fella!

[=VG= SemlerPDX]

@=VG= ciro make an account at the PR forums, then send me a friend request.  I'll make you a server moderator and you'll have access to files as they distribute them

But you can find a zip of the PR1.5 files that is not altered, somewhere on the server. (check documents/downloads folders of all user accounts like mine, my old one, etc.)

ALWAYS MAKE BACKUPS FIRST! Move any current folders somewhere or copy them, so we have references and backups to any file or system in our PR Server folders. I'm sure you'll do this, just had to say it so I can rest easy.

 

[=VG= ciro (Inactive Duty)]

cool sem, had an account there for a while but forgotten pW... waiting reply.

will look on the server.

@=VG= Fastjack ... this may be a strange question but you have created mods for this engine you may help us here:

From a map developmet prospective is there a null location i.e. an out of bounds for want of a better expression? are maps restricted to a specified size per the Refractor engine capabilities? if so are the maps that crash at the limit of those capabilities?, ...

Im struggling to find any documentation of the refractor engine but see a heap of phisics engine errors.

 

[=VG= Acro1]

12 hours ago, =VG= ciro said:

https://gaming.stackexchange.com/questions/358024/project-reality-bf2-flying-south-out-of-bounds-crashes-the-coop-server-whats

 

Nice one fella!

Thanks Though I don't think many answers will come, I'm keeping tabs on that page. Never know if something good comes out of it!

 

@=VG= SemlerPDX Does Realitymod provide you with the necessary files once you become a contributor? I'm submitting an R-CON application somewhere today.

[Rabbit]

Does it crash if you fly into the SE and SW oob terrain squares? Or just the south one.

[=VG= Fastjack]

18 hours ago, =VG= ciro said:

cool sem, had an account there for a while but forgotten pW... waiting reply.

will look on the server.

@=VG= Fastjack ... this may be a strange question but you have created mods for this engine you may help us here:

From a map developmet prospective is there a null location i.e. an out of bounds for want of a better expression? are maps restricted to a specified size per the Refractor engine capabilities? if so are the maps that crash at the limit of those capabilities?, ...

Im struggling to find any documentation of the refractor engine but see a heap of phisics engine errors.

 

That's a difficult question Ciro …..

really difficult and i dont know if the error is based on Refractor 2 codding or Mr. R-DEV ihaveallhandsfull-so-notimeforcoopgameplayupgrades Alon Tavor's pythoncodding or what they changed via Cheat-Engine Essembling in the Binaries to get that done. 

Another Problem is, it is not clear to me if this error occurs ONLY on our server or others server too like SSG or PRTA or Blackhoundz

R-DEV Mineral said it happend only by us but i heared from other people that it happens also on some other coop servers and other people saying it never happend on their coop server.

Semler said we had back january/february a fresh install and it didn't fixed it.

The question is what got modified at all like kit restrictions etc.. I dont know if there exist such an option to fuck around with OoB settings like North OoB or South OoB.

A complete list of what got changed. A typo (also empty case) can cause Errors if Melon modified something serverside but i dont believe that kitrestriction setup causing OoB server crashs.

Other thing we could do with our Server is to switch from coop to deployment and than one from us fly out of the south edge border and we will see what happens. If we also get a crash something is wrong on our side.

I running also out of ideas ….

 

EDIT: 

Quote

From a map developmet prospective is there a null location i.e. an out of bounds for want of a better expression? are maps restricted to a specified size

Maps with Helos are atleast  2 km (2048) and maps with jets  are 4km (4096) when you meant this.

I also saw Binaries Server crash test post to late. Good Job on this Binary, really good Job.

Server with Kit restrictions are affected. Humans have restrictions but not the Bot Team but what have kits todo with OoB crashs caused by pilotkits or their restrictions. We should really look for an invisible/visible typo.