Video of possible EKF fly away

[Pedals2Paddles]

This video is from a user on the Facebook group. He was cruising around the neighborhood when it went bananas. It took off at high speed in descent. Narrowly missed one house, and crashed through the window of another house. I don't see any way this was pilot error. it is quite obviously not just drifting with the wind. It clearly is clearly under power, pitched for maximum speed, probably 22mph. Unless the operator is lying and he intentionally kamikazed it in, this is definitely a fly away of the worst kind. Thankfully nobody was standing there.

It appears to be what several others have experienced after upgrading to 2.4. A Compass/GPS/IMU disagreement that results in it taking off and unable to respond to control input. From what others experienced, it will continue in whatever direction, sometimes climbing, sometimes descending, sometimes level, until the batteries failsafe kicks in. That seems to make it snap out of it, if you're lucky enough to not crash first.

I can count the number of people this has happened to on here and facebook one hand, so it's not a massive widespread issue. But that count is way more than it ever was before with any arducopter based aircraft. Others have suggested doing a compass and level calibration immediately after upgrading to avoid this. Since there are no release notes or instructions for 2.4, nobody would know to do this.

Skip to 1:00 to start just before it loses its mind. This is significant because I don't believe anyone has ever recovered the solo and had usable video. I believe it's also a first for Solo going through a window on video.

 

[Rusty Jones]

So did they send the logs in? Or look at them ? or post on FB?

 

[Pedals2Paddles]

No logs have been provided by the operator yet. The owner of the house wasn't home, so he couldn't even get it back until they got home later at night or today (not sure the timing). Many have asked him to download and provide the logs when he can. I don't know if he's even familiar with the process. The poor guy is a little preoccupied lately I'm sure...

 

[Adimuzio513]

Wow that's scary, that definitley would have messed somebody up pretty bad not to mention if it was a child. Hope they get on this fast, with these recent updates I've had a couple odd things happen as well. Nothing so serious, but it has made me less confident. When in manual mine acquired a gps lock in mid air and died for a split second, it saved itself right away thankfully. That second was scary though went from buzzzzz to silence and back in about 2 seconds. Really hope to be as confident as I was with my first solo again soon, I'm nervous to take it out at the moment. Even if their few and far between the conciquences are enough to make me question flying at all until these bugs are definitley fixed.

 

[Pedals2Paddles]

Updates. First, thankfully nobody was hurt. And ironically, the neighbor who's house the Solo crashed into with the precision of a tomahawk cruise missile is a fellow solo owner. So it sounds like there are no neighborly hard feelings.

Now, log files! The owner was kind enough to provide the telemetry logs for us to look over. It shows it was indeed a disagreement between the GPS, compass, and IMUs. The solo was having all kinds of problems on the ground with AHRS and motion errors. The solo app prompted for a level calibration, which it appears he did. The solo settled down, aquired a GPS lock, and was able to arm and take off.

As he was making that wide turn, the EKF indicator turned yellow a few times indicating something was up, but not enough to trigger an error or cause a loss of control apparently. That will not be indicated in the Solo app either. The GPS appears to be locked and compass tracking just fine.

Then as he came out of the turn, and happened to be pointing directly at the neighbor's window, it lost it's ability to determine it's orientation and position. The EKF error was for horizontal position variance. So the IMU, compass, and GPS data are conflicting and it doesn't know wtf to make of it.

It continued straight ahead with operator control lost. It was at such a steep pitch angle, that it went through the window at 45mph. That pitch angle was clearly well beyond the maximum angle parameter, which is why it was descending and going so fast.

The dataflash logs will tell us more detail. Based just on what I am seeing in this t-log, I believe we're looking at an IMU hardware failure. The IMUs were screwed up on the ground long before GPS and compass even come into play. And to allow such a steep pitch angle, it would have to be totally out of whack.

 

[SurfnSkate81]

Updates. First, thankfully nobody was hurt. And ironically, the neighbor who's house the Solo crashed into with the precision of a tomahawk cruise missile is a fellow solo owner. So it sounds like there are no neighborly hard feelings.

Now, log files! The owner was kind enough to provide the telemetry logs for us to look over. It shows it was indeed a disagreement between the GPS, compass, and IMUs. The solo was having all kinds of problems on the ground with AHRS and motion errors. The solo app prompted for a level calibration, which it appears he did. The solo settled down, aquired a GPS lock, and was able to arm and take off.

As he was making that wide turn, the EKF indicator turned yellow a few times indicating something was up, but not enough to trigger an error or cause a loss of control apparently. That will not be indicated in the Solo app either. The GPS appears to be locked and compass tracking just fine.

Then as he came out of the turn, and happened to be pointing directly at the neighbor's window, it lost it's ability to determine it's orientation and position. The EKF error was for horizontal position variance. So the IMU, compass, and GPS data are conflicting and it doesn't know wtf to make of it.

It continued straight ahead with operator control lost. It was at such a steep pitch angle, that it went through the window at 45mph. That pitch angle was clearly well beyond the maximum angle parameter, which is why it was descending and going so fast.

The dataflash logs will tell us more detail. Based just on what I am seeing in this t-log, I believe we're looking at an IMU hardware failure. The IMUs were screwed up on the ground long before GPS and compass even come into play. And to allow such a steep pitch angle, it would have to be totally out of whack.

Great analysis. Scary video.


Sent from my iPhone using Tapatalk

 

[User Name]

And ironically, the neighbor who's house the Solo crashed into with the precision of a tomahawk cruise missile is a fellow solo owner.

Holy heck, now what are the odds of that??!!??!? Wonder if the guy came home and said "what the bloody hell is my Solo doing on the floor? Hey, wait a minute..."

As for the crash, a few questions:

Do you know what firmware was on his solo? There's been the odd report of issues with every version, but 2.4 seems to have more issues than normal.

Would the issues pre-takeoff have generated any alerts or warnings in the app? Would the operator have knows that trouble was brewing?

If not, would it be correct to assume looking at logs from precious flights might have alerted him to problems?

It makes me want to pull some logs from mine to evaluate it's overall health.

Would switching to manual have enabled him to regain control, or was the discrepancy more than between the GPS and the other sensors?

 

[pyrate]

did he ever confirm what firmware he was on.
if he had the first 2.4 beta and did not recalibrate it could cause that

 

[Pedals2Paddles]

The solo app doesn't really give any of these details to the operator. Those details only show up in Mission Planner. On the ground, it would just say calibration required if anything on Solo App. I presume it did, and he did a calibration.

He is on 2.4.

 

[prowler9]

Pics of the damage done to solo it's self?

Sent from my SM-T337V using Tapatalk

 

[Don H.]

Wow, total Kamikazee mission.

I'm thinking the level and mag cal would not help. The reason why is because my first one had a flyaway after I did a factory reset per 3dr. I did the level and mag cal and the first flight after I was cruising it around and paused it. I hit RTL and what I thought was rewind was a fly away. It picked up speed and dove into the ground much like in the vid. I had no control on it. Pause didn't help, manual didn't help.
3DR reviewed the logs and sent me a replacement.

 

[fpvsteve]

Sorry guys. Not a flyaway.
I looked at the logs.
Root cause was that Solo was constantly moved around over several minutes during prearms, causing bad AHRS and position variance errors, then level cal'ed, further moved around then put finally really close to a building wall before acquisition of GPS fix and takeoff - likely adding multipath interferences to the equation.

So not a surprise that the EKF finally threw an exception and put Solo into AltHold. Solo went incommunicado for a few seconds- long enough to crash.
No hardware failure, not even software, simply user error.

Flyaways have happened with Solo, but this one wasn't one I am afraid.

 

[Pedals2Paddles]

I completely disagree with everything above. It makes absolutely no sense. You're citing evidence that he was too close to the building based on a google map that makes it look like he armed it inside the house. The map is obviously wrong. You're claim the GPS was wrong, but using the GPS position as evidence of your point... so is it right or wrong? Moving the solo is not prohibited and nothing says anything about not moving it around. The guy did absolutely nothing wrong. Followed all the instructions. Did what the app told him to do. Calibrated it. Had a good GPS lock. And took off from an area where he has flown many many times. Sorry, your conclusions just make no sense.

@fpvsteve, I have a lot of respect for you and the work you do for the Solo / Arducopter community. I recognize that in this debate (here and the FB group). We both have rather strong feelings about this incident, so I hope we can debate it and keep it non-personal.

 

[Pedals2Paddles]

After a lot of discussion in the FB group, some theories have come up. Apparently moving the aircraft around once it is powered on can on occasion mess with the pre-arm calibration and go unnoticed by the pixhawk. If that happens, it will cause an increasing compass variance during flight, and eventually lose it's mind completely. This could be an example of that issue. If that is what happened here, it still does not qualify as operator error. This is not documented in any instructions online or on the app for Solo or arducopter in general. It's known by some of the tech nerds that are really in the weeds, and that's about it. From the operator's perspective, it asked for a calibration, he did the calibration, got a GPS lock, and it said GO FLY.

At this point, i think this theory is more likely than an IMU hardware failure, which I originally suspected. But in either case, still not an error on the operator's part. Again, all theories at this point.

 

[RichWest]

From the user manual...

2.3.2
Powering
To power Solo, insert the Smart Battery into Solo’s battery bay and slide the battery forward until it clicks into place. To turn on Solo, press and hold the battery power button. When Solo powers on, the battery displays an LED animation and you hear the startup tone. Power Solo only with the designated 3DR Solo Smart Battery; using a different battery can permanently damage Solo.

Before powering on, make sure Solo is level and keep Solo still during startup and while the sensors initialize. Moving Solo during this process causes the sensors to calibrate incorrectly and can create a preflight error or affect in-flight performance.

 

[Jubalr]

Yes, this was an old discussion last year. There is a post by one of the developers last year that moving Solo after it has powered on can cause problems later in the flight. Makes sense to me and something I have always done anyway with my home builds. Just from a common sense perspective on a set of electronics that is trying to initialize several components and compass. The argument long ago was that some believed it shouldn't matter. My belief is 'shouldn't' is a very telling word and my response was always why not just play it safe and just not move it during power up? There were a few crashes last year that were similar in that they were moved after power on and before arming. May not always be a problem, but sometimes shi*t happens.

 

[Pedals2Paddles]

@RichWest, that instruction is only during startup while it is initializing. It does not say what that duration is though, so the consumer user is left to assume that. What we're talking about is moving it around after startup. I do not believe the operator here was moving it around during startup. He did move it around after startup, but then also did a level calibration. Calibration was successful, and he moved it around after doing a level calibration. None of which is recommended against in the documentation.

I think it's also worth noting that many things that would be "common sense" to us that have been building and flying these things for years is a foreign concept to a new consumer user. Because a bunch of us nerds know these things doesn't mean squat to a new consumer user. They can only know what the instructions and application tell them.

 

[SARDG]

..Because a bunch of us nerds know these things doesn't mean squat to a new consumer user. They can only know what the instructions and application tell them.[/USER]

...and generally not even that much, because I am fairly certain many don't RTFM.

 

[PdxSteve]

...and generally not even that much, because I am fairly certain many don't RTFM.

Yes, or this forum would probably only contain 1/5th the posts as it does lol.

 

[RichWest]

@RichWest, that instruction is only during startup while it is initializing. It does not say what that duration is though, so the consumer user is left to assume that. What we're talking about is moving it around after startup

I got no dog in this hunt and really don't care either way, was just trying to clarify that the manual does state not to move during start up. I'm just the messenger....

As to how long a duration to not move, I believe the user should assume to the point that the bird is fully flight ready and commanded with "Ready to FLY". Based on the manual's lack of saying otherwise. You all are splitting hairs with this one, just too many variables to say one way or another. Give the guy a 50% discount and be done.

To add: I don't think I could have hit that window square-on like that bird did, maybe with a baseball. That was an epic crash. Peace.