EKF Variance--Need Help Understanding (Log Included)

Joined
Nov 15, 2015
Messages
147
Reaction score
18
Age
49
I think we need @P2P for this one...

I was flying a very reliable Solo the other day using Tower on a mission, battery is relatively new to service with only 10-15 flights.

Shortly after reaching 50m and heading to the next waypoint at 10m/s, Tower said "battery at -1%".

And something just seemed wrong...

So I hit the PAUSE button on the controller.

Immediately the Solo shot back towards me, holding altitude. According to Tower, it was in Altitude Hold mode (not Loiter, not RTH). I believe the controller displayed MANUAL.

I was fighting the controls, and trying to put it into FLY mode.

I still had a good GPS sat count and HDOP.

Then, Tower barked out "EKF VARIANCE".

I was fighting it and eventually got it into FLY mode.

Brought it down.

Accidentally hit RTH and it started to rise--so I hit PAUSE then FLY.

Landed safely. Stress!!!

But now I am wondering just what the heck happened.

Can someone check the log and see what they can find?

Here is a link on Dropbox (too large to attach here)

PS: Not sure this means anything, but earlier in the day I was flying a different quad (APM/Pixhawk) using USB telemetry rather than UDP. That quad for some reason won't display battery percentage, but the Solo does (and Solo did properly display batt percent when it took off).
 
  • Like
Reactions: tinker bot
Joined
Oct 17, 2016
Messages
934
Reaction score
268
Age
54
Location
Naples, FL
This is a log that needs P2P's wisdom. Here is what I see and it is very strange. Shortly after takeoff your voltage is logged dropping from 16.5 volts to 15.8 and then it flat lines the rest of the flight at 15.8. Amps also flat lines at about 20 the rest of your flight. At the same time that's happening the RCOUT of motors 1 - 4 frequently are recorded at bottoming out at 1117. Yet the copter flew and did not crash. I have provided some screenshots. Strange indeed.
 

Attachments

  • Like
Reactions: 5r49cfu
Joined
Mar 12, 2016
Messages
3,700
Reaction score
2,477
What the (@&#$. That's way out of whack.

I'll look at the logs later this evening.
 
Joined
Oct 17, 2016
Messages
934
Reaction score
268
Age
54
Location
Naples, FL
No kidding!! No expert at reading logs here but something is screwed up. Flying orientation looked good but power related stuff.... not so much.
 
Joined
Nov 15, 2015
Messages
147
Reaction score
18
Age
49
No kidding!! No expert at reading logs here but something is screwed up. Flying orientation looked good but power related stuff.... not so much.
Yeah, it went haywire once I hit PAUSE on the controller.

I can’t believe I was able to keep it from flying off—I had to peg the controls in the opposite direction it was pushing to.

Note that I didn’t have an issue holding altitude, thankfully.
 
Joined
Mar 12, 2016
Messages
3,700
Reaction score
2,477
FYI when the ekf fails, the copter will no longer be able to use the GPS. If you got pause when GPS is not available, it will mode change to manual instead of "braking".
 
Joined
Nov 15, 2015
Messages
147
Reaction score
18
Age
49
FYI when the ekf fails, the copter will no longer be able to use the GPS. If you got pause when GPS is not available, it will mode change to manual instead of "braking".
The Solo was in AUTO mode when I hit PAUSE (BRAKE in the log).

I didn’t get the “EKF Variance” notice on Tower until at least a few seconds later, maybe more—after it was already shooting off back towards me barely controllable.

Note the log shows high satellite count and good HDOP the entire flight.
 
Joined
Mar 12, 2016
Messages
3,700
Reaction score
2,477
I've looked at this log. It's really strange. I think there's a few things going on here.

First the obvious. The battery data stopped flowing. That could be because the battery malfunctioned, or it could be because the battery monitor cable came unplugged from the board. I've never seen it happen. But I just did a test one of my solos that's opened up. If I unplugged the little white & green battery data wire while it's already powered up, the values remain at their last. There is no error message and no other indication that something is wrong. I confirmed this in the code too. The battery monitor reports itself as unhealthy. But ArduCopter doesn't actually take any action on that.

As to the EKF and control issues, It's hard to say. The vibrations went up significantly, although not over the limits. And maybe 10 seconds later the EKF lost it, which excessive vibration can certainly cause. I suppose vibration could also cause the battery data cable to come unplugged if it was already loose. This is the best I can come up with.

Suggestions:
  • First see if the battery data works the next time you power it up. Preferably trying more than one battery. If the battery data is still out, you'll need to open it up and see if the data cable is unplugged.
  • Redo the level calibration and compass calibration before flying again
  • Test fly and post a log for comparison.
 
Joined
Nov 15, 2015
Messages
147
Reaction score
18
Age
49
Yes, the reason I hit PAUSE was because Tower said “battery -1%” just after it got to the first waypoint.

It just didn’t seem like it was flying right.

NOTE I replaced all four motors/props a week or so ago, with at least 5-10 successful flights since (including just before this flight with a different battery).
I've looked at this log. It's really strange. I think there's a few things going on here.
...
P2P: Here is a screen recording from Tower--this should help a bit.

Note how the position on screen stops updating while it was out of control.
Note how other telemetry data does keep updating (pitch, roll, yaw, speed, altitude).
Note that it is in ALT HOLD position most of the time, I was hitting FLY but didn't go.
Note when EKF Variance is announced--long after BRAKE

Link
 
Joined
Mar 12, 2016
Messages
3,700
Reaction score
2,477
When the ekf is bad, it will not have any position data to report. And as such, fly mode is not possible, and neither is pause/brake. That's why the log has flight mode errors every time you hit fly or pause.
 
  • Like
Reactions: 5r49cfu
Joined
Nov 15, 2015
Messages
147
Reaction score
18
Age
49
When the ekf is bad, it will not have any position data to report. And as such, fly mode is not possible, and neither is pause/brake. That's why the log has flight mode errors every time you hit fly or pause.
yup, that makes sense.

I've looked at this log. It's really strange. I think there's a few things going on here.

Suggestions:
  • First see if the battery data works the next time you power it up. Preferably trying more than one battery. If the battery data is still out, you'll need to open it up and see if the data cable is unplugged.
  • Redo the level calibration and compass calibration before flying again
  • Test fly and post a log for comparison.
  • Powered it up last night, batt percent showed in both 3DR Solo App and Tower
  • Opened up the tray, the battery data cable is still connected
I'll fly it today and post the log.

Thanks for the help P2P!
 
Joined
Oct 17, 2016
Messages
934
Reaction score
268
Age
54
Location
Naples, FL
P2P I realize you are a busy guy but I have a question. I'm a novice at reading logs. After four flights or so in a day I take one log and look it over. Roll, Pitch, Yaw, Alt, all vs. desired. Battery Volts and Amp draw. Vibrations, RCOUT, Sat count, and HDOP. All just to make sure nothing has changed from my previous flight day. A few other things also. How do you tell when the EKF explodes?
 
Joined
Nov 15, 2015
Messages
147
Reaction score
18
Age
49
Ok, flew this Solo again yesterday.

  1. Batt percent was displayed in both Tower and iOS 3DR App.
  2. No EKF Variance problems-flew perfectly.
  3. Strangely, video stopped working after first flight.
    1. GPH4B v5.00 (been using this with Solo for months without issue)
    2. During first flight on screen video was blinking in and out
    3. Tried two different GoPros, same issue.
    4. Video feed worked until I hit the record button. But after a few flight no video at all.
I think the gimbal is toast.

I wonder if the gimbal could have been messing up other things in the system causing EKF Variance?
 
Joined
Oct 17, 2016
Messages
934
Reaction score
268
Age
54
Location
Naples, FL
Ok, flew this Solo again yesterday.

  1. Batt percent was displayed in both Tower and iOS 3DR App.
  2. No EKF Variance problems-flew perfectly.
  3. Strangely, video stopped working after first flight.
    1. GPH4B v5.00 (been using this with Solo for months without issue)
    2. During first flight on screen video was blinking in and out
    3. Tried two different GoPros, same issue.
    4. Video feed worked until I hit the record button. But after a few flight no video at all.
I think the gimbal is toast.

I wonder if the gimbal could have been messing up other things in the system causing EKF Variance?
Did you try restarting your tablet or whatever smart device you were using?
 
Joined
Mar 12, 2016
Messages
3,700
Reaction score
2,477
The gimbal wouldn't have caused the other issues. Video issues like that are usually a bad HDMI connection on the board. Sometimes it's a codec issue and pulling the battery from the controller helps.
 
Joined
Nov 15, 2015
Messages
147
Reaction score
18
Age
49
Did you try restarting your tablet or whatever smart device you were using?
Yes, I restarted Android/Tower, the Solo, Pulled GoPro battery, Replaced the GoPro with another.

Today at home I am testing this Solo and video feed works sometimes, other times not. Sometimes it blanks out after starting to record, other times it works during record. Tried both iOS app and Android/Tower.

The gimbal wouldn't have caused the other issues. Video issues like that are usually a bad HDMI connection on the board. Sometimes it's a codec issue and pulling the battery from the controller helps.
I'll open it up soon, if I can't find anything, I'll swap with a new Gimbal...
 

New Threads

Members online

Forum statistics

Threads
12,095
Messages
141,202
Members
14,016
Latest member
wabknobby