Open Solo 2.5-RC2 is out

[Pedals2Paddles]

Matt,
Regarding slow loops, I have AC 3.5.3 and OpenSolo running on my PixHawk 2.0 with 5V mod. I also have run (and flown) it with the exact same OpenSolo but with AC 1.5.3. I see NLon of under 150 with 1.5.3 during a 15 minute flight. With 3.5.3 I see 450-550 just sitting on the ground with sats acquired and no motors running. I have repeated this on 2 non-flights with 3.5.3 and it is consistently between 450-550.
I don't think the slow loop issue is with the Green Cube hardware.
If it were not snowing I would try flying 3.5.3 and dumping the logs. Maybe tomorrow...;>))

Where are you that it's snowing already!?

My concern is that something about the pixhawk communicating with the IMX is bogging down ArduCopter. If you were seeing it in the old FW too, but less, I think it stands to reason that the new FW is doing a lot more so there is a lot more potential to slow down. Or is this just normal and it does it on any vehicle?

 

[mike_kelly]

Matt, I need to compile Arducopter myself for Solo. I assume it is a V3 platform. Is that correct or V4?

I just found your post on Ardupilot.org

"3.5-RC2 (-v3.px4 version) is now installed on my experimental solo."

So I think I am correct to use V3.

Cheers

PS. I wish you would let me buy you a beer for a token appreciation of all the work you have done to make the Solo viable again.

 

[cynfab]

I'm in western Colorado at about 6200'. It's pretty early for snow here too.
I would assume with all the improvements to AC3.5+ that the code is doing more now than in 1.5.3,
The Ardupilot flight log analyser tool defines a loop error as when NLon is greater than 6% of NLoop. So if NLoop is
usually about 4000 then any time NLon is greater than 240 there may be an issue.
Reviewing DaveB's 3.5.3 flight log I see his NLoop never gets over 4006, but his NLon is 1250 or so while not flying
and over 2000 while the motors are turning, except at the beginning where it is about 500.

 

[cynfab]

Logs:
13.bin is OpenSolo-RC2 1.5.3 on PixHawk 2.0 with 5V mod
14.bin is first run of same hardware but with AC 3.5.3, spun motors up several times, didn't realize that there was a new parameter, DISARM_DELAY, set to 8 in the SoloParameters is a little short.
15.bin is second run where on the second motor start I actually took off for about 10 seconds. At that time I was paranoid about DaveB's shutdown, so I force landed it asap, then spun the motots up a few more times.

loop_logs.tar.gz

 

[David Boulanger]

My two motor shut downs happened about a second after Solo left the ground. Maybe less. It has not happened again since.

 

[cynfab]

This is pure speculation But...
Here is DaveB's crash2 on top and my non-flight test on the bottom where I discovered tthe DISARM_DELAY parameter of 8 seconds was shutting down my motors after I hit the fly button but didn't hit it a second time to fly. The time scales are the same as are the parameters graphed. Notice that the events occur at the same time and that DaveB's log ends when my disarm event happens.
I think that the Pixhawk crashed at that point in DaveB's flight and ended the logs. It may have to do with a poor "wipe" or somesuch. My guess only.

 

[XevetS]

This is pure speculation But...
Here is DaveB's crash2 on top and my non-flight test on the bottom where I discovered tthe DISARM_DELAY parameter of 8 seconds was shutting down my motors after I hit the fly button but didn't hit it a second time to fly. The time scales are the same as are the parameters graphed. Notice that the events occur at the same time and that DaveB's log ends when my disarm event happens.
I think that the Pixhawk crashed at that point in DaveB's flight and ended the logs. It may have to do with a poor "wipe" or somesuch. My guess only.

View attachment 7065

That is one of the parameters I had to change on my custom build stock solo cube running Master "arducopter solo_beta 1.2.x" because the motors were shutting down to quickly, if I would arm using the fly button before I had the chance to hit fly again they would shutdown so I increased the time threshold..

I never noticed or even saw the PM=slow loop issues at all?

Edit: I just checked an old log they are there

Here is one auto analysis

Log File tmpA0FA.tmp.log
Size (kb) 21251.77734375
No of lines 249140
Duration 0:11:55
Vehicletype ArduCopter
Firmware Version Solo
Firmware Hash
Hardware Type
Free Mem 0
Skipped Lines 0
Test: Autotune = UNKNOWN - No ATUN log data
Test: Brownout = GOOD -
Test: Compass = WARN - Moderate change in mag_field (31.28%)

Test: Dupe Log Data = GOOD -
Test: Empty = GOOD -
Test: Event/Failsafe = GOOD -
Test: GPS = GOOD -
Test: IMU Mismatch = GOOD - (Mismatch: 0.60, WARN: 0.75, FAIL: 1.50)
Test: Motor Balance = GOOD - Motor channel averages = [1662, 1661, 1687, 1644]
Average motor output = 1663
Difference between min and max motor averages = 43
Test: NaNs = GOOD -
Test: OpticalFlow = FAIL - FAIL: no optical flow data

Test: Parameters = FAIL - 'THR_MIN' not found
Test: PM = FAIL - 64 slow loop lines found, max 19.77% on line 64050
Test: Pitch/Roll = UNKNOWN - Unknown mode in TestPitchRollCoupling: BRAKE
Test: Thrust = GOOD -
Test: VCC = GOOD -

 

[cynfab]

And here is where I took off and flew for 10 seconds compared to DaveB's crash2. There should have been a "NOT LANDED" event at 122.7 seconds into his log. there is in mine. Without that event, maybe the disarm timer went off, and Bad Things(tm) happened.

 

[XevetS]

This is from a post by Randy MacKay at DIYDRONES

"The PM failure messages is most likely a red herring. The PM messages display the slow loops after arming because arming ties up the CPU for a couple of seconds.

Hopefully we will fix this false-positive for AC3.3."

Maybe its not fixed yet?

What about the PWM values for the open solo controller buttons? These now serve more functions Are these possibly causing an issue? Maybe the scale is outta wack?

 

[cynfab]

The snow stopped and the sun is coming out, so I flew my PixHawk 2.0 5V mod OpenSolo 2.5-RC2 Solo for a bit less than 10 minutes. Light breeze. No problems, flew well, RTH'd a few times, several takeoffs & landings.
Here is the log.

17.BIN.gz

 

[David Boulanger]

O.K. smart guys let me run this by you regarding my minor crash 1 and 2. What if this is what happened. I push and hold FLY to arm the Solo. After arming I take a glance at Solex on my tablet to check some stuff. I then hold FLY again to take off. Is it possible with the slow looping that the Solo did not realize it was off the ground and disarmed after the 8 seconds. Between the logs ending, props stopping and Gopro stopping recording it seems like a disarming to me.

 

[David Boulanger]

This is from a post by Randy MacKay at DIYDRONES

"The PM failure messages is most likely a red herring. The PM messages display the slow loops after arming because arming ties up the CPU for a couple of seconds.

Hopefully we will fix this false-positive for AC3.3."

Maybe its not fixed yet?

What about the PWM values for the open solo controller buttons? These now serve more functions Are these possibly causing an issue? Maybe the scale is outta wack?

I have a stock Solo running open solo with 1.5.3 and no looping fail on it.

 

[XevetS]

I have a stock Solo running open solo with 1.5.3 and no looping fail on it.

Yes I have 2 solo's running open solo also no looping fails, I'm quite sure its something in the master code causing this... The only thing I'm curious about is someone complained of the controller switching to a different mode or loosing connection by a button press this seems like a PWM issue but it could be anything I'm just throwing stuff out there. The reason being is I just ordered a green cube, Maybe I'm just being paranoid ah heck I know full well I'm going to chuck that cube into the solo the second I have it in my hands and follow Matts instructions to a tee and be up and flying....I have total faith......Gulp...

 

[carpy]

This is from a post by Randy MacKay at DIYDRONES

"The PM failure messages is most likely a red herring. The PM messages display the slow loops after arming because arming ties up the CPU for a couple of seconds.

Hopefully we will fix this false-positive for AC3.3."

Maybe its not fixed yet?

What about the PWM values for the open solo controller buttons? These now serve more functions Are these possibly causing an issue? Maybe the scale is outta wack?

It looks to me like the code hasn't changed in a while (well the code for the loop stuff hasn't changed):
History for Tools/LogAnalyzer/tests/TestPerformance.py - ArduPilot/ardupilot · GitHub

 

[XevetS]

It looks to me like the code hasn't changed in a while (well the code for the loop stuff hasn't changed):
History for Tools/LogAnalyzer/tests/TestPerformance.py - ArduPilot/ardupilot · GitHub

Hummm?

 

[David Boulanger]

Yes I have 2 solo's running open solo also no looping fails, I'm quite sure its something in the master code causing this... The only thing I'm curious about is someone complained of the controller switching to a different mode or loosing connection by a button press this seems like a PWM issue but it could be anything I'm just throwing stuff out there. The reason being is I just ordered a green cube, Maybe I'm just being paranoid ah heck I know full well I'm going to chuck that cube into the solo the second I have it in my hands and follow Matts instructions to a tee and be up and flying....I have total faith......Gulp...

The green cube flies great! I had some little problem and I am sure someone will get to the bottom of it. The looping problem will be solved also I'm sure. I will go back to my stock Solo's for the commercial work for now.

 

[pyrate]

that crash is indeed odd. Mine has been pretty solid so far.

 

[David Boulanger]

that crash is indeed odd. Mine has been pretty solid so far.

Mine has been solid also except for two consecutive take offs. I'll give this some time. Logs don't lie.

 

[Pedals2Paddles]

Update on this slow loop thing. First, it is incredibly unlikely it had anything to do with the shutdown just after taking off. We had a long discussion about this during and after the ArduPilot developers conference call on Monday.

The slow loop thing is indeed a problem that needs to be addressed. It is actually horrendous. But it isn't anything anyone here has done wrong. This has apparently been a problem since 3DR began the Solo and it's just been conveniently ignored since it doesn't seem be causing noticeable problems. Kind of like the ESC problems and GPS problems that management decided to let slide and give it a cute coat of paint to hide it. And this is going to be ALL SOLOS. Not just yours. Even stock solos with stock cubes, though not as bad. It is much more pronounced on the Green Cube with 3.5 because it does so much more work.

The main loop runs at 400hz. The slow loop log counts the loops that run longer than they should over the prior 10 seconds, which at 400hz is 4000 loops through the code. It is normal to see 0 to 5% slow loops. On the Solo, 30-40% of the loops are running slower than they should, with some running really really long. 8 milliseconds may not seem like a long loop. But in the wise words of Lt Commander Data, "for an android, that is an eternity." Some folks on the dev team were surprised it's not degrading flight control. So suffice to say at best there is no room left for it to get worse and something needs to be done about it.

Of course 3DR already decided a long time to ignore this, and they're certainly not going to fix it now. So looks like it will be me and the ArduPilot dev team doing something about it. Which happens to include former 3DR folks that are familiar with the issue already, which is helpful. I've narrowed the slow loop culprit to the Solo's mavlink gimbal and the mavlink dataflash logging to the IMX. Each one is responsible for 400-600 slow loops out 4000. And combined, they push it up to 1100 to 1400 slow loops per 4000.

• Disabling the gimbal logging took a good chunk out of it, taking the slow loops down to about 20%. This is generally useless data for normal operations.
  
• Finding a more efficient dataflash LOG_BITMASK parameter will reduce the amount of unneccessary data being logged, and shave that down some more.
• Other inefficiencies TBD.

The good news is this doesn't seem to effect the flight control and performance. At least not in any noticeable way. I've made it worse cruising around with fast IMU logging, and still didn't suffer any noticeable ill-efects. And this again has been a problem for for two years now. So this isn't something that requires everyone to park it. But it is something that needs to be addressed.

 

[carpy]

Excellent write up--thanks Matt.