General Discussion => Magic Lantern Forum and Site Discussion => Topic started by: a1ex on September 20, 2016, 06:47:25 PM

Jenkins is overloading the server too much for my taste lately, so I'm considering rewriting the nightly builds page as static HTML, without any JavaScript. Another reason for the rewrite: the builds page is impossible to load on slow network connections.

Any volunteers to help me with this task? I'm going to use a Python script similar to this one (https://bitbucket.org/hudson/magic-lantern/src/tip/features/), and here's what I came up with so far (edited manually, based on the previous template):

Hi, I went to the download page today to download the latest version of ML for the 60D and found that the download site is down. I had a previous version on my card a while ago, but since then my wife has formatted my card and I lost magic lantern. I haven't really needed it so I didn't worry about it, but I'm filming a wedding this weekend and needed to put it back on. I might be able to go back and search my computer for the old magic lantern, but it was literally over a year ago that I had it on. I don't know how the download site works, but is there any way someone could send me or put on dropbox or some other download site the latest stable or nightly of ML for the 60D? I would greatly appreciate it. Thanks.

I would recommend running jenkins just once every 24 hours or something to that effect. Or if the repo has webhooks (like github does) then you can have it fire a rebuild request to your build server whenever new code is merged. From there, you could do something as simple as letting your webserver list directory contents of a build folder. You can do this with apache or nginx. If you wanted a better interface than that, I'd be happy to volunteer for that. I have a platform that I use called Lightning for dynamic content. It's an extremely lightweight CMS built for speed and high volume requests. It can even run without a database, although even with it, you can handle plenty of requests. If you want to talk about it send me a PM.

A1ex, I've got a simple Python script running now, that reads a generic template file, takes the URL of the latest nightly build from Jenkins and write everything into a new HTML file. I could not manage to read anything else (e.g. build time, changeset hash or the number of downloads) from the Jenkins instance so far. I need to understand the Python JenkinsAPI better if it's even possible. To bad currently Jenkins is not reachable here. So I couldn't test out things.

A1ex, I've got a simple Python script running now, that reads a generic template file, takes the URL of the latest nightly build from Jenkins and write everything into a new HTML file.

Sounds good!

Quote

Too bad currently Jenkins is not reachable here. So I couldn't test out things.

Just restarted it, hope it's fine now. It used to lock up within minutes after starting, but since I've disabled the builds page, it's a little better. Not sure how to diagnose it - the symptom is very high CPU usage with no apparent reason (and no obvious increase in website traffic - it's actually pretty quiet).

Quote

Do you plan to use one Python script that generates multiple HTML file or do you need to call it with parameters for each camera model?

I don't mind, either way should be fine.

Quote

> There's some extra functionality I'd like to add, too.Could you tell me what you like to add in future?

Sure, most of it should be obvious from the extra menu items from the example HTML, but I'll give some details below.

* validation: I'm not good at web development, so other than the w3c validator, I'm not sure what to look for.

* top menu: - main builds: ports merged to mainline - early ports: ports that are still in progress and can be compiled from their own branches (e.g. 1200D) - additional modules: - here I'm thinking to list the modules that are not widely used, or still experimental (e.g. io_crypt, tiltcalc, raw_diag) - I can compile from user repos as well - todo: a list with what's worth including here - what about naming this just Modules, moving the modules section from the features page here, and listing additional modules below? - utilities - things like mlv_dump, raw2dng, cr2hdr, with both win/mac builds - todo: a list with what's worth including here - experiments - builds from branches like new_sound_system, crop_rec, iso-research - what about compiling pull requests automatically, so users can give feedback easier? - other stuff worth including?

* buttons: - change log - this used to be a popup box, which looks nice, but requires javascript, which I'd rather avoid - I'm open to suggestions on how to make them more obvious - what about displaying the changes for latest build right below the buttons? - the changes for old builds could be in a large HTML page; maybe only show the last month or so? - build log: plain text is fine for me; better suggestions? - old builds: large HTML page? - screenshots and tests: - proof of concept here (http://www.magiclantern.fm/forum/index.php?topic=12396.msg119592#msg119592) - we may start with some menu screenshots from qemu (should already work on some cameras like 60D, 5D3, 500D, 1200D), and do the rest later. - a simple test would be to start each binary in QEMU and check whether ML config file gets created (this should also work on models with SD card that cannot boot the GUI yet) - this section has huge potential IMO (automated burn-in tests and such), but let's start with something easy, just to get the ball rolling. - discussion: just links to the forum (for each camera)

We don't need to finish all of these in order to get it back online; just the basics (previous functionality). I'm just thinking in advance, what would be nice to have over the next few months.

That's it for now, thanks for helping on any of those items. I expect to have some more time next week, to work on it; meanwhile, I'm mostly available during evenings (European time). We may discuss on IRC as well if you prefer.

A1ex, I make some progress with the Python script. Now I could generate HTML files with the data from Jenkins, that contain almost every information, like your 550D.html page example in your first post, but for every camera currently supported by ML. Missing within the HTML is currently the hash of the Mercurial changeset from the repository, the commit messages and the number of downloads. I need to extract the first two values from a structure which appears to me like a JSON object or array.

The number of downloads must be read somehow with the JenkinsAPI, but I don't know how to do it yet. I pushed my latest progress into my development branch.https://bitbucket.org/tecgen/magic-lantern/commits/c562c7bfd54ffc088ce79d3aba9c8b0349be3970?at=unified

Before you finish a non-Javascript version. I created a Javascript-based website for people to use in the meantime.https://coloraggio.github.io/builds.magiclantern.fm/

It is as fast as Jenkins API allows (I optimized query parameters to make responses faster). I also implemented a client-side caching to save useless request when user is navigating back and forth between pages.

@tecgen: the number of downloads is not given directly by Jenkins - that's my hack, and the info is available in a separate JSON file ( http://builds.magiclantern.fm/download-stats/api/json ). It only contains the most downloaded ones, and it's a static file (recomputed periodically).

BTW, the old site is available at http://builds.magiclantern.fm/index-js.html, for reference.

Yes looks very good , but I notice you can only download the latest nightly builds , What about older builds ? How will there be accessed ? As we all know the latest nighty builds sometime can very problematic .So we my have to reverting back to a older build. Before you could show all the older builds from a dropdown menu , I think this needs to be re-implemented .

(a small HTML page with latest build, a larger HTML page with all the old builds, and a huge HTML listing all the changes as well)

The changeset info from Jenkins appears both broken (e.g. incorrect on 1100D) and incomplete (missing branch info, for example), so I guess I should take just each build's changeset from Jenkins and lookup the other changes directly on mercurial.

I'm also thinking to show only the large changes (e.g. merged pull request XYZ, with link), since a large page with all that stuff doesn't look quite easy to read. Better ideas?

Hi folks,I tested the new builds page a bit and it looks good! Great work!I just have a few things to suggest for a better readability.

1st. It would be nice to group the changeset list and make it like the Older builds one. So you can click on "changes", or whatever you want, and show the list and then hide it.2nd. Could you set the the fw version under the camera model a little bit bigger? My eyes are still good but I had to read it two times to choose the right version. :DAnother possible solution for this, that would also shorten the list, would be: Write the camera model as simple text and write one or more firmware versions as links on its right or below it.For example:

About the "other builds" page. In my opinion skipping the "short" version and directly showing the full version when user clicks "older builds" is better, since it's just an HTML page and person interested in older builds will prefer information about changes right away.

P.S. a1ex is the source of this website on bitbucket? I can contribute on HTML, CSS, typography, etc.

@a1ex thank you for everything you're doing for ML!At some point when it's on Bitbucket people (me or others) can contribute and do fixes here and there. The most important thing is that the website right now as it is looks fine and works fine.

@A1lexJust a couple of suggested wording changes (in red) to the following section on the "Builds" page.

When it will be ready?Unfortunately, Magic Lantern ports don't happen to a schedule.There is no plan, and there is no Magic Lantern organization that specifies which port happens next.If there is work being done on a camera, there will be a development thread in the forum.If there is no mention of development activity on the forum, there is no reason to ask about that camera's status: It is not supported, and there is no way for anyone to know if or when it might be supported. Your guess is as good as ours.If there is no ML available for your camera, you should act like it never will be. -- Walter Schulz

Maybe display the latest Build Date and Time under each camera model on the Nightly Build landing page or some other indicator that a new build exists. This would remove the need to drill down further unless a new build was available.

Hello,I'm brand new on this forum but I'm a young amateur independent web designer. I would like to help as much as I can, because I think that I may have some good ideas or advices and that it may be great experience too !

So, to begin softly, I just ran a quick speed test on your page, using common tools, here are the results :

I suggest to avoid links to Canon at all and use Pelican's repository for 5D2, 550D and 50D instead.

Installation instructions for 5D3.113 and 5D3.123 may be altered from"[...] If you are running 1.3.3, you can downgrade with EOS Utility."to"[...] If you are running 1.3.3 or higher, you can downgrade with EOS Utility 2.x."A link to EOS Digital Solution Disk 29.1A (containing EOS Utility 2.14.10) might be helpful. For all I know EOS Utility 3.x will not allow downgrading firmware.

Don't know if you already have this but it would be great to track the number of downloads per platform. The Nightly builds page would be an obvious place to show the number of downloads.

Reasons for this are:

1- People looking into buying a camera can see which camera and firmware version is the most popular with ML users.2- Developers can see when a new feature shows a spike in the number of downloads.

I've been posting some test builds from development branches and was surprised at the amount of interest in the 5D3.123 despite what seems to be a recommendation to use 5D3.113 for better performance or if clean HDMI out, dual monitor support and AF at f/8 with teleconverters is not required.

A bit off topic but since Walter brought it up -- when upgrading or downgrading the Canon firmware I've always put the Canon firmware on the card and done the process in camera without problems. The issue with not being able to downgrade seem to apply only to users attempting to do it using EOS Utility 3.x. Am I missing something by doing it in camera?

One more thing -- you can include the 5D3 to the list of Canon updated firmware. Version 1.3.4 was released on November 29, 2016. Pelican has already added it to his site.

The issue with not being able to downgrade seem to apply only to users attempting to do it using EOS Utility 3.x. Am I missing something by doing it in camera?

Firmware can only be downgraded to 1.1.3 from eos utility when going from 1.3.4. Do it from the card and your camera will tell you no. Do it from eos utlity and we,re ok. Thankfully.The issue doesn,t seem to occur with 1.1.3 to 1.2.3 and back.

@dfort & garry23: Canon introduced firmware downgrade check with 1.3.3 for 5D3. As Danne wrote downgrading is blocked in 1.3.3 and 1.3.4 when using the cam's menu option.@dort: Links for 5D3 - as you can check - are leading to Pelican's repository. Therefore 5D3 is not included in the list above.

@a1ex Thought I'd be helpful, did a GIF for burst.mo(I hope no one else is making the same gif as I post, but if you are, I'm sorry, but still post it because it's probably better than mine)(http://gdurl.com/YITu)Is the 5D2 screen resolution the same as others? I thought it was 640x480 but the screenshots are 720x480

Reason: current web browsers don't seem to do a good job at resizing the original (aliasing issues).

5D2 has 640x480 physical resolution, but 720x480 logical buffer. The exact mapping from buffer to display is unknown; can be probably found with a bit of patience to try some patterns.

@Licaon_Kter: do Linux binaries make any sense? (I mean, compiling host binaries on this system usually just works)

For Mac, I need some help setting up the build environment. I've tried a while ago on the old server, with advice from kichetof (https://www.magiclantern.fm/forum/index.php?topic=7139.msg126726#msg126726), but got stuck.

@Licaon_Kter: do Linux binaries make any sense? (I mean, compiling host binaries on this system usually just works)

Like on any other platform, don't just go the "if you run Linux you already compile stuff" way :) (if your response was like that)Else, that page kinda says "You'll need Windows to use Magic Lantern" otherwise ;)

Okay, pull request welcome for statically linking these binaries, preferably with this tool (http://ptspts.blogspot.com/2014/01/announcing-pts-xstatic-tool-for.html). Bonus points for x86 and ARM support in the same binary.

Is there any easy way to get to the experimentals (https://builds.magiclantern.fm/experiments.html) page from the home page (http://magiclantern.fm/) by following links? Seems like the only way to access that page is by the URL posted on various forum posts.

Are the experimentals also on a nightly schedule? Just wondering because there were some updates that haven't come through yet.

Finally, which branches are the experimentals coming from? The Crop mode recording build log says 5D3.113-crop3x-april-1st-edition which isn't in the main repository and the log doesn't have any trace of 5D3.123 yet there's a build for it.

Hey, how did you get there? I removed that page a while ago, because it was pretty outdated and some things were not exactly true. Would be nice to have it updated.

http://www.magiclantern.fm/features.html and yes I'm interested in helping out on getting you updated screenshots for the feature page since the new layouts looks really nice and easy to scroll through -- is that right, @Audionut? :)

In the experimental area would it be useful to place a link in each sub section, eg Lua_fix etc, including sections such as Lua_api_docs, that took one straight to the correct area to report things and suggest changes.

At the moment you need to know where to report, and if you don't know you are rather lost.

Main BuildsThese builds have been around for some time, and they are unlikely to cause major issues.In most cases, regressions are fixed quickly - if you report them.

I like this text, and think they are OK.

But the builds here display once a day the Warning message "This is a development snapshot for testing purposes." etc. (menuhelp.c, draw_beta_warning)

In my opinion, this is a inconsistency. I would remove this message from ML.Or an alternative would be to display the history, if ML is updated (once, not every day), like some Android Apps does, and then the Text: "Please report all bugs at www.magiclantern.fm"

(Or change the nightly build, and defined CONFIG_RELEASE_BUILD?).

I know, it's a little bit offtopic, but I think the message should match the new Download page, even if the page is not that new anymore...

2) make sure the builds are tested before posting; that was done manually years ago, when the feature set was small and I was working on 1 or 2 camera models; however, this approach no longer works (with hundreds of menu options - if not one thousand - and several modules, workflows, bleeding edge features and so on).

Made some progress for #2 - most of the nightly builds are now tested, to a limited extent, in QEMU (http://www.magiclantern.fm/forum/index.php?topic=20560).

Layout is still fine to navigate, but it's a little confusing: when some builds pass less tests than others, does that mean they failed some?How would it present failing half and passing half?Maybe say how many there were in total?

Design looks good and comprehensive (with tooltip info)Maybe you could warn users when the build failed tests if it's "dangerous" to play with it (if a test is crucial for install it for exemple, maybe with a ponderation if needed)

About print screen, what do you think to add a gallery inside the build page ? like that (http://ashleydw.github.io/lightbox/) or that (https://blueimp.github.io/Gallery/#links)

If I'll also compare the screenshots with the previous run (http://www.magiclantern.fm/forum/index.php?topic=12396.0), there will be a *lot* of screenshots - and the current tests are just scratching the surface. I'd keep a few important screenshots on the main page, then add an additional page with detailed test reports. For now I'm just linking to Jenkins log, but that's not going to scale with more than 10-20 tests.

For the main page, a gallery could be a little more user-friendly, indeed.

Meta data access is intriguing. Want to check which options are available (brain fart developing).

Page layout in https://builds.magiclantern.fm/index.html is useable. Minor: I do not like use of colours that much. Blue captions for camera builds tend to be overlooked because red and green tags will attract vision first. Should be the other way round: User is looking for the cam in use. Maybe there is way to hide/grey out metadata and only get info if cursor hoovers over the caption/cam area?

Bigger one: Wording.Most ML users (in my opinion) do not have the slightest idea what QEMU is and don't understand the concept of emulators either. Users have to know there is (at time of writing) a difference running emulations and running ML in cam and why automated tests are useful but not sufficient to avoid software errors (bugs).

Same for "regression". Let's - for simplification - just call it "bug".

Now to cam build page (example 5D Mark II 2.1.2):Not happy with layout.There are 5 lines with checkmarks for each test and then a line beneath with a ballot box and a checkmark and then changes and then ...

Meta data access is intriguing. Want to check which options are available (brain fart developing).

Anything you can think of - data from Bitbucket, from Jenkins, from parsing the source code, or the result of some emulation, or the result of some Lua script running on the camera, or a list with menu entries present in the build, or a list with included modules...

Looks good! To be sure user clicks on accordion, add a chevron in the right of the button, like that (https://www.codeply.com/go/gOBrMgWpzS/bootstrap-collapse-accordion-with-chevron-icons)Minor update with the border under the button, add a little margin (8px like padding in alert) in your css:

Collapse all of the build information to headings, so that you can see all builds when navigating to that page. Click on heading to un-collapses all of the information for that build.Linkable headings could be useful also.

I have downloaded Canon firmwares for all of the available ML models. This will be a backup (https://www.dropbox.com/sh/uvdmveo80wz8zs6/AAAFbubagD1pVWBH94h_uFSBa?dl=0), and contain all the firmwares once uploaded.

I also backed up the Canon firmware updaters. You'll find all of the applicable versions for the 5D3 on my Bitbucket downloads page (https://bitbucket.org/daniel_fort/magic-lantern/downloads/). Look for the "5D3 Canon Firmware for Testers.zip" package.

Back then, the nightly builds were built in the same way (that was the default config, I think). Some people actually thought there were daily updates, and some of them even reported differences between identical builds!