How DHE@BOINC works - READ THIS FIRST!

Message boards : Number crunching : How DHE@BOINC works - READ THIS FIRST!
Message board moderation

To post messages, you must log in.

1 · 2 · Next

AuthorMessage
Michael DHEP
Project administrator
Project developer
Project scientist

Send message
Joined: 13 Jun 18
Posts: 196
Credit: 0
RAC: 0
Message 269 - Posted: 26 Jul 2018, 12:48:44 UTC
Last modified: 5 Aug 2018, 15:24:05 UTC

A few things you should know about DHE before you start crunching with BOINC:



Now to BOINC:


    o Credit is awarded hourly for processing done within workunits.
    o Workunits are placeholder dummies of non-defined duration with no required validation.
    o Percentage progress can be ignored as is meaningless for tasks of non-defined duration.


And that's it! We hope you enjoy our project!



If you'd like more details keep on reading...


    o DHE runs a Genetic Algorithm on your computer. These run for a non-defined time until a good solution is found. Hence "progress" within a workunit is meaningless and in fact workunits are dummies and do not represent actual work done.
    o The DHE BOINC app will connect to our servers every 15 minutes and upload their progress. You can check how much you have made today (and yesterday) at: http://dhep.ga/statsdailyrankings.php. You can check how many of your cores are actively crunching by checking how many times your username (with spaces as _) appears at: http://dhep.ga/statstopology.php.
    o Credit is awarded by a Validated Trickle system and is not awarded on workunit completion, because as mentioned above workunits are dummies. Hence you will receive credit by how much work is done within a workunit.
    o Credit is awarded proportional to how much is contributed scientifically. This is roughly proportional to Ghz core/hours. So a core twice as fast will receive roughly twice the credit. Credit is awarded hourly.
    o Restarting your BOINC client will show workunits resuming at 0%. This is normal and no work has been lost because you have already been awarded credit for all previous work done within the workunit. Remember the DHE BOINC app connects to our servers every 15 minutes to upload its progress. There is little value in having a percentage completion figure for a task with a non-defined processing time.
    o All that said, workunits will occasionally end. This is when the goal at http://dhep.ga/statsgoal.php is changed which is roughly every two to four weeks.
    o Computation Error will occasionally be the outcome of workunits. In most cases this is normal: when the server restarts with different class files the client will detect this through a NoClassDefFoundException. This will trigger the client to shutdown and restart as another WU to pick up the new class files.
    o Workunits can be aborted at any time without any loss of progress or credit. However a Genetic Algorithm is more efficient when islands are connected continuously.
    o Badges will be awarded and there are a variety of stats on which they will be based such as the number of times the Best of all Islands was arrived at on your island (aka 'Who's got the Knack'): all stats visible here: http://dhep.ga/statsrankings.php.
    o You can also view live progress of the current run at http://dhep.ga/statsgoal.php and live genetic strains analysis at http://dhep.ga/statsstrains.php, and a list of GPL licenced results at http://dhep.ga/statstschof.php including an online circuit simulator to try them out.
    o Any further questions may be answered at http://dhep.ga/faq.php and will be more than happy to help if not.



Happy Crunching!

Michael

ID: 269 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Steve Dodd

Send message
Joined: 25 Jul 18
Posts: 10
Credit: 3,579,976
RAC: 27,392
Message 274 - Posted: 26 Jul 2018, 21:35:26 UTC - in response to Message 269.  

Could you tell me more about the badges part? :)
ID: 274 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
lugu

Send message
Joined: 26 Jul 18
Posts: 1
Credit: 604,455
RAC: 6,016
Message 278 - Posted: 27 Jul 2018, 9:13:01 UTC

Hi Michael,

just to make this clear (for me): Restarting the BOINC Manager does not mean a loss of work? Does the Genetic Algorithm start where it was suspended earlier?

I'm referring to: " Restarting your BOINC client will show workunits resuming at 0%. This is normal and no work has been lost because you have already been awarded credit for all previous work done within the workunit."

Cheers,

lugu
ID: 278 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Contact
Avatar

Send message
Joined: 10 Jun 18
Posts: 4
Credit: 4,466,155
RAC: 15,381
Message 279 - Posted: 27 Jul 2018, 11:13:55 UTC - in response to Message 278.  
Last modified: 27 Jul 2018, 11:15:35 UTC

Restarting the BOINC Manager does not mean a loss of work? Does the Genetic Algorithm start where it was suspended earlier?


Restarting will reset the workunit to 0%, but no loss of work.
https://dhep.ga/boinc/forum_thread.php?id=31&postid=269 wrote:
o Restarting your BOINC client will show workunits resuming at 0%. This is normal and no work has been lost because you have already been awarded credit for all previous work done within the workunit. Remember the DHE BOINC app connects to our servers every 15 minutes to upload its progress. There is little value in having a percentage completion figure for a task with a non-defined processing time.

ID: 279 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Michael DHEP
Project administrator
Project developer
Project scientist

Send message
Joined: 13 Jun 18
Posts: 196
Credit: 0
RAC: 0
Message 280 - Posted: 27 Jul 2018, 15:39:41 UTC - in response to Message 274.  

Could you tell me more about the badges part? :)

:) They will make a dashing addition to your collection I'm sure. Stay tuned.
ID: 280 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Michael DHEP
Project administrator
Project developer
Project scientist

Send message
Joined: 13 Jun 18
Posts: 196
Credit: 0
RAC: 0
Message 281 - Posted: 27 Jul 2018, 15:43:51 UTC - in response to Message 278.  

just to make this clear (for me): Restarting the BOINC Manager does not mean a loss of work? Does the Genetic Algorithm start where it was suspended earlier?

Hello Iugu,

The amount of evaluations you've done has already been stored and credited, so that takes care of that.

Where the GA will resume from is a more complex question. Upon reconnection you will be assigned a new location on the topology: https://dhep.ga/statstopology.php . Locations are reused when the server hasn't heard from the previously assigned island for over 48 hours. In this case your GA would resume from where that one left off - ie their population would be loaded. If you are assigned a new location no one's been at before your GA starts from scratch. However due to migration it will soon receive a fit migrant and will resume progress near the front of the fitness curve.

Kind regards,
Michael
ID: 281 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
M0CZY

Send message
Joined: 6 Jul 18
Posts: 2
Credit: 361,464
RAC: 202
Message 282 - Posted: 29 Jul 2018, 10:51:21 UTC

Sorry in advance if this question is in the wrong thread, or wrong forum.

I have tried out the standalone client on my Ubuntu Linux computer, and it works fine.
But if I do Ctrl+C then close the terminal and the Java windows, the program keeps running, and even if I kill the Java processes in the System Monitor, the program restarts and the Java windows reappear.

What is the correct way to safely stop the program running, apart from having to turn the computer off?
ID: 282 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Michael DHEP
Project administrator
Project developer
Project scientist

Send message
Joined: 13 Jun 18
Posts: 196
Credit: 0
RAC: 0
Message 284 - Posted: 29 Jul 2018, 12:20:34 UTC - in response to Message 282.  

Sorry in advance if this question is in the wrong thread, or wrong forum.

I have tried out the standalone client on my Ubuntu Linux computer, and it works fine.
But if I do Ctrl+C then close the terminal and the Java windows, the program keeps running, and even if I kill the Java processes in the System Monitor, the program restarts and the Java windows reappear.

What is the correct way to safely stop the program running, apart from having to turn the computer off?

Excellent question. I have made a tweak to the script. You can now stop all running threads with:
touch stop
pkill -f 'ITClient'


Kind regards,
Michael
ID: 284 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
djoser

Send message
Joined: 30 Jul 18
Posts: 5
Credit: 495,309
RAC: 8,314
Message 295 - Posted: 30 Jul 2018, 22:51:49 UTC - in response to Message 269.  

You can check how many of your cores are actively crunching by checking how many times your username (with spaces as _) appears at: http://dhep.ga/statstopology.php.


Hi!

Is there any way to limit the used cores via BOINC to crunch this project alongside other projects?
Maybe using a app_config.xml file?

I'd like to contribute only one core and crunch other projects concurrently.

Thanks and regards,
djoser.
ID: 295 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
JugNut

Send message
Joined: 2 Jul 18
Posts: 27
Credit: 62,673,001
RAC: 11,152
Message 296 - Posted: 31 Jul 2018, 0:42:15 UTC - in response to Message 295.  
Last modified: 31 Jul 2018, 1:34:02 UTC

Hi, just copy & paste the below in notepad. Then save it as app_config.xml in "C:\ProgramData\BOINC\projects\dhep.ga_boinc" folder.
Then in boinc manager click the "Options" menu then click "Read config files" to activate it.

The only problem with doing it this way is that you'll end up with extra WU's you don't need. And because the WU's are so long it can eventually put boinc into panic mode anyway. You can of course suspend the WU's your not running but then when your WU completes a new one wont start automatically. Or you can abort the WU's you don't need but then fresh WU's will download automatically to replace them. So the only option is to set NNT(no new tasks) for DHEP and then delete the WU's you don't want. But even that's not perfect as then you will not be able to get new work once your single DHEP WU complet's.

Anyone got a solution to this common problem?(when running multiple projects at the same time)

Maybe Michael can implement a scheme similar to what other projects have done whereby an option is put put in the preferences to limit the amount of downloaded work? The usual method is a drop down box with the amount of work you want. ie: a drop down box with 1 - 30 in it. Where 1 means only download 1 work unit at a time ect ect.


<app_config>
<app>
<name>jreitc</name>
<max_concurrent>1</max_concurrent>
</app>
</app_config>

Crunching today for a better tomorrow...
ID: 296 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
mikey
Avatar

Send message
Joined: 24 Jul 18
Posts: 12
Credit: 43,254,666
RAC: 713,354
Message 311 - Posted: 5 Aug 2018, 12:30:46 UTC - in response to Message 296.  

Hi, just copy & paste the below in notepad. Then save it as app_config.xml in "C:\ProgramData\BOINC\projects\dhep.ga_boinc" folder.
Then in boinc manager click the "Options" menu then click "Read config files" to activate it.

The only problem with doing it this way is that you'll end up with extra WU's you don't need. And because the WU's are so long it can eventually put boinc into panic mode anyway. You can of course suspend the WU's your not running but then when your WU completes a new one wont start automatically. Or you can abort the WU's you don't need but then fresh WU's will download automatically to replace them. So the only option is to set NNT(no new tasks) for DHEP and then delete the WU's you don't want. But even that's not perfect as then you will not be able to get new work once your single DHEP WU complet's.

Anyone got a solution to this common problem?(when running multiple projects at the same time)

Maybe Michael can implement a scheme similar to what other projects have done whereby an option is put put in the preferences to limit the amount of downloaded work? The usual method is a drop down box with the amount of work you want. ie: a drop down box with 1 - 30 in it. Where 1 means only download 1 work unit at a time ect ect.


<app_config>
<app>
<name>jreitc</name>
<max_concurrent>1</max_concurrent>
</app>
</app_config>


Would it be helpful if we used more than one cpu core per wu here? Similar to your current app_config file but adding lines like this:

<app_version>
<app_name>jreitc</app_name>
<cmdline>-t 6</cmdline>
<avg_ncpus>6</avg_ncpus>
<max_ncpus>6</max_ncpus>
</app_version>
ID: 311 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
kashi

Send message
Joined: 3 Jul 18
Posts: 4
Credit: 46,138,483
RAC: 114,729
Message 316 - Posted: 6 Aug 2018, 7:24:05 UTC - in response to Message 296.  

Hi, just copy & paste the below in notepad. Then save it as app_config.xml in "C:\ProgramData\BOINC\projects\dhep.ga_boinc" folder.
Then in boinc manager click the "Options" menu then click "Read config files" to activate it.

The only problem with doing it this way is that you'll end up with extra WU's you don't need. And because the WU's are so long it can eventually put boinc into panic mode anyway. You can of course suspend the WU's your not running but then when your WU completes a new one wont start automatically. Or you can abort the WU's you don't need but then fresh WU's will download automatically to replace them. So the only option is to set NNT(no new tasks) for DHEP and then delete the WU's you don't want. But even that's not perfect as then you will not be able to get new work once your single DHEP WU complet's.

Anyone got a solution to this common problem?(when running multiple projects at the same time)

Maybe Michael can implement a scheme similar to what other projects have done whereby an option is put put in the preferences to limit the amount of downloaded work? The usual method is a drop down box with the amount of work you want. ie: a drop down box with 1 - 30 in it. Where 1 means only download 1 work unit at a time ect ect.


<app_config>
<app>
<name>jreitc</name>
<max_concurrent>1</max_concurrent>
</app>
</app_config>


For projects where extra tasks in the cache are not required and to only download new work to replace tasks that have completed I sometimes use a Resource share setting of 0 on that project. Like quite a few others, I almost always use an app_config.xml file with max_concurrent specified for every project to fine tune how many BOINC cores to allocate to each project. So that means I decide how/when projects use resources rather than BOINC scheduling different projects to run at different times.

It sometimes doesn't work if the cache gets full with the other project and BOINC refuses to download new work, but that can often be worked around by using a higher cache setting than the task download limit of the other project.

I understand it would be more complicated and may not work properly if you were running more than 2 projects at once. Also because of the long deadlines of DHEP this kind of micromanaging of BOINC doesn't always work automatically if many BOINC cores are allocated to DHEP as then your BOINC cache can be full.

However for those like djoser who want to run only 1 BOINC core of DHEP continuously, then using Resource share of 0 and max_concurrent value of 1 and with the other project also limited by app_config.xml <max_concurrent> as detailed by JugNut in this post https://www.dhep.ga/boinc/forum_thread.php?id=39&postid=307#307 should work to allow continuous processing and downloading of a single DHEP task.

If using a Resource share of 0 on a project for this purpose, it's necessary to limit the number of BOINC cores to use for other projects otherwise the project with Resource share of 0 will not download any work at all unless the other project with Resource share of greater than 0 runs out of work.
ID: 316 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Trotador

Send message
Joined: 29 Jul 18
Posts: 2
Credit: 11,690,277
RAC: 521,294
Message 348 - Posted: 11 Aug 2018, 17:57:26 UTC

Hi,

You explain that the client connects to the server every half an hour to report the advance and the scoring advance agrees with it. However, I do not find any connection process registered in the boinc manager log and my list of hosts page show that the last connection was over 30 hours ago for the host that is processing now most of the units, the other host shows a more recent contact time because I've forced an update in the boinc manager.

Curiosity on my side. How is that those connections not registered? Is it because of the Java VMs nature? Shoudnt they be registered?

Thanks
ID: 348 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Michael DHEP
Project administrator
Project developer
Project scientist

Send message
Joined: 13 Jun 18
Posts: 196
Credit: 0
RAC: 0
Message 354 - Posted: 13 Aug 2018, 15:51:29 UTC - in response to Message 348.  

Hi,

You explain that the client connects to the server every half an hour to report the advance and the scoring advance agrees with it. However, I do not find any connection process registered in the boinc manager log and my list of hosts page show that the last connection was over 30 hours ago for the host that is processing now most of the units, the other host shows a more recent contact time because I've forced an update in the boinc manager.

Curiosity on my side. How is that those connections not registered? Is it because of the Java VMs nature? Shoudnt they be registered?

Thanks

Hi Trotador the connection is made from the application directly to the DHE server, the DHE server in turn validates progress and sends trickles to the BOINC server.

Kind regards,
Michael
ID: 354 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
enginerd

Send message
Joined: 24 Aug 18
Posts: 3
Credit: 2,112,683
RAC: 34,406
Message 425 - Posted: 11 Sep 2018, 22:10:58 UTC

Do we need to open up any special ports in the firewall (i'm using ufw on linux) for connection to the project server, or will BOINC default port 31416 work?
Thanks!
ID: 425 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Michael DHEP
Project administrator
Project developer
Project scientist

Send message
Joined: 13 Jun 18
Posts: 196
Credit: 0
RAC: 0
Message 428 - Posted: 14 Sep 2018, 11:26:26 UTC - in response to Message 425.  

Do we need to open up any special ports in the firewall (i'm using ufw on linux) for connection to the project server, or will BOINC default port 31416 work?
Thanks!

Hello yes you will need to allow outgoing ports to rmi.dhep.ga (I believe RMI will begin at 1098 and 1099 and the allocates randomly so you may need a blanket allow to that host).
ID: 428 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
LumenDan

Send message
Joined: 30 Aug 18
Posts: 4
Credit: 349,585
RAC: 9,781
Message 510 - Posted: 21 Oct 2018, 1:34:59 UTC - in response to Message 269.  

Is it possible to schedule a daily project update so that BOINC Manager refreshes it's project statistics between work unit requests?
At the moment I am manually requesting a project update every few days to keep an eye on host averages, I'm not sure if the host statistics have any effect on task scheduling but a regular update would resolve any such issue.
ID: 510 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Michael DHEP
Project administrator
Project developer
Project scientist

Send message
Joined: 13 Jun 18
Posts: 196
Credit: 0
RAC: 0
Message 517 - Posted: 23 Oct 2018, 9:55:17 UTC - in response to Message 510.  

Is it possible to schedule a daily project update so that BOINC Manager refreshes it's project statistics between work unit requests?
At the moment I am manually requesting a project update every few days to keep an eye on host averages, I'm not sure if the host statistics have any effect on task scheduling but a regular update would resolve any such issue.

Hello Dan I am unaware of any effect of the host update on task scheduling at the hourly level. You can check your live host data on this website.

Kind regards,
Michael
ID: 517 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Contact
Avatar

Send message
Joined: 10 Jun 18
Posts: 4
Credit: 4,466,155
RAC: 15,381
Message 544 - Posted: 25 Oct 2018, 21:49:37 UTC

Maybe Trickle-down messages API would work here to update the clients.
https://boinc.berkeley.edu/trac/wiki/TrickleMessages
ID: 544 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
Michael DHEP
Project administrator
Project developer
Project scientist

Send message
Joined: 13 Jun 18
Posts: 196
Credit: 0
RAC: 0
Message 564 - Posted: 29 Oct 2018, 13:41:31 UTC - in response to Message 544.  

Maybe Trickle-down messages API would work here to update the clients.
https://boinc.berkeley.edu/trac/wiki/TrickleMessages

Nice one Contact. That might do the trick. Wish we had more hands on deck to sort this kind of thing ;) ;)
ID: 564 · Rating: 0 · rate: Rate + / Rate - Report as offensive     Reply Quote
1 · 2 · Next

Message boards : Number crunching : How DHE@BOINC works - READ THIS FIRST!


©2018