Basic information

Basic information to allow the comprehension of the documentations of this directory

It is important to understand the information in this document when undertaking restarts of the station computers or certain types of boot or reboot. This station cannot be turned on by feeling. This would only result in a great loss of time and energy or even damage that would prevent observation.

Most actions are possible remotely, so in this case, please read the preamble. On site, go directly to the next chapter!

Mail for technical support

PREAMBLE CONCERNING REMOTE ACTIONS

If the router of the Euler station is inaccessible, no action is possible. The router must respond to the following command, otherwise there is no need to go any further. If there is no problem on this side of the Atlantic, only a local physical action (ESO support) will be able to solve the problems.

ping 134.171.80.170

If Euler's network is accessible via glslogin1.ls.eso.org or glslogin2.ls.eso.org (the only 2 public addresses of Euler.), it is possible to perform remotes actions by connecting to it via ssh or x2go (x2go opens a Remote Desktop and allows easy use of terminals and browsers).

Remote actions can also be carried out directly from the observatory's two sub-networks connected by VPN to Euler's network 10.10.132.*

VPN is useful or even essential if glslogin1 or glslogin2 are inaccessible..

Electronics Lab 10.10.133.*
Astrodome 10.10.134.*

The Electronics Lab Nuc has the public address129.194.67.159

You can connect to it by x2go and in this case, you can also restart the router with firefox if necessary..

Another option (which requires passwords) is to use teamwork on the station's Windows PCs (Beckhoff development) and then launch a browser.

ARCHITECTURE

Computer groups

These are:

The disk server, consisting of 2 Linux machines mounted in High Avaibility Cluster (HA): the synologyCluster (10.10.132.101 consisting of 10.10.132.105 and 10.10.132.106)
The main workstation is a NUC that acts as a DHCP server: glslogin1 10.10.132.81
Auxiliary workstations, NUCs: including the spare for the main workstation: glslogin2 (10.10.132.82). These are glsaux, glsdrs, glsmonitor
Local control units (LCUs), Beckhoff industrial PCs physically interfaced to the instruments, 5 in the LCU cabinet and 2 in the REM cabinet (in the telescope) These are glsserv, glsecam, glscora, glspc20, glspisco, glstreg, glstopt
The 2 Windows PCs for Beckhoff software development

Group 5 is not required for observation with Euler. The PCs can be left off and then switched on when a stable situation is observed.

Alimentation électrique commutable à distance

Toutes les machines sont alimentées au travers d'IP-Power (ou Internet Power Switch), c'est à dire que l'alimentation en courant électrique peut être enclenchée ou déclenchée depuis un site WEB. L'enclenchement manuel est également possible par les boutons de la face avant.

Les 3 IP-Power sont dans l'armoire Serveur (10.10.132.91 ; L105), l'armoire LCU (10.10.132.92 ; L105) et dans le REM (10.10.132.94 ; L201, embarqué sous le télescope)

Les commandes suivantes sont accessible depuis la workstation principale ou en remote:

T_ippower_stat                    // status of all IPpower

T_ippower_reset <host> [...]      // reset "intelligent" : n'éteint pas si <host> répond au ping

T_ippower_set <glsippowerXX> <port(1-8)>     // switch on a port

T_ippower_clear <glsippowerXX> <port(1-8)>   // switch off a port (!! Commande dangereuse !!)

T_ippower_reset_nocheck <glsippowerXX> <port(1-8)>   // reset a port (!! Commande dangereuse !!) (on,wait,off)

Ordre de demarrage

L'ordre de démarrage est essentiel, significant que certains groupes ne doivent pas être démarrés s'il dépendent d'un autre groupe. AInsi:

Le groupe 1 (SynologyCluster) doit fonctionner. Chaque ordinateur monte les disques du serveur. Le serveur doit être réactif pour envisager le boot des autres machines.
Le groupe 2 (glslogin1) doit fonctionner (ou glslogin2 en cas de panne de glslogin1) pour que les ordinateurs qui seront bootés après lui récupèrent leur adresse IP sur le serveur DHCP qu'il héberge
Les groupes 3, 4 et 5 (workstations annexes, LCUs et PCs) peuvent être bootés dans n'importe quel ordre à la seule condition que le SynologyCluster (1) et la workstation principale (2) soit accessibles.

En résumé pour un démarrage de la station:

serveur de disque ... attendre accessibilité
glslogin1 (ou glslogin2) ... attendre accessibilité
les autres ordinateurs, même simultanément

Switching on the machines

To switch on a machine that is electrically switched off (which has no fan/light on) the port corresponding to it must be switched to the IP Power corresponding to it (usually next to a length of cable) which must be OFF.

Switching on or off the power supply must only be done with awareness of what is being done.

Turning on the power to a computer must be done in the order described above (see Startup Order). Turning on a computer prematurely will require a shutdown, thus wasting time.

Turning off a power supply or performing a Reset is equivalent to pulling the plug out abruptly. Doing so may destroy the hard disk of the machine in question. The risk is normally non-existent on workstations of the NUC type which do not have hard disks, but SSD-type memory.

IP-Power is configured to keep its ports off after a power outage. Be careful, IP-Power are powered by a UPS, so when we talk about a power failure, it means that the UPS batteries are exhausted and the UPS no longer plays its role. Power outages of an hour or less are handled by the UPS and IP-Power is not switched off, so neither are the computers.
Computers are configured to boot when they go from off to on state (BIOS setup)

In summary, switching the port of an IP-Power to ON while it was OFF starts the connected computer.

Note about Group 1 SynologyCluster. This cluster consists of 2 machines and each one has dual power supply to increase system reliability. This system uses 4 ports on its IP-Power. In addition, and unfortunately because SynologyCluster's features may vary with the rate of updates, starting the SynologyCluster requires manual action. Some revisions of the Linux kernel allowed remote booting with WakeOnLine technique, it is currently undefined whether this technique exists or not. So a person has to manually power on the SynologyCluster if it was just started with IP-Power.

How to test the accessibility of a machine

Group 1 (disk server: SynologyCluster) has the address 10.10.132.101, so 2 solutions, the best is with a browser:

access the URL: 10.10.132.101:5000

the other method consists in sending a ping on the machine followed (as soon as the ping answers) by an ssh to see the login prompt only:

ping 10.10.132.101 ... <CTRL>-C
ssh 10.10.132.101  ... <CTRL>-C

Group 2 (workstation: glslogin1)

a ping on the machine with its IP followed by an ssh and login under your name

ping 10.10.132.81 ... <CTRL>-C
ssh 10.10.132.81  ... <CTRL>-C

The answer of these 2 groups makes it possible to envisage the complete starting of the station.

All other computers follow the example of group 2.

The following order (only on glslogin1) allows the date command to be sent to all LCUs and therefore provides an easy and complete test:

T_show_date

LW 18/05/2020