DRS startup

DRS crate assembly

• Backplane: The DRS crate comes without the backplane. It has to be screwed to the crate. When installing the backplane make sure about connecting power to proper buses. When connected, install the power supply and fan unit. Test it with power. Check for no power-related failures.
  
• Install support modules: J110, J111, J112
  
• J111 - gets trigger signal, eventID number, clear signal. Gets busy signals from J110 and J112, performs logical OR on them and sends back busy signal to the central trigger. External clock should be provided to this module. All signals are distributed over the backplane.
  
• J110 - performs logical OR from busy signals sent by DRS boards by the left side of the backplane and outputs the result to module J111. Gets sync signal for global timing synchronization and spreads it to backplane.
  
• J112 - performs logical OR from busy signals sent by DRS boards by the right side of the backplane and outputs the result to module J111.
  

Installation of DRS boards and tests

• Install DRS boards, remember about setting them proper boardID on rotary switches and configuring the jumpers (by default, no jumper should be set on the board, for more custom solutions use jumpers J7 and J8)
  
• If none of J7 and J8 jumpers set, connect clock signal to J111. If needed, connect sync to J110.
  
• Connect four LEMO cables with trigger/eventID/clear/busy signals to J111. Optionally, install Trigger Box to the same crate and provide LVDS signal from central trigger. (note 1: Trigger Box works correctly when two green LEDs shine; note 2: In case of TOF crates for DRS there is only one Trigger Box per two racks, so LEMO signals should be split by NIM fan-in fan-out modules; in case of L0 trigger signal, the splitting cables should be as short as possible; note 3: If everything is connected properly, then at power up J111 all green LEDs should be off and at least one red should be on).
  

Start of DRS readout machines

• After the boot, check the number of USB ports using command

  lsusb | grep Renesas

  . If you see 4 entries (ports), the job is done and you can switch the procedure to another machine. If not...
  
• Try rebooting the machine at least up to 5 times. After each try, check the number of ports. My experience is that the number of ports after each reboot can be pretty random, but if you can see maximum of two, then the USB expansion card might be broken and need replacement while if you can see maximum three then it is almost certain that at some point you will get four ports.
  
• If after many reboots there is still no 4 ports, the machine should be cooled. My another experience is that it is much easier to get 4 ports when the machine is not warmed up. There is a good method to bring the machine to TDAQ room which is intensively air-conditioned to 20 degrees and keep it there for at least several hours (one day is optimal).
  
• If, after all tries, you cannot get 4 ports, replace the expansion card.
  

Connecting DRS boards to their readout machines

• If the LEDs are off, then the board is most probably not powered.
  
• If the LEDs are on in a different way than expected, then the firmware version in a given board might be different.
  

• BPD1
  
• BPD2 (Sandro's crate, Sandro's backplane, our power supply, Sandro's power supply is a "hostage" in Wiener lab in Meyrin)
  
• BPD3
  
• Trigger (located next to BPD3, Sandro's crate backplane, support modules)
  
• PSD (older version of backplane and module J110)
  
• TOF-L 1-4
  
• TOF-R 1
  
• TOF-R 2 (crate originally from BPD2, no power supply)
  
• TOF-R 3 - no crate
  
• TOF-R 4 - no crate
  

• Move BPD1 crate to TOF-R 3
  
• Move Sandro's crate at BPD2 to TOF-R 4
  
• Move new Kontron crate to BPD2
  
• Move three new Kontron crates to Counting House
  

• I accidentally ripped off capacitor C483 from board SN195. It should be checked what is it responsible for.
  
• Board #07 cannot be programmed
  
• Boards SN122 and SN184 do not turn on
  

• Fix problem with off-time rejection not working correctly
  
• Protection against event ID being duplicated
  

• bpd1-03 - three ports (board x103 cannot be contacted)
  
• psd4 - two ports (board x50d cannot be contacted)
  
• tof1-01 - two ports (x407 cannot be contacted, but one can consider plugging it to other machine)
  
• tof2-03 - two ports (x411 cannot be contacted, but one can consider plugging it to other machine)
  

• For many boards in TOF crates (except TOF1) the data channels showed strange waveforms.
  
• Several boards had strange waveforms on channel 9.
  
• One board had all channels from one DRS chip dead.
  

• When turning on the DRS crate, the board boots up very slowly. It starts booting only after all other boards finish their boot.
  
• When stopping the run, the communication with the board breaks and lsusb for that board gets crazy.
  
• When performing run, the computer with that board connected quickly increases hanging event fragments.
  

• Restarted supervisor process on bpd1-01 computer -- didn't help
  
• Restarted BPD1 DRS crate -- didn't help
  
• Restarted bpd1-01 machine -- after restart the machine didn't see 4 USB ports, so I kept restarting it, after several unsuccessful attempts I decided to turn it off and asked shifters to take data without BPD1 and BPD2. Oskar turned on the machine next morning, we could see 4 USB ports.
  
• Recompiled pterodaqtyl on supervisor (compilation contained fix for the problem in previous entry)
  
• Changed back the config to the nominal margin of zero-suppression
  

• Change margins of zero-suppression.
  
• Run data taking from pterodaqtyl installation in /soft/drs.
  
• Debug and localize the line in the code where the crash happens.
  

• Change the version of PteroDAQtyl used by PSD to the one used by other systems
  
• Cooperate with Tobiasz towards a web interface to remotely restart Wiener crates
  
• What is the status of broken boards sent back to Sandro?
  
• What is the status of clock generators from Sandro?
  

• Upload the newest firmware to the boards in PSD - on-site
  
• Test companion modules delivered from Pittsburgh
  
  • J111 - bring few DRS boards, plug in J111, connect via Quartus, check if we can see/program connected DRS boards
    
  • J110 - what is the exact role(s) of it?
    
  • J112 - what is the exact role(s) of it?
    
  • put modules to their crates
    
• Check power of BPD2/3 racks and crates
  
• Did they make any kind of doors to ToF racks?
  
• Pull cables for clock signal
  
• Prepare some kind of local Grafana monitoring for the condition of the boards (data taking speed, lost events, etc.)
  

• Make sure that all data taking using DRS is stable during longer runs
  
• Prepare a manual for shifters (e.g. what to do when some things are not working)
  

• 5 80-meter cables to BPD1 marked as "DRSBPD1 0[1234]" and "DRSBPD1 CTRL"
  
• 5 70-meter cables to BPD2 marked as "DRSBPD1 0[1234]" and "DRSBPD2 CTRL"
  
• 5 70-meter cables to BPD3 marked as "DRSBPD3 0[1234]" and "DRSBPD3 CTRL"
  

• Wiener crates have currents and voltages fluctuating
  
• After several thousand of events, the DRSEventBuilder stops saving files (need to change the naming convention of the output file and the way how they are being created
  
• The internal oscillator on new DRS boards has some glitches. One can see strange sine waves on the control plots.
  

ssh -J drs@na61-tdaq-gwX drs@detectornumber

source /soft/drs/venv3/bin/activate

cd /soft/drs/pteroDRS/
supervisord -c supervisor.conf -l supervisor_$(hostname).log

cd /build/scripts/Control/
python daqinterface.py config_script_here.json
tail -f module_timestamp_here.log

load

config

start

stop

start

unconfig

exit

load -> config -> start -> datadatadatadata -> stop -> unconfig -> exit

./fwload_fx3 cyfxflashprog.img

./download_fx3 -t I2C -i second.img

su
apt-get install sudo
nano /etc/sudoers
#root ALL=(ALL:ALL) ALL
na61drs ALL=(ALL) NOPASSWD:ALL
na61tdaq ALL=(ALL) NOPASSWD:ALL

cd /opt/
sudo apt-get install libssl-dev libusb-dev libusb-1.0-0 libusb-1.0-0-dev pkg-config net-tools vim tmux

cd /etc/udev/rules.d
sudo touch 50-usb-drs-rule.rules
nano 50-usb-drs-rule.rules
SUBSYSTEM=="usb", ATTRS{idVendor}=="04b4", ATTRS{idProduct}=="00f1", MODE="0666"
sudo udevadm control --reload-rules

sudo mkdir cmake
sudo chown na61drs:na61drs cmake/
cd cmake/ 
sudo apt-get install git cmake g++
wget https://github.com/Kitware/CMake/releases/download/v3.21.0-rc2/cmake-3.21.0-rc2.tar.gz
tar -xvzf cmake-3.21.0-rc2.tar.gz 
cd cmake-3.21.0-rc2/
./bootstrap
make -j4
sudo make install

cd /opt/
sudo mkdir zmq
sudo chown na61drs:na61drs zmq/
cd zmq/
git clone https://github.com/zeromq/libzmq.git
cd libzmq/
mkdir build
cd build
cmake ..
make -j4
sudo make install

cd ../..
git clone https://github.com/zeromq/cppzmq.git
cd cppzmq/
mkdir build
cd build
cmake ..
make -j4
sudo make install

sudo apt-get install supervisor
sudo apt-get install python-pip
sudo pip install supervisor_twiddler
sudo nano /etc/supervisor/supervisord.conf
	[inet_http_server]
	port=*:9001

	[rpcinterface:twiddler]
	supervisor.rpcinterface_factory = supervisor_twiddler.rpcinterface:make_twiddler_rpcinterface

	[group:daq]
sudo systemctl restart supervisor
sudo systemctl status supervisor

cd
git clone https://gitlab.cern.ch/na61-software/online/pterodaqtyl/pterodaqtyl.git
cd pterodaqtyl/
git submodule init src/Modules/drsmodules
git submodule update src/Modules/drsmodules
cd src/Modules/drsmodules
git checkout master
git pull
cd ../../..
git submodule init src/Utilities/drsutilities
git submodule update src/Utilities/drsutilities
git checkout master
git pull
cd ../../..
mkdir build
cd build
cmake ..
make -j4

sudo apt-get install python3-pip
sudo pip3 install jsonschema
sudo pip3 install zmq

• Trying other machine
  
• Trying other PCI-E card (one was burned)
  
• Trying many options in BIOS