Recovery Procedures

This guide provides step-by-step recovery procedures for various system failures in the RCR Common Robotics Platform.

Emergency Recovery

Immediate Actions

  1. Emergency Stop

    • Press physical emergency stop button

    • Turn off main power switch

    • Assess the situation safely

  2. Safety Check

    • Ensure robot is in safe location

    • Check for any damage

    • Verify no immediate hazards

  3. Documentation

    • Note what happened

    • Record error messages

    • Document system state

System Recovery Procedures

1. Complete System Restart

Procedure:

  1. Power off robot completely

  2. Wait 10 seconds

  3. Power on robot

  4. Launch system

  5. Verify all components

Commands:

# Power off
sudo shutdown -h now

# Power on and launch
ros2 launch common_platform launch_robot.launch.py

2. Software Recovery

Restart ROS2 Nodes:

# Kill all ROS2 nodes
ros2 daemon stop
ros2 daemon start

# Restart launch file
ros2 launch common_platform launch_robot.launch.py

Reset Configuration:

# Reset to default configuration
cp config/default/* config/

# Restart system
ros2 launch common_platform launch_robot.launch.py

3. Hardware Recovery

Check Connections:

  1. Power off robot

  2. Check all cable connections

  3. Verify power connections

  4. Test sensor connections

  5. Power on and test

Reset Hardware:

# Reset USB devices
sudo modprobe -r usb_storage
sudo modprobe usb_storage

# Reset I2C bus
sudo modprobe -r i2c_dev
sudo modprobe i2c_dev

Component-Specific Recovery

Power System Recovery

Battery Issues:

  1. Check battery voltage

  2. Verify charging system

  3. Test under load

  4. Replace if necessary

Power Supply Issues:

  1. Check power connections

  2. Verify voltage levels

  3. Test power distribution

  4. Replace faulty components

Motor System Recovery

Motor Driver Recovery:

  1. Check driver connections

  2. Verify power supply

  3. Test driver functionality

  4. Replace if necessary

Encoder Recovery:

  1. Check encoder wiring

  2. Verify encoder counts

  3. Test encoder resolution

  4. Replace if necessary

Sensor Recovery

LiDAR Recovery:

  1. Check USB connection

  2. Verify power supply

  3. Test serial communication

  4. Restart LiDAR node

Camera Recovery:

  1. Check USB connection

  2. Verify power supply

  3. Test video device

  4. Restart camera node

IMU Recovery:

  1. Check I2C connection

  2. Verify power supply

  3. Test I2C communication

  4. Restart IMU node

Data Recovery

Log Recovery

System Logs:

# Check system logs
journalctl -u ros2
journalctl -u robot

# Save important logs
journalctl -u ros2 > /tmp/ros2_recovery.log

ROS2 Logs:

# Check ROS2 logs
ros2 log list
ros2 log show

# Save ROS2 logs
ros2 log save /tmp/ros2_logs

Configuration Recovery

Backup Configuration:

# Backup current configuration
cp -r config/ config_backup_$(date +%Y%m%d_%H%M%S)/

# Restore from backup
cp -r config_backup_YYYYMMDD_HHMMSS/* config/

Reset to Defaults:

# Reset configuration
git checkout config/

Network Recovery

Communication Recovery

USB Recovery:

# Reset USB subsystem
sudo modprobe -r usb_storage
sudo modprobe usb_storage

# Check USB devices
lsusb

Serial Recovery:

# Check serial devices
ls /dev/tty*

# Reset serial ports
sudo chmod 666 /dev/ttyUSB0

Network Recovery:

# Restart network
sudo systemctl restart networking

# Check network status
ip addr show

Performance Recovery

System Performance

Resource Recovery:

# Check system resources
htop
df -h
free -h

# Clean up if needed
sudo apt autoremove
sudo apt autoclean

Process Recovery:

# Kill problematic processes
sudo pkill -f ros2
sudo pkill -f robot

# Restart system
ros2 launch common_platform launch_robot.launch.py

Memory Recovery

Memory Issues:

# Check memory usage
free -h

# Clear memory cache
sudo sync
sudo echo 3 > /proc/sys/vm/drop_caches

Recovery Verification

System Verification

Component Check:

# Check all nodes
ros2 node list

# Check all topics
ros2 topic list

# Check all services
ros2 service list

Functionality Test:

# Test sensors
ros2 topic echo /${ROS_NAME}/scan
ros2 topic echo /${ROS_NAME}/camera/image_raw
ros2 topic echo /${ROS_NAME}/imu/data

# Test motors
ros2 topic pub /cmd_vel geometry_msgs/msg/Twist "{}"

Performance Verification

System Performance:

# Check system load
uptime
htop

# Check disk usage
df -h

# Check network
ping google.com

Prevention

Regular Maintenance

Daily Checks:

  • Visual inspection

  • Check connections

  • Verify data quality

Weekly Checks:

  • System performance

  • Log analysis

  • Configuration backup

Monthly Checks:

  • Full system test

  • Component inspection

  • Performance review

Monitoring

System Monitoring:

  • Monitor system resources

  • Check for errors

  • Track performance

Health Checks:

  • Implement health checks

  • Monitor sensor data

  • Track system status

Recovery Documentation

Recovery Log

Document Recovery:

  • Date and time

  • What failed

  • Recovery steps taken

  • Results

  • Lessons learned

Update Procedures:

  • Update recovery procedures

  • Document new issues

  • Improve recovery process

For common issues, see Common Issues For error codes, see Error Codes For troubleshooting, see Troubleshooting