Linux Things

Which happened to be 50. And the machine just happened to have about 5GB of RAM. Well look there, 5×50 is 250GB

If a process tries to malloc() more memory than available it will get an error right away. This mode is set by changing the sysctl value vm.overcommit_memory to 2.

This tells the kernel to try and keep 64MB of RAM free at all times. It’s useful in two main cases:

• Swap-less machines, where you don’t want incoming network traffic to overwhelm the kernel and force an OOM before it has time to flush any buffers.
• x86 machines, for the same reason: the x86 architecture only allows DMA transfers below approximately 900MB of RAM. So you can end up with the bizarre situation of an OOM error with tons of RAM free.

vm.min_freekbytes 设定值高于LowTotal 值，系统认为没有足够的lowmem，而触发OOM Killer，将进程强行杀掉。

系统中内存分为lowmem和highmem，其中lowmem为寻址内存，当lowmem耗尽时，系统会触动OOM Killer将多余进程杀掉来释放内

It’s said that altering swappiness can help you when you’re running under high memory pressure with software that tries to do its own memory management (i.e. MySQL). We’ve had limited success with this and I’d much prefer to use software which doesn’t pretend to know more about your hardware than the OS (i.e. PostgreSQL). Not that I’m bitter.

The overcommit_memory sysctl isn’t something you’ll usually have to change if your software isn’t insane, but our netboot setup uses it so I thought I’d mention it. From the documentation:

1. - Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slighly more memory in this mode. This is the default.
2. - Always overcommit. Appropriate for some scientific applications.
3. - Don’t overcommit. The total address space commit for the system is not permitted to exceed swap + a configurable percentage (default is 50) of physical RAM. Depending on the percentage you use, in most situations this means a process will not be killed while accessing pages but will receive errors on memory allocation as appropriate.

For more info on this, see the overcommit accounting documentation. https://www.kernel.org/doc/Documentation/sysctl/vm.txt

通过修改proc系统的drop_{caches清理free的cache} $echo 3 > /proc/sys/vm/drop_caches

drop_{caches的详细文档如下：} Writing to this will cause the kernel to drop clean caches, dentries and inodes from memory, causing that memory to become free.

To free pagecache:
 echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
 echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
 echo 3 > /proc/sys/vm/drop_caches

As this is a non-destructive operation, and dirty objects are notfreeable, the user should run "sync" first in order to make sure allcached objects are freed. This tunable was added in 2.6.16.

修改/etc/sysctl.conf 添加如下选项后就不会内存持续增加

vm.dirty_ratio = 1
vm.dirty_background_ratio=1
vm.dirty_writeback_centisecs=2
vm.dirty_expire_centisecs=3
vm.drop_caches=3
vm.swappiness =100
vm.vfs_cache_pressure=163
vm.overcommit_memory=2
vm.lowmem_reserve_ratio=32 32 8
kern.maxvnodes=3

上面的设置比较粗暴，使cache的作用基本无法发挥。需要根据机器的状况进行适当的调节寻找最佳的折衷。

In the default configuration, then, data written to disk will sit in memory until either a) they're more than 30 seconds old, or b) thedirty pages have consumed more than 10% of the active, working memory.

/proc/sys/vm/dirty_ratio (default 40): Maximum percentage of total memory that can be filled with dirty pages before processes are forced to write dirty buffers themselves during their time slice instead of being allowed to do more writes.

Kernel Korner: I/O Schedulers: http://www.linuxjournal.com/article/6931

• dirty_{backgroundratio}: Primary tunable to adjust, probably downward. If your goal is to reduce the amount of data Linux keeps cached in memory, so that it writes it more consistently to the disk rather than in a batch, lowering dirty_{backgroundratio} is the most effective way to do that.
• dirty_ratio: Secondary tunable to adjust only for some workloads. Applications that can cope with their writes being blocked altogether might benefit from substantially lowering this value.
• dirty_{expirecentisecs}: Test lowering, but not to extremely low levels. Attempting to speed how long pages sit dirty in memory can be accomplished here, but this will considerably slow average I/O speed because of how much less efficient this is. This is particularly true on systems with slow physical I/O to disk.
• dirty_{writebackcentisecs}: Leave alone. The timing of pdflush threads set by this parameter is so complicated by rules in the kernel code for things like write congestion that adjusting this tunable is unlikely to cause any real effect.

By default, Linux will aggressively swap processes out of physical memory onto disk in order to keep the disk cache as large as possible. This means that pages that haven't been used recently will be pushed into swap long before the system even comes close to running out of memory, which is an unexpected behavior compared to some operating systems. The /proc/sys/vm/swappiness parameter controls how aggressive Linux is in this area.

• Neil Horman, "Understanding Virtual Memory in Red Hat Enterprise Linux 4" http://people.redhat.com/nhorman/papers/rhel4_vm.pdf
• Daniel P. Bovet and Marco Cesati, "Understanding the Linux Kernel, 3rd edition", chapter 15 "The Page Cache". Available on the web at http://www.linux-security.cn/ebooks/ulk3-html/
• Robert Love, "Linux Kernel Development, 2nd edition", chapter 15 "The Page Cache and Page Writeback"
• "Runtime Memory Management", http://tree.celinuxforum.org/CelfPubWiki/RuntimeMemoryMeasurement
• "Red Hat Enterprise Linux-Specific [Memory] Information", http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/admin-guide/s1-memory-rhlspec.html
• "Tuning Swapiness", http://kerneltrap.org/node/3000
• "FAQ Linux Memory Management", http://gentoo-wiki.com/FAQ_Linux_Memory_Management
• From the Linux kernel tree:
  • Documentation/filesystems/proc.txt (the meminfo documentation there originally from http://lwn.net/Articles/28345/)
  • Documentation/sysctl/vm.txt
  • Mm/page-writeback.c

• While not directly addressed here, the I/O scheduling algorithms in Linux actually handle the writes themselves, and some knowledge or tuning of them may be synergistic with adjusting the parameters here. Adjusting the scheduler only makes sense in the context where you've already configured the page cache flushing correctly for your workload.
• D. John Shakshober, "Choosing an I/O Scheduler for Red Hat Enterprise Linux 4 and the 2.6 Kernel" http://www.redhat.com/magazine/008jun05/features/schedulers/
• Robert Love, "Kernel Korner: I/O Schedulers", http://www.linuxjournal.com/article/6931
• Seelam, Romero, and Teller, "Enhancements to Linux I/O Scheduling", http://linux.inet.hr/files/ols2005/seelam-reprint.pdf
• Heger, D., Pratt, S., "Workload Dependent Performance Evaluation of the Linux 2.6 I/O Schedulers", http://linux.inet.hr/files/ols2004/pratt-reprint.pdf

1. Cpu it self is busy/overloaded in processing things
2. Processes (typically called Blocking process) in run queue, waiting for I/O

In fact, it is precisely the CPU load that is measured, because load averages do not include any processes or threads waiting on I/O, networking, databases or anything else not demanding the CPU. It narrowly focuses on what is actively demanding CPU time. This differs greatly from the CPU percentage. The CPU percentage is the amount of a time interval (that is, the sampling interval) that the system's processes were found to be active on the CPU. ²

The load averages differ from CPU percentage in two significant ways: 1) load averages measure the trend in CPU utilization not only an instantaneous snapshot, as does percentage, and 2) load averages include all demand for the CPU not only how much was active at the time of measurement.

When the quantum timer pops, timer.c is entered at a function in the kernel called timer.c:do_timer(). Here, all interrupts are disabled so the code is not working with moving targets. The jiffies counter is incremented by 1, and the load-average calculation is checked to see if it should be computed.

The calculation code uses the HZ value to determine the calculation frequency. Specifically, the timer.c:calc_load() function will run the averaging algorithm every 5 * HZ, or roughly every five seconds.

 unsigned long avenrun[3];

   static inline void calc_load(unsigned long ticks)
   {
      unsigned long active_tasks; /* fixed-point */
      static int count = LOAD_FREQ;

      count -= ticks;
      if (count < 0) {
         count += LOAD_FREQ;
         active_tasks = count_active_tasks();
         CALC_LOAD(avenrun[0], EXP_1, active_tasks);
         CALC_LOAD(avenrun[1], EXP_5, active_tasks);
         CALC_LOAD(avenrun[2], EXP_15, active_tasks);
      }
   }

extern unsigned long avenrun[]; /* Load averages */

   #define FSHIFT   11          /* nr of bits of precision */
   #define FIXED_1  (1<<FSHIFT) /* 1.0 as fixed-point */
   #define LOAD_FREQ (5*HZ)     /* 5 sec intervals */
   #define EXP_1  1884          /* 1/exp(5sec/1min) as fixed-point */
   #define EXP_5  2014          /* 1/exp(5sec/5min) */
   #define EXP_15 2037          /* 1/exp(5sec/15min) */

   #define CALC_LOAD(load,exp,n) \
      load *= exp; \
      load += n*(FIXED_1-exp); \
      load >>= FSHIFT;

The calc_load() function is called by update_times(), also found in timer.c, and is the code responsible for supplying the calc_load() function with the ticks parameter.

The macro takes in three parameters: the load-average bucket (one of the three elements in avenrun[]), a constant exponent and the number of running/uninterruptible processes currently on the run queue. The possible exponent constants are listed above: EXP_1 for the 1-minute average, EXP_5 for the 5-minute average and EXP_15 for the 15-minute average. The important point to notice is that the value decreases with age. The constants are magic numbers that are calculated by the mathematical function shown below:

When x=1, then y=1884; when x=5, then y=2014; and when x=15, then y=2037. The purpose of the magical numbers is that it allows the CALC_LOAD macro to use precision fixed-point representation of fractions. The magic numbers are then nothing more than multipliers used against the running load average to make it a moving average. (The mathematics of fixed-point representation)

load average: 0.09, 0.05, 0.01

the three numbers represent averages over progressively longer periods of time (one, five, and fifteen minute averages)

On multi-processor system, the load is relative to the number of processor cores available. The "100% utilization" mark is 1.00 on a single-core system, 2.00, on a dual-core, 4.00 on a quad-core, etc.

grep 'model name' /proc/cpuinfo | wc -l

https://scoutapp.com/

http://datatag.web.cern.ch/datatag/howto/tcp.html

http://kaivanov.blogspot.hk/2010/09/linux-tcp-tuning.html

http://www.hep.ucl.ac.uk/~ytl/tcpip/linux/txqueuelen/datatag-tcp/

http://fasterdata.es.net/host-tuning/linux/

http://www.acc.umu.se/~maswan/linux-netperf.txt

http://www.cyberciti.biz/faq/linux-tcp-tuning/

Writing to this file results in a flush of the routing cache.

You can check your video card model by running, from terminal:

lspci | grep VGA More information can be provided by running

sudo lshw -C video

After the system reboot, you can change your X settings by going to System->Administration->NVIDIA X Server Settings. If you installed the driver manually, it may be under Applications->System Tools->NVIDIA X Server Settings. Alternatively, run from terminal:

gksudo nvidia-settings

If the driver didn't install this (Envy/EnvyNG surely does), you can install it manually with

sudo apt-get install nvidia-settings

I prefer to uncheck Merge with existing file. Now click Save. If it gives you an error, "Unable to remove old X config backup file '/etc/X11/xorg.conf.backup' then click OK and open the save dialog again. Click Show preview…, select all and copy. Now open a terminal and run

=gksudo gedit /etc/X11/xorg.conf Delete the current contents of this file and paste what is in the preview. Save and close.

sudo apt-get remove --purge nvidia-glx-new

Failed to access the USB subsystem. Could not load the Host USB Proxy service: VERR_DISKFULL.

sudo VBoxManage extpack uninstall "Oracle VM VirtualBox Extension Pack"
sudo VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-xxxxvbox-extpack
#download from oracle

If do not resolve this issue, more:http://phobosk.wordpress.com/2012/11/11/howto-fix-a-virtualbox-failed-to-access-the-usb-subsystem-error/

http://sourceforge.net/projects/loic/

http://www.thc.org/thc-ssl-dos/

http://sourceforge.net/projects/xoic/

http://staff.washington.edu/dittrich/misc/ddos/

http://www.nmrc.org/pub/faq/hackfaq/hackfaq-05.html

• APF: Advanced Policy Firewall: http://www.rfxn.com/
• BFD: Brute Force Detection: http://www.rfxn.com/
• DDoS: DDoS Deflate: http://deflate.medialayer.com/
• Rootkit: Spy and Junkware detection and removal tool: http://www.rootkit.nl/projects/rootkit_hunter.html