Apache

During 2021 and 2022 a new Apache web hosting has been deployed for our customers. Sites will use the same service accounts and configurations with a few notable modifications. Over late 2022 and 2023 we will be working with our existing customers to migrate their websites to the new infrastructure.

Changes of note:

• Users can see their disk usage with the quota command when logged onto az1-apacheint-prod01.server.ufl.edu or other interactive login servers with SSH.
• Red Hat Enterprise Linux 8, current update
• default PHP 7.4 with patches
  • Red Hat has dropped SCL so there will no longer be calls to alternate PHP startup scripts.
• default python 3.9 with patches
  • Red Hat has dropped SCL so there will no longer be calls to alternate python startup scripts.
• DirectoryIndex index.cgi index.html index.html.var index.htm index.shtml
• Shibboleth default REMOTE_USER is in order of preference: ufid eppn glid persistent-id targeted-id, depending on which attributes are released to that SP.
• If your site is protected shibboleth matches on specific attributes, the .htaccess syntax will change in RHEL 8 / Apache 2.4, enforcing lower case for some keywords and adding shib-attr between the keyword require and the name of the attribute.
  • Old syntax: Require glid allan
  • New syntax: require shib-attr glid allan
  Note: there is no compatibility mode, so this change must be made in tandem with the actual migration and DNS change of the site.
• The new RHEL 8 interactive login server which has these software versions is named:
  • az1-apacheint-prod01.server.ufl.edu
  and should be used in place of any of the older glint servers. The glint servers will be retired after the migration from RHEL 6 is complete.

Details on usage:

File upload and access

To match the new environment for the web servers we have a new access node to upload/edit content. Please use ssh/scp/sftp to access your site's account on az1-apacheint-prod01.server.ufl.edu which has the same environment as the new web cluster.

How to Control Access with Shibboleth

If your site is configured with shibboleth, you can control access based on shibboleth logins. If your site is not yet configured with shibboleth, you'll need to start with the Identity Services Authentication for Web Based Services Setup Request page and fill in the form for a UF Service Provider.

How to transition from PHP with SCL

• If your site used one of the later version of PHP, e.g. 7.0, the new environment will try to read the SCL source and fail, causing a brief but noticeable timeout before it proceeds. To prepare for the transition off SCL, any site that uses PHP 7.0 can edit the CGI script to check and only run the SCL version if it exists.

This will be an edit in FDQN/cgi-bin/php.cgi and will require editing the definition of PHP_CGI, replace the source line with this new line:

test -e /opt/rh/rh-php70/enable && source /opt/rh/rh-php70/enable

If your exec ... line already has options on it they can likely stay the same. For a nearly-stock PHP configuration (just change version) the whole file (~/FDQN/cgi-bin/php.cgi) should be:

#!/bin/bash
test -e /opt/rh/rh-php70/enable && source /opt/rh/rh-php70/enable
PHP_CGI=php-cgi
exec $PHP_CGI

Note: If your PHP scripts were running a version before 7.0, you'll need to test the scripts with the new version 7.4 when it is migrated to the new hosting system.

How to run CGI with python 3.9

• Currently python 3.9 is provided by RHEL 8, any site that uses older python 3.6 will need to test as the site is migrated. The path for the latest python 3.x is /usr/bin/python3

How to install wp-cli for WordPress

Users who maintain their own WordPress sites and are comfortable using the linux command line may be interested in using the wp-cli tool. We have notes on installing wp-cli in your service account. UFIT does not support this tool, we merely offer guidance on local installation.

During 2015 a new Apache web hosting has been deployed for our customers. New accounts will be created on the new environment. Furthermore, over the next few months we will be working with our existing customers to migrate their websites to the new infrastructure.

As much as possible for new accounts and migrations to the new environment we will be creating one user account to host each web site. This is a change from the past to increase security and cut down on leakage between sites and keep test site issues from leaking to production web sites.

Changes of note:

• Users can see their disk usage with the quota command when logged onto glint-prod06.osg.ufl.edu or other interactive login servers with SSH.
• Red Hat Enterprise Linux 6, current update
• default PHP 5.3.3 with patches
• optional PHP 7.0 (currently 7.0.27) with patches
  • PHP 7.0 is provided by Red Hat SCL, any site that uses PHP 7.0 will need to migrate (new PHP version or some future Apache Hosting version). PHP 7.0 is the last version available for RHEL 6.
• optional python 3.6 (currently 3.6.3) with patches
  • python 3.6 is provided by Red Hat SCL, any site that uses python 3.6 will need to migrate (new python version or some future Apache Hosting version) by October 2020
• DirectoryIndex index.cgi index.html index.html.var index.htm index.shtml
• Shibboleth default REMOTE_USER is UFID (rather than EPPN or GatorLink username) if the SP has that Attribute

Details on usage:

File upload and access

To match the new environment for the web servers we have a new access node to upload/edit content. Please use ssh/scp/sftp to access your site's account on glint-prod06.osg.ufl.edu which has the same environment as the new web cluster.

How to Control Access with Shibboleth

If your site is configured with shibboleth, you can control access based on shibboleth logins. If your site is not yet configured with shibboleth, you'll need to start with the Identity Services Authentication for Web Based Services – Setup Request page and fill in the form for a UF Service Provider.

How to run CGI with PHP 7.0.27 (instead of PHP 5.3.3)

• PHP 7.0 is provided by Red Hat SCL, any site that uses PHP 7.0 will need to migrate (new PHP version or some future Apache Hosting version) when the support lifecycle of this SCL ends in 2019 November

This will be an edit in FDQN/cgi-bin/php.cgi and will require editing the definition of PHP_CGI, replace the whole line with two new lines:

source /opt/rh/rh-php70/enable
PHP_CGI=php-cgi

If your exec ... line already has options on it they can likely stay the same. For a stock PHP configuration (just change version) the whole file (~/FDQN/cgi-bin/php.cgi) should be:

#!/bin/bash
source /opt/rh/rh-php70/enable
PHP_CGI=php-cgi
exec $PHP_CGI

Note Well: This change will cause any site on the 2010 hosting system to stop running PHP scripts. Do not make this change unless your site has been created or migrated to the new hosting system.

How to run CGI with python 3.6.3 (instead of python 2.6.6)

• python 3.6 is provided by Red Hat SCL, any site that uses python 3.6 will need to migrate (new python version or some future Apache Hosting version) by October 2020

To use Python 3.6 in a cgi script make sure the mode as execute and read set for the owner (your user account), the file name ends in ".cgi", and that the first line of the script is:

#!/usr/bin/scl enable rh-python36 -- python3

Background

Some websites use web content management software that generates logs. Those logs need to be managed to prevent them from growing to consume all allotted resources. We will discuss by way of example a WordPress site with a plugin that creates log files, and how to manage them.

File locations will be relative to the DocumentRoot directory, starting with htdocs/ which is in the service account under a directory named for the website. All references that start with a leading slash e.g. /h/cnswww-t01.wp.it/ will require you to substitute your own service account and website name.

Example

The popular WordPress security plugin sucuri-scanner creates log in the service account, as well as in a cloud place for recovery after a site incident. The log file htdocs/wp-content/uploads/sucuri/sucuri-oldfailedlogins.php occasionally grows without bounds, and can consume 100% of the storage quota for a site.

Web Services submitted a ticket asking for assistance in creating a log rotation process to handle the log file automatically. Since we had a recent incident of a site running out of quota, we had sample files to work with.

Cron

The cron process runs jobs based on time, and is a handy way to make something happen repeatedly. Each account can have its own cron jobs, so you can create repeating processes for your website. Cron 'jobs' are specified in a format that starts with the times to run, and then describes what action to take.

Logrotate

Logrotate can be run as an individual service account to work on files in the service account's home directory, which allows the user to manage its own logs. Requirements for the logrotate include specifying what file(s) to consider, what metrics to use, and where to write the temporary file.

Configuration File

To specify the parameters for this logrotate job, we created a configuration file in the top level of the service account, outside the document root (htdocs/) of the web site so that it isn't displayed on the web. This file should be created as text (not MS Word) and preferably on a linux host so it doesn't include Windows return characters.

We are currently looking only at the specific log file which has caused quota use to reach 100% in other sites: htdocs/wp-content/uploads/sucuri/sucuri-oldfailedlogins.php We are using full paths to all files so the cron job doesn't have to interpolate any data. This is the file for the Test hosted WordPress instance:

# cat /h/cnswww-t01.wp.it/sucuri-logrotate.conf
/h/cnswww-t01.wp.it/t01.wp.it.ufl.edu/htdocs/wp-content/uploads/sucuri/sucuri-oldfailedlogins.php {
rotate 10
missingok
dateext
dateformat _%Y%m%d-%s
compress
size 1G
}

These directives should retain 10 versions of the rotated log, names with a date instead of .1, .2, etc., and using a seconds counter so that multiple rotations in the same date do not conflict. The rotated log will be compressed (with gzip) and the trigger to perform a rotation is a file size of 1 GB or greater. We also specify that it's OK if the log file is missing, so that we don't get error messages if you manually remove the file, or it was recently rolled by logrotate and not yet re-created by sucuri.

Temporary File

If not specified, logrotate attempts to write to /var/tmp/logrotate.status, which is the location used by the system logrotate processes. To avoid conflict, we created an empty file in the website's service account home directory (a.k.a. ~/) with the unix command touch:

touch ~/logrotate.status
ls -l ~/logrotate.status
-rw-r-----. 1 cnswww-t01.wp.it cnswww-t01.wp.it 0 /h/cnswww-t01.wp.it/logrotate.status

The Cron Job

As the canonical location for service account cron jobs, we created crontab entries on glint-prod06.osg.ufl.edu as the service accounts. Note that best practice is to spell out the entire path of the home directory, do not use shorthand like ~/ or variables like $HOME since the cron process is not a regular user login. For the Test instance, we added these lines:

# hourly check & fix of expanding sucuri-oldfailedlogins.php file
55 * * * * /usr/sbin/logrotate -s /h/cnswww-t01.wp.it/logrotate.status /h/cnswww-t01.wp.it/sucuri-logrotate.conf

That will run logrotate with the configuration file and temporary file we created at :55 minutes after every hour, every day of every month. Log growth in previously affected sites indicates that the log growth was less than 1 GB per hour, but could be several gigabytes per day, so hourly checks seem reasonable.

Note: cron jobs do not invoke any login shell scripts, so using the complete path of any command or file is always appropriate and frequently necessary. When in doubt, you can use the which command to find the full path of a command like logrotate for use in the cron job.

If your site is configured for shibboleth, you can control access to directories and paths by editing the .htaccess file at the top level of the site or in a relevant subdirectory. Trivial access to any UF account holder can be granted, or more granular access may be granted based on the attributes returned in the shibboleth assertion.

Changes in apache 2.4 which comes with RHEL 8 include requiring lower case for several keywords which used to accept mixed case. In particular, anything after the word require should be lower case.

Basic Access

The most trivial access is to allow anyone who can authenticate to shibboleth Identity Provider (IdP) access to the content. This can be accomplished with as few as these lines:

AuthType shibboleth
ShibRequireSession on
Require valid-userThat allows any user who can log into the UF IdP at login.ufl.edu to access the content.

Shibboleth Attributes

Know Your Attributes

Identity Services approves a list of attributes to be released to each SP Group, so what choices you have depend on what was approved for your site. Registered contacts for an SP Group can view the list of attributes on the UF Shibboleth SP Registry page: https://admin.ict.it.ufl.edu/sp/sp.cgi. You can also use a CGI script to print the environment variables, and if it is in a protected location then the shibboleth attributes will be included in the variable list. An example of a printenv.cgi written in Perl can be placed in your site to help you determine what attributes are available, and what they are called.

Attribute-Based Access

Once you know your available attributes, you can use one or more to determine access to the site, e.g. limiting to specific users or groups of users. A simple example which requires a user to match more one of a possible list of users is:

AuthType shibboleth
ShibRequireSession on
require shib-attr eppn ~ "^(asr|allan)@ufl.edu$"Those lines require the user match either asr@ufl.edu or allan@ufl.edu to gain access.

A complex example which requires a user to match more than one attribute might look like this:

AuthType shibboleth
ShibRequireSession on
ShibRequireAll On
require shib-attr departmentNumber ~ "14300000"
require shib-attr UFADGroupsDN ~ "EIO-SYSTEMS-LIN"Those lines require the user match all, rather than any, of the patterns, and that the user both be in a specific department and had a specific UF Level of Assurance.

Attribute Variable Names

Depending on whether you are using the apache .htaccess file or other software to refer to the attribute variables, you need to know some specific things about UFIT apache hosting.

• Shibboleth standard attribute "eduPersonPrincipalName" is shortened to "eppn"
• Shibboleth default REMOTE_USER is UFID (rather than eppn or GatorLink username) if the SP has that Attribute. The value used is the first attribute that is released to the SP in this list "ufid, eppn, glid, persistent-id, targeted-id".
• Shibboleth generated environment variables are prefixed with "UFShib_" for security (and may also have a "REDIRECT_" prefix depending on how you access scripts, usually PHP scripts will have the prefix and others will not, this may change if you use .htaccess configuration). Example, to find the value of the "ufid" attribute check the environment for one of the following variables and read the value of the first one found:UFShib_ufid
REDIRECT_UFShib_ufid
• Please verify you do not have "ShibUseHeaders on" set in any .htaccess files, as this setting compromises security. A visible side effect of enabling this is the addition of HTTP_ in front of variable names.

Complex Nested Access Control

In apache 2.4 you can make more complex, nested access control stanzas in the .htaccess file. Start by requiring any of the following, with a list of equally allowable options. Then nest requiring all of the subset to enforce shibboleth and one or more specific attributes. An example of a nested check for client IP with fall-back to shibboleth authentication is:

You can build up a test nested rstriction in a subdirectory and/or test site before you eploy it, to check that the allowed options all get in, and things not in the allowed list do not.

If your site is configured for shibboleth, you can control access to directories and paths by editing the .htaccess file at the top level of the site or in a relevant subdirectory. Trivial access to any UF account holder can be granted, or more granular access may be granted based on the attributes returned in the shibboleth assertion.

Basic Access

The most trivial access is to allow anyone who can authenticate to shibboleth Identity Provider (IdP) access to the content. This can be accomplished with as few as these lines:

AuthType Shibboleth
ShibRequireSession on
Require valid-userThat allows any user who can log into the UF IdP at login.ufl.edu to access the content.

Shibboleth Attributes

Know Your Attributes

Identity Services approves a list of attributes to be released to each SP Group, so what choices you have depend on what was approved for your site. Registered contacts for an SP Group can view the list of attributes on the Service Provider (SP) Helpdesk page: https://sp.helpdesk.ufl.edu/. You can also use a CGI script to print the environment variables, and if it is in a protected location then the shibboleth attributes will be included in the variable list. An example of a printenv.cgi written in Perl can be placed in your site to help you determine what attributes are available, and what they are called.

Attribute-Based Access

Once you know your available attributes, you can use one or more to determine access to the site, e.g. limiting to specific users or groups of users. A simple example which requires a user to match more one of a possible list of users is:

AuthType Shibboleth
ShibRequireSession on
Require eppn ~ "^(namsirc|allan)@ufl.edu$"Those lines require the user match either namsirc@ufl.edu or allan@ufl.edu to gain access.

A complex example which requires a user to match more than one attribute might look like this:

AuthType Shibboleth
ShibRequireSession on
ShibRequireAll On
Require departmentNumber ~ "14300000"
Require ufloa ~ "(Silver|UFM)"Those lines require the user match all, rather than any, of the patterns, and that the user both be in a specific department and had a specific UF Level of Assurance.

Attribute Variable Names

Depending on whether you are using the apache .htaccess file or other software to refer to the attribute variables, you need to know some specific things about UFIT apache hosting.

• Shibboleth standard attribute "eduPersonPrincipalName" is shortened to "eppn"
• Shibboleth default REMOTE_USER is UFID (rather than eppn or GatorLink username) if the SP has that Attribute. The value used is the first attribute that is released to the SP in this list "ufid, eppn, glid, persistent-id, targeted-id".
• Shibboleth generated environment variables are prefixed with "UFShib_" for security (and may also have a "REDIRECT_" prefix depending on how you access scripts, usually PHP scripts will have the prefix and others will not, this may change if you use .htaccess configuration). Example, to find the value of the "ufid" attribute check the environment for one of the following variables and read the value of the first one found:UFShib_ufid
REDIRECT_UFShib_ufid
• Please verify you do not have "ShibUseHeaders on" set in any .htaccess files, as this setting compromises security. A visible side effect of enabling this is the addition of HTTP_ in front of variable names.

Once your website has been created you will recieve communication that provides you with your new username and a request to provide SSH keys for access to the account. Using your new credentials you can login to glint-prod06.osg.ufl.edu with one of the secure shell protocols: ssh/scp/sftp Because glint is on private IP, you will need to be logged in to the UF VPN or be on a host in the UF network. If you're unfamiliar with these we recommend looking into WinSCP as it seems to work well for other customers. If you're on a Mac or Linux ssh/scp utilities are available on the command line.

You can also use ssh for shell access to glint in order to edit files with utilities in the linux shell.

f you don't have an email that provides you with instructions and a request to provide SSH keys for access to the account, you can download the How-To document on creating and using SSH Keys with PuTTY on a PC. If you're on a Mac or Linux, ssh-keygen utilities are available on the command line.

This is one of the most common questions we get and it has to do with our CGI environment and how smtp.ufl.edu handles email. There's a short answer to this problem and a longer background story, so here goes...

Short answer - you need to have your mail function specify the envelope sender address (not the header from address) to the mail submission program (sendmail). When you do this be sure to use one of (1) a valid gatorlink address or (2) an address outside of the @ufl.edu namespace. If you would be so kind as to use a deliverable address we'd appreciate it because it will prevent our postmaster from receiving bounces.

To specify the envelope sender address with php's mail() function use the "additional_parameters" argument to mail(). See example #3 at http://php.net/manual/en/function.mail.php

Long story - In our web hosting environment when you submit mail by default your code will do so as whatever system user is executing the CGI code. For most users this will be the cnswww- prefixed username we've given you to manage your website- let's say "cnswww-you". So by default sendmail gets a messages submitted by cnswww-you and adds that id to the message's envelope with the assumed @ufl.edu domain. Sendmail then hands the message off to smtp.ufl.edu which sees the envelope sender address as cnswww-you@ufl.edu. To cut down on fraudulent email passing through smtp.ufl.edu it is configured to verify any @ufl.edu addresses that are attempting to send messages. So when smtp.ufl.edu tries to lookup cnswww-you more than likely that's not a valid Gatorlink mailbox, so smtp.ufl.edu will reject the message.

So to prevent this rejection from occurring you can manually set the envelope sender address (using the -f option to sendmail) to be either a valid Gatorlink mail address or some address that is deliverable on another system outside of @ufl.edu. There is another option: if you like, UFIT can setup an email alias for cnswww-you@ufl.edu and have mail to that address delivered to some real address of your choosing. This will satisfy smtp.ufl.edu for purposes of verifying the sender's address.

If you're not sure what is meant by the "envelope sender address" or how that differs from a message's "header from address" see the "Envelopes and Bodies" section on http://www.sendmail.org/resources

Customers may notice PHP sessions do not work as expected or may see a PHP warning "Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/var/lib/php/session)".

Since UFIT apache web hosting is configured in a cluster, the default method of storing PHP session info to local disk is not advisable. PHP sessions should be stored in the database (preferred) or stored in a non-web-accessible location such as

/h/<cnswww-user>/<website>/sessions/

This is accomplished by appending "-d session.save_path=<path>" to the end of the "exec" line in

/h/<cnswww-user>/<website>/cgi-bin/php.cgi

The full line would look similar to:

exec $PHP_CGI -d session.save_path=/h/<cnswww-user>/<website>/sessions/

Note: sites configured with fastcgi will require UFIT to modify the configuration to change the session save path.

Customers often want to increase the size limit on WordPress file uploads from the default 2MB.

Since WordPress is a PHP application, the easiest place to make the change is in the command line that starts PHP.

This is accomplished by appending (for example) "-d upload_max_filesize=20M -d post_max_size=20M -d memory_limit=64M" to the end of the "exec" line in

/h/<cnswww-user>/<website>/cgi-bin/php-cgi

The full line would look similar to:

exec $PHP_CGI -d upload_max_filesize=20M -d post_max_size=20M -d memory_limit=64M

While customers cannot edit the server-wide php.ini file, most PHP configuration options can be specified on the command line. This is accomplished by appending multiple "-d <option>" to the end of the "exec" line in

/h/<cnswww-user>/<website>/cgi-bin/php.cgi

Common options include the file upload size or maximum post size. Increasing these settings to 24 MB and 12 MB would look similar to:

exec $PHP_CGI -d upload_max_filesize=24M -d post_max_size=12M

Note: sites configured with fastcgi will require UFIT to modify php options such as upload_max_filesize.

If the perl library is included as part of the RHEL6 platform, you may simply contact us via Remedy ticket and ask us to configure the package for our web clusters. If the package could conflict with the default perl, or might otherwise be non-standard, you may wish to install a local perl or perl libraries in your document root.

When uploading files via glint it is likely that the files will be given the permission of "world readable". If you are uploading files that are executed in the cgi environment (e.g. PHP, Perl, Python, and many content management systems such as Wordpress, Drupal, etc.), you likely want to restrict permissions so only your hosting account can read the files. This is especially true of site configuration files that hold database credentials.

The following directory listing shows a world readable file (in Unix terminology, "other" users have "read" permission as indicated by the "r" in the third / rightmost group of columns):

$ ls -al
total 8
drwxr-x--- 2 cnswww-user cnswww-user 80 Jul 17 15:29 .
drwxr-xr-x 5 cnswww-user cnswww-user 1024 Jul 16 15:26 ..
-rw-r--r-- 1 cnswww-user cnswww-user 0 Jul 17 15:29 myfile

To make a file so it is only accessible by the owner:

$ chmod 600 myfile

which in this case is the same as removing "read" permission from "group" and "other":

$ chmod go-r myfile

After the change, the directory now looks like the following:

$ ls -al
total 8
drwxr-x--- 2 cnswww-user cnswww-user 80 Jul 17 15:29 .
drwxr-xr-x 5 cnswww-user cnswww-user 1024 Jul 16 15:26 ..
-rw------- 1 cnswww-user cnswww-user 0 Jul 17 15:29 myfile

It is also a best practice to keep your site configuration files outside of the htdocs / document root so they cannot be accidentally served up to the web. Some Content Management Systems allow this.

NOTE: Do not do a global permission change to 600 on all files under your htdocs. Statically served content (e.g. in most cases .html, images and CSS files) must still be readable by the apache user.

Edit the php.cgi file in your document root:

Before:

#!/bin/bash
PHP_CGI=/usr/bin/php-cgi
exec $PHP_CGI

After:

#!/bin/bash
PHP_CGI=/usr/bin/php-cgi
exec $PHP_CGI -d error_log=/h/cnswww-user/error.txt -d display_errors=On -d error_reporting=2147483647

(change user to your username).

The last version of PHP for RHEL 6 is 7.0.27, and you can enable it using the instructions on RHEL6 notes page. The next version of hosted apache infrastructure will have newer versions PHP and other software.

For a hosted apache site (vhost) which already exists, these are the normal steps:

1. UFIT: Create a certificate with all of the names the site uses
2. UFIT: Change the DNS for the site to an IP address which is configured to allow both ports 80 & 443 at the load balancer
3. UFIT: Apply the new certificate without enforcing HTTPS
4. Owner: Site owners test the HTTPS interface, checking for content mismatch errors, and fixing them
5. UFIT: Enforce HTTPS with a Redirect to the canonical ServerName
6. Both: Clean up any surprises, or revert the HTTPS enforcement

Normally this is a 1-2 day process that requires coordination from the site developer(s) and tester(s). Please start the process by opening a myIT ticket under Infrastructure -> Hosting -> Hosted Websites - Apache and asking to enable HTTPS for your site.

A WordPress update in 2020 caused many sites to lose their shibboleth exclude line(s) from the .htaccess file. An example of a .htaccess code block for WordPress with an exclusion for the shibboleth pseudo-path:

# Exclude Shibboleth.sso BEFORE the WordPress block
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_URI} ^/Shibboleth.sso(.*)$
RewriteRule ^.*$ - [L]
</IfModule>

Paste that stanza into the top level .htaccess file before the WordPress stanza to allow shibboleth.