Affiliate Disclosure: This blog may contain affiliate links, which means I may earn a small commission if you click on the link and make a purchase. This comes at no additional cost to you. I only recommend products or services that I believe will add value to my readers. Your support helps keep this blog running and allows me to continue providing you with quality content. Thank you for your support!
Welcome to Microsoft Visual FoxPro. Visual FoxPro is the object-oriented relational database management system that makes it possible for you to create database solutions for the desktop to the Web. Visual FoxPro provides powerful data handling capabilities, rapid application development tools for maximum productivity, and the flexibility needed to build all types of database solutions.
Guidelines for Using Visual FoxPro Foundation Classes
The Visual FoxPro .vcx visual class libraries located in the Ffc folder contain a variety of foundation classes for enhancing your Visual FoxPro applications with little or no programming. You can freely distribute the foundation classes with your applications. These foundation classes are contained in the Component Gallery. The Component Gallery provides a quick and easy way to learn more about the properties, events, and methods of each of the foundation classes.
You can also open up a foundation class in the Class Designer or Class Browser to see its structure and code. This is a great way to learn how the foundation class works as well as offering excellent insights into programming with Visual FoxPro.
The following guidelines provide information about how you can add the Visual FoxPro foundation classes to your applications.
Class Types
You need to know the Visual FoxPro base class of a foundation class before you can add the foundation class to your application. Certain foundation classes can be used only as visual objects on forms, while others are non-visual and can be run programmatically without being placed on a form. The Visual FoxPro Foundation Classes documentation indicates the base class of each foundation class so you can determine if the foundation class can be added to a form, or run programmatically in your application. Note that in the Component Gallery you can right-click a foundation class to display a shortcut menu. Choose Properties from the shortcut menu, and then choose the Class tab to display the base class.
The following table lists the Visual FoxPro base classes and how they can be added to your applications.
Category A – base classes that can be dropped onto a form
Category B – base classes that can be dropped onto a form or run programmatically in your application
Category C – base classes that can only be run programmatically in your application
Checkbox
Custom
Form
Combobox
Container
Formset
Commandbutton
Timer
Toolbar
Commandgroup
ProjectHook
Editbox
ActiveDoc
Grid
Hyperlink
Image
Label
Line
Listbox
OLE Control
Optionbutton
Optiongroup
Shape
Spinner
Textbox
Adding Foundation Classes to Forms
You will most often add foundation classes to forms. You can drag and drop foundation classes from the Component Gallery, Class Browser, Project Manager, and the Forms Control toolbar onto forms.
Note You can select a foundation class you’ve added to a form and then choose Class Browser from the Tools menu to display more information about the foundation class.
Component Gallery The Component Gallery provides the easiest way to add foundation classes to a form. For foundation classes with Category A and B base classes, you can drag the foundation class from the Component Gallery and then drop it on a form. You can also right-click a foundation class in the Component Gallery to display a shortcut menu, and then choose Add to Form to add the foundation class to the form.
Some of the foundation classes have associated builders that are automatically launched to prompt you for more information needed by the foundation class.
Class Browser You can drag foundation classes with Category A and B base classes directly from the Class Browser to a form by using the drag icon in the upper left corner of the Class Browser. Select the foundation class in the Class Browser, click the icon for the foundation class in the upper left corner of the Class Browser, and then drag the icon over the form. Release the mouse button over the form where you’d like the foundation class to appear on the form.
Foundation classes dragged from the Class Browser to a form do not launch the associated builder. However, you can launch the builder by after the foundation class has been dropped on the form. Select the foundation class on the form, and then right-click to display the shortcut menu. Choose Builder from the shortcut menu to launch the builder.
Project Manager Foundation classes with Category A and B base classes can be dragged from the Project Manager and dropped on a form.
Foundation classes dragged from the Project Manager to a form do not launch the associated builder. However, you can launch the builder by after the foundation class has been dropped on the form. Select the foundation class, and then right-click to display the shortcut menu. Choose Builder from the shortcut menu to launch the builder.
Form Controls toolbar Foundation classes with Category A and B base classes added to the Form Controls toolbar can be added to a form.
If the Builder Lock isn’t on, foundation classes dropped from the Form Controls toolbar may launch an associated builder. If the Build Lock is on, you can launch the builder by after the foundation class has been added to the form. Select the foundation class on the form, and then right-click to display the shortcut menu. Choose Builder from the shortcut menu to launch the builder.
Adding Foundation Classes to Projects
When a form containing foundation classes is added to a Visual FoxPro project, the Project Manager automatically adds the visual class libraries containing the foundation classes to the project. However, there are other cases where you may need to add foundation classes to a project. For example, your application may run a Category C foundation class, so the foundation class must be added to the application’s project.
Foundation classes can be added to a project from the Component Gallery, by dragging the .vcx visual class library containing the foundation classes from the Windows Explorer, or by choosing the Add button in the Project Manager.
Adding Foundation Classes from the Component Gallery
You can drag a foundation class from the Component Gallery to a project, or you can right-click the foundation class in the Component Gallery to display a shortcut menu, and then choose Add to Project to add the foundation class to the project. When you add a foundation class to a project, the Add Class to Project dialog box is displayed, prompting you with the following options: Add class to project Choose this option to add the foundation class and its .vcx class library to the project. Again, this is done automatically for classes dropped onto a form (Categories A and B). For certain Category B and C classes where you plan to call them programmatically from within your application, you will want to choose this option. Create a new class from selected class Choose this option to create a new subclass from the foundation class you want to add to the project. This option makes it possible for you to enhance the functionality of the original foundation class, usually by adding additional program code. Create a new form from selected class Choose this option for foundation classes with a Form base class (for example, the foundation classes in _dialogs.vcx). This option makes it possible for you to create a new form from the foundation class and enhance the functionality of the original foundation class.
Adding Foundation Classes from the Windows Explorer
A foundation class can be added to a project by dragging the .vcx visual class library containing the foundation class from the Window Explorer to the Project Manager. The visual class library is added to the Class Libraries item in the Project Manager.
Adding Foundation Classes from within the Project Manager
A foundation class can be added to a project by selecting the Classes tab and then choosing the Add button. Select the class library from the Ffc folder that contains the foundation class to add to the project.
Incorporating Classes into your Application
In many situations, most foundation classes don’t require additional programming to work with your application. However, you may need to provide additional program code for certain foundation classes (those of Category B and Category C non-visual base classes).
Non-Visual Foundation Classes
For example, foundation classes are often based on the Category B Custom class, and these require additional programming. These non-visual classes often perform common tasks such as checking information in the Windows registry, handling environment settings, managing application errors, and utilizing Automation with other applications, such as performing mail merge with Microsoft Word.
You can drop these non-visual classes onto a form, but you will need to do some additional work in order for them to work with your application. In some cases, a builder is launched when you drag a foundation class onto a form.
The following example demonstrates some of the program code typically necessary to use a non-visual foundation class in your application:
Drag the File Version foundation class from the Component Gallery (Foundation ClassesUtilities folder) onto a form.
Add a command button to the form and add the following code to its Click event: WITH THISFORM._FILEVERSION1 .cFileName = HOME( )+ ‘VFP7.EXE’ .GetVersion( ) .DisplayVersion( ) ENDWITH
Run the form and click the command button.
You can incorporate a non-visual class in your application without dropping it on a form, as long as you include it in the project used to create the application. The following code illustrates how to executed this same File Version foundation class if the class is not dropped onto a form.
LOCAL oFileVersion
oFileVersion = NewObject('_fileversion', '_utilities.vcx')
WITH oFileVersion
.cFileName = HOME( )+ 'VFP7.EXE'
.GetVersion( )
.DisplayVersion( )
ENDWITH
Note This example assumes that the code can locate the _utilities.vcx class library or from an .app file that is built containing _utilities.vcx.
When you use a non-visual foundation class, you need to know how and when the class is used within your application so it can be scoped correctly. If only a single form uses the class, you can just drag the class onto the form. However, if the class is used by many forms or is used globally by the application, the foundation class should have a global scope in the application so it remains accessible throughout the application. A global scope may also improve performance.
Visual Foundation Classes
You can also programmatically add visual foundation classes, such as those with form base classes, to your application. The following example shows how you can add code to your application to display an About dialog box box.
LOCAL oAbout
oAbout = NewObject('_aboutbox','_dialogs.vcx')
oAbout.Show( )
You can create a subclass of the dialog box foundation class for each of your applications so that you can customize the contents of the dialog box for each application. The following example demonstrates how you can subclass the Aboutbox foundation class:
Drag and drop the Aboutbox class from the Component Gallery (Foundation ClassesDialogs folder) to the project for your application.
Select Create new form from selected class in the Add Class to Project dialog box that is displayed, and enter name for the form.
Change the Caption property for the new form for your application. Save and close the form.
Add program code (DO FORM FormName) to the procedure that runs the form, such as an About menu item procedure. -or-
Drag the Run Form button class from the Component Gallery (Foundation ClassesButtons folder) onto the form. A builder is launched, and you can specify the name of the form to execute.
If you use the Visual FoxPro 7.0 Application Framework, the Application Builder automatically handles adding forms (both .scx and .vcx form classes). The new Application Wizard or the Component Gallery New Application item installs this framework in the projects they create. The Application Builder interacts directly with the framework and enables you to specify how and where the form is launched.
By using a framework built with the Application Wizard, the Application Builder, and Component Gallery, you have a rich set of tools for creating entire applications with minimal manual coding.
Class Naming Conventions
The Visual FoxPro Foundation classes and their properties and methods use the following naming conventions.
Classes and Class Libraries
Most foundation classes are subclassed from classes in the _base.vcx visual class library, which you can also find in the Ffc folder. The naming conventions for these classes reflect the base class used. For example, a subclass of the Custom class is called _Customin _base.vcx. All classes use an underscore ( _ ) to preface the name of a class in _base.vcx.
A few class libraries do not contain classes that are subclassed from _base.vcx because these classes are shared with other Visual FoxPro components such as wizards and builders. These classes are contained in class libraries without a preceding underscore, such as Registry.vcx.
Methods and Properties
Methods are often based on an action name such as RunForm. If the name contains several words, for example, RunForm, then capitalization reflects this. Properties are usually prefaced with a single letter characterizing the data type of that particular property. For example, cFileName indicates that the property is of character type. In addition, default values for properties are also set to the appropriate data type. For example, a logical property can be initialized to false (.F.), while a numeric property can be initialized to 0.
Properties of classes that shipped in earlier versions of Visual FoxPro do not strictly adhere to these property-naming conventions, and retain their earlier names to avoid compatibility conflicts with user code referencing these properties.
Enhancing or Modifying FoxPro Foundation Classes
You can enhance or modify the Visual FoxPro foundation classes to meet the needs of your application. However, we recommend that you do not modify the foundation classes themselves. The foundation classes may be periodically updated with new functionality.
Subclassing the Foundation Class
The source code is provided for the foundation classes, so you can subclass any foundation class to override or enhance properties and methods. This choice is common when the behavior of a particular foundation class varies between different applications. One application might use a foundation class directly, while another application uses a subclass of the foundation class.
Updating _base.vcx
If you want to add global changes to the Visual FoxPro foundation classes, you can modify _base.vcx. Since foundation classes are subclassed from _base.vcx, changes to this class library are automatically propagated to the foundation classes. A common set of methods and properties are provided for all the classes in _base.vcx. However, you can alter the classes in _base.vcx if they add desired behavior to your applications (unlike the foundation classes that we recommend that you do not change).
Instead of changing _base.vcx, however, you should redefine the classes in _base.vcx to inherit their behavior from your own custom base classes (rather than from the Visual FoxPro base classes currently used). If you already have a custom class library which subclasses the Visual FoxPro base classes, you can redefine the classes in _base.vcx to inherit from your custom classes so that when components use the _base classes they will inherit from your custom classes too. You can use the Class Browser to redefine the parent class for a particular class.
Note If you redefine the classes to inherit from your own custom base classes, you should add DODEFAULT( ) calls at appropriate locations if you desire that parent class method code be executed.
If you replace the entire _base.vcx class with your own, make sure that you have the same set of named classes; otherwise the foundation classes will have missing links.
Specifies the name of the variable or array element to reference in the macro substitution. Do not include the M. prefix that distinguishes variables from fields. Such inclusion causes a syntax error. The macro should not exceed the maximum statement length permitted in Visual FoxPro.
A variable cannot reference itself recursively in macro substitution. For example, the following generates an error message:
Copy Code
STORE ‘&gcX’ TO gcX ? &gcX
Macro substitution statements that appear in DO WHILE, FOR, and SCAN are evaluated only at the start of the loop and are not reevaluated on subsequent iterations. Any changes to the variable or array element that occur within the loop are not recognized. [. cExpression]
The optional period (.) delimiter and .cExpression are used to append additional characters to a macro. cExpression appended to the macro with .cExpression can also be a macro. If cExpression is a property name, include an extra period (cExpression..PropertyName).
Remarks
Macro substitution treats the contents of a variable or array element as a character string literal. When an ampersand (&) precedes a character-type variable or array element, the contents of the variable or element replace the macro reference. You can use macro substitution in any command or function that accepts a character string literal.
Tip:
Whenever possible, use a name expression instead of macro substitution. A name expression operates like macro substitution. However, a name expression is limited to passing character strings as names. Use a name expression for significantly faster processing if a command or function accepts a name (a file name, window name, menu name, and so on). For additional information on name expressions, see Name Expression Creation.
While the following commands are acceptable:
Copy Code
STORE ‘customer’ TO gcTableName STORE ‘company’ TO gcTagName USE &gcTableName ORDER &gcTagName
use a name expression instead:
Copy Code
USE (gcTableName) ORDER (gcTagName)
Macro substitution is useful for substituting a keyword in a command. In the following example, the TALK setting is saved to a variable so the setting can be restored later in the program. The original TALK setting is restored with macro substitution.
Note:
Performing concatenation with a single ampersand (&) when attempting to include double ampersands (&&) in a string literal might produce undesirable results. For example, suppose you assign the string “YYY” to a variable, BBB. Performing concatenation using “AAA&” and “&BBB” replaces “BBB” with “YYY” so instead of getting the result “AAA&&BBB”, the result is “AAA&YYY”. For more information, see && Command.
Example
Copy Code
STORE SET(‘TALK’) TO gcSaveTalk SET TALK OFF * * Additional program code * SET TALK &gcSaveTalk && Restore original TALK setting
Indicates the beginning of a nonexecuting inline comment in a program file.
&& [Comments]
Parameters
&& [ Comments]
Specifies inline comments that follow.
Remarks
Inserting inline comments to denote the end of the IF … ENDIF, DO, and FOR … ENDFOR structured programming commands greatly improves the readability of programs when including many such structures.
Caution:
Including double ampersands (&&) in a string literal, for example, “AAA&&BBB”, generates an error. Instead, to include double ampersands, use concatenation as shown: “AAA&” + “&” + “BBB”.
Note:
When using concatenation, use caution with placement of a single ampersand (&), which is used to perform macro substitution and thus might produce undesirable results. For example, suppose you assign the string “YYY” to a variable, BBB. Performing concatenation using “AAA&” and “&BBB” replaces “BBB” with “YYY”, so instead of getting the result “AAA&&BBB”, the result is “AAA&YYY”. For more information, see & Command.
To continue a comment on the following line, place a semicolon (;) at the end of the comment line to be continued.
Note:
In earlier versions of Visual FoxPro, you cannot place && and a comment after the semicolon that is used to continue a command line to an additional line.
Example
The following example includes the inline comments “20 years of monthly payments” indicated by the && command:
Copy Code
STORE (20*12) TO gnPayments && 20 years of monthly payments NOTE Initialize the page number; variable. STORE 1 to gnPageNum * Set up the loop DO WHILE gnPageNum <= 25 && loop 25 times gnPageNum = gnPageNum + 1 ENDDO && DO WHILE gnPageNum <= 25
Indicates the beginning of a nonexecuting comment line in a program file.
* [Comments]
Parameters
Comments
Specifies the comment in the comment line. For example:
Copy Code
* This is a comment
Remarks
Place a semicolon (;) at the end of each comment line that continues to a following line.
Any text added to a method or event in a Visual Class Library (VCX) or form (SCX) Code window will cause that class to have Override behavior for the method or event. Therefore, code for the method or event in a parent class will not be executed by default (it must be explicitly executed). This includes non-executable comment lines that begin with “*”.
Example
Copy Code
* Initialize the page number; variable. STORE 1 to gnPageNum * Set up the loop DO WHILE gnPageNum <= 25 && loop 25 times gnPageNum = gnPageNum + 1 ENDDO && DO WHILE gnPageNum <= 25
Evaluates the expression specified by Expression1 and sends a carriage return and line feed preceding the expression results.
The results display on the next line of the main Visual FoxPro window or the active user-defined window and are printed at the left margin of a page unless a function code specified by cFormatCodes or the _ALIGNMENT system variable specifies otherwise.
If you omit the expressions, a blank line is displayed or printed. A space is placed between expression results when multiple expressions are included.
If Expression1 is an object, the ? command returns the character string, “(Object)”. ?? Expression1
Evaluates the expression specified by Expression1 and displays the expression results on the current line at the current position of the main Visual FoxPro window, an active user-defined window, or the printer. A carriage return and line feed are not sent before the results. PICTURE cFormatCodes
Specifies a picture format in which the result of Expression1 is displayed. cFormatCodes can consist of function codes, picture codes, or a combination of both. You can use the same codes available in the Format Property and InputMask Property.
Function codes affect the overall format of the result; picture codes act on individual characters in the result. If function codes are used in cFormatCodes, they must appear before the picture codes and they must be preceded by an at (@) sign. Multiple function codes with no embedded spaces can immediately follow the @ sign. The last function code must be followed by one or more spaces. The space or spaces signal the end of the function codes and the start of the picture codes. FUNCTION cFormatCodes
Specifies a function code to include in the output from the ? and ?? commands. If the function clause is included, do not precede the function codes with an @ sign. Function codes must be preceded by a @ sign when included in PICTURE. V nWidth
Specifies a special function code that enables the results of a character expression to stretch vertically within a limited number of columns. nWidth specifies the number of columns in the output.
Copy Code
? ‘This is an example of how the V function code works.’ ; FUNCTION ‘V10’
AT nColumn
Specifies the column number where the output is displayed. This option makes it possible for you to align output in columns to create a table. The numeric expression nColumn can be a user-defined function that returns a numeric value. FONT cFontName[, nFontSize [, nFontCharSet]]
Specifies a font for output by the ? or ?? command. cFontName specifies the name of the font, and nFontSize specifies the point size. You can specify a language script with nFontCharSet. See the GETFONT( ) Function for a list of available language script values.
For example, the following command displays the system date in 16-point Courier font:
Copy Code
? DATE( ) FONT ‘Courier’,16
If you include the FONT clause but omit the point size nFontSize, a 10-point font is used.
If you omit the FONT clause, and output for the ? or ?? command is placed in the main Visual FoxPro window, the main Visual FoxPro window font is used for the output. If you omit the FONT clause, and output for the ? or ?? command is placed in a user-defined window, the user-defined window font is used for the output.
Note:
If the font you specify is not available, a font with similar font characteristics is substituted.
STYLE cFontStyle
Specifies a font style for output by the ? or ?? commands. If you omit the STYLE clause, the Normal font style is used. If the font style you specify is not available, a font style with similar characteristics is substituted.
Note:
You must include the FONT clause when you specify a font style with the STYLE clause.
The following table lists font styles that you can specify for cFontStyle.
cFontStyle
Font style
B
Bold
I
Italic
N
Normal
Q
Opaque
–
Strikeout
T
Transparent
U
Underline
You can include more than one character to specify a combination of font styles. For example, the following command displays the system date in Courier Bold Italic:
Copy Code
? DATE( ) FONT ‘COURIER’ STYLE ‘BI’
Remarks
To send the results to the printer only, use the following commands:
Copy Code
SET PRINTER ON SET CONSOLE OFF
To send the results to the printer and the main Visual FoxPro window or an active user-defined window, use the following command:
Copy Code
SET PRINTER ON
The setting of SET ALTERNATE affects the destination for the ? and ?? commands. For more information, see SET ALTERNATE Command.
The ? command displays binary data for Varbinary data types in hexadecimal format with no limitation on size. For more information, see Varbinary Data Type.
Example
The following example displays evaluates and displays the expressions specified:
Specifies the characters that are sent to the printer.
Remarks
A group of three question marks bypasses the printer driver and sends the contents of cExpression directly to the printer. cExpression must contain valid printer codes.
Printer control codes make it possible for you to reset the printer, change type styles and sizes, and enable or disable boldface printing. These codes can consist of any combination of printable or nonprintable characters that are specific to the printer you are using. You can direct control codes to the printer in several different ways:
Use combinations of CHR( ) and quoted strings concatenated with + to send ASCII characters directly to the printer.
Use quotation marks to send a string containing printer codes or ASCII characters.
Codes can be sent to the printer before printing begins and after printing ends with the _PSCODE and _PECODE system variables. For more information, see _PSCODE System Variable and _PECODE System Variable.
Printer control codes vary from printer to printer. The best source for information about printer control codes is the manual that came with your printer.
Creates a control or object that can be activated with READ.
@ nRow, nColumn CLASS ClassName NAME ObjectName
Parameters
@ nRow, nColumn
Specifies the position of the control or object. The height and width of the control or object is determined by the class default height and width values.
Rows are numbered from top to bottom. The first row is number 0 in the main Visual FoxPro window or in a user-defined window. Row 0 is the row immediately beneath the Visual FoxPro system menu bar.
Columns are numbered from left to right. The first column is number 0 in the main Microsoft Visual FoxPro window or in a user-defined window. When a control or object is placed in a user-defined window, the row and column coordinates are relative to the user-defined window, not to the main Visual FoxPro window.
A position in the main Visual FoxPro window or in a user-defined window is determined by the font of the window. Most fonts can be displayed in a wide variety of sizes; some are proportionally spaced. A row corresponds to the height of the current font; a column corresponds to the average width of a letter in the current font.
You can position the control or object using decimal fractions for row and column coordinates. CLASS ClassName
Specifies the class of the control or object. ClassName can be a Visual FoxPro base class or a user-defined class. The following table lists the Visual FoxPro base classes you can specify for ClassName. NAME ObjectName
Specifies the name of the object reference variable to create. The object-oriented properties, events, and methods of the control or object can be manipulated by referencing this variable. For a complete list of the Visual FoxPro base classes, see Base Classes in Visual FoxPro.
Remarks
@ … CLASS provides an intermediate step for converting programs and applications created in earlier versions of FoxPro to the preferred object-oriented programming methods of Visual FoxPro. For additional information about backward compatibility with FoxPro 2.x controls, see Controls and Objects Created in Earlier Versions.
The following example demonstrates how @ … CLASS can be used with programming techniques used in earlier FoxPro versions (in this example, use of READ to activate controls). @ … CLASS is used to create a text box whose properties can be changed with the Visual FoxPro object-oriented programming techniques.
ON KEY LABEL is used to display the Windows Color dialog box when you press CTRL+I. The TextBox is placed on the main Visual FoxPro window using @ … CLASS, and READ activates the text box.
Copy Code
CLEAR ON KEY LABEL CTRL+I _SCREEN.PageFrame1.Page1.goFirstName.BackColor; = GETCOLOR( ) @ 2,2 SAY ‘Press Ctrl+I to change the background color’ @ 4,2 CLASS TextBox NAME goFirstName READ CLEAR
Clears a portion of the main Visual FoxPro window or a user-defined window.
@ nRow1, nColumn1 [CLEAR CLEAR TO nRow2, nColumn2]
Parameters
@ nRow1, nColumn1 CLEAR
Clears a rectangular area whose upper-left corner begins at nRow1 and nColumn1 and continues to the lower-right corner of the main Visual FoxPro window or a user-defined window. CLEAR TO nRow2, nColumn2
Clears a rectangular area whose upper-left corner is at nRow1 and nColumn1 and whose lower-right corner is at nRow2 and nColumn2.
Remarks
If you omit CLEAR or CLEAR TO, Visual FoxPro clears nRow1 from nColumn1 to the end of the row.
Example
The following example clears the screen, main Visual FoxPro window, or user-defined window from the second row to the bottom of the window.
Copy Code
@ 2,0 CLEAR
The following example clears a rectangular region. The area from row 10 and column 0 to row 20 and column 20 is cleared.
Changes the colors of existing text within an area of the screen.
@ nRow1, nColumn1 FILL TO nRow2, nColumn2[COLOR SCHEME nSchemeNumber COLOR ColorPairList]
Parameters
@ nRow1, nColumn1
Specifies the upper-left corner of the area to change. FILL TO nRow2, nColumn2
Specifies the lower-right corner of the area to change. COLOR SCHEME nSchemeNumber
Specifies the color of the area. Only the first color pair in the specified color scheme determines the color of the area. COLOR ColorPairList
Specifies the color of the area. Only the first color pair in the specified color pair list determines the color of the area.
If you omit the COLOR SCHEME or COLOR clauses, the rectangular portion is cleared. An area can also be cleared with @ … CLEAR.
Remarks
This command changes the colors of text within a rectangular area of the main Visual FoxPro window or the active user-defined window. You can set the foreground and background color attributes for existing text only. Any text output to the same area after you issue @ … FILL appears in the default screen or window colors.
Example
The following example clears the main Visual FoxPro window and fills an area with color.
Copy Code
ACTIVATE SCREEN CLEAR @ 4,1 FILL TO 10, 8 COLOR GR+/B
When you use , the text line is preceded by a carriage return and a line feed. \ TextLine
When you use \, the text line is not preceded by a carriage return and a line feed.
Any spaces preceding and \ are not included in the output line, but spaces following and \ are included.
You can embed an expression in the text line. If the expression is enclosed in the text merge delimiters (<< >> by default) and SET TEXTMERGE is ON, the expression is evaluated and its value is output as text.
Remarks
The and \ commands facilitate text merge in Visual FoxPro. Text merge makes it possible for you to output text to a file to create form letters or programs.
Use and \ to output a text line to the current text-merge output file and the screen. SET TEXTMERGE is used to specify the text merge output file. If text merge is not directed to a file, the text line is output only to the main Visual FoxPro window or the active user-defined output window. SET TEXTMERGE NOSHOW suppresses output to the main Visual FoxPro window or the active user-defined window.
Example
Copy Code
CLOSE DATABASES OPEN DATABASE (HOME(2) + ‘Datatestdata’) USE Customer && Open customer table SET TEXTMERGE ON SET TEXTMERGE TO letter.txt <<CDOW(DATE( ))>>, <<CMONTH(DATE( ))>> \ <<DAY(DATE( ))>>, <<YEAR(DATE( ))>> Dear <<contact>> Additional text Thank you, XYZ Corporation CLOSE ALL MODIFY FILE letter.txt NOEDIT
Specifies the expression or expressions that the = command evaluates.
Remarks
The = command evaluates one or more expressions, Expression1, Expression2 …, and discards the return values. This option is particularly useful when a Visual FoxPro function or a user-defined function has a desired effect, but there is no need to assign the function’s return value to a variable, array element, or field.
For example, to turn insert mode on, you can issue the following command:
Copy Code
= INSMODE(.T.)
INSMODE normally returns a True (.T.) or False (.F.) value. In the example above, the function is executed but the return value is discarded.
If only one expression (Expression1) is included, the equal sign is optional.
Note:
There are two unrelated uses for the equal sign (=). It can be used as an operator in logical expressions to make a comparison, or to assign values to variables and array elements. In these two cases, the equal sign (=) is an operator and not a command. See Relational Operators for more information about using the equal sign (=) as an operator in logical expressions. See STORE Command for more information about using the equal sign (=) to assign values to variables and array elements.
ACTIVATE MENU MenuBarName [NOWAIT] [PAD MenuTitleName]
Parameters
MenuBarName
Specifies the name of the menu bar to activate. NOWAIT
Specifies that at run time the program should not wait for the user to choose a menu from the active menu bar or to press ESC. Instead, the program continues to execute. A menu activated with the NOWAIT option does not return program execution to the line following the ACTIVATE MENU command when DEACTIVATE MENU is issued. PAD MenuTitleName
Specifies the menu title name that is automatically selected when the menu bar is activated. If you don’t specify a menu title name, the first menu title name in the activated menu bar is activated by default.
Remarks
Displays and activates the menu bar specified with MenuBarName. This command works in conjunction with DEFINE MENU and DEFINE PAD.
Note:
When you include the Visual FoxPro system menu bar (_MSYSMENU) in an application, there is no need to activate the menu. Instead, issue SET SYSMENU AUTOMATIC.
Example
The following example uses ACTIVATE MENU to display and activate a user-defined menu system. The current system menu bar is first saved to memory with SET SYSMENU SAVE, and then all system menu titles are removed with SET SYSMENU TO.
Two menu titles are created with DEFINE PAD; DEFINE POPUP is used to create a drop-down menu for each menu title. DEFINE BAR is used to create menu items on each of the menus. When a menu title is chosen, ON PAD uses ACTIVATE POPUP to activate the corresponding menu. ACTIVATE MENU displays and activates the menu bar.
When a menu item is chosen from a menu, the CHOICE procedure is executed. CHOICE displays the name of the chosen item and the name of the menu containing the item.
Copy Code
*** Name this program ACTIMENU.PRG *** CLEAR SET SYSMENU SAVE SET SYSMENU TO ON KEY LABEL ESC KEYBOARD CHR(13) DEFINE MENU example BAR AT LINE 1 DEFINE PAD convpad OF example PROMPT ‘<Conversions’ COLOR SCHEME 3 ; KEY ALT+C, ” DEFINE PAD cardpad OF example PROMPT ‘Card <Info’ COLOR SCHEME 3 ; KEY ALT+I, ” ON PAD convpad OF example ACTIVATE POPUP conversion ON PAD cardpad OF example ACTIVATE POPUP cardinfo DEFINE POPUP conversion MARGIN RELATIVE COLOR SCHEME 4 DEFINE BAR 1 OF conversion PROMPT ‘Ar<ea’ ; KEY CTRL+E, ‘^E’ DEFINE BAR 2 OF conversion PROMPT ‘<Length’ ; KEY CTRL+L, ‘^L’ DEFINE BAR 3 OF conversion PROMPT ‘Ma<ss’ ; KEY CTRL+S, ‘^S’ DEFINE BAR 4 OF conversion PROMPT ‘Spee<d’ ; KEY CTRL+D, ‘^D’ DEFINE BAR 5 OF conversion PROMPT ‘<Temperature’ ; KEY CTRL+T, ‘^T’ DEFINE BAR 6 OF conversion PROMPT ‘T<ime’ ; KEY CTRL+I, ‘^I’ DEFINE BAR 7 OF conversion PROMPT ‘Volu<me’ ; KEY CTRL+M, ‘^M’ ON SELECTION POPUP conversion DO choice IN actimenu; WITH PROMPT( ), POPUP( ) DEFINE POPUP cardinfo MARGIN RELATIVE COLOR SCHEME 4 DEFINE BAR 1 OF cardinfo PROMPT ‘<View Charges’ ; KEY ALT+V, ” DEFINE BAR 2 OF cardinfo PROMPT ‘View <Payments’ ; KEY ALT+P, ” DEFINE BAR 3 OF cardinfo PROMPT ‘Vie<w Users’ ; KEY ALT+W, ” DEFINE BAR 4 OF cardinfo PROMPT ‘-‘ DEFINE BAR 5 OF cardinfo PROMPT ‘<Charges ‘ ; KEY ALT+C, ” ON SELECTION POPUP cardinfo; DO choice IN actimenu WITH PROMPT( ), POPUP( ) ACTIVATE MENU example DEACTIVATE MENU example RELEASE MENU example EXTENDED SET SYSMENU TO DEFAULT ON KEY LABEL ESC PROCEDURE choice PARAMETERS mprompt, mpopup WAIT WINDOW ‘You chose ‘ + mprompt + ‘ from popup ‘ + mpopup NOWAIT
Specifies the name of the menu to activate. AT nRow, nColumn
Specifies the position of the menu on the screen or in a user-defined window. The row and column coordinate applies to the upper-left corner of the menu. The position you specify with this argument takes precedence over a position you specify with the FROM argument in DEFINE POPUP. BAR nMenuItemNumber
Specifies the item in the menu that is selected when the menu is activated. For example, if nMenuItemNumber is 2, the second item is selected. The first item is selected if you omit BAR nMenuItemNumber or if nMenuItemNumber is greater than the number of items in the menu. NOWAIT
Specifies that, at run time, a program does not wait for the user to choose an item from the menu before continuing program execution. Instead, the program continues to execute. REST
A menu created with the PROMPT FIELD clause of DEFINE POPUP places records from a field into the menu. When the menu is activated, the first item in the menu is initially selected, even if the record pointer in the table containing the field is positioned on a record other than the first record.
Include REST to specify that the item selected when the menu is activated corresponds to the current record pointer position in the table.
Remarks
ACTIVATE POPUP works in conjunction with DEFINE POPUP, used to create the menu, and DEFINE BAR, used to create the items on the menu.
Example
This example uses ACTIVATE POPUP with ON PAD to activate a menu when a menu title is chosen. The current system menu bar is first saved to memory with SET SYSMENU SAVE, and then all system menu titles are removed with SET SYSMENU TO.
Two new system menu titles are created with DEFINE PAD; DEFINE POPUP is used to create a menu for each menu title. DEFINE BAR is used to create menu items on each of the menus. When a menu title is chosen, ON PAD uses ACTIVATE POPUP to activate the corresponding menu.
When an item is chosen from a menu, the CHOICE procedure is executed. CHOICE displays the name of the chosen item and the name of the menu containing the item. If the Exit item is chosen from the Card Info menu, the original Visual FoxPro system menu is restored.
Copy Code
*** Name this program ACTIPOP.PRG *** CLEAR SET SYSMENU SAVE SET SYSMENU TO DEFINE PAD convpad OF _MSYSMENU PROMPT ‘<Conversions’ COLOR SCHEME 3 ; KEY ALT+C, ” DEFINE PAD cardpad OF _MSYSMENU PROMPT ‘Card <Info’ COLOR SCHEME 3 ; KEY ALT+I, ” ON PAD convpad OF _MSYSMENU ACTIVATE POPUP conversion ON PAD cardpad OF _MSYSMENU ACTIVATE POPUP cardinfo DEFINE POPUP conversion MARGIN RELATIVE COLOR SCHEME 4 DEFINE BAR 1 OF conversion PROMPT ‘Ar<ea’ KEY CTRL+E, ‘^E’ DEFINE BAR 2 OF conversion PROMPT ‘<Length’ ; KEY CTRL+L, ‘^L’ DEFINE BAR 3 OF conversion PROMPT ‘Ma<ss’ ; KEY CTRL+S, ‘^S’ DEFINE BAR 4 OF conversion PROMPT ‘Spee<d’ ; KEY CTRL+D, ‘^D’ DEFINE BAR 5 OF conversion PROMPT ‘<Temperature’ ; KEY CTRL+T, ‘^T’ DEFINE BAR 6 OF conversion PROMPT ‘T<ime’ ; KEY CTRL+I, ‘^I’ DEFINE BAR 7 OF conversion PROMPT ‘Volu<me’ ; KEY CTRL+M, ‘^M’ ON SELECTION POPUP conversion; DO choice IN actipop WITH PROMPT(), POPUP() DEFINE POPUP cardinfo MARGIN RELATIVE COLOR SCHEME 4 DEFINE BAR 1 OF cardinfo PROMPT ‘<View Charges’ ; KEY ALT+V, ” DEFINE BAR 2 OF cardinfo PROMPT ‘View <Payments’ ; KEY ALT+P, ” DEFINE BAR 3 OF cardinfo PROMPT ‘Vie<w Users’ ; KEY ALT+W, ” DEFINE BAR 4 OF cardinfo PROMPT ‘-‘ DEFINE BAR 5 OF cardinfo PROMPT ‘<Charges’ ; KEY ALT+C, ” DEFINE BAR 6 OF cardinfo PROMPT ‘-‘ DEFINE BAR 7 OF cardinfo PROMPT ‘E<xit’; KEY ALT+X, ” ON SELECTION POPUP cardinfo; DO choice IN actipop WITH PROMPT(),POPUP() PROCEDURE choice PARAMETERS mprompt, mpopup WAIT WINDOW ‘You chose ‘ + mprompt + ; ‘ from popup ‘ + mpopup NOWAIT IF mprompt = ‘Exit’ SET SYSMENU TO DEFAULT ENDIF
Displays and activates one or more user-defined windows or Visual FoxPro system windows.
ACTIVATE WINDOW WindowName1 [, WindowName2 …] | ALL [IN [WINDOW] WindowName3 | IN SCREEN [BOTTOM | TOP | SAME] [NOSHOW]
Parameters
WindowName1[, WindowName2…]
Specifies the name of each window to activate. Separate the window names with commas. In Visual FoxPro, you can specify the name of a toolbar to activate. See SHOW WINDOW Command for a list of Visual FoxPro toolbar names. ALL
Specifies that all windows are activated. The last window activated is the active output window. IN [WINDOW] WindowName3
Specifies the name of the parent window within which the window is placed and activated. The activated window becomes a child window. A parent window can have multiple child windows. A child window activated inside a parent window cannot be moved outside the parent window. If the parent window is moved, the child window moves with it.
Note:
The parent window must be visible for any of its child windows to be visible.
IN SCREEN
Places and activates a window in the main Visual FoxPro window. A window can be placed in a parent window by including IN WINDOW in DEFINE WINDOW when the window is created. Including the IN SCREEN clause in ACTIVATE WINDOW overrides the IN WINDOW clause in DEFINE WINDOW. BOTTOM | TOP | SAME
Specifies where windows are activated with respect to other previously activated windows. By default, a window becomes the window on top when it is activated. Including BOTTOM places a window behind all other windows. TOP places it in front of all other windows. SAME activates a window without affecting its front-to-back placement. NOSHOW
Activates and directs output to a window without displaying the window.
Remarks
To successfully use this command on user-defined windows, any target user-defined window must have been created using the DEFINE WINDOW Command command.
Activating a window makes it the window on top and directs all output to that window. Output can be directed to only one window at a time. A window remains the active output window until it is deactivated or released, or until another window or the main Visual FoxPro window is activated.
The names of user-defined windows appear in the bottom section of the Window menu. The name of the active user-defined window is marked with a check mark.
More than one window can be placed in the main Visual FoxPro window at one time, but output is directed only to the last window activated. When more than one window is open, deactivating the active output window removes it from the main Visual FoxPro window and sends subsequent output to another window. If there is no active output window, output is directed to the main Visual FoxPro window.
Note:
To ensure output is directed to a specific window when you deactivate the active output window, you must explicitly activate the window you want to send output to with ACTIVATE WINDOW.
All activated windows are displayed until DEACTIVATE WINDOW or HIDE WINDOW is issued to remove them from view. Issuing either command removes windows from view but not from memory. Windows can be redisplayed by issuing ACTIVATE WINDOW or SHOW WINDOW.
To remove windows from view and from memory, use CLEAR WINDOWS, RELEASE WINDOWS, or CLEAR ALL. Windows that are removed from memory must be redefined to place them back in the main Visual FoxPro window.
You can use ACTIVATE WINDOW to place Visual FoxPro system windows in the main Visual FoxPro window or in a parent window.
The following system windows can be opened with ACTIVATE WINDOW:
Command
Call Stack
Debug
Debug Output
Document View
Locals
Trace
Watch
View
To activate a system window and or a toolbar, enclose the entire system window or toolbar name in quotation marks. For example, to activate the Call Stack debugging window in Visual FoxPro, issue the following command:
Copy Code
ACTIVATE WINDOW “Call Stack”
Historically in prior versions of Visual FoxPro, the Data Session window has always been referred to as the View window. Additionally, language used to control this window, such as HIDE WINDOW, ACTIVATE WINDOW, WONTOP( ), also refers to this window as the View window. Visual FoxPro continues to refer to the View window for the ACTIVATE WINDOW command.
Use HIDE WINDOW or RELEASE WINDOW to remove a system window from the main Visual FoxPro window or a parent window.
Example
The following example defines a window named output and activates it, placing it in the main Visual FoxPro window. The WAIT command pauses execution, the window is hidden, and then redisplayed.
Copy Code
CLEAR DEFINE WINDOW output FROM 2,1 TO 13,75 TITLE ‘Output’ ; CLOSE FLOAT GROW ZOOM ACTIVATE WINDOW output WAIT WINDOW ‘Press any key to hide window output’ HIDE WINDOW output WAIT WINDOW ‘Press any key to show window output’ SHOW WINDOW output WAIT WINDOW ‘Press any key to release window output’ RELEASE WINDOW output
Microsoft® Visual FoxPro® database development system is a powerful tool for quickly creating high-performance desktop, rich client, distributed client, client/server, and Web database applications. Employ its powerful data engine to manage large volumes of data, its object-oriented programming to reuse components across applications, its XML Web services features for distributed applications, and its built-in XML support to quickly manipulate data.
Note that Visual FoxPro 9.0 is the last version and was published in 2007.
Join the conversation and get your questions answered on the Visual FoxPro Forum on MSDN.
Visual FoxPro 9.0 Overview
With its local cursor engine, tight coupling between language and data, and powerful features, Visual FoxPro 9.0 is a great tool for building database solutions of all sizes. Its data-centric, object-oriented language offers developers a robust set of tools for building database applications for the desktop, client-server environments, or the Web. Developers will have the necessary tools to manage data—from organizing tables of information, running queries, and creating an integrated relational database management system (DBMS) to programming a fully-developed data management application for end users.
Data-Handling and Interoperability. Create .NET compatible solutions with hierarchical XML and XML Web services. Exchange data with SQL Server through enhanced SQL language capabilities and newly supported data types.
Extensible Developer Productivity Tools. Enhance your user interfaces with dockable user forms, auto-anchoring of controls, and improved image support. Personalize the Properties Window with your favorite properties, custom editors, fonts, and color settings.
Flexibility to Build All Types of Database Solutions. Build and deploy stand-alone and remote applications for Windows based Tablet PCs. Create and access COM components and XML Web Services compatible with Microsoft .NET technology.
Reporting System Features. Extensible new output architecture provides precision control of report data output and formatting. Design with multiple detail banding, text rotation, and report chaining. Output reports supported include in XML, HTML, image formats, and customizable multi-page print preview window. Backward compatible with existing Visual FoxPro reports.
Download samples, along with the final product updates including service packs for Visual FoxPro to ensure maximum productivity and performance from your Visual FoxPro development.
Visual FoxPro 9.0 Updates
Visual FoxPro 9.0 Service Pack 2 (SP2) Download Service Pack 2 for Microsoft Visual FoxPro 9.0. SP2 provides the latest updates to Visual FoxPro 9.0 combining various enhancements and stability improvements into one integrated package.
GDI+ Update for Visual FoxPro 9.0 SP1 Security update patch for Visual FoxPro 9.0 SP1 for fixing Buffer Overrun in JPEG Processing (GDI+). Note: We highly recommend that you install Service Pack 2, then apply the GDI+ SP2 update.
Visual FoxPro 9.0 ‘Sedna’ AddOns AddOn pack for Visual FoxPro 9.0. This download contains six components: VistaDialogs4COM, Upsizing Wizard, Data Explorer, NET4COM, MY for VFP and VS 2005 Extension for VFP.
XSource for Visual FoxPro 9.0 SP2 Download XSource for Visual FoxPro 9.0 SP2. XSource.zip has its own license agreement for usage, modification, and distribution of the Xbase source files included.
Microsoft OLE DB Provider for Visual FoxPro 9.0 SP2 The Visual FoxPro OLE DB Provider (VfpOleDB.dll) exposes OLE DB interfaces that you can use to access Visual FoxPro databases and tables from other programming languages and applications. The Visual FoxPro OLE DB Provider is supported by OLE DB System Components as provided by MDAC 2.6 or later. The requirements to run the Visual FoxPro OLE DB Provider are the same as for Visual FoxPro 9.0. Note: This version of the VFP OLE DB provider is the same version as the one included with Visual FoxPro 9.0 SP2.
VFPCOM Utility Extend Visual FoxPro interoperability with other COM and ADO components with the VFPCOM Utility. This utility is a COM server that provides additional functionality when you use ADO and access COM events with your Visual FoxPro 9.0 applications. For installation instructions and more details on the issues that have been addressed, consult the VFPCOM Utility readme.
Visual FoxPro ODBC Driver The VFPODBC driver is no longer supported. We strongly recommend using the Visual FoxPro OLE DB provider as a replacement. Please refer to the following article for more information and related links to issues when using the VFPODBC driver: https://support.microsoft.com/kb/277772.
Visual FoxPro 8.0 Updates
Visual FoxPro 8.0 Service Pack 1Download Microsoft Visual FoxPro 8.0 Service Pack 1 (SP1), which provides the latest updates to Visual FoxPro 8.0. SP1 combines various enhancements and stability improvements into one integrated package. The download contains all the documentation for these updates. For installation instructions and more details on SP1, consult the Service Pack 1 readme.
Visual FoxPro 8.0 SP1 Task Pane Source CodeSource code for Task Pane Manager component included in SP1 for Visual FoxPro 8.0. SP1 for VFP 8.0 included an updated Task Pane Manager component as an .APP application file but did not contain the update source code files associated with the updated version.
Visual FoxPro 8.0 Localization Toolkit OverviewOverview document of the Localization Toolkit project results for making available various language versions of the design-time IDE DLL and help documentation as add-ons to the English version of Visual FoxPro 8.0.
Visual FoxPro 7.0 Updates
Visual FoxPro 7.0 Service Pack 1Download Microsoft Visual FoxPro 7.0 Service Pack 1 (SP1), which provides the latest updates to Visual FoxPro 7.0. SP1 combines various enhancements and stability improvements into one integrated package. The download contains all the documentation for these updates. For installation instructions and more details on SP1, consult the Service Pack 1 readme.
Code Samples
.NET Samples for Visual FoxPro DevelopersThis download contains different projects and source files which are designed to show how how some common Visual FoxPro functionally is created in Visual Basic .NET.
Visual FoxPro 8.0 SamplesThis download contains different projects which are designed to show how new features in Visual FoxPro 8.0 can be used. Each project is self-contained and can be run independently of any other. There is a readme text file contained in each project that describes each sample program.
To install Microsoft Visual FoxPro 9.0, you need:Expand table
Minimum Requirements
Processor
PC with a Pentium-class processor
Operating System
Microsoft Windows 2000 with Service Pack 3 or later operating systemMicrosoft Windows XP or laterMicrosoft Windows Server 2003 or later
Memory
64 MB of RAM minimum; 128 MB or higher recommended
Hard Disk
165 MB of available hard-disk space for typical installation; 20 MB of additional hard-disk space for Microsoft Visual FoxPro 9.0 Prerequisites
Drive
CD-ROM or DVD-ROM drive
Display
Super VGA 800 X 600 or higher-resolution monitor with 256 colors
Mouse
Microsoft Mouse or compatible pointing device
Frequently Asked Questions
Article
08/29/2016
Find answers to your frequently asked questions about Visual FoxPro.
Q: What operating system is required for Visual FoxPro 9.0?
Developing applications with Visual FoxPro 9.0 is supported only on Microsoft Windows 2000 Service Pack 3 or later, Windows XP, Windows Server 2003 and Windows Vista. You can create and distribute run-time applications for Windows 98, Windows Me, Windows 2000 Service Pack 3 or later, Windows XP, Windows Server 2003 and Windows Vista. Installation on Windows NT 4.0 Terminal Server Edition is not supported.
Yes. Visual FoxPro will continue to be supported as per the lifecyle policy (https://support.microsoft.com/lifecycle/?p1=7992). Visual FoxPro 9 will be supported until 2014. In support of these products we may release patch updates from time to time. These typically fix problems discovered either internally or by a customer and reported to our product support engineers.
Q: Will there be a service pack 3 for Visual FoxPro 9?
At this time there are no plans to release a service pack for Visual FoxPro. However if there arises a need to publish a collection of fixes we may release a service pack. We will make announcements on the Visual FoxPro home page.
Q: What types of applications can I build with Visual FoxPro 9.0?
With its local cursor engine, tight coupling between language and data, and powerful features, such as object-oriented programming, Visual FoxPro 9.0 is a great tool for building database solutions of all sizes, from desktop and client/server database applications to data-intensive COM components and XML Web services.
Visual FoxPro 9.0 is an application development tool for building extremely powerful database applications and components. Its data-centric, object-oriented language offers developers a robust set of tools for building database applications on the desktop, client/server, or on the Web, through components and XML Web services. Developers will have the necessary tools to manage data from organizing tables of information, running queries, and creating an integrated relational database management system (DBMS) to programming a fully developed data management application for end users.
Q: Can I use Visual FoxPro to build Web applications?
Visual FoxPro COM components can be used with Internet Information Services (IIS) to build high-powered Internet database applications. This is because Visual FoxPro components can be called from Active Server Pages (ASP). Visual FoxPro is compatible with ASP but works even better in conjunction with the more modern ASP.NET. The components will retrieve and manipulate data, and will build some of the HTML returned to the user.
Q: Can you consume XML Web services with Visual FoxPro?
Yes, Visual FoxPro 9.0 makes it easy to consume XML Web services by integrating the SOAP Toolkit into the product.
Q: Is Visual FoxPro a part of MSDN Subscriptions?
Yes, Visual FoxPro 9.0 is included in the Professional, Enterprise, and Universal levels of MSDN Subscriptions. Visual FoxPro 9.0 is available for download to MSDN Subscribers via MSDN Subscriber downloads.
Q: How long will Visual FoxPro be supported by Microsoft?
Visual FoxPro 9.0 has standard support by Microsoft through January 2010 and extended support through January 2015 as per the developer tools lifecycle support policy.
Q: How long will the SOAP Toolkit included in Visual FoxPro 9.0 be supported by Microsoft?
Licensed users of Visual FoxPro 9.0 have a special lifecycle support plan for the SOAP Toolkit, supported by Microsoft on the same support plan as Visual FoxPro 8.0 which is through April 2008 and extended support through September 2013.
Q: Is Visual FoxPro 9.0 compatible with Visual Studio 2005 and SQL Server 2005?
Yes. We improved XML support and added new data types in Visual FoxPro 9.0 which improves .NET interop and SQL Server compatibility. Moreover the ‘Sedna’ add-on pack includes improvements to the Data Explorer and the Upsizing Wizard. These have significant improvements to support SQL Server 2005.
Q: How does Visual FoxPro 9.0 compare to SQL Server?
We do not contrast Visual FoxPro versus SQL Server. We position SQL Server as a database engine and Visual FoxPro as a developer tool. While Visual FoxPro has a database engine built-in, it is not positioned as a stand-alone database engine only. The trend is for an increasing amount of Visual FoxPro based applications to use SQL Server as the data storage in the solution. Of course, this is not required; it depends on the requirements of the solution. SQL Server offers security, reliability, replication, and many other features of a full relational database engine while the Visual FoxPro database system is an open file based DBF system that does not have many of those features. We leave it up to developers and companies to position and to compare various Microsoft products and technologies with each other and decide which ones are best for them to use when and how.
Q: Are there plans to enhance the 2 GB database size limit in Visual FoxPro?
The 2 GB limit is per table, not per database. We do not have any plans to extend the 2 GB table size limit in Visual FoxPro due to many reasons including the 32-bit architecture that already exists within the product. For large, scalable databases we recommend SQL Server 2008.
Q: Is Visual FoxPro supported on Windows Vista?
Yes. Visual FoxPro 9 Service Pack 2 is fully supported on Windows Vista.
Q: Are there plans for Visual FoxPro to support 64-bit versions of the Windows operating system?
No. While Visual FoxPro will remain 32-bit and not natively use 64-bit addressing; it will run in 32-bit compatibility mode. Visual Studio 2008 supports creating native 64-bit applications.
Q: How do you position Visual FoxPro in relation to Microsoft Access?
Microsoft Access, the database in Office, is the most broadly used and easiest-to-learn database tool that Microsoft offers. If you are new to databases, if you are building applications that take advantage of Microsoft Office, or if you want an interactive product with plenty of convenience, then choose Microsoft Access. Visual FoxPro is a powerful rapid application development (RAD) tool for creating relational database applications. If you are a database developer who builds applications for a living and you want ultimate speed and power, then choose Visual FoxPro.
Q: Is Visual FoxPro part of Visual Studio .NET?
No. Visual FoxPro 9.0 is a stand-alone database development tool which is compatible and evolutionary from previous versions of Visual FoxPro. Visual FoxPro 9.0 does not use or install the Windows .NET Framework. Visual FoxPro 9.0 is compatible with Visual Studio .NET the area of XML Web services, XML support, VFP OLE DB provider, and more. Visual FoxPro and Visual Studio are complimentary tools that work great together, such as Visual FoxPro 9.0 plus ASP.NET for adding WebForm front ends and mobile device front ends to Visual FoxPro applications.
Q: What is Microsoft’s position on Visual FoxPro related to Visual Studio and .NET?
We do not have plans to merge Visual FoxPro into Visual Studio and .NET, and there are no plans to create any sort of new Visual FoxPro .NET language. Instead, we are working on adding many of the great features found in Visual FoxPro into upcoming versions of Visual Studio, just like we’ve added great Visual Studio features into Visual FoxPro. If you want to do .NET programming, you should choose a .NET language with Visual Studio.
A Message to the Community
Article
08/29/2016
March 2007
We have been asked about our plans for a new version of VFP. We are announcing today that there will be no VFP 10. VFP9 will continue to be supported according to our existing policy with support through 2015 (https://support.microsoft.com/lifecycle/?p1=7992). We will be releasing SP2 for Visual FoxPro 9 this summer as planned, providing fixes and additional support for Windows Vista.
Additionally, as you know, we’ve been working on a project codenamed Sedna for the past year or so. Sedna is built using the extensibility model of VFP9 and provides a number of new features including enhanced connectivity to SQL Server, integration with parts of the .NET framework, support for search using Windows Desktop Search and Windows Vista as well as enhanced access to VFP data from Visual Studio.
Concurrently, the community has been using CodePlex (https://www.codeplex.com) to enhance VFP using these same capabilities in the VFPx project. Some of these community driven enhancements include:
Support for GDI+
An enhanced class browser
Support for Windows Desktop Alerts
An object oriented menu system
Integration with MSBuild
A rule-based code analysis tool similar to fxCop in Visual Studio
An Outlook Control Bar control
To reiterate, today we are announcing that we are not planning on releasing a VFP 10 and will be releasing the completed Sedna work on CodePlex at no charge. The components written as part of Sedna will be placed in the community for further enhancement as part of our shared source initiative. You can expect to see the Sedna code on CodePlex sometime before the end of summer 2007.
Welcome to Microsoft Visual FoxPro. Visual FoxPro is the object-oriented relational database management system that makes it possible for you to create database solutions for the desktop to the Web. Visual FoxPro provides powerful data handling capabilities, rapid application development tools for maximum productivity, and the flexibility needed to build all types of database solutions.
In the Visual FoxPro Documentation
What’s New in Visual FoxPro Describes the new features and enhancements included in this version of Visual FoxPro.
Getting Started with Visual FoxPro Provides information about where to find the Readme file, installing and upgrading from previous versions, configuring Visual FoxPro, and customizing the development environment.
Using Visual FoxPro Provides an overview of Visual FoxPro features, describes concepts and productivity tools for developing, programming, and managing high-performance database applications and components.
Samples and Walkthroughs Contains Visual FoxPro code samples and step-by-step walkthroughs that you can use for experimenting with and learning Visual FoxPro features.
Reference Includes Visual FoxPro general, programming language, user interface, and error message reference topics.
Product Support Provides information about Microsoft product support services for Visual FoxPro.
Additional Information
Microsoft Visual FoxPro Web Site Provides a link to the Microsoft Visual FoxPro Web site for additional information and resources for Visual FoxPro.
Microsoft Visual FoxPro Community Provides a link to Microsoft Visual FoxPro Online Community Web site for third-party community resources and newsgroups.
Getting Started with Visual FoxPro Provides information about where to find the ReadMe file and how to install and upgrade from previous versions, configure Visual FoxPro, and customize the development environment.
Using Visual FoxPro Provides an overview of Visual FoxPro features, describes concepts and productivity tools for developing, programming, and managing high-performance database applications and components, and provides walkthroughs that help get you started. With the robust tools and data-centric object-oriented language that Visual FoxPro offers, you can build modern, scalable, multi-tier applications that integrate client/server computing and the Internet.
Samples and Walkthroughs Contains Visual FoxPro code samples and step-by-step walkthroughs that you can use for experimenting with and learning Visual FoxPro features.
Reference (Visual FoxPro) Describes Visual FoxPro general, programming language, user interface, and error message reference topics.
Visual FoxPro 9’s Report System has undergone a thorough revision. This topic sketches the broad outlines of the changes, and provides you with information about where to look for details.
The following main areas of enhancements to the Report System are covered in sections of this topic.
Design-time enhancements. Multiple features and changes make designing reports in Visual FoxPro better for you and your end-users. The Report Builder Application re-organizes your design experience out-of-the-box. If you want to customize the design process, Report Builder dialog boxes and Report Designer events are fully exposed for you to do so.
Multiple detail bands. You can handle multiple child tables and data relationships more flexibly in the revised Report Designer. When you run multiple-detail-band reports, you can leverage the new bands, with associated detail headers and footers, both for appropriate presentation of these relationships and for more capable calculations.
Object-assisted run-time report processing. An entirely re-built output system, including a new base class, changes the way Visual FoxPro provides output report and label files at run time. Object-assisted reporting provides better-quality output, new types of output, and an open-architecture based on a new Visual FoxPro base class, the ReportListener. A programmable Report Preview interface interacts with ReportListeners to give you full control over report preview experience. The Report Preview Application provides improved out-of-the-box previewing facilities.
Printing, rendering, and character-set-handling improvements. Visual FoxPro 9 makes better use of the operating system’s printing features and GDI+ rendering subsystem. It also handles multiple locales and character sets better than previous versions. These changes are showcased in the Report System, and are accessible for use in custom code during report design and run-time processing.
Extensible use of report and label definition files (.frx and .lbx tables). Visual FoxPro 9 handles your existing reports and labels without modification, while allowing you to add new features and behavior to these reports easily. This backward-compatible, yet forward-thinking, migration strategy is made possible by the Report System’s newly-flexible handling of the .frx and .lbx table structure.
Design-time Enhancements
Numerous changes in the Report System help you enhance the design-time experience for developers and end-users. This section directs you to information about design-time improvements.
Report Designer Event Hooks and the Report Builder Application
The Report Designer now offers Report Builder Hooks, which enable you to intercept events occurring during a report or label design session to override and extend designer activity. The default Report Builder Application replaces many of the standard reporting dialog boxes with new ones written in Visual FoxPro code. Components of the Report Builder Application are exposed as Visual FoxPro Foundation Classes for your use.Expand table
Protection for End-User Design Sessions, and other Design-time Customization Opportunities
You can allow end-users to MODIFY and CREATE reports and labels, while setting limitations on what they can do in the Report Designer interface, using the new PROTECTED keyword. Protection is available individually by object and globally for the report. You can change what end-users see on the designer layout surface, from complex expressions to simple labels or sample data, while working in PROTECTED design mode, using Design-Time Captions. You can also provide helpful instructions, for both PROTECTED and standard design mode, by specifying Tooltips for report controls.Expand table
You can save the Data Environment you designed for a Report or Label as a visual class. You can load a Data Environment into a Report or Label design from either a visual class or a previously-saved report or label.Expand table
There have been numerous enhancements to the Report and Label Designers. Some features are subtle changes to make design sessions more efficient and more enjoyable, and others improve your choices for resulting output.Expand table
To learn about:
Read:
Improvements to the Report and Label Interactive Development Environment (IDE), such as:Enhanced Report Designer toolbar, and easier access to the Report Designer Toolbar from the View menuNew global Report Properties context menuImprovements and additions to existing context menusRevised and extended Report menu
Report document properties enable you to include information about the report in the report. Document properties are included as elements and attributes in XML and HTML output.
The Report Engine can now move through a scope of records multiple times. The records can represent related sets of detail lines in child tables, or they can be multiple passes through a single table. These multiple passes through a scope of records are represented as multiple detail bands.
Detail bands can have their own headers and footers, their own associated onEntry and onExit code, and their own associated report variables. Each detail band can be explicitly associated with a separate target alias, allowing you to control the number of entries in each detail band separately for related tables.
Multiple detail band reports provide many new ways you can represent data in reports and labels, and new ways you can calculate or summarize data, as you move through a record scope.Expand table
To learn about:
Read:
Designing reports and labels with multiple detail bands and their associated headers and footers
Visual FoxPro 9 has a new, object-assisted method of generating output from reports and labels. You can use your existing report and label layouts in object-assisted mode, to:
Generate multiple types of output during one report run.
Connect multiple reports together as part of one output result.
Improve the quality of traditional report output.
Dynamically adjust the contents of a report while you process it.
Provide new types of output not available from earlier versions of Visual FoxPro.
This section covers the array of run-time enhancements that work together to support object-assisted reporting mode.
Object-Assisted Architecture and ReportListener Base Class
The new ReportListener base class and supporting language enhancements are the heart of run-time reporting enhancements.Expand table
To learn about:
Read:
Fundamentals of the architecture, how its components work together, and what happens during an object-assisted report run
New Types of Output and the Report Output Component Set
Because you can subclass ReportListener, you can create new types of output. Visual FoxPro 9 supplies a Report Output Application to connect ReportListener subclasses with output types, as well as ReportListener-derived classes with enhanced output capabilities.Expand table
To learn about:
Read:
Requirements for Report Output Application, and how Visual FoxPro uses Report Output Applications
Migration Strategies and Changes in Output Rendering
You can use the design-time changes to improve all reports and labels, whether you choose backward-compatible or object-assisted reporting mode at run time.
When evaluating whether to switch to object-assisted reporting mode at run time, first consider items on the Reporting list of Important Changes in the Changes in Functionality for the Current Release topic, some of which are specific to this new method of creating output. .The topic includes a table of minor differences between backward-compatible and object-assisted reporting output. You can examine what effects these changes might have on individual existing reports, and use the recommendations in the table to address them. You will find additional details in the topic Using GDI+ in Reports.
Once you have experimented with your current reports, you can decide on a migration strategy for output:
You can switch applications over to use object-assisted reporting mode completely, by using the command SET REPORTBEHAVIOR 90.
You can use SET REPORTBEHAVIOR 90 but preface specific REPORT FORM commands for reports with formatting issues with SET REPORTBEHAVIOR 80, returning your application to object-assisted mode afterwards.
You can use object-assisted mode all the time, but adjust your ReportListener-derived classes’ behavior to suit specific needs. For example, you could change the default setting of the ReportListener’s DynamicLineHeight Property to False (.F.).
You can leave SET REPORTBEHAVIOR at its default setting of 80, and add an explicit OBJECT clause to specific reports at your leisure, as you have the opportunity to evaluate and adjust individual report and label layouts.
Printing, Rendering, and Character-set-handling Improvements
General changes to Visual FoxPro’s use of Windows’ printing, rendering and font-handling support the improvements in the Report System’s output. These changes enhance your ability to support multiple printers and multiple languages in reports.Expand table
To learn about:
Read:
GDI+ features and their impact on native Visual FoxPro output
Visual FoxPro reporting enhancements that allow your code to use GDI+ in object-assisted reporting mode, and Visual FoxPro Foundation Classes to get you started
Making full use of multiple character sets, or language scripts, in reports, for single report layout elements, for report defaults, or globally in Visual FoxPro
Changes to page setup dialog boxes in Visual FoxPro, improvements in your programmatic access to them, and providing overrides to Printer Environment settings in report and label files
Extensible Use of Report and Label Definition Files
Underneath all the changes to the Visual FoxPro Report System, the Report Designer and Report Engine handle your report and label definitions using the same .frx and .lbx file structures as they did in previous versions. They change the way they use certain fields, without making these reports and labels invalid in previous versions, and they also allow you to extend your use of existing fields or add custom fields.
Tip
This change is critical to your ability to create extensions of the new reporting features. For example, you might store two sets of ToolTips in two report extension fields, one set for use by developers and one for use by end-users. In a Report Builder extension, you could evaluate whether the Designer was working in protected or standard mode, and replace the actual set of ToolTips from the appropriate extension field. In previous versions, you could not add fields to report or label structure; the Designer and Engine would consider the table invalid. You also could not add custom content to unused, standard fields in various report and label records safely, because the Report Designer removed such content.
Visual FoxPro 9 provides a revised FILESPEC table for report and label files, with extensive information on the use of each column in earlier versions as well as current enhancements.
Visual FoxPro 9 also establishes a new, structured metadata format for use with reports. This format is an XML document schema shared with the Class Designer’s XML MemberData.
The XML document format allows you to pack custom reporting information into a single report or label field. The default Report Builder Application makes it easy to add Report XML MemberData to report and label records.Expand table
To learn about:
Read:
How Visual FoxPro uses .frx and .lbx tables, and how to extend these structures
Visual FoxPro contains the following additions and improvements to its data features:
Extended SQL Capabilities
Visual FoxPro contains many enhancements for SQL capabilities. For more information, see SQL Language Improvements.
New Data Types
Visual FoxPro includes the following new field and data types:
**Varchar **To store alphanumeric text without including padding by additional spaces at the end of the field or truncating trailing spaces, use the new Varchar field type.If you do not want Varchar fields translated across code pages, use the Varchar (Binary) field type. For more information, see Varchar Field Type.You can specify Varchar type mapping between ODBC, ADO, and XML data source types and CursorAdapter and XMLAdapter objects using the MapVarchar Property. You can also specify Varchar mapping for SQL pass-through technology and remote views using the MapVarchar setting in the CURSORSETPROP( ) function. For more information, see CURSORSETPROP( ) Function and CURSORGETPROP( ) Function.
**Varbinary **To store binary values and literals of fixed length in fields and variables without padding the field with additional zero (0) bytes or truncating any trailing zero bytes that are entered by the user, use the Varbinary data type. Internally, Visual FoxPro binary literals contain a prefix, 0h, followed by a string of hexadecimal numbers and are not enclosed with quotation marks (“”), unlike character strings. For more information, see Varbinary Data Type.You can specify binary type mapping between ODBC, ADO, and XML data source types and CursorAdapter and XMLAdapter objects using the MapBinary Property. You can also specify binary mapping for SQL pass-through technology and remote views using the MapBinary setting in the CURSORSETPROP( ) function. For more information, see CURSORSETPROP( ) Function and CURSORGETPROP( ) Function.
BlobTo store binary data with indeterminate length, use the Blob data type. For more information, see Blob Data Type.
Many of the Visual FoxPro language elements affected by these new data types are listed in the topics for the new data types.
Binary Index Tag Based on Logical Expressions
Visual FoxPro includes a new binary, or bitmap, index for creating indexes based on logical expressions, for example, indexes based on deleted records. A binary index can be significantly smaller than a non-binary index and can improve the speed of maintaining indexes. You can create binary indexes using the Table Designer or INDEX command. Visual FoxPro also includes Rushmore optimization enhancements in the SQL engine for deleted records.
You can convert expressions from one data type to another by using the new CAST( ) function. Using CAST( ) makes it possible for you to create SQL statements more compatible with SQL Server.
Get Cursor and Count Records Affected by SQL Pass-Thru Execution
By using the aCountInfo parameter of the SQLEXEC( ) and SQLMORERESULTS( ) functions, you can get the name of the cursor created and a count of the records affected by the execution of a SQL pass-through statement.
Roll-Back Functionality Supported when a SQL Pass-Through Connection Disconnects
Visual FoxPro now supports the DisconnectRollback property for use with the SQLSETPROP( ), SQLGETPROP( ), DBSETPROP( ), and DBGETPROP( ) functions. DisconnectRollback is a connection-level property that causes a transaction to be either rolled back or committed when the SQLDISCONNECT( ) function is called for the last connection handle associated with the connection.
The DisconnectRollback property accepts a logical value.
False (.F.) – (Default) The transaction will be committed when the SQLDISCONNECT( ) function is called for the last statement handle associated with the connection.
True (.T.) – The transaction is rolled back when the SQLDISCONNECT( ) function is called for the last statement handle associated with the connection.
The following example shows the DisconnectRollback property set in the DBSETPROP( ) and SQLSETPROP( ) functions.Copy
You can use the new SQLIDLEDISCONNECT( ) function to allow a SQL Pass-Through connection to be temporarily disconnected. Use the following syntax.Copy
SQLIDLEDISCONNECT( nStatementHandle )
The nStatementHandle parameter is set to the statement handle to be disconnected or 0 if all statement handles should be disconnected.
The SQLIDLEDISCONNECT( ) function returns the value 1 if it is successful; otherwise, it returns -1.
The function fails if the specified statement handle is busy or the connection is in manual commit mode. The AERROR( ) function can be used to obtain error information.
The disconnected connection handle is automatically restored if it is needed for an operation. The original connection data source name is used.
If a statement handle is temporarily released, the OBDChstmt property returns 0; the OBDChdbc returns 0 if the connection is temporarily disconnected. A shared connection is temporarily disconnected as soon as all of its statement handles are temporarily released.
Retrieving Active SQL Connection Statement Handles
You can retrieve information for all active SQL connection statement handles using the new ASQLHANDLES( ) function. ASQLHANDLES( ) creates and uses the specified array to store numeric statement handle references that you can use in other Visual FoxPro SQL functions, such as SQLEXEC( ) and SQLDISCONNECT( ). ASQLHANDLES( ) returns the number of active statement handles in use or zero (0) if none are available. For more information, see ASQLHANDLES( ) Function.
Obtain the ADO Bookmark for the Current Record in an ADO-Based Cursor
The ADOBookmark property is now supported by the CURSORGETPROP( ) function. Use this property to obtain the ActiveX® Data Objects (ADO) bookmark for the current record in an ADO-based cursor.
If a table is not selected and an alias is not specified, Error 52, “No table is open in the current work area,” is generated. If the cursor selected is not valid, Error 1467, “Property is invalid for local cursors,” is generated.
Obtain the Number of Fetched Records
You can obtain the number of fetched records during SQL Pass-Through execution by using the new RecordsFetched cursor property with the CURSORGETPROP( ) function.
Specifying the RecordsFetched cursor property will return the number of fetched records from an OBDC/ADO-based cursor.
If records have been deleted or appended locally, the RecordsFetched cursor property may not return the current number of records in the OBDC/ADO-based cursor. In addition, filter conditions are ignored.
You can determine if a fetch process is complete for an OBDC/ADO-based cursor by using the new FetchIsComplete cursor property with the CURSORGETPROP( ) function. Read-only at design time and run time.
This property is not supported on environment level (work area 0) cursors, tables, and local views.
The FetchIsComplete cursor property returns a logical expression True (.T.) if the fetch process is complete; otherwise False (.F.) is returned.
ISMEMOFETCHED( ) Determines Whether a Memo is Fetched
You can use the ISMEMOFETCHED( ) function to determine whether a Memo field or General field is fetched when you are using delayed memo fetching. For more information about delayed memo fetching, see Speeding Up Data Retrieval.
The ISMEMOFETCHED( ) function returns True (.T.) when the Memo field is fetched or if local data is used. ISMEMOFETCHED() returns NULL if the record pointer is positioned at the beginning of the cursor or past the last record.
The following table lists the data types along with their long type names and short type names.Expand table
Data Type
Long Type Name
Short Type Name
Character
Char, Character
C
Date
Date
D
DateTime
Datetime
T
Numeric
Num, Numeric
N
Floating
Float
F
Integer
Int, Integer
I
Double
Double
B
Currency
Currency
Y
Logical
Logical
L
Memo
Memo
M
General
General
G
Picture
Picture
P
Varchar
Varchar
V
Varbinary
Varbinary
Q
Blob
Blob
W
Visual FoxPro allows ambiguous long type names to be used with the ALTER TABLE, CREATE CURSOR, CREATE TABLE, and CREATE FROM commands. If the specified long type name is not a recognized long type name, Visual FoxPro will truncate the specified name to the first character.
Transaction Support for Free Tables and Cursors
In prior versions of Visual FoxPro, transactions using the BEGIN TRANSACTION Command were only supported for local and remote data from databases. Transactions involving free tables and cursors are now supported through use of the MAKETRANSACTABLE( ) and ISTRANSACTABLE( ) functions. For more information, see MAKETRANSACTABLE( ) Function and ISTRANSACTABLE( ) Function.
Specify a Code Page When Using the CREATE TABLE or CREATE CURSOR Commands
You can specify a code page by including the CODEPAGE clause with the CREATE CURSOR or CREATE TABLE commands.
When the CODEPAGE clause is specified, the new table or cursor has a code page specified by nCodePage. An error, 1914, “Code page number is invalid”, is generated if an invalid code page is specified.
The following example creates a table and displays its code page:Copy
CREATE TABLE Sales CODEPAGE=1251 (OrderID I, CustID I, OrderAmt Y(4))
Convert Character and Memo Data Types Using the ALTER TABLE Command
Visual FoxPro now supports automatic conversion from character data type to memo data type without loss of data when using the ALTER TABLE command along with the ALTER COLUMN clause. This conversion is also supported when making structural changes using the Table Designer. For more information, see ALTER TABLE – SQL Command.
BLANK Command Can Initialize Records to Default Value
You can initialize fields in the current record to their default values as stored in the table’s database container (DBC) by using the DEFAULT [AUTOINC] option when clearing the record with the BLANK command. For more information, see BLANK Command.
FLUSH Command Writes Data Explicitly to Disk
Visual FoxPro now includes options and parameters for the FLUSH command and FFLUSH function so you can explicitly save all changes you make to all open tables and indexes. You can also save changes to a specific table by specifying a work area, table alias, or a path and file name. For more information, see FLUSH Command and FFLUSH( ) Function.
Populate an Array with Aliases Used by a Specified Table
The new cTableName parameter for the AUSED( ) function makes it possible to filter the created array to contain only the aliases being used for a specified table.
Obtain Last Auto-Increment Value with GETAUTOINCVALUE( )
You can use the new GETAUTOINCVALUE( ) function to return the last value generated for an autoincremented field within a data session. For more information, see GETAUTOINCVALUE( ) Function.
Use SET VARCHARMAPPING to Control Query Result Set Mappings
For queries such as SELECT – SQL Command, character data is often manipulated using Visual FoxPro functions and expressions. Since the length of the resulting field value may be important for certain application uses, it is valuable to have this Character data mapped to Varchar data in the result set. The SET VARCHARMAPPING command controls whether Character data is mapped to a Character or Varchar data type. For more information, see SET VARCHARMAPPING Command.
SET TABLEVALIDATE Expanded
When a table header is locked during validation, attempts to open the table, for example, with the USE command, generate the message “File is in use (Error 3).” If the table header cannot be locked for a table open operation, you can suppress this message by setting the third bit for the SET TABLEVALIDATE command. You must also set the first bit to validate the record count when the table opens. Therefore, you need to set the SET TABLEVALIDATE command to a value of 5. Also, a fourth bit option (value of 8) is available for Insert operations which checks the table header before the appended record is saved to disk and the table header is modified.
You can specify fractions of a second for the nSeconds2 parameter to a minimum of 0.001 seconds. You can also specify the following values for the optional second parameter:
-1 – Always read data from a disk.
0 – Always use data in memory buffer but do not refresh buffer.
The Table refresh interval check box on the Data tab of the Options dialog box now also accepts fractional values.
SET REFRESH Can Differentiate Values for Each Cursor
You can use the new Refresh property with the CURSORGETPROP( ) function to differentiate the SET REFRESH values for individual cursors. The default setting is -2, which is a global value. This value is not available with the SET REFRESH command.
The Refresh property is available at the Data Session and Cursor level. The default setting for a Data Session level is -2 and the default value for a Cursor level is the current session’s level setting. If the global level setting is set to 0, the Cursor level setting is ignored.
If a table is not currently selected and an alias is not specified, Error 52, “No table is open in the current work area,” is generated.
You can use the new SYS(3092) function in conjunction with SYS(3054) to record the resulting output to a file.
SYS( 3092 [, cFileName [, lAdditive ]])
The cFileName parameter specifies the file to echo the SYS(3054) output to. Sending an empty string to cFileName will deactivate output recording to the file.
The default value for lAdditive is False (.F.). This specifies that new output will overwrite the previous contents of the specified file. To append new output to the specified file, set lAdditive to True (.T.).
SYS(3092) returns the name of the current echo file if it is active; otherwise, it returns an empty string.
SYS(3054) and SYS(3092) are global settings — in a multithreaded runtime they are scoped to a thread. Each function can be changed independently from each other.
These functions are not available in the Visual FoxPro OLE DB Provider.
Purge Cached Memory for Specific Work Area Using SYS(1104)
You can optionally specify the alias or work area of a specified table or cursor for which cached memory is purged. For more information, see SYS(1104) – Purge Memory Cache.
New Table Types for SYS(2029)
The SYS(2029) function returns new values for tables that contain Autoinc, Varchar, Varbinary or Blob fields. For more information, see SYS(2029) – Table Type.
Map Remote Unicode Data to ANSI Using SYS(987)
Use SYS(987) to map remote Unicode data retrieved through SQL pass-through or remote views to ANSI. This function can be used to retrieve remote Varchar data as ANSI for use with Memo fields. This setting is a global setting across all data sessions so should be used with care. For more information, see SYS(987) – Map Remote Data to ANSI.
Memo and Field tips in a BROWSE or Grid
When the mouse pointer is positioned over a Memo field cell in a Browse window or Grid control, a Memo Tip window displays the contents of the Memo field.
For other field types, positioning the mouse pointer over the field displays the field contents in a Field Tip window when the field is sized smaller than its contents.
Memo Tip windows display no more than 4 kilobytes of text, and are not displayed for binary data. A Memo Tip window is displayed until the mouse pointer is moved from the Memo field. The _TOOLTIPTIMEOUT System Variable determines how long a Field Tip window is displayed.
You can disable Memo Tips by setting the _SCREEN ShowTips Property to False (.F.).
Memo and Field Tips will also be displayed for Grid controls if both _SCREEN and the form’s ShowTips property are set to True (.T.). Additionally, the ToolTipText Property for the field’s grid column Textbox control must contain an empty string.
Specify Code Pages
You can specify the code page used to decode data when XML is being parsed and to encode data when UTF-8 encoded XML is generated. The following language changes are available:
nCodePage ParameterTo specify code pages, you can use the nCodePage parameter for the following XMLToTable methods:
MapVarchar Property Maps to Varchar, Varbinary, and Blob Data Types
For CursorAdapter and XMLAdapter classes, you can use the MapVarchar property to map to Varchar data types. To map to Varbinary and Blob data types, you can use the MapBinary property.
The XMLTOCURSOR( ) Function contains several new flags to support mapping of Char and base64Binary XML field types to new Fox data types.
Handling Conflict Checks with Properties for CursorAdapter Class
You can better handle conflicts when performing update and delete operations using the commands specified by the UpdateCmd and DeleteCmd properties for CursorAdapter objects by using the new ConflictCheckType and ConflictCheckCmd properties for CursorAdapter objects.
You can use ConflictCheckType to specify how to handle a conflict check during an update or delete operation. When ConflictCheckType is set to 4, you can use ConflictCheckCmd to specify a custom command to append to the end of the commands in the UpdateCmd and DeleteCmd properties.
Note
Visual FoxPro 8.0 Service Pack 1 includes the ConflictCheckType and ConflictCheckCmd properties.
The new TimestampFieldList property lets you specify a list of timestamp fields for the cursor created by the CursorAdapter. For more information see TimestampFieldList Property.
Auto-Refresh Support
There are a number of scenarios where you might want to have cursor data refreshed from a remote data source after an Insert/Update operation has occurred. These include following scenarios:
A table has an auto-increment field that also acts as a primary key.
A table has a timestamp field, and that field must be refreshed from the database after each Insert/Update in order to allow successful subsequent updates to the record when WhereType=4 (key and timestamp).
A table contains some fields which have DEFAULT values or triggers defined that will cause changes to occur.
The following new properties have been added to the CursorAdapter class for Auto-Refresh support:Expand table
Property
Description
InsertCmdRefreshFieldList
List of fields to refresh after Insert command executes.
InsertCmdRefreshCmd
Specifies the command to refresh the record after Insert command executes.
InsertCmdRefreshKeyFieldList
List of key fields to refresh in record after Insert command executes.
UpdateCmdRefreshFieldList
List of fields to refresh after Update command executes.
UpdateCmdRefreshCmd
Specifies the command to refresh the record after Update command executes.
UpdateCmdRefreshKeyFieldList
List of key fields to refresh the record after Update command executes.
RefreshTimestamp
Enables automatic refresh for fields in TimestampFieldList during Insert/Update.
In Visual FoxPro 8.0, the REFRESH( ) Function provides on demand record refresh functionality for local and remote views, however, it does not support this for the CursorAdapter. Visual FoxPro 9.0 extends REFRESH( ) support to the CursorAdapter and provides some additional capabilities:Expand table
Member
Description
RecordRefresh method
Refreshes the current field values for the target records. Use the CURVAL( ) Function to determine current field values.
BeforeRecordRefresh event
Occurs immediately before the RecordRefresh method is executed.
AfterRecordRefresh event
Occurs after the RecordRefresh method is executed.
RefreshCmdDataSourceType property
Specifies the data source type to be used for the RecordRefresh method.
RefreshCmdDataSource property
Specifies the data source to be used for the RecordRefresh method.
RefreshIgnoreFieldList property
List of fields to ignore during RecordRefresh operation
RefreshCmd property
Specifies the command to refresh rows when RecordRefresh is executed.
RefreshAlias property
Specifies the alias of read-only cursor used as a target for the refresh operation.
The CursorAdapter class has a FetchMemo Property, which when set to False (.F.) in Visual FoxPro 9.0 places the cursor in Delayed Memo Fetch mode similar to Remote Views. Delayed Memo Fetch Mode prevents the contents of Memo fields from being fetched using CursorFill Method or CursorRefresh Method. An attempt to fetch content for a Memo field is done when the application attempts to access the value. The following CursorAdapter enhancements provide support for Delayed Memo Fetch:Expand table
Member
Description
DelayedMemoFetch method
Performs a delayed Memo field fetch for a target record in a cursor in a CursorAdapter object.
FetchMemoDataSourceType property
Specifies the data source type used for the DelayedMemoFetch method.
FetchMemoDataSource property
Specifies the data source used for the DelayedMemoFetch method.
FetchMemoCmdList property
Specifies a list of Memo field names and their associated fetch commands.
The new UseTransactions property specifies whether the CursorAdapter should use transactions when sending Insert, Update or Delete commands through ADO or ODBC. For more information, see UseTransactions Property.
DEFAULT and CHECK Constraints Respected
In Visual FoxPro 9.0, DEFAULT values and table and field level CHECK constraints are supported for XML, Native, ADO and ODBC data sources. In Visual FoxPro 8.0, DEFAULT values and table and field level CHECK constraints are only supported for an XML data source. For the DEFAULT values and CHECK constraints to be applied to a cursor, call the CursorFill Method with the lUseSchema parameter set to True (.T.). For more information, see CursorSchema Property.
Remote Data Type Conversion for Logical Data
When you move data between a remote server and Visual FoxPro, Visual FoxPro uses ODBC or ADO data types to map remote data types to local Visual FoxPro data types. In Visual FoxPro 9.0, certain ODBC and ADO data types can now be mapped to a logical data type in remote views and the CursorAdapter object. For more information, see Data Type Conversion Control.
ADOCodePage Property
When working with an ADO data source for your CursorAdapter, you may want to specify a code page to use for character data translation. The new ADOCodePage property allows you to specify this code page. For more information, see ADOCodePage Property.
Read and Write Nested XML Documents
You can read to and write from your relational database into XML documents using nesting to handle the relationships between tables. You accomplish this using the RespectNesting Property of the XMLAdapter class. The XMLTable class has the Nest Method, Unnest Method and the following properties to handle nesting.
The LoadXML method accepts any XML document with a valid schema. Previously, the method required that the schema follow the format of a Visual Studio generated dataset. When you use the LoadXML method to read an XML document with a schema different from a Visual Studio generated dataset, the properties for the XMLAdapter, XMLName, and XMLPrefix properties are set to empty (“”). The XMLAdapterXMLNamespace property becomes equal to the target Namespace attribute value for the schema node and each XML element becomes a complexType and is mapped to an XMLTable object. The XMLNamespace property is set to namespaceURI for the element.
If you set the XMLAdapterRespectNesting property to True (.T.), the top level element declaration is ignored if it is referenced from some other complex element. For that case, the XMLTable object for the referenced element is nested into the XMLTable for the element that references it.
XPath Expressions Can Access Complex XML Documents
You can use XPath expressions to access complex XML documents and the new properties for reading the nodes within the document. For example, you might want to filter record nodes, restore relationships based on foreign key fields, use an element’s text as data for a field, or access XML that uses multiple XML namespaces. The following properties provide you with the ability to read the XML at the XMLAdapter level, XMLTable level, or the XMLField level.
You can use the following table to determine the node within the XML document that you want to start reading.
For example, if you use an XPath expression in the XMLName property for an XMLAdapter, reading begins at the first nodeExpand table
To read
Class
Context node
From the first found XML node:
XMLAdapter
IXMLDOMElement property
All found XML nodes and use each node as a single record:
XMLTable
XMLAdapter object
The first found XML node and use its text as a field value:
XMLField
XMLTable object
The following methods do not support the use of XPath expressions in the XMLName property:
The ApplyDiffgram and ChangesToCursor methods do not support XPath expressions for XMLAdapter and XMLTable objects.
The ToCursor method does not support an XPath expression for XMLAdapter when the IsDiffgram property is set to True (.T.).
The ToXML method does not support XPath expressions for XMLAdapter and XMLTable objects and ignores XMLField objects that use XPath expressions.
For more information about XPath expressions, see the XPath Reference in the Microsoft Core XML Services (MSXML) 4.0 SDK in the MSDN library at https://msdn.microsoft.com/library.
Cursor to XML Functions
Support for the following functions has been added to the OLE DB Provider for Visual FoxPro:
When used in the OLE DB Provider for Visual FoxPro, the _VFP VFPXMLProg property is not supported for the CURSORTOXML( ), XMLTOCURSOR( ) and XMLUPDATEGRAM( ) functions because the _VFP system variable is not supported in the OLE DB Provider.
EXECSCRIPT Supported in the Visual FoxPro OLE DB Provider
You can use the EXECSCRIPT( ) function with the Visual FoxPro OLE DB Provider. For more information, see EXECSCRIPT( ) Function.
Returning a Rowset from a Cursor in the Visual FoxPro OLE DB Provider
You can use the new SETRESULTSET( ), GETRESULTSET( ), and CLEARRESULTSET( ) functions to mark a cursor or table that has been opened by the Visual FoxPro OLE DB Provider, retrieve the work area of the marked cursor, and clear the marker flag from a marked cursor. By marking a cursor or table, you can retrieve a rowset that is created from the marked cursor or table from a database container (DBC) stored procedure when the OLE DB Provider completes command execution.
The SELECT – SQL Command and other SQL commands have been substantially enhanced in Visual FoxPro 9.0. This topic describes the enhancements made to these commands, and new commands that affect SQL performance.
Expanded Capacities
Several SELECT – SQL command limitations have been removed or increased in Visual FoxPro 9.0. The following table lists the areas where limitations have been removed or increased.Expand table
Capacity
Description
Number of Joins and Subqueries in a SELECT – SQL command
Visual FoxPro 9.0 removes the limit on the total number of join clauses and subqueries in a SELECT – SQL command. The previous limit was nine.
Number of UNION clauses in a SELECT – SQL command
Visual FoxPro 9.0 removes the limit on number of UNION clauses in a SQL SELECT statement. The previous limit was nine.
Number of tables referenced a SELECT – SQL command
Visual FoxPro 9.0 removes the limit on the number of tables and aliases referenced in a SQL SELECT statement. The previous limit was 30.
Number of arguments in an IN( ) clause
Visual FoxPro 9.0 removes the limit of 24 values in the IN (Value_Set) clause for the WHERE clause. However, the number of values remains subject to the setting of SYS(3055) – FOR and WHERE Clause Complexity. For functionality changes concerning the IN clause, see Changes in Functionality for the Current Release.
Subquery Enhancements
Visual FoxPro 9.0 provides more flexibility in subqueries. For example, multiple subqueries are now supported. The following describes the enhancements to subqueries in Visual FoxPro 9.0.
Multiple Subqueries
Visual FoxPro 9.0 supports multiple subquery nesting, with correlation allowed to the immediate parent. There is no limit to the nesting depth. In Visual FoxPro 8.0, error 1842 (SQL: Subquery nesting is too deep) was generated when more than one level of subquery nesting occurred.
The following is the general syntax for multiple subqueries.
SELECT … WHERE … (SELECT … WHERE … (SELECT …) …) …
Examples
The following example queries, which will generate an error in Visual FoxPro 8.0, are now supported in Visual FoxPro 9.0.Copy
CREATE CURSOR MyCursor (field1 I)
INSERT INTO MyCursor VALUES (0)
CREATE CURSOR MyCursor1 (field1 I)
INSERT INTO MyCursor1 VALUES (1)
CREATE CURSOR MyCursor2 (field1 I)
INSERT INTO MyCursor2 VALUES (2)
SELECT * FROM MyCursor T1 WHERE EXISTS ;
(SELECT * from MyCursor1 T2 WHERE NOT EXISTS ;
(SELECT * FROM MyCursor2 T3))
*** Another multiple subquery nesting example ***
SELECT * FROM table1 WHERE table1.iid IN ;
(SELECT table2.itable1id FROM table2 WHERE table2.iID IN ;
(SELECT table3.itable2id FROM table3 WHERE table3.cValue = "value"))
GROUP BY in a Correlated Subquery
Many queries can be evaluated by executing a subquery once and substituting the resulting value or values into the WHERE clause of the outer query. In queries that include a correlated subquery (also known as a repeating subquery), the subquery depends on the outer query for its values. This means that the subquery is executed repeatedly, once for each row that might be selected by the outer query.
Visual FoxPro 8.0 does not allow using GROUP BY in correlated subquery, and generates error 1828 (SQL: Illegal GROUP BY in subquery). Visual FoxPro 9.0 removes this limitation and supports GROUP BY for correlated subqueries allowed to return more than one record.
The following is the general syntax for the GROUP BY clause in a correlated subquery.
SELECT … WHERE … (SELECT … WHERE … GROUP BY …) …
Examples
The following example, which will generate an error in Visual FoxPro 8.0, is now supported in Visual FoxPro 9.0.Copy
CLOSE DATABASES ALL
CREATE CURSOR MyCursor1 (field1 I, field2 I, field3 I)
INSERT INTO MyCursor1 VALUES(1,2,3)
CREATE CURSOR MyCursor2 (field1 I, field2 I, field3 I)
INSERT INTO MyCursor2 VALUES(1,2,3)
SELECT * from MyCursor1 T1 WHERE field1;
IN (SELECT MAX(field1) FROM MyCursor2 T2 ;
WHERE T2.field2=T1.FIELD2 GROUP BY field3)
TOP N in a Non-Correlated Subquery
Visual FoxPro 9.0 supports the TOP N clause in a non-correlated subquery. The ORDER BY clause should be present if the TOP N clause is used, and this is the only case where it is allowed in subquery.
The following is the general syntax for the TOP N clause in a non-correlated subquery.
SELECT … WHERE … (SELECT TOP nExpr [PERCENT] … FROM … ORDER BY …) …
Examples
The following example, which will generate an error in Visual FoxPro 8.0, is now supported in Visual FoxPro 9.0.Copy
CLOSE DATABASES ALL
CREATE CURSOR MyCursor1 (field1 I, field2 I, field3 I)
INSERT INTO MyCursor1 VALUES(1,2,3)
CREATE CURSOR MyCursor2 (field1 I, field2 I, field3 I)
INSERT INTO MyCursor2 VALUES(1,2,3)
SELECT * FROM MyCursor1 WHERE field1 ;
IN (SELECT TOP 5 field2 FROM MyCursor2 order by field2)
Subqueries in a SELECT List
Visual FoxPro 9.0 allows a subquery as a column or a part of expression in a projection. A subquery in a projection has exactly the same requirements as a subquery used in a comparison operation. If a subquery does not return any records, NULL value is returned.
In Visual FoxPro 8.0, an attempt to use a subquery as a column or a part of expression in a projection would generate error 1810 (SQL: Invalid use of subquery).
The following is the general syntax for a subquery in a SELECT list.
SELECT … (SELECT …) … FROM …
Example
The following example, which will generate an error in Visual FoxPro 8.0, is now supported in Visual FoxPro 9.0.Copy
SELECT T1.field1, (SELECT field2 FROM MyCursor2 T2;
WHERE T2.field1=T1.field1) FROM MyCursor1 T1
Aggregate functions in a SELECT List of a Subquery
In Visual FoxPro 9.0, aggregate functions are now supported in a SELECT list of a subquery compared using the comparison operators <, <=, >, >= followed by ALL, ANY, or SOME. See Considerations for SQL SELECT Statements for more information about aggregate functions.
Example
The following example demonstrates the use of an aggregate function (the COUNT( ) function) in a SELECT list of a subquery.Copy
CLOSE DATABASES ALL
CREATE CURSOR MyCursor (FIELD1 i)
INSERT INTO MyCursor VALUES (6)
INSERT INTO MyCursor VALUES (0)
INSERT INTO MyCursor VALUES (1)
INSERT INTO MyCursor VALUES (2)
INSERT INTO MyCursor VALUES (3)
INSERT INTO MyCursor VALUES (4)
INSERT INTO MyCursor VALUES (5)
INSERT INTO MyCursor VALUES (-1)
CREATE CURSOR MyCursor2 (FIELD2 i)
INSERT INTO MyCursor2 VALUES (1)
INSERT INTO MyCursor2 VALUES (2)
INSERT INTO MyCursor2 VALUES (2)
INSERT INTO MyCursor2 VALUES (3)
INSERT INTO MyCursor2 VALUES (3)
INSERT INTO MyCursor2 VALUES (3)
INSERT INTO MyCursor2 VALUES (4)
INSERT INTO MyCursor2 VALUES (4)
INSERT INTO MyCursor2 VALUES (4)
INSERT INTO MyCursor2 VALUES (4)
SELECT * FROM MYCURSOR WHERE field1;
< ALL (SELECT count(*) FROM MyCursor2 GROUP BY field2) ;
INTO CURSOR MyCursor3
BROWSE
Correlated Subqueries Allow Complex Expressions to be Compared with Correlated Field
In Visual FoxPro 8.0, correlated fields can only be referenced in the following forms:
correlated field <comparison> local field
-or-
local field <comparison> correlated field
In Visual FoxPro 9.0. correlated fields support comparison to local expressions, as shown in the following forms:
correlated field <comparison> local expression
-or-
local expression <comparison> correlated field
A local expression must use at least one local field and cannot reference any outer (correlated) field.
Example
In the following example, a local expression (MyCursor2.field2 / 2) is compared to a correlated field (MyCursor.field1).Copy
SELECT * FROM MyCursor ;
WHERE EXISTS(SELECT * FROM MyCursor2 ;
WHERE MyCursor2.field2 / 2 > MyCursor.field1)
Changes for Expressions Compared with Subqueries.
In Visual FoxPro 8.0, the left part of a comparison using the comparison operators [NOT] IN, <, <=, =, ==, <>, !=, >=, >, ALL, ANY, or SOME with a subquery must reference one and only one table from the FROM clause. In case of a comparison with correlated subquery, the table must also be the correlated table.
In Visual FoxPro 9.0, comparisons work in the following ways:
The expression on the left side of an IN comparison must reference at least one table from the FROM clause.
The left part for the conditions =, ==, <>, != followed by ALL, SOME, or ANY must reference at least one table from the FROM clause.
The left part for the condition >, >=, <, <= followed by ALL, SOME, or ANY (SELECT TOP…) must reference at least one table from the FROM clause.
The left part for the condition >, >=, <, <= followed by ALL, SOME, or ANY (SELECT <aggregate function>…) must reference at least one table from the FROM clause.
The left part for the condition >, >=, <, <= followed by ALL, SOME, or ANY (subquery with GROUP BY and/or HAVING) must reference at least one table from the FROM clause.
In Visual FoxPro 9.0, the left part of a comparison that does not come from the list (for example, ALL, SOME, or ANY are not included) doesn’t have to reference any table from the FROM clause.
In all cases, the left part of the comparison is allowed to reference more than one table from the FROM clause. For a correlated subquery, the left part of the comparison does not have to reference the correlated table.
Subquery in an UPDATE – SQL Command SET List
In Visual FoxPro 9.0, the UPDATE – SQL Command now supports a subquery in the SET clause.
A subquery in a SET clause has exactly the same requirements as a subquery used in a comparison operation. If the subquery does not return any records, the NULL value is returned.
Only one subquery is allowed in a SET clause. If there is a subquery in the SET clause, subqueries in the WHERE clause are not allowed.
The following is the general syntax for a subquery in the SET clause.
UPDATE … SET … (SELECT …) …
Example
The following example demonstrates the use of a subquery in the SET clause.Copy
CLOSE DATA
CREATE CURSOR MyCursor1 (field1 I , field2 I NULL)
INSERT INTO MyCursor1 VALUES (1,1)
INSERT INTO MyCursor1 VALUES (2,2)
INSERT INTO MyCursor1 VALUES (5,5)
INSERT INTO MyCursor1 VALUES (6,6)
INSERT INTO MyCursor1 VALUES (7,7)
INSERT INTO MyCursor1 VALUES (8,8)
INSERT INTO MyCursor1 VALUES (9,9)
CREATE CURSOR MyCursor2 (field1 I , field2 I)
INSERT INTO MyCursor2 VALUES (1,10)
INSERT INTO MyCursor2 VALUES (2,20)
INSERT INTO MyCursor2 VALUES (3,30)
INSERT INTO MyCursor2 VALUES (4,40)
INSERT INTO MyCursor2 VALUES (5,50)
INSERT INTO MyCursor2 VALUES (6,60)
INSERT INTO MyCursor2 VALUES (7,70)
INSERT INTO MyCursor2 VALUES (8,80)
UPDATE MyCursor1 SET field2=100+(SELECT field2 FROM MyCursor2 ;
WHERE MyCursor2.field1=MyCursor1.field1) WHERE field1>5
SELECT MyCursor1
LIST
Sub-SELECT in the FROM Clause
A sub-SELECT is often referred to as a derived table. Derived tables are SELECT statements in the FROM clause referred to by an alias or a user-specified name. The result set of the SELECT in the FROM clause creates a table used by the outer SELECT statement. Visual FoxPro 9.0 permits the use of a subquery in the FROM clause.
A sub-SELECT should be enclosed in parentheses and an alias is required. Correlation is not supported. A sub-SELECT has the same syntax limitations as the SELECT command, but not the subquery syntax limitations. All sub-SELECTs are executed before the top most SELECT is evaluated.
The following is the general syntax for a subquery in the FROM clause.
SELECT … FROM (SELECT …) [AS] Alias…
Example
The following example demonstrates the use of a subquery in the FROM clause.Copy
SELECT * FROM (SELECT * FROM MyCursor T1;
WHERE field1 = (SELECT T2.field2 FROM MyCursor1 T2;
WHERE T2.field1=T1.field2);
UNION SELECT * FROM MyCursor2;
ORDER BY 2 desc) AS subquery
** Note that the following code will generate an error ** SELECT * FROM (SELECT TOP 5 field1 FROM MyCursor) ORDER BY field1
ORDER BY with Field Names in the UNION clause
When using a UNION clause in Visual FoxPro 8.0, you are forced to use numeric references in the ORDER BY clause. In Visual FoxPro 9.0, this restriction has been removed and you can use field names.
The referenced fields must be present in the SELECT list (projection) for the last SELECT in the UNION; that projection is used for ORDER BY operation.
Example
The following example demonstrates the use of a field names in the ORDER BY clause.Copy
CLOSE DATABASES all
CREATE CURSOR MyCursor(field1 I,field2 I)
INSERT INTO MyCursor values(1,6)
INSERT INTO MyCursor values(2,5)
INSERT INTO MyCursor values(3,4)
SELECT field1, field2, .T. AS FLAG,1 FROM MyCursor;
WHERE field1=1;
UNION ;
SELECT field1, field2, .T. AS FLAG,1 FROM MyCursor;
WHERE field1=3;
ORDER BY field2 ;
INTO CURSOR TEMP READWRITE
BROWSE NOWAIT
Optimized TOP N Performance
In Visual FoxPro 8.0 and earlier versions, when using the TOP N [PERCENT] clause all records are sorted and then the TOP N are extracted. In Visual FoxPro 9.0, performance has been improved by eliminating records that do not qualify for the TOP N from the sort process as early as possible.
Optimization requires that TOP N return no more than N records (this is not the case for Visual FoxPro 8.0 and earlier versions) which is enforced if SET ENGINEBEHAVIOR is set to 90.
TOP N PERCENT cannot be optimized unless the whole result set can be read into memory at once.
Improved Optimization for Multiple Table OR Conditions
Visual FoxPro 9.0 provides for improved Rushmore optimization involving multi-table OR conditions. Visual FoxPro uses multi-table OR conditions to Rushmore optimize filter conditions for a table as long as both sides of the condition can be optimized. The following example shows this:Copy
CLEAR
CREATE CURSOR Test1 (f1 I)
FOR i=1 TO 20
INSERT INTO Test1 VALUES (I)
NEXT
INDEX ON f1 TAG f1
CREATE CURSOR Test2 (f2 I)
FOR i=1 TO 20
INSERT INTO Test2 VALUES (I)
NEXT
INDEX ON f2 TAG f2
SYS(3054,12)
SELECT * from Test1, Test2 WHERE (f1 IN (1,2,3) AND f2 IN (17,18,19)) OR ;
(f2 IN (1,2,3) AND f1 IN (17,18,19)) INTO CURSOR Result
SYS(3054,0)
In this scenario, table Test1 can be Rushmore optimized using the following condition:
(f1 IN (1,2,3) OR f1 IN (17,18,19))and table Test2 with the following:
(f2 IN (17,18,19) OR f2 IN (1,2,3))
Support for Local Buffered Data
At times it can be beneficial to use SELECT – SQL to select records from a local buffered cursor in which the table has not been updated. Many times when creating controls like grids, list boxes, and combo boxes it is necessary to consider newly added records which have not yet been committed to disk. Currently, SQL statements are based on content that is already committed to disk.
Visual FoxPro 9.0 provides language enhancements that allow you to specify if the data returned by a SELECT – SQL command is based on buffered data or data written directly to disk.
The SELECT – SQL command now supports a WITH … BUFFERING clause that lets you specify if retrieved data is based on buffered data or data written directly to disk. For more information, see SELECT – SQL Command – WITH Clause.
If you do not include the BUFFERING clause, the retrieved data is then determined by the setting for SET SQLBUFFERING command. For more information, see the SET SQLBUFFERING Command.
In Visual FoxPro 9.0, a UNION clause is now supported in the INSERT – SQL Command.
The following is the general syntax for the UNION clause.
INSERT INTO … SELECT … FROM … [UNION SELECT … [UNION …]]
Example
The following example demonstrates the use of a UNION clause in INSERT-SQL.Copy
CLOSE DATABASES ALL
CREATE CURSOR MyCursor (field1 I,field2 I)
CREATE CURSOR MyCursor1 (field1 I,field2 I)
CREATE CURSOR MyCursor2 (field1 I,field2 I)
INSERT INTO MyCursor1 VALUES (1,1)
INSERT INTO MyCursor2 VALUES (2,2)
INSERT INTO MyCursor SELECT * FROM MyCursor1 UNION SELECT * FROM MyCursor2
SELECT MyCursor
LIST
Correlated UPDATE – SQL Commands
Visual FoxPro 9.0 now supports correlated updates with the UPDATE – SQL Command.
If a FROM clause is included in the UPDATE -SQL command, then the name after UPDATE keyword defines the target for the update operation. This name can be a table name, an alias, or a file name. The following logic is used to select the target table:
If the name matches an implicit or explicit alias for a table in the FROM clause, then the table is used as a target for the update operation.
If the name matches the alias for the cursor in the current data session, then the cursor is used as a target.
A table or file with the same name is used as a target.
The UPDATE -SQL command FROM clause has the same syntax as the FROM clause in the SELECT – SQL command with the following limitations:
The target table or cursor cannot be involved in an OUTER JOIN as a secondary table.
The target cursor cannot be a subquery result.
All other JOINs can be evaluated before joining the target table.
The following is the general syntax for a correlated UPDATE command.
UPDATE … SET … FROM … WHERE …
Example
The following example demonstrates a correlated update using the UPDATE -SQL command.Copy
CLOSE DATABASES ALL
CREATE CURSOR MyCursor1 (field1 I , field2 I NULL,field3 I NULL)
INSERT INTO MyCursor1 VALUES (1,1,0)
INSERT INTO MyCursor1 VALUES (2,2,0)
INSERT INTO MyCursor1 VALUES (5,5,0)
INSERT INTO MyCursor1 VALUES (6,6,0)
INSERT INTO MyCursor1 VALUES (7,7,0)
INSERT INTO MyCursor1 VALUES (8,8,0)
INSERT INTO MyCursor1 VALUES (9,9,0)
CREATE CURSOR MyCursor2 (field1 I , field2 I)
INSERT INTO MyCursor2 VALUES (1,10)
INSERT INTO MyCursor2 VALUES (2,20)
INSERT INTO MyCursor2 VALUES (3,30)
INSERT INTO MyCursor2 VALUES (4,40)
INSERT INTO MyCursor2 VALUES (5,50)
INSERT INTO MyCursor2 VALUES (6,60)
INSERT INTO MyCursor2 VALUES (7,70)
INSERT INTO MyCursor2 VALUES (8,80)
CREATE CURSOR MyCursor3 (field1 I , field2 I)
INSERT INTO MyCursor3 VALUES (6,600)
INSERT INTO MyCursor3 VALUES (7,700)
UPDATE MyCursor1 SET MyCursor1.field2=MyCursor2.field2, field3=MyCursor2.field2*10 FROM MyCursor2 ;
WHERE MyCursor1.field1>5 AND MyCursor2.field1=MyCursor1.field1
SELECT MyCursor1
LIST
UPDATE MyCursor1 SET MyCursor1.field2=MyCursor3.field2 FROM MyCursor2, MyCursor3 ;
WHERE MyCursor1.field1>5 AND MyCursor2.field1=MyCursor1.field1 AND MyCursor2.field1=MyCursor3.field1
SELECT MyCursor1
LIST
Correlated DELETE – SQL Commands
Visual FoxPro 9.0 now supports correlated deletions with the DELETE – SQL Command.
If a FROM clause has more than one table, the name after the DELETE keyword is required and it defines the target for the delete operation. This name can be a table name, an alias or a file name. The following logic is used to select the target table:
If the name matches an implicit or explicit alias for a table in the FROM clause, then the table is used as a target for the update operation.
If the name matches the alias for the cursor in the current data session, then the cursor is used as a target.
A table or file with the same name is used as a target.
The DELETE -SQL command FROM clause has the same syntax as the FROM clause in the SELECT – SQL command with the following limitations:
The target table or cursor cannot be involved in an OUTER JOIN as a secondary table.
The target cursor cannot be a subquery result.
It should be possible to evaluate all other JOINs before joining the target table.
The following is the general syntax for a correlated DELETE command.
DELETE [alias] FROM alias1 [, alias2 … ] … WHERE …
Example
The following example demonstrates a correlated deletion using the DELETE -SQL command.Copy
CLOSE DATABASES ALL
CREATE CURSOR MyCursor1 (field1 I , field2 I NULL,field3 I NULL)
INSERT INTO MyCursor1 VALUES (1,1,0)
INSERT INTO MyCursor1 VALUES (2,2,0)
INSERT INTO MyCursor1 VALUES (5,5,0)
INSERT INTO MyCursor1 VALUES (6,6,0)
INSERT INTO MyCursor1 VALUES (7,7,0)
INSERT INTO MyCursor1 VALUES (8,8,0)
INSERT INTO MyCursor1 VALUES (9,9,0)
CREATE CURSOR MyCursor2 (field1 I , field2 I)
INSERT INTO MyCursor2 VALUES (1,10)
INSERT INTO MyCursor2 VALUES (2,20)
INSERT INTO MyCursor2 VALUES (3,30)
INSERT INTO MyCursor2 VALUES (4,40)
INSERT INTO MyCursor2 VALUES (5,50)
INSERT INTO MyCursor2 VALUES (6,60)
INSERT INTO MyCursor2 VALUES (7,70)
INSERT INTO MyCursor2 VALUES (8,80)
CREATE CURSOR MyCursor3 (field1 I , field2 I)
INSERT INTO MyCursor3 VALUES (6,600)
INSERT INTO MyCursor3 VALUES (7,700)
DELETE MyCursor1 FROM MyCursor2 ;
WHERE MyCursor1.field1>5 AND MyCursor2.field1=MyCursor1.field1
SELECT MyCursor1
LIST
RECALL ALL
DELETE MyCursor1 FROM MyCursor2, MyCursor3 ;
WHERE MyCursor1.field1>5 AND MyCursor2.field1=MyCursor1.field1 AND MyCursor2.field1=MyCursor3.field1
SELECT MyCursor1
LIST
RECALL ALL
DELETE FROM MyCursor1 WHERE MyCursor1.field1>5
SELECT MyCursor1
list
RECALL ALL
DELETE MyCursor1 from MyCursor1 WHERE MyCursor1.field1>5
RECALL ALL IN MyCursor1
DELETE T1 ;
FROM MyCursor1 T1 JOIN MyCursor2 ON T1.field1>5 AND MyCursor2.field1=T1.field1, MyCursor3 ;
WHERE MyCursor2.field1=MyCursor3.field1
RECALL ALL IN MyCursor1
Updatable Fields in UPDATE – SQL Command
The number of fields that can be updated with the UPDATE – SQL Command is no longer limited to 128 as in prior versions of Visual FoxPro. You are now limited to 255, which is the number of fields available in a table.
SET ENGINEBEHAVIOR
The SET ENGINEBEHAVIOR Command has a new Visual FoxPro 9.0 option, 90, that affects SELECT – SQL command behavior for the TOP N clause and aggregate functions. For additional information, see the SET ENGINEBEHAVIOR Command.
Data Type Conversion
Conversion between data types (for example, conversion between memo and character fields) has been improved in Visual FoxPro 9.0. This conversion improvement applies to the ALTER TABLE – SQL Command with the COLUMN option as well as structural changes made with the Table Designer.
Visual FoxPro contains the following enhancements to classes, forms, controls and object-oriented related syntax.
Anchoring Visual Controls
You can anchor a visual control to one or more edges of its parent container using the control’s Anchor property. When you anchor a visual control to the parent container, the edges of the control remain in the same position relative to the edges of the container when you resize the container. For more information, see Anchor Property.
Docking Forms
Visual FoxPro extends docking support to user-defined forms. Docking forms works similarly to docking toolbars except that you can dock forms to Visual FoxPro Interactive Development Environment (IDE) windows and other forms, and controls on the form can still obtain focus when the form is docked.
Visual FoxPro includes the following new and updated properties, methods, and events to support docking forms.
CheckBox and OptionButton Controls Support Wordwrapping
The WordWrap property is now supported for CheckBox and OptionButton controls. The text portions of these controls now use wordwrapping. For more information, see WordWrap Property.
CommandButton Controls Can Align Text with Pictures
The Alignment property now applies to CommandButton controls when specifying an image for the Picture property and setting the PicturePosition property to a value other than the default. The Alignment property also contains new and revised settings for CommandButton, CheckBox, and OptionButton controls. For more information, see Alignment Property.
CommandButton, OptionButton, and CheckBox Controls Can Hide Captions
The PicturePostion property contains a new setting of 14 (No text) for CommandButton, OptionButton, and CheckBox controls. You can use this setting to hide the text portions of these controls without needing to set the Caption property to an empty string. This setting is particularly useful when you want to include a hotkey for a button with a graphic without displaying the Caption text. You must set the Style property to 1 (Graphical) for this new setting to apply.
In addition, the PicturePosition property now applies to CheckBox and OptionButton controls when Style is set to 1 (Graphical).
PictureMargin and PictureSpacing Properties Control Spacing and Margins on CommandButton, OptionButton, and CheckBox Controls
You can better control positioning of images on CommandButton, OptionButton, and CheckBox controls with the new PictureMargin and PictureSpacing properties. The PictureMargin property specifies margin spacing in pixels between an image and the control’s border as determined by the PicturePosition property. The PictureSpacing property specifies margin spacing in pixels between an image and text on the control.
Collection Objects Support in ComboBox and ListBox Controls
You can now specify Collection objects as the row source and row source type for the RowSource and RowSourceType properties of ComboBox and ListBox controls. For more information, see RowSource Property and RowSourceType Property.
Setting Ascending or Descending Indexes on Cursors in the DataEnvironment
You can specify ascending or descending order for a cursor index by using the new OrderDirection property for Cursor objects.
Note
OrderDirection is disregarded when the cursor’s Order property is empty.
Mouse Pointer Control for Grid Columns and Column Headers
The MousePointer and MouseIcon properties now apply to Column objects in a grid and Header objects in a column. For the MousePointer property, you can specify the new setting of 16 (Down Arrow) to reset the mouse pointer for a column header to the default down arrow.
You can use the new Rotation property to rotate Label controls. The Rotation property applies to Line and Shape controls when used with the new PolyPoints property. For more information, see Rotation Property (Visual FoxPro), PolyPoints Property, and Creating More Complex Shapes using the PolyPoints Property.
Label Controls Can Display Themed Background
For Label controls, you can set the Style property to Themed Background Only to show only themed background colors when Windows themes are turned on. The label background color is the same as the parent container for the label. For more information, see Style Property.
ListBox Controls Can Hide Scroll Bars
You can use the new AutoHideScrollBar property for ListBox controls to hide scroll bars when the list contains less than the number of items that can be visible in the list box. For more information, see AutoHideScrollBar Property.
Toolbar Controls Can Display Horizontal Separator Objects
For Separator objects, set the Style property to 1 to display a horizontal line or a vertical line, depending on how the toolbar appears. If the toolbar appears horizontally, the line displays vertically. If the toolbar appears vertically, the line displays horizontally. In versions prior to this release, setting Style to 1 displayed a vertical line only.
Note
In versions prior to this release, undocked vertical system and user-defined toolbars did not display horizontal separators. In the current release, horizontal separators now display for vertical undocked toolbars.
The Visible property now applies to Separator objects so you can control whether a Separator object displays in Toolbar controls. When used in combination with the Style property, the separator’s Visible property determines whether a space or line displays as the separator when its Style property is set to 0 (Normal – do not display a line) or 1 (display a horizontal or vertical line), respectively.
You can use the new PolyPoints property for Line and Shape controls to create polygon lines and shapes. PolyPoints specifies an array of any dimension containing coordinates with the format of X1, Y1, X2, Y2, …, organized in the order in which the polygon line or shape is drawn.
For Line controls, when you create a polygon line using the PolyPoints property, you can specify the new setting of “S” or “s” for the LineSlant property to create a Bezier curve.
You can now use the NODEFAULT command in the DropDown event for a ComboBox control. This will prevent the drop-down list portion of a ComboBox control from appearing. For more information, see NODEFAULT Command.
NEWOBJECT( ) Creates Objects without Raising Initialization Code
To mimic the behavior of a class opened in Class Designer or Form Designer, pass 0 to the cInApplication parameter. This feature allows you to create design-time tools that view the structure of a class.
By passing 0 to the cInApplication parameter for the NEWOBJECT( ) function, Visual FoxPro allows you to create an instance of a class without raising initialization code (such as code in the Init, Load, Activate, and BeforeOpenTables events). Furthermore, when the object is released, it does not raise its destructor code (such as code in the Destroy and Unload events). Only initialization and destructor code is suppressed; code in other events and methods is still called.
If you use the cInApplication parameter to suppress initialization and destructor code in an object, you also suppress it in the object’s child objects.
Specify Where Focus is Assigned in the Valid Event
To direct where focus is assigned, you can use the optional ObjectName parameter in the RETURN command of the Valid event. The object specified must be a valid Visual FoxPro object. If the specified object is disabled or cannot receive focus, then focus is assigned to the next object in the tab order. If an invalid object is specified, Visual FoxPro keeps the focus at the current object.
You can now set focus to objects in the following scenarios:
Set focus to an object on another visible form.
Set focus to an object on a non-visible Page or Pageframe control.
TextBox Controls Have Auto-Completion Functionality
You can add auto-completion functionality to your text box controls to make data entry more efficient. Auto-completion is the automatic display of a drop-down list of entries that match the string as it is typed into the text box. The entries come from a special table that tracks unique values entered into the text box, the control name that is the source of the value, and usage information.
By the setting the AutoComplete property, you determine the sort order for the entries. If you want more control over the list and where it is stored, you can use the AutoCompSource property to specify the table used to populate the automatic list. By default, the table is AUTOCOMP.DBF. You can use one table for each text box control or a single table can populate automatic lists for several text boxes.
If you use a single table, which is the default, the table uses values in the Source field for each entry to identify the text box control associated with the entry. By default, the Source field value is the name of the text box control. You can specify the Source field value using the AutoCompSource property of the text box. For example, you might want to make the same set of entries available to multiple Text box controls within the application such as address information. You can explicitly set the AutoCompTable and AutoCompSource properties for each of the controls to the same table and source field value. The same automatic list appears for each of them.
The text box control handles updating the auto-completion table for you based on the values actually entered in the text box. If you want to remove a value from the list, enter a string in a text box that matches the string you want to delete to filter the list for it. Select the entry in the list and press the DELETE key. The string remains in the table but no longer appears in the automatic list.
The following new InputMask and Format settings are available:
InputMask PropertyExpand table
cMask
Description
U
Permits alphabetic characters only and converts them to uppercase (A – Z).
W
Permits alphabetic characters only and converts them to lowercase (a – z).
Format PropertyExpand table
cFunction
Description
Z
Displays the value as blank if it is 0, except when the control has focus.Dates and DateTimes are also supported in these controls. The date and datetime delimiters are not displayed unless the control has focus.
The Image control’s new PictureVal property can be used instead of the Picture Property (Visual FoxPro) to specify a character string expression or object of an image. For an object, the format must be of an IPicture interface format compatible with LOADPICTURE( ) Function.
The CLEAR CLASSLIB command now automatically executes a CLEAR CLASS command on each class in the specified class library. Any errors that might occur during release of individual classes (e.g., class in use) are ignored.
Note
Classes in other class libraries that are used or referenced by a class in the specified class library are not cleared.
In prior versions of Visual FoxPro, the definable maximum area for a form is limited to twice the Screen Resolution for both X and Y dimensions. For example, if your monitor resolution is 1280×1024, then the max dimensions would be:Copy
Form.Width = 2552
Form.Height = 2014
Additionally, if you attempted to set Width and Height properties to these limits in design-time and then ran the form, you would see that the values have reverted to screen resolution limits (being that they were saved this way):Copy
Form.Width = 1280
Form.Height = 998
In Visual FoxPro 9.0, this limitation has been increased to approximately 32,000 pixels for each dimension and now allows for more flexibility with certain forms such as scrollable ones:Copy
New events have been added to the ProjectHook class, which allow you to perform source code control operations such as check-in and check-out of multiple files at once.
You can specify the visibility (Protected, Hidden or Public) and description of a property using the AddProperty method with new available parameters. These settings can also be controlled from the New Property Dialog Box and Edit Property/Method Dialog Box. For more information, see AddProperty Method.
WriteMethod Method Supports Design Time Settings
You can specify the visibility (Protected, Hidden or Public) and description of a method using the WriteMethod method with new available parameters. These settings can also be controlled from the New Property Dialog Box and Edit Property/Method Dialog Box. For more information, see WriteMethod Method.
In the current release of Visual FoxPro, you will find enhanced functionality via new and enhanced commands and functions. Expand and collapse this topic to see information about miscellaneous language additions and improvements.
Class Enhancements
Visual FoxPro contains significant language enhancements for classes, forms, controls, and object-oriented related features. For more information, see Class Enhancements.
Visual FoxPro contains a number of language enhancements to support new Reporting functionality:
REPORT FORM CommandDisplays or prints out a report specified by a report definitions file. This command has been enhanced to support Report Listener objects.
SET REPORTBEHAVIOR CommandControls use of Report Preview and Report Output applications with the Visual FoxPro Report System.
Additionally, there are improvements to the following related Printing language elements:
SYS(1037) – Page Setup Dialog BoxDisplays Visual FoxPro default or report Page Setup dialog box or sets printer settings for the default printer in Visual FoxPro or for the report printer environment. In this version, a new nValue parameter is available.
APRINTERS( ) FunctionReturns a five-column array with the name of the printer, connected port, driver, comment, and location. The last three columns are available if the new optional parameter is passed.
GETFONT( ) FunctionContains an additional setting to display only those fonts available on the current default printer and clarified values for the language script.
New Reporting functionality is described in more detail in separate Reporting topics. For more information, see Guide to Reporting Improvements.
Specifying Arrays with More Than 65K Elements
You can now specify arrays containing more than 65,000 elements, for example, when using the DIMENSION command. Normal arrays and member arrays have a new limit of 2GB. Arrays containing member objects retain a limit of 65,000 elements.
Note
Array sizes can also be limited by available memory, which affects performance, especially for very large arrays. Make sure your computer has enough memory to accommodate the upper limits of your arrays.
For operations such as the DO command, you can change the default number of nesting levels from 128 levels to 32 and up to 64,000 levels of nesting by including the new STACKSIZE setting in a Visual FoxPro configuration file.
Note
You can change the nesting level only during Visual FoxPro startup.
In previous versions of Visual FoxPro, the size of a procedure or program could not exceed 65K. Visual FoxPro now removes this restriction for programs and procedures. For more information, see Visual FoxPro System Capacities.
PROGCACHE Configuration File Setting
In previous versions of Visual FoxPro, you could not specify the program cache size or amount of memory reserved to run programs. This configuration file setting allows you to control this. It is especially useful for MTDLL scenarios. For more information, see Special Terms for Configuration Files.
ICASE( ) Function
The new ICASE( ) function makes it possible for you to evaluate a list of conditions and return results depending on the result of evaluating those conditions. For more information, see ICASE( ) Function.
TTOC( ) Converts DateTime Expressions to XML DateTime Format
You can convert a DateTime expression into a character string in XML DateTime format by passing a new optional value of 3 to the TTOC( ) function. For more information, see TTOC( ) Function.
SET COVERAGE Command Available at Run Time
The SET COVERAGE command is now available at run time so that you can debug errors that occur at run time but not at design time. For more information, see SET COVERAGE Command.
CLEAR ERROR Command
The new ERROR clause for the CLEAR command makes it possible to reset the error structures as if no error occurred. This affects the following functions:
The AERROR( ) function will return 0.
The ERROR( ) function will return 0.
The output from MESSAGE( ), MESSAGE(1) and SYS(2018) will return a blank string.
The CLEAR command should not be used with the ERROR clause within a TRY…CATCH…FINALLY structure. For more information, see CLEAR Commands.
Write Options Dialog Settings to Registry Using SYS(3056)
The SYS(3056) function can now be used to write out settings from the Options dialog box to the registry.
SYS(3056 [, nValue ])
The following table lists values for nValue.Expand table
nValue
Description
1
Update only from registry settings, with the exception of SET commands and file locations.
Visual FoxPro now includes the FOXOBJECT keyword for the FOR EACH … ENDFOR command to support preservation of native Visual FoxPro object types.
FOR EACH objectVar [AS Type [OF ClassLibrary ]] IN Group FOXOBJECT
Commands
[EXIT]
[LOOP]
ENDFOR | NEXT [Var]
The FOXOBJECT keyword specifies that the objectVar parameter created will be a Visual FoxPro object. The FOXOBJECT keyword only applies to collections where the collection is based on a native Visual FoxPro Collection class. Collections that are COM-based will not support the FOXOBJECT keyword.
The SET PATH command now supports the ADDITIVE keyword. The ADDITIVE keyword appends the specified path to the end of the current SET PATH list. If the path already exists in the SET PATH list, Visual FoxPro does not add it or change the order of the list. Paths specified with the ADDITIVE keyword must be strings in quotes or valid expressions.
In addition, the length of the SET PATH list has been increased to 4095 characters.
You can specify that the trim is case-insensitive using the nFlag value of 0 bit and 1.
The cParseChar parameter specifies one or more character strings to be trimmed from cExpression. A maximum of 23 strings can be specified in cParseChar.
By default, if cParseChar is not specified, then leading and trailing spaces are trimmed from character strings or 0 bytes are removed for Varbinary data types.
The cParseChar parameters are applied in the order they are entered. When a match is found, cExpression is truncated and the process repeats from the first cParseChar parameter.
The ALINES( ) function has been enhanced to provide several additional options such as case-insensitive parsing and improved handling of empty array elements. These options are available using the new nFlags parameter that replaces the older lTrim 3rd parameter. For more information, see ALINES( ) Function.
Improvements in TEXT…ENDTEXT Statement
You can use the TEXT…ENDTEXT command to eliminate line feeds using the new PRETEXT setting. A new FLAGS parameter controls additional output settings. For more information, see TEXT … ENDTEXT Command.
Include Delimiters in STREXTRACT( ) Results
The STREXTRACT( ) function has a new nFlags setting that allows you to include the specified delimiters with the returned expression. For more information, see STREXTRACT( ) Function.
STRCONV( ) Enhanced to Allow for Code Page and FontCharSet
For certain conversion settings, you can specify an optional Code Page or Fontcharset setting for use in the conversion. For more information, see STRCONV( ) Function.
TYPE( ) Determines if an Expression is an Array
The TYPE( ) function accepts the parameter, 1, to evaluate an expression to determine if it is an array.
Type(cExpression, 1)
The following character values are returned if the 1 parameter is specified.Expand table
BINTOC( ) and CTOBIN( ) Have Additional Conversion Capabilities
The BINTOC( ) and CTOBIN( ) functions have update or new parameters that provide you with more control over the output of these functions. Additionally, these enhancements offer some improvements for working with Win32 API routines. For more information, see BINTOC( ) Function and CTOBIN( ) Function.
MROW( ) and MCOL( ) Can Detect the Position of the Mouse Pointer
The MROW( ) and MCOL( ) functions now have a zero (0) parameter for detecting the position of the mouse pointer based on the currently active form instead of the form returned by the WOUTPUT( ) function. Although they are typically reference the same form, if the AllowOutput property of the form is set to False (.F.), WOUTPUT( ) does not return the current active form. The mismatch of references and can lead to unexpected results. By using the zero (0) parameter, you can avoid misplacing items, such as Shortcut menus, as the currently active form is always used.
The INPUTBOX( ) function contains an additional parameter that allows you to determine if the user canceled out of the dialog. For more information, see INPUTBOX( ) Function.
AGETCLASS( ) Supported for Runtime Applications
The AGETCLASS( ) fiunction is now supported for runtime applications. For more information, see AGETCLASS( ) Function.
Visual FoxPro will display a tip such as “CTRL+Click to follow the link” when you hover over a hyperlink in the editor. If you desire to not have this tip show, you can use SYS(3008) to turn it off. This function is also useful for international applications where you do not want to display the English text for this tip. For more information, see SYS(3008) – Hyperlink Tooltips.
SYS(3065) Internal Program Cache
You can obtain the internal program cache (PROGCACHE configuration file setting). For more information, see SYS(3065) – Internal Program Cache.
SYS(3101) COM Code Page Translation
You can now specify a code page to use for character data translation involving COM interoperability. For more information, see SYS(3101) – COM Code Page Translation.
Bidirectional Support for Tooltips and Popups
For international applications that display text from right to left, you can use the following new enhancements to control text justification:
SYS(3009) – right justifies text in ToolTips.
DEFINE POPUP…RTLJUSTIFY – right justifies items in a popup, such as a shortcut menu.
SET SYSMENU TO RTLJUSTIFY – right justifies an entire menu system.
Visual FoxPro 9.0 contains a number of enhancements that extend ability to specify a Font Language Script (or FontCharSet) along with existing Font settings:
SYS(3007) – specifies a FontCharSet for ToolTips. This is a global setting.
FONT Clause – the following table lists commands that support an optional FONT clause that allows for specification of a FontCharSet in the following format:FONT cFontName [, nFontSize [, nFontCharSet]]Expand tableCommandDEFINE MENUDEFINE POPUPDEFINE BARDEFINE PADDEFINE WINDOWMODIFY WINDOWBROWSE/EDIT/CHANGE?/??
Browse – the Font Dialog Box that you can invoke by selecting the Font menu item from the Table menu with a Browse Window active now allows for selection of a font language script. You can specify a global default font script from the IDE Tab, Options Dialog Box in the Options Dialog Box (Visual FoxPro). To do this, you must first check the Use font script checkbox.
Editors – the Font Dialog Box that you can invoke with an editor window active by selecting the Font menu item from the Format menu or right-click shortcut menu Edit Properties Dialog Box now allows for selection of a font language script. You can specify a global default font script from the IDE Tab, Options Dialog Box in the Options Dialog Box (Visual FoxPro). To do this, you must first check the Use font script checkbox.
You can specify how long a ToolTip is displayed if the mouse pointer is left stationary. For more information, see _TOOLTIPTIMEOUT System Variable.
Tablet PC Features
The following features are available to assist with applications designed to run on a Tablet PC computer.
ISPEN( ) – determines if the last Visual FoxPro application mouse event on a Tablet PC was a pen tap.
_SCREEN.DisplayOrientation – this read-write property specifies the screen display orientation for a Tablet PC. The value returned is the current orientation.
_TOOLTIPTIMEOUT – specifies how long a ToolTip is displayed if the mouse pointer is left stationary.
Visual FoxPro allows you to trap and handle window messages from the Microsoft® Windows® operating system using existing BINDEVENT functions. Some examples of common events you might wish to trap for include:
A power broadcast message used to intercept standby or power-down activities.
Media insertion and removal events, such as the insertion of a CD into a drive.
The insertion and/or removal of a Plug and Play hard disk (e.g., USB Drive).
Interception of screen saver queries to stop the screen saver from activating.
Operating system level font changes and Windows XP Theme changes.
New network connections/shares added or removed from system.
Switching between applications.
You can use the Visual FoxPro BINDEVENT functions to register (and unregister) event handlers used to intercept messages (i.e., Win32 API window messages that get processed by the Win32 WindowProc function). See MSDN for more details.
The new BINDEVENT( ) syntax requires the hWnd (integer) of the window receiving the message you desire to intercept, and the specific message itself (integer). For example, power-management events such as standby and power-down use the Win32 WM_POWERBROADCAST message (value of 536).
To provide a more integrated development environment for your projects and applications, Visual FoxPro contains the following improved functionality for the IDE.
Additional Project Manager Shortcut Menu Commands
When docked, the Project Manager window contains the following additional shortcut menu commands that are available on the Project menu:
CloseCloses the Project Manager.
Add Project to Source ControlCreates a new source control project based on the current project. Available only when a source code control provider is installed and specified on the Projects tab in the Options dialog box.
ErrorsDisplays the error (.err) file after running a build.
RefreshRefreshes the contents of the Project Manager.
Clean Up ProjectRemoves deleted records from the Project Manager (.PJX) file.
Modifying a Class Library from the Project Manager
When you select a class library (.vcx) file in the Project Manager, you can now open and browse class libraries by clicking the Modify button. The class library opens in the Class Browser. For more information, see How to: Open Class Libraries.
Set Font of Project Manager
You can change the text font settings for the Project Manager window. Right-click the Project Manager window (outside of the tree hierarchy window) and choose Font.
Generating Message Logs During Project Build and Compile
When you build a project, application, or dynamic-link library, Visual FoxPro automatically generates an error (.err) file that includes any error messages, if they exist, when the build process completes. When you select the Display Errors check box in the Build Options dialog box, Visual FoxPro displays the .err file when the build completes. Selecting the Recompile All Files check box includes compile errors in the .err file. Build status messages usually appear in the status bar. However, in previous versions, if the build process is interrupted, Visual FoxPro did not write the .err file to disk.
In the current release, Visual FoxPro writes build status and error messages to the .err file as they occur during the build process. If the build process is interrupted, you can open the .err file opens to review the errors.
Note
If no errors occur during the build, the .err file is deleted.
If the Debug Output window is open, build status and error messages appear in the window. You can save messages from the Debug Output window to a file.
Design time support for entering property values greater than 255 characters and extended characters, such as CHR(13) (carriage return) and CHR(10) (linefeed), has been added to visual class library (.vcx) and form (.scx) files. You can now enter up to 8k characters in length. NoteExtended property value support is only available through the Properties Window (Zoom dialog box) for custom user-specified properties as well as certain native ones such as CursorSchema and Value. For properties not supported, you can still specify values which are longer than 255 characters, or contain carriage returns and linefeeds by assigning them in code such as during the object’s Init Event.The Zoom dialog box and Expression Builder dialog box have been updated to support this. The Properties window includes a Zoom (Z) button that appears next to the property settings box for appropriate properties. WarningProperty values that exceed 255 characters or include carriage return and/or linefeed characters are stored in a new format inside the .vcx or .scx file. If you attempt to modify these classes in a prior version, an error occurs.This feature is particularly useful for setting the CursorAdapter CursorSchema property to any schema expression when schemas might exceed 255 characters.
The Properties window font can now be specified by the new Font shortcut menu option. This new menu replaces the Small, Medium and Large font menu items used in prior versions. This font is also used in the description pane, and object and property value dropdowns. NoteBold and italic font styles are reserved for non-default property values and read-only properties, respectively. If a bold or italic font style is chosen, then the Properties window inverts the displayed behavior. For example, if one chooses an italic font style, read-only properties appear in normal font style and all others in italic.
Colors can be specified for certain types of properties by right clicking on the Properties Window and selecting following menu items:
Non-Default Properties ColorSets color for properties whose values have changed from default setting (same properties that are displayed when the Non-Default Properties Only menu item is selected).
Custom Properties ColorSets color for custom properties.
Instance Properties ColorSets color for custom properties that have been added to the current class instance (same properties that appear in bold in the Edit Property/Method Dialog Box).
NoteIf a conflict exists between color settings, the Instance setting takes priority followed by the Non-Default one.
The MemberData extensibility architecture lets you provide metadata for class members (properties, methods and events). With MemberData, you can specify a custom property editor, display a property on the Favorites tab, or change the capitalization in the Properties Window (Visual FoxPro).
When adding a new property to a class, you can specify an initial value other than the default in the New Property dialog box. Subclasses inherit these default values unless you reset the default values to the parent class. In previous versions, you had to set the default value for the new property by selecting the property in the Properties window and setting the default value.
You can now sort items in the Document View window by name for forms and visual class libraries.
See Document View Window for more information on sorting items in the Document View Window.
Compiling Code in the Background
Visual FoxPro performs background compilation when syntax coloring is turned on in the Command window and Visual FoxPro editors for program (.prg) files, methods, stored procedures, and memos. The Expression box in the Expression Builder dialog box also includes support for background compilation and syntax coloring when turned on.
When the single and current line of code that you are typing contains invalid syntax, Visual FoxPro displays the line of code with the formatting style selected in the Editor tab of the Options dialog box.
Note
Syntax coloring must be turned on for background compilation to function. Background compilation does not detect invalid syntax in multiple lines of code, including those containing continuation characters.
Visual FoxPro now supports copying in RTF (Rich Text Format) to the clipboard. Visual FoxPro preserves the style (bold, italic, and underline) and color attributes.
RTF is supported only in the FoxPro editors that allow for syntax coloring, such as the Command window and editing windows opened with MODIFY COMMAND Command. The RTF clipboard format is only supported when syntax coloring is enabled such as from Edit Properties Dialog Box. You can disable RTF clipboard format with the _VFP EditorOptions Property.
The following improvements were made to Find support:
If a word is selected in a Visual FoxPro editor, the Find Dialog Box (Visual FoxPro) when opened now displays the word in the Look For drop-down box. If Find has not yet been used for a running instance of Visual FoxPro, a word positioned under the insertion pointer will appear in the Look For drop-down. If multiple words are selected, only the first word appears in the drop-down (use copy and paste to enter multiple words).
When a Browse window is open and you search for a word with the Find dialog box, you can search for the word again (Find Again) after the Find dialog box is closed by pressing the F3 key.
You can now use Find to search for content in Name column of the Watch and Locals debug windows (see Debugger Window). When searching object members, Find searches in these debug windows are limited to nodes that have been expanded and one level below.
View Constants in Trace Window
Constants (#DEFINE values) can be viewed in the Trace Window when you hover over it with the mouse.
Note
Visual FoxPro evaluates constants as expressions in the Trace Window and may have difficulty interpreting a specific #DEFINE when you hover over it with the mouse. Consequently, if there are multiple expressions on a line, they are all displayed in the value tip.
Printing Selected Text in Editor Windows
You can print selected text from Visual FoxPro editor windows. When you have text selected in the editor window, the Selection option in the Print dialog box is available and selected.
Note
If a partial line is selected, the entire line is printed.
To improve legibility on high-resolution monitors, Error dialog boxes and the Zoom <property> Dialog Box in the Properties window now use the Windows Message Box text font.
In Windows XP, the Windows Message Box text font is set by opening Display in the Control Panel, and then clicking Advanced on the Appearance tab.
IntelliSense Saves Settings Between User Sessions
Visual FoxPro now saves IntelliSense settings, such as turning IntelliSense on, between user sessions. These settings are controlled by the _VFP EditorOptions property. In addition, the settings in the _VFP EditorOptions property are saved in the FoxUser.dbf resource file. For more information, see EditorOptions Property.
IntelliSense in Memo Field Editor Window
Visual FoxPro includes IntelliSense support in Memo field editor windows when syntax coloring is turned on.
IntelliSense Available for Runtime Applications
Selected IntelliSense features are available at run time in distributed Visual FoxPro 9.0 applications. In order to use IntelliSense at run time, you need to set the _FOXCODE and _CODESENSE variables, and EditorOptions Property.
Note
With runtime applications, syntax coloring does not need to be turned on for an editor to support IntelliSense.
FOR EACH ObjectName [AS Type [OF ClassLibrary]] IN Group
Commands
[EXIT]
[LOOP]
ENDFOR
The Type parameter can be any valid type, including data types, class types, or ProgID. If the class name cannot be found, Visual FoxPro disregards Type and does not display IntelliSense for it.
Note
The type reference does not affect the functionality of the application at run time. The type reference is only used for IntelliSense.
The ObjectName expression can refer to a memory variable or an array.
The ClassLibrary parameter must be in a path list that is visible to Visual FoxPro. You must specify a valid class library; references to existing objects are not valid. If Visual FoxPro cannot find the specified class library, IntelliSense does not display.
Types expressed as ProgIDs and class libraries do not require quotation marks (“”) to enclose them unless their names contain spaces.
When a user types the AS keyword, IntelliSense displays a list of types registered in the FoxCode.dbf table with Type “T”. If you have specified a valid type, typing a period within a WITH … ENDWITH or a FOR EACH … ENDFOR command displays IntelliSense for that object reference.
Visual FoxPro supports IntelliSense for nested WITH … ENDWITH and FOR EACH … ENDFOR commands. The following is an example of nested WITH … ENDWITH commands in a class defined in a program (.prg) file named Program1.prg. To use, paste this code into a new program named Program1.prg, save it and then type a period (.) inside the WITH … ENDWITH block.Copy
DEFINE CLASS f1 AS form
MyVar1 = 123
ADD OBJECT t1 AS mytext
PROCEDURE Init
WITH THIS AS f1 OF program1.prg
WITH .t1 AS mytext OF program1.prg
ENDWITH
ENDWITH
ENDPROC
ENDDEFINE
DEFINE CLASS mytext as textbox
MyVar2 = 123
ENDDEFINE
IntelliSense provides limited List Values functionality for selected properties that begin with a “T” or “F” within a WITH … ENDWITH or FOR EACH … ENDFOR command. This is done to avoid possible conflicts with the common property values True (.T.) and False (.F.). If you just type “.T” or “.F” and press Enter, the word selected in the List Value drop-down does not expand. You need to type at least two letters for IntelliSense to insert the selected word.
You might want to open the following designers and find the enhancements.
Report and Label Designers
You can use the Report Builder available in the Report Designer and Label Designer to perform reporting tasks, configure settings, and set properties for reporting features such as report layout, report bands, data groups, report controls, and report variables. For example, you can perform the following tasks:
Prevent users from modifying reports, report controls, and report bands when editing the report in protected mode.
Display captions instead of expressions for Field controls at design time.
Display user-defined ToolTips for report controls.
Set the language script for reports.
Save the report data environment as a class.
By default, the Report Builder activates when you interact with the Report and Label designers. However, you can use the _REPORTBUILDER system variable to specify ReportBuilder.app. The Report Builder consolidates, replaces, and adds to the functionality found in previous Report Designer user interface elements, which remain in the product and are available by setting _REPORTBUILDER. You can write custom report builders to extend reporting functionality and output or run reports with report objects. For more information, see Working with Reports and _REPORTBUILDER System Variable.
Menu Designer
You can set the _MENUDESIGNER system variable to call your own custom designer for creating menus.Copy
You can use spaces in table names specified in SQL statements in the Query and View designers if you provide an alias. For example, editing the following statement is valid in the View and Query designers:Copy
Visual FoxPro contains the following miscellaneous enhancements. You can click Collapse All to view a list of enhancments.
Printing Dialog Boxes and Printing Language Enhancements
Visual FoxPro includes various enhancements for its printing dialog boxes and printing language.
Visual FoxPro uses the latest operating system dialogs for Printer Setup and other related printing operations. If the user is running on Windows XP, the dialogs will appear Themed.
The following language functions contain new enhancements that impact general printing operations:
GETFONT( ) FunctionContains an additional setting to display only those fonts available on the current default printer and clarified values for the language script.
Improved Support for Applications Detecting Terminal Servers
Visual FoxPro now automatically includes support for applications that are generated by the build process to detect whether they are running on a Terminal Server and prevent loading of unnecessary dynamic-link library (.dll) files that can impact performance. For more information, see BUILD EXE Command.
Updated Dr. Watson Error Reporting to 2.0
Visual FoxPro includes and updates its product error reporting to support Dr. Watson 2.0. This version includes new and improved error reporting, logging, and auditing features. For example, errors are logged while offline and are posted when you reconnect.
Anchor Editor Application
Visual FoxPro 9.0 allows you to create a custom property editor through extended metadata attributes for class members. Through this new extensibility model, you now have the ability to extend the functionality of class properties and methods, allowing you to create design-time enhancements such as a custom property editor. For more information about creating custom property editors, see MemberData Extensibility.
A sample custom property editor, Anchoreditor.app, is included in Visual FoxPro 9.0 and is located in the Wizards directory. This application is run when the Anchor property is double-clicked in the Properties window, or by choosing the Anchor property in the Properties window and clicking the ellipsis button (…).Expand table
Term
Definition
Anchor but do not resize vertically
Specifies that the center of the control is anchored to the top and bottom edges of its container but the control does not resize.
Anchor but do not resize horizontally
Specifies that the center of the control is anchored to the left and right edges of its container but the control does not resize.
Border values
Displays the current settings for the border values.
Common settings
Selects commonly used settings for the Anchor property.
Sample
Click the Sample button to test the current anchor value on a sample form.
Anchor value
The Anchor property value that is the combination of the current settings for the border values.
Class Browser
You can open and view class definitions that are specified within a program (.prg) similarly to class libraries (.vcx). You can select a program (.prg) from the File Open/Add dialog box. See Class Browser Window for more information.
CursorAdapter Builder
The CursorAdapter Builder contains a number of enhancements that correspond to improvements added to the CursorAdapter class. See CursorAdapter Builder for more information.
Toolbox
The Toolbox (Visual FoxPro) is now dockable and can be docked to the desktop or other IDE windows.
For the results grid, the Options dialog provides a new setting to show separate columns for class, method, and line, rather than concatenating them all in a single column.
You can now sort by method name by right-clicking on the method header or selecting the Sort By menu item from the right-click menu.
With the results tree list, the following new right-click menu options are available:
Expand All – expands all nodes
Collapse All – collapses all nodes
Sort by Most Recent First – puts the most recent result sets at the top of the list rather than at the bottom
Note
The results beneath a tree node are not filled until the node is expanded. This is done to increase performance if you have a large result sets.
GENDBC.PRG
The Gendbc.prg program which generates program used to recreate a database has been updated with following minor enhancements:
Support for new Varchar, Varbinary and Blob field types
Support for AllowSimultaneousFetch, RuleExpression, and RuleText properties for views
Environment Manager Task Pane
The Environment Manager Task Pane has been enhanced with the following features:
Form and Formset Template Classes – you can now specify template classes for new forms and formsets with each environment set. This is setting specified in the Forms Tab, Options Dialog Box.
Field Mapping – you can set classes to use for when you drag and drop a field onto a form with each environment set. This is setting specified in the Field Mapping Tab, Options Dialog Box.
Resource File – the Environment Manager now supports setting of a Resource File. If one does not exist, the Environment Manager will optionally create it when the environment is set.
The Environment Manager now contains a new <default field mapping> environment set. This set is created the first time the Environment Manager is run so that the original default Options dialog settings for Field Mapping and Form Template Classes can be saved and restored later if desired.
The Task Pane Manager includes the new Data Explorer Task Pane which allows you to view and work with remote data sources such as SQL Server databases.
The new MemberData Editor lets you edit MemberData for your classes. The MemberData Editor is available from the Class menu when the Class Designer is active. The MemberData Editor is also invoked silently when you right-click on an item in the Properties Window and select the Add to Favorites menu item. The MemberData Editor application is specified as a builder and can be changed in the Builder.dbf table located in your Wizards directory.
The following are new FoxPro Foundation classes added to this version of Visual FoxPro:
_REPORTLISTENER.VCX – a set of core classes you can use when creating custom report listeners.
_FRXCURSOR.VCX – a class library used for working with report (FRX) files.
_GDIPLUS.VCX – a set of classes you can use for GDI+ handling. This is intended primarily for use when creating custom report listener classes.
New Solution Samples
Visual FoxPro 9.0 contains many new samples that show off new features in the product. To see a list of these samples, select the Solution Samples task pane in the Task Pane Manager and expand the New in Visual FoxPro 9.0 node.
Visual FoxPro includes functionality that differs from previous versions and might affect existing code. These behavior changes are organized according to the following categories:
Critical Changes Functionality changes most likely to affect existing code when running under this version of Visual FoxPro. It is extremely important that you read this section.
Important Changes Functionality changes that might affect existing code when running under this version of Visual FoxPro.
Miscellaneous Changes Functionality changes you should know about but are not likely to impact existing code.
Removed Items Features or files that existed in prior versions of Visual FoxPro but are no longer included.
Critical Changes
Critical behavior changes will most likely to affect existing code when running under this version of Visual FoxPro.
SQL SELECT IN (Value_Set) Clause
In previous versions of Visual FoxPro, the IN (Value_Set) clause for the WHERE clause in the SQL SELECT command is mapped to INLIST( ) function. In the current release, Visual FoxPro might stop evaluating values and expressions in the Value_Set list when the first match is found. Therefore, if the IN clause is not Rushmore-optimized, you can improve performance by placing values most likely to match in the beginning of the Value_Set list. For more information, see the description for the IN clause in the SELECT – SQL Command topic and the INLIST( ) Function.
Conversion of INLIST( ) Function in the Query Designer and View Designer
In previous versions of Visual FoxPro, the Query Designer and View Designer convert INLIST( ) function calls in the WHERE clause of the SQL SELECT command into IN (Value_Set) clauses. In the current release, this conversion no longer occurs due to the differences between INLIST( ) and the SQL IN clause. INLIST( ) remains restricted to 24 arguments. For more information, see the description for the IN clause in the SELECT – SQL Command topic and the INLIST( ) Function.
Grids and RecordSource and ControlSource Properties
In Visual FoxPro 9.0 there is a change in Grid control behavior. When the RecordSource property for a Grid control is set, Visual FoxPro 9.0 resets all ControlSource properties to the empty string (“”) for all columns. In earlier versions of Visual FoxPro, the ControlSource properties were not properly reset, so problems could occur when a RecordSource with a different structure was later bound. This change may impact scenarios involving Access and Assign methods or BINDEVENT( ) function calls made against a Grid column’s ControlSource property.
Important Changes
Important changes might affect existing code when running under Visual FoxPro 9.0.
Reporting
Visual FoxPro contains many improvements for reporting. The following are behavior changes that could impact existing reports:
The Report Designer and Engine now make use of extensible components. You can control or eliminate use of design-time extensions by altering the value of _REPORTBUILDER System Variable. You control run-time extension use with the SET REPORTBEHAVIOR Command.
In Visual FoxPro 9’s new object-assisted reporting mode, report fields may need to be adjusted (widened) slightly. This is especially important for numeric data where a field that is not wide enough to display the entire number will show it instead as asterisks (*****). For more information about the changes to the Report System that required this change, and features of the GDI+ rendering engine and other changes related to it, see Using GDI+ in Reports. For migration strategy and recommendations, see Guide to Reporting Improvements.
For a table of additional, minor rendering differences between backward-compatible reporting mode and object-assisted reporting mode, see the table below.Expand tableRendering featureBehavior in backward-compatible modeBehavior in object-assisted modeRecommendationsTab stops (CHR(9) values included in report data)The width of a tab stop is determined by a number of characters in the font used.Tab stops are set at fixed-width positions, regardless of font.If you concatenated tabs with data in a stretching report layout element to create a table format within the element, you can often fulfill the same requirements using a second detail band in Visual FoxPro 9. Alternatively, change the number of tabs you concatenate with your data.Special characters and word-wrappingNon-breaking spaces are not respected; they are treated as normal space characters.Special characters such as non-breaking spaces (CHR(160)) and soft hyphens (CHR(173)) are correctly interpreted. As a result, words may wrap differently in output.Evaluate the results. In most cases, users will appreciate the change, because it more faithfully representing their original intentions in the text. If necessary, use the CHRTRAN( ) Function or STRTRAN( ) Function to replace these special characters with standard spaces and hyphens.Line spacing of multi-line objectsLine spacing is determined by a formula that does not take font properties into consideration. Lines in a multi-line object are individually rendered, so background colors for each line may appear to have a different width.GDI+ line spacing is dynamically determined using font characteristics. A multi-line object is rendered as a single block of text.Evaluate the results. In most cases, the change in line spaces will provide a more polished appearance, and in all cases this method of handling multi-line text provides better performance. If a report depends on the old style of spacing lines, you can adjust the ReportListener’s DynamicLineHeight Property to revert to the old behavior.Cursor images (.CUR files).CUR files can be used as image sources in reports..CUR files are not supported as image sources.Convert the cursor file to another, supported image format.Shape (Rounded Rectangle) curvatureLimited choices for curvature.More choices are available through the Report Builder Application dialog box interface, but some will not look the same way in backward-compatible mode and object-assisted mode.If reports have to run in both backward-compatible mode and object-assisted mode, or if they are designed in version 9.0 but must run in earlier versions, limit your choices of values of shape curvature to those allowed in the native Round Rectangle Dialog Box. If you are using the Style Tab, Report Control Properties Dialog Box (Report Builder) to design such reports, use the values 12, 16, 24, 32, and 99, to represent the native buttons, selecting the buttons from left to right. The default value in the Round Rectangle dialog box (second button) is 16.
When you create a Quick Report, by using the CREATE REPORT – Quick Report Command or by invoking the Quick Report… option on the Report menu, and if you have SET REPORTBEHAVIOR 90, the layout elements created by the Report Designer are sized differently from ones created for the same fields in previous versions. This change handles the additional width required by the new rendering mechanism of the report engine.
If you use the KEYBOARD Command or PLAY MACRO Command statements to address options on the Report menu, you may need to revise the keystrokes in these statements, as the menu has been reorganized.
Reports may take longer to open in the Report Designer if the report was previously saved with the Printer Environment setting enabled. You can improve performance by unchecking the Printer Environment menu item from the Report menu and re-saving the report. The saved Printer Environment is not critical for functioning of a report and is typically not recommended. Object-assisted report mode also respects different printers’ resolution settings, so saving resolution information for one printer in your report may have adverse effects in an environment with printers that have different resolutions. A saved Printer Environment may also have more adverse affects on REPORT FORM or LABEL commands invoked with the TO FILE option than they did in previous versions, if the associated printer setup is not available in the environment at runtime. In Visual FoxPro 9, the global default for this setting in the Reports Tab, Options Dialog Box, and for reports created in executable applications (.exe files), has been changed to unchecked.
Because of changes to the way Visual FoxPro 9 uses current printer settings to determine items such as print resolution and page height dynamically, a REPORT FORM or LABEL command will not run in object-assisted mode if there are no available printer setups in the environment or if the print spooler has been stopped. You will receive Error loading printer driver (Error 1958). If you need to run reports in an environment with no printer information, perhaps creating custom types of output that do not require printers, you can supply Visual FoxPro with the minimal set of information it needs to run your report by supplying a page height and page width from the appropriate Report Listener methods. For more information, see GetPageHeight Method and GetPageWidth Method.
By default, and by design, the Report Builder Application does not automatically show tables in the report’s Data Environment when you build report expressions. To better protect end-user design sessions, only tables you have explicitly opened, not all tables from the DataEnvironment, are available in the Expression Builder. With this change, you have the opportunity to set up the design session’s data exactly the way you want the end-user to see it, before you issue a MODIFY REPORT Command in your application. If you prefer the Report Designer’s old behavior, you can change the Report Builder Application to emulate it. For more information, see How to: Replace the Report Builder’s Expression Builder Dialog Box.
The ASCII keyword on the REPORT FORM Command is documented as following the <filename> parameter of the TO FILE<filename> clause. In earlier versions of Visual FoxPro, you could safely use the incorrect and unsupported syntax TO FILE ASCII<filename> instead. This incorrect syntax triggers an error in Visual FoxPro 9. Note that the ASCII keyword has no effect on object-assisted mode output provided by the Report Engine, although a ReportListener Object can be written to implement it.
The keyword NOCONSOLE has no default meaning in object-assisted reporting mode, because ReportListeners do not echo their rendering output to the current output window by default. However, a ReportListener can mimic backward-compatible mode in this respect, if desired. Refer to OutputPage Method for a complete example.
To facilitate development of run-time reporting extensions, the Report Engine now allows normal debugging procedures during a report run. If your error handling routine assumes it is impossible for a report to be suspended, this assumption will now be challenged. Refer to Handling Errors During Report Runs for a detailed look at the associated changes, and some suggestions for strategy.
REPORT FORM and LABEL commands are no longer automatically prohibited as user-interface-related commands in COM objects compiled into DLLs, when you run the commands in object-assisted mode. The restriction still applies to these commands when they are run in backward-compatible mode. (The topic Selecting Process Types explains why user-interface-related commands are prohibited in DLLs.) This change is not applicable to multi-threaded DLLs. A number of user-interface-related facilities also are not available in DLLs (whether single- or multi-threaded). For example, the TXTWIDTH( ) Function and TextWidth Method depend on a window handle to function, so they are not available in a DLL. The CREATE REPORT – Quick Report Command relies on the same facilities as TXTWIDTH(), and therefore is not available in a DLL. However, in many instances, creating custom output using a ReportListener does not require any user-interface activity, so a REPORT FORM or LABEL command can now be used productively in a DLL. Using the SYS(2335) – Unattended Server Mode function to trap for potential modal states, as well as the new SET TABLEPROMPT Command, is recommended. Refer to Server Design Considerations and Limitations for more information.
Changes have occurred to the handling of group headers and footers in multi-column reports, when the columns flow from left to right (label-style layout). The revised behavior is more useful and behaves consistently with the new detail header and footer bands as well. For a description of the change, see How to: Define Columns in Reports.
In previous versions, the NOWAIT keyword on the REPORT FORM and LABEL commands was not significant when the commands were issued in the Command window rather than in a program. In Visual FoxPro 9’s object-assisted mode, when previewing instructions are interpreted by the Report Preview Application, this keyword is significant no matter where you issue the command. The Report Preview Application uses the NOWAIT keyword, consistently, as an instruction to provide a modeless preview form. For more information about the Report Preview Application, see Extending Report Preview Functionality.
Visual FoxPro 8 introduced the NOPAGEEJECT keyword on the REPORT FORM and LABEL commands, but applied the keyword only to printed output. In Visual FoxPro 9, NOPAGEEJECT has significance for all output targets, including PREVIEW. This keyword provides chained or continued report runs for multiple REPORT FORM and LABEL commands. To facilitate this behavior in preview mode, and to allow you to apply customization instructions to multiple previews, the Report Output Application caches a single ReportListener object instance for preview output, causing a change in behavior for multiple modeless report commands (REPORT FORM … PREVIEW NOWAIT). In the past, you used multiple REPORT FORM… PREVIEW NOWAIT statements in a sequence, your commands resulted in multiple report preview windows. In Visual FoxPro 9, when SET REPORTBEHAVIOR 90, these commands will result in successive report previews being directed to a single report preview window. TipYou can easily invoke the old behavior by creating multiple ReportListener object references and associating one with each separate REPORT FORM or LABEL command, using the OBJECT keyword. For more information about using the OBJECT syntax, see REPORT FORM Command. For information about receiving multiple object references of the appropriate type from the Report Output Application, see Understanding the Report Output Application.
In the process of reviewing and overhauling the native Report Engine, a number of outstanding issues regarding band and layout element positioning were addressed. For example, a field element marked to stretch and sized to take up more than one text line’s height in the report layout might have inappropriately pushed its band’s exit events to the next page in Visual FoxPro 8. In Visual FoxPro 9, the band’s exit events occur on the same page. Additional revisions improve record-pointer-handling in footer bands, when bands stretch across pages. These changes are not specific to object-assisted output rendering. If you have relied on undocumented behavior providing exact band or layout control placement in a particular report, you should review that report’s behavior in Visual FoxPro 9.
Rushmore Optimization
When character values are indexed, all values are considered to be encoded using the table’s code page. In previous versions of Visual FoxPro, when the current Visual FoxPro code page differed from a table’s code page, any attempt to look for a character value within that table’s index resulted in an implicit translation of the value from the current Visual FoxPro code page into the table’s code page. This could cause SQL or other Rushmore optimizable commands to return or act upon incorrect records.
In Visual FoxPro 9 and later, by default, the optimization engine no longer uses existing character indexes for tables created with a non-current code page. Instead, Visual FoxPro builds temporary indexes to ensure correct results. This can result in a loss of optimization of SQL or other commands which were optimized in earlier VFP versions. To prevent this, ensure that the current Visual FoxPro code page returned by CPCURRENT( ) Function matches the table’s code page returned by CPDBF( ) Function. This requires either changing the current Visual FoxPro code page, or changing the table’s code page. For information on specifying the current Visual FoxPro code page, see Understanding Code Pages in Visual FoxPro. For information on specifying the code page for a table, see How to: Specify the Code Page of a .dbf File. If you cannot change either the Visual FoxPro codepage or the table codepage to match, you can force optimization to work as it did in Visual FoxPro 8 and prior versions using the SET ENGINEBEHAVIOR Command with either 80 or 70 as a parameter.
SQL SELECT Statements
A SELECT – SQL Command containing DISTINCT and ORDER BY clauses in which the ORDER BY field is not in the SELECT field list will generate an error in Visual FoxPro 9.0 with SET ENGINEBEHAVIOR 90 (Error 1808: SQL: ORDER BY clause is invalid.). The following example shows this:CopySET ENGINEBEHAVIOR 90 CREATE CURSOR foo (f1 int, f2 int) SELECT DISTINCT f1 FROM foo ORDER BY f2 INTO CURSOR res
A SELECT – SQL Command containing DISTINCT and HAVING clauses in which the HAVING field is not in the SELECT field list will now generate an error in Visual FoxPro 9.0 with SET ENGINEBEHAVIOR 90 (Error 1803: SQL: HAVING clause is invalid.). An error is reported because the HAVING field is not in projection and DISTINCT is used. The following example shows this:CopySET ENGINEBEHAVIOR 90 CREATE CURSOR foo (f1 int, f2 int) SELECT DISTINCT f1 FROM foo HAVING f2>1 INTO CURSOR res
The number of UNION statements that can be used in a SELECT – SQL Command is no longer limited to 9. Parentheses are not completely supported with UNION statements and unlike previous versions may generate an error. If two or more SELECT statements are enclosed in parenthesis, an error is generated during compile (Error 2196: Only a single SQL SELECT statement can be enclosed in parentheses.). This behavior is not tied to any SET ENGINEBEHAVIOR Command level. The following example shows this error:CopySELECT * FROM Table1 ; UNION ; (SELECT * FROM Table2 ; UNION ; SELECT * FROM Table3) The following example compiles without an error:CopySELECT * FROM Table1 ; UNION ; (SELECT * FROM Table2) ; UNION ; (SELECT * FROM Table3)
An error (Error 2199) is now generated when index key truncation is about to occur, typically during index creation or modification. This can happen with use of a key that contains an expression involving a Memo field, whose length in not fixed, such as in the following example:
INDEX ON charfld1 + memofld1 TAG mytag
Similar issues can also occur with the SQL engine (such as during a SQL SELECT command or View creation) where it might fail to build a temporary index to optimize a join evaluation if it is unable to accurately determine the maximum size of the key.
Visual FoxPro will now detect if a Memo field in a class library (.vcx) or form (.scx) is corrupt when you try to open up that file in the designer. If the file contains a corrupt Memo field, an Error 41 such as following will occur:
Memo file <path>myclass.VCT is missing or is invalid.
Additionally, similar Memo errors may occur if you have a Visual FoxPro table open and try to access contents of a corrupt Memo. The following sample code shows how you can detect the Error 41 memo file corruption:Copy
TRY
USE myTable EXCLUSIVE NOUPDATE
SCAN
SCATTER MEMO MEMVAR
ENDSCAN
CATCH TO loError
IF loError.ErrorNo=41
* handle error here
ENDIF
ENDTRY
USE IN myTable
While it is possible that loss of data may occur, the following sample code may assist in repairing some or the entire file:Copy
ON ERROR *
USE myclass.vcx
COPY TO myclass_bkup.vcx&&backup
COPY TO myclass2.vcx
USE
DELETE FILE myclass.vc*
RENAME myclass2.vcx TO myclass.vcx
RENAME myclass2.vct TO myclass.vct
COMPILE CLASSLIB myclass.vcx
ON ERROR
Visual Form and Class Extended Property Support
Visual FoxPro 9 allows you to create custom properties in your visual class (SCX or VCX file) whose values can contain carriage returns and/or be of length greater than 255 characters. If you specify a property with a value like this through the Properties Window (i.e., the Zoom dialog box), Visual FoxPro will store it in a format such that you will no longer be able to edit that class in older versions of Visual FoxPro.
Class Definitions
The ability to have a property assignment set to instantiated object is no longer supported in a class definition and will generate an error. The following example shows this.Copy
LOCAL oCustom
oCustom = CREATEOBJECT('cusTest')
DEFINE CLASS cusTest AS CUSTOM
oRef = CREATEOBJECT('myclass')
ENDDEFINE
DEFINE CLASS myclass AS CUSTOM
ENDDEFINE
You can instead assign a property to an instantiated object reference in the Init event of your class.
Merge Modules for Redistributable Components
Visual FoxPro includes merge modules (MSM files) for use in redistributing shared components with your runtime applications. Merge modules are used by applications that can create Windows Installer based setups. For example, Visual FoxPro ships with merge modules that contain the Visual FoxPro runtime libraries as well as some common components including a number of ActiveX controls.
For Visual FoxPro 9, the VFP9RUNTIME.MSM merge module contains the runtime libraries that you will need for your custom redistributable application. The VFP9RUNTIME.MSM merge module also has dependencies on the merge modules containing the Microsoft VC 7.1 runtime library (MSVCR71.DLL) and the GDI+ graphics library (GDIPLUS.DLL). Because of these dependencies, if you select the VFP9RUNTIME.MSM merge module in a Windows Installer tool such as InstallShield, the other dependent merge modules will automatically be selected as well.
Note For Windows XP and higher operating systems, Visual FoxPro uses the GDI+ graphics library that is installed in your Windows System folder.
For Visual FoxPro 9, the merge module containing the VC runtime library no longer installs to the Windows System directory. Instead, this file is installed to your application’s directory. This is done in compliance with recommended component versioning strategies for Windows operating systems. The GDI+ library is installed into the same directory as your Visual FoxPro runtime libraries and is only installed on operating systems later than Windows XP (XP already includes the GDI+ library in its Windows System directory).
Tip There may be circumstances where you will want to install the VC or GDI+ library to another location such as the Windows System directory. You can do this with your Windows Installer application (e.g., InstallShield) by first selecting the merge module before selecting the VFP9RUNTIME.MSM one. Once you have selected a merge module, you can change its installation path.
There are new merge modules for MSXML3 and MSXML4 XML parser components. The MSXML 3.0 component consists of the following merge modules:
There are two MSXML 4.0 modules that should be included with any custom setup:
MSXML 4.0 (msxml4sxs32.msm)
MSXML 4.0 (msxml4sys32.msm)
MTDLL Memory Allocation
Visual FoxPro contains a new PROGCACHE configuration file setting which specifies the amount of memory Visual FoxPro allocates at startup for running programs (program cache). This setting also determines memory allocated per thread for Visual FoxPro MTDLL COM Servers. In prior versions, this setting was not configurable and memory was allocated as a fixed program cache of a little over 9MB (144 * 64K). The new PROGCACHE setting allows you to set the exact size of the program cache or specify that dynamic memory allocation be used.
Since MTDLL COM Servers can use up a great amount of memory if many threads are created, it is important that memory be allocated more efficiently for these scenarios. In Visual FoxPro 9, the new default setting for MTDLL COM Servers is -2 (dynamic memory allocation). For more information, see Special Terms for Configuration Files.
Miscellaneous Changes
The following are miscellaneous changes that you should know about but are not likely to impact existing code.
CursorAdapter Changes
In the current version of Visual FoxPro, the following behavior changes apply to the CursorAdapter object:
The target record is now kept current in the ADODB.Recordset during CursorAdapter.After… events.
Grid SetFocus Supported for AllowCellSelection
You can now call a Grid control’s SetFocus Method and have the Grid receive focus when the AllowCellSelection Property is set to False (.F.) and the grid contains no records.
Additionally, Visual FoxPro 9.0 tightens syntax validation of calls made from concatenation of parameters. The following code, which worked in prior versions of Visual FoxPro, now properly causes an error because the CHR(13) character breaks the call into two lines whereas it is supposed to be part of the parameter for the EXECSCRIPT call.Copy
In the current version of Visual FoxPro, the PageUp, PageDown, Home and End keyboard keys now cause a Listbox control’s Click event to fire. In previous versions, these keys did not trigger the Click event to fire, unlike the arrow keys.
PEMSTATUS( ) Function Returns False for Hidden Native Properties
In previous versions of Visual FoxPro, the PEMSTATUS( ) function returned True (.T.) for hidden native properties of Visual FoxPro base classes when specifying a value of 5 for nAttribute. In the current release, PEMSTATUS( ) returns False (.F.) for these hidden native properties. For more information, see PEMSTATUS( ) Function.
Changes to Options Dialog Box
In the Options dialog box, the List display count option has been moved from the Editor tab to the View tab. For more information, see View Tab, Options Dialog Box.
In previous versions of Visual FoxPro, you could output all the settings in the Options Dialog Box (Visual FoxPro) to the Command Window by pressing the SHIFT key when choosing the OK button to close the dialog. In the current release, these settings are now sent to the Debug Output Window. The Debug Output window must be opened in order for the settings to be directed there.
FOXRUN.PIF
The FOXRUN.PIF file that is used by the RUN | ! Command is no longer installed in the Visual FoxPro root directory. If Visual FoxPro detects the presence of a FOXRUN.PIF file during a RUN command, it will use COMMAND.COM to execute the specified RUN command. This may not be the desired SHELL program to use for a particular operating system, especially newer ones like Windows XP in which CMD.EXE is preferable.
The current behavior for a RUN command without the existence of a FOXRUN.PIF file is that the RUN command will use the SHELL program specified by the operating system COMSPEC environment variable. With Windows XP, you can view and edit this variable by right-clicking your computer desktop’s My Computer icon and selecting the Properties dialog box (Advanced tab).
The FOXRUN.PIF file is still available in the Tools directory if you need it for a particular reason.
The SCATTER command no longer allows for ambiguous use of both MEMVAR and NAME clauses in the same command. You can only include one of these clauses. In prior versions, the following code would not generate an error:Copy
USE HOME()+"SAMPLESDatacustomer.dbf"
SCATTER MEMVAR NAME oCust
The SET DOHISTORY command, which is included for backward compatibility, was updated to send output to the Debug Output Window instead of the Command Window as in prior versions.
SCREEN ShowTips Property
The default value for _SCREEN ShowTips Property has been changed from False (.F.) to True (.T.). This change was made because new Memo and Field Tips support is now dependent on this setting.
AllowCellSelection Does Not Permit Deleting Grid Rows When Set to False
The sample Northwind database has been updated. Five of the stored procedures now include calls to the SETRESULTSET( ) Function so that the Visual FoxPro OLE DB Provider will return a cursor when they are executed.
Foundation Classes
The _ShellExecute class contained in the _Environ.vcx FFC class library has been updated to include an additional parameter in the ShellExecute method.
Wizards and Builders
The Wizard/Builder selection dialog box now properly hides deleted entries in the Wizard and Builder registration tables.
Specifying Western Language Script Values for GETFONT( ) Function
In versions prior to this release, passing 0 as the nFontCharSet value for GETFONT( ) opened the Font Picker dialog box and displayed the Script list as unavailable. You could not specify 0 (Western) as the language script value, and setting it to 1 (Default) sets nFontCharSet to the default font setting only, which is determined by the operating system.
In this release, passing 0 to GETFONT( ) opens the Font Picker dialog box with the Script list available and Western selected. The return value for GETFONT( ) also includes the return value for nFontCharSet.
Removed Items
HTML Help SDK
The HTML Help 1.3 SDK no longer ships with Visual FoxPro.
Microsoft Visual FoxPro is the object-oriented relational database management system that makes it possible for you to create state-of-the-art enterprise database solutions. Visual FoxPro includes professional productivity tools, documentation, and sample code for quickly building, managing, and deploying solutions.
Customizing the Visual FoxPro Environment Explains how you can optimize your computer system, configure Visual FoxPro and development environment settings, restore your desktop, and how people with disabilities can improve accessibility to Visual FoxPro and Microsoft Windows.
Related Sections
What’s New in Visual FoxPro Describes the new features and enhancements included in this version of Visual FoxPro.
Using Visual FoxPro Gives an overview of Visual FoxPro features, describes concepts and productivity tools for developing, programming, and managing high-performance database applications and components, and provides walkthroughs that help get you started. With the robust tools and data-centric object-oriented language that Visual FoxPro offers, you can build modern, scalable, multi-tier applications that integrate client/server computing and the Internet.
Developing Visual FoxPro Applications Includes conceptual information about how to develop Visual FoxPro applications, instructions for creating databases and the user interface, and other tasks needed to create Visual FoxPro applications.
Programming in Visual FoxPro Describes how understanding object-oriented programming techniques and the event-driven model can maximize your programming productivity and enable you to access the full power of Visual FoxPro.
Reference (Visual FoxPro) Includes Visual FoxPro general, programming language, user interface, and error message reference topics.
Samples and Walkthroughs Contains Visual FoxPro samples and walkthroughs that you can use for experimenting with and learning Visual FoxPro features.
Locating Readme Files (Visual FoxPro)
Article
07/09/2007
The Readme.htm file is stored at the root of the Microsoft Visual FoxPro CD-ROM. Use your Internet browser to open and view the files.
To locate Readme files for additional products included in the Visual FoxPro package, see the root of each product CD-ROM.
Customizing the Visual FoxPro Environment Explains how you can optimize your computer system, configure Visual FoxPro and development environment settings, restore your desktop, and how people with disabilities can improve accessibility to Visual FoxPro and Microsoft Windows.
Upgrading from Earlier Versions Describes how Visual FoxPro protects your investment in applications built in previous versions of FoxPro.
Getting Started with Visual FoxPro Provides information about where to find the Readme file, installing and upgrading from previous versions, configuring Visual FoxPro, and customizing the development environment.
Requirements for Installing Visual FoxPro
Article
07/09/2007
Visual FoxPro has the following minimum system requirements:
Computer: PC with a Pentium class processor.
Peripherals: Mouse or pointing device
Memory: 64 MB RAM (128 MB or higher recommended)
Hard disk space:
Visual FoxPro Prerequisites: 20 MB
Visual FoxPro Typical Install: 165 MB
Visual FoxPro Maximum Install: 165 MB
Video: 800 x 600 resolution, 256 colors (High color 16-bit recommended)
Operating system: Developing applications with Visual FoxPro is supported only on Microsoft Windows 2000 Service Pack 3 or later, Windows XP and Windows Server 2003. You can create and distribute run-time applications for Windows 98, Windows Me, Windows 2000 Service Pack 3 or later, Windows XP and Windows Server 2003. NoteInstallation on Windows NT 4.0 Terminal Server Edition is not supported.
You can install this version of Visual FoxPro from a CD-ROM or a network to a local hard drive. You must install Visual FoxPro on a local drive, not a mapped drive. There is no other preparation required before installing Visual FoxPro. You must have administrator privileges to install Visual FoxPro. It is recommended that you run with power-user privileges to use all the provided tools effectively.
You can safely install or uninstall using Visual FoxPro Setup. If you are upgrading Visual FoxPro, you must first uninstall the previous version of of the program. Though both versions of Visual FoxPro can exist on the same computer, you cannot install the current version of Visual FoxPro in the same directory as the previous version.
If you plan to publish XML Web services using Visual FoxPro, you might want to set up Internet Information Services (IIS) on a Windows 2000, Windows XP or Windows Server 2003 computer. Refer to your operating system documentation for instructions on how to set up and configure IIS.
Note
Visual FoxPro setup no longer installs any Windows operating system Service Packs or versions of Internet Explorer. It is highly recommended that you install the latest versions of these components before installing Visual FoxPro. Additionally, Visual FoxPro 9.0 is supported only on Windows 2000 Service Pack 3 or later. For details about installing the latest Service Pack, visit the following Microsoft Web page at https://www.microsoft.com/windows2000/.
Full installation includes all Visual FoxPro program files, online help, and samples files.
To install Visual FoxPro
Quit all open applications. NoteIf you use a virus protection program on your computer, override it or turn it off before running the Installation wizard. The Installation wizard might not run properly with virus protection turned on. After installation, be sure to restart your virus protection program.
Click Install Visual FoxPro to launch Visual FoxPro Setup.
To determine if you need additional components, click Prerequisites to display any necessary components.
Click Install Now! to install any new components. If Visual FoxPro Prerequisites needs to only update components, click Update Now!
You might need to restart your computer. When finished, click Done.Visual FoxPro Setup reappears.
To continue installation, click Visual FoxPro.
After accepting the End User License Agreement and entering the Product Key and your name, click Continue. NoteVisual FoxPro cannot be installed on a mapped drive. You must install Visual FoxPro on a local drive. Do not attempt to use the Map Network Drive functionality in Setup.
On the Options page, select the features you want to install and click Install Now! to continue.
When finished, click Done to return to Visual FoxPro Setup. Click Exit to return to the Visual FoxPro Setup start page.
If you uninstall Visual FoxPro while the previous version of Visual FoxPro exists on your computer, certain shared registry keys used by the previous version of Visual FoxPro are removed. You must reinstall these critical shared registry keys.
If you run Visual FoxPro from the Start menu, Visual FoxPro Setup automatically reinstalls these keys. If you start Visual FoxPro using other means, such as running the application executable directly, the setup program does not start automatically. You should use Add/Remove Programs in the Control Panel and the following steps to reinstall the registry keys manually:
To manually reinstall Visual FoxPro 9.0 registry keys
From the Start menu, click Control Panel.
Click Add/Remove Programs.
Click Change/Remove for Microsoft Visual FoxPro 9.0.
This release includes copies of additional software that you can install and use with Visual FoxPro. These include:
InstallShield Express Limited EditionProvides the capability to package and deploy the applications that you create using Visual FoxPro. Visual FoxPro includes the InstallShield Express 5.0 Visual FoxPro Limited Edition. NoteThe limited and full editions of InstallShield Express 5.0 are considered two versions of the same product and cannot coexist. If you install one version on a computer where another already exists, the original is uninstalled automatically. Because the limited edition contains fewer features than the full edition, you should keep the full edition on your computer.
Microsoft SOAP Toolkit 3.0 SamplesProvides samples for demonstrating how to consume and publish XML Web services. Visual FoxPro Prerequisites installs the core SOAP Toolkit 3.0 components needed to access and publish XML Web services in Visual FoxPro.
Microsoft SQL Server 2000 Desktop Engine (MSDE)Provides a personal version of SQL Server.
Follow the instructions in the InstallShield Express installation wizard.
You can also locate the Setup.exe file for InstallShield Express in the InstallShield folder on the Visual FoxPro CD.
Note
Visual FoxPro 9.0 installs its redistributable merge modules in the same location as Visual FoxPro 8.0.
The version of InstallShield Express included with Visual FoxPro 9.0 automatically uses the Visual FoxPro 9.0 merge module location.
Note
Visual FoxPro 9.0 requires certain merge modules when creating a Visual FoxPro 9.0 redistributable custom application setup program using InstallShield Express.
You need to include the following merge modules when creating your custom setup program:
Microsoft Visual FoxPro 9 Runtime Language Libraries (specific language library files that may be needed for international applications)
Reporting Applications (needed for Visual FoxPro 9.0 reporting engine)
Note
MSXML 4.0 consists of two merge modules (msxml4sxs32.msm and msxml4sys32.msm). MSXML 3.0 consists of three merge modules (msxml3_wim32.msm, msxml3inf_wim32.msm and wdstddll_wim32.msm).
Click Install Microsoft SQL Server Desktop Engine (MSDE) and follow the installation instructions that appear in the Readme file.
You can locate the Setup.exe file for MSDE in the SQLMSDE folder on the Visual FoxPro CD.
Note
Visual FoxPro includes Microsoft SQL Server 2000 Desktop Engine Service Pack 3.0a. To make sure you have the most recent version and Service Pack installed, visit the Microsoft SQL Server Web page at https://www.microsoft.com/sql. In addition, if you are distributing custom Visual FoxPro applications that require MSDE, you can obtain the redistributable merge modules from the Microsoft SQL Server Web page for use with Windows Installer-based setup programs.
You can reinstall Visual FoxPro by uninstalling it and then installing it again. You can uninstall Visual FoxPro from the Start menu or from the original installation disk.
To uninstall Visual FoxPro
On the Start menu, click Control Panel.
In the Control Panel window, double-click Add or Remove Programs.The Add or Remove Programs window opens.
In the Currently installed programs list, click the version of Microsoft Visual FoxPro you want to uninstall, and then Change/Remove.
If you reinstall Visual FoxPro or reinstall to another location, you might want to clean your user settings and other files installed by Visual FoxPro before reinstalling.
You can remove these files by deleting the contents of the …Application DataMicrosoftVisual FoxPro folder inside your user settings folder. To determine the location of the Application Data folder, type ? HOME(7) in the Command window. These files include your FoxUser.* resource files, which contain user settings, and folders for the Toolbox and Task Pane.
However, it is possible that your resource files are in another location. You can determine their location by typing the following in the Command window:Copy
? SYS(2005)
You should delete old Code Reference files that might be associated with projects in the project directories. These are labeled as projectname_ref.* files. You might also need to restore the default Visual FoxPro registry settings.
Visual FoxPro includes the VFPClean.app tool so you can make sure all core Xbase and other files are set appropriately.
To run VFPClean.app
Type the following line of code in the Command window:CopyDO HOME()+"VFPCLEAN.APP"
You might encounter the following issues when installing Visual FoxPro:
If you cannot run Visual FoxPro and do not see error messages telling you what is wrong, the problem might be in your computer’s ROM BIOS or the video driver you are using.
If you are using an extended keyboard, be sure it is compatible with the ROM BIOS. In addition, make sure that you are using a standard VGA or Super VGA Windows video driver.
If you get a “stack overflow” error message, your video driver is out of date or not designed for your video card. To correct this problem, update the video driver.
For additional information, see the Visual FoxPro Readme at the root of the Visual FoxPro installation CD.
Microsoft Visual FoxPro protects your investment in applications built with previous versions of FoxPro. In Visual FoxPro, you can run many applications that were written in earlier versions with little or no conversion. You can modify and enhance applications using the Visual FoxPro language, knowing that most extensions to the language do not affect backward compatibility. In addition, you can convert FoxPro screens, projects, and reports to Visual FoxPro format.
However, it is possible that some behavior or feature changes in the current version of Visual FoxPro might affect existing Visual FoxPro source code. Therefore, you should review the new features, enhancements, and most recent behavior changes for this version. For more information, see What’s New in Visual FoxPro and Changes in Functionality for the Current Release.
Conversion to Visual FoxPro Format
If you choose to convert your dBASE or FoxPro files to the Visual FoxPro format, you can take advantage of the unique features of Visual FoxPro. You can run many files from some previous versions of FoxPro directly; others require varying levels of conversion.
You can convert most projects or components created using previous versions of Visual FoxPro simply by opening or recompiling them in this version of Visual FoxPro. When you recompile components, such as forms, screens, or reports, some modifications may be necessary. You can make modifications to these components in the same way you modify the components of this version of Visual FoxPro.
You can find additional information about upgrading from previous versions of Visual FoxPro on the Microsoft Developer Network (MSDN) Web site at https://msdn.microsoft.com. You can search the MSDN Archive for documentation of previous versions of Visual FoxPro.
You can explicitly convert FoxPro 2.6 and Visual FoxPro 3.0 files to the current Visual FoxPro format, which is necessary when you want to use these files with later versions of Visual FoxPro. Files that are created from later versions are converted automatically.
To convert FoxPro 2.6 and Visual FoxPro 3.0 files
On the File menu, click Open.
In the Open dialog box, browse for and select the file.The Visual FoxPro Converter dialog box opens. For more information, see Visual FoxPro Converter Dialog Box.
In the Visual FoxPro Converter dialog box, select the options you want.
To complete the file conversion, click Continue. NoteIf you are converting Macintosh or MS-DOS files that have never contained Windows records, the Visual FoxPro Transporter dialog box appears. For more information, see Visual FoxPro Transporter Dialog Box.
You can also convert FoxPro 2.6 and Visual FoxPro 3.0 files by typing one of the following commands with the file name in the Command window:
After you install Visual FoxPro, you might want to customize your development environment. You can also specify configuration settings that load when you start Visual FoxPro.
Optimizing Your System Explains how to get maximum performance by optimizing your operating system, Visual FoxPro, and your application.
Visual FoxPro Configuration Explains how changing the configuration of your copy of Visual FoxPro affects the way it looks and behaves, such as establishing default locations for files used with Visual FoxPro, altering how your source code looks in an edit window, and displaying the format of dates and times.
Visual FoxPro Environment Settings Describes different ways to change Visual FoxPro environment settings such as using the Options dialog box, setting configuration options at program startup, and using command-line options. You can configure Visual FoxPro toolbars, dock windows, set editor options, and customize the appearance of your applications without altering code.
Using Visual FoxPro Explains how Visual FoxPro provides the tools you need to create and to manage high-performance database applications and components.
Samples and Walkthroughs Describes how to create different types of applications and components with step-by-step guides.
Overview of Visual FoxPro Features Describes how Visual FoxPro gives you more of everything you have come to expect in a database management system (DBMS) — speed, power, and flexibility.
Developing Visual FoxPro Applications Includes conceptual information about how to develop Visual FoxPro applications, instructions for creating databases and the user interface, and other tasks needed to create Visual FoxPro applications.
Programming in Visual FoxPro Describes how understanding object-oriented programming techniques and the event-driven model can maximize your programming productivity and enable you to access the full power of Visual FoxPro.
Development Productivity Tools Explains that Visual FoxPro provides developer tools for application development within the FoxPro application and the FoxPro language.
Optimizing Your System
Article
07/09/2007
Visual FoxPro is designed to be a fast relational database development system. However, applications you create with Visual FoxPro can have varying requirements and purposes. Therefore, you might want to optimize the operating system, Visual FoxPro, or your application for maximum performance.
Getting Started with Visual FoxPro Discusses how to get started, including information about installing, upgrading, and customizing Visual FoxPro to create state-of-the-art enterprise database solutions.
What’s New in Visual FoxPro Lists the new features and enhancements made to this version of Microsoft Visual FoxPro.
Using Visual FoxPro Provides links to information on Visual FoxPro programming features that are designed to improve developer productivity, including Access and Assign methods, support for more graphic file formats, and language to simplify programming tasks.
Developing Visual FoxPro Applications Includes conceptual information about how to develop Visual FoxPro applications, instructions for creating databases and the user interface, and other tasks needed to create Visual FoxPro applications.
Programming in Visual FoxPro Discusses how to access the full power of Visual FoxPro by creating applications. Understanding object-oriented programming techniques and the event-driven model can maximize your programming productivity.
You can optimize Visual FoxPro performance by maximizing your computer’s hardware and operating environment. The following sections describe how you can optimize these areas:
Maximizing Memory and Virtual Memory
Managing Your Hard Disk
Maximizing Memory and Virtual Memory
Providing your computer with as much memory as possible is the most effective way to optimize your system for Visual FoxPro. You can also use memory more effectively by closing all other running applications on your computer. To maximize the use of your computer’s memory while running Visual FoxPro, follow these guidelines:
Do not run other Windows applications while running Visual FoxPro.
Use only those memory-resident programs needed for operation.
Simplify the screen display.
You can free memory by simplifying the way windows and screen backgrounds display on your computer monitor.
Use a color or a pattern for the desktop background instead of wallpaper.
Use the lowest-resolution display that is practical for you. The higher resolution of the display, the more memory your computer requires and the slower your graphics and user-interface elements appear. For VGA-compatible displays that use an extended mode driver, such as Video 7 or 8514, using the standard VGA driver ensures faster display performance but provides lower resolution and less color support.
To increase the number of applications that you can run simultaneously, Microsoft Windows supports virtual memory by swapping segments of code that is the least recently used from memory to the hard disk in the form of a paging file. As a rule, the default settings in the Windows operating system for managing virtual memory meet the requirements of most users and are the recommended settings.
Note
The paging file does not improve Visual FoxPro performance and is not a substitute for more memory.
Managing Your Hard Disk
Managing your hard disk can improve overall product speed. To get the best performance from your hard disk, provide a generous amount of disk space. If your hard disk has little free space, you can increase Visual FoxPro performance by removing unnecessary data or by purchasing a hard disk with greater capacity.
Disk input/output performance degrades significantly when a hard disk is nearly full. The more free hard disk space that is available, the more likely it is that contiguous blocks of disk space are available. Visual FoxPro uses this space for changes and additions to database, table, index, memo, and temporary files. Increasing free hard disk space improves performance of any commands that change or add to your files. More disk space also decreases the time required to read those files in response to your queries.
The way that Windows and Visual FoxPro manage files on disk can greatly affect the performance of your application. The following sections discuss managing files in directories and temporary files:
Managing Files in Directories
Managing Temporary Files
Managing Files in Directories
As a directory becomes increasingly congested with files, the operating system takes longer to find files. The speed of your system when searching directories is a factor that Visual FoxPro does not control. To improve the speed of directory searches, reduce the number of files in your directories by performing the following actions:
Use the Visual FoxPro Project Manager to create and manage your files, segregate program files into separate directories, and avoid creating numerous generated files.
When you want to distribute your application, create an application or an executable (.exe) file instead of numerous individually generated files.This process decreases the number of files in your application’s subdirectories and increases performance.
If you delete a large number of files in one directory, copy the remaining files into a new directory or optimize the directory using a defragmenting utility program. NoteDeleting files from a directory does not automatically speed directory searching. When a file is deleted, the file is only marked for deletion and is still included in directory searches.
When saving files, use short file paths to increase performance.For example, suppose you have a file path “C:Program FilesMicrosoft Visual FoxPro…”, which is a very long file path. Try to use shorter file paths.
Managing Temporary Files
Visual FoxPro creates temporary files for a variety of operations. For example, Visual FoxPro creates temporary files during editing, indexing, and sorting. Text editing sessions can also create a temporary or backup (.bak) copy of the edited file. By default, Visual FoxPro creates its temporary files in the same directory that Windows stores its temporary files unless you specifically designate an alternate location.
Tip
In most cases, you should specify one location for all Visual FoxPro temporary files. Make sure that the location you specify contains enough space for all possible temporary files.
When Visual FoxPro searches for temporary files, for example, when you use the SYS(2023) – Temporary Path function to retrieve the temporary files path or when the TMPFILES, EDITWORK, PROGWORK, and SORTWORK settings in a Visual FoxPro configuration file do not specify a different location, the Windows API GetTempPath is used to search for the path containing the temporary files. GetTempPath searches a sequence of variables that differ depending on the operating system. Microsoft Windows 2000 and later include user variables that store the location of temporary files, while Microsoft Windows 95, 98, and Me include only global system environment variables for this purpose.
On Windows 2000 and later, GetTempPath, and therefore, SYS(2023), TMPFILES, EDITWORK, PROGWORK, and SORTWORK, searches the TMP user variable for the location of temporary files by default. If the TMP user variable does not specify a location, Visual FoxPro searches the following variables in a specific order:
TMP system variable.
TEMP user variable.
TEMP system variable.
If these variables do not specify a location, the location for storing temporary files defaults to the home drive and path, or the Temp folder in the user’s Documents and Settings directory.
Note
If more than one value is specified for TMP or TEMP, then the first value is used.
On Windows 95, 98, and Me, GetTempPath searches the TMP and TEMP global system variables in that order and then searches the current directory.
Though Visual FoxPro is always fast, you can optimize the startup and operating speed. This section describes enhancing Visual FoxPro performance by managing startup speed and optimizing SET commands.
Managing Startup Speed
The time required to load and start Visual FoxPro relates to the physical size of Visual FoxPro, the length of the PATH statement in effect, the number of items to be found at startup, and other factors. You can control the load size, the search path, component file locations, and the startup SET command values of Visual FoxPro.
Managing File Locations
Visual FoxPro stores the FoxUser.dbf file, which contains user settings, in the user’s Application Data directory by default. You can display this location by typing ? HOME(7) in the Command window. Visual FoxPro searches for the FoxUser.dbf and Config.fpw files in the following places:
In the startup application or executable file, if any.For example, you can start a Visual FoxPro application by typing the following code on the command line:Copy VFPversionNumber.exe MyApp.app – or –CopyVFPversionNumber.exe MyApp.exe If the startup application or executable file contains a Config.fpw file, the configuration file is always used. You can override settings in a Config.fpw file that are bound inside an application by specifying an external Config.fpw file, using the -C command-line switch when starting an application or Visual FoxPro.
In the working directory.
Along the path established with the PATH environment variable.
In the directory containing Visual FoxPro.
Controlling File Loading
You can also speed startup by preventing Visual FoxPro from loading files you don’t plan to use. If your application does not use the FoxUser or FoxHelp file, disable them in the Config.fpw file by using the following commands:Copy
RESOURCE = OFF
HELP = OFF
Visual FoxPro seeks all other Visual FoxPro components (GENXTAB, CONVERT, and so on) only in the Visual FoxPro directory. If you place components elsewhere, you must explicitly identify the path to those components in your Config.fpw file. For example, you might specify these locations:Copy
You can use the environment variable FOXPROWCFG to explicitly specify the location of Config.fpw. For details about the FOXPROWCFG variable, see Customizing the Visual FoxPro Environment.
Optimizing the Load Size of Visual FoxPro
If you don’t plan on using any of the Visual FoxPro components listed previously, set them to an empty string to speed startup.
To optimize the load size of Visual FoxPro, use the following syntax:Copy
cFileVariable = ""
Replace cFileVariable with _TRANSPORT, _CONVERT, or other variables as appropriate.
Optimizing Key SET Commands
You can optimize the operation of Visual FoxPro by tuning the values of certain SET commands.
The following table shows SET commands that have the greatest effect on performance, and their settings for maximum performance. You can specify SET command values by including them in the Config.fpw file, by typing them in the Command window, or by setting them in the Options dialog box.
Command Settings for Maximum PerformanceExpand table
When you run Visual FoxPro or Visual FoxPro applications in a multiuser environment, you can improve performance by managing storage of temporary files and controlling the way tables are shared.
Managing Temporary Files
In most multiuser environments, it is recommended that you save temporary files to local disks or memory when networked computers contain large amounts of free disk space. Redirecting storage of temporary files can improve performance by reducing frequent access to the network drive.
On small networks with older networked computers and slow hard disks, you might experience better performance by leaving Visual FoxPro temporary files on the file server; however, when in doubt, direct temporary files to the local disk. When working on large, heavily used networks, always redirect temporary files to the local disk.
By saving all temporary files to a single directory on a local hard drive, you can safely erase the contents of the temporary file directory on the file server prior to each Visual FoxPro session. This action purges the system of any temporary files that were created but not erased by Visual FoxPro due to a system reboot or power loss.
If users share tables on a network, the way you manage access to them can affect performance.
Avoid opening and closing tables repeatedly.
Buffer write operations to tables that are not shared.
Provide exclusive access to tables.
Limit the time on locking tables.
Providing Exclusive Access
You can enhance performance for the APPEND, REPLACE, and DELETE commands and operations that run at times when no other users require access to the data, for example, overnight updates, by opening data files for exclusive use. When tables are open for exclusive use, performance improves because Visual FoxPro does not need to test the status of record or file locks.
To open data files for exclusive use, use the EXCLUSIVE clause in the USE and OPEN DATABASE commands. For more information, see USE Command and OPEN DATABASE Command.
Limiting the Time on Locking Tables
You can reduce contention between users for write access to a table or record by shortening the amount of time for locking a record or table. Instead of locking a record while the user edits it, lock the record only after it has been edited. Using optimistic row buffering provides the shortest amount of time that records are locked. For more information, see Buffering Data.
The configuration of Visual FoxPro determines how your copy of Visual FoxPro looks and behaves. For example, you can establish the default locations for files used with Visual FoxPro, how your source code looks in an edit window, and the format of dates and times.
You can make changes to the Visual FoxPro configuration that exist for the current session only (temporary), or specify them as the default settings for the next time you start Visual FoxPro (permanent). If the settings are temporary, they are stored in memory and are discarded when you quit Visual FoxPro.
If you make permanent settings, they are stored in the Microsoft Windows registry or Visual FoxPro resource file. The Windows registry is a database that stores configuration information about the operating system, all Windows applications, OLE, and optional components such as ODBC. For example, the registry is where Windows stores the associations between file name extensions and applications so that when you click a file name, Windows can launch or activate the appropriate application.
For an example of how to change the registry, you can examine Registry.prg in the SamplesClasses directory, which contains numerous methods based on Windows API calls and makes it possible for you to manipulate the Windows registry.
Similarly, Visual FoxPro stores its application-specific configuration information in the registry. When you start Visual FoxPro, the program reads the configuration information in the registry and sets the configuration according to those settings. After reading the registry, Visual FoxPro also checks for a configuration file, which is a text file in which you can store configuration settings to override the defaults stored in the registry. After Visual FoxPro has started, you can make additional configuration settings using the Options Dialog Box or SET commands. For more information, see How to: View and Change Environment Settings.
Note
The run-time version of Visual FoxPro does not read the Windows registry when starting up, as registry settings are designed primarily to configure the development environment. If you intend to distribute your Visual FoxPro applications using a run-time library, you can establish configuration settings in two ways: with a configuration file, or with a program that manipulates the Windows registry on the user’s computer.
Visual FoxPro also maintains a resource file, Foxuser.dbf, that stores information about the current state of the program when you quit. For example, the resource file contains information about the location and size of the Command window, current keyboard macros, the toolbars that are displayed, and so on. The Foxuser.dbf file is an ordinary Visual FoxPro table, which you can read and change as required by your application.
Tip
If the data in the Foxuser.dbf file becomes corrupted or invalid, it can cause Visual FoxPro to behave in an erratic manner. If you do not manually store anything in the table, for example keyboard macros, deleting the table might help.
You can make changes to the Visual FoxPro environment by using the Options dialog box, editing the Windows registry, overriding default configuration settings, customizing the Project Manager, and configuring Visual FoxPro toolbars.
To view and change environment settings, use the Options dialog box. The Options dialog box contains a series of tabs representing different categories of environment options.
To display the Options dialog box
On the Tools menu, click Options.
The Options dialog box appears and displays tabs from which you can choose desired settings. For details about options you can set using each tab, see the Options Dialog Box (Visual FoxPro).
Displaying Environment Settings
When you run Visual FoxPro, you can verify environment settings by using the Options dialog box or the DISPLAY STATUS Command. Also, you can display the values of individual SET Commands to verify settings.
To display multiple environment settings
On the Tools menu, click Options to display the Options dialog box and view the current settings. – OR –
Type DISPLAY STATUS in the Command window.
To display individual environment settings
Use the SET( ) Function in the Command window to display the current value of any SET commands.
For example, to view the current status of SET TALK, type:Copy
? SET("TALK")
Note
Because settings are valid only for the current data session, you must capture your settings and place them in a program or a form’s Init event code for every private data session.
To echo Options dialog box settings to the Debug Output window
On the Tools menu, click Debugger.
Click the main Visual FoxPro window to select it and on the Tools menu, click Options.
In the Options dialog box, make setting choices.
Hold down the SHIFT key and click OK.The settings are echoed to the Debug Output window.
Click the Visual FoxPro Debugger window to select it and copy the setting commands from the Debug Output window.
Saving Environment Settings
You can save the settings you make in the Options dialog box for the current data session or as default (permanent) settings for your copy of Visual FoxPro.
To save settings for the current session only
In the Options dialog box, select your settings.
Click OK.
When you save settings for the current session only, they remain in effect until you quit Visual FoxPro (or until you change them again). To save changes permanently, save them as default settings. This action stores your settings in the Windows registry.
To save current settings as default settings
In the Options dialog box, select your settings.
Click Set As Default. NoteThe Set as Default button is disabled until you make a change to the current settings.
You can override default settings either by issuing SET commands or by specifying a configuration file when you start Visual FoxPro. For details, see Setting Configuration Options at Startup.
Setting the Environment Using the SET Command
You can programmatically modify most options displayed on the tabs in the Options dialog box using SET commands or by assigning a value to a system variable.
Note
When you configure the environment using SET commands, the settings take effect only for the current session of Visual FoxPro. When you quit the program, the system discards your settings. This means you must reissue the SET commands. However, you can automate this process by issuing SET commands at startup or using a configuration file. For details, see Setting Configuration Options at Startup.
Tip
To save a configuration made with SET commands, open the Options dialog box and save your settings there.
To set the environment programmatically
Use the SET commands that you want.
For example, the following lines of code set a default path, add a clock to the status bar and use a year-month-date (yy.mm.dd) format for dates:Copy
SET DEFAULT TO HOME()+"VFP"
SET CLOCK ON
SET DATE TO ANSI
You can specify a different location for temporary files using the Visual FoxPro interface or by using the TMPFILES, EDITWORK, PROGWORK, and SORTWORK settings in a Visual FoxPro configuration file.
To specify the location of temporary files
On the Tools menu, click Options.
In the Options dialog box, click the File Locations tab.
In the File Type list, click Temporary Files, then Modify.
In the Change File Location dialog box, type a new location or click the ellipsis (…) button to browse and select a location for temporary files.
How to: Change Configuration Settings in the Windows Registry
Article
07/09/2007
You can set the Visual FoxPro configuration by making changes directly in the Windows registry. To change the Windows registry, use the Registry Editor, a utility provided with Windows.
Note
Use caution when changing the Windows registry. Changing the wrong registry entry or making an incorrect entry for a setting can introduce an error that prevents Visual FoxPro, or even Windows itself, from starting or working properly.
To change configuration settings in the registry
In Windows, start the Registry Editor.
In HKEY_CURRENT_USER node, browse to the SoftwareMicrosoftVisual FoxPro directory and open the folder for the current version of Visual FoxPro.
In the Options folder, double-click the name of the setting to change, and then enter a new value.
Close the Registry Editor.Your change will take effect the next time you start Visual FoxPro.
You can also make changes to the registry by calling Windows APIs from a Visual FoxPro program.
You can establish configuration settings when you first start the program, which allows you to override default settings.
Using SET Commands in Applications
One way to establish configuration settings is to issue one or more SET commands when your application starts. For example, to configure your system to display a clock in the status bar when the application starts, you can issue this SET command:Copy
SET CLOCK ON
The exact point at which you issue the SET command depends on your application. In general, you issue SET commands from your application’s main program file, which is the program or form that controls access to the rest of your application. You can also issue SET commands from the Load or Init events of the form. If you are using private data sessions, it may be necessary to make these settings in the BeforeOpenTables Event of your DataEnvironment object. For details about specifying a main file for an application, see Compiling an Application.
An efficient way to manage SET commands for startup is to create a procedure that contains all the commands that you want to issue. You can then call the procedure from the appropriate point in your application. Keeping all the SETUP commands in a single procedure makes it easier to debug and maintain your configuration settings. You can also put the code in the class on which your application object is based, or the class on which your forms are based.
Using a Configuration File
In addition to setting the Visual FoxPro environment using the Options dialog box or SET commands, you can establish preferred settings and save them in one or more configuration files. A Visual FoxPro configuration file is a text file in which you can specify values for SET commands, set system variables, and execute commands or call functions. Visual FoxPro reads the configuration file when starting up, establishing the settings and executing the commands in the file. Settings made in the configuration file override default settings made in the Options dialog box (and stored in the Windows registry).
Using a configuration file provides several advantages. You can:
Override the default settings established in the Options dialog box.
Maintain several different configuration files, each with different settings, so that Visual FoxPro can load a configuration suitable to a particular user or project.
Make changes more easily than if you establish settings with the SET commands in the program initialization sequence.
Start a program or call a function automatically when Visual FoxPro starts.
When Visual FoxPro starts, you can specify a configuration file or bypass all configuration files, allowing Visual FoxPro to use its default settings.
When Visual FoxPro loads a configuration file, the settings in that file take precedence over corresponding default settings made in the Options dialog box.
To specify a configuration file
In the command line that starts Visual FoxPro, specify the -C switch and the name of the configuration file that you want to use (including a path if necessary). Do not put a space between the switch and the file name.-or-
In Windows, double-click the name of the configuration file to use. Visual FoxPro will start using the configuration file you have selected.
If you want to avoid using any configuration file, including the default file Config.fpw, you can suppress all configuration files. This causes Visual FoxPro to use only the default settings established in the Options dialog box.
To suppress a configuration file
In the command line that starts Visual FoxPro, add the -C switch with nothing after it.For example, to avoid any configuration file found in the startup directory or the system path, use this command line:CopyVFPVersionNumber.exe -C
Specifying an External Configuration File
You can use an external configuration file in addition to an internal configuration file in circumstances where you need to configure settings separately. For example, setting SCREEN=OFF should be performed in an internal configuration file.
You can set Visual FoxPro to read an external configuration file following an internal configuration file by using the new ALLOWEXTERNAL directive in the internal configuration file. When you include the setting ALLOWEXTERNAL=ON in the internal configuration file, Visual FoxPro searches for an external configuration file, usually Config.fpw, and reads its settings. You can also specify a different configuration file using the -C command-line switch when starting Visual FoxPro.
Note
For .exe and .dll file servers, Visual FoxPro supports only those configuration files bound inside the server. Therefore, Visual FoxPro disregards the ALLOWEXTERNAL setting.
To read an external configuration file after an internal one
In the internal configuration file, set the special term ALLOWEXTERNAL to on.CopyALLOWEXTERNAL = ON
When you start your program, either specify a second configuration file using the -C command-line switch or have a second configuration file in the default program path.
The settings in an external configuration file take precedence over those in the internal configuration file, if duplicate settings exist, because the external configuration file is read after the internal file. Visual FoxPro does not begin initialization until it reads both files.
If you want to specify the configuration file as read-only, place the file in your project and mark it as Included. If you want to specify that the file can be modified, place the file in your project and mark it as Excluded. You can then distribute the file separately with your application or executable file. By convention, configuration files use the .fpw extension.
To create a configuration file, use the Visual FoxPro editor, or any editor that can create text files, to create a text file in the directory where Visual FoxPro is installed. Earlier versions of Visual FoxPro created the file Config.fpw in the startup directory. Config.fpw became the default configuration file. You can create any program file and use it to establish default settings and behaviors by starting Visual FoxPro using that file either by double clicking the file or using a command line reference.
If you are creating a new configuration file, you can save it using any name you want. By convention, configuration files have the extension .fpw.
When you start Visual FoxPro, you can use a default configuration file from the following locations in order:
Current working directory
Directory where Visual FoxPro is installed
Directories listed in the DOS path
If Visual FoxPro does not find a configuration file in these locations, Visual FoxPro uses only the default settings established in the Options dialog box.
Note
For details about specifying an alternative to the default file name or location for the configuration file, see How to: Specify the Configuration File.
Enter configuration settings using one of these methods:
Make settings with the SET command.
Set system variables.
Call programs or functions.
Include special terms used only in configuration files.
To enter SET commands in a configuration file
Enter SET commands without the SET keyword and with an equal sign.For example, to set a default path type, use this format:CopyDEFAULT = HOME()+"VFP" To add a clock to the status bar, use this command:CopyCLOCK = ON
To enter a setting for a system variable, use the same syntax you would use in the Command window or in a program.
To set system variables in a configuration file
Enter the name of the system variable, an equal sign (=), and the value to set the variable to.For example, the following command specifies an alternative spell-checking program:Copy_SPELLCHK = "SPLLCHK.EXE"
You can also call functions or execute programs from within a configuration file by using the COMMAND command. For example, you can start an initialization program as part of the startup process.
To call functions or execute commands in a configuration file
Enter COMMAND, an equal sign (=), and the command to execute or function to call.For example, to include the Visual FoxPro version number in the caption of the main Visual FoxPro window, use this command:CopyCOMMAND =_SCREEN.Caption="Visual FoxPro " + VERS(4) The following command launches a specific application when Visual FoxPro starts:CopyCOMMAND = DO MYAPP.APP
You can also use special terms in a configuration file that do not correspond to SET value, system variables, or commands.
To use special terms in a configuration file
Enter the special term, an equal sign (=), and the setting.For example, to set the maximum number of variables available in Visual FoxPro, use this command:CopyMVCOUNT = 2048
You can insert commands into a configuration file that automatically launches programs when Visual FoxPro starts. You can use these commands either to start an entire application or to start just a program, such as one that initializes system variables.
To start applications from a configuration file
Assign the name of your application to the _STARTUP System Variable anywhere in the configuration file:Copy_STARTUP = MYAPP.APP -or-
Use the COMMAND command, which must be the last line in your configuration file:CopyCOMMAND = DO MYAPP.APP
The following table lists special terms you can use in configuration files. You can also SET commands, system variables, and the _STARTUP setting in configuration files to customize the Visual FoxPro environment.
Remarks
Expand table
Term and syntax
Description
ALLOWEXTERNAL ON | OFF
Specifies whether settings from an external configuration file as specified by the -C command-line switch (or located in path) are read in after those from an internal one. The ALLOWEXTERNAL term is ignored unless it is bound inside of an application.Default: OFF
BITMAP ON | OFF
Specifies whether Visual FoxPro first writes screen or form updates to an off-screen bitmap, and then performs a bit block transfer (bitblt) to the screen. BITMAP = OFF can improve performance when application are accessed using Windows Terminal Server clients.Default: ON
CODEPAGE = nValue | AUTO
Specifies a number that identifies the character set used for files. Setting CODEPAGE to AUTO selects the current operating system code page.For the possible values you can use, see Code Pages Supported by Visual FoxPro.
COMMAND = cVisualFoxProCommand
Specifies a Visual FoxPro command to execute when Visual FoxPro is started. The cVisualFoxProCommand specifies the command to execute.
EDITWORK path
Specifies where the text editor should place its work files. Because work files can become large, specify a location with plenty of free space.Default: Operating system dependent. For more information, see Optimizing the Operating Environment.
INDEX extension
Specifies the extension for Visual FoxPro index files.Default: .idx
LABEL extension
Specifies the extension for Visual FoxPro label definition files.Default: .lbx
_MENUDESIGNER = cProgramName
Specifies an external menu design application.Default: The empty string “”.
MVCOUNT
Sets the maximum number of variables that Visual FoxPro can maintain. This value can range from 128 to 65,000.Default: 16,384
OUTSHOW = ON | OFF
Disables the ability to hide all windows in front of the current output by pressing SHIFT+CTRL+ALT.Default: ON
PROGCACHE = nMemoryPages
Specifies the amount of memory (address space) in pages that Visual FoxPro allocates at startup or a Visual FoxPro MTDLL COM Server allocates per thread for the internal program cache (memory used to run programs). Each page of memory is equal to 64K so the default setting equates to an allocation a little over 9MB. As the cache is filled, Visual FoxPro will try to flush it to remove unused items. It is possible that Visual FoxPro cannot free enough memory in which case an Error 1202 is generated (Program is too large). Adjusting the PROGCACHE setting can prevent this error from occurring. NoteWhile this setting can be used for the Visual FoxPro development product or normal runtime applications, it is primarily intended for MTDLL COM Servers where many threads are often created for a single server. In Visual FoxPro 9.0, the default value for MTDLL COM Servers is -2.When the value of nMemoryPages is greater than 0, Visual FoxPro allocates a fixed program cache. You can specify between 1 and 65000.If you specify 0 for nMemoryPages, no program cache is used. Instead, Visual FoxPro uses dynamic memory allocation based on determinations made by the operating system.If you pass a value for nMemoryPages that is less than 0, Visual FoxPro uses dynamic memory allocation but is limited to the specified memory (nMemoryPages * 64K). When the limit is reach, Visual FoxPro will flush allocated programs to free memory.You can call SYS(3065) to determine the current PROGCACHE setting. CLEAR PROGRAM will attempt to clear unreferenced code regardless of this setting. NoteThe Visual FoxPro OLE DB Provider ignores this setting since it uses dynamic memory allocation (PROGCACHE=0).Default: 144 (-2 for MTDLL)
PROGWORK path
Specifies where Visual FoxPro stores the program cache file.For faster performance, especially in a multiuser environment, specify a fast disk, such as a local disk or memory, if available. Provide at least 256K for the cache, though the file can grow larger.Default: Operating system dependent. For more information, see Optimizing the Operating Environment.
REPORT extension
Specifies the extension for Visual FoxPro report definition files.Default: .frx
RESOURCE path[file] | OFF
Specifies the location of the FoxUser.dbf resource file or prevents Visual FoxPro from using a resource file. The file argument is optional. If file is omitted, Visual FoxPro searches for the FoxUser.dbf file. If the specified file does not exist, it is created.Default: Startup directory as path and FoxUser.dbf as file.
SCREEN = ON | OFF
Specifies whether the main Visual FoxPro window appears when opening Visual FoxPro.When an application consists of one or more top-level forms that are displayed in the Windows desktop, setting SCREEN to OFF can be useful, making the main Visual FoxPro window not required. For further information on top-level forms, see Controlling Form Behavior.Default: ON
SORTWORK path
Specifies where commands such as SORT and INDEX should place work files.Because work files can be up to twice as large as the tables being sorted, specify a location with plenty of free space. For faster performance, especially in a multiuser environment, specify a fast disk such as a local disk.Default: Operating system dependent. For more information, see Optimizing the Operating Environment.
STACKSIZE = nValue
Specifies the number of nesting levels from 32 to 64,000 for operations such as the DO command. NoteYou can change the nesting level only during Visual FoxPro startup.Default: 128
TEDIT [/N] editor
Specifies the name of the text editor used when you edit program files with MODIFY COMMAND or MODIFY FILE.Include the optional clause /N with TEDIT to specify a Windows text editor.Default: Visual FoxPro editor
TITLE title
Specifies the title that appears in the caption bar of the main Visual FoxPro window.Default: “Microsoft Visual FoxPro”
TMPFILES path
Specifies where temporary work files specified by EDITWORK, SORTWORK, and PROGWORK configuration file settings are stored if they are not specified.Because work files can become very large, specify a location with plenty of free space. For faster performance, especially in a multiuser environment, specify a fast disk such as a local disk.Default: Operating system dependent. For more information, see Optimizing the Operating Environment.
How to: Use Command-Line Options When Starting Visual FoxPro
Article
07/09/2007
In addition to using the SET command and a configuration file, you can specify startup options by including a command-line switch. For example, using command-line options, you can suppress the display of the Visual FoxPro splash screen, which displays at the startup of Visual FoxPro, or specify a nondefault configuration file.
To use a command-line switch
On the command line or in a shortcut, add the switch after the name of the Visual FoxPro executable file, VFPVersionNumber.exe where VersionNumber represents the version number of this release or any Visual FoxPro-created .exe file. NoteIf the command-line switch requires arguments, such as a file name, do not put a space between the switch and the argument. For example, to specify a configuration file, use a command such as:CopyC:Program FilesMicrosoft Visual FoxPro VersionNumberVFPVersionNumber.exe -CC:MYAPP.FPW Separate multiple options with single spaces.
The following table lists the command-line switches available in Visual FoxPro.Expand table
Switch
Description
-A
Ignore the default configuration file and Windows registry settings.
-BFileName,Duration
Display a custom bitmap (.bmp), .gif, or .jpg graphic file and specify its display duration in milliseconds when Visual FoxPro starts. You can also include the -B command-line switch in a Visual FoxPro shortcut. NoteIf the bitmap you specify cannot be located, the bitmap does not display when Visual FoxPro starts.
-CFileName
Specify a configuration file, including a path if necessary, other than the default file, Config.fpw.
-LFileName
Specify a resource file, including a path if necessary, other than the default, vfp*ENU.dll, so you can use Visual FoxPro in a language other than the current language specified by Windows.
-R
In earlier versions, refresh the Windows registry with information about Visual FoxPro, such as associations for Visual FoxPro files. In later versions, use /regserver.
-T
Suppress the display of the Visual FoxPro splash screen. By default, when Visual FoxPro starts, it displays a splash screen that shows the Visual FoxPro logo, version number, and other information. If you prefer that users of your application not see this splash screen, you can prevent Visual FoxPro from displaying it using the -T command-line switch.
/?
List the available command-line arguments. Available in Visual FoxPro 7.0 and later.
Visual FoxPro includes the following customizable toolbars.Expand table
Tool
Associated Toolbars
Command
Database Designer
Database
CREATE DATABASE
Form Designer
Form ControlsForm DesignerColor PaletteLayout
CREATE FORM
Print Preview
Print Preview
Query Designer
Query Designer
CREATE QUERY
Report Designer
Report ControlsReport DesignerColor PaletteLayout
CREATE REPORT
You can place as many toolbars on your screen as you need while working. You can dock toolbars to the top, bottom, or sides of your screen to customize your working environment. Visual FoxPro saves the positions of the toolbars so they remain where you last placed them.
To dock a toolbar
Drag the toolbar to the top, bottom, or side of your screen.-OR-
By default, only the Standard toolbar is visible. When you use a Visual FoxPro designer tool (for example, the Form Designer), the designer displays the toolbars that you commonly need when working with that designer tool. However, you can activate a toolbar any time you require it.
To activate a toolbar
Run the associated tool.–OR–
To deactivate a toolbar
Close the associated tool.–OR–
You can also programmatically activate and deactivate toolbars that have been previously activated by using the DEACTIVATE WINDOW or ACTIVATE WINDOW commands as in the following example.Copy
IF WVISIBLE ("Color Palette")
DEACTIVATE WINDOW("Color Palette")
ENDIF
Customizing Existing Toolbars
The easiest way to create custom toolbars is by modifying the toolbars already provided with Visual FoxPro. You can:
Modify an existing toolbar by adding or removing buttons.
Create a new toolbar that contains buttons from existing toolbars.
You can also define custom toolbars by creating a custom toolbar class using code. For details, see Designing Menus and Toolbars.
You can modify any of the toolbars provided with Visual FoxPro. For example, you might want to remove a button from an existing toolbar, or copy buttons from one toolbar to another.
To modify an existing Visual FoxPro toolbar
From the View menu, choose Toolbars.
Select the toolbar you want to customize and choose Customize.
Remove buttons from the toolbar by dragging them off of the toolbar.
Add buttons to the toolbar by selecting an appropriate category in the Customize Toolbar dialog box and then dragging the appropriate buttons onto the toolbar.
Complete the toolbar by choosing Close in the Customize Toolbar dialog box and then closing the toolbar window. TipIf you change a Visual FoxPro toolbar, you can restore it to its original configuration of buttons by selecting the toolbar in the Toolbar dialog box and then choosing Reset.
You can create your own toolbars comprised of buttons from other toolbars.
To create your own toolbar
From the View menu, choose Toolbars.
Choose New.
In the New Toolbar dialog box, name the toolbar.
Add buttons to the toolbar by selecting a category in the Customize Toolbar dialog box and then dragging the appropriate buttons onto the toolbar.
You can rearrange buttons on the toolbar by dragging them to the desired position.
Complete the toolbar by choosing Close in the Customize Toolbar dialog box and then closing the toolbar window. NoteYou cannot reset buttons on a toolbar you create.
To delete a toolbar you created
From the View menu, choose Toolbars.
Select the toolbar you want to delete.
Choose Delete.
Choose OK to confirm the deletion. NoteYou cannot delete toolbars provided by Visual FoxPro.
You can dock certain Visual FoxPro Integrated Development Environment (IDE) windows to the Visual FoxPro desktop window, each other, or user-defined forms. The following windows are dockable:
Call Stack
Command
Data Session (View)
Debugger
Document View
Locals
Output
Properties
Trace
Watch
Note
Historically in prior versions of Visual FoxPro, the Data Session window has always been referred to as the View window. Additionally, language used to control this window, such as HIDE WINDOW, ACTIVATE WINDOW, WONTOP( ), also refers to this window as the View window.
When you drag a dockable window to a Visual FoxPro desktop window boundary, the dockable window repositions itself against the chosen boundary. The state and location of docked windows persist from the last user session.
You can change the dockable status of a dockable window.
To enable or disable dockable status
Right-click the title bar of an open and dockable window.
Click Dockable to enable or disable the dockable status of the window. A check mark appears when the window is dockable.-OR-
Click the desired window to make the window active.
On the Window menu, click Dockable to enable or disable the dockable status of the window. A check mark appears when the window is dockable.
For more information about retrieving the dockable status for a window programmatically, see WDOCKABLE( ) Function.
Docking Modes
You can dock windows in three different modes:
Normal dockingWindows dock to a boundary of the Visual FoxPro desktop window.
Linked dockingWindows dock to each other and share a dockable window container.
Tabbed dockingWindows dock to each other and share the full window through the use of tabs.
You can use tabbed docking and linked docking together.
To create normal docking
Make sure the docking status of the window or windows is set to Dockable.
Drag the window title bar to a boundary of the Visual FoxPro desktop window.
To create linked docking
Make sure the docking status of the window or windows is set to Dockable.
Drag the title bar of the desired window to a boundary or docking zone of the target window.
The docking zone is indicated when the window you are dragging changes shape to fit the target window. Visual FoxPro creates an additional title bar for link-docked windows.
To create tabbed docking
Make sure the docking status of the window or windows is set to Dockable.
Drag the title bar of the desired window to the title bar of the target window.
Visual FoxPro adds tabs to the bottom boundary of the docked windows.
To undock windows
To undock normal-docked windows, drag the title bar of the desired window away from the shared window boundary.-or-
To undock link-docked windows, drag the title bar of the desired window away from the shared window.-or-
To undock tab-docked windows, drag the tab of the desired window away from the shared window.
You can disable docking behavior by holding the CTRL key while dragging a window.
Deleting or editing the FoxUser.dbf resource file, which contains your settings, restores or changes your default window settings. For more information about altering your settings, see FoxUser Resource File Structure.
You can configure the window of the Visual FoxPro editor to display text the way you want such as setting the font, text alignment, or syntax coloring. You can also make the editor easier to use by setting preferences for indentation, wordwrap, automatic backup copies, and other features.
To configure the editing window, set your preferences in the Edit Properties dialog box that appears after you open a program or text file open and select Properties from the Edit menu. For details about settings you can configure, see Edit Properties Dialog Box.
To display the Edit Properties dialog box
Open the editing window for a program, text file, or control.
From the Edit menu that becomes active, select Properties. TipYou can display the Font dialog box directly by right-clicking the editing window and choosing Font from the shortcut menu.
By default, the settings that you make in the Edit Properties dialog box persist for that file. For example, if you change the font, the font for all text in the current window changes. If you open another editing window, the default settings apply.
You can choose to save your settings so that they apply to all files of the same type, or not to save the new settings at all. If you apply your settings to similar file types, Visual FoxPro uses the settings you make when you edit files with the same extension, for example, all .prg files, or all method code in the Form Designer.
To avoid persisting changes to editor settings
In the Edit Properties window, clear the Save Preferences option and then click OK.
To apply editor options to similar files
In the Edit Properties dialog box, select Use These Preferences As Default and then click OK.
You can also set the color and font that the editor uses to identify keywords, comments, and other elements of programs. For details, see Editor Tab, Options Dialog Box.
You can display and print code with color syntax in the Command window and Visual FoxPro editors for program (.prg) files, methods, stored procedures, and memos.
The following sections contain more information about displaying and printing code in color:
Displaying Source Code in Color
Printing Source Code in Color
Displaying Source Code in Color
You can turn on syntax coloring separately for each editing window; however; the color syntax settings that you choose in the Editor tab of the Options dialog box apply to the Command window and most other editing windows. For more information, see Editor Tab, Options Dialog Box and Editing Window.
Note
To display color syntax in run-time applications, Visual FoxPro must be configured to display color syntax. Run-time applications display only the default color settings because run-time applications do not check the Windows registry for settings that you change in the Options dialog box, which specifies the default settings for color syntax.
When color syntax is turned on, Visual FoxPro performs background compilation for the current and single line of code that you are typing. When a line of code contains invalid syntax, Visual FoxPro displays the line of code with the selected formatting style.
To display color syntax in an editing window
Open the editing window that you want to display color syntax.
On the Edit menu, choose Properties.
In the Edit Properties dialog box, select the Syntax coloring check box.
Click OK.
Syntax coloring is activated for the editing window you selected. For more information, see Edit Properties Dialog Box.
You can also display source code files to the screen in color using the TYPE command. For more information, see TYPE Command.
To customize color syntax settings
On the Tools menu, choose Options.
In the Options dialog box, choose the Editor tab.
Under Syntax color settings, choose the color settings that you want.
When you are finished choosing settings, click OK.
The color settings you chose take effect.
To set formatting for invalid color syntax
Turn on syntax coloring for the editing window.
On the Tools menu, choose Options.
In the Options dialog box, choose the Editor tab.
In the Background Compile box, choose the formatting style you want.
Invalid syntax displays in the editing window with the formatting style you chose.
Printing Source Code in Color
You can print source code in color wherever color syntax appears, such as program files, methods, stored procedures, and memos.
Note
To print your code files in color, you must be connected to a color printer to select the color printing options available for your printer. Background colors set in the Editor tab of the Options dialog box are not printed. If you select color printing for source code files, hyperlinks appear underlined.
To print source code in color
Open the program file or code you want to print.
On the File menu, choose Print.
In the Print dialog box, choose your color printer, and then click Preferences.
Select the color printing options available for your printer.
When you are finished, click Print.
When connected to a color printer, you can also print source code in color using the TYPE command with the TO PRINTER clause.
If you find yourself repeating the same keystrokes, or if you want to automate some keystroke tasks for your users, you can record and save these keystrokes in macros by using the Macros dialog box.
Note
To use macros, any keyboard you use must have function keys F1 to F9, and a CTRL key or an ALT key.
Visual FoxPro Environment Settings Describes different ways to change Visual FoxPro environment settings such as using the Options dialog box, setting configuration options at program startup, using command-line options. You can configure Visual FoxPro toolbars, dock windows, set editor options, and customize the appearance of your applications without altering code.
Optimizing Your System Provides information about optimizing your operating environment.
Development Productivity Tools Discusses the different tools available to help make creating Visual FoxPro applications easier and faster.
A macro set is a defined set of keys and their associated macros stored in a file with the extension .FKY. You can create macro sets through the Macros Dialog Box.
To create a macro set
From the Macros dialog box, create individual key macros.
Choose Save.
In the Save Current Macros To box, enter a name for the macro set and choose Save.
In the Visual FoxPro Documentation What’s New in Visual FoxPro Describes the new features and enhancements included in this version of Visual FoxPro.
Getting Started with Visual FoxPro Provides information about where to find the Readme file, installing and upgrading from previous versions, configuring Visual FoxPro, and customizing the development environment.
Using Visual FoxPro Provides an overview of Visual FoxPro features, describes concepts and productivity tools for developing, programming, and managing high-performance database applications and components.
Samples and Walkthroughs Contains Visual FoxPro code samples and step-by-step walkthroughs that you can use for experimenting with and learning Visual FoxPro features.
Reference Includes Visual FoxPro general, programming language, user interface, and error message reference topics.
Product Support Provides information about Microsoft product support services for Visual FoxPro.
Provides a link to the Visual FoxPro training Web site to find information about training, books, and events for Visual FoxPro. Accessibility for People with Disabilities
Provides information about features that make Visual FoxPro more accessible for people with disabilities.
This release of Visual FoxPro contains many new features and enhancements. The following sections describe these new features and enhancements.
Provides information about where to find the ReadMe file and how to install and upgrade from previous versions, configure Visual FoxPro, and customize the development environment.Using Visual FoxPro
Provides an overview of Visual FoxPro features, describes concepts and productivity tools for developing, programming, and managing high-performance database applications and components, and provides walkthroughs that help get you started. With the robust tools and data-centric object-oriented language that Visual FoxPro offers, you can build modern, scalable, multi-tier applications that integrate client/server computing and the Internet.Samples and Walkthroughs
Contains Visual FoxPro code samples and step-by-step walkthroughs that you can use for experimenting with and learning Visual FoxPro features.Reference (Visual FoxPro)
Describes Visual FoxPro general, programming language, user interface, and error message reference topics.Product Support (Visual FoxPro)
Provides information about Microsoft product support services for Visual FoxPro.
Guide to Reporting Improvements
Visual FoxPro 9’s Report System has undergone a thorough revision. This topic sketches the broad outlines of the changes, and provides you with information about where to look for details.
The following main areas of enhancements to the Report System are covered in sections of this topic. Design-time enhancements.
Multiple features and changes make designing reports in Visual FoxPro better for you and your end-users. The Report Builder Application re-organizes your design experience out-of-the-box. If you want to customize the design process, Report Builder dialog boxes and Report Designer events are fully exposed for you to do so. Multiple detail bands.
You can handle multiple child tables and data relationships more flexibly in the revised Report Designer. When you run multiple-detail-band reports, you can leverage the new bands, with associated detail headers and footers, both for appropriate presentation of these relationships and for more capable calculations. Object-assisted run-time report processing.
An entirely re-built output system, including a new base class, changes the way Visual FoxPro provides output report and label files at run time. Object-assisted reporting provides better-quality output, new types of output, and an open-architecture based on a new Visual FoxPro base class, the ReportListener. A programmable Report Preview interface interacts with ReportListeners to give you full control over report preview experience. The Report Preview Application provides improved out-of-the-box previewing facilities. Printing, rendering, and character-set-handling improvements.
Visual FoxPro 9 makes better use of the operating system’s printing features and GDI+ rendering subsystem. It also handles multiple locales and character sets better than previous versions. These changes are showcased in the Report System, and are accessible for use in custom code during report design and run-time processing. Extensible use of report and label definition files (.frx and .lbx tables).
Visual FoxPro 9 handles your existing reports and labels without modification, while allowing you to add new features and behavior to these reports easily. This backward-compatible, yet forward-thinking, migration strategy is made possible by the Report System’s newly-flexible handling of the .frx and .lbx table structure.
Design-time Enhancements
Numerous changes in the Report System help you enhance the design-time experience for developers and end-users. This section directs you to information about design-time improvements.
Report Designer Event Hooks and the Report Builder Application
The Report Designer now offers Report Builder Hooks, which enable you to intercept events occurring during a report or label design session to override and extend designer activity. The default Report Builder Application replaces many of the standard reporting dialog boxes with new ones written in Visual FoxPro code. Components of the Report Builder Application are exposed as Visual FoxPro Foundation Classes for your use.
Protection for End-User Design Sessions, and other Design-time Customization Opportunities
You can allow end-users to MODIFY and CREATE reports and labels, while setting limitations on what they can do in the Report Designer interface, using the new PROTECTED keyword. Protection is available individually by object and globally for the report. You can change what end-users see on the designer layout surface, from complex expressions to simple labels or sample data, while working in PROTECTED design mode, using Design-Time Captions. You can also provide helpful instructions, for both PROTECTED and standard design mode, by specifying Tooltips for report controls.
You can save the Data Environment you designed for a Report or Label as a visual class. You can load a Data Environment into a Report or Label design from either a visual class or a previously-saved report or label.
There have been numerous enhancements to the Report and Label Designers. Some features are subtle changes to make design sessions more efficient and more enjoyable, and others improve your choices for resulting output.
To learn about:
Read:
Improvements to the Report and Label Interactive Development Environment (IDE), such as: Enhanced Report Designer toolbar, and easier access to the Report Designer Toolbar from the View menu New global Report Properties context menu Improvements and additions to existing context menus Revised and extended Report menu
Report document properties enable you to include information about the report in the report. Document properties are included as elements and attributes in XML and HTML output.
The Report Engine can now move through a scope of records multiple times. The records can represent related sets of detail lines in child tables, or they can be multiple passes through a single table. These multiple passes through a scope of records are represented as multiple detail bands.
Detail bands can have their own headers and footers, their own associated onEntry and onExit code, and their own associated report variables. Each detail band can be explicitly associated with a separate target alias, allowing you to control the number of entries in each detail band separately for related tables.
Multiple detail band reports provide many new ways you can represent data in reports and labels, and new ways you can calculate or summarize data, as you move through a record scope.
To learn about:
Read:
Designing reports and labels with multiple detail bands and their associated headers and footers
Visual FoxPro 9 has a new, object-assisted method of generating output from reports and labels. You can use your existing report and label layouts in object-assisted mode, to:
Generate multiple types of output during one report run.
Connect multiple reports together as part of one output result.
Improve the quality of traditional report output.
Dynamically adjust the contents of a report while you process it.
Provide new types of output not available from earlier versions of Visual FoxPro.
This section covers the array of run-time enhancements that work together to support object-assisted reporting mode.
Object-Assisted Architecture and ReportListener Base Class
The new ReportListener base class and supporting language enhancements are the heart of run-time reporting enhancements.
To learn about:
Read:
Fundamentals of the architecture, how its components work together, and what happens during an object-assisted report run
New Types of Output and the Report Output Component Set
Because you can subclass ReportListener, you can create new types of output. Visual FoxPro 9 supplies a Report Output Application to connect ReportListener subclasses with output types, as well as ReportListener-derived classes with enhanced output capabilities.
To learn about:
Read:
Requirements for Report Output Application, and how Visual FoxPro uses Report Output Applications
Migration Strategies and Changes in Output Rendering
You can use the design-time changes to improve all reports and labels, whether you choose backward-compatible or object-assisted reporting mode at run time.
When evaluating whether to switch to object-assisted reporting mode at run time, first consider items on the Reporting list of Important Changes in the Changes in Functionality for the Current Release topic, some of which are specific to this new method of creating output. .The topic includes a table of minor differences between backward-compatible and object-assisted reporting output. You can examine what effects these changes might have on individual existing reports, and use the recommendations in the table to address them. You will find additional details in the topic Using GDI+ in Reports.
Once you have experimented with your current reports, you can decide on a migration strategy for output:
You can switch applications over to use object-assisted reporting mode completely, by using the command SET REPORTBEHAVIOR 90.
You can use SET REPORTBEHAVIOR 90 but preface specific REPORT FORM commands for reports with formatting issues with SET REPORTBEHAVIOR 80, returning your application to object-assisted mode afterwards.
You can use object-assisted mode all the time, but adjust your ReportListener-derived classes’ behavior to suit specific needs. For example, you could change the default setting of the ReportListener’s DynamicLineHeight Property to False (.F.).
You can leave SET REPORTBEHAVIOR at its default setting of 80, and add an explicit OBJECT clause to specific reports at your leisure, as you have the opportunity to evaluate and adjust individual report and label layouts.
Printing, Rendering, and Character-set-handling Improvements
General changes to Visual FoxPro’s use of Windows’ printing, rendering and font-handling support the improvements in the Report System’s output. These changes enhance your ability to support multiple printers and multiple languages in reports.
To learn about:
Read:
GDI+ features and their impact on native Visual FoxPro output
Visual FoxPro reporting enhancements that allow your code to use GDI+ in object-assisted reporting mode, and Visual FoxPro Foundation Classes to get you started
Making full use of multiple character sets, or language scripts, in reports, for single report layout elements, for report defaults, or globally in Visual FoxPro
Changes to page setup dialog boxes in Visual FoxPro, improvements in your programmatic access to them, and providing overrides to Printer Environment settings in report and label files
Extensible Use of Report and Label Definition Files
Underneath all the changes to the Visual FoxPro Report System, the Report Designer and Report Engine handle your report and label definitions using the same .frx and .lbx file structures as they did in previous versions. They change the way they use certain fields, without making these reports and labels invalid in previous versions, and they also allow you to extend your use of existing fields or add custom fields.
Tip:
This change is critical to your ability to create extensions of the new reporting features. For example, you might store two sets of ToolTips in two report extension fields, one set for use by developers and one for use by end-users. In a Report Builder extension, you could evaluate whether the Designer was working in protected or standard mode, and replace the actual set of ToolTips from the appropriate extension field. In previous versions, you could not add fields to report or label structure; the Designer and Engine would consider the table invalid. You also could not add custom content to unused, standard fields in various report and label records safely, because the Report Designer removed such content.
Visual FoxPro 9 provides a revised FILESPEC table for report and label files, with extensive information on the use of each column in earlier versions as well as current enhancements.
Visual FoxPro 9 also establishes a new, structured metadata format for use with reports. This format is an XML document schema shared with the Class Designer’s XML MemberData.
The XML document format allows you to pack custom reporting information into a single report or label field. The default Report Builder Application makes it easy to add Report XML MemberData to report and label records.
To learn about:
Read:
How Visual FoxPro uses .frx and .lbx tables, and how to extend these structures
Varbinary To store binary values and literals of fixed length in fields and variables without padding the field with additional zero (0) bytes or truncating any trailing zero bytes that are entered by the user, use the Varbinary data type. Internally, Visual FoxPro binary literals contain a prefix, 0h, followed by a string of hexadecimal numbers and are not enclosed with quotation marks (“”), unlike character strings. For more information, see Varbinary Data Type. You can specify binary type mapping between ODBC, ADO, and XML data source types and CursorAdapter and XMLAdapter objects using the MapBinary Property. You can also specify binary mapping for SQL pass-through technology and remote views using the MapBinary setting in the CURSORSETPROP( ) function. For more information, see CURSORSETPROP( ) Function and CURSORGETPROP( ) Function.
Blob To store binary data with indeterminate length, use the Blob data type. For more information, see Blob Data Type.
Many of the Visual FoxPro language elements affected by these new data types are listed in the topics for the new data types.
If a table is not selected and an alias is not specified, Error 52, “No table is open in the current work area,” is generated. If the cursor selected is not valid, Error 1467, “Property is invalid for local cursors,” is generated.
The ISMEMOFETCHED( ) function returns True (.T.) when the Memo field is fetched or if local data is used. ISMEMOFETCHED() returns NULL if the record pointer is positioned at the beginning of the cursor or past the last record.
The following table lists the data types along with their long type names and short type names.
Data Type
Long Type Name
Short Type Name
Character
Char, Character
C
Date
Date
D
DateTime
Datetime
T
Numeric
Num, Numeric
N
Floating
Float
F
Integer
Int, Integer
I
Double
Double
B
Currency
Currency
Y
Logical
Logical
L
Memo
Memo
M
General
General
G
Picture
Picture
P
Varchar
Varchar
V
Varbinary
Varbinary
Q
Blob
Blob
W
Visual FoxPro allows ambiguous long type names to be used with the ALTER TABLE, CREATE CURSOR, CREATE TABLE, and CREATE FROM commands. If the specified long type name is not a recognized long type name, Visual FoxPro will truncate the specified name to the first character.
Use SET VARCHARMAPPING to Control Query Result Set Mappings
For queries such as SELECT – SQL Command, character data is often manipulated using Visual FoxPro functions and expressions. Since the length of the resulting field value may be important for certain application uses, it is valuable to have this Character data mapped to Varchar data in the result set. The SET VARCHARMAPPING command controls whether Character data is mapped to a Character or Varchar data type. For more information, see SET VARCHARMAPPING Command.
You can disable Memo Tips by setting the _SCREEN ShowTips Property to False (.F.).
Memo and Field Tips will also be displayed for Grid controls if both _SCREEN and the form’s ShowTips property are set to True (.T.). Additionally, the ToolTipText Property for the field’s grid column Textbox control must contain an empty string.
In Visual FoxPro 8.0, the REFRESH( ) Function provides on demand record refresh functionality for local and remote views, however, it does not support this for the CursorAdapter. Visual FoxPro 9.0 extends REFRESH( ) support to the CursorAdapter and provides some additional capabilities:
Member
Description
RecordRefresh method
Refreshes the current field values for the target records. Use the CURVAL( ) Function to determine current field values.
BeforeRecordRefresh event
Occurs immediately before the RecordRefresh method is executed.
AfterRecordRefresh event
Occurs after the RecordRefresh method is executed.
RefreshCmdDataSourceType property
Specifies the data source type to be used for the RecordRefresh method.
RefreshCmdDataSource property
Specifies the data source to be used for the RecordRefresh method.
RefreshIgnoreFieldList property
List of fields to ignore during RecordRefresh operation
RefreshCmd property
Specifies the command to refresh rows when RecordRefresh is executed.
RefreshAlias property
Specifies the alias of read-only cursor used as a target for the refresh operation.
The CursorAdapter class has a FetchMemo Property, which when set to False (.F.) in Visual FoxPro 9.0 places the cursor in Delayed Memo Fetch mode similar to Remote Views. Delayed Memo Fetch Mode prevents the contents of Memo fields from being fetched using CursorFill Method or CursorRefresh Method. An attempt to fetch content for a Memo field is done when the application attempts to access the value. The following CursorAdapter enhancements provide support for Delayed Memo Fetch:
Member
Description
DelayedMemoFetch method
Performs a delayed Memo field fetch for a target record in a cursor in a CursorAdapter object.
FetchMemoDataSourceType property
Specifies the data source type used for the DelayedMemoFetch method.
FetchMemoDataSource property
Specifies the data source used for the DelayedMemoFetch method.
FetchMemoCmdList property
Specifies a list of Memo field names and their associated fetch commands.
You can use the following table to determine the node within the XML document that you want to start reading.
For example, if you use an XPath expression in the XMLName property for an XMLAdapter, reading begins at the first node
To read
Class
Context node
From the first found XML node:
XMLAdapter
IXMLDOMElement property
All found XML nodes and use each node as a single record:
XMLTable
XMLAdapter object
The first found XML node and use its text as a field value:
XMLField
XMLTable object
The following methods do not support the use of XPath expressions in the XMLName property:
The ApplyDiffgram and ChangesToCursor methods do not support XPath expressions for XMLAdapter and XMLTable objects.
The ToCursor method does not support an XPath expression for XMLAdapter when the IsDiffgram property is set to True (.T.).
The ToXML method does not support XPath expressions for XMLAdapter and XMLTable objects and ignores XMLField objects that use XPath expressions.
For more information about XPath expressions, see the XPath Reference in the Microsoft Core XML Services (MSXML) 4.0 SDK in the MSDN library at http://msdn.microsoft.com/library.
When used in the OLE DB Provider for Visual FoxPro, the _VFP VFPXMLProg property is not supported for the CURSORTOXML( ), XMLTOCURSOR( ) and XMLUPDATEGRAM( ) functions because the _VFP system variable is not supported in the OLE DB Provider.
EXECSCRIPT Supported in the Visual FoxPro OLE DB Provider
The SELECT – SQL Command and other SQL commands have been substantially enhanced in Visual FoxPro 9.0. This topic describes the enhancements made to these commands, and new commands that affect SQL performance.
Expanded Capacities
Several SELECT – SQL command limitations have been removed or increased in Visual FoxPro 9.0. The following table lists the areas where limitations have been removed or increased.
Capacity
Description
Number of Joins and Subqueries in a SELECT – SQL command
Visual FoxPro 9.0 removes the limit on the total number of join clauses and subqueries in a SELECT – SQL command. The previous limit was nine.
Number of UNION clauses in a SELECT – SQL command
Visual FoxPro 9.0 removes the limit on number of UNION clauses in a SQL SELECT statement. The previous limit was nine.
Number of tables referenced a SELECT – SQL command
Visual FoxPro 9.0 removes the limit on the number of tables and aliases referenced in a SQL SELECT statement. The previous limit was 30.
Number of arguments in an IN( ) clause
Visual FoxPro 9.0 removes the limit of 24 values in the IN (Value_Set) clause for the WHERE clause. However, the number of values remains subject to the setting of SYS(3055) – FOR and WHERE Clause Complexity. For functionality changes concerning the IN clause, see Changes in Functionality for the Current Release.
Subquery Enhancements
Visual FoxPro 9.0 provides more flexibility in subqueries. For example, multiple subqueries are now supported. The following describes the enhancements to subqueries in Visual FoxPro 9.0.
Multiple Subqueries
Visual FoxPro 9.0 supports multiple subquery nesting, with correlation allowed to the immediate parent. There is no limit to the nesting depth. In Visual FoxPro 8.0, error 1842 (SQL: Subquery nesting is too deep) was generated when more than one level of subquery nesting occurred.
The following is the general syntax for multiple subqueries.
SELECT … WHERE … (SELECT … WHERE … (SELECT …) …) …
Examples
The following example queries, which will generate an error in Visual FoxPro 8.0, are now supported in Visual FoxPro 9.0.
Copy Code
CREATE CURSOR MyCursor (field1 I) INSERT INTO MyCursor VALUES (0) CREATE CURSOR MyCursor1 (field1 I) INSERT INTO MyCursor1 VALUES (1) CREATE CURSOR MyCursor2 (field1 I) INSERT INTO MyCursor2 VALUES (2) SELECT * FROM MyCursor T1 WHERE EXISTS ; (SELECT * from MyCursor1 T2 WHERE NOT EXISTS ; (SELECT * FROM MyCursor2 T3)) *** Another multiple subquery nesting example *** SELECT * FROM table1 WHERE table1.iid IN ; (SELECT table2.itable1id FROM table2 WHERE table2.iID IN ; (SELECT table3.itable2id FROM table3 WHERE table3.cValue = “value”))
GROUP BY in a Correlated Subquery
Many queries can be evaluated by executing a subquery once and substituting the resulting value or values into the WHERE clause of the outer query. In queries that include a correlated subquery (also known as a repeating subquery), the subquery depends on the outer query for its values. This means that the subquery is executed repeatedly, once for each row that might be selected by the outer query.
Visual FoxPro 8.0 does not allow using GROUP BY in correlated subquery, and generates error 1828 (SQL: Illegal GROUP BY in subquery). Visual FoxPro 9.0 removes this limitation and supports GROUP BY for correlated subqueries allowed to return more than one record.
The following is the general syntax for the GROUP BY clause in a correlated subquery.
SELECT … WHERE … (SELECT … WHERE … GROUP BY …) …
Examples
The following example, which will generate an error in Visual FoxPro 8.0, is now supported in Visual FoxPro 9.0.
Copy Code
CLOSE DATABASES ALL CREATE CURSOR MyCursor1 (field1 I, field2 I, field3 I) INSERT INTO MyCursor1 VALUES(1,2,3) CREATE CURSOR MyCursor2 (field1 I, field2 I, field3 I) INSERT INTO MyCursor2 VALUES(1,2,3) SELECT * from MyCursor1 T1 WHERE field1; IN (SELECT MAX(field1) FROM MyCursor2 T2 ; WHERE T2.field2=T1.FIELD2 GROUP BY field3)
TOP N in a Non-Correlated Subquery
Visual FoxPro 9.0 supports the TOP N clause in a non-correlated subquery. The ORDER BY clause should be present if the TOP N clause is used, and this is the only case where it is allowed in subquery.
The following is the general syntax for the TOP N clause in a non-correlated subquery.
SELECT … WHERE … (SELECT TOP nExpr [PERCENT] … FROM … ORDER BY …) …
Examples
The following example, which will generate an error in Visual FoxPro 8.0, is now supported in Visual FoxPro 9.0.
Copy Code
CLOSE DATABASES ALL CREATE CURSOR MyCursor1 (field1 I, field2 I, field3 I) INSERT INTO MyCursor1 VALUES(1,2,3) CREATE CURSOR MyCursor2 (field1 I, field2 I, field3 I) INSERT INTO MyCursor2 VALUES(1,2,3) SELECT * FROM MyCursor1 WHERE field1 ; IN (SELECT TOP 5 field2 FROM MyCursor2 order by field2)
Subqueries in a SELECT List
Visual FoxPro 9.0 allows a subquery as a column or a part of expression in a projection. A subquery in a projection has exactly the same requirements as a subquery used in a comparison operation. If a subquery does not return any records, NULL value is returned.
In Visual FoxPro 8.0, an attempt to use a subquery as a column or a part of expression in a projection would generate error 1810 (SQL: Invalid use of subquery).
The following is the general syntax for a subquery in a SELECT list.
SELECT … (SELECT …) … FROM …
Example
The following example, which will generate an error in Visual FoxPro 8.0, is now supported in Visual FoxPro 9.0.
Copy Code
SELECT T1.field1, (SELECT field2 FROM MyCursor2 T2; WHERE T2.field1=T1.field1) FROM MyCursor1 T1
Aggregate functions in a SELECT List of a Subquery
In Visual FoxPro 9.0, aggregate functions are now supported in a SELECT list of a subquery compared using the comparison operators <, <=, >, >= followed by ALL, ANY, or SOME. See Considerations for SQL SELECT Statements for more information about aggregate functions.
Example
The following example demonstrates the use of an aggregate function (the COUNT( ) function) in a SELECT list of a subquery.
Copy Code
CLOSE DATABASES ALL CREATE CURSOR MyCursor (FIELD1 i) INSERT INTO MyCursor VALUES (6) INSERT INTO MyCursor VALUES (0) INSERT INTO MyCursor VALUES (1) INSERT INTO MyCursor VALUES (2) INSERT INTO MyCursor VALUES (3) INSERT INTO MyCursor VALUES (4) INSERT INTO MyCursor VALUES (5) INSERT INTO MyCursor VALUES (-1) CREATE CURSOR MyCursor2 (FIELD2 i) INSERT INTO MyCursor2 VALUES (1) INSERT INTO MyCursor2 VALUES (2) INSERT INTO MyCursor2 VALUES (2) INSERT INTO MyCursor2 VALUES (3) INSERT INTO MyCursor2 VALUES (3) INSERT INTO MyCursor2 VALUES (3) INSERT INTO MyCursor2 VALUES (4) INSERT INTO MyCursor2 VALUES (4) INSERT INTO MyCursor2 VALUES (4) INSERT INTO MyCursor2 VALUES (4) SELECT * FROM MYCURSOR WHERE field1; < ALL (SELECT count(*) FROM MyCursor2 GROUP BY field2) ; INTO CURSOR MyCursor3 BROWSE
Correlated Subqueries Allow Complex Expressions to be Compared with Correlated Field
In Visual FoxPro 8.0, correlated fields can only be referenced in the following forms:
correlated field <comparison> local field
-or-
local field <comparison> correlated field
In Visual FoxPro 9.0. correlated fields support comparison to local expressions, as shown in the following forms:
correlated field <comparison> local expression
-or-
local expression <comparison> correlated field
A local expression must use at least one local field and cannot reference any outer (correlated) field.
Example
In the following example, a local expression (MyCursor2.field2 / 2) is compared to a correlated field (MyCursor.field1).
Copy Code
SELECT * FROM MyCursor ; WHERE EXISTS(SELECT * FROM MyCursor2 ; WHERE MyCursor2.field2 / 2 > MyCursor.field1)
Changes for Expressions Compared with Subqueries.
In Visual FoxPro 8.0, the left part of a comparison using the comparison operators [NOT] IN, <, <=, =, ==, <>, !=, >=, >, ALL, ANY, or SOME with a subquery must reference one and only one table from the FROM clause. In case of a comparison with correlated subquery, the table must also be the correlated table.
In Visual FoxPro 9.0, comparisons work in the following ways:
The expression on the left side of an IN comparison must reference at least one table from the FROM clause.
The left part for the conditions =, ==, <>, != followed by ALL, SOME, or ANY must reference at least one table from the FROM clause.
The left part for the condition >, >=, <, <= followed by ALL, SOME, or ANY (SELECT TOP…) must reference at least one table from the FROM clause.
The left part for the condition >, >=, <, <= followed by ALL, SOME, or ANY (SELECT <aggregate function>…) must reference at least one table from the FROM clause.
The left part for the condition >, >=, <, <= followed by ALL, SOME, or ANY (subquery with GROUP BY and/or HAVING) must reference at least one table from the FROM clause.
In Visual FoxPro 9.0, the left part of a comparison that does not come from the list (for example, ALL, SOME, or ANY are not included) doesn’t have to reference any table from the FROM clause.
In all cases, the left part of the comparison is allowed to reference more than one table from the FROM clause. For a correlated subquery, the left part of the comparison does not have to reference the correlated table.
Subquery in an UPDATE – SQL Command SET List
In Visual FoxPro 9.0, the UPDATE – SQL Command now supports a subquery in the SET clause.
A subquery in a SET clause has exactly the same requirements as a subquery used in a comparison operation. If the subquery does not return any records, the NULL value is returned.
Only one subquery is allowed in a SET clause. If there is a subquery in the SET clause, subqueries in the WHERE clause are not allowed.
The following is the general syntax for a subquery in the SET clause.
UPDATE … SET … (SELECT …) …
Example
The following example demonstrates the use of a subquery in the SET clause.
Copy Code
CLOSE DATA CREATE CURSOR MyCursor1 (field1 I , field2 I NULL) INSERT INTO MyCursor1 VALUES (1,1) INSERT INTO MyCursor1 VALUES (2,2) INSERT INTO MyCursor1 VALUES (5,5) INSERT INTO MyCursor1 VALUES (6,6) INSERT INTO MyCursor1 VALUES (7,7) INSERT INTO MyCursor1 VALUES (8,8) INSERT INTO MyCursor1 VALUES (9,9) CREATE CURSOR MyCursor2 (field1 I , field2 I) INSERT INTO MyCursor2 VALUES (1,10) INSERT INTO MyCursor2 VALUES (2,20) INSERT INTO MyCursor2 VALUES (3,30) INSERT INTO MyCursor2 VALUES (4,40) INSERT INTO MyCursor2 VALUES (5,50) INSERT INTO MyCursor2 VALUES (6,60) INSERT INTO MyCursor2 VALUES (7,70) INSERT INTO MyCursor2 VALUES (8,80) UPDATE MyCursor1 SET field2=100+(SELECT field2 FROM MyCursor2 ; WHERE MyCursor2.field1=MyCursor1.field1) WHERE field1>5 SELECT MyCursor1 LIST
Sub-SELECT in the FROM Clause
A sub-SELECT is often referred to as a derived table. Derived tables are SELECT statements in the FROM clause referred to by an alias or a user-specified name. The result set of the SELECT in the FROM clause creates a table used by the outer SELECT statement. Visual FoxPro 9.0 permits the use of a subquery in the FROM clause.
A sub-SELECT should be enclosed in parentheses and an alias is required. Correlation is not supported. A sub-SELECT has the same syntax limitations as the SELECT command, but not the subquery syntax limitations. All sub-SELECTs are executed before the top most SELECT is evaluated.
The following is the general syntax for a subquery in the FROM clause.
SELECT … FROM (SELECT …) [AS] Alias…
Example
The following example demonstrates the use of a subquery in the FROM clause.
Copy Code
SELECT * FROM (SELECT * FROM MyCursor T1; WHERE field1 = (SELECT T2.field2 FROM MyCursor1 T2; WHERE T2.field1=T1.field2); UNION SELECT * FROM MyCursor2; ORDER BY 2 desc) AS subquery *** Note that the following code will generate an error *** SELECT * FROM (SELECT TOP 5 field1 FROM MyCursor) ORDER BY field1
ORDER BY with Field Names in the UNION clause
When using a UNION clause in Visual FoxPro 8.0, you are forced to use numeric references in the ORDER BY clause. In Visual FoxPro 9.0, this restriction has been removed and you can use field names.
The referenced fields must be present in the SELECT list (projection) for the last SELECT in the UNION; that projection is used for ORDER BY operation.
Example
The following example demonstrates the use of a field names in the ORDER BY clause.
Copy Code
CLOSE DATABASES all CREATE CURSOR MyCursor(field1 I,field2 I) INSERT INTO MyCursor values(1,6) INSERT INTO MyCursor values(2,5) INSERT INTO MyCursor values(3,4) SELECT field1, field2, .T. AS FLAG,1 FROM MyCursor; WHERE field1=1; UNION ; SELECT field1, field2, .T. AS FLAG,1 FROM MyCursor; WHERE field1=3; ORDER BY field2 ; INTO CURSOR TEMP READWRITE BROWSE NOWAIT
Optimized TOP N Performance
In Visual FoxPro 8.0 and earlier versions, when using the TOP N [PERCENT] clause all records are sorted and then the TOP N are extracted. In Visual FoxPro 9.0, performance has been improved by eliminating records that do not qualify for the TOP N from the sort process as early as possible.
Optimization requires that TOP N return no more than N records (this is not the case for Visual FoxPro 8.0 and earlier versions) which is enforced if SET ENGINEBEHAVIOR is set to 90.
TOP N PERCENT cannot be optimized unless the whole result set can be read into memory at once.
Improved Optimization for Multiple Table OR Conditions
Visual FoxPro 9.0 provides for improved Rushmore optimization involving multi-table OR conditions. Visual FoxPro uses multi-table OR conditions to Rushmore optimize filter conditions for a table as long as both sides of the condition can be optimized. The following example shows this:
Copy Code
CLEAR CREATE CURSOR Test1 (f1 I) FOR i=1 TO 20 INSERT INTO Test1 VALUES (I) NEXT INDEX ON f1 TAG f1 CREATE CURSOR Test2 (f2 I) FOR i=1 TO 20 INSERT INTO Test2 VALUES (I) NEXT INDEX ON f2 TAG f2 SYS(3054,12) SELECT * from Test1, Test2 WHERE (f1 IN (1,2,3) AND f2 IN (17,18,19)) OR ; (f2 IN (1,2,3) AND f1 IN (17,18,19)) INTO CURSOR Result SYS(3054,0)
In this scenario, table Test1 can be Rushmore optimized using the following condition:
(f1 IN (1,2,3) OR f1 IN (17,18,19))and table Test2 with the following:
(f2 IN (17,18,19) OR f2 IN (1,2,3))
Support for Local Buffered Data
At times it can be beneficial to use SELECT – SQL to select records from a local buffered cursor in which the table has not been updated. Many times when creating controls like grids, list boxes, and combo boxes it is necessary to consider newly added records which have not yet been committed to disk. Currently, SQL statements are based on content that is already committed to disk.
Visual FoxPro 9.0 provides language enhancements that allow you to specify if the data returned by a SELECT – SQL command is based on buffered data or data written directly to disk.
The SELECT – SQL command now supports a WITH … BUFFERING clause that lets you specify if retrieved data is based on buffered data or data written directly to disk. For more information, see SELECT – SQL Command – WITH Clause.
If you do not include the BUFFERING clause, the retrieved data is then determined by the setting for SET SQLBUFFERING command. For more information, see the SET SQLBUFFERING Command.
In Visual FoxPro 9.0, a UNION clause is now supported in the INSERT – SQL Command.
The following is the general syntax for the UNION clause.
INSERT INTO … SELECT … FROM … [UNION SELECT … [UNION …]]
Example
The following example demonstrates the use of a UNION clause in INSERT-SQL.
Copy Code
CLOSE DATABASES ALL CREATE CURSOR MyCursor (field1 I,field2 I) CREATE CURSOR MyCursor1 (field1 I,field2 I) CREATE CURSOR MyCursor2 (field1 I,field2 I) INSERT INTO MyCursor1 VALUES (1,1) INSERT INTO MyCursor2 VALUES (2,2) INSERT INTO MyCursor SELECT * FROM MyCursor1 UNION SELECT * FROM MyCursor2 SELECT MyCursor LIST
Correlated UPDATE – SQL Commands
Visual FoxPro 9.0 now supports correlated updates with the UPDATE – SQL Command.
If a FROM clause is included in the UPDATE -SQL command, then the name after UPDATE keyword defines the target for the update operation. This name can be a table name, an alias, or a file name. The following logic is used to select the target table:
If the name matches an implicit or explicit alias for a table in the FROM clause, then the table is used as a target for the update operation.
If the name matches the alias for the cursor in the current data session, then the cursor is used as a target.
A table or file with the same name is used as a target.
The UPDATE -SQL command FROM clause has the same syntax as the FROM clause in the SELECT – SQL command with the following limitations:
The target table or cursor cannot be involved in an OUTER JOIN as a secondary table.
The target cursor cannot be a subquery result.
All other JOINs can be evaluated before joining the target table.
The following is the general syntax for a correlated UPDATE command.
UPDATE … SET … FROM … WHERE …
Example
The following example demonstrates a correlated update using the UPDATE -SQL command.
Copy Code
CLOSE DATABASES ALL CREATE CURSOR MyCursor1 (field1 I , field2 I NULL,field3 I NULL) INSERT INTO MyCursor1 VALUES (1,1,0) INSERT INTO MyCursor1 VALUES (2,2,0) INSERT INTO MyCursor1 VALUES (5,5,0) INSERT INTO MyCursor1 VALUES (6,6,0) INSERT INTO MyCursor1 VALUES (7,7,0) INSERT INTO MyCursor1 VALUES (8,8,0) INSERT INTO MyCursor1 VALUES (9,9,0) CREATE CURSOR MyCursor2 (field1 I , field2 I) INSERT INTO MyCursor2 VALUES (1,10) INSERT INTO MyCursor2 VALUES (2,20) INSERT INTO MyCursor2 VALUES (3,30) INSERT INTO MyCursor2 VALUES (4,40) INSERT INTO MyCursor2 VALUES (5,50) INSERT INTO MyCursor2 VALUES (6,60) INSERT INTO MyCursor2 VALUES (7,70) INSERT INTO MyCursor2 VALUES (8,80) CREATE CURSOR MyCursor3 (field1 I , field2 I) INSERT INTO MyCursor3 VALUES (6,600) INSERT INTO MyCursor3 VALUES (7,700) UPDATE MyCursor1 SET MyCursor1.field2=MyCursor2.field2, field3=MyCursor2.field2*10 FROM MyCursor2 ; WHERE MyCursor1.field1>5 AND MyCursor2.field1=MyCursor1.field1 SELECT MyCursor1 LIST UPDATE MyCursor1 SET MyCursor1.field2=MyCursor3.field2 FROM MyCursor2, MyCursor3 ; WHERE MyCursor1.field1>5 AND MyCursor2.field1=MyCursor1.field1 AND MyCursor2.field1=MyCursor3.field1 SELECT MyCursor1 LIST
Correlated DELETE – SQL Commands
Visual FoxPro 9.0 now supports correlated deletions with the DELETE – SQL Command.
If a FROM clause has more than one table, the name after the DELETE keyword is required and it defines the target for the delete operation. This name can be a table name, an alias or a file name. The following logic is used to select the target table:
If the name matches an implicit or explicit alias for a table in the FROM clause, then the table is used as a target for the update operation.
If the name matches the alias for the cursor in the current data session, then the cursor is used as a target.
A table or file with the same name is used as a target.
The DELETE -SQL command FROM clause has the same syntax as the FROM clause in the SELECT – SQL command with the following limitations:
The target table or cursor cannot be involved in an OUTER JOIN as a secondary table.
The target cursor cannot be a subquery result.
It should be possible to evaluate all other JOINs before joining the target table.
The following is the general syntax for a correlated DELETE command.
DELETE [alias] FROM alias1 [, alias2 … ] … WHERE …
Example
The following example demonstrates a correlated deletion using the DELETE -SQL command.
Copy Code
CLOSE DATABASES ALL CREATE CURSOR MyCursor1 (field1 I , field2 I NULL,field3 I NULL) INSERT INTO MyCursor1 VALUES (1,1,0) INSERT INTO MyCursor1 VALUES (2,2,0) INSERT INTO MyCursor1 VALUES (5,5,0) INSERT INTO MyCursor1 VALUES (6,6,0) INSERT INTO MyCursor1 VALUES (7,7,0) INSERT INTO MyCursor1 VALUES (8,8,0) INSERT INTO MyCursor1 VALUES (9,9,0) CREATE CURSOR MyCursor2 (field1 I , field2 I) INSERT INTO MyCursor2 VALUES (1,10) INSERT INTO MyCursor2 VALUES (2,20) INSERT INTO MyCursor2 VALUES (3,30) INSERT INTO MyCursor2 VALUES (4,40) INSERT INTO MyCursor2 VALUES (5,50) INSERT INTO MyCursor2 VALUES (6,60) INSERT INTO MyCursor2 VALUES (7,70) INSERT INTO MyCursor2 VALUES (8,80) CREATE CURSOR MyCursor3 (field1 I , field2 I) INSERT INTO MyCursor3 VALUES (6,600) INSERT INTO MyCursor3 VALUES (7,700) DELETE MyCursor1 FROM MyCursor2 ; WHERE MyCursor1.field1>5 AND MyCursor2.field1=MyCursor1.field1 SELECT MyCursor1 LIST RECALL ALL DELETE MyCursor1 FROM MyCursor2, MyCursor3 ; WHERE MyCursor1.field1>5 AND MyCursor2.field1=MyCursor1.field1 AND MyCursor2.field1=MyCursor3.field1 SELECT MyCursor1 LIST RECALL ALL DELETE FROM MyCursor1 WHERE MyCursor1.field1>5 SELECT MyCursor1 list RECALL ALL DELETE MyCursor1 from MyCursor1 WHERE MyCursor1.field1>5 RECALL ALL IN MyCursor1 DELETE T1 ; FROM MyCursor1 T1 JOIN MyCursor2 ON T1.field1>5 AND MyCursor2.field1=T1.field1, MyCursor3 ; WHERE MyCursor2.field1=MyCursor3.field1 RECALL ALL IN MyCursor1
Updatable Fields in UPDATE – SQL Command
The number of fields that can be updated with the UPDATE – SQL Command is no longer limited to 128 as in prior versions of Visual FoxPro. You are now limited to 255, which is the number of fields available in a table.
SET ENGINEBEHAVIOR
The SET ENGINEBEHAVIOR Command has a new Visual FoxPro 9.0 option, 90, that affects SELECT – SQL command behavior for the TOP N clause and aggregate functions. For additional information, see the SET ENGINEBEHAVIOR Command.
Data Type Conversion
Conversion between data types (for example, conversion between memo and character fields) has been improved in Visual FoxPro 9.0. This conversion improvement applies to the ALTER TABLE – SQL Command with the COLUMN option as well as structural changes made with the Table Designer.
By the setting the AutoComplete property, you determine the sort order for the entries. If you want more control over the list and where it is stored, you can use the AutoCompSource property to specify the table used to populate the automatic list. By default, the table is AUTOCOMP.DBF. You can use one table for each text box control or a single table can populate automatic lists for several text boxes.
If you use a single table, which is the default, the table uses values in the Source field for each entry to identify the text box control associated with the entry. By default, the Source field value is the name of the text box control. You can specify the Source field value using the AutoCompSource property of the text box. For example, you might want to make the same set of entries available to multiple Text box controls within the application such as address information. You can explicitly set the AutoCompTable and AutoCompSource properties for each of the controls to the same table and source field value. The same automatic list appears for each of them.
The text box control handles updating the auto-completion table for you based on the values actually entered in the text box. If you want to remove a value from the list, enter a string in a text box that matches the string you want to delete to filter the list for it. Select the entry in the list and press the DELETE key. The string remains in the table but no longer appears in the automatic list.
In the current release of Visual FoxPro, you will find enhanced functionality via new and enhanced commands and functions. Expand and collapse this topic to see information about miscellaneous language additions and improvements.
REPORT FORM Command Displays or prints out a report specified by a report definitions file. This command has been enhanced to support Report Listener objects.
SET REPORTBEHAVIOR Command Controls use of Report Preview and Report Output applications with the Visual FoxPro Report System.
Additionally, there are improvements to the following related Printing language elements:
SYS(1037) – Page Setup Dialog Box Displays Visual FoxPro default or report Page Setup dialog box or sets printer settings for the default printer in Visual FoxPro or for the report printer environment. In this version, a new nValue parameter is available.
APRINTERS( ) Function Returns a five-column array with the name of the printer, connected port, driver, comment, and location. The last three columns are available if the new optional parameter is passed.
GETFONT( ) Function Contains an additional setting to display only those fonts available on the current default printer and clarified values for the language script.
New Reporting functionality is described in more detail in separate Reporting topics. For more information, see Guide to Reporting Improvements.
SYS(3007) – specifies a FontCharSet for ToolTips. This is a global setting.
FONT Clause – the following table lists commands that support an optional FONT clause that allows for specification of a FontCharSet in the following format: FONT cFontName [, nFontSize [, nFontCharSet]] Command DEFINE MENU DEFINE POPUP DEFINE BAR DEFINE PAD DEFINE WINDOW MODIFY WINDOW BROWSE/EDIT/CHANGE ?/??
Editors – the Font Dialog Box that you can invoke with an editor window active by selecting the Font menu item from the Format menu or right-click shortcut menu Edit Properties Dialog Box now allows for selection of a font language script. You can specify a global default font script from the IDE Tab, Options Dialog Box in the Options Dialog Box (Visual FoxPro). To do this, you must first check the Use font script checkbox.
Interactive Development Environment (IDE) Enhancements
To provide a more integrated development environment for your projects and applications, Visual FoxPro contains the following improved functionality for the IDE.
Additional Project Manager Shortcut Menu Commands
When docked, the Project Manager window contains the following additional shortcut menu commands that are available on the Project menu:
Close Closes the Project Manager.
Add Project to Source Control Creates a new source control project based on the current project. Available only when a source code control provider is installed and specified on the Projects tab in the Options dialog box.
Errors Displays the error (.err) file after running a build.
Refresh Refreshes the contents of the Project Manager.
Clean Up Project Removes deleted records from the Project Manager (.PJX) file.
Modifying a Class Library from the Project Manager
You can change the text font settings for the Project Manager window. Right-click the Project Manager window (outside of the tree hierarchy window) and choose Font.
Generating Message Logs During Project Build and Compile
Design time support for entering property values greater than 255 characters and extended characters, such as CHR(13) (carriage return) and CHR(10) (linefeed), has been added to visual class library (.vcx) and form (.scx) files. You can now enter up to 8k characters in length. Note: Extended property value support is only available through the Properties Window (Zoom dialog box) for custom user-specified properties as well as certain native ones such as CursorSchema and Value. For properties not supported, you can still specify values which are longer than 255 characters, or contain carriage returns and linefeeds by assigning them in code such as during the object’s Init Event. The Zoom dialog box and Expression Builder dialog box have been updated to support this. The Properties window includes a Zoom (Z) button that appears next to the property settings box for appropriate properties. Caution: Property values that exceed 255 characters or include carriage return and/or linefeed characters are stored in a new format inside the .vcx or .scx file. If you attempt to modify these classes in a prior version, an error occurs. This feature is particularly useful for setting the CursorAdapter CursorSchema property to any schema expression when schemas might exceed 255 characters.
The Properties window font can now be specified by the new Font shortcut menu option. This new menu replaces the Small, Medium and Large font menu items used in prior versions. This font is also used in the description pane, and object and property value dropdowns. Note: Bold and italic font styles are reserved for non-default property values and read-only properties, respectively. If a bold or italic font style is chosen, then the Properties window inverts the displayed behavior. For example, if one chooses an italic font style, read-only properties appear in normal font style and all others in italic.
Non-Default Properties Color Sets color for properties whose values have changed from default setting (same properties that are displayed when the Non-Default Properties Only menu item is selected).
Custom Properties Color Sets color for custom properties.
If a word is selected in a Visual FoxPro editor, the Find Dialog Box (Visual FoxPro) when opened now displays the word in the Look For drop-down box. If Find has not yet been used for a running instance of Visual FoxPro, a word positioned under the insertion pointer will appear in the Look For drop-down. If multiple words are selected, only the first word appears in the drop-down (use copy and paste to enter multiple words).
When a Browse window is open and you search for a word with the Find dialog box, you can search for the word again (Find Again) after the Find dialog box is closed by pressing the F3 key.
You can now use Find to search for content in Name column of the Watch and Locals debug windows (see Debugger Window). When searching object members, Find searches in these debug windows are limited to nodes that have been expanded and one level below.
View Constants in Trace Window
Constants (#DEFINE values) can be viewed in the Trace Window when you hover over it with the mouse.
Note:
Visual FoxPro evaluates constants as expressions in the Trace Window and may have difficulty interpreting a specific #DEFINE when you hover over it with the mouse. Consequently, if there are multiple expressions on a line, they are all displayed in the value tip.
FOR EACH ObjectName [AS Type [OF ClassLibrary]] IN Group
Commands
[EXIT]
[LOOP]
ENDFOR
The Type parameter can be any valid type, including data types, class types, or ProgID. If the class name cannot be found, Visual FoxPro disregards Type and does not display IntelliSense for it.
Note:
The type reference does not affect the functionality of the application at run time. The type reference is only used for IntelliSense.
The ObjectName expression can refer to a memory variable or an array.
The ClassLibrary parameter must be in a path list that is visible to Visual FoxPro. You must specify a valid class library; references to existing objects are not valid. If Visual FoxPro cannot find the specified class library, IntelliSense does not display.
Types expressed as ProgIDs and class libraries do not require quotation marks (“”) to enclose them unless their names contain spaces.
When a user types the AS keyword, IntelliSense displays a list of types registered in the FoxCode.dbf table with Type “T”. If you have specified a valid type, typing a period within a WITH … ENDWITH or a FOR EACH … ENDFOR command displays IntelliSense for that object reference.
Visual FoxPro supports IntelliSense for nested WITH … ENDWITH and FOR EACH … ENDFOR commands. The following is an example of nested WITH … ENDWITH commands in a class defined in a program (.prg) file named Program1.prg. To use, paste this code into a new program named Program1.prg, save it and then type a period (.) inside the WITH … ENDWITH block.
Copy Code
DEFINE CLASS f1 AS form MyVar1 = 123 ADD OBJECT t1 AS mytext PROCEDURE Init WITH THIS AS f1 OF program1.prg WITH .t1 AS mytext OF program1.prg ENDWITH ENDWITH ENDPROC ENDDEFINE DEFINE CLASS mytext as textbox MyVar2 = 123 ENDDEFINE
IntelliSense provides limited List Values functionality for selected properties that begin with a “T” or “F” within a WITH … ENDWITH or FOR EACH … ENDFOR command. This is done to avoid possible conflicts with the common property values True (.T.) and False (.F.). If you just type “.T” or “.F” and press Enter, the word selected in the List Value drop-down does not expand. You need to type at least two letters for IntelliSense to insert the selected word.
GETFONT( ) Function Contains an additional setting to display only those fonts available on the current default printer and clarified values for the language script.
Visual FoxPro includes and updates its product error reporting to support Dr. Watson 2.0. This version includes new and improved error reporting, logging, and auditing features. For example, errors are logged while offline and are posted when you reconnect.
A sample custom property editor, Anchoreditor.app, is included in Visual FoxPro 9.0 and is located in the Wizards directory. This application is run when the Anchor property is double-clicked in the Properties window, or by choosing the Anchor property in the Properties window and clicking the ellipsis button (…).
Term
Definition
Anchor but do not resize vertically
Specifies that the center of the control is anchored to the top and bottom edges of its container but the control does not resize.
Anchor but do not resize horizontally
Specifies that the center of the control is anchored to the left and right edges of its container but the control does not resize.
Border values
Displays the current settings for the border values.
Common settings
Selects commonly used settings for the Anchor property.
Sample
Click the Sample button to test the current anchor value on a sample form.
Anchor value
The Anchor property value that is the combination of the current settings for the border values.
For the results grid, the Options dialog provides a new setting to show separate columns for class, method, and line, rather than concatenating them all in a single column.
You can now sort by method name by right-clicking on the method header or selecting the Sort By menu item from the right-click menu.
With the results tree list, the following new right-click menu options are available:
Expand All – expands all nodes
Collapse All – collapses all nodes
Sort by Most Recent First – puts the most recent result sets at the top of the list rather than at the bottom
Note:
The results beneath a tree node are not filled until the node is expanded. This is done to increase performance if you have a large result sets.
GENDBC.PRG
The Gendbc.prg program which generates program used to recreate a database has been updated with following minor enhancements:
Support for new Varchar, Varbinary and Blob field types
Support for AllowSimultaneousFetch, RuleExpression, and RuleText properties for views
Field Mapping – you can set classes to use for when you drag and drop a field onto a form with each environment set. This is setting specified in the Field Mapping Tab, Options Dialog Box.
Resource File – the Environment Manager now supports setting of a Resource File. If one does not exist, the Environment Manager will optionally create it when the environment is set.
The Environment Manager now contains a new <default field mapping> environment set. This set is created the first time the Environment Manager is run so that the original default Options dialog settings for Field Mapping and Form Template Classes can be saved and restored later if desired.
The following are new FoxPro Foundation classes added to this version of Visual FoxPro:
_REPORTLISTENER.VCX – a set of core classes you can use when creating custom report listeners.
_FRXCURSOR.VCX – a class library used for working with report (FRX) files.
_GDIPLUS.VCX – a set of classes you can use for GDI+ handling. This is intended primarily for use when creating custom report listener classes.
New Solution Samples
Visual FoxPro 9.0 contains many new samples that show off new features in the product. To see a list of these samples, select the Solution Samples task pane in the Task Pane Manager and expand the New in Visual FoxPro 9.0 node.
Visual FoxPro includes functionality that differs from previous versions and might affect existing code. These behavior changes are organized according to the following categories:
Critical Changes Functionality changes most likely to affect existing code when running under this version of Visual FoxPro. It is extremely important that you read this section.
Important Changes Functionality changes that might affect existing code when running under this version of Visual FoxPro.
Miscellaneous Changes Functionality changes you should know about but are not likely to impact existing code.
Removed Items Features or files that existed in prior versions of Visual FoxPro but are no longer included.
Conversion of INLIST( ) Function in the Query Designer and View Designer
In previous versions of Visual FoxPro, the Query Designer and View Designer convert INLIST( ) function calls in the WHERE clause of the SQL SELECT command into IN (Value_Set) clauses. In the current release, this conversion no longer occurs due to the differences between INLIST( ) and the SQL IN clause. INLIST( ) remains restricted to 24 arguments. For more information, see the description for the IN clause in the SELECT – SQL Command topic and the INLIST( ) Function.
Grids and RecordSource and ControlSource Properties
In Visual FoxPro 9.0 there is a change in Grid control behavior. When the RecordSource property for a Grid control is set, Visual FoxPro 9.0 resets all ControlSource properties to the empty string (“”) for all columns. In earlier versions of Visual FoxPro, the ControlSource properties were not properly reset, so problems could occur when a RecordSource with a different structure was later bound. This change may impact scenarios involving Access and Assign methods or BINDEVENT( ) function calls made against a Grid column’s ControlSource property.
In Visual FoxPro 9’s new object-assisted reporting mode, report fields may need to be adjusted (widened) slightly. This is especially important for numeric data where a field that is not wide enough to display the entire number will show it instead as asterisks (*****). For more information about the changes to the Report System that required this change, and features of the GDI+ rendering engine and other changes related to it, see Using GDI+ in Reports. For migration strategy and recommendations, see Guide to Reporting Improvements.
For a table of additional, minor rendering differences between backward-compatible reporting mode and object-assisted reporting mode, see the table below. Rendering feature Behavior in backward-compatible mode Behavior in object-assisted mode Recommendations Tab stops (CHR(9) values included in report data) The width of a tab stop is determined by a number of characters in the font used. Tab stops are set at fixed-width positions, regardless of font. If you concatenated tabs with data in a stretching report layout element to create a table format within the element, you can often fulfill the same requirements using a second detail band in Visual FoxPro 9. Alternatively, change the number of tabs you concatenate with your data. Special characters and word-wrapping Non-breaking spaces are not respected; they are treated as normal space characters. Special characters such as non-breaking spaces (CHR(160)) and soft hyphens (CHR(173)) are correctly interpreted. As a result, words may wrap differently in output. Evaluate the results. In most cases, users will appreciate the change, because it more faithfully representing their original intentions in the text. If necessary, use the CHRTRAN( ) Function or STRTRAN( ) Function to replace these special characters with standard spaces and hyphens. Line spacing of multi-line objects Line spacing is determined by a formula that does not take font properties into consideration. Lines in a multi-line object are individually rendered, so background colors for each line may appear to have a different width. GDI+ line spacing is dynamically determined using font characteristics. A multi-line object is rendered as a single block of text. Evaluate the results. In most cases, the change in line spaces will provide a more polished appearance, and in all cases this method of handling multi-line text provides better performance. If a report depends on the old style of spacing lines, you can adjust the ReportListener’s DynamicLineHeight Property to revert to the old behavior. Cursor images (.CUR files) .CUR files can be used as image sources in reports. .CUR files are not supported as image sources. Convert the cursor file to another, supported image format. Shape (Rounded Rectangle) curvature Limited choices for curvature. More choices are available through the Report Builder Application dialog box interface, but some will not look the same way in backward-compatible mode and object-assisted mode. If reports have to run in both backward-compatible mode and object-assisted mode, or if they are designed in version 9.0 but must run in earlier versions, limit your choices of values of shape curvature to those allowed in the native Round Rectangle Dialog Box. If you are using the Style Tab, Report Control Properties Dialog Box (Report Builder) to design such reports, use the values 12, 16, 24, 32, and 99, to represent the native buttons, selecting the buttons from left to right. The default value in the Round Rectangle dialog box (second button) is 16.
When you create a Quick Report, by using the CREATE REPORT – Quick Report Command or by invoking the Quick Report… option on the Report menu, and if you have SET REPORTBEHAVIOR 90, the layout elements created by the Report Designer are sized differently from ones created for the same fields in previous versions. This change handles the additional width required by the new rendering mechanism of the report engine.
If you use the KEYBOARD Command or PLAY MACRO Command statements to address options on the Report menu, you may need to revise the keystrokes in these statements, as the menu has been reorganized.
Reports may take longer to open in the Report Designer if the report was previously saved with the Printer Environment setting enabled. You can improve performance by unchecking the Printer Environment menu item from the Report menu and re-saving the report. The saved Printer Environment is not critical for functioning of a report and is typically not recommended. Object-assisted report mode also respects different printers’ resolution settings, so saving resolution information for one printer in your report may have adverse effects in an environment with printers that have different resolutions. A saved Printer Environment may also have more adverse affects on REPORT FORM or LABEL commands invoked with the TO FILE option than they did in previous versions, if the associated printer setup is not available in the environment at runtime. In Visual FoxPro 9, the global default for this setting in the Reports Tab, Options Dialog Box, and for reports created in executable applications (.exe files), has been changed to unchecked.
Because of changes to the way Visual FoxPro 9 uses current printer settings to determine items such as print resolution and page height dynamically, a REPORT FORM or LABEL command will not run in object-assisted mode if there are no available printer setups in the environment or if the print spooler has been stopped. You will receive Error loading printer driver (Error 1958). If you need to run reports in an environment with no printer information, perhaps creating custom types of output that do not require printers, you can supply Visual FoxPro with the minimal set of information it needs to run your report by supplying a page height and page width from the appropriate Report Listener methods. For more information, see GetPageHeight Method and GetPageWidth Method.
By default, and by design, the Report Builder Application does not automatically show tables in the report’s Data Environment when you build report expressions. To better protect end-user design sessions, only tables you have explicitly opened, not all tables from the DataEnvironment, are available in the Expression Builder. With this change, you have the opportunity to set up the design session’s data exactly the way you want the end-user to see it, before you issue a MODIFY REPORT Command in your application. If you prefer the Report Designer’s old behavior, you can change the Report Builder Application to emulate it. For more information, see How to: Replace the Report Builder’s Expression Builder Dialog Box.
The ASCII keyword on the REPORT FORM Command is documented as following the <filename> parameter of the TO FILE <filename> clause. In earlier versions of Visual FoxPro, you could safely use the incorrect and unsupported syntax TO FILE ASCII <filename> instead. This incorrect syntax triggers an error in Visual FoxPro 9. Note that the ASCII keyword has no effect on object-assisted mode output provided by the Report Engine, although a ReportListener Object can be written to implement it.
The keyword NOCONSOLE has no default meaning in object-assisted reporting mode, because ReportListeners do not echo their rendering output to the current output window by default. However, a ReportListener can mimic backward-compatible mode in this respect, if desired. Refer to OutputPage Method for a complete example.
To facilitate development of run-time reporting extensions, the Report Engine now allows normal debugging procedures during a report run. If your error handling routine assumes it is impossible for a report to be suspended, this assumption will now be challenged. Refer to Handling Errors During Report Runs for a detailed look at the associated changes, and some suggestions for strategy.
REPORT FORM and LABEL commands are no longer automatically prohibited as user-interface-related commands in COM objects compiled into DLLs, when you run the commands in object-assisted mode. The restriction still applies to these commands when they are run in backward-compatible mode. (The topic Selecting Process Types explains why user-interface-related commands are prohibited in DLLs.) This change is not applicable to multi-threaded DLLs. A number of user-interface-related facilities also are not available in DLLs (whether single- or multi-threaded). For example, the TXTWIDTH( ) Function and TextWidth Method depend on a window handle to function, so they are not available in a DLL. The CREATE REPORT – Quick Report Command relies on the same facilities as TXTWIDTH(), and therefore is not available in a DLL. However, in many instances, creating custom output using a ReportListener does not require any user-interface activity, so a REPORT FORM or LABEL command can now be used productively in a DLL. Using the SYS(2335) – Unattended Server Mode function to trap for potential modal states, as well as the new SET TABLEPROMPT Command, is recommended. Refer to Server Design Considerations and Limitations for more information.
Changes have occurred to the handling of group headers and footers in multi-column reports, when the columns flow from left to right (label-style layout). The revised behavior is more useful and behaves consistently with the new detail header and footer bands as well. For a description of the change, see How to: Define Columns in Reports.
In previous versions, the NOWAIT keyword on the REPORT FORM and LABEL commands was not significant when the commands were issued in the Command window rather than in a program. In Visual FoxPro 9’s object-assisted mode, when previewing instructions are interpreted by the Report Preview Application, this keyword is significant no matter where you issue the command. The Report Preview Application uses the NOWAIT keyword, consistently, as an instruction to provide a modeless preview form. For more information about the Report Preview Application, see Extending Report Preview Functionality.
Visual FoxPro 8 introduced the NOPAGEEJECT keyword on the REPORT FORM and LABEL commands, but applied the keyword only to printed output. In Visual FoxPro 9, NOPAGEEJECT has significance for all output targets, including PREVIEW. This keyword provides chained or continued report runs for multiple REPORT FORM and LABEL commands. To facilitate this behavior in preview mode, and to allow you to apply customization instructions to multiple previews, the Report Output Application caches a single ReportListener object instance for preview output, causing a change in behavior for multiple modeless report commands (REPORT FORM … PREVIEW NOWAIT). In the past, you used multiple REPORT FORM… PREVIEW NOWAIT statements in a sequence, your commands resulted in multiple report preview windows. In Visual FoxPro 9, when SET REPORTBEHAVIOR 90, these commands will result in successive report previews being directed to a single report preview window. Tip: You can easily invoke the old behavior by creating multiple ReportListener object references and associating one with each separate REPORT FORM or LABEL command, using the OBJECT keyword. For more information about using the OBJECT syntax, see REPORT FORM Command. For information about receiving multiple object references of the appropriate type from the Report Output Application, see Understanding the Report Output Application.
In the process of reviewing and overhauling the native Report Engine, a number of outstanding issues regarding band and layout element positioning were addressed. For example, a field element marked to stretch and sized to take up more than one text line’s height in the report layout might have inappropriately pushed its band’s exit events to the next page in Visual FoxPro 8. In Visual FoxPro 9, the band’s exit events occur on the same page. Additional revisions improve record-pointer-handling in footer bands, when bands stretch across pages. These changes are not specific to object-assisted output rendering. If you have relied on undocumented behavior providing exact band or layout control placement in a particular report, you should review that report’s behavior in Visual FoxPro 9.
Rushmore Optimization
When character values are indexed, all values are considered to be encoded using the table’s code page. In previous versions of Visual FoxPro, when the current Visual FoxPro code page differed from a table’s code page, any attempt to look for a character value within that table’s index resulted in an implicit translation of the value from the current Visual FoxPro code page into the table’s code page. This could cause SQL or other Rushmore optimizable commands to return or act upon incorrect records.
In Visual FoxPro 9 and later, by default, the optimization engine no longer uses existing character indexes for tables created with a non-current code page. Instead, Visual FoxPro builds temporary indexes to ensure correct results. This can result in a loss of optimization of SQL or other commands which were optimized in earlier VFP versions. To prevent this, ensure that the current Visual FoxPro code page returned by CPCURRENT( ) Function matches the table’s code page returned by CPDBF( ) Function. This requires either changing the current Visual FoxPro code page, or changing the table’s code page. For information on specifying the current Visual FoxPro code page, see Understanding Code Pages in Visual FoxPro. For information on specifying the code page for a table, see How to: Specify the Code Page of a .dbf File. If you cannot change either the Visual FoxPro codepage or the table codepage to match, you can force optimization to work as it did in Visual FoxPro 8 and prior versions using the SET ENGINEBEHAVIOR Command with either 80 or 70 as a parameter.
SQL SELECT Statements
A SELECT – SQL Command containing DISTINCT and ORDER BY clauses in which the ORDER BY field is not in the SELECT field list will generate an error in Visual FoxPro 9.0 with SET ENGINEBEHAVIOR 90 (Error 1808: SQL: ORDER BY clause is invalid.). The following example shows this: Copy Code SET ENGINEBEHAVIOR 90 CREATE CURSOR foo (f1 int, f2 int) SELECT DISTINCT f1 FROM foo ORDER BY f2 INTO CURSOR res
A SELECT – SQL Command containing DISTINCT and HAVING clauses in which the HAVING field is not in the SELECT field list will now generate an error in Visual FoxPro 9.0 with SET ENGINEBEHAVIOR 90 (Error 1803: SQL: HAVING clause is invalid.). An error is reported because the HAVING field is not in projection and DISTINCT is used. The following example shows this: Copy Code SET ENGINEBEHAVIOR 90 CREATE CURSOR foo (f1 int, f2 int) SELECT DISTINCT f1 FROM foo HAVING f2>1 INTO CURSOR res
The number of UNION statements that can be used in a SELECT – SQL Command is no longer limited to 9. Parentheses are not completely supported with UNION statements and unlike previous versions may generate an error. If two or more SELECT statements are enclosed in parenthesis, an error is generated during compile (Error 2196: Only a single SQL SELECT statement can be enclosed in parentheses.). This behavior is not tied to any SET ENGINEBEHAVIOR Command level. The following example shows this error: Copy Code SELECT * FROM Table1 ; UNION ; (SELECT * FROM Table2 ; UNION ; SELECT * FROM Table3) The following example compiles without an error: Copy Code SELECT * FROM Table1 ; UNION ; (SELECT * FROM Table2) ; UNION ; (SELECT * FROM Table3)
An error (Error 2199) is now generated when index key truncation is about to occur, typically during index creation or modification. This can happen with use of a key that contains an expression involving a Memo field, whose length in not fixed, such as in the following example:
INDEX ON charfld1 + memofld1 TAG mytag
Similar issues can also occur with the SQL engine (such as during a SQL SELECT command or View creation) where it might fail to build a temporary index to optimize a join evaluation if it is unable to accurately determine the maximum size of the key.
Visual FoxPro will now detect if a Memo field in a class library (.vcx) or form (.scx) is corrupt when you try to open up that file in the designer. If the file contains a corrupt Memo field, an Error 41 such as following will occur:
Memo file <path>myclass.VCT is missing or is invalid.
Additionally, similar Memo errors may occur if you have a Visual FoxPro table open and try to access contents of a corrupt Memo. The following sample code shows how you can detect the Error 41 memo file corruption:
Copy Code
TRY USE myTable EXCLUSIVE NOUPDATE SCAN SCATTER MEMO MEMVAR ENDSCAN CATCH TO loError IF loError.ErrorNo=41 * handle error here ENDIF ENDTRY USE IN myTable
While it is possible that loss of data may occur, the following sample code may assist in repairing some or the entire file:
Copy Code
ON ERROR * USE myclass.vcx COPY TO myclass_bkup.vcx&&backup COPY TO myclass2.vcx USE DELETE FILE myclass.vc* RENAME myclass2.vcx TO myclass.vcx RENAME myclass2.vct TO myclass.vct COMPILE CLASSLIB myclass.vcx ON ERROR
Visual Form and Class Extended Property Support
Visual FoxPro 9 allows you to create custom properties in your visual class (SCX or VCX file) whose values can contain carriage returns and/or be of length greater than 255 characters. If you specify a property with a value like this through the Properties Window (i.e., the Zoom dialog box), Visual FoxPro will store it in a format such that you will no longer be able to edit that class in older versions of Visual FoxPro.
Class Definitions
The ability to have a property assignment set to instantiated object is no longer supported in a class definition and will generate an error. The following example shows this.
Copy Code
LOCAL oCustom oCustom = CREATEOBJECT(‘cusTest’) DEFINE CLASS cusTest AS CUSTOM oRef = CREATEOBJECT(‘myclass’) ENDDEFINE DEFINE CLASS myclass AS CUSTOM ENDDEFINE
You can instead assign a property to an instantiated object reference in the Init event of your class.
Merge Modules for Redistributable Components
Visual FoxPro includes merge modules (MSM files) for use in redistributing shared components with your runtime applications. Merge modules are used by applications that can create Windows Installer based setups. For example, Visual FoxPro ships with merge modules that contain the Visual FoxPro runtime libraries as well as some common components including a number of ActiveX controls.
For Visual FoxPro 9, the VFP9RUNTIME.MSM merge module contains the runtime libraries that you will need for your custom redistributable application. The VFP9RUNTIME.MSM merge module also has dependencies on the merge modules containing the Microsoft VC 7.1 runtime library (MSVCR71.DLL) and the GDI+ graphics library (GDIPLUS.DLL). Because of these dependencies, if you select the VFP9RUNTIME.MSM merge module in a Windows Installer tool such as InstallShield, the other dependent merge modules will automatically be selected as well.
Note For Windows XP and higher operating systems, Visual FoxPro uses the GDI+ graphics library that is installed in your Windows System folder.
For Visual FoxPro 9, the merge module containing the VC runtime library no longer installs to the Windows System directory. Instead, this file is installed to your application’s directory. This is done in compliance with recommended component versioning strategies for Windows operating systems. The GDI+ library is installed into the same directory as your Visual FoxPro runtime libraries and is only installed on operating systems later than Windows XP (XP already includes the GDI+ library in its Windows System directory).
Tip There may be circumstances where you will want to install the VC or GDI+ library to another location such as the Windows System directory. You can do this with your Windows Installer application (e.g., InstallShield) by first selecting the merge module before selecting the VFP9RUNTIME.MSM one. Once you have selected a merge module, you can change its installation path.
There are new merge modules for MSXML3 and MSXML4 XML parser components. The MSXML 3.0 component consists of the following merge modules:
There are two MSXML 4.0 modules that should be included with any custom setup:
MSXML 4.0 (msxml4sxs32.msm)
MSXML 4.0 (msxml4sys32.msm)
MTDLL Memory Allocation
Visual FoxPro contains a new PROGCACHE configuration file setting which specifies the amount of memory Visual FoxPro allocates at startup for running programs (program cache). This setting also determines memory allocated per thread for Visual FoxPro MTDLL COM Servers. In prior versions, this setting was not configurable and memory was allocated as a fixed program cache of a little over 9MB (144 * 64K). The new PROGCACHE setting allows you to set the exact size of the program cache or specify that dynamic memory allocation be used.
Since MTDLL COM Servers can use up a great amount of memory if many threads are created, it is important that memory be allocated more efficiently for these scenarios. In Visual FoxPro 9, the new default setting for MTDLL COM Servers is -2 (dynamic memory allocation). For more information, see Special Terms for Configuration Files.
The target record is now kept current in the ADODB.Recordset during CursorAdapter.After… events.
Grid SetFocus Supported for AllowCellSelection
You can now call a Grid control’s SetFocus Method and have the Grid receive focus when the AllowCellSelection Property is set to False (.F.) and the grid contains no records.
Additionally, Visual FoxPro 9.0 tightens syntax validation of calls made from concatenation of parameters. The following code, which worked in prior versions of Visual FoxPro, now properly causes an error because the CHR(13) character breaks the call into two lines whereas it is supposed to be part of the parameter for the EXECSCRIPT call.
In the current version of Visual FoxPro, the PageUp, PageDown, Home and End keyboard keys now cause a Listbox control’s Click event to fire. In previous versions, these keys did not trigger the Click event to fire, unlike the arrow keys.
PEMSTATUS( ) Function Returns False for Hidden Native Properties
In previous versions of Visual FoxPro, the PEMSTATUS( ) function returned True (.T.) for hidden native properties of Visual FoxPro base classes when specifying a value of 5 for nAttribute. In the current release, PEMSTATUS( ) returns False (.F.) for these hidden native properties. For more information, see PEMSTATUS( ) Function.
Changes to Options Dialog Box
In the Options dialog box, the List display count option has been moved from the Editor tab to the View tab. For more information, see View Tab, Options Dialog Box.
In previous versions of Visual FoxPro, you could output all the settings in the Options Dialog Box (Visual FoxPro) to the Command Window by pressing the SHIFT key when choosing the OK button to close the dialog. In the current release, these settings are now sent to the Debug Output Window. The Debug Output window must be opened in order for the settings to be directed there.
FOXRUN.PIF
The FOXRUN.PIF file that is used by the RUN | ! Command is no longer installed in the Visual FoxPro root directory. If Visual FoxPro detects the presence of a FOXRUN.PIF file during a RUN command, it will use COMMAND.COM to execute the specified RUN command. This may not be the desired SHELL program to use for a particular operating system, especially newer ones like Windows XP in which CMD.EXE is preferable.
The current behavior for a RUN command without the existence of a FOXRUN.PIF file is that the RUN command will use the SHELL program specified by the operating system COMSPEC environment variable. With Windows XP, you can view and edit this variable by right-clicking your computer desktop’s My Computer icon and selecting the Properties dialog box (Advanced tab).
The FOXRUN.PIF file is still available in the Tools directory if you need it for a particular reason.
The SCATTER command no longer allows for ambiguous use of both MEMVAR and NAME clauses in the same command. You can only include one of these clauses. In prior versions, the following code would not generate an error:
Copy Code
USE HOME()+”SAMPLESDatacustomer.dbf” SCATTER MEMVAR NAME oCust
The SET DOHISTORY command, which is included for backward compatibility, was updated to send output to the Debug Output Window instead of the Command Window as in prior versions.
SCREEN ShowTips Property
The default value for _SCREEN ShowTips Property has been changed from False (.F.) to True (.T.). This change was made because new Memo and Field Tips support is now dependent on this setting.
AllowCellSelection Does Not Permit Deleting Grid Rows When Set to False
The sample Northwind database has been updated. Five of the stored procedures now include calls to the SETRESULTSET( ) Function so that the Visual FoxPro OLE DB Provider will return a cursor when they are executed.
Foundation Classes
The _ShellExecute class contained in the _Environ.vcx FFC class library has been updated to include an additional parameter in the ShellExecute method.
Wizards and Builders
The Wizard/Builder selection dialog box now properly hides deleted entries in the Wizard and Builder registration tables.
Specifying Western Language Script Values for GETFONT( ) Function
In versions prior to this release, passing 0 as the nFontCharSet value for GETFONT( ) opened the Font Picker dialog box and displayed the Script list as unavailable. You could not specify 0 (Western) as the language script value, and setting it to 1 (Default) sets nFontCharSet to the default font setting only, which is determined by the operating system.
In this release, passing 0 to GETFONT( ) opens the Font Picker dialog box with the Script list available and Western selected. The return value for GETFONT( ) also includes the return value for nFontCharSet.
Removed Items
HTML Help SDK
The HTML Help 1.3 SDK no longer ships with Visual FoxPro.
Visual FoxPro has the following minimum system requirements:
Computer: PC with a Pentium class processor.
Peripherals: Mouse or pointing device
Memory: 64 MB RAM (128 MB or higher recommended)
Hard disk space:
Visual FoxPro Prerequisites: 20 MB
Visual FoxPro Typical Install: 165 MB
Visual FoxPro Maximum Install: 165 MB
Video: 800 x 600 resolution, 256 colors (High color 16-bit recommended)
Operating system: Developing applications with Visual FoxPro is supported only on Microsoft Windows 2000 Service Pack 3 or later, Windows XP and Windows Server 2003. You can create and distribute run-time applications for Windows 98, Windows Me, Windows 2000 Service Pack 3 or later, Windows XP and Windows Server 2003. Note: Installation on Windows NT 4.0 Terminal Server Edition is not supported.
You can install this version of Visual FoxPro from a CD-ROM or a network to a local hard drive. You must install Visual FoxPro on a local drive, not a mapped drive. There is no other preparation required before installing Visual FoxPro. You must have administrator privileges to install Visual FoxPro. It is recommended that you run with power-user privileges to use all the provided tools effectively.
You can safely install or uninstall using Visual FoxPro Setup. If you are upgrading Visual FoxPro, you must first uninstall the previous version of of the program. Though both versions of Visual FoxPro can exist on the same computer, you cannot install the current version of Visual FoxPro in the same directory as the previous version.
If you plan to publish XML Web services using Visual FoxPro, you might want to set up Internet Information Services (IIS) on a Windows 2000, Windows XP or Windows Server 2003 computer. Refer to your operating system documentation for instructions on how to set up and configure IIS.
Note:
Visual FoxPro setup no longer installs any Windows operating system Service Packs or versions of Internet Explorer. It is highly recommended that you install the latest versions of these components before installing Visual FoxPro. Additionally, Visual FoxPro 9.0 is supported only on Windows 2000 Service Pack 3 or later. For details about installing the latest Service Pack, visit the following Microsoft Web page at http://www.microsoft.com/windows2000/.
Full installation includes all Visual FoxPro program files, online help, and samples files.
To install Visual FoxPro
Quit all open applications. Note: If you use a virus protection program on your computer, override it or turn it off before running the Installation wizard. The Installation wizard might not run properly with virus protection turned on. After installation, be sure to restart your virus protection program.
Insert the Visual FoxPro CD. The Visual FoxPro Setup start page appears automatically.
Click Install Visual FoxPro to launch Visual FoxPro Setup.
To determine if you need additional components, click Prerequisites to display any necessary components.
Click Install Now! to install any new components. If Visual FoxPro Prerequisites needs to only update components, click Update Now!
You might need to restart your computer. When finished, click Done. Visual FoxPro Setup reappears.
To continue installation, click Visual FoxPro.
After accepting the End User License Agreement and entering the Product Key and your name, click Continue. Note: Visual FoxPro cannot be installed on a mapped drive. You must install Visual FoxPro on a local drive. Do not attempt to use the Map Network Drive functionality in Setup.
On the Options page, select the features you want to install and click Install Now! to continue.
When finished, click Done to return to Visual FoxPro Setup. Click Exit to return to the Visual FoxPro Setup start page.
If you uninstall Visual FoxPro while the previous version of Visual FoxPro exists on your computer, certain shared registry keys used by the previous version of Visual FoxPro are removed. You must reinstall these critical shared registry keys.
If you run Visual FoxPro from the Start menu, Visual FoxPro Setup automatically reinstalls these keys. If you start Visual FoxPro using other means, such as running the application executable directly, the setup program does not start automatically. You should use Add/Remove Programs in the Control Panel and the following steps to reinstall the registry keys manually:
To manually reinstall Visual FoxPro 9.0 registry keys
From the Start menu, click Control Panel.
Click Add/Remove Programs.
Click Change/Remove for Microsoft Visual FoxPro 9.0.
This release includes copies of additional software that you can install and use with Visual FoxPro. These include:
InstallShield Express Limited Edition Provides the capability to package and deploy the applications that you create using Visual FoxPro. Visual FoxPro includes the InstallShield Express 5.0 Visual FoxPro Limited Edition. Note: The limited and full editions of InstallShield Express 5.0 are considered two versions of the same product and cannot coexist. If you install one version on a computer where another already exists, the original is uninstalled automatically. Because the limited edition contains fewer features than the full edition, you should keep the full edition on your computer.
Microsoft SOAP Toolkit 3.0 Samples Provides samples for demonstrating how to consume and publish XML Web services. Visual FoxPro Prerequisites installs the core SOAP Toolkit 3.0 components needed to access and publish XML Web services in Visual FoxPro.
Microsoft SQL Server 2000 Desktop Engine (MSDE) Provides a personal version of SQL Server.
To install InstallShield Express Limited Edition
Insert the Visual FoxPro CD. The Visual FoxPro Setup start page opens automatically.
Click Install InstallShield Express.
Follow the instructions in the InstallShield Express installation wizard.
You can also locate the Setup.exe file for InstallShield Express in the InstallShield folder on the Visual FoxPro CD.
Note:
Visual FoxPro 9.0 installs its redistributable merge modules in the same location as Visual FoxPro 8.0.
The version of InstallShield Express included with Visual FoxPro 9.0 automatically uses the Visual FoxPro 9.0 merge module location.
Note:
Visual FoxPro 9.0 requires certain merge modules when creating a Visual FoxPro 9.0 redistributable custom application setup program using InstallShield Express.
You need to include the following merge modules when creating your custom setup program:
Microsoft Visual FoxPro 9 Runtime Libraries
Microsoft Visual C Runtime Library 7.1
GDI Plus Redist
MSXML 4.0
MSXML 3.0 (needed only for CURSORTOXML functions)
Microsoft Visual FoxPro 9 Runtime Language Libraries (specific language library files that may be needed for international applications)
Reporting Applications (needed for Visual FoxPro 9.0 reporting engine)
Note:
MSXML 4.0 consists of two merge modules (msxml4sxs32.msm and msxml4sys32.msm). MSXML 3.0 consists of three merge modules (msxml3_wim32.msm, msxml3inf_wim32.msm and wdstddll_wim32.msm).
To install SOAP Toolkit 3.0 Samples
Insert the Visual FoxPro CD. The Visual FoxPro Setup start page opens automatically.
Click Install SOAP Toolkit 3.0 Samples.
Follow the instructions in the SOAP Toolkit 3.0 Samples Setup Wizard.
You can also locate the Soapsdk.msi and Soapsamp.msi files for the SOAP Toolkit in the SOAPToolkit folder on the Visual FoxPro CD.
To install MSDE
Insert the Visual FoxPro CD. The Visual FoxPro Setup start page opens automatically.
Click Install Microsoft SQL Server Desktop Engine (MSDE) and follow the installation instructions that appear in the Readme file.
You can locate the Setup.exe file for MSDE in the SQLMSDE folder on the Visual FoxPro CD.
Note:
Visual FoxPro includes Microsoft SQL Server 2000 Desktop Engine Service Pack 3.0a. To make sure you have the most recent version and Service Pack installed, visit the Microsoft SQL Server Web page at http://www.microsoft.com/sql. In addition, if you are distributing custom Visual FoxPro applications that require MSDE, you can obtain the redistributable merge modules from the Microsoft SQL Server Web page for use with Windows Installer-based setup programs.
You can reinstall Visual FoxPro by uninstalling it and then installing it again. You can uninstall Visual FoxPro from the Start menu or from the original installation disk.
To uninstall Visual FoxPro
On the Start menu, click Control Panel.
In the Control Panel window, double-click Add or Remove Programs. The Add or Remove Programs window opens.
In the Currently installed programs list, click the version of Microsoft Visual FoxPro you want to uninstall, and then Change/Remove.
If you reinstall Visual FoxPro or reinstall to another location, you might want to clean your user settings and other files installed by Visual FoxPro before reinstalling.
You can remove these files by deleting the contents of the …Application DataMicrosoftVisual FoxPro folder inside your user settings folder. To determine the location of the Application Data folder, type ? HOME(7) in the Command window. These files include your FoxUser.* resource files, which contain user settings, and folders for the Toolbox and Task Pane.
However, it is possible that your resource files are in another location. You can determine their location by typing the following in the Command window:
Copy Code
? SYS(2005)
You should delete old Code Reference files that might be associated with projects in the project directories. These are labeled as projectname_ref.* files. You might also need to restore the default Visual FoxPro registry settings.
Visual FoxPro includes the VFPClean.app tool so you can make sure all core Xbase and other files are set appropriately.
To run VFPClean.app
Type the following line of code in the Command window: Copy Code DO HOME()+”VFPCLEAN.APP”
You might encounter the following issues when installing Visual FoxPro:
If you cannot run Visual FoxPro and do not see error messages telling you what is wrong, the problem might be in your computer’s ROM BIOS or the video driver you are using.
If you are using an extended keyboard, be sure it is compatible with the ROM BIOS. In addition, make sure that you are using a standard VGA or Super VGA Windows video driver.
If you get a “stack overflow” error message, your video driver is out of date or not designed for your video card. To correct this problem, update the video driver.
For additional information, see the Visual FoxPro Readme at the root of the Visual FoxPro installation CD.
Microsoft Visual FoxPro protects your investment in applications built with previous versions of FoxPro. In Visual FoxPro, you can run many applications that were written in earlier versions with little or no conversion. You can modify and enhance applications using the Visual FoxPro language, knowing that most extensions to the language do not affect backward compatibility. In addition, you can convert FoxPro screens, projects, and reports to Visual FoxPro format.
However, it is possible that some behavior or feature changes in the current version of Visual FoxPro might affect existing Visual FoxPro source code. Therefore, you should review the new features, enhancements, and most recent behavior changes for this version. For more information, see What’s New in Visual FoxPro and Changes in Functionality for the Current Release.
Conversion to Visual FoxPro Format
If you choose to convert your dBASE or FoxPro files to the Visual FoxPro format, you can take advantage of the unique features of Visual FoxPro. You can run many files from some previous versions of FoxPro directly; others require varying levels of conversion.
You can convert most projects or components created using previous versions of Visual FoxPro simply by opening or recompiling them in this version of Visual FoxPro. When you recompile components, such as forms, screens, or reports, some modifications may be necessary. You can make modifications to these components in the same way you modify the components of this version of Visual FoxPro.
You can find additional information about upgrading from previous versions of Visual FoxPro on the Microsoft Developer Network (MSDN) Web site at http://msdn.microsoft.com. You can search the MSDN Archive for documentation of previous versions of Visual FoxPro.
You can explicitly convert FoxPro 2.6 and Visual FoxPro 3.0 files to the current Visual FoxPro format, which is necessary when you want to use these files with later versions of Visual FoxPro. Files that are created from later versions are converted automatically.
To convert FoxPro 2.6 and Visual FoxPro 3.0 files
On the File menu, click Open.
In the Open dialog box, browse for and select the file. The Visual FoxPro Converter dialog box opens. For more information, see Visual FoxPro Converter Dialog Box.
In the Visual FoxPro Converter dialog box, select the options you want.
To complete the file conversion, click Continue. Note: If you are converting Macintosh or MS-DOS files that have never contained Windows records, the Visual FoxPro Transporter dialog box appears. For more information, see Visual FoxPro Transporter Dialog Box.
You can also convert FoxPro 2.6 and Visual FoxPro 3.0 files by typing one of the following commands with the file name in the Command window:
Visual FoxPro is designed to be a fast relational database development system. However, applications you create with Visual FoxPro can have varying requirements and purposes. Therefore, you might want to optimize the operating system, Visual FoxPro, or your application for maximum performance.
Discusses how to get started, including information about installing, upgrading, and customizing Visual FoxPro to create state-of-the-art enterprise database solutions. What’s New in Visual FoxPro
Lists the new features and enhancements made to this version of Microsoft Visual FoxPro.Using Visual FoxPro
Provides links to information on Visual FoxPro programming features that are designed to improve developer productivity, including Access and Assign methods, support for more graphic file formats, and language to simplify programming tasks.Developing Visual FoxPro Applications
Includes conceptual information about how to develop Visual FoxPro applications, instructions for creating databases and the user interface, and other tasks needed to create Visual FoxPro applications.Programming in Visual FoxPro
Discusses how to access the full power of Visual FoxPro by creating applications. Understanding object-oriented programming techniques and the event-driven model can maximize your programming productivity.
Optimizing the Operating Environment
You can optimize Visual FoxPro performance by maximizing your computer’s hardware and operating environment. The following sections describe how you can optimize these areas:
Providing your computer with as much memory as possible is the most effective way to optimize your system for Visual FoxPro. You can also use memory more effectively by closing all other running applications on your computer. To maximize the use of your computer’s memory while running Visual FoxPro, follow these guidelines:
Do not run other Windows applications while running Visual FoxPro.
Use only those memory-resident programs needed for operation.
Simplify the screen display.
You can free memory by simplifying the way windows and screen backgrounds display on your computer monitor.
Use a color or a pattern for the desktop background instead of wallpaper.
Use the lowest-resolution display that is practical for you. The higher resolution of the display, the more memory your computer requires and the slower your graphics and user-interface elements appear. For VGA-compatible displays that use an extended mode driver, such as Video 7 or 8514, using the standard VGA driver ensures faster display performance but provides lower resolution and less color support.
To increase the number of applications that you can run simultaneously, Microsoft Windows supports virtual memory by swapping segments of code that is the least recently used from memory to the hard disk in the form of a paging file. As a rule, the default settings in the Windows operating system for managing virtual memory meet the requirements of most users and are the recommended settings.
Note:
The paging file does not improve Visual FoxPro performance and is not a substitute for more memory.
When Visual FoxPro searches for temporary files, for example, when you use the SYS(2023) – Temporary Path function to retrieve the temporary files path or when the TMPFILES, EDITWORK, PROGWORK, and SORTWORK settings in a Visual FoxPro configuration file do not specify a different location, the Windows API GetTempPath is used to search for the path containing the temporary files. GetTempPath searches a sequence of variables that differ depending on the operating system. Microsoft Windows 2000 and later include user variables that store the location of temporary files, while Microsoft Windows 95, 98, and Me include only global system environment variables for this purpose.
On Windows 2000 and later, GetTempPath, and therefore, SYS(2023), TMPFILES, EDITWORK, PROGWORK, and SORTWORK, searches the TMP user variable for the location of temporary files by default. If the TMP user variable does not specify a location, Visual FoxPro searches the following variables in a specific order:
TMP system variable.
TEMP user variable.
TEMP system variable.
If these variables do not specify a location, the location for storing temporary files defaults to the home drive and path, or the Temp folder in the user’s Documents and Settings directory.
Note:
If more than one value is specified for TMP or TEMP, then the first value is used.
On Windows 95, 98, and Me, GetTempPath searches the TMP and TEMP global system variables in that order and then searches the current directory.
Though Visual FoxPro is always fast, you can optimize the startup and operating speed. This section describes enhancing Visual FoxPro performance by managing startup speed and optimizing SET commands.
Managing Startup Speed
The time required to load and start Visual FoxPro relates to the physical size of Visual FoxPro, the length of the PATH statement in effect, the number of items to be found at startup, and other factors. You can control the load size, the search path, component file locations, and the startup SET command values of Visual FoxPro.
Managing File Locations
Visual FoxPro stores the FoxUser.dbf file, which contains user settings, in the user’s Application Data directory by default. You can display this location by typing ? HOME(7) in the Command window. Visual FoxPro searches for the FoxUser.dbf and Config.fpw files in the following places:
In the startup application or executable file, if any. For example, you can start a Visual FoxPro application by typing the following code on the command line: Copy Code VFPversionNumber.exe MyApp.app – or – Copy Code VFPversionNumber.exe MyApp.exe If the startup application or executable file contains a Config.fpw file, the configuration file is always used. You can override settings in a Config.fpw file that are bound inside an application by specifying an external Config.fpw file, using the -C command-line switch when starting an application or Visual FoxPro.
In the working directory.
Along the path established with the PATH environment variable.
In the directory containing Visual FoxPro.
Controlling File Loading
You can also speed startup by preventing Visual FoxPro from loading files you don’t plan to use. If your application does not use the FoxUser or FoxHelp file, disable them in the Config.fpw file by using the following commands:
Copy Code
RESOURCE = OFF HELP = OFF
Visual FoxPro seeks all other Visual FoxPro components (GENXTAB, CONVERT, and so on) only in the Visual FoxPro directory. If you place components elsewhere, you must explicitly identify the path to those components in your Config.fpw file. For example, you might specify these locations:
You can use the environment variable FOXPROWCFG to explicitly specify the location of Config.fpw. For details about the FOXPROWCFG variable, see Customizing the Visual FoxPro Environment.
Optimizing the Load Size of Visual FoxPro
If you don’t plan on using any of the Visual FoxPro components listed previously, set them to an empty string to speed startup.
To optimize the load size of Visual FoxPro, use the following syntax:
Copy Code
cFileVariable= “”
Replace cFileVariable with _TRANSPORT, _CONVERT, or other variables as appropriate.
Optimizing Key SET Commands
You can optimize the operation of Visual FoxPro by tuning the values of certain SET commands.
The following table shows SET commands that have the greatest effect on performance, and their settings for maximum performance. You can specify SET command values by including them in the Config.fpw file, by typing them in the Command window, or by setting them in the Options dialog box.
Optimizing Visual FoxPro in a Multiuser Environment
When you run Visual FoxPro or Visual FoxPro applications in a multiuser environment, you can improve performance by managing storage of temporary files and controlling the way tables are shared.
Managing Temporary Files
In most multiuser environments, it is recommended that you save temporary files to local disks or memory when networked computers contain large amounts of free disk space. Redirecting storage of temporary files can improve performance by reducing frequent access to the network drive.
On small networks with older networked computers and slow hard disks, you might experience better performance by leaving Visual FoxPro temporary files on the file server; however, when in doubt, direct temporary files to the local disk. When working on large, heavily used networks, always redirect temporary files to the local disk.
By saving all temporary files to a single directory on a local hard drive, you can safely erase the contents of the temporary file directory on the file server prior to each Visual FoxPro session. This action purges the system of any temporary files that were created but not erased by Visual FoxPro due to a system reboot or power loss.
For more information about temporary files, see Optimizing the Operating Environment and How to: Specify the Location of Temporary Files.
Sharing Tables
If users share tables on a network, the way you manage access to them can affect performance.
Avoid opening and closing tables repeatedly.
Buffer write operations to tables that are not shared.
Provide exclusive access to tables.
Limit the time on locking tables.
Providing Exclusive Access
You can enhance performance for the APPEND, REPLACE, and DELETE commands and operations that run at times when no other users require access to the data, for example, overnight updates, by opening data files for exclusive use. When tables are open for exclusive use, performance improves because Visual FoxPro does not need to test the status of record or file locks.
To open data files for exclusive use, use the EXCLUSIVE clause in the USE and OPEN DATABASE commands. For more information, see USE Command and OPEN DATABASE Command.
Limiting the Time on Locking Tables
You can reduce contention between users for write access to a table or record by shortening the amount of time for locking a record or table. Instead of locking a record while the user edits it, lock the record only after it has been edited. Using optimistic row buffering provides the shortest amount of time that records are locked. For more information, see Buffering Data.
The configuration of Visual FoxPro determines how your copy of Visual FoxPro looks and behaves. For example, you can establish the default locations for files used with Visual FoxPro, how your source code looks in an edit window, and the format of dates and times.
You can make changes to the Visual FoxPro configuration that exist for the current session only (temporary), or specify them as the default settings for the next time you start Visual FoxPro (permanent). If the settings are temporary, they are stored in memory and are discarded when you quit Visual FoxPro.
If you make permanent settings, they are stored in the Microsoft Windows registry or Visual FoxPro resource file. The Windows registry is a database that stores configuration information about the operating system, all Windows applications, OLE, and optional components such as ODBC. For example, the registry is where Windows stores the associations between file name extensions and applications so that when you click a file name, Windows can launch or activate the appropriate application.
For an example of how to change the registry, you can examine Registry.prg in the SamplesClasses directory, which contains numerous methods based on Windows API calls and makes it possible for you to manipulate the Windows registry.
Similarly, Visual FoxPro stores its application-specific configuration information in the registry. When you start Visual FoxPro, the program reads the configuration information in the registry and sets the configuration according to those settings. After reading the registry, Visual FoxPro also checks for a configuration file, which is a text file in which you can store configuration settings to override the defaults stored in the registry. After Visual FoxPro has started, you can make additional configuration settings using the Options Dialog Box or SET commands. For more information, see How to: View and Change Environment Settings.
Note:
The run-time version of Visual FoxPro does not read the Windows registry when starting up, as registry settings are designed primarily to configure the development environment. If you intend to distribute your Visual FoxPro applications using a run-time library, you can establish configuration settings in two ways: with a configuration file, or with a program that manipulates the Windows registry on the user’s computer.
Visual FoxPro also maintains a resource file, Foxuser.dbf, that stores information about the current state of the program when you quit. For example, the resource file contains information about the location and size of the Command window, current keyboard macros, the toolbars that are displayed, and so on. The Foxuser.dbf file is an ordinary Visual FoxPro table, which you can read and change as required by your application.
Tip:
If the data in the Foxuser.dbf file becomes corrupted or invalid, it can cause Visual FoxPro to behave in an erratic manner. If you do not manually store anything in the table, for example keyboard macros, deleting the table might help.
Summary: Discusses using Microsoft® Visual FoxPro® version 6.0 with MTS to develop three-tier applications. (36 printed pages).
Contents
IntroductionWhat Is Microsoft Transaction Server?Why Is MTS Important for Visual FoxPro Developers?Creating Your First MTS ServerSetting Up SecurityThe Basic Features of MTSJust-In-Time ActivationTransactionsProgramming ModelsDeploymentRemote Deployment and AdministrationSecurityShared Property ManagerMTS Support for Internet Information ServerAutomating MTS AdministrationTips and TricksExpand table
Click to copy the sample files associated with this technical article.
Introduction
No doubt you’ve heard all about Microsoft Transaction Server (MTS) and how it will make your life easier to develop three-tier applications. This article offers a good primer on using Visual FoxPro 6.0 with MTS. We cover the basics of using MTS and then extend it to using with Visual FoxPro Component Object Model (COM) Components. This document is intended to be used with the Microsoft PowerPoint® slide show included with the Visual FoxPro sample files.
MTS is a great environment for working with three-tier development. However, one should realize that it is simply not just a matter of dropping your Visual FoxPro servers into an MTS package and expecting miracles. While it is true that much of the work is already done for you, nothing comes for free. Performance and scalability are critical factors that require well-thought-out designs. Good MTS applications are designed with MTS in mind from the start!
This article assumes that you have MTS already installed. It is available in the Microsoft Windows NT® version 4.0 Option Pack, available from the Microsoft Web site at https://www.microsoft.com/windows/downloads/default.asp.
In addition, you should familiarize yourself with the basics of MTS. Information is available in the Help files provided with MTS when you install the Windows NT 4.0 Option Pack.
What Is Microsoft Transaction Server?
MTS is a component-based transaction processing system for building, deploying, and administering robust Internet and intranet server applications. In addition, MTS allows you to deploy and administer your MTS server applications with a rich graphical tool (MTS Explorer). MTS provides the following features:
The MTS run-time environment.
The MTS Explorer, a graphical user interface for deploying and managing application components.
Application programming interfaces (APIs) and resource dispensers for making applications scalable and robust. Resource dispensers are services that manage nondurable shared state on behalf of the application components within a process.
The MTS programming model provides a framework for developing components that encapsulate business logic. The MTS run-time environment is a middle-tier platform for running these components. You can use the MTS Explorer to register and manage components executing in the MTS run-time environment.
The three-tier programming model provides an opportunity for developers and administrators to move beyond the constraints of two-tier client/server applications. You have more flexibility for deploying and managing three-tier applications because:
The three-tier model emphasizes a logical architecture for applications, rather than a physical one. Any service may invoke any other service and may reside anywhere.
These applications are distributed, which means you can run the right components in the right places, benefiting users and optimizing use of network and computer resources.
Why Is MTS Important for Visual FoxPro Developers?
Microsoft is investing a great amount of resources in three-tier development because of a multitude of benefits derived from this architecture. As shown in Figure, Tier 2, the so-called “middle tier,” represents the layer where much of the Application Services/Business Logic is stored. Visual FoxPro COM components are ideally suited for this architecture and will play a key role in this tier for many years to come. This middle tier is also where MTS lives.
Figure 1. Web-enabled three-tier architecture
Future applications will consist of Web based front ends using a combination of HTML/XML. While Visual FoxPro data can be used as your database of choice for Tier 3, your applications should be written to communicate to a generic back end. This should be a test of your application’s extensibility. “How easy is it to swap back ends—let’s say Visual FoxPro database to Microsoft SQL Server™?” There are several options, including Open Database Connectivity (ODBC) and ActiveX® Data Objects (ADO), which provide generic interfaces to data. Remember, your application should be written knowing that any or all of the three tiers can be swapped out independent of each other.
So why is MTS great for Visual FoxPro developers? It should be clear now that the ability to swap out tier components at will makes for a great reusability story. Microsoft has a concept called total cost of ownership (TCO), which means the collective cost of providing and maintaining corporate Information Services. The three-tier model goes a long way toward reducing TCO.
Updating the Presentation layer is very easy because it merely involves one having to refresh his/her browser. Windows front ends consisting of Visual FoxPro/Visual Basic® forms offer more flexibility in user interface, but updating 150 sites can be time-consuming. In addition, one should expect improved UI options available in HTML.
The back-end data is usually the tier that changes the least. Having data managed centrally also reduces costs. Remember that data can be distributed and still managed from one location. It doesn’t have to be stored centrally to be managed centrally.
Finally, we get to Visual FoxPro’s role in the middle tier. Middle-tier components tend to change most often because they represent business rules, which change as the needs of the business changes. Traditional client/server and monolithic applications would often combine the first two layers into one. This was very inefficient because of the distribution costs in updating sites. Today, with browsers, much of this distribution problem goes away. However, business rules are often complex and can contain sensitive/secure information, so it’s not always wise to send these rules back with the HTML to a Web browser. In addition, it can impede performance.
So, we end up with a dilemma. We want to limit the amount of information sent back to the client, but we also want to minimize the number of back and forth trips between client and server, because bandwidth is also a big consideration (more so with the Internet versus an intranet). The best solution is one involving a so-called “Smart Client.” Traditionally, the Web browser is thought of as an unintelligent client whose job is to merely display an entire static Web page. Each time something on the page changes, we need to refresh the entire Web page. With dynamic HTML (DHTML), you no longer need to do this. Only parts of the Web page affected need updating. In addition, some of the business rules can (and should) reside on the client, thus reducing round trips to the server. For example, you may want to have your client have simple data validation rules, such as one to ensure a value is not negative. It would be more efficient to perform these sorts of checks on the client. Most of the rules, especially sensitive ones, will exist on the server away from client eyes. It is also important to realize, however, that client-side business rules are subject to change almost as frequently as those on the server. The ATSWeb application (available at https://msdn.microsoft.com/vfoxpro/ats_alpha/default.htm) offers a great example of business rules being applied to both client and server.
MTS provides an environment for hosting your Visual FoxPro middle-tier objects because it handles many of the common tasks, including resource and thread management, security, deployment, application robustness, and transactions. This leaves you, the developer, with only the responsibility of providing business logic specific to your application.
Creating Your First MTS Server
Let’s jump right in and create an MTS server, because it’s very simple if you already know how to create a Visual FoxPro COM component.
Creating a Visual FoxPro COM Component
Create a new project file called test1.pjx
Create a new program file (PRG) called test1.prg
Add the following code to this program:DEFINE CLASS server1 AS custom OLEPUBLIC PROCEDURE hello RETURN “Hello World” ENDPROC ENDDEFINE
Build the server as a DLL (for example, test1.dll). All MTS components must be created as in-process DLL servers. You now have a server that can be tested directly in Visual FoxPro:x=create(“test1.server1”) ? x.hello()
Adding the Visual FoxPro COM Component to an MTS Package
A package is a collection of components that run in the same process. Packages define the boundaries for a server process running on a server computer. For example, if you group a Sales component and a Purchasing component in two different packages, these two components will run in separate processes with process isolation. Therefore, if one of the server processes terminates unexpectedly (for instance, because of an application fatal error), the other package can continue to execute in its separate process.
This section describes the task of installing the Visual FoxPro server into the MTS environment.
Launch MTS Explorer.
In the left pane, navigate to the Computers item and select My Computer. You are now looking at the MTS environment.
Click the Packages Installed node to view all default packages installed by MTS. You can think of a Package as a set of components that perform related application functions. For example, an Inventory package might consist of two DLLs, each performing a task related to checking product inventory for a customer order.
Let’s create a new package now. Select the Action -> New -> Package menu item.
Click the Create an empty package button. Type in a name for your new package (for example, Foxtest1).
Click the Next button, and then click the Finish button. You should now see your new package added under the Packages Installed node.
Click your new package node (for example, Foxtest1). You should now see two items. The Components folder is where you add new components such as the Visual FoxPro component you just created. The Roles folder is where you set up groups of users (roles) who all share similar access privileges (security). You do not need to add anything to the Roles folder in order to use your Visual FoxPro component with MTS.
Click the Components folder and select the Action -> New -> Component menu item.
Click the Install new component(s) button. This will bring up the Install Components dialog box. Click the Add files button and go to the location where you created your Visual FoxPro server (for example, test1.dll). Select both the .dll and .tlb files. The .tlb file is the type library file containing properties and methods of your server. After selecting these two files, you should see your OLEPUBLIC component listed in the lower panel. Click Finish and you should see your server added to this folder.
At this point, your package is complete and ready to go. Later, we will talk about setting Transaction support. This can be done from the Properties dialog box of your server.
Accessing Your Component
You can now test your new MTS packaged component using a command similar to the one used to test Visual FoxPro after the DLL server was first created.
x=create("test1.server1")
? x.hello()
That’s all you need to do! If you go back into the MTS Explorer, you should see the component represented with a spinning icon. Click the Status View to see details about the state of the object.
Figure 2. New component viewed in MTS Explorer
If you release the object (RELEASE x), MTS releases its reference.
Going Forward
We’ve just discussed the basics of installing your Visual FoxPro server in MTS. Essentially, all we did was wrap the Visual FoxPro component inside an MTS process that manages security, transaction state, fault tolerance, and other common server responsibilities. All Visual FoxPro servers used with MTS are registered this way. The remainder of the article discusses how to take advantage of MTS-specific features such as security and transactions. You can write code in your components that talk directly to the MTS run-time environment. In addition, the above process can be entirely automated, because MTS exposes an administrative Automation interface.
Setting Up Security
So why are we starting out so early with security? Well, sooner or later, you’re going to fiddle with some sort of security switch and suddenly that MTS application of yours will no longer work. It’s important that you follow these instructions and refer to them later when you decide to add security to your applications.
NoteMTS 2.0 security setup is described in the Readme document. If you have MTS installed on Microsoft Windows® 95, you can skip this section.
Setting System Package Identity
Before you do anything in MTS, it is a good idea to configure the system package for administrating security. When installing MTS, set the system package identity before creating any new packages as follows:
Create a new local Windows NT group named “MTS Administrators” and a new local user named “MTS Administrator.”
Add the “MTS Administrator” user to the “MTS Administrators” and “Administrators” groups.
Set the identity of the system package to “MTS Administrator.” If this does not work, try setting this to the Administrator user.
NoteYou cannot set a package’s identity to a group.
Shut down the system package so that it will be restarted with the new identity. You can do this by right-clicking the My Computer icon in MTS Explorer and selecting Shut Down Server Processes.
Adding Security for MTS Packages
You first need to determine whether you want all or just a few components in your Package to have security. Right-click the Package and select Properties. Next, click the Security tab. Then check the Enable authorization checking check box. To enable or disable security at a component level, right-click a component and display the Properties dialog box.
If this is all you do, an “Access is denied” error message is generated when you try to access your component. You MUST associate a valid role with any component marked for security!
Right-click the package’s Roles folder and select New Role. Type in a functional role such as Managers, Accountants, and so on.
The new role is added as a subfolder. Right-click this folder to Add New User (you will get a dialog box to Add Users and Groups to Role). Select the user(s) that you want to add to your role. To finish, select the Role Membership folder under each component that is marked for security and add the new role created in step 3 by right-clicking the folder and selecting New Role.
NoteYou may still experience the “Access is denied” error message when running your components. There are a couple of possible solutions:
Sometimes adding a Group to a role does not work (step 3). You might try adding individual users instead.
The user rights for that user are not properly set. Make sure the user account for the identities of the system package and other MTS packages have the Windows NT “Log on as a service” user right. You can verify this by using the Windows NT User Manager:
From the Policies menu, select User Rights.
Click Show Advanced User Rights.
Tips for Visual FoxPro Users
Much of the security administration can easily be handled by Automation using the MTS Admin objects. You can set up Security administration in the AfterBuild event of a ProjectHook class you have tied to the project that generates your MTS COM DLL server. See the section “Using Visual FoxPro 6.0 Project Hooks” for examples.
The Basic Features of MTS
Before we jump right into using Visual FoxPro with MTS, let’s review some basic concepts that you need to know in order to make effective use of the MTS environment. For more detailed information, see MTS Help.
Activity
An activity is a collection of MTS objects that has a single distributed thread of logical execution. Each MTS object belongs to a single activity. This is a basic concept that describes how the middle-tier functions when confined to the MTS environment. In an MTS package, multiple clients can access objects, but only one object per client is running at a time on a single thread.
Context
Context is state that is implicitly associated with a given MTS object. Context contains information about the object’s execution environment, such as the identity of the object’s creator and, optionally, the transaction encompassing the work of the object. The MTS run-time environment manages a context for each object.
As a developer, think of every Visual FoxPro object that is registered in an MTS package as having an associated Context object that is created every time you instantiate the Visual FoxPro object. So, each time you issue a CreateObject command, two objects are created—your server and its associated Context. In fact, you can return an object reference to this Context object directly in your code, as in the following example:
The Context object has the following properties and methods.Expand table
Count
CreateInstance
DisableCommit
EnableCommit
IsCallerInRole
IsInTransaction
IsSecurityEnabled
Item
Security
SetAbort
SetComplete
As you can see, the properties, events, and methods (PEMs) are used to access information related to the object transaction and security context (see MTS Help for more details on specific syntax for these PEMs). It is important to understand that the Context state is inherited. An object in a package called from another object in the same package will inherit the state of its caller. Because Context is confined within the same process, state, such as security, is trusted. No object in a package needs to explicitly provide its own security. When your object is released, so is its Context.
Package
Packages, as we just described, are the building blocks of MTS. Think of them as mini applications—a set of components that perform related application functions. All components in a package run in the same MTS process.
Remember, “Good MTS applications are designed with MTS in mind from the start.” You should design your Package contents with your entire application in mind. Each package runs in its own process, so try to design packages that don’t attempt to do more than they absolutely need to. There are performance advantages to maintaining many components within in a single package, but there may also be security constraints (roles) that dictate a different architecture.
Packages are also the primary means of deployment. The MTS environment allows one to export the contents of a Package to a nice distributable setup (both client and server). We’ll discuss this in the “Deployment” section.
Role
A role is a symbolic name that defines a class of users for a set of components. Each role defines which users are allowed to invoke interfaces on a component. A role is the primary mechanism to enforce security. Role-based security is handled at the component level. It’s possible that this may be at the method level in a future version of MTS. Security cannot be enforced on the Windows 95 version of MTS.
Roles are stored at the package level. Each component in a package can belong to one of more of the defined roles. For example, an Inventory package might contain a Visual FoxPro server whose responsibility is to handle inventory. There are two roles defined in this package: Managers and Clerks. These two roles are simply collections of Windows NT users/groups with a collective name that you provide. Your server is coded so that Clerks can access inventory data for normal order entries and reporting. Managers have additional power in that they can override inventory levels to make adjustments (for example, quarterly product shrinkage estimates).
You can set up security so that it is automatically handled (for instance, users not in roles are given “Access is denied” error message), or you can manage it programmatically through code. The Context object’s IsCallerInRole method is ideal for this.
Resource Dispensers
A resource dispenser manages nondurable shared state on behalf of the application components within a process. Resource dispensers are similar to resource managers, but without the guarantee of durability. MTS provides two resource dispensers:
The ODBC resource dispenser
The Shared Property Manager
Resources are shared within the same process—same process = same package. In the section “Shared Property Manager,” we discuss programmatically accessing shared properties. This is a really cool thing for Visual FoxPro developers because it allows multiple instances of objects to share state information. For example, you could have a counter that tracks the last ID number used by a database.
ODBC resource dispenser
The ODBC resource dispenser manages pools of database connections for MTS components that use the standard ODBC interfaces, allocating connections to objects quickly and efficiently. Connections are automatically enlisted on an object’s transactions and the resource dispenser can automatically reclaim and reuse connections. The ODBC 3.0 Driver Manager is the ODBC resource dispenser; the Driver Manager DLL is installed with MTS.
Shared Property Manager
The Shared Property Manager provides synchronized access to application-defined, process-wide properties (variables). For example, you can use it to maintain a Web page hit counter or to maintain the shared state for a multiuser game.
Resource Managers
A resource manager is a system service that manages durable data. Server applications use resource managers to maintain the durable state of the application, such as the record of inventory on hand, pending orders, and accounts receivable. Resource managers work in cooperation with the Microsoft Distributed Transaction Coordinator (MS DTC) to guarantee atomicity and isolation to an application. MTS supports resource managers, such as Microsoft SQL Server version 6.5, that implement the OLE Transactions protocol.
The MS DTC is a system service that coordinates transactions. Work can be committed as an atomic transaction even if it spans multiple resource managers, potentially on separate computers. MS DTC was first released as part of SQL Server 6.5 and is included in MTS, providing a low-level infrastructure for transactions. MS DTC implements a two-phase commit protocol to ensure that the transaction outcome (either commit or abort) is consistent across all resource managers involved in a transaction. MS DTC ensures atomicity, regardless of failures.
You might be asking if Visual FoxPro is a resource manager, because it has its own native database. Unfortunately, the answer is no. Visual FoxPro transactions are native to Visual FoxPro and do not go through the MS DTC. Therefore, automatic transaction support within MTS is not supported for Visual FoxPro data. You cannot use the Context object’s SetAbort method to abort a transaction if the data is stored in Visual FoxPro databases/tables. The database must either support OLE Transactions (SQL Server) or be XA-compliant (Oracle).
Base Clients
A base client is simply a client that runs outside of the MTS run-time environment, but instantiates MTS objects. In a three-tier architecture, a base client is typically the presentation layer, such as an application form or Web page. The base client neither knows nor needs to know that MTS is used in the middle tier. It merely creates an instance of an object that exists in an MTS package and awaits a response. The following table describes some of the differences between a base client and an MTS component, such as a Visual FoxPro DLL server.Expand table
Base client
MTS component
Can be EXEs, DLLs.
Must be in-process DLL.
MTS does not manage its process.
Manages server processes that host MTS component.s
MTS does not create or manage threads used by application.
Creates and manages threads.
Does not have implicit Context object.
Each MTS object has own Context object.
Cannot use Resource Dispensers.
Can use Resource Dispensers.
Just-In-Time Activation
Just-in-Time (JIT) activation is the ability to activate an MTS object only as needed for executing requests from a client. Most Visual FoxPro developers are familiar with object instantiation, as in the following code:
A “stateful” object created by this code retains state during the lifetime of the object (until it is released). This means that property values (such as myProperty) are retained between statement execution. When the object is finally released, all object references and state are released.
There is overhead with creating objects from your Visual FoxPro components. Each time you instantiate an object, Visual FoxPro needs to allocate a certain amount of memory. In addition, the first time you create an object, Visual FoxPro takes a little extra time to load its run-time libraries. When the last instance is released, the entire Visual FoxPro run time is also released.
JIT activation addresses many of these memory issues that affect performance. The first thing JIT does is cache the server’s run-time libraries in memory, even though no outstanding object references exist. The first time you instantiate a Visual FoxPro server that’s in an MTS package, the Visual FoxPro run time loads the address space of the MTS process. When you release the object, MTS still keeps the libraries in memory for a specified amount of time. You can change this setting in the package’s property sheet (default = 3 minutes). This saves having to reload the run time when the object count hits 0.
The main thing that JIT activation offers is ability to transform your object from “stateful” to “stateless” mode. In the preceding example, you can interpret a “stateless” object as one having the initial default settings. So, in the example, the value of myProperty would be reset to its original setting. A stateless object is managed by MTS and is very lightweight, so it consumes much less memory. The only thing keeping the stateless object alive is the object reference held onto by the client. Internally, MTS recycles threads consumed by stateful objects when they go stateless. When a method is invoked on that object, it then becomes stateful again on a thread that could be different from the one originally created on.
Putting your objects into a stateless mode is handled easily by the Context object. The following code illustrates putting an object in a stateless mode:
This code is actually called from within a method of your Visual FoxPro server. You can see if your object is stateless by viewing the status of your component in the MTS Explorer. A stateless object appears in the Objects column, but not in the Activated or In Call columns.
Use the SetComplete method to put the object in a stateless mode. Use SetComplete for committing transactions (as we discuss in the next section, “Transactions”). You can also use SetAbort to make an object stateless.
Again, when you change an object to stateless, all property settings revert to their original defaults. When you invoke a method (or property set/get) on this stateless object, the object is activated (goes stateful) and the object’s INIT event is fired. When you call SetComplete, the object DESTROY event is fired.
NoteAny state that exists on the object is lost when the object is deactivated (SetComplete). If you need to save state, you should either persist information to a database or use the MTS Shared Property Manager.
Because your object’s INIT is called whenever your object goes from Stateless to Stateful, you should try to minimize the amount of code in this event.
Here is a simple scenario showing interaction between client and MTS server.
Visual FoxPro server code:
DEFINE CLASS mts2 AS Custom OLEPUBLIC
MyColor = "Green"
PROCEDURE InUsa (tcCustID)
LOCAL llInUSA,oMTX,oContext
oMtx = CreateObject("MTXAS.APPSERVER.1")
oContext = oMtx.GetObjectContext()
llInUSA = .F.
USE CUSTOMER AGAIN SHARED
LOCATE FOR UPPER(cust_id) == UPPER(tcCustID)
IF FOUND()
llInUSA = (ATC("USA",country)#0)
ENDIF
oContext.SetComplete()
RETURN llInUSA
ENDPROC
ENDDEFINE
Base client executes following code:
LOCAL oCust,cCust,lUsa
oCust = CreateObject("vfp_mts.mts2")
? oCust.MyColor
Green
oCust.MyColor = "Red"
? oCust.MyColor
Red
cCust = "JONES"
lUsa = oCust.InUsa(cCust) && object goes stateless (deactivated)
? oCust.MyColor && object is activated (stateful)
Green
RELEASE oCust && object is fully released
Notice in the preceding example how the state of oCust is lost after the InUsa method is called. The MyColor property no longer returns Red, but is instead reset to its original value of Green.
Transactions
If you have used Visual FoxPro at all, you are probably aware that Visual FoxPro supports transactions. Changes to your data can be committed or rolled back. Though transactions are critical to MTS, don’t be misled by the name; there is a lot more to it than just transactions. However, the ability to have MTS automatically handle transactions between distributed objects is quite powerful. Transactions are often discussed in terms of the ACID acronym:
Atomicity—ensures that either the entire transaction commits or nothing commits.
Consistency—a transaction is a correct transformation of the system state.
Isolation—protects concurrent transactions from seeing each other’s partial and uncommitted results.
Durability—committed updates to managed resources can survive failures.
As just mentioned, MTS transaction support is not compatible with Visual FoxPro data. It only works with databases supporting OLE transaction or XA protocols. Both SQL Server and Oracle data can be used with MTS in transactional fashion.
You should understand what we mean by a transaction and to what extent things are either committed or rolled back. Consider the following scenario (all done within confines of two components in a single MTS package):
Component A adds a new customer record to the Customer table in SQL Server.
Component A writes out new record to a Visual FoxPro database (audit log).
Component A sends e-mail notification of new customer to some manager.
Component A calls Component B.
Component B edits the Orders table with a new order in SQL Server.
Component B writes out text log file of activity.
Component B completes activity by committing the transaction (SetComplete).
Component A discovers bad credit history with customer and aborts transaction (SetAbort).
When Component B commits in step 7, not a whole lot happens because MTS manages the entire Context within the package in a distributed fashion. Component B actually inherits transaction state from Component A, so it cannot really fully commit the transaction. The real transaction terminates in step 8 when the last object with transaction state aborts. At this point, changes made to both Customer and Orders tables are rolled back because these tables are in SQL Server. Unfortunately, the Visual FoxPro table update, e-mail notification, and text log file activities are not rolled back. When a transaction is aborted/committed, only data managed through the MS DTC is affected. There is no event that is magically triggered. (Check out the MTS SDK for ideas on using Spy).
Remember, good MTS apps are written with MTS in mind from the start. Managing transactions is very important, and while much of it is handled automatically, you will need to provide a fair amount of code to effectively manage all the resources being utilized in a transaction setting.
Transaction support is set at the component level, but transactions can span multiple packages. You can set this option in the MTS Explorer from the component’s Property Sheet (see MTS Help for details on the various options). Again, the object’s Context manages and passes on transaction state for a given component. If the transaction setting of a component is marked as “Requires a transaction,” a transaction is always associated with the component. If another object that calls this component already has a transaction in effect, no new transaction is created. The component merely inherits the current one. A new transaction is only created if one does not already exist in the context.
Figure 3. Setting Transaction support
Let’s return a minute to the SetComplete and SetAbort methods. These methods actually serve two purposes. From their names, they imply functionality related to transactions. However, as already discussed, they also serve to deactivate objects (make them stateless). In fact, these methods can be used simply for JIT activation without any concern for transactional support. Again, SetComplete releases valuable resources/memory used by MTS to allow for improved scalability. The Context object also includes several other methods useful for transactions: EnableCommit, DisableCommit, and IsInTransaction. The following example shows how to handle transactions in Visual FoxPro:
LPARAMETER tcCustID
LOCAL lFound,oMTX,oContext
oMtx = CreateObject("MTXAS.APPSERVER.1")
oContext = oMtx.GetObjectContext()
USE CUSTOMER AGAIN SHARED
LOCATE FOR UPPER(cust_id) == UPPER(tcCustID)
lFound = FOUND()
IF FOUND()
oContext.SetComplete()
ELSE
oContext.SetAbort()
ENDIF
RETURN lFound
In this scenario, we assume that another component already performed an update on another table (for example, Orders). If the customer ID in the preceding code was not found, the entire transaction would be rolled back.
You’re probably wondering how transactions work in the code, which clearly appears to be against Visual FoxPro data. Actually, this example is using Remote Views against SQL Server data. Again, Visual FoxPro tables do not support OLE transactions, so you will not get MTS transaction support if you use DBF tables. However, data updates either to Remote Views or by SQL pass-through work just fine.
**Tip **Make sure that your connection to a remote data source is made without any login dialog box. If you are using a connection stored in a DBC, ensure that the Display ODBC logins prompt is set to Never. For access to remote data through SQL pass-through commands, you can use the SQLSetProp function:
SQLSETPROP(0, 'DispLogin', 3)
Programming Models
MTS supports two programming models. The TransactionContext model is intended primarily for backward compatibility. It essentially lets the base client control the transaction. The assumption is that the COM component has no MTS awareness (that is, the component was written before MTS was available). The second model is called the ObjectContext model and assumes the COM component inside the MTS package has MTS smarts and is aware of its Context object.
TransactionContext
We do not recommend using this model for new three-tier applications, because it has limited access to the full capabilities of MTS. It merely offers a way to provide some transaction support to applications whose middle-tier components were developed without MTS in mind. The burden of transaction handling rests on the base client. With this model, the base client is likely to be a smart client that has scripting capabilities (for example, an application form). The base client is less likely to be a Web page, and it always runs outside of the MTS run-time environment.
The following code snippet in a Visual FoxPro form (base client) shows this model in use. The middle-tier component is a Visual FoxPro server whose ProgID is “vfp_mts.mts1”. The assumption here is that this server knows nothing about MTS, thus requiring the base client to perform all transaction handling:
The code in the middle tier simply does a lookup in a SQL Server table for a customer’s home country. If the record was actually changed, the base client would have the capability to actually commit or roll back the transaction. The TransactionContext object only supports three methods: CreateInstance, Commit, and Abort.
ObjectContext
The ObjectContext model is the only model you should consider for new MTS application development. It relies on component awareness of MTS, but this should be your goal so that you can optimize performance and take advantage of MTS-specific features.
Unlike the TransactionContext object, which uses the following PROGID:
#DEFINE TRANS_CLASS "TxCtx.TransactionContext"
the ObjectContext object can be accessed using the following code:
#DEFINE MTX_CLASS "Mtxas.AppServer.1"
The ObjectContext object, which can be referenced in your Visual FoxPro code, as shown here:
LOCAL oMTX,oContext
oMtx = CreateObject("MTXAS.APPSERVER.1")
oContext = oMtx.GetObjectContext()
contains the following properties, events, and methods (PEMs).Expand table
PEM
Description
Count
Returns the number of Context object properties.
CreateInstance
Instantiates another MTS object.
DisableCommit
Declares that the object hasn’t finished its work and that its transactional updates are in an inconsistent state. The object retains its state across method calls, and any attempts to commit the transaction before the object calls EnableCommit or SetComplete will result in the transaction being aborted.
EnableCommit
Declares that the object’s work isn’t necessarily finished, but its transactional updates are in a consistent state. This method allows the transaction to be committed, but the object retains its state across method calls until it calls SetComplete or SetAbort, or until the transaction is completed.
IsCallerInRole
Indicates whether the object’s direct caller is in a specified role (either directly or as part of a group).
IsInTransaction
Indicates whether the object is executing within a transaction.
IsSecurityEnabled
Indicates whether security is enabled. MTS security is enabled unless the object is running in the client’s process.
Item
Returns a Context object property.
Security
Returns a reference to an object’s SecurityProperty object.
SetAbort
Declares that the object has completed its work and can be deactivated on returning from the currently executing method, but that its transactional updates are in an inconsistent state or that an unrecoverable error occurred. This means that the transaction in which the object was executing must be aborted. If any object executing within a transaction returns to its client after calling SetAbort, the entire transaction is doomed to abort.
SetComplete
Declares that the object has completed its work and can be deactivated on returning from the currently executing method. For objects that are executing within the scope of a transaction, it also indicates that the object’s transactional updates can be committed. When an object that is the root of a transaction calls SetComplete, MTS attempts to commit the transaction on return from the current method.
Deployment
Microsoft Transaction Server offers excellent tools for deploying both client- and server-side setups. Setups are made at the package level, so you should include all components for your application in a particular package. The deployment package contains all the distributed COM (DCOM) configuration settings you need, so you don’t have to fuss with the messy DCOM Configuration dialog box.
To create a setup
Click the package that you want to create setup.
Select Export… from the Action menu. The Export dialog box is displayed.
Figure 4. Exporting a package
**Important **The directions in the Export dialog box are not very clear. You should not simply type in a path as specified. If you do, the Export routine creates a file with a .pak extension in the folder location you specify. Instead, you should always type a full path and file name for the .pak file, as shown in Figure 4.
You can also use the scriptable administration objects to automate deployment and distribution of your MTS packages. See the section “Remote Deployment and Administration” to follow for more details.
The output of the Export operation consists of two setups:
Server Setup
This setup, which is placed in the folder specified in the Export dialog box, contains the .pak file and all COM DLL servers used by the package.
NoteWith Visual FoxPro servers, you will also have .tlb (type library) files included. You can install this package by selecting Install from the Package Wizard in MTS Explorer.
Figure 5. Installing package from the Package Wizard
Client Setup
The Export process creates a separate subfolder named “clients” in the folder specified in the Export Package dialog box. The Clients folder contains a single .exe file that a user can double-click to run.
The Client setup merely installs necessary files and registry keys needed by a client to access (remotely through DCOM) your MTS package and its COM servers.
Remote Deployment and Administration
The MTS Explorer allows you to manage remote components (those installed on a remote machine). The Remote Components folder contains the components that are registered locally on your local computer to run remotely on another computer. Using the Remote Components folder requires that you have MTS installed on the client machines that you want to configure. If you want to configure remote computers manually using the Explorer, add the components that will be accessed by remote computers to the Remote Components folder.
Pushing and Pulling
If both the server and client computer are running MTS, you can distribute a package by “pulling” and “pushing” components between one or more computers. You can “push” components by creating remote component entries on remote computers and “pull” components by adding component entries to your local computer. Once you create the remote component entries, you must add those component entries to your Remote Components folder on your local machine (pull the components).
Before you deploy and administer packages, set your MTS server up by doing the following:
Configure roles and package identity on the system package.
Set up computers to administer.
You must map the System Package Administrator role to the appropriate user in order to safely deploy and manage MTS packages. When MTS is installed, the system package does not have any users mapped to the administrator role. Therefore, security on the system package is disabled, and any user can use the MTS Explorer to modify package configuration on that computer. If you map users to system package roles, MTS will check roles when a user attempts to modify packages in the MTS Explorer.
Roles
By default, the system package has an Administrator role and a Reader role. Users mapped to the Administrator role of the system package can use any MTS Explorer function. Users that are mapped to the Reader role can view all objects in the MTS Explorer hierarchy but cannot install, create, change, or delete any objects, shut down server processes, or export packages. If you map your Windows NT domain user name to the System Package Administrator role, you will be able to add, modify, or delete any package in the MTS Explorer. If MTS is installed on a server whose role is a primary or backup domain controller, a user must be a domain administrator in order to manage packages in the MTS Explorer.
You can also set up new roles for the system package. For example, you can configure a Developer role that allows users to install and run packages, but not delete or export them. The Windows NT user accounts or groups that you map to that role will be able to test installation of packages on that computer without having full administrative privileges over the computer.
In order to work with a remote computer, you first need to add it to the Computers folder in the MTS Explorer:
Click the Computers folder.
Select New -> Computer from the Action menu.
Enter name of the remote computer.
ImportantYou must be mapped to the Administrator role on the remote computer in order to access it from your machine. In addition, you cannot remotely administer MTS on a Windows 95 computer from MTS on a Windows NT server.
You should now see both My Computer and the new remote computer under the Computers folder. At this point, you can push and pull components between the two machines. Think of the Remote Components folder as its own special package. You are merely adding to it components that exist in one or more packages of remote machines.
The following example pulls a component from a remote machine to My Computer.
Click the Remote Components folder of My Computer.
Select New-> Remote Component from the Action menu to display the dialog box shown here.
Figure 6. Adding a component to Remote Components
In this example, we select (and add) a component called test6.foobar2 from a package called aa on the remote machine calvinh5. This package also has another component (Visual FoxPro OLEPUBLIC class) named test6.foobar, which we do not select. When we click OK, a copy of the DLL and the type library are copied to the local machine (My Computer) and stored in a subfolder of your MTS root location (in this case, c: C:Program FilesMtsRemoteaa). In addition, the server is now registered on your machine. Note that while the DLL is copied to your machine, the .dll registered in your registry points to the remote machine.
If you encounter problems after you click OK, you may not have proper access rights to copy the server components. Ensure that the remote machine is configured with proper access privileges for you. At this point, you can go into Visual FoxPro running on the local machine and access the server:
You use MTS Explorer to view the activated object in the remote machine folder under the package it is registered in. You will not see the object activity in the Remote Components folder. See the “Working with Remote MTS Computers” topic in the MTS Help file for more details.
Security
Security in MTS is handled by roles. Roles are established at the package level. Components within that package can set up role memberships. The following MTS Explorer image shows a package called Devcon1, which contains three roles. Only the last two components contain Role Memberships.
Figure 7. Package with roles
If you navigate the Roles folder, you can see all Windows NT users or groups assigned to that particular role.
To create a new role
Click the Roles folder.
Select New-> Role from the Action menu.
Enter a new role name in the dialog box.
You can add new users/groups to a particular role as follows:
To add new users or groups
Click the Users folder of the newly added role.
Select New-> User from the Action menu.
Select users/groups from the dialog box.
MTS handles its security several different ways. The MTS security model consists of declarative security and programmatic security. Developers can build both declarative and programmatic security into their components prior to deploying them on a Windows NT security domain.
You can administer package security using MTS Explorer. This form of declarative security, which does not require any component programming, is based on standard Windows NT security. This can be done by Package- or Component-level security.
Declarative Security
You can manage Declarative security at the package and at the component level through settings available in the Security tab of the Package Properties dialog box.
Package-level security
Each package has its own security access authorization, which can be set in the Package Properties dialog box.
Figure 8. Package properties
By default, the Security check box is not marked, so you need to check this box to enable security. If you do not enable security for the package, MTS will not check roles for the component. If security is enabled, you must also enable security at the component level in order to have roles checked.
Component-level security
Each installed component can also have its own security setting. You set security for a component through the same Enable authorization checking check box on the Property dialog box in MTS Explorer. If you are enabling security at both levels and you do have defined roles, you must include one of the roles in the component’s Role Membership folder. If you do not include a role in the folder, you will get an “Access is denied” error message when you try to access a property or method of the component. Of course, if you do not have any roles, you will get the same error.
Note You can still do aCreateObjecton the component, but that is all.
oContext = CreateObject("vfp_mts.mts1")
oContext.Hello() && will generate an Access is denied error
To restrict access to a specific component within a package, you must understand how components in the package call one another. If a component is directly called by a base client, MTS checks roles for the component. If one component calls another component in the same package, MTS does not check roles because components within the same package are assumed to “trust” one another.
When you change the security settings for a particular package or component, you need to shut down server processes before changes can take place. This option is available from the Action menu when Package is selected.
Programmatic Security
You can put code in your program to check for specific security access rights. The following three properties and methods from the Context object return information regarding security for that package or component.Expand table
Methods
Description
IsCallerInRole
Indicates whether the object’s direct caller is in a specified role (either directly or as part of a group).
IsSecurityEnabled
Indicates whether security is enabled. MTS security is enabled unless the object is running in the client’s process.
Security
Returns a reference to an object’s SecurityProperty object.
The following method checks whether the called object is in a particular role. The IsCallerInRole method is useful when the roles are defined, but if your code is generic and doesn’t know the particular roles associated with a component, you must handle this through your error routine.
PROCEDURE GetRole (tcRole)
LOCAL oMTX,oContext,lSecurity,cRole,lHasRole
IF EMPTY(tcRole)
RETURN "No Role"
ENDIF
oMtx = CREATEOBJECT(MTX_CLASS)
oContext = oMtx.GetObjectContext()
IF oContext.IsSecurityEnabled
THIS.SkipError=.T.
lHasRole = oContext.IsCallerInRole(tcRole)
THIS.SkipError=.F.
DO CASE
CASE THIS.HadError
THIS.HadError = .F.
cRole="Bad Role"
CASE lHasRole
cRole="Yep"
OTHERWISE
cRole="Nope"
ENDCASE
ELSE
cRole="No Security"
ENDIF
oContext.SetComplete()
RETURN cRole
ENDPROC
Advanced users can access the SecurityProperty object to obtain more details on the user for handling security. The Security object offers the following additional methods.Expand table
Method
Description
GetDirectCallerName
Retrieves the user name associated with the external process that called the currently executing method.
GetDirectCreatorName
Retrieves the user name associated with the external process that directly created the current object.
GetOriginalCallerName
Retrieves the user name associated with the base process that initiated the call sequence from which the current method was called.
GetOriginalCreatorName
Retrieves the user name associated with the base process that initiated the activity in which the current object is executing.
What type of security should you use? Programmatic security offers more power in terms of structuring specific functionality for particular roles. You can use Case statements, as in the previous example, which perform different tasks, depending on the role. Declarative security, on the other hand, can only control access at the component level (not method or lower).
Changes to Programmatic security, however, require a new build of the component, which may not always be convenient or realistic. Controlling Component-level security for users and roles by using MTS Explorer to turn security on or off gives an administrator greater control. The optimal solution is one with utilizes both declarative and programmatic securities in the most efficient manner.
Shared Property Manager
The Shared Property Manager (SPM) MTS resource dispenser allows you to create and share properties across components. Because it is a resource dispenser, all other components in the same package can share information, but information cannot be shared across different packages. For example, if you want to keep a counter to use for generating unique IDs for objects in a package, you could create a Counter property to hold the latest unique ID value. This property would be preserved while the package was active (regardless of object state).
The SPM also represents an excellent way for an object to preserve its state before being deactivated in a stateless mode (SetComplete). Just-In-Time activation does not affect or reset the state of SPM.
The following example shows how to use the SPM with Visual FoxPro servers:
#DEFINE MTX_CLASS "MTXAS.APPSERVER.1"
#DEFINE MTX_SHAREDPROPGRPMGR "MTxSpm.SharedPropertyGroupManager.1"
PROCEDURE GetCount (lReset)
LOCAL oCount
LOCAL oMTX,oContext
LOCAL nIsolationMode,nReleaseMode,lExists
oMtx = CREATEOBJECT(MTX_CLASS)
oContext = oMtx.GetObjectContext()
oSGM = oContext.CreateInstance(MTX_SHAREDPROPGRPMGR)
nIsolationMode = 0
nReleaseMode = 1
* Get group reference in which property is contained
oSG = oSGM.CreatePropertyGroup("CounterGroup", nIsolationMode,;
nReleaseMode, @lExists)
* Get object reference to shared property
oCount = oSG.CreateProperty("nCount", @lExists)
* check if property already exists otherwise reset
IF lReset OR !lExists
oCount.Value = 1
ELSE
oCount.Value = oCount.Value + 1
ENDIF
RETURN oCount.Value
ENDPROC
The following settings are available for Isolation and Release modes.
Isolation mode
LockSetGet 0 (default)—Locks a property during a Value call, assuring that every get or set operation on a shared property is atomic. This ensures that two clients can’t read or write to the same property at the same time, but doesn’t prevent other clients from concurrently accessing other properties in the same group.
LockMethod 1—Locks all of the properties in the shared property group for exclusive use by the caller as long as the caller’s current method is executing. This is the appropriate mode to use when there are interdependencies among properties or in cases where a client may have to update a property immediately after reading it before it can be accessed again.
Release mode
Standard 0 (default)—When all clients have released their references on the property group, the property group is automatically destroyed.
Process 1—The property group isn’t destroyed until the process in which it was created has terminated. You must still release all SharedPropertyGroup objects by setting them to Nothing.
MTS Support for Internet Information Server
MTS includes several special system packages for use with Microsoft Internet Information Server (IIS). The Windows NT Options Pack 4.0 integrates MTS and IIS more closely. In the future, you can expect even better integration to play a more central role in your Web applications.
IIS Support
Transactional Active Server Pages—You can now run Scripts in Active Server Pages (ASP) within an MTS-managed transaction. This extends the benefits of MTS transaction protection to the entire Web application.
Crash Protection for IIS Applications—IIS Web applications can now run within their own MTS package, providing process isolation and crash protection for Web applications.
Transactional Events—You can embed commands in scripts on ASP pages, enabling you to customize Web application response based on transaction results.
Object Context for IIS Built-In Objects—The MTS object context mechanism, which masks the complexity of tracking user state information from the application developer, now tracks state information managed by IIS built-in objects. This extends the simplicity of the MTS programming model to Web developers.
Common Installation and Management—MTS and IIS now share common installation and a common management console, lowering the complexity of deploying and managing business applications on the Web.
IIS System Packages
If you use MTS with Internet Information Server version 4.0, the Packages Installed folder contains the following IIS-specific system packages.
IIS in-process applications
The IIS In-Process Applications folder contains the components for each Internet Information Server application running in the IIS process. An IIS application can run in the IIS process or in a separate application process. If an IIS application is running in the IIS process, the IIS application will appear as a component in the IIS In-Process Applications folder. If the IIS application is running in an individual application process, the IIS application will appear as a separate package in the MTS Explorer hierarchy.
IIS utilities
The IIS Utilities Folder contains the ObjectContext component required to enable transactions in ASP pages. For more information about transactional ASP pages, refer to the Internet Information Server documentation.
Automating MTS Administration
Microsoft Transaction Server contains Automation objects that you can use to program administrative and deployment procedures, including:
Installing a prebuilt package.
Creating a new package and installing components.
Enumerating through installed packages to update properties.
Enumerating through installed packages to delete a package.
Enumerating through installed components to delete a component.
Accessing related collection names.
Accessing property information.
Configuring a role.
Exporting a package.
Configuring a client to use Remote Components.
You can use the following Admin objects in your Visual FoxPro code.Expand table
Object
Description
Catalog
The Catalog object enables you to connect to MTS Catalog and Access collections.
CatalogObject
The CatalogObject object allows you to get and set object properties.
CatalogCollection
Use the CatalogCollection object to enumerate, add, delete, and modify Catalog objects and to access related collections.
PackageUtil
The PackageUtil object enables installing and exporting a package. Instantiate this object by calling GetUtilInterface on a Packages collection.
ComponentUtil
Call the ComponentUtil object to install a component in a specific collection and import components registered as in-process servers. Create this object by calling GetUtilInterface on a ComponentsInPackage collection.
RemoteComponentUtil
Using the RemoteComponentUtil object, you can program your application to pull remote components from a package on a remote server. Instantiate this object by calling GetUtilInterface on a RemoteComponents collection.
RoleAssociationUtil
Call methods on the RoleAssociationUtil object to associate roles with a component or interface. Create this object by calling the GetUtilInterface method on a RolesForPackageComponent or RolesForPackageComponentInterface collection.
In addition, the following collections are also supported.Expand table
Collection
LocalComputer
ComputerList
Packages
ComponentsInPackage
RemoteComponents
InterfacesForComponent
InterfacesForRemoteComponent
RolesForPackageComponent
RolesForPackageComponentInterface
MethodsForInterface
RolesInPackage
UsersInRole
ErrorInfo
PropertyInfo
RelatedCollectionInfo
If you want to get a reference to a particular collection, use the GetCollection method. The following example shows, first, getting the collection of packages and, second, getting a collection of all components in the first package:
NoteThe GetCollection method merely returns an object reference to an empty collection. You need to explicitly call the Populate method to fill the collection.
Collections are case sensitive, as in the following example code:
Visual FoxPro 6.0 is ideally suited for using MTS Automation because of the new Project Manager and Application Builder hooks support.
Using Visual FoxPro 6.0 Project Hooks
The MTS samples posted along with this document contain a special Project Hook class designed specially for MTS. This class automatically shuts down and refreshes MTS registered servers contained in that project. One of the issues that developers must consider when coding and testing servers under MTS is repeatedly opening the MTS Explorer to manually shut down processes so that servers can be rebuilt and overwritten. Using a Project Hook nicely automates this process. Here is sample code from the BeforeBuild event, which iterates through the Packages collection shutting-down processes.
* BeforeBuild event
LPARAMETERS cOutputName, nBuildAction, lRebuildAll, lShowErrors, lBuildNewGuids
#DEFINE MTS_CATALOG "MTSAdmin.Catalog.1"
#DEFINE MSG_MTSCHECK_LOC "Shutting down MTS servers...."
LOCAL oCatalog,oPackages,oUtil,i,j,oComps
LOCAL oProject,lnServers,laProgIds,lcSaveExact
THIS.lBuildNewGuids = lBuildNewGuids
oProject = _VFP.ActiveProject
lnServers = oProject.servers.count
DIMENSION THIS.aServerInfo[1]
STORE "" TO THIS.aServerInfo
IF lnServers = 0 OR nBuildAction # 4
RETURN
ENDIF
WAIT WINDOW MSG_MTSCHECK_LOC NOWAIT
DIMENSION laProgIds[lnServers,3]
FOR i = 1 TO lnServers
laProgIds[m.i,1] = oProject.servers[m.i].progID
laProgIds[m.i,2] = oProject.servers[m.i].CLSID
laProgIds[m.i,3] = THIS.GetLocalServer(laProgIds[m.i,2])
ENDFOR
ACOPY(laProgIds,THIS.aServerInfo)
* Shutdown servers
oCatalog = CreateObject(MTS_CATALOG)
oPackages = oCatalog.GetCollection("Packages")
oUtil = oPackages.GetUtilInterface
oPackages.Populate()
lcSaveExact = SET("EXACT")
SET EXACT ON
FOR i = 0 TO oPackages.Count - 1
oComps = oPackages.GetCollection("ComponentsInPackage",;
oPackages.Item(m.i).Key)
oComps.Populate()
FOR j = 0 TO oComps.Count-1
IF ASCAN(laProgIds,oComps.Item(m.j).Value("ProgID")) # 0
oUtil.ShutdownPackage(oPackages.Item(m.i).Value("ID"))
EXIT
ENDIF
ENDFOR
ENDFOR
WAIT CLEAR
SET EXACT &lcSaveExact
* User is building new GUIDs, so packages
* need to be reinstalled manually
IF lBuildNewGuids
RETURN
ENDIF
This is only one of the many possibilities provided by a Visual FoxPro Project Hook. The MTS Admin objects can save a great deal of time you normally would spend manually setting options in the MTS Explorer.
Using Visual FoxPro 6.0 Application Builders
As with the Project Hooks, you might also want to create an Application (Project) Builder that handles registration of Visual FoxPro Servers in MTS packages. The Visual FoxPro MTS samples include such a builder. (See the Readme file in the mtsvfpsample sample application for more details on setup and usage of these files.)
This Builder simply enumerates through all the servers in your Visual FoxPro project and all the available MTS packages. You can then select (or create) a particular package and registered server to install in that package. Additionally, you can set the Transaction property for each component. The Visual FoxPro code called when the user clicks OK is as follows:
#DEFINE MTS_CATALOG "MTSAdmin.Catalog.1"
#DEFINE ERR_NOACTION_LOC "No action taken."
LOCAL oCatalog,oPackages,oUtil,i,j,oComps,nPoslcPackage
LOCAL lPackageExists,oCompRef
LOCAL oProject,lnServers,laProgIds,lcSaveExact,oPackageRef,lctrans
lcPackage = ALLTRIM(THIS.cboPackages.DisplayValue)
lPackageExists = .f.
SELECT mtssvrs
LOCATE FOR include
IF !FOUND() OR EMPTY(lcPackage)
MESSAGEBOX(ERR_NOACTION_LOC)
RETURN
ENDIF
THIS.Hide
oCatalog = CreateObject(MTS_CATALOG)
oPackages = oCatalog.GetCollection("Packages")
oPackages.Populate()
FOR i = 0 TO oPackages.Count-1
IF UPPER(oPackages.Item(m.i).Name) == UPPER(lcPackage)
oPackageRef = oPackages.Item(m.i)
lPackageExists=.T.
EXIT
ENDIF
ENDFOR
IF !lPackageExists &&creating new package
oPackageRef = oPackages.Add
oPackageRef.Value("Name") = lcPackage
oPackages.SaveChanges
ENDIF
oComps = oPackages.GetCollection("ComponentsInPackage",;
oPackageRef.Key)
oUtil = oComps.GetUtilInterface
SCAN FOR include
oUtil.ImportComponentByName(ALLTRIM(progid))
ENDSCAN
oPackages.SaveChanges()
oComps.Populate()
SCAN FOR include
DO CASE
CASE trans = 1
lctrans = "Supported"
CASE trans = 2
lctrans = "Required"
CASE trans = 3
lctrans = "Requires New"
OTHERWISE
lctrans = "Not Supported"
ENDCASE
FOR j = 0 TO oComps.Count-1
IF oComps.Item(m.j).Value("ProgID")=ALLTRIM(progid)
oCompRef = oComps.Item(m.j)
oCompRef.Value("Transaction") = lctrans
oCompRef.Value("SecurityEnabled") = ;
IIF(THIS.chkSecurity.Value,"Y","N")
ENDIF
ENDFOR
ENDSCAN
oComps.SaveChanges()
oPackages.SaveChanges()
Tips and Tricks
Hopefully, this article offers enough insight into creating Visual FoxPro components that work well with your three-tier MTS applications. Here are a few final tips to consider:
Design your components with MTS in mind from the start.
Components must be in-process DLLs. Do not use Visual FoxPro EXE servers.
When adding Visual FoxPro components, make sure to select both .dll and .tlb files.
In the Project Info dialog box of Visual FoxPro DLL servers, set Instancing to MultiUse.
Don’t be afraid to mix with other components (for example, Visual Basic servers).
You must have DTC running for transaction support.
Call SetComplete regardless of whether you’re using transactions, because it places objects in stateless mode.
Your MTS object has an associated Context object. Do not place this code in the base client.
Connections must have DispLogin set to Never; for SQL pass-through, use SQLSetProp(0).
Minimize the number of PEMs on an object (protect your PEMs).
Because of page locking issues, limit the length of time you leave SQL Server 6.5 transactions uncommitted.
To use security, you must have a valid role associated with the component.
Avoid using CreateInstance on non-MTS components.
Do not pass object references of the Context object outside of the object itself.
Consider using disconnected ADO recordsets to move data between tiers.
You can pass Visual FoxPro data in strings, arrays, or ADO recordsets.
Passing Parameters:
Be careful when passing parameters.
Always use SafeArray when passing object references.
Passing by value:- Fastest and most efficient- Copies the parameters into a buffer- Sends all values at once
Passing by reference:- Sends a reference, but leaves the object back in the client.- Accessing the parameter scampers back to the client machine.
Always read the Late Breaking News! It contains important information such as Security configuration details.
By default, MTS will create a maximum of 100 apartment threads for client work (per package). In Windows NT 4.0 Service Pack 4 (and later), you can tune the MTS activity thread pool. This will not affect the number of objects than can be created. It will simply configure the number that can be simultaneously in call. To tune the MTS activity thread pool:
Open your Windows Registry using RegEdit and go to the package key:HKLM/Software/Microsoft/Transaction Server/Package/{your package GUID}
Add a REG_DWORD named value:ThreadPoolMax
Enter a value for ThreadPoolMax. Valid values are:0 to 0x7FFFFFFF
Summary: Describes how the Microsoft® Visual FoxPro® version 6.0 Application Framework, including the Application Wizard and Application Builder, can be used by the beginning developer to turn out polished applications and customized by the more experienced developer to create more detailed applications. (32 printed pages)
Contents
OverviewExamining Framework ComponentsDesignating the Classes You WantSpecifying Your Own Framework ComponentsA Closer Look at the Standard Application WizardA New Application WizardA Few Parting Thoughts about Team PracticesAppendix 1Appendix 2Appendix 3Appendix 4Expand table
Click to copy the appfrmwk sample application discussed in this article.
Overview
The Visual FoxPro 6.0 Application Framework offers a rapid development path for people with little experience in Visual FoxPro. With a few simple choices in the Application Wizard and the Application Builder, beginning developers can turn out polished and practical applications.
Under the hood, however, the framework offers experienced developers and teams much more. This article shows you how to adapt the framework components so they fit your established Visual FoxPro requirements and practices.
In the first section of this article you’ll learn about the files and components that support the framework and how they work together while you develop an application. This information is critical to moving beyond simply generating framework applications to experimenting with framework enhancements.
The second section teaches you how to apply your experiences with the framework to multiple applications. After you’ve experimented with framework enhancements for a while, you will want to integrate your changes with the framework, for standard use by your development team. By customizing the files the Application Wizard uses to generate your application, you’ll make your revisions accessible to team members—without sacrificing the framework’s characteristic ease of use.
Examining Framework Components
This section shows where the framework gets its features and components, and how these application elements are automatically adjusted during your development process.
Once you see how and where framework information is stored, you can begin to try different variations by editing the versions generated for a framework application. When you’re satisfied with your changes, you can use the techniques in the next section to migrate them to your team’s versions of the framework components.
NoteLike most Visual FoxPro application development systems, the framework is composed of both object-oriented programming (OOP) class components and non-OOP files. This distinction is important because you adapt these two types of components in different ways; classes can be subclassed, while non-OOP files must be included as is or copied and pasted to get new versions for each application. The framework is minimally dependent on non-OOP files, as you’ll see here, but these files still exist.
Throughout this article we’ll refer to the non-OOP framework files as templates, to distinguish these components from true classes.
Framework Classes
The Visual FoxPro 6.0 framework classes are of two types:
Framework-specific classes. These classes have been written especially for the application framework and provide functionality specific to the framework. The standard versions of these classes are in the HOME( )+ Wizards folder, in the _FRAMEWK.VCX class library.
Generic components. These features come from class libraries in the HOME( )+ FFC (Visual FoxPro Foundation Classes) folder.
_FRAMEWK.VCX
The _FRAMEWK.VCX class library (see Figure 1) contains all the classes written specifically to support the framework. Each framework application you create has an application-specific VCX containing subclasses of the _FRAMEWK.VCX components. The Application Wizard puts these subclasses in a class library named <Your projectname> plus a suffix to designate this library as one of the wizard-generated files. To distinguish these generated, empty subclasses, it adds a special prefix to the class names as well.
Figure 1. _FRAMEWK.VCX framework-specific class library, as viewed in Class Browser, is found in the HOME( )+ Wizards folder.
Framework superclass: _Application
The _Application class is a required ancestor class, which means that this class or a subclass of this class is always required by the framework. This class provides application-wide manager services. For example, it manages a collection of modeless forms the user has opened.
You designate a subclass of _Application simply by using CREATEOBJECT( ) or NEWOBJECT( ) to instantiate the subclass of your choice. (By default, the framework provides a main program to do this, but this PRG contains no required code.) When your designated _Application subclass has instantiated successfully, you call this object’s Show( ) method to start running the application.
NoteIn this article, we’ll refer to the object you instantiate from a subclass of _Application as the application object. We’ll continue to refer to “your subclass of _Application” to mean the class definition instantiating this object, which will be in a VCX belonging to your application (not _FRAMEWK.VCX). You’ll also see references to “_Application“, that refer specifically to code and properties you’ll find in the superclass located in _FRAMEWK.VCX.
At run time, the application object instantiates other objects as necessary to fill all the roles represented by the other classes in _FRAMEWK.VCX except _Splash. The framework identifies these roles as important to various application functions, but, as you’ll see in this section, you have full control over how the roles are carried out.
NoteThe _Splash class is an anomaly in _FRAMEWK.VCX; it isn’t instantiated or used by the framework application directly. (If it were instantiated by the application object, your splash screen would appear too late to be useful.) Instead, _Splash merely provides a default splash screen with some of the same attributes as _Application (for example, your application name and copyright). The Application Builder transfers these attributes to your application’s subclass of _Splash at the same time it gives them to your application’s subclass of _Application, so they stay synchronized. The default main program delivered with a framework gives you one way to instantiate this splash screen before you instantiate your application object.
You certainly don’t need to use the method shown in the default main program for your splash screen. In fact, many applications do not need a splash screen at all. For those that do, you may prefer to use the Visual FoxPro –b<file name> command-line switch, which displays a bitmap of your choice during startup, rather than a Visual FoxPro form of any description.
Framework superclass: _FormMediator
You’ll grasp most of the “roles” played by the subsidiary classes in _FRAMEWK.VCX easily, by reading their class names and descriptions. (If you can’t read the full class description when you examine _FRAMEWK.VCX classes in a project, try using the Class Browser.) However, you’ll notice a _FormMediator class whose purpose takes a little more explaining.
You add an object descended from the _FormMediator custom class to any form or form class, to enable the form to communicate efficiently with the application object. This section will show you several reasons the form might want to use services of the application object. With a mediator, your form classes have access to these services, but the forms themselves remain free of complex framework-referencing code.
The _FormMediator class is low-impact. It doesn’t use a lot of resources, and its presence will not prevent your forms from being used outside a framework application. Using this strategy, the framework can manage any forms or form classes your team prefers to use, without expecting them to have any special inheritance or features.
Like _Application, _FormMediator class is a required ancestor class. You can create other mediator classes, as you can subclass _Application to suit your needs, but your mediators must descend from this ancestor.
We’ll refer to _FormMediator and its descendents as the mediator object, because (strictly speaking) your forms will see it as the “application mediator” while the application object treats it as a “form mediator.”
The Visual FoxPro 6.0 Form Wizards create forms designed to take advantage of mediators when the framework is available. You can see some simple examples of mediator use in the baseform class of HOME( )+ WizardsWIZBASE.VCX.
Examine _FormMediator‘s properties and methods, and you’ll see that you can do much more with the mediator in your own form classes. For example, the application object calls mediator methods and examines mediator properties during its DoTableOutput( ) method. (This method allows quick output based on tables in the current data session.) Your mediator for a specific form could:
SELECT a particular alias to be the focus of the output.
Prepare a query specifically for output purposes (and dispose of it after the output).
Inform the application object of specific classes and styles to be used by _GENHTML for this form.
Change the output dialog box caption to suit this form.
The mediator also has methods and properties designed to specify context menus for the use of a particular form. If the application object receives this information from the mediator, it handles the management of this menu (sharing it between forms as necessary).
You’ll find one example of mediator use in the ErrorLogViewer class. (This use is described in Appendix 1, which covers the options system.) A full discussion of the _FormMediator class is beyond the scope of this document. The more information you give a mediator or mediator subclass, however, the more fully your forms can use framework’s features, without making any significant changes to the forms themselves.
NoteThe _Application class includes a property, lEnableFormsAtRuntime (defaulting to .T.), which causes the application object to add mediators at run time to any form not having a mediator of its own. You can specify the mediator subclass that the application adds to a form at run time. Keep in mind, however, that mediators added at design time will have a more complete relationship with their form containers, because these forms can include code referencing their mediator members. During a form’s QueryUnload event, for example, the form can use the mediator to determine whether the form contains any unconfirmed changes. Without code in the form’s QueryUnload method, the mediator can’t intercede at this critical point.
Additional _FRAMEWK.VCX classes
The other classes in _FRAMEWK.VCX are all dialog box and toolbar classes to perform common functions within an application. None of these classes are required ancestors; you can substitute your own user interfaces and class hierarchies for these defaults at will. Two of them (_Dialog and _DocumentPicker) are abstract; that is, they are never instantiated directly, existing only to provide properties and methods to their descendent classes. Others will not instantiate unless you pick specific application characteristics. For example, if you don’t write “top form” applications (MDI applications in their own frames) you will never use _TopForm, the _FRAMEWK.VCX class that provides the MDI frame window object.
Once you have examined these classes, and identified their roles, you will know which ones supply the types of services you need in applications you write—and, of these, you will identify the ones you wish to change.
Designating the Classes You Want
For each class role identified by the framework, the application object uses corresponding xxxClass and xxxClassLib properties to determine the classes you want. To change which class is instantiated for each role, you change the contents of these properties in your subclass of _Application.
For example, _Application has cAboutBoxClass and cAboutBoxClassLib properties, and it uses these properties to decide what dialog box to show in its DoAboutBox( ) method (see Figure 2).
Figure 2. Class and ClassLib property pairs in the _Application object
If you fill out a class property but omit the matching Classlib property, _Application assumes that your designated class is in the same library as the _Application subclass you instantiated. If your _Application subclass is in the MyApplication.vcx and cAboutBoxClass has the value “MyAboutBox” but cAboutBoxClassLib is empty, a call to the Application object’s DoAboutBox( ) method instantiates a class called MyAboutBox in MyApplication.vcx.
If you call the method instantiating one of the subsidiary classes when the matching class property is empty, _Application attempts to provide appropriate behavior to the specific situation. For example, if the cAboutBoxClass property is empty, DoAboutBox( ) will simply do nothing, because it has no alternative. By contrast, if the cErrorViewerClass property is empty, the _Application DisplayErrorLog( ) method will ask its cusError member object to use its default error log display instead.
Except for the cMediatorClass and cMediatorClassLib properties, which must specify a class descending from _FormMediator in _FRAMEWK.VCX, remember that there are no restrictions on these dialog boxes and toolbars. You don’t have to subclass them from the classes in _FRAMEWK.VCX, or even follow their examples, in your own classes fulfilling these framework roles.
Even when you design completely different classes, you will still benefit from investigating the defaults in _FRAMEWK.VCX, to see how they take advantage of their relationship with the framework. For example, all the classes descended from _Dialog have an ApplyAppAttributes( ) method. When the framework instantiates these classes, it checks for the existence of this method. If the ApplyAppAttributes( ) method exists, the application object passes a reference to itself to the form, using this method, before it calls the Show( ) method. In this way, the dialog box can derive any framework-specific information it needs before it becomes visible. For instance, the About Box dialog box might adjust its caption using the _Application.cCaption property.
If the ApplyAppAttributes( ) method does not exist in yourcAboutBoxClass class, no harm is done. The _Application code still tries to harmonize your dialog box with its interface, in a limited way, by checking to see whether you’ve assigned any custom value to its Icon property. If you haven’t, _Application assigns the value in its cIcon property to your dialog box’s icon before calling its Show( ) method.
NoteThis strategy typifies the framework’s general behavior and goals:
It tries to make the best use of whatever material you include in the application.
When possible, it does not make restrictive assumptions about the nature of this material.
It avoids overriding any non-default behavior you may have specified.
Investigating the default _Options dialog box class and _UserLogin default dialog boxes will also give you insight into the _Application options and user systems. While the dialog boxes themselves are not required, you will want to see how they interact with appropriate _Application properties and methods, so your own dialog boxes can take advantage of these framework features. In particular, the _Application options system has certain required elements, detailed in Appendix 1.
FoxPro Foundation Generic Classes
You may be surprised that _FRAMEWK.VCX contains only two required classes (the application and mediator objects), and in fact even when you add the other subsidiary classes, _FRAMEWK.VCX doesn’t contain much of the functionality you may expect in a Visual FoxPro application. You will not find code to perform table handling. You won’t find dialog boxes filling standard Visual FoxPro roles, such as a dialog box to select report destinations. You won’t find extensive error-handling code.
_FRAMEWK.VCX doesn’t include this functionality because there is nothing framework-specific about these requirements. Instead, it makes use of several Visual FoxPro Foundation Classes libraries, useful to any framework or application, to perform these generic functions. The _Application superclass contains several members descending from FFC classes, and it instantiates objects from other FFC classes at run time as necessary. Then it wraps these objects, setting some of their properties and adding some specific code and behavior to make these instances of the FFC classes especially useful to the framework.
For example, _Application relies on its cusError member, descended from the _Error object in FFC_APP.VCX, to do most of its error handling, and to create an error log. However, as mentioned earlier, _Application code displays the error log using a framework-specific dialog box. The application object also sets the name and location of the error log table to match its own needs, rather than accepting _Error‘s default.
The framework uses four FFC class libraries: _APP.VCX, _TABLE.VCX, _UI.VCX, and _REPORTS.VCX. Figure 3 shows these libraries in Class Browser views, as well as in a Classes tab for a framework application project.
Figure 3. A framework application uses generic Visual FoxPro Foundation Classes, from HOME( )+ FFC folder, to supplement the framework-specific classes in _FRAMEWK.VCX.
Unlike the subsidiary classes in _FRAMEWK.VCX, the FFC classes and their complex attributes are used directly by _Application, so you don’t specify alternative classes or class libraries for these objects. You can still specify your own copies of these class libraries, as you’ll see in the next section.
If you examine the Project tab in Figure 3, or the project for any framework application, you’ll find this list of libraries built in. You’ll see _FRAMEWK.VCX, and there will be at least one class library containing the subclasses of _FRAMEWK.VCX for this application.
You’ll see one more FFC library: _BASE.VCX, which contains the classes on which _FRAMEWK.VCX and all the FFC libraries are based. Your framework project must have access to a library called _BASE, containing all the classes found in _BASE. However, neither the framework nor the four FFC class libraries it uses require any specific behavior or attributes from these classes. You are free to create an entirely different _BASE.VCX with classes of the same name, perhaps descending from your team’s standard base library.
Framework Templates
The framework templates are of three types:
Menu templates, a collection of Visual FoxPro menu definition files (.mnx and .mnt extensions)
Metatable, an empty copy of the table the framework uses to store information about the documents (forms, reports, and labels) you use in your application
Text, a collection of ASCII supporting files
Unlike the .vcx files used by the framework, Visual FoxPro doesn’t deliver separate versions of these templates on disk. Because the templates are copied, rather than subclassed, for framework applications, the templates don’t need to be available to your project as separate files. Instead, these items are packed into a table, _FRAMEWK.DBF, found in the HOME( )+ Wizards folder. The Application Wizard unpacks the files when it generates your new application (see Figure 4).
Figure 4. The Application Wizard copies template files from this _FRAMEWK.DBF table in HOME( )+ Wizards folder.
Because the files don’t exist on disk, their template file names are largely irrelevant, except to the Application Wizard. Although we’ll use the template names here, keep in mind that their copies receive new names when the Wizard generates your application.
Menu Templates
Just as the framework identifies “dialog box roles” and supplies sample dialog boxes to fill those roles, it identifies some “menu roles,” and comes equipped with standard menus to meet these requirements. The roles are startup (the main menu for your application) and navigation (a context menu for those forms you identify as needing navigation on the menu).
There are three template startup menus, each corresponding to one of the three application types described by the Application Builder as normal, top form, and module. T_MAIN.MNX, is a standard “replace-style” Visual FoxPro menu. It’s used for normal-style applications, which take over the Visual FoxPro environment and replace _MSYSMENU with their own menu. T_TOP.MNX, for top form applications, looks identical to T_MAIN.MNX, but has some code changes important to a menu in an MDI frame. T_APPEND.MNX is an “append-style” menu, characteristic of modules, which are applications that add to the current environment rather than controlling it.
There is one navigation menu template, T_GO.MNX. Its options correspond to the options available on the standard navigation toolbar (_NavToolbar in _FRAMEWK.VCX).
NoteBecause both T_GO.MNX and T_APPEND.MNX are “append-style” menus, they can exist as part of either _MSYSMENU or your top form menu. The Application Builder synchronizes your copy of T_GO.MNX to work with your normal- or topform-type application. However, if you change your application type manually rather than through the Application Builder, or if you want a module-type application that adds to an application in a top form, you may need to tell these menus which environment will hold them.
You make this change in the General Options dialog box of the Menu Designer (select or clear the Top-Level Form check box). If you prefer, you can adjust the ObjType of the first record in the MNX programmatically, as the Application Builder does. See the UpdateMenu( ) method in HOME( )+ WizardsAPPBLDR.SCX for details.
Like the document and toolbar classes in _FRAMEWK.VCX, the menu templates are not required. They simply provide good examples, and should give you a good start on learning how to use menus in a framework application.
In particular, you’ll notice that the menus do not call procedural code directly, only application object methods. This practice ensures that the code is properly scoped, regardless of whether the MPR is built into an app, or whether the .app or .exe holding the MPR is still in scope when the menu option runs.
Because Visual FoxPro menus are not object-oriented, they can’t easily hold a reference to the application object. To invoke application object methods, the menus use the object’s global public reference. This reference is #DEFINEd as APP_GLOBAL, in an application-specific header file, like this:
#DEFINE APP_GLOBAL goApp
Here is an example menu command using the #DEFINEd constant (the Close option on the File menu):
Each template menu header #INCLUDEs this header file. You can change the #DEFINE and recompile, and your menus will recognize the new application reference.
NoteThe application object can manage this public reference on its own (you don’t need to declare or release it). It knows which variable name to use by consulting its cReference property, which holds this name as a string. You can either assign the value in the program that instantiates your application object (as shown in the default main program) or you can assign this string to the cReference property of your _Application subclass at design time.
The template menus are the only part of the framework using this global reference. If you wish, your forms and other objects can use the reference, too, but there are rarely good reasons to do this. Before you opt to use the global reference, think about ways you might pass and store a reference to the application object in your forms instead. If your forms have mediator objects, they have a built-in method to receive this reference any time they need it.
Metatable Template
_FRAMEWK.DBF contains records for T_META.DBF/FPT/CDX, the table holding information about documents for your application. Records in this table indicate whether a document should be treated as a “form” or “report”—and you can create other document types on your own.
The document type designation is used by the framework dialog boxes descending from _DocumentPicker, to determine which documents are displayed to the user at run time. For example, the _ReportPicker dialog box will not display documents of “form” type, but the _FavoritePicker dialog box displays both forms and reports.
However, document type as specified in the metatable does not dictate file type. A “report” type document might be a PRG, which called a query dialog box and then ran a report based on the results.
The Application Builder creates and edits metatable records when you use the Builder to add forms and documents to the application. If you manually add a form or document to a framework project, the Project Hook object invokes the Builder to ask you for details about this document and fill out the metatable accordingly. Of course, you can also add records to the metatable manually.
The Application Builder and the _FRAMEWK.VCX dialog boxes descending from _DocumentPicker rely on the default structure of this metatable. (You’ll find its structure detailed in**Appendix 2.) The dialog boxes derive from this table the information they need to invoke each type of document, including the options you’ve set in the Application Builder for each document. (Appendix 3 gives you a full list of _DocumentPicker subclasses and their assigned roles.)
Just as you don’t have to use the _DocumentPicker dialog boxes, you don’t have to use the default metatable structure in a framework application. If you like the idea of the table, you could design a different structure and use it with dialog boxes with different logic to call the _Application methods that start forms and reports.
NoteIf you design a metatable with a different structure from the default, the application object can still take care of it for you. On startup, the metatable is validated for availability and appropriate structure. Once the metatable is validated, the application object holds the metatable name and location so this information is available to your application elements later, even though the application object makes no use of the metatable directly.
Edit your _Application subclass’s ValidateMetatable( ) method to reflect your metatable structure if it differs from the default. No other changes to the standard _Application behavior should be necessary to accommodate your metatable strategy.
You can also dispense entirely with a metatable in a framework application. No part of the framework, except the_DocumentPickerdialog boxes, expects the metatable to be present.
For instance, you might have no need for the dialog boxes or data-driven document access in a simple application. In this case, you can eliminate the metatable and invoke all your reports and forms directly from menu options. Simply provide method calls such as APP_GLOBAL.DoForm( ) and APP_GLOBAL.DoReport( ) as menu bar options. Fill out the arguments in these methods directly in the command code for each menu option, according to the requirements of each form and report.
Additional Text Templates
_FRAMEWK.DBF holds copies of some additional text files copied for your application’s use.
T_START.PRG is the template for the program that initializes your application object and shows the splash screen. Its behavior is well documented in comments you’ll find in the application-specific header file, described later. In addition, as just mentioned, it is not necessary. The program that creates your application object does not have to be the main program for your application, nor does it have to do any of the things that T_START.PRG does.
For example, suppose your application is a “module type,” handling a particular type of chore for a larger application. Because it is a module, it does not issue a READ EVENTS line or disturb your larger application’s environment. It may or may not need to use the framework’s user log on capabilities; you may have set up a user logging system in the outer program. The outer application may be a framework application, or it may not. All these things will help you decide what kind of startup code you need for this application object.
Let’s look at some sample code you might want to use for an accounting application. This .exe file is not a framework application, but it has a framework module added to it, which performs supervisor-level actions. Only some users are allowed to have access to this module. When your accounting application starts up, it may have an application manager object of its own, which performs its own login procedures. The method that decides whether to instantiate the framework module might look like this:
IF THIS.UserIsSupervisor( )
THIS.oSupervisorModule = ;
NEWOBJECT(THIS.cMyFrameworkModuleSupervisorClass,;
THIS.cMySupervisorAppClassLib)
IF VARTYPE(THIS.oSupervisorModule) = "O"
* success
ELSE
* failure
ENDIF
ELSE
IF VARTYPE(THIS.oSupervisorModule) = "O"
* previous user was a supervisor
THIS.oSupervisorModule.Release()
ENDIF
ENDIF
This code does not handle the public reference variable, a splash screen, or any of the other items in T_START.PRG.
You may not need the public reference variable at all because, in this example, your framework application is securely scoped to your larger application manager object. However, if your module application has menus that use the global reference to invoke your application object, you might assign the correct variable name to THIS.oSupervisorModule.cReference just above the first ELSE statement in the preceding sample code (where you see the “* success” comment). This is the strategy you see in T_START.PRG.
NoteIf many different outer applications will use this module, you will prefer to assign the appropriate cReference string in the class, rather than in this method (so you only need to do it once). You can assign this value to cReference either in the Properties window or in code during startup procedures for the application object. Either way, an assign method on the cReference property in _Application does the rest.
T_META.H is the template name for the application-specific header file, just mentioned in the section on menu templates. Only the menus and T_START.PRG use this header file, so it is up to you whether you use it, and how you use it. In the preceding example, you might not use it at all, or you might use only its APP_GLOBAL define to set the application object’s global reference.
The framework uses a few more text templates:
T_CONFIG.FPWNot surprisingly, provides a template for the config.fpw generated for your application. The template version gives new Visual FoxPro developers some ideas about what the config.fpw is for (it’s mostly comments); you will almost certainly wish to edit this file to meet your own standards.
T_LOG.TXTProvides a startup file for the “action log” the Project Hook will write during the life of your application to let you know what changes it has made to your application while you worked with the project.
T_HEAD.TXTProvides a standard header that the Application Wizard uses when generating your application-specific copies of framework templates. You might want to revise T_HEAD.TXT to include your own copyright notices, especially after you’ve edited the rest of the templates.
Specifying Your Own Framework Components
If you’ve done any development at all, you’ve undoubtedly experienced moments in which you identify something you wish to abstract from the process of developing a single application. You’ve done it too many times, you know how to do it, and now it’s time you figure out the best way to do it—so you never have to do it again.
In OOP terms, this is the time to develop a superclass to handle this function, so you can reuse its features. In template terms, this is the time to edit the template you copy for each application’s use. In the Visual FoxPro 6.0 application framework’s mixed environment, as you know, we have both types of components.
We’ll quickly review how these components are managed automatically by the Application Wizard and Builder during your development cycle. Then we’ll turn our attention to how you integrate your own superclasses and edited templates into this system.
Framework Components During Your Application Lifecycle
When you choose to create a new framework application, the Application Wizard takes your choices for a location and project name and generates a project file. If you select the Create project directory structure check box, the Application Wizard also creates a directory tree under the project directory. It adds _FRAMEWK.VCX and the required foundation class libraries to this project. It also adds a class library with appropriate application-specific subclasses of _FRAMEWK.VCX.
The Application Wizard then adds template-generated, application-specific versions of all the non-OOP components the application needs. As you probably realize, the Application Wizard copies these files out of the memo fields in _FRAMEWK.DBF.
_FRAMEWK.DBF contains two more records we haven’t mentioned yet: T_META.VCX and T_META.VCT. These records hold straight subclasses of the classes in _FRAMEWK.VCX, and they are copied out to disk to provide your application-specific class library.
NoteT_META.VCX is not a template. It is just a convenient way for the Application Wizard to hold these subclasses, and is not part of your classes’ inheritance tree. Your subclasses descend directly from _FRAMEWK.VCX when the Application Wizard creates them, and thereafter will inherit directly from _FRAMEWK.VCX.
Once your new framework project exists, the Application Wizard builds it for the first time. It also associates this project with a special Project Hook object, designed to invoke the Application Builder. The Application Wizard shows you the new project and invokes the Application Builder.
At this point, the Application Builder takes over. The Application Builder provides an interface you can use to customize the framework aspects of any framework-enabled project, throughout the life of the project.
You can use the Application Builder to customize various cosmetic features of the application object, such as its icon. When you make these choices, the Application Builder stores them in the appropriate properties of your _Application subclass. (In some cases, it also stores them in the matching _Splash subclass properties.)
In addition, the Application Builder gives you a chance to identify data sources, forms, and reports you’d like to associate with this project. It gives you convenient access to the data, form, and report wizards as you work, in case you want to generate new data structures and documents. For inexperienced developers, the Application Builder provides a visual way to associate data structures directly with forms and reports, by providing options to invoke report and form wizards each time you add a new data source.
Whether you choose to generate reports and forms using the wizards or to create your own, the Application Builder and its associated Project Hook object help you make decisions about framework-specific use of these documents. (Should a report show up in the Report Picker dialog box, or is it only for internal use? Should a form have a navigation toolbar?) It stores these decisions in your framework metatable.
As you think about these automated elements of a framework development cycle, you’ll see a clear difference between the changes you can effect if you change the Application Wizard, or generation process, and the changes you can effect by editing the Application Builder and Project Hook. The files provided by the Wizard, in advance of development, represent your standard method of development. The changes made thereafter, through the Builder and Project Hook, represent customization you can do for this single application.
The balance of this article concentrates on enhancing the Wizard to provide the appropriate framework components when you begin a new application. Once you have established how you want to enhance the startup components, you will think of many ways you can change the Builder and the Project Hook, to take advantage of your components’ special features, during the rest of the development cycle.
NoteAn important change in versions after Visual FoxPro 6.0 makes it easy for you to customize the Application Builder to match your style of framework use. Rather than directly invoking the default appbldr.scx, the default Application Builder in later versions is a PRG.
The PRG makes some critical evaluations before it displays a Builder interface. For example, it checks to see whether the project has an associated Project Hook object, and whether this Project Hook object specifies a builder in its cBuilder property. See HOME( )+ WizardsAPPBLDR.PRG for details. You will find it easy to adopt this strategy, or to edit appbldr.prg to meet your own needs for displaying the Builder interface of your choice.
A preview version of appbldr.prg is included with the source for this article. See appbldr.txt for instructions on making this new Application Builder available automatically from the VFP interface, similar to the new wizard components delivered as part of the document.
A Closer Look at the Standard Application Wizard
You’ll find the Visual FoxPro 6.0 Application Wizard files in your HOME( )+ Wizards folder. When you invoke the Application Wizard from the Tools menu, it calls appwiz.prg, which in turn invokes the dialog box in Figure 5, provided by appwiz.scx.
Figure 5. The standard Visual FoxPro 6.0 Application Wizard dialog box provided by appwiz.scx
When you choose a project name and location, appwiz.prg invokes HOME( )+ WizardsWZAPP.APP, the Visual FoxPro 5.0 Application Wizard, with some special parameters.
The older wizard contained in wzapp.app does most of the work of creating your new project files. The Visual FoxPro 5.0 Application Wizard determines that you are in a special automated mode from the object reference it receives as one parameter and does not show its original interface. It evaluates a set of preferences received from this object reference, and proceeds with the generation process.
The standard implementation has a number of constraints:
Your application subclasses descend directly from _FRAMEWK.VCX. This prevents your adding superclass levels with your own enhancements to the framework, and you certainly can’t specify different superclasses when you generate different “styles” of applications.
Your copies of the ancestor classes, in _FRAMEWK.VCX and FFC libraries, are presumed to be in the HOME( )+ Wizards and HOME( )+ FFC directories. Because these ancestor classes are built into your framework applications, and therefore require recompilation during a build, you have to give all team members write privileges to these locations or they can’t use the Application Wizard to start new framework applications. In addition, the fixed locations hamper version control; you may wish to retain versions of ancestor classes specific to older framework applications, even when Microsoft delivers new FFC and Wizards folders.
Your non-OOP components are always generated out of HOME( )+ Wizards_FRAMEWK.DBF. The templates are not easily accessible for editing. The assumed location of _FRAMEWK.DBF prevents you from using different customized template versions for different types of apps, and also presents the same location problems (write privileges and versioning) that affect your use of the framework class libraries. As with your application subclasses, you can’t designate different templates when you generate different types of applications.
You have no opportunity to assign a custom Project Hook to the project.
To allow you to design and deploy customized framework components, a revised Application Wizard should, at minimum, address these points.
You can make the required changes without major adjustment of the current Application Wizard code, but some additional architectural work provides more room for other enhancements later.
A New Application Wizard
If you DO NEWAPPWIZ.PRG, provided in the source code for this article, you will get a dialog box almost identical to Figure 5, and functionally equivalent to the original dialog box. The only difference you’ll notice is a request, on startup, asking you if you wish to register this wizard in your HOME( )+ WizardsWIZARD.DBF table for future use (see Figure 6).
Figure 6. The Newappwiz.prg wizard classes can be registered to HOME( )+ WizardsWIZARD.DBF so you can choose them from the Tools Wizards menu later.
Though your newly instantiated wizard class calls the old Visual FoxPro 5.0 Wizard code just as the original one did, its internal construction allows completely new generation code to replace this approach in a future version.
You can call newappwiz.prg with a great deal of information packed into its second parameter, to indicate what wizard class should instantiate and what this wizard class should do once instantiated.
Why the second parameter, rather than the first? Newappwiz.prg, like appwiz.prg, is designed with the standard wizard.app in mind. wizard.app, the application invoked by the Tools Wizards menu option for all wizard types, uses its registration table, HOME( )+ WizardsWIZARD.DBF to find the appropriate wizard program to run. Wizard.app passes other information in its first parameter to the wizard program (in this case, newappwiz.prg). Wizard.app passes the contents of the Parms field of wizard.dbf, as the second parameter.
If you choose Yes in the dialog box in Figure 6, the NewAppWizBaseBehavior class becomes a new choice in the registration table, and fills out its options in the Parms field. Additional NewAppWizBaseBehavior subclasses will do the same thing, registering their own subclasses as separate entries. Once a class is registered in wizard.dbf, you don’t have to call newappwiz.prg directly again.
If you’ve chosen Yes in the dialog box in Figure 6 and also choose to register the wizard subclass we investigate in the next section, when you next choose the Application Wizard from the Tools menu, you’ll get a choice, as you can see in Figure 7.
Figure 7. Select your Application Wizard du jour from the Tools Wizards option—once you have more than a single Application Wizard listed in your HOME( )+ WizardsWIZARD.DBF table.
An Extended Subclass of the New Wizard: AppWizReinherit
With an enhanced architecture in place, we can address the issues of component-generation we’ve raised.
Run newappwiz.prg again, this time with a second parameter indicating a different wizard subclass to instantiate:
You should get another message box, similar to Figure 6, asking you if you want to register this subclass in the wizard.dbf table. When you’ve dismissed the message box, you see the dialog box in Figure 8.
The first page of this dialog box contains exactly the same options as the standard Application Wizard.
NoteYou’ll find all the visual classes used in the new wizards in newappwiz.vcx, as part of the source code for this article. The container you see on this page of the AppWizFormReinherit class is the same container class used in AppWizFormStandard. You can read more about these dialog box classes in Appendix 4.
Each subsequent page of the dialog box addresses one of our concerns with the way the original Application Wizard delivers components, and includes some information about how it works. (Figure 9 shows you pages 2 and 3.) Each option defaults to the same behavior you’d get from the original Application Wizard—you don’t need to fill out information on all pages.
Figure 9. Pages 2 and 3 of the Re-inherit App Wizard provide a layer of superclasses and the locations of your FFC and _FRAMEWK.VCX libraries for this framework application.
If you change the parent VCX as suggested on the second page of the dialog box, you can have one or more layers of superclasses between your application’s subclasses of _FRAMEWK.VCX. You’ll create team-specific enhancements in these layers.
NoteThis version of the Application Wizard will create the initial classes for you, as subclasses of the components in _FRAMEWK.VCX, if you specify a VCX name that does not exist. Later, you can create more layers of subclasses from the one the Application Wizard derived from _FRAMEWK.VCX, and designate your subclass layer in this dialog box as appropriate. The VCX you designate on the second page of this dialog box should always conform to the following rules:
Be the immediate superclasses (parent classes) of the application-specific VCX for this application.–and–
Include all the required subclasses of _FRAMEWK.VCX, with the same names as the _FRAMEWK ancestor classes.
You may want several different branches of your team-specific class levels, to match different types of framework applications you commonly create. For example, you could have one superclass set with your team’s options for a framework module and another one with your team’s topform custom attributes (including the class and classlibrary for your subclass of _topform to provide the correct frame).
NoteThese branches, or types, are not restricted to the “styles” or options you see represented in the Application Builder. They are just part of the normal process of subclassing and enhancing a class tree.
For example, you may decide to create Active Documents as framework applications. To do so, you’ll need an _Application subclass that is aware of its hosted environment, and makes certain interface decisions accordingly. You’ll also need an ActiveDoc subclass that is aware of the framework’s capabilities and calls application object methods in response to browser-triggered events, just as the menu templates invoke framework behavior.
Now that you can insert class levels between _FRAMEWK.VCX and your application-specific level, you can make the implementation of these features standard across applications.
If you change the locations of the FFC and _FRAMEWK.VCX libraries on the “Ancestors” page, the Application Wizard will place appropriate copies of the required class libraries in your specified locations if they don’t exist. The Application Wizard also ensures that your copy of _FRAMEWK.VCX inherits from the proper version of FFC, and that your parent classes point to the proper version of _FRAMEWK.VCX.
NoteAs mentioned in the section “FoxPro Foundation Generic Classes,” your FFC location can include your own version of _BASE.VCX. Your _BASE.VCX does not have to have the same code or custom properties as the original _BASE.VCX, but like your parent classes, your _BASE must include classes descended from the same Visual FoxPro internal classes, with the same names, as the classes in the original _BASE.
Other FFC libraries, not used in the framework and not described in this article, will not necessarily work with your own _BASE.VCX. For example, if your application uses _GENHTML, the _HTML.VCX library relies on code in the HOME( ) + FFC_BASE.VCX library. If you use other FFC libraries in your framework application, you may have two _BASE.VCXs included in your project—this is perfectly normal.
The Application Wizard then focuses on your template files on the next page of the dialog box. If you set a location for your template files, the Application Wizard will create fresh copies of these files (by copying them from the original _FRAMEWK.DBF), ready for you to edit.
In each case, if the files are already in the locations you supply, the Application Wizard will use the ones you have.
The last page of the dialog box allows you to pick a Project Hook. The original AppHook class in HOME( ) + WizardsAPPHOOK.VCX is the required ancestor class for a Project Hook designed to work with this application framework, but you can add a lot of team-specific features to your Project Hook subclass. The Application Wizard attempts to verify that the class you specify on this page descends from the appropriate AppHook class.
When you generate your application, the Application Wizard will create a new set of straight subclasses from your parent VCX (or _FRAMEWK.VCX, if you haven’t changed the default on the “Parents” page). These subclasses become the new T_META.VCX/VCT records in _FRAMEWK.DBF. The Wizard appends new contents for all the other template records of _FRAMEWK.DBF from the template folder, if you’ve named one.
NoteThe first time you and the Application Wizard perform these tasks, it won’t make much difference to the final results. Once the Wizard gives you editable superclass layers and your own copies of the templates, however, you have all the architecture necessary to customize the framework for subsequent uses of the Application Wizard.
Having replaced _FRAMEWK.DBF records, the Application Wizard proceeds to create your new application much as before, inserting information about your designated Project Hook class at the appropriate time.
All the “enhanced” Wizard actions are tuned to respect the current setting of the lDelegateToOriginalAppWizard switch, which indicates whether the Visual FoxPro 5.0 Application Wizard code is running or if new code is creating the project. For example, because the original code only looks in the HOME( )+ Wizards folder for _FRAMEWK.DBF, if you have indicated a different place for your _FRAMEWK.DBF (on the “Templates” page) this table will be copied to HOME( )+Wizards before wzapp.app runs. (The first time this occurs, the new Wizard copies your original _FRAMEWK.DBF to a backup file in the HOME( ) + Wizards folder.) Presumably, newer code simply uses your templates table wherever you’ve placed it.
When you use this Wizard to generate a framework application it saves information about your preferred parent classes, as well as the locations of your FFC and _FRAMEWK libraries and template files, to special _FRAMEWK.DBF records. You won’t need to enter this information, unless you wish to change it. This release of the Application Wizard doesn’t save information about the custom Project Hook subclass you may have specified. However, the next section will show you how to put this information into the Parms of wizard.dbf for default use.
NoteBecause the Application Wizard reads its stored information out of _FRAMEWK.DBF, it can’t get the location of _FRAMEWK.DBF from a stored record! However, you can put this information into the Parms field of wizard.dbf, as described in the next section, so all your developers use the proper version of _FRAMEWK.DBF without having to look for it.
You may even decide to use a version of this Wizard class, or of its associated dialog box, that only allows some developers to change the “advanced” pages. Other team members can fill out standard information on Page 1, but they’ll still get your improved versions of all the framework components.
Registering Additional Wizard Subclasses and Customized Records
The new Application Wizard provides the opportunity to register each subclass of its superclass separately in the wizard.dbf table. The wizard stores its class name and location in the Parms field of its own wizard.dbf record.
However, you can add more information in the Parms field. You can even store multiple entries in the wizard.dbf for a single subclass, with differently tuned Parms values. The Application Wizard, once instantiated, uses this additional information.
Here’s the full list of nine options you can pass in the second parameter, or place in the Parms field, for use by NewAppWizBaseBehavior and its subclasses. All #DEFINEs mentioned in this list are in the newappwiz.h header file associated with newappwiz.prg:
These three options instantiate the Wizard:
Wizard classMust descend from #DEFINEd APPWIZSUPERCLASS, defaults to NEWAPPWIZSUPERCLASS.
Wizard classlibLibrary containing wizard class, defaults to NEWAPPWIZ.PRG.
.App or .exe file nameOptional file, containing the wizard class library.
These six options are used by the Application Wizard after it instantiates:
Wizard form classMust descend from #DEFINEd APPWIZFORMSUPERCLASS, defaults to #DEFINEd NEWAPPWIZFORMSTANDARD.
Wizard form classlibLibrary containing the form class, defaults to NEWAPPWIZ.VCX.
.App or .exe file nameOptional file containing the wizard form class library.
Project Hook classThe Project Hook class you want to associate with this project, if you don’t want to use the default Project Hook class associated with framework-enabled projects. This class should descend from the AppHook class in HOME( )+ “WizardsAPPHOOK.VXC”, so it includes the default functionality, but can include enhancements required by your team.
Project Hook classlibThe class library containing the Project Hook class you choose to associate with this project.
Template DBFHolding application components, defaults to HOME( )+ Wizards_FRAMEWK.DBF (#DEFINED as APPWIZTEMPLATETABLE).
Store these values delimited by commas or carriage returns in the Parms field of wizard.dbf. Similarly, if you call newappwiz.prg directly, you can pass all this information as the program’s second parameter, as a single string delimited with commas or carriage returns.
After you’ve registered the AppWizReinherit class, the Parms field for this class’ record in wizard.dbf contains the following information:
APPWIZREINHERIT,<fullpath>newappwiz.fxp,,AppWizFormReinherit, <fullpath>NEWAPPWIZ.VCX,,APPHOOK, <fullpath of HOME()+ "Wizards"> APPHOOK.VCX, <fullpath of HOME()+ "Wizards"> _framewk.DBF
You could run the NEWAPPWIZ program, passing the same string as its second parameter, to get AppWizReinherit‘s default behavior.
Using our ActiveDoc example just shown, you could create a wizard.dbf entry that invokes the same Wizard class but defaults to a different parent VCX and different menu templates than the rest of your framework applications.
To accomplish this, you’d edit the information in the ninth value for this row of the wizard.dbf table, which indicates Template DBF, by editing the Parms field.
Your new row in the table contains the same string in the Parms field, except for the section following the last comma, which points to a new template table. Your special ActiveDoc copy of _FRAMEWK.DBF holds your special Active Document menu templates and superclass information.
Next, suppose you decide that your ActiveDocument framework applications need a special Project Hook subclass, not just special superclasses and menu templates. You could specify this hook automatically, in the seventh and eighth sections of the Parms field. You might even subclass the AppWizFormReinherit dialog box, to disable the last page of this dialog box for ActiveDocument-type applications, by changing the fourth and fifth sections of the Parms field. (This way, your team members would always use the right Project Hook class when generating this type of framework application.)
If you made all these changes, this new entry in the wizard.dbf table might have a Parms field that looked like this:
You would also edit the Name field in wizard.dbf for this entry, perhaps to something like “Active Document Framework Application,” to distinguish this entry from your standard values for the AppWizReinherit class.
When one of your team members accessed the Tools Wizards option from the system menu, “Active Document Framework Application” would now appear on the list of available Wizards, as part of the list you saw in Figure 7. The developer could automatically create the right type of framework application, without making any special choices.
A Few Parting Thoughts about Team Practices
You’ll notice a check box in the Reinheritance Wizard‘s dialog box, indicating that you can omit message boxes and generate your new application with no warning dialog boxes or user interaction. Although this is a helpful option once you’ve used this Wizard a few times, please be sure to read all the message boxes, and the information in the edit boxes on the various pages of this dialog box, at least once.
Any developer’s tool, especially one that edits visual class libraries and other metafiles as extensively as this one does, can potentially cause problems if the system is low on resources. The Help text available within this Wizard attempts to point out its potential trouble spots, so you can close other applications as needed, and have a good idea of what to expect at each step. Other caveats, such as incompletely validated options in this preliminary version, are indicated in the Help text as well.
You also see a More Info button, which provides an overview of the issues this class is meant to address, and how you can expect it to behave (see Figure 10).
Figure 10. Wizard documentation under the More Info button
Beyond its stated purpose to enhance the Application Wizard, AppWizReinherit and its dialog box class try to give you a good model for tool documentation, both at design and run time. The dialog box’s NewAppWiz_Documentation( ), GetUserInfo( ), and DisplayDocumentation( ) methods should give you several ideas for implementation of run-time documentation. Newappwiz.prg has a demonstration procedure, BuilderGetDocumentation( ), which shows you how you can apply these ideas to design time documentation for Builders as well. A final demonstration procedure in newappwiz.prg, ReadDocs( ), shows you another aspect of this process.
Each documentation idea demonstrated here is a variation on a theme: Text is held (using various methods) within the VCX, so it travels with the VCX and will not get lost no matter how widely you distribute the library.
Whether you use these particular implementations is not important; in many cases you’ll be just as well off if you create a text file with documentation and use Visual FoxPro’s FileToString( ) method to read this information for display by the tool whenever necessary.
No matter how you decide to implement it, documentation that helps your team better understand the intended use, extension possibilities, and limitations of the tools you build is critical to their adoption and successful use.
A framework is, in itself, a kind of abstraction, a level above daily activities. Enhancements to a framework represent yet another level of abstraction. Your team will benefit from all the extra attention you can give to communicating your goals for this process.
With any framework, you can efficiently prototype applications and build complete lightweight applications. With a framework set up the way your team operates, you can accomplish these goals without sacrificing quality, depth, or your normal habits of development. With a framework set to deliver your standard components and practices automatically, even new developers can make meaningful, rewarding contributions to your team effort.
Appendix 1: The User Option System
The framework employs a user-registration system based on a user table that is created by the application object if not found at run time. The application object uses the cUserTableName property to set the name and location of this table. If no path is supplied in this property, the location will be set by the cAppFolder property.
Note By default, the application object sets cAppFolder to the location of the APP or EXE that instantiated it. If, for some reason, the application object was instantiated outside a compiled APP or EXE container, cAppFolder contains the location of the application object’s VCX.
If necessary, the application object creates this table in the appropriate location, using the following code (excerpted from the CreateUserTable( ) method):
lcIDField = THIS.cUserTableIDField
lcLevelField = THIS.cUserTableLevelField
* names of two generic-requirement fields,
* User ID and level, are specified by
* _Application properties in case you
* wish to match them to some existing system
CREATE TABLE (tcTable) ;
((lcIDField) C(60), ;
(lcLevelField) I, ;
UserPass M NOCPTRANS, ;
UserOpts M NOCPTRANS, ;
UserFave M NOCPTRANS, ;
UserMacro M NOCPTRANS, ;
UserNotes M )
INDEX ON PADR(ALLTR(&lcIDField.),60) TAG ID
* create a case-sensitive, exact word match
INDEX ON PADR(UPPER(ALLTR(&lcIDField.)),60) TAG ID_Upper
* create a case-insensitive, exact word match
INDEX ON DELETED( ) TAG IfDeleted
If you don’t opt to have users log in and identify themselves in this application, this table is still created. In this case it supplies a default record, representing “all users,” so user macros, favorites, and options can still be stored in this table on an application-wide basis.
NoteBecause of their “global” nature in Visual FoxPro, user macro saving and setting features are only available to framework applications that issue READ EVENTS. Module applications are not allowed to edit the macro set.
When a user logs in, his password is evaluated using the user table’s UserPass field. A SetUserPermissions( ) method, abstract in the base, is called at this time so the user’s level can be checked in order to make appropriate changes to the application and menu options as well.
If the login is successful (or when the application starts up assuming no user login for this application), user name and level are stored in the cCurrentUser and iCurrentUserLevel properties.
User macros, favorites, and options are set from the user’s record in the user table. The _Application code handling macros rely on standard Visual FoxPro abilities to SAVE and RESTORE macros to and from the UserMacro memo field. The favorites system uses an easy-to-read ASCII format in the UserFave memofield. However the options system and the UserOptions field deserve more explanation.
The user table stores option information in its UserOptions memo field, by SAVEing the contents of a local array. This local array is RESTOREd and copied into a member array, aCurrentUserOpts, to establish user options when the current user is set.
The array format is fixed, and yet extremely flexible in the types of user options that can be stored. The allowable options include SETs and member properties, and the options should be specified as being “global” to the application or private to a datasession. The array is laid out, to specify these attributes of each option, in four columns, as follows.Expand table
User Option Array Column 1
Column 2
Column 3
Column 4
Item nameFor a SET command, the item you’re setting, same as what you’d pass to the SET( ) function. For an object, the property you wish to set. Can be the Member.Property you wish to set.
Value for this item
Property (.F.) or SET (.T.) ?
Session (.F.) or Global (.T.) ?
Each time a user logs in, the application method ApplyGlobalUserOptions( ) applies SET options and application object property values for all array rows with .T. in the fourth column. The mediator object has the responsibility to call the application method ApplyUserOptionsForSession( ), on your instructions, passing a reference to its parent form. This method applies SET options and form property values for all array rows with .F. in the fourth column.
The _Options dialog box supplied in _FRAMEWK.VCX gives you examples of all the combinations that can be created for a user option using this array, although its contents are merely examples. It shows you how the user options stored in an array can be expressed as a user interface, giving the user a chance to make changes. It also shows how results of a user-option-setting can be “translated” back into the user options array for use during this login, or saved as defaults to the user preference table.
You will note that, when the user options to apply changes to the current settings, the Options dialog box reinvokes ApplyGlobalUserOptions( ) and then iterates through the available forms, giving their mediators a chance to reapply session settings if they’re set to do so.
In many cases, a “global” setting can transferred to forms as well. For example, the _ErrorLogViewer dialog box has a mediator that checks the application’s cTextDisplayFont setting. This is a global user option, because it provides a chance for the user to specify a text font across all the UI of an application. The mediator transfers the value of the cTextDisplayFont to a property of the same name belonging to its parent dialog box. An assign method on this property then applies the fontname value to all members of the dialog box that should reflect the setting.
Appendix 2: The Default Metatable Structure
This table shows you the default structure of the framework’s metatable. Appendix 3 shows you how the default _FRAMEWK.VCX dialog boxes use this information.Expand table
FieldName
Type
Use
Doc_type
C
This field contains a character to distinguish between document types. Currently, “F” is used for “forms” and “R” is used for “reports.” But this designation just determines how the document type is presented in the interface, not necessarily what type of Visual FoxPro source code file underlies the document. See Alt_Exec and Doc_wrap fields, below.More document types may be added. The framework already contains one extra type, “A,” specifically reserved for you to add application information. The framework will not use “A”-type metatable records in any way, so the reservation of this type simply allows you to use metatable records, or perhaps one metatable header record, as a convenient place for system storage. In most cases, you would want to transfer the contents of such a record to application properties on startup.
Doc_descr
C
The “caption” or long description you want to show up in document picker lists.
Doc_exec
M
The name of the file to be run, usually an .scx or .frx file. In the case of a class to be instantiated, this is the .vcx file name.For Form-type documents, the file extension is assumed to be .scx unless this entry is marked “Doc_wrap” (see below) or the Doc_class field is filled out, in which case the extension is assumed to be .vcx.For Report-type documents, the file extension will default to .frx unless this entry is marked “Doc_wrap”. If no .frx file exists by that name, the application object looks for an .lbx file.In all cases, you may also fill out the file extension explicitly.In all cases, if you Include the file to be run in the project, you need not use paths in this field. If you wish to Exclude the file from the project, you may use path information. Assuming your applications install their subsidiary Excluded files to the appropriately located folder, relative pathing should work in the metatable, and is probably the best policy in this case!
Doc_class
M
The class to be instanced, where the Doc_exec is a .vcx file
Doc_new
L
Mark this .T. for a Form-type document you wish to show up in the FileNew list. When the application object instantiates a form from the FileNew list, it sets its own lAddingNewDocument property to .T. This practice gives the form a chance to choose between loading an existing document or a blank document during the form’s initialization procedures.In many cases, the form delegates this process to its mediator object. The mediator object saves this information for later use.If you do not use a mediator, you may wish to save this information to a form property; you can’t expect the application object’s lAddingNewDocument to reflect the status of any particular form except during the initialization process of that form.For a Report-type document, this field denotes an editable report (new report contents, or even a new report from a template). This capability isn’t currently implemented.
Doc_open
L
Mark this .T. for a Form-type document you wish to show up in the FileOpen list.For a Report-type document, this field denotes a runnable report or label and will place the item in the report picker list.
Doc_single
L
Mark this .T. for a Form-type document that is modeless but should only have one instance. The application object will bring it forward, rather than create a second instance, if the user chooses it a second time.
Doc_noshow
L
Mark this .T. for a Form-type document that you wish to .Show( ) yourself after additional manipulation, rather than allowing the DoForm( ) method to perform the .Show( ).NoteYou will have to manipulate the application’s forms collection or the current _SCREEN.Forms( ) contents to get a reference to this form, so you can manipulate the form and then .Show it when you are ready. If you need this reference immediately, the best place to get it is probably the application object’s aForms[] member array. At this moment, the application object’s last-instantiated form is the one for which you want the reference, and the application object’s nFormCount property has just been refreshed. Therefore, .aForms[THIS.nFormCount] gives you the reference you need when you’re in an application object method (in other code, replace THIS with a reference to the application object). You can see an example of this usage in the _Application‘s DoFormNoShow( ) method.You can create Doc_Wrap programs as described in the entry for the next field. Your wrapper program can take advantage of the DoFormNoShow( ) method, receive its return value (a reference to the form or formset object), and proceed to do whatever you want with it.
Doc_wrap
L
If this field is marked .T. indicating a “wrapped” document, the application’s DoProgram( ) method will run instead of its DoReport( )/DoLabel( ) or DoForm( ) method.If you omit the file extension, the DoProgram( ) method uses the standard Visual FoxPro extension hierarchy to figure out what file you wish to run (“.exe .app .fxp .prg”).
Doc_go
L
If this field is marked .T. and the document is “Form”-type, the form uses the framework’s standard Go context menu for navigation. The menu name is configurable using the application object’s cGoMenuFile property. This field is not used for report-type documents.
Doc_nav
L
If this field is marked .T. and the document is “Form”-type, the form uses the framework’s standard navigation toolbar for navigation. The class is configurable using the application object’s cNavToolbarClass and cNavToolbarClassLib properties. This field is not used for report-type documents.
Alt_exec
M
If this field is filled out, it takes precedence over the Doc_exec field just described. When the user makes a document choice, the _DocumentPicker’s ExecDocument( ) method converts the contents of this field into a string and runs that string as a macro.Your Alt_exec statement can be anything you choose, and it can use attributes of the metatable, including the Properties field (below) however you want. For example, you can choose to have the metatable editable (on disk) rather than included in the APP/EXE, and you can place information in the Properties field dynamically at run time. Your document would then be able to be “aware” of this information by examining the current contents of the Properties field.
Properties
M
This memo field is not used by the framework in any way. It’s for developer use, primarily in conjunction with the Alt_exec field.
User_notes
M
This memo field is not used by the framework in any way. It can be used for notes that would be displayed as Help text for a particular form or report, and so on.
Appendix 3: Default Document- Management Elements of the Framework
The framework accesses metatable information through the _DocumentPicker classes. _DocumentPicker is an abstract standard dialog box class, which contains a picklist and a couple of buttons. The working _DocumentPicker subclasses each have their own way of using the information in the metatable to perform two tasks:
Show the documents in the picklist.
Run the appropriate action when the user picks a document.
Each subclass stores the relevant metatable fields into an array, which serves as the data source for the list box in the dialog box. The same array holds the metatable information that will eventually act on the user’s choice.
The _DocumentPicker superclass has an abstract FillDocumentArray( ) method, designed to perform the first service during the dialog box Init( ), and another abstract method called ExecDocument( ), which is triggered whenever/however the user makes a selection from the document list.
The _DocumentPicker class receives a parameter from the application object. Each subclass of _DocumentPicker uses the parameter to determine which of two states it is supposed to be in when it displays its document list and acts on the user’s choice of a document from the list. The _DocumentPicker superclass simply makes note of this logical value, leaving it to the subclasses to interpret it.
The various _DocumentPicker’s FillDocumentArray( ) methods concentrate on different document types, and fill the array with the appropriate information for that type. Their ExecDocument( ) methods call different application object methods depending on their document type and the dialog box’s current state, sending information from the metatable from the array to method arguments as needed.
The first two columns in the table below show you the names of these working classes and the document types that will appear in their lists, courtesy of their FillDocumentArray( ) method. The other columns show the application methods that call them, and the meaning assigned to their two states when ExecDocument( ) is triggered. Each application method listed here takes a logical parameter (defaulting to .F., State 1) to indicate for what purpose the class presents its document list.Expand table
_DocumentPicker Subclass
_Document types
Associated _Application method
State 1 action
State 2 action
_NewOpen
forms
DoNewOpen( )
Edit
Add
_ReportPicker
reports and labels
DoReportPicker( )
Run report/label
Modify/Add not implemented in _Application superclass.
_FavoritePicker
documents and files of any type
DoStartupForm( )
Run document/file
Put document / file on Favorites menu for quick access.
Appendix 4: Using the NEWAPPWIZ Visual Classes
AppWizFormReinherit, the dialog box called by AppWizReinherit, and AppWizFormStandard, the default dialog box with the same interface as the original wizard, both descend from the same superclass, AppWizFormBaseBehavior (see Figure 11).
Figure 11. Newappwiz.vcx in the Class Browser
AppWizFormBaseBehavior is the required superclass for any dialog box provided as the UI of a NewAppWizBaseBehavior or its descendents. The Application Wizard superclass validates your dialog box class when it instantiates the dialog box as descending from this superclass dialog box.
NewAppWizBaseBehavior contains only the very simple required behavior, no visible controls. It has three custom properties to represent required wizard information (project name, location, and whether or not the Wizard should generate project directory structure). It receives this information from an object reference the Wizard passes. It has a Finish( ) method which passes this information back to the Application Wizard.
In your subclass of AppWizFormBaseBehavior, you simply databind the interface controls of your choice to these three custom properties. You create other controls and custom properties to represent your enhanced options. Your dialog box calls the Finish( ) method when you’re ready to generate. (Both AppWizFormReinherit and AppWizFormStandard use the OKButton class you see in Figure 11, which contains the call to its parent form’s Finish( ) method.)
You can augment Finish( ) to pass more options from the dialog box back to your Wizard subclass as necessary.
You’ll find more information in the NewAppWiz_Documentation method of the superclass. The default AppWizFormStandard subclass shows you a simple example of how to make it work.
ADO Jumpstart for Microsoft Visual FoxPro Developers
Summary: Provides Microsoft Visual FoxPro developers with an overview of ActiveX Data Objects (ADO) and shows how to incorporate ADO into Visual FoxPro applications. Discusses the ADO object model and implementing Remote Data Services (RDS). (52 printed pages)
Contents
IntroductionWhat are OLE DB and ADO?Why Incorporate ADO into a Visual FoxPro Application?ADO Object ModelRemote Data ServicesSummary
Introduction
Microsoft®ActiveX® Data Objects (ADO) is perhaps the most exciting new Microsoft technology in quite some time. Because ADO is concerned with data, this new technology is of particular interest to Microsoft® Visual FoxPro® developers. Of course, you may ask, “Why do I need ADO? Visual FoxPro already has a high-performance local data engine.” It’s a good question.
This paper provides the Visual FoxPro developer with a background of what ADO is and how to incorporate ADO into Visual FoxPro applications. After reading this paper, you should have enough information to readily answer the question: “Why do I need ADO?”
A Brief Word About ADO Events
One limitation of Visual FoxPro has been an inability to surface COM events. While Visual FoxPro can respond to events raised by ActiveX controls, objects created with the CreateObject function cannot. In Microsoft®Visual Basic®, COM Events are handled by using the WithEvents keyword. In Visual FoxPro, the new VFPCOM.DLL achieves the same results. The topics VFPCOM, ADO Events, and how to integrate ADO and Visual FoxPro will be discussed in another white paper. This paper is dedicated to providing the Visual FoxPro developer, with a comprehensive overview of ActiveX Data Objects, Remote Data Services (RDS), their respective objects, and how those objects work.
This paper covers the following topics:
What are ADO and OLE DB?
Why incorporate ADO into a Visual FoxPro application?
The ADO object model
Remote Data Services
What Are OLE DB and ADO?
When discussing ADO, we are really talking about two distinct elements: the ActiveX data objects themselves and Microsoft Universal Data Access technology, more commonly known as OLE DB.
OLE DB and Universal Data Access
In simple terms, OLE DB is the succeeding technology to the Open Database Connectivity (ODBC) standard. OLE DB is a set of low-level interfaces that facilitate the Microsoft Universal Data Access strategy. ADO is a set of high-level interfaces for working with data.
While both ODBC and OLE DB have the ability to make data available to a client, the capabilities of the two technologies are very different. ODBC is primarily designed for use on relational data. However, data exists in nonrelational as well as relational formats. In addition to new data formats, data resides in new places such as the Internet. Finally, the Microsoft Component Object Model (COM) framework requires better data access technology. Clearly, ODBC does not address these needs; a new technology is needed. That technology is OLE DB, and it is here to stay.
The following graphic best illustrates how OLE DB and ADO work together. Clients can work directly with OLE DB or can work with OLE DB through the ADO interface (the latter is typically the case). Note that OLE DB can access SQL data either directly or through ODBC. An OLE DB provider provides direct access by OLE DB. Also note that OLE DB can also be used to access a variety of non-SQL data, as well as data that exists in mainframes. The ability to access data through a common interface, without regard to data location or structure, is the real power behind ADO and OLE DB.
Whereas ODBC uses drivers, OLE DB uses providers. A provider is a software engine that provides a specific type of data that matches the OLE DB specification. Several OLE DB providers exist today, including those for Microsoft SQL Server™ and Oracle. Because there is such widespread use of ODBC, an OLE DB provider for ODBC has also been created in order to ease the migration from ODBC to OLE DB. Several nonrelational providers are currently under development. Perhaps the most anticipated of these is the OLE DB Provider for Microsoft Outlook®. A special provider, MS Remote, allows direct data access over the Internet. This brief list of providers shows the third-party community commitment to OLE DB, and many new providers are currently under development. For the latest news on available providers, refer to https://www.microsoft.com/data/.
ADO Overview
OLE DB is then a set of low-level interfaces that provide access to data in a variety of formats and locations. While powerful, OLE DB interfaces can be cumbersome to work with directly. Fortunately, ADO provides a set of high-level, developer-friendly interfaces that make working with OLE DB and universal data access a relatively simple task. Regardless of the programming environment you use, any Visual Studio® or Microsoft Office product such as Visual FoxPro, Visual Basic, Visual C++®, or Word, the interface you will use to access data remains constant. That interface is ADO, which in turn uses OLE DB.
ADO itself is just a set of objects. By itself, ADO is not capable of anything. In order to provide any functionality, ADO needs the services of an OLE DB provider. The provider in turn uses the low-level OLE DB interface to access and work with data. One ADO connection may use a SQL Server OLE DB provider and another ADO connection may use an Oracle OLE DB provider. While the interface is constant, the capabilities may be very different because OLE DB providers are very different, which highlights the polymorphic nature of OLE DB.
As developers, we crave consistency. ADO provides us with a consistent interface for our program code.
ADO Version Summary
The current version of ADO (2.1) is the fourth version of ADO to be released in less than two years. ADO 1.0 was primarily limited to working with Active Server pages. Only one OLE DB provider existed, the OLE DB Provider for ODBC Drivers.
ADO (2.1)—Ships with the newest version of Microsoft Web browser, Internet Explorer 5.0. When discussing data or anything related to the Internet, it is almost impossible to do so without mentioning XML. XML, the Extensible Markup Language, is a mark-up language that allows users to create custom tags to describe data. XML is quickly becoming the universal format for storing and streaming data. The primary storage format in Office 2000 for document data will be XML. ADO (2.1) client-side recordsets can be saved as XML documents.
ADO (2.0)—Represented a huge gain in functionality. One of the most notable new features was the ability to create client-side recordsets. To go along with this, also added were the abilities to create filters and indexes, and the ability to sort recordsets. These abilities are very much the same as those that exist with Visual FoxPro cursors. Finally, the ability to persist client-side recordsets was also added. In effect, data could be acquired from a server into a client-side recordset. The client-side recordset could then be saved as a file on the local hard-drive that could be opened at a later time without being connected to the network.
ADO (1.5)—Introduced new capabilities and providers to ADO. Among the new providers was the OLE DB Provider for Jet (the JOLT Provider). The MS Remote Provider, which powers the Remote Data Services (RDS), was introduced as well. This version also introduced the ability to create disconnected recordsets.
What You Need to Get Started
In order to work through the examples presented in this paper, you will need the following:
SQL Server 6.5 or 7.0 with the sample Northwind database installed
A system DSN called TasTrade that points to the TasTrade Visual FoxPro Sample Database
A system DSN called Northwind that points to the SQL Server Northwind database
Why Incorporate ADO into a Visual FoxPro Application?
Have you ever wanted to pass a cursor as an argument to a function or class method? Or have you wanted to pass data to automation server applications such as Microsoft Word or Excel? Perhaps you have created a Visual FoxPro DLL and have needed a way to pass data from the user interface to a class method in the DLL. Maybe you have been looking for a way to stream data across the Web. If your answer is “yes” to at least one of these, ADO can help you today!
Until now, the world of component-based development has lacked one thing: a method of effectively moving data between processes. Now, whether ADO is hosted by Visual FoxPro, Visual Basic, Excel, or Word, the interface is consistent. The new COM capabilities of Visual FoxPro 6.0 enable creating of ADO recordsets, populating them with data, and passing them to a variety of processes. This all goes to support the strategic positioning of Visual FoxPro, a creator of middle-tier components.
Just about everything in Visual FoxPro is an object, except for reports, menus, and data. One of the biggest feature requests from Visual FoxPro developers has been the ability to work with data as a set of objects. Data objects provide several benefits, including an enhanced event model and the ability to overcome limitations of Visual FoxPro cursors. While many limitations are gone, many benefits of Visual FoxPro cursors have been retained. As you work with ADO, there’s good reason to think are many similarities to Visual FoxPro; ADO is based on the Visual FoxPro cursor engine. So, for those who have wanted data objects in Visual FoxPro, the wait is over with ADO.
ADO is not a replacement for Visual FoxPro cursors. Rather, Visual FoxPro cursors and ADO are complementary. When used together, very powerful applications can result. The following pages detail the ADO object model and the common properties and methods you will work with, including:
Remote Data Services (RDS), technology which allows for the streaming of data over the Internet via HTTP.
VFPCOM.DLL, which enables the handling of COM events in Visual FoxPro.
ADO Integration into Visual FoxPro.
This section has several comprehensive examples on strategies you may employ when integrating ADO into your Visual FoxPro Applications.
ADO Object Model
Connection Object
ProgID: ADODB.Connection
The purpose of the Connection object is to provide access to a data store. To illustrate, the following code creates an ADO Connection object:
oConnection = CreateObject("adodb.connection")
Once an ADO Connection object has been created, you can access its data store. An active connection can be established by providing a few pieces of key information and invoking the Open( ) method of the Connection object. The following code opens a connection to the Visual FoxPro TasTrade database:
oConnection.Open("TasTrade")
Alternatively, the following code accesses the SQL Server Northwind database:
oConnection.Open("Northwind","sa","")
These two examples work with the OLE DB Provider for ODBC drivers. Different OLE DB providers can be used as well. The following example sets some common properties of the Connection object and uses the OLE DB Provider for SQL Server:
The syntax of the ConnectionString property appears complicated. Fortunately, you don’t have to code this by hand. When you install the Microsoft Data Access Components (MDAC), you can create a data link file.
To create a data link file:
Right-click your desktop and choose NewMicrosoft Data Link from the pop-up menu.
Specify a name for the file.
Right-click and select Properties to modify the file properties.
In the Properties dialog box, click the Provider tab, and choose a provider. The OLE DB Provider for ODBC is the default choice. For this example, select the OLE DB Provider for SQL Server.
Click the Connection tab.
Specify the name of the server, your user name and password, and the name of the database you wish to connect to.
Open the UDL file in Notepad.Now, it is just a matter of copying and pasting the information. Alternatively, you can use the file itself:oConnection.Open(“File Name=c:temptest.udl”)
ADO recognizes four arguments in the ConnectionString:
File Name: Specifies the name of a UDL file to use.
Provider: Specifies the name of an OLE DB provider to use.
Remote Provider: Specifies the name of a provider to use with Remote Data Services (RDS).
Remote Server: Specifies the server on which data resides when using Remote Data Services (RDS).
Any additional arguments passed in the ConnectionString are passed through to the OLE DB provider being used.
In addition to the Open method, the following are the common methods you are likely to use with the Connection object:
BeginTrans, CommiTrans, and RollBackTrans—These methods work like the Begin Transaction, End Transaction, and RollBack statements in Visual FoxPro. The Connection object controls all transaction processing. For more detail, see the section Transactions/Updating Data. Note that not all OLE DB providers support transaction processing.
Close—This method closes an open Connection object.
Execute—This method runs a SQL statement, stored procedure, or OLE DB provider-specific command. In reality, a Command object, which actually does the work of executing the command, is created on the fly. More on the Command object and the flat object hierarchy of ADO later in this paper.
OpenSchema—This method returns information regarding defined tables, fields, catalogs, and views into an ADO Recordset object. This method works like the DBGetProp( ) function in Visual FoxPro.
Errors collection
ADO does not trap errors, nor does it have an error handler. Instead, ADO can record the occasions when errors occur. It is up to the host application, Visual FoxPro in this case, to both trap and handle the error. ADO only reports what errors have occurred. Note that the error is actually reported by the specific OLE DB provider. ADO is merely a vehicle to report the error.
The Errors collection is part of the Connection object and consists of zero or more Error objects. When an error occurs, an Error object is appended to the Errors collection. The following code illustrates how the Errors collection works. In this example, the name of the database has been misspelled purposely in order to generate an error:
oConnection = CreateObject("adodb.connection")
With oConnection
.Provider = "SQLOLEDB.1"
.ConnectionString = "Persist Security Info=False;User
ID=sa;Initial Catalog=Nothwind;Data Source=JVP"
.Open
EndWith
*/ At this point an error will occur – causing VFP's default error
*/ handler – or the active error handler to invoke
*/ At this point, we can query the Errors Collection of the
*/ Connection Object
For Each Error In oConnection.Errors
?Error.Description,Error.Number
Next Error
Recordset Object
ProgID: ADODB.Recordset
Once you establish an ADO connection, you can open a recordset of data. The Recordset object is very much like a Visual FoxPro cursor. Like the Visual FoxPro cursor, an ADO recordset consists of rows of data. The recordset is the primary object that you will use while working with ADO. Like the Connection object, the Recordset object also provides an Open method. To illustrate, the following code opens the Customer table of the Visual FoxPro Tastrade database:
oRecordSet = CreateObject("adodb.recordset")
oRecordSet.Open("Select * From Customer",oConnection)
The first argument of the Open method specifies the source of data. As you will see, the source can take on several forms. The second argument of the Open method specifies a connection to use for retrieving the data specified by the source. At a minimum, this is all you need to open a recordset. Additional examples will expand on the additional arguments the Open method accepts.
With a Recordset object created, one of the most common actions you will perform is navigating through records. Depending on the type of ADO recordset that has been created, certain navigational capabilities may or may not be available. The different types of possible ADO recordsets will be discussed shortly. The following code illustrates how to navigate through an ADO recordset:
Do While !oRecordSet.Eof
oRecordset.MoveNext
EndDo
The following paragraphs briefly describe the most common recordset properties and pethods you are likely to use. It is by no means a replacement for the ADO documentation, which gives both a complete description of the properties and methods and complete descriptions of acceptable enumerated types and arguments. ADO is well documented in the Microsoft Data Access Components (MDAC) SDK. You can download the MDAC SDK from https://www.microsoft.com/data.
In addition, I highly recommend ADO 2.0 Programmers Reference, by David Sussman and Alex Homer, from Wrox Press.
RecordSet types
You can create four types of recordsets in ADO:
Forward Only—This type of recordset can be navigated only in a forward direction. It is ideal when only one pass through a recordset is required. Examples include populating a List box or a Combo box. The RecordCount property is irrelevant with this type of recordset.
Keyset—This type of recordset keeps acquired data up to date. For example, if you retrieve 100 records, data modified by other users to those 100 records will be visible in your recordset. However, modifications regarding new or deleted records made by other users will not be visible in your recordset. Both forward and backward navigation are supported. The RecordCount property returns a valid value with this type of recordset.
Dynamic—With this type of recordset, all underlying data is visible to the Recordset object. Because the number of records in the underlying table can change, the RecordCount property is irrelevant with this type of cursor. However, forward and backward navigation are supported.
Static—Both the number of records and data are fixed at the time the Recordset object is created. The only way to get the latest version of data and all records is to explicitly invoke the Requery method. You can use the RecordCount property. In addition, both forward and backward navigation is permitted.
RecordSet locations
Recordset objects can exist in either of two locations, the server or the client:
Server—The most common examples of server-side ADO recordsets are those created through Active Server Pages (ASP).
Client—A recordset that resides on a workstation is useful when creating disconnected recordsets or recordsets on which you wish to apply filters, sorts, or indexes.
The most common properties you are likely to use with ADO recordsets include the following:
ActiveCommand property—An object reference to the Command object that created the recordset.
ActiveConnection property—An object reference, to the Connection object, that provides the link to an underlying data source.
AbsolutePosition property—Specifies the relative position of a record in an ADO recordset. Unlike the Bookmark property, which does not change, the AbsolutePosition property can change depending on the active sort and filter.
Bookmark property—A unique record identifier that, like the record number in a Visual FoxPro cursor or a record number in Visual FoxPro, does not change during the life of a recordset.
BOF/EOF properties—Beginning of File and End of File, respectively, that work just like the BOF( ) and EOF( ) functions in Visual FoxPro.
EditMode property—Specifies the editing state of the current record in an ADO recordset.
Filter property—The string that represents the current filter expression. This property is like the SET FILTER statement in Visual FoxPro. Unlike the Find method, multiple expressions linked with AND or OR operators are allowed. This property is only applicable to client-side recordsets.
Sort property—A comma-delimited set of fields that specifies how the rows in an ADO recordset are sorted. This property is only applicable to client-side recordsets.
State property—Specifies the state of an ADO recordset. Valid State properties are closed, open, connecting, executing, or fetching.
Status property—Specifies the editing status of the current record. Valid Status properties include unmodified, modified, new, and deleted. This property can be any one of the values contained in RecordStatusEnum.
MarshalOptions property—Specifies how records are returned (marshaled) to the server. Either all or only modified records can be returned. This property is only applicable to client-side disconnected recordsets
MaxRecords property—Specifies the total number of records to fetch from a data source.
RecordCount property—Specifies the number of records in a recordset. This property is like the Recc( ) function in Visual FoxPro.
Source property—Specifies the command or SQL statement that provides data for the recordset.
NoteThe type and location of a cursor as well as the OLE DB provider you select will affect the recordset properties that are available.
Use the following table as a guide to help you make the right recordset type and location decision:
Table 1. PropertiesExpand table
Type
Bookmark
RecordCount
Sort
Filter
MarshalOptions
Forward Only
Key Set
4
4
Dynamic
Static: Client
4
4
4
4
4
Static: Server
4
4
Only client-side recordsets can be sorted and filtered. If the CursorLocation property of ForwardOnly, KeySet, and Dynamic recordset types is set to adUseClient, making them client-side cursors, the CursorType property is automatically coerced to the Static Cursor type.
NoteThis is the behavior of the OLE DB Provider for SQL Server. The OLE DB Provider for ODBC supports only ForwardOnly and Static recordsets, regardless of where the recordset resides.
As with properties, method availability can also vary:
Table 2. Available MethodsExpand table
Type
MoveFirst
MovePrevious
MoveNext
MoveLast
Resync
Requery
Forward Only
4
4
Key Set
4
4
4
4
4
Dynamic
4
4
4
4
4
Static – Client
4
4
4
4
4
4
Static – Server
4
4
4
4
4
The following list describes some of the common methods you will use in the ADO Recordset object:
MoveFirst, MovePrevious, MoveNext, MoveLast, and Move methods—Navigational methods that work as their respective names imply. The Move method accepts two arguments, the number of records to move and the position from which to begin the move. The Move method is similar to the Go statement in Visual FoxPro. MoveFirst and MoveLast work like Go Top and Go Bottom, respectively. Finally, MovePrevious and MoveNext work like Skip 1 and Skip –1, respectively.
Find method—Accepts a criterion string as an argument and searches the recordset for a match. If a match is not found, depending on the search direction, either the BOF or EOF property will evaluate to true (.T.). This method works much the same way as the Seek and Locate statements in Visual FoxPro. Unlike the Filter property and the Seek and Locate statements in Visual FoxPro, the ADO Recordset object does not allow multiple search values joined by the And or the Or operator. Using anything other than a single search value will result in an error.
Open method—Opens an existing ADO Recordset object. This method accepts several arguments and is discussed in detail later in this section.
Close method—Closes an ADO Recordset object. Many properties, such as CursorType and LockType, although read/write, cannot be modified while the recordset is open. The Close method must be invoked before those and other properties are modified.
Update and UpdateBatch methods—Update writes changes for the current record to the underlying data source; UpdateBatch writes pending changes for all modified records to the underlying data source. The UpdateBatch method is only relevant when Optimistic Batch Locking is used.
Cancel and CancelBatch methods—The Cancel method cancels modifications made to the current record; the CancelBatch method cancels pending changes to all modified records.
Resync method—Refreshes the Recordset object with data from the underlying data source. Invoking this method does not rerun the underlying command. Options exist for which records are actually refreshed.
Requery method—Unlike the Resync method, reruns the underlying command, which causes any pending changes to be lost. In effect, issuing a Requery is like invoking the Close method then immediately invoking the Open method.
Supports method—Specifies whether or not the recordset supports a function, based on a passed argument. For example, you can use this method to specify whether a recordset supports bookmarks, or the addition or deletion of records, or the Find, Update, and UpdateBatch methods, to name a few. Because what is supported is depends on the OLE DB provider used, it is a good idea to use this method to make sure a needed function is supported.
GetRows method—Returns a set of records into an array.
GetString method—Returns a set of records into a string.
The moral of the story is that before relying on the existence of anything in ADO, know and understand the OLE DB provider you are using, because the capabilities available to you can vary dramatically.
Lock types
There are four different locking schemes in ADO recordsets. These locking schemes are similar to those in Visual FoxPro.
Read-Only—As the name indicates, the recordset is opened for read-only purposes only. When you don’t need to modify data, this is the best locking scheme to use from a performance standpoint. This scheme applies to both server and client-side recordsets.
Lock Pessimistic—In this scheme, a lock attempt is attempted as soon as an edit is performed. This locking scheme is not relevant for client-side recordsets. Pessimistic Locking in an ADO recordset is like Pessimistic Locking with Row Buffering in a Visual FoxPro cursor.
Lock Optimistic—In this scheme, a lock attempt is made when the Update method is invoked. This locking scheme applies to both server and client-side recordsets. Optimistic Locking in an ADO recordset is like Optimistic Locking with Row Buffering in a Visual FoxPro cursor.
Lock Batch Optimistic—This scheme is like the Lock Optimistic scheme, except that more than one row of data is involved. In this scheme, a lock is attempted on modified records when the UpdateBatch method is invoked. This scheme is like Optimistic Locking with Table Buffering in a Visual FoxPro cursor.
The following table illustrates the availability of some common methods depending on the locking scheme used:
Table 3. Method Availability (Depending on Lock Type)Expand table
Lock Type
Cancel
CancelBatch
Update
UpdateBatch
Read Only
4
Pessimistic
4
4
4
4
Optimistic
4
4
4
4
Optimistic Batch
4
4
4
4
With the concepts of cursor types, locations, and locking schemes out of the way, we can discuss the real abilities of ADO recordsets. The most notable of these abilities are updating, sorting, and filtering of data. Before undertaking that discussion, however, take a few moments to review the Fields Collection object.
Fields collection object
Associated with the Recordset object, is the Fields Collection object. The Fields Collection object contains zero or more Field objects. The following code enumerates through the Fields Collection of a Recordset object:
For Each ofield In oRecordset.Fields
With oField
?.Name,.Value,.Type,.DefinedSize
?.ActualSize,.NumericScale,.Precision
EndWith
Next oField
The common Field properties you will work with:
Name—Specifies the name of the Field object. This corresponds to the name of the data element in the underlying data source. It is easy to define the name element as the name of the field in the underlying table. However, note that ADO and OLE DB work with both relational and nonrelational data. Given that, while you may be working with ADO, the underlying data may come from Outlook, Excel, Word, or Microsoft® Windows NT® Directory Services.
Value—Indicates the current value of the Field object.
OriginalValue—Indicates the Value property of the Field object before any modifications where made. The OriginalValue property returns the same value that would be returned by the OldVal( ) function in Visual FoxPro. When you invoke the Cancel or CancelUpdate methods of the Recordset object, the Value property of the Field object is replaced by the contents of the OriginalValue property. This behavior is similar to that exhibited when TableRevert( ) is issued against a Visual FoxPro cursor.
UnderlyingValue—Indicates the current value in the data source. This property corresponds most closely to the CurVal( ) function in Visual FoxPro. To populate the Value property of each Field object in the Fields collection, you need to invoke the Resync method of the Recordset object. With a client-side cursor, this property will return the same value as the OriginalValue property, since the recordset may or may not have an active connection.
Type—Indicates the data type of the Field object. The value of this property corresponds to a value contained in DataTypeEnum. Examples of values in DataTypeEnum are adBoolean, adInteger, and adVarChar.
Defined Size—Specifies the size of the field containing a data element in the data source. For example, in SQL Server, the Country field in the Customers table of the Northwind database is 15 characters long. Therefore, the DefinedSize property of the Country Field object is 15.
ActualSize—Represents the length of the actual data element in a datasource. To illustrate, consider the Country Field object again. In the case where the value is Germany, the ActualSize property is 7, while the DefinedSize property is still 15.
NumericScale—Specifies how many digits to the right of the decimal place are stored.
Precision—Specifies the maximum number of digits to be used for numeric values.
In addition to these properties, GetChunk is one interesting method you are likely to use. This method allows you to progressively fetch portions of the contents of a field object. This method is very useful when dealing with large text fields. It can be used only on fields where the adFldLong Bit set of the Attributes property is set to true (.T.). See the next section for details on the Attributes property. Understand that fields of the type ADLongVarChar have the adFldLong Bit set. The Notes field of the Employees table is of the type adLongVarChar.
The following code fetches data from the notes field in 10-byte chunks:
Local nBytesRead,cChunkRead
nBytesRead = 0
cChunkRead = Space(0)
Do While .T.
nBytesRead = nBytesRead + 10
cChunkRead = oRecordset.Fields("notes").GetChunk(10)
If IsNull(cChunkRead) Or;
nBytesRead > oRecordset.Fields("notes").ActualSize
Exit
Else
?cChunkRead
Endif
EndDo
Successive calls to GetChunk continue where the previous call ended. The GetChunk method is very useful when you need to stream data or only need to see the first few characters of a large text field.
Along with GetChunk, examine the AppendChunk method. The first time this method is called for a field, it overwrites any data in the field. Successive calls then append the data, until pending edits are cancelled or updated. The following code illustrates how this method works:
For x = 1 To 100
oRecordset.Fields("notes").AppendChunk(Str(x)+Chr(10)+Chr(13))
Next x
Both the GetChunk and AppendChunk methods are ideal for dealing with low memory scenarios.
The Attributes property
An attribute specifies the characteristics of something. As a person, you have many attributes, eye color, height, weight, and so forth. In the OOP world, objects have many attributes. Most of the time, attributes are exposed in the form of properties. A Visual FoxPro form has several properties such as Width, Height, and BackColor, just to name a few. The same is true for objects in ADO. Sometimes, however, it is not convenient to have a one-to-one correspondence between attributes and properties. Often, you can pack large amounts of information into a smaller space through the power of setting bits. A bit is much like a switch. It is either on or off or 1 or 0. If you string these bits together, you gain the ability to store multiple values in a small space. This is how the Attributes property works.
The Connection, Parameter, Field, and Property objects all have an Attributes property. If you have never worked with bit operations before, working with this property can be quite challenging. In some situations, as is the case with the GetChunk and AppendChunk methods, you will need to refer to the Attributes property of the Field object to determine whether those methods are available.
Using the Field object to illustrate how the Attributes property works, you can associate the following attributes with a Field object and its associated binary values:
AdFldMayDefer—Indicates that the field contents are retrieved only when referenced—0x00000002
adFldupdateable—Indicates that the field can be updated—0x00000004
adFldUnkownupdateable—Indicates that the provider does not know whether the field is updateable—0x00000008
adFldFixed—Indicates that the field contains fixed length data—0x00000010
adFldIsNullable—Indicates that the field can accept a null value during a write operation—0x00000020
adFldMayBeNullable—Indiates that the field may contain a null value—0x00000040
adFldlong—Indicates that the field contains long binary data and that the GetChunk and AppendChunk methods can be used—0x00000080
adFldRowID—Indicates that the field contains a row ID and cannot be updated. This does not relate to a field that may contain the identity value or some other auto-incrementing value. Rather, it relates to a ROW ID that is unique across the database. Oracle has this feature—0x00000100
adFldRowVersion—Indicates whether the field indicates the version of the row. For example, a SQL TimeStamp field may have this attribute set—0x00000200
adFldCachedDeferred—Indicates that once this field has been read, future references will be read from the cache—0x00001000
Usually, more than one of these attributes are present at any given time. Yet the Attributes property is a single value. Using the Employees table Notes field as an example, you will see that the Attributes property yields a value of 234. The value 234 represents the sum of the attributes for that field. For example, nullable and long attributes have decimal values of 32 and 128 respectively. This means that the Attributes property evaluates to 160. This works like the Windows Messagebox dialog box with regard to specifying the icon and types of buttons that are present.
Knowing that the Attributes property is a sum of the attributes of a Field object does not help in determining whether a specific attribute is present. This is where understanding bit operations comes in handy. The first step is to convert the sum (such as 234, above) into a binary equivalent:
11101010
Working from right to left, (or from the least significant bit to the most significant)—and beginning with zero, see that bits 1, 3, 5, 6, and 7 are set, (indicated by their values of 1 in those positions). Bits 0, 2, and 4 are not set. The next step is to determine whether a field is “long.”
To determine whether a field is a long field, we must first convert the adFldLong constant, which specifies which bit if set, indicates that the field is long. The adFldLong constant has a hex value of 0x00000080. This translates into a decimal value of 128. The following is the binary equivalent:
10000000
Converting a hex value to decimal in Visual FoxPro is simple. The following code illustrate how to convert hexadecimal values to decimal:
x = 0x00000080
?x && 128
And, if you ever need to convert back to hexadecimal:
?Transform(128,"@0") && 0x00000080
Using our original hex value, 11101010, and working from right to left and beginning with zero, see that the seventh bit is set. Therefore, the seventh bit of the Attributes property, if set, means the field is long. Going further, whatever attributes occupy bits 1, 3, 5, and 6, also apply to this field. The following table of field attributes should help to sort things out:
Table 4. Field AttributesExpand table
Hex Value
Decimal Value
Field Attribute Constant
Bit
0x00000002
2
AdFldMayDefer
1
0x00000004
4
AdFldupdateable
2
0x00000008
8
AdFldUnkownUpdateable
3
0x00000010
16
AdFldFixed
4
0x00000020
32
AdFldIsNullable
5
0x00000040
64
AdFldMayBeNull
6
0x00000080
128
AdFldLong
7
0x00000100
256
AdFldRowID
8
0x00000200
512
AdFldRowVersion
9
0x00001000
4096
AdFldCacheDeferred
12
So, along with being a long field, the field is deferred, updateable, can have a null written to it, and it may also already contain a null value. Visually, this makes sense. How can you do this programmatically?
If you refer to online examples (almost always programmed in Visual Basic), you will see code like this:
If (oField.Attribute AND adFldLong) = adFldLong
' The field is long
End If
This is pretty slick in that you can test for whether a specific attribute bit is set by using the AND operator with the attribute property and the constant. If you try this in Visual FoxPro, you will get data type mismatch errors. Fortunately, there is a way. Visual FoxPro contains a host of bit functions. One function, BITTEST, does as its name implies. It tests whether a specified bit in a passed argument is set. To review, we need to see if the seventh bit in the value 234 is set. The following Visual FoxPro code demonstrates how to use the BITTEST function:
If BitTest(234,7)
*/ The Field is long
Endif
To find out if the field is nullable:
If BitTest(234,5)
*/ The Field is long
Endif
The Attributes property of the Connection, Parameter, and Property objects works in the same manner as illustrated above. The differences are the names and quantity of attributes that are present.
ADO and COM defined constants
ADO and OLE DB, like any COM components, make extensive use of defined constants in the examples that document the usage of properties, events, and methods. Other development environments in Visual Studio such as Visual Basic and Visual Interdev provide IntelliSense technology, because of their respective abilities to interact directly with the type libraries of COM components. For these development environments, you can reference defined constants just as if they were a part of the native language. So, working with published examples is a fairly trivial task. On the other hand, in the Visual FoxPro development environment there is, in fact, a bit of a challenge. The question always seems to be “How can I use the Visual Basic samples in Visual FoxPro?” The biggest stumbling block is usually in finding the value of the defined constants. In Visual FoxPro, you need to use the #Define statement for each constant.
One solution for obtaining the value of the ADO defined constants is to obtain the MDAC SDK from Microsoft. The MDAC SDK can be downloaded from https://www.microsoft.com/data/download.htm.
Once you install the SDK, locate the IncludeADO directory. In that directory, you will find the ADOINT.H file, which contains all of the enumerated types and the values for the defined constants.
A second, and perhaps easier, solution is to use the resources already installed on your machine. If you are working through the sample code in this paper, you already have the Microsoft Data Access Components installed on your workstation. The Visual Basic Development Environment (both the full Visual Basic IDE and the Visual Basic Editor in desktop applications like Word and Excel) has a great resource called the Object Browser. This could, in fact, be the most underutilized tool on the planet.
To illustrate its functionality, open any desktop application that uses Visual Basic, such as Word or Excel. Or, if you have the Visual Basic Programming System installed, you can open that as well.
If you opened a VBA application
From the View menu, choose Toolbars.
From the Toolbars menu, choose Visual Basic.
On the Visual Basic toolbar, click Visual Basic Editor.
From the Tools menu, choose References.
Check the Microsoft Data Access Objects 2.x Library.
If you opened the Visual Basic IDE
Create an empty project.
From the Project menu, select References.
Check the Microsoft Data Access Objects 2.x Library.
Now, whether you are in the VBA Editor or the VB IDE
Press F2 to display the Object Browser.
In the first combo box, select ADODB.
In the second box, type ADVARCHAR.
Press Search or Press Enter.
Clearly, the Object Browser is a powerful tool for the developer who works with COM components. Not only are the defined properties, events, and methods accessible in the Object Browser, so also are the defined constants and their respective values. Notice the value of adVarChar in the lower pane of the Object Browser.
Opening, sorting, and filtering data
One of the big advantages of using a development platform such as Visual FoxPro is its local data engine. Not only does the engine provide superior query performance, but it also provides some very flexible capabilities when it comes to both working with and presenting data. There isn’t a Visual FoxPro application that fails to sort or filter data to some degree. In Visual FoxPro, sorting is accomplished by creating a set of index tags for a table. Filtering is accomplished by using the Set Filter command. Fortunately, ADO has these capabilities as well.
You can see in the Field Attribute table that the availability of features depends on the location in which the recordset is created. It is clear that we must ensure that a client-side recordset is created.
For example, create a Connection object to the TasTrade or SQL Server Northwind database. The following code assumes that the Connection object, oConnection, has been created before you open the Recordset object.
First, we need to implement a few required #Defines:
oRecordset.Open("Select * From Customer",;
oConnection,;
adUseClient,;
adLockBatchOptimistic)
SQL Server and Visual FoxPro open data differently. Remember that when using SQL Server, you are using the OLE DB Provider for SQL Server. When you access data in Visual FoxPro, use the OLE DB Provider for ODBC, since there is no native OLE DB provider for Visual FoxPro.
The difference rests with the optional fifth argument of the Open method. The SQL Server OLE DB Provider is designed to recognize when you pass just a table name. With the ODBC OLE DB Provider, you must specify how it should interpret the Source property. By default, the ODBC OLE DB Provider expects a SQL statement. When you pass a SQL statement, there is no need to explicitly state how the provider should interpret things. The Visual FoxPro ODBC driver generates an “Unrecognized Command Verb” error message if you only specify a table name as the source and you fail to use the optional fifth argument. Note that if you use the ODBC OLE DB Provider to access SQL Server, you must employ the same technique that is needed for Visual FoxPro.
Which method should you employ when you populate the properties individually before invoking the Open method or passing the arguments to the Open method? Once again, it is a matter of preference. Of the two, manually populating the properties makes for more readable code.
Sorting and filtering data are just matters of manipulating the Sort and Filter properties respectively. The following code sorts the recordset created from TasTrade in the example above, by country, ascending, then by region, descending:
oRecordset.Sort = "Country,Region Desc"
The following code displays the sort and the functionality of the AbsolutePosition and Bookmark properties.
oRecordset.MoveFirst
Do While Not oRecordset.Eof
With oRecordset
?.Fields("country").Value,;
.Fields("region").Value,;
.AbsolutePosition,;
.Bookmark
.MoveNext
EndWith
EndDo
Setting a filter is as easy as setting the sort. The following code filters for records where the country is Germany:
oRecordset.Filter = "Country = 'Germany'"
The Filter property also supports multiple values:
oRecordset.Filter = "Country = 'Germany' Or Country = 'Mexico'"
Finally, wild card characters are also supported:
oRecordset.Filter = "Country Like 'U*'"
To reset either the Filter or Sort properties, set them equal to an empty string:
oRecordset.Sort = ""
oRecordset.Filter = ""
Finding data
Another important capability of an ADO recordset is the ability to find records based on a search string. This capability works like searching for records in a Visual FoxPro cursor. Unlike the Seek or Locate statement in Visual FoxPro, the Find method provides control over the scope of records that are searched. The following code searches for a country that begins with the letter “B.”
oRecordset.Find("country Like 'B%'")
Although multiple criteria are not allowed, wild card searches are permitted:
oRecordset.Find("country Like 'U*'")
Searches for multiple criteria, such as the following, would result in an error:
oRecordset.Find("country Like 'G*' Or country Like 'B*'")
Transactions/updating data/conflict resolution
Updating data in an ADO recordset is a fairly simple process. As in any environment, conflict resolution in multi-user environments is always an issue to be dealt with. This is where the Errors collection comes into play. Error trapping and handling needs to become an integral part of your ADO-related code. The following code samples employ a simple error handling scenario and use the Errors collection to determine whether conflicts have occurred. For a complete list and description of ADO error codes, consult the online documentation.
When you update data, you can update either a single row, or several rows at a time in batch mode. These methods most closely correspond to row and table buffering, respectively, in Visual FoxPro. Building on the recordset already created, the lock type is Batch Optimistic. While updates are normally conducted in batches, you can also update one row at a time, just as in Visual FoxPro.
The following code modifies the CompanyName field and attempts to update the SQL Server data source:
Depending on a variety of scenarios, this code may or may not work. Perhaps a contention issue exists? Perhaps the user does not have rights to modify data. Hundreds of issues can cause an attempted update to fail. Therefore, anytime you attempt an update, you should employ error trapping. The following code expands the previous example and makes it a bit more robust:
Local Err,cOldErr,oError
cOldError = On("Error")
On Error Err = .T.
oRecordset.Fields("companyname").Value = "Ace Tomato Company"
oRecordset.Update
If Err
For Each oError In oRecordset.ActiveConnection.Errors
With oError
?.Number,.Description
EndWith
Next oError
Endif
On Error &cOldErr
If you are thinking, “Hey, maybe I should write a wrapper class to better encapsulate and centralize code,” you’re on the right track. The following code creates a custom class that can serve as a starting point:
Local oRecordsetHandler
oRecordsetHandler = CreateObject("RecordsetHandler")
oRecordset.Fields("companyname").Value = "Alfreds Futterkiste"
If !oRecordsetHandler.Update(oRecordset)
oRecordsetHandler.Cancel(oRecordset)
Endif
Define Class RecordsetHandler As Custom
Protected oRecordset
Protected ErrFlag
Procedure Update(oRecordset)
This.oRecordset = oRecordset
oRecordset.UpdateBatch
Return !This.ErrFlag
EndProc
Procedure Cancel(oRecordset)
This.oRecordset = oRecordset
oRecordset.Cancel
Return !This.ErrFlag
EndProc
Procedure Error(nError, cMethod, nLine)
Local oError
For Each oError In This.oRecordset.ActiveConnection.Errors
With oError
?.Number,.Description
EndWith
Next oError
This.ErrFlag = .T.
EndProc
EndDefine
There’s a better way to determine whether an update proceeded successfully. The preferred approach is to trap events that ADO fires. Visual FoxPro by itself does not surface COM Events. Fortunately, the new VFPCOM.DLL component provides this capability to Visual FoxPro. The previous example can be modified to show how using COM Events makes for more robust code and class design.
Now we can improve the code of our example. Most of the time, for efficiency, you will want to batch your updates that comprise multiple records. Often, when you update multiple records, transaction processing is required. In other words, either updates to all records must succeed or none should occur. To illustrate, let’s say you must apply a 10 percent price increase to the products you sell. The prime requirement is that all records in the Products table need modification. Without transactional capabilities, the possibility exists that, for example, after the first 10 records are updated, an error generated on the eleventh record prevents a complete update. Transaction processing provides the ability to rollback changes.
The following example incorporates error trapping and the three transaction methods of the Connection object:
Local Err,cOldErr
cOldErr = On("error")
On Error Err = .T.
oRecordset.ActiveConnection.BeginTrans
Do While !oRecordset.Eof
If Err
Exit
Else
With oRecordset
.Fields("unitprice").Value = ;
.Fields("unitprice").Value * 1.1
.Movenext
EndWith
Endif
EndDo
oRecordSet.UpdateBatch
If Err
oRecordset.ActiveConnection.RollBackTrans
oRecordset.CancelBatch
Else
oRecordset.ActiveConnection.CommitTrans
Endif
On Error &cOldErr
Additional operations you are likely to employ with recordsets deal with adding new records and deleting existing records. Both of these processes are very simple. The following code adds a new record:
oRecordset.AddNew
As in Visual FoxPro, in ADO the new record becomes current. Once the AddNew method is invoked, the field can be populated and, depending on the LockType, you then invoke either the Update or UpdateBatch methods to modify the data source.
Deleting records is just as easy. The following code deletes the current record:
oRecordset.Delete
Once again, after deleting the record, a call to Update or UpdateBatch will update the data source.
SQL Server identity fields and parent/child relationships
SQL Server, like most server RDBMSs and Microsoft® Access®, creates an auto-incrementing field that can serve as a primary key for a table. Typically, the data type for this field is Integer. In SQL Server, this type of field is called the Identity field. Fields of this type are read-only. It begs the question, “When adding records, how can one determine what these values are?” Knowing that the next generated value is a requirement for maintaining referential integrity when child tables are involved. The following example code shows a recordset in which the first field, ID, is the auto-incrementing field. After new field is added, checking the value of the ID field yields a character with a length of zero. Attempting to update the field results in an error. However, once the recordset is updated, checking the value again will yield a valid identity value.
oRecordset.AddNew
?oRecordset.Fields("id").Value && empty string
oRecordset.UpdateBatch
?oRecordset.Fields("id").Value && returns new identity value
With the new identity value available, you can add records in child tables, using the identity value in the parent table as the foreign key in the child tables.
But, what do you do in cases where you have disconnected recordsets?
This section details an important capability in ADO—the ability to have recordsets without an active connection to the backend data source. At this point you can freely add new records to disconnected records. When the recordset is eventually reconnected, those newly added records are then sent to the backend data source. How do you know what the identity value will be in those cases? Simply put, you don’t know. At the same time, however, you still need to be able to add both parent and child records locally. You need some method that maintains the relationship locally, while at the same time, supporting the use of the identity value when the data is sent to the backend.
The simplest solution to this problem is to include a field in each table that serves as the local ID. You need this extra field because the identity field will be read-only. On the client side, you can use several methods for producing an ID that is unique. One approach is to use the Windows API to fetch the next Global Unique Identifier (GUID). The following procedure outlines how the local process unfolds:
Add a new parent record.
Fetch the next GUID.
Update the local primary key column with the GUID.
Add a new child record.
Update the local primary key column with the GUID.
Update the foreign key column of the child with the GUID from its parent.
At some point, you will reconnect to the server. The update process could be performed within the context of a transaction, done one row at a time by navigating through each record. Checking the recordset Status property, which indicates whether the current record has been newly created, modified, deleted, and so on, determines whether the current row should be sent back to the server. If the record should be sent back, the parent record can be updated via the UpdateBatch method. The UpdateBatch method accepts an optional argument that specifies that only the current record be updated. By default, UpdateBatch works on all records. If the value of one is passed—corresponding to the adAffectCurrent constant—only the current record is updated. Once the update occurs, the identity value generated by the server is available. This value would then be used to update the foreign key columns of any related children. Once that process is complete, the records for that parent would be sent back to the server as well. This same process would be used if grandchild and great-grandchild relationships also existed.
The following Visual FoxPro code, from Visual FoxPro 6 Enterprise Development, by Rod Paddock, John V. Petersen, and Ron Talmage (Prima Publishing), illustrates how to generate a GUID:
Local oGuid
oGuid = CreateObject("guid")
?oGuid.GetNextGuid( )
*/ Class Definition
Define Class guid AS Custom
*/ Create protected members to hold parts of GUID
Protected data1
Protected data2
Protected data3
Protected data4
Procedure GetNextGuid
*/ The only public member. This method will return the next GUID
Local cGuid
cGuid = This.Export( )
UuidCreate(@cGuid)
This.Import(cGuid)
cGuid = This.Convert(cGuid)
Return cGuid
EndProc
Protected Procedure bintoHex(cBin)
*/ This method converts a binary value to Char by calling the Hextochar
*/ Method
Local cChars, nBin
cChars = ""
For nDigit = 1 To Len(cBin)
nBin = Asc(Substr(cBin, nDigit, 1))
cChars = cChars + This.Hex2Char(Int(nBin/16)) + ;
This.Hex2Char(Mod(nBin,16))
EndFor
Return(cChars)
EndProc
Protected Procedure hex2char(nHex)
*/ This method converts a hex value to ASCII
Local nAsc
Do Case
Case Between(nHex,0,9)
nAsc = 48 + nHex
Case Between(nHex,10,15)
nAsc = 65 + nHex - 10
EndCase
Return(Chr(nAsc))
EndProc
Procedure import(cString)
*/ This method takes the binary string and populates the 4 data
*/ properties
With This
.Data1 = Left(cString, Len(.Data1))
cString = SubStr(cString, Len(.Data1)+1)
.Data2 = Left(cString, Len(.Data2))
cString = SubStr(cString, Len(.Data2)+1)
.Data3 = Left(cString, Len(.Data3))
cString = SubStr(cString, Len(.Data3)+1)
.Data4 = Left(cString, Len(.Data4))
EndWith
Return cString
EndProc
Protected Procedure export
*/ This method creates the buffer to pass to the GUID API.
With This
.Data1 = Space(4)
.Data2 = Space(2)
.Data3 = Space(2)
.Data4 = Space(8)
EndWith
Return(This.Data1 + This.Data2 + This.Data3 + This.Data4)
EndProc
Protected Procedure Convert(cGuid)
*/ This method makes the call to the BinToHex that
*/ converts the data in the 4 data properties from
With This
cGuid = .BinToHex(.Data1) + "-" + .BinToHex(.Data2) + "-" + ;
.BinToHex(.Data3) + "-" + .BinToHex(.Data4)
Return cGuid
Endwith
EndProc
Procedure Init
*/ Declare the function in the DLL
Declare Integer UuidCreate ;
In C:WinntSystem32RPCRT4.DLL String @ UUID
Return
EndProc
EndDefine
Output is produced as follows:
Disconnected/Persisted Recordsets
One of the most powerful features of ADO is the ability to create both disconnected and persisted recordsets. A disconnected recordset is a client-side recordset that does not have a current ActiveConnection. SQL data sources, such as SQL Server, Oracle, and so on, are licensed according to the number of concurrent connections. For example, the number of people that using an application connected to SQL Server is 300. However, it has been determined that at any time, only 50 users actually use the services of a connection. A connection is needed only when data is being requested, updates are made, or a stored procedure on the database server is invoked. From a financial standpoint, it is far less expensive for a company to only purchase 50 licenses than to purchase 300. From a resource standpoint, performance should improve because the server only has the overhead of 50 connections instead of 300, of which 250 are idle at any time.
Using the ADO recordset of customer data already created, the following code disconnects the client-side recordset:
oRecordSet.ActiveConnection = Null
If you attempt to do this with a server-side recordset, an error occurs stating that the operation is not allowed on an open recordset. Once the recordset is disconnected, you can continue to work with and modify records. The following code will work:
oRecordset.MoveFirst
Do While !oRecordset.Eof
?oRecordset.Fields("companyname").Value
oRecordset.Fields("companyname").Value = ;
Upper(oRecordset.Fields("companyname").Value)
oRecordset.MoveNext
EndDo
With modified records in a client-side recordset, three basic options exist.
Cancel local changes
Marshall local changes to the server
Save (persist) the recordset locally.
You can save (persist) the recordset locally for both later use and, ultimately, for marshalling those persisted changes back to the server.
The first choice is pretty simple to implement, since it takes one line of code:
oRecordset.CancelBatch
The second choice is also simple to implement. Much of the work in updating multiple records and transactions has already been detailed. This procedure really involves two separate steps:
Then the code marshals the records by attempting the updates
Local Err,cOldErr
cOldErr = On("error")
On Error Err = .T.
With oRecordset
.ActiveConnection.BeginTrans
.UpdateBatch
If Err
.ActiveConnection.RollBackTrans
.CancelBatch
Else
.ActiveConnection.CommitTrans
Endif
EndWith
On Error &cOldErr
Often, however, there’s a need to shut things down and then reopen the recordset at another time. To be effective, the recordset must reflect incremental changes. This cycle may repeat any number of times.
To illustrate how to persist a recordset, consider again the following code that modifies records in a Recordset object:
oRecordset.MoveFirst
Do While !oRecordset.Eof
?oRecordset.Fields("companyname").Value
oRecordset.Fields("companyname").Value = ;
Upper(oRecordset.Fields("companyname").Value)
oRecordset.MoveNext
EndDo
Now you can invoke the Save method to persist the recordset:
oRecordset.Save("c:tempcustomers.rs")
At a later time, you can open the persisted recordset:
After the persisted recordset is reopened, you can use the same code, which establishes a connection to a disconnected recordset, to make additional modifications. You can marshal changes made in the persisted recordset to the underlying data source.
Hierarchical/Shaped Recordsets
Visual FoxPro not only provides the ability to work with local data, but also the ability to set up relations using the Set Relation command. When you move the record pointer in the parent table, the record pointer automatically moves in any child tables that exist. This makes working with and building interfaces for one to many relationships very simple in Visual FoxPro. Fortunately, the same capability exists in ADO, in the form of hierarchical recordsets, also referred to as shaped recordsets.
There are two necessary components when creating and working with hierarchical recordsets:
The Microsoft DataShape OLE DB Provider, MSDataShape
The Shape language, a superset of the SQL syntax
The first requirement is fairly easy to fulfill because it only entails setting the Provider property of the ADO Connection object to the proper value:
oConnection.Provider = "MSDataShape"
The second requirement, using the Data Shape language, is a bit more challenging. When you first see Data Shape language, it can be fairly intimidating, just as FoxPro may have been when you first worked with it. But like anything else, with a bit of practice and patience, Microsoft Data Shape language will become second nature.
To examine Shape language, consider a parent-child common scenario of customers and orders. For each customer, zero or more orders can exist. In turn, each order can contain one or more line items. The following code employs Shape syntax to relate customers and orders in the SQL Server Northwind database:
SHAPE {SELECT * FROM "dbo"."Customers"} AS Customers APPEND ({SELECT *
FROM "dbo"."Orders"} AS Orders RELATE "CustomerID" TO "CustomerID") AS
Orders
If your first thought is, “Gee, this is like setting relations in Visual FoxPro,” you are indeed correct. It is exactly the same principle. If the Shape syntax is broken down, the task becomes manageable. The first clause in the code begins with the keyword SHAPE, to signify that what follows is not pure SQL, but rather, Data Shape language. The Data Shape language is a super-set of SQL, which is why you need to use MSDataShape as the OLE DB provider. MSDataShape can interpret and execute Shape commands. Finally, the last portion of the first command specifies that the results of the SQL statement are to be aliased as Customers.
In the next set of commands, things get a bit complicated, especially when the hierarchy is nested an additional one or two levels (this is the case when order details are added, as we’ll do in the next example).
You can interpret the keyword APPEND as “Append the results of the next SQL statement to the results of the previous SQL statement.” Of course, just appending records won’t suffice. Rather, you must provide a rule that specifies how the records are to be related. This is where the RELATE keyword comes into play.
You can interpret the RELATE keyword as, “When appending records, do so based on these join fields.” In this case, the join is between the CustomerID column in the Customers table and the CustomerID column in the Orders table.
Finally, we need to alias the data that was just appended as Orders. The following code sets up the objects and creates the hierarchical recordset:
#Include adovfp.h
Local oRecordset,oConnection,oCommand, cShpStr
oRecordset = CreateObject("adodb.recordset")
oConnection = CreateObject("adodb.connection")
cShpStr = 'SHAPE {SELECT * FROM "dbo"."Customers"} AS Customers '
cShpStr = cShpStr + 'APPEND ({SELECT * FROM "dbo"."Orders"} ;
AS Orders '
cShpStr = cShpStr + 'RELATE "CustomerID" TO "CustomerID") AS Orders'
With oConnection
.Provider = "MSDataShape"
.ConnectionString = "Data Provider=SQLOLEDB.1;Persist Security ;
Info=False;User ID=sa;Initial Catalog=Northwind;Data Source=JVP"
.Open
EndWith
With oRecordset
.ActiveConnection = oConnection
.Source = cShpStr
.CursorType = adOpenStatic
.LockType = adLockBatchOptimistic
.CursorLocation = adUseClient
.Open
EndWith
The question at this point is, “How is the data appended?” The technique is rather clever. When you append a recordset to another recordset, you do so through a Field object. If you query the Count property of the Fields collection, you discover that the value of 12 is returned. However, in SQL Server, you see that the Customers table only has 11 fields. The twelfth field, in this case, is actually a pointer to the Orders recordset. The rows in the Orders recordset for a given row in the Customers recordset are only those for that customer. The following code illustrates just how powerful hierarchical recordsets are:
oRecordset.MoveFirst
Do While !oRecordset.Eof
With oRecordset
?.Fields("Customerid").Value,.Fields("CompanyName").Value
EndWith
oOrders = oRecordset.Fields("orders").Value
Do While !oOrders.Eof
With oOrders
?Chr(9),.Fields("Customerid").Value,.Fields("orderdate").Value
.MoveNext
EndWith
EndDo
oRecordset.MoveNext
EndDo
With the basics of hierarchical recordsets out of the way, we can turn our attention to a more complicated, real-life example. The following example adds several dimensions to the recordset.
First, the Order Details table is appended to the Orders child recordset. In this case, a new field that will in turn point to the OrderDetails recordset, is added to the Orders recordset. The Products table is then appended to the OrderDetails recordset providing three levels of nesting. Appended to the Products recordset are two tables, Categories and Suppliers. Traversing up the hierarchy to the Orders recordset appends the Employees table.
This list illustrates the hierarchy and shows all the tables involved as well as the nesting scheme. When creating reports, it is quite possible that you will need all of these tables. The ability to relate tables in this fashion and the ability to display the data in a user interface or a report have always been true powers of Visual FoxPro. Before ADO, attempting all this work outside Visual FoxPro was extremely difficult, sometimes bordering on the impossible.
The following is the Shape syntax to create the hierarchical recordset:
SHAPE {SELECT * FROM "dbo"."Customers"} AS Customers APPEND (( SHAPE
{SELECT * FROM "dbo"."Orders"} AS Orders APPEND (( SHAPE {SELECT * FROM
"dbo"."Order Details"} AS OrderDetails APPEND (( SHAPE {SELECT * FROM
"dbo"."Products"} AS Products APPEND ({SELECT * FROM "dbo"."Categories"}
AS Categories RELATE 'CategoryID' TO 'CategoryID') AS Categories,({SELECT
* FROM "dbo"."Suppliers"} AS Suppliers RELATE 'SupplierID' TO
'SupplierID') AS Suppliers) AS Products RELATE 'ProductID' TO
'ProductID') AS Products) AS OrderDetails RELATE 'OrderID' TO 'OrderID')
AS OrderDetails,(( SHAPE {SELECT * FROM "dbo"."Employees"} AS Employees
APPEND (( SHAPE {SELECT * FROM "dbo"."EmployeeTerritories"} AS
EmployeeTerritories APPEND (( SHAPE {SELECT * FROM "dbo"."Territories"}
AS Territories APPEND ({SELECT * FROM "dbo"."Region"} AS Region RELATE
'RegionID' TO 'RegionID') AS Region) AS Territories RELATE 'TerritoryID'
TO 'TerritoryID') AS Territories) AS EmployeeTerritories RELATE
'EmployeeID' TO 'EmployeeID') AS EmployeeTerritories) AS Employees RELATE
'EmployeeID' TO 'EmployeeID') AS Employees,({SELECT * FROM
"dbo"."Shippers"} AS Shippers RELATE 'ShipVia' TO 'ShipperID') AS
Shippers) AS Orders RELATE 'CustomerID' TO 'CustomerID') AS Orders
This is just about as complicated as it gets. Nobody in their right mind would want to hammer this code out manually. Fortunately, there is a visual way to build this code. The DataEnvironment designer that ships with Visual Basic allows you to visually design ADO connections, recordsets, and hierarchical recordsets. The following illustrates how this hierarchical recordset appears in the designer:
The extensive Shape syntax can be copied and pasted into Visual FoxPro, or any other environment that can host ADO. For complete details on how to use the DataEnvironment designer, consult the Visual Basic documentation on the MSDN CDs that ship with Visual Studio.
The following Visual FoxPro code traverses the hierarchical recordset and displays the data:
#Include adovfp.h
oRecordset = CreateObject("adodb.recordset")
oConnection = CreateObject("adodb.connection")
cShpStr = 'SHAPE {SELECT * FROM "dbo"."Customers"} AS Customers APPEND'
cShpStr = cShpStr + '(( SHAPE {SELECT * FROM "dbo"."Orders"} AS Orders '
cShpStr = cShpStr + 'APPEND (( SHAPE {SELECT * FROM "dbo"."Order
Details"} AS OrderDetails '
cShpStr = cShpStr + 'APPEND (( SHAPE {SELECT * FROM "dbo"."Products"}
AS Products '
cShpStr = cShpStr + 'APPEND ({SELECT * FROM "dbo"."Categories"} AS
Categories '
cShpStr = cShpStr + 'RELATE "CategoryID" TO "CategoryID") AS
Categories,'
cShpStr = cShpStr + '({SELECT * FROM "dbo"."Suppliers"} AS Suppliers '
cShpStr = cShpStr + 'RELATE "SupplierID" TO "SupplierID") AS Suppliers)
AS Products '
cShpStr = cShpStr + 'RELATE "ProductID" TO "ProductID") AS Products) AS
OrderDetails '
cShpStr = cShpStr + 'RELATE "OrderID" TO "OrderID") AS OrderDetails,'
cShpStr = cShpStr + '(( SHAPE {SELECT * FROM "dbo"."Employees"} AS
Employees '
cShpStr = cShpStr + 'APPEND (( SHAPE {SELECT * FROM
"dbo"."EmployeeTerritories"} AS EmployeeTerritories '
cShpStr = cShpStr + 'APPEND (( SHAPE {SELECT * FROM "dbo"."Territories"} AS Territories '
cShpStr = cShpStr + 'APPEND ({SELECT * FROM "dbo"."Region"} AS Region '
cShpStr = cShpStr + 'RELATE "RegionID" TO "RegionID") AS Region) AS
Territories '
cShpStr = cShpStr + 'RELATE "TerritoryID" TO "TerritoryID") AS
Territories) AS EmployeeTerritories '
cShpStr = cShpStr + 'RELATE "EmployeeID" TO "EmployeeID") AS
EmployeeTerritories) AS Employees '
cShpStr = cShpStr + 'RELATE "EmployeeID" TO "EmployeeID") AS Employees,'
cShpStr = cShpStr + '({SELECT * FROM "dbo"."Shippers"} AS Shippers '
cShpStr = cShpStr + 'RELATE "ShipVia" TO "ShipperID") AS Shippers) AS
Orders '
cShpStr = cShpStr + 'RELATE "CustomerID" TO "CustomerID") AS Orders '
With oConnection
.Provider = "MSDataShape"
.ConnectionString = "Data Provider=SQLOLEDB.1;Persist Security
Info=False;User ID=sa;Initial Catalog=Northwind;Data Source=JVP"
.Open
EndWith
With oRecordset
.ActiveConnection = oConnection
.Source = cShpStr
.CursorType = adOpenStatic
.LockType = adLockBatchOptimistic
.CursorLocation = adUseClient
.Open
EndWith
Do While !oRecordset.Eof
With oRecordset
?.Fields("CustomerID").Value,.Fields("CompanyName").Value
EndWith
oOrders = oRecordset.Fields("orders").Value
Do While !oOrders.Eof
oShippers = oOrders.Fields("shippers").Value
oEmployee = oOrders.Fields("employees").Value
oEmployeeTerritories =
oEmployee.Fields("employeeterritories").Value
oTerritories = oEmployeeTerritories.Fields("territories").Value
oRegion = oTerritories.Fields("region").Value
?"Order ID: ",oOrders.Fields("orderid").Value,;
"Order Date: ",oOrders.Fields("orderdate").Value
oOrderDetails = oOrders.Fields("orderdetails").Value
?"Territory: ",
oTerritories.Fields("territorydescription").Value,;
"Region: ",oRegion.Fields("RegionDescription").Value
?"Shipper: ",oShippers.Fields("companyname").Value
oEmployee = oOrders.Fields("employees").Value
With oEmployee
?"Employee: ",.Fields("employeeid").Value,;
.Fields("firstname").Value + " " + .Fields("lastname").Value
EndWith
?"Order Details: "
Do While !oOrderDetails.Eof
oProducts = oOrderDetails.Fields("Products").Value
oCategories = oProducts.Fields("categories").Value
oSuppliers = oProducts.Fields("suppliers").Value
?Chr(9),;
oProducts.Fields("productname").Value,;
oSuppliers.Fields("companyname").Value,;
oCategories.Fields("categoryname").Value,;
oOrderDetails.Fields("Quantity").Value,;
oOrderDetails.Fields("UnitPrice").Value
oOrderDetails.MoveNext
EndDo
oOrders.MoveNext
EndDo
oRecordset.MoveNext
EndDo
The output appears as follows:
Because a hierarchy exists, the ability to create drill-down interfaces becomes a fairly simple task. The preceding Visual FoxPro code illustrates how to traverse the hierarchy.
Perhaps you want to use Microsoft Word or Excel as a reporting tool. With a combination of Visual FoxPro COM servers, ADO, and Automation, the process becomes manageable. The first and third parts of the solution have been around. However, only now that a set of COM objects exists to handle and work with data as Visual FoxPro does natively can the solution become a reality.
Hierarchical recordsets and recursive relationships
One of the nice features of SQL Server, and of most other server back ends is provision for recursive relations. The following is the SQL Server 7.0 database diagram for the Northwind database:
In the Northwind database, the Employees table employs recursion to support a manager/staff relationship. Both managers and staff are employees. In some cases, it happens that some employees report to other employees. In Visual FoxPro, you can create the same sort of relation by opening a table twice using two different aliases. In ADO, the task is totally supported and is quite easy to implement. The following is the Shape syntax:
SHAPE {SELECT * FROM "dbo"."Employees"} AS Managers APPEND ({SELECT *
FROM "dbo"."Employees"} AS Staff RELATE 'EmployeeID' TO 'ReportsTo') AS
Staff
The following Visual FoxPro code displays a list of managers and the staff that reports to each manager:
#Include adovfp.h
oRecordset = CreateObject("adodb.recordset")
oConnection = CreateObject("adodb.connection")
cShpStr = 'SHAPE {SELECT * FROM "dbo"."Employees"} AS Managers '
cShpStr = cShpStr + 'APPEND ({SELECT * FROM "dbo"."Employees"} AS Staff '
cShpStr = cShpStr + 'RELATE "EmployeeID" TO "ReportsTo") AS Staff '
With oConnection
.Provider = "MSDataShape"
.ConnectionString = "Data Provider=SQLOLEDB.1;Persist Security
Info=False;User ID=sa;Initial Catalog=Northwind;Data Source=JVP"
.Open
EndWith
With oRecordset
.ActiveConnection = oConnection
.Source = cShpStr
.CursorType = adOpenStatic
.LockType = adLockBatchOptimistic
.CursorLocation = adUseClient
.Open
EndWith
Do While !oRecordset.Eof
oStaff = oRecordset.Fields("staff").Value
If oStaff.Recordcount > 0
With oRecordset
?.Fields("firstname").Value + " " + ;
.Fields("lastname").Value ,;
.Fields("Title").Value
Do While !oStaff.Eof
With oStaff
?Chr(9),;
.Fields("firstname").Value + " " + ;
.Fields("lastname").Value ,;
.Fields("Title").Value
EndWith
oStaff.MoveNext
EndDo
EndWith
Endif
oRecordset.MoveNext
EndDo
The output appears as follows:
Finally, note that hierarchical recordsets are updateable. The following code expands the previous example to illustrate how to make a simple update:
Do While !oRecordset.Eof
oStaff = oRecordset.Fields("staff").Value
If oStaff.Recordcount > 0
With oRecordset
Do While !oStaff.Eof
With oStaff
.Fields("firstname").Value = ;
Upper(.Fields("firstname").Value)
.Fields("lastname").Value = ;
Upper(.Fields("lastname").Value)
.Fields("Title").Value = ;
Upper(.Fields("Title").Value)
EndWith
oStaff.MoveNext
EndDo
*/ Write changes to Staff recordset
oStaff.UpdateBatch
EndWith
Endif
oRecordset.MoveNext
EndDo
The ability to view related records, coupled with the ability to make updates, places the ADO hierarchical recordset capability on par with similar capabilities in Visual FoxPro.
Multiple recordsets
Use of hierarchical recordsets represents only one method for returning data from multiple recordsets in one object. For starters, building hierarchical recordsets is not the most straightforward of propositions. In many cases, a simpler alternative may be all that is required.
Consider the case where you need a specific customer record and the orders for that customer. Yes, you could use a hierarchical recordset. But, there is a simpler way: run two SQL statements.
Some OLE DB providers can process multiple SQL Statements. The OLE DB Provider for SQL Server has this capability. Attempting to do this with Visual FoxPro tables via the OLE DB Provider for ODBC will not work.
When using this technique, you have two choices on where the logic exists to perform the task. One choice is to build the SQL on the client and pass it to the server through a Command object. The other choice is to invoke a stored procedure on the database server through a Command object. I’ll illustrate both techniques. The Command object will be discussed in detail later in this paper.
To illustrate the stored procedure method, the following stored procedure must be created on the SQL Server Northwind database:
CREATE PROCEDURE CustomerAndOrders @CustomerID nchar(5)
AS
Select * From Customers Where Customers.CustomerID = @CustomerID
Select * From Orders Where Orders.CustomerID = @CustomerID
With the stored procedure created, the following code will create the recordset:
#Include adovfp.h
oConnection = CreateObject("adodb.connection")
oCommand = CreateObject("adodb.command")
With oConnection
.Provider = "SQLOLEDB.1"
.ConnectionString = ;
"Persist Security Info=False;User ID=sa;Initial
Catalog=Northwind;Data Source=JVP"
.Open
EndWith
With oCommand
.CommandText = "CustomerAndOrders"
.ActiveConnection = oConnection
.CommandType = adCmdStoredProc
EndWith
oCommand.Parameters("@CustomerID").Value = "ALFKI"
oRecordset = oCommand.Execute
Do While !oRecordset.Eof
?oRecordset.Fields(1).Value
oRecordset.MoveNext
EndDo
oRecordset = oRecordset.NextRecordset
Do While !oRecordset.Eof
?oRecordset.Fields(0).Value
oRecordset.MoveNext
EndDo
Like any recordset, the recordset just produced can be navigated. Once the first set of records from the Customers table have been navigated, the NextRecordset method is invoked. This causes the recordset produced by the second SQL statement to become available. Thus, the next set of commands loops through the records from the Orders table. This technique is ideal in those situations where you may need to populate Combo or ListBox controls.
The previous example references a collection that has not been discussed yet, the Parameters collection. The Parameters collection and the individual Parameter objects that it contains serve several purposes. One purpose is to provide the capacity to create parameterized queries. Another purpose is to provide the ability to send arguments to, and return data from, a stored procedure. For more information on the Parameters collection, see the Command Object section of this paper.
Alternatively, you can produce the SQL on the client if you wish. The following code illustrates the difference:
With oCommand
.CommandText = "Select * From Customers Where CustomerID =
'ALFKI'" + Chr(13) + "Select * From Orders Where CustomerID =
'ALFKI'"
.ActiveConnection = oConnection
.CommandType = adCmdText
EndWith
oRecordset = oCommand.Execute
The same result is achieved. The difference lies in how the result is achieved.
Which approach is better?
It depends on what your requirements are. The first option, which uses stored procedures, is more secure; the code is set and you can assign permissions with regard to who can execute the stored procedure. The second option provides more flexibility, but less security.
Fabricated recordsets
Up to this point, recordset objects have been presented in the context of origination from an ADO connection. In many cases, you may want to create an ADO recordset with data that does not come from a data source, just as you may in some cases use the Create Cursor command in Visual FoxPro. For example, you may have an application that works with a small amount of data, such as an array or Visual FoxPro cursor. Perhaps you need to dynamically build a table structure. Whatever the reason, the ability to create ADO recordsets from scratch is powerful.
To illustrate this capability, consider the need to fetch a list of files from a specified directory. In Visual FoxPro, a handy function, ADIR( ), performs this sort of task. However, what if you need to pass the data to another application? Or, perhaps you need to persist the list to a file on disk. While Visual FoxPro arrays are powerful, ADO recordsets provide a compelling alternative. The following code fetches a list of files from a specified directory, fabricates a recordset, and copies the values from the array into the newly created recordset:
*/GetFiles.prg
#INCLUDE "adovfp.h"
Local Array aFiles[1]
Local nFiles,nField,nFile,oRS
nFiles = Adir(aFiles,Getdir( )+"*.*")
oRS=Createobject("adodb.recordset")
With oRS
.CursorLocation=ADUSECLIENT
.LockType=ADLOCKOPTIMISTIC
*/ Adding new fields is a matter of appending
*/ new field objects to the Fields Collection.
.Fields.Append("File",ADCHAR,20)
.Fields.Append("Size",ADDOUBLE,10)
.Fields.Append("DateTime",ADDBTIME,8)
.Fields.Append("Attributes",ADCHAR,10)
.Open
EndWith
For nFile = 1 To nFiles
*/ Add a new record. This automatically makes
*/ the new record the current record - just
*/ like VFP.
oRS.AddNew
With ors
.Fields("File").Value = aFiles[nFile,1]
.Fields("Size").Value = aFiles[nFile,2]
.Fields("DateTime").Value = ;
Ctot(Dtoc(aFiles[nFile,3]) + " " + aFiles[nFile,4])
.Fields("Attributes").Value = aFiles[nFile,5]
EndWith
Next nItem
Return oRS
With the new recordset created and populated, it can be navigated like any other recordset:
oFiles = GetFiles ( )
Do While !oFiles.Eof
?oFiles.Fields("File").Value
oFiles.movenext
EndDo
ADO recordsets instead of arrays
Referring to the previous example, let’s say that the list needs to be sorted by file size, descending. Arrays in Visual FoxPro can be sorted, when all columns in the array are of the same data type. In this case, there are three data types: Character, Numeric, and DateTime. With a client-side ADO recordset, the process becomes simple. The following code does the trick:
oRS.Sort = "Size Desc"
Sorts are not limited to just one column. Perhaps you need to sort by size, descending, and then by file, ascending:
oRS.Sort = "Size Desc,File"
And, when it comes to sorting, such properties as Bookmark and AbsolutePosition that have already been demonstrated are available here as well.
Perhaps you need to find a specific value. The ASCAN( ) function in Visual FoxPro enables you to do this. However, it does not allow you to specify a particular column to search. Rather, once the first occurrence of a specified value is found, regardless of the column, the search is stopped. With ADO recordsets, more granular control is provided. The following code checks to see if a file called VFP6.EXE is in the recordset:
oRS.Find("File Like 'VFP6.EXE'")
If !oRS.Eof
*/ Found it
Else
*/ Not found
Endif
Finally, you may wish to filter the list based on the file size being greater than a specified value:
oRS.Filter = "size > 50000"
When evaluating the tools at your disposal for local data handling, be sure to consider fabricated ADO recordsets. Also, if you find yourself running into obstacles with Visual FoxPro arrays, fabricated ADO recordsets may provide a sound alternative.
Command Object
ProgID: ADODB.Command
The purpose of the Command object is just as the its name implies, to run commands. For example, you may need to run a SQL update against a SQL Server table. To illustrate, the following code applies a 10 percent increase in the UnitPrice field in the Products table of the SQL Server Northwind database:
oCommand = CreateObject("adodb.command")
With oCommand
.ActiveConnection = oConnection
.CommandText = "Update Products Set unitprice = unitprice * 1.1"
.Execute
EndWith
The ActiveConnection property
To review, both the Command object and Recordset object have the ActiveConnection property. A Command object needs to know what data source it is to execute commands against. A Recordset object needs to know what data source contains the data it is to retrieve. The way you accomplish this is by setting the ActiveConnection property.
The ActiveConnection property presents a great opportunity to talk about the flexible nature of the ADO object model. The ADO object model is very flat, in that you do not have to create a series of objects in order to gain access to other objects. For example, the following is one way to create and open both a Connection and a Recordset object:
Now, you can reference the Connection object because it has been implicitly created from the passed connection string:
?oRecordset.ActiveConnection.ConnectionString
The same is true for the Command object. While a Command object was not explicitly created, a Command object was in fact created and actually did the work of creating the recordset. Using the recordset just created, the following command will yield “Products” as the CommandText:
?oRecordset.ActiveCommand.CommandText
Which method should you use?
It is really a matter of preference. The latter method, which uses only the RecordSet object, is somewhat overloaded. It carries the same overhead as the former method because you must still create a Connection object. The former method is probably a better way to go as it makes for more readable code.
Parameters collection
The Parameters collection works with the Command object. The primary use of the Parameters Collection is to both pass arguments to, and accept return values from stored procedures. To illustrate, consider the CustOrderHist stored procedure in the SQL Server Northwind database:
CREATE PROCEDURE CustOrderHist @CustomerID nchar(5)
AS
SELECT ProductName, Total=SUM(Quantity)
FROM Products P, [Order Details] OD, Orders O, Customers C
WHERE C.CustomerID = @CustomerID
AND C.CustomerID = O.CustomerID AND O.OrderID = OD.OrderID AND
OD.ProductID = P.ProductID
GROUP BY ProductName
To illustrate how the Parameters collection is used in conjunction with the Command object, consider the following comprehensive example:
At this point, information can be obtained from the Parameters collection:
For Each Parameter in oCommand.Parameters
?Parameter.Name,Parameter.Size,Parameter.Type
Next Parameter
The first Parameter object is reserved for the value that the stored procedure may return. Regardless of whether the stored procedure explicitly returns a value, this Parameter object will be created. Examining the CustOrderHist stored procedure, note that a single argument, a customer ID, is accepted.
With a Command object and Parameter object in place, the real work can begin. To get things rolling, a value needs to be assigned to the Parameter object that will in turn be passed to the stored procedure. In this case, a SQL statement is executed that totals the quantity, by product, that a specified customer has purchased. The following code provides a customer ID and executes the stored procedure:
Yet another way to produce a Recordset object is through the execution of a stored procedure. The resulting Recordset object contains two fields that correspond to the select statement in the CustOrderHist stored procedure. Need a different history? Just update the Value property of the Parameter object and invoke the Execute method of the Command object.
The Parameters collection also comes into play in the area of parameterized queries. Consider the following SQL Statement:
Select * ;
From Customer ;
Where country = ? And max_order_amt > ?
As with views, either local or remote, in Visual FoxPro, so too can queries be parameterized in ADO. In ADO, the question mark acts as a placeholder for parameters. The following example illustrates how to put this all together.
First, a connection and a Command object need to be created:
The arguments for the CreateParameter method are as follows:
Name—The name of the parameter.
Type—The data type of the parameter. A list of valid values is contained in DataTypeEnum.
Direction—The direction of the parameter. Parameters sent to a command are input parameters. Arguments passed back from a command are output parameters. A list of valid values is contained in ParameterDirectionEnum.
Size—The length of the parameter.
Value—The initial value of the parameter.
Alternatively, the parameter could have been created like this:
Once the parameter has been created, it needs to be appended into the Parameters collection of the Command object:
oCommand.Parameters.Append(oCountryParameter)
With the parameter in place, the value of the parameter can be set. In this case, the parameter will be set so that any country that begins with the letter U will be returned into a Recordset object:
With oCountryParameter
.Size = 2
.Value = "U%"
EndWith
Now, a Recordset object can be created:
oRecordset = oCommand.Execute
A useful feature of specifying parameters is that this enforces characteristics such as size, data type, and so on. For example, the preceding parameter was defined as a character. If a value based on a different data type was assigned to the Value property of the Parameter object, an error would result. The same is true if the assigned value is greater in length than what has been specified by the Size property.
Finally, if a list of customers in Mexico were required, the following code would complete the task:
Recall the earlier assertion that, by itself, ADO is incapable of doing anything? ADO in fact just provides an interface. OLE DB providers give ADO the ability to do anything. So then, what distinguishes one OLE DB provider from another? More specifically, how can you determine what an OLE DB provider can and cannot do, or what attributes it does or does not possess? Depending on the OLE DB provider you use, or the type of recordset you use (client or server), what is supported will likely differ.
The Properties collection applies to the Connection, Recordset, and Field objects. The Command object also has a Properties collection, which is identical to the Recordset object Properties collection.
Multiple result sets provide a good example of varying OLE DB provider support. To determine if multiple result sets can be obtained, you can refer to the “Multiple Results” properties:
If oConnection.Properties("Multiple Results").Value = 1
*/ Supports multiple result sets
EndIf
While the OLE DB providers for SQL Server and ODBC both support multiple results, the OLE DB provider for Jet does not. To illustrate, the following is valid syntax for SQL Server:
oRecordset.Source="SELECT * FROM customers;"+"SELECT * FROM orders"
oRecordset.Open
?oRecordSet.Fields.Count && number of fields in customers table
oRecordset = oRecordset.NextRecordSet
?oRecordSet.Fields.Count && number of fields in orders table
In this case, the OLE DB Provider for SQL Server can return multiple recordsets. If you attempt the same thing with the OLE DB Provider for ODBC, which you need to use when accessing Visual FoxPro data, you will receive an error message stating that the requested action is not supported by the OLE DB provider.
Another example involves the way in which the Properties collection deals with the location of a Recordset object. Recordsets can either exist locally as client-side recordsets or they can exist remotely as server-side recordsets. Client-side recordsets, as will be discussed shortly, have several capabilities that server-side recordsets do not have. One of these abilities is to create indexes. The following code creates a client-side recordset:
In the absence of a declaration of where a Recordset object should reside, the Recordset object, by default, resides on the server. Attempting to reference the Optimize property results in an error stating that the specified property could not be found in the collection.
While the ADO interface is constant, depending on the provider you use, the capabilities may be very different. Be sure to consult your provider’s documentation.
Remote Data Services
One of the most powerful data access capabilities introduced by Microsoft is Remote Data Services (RDS). Although a separate set of objects exists for RDS, RDS is really just another component for use with ADO. There are two ways you can implement RDS.
Use the same ADO objects described in this paper
Use the RDS data control
Let’s discuss the RDS data control option first, since it represents some uncharted territory.
The RDS Data Control
The following code creates an instance of the RDS data control:
oRDSDataControl = Createobject("rds.datacontrol")
Once the data control is created, only three properties need to be populated: Server, Connect, and SQL.
Because we’re using the SQL Server OLE DB Provider, the SQL property can consist of just the table name. The following code retrieves the same recordset, but does so with the OLE DB provider for ODBC:
From this point on, you can work with the recordset the same way you work with any other ADO client-side recordset:
Do While !oRecordset.Eof
orecordset.Fields(1).value = ;
Proper(orecordset.Fields(1).value)
oRecordset.Movenext
EndDo
oRecordset.Updatebatch
Alternatively, you can replace the last line of code with a call to the SubmitChanges method of the RDS data control:
oRDSDataControl.SubmitChanges
Implementing RDS Through the ADO Interface
You can invoke RDS by using the same ADO Connection object discussed above. As with hierarchical recordsets, the first step involves the selection of an OLE DB provider. In this case, the MSRemote provider is required. The following code sets up the Connection object:
The ADO ConnectionString property supports only four arguments. The first two, Provider and File Name, have already been discussed. The third and fourth, Remote Provider and Remote Server, are used by the RDS in the example above. The Remote Provider is the same OLE DB provider used when you create local connections. The additional parameters that specify the database, user ID, and password are used by the OLE DB Provider for SQL Server that in turn is located on the remote server. The following code connects the Recordset object and Connection object and with one difference, is basically the same as the previous examples in this paper:
The only difference is that properties such as CursorLocation and CursorType are omitted since all recordsets created through RDS must exist on the client. Additionally, all client-side recordsets are static types. If you like, you can still specify the properties explicitly. Any incompatible properties will be coerced to a valid value. For example, if you specify the CursorType to be a ForwardOnly cursor and you specify the recordset exists on the client, when the Open method is fired, ADO forces the cursor type to be static. The same is true if you specify the CursorLocation to be on the server and you use the MSDataShape provider. Since all hierarchical pecordsets must exist on the client, the CursorLocation is coerced to the proper value.
Summary
The goal of this paper has been to provide you with a fairly comprehensive overview of both ADO and RDS from the perspective of Visual FoxPro applications. Note that ADO is not a replacement for the Visual FoxPro Cursor Engine. Rather, regard it as another tool at your disposal. Both Visual FoxPro cursors and ADO recordsets have their relative strengths and weaknesses.
ADO is ideal in situations where your application is component based, or in situations where you need to pass data to other applications such as Excel in automation operations. Fabricated ADO recordsets can provide an interesting alternative to arrays when more robust data handling requirements are necessary.
For most local data handling operations however, Visual FoxPro cursors will usually provide better results.
John V. Petersen, MBA, is president of Main Line Software, Inc., based in Philadelphia, Pennsylvania. John’s firm specializes in custom software development and database design. He is a Microsoft Most Valuable Professional and has spoken at many developer events, including Visual FoxPro Developers Conference, FoxTeach, the Visual FoxExpress Developer’s Conference, DevDays, and TechEd. In addition, John has written numerous articles for FoxTalk and FoxPro Advisor. John is co-author of Visual FoxPro 6 Enterprise Development and Hands-on Visual Basic 6—Web Development, both from Prima Publishing. John’s latest project is the ADO Developer’s Handbook, from Sybex Publishing, due September 1999.
E-mail: jpetersen@mainlinesoftware.com
Books
“Programming FoxPro 2.5” by Lisa Slater and Steven D. Arnott
A foundational book for beginners and intermediate users of FoxPro 2.5. Covers database management and programming techniques.
“The Revolutionary Guide to Visual FoxPro 3.0” by Kevin McNeish and Marilyn McLain
This book explores Visual FoxPro 3.0, detailing object-oriented programming features and database development.
“Visual FoxPro 6.0 Programmer’s Guide” by Tamar E. Granor, Ted Roche, Doug Hennig, and Dave Fulton
A comprehensive guide to using Visual FoxPro 6.0 for professional database application development.
“Advanced Object-Oriented Programming with Visual FoxPro 6.0” by Markus Egger
Focuses on object-oriented techniques, design patterns, and advanced development strategies.
“Hacker’s Guide to Visual FoxPro 7.0” by Tamar E. Granor and Ted Roche
A deep dive into advanced features and lesser-documented aspects of Visual FoxPro 7.0.
“Microsoft Visual FoxPro: Language Reference”
Official Microsoft documentation providing a detailed reference to the Visual FoxPro language, commands, and functions.
“Fundamentals of FoxPro 2.x Programming” by George Goley
Covers foundational programming techniques and database management in early FoxPro versions.
Affiliate Disclosure: This blog may contain affiliate links, which means I may earn a small commission if you click on the link and make a purchase. This comes at no additional cost to you. I only recommend products or services that I believe will add value to my readers. Your support helps keep this blog running and allows me to continue providing you with quality content. Thank you for your support!
This text provides a comprehensive introduction to data science, covering its growth, career opportunities, and required skills. It explores various data science tools, programming languages (like Python and R), and techniques such as machine learning and deep learning. The materials also explain how to work with different data types, perform data analysis, build predictive models, and present findings effectively. Finally, it examines the role of generative AI in enhancing data science workflows.
Python & Data Science Study Guide
Quiz
What is the purpose of markdown cells in Jupyter Notebooks, and how do you create one?
Markdown cells allow you to add titles and descriptive text to your notebook. You can create a markdown cell by clicking ‘Code’ in the toolbar and selecting ‘Markdown.’
Explain the difference between int, float, and string data types in Python and provide an example of each.
int represents integers (e.g., 5), float represents real numbers (e.g., 3.14), and string represents sequences of characters (e.g., “hello”).
What is type casting in Python, and why is it important to be careful when casting a float to an integer?
Type casting is changing the data type of an expression (e.g., converting a string to an integer). When converting a float to an int, information after the decimal point is lost, so you must be careful.
Describe the role of variables in Python and how you assign values to them.
Variables store values in memory, and you assign a value to a variable using the assignment operator (=). For example, x = 10 assigns 10 to the variable x.
What is the purpose of indexing and slicing in Python strings and give an example.
Indexing allows you to access individual characters in a string using their position (e.g., string[0]). Slicing allows you to extract a substring (e.g., string[1:4]).
Explain the concept of immutability in the context of strings and tuples and how it affects their manipulation.
Immutable data types cannot be modified after creation. If you want to change a string or a tuple you create a new string or tuple.
What are the key differences between lists and tuples in Python?
Lists are mutable, meaning you can change them after creation; tuples are immutable. Lists are defined using square brackets [], while tuples use parentheses ().
Describe dictionaries in Python and how they are used to store data using keys and values.
Dictionaries store key-value pairs, where keys are unique and immutable and the values are the associated information. You use curly brackets {} and each key and value are separated by a colon (e.g., {“name”: “John”, “age”: 30}).
What are sets in Python, and how do they differ from lists or tuples?
Sets are unordered collections of unique elements. They do not keep track of order, and only contain a single instance of any item.
Explain the difference between a for loop and a while loop and how each can be used.
A for loop is used to iterate over a sequence of elements, like a list or string. A while loop runs as long as a certain condition is true, and does not necessarily require iterating over a sequence.
Quiz Answer Key
Markdown cells allow you to add titles and descriptive text to your notebook. You can create a markdown cell by clicking ‘Code’ in the toolbar and selecting ‘Markdown.’
int represents integers (e.g., 5), float represents real numbers (e.g., 3.14), and string represents sequences of characters (e.g., “hello”).
Type casting is changing the data type of an expression (e.g., converting a string to an integer). When converting a float to an int, information after the decimal point is lost, so you must be careful.
Variables store values in memory, and you assign a value to a variable using the assignment operator (=). For example, x = 10 assigns 10 to the variable x.
Indexing allows you to access individual characters in a string using their position (e.g., string[0]). Slicing allows you to extract a substring (e.g., string[1:4]).
Immutable data types cannot be modified after creation. If you want to change a string or a tuple you create a new string or tuple.
Lists are mutable, meaning you can change them after creation; tuples are immutable. Lists are defined using square brackets [], while tuples use parentheses ().
Dictionaries store key-value pairs, where keys are unique and immutable and the values are the associated information. You use curly brackets {} and each key and value are separated by a colon (e.g., {“name”: “John”, “age”: 30}).
Sets are unordered collections of unique elements. They do not keep track of order, and only contain a single instance of any item.
A for loop is used to iterate over a sequence of elements, like a list or string. A while loop runs as long as a certain condition is true, and does not necessarily require iterating over a sequence.
Essay Questions
Discuss the role and importance of data types in Python, elaborating on how different types influence operations and the potential pitfalls of incorrect type handling.
Compare and contrast the use of lists, tuples, dictionaries, and sets in Python. In what scenarios is each of these data structures more beneficial?
Describe the concept of functions in Python, providing examples of both built-in functions and user-defined functions, and explaining how they can improve code organization and reusability.
Analyze the use of loops and conditions in Python, explaining how they allow for iterative processing and decision-making, and discuss their relevance in data manipulation.
Explain the differences and relationships between object-oriented programming concepts (such as classes, objects, methods, and attributes) and how those translate into more complex data structures and functional operations.
Glossary
Boolean: A data type that can have one of two values: True or False.
Class: A blueprint for creating objects, defining their attributes and methods.
Data Frame: A two-dimensional data structure in pandas, similar to a table with rows and columns.
Data Type: A classification that specifies which type of value a variable has, such as integer, float, string, etc.
Dictionary: A data structure that stores data as key-value pairs, where keys are unique and immutable.
Expression: A combination of values, variables, and operators that the computer evaluates to a single value.
Float: A data type representing real numbers with decimal points.
For Loop: A control flow statement that iterates over a sequence (e.g., list, tuple) and executes code for each element.
Function: A block of reusable code that performs a specific task.
Index: Position in a sequence, string, list, or tuple.
Integer (Int): A data type representing whole numbers, positive or negative.
Jupyter Notebook: An interactive web-based environment for coding, data analysis, and visualization.
Kernel: A program that runs code in a Jupyter Notebook.
List: A mutable, ordered sequence of elements defined with square brackets [].
Logistic Regression: A classification algorithm that predicts the probability of an instance belonging to a class.
Method: A function associated with an object of a class.
NumPy: A Python library for numerical computations, especially with arrays and matrices.
Object: An instance of a class, containing its own data and methods.
Operator: Symbols that perform operations such as addition, subtraction, multiplication, or division.
Pandas: A Python library for data manipulation and analysis.
Primary Key: A unique identifier for each record in a table.
Relational Database: A database that stores data in tables with rows and columns and structured relationships between tables.
Set: A data structure that is unordered and contains only unique values.
Sigmoid Function: A mathematical function used in logistic regression that outputs a value between zero and one.
Slicing: Extracting a portion of a sequence (e.g., list, string) using indexes (e.g., [start:end:step]).
SQL (Structured Query Language): Language used to manage and manipulate data in relational databases.
String: A sequence of characters, defined with single or double quotes.
Support Vector Machine (SVM): A classification algorithm that finds an optimal hyperplane to separate data classes.
Tuple: An immutable, ordered sequence of elements defined with parentheses ().
Type Casting: Changing the data type of an expression.
Variable: A named storage location in a computer’s memory used to hold a value.
View: A virtual table based on the result of an SQL query.
While Loop: A control flow statement that repeatedly executes a block of code as long as a condition remains true.
Python for Data Science
Okay, here’s a detailed briefing document summarizing the provided sources, focusing on key themes and ideas, with supporting quotes:
Briefing Document: Python Fundamentals and Data Science Tools
I. Overview
This document provides a summary of core concepts in Python programming, specifically focusing on those relevant to data science. It covers topics from basic syntax and data types to more advanced topics like object-oriented programming, file handling, and fundamental data analysis libraries. The goal is to equip a beginner with a foundational understanding of Python for data manipulation and analysis.
II. Key Themes and Ideas
Jupyter Notebook Environment: The sources emphasize the practical use of Jupyter notebooks for coding, analysis, and presentation. Key functionalities include running code cells, adding markdown for explanations, and creating slides for presentation.
“you can now start working on your new notebook… you can create a markdown to add titles and text descriptions to help with the flow of the presentation… the slides functionality in Jupiter allows you to deliver code visualization text and outputs of the executed code as part of a project”
Python Data Types: The document systematically covers fundamental Python data types, including:
Integers (int) & Floats (float):“you can have different types in Python they can be integers like 11 real numbers like 21.23%… we can have int which stands for an integer and float that stands for float essentially a real number”
Strings (str):“the type string is a sequence of characters” Strings are explained to be immutable, accessible by index, and support various methods.
Booleans (bool):“A Boolean can take on two values the first value is true… Boolean values can also be false”
Type Casting: The sources teach how to change one data type to another. “You can change the type of the expression in Python this is called type casting… you can convert an INT to a float for example”
Expressions and Variables: These sections explain basic operations and variable assignment:
Expressions:“Expressions describe a type of operation the computers perform… for example basic arithmetic operations like adding multiple numbers” The order of operations is also covered.
Variables: Variables are used to “store values” and can be reassigned, and they benefit from meaningful naming.
Compound Data Types (Lists, Tuples, Dictionaries, Sets):
Tuples: Ordered, immutable sequences using parenthesis. “tuples are an ordered sequence… tupples are expressed as comma separated elements within parentheses”
Lists: Ordered, mutable sequences using square brackets. “lists are also an ordered sequence… a list is represented with square brackets” Lists support methods like extend, append, and del.
Dictionaries: Collection with key-value pairs. Keys must be immutable and unique. “a dictionary has keys and values… the keys are the first elements they must be immutable and unique each each key is followed by a value separated by a colon”
Sets: Unordered collections of unique elements. “sets are a type of collection… they are unordered… sets only have unique elements” Set operations like add, remove, intersection, union, and subset checking are covered.
Control Flow (Conditions & Loops):
Conditional Statements (if, elif, else):“The if statement allows you to make a decision based on some condition… if that condition is true the set of statements within the if block are executed”
For Loops: Used for iterating over a sequence.“The for Loop statement allows you to execute a statement or set of statements a certain number of times”
While Loops: Used for executing statements while a condition is true. “a while loop will only run if a condition is me”
Functions:
Built-in Functions: len(), sum(), sorted().
User-defined Functions: The syntax and best practices are covered, including documentation, parameters, return values, and scope of variables. “To define a function we start with the keyword def… the name of the function should be descriptive of what it does”
Object-Oriented Programming (OOP):
Classes & Objects:“A class can be thought of as a template or a blueprint for an object… An object is a realization or instantiation of that class” The concepts of attributes and methods are also introduced.
File Handling: The sources cover the use of Python’s open() function, modes for reading (‘r’) and writing (‘w’), and the importance of closing files.
“we use the open function… the first argument is the file path this is made up of the file name and the file directory the second parameter is the mode common values used include R for reading W for writing and a for appending” The use of the with statement is advocated for automatic file closing.
Libraries (Pandas & NumPy):
Pandas: Introduction to DataFrames, importing data (read_csv, read_excel), and operations like head(), selection of columns and rows (iloc, loc), and unique value discovery. “One Way pandas allows you to work with data is in a data frame” Data slicing and filtering are shown.
NumPy: Introduction to ND arrays, creation from lists, accessing elements, slicing, basic vector operations (addition, subtraction, multiplication), broadcasting and universal functions, and array attributes. “a numpy array or ND array is similar to a list… each element is of the same type”
SQL and Relational Databases: SQL is introduced as a way to interact with data in relational database systems using Data Definition Language (DDL) and Data Manipulation Language (DML). DDL statements like create table, alter table, drop table, and truncate are discussed, as well as DML statements like insert, select, update, and delete. Concepts like views and stored procedures are also covered, as well as accessing database table and column metadata.
“Data definition language or ddl statements are used to define change or drop database objects such as tables… data manipulation language or DML statements are used to read and modify data in tables”
Data Visualization, Correlation, and Statistical Methods:
Pivot Tables and Heat Maps: Techniques for reshaping data and visualizing patterns using pandas pivot() method and heatmaps. “by using the pandas pivot method we can pivot the body style variable so it is displayed along the columns and the drive wheels will be displayed along the rows”
Correlation: Introduction to the concept of correlation between variables, using scatter plots and regression lines to visualize relationships. “correlation is a statistical metric for measuring to what extent different variables are interdependent”
Pearson Correlation: A method to quantify the strength and direction of linear relationships, emphasizing both correlation coefficients and p-values. “Pearson correlation method will give you two values the correlation coefficient and the P value”
Chi-Square Test: A method to identify if there is a relationship between categorical variables. “The Ki Square test is intended to test How likely it is that an observed distribution is due to chance”
Model Development:
Linear Regression: Introduction to simple and multiple linear regression for predictive modeling with independent and dependent variables. “simple linear regression or SLR is a method to help us understand the relationship between two variables the predictor independent variable X and the target dependent variable y”
Polynomial Regression: Introduction to non linear regression models.
Model Evaluation Metrics: Introduction to evaluation metrics like R-squared (R2) and Mean Squared Error (MSE).
K-Nearest Neighbors (KNN): Classification algorithm based on similarity to other cases. K selection and distance computation are discussed. “the K near nearest neighbors algorithm is a classification algorithm that takes a bunch of labeled points and uses them to learn how to label other points”
Evaluation Metrics for Classifiers: Metrics such as the Jaccard index, F1 Score and log loss are introduced for assessing model performance.
“evaluation metrics explain the performance of a model… we can Define jackard as the size of the intersection divided by the size of the Union of two label sets”
Decision Trees: Algorithm for data classification by splitting attributes, recursive partitioning, impurity, entropy and information gain are discussed.
“decision trees are built using recursive partitioning to classify the data… the algorithm chooses the most predictive feature to split the data on”
Logistic Regression: Classification algorithm that uses a sigmoid function to calculate probabilities and gradient descent to tune model parameters.
“logistic regression is a statistical and machine learning technique for classifying records of a data set based on the values of the input Fields… in logistic regression we use one or more independent variables such as tenure age and income to predict an outcome such as churn”
Support Vector Machines: Classification algorithm based on transforming data to a high-dimensional space and finding a separating hyperplane. Kernel functions and support vectors are introduced.
“a support Vector machine is a supervised algorithm that can classify cases by finding a separator svm works by first mapping data to a high-dimensional feature space so that data points can be categorized even when the data are not otherwise linearly separable”
III. Conclusion
These sources lay a comprehensive foundation for understanding Python programming as it is used in data science. From setting up a development environment in Jupyter Notebooks to understanding fundamental data types, functions, and object-oriented programming, the document prepares learners for more advanced topics. Furthermore, the document introduces data analysis and visualization concepts, along with model building through regression techniques and classification algorithms, equipping beginners with practical data science tools. It is crucial to delve deeper into practical implementations, which are often available in the labs.
Python Programming Fundamentals and Machine Learning
Python & Jupyter Notebook
How do I start a new notebook and run code? To start a new notebook, click the plus symbol in the toolbar. Once you’ve created a notebook, type your code into a cell and click the “Run” button or use the shortcut Shift + Enter. To run multiple code cells, click “Run All Cells.”
How can I organize my notebook with titles and descriptions? To add titles and descriptions, use markdown cells. Select “Markdown” from the cell type dropdown, and you can write text, headings, lists, and more. This allows you to provide context and explain the code.
Can I use more than one notebook at a time? Yes, you can open and work with multiple notebooks simultaneously. Click the plus button on the toolbar, or go to File -> Open New Launcher or New Notebook. You can arrange the notebooks side-by-side to work with them together.
How do I present my work using notebooks? Jupyter Notebooks support creating presentations. Using markdown and code cells, you can create slides by selecting the View -> Cell Toolbar -> Slides option. You can then view the presentation using the Slides icon.
How do I shut down notebooks when I’m finished? Click the stop icon (second from top) in the sidebar, this releases memory being used by the notebook. You can terminate all sessions at once or individually. You will know it is successfully shut down when you see “No Kernel” on the top right.
Python Data Types, Expressions, and Variables
What are the main data types in Python and how can I change them? Python’s main data types include int (integers), float (real numbers), str (strings), and bool (booleans). You can change data types using type casting. For example, float(2) converts the integer 2 to a float 2.0, or int(2.9) will convert the float 2.9 to the integer 2. Casting a string like “123” to an integer is done with int(“123”) but will result in an error if the string has non-integer values. Booleans can be cast to integers where True is converted to 1, and False is converted to 0.
What are expressions and how are they evaluated? Expressions are operations that Python performs. These can include arithmetic operations like addition, subtraction, multiplication, division, and more. Python follows mathematical conventions when evaluating expressions, with parentheses having the highest precedence, followed by multiplication and division, then addition and subtraction.
How do I store values in variables and work with strings? You can store values in variables using the assignment operator =. You can then use the variable name in place of the value it stores. Variables can store results of expressions, and the type of the variable can be determined with the type() command. Strings are sequences of characters and are enclosed in single or double quotes, you can access individual elements using indexes and also perform operations like slicing, concatenation, and replication.
Python Data Structures: Lists, Tuples, Dictionaries, and Sets
What are lists and tuples, and how are they different? Lists and tuples are ordered sequences used to store data. Lists are mutable, meaning you can change, add, or remove elements. Tuples are immutable, meaning they cannot be changed once created. Lists are defined using square brackets [], and tuples are defined using parentheses ().
What are dictionaries and sets? Dictionaries are collections that store data in key-value pairs, where keys must be immutable and unique. Sets are collections of unique elements. Sets are unordered and therefore do not have indexes or ordered keys. You can perform various mathematical set operations such as union, intersection, adding and removing elements.
How do I work with nested collections and change or copy lists? You can nest lists and tuples inside other lists and tuples. Accessing elements in these structures uses the same indexing conventions. Because lists are mutable, when you assign one list variable to another variable both variables refer to the same list, therefore, changes to one list impact the other this is called aliasing. To copy a list and not reference the original, use [:] (e.g., new_list = old_list[:]) to create a new copy of the original.
Control Flow, Loops, and Functions
How do I use conditions and branching in Python? You can use if, elif, and else statements to perform different actions based on conditions. You use comparison operators (==, !=, <, >, <=, >=) which return True or False. Based on whether the condition is True, the corresponding code blocks are executed.
What is the difference between for and while loops? for loops are used for iterating over a sequence, like lists or tuples, executing a block of code for every item in that sequence. while loops repeatedly execute a block of code as long as a condition is True, you must make sure your condition will become False or it will loop forever.
What are functions and how do I create them? Functions are reusable blocks of code. They are defined with the def keyword followed by the function name, parentheses for parameters, and a colon. The function’s code block is indented. Functions can take inputs (parameters) and return values. Functions are documented in the first few lines using triple quotes.
What are variable scope and global/local variables? The scope of a variable is the part of the program where the variable is accessible. Variables defined outside of a function are global variables and are accessible everywhere. Variables defined inside a function are local variables and are only accessible within that function, there is no conflict if a local variable has the same name as a global one. If you would like to have a local variable update a global variable you can use the global keyword inside the function’s scope and assign the name of the global variable.
Object Oriented Programming, Files, and Libraries
What are classes and objects in Python? Classes are templates for creating objects. An object is a specific instance of a class. You can define classes with attributes (data) and methods (functions that operate on that data) using the class keyword, you can instantiate multiple objects of the same class.
How do I work with files in Python? You can use the open() function to create a file object, you use the first argument to specify the file path and the second for the mode (e.g., “r” for reading, “w” for writing, “a” for appending). Using the with statement is recommended, as it automatically closes the file after use. You can use methods like read(), readline(), and write() to interact with the file.
What is a library and how do I use Pandas for data analysis? Libraries are pre-written code that helps solve problems, like data analysis. You can import libraries using the import statement, often with a shortened name (as keyword). Pandas is a popular library for data analysis that uses data frames to store and analyze tabular data. You can load files like CSV or Excel into pandas data frames and use its tools for cleaning, modifying, and exploring data.
How can I work with numpy? Numpy is a library for numerical computing, it works with arrays. You can create Numpy arrays from Python lists, you can access and slice data using indexing and slicing. Numpy arrays support many mathematical operations which are usually much faster and require less memory than regular python lists.
Databases and SQL
What is SQL, a database, and a relational database? SQL (Structured Query Language) is a programming language used to manage data in a database. A database is an organized collection of data. A relational database stores data in tables with rows and columns, it uses SQL for its main operations.
What is an RDBMS and what are the basic SQL commands? RDBMS (Relational Database Management System) is a software tool used to manage relational databases. Basic SQL commands include CREATE TABLE, INSERT (to add data), SELECT (to retrieve data), UPDATE (to modify data), and DELETE (to remove data).
How do I retrieve data using the SELECT statement? You can use SELECT followed by column names to specify which columns to retrieve. SELECT * retrieves all columns from a table. You can add a WHERE clause followed by a predicate (a condition) to filter data using comparison operators (=, >, <, >=, <=, !=).
How do I use COUNT, DISTINCT, and LIMIT with select statements? COUNT() returns the number of rows that match a criteria. DISTINCT removes duplicate values from a result set. LIMIT restricts the number of rows returned.
How do I create and populate a table? You can create a table with the CREATE TABLE command. Provide the name of the table and, inside parentheses, define the name and data types for each column. Use the INSERT statement to populate tables using INSERT INTO table_name (column_1, column_2…) VALUES (value_1, value_2…).
More SQL
What are DDL and DML statements? DDL (Data Definition Language) statements are used to define database objects like tables (e.g., CREATE, ALTER, DROP, TRUNCATE). DML (Data Manipulation Language) statements are used to manage data in tables (e.g., INSERT, SELECT, UPDATE, DELETE).
How do I use ALTER, DROP, and TRUNCATE tables? ALTER TABLE is used to add, remove, or modify columns. DROP TABLE deletes a table. TRUNCATE TABLE removes all data from a table, but leaves the table structure.
How do I use views in SQL? A view is an alternative way of representing data that exists in one or more tables. Use CREATE VIEW followed by the view name, the column names and AS followed by a SELECT statement to define the data the view should display. Views are dynamic and do not store the data themselves.
What are stored procedures? A stored procedure is a set of SQL statements stored and executed on the database server. This avoids sending multiple SQL statements from the client to the server, they can accept input parameters, and return output values. You can define them with CREATE PROCEDURE.
Data Visualization and Analysis
What are pivot tables and heat maps, and how do they help with visualization? A pivot table is a way to summarize and reorganize data from a table and display it in a rectangular grid. A heat map is a graphical representation of a pivot table where data values are shown using a color intensity scale. These are effective ways to examine and visualize relationships between multiple variables.
How do I measure correlation between variables? Correlation measures the statistical interdependence of variables. You can use scatter plots to visualize the relationship between two numerical variables and add a linear regression line to show their trend. Pearson correlation measures the linear correlation between continuous numerical values, providing the correlation coefficient and P-value. Chi-square test is used to identify if an association between two categorical variables exists.
What is simple linear regression and multiple linear regression? Simple linear regression uses one independent variable to predict a dependent variable using a linear relationship, Multiple linear regression uses several independent variables to predict the dependent variable.
Model Development
What is a model and how can I use it for predictions? A model is a mathematical equation used to predict a value (dependent variable) given one or more other values (independent variables). Models are trained with data that determines parameters for an equation. Once the model is trained you can input data and have the model predict an output.
What are R-squared and MSSE, and how are they used to evaluate model performance? R-squared measures how well the model fits the data and it represents the percentage of the data that is closest to the fitted line and represents the “goodness of fit”. Mean squared error (MSE) is the average of the square difference between the predicted values and the true values. These scores are used to measure model performance for continuous target values and are called in-sample evaluation metrics, as they use training data.
What is polynomial regression? Polynomial regression is a form of regression analysis in which the relationship between the independent variable and the dependent variable is modeled as an nth degree polynomial. This allows more flexibility in the curve fitting.
What are pipelines in machine learning? Pipelines are a way to streamline machine learning workflows. They combine multiple steps (e.g., scaling, model training) into a single entity, making the process of building and evaluating models more efficient.
Machine Learning Classification Algorithms
What is the K-Nearest Neighbors algorithm and how does it work? The K-Nearest Neighbors algorithm (KNN) is a classification algorithm that uses labeled data points to learn how to label other points. It classifies new cases by looking at the ‘k’ nearest neighbors in the training data based on some sort of dissimilarity metric, the most popular label among neighbors is the predicted class for that data point. The choice of ‘k’ and the distance metric are important, and the dissimilarity measure depends on data type.
What are common evaluation metrics for classifiers? Common evaluation metrics for classifiers include Jaccard Index, F1 Score, and Log Loss. Jaccard Index measures similarity. F1 Score combines precision and recall. Log Loss is used to measure the performance of a probabilistic classifier like logistic regression.
What is a confusion matrix? A confusion matrix is used to evaluate the performance of a classification model. It shows the counts of true positives, true negatives, false positives, and false negatives. This helps evaluate where your model is making mistakes.
What are decision trees and how are they built? Decision trees use a tree-like structure with nodes representing decisions based on features and branches representing outcomes, they are constructed by partitioning the data by minimizing the impurity at each step based on the attribute with the highest information gain, which is the entropy of the tree before the split minus the weighted entropy of the tree after the split.
What is logistic regression and how does it work? Logistic regression is a machine learning algorithm used for classification. It models the probability of a sample belonging to a specific class using a sigmoid function, it returns a probability of the outcome being one and (1-p) of the outcome being zero, parameter values are trained to find parameters which produce accurate estimations.
What is the Support Vector Machine algorithm? A support vector machine (SVM) is a classification algorithm used for classification that works by transforming data into a high-dimensional space so that data can be categorized by drawing a separating hyperplane, the algorithm optimizes its output by maximizing the margin between classes and using data points closest to the hyperplane for learning, called support vectors.
A Data Science Career Guide
A career in data science is enticing due to the field’s recent growth, the abundance of electronic data, advancements in artificial intelligence, and its demonstrated business value [1]. The US Bureau of Labor Statistics projects a 35% growth rate in the field, with a median annual salary of around $103,000 [1].
What Data Scientists Do:
Data scientists use data to understand the world [1].
They investigate and explain problems [2].
They uncover insights and trends hiding behind data and translate data into stories to generate insights [1, 3].
They analyze structured and unstructured data from varied sources [4].
They clarify questions that organizations want answered and then determine what data is needed to solve the problem [4].
They use data analysis to add to the organization’s knowledge, revealing previously hidden opportunities [4].
They communicate results to stakeholders, often using data visualization [4].
They build machine learning and deep learning models using algorithms to solve business problems [5].
Essential Skills for Data Scientists:
Curiosity is essential to explore data and come up with meaningful questions [3, 4].
Argumentation helps explain findings and persuade others to adjust their ideas based on the new information [3].
Judgment guides a data scientist to start in the right direction [3].
Comfort and flexibility with analytics platforms and software [3].
Storytelling is key to communicating findings and insights [3, 4].
Technical Skills:Knowledge of programming languages like Python, R, and SQL [6, 7]. Python is widely used in data science [6, 7].
Familiarity with databases, particularly relational databases [8].
Understanding of statistical inference and distributions [8].
Ability to work with Big Data tools like Hadoop and Spark [2, 9].
Experience with data visualization tools and techniques [4, 9].
Soft Skills:Communication and presentation skills [5, 9].
Critical thinking and problem-solving abilities [5, 9].
Creative thinking skills [5].
Collaborative approach [5].
Educational Background and Training
A background in mathematics and statistics is beneficial [2].
Training in probability and statistics is necessary [2].
Knowledge of algebra and calculus is useful [2].
Comfort with computer science is helpful [3].
A degree in a quantitative field such as mathematics or statistics is a good starting point [4]
Career Paths and Opportunities:
Data science is relevant due to the abundance of available data, algorithms, and inexpensive tools [1].
Data scientists can work across many industries, including technology, healthcare, finance, transportation, and retail [1, 2].
There is a growing demand for data scientists in various fields [1, 9, 10].
Job opportunities can be found in large companies, small companies, and startups [10].
The field offers a range of roles, from entry-level to senior positions and leadership roles [10].
Career advancement can lead to specialization in areas like machine learning, management, or consulting [5].
Some possible job titles include data analyst, data engineer, research scientist, and machine learning engineer [5, 6].
How to Prepare for a Data Science Career:
Learn programming, especially Python [7, 11].
Study math, probability, and statistics [11].
Practice with databases and SQL [11].
Build a portfolio with projects to showcase skills [12].
Network both online and offline [13].
Research companies and industries you are interested in [14].
Develop strong communication and storytelling skills [3, 9].
Consider certifications to show proficiency [3, 9].
Challenges in the Field
Companies need to understand what they want from a data science team and hire accordingly [9].
It’s rare to find a “unicorn” candidate with all desired skills, so teams are built with diverse skills [8, 11].
Data scientists must stay updated with the latest technology and methods [9, 15].
Data professionals face technical, organizational, and cultural challenges when using generative AI models [15].
AI models need constant updating and adapting to changing data [15].
Data science is a process of using data to understand different things and the world, and involves validating hypotheses with data [1]. It is also the art of uncovering insights and using them to make strategic choices for companies [1]. With a blend of technical skills, curiosity, and the ability to communicate effectively, a career in data science offers diverse and rewarding opportunities [2, 11].
Data Science Skills and Generative AI
Data science requires a combination of technical and soft skills to be successful [1, 2].
Technical Skills
Programming languages such as Python, R, and SQL are essential [3, 4]. Python is widely used in the data science industry [4].
Database knowledge, particularly with relational databases [5].
Understanding of statistical concepts, probability, and statistical inference [2, 6-9].
Experience with machine learning algorithms [2, 3, 6].
Familiarity with Big Data tools like Hadoop and Spark, especially for managing and manipulating large datasets [2, 3, 7].
Ability to perform data mining, and data wrangling, including cleaning, transforming, and preparing data for analysis [3, 6, 9, 10].
Data visualization skills are important for effectively presenting findings [2, 3, 6, 11]. This includes using tools like Tableau, PowerBI, and R’s visualization packages [7, 10-12].
Knowledge of cloud computing, and cloud-based data management [3, 12].
Experience using libraries such as pandas, NumPy, SciPy and Matplotlib in Python, is useful for data analysis and machine learning [4].
Familiarity with tools like Jupyter Notebooks, RStudio, and GitHub are important for coding, collaboration and project sharing [3].
Soft Skills
Curiosity is essential for exploring data and asking meaningful questions [1, 2].
Critical thinking and problem-solving skills are needed to analyze and solve problems [2, 7, 9].
Communication and presentation skills are vital for explaining technical concepts and insights to both technical and non-technical audiences [1-3, 7, 9].
Storytelling skills are needed to translate data into compelling narratives [1, 2, 7].
Argumentation is essential for explaining findings [1, 2].
Collaboration skills are important, as data scientists often work with other professionals [7, 9].
Creative thinking skills allow data scientists to develop innovative approaches [9].
Good judgment to guide the direction of projects [1, 2].
Grit and tenacity to persevere through complex projects and challenges [12, 13].
Additional skills:
Business analysis is important to understand and analyze problems from a business perspective [13].
A methodical approach is needed for data gathering and analysis [1].
Comfort and flexibility with analytics platforms is also useful [1].
How Generative AI Can Help
Generative AI can assist data scientists in honing these skills [9]:
It can ease the learning process for statistics and math [9].
It can guide coding and help prepare code [9].
It can help data professionals with data preparation tasks such as cleaning, handling missing values, standardizing, normalizing, and structuring data for analysis [9, 14].
It can assist with the statistical analysis of data [9].
It can aid in understanding the applicability of different machine learning models [9].
Note: It is important to note that while these technical skills are important, it is not always necessary to be an expert in every area [13, 15]. A combination of technical knowledge and soft skills with a focus on continuous learning is ideal [9]. It is also valuable to gain experience by creating a portfolio with projects demonstrating these skills [12, 13].
A Comprehensive Guide to Data Science Tools
Data science utilizes a variety of tools to perform tasks such as data management, integration, visualization, model building, and deployment [1]. These tools can be categorized into several types, including data management tools, data integration and transformation tools, data visualization tools, model building and deployment tools, code and data asset management tools, development environments, and cloud-based tools [1-3].
Data Management Tools
Relational databases such as MySQL, PostgreSQL, Oracle Database, Microsoft SQL Server, and IBM Db2 [2, 4, 5]. These systems store data in a structured format with rows and columns, and use SQL to manage and retrieve the data [4].
NoSQL databases like MongoDB, Apache CouchDB, and Apache Cassandra are used to store semi-structured and unstructured data [2, 4].
File-based tools such as Hadoop File System (HDFS) and cloud file systems like Ceph [2].
Elasticsearch is used for storing and searching text data [2].
Data warehouses, data marts and data lakes are also important for data storage and retrieval [4].
Data Integration and Transformation Tools
ETL (Extract, Transform, Load) tools are used to extract data from various sources, transform it into a usable format, and load it into a data warehouse [1, 4].
Apache Airflow, Kubeflow, Apache Kafka, Apache NiFi, Apache Spark SQL, and Node-RED are open-source tools used for data integration and transformation [2].
Informatica PowerCenter and IBM InfoSphere DataStage are commercial tools used for ETL processes [5].
Data Refinery is a tool within IBM Watson Studio that enables data transformation using a spreadsheet-like interface [3, 5].
Data Visualization Tools
Tools that present data in graphical formats, such as charts, plots, maps, and animations [1].
Programming libraries like Pixie Dust for Python, which also has a user interface that helps with plotting [2].
Hue which can create visualizations from SQL queries [2].
Kibana, a data exploration and visualization web application [2].
Apache Superset is another web application used for data exploration and visualization [2].
Tableau, Microsoft Power BI, and IBM Cognos Analytics are commercial business intelligence (BI) tools used for creating visual reports and dashboards [3, 5].
Plotly Dash for building interactive dashboards [6].
R’s visualization packages such as ggplot, plotly, lattice, and leaflet [7].
Data Mirror is a cloud-based data visualization tool [3].
Model Building and Deployment Tools
Machine learning and deep learning libraries in Python such as TensorFlow, PyTorch, and scikit-learn [8, 9].
Apache PredictionIO and Seldon are open-source tools for model deployment [2].
MLeap is another tool to deploy Spark ML models [2].
TensorFlow Serving is used to deploy TensorFlow models [2].
SPSS Modeler and SAS Enterprise Miner are commercial data mining products [5].
IBM Watson Machine Learning and Google AI Platform Training are cloud-based services for training and deploying models [1, 3].
Code and Data Asset Management Tools
Git is the standard tool for code asset management, or version control, with platforms like GitHub, GitLab, and Bitbucket being popular for hosting repositories [2, 7, 10].
Apache Atlas, ODP Aeria, and Kylo are tools used for data asset management [2, 10].
Informatica Enterprise Data Governance and IBM provide tools for data asset management [5].
Development Environments
Jupyter Notebook is a web-based environment that supports multiple programming languages, and is popular among data scientists for combining code, visualizations, and narrative text [4, 10, 11]. Jupyter Lab is a more modern version of Jupyter Notebook [10].
RStudio is an integrated development environment (IDE) specifically for the R language [4, 7, 10].
Spyder is an IDE that attempts to mimic the functionality of RStudio, but for the Python world [10].
Apache Zeppelin provides an interface similar to Jupyter Notebooks but with integrated plotting capabilities [10].
IBM Watson Studio provides a collaborative environment for data science tasks, including tools for data pre-processing, model training, and deployment, and is available in cloud and desktop versions [1, 2, 5].
Visual tools like KNIME and Orange are also used [10].
Cloud-Based Tools
Cloud platforms such as IBM Watson Studio, Microsoft Azure Machine Learning, and H2O Driverless AI offer fully integrated environments for the entire data science life cycle [3].
Amazon Web Services (AWS), Google Cloud, and Microsoft Azure provide various services for data storage, processing, and machine learning [3, 12].
Cloud-based versions of existing open-source and commercial tools are widely available [3].
Programming Languages
Python is the most widely used language in data science due to its clear syntax, extensive libraries, and supportive community [8]. Libraries include pandas, NumPy, SciPy, Matplotlib, TensorFlow, PyTorch, and scikit-learn [8, 9].
R is specifically designed for statistical computing and data analysis [4, 7]. Packages such as dplyr, stringr, ggplot, and caret are widely used [7].
SQL is essential for managing and querying databases [4, 11].
Scala and Java are general purpose languages used in data science [9].
C++ is used to build high-performance libraries such as TensorFlow [9].
JavaScript can be used for data science with libraries such as tensorflow.js [9].
Julia is used for high performance numerical analysis [9].
Generative AI Tools
Generative AI tools are also being used for various tasks, including data augmentation, report generation, and model development [13].
SQL through AI converts natural language queries into SQL commands [12].
Tools such as DataRobot, AutoGluon, H2O Driverless AI, Amazon SageMaker Autopilot, and Google Vertex AI are used for automated machine learning (AutoML) [14].
Free tools such as AIO are also available for data analysis and visualization [14].
These tools support various aspects of data science, from data collection and preparation to model building and deployment. Data scientists often use a combination of these tools to complete their work.
Machine Learning Fundamentals
Machine learning is a subset of AI that uses computer algorithms to analyze data and make intelligent decisions based on what it has learned, without being explicitly programmed [1, 2]. Machine learning algorithms are trained with large sets of data, and they learn from examples rather than following rules-based algorithms [1]. This enables machines to solve problems on their own and make accurate predictions using the provided data [1].
Here are some key concepts related to machine learning:
Types of machine learning:Supervised learning is a type of machine learning where a human provides input data and correct outputs, and the model tries to identify relationships and dependencies between the input data and the correct output [3]. Supervised learning comprises two types of models:
Regression models are used to predict a numeric or real value [3].
Classification models are used to predict whether some information or data belongs to a category or class [3].
Unsupervised learning is a type of machine learning where the data is not labeled by a human, and the models must analyze the data and try to identify patterns and structure within the data based on its characteristics [3, 4]. Clustering models are an example of unsupervised learning [3].
Reinforcement learning is a type of learning where a model learns the best set of actions to take given its current environment to get the most rewards over time [3].
Deep learning is a specialized subset of machine learning that uses layered neural networks to simulate human decision-making [1, 2]. Deep learning algorithms can label and categorize information and identify patterns [1].
Neural networks (also called artificial neural networks) are collections of small computing units called neurons that take incoming data and learn to make decisions over time [1, 2].
Generative AI is a subset of AI that focuses on producing new data rather than just analyzing existing data [1, 5]. It allows machines to create content, including images, music, language, and computer code, mimicking creations by people [1, 5]. Generative AI can also create synthetic data that has similar properties as the real data, which is useful for training and testing models when there isn’t enough real data [1, 5].
Model training is the process by which a model learns patterns from data [3, 6].
Applications of Machine Learning
Machine learning is used in many fields and industries [7, 8]:
Predictive analytics is a common application of machine learning [2].
Recommendation systems, such as those used by Netflix or Amazon, are also a major application [2, 8].
Fraud detection is another key area [2]. Machine learning is used to determine whether a credit card charge is fraudulent in real time [2].
Machine learning is also used in the self-driving car industry to classify objects a car might encounter [7].
Cloud computing service providers like IBM and Amazon use machine learning to protect their services and prevent attacks [7].
Machine learning can be used to find trends and patterns in stock data [7].
Machine learning is used to help identify cancer using X-ray scans [7].
Machine learning is used in healthcare to predict whether a human cell is benign or malignant [8].
Machine learning can help determine proper medicine for patients [8].
Banks use machine learning to make decisions on loan applications and for customer segmentation [8].
Websites such as Youtube, Amazon, or Netflix use machine learning to develop recommendations for their customers [8].
How Data Scientists Use Machine Learning
Data scientists use machine learning algorithms to derive insights from data [2]. They use machine learning for predictive analytics, recommendations, and fraud detection [2]. Data scientists also use machine learning for the following tasks:
Data preparation: Machine learning models benefit from the standardization of data, and data scientists use machine learning to address outliers or different scales in data sets [4].
Model building: Machine learning is used to build models that can analyze data and make intelligent decisions [1, 3].
Model evaluation: Data scientists need to evaluate the performance of the trained models [9].
Model deployment: Data scientists deploy models to make them available to applications [10, 11].
Data augmentation: Generative AI, a subset of machine learning, is used to augment data sets when there is not enough real data [1, 5, 12].
Code generation: Generative AI can help data scientists generate software code for building analytic models [1, 5, 12].
Data exploration: Generative AI tools can explore data, uncover patterns and insights and assist with data visualization [1, 5].
Machine Learning Techniques
Several techniques are commonly used in machine learning [4, 13]:
Regression is a technique for predicting a continuous value, such as the price of a house [13].
Classification is a technique for predicting the class or category of a case [13].
Clustering is a technique that groups similar cases [4, 13].
Association is a technique for finding items that co-occur [13].
Anomaly detection is used to find unusual cases [13].
Sequence mining is used for predicting the next event [13].
Dimension reduction is used to reduce the size of data [13].
Recommendation systems associate people’s preferences with others who have similar tastes [13].
Support Vector Machines (SVM) are used for classification by finding a separator [14]. SVMs map data to a higher dimensional feature space so data points can be categorized [14].
Linear and Polynomial Models are used for regression [4, 15].
Tools and Libraries
Machine learning models are implemented using popular frameworks such as TensorFlow, PyTorch, and Keras [6]. These learning frameworks provide a Python API and support other languages such as C++ and Javascript [6]. Scikit-learn is a free machine learning library for the Python programming language that contains many classification, regression, and clustering algorithms [4].
The field of machine learning is constantly evolving, and data scientists are always learning about new techniques, algorithms and tools [16].
Generative AI: Applications and Challenges
Generative AI is a subset of artificial intelligence that focuses on producing new data rather than just analyzing existing data [1, 2]. It allows machines to create content, including images, music, language, computer code, and more, mimicking creations by people [1, 2].
How Generative AI Operates
Generative AI uses deep learning models like Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) [1, 2]. These models learn patterns from large volumes of data and create new instances that replicate the underlying distributions of the original data [1, 2].
Applications of Generative AI Generative AI has a wide array of applications [1, 2]:
Natural Language Processing (NLP), such as OpenAI’s GPT-3, can generate human-like text, which is useful for content creation and chatbots [1, 2].
In healthcare, generative AI can synthesize medical images, aiding in the training of medical professionals [1, 2].
Generative AI can create unique and visually stunning artworks and generate endless creative visual compositions [1, 2].
Game developers use generative AI to generate realistic environments, characters, and game levels [1, 2].
In fashion, generative AI can design new styles and create personalized shopping recommendations [1, 2].
Generative AI can also be used for data augmentation by creating synthetic data with similar properties to real data [1, 2]. This is useful when there isn’t enough real data to train or test a model [1, 2].
Generative AI can be used to generate and test software code for constructing analytic models, which has the potential to revolutionize the field of analytics [2].
Generative AI can generate business insights and reports, and autonomously explore data to uncover hidden patterns and enhance decision-making [2].
Types of Generative AI Models
There are four common types of generative AI models [3]:
Generative Adversarial Networks (GANs) are known for their ability to create realistic and diverse data. They are versatile in generating complex data across multiple modalities like images, videos, and music. GANs are good at generating new images, editing existing ones, enhancing image quality, generating music, producing creative text, and augmenting data [3]. A notable example of a GAN architecture is StyleGAN, which is specifically designed for high-fidelity images of faces with diverse styles and attributes [3].
Variational Autoencoders (VAEs) discover the underlying patterns that govern data organization. They are good at uncovering the structure of data and can generate new samples that adhere to inherent patterns. VAEs are efficient, scalable, and good at anomaly detection. They can also compress data, perform collaborative filtering, and transform the style of one image into another [3]. An example of a VAE is VAEGAN, a hybrid model combining VAEs and GANs [3].
Autoregressive models are useful for handling sequential data like text and time series. They generate data one element at a time and are good at generating coherent text, converting text into natural-sounding speech, forecasting time series, and translating languages [3]. A prominent example of an autoregressive model is Generative Pre-trained Transformer (GPT), which can generate human-quality text, translate languages, and produce creative content [3].
Flow-based models are used to model the probability distribution of data, which allows for efficient sampling and generation. They are good at generating high-quality images and simulating synthetic data. Data scientists use flow-based models for anomaly detection and for estimating probability density function [3]. An example of a flow-based model is RealNVP, which generates high-quality images of human faces [3].
Generative AI in the Data Science Life Cycle
Generative AI is a transformative force in the data science life cycle, providing data scientists with tools to analyze data, uncover insights, and develop solutions [4]. The data science lifecycle consists of five phases [4]:
Problem definition and business understanding: Generative AI can help generate new ideas and solutions, simulate customer profiles to understand needs, and simulate market trends to assess opportunities and risks [4].
Data acquisition and preparation: Generative AI can fill in missing values in data sets, augment data by generating synthetic data, and detect anomalies [4].
Model development and training: Generative AI can perform feature engineering, explore hyperparameter combinations, and generate explanations of complex model predictions [4].
Model evaluation and refinement: Generative AI can generate adversarial or edge cases to test model robustness and can train a generative model to mimic model uncertainty [4].
Model deployment and monitoring: Generative AI can continuously monitor data, provide personalized experiences, and perform A/B testing to optimize performance [4].
Generative AI for Data Preparation and Querying Generative AI models are used for data preparation and querying tasks by:
Imputing missing values: VAEs can learn intricate patterns within the data and generate plausible values [5].
Detecting outliers: GANs can learn the boundaries of standard data distributions and identify outliers [5].
Reducing noise: Autoencoders can capture core information in data while discarding noise [5].
Data Translation: Neural machine translation (NMT) models can accurately translate text from one language to another, and can also perform text-to-speech and image-to-text translations [5].
Natural Language Querying: Large language models (LLMs) can interpret natural language queries and translate them into SQL statements [5].
Query Recommendations: Recurrent neural networks (RNNs) can capture the temporal relationship between queries, enabling them to predict the next query based on a user’s current query [5].
Query Optimization: Graph neural networks (GNNs) can represent data as a graph to understand connections between entities and identify the most efficient query execution plans [5].
Generative AI in Exploratory Data Analysis
Generative AI can also assist with exploratory data analysis (EDA) by [6]:
Generating descriptive statistics for numerical and categorical data.
Generating synthetic data to understand the distribution of a particular variable.
Modeling the joint distribution of two variables to reveal their potential correlation.
Reducing the dimensionality of data while preserving relationships between variables.
Enhancing feature engineering by generating new features that capture the structure of the data.
Identifying potential patterns and relationships in the data.
Generative AI for Model Development Generative AI can be used for model development by [6]:
Helping select the most appropriate model architecture.
Assessing the importance of different features.
Creating ensemble models by generating diverse representations of data.
Interpreting the predictions made by a model by generating representatives of the data.
Improving a model’s generalization ability and preventing overfitting.
Tools for Model Development
Several generative AI tools are used for model development [7]:
DataRobot is an AI platform that automates the building, deployment, and management of machine learning models [7].
AutoGluon is an open-source automated machine learning library that simplifies the development and deployment of machine learning models [7].
H2O Driverless AI is a cloud-based automated machine learning platform that supports automatic model building, deployment, and monitoring [7].
Amazon SageMaker Autopilot is a managed service that automates the process of building, training, and deploying machine learning models [7].
Google Vertex AI is a fully managed cloud-based machine learning platform [7].
ChatGPT and Google Bard can be used for AI-powered script generation to streamline the model building process [7].
Considerations and Challenges When using generative AI, there are several factors to consider, including data quality, model selection, and ethical implications [6, 8]:
The quality of training data is critical; bias in training data can lead to biased results [8].
The choice of model and training parameters determines how explainable the model output is [8].
There are ethical implications to consider, such as ensuring the models are used responsibly and do not contribute to malicious activities [8].
The lack of high quality labeled data, the difficulty of interpreting models, the computational expense of training large models, and the lack of standardization are technical challenges in using generative AI [9].
There are also organizational challenges, including copyright and intellectual property issues, the need for specialized skills, integrating models into existing systems, and measuring return on investment [9].
Cultural challenges include risk aversion, data sharing concerns, and issues related to trust and transparency [9].
In summary, generative AI is a powerful tool with a wide range of applications across various industries. It is used for data augmentation, data preparation, data querying, model development, and exploratory data analysis. However, it is important to be aware of the challenges and ethical considerations when using generative AI to ensure its responsible deployment.
Data Science Full Course – Complete Data Science Course | Data Science Full Course For Beginners IBM
Affiliate Disclosure: This blog may contain affiliate links, which means I may earn a small commission if you click on the link and make a purchase. This comes at no additional cost to you. I only recommend products or services that I believe will add value to my readers. Your support helps keep this blog running and allows me to continue providing you with quality content. Thank you for your support!
These resources provide a comprehensive pathway for aspiring database engineers and software developers. They cover fundamental database concepts like data modeling, SQL for data manipulation and management, database optimization, and data warehousing. Furthermore, they explore essential software development practices including Python programming, object-oriented principles, version control with Git and GitHub, software testing methodologies, and preparing for technical interviews with insights into data structures and algorithms.
Introduction to Database Engineering
This course provides a comprehensive introduction to database engineering. A straightforward description of a database is a form of electronic storage in which data is held. However, this simple explanation doesn’t fully capture the impact of database technology on global industry, government, and organizations. Almost everyone has used a database, and it’s likely that information about us is present in many databases worldwide.
Database engineering is crucial to global industry, government, and organizations. In a real-world context, databases are used in various scenarios:
Banks use databases to store data for customers, bank accounts, and transactions.
Hospitals store patient data, staff data, and laboratory data.
Online stores retain profile information, shopping history, and accounting transactions.
Social media platforms store uploaded photos.
Work environments use databases for downloading files.
Online games rely on databases.
Data in basic terms is facts and figures about anything. For example, data about a person might include their name, age, email, and date of birth, or it could be facts and figures related to an online purchase like the order number and description.
A database looks like data organized systematically, often resembling a spreadsheet or a table. This systematic organization means that all data contains elements or features and attributes by which they can be identified. For example, a person can be identified by attributes like name and age.
Data stored in a database cannot exist in isolation; it must have a relationship with other data to be processed into meaningful information. Databases establish relationships between pieces of data, for example, by retrieving a customer’s details from one table and their order recorded against another table. This is often achieved through keys. A primary key uniquely identifies each record in a table, while a foreign key is a primary key from one table that is used in another table to establish a link or relationship between the two. For instance, the customer ID in a customer table can be the primary key and then become a foreign key in an order table, thus relating the two tables.
While relational databases, which organize data into tables with relationships, are common, there are other types of databases. An object-oriented database stores data in the form of objects instead of tables or relations. An example could be an online bookstore where authors, customers, books, and publishers are rendered as classes, and the individual entries are objects or instances of these classes.
To work with data in databases, database engineers use Structured Query Language (SQL). SQL is a standard language that can be used with all relational databases like MySQL, PostgreSQL, Oracle, and Microsoft SQL Server. Database engineers establish interactions with databases to create, read, update, and delete (CRUD) data.
SQL can be divided into several sub-languages:
Data Definition Language (DDL) helps define data in the database and includes commands like CREATE (to create databases and tables), ALTER (to modify database objects), and DROP (to remove objects).
Data Manipulation Language (DML) is used to manipulate data and includes operations like INSERT (to add data), UPDATE (to modify data), and DELETE (to remove data).
Data Query Language (DQL) is used to read or retrieve data, primarily using the SELECT command.
Data Control Language (DCL) is used to control access to the database, with commands like GRANT and REVOKE to manage user privileges.
SQL offers several advantages:
It requires very little coding skills to use, consisting mainly of keywords.
Its interactivity allows developers to write complex queries quickly.
It is a standard language usable with all relational databases, leading to extensive support and information availability.
It is portable across operating systems.
Before developing a database, planning the organization of data is crucial, and this plan is called a schema. A schema is an organization or grouping of information and the relationships among them. In MySQL, schema and database are often interchangeable terms, referring to how data is organized. However, the definition of schema can vary across different database systems. A database schema typically comprises tables, columns, relationships, data types, and keys. Schemas provide logical groupings for database objects, simplify access and manipulation, and enhance database security by allowing permission management based on user access rights.
Database normalization is an important process used to structure tables in a way that minimizes challenges by reducing data duplication and avoiding data inconsistencies (anomalies). This involves converting a large table into multiple tables to reduce data redundancy. There are different normal forms (1NF, 2NF, 3NF) that define rules for table structure to achieve better database design.
As databases have evolved, they now must be able to store ever-increasing amounts of unstructured data, which poses difficulties. This growth has also led to concepts like big data and cloud databases.
Furthermore, databases play a crucial role in data warehousing, which involves a centralized data repository that loads, integrates, stores, and processes large amounts of data from multiple sources for data analysis. Dimensional data modeling, based on dimensions and facts, is often used to build databases in a data warehouse for data analytics. Databases also support data analytics, where collected data is converted into useful information to inform future decisions.
Tools like MySQL Workbench provide a unified visual environment for database modeling and management, supporting the creation of data models, forward and reverse engineering of databases, and SQL development.
Finally, interacting with databases can also be done through programming languages like Python using connectors or APIs (Application Programming Interfaces). This allows developers to build applications that interact with databases for various operations.
Understanding SQL: Language for Database Interaction
SQL (Structured Query Language) is a standard language used to interact with databases. It’s also commonly pronounced as “SQL”. Database engineers use SQL to establish interactions with databases.
Here’s a breakdown of SQL based on the provided source:
Role of SQL: SQL acts as the interface or bridge between a relational database and its users. It allows database engineers to create, read, update, and delete (CRUD) data. These operations are fundamental when working with a database.
Interaction with Databases: As a web developer or data engineer, you execute SQL instructions on a database using a Database Management System (DBMS). The DBMS is responsible for transforming SQL instructions into a form that the underlying database understands.
Applicability: SQL is particularly useful when working with relational databases, which require a language that can interact with structured data. Examples of relational databases that SQL can interact with include MySQL, PostgreSQL, Oracle, and Microsoft SQL Server.
SQL Sub-languages: SQL is divided into several sub-languages:
Data Definition Language (DDL): Helps you define data in your database. DDL commands include:
CREATE: Used to create databases and related objects like tables. For example, you can use the CREATE DATABASE command followed by the database name to create a new database. Similarly, CREATE TABLE followed by the table name and column definitions is used to create tables.
ALTER: Used to modify already created database objects, such as modifying the structure of a table by adding or removing columns (ALTER TABLE).
DROP: Used to remove objects like tables or entire databases. The DROP DATABASE command followed by the database name removes a database. The DROP COLUMN command removes a specific column from a table.
Data Manipulation Language (DML): Commands are used to manipulate data in the database and most CRUD operations fall under DML. DML commands include:
INSERT: Used to add or insert data into a table. The INSERT INTO syntax is used to add rows of data to a specified table.
UPDATE: Used to edit or modify existing data in a table. The UPDATE command allows you to specify data to be changed.
DELETE: Used to remove data from a table. The DELETE FROM syntax followed by the table name and an optional WHERE clause is used to remove data.
Data Query Language (DQL): Used to read or retrieve data from the database. The primary DQL command is:
SELECT: Used to select and retrieve data from one or multiple tables, allowing you to specify the columns you want and apply filter criteria using the WHERE clause. You can select all columns using SELECT *.
Data Control Language (DCL): Used to control access to the database. DCL commands include:
GRANT: Used to give users access privileges to data.
REVOKE: Used to revert access privileges already given to users.
Advantages of SQL: SQL is a popular language choice for databases due to several advantages:
Low coding skills required: It uses a set of keywords and requires very little coding.
Interactivity: Allows developers to write complex queries quickly.
Standard language: Can be used with all relational databases like MySQL, leading to extensive support and information availability.
Portability: Once written, SQL code can be used on any hardware and any operating system or platform where the database software is installed.
Comprehensive: Covers all areas of database management and administration, including creating databases, manipulating data, retrieving data, and managing security.
Efficiency: Allows database users to process large amounts of data quickly and efficiently.
Basic SQL Operations: SQL enables various operations on data, including:
Creating databases and tables using DDL.
Populating and modifying data using DML (INSERT, UPDATE, DELETE).
Reading and querying data using DQL (SELECT) with options to specify columns and filter data using the WHERE clause.
Sorting data using the ORDER BY clause with ASC (ascending) or DESC (descending) keywords.
Filtering data using the WHERE clause with various comparison operators (=, <, >, <=, >=, !=) and logical operators (AND, OR). Other filtering operators include BETWEEN, LIKE, and IN.
Removing duplicate rows using the SELECT DISTINCT clause.
Performing arithmetic operations using operators like +, -, *, /, and % (modulus) within SELECT statements.
Using comparison operators to compare values in WHERE clauses.
Utilizing aggregate functions (though not detailed in this initial overview but mentioned later in conjunction with GROUP BY).
Joining data from multiple tables (mentioned as necessary when data exists in separate entities). The source later details INNER JOIN, LEFT JOIN, and RIGHT JOIN clauses.
Creating aliases for tables and columns to make queries simpler and more readable.
Using subqueries (a query within another query) for more complex data retrieval.
Creating views (virtual tables based on the result of a SQL statement) to simplify data access and combine data from multiple tables.
Using stored procedures (pre-prepared SQL code that can be saved and executed).
Working with functions (numeric, string, date, comparison, control flow) to process and manipulate data.
Implementing triggers (stored programs that automatically execute in response to certain events).
Managing database transactions to ensure data integrity.
Optimizing queries for better performance.
Performing data analysis using SQL queries.
Interacting with databases using programming languages like Python through connectors and APIs.
In essence, SQL is a powerful and versatile language that is fundamental for anyone working with relational databases, enabling them to define, manage, query, and manipulate data effectively. The knowledge of SQL is a valuable skill for database engineers and is crucial for various tasks, from building and maintaining databases to extracting insights through data analysis.
Data Modeling Principles: Schema, Types, and Design
Data modeling principles revolve around creating a blueprint of how data will be organized and structured within a database system. This plan, often referred to as a schema, is essential for efficient data storage, access, updates, and querying. A well-designed data model ensures data consistency and quality.
Here are some key data modeling principles discussed in the sources:
Understanding Data Requirements: Before creating a database, it’s crucial to have a clear idea of its purpose and the data it needs to store. For example, a database for an online bookshop needs to record book titles, authors, customers, and sales. Mangata and Gallo (mng), a jewelry store, needed to store data on customers, products, and orders.
Visual Representation: A data model provides a visual representation of data elements (entities) and their relationships. This is often achieved using an Entity Relationship Diagram (ERD), which helps in planning entity-relational databases.
Different Levels of Abstraction: Data modeling occurs at different levels:
Conceptual Data Model: Provides a high-level, abstract view of the entities and their relationships in the database system. It focuses on “what” data needs to be stored (e.g., customers, products, orders as entities for mng) and how these relate.
Logical Data Model: Builds upon the conceptual model by providing a more detailed overview of the entities, their attributes, primary keys, and foreign keys. For mng, this would involve defining attributes for customers (like client ID as primary key), products, and orders, and specifying foreign keys to establish relationships (e.g., client ID in the orders table referencing the clients table).
Physical Data Model: Represents the internal schema of the database and is specific to the chosen Database Management System (DBMS). It outlines details like data types for each attribute (e.g., varchar for full name, integer for contact number), constraints (e.g., not null), and other database-specific features. SQL is often used to create the physical schema.
Choosing the Right Data Model Type: Several types of data models exist, each with its own advantages and disadvantages:
Relational Data Model: Represents data as a collection of tables (relations) with rows and columns, known for its simplicity.
Entity-Relationship Model: Similar to the relational model but presents each table as a separate entity with attributes and explicitly defines different types of relationships between entities (one-to-one, one-to-many, many-to-many).
Hierarchical Data Model: Organizes data in a tree-like structure with parent and child nodes, primarily supporting one-to-many relationships.
Object-Oriented Model: Translates objects into classes with characteristics and behaviors, supporting complex associations like aggregation and inheritance, suitable for complex projects.
Dimensional Data Model: Based on dimensions (context of measurements) and facts (quantifiable data), optimized for faster data retrieval and efficient data analytics, often using star and snowflake schemas in data warehouses.
Database Normalization: This is a crucial process for structuring tables to minimize data redundancy, avoid data modification implications (insertion, update, deletion anomalies), and simplify data queries. Normalization involves applying a series of normal forms (First Normal Form – 1NF, Second Normal Form – 2NF, Third Normal Form – 3NF) to ensure data atomicity, eliminate repeating groups, address functional and partial dependencies, and resolve transitive dependencies.
Establishing Relationships: Data in a database should be related to provide meaningful information. Relationships between tables are established using keys:
Primary Key: A value that uniquely identifies each record in a table and prevents duplicates.
Foreign Key: One or more columns in one table that reference the primary key in another table, used to connect tables and create cross-referencing.
Defining Domains: A domain is the set of legal values that can be assigned to an attribute, ensuring data in a field is well-defined (e.g., only numbers in a numerical domain). This involves specifying data types, length values, and other relevant rules.
Using Constraints: Database constraints limit the type of data that can be stored in a table, ensuring data accuracy and reliability. Common constraints include NOT NULL (ensuring fields are always completed), UNIQUE (preventing duplicate values), CHECK (enforcing specific conditions), and FOREIGN KEY (maintaining referential integrity).
Importance of Planning: Designing a data model before building the database system allows for planning how data is stored and accessed efficiently. A poorly designed database can make it hard to produce accurate information.
Considerations at Scale: For large-scale applications like those at Meta, data modeling must prioritize user privacy, user safety, and scalability. It requires careful consideration of data access, encryption, and the ability to handle billions of users and evolving product needs. Thoughtfulness about future changes and the impact of modifications on existing data models is crucial.
Data Integrity and Quality: Well-designed data models, including the use of data types and constraints, are fundamental steps in ensuring the integrity and quality of a database.
Data modeling is an iterative process that requires a deep understanding of the data, the business requirements, and the capabilities of the chosen database system. It is a crucial skill for database engineers and a fundamental aspect of database design. Tools like MySQL Workbench can aid in creating, visualizing, and implementing data models.
Understanding Version Control: Git and Collaborative Development
Version Control Systems (VCS), also known as Source Control or Source Code Management, are systems that record all changes and modifications to files for tracking purposes. The primary goal of any VCS is to keep track of changes by allowing developers access to the entire change history with the ability to revert or roll back to a previous state or point in time. These systems track different types of changes such as adding new files, modifying or updating files, and deleting files. The version control system is the source of truth across all code assets and the team itself.
There are many benefits associated with Version Control, especially for developers working in a team. These include:
Revision history: Provides a record of all changes in a project and the ability for developers to revert to a stable point in time if code edits cause issues or bugs.
Identity: All changes made are recorded with the identity of the user who made them, allowing teams to see not only when changes occurred but also who made them.
Collaboration: A VCS allows teams to submit their code and keep track of any changes that need to be made when working towards a common goal. It also facilitates peer review where developers inspect code and provide feedback.
Automation and efficiency: Version Control helps keep track of all changes and plays an integral role in DevOps, increasing an organization’s ability to deliver applications or services with high quality and velocity. It aids in software quality, release, and deployments. By having Version Control in place, teams following agile methodologies can manage their tasks more efficiently.
Managing conflicts: Version Control helps developers fix any conflicts that may occur when multiple developers work on the same code base. The history of revisions can aid in seeing the full life cycle of changes and is essential for merging conflicts.
There are two main types or categories of Version Control Systems: centralized Version Control Systems (CVCS) and distributed Version Control Systems (DVCS).
Centralized Version Control Systems (CVCS) contain a server that houses the full history of the code base and clients that pull down the code. Developers need a connection to the server to perform any operations. Changes are pushed to the central server. An advantage of CVCS is that they are considered easier to learn and offer more access controls to users. A disadvantage is that they can be slower due to the need for a server connection.
Distributed Version Control Systems (DVCS) are similar, but every user is essentially a server and has the entire history of changes on their local system. Users don’t need to be connected to the server to add changes or view history, only to pull down the latest changes or push their own. DVCS offer better speed and performance and allow users to work offline. Git is an example of a DVCS.
Popular Version Control Technologies include git and GitHub. Git is a Version Control System designed to help users keep track of changes to files within their projects. It offers better speed and performance, reliability, free and open-source access, and an accessible syntax. Git is used predominantly via the command line. GitHub is a cloud-based hosting service that lets you manage git repositories from a user interface. It incorporates Git Version Control features and extends them with features like Access Control, pull requests, and automation. GitHub is very popular among web developers and acts like a social network for projects.
Key Git concepts include:
Repository: Used to track all changes to files in a specific folder and keep a history of all those changes. Repositories can be local (on your machine) or remote (e.g., on GitHub).
Clone: To copy a project from a remote repository to your local device.
Add: To stage changes in your local repository, preparing them for a commit.
Commit: To save a snapshot of the staged changes in the local repository’s history. Each commit is recorded with the identity of the user.
Push: To upload committed changes from your local repository to a remote repository.
Pull: To retrieve changes from a remote repository and apply them to your local repository.
Branching: Creating separate lines of development from the main codebase to work on new features or bug fixes in isolation. The main branch is often the source of truth.
Forking: Creating a copy of someone else’s repository on a platform like GitHub, allowing you to make changes without affecting the original.
Diff: A command to compare changes across files, branches, and commits.
Blame: A command to look at changes of a specific file and show the dates, times, and users who made the changes.
The typical Git workflow involves three states: modified, staged, and committed. Files are modified in the working directory, then added to the staging area, and finally committed to the local repository. These local commits are then pushed to a remote repository.
Branching workflows like feature branching are commonly used. This involves creating a new branch for each feature, working on it until completion, and then merging it back into the main branch after a pull request and peer review. Pull requests allow teams to review changes before they are merged.
At Meta, Version Control is very important. They use a giant monolithic repository for all of their backend code, which means code changes are shared with every other Instagram team. While this can be risky, it allows for code reuse. Meta encourages engineers to improve any code, emphasizing that “nothing at meta is someone else’s problem”. Due to the monolithic repository, merge conflicts happen a lot, so they try to write smaller changes and add gatekeepers to easily turn off features if needed. git blame is used daily to understand who wrote specific lines of code and why, which is particularly helpful in a large organization like Meta.
Version Control is also relevant to database development. It’s easy to overcomplicate data modeling and storage, and Version Control can help track changes and potentially revert to earlier designs. Planning how data will be organized (schema) is crucial before developing a database.
Learning to use git and GitHub for Version Control is part of the preparation for coding interviews in a final course, alongside practicing interview skills and refining resumes. Effective collaboration, which is enhanced by Version Control, is a crucial skill for software developers.
Python Programming Fundamentals: An Introduction
Based on the sources, here’s a discussion of Python programming basics:
Introduction to Python:
Python is a versatile and high-level programming language available on multiple platforms. It’s used in various areas like web development, data analytics, and business forecasting. Python’s syntax is similar to English, making it intuitive and easy for beginners to understand. Experienced programmers also appreciate its power and adaptability. Python was created by Guido van Rossum and released in 1991. It was designed to be readable and has similarities to English and mathematics. Since its release, it has gained significant popularity and has a rich selection of frameworks and libraries. Currently, it’s a popular language to learn, widely used in areas such as web development, artificial intelligence, machine learning, data analytics, and various programming applications. Python is easy to learn and get started with due to its English-like syntax. It also often requires less code compared to languages like C or Java. Python’s simplicity allows developers to focus on the task at hand, making it potentially quicker to get a product to market.
Setting up a Python Environment:
To start using Python, it’s essential to ensure it works correctly on your operating system with your chosen Integrated Development Environment (IDE), such as Visual Studio Code (VS Code). This involves making sure the right version of Python is used as the interpreter when running your code.
Installation Verification: You can verify if Python is installed by opening the terminal (or command prompt on Windows) and typing python –version. This should display the installed Python version.
VS Code Setup: VS Code offers a walkthrough guide for setting up Python. This includes installing Python (if needed) and selecting the correct Python interpreter.
Running Python Code: Python code can be run in a few ways:
Python Shell: Useful for running and testing small scripts without creating .py files. You can access it by typing python in the terminal.
Directly from Command Line/Terminal: Any file with the .py extension can be run by typing python followed by the file name (e.g., python hello.py).
Within an IDE (like VS Code): IDEs provide features like auto-completion, debugging, and syntax highlighting, making coding a better experience. VS Code has a run button to execute Python files.
Basic Syntax and Concepts:
Print Statement: The print() function is used to display output to the console. It can print different types of data and allows for formatting.
Variables: Variables are used to store data that can be changed throughout the program’s lifecycle. In Python, you declare a variable by assigning a value to a name (e.g., x = 5). Python automatically assigns the data type behind the scenes. There are conventions for naming variables, such as camel case (e.g., myName). You can declare multiple variables and assign them a single value (e.g., a = b = c = 10) or perform multiple assignments on one line (e.g., name, age = “Alice”, 30). You can also delete a variable using the del keyword.
Data Types: A data type indicates how a computer system should interpret a piece of data. Python offers several built-in data types:
Numeric: Includes int (integers), float (decimal numbers), and complex numbers.
Sequence: Ordered collections of items, including:
Strings (str): Sequences of characters enclosed in single or double quotes (e.g., “hello”, ‘world’). Individual characters in a string can be accessed by their index (starting from 0) using square brackets (e.g., name). The len() function returns the number of characters in a string.
Lists: Ordered and mutable sequences of items enclosed in square brackets (e.g., [1, 2, “three”]).
Tuples: Ordered and immutable sequences of items enclosed in parentheses (e.g., (1, 2, “three”)).
Dictionary (dict): Unordered collections of key-value pairs enclosed in curly braces (e.g., {“name”: “Bob”, “age”: 25}). Values are accessed using their keys.
Boolean (bool): Represents truth values: True or False.
Set (set): Unordered collections of unique elements enclosed in curly braces (e.g., {1, 2, 3}). Sets do not support indexing.
Typecasting: The process of converting one data type to another. Python supports implicit (automatic) and explicit (using functions like int(), float(), str()) type conversion.
Input: The input() function is used to take input from the user. It displays a prompt to the user and returns their input as a string.
Operators: Symbols used to perform operations on values.
Math Operators: Used for calculations (e.g., + for addition, – for subtraction, * for multiplication, / for division).
Logical Operators: Used in conditional statements to determine true or false outcomes (and, or, not).
Control Flow: Determines the order in which instructions in a program are executed.
Conditional Statements: Used to make decisions based on conditions (if, else, elif).
Loops: Used to repeatedly execute a block of code. Python has for loops (for iterating over sequences) and while loops (repeating a block until a condition is met). Nested loops are also possible.
Functions: Modular pieces of reusable code that take input and return output. You define a function using the def keyword. You can pass data into a function as arguments and return data using the return keyword. Python has different scopes for variables: local, enclosing, global, and built-in (LEGB rule).
Data Structures: Ways to organize and store data. Python includes lists, tuples, sets, and dictionaries.
This overview provides a foundation in Python programming basics as described in the provided sources. As you continue learning, you will delve deeper into these concepts and explore more advanced topics.
Database and Python Fundamentals Study Guide
Quiz
What is a database, and what is its typical organizational structure? A database is a systematically organized collection of data. This organization commonly resembles a spreadsheet or a table, with data containing elements and attributes for identification.
Explain the role of a Database Management System (DBMS) in the context of SQL. A DBMS acts as an intermediary between SQL instructions and the underlying database. It takes responsibility for transforming SQL commands into a format that the database can understand and execute.
Name and briefly define at least three sub-languages of SQL. DDL (Data Definition Language) is used to define data structures in a database, such as creating, altering, and dropping databases and tables. DML (Data Manipulation Language) is used for operational tasks like creating, reading, updating, and deleting data. DQL (Data Query Language) is used for retrieving data from the database.
Describe the purpose of the CREATE DATABASE and CREATE TABLE DDL statements. The CREATE DATABASE statement is used to create a new, empty database within the DBMS. The CREATE TABLE statement is used within a specific database to define a new table, including specifying the names and data types of its columns.
What is the function of the INSERT INTO DML statement? The INSERT INTO statement is used to add new rows of data into an existing table in the database. It requires specifying the table name and the values to be inserted into the table’s columns.
Explain the purpose of the NOT NULL constraint when defining table columns. The NOT NULL constraint ensures that a specific column in a table cannot contain a null value. If an attempt is made to insert a new record or update an existing one with a null value in a NOT NULL column, the operation will be aborted.
List and briefly define three basic arithmetic operators in SQL. The addition operator (+) is used to add two operands. The subtraction operator (-) is used to subtract the second operand from the first. The multiplication operator (*) is used to multiply two operands.
What is the primary function of the SELECT statement in SQL, and how can the WHERE clause be used with it? The SELECT statement is used to retrieve data from one or more tables in a database. The WHERE clause is used to filter the rows returned by the SELECT statement based on specified conditions.
Explain the difference between running Python code from the Python shell and running a .py file from the command line. The Python shell provides an interactive environment where you can execute Python code snippets directly and see immediate results without saving to a file. Running a .py file from the command line executes the entire script contained within the file non-interactively.
Define a variable in Python and provide an example of assigning it a value. In Python, a variable is a named storage location that holds a value. Variables are implicitly declared when a value is assigned to them. For example: x = 5 declares a variable named x and assigns it the integer value of 5.
Answer Key
A database is a systematically organized collection of data. This organization commonly resembles a spreadsheet or a table, with data containing elements and attributes for identification.
A DBMS acts as an intermediary between SQL instructions and the underlying database. It takes responsibility for transforming SQL commands into a format that the database can understand and execute.
DDL (Data Definition Language) helps you define data structures. DML (Data Manipulation Language) allows you to work with the data itself. DQL (Data Query Language) enables you to retrieve information from the database.
The CREATE DATABASE statement establishes a new database, while the CREATE TABLE statement defines the structure of a table within a database, including its columns and their data types.
The INSERT INTO statement adds new rows of data into a specified table. It requires indicating the table and the values to be placed into the respective columns.
The NOT NULL constraint enforces that a particular column must always have a value and cannot be left empty or contain a null entry when data is added or modified.
The + operator performs addition, the – operator performs subtraction, and the * operator performs multiplication between numerical values in SQL queries.
The SELECT statement retrieves data from database tables. The WHERE clause filters the results of a SELECT query, allowing you to specify conditions that rows must meet to be included in the output.
The Python shell is an interactive interpreter for immediate code execution, while running a .py file executes the entire script from the command line without direct interaction during the process.
A variable in Python is a name used to refer to a memory location that stores a value; for instance, name = “Alice” assigns the string value “Alice” to the variable named name.
Essay Format Questions
Discuss the significance of SQL as a standard language for database management. In your discussion, elaborate on at least three advantages of using SQL as highlighted in the provided text and provide examples of how these advantages contribute to efficient database operations.
Compare and contrast the roles of Data Definition Language (DDL) and Data Manipulation Language (DML) in SQL. Explain how these two sub-languages work together to enable the creation and management of data within a relational database system.
Explain the concept of scope in Python and discuss the LEGB rule. Provide examples to illustrate the differences between local, enclosed, global, and built-in scopes and explain how Python resolves variable names based on this rule.
Discuss the importance of modules in Python programming. Explain the advantages of using modules, such as reusability and organization, and describe different ways to import modules, including the use of import, from … import …, and aliases.
Imagine you are designing a simple database for a small online bookstore. Describe the tables you would create, the columns each table would have (including data types and any necessary constraints like NOT NULL or primary keys), and provide example SQL CREATE TABLE statements for two of your proposed tables.
Glossary of Key Terms
Database: A systematically organized collection of data that can be easily accessed, managed, and updated.
Table: A structure within a database used to organize data into rows (records) and columns (fields or attributes).
Column (Field): A vertical set of data values of a particular type within a table, representing an attribute of the entities stored in the table.
Row (Record): A horizontal set of data values within a table, representing a single instance of the entity being described.
SQL (Structured Query Language): A standard programming language used for managing and manipulating data in relational databases.
DBMS (Database Management System): Software that enables users to interact with a database, providing functionalities such as data storage, retrieval, and security.
DDL (Data Definition Language): A subset of SQL commands used to define the structure of a database, including creating, altering, and dropping databases, tables, and other database objects.
DML (Data Manipulation Language): A subset of SQL commands used to manipulate data within a database, including inserting, updating, deleting, and retrieving data.
DQL (Data Query Language): A subset of SQL commands, primarily the SELECT statement, used to query and retrieve data from a database.
Constraint: A rule or restriction applied to data in a database to ensure its accuracy, integrity, and reliability. Examples include NOT NULL.
Operator: A symbol or keyword that performs an operation on one or more operands. In SQL, this includes arithmetic operators (+, -, *, /), logical operators (AND, OR, NOT), and comparison operators (=, >, <, etc.).
Schema: The logical structure of a database, including the organization of tables, columns, relationships, and constraints.
Python Shell: An interactive command-line interpreter for Python, allowing users to execute code snippets and receive immediate feedback.
.py file: A file containing Python source code, which can be executed as a script from the command line.
Variable (Python): A named reference to a value stored in memory. Variables in Python are dynamically typed, meaning their data type is determined by the value assigned to them.
Data Type (Python): The classification of data that determines the possible values and operations that can be performed on it (e.g., integer, string, boolean).
String (Python): A sequence of characters enclosed in single or double quotes, used to represent text.
Scope (Python): The region of a program where a particular name (variable, function, etc.) is accessible. Python has four main scopes: local, enclosed, global, and built-in (LEGB).
Module (Python): A file containing Python definitions and statements. Modules provide a way to organize code into reusable units.
Import (Python): A statement used to load and make the code from another module available in the current script.
Alias (Python): An alternative name given to a module or function during import, often used for brevity or to avoid naming conflicts.
Briefing Document: Review of “01.pdf”
This briefing document summarizes the main themes and important concepts discussed in the provided excerpts from “01.pdf”. The document covers fundamental database concepts using SQL, basic command-line operations, an introduction to Python programming, and related software development tools.
I. Introduction to Databases and SQL
The document introduces the concept of databases as systematically organized data, often resembling spreadsheets or tables. It highlights the widespread use of databases in various applications, providing examples like banks storing account and transaction data, and hospitals managing patient, staff, and laboratory information.
“well a database looks like data organized systematically and this organization typically looks like a spreadsheet or a table”
The core purpose of SQL (Structured Query Language) is explained as a language used to interact with databases. Key operations that can be performed using SQL are outlined:
“operational terms create add or insert data read data update existing data and delete data”
SQL is further divided into several sub-languages:
DDL (Data Definition Language): Used to define the structure of the database and its objects like tables. Commands like CREATE (to create databases and tables) and ALTER (to modify existing objects, e.g., adding a column) are part of DDL.
“ddl as the name says helps you define data in your database but what does it mean to Define data before you can store data in the database you need to create the database and related objects like tables in which your data will be stored for this the ddl part of SQL has a command named create then you might need to modify already created database objects for example you might need to modify the structure of a table by adding a new column you can perform this task with the ddl alter command you can remove an object like a table from a”
DML (Data Manipulation Language): Used to manipulate the data within the database, including inserting (INSERT INTO), updating, and deleting data.
“now we need to populate the table of data this is where I can use the data manipulation language or DML subset of SQL to add table data I use the insert into syntax this inserts rows of data into a given table I just type insert into followed by the table name and then a list of required columns or Fields within a pair of parentheses then I add the values keyword”
DQL (Data Query Language): Primarily used for querying or retrieving data from the database (SELECT statements fall under this category).
DCL (Data Control Language): Used to control access and security within the database.
The document emphasizes that a DBMS (Database Management System) is crucial for interpreting and executing SQL instructions, acting as an intermediary between the SQL commands and the underlying database.
“a database interprets and makes sense of SQL instructions with the use of a database management system or dbms as a web developer you’ll execute all SQL instructions on a database using a dbms the dbms takes responsibility for transforming SQL instructions into a form that’s understood by the underlying database”
The advantages of using SQL are highlighted, including its simplicity, standardization, portability, comprehensiveness, and efficiency in processing large amounts of data.
“you now know that SQL is a simple standard portable comprehensive and efficient language that can be used to delete data retrieve and share data among multiple users and manage database security this is made possible through subsets of SQL like ddl or data definition language DML also known as data manipulation language dql or data query language and DCL also known as data control language and the final advantage of SQL is that it lets database users process large amounts of data quickly and efficiently”
Examples of basic SQL syntax are provided, such as creating a database (CREATE DATABASE College;) and creating a table (CREATE TABLE student ( … );). The INSERT INTO syntax for adding data to a table is also introduced.
Constraints like NOT NULL are mentioned as ways to enforce data integrity during table creation.
“the creation of a new customer record is aborted the not null default value is implemented using a SQL statement a typical not null SQL statement begins with the creation of a basic table in the database I can write a create table Clause followed by customer to define the table name followed by a pair of parentheses within the parentheses I add two columns customer ID and customer name I also Define each column with relevant data types end for customer ID as it stores”
SQL arithmetic operators (+, -, *, /, %) are introduced with examples. Logical operators (NOT, OR) and special operators (IN, BETWEEN) used in the WHERE clause for filtering data are also explained. The concept of JOIN clauses, including SELF-JOIN, for combining data from tables is briefly touched upon.
Subqueries (inner queries within outer queries) and Views (virtual tables based on the result of a query) are presented as advanced SQL concepts. User-defined functions and triggers are also introduced as ways to extend database functionality and automate actions. Prepared statements are mentioned as a more efficient way to execute SQL queries repeatedly. Date and time functions in MySQL are briefly covered.
II. Introduction to Command Line/Bash Shell
The document provides a basic introduction to using the command line or bash shell. Fundamental commands are explained:
PWD (Print Working Directory): Shows the current directory.
“to do that I run the PWD command PWD is short for print working directory I type PWD and press the enter key the command returns a forward slash which indicates that I’m currently in the root directory”
LS (List): Displays the contents of the current directory. The -l flag provides a detailed list format.
“if I want to check the contents of the root directory I run another command called LS which is short for list I type LS and press the enter key and now notice I get a list of different names of directories within the root level in order to get more detail of what each of the different directories represents I can use something called a flag flags are used to set options to the commands you run use the list command with a flag called L which means the format should be printed out in a list format I type LS space Dash l press enter and this Returns the results in a list structure”
CD (Change Directory): Navigates between directories using relative or absolute paths. cd .. moves up one directory.
“to step back into Etc type cdetc to confirm that I’m back there type bwd and enter if I want to use the other alternative you can do an absolute path type in CD forward slash and press enter Then I type PWD and press enter you can verify that I am back at the root again to step through multiple directories use the same process type CD Etc and press enter check the contents of the files by typing LS and pressing enter”
MKDIR (Make Directory): Creates a new directory.
“now I will create a new directory called submissions I do this by typing MK der which stands for make directory and then the word submissions this is the name of the directory I want to create and then I hit the enter key I then type in ls-l for list so that I can see the list structure and now notice that a new directory called submissions has been created I can then go into this”
TOUCH: Creates a new empty file.
“the Parent Directory next is the touch command which makes a new file of whatever type you specify for example to build a brand new file you can run touch followed by the new file’s name for instance example dot txt note that the newly created file will be empty”
HISTORY: Shows a history of recently used commands.
“to view a history of the most recently typed commands you can use the history command”
File Redirection (>, >>, <): Allows redirecting the input or output of commands to files. > overwrites, >> appends.
“if you want to control where the output goes you can use a redirection how do we do that enter the ls command enter Dash L to print it as a list instead of pressing enter add a greater than sign redirection now we have to tell it where we want the data to go in this scenario I choose an output.txt file the output dot txt file has not been created yet but it will be created based on the command I’ve set here with a redirection flag press enter type LS then press enter again to display the directory the output file displays to view the”
GREP: Searches for patterns within files.
“grep stands for Global regular expression print and it’s used for searching across files and folders as well as the contents of files on my local machine I enter the command ls-l and see that there’s a file called”
CAT: Displays the content of a file.
LESS: Views file content page by page.
“press the q key to exit the less environment the other file is the bash profile file so I can run the last command again this time with DOT profile this tends to be used used more for environment variables for example I can use it for setting”
VIM: A text editor used for creating and editing files.
“now I will create a simple shell script for this example I will use Vim which is an editor that I can use which accepts input so type vim and”
CHMOD: Changes file permissions, including making a file executable (chmod +x filename).
“but I want it to be executable which requires that I have an X being set on it in order to do that I have to use another command which is called chmod after using this them executable within the bash shell”
The document also briefly mentions shell scripts (files containing a series of commands) and environment variables (dynamic named values that can affect the way running processes will behave on a computer).
III. Introduction to Git and GitHub
Git is introduced as a free, open-source distributed version control system used to manage source code history, track changes, revert to previous versions, and collaborate with other developers. Key Git commands mentioned include:
GIT CLONE: Used to create a local copy of a remote repository (e.g., from GitHub).
“to do this I type the command git clone and paste the https URL I copied earlier finally I press enter on my keyboard notice that I receive a message stating”
LS -LA: Lists all files in a directory, including hidden ones (like the .git directory which contains the Git repository metadata).
“the ls-la command another file is listed which is just named dot get you will learn more about this later when you explore how to use this for Source control”
CD .git: Changes the current directory to the .git folder.
“first open the dot get folder on your terminal type CD dot git and press enter”
CAT HEAD: Displays the reference to the current commit.
“next type cat head and press enter in git we only work on a single Branch at a time this file also exists inside the dot get folder under the refs forward slash heads path”
CAT refs/heads/main: Displays the hash of the last commit on the main branch.
“type CD dot get and press enter next type cat forward slash refs forward slash heads forward slash main press enter after you”
GIT PULL: Fetches changes from a remote repository and integrates them into the local branch.
“I am now going to explain to you how to pull the repository to your local device”
GitHub is described as a cloud-based hosting service for Git repositories, offering a user interface for managing Git projects and facilitating collaboration.
IV. Introduction to Python Programming
The document introduces Python as a versatile programming language and outlines different ways to run Python code:
Python Shell: An interactive environment for running and testing small code snippets without creating separate files.
“the python shell is useful for running and testing small scripts for example it allows you to run code without the need for creating new DOT py files you start by adding Snippets of code that you can run directly in the shell”
Running Python Files: Executing Python code stored in files with the .py extension using the python filename.py command.
“running a python file directly from the command line or terminal note that any file that has the file extension of dot py can be run by the following command for example type python then a space and then type the file”
Basic Python concepts covered include:
Variables: Declaring and assigning values to variables (e.g., x = 5, name = “Alice”). Python automatically infers data types. Multiple variables can be assigned the same value (e.g., a = b = c = 10).
“all I have to do is name the variable for example if I type x equals 5 I have declared a variable and assigned as a value I can also print out the value of the variable by calling the print statement and passing in the variable name which in this case is X so I type print X when I run the program I get the value of 5 which is the assignment since I gave the initial variable Let Me Clear My screen again you have several options when it comes to declaring variables you can declare any different type of variable in terms of value for example X could equal a string called hello to do this I type x equals hello I can then print the value again run it and I find the output is the word hello behind the scenes python automatically assigns the data type for you”
Data Types: Basic data types like integers, floats (decimal numbers), complex numbers, strings (sequences of characters enclosed in single or double quotes), lists, and tuples (ordered, immutable sequences) are introduced.
“X could equal a string called hello to do this I type x equals hello I can then print the value again run it and I find the output is the word hello behind the scenes python automatically assigns the data type for you you’ll learn more about this in an upcoming video on data types you can declare multiple variables and assign them to a single value as well for example making a b and c all equal to 10. I do this by typing a equals b equals C equals 10. I print all three… sequence types are classed as container types that contain one or more of the same type in an ordered list they can also be accessed based on their index in the sequence python has three different sequence types namely strings lists and tuples let’s explore each of these briefly now starting with strings a string is a sequence of characters that is enclosed in either a single or double quotes strings are represented by the string class or Str for”
Operators: Arithmetic operators (+, -, *, /, **, %, //) and logical operators (and, or, not) are explained with examples.
“example 7 multiplied by four okay now let’s explore logical operators logical operators are used in Python on conditional statements to determine a true or false outcome let’s explore some of these now first logical operator is named and this operator checks for all conditions to be true for example a is greater than five and a is less than 10. the second logical operator is named or this operator checks for at least one of the conditions to be true for example a is greater than 5 or B is greater than 10. the final operator is named not this”
Conditional Statements: if, elif (else if), and else statements are introduced for controlling the flow of execution based on conditions.
“The Logical operators are and or and not let’s cover the different combinations of each in this example I declare two variables a equals true and B also equals true from these variables I use an if statement I type if a and b colon and on the next line I type print and in parentheses in double quotes”
Loops: for loops (for iterating over sequences) and while loops are introduced with examples, including nested loops.
“now let’s break apart the for Loop and discover how it works the variable item is a placeholder that will store the current letter in the sequence you may also recall that you can access any character in the sequence by its index the for Loop is accessing it in the same way and assigning the current value to the item variable this allows us to access the current character to print it for output when the code is run the outputs will be the letters of the word looping each letter on its own line now that you know about looping constructs in Python let me demonstrate how these work further using some code examples to Output an array of tasty desserts python offers us multiple ways to do loops or looping you’ll Now cover the for loop as well as the while loop let’s start with the basics of a simple for Loop to declare a for loop I use the four keyword I now need a variable to put the value into in this case I am using I I also use the in keyword to specify where I want to Loop over I add a new function called range to specify the number of items in a range in this case I’m using 10 as an example next I do a simple print statement by pressing the enter key to move to a new line I select the print function and within the brackets I enter the name looping and the value of I then I click on the Run button the output indicates the iteration Loops through the range of 0 to 9.”
Functions: Defining and calling functions using the def keyword. Functions can take arguments and return values. Examples of using *args (for variable positional arguments) and **kwargs (for variable keyword arguments) are provided.
“I now write a function to produce a string out of this information I type def contents and then self in parentheses on the next line I write a print statement for the string the plus self dot dish plus has plus self dot items plus and takes plus self dot time plus Min to prepare here we’ll use the backslash character to force a new line and continue the string on the following line for this to print correctly I need to convert the self dot items and self dot time… let’s say for example you wanted to calculate a total bill for a restaurant a user got a cup of coffee that was 2.99 then they also got a cake that was 455 and also a juice for 2.99. the first thing I could do is change the for Loop let’s change the argument to quarks by”
File Handling: Opening, reading (using read, readline, readlines), and writing to files. The importance of closing files is mentioned.
“the third method to read files in Python is read lines let me demonstrate this method the read lines method reads the entire contents of the file and then returns it in an ordered list this allows you to iterate over the list or pick out specific lines based on a condition if for example you have a file with four lines of text and pass a length condition the read files function will return the output all the lines in your file in the correct order files are stored in directories and they have”
Recursion: The concept of a function calling itself is briefly illustrated.
“the else statement will recursively call the slice function but with a modified string every time on the next line I add else and a colon then on the next line I type return string reverse Str but before I close the parentheses I add a slice function by typing open square bracket the number 1 and a colon followed by”
Object-Oriented Programming (OOP): Basic concepts of classes (using the class keyword), objects (instances of classes), attributes (data associated with an object), and methods (functions associated with an object, with self as the first parameter) are introduced. Inheritance (creating new classes based on existing ones) is also mentioned.
“method inside this class I want this one to contain a new function called leave request so I type def Leaf request and then self in days as the variables in parentheses the purpose of the leave request function is to return a line that specifies the number of days requested to write this I type return the string may I take a leave for plus Str open parenthesis the word days close parenthesis plus another string days now that I have all the classes in place I’ll create a few instances from these classes one for a supervisor and two others for… you will be defining a function called D inside which you will be creating another nested function e let’s write the rest of the code you can start by defining a couple of variables both of which will be called animal the first one inside the D function and the second one inside the E function note how you had to First declare the variable inside the E function as non-local you will now add a few more print statements for clarification for when you see the outputs finally you have called the E function here and you can add one more variable animal outside the D function this”
Modules: The concept of modules (reusable blocks of code in separate files) and how to import them using the import statement (e.g., import math, from math import sqrt, import math as m). The benefits of modular programming (scope, reusability, simplicity) are highlighted. The search path for modules (sys.path) is mentioned.
“so a file like sample.py can be a module named Sample and can be imported modules in Python can contain both executable statements and functions but before you explore how they are used it’s important to understand their value purpose and advantages modules come from modular programming this means that the functionality of code is broken down into parts or blocks of code these parts or blocks have great advantages which are scope reusability and simplicity let’s delve deeper into these everything in… to import and execute modules in Python the first important thing to know is that modules are imported only once during execution if for example your import a module that contains print statements print Open brackets close brackets you can verify it only executes the first time you import the module even if the module is imported multiple times since modules are built to help you Standalone… I will now import the built-in math module by typing import math just to make sure that this code works I’ll use a print statement I do this by typing print importing the math module after this I’ll run the code the print statement has executed most of the modules that you will come across especially the built-in modules will not have any print statements and they will simply be loaded by The Interpreter now that I’ve imported the math module I want to use a function inside of it let’s choose the square root function sqrt to do this I type the words math dot sqrt when I type the word math followed by the dot a list of functions appears in a drop down menu and you can select sqrt from this list I passed 9 as the argument to the math.sqrt function assign this to a variable called root and then I print it the number three the square root of nine has been printed to the terminal which is the correct answer instead of importing the entire math module as we did above there is a better way to handle this by directly importing the square root function inside the scope of the project this will prevent overloading The Interpreter by importing the entire math module to do this I type from math import sqrt when I run this it displays an error now I remove the word math from the variable declaration and I run the code again this time it works next let’s discuss something called an alias which is an excellent way of importing different modules here I sign an alias called m to the math module I do this by typing import math as m then I type cosine equals m dot I”
Scope: The concepts of local, enclosed, global, and built-in scopes in Python (LEGB rule) and how variable names are resolved. Keywords global and nonlocal for modifying variable scope are mentioned.
“names of different attributes defined inside it in this way modules are a type of namespace name spaces and Scopes can become very confusing very quickly and so it is important to get as much practice of Scopes as possible to ensure a standard of quality there are four main types of Scopes that can be defined in Python local enclosed Global and built in the practice of trying to determine in which scope a certain variable belongs is known as scope resolution scope resolution follows what is known commonly as the legb rule let’s explore these local this is where the first search for a variable is in the local scope enclosed this is defined inside an enclosing or nested functions Global is defined at the uppermost level or simply outside functions and built-in which is the keywords present in the built-in module in simpler terms a variable declared inside a function is local and the ones outside the scope of any function generally are global here is an example the outputs for the code on screen shows the same variable name Greek in different scopes… keywords that can be used to change the scope of the variables Global and non-local the global keyword helps us access the global variables from within the function non- local is a special type of scope defined in Python that is used within the nested functions only in the condition that it has been defined earlier in the enclosed functions now you can write a piece of code that will better help you understand the idea of scope for an attributes you have already created a file called animalfarm.py you will be defining a function called D inside which you will be creating another nested function e let’s write the rest of the code you can start by defining a couple of variables both of which will be called animal the first one inside the D function and the second one inside the E function note how you had to First declare the variable inside the E function as non-local you will now add a few more print statements for clarification for when you see the outputs finally you have called the E function here and you can add one more variable animal outside the D function this”
Reloading Modules: The reload() function for re-importing and re-executing modules that have already been loaded.
“statement is only loaded once by the python interpreter but the reload function lets you import and reload it multiple times I’ll demonstrate that first I create a new file sample.py and I add a simple print statement named hello world remember that any file in Python can be used as a module I’m going to use this file inside another new file and the new file is named using reloads.py now I import the sample.py module I can add the import statement multiple times but The Interpreter only loads it once if it had been reloaded we”
Testing: Introduction to writing test cases using the assert keyword and the pytest framework. The convention of naming test functions with the test_ prefix is mentioned. Test-Driven Development (TDD) is briefly introduced.
“another file called test Edition dot Pi in which I’m going to write my test cases now I import the file that consists of the functions that need to be tested next I’ll also import the pi test module after that I Define a couple of test cases with the addition and subtraction functions each test case should be named test underscore then the name of the function to be tested in our case we’ll have test underscore add and test underscore sub I’ll use the assert keyword inside these functions because tests primarily rely on this keyword it… contrary to the conventional approach of writing code I first write test underscore find string Dot py and then I add the test function named test underscore is present in accordance with the test I create another file named file string dot py in which I’ll write the is present function I Define the function named is present and I pass an argument called person in it then I make a list of names written as values after that I create a simple if else condition to check if the past argument”
V. Software Development Tools and Concepts
The document mentions several tools and concepts relevant to software development:
Python Installation and Version: Checking the installed Python version using python –version.
“prompt type python dash dash version to identify which version of python is running on your machine if python is correctly installed then Python 3 should appear in your console this means that you are running python 3. there should also be several numbers after the three to indicate which version of Python 3 you are running make sure these numbers match the most recent version on the python.org website if you see a message that states python not found then review your python installation or relevant document on”
Jupyter Notebook: An interactive development environment (IDE) for Python. Installation using python -m pip install jupyter and running using jupyter notebook are mentioned.
“course you’ll use the Jupiter put her IDE to demonstrate python to install Jupiter type python-mpip install Jupiter within your python environment then follow the jupyter installation process once you’ve installed jupyter type jupyter notebook to open a new instance of the jupyter notebook to use within your default browser”
MySQL Connector: A Python library used to connect Python applications to MySQL databases.
“the next task is to connect python to your mySQL database you can create the installation using a purpose-built python Library called MySQL connector this library is an API that provides useful”
Datetime Library: Python’s built-in module for working with dates and times. Functions like datetime.now(), datetime.date(), datetime.time(), and timedelta are introduced.
“python so you can import it without requiring pip let’s review the functions that Python’s daytime Library offers the date time Now function is used to retrieve today’s date you can also use date time date to retrieve just the date or date time time to call the current time and the time Delta function calculates the difference between two values now let’s look at the Syntax for implementing date time to import the daytime python class use the import code followed by the library name then use the as keyword to create an alias of… let’s look at a slightly more complex function time Delta when making plans it can be useful to project into the future for example what date is this same day next week you can answer questions like this using the time Delta function to calculate the difference between two values and return the result in a python friendly format so to find the date in seven days time you can create a new variable called week type the DT module and access the time Delta function as an object 563 instance then pass through seven days as an argument finally”
MySQL Workbench: A graphical tool for working with MySQL databases, including creating schemas.
“MySQL server instance and select the schema menu to create a new schema select the create schema option from the menu pane in the schema toolbar this action opens a new window within this new window enter mg underscore schema in the database name text field select apply this generates a SQL script called create schema mg schema you 606 are then asked to review the SQL script to be applied to your new database click on the apply button within the review window if you’re satisfied with the script a new window”
Data Warehousing: Briefly introduces the concept of a centralized data repository for integrating and processing large amounts of data from multiple sources for analysis. Dimensional data modeling is mentioned.
“in the next module you’ll explore the topic of data warehousing in this module you’ll learn about the architecture of a data warehouse and build a dimensional data model you’ll begin with an overview of the concept of data warehousing you’ll learn that a data warehouse is a centralized data repository that loads integrates stores and processes large amounts of data from multiple sources users can then query this data to perform data analysis you’ll then”
Binary Numbers: A basic explanation of the binary number system (base-2) is provided, highlighting its use in computing.
“binary has many uses in Computing it is a very convenient way of… consider that you have a lock with four different digits each digit can be a zero or a one how many potential past numbers can you have for the lock the answer is 2 to the power of four or two times two times two times two equals sixteen you are working with a binary lock therefore each digit can only be either zero or one so you can take four digits and multiply them by two every time and the total is 16. each time you add a potential digit you increase the”
Knapsack Problem: A brief overview of this optimization problem is given as a computational concept.
“three kilograms additionally each item has a value the torch equals one water equals two and the tent equals three in short the knapsack problem outlines a list of items that weigh different amounts and have different values you can only carry so many items in your knapsack the problem requires calculating the optimum combination of items you can carry if your backpack can carry a certain weight the goal is to find the best return for the weight capacity of the knapsack to compute a solution for this problem you must select all items”
This document provides a foundational overview of databases and SQL, command-line basics, version control with Git and GitHub, and introductory Python programming concepts, along with essential development tools. The content suggests a curriculum aimed at individuals learning about software development, data management, and related technologies.
Affiliate Disclosure: This blog may contain affiliate links, which means I may earn a small commission if you click on the link and make a purchase. This comes at no additional cost to you. I only recommend products or services that I believe will add value to my readers. Your support helps keep this blog running and allows me to continue providing you with quality content. Thank you for your support!
These resources provide a comprehensive pathway for aspiring database engineers and software developers. They cover fundamental database concepts like data modeling, SQL for data manipulation and management, database optimization, and data warehousing. Furthermore, they explore essential software development practices including Python programming, object-oriented principles, version control with Git and GitHub, software testing methodologies, and preparing for technical interviews with insights into data structures and algorithms.
Introduction to Database Engineering
This course provides a comprehensive introduction to database engineering. A straightforward description of a database is a form of electronic storage in which data is held. However, this simple explanation doesn’t fully capture the impact of database technology on global industry, government, and organizations. Almost everyone has used a database, and it’s likely that information about us is present in many databases worldwide.
Database engineering is crucial to global industry, government, and organizations. In a real-world context, databases are used in various scenarios:
Banks use databases to store data for customers, bank accounts, and transactions.
Hospitals store patient data, staff data, and laboratory data.
Online stores retain profile information, shopping history, and accounting transactions.
Social media platforms store uploaded photos.
Work environments use databases for downloading files.
Online games rely on databases.
Data in basic terms is facts and figures about anything. For example, data about a person might include their name, age, email, and date of birth, or it could be facts and figures related to an online purchase like the order number and description.
A database looks like data organized systematically, often resembling a spreadsheet or a table. This systematic organization means that all data contains elements or features and attributes by which they can be identified. For example, a person can be identified by attributes like name and age.
Data stored in a database cannot exist in isolation; it must have a relationship with other data to be processed into meaningful information. Databases establish relationships between pieces of data, for example, by retrieving a customer’s details from one table and their order recorded against another table. This is often achieved through keys. A primary key uniquely identifies each record in a table, while a foreign key is a primary key from one table that is used in another table to establish a link or relationship between the two. For instance, the customer ID in a customer table can be the primary key and then become a foreign key in an order table, thus relating the two tables.
While relational databases, which organize data into tables with relationships, are common, there are other types of databases. An object-oriented database stores data in the form of objects instead of tables or relations. An example could be an online bookstore where authors, customers, books, and publishers are rendered as classes, and the individual entries are objects or instances of these classes.
To work with data in databases, database engineers use Structured Query Language (SQL). SQL is a standard language that can be used with all relational databases like MySQL, PostgreSQL, Oracle, and Microsoft SQL Server. Database engineers establish interactions with databases to create, read, update, and delete (CRUD) data.
SQL can be divided into several sub-languages:
Data Definition Language (DDL) helps define data in the database and includes commands like CREATE (to create databases and tables), ALTER (to modify database objects), and DROP (to remove objects).
Data Manipulation Language (DML) is used to manipulate data and includes operations like INSERT (to add data), UPDATE (to modify data), and DELETE (to remove data).
Data Query Language (DQL) is used to read or retrieve data, primarily using the SELECT command.
Data Control Language (DCL) is used to control access to the database, with commands like GRANT and REVOKE to manage user privileges.
SQL offers several advantages:
It requires very little coding skills to use, consisting mainly of keywords.
Its interactivity allows developers to write complex queries quickly.
It is a standard language usable with all relational databases, leading to extensive support and information availability.
It is portable across operating systems.
Before developing a database, planning the organization of data is crucial, and this plan is called a schema. A schema is an organization or grouping of information and the relationships among them. In MySQL, schema and database are often interchangeable terms, referring to how data is organized. However, the definition of schema can vary across different database systems. A database schema typically comprises tables, columns, relationships, data types, and keys. Schemas provide logical groupings for database objects, simplify access and manipulation, and enhance database security by allowing permission management based on user access rights.
Database normalization is an important process used to structure tables in a way that minimizes challenges by reducing data duplication and avoiding data inconsistencies (anomalies). This involves converting a large table into multiple tables to reduce data redundancy. There are different normal forms (1NF, 2NF, 3NF) that define rules for table structure to achieve better database design.
As databases have evolved, they now must be able to store ever-increasing amounts of unstructured data, which poses difficulties. This growth has also led to concepts like big data and cloud databases.
Furthermore, databases play a crucial role in data warehousing, which involves a centralized data repository that loads, integrates, stores, and processes large amounts of data from multiple sources for data analysis. Dimensional data modeling, based on dimensions and facts, is often used to build databases in a data warehouse for data analytics. Databases also support data analytics, where collected data is converted into useful information to inform future decisions.
Tools like MySQL Workbench provide a unified visual environment for database modeling and management, supporting the creation of data models, forward and reverse engineering of databases, and SQL development.
Finally, interacting with databases can also be done through programming languages like Python using connectors or APIs (Application Programming Interfaces). This allows developers to build applications that interact with databases for various operations.
Understanding SQL: Language for Database Interaction
SQL (Structured Query Language) is a standard language used to interact with databases. It’s also commonly pronounced as “SQL”. Database engineers use SQL to establish interactions with databases.
Here’s a breakdown of SQL based on the provided source:
Role of SQL: SQL acts as the interface or bridge between a relational database and its users. It allows database engineers to create, read, update, and delete (CRUD) data. These operations are fundamental when working with a database.
Interaction with Databases: As a web developer or data engineer, you execute SQL instructions on a database using a Database Management System (DBMS). The DBMS is responsible for transforming SQL instructions into a form that the underlying database understands.
Applicability: SQL is particularly useful when working with relational databases, which require a language that can interact with structured data. Examples of relational databases that SQL can interact with include MySQL, PostgreSQL, Oracle, and Microsoft SQL Server.
SQL Sub-languages: SQL is divided into several sub-languages:
Data Definition Language (DDL): Helps you define data in your database. DDL commands include:
CREATE: Used to create databases and related objects like tables. For example, you can use the CREATE DATABASE command followed by the database name to create a new database. Similarly, CREATE TABLE followed by the table name and column definitions is used to create tables.
ALTER: Used to modify already created database objects, such as modifying the structure of a table by adding or removing columns (ALTER TABLE).
DROP: Used to remove objects like tables or entire databases. The DROP DATABASE command followed by the database name removes a database. The DROP COLUMN command removes a specific column from a table.
Data Manipulation Language (DML): Commands are used to manipulate data in the database and most CRUD operations fall under DML. DML commands include:
INSERT: Used to add or insert data into a table. The INSERT INTO syntax is used to add rows of data to a specified table.
UPDATE: Used to edit or modify existing data in a table. The UPDATE command allows you to specify data to be changed.
DELETE: Used to remove data from a table. The DELETE FROM syntax followed by the table name and an optional WHERE clause is used to remove data.
Data Query Language (DQL): Used to read or retrieve data from the database. The primary DQL command is:
SELECT: Used to select and retrieve data from one or multiple tables, allowing you to specify the columns you want and apply filter criteria using the WHERE clause. You can select all columns using SELECT *.
Data Control Language (DCL): Used to control access to the database. DCL commands include:
GRANT: Used to give users access privileges to data.
REVOKE: Used to revert access privileges already given to users.
Advantages of SQL: SQL is a popular language choice for databases due to several advantages:
Low coding skills required: It uses a set of keywords and requires very little coding.
Interactivity: Allows developers to write complex queries quickly.
Standard language: Can be used with all relational databases like MySQL, leading to extensive support and information availability.
Portability: Once written, SQL code can be used on any hardware and any operating system or platform where the database software is installed.
Comprehensive: Covers all areas of database management and administration, including creating databases, manipulating data, retrieving data, and managing security.
Efficiency: Allows database users to process large amounts of data quickly and efficiently.
Basic SQL Operations: SQL enables various operations on data, including:
Creating databases and tables using DDL.
Populating and modifying data using DML (INSERT, UPDATE, DELETE).
Reading and querying data using DQL (SELECT) with options to specify columns and filter data using the WHERE clause.
Sorting data using the ORDER BY clause with ASC (ascending) or DESC (descending) keywords.
Filtering data using the WHERE clause with various comparison operators (=, <, >, <=, >=, !=) and logical operators (AND, OR). Other filtering operators include BETWEEN, LIKE, and IN.
Removing duplicate rows using the SELECT DISTINCT clause.
Performing arithmetic operations using operators like +, -, *, /, and % (modulus) within SELECT statements.
Using comparison operators to compare values in WHERE clauses.
Utilizing aggregate functions (though not detailed in this initial overview but mentioned later in conjunction with GROUP BY).
Joining data from multiple tables (mentioned as necessary when data exists in separate entities). The source later details INNER JOIN, LEFT JOIN, and RIGHT JOIN clauses.
Creating aliases for tables and columns to make queries simpler and more readable.
Using subqueries (a query within another query) for more complex data retrieval.
Creating views (virtual tables based on the result of a SQL statement) to simplify data access and combine data from multiple tables.
Using stored procedures (pre-prepared SQL code that can be saved and executed).
Working with functions (numeric, string, date, comparison, control flow) to process and manipulate data.
Implementing triggers (stored programs that automatically execute in response to certain events).
Managing database transactions to ensure data integrity.
Optimizing queries for better performance.
Performing data analysis using SQL queries.
Interacting with databases using programming languages like Python through connectors and APIs.
In essence, SQL is a powerful and versatile language that is fundamental for anyone working with relational databases, enabling them to define, manage, query, and manipulate data effectively. The knowledge of SQL is a valuable skill for database engineers and is crucial for various tasks, from building and maintaining databases to extracting insights through data analysis.
Data Modeling Principles: Schema, Types, and Design
Data modeling principles revolve around creating a blueprint of how data will be organized and structured within a database system. This plan, often referred to as a schema, is essential for efficient data storage, access, updates, and querying. A well-designed data model ensures data consistency and quality.
Here are some key data modeling principles discussed in the sources:
Understanding Data Requirements: Before creating a database, it’s crucial to have a clear idea of its purpose and the data it needs to store. For example, a database for an online bookshop needs to record book titles, authors, customers, and sales. Mangata and Gallo (mng), a jewelry store, needed to store data on customers, products, and orders.
Visual Representation: A data model provides a visual representation of data elements (entities) and their relationships. This is often achieved using an Entity Relationship Diagram (ERD), which helps in planning entity-relational databases.
Different Levels of Abstraction: Data modeling occurs at different levels:
Conceptual Data Model: Provides a high-level, abstract view of the entities and their relationships in the database system. It focuses on “what” data needs to be stored (e.g., customers, products, orders as entities for mng) and how these relate.
Logical Data Model: Builds upon the conceptual model by providing a more detailed overview of the entities, their attributes, primary keys, and foreign keys. For mng, this would involve defining attributes for customers (like client ID as primary key), products, and orders, and specifying foreign keys to establish relationships (e.g., client ID in the orders table referencing the clients table).
Physical Data Model: Represents the internal schema of the database and is specific to the chosen Database Management System (DBMS). It outlines details like data types for each attribute (e.g., varchar for full name, integer for contact number), constraints (e.g., not null), and other database-specific features. SQL is often used to create the physical schema.
Choosing the Right Data Model Type: Several types of data models exist, each with its own advantages and disadvantages:
Relational Data Model: Represents data as a collection of tables (relations) with rows and columns, known for its simplicity.
Entity-Relationship Model: Similar to the relational model but presents each table as a separate entity with attributes and explicitly defines different types of relationships between entities (one-to-one, one-to-many, many-to-many).
Hierarchical Data Model: Organizes data in a tree-like structure with parent and child nodes, primarily supporting one-to-many relationships.
Object-Oriented Model: Translates objects into classes with characteristics and behaviors, supporting complex associations like aggregation and inheritance, suitable for complex projects.
Dimensional Data Model: Based on dimensions (context of measurements) and facts (quantifiable data), optimized for faster data retrieval and efficient data analytics, often using star and snowflake schemas in data warehouses.
Database Normalization: This is a crucial process for structuring tables to minimize data redundancy, avoid data modification implications (insertion, update, deletion anomalies), and simplify data queries. Normalization involves applying a series of normal forms (First Normal Form – 1NF, Second Normal Form – 2NF, Third Normal Form – 3NF) to ensure data atomicity, eliminate repeating groups, address functional and partial dependencies, and resolve transitive dependencies.
Establishing Relationships: Data in a database should be related to provide meaningful information. Relationships between tables are established using keys:
Primary Key: A value that uniquely identifies each record in a table and prevents duplicates.
Foreign Key: One or more columns in one table that reference the primary key in another table, used to connect tables and create cross-referencing.
Defining Domains: A domain is the set of legal values that can be assigned to an attribute, ensuring data in a field is well-defined (e.g., only numbers in a numerical domain). This involves specifying data types, length values, and other relevant rules.
Using Constraints: Database constraints limit the type of data that can be stored in a table, ensuring data accuracy and reliability. Common constraints include NOT NULL (ensuring fields are always completed), UNIQUE (preventing duplicate values), CHECK (enforcing specific conditions), and FOREIGN KEY (maintaining referential integrity).
Importance of Planning: Designing a data model before building the database system allows for planning how data is stored and accessed efficiently. A poorly designed database can make it hard to produce accurate information.
Considerations at Scale: For large-scale applications like those at Meta, data modeling must prioritize user privacy, user safety, and scalability. It requires careful consideration of data access, encryption, and the ability to handle billions of users and evolving product needs. Thoughtfulness about future changes and the impact of modifications on existing data models is crucial.
Data Integrity and Quality: Well-designed data models, including the use of data types and constraints, are fundamental steps in ensuring the integrity and quality of a database.
Data modeling is an iterative process that requires a deep understanding of the data, the business requirements, and the capabilities of the chosen database system. It is a crucial skill for database engineers and a fundamental aspect of database design. Tools like MySQL Workbench can aid in creating, visualizing, and implementing data models.
Understanding Version Control: Git and Collaborative Development
Version Control Systems (VCS), also known as Source Control or Source Code Management, are systems that record all changes and modifications to files for tracking purposes. The primary goal of any VCS is to keep track of changes by allowing developers access to the entire change history with the ability to revert or roll back to a previous state or point in time. These systems track different types of changes such as adding new files, modifying or updating files, and deleting files. The version control system is the source of truth across all code assets and the team itself.
There are many benefits associated with Version Control, especially for developers working in a team. These include:
Revision history: Provides a record of all changes in a project and the ability for developers to revert to a stable point in time if code edits cause issues or bugs.
Identity: All changes made are recorded with the identity of the user who made them, allowing teams to see not only when changes occurred but also who made them.
Collaboration: A VCS allows teams to submit their code and keep track of any changes that need to be made when working towards a common goal. It also facilitates peer review where developers inspect code and provide feedback.
Automation and efficiency: Version Control helps keep track of all changes and plays an integral role in DevOps, increasing an organization’s ability to deliver applications or services with high quality and velocity. It aids in software quality, release, and deployments. By having Version Control in place, teams following agile methodologies can manage their tasks more efficiently.
Managing conflicts: Version Control helps developers fix any conflicts that may occur when multiple developers work on the same code base. The history of revisions can aid in seeing the full life cycle of changes and is essential for merging conflicts.
There are two main types or categories of Version Control Systems: centralized Version Control Systems (CVCS) and distributed Version Control Systems (DVCS).
Centralized Version Control Systems (CVCS) contain a server that houses the full history of the code base and clients that pull down the code. Developers need a connection to the server to perform any operations. Changes are pushed to the central server. An advantage of CVCS is that they are considered easier to learn and offer more access controls to users. A disadvantage is that they can be slower due to the need for a server connection.
Distributed Version Control Systems (DVCS) are similar, but every user is essentially a server and has the entire history of changes on their local system. Users don’t need to be connected to the server to add changes or view history, only to pull down the latest changes or push their own. DVCS offer better speed and performance and allow users to work offline. Git is an example of a DVCS.
Popular Version Control Technologies include git and GitHub. Git is a Version Control System designed to help users keep track of changes to files within their projects. It offers better speed and performance, reliability, free and open-source access, and an accessible syntax. Git is used predominantly via the command line. GitHub is a cloud-based hosting service that lets you manage git repositories from a user interface. It incorporates Git Version Control features and extends them with features like Access Control, pull requests, and automation. GitHub is very popular among web developers and acts like a social network for projects.
Key Git concepts include:
Repository: Used to track all changes to files in a specific folder and keep a history of all those changes. Repositories can be local (on your machine) or remote (e.g., on GitHub).
Clone: To copy a project from a remote repository to your local device.
Add: To stage changes in your local repository, preparing them for a commit.
Commit: To save a snapshot of the staged changes in the local repository’s history. Each commit is recorded with the identity of the user.
Push: To upload committed changes from your local repository to a remote repository.
Pull: To retrieve changes from a remote repository and apply them to your local repository.
Branching: Creating separate lines of development from the main codebase to work on new features or bug fixes in isolation. The main branch is often the source of truth.
Forking: Creating a copy of someone else’s repository on a platform like GitHub, allowing you to make changes without affecting the original.
Diff: A command to compare changes across files, branches, and commits.
Blame: A command to look at changes of a specific file and show the dates, times, and users who made the changes.
The typical Git workflow involves three states: modified, staged, and committed. Files are modified in the working directory, then added to the staging area, and finally committed to the local repository. These local commits are then pushed to a remote repository.
Branching workflows like feature branching are commonly used. This involves creating a new branch for each feature, working on it until completion, and then merging it back into the main branch after a pull request and peer review. Pull requests allow teams to review changes before they are merged.
At Meta, Version Control is very important. They use a giant monolithic repository for all of their backend code, which means code changes are shared with every other Instagram team. While this can be risky, it allows for code reuse. Meta encourages engineers to improve any code, emphasizing that “nothing at meta is someone else’s problem”. Due to the monolithic repository, merge conflicts happen a lot, so they try to write smaller changes and add gatekeepers to easily turn off features if needed. git blame is used daily to understand who wrote specific lines of code and why, which is particularly helpful in a large organization like Meta.
Version Control is also relevant to database development. It’s easy to overcomplicate data modeling and storage, and Version Control can help track changes and potentially revert to earlier designs. Planning how data will be organized (schema) is crucial before developing a database.
Learning to use git and GitHub for Version Control is part of the preparation for coding interviews in a final course, alongside practicing interview skills and refining resumes. Effective collaboration, which is enhanced by Version Control, is a crucial skill for software developers.
Python Programming Fundamentals: An Introduction
Based on the sources, here’s a discussion of Python programming basics:
Introduction to Python:
Python is a versatile and high-level programming language available on multiple platforms. It’s used in various areas like web development, data analytics, and business forecasting. Python’s syntax is similar to English, making it intuitive and easy for beginners to understand. Experienced programmers also appreciate its power and adaptability. Python was created by Guido van Rossum and released in 1991. It was designed to be readable and has similarities to English and mathematics. Since its release, it has gained significant popularity and has a rich selection of frameworks and libraries. Currently, it’s a popular language to learn, widely used in areas such as web development, artificial intelligence, machine learning, data analytics, and various programming applications. Python is easy to learn and get started with due to its English-like syntax. It also often requires less code compared to languages like C or Java. Python’s simplicity allows developers to focus on the task at hand, making it potentially quicker to get a product to market.
Setting up a Python Environment:
To start using Python, it’s essential to ensure it works correctly on your operating system with your chosen Integrated Development Environment (IDE), such as Visual Studio Code (VS Code). This involves making sure the right version of Python is used as the interpreter when running your code.
Installation Verification: You can verify if Python is installed by opening the terminal (or command prompt on Windows) and typing python –version. This should display the installed Python version.
VS Code Setup: VS Code offers a walkthrough guide for setting up Python. This includes installing Python (if needed) and selecting the correct Python interpreter.
Running Python Code: Python code can be run in a few ways:
Python Shell: Useful for running and testing small scripts without creating .py files. You can access it by typing python in the terminal.
Directly from Command Line/Terminal: Any file with the .py extension can be run by typing python followed by the file name (e.g., python hello.py).
Within an IDE (like VS Code): IDEs provide features like auto-completion, debugging, and syntax highlighting, making coding a better experience. VS Code has a run button to execute Python files.
Basic Syntax and Concepts:
Print Statement: The print() function is used to display output to the console. It can print different types of data and allows for formatting.
Variables: Variables are used to store data that can be changed throughout the program’s lifecycle. In Python, you declare a variable by assigning a value to a name (e.g., x = 5). Python automatically assigns the data type behind the scenes. There are conventions for naming variables, such as camel case (e.g., myName). You can declare multiple variables and assign them a single value (e.g., a = b = c = 10) or perform multiple assignments on one line (e.g., name, age = “Alice”, 30). You can also delete a variable using the del keyword.
Data Types: A data type indicates how a computer system should interpret a piece of data. Python offers several built-in data types:
Numeric: Includes int (integers), float (decimal numbers), and complex numbers.
Sequence: Ordered collections of items, including:
Strings (str): Sequences of characters enclosed in single or double quotes (e.g., “hello”, ‘world’). Individual characters in a string can be accessed by their index (starting from 0) using square brackets (e.g., name). The len() function returns the number of characters in a string.
Lists: Ordered and mutable sequences of items enclosed in square brackets (e.g., [1, 2, “three”]).
Tuples: Ordered and immutable sequences of items enclosed in parentheses (e.g., (1, 2, “three”)).
Dictionary (dict): Unordered collections of key-value pairs enclosed in curly braces (e.g., {“name”: “Bob”, “age”: 25}). Values are accessed using their keys.
Boolean (bool): Represents truth values: True or False.
Set (set): Unordered collections of unique elements enclosed in curly braces (e.g., {1, 2, 3}). Sets do not support indexing.
Typecasting: The process of converting one data type to another. Python supports implicit (automatic) and explicit (using functions like int(), float(), str()) type conversion.
Input: The input() function is used to take input from the user. It displays a prompt to the user and returns their input as a string.
Operators: Symbols used to perform operations on values.
Math Operators: Used for calculations (e.g., + for addition, – for subtraction, * for multiplication, / for division).
Logical Operators: Used in conditional statements to determine true or false outcomes (and, or, not).
Control Flow: Determines the order in which instructions in a program are executed.
Conditional Statements: Used to make decisions based on conditions (if, else, elif).
Loops: Used to repeatedly execute a block of code. Python has for loops (for iterating over sequences) and while loops (repeating a block until a condition is met). Nested loops are also possible.
Functions: Modular pieces of reusable code that take input and return output. You define a function using the def keyword. You can pass data into a function as arguments and return data using the return keyword. Python has different scopes for variables: local, enclosing, global, and built-in (LEGB rule).
Data Structures: Ways to organize and store data. Python includes lists, tuples, sets, and dictionaries.
This overview provides a foundation in Python programming basics as described in the provided sources. As you continue learning, you will delve deeper into these concepts and explore more advanced topics.
Database and Python Fundamentals Study Guide
Quiz
What is a database, and what is its typical organizational structure? A database is a systematically organized collection of data. This organization commonly resembles a spreadsheet or a table, with data containing elements and attributes for identification.
Explain the role of a Database Management System (DBMS) in the context of SQL. A DBMS acts as an intermediary between SQL instructions and the underlying database. It takes responsibility for transforming SQL commands into a format that the database can understand and execute.
Name and briefly define at least three sub-languages of SQL. DDL (Data Definition Language) is used to define data structures in a database, such as creating, altering, and dropping databases and tables. DML (Data Manipulation Language) is used for operational tasks like creating, reading, updating, and deleting data. DQL (Data Query Language) is used for retrieving data from the database.
Describe the purpose of the CREATE DATABASE and CREATE TABLE DDL statements. The CREATE DATABASE statement is used to create a new, empty database within the DBMS. The CREATE TABLE statement is used within a specific database to define a new table, including specifying the names and data types of its columns.
What is the function of the INSERT INTO DML statement? The INSERT INTO statement is used to add new rows of data into an existing table in the database. It requires specifying the table name and the values to be inserted into the table’s columns.
Explain the purpose of the NOT NULL constraint when defining table columns. The NOT NULL constraint ensures that a specific column in a table cannot contain a null value. If an attempt is made to insert a new record or update an existing one with a null value in a NOT NULL column, the operation will be aborted.
List and briefly define three basic arithmetic operators in SQL. The addition operator (+) is used to add two operands. The subtraction operator (-) is used to subtract the second operand from the first. The multiplication operator (*) is used to multiply two operands.
What is the primary function of the SELECT statement in SQL, and how can the WHERE clause be used with it? The SELECT statement is used to retrieve data from one or more tables in a database. The WHERE clause is used to filter the rows returned by the SELECT statement based on specified conditions.
Explain the difference between running Python code from the Python shell and running a .py file from the command line. The Python shell provides an interactive environment where you can execute Python code snippets directly and see immediate results without saving to a file. Running a .py file from the command line executes the entire script contained within the file non-interactively.
Define a variable in Python and provide an example of assigning it a value. In Python, a variable is a named storage location that holds a value. Variables are implicitly declared when a value is assigned to them. For example: x = 5 declares a variable named x and assigns it the integer value of 5.
Answer Key
A database is a systematically organized collection of data. This organization commonly resembles a spreadsheet or a table, with data containing elements and attributes for identification.
A DBMS acts as an intermediary between SQL instructions and the underlying database. It takes responsibility for transforming SQL commands into a format that the database can understand and execute.
DDL (Data Definition Language) helps you define data structures. DML (Data Manipulation Language) allows you to work with the data itself. DQL (Data Query Language) enables you to retrieve information from the database.
The CREATE DATABASE statement establishes a new database, while the CREATE TABLE statement defines the structure of a table within a database, including its columns and their data types.
The INSERT INTO statement adds new rows of data into a specified table. It requires indicating the table and the values to be placed into the respective columns.
The NOT NULL constraint enforces that a particular column must always have a value and cannot be left empty or contain a null entry when data is added or modified.
The + operator performs addition, the – operator performs subtraction, and the * operator performs multiplication between numerical values in SQL queries.
The SELECT statement retrieves data from database tables. The WHERE clause filters the results of a SELECT query, allowing you to specify conditions that rows must meet to be included in the output.
The Python shell is an interactive interpreter for immediate code execution, while running a .py file executes the entire script from the command line without direct interaction during the process.
A variable in Python is a name used to refer to a memory location that stores a value; for instance, name = “Alice” assigns the string value “Alice” to the variable named name.
Essay Format Questions
Discuss the significance of SQL as a standard language for database management. In your discussion, elaborate on at least three advantages of using SQL as highlighted in the provided text and provide examples of how these advantages contribute to efficient database operations.
Compare and contrast the roles of Data Definition Language (DDL) and Data Manipulation Language (DML) in SQL. Explain how these two sub-languages work together to enable the creation and management of data within a relational database system.
Explain the concept of scope in Python and discuss the LEGB rule. Provide examples to illustrate the differences between local, enclosed, global, and built-in scopes and explain how Python resolves variable names based on this rule.
Discuss the importance of modules in Python programming. Explain the advantages of using modules, such as reusability and organization, and describe different ways to import modules, including the use of import, from … import …, and aliases.
Imagine you are designing a simple database for a small online bookstore. Describe the tables you would create, the columns each table would have (including data types and any necessary constraints like NOT NULL or primary keys), and provide example SQL CREATE TABLE statements for two of your proposed tables.
Glossary of Key Terms
Database: A systematically organized collection of data that can be easily accessed, managed, and updated.
Table: A structure within a database used to organize data into rows (records) and columns (fields or attributes).
Column (Field): A vertical set of data values of a particular type within a table, representing an attribute of the entities stored in the table.
Row (Record): A horizontal set of data values within a table, representing a single instance of the entity being described.
SQL (Structured Query Language): A standard programming language used for managing and manipulating data in relational databases.
DBMS (Database Management System): Software that enables users to interact with a database, providing functionalities such as data storage, retrieval, and security.
DDL (Data Definition Language): A subset of SQL commands used to define the structure of a database, including creating, altering, and dropping databases, tables, and other database objects.
DML (Data Manipulation Language): A subset of SQL commands used to manipulate data within a database, including inserting, updating, deleting, and retrieving data.
DQL (Data Query Language): A subset of SQL commands, primarily the SELECT statement, used to query and retrieve data from a database.
Constraint: A rule or restriction applied to data in a database to ensure its accuracy, integrity, and reliability. Examples include NOT NULL.
Operator: A symbol or keyword that performs an operation on one or more operands. In SQL, this includes arithmetic operators (+, -, *, /), logical operators (AND, OR, NOT), and comparison operators (=, >, <, etc.).
Schema: The logical structure of a database, including the organization of tables, columns, relationships, and constraints.
Python Shell: An interactive command-line interpreter for Python, allowing users to execute code snippets and receive immediate feedback.
.py file: A file containing Python source code, which can be executed as a script from the command line.
Variable (Python): A named reference to a value stored in memory. Variables in Python are dynamically typed, meaning their data type is determined by the value assigned to them.
Data Type (Python): The classification of data that determines the possible values and operations that can be performed on it (e.g., integer, string, boolean).
String (Python): A sequence of characters enclosed in single or double quotes, used to represent text.
Scope (Python): The region of a program where a particular name (variable, function, etc.) is accessible. Python has four main scopes: local, enclosed, global, and built-in (LEGB).
Module (Python): A file containing Python definitions and statements. Modules provide a way to organize code into reusable units.
Import (Python): A statement used to load and make the code from another module available in the current script.
Alias (Python): An alternative name given to a module or function during import, often used for brevity or to avoid naming conflicts.
Briefing Document: Review of “01.pdf”
This briefing document summarizes the main themes and important concepts discussed in the provided excerpts from “01.pdf”. The document covers fundamental database concepts using SQL, basic command-line operations, an introduction to Python programming, and related software development tools.
I. Introduction to Databases and SQL
The document introduces the concept of databases as systematically organized data, often resembling spreadsheets or tables. It highlights the widespread use of databases in various applications, providing examples like banks storing account and transaction data, and hospitals managing patient, staff, and laboratory information.
“well a database looks like data organized systematically and this organization typically looks like a spreadsheet or a table”
The core purpose of SQL (Structured Query Language) is explained as a language used to interact with databases. Key operations that can be performed using SQL are outlined:
“operational terms create add or insert data read data update existing data and delete data”
SQL is further divided into several sub-languages:
DDL (Data Definition Language): Used to define the structure of the database and its objects like tables. Commands like CREATE (to create databases and tables) and ALTER (to modify existing objects, e.g., adding a column) are part of DDL.
“ddl as the name says helps you define data in your database but what does it mean to Define data before you can store data in the database you need to create the database and related objects like tables in which your data will be stored for this the ddl part of SQL has a command named create then you might need to modify already created database objects for example you might need to modify the structure of a table by adding a new column you can perform this task with the ddl alter command you can remove an object like a table from a”
DML (Data Manipulation Language): Used to manipulate the data within the database, including inserting (INSERT INTO), updating, and deleting data.
“now we need to populate the table of data this is where I can use the data manipulation language or DML subset of SQL to add table data I use the insert into syntax this inserts rows of data into a given table I just type insert into followed by the table name and then a list of required columns or Fields within a pair of parentheses then I add the values keyword”
DQL (Data Query Language): Primarily used for querying or retrieving data from the database (SELECT statements fall under this category).
DCL (Data Control Language): Used to control access and security within the database.
The document emphasizes that a DBMS (Database Management System) is crucial for interpreting and executing SQL instructions, acting as an intermediary between the SQL commands and the underlying database.
“a database interprets and makes sense of SQL instructions with the use of a database management system or dbms as a web developer you’ll execute all SQL instructions on a database using a dbms the dbms takes responsibility for transforming SQL instructions into a form that’s understood by the underlying database”
The advantages of using SQL are highlighted, including its simplicity, standardization, portability, comprehensiveness, and efficiency in processing large amounts of data.
“you now know that SQL is a simple standard portable comprehensive and efficient language that can be used to delete data retrieve and share data among multiple users and manage database security this is made possible through subsets of SQL like ddl or data definition language DML also known as data manipulation language dql or data query language and DCL also known as data control language and the final advantage of SQL is that it lets database users process large amounts of data quickly and efficiently”
Examples of basic SQL syntax are provided, such as creating a database (CREATE DATABASE College;) and creating a table (CREATE TABLE student ( … );). The INSERT INTO syntax for adding data to a table is also introduced.
Constraints like NOT NULL are mentioned as ways to enforce data integrity during table creation.
“the creation of a new customer record is aborted the not null default value is implemented using a SQL statement a typical not null SQL statement begins with the creation of a basic table in the database I can write a create table Clause followed by customer to define the table name followed by a pair of parentheses within the parentheses I add two columns customer ID and customer name I also Define each column with relevant data types end for customer ID as it stores”
SQL arithmetic operators (+, -, *, /, %) are introduced with examples. Logical operators (NOT, OR) and special operators (IN, BETWEEN) used in the WHERE clause for filtering data are also explained. The concept of JOIN clauses, including SELF-JOIN, for combining data from tables is briefly touched upon.
Subqueries (inner queries within outer queries) and Views (virtual tables based on the result of a query) are presented as advanced SQL concepts. User-defined functions and triggers are also introduced as ways to extend database functionality and automate actions. Prepared statements are mentioned as a more efficient way to execute SQL queries repeatedly. Date and time functions in MySQL are briefly covered.
II. Introduction to Command Line/Bash Shell
The document provides a basic introduction to using the command line or bash shell. Fundamental commands are explained:
PWD (Print Working Directory): Shows the current directory.
“to do that I run the PWD command PWD is short for print working directory I type PWD and press the enter key the command returns a forward slash which indicates that I’m currently in the root directory”
LS (List): Displays the contents of the current directory. The -l flag provides a detailed list format.
“if I want to check the contents of the root directory I run another command called LS which is short for list I type LS and press the enter key and now notice I get a list of different names of directories within the root level in order to get more detail of what each of the different directories represents I can use something called a flag flags are used to set options to the commands you run use the list command with a flag called L which means the format should be printed out in a list format I type LS space Dash l press enter and this Returns the results in a list structure”
CD (Change Directory): Navigates between directories using relative or absolute paths. cd .. moves up one directory.
“to step back into Etc type cdetc to confirm that I’m back there type bwd and enter if I want to use the other alternative you can do an absolute path type in CD forward slash and press enter Then I type PWD and press enter you can verify that I am back at the root again to step through multiple directories use the same process type CD Etc and press enter check the contents of the files by typing LS and pressing enter”
MKDIR (Make Directory): Creates a new directory.
“now I will create a new directory called submissions I do this by typing MK der which stands for make directory and then the word submissions this is the name of the directory I want to create and then I hit the enter key I then type in ls-l for list so that I can see the list structure and now notice that a new directory called submissions has been created I can then go into this”
TOUCH: Creates a new empty file.
“the Parent Directory next is the touch command which makes a new file of whatever type you specify for example to build a brand new file you can run touch followed by the new file’s name for instance example dot txt note that the newly created file will be empty”
HISTORY: Shows a history of recently used commands.
“to view a history of the most recently typed commands you can use the history command”
File Redirection (>, >>, <): Allows redirecting the input or output of commands to files. > overwrites, >> appends.
“if you want to control where the output goes you can use a redirection how do we do that enter the ls command enter Dash L to print it as a list instead of pressing enter add a greater than sign redirection now we have to tell it where we want the data to go in this scenario I choose an output.txt file the output dot txt file has not been created yet but it will be created based on the command I’ve set here with a redirection flag press enter type LS then press enter again to display the directory the output file displays to view the”
GREP: Searches for patterns within files.
“grep stands for Global regular expression print and it’s used for searching across files and folders as well as the contents of files on my local machine I enter the command ls-l and see that there’s a file called”
CAT: Displays the content of a file.
LESS: Views file content page by page.
“press the q key to exit the less environment the other file is the bash profile file so I can run the last command again this time with DOT profile this tends to be used used more for environment variables for example I can use it for setting”
VIM: A text editor used for creating and editing files.
“now I will create a simple shell script for this example I will use Vim which is an editor that I can use which accepts input so type vim and”
CHMOD: Changes file permissions, including making a file executable (chmod +x filename).
“but I want it to be executable which requires that I have an X being set on it in order to do that I have to use another command which is called chmod after using this them executable within the bash shell”
The document also briefly mentions shell scripts (files containing a series of commands) and environment variables (dynamic named values that can affect the way running processes will behave on a computer).
III. Introduction to Git and GitHub
Git is introduced as a free, open-source distributed version control system used to manage source code history, track changes, revert to previous versions, and collaborate with other developers. Key Git commands mentioned include:
GIT CLONE: Used to create a local copy of a remote repository (e.g., from GitHub).
“to do this I type the command git clone and paste the https URL I copied earlier finally I press enter on my keyboard notice that I receive a message stating”
LS -LA: Lists all files in a directory, including hidden ones (like the .git directory which contains the Git repository metadata).
“the ls-la command another file is listed which is just named dot get you will learn more about this later when you explore how to use this for Source control”
CD .git: Changes the current directory to the .git folder.
“first open the dot get folder on your terminal type CD dot git and press enter”
CAT HEAD: Displays the reference to the current commit.
“next type cat head and press enter in git we only work on a single Branch at a time this file also exists inside the dot get folder under the refs forward slash heads path”
CAT refs/heads/main: Displays the hash of the last commit on the main branch.
“type CD dot get and press enter next type cat forward slash refs forward slash heads forward slash main press enter after you”
GIT PULL: Fetches changes from a remote repository and integrates them into the local branch.
“I am now going to explain to you how to pull the repository to your local device”
GitHub is described as a cloud-based hosting service for Git repositories, offering a user interface for managing Git projects and facilitating collaboration.
IV. Introduction to Python Programming
The document introduces Python as a versatile programming language and outlines different ways to run Python code:
Python Shell: An interactive environment for running and testing small code snippets without creating separate files.
“the python shell is useful for running and testing small scripts for example it allows you to run code without the need for creating new DOT py files you start by adding Snippets of code that you can run directly in the shell”
Running Python Files: Executing Python code stored in files with the .py extension using the python filename.py command.
“running a python file directly from the command line or terminal note that any file that has the file extension of dot py can be run by the following command for example type python then a space and then type the file”
Basic Python concepts covered include:
Variables: Declaring and assigning values to variables (e.g., x = 5, name = “Alice”). Python automatically infers data types. Multiple variables can be assigned the same value (e.g., a = b = c = 10).
“all I have to do is name the variable for example if I type x equals 5 I have declared a variable and assigned as a value I can also print out the value of the variable by calling the print statement and passing in the variable name which in this case is X so I type print X when I run the program I get the value of 5 which is the assignment since I gave the initial variable Let Me Clear My screen again you have several options when it comes to declaring variables you can declare any different type of variable in terms of value for example X could equal a string called hello to do this I type x equals hello I can then print the value again run it and I find the output is the word hello behind the scenes python automatically assigns the data type for you”
Data Types: Basic data types like integers, floats (decimal numbers), complex numbers, strings (sequences of characters enclosed in single or double quotes), lists, and tuples (ordered, immutable sequences) are introduced.
“X could equal a string called hello to do this I type x equals hello I can then print the value again run it and I find the output is the word hello behind the scenes python automatically assigns the data type for you you’ll learn more about this in an upcoming video on data types you can declare multiple variables and assign them to a single value as well for example making a b and c all equal to 10. I do this by typing a equals b equals C equals 10. I print all three… sequence types are classed as container types that contain one or more of the same type in an ordered list they can also be accessed based on their index in the sequence python has three different sequence types namely strings lists and tuples let’s explore each of these briefly now starting with strings a string is a sequence of characters that is enclosed in either a single or double quotes strings are represented by the string class or Str for”
Operators: Arithmetic operators (+, -, *, /, **, %, //) and logical operators (and, or, not) are explained with examples.
“example 7 multiplied by four okay now let’s explore logical operators logical operators are used in Python on conditional statements to determine a true or false outcome let’s explore some of these now first logical operator is named and this operator checks for all conditions to be true for example a is greater than five and a is less than 10. the second logical operator is named or this operator checks for at least one of the conditions to be true for example a is greater than 5 or B is greater than 10. the final operator is named not this”
Conditional Statements: if, elif (else if), and else statements are introduced for controlling the flow of execution based on conditions.
“The Logical operators are and or and not let’s cover the different combinations of each in this example I declare two variables a equals true and B also equals true from these variables I use an if statement I type if a and b colon and on the next line I type print and in parentheses in double quotes”
Loops: for loops (for iterating over sequences) and while loops are introduced with examples, including nested loops.
“now let’s break apart the for Loop and discover how it works the variable item is a placeholder that will store the current letter in the sequence you may also recall that you can access any character in the sequence by its index the for Loop is accessing it in the same way and assigning the current value to the item variable this allows us to access the current character to print it for output when the code is run the outputs will be the letters of the word looping each letter on its own line now that you know about looping constructs in Python let me demonstrate how these work further using some code examples to Output an array of tasty desserts python offers us multiple ways to do loops or looping you’ll Now cover the for loop as well as the while loop let’s start with the basics of a simple for Loop to declare a for loop I use the four keyword I now need a variable to put the value into in this case I am using I I also use the in keyword to specify where I want to Loop over I add a new function called range to specify the number of items in a range in this case I’m using 10 as an example next I do a simple print statement by pressing the enter key to move to a new line I select the print function and within the brackets I enter the name looping and the value of I then I click on the Run button the output indicates the iteration Loops through the range of 0 to 9.”
Functions: Defining and calling functions using the def keyword. Functions can take arguments and return values. Examples of using *args (for variable positional arguments) and **kwargs (for variable keyword arguments) are provided.
“I now write a function to produce a string out of this information I type def contents and then self in parentheses on the next line I write a print statement for the string the plus self dot dish plus has plus self dot items plus and takes plus self dot time plus Min to prepare here we’ll use the backslash character to force a new line and continue the string on the following line for this to print correctly I need to convert the self dot items and self dot time… let’s say for example you wanted to calculate a total bill for a restaurant a user got a cup of coffee that was 2.99 then they also got a cake that was 455 and also a juice for 2.99. the first thing I could do is change the for Loop let’s change the argument to quarks by”
File Handling: Opening, reading (using read, readline, readlines), and writing to files. The importance of closing files is mentioned.
“the third method to read files in Python is read lines let me demonstrate this method the read lines method reads the entire contents of the file and then returns it in an ordered list this allows you to iterate over the list or pick out specific lines based on a condition if for example you have a file with four lines of text and pass a length condition the read files function will return the output all the lines in your file in the correct order files are stored in directories and they have”
Recursion: The concept of a function calling itself is briefly illustrated.
“the else statement will recursively call the slice function but with a modified string every time on the next line I add else and a colon then on the next line I type return string reverse Str but before I close the parentheses I add a slice function by typing open square bracket the number 1 and a colon followed by”
Object-Oriented Programming (OOP): Basic concepts of classes (using the class keyword), objects (instances of classes), attributes (data associated with an object), and methods (functions associated with an object, with self as the first parameter) are introduced. Inheritance (creating new classes based on existing ones) is also mentioned.
“method inside this class I want this one to contain a new function called leave request so I type def Leaf request and then self in days as the variables in parentheses the purpose of the leave request function is to return a line that specifies the number of days requested to write this I type return the string may I take a leave for plus Str open parenthesis the word days close parenthesis plus another string days now that I have all the classes in place I’ll create a few instances from these classes one for a supervisor and two others for… you will be defining a function called D inside which you will be creating another nested function e let’s write the rest of the code you can start by defining a couple of variables both of which will be called animal the first one inside the D function and the second one inside the E function note how you had to First declare the variable inside the E function as non-local you will now add a few more print statements for clarification for when you see the outputs finally you have called the E function here and you can add one more variable animal outside the D function this”
Modules: The concept of modules (reusable blocks of code in separate files) and how to import them using the import statement (e.g., import math, from math import sqrt, import math as m). The benefits of modular programming (scope, reusability, simplicity) are highlighted. The search path for modules (sys.path) is mentioned.
“so a file like sample.py can be a module named Sample and can be imported modules in Python can contain both executable statements and functions but before you explore how they are used it’s important to understand their value purpose and advantages modules come from modular programming this means that the functionality of code is broken down into parts or blocks of code these parts or blocks have great advantages which are scope reusability and simplicity let’s delve deeper into these everything in… to import and execute modules in Python the first important thing to know is that modules are imported only once during execution if for example your import a module that contains print statements print Open brackets close brackets you can verify it only executes the first time you import the module even if the module is imported multiple times since modules are built to help you Standalone… I will now import the built-in math module by typing import math just to make sure that this code works I’ll use a print statement I do this by typing print importing the math module after this I’ll run the code the print statement has executed most of the modules that you will come across especially the built-in modules will not have any print statements and they will simply be loaded by The Interpreter now that I’ve imported the math module I want to use a function inside of it let’s choose the square root function sqrt to do this I type the words math dot sqrt when I type the word math followed by the dot a list of functions appears in a drop down menu and you can select sqrt from this list I passed 9 as the argument to the math.sqrt function assign this to a variable called root and then I print it the number three the square root of nine has been printed to the terminal which is the correct answer instead of importing the entire math module as we did above there is a better way to handle this by directly importing the square root function inside the scope of the project this will prevent overloading The Interpreter by importing the entire math module to do this I type from math import sqrt when I run this it displays an error now I remove the word math from the variable declaration and I run the code again this time it works next let’s discuss something called an alias which is an excellent way of importing different modules here I sign an alias called m to the math module I do this by typing import math as m then I type cosine equals m dot I”
Scope: The concepts of local, enclosed, global, and built-in scopes in Python (LEGB rule) and how variable names are resolved. Keywords global and nonlocal for modifying variable scope are mentioned.
“names of different attributes defined inside it in this way modules are a type of namespace name spaces and Scopes can become very confusing very quickly and so it is important to get as much practice of Scopes as possible to ensure a standard of quality there are four main types of Scopes that can be defined in Python local enclosed Global and built in the practice of trying to determine in which scope a certain variable belongs is known as scope resolution scope resolution follows what is known commonly as the legb rule let’s explore these local this is where the first search for a variable is in the local scope enclosed this is defined inside an enclosing or nested functions Global is defined at the uppermost level or simply outside functions and built-in which is the keywords present in the built-in module in simpler terms a variable declared inside a function is local and the ones outside the scope of any function generally are global here is an example the outputs for the code on screen shows the same variable name Greek in different scopes… keywords that can be used to change the scope of the variables Global and non-local the global keyword helps us access the global variables from within the function non- local is a special type of scope defined in Python that is used within the nested functions only in the condition that it has been defined earlier in the enclosed functions now you can write a piece of code that will better help you understand the idea of scope for an attributes you have already created a file called animalfarm.py you will be defining a function called D inside which you will be creating another nested function e let’s write the rest of the code you can start by defining a couple of variables both of which will be called animal the first one inside the D function and the second one inside the E function note how you had to First declare the variable inside the E function as non-local you will now add a few more print statements for clarification for when you see the outputs finally you have called the E function here and you can add one more variable animal outside the D function this”
Reloading Modules: The reload() function for re-importing and re-executing modules that have already been loaded.
“statement is only loaded once by the python interpreter but the reload function lets you import and reload it multiple times I’ll demonstrate that first I create a new file sample.py and I add a simple print statement named hello world remember that any file in Python can be used as a module I’m going to use this file inside another new file and the new file is named using reloads.py now I import the sample.py module I can add the import statement multiple times but The Interpreter only loads it once if it had been reloaded we”
Testing: Introduction to writing test cases using the assert keyword and the pytest framework. The convention of naming test functions with the test_ prefix is mentioned. Test-Driven Development (TDD) is briefly introduced.
“another file called test Edition dot Pi in which I’m going to write my test cases now I import the file that consists of the functions that need to be tested next I’ll also import the pi test module after that I Define a couple of test cases with the addition and subtraction functions each test case should be named test underscore then the name of the function to be tested in our case we’ll have test underscore add and test underscore sub I’ll use the assert keyword inside these functions because tests primarily rely on this keyword it… contrary to the conventional approach of writing code I first write test underscore find string Dot py and then I add the test function named test underscore is present in accordance with the test I create another file named file string dot py in which I’ll write the is present function I Define the function named is present and I pass an argument called person in it then I make a list of names written as values after that I create a simple if else condition to check if the past argument”
V. Software Development Tools and Concepts
The document mentions several tools and concepts relevant to software development:
Python Installation and Version: Checking the installed Python version using python –version.
“prompt type python dash dash version to identify which version of python is running on your machine if python is correctly installed then Python 3 should appear in your console this means that you are running python 3. there should also be several numbers after the three to indicate which version of Python 3 you are running make sure these numbers match the most recent version on the python.org website if you see a message that states python not found then review your python installation or relevant document on”
Jupyter Notebook: An interactive development environment (IDE) for Python. Installation using python -m pip install jupyter and running using jupyter notebook are mentioned.
“course you’ll use the Jupiter put her IDE to demonstrate python to install Jupiter type python-mpip install Jupiter within your python environment then follow the jupyter installation process once you’ve installed jupyter type jupyter notebook to open a new instance of the jupyter notebook to use within your default browser”
MySQL Connector: A Python library used to connect Python applications to MySQL databases.
“the next task is to connect python to your mySQL database you can create the installation using a purpose-built python Library called MySQL connector this library is an API that provides useful”
Datetime Library: Python’s built-in module for working with dates and times. Functions like datetime.now(), datetime.date(), datetime.time(), and timedelta are introduced.
“python so you can import it without requiring pip let’s review the functions that Python’s daytime Library offers the date time Now function is used to retrieve today’s date you can also use date time date to retrieve just the date or date time time to call the current time and the time Delta function calculates the difference between two values now let’s look at the Syntax for implementing date time to import the daytime python class use the import code followed by the library name then use the as keyword to create an alias of… let’s look at a slightly more complex function time Delta when making plans it can be useful to project into the future for example what date is this same day next week you can answer questions like this using the time Delta function to calculate the difference between two values and return the result in a python friendly format so to find the date in seven days time you can create a new variable called week type the DT module and access the time Delta function as an object 563 instance then pass through seven days as an argument finally”
MySQL Workbench: A graphical tool for working with MySQL databases, including creating schemas.
“MySQL server instance and select the schema menu to create a new schema select the create schema option from the menu pane in the schema toolbar this action opens a new window within this new window enter mg underscore schema in the database name text field select apply this generates a SQL script called create schema mg schema you 606 are then asked to review the SQL script to be applied to your new database click on the apply button within the review window if you’re satisfied with the script a new window”
Data Warehousing: Briefly introduces the concept of a centralized data repository for integrating and processing large amounts of data from multiple sources for analysis. Dimensional data modeling is mentioned.
“in the next module you’ll explore the topic of data warehousing in this module you’ll learn about the architecture of a data warehouse and build a dimensional data model you’ll begin with an overview of the concept of data warehousing you’ll learn that a data warehouse is a centralized data repository that loads integrates stores and processes large amounts of data from multiple sources users can then query this data to perform data analysis you’ll then”
Binary Numbers: A basic explanation of the binary number system (base-2) is provided, highlighting its use in computing.
“binary has many uses in Computing it is a very convenient way of… consider that you have a lock with four different digits each digit can be a zero or a one how many potential past numbers can you have for the lock the answer is 2 to the power of four or two times two times two times two equals sixteen you are working with a binary lock therefore each digit can only be either zero or one so you can take four digits and multiply them by two every time and the total is 16. each time you add a potential digit you increase the”
Knapsack Problem: A brief overview of this optimization problem is given as a computational concept.
“three kilograms additionally each item has a value the torch equals one water equals two and the tent equals three in short the knapsack problem outlines a list of items that weigh different amounts and have different values you can only carry so many items in your knapsack the problem requires calculating the optimum combination of items you can carry if your backpack can carry a certain weight the goal is to find the best return for the weight capacity of the knapsack to compute a solution for this problem you must select all items”
This document provides a foundational overview of databases and SQL, command-line basics, version control with Git and GitHub, and introductory Python programming concepts, along with essential development tools. The content suggests a curriculum aimed at individuals learning about software development, data management, and related technologies.
Affiliate Disclosure: This blog may contain affiliate links, which means I may earn a small commission if you click on the link and make a purchase. This comes at no additional cost to you. I only recommend products or services that I believe will add value to my readers. Your support helps keep this blog running and allows me to continue providing you with quality content. Thank you for your support!
Affiliate Disclosure: This blog may contain affiliate links, which means I may earn a small commission if you click on the link and make a purchase. This comes at no additional cost to you. I only recommend products or services that I believe will add value to my readers. Your support helps keep this blog running and allows me to continue providing you with quality content. Thank you for your support!
This document provides an extensive overview of linear algebra, focusing on its foundational concepts and practical applications, particularly within machine learning. It introduces systems of linear equations and their representation using vectors and matrices, explaining key properties like singularity, linear dependence, and rank. The text details methods for solving systems of equations, including Gaussian elimination and row reduction, and explores matrix operations such as multiplication and inversion. Finally, it connects these mathematical principles to linear transformations, determinants, eigenvalues, eigenvectors, and principal component analysis (PCA), demonstrating how linear algebra forms the backbone of various data science techniques.
Matrices: Foundations, Properties, and Machine Learning Applications
Matrices are fundamental objects in linear algebra, often described as arrays of numbers inside a rectangle. They are central to machine learning and data science, providing a deeper understanding of how algorithms work, enabling customization of models, aiding in debugging, and potentially leading to the invention of new algorithms.
Here’s a comprehensive discussion of matrices based on the sources:
Representation of Systems of Linear Equations
Matrices provide a compact and natural way to express systems of linear equations. For example, a system like “A + B + C = 10” can be represented using a matrix of coefficients multiplied by a vector of variables, equaling a vector of constants.
In a matrix corresponding to a system, each row represents an equation, and each column represents the coefficients of a variable. This is particularly useful in machine learning models like linear regression, where a dataset can be seen as a system of linear equations, with features forming a matrix (X) and weights forming a vector (W).
Properties of Matrices
Singularity and Non-Singularity: Just like systems of linear equations, matrices can be singular or non-singular.
A non-singular matrix corresponds to a system with a unique solution. Geometrically, for 2×2 matrices, this means the lines corresponding to the equations intersect at a unique point. For 3×3 matrices, planes intersect at a single point. A non-singular system is “complete,” carrying as many independent pieces of information as sentences/equations.
A singular matrix corresponds to a system that is either redundant (infinitely many solutions) or contradictory (no solutions). For 2×2 matrices, this means the lines either overlap (redundant, infinitely many solutions) or are parallel and never meet (contradictory, no solutions). For 3×3 matrices, singular systems might result in planes intersecting along a line (infinitely many solutions) or having no common intersection.
Crucially, the constants in a system of linear equations do not affect whether the system (or its corresponding matrix) is singular or non-singular. Setting constants to zero simplifies the visualization and analysis of singularity.
Linear Dependence and Independence: This concept is key to understanding singularity.
A matrix is singular if its rows (or columns) are linearly dependent, meaning one row (or column) can be obtained as a linear combination of others. This indicates that the corresponding equation does not introduce new information to the system.
A matrix is non-singular if its rows (or columns) are linearly independent, meaning no row (or column) can be obtained from others. Each equation provides unique information.
Determinant: The determinant is a quick formula to tell if a matrix is singular or non-singular.
For a 2×2 matrix with entries A, B, C, D, the determinant is AD – BC.
For a 3×3 matrix, it involves summing products of elements along main diagonals and subtracting products along anti-diagonals, potentially with a “wrapping around” concept for incomplete diagonals.
A matrix has a determinant of zero if it is singular, and a non-zero determinant if it is non-singular.
Geometric Interpretation: The determinant quantifies how much a linear transformation (represented by the matrix) stretches or shrinks space. For a 2×2 matrix, the determinant is the area of the image of the fundamental unit square after transformation. If the transformation maps the plane to a line or a point (singular), the area (determinant) is zero.
Properties of Determinants: The determinant of a product of matrices (A * B) is the product of their individual determinants (Det(A) * Det(B)). If one matrix in a product is singular, the resulting product matrix will also be singular. The determinant of an inverse matrix (A⁻¹) is 1 divided by the determinant of the original matrix (1/Det(A)). The determinant of the identity matrix is always one.
Rank: The rank of a matrix measures how much information the matrix (or its corresponding system of linear equations) carries.
For systems of sentences, rank is the number of pieces of information conveyed. For systems of equations, it’s the number of new, independent pieces of information.
The rank of a matrix is the dimension of the image of its linear transformation.
A matrix is non-singular if and only if it has full rank, meaning its rank equals the number of rows.
The rank can be easily calculated by finding the number of ones (pivots) in the diagonal of its row echelon form.
Inverse Matrix: An inverse matrix (denoted A⁻¹) is a special matrix that, when multiplied by the original matrix, results in the identity matrix.
In terms of linear transformations, the inverse matrix “undoes” the job of the original matrix, returning the plane to its original state.
A matrix has an inverse if and only if it is non-singular (i.e., its determinant is non-zero). Singular matrices do not have an inverse.
Finding the inverse involves solving a system of linear equations.
Matrix Operations
Transpose: This operation converts rows into columns and columns into rows. It is denoted by a superscript ‘T’ (e.g., Aᵀ).
Scalar Multiplication: Multiplying a matrix (or vector) by a scalar involves multiplying each element of the matrix (or vector) by that scalar.
Dot Product: While often applied to vectors, the concept extends to matrix multiplication. It involves summing the products of corresponding entries of two vectors.
Matrix-Vector Multiplication: This is seen as a stack of dot products, where each row of the matrix takes a dot product with the vector. The number of columns in the matrix must equal the length of the vector for this operation to be defined. This is how systems of equations are expressed.
Matrix-Matrix Multiplication: This operation combines two linear transformations into a third one. To multiply matrices, you take rows from the first matrix and columns from the second, performing dot products to fill in each cell of the resulting matrix. The number of columns in the first matrix must match the number of rows in the second matrix.
Visualization as Linear Transformations
Matrices can be powerfully visualized as linear transformations, which send points in one space to points in another in a structured way. For example, a 2×2 matrix transforms a square (basis) into a parallelogram.
This perspective helps explain concepts like the determinant (area/volume scaling) and singularity (mapping a plane to a lower-dimensional space like a line or a point).
Applications in Machine Learning
Linear Regression: Datasets are treated as systems of linear equations, where matrices represent features (X) and weights (W).
Neural Networks: These powerful models are essentially large collections of linear models built on matrix operations. Data (inputs, outputs of layers) is represented as vectors, matrices, and tensors (higher-dimensional matrices). Matrix multiplication is used to combine inputs with weights and biases across different layers. Simple neural networks (perceptrons) can act as linear classifiers, using matrix products followed by a threshold check.
Image Compression: The rank of a matrix is related to the amount of space needed to store an image (which can be represented as a matrix). Techniques like Singular Value Decomposition (SVD) can reduce the rank of an image matrix, making it take up less space while preserving visual quality.
Principal Component Analysis (PCA): This dimensionality reduction algorithm uses matrices extensively.
It constructs a covariance matrix from data, which compactly represents relationships between variables.
PCA then finds the eigenvalues and eigenvectors of the covariance matrix. The eigenvector with the largest eigenvalue indicates the direction of greatest variance in the data, which is the “principal component” or the line/plane onto which data should be projected to preserve the most information.
The process involves centering data, calculating the covariance matrix, finding its eigenvalues and eigenvectors, and then projecting the data onto the eigenvectors corresponding to the largest eigenvalues.
Discrete Dynamical Systems: Matrices can represent transition probabilities in systems that evolve over time (e.g., weather patterns, web traffic). These are often Markov matrices, where columns sum to one. Multiplying a state vector by the transition matrix predicts future states, eventually stabilizing into an equilibrium vector, which is an eigenvector with an eigenvalue of one.
The instructor for this specialization, Luis Serrano, who has a PhD in pure math and worked as an ML engineer at Google and Apple, is thrilled to bring math to life with visual examples. Andrew Ng highlights that understanding the math behind machine learning, especially linear algebra, allows for deeper understanding, better customization, effective debugging, and even the invention of new algorithms.
Think of a matrix like a versatile chef’s knife in a machine learning kitchen. It can be used for many tasks: precisely slicing and dicing your data (matrix operations), combining ingredients in complex recipes (neural network layers), and even reducing a huge block of ingredients to its essential flavors (PCA for dimensionality reduction). Just as a sharp knife makes a chef more effective, mastering matrices makes a machine learning practitioner more capable.
Matrices as Dynamic Linear Transformations
Linear transformations are a powerful and intuitive way to understand matrices, visualizing them not just as static arrays of numbers, but as dynamic operations that transform space. Luis Serrano, the instructor, emphasizes seeing matrices in this deeper, more illustrative way, much like a book is more than just an array of letters.
Here’s a discussion of linear transformations:
What is a Linear Transformation?
A linear transformation is a way to send each point in the plane into another point in the plane in a very structured way. Imagine two planes, with a transformation sending points from the left plane to the right plane.
It operates on a point (represented as a column vector) by multiplying it by a matrix.
A key property is that the origin (0,0) always gets sent to the origin (0,0).
For a 2×2 matrix, a linear transformation takes a fundamental square (or a basis) and transforms it into a parallelogram. This is also referred to as a “change of basis”.
Matrices as Linear Transformations
A matrix is a linear transformation. This means that every matrix has an associated linear transformation, and every linear transformation can be represented by a unique matrix.
To find the matrix corresponding to a linear transformation, you only need to observe where the fundamental basis vectors (like (1,0) and (0,1)) are sent; these transformed vectors become the columns of the matrix.
Properties and Interpretations Through Linear Transformations
Singularity:
A transformation is non-singular if the resulting points, after multiplication by the matrix, cover the entire plane (or the entire original space). For example, a 2×2 matrix transforming a square into a parallelogram that still covers the whole plane is non-singular.
A transformation is singular if it maps the entire plane to a lower-dimensional space, such as a line or even just a single point.
If the original square is transformed into a line segment (a “degenerate parallelogram”), the transformation is singular.
If it maps the entire plane to just the origin (0,0), it’s highly singular.
This directly relates to the matrix’s singularity: a matrix is non-singular if and only if its corresponding linear transformation is non-singular.
Determinant:
The determinant of a matrix has a powerful geometric interpretation: it represents the area (for 2D) or volume (for 3D) of the image of the fundamental unit square (or basis) after the transformation.
If the transformation is singular, the area (or volume) of the transformed shape becomes zero, which is why a singular matrix has a determinant of zero.
A negative determinant indicates that the transformation has “flipped” or reoriented the space, but it still represents a non-singular transformation as long as the absolute value is non-zero.
Determinant of a product of matrices: When combining two linear transformations (which is what matrix multiplication does), the determinant of the resulting transformation is the product of the individual determinants. This makes intuitive sense: if the first transformation stretches an area by a factor of 5 and the second by a factor of 3, the combined transformation stretches it by 5 * 3 = 15.
Determinant of an inverse matrix: The determinant of the inverse of a matrix (A⁻¹) is 1 divided by the determinant of the original matrix (1/Det(A)). This reflects that the inverse transformation “undoes” the scaling of the original transformation.
The identity matrix (which leaves the plane intact, sending each point to itself) has a determinant of one, meaning it doesn’t stretch or shrink space at all.
Inverse Matrix:
The inverse matrix (A⁻¹) is the one that “undoes” the job of the original matrix, effectively returning the transformed plane to its original state.
A matrix has an inverse if and only if its determinant is non-zero; therefore, only non-singular matrices (and their corresponding non-singular transformations) have an inverse.
Rank:
The rank of a matrix (or a linear transformation) measures how much information it carries.
Geometrically, the rank of a linear transformation is the dimension of its image.
If the transformation maps a plane to a plane, its image dimension is two, and its rank is two.
If it maps a plane to a line, its image dimension is one, and its rank is one.
If it maps a plane to a point, its image dimension is zero, and its rank is zero.
Eigenvalues and Eigenvectors:
Eigenvectors are special vectors whose direction is not changed by a linear transformation; they are only stretched or shrunk.
The eigenvalue is the scalar factor by which an eigenvector is stretched.
Visualizing a transformation through its eigenbasis (a basis composed of eigenvectors) simplifies it significantly, as the transformation then appears as just a collection of stretches, with no rotation or shear.
Along an eigenvector, a complex matrix multiplication becomes a simple scalar multiplication, greatly simplifying computations.
Finding eigenvalues involves solving the characteristic polynomial, derived from setting the determinant of (A – λI) to zero.
Applications in Machine Learning
Understanding linear transformations is crucial for various machine learning algorithms.
Neural Networks: These are fundamentally large collections of linear models built on matrix operations that “warp space”. Data (inputs, outputs of layers) is represented as vectors, matrices, and even higher-dimensional tensors, and matrix multiplication is used to combine inputs with weights and biases across layers. A simple one-layer neural network (perceptron) can be directly viewed as a matrix product followed by a threshold check.
Principal Component Analysis (PCA): This dimensionality reduction technique leverages linear transformations extensively.
PCA first computes the covariance matrix of a dataset, which describes how variables relate to each other and characterizes the data’s spread.
It then finds the eigenvalues and eigenvectors of this covariance matrix.
The eigenvector with the largest eigenvalue represents the direction of greatest variance in the data.
By projecting the data onto these principal eigenvectors, PCA reduces the data’s dimensions while preserving as much information (spread) as possible.
Discrete Dynamical Systems: Matrices, especially Markov matrices (where columns sum to one, representing probabilities), are used to model systems that evolve over time, like weather patterns. Multiplying a state vector by the transition matrix predicts future states. The system eventually stabilizes into an equilibrium vector, which is an eigenvector with an eigenvalue of one, representing the long-term probabilities of the system’s states.
Think of linear transformations as the fundamental dance moves that matrices perform on data. Just as a dance can stretch, shrink, or rotate, these transformations reshape data in predictable ways, making complex operations manageable and interpretable, especially for tasks like data compression or understanding the core patterns in large datasets.
Eigenvalues and Eigenvectors: Machine Learning Foundations
Eigenvalues and eigenvectors are fundamental concepts in linear algebra, particularly crucial for understanding and applying various machine learning algorithms. They provide a powerful way to characterize linear transformations.
What are Eigenvalues and Eigenvectors?
Definition:
Eigenvectors are special vectors whose direction is not changed by a linear transformation. When a linear transformation is applied to an eigenvector, the eigenvector simply gets stretched or shrunk, but it continues to point in the same direction.
The eigenvalue is the scalar factor by which an eigenvector is stretched or shrunk. If the eigenvalue is positive, the vector is stretched in its original direction; if negative, it’s stretched and its direction is flipped.
Mathematical Relationship: The relationship is formalized by the equation A * v = λ * v.
Here, A represents the matrix (linear transformation).
v represents the eigenvector.
λ (lambda) represents the eigenvalue (a scalar).
This equation means that applying the linear transformation A to vector v yields the same result as simply multiplying v by the scalar λ.
Significance and Properties
Directional Stability: The most intuitive property is that eigenvectors maintain their direction through a transformation.
Simplifying Complex Operations: Along an eigenvector, a complex matrix multiplication becomes a simple scalar multiplication. This is a major computational simplification, as matrix multiplication typically involves many operations, while scalar multiplication is trivial.
Eigenbasis: If a set of eigenvectors forms a basis for the space (an “eigenbasis”), the linear transformation can be seen as merely a collection of stretches along those eigenvector directions, with no rotation or shear. This provides a greatly simplified view of the transformation.
Geometric Interpretation: Eigenvectors tell you the directions in which a linear transformation is just a stretch, and eigenvalues tell you how much it is stretched. For instance, a transformation can stretch some vectors by a factor of 11 and others by a factor of 1.
Applicability: Eigenvalues and eigenvectors are only defined for square matrices.
How to Find Eigenvalues and Eigenvectors
The process involves two main steps:
Finding Eigenvalues (λ):
This is done by solving the characteristic polynomial.
The characteristic polynomial is derived from setting the determinant of (A – λI) to zero. I is the identity matrix of the same size as A.
The roots (solutions for λ) of this polynomial are the eigenvalues. For example, for a 2×2 matrix, the characteristic polynomial will be a quadratic equation, and for a 3×3 matrix, it will be a cubic equation.
Finding Eigenvectors (v):
Once the eigenvalues (λ) are found, each eigenvalue is substituted back into the equation (A – λI)v = 0.
Solving this system of linear equations for v will yield the corresponding eigenvector. Since any scalar multiple of an eigenvector is also an eigenvector for the same eigenvalue (as only the direction matters), there will always be infinitely many solutions, typically represented as a line or plane of vectors.
Number of Eigenvectors:
For a matrix with distinct eigenvalues, you will always get a distinct eigenvector for each eigenvalue.
However, if an eigenvalue is repeated (e.g., appears twice as a root of the characteristic polynomial), it’s possible to find fewer distinct eigenvectors than the number of times the eigenvalue is repeated. For instance, a 3×3 matrix might have two eigenvalues of ‘2’ but only one distinct eigenvector associated with ‘2’.
Applications in Machine Learning
Eigenvalues and eigenvectors play critical roles in several machine learning algorithms:
Principal Component Analysis (PCA):
PCA is a dimensionality reduction algorithm that aims to reduce the number of features (columns) in a dataset while preserving as much information (variance) as possible.
It achieves this by first calculating the covariance matrix of the data, which describes how variables relate to each other and captures the data’s spread.
The eigenvalues and eigenvectors of this covariance matrix are then computed.
The eigenvector with the largest eigenvalue represents the direction of greatest variance in the data. This direction is called the first principal component.
By projecting the data onto these principal eigenvectors (those corresponding to the largest eigenvalues), PCA effectively transforms the data into a new, lower-dimensional space that captures the most significant patterns or spread in the original data.
Discrete Dynamical Systems (e.g., Markov Chains):
Matrices, specifically Markov matrices (where columns sum to one, representing probabilities), are used to model systems that evolve over time, like weather patterns or website navigation.
Multiplying a state vector by the transition matrix predicts future states.
Over many iterations, the system tends to stabilize into an equilibrium vector. This equilibrium vector is an eigenvector with an eigenvalue of one, representing the long-term, stable probabilities of the system’s states. Regardless of the initial state, the system will eventually converge to this equilibrium eigenvector.
Think of eigenvalues and eigenvectors as the natural modes of motion for a transformation. Just as striking a bell makes it vibrate at its fundamental frequencies, applying a linear transformation to data makes certain directions (eigenvectors) “resonate” by simply stretching, and the “intensity” of that stretch is given by the eigenvalue. Understanding these “resonances” allows us to simplify complex data and systems.
Principal Component Analysis: How it Works
Principal Component Analysis (PCA) is a powerful dimensionality reduction algorithm that is widely used in machine learning and data science. Its primary goal is to reduce the number of features (columns) in a dataset while preserving as much information as possible. This reduction makes datasets easier to manage and visualize, especially when dealing with hundreds or thousands of features.
How PCA Works
The process of PCA leverages fundamental concepts from statistics and linear algebra, particularly eigenvalues and eigenvectors.
Here’s a step-by-step breakdown of how PCA operates:
Data Preparation and Centering:
PCA begins with a dataset, typically represented as a matrix where rows are observations and columns are features (variables).
The first step is to center the data by calculating the mean (average value) for each feature and subtracting it from all values in that column. This ensures that the dataset is centered around the origin (0,0).
Calculating the Covariance Matrix:
Next, PCA computes the covariance matrix of the centered data.
The covariance matrix is a square matrix that compactly stores the relationships between pairs of variables.
Its diagonal elements represent the variance of each individual variable, which measures how spread out the data is along that variable’s axis.
The off-diagonal elements represent the covariance between pairs of variables, quantifying how two features vary together. A positive covariance indicates that variables tend to increase or decrease together, while a negative covariance indicates an inverse relationship.
A key property of the covariance matrix is that it is symmetric around its diagonal.
Finding Eigenvalues and Eigenvectors of the Covariance Matrix:
This is the crucial step where linear algebra comes into play. As discussed, eigenvectors are special vectors whose direction is not changed by a linear transformation, only scaled by a factor (the eigenvalue).
In the context of PCA, the covariance matrix represents a linear transformation that characterizes the spread and relationships within your data.
When you find the eigenvalues and eigenvectors of the covariance matrix, you are identifying the “natural modes” or directions of variance in your data.
The eigenvectors (often called principal components in PCA) indicate the directions in which the data has the greatest variance (spread).
The eigenvalues quantify the amount of variance along their corresponding eigenvectors. A larger eigenvalue means a greater spread of data along that eigenvector’s direction.
For a symmetric matrix like the covariance matrix, the eigenvectors will always be orthogonal (at a 90-degree angle) to one another.
Selecting Principal Components:
Once the eigenvalues and eigenvectors are computed, they are sorted in descending order based on their eigenvalues.
The eigenvector with the largest eigenvalue represents the first principal component, capturing the most variance in the data. The second-largest eigenvalue corresponds to the second principal component, and so on.
To reduce dimensionality, PCA selects a subset of these principal components – specifically, those corresponding to the largest eigenvalues – and discards the rest. The number of components kept determines the new, lower dimensionality of the dataset.
Projecting Data onto Principal Components:
Finally, the original (centered) data is projected onto the selected principal components.
Projection involves transforming data points into a new, lower-dimensional space defined by these principal eigenvectors. This is done by multiplying the centered data matrix by a matrix formed by the selected principal components (scaled to have a norm of one).
The result is a new, reduced dataset that has the same number of observations but fewer features (columns). Crucially, this new dataset still preserves the maximum possible variance from the original data, meaning it retains the most significant information and patterns.
Benefits of PCA
Data Compression: It creates a more compact dataset, which is easier to store and process, especially with high-dimensional data.
Information Preservation: It intelligently reduces dimensions while minimizing the loss of useful information by focusing on directions of maximum variance.
Visualization: By reducing complex data to two or three dimensions, PCA enables easier visualization and exploratory analysis, allowing patterns to become more apparent.
Think of PCA like finding the best angle to take a picture of a scattered cloud of points. If you take a picture from an arbitrary angle, some points might overlap, and you might lose the sense of the cloud’s overall shape. However, if you find the angle from which the cloud appears most stretched out or “spread,” that picture captures the most defining features of the cloud. The eigenvectors are these “best angles” or directions, and their eigenvalues tell you how “stretched” the cloud appears along those angles, allowing you to pick the most informative views.
Linear Algebra for Machine Learning and Data Science
Affiliate Disclosure: This blog may contain affiliate links, which means I may earn a small commission if you click on the link and make a purchase. This comes at no additional cost to you. I only recommend products or services that I believe will add value to my readers. Your support helps keep this blog running and allows me to continue providing you with quality content. Thank you for your support!
The sources provided offer a comprehensive exploration of database concepts, beginning with foundational elements of Entity-Relationship (ER) models, including entities, attributes, and relationships. They distinguish between various types of attributes (derived, multi-valued, composite, descriptive) and keys (super, candidate, primary, foreign), explaining their roles in uniquely identifying and linking data. The text transitions into relational models, detailing how ER constructs are converted into tables and the importance of referential integrity. A significant portion focuses on relational algebra as a procedural query language, breaking down fundamental operators like selection, projection, union, set difference, Cartesian product, and joins (inner and outer), and illustrating their application through practical examples. Finally, the sources touch upon relational calculus (tuple and domain) as non-procedural alternatives and introduce SQL, emphasizing its syntax for data retrieval and modification (insert, delete, update).
Data Modeling: ER and Relational Models Explained
Data modeling is a fundamental concept in database management systems (DBMS) that serves as a blueprint or structure for how data is stored and accessed. It provides conceptual tools to describe various aspects of data:
Data itself.
Data relationships.
Consistency constraints.
Data meaning (semantics).
The goal of data modeling is to establish a structured format for storing data to ensure efficient retrieval and management. It is crucial because information derived from processed data is highly valuable for decision-making, which is why companies invest significantly in data.
There are primarily two phases in database design that involve data modeling:
Designing the ER (Entity-Relationship) Model: This is the first, high-level design phase.
Converting the ER Model into a Relational Model: This phase translates the high-level design into a structured format suitable for relational databases.
Let’s delve into the types and key aspects of data models discussed in the sources:
Types of Data Models
The sources categorize data within a database system into two broad types: structured and unstructured.
Structured Data: This type of data has a proper format, often tabular. Examples include data from Indian railways or university data. Different patterns for storing structured data include:
Key-value pairs: Used for high-speed lookups.
Column-oriented databases: Store data column by column instead of row by row.
Graph databases: Data is stored in nodes, with relationships depicted by edges (e.g., social media recommendation systems).
Document-oriented databases: Used in systems like MongoDB.
Object-oriented databases: Store data as objects.
Unstructured Data: This data does not have a proper format, such as a mix of videos, text, and images found on a website.
For strictly tabular and structured data, a relational database management system (RDBMS) is considered the best choice. However, for better performance, scalability, or special use cases, other database types can serve as alternatives.
The Entity-Relationship (ER) Model
The ER model is a high-level data model that is easily understandable even by non-technical persons. It is based on the perception of real-world objects and the relationships among them. The ER model acts as a bridge to understand the relational model, allowing for high-level design that can then be implemented in a relational database.
Key constructs in the ER model include:
Entities: Represent real-world objects (e.g., student, car). Entities can be:
Entity Type: The class blueprint or table definition (e.g., “Student” table).
Entity Instance: A specific record or row with filled values (e.g., a specific student’s record).
Entity Set: A collection of all entity instances of a particular type.
Strong Entity Type: Can exist independently and has its own primary key (also called regular or independent entity type).
Weak Entity Type: Depends on the existence of a strong entity type and does not have its own primary key (also called dependent entity type). Its instances are uniquely identified with the help of a discriminator (a unique attribute within the weak entity) and the primary key of the strong entity type it depends on. A weak entity type always has total participation in its identifying relationship.
Attributes: These are the properties that describe an entity type (e.g., for a “Fighter” entity, attributes could be ranking, weight, reach, record, age). Each attribute has a domain (set of permissible values), which can be enforced by domain constraints. Attributes can be categorized as:
Simple: Atomic, cannot be subdivided (e.g., gender).
Composite: Can be subdivided (e.g., address into street, locality).
Single-valued: Holds a single value (e.g., role number).
Multivalued: Can hold multiple values (e.g., phone number, email).
Stored: Cannot be derived from other attributes (e.g., date of birth).
Derived: Can be calculated or derived from other stored attributes (e.g., age from date of birth).
Descriptive: Attributes of a relationship (e.g., “since” in “employee works in department”).
Relationships: Represent an association between instances of different entity types (e.g., “customer borrows loan”). Relationships have a degree (unary, binary, ternary) and cardinality ratios (based on maximum participation like one-to-one, one-to-many, many-to-one, many-to-many, and minimum participation like total or partial).
Total Participation: Means every instance of an entity type must participate in the relationship (minimum cardinality of one).
Partial Participation: Means instances of an entity type may or may not participate in the relationship (minimum cardinality of zero), which is the default setting.
The ER model is not a complete model on its own because it does not define the storage format or manipulation language (like SQL). However, it is a crucial conceptual tool for designing high-level database structures.
The Relational Model
Developed by E.F. Codd in 1970, the relational model dictates that data will be stored in a tabular format. Its popularity stems from its simplicity, ease of use and understanding, and its strong mathematical foundation.
In the relational model:
Tables (relations): Practical forms where data of interest is stored.
Columns (attributes, fields): Represent properties of the data.
Schema: The blueprint of the database, including attributes, constraints, and relationships.
Integrity Constraints: Rules to ensure data correctness and consistency. These include domain constraints, entity integrity (primary key unique and not null), referential integrity (foreign key values are a subset of parent table’s primary key values), null constraints, default value constraints, and uniqueness constraints.
The relational model is considered a complete model because it answers the three fundamental questions of data modeling:
Storage Format: Data is stored in tables.
Manipulation Language: SQL (Structured Query Language) is used for data manipulation.
Integrity Constraints: It defines various integrity rules for data correctness.
When converting an ER model to a relational model, each entity type (strong or weak) is typically converted into a single table. Multivalued attributes usually require a separate table, while composite attributes are flattened into the original table. Relationships are represented either by incorporating foreign keys into existing tables or by creating separate tables for the relationships themselves, depending on the cardinality and participation constraints.
In summary, data modeling is the conceptual process of organizing data and its relationships within a database. The ER model provides a high-level design, serving as a conceptual bridge to the more detailed and mathematically rigorous relational model, which defines how data is physically stored and manipulated in tables using languages like SQL.
Relational Algebra: Operators and Concepts
Relational Algebra is a foundational concept in database management systems, serving as a procedural query language that specifies both what data to retrieve and how to retrieve it. It forms the theoretical foundation for SQL and is considered a cornerstone for understanding database concepts, design, and querying. This mathematical basis is one of the key reasons for the popularity of the relational model.
In relational algebra, operations deal with relations (tables) as inputs and produce new relations as outputs. The process involves three main components: input (one or more relations), output (always exactly one relation), and operators.
Types of Operators
Relational algebra operators are categorized into two main types: Fundamental and Derived. Derived operators are built upon the fundamental ones.
Fundamental Operators
Selection ($\sigma$):
Purpose: Used for horizontal selection, meaning it selects rows (tuples) from a relation based on a specified condition (predicate).
Nature: It is a unary operator, taking one relation as input and producing one relation as output.
Syntax: $\sigma_{condition}(Relation)$.
Effect on Schema: The degree (number of columns) of the output relation is equal to the degree of the input relation, as only rows are filtered.
Effect on Data: The cardinality (number of rows) of the output relation will be less than or equal to the cardinality of the input relation.
Properties: Selection is commutative, meaning the order of applying multiple selection conditions does not change the result. Multiple conditions can also be combined using logical AND ($\land$) operators.
Null Handling: Null values are ignored in the selection operator if the condition involving them evaluates to null or false. Only tuples that return true for the condition are included.
Projection ($\pi$):
Purpose: Used for vertical selection, meaning it selects columns (attributes) from a relation.
Nature: It is a unary operator, taking one relation as input and producing one relation as output.
Effect on Schema: The degree (number of columns) of the output relation is less than or equal to the degree of the input relation, as only specified columns are projected.
Effect on Data: Projection eliminates duplicates in the resulting rows. Therefore, the cardinality of the output relation may be less than or equal to the cardinality of the input relation.
Properties: Projection is not swappable with selection if the selection condition relies on an attribute that would be removed by projection.
Null Handling: Null values are not ignored in projection; they are returned as part of the projected column.
Union ($\cup$):
Purpose: Combines all unique tuples from two compatible relations.
Compatibility: Both relations must be union compatible, meaning they have the same degree (number of columns) and corresponding columns have same domains (data types). Column names can be different.
Properties: Union is commutative ($A \cup B = B \cup A$) and associative ($A \cup (B \cup C) = (A \cup B) \cup C$).
Effect on Schema: The degree remains the same as the input relations.
Effect on Data: Eliminates duplicates by default. The cardinality of the result is $Cardinality(R1) + Cardinality(R2)$ minus the number of common tuples.
Null Handling: Null values are not ignored; they are treated just like other values.
Set Difference ($-$):
Purpose: Returns all tuples that are present in the first relation but not in the second relation. ($A – B$) includes elements in A but not in B.
Compatibility: Relations must be union compatible.
Properties: Set difference is neither commutative ($A – B \neq B – A$) nor associative.
Effect on Schema: The degree remains the same as the input relations.
Effect on Data: The cardinality of the result ranges from 0 (if R1 is a subset of R2) to $Cardinality(R1)$ (if R1 and R2 are disjoint).
Null Handling: Null values are not ignored.
Cartesian Product ($\times$):
Purpose: Combines every tuple from the first relation with every tuple from the second relation, resulting in all possible tuple combinations.
Syntax: $R1 \times R2$.
Effect on Schema: The degree of the result is the sum of the degrees of the input relations ($Degree(R1) + Degree(R2)$). If columns have the same name, a qualifier (e.g., TableName.ColumnName) is used to differentiate them.
Effect on Data: The cardinality of the result is the product of the cardinalities of the input relations ($Cardinality(R1) \times Cardinality(R2)$).
Use Case: Often used as a preliminary step before applying a selection condition to filter for meaningful combinations, effectively performing a “join”.
Renaming ($\rho$):
Purpose: Used to rename a relation or its attributes. This is useful for self-joins or providing more descriptive names.
Syntax: $\rho_{NewName}(Relation)$ or $\rho_{NewName(NewCol1, NewCol2, …)}(Relation)$.
Derived Operators
Derived operators can be expressed using combinations of fundamental operators.
Intersection ($\cap$):
Purpose: Returns tuples that are common to both union-compatible relations.
Derivation: Can be derived using set difference: $R1 \cap R2 = R1 – (R1 – R2)$.
Compatibility: Relations must be union compatible.
Effect on Schema: The degree remains the same.
Effect on Data: The cardinality of the result ranges from 0 to the minimum of the cardinalities of the input relations.
Null Handling: Null values are not ignored.
Join (Various Types): Joins combine tuples from two relations based on a common condition. They are derived from Cartesian product and selection.
Theta Join ($\Join_{\theta}$): Performs a Cartesian product followed by a selection based on any comparison condition ($\theta$) (e.g., greater than, less than, equals).
Syntax: $R1 \Join_{condition} R2$.
Effect on Schema: Sum of degrees.
Effect on Data: Ranges from 0 to Cartesian product cardinality.
Equijoin ($\Join_{=}$): A special case of Theta Join where the condition is restricted to equality ($=$).
Natural Join ($\Join$):
Purpose: Equijoins relations on all common attributes, automatically. The common attributes appear only once in the result schema.
Properties: Natural join is commutative and associative.
Effect on Schema: Degree is sum of degrees minus the count of common attributes.
Effect on Data: Cardinality ranges from 0 to the maximum (can be Cartesian product if no common attributes or if all common attributes have same values across all tuples). Tuples that fail to find a match are called dangling tuples.
Semi-Join ($\ltimes$):
Purpose: Performs a natural join but keeps only the attributes of the left-hand side relation. It effectively filters the left relation to only include tuples that have a match in the right relation.
Anti-Join ($\rhd$):
Purpose: Performs a natural join but keeps only the attributes of the left-hand side relation for tuples that do not have a match in the right relation [This is an external clarification, source says “keep the attributes of right hand side relation only” for anti-join, which contradicts the common definition of anti-join]. Correction based on source direct statement: “we have to keep the attributes of right hand side relation only” for anti-join. This is a bit unusual compared to standard anti-join (which typically returns tuples from the left that don’t have a match on the right, retaining left attributes). However, sticking to the provided source:
Purpose (per source): “keep the attributes of right hand side relation only”.
Effect: It implies a filtering operation, but the source’s description for anti-join might be a specific interpretation or a typo compared to conventional anti-join. I’ll highlight the source’s wording.
Outer Join (Left, Right, Full):
Purpose: Similar to inner joins, but they also include non-matching (dangling) tuples from one or both relations, padding missing attribute values with null.
Left Outer Join ($\Join^{L}$): Includes all matching tuples and all dangling tuples from the left relation.
Right Outer Join ($\Join^{R}$): Includes all matching tuples and all dangling tuples from the right relation.
Full Outer Join ($\Join^{F}$): Includes all matching tuples and dangling tuples from both left and right relations.
Effect on Data: Cardinality of Left Outer Join is at least $Cardinality(R1)$. Cardinality of Right Outer Join is at least $Cardinality(R2)$. Cardinality of Full Outer Join is at least $Cardinality(R1 \cup R2)$ (if treating attributes as sets).
Null Handling: Nulls are explicitly used to represent missing values for non-matching tuples.
Division ($\div$):
Purpose: Finds tuples in one relation that are “associated with” or “match all” tuples in another relation based on a subset of attributes. Often used for “for all” type queries.
Prerequisite: $R1 \div R2$ is only possible if all attributes of $R2$ are present in $R1$, and $R1$ has some extra attributes not present in $R2$.
Effect on Schema: The degree of the result is $Degree(R1) – Degree(R2)$ because attributes of $R2$ are removed from $R1$ in the output.
Derivation: Division is a derived operator and can be expressed using projection, Cartesian product, and set difference.
Relationship with Relational Calculus and SQL
Relational Algebra is a procedural language, telling the system how to do the retrieval, in contrast to Relational Calculus (Tuple Relational Calculus and Domain Relational Calculus), which are non-procedural and only specify what to retrieve. Relational algebra has the same expressive power as safe relational calculus. This means any query expressible in relational algebra can also be written in safe relational calculus, and vice versa. However, relational calculus (in its full, unsafe form) can express queries that cannot be expressed in relational algebra or SQL.
SQL’s SELECT, FROM, and WHERE clauses directly map to relational algebra’s Projection, Cartesian Product, and Selection operators, respectively. SQL is considered relationally complete, meaning any query expressible in relational algebra can also be written in SQL.
Key Concepts in Relational Algebra
Relation vs. Table: A relation is a mathematical set, a subset of a Cartesian product, containing only tuples that satisfy a given condition. A table is the practical form of a relation used in DBMS for storing data of interest. In tables, null and duplicate values are allowed for individual columns, but a whole tuple in a relation (mathematical sense) cannot be duplicated.
Degree and Cardinality: Degree refers to the number of columns (attributes) in a relation, while cardinality refers to the number of rows (tuples/records).
Null Values: In relational algebra, null signifies an unknown, non-applicable, or non-existing value. It is not treated as zero, empty string, or any specific value. Comparisons involving null (e.g., null > 5, null = null) typically result in null (unknown). This behavior impacts how selection and join operations handle tuples containing nulls, as conditions involving nulls usually do not evaluate to true. Projection, Union, Set Difference, and Intersection, however, do not ignore nulls.
Efficiency: When writing complex queries involving Cartesian products, it is generally more efficient to minimize the number of tuples in relations before performing the Cartesian product, as this reduces the size of the intermediate result. This principle is often applied by performing selections (filtering) early.
Relational Calculus: Principles, Types, and Applications
Relational Calculus is a non-procedural query language used in database management systems. Unlike procedural languages such as Relational Algebra, it specifies “what data to retrieve” rather than “how to retrieve” it. This means it focuses on describing the desired result set without outlining the step-by-step process for obtaining it.
Comparison with Relational Algebra and SQL
Relational Algebra (Procedural): Relational Algebra is considered a procedural language because it answers both “what to do” and “how to do” when querying a database.
Expressive Power:
Safe Relational Calculus has the same expressive power as Relational Algebra. This means any query that can be formulated in safe Relational Calculus can also be expressed in Relational Algebra, and vice versa.
However, Relational Calculus, in its entirety, has more expressive power than Relational Algebra or SQL. This additional power allows it to express “unsafe queries” – queries whose results include tuples that are not actually present in the database table.
Consequently, every query expressible in Relational Algebra or SQL can be represented using Relational Calculus, but there exist some queries in Relational Calculus that cannot be expressed using Relational Algebra.
Theoretical Foundation: SQL is theoretically based on both Relational Algebra and Relational Calculus.
Types of Relational Calculus
Relational Calculus is divided into two main parts:
Tuple Relational Calculus (TRC)
Domain Relational Calculus (DRC)
Tuple Relational Calculus (TRC)
Tuple Relational Calculus uses tuple variables to represent an entire row or record within a table.
Representation: A TRC query is typically represented as S = {T | P(T)}, where S is the result set, T is a tuple variable, and P is a condition (or predicate) that T must satisfy. The tuple variable T iterates through each tuple, and if the condition P(T) is true, that tuple is included in the result.
Attribute Access: Attributes of a tuple T are denoted using dot notation (T.A) or bracket notation (T[A]), where A is the attribute name.
Relation Membership: T belonging to a relation R is represented as T ∈ R or R(T).
Quantifiers in TRC: TRC employs logical quantifiers to express conditions:
Existential Quantifier (∃): Denoted by ∃ (read as “there exists”).
It asserts that there is at least one tuple that satisfies a given condition.
Unsafe Queries: Using the existential quantifier with an OR operator can produce unsafe queries. An unsafe query can include tuples in the result that are not actually present in the source table. For example, a query like T | ∃B (B ∈ Book ∧ (T.BookID = B.BookID ∨ T.Year = B.Year)) (where Book is a table) might include arbitrary combinations of BookID and Year that aren’t real entries if either part of the OR condition is met.
The EXISTS keyword in SQL is conceptually derived from this quantifier, returning true if a subquery produces a non-empty result.
Universal Quantifier (∀): Denoted by ∀ (read as “for all”).
It asserts that a condition must hold true for every tuple in a specified set.
Using ∀ with an AND operator can be meaningless for direct output projection.
It is often used in combination with negation (¬) or implication (→) to express queries like “find departments that do not have any girl students”.
Examples in TRC (from sources):
Projection:
To project all attributes of the Employee table: {T | Employee(T)}.
To project specific attributes (e.g., EName, Salary) of the Employee table: {T.EName, T.Salary | Employee(T)}.
Selection:
Find details of employees with Salary > 5000: {T | Employee(T) ∧ T.Salary > 5000}.
Find Date_of_Birth and Address of employees named “Rohit Sharma”: {T.DOB, T.Address | Employee(T) ∧ T.FirstName = ‘Rohit’ ∧ T.LastName = ‘Sharma’}.
Join (referencing multiple tables):
Find names of female students in the “Maths” department: {S.Name | Student(S) ∧ S.Sex = ‘Female’ ∧ ∃D (Department(D) ∧ D.DeptID = S.DeptNo ∧ D.DeptName = ‘Maths’)}.
Find BookID of all books issued to “Makash”: {T.BookID | ∃U (User(U) ∧ U.Name = ‘Makash’) ∧ ∃B (Borrow(B) ∧ B.CardNo = U.CardNo ∧ T.BookID = B.BookID)}.
Domain Relational Calculus (DRC)
Domain Relational Calculus uses domain variables that represent individual column attributes, rather than entire rows.
Representation: A DRC query is typically represented as Output_Table = {A1, A2, …, An | P(A1, A2, …, An)}, where A1, A2, …, An are the column attributes (domain variables) to be projected, and P is the condition they must satisfy.
Concept: Instead of iterating through tuples, DRC defines the domains of the attributes being sought.
Examples in DRC (from sources):
Projection:
Find BookID and Title of all books: {BookID, Title | (BookID, Title) ∈ Book}.
Selection:
Find BookID of all “DBMS” books: {BookID | (BookID, Title) ∈ Book ∧ Title = ‘DBMS’}.
Join:
Find title of all books supplied by “Habib”: {Title | ∃BookID, ∃SName ((BookID, Title) ∈ Book ∧ (BookID, SName) ∈ Supplier ∧ SName = ‘Habib’)}.
Safety of Queries
As mentioned, Relational Calculus can express unsafe queries. An unsafe query is one that, when executed, might include results that are not derived from the existing data in the database, potentially leading to an infinite set of results. For instance, a query to “include all those tuples which are not present in the table book” would be unsafe because there are infinitely many tuples not in a finite table.
SQL: Relational Database Querying and Manipulation
SQL (Structured Query Language) queries are the primary means of interacting with and manipulating data in relational database management systems (RDBMS). SQL is a non-procedural language, meaning it specifies what data to retrieve or modify rather than how to do it. This design allows the RDBMS to manage the efficient retrieval of data.
The theoretical foundation of SQL is based on both Relational Algebra (a procedural language) and Relational Calculus (a non-procedural language). SQL is considered a fourth-generation language, making it closer to natural language compared to third-generation languages like C++.
Core Components of SQL Queries
At its most basic level, an SQL query consists of three mandatory keywords for data retrieval: SELECT, FROM, and WHERE.
SELECT Clause:
Corresponds conceptually to the projection operator in Relational Algebra.
By default, SELECT retains duplicate values (projection with duplicacy).
To obtain distinct (unique) values, the DISTINCT keyword must be explicitly used (e.g., SELECT DISTINCT Title FROM Book).
If the default setting is changed to DISTINCT, ALL can be used to explicitly retain duplicates (e.g., SELECT ALL Title FROM Book).
Attributes or columns to be displayed are listed here.
FROM Clause:
Specifies the tables from which data is to be retrieved.
Conceptually, listing multiple tables in the FROM clause (e.g., FROM User, Borrow) implies a Cartesian Product between them.
The FROM clause is mandatory for data retrieval.
Tables can be renamed using the AS keyword (e.g., User AS U1), which is optional for tables but mandatory for renaming attributes.
WHERE Clause:
Used to specify conditions that rows must satisfy to be included in the result.
Corresponds to the selection operator in Relational Algebra (horizontal row selection).
The WHERE clause is optional; if omitted, all rows from the specified tables are returned.
SQL queries can become complex using various clauses and operators:
Set Operations:
UNION: Combines the result sets of two or more SELECT statements. By default, UNION eliminates duplicate rows.
UNION ALL: Combines results and retains duplicate rows.
INTERSECT: Returns only the rows that are common to both result sets. By default, INTERSECT eliminates duplicates.
EXCEPT (or MINUS): Returns rows from the first query that are not present in the second. By default, EXCEPT eliminates duplicates.
For all set operations, the participating queries must be union compatible, meaning they have the same number of columns and compatible data types in corresponding columns.
Aggregate Functions:
Used to perform calculations on a set of rows and return a single summary value. Common functions include:
COUNT(): Counts the number of rows or non-null values in a column. COUNT(*) counts all rows, including those with nulls.
SUM(): Calculates the total sum of a numeric column.
AVG(): Calculates the average value of a numeric column.
MIN(): Returns the minimum value in a column.
MAX(): Returns the maximum value in a column.
All aggregate functions ignore null values, except for COUNT(*).
GROUP BY Clause:
Used to logically break a table into groups based on the values in one or more columns.
Aggregate functions are then applied to each group independently.
All attributes in the SELECT clause that are not part of an aggregate function must also be included in the GROUP BY clause.
Any attribute not in GROUP BY that needs to be displayed in the SELECT clause must appear inside an aggregate function.
HAVING Clause:
Used to filter groups created by the GROUP BY clause.
Similar to WHERE, but HAVING operates on groups after aggregation, while WHERE filters individual rows before aggregation.
Aggregate functions can be used directly in the HAVING clause (e.g., HAVING COUNT(*) > 50), which is not allowed in WHERE.
Subqueries (Nested Queries):
A query embedded within another SQL query.
Used with operators like IN, NOT IN, SOME/ANY, ALL, EXISTS, NOT EXISTS.
IN: Returns true if a value matches any value in a list or the result of a subquery.
SOME/ANY: Returns true if a comparison is true for any value in the subquery result (e.g., price > SOME (subquery) finds prices greater than at least one price in the subquery).
ALL: Returns true if a comparison is true for all values in the subquery result (e.g., price > ALL (subquery) finds prices greater than the maximum price in the subquery).
EXISTS: Returns true if the subquery returns at least one row (is non-empty). It’s typically used to check for the existence of related rows.
NOT EXISTS: Returns true if the subquery returns no rows (is empty).
UNIQUE: Returns true if the subquery returns no duplicate rows.
ORDER BY Clause:
Used to sort the result set of a query.
Sorting can be in ASC (ascending, default) or DESC (descending) order.
When sorting by multiple attributes, the first attribute listed is the primary sorting key, and subsequent attributes are secondary keys for tie-breaking within primary groups.
Sorting is always done tuple-wise, not column-wise, to avoid creating invalid data.
JOIN Operations:
Used to combine rows from two or more tables based on a related column between them.
INNER JOIN: Returns only the rows where there is a match in both tables. Can be specified with ON (any condition) or USING (specific common columns). INNER keyword is optional.
THETA JOIN: An inner join with an arbitrary condition (e.g., R1.C > R2.D).
EQUI JOIN: A theta join where the condition is solely an equality (=).
NATURAL JOIN: An equi join that automatically joins tables on all columns with the same name and data type, and eliminates duplicate common columns in the result.
OUTER JOIN: Includes matching rows and non-matching rows from one or both tables, filling non-matches with NULL values.
LEFT OUTER JOIN: Includes all rows from the left table and matching rows from the right table.
RIGHT OUTER JOIN: Includes all rows from the right table and matching rows from the left table.
FULL OUTER JOIN: Includes all rows from both tables, with NULL where there’s no match.
Database Modification Queries
SQL provides commands to modify the data stored in tables:
INSERT:
Adds new rows (tuples) to a table.
Syntax includes INSERT INTO table_name VALUES (value1, value2, …) or INSERT INTO table_name (column1, column2, …) VALUES (value1, value2, …).
DELETE:
Removes one or more rows from a table.
Syntax is DELETE FROM table_name [WHERE condition].
If no WHERE clause is specified, all rows are deleted.
TRUNCATE TABLE: A DDL command that quickly removes all rows from a table, similar to DELETE without a WHERE clause, but it is faster as it deletes the whole table in one go (rather than tuple by tuple) and resets identity columns. TRUNCATE cannot use a WHERE clause.
UPDATE:
Modifies existing data within a row (cell by cell).
Syntax is UPDATE table_name SET column1 = value1, … [WHERE condition].
Other Important Concepts Related to Queries
Views (Virtual Tables):
A virtual table based on the result-set of an SQL query.
Views are not physically stored in the database (dynamic views); instead, their definition is stored, and the view is evaluated when queried.
Views are primarily used for security (data hiding) and simplifying complex queries.
Views can be updatable (allowing INSERT, UPDATE, DELETE on the view, which affects the base tables) or read-only (typically for complex views involving joins or aggregates).
Materialized Views are physical copies of a view’s data, stored to improve performance for frequent queries.
NULL Values:
NULL represents unknown, non-existent, or non-applicable values.
NULL is not comparable to any value, including itself (e.g., SID = NULL will not work).
Comparison with NULL is done using IS NULL or IS NOT NULL.
NULL values are ignored by aggregate functions (except COUNT(*)).
In ORDER BY, NULL values are treated as the lowest value by default.
In GROUP BY, all NULL values are treated as equal and form a single group.
Pattern Matching (LIKE):
Used for string matching in WHERE clauses.
% (percentage sign): Matches any sequence of zero or more characters.
_ (underscore): Matches exactly one character.
The ESCAPE keyword can be used to search for the literal % or _ characters.
DDL Commands (Data Definition Language):
While not strictly queries that retrieve data, DDL commands define and manage the database schema.
CREATE TABLE: Defines a new table, including column names, data types, and constraints (like PRIMARY KEY, NOT NULL, FOREIGN KEY, DEFAULT).
ALTER TABLE: Modifies an existing table’s structure (e.g., adding/dropping columns, changing data types, adding/deleting constraints).
DROP TABLE: Deletes an entire table, including its data and schema.
DCL Commands (Data Control Language):
Manage permissions and access control for database users.
GRANT: Assigns specific privileges (e.g., SELECT, INSERT, UPDATE, DELETE) on database objects to users or roles.
REVOKE: Removes previously granted privileges.
SQL: Data Modification, Definition, and Control
SQL (Structured Query Language) provides powerful commands for modifying data stored in relational database management systems (RDBMS). These modifications are distinct from data retrieval queries (like SELECT) and fall under various categories within SQL, primarily Data Manipulation Language (DML) for data content changes and Data Definition Language (DDL) for schema structure changes.
Data Manipulation Commands (DML)
The core DML commands for modifying database content operate on a tuple-by-tuple or cell-by-cell basis.
Deletion (DELETE)
Purpose: DELETE is used to remove one or more rows (tuples) from a table.
Syntax: The basic syntax is DELETE FROM table_name [WHERE condition].
Conditional Deletion: If a WHERE clause is specified, only rows satisfying the condition are deleted. If omitted, all rows are deleted from the table.
Relational Algebra Equivalent: In relational algebra, deletion is represented using the set difference operator (R – E), where R is the original relation and E is a relational algebra expression whose output specifies the tuples to be removed. The resulting new relation is then assigned back to the original relation. This requires E to be union compatible with R (same degree and domain for corresponding attributes).
Example: To delete all entries from the borrow relation corresponding to card number 101, one would subtract a relation containing all tuples where card_number = 101 from the borrow relation.
Insertion (INSERT)
Purpose: INSERT is used to add new rows (tuples) to a table.
Syntax:
INSERT INTO table_name VALUES (value1, value2, …): Values must be in the order of the table’s columns.
INSERT INTO table_name (column1, column2, …) VALUES (value1, value2, …): Allows specifying columns, useful if not inserting values for all fields or if the order is not strictly followed.
Null Values: If not all fields are inserted, the remaining fields will by default be set to NULL.
Relational Algebra Equivalent: In relational algebra, insertion is performed using the union operator (R UNION E), where R is the original relation and E represents the tuples to be inserted. The new relation is then assigned to the old one. Union compatibility is also required here.
Example: To insert an entry into the book table with book_ID B101, year_of_publication 2025, and title A, you would use INSERT INTO book VALUES (‘B101’, ‘A’, 2025) or INSERT INTO book (book_ID, title, year_of_publication) VALUES (‘B101’, ‘A’, 2025).
Update (UPDATE)
Purpose: UPDATE is used to modify existing data within rows. Unlike INSERT and DELETE which work tuple-by-tuple, UPDATE works cell-by-cell.
Conditional Updates: The WHERE clause specifies which rows to update.
Calculations: The SET clause can include calculations (e.g., applying a discount).
Relational Algebra Equivalent: Conceptually, updating a single cell in relational algebra involves deleting the old tuple and inserting a new tuple with the modified value, while retaining other values.
Example: To give a 5% discount on all books supplied by ABC having a price greater than 1,000, you would UPDATE supplier SET price = 0.95 * price WHERE s_name = ‘ABC’ AND price > 1000.
Schema Modification Commands (DDL)
DDL commands are used to define and modify the database schema (structure).
TRUNCATE TABLE
Purpose: TRUNCATE TABLE is a DDL command that removes all rows from a table.
Key Differences from DELETE:
Speed: TRUNCATE is faster than DELETE because it deletes the whole table in one go, rather than row by row.
WHERE Clause: TRUNCATE cannot use a WHERE clause; it always removes all rows.
Logging/Transactions: TRUNCATE typically involves less logging and cannot be rolled back easily in some systems, while DELETE (being DML) is part of transactions and can be rolled back.
Identity Columns: TRUNCATE often resets identity columns (auto-incrementing IDs).
DDL vs. DML: TRUNCATE is DDL, DELETE is DML.
Schema Preservation: Both DELETE (without WHERE) and TRUNCATE preserve the table’s schema (structure).
DROP TABLE
Purpose: DROP TABLE deletes an entire table, including its data and schema (structure). This is a more permanent and impactful operation compared to DELETE or TRUNCATE.
ALTER TABLE
Purpose: ALTER TABLE is used to modify the structure of an existing table.
Common Operations:
Adding/Dropping Columns: You can add new columns with ADD COLUMN column_name data_type or remove existing ones with DROP COLUMN column_name.
Modifying Columns: Change the data type or properties of an existing column with MODIFY COLUMN column_name new_data_type.
Adding/Dropping Constraints: Constraints (like PRIMARY KEY, FOREIGN KEY, NOT NULL) can be added or removed. Naming constraints with the CONSTRAINT keyword allows for easier modification or deletion later.
Infrequent Use: Schema changes are rarely done frequently because they can affect numerous existing tuples and related application programs.
RESTRICT vs. CASCADE with DROP COLUMN:
RESTRICT: If a column being dropped is referenced by another table (e.g., as a foreign key), RESTRICT will prevent the deletion.
CASCADE: If a column being dropped is referenced, CASCADE will force the deletion and also delete the referencing constraints or even the dependent tables/relations.
Data Control Language (DCL)
DCL commands manage permissions and access control for database users.
GRANT
Purpose: GRANT is used to assign specific privileges on database objects (like tables, views) to users or roles.
Common Privileges:
SELECT: Allows users to retrieve data.
INSERT: Allows users to add new data.
UPDATE: Allows users to modify existing data.
DELETE: Allows users to remove data.
REFERENCES: Allows users to create foreign key relationships referencing the object.
ALL PRIVILEGES: Grants all available permissions.
Syntax: GRANT privilege_name ON object_name TO username.
Example: GRANT INSERT, UPDATE ON student TO Gora gives Gora permission to insert and update data in the student table.
REVOKE
Purpose: REVOKE is used to remove previously granted privileges from users or roles.
Syntax: REVOKE privilege_name ON object_name FROM username.
Example: REVOKE DELETE ON student FROM Gora removes the delete privilege from Gora on the student table.
GRANT and REVOKE are crucial for database security and controlling who can perform specific actions with the data. Views, which are virtual tables, are often used in conjunction with DCL for security, as permissions can be granted on a view rather than directly on the underlying base tables, allowing for data hiding and simplified interaction.
Affiliate Disclosure: This blog may contain affiliate links, which means I may earn a small commission if you click on the link and make a purchase. This comes at no additional cost to you. I only recommend products or services that I believe will add value to my readers. Your support helps keep this blog running and allows me to continue providing you with quality content. Thank you for your support!
This text provides a comprehensive introduction to data science, covering its growth, career opportunities, and required skills. It explores various data science tools, programming languages (like Python and R), and techniques such as machine learning and deep learning. The materials also explain how to work with different data types, perform data analysis, build predictive models, and present findings effectively. Finally, it examines the role of generative AI in enhancing data science workflows.
Python & Data Science Study Guide
Quiz
What is the purpose of markdown cells in Jupyter Notebooks, and how do you create one?
Markdown cells allow you to add titles and descriptive text to your notebook. You can create a markdown cell by clicking ‘Code’ in the toolbar and selecting ‘Markdown.’
Explain the difference between int, float, and string data types in Python and provide an example of each.
int represents integers (e.g., 5), float represents real numbers (e.g., 3.14), and string represents sequences of characters (e.g., “hello”).
What is type casting in Python, and why is it important to be careful when casting a float to an integer?
Type casting is changing the data type of an expression (e.g., converting a string to an integer). When converting a float to an int, information after the decimal point is lost, so you must be careful.
Describe the role of variables in Python and how you assign values to them.
Variables store values in memory, and you assign a value to a variable using the assignment operator (=). For example, x = 10 assigns 10 to the variable x.
What is the purpose of indexing and slicing in Python strings and give an example.
Indexing allows you to access individual characters in a string using their position (e.g., string[0]). Slicing allows you to extract a substring (e.g., string[1:4]).
Explain the concept of immutability in the context of strings and tuples and how it affects their manipulation.
Immutable data types cannot be modified after creation. If you want to change a string or a tuple you create a new string or tuple.
What are the key differences between lists and tuples in Python?
Lists are mutable, meaning you can change them after creation; tuples are immutable. Lists are defined using square brackets [], while tuples use parentheses ().
Describe dictionaries in Python and how they are used to store data using keys and values.
Dictionaries store key-value pairs, where keys are unique and immutable and the values are the associated information. You use curly brackets {} and each key and value are separated by a colon (e.g., {“name”: “John”, “age”: 30}).
What are sets in Python, and how do they differ from lists or tuples?
Sets are unordered collections of unique elements. They do not keep track of order, and only contain a single instance of any item.
Explain the difference between a for loop and a while loop and how each can be used.
A for loop is used to iterate over a sequence of elements, like a list or string. A while loop runs as long as a certain condition is true, and does not necessarily require iterating over a sequence.
Quiz Answer Key
Markdown cells allow you to add titles and descriptive text to your notebook. You can create a markdown cell by clicking ‘Code’ in the toolbar and selecting ‘Markdown.’
int represents integers (e.g., 5), float represents real numbers (e.g., 3.14), and string represents sequences of characters (e.g., “hello”).
Type casting is changing the data type of an expression (e.g., converting a string to an integer). When converting a float to an int, information after the decimal point is lost, so you must be careful.
Variables store values in memory, and you assign a value to a variable using the assignment operator (=). For example, x = 10 assigns 10 to the variable x.
Indexing allows you to access individual characters in a string using their position (e.g., string[0]). Slicing allows you to extract a substring (e.g., string[1:4]).
Immutable data types cannot be modified after creation. If you want to change a string or a tuple you create a new string or tuple.
Lists are mutable, meaning you can change them after creation; tuples are immutable. Lists are defined using square brackets [], while tuples use parentheses ().
Dictionaries store key-value pairs, where keys are unique and immutable and the values are the associated information. You use curly brackets {} and each key and value are separated by a colon (e.g., {“name”: “John”, “age”: 30}).
Sets are unordered collections of unique elements. They do not keep track of order, and only contain a single instance of any item.
A for loop is used to iterate over a sequence of elements, like a list or string. A while loop runs as long as a certain condition is true, and does not necessarily require iterating over a sequence.
Essay Questions
Discuss the role and importance of data types in Python, elaborating on how different types influence operations and the potential pitfalls of incorrect type handling.
Compare and contrast the use of lists, tuples, dictionaries, and sets in Python. In what scenarios is each of these data structures more beneficial?
Describe the concept of functions in Python, providing examples of both built-in functions and user-defined functions, and explaining how they can improve code organization and reusability.
Analyze the use of loops and conditions in Python, explaining how they allow for iterative processing and decision-making, and discuss their relevance in data manipulation.
Explain the differences and relationships between object-oriented programming concepts (such as classes, objects, methods, and attributes) and how those translate into more complex data structures and functional operations.
Glossary
Boolean: A data type that can have one of two values: True or False.
Class: A blueprint for creating objects, defining their attributes and methods.
Data Frame: A two-dimensional data structure in pandas, similar to a table with rows and columns.
Data Type: A classification that specifies which type of value a variable has, such as integer, float, string, etc.
Dictionary: A data structure that stores data as key-value pairs, where keys are unique and immutable.
Expression: A combination of values, variables, and operators that the computer evaluates to a single value.
Float: A data type representing real numbers with decimal points.
For Loop: A control flow statement that iterates over a sequence (e.g., list, tuple) and executes code for each element.
Function: A block of reusable code that performs a specific task.
Index: Position in a sequence, string, list, or tuple.
Integer (Int): A data type representing whole numbers, positive or negative.
Jupyter Notebook: An interactive web-based environment for coding, data analysis, and visualization.
Kernel: A program that runs code in a Jupyter Notebook.
List: A mutable, ordered sequence of elements defined with square brackets [].
Logistic Regression: A classification algorithm that predicts the probability of an instance belonging to a class.
Method: A function associated with an object of a class.
NumPy: A Python library for numerical computations, especially with arrays and matrices.
Object: An instance of a class, containing its own data and methods.
Operator: Symbols that perform operations such as addition, subtraction, multiplication, or division.
Pandas: A Python library for data manipulation and analysis.
Primary Key: A unique identifier for each record in a table.
Relational Database: A database that stores data in tables with rows and columns and structured relationships between tables.
Set: A data structure that is unordered and contains only unique values.
Sigmoid Function: A mathematical function used in logistic regression that outputs a value between zero and one.
Slicing: Extracting a portion of a sequence (e.g., list, string) using indexes (e.g., [start:end:step]).
SQL (Structured Query Language): Language used to manage and manipulate data in relational databases.
String: A sequence of characters, defined with single or double quotes.
Support Vector Machine (SVM): A classification algorithm that finds an optimal hyperplane to separate data classes.
Tuple: An immutable, ordered sequence of elements defined with parentheses ().
Type Casting: Changing the data type of an expression.
Variable: A named storage location in a computer’s memory used to hold a value.
View: A virtual table based on the result of an SQL query.
While Loop: A control flow statement that repeatedly executes a block of code as long as a condition remains true.
Python for Data Science
Okay, here’s a detailed briefing document summarizing the provided sources, focusing on key themes and ideas, with supporting quotes:
Briefing Document: Python Fundamentals and Data Science Tools
I. Overview
This document provides a summary of core concepts in Python programming, specifically focusing on those relevant to data science. It covers topics from basic syntax and data types to more advanced topics like object-oriented programming, file handling, and fundamental data analysis libraries. The goal is to equip a beginner with a foundational understanding of Python for data manipulation and analysis.
II. Key Themes and Ideas
Jupyter Notebook Environment: The sources emphasize the practical use of Jupyter notebooks for coding, analysis, and presentation. Key functionalities include running code cells, adding markdown for explanations, and creating slides for presentation.
“you can now start working on your new notebook… you can create a markdown to add titles and text descriptions to help with the flow of the presentation… the slides functionality in Jupiter allows you to deliver code visualization text and outputs of the executed code as part of a project”
Python Data Types: The document systematically covers fundamental Python data types, including:
Integers (int) & Floats (float):“you can have different types in Python they can be integers like 11 real numbers like 21.23%… we can have int which stands for an integer and float that stands for float essentially a real number”
Strings (str):“the type string is a sequence of characters” Strings are explained to be immutable, accessible by index, and support various methods.
Booleans (bool):“A Boolean can take on two values the first value is true… Boolean values can also be false”
Type Casting: The sources teach how to change one data type to another. “You can change the type of the expression in Python this is called type casting… you can convert an INT to a float for example”
Expressions and Variables: These sections explain basic operations and variable assignment:
Expressions:“Expressions describe a type of operation the computers perform… for example basic arithmetic operations like adding multiple numbers” The order of operations is also covered.
Variables: Variables are used to “store values” and can be reassigned, and they benefit from meaningful naming.
Compound Data Types (Lists, Tuples, Dictionaries, Sets):
Tuples: Ordered, immutable sequences using parenthesis. “tuples are an ordered sequence… tupples are expressed as comma separated elements within parentheses”
Lists: Ordered, mutable sequences using square brackets. “lists are also an ordered sequence… a list is represented with square brackets” Lists support methods like extend, append, and del.
Dictionaries: Collection with key-value pairs. Keys must be immutable and unique. “a dictionary has keys and values… the keys are the first elements they must be immutable and unique each each key is followed by a value separated by a colon”
Sets: Unordered collections of unique elements. “sets are a type of collection… they are unordered… sets only have unique elements” Set operations like add, remove, intersection, union, and subset checking are covered.
Control Flow (Conditions & Loops):
Conditional Statements (if, elif, else):“The if statement allows you to make a decision based on some condition… if that condition is true the set of statements within the if block are executed”
For Loops: Used for iterating over a sequence.“The for Loop statement allows you to execute a statement or set of statements a certain number of times”
While Loops: Used for executing statements while a condition is true. “a while loop will only run if a condition is me”
Functions:
Built-in Functions: len(), sum(), sorted().
User-defined Functions: The syntax and best practices are covered, including documentation, parameters, return values, and scope of variables. “To define a function we start with the keyword def… the name of the function should be descriptive of what it does”
Object-Oriented Programming (OOP):
Classes & Objects:“A class can be thought of as a template or a blueprint for an object… An object is a realization or instantiation of that class” The concepts of attributes and methods are also introduced.
File Handling: The sources cover the use of Python’s open() function, modes for reading (‘r’) and writing (‘w’), and the importance of closing files.
“we use the open function… the first argument is the file path this is made up of the file name and the file directory the second parameter is the mode common values used include R for reading W for writing and a for appending” The use of the with statement is advocated for automatic file closing.
Libraries (Pandas & NumPy):
Pandas: Introduction to DataFrames, importing data (read_csv, read_excel), and operations like head(), selection of columns and rows (iloc, loc), and unique value discovery. “One Way pandas allows you to work with data is in a data frame” Data slicing and filtering are shown.
NumPy: Introduction to ND arrays, creation from lists, accessing elements, slicing, basic vector operations (addition, subtraction, multiplication), broadcasting and universal functions, and array attributes. “a numpy array or ND array is similar to a list… each element is of the same type”
SQL and Relational Databases: SQL is introduced as a way to interact with data in relational database systems using Data Definition Language (DDL) and Data Manipulation Language (DML). DDL statements like create table, alter table, drop table, and truncate are discussed, as well as DML statements like insert, select, update, and delete. Concepts like views and stored procedures are also covered, as well as accessing database table and column metadata.
“Data definition language or ddl statements are used to define change or drop database objects such as tables… data manipulation language or DML statements are used to read and modify data in tables”
Data Visualization, Correlation, and Statistical Methods:
Pivot Tables and Heat Maps: Techniques for reshaping data and visualizing patterns using pandas pivot() method and heatmaps. “by using the pandas pivot method we can pivot the body style variable so it is displayed along the columns and the drive wheels will be displayed along the rows”
Correlation: Introduction to the concept of correlation between variables, using scatter plots and regression lines to visualize relationships. “correlation is a statistical metric for measuring to what extent different variables are interdependent”
Pearson Correlation: A method to quantify the strength and direction of linear relationships, emphasizing both correlation coefficients and p-values. “Pearson correlation method will give you two values the correlation coefficient and the P value”
Chi-Square Test: A method to identify if there is a relationship between categorical variables. “The Ki Square test is intended to test How likely it is that an observed distribution is due to chance”
Model Development:
Linear Regression: Introduction to simple and multiple linear regression for predictive modeling with independent and dependent variables. “simple linear regression or SLR is a method to help us understand the relationship between two variables the predictor independent variable X and the target dependent variable y”
Polynomial Regression: Introduction to non linear regression models.
Model Evaluation Metrics: Introduction to evaluation metrics like R-squared (R2) and Mean Squared Error (MSE).
K-Nearest Neighbors (KNN): Classification algorithm based on similarity to other cases. K selection and distance computation are discussed. “the K near nearest neighbors algorithm is a classification algorithm that takes a bunch of labeled points and uses them to learn how to label other points”
Evaluation Metrics for Classifiers: Metrics such as the Jaccard index, F1 Score and log loss are introduced for assessing model performance.
“evaluation metrics explain the performance of a model… we can Define jackard as the size of the intersection divided by the size of the Union of two label sets”
Decision Trees: Algorithm for data classification by splitting attributes, recursive partitioning, impurity, entropy and information gain are discussed.
“decision trees are built using recursive partitioning to classify the data… the algorithm chooses the most predictive feature to split the data on”
Logistic Regression: Classification algorithm that uses a sigmoid function to calculate probabilities and gradient descent to tune model parameters.
“logistic regression is a statistical and machine learning technique for classifying records of a data set based on the values of the input Fields… in logistic regression we use one or more independent variables such as tenure age and income to predict an outcome such as churn”
Support Vector Machines: Classification algorithm based on transforming data to a high-dimensional space and finding a separating hyperplane. Kernel functions and support vectors are introduced.
“a support Vector machine is a supervised algorithm that can classify cases by finding a separator svm works by first mapping data to a high-dimensional feature space so that data points can be categorized even when the data are not otherwise linearly separable”
III. Conclusion
These sources lay a comprehensive foundation for understanding Python programming as it is used in data science. From setting up a development environment in Jupyter Notebooks to understanding fundamental data types, functions, and object-oriented programming, the document prepares learners for more advanced topics. Furthermore, the document introduces data analysis and visualization concepts, along with model building through regression techniques and classification algorithms, equipping beginners with practical data science tools. It is crucial to delve deeper into practical implementations, which are often available in the labs.
Python Programming Fundamentals and Machine Learning
Python & Jupyter Notebook
How do I start a new notebook and run code? To start a new notebook, click the plus symbol in the toolbar. Once you’ve created a notebook, type your code into a cell and click the “Run” button or use the shortcut Shift + Enter. To run multiple code cells, click “Run All Cells.”
How can I organize my notebook with titles and descriptions? To add titles and descriptions, use markdown cells. Select “Markdown” from the cell type dropdown, and you can write text, headings, lists, and more. This allows you to provide context and explain the code.
Can I use more than one notebook at a time? Yes, you can open and work with multiple notebooks simultaneously. Click the plus button on the toolbar, or go to File -> Open New Launcher or New Notebook. You can arrange the notebooks side-by-side to work with them together.
How do I present my work using notebooks? Jupyter Notebooks support creating presentations. Using markdown and code cells, you can create slides by selecting the View -> Cell Toolbar -> Slides option. You can then view the presentation using the Slides icon.
How do I shut down notebooks when I’m finished? Click the stop icon (second from top) in the sidebar, this releases memory being used by the notebook. You can terminate all sessions at once or individually. You will know it is successfully shut down when you see “No Kernel” on the top right.
Python Data Types, Expressions, and Variables
What are the main data types in Python and how can I change them? Python’s main data types include int (integers), float (real numbers), str (strings), and bool (booleans). You can change data types using type casting. For example, float(2) converts the integer 2 to a float 2.0, or int(2.9) will convert the float 2.9 to the integer 2. Casting a string like “123” to an integer is done with int(“123”) but will result in an error if the string has non-integer values. Booleans can be cast to integers where True is converted to 1, and False is converted to 0.
What are expressions and how are they evaluated? Expressions are operations that Python performs. These can include arithmetic operations like addition, subtraction, multiplication, division, and more. Python follows mathematical conventions when evaluating expressions, with parentheses having the highest precedence, followed by multiplication and division, then addition and subtraction.
How do I store values in variables and work with strings? You can store values in variables using the assignment operator =. You can then use the variable name in place of the value it stores. Variables can store results of expressions, and the type of the variable can be determined with the type() command. Strings are sequences of characters and are enclosed in single or double quotes, you can access individual elements using indexes and also perform operations like slicing, concatenation, and replication.
Python Data Structures: Lists, Tuples, Dictionaries, and Sets
What are lists and tuples, and how are they different? Lists and tuples are ordered sequences used to store data. Lists are mutable, meaning you can change, add, or remove elements. Tuples are immutable, meaning they cannot be changed once created. Lists are defined using square brackets [], and tuples are defined using parentheses ().
What are dictionaries and sets? Dictionaries are collections that store data in key-value pairs, where keys must be immutable and unique. Sets are collections of unique elements. Sets are unordered and therefore do not have indexes or ordered keys. You can perform various mathematical set operations such as union, intersection, adding and removing elements.
How do I work with nested collections and change or copy lists? You can nest lists and tuples inside other lists and tuples. Accessing elements in these structures uses the same indexing conventions. Because lists are mutable, when you assign one list variable to another variable both variables refer to the same list, therefore, changes to one list impact the other this is called aliasing. To copy a list and not reference the original, use [:] (e.g., new_list = old_list[:]) to create a new copy of the original.
Control Flow, Loops, and Functions
How do I use conditions and branching in Python? You can use if, elif, and else statements to perform different actions based on conditions. You use comparison operators (==, !=, <, >, <=, >=) which return True or False. Based on whether the condition is True, the corresponding code blocks are executed.
What is the difference between for and while loops? for loops are used for iterating over a sequence, like lists or tuples, executing a block of code for every item in that sequence. while loops repeatedly execute a block of code as long as a condition is True, you must make sure your condition will become False or it will loop forever.
What are functions and how do I create them? Functions are reusable blocks of code. They are defined with the def keyword followed by the function name, parentheses for parameters, and a colon. The function’s code block is indented. Functions can take inputs (parameters) and return values. Functions are documented in the first few lines using triple quotes.
What are variable scope and global/local variables? The scope of a variable is the part of the program where the variable is accessible. Variables defined outside of a function are global variables and are accessible everywhere. Variables defined inside a function are local variables and are only accessible within that function, there is no conflict if a local variable has the same name as a global one. If you would like to have a local variable update a global variable you can use the global keyword inside the function’s scope and assign the name of the global variable.
Object Oriented Programming, Files, and Libraries
What are classes and objects in Python? Classes are templates for creating objects. An object is a specific instance of a class. You can define classes with attributes (data) and methods (functions that operate on that data) using the class keyword, you can instantiate multiple objects of the same class.
How do I work with files in Python? You can use the open() function to create a file object, you use the first argument to specify the file path and the second for the mode (e.g., “r” for reading, “w” for writing, “a” for appending). Using the with statement is recommended, as it automatically closes the file after use. You can use methods like read(), readline(), and write() to interact with the file.
What is a library and how do I use Pandas for data analysis? Libraries are pre-written code that helps solve problems, like data analysis. You can import libraries using the import statement, often with a shortened name (as keyword). Pandas is a popular library for data analysis that uses data frames to store and analyze tabular data. You can load files like CSV or Excel into pandas data frames and use its tools for cleaning, modifying, and exploring data.
How can I work with numpy? Numpy is a library for numerical computing, it works with arrays. You can create Numpy arrays from Python lists, you can access and slice data using indexing and slicing. Numpy arrays support many mathematical operations which are usually much faster and require less memory than regular python lists.
Databases and SQL
What is SQL, a database, and a relational database? SQL (Structured Query Language) is a programming language used to manage data in a database. A database is an organized collection of data. A relational database stores data in tables with rows and columns, it uses SQL for its main operations.
What is an RDBMS and what are the basic SQL commands? RDBMS (Relational Database Management System) is a software tool used to manage relational databases. Basic SQL commands include CREATE TABLE, INSERT (to add data), SELECT (to retrieve data), UPDATE (to modify data), and DELETE (to remove data).
How do I retrieve data using the SELECT statement? You can use SELECT followed by column names to specify which columns to retrieve. SELECT * retrieves all columns from a table. You can add a WHERE clause followed by a predicate (a condition) to filter data using comparison operators (=, >, <, >=, <=, !=).
How do I use COUNT, DISTINCT, and LIMIT with select statements? COUNT() returns the number of rows that match a criteria. DISTINCT removes duplicate values from a result set. LIMIT restricts the number of rows returned.
How do I create and populate a table? You can create a table with the CREATE TABLE command. Provide the name of the table and, inside parentheses, define the name and data types for each column. Use the INSERT statement to populate tables using INSERT INTO table_name (column_1, column_2…) VALUES (value_1, value_2…).
More SQL
What are DDL and DML statements? DDL (Data Definition Language) statements are used to define database objects like tables (e.g., CREATE, ALTER, DROP, TRUNCATE). DML (Data Manipulation Language) statements are used to manage data in tables (e.g., INSERT, SELECT, UPDATE, DELETE).
How do I use ALTER, DROP, and TRUNCATE tables? ALTER TABLE is used to add, remove, or modify columns. DROP TABLE deletes a table. TRUNCATE TABLE removes all data from a table, but leaves the table structure.
How do I use views in SQL? A view is an alternative way of representing data that exists in one or more tables. Use CREATE VIEW followed by the view name, the column names and AS followed by a SELECT statement to define the data the view should display. Views are dynamic and do not store the data themselves.
What are stored procedures? A stored procedure is a set of SQL statements stored and executed on the database server. This avoids sending multiple SQL statements from the client to the server, they can accept input parameters, and return output values. You can define them with CREATE PROCEDURE.
Data Visualization and Analysis
What are pivot tables and heat maps, and how do they help with visualization? A pivot table is a way to summarize and reorganize data from a table and display it in a rectangular grid. A heat map is a graphical representation of a pivot table where data values are shown using a color intensity scale. These are effective ways to examine and visualize relationships between multiple variables.
How do I measure correlation between variables? Correlation measures the statistical interdependence of variables. You can use scatter plots to visualize the relationship between two numerical variables and add a linear regression line to show their trend. Pearson correlation measures the linear correlation between continuous numerical values, providing the correlation coefficient and P-value. Chi-square test is used to identify if an association between two categorical variables exists.
What is simple linear regression and multiple linear regression? Simple linear regression uses one independent variable to predict a dependent variable using a linear relationship, Multiple linear regression uses several independent variables to predict the dependent variable.
Model Development
What is a model and how can I use it for predictions? A model is a mathematical equation used to predict a value (dependent variable) given one or more other values (independent variables). Models are trained with data that determines parameters for an equation. Once the model is trained you can input data and have the model predict an output.
What are R-squared and MSSE, and how are they used to evaluate model performance? R-squared measures how well the model fits the data and it represents the percentage of the data that is closest to the fitted line and represents the “goodness of fit”. Mean squared error (MSE) is the average of the square difference between the predicted values and the true values. These scores are used to measure model performance for continuous target values and are called in-sample evaluation metrics, as they use training data.
What is polynomial regression? Polynomial regression is a form of regression analysis in which the relationship between the independent variable and the dependent variable is modeled as an nth degree polynomial. This allows more flexibility in the curve fitting.
What are pipelines in machine learning? Pipelines are a way to streamline machine learning workflows. They combine multiple steps (e.g., scaling, model training) into a single entity, making the process of building and evaluating models more efficient.
Machine Learning Classification Algorithms
What is the K-Nearest Neighbors algorithm and how does it work? The K-Nearest Neighbors algorithm (KNN) is a classification algorithm that uses labeled data points to learn how to label other points. It classifies new cases by looking at the ‘k’ nearest neighbors in the training data based on some sort of dissimilarity metric, the most popular label among neighbors is the predicted class for that data point. The choice of ‘k’ and the distance metric are important, and the dissimilarity measure depends on data type.
What are common evaluation metrics for classifiers? Common evaluation metrics for classifiers include Jaccard Index, F1 Score, and Log Loss. Jaccard Index measures similarity. F1 Score combines precision and recall. Log Loss is used to measure the performance of a probabilistic classifier like logistic regression.
What is a confusion matrix? A confusion matrix is used to evaluate the performance of a classification model. It shows the counts of true positives, true negatives, false positives, and false negatives. This helps evaluate where your model is making mistakes.
What are decision trees and how are they built? Decision trees use a tree-like structure with nodes representing decisions based on features and branches representing outcomes, they are constructed by partitioning the data by minimizing the impurity at each step based on the attribute with the highest information gain, which is the entropy of the tree before the split minus the weighted entropy of the tree after the split.
What is logistic regression and how does it work? Logistic regression is a machine learning algorithm used for classification. It models the probability of a sample belonging to a specific class using a sigmoid function, it returns a probability of the outcome being one and (1-p) of the outcome being zero, parameter values are trained to find parameters which produce accurate estimations.
What is the Support Vector Machine algorithm? A support vector machine (SVM) is a classification algorithm used for classification that works by transforming data into a high-dimensional space so that data can be categorized by drawing a separating hyperplane, the algorithm optimizes its output by maximizing the margin between classes and using data points closest to the hyperplane for learning, called support vectors.
A Data Science Career Guide
A career in data science is enticing due to the field’s recent growth, the abundance of electronic data, advancements in artificial intelligence, and its demonstrated business value [1]. The US Bureau of Labor Statistics projects a 35% growth rate in the field, with a median annual salary of around $103,000 [1].
What Data Scientists Do:
Data scientists use data to understand the world [1].
They investigate and explain problems [2].
They uncover insights and trends hiding behind data and translate data into stories to generate insights [1, 3].
They analyze structured and unstructured data from varied sources [4].
They clarify questions that organizations want answered and then determine what data is needed to solve the problem [4].
They use data analysis to add to the organization’s knowledge, revealing previously hidden opportunities [4].
They communicate results to stakeholders, often using data visualization [4].
They build machine learning and deep learning models using algorithms to solve business problems [5].
Essential Skills for Data Scientists:
Curiosity is essential to explore data and come up with meaningful questions [3, 4].
Argumentation helps explain findings and persuade others to adjust their ideas based on the new information [3].
Judgment guides a data scientist to start in the right direction [3].
Comfort and flexibility with analytics platforms and software [3].
Storytelling is key to communicating findings and insights [3, 4].
Technical Skills:Knowledge of programming languages like Python, R, and SQL [6, 7]. Python is widely used in data science [6, 7].
Familiarity with databases, particularly relational databases [8].
Understanding of statistical inference and distributions [8].
Ability to work with Big Data tools like Hadoop and Spark [2, 9].
Experience with data visualization tools and techniques [4, 9].
Soft Skills:Communication and presentation skills [5, 9].
Critical thinking and problem-solving abilities [5, 9].
Creative thinking skills [5].
Collaborative approach [5].
Educational Background and Training
A background in mathematics and statistics is beneficial [2].
Training in probability and statistics is necessary [2].
Knowledge of algebra and calculus is useful [2].
Comfort with computer science is helpful [3].
A degree in a quantitative field such as mathematics or statistics is a good starting point [4]
Career Paths and Opportunities:
Data science is relevant due to the abundance of available data, algorithms, and inexpensive tools [1].
Data scientists can work across many industries, including technology, healthcare, finance, transportation, and retail [1, 2].
There is a growing demand for data scientists in various fields [1, 9, 10].
Job opportunities can be found in large companies, small companies, and startups [10].
The field offers a range of roles, from entry-level to senior positions and leadership roles [10].
Career advancement can lead to specialization in areas like machine learning, management, or consulting [5].
Some possible job titles include data analyst, data engineer, research scientist, and machine learning engineer [5, 6].
How to Prepare for a Data Science Career:
Learn programming, especially Python [7, 11].
Study math, probability, and statistics [11].
Practice with databases and SQL [11].
Build a portfolio with projects to showcase skills [12].
Network both online and offline [13].
Research companies and industries you are interested in [14].
Develop strong communication and storytelling skills [3, 9].
Consider certifications to show proficiency [3, 9].
Challenges in the Field
Companies need to understand what they want from a data science team and hire accordingly [9].
It’s rare to find a “unicorn” candidate with all desired skills, so teams are built with diverse skills [8, 11].
Data scientists must stay updated with the latest technology and methods [9, 15].
Data professionals face technical, organizational, and cultural challenges when using generative AI models [15].
AI models need constant updating and adapting to changing data [15].
Data science is a process of using data to understand different things and the world, and involves validating hypotheses with data [1]. It is also the art of uncovering insights and using them to make strategic choices for companies [1]. With a blend of technical skills, curiosity, and the ability to communicate effectively, a career in data science offers diverse and rewarding opportunities [2, 11].
Data Science Skills and Generative AI
Data science requires a combination of technical and soft skills to be successful [1, 2].
Technical Skills
Programming languages such as Python, R, and SQL are essential [3, 4]. Python is widely used in the data science industry [4].
Database knowledge, particularly with relational databases [5].
Understanding of statistical concepts, probability, and statistical inference [2, 6-9].
Experience with machine learning algorithms [2, 3, 6].
Familiarity with Big Data tools like Hadoop and Spark, especially for managing and manipulating large datasets [2, 3, 7].
Ability to perform data mining, and data wrangling, including cleaning, transforming, and preparing data for analysis [3, 6, 9, 10].
Data visualization skills are important for effectively presenting findings [2, 3, 6, 11]. This includes using tools like Tableau, PowerBI, and R’s visualization packages [7, 10-12].
Knowledge of cloud computing, and cloud-based data management [3, 12].
Experience using libraries such as pandas, NumPy, SciPy and Matplotlib in Python, is useful for data analysis and machine learning [4].
Familiarity with tools like Jupyter Notebooks, RStudio, and GitHub are important for coding, collaboration and project sharing [3].
Soft Skills
Curiosity is essential for exploring data and asking meaningful questions [1, 2].
Critical thinking and problem-solving skills are needed to analyze and solve problems [2, 7, 9].
Communication and presentation skills are vital for explaining technical concepts and insights to both technical and non-technical audiences [1-3, 7, 9].
Storytelling skills are needed to translate data into compelling narratives [1, 2, 7].
Argumentation is essential for explaining findings [1, 2].
Collaboration skills are important, as data scientists often work with other professionals [7, 9].
Creative thinking skills allow data scientists to develop innovative approaches [9].
Good judgment to guide the direction of projects [1, 2].
Grit and tenacity to persevere through complex projects and challenges [12, 13].
Additional skills:
Business analysis is important to understand and analyze problems from a business perspective [13].
A methodical approach is needed for data gathering and analysis [1].
Comfort and flexibility with analytics platforms is also useful [1].
How Generative AI Can Help
Generative AI can assist data scientists in honing these skills [9]:
It can ease the learning process for statistics and math [9].
It can guide coding and help prepare code [9].
It can help data professionals with data preparation tasks such as cleaning, handling missing values, standardizing, normalizing, and structuring data for analysis [9, 14].
It can assist with the statistical analysis of data [9].
It can aid in understanding the applicability of different machine learning models [9].
Note: It is important to note that while these technical skills are important, it is not always necessary to be an expert in every area [13, 15]. A combination of technical knowledge and soft skills with a focus on continuous learning is ideal [9]. It is also valuable to gain experience by creating a portfolio with projects demonstrating these skills [12, 13].
A Comprehensive Guide to Data Science Tools
Data science utilizes a variety of tools to perform tasks such as data management, integration, visualization, model building, and deployment [1]. These tools can be categorized into several types, including data management tools, data integration and transformation tools, data visualization tools, model building and deployment tools, code and data asset management tools, development environments, and cloud-based tools [1-3].
Data Management Tools
Relational databases such as MySQL, PostgreSQL, Oracle Database, Microsoft SQL Server, and IBM Db2 [2, 4, 5]. These systems store data in a structured format with rows and columns, and use SQL to manage and retrieve the data [4].
NoSQL databases like MongoDB, Apache CouchDB, and Apache Cassandra are used to store semi-structured and unstructured data [2, 4].
File-based tools such as Hadoop File System (HDFS) and cloud file systems like Ceph [2].
Elasticsearch is used for storing and searching text data [2].
Data warehouses, data marts and data lakes are also important for data storage and retrieval [4].
Data Integration and Transformation Tools
ETL (Extract, Transform, Load) tools are used to extract data from various sources, transform it into a usable format, and load it into a data warehouse [1, 4].
Apache Airflow, Kubeflow, Apache Kafka, Apache NiFi, Apache Spark SQL, and Node-RED are open-source tools used for data integration and transformation [2].
Informatica PowerCenter and IBM InfoSphere DataStage are commercial tools used for ETL processes [5].
Data Refinery is a tool within IBM Watson Studio that enables data transformation using a spreadsheet-like interface [3, 5].
Data Visualization Tools
Tools that present data in graphical formats, such as charts, plots, maps, and animations [1].
Programming libraries like Pixie Dust for Python, which also has a user interface that helps with plotting [2].
Hue which can create visualizations from SQL queries [2].
Kibana, a data exploration and visualization web application [2].
Apache Superset is another web application used for data exploration and visualization [2].
Tableau, Microsoft Power BI, and IBM Cognos Analytics are commercial business intelligence (BI) tools used for creating visual reports and dashboards [3, 5].
Plotly Dash for building interactive dashboards [6].
R’s visualization packages such as ggplot, plotly, lattice, and leaflet [7].
Data Mirror is a cloud-based data visualization tool [3].
Model Building and Deployment Tools
Machine learning and deep learning libraries in Python such as TensorFlow, PyTorch, and scikit-learn [8, 9].
Apache PredictionIO and Seldon are open-source tools for model deployment [2].
MLeap is another tool to deploy Spark ML models [2].
TensorFlow Serving is used to deploy TensorFlow models [2].
SPSS Modeler and SAS Enterprise Miner are commercial data mining products [5].
IBM Watson Machine Learning and Google AI Platform Training are cloud-based services for training and deploying models [1, 3].
Code and Data Asset Management Tools
Git is the standard tool for code asset management, or version control, with platforms like GitHub, GitLab, and Bitbucket being popular for hosting repositories [2, 7, 10].
Apache Atlas, ODP Aeria, and Kylo are tools used for data asset management [2, 10].
Informatica Enterprise Data Governance and IBM provide tools for data asset management [5].
Development Environments
Jupyter Notebook is a web-based environment that supports multiple programming languages, and is popular among data scientists for combining code, visualizations, and narrative text [4, 10, 11]. Jupyter Lab is a more modern version of Jupyter Notebook [10].
RStudio is an integrated development environment (IDE) specifically for the R language [4, 7, 10].
Spyder is an IDE that attempts to mimic the functionality of RStudio, but for the Python world [10].
Apache Zeppelin provides an interface similar to Jupyter Notebooks but with integrated plotting capabilities [10].
IBM Watson Studio provides a collaborative environment for data science tasks, including tools for data pre-processing, model training, and deployment, and is available in cloud and desktop versions [1, 2, 5].
Visual tools like KNIME and Orange are also used [10].
Cloud-Based Tools
Cloud platforms such as IBM Watson Studio, Microsoft Azure Machine Learning, and H2O Driverless AI offer fully integrated environments for the entire data science life cycle [3].
Amazon Web Services (AWS), Google Cloud, and Microsoft Azure provide various services for data storage, processing, and machine learning [3, 12].
Cloud-based versions of existing open-source and commercial tools are widely available [3].
Programming Languages
Python is the most widely used language in data science due to its clear syntax, extensive libraries, and supportive community [8]. Libraries include pandas, NumPy, SciPy, Matplotlib, TensorFlow, PyTorch, and scikit-learn [8, 9].
R is specifically designed for statistical computing and data analysis [4, 7]. Packages such as dplyr, stringr, ggplot, and caret are widely used [7].
SQL is essential for managing and querying databases [4, 11].
Scala and Java are general purpose languages used in data science [9].
C++ is used to build high-performance libraries such as TensorFlow [9].
JavaScript can be used for data science with libraries such as tensorflow.js [9].
Julia is used for high performance numerical analysis [9].
Generative AI Tools
Generative AI tools are also being used for various tasks, including data augmentation, report generation, and model development [13].
SQL through AI converts natural language queries into SQL commands [12].
Tools such as DataRobot, AutoGluon, H2O Driverless AI, Amazon SageMaker Autopilot, and Google Vertex AI are used for automated machine learning (AutoML) [14].
Free tools such as AIO are also available for data analysis and visualization [14].
These tools support various aspects of data science, from data collection and preparation to model building and deployment. Data scientists often use a combination of these tools to complete their work.
Machine Learning Fundamentals
Machine learning is a subset of AI that uses computer algorithms to analyze data and make intelligent decisions based on what it has learned, without being explicitly programmed [1, 2]. Machine learning algorithms are trained with large sets of data, and they learn from examples rather than following rules-based algorithms [1]. This enables machines to solve problems on their own and make accurate predictions using the provided data [1].
Here are some key concepts related to machine learning:
Types of machine learning:Supervised learning is a type of machine learning where a human provides input data and correct outputs, and the model tries to identify relationships and dependencies between the input data and the correct output [3]. Supervised learning comprises two types of models:
Regression models are used to predict a numeric or real value [3].
Classification models are used to predict whether some information or data belongs to a category or class [3].
Unsupervised learning is a type of machine learning where the data is not labeled by a human, and the models must analyze the data and try to identify patterns and structure within the data based on its characteristics [3, 4]. Clustering models are an example of unsupervised learning [3].
Reinforcement learning is a type of learning where a model learns the best set of actions to take given its current environment to get the most rewards over time [3].
Deep learning is a specialized subset of machine learning that uses layered neural networks to simulate human decision-making [1, 2]. Deep learning algorithms can label and categorize information and identify patterns [1].
Neural networks (also called artificial neural networks) are collections of small computing units called neurons that take incoming data and learn to make decisions over time [1, 2].
Generative AI is a subset of AI that focuses on producing new data rather than just analyzing existing data [1, 5]. It allows machines to create content, including images, music, language, and computer code, mimicking creations by people [1, 5]. Generative AI can also create synthetic data that has similar properties as the real data, which is useful for training and testing models when there isn’t enough real data [1, 5].
Model training is the process by which a model learns patterns from data [3, 6].
Applications of Machine Learning
Machine learning is used in many fields and industries [7, 8]:
Predictive analytics is a common application of machine learning [2].
Recommendation systems, such as those used by Netflix or Amazon, are also a major application [2, 8].
Fraud detection is another key area [2]. Machine learning is used to determine whether a credit card charge is fraudulent in real time [2].
Machine learning is also used in the self-driving car industry to classify objects a car might encounter [7].
Cloud computing service providers like IBM and Amazon use machine learning to protect their services and prevent attacks [7].
Machine learning can be used to find trends and patterns in stock data [7].
Machine learning is used to help identify cancer using X-ray scans [7].
Machine learning is used in healthcare to predict whether a human cell is benign or malignant [8].
Machine learning can help determine proper medicine for patients [8].
Banks use machine learning to make decisions on loan applications and for customer segmentation [8].
Websites such as Youtube, Amazon, or Netflix use machine learning to develop recommendations for their customers [8].
How Data Scientists Use Machine Learning
Data scientists use machine learning algorithms to derive insights from data [2]. They use machine learning for predictive analytics, recommendations, and fraud detection [2]. Data scientists also use machine learning for the following tasks:
Data preparation: Machine learning models benefit from the standardization of data, and data scientists use machine learning to address outliers or different scales in data sets [4].
Model building: Machine learning is used to build models that can analyze data and make intelligent decisions [1, 3].
Model evaluation: Data scientists need to evaluate the performance of the trained models [9].
Model deployment: Data scientists deploy models to make them available to applications [10, 11].
Data augmentation: Generative AI, a subset of machine learning, is used to augment data sets when there is not enough real data [1, 5, 12].
Code generation: Generative AI can help data scientists generate software code for building analytic models [1, 5, 12].
Data exploration: Generative AI tools can explore data, uncover patterns and insights and assist with data visualization [1, 5].
Machine Learning Techniques
Several techniques are commonly used in machine learning [4, 13]:
Regression is a technique for predicting a continuous value, such as the price of a house [13].
Classification is a technique for predicting the class or category of a case [13].
Clustering is a technique that groups similar cases [4, 13].
Association is a technique for finding items that co-occur [13].
Anomaly detection is used to find unusual cases [13].
Sequence mining is used for predicting the next event [13].
Dimension reduction is used to reduce the size of data [13].
Recommendation systems associate people’s preferences with others who have similar tastes [13].
Support Vector Machines (SVM) are used for classification by finding a separator [14]. SVMs map data to a higher dimensional feature space so data points can be categorized [14].
Linear and Polynomial Models are used for regression [4, 15].
Tools and Libraries
Machine learning models are implemented using popular frameworks such as TensorFlow, PyTorch, and Keras [6]. These learning frameworks provide a Python API and support other languages such as C++ and Javascript [6]. Scikit-learn is a free machine learning library for the Python programming language that contains many classification, regression, and clustering algorithms [4].
The field of machine learning is constantly evolving, and data scientists are always learning about new techniques, algorithms and tools [16].
Generative AI: Applications and Challenges
Generative AI is a subset of artificial intelligence that focuses on producing new data rather than just analyzing existing data [1, 2]. It allows machines to create content, including images, music, language, computer code, and more, mimicking creations by people [1, 2].
How Generative AI Operates
Generative AI uses deep learning models like Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs) [1, 2]. These models learn patterns from large volumes of data and create new instances that replicate the underlying distributions of the original data [1, 2].
Applications of Generative AI Generative AI has a wide array of applications [1, 2]:
Natural Language Processing (NLP), such as OpenAI’s GPT-3, can generate human-like text, which is useful for content creation and chatbots [1, 2].
In healthcare, generative AI can synthesize medical images, aiding in the training of medical professionals [1, 2].
Generative AI can create unique and visually stunning artworks and generate endless creative visual compositions [1, 2].
Game developers use generative AI to generate realistic environments, characters, and game levels [1, 2].
In fashion, generative AI can design new styles and create personalized shopping recommendations [1, 2].
Generative AI can also be used for data augmentation by creating synthetic data with similar properties to real data [1, 2]. This is useful when there isn’t enough real data to train or test a model [1, 2].
Generative AI can be used to generate and test software code for constructing analytic models, which has the potential to revolutionize the field of analytics [2].
Generative AI can generate business insights and reports, and autonomously explore data to uncover hidden patterns and enhance decision-making [2].
Types of Generative AI Models
There are four common types of generative AI models [3]:
Generative Adversarial Networks (GANs) are known for their ability to create realistic and diverse data. They are versatile in generating complex data across multiple modalities like images, videos, and music. GANs are good at generating new images, editing existing ones, enhancing image quality, generating music, producing creative text, and augmenting data [3]. A notable example of a GAN architecture is StyleGAN, which is specifically designed for high-fidelity images of faces with diverse styles and attributes [3].
Variational Autoencoders (VAEs) discover the underlying patterns that govern data organization. They are good at uncovering the structure of data and can generate new samples that adhere to inherent patterns. VAEs are efficient, scalable, and good at anomaly detection. They can also compress data, perform collaborative filtering, and transform the style of one image into another [3]. An example of a VAE is VAEGAN, a hybrid model combining VAEs and GANs [3].
Autoregressive models are useful for handling sequential data like text and time series. They generate data one element at a time and are good at generating coherent text, converting text into natural-sounding speech, forecasting time series, and translating languages [3]. A prominent example of an autoregressive model is Generative Pre-trained Transformer (GPT), which can generate human-quality text, translate languages, and produce creative content [3].
Flow-based models are used to model the probability distribution of data, which allows for efficient sampling and generation. They are good at generating high-quality images and simulating synthetic data. Data scientists use flow-based models for anomaly detection and for estimating probability density function [3]. An example of a flow-based model is RealNVP, which generates high-quality images of human faces [3].
Generative AI in the Data Science Life Cycle
Generative AI is a transformative force in the data science life cycle, providing data scientists with tools to analyze data, uncover insights, and develop solutions [4]. The data science lifecycle consists of five phases [4]:
Problem definition and business understanding: Generative AI can help generate new ideas and solutions, simulate customer profiles to understand needs, and simulate market trends to assess opportunities and risks [4].
Data acquisition and preparation: Generative AI can fill in missing values in data sets, augment data by generating synthetic data, and detect anomalies [4].
Model development and training: Generative AI can perform feature engineering, explore hyperparameter combinations, and generate explanations of complex model predictions [4].
Model evaluation and refinement: Generative AI can generate adversarial or edge cases to test model robustness and can train a generative model to mimic model uncertainty [4].
Model deployment and monitoring: Generative AI can continuously monitor data, provide personalized experiences, and perform A/B testing to optimize performance [4].
Generative AI for Data Preparation and Querying Generative AI models are used for data preparation and querying tasks by:
Imputing missing values: VAEs can learn intricate patterns within the data and generate plausible values [5].
Detecting outliers: GANs can learn the boundaries of standard data distributions and identify outliers [5].
Reducing noise: Autoencoders can capture core information in data while discarding noise [5].
Data Translation: Neural machine translation (NMT) models can accurately translate text from one language to another, and can also perform text-to-speech and image-to-text translations [5].
Natural Language Querying: Large language models (LLMs) can interpret natural language queries and translate them into SQL statements [5].
Query Recommendations: Recurrent neural networks (RNNs) can capture the temporal relationship between queries, enabling them to predict the next query based on a user’s current query [5].
Query Optimization: Graph neural networks (GNNs) can represent data as a graph to understand connections between entities and identify the most efficient query execution plans [5].
Generative AI in Exploratory Data Analysis
Generative AI can also assist with exploratory data analysis (EDA) by [6]:
Generating descriptive statistics for numerical and categorical data.
Generating synthetic data to understand the distribution of a particular variable.
Modeling the joint distribution of two variables to reveal their potential correlation.
Reducing the dimensionality of data while preserving relationships between variables.
Enhancing feature engineering by generating new features that capture the structure of the data.
Identifying potential patterns and relationships in the data.
Generative AI for Model Development Generative AI can be used for model development by [6]:
Helping select the most appropriate model architecture.
Assessing the importance of different features.
Creating ensemble models by generating diverse representations of data.
Interpreting the predictions made by a model by generating representatives of the data.
Improving a model’s generalization ability and preventing overfitting.
Tools for Model Development
Several generative AI tools are used for model development [7]:
DataRobot is an AI platform that automates the building, deployment, and management of machine learning models [7].
AutoGluon is an open-source automated machine learning library that simplifies the development and deployment of machine learning models [7].
H2O Driverless AI is a cloud-based automated machine learning platform that supports automatic model building, deployment, and monitoring [7].
Amazon SageMaker Autopilot is a managed service that automates the process of building, training, and deploying machine learning models [7].
Google Vertex AI is a fully managed cloud-based machine learning platform [7].
ChatGPT and Google Bard can be used for AI-powered script generation to streamline the model building process [7].
Considerations and Challenges When using generative AI, there are several factors to consider, including data quality, model selection, and ethical implications [6, 8]:
The quality of training data is critical; bias in training data can lead to biased results [8].
The choice of model and training parameters determines how explainable the model output is [8].
There are ethical implications to consider, such as ensuring the models are used responsibly and do not contribute to malicious activities [8].
The lack of high quality labeled data, the difficulty of interpreting models, the computational expense of training large models, and the lack of standardization are technical challenges in using generative AI [9].
There are also organizational challenges, including copyright and intellectual property issues, the need for specialized skills, integrating models into existing systems, and measuring return on investment [9].
Cultural challenges include risk aversion, data sharing concerns, and issues related to trust and transparency [9].
In summary, generative AI is a powerful tool with a wide range of applications across various industries. It is used for data augmentation, data preparation, data querying, model development, and exploratory data analysis. However, it is important to be aware of the challenges and ethical considerations when using generative AI to ensure its responsible deployment.
Data Science Full Course – Complete Data Science Course | Data Science Full Course For Beginners IBM
Affiliate Disclosure: This blog may contain affiliate links, which means I may earn a small commission if you click on the link and make a purchase. This comes at no additional cost to you. I only recommend products or services that I believe will add value to my readers. Your support helps keep this blog running and allows me to continue providing you with quality content. Thank you for your support!
These documents function as a tutorial on data science in R, covering topics from the fundamentals of the R environment and data manipulation to advanced concepts in machine learning. The material explores visualizing data using various plotting techniques, including base graphics, ggplot2, and box plots, to gain insights into data distribution and relationships. Furthermore, it introduces regression models, specifically linear and logistic regression, explaining their mathematical basis and practical application in R for prediction. Finally, the sources discuss clustering algorithms, like hierarchical and k-means clustering, for grouping similar data points and touch upon time series analysis for understanding data trends over time, all while highlighting the essential skills and job roles within the data science field that often utilize R.
Podcast
Essential Data Science Skills and R Applications
R for Data Science Fundamentals
Based on the provided source, here is a discussion of R for data science:
R as a Programming Language for Data Science
R is described as a widely used programming language for data science. It is considered more than just a programming language; it is also a programming tool for performing analytics on data. R is an open-source and free software environment for statistical computing and graphics. It supports most machine learning algorithms for data analytics like regression, association, clustering, and more. While Python is noted as the main programming language in data science currently, R is considered powerful for doing a very quick display. Becoming proficient in R analytics can make transferring those skills to another language fairly easy, although R doesn’t have the same breadth of general code access as Python.
Key Features and Advantages of R
Several advantages of using R are highlighted:
Open Source: R is completely free and open source with active community members.
Extensible: It offers various statistical and graphical techniques.
Compatible: R is compatible across all platforms, including Linux, Windows, and Mac. Its compatibility is continually growing, integrating with systems like cluster computing and Python.
Extensive Library: R has an extensive library of packages for machine learning and data analysis. The Comprehensive R Archive Network (CRAN) hosts around 10,000 packages focused on data analytics.
Easy Integration: R can be easily integrated with popular software like Tableau, SQL Server, etc..
Diversity and Ease of Use: The diverse capabilities and extensive libraries make R a very diverse and easy-to-use coding source for analyzing data. It’s very easy and quick to go through and do different functions on the data and analyze it. R makes it easy to explore data.
R Environment: RStudio
RStudio is presented as a popular Integrated Development Environment (IDE) for R. It automatically opens up extra windows, which is nice. Typically, RStudio displays a console on the left (the main workspace), environmental information, and plots on the right. You can also use a script file in the upper left panel and execute the script, which runs in the console on the bottom left.
R Packages
Packages are essential in R as they provide pre-assembled collections of functions and objects. Each package is hosted on the CRAN repository. Not all packages are loaded by default, but they can be installed on demand using install.packages() and accessed using the library() function. Installing only necessary packages saves space.
Key packages mentioned for data science include:
dplyr: Used to transform and summarize tabular data. It’s described as much faster and easier to read than base R. Functions include grouping by data, summarizing, adding new variables (mutate), selecting columns (select), filtering data (filter), sorting (arrange), and sampling (sample_n, sample_fraction).
tidyr: Makes it easy to “tidy” data. It includes functions like gather (stacks multiple columns into a single column), spread (spreads single rows into multiple columns), separate (splits a single column into multiple), and unite (combines multiple columns). It’s also used for handling missing values, such as filling them.
ggplot2: Implements the grammar of graphics. It’s a powerful and flexible tool for creating sophisticated visualizations with little code. It’s part of the tidyverse ecosystem. You can build graphs by providing components like data, aesthetics (x, y axes), and geometric objects (geom). It uses sensible defaults if details aren’t provided. Different geom types are used for different graphs, e.g., geom_bar for bar charts, geom_point for scatter plots, geom_boxplot for box plots. You can customize elements like colors and sizes.
rpart: Used for partitioning data and creating decision trees.
rpart.plot: Helps in plotting decision trees created by rpart.
fSelector: Computes measures like Chi-squared, information gain, and entropy used in decision tree algorithms.
caret: A package for splitting data into training and test sets, used in machine learning workflows.
randomForest: The package for implementing the random forest algorithm.
e1071: A library containing support vector machine (SVM) functions.
dmwr: Contains the regress.eval function to compute error metrics like MAE, MSE, RMSE, and MAPE for regression models.
plotrix: Used for creating 3D pie charts.
caTools: Includes the sample.split function used for splitting data sets into training and test sets.
xlsx: Used to import data from Microsoft Excel spreadsheets.
elements.learn: Mentioned as a standard R library.
mass: A package containing data sets like the US serial data frame used for examples.
plot_ly: Creates interactive web-based graphs via a JavaScript library.
Data Structures in R
R supports various data structures, including vectors (the most basic), matrices, arrays, data frames, and lists. Vectors can contain numerous different values. Data frames are tabular data with rows and columns.
Data Import and Export
R can import data from various sources, including Excel, Minitab, CSV, table, and text files. Common functions for importing include read.table() for table files and read.csv() for CSV files, often specifying if the file has a header. Even if a file is saved as CSV, it might be separated by spaces or tabs, requiring adjustments in the read function. Exporting data is also straightforward using functions like write.table() or write.csv(). The xlsx package allows importing directly from .xlsx files.
Data Wrangling/Manipulation
Data wrangling is the process of transforming raw data into an appropriate format for analytics; it involves cleaning, structuring, and enriching data. This is often considered the least favorite but most time-consuming aspect of data science. The dplyr and tidyr packages are specifically designed for data manipulation and tidying. dplyr functions like filter for filtering data, select for choosing specific columns, mutate for adding new variables, and arrange for sorting are key for data transformation. Tidyr functions like gather, spread, separate, and unite help restructure data. Handling missing values, such as using functions from tidyr to fill NA values, is part of data wrangling.
Data Visualization
Data visualization in R is very powerful and quick. Visualizing data helps in understanding patterns. There are two types: exploratory (to understand the data yourself) and explanatory (to share understanding with others). R provides tools for both.
Types of graphics/systems in R:
Base graphics: Easiest to learn, used for simple plots like scatter plots using the plot() function.
Grid graphics: Powerful modules for building other tools.
Lattice graphics: General purpose system based on grid graphics.
ggplot2: Implements grammar of graphics, based on grid graphics. It’s a method of thinking about complex graphs in logical subunits.
Plot types supported in R include:
Bar chart (barplot(), geom_bar)
Pie chart (pie(), pi3d() from plotrix)
Histogram (hist(), geom_histogram)
Kernel density plots
Line chart
Box plot (boxplot(), geom_boxplot). These display data distribution based on minimum, quartiles, median, and maximum, and can show outliers. Box plots grouped by time periods can explore seasonality.
Heat map
Word cloud
Scatter plot (plot(), geom_point). These graph values of two variables (one on x, one on y) to assess their relationship.
Pairs plots (pairs()).
Visualizations can be viewed on screen or saved in various formats (pdf, png, jpeg, wmf, ps). They can also be copied and pasted into documents like Word or PowerPoint. Interactive plots can be created using the plot_ly library.
Machine Learning Algorithms in R
R supports various machine learning algorithms. The process often involves importing data, exploring/visualizing it, splitting it into training and test sets, applying the algorithm to the training data to build a model, predicting on the test data, and validating the model’s performance.
Linear Regression: A statistical analysis that attempts to show the linear relationship between two continuous variables. It creates a predictive model on data showing trends, often using the least square method. In R, the lm() function is used to create a linear regression model. It is used to predict a number (continuous variable). Examples include predicting rent based on area or revenue based on traffic sources (paid, organic, social). Model validation can use metrics like RMSE (Root Mean Squared Error), calculated from the square root of the mean of the squared differences between predicted and actual values. The regress.eval function in the dmwr package provides multiple error metrics.
Logistic Regression: A classification algorithm used when the dependent variable is categorical (e.g., yes/no, true/false). It uses a sigmoid function to model the probability of belonging to a class. A threshold (usually 50%) is used to classify outcomes based on the predicted probability. The college admission problem (predicting admission based on GPA and rank) is presented as a use case.
Decision Trees: A classification algorithm that splits data into nodes based on criteria like information gain (using algorithms like ID3). It has a root node, branch nodes, and leaf nodes (outcomes). R packages like rpart, rpart.plot, and fSelector are used. The process involves loading libraries, setting a working directory, importing data (potentially from Excel using xlsx), selecting relevant columns, splitting the data, creating the tree model using rpart, and visualizing it using rpart.plot. Accuracy can be evaluated using a confusion matrix. The survival prediction use case (survived/died on a ship based on features like sex, class, age) is discussed.
Random Forest: An ensemble method that builds multiple decision trees (a “forest”) and combines their outputs. It can be used for both classification and regression. Packages like randomForest are used in R. Steps include loading data, converting categorical variables to factors, splitting data, training the model with randomForest, plotting error rate vs. number of trees, and evaluating performance (e.g., confusion matrix). The wine quality prediction use case is used as an example.
Support Vector Machines (SVM): A classification algorithm used for separating data points into classes. The e1071 package in R contains SVM functions. This involves reading data, creating indicator variables for classes (e.g., -1 and 1), creating a data frame, plotting the data, and running the svm model. The horse/mule classification problem is a use case.
Clustering: Techniques used to group data points based on similarity. The process can involve importing data, creating scatter plots (pairs) to visualize potential clusters, normalizing the data so metrics aren’t biased by scale, calculating distances between data points (like Euclidean distance), and creating a dendrogram to visualize the clusters. The use case of clustering US states based on oil sales is provided.
Time Series Analysis: Analyzing data collected over time to identify patterns, seasonality, trends, etc.. This involves loading time-stamped data (like electricity consumption, wind/solar power production), creating data frames, using the date column as an index, visualizing the data (line plots, plots of log differences, rolling averages), exploring seasonality using box plots grouped by time periods (e.g., months), and handling missing values.
R in Data Science Skills and Roles
R is listed as an essential programming tool for performing analytics in data science. A data science engineer should have programming experience in R (or Python). While proficiency in one language is helpful, having a solid foundation in R and being well-rounded in another language (like Python, Java, C++) for general programming is recommended. Data scientists and data engineers often require knowledge of R, among other languages. The role of a data scientist includes performing predictive analysis and identifying trends and patterns. Data analytics managers also need to possess specialized knowledge, which might include R. The job market for data science is growing, and R is a relevant skill for various roles. Knowing R is beneficial even if you primarily use other tools like Python or Hadoop/Spark for quick data display or basic exploration.
Data Visualization Techniques in R
Data visualization is a core aspect of data science that involves the study and creation of visual representations of data. Its primary purpose is to leverage our highly developed ability to see patterns, enabling us to understand data better. By using graphical displays, such as algorithms, statistical graphs, plots, and information graphics, data visualization helps to communicate information clearly and effectively. For data scientists, being able to visualize models is very important for troubleshooting and understanding complex models. Mastering this skill is considered essential for a data scientist, as a picture is often worth a thousand words when communicating findings.
The sources describe two main types of data visualization:
Exploratory data visualization helps us to understand the data itself. The key is to keep all potentially relevant details together, and the objective is to help you see what is in your data and how much detail can be interpreted. This can involve plotting data before exploring it to get an idea of what to look for.
Explanatory visualization helps us to share our understanding with others. This requires making editorial decisions about which features to highlight for emphasis and which might be distracting or confusing to eliminate.
R is a widely used programming language for data science that includes powerful packages for data visualization. Various tools and packages are available in R to create data visualizations for both exploratory and explanatory analysis. These include:
Base graphics: This is the easiest type of graphics to learn in R. It can be used to generate simple plots, such as scatter plots.
Grid graphics: This is a powerful set of modules for building other tools. It has a steeper learning curve than base graphics but offers more power. Plots can be created using functions like pushViewport and rectangle.
Lattice graphics: This is a general-purpose system based on grid graphics.
ggplot2: This package implements the “grammar of graphics” and is based on grid graphics. It is part of the tidyverse ecosystem. ggplot2 enables users to create sophisticated visualizations with relatively little code using a method of thinking about and decomposing complex graphs into logical subunits. It requires installation and loading the library. Functions within ggplot2 often start with geom_, such as geom_bar for bar charts, geom_point for scatter plots, geom_boxplot for box plots, and geom_line for line charts.
plotly (plot ly): This library creates interactive web-based graphs via an open-source JavaScript graphing library. It also requires installation and loading the library.
plotrix: This is a package that can be used to create 3D pie charts.
R supports various types of graphics. Some widely used types of plots and graphs mentioned include:
Bar charts: Used to show comparisons across discrete categories. Rectangular bars represent the data, with the height proportional to the measured values. Stacked bar charts and dodged bar charts are also possible.
Pie charts: Used to display proportions, such as for different products and units sold.
Histograms: Used to look at the distribution and frequency of a single variable. They help in understanding the central tendency of the data. Data can be categorized into bins.
Kernel density plots.
Line charts: Used to show trends over time or sequences.
Box plots (also known as whisker diagrams): Display the distribution of data based on the five-number summary: minimum, first quartile, median, third quartile, and maximum. They are useful for exploring data with little work and can show outliers as single dots. Box plots can also be used to explore the seasonality of data by grouping data by time periods like year or month.
Heat maps.
Word clouds.
Scatter plots: Use points to graph the values of two different variables, one on the x-axis and one on the y-axis. They are mainly used to assess the relationship or lack of relationship between two variables. Scatter plots can be created using functions like plot or geom_point in ggplot2.
Dendrograms: A tree-like structure used to represent hierarchical clustering results.
Plots can be viewed on screen, saved in various formats (including pdf, png, jpeg, wmf, and ps), and customized according to specific graphic needs. They can also be copied and pasted into other files like Word or PowerPoint.
Specific examples of using plotting functions in R provided include:
Using the basic plot function with x and y values.
Using the boxplot function by providing the data.
Importing data and then graphing it using the plot function.
Using plot to summarize the relationship between variables in a data frame.
Creating a simple scatter plot using plot with xlab, ylab, and main arguments for labels and title.
Creating a simple pie chart using the pie function with data and labels.
Creating a histogram using the hist function with options for x-axis label, color, border, and limits.
Using plot to draw a scatter plot between specific columns of a data frame, such as ozone and wind from the airquality data set. Labels and titles can be added using xlab, ylab, and main.
Creating multiple box plots from a data frame.
Using ggplot with aesthetics (aes) to map variables to x and y axes, and then adding a geometry layer like geom_boxplot to create a box plot grouped by a categorical variable like cylinders. The coordinates can be flipped using coord_flip.
Creating scatter plots using ggplot with geom_point, and customizing color or size based on variables or factors.
Creating bar charts using ggplot with geom_bar and specifying the aesthetic for the x-axis. Stacked bar charts can be created using the fill aesthetic.
Using plotly to create plots, specifying data, x/y axes, and marker details.
Plotting predicted versus actual values after training a model.
Visualizing the relationship between predictor and response variables using a scatterplot, for example, speed and distance from the cars data set.
Visualizing a decision tree using rpart.plot after creating the tree with the rpart package.
Visualizing 2D decision boundaries for a classification dataset.
Plotting hierarchical clustering dendrograms using hclust and plot, and adding labels.
Analyzing time series data by creating line plots of consumption over time, customizing axis labels, limits, colors, and adding titles. Log values and differences of logs can also be plotted. Multiple plots can be displayed in a single window using the par function. Time series data can be narrowed down to a single year or shorter period for closer examination. Grid lines (horizontal and vertical) can be added to plots to aid interpretation, for example, showing consumption peaks during weekdays and drops on weekends. Box plots can be used to explore time series seasonality by grouping data by year or month. Legends can be added to plots using the legend function.
Overall, the sources emphasize that data visualization is a critical skill for data scientists, enabling them to explore, understand, and effectively communicate insights from data using a variety of graphical tools and techniques available in languages like R.
Key Machine Learning Algorithms for Data Science
Based on the sources, machine learning algorithms are fundamental techniques used in data science to enable computers to predict outcomes without being explicitly programmed. These algorithms are applied to data to identify patterns and build predictive models.
A standard process when working with machine learning algorithms involves preparing the data, often including splitting it into training and testing datasets. The model is trained using the training data, and then its performance is evaluated by running the test data through the model. Validating the model is crucial to see how well it performs on unseen data. Metrics like accuracy, RMSE (Root Mean Squared Error), MAE (Mean Absolute Error), MSE (Mean Squared Error), and MAPE are used for validation. Being able to visualize models and troubleshoot their code is also very important for data scientists. Knowledge of these techniques is useful for various data science job roles.
The sources discuss several specific machine learning algorithms and related techniques:
Linear Regression: This is a type of statistical analysis and machine learning algorithm primarily used for predicting continuous variables. It attempts to show the relationship between two variables, specifically modeling the relation between a dependent variable (y) and an independent variable (x). When there is a linear relationship between a continuous dependent variable and a continuous or discrete independent variable, linear regression is used. The model is often found using the least square method, which is the most commonly used method. Examples include predicting revenue based on website traffic or predicting rent based on area. In R, the lm function is used to generate a linear model.
Logistic Regression: Despite its name, logistic regression is a classification algorithm, not a continuous variable prediction algorithm. It is used when the response variable has only two outcomes (yes/no, true/false), making it a binary classifier. Instead of a straight line like linear regression, it uses a sigmoid function (sigmoid curve) as the line of best fit to model the probability of an outcome, which is always between zero and one. Applications include predicting whether a startup will be profitable or not, whether trees will get infested with bugs, or predicting college admission based on GPA and rank. In R, the glm (general linear model) function with the family=binomial argument is used for logistic regression.
Decision Trees: This is a tree-shaped algorithm used to determine a course of action and can solve both classification and regression problems. Each branch represents a possible decision, occurrence, or reaction. An internal node in the tree is a test that splits objects into different categories. The top node is the root node, and the final answers are represented by leaf nodes or terminal nodes. Key concepts include entropy, which measures the messiness or randomness of data, and information gain, which is used to calculate the tree splits. The ID3 algorithm is a common method for calculating decision trees. R packages like rpart and rpart.plot are used to create and visualize decision trees. Examples include predicting survival or classifying flower types.
Random Forests: This is an ensemble machine learning algorithm that operates by building multiple decision trees. It can be used for both classification and regression problems. For classification, the final output is the one given by the majority of its decision trees; for regression, it’s the majority output (implied average/aggregation of values). Random forests have various applications, including predicting fraudulent customers, diagnosing diseases, e-commerce recommendations, stock market trends, and weather prediction. Predicting the quality of wine is given as a use case. R packages like randomForest are used.
k-Nearest Neighbors (KNN): This is a machine learning technique mentioned as useful for certain job roles. It is described as grouping things together that look alike.
Naive Bayes: Mentioned as one of the diverse machine learning techniques that can be applied.
Time Series Analysis: While not a single algorithm, this involves techniques used for analyzing data measured at different points in time. Techniques include creating line plots to show trends over time, examining log values and differences of logs, and using box plots to explore seasonality by grouping data by time periods.
Clustering: This technique involves grouping data points together. It is useful for tasks like customer segmentation or social network analysis. Two main types are hierarchical clustering and partial clustering. Hierarchical clustering can be agglomerative (merging points into larger clusters) or divisive (splitting a whole into smaller clusters). It is often represented using a dendrogram, a tree-like structure showing the hierarchy of clusters. Partial clustering algorithms like k-means are also common. Calculating distances between points (like Euclidean or Manhattan distance) is a key step. Normalization of data is important for clustering to prevent bias from different scales. A use case is clustering US states based on oil sales.
Support Vector Machine (SVM): SVM is a machine learning algorithm primarily used for binary classification. It works by finding a decision boundary (a line in 2D, a plane in 3D, or a hyperplane in higher dimensions) that best separates the data points of two classes. The goal is to maximize the margin, which is the distance between the decision boundary and the nearest points from each class (called support vectors). If data is linearly separable, a linear SVM can be used. For data that is not linearly separable, kernel SVM uses kernel functions (like Gaussian RBF, sigmoid, or polynomial) to transform the data into a higher dimensional space where a linear separation becomes possible. Use cases include classifying cricket players as batsmen or bowlers or classifying horses and mules based on height and weight. Other applications include face detection, text categorization, image classification, and bioinformatics. The e1071 library in R provides SVM functions.
Overall, the sources highlight that a strong understanding of these algorithms and the ability to apply them, often using languages like R, is essential for data scientists.
Time Series Analysis: Concepts, Techniques, and Visualization
Based on the sources, Time series analysis is a data science technique used to analyze data where values are measured at different points in time,. It is listed among the widely used data science algorithms. The goal of time series analysis is to analyze and visualize this data to find important information or gather insights.
Time series data is typically uniformly spaced at a specific frequency, such as hourly weather measurements, daily website visit counts, or monthly sales totals. However, it can also be irregularly spaced and sporadic, like time-stamped data in computer system event logs or emergency call history.
A process for working with time series data involves using techniques such as time-based indexing, resampling, and rolling windows. Key steps include wrangling or cleaning the data, creating data frames, converting the date column to a date time format, and extracting time components like year, month, and day,,,,,. It’s also important to look at summary statistics for columns, check for and potentially handle missing values (NA), for example, by using forward fill,,,,. Accessing specific rows by date or index is also possible. The R programming language, often within the RStudio IDE, is used for this analysis,,. Packages like dplyr are helpful for data wrangling tasks like arranging, grouping, mutating, filtering, and selecting data,,,,.
Visualization is a crucial part of time series analysis, helping to understand patterns, seasonality, and trends,,,,. Various plotting methods and packages in R are used:
Line plots can show the full time series,,,.
The base R plot function allows for customizing the x and y axes, line type, width, color, limits, and adding titles,,,,. Using log values and differences of logs can sometimes reveal better patterns,.
It’s possible to display multiple plots in a single window using functions like par,,.
You can zoom into specific time periods, like plotting data for a single year or a few months, to investigate patterns at finer granularity,,,,,. Adding grids and vertical or horizontal lines can help dissect the data,,.
Box plots are particularly useful for exploring seasonality by grouping data by different time periods (yearly, monthly, or daily),,,,,,,,. They provide a visual display of the five-number summary (minimum, first quartile, median, third quartile, and maximum) and can show outliers,,.
Other visualization types like scatter plots, heat maps, and histograms can also be used for time series data.
Packages like ggplot2 and plotly are also available for creating sophisticated visualizations, although the plot function was highlighted as choosing good tick locations for time series,,,,,,,,. Legends can be added to plots to identify different series.
Analyzing time series data helps identify key characteristics:
Seasonality: Patterns that repeat at regular intervals, such as yearly, monthly, or weekly oscillations,,,,,,,,,. Box plots grouped by year or month clearly show this seasonality,,,. Weekly oscillations in consumption are also evident when zooming in,,,.
Trends: Slow, gradual variability in the data over time, in addition to higher frequency variations,,,. Rolling means (or rolling averages) are a technique used to visualize these trends by smoothing out higher frequency variations and seasonality over a defined window size (e.g., 7-day or 365-day rolling mean),,,,,,,. A 7-day rolling mean smooths weekly seasonality but keeps yearly seasonality, while a 365-day rolling mean shows the long-term trend,,. The zoo package in R is used for calculating rolling means.
Using an electricity consumption and production dataset as an example,, time series analysis revealed:
Electricity consumption shows weekly oscillations, typically higher on weekdays and lower on weekends,,,.
There’s a drastic decrease in consumption during early January and late December holidays,.
Both solar and wind power production show yearly seasonality,. Solar production is highest in summer and lowest in winter, while wind power production is highest in winter and drops in summer. There was an increasing trend in wind power production over the years.
The long-term trend in overall electricity consumption appeared relatively flat based on the 365-day rolling mean,.
Data Science Careers and Required Skills
Based on the sources, the field of data science offers a variety of career paths and requires a diverse skill set. Data scientists and related professionals play a crucial role in analyzing data to gain insights, identify patterns, and make predictions, which can help organizations make better decisions. The job market for data science is experiencing significant growth.
Here are some of the roles offered in data science, as mentioned in the sources:
Data Scientist: A data scientist performs predictive analysis and identifies trends and patterns to aid in decision-making. Their role involves understanding system challenges and proposing the best solutions. They repetitively apply diverse machine learning techniques to data to identify the best model. Companies like Apple, Adobe, Google, and Microsoft hire data scientists. The median base salary for a data scientist in the U.S. can range from $95,000 to $165,000, with an average base pay around $117,000 according to one source. “Data Scientist” is listed as the most common job title.
Machine Learning Engineer: This is one of the roles available in data science. Knowledge of machine learning techniques like supervised machine learning, decision trees, linear regression, and KNN is useful for this role.
Deep Learning Engineer: Another role mentioned within data science.
Data Engineer: Data engineers develop, construct, test, and maintain architectures such as databases and large-scale processing systems. They update existing systems with better versions of current technologies to improve database efficiency. Companies like Amazon, Spotify, and Facebook hire data engineers.
Data Analyst: A data analyst is responsible for tasks such as visualization, optimization, and processing large amounts of data. Companies like IBM, DHL, and HP hire data analysts.
Data Architect: Data architects ensure that data engineers have the best tools and systems to work with. They create blueprints for data management, emphasizing security measures. Companies hiring data architects include Visa, Logitech, and Coca-Cola.
Statistician: Statisticians create new methodologies for engineers to apply. Their role involves extracting and offering valuable reports from data clusters through statistical theories and data organization. Companies like LinkedIn, Pepsico, and Johnson & Johnson hire statisticians.
Database Administrator: Database administrators monitor, operate, and maintain databases, handle installation and configuration, define schemas, and train users. They ensure databases are available to all relevant users and are kept safe. Companies like Tableau, Twitter, and Reddit hire database administrators.
Data and Analytics Manager: This role involves improving business processes as an intermediary between business and IT. Managers oversee data science operations and assign duties to the team based on skills and expertise.
Business Analytics/Business Intelligence: This area involves specializing in a business domain and applying data analysis specifically to business operations. Roles include Business Intelligence Manager, Architect, Developer, Consultant, and Analyst. They act as a link between data engineers and management executives. Companies hiring in this area include Oracle, Uber, and Dell. Business intelligence roles are noted as having a high level of jobs.
To succeed in these data science careers, a strong skill set is necessary, encompassing both technical and non-technical abilities.
Key Technical Skills:
Programming Languages: Proficiency in languages like R and Python is essential. Other languages mentioned as useful include SAS, Java, C++, Perl, Ruby, MATLAB, SPSS, JavaScript, and HTML. R is noted for its strengths in statistical computing and graphics, supporting most machine learning algorithms for data analytics. Python is highlighted as a general-purpose language with libraries like NumPy and SciPy central to data science. Mastering at least one specific programming language is important.
SQL and Database Knowledge: A strong understanding of SQL (Structured Query Language) is considered mandatory for extracting large amounts of data from datasets. Knowledge of database concepts is fundamental. Various SQL forms exist, and a solid basic understanding is very important as it frequently comes up.
Big Data Technologies: Experience with big data, including technologies like Hadoop and Spark, is required. Hadoop sits on top of SQL and is used for creating huge clusters of data. Spark often sits on top of Hadoop for high-end processing.
Data Wrangling/Preparation: This is a process of transforming raw data into an appropriate format for analytics and is often considered the most time-consuming aspect. It involves cleaning (handling inconsistent data types, misspelled attributes, missing values, duplicates), structuring, and enriching data. Functions like arranging, grouping, mutating, filtering, and selecting data are part of this process. Techniques for handling missing values like forward fill are also used.
Machine Learning Algorithms: Knowledge of diverse machine learning techniques is crucial. This includes algorithms like Linear Regression (for continuous variables), Logistic Regression (a classification algorithm for binary outcomes), Decision Trees (for classification and regression), Random Forests (an ensemble method for classification and regression), k-Nearest Neighbors (KNN), Naive Bayes, Clustering (like hierarchical clustering and k-means), and Support Vector Machines (SVM) (often for binary classification). Applying these algorithms to data to identify patterns and build predictive models is core to data science.
Data Visualization: This involves creating visual representations of data using algorithms, statistical graphs, plots, and other tools to communicate information effectively. Being able to visualize models is important for troubleshooting. Various plots like line plots, bar charts, histograms, scatter plots, box plots, heat maps, pie charts, and dendrograms for clustering are used. Tools like Tableau, Power BI, and QlikView are used for creating reports and dashboards. R provides packages and functions for visualization, including base graphics, grid graphics, plot, and ggplot2.
Statistics: A data scientist needs to know statistics, which deals with collecting, analyzing, and interpreting data. Understanding probabilities, p-scores, f-scores, mean, median, mode, and standard deviation is necessary.
Model Validation: Evaluating the performance of models is crucial, using metrics like accuracy, RMSE, MAE, MSE, and MAPE.
Key Non-Technical Skills:
Intellectual Curiosity: This is highlighted as a highly important skill due to the rapidly changing nature of the field. It involves updating knowledge by reading content and books on data science trends.
Business Acumen/Intuition: Understanding how the problem solved can impact the business is essential. Knowing the company’s needs and where the analysis is going is crucial to avoid dead ends.
Communication Skills: The ability to clearly and fluently translate technical findings to non-technical teams is vital. Explaining complex concepts in simple terms is necessary when communicating with stakeholders and colleagues who may not have a data science background.
Versatile Problem Solver: Data science roles require strong analytical and quantitative skills.
Self-Starter: As the field is sometimes not well-defined within companies, data scientists need to be proactive in figuring out where to go and communicating that back to the team.
Teamwork: Data science professionals need to work well with others across the organization, including customers.
Ability to Visualize Models and Troubleshoot Code: This specific skill goes beyond just visualization for communication; it’s about breaking down and debugging complex models.
Career Outlook and Resume Tips:
The sources indicate significant growth in data science job listings.
For building a resume, key elements include a summary that ties your skills and experience to the specific company. Including links to professional profiles like LinkedIn and GitHub is important. The resume should be concise, ideally taking only about 30 seconds to a minute to glance over. Sections typically include experience, education, skills, and certifications. The order can be adjusted based on experience level and the specific job requirements. Highlighting experiences relevant to data science is advised. Remember to keep the resume simple, short, and direct.
R For Data Science Full Course Data Science With R Full Course Data Science Tutorial Simplilearn
Affiliate Disclosure: This blog may contain affiliate links, which means I may earn a small commission if you click on the link and make a purchase. This comes at no additional cost to you. I only recommend products or services that I believe will add value to my readers. Your support helps keep this blog running and allows me to continue providing you with quality content. Thank you for your support!