Tag: ml

From Data

Posted on by lechuck park

From Claude with some prompting
following the overall sequence from data collection to AI systems development.

  1. Data Collection and Processing (Upper “From Data” section): a) Collecting data from people worldwide b) “Get Data”: Acquiring raw data c) “Gathering Data”: Converting data into binary format d) “Statistics Analysis”: Performing data analysis e) “Making Rules/Formula”: Generating rules or formulas based on analysis
  2. Evolution of AI Systems (Lower “Human-made AI (Legacy)” section): a) Human-centered analysis:
    • “Combine formulas”: Combining rules and formulas directly created by humans
    b) Machine Learning-based analysis:
    • Rule-based Machine Learning: • Utilizes Big Data • Generates rules/formulas through machine learning • Results evaluated as “True or False”
    • Statistical Machine Learning (e.g., LLM): • Utilizes Big Data • Performs statistical analysis using advanced machine learning • Results evaluated as “Better or Worse”

Key Points Summary:

  1. Data Processing Flow: Illustrates the step-by-step process from raw data collection to rule/formula generation.
  2. AI System Evolution:
    • Begins with human-centered rule-based systems
    • Progresses to machine learning models that learn rules from data
    • Advances to sophisticated statistical models (like LLMs) that recognize complex patterns and provide nuanced results
  3. Shift in Result Interpretation:
    • Moves from simple true/false outcomes
    • To relative and context-dependent “better/worse” evaluations

This image effectively demonstrates the progression of data processing and AI technology, particularly highlighting how AI systems have become more complex and sophisticated. It shows the transition from human-derived rules to data-driven machine learning approaches, culminating in advanced statistical models that can handle nuanced analysis and produce more contextualized results.

A huge distinction

Posted on by lechuck park

From Claude with some prompting
Image Analysis: “A huge distinction”

  1. Human Data Processing:
    • “Feel a difference”: Humans start by recognizing basic differences.
    • “Make one diff (0 vs 1)”: Creating the most fundamental distinction.
    • “Found relations with many diff”: Discovering relationships among various differences.
    • “Make a formula”: Developing formulas based on discovered relationships.
    • “Make a Rules”: Establishing rules based on these formulas.
    • “Human-made All Data”: Accumulation of data created through this entire process.
  2. Process Recording:
    • “Logging all processes”: The data creation process itself is recorded as data.
  3. AI Data Processing:
    • “Sensing & Related Data”: AI receives basic sensory data and related information.
    • “Human-made All Data”: All human-created data serves as input for AI.
    • “Finding a Relations with huge diff”: AI analyzes relationships and differences within this vast dataset.
  4. Result:
    • AI icon: Represents the final derivation of insights through AI.

Evaluation:

  1. Insightfulness: The diagram effectively illustrates the fundamental difference between human data processing methods and AI’s data processing capabilities. It highlights how humans follow a step-by-step logical process, while AI can process large-scale data simultaneously.
  2. Process Continuity: The diagram well expresses how human data processing forms the foundation for AI learning. This conveys the important concept that AI develops based on human knowledge and experience.
  3. Importance of Data: By emphasizing the importance of digitalizing all processes, the diagram effectively represents the core of our modern data-centric society.
  4. Visualization Effectiveness: Complex concepts are represented using simple icons and arrows, making them easy to understand.
  5. Future Expectation: We look forward to seeing additional explanations about AI’s output results or its potential applications. This would enhance the comprehensiveness of the diagram, providing a fuller picture of the AI process from input to output and its real-world impact.

Overall, this image effectively contrasts human data processing with AI’s data analysis capabilities, showcasing how these two domains operate complementarily. It provides a clear visual representation of the journey from basic human distinctions to complex AI-driven insights, highlighting the ‘huge distinction’ in scale and capability between human and artificial intelligence processing.

Both are equally unexplainable

Posted on by lechuck park

From Claude with some prompting
This image compares human intelligence and artificial intelligence, emphasizing that both are “equally unexplainable” in certain aspects:

  1. Human Intelligence:
    • Uses 100% math and logic, but based on limited experience and data.
    • Labeled “Not 100% depend on Experience,” indicating experience alone is insufficient.
    • When decision-making under time constraints, humans make the “best choice” rather than a 100% perfect choice.
    • Shows a process of: Event → Decision with Time Limit → Action.
  2. Artificial Intelligence:
    • Based on big data, GPU/CPU processing, and AI models (including LLMs).
    • Labeled as “Unexplainable AI Model,” highlighting the difficulty in fully interpreting AI decision-making processes.
    • Demonstrates a flow of: Data input → Neural network processing → “Nice but not 100%” output.
    • Like human intelligence, AI also makes best choices within limited data and time constraints.
  3. Key Messages:
    • AI is not a simple logic calculator but a system mimicking human intelligence.
    • AI decisions, like human decisions, are not 100% perfect but the best choice under given conditions.
    • We should neither overestimate nor underestimate AI, but understand its limitations and possibilities in a balanced way.
    • Both human and artificial intelligence have unexplainable aspects, reflecting the complexity and limitations of both systems.

This image emphasizes the importance of accurately understanding and appropriately utilizing AI capabilities by comparing it with human intelligence. It reminds us that while AI is a powerful tool, human judgment and ethical considerations remain crucial. The comparison underscores that AI, like human intelligence, is making the best possible decisions based on available data and constraints, rather than providing infallible, 100% correct answers.

Finding Rules

Posted on by lechuck park

From Claude with some prompting
This image, titled “Finding Rules,” illustrates the contrast between two major learning paradigms:

  1. Traditional Human-Centric Learning Approach:
    • Represented by the upper yellow circle
    • “Human Works”: Learning through human language and numbers
    • Humans directly analyze data and create rules
    • Leads to programming and legacy AI systems
  2. Machine Learning (ML) Approach:
    • Represented by the lower pink circle
    • “Machine Works”: Learning through binary digits (0 and 1)
    • Based on big data
    • Uses machine/deep learning to automatically discover rules
    • “Finding Rules by Machines”: Machines directly uncover patterns and rules

The diagram showcases a paradigm shift:

  • Two coexisting methods in the process from input to output
  • Transition from human-generated rules to machine-discovered rules
  • Emphasis on data processing in the “Digital World”

Key components:

  • Input and Output: Marking the start and end of the process
  • Analysis: Central to both approaches
  • Rules: Now discoverable by both humans and machines
  • Programming & Legacy AI: Connected to the human-centric approach
  • Machine/Deep Learning: Core of the ML approach

This visualization effectively demonstrates the evolution in data analysis and rule discovery brought about by advancements in artificial intelligence and machine learning. It highlights the shift from converting data into human-readable formats for analysis to leveraging vast amounts of binary data for machine-driven rule discovery.

More abstracted Data & Bigger Error possibility

Posted on by lechuck park

From Claude with some prompting
This image illustrates the data processing, analysis, and machine learning application process, emphasizing how errors can be amplified at each stage:

  1. Data Flow:
    • Starts with RAW data.
    • Goes through multiple ETL (Extract, Transform, Load) processes, transforming into new forms of data (“NEW”) at each stage.
    • Time information is incorporated, developing into statistical data.
    • Finally, it’s processed through machine learning techniques, evolving into more sophisticated new data.
  2. Error Propagation and Amplification:
    • Each ETL stage is marked with a “WHAT {IF.}” and a red X, indicating the possibility of errors.
    • Errors occurring in early stages propagate through subsequent stages, with their impact growing progressively larger, as shown by the red arrows.
    • The large red X at the end emphasizes how small initial errors can have a significant impact on the final result.
  3. Key Implications:
    • As the data processing becomes more complex, the quality and accuracy of initial data become increasingly crucial.
    • Thorough validation and preparation for potential errors at each stage are necessary.
    • Particularly for data used in machine learning models, initial errors can be amplified, severely affecting model performance, thus requiring extra caution.

This image effectively conveys the importance of data quality management in data science and AI fields, and the need for systematic preparation against error propagation. It highlights that as data becomes more abstracted and processed, the potential impact of early errors grows, necessitating robust error mitigation strategies throughout the data pipeline.

“if then” by AI

Posted on by lechuck park

From Claude with some prompting
This image titled “IF THEN” by AI illustrates the evolution from traditional programming to modern AI approaches:

  1. Upper section – “Programming”: This represents the traditional method. Here, programmers collect data, analyze it, and explicitly write “if-then” rules. This process is labeled “Making Rules”.
    • Data collection → Analysis → Setting conditions (IF) → Defining actions (THEN)
  2. Lower section – “AI”: This shows the modern AI approach. It uses “Huge Data” to automatically learn patterns through machine learning algorithms.
    • Large-scale data → Machine Learning → AI model generation

Key differences:

  • Traditional method: Programmers explicitly define rules
  • AI method: Automatically learns patterns from data to create AI models that include basic “if-then” logic

The image effectively diagrams the shift in programming paradigms. It demonstrates how AI can process and learn from massive datasets to automatically generate logic that was previously manually defined by programmers.

This visualization succinctly captures how AI has transformed the approach to problem-solving in computer science, moving from explicit rule-based programming to data-driven, pattern-recognizing models.

Linux with ML

Posted on by lechuck park

From Claude with some prompting
This image illustrates the process of utilizing Machine Learning (ML) and AutoML techniques for system optimization in Linux.

It starts with collecting data through profiling techniques that gather statistics on CPU, memory, I/O, network resource usage, hardware counters, scheduling information, etc. Tracing is also employed to capture kernel/system/interrupt events and process call traces.

The collected data is then used to train machine learning models. This step requires analysis and verification by Linux system experts.

The trained models help determine optimal values, which are then applied to optimize various system components such as the scheduler, memory management, network traffic, and disk I/O. Optimization can also target security and automation aspects.

The eBPF (Enhanced Berkeley Packet Filter) sandbox, situated in the center, allows safe execution within the kernel, enabling eBPF programs to interact with the kernel.

Kernel modules provide another way to implement optimization logic and integrate it directly into the kernel.

Finally, kernel parameters can be tuned from user space to perform optimizations.

In summary, the image depicts an AutoML-based process that leverages data collection, machine learning modeling, deriving optimal values, eBPF, kernel modules, and parameter tuning to automate system optimization in Linux across various kernel subsystems like the scheduler, memory management, network, and disk I/O.C